Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models
Authors: HANWEN LIANG
Yuyang Yin
Dejia Xu
hanxue liang
Zhangyang Wang
Konstantinos N Plataniotis
Yao Zhao
Yunchao Wei
Venue: NeurIPS 2024 poster (2024)
Keywords: 4D generation
diffusion model
spatial-temporal consistency
Primary area: generative_models

Abstract: The availability of large-scale multimodal datasets and advancements in diffusion models have significantly accelerated progress in 4D content generation. Most prior approaches rely on multiple images or video diffusion models, utilizing score distillation sampling for optimization or generating pseudo novel views for direct supervision. However, these methods are hindered by slow optimization speeds and multi-view inconsistency issues. Spatial and temporal consistency in 4D geometry has been extensively explored respectively in 3D-aware diffusion models and traditional monocular video diffusion models. Building on this foundation, we propose a strategy to migrate the temporal consistency in video diffusion models to the spatial-temporal consistency required for 4D generation. Specifically, we present a novel framework, \textbf{Diffusion4D}, for efficient and scalable 4D content generation. Leveraging a meticulously curated dynamic 3D dataset, we develop a 4D-aware video diffusion model capable of synthesizing orbital views of dynamic 3D assets. To control the dynamic strength of these assets, we introduce a 3D-to-4D motion magnitude metric as guidance. Additionally, we propose a novel motion magnitude reconstruction loss and 3D-aware classifier-free guidance to refine the learning and generation of motion dynamics. After obtaining orbital views of the 4D asset, we perform explicit 4D construction with Gaussian splatting in a coarse-to-fine manner. Extensive experiments demonstrate that our method surpasses prior state-of-the-art techniques in terms of generation efficiency and 4D geometry consistency across various prompt modalities.

Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models
Hanwen Liang'; Yuyang Yin Dejia Xu', Hanxue Liang', Zhangyang Wang', Konstantinos N. Plataniotis' Yao Zhao?, Yunchao Wei 'University of Toronto, 'Beijing Jiaotong University, University of Texas at Austin, *University of Cambridge
Abstract
The availability of large-scale multimodal datasets and advancements in diffusion models have significantly accelerated progress in 4D content generation. Most prior approaches rely on multiple images or video diffusion models, utilizing score distillation sampling for optimization or generating pseudo novel views for direct supervision. However, these methods are hindered by slow optimization speeds and multi-view inconsistency issues. Spatial and temporal consistency in 4D geometry has been extensively explored respectively 3D-aware diffusion models and traditional monocular video diffusion models. Building on this foundation, we propose a strategy to migrate temporal consistency video diffusion models the auntial-temporal consistency required for 4D generation. Specifically, present a novel framework, Diffusion4D, for efficient and scalable 4D content generation. Leveraging meticulously curated dynamic 3D dataset, develop 4D-aware video diffusion model capable of synthesizing orbital views of dynamic 3D assets. To control the dynamic strength of these assets, introduce 3D- to-4D motion magnitude metric as guidance. Additionally, we propose a novel motion magnitude reconstruction loss and 3D-aware classifier-free guidance refine the learning and generation of motion dynamics. After obtaining orbital views of the 4D asset, perform explicit 4D construction with Gaussian splatting in a coarse-to- to-fee manner. Extensive experiments demonstrate that our method surpasses prior state-of-the-ard techniques terms of generation efficiency and 4D geometry consistency across various prompt modalities. Our project page is https://vita- htt.:opp.iittuboo//ooouuuuu
Introduction
The availability of internet-scale image-text-video datasets, coupled with progress in diffusion model techniques [37 32 31). has propelled significant advancements in generating diverse and high-quality
visual content, including images, videos, and 3D assets [23 13 have further fostered much progress in the realm of 4D content generation [47 16 I 30
19 34 48
2 These advancements
36 7 (52. which has gained widespread attention across both research and industrial domains. The ability to generate high-quality 4D content is key to various applications, ranging from the artistic realms of animation and film production the dynamic worlds of augmented reality.
However, generating high-quality 4D content efficiently and consistently remains a significant challenge. Due the scarcity of large-scale multi-view consistent 4D datasets, earlier 4D synthesis works 57 36 borrow appearance and motion priors from pretrained image- or video-diffusion models, and leverage score distillation sampling (SDS) (28) for optimization. This strategy is
Equua contribution. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prompt
3D-Awarre Diffusion
Spatial Consistency
View
4D Awware Diffusion
Spatial-Tempont Consistency
Video Diffusion
Temporal Consistency
Space Time
Time
Figure 1: Decomposition of spatial-temporal consistency in 4D generation. The proposed Diffusion4D embeds geometrical consistency and temporal coherence into a single network.
time-consuming and computationally inefficient due the heavy supervision back-propage proppppppagttion limiting its widespread applicability. [51 [4] explored video-conditioned 4D generation and utilized 3D-aware image diffusion models to acquire pseudo multi-view data for photometric supervision. Yet the issue of slow optimization speed persists. @ Another primary challenge in 4D synthesis to guarantee 4D geometry consistency, which can be decomposed into two components: spatial consistency and temporal consistency. As illustrated in Fig. [ spatial consistency implies that the object maintains consistent 3D geometry at each distinct timestamp, while temporal consistency indicates that the object's appearance and movement exhibit coherence and smoothness across timestamps. These two components have been thoroughly explored separately in static multi-view 3D synthesis [19]
Multi-view 3D diffusion models possess robust knowledge of geometrical consistency, whereas video diffusion models encapsulate temporal appearance coherence and smooth motion priors. Recent approaches (27 49) tried combining these two generative models to enhance efficiency and consistency in 4D generation. However, their underlying assumption of spatial-temporal conditional independence, the utilization of multi-model supervision, or inference at distinct timestamps inherently leads to 4D geometry inconsistencies. This raises the question: can we integrate spatial and temporal consistency into a single network, and obtain multi-timestamp cross-view supervision in one shot? Inspired by recent works in static 3D generation [1] 588 that repurposed temporal consistency in video generation to spatial consistency in 3D generation, we design a strategy that achieves spatial-temporal 4D consistency with a singular 4D-aware video diffusion model.
39
34 and monocular video generation [
35 4
To this end, we propose a novel framework, Diffuusio4DD to achieve efficient and consistent genera- tion of 4D content. First, to overcome the data scarcity issue, meticulously curate large-scale, high-quality 4D dataset from a vast corpus of 3D datasets 1O. 9. Utilizing the curated dataset, we develop a 4D-aware video diffusion model capable of generating orbital videos of dynamic 3D objects, imitating the photographing process of 4D assets. To the best of our knowledge, this is the first endeavor adapt a video diffusion model and train on 4D dataset for explicit novel view synthesis of 4D assets. To explicitly control the dynamic strength of the assets, we propose a novel 3D-to-4D motion magnitude metric and embed it as conditions into the diffusion network. We also incorporate the motion magnitude reconstruction loss to encourage the model to directly learn the object's 3D-to-4D dynamics. During the sampling stage, we propose a novel 3D-aware classifier-fre guidance to further enhance the dynamics of 3D assets. Leveraging the flexibility of the video diffusion model architecture, our framework can seamlessly accommodate various prompts, including text, single image, and static 3D content. It demonstrates the capability to generate highly spatial- temporal consistent orbital views around 4D assets, thus providing comprehensive information for the construction of 4D assets. Based on the generated orbital views of 4D assets, perform explicit 4D construction by optimizing Gaussian splatting representations with off-the-shell 4D construction pipelines [6 15]. We develop coarse-to to-fie optimization strategy with photometric losses. The synthesized multi-view consistent 4D image set enables us to swiftly generate high-fidelity and diverse 4D assets. In summary, contributions be summarized into three folds:
We present a novel 4D content generation framework that, for the first time, adapts video diffusion models for explicit synthesis of spatial-temporal consistent novel views of 4D assets. The 4D-aware video diffusion model can seamlessly integrate with the off-the-shelf modern 4D construction pipelines to efficiently create 4D content.

---

We introduce the 3D-to-4D motion magnitude metric to enable explicit control over the dynamic strength and propose the motion magnitude reconstruction loss and 3D-aware classifer-free guid- ance to refine the dynamics learning and generation.
Extensive experiments demonstrate that the proposed framework outperforms previous approaches in terms of generation efficiency and 4D geometry consistency, establishing new benchmark for 4D generation under different types of prompts, including text, single image, and static 3D content.
2 Background
4D generation aims at synthesizing dynamic 3D content from various inputs, such text, single images, monocular videos, or static 3D) assets. Extending 3D generation into the space-time domain, this task requires not only consistent geometry prediction but also the generation of temporally consistent dynamics. There are many works dedicated to this task. MAV3D[36) deals with text- conditioned 4D generation by utilizing SDS derived from video diffusion models to optimize dynamic NeRF representation. 4DFY tackles this problem by combining supervision signals from image, video, and 3D-aware diffusion models. Animate 124 [57] leverages both text and image prompts to improve the appearance of the dynamic 3D assets. DreamGaussian4D [0] adopts the deformable 3D Gaussians for 4D representations and tackles the image-conditioned generation with a driving video. Similarly, 4DGen [51] proposes utilizing spatial-temporal pseudo labels and obtains anchor frames with multi-view diffusion model. Consistent4DD [16] leverages the object-level 3D-aware image diffusion model for the primary supervision and proposes cascade DyNeRF to facilitate stable training. More recent work STAG4D (54) uses multi-view diffusion model to initialize multi-view images anchored on input video frames and introduces fusion strategy to improve the temporal consistency. However, the use of SDS loss in these works results in slow optimization speed and limits their applicability. To improve the efficiency, Efficient4D [27] generates
multi-view captures of dynamic 3D objects through SyneDreamer-T, while Diffusion" 2 49 leverages orthogonal diffusion models to sample dense views. Despite the high efficiency. the assumptions of spatial-tempont conditional independence, multiple model utilization, and tistsant- inference design inherently lead to 4D geometry inconsistencies. Unlike previous methods, our framework is the first to integrate 4D spatial-temporal consistency into a singular video diffusion model. More discussions about related works are in App. A.
3 Method
In this section, we provide a detailed illustration of the proposed framework, Diffusion4D, designed for the efficient and consistent generation of 4D content. We initiate by formulating the problem and outlining the key objectives in 3.1 Then, we introduce our data curation strategy pivotal in acquiring a large-scale, high-quality 4D dataset in 3.2 Subsequently, we delve into the methodology of applying the curated dataset to develop 4D-aware video diffusion models, capable of synthesizing orbital views of dynamic 3D assets conditioning various forms of prompts in 3.3 Finally, we introduce the explicit construction of 4D assets using 4D models based on the Gaussian Splitting representations in 3.4
3.1 Problem Setting and Key Objectives
Formally, given a prompt y, we aim to generate an orbital video V= (7, E R#XW |iz around a dynamic 3D asset. This video comprises T multi-view images captured at T consecutive timestamps {- along predefined camera pose trajectory, where H and W are the height and width dimensions of images. To simplify the problem, we put constraints on the camera pose trajectory, assuming that the camera always looks at the center of an object (origin of the world coordinates) with fixed elevation angle and camera distance. Thus, the viewpoint can be uniquely specified by the azimuth angle. The azimuth angle uniformly increases from 0 to 360 degrees along T timestamps, constructing a complete orbital video. We aim to generate this orbital video with a 4D-aware video diffusion model that can iteratively denoise samples from the learned conditional distribution p(Vly). where y can be text, single images, or static 3D content. At the end, explicit 4D construction performed with generated orbital video V as supervisions.
3

---

CLIP
"A Mc,oSE bows its head"
Latent embedding
-
Hunder
Beskping
Zt
Z0
V
Denoising U-Nel.
A Orbital Views
3D-to-4D Motion Magnitude
Embrdding
Prompt
Embedding
Addition
ApatialAttn
Temporal
View
Timestamp
4D-Aware Video Diffusion Model
4D Construction
Figure 2: Our proposed Diffusion4D consists of a 4D-aware video diffusion model and explicit 4D construction, capable of synthesizing 4D assets conditioned on text, single images, static 3D assets.
3.2 Data Curation
The development of the proposed 4D-aware video diffusion model requires a substantial amount of high-quality 4D assets. We curate large-scale, high-quality dynamic 3D dataset sourced from the vast 3D data corpus of Objaverse- 1.0 [10] and Objaverse-XX XL [9]. Since the original Objaverse dataset primarily consists of static 3D assets and many of them are of low quality (e.g., partial seans, missing textures) (3), we applied a series of empirical rules to filter the dataset. The curation process includes an initial selection with dynamic label, removal of assets with subtle overly dramatic motion, and out-of-boundary detection. In the initial step, we select 3D) assets labeled "dynamic". However, upon closer examination, imperceptible movement, limiting their utility in learning dynamic properties in the proposed task. To mitigate this problem, we employ Structural Similarity Index Measure (SSIM) evaluate the temporal dynamics of the assets. Specifically, for each 4D asset, we fix the camera pose at the front view and render three images at three distinct timestamps (To, TT/4, and Tr/2), then compute two SSIM scores between them. Assets with both SSIM scores higher than a predetermined threshold Shigh 0.95, suggesting high resemblance and little movement, are discarded. We also observe that many samples exhibit significant geometric distortion or drastic appearance changes over time. Therefore, we empirically set an SSIM score of small value Stow 0.4 as the lower bound to filter out cases of poor quality. Subsequently, for each remaining 4D asset, we render 7 24 multi-view images following the camera pose trajectory and timeline settings as we defined in Sec. 3.1 In the final step, we handle cases where assets exhibit over-dramatic movements that extend out of the boundaries of the scene. We employ alpha maps to identify and remove such cases and ensure that only appropriately positioned dynamic assets are included in the curated dataset, enhancing the overall quality and coherence of the generated content. Finally, we also conduct a manual inspection Objaverse- 1.0 [10] to remove low-quality cases that do not make sense semantically. This comprehensive strategy results in a total 81K high-quality animated assets. We released the IDs of the dynamic 3D assets in curated dataset.
observe that many assets exhibit subtle
3.3 4D-Aware Video Generation
Utilizing the curated dataset, we can render vast number of orbital videos of dynamic 3D assets to train a 4D-aware video diffusion model. For pretrained model selection, given that conventional video diffusion models designed for monocular video generation lack 3D geometry priors, resort to the recent works of 3D-aware video generation models 4T 58). In the following, we first introduce the pretrained 3D-aware video diffusion model. Then, describe how to adapt the model to our 4D-aware orbital video generation task. At this point, we focus on explicitly specifying the motion magnitude guidance, directly learning the 3D-to-4D dynamics in the training stage, and further augmenting the 3D object dynamics in the inference stage. Finally, we introduce how we utilize the curated dataset and customize the model architecture accommodate various condition modalities.
Pretrained 3D-aware video diffusion models. The main idea of 3D-aware video diffusion mod- els [1 58 is to repurpose the temporal consistency in video generation models for the spatial 3D

---

consistency of static 3D objects. Capitalizing on pretrained video diffusion models B 42 |6] these models are finetuned on large-scale datasets of high-quality 3D assets. It facilitates them to generate smooth and consistent orbital views of static 3D objects with user-specified camera pose conditions. In our method, we inherit the 3D geometry consistency in these models and extend it to modeling the spatial-temporal consistency in orbital views of 4D objects.
Vanilla 4D- video difffusinn models. Modern video diffusion models typically carry out diffusion and denoising processes in latent space 53 In our specific task, given an orbital video V around a 4D asset, which is rendered from curated dataset following the camera position trajectory introduced in Sec. 3.1 we first use a pretrained encoder to encode images into compressed latent 20 x xxx The h, w. c respectively denote the height, width, and channel dimension of the latent representation. The diffusion forward process samples time step t and adds noise e, to source input and obtain zt- A denoising network 60. parameterized by trained to predict the added noise conditioned on y, with a noise prediction loss Lidm = ||E: - eo (= y, e) In the inference stage, as shown in Fig. [ given an initial random gaussian noise and prompt condition y. the denoising network predicts the added noise to iteratively denoise the latent embedding. denoised latent &g is finally obtained, which is decoded via a pretrained decoder to recover high- sighffideitty orbital video.
3D-to-4D motion magnitude guidance and reconstruction. For the 4D-aware video generation, one of the most interesting attributes is the dynamic strength of the 3D assets. In an effort to enhance the control over the 31D-to-4D dynamic strength, we introduce the motion magnitude guidance into the diffusion model. To begin with, we need to quantitatively measure the motion magnitude of the moving objects. In contrast to the previous video models [8] that measure the motion magnitude by computing the inter-frame differences, the appearance differences across frames in V originate from both camera-view changes and object movement. To remove the influence from the camera-view
) changes, for each dynamic 3D object, we also render orbital video V {4 x Jo=1 around the static 3D asset, consisting of T multi-view images captured at the same camera poses as V at timestamp TO. Consecutively, we propose metric named 3D-to-4D motion magnitude m measuring the dynamic strength:
1 T Σ 14-440 44.
m(v)
Motivated by previous works (58), we use a two-layer multi-layer perception (MLP) to extract motion magnitude embedding, which is combined with the time embedding and injected into the denosing network. The noise prediction function by the denoising network accordingly changes to c0 y. m, t). This design allows the user to explicitly control the motion magnitude in inference stage. The impact of the motion magnitude guidance on the generation results is illustrated in Fig. [5]
In the training phase, to encourage the denoising network to learn the 3D-to-4D dynamics, we incorporate the motion magnitude reconstruction loss. This loss facilitates the direct learning of dynamic strength with explicit supervision over motion magnitude from 3D to 4D dynamics. For the sake of computation cost, we apply the reconstruction loss in the latent space and it is formulated as:
1 C = llm(2)) - m( m(z)) = + =0 - 20112.
(2)
where åg denotes the estimated clean video latent during training, the Eo denotes the compressed latent of V by the pretrained encoder. Our diffusion model training loss is formulated by combining latent diffusion loss and motion magnitude reconstruction loss with weight w: L = Lidm + wh
3D-aware classifier-free guidance. To further augment the dynamic strength of 3D objects, we draw inspiration from classifier-free guidance [14] and propose 3D-aware classifier- free guidance. It uses the pretrained 3D-aware video diffusion model, formulated as Eg, to provide classifier-fres guidance during the inference stage. Combining the prompt condition y, motion magnitude guidance m, the unconditional noise prediction eø (zz, t) and 3D-aware noise prediction Eø (Zt, y, t). the denoising step is reformulated as:
(ze, ()) + w2(e0 St. Us u, 1) Eo zt, V, 1)) éo (zt, y, m, !) oo D. m. l) + wi (E0(2E, y, m, () co(=,,
(3)
where Eo (22, m, t) is joint noise prediction, and wi and we are the classfier-free guidance scalers.
5

---

Generation with various condition modalities. In the above formulations, we use as a general prompt condition. Thanks to the versatility of our high-quality 4D dataset and the flexibility of the 3DD-aware video diffusion model architecture, our framework can readily accommodate diverse prompt modalities, including text, single images, or static 3D contents. For the text condition, we obtain the text description of each dynamic 3D asset from the prior work (22), and the text embedding extracted by CLIP model is fed into the diffusion model via the cross-attention mechanism. You can refer to 42 for more details. For the image condition, we use the first view image To in V captured at timestamp To as the reference image. The image condition is injected into the diffusion model with both cross- cooss-attention mechanism and feature concatenation. Please refer to [56 B1 for more details. For static 3D content condition, we use V as the reference video. Similar to the image condition, the video features extracted by pretrained encoder and and fed into the diffusion model via feature concatenation. This versatility allows our framework to effectively respond to different condition modalities, facilitating seamless integration into wide range of applications and scenarios.
3.4 Coarse-to-Fine 4D Construction
The spatial-temporal consistent multi-view videos generated by our 4D-aware video diffusion model offer comprehensive information about the 3D geometry and motions of dynamic 3D assets. At this stage, we explicitly model the 4D assets with Gaussian splatting(GS) [8] owing to its explicit representation, powerful fitting capabilities, and efficient optimization with dense-view supervision. We resort to the of-h--a---e-rrtttt 4D construction pipelines, i.e. 4DGS [46) and SC-GS [15]. When we directly train the GS on the generated multi-view video V. we find that the model does not perform well in capturing the 3D geometry details. The visualization can be found in Fig. [ie).. Therefore, we augment the supervision signals and propose a coarse-to- to-fine construction strategy. In the coarse stage, given the first-view image To in V, we use the pretrained 3D-aware video diffusion model to generate an orbital-view video V of the static 3D object. As our 4D-aware video diffusion model is finetuned on this model, we observe high 3D geometry consistency between V and V. We merge them together to train the GS in the coarse training stage. Note that the two sets of images (V and V' are of the same orbital views at differrnt timestamps, V ranging from to 1, and V at 0 only. Actually, readily replace V with conditional signal V into
for static 3D content conditioned generation, the image set for coarse training. In the fine training stage, we use V only to further tune the GS to improve the spatial-temporal consistency. Thanks to the 4D consistency in our generated videos, we can achieve precise pixel-level matching across different views and timestamps. We follow [46) and use I. and Lipips (55) losses for optimization. To enforce geometry smoothness, we also involve depth smoothness loss as regularizer [5] 25). For 4DGS-based reconstruction [46]. the total loss is formulated as Cgz Ci + Aipipa Lipips + ddeepth Edepthh For SC-GS based reconstruction [15), Carap is also involved. The At1 Atpips Aarap, and Adepth are losses weights.
4
Experimentation
4.1 Experiment Setup
We employ Diffusion4D to generate 4D assets conditioned on multiple modalities of prompts, i.e. text, single image, and static 3D content. Leveraging the curated dataset, for each dynamic 3D asset, we render two 360-degree orbital videos V's with T = 24, one starting at the front view and the other starting at a random azimuth angle. We fix the elevation angle as 0 and the camera distance as 2-meter. The image resolution is chosen at 256 256 across the experiments. In preparation for static 3D content-conditioned generation, following the camera poses of each rendered video V. we also render two 360-degree orbital video V of each static 3D asset fixed at initial timestamp To. We use VideoMV (58) as the pretrained 3D-aware video diffusion model, which is trained on a large scale of static 3D assets of superior quality. Specifically, for rext-conditioned 4D-aware video generation, we adopt the text-to-3D video generator (38) that is based on the ModelScopeT22 [42 model architecture. For image and video conditions, we adopt the image-to- -o- video generator [58] that is finetuned from 12VGen-XXI XL [56). We train our 4D-aware video diffusion model for 6k iterations with a constant learning rate of 3 x 10-5, We use valid batch size of 128 and train on NVIDIA A 100 GPUs During the sampling stage, we use text, view images, front-view started orbital videos of static 3D assets as conditions. we use DDIM (7) sampling with sampling step 50, and W1 7.0 and wa = 0.5 in classifier-free guidance. In the 4D construction stage, for 4DGS,

---

Text-To-4D
A cartoon monkey in red hat raising hands
Batman with black cape running
4DFY
Animate124
4DFY
Animate 124
Diflusion41>
Diffusson4DD
Red-clod emitting energy
Iron
Man suit
4DFY
Animate 124
4DFY
Animate 124
D
Diflusion4DD
Image-To-Be To-4D
Reference
4DXGen
STAG4D
Reference
4Duie
STAG4D
Diffusion-I
Diffuusin&4D
Reference
4DGen
STAG4D
Reference
4DGen
STAG4D
Diflusion-D
DiffiusonnDD
Reference
4DGen
STAG4D
Reference
4DGen
STAG4D
D
DD
Figure 3: Qualitative comparisons between Diffusion4D and other baselines in Text-to-4D (upper) and Image-to-4D (lower) generation. For method, show five views from consecutive timestamps. ( results from 4D-aware video diffusion model).7

---

Reference
Image
conditioned
Static-3D
2
conditioned
Static-3D)
conditioned
Figure 4: Visualization of Static-3D conditioned Diffusion4D. The first row shows the conditions, and the rest shows the results. (* results from 4D-aware video diffusion model.)
optimized the GS representation for 5k iterations in the coarse stage and 2k iterations the fine stage. For SC-GS, we optimized the GS for 16k iterations in the coarse stage and 4k iterations in the fine stage. Following prior works 3D 51, by default, we use reconstructions from 4DGS for quantitative and qualitative comparisons with the baselines.
4.2 Metric
From the curated dataset, we leave out 20 cases as test set for evaluation. For quantitative assessment, we first use the CLIP [29] score to evaluate the semantic correlation between the prompts (reference) and synthesized images (target). In each case, depending on the condition modality, the text description front-view image serves as the reference. For the diffusion generation, we use all 24 images in orbital videos as targets, and for 4D construction, we uniformly render orbital views from constructed 4D assets targets (CLIP-O), We also measure using only front-view images as (argets(CLIP-F), To evaluate the appearance and texture quality and the spatial-temporal consistency of the synthesized images, also use the following metries, i.e. LPIPS [55]. PSNR, SSIM [45]. and FVD [40], to help assess image- and static 3D content-conditioned generation. For these evaluations, use ground truth images rendered from the dynamic 3D assets references. Images generated by the diffusion model or rendered from constructed 4D assets are used as targets. Metrics are computed between ground truth images and synthesized images from the same camera pose. The same procedure is applied to calculate the scores for baseline methods. We also conduct qualitative comparisons through user study involving 30 human evaluators from diverse backgrounds. Participants to specify their preference for rendered orbital videos around 4D assets based on four properties, following the approach in (I] 3D geometry consistency (3DC), appearance quality(AQ), motion (iF),, and text alignmen((TA). We report the percentage of users who prefer each method overall and based on each of the four properties.
asked
4.3 Main Results
Text-conditioned and image-conditioned 4D generation. We take 4DFY 1 and Animate 124 (57 baselines for text-conditioned generation, and 4DGen(51) and STAG4D 54 baselines for image-conditioned generation, due to their remarkable performance and adaptability to the task settings. In Fig. 3 we provide detailed comparison between our method and baselines with various prompts. For the baselines, we provide two views at starting and ending timestamps. For our method, we visualize the spatial-temporal renderings from 4D constructions at five consecutive timestamps in multiple views. We also show the generated multi-view images from our video diffusion models (denoted with *). Our efficient and elegant pipeline is capable of generating diverse and realistic 4D assets with satisfactory geometrical and temporal consistency. While the baselines also synthesize 4D assets, their results exhibit very limited or even invisible motion. contrast, our results show apparent motions of great fidelity. As highlighted by the contours in Fig. 3 our animations include cartoon characters and humans stepping forward, running or raising arms, birds flapping wings, and lights changing. Our method also provides more detailed appearances. The quantitative results in

---

Table 1: Quantitative comparison between our method with other baselines under different task settings. The human study includes 3D geometry consistency(DDC), appearance quality(AQ), motion fidelity(MF), text alignment(TA), and overall score. By default, Diffusion4D suggests results from 4D construction, and Diffusion4D* suggests results from our diffusion models.
Metricy
Hamun Preferaay
CLIP-O
time]
SSIM
PSNR
TVDL
Overal
Rtar-sp-4D
23h
29%
Asimue124
22%
DillaiinnD
8m
49%
DiffusionDD
0.82
0.69
Image-to-12
4DGen [
0.84
0.71
1h:30m
0.69
14.4
0.31
736.6
18%
22%
25%
225
STAG4D 54
0.56
0.72
2h.3tmm
0.76
15.2
0.27
675.4
15%
25%
2618
33%
24%
DiflusionD
0.89
0.75
8m
0J83
16.7
0.21
560.8
67%
53%
56.%
38%
54%
DiffusionIDD
0.90
0.50
0.82
16.8
0.19
490.2
to
DiffusionD
0.88
0.77
3m
0.82
16.8
0.19
546.7
Diflusion4D
0.91
0.81
0.83
17.2
0.18
482.4
1- 0.2
0.2
static
(a)
(b) m 5 fmer
(c) m 15 Eye
(d) m 15
(c) w/o coarse (I) w/o fine
(g) Full
Lyer +3d-ff
(1) 4D-aware video diffusion model
(2) 4D construction
Figure 5: Ablation study of different components of our proposed framework.
Tab. D demonstrate that our method outperforms previous approaches across all metrics and user preferences. Compared with the state-of-the-arr SDS-based methods, which involve sophisticated and time-consuming optimization, our method is much more efficient and produces more favorable results. Users showed a strong preference for Diffusion4D over other baselines, especially in 3D geometry consistency and motion fidelity. Also, as 4D construction is based on the images from the diffusion models, we can observe that diffusion outputs perform slightly better than rendered outputs.
Static 3D content-condditioned 4D generation. We visualized the results of static 3D content- conditioned generation in Fig. [ Our framework successfully activates static 3D assets and generates spatial-temporal consistent 4D assets. Due to the lack of direct baselines, we compare our image- conditioned 4D generation with 3D content-conditioned generation. The difference between these two settings is that the former relies only on a single image, whereas the latter has access to dense views of the 3D) assets. While both can generate realistic dynamic 3D) assets, the latter can generate more objects. As shown in the last two rows in Fig. Mario and anthropomorphic
3D geometry-consiiete mouse closely follows the appearance and geometry of the prompts, particularly from the side and back views. The quantitative results in Tab. Tindicate that the static 3D content-conditioned setting achieves the best performance among all tasks, thanks to the access to dense views of source assets.
4.4 Ablation Study
We conduct an analysis on the effect of various design components in our framework with the typical image-to-4) generation task, and the results are shown in Fig-5land Tab, [ In the 4D-aware diffusion model, we incorporated multiple features to enhance the dynamics of 3D assets. We added each feature incrementally to demonstrate their impact. Fig. (1), we can observe that training without motion magnitude reconstruction loss results in nearly invisible movement (a), while incorporating it introduces subtle motion (b). Increasing the motion magnitude guidance (m) augments the dynamics of the kid (c). The involvement of 3D-aware fassifier-frre guidance significantly augments the

---

Table Ablation study on the effect of proposed components on 4D construction. * denotes results from diffusion models.
Image-to-4D
CLIP-Ft
CLIP-O1
PSNR
LPIPS.
FVD1
w/o Cur
0.84
0.73
0.78
15.4
0.26
602.5
w/o Cer"
0.86
0.77
0.79
15.8
0.23
524.6
w/o coarse
0.79
0.70
0.72
14.5
0.32
651.2
stage
Full model 0.89
0.75
0.83
16.7
0.21
560.8
model"
16.8
dynamics of the kid (d). In the 4D construction stage, as you can observe in Fig. 5 (2), training without the coarse stage leads to incomplete 3D geometry (e). Comparing (f) and (g), we can observe that adding the fine stage enhances the appearance and texture details of the generated 4D assets. Results in Tab. Qindicate that the removal of each proposed component leads to a noticeable drop in metrics. In summary, the full model delivers the best results both quantitatively and qualitatively.
5 Limitation and Future work
While our proposed Diffusion4D framework demonstrates significant advancements in terms of efficiency and consistency for 4D content generation, acknowledge the following limitations to provide foundation for future work. First, we use a video size of 24 x 256 x 256. While this resolution provides good balance between quality and computational feasibility, higher resolution and longer temporal sequences could further enhance the detail and realism of 4D content generation. Image resolution could be improved using off-the-sheli super-resolution models. Second, though curated substantial amount of high-quality data, the dataset still lacks diversity. The asset textures simple, with actions restricted to basic movements such walking, jumping, and stretching arms. Although there a large numher of characters, each character appears only few times, performing limited range of actions. Last, in the 4D reconstruction stage, the performance is subject to the capacity of off-the-shelf 4D construction pipelines. In our settings, we empirically find that both of the chosen 4D construction pipelines work well in constructing consistent 4D assets from orbital views. Comparatively, SC-GS performs better than 4DGS from more novel views. Due to the limited views for supervision, the constructed 4D assets may meet floaters and blurry problems when the objects have large motions. In future research, we will try to expand the diversity and quality of the dynamic 3D dataset. We will focus on exploring 4D reconstruction improve the model's ability to interpret and leverage orbital-view supervisions, which will be beneficial for generating 4D assets with better consistency across novel views. We will put more effort into generating longer and higher-resolution 4D orbital videos to support more detailed 4D) construction, potentially broadening the applications for Diffusion4D.
Conclusion
In this work, we introduced Diffusion4D, efficient and scalable framework for spatial-temporal consistent 4D generation. Motivated by the prior explorations using video diffusion model to generate orbital views of static 3D assets, migrate the temporal consistency in the video diffusion model to
consistency in 4D generation. Leveraging a meticulously curated dynamic 3D dataset, we developed a 4D-aware video diffusion model capable of synthesizing orbital views of dynamic 3D assets. We incorporate 3D-to-4D motion magnitude guidance and the motion magnitude reconstruction loss enhance the dynamics learning and control. 3D-aware classifier-fres guidance is introduced to further augment the dynamic strength. We fulfill explicit 4D construction with coarse-to- learning strategy. Extensive experiments demonstrated that our method surpasses prior state-of- -hettrrrt techniques in terms of generation efficiency and 4D geometry consistency across various prompt modalities. This innovative approach not only addresses the limitations of previous methods but also sets new benchmark for 4D content generation.
7 Acknowledgment
This research was funded by the Fundamental Research Funds for the Central Universities (2024XKRC882).
10

---

References
[1] Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. arXiv preprint arXiv:2311. 2023.
[2] Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. arXiv preprint arXiv:2407. 12781, 2024.
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231..15127 2023.
[4] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhom, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563 22575,, 2023.
[5] Pierre Charbonnier, Laure Slanc-Feraude Gilles Aubert, and Michel Barlaud. Two determin- istic half-quadratic regularization algorithms for computed imaging. In Proceedings of 1st international conference on image processing, volume 2, pages 168-172. IEEE, 1994.
[6] Zhixiang Chi, Li Gu, Tao Zhong, Huan Liu, Yuanhao Yu, Konstantinos N Plataniotis, and Yang Wang. Adapting to distribution shift by visual domain prompt generation. arXiv preprint arXiv:2405.02797, 2024.
[7] Wen-Hsuan Chu, Lei Ke, and Katerina Fragkiadaki. Dreamscene4d: Dynamic multi-object scene generation from monocular videos. arXiv preprint 1rXiv:2405.022880 2024.
[8] Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Fine-grained open domain image animation with motion guidance. arXiv preprint wxXi:22311112886 2023.
[9] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024.
[10] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142-13153, 2023.
[11] Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang. and Ulrich Neumann. Gaussianflow: Splatting gaussian dynamics for 4d content creation. arXiv preprint arXiv:2403.1224 12365, 2024.
[12] Agrim Gupta, Lijun Yu. Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint vrXiv:2312.0666 2023.
[13] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210002303: 2022.
[14] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207. /2598 2022.
arXiv preprint
[15] Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang. Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4220 2024.
11

---

[16] Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, and Yao Yao. Consistent4d: Consistent 360 ('dee] dynamic object generation from monocular video. arXiv preprint arXiv:2311.02845. 2023.
[17] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprini M.Xiv:23055.22665 2023.
[18] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküüher, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph.,
2023.
[19] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: Zero-shot one image 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298-9998999 9309,,90,,000 2023.
[20] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview--conister images from a single-view image. arXiv preprint TrXiiv:2309.0355: 2023.
[21] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15000 2023.
[22] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. Advances in Neural Information Processing Systems, 36, 2024.
[23] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12. 2021.
[24] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.
[25) Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5480-5490, 2022.
[26] Zijie Pan, Jiachen Lu, Xiatian Zhu, and Li Zhang. Enhancing high-resolution 3d generation through pixel-wise gradient clipping. arXiv preprint vrXiv:2310.12474, 2023.
[27] Zijie Pan, Zeyu Yang. Xiatian Zhu, and Li Zhang. Fast dynamic 3d object generation from single-view video. arXiv preprint arXiv:2401.00 08742, 2024.
[28] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall, Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint (tt::2209.14888 2022.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 2021.
pages
[30] Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang. Ang Cao, Gang Zeng, and Ziwei Liu. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint arXiv:2312.17142. 2023.
[31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[32] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-223 10, 2023.
12

---

[33] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++ a single image to consistent multi-view diffusion base model. arXiv preprint 2rXiv:2310.1517 15/ 2023.
[34] Yichun Shi, Peng Wang. Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi- view diffusion for 3d generation. arXiv preprini arXiv:2308. /65/2, 2023.
[35] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXi::2209
2022.
[36] Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. arXiv preprint arXiv:2301. 2023.
[37] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[38] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054 2024.
[39] Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan. Mydiffusion++: A dense high-resolution multi-view diffusion model for single or sparse-view 3d object reconstruction. arXiv preprint arX1v:2402.12712, 2024.
[40] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
[41] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint arXiv:2403. 12008. 2024.
[42] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.066 2023.
[43] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201, 2023.
[44] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
[45] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
[46] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xic. Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.
[47] Dejia Xu, Hanwen Liang, Neel P Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N Plataniotis, and Zhangyang Wang. Comp4d: LIm-guides compositional 4d scene generation. arXiv preprint arXiv:2403. 16993, 2024.
[48] Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vah- dat. Camco: Camera-controllable 3d-consistent image-to-video generation. arXiv preprint 2065:22406.02509, 2024.
[49] Zeyu Yang, Zijie Pan, Chun Gu, and Li Zhang. Diffusion": Dynamic 3d content generation via score composition of orthogonal diffusion models. arXiv preprint 4rXiv:2404.02148, 2024.
13

---

[50] Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer. Fast generation from text 3d gaussian splatting with point cloud priors. arXiv preprint 1o.0023300.88529, 2023.
[51] Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arXiv:2312.17225, 2023.
[52] Heng Yu, Chaoyang Wang, Peiye Zhuang, Willi Menapace, Aliaksandr Siarohin, Junli Cao, Laszlo A Jeni, Sergey Tulyakov, and Hsin-Ying Ying Lee. 4real: Towards photorealistic 4d scene generation via video diffusion models. arXiv preprint arXiv:2406.07472. 2024.
[53] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 18456-18466 2023.
[54] Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao. Stag4d: Spatial-temporal anchored generative gaussians. arXiv preprint arXiv:2403. 14939, 2024.
[55] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[56] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311 11.3111.0444 04140,144555. 2023.
[57] Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. An- imate 124: Animating one image to 4d dynamic scene. arXiv preprint arXiv:2311.1400 2023.
[58] Qi Zuo, Xiaodong Gu, Lingteng Qiu, Yuan Dong, Zhengyi Zhao, Weihao Yuan, Rui Peng, Siyu Zhu, Zilong Dong, Liefeng Bo, et al. Videomv: Consistent multi view generation based on large video generative model. arXiv preprint arXiv:2403. 2024.
14

---

A Appendices
A.1 Related Work
A.1.1
3D Generation
3D generation aims to create static 3D content from text or images. DreamFusion (28), first introduces the score distillation sampling (SDS) loss optimize NeRF with diffusion models. However, The original form of SDS encounters challenges such as multi-face Janus issues and slow optimization. 26 33
Subsequent works 19 20
34
50 have attempted to refine the mechanism to address 21
20 39 focus on generating dense multi-view images with consistent 3D properties directly. They achieve this by training or fine-tuning 2D diffusion models on 3D datasets to better suit 3D generation tasks. Zero12) [19] generates a 2D image from an images from single input image. Some later works [3 34 explicitly generate fixed multi-view images in one diffusion pass. The generated images can be used for reconstructing 3D content. Some other works like Point-E [4] and Shap-E [17] train models to directly generate 3D point clouds meshes.
these issues. Another research direction [9
unseen view based on a single image. SyneDreamer (20) can generate multiview-consisse
A.1.2 Video Generation
Video generation has obtained significant advancements. Diffusion has been employed for video pre- diction tasks in recent years, which have achieved great levels of realism, diversity, and controllability. Among them, video LDM 4 applics the latent diffusion framework for video generation. SVD 3 follows the same architecture and inserts temporal convolution and attention layers after every spatial convolution and attention layer, achieving effective performance improvement. W.A.L.T [12 introduces window attention architecture for joint spatial and spatio-temporal generative modeling. Some recent works [1 588 have tried to leverage temporal consistency [ ] from video generation but are limited to static 3D generation tasks. Inspired by their success, our work focuses on utilizing spatial consistency from video diffusion models for 4D generation tasks.
A.1.3 4D Generation
Compared to 3D generation, 4D generation requires only predicting consistent geometry but also generating temporal-consistn line of research works predict 4D representations conditioned on single image and text description. MAV3D [36] deals with text-to-4D problem by utilizing score distillation sampling derived from video diffusion models to optimize dynamic NeRF based on textual prompts. Animate 124 [57] leverages a dynamic NeRF-based representation to tackle this problem. 4DFY [ addresses text-to- 4D synthesis by combining supervision signals from image, video and 3D- 3D-aware diffusion models. Dream PreamGaussianDD [30] adopts the deformable 3D Gaussian as 4D representations. 4DGen [51] proposes novel pipeline where they utilize spatial-temporal pseudo labels into anchor frames with multi-view diffusion model. Another line of work predicts dynamic objects from single-view videos. Consistent4D [6] leverages the object-level 3D-aware image diffusion model as the primary supervision signal for training Dynamic Neural Radiance Fields, and proposing cascade DyNeRF to facilitate stable training. More recent work STAG4D [54] utilizes a multi-view diffusion model to initialize multi-view images anchoring the input video frames and introduce fusion strategy to ensure temporal consistency. Efficient4D (27 generates multi-frame multivvew captures of dynamic 3D objects through SyncDreamer-T and reconstructs 4D representations with them.
dynamics. There are a few recent works dedicated to this task. One
A.2 Experimental Details about Baselines implementation
For the baselines, we followed the instructions from their public websites to do experiments. 4DFY is designed for text-to-4[) generation. We fed in the text prompts directly to get the generation results. Animate 124 [57] requires a pair of text and image prompts for generation. Therefore, we use the first front-view image in our generated video as the image prompt for generation. In the image-to-4D generation, both 4DGen [51] and STAG4D 54 propose to use pretrained video diffusion models to generate monocular videos as the prompt conditions. To boost their performance, we directly render ground images at the front view from the source dynamic 3D datasets and obtain the monocular dynamic videos as prompts.
15

---

NeurIPS Paper Checklist
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction both outline the challenges in 4D content generation. It then introduces novel framework, Diffusion4D, and highlights key innovations. These contributions are accurately detailed in the main body of the paper, where each component thoroughly explained, implemented, and evaluated.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discussed about the limitations and future works of this submission in APP.57 This includes the need for higher image resolution and longer temporal sequences to enhance visualization and detail 4D construction.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
16

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not include theoretical results our submission.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe our method in great details in Sec S.c experimental details in See 4.1 and App. A.2 the comprehensive evaluation metrics in Sec. 4.2 and the data curation
strategies in Sec. 3.2 and App. ??
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
17

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will release the code and the dynamic 3D assets idx of the dataset for reproduction of the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/gust wiwliccgggig///ee Swtee//uiiinsiiooooooooooooooooa
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the details in Sec. 4.1 and App. A.2 model parameters, dataset used, and metrics discussed.
All training details such as
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
18

---

Answer: [No].
Justification: In this particular literature, error bars and confidence intervals are not typically reported. It is clear what assumptions should be made on distributions. So do not report error bars
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide information about compute resources in Table [ Our 4D con- struction model is trained on A100 in minutes to the results. We also provide the details about cost training diffusion model in App. A.2
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported paper (e.g., preliminary or failed experiments that didn't 'tt make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The paper demonstrates high level of research integrity by providing detailed methodologies, transparent data curation processes, and thorough analyses. We also empha- size reproducibility by fully disclosing the information needed to replicate the experiments. The research adheres to all relevant legal and ethical standards regarding the use of data and research practices.
Guidelines:
19

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We disucss our positive societal impacts in Sec. Our work has a positive effect the generating area, helping artists and designers to create high-quality 4D content quickly. We do not find that there are direct negative societal impacts of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification: By providing detailed information about the curation process and the safe- guards implemented, the paper promotes transparency. This allows other researchers and practitioners to understand the steps taken to ensure ethical use and to replicate these safeguards in their own work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
20

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners the assets used in the paper are properly credited, and the license and terms of use explicitly mentioned and properly respected. The paper clearly credits the original source the 3D data corpus, and provides citations for all the models, methods, and algorithms it builds upon compares against.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwprspppeesteeeeeee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We did not provide any new assets alongside this paper. The paper describes in detail the curation process of a high-quality dynamic 3D dataset sourced from the Objaverse dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: For the user study, we provide the details of this part in Sec. 4.3
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did study human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "2VI62GuyuF", "reviewer": "Reviewer_G7WS", "summary": "This paper studies 4D object generation. Instead of using SDS, this paper proposes to use a video diffusion model for generating multi-view images first and then a 4D Gaussian Splatting representation is learned using the multi-view images. To facilitate the training of video diffusion model, this paper uses a filtered subset of Objaverse as pretraining data. It also adds an motion magnitude embedding to the diffusion model as extra condition for guiding the generation process. \n\nThe developed framework Diffusion4D is able to generate 4D objects from various condition: image, text, static-3D.  It's much faster than the previous SDS based method, and achieves better results on rendering metrics and human preference benchmark.", "questions": "- For different conditions such as text, image, and static-3D, are different diffusion model used for each input?\n- Limiting the camera with only 1 DOF (azimuth angle) seems to be unreasonable, especially during test time. For the generated 3D/4D content, there is no reason we only evaluate the rendering results at a fixed elevation and distance. \n- The backbone/pre-trained diffusion models are extremely important, and I'm wondering why current set of models (Sec.4.1) are selected, and what are the insights and benefits. \n- Why filtering out the objects with big motion in the dataset? These data are the challenging ones and should serve as the hard examples of the proposed methods. A good dataset should has long tail distribution of these hard examples. Removing them from the data might makes the established benchmark unfair.", "limitations": "yes, the authors talked about limitation in Appendix such as the limited spatial temporal resolutions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper has a few strengths:\n- It tries to solve the slow convergence problem of SDS, and proposes a solution based on multi-view images generation and Gaussian splatting re-training. \n- To address temporal consistency, this paper adopts a video diffusion model for generating the multi-view images.\n- A motion magnitude metric is developed as metric and also extra condition to the framework for improving the performance.\n- The proposed framework is able to generate 4D objects from various condition including image, text, static-3D.  \n- The proposed framework also achieves better results on generation metrics and human preference.\n- The authors curated a filtered subset from objaverse, which will be released to the community of 4D generation research.", "weaknesses": "In my opinion, I found below things in this paper can be improved:\n- Unclear and insufficient comparison to related work in Sec.2. 4D content generation is an active research field, and multiple recent work are mentioned in Sec.2. However, I found it's unclear 1) what are the differences between the proposed method to these related work, 2) why aren't all of them considered in the experiments? A comparison table that clearly listing the pros and cons between the proposed methods and all related work will greatly improve the reability. The authors might even consider to include some concurrent arxiv works such as L4GM/DreamGaussian4D/Gaussian4D/etc (not required though).\n- Limited visual quality of the results. When I check the results in both paper and supplementary material (video), I found the quality are limited in two ways: 1) the rotating videos show that most generated objects are still suffering from view/time inconsistency. 2) The generated objects lack local fine details and are usually of very simplified texture. Also the local details are not sharp and often looks blurred. The 256x256 resolution might be one factor of this. 3) In the figures of this paper, the differences between the proposed method and the baselines are also hard to see. Overall, the generated objects are of low resolution and lack fine details. The inconsistency is also obvious in videos.\n- The method section isn't very clear. Specifically\n  - In Eq.1 m takes as input a video, but in Eq.2 m takes as input one latent $z_0$. What does m do exactly? And how is the motion magnitude embedding extracted from m?\n  - In Eq.2, both m terms in $L_{mr}$ has the same offset $\\bar{z}_0$, isn't this loss equivalent to $\\frac{1}{T}||z_0-\\hat{z}_0||^2$? Why do we need to write in the current form where two squared terms inside a squared term?\n- Visually, what's the effect of the motion magnitude guidance term isn't clear. It's unknown to me which artifacts this term help to address and how it improves the overall results.\n- The entire Coarse-to-Fine 4D construction seems to be a model-specific thing, it's proposed to address the issue of 4D Gaussian Splatting (4DGS). Is this module really necessary if different 4D representation is used? Or what's the benefit of using 4DGS over other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer G7WS, we would like to thank you for taking the time to review our work and providing valuable feedback. We also thank you for recognizing the importance of our targeted issue, the effectiveness and versatility of our framework, and the value of our curated dataset. \n\nWe thoroughly considered your questions, and we are committed to addressing all your concerns as follows:\n\n**Q1.  Comparison to related works:**\n1) We have elaborated the motivations of our method and its distinctions from prior works in the 2nd paragraph of Sec.1 and L99, 102-106 in Sec.2. Also, in response to your request, we provide a clearer summarization of prior 4D generation works in global rebuttal-(3). Our approach differs from priors in terms of optimization and consistency enhancement strategies. The innovative design of 4D-aware video diffusion model enables us to swiftly generate spatial-temporal consistent 4D assets. Please refer to global rebuttal-(3) for more details.\n2) We compared our method with the most relevant, latest, and well-recognized works. For text-to-4D, we compared with 4dfy (CVPR2024) and animated124(arXiv, Nov.2023). For image-to-4D, we compared with Stag4D(ECCV2024) and 4DGen(arXiv, Dec.2023). Notably, Stag4D claimed better performance than DreamGaussian4D(arXiv, Dec.2023) in its Fig.5 of Sec 4.3. L4GM(arXiv, June.2024) focused on video-to-4D generation task and was released after NIPS submission deadline, making it impossible for us to include a comparison.\n\n**Q2.  Visual quality**\n1) For generation consistency, we quantitatively assessed the results with FVD and conducted qualitative comparisons via extensive user study. Results in Tab.1 in main.pdf comprehensively demonstrated our superiority over baselines in geometry consistency. We can observe from the supplementary videos that ‘Diffusion4D’ demonstrates consistent 3D geometry, the appearances and motions exhibiting coherence and smoothness across 360°. This can be attributed to the extensive spatial-temporal attentions in the video model architecture.\n2) We trained our model on objaverse dataset consisting of simple objects and this is the reason for the simplified texture. The 256² resolution was also adopted in prior works[1, 12, 27, 34, 48, 50, 52, 55]. With the same resolution, as shown in Fig.A in rebuttal.pdf, our method can generate assets with more complex textures, clearer appearance, and more consistent geometry compared with baselines. We appreciate your feedback and are committed to exploring more complex textures and higher resolutions in the future study. \n3) For qualitative comparison with baselines, we provide a detailed explanation in global rebuttal-(1) and (2). Please refer to global rebuttal-(1) (2) and Fig.A in rebuttal.pdf for more details.\n\n**Q3.  Method section details**\n1) _m_ is the proposed metric of 3D-to-4D motion magnitude and it can be evaluated in both pixel space (Eq.1) and latent space (Eq.2). In Eq.1, _m_ is computed by averaging the difference across T images $I_i$ in a video; In Eq.2, _m_ is also computed by averaging T image latents in _z_. These two operations are technically consistent so we use the same notation. We will revise the notations to avoid confusion. The motion magnitude is embedded in the same way as timestamp embedding. We project the motion magnitude into a positional embedding followed by two MLP layers.\n2) As _z_ is a matrix, the mathematical form in the paper is computationally different from the suggested form. We derive the $L_{mr}$ in this form to encourage the model to reconstruct the motion magnitude explicitly. We empirically found the proposed form leads to better learning of motion magnitude.\n\n**Q4.  Effect of motion magnitude guidance.** \nIn ablation study, comparing (b) and (c) in Fig.5, we can observe that increasing _m_ augments the dynamics of the kid. In Fig.B in rebuttal.pdf, we show more cases with varying motion magnitudes. We can observe that increasing _m_ augments the dynamics of the generated assets.\n\n**Q5.  Benefit of 4DGS.**\nMany previous works [37,46,49] have demonstrated that Gaussian Splatting benefits from explicit representation, better construction details, faster rendering, and less memory usage compared to NeRF. Therefore, to achieve fast and high-quality 4D generation, we select 4DGS for 4D construction.\n\n**Q6.  More questions**\n1) *Different conditions*. Yes, we trained different diffusion models for distinct input modalities. The flexibility of the video diffusion model architecture allows our framework to accommodate various prompt modalities with minor modification. Please refer to L196-L208 for more details. \n2) *Camera pose settings during training and evaluation*. Generating consistent 4D assets across 360-degree is a challenging task. Therefore, **in the training stage**, we mitigate the modeling burden on the video diffusion model by fixing elevation angle and distance, making the model focus on learning the motion and geometry changes along orbital views. We can observe the same training settings in prior works [27,52,56]. **In the test stage**, we also adhered to the evaluation settings in prior works [1, 12, 34, 50, 52, 55] and rendered from constructed 4D assets with a fixed elevation and distance.\n3) We selected VideoMV as it is a 3D-geometry-aware video diffusion model. We did extensive testing experiments and demonstrated that this model is stable and reliable in generating consistent orbital videos of static 3D assets.\n4) *Data curation details*. We want to clarify that we do not filter out ‘big-motion’ assets. In the final curation stage, we remove cases with dramatic location changes where the objects exit the scene boundaries (leaving only partial parts within the scene). Such instances would deteriorate the model in learning stable and reasonable motions. We will clarify this point in the revised version.\n\nWe are willing to reply to your further questions in this time window if you have any feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IZm8wzVD1n", "reviewer": "Reviewer_fabU", "summary": "The paper presents Diffusion4D, a novel method for generating 4D content that maintains spatial and temporal consistency. It improves upon previous approaches by embedding spatial-temporal consistency within a single video diffusion model, enabling efficient creation of 4D assets. The technique uses a curated dataset of high-quality dynamic 3D objects to train a 4D-aware video diffusion model, which can synthesize orbital views of moving 3D assets. The method outperforms existing techniques in terms of generation speed and 4D geometry consistency.", "questions": "1. The qualitative comparisons are not that convincing, the generated motions are quite small and easy, the authors are encouraged to provide more experimental results in the rebuttal.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow\n2. The proposed subset of objaverse and objaverse-xl is valuable for subsequent research\n3. The quantitative experiments presented in the main text demonstrate the superiority of the proposed method.", "weaknesses": "1. The qualitative comparisons are not that convincing, the generated motions are quite small and easy, the authors are encouraged to provide more experimental results in the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fabU, thank you very much for taking the time to review our work and providing constructive feedback. We also thank you for recognizing our paper with **novel method, valuable data contribution for subsequent research, good writing, and superior quantitative results**. We would like to take this opportunity to address your comments regarding qualitative comparisons.\n\n**Provide more qualitative comparisons**\n**1) Comparisons with baselines.** Please refer to Fig.3 in main.pdf and Fig. A in rebuttal.pdf for qualitative comparisons between our method and the baselines.\n- **Larger motion magnitude.** In Fig.3, due to space constraints, we only showed two viewpoints at starting and ending timestamps as representatives for the baselines. It is evident that the objects’ poses at two timestamps are almost identical, as the baselines generated samples with almost invisible motions along the whole timeline. In contrast, the motions from our method are much more obvious and pronounced. Also in Fig.A in rebuttal.pdf, we show more cases with large motions, including the pose change of a knight, the flapping of butterfly wings, a turtle swimming, a kid falling, and the arm movements of a cartoon dog and Mario. Our method is capable of generating 4D assets with very much larger motions.\n- **More complex texture, smoother appearance, and more consistent geometry.** For the generation details, by zooming in, we can observe that the baselines suffer from blurry texture, noisy appearances, excessive artifacts, and incomplete geometry. On the other hand, our method provides more complex texture, smoother appearances, and more consistent geometry. These phenomena can also be observed in both Fig.3 in main.pdf and Fig.A in rebuttal.pdf.\n\n**2) Consistent novel views.**\n- In Fig.A in rebuttal.pdf, in addition to orbital views with elevation angle of 0°, we also provide more novel views by rendering images from the constructed 4D assets. We show orbital views with elevation angle of 30° and front view images along the timeline. The motions are much more apparent by fixing the viewpoint at the front view. We can observe that the constructed 4D assets demonstrate large motions, coherent appearance, and consistent geometries across timestamps and viewpoints.\n\nWe believe that these demonstrations, along with the visualizations in the main submission, could provide a well-rounded view of our generation results and successfully address your concerns. And we are pleased to offer more video demonstrations if policy allows. Thank you again!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wt6rEI604q", "reviewer": "Reviewer_BwxH", "summary": "The paper tackles 4D asset generation. Its pipeline consists of two parts: a 4D-aware video diffusion model, where orbital videos circling the 4D assets can be seamlessly generated, and a coarse-to-fine 4D construction stage afterward. The video diffusion model is trained on a novel dynamic dataset curated by the authors from Objaverse. The method can generate pretty high-quality 4D assets while being quite efficient compared with SDS-based methods.", "questions": "N/A", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The task of 4D asset generation is extremely challenging and well-motivated.\n2. The paper is nicely written, and ablation studies were done to justify its design choices.\n3. Evaluations were done with reasonable metrics and against SOTA methods.\n4. Tremendous engineering efforts can be observed from all the design choices, such as the motion magnitude conditioning and guidance, details about the 4D construction, etc.", "weaknesses": "1. The effectiveness of the Diffusion4D model heavily relies on a meticulously curated high-quality 4D dataset. The dependency on large-scale, high-quality datasets might limit its applicability in scenarios where such data is scarce or hard to generate.\n2. The current resolution and temporal sequence length significantly limit the quality and realism of the generated 4D content --- I could barely tell whether there is motion in many samples until after looking at them several times.\n3. The introduction of a novel 3D-to-4D motion magnitude metric and the corresponding reconstruction loss is innovative; however, the paper could explore in more detail how sensitive the model’s performance is to the tuning of these parameters. It’s also not clear how much the performance depends on the exact settings of these loss functions, which could be critical for users trying to replicate or build upon this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer BwxH, \nWe would like to extend our great thanks for your time and effort in reviewing our work and providing insightful comments. We thank you for recognizing the values and contributions of our work, noting that **the paper is nicely written, the solution to the challenging task is well-motivated, the experimental evaluations and ablation studies are comprehensive and convincing, and the generated assets are of pretty high quality. We are also grateful for your acknowledgment of our tremendous engineering efforts in high-quality data curation and experimental design.** \n\nWe have carefully considered your feedback and would like to address your comments as follows:\n\n**Q1. Dependency on 4D dataset.**\nWe acknowledge that the success of our method relies on a curated high-quality dataset. To the best of our knowledge, our work is the first that trains a singular video diffusion model directly on 4D datasets for explicit synthesis of spatial-temporal consistent views of 4D assets. We adopt a data-driven optimization strategy that reduces the generation time from a couple of hours to several minutes. The data scarcity is an issue in this research field. To mitigate this problem, we made significant efforts to curate a high-quality 4D dataset from Objaverse. We believe that this dataset can make contributions to future research in this community.\n\n**Q2. Resolutions, temporal length, and large motion.**\n1) *Resolution and temporal length*. We followed prior works [1, 12, 27, 34, 50, 52, 55] and chose a resolution of 256², considering GPU memory. With the same resolution, as shown by more examples in Fig.A of rebuttal.pdf, our method can generate assets with **larger motion, more complex textures, clearer appearance, and more consistent geometry** compared with baselines. We have adopted a longer temporal length of 24, compared to 16 in most prior works. We appreciate your feedback and are committed to exploring higher resolutions in the future study. \n2) *Large motion*. Please refer to Fig.3 in main.pdf and Fig.A in rebuttal.pdf for more demonstrations of our method and the baselines.\n- In Fig.3 in main.pdf, as highlighted by the contours, our animations include cartoon characters and humans stepping forward, running or raising arms, birds flapping wings, and lights changing. Due to space constraints, for the baselines, we only showed two representative viewpoints at starting and ending timestamps. The objects’ poses at two timestamps are almost identical, as the baselines generated samples with invisible motions along the whole timeline. In contrast, the motions from our method are much more obvious and pronounced.\n- In Fig.A in rebuttal.pdf, we provide more samples with larger motion magnitudes, including the pose change of a knight, flapping of butterfly wings, turtle swimming, the kid falling, etc. In addition to orbital views that illustrate the 3D geometry consistency, we also exhibit novel front views. The generated motions are much more apparent seen from the front view. We can observe that our model is capable of generating 4D assets with very large motions and great geometry consistency.\n\nWe believe that these demonstrations, along with the visualizations in the main submission, could provide a well-rounded view of our generation results and successfully address your concerns. We also kindly invite you to refer to global rebuttal-(1) and (2) for more details.\n\n**Q3. More analysis on the effect and the sensitivity of the design components.**\n\nThank you for your recognition of the novelty of the proposed 3D-to-4D motion magnitude metric (_m_) and innovation of our motion magnitude reconstruction loss ($L_{mr}$). We are pleased to receive your constructive comment and provide a more in-depth analysis of the effect and the sensitivity of the design components as follows.\n1) 3D-to-4D motion magnitude metric (_m_). In the ablation study, we conducted a qualitative analysis on the effect of _m_. By comparing (b) and (c) in Fig.5 of main.pdf, we can observe that increasing _m_ could augment the dynamics of the kid. We also provide more examples in Fig.B of rebuttal.pdf. We can observe that increasing _m_ augments the dynamics of the generated assets.\n2) Effect of motion magnitude reconstruction loss ($L_{mr}$). In the ablation study, we conducted both quantitative and qualitative analysis on the effects of $L_{mr}$ in Tab.2 and Fig.5 of main.pdf. We can observe that adding in the $L_{mr}$ is critical to animating the static objects, and improving the photometric performance. We further evaluate the effect of $L_{mr}$ on the model performance by monitoring the validation loss along the training process. Please refer to Fig.C in rebuttal.pdf. \nSpecifically, we trained two image-to-4D generation models under the same settings, except that one included $L_{mr}$ and the other did not. At each training step, we evaluate the model’s denoising capability with the validation set. We applied the same magnitude of noise (controlled by $t$) to the clean validation set in the diffusion process, and computed the noise prediction loss $L_{ldm}$ and motion magnitude reconstruction error (*mre*) (Eq.(2)) after denosing. As shown in Fig.C in rebuttal.pdf, **training with $L_{mr}$ leads to faster convergency in diffusion learning and better performance in motion magnitude reconstruction.** (Note that the curves are smoothed with a striding window of size 5.) Additionally, we empirically found that our framework is robust to the setting of the weight _w_ of $L_{mr}$. \n3) For the explicit 4D construction stage, we followed [46] and adopted the same weights for $L_{1}$ and $L_{lpips}$. We empirically found that the model performance is robust to the choice of weight of $L_{depth}$, which serves as a regularizer.\n\nWe will release our codebase soon to facilitate replication and further development upon our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uYnHkVEC0C", "reviewer": "Reviewer_mXZW", "summary": "The paper presents a novel framework for generating 4D content, leveraging video diffusion models, and introduces motion magnitude reconstruction loss and 3D-aware classifier-free guidance to refine dynamics learning and generation. The framework is shown to enhance generation efficiency and 4D geometry consistency relative to existing methods.", "questions": "See weaknesses", "limitations": "The authors have acknowledged some limitations in their method.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is the first to directly generate 4D videos using a video diffusion model, partially addressing the spatial-temporal inconsistencies caused by using separate multi-view 3D synthesis models and monocular video generation models in previous works.\n2.\tCurate a large-scale, high-quality dynamic 3D dataset sourced from Objaverse to facilitate future research.", "weaknesses": "1.\tThis method still uses two models (VideoMV and Diffusion4D). Although it is claimed that the outputs of the two models are very similar, the supplementary video still shows considerable inconsistencies, which to some extent undermine the overall consistency.\n2.\tIn the paper, both numerical and visual comparisons only used 24 images in orbital videos as targets. However, this reduces the difficulty because the viewpoints of the final 4D renderings are almost the same as those generated by Diffusion4D, and the viewpoints generated by Diffusion4D are too limited compared to other methods. Maybe the author could provide more results to demonstrate that the 4D Gaussian obtained by this method can still render high-quality images from different viewpoints. For example, they could fix the camera at a frontal view to render images at different time points or use viewpoints opposite to those generated by diffusion4D.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mXZW, many thanks to you for taking the time and effort to review our work and providing constructive comments. We are grateful for your recognition of the novelty and contributions of our research, noting that **it is the first work training a singular video diffusion model directly on 4D datasets for explicit synthesis of spatial-temporal consistent views of 4D assets. We also appreciate your acknowledgment of the great value of our curated high-quality dataset in facilitating future research.**\n\nWe have thoroughly considered your constructive feedback, and we are committed to addressing all your concerns as follows:\n\n**Q1. VideoMV and Diffusion4D outputs show differences.**\n\nIn the 4D construction stage, we apply the dynamic images from our 4D-aware video diffusion model to optimize the 4DGS. We also involved static images from VideoMV to help initialize the 3D geometry. Since our 4D-aware video diffusion model was finetuned from VideoMV, we noticed that the inconsistency between two sets of images is very limited. **We proposed a coarse-to-fine optimization strategy in the 4D construction stage, and we carried out extensive experiments and validated that this strategy can effectively resolve the potential inconsistency.** \n\nSpecifically, In the coarse stage, the static images from VideoMV are only used to initialize the 3D geometry (the attributes of gaussians), and are not used to train the deformation network for motion learning. The potential inconsistency is mitigated in the fine stage, where only dynamic images from Diffusion4D are used to finetune both the attributes of gaussians and the deformation network to guarantee spatial-temporal consistency.\n\n**Q2. More numerical and visual comparisons on novel views**\n\nWe highly appreciate your comments at this point which have greatly contributed to enhancing the quality and clarity of our work. In response, we have conducted more extensive numerical comparisons and visual demonstrations on novel views, as detailed below.\n\n**1) Numerical Comparisons.** \n\nWe first clarify the details regarding numerical comparison in our main submission. As shown in Tab.1 of main.pdf, we evaluated both the images from 4D-aware video diffusion model (Diffusion4D*) and the renderings from constructed 4D assets (Diffusion4D). For the latter, we uniformly rendered 36 orbital viewpoints along the timeline as the targets (L252), which resulted in most of them being at novel azimuth angles distinct from the 24-frame videos. At this point, our comprehensive evaluation covered a set of metrics including CLIP-O, SSIM, PSNR, LPIPS, and FVD. Additionally, we measured the novel front views using CLIP (CLIP-F in Tab.1), where we fixed the viewpoint at the front view and similarly, uniformly rendered 36 images along the timeline as targets (L253). Thus, most of these images are at novel timestamps distinct from the 24-frame videos. We will clarify this in the refined version.\n\nTo strengthen our quantitative evaluation, we used our validation set and further evaluated **two sets of novel viewpoints with all the metrics**. The first set is the previously used 36-frame front-view renderings (F); the second set is a new 36-frame orbital-view renderings with an elevation angle fixed at 30° (O-30°). For both sets, we used ground truth images rendered from the dynamic 3D assets in Objaverse as references. Metrics were computed between ground truth images and rendered images from the same viewpoints and timestamps. The results are shown in the table below:\n\n||CLIP-F↑|SSIM-F↑|PSNR-F↑|LPIPS-F↓|FVD-F↓|CLIP-O-30°↑|SSIM-O-30°↑|PSNR-O-30°↑|LPIPS-O-30°↓|FVD-O-30°↓|\n|-|-|-|-|-|-|-|-|-|-|-|\n|(text) 4D-fy|0.78|-|-|-|-|0.55|-|-|-|-|\n|(text) Animate124   |0.75|-|-|-|-|0.51|-|-|-|-|\n|(text) Diffusion4D  |**0.81**|-|-|-|-|**0.62**|-|-|-|-|\n|(image) 4DGen|0.84| 0.72|15.1| 0.28     | 691.5     | 0.66        | 0.66        | 13.9        | 0.34         | 760.5      |\n|(image) Stag4D|0.86|0.78| 16.0     | 0.25     | 624.7     | 0.69        | 0.74        | 15.1        | 0.30         | 705.8      |\n|(image) Diffusion4D | **0.89** | **0.80**     | **16.2** | **0.24** | **594.5** | **0.72**    | **0.81**    | **16.5**    | **0.23**     | **526.4**  |\n\nNote that CLIP-F is adopted from Tab.1 in main.pdf. Comparing these results along with the results in Tab.1, we can observe the following:\n- Our method consistently outperforms the baselines across all the metrics at the novel viewpoints. \n- For the semantic metric(CLIP), all the methods perform better in front views than orbital views;\n- For the photometric metrics, the competitors (4DGen, Stag4D) perform better in front views than orbital views, while our method performs better in orbital views than front views. This is because the competitors involved the ground-truth front-view videos in their training for 4D construction, whereas our method utilized the orbital videos generated by our 4D-aware video diffusion model in 4D construction.\n\n**2) Visual comparisons.** \n\nFirst, we would like to clarify that each rendered video in the supplementary material consists of 160 frames around the constructed 4D asset, with most frames presenting novel viewpoints not seen in the 24-frame videos. Also, we kindly invite you to refer to Fig.A in rebuttal.pdf, where we provide renderings of samples from more novel views. In addition to orbital-view at elevation angle of 0°, we also visualize the renderings from orbital-view at an elevation angle of 30°, and monocular view by fixing the camera at the front view. \n\nCompared to the baselines, we can observe that our constructed 4D assets exhibit **larger motion magnitude, more complex texture, smoother appearance, and more consistent geometry** across multiple viewpoints. Please refer to global rebuttal (1) and (2) for detailed discussions.\n\nWe are willing to address any further questions or concerns you might have within this review window.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Area Chair and Reviewers,

We would like to express our sincere gratitude for your valuable time and efforts in reviewing our work and providing insightful feedback. We are encouraged that reviewers find that our paper is well-written (fabU, BwxH), the usage of video diffusion model to address inefficiency and inconsistency issues is novel and clearly illustrated (G7WS, fabU, mXZW), the introduction of 3D-to-4D motion magnitude metric is innovative (BwxH), and the experimental evaluations are comprehensive and convincing (fabU, BwxH). We are also pleased that all reviewers recognize the value of our curated 4D dataset in facilitating future research in the community. 

Primarily, the reviewers requested more visualizations of the generated assets, in terms of comparisons with baselines and novel views. Therefore, we provided more cases in Fig.A in rebuttal.pdf and elaborated in detail as follows:

**(1) Comparisons with baselines.** 
- **Larger motion magnitude.** In Fig.3 of main.pdf and Fig.A of rebuttal.pdf, we provided a qualitative comparison between our method and baselines. In Fig.3, due to space constraints, for the baselines, we only showed two representative viewpoints at starting and ending timestamps. It is evident that the objects’ poses at two timestamps are almost identical, as they generated samples with invisible motions along the whole timeline. In contrast, the motions from our method are much more obvious and pronounced. Additionally, in Fig.A, we show more cases with large motions, including the pose change of a knight, the flapping of butterfly wings, a turtle swimming, a kid falling, and the arm movements of a cartoon dog and Mario. Our method is capable of generating 4D assets with significantly larger motions.
- **More complex texture, smoother appearance, and more consistent geometry.** For the generation details, by zooming in, we can observe that the baselines suffer from blurry texture, noisy appearances, excessive artifacts, and incomplete geometry. On the other hand, our method provides more complex texture, smoother appearances, and more consistent geometry. These phenomena can also be observed in both Fig.3 and Fig.A. 
- **More efficient generation and superior quantitative performance.** Also, in Tab.1 of main.pdf, extensive quantitative comparisons and human preference evaluations suggest the superiority and efficiency of our method, which can produce more favorable and high-quality 4D assets within several minutes, compared with the baselines that require hours of optimization.

**(2) Consistent novel views**.

In Fig.A in rebuttal.pdf, we provided more samples with large motion magnitudes from more novel views. In addition to our default setting of orbital views with elevation angle 0°, we also rendered more novel views from the constructed 4D assets, including orbital views with elevation angle 30° and front views. We can observe that the constructed 4D assets demonstrate large motions, coherent appearance, and consistent geometries across timestamps and viewpoints.The motions are more apparent by fixing the camera at front view.

We believe that these demonstrations, along with the visualizations in the main submission, could provide a well-rounded view of our generations. And we are pleased to offer more video demonstrations if policy allows. 

**(3) Summarization of related 4D generation works.**

To give a clearer illustration about the method differences btw. our work and previous works, we provide a summarization of prior 4D generation works in the table below. Our approach differs from previous methods in terms of optimization and consistency enhancement strategies. Most works depend on 2D/3D diffusion priors and use SDS optimization, resulting in hours of optimization cost, including some text-to-4D generation works(4D-fy[1], Animate124 [55]) and image/video-to-4D generation works(4DGen [50], DreamGaussian4D[27], Stag4D[52]). Concurrent arXiv work (Diffusion^2 [48]) eliminates SDS and combines multiple diffusion priors for joint denoising, but it ignores geometry consistency in the generation process. **In contrast, our work a) avoids SDS optimization and largely reduces the generation time from hours to several minutes; and b) utilizes a singular video diffusion model to ensure spatial-temporal consistency.** To the best of our knowledge, this is the first work training on 4D datasets to generate 4D assets, which can be termed as '4D Native' method [57]. Our framework is also the first to integrate 4D spatial-temporal consistency into a singular video diffusion model, enabling the generation of multi-timestamp cross-view supervisions in one shot. 

| | 2D/3D Priors | 2D/3D Priors|2D/3D Priors |2D/3D Priors |2D/3D Priors |2D/3D Priors |4D Native|
|-|-|-|-|-|-|-|-|
||4D-fy[1]|Animate124[55]|4DGen[50]|DreamGaussian4D[27]|Stag4D[52]|Diffusion^2[48]|**Diffusion4D**|
|Text-to-4D generation|√|√|×|×|√|×|√|
|Image-to-4D generation|×|√|√|√|√|√|√|
|SDS optimization|√|√|√|√|√|×|×|
|Training on 4D data|×|×|×|×|×|×|√|
|# of diffusion models used|3|3|2|2|2|2|1|

In the following, we provide more experiments and explanations to address all the questions and concerns raised by the reviewers. And we will incorporate them in the revised version.

Best regards,

Paper 3414 Authors

[57] Liu J, Huang X, Huang T, et al. A comprehensive survey on 3D content generation[J]. arXiv preprint arXiv:2402.01166, 2024.

Meta-review: All the reviewers are positive about the paper and recommends acceptance. The AC agrees with the decision. Please read the weakness comments carefully and prepare the camera ready.
Decision: Accept (poster)