4Diffusion: Multi-view Video Diffusion Model for 4D Generation
Authors: Haiyu Zhang
Xinyuan Chen
Yaohui Wang
Xihui Liu
Yunhong Wang
Yu Qiao
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Model
4D Generation
NeRF
Primary area: generative_models
TL;DR: we present a novel 4D generation pipeline, 4Diffusion, to create high-quality spatial-temporally consistent 4D content from a monocular video.

Abstract: Current 4D generation methods have achieved noteworthy efficacy with the aid of advanced diffusion generative models. However, these methods lack multi-view spatial-temporal modeling and encounter challenges in integrating diverse prior knowledge from multiple diffusion models, resulting in inconsistent temporal appearance and flickers. In this paper, we propose a novel 4D generation pipeline, namely $\textbf{4Diffusion}$, aimed at generating spatial-temporally consistent 4D content from a monocular video. We first design a unified diffusion model tailored for multi-view video generation by incorporating a learnable motion module into a frozen 3D-aware diffusion model to capture multi-view spatial-temporal correlations. After training on a curated dataset, our diffusion model acquires reasonable temporal consistency and inherently preserves the generalizability and spatial consistency of the 3D-aware diffusion model. Subsequently, we propose 4D-aware Score Distillation Sampling loss, which is based on our multi-view video diffusion model, to optimize 4D representation parameterized by dynamic NeRF. This aims to eliminate discrepancies arising from multiple diffusion models, allowing for generating spatial-temporally consistent 4D content. Moreover, we devise an anchor loss to enhance the appearance details and facilitate the learning of dynamic NeRF. Extensive qualitative and quantitative experiments demonstrate that our method achieves superior performance compared to previous methods.

4Diffusion: Multi-view Video Diffusion Model for 4D Generation
Haiyu Zhang! Xinyuan Chen', Yaohui Wang", Xihui Liu', Yunhong Wang', Yu Qiao
'Beihang University "Shanghai AI Laboratory 'The University of Hong Kong
(zhyzhy, y.wang) (auaaa) edu. cn "(chenxinyuan, wangyaohui, qiaboy)epjaabb.oooo. org. cn "xihuiliu@eee, hku. hk
sccyzhy,yywang//
https://aejion, http:///aeiion.gihubb.... iorb.io/44 io/4diffusion
Monocular Video
Viewpoint
Time
Time
Time
Figure 1: 4Diffusion generates spatial-temporally consistent 4D contents from monocular videos.
Abstract
Current 4D generation methods have achieved noteworthy efficacy with the aid of advanced diffusion generative models. However, these methods lack multi- view spatial-tene -patial-temporal modeling and encounter challenges in integrating diverse prior knowledge from multiple diffusion models, resulting inconsistent temporal appearance and flickers. In this paper, we propose novel 4D generation pipeline, namely 4Diffusion, aimed at generating spatial-temporally consistent 4D content from monocular video. We first design unified diffusion model tailored for multi- view video generation by incorporating learnable motion module into frozen 3D- aware diffusion model to capture multi-view spatial-temporal correlations. After training on a curated dataset, our diffusion model acquires reasonable temporal consistency and inherently preserves the generalizability and spatial consistency the 3D-aware diffusion model. Subsequently, we propose 4D-aware Score Distillation Sampling loss, which is based model, to optimize 4D representation parameterized by dynamic NeRF, This aims eliminate discrepancies arising from multiple diffusion models, allowing for generating spatial-temporaal consistent 4D content. Moreover, we devise an anchor loss to enhance the appearance details and facilitate the learning of dynamic
multi-view video diffusion
'Wok done when Haiyu Zhang interned Shanghai Al Laboratory. Corresppnning author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

NeRF. Extensive qualitative and quantitative experiments demonstrate that our method achieves superior performance compared previous methods.
Introduction
In recent years, diffusion models have significantly impacted the of image, video, and 3D generation. With the support of large-scale text-to-image diffusion models [43, and 3D-aware diffusion models [44, 29, 52], many works [26, 39, 51, 9, 34, 41, 50, 30, 55, 27] leverage Score Distillation Sampling (SDS) [38] to distill the prior knowledge from diffusion models to optimize 3D shape parameterized by NeRF [35] 3DGS [21]. Although they have attained faithful results, they only focus on creating static 3D shapes, neglecting the dynamics of objects in the real world.
Generating 4D content, i... dynamic 3D content, holds diverse applications in the virtual realm, including digital human, gaming, media, and AR/VR. The main challenge lies in creating 4D content with vivid motion and high-quality spatial-temporal consistency. The pioneering study MAV3D [47] introduces a two-stage method, which first learns a static 3D shape with a text-to- text-to-image diffusion model and then deforms the static 3D shape with text-to- rext-to-video diffusion model [46]. However, MAV3D encounters the Janus problem and generates 4D contents with poor appearance and motion [4]. To overcome these issues, the following works [4, 66, 65, 28] employ multiple diffusion models for distinct purposes. Specifically, these methods leverage 3D-aware diffusion models [44, 29] and text-to-image diffusion models [43] to achieve spatial consistency and visually appealing appearance. Akin to MAV3D, they utilize video diffusion models [2, 54, 46] to add motion to create 4D content.
The aforementioned methods utilize multiple diffusion models for 4D generation. As Fig. 2 illustrates, when diffusing images rendered from a 3D model, the 3D-aware diffusion model [44] generates multi-view images address the spatial ambiguity. On the other hand, the 2D image diffusion model [43] produces a clean image with subtle details to refine appearance. The 2D video diffusion model [2] generates dynamic frames to ensure temporal consistency within the same viewpoint. However, there is no accurate guidance to ensure multi-view spatial-tempora consistency due to the lack of multi-view spatial-temporal modeling. Moreover, it is challenging to integrate diverse prior knowledge from multiple diffusion models, often leading to inconsistent temporal appearance and flickers as shown in the second row of Fig. 7.
Noised
Stable Diffusion
MVDream
Condition
A monkey eating candy bar
ZeroScope
Denolsed
Discrepancies arising from multiple diffusion models
Figure 2: Challenges. The denoised images from Stable Diffusion (SD) [43], MVDream [44], and ZeroScope [2]. These diffusion models can not provide multi-view spatial-temporal guidance and exhibit discrepancies, making their integration challenging.
In this paper, we present novel 4D generation pipeline, namely 4Diffusion, to create high-quality -patial-temporally consistent 4D content from monocular video. Specifically, we propose a unified diffusion model, 4DM, capture multi-view spatial-temporal correlations for multi-view video generation. To achieve this, we construct 4DM based on the powerful pre-trained 3D-aware diffusion model [52], which already ensures high-quality multi-view spatial consistency. We then seamlessly integrate motion module into the 3D-aware diffusion model to extend the temporal modeling capability. Contrary to previous attempts [16, 14] that typically demand extensive large-scale video datasets for tuning the motion module, 4DM achieves reasonable temporal consistency and

---

captures multi-view spatial temporll correlations after training only hundreds of multi-view videos. Importantly, we keep the parameters of the 3D-aware diffusion model unchanged to preserve the generalization ability and spatial consistency. 4DM provides multi-view spatial-temporal guidance for 4D generation. Therefore, we propose 4D-aware SDS loss to distill prior knowledge from 4DM to optimize 4D content parameterized by dynamic NeRF. This approach eliminates discrepancies arising from multiple diffusion models and stabilizes the optimizing process. Moreover, we use 4DM to generate anchor videos conditioned on the input monocular video and devise an anchor loss to enhance the appearance details, facilitating the learning of dynamic NeRF, Finally, we generate 4D content -emporlll consistency and vibrant motion coherence with the input video
with high-quat high-quality as shown in Fig. 1. Qualitative and quantitative experiments demonstrate that our method achieves state-of-tee thet performance multi-view video generation and 4D generation from monocular videos.
To summarize, our contributions are as follows: 1) We present 4Diffusion, a novel 4D generation pipeline that generates high-quality spatial-temporal consistent 4D content from a monocular video with multi-view video diffusion model. 2) We propose multi-view video diffusion model, 4DM, which provides multi-view spatial-temporal guidance for 4D generation. It trains on only hundreds of curated high-quality multi-view videos capture multi-view spatial-temporal correlations. 3) We combine 4D-aware SDS loss and an anchor loss based on 4DM to optimize dynamic NeRF, which stabilizes the training process and allows for generating high-quality 4D content.
2
Related Work
Recent breakthroughs in multiple research domains have significantly accelerated progress in 4D generation task. Here, we discuss the most relevant fields, including 3D generation, video and 3D-aware diffusion models, and 4D generation.
3D Generation. Recent studies in 3D generation can be classified into three categories: 3D generative methods [53, 17, 45, 36, 60, 3, 58, 12], feed forward methods [18, 49, 23, 67], and diffusion prior- based methods [26, 39, 51, 9, 34, 41. 50, 30, 55, 27]. Inspired by the advancements in 2D content creation, 3D generative methods utilize the robust diffusion [53] or flow-based [60] backbone to
generate 3D data represented by Signed Distance Function (SDF) [60]. voxel grid [36]. triplane
[8, 17, 45], or weights of neural network [12]. However, these methods require time-consuming pre-training to fit each 3D data and are limited to creating single category. Feed forward methods
[18, 23] adopt image features extracted from the pre-trained visual encoder DINO [7] to reconstruct 3D representations through a highly scalable and efficient transformer-basee decoder. Although they can produce 3D shape in a few seconds, they demand extensive training on large-scale 3D datasets, which is impractical with limited 4D datasets for 4D generation. Furthermore, diffusion prior-based methods distill prior knowledge from diffusion generative models via SDS [38] to optimize 3D representations, enabling the generation of high-quality 3D shapes with strong generalizability. In contrast to static 3D generation, our method focuses on creating 4D content.
Video and 3D-aware Diffusion Models. With the success of large-scale text-to-image diffusion models [43, 1], recent works attempt to use diffusion models to generate more complex signals, including video and 3D. AnimateDiff [16] inserts learnable motion module into the frozen text-to- image model for video generation, which preserves the efficacy of the ext-to--mage model while successfully modeling temporal information. Recent 3D-aware diffusion model Zero- 1-to-3 [29] adopts a stable diffusion model conditioned on relative camera pose and a single image for novel view synthesis. However, this method still suffers from the Janus problem and content drafting problem [44] due to the lack of explicit 3D modeling. Approaches like [31, 44, 59, 32, 52] leverage 3D-aware attention block to model the joint probability distribution of multi-view images, leading spatially consistent generation. However, these approaches are incapable of producing multi-view consistent videos, due to the absence of temporal or spatial modeling.
4D Generation.
Recently, several works have delved into 4D generation from various user-friendly prompts, such as text [47, 4, 66, 28], single image [65, 66]. and a monocular video [42, 20, 62]. The pioneering study MAV3D [47] proposes a two-stage method to optimize 4D representation, i.e. Hexplane [6], with both text-to-image and text-to-video diffusion models in static-to-dynand manner. To generate 4D contents with realistic appearance, Dream-in-4DD [66] and 4D-fy [4] combine hybrid diffusion models. Specifically, they utilize 3D-aware and 2D diffusion guidance to learn
3

---

4D Representation (Dynamic NeRF)
Multi-viee view Videos
Monocular Video
Viewpoint
Tol: Apockay biscky
Volume Rendering
Motion Module
Spoth Mccou de
Sputial ModilMMollli
Motto Modili
Ray
Spalta
Video
GD (D-aware SDS
Anchor Video
Camera Embedding
Novel View
4,
Source View
Figure 3: 4Diffusion overview. Our method first trains unified diffusion, named 4DM, by inserting a learnable motion module at the end of each frozen spatial module of ImageDream to capture multi- view spatial-temporal correlations. Given monocular video and text prompt, 4DM produce consistent multi-view videos. Then, we combine 4D-aware SDS and an anchor loss based on 4DM to optimize 4D content parameterized by Dynamic NeRF.
static 3D representation and incorporate video diffusion guidance to add motion. However, these diffusion models can not offer multi-vier integrate diverse prior knowledge from multiple diffusion models, resulting in suboptimal results. In contrast these approaches, design unified model capture multi-view spatial-temporal correlations for 4D generation.
consistent guidance and is difficult to
Similar to us, [20, 42, 63, 62, 57] generate 4D content from monocular video. Consistent4D [20] introduces an interpolation-driven loss between two adjacent frames to enhance spatial-temporal consistency. However, Consistent4D lacks temporal modeling cross frames. DreamGaussian4D [42],
4DGen [62], and SC4D [57] combine 4D Gaussian Splatting [56, 19] into 4D generation pipeline. Although they notably reduce optimization time, they may result in blurred appearance and inaccurate geometry due to the explicit characteristics of Gaussians. STAG4D [63] proposes a training- freee strategy to generate sparse anchor multi-view videos for 4D generation. In contrast, we propose multi-view video diffusion model to provide multi-view 4D generation.
consistency guidance for
3 Method
Given monocular video V = {451 j = 4.2,...,T' with T frames and an optional textual caption, our temporalyy consistent 4D content, capable of rendering from
goal is to generate a high-quality any novel viewpoint across the temporal dimension. In Sec. 3.1, we talk about 3D-aware diffusion models, employed as the initialization our unified diffusion model. In Sec. 3.2, we propose unified diffusion model 4DM to capture multi-view spatial-tene -ential-eempora correlations for multi-view video generation. Subsequently, we elaborate on distilling prior knowledge from 4DM to optimize 4D content parameterized by dynamic NeRF and devise anchor loss to enhance the appearance details, as detailed in Sec. 3.3. Fig. 3 shows the overall pipeline our method.
3.1 Preliminary: 3D-aware Diffusion Models
3D-aware diffusion models learn spatial relationships from multi-views images for 3D generation and can serve as an initialization of our unified diffusion model. Recent works [31, 59, 32] mainly focus on generating multi-view images from predetermined sparse viewpoints and necessitate additional algorithms 3D reconstruction. Although we can extend these methods to generate multi-view videos and employ 4D reconstruction algorithms, it is challenging to reconstruct high-quality 4D content from a limited number of viewpoints. Therefore, we design our unified diffusion model to generate multi-vie view videos from arbitrary viewpoints and choose ImageDream [52] as initialization.

---

Given four arbitrary orthogonal viewpoints under canonical coordination and single image with an optional textual caption, ImageDream can synthesize four multi-view images that align coherently with the input. Specifically, IrageDream utilizes an adapter similar to IP-Adapter [61] to inject image prompts and a 3D self-attention module to capture spatial relationships.
3.2
4DM: Multi-view Video Diffusion Model
To maintain the spatial consistency and mitigate training complexity, we design our multi-view video diffusion model 4DM based pre-trained 3D-aware diffusion model (i.e., ImageDream [52]). Given a monocular video with an optional text prompt and four orthogonal novel viewpoints under canonical coordination, 4DM aims to generate four spatial-temporaly consistent videos.
Although we can directly use the original IrageDream to generate a set of individual multi-view images to form multi-view videos, the result lacks temporal consistency as IrageDream has no layer for temporal modeling, as shown in Fig. 8. We thus add a zero-initialized motion module at the end of each block of the UViT network of ImageDream. Specifically, each motion module begins with group normalization and a linear projection, followed by two self-attention blocks and one feed-forward block. A final linear projection is then applied, after which the residual hidden feature is added back at the end of each motion module as detailed in Fig. 4. Then, each attention block of 4DM includes spatial module and motion module imm. The spatial module comprises a 3D self-atte oef-attention module I; and a cross-ar cross- attention module. We first concatenate the monocular video latent and four multi-view video latents encoded by VAE [22] to obtain batch B of latents Z E IR BxFF FXN X'x XXW where C is the number of channels, H and W are spatial resolutions, N = 5 is the number of viewpoints, and F is the number of frames. Subsequently, we reshape the temporal axis into the batch dimension and independently process multi-view video latents through the 3D self-attention module,
Z f- Reshape(Z, Z. B F NNCHWW + (B C). Z (Za),
(1)
(2)
Z + Reshape Zn, DD)NHWC-BBNNC
Then, we use the adapter in ImageDream to individually process the input video frames and output the video features to perform cross-attention operations. Here, we also reshape the temporal axis into the batch dimension to prevent dimensional confusion. Furthermore, for the motion module, we perform selffattentton exclusively along the temporal axis by reshaping the spatial dimensions and the viewpoint dimension into the batch dimension,
Z' Reshape ZZ B F NNCHWW -
(BN W) FC).
(4)
Z'
i (Z'),
(5)
Z' Reshape Z' (B
(6)
We utilize Objaverse dataset [11] to train 4DM. Although Objaverse provides nearly 44K animated
3D shapes, rendering multi-view videos and training diffusion model are time- and computation- consuming using the entire dataset. Moreover, it worth noting that the Objaverse dataset contains significant amount of flawed data. Consequently, we manually select curated subset of 926 high-quality animated 3D shapes from Objaverse dataset [1]]. We render multi-view videos from those animated 3D shapes to tune our motion module while holding the parameters of the origin ImageDream frozen. Surprisingly, 4DM successfully learns reasonable temporal dynamics and preserves the characteristics of the origin ImageDream model, including generalization ability, spatial consistency, and image understanding ability, even when trained on a small curated dataset. As Fig. illustrates, 4DM generates multi-view spatial-temporal consistent videos, surpassing the performance of IrageDream. For more details on our dataset, please refer to supplementary material.
Training Objectives. For each animated 3D shape from our dataset, we render a monocular video Vn. with a random viewpoint and four videos V, with orthogonal viewpoints cy,, and select F = 8 frames from each video at a stride of 4 to create our multi-view video dataset X = {(x V, x em J. Here, x and " represent the video clips from Vm and Vo- y is the text prompt captioned by Cap3D [33]. Then, we use X following the diffusion loss to train 4DM,
LMV (0, Xmw) = Ex.y, C, [[]e
e0 (x2: xP, c², 20)
(Xmox 0, 0,O),
with probability p
where, (xP, xL, c²)=
(Xmor x" ) with probability 1- p
5

---

TxB
TAE
Tat
THX
Modiale
Reshape
E gaaerrsssda
GroupNorm
Position Encoclng
Linear Projection
Seft-Attn
Feed Forword
Linear
Tab
Txld
Tab
Tol
Dropout
Linerr Projection
Reshape
Input
front-vie
back-veu
2, Pa Calhady
Figure 5: The illustration of multi-view video generation when input video exceeds 8 frames.
Figure 4: The detailed overview of the archi- tecture of motion module.
Only
Modeos
Training View
Figure 6: Illustration of directly optimizing on the generated multi-view videos.
here, Xme represent the noisy video latents derived from x These latents initially encoded by VAE and subsequently noised by random noise e at diffusion timestep For more details about the noising process, please refer [43]. en is 4DM model parametrized 9. During the training of 4DM, we ensure that our training data does not overlap with the test data used in our experiments.
3.3 4D Generation
Dynamic NeRF Representation. Recent methods [25, 13] use neural networks or explicit spatial grids to map a 6D spatial-temporal coordinate (xr d, t) density r(x, t) R4 and view-dependent color c(x, d, t) R. of dynamic scenes, where originating at o with direction and t denotes timestamp. Then, they leverage volumetric rendering to render images,
fd (E >>>>>>>> 0) are sampled points along ray
C= Σ Wilts where wi = E (5==) (( -1000))
(8)
Following 4D-fy [13] and iNGP [37], use one multi-resolution spatial grid P Zyy and one spatial- time planes P zyzz as 4D representation. Here, both P zy= and P Payse use hash tables to store learnable features. Then, we acquire spatial-ting -ime features f through interpolation and hash lookup on zys and P myzt. Finally, f are decoded into density and view- niw-indepeendent color using tiny MLPs,
tb fff f
The entire set of trainable parameters is denoted as 9aD. We can optimize our dynamic NeRF by using the multi-vies view videos generated from 4DM. however, 4DM can only produce four orthogonal viewpoints at one time. Training with such sparse views often results overfitting to the training viewpoints, as presented in Fig. 6. To mitigate this, we leverage 4D-aware SDS to optimize the dynamic NeRF enabling effective rendering from novel viewpoints across the temporal dimension. which is crucial for 4D generation.

---

4D-aware SDS. Once 4DM is trained, we employ 4D-awar SDS loss to guide the optimization of our 4D representation. To be concrete, we utilize Eqn. 8 render four frames video V,, with timestamps t. = (t1. 12,... ...: (p] from four orthogonal viewpoints Cmv- Our 4D-aware SDS injects Gaussian noise e into V,, at diffusion timestep and passes to our multi-view video diffusion to provide gradients to update 0aD.
V 8ap CAD-SDS Re (,,, 2(V, 10,,-Vo Vo) av, Ø0AD
(10)
where Vo denotes the pseudo ground truth denoised from 4DM with the input video V and viewpoints Cmv as condition. Here, we replace original e-based SDS loss with =n-reconstruction loss as in [44].
Anchor Loss. Accurately estimating the elevation and azimuth of input monocular video within the canonical coordination is challenging, making it difficult to use the input video directly as supervision signals. Therefore, we utilize 4DM to produce four orthogonal videos conditioned on the input video and select the one with the viewpoint closest to that of the input video as the anchor video. This approach ensures that the anchor video maintains the same quality as the input and improves the results. Moreover, 4DM is currently limited generating multi-view videos with 8 frames. When the input video exceeds 8 frames, we must apply our multi-view video diffusion model multiple times to generate anchor videos. However, this process may lead to temporally inconsistent results due to the stochasticity of the diffusion model, particularly when the viewpoint is far from the input video as shown Fig. 5. This inconsistency would degrade the 4D generation performance. Finally, we devise an anchor loss La based on the anchor video to enhance the appearance details and facilitate the learning of dynamic NeRF Since is challenging for 4DM to ensure pixel-to to-pixel alignment of the anchor video, we follow [10] to use image-level perceptual loss, i.e., LPIPS [64] and SSIM, for dynamic NeRF optimization,
C2 AILPIIS Ia) + AzD-SSIM Ia),
(11)
where I, and La represent the rendered video and anchor video, is the loss weight. Consequently, our total loss function for 4D generation is,
LAD = LADASDS + La + AyCorient + X4 Copucay + As As
(12)
here Corient, Lopacity, and Lsparse are regularization loss in DreamFusion [38].
4 Experiments
Implementation Details. We implement 4DM under the Stable Diffusion framework and initialize from the checkpoint of ImageDream. We train 4DM with multi-view videos with 256x256 resolutions for 30,000 steps with a batch size of 32, using the AdamW optimizer with a learning rate of le-4. The training takes about 2 days with 16 NVIDIA Tesla A100 GPUs. Additionally, for 4D generation experiments, we optimize dynamic NeRF representation in an end-to-end manner, avoiding utilizing multiple stages as in previous works.
Baselines. To evaluate our method, we compare to two video-to-4D approaches, namely Consistent4D [20] and DreamGaussian44 [42], and one text-to-4D approach 4D-fy [4]. We extend 4D-fy to video- prompt 4D generation by using ImageDream as the 3D-aware diffusion model. 4D-fy introduces hybrid SDS to blend gradients from multiple pre-trained diffusion models to create 4D contents. Consistent4D is the first study focusing on the video-to-4D task. They utilize a 3D-aware diffusion model to optimize cascade dynamic NeRF and propose consistency loss to address spatial-temporal inconsistency. DreamGaussian4D leverages 4D Gaussian Splatting for faster training.
4.1 Comparisons on 4D Generation
Qualitative Evaluation. To validate 4Diffusion for 4D generation, we compare it to Consistent4D [20]. DreamGaussian4D [42], and 4D-fy [4] on monocular video-to-4D task. Here, we use 3 real- world videos and 3 synthetic videos from the Consistent4D dataset, as well as 3 images from the ImageDream. As discussed in Sec.A.1 of the supplementary materials, for text-image pairs from Iream, we utilize SVD to generate input videos. We illustrate the results in Fig. 7. 4D-fy
[4] achieves ofate-of-the-an results on text-to-4D task and can be simply extended to video-prompt
7

---

Reforenca
4D-f
CondstentaD
I Maw
4D-fy
CondistentDD
View 2
time
time
time
time
Figure 7: 4D generation comparisons with 4D-fy [4], Consistent4D [20], and DreamGaussian4D [42].
4D generation by replacing 3D-awar diffusion model. Here, we utilize ImageDream as the 3D- aware diffusion model in 4D-fy. 4D-fy produces 4D contents with inconsistent temporal appearance, sometimes diverging significantly from the input video, as depicted in the first two columns of Fig. This is primarily because integrating gradients from multiple diffusion models is difficult and they face challenges in multi-view spatial-temporal modeling. Consistent4D the first work for 4D generation from monocular video. They employ interpolation loss between two frames to enhance spatial-temporal consistency. However, they lack temporal consistency across frames, leading to poor appearance quality and flickers. DreamGaussian41 generates 4D contents with blurred appearance and inaccurate geometry because GS struggles to model thin structures and large motions under unconstrained situations. In contrast, 4Diffusion generates high-quality 4D content with 4DM, which captures multi-view superior results, demonstrating its effectiveness. For more visualization results, please refer to supplementary materials.
correlations a unified manner. Overall, our method achieves
Quantitative Evaluation. We select test cases from Objaverse, each consisting of a monocular input video and four orthogonal ground truth videos, which are not included in the training data, to evaluate our model. To evaluate image quality, we leverage CLIP-I [40] to measure the similarity.

---

Table 1: Quantitative evaluation on 4D generation.
Image quality
Tem. Con.
Video quality
Spa. Con.
CLIP-It
CLIP-C1
FVD
LPIPS
PSNR
4D-fy[4]
0.8658
0.9487
1042.3
0.2254
14.24
Consistent4DD200 [00]
0,9216
0.9723
706.07
0.1593
16.70
DreamGaussian44((42
0.8898
0.9710
760.18
0.1793
15.97
Ours(w/o LID-SDS)
0.8195
0.9503
1546.4
0.2356
13.92
Ours(w/o L.) Ours
0.8823
0.9720
853.57
0.1589
17.20
0.9310
0.9798
417.63
0.1199
19.07
Input Monocular Video
Back-View
Side-View
Image ImageDream Ours ImageDream Ours
T
T
T
1/2
Figure 8: The illustration of synthesized multi-view videos from 4DM and ImageDream [52]. 4DM produces more apatial-temporty consistent results than ImageDream. T denotes the timestep of video clips. All results are generated from DDIM [48] sampler.
We also calculate FVD to evaluate the video quality. We compute LPIPS [64] and PSNR metrics to evaluate the spatial consistency. Here, we use ground truth videos for novel viewpoints to compute the above metrics. Moreover, we compute CLIP-C between frames each synthetic video to evaluate temporal consistency. Tab. presents the results, clearly demonstrating that 4Diffusion outperforms other methods on all metrics.
4.2 Multi-view Video Generation
Qualitative Evaluation. In this section, we evaluate the multi-view video generation quality produced by 4DM using the same input videos as described in qualitative evaluation in Sec 4.1. We employ ImageDream to synthesize a set of multi-view images pseudo multi-view video by taking each frame of the input monocular video as an image prompt. Fig. illustrates results with the first two columns corresponding the input monocular video. Although ImageDream excels synthesizing spatially consistent images, struggles to model temporal correlations, leading to inconsistent temporal appearances, such as the icon on the back of Spiderman. Comparatively. 4DM effectively captures reasonable temporal information using the motion module, even when trained on a small curated dataset. Moreover, us to generate high-fidelity multi-view videos, even beyond the distribution of our training dataset. As the last two rows of Fig. show, 4DM produces spatially consistent videos by sharing information across spatial and temporal dimensions to constraint the generation process while ImageDream occasionally fails to generate videos coherent to the viewpoint.
model preserves the generalization ability of ImageDream, allowing
Quantitative Evaluation. We use the same test cases described in quantitative evaluation Sec 4.1, alongside the test data provided by Consistent4D, to evaluate 4DM. To account for the stochasticity of the diffusion model, we conduct five runs for each test case and report the average metrics. Tab. shows comparative results. Despite the comparable performance in CLIP-I, 4DM excels in

---

Table 2: Quantitative evaluation on multi-view video generation. Here, we employ Consistent4D test dataset evaluate 4DM and Irege. 'Spa. Con'' and "Tem. Con." refer spatial consistency and temporal consistency, respectively.
Image quality
Tem. Con.
Video Quality
Spa. Con.
CLIP-It
CLIP-Ct
FVD1
LPIPS|
PSNR
ImageDream[52]
0.9165
0,9320
465.94
0.1536
16.57
Ours(w/ whole)
0.8872
0.9478
583.79
0.1763
15.28
Ours(4DM)
0.9260
0.9601
427.34
0.1346
17.88
generating spatial-temporally consistent multi-view videos, primary focus of our research. This is evidenced by the superior performance on metrics such as CLIP-C, FVD, LPIPS, and PSNR, which better capture the spatial and temporal fidelity of video content. These metrics demonstrate that our method effectively balances image quality with temporal consistency, making it robust solution for multi-view video generation.
4.3 Ablation study and analysis
Effectiveness of the Curated Multi-view Video Dataset. To evaluate the importance and effectiveness of the selected high-quality multi-
view vidcos, use the entire animated 3D shapes from Objaverse and render mieti-view videos to fine-tune 4DM (Ours w/ whole). The results are shown Tab. 2. Given the pres- ence of numerous flawed data within the entire dataset, compromises the image quality of ImageDreamm and encounters challenges in pre- cisely capturing spatial-temporal correlations, demonstrating the importance of high-quality datasets for fine-tuning 4DM.
w/oo Loa-ins
Full
Time
4D 4D-aware SDS Loss. To evaluate the effect of our 4D-aware SDS loss, we substitute the 4DM with ImageDream and use 3D- 3D-aware SDS loss based on IrageDream to optimize dynamic NeRF representation. As Fig. depicted, inconsistent temporal textures. such as the leg of the squirrel, emerge due to the lack of temporal modeling of imageDrean underscoring the significance of capturing spatial-temporal correlations in 4DM. The quantitative results presented in Tab. 1 indicate the significance of our 4D-aware SDS loss.
Figure 9: Ablation studies on 4D-aware SDS loss and the anchor loss.
Anchor Loss. We also assess the impact of the proposed anchor loss. As illustrated in Fig. 9, capturing detailed appearance features, such as the eyes of the squirrel, proves challenging without the anchor loss. Conversely, the anchor images furnish visual clues to facilitate the learning of 4D representation, resulting in high-quality 4D content. The quantitative results Tab. demonstrate the erucial role of our anchor loss.
5 Conclusion
In this paper, we present 4Diffusion for 4D generation from a monocular video. Our method proposes multi-view video diffusion model 4DM based 3D-aware diffusion model for multi-view video generation and provides multi-view spatial-temp metiaal-emmporl guidance for 4D generation. 4DM captures spatial-temporal correlations and preserves the characteristies of the origin 3D-aware diffusion model even when training on small curated dataset. Then, we combine 4D-aware SDS loss and an anchor loss based on 4DM to optimize our hash-encoded dynamic NeRF, resulting in mntialltemporally consistent 4D contents coherent with the input monocular video.
10

---

Acknowlegements
The work is supported by the National Key R&D Program of China (No. 2022ZD0160102) the National Natural Science Foundation of China under Grant No. 62102150, and the Science and Technology Commission of Shanghai Municipality under Grant No. 23QD1400800
References
[1] Deepfloyd. https://gitae com/deep-floyd/Ir 2023.
[2] Zeroscope text-to-video model. https::/huuginngaace.o co/ httpe//eee.eepeeeeeeerrppppppp.. 576w. 2023.
[3] Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In CVPR, pages 12608-1266 12618, 2023.
[4] Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. In CVPR, pages 7996-8000 2024.
[5] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Rodin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint rrXiv:2311.15122 2023.
[6] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes, In CVPR, pages 130-141, 2023.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pages 9650-9660. 2021.
[8] Eric R Chan, Connor Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-awart 3d generative adversarial networks. In CVPR, pages 16123-16133, 2022.
[9] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In ICCV, pages 22246-22256, 2023.
[10] Zilong Chen, Yikai Wang, Feng Wang. Zhengyi Wang. and Huaping Liu. V3d: Video diffusion models are effective 3d generators. arXiv preprint arXiv:2403.06738, 2024.
[11] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, pages 13142-13153, 2023.
[12] Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. In ICCV, pages 14300-14310, 2023.
[13] Sara Fridovich-Ked Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, pages 12479-12488, 2023.
[14] Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Pengfei Wan, Di Zhang, Yufan Liu, Weiming Hu, Zhengjun Zha, Haibin Huang, and Chongyang Ma. 12v-adapter: A general image-to-video adapter for video diffusion models. arXiv preprint arXiv:23 /2. 16693, 2023.
[15] Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia-Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A uni- fied framework for 3d content generation. https:/ //s::////// tigut.u...... com/ www.eeeesttioooooojeeee threestudio, 2023.
[16] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Ani- matediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024.
11

---

[17] Anchit Gupta, Wenhan Xiong, Yixin Nie, lan Jones, and Barlas Oğuz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.05377, 2023.
[18] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi. Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lmm: Large reconstruction model for single image to 3d. In ICLR, 2024.
[19] Yi-Hua Huang. Yang- Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: cparse-controlled gaussian splatting for editable dynamic scenes. In CVPR, pages -230, 2024.
[20] Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, and Yao Yao. Consistent4d: Consistent 360 dynamic object generation from monocular video. In /CLR, 2024.
[21] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküühler and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. TOG, 42(4), 2023.
[22] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[23] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In ICLR, 2024.
[24] Ruilong Li, Hang Gao. Matthew Tancik, and Angjoo Kanazawa. Nerfacc: Efficient sampling accelerates nerfs. In ICCV, pages 18537-18546, 2023.
[25] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, and Zhaoyang Lv. Neural 3d video synthesis from multi-view video. In CVPR, pages 5521-5531, 2022.
[26] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yt Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, pages 300-309, 2023.
[27] Yukang Lin, Haonan Han, Chaoqun Gong, Zunnan Xu, Yachao Zhang, and Xiu Li. Consis- tent123: One image highly consistent 3d asset using case-aware diffusion priors. arXiv preprint arXiv:2309. 272, 202..
[28] Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. In CVPR, pages -586,8888,.1 2024.
[29] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image to 3d object. In ICCV, pages 9298 900889939,,00 2023.
[30] Xian Liu, Xiaohang Zhan, Jiaxiang Tang. Ying Shan, Gang Zeng, Dahua Lin, Xihui Liu, and Ziwei Liu. Humangaussian: Text-driven 3d human generation with gaussian splatting. In CVPR, pages 6646-6657, 2024.
[31] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synedreamer: Generating multiview-consistenn images from a single-view image. In
ICLR, 2024.
[32] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, and wenping Wang. Wonder3d: Single image to 3d using cross-domain diffusion. In CVPR, pages 9970-9980, 2024.
[33] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. In NeurlPS, 2023.
[34] Luke Melas-Kyriaze Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360 reconstruction of any object from a single image. In CVPR, pages 8446-8455 2023. [35] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[36] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guidee 3d radiance field diffusion. In CVPR. pages 4328-4338 2023.
[37] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. TOG, 41(4):1-15, 2022.
12

---

[38] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
[39] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Hsin-Ying Lee, Ivan Skorokhodov. Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magic123: One image high-quality 3d object generation using both 2d and 3d diffusion priors. In ICLR, 2024.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Sutskever lya. Learning transferable visual models from natural language supervision. In ICML, pages 8748 8763, 2021.
[41] Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, Yuanzhen Li, and Varun Jampani. Dreambooth3d: Subject-driven text-to-3d generation. ICCV, 2023.
[42] Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint arXiv:2312.17142, 2023.
[43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[44] Yichun Shi, Peng Wang. Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi- view diffusion for 3d generation. In ICLR, 2024.
[45] J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In CVPR, pages 20875-20886, 2023.
[46] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023.
[47] Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, lurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, and Yaniv Taigman. Text-to-4d dynamic scene generation. In ICML, pages 31915-31999 2023.
[48] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[49] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In ECCV, pages 1-18, 2024.
[50] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2024.
[51] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. In ICCV, 2023.
[52] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi- diffusion for 3d generation. arXiv preprint arXiv:2312.02200 2023.
[53] Tengfei Wang. Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, and Baining Guo. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In CVPR, pages 4563 4573, 2023.
[54] Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation. arXiv preprint arXi::2305.
2023.
[55] Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurIPS. 2023.
[56) Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering, In CVPR, pages 20310-20322 2024.
[57] Zijie Wu, Chaohui Yu, Yanqin Jiang, Chenjie Cao, Fan Wang, and Xiang Bai. Sc4d: Sparse- controlled video-to-4d generation and motion transfer. arXiv preprint axXw::204.033336, 2024.
13

---

[58] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang. Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. In ICIR, 2024.
[59] Jiayu Yang, Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong Li. Consistnet: Enforcing 3d consistency for multi-view images diffusion. In CVPR, pages 7079-7088, 2024.
[60] Lior Yariv, Omri Puny, Natalia Neverova, Oran Gafni, and Yaron Lipman. Mosaic-sdf for 3d generative models. In CVPR, pages 4630-4639, 2024.
[61] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0662) 2023.
[62] Yuyang Yin, Dejia Xu, Zhangyang Wang. Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arXiv:2312.17225 2023.
[63] Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao. Stag4d: Spatial-tempont anchored generative gaussians. arXiv preprint arXiv:2403.14993 2024.
[64] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The un- reasonable effectiveness of deep features as a perceptual metric. CVPR, pages 586-595, 2018.
[65] Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. An- imate 124: Animating one image to 4d dynamic scene. arXiv preprint arXiv:231111400,, 2023.
[66] Yufeng Zheng, Xueting Li, Koki Nagano, Sifei Liu, Otmar Hilliges, and Shalini De Mello. A unified approach for text-and mage-guided 4d scene generation. In CVPR, pages 7300-7309, 2024.
[67] Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song- Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. CVPR, pages 10324-10335, 20234.
14

---

A Supplemental material
A.1 More Implementation Details
Table 3: Hash encoding parameters of Pays and Pa Pzyzt
Parameter
Value
Number of levels
16
Hash table size
2
Number of feature dimensions per level
Coarsest resolution
16
Finest resolution
4096
Low-quality
High-quality
Static
+
Out-of-scene Movement
Rotated Camera
Figure 10: The illustration of our training dataset. We manually filter out animated 3D data with static motion, out-of-scene movement, rotated camera to curate dataset with high-quality appearance and realistic motion.
Datasets. We utilize Objaverse dataset [11] to train our multi-view diffusion model, described in Sec. 3.2. Objaverse dataset comprises vast collection of 3D shapes with descriptive captions, tags, and animations. We manually filter out animated 3D shapes that contain static objects, out-of-scene movement, rotated cameras, or meaningless objects, resulting models, depicted in Fig. 10. We apply Blender render 32 videos with azimuth angles uniformly ranging from 1-100, 180%] and an clevation angle of 0" for each animated 3D model. our experiments, use the dataset released by Consistent4D [20], test cases from Objaverse, and text-image pairs from IrageDream [52] project page. Specifically, for text-image pairs, we leverage Stable Video Diffusion V1.1 [5] to produce monocular videos for 4D generation.
926 high-quality 3D animated
4D Generation. We implement our 4D generation model under threestudio framework [15]. Our hash-encoded dynamic NeRF representation utilizes the parameters detailed in Tab. 3. Following [55, 4], we anneal the timesteps of diffusion models from E [0.98, 0.98) to E [0.02, 0.25] over initial 5,000 iterations and set the diffusion CFG to 5.0. The loss weights Å1 A2, X4 are set to 200, 100. and 100, respectively. Addtionally, Ag linearly increases from to 1000 during the first 5,000 iterations and A is fixed at 100 after the initial 10,000 iterations. The model is trained with AdamW optimizer for 35,000 iterations with a learning rate of le-2 except for the decoded MLPs, where the learning rate is adjusted to le-3. It takes around 12 hours to train the model on one NVIDIA Tesla A100 GPU.
Volume Rendering. We employ NerfAce [24] our rendering pipeline, which leverages an occupancy grid to store the opacity of scene. This approach accelerates volume rendering and reduces computations. We adopt shared occupancy grid by representing the maximum opacity of the scene across all frames, facilitating application dynamic scenes. Additionally, we set the background of the rendered images to white. For the resolution of rendered images, follow the
15

---

Time
Reference
ViewI
Ours
View2
Reference
View1
Ours
View2
Reference
View1
Ours
View2
Figure 11: The illustration of 4D generation results of 4Diffusion.
configuration of ImageDream [52]. We maintain fixed camera distances at 1.1 to enhance the stability of the optimization process.
A.2 More Results
4D Generation. In Fig. 11, we showcase additional results our 4D generation results. To gain more intuitive understanding, encourage readers to view the supplementary videos. Moreover,
16

---

Input
4D-fy
Dream Gaussian4D
Consistent4D
Ours
Figure 12: 4D generation comparisons with 4D-fy, DreamGaussian4D, Consistent4D. These test cases selected from Objaverse dataset, which are not included in the training data of 4DM.
show more comparisons on 4D generation as shown in Fig. 12, Consistent4D and DreamGaussian4D encounter the multi-face problem while our method generates temporllly consistent contents.
Text-to-4D. To Further validate the effectiveness 4Diffusion, conduct experiments on text- to-4D task. Specifically, we first employ SDXL to generate images conditioned on text prompts. Subsequently, we utilize SVD V1.1 to produce monocular videos for 4D generation. Finally, we follow the procedure outlined in the main paper to generate 4D content from the monocular videos. As illustrated Figure 13, our approach yields high-quality 4D content from text prompts, thereby demonstrating its effectiveness.
A.3 Limitations
Our method can be improved in the following aspects: 1) Our multi-view video diffusion model is constrained by the capability of the base model and the scale of high-quality training data. We believe that improving the base model and scaling up the high-quality dataset can obtain better model. 2) Our 4D generation pipeline relies on heavily volumetric rendering, causing slow training speed. We believe advances in 3D and GS can potentially solve these problems.
A.4 Broader Impacts
Our work paves the way for high-quality 4D content generation, reducing the extensive manual effort for artists and novices. Although our method is not designed for generating humans, may be extended and misused, potentially influencing human perceptions.
17

---

Time
"A photo of a horse walking, toy, 3d asset"
"Astronaut walking in space, full body, 3d asset"
Figure 13: The illustration of text-to-4D results of 4Diffusion.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction claim the contributions and describe the chal- lenges, which our method tries to solve.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In our supplementary material, we discuss the limitations of our method and the potential future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We describe the implementation details for our experiments in our paper and all the results can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We release our code and data at htttp:://aajioo http://///eiioniiiub..bb.. github..oo4ddff.......
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss httppsccccpbccccccceececcccccccc httee/guuueeeeeeeuubiiaeeeee /Cooeeeooiissseeeeeee
cessionPoocyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide training and test details in experiment which can help readers to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: In our experiments, we do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computation resources required to reproduce our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: / //nuurrrrsp.. cc cc/ppbblc//eeeeeeeeeeeeeeeeeeeet ctw.teeeeeehiiiieeeeeeeeeeeee
didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: Our paper follows the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in our supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our pre-tree pee-trained model and method present low risk of misuse dual-use, as the model is trained on object-centric datasets that contain no human data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow the license of each we have used and cite them properly. Moreover, we also state the version of these assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper provides assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "iuWlyHssyl", "reviewer": "Reviewer_ioTR", "summary": "The paper proposes a 4D generation method that aims to generate 4D content from a monocular video. A video-to-multi-view-video diffusion model is presented to create multi-view videos given a monocular video, a text prompt, and a sequence of camera poses. The trained multi-view-video diffusion model is leveraged to optimize 4D representation, i.e., dynamic NeRF.  In addition, 4D-aware SDS loss and an anchor loss are introduced to train dynamic NeRF. Experimental results show the proposed method achieves the best performance, compared with state-of-the-art methods.", "questions": "Please refer to my comments above", "limitations": "The paper provides the limitations and societal impact of the proposed work", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n\n2. A multi-view-video diffusion model is presented to generate multi-view videos from a monocular video\n\n3. The paper addresses an interesting problem, and  4D generation significantly impacts various applications.", "weaknesses": "1. Some technical details are unclear. The paper builds a multi-view-video diffusion model by inserting a learnable motion module into ImageDream. The learnable motion module is critical to the proposed method. However, the paper does not provide detailed information about the motion module, such as the architecture and the layer information. Without this information, it is difficult to reproduce the proposed method.\n\n2.  The paper only uses 996 training data to train the multi-view-video diffusion model or the motion module, while the training takes two days on 16 NVIDIA Tesla A100 GPUs. Does such a small data set and such an extensive training cost lead to significant overfitting? How many parameters are in the motion modules?\n\n3.  Instead of the input monocular video, the anchor loss chooses a monocular video generated by the presented multi-view-video diffusion model as an anchor, due to the difficulty in estimating the camera pose of the input video. Why not use all videos generated by the multi-view-video diffusion? Would this operation degrade the 4D generation performance?  In addition, the input video typically has better quality than the generated one.\n\n4. Table 2 shows that using ImageDream achieves better CLIP-I than using the present multi-view-video diffusion model. Could the authors provide more explanations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating and acknowledging our work. We address your constructive comments below:\n\n**W1: Some technical details are unclear.**\n\nWe will provide detailed information about the motion module in the revised version. Specifically, we incorporate a zero-initialized motion module at the end of each spatial attention block in ImageDream, as illustrated in Fig.3. Each motion module begins with group normalization and a linear projection, followed by two self-attention blocks and one feed-forward block. A final linear projection is then applied, after which the residual hidden feature is added back at the end of each motion module. For a more detailed architecture overview, please refer to Fig. R1 in the attached PDF. Moreover, we will open our code to enable researchers to reproduce our results and further build upon our work after the completion of the anonymous review process.\n\n**W2: Overfitting and parameters of the motion modules.**\n\nThank you for your insightful question. Our motion module contains 453.209M parameters. We have evaluated our model on real-world videos from the Consistent4D dataset, including the squirrel in Fig. 1, the egret and robot in Fig. 4, and the jay in Fig. 8, indicating that our model does not overfit the training data. To be more concrete, the reasons are as follows:\n- Our training dataset comprises 966 multi-view videos, with each video containing 32 viewpoints. This setup results in approximately 741,000 frames in total. Additionally, the average number of frames per video is 24. During the training iteration of 4DM, we randomly sample 4 orthogonal viewpoints with 8 frames from the training data, this strategy effectively augments our training data, helping to mitigate the risk of overfitting.\n- Our 4DM model employs several 3D self-attention layers and 1D temporal self-attention layers to capture spatial-temporal relationships, which requires approximately 6 seconds to optimize per step. As a result, two days of training (30,000 iterations) is insufficient for 4DM to overfit the dataset.\n- In 4DM, we only finetune the parameters of the motion module while keeping the parameters of the original ImageDream model frozen, thereby preserving ImageDream's generalization ability, even when trained on a small curated dataset.\n\n**W3: Anchor videos.**\n\nThank you for your comments. As the reviewer pointed out, the input video is generally of better quality than the generated videos. Therefore, we select the viewpoint closest to the input video as the anchor. This approach ensures that the anchor video maintains the same quality as the input, which improves the results. Moreover, our multi-view video diffusion model is currently limited to generating multi-view videos with 8 frames. When the input video exceeds 8 frames, we must apply our multi-view video diffusion model multiple times to generate anchor videos. However, this process may lead to temporally inconsistent results due to the stochasticity of the diffusion model, particularly when the viewpoint is far from the input video as shown in Fig.R2 in the attached PDF. This inconsistency would degrade the 4D generation performance.\n\n**W4: CLIP-I score.**\n\nIn Tab.2 in the main paper, our method performs only slightly lower (-0.0146) than ImageDream in the CLIP-I metric, likely due to the stochasticity of the diffusion model. However, our method surpasses ImageDream in the LPIPS metric, which also reflects image quality. Moreover, our method exhibits comparable visual quality to that of ImageDream as shown in Fig.5 in the main paper. To further assess our model, we select 5 additional test cases from Objaverse, alongside the test data provided by Consistent4D (None of these test data are included in our training dataset). To account for the stochasticity of the diffusion model, we conduct five runs for each test case and report the average metrics as follows:\n\n\n| **Model**                     | **CLIP-I↑**   | **LPIPS↓** |**CLIP-C↑**| **FVD↓**      | \n|------------------------------|---------------|-----------|---------------|--------------------\n| **ImageDream**                     |   0.9165      |   0.1536   |    0.9320     |       465.94       |\n| **Ours(4DM)**                      |  **0.9260**   | **0.1346**| **0.9601**    | **427.34**         |\n\nDespite the comparable performance in CLIP-I, our model excels in generating spatial-temporally consistent multi-view videos, a primary focus of our research. This is evidenced by the superior performance on metrics such as LPIPS, CLIP-C, and FVD, which better capture the spatial and temporal fidelity of video content. These metrics demonstrate that our method effectively balances image quality with temporal consistency, making it a robust solution for multi-view video generation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dcSLssYp9U", "reviewer": "Reviewer_Y3vx", "summary": "This paper tackles the task of 4D reconstruction from monocular video. It introduces a training approach for a multi-view video generative model using a synthetic dataset of multi-view videos. The architecture uses a 3D-aware denoising diffusion model previously applied to multi-view images and extends it to accommodate multi-view videos. The model is fine-tuned using 1,000 synthetic multi-view videos from the Objaverse dataset. Then, score-distillation sampling (SDS) is used to generate a dynamic radiance field. The evaluation on videos of synthetic object-centric scenes demonstrates a slight improvement in terms of CLIP and FVD metrics over the recent Consistent4D work on the task of novel view synthesis from monocular video. Although the qualitative results show minor enhancement over baselines, concerns remain about the generalization to real-world videos and the evaluation, especially regarding potential training data leakage and significance of improvement over Consistent4D. Addressing these issues would warrant the acceptance of the paper.", "questions": "The paper assumes access to a multi-view video dataset. The rationale behind the need for SDS when a multi-view video diffusion model is already available is unclear. Could you explain why not fit the dynamic NeRF directly on the generated multi-view videos?", "limitations": "Authors addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses the significant and timely issue of generating 4D content using diffusion models.\n- The architectural extension of the 3D-aware diffusion model and its fine-tuning are good contributions that would be useful to know for the community. \n- The technical contribution is highlighted by impressive generalization performance (assuming no train data leakage). \n- This also highlights the scalability potential of synthetic Objaverse dataset for fine-tuning video diffusion models to perform 4D generation.\n- Both qualitative and quantitative results indicate improvements over the baselines, albeit modest compared to Consistent4D.", "weaknesses": "- The training requires a multi-view video dataset which is difficult to obtain.\n- The evaluation is limited to synthetic, object-centric toy scenes without backgrounds.\n- I haven’t found a description of the validation and test dataset for experiments in Section 4.1\n- It is not clear whether assets in test-videos are unseen during training of all models (as some of them are also trained on objaverse).\n- Evaluation in Sec 4.1 is limited to CLIP and FVD metrics. Since multi-view video datasets were used for training, one could evaluate models for the novel view synthesis task using standard metrics such as LPIPS/PSNR (taking best of 10 due probabilistic nature of the task).\n- Minor: Given the small improvement over Consistent4D, further evaluation of statistical significance is needed.\n- Minor: The paper would benefit from more precise writing; particularly, the training description in lines 176-182 needs more clarity on each variable and the noising process, and Equation 10 lacks clarity regarding sampled variables used in expectations. The method description is overly complex, and the language is difficult to follow, containing several unclear sentences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing and valuing our work. We address your constructive comments as follows:\n\n**W1: Difficult to obtain training data.**\n\nWe have manually curated a high-quality subset of Objaverse, which will released to support community development. Despite data limitations, our method has achieved promising results by leveraging a pre-trained 3D-aware diffusion model. Additionally, as the 4D field continues to advance, we anticipate the emergence of more multi-view video datasets, which have been used in L4GM[1] and Diffusion4D[2].\n\n**W2: The evaluation is limited to synthetic, object-centric toy scenes without backgrounds.**\n\nConcurrently, most research in 4D generation focuses on creating object-centric scenes without backgrounds. Because maintaining spatial-temporal consistency is challenging. We address this challenge by designing a multi-view video diffusion model for 4D generation, which achieves superior performance compared to other methods. Additionally, our evaluation extends beyond synthetic data. We also evaluated on real-world videos from the Consistent4D dataset, including the squirrel in Fig. 1, the egret and robot in Fig. 4, and the jay in Fig. 8.\n\n**W3: Description of test dataset for experiments in Section 4.1.**\n\nWe conduct experiments in Sec.4.1 by using 3 real-world videos and 3 synthetic videos from the Consistent4D dataset, as well as 3 images from the ImageDream. As discussed in Sec.A.1 of the supplementary materials, for text-image pairs from ImageDream, we utilize SVD to generate input videos. We will add the description in our revised version.\n\n**W4: Whether assets in test-videos are unseen during the training of all models.**\n\nSorry for making this unclear. During the training of 4DM, we ensure that our training data does not overlap with the test data. Regarding the pre-trained ImageDream model, the training data has not been released. However, it is known that the model was trained exclusively on multi-view images from Objaverse, which means it has not seen the test videos either.\n\n**W5: Evaluation in Sec 4.1 is limited to CLIP and FVD metrics.**\n\nThanks for your constructive comments. As detailed in **W3**, we utilize monocular videos from Consistent4D and ImageDream to evaluate our method. However, these datasets do not provide ground truth videos for novel viewpoints, preventing us from calculating LPIPS and PSNR metrics. To further evaluate our model, we select 5 multi-view videos that are not included in the training data and conduct experiments.  The following table presents the results, where metrics such as FVD, LPIPS, and PSNR clearly demonstrate that our method significantly outperforms the baselines. \n\n| **Model**                     | **CLIP-I↑**   | **CLIP-C↑**| **FVD↓**      | **LPIPS↓**         | **PSNR↑**         |\n|------------------------------|---------------|-----------|---------------|--------------------|-------------------|\n| **Consistent4D**              | 0.9216        | 0.9723    | 706.07        | 0.1593             | 16.70             |\n| **DreamGaussian4D**           | 0.8898        | 0.9710    | 760.18        | 0.1793             | 15.97             |\n| **4D-fy**                     | 0.8658        | 0.9487    | 1042.3        | 0.2254             | 14.24             |\n| **Ours**                      | **0.9310**    | **0.9798**| **417.63**    | **0.1199**         | **19.07**         |\n\nHere, we use ground truth videos for novel viewpoints to compute these metrics. As illustrated in the first two rows of Fig.R3 in the attached PDF, Consistent4D and DreamGaussian4D encounter the multi-face problem. Consequently, when computing metrics between the input video and synthesized videos, as shown in Tab.1 of the main paper, the performance gap between our method and the baselines may not be apparent. Due to time constraints, we selected the best of two runs when conducting this experiment.\n\n**W6: Minor: Given the small improvement over Consistent4D, further evaluation of statistical significance is needed.**\n\nAs discussed in **W5**, we conduct additional experiments to further evaluate our method. As shown in the table above and Fig. R3, our method significantly outperforms Consistent4D.\n\n**W7: Minor: The paper would benefit from more precise writing.**\n\nThank you for pointing out the writing problems. We will revise the training details in lines 176-182 and clarify the annotations in Eq.10. Furthermore, we will simplify and refine the method section to enhance readability.\n\n**Q1: The paper assumes access to a multi-view video dataset.**\n\nThis question is related to **W1**. Please refer to **W1** for detailed response.\n\n**Q2: The rationale behind the need for SDS.**\n\nWe utilize SDS to optimize dynamic NeRF, enabling effective rendering from any novel viewpoint across the temporal dimension, which is essential for 4D generation. This is particularly challenging for our multi-view video diffusion model, as it is limited to generating only four orthogonal viewpoints at a time, which hinders its ability to produce consistent videos from any novel viewpoint.\n\n**Q3: Why not fit the dynamic NeRF directly on the generated multi-view videos？**\n\nOur multi-view video diffusion model can only produce four orthogonal viewpoints at one time. Training with such sparse views often results in overfitting to the training viewpoints, as proved in SPARF[3]. To further demonstrate this challenge, we conducted an experiment where we used only four generated videos to optimize dynamic NeRF. The results, which illustrate the overfitting problem, are presented in Fig.R4 in the attached PDF.\n\n[1]Ren J, et al. L4GM: Large 4D Gaussian Reconstruction Model. arXiv preprint arXiv:2406.10324, 2024.\n\n[2]Liang H, et al. Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models. arXiv preprint arXiv:2405.16645, 2024.\n\n[3]Truong P, et al. SPARF: Neural radiance fields from sparse and noisy poses. CVPR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5RtyZz6UF9", "reviewer": "Reviewer_4Nq2", "summary": "The paper proposes 3D-aware diffusion model trained on a curated 4D dataset for video-to-4D generation. 4D-aware Score Distillation Sampling loss is introduced to optimize 4D representation parameterized by dynamic NeRF. The proposed framework outperforms optimzation-based baselines.", "questions": "- I appreciate the authors' effort in manually filtering the dataset. I'm interested in how many objects were there before filtering out the 966 objects?\n- How are the input videos for evaluation constructed? Are they ground truth videos from the Objaverse dataset?\n- What are the reference videos for calculating the FVD metrics? Are they ground truth videos from the Objaverse dataset? For table 2, are the reference videos the same?", "limitations": "Please refer to weakness.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- A new subset of animatable Objaverse is presented and improves the model's generation ability as shown in Table 2.\n- The proposed 4D diffusion model outperforms optimization-based baselines.", "weaknesses": "- The proposed method requires 12 hours on A100, which is significantly longer than baselines.\n- I am confused by the quantitative evaluation design as mentioned in L244. \"we calculate FVD between the input video and \nsynthesized videos to evaluate the video quality\". What's the purpose of comparing against input videos? If my understanding is correct, the ground truth 4D objects from Objaverse can be rendered into ground truth videos and can be used to calculate FVD, right?\n- The proposed anchor loss seems very similar to the ones proposed in 4DGen[62] and is not properly discussed in L200.\n- The back views in Fig. 5 seem very blurry and contain transparent edges. What might be the reason for these artifacts? Is this because of SDS loss?\n- I am mainly concerned about the test set construction. Does this manually filtered training set overlap with Consistent4D test set? Many of the Consistent4D test sets are originally from Sketchfab.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for providing insightful suggestions. Below, we address your constructive comments individually.\n\n**W1: The proposed method requires 12 hours on A100, which is significantly longer than baselines.**\n\nOur method focuses on generating high-quality, spatial-temporally consistent 4D content. Therefore, we use dynamic NeRF, which is compact and high-capacity. However, it involves time-consuming volume rendering that extends the optimization time compared to other methods. Although 4DGS can significantly reduce optimization time, it may produce blurred appearances and inaccurate geometries due to the explicit characteristics of Gaussians, as illustrated in Fig.4.\n\n**W2: The quantitative evaluation design.**\n\nSorry for the confusion. Here, we follow the setting of Consistent4D. For each generated 4D content, it is challenging to accurately assess video quality when computing FVD with ground truth videos rendered from Objaverse due to the potential distribution gap. Therefore, we use the input video as the reference video to compute the FVD, which provides a more precise evaluation for each test case.\n\n**W3: The proposed anchor loss seems very similar to the ones proposed in 4DGen[62] and is not properly discussed in L200.**\n\nThanks for your important comments. Our anchor loss differs from 4DGen in several key aspects:\n- **Model and Consistency:** 4DGen employs a 3D-aware diffusion model, SyncDreamer, to generate multi-view images for each frame of the input video. This approach may lead to temporal inconsistencies. In contrast, our method uses a multi-view video diffusion model to produce videos with improved spatial-temporal consistency, resulting in better overall performance for 4D generation.\n- **Viewpoint Selection:** 4DGen utilizes all viewpoints generated by SyncDreamer to supervise the optimization process. However, viewpoints that are far from the input video may have lower quality, potentially degrading performance. To mitigate this issue, we select the viewpoint closest to the input video as the anchor. This strategy ensures that the anchor video maintains high quality and consistency as illustrated in Fig.R2 in the attached PDF.\n\nWe will discuss this in our revised version.\n\n\n**W4: What might be the reason for these artifacts in Fig.5? Is this because of SDS loss?**\n\nFigure 5 presents the multi-view video generation results from 4DM, where DDIM was used for sampling. It is important to note that SDS loss was not applied during the sampling process, so the observed artifacts are not due to SDS loss. Despite these artifacts, our approach achieves results that are comparable to, and even better than, those obtained with ImageDream as shown in Tab.2. We believe these artifacts may be attributable to limitations in the base model, which also produces results with transparent edges. Using a stronger base model could potentially enhance performance.\n\n**W5: Main concern: Test set construction.**\n\nSorry for making these unclear. During the training of our multi-view video diffusion model, we ensure that our training data does not overlap with the test data used for evaluating our method.\n\n**Q1: How many objects were there before filtering out the 966 objects?**\n\nThank you for your appreciation. Before filtering, the dataset contained a total of 44,000 objects, which represents the number of animated shapes from Objaverse 1.0.\n\n**Q2: How are the input videos for evaluation constructed? Are they ground truth videos from the Objaverse dataset?**\n\nSpecifically, for experiments in Section 4.1, We use 3 real-world videos and 3 synthetic videos from the Consistent4D dataset, as well as 3 images from the ImageDream project page. As discussed in Section A.1 of the supplementary materials, for text-image pairs from ImageDream, we utilize SVD to generate monocular videos for 4D generation.\n\nFor the qualitative experiments in Sec.4.2, we use the same input videos as in Sec.4.1. For the quantitative experiments in Sec.4.2, we utilize six monocular videos from the Consistent4D test dataset, which provide ground truth for novel viewpoints. While some of these input videos for evaluation may come from the Objaverse dataset, none of them were used in training our multi-view video diffusion model.\n\n**Q3: What are the reference videos for calculating the FVD metrics? Are they ground truth videos from the Objaverse dataset? For table 2, are the reference videos the same?**\n\nIn Tab.1, the reference videos used for calculating the FVD metric are the input videos, as there is no ground truth available for novel viewpoints. In Tab.2, the reference videos for the FVD metric are the ground truth videos that share the same viewpoints as the rendered videos. Although these input videos for evaluation may come from Objaverse dataset, none of them are used for training our multi-view video diffusion model. The reference videos are the same for each method in Tab.2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mNGRqlKOYP", "reviewer": "Reviewer_MfEx", "summary": "The paper propose a 4D generation pipeline, namely 4Diffusion, aimed at generating spatial-temporally consistent 4D content from a monocular video.The authors design a unified diffusion model tailored for multiview video generation by incorporating a learnable motion module into a frozen 3Daware diffusion model to capture multi-view spatial-temporal correlations. After training on a curated dataset, the diffusion model acquires reasonable temporal consistency and inherently preserves the generalizability and spatial consistency of the 3D-aware diffusion model.", "questions": "1. The improvement from this multi-view generation model is limited, the colors are still flickering, especially from the generated multview frames. \n\n2. I wonder if the limited consistency is due to insufficient amount of training data.", "limitations": "The paper hasn't show the full potential of the multi-view video for 4D generation, sourcing more synthetic data is important to improve the results.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes to generate multiview video to guide the 4d generation, the direction is reasonable.\n2. The paper is easy to follow.", "weaknesses": "1. The paper's novelty wouldn't be its biggest strength, but training a multiview-video module is a good direction so, this point is moderately pass the bar of NeurIPS.\n2. The results look temporally inconsistent (color flickering), see the frog man's eye and wolf (w/ rider) 's tail. This might due to insufficient training samples.\n3. As another nerf based model, the results are not much better than consistent4D.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for providing insightful comments. Below, we address your constructive comments individually.\n\n**W1: The paper's novelty wouldn't be its biggest strength, but training a multiview-video module is a good direction so, this point is moderately pass the bar of NeurIPS.**\n\nThank you for your recognition. Our paper primarily focuses on generating 4D content using our multi-view video diffusion model. Although the multi-view video diffusion model is an important contribution of our method, it can only generate four orthogonal viewpoints at a time. This restriction makes it challenging to produce consistent videos from any novel viewpoint, which is essential for 4D generation. Therefore, the 4D-aware SDS loss and anchor loss are also important for our model, which can enhance the spatial and temporal consistency of the generated content, allowing for effective rendering from any novel viewpoint across the temporal dimension.\n\n**W2: The results look temporally inconsistent (color flickering), see the frog man's eye and wolf (w/ rider) 's tail. This might due to insufficient training samples.**\n\nWhile we acknowledge that there is room for improvement in our results, our method still outperforms the baselines, demonstrating its effectiveness. Additionally, we have conducted further experiments to assess our method more thoroughly. We select 5 multi-view videos that are not included in the training data as test data. The following table presents the results, where metrics such as FVD, LPIPS, and PSNR clearly demonstrate that our method significantly outperforms the baselines. \n\n| **Model**                     | **CLIP-I↑**   | **CLIP-C↑**| **FVD↓**      | **LPIPS↓**         | **PSNR↑**         |\n|------------------------------|---------------|-----------|---------------|--------------------|-------------------|\n| **Consistent4D**              | 0.9216        | 0.9723    | 706.07        | 0.1593             | 16.70             |\n| **DreamGaussian4D**           | 0.8898        | 0.9710    | 760.18        | 0.1793             | 15.97             |\n| **4D-fy**                     | 0.8658        | 0.9487    | 1042.3        | 0.2254             | 14.24             |\n| **Ours**                      | **0.9310**    | **0.9798**| **417.63**    | **0.1199**         | **19.07**         |\n\nHere, we use ground truth videos for novel viewpoints to compute these metrics. As illustrated in the first two rows of Fig.R3 in the attached PDF, Consistent4D and DreamGaussian4D encounter the multi-face problem while our method generates spatial-temporally consistent contents.\n\nWe believe that curating more high-quality multi-view video datasets for training our multi-view video diffusion model could further enhance performance. However, there were very few high-quality multi-view video datasets available at the time of our project. We made every effort to curate as many datasets as possible to train our model.\n\n**W3: As another nerf based model, the results are not much better than consistent4D.**\n\nThis weakness is related to **W2**. We have conducted further experiments to evaluate our method. As shown in the table above and Fig. R3 in the attached PDF, our method significantly outperforms the Consistent4D and other baselines. As illustrated in the first two rows of Fig.R3 in the attached PDF, Consistent4D and DreamGaussian4D encounter the multi-face problem. Consequently, when computing metrics between the input video and synthesized videos, as shown in Tab.1 of the main paper, the performance gap between our method and the baselines may not be apparent.\n\n**Q1: The improvement from this multi-view generation model is limited, the colors are still flickering, especially from the generated multview frames.**\n\nThis question is related to **W2**. Please refer to **W2** for detailed response.\n\n**Q2: I wonder if the limited consistency is due to insufficient amount of training data. The paper hasn't show the full potential of the multi-view video for 4D generation, sourcing more synthetic data is important to improve the results.**\n\nWe believe that curating more high-quality multi-view video datasets for training our multi-view video diffusion model could indeed enhance performance. However, there were very few high-quality multi-view video datasets available at the time of our project. We made every effort to curate as many datasets as possible to train our model. Moreover, we believe that leveraging more robust and powerful 4D representations could further improve results. We plan to explore these enhancements in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the detailed and constructive feedback from all the reviewers. We are pleased that reviewers recognize our contribution (Y3vx, 4Nq2) and the effort involved in filtering the dataset (4Nq2). Additionally, the reviewers acknowledge the significance (Y3vx) and interest (ioTR) of the addressed problem, and consider the proposed method reasonable (MfEx). Furthermore, reviewers find our paper well-written (ioTR, MfEx) and easy to follow (ioTR).

Before addressing the individual reviews, we briefly summarize our responses as follows:

- Experiment Improvements:
    - We utilize more accurate settings to evaluate our method on multi-view video generation.
    - We have conducted additional experiments on 4D generation, using LPIPS and PSNR metrics to evaluate our method against baselines.
    - We demonstrate the effectiveness of 4D-aware SDS by directly optimizing on generated videos.
- Clarification of Technical Details:
    - Provide a detailed explanation of the design of our motion module.
    - Clarify the construction of our training dataset for the multi-view video diffusion model.
    - Describe the construction of our test dataset for each experiment.
    - Detail the reason for selecting anchor videos.
- Writing Improvements:
    - We will clarify the annotations used in the paper and simplify and refine the method section to enhance readability in the revised version.

We provide the qualitative results of additional experiments and the detailed architecture of the motion module in the attached PDF.

Meta-review: This paper tackles the task of 4D reconstruction from monocular video. The approach extends a pretrained 3D-aware diffusion model to work with multi-view viedos by incorporating a learnable motion module into the frozen 3D-aware backbone to capture multi-view spatial-temporal correlations. The model is fine-tuned using 1,000 synthetic multi-view videos from the Objaverse dataset. Then, score-distillation sampling (SDS) is used to generate a dynamic radiance field.

The paper received borderline ratings. The reviewers consider the addressed problem timely and relevant and the technical contribution interesting to the community. Initially, the main concerns of the reviewers were the marginal improvements over one of the baselines (Consistent4D), and lacking details for training / test split of the data. The authors resolved these concerns with their rebuttal. 

I hence recommend acceptance.
Decision: Accept (poster)