Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models
Authors: Zeyu Yang
Zijie Pan
Chun Gu
Li Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: 4D generation
diffusion model
generative model
video diffusion
Primary area: generative models

Abstract: Recent advancements in 3D generation are predominantly propelled by improvements in 3D-aware image diffusion models. These models are pretrained on Internet-scale image data and fine-tuned on massive 3D data, offering the capability of producing highly consistent multi-view images. However, due to the scarcity of synchronized multi-view video data, it remains challenging to adapt this paradigm to 4D generation directly. Despite that, the available video and 3D data are adequate for training video and multi-view diffusion models separately that can provide satisfactory dynamic and geometric priors respectively. To take advantage of both, this paper presents Diffusion$^2$, a novel framework for dynamic 3D content creation that reconciles the knowledge about geometric consistency and temporal smoothness from these models to directly sample dense multi-view multi-frame images which can be employed to optimize continuous 4D representation. Specifically, we design a simple yet effective denoising strategy via score composition of pretrained video and multi-view diffusion models based on the probability structure of the target image array. To alleviate the potential conflicts between two heterogeneous scores, we further introduce variance-reducing sampling via interpolated steps, facilitating smooth and stable generation. Owing to the high parallelism of the proposed image generation process and the efficiency of the modern 4D reconstruction pipeline, our framework can generate 4D content within few minutes. Notably, our method circumvents the reliance on expensive and hard-to-scale 4D data, thereby having the potential to benefit from the scaling of the foundation video and multi-view diffusion models. Extensive experiments demonstrate the efficacy of our proposed framework in generating highly seamless and consistent 4D assets under various types of conditions.

Published as a conference paper at ICLR 2025
DIFFUSION?: DYNAMIC 3D CONTENT GENERATION VIA SCORE COMPOSITION OF VIDEO AND MULTI- VIEW DIFFUSION MODELS
Zeyu Yang, Zijie Pan; Chun Gu, Li Zhang"
Fudan University
ABSTRACT
Recent advancements in 3D generation are predominantly propelled by improve- ments in 3D-aware image diffusion models. These models are pretrained on internet-scale image data and fine-tuned on massive 3D data, offering the capa- bility of producing highly consistent multi-view images. However, due to the scarcity of synchronized multi-view video data, it remains challenging to adapt this paradigm to 4D generation directly. Despite that, the available video and 3D data are adequate for training video and multi-view diffusion models separately that can provide satisfactory dynamic and geometric priors respectively. To take advantage of both, this paper presents Diffusion", a novel framework for dynamic 3D content creation that reconciles the knowledge about geometric consistency and temporal smoothness from these models to directly sample dense multi-view multi-frame images which can be employed to optimize continuous 4D represen- tation. Specifically, we design a simple yet effective denoising strategy via score composition of pretrained video and multi-view diffusion models based on the probability structure of the target image array. To alleviate the potential conflicts between two heterogeneous scores, we further introduce variance-reducing sam- pling via interpolated steps, facilitating smooth and stable generation. Owing the high parallelism of the proposed image generation process and the efficiency of the modern 4D reconstruction pipeline, framework can generate 4D content within few minutes. Notably, our method circumvents the reliance on expensive and hard-to-scale 4D data, thereby having the potential benefit from the scaling of the foundation video and multi-view diffusion models. Extensive experiments demonstrate the efficacy of our proposed framework in generating highly seamless and consistent 4D assets under various types of conditions.
INTRODUCTION
Spurred by the advances from generative image models (Ho et al., 2020; Song et al., 202 Karras et al., 2022; Zhang et al., 2023), automatic 3D content creation (Poole et al., 2023; Wang et al., 2023; Tang et al., 2024b; Hong et al., 2024) has witnessed remarkable progress in terms of efficiency, fidelity, diversity, and controllability. Coupled with the breakthroughs in 4D representation (Yang et al., 2024; Wu et al., 2024a; Duan et al., 2024; Li et al., 2024), these advances further foster substantial development in dynamic 3D content generation (Singer et al., 2023; Bahmani et al., 2024; Jiang et al., 2024b; Zhao et al., 2023; Ren et al., 2023; Gao et al., 2024), which holds significant value across wide range of applications in animation, film, game, and Versee
Recently, 3D content generation has achieved considerable breakthroughs in efficiency. Some works (Liu et al., 2023; 2024; Shi et al., 2023; Wang Shi, 2023: Tang et al., 2024c) inject stereo knowledge into the image generation model, enabling these 3D-aware image generators to
'Equally contributed 11.i Zhang (lizhangfd@ aaudnnedu.ccm is the corresponding author with School of Data Science, Fudan Uni- versity.

---

Published as a conference paper at ICLR 2025
Figure 1: Diffusion" is designed to generate dynamic 3D content by creating dense multi-frame multi-view image matrix in highly parallel denoising process by combining the foundation video and multi-view diffusion model. Please refer to supplementary demo video for more results.
produce consistent multi-view images, thereby effectively stabilizing and accelerating the optimiza- tion. Other efforts (Hong et al., 2024; Chen et al., 2024; Voleti et al., 2024; Zuo et al., 2024; Tang al., 2024a) attempt to directly generate 3D representations, such triplane (Chen et al., 2022) or 3DGS. However, the efficiency improvement from these works is largely data-driven (Wu et al., 2023; Reizenstein al., 2021; Yu et al., 2023; Deitke al., 2023). Consequently, it infeasible to adapt them to 4D generation due to the scarcity of synchronized multi-view video data. Therefore, most existing 4D generation works (Jiang et al., 2024b; Yin et al., 2023; Ren et al., 2023) still adopt the score distillation sampling (SDS) approach and suffer from slow or unstable optimization.
However, despite the paucity of 4D data, there are vast available monocular video data and static multi-view image data. Existing works have demonstrated that it is feasible to train diffusion-based generative models learning the distribution of these two classes of data independently (Voleti al., 2024; Liu et al., 2024; Tang et al., 2024c; Blattmann et al., 2023b;a). Considering that video dif- fusion model stores the prior of motion and temporal smoothness, and multi-view diffusion model has sound knowledge of geometrical consistency, combining the two types of generative models to generate 4D assets becomes a highly promising and appealing approach.
To leverage both of them, we propose novel 4D generation framework, Diffusson?, which rec- onciles the video and multi-view diffusion priors to directly sample multi frame multi-view im- age arrays imitating the photographing process of 4D object. Instead of unleashing the pre-trained knowledge stored in model parameters through fine-tuning (Xie et al., 2024; Ren et al., 2024), Dif- fusion2 achieves this goal that the score function for the joint distribution of elements in image array can be approximated by the convex combination of scores for each view and frame due the conditional independence within the image matrix. Therefore, we can directly sample multi-view multi-frame images within the framework of score-based generative models with off -he-hhh-heff score estimators.
training-free free and architettur--gnoosiic manner. Our key insight is
In practice, due to the potential divergence between the learned distributions of two foundational models caused by imbalanced training data, conflicts could emerge during the denoising process. Such conflicts are primarily manifested as two distinct modes in the denoising results, which may compromise the completeness and consistency of the generated images with continuously reducing noise levels. However, we found that this issue can be effectively resolved by leveraging the inherent properties of the diffusion framework, without requiring any additional training. Based on this idea, we further propose the variance-reducing sampling (VRS), which can effectively mitigate such conflicts at the cost of negligible additional time consumption by reconciling the distinct modes at a higher noise level. With this technique, Diffussion can produce smooth multi-view videos with
2

---

Published as a conference paper at ICLR 2025
high spatial-temporal consistency. Furthermore, we introduce a dynamic and texture- reconstruction strategy that transforms the generated discrete image matrix into a continuous 4D representation while minimizing the potential detail loss caused by aggressive variance reduction, especially on images that are far from the reference view. Thanks to the highly parallel denoising in the image matrix synthesis stage and the efficiency of the reconstruction stage, our method can generate figh-fidelity and diverse 4D assets within just few minutes.
Our contributions can be summarized as follows: (i) We develop a novel 4D generation framework that can generate highly consistent multi-view videos in a single pass of reverse diffusion process using only existing multi-view and video diffusion models, without relying on any 4D dataset. The core of this framework is to estimate the score function of the image matrix by a convex combination of the scores predicted by orthogonal diffusion models. We identify the conditional independence within elements constituting the image arrays and establish the theoretical soundness of the proposed approaches based on this property. (ii) We propose the VRS for reconciling heterogeneous scores during denoising, which can effectively alleviate potential conflicts and facilitate the generation of seamless results with high consistency across multiple views and frames. In synergy with it, we also propose a dynamic and texture decoupling reconstruction pipeline better preserve texture details. (iii) Systematic experiments demonstrate that our proposed method achieves impressive results under different types of prompts. (iv) Notably, our work first proves that it is practical to directly sample highly spatial-temporal consistent multi-view videos of dynamic 3D objects within a single-pass pass of denoising diffusion procedure, while only using the existing multi-view and video diffusion prior without any additional training on expensive and hard-to-scale 4D dataset.
2 RELATED WORK
3D generation 3D generation aims at creating static 3D content from different type of condi- tions like text or reference image. Early efforts employed GAN-based approaches (Gao et al., 2022; Schwarz et al., 2020). Recently, significant breakthroughs have been achieved with diffusion mod- els (Ho et al., 2020). DreamFusion (Poole et al., 2023) introduced score distillation sampling (SDS) to unleash the creativity in diffusion models. A series of subsequent works ( (Wang et al., 2023; Shi et al., 2023; Wang & Shi, 2023; Pan al., 2024a; Tang et al., 2024b; Yi et al., 2024) continuously address challenges such as multi-face Janus issues and slow optimization. On the other hand, with the development of large scale 3D datasets (Yu et al., 2023; Deitke et al., 2023), many works try to directly generate 3D contents by using diffusion models. Some studies (Nichol et al., 2022; Jun & Nichol, 2023; Hong et al., 2024; Tang et al., 2024a; Wang et al., 2024) have explored the di- rect generation of 3D representations. Another line of research (Liu et al., 2023; 2024; Long et al., 2023; Chen et al., 2024; Tang et al., 2024c) focuses on generating multi-view images with sufficient 3D consistency for reconstruction. Our method also directly generates consistent images for recon- struction. But unlike 3D counterparts, there is no large-scale synchronized multi-view video data. Therefore, opt combine existing geometrical consistency priors and video dynamic priors.
Video generation Recent diffusion--aasee video generation methods have exhibited unprecedented levels of realism, diversity, and controllability. Video LDM (Blattmann et al., 2023h) is the pioneer work to apply the latent diffusion framework (Rombach et al., 2022) to video generation. The sub- sequent work SVD (Blattmann al., 2023a) followed its architecture and made effective improve- ments to the training recipe. W.A.L.T (Gupta et al., 2023) employed a transformer with window at- tention tailored for spatiotemporal generative modeling to generate high-resolution videos. VDT (Lu et al., 2024) introduced the video diffusion transformer and a spatial-temporal mask mechanism to flexibly capture long-distan spatiotemporal context. The recently introduced SORA (Brooks et al., 2024) demonstrated remarkable capability to generate long videos with intuitively physical fidelity. Models trained on large-scale video data can synthesize realistic dynamics. Besides, video diffusion models can also provide effective prior for 3D generators (Chen ct al., 2024; Han et al., 2024; Voleti et al., 2024). Therefore, we build our method on this flourishing domain.
4D generation Animating category-agnostic objects is challenging and has drawn considerable attention from both academia and industry. Unlike 3D generation, 4D generation requires both con- sistent geometry and realistic dynamics. Recent works on this domain can be categorized based on the input condition. Some of them create 4D content from text or single image. For instance, MAV3D (Singer et al., 2023) first employs SDS in text-to-4D tasks. 4D-fy (Bahmani et al., 2024)
3

---

Published as a conference paper at ICLR 2025
Denolser
Nult-View Diftusion Model
RetegpOlus)
Intont
reprosentation
Diflusion
Mutti Viow
Danolzer
Orthegonal condition generation
(8) Paraliel denoising for synchronized nultivicw videos
(i0 Optimization of 4D represantation
The overall pipeline of Diffission (i) Given a reference image, Diffusion first indepen- dently generates the animation under the reference view (denoted Iı.v:o) and the multi-view images at the reference time (denoted Io as the condition for the subsequent generation of the full matrix, denoted I. Depending on the type of given prompt, the condition images Iı. or I: can be specified by users. (ii) Then, Dijffusion" directly samples dense multi-fraan fult-fframe multi-view image array by blending the estimated scores with weighting scheduler from pretrained video and multi-view diffusion models the reverse diffusion process. (iii) The generated image arrays are employed as supervision to optimize a continuous 4D content representation.
Figure 2:
hybridizes different diffusion priors during SDS training. But they suffer from extremely slow gen- eration. DreamGaussian4D (Ren al., 2023) adopts deformable 3D Gaussian (Yang et al., 2024) as the underlying 4D representation and exports mesh for texture refinement, significantly improv- ing efficiency. Another line of work (Jiang et al., 2024b; Wu et al., 2024b; Yin et al., 2023; Pan et al., 2024b) predicts dynamic objects from a single- video with largely dictated motion. Con- sistent4D (Jiang al., 2024b) proposes to use SDS approach for geometry consistency and frame interpolation loss for temporal continuity. 4DGen (Yin et al., 2023) further grounds the 4D con- tent creation with pseudo labels. Meanwhile, L4GM (Ren et al., 2024) directly predicts consistent 3D Gaussians for each frame. And Jiang et al. (2024a) animates existing 3D assets given their multi-view rendering with 4D-SDS. Efficient4D (Pan et al., 2024b) mimics a photogrammetry-baeee neural volumetrie video reconstruction pipeline by directly generating multi-view videos for recon- struction. Some recent works opt for training diffusion models to generate monocular video (Liang al., 2024) or multi-view videos (Xie et al., 2024) with spatiotemporal consistency for subsequent explicit reconstruction. Compared to previous works. our framework can efficiently generate di- verse 4D contents from different input prompts, avoiding the slow and unstable optimization and have potential to continuously benefit from the scalability of underlying diffusion models.
3 METHOD
As depicted in figure 2, the proposed 4D generation framework Diffusion2 adopts a two-stage pipeline: dense observation synthesis followed by reconstruction. In this section, we will first elab- orate on how to generate dense multi-view multi-frame images for reconstruction through highly parallelizable denoising process by reconciling the pre-trained video diffusion prior and multi-view diffusion prior while pointing out why it is feasible (stage- 1), and then briefly introduce how to robustly reconstruct 4D content from the sampled images (stage-2).
3.1 ESTIMATING SCORES OF THE IMAGE MATRIX
In this stage, our goal is to generate highly consistent dense multi-frame multi-view images for reconstruction, which can be denoted as matrix images:
I
:
I= {1g E RxxWxx V.F
4
I
I
(1)
Ji=1,==
:
Iv
Iv.
4

---

Published as a conference paper at ICLR 2025
where V is the number of views, F is the number of video frames, and (H, W) is the size of images. We aim to construct a generative model that allows us to directly sample natural I ~ P(Z).
Now, let us first divert our focus to reviewing existing diffusion- basodd generators for video and multi- view images, which can be utilized for sampling realistic image sequence through the probabilistic flow ODE as below:
dx = - logp (()) dt.
(2)
Here, x = (1: E [HxWx3] J is series of images with N frames or N views, Vx lologp (x: σ)) is the score functtin, which can be estimated as Vx log (x; s (Do(x; a)) ( /o2 (Karras et al., 2022; Blattmann et al., 2023a), where ox;x;r) is neural network trained via denoising score matching. We want to extend the above formulation to the generation of the whole image matrix, i.e., sam- pling I through equation 2 according its score function V log p (I; (t)). The question is, how do we estimate the score of the joint distribution of V X F images? Due to the scarcity of available synchronized multi-view videos and the potentially huge memory footprint of simultaneously con- suming V x F' images, it is impractical to train a neural network to directly predict the score function of an image matrix with densely distributed views and sufficient frames for substantial motion.
Fortunately, image matrix I in nature has a nice property, such that we can approximate the V log p (I;o(() by combining existing video and multi-view score estimators, as claimed in the following theorem:
Theorem 3.1. For x ligj we have
Vx logp(I;o(()) Vo
(ogpoooo)))ttooooooooo....o.oo. - Vzlog(((,sio(()). (3)
Here Vx log,(Tr,,1:1) and V log [xlog((((rv)))) can be predicted by the off-the-shelf video diffusion model and the vieti-view diffusion model. This assertion implies that we can sample the desired image matrix by progressively denoising from pure Gaussian noise using the summation of two estimated scores for its row and column, which can be obtained from the pre-trained multi-view and video diffusion models respectively. The theorem 3.1 can be derived from the following assumption about the structure of p(I) (refer to appendix A.3 for detailed proof):
Assumption 3.1. images captured from different viewpoints and different times are conditionally independent given the image the intersection of their respective rows and columns i.e.
p ( (T.s) = p (Te,y (1.5) with 1' +i,5' #
(4)
We provide an example in figure 3 to illustrate this assumption. Denote Itj as the front view observed currently, it can indeed dictate some aspects of the back view at the same frame Ir. such as its contour. The conditional indepen- dence means that the future front view L, can- provide more information about the T be- yond those have been provided by the current front view. aligns with our intuition, as the differences between li and Li, are mainly in- duced by the hand waving, which is incapable of bringing extra information gains about the back view. Similarly, the texture of the back at the current moment is also not helpful in infer- ring the future dynamics.
Shape information
Texture information
lij
Unknown texture
Unknown
movement
lij
Actually, the assumption of conditional inde- pendence also implicitly held by many other 4D generation pipelines. For instance, the le- gitimacy of independently synthesizing multi- view reference and driving video (Yin et al., 2023; Xie et al., 2024) from a single image also relies on this hypothesis.
Figure 3: Schematic illustration for the condi- tional independence. Compared to It,j the ad- ditional information (future dynamic) in Ii does not help in inferring the unknown texture of the back view Tr. j. and vice versa.
5

---

Published as a conference paper at ICLR 2025
Sampling in latent space For simplicity, the previous discussion assumes that sample images directly in the RGB space. However, modern high-resolution diffusion models typically gener- ate images in latent space encoded by pre-trained Auto-Eneoe Encoder (Kingma & Welling, 2014; Van Den Oord et al., 2017). The legitimacy of the aforementioned derivation requires that the multi- view and video generative models share the same latent space. Therefore, we employ SV3D (Voleti et al, 2024) and SVD (Blattmann et al., 2023a) as the respective generators in this work. We believe that the requirement will be increasingly satisfied by more models in the future. As pointed out by Blattmann et al. (2023a); Voleti et al. (2024); Chen et al. (2024), video generative models trained on large-scald video datasets have learned a strong geometry prior and thus capable of providing a better pre-training for multi-view diffusion models than those trained solely on image data.
3.2 GENERATION UNDER VARIOUS CONDITIONS
Note that the formulation described above is based on unconditional generation. However, we are more interested in controllable generation in practice. In this section, will discuss how to extend the above process to the conditional generation.
First of all, since the Vx p(.,)) is intractable in the conditional generation, combination of Vx lo.g((I.(...p)) and Vx log (((.() to replace it, and revise the equation 3 as:
the convex
Vx log (I; = sV (ogp(Z
(1 (og (t)).
(5)
Then we formulating our conditional generation pipeline like a "inpainting" process for the aug- mented matrix Iugg defined as
10.0 [[I:F),0
Io. I
Zang
(6)
Compared to the unconditional generation, the augmented matrix Iaug includes the optional inputs Iu,o, [Iv and I(I:F).0 representing reference geometry and dynamic target object.
In the proposed generation pipeline, Io. (1:V). : according to the given input. Then the rest part of Zaug will be sampled given these orthogonal con- ditions. To incorporate them into the generation of I, we condition the score estimator for demoising each row/column in equation 5 on the first element of the same row/column in Tang- This formu- lation enables us to handle various 4D generation tasks depending on different forms of conditions specified by users. For example, in the image-to-4D task, given single image 1o,0 as input, both [1)( and Z(I=F), should be generated by multi-view and video diffusion models respectively. In the video-to-4D task, we just need to leave the input single-view video as I(o:F)... and use its last frame Io.0 as the condition for multi-view diffusion model generate (I.().)- These process can also be similarly extended to the text-to-4D) tasks and end-to-end animating static 3D objects. Once (I.V I[1:F).0 are obtained, we can sample the full matrix I using equations (3) and (5).
should be first created as the known part
Parallel denoising Assumption 3.1 ensures the safety of independently generating the geometry [I.( and the motion I(1:F).- Since these two generation processes have no computational or data dependency, their total time cost could be reduced to a single reverse diffusion process. Additionally, when we denoise the rest part of Iang. i.e., I, the score estimation for each row and column can also be parallelized in each denoising step. Therefore, with sufficient GPU memory, the total time spent figure 2 (ii) remains the same as that for generating a single video.
3.3 RECONCILING JOINT DENOISING
In practice, the training data for foundational multi-view and video generation models may exhibit a domain gap and imbalance in quantity; for instance, the multi-view diffusion model employed in this work is fine-tuned on an object-centric dataset which is much smaller than the large-scale general video dataset used to train the video diffusion model. Consequently, the learned distributions may not aligned ideally, leading to potential conflicts when directly fusing their scores without taking some measures to harmonize their denoising trajectories. A straightforward solution is to fine- tune the utilized video diffusion model on single-view videos of dynamic objects. However, such conflicts can also be mitigated by harnessing the capabilities inherent in the diffusion framework without extra training. Consequently, to preserve our training- frree merit and avoid reliance on multi- view video data, we propose the following inference-time approach reconcile them.

---

Published as a conference paper at ICLR 2025
Variance-reducing sampling via interpolated steps Due to the potential conflicts between the heterogeneous scores mentioned above, I' may not always consistently conform to the ideal distri- bution of p(I, (t)). Obvious flicker and ghosting artifacts will appear with the accumulation of the divergence step by step. Therefore, we proposed an ingenious strategy to resolve this issue, which involves incorporating some "intermediate" steps into the reverse diffusion process. Specifically, we observe that although I' may deviate from p(I, (t)), , it still possesses a high likelihood at a larger noise level. This can be understood from the following perspective: deviations between I' and p(Z, (t)) are primarily exhibited as two isolated modes driven by independent score estima- tors. However, these can be mixed in a mollified distribution with a higher noise level, which is actually one of the initial motivations of the diffusion model (Song & Ermon, 2019). Therefore, the core insight of the proposed strategy is to deceive the diffusion model to denoise noisy latent with smaller variance at higher noise level. To balance efficiency and quality, we achieve this goal using the following approach. In the t-th denoising step, we first update I' to I++1 based on the estimated score from equation 5. Then,
a forward process to resample noisy latent at noise levels between t and + 1. We then use it as I' to repeat the previous denoising step. Algorithm provides the pseudo-code for the sampling process. Since the divergence primarily occurs at the early stage of denoising, we only need to perform the rollback in the first five denoising steps, which only brings about 1/10 extra inference steps, and the experiments demonstrate that it is sufficient to achieve very seamless and consistent results.
3.4 RECONSTRUCTION FROM SAMPLED IMAGE MATRIX
Given the multi-view videos synthesized in stage-1, 1,, it is imperative to transform them into a tinuous 4D representation for facilitating real-world applications. Beyond the functional consider- ations, i.e., enabling renderings from arbitrary viewpoints beyond the images generated in stage- the reconstruction phase (stage-2) is also responsible for enhancing texture details in views that are distant from the reference views and frames, since we have sacrificed some texture diversity on these views with higher uncertainty with the aggressive variance-reducing strategy in stage- for efficiency. Therefore, we propose geometry and texture decoupled scheme to train dynamic Gaussians on the images generated in stage-1 I to minimize the negative impact of this trade-off.
4D Gaussian optimization Specifically, we represent dynamic objects as set of Gaussian primi- tives in canonical space, driven by a learnable deformation field. The deformation field is responsible for predicting the 6-DoF transformation of each Gaussian given its mean and the queried times- tamps. We model it with an MLP (Yang et al., 2024) because of its inherent inductive bias about low-frequency and topological invariance which can effectively regularize the learning of dynamics.
In the optimization, during the initial 3,000 warm-up iterations, we only optimize the underlying static Gaussians using synthetic multi-view images at the reference frame, i.e., Ti..ry. with photo- metric losses. Compared to warm-up with monocular views (Ren et al., 2023), consistent multi-view static images allow for rapidly achieving very high-quality geometry in this stage and bind the refer- ence frame to the canonical space to some extent, thus making it easier for the subsequent potential extraction of deformed mesh. After warm-up, we use the full image matrix as ground truth to op- timize the deformation field while fine-tuning the canonical Gaussians at a reduced learning rate and loss scale away from the reference view and frame. Unlike Ren et al. (2023) optimizing the deformation field with a single-view driving video, our consistent multi-view videos can lead to more stable optimization. Benefiting from the strong fitting capabilities of the Gaussian approach, the trained Gaussian model can achieve high- fidelity details while supporting real-tine -ime viewing.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
In the first stage, we employ Stable Video Diffusion (Blattmann et al., 2023a) as our founda- tion video diffusion model, predicting 25 frames each time conditioning on the reference image. SV3D" (Voleti et al., 2024) is chosen our foundation multi-view diffusion model. For simplicity, we only generate orbital videos with 21 uniformly spaced azimuths and fixed elevation. By default, set the number of sampling steps to 50 for both models. In the reconstruction stage, we assume
7

---

Published as a conference paper at ICLR 2025
INPUT
STAG4D
DGN
Emicienia
Ours sasgge 1) 2und
Ours (stage--2 2)
Eme2
$ie
Emoo
Lma
tind
time
tme
Sine
Time
Figure 4: Qualitative comparisons. The left 3 cases are generated from input videos, whilst the right 2 cases are generated given single input images. The results of our both stages exhibit better coherence, appear more natural, and have fewer artifacts. More results in supplementary video.
that the virtual camera whose FOV is set 33.8 orbiting around the object center with fixed radius of 2m. Finally, We optimized Gaussian Splatting in totally 5,000 iterations in our reconstruc- tion stage. The image size is (576 576) in both stages. All the experiments conducted on 8 NVIDIA A6000 GPUs. Please refer to section A.5 for more details.
4.2 QUALITATIVE COMPARISONS
To demonstrate the effectiveness of the proposed method, we qualitatively compared our method with three representative baselines, ranging across optimization-based. photogrammetty-based, dif- ferent types of diffusion prior and 4D representation. The results on the two commonly used 4D generation task, i.e., video-to-4D and image-to-4D task shown in figure 4. For fair comparison, we use the same reference video for all methods in the image-to-4d task. In terms generation quality, simple and elegant pipeline is capable of generating assets with remarkably superior
8

---

Published as a conference paper at ICLR 2025
User study on image-to-4D generation. The proportions of different methods that best match user preferences under four criteria are reported.
Table I:
Consistent4D
STAG4D
DG4D
Efficient4D
Ours
Ref. consistency
3.1%
5.5%
2.0%
1.1%
88.3 %
Multi-view consistency
6.7%
22.9%
3.1% %
8.4%
58.9 %
Temporal smoothness
8,0%
36.2%
2.0%
10.7%
43.1 %
Overall model quality
8.2%
18.2%
2.0%
12.0%
59.6 %
Table 2: Quantitative comparisons video-to-4D generation. All metrics averaged four ground truth novel views and one input view within the first 25 frames.
Consistent4D
STAG4D
DG4D
Etficient4D
Ours
CLIP score f
0.905
0.920
0.885
0.917
0.922
LPIPS 4
12.81
12.78
16.17
14.66
12.35
FVD L
893.67
855.84
1143.59
873.25
831.32
Gen. time L
120 mins
90 mins
11 mins
12 mins
12 mins
smoothness and temporal consistency, while exhibiting fewer floaters and more complete geometry. The fewer artifacts arise from the ability to directly sample the natural image matrix by harnessing the orthogonal diffusion priors. Additionally, the closeness between both stages indicates that our method achieves commendable spatiotemporal consistency, which is also manifested by the cleaner geometry demonstrated in the additional results from supplementary videos and the appendix. Fur- thermore, the denoising process of the proposed method is highly parallelizable, which provides significant efficiency advantage over those that need recurrent evaluation of diffusion UNet.
4.3 QUANTITATIVE COMPARISONS
We also present quantitative results on two widely explored 4D generation tasks, video-to-4D and image-to-4DD. For video-to-4I) task, we report the LPIPS and CLIP similarity on the test split fol- lowing the previous work (Jiang et al., 2024b), which mainly focus on semantic and perceptual consistency. These metrics are measured 195 within the first 25 frames. The results in table 2 indicate that our method is able to more accu- rately infer geometry from the reference video with good trade-off between generation quality and time. And the lower FVD suggests that our method achieves better temporal coherence. We also conduct
the the views azimuths of 0", -75", 15", 105%, and
study for both video-to-4D and image-to-4D tasks. The survey primarily focused on three aspects of 4D consistency, i.e., the consistency with the reference view (Ref. consistency) and frames (temporal smoothness); addition-
between the different views (Multi-view consistency) ally, we also requested participants to report on their subjective preference of overall model quality. The results are shown in table 1. It suggests that our method garnered the highest preference both the consistency dimension and overall model quality. Under the setting described in section 4.1, our method can complete the generation process within 20 minutes, where the first takes 8m31s. It is significantly faster than the state-of-the-ane SDS- methods with sophisticated multi-stage optimization. is noteworthy that the parallel denoising characteristic could offer an extra trade- off between memory footprint and inference time. The reported times are measured on 8 NVIDIA RTX A6000 GPUs, which have not fully exploited their parallel capabilities. When the number of GPUs is increased 16, the generation time is expected to be halved. On the other hand, can also sacrifice some parallelism to achieve a more friendly memory footprint. Further substan- tial efficiency improvements be achieved by optimizing the multi-GPU communication, which contributes mostly to the gap between the theoretically optimal inference time.
4.4 ABLATION STUDIES
We perform the following ablations to verify the effectiveness of our core designs, and present the results in figure 6. Please refer the videos in supplementary materials for more comprehensive and intuitive comparison.
The impact of variance-reducing sampling We first analyze the effect of the proposed variance- reducing sampling (VRS). In figure 6, we show the images synthesized in the first stage with and without applying this technique. It can be obviously observed that without the variance-reducing

---

Published as a conference paper at ICLR 2025
time
time 2
wio VRS
w/ VRS
Multi-viww diffusion Attention feature blend
Score compasition
wio decoupled
wi docoupled
only
(Ours)
Figure 6: Ablation studies for variance-reducing sampling (left), score composition (middle), and texture- detture-ynamic decoupled reconstruction (right).
sampling, the generated images may exhibit severe ghosting artifacts. This arises from the discrep- ancy between the learned distribution of the two models, which may lead to substantial deviation in the denoising directions, especially the first few steps. To support this claim, we visualize the predicted 20 of the two models in the 4-th step in figure 5, where two distinct modes be seen. These discrepancies can harmonized at higher noise levels via variance-reducing sampling, which ascribed for the effectiveness of VRS.
The impact of score composition The score compo- sition is essential in our framework. To demonstrate its efficacy, we compared the images synthesized three al- ternative settings in stage-1: (1) Generating multi-view images independently for each frame using SV3D; (2)
Blending the key and value of the spatial attention within SV3D in the time dimension using ID conv; (3) Using our score composition as equation 5. We enable variance- reducing sampling for all three settings for fair compari- son. couple of observations can be drawn from figure
Visualization of two isolated modes in the early denoising process.
Figure 5:
(1) While pariance-reducing sampling can control variance to some extent, significant inconsistency still can observed at the most under-constraint back views with only multi-view generative model; (2) Due to the limited receptive field and hand-crafted fixed conv weights, blending the attention feature between adjacent frames each step of denoising is challenging to maintain long-range consistency. Another similar scheme blending the feature of each frame with the reference frame. However, such a pixel-wise mixing approach also struggles to handle the long-range spatial context, resulting in ghosting effects on large movements. In contrast, the score composition can support arbitrary length of temporal and spatial context, thus can generate seamless videos.
The impact of decoupled reconstruction Additionally, investigate the effect of decoupled construction of texture and geometry in the second stage. In the first stage, we adopted an aggressive impact on efficiency, but this led gradual
setting for variance-reducing sampling to minimize blurring of details in images as they moved farther from the reference view. Nevertheless, these details were still relatively well presepereer in the reference frame. Therefore, as depicted in figure 6, our stage-2 2 was capable of effectively restoring them through decoupled reconstruction even for the view farthest from the references.
CONCLUSION
In this work, we introduce Diffusion?, novel framework for creating dynamic 3D content. This framework first generates dense array of multi-view and multi-frame images with high paralleliza- tion, which are then used to build full 4D representation through reconstruction pipeline. The core assumption behind this framework is that the motion of an object viewed from one angle and its appearance from another are conditionally independent. Based on this assumption, we prove that can directly sample synchronized multi-view videos in denoising process by combining pre- trained video and multi-view diffusion models. Our experimental results demonstrate the flexibility and effectiveness of our framework, showing that can adapt to various prompts to produce high- quality 4D content efficiently and effectively. We hope that our work can inspire future research on unleashing the geometrical and dynamic priors from foundation 3D and video diffusion models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work was supported in part by National Natural Science Foundation of China (Grant No. 62376060).
REFERENCES
Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. CVPR, 2024. 1,3
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint, 2023a. 2, 3, 6,7
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023b. 2,3
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. https://openas URL https://
v sonerracc-ooommmdel------------- -or-a--soo------mllatoo-- 2024.
openai.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance
fields. In ECCV, 2022. 2
Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, and Huaping Liu. V3d:
Video diffusion
models are effective 3d generators. arXiv preprins, 2024. 2. 3.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In CVPR, 2023. 2,
Yuanxing Duan, Fangyin Wei, Qiyu Dai, Yuhang He, Wenzheng Chen, and Baoquan Chen. 4d gaussian splatting: Towards efficient novel view synthesis for dynamic scenes. arXiv preprint, 2024.1
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeie, and Sanja Fidler. Get3d: generative model of high quality 3d textured shapes learned from images. In NeurlPS, 2022. 3
Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, and Ulrich Neumann. Gaussianflow: Splatting gaussian dynamics for 4d content creation. arXiv preprint, 2024. 1
Agrim Gupta. Lijun Yu, Kihyuk Sohn. Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models, arXiv preprint, 2023.
Junlin Han, Filippos Kokkinos, and Philip Torr. Vfusion3d: Learning scalable 3d generative models from video diffusion models. arXiv preprint, 2024. 3
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020. 3,3
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In ICLR, 2024. 1, 2
Yanqin Jiang, Chaohui Yu, Chenjie Cao, Fan Wang, Weiming Hu, and Jin Gao. Animate3d: Ani- mating any 3d model with multi-view video diffusion. In NeurlPS, 2024a. 4

---

Published as a conference paper at ICLR 2025
Yanqin Jiang, Li Zhang. Jin Gao, Weimin Hu, and Yao Yao. Consistent4d: Consistent 360 dynamic object generation from monocular video. In ICLR, 2024b. 4, 18
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint, 2023. 3
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. NeurIPS, 2022. 1,5
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014. 6
Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In CVPR, 2024. 1, 19
Hanwen Liang, Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N Platanio- tis, Yao Zhao, and Yunchao Wei. Diffusion4d: Fast spatial-temporal consistent 4d generation via video diffusion models. arXiv preprint, 2024. 4
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; Zero-shot one image 3d object. In ICCV, 2023. 1,
Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang- Syncdreamer: Generating multiview-consistere images from single-view image. In ICLR, 2024. 1, 2,3
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint, 2023. 3
Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: General- video diffusion transformers via mask modeling. In ICLR, 2024. 3
Alex Nichol, Hcewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: system for generating 3d point clouds from complex prompts. arXiv preprint, 2022. 3
Zijie Pan, Jiachen Lu, Xiatian Zhu, and Li Zhang. Enhancing high-resolution 3d generation through pixel-wise gradient clipping. In ICLR, 2024a.
Zijie Pan, Zeyu Yang. Xiatian Zhu, and Li Zhang. Fast dynamic 3d object generation from single- view video. arXiv preprint, 2024b.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. ICLR, 2023. 1,3
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cate- gory reconstruction. In ICCV, 2021. 2
Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dream- gaussian4d: Generative 4d gaussian splatting. arXiv preprint, 2023. 2, 4,
Jiawei Ren, Kevin Xie, Ashkan Mirzaei, Hanxue Liang, Xiaohui Zeng, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, et al. L4gm: Large 4d gaussian reconstruction model. arXiv preprint, 2024. 2,4 4
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 3
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In NeurIPS, 2020.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long. Kejic Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint, 2023. 1,3
12

---

Published as a conference paper at ICLR 2025
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, lurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. arXiv preprint, 2023. 1,
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019. 7
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. gmm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint, 2024a. 2,
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2024b. 1,3, 18
Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan. Mydiffusion+++: A dense high-resolution multi-view diffusion model single or sparse-view 3d object reconstruction. arXiv preprint, 2024c. 2,
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. In NeurlPS, 2017. 6
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Chris- tian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion. arXiv preprint, 2024. 2, 3, 6,
Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation.
arXiv preprint, 2023. 1,3
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NewrlPS, 2023. 1,3
Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image 3d textured mesh with convolutional reconstruction model. arXiv preprint, 2024. 3
Guanjun WU, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In CVPR, 2024a.
Tong Wu, Jiarui Zhang. Xiao Fu, Yuxin Wang. Liang Pan Jiawei Ren, Wayne Wu, Lei Yang, Jiaqi 3d object dataset
Wang, Chen Qian, Dahua Lin, and Ziwei Liu. Omniobject3d: for realistic perception, reconstruction and generation. In CVPR, 2023. 2
Zijie Wu, Chaohui Yu, Yanqin Jiang, Chenjie Cao, Fan Wang, and Xiang Bai. Sc4d: Sparse- controlled video-to-4d generation and motion transfer. In ECCV, 2024b. 4
Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, and Varun Jampani. Sv4d: Dynamic 3d content generation with multi-frame and multi-view consistency. arXiv preprint, 2024. 2, 5
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In CVPR, 2024. 1,4,
Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. In CVPR, 2024.
13

---

Published as a conference paper at ICLR 2025
Yuyang Yin, Dejia Xu, Zhangyang Wang. Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial- consistency. arXiv preprint, 2023. 2, 4,
Xianggang Yu, Mutian Xu. Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvingnet: A large-scale dataset of multi-view images. In CVPR, 2023. 2,
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. Animate 124; Animating image 4d dynamic scene. arXiv preprint, 2023. 18
Qi Zuo, Xiaodong Gu, Lingteng Qiu, Yuan Dong, Zhengyi Zhao, Weihao Yuan, Rui Peng, Siyu Zhu, Zilong Dong, Liefeng Bo, et al. Videomv: Consistent multi-view generation based on large video generative model. arXiv preprint, 2024. 2
A APPENDIX
A.1 LIMITATIONS
Beyond demonstrating the efficacy of the proposed approach, as a preliminary exploration of photogrammetry- bassd 4D generation via zero-shot sampling of multi-view synchronized videos within a single pass of denoising diffusion procedure, this paper acknowledges that it still has some limitations.
First, assumption 3.1 may not strictly hold for some cases. However, minor violations do not sub- stantially affect the quality of generated results. As shown in the experimental results, score composition framework is capable of generating smooth and consistent multi-view videos even for cases adhering this assumption. We provide detailed discussion about its necessity on the correctness of thereom 3.1 in section A.4.
Second, our framework's 's superiority in efficiency originates from its parallelism. actually in- troduces a trade-off between memory and efficiency, which leads to higher maximum inference efficiency crucial for some online tasks, while not resulting in similar advantages on throughput which is important for some offline tasks. We illustrate this trade-off in figure 7. It can be ob- served that even using a single GPU, our method still surpasses representative SDS-based alternates in efficiency by considerable margin.
120
100
generatton bme (min)
80
20
ConsistenHO
STAG4D
number GPUs
Figure 7: Inference time of the first stage with different number of GPUs. For comparison, we also present the inference time two representative SDS-base solutions. They are shown as horizontal lines, because their efficiency cannot benefit from more computes.
14

---

Published as a conference paper at ICLR 2025
Last, since we adopt radical VRS setting for efficiency, texture diversity for some views in image matrix far from the references may be sacrificed for some cases. For example, as can be seen from the second example in figure 4 (a), the model has generated a weird texture-less back for the cat. (Actually, for some input images out training distribution of multi-view prior, the generated multi-view references (1.V) already pose textureless backside view. To more clearly distinguish the extent to which this can be attributed to the proposed sampling strategy, in figure 8, we show the multi-view references of the samples conditioned on generated images in figure 4.)
Figure 8: Multi-view references I.ter of samples from out-of-distribution images in figure 4.
Fortunately, due to details are still preserved in the reference frame, some techniques can be em- ployed to transfer them from the reference frame into those texture-less views during the recon- struction process. We adopt Gaussian-based dynamic-texture decoupling reconstruction pipeline to address this issue. Besides, since this phenomenon does not impact the training of geometry and deformation fields due to its low-frequency nature, further decoupling can be achieved by extracting deformable meshes after their optimization, and independently refining textures using the reference frame. Beyond that, the issue also holds potential for avoidance the first stage, which we leave for future work.
We sincerely hope that our exploration and results could demonstrate the feasibility and potential of zero-shot sampling of multi-view videos using existing diffusion priors, thereby opening up brand- new path for 4D generation that is not constrained by the bottleneck of expensive and hard-to-scale 4D data.
A.2 PSEUDOCODE FOR JOINT DENOISING OF MULTI-VIEN MULTI-FR AME IMAGE MATRIX
VIA SCORE COMPOSITION
A.3 THE PROOF FOR MAIN THEOREM 3.1
Recall that the assumption 3.1 add conditional independence property on the probability structure of Z. However, in the denoising process, the image matrix I is perturbed with Gaussian noise, so we need similar property on the noisy version of I, which leads to corollary A.I:
Corollary A.1. Denote î as the noisy version of I, i.e.,
i = ( HR
V.F HEll== with 11 = cTi. + Eijo
where a E IR. is a constant and Ei, llxx333 are independent Gaussian noises. Then equation also holds for I:
(i.)) - (Zn.
(8)
Then set out proving the main theorem 3.1.
15

---

Published as a conference paper at ICLR 2025
Algorithm Joint denoising for multi-view multi-frame image matrix via score composition.
Require: Initial noise 7 multi view denoiser Dy(æ; σ), video denoiser D& (s(2; o), discrete noise levels (ee))e(--- scale schedule s, rollback steps N,
1: for all 10, ... . 1 do
2: + 2 if E {0. ... N2- 1} else
3: for all j (0.... R 1} do
4:
d' <- zero zeroorllkee (T')
5:
for all 10.... .V 1-] do
6:
Do Do(T(o,)io:))/0 d( t d [0.) + + (L))s())([[ 1o,t end for
7:
8:
for all ʃ 10.... ... F do
9:
d' ( + d (c}} + + Do Do(T(
10:
end for
11:
- o.)di I - I + -0.41-0;)
j R R-11 then
13: sample e ~ (0,
I + (ot d4t1)e
15: end if
end for
17:
end for
18:
return IN
Proof. We first decompose p (
p -1)= = (T... (1.si,t tts,tt,-s,ttt (, ([s,tir-(ts,, p (I)s
(9)
Note that Virs E I and J5 are independent conditioned on Iv.j by corollary A.1, hence
It.1)) = (i ( (holi(1stth)))
P
(10)
Since p (i..-)It- Ih,) does not contain (=I,,)), its derivative with respect to x is zero thus this term does not contribute to the score of Ttj- Then combined with equation 9 and equation 10, taking the derivative of log ( with respect to x, we achieve
V log (t) V logg (in tn p (Insstic-s)
(11)
Vx logp (iej,tiis ti.
Finally, by further decomposing p (.,,(,,,, and directly applying corollary A.1, we obtain
V log p (I)
Vx logp
(Z - p (Irs)
(Inslles) a (Ze.sllass) (Ind)
log
(I(WV)s) P (I,((1F3)
(12)
Vx log
P (ir1))
Vo (I(wv))) + Vx logo (In(1:F)) - Vx logp (Trs).
16

---

Published as a conference paper at ICLR 2025
A.4 THE APPLICABILITY OF THE ASSUMPTION 3.1 AND PROOF FOR THE THEOREM 3.1 IN GENERAL CASE
Another risk that needs be pointed out is that our assumption about the conditional independence is too strong to apply in some cases. For example, if the target object rotates 180" over a period of time, the back view at the current moment and the front view after period of time should not be conditionally independent given the front view at the current moment. Fortunately, existing video diffusion models typically generate videos from a single view without rotational movements, and our pipeline allows users to specify conditions to exclude such cases.
In addition, is worth noting that even in such cases that violate assumption 3.1, our method can still work well. This is because the main theorem remains valid under these case and its correctness does not essentially rely on such strong assumption. Although extreme rotations might exist when considering frames with large intervals, they unlikely to occur between adjacent frames in natural video. Now considering that the distribution of Í.,j is almost entirely determined by its adjacent frames and views, we have the continuity assumption:
Assumption A.I. (Continuity assumption) Denote Ir E II E &i. are the views and frames close enough to L to provide sufficient information for the distribution of it while keeping sufficiently continuous, then
(in) (fasttumm) p [1 (eg) (Fisltioun)
(13)
(14)
p T Fimmnt Iri less) ( Indmm)
(15)
where Iv & U1ig Drwandd Disa
Further note that the near index is symmetric, i.e,, if f 7 Te then hig E Te Then relax the assumption 3.1 as follows:
Assumption A.2. (Weak version of assumption 3.1 Given any image Isg. the nearby geometry Iy E I and the nearby dynamics 1s.3" E Je are conditionally independent, iee. p (Tv.s (Ts)) = (Ir (1 ( (13))
(16)
Similar condition also holds for I.
Finally, we can establish our main theorem 3.1 under the additional assumptions.
Proof of theorem 3.1 under assumption A.I and assumption A.2.
Following the same derivation of the original proof A.3, equations (9), (10) and (11) remain un- changed but all "=" in (10) and (11) should be "z²" because equation 10 is now established on cquation 15 of assumption A. instead of assumption 3.1.
Then integrating new assumptions to correct equation 12, we obtain:
(1) p (I=.),
Vx log
Vx log
Vx logg ال
(,, + Vx logp
~ Vx logp [inliimenn In
+ Vx logp (Zeani
(17)
Vx logp (Ir Thanire I Inieee p (Ireentr Ti
Vx logp [iry Inunss Inndine)
This suggests the score of I, is only related to Tmaors and Inijner Similar to the derivation in equa- tion 12, we can achieve:
17

---

Published as a conference paper at ICLR 2025
P (Lrs (heees) a (Jess
vx log (i) = Vx log
(18)
(4rs)
Finally, by further decomposing the numerator and applying the condition of continuity, we have:
(Irmns)
(Ir. (Tsse))))))
(Frssoer)
Vx logg (z) Vx log
p [Tis
(Irstiees) (Imis)
p (holtin (2r--s
Vx log
P
(hss
(19)
(I (In,(113))
Vx log
P (1.s
vx logp (Frvy)s) + Vx log (LI(EFT) V logp (hrs
Now we have verified the correctness of the conclusion in equation 12. Based on the similar deriva- tion, also ensure the safety of equation 10. This guarantees the correctness of our main theorem.
A.5 ADDITIONAL EXPERIMENTAL DETAILS
Dataset and evaluate setting In the main text, we the proposed approach on two types of conditions: reference image and single-view video. The input images and videos used in the qual- itative comparisons are sourced from the previous works Zhao et al. (2023); Jiang et (2024b); Tang et al. (2024b). For the quantitative evaluation in the video-to-4D task, we evaluate all methods on the Consistent4D test split and report the CLIP similarity, LPIPS, and FVD averaged on five views with accessible ground truth within the first 25 frames. Note that in equation 5, we introduced a coefficient s to modulate the contribution of both models within the convex combination.
Scale schedule Note that in equation 5, introduced coefficient modulate the contribution of both models within the convex combination. Its schedule in denoising process can provide addi- tional design space for enhaneing the details at specific views. However, to ease the reproduction and reduce the reliance on manual priors, we fix it at 0.5 in all steps which is sufficient to achieve satisfactory results for most cases.
User study Figure 9 shows the template we used for user study. The participants are asked to pick one generation results from five candidates (displayed in video format) according to four di- mensions: consistency with the reference video, multi-view consistency, temporal smoothness and overall model quality. Total 50 participants from diverse backgrounds are asked to do the question- naire. And there are 11 cases for user study, including anya, cat-wave, patrick-star, iring-ironman, sighing-rrog tiger-guital trump. walking- astrronatt. All malking-astronauut of them collected from the previous works (Jiang al., 2024b; Zhao et al., 2023; Tang et al., 2024b).
luigi, panda-dance mmuning-triceratopy
A.6 ADDITIONAL RESULTS
A.6.1 QUANTITATIVE ABLATIONS
In section 4.4 and the supplementary video, we have validated the effectiveness of the core com- ponents through qualitative ablations, where the improvements over naïve baselines are clearly and intuitively presented from the provided samples. To provide a more comprehensive evaluation, we present quantitative ablations in table 3, where all metrics averaged across all seven samples in the Consistent4D (Jiang et al., 2024b) test dataset. We also report the result using only the multi-view generator. While this approach leads in image-based metrics (e.g., LPIPS, CLIP), due
18

---

Published as a conference paper at ICLR 2025
Please watch the videos and rate the generation results according to the following questions.
Input vidco
D
E
B
c
D
E
Which one is most consistent with the input video?
O
Which one most multi-view consistent?
0
0
3. Which one has the highest temporal smoothness?
0
0
4. Please pick your most perferred one according the overall model quality.
0 0
0
Figure 9: The layout of our user study..
Table 3: Quantitative ablations. Each method is benchmarked in the same way as table 2. Dis-
abling score composition means multi-view diffusion only.
Score composition
Attention feature blend
VRS
LPIPS +
CLIP score f
FVD
mv mv-generation only recon with Li al. (2024)
I1.63
0,948
1248.49
14.86
0.896
1412.00
12.79
0.919
1050.68
12.65
0.922
898.66
11.86
0.943
1170.96
12.35
0.922
831.32
(g)
(f) w/o decoupled recon.
12.57
0.921
833.37
the inherent stochasticity in the diffusion denoising process, its output exhibits significant temporal inconsistencies, making it challenging be reconstructed into meaningful 4D assets, even for the state-of-the-ar 4D reconstruction method (Li ct al., 2024) with superior fitting capability.
A.6.2 MORE VISUALIZATIONS
In figure 10, we provided more cases for qualitative comparison. Similar conclusion to section 4.2 can be achieved that our results outperform the others. Figures 11 and 12 further show additional results on custom data, with RGB and depth images from two views and three timestamps displayed. We recommend readers to watch the videos in our supplementary materials.
19

---

Published as a conference paper at ICLR 2025
Input
STAG4D
DG4D
EmolenAD
SV4D
Ours (stage-1) Vee
Ous (stage-2)
Sme
time
time 1
time
time
time 2
time
time
Figure 10: Qualitative comparisons. For each method, show images from two views and two timestamps.
20

---

Published as a conference paper at ICLR 2025
Figure 11: More visualization results.
21

---

Published as a conference paper at ICLR 2025
Figure 12: More visualization results.
22


Reviews:
[{"review_id": "By2Tw7Wxf9", "reviewer": "Reviewer_s4pt", "summary": "This paper presents Diffusion^2, an interesting framework that aims to combine video and multi-view diffusion models through score composition for 4D content generation. The method proposes parallel generation of multi-view videos without requiring synchronized 4D training data, offering a novel theoretical perspective on combining different types of diffusion priors.", "questions": "- Have the authors considered strategies to reduce the computational requirements?\n- Could the authors provide insights on how the method might handle more complex motion patterns?\n- What are the main challenges in extending this approach to more diverse scenarios?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Innovative conceptual approach to leveraging video and multi-view diffusion priors\n- Well-formulated theoretical framework for score composition\n- Thoughtful parallel generation architecture design\n- Clear and thorough ablation studies", "weaknesses": "The method has some practicality issues:\n- Requires 8-16 A6000 GPUs for basic operation\n- Still takes 20 minutes per generation even with massive compute\n- Memory-speed tradeoff makes it impractical for most real applications\nIt would be good to hear the authors discussing about the path to reduce these extreme hardware requirements.\n\n\nThe experimental validation could be more comprehensive:\n- Most demonstrations focus on relatively simple motions\n- Additional results on more complex scenarios (e.g., rotational movements, articulated motions) would help better understand the method's capabilities\n- Further exploration of challenging cases would strengthen the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d82bEOgQ6A", "reviewer": "Reviewer_AvRx", "summary": "This paper aims to leverage existing multi-view 3D generation models and video generation models for 4D generation. For this, it designs a score composition strategy to make use of these two kinds of generation models for directly generating multi-frame multi-view image arrays. To alleviate potential conflicts between these two kinds of generation models, a variance-reducing sampling strategy is proposed. With the generated multi-frame multi-view image arrays, the proposed method introduces a dynamic and texture-decoupled\nreconstruction strategy to reconstruct 4D videos. Experiments show that the proposed method can make use of these two kinds of generation models for 4D generation.", "questions": "1. Can you try the baseline mentioned in W1 to further demonstrate the effectiveness of the proposed method?\n2. Can the proposed method generalize to other multi-view 3D generation and video generation methods?\n3. Can you show the time consumption for stage-1 and stage-2?\n4. Is it possible for the proposed method to generate 4D contents with multiple objects?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This proposed method takes advantage of existing multi-view 3D generation and video generation for 4D generation and does not require extra 4D training data. \n2. Through the analysis, the score composition for denoising is reasonable to achieve the above goal.\n3. Considering the potential conflicts between two kinds of generation models, the proposed method introduces variance-reducing sampling to reconcile the distinct modes at a higher noise level.", "weaknesses": "1. Since the proposed method leverages both multi-view 3D generation models and video generation models, maybe a reasonable baseline is to generate a video first via video generation models, then generate multi-view images based on the frames in this video using the multi-view 3D generation models. Finally, 4D reconstruction methods, such as [1], can be applied to generate 4D contents. It will be better to use this baseline to highlight the effectiveness of the proposed score composition.         \n[1] Li, Zhan, et al. \"Spacetime gaussian feature splatting for real-time dynamic view synthesis.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.   \n2. The experiments are weird. In Figure 4, the qualitative results for the stage-1 and stage-2 are almost the same. As mentioned in L524-525, the images generated in the first stage will show blurring details. Therefore, why are the results in two stage almost the same?\n3. For the ablation studies, it is better to show quantitative results like Table 1 and Table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LfBvnULoLC", "reviewer": "Reviewer_rKU7", "summary": "This paper introduces a new algorithm to generate 4D content from a single image or video of a foreground object.\nThe key idea is to combine a video diffusion model with a multiview diffusion model to generate a set of multiview-consistent videos.\nThe generation process involves interleaving the denoising of the video diffusion model and the denoising of the multiview diffusion model. To further improve the generation quality, a variance reduction method is proposed to re-denoise some views. Finally, a dynamic Gaussian field is reconstructed from the generated multiview videos. \nExperiments demonstrate that the proposed method is able to produce multiview consistent videos and generate 4D contents.", "questions": "I'm not sure whether a finetuning of SVD on videos containing only foreground objects would be helpful in improving the quality. In my experience, SVD performs poorly in generating videos of only one dynamic foreground object.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of the paper is that the idea of combining two diffusion models to generate 4D content is novel and interesting.\nDue to the lack of 4D data, it is difficult to train a 4D diffusion model directly. This method only takes advantage of video diffusion models and multiview diffusion models, which can be trained separately on large-scale 3D object datasets or video datasets. Thus, this method is promising to avoid directly using 4D data but still allow 4D generation. \n\nThe theoretical analysis of the proposed method is reasonable and convincing. The assumptions are provided for a better understanding of the mechanism of the interleaved denoising of two diffusion models.", "weaknesses": "The main weakness is the quality of the 4D generation.\nThe generation of the proposed method seems to be oversmoothed on unseen viewpoints. As we can see in both figures and the supplementary video, the backside all seems to be oversmoothed without any details, especially when the variance reduction is applied. \nI guess this may be due to the difficulty in making two diffusion models agree with each other. The assumption of independent probability distribution may be a little bit strong. \nThe results are even worse than the models with SDS losses or some other models directly trained on limited 4D data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pDZP89J8qw", "reviewer": "Reviewer_UTNE", "summary": "This proposal design a framework for combination between temporally aligned VDM(Video Diffusion Model) and spatially aligned multi-view generation model. It utilizes runtime optimization instead of training to build up 4D objects. Among recent SoTA 4D generation methods, this proposal is free of training and achieves competitive performance. Also, it proves that we can use off-the-shell models for 4D content generation without further training.", "questions": "1、Figure3，If the viewpoints differ by only 30 degrees, can they still be considered independent variables?\n2、Variance-reducing sampling，the rollback operation need more specific explanation of why it works? And demonstrate why this would reduce the variance of sampling results? Also, If we set the sampling steps larger, will it achieves similar effect with Variance-reducing sampling?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1、The framework setting needs no further training. \n2、The authors write clear proofs for the denoise process and 4D reconstruction process.\n3、Experiments are sufficient to prove the anticipated conclusion.", "weaknesses": "Some conclusions are not that rigorous, see questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)