Vivid-ZOO: Multi-View Video Generation with Diffusion Model
Authors: Bing Li
Cheng Zheng
Wenxuan Zhu
Jinjie Mai
Biao Zhang
Peter Wonka
Bernard Ghanem
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-view video
generative model
Primary area: generative_models
TL;DR: A diffusion model to generate multi-view videos for dynamic objects.

Abstract: While diffusion models have shown impressive performance in 2D image/video generation, diffusion-based Text-to-Multi-view-Video (T2MVid) generation remains underexplored. The new challenges posed by T2MVid generation lie in the lack of massive captioned multi-view videos and the complexity of modeling such multi-dimensional distribution. To this end, we propose a novel diffusion-based pipeline that generates high-quality multi-view videos centered around a dynamic 3D object from text. Specifically, we factor the T2MVid problem into viewpoint-space and time components. Such factorization allows us to combine and reuse layers of advanced pre-trained multi-view image and 2D video diffusion models to ensure multi-view consistency as well as temporal coherence for the generated multi-view videos, largely reducing the training cost. We further introduce alignment modules to align the latent spaces of layers from the pre-trained multi-view and the 2D video diffusion models, addressing the reused layers' incompatibility that arises from the domain gap between 2D and multi-view data. In support of this and future research,  we further contribute a captioned multi-view video dataset. Experimental results demonstrate that our method generates high-quality multi-view videos, exhibiting vivid motions, temporal coherence, and multi-view consistency, given a variety of text prompts.

Vivid-Z00: Multi-Vies Video Generation with Diffusion Model
Bing Li
Cheng Zheng"
Wenxuan Zhu"
Jinjie Mai Biao Zhang
Peter Wonka
Bernard Ghanem
King Abdullah University of Science and Technology
http:://hiizhhngghhgy....... wtt.hbb..bo//viidzoo/////
Abstract
While diffusion models have shown impressive performance in 2D image/video generation, diffusion-based mains underexplored. The new challenges posed by T2MVid generation lie in the lack of massive captioned multi-view videos and the complexity of modeling such multi-dimensiona distribution. To this end, we propose a novel diffusion-basee pipeline that generates high-quality multi-view videos centered around dynamic 3D object from text. Specifically, we factor the T2MVid problem into viewpoint- space and time components. Such factorization allows us to combine and reuse layers of advanced pre-trate pre-trained multi-view image and 2D video diffusion models to ensure multi-view consistency as well as temporal coherence for the generated multi- videos, largely reducing the training cost. We further introduce align- ment modules to align the latent spaces of layers from the pre-trained multi-view and the 2D video diffusion models, addressing the reused layers' incompatibility that arises from the domain gap between 2D and multi-view data. support of this and future research, we further contribute captioned multi-view video dataset. Experimental results demonstrate that our method generates high-quality multi-view videos, exhibiting vivid motions, temporal coherence, and multi-view consistency, given a variety text prompts.
Video (T2MVid) generation
Introduction
Multi-view videos capture scene/object from multiple cameras with different poses simultaneously, which are critical for numerous downstream applications [5, 55, 62, 39, 40] such as AR/VR, 3D/4D modeling, media production, and interactive entertainment. More importantly, the availability of such data holds substantial promise for facilitating progress in research areas such 4D reconstruction [44, 48], 4D generation [3, 49], and long video generation [9, 101] with 3D consistency. However, collecting multi-view videos often requires sophisticated setups [1] to synchronize and calibrate multiple cameras, resulting in significant absence of datasets and generative techniques for multi- view videos.
In the meantime, diffusion models have shown great success in 2D image/video generation. For example, 2D video diffusion models [6, 23, 28, 81] generate high-quality 2D videos by extending image diffusion models [74, 83). Differently, multi-view image diffusion models [80, 34, 53, 93] proposed to generate multi-view images of 3D objects, which have demonstrated significant impact in 3D object generation [45], 3D reconstruction [66], and related fields. However, to the best of our knowledge, other works have explored Muattto-Muuii-vee--Weee (T2MVid) diffusion
Equaal contributions. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

View
View
View
View
Text prompt: yellow and black striped wasp bee, 3d asset
Figure 1: The proposed Vivid-700 generates high-quality multi-view videos of a dynamic 3D object from text. Each row illustrates six frames drawn from generated video for viewpoint.
models. Motivated by recent 2D video and multi-view image diffusion models, we aim to propose diffusion-based method that generates multi-view videos of dynamic objects from text (see Fig. 1).
Compared 2D video generation, T2MVid generation poses ing multi-view videos is complex due to their nuu-ddmensional nature, which involves different nontrivial for
challenges. First, model-
viewpoints as well as the dimensions of time and space (2D). Consequently, diffusion models to model such intricate data from scratch without extensive captioned multi-view video datasets. Second, there are no publicly available large-scale datasets of captioned multi-view videos, but it has been shown that billions of text and 2D image pairs are essential for powerful image diffusion models [74, 76, 83]. For example, Stable Diffusion [76] is trained on the massive LAION-5I 5B dataset [78]. Unlike downloading 2D images available on the Internet, collecting large quantity of multi-view videos labor-intensive and time-consuming. This challenge is further compounded when high-quality captioned videos are needed, hindering the extension of diffusion models to V2MVid generation.
In this paper, instead of the aabor-intensive task of collecting a large amount of captioned multi-view video data, we focus on the problem of enabling diffusion models to generate multi-view videos from text using only comparable small dataset of captioned multi-view videos. This problem has not been taken into account by existing diffusion-bain based methods (e.g., [6][23]). However, studies have revealed that naively fine-tuning a large pre- preitrd model on limited data can result in overfitting [30, 75, 115]. Our intuition that we can factor the multi-view video generation problem into viewpoint-space and time components. The viewpoint- are geometrically consistent and aligned with the input text, the temporal component ensures temporal coherence. With such factorization, a straightforward approach is to leverage large-scale multi-view image datasets (e.g., [51] [67]) and 2D video datasets (e.g.. Web1OM [4]) to pre-train the viewpoint-space component and temporal component, respectively. However, while this approach can largely reduce the reliance on extensive captioned multi-view videos, remains costly in terms of training resources.
component ensures that the generated multi-view videos
Instead, we explore a new question: can we jointly combine and reuse the layers of pre-trained 2D video and multi-view image diffusion models to establish a T2MVid diffusion model? The large-scale pre-trained multi-view image diffusion models (e-g-, MVdream [80]) have learned how to model multi-view images, and the 2D temporal layers of powerful pre-trained video diffusion models (e.g., Difm [23]) learned rich motion knowledge. However, new challenges are posed. We observe that naively combining the layers from these two kinds of diffusion models leads to poor generation

---

results. More specifically, the training data of multi-view image diffusion models are mainly rendered from synthetic 3D objects (e.g.. Objaverse [17, 16] ). while 2D video diffusion models are mainly trained on real-world 2D videos, posing large domain gap issue.
To bridge this gap. propose a novel diffusion-based pipeline, namely, Vivid-ZOO, for V2M generation. The proposed pipeline effectively connects the pre-trained multi-view image diffusion model [80] and 2D temporal layers' of the pre-trained video model by introducing two kinds of layers, named 3D-2D alignment layers and 2D-3D alignment layers, respectively. The 3D-2D alignment layers are designed to align features to the latent space of the pre-trained 2D temporal layers, and the introduced 2D-3D alignment layers project the features back. Furthermore, we construct a comparable small dataset consisting of 14,271 captioned multi-view videos to facilitate this and future research line. Although our dataset is much smaller compared to the billion-scale 2D image dataset (LAIOON (78]) and the million-scan 2D video dataset (e.g., WebVidIOM [4]), our pipeline allows us to effectively train large-scale ViM diffusion model using such limited data. Extensive experimental results demonstrate that our method effectively generates high-quality multi-view videos given various text prompts.
We summarize our contributions as follows:
We present novel diffusion- -bsed pipeline that generates high-quality multi-view videos from text prompts. This is the first study on T2MVid diffusion models.
We show how to combine and reuse the layers of the pre-trained 2D video and multi-view image diffusion models for T2MVid diffusion model. The introduced 3D-2D alignment and 2D-3D alignment are simple yet effective, enabling our method to utilize layers from the two diffusion models across different domains, ensuring both temporal coherence and multi-view consistency.
We contribute a multi-view video dataset that provides multi-view videos, text descriptions, and corresponding camera poses, which helps to advance the field of T2MVid generation.
2 Related work
2D video diffusion model. Many previous approaches have explored autoregressive transformers (e.g.. [18, 29, 107]), physical models [104] or GANs (e.g., [8, 56, 77, 41J) for video generation. Recently, more and more efforts have been devoted to biffusion-based video generation [21, 26, 65, 94, 99, 102, 110, 121, 37], inspired by the impressive results of image diffusion models [11, 12, 74, 76, 83, 115].
The amount of available captioned 2D video data significantly less than the vast number of 2D image-text pairs available on the Internet. Most methods [7, 22, 23, 90, 91, 92] extend pre-trained 2D image diffusion models to video generation to address the challenge of limited training data. Some methods employ pre- rreitrined 2D image diffusion models (e.g., [76]) to generate 2D video from texts in a zero-shot manner [36][] [ or using few-shot tuning strategies [103]. These methods avoid the requirement of large-scale training data. Differently, another research line is to augment pre- trained 2D image diffusion models with various temporal modules or trainable parameters, showing impressive temporal coherence performance. For example, Ho et al. [28] extend the standard image diffusion architecture by inserting a temporal attention block. Animatedif[[33] and AYL [7] freeze 2D image diffusion model and solely train additional motion modules large-scale datasets of captioned 2D videos such as Web Vid [4]. In addition, imagc-to- generation methods [6, 71, 105, 117] are proposed based on diffusion models to generate monocular video from an image. Methods [95] focus on controllable video generation through different conditions such as pose and depth. MotionCtrl [98] and Direct-a-video [109] can generate videos conditioned by the camera and object motion. CameraCtrl [24] can also control the trajectory of a moving camera for generated videos. However, these -ex-ioo-DD-video diffusion models are designed for monocular video generation, which does not explicitly consider the spatial 3D consistency of multi-view videos.
Multi-view image diffusion model. Recent works have extended 2D image diffusion models for multi-view image generation. Zero [51] and Zerol23++ [79] propose to fine-tune an image- conditioned diffusion model so as to generate novel view from a single image. Inspired by this,
Forr clarification, we add "2D" when referring to the layers of the 2D video diffusion models, while we add "3D" when referring to the multi view image diffusion model.
3

---

many novel view synthesis methods [19, 32, 52, 53, 59, 93, 97, 100, 108, 112, 120] are proposed based on image diffusion models. For example, IM3D [59] and Free3D [120] generate multiple novel views simultaneously to improve spatial 3D consistency among different views. Differently, few methods [13, 33, 89] adapt pre-trained video diffusion models (e.g., [6]) to generate multi-view images from a single image. MVDream [80] presents text-to-multic -ilo-uul--view-mmaeee diffusion model to generate four views of an object each time given a text, while SPAD [34] generates geometrically consistent images for more views. Richdreamer [67] trains a diffusion model to generate depth, normal, and albedo.
4D generation using diffusion models. Many approaches [47, 51, 64, 69. 79, 80, 85, 96, 2] have exploited pre-trained diffusion models to train 3D representations for 3D object generation via score distillation sampling [64]. Recently, a few methods [3, 49. 70, 82. 113] leverage pre-trained diffusion models to train 4D representations for dynamic object generation. For example, Ling er al. [49] represent a 4D object as Gaussian spatting [35], while Bahmani et al] adopt a NeRF-based representation [60, 61, 86). Then, pre-trained 2D image, 2D video, and multi-view image diffusion models are employed jointly train the 4D representations. In addition, diffusion models are used to generate 4D objects from monocular videos [14, 31J. Diffusion4D [46] presents a diffusion model that generates an orbital video around 4D content, and 4Diffusion [1 14] presents a video-conditioned diffusion model that generates MV videos from a monocular video. Different from all these methods, approach focuses presenting T2MVid diffusion model. LMM [116] generates 3D motion for given 3D human models. DragAPart [43] can generate part-level motion for articulated objects. Unlike our method, Kuang et [38] focuses generating multiple videos of the same scene given multiple camera trajectories.
3 Multi-view video diffusion model
Problem definition. Our goal for T2MVid generation to generate a set of multi-view videos centered around dynamic object from a text prompt. Motivated by the success of diffusion models in 2D video/image generation, we aim to design a T2M Vid diffusion model. However, T2MVid generation is challenging due to the complexity of modeling multi-view videos and the difficulty of collecting massive captioned multi-view videos for training.
We address the above challenges by exploring two questions. (1) Can we design diffusion model that effectively learns T2MVid generation, yet only needs a comparable small dataset of multi-view video data? (2) Can we jointly leverage, combine, and reuse the layers of pre-trained 2D video and multi-vien view image diffusion models to establish a T2MVid diffusion model? Addressing these questions can reduce the reliance on large-scale training data and decrease training costs. However, this question remains unexplored for diffusion-based V2MVid.
Overview. We address the above questions by factoring the T2MVid generation problem over viewpoint-space and time. With the factorization, we propose a diffusion- based pipeline for T2MVid generation (see Fig 2), including the multi-view spatial modules and multi-view temporal modules. See 3.1 describes how we adapt pre-trained multi-view image diffusion model as the multi-view spatial modules. Multi-view temporal modules effectively leverage temporal layers of the pre-trained 2D video diffusion model with the newly introduced 3D-2D alignment layers and 2D-3D alignment layers (Sec 3.2). Finally, we describe training objectives in Sec 3.3 and the dataset construction to support our pipeline for T2MVid generation in Sec 3.4.
3.1 Multi-view spatial module
Our multi-view spatial modules ensure that the generated multi-view videos are geometrically consistent and aligned with the input text. Recent multi image diffusion models [34, 80] generate high-quality multi-view images by fine-tuning Stable Diffusion and modifying its self- attention layers. We adopt the architecture of Stable Diffusion for our multi-view spatial modules. Furthermore, we leverage a pre-trained multi-view image diffusion model based on Stable Diffusion by reusing its pre-trained weights in our spatial modules, which avoids training from scratch and reduces the training cost. However, the self-attention layers of Stable Diffusion are not designed for multi-view videos. We adapt these layers for multi-view self-attention as below.

---

Text Pronpt
CLIP
Pondd Shory
Mait view images of 3D object
Itain
Pre-tmined Malli-View Spatial Medule
30-20 aligrmen
Domanin Gap
ID-5D algennen
temporal layers
Trainable
MLP
2D temporal layers
Timestop
ML.P
Camera pose
Addtiion
Mils- View Temporal Module
Multi-View Sputial Module
20-30 aligaunnn
313-23 aligament
Real-word 2D videos
Figure 2: Overview of the proposed Vivid-Zoo. Left: Given text prompt, our diffusion model generates multi-view videos. Instead of training from scratch, the multi-view spatial module reuses the pre-trained multi-view image diffusion model, and the multi-view temporal module leverages the 2D temporal layers of the pre- pre-trained 2D video diffusion model to enforce temporal coherence. Right: Jointly reusing the pre-trained multi-view image diffusion model and temporal 2D layers poses new challenges due to the large gap between their training data (multi-vier images of synthetic 3D objects versus real-world 2D videos). We introduce 3D-2D alignment and 2D-3D alignment to address the domain gap issue.
Multi-view self-attention. We inflate self-attention layers to capture geometric consistency among generated multi-view videos. Let F E R R N hx denote the 6D feature tensor of multi-view videos in the diffusion model, where b, K. N, and h x w are batch size, view number, frame number, feature channel and spatial dimension, respectively. Inspired by [34, 80), we reshape F into shape of (b N) d KK N) is the batch size, Fnn denotes feature map representing all views at frame index n, and (K h x w) is the spatial size. We then feed the reshaped feature maps F into self aeff-attention layers. Since F" consists all views at frame index n, the self-attention layers learn to capture geometrical consistency among different views. We also inflate other layers of stable diffusion (see Appendix E) so that we can reuse their pre-trained weight.
w). leading to batch of feature maps F of 2D images, where
Camera pose embedding. Our diffusion model is controllable by camera poses, achieved by incorporating camera pose sequence as input. These poses are embedded by MLP layers and then added to the timestep embedding, following drdram [80]. Here, our multi-view spatial module reuses the pre-trained multi-view image diffusion model MVDream [80].
3.2 Multi-view temporal module
Besides spatial 3D consistency, it is crucial for V2MVid diffusion models to maintain the temporal coherence of generated multi-view videos simultaneously. Improper temporal constraints would break the synchronization among different views and introduce geometric inconsistency. Moreover, training a complex temporal module from scratch typically requires a large amount of training data.
Instead, we propose to leverage the 2D temporal layers of large pre-trained 2D video diffusion models (e.g., [23]) ensure temporal coherence for T2MVid generation. These 2D temporal layers have learned rich motion priors, as they have been trained on millions of 2D videos (e.g., [4]). Here, we employ the 2D temporal layers of AnimateDiff [23] due to its impressive performance in generating temporal coherent 2D videos.
However, we observed that naively combining the pre-trained 2D temporal layers with the multi-view spatial module leads to poor results. The incompatibility is due to the fact that the pre-trained 2D temporal layers and the multi-view spatial modules are trained on data from different domains (i.e., real 2D and synthetic multi-view data) that have a large domain gap. address the domain gap issue, one approach is to fine-tune all 2D temporal layers of pre- trained 2D video diffusion model
5

---

on multi-view video data. However, such an approach not only needs to train many parameters but can also harm the learned motion knowledge [30] if small training dataset is given. We present a multi-view temporal module (see Fig. 3) that reuses and freezes all 2D temporal layers to maintain the learned motion knowledge and introduce the 3D-2D alignment layer and the 21>-3D alignment layer.
3D-2D alignment. We introduce the 3D-2D alignment layers effectively combine the pre-trained 2D temporal layers with the multi-view spatial module. Recently, a few methods [23, 6] add motion LoRA to 2D temporal attention for personalizeed/customizee video generation tasks. However, our aim is different, i.e., we expect to preserve the learned motion knowledge of 2D temporal layers, such that our multi-view temporal module can leverage the knowledge for ensuring temporal coherence.
Since motion prior knowledge is captured by the pre-trained 2D temporal attention layers, we insert the 3D>-2D alignment layers before the 2D temporal attention layers. The 3D-21) alignment layers are learned to align the features into the latent space of the pre-trained 2D temporal layers. Furthermore, inspired by ControlNet [115] and [25], the 3D-2D alignment layers are inserted via residual connections and are zero-initialized, providing an identity mapping at the beginning of training. The process described as follows:
F F= (F) + a. 3D-
(1)
where a. is the 3D-2D alignment layer. a² the 2D temporal layer followed by the 2D temporal attention layers and we refer to it as 2D in-layer (see more details in Appendix), The 3D-2D alignment layer plug-and-play and is simply implemented as an MLP.
Multi-view temporal coherence. We reuse and freeze the pre-trained 2D temporal layers in our multi-view temporal module to ensure the temporal coherence of each generated video. However, the 2D temporal layer is designed to handle 2D videos. We inflate the 2D temporal layer by reshaping the feature F the 2D video dimension via the rearrange operation [73]. Then, 2D temporal layers 7(-) model tem- poral coherence across frames by calculating the attention of points at the same spatial location in F across frames for each video:
2D In Layer
3D-2D
Zoro Intt
Alignment
2D Temporal Attention
F rearrange 6K N h d + (bK h W) N d)
(2)
2D Out Layer
2D-3D Alignment
Zero
F (F)
F rearrange (F (b K h w) N d + bKNhw d) (4)
2D-3D alignment. We add the 2D-3D alignment layers after 2D temporal attention layers to project the feature back to the feature space of the multi-view spatial modules.
Our multi-view temporal module, where 3D-2D alignment lay- ers are trained to align features to the latent space of the 2D temporal atten- tion layers, and the 2D-3D alignment layers project them back.
Figure 3:
(5) where 33D-2D is the 2D-3D alignment layer. gaD the 2D temporal layer following the 2D temporal attention layer. The 2D-3D alignment layers are implemented as an MLP.
F- (22(F)
(2D-3D (F)
3.3 Training objectives
We train our diffusson model to generate multi-view videos. Note that we freeze most layers/modules in the diffusion model and only train the 3D-2D and 2D-3D alignment layers during training, which largely reduces the training cost and reliance on large-scale data. Let X' denote the training dataset, where a training sample {x, y. c} consists of N multi-view videos x = {x) N corresponding camera poses c, and a text prompt y. The training objective L on X' is defined as follows:
L = s.e. wee. [|c - ca (z) tt re(y).
(6)
where (o(-) is text encoder that encodes the text into text embedding, eo (-) is the denoising network. zi is the latent code of multi-view video sequence and z is its noisy code with added noise e.

---

MVdream
MVdream IP-AnimateDiff
Ours
View 0
View
View 2
View 3
Text prompt: Beautiful, intricate butterfly, 3d asset.
Figure Comparison on T2MVid generation. Although MVDream generates spatially 3D consistent images among views (the 1st column), MVDream DP-AnimateDiff breaks the spatial 3D consistency among its generated videos. Instead, our method generates high-quality multi-view videos with large motions while maintaining temporal coherence and spatial 3D consistency.
3.4 Multi-view video dataset
Different from 2D images that are available in vast numbers on the Internet, it is much more difficult and expensive to collect a large amount of multi-view videos centered around 3D objects and corresponding text captions. Recently, multi-view image datasets (e.g., [51, 67]), rendered from synthetic 3D models, have shown a significant impact on various tasks such as novel view synthesis [51, 93], , 3D generation (Gaussian Splatting [84], large reconstruction model [50]), multi-view image generation [80] and associated applications. Motivated by this, we resort rendering multi-view videos from synthetic 4D models (animated 3D models).
We construct dataset named MV-VideoNet that provides 14,271 triples of multi-view video sequence, its associated camera pose sequence, and a text description. In particular, we first select animated objects from Objaverse [17]. Objaverse an open-source dataset that provides high-quality 3D objects and animated ones (i.e., 4D object). We select 4D objects from the Objaverse dataset and discard those without motions with imperceptible motions. Given each selected 4D object, render 2iev videos from it, where the azimuth angles of camera poses are uniformly distributed. To improve the quality of our dataset, we manually filter multi-view videos with low-quality e-g., distorted shapes or motions, very slow or rapid movement. For text descriptions, we adopt the captioning method Cap3D [57, 58] to caption a multi-vies view video sequence. Cap3D leverages BLIP2 [42] and GPT4 [63] to fuse information from multi-view images, generating text descriptions.
4 Experiments
Implementation details. We reuse the pre-trained MVDream V1.5 in our multi-view spatial module and reuse the pre-trained 2D temporal layers AnimateDiff V2.0 in our multi-view temporal module. We train model using AdamW [54] with learning rate of 10 During training, process the training data by randomly sampling 4 views that are orthogonal to each other from a multi-view video sequence, reducing the spatial resolution of videos to 256 256, and sample video frames with a stride of 3. Following Diff, we use linear beta schedule with 3start 0.00085 and Bend 0.012. (Please refer to the Appendix for more details).
Evaluation metrics. Quantitatively evaluating multi-view consistency and temporal coherence remains open problem for T2MVid generation. We quantitatively evaluate text alignment via CLIP [68] and temporal coherence via Frechet Video Distance (FVD) [87]. Yet, Ge al. [20] pointed out FVD leans more towards per-frame quality than temporal consistency. To compensate for FVD,

---

w/o MS SD
Our
View
View
View 2
View
Text prompt: a dog wearing an outfit, 3d asset
Figure 5: Visual comparison of the contributions of our multi-view spatial module
we conduct a user study to evaluate the overall performance incorporating text alignment, temporal coherence, and multi-view consistency according human preference (H. Pref.). CLIP and FVD scores Tab. are computed from 25 multi-view videos, where most input prompts used to generate these videos separate from the training set, and only two prompts from the training set. For ablation study, there are five methods and ten subjects for evaluating human preference, leading to 5x2x10 = =100 questionnaires per input text prompt. To reduce the cost, input prompts to evaluate human preference in the ablation.
4.1 Qualitative and quantitative results
To the best of our knowledge, no studies have explored T2MVid diffusion models before. We establish baseline method named MVDream + IP-AnimateDiff for comparison. MVDream + IP-AnimateDiff combines the pre-trained multi-view image diffusion model MVDream [80] and the 2D video diffusion model AnimatelDif' [23], since MVDream generates high-quality multi-view images and AnimateDiff generates temporal coherent 2D videos. Following [119], we combine AnimateDiff with IP-adaptor [111] enable AnimateDiff to take image input.
Given a text prompt, MVDream IP-AnimateDiff generates multi-view videos two stages, where MVDream generates multi-view images in the first stage, and IP-AnimateDiff animates each generated image from view into a 2D video in the second stage.
Fig. 4 and Tab. show that MVDream + IP-AnimateDiif achieves slightly better CLIP values. However, our method outperforms MVDream + IP-AnimateDiff' by large margin in FVD and overall performance. MVDream IP-AnimateDiff introduces the noticeable 3D inconsistency among different views. For example, both appearances and motions of the butterfly in the view 0 video are inconsistent with those of view 3. In contrast, our method not only achieves better performance in maintaining multi-view consistency, but also generates larger and more vivid motions the butterfly, thanks to our pipeline and dataset. In addition, different from MVDream IP-AniiateDDif employing two kinds of diffusion models and generating results in two stages, our method provides unified diffusion model generating high-quality multi-view videos in only stage. Please refer to the Appendix for more results.
4.2 Ablation study and discussions
We conduct the ablation study show the effectiveness of the design in our multi-view spatial and temporal modules, well the proposed 3D-2D and 2D-3D alignment.

---

Table 1: Multi-view video generation. Best in bold.
Method
FVD |
CLIP
Overall f
MVDream IP-AnimateDiff
2038.66 + 44.36
32.71 + 0.67
28% 72%
Ours
1634.28 + 45.24
32.24 0.78
w/o MT TM LoRA
Ours
View
View
View 2
View 3
Text prompt: a sea turtle, 3d asset.
Figure 6: Visual comparison of the contributions of our multi-view temporal module
Design of multi-view spatial module. We build baseline named w/o MS w SD that employs original Stable Diffusion 1.5 [76] as our multi-view spatial module and reuses its pre-trained weights. We also insert the camera embedding into the Stable Diffusion to enable viewpoint control. That is, w/o MS w SD is to generate a single-view video (2D) conditioned on input text and camera poses. We train w/o MS SD on dataset, where single-view videos used training data.
Since single-view video generation much simpler than multi-view video generation, w/o MS SD achieves high performance in video quality. However, w/o MS w SD fails to maintain multi-view consistency among different views (see Fig. 5) and has degraded overall generation performance (see Tab. 2). For example, the motion and shapes of the dragon significantly inconsistent among views. Instead, by simply adapting a pre- pre-trained multi- view image diffusion model as our spatial module, method effectively ensures multi-view consistency.
Design of multi-view temporal module. Recent methods apply LoRA [30] to the 2D temporal attention layers of a pre-trained 2D video diffusion model and fine-tune only LoRA for personalized and customized 2D video gener- ation tasks [23, 6, 72]. Following these methods, we build temporal module named TM LoRA by inflating 2D tem- poral layers of AnimateDiff to handle multi-view videos and adding LoRA to the 2D temporal attention layers. We replace our multi-view temporal module with TM LoRA, and denote it by w/o MT TM LoRA. Fig. 6 and Tab. 2 shows w/o MT w TM LoRA generates low-quality results, despite being alignment and 2D-3D alignment layers before and after the 2D temporal attention layers, enabling the multi-view temporal module to be compatible with the multi-view spatial module.
Table 2: The ablation study results. The overall performance is assessed by a user study using paired comparison [3, 15].
Method
Overall f
w/o MS w SD
44.88%
w/o MT w TM LoRA
11.25%
w/o 3D-2D alignment
53.50%
w/o 2D-3D alignment
54.50%
Ours
80.25
dataset. Instead,
multi-view temporal module inserts 3D-2D
Effect of 3D-2D alignment. We remove the proposed 3D-2D alignment from our model and train the model our datasct with the same settings. Tab. shows w/o 3D-2D alignment degrades

---

our temporal coherence and video quality performance. Instead, by projecting the feature to the latent space the pre-trained 2D attention layers, our 3D-2D alignment layer effectively enables the 2D attention layers to align temporally correlated content, ensuring the video quality and temporal coherence.
Effect of 2D-3D alignment. As shown in Tab. 2, "w/o 2D-3D temporal alignment" achieves lower performance with the same training settings due the removal of 2D-3D temporal alignment. The results indicate that only 3D-2D alignment is insufficient in jointly leveraging the pre-trained 2D temporal layers [23] and the multi-view image diffusion model [80] our diffusion model. Instead, our 2D-3D alignment projects the features processed by the pre-trained 2D temporal layers back to the latent space of the multi-view image diffusion model, leading high-quality results.
Training cost. MVDream is trained on 32 Nvidia Tesla A100 GPUs, which takes 3 days, and AnimateDiff takes around days 8 A100 GPUs. By combining and reusing the layers of MVDream and AnimateDiff, our method only needs to train the proposed 3D-2D alignment and 2D-3D layers, reducing the training cost around 2 days with A100 GPUs.
5
Conclusions
In this paper, propose novel eeffusion-baseed pipeline named Vivid-ZOO that generates high- quality multi-view videos centered around dynamic 3D object from text. The presented multi-view spatial module ensures the multi-view consistency of generated multi-view videos, while the multi- view temporal module effectively enforces temporal coherence. By introducing the proposed 3D-2D temporal alignment and 2D-3D temporal alignment layers, pipeline effectively leverages the layers of the pre-trained multi-view image and video diffusion models, reducing the training cost and accelerating the training of our diffusion model. We also construct dataset of captioned multi-view vidcos, which facilitates future research this emerging area.
Acknowledgments and Disclosure of Funding
The research reported in this publication was supported by funding from King Abdullah University of Science and Teccnolooyy (KAUST)) Center Excellence for Generative AI, under award number 5940 and the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence. We thank Zhangjic Wu and Mengmeng Xu for their valuable constructive suggestions and help.
10

---

References
[1] Liang An, Jilong Ren, Tao Yu, Tang Hai, Yichang Jia, and Yebin Liu. Three-dimensional surface motion capture of multiple freely moving pigs using mammal. Nature Communications, 14(1):7727, 2023.
[2] Sherwin Bahmani, Xian Liu, Yifan Wang, Ivan Skorokhodov, Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, et al. Te4d: Trajectory-conditionee text-to-4d generation. arXiv preprint arXiv:2403.17920 2024.
[3] Sherwin Bahmani, Ivan Skorokhodov, Vietor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B. Lin- dell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling, arXiv preprint arXiv:2311.17984 2023.
[4] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
[5] Aayush Bansal, Minh Vo, Yaser Sheikh, Deva Ramanan, Srinivasa Narasimhan. 4d visualization of dynamic events from unconstrained multi-view videos. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[6] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023.
[7] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22563-22574 2023.
[8] Tim Brooks, Janne Hellsten, Miika Aittala, Ting chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yo Yu Liu, Alexei Efros, and Tero Karras. Generating long videos of dynamic scenes. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024.
[10] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308 2017.
[11] Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart-delta: Fast and controllable image generation with latent consistency models, 2024.
[12] Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-SlalphaS: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
[13] Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, and Huaping Liu. V3d: Video diffusion models are effective 3d generators, 2024.
[14] Wen-Hsuan Chu, Lei Ke, and Katerina Fragkiadaki. Dreamscene4d: Dynamic multi-object scene generation from monocular videos. arXiv preprint arXiv:2405.02280 2024.
[15] Herbert Aron David. The method of paired comparisons, volume 12. 1963.

---

[16] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte. Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: universe of 10m+ 3d objects. arXiv preprint arXiv:2307.0566.3 2023.
[17] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe annotated 3d objects. arXiv preprint arXiv:22/2.08055 2022.
[18] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-1o- image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 35: 18900-166,,,,, 2022.
[19] Ruiqi Gao*, Aleksander Holynski*, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole*, Cat3d: Create anything in 3d with multi-view diffusion models. arXiv, 2024.
[20] Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun- Man Zhu, and Jia-Bin Huang. On the content bias in fréchet video distance. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[21] Michal Geyer, Omer Bar- Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint 2xxiv:2307.10733 2023.
[22] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning, 2023.
[23] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang. Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image dif- fusion models without specific tuning. International Conference on Learning Representations, 2024.
[24] Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameraetrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024.
[25] Kaiming He, Xiangyu Zhang, Shanqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[26] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma. Ben Poole, Mohammad Norouzi. David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300 2022.
[27] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 6840-6851, 2020.
[28] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 2022.
[29] Wenyi Hong. Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for -ixdeo generation via transformers. arXiv preprint arXiv:2205. 15868. 2022.
[30] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[31] Yanqin Jiang, Li Zhang. Jin Gao, Weiming Hu, and Yao Yao. Consistent4d: Consistent 360 dynamic object generation from monocular video. In The Twelfth International Conference on Learning Representations, 2024.
12

---

[32] Yifan Jiang, Hao Tang, Hen-Hao Rick Chang, Liangchen Song, Zhangyang Wang, and Lian- gliang Cao. Efficient-3dim: Learning generalizable single-image novel-view synthesizer in one day. In The Twelfth International Conference on Learning Representations, 2024.
[33] Philip Torr Junlin Han, Filippos Kokkinos. Vfusion3d: Learning scalable 3d generative models from video diffusion models, arXiv preprini urXiv:2403. 00.4403./2344 /2034, 2024.
[34] Yash Kant, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, Sergey Tulyakov, Igor Gilitschenski, and Aliaksandr Siarohin. Spad Spatially aware multiview diffusers, 2024.
[35] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküihler and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 12(4))1-14, 2023.
[36] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion mod- els are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15954-15964, 2023.
[37] Jihwan Kim, Junoh Kang, Jinyoung Choi, and Bohyung Han. Fifo-diffusion: Generating infinite videos from text without training, 2024.
[38] Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, and Gordon. Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. In arXiv, 2024.
[39] Bing Li, Chia-Wen Lin, Boxin Shi, Tiejun Huang, Wen Gao, and C-C Jay Kuo. Depth-aware stereo video retargeting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6517-6525, 2018
[40] Bing Li, Chia- Wen Lin, Cheng Zheng, Shan Liu, Junsong Yuan, Bernard Ghanem, and C- C Jay Kuo. High quality disparity remapping with two-stage warping. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2269-2278, 2021.
[41] Bing Li, Yuanlue Zhu, Yitong Wang. Chia- Lin, Bernard Ghanem, and Linlin Shen. Anigan: Style-guided generative adversarial networks for unsupervised anime face generation. IEEE Transactions on Multimedia, 24:4077-4099 2021.
[42] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 19730-19742, 2023.
[43] Ruining Li, Chuanxia Zheng, Christian Rupprecht, and Andrea Vedaldi. Dragapart: Learning a leve- motion prior for articulated objects. arXiv preprint arXiv:2403. 15382, 2024.
[44] Tianye Li, Mira Slavcheva, Michael Zollhöfer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, and Zhaoyang Lx. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CV Conference Computer Vision and Pattern Recognition (CVPR), pages 5521-5531, June 2022.
[45] Zhiqi Li, Yiming Chen, Lingzhe Zhao, and Peidong Liu. Controllable text-to-3d generation via surface-aligned gaussian splatting. 2024.
[46] Hanwen Liang, Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N Plataniotis, Yao Zhao, and Yunchao Wei. Diffusion4d: Fast spatial-temporal consistent 4d generation via video diffusion models. arXiv preprint arXiv:2405. 16645, 2024.
[47] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung-Yi Yi Lin. Magic3d: High-resolution text-to-3d content creation. 2023 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 300-309, 2022.
13

---

[48] Haotong Lin, Sida Peng, Zhen Xu, Tao Xie, Xingyi He, Hujun Bao, and Xiaowei Zhou. and real-time novel view synthesis for dynamic scenes. In SIGGRAPH Asia Conference Proceedings, 2023.
[49] Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. arXiv preprint arXiv:2312. 2023.
[50] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
[51] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Kero-I-to-3: Zero-shot one image 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298-9309, 2023.
[52] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjee Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprini arXiv:2309.0345555 2023.
[53] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma. Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310. 15008, 2023.
[54] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[55] Jian Jian-Guang Lou, Hua Cai, and Jiang Li. A real-time interactive multi-view video system. In Proceedings of the 13th annual ACM international conference on Multimedia, pages 161-170, 2005.
[56] Pauline Luc, Aidan Clark, Sander Dicleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. coansfooratiionbaseee adversarial video prediction on large-scale data. arXiv preprint arXiv:2003. 04035, 2020.
[57] Tiange Luo. Justin Johnson, and Honglak Lee. View selection for 3d captioning via diffusion ranking, arXiv preprint arXiv:2404.07984 2024.
[58] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. arXiv preprint arXiv:2306.07279 2023.
[59] Luke Melas-Kyriaze Iro Laina, Christian Rupprecht, Natalia Neverova. Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos. Im-3d: Iterative multiview diffusion and reconstruction for high-quality 3d generation, 2024.
[60] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[61] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (TOG), 41:1 15, 2022.
[62] Armin Mustafa, Hansung Kim, Jean- Yves Guillemaut, and Adrian Hilton. General dynamic scene reconstruction from multiple view video. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
[63] OpenAL, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett- Shaaiiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell,
14

---

Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning. Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus. Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He. Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, t.kass Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Eukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil. David Medina, Aalok Mehta, Jacob Menick, Luke Metz. Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Retichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen OeKeee, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rac. Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng. Matt Wiethoff, Dave Willner. Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
[64] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2023.
[65] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv:2303.0955 2023.
[66] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magic One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[67] Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. arXiv preprint 27X1 2023.
[68] Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkín, Jack Clark, et al. Learning transferable visual
15

---

models from natural language supervision. In International conference on machine learning. pages 8748- PMLR, 2021.
[69] Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan T. Barron, Yuanzhen Li, and Varun Jampani. Dreambooth3d: Subject-driven text-to-3d generation. 2023 IEEE/CVE International Conference on Computer Vision (ICCV), pages
2023.
[70] Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint arXiv:2312.17142 2023.
[71] Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation. arXiv preprint 2xXi::2402.04324, 2024.
[72] Yixuan Ren, Yang Zhou, Jimei Yang, Jing Shi, Difan Liu, Feng Liu, Mingi Kwon, and Abhinav Shrivastava. Customize-a-videw One-shot motion customization of text-to-vides diffusion models, 2024.
[73] Alex Rogozhnikov. Einops:
Clear and reliable tensor manipulations with einstcin- notation. In International Conference on Learning Representations, 2022.
[74] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 10684-10695, 2022.
[75] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22510, 2023.
[76] MunwayML. Stable diffusion v1.5 model card. https: https://huggingface co/runwamm/ stable-diffusion
2022.
[77] Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, 128( 10):2586-2666, 2020.
[78] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systemx, 35:25278-2521 25294, 2022.
[79] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model, 2023.
[80] Yichun Shi, Peng Wang. Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv:2308. 165 2023.
[81] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An. Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.1477
2022.
[82] Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, lurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. In International Conference on Machine Learning, pages 31915-31929, PMLR, 2023.
[83] Stability. Stable diffusion v2 model card. http:://moggingfaae.ooosttatti 2022. stable-diffusion2-ddphh.
[84] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Igm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.0500 050544 2024.
16

---

[85] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. ArXiv, abs/2309.16655. 2023.
[86] Haithem Turki, Jason Y. Zhang, Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12375-12385, 2023.
[87] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michal- ski, and Sylvain Gelly. FVD: new metric for video generation, 2019.
[88] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, 2017.
[89] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion, 2024.
[90] Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-I-video: Multi-text to long video generation via temporal co-denoising, arXiv preprint arXiv:2305. 18264, 2023.
[91] Fu-Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Guanglu Song, Yu Liu, and Hongsheng Li. Animatelem: Accelerating the animation of personalized diffussin models and adapters with decoupled consistency learning. arXiv preprint arXiv:2402.00769 2024.
[92] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-vides technical report. arXiv preprint arXiv:2308.0657/ 2023.
[93] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d genera- tion. arXiv preprint arXiv:2312.02201, 2023.
[94] Qian Wang, Weiqi Li, Chong Mou, Xinhua Cheng, and Jian Zhang. 360dvd: Control- lable panorama video generation with 360-degree video diffusion model. arXiv preprint WrXiv:2401.06578 2024.
[95] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang. Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Vidcocomposer: Compositional video synthesis with motion controllability. 2023.
[96] Zhengyi Wang, Cheng Lu, Yikai Wang. Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu, Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
[97] Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403.055 050344 2024.
[98] Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.0366 2023.
[99] Wenming Weng, Ruoyu Feng. Yanhui Wang, Qi Dai, Chunyu Wang, Dacheng Yin. Zhiyuan Zhao, Kai Qiu, Jianmin Bao. Yuhui Yuan, Chong Luo, Yueyi Zhang, and Zhiwei Xiong. Artev: Auto-regressive text-to-video generation with diffusion models. arXiv preprint arXiv:2311.188.2 2023.
[100] Sangmin Woo, Byeongjun Park, Hyojun Go, Jin- Jin-Young Kim, and Changick Kim. Har- monyview: Harmonizing consistency and diversity in one-image-to-3d 2023.
[101] Chao- Yuan Wu and Philipp Krahenbuhl. Towards long-form video understanding. In Pro- ceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 1884-1894, 2021.
17

---

[102] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 7623-7633, 2023.
[103] Ruiqi Wu, Liangyu Chen, Tong Yang, Chunle Guo, Chongyi Li, and Xiangyu Zhang. Lamp: Learn motion pattern for few-shot-based video generation. arXiv preprint arXiv:2310.1076, 2023.
[104] Wenpeng Xiao, Wentao Liu, Yitong Wang, Bernard Ghanem, and Bing Li. Automatic animation of hair blowing in still portrait photos. In ICCV, 2023.
[105] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors, arXiv preprint arXiv:2310.12/90 2023.
[106] Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient video generation, 2023.
[107] Wilson Yan, Yunzhi Zhang. Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2 10157, 2021.
[108] Jiayu Yang, Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong Li. Consistnet: Enforcing 3d consistency for multi-view images diffusion. arXiv, 2023.
[109] Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. arXiv preprint wxXi:2240..03162 2024.
[110] Shuai Yang, Yifan Zhou, Ziwei Liu,, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-vider translation. In ACM SIGGRAPH Asia Conference Proceedings, 2023.
[111] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 2023.
[112] Jianglong Ye, Peng Wang, Kejie Li, Yichun Shi, and Heng Wang. Consistent- 1-to-3: Consistent image to 3d view synthesis via geometry-awarr diffusion models, 2023.
[113] Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arXiv:2312.17225, 2023.
[114] Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, and Yu Qiao. 4diffu- sion: Multi-view video diffusion model for 4d generation. arXiv preprint arXiv:2405.20674 2024.
[115] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
[116] Mingyuan Zhang, Daisheng Jin, Chenyang Gu, Fangzhou Hong, Zhongang Cai, Jingfang Huang, Chongzhi Zhang, Xinying Guo, Lei Yang, Ying He, et al. Large motion model for unified multi-modal motion generation. arXiv preprint 2rXiv:2404.01284, 2024.
[117] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang. Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:231 1.04145, 2023.
[118] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-fin controllable text-to-video generation. arXiv preprint arXiv:2305.13077 2023.
[119] Yiming Zhang, Zhening Xing, Yanhong Zeng, Youqing Fang, and Kai Chen. Pia: Your personalized image animator via plug-and-play modules in text-to-image models. arXiv preprint arXiv:2312.1396. 2023.
18

---

[120] Chuanxia Zheng and Andrea Vedaldi. Free3d: Consistent novel view synthesis without 3d representation. arXiv, 2023.
[121] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.1018,, 2022.
19

---

Vivid-ZOO: Multi-View Video Generation with Diffusion Model
- Supplementaryy Material
In this appendix, we provide additional content to complement the main manuseript:
Appendix A: Discussions about limitations of the current method and possible future improvements.
Appendix B: Preliminaries about diffusion and latent diffusion models.
Appendix C: Evaluation metrics for multi-view video generation.
Appendix D: Details about the multi-view captioned video dataset construct.
Appendix E: More details of our proposed model, including the spatial and temporal module.
Appendix F: Additional qualitative visualized results.
Appendix G: Societal impact, ethic concerns, dataset copyrights, and our safeguard policies.
A Limitations and future works
While our method takes step forward in T2MVid generation, our method can be improved few aspects.
A.1 Qualitative quality
For example, the visual quality of generated videos is not as high as that of multi-view image diffusion models due to the complexity of modeling multi-view videos. The spatial module of method can be replaced with more advanced multi-view image diffusion models [34], to improve the performance of multi-view consistency. A large dataset of multi-view videos can be constructed, which further improves our method.
A.2 Lighting
For lighting and rendering, we followed the settings [80] to ensure fair comparisons. Since they used point light sources, our learned model also generates multi-view videos under the assumption of point light sources, which may result in different exposures across the viewpoints the videos, as shown in Fig. III. Future work could explore generating videos that simulate more complex ambient lighting settings and even achieve controllable lighting for different viewpoints in the generation process.
A.3 Topic of generation
Currently, the proposed Vivid-ZOO mainly supports generating multi-view videos for dynamic creatures with natural motions. Though we can also generate some categories like humans (astronaut and horse, Fig. II) and common objects (waving flag, Fig. IV), we believe our research can further inspire the community develop more T2MVid techniques for the generation of more diverse and complex topics, similar to image diffusion model counterparts. For example, more specialized models that generate man-made artifacts like moving cars or articulated furniture, more powerful models that generate multiple dynamic objects with complex motions, and more diverse models like Vivid-Scene that generate dynamic scenes like a stormy sea, an erupting volcano.
B Background
Diffusion model. Diffusion models [27] learn to model a data distribution by iteratively recovering original data from noisy one, which comprises forward and backward phases. Given clean sample de from the data distribution Paata, the forward process gradually adds Gaussian noise to the sample, generating random latent variables 22 at each time step E [0, T):
q( 1T-1) A(x /I B, 3,I)
20

---

where Bt is hyperparameter that determines the noise schedule. With a large time step, 2T is assumed be perturbed into a standard Gaussian noise. Given 27, the denoising network trained to gradually remove the noise and recover the original data:
p=(
(4 t), (2x,
(8)
where 0 denote parameters of the denoising network, and Σ are mean and variance, respectively.
Latent diffusion model (LDM). By embedding images into low-dimensional latent codes, latent diffusion models [74] (LDMs) perform the diffusion process in the latent space of latent codes, significantly reducing the computational cost. Typically, LDMs employ a pre-trained autoencoder (e.g., VQ- VAE [88], which consists of an encoder and decoder, where the encoder transforms images into the latent space and the decoder maps denoised latent codes back to the pixel space.
C Details about evaluation metrics
Multi-view Text alignment: Text-to-2D-videe diffusion models (e.g.. [23, 36, 102]) adopt CLIP score [68] to measure the alignment between an input text and corresponding generated 2D video. To evaluate the alignment between the input text and generated multi-view videos, we first measure the CLIP score for each view and then average the CLIP scores for all views.
Video quality: We adopt Frechet Video Distance (FVD) to measure the quality of generated multi- view videos. FVD is the standard metric adopted by many 2D video generation [95, 106] and animation methods [6, 105]. FVD measures the quality of generated videos by measuring the data distribution between generated and training videos, where I3D networks pre-trained on the Kinetics dataset [10] are employed to extract features.
Human preference. We conduct a user study to measure the overall quality of generated multi-view video text alignment, temporal coherence and multi-view consistency.
We adopt paired comparison. We invite 10 subjects to participate the user study. For each subject, we display two multi-view video sequences generated by different generation methods as well as the corresponding input text prompt, where the two results are arranged in an up-and-down order, and resulting multi-view video sequence is displayed in a single row. We told each subject that the task is to generate four orthogonal views dynamic 3D object. Then the subject was asked to choose a multi-view video sequence whose overall quality is better in text alignment, temporal coherence and multi-view consistency. For comparison with MVdream + Diff,,, we use 10 input prompts to generate multi-view videos for the user study. For the ablation study, there five methods in total, leading to more combinations of paired comparisons. We hence use 5 input prompts in the use study for the ablation study.
D More details about our multi-view video dataset
2D video diffusion models [6] have pointed out that data curation is essential to improve the generation performance of diffusion models. The training of 2D Video diffusion models can be degraded if the training dataset contains many static 2D videos [6]. Hence, we developed an animated 3D object selection tool that automatically discards 4D objects that are static or close to static. In particular, given a 4D object, we first render a video from a single viewpoint. For efficiency, instead of using advanced optical flow algorithms, we calculate the pixel difference between different frames in each video, in order to identify whether 4D objects are static.
With the selected 4D objects, we employ Cycles as the rendering engine to render multi-view videos. Given a 4D object, we render 24-view videos with the resolution of 512 x 512. We first uniformly distribute the camera poses around the normalized 4D object and then add subtle disturbances. The radius of a camera to a 4D object is in the range of [2.2 2.6], and a camera's 's height is in [0.8 1.2]. The background of multi-view video sequence is randomly filled in gray color. The frame number of multi-view video sequences is diverse, depending on its 4D objects.
To further improve the quality of our dataset, we manually discard low low- www-qualty data from our dataset. We found that many multi-view videos contain distorted shapes or motions. We remove these low- quality data to avoid their negative effect on the training generation models. On the other hand, we
hhaps:/wwww.ccllesseeeeeeee
21

---

also remove multi-view videos that contain large translation motions. Due to the large translation motions, objects disappear in a few frames, which leads to these frames having only a background. In addition, many 4D objects are textureless in Objaverse [17]. We keep 10% of these textureless objects. For caption generation, we adopt a caption method i.e., Cap3D which is designed to caption multi-view images of 3D object. The Cap3D) is used to describe multi-view frame sequence sampled from a multi-view video sequence.
E
More implementation details
Training settings. Table II provides detailed information on the hyperparameter settings and hardware configuration used for model training. During training, four orthogonal views are randomly chosen, leading to four-view videos. For a video from viewpoint, the starting frame is randomly selected, and then we extract one frame every 3 frames. The frame size is 256 x 256 and the frame number set to 16 (see Table I).
Table I: Training dataset settings
Name
Parameter value
view number sample size
4
256 x 256
sample stride
3
frame number
16
Table II: Training settings
Name
Parameter value
noise scheduler type
DDIMScheduler
noise scheduler timesteps number noise scheduler start beta noise scheduler end beta noise scheduler beta schedule noise scheduler steps offset noise scheduler clip sample optimizer
1000
0.00085
0.012
linear 1
false
AdamW
learning rate
0.0001
train step number
100000
batch size
16
CPU memory size in total
320G
GPU type
NVIDIA A100
GPU number
Inference settings. Table III lists the hyperparameters and hardware configurations in the inference stage. The resolution and number of frames in the generated video are the same as those in the training settings.
Table III: Inference settings
Name
Parameter value
sample step number
50
CFG weight
7.5
CPU memory
30G
GPU type
NVIDIA A100
GPU number
1
22

---

View axtis
View
Temporal
Temporal
Multi- View consistency using Multi- View Spatial Module
Temporal coherence
using Multi-Viee Temporal Module
Figure I: The multi view spatial module of our method ensures multi-view consistency of generated multi-view videos via capturing correlations across different views. The multi-viow temporal module enforces temporal coherence via capturing temporal correlations among frames in video of viewpoint.
E.1
More details about multi-view spatial module
Our multi-view spatial module adapts Stable Diffussinn to handle multi-view videos, and reuses the pre-trained weight of MVDream[80J. In this main paper, we have elaborated how adapt Stable Diffusion's self-attention layers to handle multi-view videos' 6D feature tensors. With the adaption, the self-atention layers model multi-view consistency among views, shown in Fig. I. For other layers, we first reshape the features of multi-view videos using the rearrange operation: rearrange F, K N w d + (b N) temporal module, we transform the dimensions of the output feature F' back with rearrange operation: (F', (b N)K h w d K N d).
d). When the features are fed the multi-view
E.2 More details of multi-view temporal module
Fig. shows how our multi-view temporal module leverages 2D temporal layers to caption temporal correlations among frames in each view video.
Both 3D-2D Alignment layer and 2D-3D Alignment layer are implemented using a Linear MLP layer. We experimented with 2-1ayer/3-layer setups, but there was no improvement in performance. Therefore, we use a simple single layer for implementation. In this paper, we reuse AnimateDir the multi-view temporal module, where the 2D-in-layer refers the "project_in" in" layer and 2D-out-layy layer refers to " "ouut" layer in the AnimateDiff.
F
Additional results
F1 Additional text multi-view video examples
Some additional experimental results are presented from Fig. to Fig. VIII. We strongly recommend the readers watch the corresponding videos anonymous website to get better feel for the movement of the objects in the picture.
F.2
More ablation study results
Fig. IX and Fig. X show the contributions of the 3D-2D and 2D-3D alignment layers respectively.
23

---

View
View
View
View
Figure II:
Text prompt: astronaul riding horse, 3d asset
View
View
View
View 4
Figure III: Text prompt: A full-bodied tiger walking, 3d asset
G Societal impact and ethic concerns
G.1 Positive societal impact
Our method is able to generate vivid multi-view videos for dynamic creatures. Therefore, our method can be directly applied to enhance creativity and entertainment, e.g., creating AR/VR and game assets. Due to the availability of multi-view videos, our method can also be used for art creation and interactive educational content, which could benefit many people, like artists, designers, educators, and film and television creators. Researchers in fields such as biology, ecology, and zoology can benefit from this technology by creating accurate multi-view visualizations of dynamic creatures, which can aid in research, analysis, and presentations.

---

View
View
View
View
Figure IV: Text prompt: blue flag attached flagpole with smooth curve, 3d asset
View
View
View
View 4
Figure V: Text prompt: spiked sea turtle, 3d asset
G.2
Negative societal impact
diffusion method is based on one existing text-to-multi-view image model and one text-to-video model. Therefore, its internal representation may inherit some bias from these two base models. Our multi-view video generation could be exploited to create highly realistic deepfakes. These fake videos can be used for malicious purposes such as spreading disinformation. manipulating public opinion, or creating fake profiles for fraudulent activities. If users input provoca- tive prompts or maliciously fin-tune the model parameters, our model could potentially generate harmful videos, such as those containing vulgarity, gore, or violence. However, since our model fine-tuned on the dynamic creature dataset, we believe the risk of such content is significantly lower compared to previous open-domain generative models like MV-Dream [80] and SVD [6]. Additionally, we will implement gated access and usage guidelines for our model and continuously monitor community usage and feedback to prevent such harmful content much possible.
Our Serrtto--uuti-viee--Weee
25

---

View
View
View
View
Figure VI: Text prompt: dog wearing outfit, 3d asset
View
View
View
View
Figure VII: Text prompt: panda is dancing
G.3 Copyright
Our dynamic dataset is directly sourced from the already open open-sourced and published Objaverse [17] dataset and used solely for scientific research purposes. Therefore, does not infringe on the legal rights and copyrights of the original 3D/4D model creators and collectors.
26

---

View
View
View
View
Figure VIII:
Text prompt: a pixelated Minecraft character walking, 3d asset
w/o 3D-2D alignment
Ours
View O
View
View
View
Text prompt: a blue-winged dragon, also depicted flying monster, 3d asset
Figure IX: Visual comparison of the contributions of our 3D-2D alignment layers
27

---

w/o 2D-3D alignment
Ours
View O
View
View
View 3
Text prompt: an astronaut riding a horse, 3d assel
Figure X: Visual comparison of the contributions of our 2D-3D alignment layers
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims accurately reflect the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We describe the limitations of our work Limitations (Sec. A).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper contains the overall structure of the model Sec. 3, and Sec. E of the appendix contains detailed model parameters.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer:
[No]
Justification:
We will release the code and pretrained models upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm
mublic/guides htte//oooeeuSiiiseeeeeeee httiat//uuiie//ooouuubiiisssss
cetibmisssoonPoii oeiionnopiyy ooiicylyyyy)oo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /ttppp.ccppppaaaap ctcccccpp.o wwtpccccppcccceeec.cc.ccc...clct gttp//gwiee//eeeeeeeeee /gwddes/ooeeuubmiii
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify these details in Sec. E. We will make all the code and pretrained models publicly available upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars in Tab. 1. Yet. Our method is for the multi-view video generation task, and there are no standard qualitative metrics available.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
31

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in Sec. 4.2 and Sec. E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://neen /ttear///////pe///eee
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: have discussed Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We have discussed the safeguards in Sec. G.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets used in the paper are cited and credited. The license and terms of are mentioned and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
33

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "DqsfM6VZ7A", "reviewer": "Reviewer_MoTs", "summary": "The paper presents a novel diffusion-based pipeline designed to generate multi-view videos from text prompts. The core challenge addressed by the paper is the generation of dynamic 3D object videos from multiple viewpoints, a task not extensively explored with existing diffusion models. The authors factor the problem into viewpoint-space and time components, allowing for the reuse of layers from pre-trained multi-view image and 2D video diffusion models. This reuse is intended to ensure consistency across views and temporal coherence, thereby reducing training costs. The paper also introduces 3D-2D and 2D-3D alignment layers to bridge the domain gap between the training data of multi-view images and 2D videos. Additionally, a new dataset of captioned multi-view videos was created.", "questions": "N/A", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a relatively unexplored area of diffusion models, extending their application to multi-view video generation from text prompts. The task is extremely challenging and important.\n2. By reusing layers from existing models, the approach leverages the strengths of pre-trained systems while minimizing the requirement for large-scale data and extensive computational resources, addressing a common barrier in training new machine learning models.\n3. The introduction of 3D-2D and 2D-3D alignment layers is an interesting solution to the issue of domain gaps between different types of training data.\n4. The creation of a new, although smaller, dataset for training and testing in this niche area is crucial, as it provides a resource that can be built upon by subsequent research. However, this does not seem to be license-free?", "weaknesses": "1. The paper acknowledges the challenge of not having a large-scale, captioned multi-view video dataset, which is crucial for training robust diffusion models. The authors attempt to mitigate this by constructing a smaller dataset, but this could limit the model’s generalization capabilities and performance across diverse scenarios. Further, this may be very challenging if we would like to further scale to scene-level contents.\n2. The alignment between pre-trained 2D video and multi-view image diffusion models is crucial, yet challenging. The proposed 3D-2D and 2D-3D alignment layers are a solution, but the effectiveness of these layers in truly bridging the domain gap might not fully compensate for the inherent differences in training data types (real-world 2D videos versus synthetic multi-view data), potentially affecting the quality of generated videos. Since the model is trained on an entirely different domain of object-centric videos, there is very little we can tell in terms of video quality degradation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the positive comments and constructive suggestions. We are encouraged that the reviewer agrees that \"The task is extremely challenging and important\", \"tackles a relatively unexplored area of diffusion models,\", \"a novel diffusion-based pipeline\", \"a new dataset\",  \"minimizing the requirement for large-scale data and extensive computational resources\", and \"an interesting solution\".\n\n**S4. License of the new dataset.**\n\nWe greatly appreciate your recognition of our contributions. Our dataset is built from 4D assets provided by Objaverse [15], which are released under the ODC-BY license. This license allows us to make our dataset available using the same license.\n\n**W1: The authors attempt to mitigate the challenge of lacking a large-scale dataset by constructing a smaller dataset, but this could limit the model’s generalization capabilities and performance across diverse scenarios.**\n\nThank you for your valuable comments. Yes, ideally, if we could construct a large-scale dataset of millions or billions of Multi-View (MV) video sequences, similar to 2D image/video diffusion methods, the method could achieve excellent generalization.\nHowever, the cost of constructing such a large-scale dataset is enormous. \n\n\nInstead, although we create a smaller dataset, we resort to re-using the layers of the pre-trained MV image and 2D video diffusion model which have learned rich knowledge from large-scale data of MV images and 2D videos, to improve the generalization performance of our method. We observe that our method can generate meaningful results that are rather irrelevant to our MV video dataset, since our method effectively leverages the knowledge learned by pre-trained MV image and 2D video diffusion models.  Our method opens up a new path, though its generalization performance is not yet optimal.\n\n\nWe appreciate your suggestions. In our future work, we will further improve our dataset, including its size and diversity. For scene-level contents, we can similarly construct a small dataset of scene-based MV videos to train our method while replacing MVDream with a pre-trained scene-based MV image diffusion model.\n\n\n\n**W2: The alignment between pre-trained 2D video and multi-view image diffusion models is crucial, yet challenging.**\n\nThank you for your comments. We agree that it is challenging to bridge the domain gap between real-world 2D videos and synthetic multi-view data, due to the inherent differences. To reduce the difficulty in bridging the domain gap, \nour 3D-2D alignment is to project the feature into the latent space of the pre-trained 2D video diffusion model's 2D temporal attention layers, instead of all temporal layers. In other words, the temporal attention layers capture temporal correlation among features, which can reduce the difficulty of alignment to some extent.\nConsequently, although a 3D-2D alignment layer is implemented as only an MLP, our method can generate high-quality MV videos. \n\nWe are inspired by your valuable comments. To further enhance the quality of the generated videos, we will jointly train our model on real-world 2D videos, and our dataset, such that real-world 2D video data is leveraged to preserve the original video generation performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cxSqfFNsBC", "reviewer": "Reviewer_K81K", "summary": "This paper focuses on generating multi-view consistent videos given a text prompt, specifically from 4 orthogonal views. It fine-tunes MVDream with a temporal layer adopted from AnimateDiff. To mitigate the domain gap between the two layers, some connected layers called 3D-2D and 2D-3D alignment layers are introduced. They train the newly introduced layers on a 4D subset of Objaverse. The proposed model can generate multi-view videos. The authors conduct evaluations to verify the new layers and choices of the base models.", "questions": "1. As far as I know, MVDream is fine-tuned (fully?) from SD2.1, while AnimateDiff v2 is fine-tuned from SD1.5; this could cause an additional discrepancy when combining the two models, right?\n2. In the ablation study of \"Design of multi-view spatial module\", what is the difference between what you do and directly using AnimateDiff v2?", "limitations": "The authors discuss the limitations and negative societal impact in the appendix.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* This is the first paper to study the problem of text-conditioned multi-view video generation, based on my best knowledge. Perhaps [1] is a concurrent work and should be acknowledged.\n* The paper is easy to follow.\n* A multiview video dataset extracted from Objaverse is collected.\n\n[1] Kuang, Zhengfei, et al. \"Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control.\" arXiv preprint arXiv:2405.17414 (2024).", "weaknesses": "Despite the fact that the paper is the first to study text-conditioned multi-view video generation, I think there could still be room for improvement in the evaluation section in terms of the justification of the method as stated below:\n\n1. There are very few generated videos (8 in the supplementary material) shown in the paper. Is this because the success rate is low? Since it is a forward-based method, I expect the time to generate the videos will not be too long. If so, it would be good to discuss the failure cases in the limitation section.\n\n2. As for human evaluation, only five videos are used for comparison.\n\n3. There is a lack of details about how many videos are used to compute CLIP and FVD scores. Also, are the prompts used to generate these videos separate from the training set? FVD was used to evaluate temporal coherence, but FVD is shown to be not good at it (see [2]).\n\n    [2] Ge, Songwei, et al. \"On the Content Bias in Fréchet Video Distance.\" CVPR. 2024.\n\n4. The motions in most generated videos are small and are not indicated even in the text prompt. This could limit the use case of the model in generating specific motions.\n\n5. All the generated videos contain a grayish background. I'm fine with it if the ultimate goal is to generate a 4D asset, and I think it is a reasonable goal. Ideally, like MVDream, the authors may want to compare with 4D generation methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive comments. We are encouraged that the reviewer agrees that our work is \"the first paper to study\",  \"easy to follow\", and \"A multiview video dataset ... is collected\".\n \n**S1: This is the first paper to study the problem of text-conditioned multi-view video generation.** \n\nWe appreciate your recognition of our contributions. Unlike our method, concurrent work [Kuang et al.] focuses on generating multiple videos of the same scene given multiple camera trajectories. We will cite [Kuang et al.].\n\n\n\n**W1: Success rate of the proposed method.** \n\nThe success rate of our method is not low. We used diverse prompts to challenge our method, including some from existing methods: the prompt used in Fig. 4 in the main paper is from MVDream, the prompt in Fig. II is from MVDream and 4D-fy [3], and the prompt in Fig. VII is from AYG [43] and MAV3D [76].\n\nThe 25 Multi-View (MV) videos generated by our method (11 videos on the website mentioned in the abstract) achieve better performance in Tab. 1 of the main paper, showcasing the effectiveness of our method.  We provide more results in the PDF of the global response, as we cannot upload videos due to the NeurIPS 2024 regulations. \n\n\n**W2: As for human evaluation, only five videos are used for comparison.**\n\nAs stated in line 783 in Appendix C, we used **ten** sequences of MV videos (text prompts), for **comparison**.\nAs to the ablation study, there are five methods and ten subjects, leading to 5×2×10 =100 questionnaires per MV video sequence. To reduce the cost, we followed 4D-fy [3] and used five MV videos in the ablation.\nAs suggested, we increase the video number to 10 in the table below, showing our method outperforms the baselines in the ablation. \n\n| Method                | Human evaluation↑ |\n|--|--|\n| w/o MS w SD           | 44.88%            |\n| w/o MT w TM LoRA      | 11.25%            |\n| w/o 3D-2D alignment   | 53.50%            |\n| w/o 2D-3D alignment   | 54.50%            |\n| Ours                  | **80.25%**            |\n\n\n\n**W3.1: Number of MV videos to compute the CLIP and FVD scores.** \n\nThank you for your valuable suggestion. The number is 25. We will clarify this in our final version.\n\n\n**W3.2: Are prompts separate from the training set?**\n\nYes. Most prompts used to generate these videos are separate from the training set, and only two prompts are from the training set. We will clarify this in our final version.\n\n\n**W3.3: FVD is shown to be not good at evaluating temporal coherence [R2].**\n\nwe adopt FVD, since many 2D video generation methods (e.g. [6, 89, 98, 99]) use it to evaluate temporal coherence. To compensate for FVD, we also used human evaluation in Tab.1.  Following the suggested paper [Ge et al.], we calculate FVD-new by replacing I3D with videoMAE. The table below shows that our method consistently achieves better performance on 25 samples.\n\n| Method                  | FVD-new [Ge et al.] ↓ |\n|--|--|\n| MVDream + IP-AnimateDiff| 466.82           |\n| Ours                    | **459.31**                   |\n\n\n**W4: The motions in most generated videos are small and are not indicated even in the text prompt.** \n\nThe motions in at least **five** generated videos are not small (see the web mentioned in the abstract and Figs. 4, III,  VII, VIII, IX). And the motions in the **five** generated videos are indicated in text prompts (see Figs. II, III,  VII, VIII, IX).   \nWe intentionally omit motion descriptions in some prompts to evaluate how our method performs with various text prompts. Despite such omission, our method still generates natural and vivid motions for the corresponding objects rather than static or improper ones.\nWe provide more results that contain motion descriptions in Fig. C (see PDF of global response). \n\n\nIt is worth noting that our method re-uses the pre-trained temporal layers of AnimateDiff in our temporal module, instead of training from scratch, **\"minimizing the requirement for large-scale data and extensive computational resources\"**, as recognized by reviewer MoTs. Despite such reductions, we find that the **motion magnitude** of generation results by **our method is much larger** than MVdream + IP-AnimateDiff (see Fig. 4), thanks to our dataset and our 3D-2D and 2D-3D alignment layers.\nOur motion generation performance can be further improved by replacing AnimateDiff with a 2D video diffusion model with better motion capture performance.\n\n\n**W5: Comparison with 4D generation methods.**  \n\nDue to the time limitation,  according to constructive suggestions raised by Reviewer CcuB, we compare our method with 4D generation methods 4D-fy, AYG [43], and MAV3D [76] by treating them as MV video generation methods.\nSince AYG [43] and MAV3D [76] have not released their code, we take generated videos from their website to conduct qualitative experiments. As shown in Figs. A and B in the PDF of global response, our method outperforms both MAV3D and AYG.  AYG uses motion amplification to amplify its generated motions; however, it distorts the magic wand (see Fig.A in the global response).\n\n\nThe table below shows that our method outperforms 4D-fy on ten samples. Since 4D-fy needs around 18 hours to train a 4D object, due to the time limitation, we obtain only ten 4D-fy results.\n\n| Method | FVD-new [Ge et al.] ↓ | Overall↑ |\n|--|--|--|\n| 4D-fy  | 661.57               | 37%      |\n| Ours   | **443.99**           | **63%**  |\n\n\n\n**Q1: MVDream is fully fine-tuned from SD2.1.**\n\nMVDream had indeed provided a model fine-tuned from SD 1.5 on its GitHub page. Hence, there is no additional discrepancy.\n\n**Q2: In the ablation study of \"Design of multi-view spatial module\", what is the difference between what you do and directly using AnimateDiff v2.**\n\n\nThe largest difference is that AnimateDiff v2 does not have camera embedding, which can not enable viewpoint control. Moreover, we finetune the baseline on our dataset, while  AnimateDiff v2 is trained on Web10M.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j2U3EMbQUf", "reviewer": "Reviewer_CcuB", "summary": "This paper proposes a Text-to-Multi-view-Video generation algorithm capable of generating multi-view video content based on text descriptions. The method decouples the multi-view spatial and temporal dimensions, utilizing pretrained models like MVDream for multi-view generation and animatediff for video generation. It addresses domain gaps between different features through 3D-2D/2D-3D alignment. The authors trained on captioned 4D data from objverse and conducted various ablation studies to validate their approach.", "questions": "1. Regarding the 'overall' metric in Table 1, does it refer to the results of a user study, or is it a comprehensive representation of the previous two results?\n2. What is the test dataset in Table 1? How large is the dataset?\n3. Will the 4D dataset proposed in the paper be made publicly available?\n4. In the current framework, what are the results if MVDream and AnimateDiff are not fixed, or if fine-tuning is done with a very small learning rate?\n5. Did the authors conduct experiments on generating 4D contents based on image prompt, and are there any visual results?", "limitations": "Currently, the results only compare against a self-designed baseline, lacking a broader comparison with other algorithms.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method proposed by the authors achieved impressive visual results, with performance metrics showing improvements over naively using MVDream + AnimateDiff.\n2. The 3D-2D/2D-3D alignment proposed by the authors reduces the number of parameters and training time needed. Given the limited training data, this approach also helps mitigate degradation in pretrained models.", "weaknesses": "1The authors claim their method is the first to do text-to-multi-view video generation, but as far as I know, there are many works related to 4D generation currently, such as MAV3D, AYG, 4D-fy, including recent works like 4Diffusion and Diffusion4D. I believe the authors may want to emphasize that their work focuses on novel view synthesis rather than involving 4D representation, such as Deformable GS, Dynamic Nerf, etc. However, these are just differences at the algorithmic level; fundamentally, they are all engaged in multi-view video generation. Therefore, the authors' comparison in Table 1 is insufficient, and relevant 4D generation works should also be included for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the positive comments and constructive suggestions. We are encouraged that the reviewer agrees that our work \"achieved impressive visual results\", \"performance metrics showing improvements\", \"reduces the number of parameters and training time needed\", and \"mitigate degradation in pretrained models.\"\n\n\n**W1.1. The authors claim their method is the first to do text-to-multi-view video generation.** \n\nThank you for your comments. Our claim is that our method is ``the first study on T2MVid **(Text-to-Multi-view-Video) diffusion models**\", instead of the first to do text-to-multi-view video generation,  as stated in line 82 of the main paper. \n\nDifferent from MAV3D, AYG and 4D-fy which use the pre-trained diffusion models to optimize 4D representations, we focus on presenting a diffusion model that generates Multi-View (MV) videos from texts. \nThe concurrent work Diffusion4D presents a diffusion model that generates **an orbital video** around 4D content, and 4Diffusion presents a **video-conditioned** diffusion model that generates MV videos from a monocular video. The focus of these two methods is different from ours. We will cite the Diffusion4D and 4Diffusion.\nSince the concurrent Diffusion4D and 4Diffusion appeared after our submission, we will provide detailed discussions in our final version.\n\n\n\n\n**W1.2. Including relevant 4D generation works for comparison.**\n\nThank you for your constructive suggestions. Following your suggestions, we treat 4D-fy, AYG [43], and MAV3D [76] as MV video generation methods and then compare our method with these methods.\n\nSince AYG [43] and MAV3D [76] have not released their code, we obtained their generated videos from their websites and conducted qualitative experiments. As shown in Figs. A and B of the global response, our method outperforms both MAV3D and AYG. AYG employs motion amplification to enhance its generated motions, yet, this distorts the magic wand (see Fig.A in the global response).\nThe table below shows our method achieves better performance than 4D-fy on ten testing samples.\n \n| Method | FVD ↓ | Overall↑ |\n|--------|-------|----------|\n| 4D-fy  | 2189.87 | 37%     |\n| Ours   | **1621.92** | **63%** |\n\n\n\n**Q1. 'overall' metric in Table 1.**\n\nThe overall metric refers to the results of a user study using paired comparison. The evaluation metric in lines 264-265 of the main paper provides more details. \nWe will clarify the overall metric in Table 1 in our final version.\n\n\n\n**Q2. Testing dataset in Table 1.**\n\nTo the best of our knowledge, there is no established and widely used testing dataset for testing text-to-multi-view video generation. \nFurthermore, the testing data in existing text-to-4D generation methods (e.g., 4D-fy, MAV3D, AYG) are different from each other.  \n\nIn Table 1, we used **25** diverse text prompts (i.e., testing dataset) to evaluate our method, where 11 prompts are shown on the web mentioned in the abstract. Noting that some text prompts are taken from existing 3D or 4D generation methods to challenge our method, for example, the prompt used in Fig. 4 in the main paper comes from MVDream, the prompt in Fig. II comes from MVDream and 4D-fy [3], and the prompt in Fig. VII comes from AYG [43] and MAV3D [76]. We will release our testing dataset in our final version.\n\n\n**Q3. Will the 4D dataset proposed in the paper be made publicly available?**\n\nYes. Our 4D dataset will be made publicly available. \n\n**Q4. What are the results if MVDream and AnimateDiff are not fixed.**\n\nWe only re-use the temporal layers of AnimateDiff instead of all its layers. When both MVDream and the temporal layers of AnimateDiff are not fixed, the GPU memory required for model training is substantially increased, since the number of trainable parameters is significantly increased. Training such a model, **runs out of memory** on 8 A100 (80G) using the same setting as our method. Similarly, when MVDream is not fixed, the model also **runs out of memory** with the same setting.\n\nAdditionally, we build another baseline, namely, Ours (trainable T-AnimateDiff), that does not fix the temporal layers of AnimateDiff in our method.  The trainable parameters of this baseline increase by 600\\%,   significantly escalating the difficulties and computational cost of training the diffusion model. The table below shows the performance of our method is largely degraded, when the temporal layers of AnimateDiff are not fixed.  \n\n| Method                       | FVD ↓   |\n|------------------------------|---------|\n| Ours (trainable T-AnimateDiff) | 2796.86 |\n| Ours                          | **1634.28**  |\n\n\n**Q5. Did the authors conduct experiments on generating 4D content based on image prompt**\n\nWe haven't conducted experiments based on image prompts. In this paper, we mainly focus on the text-to-multi-view-video diffusion model. Thank you for your suggestions.   Image-to-4D is another interesting and meaningful topic, we will explore it in our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jDSrspM22D", "reviewer": "Reviewer_VRVx", "summary": "The paper introduces \"Vivid-ZOO,\" an innovative diffusion model designed for Text-to-Multi-view-Video (T2MVid) generation. This is a novel approach that addresses the challenges of generating high-quality multi-view videos from textual descriptions. Specifically, the authors propose a new method that leverages diffusion models to generate multi-view videos centered around dynamic 3D objects from text prompts. The T2MVid generation problem is effectively factorized into viewpoint-space and time components, allowing for the combination and reuse of layers from advanced pre-trained multi-view image and 2D video diffusion models. And the authors claim that it is the first study that explores the application of diffusion models to the T2MVid generation task.", "questions": "Please refer to weakness", "limitations": "The authors discuss the limitations and potential negative societal impacts of their work in the paper.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel diffusion-based approach to generate multi-view videos from text prompts, which is a relatively unexplored area in the literature.The creative factorization of the problem into viewpoint-space and time components is an innovative way to tackle the complexity of multi-view video generation.The introduction of alignment modules to integrate pre-trained models is a clever solution to address domain gaps and reuse of layers.\n2. Furthermore, by providing a captioned multi-view video dataset, the authors not only support their own research but also contribute to the wider research community, facilitating future work and lowering the entry barriers for other researchers in this domain. \n3. The proposed solution adeptly addresses a notable gap in the current literature by offering an efficient approach that does not rely on massive datasets, thereby making sophisticated multi-view video generation more accessible and applicable to a wider array of problems", "weaknesses": "1. The visual quality of the generated videos might not match state-of-the-art single-view image generation models. Future work could focus on improving the visual fidelity and realism of the generated multi-view videos.\n2. The model currently assumes point light sources, which might not always produce the most realistic renderings. Enhancements in simulating complex lighting conditions could improve the model's applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the positive comments and constructive suggestions.  We are encouraged that the reviewer agrees that  \"an innovative diffusion model\",  \"novel approach\", \"a clever solution to address domain gaps\", \"contribute to the wider research community”, and \"offering an efficient approach\".\n\n\n**W1: Future work could focus on improving the visual fidelity and realism of the generated multi-view videos.**\n\nThank you for your constructive suggestions. In our future work, we will jointly train our model on real-world 2D videos and our dataset, leveraging real-world 2D videos to improve the visual fidelity and realism of the generated multi-view videos.\n\n**W2: Enhancements in simulating complex lighting conditions could improve the model's applicability.**\n\nThank you for the suggestion. We haven't considered lighting conditions. We will explore complex lighting conditions to improve the model's applicability. Inspired by your suggestion, we will also improve our dataset by exploring complex lighting conditions. We plan to explore more advanced lighting models to better simulate complex lighting scenarios, such that a higher-quality and more realistic dataset can further improve the generation performance of the model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive comments and valuable suggestions. We are encouraged by the reviewers' positive feedback on novelty, writing, methodology, and  experiment, such as

- novelty: "a novel approach", (VRVx), "a novel diffusion-based pipeline" (MoTs),  "an innovative diffusion model" (VRVx), "an interesting solution" (MoTs), and "the first paper to study" (K81K).

- writing: "easy to follow" (K81K).

- methodology: "minimizing the requirement for large-scale data and extensive computational resources" (MoTs),     "reduces the number of parameters and training time needed" (CcuB), and "a clever solution to address domain gaps" (VRVx).

- experiment: "achieved impressive visual results" (CcuB), and "performance metrics showing improvements" (CcuB). 




**To Reviewer K81K.**

**Q1: More multi-view videos generated by our method.**
Following your suggestion, we provide additional multi-view generation results of our method in Fig. C of the attached PDF. Moreover, motion descriptions are included in the text prompts. Fig. C shows our method achieves high generation performance in terms of large motion magnitude and text alignment.


**To Reviewers K81K and CcuB.**

**Q1: Compared with 4D generation method.**
Due to the time limitation, following constructive suggestions raised by Reviewer CcuB, we compare our method with 4D generation methods 4D-fy, AYG [43] and MAV3D [76] by treating them as multi-view video generation methods.  As shown in Figs. A and B in the attached PDF, While AYG employs motion amplification to enhance its generated motions, this approach results in distortion of the magic wand, as shown in Fig. A of the global response. The table below shows that our method outperforms 4D-fy on ten samples.
| Method | FVD ↓ | Overall↑ |
|--------|-------|----------|
| 4D-fy  | 2189.87 | 37%     |
| Ours   | **1621.92** | **63%** |

Meta-review: The paper received two Accepts and two Borderline Accepts. The reviewers appreciated the new task, the proposed dataset, and some of the results and ideas. However, there are still concerns about the comparisons with existing about generalizations / performance at scale and comparisons. I agree with all these points mentioned by the reviewers and recommend Accept as poster.
Decision: Accept (poster)