Vivid-ZOO: Multi-View Video Generation with Diffusion Model

Authors: bing  li
cheng  zheng
wenxuan  zhu
jinjie  mai
biao  zhang
peter  wonka
bernard  ghanem
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-27 05:26:00

Abstract: While diffusion models have shown impressive performance in 2D image/video generation, diffusion-based Text-to-Multi-view-Video (T2MVid) generation remains underexplored. The new challenges posed by T2MVid generation lie in the lack of massive captioned multi-view videos and the complexity of modeling such multi-dimensional distribution. To this end, we propose a novel diffusion-based pipeline that generates high-quality multi-view videos centered around a dynamic 3D object from text. Specifically, we factor the T2MVid problem into viewpoint-space and time components. Such factorization allows us to combine and reuse layers of advanced pre-trained multi-view image and 2D video diffusion models to ensure multi-view consistency as well as temporal coherence for the generated multi-view videos, largely reducing the training cost. We further introduce alignment modules to align the latent spaces of layers from the pre-trained multi-view and the 2D video diffusion models, addressing the reused layers' incompatibility that arises from the domain gap between 2D and multi-view data. In support of this and future research,  we further contribute a captioned multi-view video dataset. Experimental results demonstrate that our method generates high-quality multi-view videos, exhibiting vivid motions, temporal coherence, and multi-view consistency, given a variety of text prompts.

Main paper: 
 introduction: 
 Multi-view videos capture a scene/object from multiple cameras with different poses simultaneously, which are critical for numerous downstream applications [5, 55, 62, 39, 40] such as AR/VR, 3D/4D modeling, media production, and interactive entertainment. More importantly, the availability of such data holds substantial promise for facilitating progress in research areas such as 4D reconstruction [44, 48] , 4D generation [3, 49] , and long video generation [9, 101] with 3D consistency. However, collecting multi-view videos often requires sophisticated setups [1] to synchronize and calibrate multiple cameras, resulting in a significant absence of datasets and generative techniques for multiview videos.In the meantime, diffusion models have shown great success in 2D image/video generation. For example, 2D video diffusion models [6, 23, 28, 81] generate high-quality 2D videos by extending image diffusion models [74, 83] . Differently, multi-view image diffusion models [80, 34, 53, 93] are proposed to generate multi-view images of 3D objects, which have demonstrated significant impact in 3D object generation [45] , 3D reconstruction [66] , and related fields. However, to the best of our knowledge, no other works have explored Text-to-Multi-view-Video (T2MVid) diffusion Text prompt: a yellow and black striped wasp bee, 3d asset models. Motivated by recent 2D video and multi-view image diffusion models, we aim to propose a diffusion-based method that generates multi-view videos of dynamic objects from text (see Fig. 1 ).Compared to 2D video generation, T2MVid generation poses two new challenges. First, modeling multi-view videos is complex due to their four-dimensional nature, which involves different viewpoints as well as the dimensions of time and space (2D). Consequently, it is nontrivial for diffusion models to model such intricate data from scratch without extensive captioned multi-view video datasets. Second, there are no publicly available large-scale datasets of captioned multi-view videos, but it has been shown that billions of text and 2D image pairs are essential for powerful image diffusion models [74, 76, 83] . For example, Stable Diffusion [76] is trained on the massive LAION-5B dataset [78] . Unlike downloading 2D images available on the Internet, collecting a large quantity of multi-view videos is labor-intensive and time-consuming. This challenge is further compounded when high-quality captioned videos are needed, hindering the extension of diffusion models to T2MVid generation.In this paper, instead of the labor-intensive task of collecting a large amount of captioned multi-view video data, we focus on the problem of enabling diffusion models to generate multi-view videos from text using only a comparable small dataset of captioned multi-view videos. This problem has not been taken into account by existing diffusion-based methods (e.g., [6][23] ). However, studies have revealed that naively fine-tuning a large pre-trained model on limited data can result in overfitting [30, 75, 115] .Our intuition is that we can factor the multi-view video generation problem into viewpoint-space and time components. The viewpoint-space component ensures that the generated multi-view videos are geometrically consistent and aligned with the input text, and the temporal component ensures temporal coherence. With such factorization, a straightforward approach is to leverage large-scale multi-view image datasets (e.g., [51] [67] ) and 2D video datasets (e.g., Web10M [4] ) to pre-train the viewpoint-space component and temporal component, respectively. However, while this approach can largely reduce the reliance on extensive captioned multi-view videos, it remains costly in terms of training resources.Instead, we explore a new question: can we jointly combine and reuse the layers of pre-trained 2D video and multi-view image diffusion models to establish a T2MVid diffusion model? The large-scale pre-trained multi-view image diffusion models (e.g., MVdream [80] ) have learned how to model multi-view images, and the 2D temporal layers of powerful pre-trained video diffusion models (e.g., AnimateDiff [23] ) learned rich motion knowledge. However, new challenges are posed. We observe that naively combining the layers from these two kinds of diffusion models leads to poor generation results. More specifically, the training data of multi-view image diffusion models are mainly rendered from synthetic 3D objects (e.g., Objaverse [17, 16] ), while 2D video diffusion models are mainly trained on real-world 2D videos, posing a large domain gap issue.To bridge this gap, we propose a novel diffusion-based pipeline, namely, Vivid-ZOO, for T2MVid generation. The proposed pipeline effectively connects the pre-trained multi-view image diffusion model [80] and 2D temporal layers2 of the pre-trained video model by introducing two kinds of layers, named 3D-2D alignment layers and 2D-3D alignment layers, respectively. The 3D-2D alignment layers are designed to align features to the latent space of the pre-trained 2D temporal layers, and the introduced 2D-3D alignment layers project the features back. Furthermore, we construct a comparable small dataset consisting of 14,271 captioned multi-view videos to facilitate this and future research line. Although our dataset is much smaller compared to the billion-scale 2D image dataset (LAION [78] ) and the million-scale 2D video dataset (e.g., WebVid10M [4] ), our pipeline allows us to effectively train a large-scale T2MVid diffusion model using such limited data. Extensive experimental results demonstrate that our method effectively generates high-quality multi-view videos given various text prompts.We summarize our contributions as follows:• We present a novel diffusion-based pipeline that generates high-quality multi-view videos from text prompts. This is the first study on T2MVid diffusion models.• We show how to combine and reuse the layers of the pre-trained 2D video and multi-view image diffusion models for a T2MVid diffusion model. The introduced 3D-2D alignment and 2D-3D alignment are simple yet effective, enabling our method to utilize layers from the two diffusion models across different domains, ensuring both temporal coherence and multi-view consistency.• We contribute a multi-view video dataset that provides multi-view videos, text descriptions, and corresponding camera poses, which helps to advance the field of T2MVid generation. 
related work: 
 2D video diffusion model. Many previous approaches have explored autoregressive transformers (e.g., [18, 29, 107] ), physical models [104] or GANs (e.g., [8, 56, 77, 41] ) for video generation.Recently, more and more efforts have been devoted to diffusion-based video generation [21, 26, 65, 94, 99, 102, 110, 121, 37] , inspired by the impressive results of image diffusion models [11, 12, 74, 76, 83, 115] .The amount of available captioned 2D video data is significantly less than the vast number of 2D image-text pairs available on the Internet. Most methods [7, 22, 23, 90, 91, 92] extend pre-trained 2D image diffusion models to video generation to address the challenge of limited training data. Some methods employ pre-trained 2D image diffusion models (e.g., [76] ) to generate 2D video from texts in a zero-shot manner [36][118] or using few-shot tuning strategies [103] . These methods avoid the requirement of large-scale training data. Differently, another research line is to augment pretrained 2D image diffusion models with various temporal modules or trainable parameters, showing impressive temporal coherence performance. For example, Ho et al. [28] extend the standard image diffusion architecture by inserting a temporal attention block. Animatediff [23] and AYL [7] freeze 2D image diffusion model and solely train additional motion modules on large-scale datasets of captioned 2D videos such as WebVid10M [4] . In addition, image-to-2D-video generation methods [6, 71, 105, 117] are proposed based on diffusion models to generate a monocular video from an image. Methods [95] focus on controllable video generation through different conditions such as pose and depth. MotionCtrl [98] and Direct-a-video [109] can generate videos conditioned by the camera and object motion. CameraCtrl [24] can also control the trajectory of a moving camera for generated videos. However, these text-to-2D-video diffusion models are designed for monocular video generation, which does not explicitly consider the spatial 3D consistency of multi-view videos.Multi-view image diffusion model. Recent works have extended 2D image diffusion models for multi-view image generation. Zero123 [51] and Zero123++ [79] propose to fine-tune an imageconditioned diffusion model so as to generate a novel view from a single image. Inspired by this, many novel view synthesis methods [19, 32, 52, 53, 59, 93, 97, 100, 108, 112, 120] are proposed based on image diffusion models. For example, IM3D [59] and Free3D [120] generate multiple novel views simultaneously to improve spatial 3D consistency among different views. Differently, a few methods [13, 33, 89] adapt pre-trained video diffusion models (e.g., [6] ) to generate multi-view images from a single image. MVDream [80] presents a text-to-multi-view-image diffusion model to generate four views of an object each time given a text, while SPAD [34] generates geometrically consistent images for more views. Richdreamer [67] trains a diffusion model to generate depth, normal, and albedo.4D generation using diffusion models. Many approaches [47, 51, 64, 69, 79, 80, 85, 96, 2] have exploited pre-trained diffusion models to train 3D representations for 3D object generation via score distillation sampling [64] . Recently, a few methods [3, 49, 70, 82, 113] leverage pre-trained diffusion models to train 4D representations for dynamic object generation. For example, Ling et al. [49] represent a 4D object as Gaussian spatting [35] , while Bahmani et al. [3] adopt a NeRF-based representation [60, 61, 86] . Then, pre-trained 2D image, 2D video, and multi-view image diffusion models are employed to jointly train the 4D representations. In addition, diffusion models are used to generate 4D objects from monocular videos [14, 31] . Diffusion4D [46] presents a diffusion model that generates an orbital video around 4D content, and 4Diffusion [114] presents a video-conditioned diffusion model that generates MV videos from a monocular video. Different from all these methods, our approach focuses on presenting a T2MVid diffusion model. LMM [116] generates 3D motion for given 3D human models. DragAPart [43] can generate part-level motion for articulated objects. Unlike our method, Kuang et al. [38] focuses on generating multiple videos of the same scene given multiple camera trajectories.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We describe the limitations of our work in Limitations (Sec. A).Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
multi-view video diffusion model: 
 Problem definition. Our goal for T2MVid generation is to generate a set of multi-view videos centered around a dynamic object from a text prompt. Motivated by the success of diffusion models in 2D video/image generation, we aim to design a T2MVid diffusion model. However, T2MVid generation is challenging due to the complexity of modeling multi-view videos and the difficulty of collecting massive captioned multi-view videos for training.We address the above challenges by exploring two questions. (1) Can we design a diffusion model that effectively learns T2MVid generation, yet only needs a comparable small dataset of multi-view video data? (2) Can we jointly leverage, combine, and reuse the layers of pre-trained 2D video and multi-view image diffusion models to establish a T2MVid diffusion model? Addressing these questions can reduce the reliance on large-scale training data and decrease training costs. However, this question remains unexplored for diffusion-based T2MVid.Overview. We address the above questions by factoring the T2MVid generation problem over viewpoint-space and time. With the factorization, we propose a diffusion-based pipeline for T2MVid generation (see Fig 2) , including the multi-view spatial modules and multi-view temporal modules. Sec 3.1 describes how we adapt a pre-trained multi-view image diffusion model as the multi-view spatial modules. Multi-view temporal modules effectively leverage temporal layers of the pre-trained 2D video diffusion model with the newly introduced 3D-2D alignment layers and 2D-3D alignment layers (Sec 3.2). Finally, we describe training objectives in Sec 3.3 and the dataset construction to support our pipeline for T2MVid generation in Sec 3.4.Our multi-view spatial modules ensure that the generated multi-view videos are geometrically consistent and aligned with the input text. Recent multi-view image diffusion models [34, 80] generate high-quality multi-view images by fine-tuning Stable Diffusion and modifying its selfattention layers. We adopt the architecture of Stable Diffusion for our multi-view spatial modules. Furthermore, we leverage a pre-trained multi-view image diffusion model based on Stable Diffusion by reusing its pre-trained weights in our spatial modules, which avoids training from scratch and reduces the training cost. However, the self-attention layers of Stable Diffusion are not designed for multi-view videos. We adapt these layers for multi-view self-attention as below. Multi-view self-attention. We inflate self-attention layers to capture geometric consistency among generated multi-view videos. Let F ∈ R b×K×N ×d×h×w denote the 6D feature tensor of multi-view videos in the diffusion model, where b, K, N , d and h × w are batch size, view number, frame number, feature channel and spatial dimension, respectively. Inspired by [34, 80] , we reshape F into a shape of (b × N ) × d × (K × h × w), leading to a batch of feature maps Fn of 2D images, where (b × N ) is the batch size, Fn denotes a feature map representing all views at frame index n, and (K × h × w) is the spatial size. We then feed the reshaped feature maps Fn into self-attention layers.Since Fn consists of all views at frame index n, the self-attention layers learn to capture geometrical consistency among different views. We also inflate other layers of stable diffusion (see Appendix E) so that we can reuse their pre-trained weight.Camera pose embedding. Our diffusion model is controllable by camera poses, achieved by incorporating a camera pose sequence as input. These poses are embedded by MLP layers and then added to the timestep embedding, following MVdream [80] . Here, our multi-view spatial module reuses the pre-trained multi-view image diffusion model MVDream [80] .Besides spatial 3D consistency, it is crucial for T2MVid diffusion models to maintain the temporal coherence of generated multi-view videos simultaneously. Improper temporal constraints would break the synchronization among different views and introduce geometric inconsistency. Moreover, training a complex temporal module from scratch typically requires a large amount of training data.Instead, we propose to leverage the 2D temporal layers of large pre-trained 2D video diffusion models (e.g., [23] ) to ensure temporal coherence for T2MVid generation. These 2D temporal layers have learned rich motion priors, as they have been trained on millions of 2D videos (e.g., [4] ). Here, we employ the 2D temporal layers of AnimateDiff [23] due to its impressive performance in generating temporal coherent 2D videos.However, we observed that naively combining the pre-trained 2D temporal layers with the multi-view spatial module leads to poor results. The incompatibility is due to the fact that the pre-trained 2D temporal layers and the multi-view spatial modules are trained on data from different domains (i.e., real 2D and synthetic multi-view data) that have a large domain gap. To address the domain gap issue, one approach is to fine-tune all 2D temporal layers of a pre-trained 2D video diffusion model on multi-view video data. However, such an approach not only needs to train many parameters but can also harm the learned motion knowledge [30] if a small training dataset is given. We present a multi-view temporal module (see Fig. 3 ) that reuses and freezes all 2D temporal layers to maintain the learned motion knowledge and introduce the 3D-2D alignment layer and the 2D-3D alignment layer.3D-2D alignment. We introduce the 3D-2D alignment layers to effectively combine the pre-trained 2D temporal layers with the multi-view spatial module. Recently, a few methods [23, 6] add motion LoRA to 2D temporal attention for personalized/customized video generation tasks. However, our aim is different, i.e., we expect to preserve the learned motion knowledge of 2D temporal layers, such that our multi-view temporal module can leverage the knowledge for ensuring temporal coherence.Since motion prior knowledge is captured by the pre-trained 2D temporal attention layers, we insert the 3D-2D alignment layers before the 2D temporal attention layers. The 3D-2D alignment layers are learned to align the features into the latent space of the pre-trained 2D temporal layers. Furthermore, inspired by ControlNet [115] and [25] , the 3D-2D alignment layers are inserted via residual connections and are zero-initialized, providing an identity mapping at the beginning of training. The process is described as follows:EQUATIONwhere α 3D⇀2D is the 3D-2D alignment layer. α 2D is the 2D temporal layer followed by the 2D temporal attention layers and we refer to it as 2D in-layer (see more details in Appendix). The 3D-2D alignment layer is plug-and-play and is simply implemented as an MLP. Multi-view temporal coherence. We reuse and freeze the pre-trained 2D temporal layers in our multi-view temporal module to ensure the temporal coherence of each generated video. However, the 2D temporal layer is designed to handle 2D videos. We inflate the 2D temporal layer by reshaping the feature F to the 2D video dimension via the rearrange operation [73] . Then, 2D temporal layers γ(•) model temporal coherence across frames by calculating the attention of points at the same spatial location in F across frames for each video:F = rearrange(F, b K N h w d → (b K h w) N d) (2) F = γ(F) (3) F = rearrange(F, (b K h w) N d → b K N h w d) (4)2D-3D alignment. We add the 2D-3D alignment layers after 2D temporal attention layers to project the feature back to the feature space of the multi-view spatial modules.EQUATIONwhere β 3D⇀2D is the 2D-3D alignment layer. β 2D is the 2D temporal layer following the 2D temporal attention layer. The 2D-3D alignment layers are implemented as an MLP.We train our diffusion model to generate multi-view videos. Note that we freeze most layers/modules in the diffusion model and only train the 3D-2D and 2D-3D alignment layers during training, which largely reduces the training cost and reliance on large-scale data. Let X denote the training dataset, where a training sample {x, y, c} consists of N multi-view videos x = {x} N 1 , N corresponding camera poses c, and a text prompt y. The training objective L on X is defined as follows:EQUATIONwhere τ θ (•) is a text encoder that encodes the text into text embedding, ϵ θ (•) is the denoising network. z v 0 is the latent code of a multi-view video sequence and z v t is its noisy code with added noise ϵ. Different from 2D images that are available in vast numbers on the Internet, it is much more difficult and expensive to collect a large amount of multi-view videos centered around 3D objects and corresponding text captions. Recently, multi-view image datasets (e.g., [51, 67] ), rendered from synthetic 3D models, have shown a significant impact on various tasks such as novel view synthesis [51, 93] , 3D generation (Gaussian Splatting [84] , large reconstruction model [50] ), multi-view image generation [80] and associated applications. Motivated by this, we resort to rendering multi-view videos from synthetic 4D models (animated 3D models).We construct a dataset named MV-VideoNet that provides 14,271 triples of a multi-view video sequence, its associated camera pose sequence, and a text description. In particular, we first select animated objects from Objaverse [17] .Objaverse is an open-source dataset that provides high-quality 3D objects and animated ones (i.e., 4D object). We select 4D objects from the Objaverse dataset and discard those without motions or with imperceptible motions. Given each selected 4D object, we render 24-view videos from it, where the azimuth angles of camera poses are uniformly distributed.To improve the quality of our dataset, we manually filter multi-view videos with low-quality e.g., distorted shapes or motions, very slow or rapid movement. For text descriptions, we adopt the captioning method Cap3D [57, 58] to caption a multi-view video sequence. Cap3D leverages BLIP2 [42] and GPT4 [63] to fuse information from multi-view images, generating text descriptions.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiments: 
 Implementation details. We reuse the pre-trained MVDream V1.5 in our multi-view spatial module and reuse the pre-trained 2D temporal layers of AnimateDiff V2.0 in our multi-view temporal module. We train our model using AdamW [54] with a learning rate of 10 Evaluation metrics. Quantitatively evaluating multi-view consistency and temporal coherence remains an open problem for T2MVid generation. We quantitatively evaluate text alignment via CLIP [68] and temporal coherence via Frechet Video Distance (FVD) [87] . Yet, Ge et al. [20] pointed out FVD leans more towards per-frame quality than temporal consistency. To compensate for FVD, Text prompt: a dog wearing an outfit, 3d asset To the best of our knowledge, no studies have explored T2MVid diffusion models before. We establish a baseline method named MVDream + IP-AnimateDiff for comparison. MVDream + IP-AnimateDiff combines the pre-trained multi-view image diffusion model MVDream [80] and the 2D video diffusion model AnimateDiff [23] , since MVDream generates high-quality multi-view images and AnimateDiff generates temporal coherent 2D videos. Following [119] , we combine AnimateDiff with IP-adaptor [111] to enable AnimateDiff to take an image as input.Given a text prompt, MVDream + IP-AnimateDiff generates multi-view videos in two stages, where MVDream generates multi-view images in the first stage, and IP-AnimateDiff animates each generated image from view into a 2D video in the second stage.Fig. 4 and Tab. 1 show that MVDream + IP-AnimateDiff achieves slightly better CLIP values. However, our method outperforms MVDream + IP-AnimateDiff by a large margin in FVD and overall performance. MVDream + IP-AnimateDiff introduces the noticeable 3D inconsistency among different views. For example, both appearances and motions of the butterfly in the view 0 video are inconsistent with those of view 3. In contrast, our method not only achieves better performance in maintaining multi-view consistency, but also generates larger and more vivid motions for the butterfly, thanks to our pipeline and dataset. In addition, different from MVDream + IP-AnimateDiff employing two kinds of diffusion models and generating results in two stages, our method provides a unified diffusion model generating high-quality multi-view videos in only one stage. Please refer to the Appendix for more results.We conduct the ablation study to show the effectiveness of the design in our multi-view spatial and temporal modules, as well as the proposed 3D-2D and 2D-3D alignment. Text prompt: a sea turtle, 3d asset. Since single-view video generation is much simpler than multi-view video generation, w/o MS w SD achieves high performance in video quality. However, w/o MS w SD fails to maintain multi-view consistency among different views (see Fig. 5 ) and has degraded overall generation performance (see Tab. 2). For example, the motion and shapes of the dragon are significantly inconsistent among views. Instead, by simply adapting a pre-trained multi-view image diffusion model as our spatial module, our method effectively ensures multi-view consistency. despite being fine-tuned on our dataset. Instead, our multi-view temporal module inserts 3D-2D alignment and 2D-3D alignment layers before and after the 2D temporal attention layers, enabling the multi-view temporal module to be compatible with the multi-view spatial module.Effect of 3D-2D alignment. We remove the proposed 3D-2D alignment from our model and train the model on our dataset with the same settings. Tab. 2 shows w/o 3D-2D alignment degrades our temporal coherence and video quality performance. Instead, by projecting the feature to the latent space of the pre-trained 2D attention layers, our 3D-2D alignment layer effectively enables the 2D attention layers to align temporally correlated content, ensuring the video quality and temporal coherence.Effect of 2D-3D alignment. As shown in Tab. 2, "w/o 2D-3D temporal alignment" achieves lower performance with the same training settings due to the removal of 2D-3D temporal alignment. The results indicate that only 3D-2D alignment is insufficient in jointly leveraging the pre-trained 2D temporal layers [23] and the multi-view image diffusion model [80] in our diffusion model. Instead, our 2D-3D alignment projects the features processed by the pre-trained 2D temporal layers back to the latent space of the multi-view image diffusion model, leading to high-quality results.Training cost. MVDream is trained on 32 Nvidia Tesla A100 GPUs, which takes 3 days, and AnimateDiff takes around 5 days on 8 A100 GPUs. By combining and reusing the layers of MVDream and AnimateDiff, our method only needs to train the proposed 3D-2D alignment and 2D-3D layers, reducing the training cost to around 2 days with 8 A100 GPUs. 
conclusions: 
 In this paper, we propose a novel diffusion-based pipeline named Vivid-ZOO that generates highquality multi-view videos centered around a dynamic 3D object from text. The presented multi-view spatial module ensures the multi-view consistency of generated multi-view videos, while the multiview temporal module effectively enforces temporal coherence. By introducing the proposed 3D-2D temporal alignment and 2D-3D temporal alignment layers, our pipeline effectively leverages the layers of the pre-trained multi-view image and 2D video diffusion models, reducing the training cost and accelerating the training of our diffusion model. We also construct a dataset of captioned multi-view videos, which facilitates future research in this emerging area. models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have discussed in Sec. G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We have discussed the safeguards in Sec. G.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All assets used in the paper are cited and credited. The license and terms of use are mentioned and respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this appendix, we provide additional content to complement the main manuscript:• Appendix A: Discussions about limitations of the current method and possible future improvements.• Appendix B: Preliminaries about diffusion and latent diffusion models.• Appendix C: Evaluation metrics for our multi-view video generation.• Appendix D: Details about the multi-view captioned video dataset we construct.• Appendix E: More details of our proposed model, including the spatial and temporal module.• Appendix F: Additional qualitative visualized results.• Appendix G: Societal impact, ethic concerns, dataset copyrights, and our safeguard policies.While our method takes a step forward in T2MVid generation, our method can be improved in a few aspects.For example, the visual quality of generated videos is not as high as that of multi-view image diffusion models due to the complexity of modeling multi-view videos. The spatial module of our method can be replaced with more advanced multi-view image diffusion models [34] , to improve the performance of multi-view consistency. A large dataset of multi-view videos can be constructed, which further improves our method.For lighting and rendering, we followed the settings of [80] to ensure fair comparisons. Since they used point light sources, our learned model also generates multi-view videos under the assumption of point light sources, which may result in different exposures across the viewpoints in the videos, as shown in Fig. III . Future work could explore generating videos that simulate more complex ambient lighting settings and even achieve controllable lighting for different viewpoints in the generation process.Currently, the proposed Vivid-ZOO mainly supports generating multi-view videos for dynamic creatures with natural motions. Though we can also generate some categories like humans (astronaut and horse, Fig. II ) and common objects (waving flag, Fig. IV ), we believe our research can further inspire the community to develop more T2MVid techniques for the generation of more diverse and complex topics, similar to image diffusion model counterparts. For example, more specialized models that generate man-made artifacts like moving cars or articulated furniture, more powerful models that generate multiple dynamic objects with complex motions, and more diverse models like Vivid-Scene that generate dynamic scenes like a stormy sea, an erupting volcano.Diffusion model. Diffusion models [27] learn to model a data distribution by iteratively recovering original data from noisy one, which comprises forward and backward phases. Given a clean sample x 0 from the data distribution p data , the forward process gradually adds Gaussian noise to the sample, generating random latent variables x t at each time step t ∈ [0, T ]:EQUATIONwhere β t is a hyperparameter that determines the noise schedule. With a large time step, x T is assumed to be perturbed into a standard Gaussian noise. Given x T , the denoising network is trained to gradually remove the noise and recover the original data:EQUATIONwhere θ denote parameters of the denoising network, µ and Σ are mean and variance, respectively.. By embedding images into low-dimensional latent codes, latent diffusion models [74] (LDMs) perform the diffusion process in the latent space of latent codes, significantly reducing the computational cost. Typically, LDMs employ a pre-trained autoencoder (e.g., VQ-VAE [88] ), which consists of an encoder and decoder, where the encoder transforms images into the latent space and the decoder maps denoised latent codes back to the pixel space.• Multi-view Text alignment: Text-to-2D-video diffusion models (e.g., [23, 36, 102] ) adopt CLIP score [68] to measure the alignment between an input text and a corresponding generated 2D video.To evaluate the alignment between the input text and generated multi-view videos, we first measure the CLIP score for each view and then average the CLIP scores for all views. • Video quality: We adopt Frechet Video Distance (FVD) to measure the quality of generated multiview videos. FVD is the standard metric adopted by many 2D video generation [95, 106] and animation methods [6, 105] . FVD measures the quality of generated videos by measuring the data distribution between generated and training videos, where I3D networks pre-trained on the Kinetics dataset [10] are employed to extract features. • Human preference. We conduct a user study to measure the overall quality of generated multi-view video text alignment, temporal coherence and multi-view consistency. We adopt paired comparison. We invite 10 subjects to participate in the user study. For each subject, we display two multi-view video sequences generated by different generation methods as well as the corresponding input text prompt, where the two results are arranged in an up-and-down order, and a resulting multi-view video sequence is displayed in a single row. We told each subject that the task is to generate four orthogonal views of a dynamic 3D object. Then the subject was asked to choose a multi-view video sequence whose overall quality is better in text alignment, temporal coherence and multi-view consistency. For comparison with MVdream + IP-AnimateDiff, we use 10 input prompts to generate multi-view videos for the user study. For the ablation study, there are five methods in total, leading to more combinations of paired comparisons. We hence use 5 input prompts in the use study for the ablation study.2D video diffusion models [6] have pointed out that data curation is essential to improve the generation performance of diffusion models. The training of 2D Video diffusion models can be degraded if the training dataset contains many static 2D videos [6] . Hence, we developed an animated 3D object selection tool that automatically discards 4D objects that are static or close to static. In particular, given a 4D object, we first render a video from a single viewpoint. For efficiency, instead of using advanced optical flow algorithms, we calculate the pixel difference between different frames in each video, in order to identify whether 4D objects are static.With the selected 4D objects, we employ Cycles3 as the rendering engine to render multi-view videos. Given a 4D object, we render 24-view videos with the resolution of 512 × 512. We first uniformly distribute the camera poses around the normalized 4D object and then add subtle disturbances. The radius of a camera to a 4D object is in the range of [2.2 2.6], and a camera's height is in [0.8 1.2].The background of a multi-view video sequence is randomly filled in gray color. The frame number of multi-view video sequences is diverse, depending on its 4D objects.To further improve the quality of our dataset, we manually discard low-quality data from our dataset. We found that many multi-view videos contain distorted shapes or motions. We remove these lowquality data to avoid their negative effect on the training generation models. On the other hand, we also remove multi-view videos that contain large translation motions. Due to the large translation motions, objects disappear in a few frames, which leads to these frames having only a background. In addition, many 4D objects are textureless in Objaverse [17] . We keep 10% of these textureless objects. For caption generation, we adopt a caption method i.e., Cap3D which is designed to caption multi-view images of a 3D object. The Cap3D is used to describe a multi-view frame sequence sampled from a multi-view video sequence.Training settings. Table II provides detailed information on the hyperparameter settings and hardware configuration used for model training. During training, four orthogonal views are randomly chosen, leading to four-view videos. For a video from a viewpoint, the starting frame is randomly selected, and then we extract one frame every 3 frames. The frame size is 256 × 256 and the frame number is set to 16 (see Table I ). Both a 3D-2D Alignment layer and 2D-3D Alignment layer are implemented using a Linear MLP layer. We experimented with 2-layer/3-layer setups, but there was no improvement in performance. Therefore, we use a simple single layer for implementation. In this paper, we reuse AnimateDiff in the multi-view temporal module, where the 2D-in-layer refers to the "project_in" layer and 2D-out-layer refers to "_out" layer in the AnimateDiff. Justification: The main claims accurately reflect the paper's contributions.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.For clarification, we add "2D" when referring to the layers of the 2D video diffusion models, while we add "3D" when referring to the multi-view image diffusion model.https://www.cycles-renderer.org/ 


Decision: The paper received two Accepts and two Borderline Accepts. The reviewers appreciated the new task, the proposed dataset, and some of the results and ideas. However, there are still concerns about the comparisons with existing about generalizations / performance at scale and comparisons. I agree with all these points mentioned by the reviewers and recommend Accept as poster.

Reviews: review: paper_summary: paper_summary: The paper presents a novel diffusion-based pipeline designed to generate multi-view videos from text prompts. The core challenge addressed by the paper is the generation of dynamic 3D object videos from multiple viewpoints, a task not extensively explored with existing diffusion models. The authors factor the problem into viewpoint-space and time components, allowing for the reuse of layers from pre-trained multi-view image and 2D video diffusion models. This reuse is intended to ensure consistency across views and temporal coherence, thereby reducing training costs. The paper also introduces 3D-2D and 2D-3D alignment layers to bridge the domain gap between the training data of multi-view images and 2D videos. Additionally, a new dataset of captioned multi-view videos was created.
strength_weakness: strength_weakness: 1. The paper acknowledges the challenge of not having a large-scale, captioned multi-view video dataset, which is crucial for training robust diffusion models. The authors attempt to mitigate this by constructing a smaller dataset, but this could limit the model’s generalization capabilities and performance across diverse scenarios. Further, this may be very challenging if we would like to further scale to scene-level contents.
2. The alignment between pre-trained 2D video and multi-view image diffusion models is crucial, yet challenging. The proposed 3D-2D and 2D-3D alignment layers are a solution, but the effectiveness of these layers in truly bridging the domain gap might not fully compensate for the inherent differences in training data types (real-world 2D videos versus synthetic multi-view data), potentially affecting the quality of generated videos. Since the model is trained on an entirely different domain of object-centric videos, there is very little we can tell in terms of video quality degradation.
questions: questions: N/A
limitations: limitations: Yes
paper_summary: paper_summary: This paper focuses on generating multi-view consistent videos given a text prompt, specifically from 4 orthogonal views. It fine-tunes MVDream with a temporal layer adopted from AnimateDiff. To mitigate the domain gap between the two layers, some connected layers called 3D-2D and 2D-3D alignment layers are introduced. They train the newly introduced layers on a 4D subset of Objaverse. The proposed model can generate multi-view videos. The authors conduct evaluations to verify the new layers and choices of the base models.
strength_weakness: strength_weakness: Despite the fact that the paper is the first to study text-conditioned multi-view video generation, I think there could still be room for improvement in the evaluation section in terms of the justification of the method as stated below:

1. There are very few generated videos (8 in the supplementary material) shown in the paper. Is this because the success rate is low? Since it is a forward-based method, I expect the time to generate the videos will not be too long. If so, it would be good to discuss the failure cases in the limitation section.

2. As for human evaluation, only five videos are used for comparison.

3. There is a lack of details about how many videos are used to compute CLIP and FVD scores. Also, are the prompts used to generate these videos separate from the training set? FVD was used to evaluate temporal coherence, but FVD is shown to be not good at it (see [2]).

    [2] Ge, Songwei, et al. "On the Content Bias in Fréchet Video Distance." CVPR. 2024.

4. The motions in most generated videos are small and are not indicated even in the text prompt. This could limit the use case of the model in generating specific motions.

5. All the generated videos contain a grayish background. I'm fine with it if the ultimate goal is to generate a 4D asset, and I think it is a reasonable goal. Ideally, like MVDream, the authors may want to compare with 4D generation methods.
questions: questions: 1. As far as I know, MVDream is fine-tuned (fully?) from SD2.1, while AnimateDiff v2 is fine-tuned from SD1.5; this could cause an additional discrepancy when combining the two models, right?
2. In the ablation study of "Design of multi-view spatial module", what is the difference between what you do and directly using AnimateDiff v2?
limitations: limitations: The authors discuss the limitations and negative societal impact in the appendix.
paper_summary: paper_summary: This paper proposes a Text-to-Multi-view-Video generation algorithm capable of generating multi-view video content based on text descriptions. The method decouples the multi-view spatial and temporal dimensions, utilizing pretrained models like MVDream for multi-view generation and animatediff for video generation. It addresses domain gaps between different features through 3D-2D/2D-3D alignment. The authors trained on captioned 4D data from objverse and conducted various ablation studies to validate their approach.
strength_weakness: strength_weakness: 1The authors claim their method is the first to do text-to-multi-view video generation, but as far as I know, there are many works related to 4D generation currently, such as MAV3D, AYG, 4D-fy, including recent works like 4Diffusion and Diffusion4D. I believe the authors may want to emphasize that their work focuses on novel view synthesis rather than involving 4D representation, such as Deformable GS, Dynamic Nerf, etc. However, these are just differences at the algorithmic level; fundamentally, they are all engaged in multi-view video generation. Therefore, the authors' comparison in Table 1 is insufficient, and relevant 4D generation works should also be included for comparison.
questions: questions: 1. Regarding the 'overall' metric in Table 1, does it refer to the results of a user study, or is it a comprehensive representation of the previous two results?
2. What is the test dataset in Table 1? How large is the dataset?
3. Will the 4D dataset proposed in the paper be made publicly available?
4. In the current framework, what are the results if MVDream and AnimateDiff are not fixed, or if fine-tuning is done with a very small learning rate?
5. Did the authors conduct experiments on generating 4D contents based on image prompt, and are there any visual results?
limitations: limitations: Currently, the results only compare against a self-designed baseline, lacking a broader comparison with other algorithms.
paper_summary: paper_summary: The paper introduces "Vivid-ZOO," an innovative diffusion model designed for Text-to-Multi-view-Video (T2MVid) generation. This is a novel approach that addresses the challenges of generating high-quality multi-view videos from textual descriptions. Specifically, the authors propose a new method that leverages diffusion models to generate multi-view videos centered around dynamic 3D objects from text prompts. The T2MVid generation problem is effectively factorized into viewpoint-space and time components, allowing for the combination and reuse of layers from advanced pre-trained multi-view image and 2D video diffusion models. And the authors claim that it is the first study that explores the application of diffusion models to the T2MVid generation task.
strength_weakness: strength_weakness: 1. The visual quality of the generated videos might not match state-of-the-art single-view image generation models. Future work could focus on improving the visual fidelity and realism of the generated multi-view videos.
2. The model currently assumes point light sources, which might not always produce the most realistic renderings. Enhancements in simulating complex lighting conditions could improve the model's applicability.
questions: questions: Please refer to weakness
limitations: limitations: The authors discuss the limitations and potential negative societal impacts of their work in the paper.

Comments: title: Please discuss
Please discuss
Response
comment: Dear Reviewer VRVx,

We sincerely thank the reviewer for your positive feedback.  Your insightful comments help us further improve the paper's quality.

Best,

The Authors.
Dear Reviewer MoTs,

We sincerely thank the reviewer for your positive feedback. Your constructive review comments help us enhance the strength of our work.

Best regards,

The Authors.
Dear Reviewer CcuB,

We sincerely thank the reviewer for your positive feedback. Your constructive suggestions help us improve the paper's quality. 

Best,

The Authors.
Dear Reviewer K81K,

We sincerely thank the reviewer for your positive feedback. Your constructive review comments help us improve the quality of the manuscript.

Best regards,

The Authors.
Dear reviewer,

The discussion period is coming to a close soon. Please do your best to engage with the authors.

Thank you,
Your AC
Dear reviewer,

The discussion period is coming to a close soon. Please do your best to engage with the authors.

Thank you,
Your AC
Thank you for the thorough response. The rebuttal has addressed most of my concerns. I'd like to keep my initial score
Thank you for your time and effort in addressing the unclear details, adding additional experiments, and performing additional evaluations. I have got most of my concerns addressed and updated my rating.