Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control

Authors: zhengfei  kuang
shengqu  cai
hao  he
yinghao  xu
hongsheng  li
leonidas  guibas
gordon  wetzstein
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 23:31:59

Abstract: Research on video generation has recently made tremendous progress, enabling high-quality videos to be generated from text prompts or images. Adding control to the video generation process is an important goal moving forward and recent approaches that condition video generation models on camera trajectories take an important step towards this goal. Yet, it remains challenging to generate a video of the same scene from multiple different camera trajectories. Solutions to this multi-video generation problem could enable large-scale 3D scene generation with editable camera trajectories, among other applications. We introduce collaborative video diffusion (CVD) as an important step towards this vision. The CVD framework includes a novel cross-video synchronization module that promotes consistency between corresponding frames of the same video rendered from different camera poses using an epipolar attention mechanism. Trained on top of a state-of-the-art camera-control module for video generation, CVD generates multiple videos rendered from different camera trajectories with significantly better consistency than baselines, as shown in extensive experiments.

Main paper: 
 introduction: 
 With the impressive progress of diffusion models [22, 51, 28, 44, 46, 43] , video generation has significantly advanced [12, 17, 26, 5, 6, 15, 64, 23, 32] , with a transformative impact on digital content creation workflows. Recent models like SORA [7] exhibit the ability to generate long highquality videos with complex dynamics. Yet, these methods typically leverage text or image inputs to control the generation process and lack precise control over content and motion, which is essential for practical applications. Prior efforts explore the use of other input modalities, such as flow, keypoints, and depths, and develop novel control modules to incorporate these conditions effectively, enabling precise guidance of the generated contents [60, 63, 67, 26, 56, 9] . Despite these advancements, these methods still fail to provide camera control to the video generation process.Recent works have started to focus on camera control using various techniques, such as motion LoRAs [25, 17] or scene flows [63, 67] . Some representative works such as MotionCtrl [60] and CameraCtrl [18] offer more flexible camera control by conditioning the video generative models on a sequence of camera poses, showing the feasibility of freely controlling the camera movements of videos. However, these methods are limited to single-camera trajectories, leading to significant inconsistencies in content and dynamics when generating multiple videos of the same scene from different camera trajectories. Consistent multi-video generation with camera control is desirable in many downstream applications, such as large-scale 3D scene generation. Training video generation models for consistent videos with different camera trajectories, however, is very challenging, partly due to the lack of large-scale multi-view dynamic in-the-wild scene data.In this paper, we introduce CVD, a plug-and-play module capable of generating videos with different camera trajectories sharing the same underlying content and motion of a scene. CVD is designed on a collaborative diffusion process that generates consistent pairs of videos with individually controllable camera trajectories. Consistency between corresponding frames of a video is enabled using epipolar attention, introduced by a learnable cross-view synchronization module. To effectively train this module, we propose a new pseudo-epipolar line sampling scheme to enrich the epipolar geometry attention. Due to the shortage of large-scale training data for 3D dynamic scenes, we propose a hybrid training scheme where multi-view static data from RealEstate10k [68] and monocular dynamic data from WebVid10M [1] are utilized to learn camera control and motion, respectively. To our knowledge, CVD is the first approach to generate multiple videos with consistent content and dynamics while providing camera control. Through extensive experiments, we demonstrate that CVD ensures strong geometric and semantic consistencies, significantly outperforming relevant baselines. We summarize our contributions as follows:• To our knowledge, our CVD is the first video diffusion model that generates multi-view consistent videos with camera control; • We introduce a novel module called the Cross-Video Synchronization Module, designed to align features across diverse input videos for enhanced consistency; • We propose a new collaborative inference algorithm to extend our video model trained on video pairs to arbitrary numbers of video generation; • Our model demonstrates superior performance in generating multi-view videos with consistent content and motion, surpassing all baseline methods by a significant margin. 
related work: 
 Video Diffusion Models. Recent efforts in training large-scale video diffusion models have enabled high-quality video generation [15, 6, 23, 21, 17, 7, 12, 50] . Video Diffusion Model [23] , utilizes a 3D UNet to learn from images and videos jointly. With the promising image quality obtained by text-to-image (T2I) generation models, such as StableDiffusion [44] , many recent efforts focus on extending pretrained T2I models by learning a temporal module. Align-your-latents [6] proposes to inflate the T2I model with 3D convolutions and factorized space-temporal blocks to learn video dynamics. Similarly, AnimateDiff [17] builds upon StableDiffusion [44] , adding a temporal module after each fixed spatial layer to achieve plug-and-play capabilities that allow users to perform personalized animation without any finetuning. Pyoco [15] proposes a temporally coherent noise strategy to effectively model temporal dynamics. More recently, SORA [7] shows a great step towards photo-realistic long video generation by utilizing space-time diffusion with a transformer architecture.Controllable Video Generation. The ambiguity of textual conditions often results in weak control for text-to-video models (T2V). To provide precise guidance, some approaches utilize additional conditioning signals such as depth, skeleton, and flow to control the generated videos [12, 59, 26, 48, 27, 8, 53] . Recent efforts like SparseCtrl [65] and SVD incorporate images as control signals for video generation. To further control motions and camera views in the output video, DragNUWA [63] and MotionCtrl [60] inject motion and camera trajectories into the conditioning branch, where the former uses a relaxed version of optical flow as stroke-like interactive instruction, and the later directly concatenate camera parameters as additional features. CameraCtrl [18] proposes to overparameterize the camera parameters using Plücker Embeddings [39] and achieves more accurate camera conditioning. Alternatively, AnimateDiff [17] trains camera-trajectory LoRAs [25] to achieve viewpoint movement conditioning, while MotionDirector [67] also utilizes LoRAs [25] but to overfit to specific appearances and motions to gain their decoupling.Multi-View Image Generation. Due to the lack of high-quality scene-level 3D datasets, a line of research focuses on generating coherent multi-view images. Zero123 [35] learns to generate novel-view images from pose conditions, and subsequent works extend it to multi-view diffusion [11, 36, 37, 49, 54, 55, 62, 31] for better view consistency. However, these methods are only restricted to objects and consistently fail to generate high-quality large-scale 3D scenes. Mul-tiDiffusion [3] and DiffCollage [66] facilitates 360-degree scene image generation, while Sce-neScape [14] generates zooming-out views by warping and inpainting using diffusion models. Similarly, Text2Room [24] generates multi-view images of a room, where the images can be projected via depths to get a coherent room mesh. DiffDreamer [9] follows the setups in Infinite-Nature [34, 33] and iteratively performs projection and refinement using a conditional diffusion model. A recent work, PoseGuided-Diffusion [56] , performs novel view synthesis from a single image by training and adding an epipolar line bias to its attention masks on multi-view datasets with camera poses provided (RealEstate10k [68] ). However, this method by construction does not generalize to in-the-wild or dynamic scenes, as its prior is solely learned from well-defined static indoor data.A comprehensive survey of recent advances in diffusion models for visual computing is provided by Po et al. [40] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Sec. 6.1Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
collaborative video generation: 
 Conventionally, video diffusion models (VDMs) aim to generate videos from randomly sampled Gaussian noise with multiple denoising steps, given conditions such as text prompts, frames, or camera poses. Specifically, let v 0 ∼ q data (v) be a data point sampled from the data distribution; the forward diffusion process continuously adds noises to v 0 to get a series of v t , t ∈ 1, ..., T until it becomes Gaussian noise. Using the reparameterization trick from Ho et al. [22] , the distribution of v t can be represented asq(v t | v 0 ) = N (v t ; √ ᾱt v 0 , (1 -ᾱt )I), where ᾱt ∈ (0, 1] are the noise scheduling parameters, which are monotonously increasing, and ᾱT = 1. The video diffusion model, typically denoted as p θ (v t-1 |v t ), is a model parameterized by θ that is trained to estimate the backward distribution q(v t-1 |v t , v 0 ). According to Ho et al. [22] , the optimization of p θ (v t-1 |v t ) results in minimizing the following loss function:EQUATIONwherev t = √ ᾱt v 0 + (1 -ᾱt )ϵis the noisy video feature generated from v 0 and a random sampled Gaussian noise ϵ, ϵ θ (v t , t) is the noise prediction of the VDM, and c is the video condition. During inference time, one can start from a normalized Gaussian noise v T ∼ N (0, I) and apply the noise prediction model ϵ θ (v t , t) multiple times to denoise it until v 0 .Empowered by readily available large-scale video datasets, many state-of-the-art VDMs have successfully shown ability to produce temporally consistent and realistic videos [23, 5, 7, 17, 26, 21, 12, 6, 15] . However, one of the key drawbacks of all these existing methods is the inability to generate consistently coherent multi-view videos. As Fig. 1 shows, videos generated from a VDM under the same textual conditions exhibit content and spatial arrangement disparities. One can use inference-stage tricks, such as extended attention [8] , to increase the semantic similarities between the videos, yet this does not address the problem of structure consistency. To address this issue, we introduce a novel objective for VDMs to generate multiple structurally consistent videos simultaneously given certain semantic conditions and dub it Collaborative Video Diffusion (CVD). In contrast to conventional video diffusion models, CVD seeks to find an arbitrary number of videos v i , i ∈ 1, ..., M that comply with the unknown data distribution q data (v 1,...,M ) given separate conditions c 1,...,M . Similarly, the CVD model can be represented as p θ (v 1,...,M |c 1,...,M ). An example includes multi-view videos synchronously captured from the same dynamic 3D scene. Similarly, the loss function for a collaborative video diffusion model is defined as:EQUATIONIn practice, however, the scarcity of large Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
collaborative video diffusion with camera control: 
 We seek to build a diffusion model that takes a text prompt y and a set of camera trajectories cam 1,...,M and generates the same number of collaborative videos v 1,...,M . To ease the generation of consistent videos, in this work we train our model with video pairs (M = 2), we make the assumption that the videos are synchronized (i.e., corresponding frames are captured simultaneously), and set the first pose of every trajectory to be identical, forcing the first frame of all videos to be the same.Inspired by [18, 17] , our model is designed as an extension of the camera-controlled video model CameraCtrl [18] . As shown in Fig. 2 , our model takes two (or more) noisy video feature inputs and generates the noise prediction in a single pass. The video features pass through the pretrained weights of CameraCtrl and are synchronized in our proposed Cross-View Synchronization Modules (Sec. 4.1). The model is trained with two different datasets: RealEstate10K [68] , which consists of camera-calibrated video on mostly static scenes, and WebVid10M [1] , which contains generic videos without poses. This leads to our two-phase training strategy introduced in Sec. 4.2. The learned model can infer arbitrary numbers of videos using our proposed inference algorithm, which will be described in Sec. 4.3.State-of-the-art VDMs commonly incorporate various types of attention mechanisms defined on the spatial and temporal dimension: works such as AnimateDiff [17] , SVD [5] , LVDM [19] disentangles space and time and applies separate attention layers; the very recent breakthrough SORA [7] processes both dimensions jointly on its 3D spatial-temporal attention modules. Whilst the operations defined on the spatial and temporal dimensions bring a strong correlation between different pixels of different frames, capturing the context between different videos requires a new operation: cross-video attention.Thankfully, prior works [10, 8] have shown that the extended attention technique, i.e., concatenating the key and values from different views together, is evidently efficient for preserving identical semantic information across videos. However, it refrains from preserving the structure consistency among them, leading to totally different scenes in terms of geometry. Thus, inspired by [56] , we introduce the Cross-View Synchronization Module based on the epipolar geometry to shed light on the structure relationship between cross-video frames during the generation process, aligning the videos towards the same geometry.Fig. 2 demonstrates the design of our cross-view module for two videos. Taking a pair of feature sequences z 1 1,..,N , z 2 1,..,N of N frames as input, our module applies a cross-video attention between the same frames from the two videos. Specifically, we define our module as:EQUATIONEQUATION)where k is the frame index, W Q , W K , W V are the query, key, value mapping matrices, M is the attention mask, Attn(Q, K, V, M) is the attention operator introduced from the Transformer [58] , ff is the feed-forward function and F 1→2 k is the fundamental matrix between cam 1 k and cam 2 k . The attention mask M between any two pixels x 1 , x 2 is determined by the epipolar distance betweenx 1 and x 2 , i.e. the shortest distance between x 1 and the epipolar line of x 2 in x 1 's frame, which is set to 1 if the epipolar distance is smaller than a given threshold τ epi (set to 3 in all of our experiments) and vise versa. The outputs of these modules are used as residual connections with corresponding original inputs to ensure no loss of originally learned signals. The key insight of this module is as the two videos are assumed to be synchronized to each other, the same frame from the two videos is supposed to share the same underlying geometry and hence can be correlated by their epipolar geometry defined by the given camera poses. For the first frames where the camera poses are set to be identical since the fundamental matrix is undefined here, we generate pseudo epipolar lines for each pixel with random slopes that go through the pixels themselves. In the scenario where multi-view datasets are available, the modules can be further adapted to more videos by extending the cross-view attention from 1-to-1 to 1-to-many. Our study shows that epipolar-based attention remarkably increases the geometry integrity of the generated video pairs. Considering the fact that there is no available large-scale real-world dataset for video pairs, we opt to make use of the two popular monocular datasets, RealEstate10K [68] and Web-Vid10M [1] , to develop a hybrid training strategy for video pair generation models.RealEstate10K with Video Folding. The first phase of the training involves RealEstate10K [68] , a dataset consisting of video clips capturing mostly static indoor scenes and corresponding camera poses. We sample video pairs by simply sampling subsequences of 2N -1 frames from a video in the dataset, then cutting them from the middle and reversing their first parts to form synchronized video pairs. In other words, the subsequences are folded into two video clips sharing the same starting frame.WebVid10M with Homography Augmentation. While RealEstate10K [68] provides a decent geometry prior, training our model only on this dataset is not ideal since it does not provide any knowledge regarding dynamics and only contains indoor scenes. On the other hand, WebVid10M, a large-scale video dataset, consists of all kinds of videos and can be used as a good supplement to RealEstate10K. To extract video pairs, we clone the videos in the dataset and then apply random homography transformations to the clones. Nonetheless, The WebVid10M dataset contains no camera information, making it unsuitable for camera-conditioned model training. To address this problem, we propose a two-phase training strategy to adapt both datasets (with or without camera poses) for the same model.As previously mentioned, our model is built upon the existing cameracontrolled VDM CameraCtrl [18] . It is an extended version of AnimateDiff [17] that adds a pose encoder and several pose feature injectors for the temporal attention layers to the original model. Both AnimateDiff [17] and CameraCtrl [18] are based on Stable Diffusion [44] . This implies that they incorporate the same latent space domain, and thus, it is feasible to train a module that can be universally adapted. Therefore, our training procedure follows a two-phase scheme Fig. 3 shows. Specifically, we build a hybrid dataset that combines the data from both sources. Then in each training iteration, if the training data is from RealEstate10K, we use CameraCtrl with LoRA fine-tuned on RealEstate10K as the backbone and applying the ground truth epipolar geometry in the cross-video module. Otherwise, we use AnimateDiff with LoRA fine-tuned on WebVid10M as the backbone, and apply the pseudo epipolar geometry (the same strategy used for the first frames in RealEstate10K dataset) in the cross-video module. The two training phases are applied alternatively to the same instance of CVSM in a single training procedure. Experiments show that the hybrid training strategy greatly helps the model generate videos with synchronized motions and great geometry consistency.With the CVD trained on video pairs, during inference, we can generate multiple videos (for example, M videos where M > 2) that share consistent content and motions. To achieve that, we start from M individual gaussian noise maps and denoise them in multiple steps. At each denoising step t, we select P feature pairs P = {v i1,j1 t , v i2,j2 t , ..., v i P ,j P t | i 1,...,P , j 1,...,P ∈ 1, ..., M } among all M video features. We then use the trained network to predict the noise of each feature pair, and averaging them w.r.t. each video feature. That is, the output noise for the ith video feature is defined as:ϵ out (v i t ) = Avg v i,j ∈P (ϵ i θ (v i,j t , t, cam i,j )), where ϵ i θ (v i,j t , t, cam i,j ) is the noise prediction for v i t given the video pair input v i,j t . For pair selection, we propose the following strategies:• Exhaustive Strategy: Select all M (M -1)/2 pairs.• Partitioning Strategy: Randomly divide M noisy video inputs into M 2 pairs. • Multi-Partitioning Strategy: Repeat the Partitioning Strategy multiple times and combine all selected pairs. The exhaustive strategy has a higher computational complexity of O(M 2 ) compared to the partitioning one (O(M )) but covers every pair among M videos and thus can produce more consistent videos. The multi-partitioning strategy, on the other hand, is a trade-off between the two strategies. We also embrace the recurrent denoising method introduced by Bansal et al. [2] that does multiple recurrent iterations on each denoising timestep. We provide the pseudo-code of our inference algorithm and detailed mathematical analysis in our supplementary. 
experiments: 
 5.1 Qualitative ResultsQualitative comparisons are shown in Fig. 4 . Following our quantitative comparisons in Sec. 5.2, we compare against CameraCtrl [18] and its combination with SparseCtrl [16] , MotionCtrl [60] and its combination with SVD [5] . The results indicate our method's superiority in aligning the content within the videos, including dynamic content such as lightning, waves, etc. More qualitative results are provided in our supplemental material and video.We also show the results of arbitrary view generation shown in Fig. 5 . Using the algorithm introduced in Sec. 4.3, our model can generate groups of different camera-conditioned videos that share the same contents, structure, and motion. Please refer to our supplementary video for animated results.We compare our model with two state-of-the-art camera-controlled video diffusion models for quantitative evaluation: CameraCtrl [18] and MotionCtrl [60] . Both of the two baselines are trained on the RealEstate10K [68] for camera-controlled video generation. We conduct the following experiments to test the geometric consistency, semantic consistency, and video fidelity of all models:Table 1 : Quantitative Results on Geometry Consistency. Following SuperGlue [47] , we report the area under the cumulative error curve (AUC) of the predicted camera rotation and translation under certain thresholds (5 • , 10 • , 20 • ), and the precision (P) and matching score (MS) of the Super-Glue correspondences. We feed the models with prompts from RealEstate10K [68] (RE10K) and WebVid10M [1] (WV10M) in two experiments separately. For RealEstate10K scenes, we also run SuperGlue on the original RealEstate10K [68] frames as reference. Our model achieves the highest scores on all metrics compared to baselines. Per-video geometric consistency on estate scenes. Following CameraCtrl [18] , we first test the geometry consistency across the frames in the video generated from our model, using the camera trajectories and text prompts from RealEstate10K [68] (which mostly consists of static scenes). Specifically, we first generate 1000 videos from randomly sampled camera trajectory pairs (two camera trajectories with the same starting transformation) and text captions. All baselines generate one video at a time; our model generates two videos simultaneously. For each generated video, we apply the state-of-the-art image matching algorithm SuperGlue [47] to extract the correspondences between its first frame and following frames and estimate their relative camera poses using the RANSAC [13, 42] algorithm. To evaluate the quality of correspondences and estimated camera poses, we adopt the same protocol from SuperGlue [47] , which 1) evaluates the poses by the angle error of their rotation and translation and 2) evaluates the matched correspondences by their epipolar error (i.e., the distance to the ground truth epipolar line). The results are shown in Tab. 1, where our model significantly outperforms all baselines. More details are provided in our supplementary materials.Scenes Methods Rot. AUC ↑ Trans. AUC ↑ Prec. ↑ M-S. ↑ (@5 • /10 • /20 • ) (@5 • /10 • /20Cross-video geometric consistency on generic scenes. Aside from evaluating the consistency between frames in the same video, we also test our model's ability to preserve the geometry information across different videos. To do that, we randomly sample 500 video pairs (1000 videos in total) using camera trajectory pairs from RealEstate10K [68] and text prompts from WebVid10M's captions [1] .To the best of our knowledge, there is no available large video diffusion model that is designed to generate multi-view consistent videos for generic scenes. Hence, we modify the CameraCtrl [18] and MotionCtrl [60] to generate video pairs as baselines. Here, we use the text-to-video version of each model to generate a reference video first, then take its first frame as the input of their image-to-video version (i.e., their combination with SparseCtrl [16] and SVD [5] ) to derive the second video. We use the same metrics as in the first experiment but instead evaluate between the corresponding frames from the two videos. Results are shown in Tab. 1, where our model greatly outperforms all baselines.Semantic and fidelity evaluations. Following the standard practice of prior works [17, 61, 29, 9, 10, 8] , we report CLIP [41] embedding similarity between 1) each frame of the output video and the corresponding input prompt and 2) pairs of frames across video pairs. The former metric, denoted as CLIP-T, is to show that our model does not destroy the appearance/content prior of our base model, and the latter, denoted as CLIP-F, is aimed to show that the cross-view module can improve the semantic and structural consistency between the generated video pair. For these purposes, we randomly sample 1000 videos using camera trajectory pairs from RealEstate10K, along with text captions from WebVid10M (2000 videos generated in total). To further demonstrate our method's ability to maintain high-fidelity generation contents, we report FID [20] and KID [4] ×100 using the implementation [38] , and FVD [57] . We do not compare against models that do not share the same base model as us for FID [20] , KID [4] and FVD [57] , since these metrics are strongly influenced by the abilities of the base models. Following prior work [18] , we evaluate these two metrics on RealEstate10k [68] because of the strong undesired bias, e.g., watermarks, on WebVid10M [1] . As shown in Tab. 2, our model surpasses all baselines for the CLIP [41] -based metrics. This proves our model's ability to synthesize collaborative videos that share a scene while maintaining and improving fidelity according to the prompt. Our model also performs better than or on par with all prior works on fidelity metrics, which indicates robustness to the appearances and content priors learned by our base models. We perform a thorough ablation study in Tab. 3 to verify our design choices, where the variants are: 1) No epipolar line constraints (Ours w/o Epi), where we perform a normal self-attention instead of epipolar attention in our Cross-View Synchronization Module; 2) No mixed training (Ours RE10K only), where we follow the setups in CameraCtrl [18] and train the model only on RealEstate10k [68] ; 3) No homography augmentation (Ours w/o HG), where we switch off the homography transformations applied to WebVid10M [1] videos during training; and 4) using only 1 Cross-View Synchronization Module instead of 2 (Ours 1 Layer). The ablation study indicates that while we can get semantically consistent outputs without epipolar constraints, they are essential to gain geometrical consistency. We also observe that the mixed training strategy and homography augmentation greatly improve all metrics, including semantic consistency, further verifying their purpose of closing the gap between static training scenes and desired dynamic outputs. We believe there are two reasons why our full model outperforms the model trained on RealEstate10K [68] . The first reason is our epipolar attention design. In the WebVid10M [1] training stage, while there are no camera poses available, we use pseudo-gt epipolar lines (i.e. lines calculated from homography matrix H. The line of pixel x in the warped frame goes through the pixel Hx) to describe the spatial relationship between video frames. This enhances the model's ability to generate videos that satisfy the given line conditions. Hence, in a camera-control setting, the full model is more constrained to the epipolar lines and generates videos that align better with the camera poses. Secondly, since RealEstate10K [68] mostly consists of static indoor scenes, models trained on RealEstate10K [68] may suffer from data bias and may not perform well on general scenes, thus resulting in poor evaluation performance in this experiment. 
discussion: 
 We introduce CVD, a novel framework facilitating collaborative video generation. It ensures seamless information exchange between video instances, synchronizing content and dynamics. Additionally, CVD offers camera customization for comprehensive scene capture with multiple cameras. The core innovation of CVD is its utilization of epipolar geometry, derived from reconstruction pipelines, as a constraint. This geometric framework fine-tunes a pre-trained video diffusion model. The training process is enhanced by integrating dynamic, single-view, in-the-wild videos to maintain a diverse range of motion patterns. During inference, CVD employs a multi-view sampling strategy to facilitate efficient information sharing across videos, resulting in a "collaborative diffusion" effect for unified video output. To our knowledge, CVD represents the first approach to tackle the complexities of multi-view or multi-trajectory video synthesis. It significantly advances beyond existing multi-view image generation technologies, such as Zero123 [35] , by also ensuring consistent dynamics across all videos produced. This breakthrough marks a critical development in video synthesis, promising new capabilities and applications.CVD faces certain limitations. Primarily, the effectiveness of CVD is inherently linked to the performance of its base models, AnimateDiff [17] and CameraCtrl [18] . While CVD strives to facilitate robust information exchange across videos, it does not inherently solve the challenge of internal consistency within individual videos. As a result, issues such as uncanny shape shifting and dynamic inconsistencies that are presented in the base models may persist, affecting the overall consistency across the video outputs. Additionally, it cannot synthesize videos in real time, owing to the computationally intensive nature of diffusion models. Nevertheless, the field of diffusion model optimization is rapidly evolving, and forthcoming advancements are likely to enhance the efficiency of CVD significantly.Our approach represents a significant advancement in multi-camera video synthesis, with wideranging implications for industries such as filmmaking and content creation. However, we are mindful of the potential misuse, particularly in creating deceptive content like deepfakes. We categorically oppose the exploitation of our methodology for any purposes that infringe upon ethical standards or privacy rights. To counteract the risks associated with such misuse, we advocate for the continuous development and improvement of deepfake detection technologies.A.1 More Analysis on Collaborative Video Generation.For simplicity, the video conditions are omitted in this section without loss of generality. In the paper, we describe the collaborative video diffusion model as a multivariate denoising function p θ (v 1,...,M t ) that estimates the real distribution q(v 1,...,M t-1 |v 1,...,M t , v 1,...,M 0). Following Ho et.al. [22] , the problem can be transformed into the optimizing a noise prediction network ϵ θ (v 1,...,M t ) to predictϵ t = 1 √ 1-ᾱt v 1,...,M t - √ ᾱt √ 1-ᾱt v 1,...,M 0. On the other hand, Song et.al. [52] demonstrated the relation between noise prediction and the score function s θ (v 1,...,M t , t) ≈ ∇ log q(v 1,...,M t ) is:EQUATION)As discussed in the paper, directly training s θ (v 1,...,M t , t) for an arbitrary M is intractable due to the lack of multi-view datasets, so we reduce the problem into generating video pairs (M = 2) instead. Specifically, we train a score function s θ (v i,j ) using video pair datasets and apply it to infer all M videos. Our collaborative video score function, denoted as s CVD (v 1,...,M ), is defined as:EQUATIONwhere P is the set of all selected video pairs, and s θ (v i,j ) i = e i s θ (v i,j ) represents the i'th video component of the score function. Note that the order of i, j is irrelevant.In essence, we utilize the weighted sum of video pair score functions to depict the score function of all videos. We demonstrate that the defined score function s CVD (v 1,...,M ) can estimate the real score function ∇ log q(v 1,...,M ), only if (i,j)∈P w i,j i = 1 for all i ∈ 1, ..., M .Lemma A.1. Let S be a subset of {1, ..., M }, and q(v S t ) be the density function of a set of video features v S t = {v k t |k ∈ S} derived from the forward diffusion process, that is,q(v S t | v S 0 ) = N (v S t ; √ ᾱt v S 0 , (1 -ᾱt )I). Then ∇ v k t log q(v S t | v S 0 ) = 1(k∈S) 1-ᾱt ( √ ᾱt v k 0 -v k t ), where 1(k ∈ S) equals to 1 if k ∈ S and 0 otherwise. Proof.EQUATIONLemma A.2 (Updated Tweedies's Formula). Let S be a subset of {1, ..., M }, and q(v S t ) be the density function of a set of video features v S t = {v k t |k ∈ S} derived from the forward diffusion process, then∇ v k t log q(v S t ) = 1(k∈S) 1-ᾱt ( √ ᾱt E q (v k 0 |v S t ) -v k t ).Proof.EQUATIONEQUATIONEQUATIONTheorem A.3. The function s CVD (v 1,...,M t ) can be an unbiased approximation of the real score function ∇ log q(v 1,...,M t ) for all timesteps t ∈ 1, ..., T , only if (i,j)∈P w i,j i = 1 for all i ∈ 1, ..., M .Proof. For any k ∈ 1, .., M , the k'th component of s CVD (v 1,...,M t ) can be written as:EQUATION=(EQUATIONEQUATIONEQUATIONTo unbiasedly estimate∇ x k t log q(v 1,...,M t ) = 1 1-ᾱt ( √ ᾱt E q (v k 0 |v 1,...,M t ) -v k t )from Eq. 29 w.r.t. all t and v k t , there must be (k,j)∈P w k,j k v k t = v k t , which means (k,j)∈P w k,j k = 1.In addition, we can observe that the accuracy of the estimation from Eq. 29 heavily relies on the similarity between (k,j)∈P w k,j k E q (v k 0 |v k,j t ) and E q (v k 0 |v 1,...,M t ). That means, when we apply a denoising step to a noisy input v 1,...,M t , the output v 1,...,M t-1is more likely to align with the true distribution if the prediction of v k 0 from v k,j t resembles the prediction of v k 0 from all v 1,...,M t . We think this is fairly reasonable in the context of consistent camera-controlled video generation, as the underlying geometry of captured videos can often be discerned from just a few views. We believe this is the key reason why our model can generate consistent multi-view videos trained from video pair data only.We built our pipeline on top of AnimateDiff [17] , a popular open-source T2V model that is widely used among artists. We additionally deploy CameraCtrl [18] to utilize its camera conditioning ability. Following this line of works, we benefit from their plug-and-play property and can swap our base model with a fine-tuned version, e.g., via DreamBooth [45] or LORA [25] .We select 65,000 videos from RealEstate10K [68] and 2,400,000 videos from WebVid10M [1] to train our model. Each data point consists of two videos of 16 frames and their corresponding camera extrinsic and intrinsic parameters. For RealEstate10K, we randomly sample a 31-frame clip from the original video and split it into two videos using the method described in the paper. For WebVid10M, we sample a 16-frame clip, duplicate it to create two videos, and then apply random homography deformations to the second video. The homography transformation matrix H = H t H r H s H sh H p is defined as the composition of a series of transformations: translation, rotation, scaling, shearing, and projection, where:EQUATIONare transformation matrices parameterized by controlling vectors t, θ, s, sh, p. We aim for the first frame of the deformed video to remain unchanged, with the deformation gradually increasing in subsequent frames. To achieve this, we randomly sample the controlling vectors for the last frame from normal distributions. Then, we interpolate these vectors from 0 to the sampled values to obtain the vectors for each intermediate frame and calculate the corresponding matrices.Following [18] , we use the Adam optimizer [30] with learning rate 1e -4. During training, we freeze the vanilla parameters from our backbones and optimize only our newly injected layers. We mix the data points from RealEstate10K and WebVid10M under the ratio of 7 : 3 and train the model in two phases alternatively. All models are trained on 8 NVIDIA A100 GPUs for 100k iterations using an effective batch size 8. The training takes approximately 30 hours.We use DDIM [51] scheduler with 1000 steps during training and 25 steps during inference. Assuming w k,j k is independent with j, we havew k,j k = w k = 1 |(k,j)∈P| .Our algorithm is shown in Alg. 1. We use the partitioning strategy in all of our experiments. For 2-view (video pair) results, we use R = 1(no recurrent denoising) and P = 1; For 4-view results, we use R = 4, P = 1; and for 6-view results, we use R = 6, P = 2. We demonstrate multi-view video generation results in our supplementary videos. Additionally, we show three potential applications of our algorithm: long looping videos, view switching, and potential 3D generation. In Fig. 15 , we show visualization results of our homography warping augmentation applied in our WebVid10M [1] phase. During our training, we removed the L2 loss in the potentially unseen pixels (black regions) of the cloned video for data integrity. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Sec. 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited all works of codes and datasets used in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix:  


Decision: The reviewers raised concerns mainly about the training strategy's clarity, the mathematical derivations' correctness, the integration and generalization of LoRA modules during inference, and the lack of ablation studies to evaluate the impact of key components. They also suggested including recent related works on consistent multi-view content generation. 

The authors responded comprehensively by clarifying the training phases, explaining the epipolar attention mechanism and LoRA selection process, addressing the derivation issues, and providing additional comparisons and qualitative results. They also acknowledged the need for more discussions on related works and ablation studies, which they committed to addressing in their revision. 

The ACs agree that the authors have adequately addressed the concerns, and the paper should be accepted.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel framework that generates multi-view videos from text input. The model builds upon the CameraCtrl pipeline and includes a Cross-View Synchronization Module to enforce consistency guided by a fundamental matrix.
strength_weakness: strength_weakness: - Random homography transformations are applied to the clones for WebVid videos. It would be great if examples of the augmented clones were visualized. I'm mainly worried that the augmented clones seem to introduce a lot of black (unknown) regions and how much it affects the performance of video generation.
- Table 2 should include video-based metrics, such as FVD.
questions: questions: - How is the 3D reconstruction (shown in supp_video.mp4) obtained using the generated multi-view videos?
- How is Table 3 obtained? Why does training on WebVid10M (no camera pose) improve Rot. AUC Trans. AUC (camera accuracy) as compared to training on RE10K only? I'm confused by the results in Table 3.
limitations: limitations: Please refer to the weakness.
paper_summary: paper_summary: This paper proposes a diffusion-based video generation method that generates multiple videos of the same scene simultaneously from camera trajectories and a text prompt. A cross-video synchronization module is proposed, where epipolar attention is introduced to improve the consistency across multiple videos. Experimental results show that the proposed method outperforms state-of-the-art approaches, especially in cross-video geometric consistency and semantic consistency.
strength_weakness: strength_weakness: 1. When the proposed model is trained on different datasets (RE10K LoRA, CameraCtrl Pose LoRA, WV10M LoRA), different LoRA modules are used in the two-phase hybrid training. How are these different LoRA modules used in the final inference step? Are they used together, or is only one LoRA module chosen according to some settings?

2. Eq. 4 is confusing. The variables of Eq. 4 are x_i and x_2. However, x_i is not used on the right side of Eq. 4.

3. It's interesting to exploit the epipolar geometry to ensure cross-video geometric consistency. However, some technical details are unclear in Sec. 4.1. It is not very clear how the epipolar geometry is used to generate the attention mask M in Eq. 4.  It would be better if the authors provided more details and more explanations. Otherwise, it is difficult to reproduce the proposed method.

4. The attention mask is additionally introduced in the cross-view synchronization module. Yet, it is confusing why this mask is being introduced. Does the attention mask indicate correspondences between frames  of different videos?  It would better to provide some statistical results (average percentage ) for the attention mask. 

5. Given a video diffusion model perfectly ensures temporal coherence and has excellent camera control performance， is it still necessary to generate multiple videos simultaneously? Can we use the video diffusion model to generate one video at a time instead?
questions: questions: Please refer to my comments above
limitations: limitations: The authors have addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: The paper introduces a framework to generate consistent multi-view videos of the same scene. Existing models lack precise camera control and consistency across views. The proposed CVD framework uses a cross-video synchronization module with an epipolar attention mechanism to maintain frame consistency from different camera trajectories. Trained with a combination of static multi-view and dynamic monocular data, CVD outperforms existing methods in generating consistent multi-view videos, as claimed by the authors.
strength_weakness: strength_weakness: - In Eq.(4), what does the x_i represent? Should it be x_1 here? I guess this is a typo. I also don’t find the notation illustration for x, which makes it hard to understand the purpose of using Eq.(4) to generate the attention mask. I suggest the authors to provide more details on this.
- For WebVid10M video dataset, the authors use homography warping for the training video generation. However, warping has many limitations. Technically, it only deals with planar points. Applying it to all video frames for “fake new view synthesis” will suffer from lost depth, perspective distortion, and incorrect parallax. Also, homography warping makes it hard to simulate new views from large rotations and translations. The visualized examples provided in the material also don’t show large camera motions. I notice that concurrent works usually construct a “real” multi-view dataset consisting of videos rendered from multi-view from 3D objects. Actually, the authors have a similar solution by re-organizing the videos from the multiview video dataset RealEstate10K and proposing the two-stage training, but the camera motion is still limited. Do the authors have some comments on this?
- The authors mentioned that “For the RealEstate10K dataset, we use CameraCtrl with LoRA fine-tuned on RealEstate10K as the backbone and applying the ground truth epipolar geometry in the cross-video module. For the WebVid10M dataset, we use AnimateDiff with LoRA fine-tuned on WebVid10M as the backbone, and apply the pseudo epipolar geometry (the same strategy used for the first frames in RealEstate10K dataset) in the cross-video module.” It seems that the authors are applying different model structures to train the model for the two datasets. In the inference, how do we combine these two separate models into one? Since the first one is supposed to learn static multiview geometry transform, the other one is supposed to learn more about video dynamics like effect or object motion. How do the learned priors combine together with consistency?
questions: questions: See weakness above.

Justification for rating:

This paper proposes a new pipeline for a relatively new field, i.e., collaborative video diffusion for camera-controllable multi-view video generation. The results provided by the authors show some improvements compared to the baseline CameraCtrl. However, due to the questions and limitations I raised above, I give a slightly positive score here.
limitations: limitations: See above.
paper_summary: paper_summary: This paper studies video generation with camera trajectories. The proposed method improves the consistency across multiple views via a cross-video synchronization module, which is equipped with the existing epipolar attention. Training of the proposed method consists of two phases: training to learn geometric consistency (3D) using ReadEstate10K and training with WebVid10M for learning motions. The authors utilized the augmentation methods such as video folding and homography augmentation for more effective learning addressing the scarcity of data. The proposed method achieves superior performance over baseline methods.
strength_weakness: strength_weakness: - **Ablation study**: The data augmentations, video folding and homography augmentation, are applied. These augmentations are crucial to improve performance. The data augmentation strategies may be effective for the baselines CameraCtrl and MotionCtrl. However, due to the lack of ablation studies, it is hard to evaluate the impact of these components.
- **Lack of technical novelty**: Key value injection and epipolar attention have been proposed in previous works. Cross-view synchronization is merely a combination of existing techniques.
- **Qualitative analysis of trajectories**: Camera control is a challenging problem, and existing methods actually generate seemingly okay videos. However, the generated videos, especially when trajectories are unseen/new, do not precisely follow the input trajectories. Beyond AUC, camera pose comparisons will be helpful for readers to understand the quality of generated videos.
questions: questions: - **Inference.** Multi-stage training has been proposed with/without additional parameters depending on datasets. It is not clear how to perform the inference. It needs a little bit more details.
- **Multiple views beyond two views and section 4.3.** How do you generate videos of multiple views beyond two views and Section 4.3 is not clear. What do you mean by M video features? Are they from different networks?
- **Editing.** Is it possible to apply the proposed pipeline to video editing? Given an input source video, is it possible to generate a new video with the same contents following two camera trajectories?
limitations: limitations: Limitations and Broader impacts are properly discussed in Section 6.1 and 6.2 The performance of the proposed method heavily relies on the performance of base video diffusion models. Also, because of the dependency, the proposed method cannot be applied to real-time applications, yet. Regarding negative societal impacts, the authors discussed deceptive content. The issue could be alleviated by better deepfake detectors.
paper_summary: paper_summary: This paper tackles the problem of consistent multi-video generation, i.e., generating multiple videos capturing the same scene from various camera trajectories. For this, it proposes a cross-view synchronization module (CVD) based on the epipolar geometry. Training-wise, it proposes a hybrid training strategy, exploiting various video datasets. Quantitative and qualitative experiments demonstrate the effectiveness of the proposed approach for generating consistent videos.
strength_weakness: strength_weakness: ## 1. Clarifications about the training strategy

I am confused about how the training strategy in Sec. 4.1 is applied to the trainable parameters, i.e., cross-view synchronization module (CVSM). To be specific, are CVSM blocks shown in Fig. 3 the same model? Namely, will the CVSM in Fig. 3.(b) initialize from the trained CVSM in Fig. 3.(a)?

## 2. Problems in Derivation

I think there are issues from Eq. (16) to (17) in Sec. A of the supplementary. Specifically, this step directly changes $q(v_0^S \vert v_t^S)$ to $q(v_0^k \vert v_t^S)$, which is not suitable.

The main concern I have is that $q(v_0^S \vert v_t^S) \neq \prod_{k\in S} q(v_0^k \vert v_t^S)$ since the videos $\\{ v_0^k \\}_{k \in S}$ captures the same scene and are not independent. 

Please correct or clarify.

## 3. Missing related works

Recently, there have been several works tackling the problem of generating multi-view consistent content via manipulating the latent space, e..g, [a, b]. Please provide some discussions for comparison to such line of work.

[a] Kapon et al., MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusion. CVPR 2024
[b] Jiang et al., MVHuman: Tailoring 2D Diffusion with Multi-view Sampling For Realistic 3D
Human Generation. ArXiv 2023
questions: questions: See "weakness".
limitations: limitations: Yes.

Comments: title: Reply to Reviewer MCcD
Reply to the rebuttal
Response to Rebuttal
comment: Thank the authors for the discussion and for addressing most of my concerns. It may be better to provide more details of the presented datasets.   I will keep my positive score.
We thank the reviewer for the reply and response the new issues below.

## Q1: Non-overlapping regions due to large camera changes.

Yes, this is indeed possible and composes only a small portion of our training data. In this case, consistency becomes trivial as there is limited overlapping content, but our model is still fully compatible since it can simply generate high-quality new content for the non-overlapping regions too. This is because our model does not alter the parameters of our base video generative models and is trained with both RealEstate10K and WebVid10M to handle generation effectively, with generation performances on par with our baselines as indicated in Tab. 2. To summarize, our model can handle large camera motion and non-overlapping scenarios with no problem, so the non-overlapping large camera movement videos in our training set do not affect performances of our model.

## Q2: Camera Differences in Homography Transformation.

We, in fact, adjust the homography transformation parameter according to RealEstate10K's camera movements so that it's a relatively close simulation. Therefore, whatever we have for the homography augmentation should be similar to the camera trajectories in RealEstate10K and will not restrict the difference in camera trajectories.

## Q3: Fireworks are not so Synchronized.

As we mentioned in our limitation section, our method is heavily dependent on the performances of our base models. Since we built CVD on AnimateDiff, which has quite a lot of flicking and content shifting itself, our results are potentially influenced too and are certainly not perfect --- we may suffer from the same type of issues that cause some small inconsistency and non-synchronized parts. We believe with newer, more consistent video generation models with fewer flickings such as OpenSORA/CogVideoX, our method should transfer, and potentially, the small issues will address themselves.
Thank the authors for answering my questions.
- RealEstate10K videos.  Some videos in this dataset have larger camera motions. Could it be that the i-th frames of the two videos have only a few corresponding regions?
- WebVid10M videos. The paper uses random homography transformations to create video pairs.  Does this restrict the difference in camera trajectories between the two videos from being too significant

In addition,  the fireworks are not so synchronized, as shown in the last two rows of Fig. 13 and the demo video
I appreciate the authors' detailed responses. They have addresses all my minor concerns. Originally, I did not have any major concerns. I will keep my original rating.
Dear Reviewer,

Thank you for your insightful feedback and for recognizing the value of our proposed method. We appreciate the opportunity to clarify the concerns regarding the synchronization of generated multi-videos and the training of our Cross-View Synchronization Module (CVSM).

We train our CVSM module using two data sources:
- RealEstate10K videos, which are static scene captures.
- WebVid10M videos, which are dynamic in-the-wild videos.

As outlined in our Sec. 4.2, **the middle-cut and reverse scheme is only applied to the static RealEstate10K videos** — the static nature of these videos ensures that epipolar geometry always holds, therefore the above-mentioned scenario, where a chair moves (e.g. appears in the first part but disappear in the corresponding frame in the second part) will never happen, as there is no dynamic movement in these training videos. The only possibility that such a scenario may arise is if the camera itself moves beyond the visibility of the chair, which is commonly presented in our training data.  In this scenario, the content shared across two views would be synchronized by our epipolar attention in CVSM. For the non-overlapping regions, consistency becomes trivial and it typically relies more on the model’s generation capability to construct new content. Our model does not alter the parameters of our base video generative models and is trained with both RealEstate10K and WebVid10M to handle generation effectively, with generation performances on par with our baselines as indicated in Tab. 2. **For the dynamic WebVid10M videos, we intentionally avoid applying the middle-cut and reverse scheme** to prevent the exact issues you highlighted. Instead, we create paired data by duplicating a clip and applying homography augmentation. In this case, the paired videos are perfectly synchronized, as they are essentially copies of each other with perfectly consistent timestamps.

Therefore, our CVSM module, derived from epipolar geometry, is always trained on paired videos that hold perfect epipolar geometry and are therefore always perfectly synchronized. This ensures that as a generative model, the output of our model is also synchronized.
Has the author's rebuttal addressed your concerns?

Please provide your feedback on the rebuttal and make your final decision.

AC
Has the author's rebuttal addressed your concerns? 

Please provide your response to the rebuttal and make your final decision.

AC
Thank the authors for answering my questions.  I still have questions about whether the proposed method can ensure the Synchronization of generated multi-videos (also the Cross-View Synchronization Module).  The concern is from the training data.  As stated in the paper, the paper constructs a pair of training data by first sampling 2N − 1 frames from a video in the dataset and divides them into two clips (e.g., N,-1 ) from the middle, where the first part is reversed.  There is no guarantee that the content of the i-th frames in both video clips is synchronized or consistent. For example, a chair may appear in the first part and disappear in the second part.
We thank the reviewer for the reply and respectfully disagree with some of the claims.

## (Q1, Q3) Our model does not have enough ability to generate large-scale camera changes: 

**We first argue that our current model can handle large camera motions to some extent, as shown in our results**. For example, sample #3 in the first generation results in our demo video (0:01) are controlled by very different cameras; In the first demo of our 6-view video generation (1:35 in the demo video), the differences between the view on the left-top and the one on the bottom-right are also significant. These examples represent some of the most diverse and extreme camera trajectory variations possible that we can think of, within a 16-frame sequence, demonstrating that our method performs well even when the camera moves in opposite directions (e.g., flying in vs. flying out, rotating left vs. rotating right).

Second, CVD is grounded in the currently limited infrastructures of open-source video generation and available datasets, functioning as a proof-of-concept that maximizes the potential of existing resources. This small-scale setup naturally affects the model’s capability to support large camera motions for two reasons:
The capabilities of our method are inherently constrained by the performance of our base models, namely CameraCtrl. Consequently, the extent of motion and the range of camera paths we can manage are limited by the pretrained AnimateDiff and CameraCtrl models. Since the camera path conditioning is derived from RealEstate10K’s camera paths in CameraCtrl (and other models such as MotionCtrl, VD3D), our method cannot accommodate dramatic camera changes beyond what is covered in the RealEstate10K domain. As an extension of existing camera-conditioned video generation techniques, our approach necessarily inherits these limitations.
At the time of our paper’s development, open-source video generation models were generally limited to producing only a few frames. For example, AnimateDiff, the base model used in our work, is capable of generating only 16-frame videos. Within this frame limit, accommodating large motions and significant camera movements is particularly challenging. 
Although CVD is relatively small in scale, we are inspired by our results and see significant potential in a scaled-up version incorporating systems like SORA or DreamMachine. We believe our work will inspire and motivate future research and development efforts in multi-view and multi-camera video generation.




## (Q2) Our model lacks generalization by using different LoRA for different experiments: 

We want to clarify that it is a common practice for diffusion models to apply different LoRA in different tasks. Both CameraCtrl and AnimateDiff are trained with specific LoRA to adapt the training data distribution, but are combined with unseen LoRAs in their qualitative results. 
While we apply different LoRAs for the spatial attention layers (which are pretrained models from previous works) for each task, our CVSM module remains the same across all experiments. We also want to emphasize that, although during the training our model **has never seen RealVision’s LoRA module**, it can naturally adapt this LoRA during the inference. These indicate that our CVSM can be generalized in various tasks.

On the other hand, even though we have shown that our model generalizes well to unseen LoRAs, this does not mean our method does not work well without a LoRA. To support this, we also provide additional results without using any appearance LoRA (along with results in Table 1. in our paper):

Model w/o LoRA:

Rot. AUC: 21.9 / 34.6 / 49.4

Trans. AUC: 2.7 / 7.4 / 16.3

Prec.: 44.0

M-S: 17.6

Full Model:

Rot. AUC: 25.2 / 40.7 / 57.5

Trans. AUC: 3.7 / 9.6 / 19.9

Prec.: 51.0

M-S: 23.5

MotionCtrl [56]+SVD:

Rot. AUC: 12.2 / 28.2 / 48.0

Trans. AUC: 1.2 / 4.9 / 13.5

Prec.: 23.5

M-S: 12.8

Our model without any LoRA slightly underperforms our full model but still performs much better than the baselines.

We also notice that merging different LoRAs into a single model is an active area of research (e.g. Work from Gu et al. [1] and Po et al. [2]). Therefore, while our setting matches standard practice in the field, we believe it would be interesting to explore CVD in combination with very recent advances, like MoS or orthogonal adaptation, which we will clarify in the discussion.

[1] Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models, Gu et al. 2024
[2] Orthogonal adaptation for modular customization of diffusion models, Po et al., 2024

Thank you again for your time and consideration.
Thanks to the authors for the clarification. 

- Thanks for the explanation on the warping. However, I still think that the current training curriculum and generated results lack enough ability to enable larger camera motion. The discussed solutions may work but they have been done and verified in this submitted version.
- I notice that the proposed method can't train a consistent and generalizable model to treat different scenarios. The proposed method needs to switch different LoRAs trained from datasets from different domains to generate different content for the best performance. This is a major flaw that seriously harms the generalization ability of the method.
- Also, I doubt whether the current model can handle diverse and distant camera trajectories for collaborative (or multi-view) video generation. Most of the given visualized examples show connected or spatially close camera trajectories.

Considering the above concerns, I keep my final rating on the borderline.
I thank the authors' time and effort in addressing my concerns.

I have carefully read all reviews and the rebuttal and am inclined to maintain my positive score for the submission.

Please update the training and inference details in the final version as it seems most reviewers are confused.