Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control
Authors: Zhengfei Kuang
Shengqu Cai
Hao He
Yinghao Xu
Hongsheng Li
Leonidas Guibas
Gordon Wetzstein
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Rendering
Video Diffusion Models
Multi-View Generation
Primary area: diffusion_based_models

Abstract: Research on video generation has recently made tremendous progress, enabling high-quality videos to be generated from text prompts or images. Adding control to the video generation process is an important goal moving forward and recent approaches that condition video generation models on camera trajectories take an important step towards this goal. Yet, it remains challenging to generate a video of the same scene from multiple different camera trajectories. Solutions to this multi-video generation problem could enable large-scale 3D scene generation with editable camera trajectories, among other applications. We introduce collaborative video diffusion (CVD) as an important step towards this vision. The CVD framework includes a novel cross-video synchronization module that promotes consistency between corresponding frames of the same video rendered from different camera poses using an epipolar attention mechanism. Trained on top of a state-of-the-art camera-control module for video generation, CVD generates multiple videos rendered from different camera trajectories with significantly better consistency than baselines, as shown in extensive experiments.

Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control
Zhengfei Kuang* Hongsheng Li
Shengqu Cai*! Leonidas Guibas!
Hao He?
Yinghao Xu'
Gordon Wetzstein
Stanford Univerity
CCHK
Abstract
Research video generation has recently made tremendous progress, enabling high-quality videos to be generated from text prompts or images. Adding con- trol to the video generation process is an important goal moving forward and recent approaches that condition video generation models on camera trajecto- ries make strides towards it. Yet, it remains challenging to generate a video of the same scene from multiple different camera trajectories. Solutions to this multi-video generation problem could enable large-scale 3D scene generation with editable camera trajectories, among other applications. We introduce collabora- tive video diffusion (CVD) as an important step towards this vision. The CVD framework includes a novel cross-video synchronization module that promotes consistency between corresponding frames the same video rendered from dif- ferent camera poses using epipolar attention mechanism. Trained top of a ofate-of-theeart camera-control module for video generation, CVD generates multiple videos rendered from different camera trajectories with significantly bet- ter consistency than baselines, as shown in extensive experiments. Project page: fusii.. github.
Introduction
With the impressive progress of diffusion models [22, 51, 28, 44, 46, 43], video generation has significantly advanced [12, 17, 26, 5, 6, 15, 64, 23, 32], with transformative impact digital content creation workflows. Recent models like SORA [7] exhibit the ability to generate long high- quality videos with complex dynamics. Yet, these methods typically leverage text image inputs to control the generation process and lack precise control over content and motion, which is essential for practical applications. Prior efforts explore the use of other input modalities, such as flow, keypoints, and depths, and develop novel control modules to incorporate these conditions effectively, enabling precise guidance of the generated contents [60, 63, 67. 26, 56, 9]. Despite these advancements, these methods still fail to provide camera control to the video generation process.
control using various techniques, such motion LoRAs [25, 17] or scene flows [63, 67]. Some representative works such as MotionCtrl [60] and CameraCtrl [18] offer more flexible camera control by conditioning the video generative models on a sequence of camera poses, showing the feasibility of freely controlling the camera movements of videos. However, these methods are limited single-camera trajectories, leading significant inconsistencies in content and dynamics when generating multiple videos of the same scene from different camera trajectories. Consistent multi-video generation with camera control desirable in many downstream upplications, such as large-scale 3D scene generation. Training video generation models for consistent videos with different camera trajectories, however, is very challenging, partly due to the lack of large-scale multi-view dynamic in-the-wild scene data.
Recent works have started to focus on
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we introduce CVD, a plug-and and-play module capable of generating videos with different camera trajectories sharing the same underlying content and motion of scene. CVD is designed on a collaborative diffusion process that generates consistent pairs of videos with individually controllable camera trajectories. Consistency between corresponding frames of a video enabled using epipolar attention, introduced by a learnable cross-view synchronization module. To effectively train this module, we propose new pseudo-epipolar line sampling scheme to enrich the epipolar geometry attention. Due to the shortage of large-scale training data for 3D dynamic scenes, we propose hybrid training scheme where multi-view static data from RealEstate 10k [68] and monocular dynamic data from VidVid10M [1] are utilized to learn camera control and motion, respectively. To our knowledge, CVD is the first approach to generate multiple videos with consistent content and dynamics while providing camera control. Through extensive experiments, we demonstrate that CVD ensures strong geometric and semantic consistencies, significantly outperforming relevant baselines. We summarize contributions follows:
To our knowledge, our CVD is the first video diffusion model that generates multi-view consistent videos with camera control;
We introduce a novel module called the Cross-Vides Synchronization Module, designed to align features across diverse input videos for enhanced consistency;
We propose a new collaborative inference algorithm to extend our video model trained on video pairs to arbitrary numbers of video generation;
Our model demonstrates superior performance in generating multi-view videos with consis- tent content and motion, surpassing all baseline methods by significant margin.
2
Related Work
Video Diffusion Models. Recent efforts in training large-scale video diffusion models have enabled
high-quality video generation [15, 6, 23, 21, 17, 12, 50]. Video Diffusion Model [23], utilizes
3D UNet to learn from images and videos jointly. With the promising image quality obtained by text-to-image (T2I) generation models, such StableDiffusion [44], many recent efforts focus on extending pretrained T2I models by learning a temporal module. Align- lour-latentss [6] proposes to inflate the T2I model with 3D convolutions and factorized space-temporal blocks to learn video dynamics. Similarly, AnimateDiff [17] builds upon StableDiffusion [44], adding a temporal module after each fixed spatial layer to achieve plug-and-play capabilities that allow users to perform personalized animation without any finetuning. Pyoco [15] proposes temporally coherent noise strategy to effectively model temporal dynamies. More recently, SORA [7] shows great step towards photo-realistic long video generation by utilizing space-time diffusion with transformer architecture.
Controllable Video Generation. The ambiguity of textual conditions often results in weak control for text-to-video models (T2V). To provide precise guidance, some approaches utilize additional conditioning signals such as depth, skeleton, and flow control the generated videos [12, 59, 26, 48, 27, 8, 53]. Recent efforts like SparseCtrl [65] and SVD incorporate images as control signals for video generation. To further control motions and camera views in the output video, DragNUWA [63] and MotionCtri [60] inject motion and camera trajectories into the conditioning branch, where the former uses a relaxed version of optical flow as stroke-like interactive instruction, and the later directly concatenate camera parameters as additional features. CameraCtrl [18] proposes to over- parameterize the camera parameters using Plücker Embeddings [39] and achieves more accurate camera conditioning. Alternatively, AnimateDiff [17] trains camera-traeectory LoRAs [25] to achieve viewpoint movement conditioning, while MotionDirector [67] also utilizes LoRAs [25] but to overfit to specific appearances and motions to gain their decoupling.
Multi-View Image Generation. Due the lack of high-quality scene-level 3D datasets, a line of research focuses on generating coherent multi-view images. Zero123 [35] learns to generate novel-view images from pose conditions, and subsequent works extend it to multi-view diffu- sion [11, 36, 37, 49, 54, 55, 62, 31] for better view consistency. However, these methods are only restricted to objects and consistently fail to generate high-quality large-scale 3D scenes. Mul- tiDiffusion [3] and DiffCollage [66] facilitates 360-degree scene image generation, while Sce- neScape [14] generates zooming-out views by warping and inpainting using diffusion models. Simi- larly, Text2Room [24] generates multi-view images of room, where the images can be projected via

---

Separate
Shared Content
Scene
CVD or
Semantic
Nizer Park
VIENG
CVD
Conventional
Collaborative
Video
Video
CVD for More Videos
Generation
Diffussin
Figure I: An illustration of pairwise collaborative video generation. Existing video diffusion models generate videos separately. which may result in inconsistent frame contents (e.g.. geometries, objects, motions) across videos (Left); Collaborative video generation aims produce videos sharing the same underlying content (Middle); In this work, we train our model on video pair datasets, and extend it generate more collaborative videos (Right).
depths to get coherent room mesh. DifiDreamer [9] follows the setups in Infinite-Nature [34, 33] and iteratively performs projection and refinement using a conditional diffusion model. A recent work, BaseGudddd-iiffusiii
[56], performs novel view synthesis from single image by training and adding an epipolar line bias to its attention masks on multi-view datasets with camera poses pro- vided (RealEstate 1k [68]). However, this method by construction does not generalize to in-the-wild or dynamic scenes, as its prior is solely learned from well-defined static indoor data.
A comprehensive survey of recent advances in diffusion models for visual computing is provided by Po et al. [40].
3 Collaborative Video Generation
Conventionally, video diffusion models (VDMs) aim to generate videos from randomly sampled Gaussian noise with multiple denoising steps, given conditions such as text prompts, frames, or camera poses. Specifically,
do ~ Idma (v) be a data point sampled from the data distribution; the forward diffusion process continuously adds noises to Un to get series of Ur, E 1...., T until becomes Gaussian noise. Using the reparameterization trick from Ho et [22] the distribution of can be represented as q(v, vo) NV,,;; Va, (1 ()) where noise scheduling parameters, which are monotonously increasing, and ãr 1. The video diffusion model, typically denoted as po(vt-1/e,) is model parameterized by A that is trained to estimate the backward distribution q(v:-1 v,, vo). According to Ho et al. [22], the optimization of pe (0,-1 results in minimizing the following loss function:
E (0, 1] are the
= Ee, - co(u, t, c.)),
(I)
where Vir + (1- de is the noisy video feature generated from vo and random sampled Gaussian noise e, en <0(";, 1) is the noise prediction of the VDM, and e the video condition. During inference time, one can start from normalized Gaussian noise ~ N(O /) and apply the noise prediction model eg (ve, t) multiple times to denoise it until vo-
Empowered by readily available large-scale video datasets, many state-of-the-ee VDMs have success- fully shown ability to produce temporally consistent and realistic videos [23, 17, 26, 21, 12. 6. 15). However, one of the key drawbacks of all these existing methods is the inability to generate consis- tently coherent multi-vier -iew videos. As Fig. 1 shows, videos generated from a VDM under the same textual conditions exhibit content and spatial arrangement disparities. One can use inference-stage tricks, such as extended attention [8], to increase the semantic similarities between the videos, yet this does not address the problem of structure consistency. To address this issue, we introduce a novel objective for VDMs to generate multiple structurally consistent videos simultaneously given certain semantic conditions and dub it Collaborative Video Diffusion (CVD).
3

---

xtx
X
hnntemerral
Prodectd Nomouue
Poune
2003
Makill Altesiio
Munked Altetion
Noiiy latcess
Notuce?
OuG
Sual Difaics Cones
Ciea-Vly www.heeeoneeacceooo (Traael
Flcker Exteedaa
Pr
(a) Network Overview
(b) Cross-View Synchronization Module
Figure 2: Architecture of collaborative video diffusion. Left: The model takes two (or more) noisy video features and camera trajectories input and generates the noise prediction for both videos. Note that the image autoencoder Stable Diffusion omitted here; Right: Our Cross- View Synchronization Module takes the same frames from the videos along with the corresponding fundamental matrix as input, and applies masked cross-view attention between the frames.
In contrast to conventional video diffusion models, CVD seeks to find an arbitrary number of videos v E 1..... .M that comply with the unknown data distribution Gdma (v I (ala,, given separate conditions g......dd, Similarly, the CVD model can be represented as p (v-...., I (f ). An example includes multi-view videos synchronously captured from the same dynamic 3D scene. Similarly, the loss function for a collaborative video diffusion model is defined as:
LCVD = Ee.....,MMM M..... ,,clle" I...... (v, !....M .t, ,t.c..,)
(2)
In practice, however, the scarcity of large-scale multi-view video data prevents from directly training model for an arbitrary quantity of videos. Therefore, we build our training dataset of consistent video pairs (i.e., .M = 2) from existing monocular video datasets, and train the diffusion model to generate pairs of videos sharing the same underlying contents and motions (see details in Secs. 4.1 and 4.2). Our model is designed to accommodate any number of input video features, however, and we develop an inference algorithm generate an arbitrary number of videos from our pre-trained pairwise CVD model (see Sec. 4.3).
4 Collaborative Video Diffusion with Camera Control
We seek to build a diffusion model that takes a text prompt y and set of camera trajectories cum and generates the same number of collaborative videos vM......M To ease the generation of consistent videos, in this work we train our model with video pairs (M = 2), we make the assumption that the videos synchronized (i.c., corresponding frames captured simultaneously), and set the first pose of every trajectory to be identical, forcing the first frame all vidcos to be the same.
Inspired by [18, 17], our model is designed as extension of the controecontoolled video model CameraCtrl [18]. shown in Fig. 2, our model takes two (or more) noisy video feature inputs and generates the noise prediction in a single pass. The video features pass through the pretrained weights of CameraCtrl and are synchronized in our proposed Cross- View Synchronization Modules (Sec. 4.1.. The model is trained with two different datasets: RealEstate 10K [68], which consists of camera-calibrated video on mostly static scenes, and WebVid10M [1], which contains generic videos without poses. This leads to our two-phase training strategy introduced in Sec. 4.2. The learned model can infer arbitrary numbers of videos using our proposed inference algorithm. which will be described in Sec. 4.3.
4.1 Cross- View Synchronization Module
State-of-the-art VDMs commonly incorporate various types of attention mechanisms defined on the spatial and temporal dimension: works such as AnimateDiff [17], SVD [5], LVDM [19] disentangles space and time and applies separate attention layers; the very recent breakthrough SORA [7] processes both dimensions jointly on its 3D spatial-temporal attention modules. Whilst the operations defined on the spatial and temporal dimensions bring strong correlation between different pixels of different frames, capturing the context between different videos requires new operation: cross-video attention.

---

Thankfully, prior works [10, 8] have shown that the extended attention technique, i.e., concatenating the key and values from different views together, evidently efficient for preserving identical semantic information across videos. However, it refrains from preserving the structure consistency among them, leading to totally different scenes in terms of geometry. Thus, inspired by [56], we introduce the Cross- View Synchronization Module based on the epipolar geometry to shed light on the structure relationship between cross-video frames during the generation process, aligning the videos towards the same geometry.
Fig. 2 demonstrates the design of our cross-view module for two videos. Taking a pair of feature sequences z z, zi...., N of N frames as input, our module applies cross-video attention between the same frames from the two videos. Specifically, we define our module as:
out (Attt((t(WW)) Wo W Wxzk, Wuzk, M (x1, x2) = 1xxx F < Tepi)
Ak L,,,,
(4)
where k is the frame index, W Q. W Wv are the query, key, value mapping matrices, MM is the attention mask, Attn Q, K, V, M) is the attention operator introduced from the Transformer [58], ff is the feed-forward function and F is the fundamental matrix between cam,, and cam. The attention mask M between any two pixels X1, X2 is determined by the epipolar distance betweenx, and X2, L.e. the shortest distance between X1 and the epipolar line of X2 in X1 's frame, which is set to if the epipolar distance is smaller than a given threshold Tepi (set to 3 in all of our experiments) and vise versa. The outputs of these modules are used as residual connections with corresponding original inputs to ensure no loss of originally learned signals. The key insight of this module is as the two videos assumed be synchronized to each other, the same frame from the two videos is supposed to share the same underlying geometry and hence can be correlated by their epipolar geometry defined by the given camera poses. For the first frames where the camera poses are set to be identical since the fundamental matrix is undefined here, we generate pseudo epipolar lines for each pixel with random slopes that go through the pixels themselves. In the scenario where multi-view datasets are available, the modules can be further adapted to more videos by extending the cross-view attention from 1-to-1 to 1-to-many. Our study shows that epipolar based attention remarkably increases the geometry integrity of the generated video pairs.
4.2 Hybrid Training Strategy from Two Datasets
Considering the fact that there is no available large-scale real-world dataset for video pairs, we opt make of the two popular monoc- ular datasets, RealEstate 10K [68] and Web- VidIOM [1], to develop a hybrid training strat- egy for video pair generation models.
Dataset
Reveno
Depaull Video
Vilao
Vidoo
Oriiial Vidia
Vidoo P1
Video
RealEstate with Video Folding. The first phase of the training involves RealEstate 10K [68], dataset consisting of video clips capturing mostly static indoor scenes and corresponding camera poses. We sample video pairs by simply sampling subsequences of 2N frames from a video in the dataset, then cutting them from the middle and reversing their first parts to form synchronized video pairs. In other words, the subsequences are folded into two video clips sharing the same starting frame.
Model
Casurs
stal Wewiic
Stale Dalaai Lyan
Eppdar
CVSM
CVSM
(8) RcalEstate100 10K Phase
(b) WebVidIOM Phase
Figure 3: Two-Phase Hybrid Training. We use different data processing schemes to handle the two datasets (Top) and apply separate model structures to train in corresponding phases (Bottom).
Web Vid10M with Homography Augmentation. While RealEstate [68] provides a decent geometry prior, training our model only on this dataset is not ideal since it does not provide any knowledge regarding dynamics and only contains indoor scenes. On the other hand, Web VidIOM, a large-scale video dataset, consists of all kinds of videos and can be used as a good supplement to RealFsttaee10K 10K. To extract video pairs, we clone the videos in the dataset and then apply random homography transformations to the clones. Nonetheless, The Vid100 dataset contains no camera information, making it unsuitable for camerra-coditioneed model training. To address this problem,
5

---

we propose a two-phase training strategy to adapt both datasets (with or without camera poses) for the same model.
Two-Phase Training. As previously mentioned, our model is built upon the existing camera- controlled VDM CameraCtrl [18]. It an extended version of AnimateDiff [17] that adds a pose encoder and several pose feature injectors for the temporal attention layers to the original model. Both AnimateDiff [17] and CameraCtrl [18] are based on Stable Diffusion [44]. This implies that they incorporate the same latent space domain, and thus, it is feasible to train module that can be universally adapted. Therefore, our training procedure follows a two-phase scheme Fig. 3 shows. Specifically, build hybrid dataset that combines the data from both sources. Then in each training iteration, if the training data is from RealEstate 10KK, we use CameraCtrl with LoRA fine-tuned on RealEstate 10K as the backbone and applying the ground truth epipolar geometry in the video module. Otherwise, we use AnimateDiff with LoRA fine-tuned on WebVid as the backbone, and apply the pseudo epipolar geometry (the same strategy used for the first frames in RealEstate 10K dataset) in the cross-video module. The two training phases are applied alternatively to the same instance of CVSM in a single training procedure. Experiments show that the hybrid training strategy greatly helps the model generate videos with synchronized motions and great geometry consistency.
4.3 Towards More Videos
With the CVD trained video pairs, during inference, can generate multiple videos (for example, M videos where M > 2) that share consistent content and motions. To achieve that, we start from M individual gaussian noise maps and denoise them in multiple steps. At each denoising step
select P feature pairs P (1.51.11, v.2 of J. (i,....,; Ji.....' P E 1, ..... M) among all M video features. We then use the trained network to predict the noise of each feature pair, and averaging them w.r.t. each video feature. That is, the output noise for the ith video feature is defined
as: Cout (x:) Avg eP (c) (u) ,t, camf")), where ca(2, f. tt cam²") ) is the noise prediction for oy given the video pair input vt For pair selection, we propose the following strategies:
Exhaustive Strategy: Select all (M 1)/2 pairs.
Partitioning Strategy: Randomly divide noisy video inputs into M pairs.
2 Multi- PattitPaatitionin Strategy: Repeat the Partitioning Strategy multiple times and combine all selected pairs
The exhaustive strategy has a higher computational complexity of ((M²) compared to the partitioning one (O(M)) but covers every pair among M videos and thus produce more consistent videos. The nunti-partitionnig strategy, on the other hand, a trade-off between the two strategies. We also embrace the recurrent denoising method introduced by Bansal al. [2] that does multiple recurrent iterations on each denoising timestep. We provide the pseudo-code of our inference algorithm and detailed mathematical analysis in our supplementary.
5
Experiments
5.1
Qualitative Results
5.1.1
Comparison with Baselines
Qualitative comparisons are shown in Fig. 4. Following our quantitative comparisons in Sec. 5.2, we compare against CameraCtrl [18] and its combination with SparseCtrl [16], MotionCtrl [60] and its combination with SVD [5]. The results indicate our method's 's superiority aligning the content within the videos, including dynamic content such as lightning, waves, etc. More qualitative results are provided our supplemental material and video.
5.1.2 Additional results for arbitrary views generation
We also show the results of arbitrary view generation shown in Fig. 5. Using the algorithm introduced in Sec. 4.3, our model can generate groups of different camera-conditioned videos that share the same contents, structure, and motion. Please refer to supplementary video for animated results.

---

5.2 Quantitative Results
compare our model with two state-of-the-art camera-controlled video diffusion models for quantitative evaluation: CameraCtrl [18] and MotionCtrl [60]. Both of the two baselines are trained on the RealEstate 10K [68] for camera-controlled video generation. We conduct the following experiments to test the geometric consistency, semantic consistency, and video fidelity of all models:
Table I: Quantitative Results Geometry Consistency. Following SuperGlue [47], report the area under the cumulative error curve (AUC) of the predicted camera rotation and translation under certain thresholds (5", 10 20%), and the precision (P) and matching score (MS) of the Super- Glue correspondences. We feed the models with prompts from RealEstate [68] (RE10K) and Web [1] (WV10M) in two experiments separately. For RealEstate 10K scenes, also run SuperGlue on the original RealEstate 10K [68] frames as reference. Our model achieves the highest scores all metrics compared baselines.
Rot. AUC (@5" /10" /20")
Trans, AUC (@5' /10" /20")
Scenes
Methods
Pree.
M-S.
Reference
6../ 61.4/77.2/87.8
60.2
36.5
CameraCull [18]
34.8/552/72.4
2.3/6.6/ 17.0
50.8
27.3
RE10K
MotionCtrl [60]
49.0/ 68.0/81.2
3,4/ 10.2
64.6
38.9
Ours
155/718/83..
5.6 /3.9
76.9
42.3
CameraCtrl Sparseen [16]
6.2/14.37255
0.5/1.774.7
16.5
5.4
WV10M
MotionCtul [60]+SVD [5]
28.2/28.//8.00
1,2499 13.5
23.5
12.8
Ours
46.2/40./57.
3.7 19.9
51.0
23.5
Table 2: Quantitative Results for semantic & fidelity metrics. The semantic metrics are evaluated on Web VidIOM [1] and the fidelity metrics performed on RealEsstae [68]. As shown in the table, our method is better than or on par with all prior work regarding semantic matching with the prompt, cross-video consistency, and frame fidelity.
Semantic Consistency CLIP-T
Fidelity KID
CLIP-F
FID
FVD
MotionCtrl [60]+SVD [5]
0.81
CameraCtrl [18]
0.28
0.79
32.10
0.79
277
AnimateDiff Sparsepa [16]
0.29
0.86
51.97
1.86
327
CameraCtrl [18]+SparseCtrl [16]
0.29
0.85
61.68
2.47
430
Ours
0.30
0.93
32.90
0.61
285
Per-video geometric consistency on estate scenes. Following CameraCtrl [18], we first test the geometry consistency across the frames in the video generated from our model, using the camera trajectories and text prompts from RealEstate 10KK [68] (which mostly consists of static scenes). Specifically, we first generate 1000 videos from randomly sampled camera trajectory pairs (two camera trajectories with the same starting transformation) and text captions. All baselines generate one video at a time; our model generates two videos simultaneously. For each generated video, we apply the state-of-the-an image matching algorithm SuperGlue [47] to extract the correspondences between its first frame and following frames and estimate their relative camera poses using the RANSAC [13, 42] algorithm. To evaluate the quality of correspondences and estimated camera poses, we adopt the same protocol from SuperGlue [47], which 1) evaluates the poses by the angle error of their rotation and translation and 2) evaluates the matched correspondences by their epipolar error (i.e., the distance to the ground truth epipolar line). The results are shown Tab. 1, where our model significantly outperforms all baselines. More details are provided in our supplementary materials.
Cross-video geometric consistency generic scenes. Aside from evaluating the consistency be- tween frames the same video, we also test our model's ability to preserve the geometry information across different videos. To do that, we randomly sample 500 video pairs (1000 videos in total) using camera trajectory pairs from RealEsttate 10K [68] and text prompts from Web 1M's captions [1]. To the best of our knowledge, there is available large video diffusion model that is designed to generate multi-view consistent videos for generic scenes. Hence, we modify the CameraCtrl [18] and MotionCtrl [60] to generate video pairs baselines. Here, use the text-to-video version of each
7

---

MotionnCet
Cameraaat
trajectories
CareerCCaal C Sparseer
Oura
plasto coautine, rudes weather, wind, lightning. uhuld, diir, lighting. high quality, Sim grain, zujfiin NYA..
clane platoo of rubhi, fores, hume, balation, bllor, drumatic simusphere, contred, thirah, 200 2.4f marro shat.
Figure 4: Qualitative comparison. Our method maintains consistency across videos for static and dynamic scenes, while no prior work can generate synchronized different realizations. * Despite our best efforts, we incapable of getting MotionCtrl [60]+SVD [5] to generate any motion beyond the simplest static camera zooming in-and-out, Please refer to our supplemental video for illustration.
model generate reference video first, then take its first frame the input of their image-to-video version (i.e., their combination with SparseCtrl [16] and SVD (5]) to derive the second video. We use the same metries in the first experiment but instead evaluate between the corresponding frames from the two videos. Results are shown in Tab. 1, where our model greatly outperforms all baselines.
Semantic and fidelity evaluations. Following the standard practice of prior works [17, 61, 29, 9, 10, 8], we report CLIP [41] embedding similarity between 1) each frame the output video and the corresponding input prompt and 2) pairs of frames across video pairs. The former metric, denoted as CLIP-T, is to show that our model does not destroy the appearance/content prior of our base model, and the latter, denoted as CLIP-R, is aimed to show that the cross-view module can improve the semantic and structural consistency between the generated video pair. For these purposes, we randomly sample 1000 videos using camera trajectory pairs from RealEstate 10K, along with text

---

captions from WebVidIOM (2000 videos generated in total). To further demonstrate our method's ability maintain high-fid fidelity generation contents, we report FID [20] and KID [4] x x100 using the implementation [38], and FVD [57]. We do not compare against models that do not share the same base model as us for FID [20], KID [4] and FVD [57], since these metrics are strongly influenced by the abilities of the base models. Following prior work [18], we evaluate these two metrics on RealEstate 10k [68] because of the strong undesired bias, c.g., watermarks, on Web VidVidI0M [1]. As shown in Tab. 2, our model surpasses all baselines for the CLIP [41]-based metrics. This proves model's ability to synthesize collaborative videos that share a scene while maintaining and improving fidelity according to the prompt. Our model also performs better than or par with all prior works on fidelity metrics, which indicates robustness to the appearances and content priors learned by our base models.
5.3 Ablation Study
Table 3: Ablation Study conducted on generic scenes (prompts from WebVid [1]), where we deactivate each of our introduced modules. Results indicate that our full pipeline outperforms the ablation settings for both geometric and semantic consistencies.
Rot. AUC
Trans. AUC (@5' /100 /209)
Semantic Consistency CLIP-T
(@5 /10% /20")
CLIP-F f
Ours w/o Epi
16.8/31//49
1..5
0.30
0.91
Ours REIOK only
17.9 23.8/333
1.7/5.3/13.2
0.29
0,90
Ours w/o HG
22.0 35.5/
4.3/611/4.5
0.29
0.92
Ours I Layer
22.7 / / 54.3
3.1
19.2
0.29
0.92
Ours
25.2/ 40.7 57.5
3.7/9.6/ 19.9
0.30
0.93
We perform a thorough ablation study in Tab. 3 to verify our design choices, where the variants are: No epipolar line constraints (Ours w/o Epi), where we perform a normal self-attention instead of epipolar attention in our Cross-Vien Synchronization Module; 2) No mixed training (Ours REIOK only), where we follow the setups in CameraCtrl [18] and train the model only on RealEstate 10k [68]; 3) No homography augmentation (Ours w/o HG). where we switch off the homography transformations applied to Web Vid10i [1] videos during training: and 4) using only 1 Cross- View Synchronization Module instead of 2 (Ours / Layer). The ablation study indicates that while we can get semantically consistent outputs without epipolar constraints, they are essential to gain geometrical consistency. We also observe that the mixed training strategy and homography augmentation greatly improve all metries, including semantic consistency, further verifying their purpose of closing the gap between static training scenes and desired dynamic outputs. We believe there are two reasons why our full model outperforms the model trained on RealEstate [68]. The first reason is our epipolar attention design. In the Vid 10M [1] training stage, while there are no camera poses available, we use pseudo-gt epipolar lines (i.e. lines calculated from homography matrix H. The line of pixel in the warped frame goes through the pixel Hx) to describe the spatial relationship between video frames. This enhances the model's 's ability to generate videos that satisfy the given line conditions. Hence, in camera-contro setting, the full model is more constrained to the epipolar lines and generates videos that align better with the camera poses. Secondly, since ReallEstaeeOOK [68] mostly consists of static indoor scenes, models trained on RealFistate 10K [68] may suffer from data bias and may not perform well on general scenes, thus resulting in poor evaluation performance in this experiment.
6 Discussion
We introduce CVD, a novel framework facilitating collaborative video generation. It ensures seamless information exchange between video instances, synchronizing content and dynamics. Additionally, CVD offers camera customization for comprehensive scene capture with multiple cameras. The core innovation of CVD is its utilization of epipolar geometry, derived from reconstruction pipelines, as a constraint. This geometric framework fine-tunes a pre-trained video diffusion model. The training process is enhanced by integrating dynamic, single-view, in-the-wild videos to maintain diverse range of motion patterns. During inference, CVD employs multi-view sampling strategy to facilitate efficient information sharing across videos, resulting in a "collaborative diffusion' " effect for unified video output. To our knowledge, CVD represents the first approach to tackle the complexities of

---

trajectories
trajectories
high-ddh mamure, and through seraat haphnnos
Figure 5: Multi-view Video Generation. Left: The cameras move towards 4 directions, while all cameras looking at the same 3D point; Right: The trajectories interpolated from one trajectory (Ist Row) to another (4th Row).
multi-view multi-trajecton video synthesis. significantly advances beyond existing multi-view image generation technologies, such Zero123 [35], by also ensuring consistent dynamics across all videos produced. This breakthrough marks critical development in video synthesis, promising new capabilities and applications.
6.1
Limitations
CVD faces certain limitations. Primarily, the effectiveness of CVD inherently linked to the performance of its base models, AnimateDiff [17] and CameraCtrl [18]. While CVD strives to facilitate robust information exchange across videos, does not inherently solve the challenge of internal consistency within individual videos. As result, issues such uncanny shape shifting and dynamic inconsistencies that are presented in the base models may persist, affecting the overall consistency across the video outputs. Additionally, it cannot synthesize videos in real time, owing to the computationally intensive nature diffusion models. Nevertheless, the field of diffusion model optimization is rapidly evolving, and forthcoming advancements are likely to enhance the efficiency of CVD significantly.
6.2 Broader Impacts
Our approach represents a significant advancement in multi-camera video synthesis, with wide- ranging implications for industries such filmmaking and content creation. However, we are mindful of the potential misuse, particularly in creating deceptive content like deepfakes. We categorically oppose the exploitation of our methodology for any purposes that infringe upon ethical standards or privacy rights. To counteract the risks associated with such misuse, we advocate for the continuous development and improvement deepfake detection technologies.
Acknowledgement
This project was partly supported by Google and Samsung.
10

---

References
[1] M. Bain, A. Nagrani, G. Varol, and A. Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021.
[2] A. Bansal, H.-M. Chu, A. Schwarzschild, S. Sengupta, M. Goldblum, J. Geiping, and T. Goldstein. Universal guidance for diffusion models. In CVPR, 2023.
[3] O. Bar-Tal, Yariv, Y. Lipman, and T. Dekel. Multidiffusion: Fusing diffiusion paths for controlled image
generation. In Xiv, 2023.
M. Biríkowski, D. J. Sutherland, M. Arbel, and A. Gretton. Demystifying mmd gans. In arXiv, 2018.
[5] A. Blattmann, T. Dockhom, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti,
A. Letts, V. Jampani, and R. Rombach. Stable vidco diffusion: Scaling latent video diftusion models to large datasets. In arXiv, 2023.
[6] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align your latents: High-rcsolution video synthesis with latent diffusion models. In CVPR, 2023.
[7] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, Taylor, T. Luhman, E. Luhman,
C. Ng. R. Wang, and A. Ramesh. Vidco generation models as world simulators. 2024.
[8] S. Cai. D. Ceylan, M. Gadelha. C.-HI Huang, T. Wang, and G. Wetzstein. Generative rendering: Control-
lable 4d-guided video generation with 2d diffusion models, In CVPR, 2024.
[9] S. Cai, E. R. Chan, S. Peng, M. Shahbazi, A. Obukhov, L. Van Gool, and G. Wetzstein. Diffdreamer:
Towards consistent unsupervised single-view scene extrapolation with conditional diffusson models. In ICCV, 2023.
[10] D. Ceylan, C.-II Huang, and N. J. Mitra. Pix2video: Video editing using image diffuusion. In ICCV, 2023.
[11] E. R. Chan, K. Nagano, M. A. Chan, A. W. Bergman, J. Park, A. Levy, M. Aittala, S. D. Mello, T. Karras, and G. Wetzstein, GeNVS: Generative novel view synthesis with 3D-aware diffusion models. In ICCV, 2023.
[12] P. Esser, J. Chiu, P. Atighchchian, J. Granskog, and A. Germanidis. Structure and content-guided vidco synthesis with diffusion models. In ICCV, 2023.
[13] M. A. Fischler and R. C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6))381-395, 1981. [14] R. Fridman, A. Abecasis, Kasten, and T. Dekel. Scenescape: Text-driven consistent scene generation. In arXiv, 2023.
[15] S. Ge, S. Nah, G. Liu, T. Poon, A. Tao, B. Catanzaro, D. Jacobs, J.-B. Huang, M.-Y. Liu, and Y. Balaji. Preserve your own correlation: noise prior for video diffusion models. In ICCV, 2023.
[16] Y. Guo, C. Yang. A. Rao, M. Agrawala, D. Lin, and B. Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. In arXiv, 2023.
[17] Y. Guo, C. Yang, A. Rao, Y. Wang. Y. Qiao, D. Lin, and B. Dai. Animatediff: Animate your personalized
text-to-image diffusion models without specific tuning. In arXiv, 2023.
[18] H. He, Y. Xu, Y. Guo, G. Wetzstein, B. Dai, H. Li, and C. Yang. Cameractrl: Enabling camera control for text-to-video generation. In arXiv,, 2024.
[19] Y. He, T. Yang. Y. Zhang, Y. Shan, and Q Chen. Latent video diffusion models for figh-fidelity long video generation. In arviv, 2022.
[20] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NeurIPS, 2017.
[21] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans. Imagen video: High definition video generation with diffusion models. In arXiv, 2022.
[22] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NewrlPS, 2020.
[23] Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models. In arXiv,
2022.
[24] L. Höllein, A. Cao, A. Owens, Johnson, and M. NicBner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. /CCV, 2023.
[25] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language modcls. In ICLR, 2022.
[26] L. Hu, X. Gao, P. Zhang, K. Sun, B. Zhang, and L. Bo. Animate anyone: Consistent and controllable
image-to-video synthesis for character animation. In arXiv, 2023.
[27] R. Kapon, G. Tevet, D. Cohen-Or, and A. H. Bermano. Mas: Multi-view ancestral sampling for 3d motion
generation using 2d diffusion. In CVPR, 2024.
[28] T. Karras, M. Aittala, T. Aila, and S. Laine. Flucidating the design space of diffusion-hased generative models. In NeurlPS, 2022.
[29] I.. Khachatryan, A. Movsisyan, V. Tadevosyan, R. Henschel, 7.. Wang, Navasardyan, and H. Shi. Text2viden-zero: Text-to-image diffusion models are zero-shot video generators. In arXiv, 2023.
[30] D. P. Kingma and J. Ba. Adam: method for stochastic optimization. In ICLR, 2015.
11

---

[31] J. Li. H. Tan, K. Zhang, Z. Xu. F. Luan, Y. Xu, Y. Hong, K. Sunkavalli, G. Shakhnarovich, and S. Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In arXiv, 2023.
[32] X. Li, W. Chu, Y. Wu, W. Yuan, F. Liu, Q. Zhang, F. Li, H. Feng. E. Ding, and J. Wang. Videogen: A
reference-guidee latent diffusion approach for high definition text-to-video generation. In arXiv, 2023.
[33] Z. Li, Q. Wang, N. Snavely, and A. Kanazawa, Infinitenature-zent Learning perpetual view generation of natural scenes from single images. In ECCV, 2022.
[34] A. Liu, R. Tucker, V. Jampani, A. Makadia, N. Snavely, and A. Kanazawa, Infinite nature: Perpetual view generation of natural scenes from a single image. In ICCV, 2021.
[35] R. Liu, R. Wu, B. V. Hoorick, P. Tokmakov, S. 7akharov, and C. Vondrick. Zero-1-tod Zero-shot one image to 3d object. In arXiv, 2023.
[36] Y. Liu, C. Lin, 7. Zeng, X. Long, I. Liu, T. Komura, and W. Wang. Synedreamer: Generating multiview-
consistent images from a single-view image. In arXiv, 2023.
[37] X. Long. Y.-. Guo. C. Lin, Y. Liu. Z. Dou, L. Liu, Y. Ma, S.-H. Zhang, M. Habermann, C. Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In arXiv, 2023.
[38] A. Obukhov, M. Seitzer, P.-W. Wu, S. Zhydenko. J. Kyl. and E. Y.J. Lin. High-fidelity performance metrics for generative models in pytorch, 2020.
[39] J. Plücker. Anddyiishh-Geometrisst Entwicklungen. GD Baedeker, 1828.
[40] R. Po, W. Yifan, V. Golyanik, K. Aberman, J. T. Barron, A. H. Bermano, E. R. Chan, T. Dekel, A. Holynski,
A. Kanazawa, et al. State of the art on diffusion models for visual computing. In arXiv, 2023, [41] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language
supervision. In CoRR, 2021.
[42] R. Raguram, J.-M. Frahm, and M. Pollefeys. A comparative analysis of ransac techniques leading to adaptive real-time random sample consensus. In ECCV, 2008.
[43] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. In arXiv, 2022.
[44] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[45] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning
text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
[46] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S.
Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi. Photorcalistic text-to-image diffusion models with deep language understanding, In arXiv, 2022.
[47] P.-E. Sarlin, D. DeTone, T. Malisiewiez, and A. Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In CVPR, 2020.
[48] R. Shao, Y. Pang. Z. Zheng, J. Sun, and Y. Liu. Human4dit: 360-degree human video generation with 4d diffusion transformer. In SIGGRAPH Asia, 2024.
[49] R. Shi, H. Chen, Z. Zhang, M. Liu, C. Xu, X. Wei, L. Chen, C. Zeng, and H. Su. Zero123++: a single image to consistent multi-view diffusion base model. In arXiv, 2023.
[50] U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni. D. Parikh, S. Gupta, and Y. Taigman. Make-a-video: Text--o-video generation without text-video data. In arXiv, 2022.
[51] J. Song, C. Meng. and S. Ermon. Denoising diffusion implicit models. In ICLR, 2020.
[52] Y. Song, J. Sohl-Dickstein, D. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2020.
[53] J. Suyi. L. Haimin, J. Haoran, W. Ziyu, Y. Jingyi, and X. Lan. Mvhuman: Tailoring 2d diffusion with multi-view sampling for realistic 3d human generation. In arXiv, 2024.
[54] J. Tang, T. Wang. B. Zhang. T. Zhang. R. Yi, L. Ma, and D. Chen. Make-it-3d: High-fidelity 3d creation from single image with diffusion prior. In ICCV, 2023.
[55] A. Tewari, T. Yin, G. Cazenavette, S. Rezchikov, J. B. Tenenbaum, F. Durand, W. T. Freeman, and V. Sitz- mann. Diffusion with forward models: Solving stochastic inverse problems without direct supervision. In
arXiv, 2023.
[56] H.-Y. Tseng, Q. Li, C. Kim, S. Alsisan, J.-B. Huang, and J. Kopf. Consistent view synthesis with pose-guided diffusion models. In CVPR, 2023.
[57] T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. FVD: A new metric for vidco generation. In ICLR Workshop. 2019.
[58] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I.. Jones, A. N. Gomez, F.. Kaiser, and I. Polosukhin.
Attention is all you need. In NeurlPS, 2017.
[59] X. Wang, H. Yuan. S. Zhang, D. Chen, J. Wang, Y. Zhang, Y. Shen, D. Zhao, and J. Zhou. Videocomposer: Compositional video synthesis with motion controllability. In NeurlPS, 2023.
[60] Z. Wang. Z. Yuan, X. Wang. T. Chen, M. Xia, P. Luo, and Y. Shan. Motionctrl: A unified and flexible motion controller for video generation. In arXiv, 2023.
12

---

[61] J. Z. Wu, Y. Ge, X. Wang. S. W. Lei, Y. Gu, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou. Tune-a-video:
One-shot tuning of image difflusion models for text-to-video generation. In arXiv, 2022.
[62] Y. Xu, H. Tan, F. Luan, S. Bi, P. Wang. Li. Z. Shi, K. Sunkavalli, G. Wetzstein, Z. Xu, and K. Zhang.
Dmv3d: Denoising multi-view diffusson using 3d large reconstruction model. In arXiv, 2023.
[63] S. Yin, C. Wu, Liang. Shi, H. Li. G. Ming. and Duan. Dragnuwa: Fine-grained control in video
generation by integrating text, image, and trajectory. In arXiv, 2023.
[64] S. Yu, K. Sohn, S. Kim, and J. Shin. Video probabilistic diffusion models in projected latent space. In
CVPR, 2023.
[65]
1.. Zhang, A. Rao, and M. Agrawala, Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[66]
Q. Zhang, J. Song, X. Huang, Y. Chen, and M. yu Liu. Diffcollage: Parallel generation of large content with diffusion models. In CVPR, 2023.
[67] R. Zhao, Y. Gu, Wu, Zhang. Liu, Wu, J. Keppo, and M. Z. Shou. Motiondirector. Motion customization of text-to-video diffusion models. In arXiv, 2023.
[68] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely. Stereo magnification: Learning view synthesis using multiplane images. SIGGRAPH, 2018.
13

---

A Appendix supplemental material
A.1 More Analysis on Collaborative Video Generation.
For simplicity, the video conditions omitted in this section without loss of generality. In the paper. we describe the collaborative video diffusion model as a multivariate denoising function po(o, (o......A)) that estimates the real distribution q<vl_1 |07 ?o?.....MM % "......)) Following Ho et.al. [22], the problem be transformed into the optimizing a noise prediction network E c (v)"...M) predict -I-a;" 24 1. /0,
Vi-a. 1o ....MM On the other hand, Song et.al. [52] demonstrated the relation between noise prediction and the score function so A........, .1) R V log ((p)....M))) is:
M .1) so(n.r.. -]= = 60 Eo( M v1 ae
As discussed in the paper, directly training sp( No t) for arbitrary M is intractable due the lack of multi-view datasets, so we reduce the problem into generating video pairs (M = 2) instead. Specifically, we train score function sp(v()) using video pair datasets and apply it to infer all M vidcos. Our collaborative video score function, denoted as SCVD(V) (ul..,,M) ), is defined as:
M
SCVD( (c)(( = Σ
As so(("))) + w se( w
(6)
(UJ)P
where the of all selected video pairs, and so (v ( represents the i'th video component of the score function. Note that the order of i is irrelevant.nn In essence. we utilize the weighted sum of video pair score functions to depict the score function of all videos. We demonstrate that the defined score function ScvD( can estimate the real score function V log g( (( only if (..))PP wh = 1 for all i Elliel,.... I,..,
Lemma A.1. Let S be a subset of {L,... M), and q(v)) be the density function of set of video features viS {wf lk E 8} derived from the forward diffusion process, that is, q( vg)) (Vare) - -vf), where 1(k E S)
N(vi: Varug (1 a.) Then Va log q(v;") vg) = L(kES)
I-a:
equals to lif E S and 0 otherwise.
Proof.
V. log logg(v;(oo)) =V,; log(N(u, varug (1 a))
(7)
=V (v. v Varu)
2(1 - (xt))
1(k E S)
(V / v.
(9)
1 a,
Lemma A.2 (Updated Tweedies's Formula). Let S be a subset of (1,..., M) }. and q( be the density function of a set of video features vi = fo/l E S\ derived from the forward diffusion process, then V ot log g( 1(kE8) (Vo, (utbo)) ot))
14

---

Proof.
V Voig(ns)
oo log q(,))
(10)
{pp))
V Eug (1(1))
(11)
(1,))
og (o:
(12)
/
alvi) V ( ug)) dug
(13)
q(vi)
/ aavg o ) Vo log g(v (o5) duy (Bayes' Theorem)
(14)
/ ( p0))
1 E 5)
1- āi (Vāreg - =2,) dog (Lemma. A.1)
(15)
1 E S) (Vār / (108810 dug - ot))
(16)
1- at
1(k: E S) (Vār
+t)
(17)
1- Ōt
1(k E S)
(Vā.)
( 5/R No. v drg - v())
(18)
1- āt
1(k E S) (Vā.)
( / aiv aivgg///v Ivo v2) dr dv vt))
(19)
1 1- ai
1(k E S)
(Vā.
9 dv. -())
(00)
1 1-
1(k (( (Vass
((v0l1,) 11(00 5/k lo uz doy -0,)
(2))
1- -tt
1(k [() (Vorh
((obll))) / (1100 8/ 100 vz)) dry g du vt
(22)
1 at
1(k E S)
(Vās / ( ((vtllz))e jeğ duž ot
(23)
1 at
1 1(k E S)
(V aa E, - 03)
(24)
1-a:
Theorem A.3. The function (c:
M) can be unbiased approximation of the real score function V log (.M, for all timesteps E I,...,T, T, only if E(.)EE w is forr all i 1..., M.
Proof. For any E 1,.,, M, the k'th component of sCvD( RevD(o!....M MM can be written as:
ScvD( (u) M )R
(25)
-( Σ
w )o(00))) + w. ()())))))e
(26)
(1.)EP
Σ
w
kit
(0(D )x
(27)
(k.j)P
Σ (KJ))P
wk
V,, n² log ())))) (Score Matching)
(28)
Σ
u Aad (Vor (o( - ut) (Lemma. A.2)
(29)
1 āt (k.j)EP
15

---

To unbiasedly estimate V xx log q(. = I-m ( ( w. vf¹ from Eq. 29 w.r.t. all l and of. there must be Ea.eer wk
Rij of = vf. which means E(Kj)eP Wk kij
=
In addition, we can observe that the accuracy of the estimation from Eq. 29 heavily relies on the similarity between [(k.jee Wk RJ (ufi oy and (u That means, when we apply a denoising step to a noisy input 8, ........M the output w/-1 is more likely to align with the true distribution if the prediction of v from o. resembles the prediction of et from all o, u........ We think this is fairly reasonable in the context of consistent camera-controlled video generation, as the underlying geometry of captured videos can often be discerned from just few views. We believe this is the key reason why our model can generate consistent multi-view videos trained from video pair data only.
A.2 Implementing Details
We built pipeline top of AnimateDiff [17], popular open-source T2V model that is widely used among artists. We additionally deploy CameraCtrl [18] to utilize its camera conditioning ability. Following this line of works, we benefit from their plug-and-play property and swap base model with fine-tuned version, e.g., via DreamBooth [45] or LORA [25].
A.2.1 Training
We select 65,000 videos from RealEstate 10K [68] and 2,400,000 videos from WebVid IOM [1] to train model. Each data point consists of two videos of 16 frames and their corresponding camera extrinsic and intrinsic parameters. For RealEstate we randomly sample a 31 -frame clip from the original video and split it into two videos using the method described in the paper. For WebVid10M, we sample a 16-frame clip, duplicate it to create two videos, and then apply random homography deformations the second video. The homography transformation matrix H H, H, H Hp is defined as the composition of series of transformations: translation, rotation, scaling, shearing, and projection, where:
to
Cos
sin ( cos
0
H, =
0
H
sin 0)
(0)
0
1
1
(30)
1 + 80
0
0
1
sho
0
1
0
H, =
1 1+88 81
Hsh =
sh1
H
=
0
0
0
1
0 0
1
Po
P1
1
are transformation matrices parameterized by controlling vectors 0, sh, p. We aim for the first frame of the deformed video to remain unchanged, with the deformation gradually increasing in subsequent frames. To achieve this, we randomly sample the controlling vectors for the last frame from normal distributions. Then, we interpolate these vectors from 0 to the sampled values to obtain the vectors for each intermediate frame and calculate the corresponding matrices.
Following [18], we use the Adam optimizer [30] with learning rate 1e - 4. During training, we freeze the vanilla parameters from backbones and optimize only our newly injected layers. We mix the data points from RealEstate and WebVidl under the ratio of 7 3 and train the model two phases alternatively. All models trained on 8 NVIDIA A100 GPUs for 100k iterations using an effective batch size 8. The training takes approximately 30 hours.
A.2.2 Inference
We use DDIM [51] scheduler with 1000 steps during training and 25 steps during inference. Assuming
k.j independent with j. we have w = wy = パ.JাリP. Our algorithm shown in Alg. 1. We use the partitioning strategy in all of our experiments. For 2-view (video pair) results, we use R = (no recurrent denoising) and P = 1; For 4-view results, we use R = 4, P and for 6-view results, we use R = 6, P = 2. We demonstrate multi-view video generation results in our supplementary videos. Additionally, we show three potential applications of our algorithm: long looping vidcos, view switching, and potential 3D generation.
16

---

Algorithm 1: Algorithm for arbitrary number of videos generation
Parameter: Denoising steps T, recurrent steps R, video number M, noise scheduling
parameters (az TT pair selecting strategy Sig E (Exhaustive, Partition), partition
number Q
Input: V7 ".r..M M sampled from (N I), video pair diffusion model eæ, text prompt y, camera
trajectories cam......M
for t = T.T 1..,. do
I.....M t- 0:
Cout
for r = 0, 0..,,...
do if Sig is Exhaustive then
- {(,,) i,j E 1, M]
Selecting all pairs */
denom t M -
else
+ (): for q = 0, 1, 1.... ......... Q 1
P.Extend (axtenn(Randoomaiiraaatt 2., M))
end
denom + Q: end
for
(i,j) E P do
Cuul - c + ei( ( t, cam
Eont Eot egg (v) ,t,,tt,o cam"
end
NoiseSchedule( 1....MM (denm///enmnn u...... ,t);
+ then
è' ~ (), Munn + VI az/ār_1é'; U - v 10//0-111
Renoise */
end
end
end
A.3 Results of Attention Maps
We show an exemplar visualization of our epipolar-based attention in Fig. 6, where we take the highlighted pixel from the left image, and visualize its corresponding attention probability after softmax. We can observe that information is taken from the second image according to the epipolar line, and specifically, the corresponding region is being attended to.
A.4 Performances with identical camera trajectories
In Fig. 7, show that our model generate identical videos if the input camera trajectories identical, while none of the prior works communicates cross-videos, hence incapable of generating identical contents. Quantitatively. our model reaches an MSE of 0.01, significantly outperforming CameraCtrl at 0.07 and CameraCCnI+Sparseee when the camera trajectory pair and prompt are identical in Fig. 8
at 0.06. We show more realizations of our model
A.5 Additional results for LoRA fine-tuned models
Our model exhibits strong plug-and-play properties and can directly generalize to different fine-tuned models, e.g., using Dreambooth [45] or LoRA [25]. We show few rendering results Fig. For better illustration, please refer to supplemental vidco.
17

---

Attention Weights
prompt: A fish swimming aquarium tank.
prompt: Nest singlt archteerc,,,,,eur,, blue_sky, cloudy_syy day, fantasy, fence, field,
Figure 6: Exemplar visualization of epipolar-attention map.
CameraCri
trajectories
Ours
prompt: forbidden castle high up in the mountains, pixel art, game key visual, surreal.
prompt: A fish is swimming in the aquarium tank.
Figure 7: Qualitative comparison with our baselines where the two camera trajectories are identical.
A.6 More Qualitative Results
Figures 10, 11, 12, 13 and 14 show more qualitative results, where we generate video pairs with different realizations and camera trajectories for each prompt. Please refer to our supplementary video for better illustrations.
A.7 Homography warping Visualization
In Fig. 15, we show visualization results of our homography warping augmentation applied in our Ved [1] phase. During our training, we removed the L2 loss the potentially unseen pixels (black regions) of the cloned video for data integrity.
18

---

Figure 8: Qualitative results where the two camera trajectories are identical.
19

---

RealisticVision
trajectories
prompt: b&w photo of 42 y.o man in black clothes, Fujifilm XT3.
Lyriet
trajectories
prompt: dark shot, epic realistic, portrait of hato, sunglasses, blue eyes, tartan scarf. .... white hair.
Toonyou
trajectories
prompt: masterpiece, best quality, Igirl, solo, cherry blossoms, hanami, ..., pink flower.
Majicmix Realistic
trajectories
prompt: best quality, masterpiece, photorealistic, 1boy, 50 years beard, dramatic lighting.
Figure 9: Exemplar outputs from Dreambooth [45]/LoRA [25] fine-tuned models.
20

---

trajectories
prompt: fish swimming in the aquarium tank.
Figure 10: Additional Qualitative Results with different camera trajectories and realizations.
21

---

trajectories
prompt: aerial view of cyberpunk city, night time, lights, masterpiece, high quality.
Figure 11: Additional Qualitative Results with different camera trajectories and realizations.
22

---

trajectories
prompt: A forobidden castle high up in mountains, pixel art, game art, key visual, surreal.
Figure 12: Additional Qualitative Results with different camera trajectories and realizations.
23

---

trajectories
prompt: photo of coastline, rocks, storm weather, wind, waves, lightning, 8k uhd, dsir, soft lighting, high quality, film grain, Pujifilm XT3.
Figure 13: Additional Qualitative Results with different camera trajectories and realizations.
24

---

trajectories
prompt: fireworks display illuminating the night sky.
Figure 14: Additional Qualitative Results with different camera trajectories and realizations.
25

---

Original Views
#1
Warped Views
Original Views
Clip #2
Warped Views
Frums #1
Frame m4
Frame 88
Frume #12
Frame #16
Visualization of homography warping for WebVid10M data. For each video clip, the top row represents the original frames from the video, and the bottom row represents the frames warped by homography transformations.
Figure 15:
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract and Sec. 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Sec. 6.1
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
See Sec. A.1.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Sec. 5 and Sec. A.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer:
[Yes]
Justification: We use public datasets and data processing details are provided in Sec. 4.2 and Sec. A.2.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
Seteii/guiidee/oeeuuiiiiee soioooicsin for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
CoodeSubmiis ssi wwwinnossnoiiiiioooooooooooioe oniioiiyl) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: See Sec. A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Our evaluation uses protocols from previous works which do not use error
bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Sec. A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpas/////ui.../eeeeee/ee httppppp.ccpppp/pateeeeee httpppuicc/thhhcccueeleeee..eeee ctw.idheellellleeccccccc ines?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Sec. 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited all works of codes and datasets used in the paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not conduct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "ozi1DwfbXU", "reviewer": "Reviewer_3sRf", "summary": "The paper proposes a novel framework that generates multi-view videos from text input. The model builds upon the CameraCtrl pipeline and includes a Cross-View Synchronization Module to enforce consistency guided by a fundamental matrix.", "questions": "- How is the 3D reconstruction (shown in supp_video.mp4) obtained using the generated multi-view videos?\n- How is Table 3 obtained? Why does training on WebVid10M (no camera pose) improve Rot. AUC Trans. AUC (camera accuracy) as compared to training on RE10K only? I'm confused by the results in Table 3.", "limitations": "Please refer to the weakness.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The performance of the proposed method is impressive, generating multi-view videos from text input.\n- The framework leverages both static videos and dynamic videos for joint training.", "weaknesses": "- Random homography transformations are applied to the clones for WebVid videos. It would be great if examples of the augmented clones were visualized. I'm mainly worried that the augmented clones seem to introduce a lot of black (unknown) regions and how much it affects the performance of video generation.\n- Table 2 should include video-based metrics, such as FVD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response of Reviewer 3sRF’s Review\n\n## Q1: The black regions might affect the performance of video generation.\n\nDuring our training, we removed the L2 loss in the unseen pixels (black regions) of the cloned video for data integrity. We show examples of homography warped videos in our attached PDF (Fig. A2) and will clarify the loss in black regions in our revision.\n\n## Q2: Add FVD metric evaluation.\n\nThanks for pointing this out. We evaluate our method and all baselines using the FVD metric, and here are the results:\n- CameraCtrl: 277\n- AnimateDiff+SparseCtrl: 327\n- CameraCtrl+SparseCtrl: 430\n- Ours: 285\n\nSimilar to our experiments on the FID metric, our method is on par with CameraCtrl and performs better than other baselines. \n\n## Q3: How is the 3D reconstruction obtained using the generated multi-view videos?\n\nWe use NeRFactor [Tancik et al. 2023] to reconstruct 3D from our multi-view videos. We will clarify more details in the revised version.\n\n## Q4: How is Tab.3 obtained? Why does training on WebVid10M improve Rot. AUC Trans. AUC as compared to training on RealEstate10K only?\n\nThis is an insightful question! For the evaluation results in Tab.3, each model generates video pairs given the same text prompt and camera trajectories. For each video pair, we calculate the SuplerGlue [Sarlin et al. 2020] matches between each frame pair (i.e. two frames captured at the same time from the video pair). Then, we run RANSAC on these matches to calculate the relative camera poses between the two frames in each frame pair and compare the camera poses with the ground truth to get the AUC score.  \n\nWe believe there are two reasons why our full model outperforms the model trained on RealEstate10K. Firstly, we think the credit goes to our epipolar attention. In the WebVid10M training stage, while there are no camera poses available, we managed to use pseudo-gt epipolar lines (i.e. lines calculated from homography matrix H. The line of pixel x in the warped frame goes through the pixel Hx) to describe the spatial relationship between video frames. This enhances the model’s ability to generate videos that satisfy the given line conditions. Hence, in a camera-control setting, the full model is more constrained to the epipolar lines and generates videos that align better with the camera poses. Secondly, since RealEstate10K mostly consists of static indoor scenes, models trained on RealEstate10K may suffer from data bias and may not perform well on general scenes,  thus resulting in poor evaluation performance in this experiment. We will clarify this in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bE1pgAtryn", "reviewer": "Reviewer_WFsL", "summary": "This paper proposes a diffusion-based video generation method that generates multiple videos of the same scene simultaneously from camera trajectories and a text prompt. A cross-video synchronization module is proposed, where epipolar attention is introduced to improve the consistency across multiple videos. Experimental results show that the proposed method outperforms state-of-the-art approaches, especially in cross-video geometric consistency and semantic consistency.", "questions": "Please refer to my comments above", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a new task that is to simultaneously generate videos of the same scene given multiple camera trajectories.\n\n2. The paper is well-organized and easy to follow.\n\n3.  A cross-video synchronization module is proposed to ensure cross-video geometric consistency.", "weaknesses": "1. When the proposed model is trained on different datasets (RE10K LoRA, CameraCtrl Pose LoRA, WV10M LoRA), different LoRA modules are used in the two-phase hybrid training. How are these different LoRA modules used in the final inference step? Are they used together, or is only one LoRA module chosen according to some settings?\n\n2. Eq. 4 is confusing. The variables of Eq. 4 are x_i and x_2. However, x_i is not used on the right side of Eq. 4.\n\n3. It's interesting to exploit the epipolar geometry to ensure cross-video geometric consistency. However, some technical details are unclear in Sec. 4.1. It is not very clear how the epipolar geometry is used to generate the attention mask M in Eq. 4.  It would be better if the authors provided more details and more explanations. Otherwise, it is difficult to reproduce the proposed method.\n\n4. The attention mask is additionally introduced in the cross-view synchronization module. Yet, it is confusing why this mask is being introduced. Does the attention mask indicate correspondences between frames  of different videos?  It would better to provide some statistical results (average percentage ) for the attention mask. \n\n5. Given a video diffusion model perfectly ensures temporal coherence and has excellent camera control performance， is it still necessary to generate multiple videos simultaneously? Can we use the video diffusion model to generate one video at a time instead?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response of Reviewer WFsL’s Review\n\n## Q1: How are the LoRA modules used in the inference step?\n\nAs our model inherits the ability from AnimateDiff to generalize on different LoRAs, during inference we select the LoRA that best satisfies the needs in different settings. For the quantitative comparisons, we use LoRA trained on RealEstate10K (provided by CameraCtrl) for RealEstate10K scene generation and LoRA trained on WebVid10M (provided by AnimateDiff) for WebVid10M scene generation; For qualitative comparisons, without specification, we use the RealisticVision LoRA for its superior performance in generating high-quality images and videos. LoRA from CameraCtrl for temporal layers is applied in all experiments. \n\n## Q2: Wrong notation in Eq. 4. (x_i should be x_1).\n\nThanks for pointing this out. We will fix this typo in our revision.\n\n## Q3: Clarification on how the attention masks are generated and used.\n\nOur attention masks are added as an inductive bias to the self-attention modules, encouraging them to extract information from the epipolar geometry. For this reason, our attention masks function as a 0/1 masking on the full self-attention score, where only the pixels on the corresponding epipolar lines have attention weights, as shown in Fig.6 of our supplementary. This process does not directly indicate correspondences (since we never possess ground truth pixel correspondences during training and inference) but rather introduces the epipolar line inductive bias about where to find the correspondences. As shown in our ablation study in Tab.3, having this inductive bias is very helpful in terms of aligning the content across the two frames. Empirically, we find the hit rate of a pixel’s corresponding pixels being picked up by the attention module (at some diffusion step) close to 100%.\n\n## Q4: Comparison between our model and running monocular video diffusion model multiple times.\n\nIndeed, running a powerful monocular VDM multiple times with the same condition (ideally the same keyframes) is a rather reasonable method for multi-view video generation. Yet, this approach faces a major issue: It will suffer from generating inconsistent motion and different content that is unseen in the keyframes. The reason is that monocular VDMs are not trained to introduce consistency across views. For example, if we let a monocular VDM generate multiple firework videos given the same condition (text prompt or starting frame), it will very likely generate different firework patterns in each pass. Our model instead jointly predicts all videos together with their information being shared in the cross-view modules, thus it can produce more view-consistent results. In our experiments (Tab.2), we compare our model with the baseline “CameraCtrl+SparseCtrl”, which can be considered as the monocular VDM approach. Our model outperforms the latter by a large margin.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2QCkoX3GLy", "reviewer": "Reviewer_MCcD", "summary": "The paper introduces a framework to generate consistent multi-view videos of the same scene. Existing models lack precise camera control and consistency across views. The proposed CVD framework uses a cross-video synchronization module with an epipolar attention mechanism to maintain frame consistency from different camera trajectories. Trained with a combination of static multi-view and dynamic monocular data, CVD outperforms existing methods in generating consistent multi-view videos, as claimed by the authors.", "questions": "See weakness above.\n\nJustification for rating:\n\nThis paper proposes a new pipeline for a relatively new field, i.e., collaborative video diffusion for camera-controllable multi-view video generation. The results provided by the authors show some improvements compared to the baseline CameraCtrl. However, due to the questions and limitations I raised above, I give a slightly positive score here.", "limitations": "See above.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This work explores trajectory controllable multi-view video generation, while many of the existing works solely consider single trajectory generation. The research problem is a new setup.\n- The authors propose a two-stage training strategy to learn multi-view camera motion in the real world. Though this may have some limitations, this is a new practical solution since there is a lack of large-scale multi-view video datasets with camera poses for dynamic scenes in the community.", "weaknesses": "- In Eq.(4), what does the x_i represent? Should it be x_1 here? I guess this is a typo. I also don’t find the notation illustration for x, which makes it hard to understand the purpose of using Eq.(4) to generate the attention mask. I suggest the authors to provide more details on this.\n- For WebVid10M video dataset, the authors use homography warping for the training video generation. However, warping has many limitations. Technically, it only deals with planar points. Applying it to all video frames for “fake new view synthesis” will suffer from lost depth, perspective distortion, and incorrect parallax. Also, homography warping makes it hard to simulate new views from large rotations and translations. The visualized examples provided in the material also don’t show large camera motions. I notice that concurrent works usually construct a “real” multi-view dataset consisting of videos rendered from multi-view from 3D objects. Actually, the authors have a similar solution by re-organizing the videos from the multiview video dataset RealEstate10K and proposing the two-stage training, but the camera motion is still limited. Do the authors have some comments on this?\n- The authors mentioned that “For the RealEstate10K dataset, we use CameraCtrl with LoRA fine-tuned on RealEstate10K as the backbone and applying the ground truth epipolar geometry in the cross-video module. For the WebVid10M dataset, we use AnimateDiff with LoRA fine-tuned on WebVid10M as the backbone, and apply the pseudo epipolar geometry (the same strategy used for the first frames in RealEstate10K dataset) in the cross-video module.” It seems that the authors are applying different model structures to train the model for the two datasets. In the inference, how do we combine these two separate models into one? Since the first one is supposed to learn static multiview geometry transform, the other one is supposed to learn more about video dynamics like effect or object motion. How do the learned priors combine together with consistency?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response of Reviewer MCcD’s Review\n\n## Q1: Clarification of Eq.(4) (how the attention masks are generated).\n\nThanks for pointing this out. The attention masks are calculated as follows: For each pair of pixels coordinated at x_1 and x_2 in two frames, respectively, the attention mask from x_1 to x_2 is calculated by the epipolar distance between x_1 and x_2 (i.e. the shortest distance between x_1 and the epipolar line of x_2 in x_1’s frame). We will fix the typo and add the clarification in our revision.\n\n## Q2: Limitations of homography warping and other possible dataset candidates.\n\nThanks for raising this important question. Technically, the homography warping phase is introduced to enhance the model’s capability to produce synchronized motion across multi-view videos, which is an action taken out of necessity due to the lack of large-scale generic 4D datasets. Indeed, as the reviewer suggested, homography warping has many downsides, such as distortion of shape, incorrect perspective, and lack of view-dependent appearance and depth information. However, despite the limitations, homography warping serves as a pseudo-bridge attempting to close the gap between the training of static RealEstate10K videos and dynamic WebVid10M videos. Further, we argue that the perspective issue can be addressed by the pseudo epipolar lines: They are similarly distorted as the warped geometry, hence canceling out its effect on the model. In other words, it pushes the video model to strictly follow the given epipolar lines, making it eventually generate decent videos when the lines become correct. The combination of homography warping and epipolar-based inductive bias both effectively improves the transferring of learned correspondence information between our two training stages, as indicated in Tab.3 in our supplementary.\n\nIn our work, we choose to follow CameraCtrl and MotionCtrl and use RealEstate10K, which has less diversity in camera trajectories since it only consists of monocular videos. But we believe our model can be further improved by integrating more datasets into the training, such as 3D/4D object datasets (Objaverse, OmniObject3D, MvImgNet), 3D landscape datasets (MegaScene, Acid), and small-scale 4D scenes (Multi-view video datasets). On the other hand, there exist many real-world dynamic motions that are hard to capture in multi-view, such as the motion of fireworks, waves, or wild animals. In these cases, monocular videos are still the only available large-scale resources. We believe there is great potential for exploring future works based on our method with additional datasets, and will add more discussions in the limitation section of our paper.\n\n## Q3: How do you combine the models trained in different phases? \n\nAs our model inherits the ability from AnimateDiff to generalize on different LoRAs, during inference we select the LoRA that best satisfies the needs in different settings. For the quantitative comparisons, we use LoRA trained on RealEstate10K (provided by CameraCtrl) for RealEstate10K scene generation and LoRA trained on WebVid10M (provided by AnimateDiff) for WebVid10M scene generation; For qualitative comparisons, without specification, we use the RealisticVision LoRA for its superior performance in generating high-quality images and videos. On the other hand, LoRA from CameraCtrl for temporal layers is applied in all experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1yPD0FPHdh", "reviewer": "Reviewer_EujL", "summary": "This paper studies video generation with camera trajectories. The proposed method improves the consistency across multiple views via a cross-video synchronization module, which is equipped with the existing epipolar attention. Training of the proposed method consists of two phases: training to learn geometric consistency (3D) using ReadEstate10K and training with WebVid10M for learning motions. The authors utilized the augmentation methods such as video folding and homography augmentation for more effective learning addressing the scarcity of data. The proposed method achieves superior performance over baseline methods.", "questions": "- **Inference.** Multi-stage training has been proposed with/without additional parameters depending on datasets. It is not clear how to perform the inference. It needs a little bit more details.\n- **Multiple views beyond two views and section 4.3.** How do you generate videos of multiple views beyond two views and Section 4.3 is not clear. What do you mean by M video features? Are they from different networks?\n- **Editing.** Is it possible to apply the proposed pipeline to video editing? Given an input source video, is it possible to generate a new video with the same contents following two camera trajectories?", "limitations": "Limitations and Broader impacts are properly discussed in Section 6.1 and 6.2 The performance of the proposed method heavily relies on the performance of base video diffusion models. Also, because of the dependency, the proposed method cannot be applied to real-time applications, yet. Regarding negative societal impacts, the authors discussed deceptive content. The issue could be alleviated by better deepfake detectors.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Strong performance**: The proposed pipeline achieves a significant performance gain compared to baselines: CameraCtrl and MotionCtrl. The camera control in video diffusion models is a relatively new topic and less explored in the literature. In both metrics, such as geometry and semantic consistencies, the proposed method outperforms baselines by a significant margin.\n- **Comprehensive survey/related works**: This paper includes the most recent advances in video generation and discusses their strengths and weaknesses.\n- **Analysis**: The appendix provides comprehensive analyses. First, the visualization of epipolar attention is included in the appendix. Also, qualitative results/generated videos (mp4) were helpful in understanding the quality of the generated videos.", "weaknesses": "- **Ablation study**: The data augmentations, video folding and homography augmentation, are applied. These augmentations are crucial to improve performance. The data augmentation strategies may be effective for the baselines CameraCtrl and MotionCtrl. However, due to the lack of ablation studies, it is hard to evaluate the impact of these components.\n- **Lack of technical novelty**: Key value injection and epipolar attention have been proposed in previous works. Cross-view synchronization is merely a combination of existing techniques.\n- **Qualitative analysis of trajectories**: Camera control is a challenging problem, and existing methods actually generate seemingly okay videos. However, the generated videos, especially when trajectories are unseen/new, do not precisely follow the input trajectories. Beyond AUC, camera pose comparisons will be helpful for readers to understand the quality of generated videos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response of Reviewer EujL’s Review\n\n## Q1: Lack of ablation study.\n\nWe would like to emphasize that our cross-video augmentations are applied to the pairs of videos, which serve as the training input throughout our experiments; It is infeasible to apply such augmentations to monocular video generation methods such as MotionCtrl and CameraCtrl since they always only take one video as training input. To verify our design choices, including the effectiveness of our cross-video augmentations, we supply ablation study results on our method in Tab.3 in our supplementary, where our full model outperforms all ablative variants by a large margin, demonstrating the effectiveness of our design. For example, **our full model** achieves 25.2 scores at the Rot. AUC 5%, and scores of other variants are 16.8 for the **model without epipolar attention**; 17.9 for the **model without WebVid10M training**; 22.0 for the **model without homography transformation**; \n\n## Q2: Qualitative Analysis of camera trajectories.\n\nIn our paper, we show both quantitative and qualitative comparisons regarding camera trajectories. In Tab.1, the AUC scores evaluate the accuracy of camera poses optimized from our video predictions. In Fig.4, we show the results of our model and all baselines given the input camera trajectories (shown on the left). We think these experiments demonstrate the capability of our model to follow the input cameras. \n\n## Q3: More details for inference (how to generate multiple views).\n\nThanks for pointing this out. To generate more than two videos with our model, we designed a sampling algorithm as described in Section A.3.2. In general, the algorithm first initializes M noise maps from Gaussian (corresponding to the videos of M camera trajectories). In each denoising step, multiple pairs among the noise maps are selected, and the network is run on each selected pair for noise prediction. The predictions are then averaged over all pairs and applied to each noise map individually for denoising.\n\nDue to the page limit, we moved most of our inference details to our supplementary (Section A.3.2). If accepted, we will add more details to the main paper with the additional page quota.   \n\n## Q4: Is it possible to edit videos using our method (i.e., generate another video given a source video)?\n\nWe think this is a very interesting and doable direction. While our model is trained to generate two videos simultaneously, it can also be adapted into a model conditioned on a source video. To do that, we can train it with the same training dataset but feed the model with one video as a reference and let it predict the other one. \n\nWe also notice that a concurrent work from Hoorick et al. [1] also made a good attempt at this setting. We believe this demonstrates the great potential of multi-view video generation models to be applied in many applications in the future.\n\n## Q5: Novelty of our paper.\n\nWe believe our work has several significant distinctions from previous works. To our knowledge, we are the first work attempting the task of general multi-view video generation; We introduce the cross-view synchronization module (CVSM), which is inspired by previous works on epipolar attention, but has a very different approach aiming for the synchronization across videos in the task of multi-view video generation; And we propose a novel training-free algorithm to sample more consistent views from our model at inference.\n\n[1] Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis, Hoorick et al., 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Wv7dusUE1", "reviewer": "Reviewer_v3P6", "summary": "This paper tackles the problem of consistent multi-video generation, i.e., generating multiple videos capturing the same scene from various camera trajectories. For this, it proposes a cross-view synchronization module (CVD) based on the epipolar geometry. Training-wise, it proposes a hybrid training strategy, exploiting various video datasets. Quantitative and qualitative experiments demonstrate the effectiveness of the proposed approach for generating consistent videos.", "questions": "See \"weakness\".", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- originality-wise: the idea of utilizing epipolar geometry to encourage consistency across generated videos is interesting;\n- quality-wise: the provided qualitative results demonstrate high-quality consistency\n- clarity-wise: the paper is well-written and easy to follow. The derivation provided in the supplementary is clear.\n- significance-wise: the problem of generating consistent video content is vital for enabling fine-grained control for video generation", "weaknesses": "## 1. Clarifications about the training strategy\n\nI am confused about how the training strategy in Sec. 4.1 is applied to the trainable parameters, i.e., cross-view synchronization module (CVSM). To be specific, are CVSM blocks shown in Fig. 3 the same model? Namely, will the CVSM in Fig. 3.(b) initialize from the trained CVSM in Fig. 3.(a)?\n\n## 2. Problems in Derivation\n\nI think there are issues from Eq. (16) to (17) in Sec. A of the supplementary. Specifically, this step directly changes $q(v_0^S \\vert v_t^S)$ to $q(v_0^k \\vert v_t^S)$, which is not suitable.\n\nThe main concern I have is that $q(v_0^S \\vert v_t^S) \\neq \\prod_{k\\in S} q(v_0^k \\vert v_t^S)$ since the videos $\\\\{ v_0^k \\\\}_{k \\in S}$ captures the same scene and are not independent. \n\nPlease correct or clarify.\n\n## 3. Missing related works\n\nRecently, there have been several works tackling the problem of generating multi-view consistent content via manipulating the latent space, e..g, [a, b]. Please provide some discussions for comparison to such line of work.\n\n[a] Kapon et al., MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusion. CVPR 2024\n[b] Jiang et al., MVHuman: Tailoring 2D Diffusion with Multi-view Sampling For Realistic 3D\nHuman Generation. ArXiv 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response of Reviewer v3P6’s Review\n\n## Q1: Clarification of training Strategy.\n\nThe two training phases are applied to the same CVSM parameters in a single training pass. (The CVSM in Fig.3a and Fig.3b are the same modules.) Specifically, we blend the data from Webvid10M or RealEstate10K together. Then in each training iteration, we randomly pick one data batch from our hybrid dataset and train our model in the corresponding phase. \n\n## Q2: Problem in Derivation from Eq. (16) to Eq. (17):\n\nThe transformation from Eq. (16) to Eq. (17) can be expanded as:\n\n$$ \\frac{\\textbf{1}(k\\in S)}{1-\\bar\\alpha_t} ( \\sqrt{\\bar \\alpha_t} \\int q(\\textbf{v}_0^S | \\textbf{v}_t^S) \\textbf{v}_0^k \\ d\\textbf{v}^S_0 - \\textbf{v}_t^k) $$\n$$ = \\frac{\\textbf{1}(k\\in S)}{1-\\bar\\alpha_t} ( \\sqrt{\\bar \\alpha_t} \\int q(\\textbf{v}_0^k | \\textbf{v}_t^S) q(\\textbf{v}_0^{S/k} | \\textbf{v}_0^{k}, \\textbf{v}_t^S) \\textbf{v}_0^k \\ d\\textbf{v}^S_0 - \\textbf{v}_t^k) $$\n$$ =  \\frac{\\textbf{1}(k\\in S)}{1-\\bar\\alpha_t} ( \\sqrt{\\bar \\alpha_t} \\int q(\\textbf{v}_0^k | \\textbf{v}_t^S) \\textbf{v}_0^k \\int q(\\textbf{v}_0^{S/k} | \\textbf{v}_0^{k}, \\textbf{v}_t^S) \\ d\\textbf{v}^{S/k}_0 \\ d\\textbf{v}^k_0 - \\textbf{v}_t^k) $$\n$$ = \\frac{\\textbf{1}(k\\in S)}{1-\\bar\\alpha_t} ( \\sqrt{\\bar \\alpha_t} \\int q(\\textbf{v}_0^k | \\textbf{v}_t^S) \\textbf{v}_0^k \\ d\\textbf{v}^k_0 - \\textbf{v}_t^k) $$\n\nWe will add the intermediate steps in our revision to make it more accessible.\n\n## Q3: Missing references of related works.\nThanks for pointing it out. MAS and MVHuman are both interesting recent works that focus on 3D human/animal generation, which is a similar but different task from ours. We will add the references and discussion in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the thorough review and constructive feedback provided on our work. We are happy to see that the reviewers recognize our work as a novel attempt at a new and interesting task (Reviewer **WFsL, MCcD, v3P6**), our model is interesting (Reviewer **WFsL**) and achieves strong performance (Reviewer **3sRf, EujL, v3P6**), and our paper is well-written (Reviewer **WFsL, v3P6, EujL**). We will fix all typos and missing references in our revision.

We want to emphasize that our work aims to solve the very new task of multi-view video generation. Compared to other generative models for images, videos, and 3D objects, this task is much more challenging due to the lack of large-scale multi-view video datasets. 
By introducing this new problem setting, we aim to encourage further exploration of how video generation models can be utilized for future scene-level dynamic 3D and 4D generation tasks.
With the rapid progress of large-scale video diffusion models, we strongly believe the multi-video generative model has great potential to evolve and benefit many downstream applications, such as immersive content creation, video editing, and communications. 

We incorporate some fruitful suggestions from the reviewers and provide additional results in this rebuttal:
- Comparison with baselines on the FVD metric: Our model is on par with CameraCtrl and outperforms other baselines. This matches the results on FID and KID in our paper as well.
  - CameraCtrl: 277
  - AnimateDiff+SparseCtrl: 327
  - CameraCtrl+SparseCtrl: 430
  - Ours: 285
- Additional qualitative comparisons on new camera trajectories (Fig.A1 in PDF)
- Visualization of the homography warping applied in our WebVid10M phase. (Fig.A2 in PDF).

Here we address some of the most common issues.

## Q1: Clarification for the epipolar attention mechanism (Eq. 4).

The epipolar attention mechanism is added to provide inductive bias from input camera poses to the self-attention modules, encouraging them to extract information from the epipolar geometry. Specifically, it incorporates attention masks that function as a 0/1 masking on the full self-attention score, where only the pixels on the corresponding epipolar lines have attention weights, as shown in Fig.6 of our supplementary. Eq. 4 shows how the attention mask is calculated: For each pair of pixels located at x_1 and x_2 in the two input frames, respectively, the attention mask from x_1 to x_2 is determined by their epipolar distance (i.e. how far is it from x_1 to the epipolar line of x_2 in x_1’s frame). We will fix the typo (x_i -> x_1) in Eq. (4) and add more explanation in our revision.

## Q2: More details of our training phases.

The two training phases are applied to the same instance of CVSM in a single training process. Specifically, we load the LoRAs in both phases to the pipeline, and then, for each training step, a data batch from the joint of Webvid10M (MV10M) and RealEstate10K is sampled. The corresponding LoRAs are activated for training based on which dataset the data comes from.

## Q3: Clarification of the LoRA selection during inference.

Different LoRA selecting strategies are applied to the Stable Diffusion layers (i.e. Spatial attention layers) and the AnimateDiff layers (i.e. Temporal attention layers). For Stable Diffusion layers, as our model inherits the capability of AnimateDiff to adapt various LoRAs, we use different LoRAs that best fit the task in different experiments. In the quantitative experiments, RealEstate10K’s LoRA is used for RealEstate10K’s test case, and WebVid10M’s LoRA is used for WebVid10M’s test case; In the qualitative experiments, without specification, RealisticVision’s LoRA is applied for its great performance to generate high-quality images/videos. For temporal attention layers, the LoRA from CameraCtrl is used in all experiments.

Meta-review: The reviewers raised concerns mainly about the training strategy's clarity, the mathematical derivations' correctness, the integration and generalization of LoRA modules during inference, and the lack of ablation studies to evaluate the impact of key components. They also suggested including recent related works on consistent multi-view content generation. 

The authors responded comprehensively by clarifying the training phases, explaining the epipolar attention mechanism and LoRA selection process, addressing the derivation issues, and providing additional comparisons and qualitative results. They also acknowledged the need for more discussions on related works and ablation studies, which they committed to addressing in their revision. 

The ACs agree that the authors have adequately addressed the concerns, and the paper should be accepted.
Decision: Accept (poster)