SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints
Authors: Jianhong Bai
Menghan Xia
Xintao Wang
Ziyang Yuan
Zuozhu Liu
Haoji Hu
Pengfei Wan
Di ZHANG
Venue: ICLR 2025 Poster (2025)
Keywords: Video Generation
Diffusion Model
Primary area: generative models
TL;DR: A efficient method is proposed to lift pre-trained text-to-video models for open-domain multi-camera video generation from diverse viewpoints

Abstract: Recent advancements in video diffusion models demonstrate remarkable capabilities in simulating real-world dynamics and 3D consistency. This progress motivates us to explore the potential of these models to maintain dynamic consistency across diverse viewpoints, a feature highly sought after in applications like virtual filming. Unlike existing methods focused on multi-view generation of single objects for 4D reconstruction, our interest lies in generating open-world videos from arbitrary viewpoints, incorporating six degrees of freedom (6 DoF) camera poses.
To achieve this, we propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring consistent content across different viewpoints. Specifically, we introduce a multi-view synchronization module designed to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of high-quality training data, we also propose a progressive training scheme that leverages multi-camera images and monocular videos as a supplement to Unreal Engine-rendered multi-camera videos. This comprehensive approach significantly benefits our model.
Experimental results demonstrate the superiority of our proposed method over existing competitors and several baselines. Furthermore, our method enables intriguing extensions, such as re-rendering a video from multiple novel viewpoints. Project webpage: https://jianhongbai.github.io/SynCamMaster/

Published as a conference paper at ICLR 2025
SYNCAMMASTER: SYNCHRONIZING MULTI-CAMERA VIDEO GENERATION FROM DIVERSE VIEWPOINTS
Jianhong Bai" Menghan Xia? Xintao Wang", Ziyang Yuan', Zuozhu Liu', Haoji Hu
Pengfei Wan Di Zhang'
'Zhejiang University, Kuaishou Technology, 'Tsinghua University
Project webpage: https:// http.//....o/osyyaaaaatttt
Azimuth
Difference
Viewpalntz Viespolat
A young and besutiful gir dressed in pink dress, playing grund piano.
A man en thusiastiiaall devouring steaming plate of spaghetti.
Wide Shot
60" Difference Azimuth 30" Difference in Elevation
Viewpolat Vieepolati
An elephant wearing coloctel birthday hat walkingolong the sandy beach.
Achefis experty chopping onloes a well equipped kitchen.
15" Difference in Elevation (4 viewpoints)
Viewpoint 3 Viewpolax
Accuple sitting dazy setting enjoying Wistern-style wearingo wonan wearing
Figure 1: Examples synthesized by SynCamMaster. SynCamMaster generates multiple videos of the same dynamic scene from diverse viewpoints. Videos results are our project page.
ABSTRACT
Recent advancements in video diffusion models have shown exceptional abili- ties in simulating real-world dynamics and maintaining 3D consistency. This progress inspires us to investigate the potential of these models to ensure dy- namic consistency across various viewpoints, highly desirable feature for ap- plications such as virtual filming. Unlike existing methods focused on multi- view generation of single objects for 4D reconstruction, our interest lies in gen- erating open-world videos from arbitrary viewpoints, incorporating 6 DoFF cam- era poses. To achieve this, we propose a plug-and-plaz module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring con- sistent content across different viewpoints. Specifically, we introduce a multi- view synchronization module to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of high-quality training data, we de- sign hybrid training scheme that leverages multi-camera images and monoc- ular videos to supplement Unreal Engine-rendered multi-camera videos. Fur- thermore, method enables intriguing extensions, such re-rendering video from novel viewpoints. We also release a multi-view synchronized video dataset, named SynCamVideo-Datasee Our code is available https: /atp::///iite hhg. com/ i VGI SSnnaaaattt.r..... con//
Work done during an intersship KwaiVGI, Kuaishou Technology. Corresponding authors.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
As one of the most prominent generative models today, diffusion models have significantly ad- vanced video generation technology. From the initial UNet-based explorations (Wang et al., 2023a; Blattmann et al., 2023; Chen et al., 2024; Wang et al., 2023b; Zhang et al., 2023a), to recent transformer-baute based scaling laws (Ma et al., 2024a; Yang et al., 2024b), state-of-thh-arr video diffu- sion models (Sora, 2024; Gen-3, 2024; Kling, 2024) are capable of producing dynamic simulations and 3D consistent videos that align with textual descriptions and adhere to real- woolld physical laws. This progress has sparked visions of future video creation paradigms, where controllable generation technology is poised to play an indispensable role in this evolution. Current explorations in control- lable generation include structure control (Zhang et al., 2023b; Guo et al., 2023a), ID control (Jiang et al., 2024; Ma et al., 2024b), style control (Yang et al., 2023; Liu et al., 2023a), and single-camera control (Wang et al., 2024; He et al., 2024). However, multi-camera synchronized video generation, crucial for achieving combinational shots in virtual filming, remains under-explored.
Previous efforts in multi multi-camera generation have primarily focused 4D object generation (Xie et al., 2024; Li et al., 2024). While these methods have shown promising results, they are limited to generating multi-view videos from fixed positions, such as sampling equal intervals along an orbit around an object. Additionally, they are restricted to single-object domains and do not support open-domain scene generation. Recently, a concurrent work, CVD (Kuang et al., 2024), has ex- plored synthesizing videos with multiple camera trajectories starting from the same pose. However, this approach has been studied only in the context of narrow viewpoints due limitations in dataset construction. In this study, we focus on open-domain multi-camera video generation from arbitrary viewpoints. This task presents new challenges due to the goal of accommodating open-domain sce- narios and the large discrepancies in viewpoints. Specifically, we face two main challenges: (i) dynamically synchronizing across multiple viewpoints, which introduces the complexity main- taining 4D consistency, and (ii) the scarcity of multi-camera videos with diverse poses.
To address these challenges, we leverage the generative capabilities of a pre-trained text-to-video model by introducing plug-and-play modules. Specifically, given the extrinsic parameters of the desired cameras, normalized by setting one camera as the global coordinate system, we first encode these parameters into the camera embedding space using camera encoder. We then perform inter- view feature attention computation within a multi-view synchronization module, which is integrated into each Transformer block of the pre-trained Diffusion Transformer (DiT), Additionally, we col- lected a hybrid training dataset consisting of multi-view images, general single-view videos, and multi-view videos rendered by Unreal Engine (Sanders, 2016) (UE). While the manually prepared UE data suffers from -ppaiic issues and limited quantity, publicly available general videos enhance generalization to open-domain scenarios, and multi-view images promote geometric and visual consistency between viewpoints. Accordingly, we developed a tailored hybrid-data training scheme to optimize performance across these aspects.
Extensive experiments show that SynCamMaster can generate consistent content from different viewpoints of the same scene, and achieves excellent inter-view synchronization. Ablation stud- ies highlight the advantages of our key design choices. Furthermore, our method can be easily extended for novel view synthesis in videos by introducing a reference video to our multi-camera video generation model. Our contribution can be summarized as follows:
To our knowledge, SynCamMaster pioncered multi-camera real-world video generation.
We design an efficient approach to achieve riew-synchronized video generation across ar- bitrary viewpoints and support open-domain text prompts.
We propose a hybrid-data construction and training paradigm to overcome the scarcity of multi-camera videos and achieve robust generalization.
We extend our approach novel view video synthesis to re-render input video from novel viewpoints. Extensive experiments show the proposed SynCamMaster outperforms baselines by large margin.
2
RELATED WORKS
Controllable Video Generation. With the success of text-to-video generation models (Blattmann al., 2023; Wang et al., 2023b; Chen al., 2024; Menapace et al., 2024), more accurate guidance
2

---

Published as a conference paper at ICLR 2025
accompanied with text is often required in real-world applications. Previous research has success- fully incorporated various conditional signals in video generation models, such as motion trajectory (Yin et al., 2023; Fu et al., 2024), sketch and depth (Guo et al., 2023a; Xing et al., 2024).
Several works (Guo et al., 2023b; Yang et al., 2024a; Bahmani et al., 2024) also integrate camera pose control into video diffusion models. AnimateDiff (Guo et al., 2023b) introduces various motion LoRAs (Hu et al., 2021) to learn specific patterns of camera movements. MotionCtrl (Wang et al,, 2024) decouples camera motion and object movement and trains control modules to independently control both kinds of motion. CameraCtrl (He et al., 2024) further improves the accuracy and gen- eralizabilityy of Linglessquennee camera control with the dedicatedly designed camera encoder. The following work CVD (Kuang et al., 2024), expands CameraCtrl to multi-sequence camera control with the proposed cross-video synchronization module. Different from previous works, we focus on controllable multi-view video generation, rather than camera trajectory control in time dimension.
Multi- View Image Generation. Multi-view image generation (Shi et al., 2023; Liu et al., 2023c; Kant et al., 2024) is well-studied in the scenario of 3D synthesis. Given one image as input, (Liu et al., 2023b;e; Chan et al., 2023) achieves 3D reconstruction by multiview-consistenn image gener- ation. ViewDiff (Höllein et al., 2024) enabling more realistic multi-view synthesis with background via training on real-world images and the proposed 3D projection layer. Despite the promising performance, it exhibits limited generalization capabilities.
Multi- View Video Generation. Current studies on multi-view video generation primarily focus on 4D-assct synthesis (Liang et al., 2024; Li et al., 2024; Xie et al., 2024). SV4D (Xie et al., 2024) leverages both 3D priors in multi-view image generation models and motion priors in video gener- ation models to synthesize multi-view videos from a reference video. Vivid-ZOO (Li et al., 2024) introduces trainable LoRAs (Hu al., 2021) to alleviate the domain misalignment issue between 3D objects and real-world videos. In general, these approaches are expertise in object-level 4D synthe- sis, but could not generalized to
video with arbitrary viewpoints. A concurrent work GCD (Van Hoorick et al., 2024), dives into novel-view video synthesis in real-world scenarios, which is accomplished by training a video diffusion model conditioned on the camera pose and input video. Our method deviates from them in synthesizing multi-view videos with a single text prompt and desired viewpoints, while GCD generates monocular videos with an input video as the reference.
METHOD
Our goal is to achieve an open-domain multi-camera video generation model that can synthesize synchronized videos [V) ...., V") E R=xfxoxkxx with f frames following the text prompt Pt and n specified viewpoints {cam
cam"). The viewpoint is represented as the extrinsic parameters of the camera, i.e., cam; [R, t] R3x4, where R Rax3 refers to the rotation matrix and t E I 1xx is the translation vector. For simplificatio,, we assume that the viewpoints remain constant across frames. To realize this, we propose utilize the capability of pre-trained video diffusion models (Wang et al., 2023b; Chen et al., 2024) in 3D-consistent dynamic content synthesis and introduce and-pay multi-view synchronization module modulate the inter- view geometric and visual coherence. The overview of the model is depicted in Figure 2.
3.1 PRELIMINARY: TEXT-TO-VIDE VIDEO BASE MODEL
Our study is conducted over an internal pre-trained text-to-video foundation model. is a latent video diffusion model, consisting of 3D Variational Auto-Encoder (VAE) (Kingma & Welling, 2014) and Transformer-based diffusion model (DiT) (Peebles & Xie, 2023). Typically, each Trans- former block is instantiated cross-attention modules. The generative model adopts Rectified Flow framework (Esser et al., 2024) for the noise schedule and denoising process. The forward process is defined as straight paths be- tween data distribution and a standard normal distribution, i.e.
sequence of spatial attention, 3D (spatial-temporal) attention, and
= (1 - t)=o + te,
(1)
where e e (0, and t denotes the iterative timestep. To solve the denoising processing, we define mapping between samples 21 from noise distribution p1 to samples 2g from data distribution
3

---

Published as a conference paper at ICLR 2025
Procipt
F Rhxaxfees
cam E R
Block siV
reshane
30-attn
Cros-ettin
F p(xxfixxxa
cam' Rt dxxxd
Spart
repori
View
F Riox
Spatial-tth
Projector
Cross Goos-atth
View Attention
Linere Layer
Procapt
View View-Synchroiized Features
(a) Overview of SyaCamMaster
(b) Multi-View Synchronization Module
Figure 2: Overview of SynCamMaster. Based on pre-trained text-to-video model, two compo- nents are newly introduced: the camera encoder projects the normalized camera extrinsic parame- ters into embedding space; the multi-view synchronization module, as plugged in each Transformer block, modulates inter-view features under the guidance of inter-camera relationship. Only new components are trainable, while the pre-trained text-to-video model remains frozen.
Po in terms of ordinary differential equation (ODE), namely:
(2) where the velocity e is parameterized by the weights a neural network. For training, we regress vector field u, that generates a probability path between po and pı via Conditional Flow Matching (Lipman et al., 2023):
dæ. (= l)dl,
LICM Bepp(a.e)pptellleeet:..
where (2 ( (x)o)c)) with c-(-)c) denotes the function of Eq. 1. For inference, employ Euler discretization for Eq.2 and perform discretization over the timestep interval at [0, 1]. starting at
We then processed with iterative sampling with:
2 v0(=-11 () At.
(4)
3.2 MULTI-VIEW SYNCIRONIZATION MODULE
To achieve multi-view video synthesis, we train the multi-view synchronization (MVS) modules on top of the T2V generation model and leave the base model frozen. Note that, the operations below are conducted per frame across viewpoints, therefore we omit frame index for simplicity. The MVS module takes spatial features F (Fí, (F....., F Rax/xexd (where token sequence size h w) and camera extrinsic parameters cam = camm ..... cam"} E R=x122 of n videos as input, and output view-consistent features F" {F) [F...... F.J [axxfxxd to the subsequent layers in the base T2V model.
Specifically, at each block in the video diffusion model, the 12-dimensional extrinsic parameters of the i-th camera are first embedded with a camera encoder & to have the same dimension as the spatial features, and then clement-wise added to the corresponding spatial features. Then, propose to leverage a cross-view self-attention layer for multi-view synchronization. Unlike layers in the T2V base model which operate within single view feature, the additional cross-view attention layer takes multi-view features of the same frame as input, enabling cross-view feature aggregation. Finally, the aggregated features are then projected back the spatial feature domain with a linear layer and residual connections. In summary, the multi-view synchronization (MVS) module is formulated as:
F F + (camm))
(5)
F F + projector(attn view(Fi,.... F.)...
(6)
where we instantiate fully connected layer with input dimension of 12 and output dimension of d as camera encoder E, each block. Note that we insert the MVS module introduced above into each basic block of the DiT model to realize fine-grained control.
4

---

Published as a conference paper at ICLR 2025
Video
Video
Sampling
Surpling
Sumpling
Sampling
tampinng
Sampling
Camera Sequence
Camera Pose
Copy times
Construction Multi-view Image
Remdered Multi-view Video Data
General
Figure 3:
Data collection process. (a) Illustration extracting multi-view image data from videos with camera movements, images are from DL3DV-10K (Ling et al., 2024); (b) Example of the ren- dered multi-view videos from diverse viewpoints; (c) Utilizing general video data as regularization.
3.3 DATA COLLECTION
The scarcity multi-view video data is one the main challenges hindering the training of multi- view video generation models. Existing multi-view vidco data primarily consists of 1) videos ren- dered from 4D assets from various views (Liang et al., 2024; Zhang et al., 2024), and 2) human- centric motion capture datasets (lonescu et al., 2013; Joo et al., 2015; Shahroudy et al., 2022). However, these datasets do not adequately meet the needs of the task in this paper. For videos rendered from 4D assets (Liang al., 2024), there is significant domain gap between them and real-world videos, which can lead to severe degradation in video quality. For motion cap- ture datasets, they mainly captured the videos from several fixed viewpoints (e.g.. 31 viewpoints distributed over hemispherical surface in Panoptic studio (Joo et al., 2015) and 4 in Human3 6M (lonescu al., 2013)), which hinders the model's generalization across arbitrary viewpoints.
al., 2016; Lin
To this end, we propose three-step solution, as illustrated in Fig. 3. Firstly, we leverage single-camera videos as multi-view image data, and transfer the knowledge of geometry corre- spondence between different viewpoints video generation. Specifically, RealEstate-110K (Zhou al., 2018) and DL3DV-IOK (Ling et al., 2024) contain videos with camera movements and their corresponding camera parameters across frames. We propose to sample n video frames from the video as available multi-view image data. Secondly, we manually render a small number (500 scenes, 36 cameras each) of videos using the UE engine, featuring 3D assets such humans and animals moving within urban environments. We enhance the model's generalization ability across arbitrary viewpoints by randomly placing camera positions. Lastly, we also incorporate high-quality general video data (without the corresponding camera information) as regularization during training. The construction process of our rendered multi-view video data follows.
Construction of the Multi-View Video Data Firstly, we collect 70 3D assets of humans and animals as the main subjects and select 500 dif- ferent locations in 3D scenes background. Secondly, we randomly sample 1-2 main sub- jects to place them in each location and let them move along several pre-defined trajecto- ries. Thirdly,
up 36 cameras differ- ent locations in each scene and rendered 100 frames synchronously. As result, the multi- view video dataset is constructed of 500 sets of synchronized videos with 36 cameras each. The cameras in each scene are placed on hemispher- ical surface at distance to the center of 3.5m 9m. To ensure the rendered videos have minimal domain shift with real-world videos, we constraint the elevation of each camera between 0 45%, and the azimuth between 0% 360", To support SynCamMaster in synthesizing videos from arbitrary viewpoints, each camera is randomly sampled within the constraints, rather than using the same set of camera positions across scenes. Fig. 4 shows example of one scene, where the red star in- dicates the center point of the scene (slightly above the ground), and the videos are rendered from the synchronized cameras capture the movements of the main subjects (a goat and bear in the
Figure 4: Illustration of the rendering scene.
5

---

Published as a conference paper at ICLR 2025
case). In the paper, we refer to this batch of rendered multi-view data as 'M. V. Video'. A detailed description and analysis of datasets are in Appendix B.
3.4 TRAINING STRATEGY
Progressive Training. To effectively learn the geometry correspondence between different view- points, we found it's crucial to start with feeding the model views with relatively small angle dif- ferences, and progressively increase the differences during training. Simply performing random sampling from different cameras in the same scene will lead to significant performance degradation in viewpoint following capabilities when input viewpoints with large relative angles (Fig. 7). A detailed description of training details is in Appendix C.
Joint Training with Multi-Viewy Image Data. To alleviate the lack of multi-cam video data, we construct mulli-view image data by sampling from single-camert video data as introduced in 3.3. We leverage DL3DV-IOK (Ling et al., 2024) as the auxiliary image data, which includes ~10K videos featuring wide-angle camera movements in both indoor and outdoor scenes. Our findings indicate that joint training with multi-view image data significantly enhances the generalization capability of SynCamMaster. This improvement is largely attributed to the diversity and scale of DL3DV- 10K compared to our multi-view video data (10K vs. 500). Furthermore, the viewpoint-following capability is agnostic to the type of data, whether image or video, and is transferable between them.
Joint Training with Single-Vies View Video Data. To improve the visual quality of the synthesized videos, propose incorporate high-quality video data (without camera information) as regular- ization. Given a single-view video, we augment it into multi-view videos by copying it v times and setting the camera parameters the same across views. In other words, single-view videos are sidered as videos with u overlapping views during training. Moreover, we observe performance degradation when simply using videos with arbitrary camera movements, it can be caused by dis- tribution misalignment since SynCamMaster aims to generate videos from a fixed viewpoint. To this end, we filter out static camera video data using the following three steps: First, we downsam- ple the video to 8 fps and use SAM (Kirillov et al., 2023) to segment the first frame. obtaining 64 segmentation masks. Next, we select the center point of each segmented region as the anchor point and use the video point tracking method CoTracker (Karaev et al., 2023) to calculate the position coordinates of each anchor point in all frames. Finally, we determine whether the displacement of all points is below a certain threshold. As result, we filtered out 12,000 static camera videos, which were added as a regularization term during training.
3.5 EXTENSION TO NOVEL VIEW VIDEO SYNTHESSS
Additionally, we extend SynCamMaster to the task of novel view video synthesis (Van Hoorick al., 2024; Zhang et al., 2024), which aims to generate videos captured from different view- points based a reference video. To achieve this, we introduce modifications to turn SynCam- Master into a video-to-multvvee--vie generator. During training, given the noised latent features (z!...., 2(" E [ of multi-view videos at timestep f,, we regard the first view video as reference and replace the noisy latent of the video with its original one with the probability p 90%, i..., 2 z/. To this end, videos from novel views (i = 2, ... n1) could effectively aggregate fea- tures from the reference view via the proposed multi-view synchronization module introduced in Section 3.2. At the inference stage, we first extract the latent feature of the input video with the pre-trained video encoder and then perform feature replacement at each timestep / = T, O. Meanwhile, implement weighted classifier-free guidance on text condition er and video condi- tion cy similar to Instruct-Pix2pix (Brooks et al., 2023):
eie ev = vve22. 2.0)
+ sv (v CV, Ø) -oe-- 2,2))
ST (ve ev, cv, CT) no zz, cv, 2)),
where sy and sy are the weighting scores of the text condition and video condition respectively, set 8y = 7.5 and sv = 1.8 in practice. To this end, the proposed SynCamMaster can effectively re-render video consistent with the text prompt and camera poses, exhibited in Fig. 8.

---

Published as a conference paper at ICLR 2025
90" Difference in Azimuth
20 Difference in Elevation
M.V. Image
SVD-XY
181
M.V. Image
M.V. Image 12V-0urs
SynCamMaster
An elephant wearing colorful birthday hat walking along the sandy beach.
blue bus drives across the iconic Tower Dridge in London.
Figure 5: Comparison with state-of-the-ars methods. The reference multi-view images of baseline methods (indicated in the blue box) are generated by SynCamMaster. It shows that SynCamMaster generates consistent content (e.g., the details in the red box) from different viewpoints of the same scene, and achieves excellent niew synchronization.
4
EXPERIMENTAL RESULTS
4.1 EXPERIMENT SETTINGS
Implementation Details We joint train our model on multi-view video data, multi-view image data, and single-view video data with the probability of 0.6, 0.2, and 0.2 respectively. We train the model of 50K steps at the resolution of 384x672 with a learning rate of 0.0001, batch size 32. The view-attention module initialized with the weight of the temporal-attention module, and the camera encoder and the projector are vero-initialized.
Evaluation Metries We mainly evaluate the proposed method in terms of cross-view synchro- nization and visual quality. In terms of cross-view synchronization, we utilize the state-of-the-ar image matching method GIM (Shen et al., 2024) to calculate: 1) the number of matching pixels with confidence greater than the threshold, denoted as Mat. Pix., and 2) the average error between the rotation matrix and translation vector estimated by GIM of each frame and their ground truth, denoted as RotEr and TransErr respectively (He et al., 2024). Furthermore, we calculate the FVD- V score in SV4D (Xie et al., 2024), and the average CLIP similarity between multi-view frames at the same timestamp. denoted as CLIP-V (Kuang et al., 2024). For visual quality, we divide into fidelity, coherence with text, and temporal consistency. and quantified them with Fréchet Image Distance (Heusel et al., 2017) (FID) and Fréchet Video Distance (Unterthiner et al., 2019) (FVD), CLIP-T, and CLIP-F respectively. CLIP-T refers the average CLIP (Radford et al., 2021) simi-

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison with state-of-the-art methods.
Visual Quality
View Synchronization
Method
FID + FVD + CLIP-T CLIP-F MaiP-FF|Maa Mat. Pix.(K) 1 FVD-V 4 CLIP-V V
M.V. Image SVD-XT
137.3
1755
97.56
150.4
1742
89.14
M.V. Image + CameraCtrl 152.8
2203
98.32
172.9
1661
89.33
M.V. Image + 12V-Ours 113.1
1376
33.48
99.27
116.8
1930
90.01
SynCamMaster
116.7
1401
33,40
99.36
527.1
1470
93.71
Table 2: Quantitative ablation on the joint training strategy.
Visual Quality
View Synchronization
Method
FID
FVD & CLIP-T f CLIP-F t
Mat. (Kx. f FVD-V 4 CLIP-V
Multi- Video
149.9
1971
30.97
99.37
460.5
1668
89.68
Multi-View Image
121.5
1655
33.02
99.36
533.0
1482
93.15
+ General Video
122.4
1608
32.54
99.38
471.9
1514
90.12
Both
116.7
1401
33.40
99.36
527.1
1470
93.71
larity of each frame and its corresponding prompt, and CLIP-F is the average CLIP similarity of adjacent frames. We construct the evaluation set with 100 manually collected text prompts, and inference with viewpoints each, resulting 400 videos in total.
4.2 COMPARISON WITH STATE-OF-THE-ARR METHODS
Baselines To the best of our knowledge, multi-view real-world video generation has not been explored by previous works. To this end, we establish baseline approaches by first extracting the first frame of each view generated by SynCamMaster, and then feeding them into 1) image-to-vides (12V) generation method, i.e., SVD-XT (Blattmann al., 2023) state-of-the-art single-video camera control approach CameraCtrl (He et al., 2024) based on SVD-XT. Since CameraCtrl would have non-optimal performance when conditions on static camera trajectory, we use a trajectory with limited movement as input. To ensure fair comparison, we additionally train an 12V generation model based on the same T2V model used by SynCamMaster, the I2V model is fine-tuned with the approach similar to EMU Video (Girdhar et al., 2023) for 50K steps. During training, we extend and concatenate the latent feature of the first frame with the noised video latent along the channel dimension, and expand the dimension of the input convolutional layer with zero-initialized weights. We also replace the image latent with zeros the probability 0.I. At the inference stage, we implement weighted classifier-free guidance for the image and text condition (Brooks et al., 2023).
Qualitative Results We present synthesized examples of SynCamMaster in Fig. (additional examples in Fig. 13-14 of Appendix E). Please visit our project page for more videos. SynCam- Master demonstrates the ability 1) generate consistent content from diverse viewpoints of the same scene; 2) achieve excellent inter-view synchronization; and 3) maintain the generation ability of base model across various text prompts.
We compare SynCamMaster with state-of-the-ar methods in Fig. 5 (further comparisons in Fig. 12 of Appendix E). Note that use SynCamMaster to synthesize multi-view images (M.V. V. Images) as baseline methods' reference images (indicated in the blue box) since they cannot generate videos from various viewpoints. It's observed that the baseline methods failed to generate coherent videos across viewpoints. For example, the blue bus may stay in place in one shot, moving forward in another. While SynCamMaster can synthesize view-aligned videos that adhere to both the camera poses and text prompts.
Quantitative Results We quantitatively evaluate method using various automatic metrics, the summarized results are in Tab. 1. For view synchronization, we calculate the clip similarity score and FVD between video frames of different viewpoints within scene, denoted as CLIP-V and
8

---

Published as a conference paper at ICLR 2025
Difference Arimuth
Viewpuint
Viewpoint 2
Viewpiitt
Viewpoint 2
Video
MAC Vidte DANVI 2AN
N.V. Video
Noroo Video
Monno Vileo - Inagge Woon Softo
ACIC Videoo
M.K Vidco
-MLXK imrage
MLD. Vidio
Wooth Video
-M X warge
MEK Ville
walks strect
young and becutitu gtri dressed In pink dress, sitting elogontyy ot grand plano.
night accompatiiol brown dog.
Resalts
Figure 6: Ablation on the joint training strategy. The captions on both sides represent the com- position of the training set, where "Mono. Video" refers to general monocular videos. It shows that training with the auxiliary multi-view image data and general video data significantly improves the generalization ability and fidelity of the synthesized videos.
Table 3: Results of novel view video synthesis.
Table 4: Accuracy of camera control.
Setting
LPIPS L PSNR SSIM 1
Method
RotErr TransErr
sy 1.2, 8T 550 0.4899
16.29
0.4754
M.V. Image + SVD-XT
0.25
0.72
sy 1.2, 8T 775 0.4901
16.60
0.4783
M.V. Image + CameraCtrl
0.16
0.67
sy 1.8, 7.5 0.4761
16.47
0.4935
M.V. Image + 12V-Ours
0.26
0.80
8v 2.5, ST 7.5 0.5022
14.55
0.4667
SynCamMaster
0.12
0.58
FVD-V. It's observed that the proposed SynCamMaster significantly outperforms baseline methods, which is aligned with qualitative results Fig. 5. To further eliminate the effect different base models, we also fine-tuned 12V generation model top of the same base model used by SynCam- Master, denoted as '12V-Ours'. As observed, SynCamMaster achieves comparable performance in terms of fidelity, text alignment, and frame consistency, while having better synchronization across views. Furthermore, we use 100 videos containing camera pose differences in azimuth or elevation calculate the accuracy of camera control. The average eтrо along frames is reported in Tab. 4, where SynCamMaster has superior performance compared to baselines. Note that we normalize the relative translation vector of two viewpoints to have a norm of 1.0 to align the scale between estimated poses and ground truth (Xu et al., 2024).
4.3 MORE ANALYSIS AND ABLATION STUDIES
The Effectiveness of Joint Training As introduced Section 3.4, we utilize multi-view im- ages and general videos to alleviate the scarcity of available multi-view video data, and further improve the generalization ability and visual quality of SynCamMaster. To verify the effective- ness, we make qualitative comparisons in Fig. 6 and exhibit quantitative results in Tab. 2. In Fig. 6(a), we visualize the generation results of the proposed method on the validation set by con- ditioning on two cameras with a 90-degree difference in azimuth. It's observed that only train- ing SynCamMaster on the rendered multi-view video dataset is sufficient to generate synchronized videos the training domain, which demonstrates the proposed multi-view synchronization mod- ule could effectively generate consssonsitt features. However, due to the domain gap between the rendered data and the diverse real-world videos, merely training multi-view videos could result in poor performance when transferring the multi-view synthesis ability to general videos. As shown in Fig. 6(b), we observe severe performance degradation in pose following and syn- chronization when inference with test set prompts. In this case, the girl's arms are at different positions across the two viewpoints, with the camera pose misalignment. The issue is signifi- cantly alleviated when we engage multi-view image data in training, as exhibited in the second line. Furthermore, we enhance the fidelity of SynCamMaster with diverse general video data.

---

Published as a conference paper at ICLR 2025
30" Difference Azimuth
Video
Ret.
Viddon
5
1.2
Novel View
Av 2.5. bq
7.5
man is enthusiastically eating colorfal rainbow- striped cake.
30" Difference Azimuth
7.5 7
Ref. Vidoo
Novel
Tresh
A man is standing in kitchen, wearing gray hoodle and black t-shirt.
Results of Novel View Video Synthesis.
Ablation Weighting Factor.
Figure Results of the extension on novel view video synthesis.
The Effectiveness of Progressive Training
View synchronized synthesis is particularly challenging when aiming generate videos with large viewpoint differences since there are fewer matching features across views. To tackle this problem, we propose a progressive train- ing strategy that gradually increases the rel- ative angle between different viewpoints dur- ing training. In Fig. 7, we visualize the syn- thesized results of training w/ w/o the pro-
120' Difference Asimuth
w/A /o Progressivv Training
w/ Progressive Training
posed progressive training strategy. It's ob-
served that while simply training with samples that have arbitrary relative angles can also gen- erate fiew-consistent videos, they would fail in pose-following ability when conditioning on cameras with large relative angles.
clese-up
cravling across a sandy
Figure 7: Ablation on progressive training.
Ablation on Classifier-Frrs Guidance in Novel View Video Synthesis In Section 3.2, We extend the proposed text-to-multiveew video generation method novel view video synthesis, i.c., video- to-multiview video generation. We present the synthesized videos in Fig. 8(b) on the validation set with different sets of weighting scores sy and sT Eq. 7. As shown, SynCamMaster can re-render the input video at arbitrary viewpoints, while maintaining excellent view synchronization by setting weighting scores certain range. Quantitatively, we calculate the LPIPS, PSNR, and SSIM scores on 100 randomly selected video pairs in Tab. 3, and we observe comparable results with previous work GCD (Van Hoorick et al., 2024). We set sv = 1.8 and ST = 7.5 practice.
CONCLUSION AND LIMITATIONS
In this paper, we propose SynCamMaster to generate synchronized real-world videos from arbitrary viewpoints. To achieve this, we leverage the pre-trained text-to-video generation model and design multi-view synchronization module to maintain appearance and geometry consistency across differ- ent viewpoints. We also extend our method to novel view video synthesis to re-render an input video. There are nevertheless some limitations. Firstly, when generating videos with complex scenes, the content in the synthesized multi-view videos may exhibit inconsistencies in details. Secondly, since SynCamMaster is based on T2V models, it also inherits some the shortcomings the base model, such inferior performance in hand generation. We exhibit the failure cases in Fig. 15.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Jinwen Cao, Yisong Guo, Haowen Ji, Jichao Wang, and Yi Wang from Kuaishou Tech- nology for their invaluable help in constructing the multi-view video dataset. We thank the authors of 4DGS (Wu et al., 2024) and Jiangnan Ye for their help in running 4D reconstruction.
REFERENCES
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin- Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. arXiv preprint 10Xiv:2407.12781, 2024.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint arXiv:2311. 2023.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pixx Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view syn- thesis with 3d-aware diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4217-4229, 2023.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7310-7320, 2024.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning (ICML), 2024.
Xiao Fu, Xian Liu, Xintao Wang, Sida Peng. Menghan Xia, Xiaoyu Shi, Ziyang Yuan, Pengfei Wan, Di Zhang, and Dahua Lin. 3dtrajmaster: Mastering 3d trajectory for multi-entity motion in video generation. arXiv preprint, 2024.
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Bautin-Brualla, Pratul Srinivasan, Jonathan Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405. 10314, 2024.
Gen-3. Gen-3. Gen-3. Accessed Sept.30, 2024 [Online] https://runwayntt wwm.aammmm.mmmm...mmmmmmmm..m.mm 3- 3-alpha, 2024. URL https://runwayml. com/
research/introdueiin gen-
researchh/ittoouccee -en- 3-a3--llll alpha.
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Ramb- hatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-vides generation by explicit image conditioning. arXiv preprint urXiv:2311.10709 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint arXiv:2311.1693, 2023a.
11

---

Published as a conference paper at ICLR 2025
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint WxXfv:2307.04725, 2023b.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101 2024.
Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar transformers. In Proceedings of the ieee/cvf conference computer vision and pattern recognition, pp. 7779-7788, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Lukas Höllein, Aljaž Božič, Norman Müller, David Novotny, Hung-Yo Yu Tseng, Christian Richardt, Michael Zollhöfer, and Matthias Nießner. Viewdiff: 3d-consistent image generation with text- to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5043-5052, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint vrXiv:2106.0966 2021.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7): 1375-1393,,3 2013.
Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu. Videobooth: Diffusion-based video generation with image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6689-6700, 2024.
Haian Jin, Hanwen Jiang. Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, and Zexiang Xu. Lvsm: A large view synthesis model with minimal 3d inductive bias. arXiv preprint 10 17242, 2024.
Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, lain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In Proceedings of the IEEE international conference on computer vision, pp. 3334-3342 2015.
Yash Kant, Aliaksandr Siarohin, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, Sergey Tulyakov, and Igor Gilitschenski. Spad: Spatially aware multi- view diffusers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10026-10038, 2024.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: better to track together. arXiv preprint arXiv:2307.07635 2023.
Diederík P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Kling, Kling. Kling. Accessed Sept.30, 2024 [Online] sttps://kles kuai com/en, 2024. URL.https: //kl ii............. kua shou. com/en.
12

---

Published as a conference paper at ICLR 2025
Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-vidco generation with camera control. arXiv preprint ar 4rXiv:2405.17414 2024.
Bing Li, Cheng Zheng, Wenxuan Zhu, Jinjie Mai, Biao Zhang, Peter Wonka, and Bernard Ghanem. Vivid-zoo: Multi-view video generation with diffusion model. arXiv preprint:220660000 2024.
Hanwen Liang, Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N Platanio- tis, Yao Zhao, and Yunchao Wei. Diffusion4d: Fast ppatial-tempore consistent 4d generation via video diffusion models. arXiv preprint 2rXiv:2405.16644 16645, 2024.
Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Effi- cient neural radiance fields for interactive free-viewpoint video. In SIGGRAPH Asia Conference Proceedings, 2022.
Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D13dv-10k: A large-scale scene dataset for deep learning- bbased 3d vision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22160-22169, 2024.
Yaron Lipman, Ricky Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023.
Gongye Liu, Menghan Xia, Yong Zhang, Haoxin Chen, Jinbo Xing, Xintao Wang, Yujiu Yang, and Ying Shan. Stylecrafter: Enhancing stylized text-to-video generation with style adapter. arXiv preprint arXiv:2312.003:: 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9298-9309, 2023b.
Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistenn images from a single-view image. arXiv preprint urXiv:2309.034555 2023c.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03045 2024a.
Ze Ma, Daquan Zhou, Chun-Hsiao Yeh, Xue-She Wang, Xiuyu Li, Huanrui Yang, Zhen Dong, Kurt Keutzer, and Jiashi Feng. Magic-me: Identity-specific video customized diffusion. arXiv preprint arXiv:2402.093683 2024b.
Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shies Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotem- poral transformers for text-to-video synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7038-7048, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 4205-4205, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 9748-8763. PMLR, 2021.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cat- egory reconstruction. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 10901-10911. 2021.
Andrew Sanders. An introduction to Unreal engine 4. AK Peters/CRC Press, 2016.
13

---

Published as a conference paper at ICLR 2025
Amir Shahroudy, Jun Liu, Tian-Tsone Ng, and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1010-1019, 2016.
Xuelun Shen, Zhipeng Cai, Wei Yin, Matthias Müller, Zijun Li, Kaixuan Wang, Xiaozhi Chen, and Cheng Wang. Gim: Learning generalizable image matcher from internet videos. In The Twelfth International Conference on Learning Representations, 2024.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long. Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.16511 2023.
Sora.
Sora. Sora. Accessed Sept.30, 2024 [Online] https:/ /ttppp::///ppaa..... cariaaai.ooo//nee/e//ee
video- setaoo-geeeattoo--mmdeeaa------- wtw.llddddsmmaaattrsoooooo 2024. URL https:// satissaaaa-oooo--mmmamaatrrr///
openai com/ wwtp:///ndderrreeeeeeeeetaaaooor wttp://iiwoeeeeeeiii......
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. Openreview, 2019.
Basile Van Hoorick, Rundi Wu, Ege Ozguroglu, Kyle Sargent, Ruoshi Liu, Pavel Tokmakov, Achal Dave, Changxi Zheng, and Carl Vondrick. Generative camera dolly: Extreme monocular dynamic novel view synthesis. arXiv preprint arXiv:2405.14866 2024.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang. and Shiwei Zhang. Mod- clscope text-to-video technical report. arXiv preprint arXiv:2308 2023a.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang. Yinan Hc, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality vidco generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 2023b.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionetrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20310-20320, 2024.
Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, and Varun Jampani. Sv4d: Dy- namic 3d content generation with multi-frame and multi-view consistency. arXiv preprint arXiv:2407.17470, 2024.
Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Yong Zhang, Yingqing He, Hanyuan Liu, Haoxin Chen, Xiaodong Cun, Xintao Wang, et al. Make-your-video: Customized video gener- ation using textual and structural guidance. IEEE Transactions on Visualization and Computer Graphics, 2024.
Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Cameralcontrollable 3d-consistent image-to-video generation. arXiv preprint arXiv:2406.02509 2024.
Shiyuan Yang. Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-12, 2024a.
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text text-guided video-to-video translation. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-11, 2023.
Zhuoyi Yang. Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint arXiv:2408. 2400.00072 2024b.
14

---

Published as a conference paper at ICLR 2025
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine- Fine-grained control in video generation by integrating text, image, and trajectory. arXiv 2023.
preprint
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9150-9161, 2023.
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to- video generation. arXiv preprint arXiv:2309.15878, 2023a.
Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, and Yu Qiao. 4diffusion: Multi-view video diffusion model for 4d generation. arXiv preprint arXiv:2405.20674 2024.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305. 13077, 2023b.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images, arXiv preprint 1rXiv:1805.0988/ 2018.
15

---

Published as a conference paper at ICLR 2025
A INTRODUCTION OF THE BASE TEXT-TO- VIDEO GENERATION MODEL
Skip Addition
3D VAE Encoder
Trunsfomner Block
Spatial Self'- Self-AAtennie Attention
RMSNoom & Scale
RMSNomu & Scale
3D Self- Attentio
RMSNom Scale
Cross- Cross-Atrention Attention
RMSNorm & Scale
Feed- Feed-Forward Forward NN
Zo
Noise
3D VAE
15 Encoder
"Text prompt"
TimeStep
Figure 9: Overview of the base text-to-video generation model.
We use a transformer-Dee based latent diffusion model (Peebles & Xie, 2023) as the base T2V generation model, as illustrated in Fig. 9. Initially, we employ 3D-VAE to transform videos from the pixel space to latent space, upon which we construct a transformer- based video diffusion model. Unlike previous models that rely on UNets or transformers, which typically incorporate an additional 1D temporal attention module for video generation, such spatially-temporall yield optimal results. We replace the ID temporal attention with 3D self-attention, enabling the model to more effectively perceive and process spatiotemporal tokens, thereby achieving a high- quality and coherent video generation model. Specifically, before each attention feed-forward network (FFN) module, we map the timestep to a scale, thereby applying RMSNorm to the spa- tiotemporal tokens.
separated designs do not
B DATA CONSTRUCTION
B. TRAINING DATASETS
Recall in Section 3.3, we train the proposed SynCamMaster with data from different sources. In this section, we provide a detailed description of the construction process various datasets.
Multi- View Video Data Firstly, we collect 70 3D assets of persons and animals the main sub- jects and select 500 different locations in 3D scenes as background. Secondly, we randomly sample 1-2 main subjects to place them in each location and let them move along several pre-defined trajec- tories. Thirdly, we set up 36 cameras at different locations in each scene and rendered 100 frames synchronously. As a result, the multi-view video dataset is constructed of 500 of synchronized videos with 36 cameras each.
The cameras in each scene are placed on a hemispherical surface a distance to the center of 3.5m- 9m. To ensure the rendered videos have minimal domain shift with real-world videos, we constraint the elevation of each camera between 0º 45", and the azimuth between 0% 360", To support SynCamMaster in synthesizing videos from arbitrary viewpoints, each camera is randomly sampled within the constraints described above, rather than using the same set of camera positions across scenes. Fig. 4 shows example of one scene, where the red star indicates the center point of the scene (slightly above the ground), and the videos are rendered from the synchronized cameras to capture the movements of the main subject.
16

---

Published as a conference paper at ICLR 2025
Multi- View Image Data In this paper, construct multi-view image data from DL3DV- 10K (Ling et al., 2024). DL3DV-1KK is composed of 10,510 videos from both indoor and outdoor scenes, with their corresponding camera poses estimated by structure from motion (SFM) methods. We utilize its 960P version and downsample the frames to the resolution of 384x672.
General Video Data We also utilize single-view video data to further improve the visual quality of the synthesized videos. We observe a performance degradation when simply using videos with arbitrary camera movements, it can be caused by distribution misalignment since SynCamMaster aims to generate videos from a fixed viewpoint. To this end, we filter out static camera video data using the following three steps: First, we downsample the video to 8 fps and use SAM (Kirillov et al., 2023) to segment the first frame, obtaining 64 segmentation masks. Next, we select the center point of each segmented region as the keypoint and use the video point tracking method CoTracker (Karaev et al., 2023) to calculate the position coordinates of each keypoint in all frames. Finally, determine whether the displacement of all points is below certain threshold. As a result, filtered out 12,000 static camera videos, with an average duration of 9 seconds, which were added regularization term during training.
B.2 DISCUSSION ON OTHER MULTI- DATASETS
Multi- Images Multi-view images are also utilized in image novel view synthesis. Co3D (Reizenstein et al., 2021) and MVImgNett (Yu et al., 2023) are object-centric datasets including multi-view frames from multiple object classes. Despite their large scale, they do not meet the requirements of our task in two aspects. On the one hand, aim to synthesize multi-view real- world videos, which have a domain gap between the object-centric frames. On the other, most of the backgrounds in these datasets do not have obvious features. For example, most objects are placed on solid-colored tables or roads, which makes it more challenging to learn geometry correspondence. We observe inferior performance in terms of camera pose following ability when integrating Co3D and MV WegggNet into training. Furthermore, similar to DL3DV-10K, RealEstate- (Zhou et al., 2018) is also a commonly used dataset at scene level. Compared to RealEstate- 10K, DL.3DV-10 10K has more videos with rotating perspectives, which benefits SynCamMaster from synthesizing videos with large differences amizuth.
Multi- View Videos Previous works have established frame-synchronized multi-view video data (lonescu al., 2013; Joo et al., 2015) for human pose estimation, action recognition, etc. Neverthe- less, the cameras are fixed across different videos, which hinders SynCamMaster's ability to learn to generate videos from arbitrary viewpoints. Additionally, some recent works (Xie al., 2024; Zhang et al.. 2024) obtain multi-view video by filtering out 4D assets from Objaverse (Deitke et al., 2023) and rendering them with multiple cameras. While these data are helpful for 4D object generation, the significant domain gap is insufficient to support the generation of multi-view view real-world videos.
C IMPLEMENTATION DETAILS
Sampling Strategy To effectively learn the geometric correspondences between different view- points, we have carefully designed sampling strategy during training. For multi-viee view image data, to ensure overlap in the field of view between selected frames, we limit the maximum distance between u different frames to 100 frames during sampling. For multi-view image data, we first cal- culate the azimuth angle of each camera, and then ensure that the relative elevation angles between each pair of the selected v video clips fall within the interval [0t, 0z). The values of 0; and 0₂ are determined based on the number of training steps. In practice, we use 0; = 0, 0, = 60 for 0-10K steps, 6 30, f,, = 90 for 10K-20K steps, and A; 60, 9,, 120 for steps greater than 20K. We precompute all possible combinations before training to enhance efficiency.
Training Configurations The proposed SynCamMaster is jointly trained multi-view video, multi-view image, and general video data as introduced in Section 3.3. At the beginning of each step, we randomly select the type of data to be used for that step based on predefined probabilities (0.6, 0.2, 0.2 in practice). We use a batch size of 32 and a learning rate of 0.0001 to train the model for 50k steps at the resolution of 384x672.
17

---

Published as a conference paper at ICLR 2025
Construction of Evaluation Set evaluate SynCamMaster's accuracy camera control, set up several groups of camera poses with variations in either azimuth or elevation angles. We used the image matcher method GIM (Shen et al., 2024) to obtain the estimated homography matrix, which was further decomposed to derive the relative extrinsic matrix. Then we computed the rotation error and translation 2024). Specifically, under the setting of generating four synchronized videos (v = 4), we configured adjacent cameras with azimuth angle differences of {10", 20", 30%) elevation angle differences of {10", 15%, resulting in total of 5 different camera setups, with 20 videos per setup. The evaluation results are summarized in Tab. 4.
respectively by following CameraCtrl (He et al., 2024) and CamCo (Xu al.,
D MORE ANALYSIS AND ABLATION STUDIES
The Choice of Camera Representation In this paper, used the camera extrinsic matrix as the camera representation. The ray positional embedding is also widely used in 3D reconstruction (Gao et al., 2024) and camera control (He et al., 2024) techniques. We evaluate the impact of using different representations on camera accuracy, and present the qualitative and quantitative results in Fig. 10 and Tab. 5 respectively. As shown, there is no significant difference in the generated results between the two representations. We assume this is because the UE data (rendered multi-camera synchronized videos) has the same camera intrinsics, and in this case, both representations contain consistent information.
20" Difference Elevation
15' Difference in Elevation
Extrinsie Embedding
Plucker Embedding
A chef expertly chopping onions in well eell-equipped kitchen.
The setting is workshop or studio environment with the focus oc individual engaged puttery making.
Figure 10: Comparasion using different camera representations.
Table 5: Accuracy of camera control with different camera representations.
Camera Representation
RotEr
TransErr
Plucker Embedding
0.12
0.60
Extrinsic Embedding
0.12
0.58
Ablation on Epipolar Attention To effectively learn the spatial geometry, previous studies (He et al., 2020; Kant et al., 2024) explicitly model 3D information by using epipolar spistlar-oonstrained atten- tion layers. In Fig. 11 and Tab. 6, we compare the results w/ and w/o epipolar attention in view attention layers. Specifically, when implementing epipolar attention, for the token at spatial position (x, v) in view i, only aggregates features from all tokens within view and tokens on the epipolar line in other views. In contrast, each token attends to all tokens in all views in the full attention setting. We found that although epipolar attention has low rotation error, it can result inconsis- tency with the text prompt semantics (shown in Fig. 11) compared to the full-attention design. On the other, we found that is sufficient to learn the spatial correspondence a data-driven manner without explicit 3D modeling, which consistent with the findings of recent work (Jin al., 2024).
18

---

Published as a conference paper at ICLR 2025
20" Difference in Azimuth
30" Difference in Azimuth
Epipolar Atlention
Full Attention
young and besutiful girl dressed in pink dress, sitting elegantly at grand piano.
horse and ricer in motion along the shoreline of beach.
Figure 11: Performance comparison of SynCamMaster with epipolar attention and full attention.
Table 6: Accuracy of camera control with epipolar attention and full attention.
Camera Representation
RotErr
TransErr
Epipolar Attention
0.10
0.59
Full Attention
0.12
0.58
Discussion on the Data Mixture Strategy the paper, we jointly train our model on multi-view video data, multi-view image data, and single-view video data with probabilities of 0.6, 0.2, and 0.2, respectively. We also explored the impact of different mixing ratios on the generated results. We found that higher proportion of multi-view image data disrupts the temporal continuity of the videos. On the other hand, using too much single-view video data causes the model to favor synthesizing views with small relative angles, affecting the camera accuracy. Therefore, sample multi-view image data and single-view video data with small probabilities.
E
MORE RESULTS
E. MORE COMPARISON WITH STATE-OF-THE-ART METHODS
In addition to using FID, FVD, and CLIP scores to evaluate the visual quality of the generated videos in Tab. I, we also assessed the generation quality of our method and the baselines using the video generation evaluation method VBench (Huang et al., 2024) as supplementary. Specifically, we randomly sample 100 prompts from the 300 prompts provided by VBench the categories of "animal', "human", and 'vehicles" for evaluation. We did not sample from categories such as 'plant' and 'scenery' because, given that we generate videos from fixed camera positions, static scenes would result in videos that stationary over time. The evaluation results are presented in Tab. 7. Our method outperforms SVD-XT (Blattmann et al., 2023) and CameraCtrl (He et al., 2024) across all metric dimensions and demonstrates performance comparable
trained I2V model.
Table 7: Quantitative comparison with baseline methods on VBench (Huang et al., 2024).
Subject Background Acsthetic Imaging Temporal Motion Consistency Consistency Quality
Method
Quality
Flickering Smoothness
M.V. Image SVD-XT
94.32
94.23
48.85
52.80
95.79
98.26
M.V. Image + CameraCtrl
95.91
96.40
50.44
52.85
96.79
98.73
M. V. Image + 12V-Ours
93.53
92.93
49.07
58.49
95.20
98.13
SysterMaster
97.84
96.55
50.50
58.30
98.95
99.27
Please refer Fig. 12 for qualitative comparison with the state-of-the-art methods.
19

---

Published as a conference paper at ICLR 2025
90" Difference in Azimuth
Wide Shot and Close-Up
M.V. Image
SVD-XT
M.V. Image CameraCtri
Image 12V-Ours
SynCamMaster
A gorilla is sitting on the ground, holding the carrot and taking small bitcs.
Alleek orange sports car on long straight truck. tires screeching accelerates, leaving behind trail of smoke and clust.
Figure 12: More comparison with state-of- thettrrrrt methods.
E.2 MORE RESULTS OF SYNCAMMASTER
More synthesized results of SynCamMaster are presented in Fig. 13-14.
20

---

Published as a conference paper at ICLR 2025
20 Difference in Elevation
30" Difference in Azimuth
Visuppiitt 2 Viswpelnt
Vinspoiint Viaepoint'2
An elephant wearing colorful birthday hat is walking along the sandy beach.
A yong girl is playing guitar:
Close- Up Shot
60" Difference in Azimuth
Vieeppintz Visupotats
Wewpooint: 1 Visupolut2
A chef expertly chopping onions well- equipped kitchen.
train traveling the tracks, moving slowly through the countryside.
120' Difference Azimuth
Close- Up and Wide Shot
Visupointt 2 Vionpoont
Viewpointt Visepaint2
close-up view of turtle slowly crawling across sandy beach.
gorilla sitting the ground, holding the carrot and taking small bites.
Close-Up Up and Wide Shot
Close-Up and Wide Shot
Visupptttt Viowpointts
Viespoint Visepointz
A monkey is skillfully creating latte art small coffee machine.
sports racing along coastal road.
Figure 13: More synthesized results of SynCamMaster.
21

---

Published as a conference paper at ICLR 2025
Close-Up and Wide Shot
Viewpeints Vionpalnt
Virsppoint' 2
sleek orange sports long, straight track, its screeching accelerates, leaving behind trail smoke and dust.
30" Difference in Azimuth
Visspoints Vloapoint
Viewpoint'2
A majestic eagle soaring through the clear blue sky:
30" Difference in Azimuth
Vinspeinss Vioapolntt
Virwpoint's 2
middle-aged woman, elegantly dressed traditional white cheongsam, carefully selecting bouquets
rose section of flower shop.
30" Difference in Azimuth
Viospoints Visupolats
Viewpointt 2
young child playing park crisp day.
15" Difference in Elevation
Vinapoint 3 Viouppoint
Viewppoint 2 Viewpeini
tractor is plowing in field, tractor moving slowly and steadily, leaving a trail freshly plowed soil behind
Figure 14: More synthesized results of SynCamMaster
22

---

Published as a conference paper at ICLR 2025
E.3 FAILURE CASES VISUALIZATION
We present the failure cases in Fig. 15. Firstly. we observe that inconsistencies in details may oceur when generating complex scenes, for example, in the first and second rows of Fig. 15, the bowls and plates on the table show content discrepancies between the two viewpoints. Secondly, since our model built upon a text-to-video base model, we also inherit some of the base model's '' short- comings. For instance, the generated hand movements of characters may exhibit inferior quality, as shown in the third and fourth rows.
30" Difference in Azimuth
Vieiapoiint 2 Virapoiatt
A fimily yothering Ingether in hawe a meanion dinnee on Chireee Nert Year's Erc.
Close-Up and Wide Shot
Viewpoint Vitrapointt
a holding writing a repuri.
Figure 15: Visualization of failure cases.
23


Reviews:
[{"review_id": "JhD5zyTeeu", "reviewer": "Reviewer_zZFD", "summary": "In this manuscript, the authors proposed a plug-and-play module that enhances a pretrained text-to-video model for multi-camera video generation. To achieve this goal, a multi-view synchronization module to ensure appearance and geometry consistency across different viewpoints. The synthesized videos using UE are generated as supplement for training. The proposed SynCamMaster is compared with related works on multi-view video generation, and provides competitive performance.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed method is simple yet effective for multi-camera video generation. \nThe paper is well-written and easy to understand.\nThe results are impressive.", "weaknesses": "[Pioneer Status] In terms of pioneering work, it has come to my attention that Sora possesses multi-camera video generation capabilities. Therefore, the claim in the article that \"To our knowledge, SynCamMaster pioneered multi-camera real-world video generation\" warrants reconsideration.\n\n[Novelty of SynCamMaster] The core design of SynCamMaster revolves around the inter-view synchronization module, which can be integrated into a pretrained text-to-video model. However, similar modules are commonly found in other generation models employing cross-view attention, which suggests that the novelty is incremental. It would be beneficial to emphasize how SynCamMaster distinguishes itself from existing methods.\n\n[Camera Encoder] The process for converting 12-dimensional extrinsic parameters to align with the dimensions of spatial features remains unclear. The encoding of camera extrinsics significantly influences the final outcome, so providing additional crucial details would be appreciated.\n\n[Typos] M.V. Images in table 1 and 4, they are mentioned in the manuscript. It is better to make it clear what is it. \n\n[Supplementary files] If the generated video files by all comparison methods are provided as supplementary files, it will provide a better and more intuitive way to understand the proposed SynCamMaster.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H2O4aIkiHa", "reviewer": "Reviewer_edwQ", "summary": "This paper focuses on the multi-view video generation with image/video and camera pose  conditions. Specifically, this paper propose a plug-and-play multi-view synchronization module to maintain appearance and geometry consistency. In addition, they collect a small set of 4D data and combine it together with 3D data to improve the generalization ability.", "questions": "My concern is mainly about the evaluation of multi-camera videos. Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method is novel and each module and design are reasonable with clear motivation.\n\n2. The proposed method achieved good performance both qualitatively and quantitatively.\n\n3. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed techniques.", "weaknesses": "1. I am kind of confused with the motivation of this task. The multi-camera videos generated by this work are several videos with fixed cameras. However, the videos we see and use in our daily life are commonly one monocular video with a dynamic camera, like the task that MotionCtrl focuses on. I hope the authors could provide more explanation about the task motivation.\n\n2. As claimed in the introduction section, multi-camera synchronized video generation is crucial for virtual filming. I am not an expert in virtual filming, but I feel it requires highly spatio-temporal consistent videos. Therefore, a good way to demonstrate the 4D consistency is to lift the generated samples to 4D representations, such as 4D gaussian splatting, and to show the rendered views. Plenoptic dataset [A] can be used for this evaluation.\n\n\n3. Evaluation data and metrics should be explained. The details of the evaluation data are not provided. It is more convincing to evaluate on the public dataset. For multi-camera video evaluation, the PSNR, SSIM, and LPIPS metrics on Plenoptic [A]  D-NeRF [B] datasets can be used. For video quality evaluation, VBench [C] would be better than ``100 manually collected text prompts’’. In addition, when evaluating the accuracy of camera control with TansErr, the estimated camera should be normalized before estimating the loss (following [D]), since the SfM is up to a scale and the scale mismatch can greatly increase the TransErr. In CameraCtrl, the scale problem is not significant as the compared methods are all trained on the Re10K dataset. However, this paper trains on a different dataset (DL3DV-10K) with MotionCtrl (Re10K) and the scale problem might be enhanced.\n\n*Reference*\n\n[A] Neural 3D Video Synthesis from Multi-view Video\n\n[B] D-NeRF: Neural Radiance Fields for Dynamic Scenes\n\n[C] VBench: Comprehensive Benchmark Suite for Video Generative Models\n\n[D] CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jl8HuTtslr", "reviewer": "Reviewer_bc9Q", "summary": "This paper presents a multi-view video generation approach based on an input textual prompt or monocular video. To achieve this goal, a multi-view synchronization module is incorporated into a pre-trained text-to-video model, where the goal is to maintain appearance and geometry consistency across multiple viewpoints. To address the scarcity issue of the training data, a progressive training scheme is designed that leverages multi-camera images, synthetic multi-view videos, and real-world monocular videos.", "questions": "1. Around the line #189, should $\\phi_t$ be $\\psi_t$? And what do $\\psi_t^{-1}$ and $\\psi_t'$ mean, respectively?\n\n2. Could you please provide more information about the real-world monocular videos used in this paper? Like duration, amount, how they are curated, etc.\n\n3. In the line #140, does $f$ indicate the number of frames?\n\n4. Is there any plan to release the code?\n\n5. Can more visual examples of the generated multi-view synthetic videos be shown?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The multi-view video generation is a novel task. This paper shows some promising results in this direction, where the temporal consistency within each single view and spatial consistency across different views are maintained reasonably well.\n\n2. The proposed way of using multi-view images, synthetic multi-view videos, and real-world monocular videos to train the model is novel. Ideally, we could train the model with multi-view videos, which are however hard to obtain, especially the high-quality real-world ones. The proposed approach shows an alternative approach to solve the data scarcity issue.", "weaknesses": "1. The proposed approach can only generate a single video at a novel viewpoint each time. Compared with generating multiple videos at different viewpoints simultaneously, the view consistency among the generated multi-view videos may not be good. Of course, the proposed approach can be used progressively generate further and further novel-view videos. But the view inconsistency may accumulate.\n\n2. I wish the authors could show more visual results so the quality of the generated videos could be more comprehensively gauged. In the current results (both in the paper and on the project webpage), the same example (e.g., the gorilla eating carrot, the elephant walking on a beach) appears many times.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k7LNLe25EM", "reviewer": "Reviewer_KEwM", "summary": "This paper introduces SynCamMaster, a pioneering framework for generating real-world videos across multiple camera perspectives. SynCamMaster also extends to novel view synthesis, allowing re-rendering of input videos from new viewpoints. To address the scarcity of multi-camera video data, the authors propose a hybrid data construction and training paradigm that enhances the model's generalization capabilities. Extensive experiments demonstrate that SynCamMaster significantly outperforms baseline methods, with particularly impressive visualization results.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Simplicity and Efficiency: The proposed method is both straightforward and efficient. This paper introduces a multi-view synchronization module designed to ensure camera-conditioned video consistency, enhancing realism and alignment across views.\n\n2. Innovative Data Construction: This paper addresses the scarcity of multi-view video data by developing a robust hybrid-data collection pipeline. This process effectively collects diverse and comprehensive training data.\n\n3. Thoughtful Training Strategy: The paper provides a clear breakdown of each stage in the training strategy, along with detailed insights into the motivations behind each stage. These insights offer readers a better understanding of the model's optimization process and rationale.\n\n4. High-Quality Visualizations: The visualization results presented in the paper are impressive.\n\n5. Clarity of Presentation: The paper is well-organized and articulated in a way that is accessible and easy to understand.", "weaknesses": "Lack of Comparison with MotionCtrl: A drawback is the absence of a comparative evaluation with MotionCtrl (\"MotionCtrl: A Unified and Flexible Motion Controller for Video Generation\").", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qSCo5eW4my", "reviewer": "Reviewer_bsPL", "summary": "The paper \"SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints\" introduces a novel approach for generating synchronized multi-view videos from arbitrary viewpoints, addressing challenges in open-domain video generation that require consistent geometry and appearance across diverse camera angles. The authors leverage a pre-trained text-to-video model and enhance it with a multi-view synchronization module, designed to integrate inter-view feature consistency, which is achieved by incorporating a camera encoder and a unique attention mechanism. Additionally, they introduce a hybrid dataset and progressive training techniques to improve the generalization of the model to open-domain scenarios, effectively overcoming the scarcity of synchronized multi-view video data. Their experiments demonstrate superior cross-view synchronization and visual fidelity compared to baseline methods.", "questions": "1.The citations for each method in tables should be added.\n2.It appears a new synthetic dataset was proposed in this work for training the model. Will you decide to release this dataset? The dataset could be considered a contribution to the paper, so I suggest naming this synthetic dataset for readability.\n3.In the qualitative experiments, you mention a polar bear case, but Figure 5 uses an elephant and a bus, and I couldn’t find polar bear images in the appendix.\n4.In the quantitative experiments, your method performs worse than I2V-ours in terms of visual quality. I would like to see some analysis on this comparison between your results and those of I2V-ours.\n5.Table 1 entries for I2V and SVD-XT should be unified if they refer to the same method for consistency.\n6.In Figure 6(a), joint training strategy ablation study results are shown, but without ground truth to assess and with images that are too small, it is difficult to judge if the strategy improved the generation results, or if the effect is simply not significant. In contrast, Figure 6(b) shows clear improvements.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "This paper introduces a forward-looking approach by achieving multi-camera real-world video generation, a pioneering advancement that is likely to drive significant progress in several areas in computer vision. Extensive experiments and ablation studies support the reliability of the proposed model, and the project website showcases video results that validate the method's effectiveness.", "weaknesses": "1.I believe the challenge identification section is unclear. Initially, the authors mention two challenges in this field, then state, \"our research targets open-domain video generation from multiple arbitrary viewpoints, which has not been investigated.\" However, they do not clearly explain the reason for this research, nor do they clarify the limitations of existing methods and the higher demands that necessitate this research.\n2.The explanation of the model structure and model formulation is severely lacking, making it difficult to connect the input-to-output process in the model with the text description, including the loss function and optimization mechanism. Specifically, the figure should indicate what the input is, which step produces the features, and how the loss is calculated, among other key details.\n3.How does SynCamMaster determine the rotation center and radius when generating rotating camera viewpoints? I think that for a method designed to generate multi-view videos, this is an aspect that requires clarification.\n4. Although the paper provides results for camera rotations like 90 degrees or 20 degrees, how can you quantitatively assess that the generated images match the rotation you specified? Without ground truth for reference, it is challenging to determine whether the rotation results are accurate. Could the authors use a COLMAP or similar method to register the spatial relationships of the generated multi-view images in camera space?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YPoRx0WCCh", "reviewer": "Reviewer_zv9N", "summary": "This paper presents a multiview video dataset rendered using Unreal Engine. Additionally, it proposes a multiview video generative model architecture and training strategy to adapt a pretrained video diffusion model for simultaneous multiview generation.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) A multiview video dataset is presented.\n(2) A multiview video diffusion model is trained using the proposed dataset.\n(3) Comprehensive visualization results of the proposed method are provided.", "weaknesses": "(1) Encoding the camera pose matrix directly into the diffusion process as a condition is overly coarse. The authors should consider transforming camera poses into more explicit representations, such as ray positional encoding [1].\n(2) Since the multi-view video is rendered rather than directly collected from real-world scenes, a significant concern is ensuring that the model learns only multi-view-related information rather than features specific to Unreal Engine-rendered models, materials, and textures. For instance, in Fig. 6 (a), how to avoid the output as the rendered scene? Authors should also introduce some real-world multiview videos to help balance the training of the diffusion model.\n(3) The rotation angle in Fig. 6 (b) appears noticeably less than 90 degrees. Additionally, the precision of camera control is low, as indicated in Table 4.\n(4) The proposed multi-view synchronization module is essentially a cross-attention layer, which lacks explicit modeling of 3D information.\n(5) Given that camera poses are static in each rendered video under the proposed setup, it's unnecessary to render arbitrary camera poses. We can greatly lessen the burden of generative models and improve it accuracy by capturing and training more fewer camera poses, e.g., 6 views evenly distributed around 360 degrees, and applying 3D reconstruction on the generated views to get other different views.\n[1] Gao, Ruiqi, et al. \"Cat3d: Create anything in 3d with multi-view diffusion models.\" arXiv preprint arXiv:2405.10314 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZSQbAzniA4", "reviewer": "Reviewer_kLvx", "summary": "This paper introduces a plug-and-play module for generating consistent videos from arbitrary viewpoints using 6 DoF camera poses. Unlike methods focused on 4D reconstruction of single objects, this approach enhances a pre-trained text-to-video model for synchronized multi-camera generation, ensuring appearance and geometry consistency across viewpoints. To overcome limited training data, this paper uses a progressive training scheme, leveraging multi-camera and monocular videos along with Unreal Engine-generated data.", "questions": "1. Will the data preparation process and resulting dataset be made publicly available? The paper states that data was collected from a game engine, which could be valuable for advancing research in multi-view video generation. Public access to this data would enable reproducibility, facilitate benchmarking, and help other researchers develop and test models in a similar setting.\n\n2. How were the data mixture probabilities (0.6, 0.2, 0.2) chosen, and what role do they play in the model’s performance? It would be helpful to understand the rationale behind this distribution and whether alternative ratios were explored. Insights into how these probabilities impact training dynamics and model generalization would also clarify the importance of the data mixture balance in achieving optimal results.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The approach of generating synchronized multi-view videos is a strong point, as these videos offer applications beyond traditional 4D reconstruction, such as enhancing virtual filming. By maintaining dynamic consistency across views, this method broadens the potential use cases in various real-time and interactive settings.\n\n2. The visual quality of the generated videos is impressive, particularly with respect to the wide camera baselines achieved. This indicates the model’s robustness in handling significant viewpoint shifts, preserving both appearance and geometric consistency across diverse perspectives, which is essential for applications requiring high-fidelity visual continuity.\n\n3. Leveraging multi-view video data from game engines for fine-tuning is a smart approach, especially given the scarcity of high-quality multi-camera real-world datasets. The method demonstrates effective generalization to real-world data, showing that game engine data can be a viable supplement for improving model performance in open-world video generation tasks.", "weaknesses": "1. Although 4D video reconstruction is highlighted as a key application, the paper does not provide specific results on this. Including an evaluation of 4D reconstruction performance (like 4DGS) on the output synchronized videos would strengthen the contribution and showcase the practical utility of the generated multi-view videos in real-world 4D reconstruction tasks.\n\n2. The use of camera conditioning is potentially limited, as it may not handle large out-of-distribution camera pose translations during inference (e.g., for training only 3.5m - 9m shifts). Testing the model on such out-of-distribution poses would provide valuable insight into its robustness and adaptability to extreme viewpoint variations, which are common in real-world applications.\n\n3. Multi-view consistency could be better assessed. A qualitative evaluation, such as a cycle consistency check, could help—rendering videos from generated camera poses and then re-rendering from the original poses would allow comparison. The differences between these two renderings could serve as a metric for the model’s ability to maintain consistent content across varying views, giving a clearer measure of its multi-view coherence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)