SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints

Authors: Jianhong Bai, Menghan Xia, Xintao Wang, Ziyang Yuan, Zuozhu Liu, Haoji Hu, Pengfei Wan, Di ZHANG
Primary area: generative models
Keywords: Video Generation, Diffusion Model

Abstract:
Recent advancements in video diffusion models demonstrate remarkable capabilities in simulating real-world dynamics and 3D consistency. This progress motivates us to explore the potential of these models to maintain dynamic consistency across diverse viewpoints, a feature highly sought after in applications like virtual filming. Unlike existing methods focused on multi-view generation of single objects for 4D reconstruction, our interest lies in generating open-world videos from arbitrary viewpoints, incorporating six degrees of freedom (6 DoF) camera poses.
To achieve this, we propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring consistent content across different viewpoints. Specifically, we introduce a multi-view synchronization module designed to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of high-quality training data, we also propose a progressive training scheme that leverages multi-camera images and monocular videos as a supplement to Unreal Engine-rendered multi-camera videos. This comprehensive approach significantly benefits our model.
Experimental results demonstrate the superiority of our proposed method over existing competitors and several baselines. Furthermore, our method enables intriguing extensions, such as re-rendering a video from multiple novel viewpoints. Project webpage: https://jianhongbai.github.io/SynCamMaster/

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730687203275

thread_type:
official_review

<thread_object>
creator: reviewer_klvx

<content>
summary:
This paper introduces a plug-and-play module for generating consistent videos from arbitrary viewpoints using 6 DoF camera poses. Unlike methods focused on 4D reconstruction of single objects, this approach enhances a pre-trained text-to-video model for synchronized multi-camera generation, ensuring appearance and geometry consistency across viewpoints. To overcome limited training data, this paper uses a progressive training scheme, leveraging multi-camera and monocular videos along with Unreal Engine-generated data.

strengths:
1. The approach of generating synchronized multi-view videos is a strong point, as these videos offer applications beyond traditional 4D reconstruction, such as enhancing virtual filming. By maintaining dynamic consistency across views, this method broadens the potential use cases in various real-time and interactive settings.

2. The visual quality of the generated videos is impressive, particularly with respect to the wide camera baselines achieved. This indicates the model’s robustness in handling significant viewpoint shifts, preserving both appearance and geometric consistency across diverse perspectives, which is essential for applications requiring high-fidelity visual continuity.

3. Leveraging multi-view video data from game engines for fine-tuning is a smart approach, especially given the scarcity of high-quality multi-camera real-world datasets. The method demonstrates effective generalization to real-world data, showing that game engine data can be a viable supplement for improving model performance in open-world video generation tasks.

weaknesses:
1. Although 4D video reconstruction is highlighted as a key application, the paper does not provide specific results on this. Including an evaluation of 4D reconstruction performance (like 4DGS) on the output synchronized videos would strengthen the contribution and showcase the practical utility of the generated multi-view videos in real-world 4D reconstruction tasks.

2. The use of camera conditioning is potentially limited, as it may not handle large out-of-distribution camera pose translations during inference (e.g., for training only 3.5m - 9m shifts). Testing the model on such out-of-distribution poses would provide valuable insight into its robustness and adaptability to extreme viewpoint variations, which are common in real-world applications.

3. Multi-view consistency could be better assessed. A qualitative evaluation, such as a cycle consistency check, could help—rendering videos from generated camera poses and then re-rendering from the original poses would allow comparison. The differences between these two renderings could serve as a metric for the model’s ability to maintain consistent content across varying views, giving a clearer measure of its multi-view coherence.

questions:
1. Will the data preparation process and resulting dataset be made publicly available? The paper states that data was collected from a game engine, which could be valuable for advancing research in multi-view video generation. Public access to this data would enable reproducibility, facilitate benchmarking, and help other researchers develop and test models in a similar setting.

2. How were the data mixture probabilities (0.6, 0.2, 0.2) chosen, and what role do they play in the model’s performance? It would be helpful to understand the rationale behind this distribution and whether alternative ratios were explored. Insights into how these probabilities impact training dynamics and model generalization would also clarify the importance of the data mixture balance in achieving optimal results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694047998

thread_type:
official_review

<thread_object>
creator: reviewer_zv9n

<content>
summary:
This paper presents a multiview video dataset rendered using Unreal Engine. Additionally, it proposes a multiview video generative model architecture and training strategy to adapt a pretrained video diffusion model for simultaneous multiview generation.

strengths:
(1) A multiview video dataset is presented.
(2) A multiview video diffusion model is trained using the proposed dataset.
(3) Comprehensive visualization results of the proposed method are provided.

weaknesses:
(1) Encoding the camera pose matrix directly into the diffusion process as a condition is overly coarse. The authors should consider transforming camera poses into more explicit representations, such as ray positional encoding [1].
(2) Since the multi-view video is rendered rather than directly collected from real-world scenes, a significant concern is ensuring that the model learns only multi-view-related information rather than features specific to Unreal Engine-rendered models, materials, and textures. For instance, in Fig. 6 (a), how to avoid the output as the rendered scene? Authors should also introduce some real-world multiview videos to help balance the training of the diffusion model.
(3) The rotation angle in Fig. 6 (b) appears noticeably less than 90 degrees. Additionally, the precision of camera control is low, as indicated in Table 4.
(4) The proposed multi-view synchronization module is essentially a cross-attention layer, which lacks explicit modeling of 3D information.
(5) Given that camera poses are static in each rendered video under the proposed setup, it's unnecessary to render arbitrary camera poses. We can greatly lessen the burden of generative models and improve it accuracy by capturing and training more fewer camera poses, e.g., 6 views evenly distributed around 360 degrees, and applying 3D reconstruction on the generated views to get other different views.
[1] Gao, Ruiqi, et al. "Cat3d: Create anything in 3d with multi-view diffusion models." arXiv preprint arXiv:2405.10314 (2024).

questions:
Please refer to the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695200367

thread_type:
official_review

<thread_object>
creator: reviewer_bspl

<content>
summary:
The paper "SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints" introduces a novel approach for generating synchronized multi-view videos from arbitrary viewpoints, addressing challenges in open-domain video generation that require consistent geometry and appearance across diverse camera angles. The authors leverage a pre-trained text-to-video model and enhance it with a multi-view synchronization module, designed to integrate inter-view feature consistency, which is achieved by incorporating a camera encoder and a unique attention mechanism. Additionally, they introduce a hybrid dataset and progressive training techniques to improve the generalization of the model to open-domain scenarios, effectively overcoming the scarcity of synchronized multi-view video data. Their experiments demonstrate superior cross-view synchronization and visual fidelity compared to baseline methods.

strengths:
This paper introduces a forward-looking approach by achieving multi-camera real-world video generation, a pioneering advancement that is likely to drive significant progress in several areas in computer vision. Extensive experiments and ablation studies support the reliability of the proposed model, and the project website showcases video results that validate the method's effectiveness.

weaknesses:
1.I believe the challenge identification section is unclear. Initially, the authors mention two challenges in this field, then state, "our research targets open-domain video generation from multiple arbitrary viewpoints, which has not been investigated." However, they do not clearly explain the reason for this research, nor do they clarify the limitations of existing methods and the higher demands that necessitate this research.
2.The explanation of the model structure and model formulation is severely lacking, making it difficult to connect the input-to-output process in the model with the text description, including the loss function and optimization mechanism. Specifically, the figure should indicate what the input is, which step produces the features, and how the loss is calculated, among other key details.
3.How does SynCamMaster determine the rotation center and radius when generating rotating camera viewpoints? I think that for a method designed to generate multi-view videos, this is an aspect that requires clarification.
4. Although the paper provides results for camera rotations like 90 degrees or 20 degrees, how can you quantitatively assess that the generated images match the rotation you specified? Without ground truth for reference, it is challenging to determine whether the rotation results are accurate. Could the authors use a COLMAP or similar method to register the spatial relationships of the generated multi-view images in camera space?

questions:
1.The citations for each method in tables should be added.
2.It appears a new synthetic dataset was proposed in this work for training the model. Will you decide to release this dataset? The dataset could be considered a contribution to the paper, so I suggest naming this synthetic dataset for readability.
3.In the qualitative experiments, you mention a polar bear case, but Figure 5 uses an elephant and a bus, and I couldn’t find polar bear images in the appendix.
4.In the quantitative experiments, your method performs worse than I2V-ours in terms of visual quality. I would like to see some analysis on this comparison between your results and those of I2V-ours.
5.Table 1 entries for I2V and SVD-XT should be unified if they refer to the same method for consistency.
6.In Figure 6(a), joint training strategy ablation study results are shown, but without ground truth to assess and with images that are too small, it is difficult to judge if the strategy improved the generation results, or if the effect is simply not significant. In contrast, Figure 6(b) shows clear improvements.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730945602262

thread_type:
official_review

<thread_object>
creator: reviewer_kewm

<content>
summary:
This paper introduces SynCamMaster, a pioneering framework for generating real-world videos across multiple camera perspectives. SynCamMaster also extends to novel view synthesis, allowing re-rendering of input videos from new viewpoints. To address the scarcity of multi-camera video data, the authors propose a hybrid data construction and training paradigm that enhances the model's generalization capabilities. Extensive experiments demonstrate that SynCamMaster significantly outperforms baseline methods, with particularly impressive visualization results.

strengths:
1. Simplicity and Efficiency: The proposed method is both straightforward and efficient. This paper introduces a multi-view synchronization module designed to ensure camera-conditioned video consistency, enhancing realism and alignment across views.

2. Innovative Data Construction: This paper addresses the scarcity of multi-view video data by developing a robust hybrid-data collection pipeline. This process effectively collects diverse and comprehensive training data.

3. Thoughtful Training Strategy: The paper provides a clear breakdown of each stage in the training strategy, along with detailed insights into the motivations behind each stage. These insights offer readers a better understanding of the model's optimization process and rationale.

4. High-Quality Visualizations: The visualization results presented in the paper are impressive.

5. Clarity of Presentation: The paper is well-organized and articulated in a way that is accessible and easy to understand.

weaknesses:
Lack of Comparison with MotionCtrl: A drawback is the absence of a comparative evaluation with MotionCtrl ("MotionCtrl: A Unified and Flexible Motion Controller for Video Generation").

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731035911462

thread_type:
official_review

<thread_object>
creator: reviewer_bc9q

<content>
summary:
This paper presents a multi-view video generation approach based on an input textual prompt or monocular video. To achieve this goal, a multi-view synchronization module is incorporated into a pre-trained text-to-video model, where the goal is to maintain appearance and geometry consistency across multiple viewpoints. To address the scarcity issue of the training data, a progressive training scheme is designed that leverages multi-camera images, synthetic multi-view videos, and real-world monocular videos.

strengths:
1. The multi-view video generation is a novel task. This paper shows some promising results in this direction, where the temporal consistency within each single view and spatial consistency across different views are maintained reasonably well.

2. The proposed way of using multi-view images, synthetic multi-view videos, and real-world monocular videos to train the model is novel. Ideally, we could train the model with multi-view videos, which are however hard to obtain, especially the high-quality real-world ones. The proposed approach shows an alternative approach to solve the data scarcity issue.

weaknesses:
1. The proposed approach can only generate a single video at a novel viewpoint each time. Compared with generating multiple videos at different viewpoints simultaneously, the view consistency among the generated multi-view videos may not be good. Of course, the proposed approach can be used progressively generate further and further novel-view videos. But the view inconsistency may accumulate.

2. I wish the authors could show more visual results so the quality of the generated videos could be more comprehensively gauged. In the current results (both in the paper and on the project webpage), the same example (e.g., the gorilla eating carrot, the elephant walking on a beach) appears many times.

questions:
1. Around the line #189, should $\phi_t$ be $\psi_t$? And what do $\psi_t^{-1}$ and $\psi_t'$ mean, respectively?

2. Could you please provide more information about the real-world monocular videos used in this paper? Like duration, amount, how they are curated, etc.

3. In the line #140, does $f$ indicate the number of frames?

4. Is there any plan to release the code?

5. Can more visual examples of the generated multi-view synthetic videos be shown?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731066409726

thread_type:
official_review

<thread_object>
creator: reviewer_edwq

<content>
summary:
This paper focuses on the multi-view video generation with image/video and camera pose  conditions. Specifically, this paper propose a plug-and-play multi-view synchronization module to maintain appearance and geometry consistency. In addition, they collect a small set of 4D data and combine it together with 3D data to improve the generalization ability.

strengths:
1. The method is novel and each module and design are reasonable with clear motivation.

2. The proposed method achieved good performance both qualitatively and quantitatively.

3. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed techniques.

weaknesses:
1. I am kind of confused with the motivation of this task. The multi-camera videos generated by this work are several videos with fixed cameras. However, the videos we see and use in our daily life are commonly one monocular video with a dynamic camera, like the task that MotionCtrl focuses on. I hope the authors could provide more explanation about the task motivation.

2. As claimed in the introduction section, multi-camera synchronized video generation is crucial for virtual filming. I am not an expert in virtual filming, but I feel it requires highly spatio-temporal consistent videos. Therefore, a good way to demonstrate the 4D consistency is to lift the generated samples to 4D representations, such as 4D gaussian splatting, and to show the rendered views. Plenoptic dataset [A] can be used for this evaluation.


3. Evaluation data and metrics should be explained. The details of the evaluation data are not provided. It is more convincing to evaluate on the public dataset. For multi-camera video evaluation, the PSNR, SSIM, and LPIPS metrics on Plenoptic [A]  D-NeRF [B] datasets can be used. For video quality evaluation, VBench [C] would be better than ``100 manually collected text prompts’’. In addition, when evaluating the accuracy of camera control with TansErr, the estimated camera should be normalized before estimating the loss (following [D]), since the SfM is up to a scale and the scale mismatch can greatly increase the TransErr. In CameraCtrl, the scale problem is not significant as the compared methods are all trained on the Re10K dataset. However, this paper trains on a different dataset (DL3DV-10K) with MotionCtrl (Re10K) and the scale problem might be enhanced.

*Reference*

[A] Neural 3D Video Synthesis from Multi-view Video

[B] D-NeRF: Neural Radiance Fields for Dynamic Scenes

[C] VBench: Comprehensive Benchmark Suite for Video Generative Models

[D] CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation

questions:
My concern is mainly about the evaluation of multi-camera videos. Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731158058173

thread_type:
official_review

<thread_object>
creator: reviewer_zzfd

<content>
summary:
In this manuscript, the authors proposed a plug-and-play module that enhances a pretrained text-to-video model for multi-camera video generation. To achieve this goal, a multi-view synchronization module to ensure appearance and geometry consistency across different viewpoints. The synthesized videos using UE are generated as supplement for training. The proposed SynCamMaster is compared with related works on multi-view video generation, and provides competitive performance.

strengths:
The proposed method is simple yet effective for multi-camera video generation. 
The paper is well-written and easy to understand.
The results are impressive.

weaknesses:
[Pioneer Status] In terms of pioneering work, it has come to my attention that Sora possesses multi-camera video generation capabilities. Therefore, the claim in the article that "To our knowledge, SynCamMaster pioneered multi-camera real-world video generation" warrants reconsideration.

[Novelty of SynCamMaster] The core design of SynCamMaster revolves around the inter-view synchronization module, which can be integrated into a pretrained text-to-video model. However, similar modules are commonly found in other generation models employing cross-view attention, which suggests that the novelty is incremental. It would be beneficial to emphasize how SynCamMaster distinguishes itself from existing methods.

[Camera Encoder] The process for converting 12-dimensional extrinsic parameters to align with the dimensions of spatial features remains unclear. The encoding of camera extrinsics significantly influences the final outcome, so providing additional crucial details would be appreciated.

[Typos] M.V. Images in table 1 and 4, they are mentioned in the manuscript. It is better to make it clear what is it. 

[Supplementary files] If the generated video files by all comparison methods are provided as supplementary files, it will provide a better and more intuitive way to understand the proposed SynCamMaster.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732218108690

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revision

comment:
We thank all the reviewers for your time and thoughtful feedback. We revised our manuscript and submitted a new version for review. We also add more video results on our anonymous project page (https://syncammaster.github.io/SynCamMaster/) for your reference.

## Summary of Revision
### Project Page
1. Including more synthesized results of SynCamMaster in "Demos" and "More Results" (Reviewer bc9Q).
2. Adding video comparison of SynCamMaster and baseline methods including MotionCtrl [1] (Reviewer zZFD, KEwM).
3. Results on 4D reconstruction (Reviewer edwQ, kLvx).
4. Results with Out-of-Distribution Cameras (Reviewer kLvx).
### Main Paper
1. We revise the introduction section to better clarify the limitations of existing methods (Reviewer bsPL).
2. We refine the method pipeline figure for a clear correspondence with the text (Reviewer bsPL).
3. We add the introduction of the camera encoder (Reviewer zZFD).
4. We normalize the translation vector following [2], update the results in Table 3, and add the corresponding explanation (Reviewer edwQ).
5. We fix some typos.
### Appendix
1. Detailed introduction of the base text-to-video generation model in Appendix A (Reviewers bsPL).
2. Description of the evaluation set construction in Appendix C (Reviewers edwQ).
3. Comparison of using different camera representations in Appendix D (Reviewers zv9N).
4. Results of using epipolar attention as explicit 3D modeling in Appendix D (Reviewers zv9N).
5. Discussion on the data mixture strategy in Appendix D (Reviewers kLvx).
6. Comparison with baseline methods on VBench in Appendix E (Reviewers edwQ).

[1] Wang, Zhouxia, et al. "Motionctrl: A unified and flexible motion controller for video generation." ACM SIGGRAPH 2024 Conference Papers. 2024.

[2] Xu, Dejia, et al. "CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation." arXiv preprint arXiv:2406.02509 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732565848127

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks for Your Review

comment:
Dear Reviewers,

Thanks again for the constructive comments and the time you dedicate to the paper! Many improvements have been made according to your suggestions and other reviewers' comments. We also updated the manuscript and summarized the changes below. We hope that our responses and the revision will address your concerns.

Since the discussion is about to close, we would be grateful if you could take some time to review our responses and the revised manuscript. We are glad to follow up with your further comments.

Thanks a lot again, and with sincerest best wishes

Submission 1274 Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732859185203

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Friendly Reminder

comment:
Dear Reviewers:

Thanks again for the constructive comments and the time you dedicate to the paper! 

**Since the discussion is about to close (Dec. 2nd)**, we would be grateful if you would kindly let us know of any other concerns and if we could further assist in clarifying any other issues.

Thanks a lot, and with sincerest best wishes

Submission 1274 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper receives final ratings of 6,6,6,8,6,3,6. The AC follows the majority of the reviewers to accept the paper. The reasons for accepting the paper are: 1) Novelty of the method and there are clear motivations in the design of each component. The proposed method is simple and efficient. A thoughtful training strategy is proposed. 2) The multi-view video generation is a novel task. 3) The proposed approach solves the data scarcity issue. 4) The experimental results show that the proposed method achieved good performance both qualitatively and quantitatively. 5) There are extensive experiments and ablation studies to support the effectiveness of the proposed techniques. 

The reviewer who gave a 3 did not give any strong reasons for rejection and did not respond to the discussion and clarification of the authors. Considering the reasons to accept the paper is more compelling from other reviewers, the reviewer decided to disregard the 3 and accept the paper.