SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix

Authors: Peng Dai, Feitong Tan, Qiangeng Xu, David Futschik, Ruofei Du, Sean Fanello, XIAOJUAN QI, Yinda Zhang
Primary area: generative models
Keywords: video generation, stereoscopic video, inpainting, diffusion model

Abstract:
Video generation models have demonstrated great capability of producing impressive monocular videos, however, the generation of 3D stereoscopic video remains under-explored. We propose a pose-free and training-free approach for generating 3D stereoscopic videos using an off-the-shelf monocular video generation model. Our method warps a generated monocular video into camera views on stereoscopic baseline using estimated video depth, and employs a novel frame matrix video inpainting framework. The framework leverages the video generation model to inpaint frames observed from different timestamps and views. This effective approach generates consistent and semantically coherent stereoscopic videos without scene optimization or model fine-tuning. Moreover, we develop a disocclusion boundary re-injection scheme that further improves the quality of video inpainting by alleviating the negative effects propagated from disoccluded areas in the latent space. We validate the efficacy of our proposed method by conducting experiments on videos from various generative models, including Sora [4], Lumiere [2], WALT [8], and Zeroscope [12]. The experiments demonstrate that our method has a significant improvement over previous methods. Project page at https://daipengwa.github.io/SVG_ProjectPage/

Decision: accept (poster)
Reviewer scores: 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729082664080

thread_type:
official_review

<thread_object>
creator: reviewer_dhei

<content>
summary:
This paper proposes a training-free framework to generate stereo video from single video sequence by using retrained video generation model. The proposed frame matrix leverages the power of video generation model and the joint optimization idea to generate semantically consistent and temporally smooth content, which offers valuable insights and can provide reference directions for future work.. However, besides the frame matrix, the other modules mentioned in methods are more like tricks and are all followed by existing works. The extensive results show that this paper achieve SOTA performance in training-free manner. However, the drawbacks are obvious, the performance is significantly worse than the training methods, and the process is slow due to the need for multiple iterations.

strengths:
1. This paper proposes a training-free manner to generate stereo video from monocular video and achieve SOTA performance in training-free manner.

2. The proposed denoising frame matrix uses pre-trained video generation model as the inpainting model, which is the first one to do this in stereo video generation field and offers insight for leveraging video generation model to assist this task.

weaknesses:
1. The analysis for denoising frame matrix is insufficient. Although the authors provide the reason for using video generation model, the theory and the high-level reason of why it works are not clear, please show the theoretical analysis of the proposed frame matrix.

2. Lack citations for the methodology followed by other methods. The presentation in the paper contains certain misleading and deceptive elements. Line 153-161, the viewpoint transfer part is widely used in novel view synthesis [R1, R2] and stereo image generation [R3]. Additionally, the proposed boundary re-injection is just a trick, which can not be a contribution. These parts should be revised for better presentation.

[R1] Tucker, R.; and Snavely, N. 2020. Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 551–560.

[R2] Han, Y.; Wang, R.; and Yang, J. 2022. Single-view view synthesis in the wild with learned adaptive multiplane images. In ACM SIGGRAPH 2022 Conference Proceedings, 1–8.

[R3] Wang, X.; Wu, C.; Yin, S.; Ni, M.; Wang, J.; Li, L.; Yang, Z.; Yang, F.; Wang, L.; Liu, Z.; et al. 2023b. Learning 3D Photography Videos via Self-supervised Diffusion on Single Images. arXiv preprint arXiv:2302.10781.

3. Insufficient evaluation metics. In [R1], [R2], [R3], they will use SSIM, PSNR, LPIPS evaluation metrics in novel-view image, which can also be used in stereo video generation. What's more, the temporal consistency is just justified in user study, the quantitative results such as Fréchet Video Distance should also be given.

4. Lack comparison with highly related methods. The methods compared in this paper is too simple. As mentioned in Q2, [R1], [R2], [R3] are the highly related stereo image generation methods. Although they need to train, they own the advantages of high performance and fast inference speed, the authors should give the result in the paper and clarify the strength of their methods.

questions:
See the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730520898567

thread_type:
official_review

<thread_object>
creator: reviewer_zjcl

<content>
summary:
The paper addresses a gap in the generation of stereoscopic videos, particularly corresponding to the advancements in VR/AR technologies. The authors introduce an interesting pose-free and training-free framework that aims to improve the generation of high-quality 3D stereoscopic videos from monocular video inputs. The author utilizes video generation models to enhance 3D consistency while addressing challenges such as occlusion and temporal stability.

strengths:
- The paper is well-written and easy to follow. 
- The proposed frame matrix representation is reasonable, and the extensive experiments support its functionality. Therefore, this paper is sufficiently novel. 
- The proposed method demonstrates a strong understanding of the challenges specific to 3D video generation, including issues with depth estimation and video inpainting. 
- I believe that sufficient experiments and ablation studies are presented to support the approach.

weaknesses:
The main weaknesses of the proposed method are the disocclusion boundary artifacts, slightly lower temporal consistency compared to Deep3D, and the need for further improvements in holistic perceptual consistency, especially for certain subjects like human faces.

questions:
Missing some reference about multi-view synthesis, please consider reference them:

[1] Chen, Zilong, et al. "V3d: Video diffusion models are effective 3d generators." arXiv preprint arXiv:2403.06738 (2024).
[2] Zuo, Qi, et al. "Videomv: Consistent multi-view generation based on large video generative model." arXiv preprint arXiv:2403.12010 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705469542

thread_type:
official_review

<thread_object>
creator: reviewer_p3fm

<content>
summary:
This paper aims to synthesize 3D stereoscopic videos using an off-the-shelf monocular video generation model without finetuning. Concretely, the authors leverage the estimated video depth and propose a novel frame matrix to denoise both spatially and temporally, and thus the model is aware of the exsiting information on the left view and synthesizes consistent right-view videos accordingly. A disocclusion boundary re-injection scheme is also proposed to solve the boundary problem.

strengths:
1. The proposed method is pose-free and training-free.

2. This paper is clearly written and easy to understand.

3. Extensive experiments demonstrate the effectiveness of the proposed frame matrix and the disocclusion boundary re-injection scheme.

weaknesses:
1. As the model denoises along both temporal and spatial dimensions, one experiment that is missing is the investigation of varying the number of cameras between the left and right views. How does this variation impact the final quality and overall efficiency of the process? Is it feasible to use fewer internal camera views to save time?
2. Currently all the experiments have been conducted on the synthesized videos. It would be beneficial to explore how the results look like when applied to real-world videos.
3. The model heavily depends on a pre-trained depth estimation model, which can overlook thin structures and sometimes produce inaccurate results.

questions:
1. Most videos feature a single movable object or minor movements. How does the proposed method perform when applied to more complex scenes, such as those with multiple moving objects and significant motions?
2. Typos:

L225 Denosing -> Denoising

L266 refence → reference
</content>
</thread_object>

</thread>
<thread>
creation date:
1732394233304

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thanks reviewers and ACs for their time and efforts.

Our rebuttal materials include official comments and a revised PDF. In the new PDF, we have made the following changes, which are highlighted in blue for easy distinction from the previous content.

- In Table 2, we added a new metric FVD. 

- In Fig. 18, we incorporated our boundary re-injection design into the existing image inpainting method.

- In Fig. 19, we studied the number of cameras used between the left and right views.

- in Fig. 20 and Fig. 21, we included comparisons with training-based single-image view synthesis methods.

- In Fig. 22, we displayed diverse outputs of our method.

- In Fig. 23, we conducted experiments on complex real-world videos, including fast-moving objects as well as multiple moving objects.

- We corrected several typos and included the missing references
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: This paper presents a method for synthesizing 3D stereoscopic videos using a pre-trained monocular video generation model. The basic idea is to create a warped version of a video using estimated video depth and inpaint it using a video generation model. The core contribution is a novel frame matrix to denoise both spatially and temporally and, therefore, achieve coherent stereoscopic video generation. 

Strength:
- Pose-free and training-free method.
- Extensive experimental validation (including videos from multiple models, e.g., Sora, Lumiere, WALT, and Zeroscope)

Weakness: 
- Most videos tested are not complex, e.g., only single movable objects.
- There are artifacts at the disocclusion boundary and temporal inconsistency. 
- Insufficient evaluation metrics

Justification: 
- All three reviewers are positive about this paper. This work fills a gap in the existing literature on generating 3D stereoscopic videos. The method of using depth-warped video and inpainting is not new. However, using the proposed frame matrix in the video noising framework is interesting and novel. Most of the raised concerns by reviewers have been resolved in the rebuttal stage. The AC thus recommends to accept.