SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix
Authors: Peng Dai
Feitong Tan
Qiangeng Xu
David Futschik
Ruofei Du
Sean Fanello
XIAOJUAN QI
Yinda Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: video generation
stereoscopic video
inpainting
diffusion model
Primary area: generative models

Abstract: Video generation models have demonstrated great capability of producing impressive monocular videos, however, the generation of 3D stereoscopic video remains under-explored. We propose a pose-free and training-free approach for generating 3D stereoscopic videos using an off-the-shelf monocular video generation model. Our method warps a generated monocular video into camera views on stereoscopic baseline using estimated video depth, and employs a novel frame matrix video inpainting framework. The framework leverages the video generation model to inpaint frames observed from different timestamps and views. This effective approach generates consistent and semantically coherent stereoscopic videos without scene optimization or model fine-tuning. Moreover, we develop a disocclusion boundary re-injection scheme that further improves the quality of video inpainting by alleviating the negative effects propagated from disoccluded areas in the latent space. We validate the efficacy of our proposed method by conducting experiments on videos from various generative models, including Sora [4], Lumiere [2], WALT [8], and Zeroscope [12]. The experiments demonstrate that our method has a significant improvement over previous methods. Project page at https://daipengwa.github.io/SVG_ProjectPage/

Published as a conference paper at ICLR 2025
SVG: 3D STEREOSCOPIC VIDEO GENERATION VIA DENOISING FRAME MATRIX
Peng Dai Feitong Tan'1 Qiangeng Xu
David Futschik'
Ruofei Du
Sean Fanello'
Xiaojuan Qi
Yinda Zhang!
!Google The University of Hong Kong
ABSTRACT
Video generation models have demonstrated great capability of producing impres- sive monocular videos, however, the generation of 3D stereoscopic video remains under-explored. We propose pose-free and training-free approach for gener- ating 3D stereoscopic videos using an off-the-shelf monocular video generation model. Our method warps generated monocular video into camera views on stereoscopic baseline using estimated video depth, and employs a novel frame matrix video inpainting framework. The framework leverages the video genera- tion model to inpaint frames observed from different timestamps and views. This effective approach generates consistent and semantically coherent stereoscopic videos without scene optimization or model fine-tuning. Moreover, we develop a disocclusion boundary re-injection scheme that further improves the quality of video inpainting by alleviating the negative effects propagated from disoccluded areas in the latent space. We validate the efficacy of our proposed method by conducting experiments on videos from various generative models, including Sora (Brooks et al., 2024), Lumiere (Bar-TTa al., 2024), WALT (Gupta et al., (023), and Zeroscope (Wang et al.) (023a). The experiments demonstrate that our method has a significant improvement over previous methods. Project page at hattps://aaipeqqeee github. wot.sc_oovvvv. Bttp:/vv__Poooeceeeeeeeee pec_tPreeetttaaeee
INTRODUCTION
As VR/AR technology advances, the demand for creating stercoscopic content and delivering im- mersive 3D experiences to users continues to grow. Due visual sensitivity, binocular stereoscopic content should feature flawless 3D and semantic consistency between both eye views, as well as seamless temporal consistency across frames. While monocular video generation models have been extensively researched and methods are now capable of synthesizing high-fidelity videos that adhere complex text prompts (Brooks al., 2024), there has not been much progress in the realm of generating 3D stereoscopic videos at the scene level. One reason for this gap lies in the substantial amount of monocular video data that is readily available, contrasted with the scarcity of stereo video data for training models to generate stereoscopic videos directly.
An emergent solution is to convert generated monocular videos into stereoscopic videos using novel view synthesis [Li et al. 2023 Liu et al.) 2023b). However, these methods usually overly rely on camera pose estimation, which is challenging task on its own either using SFM (Schönberger
Frahm 2016) or joint optimization (Liu al. 2023b), and result tend to be unstable, particularly in dynamic scenes where cameras experience subtle motions or when the content dominated by dynamic objects with temporally varying appearances, both of which are prevalent in generated videos. Consequently, these methods fail in optimizing 3D scenes and offer low-qualit, solutions to the task (see Fig. 3) Moreover, these approaches are based reconstruction, lacking the generative ability to hallucinate occluded regions the novel views that do not appear in any of the remaining video frames.
In this paper, we propose an alternative pose-free and wriniin-free framework, for the sake of robustness and generalization capability, that operates solely by exploiting inference of off-the-
a Work done when Peng Dai was an intern at Google Equal contribution

---

Published as a conference paper at ICLR 2025
shelf video generation model (Wang al. 2023a) to generate high-quality 3D stereoscopic videos. Our initial attempt follows a typical 2D to 3D image uplifting methodology (Höllein et al. 2023)
and extends it into the video domain. Specifically, view, which is then warped into the right view using estimated monocular depths (Yang et al. 2024
first generate monocular video as the left
where apply temporal-spatial smoothing improve the consistency of the estimated depths.
Subsequently, we leverage an off-the-shelff video generation model's ability (Wang et al. 2023a) to generate natural videos, by adding noise and denoising the warped video frames to inpaint the disoccluded regions, inspired by diffusion- bassd image inpainting (Avrahami et al. 2023).
However, this naive pipeline does not produce appealing results: inpainting the right-view video frames independently. without referencing the left view, typically generates semantically mismatched content. To address this problem, we propose a novel representation, called the frame matrix, which contains frame sequences observed from a number of viewpoints evenly distributed along the baseline between two eyes. The frame sequences along the view direction (rows of the matrix) form videos with camera motion, while the frame sequences along the time direction (columns of the matrix) form videos with scene motions (see Fig. T second column). Since the video diffusion model has video prior for both scene and camera motions, we propose jointly update the entire frame matrix from both directions. In each denoising step. we use resample techniques (Lugmayr et al.) (022)) by alternatively denoising frame sequences along the view and the time directions. Finally, obtain semantically consistent and temporally smooth 3D stereoscopic video by taking the leftmost and the rightmost frame sequences to represent the left-eye view and the right-eye view, respectively.
Furthermore, we note that the inevitable resolution downsampling operation in most video generation models with latent encoding (Brooks et al. 2024 Bar-Tal al., 2024; Wang al., 2023a] Gupta et al. 2023) is detrimental to the video inpainting task. During encoding, the dark pixels created by disocclusion can degrade the features near the disocclusion boundary, leading to undesirable artifacts (see Fig. 5). Instead of following the inpainting scheme proposed in previous work (Avrahami et al., 2023). which encodes the latent feature only once, we iteratively update both the disoccluded regions in the image space and the latent feature map with generated content during the diffusion process. This approach re-injects the generated content into the disocclusion boundary, which mitigates the negative impact of disoccluded regions in downsampling and effectively prevents the artifacts.
To validate the efficacy of our proposed method, we generate stereoscopic videos from monocular videos generated by Sora, Lumiere, WALT, and Zeroscope. Both qualitative and quantitative evalua- tions suggest that our approach outperforms other baselines in 3D stereoscopic video generation. Our contributions are summarized as follows:
We design a novel pipeline to generate 3D stereoscopic videos. Unlike previous work, our method does not need camera pose estimation fine-tuning on specific datasets.
We propose a novel frame matrix representation that regularizes the diffusion- bbaed video inpainting to generate semantically consistent and temporally smooth content.
We propose a re-injection scheme that drastically reduces the negative influence of disoc- cluded regions in latent space and produces high-quality results.
We conduct comprehensive experiments that show the superiority of our approach over previous methods for 3D stereoscopic video generation.
2 RELATED WORK
Video Generation. Video generation Wang et al. 2023a Brooks et al. 2024 Bar- Tal et al. 2024
Gupta et al.) | Harvey et al.) [022 Ho et al. 2022a b Singer et al. 2022) has achieved tremendous progress since the advent of the diltusion model (Ho et al. 2020). Taking into account the dataset
requirements and scarcity of tagged videos, a prominent approach for video generation to extend
pre-trained image generation models (Rombach et al., 2022, Saharii et al.) 2022 Ramesh et al.
2022) by inserting additional temporal layers and then fine-tuning them on video data (Guo et al.
2023 Blattmann et al. 2023: Wu et al. 2023b). To further improve the compute efficiency and enable long clip processing, WALT (Gupta et al. 2023) and Lumiere (Bar- Tal et al. 2024) proposed to compress the video in both the temporal and spatial dimensions. More recently, Sora (Brooks
jet al. (204)) adopted a transformer diffusion architecture (Peebles & Xie (023) and was trained on large-scale video datasets to produce impressive video generation results. Different from previous
2

---

Published as a conference paper at ICLR 2025
video generation models focusing producing higher-quality and longer monocular videos, our method orthogonally explores the possibility of leveraging pre-trained video generation models for stereoscopic 3D video generation.
Novel View Synthesis. Great progress has been made for novel view synthesis in both static and
dynamic scenes (Mildenhall et al. [202T] Yoon et al. 2020 Li et al. 2022b Kerbl et al. [023] Müller
ett al. 2022 Tuckerr& & Snavely] [220]] Han et al. 2022 Li 20,2: Wang et al]] (023b)] Tucker &
Snavely (2020) convert a single image into a multi-plane representation for view synthesis. Mildenhall
et al. [2021) proposed to encode the static scene into neural radiance fields (NeRF), which were
then used for novel view synthesis through volume rendering. For more challenging scenes with
dynamic content, follow-up works additionally optimized a deformation field (Park et al. 2202aa)
Huang et al. 2023 Park al. 202 or scene flow fields (Li et al. 202 to handle the motion of dynamic objects. Instead of encoding the scene into a NeRF, Dyn BaR (Li et al., 2023) leveraged
nearby frames for rendering novel view images, and dynamic objects were handled by optimized motion fields. Different from methods requiring pre-computed camera poses, RoDynRF (Liu et al.
2023b jointly optimized the NeRF and camera poses from scratch. Concurrently, FVS (Lee et al. 2023) achieves novel view video synthesis using plane-based scene representation. Although these approaches produce high-quality renderings, they are limited to scenes where the camera pose can be accurately estimated and have limited synthesis capability. In contrast, our method explicitly avoids having to estimate camera poses and possesses the ability to hallucinate unseen content.
3D Content Creation and Inpainting, Automated 3D content creation (Höllein al., 2023 Dai
iet al. 2024 Gao et al. [2024] Yu et al. [2033] Chen et al. 2024) is another related area, with emerging approaches such as inpainting (Ho al. 2022a) or mulli-view generators (Liu al. 2023a Wang
ctt al. 2024 Zuo et al. (024). Recently, Text2Room Höllein et al. (023) proposed creating a 3D room
by warping an image into novel views and using a text- text-guided inpainter deal with disocclusions. WonderJourney (Yu et al.) (023) made this process automatic by including large language model in the loop. Similar to creating static scenes, we could use pretrained video inpainter (Zhou et al. 2023 Li et al.) (022c) for dynamic 3D content creation, however, these models suffer from generalization problems in creating high-quality, consistent 3D content. Lastly, Deep3D (Xic et al. 2016) is trained using 3D movies, with the goal of converting 2D videos into stereoscopic videos. However, the training data is not publicly available and it lacks the flexibility to modify videos for creative purposes, such different stereo baselines. In this paper, we explore the possibilities of using video generation models for 3D video creation without training on specific, hard-to-obtain datasets.
STEREOSCOPIC VIDEO GENERATION
Conditioned on text prompt or single image c, our method aims to generate 3D stereoscopic video 1X, X, consisting of two monocular sequences. The most straightforward way is to use a diffusion-based generation model G:
X,) = 9 (fez 1= ((,,,, T).),
where 6 ~ (O, is the sampled noise at step t. The generated stereoscopic videos should possess the following characteristics: First, the appearance and semantics between the left eye view X, and right eye view X, should be consistent and be temporally stable. Second, the stereo effect should be prominent and immersive. Last, the generated content should be diverse and controllable with the given conditioning.
However, training a G that can directly generate stereo videos (X,, X with the desired properties requires a vast dataset of stereo videos with diverse content. Due to the scarcity of such data, we propose training-
approach that relies on an off-tth-shelff depth estimator YYang et al. 2024) and diffusion-barr -based monocular video generation model G such as Zeroscope (Wang et al.) (023a).
We first generate a monocular video for one eye using a video diffusion model (Eq. 2). then obtain the other video view by conditioning on the first video. To automatically preserve 3D consistency, we implement this conditioning by estimating depth d, for the left video and warp its content to obtain the right view sequence X - with disocclusion masks M, (Eq. 3) according to the stereoscopic baseline (Wang et al. [2023b] Han et al. (022)). Then, we use 9 again to inpaint the disoccluded parts by denoising inpainting process (Avrahami al. 2023
Lugmayr et al. 2022) (Eq. 4. obtaining the
3

---

Published as a conference paper at ICLR 2025
Spatial
space riding homs""
Decouur
Lafent m
Denoising
Freme Matrix
Frame Matrix
Genenttd mono vidao
Warp
Rosas
right view
3D Sterenscopiic Video
Estimated depth
Disocclusion Manks
Sample
time
Denoising Inpainsing
FT
XN (add notse back to
JxTstep
Figure I: Overview Top: Given a text prompt, our method first uses a video generation model to generate monocular video, which is warped (using estimated depth) into pre-defined camera views to form a frame matrix with disocclusion masks M. Then, the disoccluded regions are inpainted by denoising the frame sequences within the frame matrix. After denoising, select the leftmost and the rightmost columns and decode them to obtain a 3D stereoscopic video. Bottom: Details of denoising frame matrix. We initialize the latent matrix 2r random noise map. For each noise level, we extend the resampling mechanism (Karnewar et al. 2023 Lugmayr et al.) to alternatively denoise temporal (column) sequences and spatial (row) sequences N times. Each time, row or column sequences are denoised and inpainted (see Fig 2). By denoising along both spatial and temporal directions, we obtain an inpainted latent Zg which can decoded into temporally smooth and semantically consistent sequences.
other eye view video X,
X;
X M, (X,,
(3)
X, /(fe T), Xi.r,M,
In Sec. 3.1 we describe the video depth warping. In Sec. 3.2 we introduce the frame matrix representation for the video inpainting. Our denoising frame matrix drastically improves the semantic similarity between X₁ and X, and helps preserve temporal smoothness. Last but not least, disocclusion boundary re-injection mechanism is introduced to further improve the inpainting quality An overview our method is displayed Fig.
in Sec. 3.3
3.1
MONOCULAR VIDEO DEPTH WARPING
The depth estimation model Yang et al. 2024 is applied predict all frames' depth values, which will be smoothed to produce more consistent video depths. Specifically, we utilize the estimated optic flows (Teed Deng 2020) to align consecutive depth frames. The outliers in predicted depths will be suppressed by convolving with Gaussian kernel along the time axis. After obtaining RGB-D frames, we can warp them into target camera views where disoccluded regions appear. In addition, the warped images usually contain isolated pixels, and the foreground and background are entangled, which jeopardizes video quality (Dai et al.) 2020). To handle these problems, we follow [Dai al.) (2020) to project points into multi-plane images (Zhou et al.) 2018), then remove isolated pixels and cracks and finally obtain voisy-points-fie free image. (See supplemental material Sec. C for details).
3.2
VIDEO INPAINTING WITH FRAME MATRIX
The inpainting pipeline plays a key role in ensuring spatial/semantic and temporal consistency. While image inpainting approaches (Avrahami al. 2023 Lugmayr al.) (2022) provide reasonable baseline, the results lack temporal and spatial stability. Therefore, we introduce Frame Matrix representation, which addresses both issues.
Single Video Denoising Inpainting Inspired by RePaint (Lugmayr et al. 2022). we extend the diffusion-based image inpainting video inpainting. We the video generation model & (i.e.,
4

---

Published as a conference paper at ICLR 2025
Zeroscope (Wang al. 2023al) ) inpainting tool, which is latent diffusion model consisting of a VAE encoder & a decoder and a latent denoiser (cg. Eg}. First, the warped video is fed into the VAE encoder to obtain video latent features z) =/ E (X Then, resize the image disocclusion masks M, to the resolution of the latent and obtain latent disocclusion masks m. During the denoising process, step i, we sample new intermediate noisy latent map from Zo (Eq. (5 denoises the latent map from the last step Z, (Eq. 6 and combine them with m obtain the z (Eq. 7 We visualize the following steps in Fig. (b):
start from random noisy latent map Zr (0, I). For each subsequent
known 2
N (vazz) known (1 at)1).
(5)
2t vI co (e e, Eolla,a)))
Br
2 1-1
unknoww
N
(6)
B3
Zt-1 m
+ (1 m)
2z-1
unknown
where de and 9g denote the total noise variance and one step noise variance t. respectively; co(z,, c, and [z( c, are predicted noise and variance for noisy latent map at f - 1 step. Finally, we can obtain the inpainted right view sequence X, by decoding the denoised latent X, D(zo).
By applying the above video inpainting scheme for the right view, we implement Eq. 4 and suc- cessfully hallucinate the disoccluded (unknown) regions while preserving the unoccluded (known) regions. The video diffusion model also ensures temporal smoothness. However, the inpainted content on the right view usually lacks semantic consistency w.r.t. the left view, as shown in the third column of Fig. 4 This is because only condition the left view by depth warping, while dropping the conditioning during inpainting.
Frame Matrix Representation. We propose novel representation- -rame matrix, which targets consistent dynamic content generation across space and time. As shown in Fig. top. it is matrix consisting of multiple frames, where each row represents frames observed from different camera poses at the same time stamp, and each column is a video recorded by fixed camera at different time stamps. Consequently, the frame matrix can be defined as;
(
X
X
(
where S and V are the largest indices of time steps and views, respectively. A view sequence (row) (e.)) forms video with camera motions, while time sequence (column) X(..)) forms video with time-varying scene motions. Since the video diffusion model can denoise a sequence to a temporally and semantically consistent video, jointly denoise the rows and columns ensure consistency spatially and temporally. Finally, we can obtain a 3D stereoscopic video by taking the leftmost and the rightmost time sequences X (1,0), X ((.V)-
Constructing Frame Matrix. We evenly add V camera views distributed along the baseline between the two eyes with the same orientation of the reference view. Then, we warp the ref- erence video (the Oth column) based on depth (Sec. 3.1) into these views and obtain Xworp = [X X (1.0-1), X (.0-v)] with a disocclusion masks matrix M.
Denoising Frame Matrix. Similar to single video sequence inpainting, we encode frame matrix into a latent frame matrix known E(Xwarp). and resize M to obtain latent disocclusion map m. We also initialize 2y (o I). As shown Fig T (Bottom), for each noise level, we extend the resampling mechanism (Lugmayr et al. (022) to alternatively denoise column sequences and row sequences times. Each time, row or column sequences are denoised following Eq. 5 and we add back noise between every resampling iteration:
Z N( B,-1 11-11)
Please refer to See B in the supplemental material. By denoising along these two directions alterna- tively, the spatial and temporal sequences will proceed toward harmonic state.
High-level motivation of Frame Matrix. In practice, 3D stereoscopic videos can be produced by recording with two cameras (time-direction videos). Since both cameras are capturing the same
5

---

Published as a conference paper at ICLR 2025
Decooae
Deneiser t
2
%
Fieodder
Re-1
Al
knswn
Update
Add Nolka
Tiaars
Disocclusion Boundary Re-Injection
(b) Single Denoising Inpainting Step
Figure 2: Denoising Inpainting. This figure visualizes the operations in the purple box Fig (a) We re-inject the generated content from denoised latent Zg to update Z known and reduce its feature corruption on the disocclusion boundary. (b) A noisy latent z, is denoised to z disoccluded region and combine it with the unoccluded region of zb known
zunknownn
We take its
scene, gradually moving the left camera toward the right camera also results in coherent video (space-direction video). Likewise, it is essential to consider both the time and spatial directions when generating the 3D stereoscopic video to ensure that both perspectives represent the same scene. Additionally, compared to direct inpainting a large region all at once, gradually expanding the inpainting area (spatial direction) tends be easier and can lead more stable and plausible results. More analysis is in suppicmentary material Sec. D
3.3 DISOCCLUSION BOUNDARY RE-INJECTION
Since most video generation models using latent diffusion, the disoccluded dark regions of Xwarp will be propagated beyond the latent mask m during VAE encoding (e.g.. Zeroscope downsamples by 8x), leading to defective latent features on 2 kkoown's disocclusion boundary. This would lead to artifacts in the final results (Fig. left).
We propose to re-inject the denoised information in the disoccluded regions improve the latents on this boundary. Specifically, we predict the denoised latent features (Ho et al.) 2020), which are decoded into denoised video (Eq. ( Then, replace its unoccluded regions with warped pixels to form video that is faithful to the reference view but with better disocclusion pixels. By encoding this video, we can get a updated z
zknunn (Eq. 10) which alleviates corruption on the boundary:
1
X, = D(Za), where %o
(Zt v1 Ciee (Zt, (z),
Vot
Z
& M O X Xwarp +11 (1-M)
M) O Xo)
(10)
After this, this improved znnoo can be used in Eq. for the next iteration.
EXPERIMENTS
Datasets. To validate the effectiveness of our method, we conduct experiments using a variety of recent video generation models, including Sora (Brooks et al. (024), Lumiere (Bar-Tal et al. (024), WALT (Gupta et al., 2023), and Zeroscope (Wang et al. 2023a). These models produce diverse left videos from a wide range of input text prompts, covering subjects such humans, animals, buildings, and imaginary content.
Implementation Details. To ensure the stereo effect appears realistic, we normalize the up-to-scale depth values predicted by the depth estimation model (Yang et al.) 2024 to range of (1, 10) and set the baseline between left and right views to 0.08. The frame matrix constructed by evenly placing 8 cameras between the left and right views, with each camera corresponding to warped video sequence. Due to the limitations of the Zeroscope model, we currently conduct experiments on video sequences with 16 frames. Following the approach of RePaint (Lugmayr al., (022), employ DDPM (Ho et al.) 2020) as our denoising scheduler with 1000 total time steps 7 and 50 denoising steps, resulting 20 time step jumps per denoising step. During the initial 25 denoising steps (50 to 25), we resample 8 times at each step to establish reasonable structure in disoccluded regions. For the remaining steps, we reduce resampling to 4 times and denoise only the right view for improved efficiency while generating stereoscopic videas. Currently, our implementation runs on an A6000 GPU and takes 8 minutes to generate stereoscopic video using only ~ 10GB of RAM.

---

Published as a conference paper at ICLR 2025
Baselines. We compare method with two families of approaches: video inpainting, and novel view synthesis from a monocular vidco. For video inpainting approaches, we generate the right view in the same manner our method using depth-guided warping. We then apply state-of-the-an methods ProPainter (Zhou et al. 2023) and E2FGVI (Li et al. (022c) to inpaint the right views. For novel view synthesis methods,
compare our results with RoDynRF (Liu et al.) 2023b) and DynlBaR (Li et al., 2023), which optimize scene representations relying on camera poses. To ensure fair comparison, given the differing 3D scales between their reconstructed scenes and our estimated depth, we select the baseline for rendering the right view by matching the median disparity of foreground regions in the resulting disparity map to that of our methods. We are also aware of approaches trained on dedicated datasets that directly produce the right view given the left view like Deep3D (Xie et al. 2016). However, does not generalize well to the generated video, especially those in non-realistic styles, and the comparison could be found in the supplemental material.
4.1 QUALITATIVE RESULTS
Qualitative Comparisons. We show qualitative comparisons in Fig. 3 Previous video inpainting methods suffer from common problem - the generated content in disoccluded regions is blurry, such as the knight's 's arm, horse's s tail, and corgi's face, presumably because that these methods are trained on limited datasets. On the other hand, novel view synthesis methods suffer from unstable camera pose estimation (e.g., DynlBaR fails on some videos). Though good at reconstructing visible content from the monocular video, they are typically poor at synthesizing novel contents in the disoccluded regions that are not observed in any frames (e-g-, ghost effect near the boundary in the RoDynRF result on the corgi example). In contrast, our approach takes advantage of the generative capability of video diffusion models trained on massive scale datasets and does not require camera poses of the input video as inputs, thereby generating high-quality content in various types of scenarios (last row of Fig. 3) and consistently outperforms baseline methods. Additionally, we visualize the stereo effects of different methods on the corgi case using a stereo depth estimator (Li et al. 2021a), which predicts disparity values from the stereo images. As shown in Fig. [10 RoDynRF and DynIBaR exhibit less depth variation, indicating weaker stereo effects. This occurs when the camera is wrong and the training process overfits the training views, resulting in a sub-optimal 3D representation.
4.2 QUANTITATIVE RESULTS
In this part, we show quantitative comparisons with other baselines. We primarily rely on dedicatedly designed user study to evaluate the quality of generated stereoscopic video on various quality axes. We also provide an objective metric to measure the semantie similarity between the left and right views using pre-trained CLIP models.
Human Perception. To assess the perceived visual quality, we conducted a user study with 20 6.2). On VR headset, each participant viewed and
participants (9 female, age = 33, evaluated five generated videos (out of 20 in total) by all five methods on stereo effect, temporal consistency, image quality, and overall experience using 7-point Likert scale (Likert) 1932). A total of 435 evaluations (DynIBaR failed to generate 13 videos) were counterbalanced and randomly shuffled. We also included training session to eliminate novelty effects. Results summarized in [ with details in the supplemental material. Our method outperforms other baselines on measured metrics.
E2FGVI
ProPainter
RoDynRF
DynlBaR
Ours
Stereo Effect
4.79 (1.08)
4.81 (1.13)
2.97 (1.34)
1.86 (1.25)
5.24 (0.94)
Temporal Consistency f
4.74 (1.33)
4.74 (1.22)
3.35 (1.66)
1.89 (1.33)
5.15 (1.22)
Image Quality 1
4.42 (1.27)
4.38 (1.28)
2.84 (1.60)
1.67 (1.07)
5.12 (1.33)
Overall Experience t
4.67 (1.04)
4.66 (1.09)
2.92 (1.43)
1.72 (1.06)
5.35 (0.99)
Table Human perception. This table reports the results of human perception experiments as mean
(std). Our method outperforms other baselines on all metrics. Kruskal-Walli tests (Kruskal & Wallis 1952) reveal significant effects of group on all metrics (x2 > 13.3, p < 0.001 "0]) Post-hoe tests using
tests (Mann & Whitney 1947) with Bonferroni correction reveal significant effects (p < 0.05*, |r| > 0.I) for each pairwise comparison, except E2FGVI vs. ProPainters yield comparable results.
7

---

Published as a conference paper at ICLR 2025
Sora
WALT
Zeroscope
Left
E2FGVI
ProPainter
RoDynRF
FAILED
DynIBaR
Ours
Figure 3: Qualitative comparisons. The first row shows left-view images. The video inpainting methods E2FGVI and ProPainter tend to generate blurry content in disoccluded regions, such as knight's arm and corgi's face. RoDynRF lacks the generation ability, thus content on the right side of the corgi case is poor. DynlBaR's ' results contain artifacts, and it requires camera poses as inputs, which failed in some scenarios. On the contrary, our method takes advantage of video generation models and is pose-free, thus generating high-quality content in different scenarios.
8

---

Published as a conference paper at ICLR 2025
Semantic Consistency. We additionally check the semantic consistency between the left and the right view. We use pre-trained CLIP model (Radford et (021)) to extract features for both left views and right views of stereoscopic video, and then calculate the teature distance following Zhengwentai (2023) to obtain the semantic consistency score. Table 2 our method attains the best semantic consistency (96.44) over other baselines.
Quality Assessment. We measure the quality generated video using aesthetic score (Schuhmann al. 2022), DOVER (Wu al., 2023a), and FVD (Unterthiner al. (219), which are in line with human preferences. In Table 2 our approach achieves the best performance (5.27, 0,584, and 599), aligning with the conclusion from the user study experiment in Table
4.3 ABLATION STUDIES
Effect of Frame Matrix. In Fig. 4 we showcase that using frame matrix benefits semantic con- sistency between the left and right views. Without using frame matrix, the disoceluded regions in warped images can be inpainted with unconstrained contents, which are likely to be inconsistent with the left view given impressive generative capability of the diffusion model, such as the hair of the man and the head of the horse. This is also revealed in Table 2 where CLIP Score drops from 96.44 to 95.81 when disabling frame matrix. Thanks to constraints from other frames within the frame matrix, method generates both reasonable foreground and background contents in the disoccluded regions. More studies of frame matrix are included in Sec F and Sec. of the supplemental material.
Reference
Warped
Wio frame matrix
With fram matrix
Figure 4: Semantically consistent content generation. The reference frames are warped into the target view with disoccluded regions set to be black. Without using frame matrix, the generated content does not match the reference, such as the book and the face of horse. With frame matrix, the inpainted contents are more semantically reasonable.
Effects of Disocclusion Boundary Re-Injection. In Fig. 5 demonstrate the importance of updating unoccluded latent features for high-quality results. Without this update, the disoccluded region inpainted with unnatural textures that don't blend well with the surrounding content. With the update, the inpainted content blends seamlessly. This is reflected quantitatively in Table 2 where the aesthetic score drops from 5.27 5.18 when unoccluded feature updates are discarded.
Wio Disoeclusion Boundary Re-Injection
With Disoczlasion Boundary Re- lejennecnoon
Figure 5: Disocclusion Boundary Re-injection. Without disocclusion boundary re-injection, the inpainted images usually contain artifacts. Bottom-left corner shows the warped image.
Effects of Handling Isolated Pixels and Cracks. In Fig. left, obvious artifacts in warped images, such as isolated points and cracks where the foreground car entangled with the background gray road, and these artifacts remain in the final generated results. After applying warping designs in Sec. C our results are clean, handling warping-related artifacts as shown in Fig. 6 right side.
Different Stereo Baselines. Fig. 7 shows increasing the stereo baseline makes inpainting harder and degrades stereoscopic video quality, reflected by CLIP score. Our method is resilient to

---

Published as a conference paper at ICLR 2025
Method
E2FGVI
ProPainter
RoDynRF
DynlBaR
Ours FM
Ours
DBR
Ours
CLIP
94.34
95.29
96.03
93.24
95.81
95.60
96.44
Aesthetic
5.06
5.07
4.97
4.66
5.25
5.18
5.27
DOVER 1
0.547
0.535
0.352
0,365
0.565
0,560
0.584
FVD 1
638
606
727
1208
614
699
599
Table 2: Quantitative comparisons. We show the semantic consistency using CLIP feature simi-
larity (Hessel al., (02T)) between the left and right view. Additionally, the quality of generated videos is measured by aesthetic score (Schuhmann et al. 2022), DOVER (Wu et al. 2023a) and FVD (Unterthiner al. 2019). Our method outperforms previous methods well as ablated cases.
W/o data peppocessnn
Ours
Figure 6: Isolated points and cracks. Left:: without handling isolated points and entangled fore- ground and background (the gray road can be seen through the cracks) in warped images, these artifacts remain in the final results. Right: our results are clean.
7cm
15cm
23cm
32cm
Sonsntic consisteney
Figure 7: Different stereo baselines. Unnatural artifacts begin to appear the baseline expands. Our method performs well for stereoscopic video generation where baseline is usually less than 7cm.
larger baselines, failing beyond 20cm (depth normalized to 1.0-10.0m). This range is sufficient for generating 3D stereoscopic video for most people, given typical inter-pupillary distances of 5-7cm.
LIMITATIONS AND DISCUSSIONS
Although results demonstrate the possibility of generating 3D stereoscopic videos using pre- trained video diffusion models, challenges remain. For one, we did not study longer videos because the architecture of typical video diffusion model supports the generation of videos only few seconds long. One possible solution for long 3D stereoscopic video generation is to use stronger foundational models, such Sora (Brooks al. (024).. Alternatively, we could gradually generate longer videos by overlapping frames of shorter videos. Additionally, our method depends on depth estimation model, which may fail, e.g., when dealing with thin structures or boundaries. practice, the depth estimation model showcases decent depth consistency after training on large-scale datasets containing images extracted from the same scene and videos, and our flow-based smoothing operation can further mitigate depth perturbations that occasionally arise. To obtain more consistent video depth estimation, fine-tuning video generation models with synthetic video depths is worth exploring. Note that method is orthogonal the depth estimation and can benefit from its advancements. Furthermore, our method utilizes auxiliary views to assist in generation, necessitating additional denoising time. This challenge be addressed by leveraging advancements in efficient denoising techniques and diffusion model architecture, potentially enhancing computational efficiency.
6 CONCLUSION
We proposed a complete system for stereoscopic video generation, using a video diffusion model and our frame matrix inpainting scheme. Given the fast adoption of video generation, our approach bridges the gap between the current ability to generate monocular and stereoscopic videos. In particular, we showed that our frame matrix formulation significantly advances the state-of satte-of-the-arr the-hrt for generative stereoscopic video, and can be adopted by existing and future video diffusion models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended Latent Diffusion. ACM Transactions on Graphics (TOG), 42(4):1-11, 2023.
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: space-time diffusion model for video generation. arXiv preprint arXiv:2401. 12945, 2024.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563 22575, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL /ttttpss:/oppaaiii. wwwppnnaicco//ooooeeooooomoooeoe www.cco//eeeaae research/
video-generation -otea-oooolllssssseeeeeeeee----- as-sssssswworloooo www.wwowdddsmmaaoooooooooooooooo
Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang. and Huaping Liu. V3D: Video Diffusion Models are Effective 3D Generators. arXiv preprint arXiv:2403.06738, 2024.
Peng Dai, Yinda Zhang, Zhuwen Li, Shuaicheng Liu, and Bing Zeng. Neural point cloud rendering multi- projection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7830-7839, 2020.
Peng Dai, Feitong Tan, Xin Yu, Yinda Zhang, and Xiaojuan Qi. GO-NeRF: Generating Virtual Objects in Neural Radiance Fields. arXiv preprint 1xXiv:2401.057500 2024.
Quankai Gao, Qiangeng Xu, The Cao, Ben Mildenhall, Wenchao Ma, Chen, Danhang Tang, and UIrich Neumann. GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation. arXiv preprint 123.: 2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.044 04725, 2023.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 2023.
Yuxuan Han, Ruicheng Wang, and Jiaolong Yang. Single-view view synthesis in the wild with learned adaptive multiplane images. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-8, 2022.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling long videos. Advances Neural Information Processing Systems, 35: 27953-27965, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference-free evaluation metric for image captioning. In EMNLP, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 2022a.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8644 2022b.
11

---

Published as a conference paper at ICLR 2025
Lukas Höllein, Ang Cao. Andrew Owens, Justin Johnson, and Matthias Nießner. Text2Room: Extracting Textured 3D Meshes from 2D text-to-Image Models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7909-79200 2023.
Yi-Hua Huang, Yang-Tiae Tian Sun, Ziyi Yang. Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: cparse-controlled gaussian splatting for editable dynamic scenes, arXiv preprint ar arX47:2312.14937, 2023.
Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy Mitra. Holodiffusion: Training 3d diffusion model using 2d images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18423-18444 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splatting for 2023.
Radiance Field Rendering, ACM Transactions Graphics, 42(4):1-14,
William H Kruskal and W Allen Wallis. Use of Ranks in One-criterion Variance Analysis. Journal of the American Statistical Association, 45611)055272226658---626222 1952.
Yao-Chih Lee, Zhoutong Zhang, Kevin Blackburn- Simon Niklaus, Jianming Zhang, Jia-Bin Huang, and Feng Liu. Fast view synthesis of casual videos. arXiv preprint a3X1v:2312.02135. 2023.
David Li, Yinda Zhang, Christian Häne, Danhang Tang, Amitabh Varshney, and Ruofei Du. OmniSyn: Intermediate View Synthesis Between Wide-baseline Panoramas. In 2022 IEEE Conference Virtual Reality and 3D User Interfaces, 2022a.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3D Video Synthesis from Multi-view Video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5521-5531, 2022b.
Zhaoshuo Li, Xingtong Liu, Nathan Drenkow, Andy Ding, Francis X. Creighton, Russell H. Taylor, and Mathias Unberath. Revisiting stereo depth estimation from a sequence-to-seqeene perspective with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 6197-6206, October 2021a.
Zhen Li, Cheng-Ze Ze Lu, Jianhua Qin, Chun-Le Guo, and Ming-Ming Cheng. Towards an end-to-end framework for flow-guided video inpainting. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 17562-17571, 2022c.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6498-6508, 2021b.
Zhenggi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4273 4283-4284, 2023.
Rensis Likert. A Technique for the Measurement of Attitudes. Archives of Psychology, 1932.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: Zero-shot one image to 3D object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9298 2998-9909, 2023a.
Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13-23, 2023b.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 11461-11471 11411, 2022.
12

---

Published as a conference paper at ICLR 2025
Henry B Mann and Donald R Whitney. On Test of Whether One of Two Random Variables Is Stochastically Larger Than the Other. The Annals of Mathematical Statistics, pp. 50 50-60, 1947.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics (TOG), 41(4): 1-15, 2022.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5865-5865777
2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla and Steven Seitz. Hypernerf: higher-dimensionall representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228, 2021b.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 1195-4205, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06123 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 11684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 15:33449-36499 2022.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure- from-motionn revisited. In Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25225 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.144: 14792, 2022.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Eision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Pari II 16, pp. 402-419. Springer, 2020.
Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 551-560, 2020.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski. and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
Jiuniu Wang. Hangjie Yuan, Dayou Chen, Yingya Zhang. Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint 266.::230000671,,, 2023a.
13

---

Published as a conference paper at ICLR 2025
Lezhong Wang, Jeppe Revall Frisvad, Mark Bo Jensen, and Siavash Arjomand Bigdeli. Stereodiffu- sion: Training-free free stereo image generation using latent diffusion models, 2024.
Xiaodong Wang, Chenfei Wu, Shengming Yin, Minheng Ni, Jianfeng Wang, Linjie Li, Zhengyuan Yang. Fan Yang, Lijuan Wang, Zicheng Liu, et al. Learning 3d photography videos via self- supervised diffusion on single images. arXiv preprint arXiv:2302. 10781, 2023b.
Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 20144-20155 54, 2023a.
Jay Zhangjie WU, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVT International Conference on Computer Vision, pp. 7623-7633, 2023b.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Deep3d: Fully automatic 2d-to-3d video conversion with deep convolutional neural networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 842-857. Springer, 2016.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. pp. 10371-10381 2024.
Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from monocular camera. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5336-5345, 2020.
Hong-Xing Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T Freeman, Forrester Cole, Deqing Sun, Noah Snavely, Jiajun Wu, et al. Wonderjourney: Going from anywhere to everywhere. arXiv preprint arXiv:2312.0388
2023.
SUN Zhengwentai. clip-score: CLIP Score for PyTorch. httpp:://githbb.comm/stss
c clip 1lip-score, March 2023. Version 0.1.1.
Shangchen Zhou, Chongyi Li, Kelvin CK Chan, and Chen Change Loy. Propainter: Improving propagation and transformer for video inpainting. In Proceedings of the IEEÉ/CVF International Conference on Computer Vision, pp. 10477-10486, 2023.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.098// 2018.
Qi Zuo, Xiaodong Gu, Lingteng Qiu, Yuan Dong, Zhengyi Zhao, Weihao Yuan, Rui Peng, Siyu Zhu, Zilong Dong, Liefeng Bo, et al. VideoMV: Consistent Multi-View Generation Based on Large Video Generative Model. arXiv preprint arXiv /2010, 2024.
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
In the supplementary sections, we provide more studies and details of the proposed method.
Sec.
provides pseudocode describing frame matrix denoising inpainting.
Sec. C
includes details on depth smoothing and warping-related artifacts handling.
Sec. provides more analysis of frame matrix.
Sec. D elaborates details of human perception experiments and provides additional compar- isons with Deep3D.
Sec. contains more studies of our method, including visualization of stereo effects, the ability utilize temporal context, ablations on the number of cameras used, the efficacy of depth smoothing, intermediate results during the denoising process, consistency across different views, generalization ability of boundary re-injection, and comparisons with single- image view synthesis.
Sec. G displays different trajectory results sampled from the frame matrix and cases in various scenarios.
B ALGORITHM DETAILS
In the algorithm below, we present the detailed steps to denoise the Frame Matrix with spatial-temporal resampling, where we set μi [lo(2; Zt; C, 1) -i-p((2 VI-a, co C, ()), following DDPM (Ho et
2020
Algorithm Frame Matrix Inpainting
Input: 2T No0, I): Initial noisy latent maps Zo: Initial clean latent maps
for t T, T..... do
for = N....NN if odd then
N do
Denoise time sequences (A(.vj/lU = O,..., V):
for e = 0, do
known
(V (1 (1-ai,) a)
[:,]]-1
unknown (s,)e-1
NNoo (Z(r,)) C, 1), Ex t))
Z (sv)t-1 m(,))
known (::))-1 + (1 m(n))
((v)t
unknown
end for else
Denoise view sequences [Z(s.)rls = 0, O,..., S):
for s O, S do
known (s:)t-1 N N(Var (1 (1- (1-ū))
(zo(2(s)))) c, t), Eg (z((Z(a.)) c,
(s,)t-1
(s.))t11 m(.:)
known (
+ (1 (s.))
(s.))-1
end for
end if
if < then
Add back one noise step for resampling:
2, N 3r- Z,
I
end if
end for
end for
15

---

Published as a conference paper at ICLR 2025
C DETAILS OF MONOCULAR VIDEO DEPTH WARPING
C.I DETAILS OF TEMPORAL DEPTH SMOOTHING
In practice, three consecutive frames to stabilize depth changes. At timestamp first estimate optical flows flow;-1--t and flow;+1->h which help align the predicted depth maps at different timestamps. Finally, the refined depth timestamp / is obtained via convolving aligned depths (Di-1-t: Dr: D(-1-1) with a one-dimensional Gaussian kernel of shape 3 x 1.
C.2 DETAILS OF HANDLING ISOLATED PIXELS AND CRACKS
Puati-Plaaee projection. Given RGB-D images, we warp them into a target camera view. Instead of projecting all pixels onto one image plane and handling occlusions using z-buffer, we divide the camera view space into multi-plane images (1atez0 N....IN I..))) (N=4 in this paper) according to near and far depths, then each pixel is projected onto the image plane closest to it. We use [M ..... ) to indicate valid pixel positions on each image plane. By doing this, the foreground and background are separated in different planes, which makes artifacts (i.e., isolated points and entangled foreground and background in Fig. leleet) easily to be handled.
Remove isolated points. Due the inaccuracy of depth values around image boundaries, these pixels are warped into wrong positions leading to isolated pixels (see red box in fleft). Intuitively, isolated pixels have no or very few neighbors, thus we detect isolated pixels based on this observation. Specifically, we apply convolution to each mask image M, using 3 kernel, after which isolated pixels are empirically determined where the values after convolution are less than 0.5. We remove these isolated pixels on both RGB and mask images to obtain updated [Totep] rstepl and [Mpeepl ..... MN. M
Remove cracks. Since the depth image is not a watertight representation, the warped image usually contains cracks/holes that confuse foreground and background content. For example, the gray road can be seen through the dog's ear in Fig. left. Similar to handling isolated pixels, we use a 3 x Gaussian kernel to perform convolution on each mask image M.tept When there are cracks, the values after convolution will be less than 1. In this paper, positions with no pixel values (0 in M,) but with greater values than 0.2 after convolution are considered cracks. We fill cracks by interpolating nearby pixels in each image and obtain new multi-plane images (1et .... 1 stepp? and [M M x
After handling artifacts in each image plane, all image planes are blended into a final image (c.g.. Fig. 6 ours left) in back-to- front order using Eq. II] where the content of front plane occludes content of the back plane.
I (1 (1
Moter2) + rotep2 x Mctep2 for i in [N,..., 1].
(11)
D MORE ANALYSIS OF FRAME MATRIX
Based on the high-level motivation in the main paper, each frame z(i,j) in the frame matrix should be as consistent as the denoising results from both time and spatial directions. Mathematically:
[ii,j)=llz, -1| (i,
z
spatial ial
(i.j)ll2 + (i.,)
2
temporal (i,
(12)
2t-1
spatial spatial (z+); 2
temporal
@ temmporal (z+):
(13)
Where θ is pre-trained video diffusion model, e Opptiial and Q temporal indicate denoising frame matrix in the time and spatial directions, respectively. Here, L is a quadratic Least-Squares (LS) where the solution as close as possible to all diffusion samples 2 spptial (i.j) and 2 In practice, simultaneously denoise in spatial and time directions obtain the optimal results.
temporal (i.
16

---

Published as a conference paper at ICLR 2025
E DETAILS OF HUMAN PERCEPTION STUDY
Participants. To evaluate the perceived quality of the generated stereoscopic videos, we recruited 20 participants (9 females) at least 18 years old (n1 = 33, a 6.2) with normal or corrected- corrected-to-normall vision at an anonymous institution via email lists and group communication software. The majority of participants had some experience with virtual reality. None of the participants was involved with this project prior to the user study,
Study setup. The study was conducted in a quiet meeting room with a commercial VR headset as the primary apparatus. The study software is implemented in Unity 2023.3.0b and we render stereoscopic videos with custom shaders on a 1.8m x 1.0m quad that is three meters away from the participant in the world space, which occupies approximately 33.4 degrees in width and 18.92 degrees in height initially. Users have the freedom to move themselves within the meeting room to examine the stereoscopic video. This setup allowed participants to experience the stereoscopic videos in virtual reality settings and provided a controlled environment for the user study.
Each study session consists of a demographies interview with consent forms, training session, and an evaluation session. To eliminate the ordering effect, we randomly counter- balanced all five methods for each video and assigned five random videos (out of 20 videos) with five conditions to each participant. However, since DynlBaR method failed to generate 13 videos, we collected total of 100 human evaluations for each method except DynlBaR. During the training session, we randomly picked a video that was outside of the assigned videos to the participant and asked the participant to rate the stereoscopic effect, temporal consistency, graphical quality, and overall experience on a 7-point Likert scale (Likert 1932). with 1 being the lowest, 7 being the highest, and being the average. This procedure helps eliminate the novelty effect and calibrate the user's rating before the formal evaluation session. In the formal evaluation, we prompted the participant with the question like "How would you like rale the stereoscopic effect of the video on 7-point scale, with / being the lowest, 7 being the highest, and 4 being the average?" ? and asked the user the reason behind the rating.
Study protocol.
x 20 13 x 435 evaluations from 20 participants, resulting in
Metrics.. We evaluate the perceived quality of generated stereo videos based on three key aspects: 1. Stereo Effect. This refers to the perception of depth achieved by presenting slightly different images to each eye. A strong stereo effect makes objects appear closer or farther away, enhancing the 3D experience. Example questions: "How strong was the 3D effect in the video?" and "Which video felt more immersive due to the 3D effect?" 2. Temporal Consistency. This aspect assesses the smoothness of scene motion and the absence of artifacts such as jitter or ghosting over time. Example questions: "How smooth and natural did the motion of objects appear?" and "Did you notice any flickering, jumpiness, or distortions in the video?" 3. Graphical Quality. This evaluates the overall visual appeal of the video, including the quality of details, textures, lighting, and color fidelity. Example questions: "How would you rate the visual quality of the video?" and "Which video had more detailed and realistic textures?"
Study results. Overall, despite the missing data points for the DynlBaR method in some videos, Kruskal-Walli Wallis tests (Kruskal & Wallis 1952) reveals significant effects of group on all metrics respectively (x2 > 13.3, < 0.01): with stereo effect x2 186.3. p < 0.001, temporal consistency x²= == 121.3, p < 0.001, graphical quality x2 = 153.2, p 0.001, and overall experience x² 192.9, p < 0.001. We further performed post-hoc tests using 1947) with Bonferroni correction, which revealed significant effects (p < 0.05, |r| > 0.1) for each pairwise comparison, except E2FGVI vs. ProPainters. Specifically, for Ours vs. E2FGVI, p = 0.002 on stereo effect, p- = 0.030 on temporal consistency, p 0.001 on graphical quality and overall experience. For Ours vs. ProPainter,
tests (Mann & Whitney
0.004 on stereo effect, p 0.017 on temporal consistency, p < 0.001 on graphical quality and overall experience.
Study findings. Our results suggest that our methods achieve significantly better perceived stereo- scopic effect than all other methods, while improvement in graphical quality and overall experience is more evident over stereoscopic effect; and stereo effect more evident over temporal consistency. During the study, we also observed many positive comments about our methods like "the contour is more clear". "the graphics neutral feedback like "some part really works and some parts don't 't: one side of the turtle face is wrong". " and "I see no difference (on the faces)" from two participants. This suggests future research
sharper with fewer artifacts", "; however,
also observed negative or
17

---

Published as a conference paper at ICLR 2025
Leftviss VIEW
Figh
Disparity
Figure 8: Results Deep3D. Deep3D does not provide the function to change the stereo baseline, and the vague disparity map on the right side demonstrates its weak stereo effects.
investigate holistic perceptual consistency in stereoscopic videos and finetune models for special subjects like human beings.
Additional user study on Ours vs. Deep3D. We further conducted human evaluation between Ours and Deep3D across the same metrics with a total of 190 random evaluations over 20 random videos, following the same protocol. Pairwise Mann-Whitney tests with Bonferroni correction reveal significant effects on stereo effect (p < 0.001), overall experience (p 0.001), and temporal consistency (p 0.015). We found our method outperforms Deep3D stereo effect and overall experience, yet falling slightly short in temporal consistency.
Similar Fig. TO we visualize Deep3D's disparity map Fig. X
The vague disparity map in
the third column demonstrates weak stereo effects, which matches the statistic results in Table 3 By manually modifying the disparity map or changing the stereo baseline, 3D effects may become apparent. However, Deep3D does not support these functions.
Deep3D
Ours
p-value
|r| (effect size)
Stereo Effect
2.29 (1.63)
5.29 (1.09)
0.001
0.60 (large)
Temporal Consistency
5.37 (1.23)
5.06 (1.25)
0,015 *
0.49 (medium)
Image Quality
5.27 (1.17)
5.12 (1.19)
0.103
0.10 (small)
Overall Experience
3.68 (1.36)
5.08 (1.09)
< 0.001 ***
0.57 (large)
Table Human perception. This table reports results of human perception experiments as mean (std) between Deep3D and Ours. Our method outperforms Deeph3D in stereo effect and overall experience, yet falls slightly short temporal consistency. Mann-Whitney tests with Bonferroni correction reveals significant effects stereo effect (p < 0.001, Z -8224), overall experience (p < 0.001, = - -7922,, and temporal consistency (p = 0.015, = -6.72).
F MORE EXPERIMENTS
Stereo effects visualization. To further verify the quality of stereo effects, we employed the Stereo Transformer (Li al., (021a) to predict disparity maps for our generated stereo videos. As illustrated in Fig. 9 (a). the resulting disparity maps are sharp. with a clear distinction between foreground and background elements, indicating the reasonable 3D effect of our results. Additionally, since majority of pixels are generated by warping the monocular estimated depth and the diffusion model tends to preserve the known pixels, the generated binocular pair maintains the monocular depth very well. This is verified by the example in Fig. (b), which shows disparity maps from monocular estimation (first column) and binocular estimation (second column), and the difference between them (last column). The quantitative difference is 0.63 pixels, which indicates overall good consistency.
Ability to utilize temporal context for inpainting. Our method able harmonize image contents between different temporal frames during inpainting and thus enhance temporal consistency. Figure [] shows one example. When inpainting the right-view frame at f, our method successfully creates content that is consistent with the left-view frame at + 1 (see the generated character "R" in the disoccluded region). Note that such consistency is maintained automatically thanks to frame matrix based denoising, since all temporal frames are taken into account.
Number of cameras used. We reduce the number of cameras between left and right views and show results in Fig. 12 Artifacts (e.g., the fifth leg) tend to occur when the number of cameras is fewer
18

---

Published as a conference paper at ICLR 2025
(a)
(b)
Figure 9: Stereo effects visualization. (a) The generated stereo results are used for predicting disparity maps (Li et al.) (021aa). The distinction between foreground and background elements indicates reasonable 3D effects. (b) Disparity maps are estimated from monocular and binocular images. The error map on the right side demonstrates their consistency.
E2EGVI
ProPainter
RoDDyRR
DynlBaR
Ours
Figure 10: Estimated disparity maps of different methods. The disparity values change reasonably in our method.
than four in this horse case, limited number of cameras can lead rapid changes between video frames that exceed the capabilities of current video generation models.
Efficacy of temporal depth smoothing, In Fig. their depth changes with and without using our depth smoothing operation. From Fig. 14 right side, our approach effectively stabilizes depth changes across frames.
track pixel across video frames and display
Results at different denoising steps. In Fig. 13 we present results at different denoising steps to enhance the understanding of the denoising inpainting process. Without the disocclusion boundary re-injection, noticeable artifacts emerge throughout the denoising process. In contrast, our method progressively fills disoccluded regions with harmonious content.
Left
Right view
BR
(warped)
(impuinted)
Figure 11: Ability utilize unobserved content. Left view: two consecutive images observed by the left view. Right view: the warped and inpainted images at time t. Note that the black region is inpainted with the character "R", matching characters in the second image at time I+1.
19

---

Published as a conference paper at ICLR 2025
3 cameras
Figure 12: Number of cameras between left and right views. Artifacts arise when the number of cameras is too small to exceed the capabilities of video generation models.
Without
Outs
800
500
Figure 13: Results at different denoising steps. Without the use of disocclusion boundary re- injection, artifacts persist throughout the denoising process. contrast, our method gradually fills disoccluded regions with harmonious content.
Alke suniby
Frame 2
Frame 2
Frame
Frame
Images
Depths
Depth changes
Figure 14: Temporal depth smoothing, We track a pixel's depth values across frames and visualize the depth changes. By applying smoothing operations, the depth changes more smoothly.
Consistency across different viewpoints. In Fig. 17 the first row is warped images under different camera views. We generate each view independently and show results in the second row, where the content is not consistent across different views, such as the dragon's 's wing. With the help of the frame matrix, which also regularizes generation in the direction of camera motion, our results in the third row are more consistent.
Apply boundary re-injection to image inpainting (Avrahami et al. 2023). Our method is also applicable previous image inpainting approaches that utilize latent diffusion backbones. In Fig. 19 the masked region is inpainted as a stone, and our method showcases smoother transition between original and new content (e.g., the grassland), thanks to the boundary re-injection designs that update latent features around the mask boundary.
Compare with single-image view synthesis methods. Stereoscopic video be obtained by dealing with each frame individually using a single-image view synthesis method. Here, we compare our approach with SVM (Tucker & Snavely (020)) and AdaMPI (Han al. 2022). and show results at different time stamps in Fig. 20 At specific time stamp (e.g.. ti), both SVG and AdaMPI produce blurry content next to the character "B", whereas our method generates a sharp character "R". This improvement is attributable to our approach's 's ability to leverage information from other frames (i.e., left-view frame time 12). Moreover, the results from AdaMPI and SVM are not temporally consistent because each frame is processed individually. For example, the content next to the character "B" differs between 1 and f2. On the contrary, our approach leverages the video generation model to ensure temporal consistency, allowing the characters in results align with those the left-view frames. More cases are included Fig. 22 Quantitative comparisons are displayed in Table 4
20

---

Published as a conference paper at ICLR 2025
SVM
AdaMPI
Ours
DOVER FVD 1
0.215
0.245
0.584
784
718
599
Table Compare with single-image view synthesis. Our method outperforms SVM and AdaMPI on aesthetic scores, measured by DOVER and FVD.
Ts
WALLZ
62
View
View
Figure 15: Frames sampled from frame matrix. In both cases, each column represents generated video from a camera, while each row corresponds to the generated frames from different cameras at specific timestamp.
G MORE RESULTS
Other trajectories in frame matrix. In the main paper, show generated left and right views. Here, we additionally show the results other trajectories. In Fig. [5] we selectively display frames generated within the frame matrix at different timestamps (3 out of 16) in different camera views (3
21

---

Published as a conference paper at ICLR 2025
View
View2
View3
Figure 16: Frames sampled from frame matrix constructed using a spiral trajectory. Warped and generated frames in different cameras different timestamps.
Warped
Wio FM
Ours
View1
View2
View3
Figure 17: Frame matrix benefits consistent content generation. When each view is processed independently, the dragon's wing varies across different camera viewpoints.
22

---

Published as a conference paper at ICLR 2025
Lunnten
Lumerc
Left
Warped image
Right
Figure 18: We display more generated results in various scenarios.
of 8). From the results, both foreground and background content are coherent across different frames. Furthermore, instead of building the frame matrix using camera moving from left to right, we move the camera along a spiral trajectory. Fig. first and third rows, we selectively show warped images in different camera views (3 out of 16), where disocclusions appear around the plane. Under each warped image, we display the corresponding image with disocclusions filled.
Diverse outputs. As shown in Fig. 21 the output is not unique because the disoccluded regions can be generated with different plausible contents, such as cliffs or trees.
More cases. In this part, more generated results are displayed in Fig. 18 The proposed method works on different scenarios, such as the beautiful church, imaginary scenes, and ships the storm where the whole scene is dynamic. The high-quality generated results in Fig. 18 right column demonstrate the generalization ability of the proposed method. Moreover, Fig. 23 includes results from two complex real-world scenes. The dolphin case features multiple moving objects, while the train
23

---

Published as a conference paper at ICLR 2025
Input
Lakent Diffisson ("A stonc")
Ours ("A stone")
Figure 19: Boundary re-injection in image inpainting. Our approach (blended latent diffusion + disocclusion boundary reinjection) provides a smoother transition between original and inpainted content, such as the grassland.
Left View
BRI
BRIT
SVM
BRY
BRIT
AdaMPI
BRI
BRIDI
Olrs
Figure 20: Compare with single-image view synthesis. SVM and AdaMPI tend to produce blurry and temporally inconsistent results, please note the content next to the character "B" across different time stamps.
Warped image mask
Ours 1
Ours 2
Ours 3
Figure 21: Diverse outputs. The disoccluded regions can be generated with different content.
involves fast-moving object. Our method performs effectively in both scenarios and outperforms results without adopting boundary re-injection design.
24

---

Published as a conference paper at ICLR 2025
SVM
AdaMPI
Ours
Figure 22: More comparisons with single-image view synthesis. SVM and AdaMPI tend
produce blurry results in disoccluded regions.
Warpod
Ouns
Wo DBR
Warped
Outs
Wio WioDB DBR
Figure 23: Complex real-world scenes. Our method is effective in real-world scenes featuring multiple moving objects, such as dolphins, as well as fast-moving objects, like trains. Without using boundary re-injection, results have obvious artifacts.
25


Reviews:
[{"review_id": "nxCxsR2yCB", "reviewer": "Reviewer_P3fm", "summary": "This paper aims to synthesize 3D stereoscopic videos using an off-the-shelf monocular video generation model without finetuning. Concretely, the authors leverage the estimated video depth and propose a novel frame matrix to denoise both spatially and temporally, and thus the model is aware of the exsiting information on the left view and synthesizes consistent right-view videos accordingly. A disocclusion boundary re-injection scheme is also proposed to solve the boundary problem.", "questions": "1. Most videos feature a single movable object or minor movements. How does the proposed method perform when applied to more complex scenes, such as those with multiple moving objects and significant motions?\n2. Typos:\n\nL225 Denosing -> Denoising\n\nL266 refence → reference", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is pose-free and training-free.\n\n2. This paper is clearly written and easy to understand.\n\n3. Extensive experiments demonstrate the effectiveness of the proposed frame matrix and the disocclusion boundary re-injection scheme.", "weaknesses": "1. As the model denoises along both temporal and spatial dimensions, one experiment that is missing is the investigation of varying the number of cameras between the left and right views. How does this variation impact the final quality and overall efficiency of the process? Is it feasible to use fewer internal camera views to save time?\n2. Currently all the experiments have been conducted on the synthesized videos. It would be beneficial to explore how the results look like when applied to real-world videos.\n3. The model heavily depends on a pre-trained depth estimation model, which can overlook thin structures and sometimes produce inaccurate results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ETqZj1EWTz", "reviewer": "Reviewer_zJCL", "summary": "The paper addresses a gap in the generation of stereoscopic videos, particularly corresponding to the advancements in VR/AR technologies. The authors introduce an interesting pose-free and training-free framework that aims to improve the generation of high-quality 3D stereoscopic videos from monocular video inputs. The author utilizes video generation models to enhance 3D consistency while addressing challenges such as occlusion and temporal stability.", "questions": "Missing some reference about multi-view synthesis, please consider reference them:\n\n[1] Chen, Zilong, et al. \"V3d: Video diffusion models are effective 3d generators.\" arXiv preprint arXiv:2403.06738 (2024).\n[2] Zuo, Qi, et al. \"Videomv: Consistent multi-view generation based on large video generative model.\" arXiv preprint arXiv:2403.12010 (2024).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. \n- The proposed frame matrix representation is reasonable, and the extensive experiments support its functionality. Therefore, this paper is sufficiently novel. \n- The proposed method demonstrates a strong understanding of the challenges specific to 3D video generation, including issues with depth estimation and video inpainting. \n- I believe that sufficient experiments and ablation studies are presented to support the approach.", "weaknesses": "The main weaknesses of the proposed method are the disocclusion boundary artifacts, slightly lower temporal consistency compared to Deep3D, and the need for further improvements in holistic perceptual consistency, especially for certain subjects like human faces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ll2hzuZy0W", "reviewer": "Reviewer_dHEi", "summary": "This paper proposes a training-free framework to generate stereo video from single video sequence by using retrained video generation model. The proposed frame matrix leverages the power of video generation model and the joint optimization idea to generate semantically consistent and temporally smooth content, which offers valuable insights and can provide reference directions for future work.. However, besides the frame matrix, the other modules mentioned in methods are more like tricks and are all followed by existing works. The extensive results show that this paper achieve SOTA performance in training-free manner. However, the drawbacks are obvious, the performance is significantly worse than the training methods, and the process is slow due to the need for multiple iterations.", "questions": "See the weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. This paper proposes a training-free manner to generate stereo video from monocular video and achieve SOTA performance in training-free manner.\n\n2. The proposed denoising frame matrix uses pre-trained video generation model as the inpainting model, which is the first one to do this in stereo video generation field and offers insight for leveraging video generation model to assist this task.", "weaknesses": "1. The analysis for denoising frame matrix is insufficient. Although the authors provide the reason for using video generation model, the theory and the high-level reason of why it works are not clear, please show the theoretical analysis of the proposed frame matrix.\n\n2. Lack citations for the methodology followed by other methods. The presentation in the paper contains certain misleading and deceptive elements. Line 153-161, the viewpoint transfer part is widely used in novel view synthesis [R1, R2] and stereo image generation [R3]. Additionally, the proposed boundary re-injection is just a trick, which can not be a contribution. These parts should be revised for better presentation.\n\n[R1] Tucker, R.; and Snavely, N. 2020. Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 551–560.\n\n[R2] Han, Y.; Wang, R.; and Yang, J. 2022. Single-view view synthesis in the wild with learned adaptive multiplane images. In ACM SIGGRAPH 2022 Conference Proceedings, 1–8.\n\n[R3] Wang, X.; Wu, C.; Yin, S.; Ni, M.; Wang, J.; Li, L.; Yang, Z.; Yang, F.; Wang, L.; Liu, Z.; et al. 2023b. Learning 3D Photography Videos via Self-supervised Diffusion on Single Images. arXiv preprint arXiv:2302.10781.\n\n3. Insufficient evaluation metics. In [R1], [R2], [R3], they will use SSIM, PSNR, LPIPS evaluation metrics in novel-view image, which can also be used in stereo video generation. What's more, the temporal consistency is just justified in user study, the quantitative results such as Fréchet Video Distance should also be given.\n\n4. Lack comparison with highly related methods. The methods compared in this paper is too simple. As mentioned in Q2, [R1], [R2], [R3] are the highly related stereo image generation methods. Although they need to train, they own the advantages of high performance and fast inference speed, the authors should give the result in the paper and clarify the strength of their methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)