ImmersePro: End-to-End Stereo Video Synthesis Via Implicit Disparity Learning

## Abstract

Abstract We introduce ImmersePro , an innovative framework specifically designed to transform single-view videos into stereo videos. This framework utilizes a novel dual-branch architecture comprising a disparity branch and a context branch on video data by leveraging spatial-temporal attention mechanisms. ImmersePro employs implicit disparity guidance, enabling the generation of stereo pairs from video sequences without the need for explicit disparity maps, thus reducing potential errors associated with disparity estimation models.
In addition to the technical advancements, we introduce the YouTube-SBS dataset, a comprehensive collection of 423 stereo videos sourced from YouTube. This dataset is unprecedented in its scale, featuring over 7 million stereo pairs, and is designed to facilitate training and benchmarking of stereo video generation models. Our experiments demonstrate the effectiveness of ImmersePro in producing high-quality stereo videos, offering significant improvements over existing methods.
Compared to the best competitor stereo-from-mono we quantitatively improve the results by 11.76% (L1), 6.39% (SSIM), and 5.10% (PSNR).

## 1 Introduction

A stereo movie, also known as a 3D movie, provides three-dimensional visual effects by employing stereoscopic techniques. By capturing or creating separate views for the left and right eyes, a 3D immersive experience can be achieved by using dedicated hardware such as head-mounted displays or autostereoscopic displays.
The disparity between the two views perceived by the viewer’s brain creates the illusion of depth, making the objects in the movie appear at varying distances, thereby enhancing the immersive experience of the film.
Shooting stereo movies in the film industry often involves high costs due to the need for specialized equipment and meticulous post-production processes.
Alternatively, the stereoscopic effect can be created through a post-production process for videos that are shot with monocular cameras.
This post-production process uses stereo conversion, which adds the binocular disparity depth cue to digital images.
It requires significant manual work by artists since inaccurate depth mapping and misrepresentations of occluded areas can cause visual discomfort Devernay & Beardsley ().
In this paper, we propose an automated system that can reduce the time and expense associated with the conversion process, making it more accessible and economically feasible for more films.

Figure: (a) 3D Photo
Refer to caption: extracted/5891240/misc/comp/3dphoto-yt_q-IatK6VsQQ-frame_00004984_left.png

Traditional stereo conversion involves creating disparity maps from single images or sequences and then using them to generate the corresponding stereo pair for the other eye, creating the illusion of depth for stereoscopic viewing.
Recently, many deep learning-based methods (Xie et al., ; Wang et al., ; Shih et al., ; Watson et al., ; Ranftl et al., ) are primarily proposed for image-based stereo conversions, aiming to improve disparities and enhance inpainting effectiveness on occluded areas.
Unlike image data, video data provides additional temporal information, which can yield more detailed disparities and occlusion insights by leveraging information across frames.
To handle video inputs, Chen et al. () synthesizes right-view video sequences by estimating a displacement map to move each pixel to a new location, with a 3D DenseNet.
Temporal3D (Zhang & Wang, ) compromises to use three adjacent left-view frames to predict the single right-view of the middle frame.
Based on our analysis, current stereo conversion frameworks for video sequences are not robust and have several drawbacks. We believe the area is underexplored and there is a large room for improvement. At the same time, we believe the topic will gain in importance due to recent efforts to manufacture stereo displays, e.g., from Apple and Magic Leap.

We introduce ImmersePro, a novel approach designed specifically for video stereo conversion that utilizes the contextual information available across video frames to enhance stereo disparity consistency across the temporal dimension.
For doing so, we collectively build a large-scale stereo movie dataset, Youtube-SBS, with over 7 million stereo pairs from a collection of stereo movies, game films, and music videos.
Due to the absence of ground truth disparities, we propose to use implicit disparities to guide the generation of layered disparities, which outperforms the explicit disparity guidance (*e.g*. a depth estimation model) that was commonly used in previous work. We propose to use a layered disparity representation that refers to a stack of disparity maps corresponding to one image. Each pixel that appears in the image can be reused multiple times, avoiding creating black holes after the warping operation.
This approach ensures that the generated stereo parts strictly adhere to the semantics of the input video, minimizing the need for improvisation and thus preserving the original narrative and visual intent.
As a result, ImmersePro not only maintains the semantic integrity of the original video but also intelligently infers the geometry of occluded areas, enabling consistent right-view generation.
As shown in Figure 1, previous methods may generate artifacts such as texture misalignment or object deformation, whereas our ImmersePro can keep the semantic integrity from the left-view image.
Our main contributions are as follows:

- •
We introduce the YouTube-SBS dataset, an extensive collection of stereo videos sourced from YouTube, featuring over 7 million stereo pairs. This dataset fills the gap to serve as a benchmark for training and evaluating stereo video generation models.
- •
We introduce ImmersePro, specifically tailored for converting single-view videos into stereo videos using layered disparity warping via implicit disparity guidance. Compared to the best competitor stereo-from-mono we quantitatively improve the results by 11.76% (L1), 6.39% (SSIM), and 5.10% (PSNR).

## 2 Background

We discuss previous stereo conversion methods and stereo datasets in this section.

### 2.1 Stereo Conversion Methods

Image-Based Stereo Conversion.
Deep3D (Xie et al., ) relaxes the disparity map into a multi-layer probabilistic map and then multiplies it with several horizontally shifted copies of the input image, which relaxes the non-differentiable warping operation.
Watson et al. () used a warping-and-inpainting framework, which creates stereo training pairs from single RGB images to improve the modern monocular depth estimators.
However, a non-differentiable strategy is used and the inpainting randomly selects the texture from the training set.
Apart from using pretrained depth estimation models, Wang et al. (); Ranftl et al. () use FlowNet2.0 (Ilg et al., ) to estimate optical flows as ground truth disparities.
StereoDiffusion (Wang et al., ) proposes a training-free approach to generate stereo pairs by directly warping the latent space of diffusion models. It requires inversion methods to produce the latents to generate the stereo pair of a given image. The fine details of the resulting photo may vary due to the direct modification of the latent space.
Shih et al. () proposed a layered depth inpainting method that generates a 3D representation by intelligently estimating and filling depth information, particularly in areas where it is missing or uncertain.
Our work does not rely on explicit disparity computation, with the additional consideration of the context within video frames.

Video-Based Stereo Conversion.
Chen et al. () adopts a reconstruction-based approach by using a 3D DenseNet to estimate the disparity map of an input sequence.
Temporal3D (Zhang & Wang, ) estimates the middle frame using three adjacent frames, with the output being a weighted sum of three disparity-warped images.
Additionally, methods such as NVDS (Wang et al., ) may be adopted for consistent depth estimations across video frames.
However, those methods assume the pixels within the left image are adequate for the right image.
Mehl et al. () adopted the warping-inpainting approach with a pretrained depth estimation method (*i.e*. MiDaS (Birkl et al., )) for warping and inpainting with multiple adjacent frames.
Still, this method relies on a single frame depth estimation model that can likely break the temporal consistency between frames. In this work, we propose an end-to-end video stereo conversion method based on implicit disparity guidance across the temporal dimension.

### 2.2 Stereo Datasets

There are limited resources on video-based stereo datasets.
Sintel (Butler et al., ) contains 1064 synthetic stereo images with accurate disparities.
KITTI (Menze & Geiger, ) offers 8.4K frames captured from the real world for autonomous driving.
Wang et al. () introduces a WSVD dataset and proposes to use optical flow as disparities as ground truth for supervision.
Similarly, Ranftl et al. () collected a private 3D movie dataset and extracted ground truth disparities by estimated optical flows to improve depth estimation.
Since different levels of stereoscopic effects may exist for different purposes of a dataset, a movie-specific benchmark dataset is preferable. Ranftl et al. () is the only relevant dataset but it is built on top of real movies with intellectual property right issues. Therefore, we propose a benchmark stereo dataset that contains publicly available content.

**Table 1: Relevant datasets.**
| \topruleDataset | content | GT depth | available | No. frames |
| --- | --- | --- | --- | --- |
| \midruleKITTI (Menze & Geiger, ) | autonomous driving | metric | Y | 8.4K |
| WSVD (Wang et al., ) | mixed | NA | Y | 1.5M |
| 3D Movies (Ranftl et al., ) | movies | NA | N | 75K |
| Sintel (Butler et al., ) | synthetic | metric | Y | 1064 |
| \midruleYoutube-SBS | movies | NA | Y | 7M |
| \bottomrule |  |  |  |  |

## 3 Youtube-SBS

We aim to set up a large-scale publicly accessible benchmark dataset.
The direct collection of 3D movies often encounters legal challenges to publish as an open-source dataset. Therefore, we present Youtube-SBS, an open-source dataset collected from YouTube. This dataset contains over 400 3D side-by-side (SBS) videos. With a particular interest in stereo movies, our dataset primarily consists of movie trailers, game films, and music videos. We explicitly excluded 360-degree virtual reality videos and gameplay videos (that contain user interfaces).
To ensure accessibility for future research, we select videos that (1) have existed for at least one year, and (2) from accounts that have at least 500 followers.
This curated selection includes 423 videos at a standard resolution of 1920x1080.
During the frame extraction, as some videos include a non-stereo intro section, we skip the first 600 frames to capture valid stereo pairs.

To measure the general stereo effects of our dataset, we propose to compute a metric that evaluates the left-right consistency of the disparity.
For a stereo pair with subtle stereo effects, the disparity maps for the left and right images should be almost symmetrical with one another. That is, a point in the left image should have a corresponding point in the right image at the same row but shifted horizontally according to the disparity. For large stereo effects there is an increasing number of occluded and disoccluded areas. In these regions, the right image can no longer be reconstructed from the left image with simple warping (and vice versa).
To compute our metric, we use the optical flow method RAFT (Teed & Deng, ). We also evaluated STTR (Li et al., ) and RAFT-Stereo (Lipson et al., ), but these two methods produced worse results. Note that high consistency means that the left-to-right optical flow $F_{l\rightarrow r}$ and right-to-left optical flow $F_{r\rightarrow l}$ are the negative of each other.
We calculate the consistency $\varepsilon$ as follows:

$$ $\mathcal{E}_{p}=||F_{l\rightarrow r}(p))+F_{r\rightarrow l}(p+F_{l\rightarrow r }(p))||,$ (1) $$

where $p$ is the pixel position of a frame.
We provide a breakdown to demonstrate consistency metric in Table 2 to present the general stereo effects of the dataset.
We compute occluded areas with $\sum_{p}1(\mathcal{E}_{p}>\epsilon)$.
We use $\epsilon=4$ for improved stability on RAFT-computed optical flows.
We present a visual demonstration of different levels of stereo effects in Figure 6.

**Table 2: Flow-based consistency check results. Most frames present subtle stereo effects in the dataset.**
| \topruleoccluded area | $<10\%$ | $<20\%$ | $<30\%$ | $<40\%$ |
| --- | --- | --- | --- | --- |
| \midrulePercentage | 71.27% | 84.60% | 91.30% | 94.71% |
| \bottomrule |  |  |  |  |

## 4 Method

A stereo video sequence $I=\{I^{l},I^{r}\}$ contains left and right video sequences of $I^{l}\in R^{T\times H\times W\times 3}$ and $I^{r}\in R^{T\times H\times W\times 3}$, respectively.
We use $T,H,W$ to denote the video sequence length, video height, and video width, respectively.
We aim to predict a right video sequence $\hat{I}^{r}$ based on the input left video sequence $I^{l}$ to make $\hat{I}=\{I^{l},\hat{I}^{r}\}$ presents similar stereo effects as ${I}$.

As shown in Figure 2, our method compromises six stages. First, we use a dual branch architecture (section 4.1) that consists of a disparity branch and a context branch, to extract disparity and semantic features, respectively. Second, we apply spatial-temporal self-attention (section 4.2) on each scale feature to achieve multi-frame awareness. Third, we fuse the multi-scale features to obtain implicit disparity features (section 4.3). Fourth, we then use a spatial-temporal cross-attention module (section 4.2) to inject contextual information into the implicit disparity features to obtain layered disparity features (section 4.4). Fifth, right-view video sequences can be estimated by warping through layered disparities. Finally, we enrich the estimated right-view sequences with a context fusion module.

Figure: Figure 2: Illustration of ImmersePro framework. Our network contains six parts: (1) dual-branch feature extractors for extracting disparity features and context features (section 4.1), (2) multi-scale spatial-temporal self-attention to refine disparity features (section 4.2), (3) implicit disparity to generate stereo images without explicit disparities (section 4.3), (4) spatial-temporal cross attention block to inject contextual information into the implicit disparity features (section 4.2), (5) layered disparity to obtain the estimated right view video sequences (section 4.4), and (6) context fusion to enrich the estimated right view video sequences with detailed semantic information (section 4.5).
Refer to caption: x1.png

### 4.1 Dual Branch Architecture

We use a dual-branch architecture to enhance stereo video conversion by separately processing disparity and contextual information, as shown in Figure 2.
We employ a pretrained DepthAnything (Yang et al., ) model for the disparity branch to extract disparity-oriented feature maps, while a context feature extractor with the same architecture from Zhou et al. (); Li et al. ()’s encoder is used to extract contextual semantic features.

The disparity branch operates on multiple scales, extracting features at $1/2$ and $1/4$ resolutions of the original input to capture detailed disparity information.
The disparity features are from the decoder of the model(^1^11We use the output from the neck of the model, as implemented by [https://github.com/huggingface/transformers](https://github.com/huggingface/transformers).).
This branch utilizes spatial-temporal self-attention modules (section 4.2) to prioritize relevant spatial and temporal details on different scales, ensuring that the model focuses on areas with significant disparity changes or movement.
After combining the multi-scale features into $1/2$ resolution with a fusion block, we apply softmax to these features to create a probability distribution that represents the implicit disparities.
The implicit disparity is used to select the appropriate pixels from a stack of the multiple horizontally shifted copies of the input image (section 4.3). By encouraging accurate selection, these features implicitly represent the disparity for stereo conversion.

Figure: Figure 3: Visual demonstration of the implicit disparity guidance. We can observe that (1) the implicit disparity module tries to resolve the disparity from the given image, and (2) our method can significantly rectify the error introduced by the implicit disparity estimations. Our method offers a significant improvement regarding clarity with less irregular texture deformation on the image. The implicit disparity map contains multiple channels and we apply $argmax$ to obtain the visual output.
Refer to caption: x2.png

Concurrently, we use a stack of convolution layers as the context encoder. We experimented with multiple encoder architectures and settled on the architecture without aggressive downsampling. The details for the context encoder are presented in Section A.2.
The context branch focuses solely on capturing texture information. This branch processes texture at $1/2$ the original resolution, aligning with the disparity branch’s output.
Finally, with spatial-temporal cross-attention modules to fuse the implicit disparity and texture information, we apply a layered disparity warping (section 4.4) to obtain the final predicted right-view.

### 4.2 Spatial-Temporal Attention

Video transformers have demonstrated excellent performances in video-based tasks such as video segmentation (Duke et al., ), video-text feature mapping (Li et al., ), and video inpainting (Li et al., ; Zhou et al., ).
This work builds sparse video transformers on top of the ProPainter’s version, considering the highly redundant and repetitive textures in adjacent frames.
We remove the mask guidance in the original model and use a temporal stride of 2 to avoid redundant key/value tokens within each transformer block and to improve the computational efficiency.
Aside from spatial-temporal self-attention, we also use spatial-temporal cross-attention to fuse features from different sources.

Given a video feature sequence $E_{s}\in\mathbb{R}^{T_{s}\times H_{s}\times W_{s}\times C}$, we first perform soft split (Liu et al., ) to generate patch embeddings $Z\in\mathbb{R}^{T_{s}\times M\times N\times C_{z}}$.
Subsequently, $Z$ is partitioned into $m\times n$ non-overlapping windows, yielding the partitioned embedding features $Z_{w}\in\mathbb{R}^{T_{s}\times m\times n\times h\times w\times C_{z}}$, where $m\times n$ denotes the number of windows and $h\times w$ denotes their size.
For self-attention transformer blocks, we obtain the query $Q$, key $K$, and value $V$ from $Z_{w}$ through three linear layers, respectively.
For cross-attention transformer blocks, we repeat the above process to obtain embeddings $Z_{c}\in\mathbb{R}^{T_{s}\times m\times n\times h\times w\times C_{z}}$ from another feature sequence $E_{c}\in\mathbb{R}^{T_{s}\times H_{s}\times W_{s}\times C}$. Note that $Z_{c}$ shares the same shape with $Z_{w}$. Then $Q$ is extracted from $Z_{w}$ whilst $K$ and $V$ are extracted from $Z_{c}$.
For both self-attention and cross-attention mechanisms, the final embedding features are gathered using soft composition Liu et al. () for further processing.

### 4.3 Implicit Disparity

For stereo vision, different from common generative tasks, the generated right view requires a precise match to the input view with as little improvisation as possible.
The stereo pair of an image is commonly constructed by obtaining the disparity map to find the shifting distances of each pixel within the input view.
Assuming $d_{i,j}$ is the disparity value at pixel location $(i,j)$ in the left image, the corresponding pixel in the right image is:

$$ $I^{r}_{i,j}=I^{l}_{i,j+d_{i,j}}.$ (2) $$

It is typically a non-differentiable operation due to its piecewise nature. Jaderberg et al. () propose to use sub-gradients for backpropagation through spatial transformations to handle such non-smooth operations, enabling differentiable warping.

Xie et al. () proposed another approach to use a depth selection layer to align the output right view to the source input view structure. Subsequent works such as Zhang et al. () follow a similar idea. We employ it as auxiliary supervision. We found this method to be suitable for guidance only. Directly using it to compute the output leads to blurry results.
*Implicit disparity* predicts a probability distribution across possible disparity values $d$ at each pixel location. $p^{d}_{i,j}$, with $\sum_{d}p^{d}_{i,j}=1$, denotes the probability of pixel $(i,j)$ having disparity $d$.
We denote an image that is shifted by $d$ pixels horizontally as $I^{d}_{i,j}=I_{i,j-d}$. We then obtain the right-view pixel values as:

$$ $\hat{I}^{aux}_{i,j}\sum_{d}=I^{d}_{i,j}p^{d}_{i,j}.$ (3) $$

where $\hat{I}^{aux}$ is the auxiliary predicted right view. We use $V^{d}_{i,j}=I^{d}_{i,j}D^{d}_{i,j}$ for subsequent computations. This approach estimates the stereo pair of a given image without an explicit disparity map, serving as a relaxation of the warping operation in Equation 2.
Without implicit disparity, our model can hardly converge as shown in Section 5.1.

Figure: (a) Left image
Refer to caption: extracted/5891240/misc/layered2/visual_2_raw_left.png

Figure: ALGORITHM 1 Synthesis from layered disparities.

### 4.4 Layered Disparity

The implicit disparity is a summation-based approach that computes pixel colors as a blend of other pixel colors, weighted by the estimated probabilities. This may produce good results with a correct estimation, but it may introduce artifacts such as blurring if the estimation is inaccurate. The final output visually improves if each pixel location is selected from a set of candidate disparity layers, rather than blending all the layers.
The proposed *Layered Disparity* uses a smaller stack of candidate layers, and each layer represents disparity information.
Therefore, our layered disparity representation is a stack of disparity maps. We use a differentiable warping (Jaderberg et al., ) operation to warp the input image to an output image. While a single disparity map already defines a solution to the problem, there may be problems due to occlusion and disocclusion artifacts. These problems can then be fixed by other layers.
Our approach avoids the mentioned blending problem. Meanwhile, we maximize the reuse of pixel information within the image while at the same time avoiding generating image holes.

We use implicit disparity $V^{d}_{i,j}$ as a guidance to generate layered disparities.
First, we employ three Conv-ReLU blocks to refine the $V^{d}_{i,j}$ to shrink them from $\mathbb{R}^{T_{s}\times H\times W\times D}$ into $\mathbb{R}^{T_{s}\times\frac{H}{2}\times\frac{W}{2}\times D}$, where $D$ is the number of stacked disparities. We then apply the spatial-temporal cross-attention process, as mentioned in Section 4.2. With the attention-applied features, a deconvolution operation and three Conv-ReLU blocks are used to obtain the final layered disparity $LD_{i,j}^{d}$. Here, $d=7$ since we use 7 disparity layers in our work. We then apply the differentiable warping operation with the layered disparity to obtain layered warped images $\hat{I}_{i,j}$ and masks $\hat{M}_{i,j}$, respectively. We select pixel values according to the layered masks as in Algorithm 1. As shown in Figure 3, our proposed approach significantly improved the visual quality compared to the output from the implicit disparity layer. Figure 4 visualizes an example of learned disparity maps from the proposed layered disparity representation.

### 4.5 Context Fusion

The final stage of our network focuses on enriching semantic details while maintaining the learned right-view structure.
The context fusion module integrates semantic and disparity features from a video sequence by concatenating the encoder feature map with layered disparity features to form a fused representation. These fused features are then processed through spatial-temporal attention (section 4.2), enabling global context awareness.
We apply spatial-temporal attention modules at 1/2 the original resolution, as mentioned in section 4.1.
To retain structural integrity, a residual connection reintroduces the refined transformer output into the original fused feature map.
We then apply a deconvolution to obtain a texture map in the original resolution, then enrich the texture map by three Conv-ReLU blocks. Next, the module supplements the layered disparity-warped images from section 4.4 with the enriched feature map. To be specific, a median blur with $3\times 3$ kernels is first applied to the warped images to reduce noise and improve local smoothness before concatenating them with the enriched feature map. A semantic residual is then derived by passing this combined map through three Conv-ReLU blocks. The final output is produced by combining the blurred image with the semantic residual.
This approach ensures that the final result maintains sharp textures while preserving structural consistency, achieving a balance between local detail and global coherence.

## 5 Results

We implement our method using Pytorch and train on four NVIDIA A100 (80G) GPUs for 50,000 iterations (approx. 3 days). Models are trained for 40,000 iterations for our ablations. At training time, we first resize the input sequence to $422\times 422$ and then randomly crop the resized video sequence to $384\times 384$. Each input sequence contains 8 frames.
We use L1 loss during training to encourage an accurate reconstruction of the right-view images using both implicit and layered disparities. In addition, an LPIPS (Zhang et al., ) loss is used for better reconstruction results.
An AdamW (Loshchilov & Hutter, ) optimizer is used. We use $3e-5$ learning rate while image losses are computed within the range of $(-127.5,127.5)$.
We evaluated our method on our test set which includes 43 video sequences with 558K frames.

**Table 3: Benchmark results. The best and second-best results are highlighted in green and yellow, respectively. $*$ indicates the model is not converged.**
| \toprule | L1 $\downarrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ |
| --- | --- | --- | --- |
| \midruleDeep3D | 0.2215 | 0.1935 | 11.9089 |
| 3D Photo | 0.1069 | 0.3463 | 16.3658 |
| Stereo Diffusion | 0.0816 | 0.4651 | 18.6684 |
| stereo-from-mono | 0.0646 | 0.5685 | 20.7788 |
| \midruleOurs w/o implicit disparity $*$ | n/a | n/a | n/a |
| Ours w/o layered disparity | 0.0885 | 0.4717 | 19.0523 |
| Ours w/o attention blocks | 0.0593 | 0.5894 | 21.4162 |
| Ours w/o context fusion | 0.0588 | 0.5959 | 21.6649 |
| \midruleOurs | 0.0570 | 0.6048 | 21.8387 |
| \bottomrule |  |  |  |

### 5.1 Comparison with State-of-art models

Benchmark methods. We compare our method with three state-of-the-art methods including Stereo-from-mono (Watson et al., ), 3D Photography (Shih et al., ), and StereoDiffusion (Wang et al., ). Note that those methods are designed for image-based stereo conversion purposes. We are not aware of any open-source implementations for video stereo conversion. We use official implementations for the selected methods.

Benchmark settings.
Due to the high runtime of those methods (especially for StereoDiffusion which is required to perform inversion (Mokady et al., ) for each image), we compare those methods with a subsampled dataset every 3 seconds (72 frames). At test time, we process 8 frames as input where the last 2 frames are taken as reference frames. We use widely employed L1, SSIM, and PSNR to evaluate the quality of the generated stereo pairs.

Figure: Figure 5: Visual demonstration of the disparity analysis results. Our network predicts reasonable stereo effects but may be stronger or weaker if compared to the ground truth. The L2R disparity computes the left-to-right disparity using RAFT-Stereo (Lipson et al., ).
Refer to caption: extracted/5891240/misc/disp_eval/yt_qju6fl03MYo-frame_00001000/left_img.png

Benchmark results.
Our qualitative and quantitative results are shown in Figure 1 and Table 3, respectively. The visual results show that other methods tend to generate right-view images with texture deformations.
To be specific, 3D photo struggles to find accurate depth cues with MiDaS (Birkl et al., ) depth estimation model, resulting in inaccurate warping on given images. Stereo-from-mono can generate images well but often comes with unpleasant black dots around the warping shapes.
StereoDiffusion requires using null-text inversion Mokady et al. () to convert a given image to the latent space and then warp the latent features to create the right-view image. It highly depends on the performance of the inversion, which creates unstable performances. As shown in our table, our method yields better numerical results. This finding aligns with the visual results. In addition, our accompanying videos demonstrate better stability in terms of jittering and shaking. Please watch the accompanying videos with 0.5 speed to see the artifacts generated by the different methods.

Ablation results. Table 3 shows our ablation results. We show that our method is not going to converge without using implicit disparity guidance, while a significant performance drop may occur when removing our proposed layered disparity. We show that our layered disparity generates better visual quality in Figure 3 compared to the outputs from implicit disparities. Though not significant, the attention blocks can slightly improve the overall performance, while the context fusion module contributes significantly.
Additional experiments including alternative masking strategies, the inclusion of the context fusion module, flow-guided feature propagation, and different backbone choices are included in our supplementary material.
Lastly, we show our method may generate different levels of stereo effects in Figure 5 compared to the ground truth, but this is expected due to the underdetermined nature of the problem, and we consider our solution also as reasonable.

## 6 Discussion And Limitation

To enhance the viewing experience, films sometimes employ a stronger stereoscopic effect at the start and end, while moderating it in the middle to ensure viewer comfort Neuman (); Ranftl et al. (). Thus, the stereo parameters such as focal length, are hard to retrieve even for the same film.
Theoretically, the precise reproduction of the right view is impossible without knowing the stereo parameters in advance.
By learning through a large-scale dataset, ImmersePro estimates its average disparity, then tries to create an average-level stereo effect for input videos rather than reproduce the precise right pair.
Therefore, as shown in Figure 5, our model may produce reasonable but “inaccurate” stereo effects if compared with the ground truth.

A reasonable stereo conversion pipeline involves a warping-and-inpainting process, where the inpainting operation fills the black holes created by the warping operation. One sample work is stereo-from-mono (Watson et al., ) that performs inpainting with a randomly sampled image from the training dataset. In a way, our method can be seen as an improvement to stereo-from-mono by intelligently selecting the correct regions for inpainting.
However, this strategy works for creating stereo movies with “subtle” stereo effects without the need for significant inpainting.
As we observed in most 3D movie examples, very few movies contain strong stereo effects.
Notably, our method cannot produce strong stereo effects due to the limited dataset and limited inpainting capabilities. In future work, we would like to investigate how Nerf (Mildenhall et al., )-based inpainting can be used for stereo-movie generation.

## 7 Conclusion

This work presents an end-to-end video-based stereo conversion method that generates right-view video sequences according to the input video. Our method automatically utilizes layered disparity maps on top of implicit disparities. Additionally, we propose Youtube-SBS, a large-scale stereo dataset that is publicly available for benchmarking purposes.
Extensive qualitative and quantitative evaluations demonstrated the robustness of our approach against previous works.

## SUPPLEMENTARY MATERIAL

We present implementation details and additional experiments in our supplementary material. Please watch the accompanying videos with 0.5 speed to see the artifacts generated by the different methods.

## Appendix A Technical Details

### A.1 Visual Reference for Stereoeffects

We provide a visual reference for the optical flow analysis as below:

Figure: Figure 6: Visual demonstration of the different levels of stereo effects with the dataset.
Refer to caption: extracted/5891240/misc/dataset/fc-yt_kfquE1nqM6A-left-frame_00018648_-0.000202.png

### A.2 Context Encoder

Our context encoder uses a stack of convolutional layers to obtain semantic features from the input image as below. Starting from the $5^{th}$ convolution layer, the extracted features are the concatenation of the features from the current and previous layers.

## Appendix B Additional Experiments

### B.1 Alternative Mask Selection Algorithm

To avoid having multiple pixels being mapped to the same pixel location $i,j$, we use an algorithm to produce $[0,1]$ masks so that different layers cannot interfere with each other as shown in Algorithm 1.
In addition, we further tested another design where the mask value selection algorithm Algorithm 2 generates mask values $\in\{-1,0,1\}$, to allow more interactions between layers.
However, though Algorithm 2 can better resolve complicated scenarios, we found the intermediate implicit disparity layers often fail to resolve disparities correctly, as shown in Figure 7.
In general, we found that the Algorithm 2 tries to weaken the disparity cues, resulting in smoother output with weaker or wrong disparity maps.

Figure: ALGORITHM 2 Synthesis from layered disparities.

Figure: Figure 7: Visual demonstration of the implicit disparity output for different masking strategies. The implicit disparity map contains multiple channels and we apply $argmax$ to obtain the visual output.
Refer to caption: x27.png

### B.2 Context Fusion Module

As shown in Table 3, the inclusion of the context fusion module significantly enhances the overall statistical performance. Moreover, as demonstrated in the accompanying videos, this module greatly improves the temporal consistency of the generated videos. However, we observed potential artifacts in frames with complex feature patterns, as illustrated in Figure 8. We suspect that these edge cases could be mitigated with a larger training dataset that includes greater internal variance, allowing the model to better handle such intricate scenarios.

Figure: Figure 8: Visual demonstration of the failed edge cases of context fusion.
Refer to caption: x39.png

### B.3 Flow-Guided Feature Propagation

Video feature propagation and deformation have shown their effectiveness for many video-based tasks Xue et al. (); Wang et al. (); Haris et al. ().
The flow-guided deformation concept is particularly suitable for the stereo conversion scenario as the pixel shifting nature according to the disparities.
Similar to E^2FGVI Li et al. () and ProPainter Zhou et al. (), we use a similar design of flow-guided feature propagation module, that features bi-directional optical flow-guided deformable alignments that built on top of deformable convolution networks (DCN) Dai et al. (); Zhu et al. ().

Given extracted features $\{E_{t}|t=1...T\}$ from a feature encoder where $T$ is the total number of frames.
Under the context of stereo conversion, the forward flow $F_{t\rightarrow t+1}$ helps to track the movement of occluded regions from frame $t$ to frame $t+1$. When the pixels within the occluded areas of frame $t$ are found in the valid regions of frame $t+1$, this information can be utilized effectively by warping the backward propagation feature $\hat{E}_{b}^{t+1}$ from frame $t+1$ back to frame $t$, guided by the forward flow ${F}_{t\rightarrow t+1}$.
On top of E^2FGVI’s approach, we include flow validation maps $M_{t+1\rightarrow t}$ by consistency check introduced by ProPainter.
The consistency check compares the forward and backward optical flows to ensure the correctness of the used optical flows. Similar to Equation 1, the consistency error is computed as follows:

$$ $\mathcal{E}_{t\rightarrow t+1}(p)=||F_{t\rightarrow t+1}(p))+F_{t+1\rightarrow t }(p+F_{t\rightarrow t+1}(p))||_{2}^{2},$ (4) $$

where $p$ is pixel positions of the frame.
Then the flow deformation offsets $\tilde{o}_{t\rightarrow t+1})$ are computed with the DCN network, where a concatenation of the forward flow ${F}_{t\rightarrow t+1}$, backward propagation feature $\hat{E}_{b}^{t+1}$, warped backward feature $\mathcal{W}(\hat{E}_{b}^{t+1},{F}_{t\rightarrow t+1})$, and flow validation maps $M_{t+1\rightarrow t}$ is used as the condition, where $\mathcal{W}$ is warping operation. The flow-guided alignment propagation is then:

$$ $\hat{E}_{b}^{t}=\mathcal{R}(\mathcal{D}(\hat{E}_{b}^{t+1};F_{t\rightarrow t+1} +\tilde{o}_{t\rightarrow t+1}),f_{t}),$ (5) $$

where $\mathcal{D}(\cdot)$ is the deformable convolution layers and $\mathcal{R}(\cdot)$ fuses the aligned and current features.

However, we found the disparity cannot be learned with those flow-guided propagation modules. We suspect the feature map deformation and alignment can break the internal disparity features, resulting in a failed learning of the implicit disparity maps.

### B.4 Different Backbones For the Depth Branch

We provide additional results in table 4. We experimented with MiDaS instead of DepthAnything. The results indicate that different depth estimation backbones do not affect the performance of our proposed method.

**Table 4: Additional results. The best results are highlighted in green.**
| \toprule | Depth Backbone | L1 $\downarrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ |
| --- | --- | --- | --- | --- |
| \midruleOurs w/o context fusion | MiDaS | 0.0590 | 0.6014 | 21.6572 |
| Ours w/o context fusion | DepthAnything | 0.0588 | 0.5959 | 21.6649 |
| \bottomrule |  |  |  |  |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weakness section.
soundness: 3
strengths: 1. These paper introduces a large dataset for stereo video generation, which will benefit the research in this area.

2. The design for the implicit disparity is reasonable, which is also consistent with the findings from Vimeo that the direct optical flow is not good for image systhesization.
summary: This paper introduces a method to convert single-view videos into stereo videos. The authors propose a dual-branch architecture with spatial-temporal attention layers, eliminating the need for explicit disparity maps. The authors also build a large scale stereo video dataset, containing 7M stereo frames from 423 videos. The experiments demonstrates the effectiveness of the method and dataset.
weaknesses: 1. The dataset details are missing. 

(a) How many frames per video? Does the videos have the same length? If not, what's the length distribution?

(b) Stereo videos often contain multiple scenes. These video clips are not consistent. The author only measured the spatial consistency between the left and right view. However, it seems that the author missed the temporal consistency, which is important for the stereo videos. How does the author handle it?

(c) In Eq. 1, the authors leveraged the optical flow estimated by RAFT to measure the error. However, optical flow is not accurate in the stereo pairs as it containing the movement in the vertical direction. The author should only consider the horizential direction. Although authors claimed that "We also evaluated STTR (Li et al., 2021) and RAFT-Stereo (Lipson et al., 2021), but these two methods produced worse results.", the aurhors should find a more accurate disparity estimation method or constrain RAFT in the horizential direction.

2. Experiments

(a) For a fair comparison, the authors are required to re-train or finetune the state-of-the-art methods on the new datasets. 

(b) To check the effectiveness of the proposed dataset, the authors need to evaluate image-based models trained on the new dataset on the previous image-based dataset. 


Minor

1. "6.39%(SSIM), and 5.10% (PSNR)" is not professional. Most of the time, we only report the differences, e.g. 1.1 dB in term of PSNR. As PSNR is based on the log function, percentage is not correct.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1) What is the computational complexity of predicting a view in the proposed framework? Is there any latency?
2) Why does the method fail to converge when “implicit disparity” is removed? Can this component be replaced with an alternative approach?
soundness: 2
strengths: + The introduction of a dual-branch architecture and implicit disparity guidance for stereo video generation 
+ Introduction of the YouTube-SBS dataset 
+ Ablation studies on various modules of the framework
summary: The paper presents a framework for converting single-view videos into stereo videos using a dual-branch architecture ((1) disparity branch to extract disparity-oriented feature maps, (2)  context feature extractor to extract contextual semantic features. ) with spatial-temporal attention. This approach leverages implicit disparity guidance to generate stereo pairs without the need for explicit disparity maps, thereby reducing errors. Additionally, the authors introduced the YouTube-SBS dataset, which includes 423 stereo videos and over 7 million stereo pairs, for training and benchmarking purposes. Experimental results demonstrate that ImmersePro enhances the quality of stereo videos compared to existing methods.
weaknesses: - Lack of complexity analysis and comparison with other methods, such as “stereo-from-mono.”
- Ablation studies do not provide conclusive evidence of the necessity of specific modules. For instance, the ablation on context fusion shows a PSNR drop of less than 0.2 dB.
- The paper lacks sufficient novelty in theoretical contributions or architectural design. 
- The main figure on architecture is not very readable; a zoom-in on implicit disparity (as the contribution) would help in understanding the architecture.
- The complexity of the stack of multiple horizontally shifted copies is not discussed, and it is not explained why only horizontal shifts are sufficient for prediction.
- Evaluation on other benchmark datasets is not shown.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: * Was the YouTube-SBS dataset filtered to include only videos with public domain and CC-BY license or does it also include content with the standard youtube license?  Can you give a breakdown of percentage of each license?
soundness: 2
strengths: * Timely problem
* Method is clear and intuitive
* YouTube SBS dataset preprocessing is a significant contribution
* Results look good
summary: This paper presents both a new dataset for mono to stereo video conversion based on permissively licensed stereo youtube videos as well as a method for performing the conversion automatically.  The authors claim the following contributions:
* YouTube-SBS dataset which includes 7 million stereo pairs from over 400 videos.
* ImmersePro - a mono to stereo conversion method

The authors also present a number of qualitative and quantitative experiments validating the approach.
weaknesses: * Figure 3 rows 2 and 3 illustrating implicit disparity and the associated warped image looks more like a bug rather than a valid result.  Is it the case that row 3 was obtained by training a network with layered disparity included, but then omitted in the final rendering?  Or does it represent a model where layered disparity was excluded from both training and rendering?  Row 3 shows more than just "blurring" artifacts as suggested in section 4.4 and could impact the ablations in table 3.

* Two of the baselines are pretty old at this point and the proposed method is making use of Depth Anything, which presumably offers quite an advantage.  I think a very simple baseline would be to use Depth Anything v2 on the left view and then forward warp it to produce the right view.  How the holes from the forward warp are filled in is an open question, but you could generate a mesh for warping to make sure all pixels are filled.

* There are no metrics related to temporal consistency.  I'd recommend reading Luo, et.al, "Consistent Video Depth Estimation" SIGGRAPH 2020 to get an idea on how to illustrate temporal consistency.

* Section 6 paragraph 1 raises concerns about the metrics used to compare the proposed method and baselines.  Many of these methods do indeed need to be treated as "scale invariant" in one way or another (whether it be the focal length, IPD, or something else).  Was there care taken when doing comparisons with the baselines to ensure a scale was selected that best minimized the error reported in the metrics?

* The "GT" presented in figure 5 clearly should not be interpreted as GT.  The predicted L2R disparity also appears to be well below the bar for modern disparity prediction.

* I think if the authors want to show GT disparity, then they should use a dataset like MPI Sintel.  They could also use this for the temporal consistency evaluation.  Reading Kopf, et.al. "Robust Consistent Video Depth Estimation" CVPR 2021 might be useful here.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: See the weaknesses.
soundness: 2
strengths: 1. The proposed implicit disparity and layered disparity improve the results.

2. This paper collect a new stereo video dataset.
summary: This paper propose an implicit disparity map to avoid the texture artifact problem occurred in explicit disparity and a layered disparity selection module to improve the result. Then the authors collect a new stereo video dataset to assist stereo video generation. Some self-attention, cross-attention and context fusion modules are added to enrich the paper. The extensive experiments show they achieve great spatial and temporal consistency in stereo video.
weaknesses: 1. The problem of existing methods is unclear. As shown in Line 051-052, what drawbacks? The authors should give detailed discussion and show the visualization or experiment at the beginning to better understanding.

2. Vertical distortion in the dataset. In section 3 Line 180-181, the authors say that the performance of RAFT-Stereo is worse than RAFT, which does not align with intuition as RAFT-Stereo is specifically designed to implement horizontal offsets for stereo matching. The probable reason is that the proposed dataset contains vertical offset, which is not suitable for stereo video synthesis. The author should do more analysis about the proposed dataset to show the accuracy of their dataset.

3. The presentation of the method section is poor. 

       a) Line 348 Eq. 3, typo in equation.
       b) Line 351, what is $D_{i,j}^d$
       c) How the dimension D (7) is obtained.
       d) Why the operation of layered disparity can be used to select the layer. The authors should give the illustration in detail (not just an algorithm).
4. Question about the novelty. The disparity index selection is not a novel operation as it is widely used in stereo matching Fast-ACV [1], the authors should clarify the novelty of their layered disparity compared with the one used in stereo matching. What's more, why implicit disparity works better than explicit disparity, as in operation, the difference is just the turn (calculating the disparity by weighted-summation first and using warp function (explicit), using warping function first and calculating the image by weighted-summation). The authors should give more theoretical analysis and high-level insight to validate the motivation.

[1] Xu, G., Wang, Y., Cheng, J., Tang, J., Yang, X. (2023). Accurate and efficient stereo matching via attention concatenation volume. IEEE Transactions on Pattern Analysis and Machine Intelligence.

5. Lack experiments.

       a) Lacking comparison with explicit disparity. The explicit disparity can also be used and the disparity index selection can be applied, this experiment is vital for showing the motivation of this paper.
       b) Lacking comparison with recent SOTA methods. This paper in Line 051-053, 135-136 claims that warping-inpainting based methods work bad. However, they does not compare with this methods which weakens the contribution of this paper. The author should add comparison with following SOTA methods [2], [3], [4]. Note that although these methods do the task of stereo image generation from mono, the performance are great when directly applying to video.
[2] Tucker, R.; and Snavely, N. 2020. Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 551–560.

[3] Han, Y.; Wang, R.; and Yang, J. 2022. Single-view view synthesis in the wild with learned adaptive multiplane images. In ACM SIGGRAPH 2022 Conference Proceedings, 1–8.

[4] Wang, X.; Wu, C.; Yin, S.; Ni, M.; Wang, J.; Li, L.; Yang, Z.; Yang, F.; Wang, L.; Liu, Z.; et al. 2023b. Learning 3D Photography Videos via Self-supervised Diffusion on Single Images. arXiv preprint arXiv:2302.10781.