Deep 3D Pan via local adaptive "t-shaped" convolutions with global and local adaptive dilations

1 INTRODUCTION . Recent advances in deep learning have pushed forward the state-of-the-art performance for novel view synthesis problems . Novel view synthesis is the task of generating a new view seen from a different camera position , given a single or multiple input images , and finds many applications in robotics , navigation , virtual and augmented reality ( VR/AR ) , cinematography , etc . In particular , the challenging task of generating stereo images given a single input view is of great interest as it enables 3D visualization of the 2D input scene . In addition , the falling price and the increasing availability of the equipment required for VR/AR has fueled the demand for stereoscopic contents . The previous works , such as Deep3D ( Xie et al. , 2016 ) , have addressed the right-view generation problem in a fully supervised fashion when the input is the left-view to which the output is the synthetic right-view at a fixed camera shift . In contrast , our proposed Deep 3D Pan pipeline enables the generation of new views at arbitrary camera positions along the horizontal X-axis of an input image with far better quality by incorporating adaptive “ t-shaped ” convolutions with globally and locally adaptive dilations . Our proposed “ t-shaped ” kernel with adaptive dilations takes into account the camera shift amount and the local 3D geometries of the target pixels . Panning at arbitrary camera positions allows our proposed model to adjust the baseline ( distance between cameras ) for different levels of 3D sensation . Additionally , arbitrary panning unlocks the possibility to adjust for different inter-pupillary distances of various persons . Figure 1 shows some generated left and right ∗https : //www.VICLAB.kaist.ac.kr view images for a given single image input by our proposed Deep 3D Pan pipeline , which we call “ monster-net ” ( monocular to stereo network ) . In this paper , we define “ panning ” in the context of 3D modeling , implying that camera movement is in parallel to the center view camera plane . In the following sections , we review the related works to stereoscopic view synthesis and discuss the differences with our proposed method , followed by the formulation of our Deep 3d Pan pipeline and finally , we present outstanding results on various challenging stereo datasets , showing superior performance against the previous state-of-the-art methods . 2 RELATED WORK . Novel view synthesis is a well-studied problem in deep learning-based computer vision , and has already surpassed the classical techniques for both cases of the multiple-image ( Woodford et al. , 2007 ; Liu et al. , 2009 ; Chaurasia et al. , 2013 ) and single-image input ( Horry et al. , 1997 ; Hoiem et al. , 2005 ) . The latter , single-image based novel view synthesis , is known to be a much more complex problem compared to multiple-image based ones . Previous deep learning-based approaches usually tend to utilize one of the two techniques to generate a novel view : ( i ) optical flow guided image warping , and ( ii ) a “ flavor ” of kernel estimation , also known as adaptive convolutions . The first technique , optical flow guided image warping , has been adopted by several authors to train convolutional neural networks ( CNNs ) for optical flow or disparity estimation from single or stereo images in an unsupervised fashion . However , their final goal was not to synthesize novel views . These works include those of ( Godard et al. , 2017 ; Zhou et al. , 2016 ; Gonzalez & Kim , 2019b ; Tosi et al. , 2019 ; Liu et al. , 2019 ; Wang et al. , 2019b ; Ranjan et al. , 2019 ; Lai et al. , 2019 ) . Not all previous existing works have used flow-guided warping for unsupervised training or to regularize supervised methods for optical flow estimation . The work of Im et al . ( 2019 ) implemented plane sweep at the feature level to generate a cost volume for multi-view stereo depth estimation . Such plane sweep can be seen as a type of 1D convolution , similar to the 1D kernel utilized in the second approach of kernel estimation for new view synthesis . On the other hand , the second approach , kernel estimation or adaptive convolutions , has proved to be a superior image synthesis technique and has been incorporated in several different ways . For example : ( 1 ) Flynn et al . ( 2016 ) , in their early DeepStereo , formulated a CNN capable of synthesizing a middle view by blending multiple plane-swept lateral view inputs weighted by a “ selection volume ” , which can be interpreted as a 1D ( or line-shaped ) adaptive convolution ; ( 2 ) in a similar way , Xie et al . ( 2016 ) devised the Deep3D , a non fully-convolutional network that estimates a series of “ probabilistic disparity maps ” that are then used to blend multiple shifted versions of the left-view input to generate a synthetic right-view image ; ( 3 ) The adaptive separable convolutions ( SepConv ) in the work of Niklaus et al . ( 2017 ) approximated adaptive 2D convolutions by two ( vertical and horizontal ) 1D kernels that are applied sequentially to the input current and previous frames ( t0 and t1 ) for the video interpolation problem ; ( 4 ) In the works of ( Zhou et al. , 2018 ; Srinivasan et al. , 2019 ) , although with additional considerations , their multiplane image representation approach can be loosely understood as a 1D adaptive convolution as the final operation involves the reduction of a plane sweep volume ; ( 5 ) Geometric-aware networks in the work of Liu et al . ( 2018 ) indirectly achieved adaptive convolutions by learning a fixed number of affine transformations on an input image , where the resulting affine-transformed images are then blended together to generate one output image ; and finally , ( 6 ) in the work of Gonzalez & Kim ( 2019a ) , the authors developed the Deep 3D Zoom Net , which estimates a selection volume for the “ blending of multiple upscaled versions of the input image ” , which can be treated as an special case of a 1D adaptive convolution . The ( Flynn et al. , 2016 ) and ( Zhou et al. , 2018 ) approaches require two or more images as inputs , thus , greatly reducing the complexity of the synthesis task as most ambiguities are removed by counting on multiple views . In our work , we focus on the single-image based stereoscopic view synthesis task , which is a far more difficult problem as the network needs to understand the 3D geometry in the scene , and to handle complex occlusions , ambiguities and non-Lambertian surfaces . Although the aforementioned methods are distinguished one another , as the different synthesis techniques have their own properties , they can be all interpreted as belonging to a category of adaptive convolutions which are visualized in Figure 2 . As observed in Figure 2- ( a ) , DeepStereo ( Flynn et al. , 2016 ) and Deep3D ( Xie et al. , 2016 ) share the same shape of kernel , that is , a 1D horizontal-only kernel that samples pixels at a fixed interval , or dilation , along the X-axis for each target output pixel . A 1D horizontal-only constant-dilation kernel suffers from three major drawbacks : 1 . Inefficient usage of kernel values . When sampling the positions opposite to the camera movement ( which are the pixel locations corresponding to a1-a3 in Figure 2- ( a ) , assuming a rightward camera shift ) , experiments showed that these kernel values would often be zeros . The same effect repeats when sampling the positions further away from the maximum disparity value of the given scene ( which corresponds to the pixel location at a7 , assuming that the maximum disparity is 2 and the dilation is 1 ) as the network is not able to find valid stereo correspondences for these kernel positions ; 2 . Right-view synthesis is limited to the trained baseline ( distance between stereo cameras ) , as the models over-fit to a specific training dataset with a fixed baseline ; and 3 . The 1D line kernel has limited occlusion handling capabilities , as the network will try to fill in the gaps with the information contained only along the horizontal direction , limiting the reconstruction performance of the models on the occluded areas . In contrast , the kernels predicted by the geometric-aware networks ( Liu et al. , 2018 ) have deformable structures adaptive to the given input images , as shown in Fig . 2- ( b ) . However , only one deformed kernel shape is predicted and shared to synthesize all target output pixels , leading to limited performance . Another drawback of the geometric-aware networks is their complexity , as they require three sub-networks and a super-pixel segmentation step as pre-processing , hindering the processing of high-resolution images . For the Deep 3D Zoom Net ( Gonzalez & Kim , 2019a ) case ( Fig . 2- ( c ) ) , the 1D kernel tends to point to the center of the image , as it performs a blending operation of multiple upscaled versions of the input image . The dilation size of this 1D kernel is adaptive according to the desired 3D-zoom factor . Finally , for the video interpolation case , the SepConv ( Niklaus et al. , 2017 ) approximates an NxN adaptive kernel via a 1xN and an Nx1 component ( see Fig . 2- ( d ) ) which are sequentially applied to the input images to generate the synthetic output . SepConv has , by design , limited receptive fields , as the dilation size is fixed to 1 . Besides , the sequential nature of the kernel forces the vertical component to sample pixels from the output of the horizontal convolution , which could be already degraded due to heavy deformations introduced by the horizontal component . Recent works have also attempted to improve upon the stereoscopic view synthesis by improving the loss functions involved in the CNN ’ s training . The work of Zhang et al . ( 2019 ) proposed a multiscale adversarial correlation matching ( MS-ACM ) loss that learns to penalize structures and ignore noise and textures by maximizing and minimizing the correlation-l1 distance in the discriminator ’ s feature-space between the generated right-view and the target-view in an adversarial training setup . Whereas the objective function is a key factor in training any CNN , we believe that , at its current state , the stereoscopic view synthesis problem can benefit more from a better pipeline that can handle the previously mentioned issues and using the widely accepted l1 and perceptual losses ( Johnson et al. , 2016 ) for image reconstruction , rather than a more complex loss function . Our proposed dilation adaptive “ t-shaped ” convolutions incorporate global ( new camera position along the X-axis ) and local ( 3D geometries of specific target pixels ) information of the input scene into the synthesis of each output pixel value , by not only learning the specific kernel that will generate each output pixel , but also by learning the proper dilation value for each kernel . The “ t ” shape of the kernel allows the network to account for occlusions by filling-in the gaps ( missing information in the output ) due to shifted camera positions using not only left-and-right pixels ( like DeepStereo and Deep3D ) , but also up-and-down neighboring pixel information . In addition , the notions of global and local dilations allow our proposed monocular to stereo network , the monster-net , to generate arbitrarily 3D panned versions of the input center view along the X-axis , a useful feature not present in previous works that allows adjusting for eye-to-eye separation and/or level of 3D sensation .

Summary: This paper proposes a deep learning method to produce "pans" of an input image. That is, simulated images of the scene from translated viewpoints. Unlike some previous work that considers only a fixed baseline (such as the 2nd view of a stereo camera), this approach allows generation of a range of views. A specially crafted convolutional architecture is shown to be well-suited to this problem. Results demonstrate visually pleasing image generation and low metric errors on several datasets.