Deep 3D Pan via local adaptive "t-shaped" convolutions with global and local adaptive dilations
Authors: Juan Luis Gonzalez Bello
Munchurl Kim
Venue:  (2020)
Keywords: Deep learning
Stereoscopic view synthesis
Monocular depth
Deep 3D Pan

Abstract: Recent advances in deep learning have shown promising results in many low-level vision tasks. However, solving the single-image-based view synthesis is still an open problem. In particular, the generation of new images at parallel camera views given a single input image is of great interest, as it enables 3D visualization of the 2D input scenery. We propose a novel network architecture to perform stereoscopic view synthesis at arbitrary camera positions along the X-axis, or “Deep 3D Pan”, with “t-shaped” adaptive kernels equipped with globally and locally adaptive dilations. Our proposed network architecture, the monster-net, is devised with a novel t-shaped adaptive kernel with globally and locally adaptive dilation, which can efficiently incorporate global camera shift into and handle local 3D geometries of the target image’s pixels for the synthesis of naturally looking 3D panned views when a 2-D input image is given. Extensive experiments were performed on the KITTI, CityScapes, and our VICLAB_STEREO indoors dataset to prove the efficacy of our method. Our monster-net significantly outperforms the state-of-the-art method (SOTA) by a large margin in all metrics of RMSE, PSNR, and SSIM. Our proposed monster-net is capable of reconstructing more reliable image structures in synthesized images with coherent geometry. Moreover, the disparity information that can be extracted from the “t-shaped” kernel is much more reliable than that of the SOTA for the unsupervised monocular depth estimation task, confirming the effectiveness of our method.

Published as a conference paper at ICLR 2020
DEEP 3D PAN VIA LOCAL ADAPTIVE "T-SHAPED" CONVOLUTIONS WITH GLOBAL AND LOCAL ADAPTIVE DILATIONS
Juan Luis Gonzalez. Bello Munchurl Kim Department of Electrical Engineering Korea Advanced Institute Science and Technology (juanl ui sig,, mk mee) @ka sss.s... ac. kr
ABSTRACT
Recent advances in deep learning have shown promising results in many low-level vision tasks. However, solving the minll-image-basse view synthesis is still an open problem. In particular, the generation new images at parallel camera views given single input image is of great interest, it enables 3D visualization of the 2D input scenery. We propose a novel network architecture to perform stereo- scopic view synthesis arbitrary camera positions along the X-axis, or "Deep 3D Pan", with "t-shaped" adaptive kernels equipped with globally and locally adap- tive dilations. Our proposed network architecture, the monster-net, is devised with a novel t-shaped adaptive kernel with globally and locally adaptive dilation, which can efficiently incorporate global camera shift into and handle local 3D geometries of the target image's 's pixels for the synthesis of naturally looking 3D panned views when 2-D input image given. Extensive experiments were performed on the KITTI, CityScapes, and our VICLAB STEREO indoors dataset to prove the effi- cacy of our method. Our monster-net significantly outperforms the state-of-the-art method (SOTA) by a large margin in all metrics of RMSE, PSNR, and SSIM. Our proposed monster-net is capable of reconstructing more reliable image structures in synthesized images with coherent geometry. Moreover, the disparity informa- tion that can be extracted from the "t-shaped" kernel is much more reliable than that of the SOTA for the unsupervised monocular depth estimation task, confirm- ing the effectiveness our method.
INTRODUCTION
Recent advances deep learning have pushed forward the state-of-the-art performance for novel view synthesis problems. Novel view synthesis is the task of generating new view from different camera position, given a single or multiple input images, and finds many applications in robotics, navigation, virtual and augmented reality (VR/AR), cinematography, etc. In particular, the challenging task of generating stereo images given a single input view is of great interest as enables 3D visualization of the 2D input scene. In addition, the falling price and the increasing availability of the equipment required for VR/AR has fueled the demand for stereoscopic contents.
The previous works, such as Deep3D (Xie al. 2016), have addressed the right-view generation problem in fully supervised fashion when the iput is the left-view to which the output is the synthetic right-view at fixed camera shift. In contrast, our proposed Deep 3D Pan pipeline enables the generation of new views at arbitrary camera positions along the horizontal X-axis of an input image with far better quality by incorporating adaptive "t-shaped" convolutions with globally and locally adaptive dilations. Our proposed "t-shaped" kernel with adaptive dilations takes into account the camera shift amount and the local 3D geometries of the target pixels. Panning at arbitrary camera positions allows our proposed model to adjust the baseline (distance between cameras) for different levels of 3D sensation. Additionally, arbitrary panning unlocks the possibility to adjust for different inter- pupillary distances of various persons. Figure] Thoows shows some generated left and right
"htps://www. VICLAB. W.CC.B..aist.ac..k

---

Published as a conference paper at ICLR 2020
Leftward
Center
Rightward
Figure Generated left and right images by our proposed Deep 3D Pan for an input center image.
view images for a given single image input by our proposed Deep 3D Pan pipeline, which we call "monster-net" (monocular to stereo network). In this paper, define "panning" in the context of 3D modeling, implying that camera movement is in parallel to the center view camera plane.
In the following sections, we review the related works to stereoscopic view synthesis and discuss the differences with our proposed method, followed by the formulation of our Deep 3d Pan pipeline and finally, we present outstanding results on various challenging stereo datasets, showing superior performance against the previous state-of-the-ee methods.
2 RELATED WORK
Novel view synthesis is a well-studied problem in deep learning-based computer vision, and has already surpassed the classical techniques for both cases of the multiple-image (Woodford et al.)
2007 Liu et al. 2009] Chaurasia et al. 2013) and single-image input (Horry et al. 1997] Hoiem al, 2005). The latter, single-image based novel view synthesis, is known to be much more complex problem compared to multiple-image based ones. Previous deep learning- based approaches usually tend utilize one of the two techniques to generate a novel view: (i) optical flow guided image warping, and (ii) a "flavor" of kernel estimation, also known as adaptive convolutions.
The first technique, optical flow guided image warping, has been adopted by several authors to train convolutional neural networks (CNNs) for optical flow or disparity estimation from single or stereo images in an unsupervised fashion. However, their final goal was not to synthesize novel views. These works include those of (Godard et al.
2017 Zhou et al.) 2016 Gonzalez & Kim 2019b Tosi et al.] [0119] Liu et al.] 2019 Wang et al. 2019b; Ranjan et al. 2019 Lai
Lai et al. 2019) Not all previous existing works have used flow-guided warping for unsupervised training to reg- ularize supervised methods for optical flow estimation. The work of Im et al. (2019) implemented plane sweep the feature level to generate a cost volume for multi-view stereo depth estimation. Such plane sweep can be seen as a type of ID convolution, similar to the ID kernel utilized in the second approach of kernel estimation for new view synthesis.
On the other hand, the second approach, kernel estimation or adaptive convolutions, has proved be superior image synthesis technique and has been incorporated in several different ways. For example: (1) Flynn et al. (201)). in their early DeepStereo, formulated a CNN capable of synthe- sizing a middle view by blending multiple plane-swept lateral view inputs weighted by a "selection volume", which can be interpreted as a 1D (or line-shaped) adaptive convolution; (2) in similar way, et al. (2016) devised the Deep3D, a non fully-convolutional network that estimates a series of "probabilistic disparity maps" that are then used to blend multiple shifted versions of the left-view input to generate a synthetic view image; (3) The adaptive separable convolutions (SepConv) in the work of Niklaus et al.) (2017) approximated adaptive 2D convolutions by two (vertical and horizontal) ID kernels that are applied sequentially to the input current and previous frames (fp and 41) for the video interpolation problem; (4) In the works of (Zhou al. 2018 Srinivasan et al.
2

---

Published as a conference paper at ICLR 2020
Constant
Locations vary according to affine
dilation
transforms shared for every outpul pixel
(a)
DeepStereo and Deep3D
(b) Geometric-aware
A Adaptive dilation
Nxi and
Receptive
according to the
1xN
field is
zoom factor
approximatt
000
limited to
a NxN
+ +
the kernel
+7 tH
kemel
D
size
(c) Deep3D Zoom Net
(d) Adaptive Separable Convolutions
Figure 2: Synthesis techniques based on adaptive convolutions. The background is the input image.
Red dots represent target pixel locations in output images. Green (along with red) dots represent sampling positions where the corresponding pixels are used to generate one target pixel.
2019). although with additional considerations, their multiplane image representation approach can be loosely understood as a ID adaptive convolution the final operation involves the reduction of a plane sweep volume; (5) Geometric-aware networks in the work of Liu et al.] (2018) indirectly achieved adaptive convolutions by learning fixed number of affine transformations on an input im- age, where the resulting transforrddf images are then blended together to generate one output image; and finally, (6) in the work of [Gozzaezz & Kim (2019)). the authors developed the Deep 3D Zoom Net, which estimates a selection volume for the "blending of multiple upscaled versions of the input image", which can be treated as an special case of ID adaptive convolution. The (Flynn et al., 6) and (Zhou et al. 2018) approaches require two more images inputs, thus, greatly reducing the complexity of the synthesis task as most ambiguities are removed by counting multiple views. In work, we focus the single-image based stereoscopic view synthesis task, which is a far more difficult problem as the network needs to understand the 3D geometry in the scene, and to handle complex occlusions, ambiguities and non- namberrmn surfaces.
Although the aforementioned methods are distinguished one another, as the different synthesis tech- niques have their own properties, they can be all interpreted belonging convolutions which are visualized in Figure As observed in (ag, DeepStereo (Flynn et al.] 2016) and Deep3D (Xie et al.) 2016) share the same shape of kernel, that is, a ID horizontal-only kernel that samples pixels at a fixed interval, or dilation, along the X-axis for each target output pixel. A ID horizontal-one only constantddilation kernel suffers from three major drawbacks:
category of adaptive
1. Inefficient usage of kernel values. When sampling the positions opposite the camera movement (which are the pixel locations corresponding to 01-03 in (a) assuming rightward camera shift), experiments showed that these kernel values would often be zeros. The same effect repeats when sampling the positions further away from the maximum disparity value of the given scene (which corresponds to the pixel location a7, assuming that the maximum disparity is 2 and the dilation is 1) the network is not able to find valid stereo correspondences for these kernel positions;
2. Right-view synthesis is limited to the trained baseline (distance between stereo cameras), the models over-fit specific training dataset with fixed baseline; and
3. The 1D line kernel has limited occlusion handling capabilities, as the network will try to fill in the gaps with the information contained only along the horizontal direction, limiting the reconstruction performance the models on the occluded areas.
In contrast, the kernels predicted by the gcometric-awane networks (Liu et al. al. 2018 have deformable structures adaptive to the given input images, as shown in Fig. (b). However, only one deformed kernel shape is predicted and shared to synthesize all target output pixels, leading to limited per- formance. Another drawback of the geometric-awee aware networks is their complexity. they require three sub-networks and a super- suxer-pixel segmentation step as pre-processing, hindering the processing of high-resolution images. For the Deep 3D Zoom Net (Gonzalez & Kim 2019a) case (Fig. (c).
3

---

Published as a conference paper at ICLR 2020
the ID kernel tends to point to the center of the image, as it performs blending operation of multiple upscaled versions of the input image. The dilation size of this ID kernel is adaptive according to the desired 3D-zoom factor. Finally, for the video interpolation case, the SepConv (Niklaus et al. (017) approximates an NxN adaptive kernel via a IxN and an Nx1 component (see Fig. (d) which are sequentially applied the input images generate the synthetic output. SepConv has, by design, limited receptive fields, as the dilation size is fixed to I. Besides, the sequential nature of the kernel forces the vertical component sample pixels from the output of the horizontal convolution, which could be already degraded due to heavy deformations introduced by the horizontal component.
Recent works have also attempted improve upon the stereoscopic view synthesis by improving the loss functions involved in the CNN's training. The work of et al. (2019) proposed a multi- scale adversarial correlation matching (MS-ACM) loss that learns to penalize structures and ignore noise and textures by maximizing and minimizing the correlation-ly distance in the discriminator's feature-space between the generated right-view and the target-view in an adversarial training setup. Whereas the objective function is key factor in training any CNN, we believe that, at its current state, the stereoscopic view synthesis problem can benefit more from a better pipeline that can handle the previously mentioned issues and using the widely accepted l and perceptual losses (Johnson iet al. 2016) for image reconstruction, rather than a more complex loss function.
Our proposed dilation adaptive "t-shaped" convolutions incorporate global (new camera position along the X-axis) and local (3D) geometries of specific target pixels) information of the input scene into the synthesis of each output pixel value, by not only learning the specific kernel that will gener- ate each output pixel, but also by learning the proper dilation value for each kernel. The "(" shape of the kernel allows the network to account for occlusions by filling-in the gaps (missing information in the output) due to shifted camera positions using not only left-and-right pixels (like DeepStereo and Deep3D), but also up-and-down neighboring pixel information. In addition, the notions of global and local dilations allow our proposed monocular to stereo network, the monster-net, to generate arbitrarily 3D panned versions of the input center view along the X-axis, useful feature not present in previous works that allows adjusting for eye-to-eye separation and/or level of 3D sensation.
3 METHOD
In order to effectively synthesize an arbitrary 3D panned image, we propose a global dilation filter as shown in Figure
3 Our proposed cross- pross-shaped global dilation filter Ta(p) a target pixel location p (x, v) E l.s where 12 is generated image, is defined
(p()) = TT v). [T.. To, T, |1]
where T2(x.y) y) is the filter parameter value of Ta(p) at the center location p. The upper, bottom, left and right wing parameters (T.,, To, T,, T, of the cross-shaped dilation (d) filter are defined as
T, [T,(,, y - d), (zr (x( ni, d))"
T [To(x, y + d) (2, 2d) To(2, + n1
(2)
T, = [T) d) Ti 2d, vi(r..,,Ti( - 1iii--nd,11 v)l"
T, [z + d, v), Ti 2d, y) ......,,TT... T,(x n.d, 2))²
where Nu. nlg ny and n, indicate the numbers of filter parameters in T To, T, and T,,, respectively. For the shased dilation filter shown in Figure 3 it is more appropriate to have a longer length of the right (left) filter wing than the other three wings when the camera panning is rightward (leftward), as it allows capturing more useful information for the synthesis of a right (left) panned image. In this case, n, (n)) set to greater than n; (n,), n,, and пp. such that the global dilation filter showed in Figure 3 can be claborated as a "t-shaped" kernel which can then take into account the camera panning direction for synthesis. Figure 4 shows examples of "t-shaped" kernels overlaid on top of input center image. As shown in Figure (a). the "t-shaped" kernel has longer left wing of filter parameters for the synthesis of a leftward camera panning while in Figure (b) it shows a longer right-wi wing of filter parameters for the synthesis of rightward camera panning.
Why "I" shape? Experiments with symmetric kernel shapes (e.g., " shape) were performed first, but it was noted that most of the elements on the left (right), upper and bottom sides against the
4

---

Published as a conference paper at ICLR 2020
Upper wing filter parameters Center filter parameter
Tx
T)
Tx(x,y-d)
Ti(x R2 d, T₁(x- 2xxx
T (x+ T, (x++nrd,y)
Tx(x-d,y)
T, (x+d,y)
Left wing filter parameters
(xx,y++d)
Bottom wing filter parameters yxxx,
Figure 3: Our proposed global dilation (d) filter with general cross shape.
Dilation
globally and
locally
adaptive
a
Leftward panning "t" kernel
(b) Rightward panning "t" kemel
2g /
8 /
(c) Large dilation (d, (d) Med dilation (d,-2g,//)) (c) Small dilation (d (d, (3)
Figure 4: Our proposed "t-shaped" kernels are overlaid on top of a center input image. The distance between samples (dilation) adaptive according to the amount and direction of 3D panning to be applied to the input image and the local 3D geometry of the scene.
centered red dot the kernel tended to have very small values close to zeros for most target pixels for the rightward (leftward) movement of the camera. Similar SepConv (Niklaus et al. 2017). the experiments with a horizontal kernel applied first followed by a vertical kernel were performed, yielding poor results. It discovered that the "I" shaped kernel is more efficient than the "+" shaped kernel as it picks up more effective sampling positions with a fewer parameters than the standard adaptive convolutions such those in SepConv. As depicted in Figure 5 the "t-shaped" kernels can embed useful information like disparity and occlusion from a monocular image into the stereo synthesis process.
The longer right (left) wing of the "t-shaped" kernel contains disparity information, as it will try to sample pixels from the right (left) side to the target pixel when the camera assumed move in the rightward (leftward) direction. Figure (a) depicts primitive disparity map Dp that was constructed by the weighted sum of the kernel values in the longer kernel wing, as described by
(p)
(x + id,
n2y
where [x + id, 3) the i-th value of the longer wing T, at pixel location = (a, 3) for the rightward 3D panning of input center image I.. Note that D, is normalized in the range [0, 1]. In- terestingly, as shown in 5 (a), the generated disparity map looks very natural and appropriate, which implies the effectiveness of our "t-shaped" kernel approach.
The short left (right), upper and bottom wings of the "t-shaped" kernel contain occlusion information, the network will try to fill in the gaps utilizing surrounding information that is not present in the long part of the "t-shaped" kernel. is also interesting to see the occlusion map in Figure (b)b) where primitive rightward occlusion map O was constructed by summing up the "t-shaped" kernel values in the short wing parts according to the following:
742
(p()
Σ Ti(x - id, v) + E Tu (z.y- - id) + E To y iid
(4)
The bright regions or spots in Figure (b) indicate the occlusions due to the camera shift along the horizontal axis of the input center image, which likely to happen for the case of the camera's
5

---

Published as a conference paper at ICLR 2020
(a) Input center image and disparity map (b) Output panned image and occlusion map
Figure 5: Disparity (Dy)) and occlusion (0r) maps generated from the proposed "t-shaped" kemel.
rightward panning. For both Equations (3) and 4 the primitive disparity and occlusion maps for the leftward panning case can be obtained by swapping the and indices.
3.1 GLOBBLLY AND LOCALLY ADAPTIVE DILATIONS FOR THE SYNTHESIS OF A NEW VIEW IMAGE AT A SHIFTED CAMERA POSITION
In general, the disparity amounts between stereo images are variable different pixel locations according to the distance between stereo cameras and the local scene geometries. Therefore, it is necessary take into account the variable disparity in synthesizing a new view in globally and locally adaptive fashion. For this, a "t-shaped" kernel is introduced with a controllable dilation fac- tor by which both camera shift and local changes in image geometry can be effectively taken into account when synthesizing new (left or right) view for the input center image. Any kernel with a fixed dilation may cause a limited accuracy in synthesizing novel view because the disparity amounts vary over the whole image according to the cameras' baseline and the local geometries. So, our "t-shaped" kernel is proposed make the synthesis of novel views not only globally, but lo- cally adaptive to the camera shift and its local changes in image geometry by controlling its dilation size per-pixel in the output image. Globally, short dilation value is more appropriate when slightly shifting the camera, while a high dilation value is desirable when largely shifting the camera posi- tion. In a local manner, a small dilation value is appropriate for far-away objects from the camera while very close objects the camera be better reconstructed with a larger dilation value.
3.1.1 GLOBAL DILATION
We define the global dilation Pa as the pixel distance between two consecutive kernel sampling positions, which is given by the pan amount Pa to be applicd to the input center image I. divided by the total number of filter parameters in the longer "t-shaped" kernel wing (nı n,,). P. has unit of pixels mapped in the image corresponding to the camera shift into the left or right direction and takes floating numbers. Therefore, the global dilation ga is given by
(5) where Pa takes on positive (negative) values for the rightward (leftward) panning scenario. The pan amount needed to generate left-view or right-view is determined during training according to the closest possible objects to the camera. The "closest possible objects" vary over different training datasets. For our novel view synthesis task, like in (Godard al. 2017 Gonzalez & Kim 2019b), we assume the KITTI dataset have a maximum or "closest possible object" disparity 153 pixels. During training, Pa is set to 153 and -153 for the rightward and leftward panning, respectively.
gd Pa/n, Pa >0 O, P./ma if
3.1.2 LOCAL DILATION
While global dilation allows the "t-shaped" kernel to take into account the global camera shift, a locally adaptive mechanism is needed synthesize new views of locally variable disparity. Such mechanism is realized by first generating multiple images with the "t-shaped" kernel at N different dilations and blending them per-pixel in locally adaptive manner. The blending is a weighted sum of filtered images by the "t-shaped" kernel with N different dilations, where the blending weights (W1, W22 wx) control the local dilation per-pixel and are learned via a convolutional neural network (CNN) along with the parameter values of the "t-shaped" kernel. Let Igul be the maximum dilation value that is a fractional number. Figures (c), -(d) and -(c) illustrative three "t-shaped" kernels with maximum dilation |aal and two dilation values less than (gal- To generate output

---

Published as a conference paper at ICLR 2020
Pa: Pan amount
c49 Horizontal weights TT. T.
c50 c81 Vertical weights [Tu: Tb]
Ig Center Image
IIo
Panned Image Pai pan amount
le
Center
image
c32
c64
cl28
conv
c256
ELU
c512
<512
c512
c512
c256
c128
c145
c3 Dilation weights [w]. WN)
Conv+ELU
Stride ELU
Residual
Nearest 2x upscale
Coav
ELU
Concat skip
connection
Conv sofmax
Figure 6: Our t-net architecture. The t-net estimates the kernel values and the dilation weights used for the local adaptive convolutions with global and local adaptive dilation.
image I' panned to the rightward direction (gs 0) or to the leftward direction (ga < 0), the input center image Ig is first filtered by N "t-shaped" kernels Ta, of different dilations (di...., ,.x...,dd..... Then, local adaptive dilations are calculated by linearly combining the resulting N intermediate filtered images according to the corresponding blending weights (W1, 422 WN)) Based on the N different global dilations, the output image value (p) at pixel location p can be calculated
N
(p)
Σ
wx(p) [1. Ta, (p)
(6)
where [le (p) is "t-shaped" convolution location p between le and Ta, of dilation d; (1 + (1 g) for i = .....,.. w; (p) indicates a blending weight for the i-th global dilation.
3.2 NETWORK ARCHITECTURE
We propose an end-to-end trainable CNN, called the "monster-net" (monocular stereo net). The monster-net is made of two main building blocks, novel view synthesis network, the "t-net", and resolution restoration block, the "sr-block". Given input center image I, and pan amount Pa. the final output panned image Io is obtained by sequentially applying the aforementioned modules by
Io monster-net(le: Pa) gr-block(t-net((( Pa: 0o), (..))
(7)
where 6, and 0.. parameterize the t-net and the sr- blobkock respectively. (1e,) is the stack of progres- sively shifted-dowen -owfeeddowwscaaeed versions the input center image I, described the SR-BLOCK section.
The t-net. The "t-net" estimates both the "t-shaped" global dilation kernel parameters (T.) and the adaptive local dilation weights (w,, - wx The t-net is designed have large receptive fields to synthesize detailed image structures of a new view image, which corresponds to a shifted camera position. Such large receptive fields are useful in capturing the global image structure and contex- tual information needed for new view image to be synthesized. For this, nuto-encode with skip connections (not a U-net structure) is adopted. which allows the t-net to have effectively large re- ceptive fields and to efficiently fuse global and local (fine details) information For better feature extraction, we adopt the residual connections in the encoder side, as proposed by (Gonzalez & Kim 2019h). The t-net estimates all necessary values perform the operation de- scribed by Equation (6). The t-net, depicted in Figure 6) has two output branches: the first output branch yields 81 channels, where the first 49 are horizontal parameter maps, and the following 32 vertical parameter maps; the second output branch generates the 3-channel blending weight maps for the local adaptive dilation. That is, each channel-wise vector pixel location for the first output branch corresponds to the t-kernel parameter values [T., T/" T.r TT T, and each channel-wise vector for the second output branch corresponds to the blending weights [w1, W2..... WN), utilized for local dilations in Equation (6). As our t-net is devised to generate arbitrarily panned novel views, feeding the pan amount take into account the varying pan direction, and the amount of occlusion on the 3D panned output. The effect of feeding the pan amount is further discussed in appendix A-1.
the decoder stage.
I-channel constant feature map (P.(p) P.Y p) helps the network
Super resolution (SR) block. As generating a full resolution dilation dilation-adaptive t-kernel would be computationally too expensive, we propose estimate it at a low resolution (LR) for the synthesis

---

Published as a conference paper at ICLR 2020
k5c32
Conv+ ReUU
Nearest 2x
n=
upscale Conv
(T cst +
lo: Final
lc
1--2
Concat
Add
panned
c64
c64
c32
k5c3
image
(a) (Left) Target resolution center view. (Right) half- resolution shifted (to the left) versions of the center view
c64
c32
(b) Our sr-block
Figure 7: (a) Shifted-LR versions of the center-vien contain different information as they are sam- pled from different groups of pixels via bilinear interpolation depending the stride (controlled by the maximum disparity). (b) Our light sr-block. All convs have 3x3 kernels otherwise specified.
Table 1: Stereoscopic view synthesis performance on the 400 KITTI2015 training images (left) and
the 500 CityScapes validation images (right). 4. perceptual loss. 14 indicate the better performance.
Model
training dataset
loss
RMSEJ
PSNR1
SSIM
RMSET
PSNRT
SSIM1
Deep3D
4
26.13
20.07
0.637
30.10
18.82
0.655
Deep3D-B
K
1
+ 4p
26.00
20.10
0.633
31.34
18.46
0.636
SepConv
K
4
27.22
19.73
0.633
27.77
19.54
0.660
SepConv-D
K
4 + lp
26.36
20.02
0.626
29.66
18.95
0.647
monstet-net
1
25.61
20.24
0.641
20.28
22.34
0.710
monster-net
K+CS
4
24.11
20.76
0.667
12.87
26.36
0,816
monster-net (full)
K+CS
11 1
24.44
20.64
0.651
13.12
26.20
0.805
monster-net
K+CS+VL
4 +
24.62
20.55
0.645
of half-resolution novel view, and then to apply deep learning based SR techniques to bring the LR novel view to the high (or original) resolution (HR). In comparison. in Deep3D and SepConv, the estimated LR kernel is upscaled with conventional methods the HR and then applied to the input image(s), which a costly operation as it is carried out in the HR dimensions and can lead to blurred areas as the kernel is just bilinearly interpolated. In our proposed pipeline, instead of utilizing common single image SR methods like (Dong et al. 2015 Shi et al.. 2016 Kim et al. 2016), we propose to apply a stereo-SR method. The stereo-SR technique in (Jeon et al., 2018) takes LR stereo pair (left and right views) as input and progressively shifts the right- producing a stack that is concatenated with the left-view and later processed by CNN to obtain the super-resolved leet-view. This process is made at an arbitrary and fixed stride (e.g. 1 pixel at every step of the stack) and does not take into account the maximum disparity between the input views. For our Deep 3D> Pan pipeline, wing of the t-kernel to dynamically set the shifting stride. Additionally, instead of interpolating and processing the low resolution panned view (p) the HR dimensions, we progressively shift and then downscale the high-resolution center view I by a factor of x2. This allows our sr-block operate on the LR dimensions without performance degradation, in the horizontal axis is not lost but distributed along the levels of the shifted center view stack as depicted in Figure 7 (a). Our block,ck, depicted Figure (b). simple, yet effective module that takes input the LR I view and the shifted-downsealedd center view stack 1a described by
propose to the maximum disparity prior that can be obtained from the long
high frequency information
nP. N₂
F gle,
mar (D,))
(8)
where g(1, s) is s-strided horizontal-shift and 2x down-scaling operator applied image I. The stride a can take any real number and the resulting image obtained via bilinear interpolation. N. is the depth of the stack, and was set to N,- 32 for all our experiments). The stack is concatenated with the LR I and passed trough four Conv-ReLU layers followed by residual connection as shown in Figure (b). The final step up-scales the resulting features into the target resolution via nearest interpolation followed by a convolutional layer. The last layer reduces the number of channels to three for the final RGB output I,- Nearest upscaling was adopted as it yields no checkerboard artifacts in contrast with transposed sub-pixel convolutions (Niklaus al. 2017).
8

---

Published as a conference paper at ICLR 2020
D
DB
SepConv
SepCone-D
Monster-net (K)
Monster-net (K+CS)
GT
Figure 8: Comparison against the state-of-the-an methods for stereoscopic view synthesis.
EXPERIMENTS AND RESULTS
To demonstrate the effectiveness of our "I-shaped"-dilaaee -adaitiv- kernel, we performed several experiments on the challenging KITTI2012 (Geiger et al. 2012), KITTI2015 (Menze & Geiger 2015), and CityScapes (Cordts et al. 2016) datasets. As these stereo datasets only consist of out- door scenes, also performed experiments dataset. Surprisingly, to our best knowledge, this is the first stereo dataset available that focuses on the indoor scene, which is planned to be publicly available for research. Additionally, our formu- lation of global and local adaptive dilations allows our monster-net to be trained on multiple stereo datasets at the same time, even if these have different baselines. Instead of wer-fitting on a single camera baseline like the previous methods (Xie et al. (2016); [odard et al. (2017); Zhang et al. (2019; Luo al. (2018)). our monster-net can build knowledge when simultaneously trained on many datasets. To our best knowledge, our Deep 3D Pan pipeline is the first method designed to be trained on multiple baseline datasets concurrently for the stereoscopic view synthesis problem where unsupervised monocular depth estimation is even used particularly. For more details about the datasets and multi- dataset training, please see the appendix A-3.
indoors dataset, called the VICLAB TICLAB_STEREO STEREO
We compare monster-net against the stereoscopic view synthesis SOTA: Deep3D Xie et al. (2016) and a version of SepConv (Niklaus et al. 2017) modified for right-view synthesis. Firstly, for fair comparison, the backbone convolutional auto-encoders for the Deep3l) and SepConv were set up to be equivalent to our t-net's, that is, a six-stage encoder-decoder with skip connections and residual blocks in the encoder side. Secondly, we compare our monster-net with Deep3D-B, a "Bigger" version of Deep3D, where, instead of 32 elements in the ID kernel as in its original 49 elements to match the number of horizontal kernel values our t-net. Thirdly, we compare against SepConv-D, dilated version of the SepConv such that the receptive field of the separable convolutions has size of 153x153. The Deep3D and the SepConv models are trained without using perceptual loss as in their original works. For more meaningful comparison, the Deep3D-B and the SepConv-D are trained with a combination of I and perceptual loss 4, [Johnson) al. 2016), and demonstrate that a better loss function than l does not contribute enough to the stereoscopic view synthesis problem. For more implementation details, reefer to the appendix A-4.
work,
Additionally, we compare the quality of the embedded disparity in the long wing of the "t-shaped" kernel with those of the state-of-thh-ane models for the monocular depth estimation task. For that, first define disparity refinement sub-network. that uses the primitive disparity obtained from the long wing of the "t-shaped" kernel as prior information. Secondly, we define a special post- processing (spp) step, which, instead of relying naive element wise summation in Godard

---

Published as a conference paper at ICLR 2020
Table 2: Depth metrics (Figen al., 2014) for KITT12015. Models are trained with video (V),
stereo (S), semi global matching (SMG) or GT depth (Supp). Top models in terms of a accuracy are highlighted. Simplified table, see appendix A.9 for the full version.
Model
Supp V S dataset abs rel., sq rell. rms,,
log rms..
a'
a²
a²
Wang et al. (2019a) (9-view)
X
K
0.112
0.418
2.320
0.153
0.882
0.974 0.992
Tost ct al. ( (2119) (pp)
SMG
X K+CS
0.096
0.673
4.351
0.184
0.890
0.961 0,981
ours with refine block (spp)
K+CS
0,099
0,950
4.739
0.160
0.900
0,971 0,989
Gur & Woli 2019)
K
0.110
0.666
4.186
0.168
0.880
0,966 0,988
Luo et al. (2018)
X
0.094
0.626
4.252
0.177
0.891
0.965 0.984
Wang el al. (2019a) (1/9-view)
X
X
K
0.088
0.245
1.949
0.127
0.915
0.984 0.996
et al. (2017), takes into account the ambiguities of the first and second forward passes to generate a remarkable sharp and consistent disparity map. For more details on the refinement block and our special post-processing, reefer the appendix A-2.
4.1 RESULTS THE KITTI, CITYSCAPES AND THE VICLAB DATASETS
Table shows the performance comparison for our method and previous works. It is important to mention that our monster-net performs inference on full resolution images, while the previous ap- proaches for single-view novel view synthesis perform estimation on reduced resolution inputs. Our method outperforms the Deep3D baseline by a considerable margin O. PSNR, 2.0 in RMSE, and 0.03 in SSIM. The gualitative results are shown in Figure Our method produces superior look- ing images. In Deep3D and SepConv, many objects appear too blurred such that their boundaries can hardly be recognized in the synthetic images (c.g the motorcycle, persons, traffic signs, etc.). We challenged the models trained on KITTI (K) to perform inference on the CityScapes validation split (CS), and observed that method generalizes much better than the Deep3D baseline with up to 3dB higher in PSNR. When training the monster-net with K+CS, we get an additional improvement of 4dB PSNR in the validation CS dataset. Incorporating an indoor dataset to our training pipeline is also possible, making our network applicable to a wide variety of scenarios. We added the VI- CLAB_STERES (VL) dataset to the training, that is K+CS+VL, and observed little impact on the K dataset performance as shown in [ We also tested the performance of our monster-net on the validation split of the VI. dataset. We observed that our full monster-net trained on K+CS achieved mean PSNR of 19.92dB, while achieving mean PSNR of 21.78 dB when trained on K+CS+VI. For a network trained on the outdoors dataset only it is difficult to generalize to the indoors case, as the latter contains mainly homogeneous areas, whereas the outdoors case mainly contains texture rich scenes. Visualizations on CS and VL. and ablation studies that prove the efficacy of each of our design choices can be found in the appendices A-5, A-6 and A-8.
4.2 RESULTS ON DISPARITY ESTIMATION
With the addition of a relatively shallow disparity refinement sub-network, the monster-net remark- ably outperforms all the state-of-the-ar models for the unsupervised monocular depth estimation task, as shown in Table Our monster-net with disparity refinement even outperforms the super- vised monocular disparity estimation methods such as (Luo et al., 2018 Gur Wolf 2019) and multiple view unsupervised methods such as (Wang et 2019a] Ranjan et al. 2019).
5 CONCLUSION
We presented an adaptive "t-shaped" kernel equipped with globally and locally adaptive dilations for the Deep 3D Pan problem, defined as the task of arbitrarily shifting the camera position along the X-axis for stereoscopic view synthesis. Our proposed monster-net showed superior performance to the SOTA for right-view generation on the KITTI and the SctySs datasets. Our monster- net also showed very good generalization capabilities with 3dB gain in PSNR against the Deep3D baseline. In addition, our method presents no-discontinuities, consistent geometries, good contrast, and naturally looking left or right synthetic panned images. Our monster-net can be extended for image registration, monocular video to stereo video, and generation of novel views at any camera translation by just allowing wixel-wise rotation of our "t-shaped" kernel.
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea govermment (MSIT) (No. 2017-0-00419, Intelligent High Realistic Visual Processing for Smart Broadcasting Media).
REFERENCES
Yochai Blau and Tomer Michaeli. The perception serception-distotionn tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 6228-6237, 2018.
Gaurav Chaurasia, Sylvain Duchêne, Olga Horkine- and George Drettakis. Depth synthesis and local warps for plausible image- navigation. ACM Transactions on Graphics, 32, 2013. URL htttp://wet sop httea///e//eeeeaaaaa/////1/3331 at SIGGRAPH 2013.
to be presented
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2): 295-307, 2015.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using multi-scale deep network. In Advances neural information processing systems, pp. 2366-2374, 2014.
Jose M. Facil, Benjamin Ummenhofer, Huizhong Zhou, Luis Montesano, Thomas Brox, and Javier Civera. Cam-convs: Camera-aware multi-scale convolutions for single-view depth. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2019.
John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. Deepstereo: Learning predict new views from the world's ' imagery. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
Clement Godard, Oisin Mac Aodha, and Gabriel J. Brostow. Unsupervised monocular depth es- timation with left-right consistency. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
J.L. Gonzalez and M. Kim. Deep 3D-Zoom Net: Unsupervised Learning of Photo-Realistic 3D- Sep 2019a.
Zoom. arXiv e-prints, art.
J.L. Gonzalez and M. Kim. A novel monocular disparity estimation network with domain trans- formation and ambiguity learning. In 20/9 IEEE International Conference on Image Processing (ICIP), pp. 474 478, Sep. 2019b. doi: 111110///11222198803333
Shir Gur and Lior Wolf. Single image depth estimation trained via depth from defocus cues. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Derek Hoiem, Alexei A. Efros, and Martial Hebert. Automatic photo pop-up. In ACM SIGGRAPH 2005 Papers, SIGGRAPH '05, pp. 577-584, New York, NY, USA, 2005. ACM. doi: 10.1 1073222 1186822.1073232. URL http: doi acm. org/10
Youichi Horry, Ken Anjyo, and Kiyoshi Arai. Tour into the picture: Using spidery mesh interface to make animation from single image". pp. 225-232, 01 1997. doi: 10. 10.1145/258734255888

---

Published as a conference paper at ICLR 2020
Sunghoo Im, Hae-Gon Jeon, Stephen Lin, and In So Kweon. DPSNet: End-to-end deep plane sweep stereo. In International Conference on Learning Representations, 2019. URL https: /oopeneeieww..o http://www.....ommmmaa.aereeeeaa
Daniel S. Jeon, Seung- Hwan Baek, Inchang Choi, and Min H. Kim. Enhancing the spatial resolution of stereo images using a parallax prior. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super- super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646-1654, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.69800, 2014. URL http: wttpaaaw.....aaaaaaaaaa wttp//ggg/b///..../////// 6980
Hsueh- Ying Lai, Yi-Hsuan Tsai, and Wei-Chen Chiu. Bridging stereo matching and optical flow via spatiotemporal correspondence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1890-1899, 2019.
Feng Liu, Michael Gleicher, Hailin Jin, and Aseem Agarwala. Content-preserving warps for 3d video stabilization. In ACM SIGGRAPH 2009 Papers, SIGGRAPH '09, pp. 44:1-44:9, New York, NY, USA. 2009. ACM. ISBN 978-1-60558-726-44 doi: 0..144451576244.551133 URL
http:://do: acm. org www./og/1.. /10
1531350)
Miaomiao Liu, Xuming He, and Mathieu Salzmann. Geometry-aware deep network for single- image novel view synthesis. In The IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), June 2018.
Pengpeng Liu, Michael Lyu, Irwin King, and Jia Xu. Selflow: Self-supervised learning of optical flow. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Yue Luo, Jimmy Ren, Mude Lin, Jiahao Pang, Wenxiu Sun, Hongsheng Li, and Liang Lin. Single view stereo matching. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018
Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable convo- lution. The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Andrea Pilzer, Stephane Lathuiliere, Nicu Sebe, and Elisa Ricci. Refine and distill: Exploiting sycle-inconsistency and knowledge distillation for unsupervised monocular depth estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J. Black. Competitive collaboration: Joint unsupervised learning of depth, camera mo- tion, optical flow and motion segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874-1883. 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.155 2014.
Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc Huszár. Amortised MAP inference for image super-resolution. In 5th International Conference Learning Repre- sentations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings, 2017. URL htttsp:://ppenee htt.hhttps///ppeneeee
12

---

Published as a conference paper at ICLR 2020
Pratul P. Srinivasan, Richard Tucker, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. Pushing the boundaries of view extrapolation with multiplane images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Fabio Tosi, Filippo Aleotti, Matteo Poggi, and Stefano Mattoccia. Learning monocular depth esti- mation infusing traditional stereo knowledge. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Rui Wang, Stephen M. Pizer, and Jan- lan-Michael Frahm. Recurrent neural network for (un-)supervised learning of monocular video visual odometry and depth. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019a.
Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang, and Wei Xu. Unos: Unified unsu- pervised pllw and stereo-depth estimation by watching videos. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019b.
O. J. Woodford, I. D. Reid, P. H. S. Torr, and A. W. Fitzgibbon. On new view synthesis using multiview stereo. In In Proc. BMVC, pp. 1120-1129, 2007.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Deep3d: Fully automatic 2d-to-3d video conversion with deep convolutional neural networks. In European Conference on Computer Vision, pp. 842- 857. Springer, 2016.
Yu Zhang, Dongqing Zou, Jimmy S. Ren, Zhe Jiang, and Xiaohao Chen. Strcture-preservingg stereoscopic view symthesis with multi-scale adversarial correlation matching. In The IEEE Con- ference Computer Vision and Pattern Recognition (CVPR), June 2019.
Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In European conference on computer vision, pp. 286-301. Springer, 2016.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. SIGGRAPH, 2018
A APPENDIX
A.1 EFFECT OF FEEDING THE PAN AMOUNT TO THE T-NET
The larger the pan amount, the greater the occlusions to be handled in the synthetic output image. The effect of feeding the pan amount Pa as a one-channel constant feature to the t-net can be visu- alized in Figure 9 where multiple primitive disparity and occlusion maps are depicted for different pan amounts. As shown in Figure 9 the network generates different maps for different magnitudes and directions of Pa while keeping the input center image I. unchanged, confirming the effect of the pan amount as prior knowledge to the network. The difference between the disparity maps can be appreciated in the "holes" or "shadows" casted in the objects borders, as they represent the occluded content seen from the output 3D panned image. In the red box Figure 9 it is observed that the shadows casted by leftward and rightward camera panning appear in opposite sides ob the objects. In the ycllow box, it is observed that the larger the Pa. the larger the shadows projected, as more occlusions are to be handled.
A.2 DISPARITY REFINEMENT SUB-NETWORK
The disparity refinement network architecture, as depicted in Figure 10 has two input branches: one takes a stack of the bilinearly upscaled center image disparity prior (Dep) and the RGB center image (l.); and the other is fed with stack of the 2x bilinearly upscaled output panned view disparity prior (Do)) and the generated panned view (l.). The disparity refinement block is relatively shallow auto-encoder with skip connections and rectangular convolutions as fusion layers in the decoder stage. This allows to increase the receptive field size in the horizontal axis, thus improving the stereo matching performance, suggested by (Gonzalez & Kim, 2019b). We configure the output layer of our refinement network with the last layer of Gonzalez & Kim (2019b)'s 's architecture, which allows to do ambiguity learning in our refinement block. Ambiguity learning allows the network to
13

---

Published as a conference paper at ICLR 2020
unsupervisedly account for occlusions and complex clutered regions that difficult to minimize in the photometric reconstruction loss (Gonzalez & Kim, 2019b). We train the refinement block with the loss functions defined in (Gonzalez & Kiml (019bb) and new additional loss towards producing the refined disparity maps similar to the primitive disparity maps Dep and Dop- The refinement network is encouraged to produce refined center and panned disparity maps D. and D. similar to their primitive counterparts of half the resolution (as they are estimated from the t-net), by minimizing the following primitive disparity loss
1pp = DDop D
1 + | - D
where D and D are the bilinearly downscaled and refined center and panned disparity maps by a factor of 1/2, respectively. We give a weight of 0.5 this new loss term. The disparity refinement block can be trained end-to-end along with the monster-net or from pre-trained monster-net.
1. SPECIAL POST-PROCESSING
Instead of relying on naive post-processing approaches like in (Godard et al. 2017). which consist on running the disparity estimation twice with normal and horizontally flipped inputs and then tak- ing the average depth, we define novel special post-processing step (spp) by taking into account the ambiguities in the first and second forward pass. We noted that the ambiguity learning from (Gon-) zalez& & Kiml 2019b), which we incorporate in our disparity refinement block, can be used to blend the resulting disparities from the first and the second forward pass such that only the best disparity estimation (or ambiguity free) from each forward pass is kept on the final post-pon processed disparity. Figure depicts our novel post-processing step, which consist on running the forward pass of our monster-net with disparity refinement block with P. 153 and Pa -153, for the first and the second pass respectively. Then, the generated ambiguity masks of each forward pass are concate- nated create two-channel tensor and passed through softmax operation along the channel axis. The resulting soft-maxed ambiguities are used to linearly combine the disparity maps of each for- ward pass. As can be observed in Figure IT] the soft-maxed ambiguity mask effectively segment the best disparity estimation from each forward pass. Figure shows the primitive disparity map, the subsequent refinement step, the naive post-processing (pp) and our novel post-processing (spp).
A.3 THE KITTI, CITYSCAPES AND VICLAB STEREO DATASETS
The KITTI dataset is a very large collection of mid-resolution 370x 1226 stereo images taken from a driving perspective. We used the KITTI split suggested in (Godard et al. 2017), which consists of 29,000 stereo pairs from 33 different scenes of the KITTI2012 dataset. We set apart the KITTI2015 dataset for validation as it contains 400 images excluded from the KITTI split. Additionally, the KITTI2015 contains sparse disparity ground truths (GTss which are obtained from LIDAR and then refined by car CAD models. We use these GTs to evaluate the quality of the estimated disparity that can be extracted from the long wing of the t-kernel. CityScapes is a higher resolution stereo dataset with 24,500 stereo pairs that we extract from the train, test and train.crtra directories for training. The val directory is left for validation with 500 stereo pairs. We pre-process the CityScapes dataset for faster and more robust training. We first remove the top 25, bottom 200
-153
P,, 76.5
P = 153
Figure 9: Effect of feeding different values of Pa while keeping the input image unchanged. Differ- ent values of P. generate different occlusions and different holes in disparities (see red and yellow boxes), which indicate the occluded regions in the target panned image.
14

---

Published as a conference paper at ICLR 2020
Shared weights
el Center (c) disparity
map De
Add
S
el Panned dispurity map Do
Concal
+
Add
1g Dep
c256
c128
c128
c2 Center and panned view ambiguity masks
c64
k3x5
Concul
c32
c32
1,. Dop
Conv sigmoid
2x Pixel shuffle
Figure 10: Disparity refinement block
Refined disparity and umbiguity mask first run P, 153
Element-wise multiplication
Add
Second rua P, -153
Specially Post Processed (spp) refined disparity map
chaunel wise softmax
Element-wise multiplication
Figure 11: Our novel special post-processing step (spp)
(a) Primitive disparity map D,
(b) Refinement block
(c) Naive post processing (pp)
(d) Our post processing (spp)
Figure 12: Primitive disparity and different refinement options.
15

---

Published as a conference paper at ICLR 2020
and left 100 pixels to avoid car hoods and rectification artifacts, thus yielding final image size of 799x1948. Secondly, we save the cropped images in .jpg format to accelerate loading times during training. The VICLAB STEREO is an indoor left and right view dataset that was captured in 18 different buildings on a static pedestal using a Stereolabs's 's TttreM camera. The captured 2,051 high-resolution stereo pairs were rectified via checkerboard calibration as described in this MATLAB websitq] After rectification, the images in the VICLAB A.STEEEOO yield a final resolution of 1247x2454, We use 2,035 stereo pairs spanning 17 buildings for training, and the remaining 16 images for validation.
A.3.1 TRAINING ON MULTIPLE DATASETS
To our best knowledge, our Deep 3D Pan pipeline is the first method designed to be trained on multiple baseline datasets at the same time for the stereoscopic view synthesis problem and the unsupervised monocular depth estimation task. should be noted that the work of Facii et al. 2019) only handled the supervised monocular depth estimation task for multiple datasets with dif- ferent camera intrinsics utilizing "CAM-Convs", which is simpler problem than our unsupervised problem. While they require to know the intrinsic matrix for each dataset, along with added com- putational complexity to perform the "CAM-Coov's our method only requires to know the dataset baseline (distance between cameras). To train multiple datasets, the only required step is to mul- tiply the Pa by the relative baseline with respect to a reference dataset. For instance, the baseline in the KITTI dataset is about 54cm, and, mentioned before, we have set this baseline to correspond to PK = 153. Then for the CityScapes dataset, whose baseline is 22cm, its pan amount will be given by Pcs = (22/54)
For the VICLAB STEREO dataset with the baseline of 12em long, its pan amount becomes Aa pVy = (12/54) When training on KITTI + CityScapes (K+CS), batch of size 8 contains 4 images from each dataset. When training on KITTI CityScapes VICLAB (K+CS+VL), each batch of size 8 contains 3, 3 and 2 images, respectively. For the comparison against the recent works, we train our model and the state-of-the-ar models with KITTI (K) and KITTI CITYSCAPES (K+CS) only, and evaluate the resulting trained models on the 400 images from the KITTI2015.
A.4 IMPLEMENTATION DETAILS
For the training of all models, we used the Adam optimizer (Kingma & Ba 2014) with the recom- mended 3's (0.5 and 0.999) for the regression task with a batch size of for 50 epochs. The initial learning rate was set to 0.0001 and was halved at epochs 30 and 40. The following data augmen- tations on-the-fly were performed: Random resize with a factor between 0.5 and 1 conditioned by the subsequent 256x512 12 random crop; random horizontal flip, random gamma, and random color and RGB brightness. It was observed that vertical flip has made the learning more difficult, thus it was avoided. When training our model (the monster-net), the training images were sampled with 50% chance for rightward (Pa >> 0) or leftward (P. < 0) panning. Additionally, random resolu- tion degradation was applied to the input only by down-scaling followed by up-scaling back to its original size using bicubic kernel with scaling factor between and 1/3 while keeping the target view unchanged. Random resolution degradation has improved our results by making the network more sensitive to edges and forcing it to focus more on structures and less on textures. Similar "tricks" have been used in previous works in the form of adding noise to the input of discriminators (Sonderby et al.] 2017 Zhang et al. 2019) to make them invariant to noise, and more sensible to structures. When training our monsterr-et, either the left or the right view can be used as the input during training. When the left-view is used as the center input view Ie, the pan amount P. is set to 153 and the ground truth Zge is set to be the right image. In the opposite case, when the center view is the right image, Pa is set -153 and the left-view is set the GT. For our monster-net, the "t-shaped" kernel was set to have short wings of 16 elements and a long wing of 32 elements plus one center element T.. For the Deep3D, the ID kernel is set have 33 elements and 49 elements for the Decp3D-B variant. For the SepCony and the SepConv-D cases, we set the horizontal and vertical ID kernels to have 1x51 and 51x1 shape, respectively, as in (Niklaus et al. 2017
16

---

Published as a conference paper at ICLR 2020
Deep3D (K)
monster-net (K)
monster-net (4, K-CS)
monster-net Ui+ fo K+CS)
Figure 13: Results on the CityScapes dataset. Our method trained on KITTI-only (K), generalizes very well on the unseen images with an improvement over 3dB against the Deep3D baseline.
A.4.1 LOSS FUNCTION
We train monster-net with combination of l loss and perceptual loss (Johnson et al.) 2016 The later measures the distance between the generated view (l. I2) and the ground truth (ggt) images in the deep feature space of a pre-trained network for image classification. The perceptual loss is especially good to penalize deformations, textures and lack of sharpness. The mean square error of the output of the first three max-pooling layers from the pre-trained VGG19 (Simonyan & Zisserman, 2014), denoted by ø', was utilized the perceptual loss function. To balance the contributions of the ( and perceptual losses, a constant ap 0.01 was introduced. This combination of loss terms was applied to both the low-resolution panned image I and super-resolved panned image I, yield the total loss function Imaa as follows:
3
Lpan = I|let lolli + 1/2 1111 + aap
11 (1gx) o' (1(11)112 + 11 110 Φ (1)1 (10)
where 1/2 gt is the bilinearly downscaled version of the ground truth by factor of 1/2.
A.5 RESULTS ON THE CITYSCAPES DATASET
Visualizations on the CittyScapes (CS) datasets for our monster-net trained on KITTI (K) and on KITT + CityScapes (K+CS) are depicted in Figure 13 The First row of Figure 13 shows the synthe- sized images from the Deep3D baseline, it can be noted that over-fits to the training baseline of the KITTI dataset, performing very poorly on CityScupes. The subsequent rows show the results for our monster-net when trained without and with the CityScapes dataset. Our models generate very good structures and sharp panned views as depicted the red highlighted regions on Figure [3 for both cases of with and without CityScapes training. When trained on KITTI-only, our method general- izes very well on the CityScapes dataset, with performance improvement 3dB over the Deep3D baseline as shown in Table When trained on K+CS, we obtain additional improvement of 4dB against the KITTY-only traiedd monster-net. Additionally, we present results for our monster-net trained with and without perceptual loss, Sharper results with clear edges and structures are achieved when utilizing the perceptual loss, as depicted in the highlighted areas in Figure 13
(L₃) and (11 + Ip) respectively, on the CityScapes dataset.
17

---

Published as a conference paper at ICLR 2020
GT (mean PSNR / SSIM)
monster-net K+CS+VL (21.78dB 0.718)
monster-net K+CS (19.92dB 0.635)
Figure 14: Results the VICLAB STEREO (VL) dataset. The monster-net trained the
K+CS+VL datasets achieves better structures in homogeneous areas (highlighted in red).
A.6 RESULT ON THE VICLAB STEREO INDOORS DATASET
A network that is trained on outdoor datasets only is not able to generalize well on highly homoge- neous areas that are common in the indoors datasets but rare in the outdoor scenes. Visualization of the synthetic views generated for the VICLAB STEREO (VL) indoors dataset is provided in Figure We compare the results our network trained on K+CS versus those our monster-net trained on K+CS+VL. The latter achieves better generalization, as depicted in Figure [4] with marginal performance decrease in the KITTI dataset (-0.09dB) and considerable quality improvement on the VICLAB STEREO dataset (+ 1.86dB), showed in the last row of Table
14
A.7 3D PANNING BEYOΝD BASELINE
As our method allows for arbitrary camera panning, it is possible perform 3D pan beyond the baseline as depicted in Figure [5 where the pan amount was set to go 30% beyond the baseline for both leftward and rightward camera panning, that P. 200 and P. 200 for the per-scene top and bottom samples respectively, where the input image for both pan amounts was set to be the left-view. is observed that the monster-net with adaptive dilations generates naturally looking new views with consistent structures and without discontinuities even at beyond training baseline 3D panning.
A.8 ABLATION STUDIES
We demonstrate the contribution of our design choices this section. Our main contribution is the adaptive "t-shaped" kernel equipped with globally and locally adaptive dilations. Figure (a) shows the effect of adaptive dilations in comparison with the fixed dilation. As can be observed, the resulting synthesized image by fixed local dilation kernel shows unnaturally looking regions with low contrast, discontinuities and blurs. Unlike any previous work, our pipeline can greatly benefit from training on multiple datasets at the same time, as shown Figure greatly benefits from training on two datasets (KITTI and CityScapes) as it exploits the baseline information via its global adaptive dilation property. Training on both KITTI and CityScapes con- tributes to improved geometry reconstruction the network is exposed to wide variety of objects at many different resolutions where, in addition to applying random resizing during training, the resolutions and baselines of these two datasets very different. Figure (c) shows the effect of utilizing our sr-block. Even if the quality of the panned image I is good in terms of structure, the sharpness is further improved by the addition of the super resolution block. Finally, we analyze the effect of the perceptual loss. By utilizing the perceptual loss, our monter-net is able to better synthe- size rich textures and complex or thin structures, as depicted in I6 (d), even though the PSNR and SSIM slightly lower shown in Table [ The last known the "penception-distorrin
That is, our method
18

---

Published as a conference paper at ICLR 2020
Left view (input)
monstcr-net fixed local dilation
monstcr-nct
200
Right view
200
Left view (input)
P.- 200
Right view
P-= 200
200
Left view (input)
P.- -20
200
Right view
P. = 200
Figure 15: Our models generating 3D panned views 30% beyond the baseline for the leftward and rightward camera panning. The magnification of the figures helps to better compare between the fixed local dilation monster-net and adaptive local dilation monster-net.
(a) monster-net
nt
(b) Fixed local dilation
(c) KITTI only
(d) Wo block (T.)
Wo perceptual loss
Figure 16: Ablation studies
tradeoff' (Blau & Michaeli yield higher PSNR/SSIM.
2018 which suggests that better synthetic looking images not always
A.9 DISPARITY/DEPTH ESTIMATION RESULTS
Our monster-net with refinement block beats the current state-of-the-art methods for unsupervised monocular depth estimation in terms prediction accuracy for the KITTI2015 dataset. As shown in T the primitive disparity Dp. that can be extracted from the longer wing of the "t-shaped" kernel, is already among the best performing unsupervised methods. When we add the refinement block with ambiguity learning, more, we get an additional improvement in the a accuracy metric when we add our novel special- post-processing (spp) step. The qualitative comparison against previous methods and the ground truth disparity is shown in Figure [7] It is noted that our monster-net with disparity refinement and special- cistall-ost-pure poss--processig generates very reliable disparity maps even on thin structures and image borders. Additionally, our method benefits from very good detection of far away objects
model results outperform those of the state-of-the-art. Further-
19

---

Published as a conference paper at ICLR 2020
Table 3: Disparity estimation performance on the KITTI2015 metrics from (Eigen et al. 2014). Models are trained with (V) video, (S) stereo, semi global matching (SMG) GT depth (Supp), and can take stereo inputs (s), nine consecutive input frames during testing and training (9-view), or one input frame during testing and nine consecutive views supervision during training (1/9-view). Additionally, (st) stands for student, (1) for teacher, (pp) for post-processing and (spp) for special post- prosesrocs...ng. The best performing models in terms of a' threshold, which is the percentage of disparity values with relative error less than 0.25, are highlighted in bold.
Model
Supp dataset
abs rel., sq rel.. rms..
log rms..
a
a²
a²
Pilzer et al.
2019
(st)
K
0.142
1.231
5.785
0.239
0.795
0,924 0,968
Kanjan
2019)
K
0.140
1.070
5.326
0.217
0.826
0.941 0.975
Ramjan
K+CS
0.139
1.032
5.199
0.213
0.827
0.943 0.977
Godard ct al
2017
X
0.149
2.565
6.645
0.245
0.849 0.936 0.969
Godard ct a
2017
(pp)
K 0.114
1.138
5.452
0.204
0.859 0,946 0,977
Tost et al. 2019)
SMG
K 0.111
0.867
4.714
0.199
0.864 0.954 0,979
Gonzalez & Kim
(019b)
K 0.113
1.114
5.364
0.195
0.866 0.951 0.981
Godard [
(pp)
K+CS
0.100
0.934
5.141
0.178
0.878 0.961 0.986
Wang ct
20199
(9-view)
X
K
0.112
0.418
2.320
0.153
0.882 0.974 0.992
Pilzer et al.
2019 (t)
K
0.098
0.831
4.656
0.202
0.882 0,948 0,973
losı ct al.
(pp)
SMG
K+CS
0.096
0.673
4.351
0.184
0.890 0,961 0,981
ours Teline block
K+CS
0.121
1.028
4.917
0.174
0.885 0.969 0.989
ours with refine block
K+CS
0.098
0.893
4.836
0.166
0.894 0.967 0.988
ours with refine block (pp)
K+CS
0.095
0.793
4.634
0.159
0.896 0.969 0.989
ours with refine block (spp)
K+CS
0.099
0,950
4.739
0.160
0.900 0.971 0,989
Gur & Wolf (2019)
K
0.1TO
0.666
4.186
0.168
0,880
0.966 0.988
Luo ct al. (2018)
0.094
0.626
4.252
0.177
0.891
0.965 0.984
Wang et al. (201 9a) (1/9-view)
X
X
K
0.088
0.245
1.949
0.127
0.915
0.984 0.996
Godard ct al. (201) (8)
K
0.068
0.835
4.392
0.146
0.942
0.978 0.989
(201 92019)
0.062
0.747
4.113
0.146
0.948
0.979 0.990
Wang et al. (2019b)
(8)
X
X
0.049
0.515
3.404
0.121
0.965
0.984 0.992
GT
Gedard (2017)
Genzalez and Kim (2019)
Ours (spp)
Figure 17: Qualitative comparison between our method and the SOTA for the unsupervised monoc- ular depth estimation task on the KITTI2015 dataset.
20


Reviews:
[{"review_id": "ryeKDII6YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes a method to perform stereoscopic view synthesis. The method consists of a neural network model that estimates a novel viewpoint either to the left or to the right of a given image. The two key insights of the proposed method is 1) to learn the weights of a t-shaped kernel when performing novel view synthesis, and 2) to estimate and use adaptive dilations on those kernels.\n\nThe proposed approach is sound and the evaluation methodology used is adequate. The technique proposed is somewhat related to the recent interest in the community to apply CNNs to non-regular grids [1,2,3].\n\nSec. 4.2 states that the proposed method outperforms other existing methods on monocular depth estimation, while the table seems to indicate that other methods (e.g., Wang et al. 2019) obtain a better a1 measure.\n\nI would refrain from calling a 0.004 increase in a1 “a remarkable improvement” (sec. A.9).\n\nMinor details\n- I would refrain from using the word “prove” (abstract, sec. 4), since no proof is provided. “demonstrate”\n- p. 2 “is open known to be a much more complex problem”: I think the authors meant either “is known to be a much more complex problem” or “is still an open problem”?\n- p. 9 “planed” should be written “planned”.\n- p. 9 “the receptive field [...] is of the 153x153 size”: should be “has a size of 153x153”.\n\n\n[1] Su, Yu-Chuan, and Kristen Grauman. \"Learning spherical convolution for fast features from 360 imagery.\" Advances in Neural Information Processing Systems. 2017.\n[2] Coors, Benjamin, Alexandru Paul Condurache, and Andreas Geiger. \"Spherenet: Learning spherical representations for detection and classification in omnidirectional images.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n[3] Zhao, Qiang, et al. \"Distortion-aware CNNs for Spherical Images.\" IJCAI. 2018.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJem4jsAtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a deep learning method to produce \"pans\" of an input image. That is, simulated images of the scene from translated viewpoints. Unlike some previous work that considers only a fixed baseline (such as the 2nd view of a stereo camera), this approach allows generation of a range of views. A specially crafted convolutional architecture is shown to be well-suited to this problem. Results demonstrate visually pleasing image generation and low metric errors on several datasets.\n\nStrengths:\n- The justifications for the design choices in this paper, in particular the convolution structure and connection to image geometry, was quite strong compared to recent papers (although, many of the presented ideas are known in more classical, non-learning, techniques). \n- All presented empirical results are impressive, and show the method is likely to \"really work\" and be reproducible, judging from the number of experiments where the method has consistently outperformed. \n- The method is clear and straightforward to implement either on its own, or as a module/architecture within a larger pipeline.\n\nAreas for Improvement and Detailed Suggestions:\n- The problem of panned view generation is a bit more narrow than some authors are lately considering (generate any viewpoint including off-axis rotations). \n- The t-shaped network architecture here is largely presented as only appropriate to handing image panning. Could a more general network be created, perhaps parameterized by the type of rigid motion occurring? Better, could more flexible networks be proposed with sparsity constraints that allow sub-patterns like the t-network to be learned in a data-driven fashion? \n- Please try to cite referenced work more consistently. For several methods, such as Deep 3D Zoom Net, you have begun to discuss the method using name only for large stretches. It would be helpful to keep using the citation at least once per paragraph to remind the reader of the source of these ideas. \n\nDecision and Justification: \nWeak reject due to the lack of generality in the approach. I suspect the impact of this work will be a bit less than the very competitive bar for ICLR this year. However, I must note that I am the least expert in this area, out of any paper in my stack. \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1euNLVnFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the problem of performing stereoscopic view synthesis (i.e., generating a new view seen from a different camera position) at an arbitrary position along the X-axis from a single input image only. This is an important problem as it enables 3D visualization of a 2D input scene. The paper focuses on the particular problem of generating a stereoscopic view from a single image (i.e., a right and left view from a center image). \nFor this purpose, the paper proposes a t-net architecture which is an autoencoder or U-net like architecture that estimates the values for the t-convolutions proposed in the paper. The network (called monster-net) takes a center image and a pan amount as input, and from those synthesizes the image with the respective view.\n\nThe paper demonstrates that their idea of t-convolutions outperforms recent competing approaches such as deep 3D on available datasets as well as on an in-house collected dataset. The figures provided demonstrate that the views generated by the proposed Monster-net visibly look slightly better than those generated by the competing approaches DeepD and SepConv. In addition, the paper is well written and easy to follows. I therefore recommend acceptance of this paper. I would like to emphasize that while I work in deep learning, I don't work on view synthesis and therefore it is difficult for me to evaluate the novelty of the proposed approach as well as the difficulty of the problem.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers recommend acceptance while one is negative. The authors propose t-shaped kernels for view synthesis, focusing on stereo images. AC finds the problem and method interesting and the results to be sufficiently convincing to warrant acceptance.
Decision: Accept (Poster)