VA-DepthNet: A Variational Approach to Single Image Depth Prediction
Authors: Ce Liu
Suryansh Kumar
Shuhang Gu
Radu Timofte
Luc Van Gool
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Single Image Depth Estimation
Variational Approach.

Abstract: We introduce VA-DepthNet, a simple, effective, and accurate deep neural network approach for the single-image depth prediction (SIDP) problem. The proposed approach advocates using classical first-order variational constraints for this problem. While state-of-the-art deep neural network methods for SIDP learn the scene depth from images in a supervised setting, they often overlook the invaluable invariances and priors in the rigid scene space, such as the regularity of the scene. The paper's main contribution is to reveal the benefit of classical and well-founded variational constraints in the neural network design for the SIDP task. It is shown that imposing first-order variational constraints in the scene space together with popular encoder-decoder-based network architecture design provides excellent results for the supervised SIDP task. The imposed first-order variational constraint makes the network aware of the depth gradient in the scene space, i.e., regularity. The paper demonstrates the usefulness of the proposed approach via extensive evaluation and ablation analysis over several benchmark datasets, such as KITTI, NYU Depth V2, and SUN RGB-D. The VA-DepthNet at test time shows considerable improvements in depth prediction accuracy compared to the prior art and is accurate also at high-frequency regions in the scene space.  At the time of writing this paper, our method---labeled as VA-DepthNet, when tested on the KITTI depth-prediction evaluation set benchmarks, shows state-of-the-art results, and is the top-performing published approach.

Published as a conference paper at ICLR 2023
VA-DEPTHÑET: A VARIATIONAL APPROACH TO SIN- GLE IMAGE DEPTH PREDICTION
Ce Liu Suryansh Kumar Shuhang Gu² Radu Timofte Luc Van Gool 1.4
CCVI,ETH Zürich UUSTC China Banias""iiii UUniversity Würzburg *KU Leuven
sukumar, vangool] @vvi ee ee. ee..thz.cch shuhanggueuestc.edu
ABSTRACT
We introduce VA-DepthNet, a simple, effective, and accurate deep neural network approach for the single-image depth prediction (SIDP) problem. The proposed approach advocates using classical first-order variational constraints for this prob- lem. While state-of-the-an deep neural network methods SIDP learn the scene depth from images in supervised setting, they often overlook the invaluable in- variances and priors in the rigid scene space, such as the regularity of the scene. The paper's 's main contribution is to reveal the benefit of classical and well-founded variational constraints the neural network design for the SIDP task. shown that imposing first-order variational constraints in the scene space together with popular recoder-decoder-beee
network architecture design provides excellent re- sults for the supervised SIDP task. The imposed first-order variational constraint makes the network aware of the depth gradient in the scene space, i.c., regular- ity. The paper demonstrates the usefulness the proposed approach via extensive evaluation and ablation analysis over several benchmark datasets, such as KITTI, NYU Depth V2, and SUN RGB-D. The VA-DepthNet test time shows consid- erable improvements in depth prediction accuracy compared to the prior art and is accurate also at high-frequency regions the scene space. At the time writ- ing this paper, our depth-prediction evaluation set benchmarks, shows state-of-ther results, and is the top-performing published approach"
as VA-DepthNet, when tested on the KITTI
INTRODUCTION
Over the last decade, neural networks have introduced a new prospect for the 3D computer vision field. It has led significant progress on many long-standing problems in this field, such as multi- view stereo (Huang et al., 2018; Kaya et al., 2022). visual simultaneous localization and mapping (Teed & Deng, 2021), novel view synthesis (Mildenhall et al., 2021), etc. Among several 3D vision problems, one of the challenging, if not impossible, to solve is the single-image depth prediction (SIDP) problem. SIDP is indeed ill-posed- a strict geometric sense, presenting an extraordinary challenge to solve this inverse problem reliably. Moreover, since we do not have access to multi-view images, it is hard to constrain this problem via well-known geometric constraints (Longuet-Higgnng 1981; Nistér, 2004; Furukawa & Ponce, 2009; Kumar et al., 2019; 2017). Accordingly, the SIDP problem generally boils down to an ambitious fitting problem, to which deep learning provides a suitable way to predict an acceptable solution to this problem (Yuan et al., 2022; Yin al., 2019).
Impressive earlier methods use Markov Random Fields (MRF) to model monocular cues and relation between several oeer-segmented image parts (Saxena et al., 2007; 2008). Nevertheless, with the recent surge in neural network architectures (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He al., 2016), which has extraordinary capability perform complex regression, many current works use deep learning to solve SIDP and have demonstrated high-quality results (Yuan al., 2022; Aich et al., 2021; Bhat et al., 2021; Eigen al., 2014: Fu al., 2018; Lee al., 2019;
"Corresponding Author
'kitti aetri_ddpth__aae tri_ddpht_pedicctionnnn benchmark prediction_benccharrk
For official code refer here

---

Published as a conference paper at ICLR 2023
2021). Popular recent methods for SIDP are mostly supervised. But even then, they are used less in real-world applications than geometric multiple view methods (Labbé & Michaud, 2019; Müller et al., 2022). Nonetheless, good solution to SIDP is highly desirable in robotics (Yang et al., 2020), virtual-reality (Hoiem et al., 2005), augmented reality (Du el. 2020), view synthesis (Hoiem et al., 2005) and other related vision tasks (Liu et al., 2019).
In this paper, we advocate that despite the supervised approach being encouraging, SIDP advance- ment should not wholly rely on the increase of dataset sizes. Instead, geometric and scene priors could help improve the SIDP results. Not that scene priors have not been studied to improve SIDP accuracy in the past. For instance, Chen et al. (2016) uses pairwise ordinal relations between points to learn scene depth. Alternatively, Yin et al. (2019) uses surface normals as an auxiliary loss to improve performance. Other heuristic approaches, such as Qi et al. (2018), jointly exploit the sopth-to-normal relation to recover scene depth and surface normals. Yet. such state-of-the-an SIDP methods have limitations: for example, the approach in Chen et al. (2016) using ordinal relation to learn depth- over-smooths the depth prediction results, thereby failing to preserve high- fighffreueecc surface details. Conversely, Yin et al. (2019) relies on good depth map prediction from a deep network and the idea of virtual normal. The latter is computed by randomly sampling three non- collinear points with large distances. This is rather complex and heuristic in nature. Qi et al. (2018) uses depth and normal consistency, which is good, yet it requires good depth map initialization.
This brings us to the point that further generalization of the regression- based SIDP pipeline is re- quired. As mentioned before, existing approaches in this direction have limitations and are complex. In this paper, we propose a simple approach that provides better depth accuracy and generalizes well across different scenes. To this end, we resort to the physics of variation (Mollenhoff et al., 2016; Chambolle et al., 2010) in the neural network design for better generalization of the SIDP network, which by the way, keeps the essence of affine invariance (Yin et al., 2019). An image of general scene-indoor or outdoor, has a lot of spatial regularity. And therefore, introducing a variational constraint provides convenient way to ensure spatial regularity and to preserve information related to the scene discontinuities (Chambolle et al., 2010). Consequently, the proposed network is trained in a fully-supervised manner while encouraging the network be mindful of the scene regularity where the variation in the depth is large (cf. Sec.3.1). In simple terms, depth regression must be more than parameter fitting, and at some point, a mindful decision must be either by imag- ing features or by scene depth variation, or both. As we demonstrate later in the paper, such an idea boosts the network's 's depth accuracy while preserving the high-frequency and cow-frequency scene information (see 1).
Our neural network for SIDP disentangles the absolute scale from the metric depth map. models an unscaled depth map as the optimal solution to the pixel-level variational constraints via weighted first-order differences, respecting the neighboring pixel depth gradients. Compared to previous methods, the network's 's task has been shifted away from pixel-wise metric depth learning to learn- ing the first-order differences of the scene, which alleviates the scale ambiguity and favors scene regularity. To realize that, initially employ a neural network to predict the first-order differences of the depth map. Then, we construct the partial differential equations representing the variational constraints by reorganizing the differences into a large matrix, i.e., an over-determined system of equations. Further, the network learns a weight matrix to eliminate redundant equations that do not favor the introduced first-order difference constraint. Finally, the closed-form depth map solution is recovered via simple matrix operations.
When tested the KITTI (Geiger al., 2012) and NYU Depth V2 (Silberman et al., 2012) test sets, our method outperforms prior art depth prediction accuracy by a large margin. Moreover, our model pre-tr on NYU Depth V2 better generalizes to the SUN RGB-D test set.
2
PRIOR WORK
Depth estimation is a longstanding task in computer vision. In this work, we focus on a fully- supervised, single-image approach, and therefore, we discuss prior art that directly relates to such approach. Broadly, we divide the popular supervised SIDP methods into three sub-categories.
(i) Depth Learning using Ranking Ordinal Relation Constraint. Zoran et al. (2015) and Chen et al. (2016) argue that the ordinal relation between points is easier to learn than the metric depth. To this end, Zoran et al. (2015) proposes constrained quadratic optimization while Chen
2

---

Published as a conference paper at ICLR 2023
(a) Image
(b) AdaBins
(c) NeWCRFs
(d) Ours
Figure 1: Qualitative comparison our method's depth result with recent state-of-the-art methods such as AdaBins (Bhat et al., 2021), NeWCRFs (Yuan et al., 2022) on NYU Depth V2 test set (Silherman et al., 2012). It can be observed that our method predicts high-frequency details better than other recent methods.
et al. (2016) relies on the variation of the inception module to solve the problem. Later, Xian et al. (2020) proposes gructure-guider sampling strategies for point pairs improve training efficiency. Recently, Lienen et al. (2021) elaborates on the use of listwise ranking method based on the Plackett- Luce model (Luce, 2012). The drawback of such approaches is that the ordinal relationship and ranking over smooth the depth solution making accurate metric depth recovery challenging.
(ii) Depth Learning using Surface Normal Constraint. Hu et al. (2019) introduces normal loss in addition to the depth loss to overcome the distorted and blurry edges in the depth prediction. Yin al. (2019) proposes the concept of virtual normal impose 3D scene constraint explicitly and to capture the long-range relations in the depth prediction. The long-range dependency in 3D introduced via random sampling of three non-colinear points large distance from the virtual plane. Lately, Long et al. (2021) proposes an adaptive strategy to compute the local patch surface normals at train time from a set of randomly sampled candidates and overlooks it during test time.
(iii) Depth Learning using other Heuristic Refinement Constraint. There has been numerous works attempt to refine the depth prediction as a post-processing step. Liu al. (2015), Li et al. (2015) and Yuan et al. (2022) propose to utilize the Conditional Random Fields (CRF) to smooth the depth map. Lee al. (2019) utilizes the planar assumptions to regularize the predicted depth map. Qi et al. (2018) adopts an auxiliary network to predict the surface normal, and then refine the predicted depth map following their proposed heuristic rules. There are mainly two problems with such approaches: Firstly. these approaches rely on good depth map initialization. Secondly, the heuristic rules and the assumptions might result in over-smoothed depth values at objects boundaries.
Meanwhile, few works, such as Ramamonjisoa et al. (2020), Cheng et al. (2018), 8), Li et al. (2017) were proposed in the past with similar inspirations. Ramamonjisoa et al. (2020), Cheng et al. (2018) methods are generally motivated towards depth map refinement predicted from an off-the-shelf net- work. On the other hand, Cheng et al. (2018) proposes to use an affinity matrix that aims to learn the relation between each pixel's depth value and its neighbors' depth values. However, the affinity matrix has no explicit supervision, which could lead to imprecise learning of neighboring relations providing inferior results. On the contrary, our approach is mindful of imposing the first-order dif- ference constraint leading to better performance. Earlier, Li et al. (2017) proposed two strategies for SIDP, i.e., fusion in end-to-end network and fusion via optimization. The end-to-end strategy fuses the gradient and the depth map via convolution layers without any constraint on convolution weights, which may not be an apt choice for a depth regression problem such as SIDP. On the other hand, the fusion via optimization strategy is based on a non non-differenniable strategy, leading to a non- end-to-end network loss function. Contrary to that, our method well-constrained and performs quite well with a loss function that helps end-to-end learning of our proposed network. Not long ago, Lee & Kim (2019) proposed to estimate relative depths between pairs of images and ordinary depths at a different scale. By exploiting the rank-1 property of the pairwise comparison matrix, it recovers the relative depth map. Later, relative and ordinary depths are decomposed and fused to recover the depth. On a slightly different note, Lee & Kim (2020) studies the effectiveness of various losses and how to combine them for better monocular depth prediction.
To sum up. our approach allows learning of confidence weight to select reliable gradient estima- tion in a fully differentiable manner. Further, it proffers the benefits of the variational approach to overcome the limitations of the existing offte-of-the-an art methods. More importantly, the proposed method can provide excellent depth prediction without making extra assumptions such as good depth initialization, piece-wise planar scene, and assumptions used by previous works mentioned above.
3

---

Published as a conference paper at ICLR 2023
Weight Map
Depth Map
Differenor Map
Weight Marrix
Difference Openator
Depth
Difference
Zap Vistor
X
Zu.
-
Z
V Z
(Z
23 1
S(PZ,
(a) First order depth variation along x-axis.
(b) Overall matrix by ordering the terms.
Figure 2: Illustration of the idea. (a) Depth gradient constraint along axis at location (i, in 4 x 4 matrix
[orm. (b) Construction of the overall matrix formulation with constraints all the pixel locations.
3 METHOD
In this section, we first describe proposed variational constraint and then present the overall network architecture leading to the overall loss function.
3.1 VARIATIONAL CONSTRAINT
Here we introduce our variational constraint and how it can be useful for depth estimation. Consider an unscaled depth map as Zu R#*W with (H, W) symbolizing the height and width, respec- tively. Assuming rn E I NXW and r y E R XTx as the gradient of Z,, in the x and axis, we write
- T...
(1)
Here, and y subscript corresponds the direction from left to right (æ-axis) and top bottom of the image (y-axis), respectively. Elaborating on this, we can write
P 2
Zid Zi,+1 Zid; P&j = Vy = Zit15 2
(2)
Suppose we augment (2) expression for all (i,j). (I...., and j E {1,..., WJ. In that case, we will end up with an over-determined system with 2/W equations in total. Given the predicted r2 and yy, we aim to recover the HW unknown variables in Zu. However, some of the equations could be spurious and deteriorate the overall depth estimation result rather than improving it. As a result, we must be mindful about selecting the equation that respects the imposed first-order constraint and maintains the depth gradient to have meaningful fitting for better generalization. To that end, we introduce confidence weight Σg E [0, H|H HW Ey E [0, 1 for gradient along x, direction. Consequently, we multiply the above two equations by the confidence weight term E and Σ respectively. On one hand, if the confidence is close 1, the equation will have priority to be satisfied by the optimal Z,, On the other hand, if the confidence is close to O, we must ignore the equation. For better understanding. construction in F (a) and Fig.2 (b).
illustrate the first-order difference and weighted matrix
Next, we reshape the Eg, Ey. Пz, Ty, and Zu into column vectors ΣS E [ Σ, E [0, # Fs E nWx1, r, E #wx1, and Z, E #l respectively. Organizing R2#wx1, we can write the
diag([É.; Σn,) E R2HW X28W and I = concat[Ta;i; overall expression in a compact matrix form using simple algebra as follows
ŠPŻ. Ër
(3)
where P E {1,0, - XHW is the first-order difference operator. Specifically, is a sparse matrix with only few elements as 1 -1. The 4h row of P provides the first-order difference operator for the ęth equation. The position of and indicates which pair of neighbors to be considered for the constraint. F (b) provides visual intuition about this matrix equation.
E3)(3) can utilized to recover Z, from the predicted Fz. Hy. Σg. and Σ,. As alluded to above, have more equations than unknowns, hence, we resort to recovering the optimal depth map
4

---

Published as a conference paper at ICLR 2023
Difference Operator
reArrange
Image
(b) V-Layer
Encoder
Ex.Ey Ey
Weight
Kgf
Eq. (5)
(b) V-Layer
Depth Zu
Difference Vector 1
Depth Z
Feature
Feature
Fenture
1 Upsampling
(c) Upsample
Upsample Refine
Refine
Metrie Layer
Refine
Refine
Refine
Depth
Depth
Depth
Depth
Overall Architecture
Figure 3: Overview our framework. Given an input image, first an encoder employed extract features. Then we predict the depth map by the V-layer. Next, we gradually upsample and refine the depth map. In the end, we recover the metric depth by the metric layer.
ž IR nWxx by minimizing the following equation:
" arg min 1
(4)
Ze
Refer Appendix for the full derivation. The closed-form solution can be written as follows:
K2
( PT
(5)
Denote Kg & (P'E' in (5), we write overall equation as ž = KgT. Next, we
describe the overall network architecture.
3.2 OVERALL NETWORK ARCHITECTURE
Our overall network architecture is composed of four main modules as follows.
(a) Encoder. Given input image, the encoder computes the hierarchical feature maps through series of stages. To be precise, our encoder has four stages. Each stage contains transformer blocks (Liu et al., 2021b). At the end of each stage, collect the final feature map as the output of the encoder resulting in the encoded feature maps with strides 4, 8, 16, and 32, respectively. Our encoder module is inspired by Liu al. (2021b), recent state-of-the-ar transformer network design. We use it as our backbone by removing the final global pooling layer and fully connected layer.
(b) Variational Layer (V-Layer). The goal of this layer is to compute a map from encoded feature maps to unscaled depth map, which adheres to the first-order variational constraint. As of V-layer, we feed the feature maps of strides 16 and as input which the output of the encoder. Since these features are at different resolutions, bi-linear interpolation and concatenate to construct / E RCC the height, width, and the number of channels, respectively. Note that H, W is not the same as the original resolution of the ground- -ooun depth and images. We use of two convolutional layers on la. to predict the depth gradient and corresponding weight for each pixel follows:
upsample the feature map of stride 32 to stride 16 via
where (H, W. C)) symbolizing
[T.. I Ez, Ey} f(T::
where, f(lo: ; f) denotes the convolutional layers with parameters 6. The predicted depth gradients Tz and T, are observed to be more accurate smooth surface than at boundaries. This brings

---

Published as a conference paper at ICLR 2023
us again to the point made above that must take care of which first-order constraint must be included and discarded during regression. Using the Eq.(6) prediction, we construct the variational constraint (3)( and obtain the unscaled depth map following Eq.( The resulting depth map has a resolution of 1/16 to the original image, which is later upsampled to the appropriate resolution.
To capture more scene features, we generate multiple channels (denoted as S) of [T',,'' 2g اny using (6) As a result, we have a group of depth maps stacked along the channel dimension. For feature map with spatial resolution H x W, our V-layer has a complexity of O W To overcome complexity issue, we perform V-layer operation on feature maps with stride 16 and then upsample and refine the depth maps in the later stage. The V-layer pipeline is shown in Fig.3(b).
(c) U Upsample and Refine. This module upsamples and refines the input depth map via encoded features given depth map resolution. To this end, we perform refinement at three different reso- lutions in a hierarchical manner. Given the V-layer depth map at 1/16 resolution, we first refine the depth via encoded features this resolution. Concretely, this refinement is done using the following set of operations.
(1) concatenate the feature map and the depth map: (2) use one convolutional layer with ReLU activation fuse the feature and depth information; and (3) predict refined feature and depth map via a convolutional layer. Later, the refined feature and depth map are upsampled and fed into 1/8 for later refinement using the same above set of operations. Finally, the exact is done at 1/4 resolution. Note that these steps are performed in sequel. At the end of this module, we have depth map of 1/4 of the actual resolution. The upsample and refine procedure are shown in Fig.3(c).
(d) Metric Layer. We must infer the global scene scale and shift to recover the metric depth. For this, we perform global max pooling on the encoded feature map of stride 32. The resulting vector is fed into a stack of fully connected layers to regress the two scalars, i.e., one representing the scale and while other representing the shift. Using the feature map of stride 32 is motivated by the observation that we have much richer global scene context using it than at higher depth resolution. also provides a good compromise between computational complexity and accuracy.
3.3 LOSS FUNCTION
Depth Loss. It estimates the scale-invariant difference between the ground-truth depth and predic- tion at train time (Eigen et al., 2014). The difference is computed by upsampling the predicted depth map to the same resolution as the ground truth via bi-linear interpolation. Denoting the predicted Rmxx compute the depth loss as follows
and ground-truth depth 2 E m
Zet
Cdepth (2, Zgt)=
1 N
Σ
22 (b.))
e.)), where, edd llog 2:: log Z-d.
=
N2
Here, N is the number of positions with valid measurements and a [0, 1] is parameear Note that the above loss is used for valid measurements only.
Variational Loss. We detine this loss using the output of V-layer. Suppose the ground-truth depth map to be Lyt E _ and the predicted depth map for S channels as Zu E R S Since the depth resolution is not same this layer, downsample the ground truth. It is observed via empirical study that low-resolution depth map in fact help capture the first-order variational loss among distant neighbors. Accordingly, we downsample the Zye instead of upsamping Z.. We downsample Zye denoted as Qut TR HxW by random pooling operation, i.e., we randomly select location where we have a valid measurement since ground-truth data may have pixels with no depth values. The coordinates of selected location in Zge Zu E Sx and the corresponding depth value is put in Q E Sxx via bi-linear interpolation. We compute the variational loss as
Lear Quar(Q
1
Σ [Conv(Q) VQ (1))
N'
gt
where N' is the number of positions having valid measurements, V symbolises the first-order dif- ference operator, and Conv refers to the convolutional layer. Here, we use the Conv layer to fuse S depth maps into single depth map and also compute its horizontal and vertical gradient.
Total Loss. We define the total loss as the sum of the depth loss and the variational loss i.c., f = Edepth + XL where À is the regularization parameter set 0.1 for all our experiments.

---

Published as a conference paper at ICLR 2023
Table 1: Comparison with the state-of of the-an methods on the NYU test set (Silberman et al., 2012). Please refer to Sec.4.1 for details.
Method
Backbone
olL
Abs Rel 4
RMS
RMS log!
61 1
GeoNet (Q: et 2018)
50
0.128
0.960
DORN (Fu et al, 2018)
ResNet-100
0.115
0.509
0.828
0.965
VNL (Yin et 2019)
ResNeXt-101
0.108
0,416
0.875
0.976
TransDepth (Yang 2022)
VIT-B
0.106
0.365
0,900
0.983
ASN (Long al., 2021)
HRNes-48
0.101
0.377
0.890
0.982
(Lec 2019)
DenseNot-161
11.533
0.110
0.142
0.885
0.978
DPT-Ilybird (Ranftl et al., 2021)
VIT-B
0.110
0.357
0.904
0.988
AdaBins (Bhat et al., 2021)
EANet mini
10.570
0.103
0.983
ASTrans (Chang et al., 2021)
VIT-B
10.429
0.103
0.374
0.132
0.902
0.985
NoWCRFs (Yuan al., 2022)
Swin-L
9.102
0.095
0.331
0.119
0,922
0.992
Swin-L
8.198
0,086
0.937
0,992
% Improvement
-9.93%
-9.4%%
-8.16/5
-9.24%
+1,63%
+0.00%
Table 2: Comparison with the statc-of-thc-as methods on the the KITTI official test set (Geiger et al., 2012).
We only list the results from the published methods. Please refer Scc.4. for details. Mcthod
Backbone
S1Log
Abs Rell
Rcll
IRMS.
DLE [Lau at., 2021a)
ReaNet-34
11.81
9.09
2.22
12.49
DORN (Fu el al., 2018)
ResNet-101
11.80
8.93
2.19
13.22
BTS (Lee al., 2019)
DenseNet-16 161
11.67
9.04
2.21
12.23
BANet (Aich et al., 2021)
DenseNet-161
11.55
9,34
2.31
12.17
PWA (Loc al., 2021)
ResNeXt-101
2.30
12.32
ViP-Deepl.ab (Qiao et al., 2021)
10.80
8,94
2.19
11.77
NeWCRFs (Yoan et al., 2022)
Swin-L
10.39
8.37
1.83
11.03
Ours
Swin-T.
9.84
7.96
1.66
10.44
& Improvement
-5.29%
-4.0%
-9.29%
-5.35%
4 EXPERIMENTS AND RESULTS
Implementation Details We implemented our method in PyTorch 1.7.1 (Python 3.8) with CUDA 11.0. The software evaluated on computing machine with Quadro-RTX-6000 GPU.
Datasets. We performed experiments on three benchmark datasets namely NYU Depth V2 (Silber- man et al., 2012), KITTI (Geiger et al., 2012), and SUN RGB-D (Song et al., 2015). (a) NYU Depth V2 contains images with 480 x 640 resolution with depth values ranging from 0 to 10 meters. We follow the train and test set split from Lee al. (2019), which contains 24,231 train images and 654 test images. (b) KITTI contains images with 352 x 1216 resolution where depth values range from 0 to 80 meters. The official split provides 42, 42,949 train, 1,000 validation, and 500 test images. Eigen et al. (2014) provides another train and test set split for this dataset which has 23,488 train and 697 test images. (c) SUN RGB-D We preprocess its images to 480 x 640 resolution for consistency. The depth values range from 0 to 10 meters. We use the official test set (5050 images) for evaluation.
Training Details. We use (Liu al., 2021b) network as our backbone, which pre-trained on ImageNet (Deng et al., 2009). We use the Adam optimizer (Kingma & Ba, 2014) without weight decay. We decrease the learning rate from 35 to le by the cosine annealing scheduler. To avoid over-fitting, we augment the images by horizontal flipping. For KITTI (Geiger et al., 2012), the model trained for 10 epochs for the official split and 20 epochs for the Eigen split (Eigen et al., 2014). For NYU Depth V2 (Silberman al., 2012), the model is trained for 20 epochs.
Evaluation Metries. We report statistical results on popular evaluation metries such as square root of the Scale Invariant Logarithmic error (SILog), Relative Squared error (Sq Rel), Relative Absolute Error (Abs Rel), Root Mean Squared error (RMS), and threshold accuracy. Mathematical definition related to each one of them is provided in the Appendix.
4.1 COMPARISON TO STATE OF THE ART
(1b. (2), (3), and (4b.(4) provide statistical comparison results with the competing meth- ods on NYU Depth V2, KITTI official split, KITTI Eigen split, and SUN RGB-D, respectively. Our proposed approach shows the best results for all the evaluation metrics. Particularly set, we reduce the SILog eπror from the previous best result, 9.102 to 8.198, and increase o, from 0.922 to 0.937. More qualitative results including V-layer output presented in the Appendix.
the NYU test
7

---

Published as a conference paper at ICLR 2023
Table Comparison with the state-oftthe- methods on KITTI Eigen test set (Eigen et al., 2014).
Method
Backhone
Sfo.og
RMS.
log!
81
6g
DORN (Fu 2018)
ResNet-101
0,072
0.275
0.120
0.932
0.984
VNI (Vin et al., 2019)
ResNeXt-100
0.072
0.326
0.117
0.938
0.990
Depns (Yang al. 2011)
VTI-B
8,900
0.275
0.994
BTS (Lee et al., 2019)
DenseNet- 161
8.933
0.060
0.280
0.096
0.955
0.993
DPT-Hybird (Ranfil et 2021)
VIT-B
0.062
0.257
0.995
AdaBins (Bhat et al., 2021)
BffNet-BS- -VT-minii
8.022
0,058
0.236
0.089
0,964
0.995
ANTrams (Chang et sl., 2021)
VIT-B
7.897
0.058
0.269
0.089
0,963
0.995
NeWs (Yuan al., 2022)
Swin L
6,986
0.052
0.213
0.079
0,974
0.997
Ours
Swin.T.
6.817
0,050
0.209
0.076
0.977
0.997
Improvement
-2.42%
-3.85%
-1.88%
-3.80%
+0.03%
+0.00%
Table 4: Comparison with AdaBins and NeWCRFs on SUN RGB-D set. All methods trained on NYU
Depth V2 train set without fine-tuning on SUN RGB-D.
Method
Backbone
SILog 1
Abs Rel 4
RMS
RMS log.
61
82 1
AdaBins(Bhat al. 2021)
BiNNa--5+VVi-miit
13.652
0.130
0.321
0.137
0.906
0.982
NeWCRE's (Yuan et al., 2022)
Swin-L
13.695
0.105
0.322
0.138
0.920
0.980
Ours
Swin-L
12.59
0.094
0.299
0.127
0.929
0,983
Improvement
-7.73%
-10.88%
-6.85%
-7.30%
+0.98%
+0.10%
For the SUN RGB-D test set, all competing models, including ours, are trained on the NYU Depth V2 training set (Silberman et al., 2012) without fine-tuning on the SUN RGB-D. In addition, we align the predictions from all the models with the ground truth by a scale and shift following Ranftl al. (2020). (ab.(4) results show our method's better generalization capability than other approaches. Extensive visual results are provided in the Appendix and supplementary video.
4.2 ABLATION STUDY
All the ablation presented below is conducted NYU Depth V2 test set (Silberman et al., 2012).
(i) Effect of V-Layer. To understand the benefit and outcome of our variational layer compared to other popular alternative layers in deep neural networks for this problem, we performed this ablation study. We replace our V-layer firstly with convolutional layer and later with a self-attention layer. (5) provides the depth prediction accuracy for this ablation. For each introduced layer in Tab.(5), the first and second rows show the performance of the depth map predicted with (w) and without (w/o) subsequent refinements (ef. Sec.3.2 (c)), respectively. For the self-attention layer, we follow the ViT (Dosovitskiy et al., 2021) and set the patch size to be one as we use the feature map with stride 16. We also adopt the learnable position embedding (PE) with 128 dimensions. We set the number of heads to be 4 and the number of hidden units to be 512. As shown in Tab. our V-layer indeed helps improve the accuracy depth prediction compared to other well-known layers. More experiments on KITTI and SUN RBG-D are provided in the Appendix.
(ii) Performance with Different Network Backbone, We evaluate the effects of our V-layer with different types of network backbones. For this ablation, we use Swin-Large (Liu et al., 2021b), Swin-Small (Liu et al., 2021b), and ConvNeXt-Smaal (Liu al., 2022). The SILog error is shown in Fig. 4. The results show that our V-layer improves the transformer and the convolutional network performance. An important observation is that our V-layer shows excellent improvements in depth prediction accuracy on weaker network backbones.
(iii) Performance with Change the Value of S. For this ablation, we change the value of S in the V-layer and observe its effects (cf. Sec.3.2 (b)). By increasing S, we generate more channels of I and E which in-effect increases V-layer parameters. In the subsequent step, we expand the
Table 5: Benefit of V-layer. We replace the proposed V-layer with a single convolutional layer and self- attention layer, and evaluate the accuracy of depth map predicted with and without subsequent refinements.
Layer
Reline
SiLog
Abs Rei
RMSL
RMS log 1
8,
dg
Convolution
wio
8,830
0.090
0.325
0.114
0.927
0.900
8.688
0.089
0.991
Sell-Atienion PE
8.790
0.090
0,318
0.114
0.927
0.99)
w/
8.595
0.089
0.316
0.112
0.929
0.991
V-Layer
w/
8.198
0.086
0.304
0.108
0.937
0.992
8

---

Published as a conference paper at ICLR 2023
number of channels 128 by convolutional layer to use the subsequent layers they are. The results are shown in (6). For reference, we also present the result by replacing the V-layer with convolutional layer in the first row in (6). By increasing S, reduce the SILog error, the price of the speed (FPS). Yet, no real benefit is observed with S more than 16.
(iv) Effect of Confidence Weight Matrix & Difference Op- erator in V-Layer: For this ablation, we study the network's s depth prediction under four different settings. (a) without V- layer and replace it with convolutional layer (b) without the confidence weight matrix (c) with learnable difference operator and (d) our full model. The depth prediction accuracy observed under these settings is provided in T7b. Clearly, our full model has better accuracy. An important empirical observation, we made during this test when we keep learnable V-layer has more learnable parameters, the performance becomes worse than with fixed difference operator.
w/
9.50
659'6
SlL.gg Brar
9.00
850'6
8.50
8.00
Swit-L.
Swin-8
Con
Figure 4: Evaluation Swin Swin
ConvNeXi-S w/ and w/o the V-layer.
Table 6: Analysis of the number of feature groups. More groups reduce the SILog error.
Table 7: Analysis of the confidence weight matrix
Σ and the difference operator P.
SILogg
Abs Rel.
RMS.
FPS 1
STLogg
Ahs Rell
RMS J.
w/o V-layer
8.688
0.317
9.343
wo V-layer
8,688
0,089
0.317
8.456
0.088
0.310
8.175
(b) w/o
8.537
0.089
0,316
16
8.198
0.086
0.304
7,032
learnable P
8.355
0.088
0.310
128
0.309
3.320
(d) full
8.198
0.086
0.304
4.3 NETWORK PROCESSING TIME & PARAMETERS
We compared our method's inference time and the number of model parameters to the Ad- aBins (Bhat et al., 2021) and the NeWCRFs (Yuan et al., 2022). The inference time is measured on the NYU Depth V2 test set with batch size 1. We have removed the ensemble tricks in Ad- aBins and NeWCRFs for an unbiased evaluation, resulting compared to (1b statistics. As is shown in (8), our method is faster and better than Ad- aBins and NeWCRFs using Swin-Small backbone. With the same backbone as the NeWCRFs, i.e., Swin-Large, we achieve much better depth prediction results. Hence, our method with Swin-Small backbone provides a better balance between accuracy, speed and memory foot-print.
a slight increase in SILog error as
Table Comparison of the inference time and parameters to AdaBins and NeWCRFs on NYU Depth V2. show our results using the Swin-Small and Swin-Large backhone.
AdaBins
NeWCRFss
Ours
Ours (Large)
(Rhat al., 2021)
(Yuan al., 2022)
(Small)
SiLog Eiror
10.651
9.171
9,069
8,198
Speed (IPS)
5.638
10.551
11.891
7.032
Param (M)
75
258
5 CONCLUSION
In conclusion, simple and effective approach for inferring scene depth from single image is in- troduced. The proposed SIDP approach is shown to better exploit the rigid scene prior, which is generally overlooked by the existing neural network-based methods. Our approach does not make explicit assumptions about the scene other than the scene gradient regularity, which holds for typical indoor or outdoor scenes. When tested on popular benchmark datasets, our method shows signifi- cantly better results than the prior art, both qualitatively and quantitatively.
LEDNOWLED
This work was partly supported by ETH General Fund (OK), Chinese Scholarship Council (CSC), and The Alexander von Humboldt Foundation.

---

Published as a conference paper at ICLR 2023
REFERENCES
Shubhra Aich, Jean Marie Uwabeza Vianney, Md Amirul Islam, and Mannat Kaur Bingbing Liu. Bidirectional attention network for monocular depth estimation. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 1774-117522 IEEE, 2021.
Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adap- tive bins. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 4009-4018, 2021.
Antonin Chambolle, Vicent Caselles, Daniel Cremers, Matteo Novaga, and Thomas Pock. An intro- duction to total variation for image analysis. Theoretical foundations and numerical methods for sparse recovery, 9(263-340):22) 2010.
Wenjie Chang, Yueyi Zhang, and Zhiwei Xiong. Transformer- based monocular depth estimation with attention supervision. In 32nd British Machine Vision Conference (BMVC 2021). 2021.
Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception in the wild. Advances in neural information processing systems, 29, 2016.
Xinjing Cheng, Peng Wang, and Ruigang Yang. Depth estimation via affinity learned with convo- lutional spatial propagation network. Proceedings of the European Conference on Computer Vision (ECCV), pp. 103-119, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, et al. Depthlab: Real time- 3d interaction with depth maps for mobile augmented reality. In Proceedings of the 33rd Annual ACM Symposium User Interface Software and Technology. pp. 829-8433 2020.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using a multi-scale deep network. Advances in neural information processing systems, 27, 2014.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
Yasutaka Furukawa and Jean Ponce. Accurate dense, and robust multiview stereopsis. IEEE trans- actions on pattern analysis and machine intelligence, 32(8):1362-1333 1376, 2009.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3354-3361. IEEE, 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pallern recognition, pp- 770-778, 2016.
Derek Hoiem, Alexei A Efros, and Martial Hebert. Automatic photo pop-up. In ACM SIGGRAPH 2005 Papers, pp. 577-584. Association for Computing Machinery, 2005.
Junjie Hu, Mete Ozay. Yan Zhang, and Takayuki Okatani. Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries. In 2019 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 1043-1051. IEFF, 2019.
10

---

Published as a conference paper at ICLR 2023
Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi- view stereopsis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2821-28300 2018.
Berk Kaya, Suryansh Kumar, Carlos Oliveira, Vittorio Ferrari, and Luc Van Gool. Uncertainty- aware deep multi-view photometric stereo. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Patlern Recognition, pp. 12601-12611, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint WrXiv:1412.69880 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Suryansh Kumar. Yuchao Dai, and Hongdong Li. Monocular dense 3d reconstruction of a complex dynamic scene from two perspective frames. In Proceedings of the IEEE international conference on computer vision (ICCV), pp. 4649 4657,667,220 2017.
Suryansh Kumar, Yuchao Dai, and Hongdong Li. Superpixel soup: Monocular dense reconstruc- tion of complex dynamic scene. IEEE transuctions on paltern analysis and machine intelligence (TPAMI), 43(5) 1717, 2019.
Mathieu Labbé and François Michaud. Rtab-map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. Journal of Field Robotics, 36(2):416-446, 2019.
Hae Lee and Chang-Su Kim. Monocular depth estimation using relative depth maps. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2019.
Jac-Han Lee and Chang-Su Kim. Multi-loss rebalancing algorithm for monocular depth estimation. In European Conference on Computer Vision, pp. 785-801. Springer, 2020.
Jin Han Lee, Myung-Ky Kyu Han, Dong Wook Ko, and Il Hong Suh. From big small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv: 10326, 2019.
Sihaeng Lee, Janghyeon Lee, Byungju Kim, Eojindl Yi, and Junmo Kim. Patch-wise attention network for monocular depth estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1873-1881, 2021.
Bo Li, Chunhua Shen, Yuchao Dai, Anton van den Hengel, and Mingyi He. Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2015.
Jun Li, Reinhard Klein, and Angela Yao. A two-streamed network for estimating fine-scaled depth maps from single rgb images. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3372-3380, 2017.
Julian Lienen, Eyke Hüllermeier, Ralph Ewerth, and Nils Nommensen. Monocular depth estimation via listwise ranking using the plackett-luce model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14595-14600 June 2021.
Ce Liu, Shuhang Gu, Lue Van Gool, and Radu Timofte. Deep line encoding for monocular 3d object detection and depth prediction. 32nd British Machine Vision Conference (BMVC 2021). pp- 354, 202 la.
Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian Reid. Learning depth from single monocular images using deep convolutional neural fields. IEEE transactions on pattern analysis and machine intelligence, 38(10):202420333 2015.
Xingtong Liu, Ayushi Sinha, Masaru Ishii, Gregory D Hager, Austin Reiter, Russell H Taylor, and Mathias Unberath. Dense depth estimation in monocular endoscopy with self-supervised learning methods. IEEE transactions medical imaging, 39(5): 14438-14477 2019.

---

Published as a conference paper at ICLR 2023
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Viston, pp. 10012-10022, 2021b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 11976-1 1986, 2022.
Xiaoxiao Long, Cheng Lin, Lingji Liu, Wei Li, Christian Theobalt, Ruigang Yang, and Wenping Wang. Adaptive surface normal constraint for depth estimation. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 12849-12885 2021.
Christopher Longuet- Higgins.. A computer algorithm for reconstructing a scene from two projec- tions. Nature, 293(5828):133-133 1981.
Duncan Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2012.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Thomas Mollenhoff, Emanuel Laude, Michael Moeller, Jan Lellmann, and Daniel Cremers. Sublabel-accurate relaxation nonconvex energies. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3948-3956, 2016.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies prim- itives with multiresolution hash encoding. ACM Trans. Graph., 41(4):102:11002:55, July 2022. doi: 10.1145/35222333530222
David Nistér. An efficient solution to the five-point relative pose problem. IEEE transactions on pattern analysis and machine intelligence, 26(6):756-770, 2004.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia. Geonet: Geometric neural network for joint depth and surface normal estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 283-291,77 2018.
Siyuan Qiao, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang- Ching Chen. Vip-deeplab: Learn- ing visual perception with depth-aware video panoptic segmentation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3997-4008, 2021.
Michael Ramamonjisoa, Yuming Du, and Vincent Lepetit. Predicting sharp and accurate occlu- sion boundaries in monocular depth estimation using displacement fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14648-14657, 2020.
René Ranfil, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transac- tions Pattern Analysis and Machine Intelligence (TPAMI), 2020.
René Ranfil, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 12179-12188 2021.
Ashutosh Saxena, Min Sun, and Andrew Y Ng. Learning 3d scene structure from single still image. In 2007 IEEE 11th international conference on computer vision, pp. 1-8. IEEE, 2007.
Ashutosh Saxena. Min Sun. and Andrew Y Ng. Make3d: Learning 3d scene structure from single still image. IEEE transactions on pattern analysis and machine intelligence, 31(5):824-88) 840, 2008.
12

---

Published as a conference paper at ICLR 2023
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In European conference on computer vision, pp. 746-760. Springer, 2012.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 567-576, 2015.
Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras.
Advances in Neural Information Processing Systems, 34: 16358-1666,, 2021.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.0802,, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Ke Xian, Jianming Zhang, Oliver Wang, Long Mai, Zhe Lin, and Zhiguo Cao. Structure- guided ranking loss for single image depth prediction. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp- 61 1-620, 2020.
Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe, and Elisa Ricci. Transformer-based attention networks for continuous pixel-wise prediction. In Proceedings of the IEFE/CVF International Conference Computer Vision, pp. 16269-16279, 2021.
Nan Yang. Lukas von Stumberg, Rui Wang, and Daniel Cremers. D3vo: Deep depth, deep pose and deep uncertainty for monocular visual odometry. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1281-1292, 2020.
Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan. Enforcing geometric constraints of vir- tual normal for depth prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5684-5693, 2019.
Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Neural window fully-connected crfs for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3916-3925, June 2022.
Daniel Zoran, Phillip Isola, Dilip Krishnan, and William T Freeman. Learning ordinal relationships for mid-level vision. In Proceedings of the IEEE international conference on computer vision, pp. 388-396, 2015.
A APPENDIX
A.1 TRAINING DETAILS
We implement our framework in PyTorch (Paszke et al., 2019). We adopt the Swin-Large (Liu al.. 2021b) as our backbone conduct ablation experiments and compare with the state of the arts. And the backbone is pre-trained on ImageNet-22K (Deng et al., 2009) by image classification. For training, we use the Adam optimizer (Kingma & Ba, 2014) without weight decay. We decrease the learning rate from 36 to 1c-5 by the cosine annealing scheduler. To avoid over-fitting, we augment the images by horizontal flipping. For both train and test, we keep the resolution of images to be 352 x 1216 in KITTI (Geiger et al., 2012), and 480 x 640 in both NYU Depth V2 (Silberman et al., 2012) and SUN RGB-D (Song et al., 2015). In KITTI, the model trained for 10 epochs for the official split and 20 epochs for the Eigen split (Eigen al., 2014). In NYU Depth V2, the model trained for 20 epochs. We set the batch size to be 4 and respectively for ablation experiments and comparison to the state of the arts.
13

---

Published as a conference paper at ICLR 2023
A.2 EVALUATION METRICS
Suppose the predicted and ground-truth depth to be 2 E Rm*x" and ge E Rm respectively, and the number of valid pixels be N. We follow the existing methods (Yuan al., 2022) and utilize the following measures for quantitative evaluation:
square root of the Scale Invariant Logarithmic error (SILog): # Eigletiee
N' DEirehyyy where ed log Zis log Zzi:
Relative Squared enror (Sq Rel): #/2 Σi, (2 Zgy /2/2z/
Relative Absolute Error (Abs Rel): 2/ Eis 12 Zgd 1/
Root Mean Squared error (RMS): //2 V ( ?:yo
Root Mean Squared Logarithmic error (RMS log): 1/ /V
threshold accuracy (ok): percentage of 2rd s.t. max 2:3 Zg
7 2 1.25*
A.3 DERIVATION OF THE VARIATIONAL CONSTRAINT
In this part, we introduce the derivation of the variational constraint in more detail. Firstly, present the detailed form of the difference operator P, the confidence weight matrix Σ, the difference vector I' and the depth vector Z.. when the depth map Z,, is at resolution 2 x 2. Secondly we show the derivation of the optimal solution ž;-
Overall Matrix Form. Suppose the unscaled depth map to be Zu R2x2, the difference map to be r E IR and ry E 2x2 and the corresponding confidence weight map to be Σ, E [0, 1 1 and Mn E [0, 1=2x2_ We first reorganize the elements in Ia Ty Vz, Mn to construct the difference
vector r E R and the confidence weight matrix E E Bx888
r1.1
1
0
0
ri,2
1,2
0
0
0
p
p
0
2
211
0
0
0
0
f
rl.1
-=
0
(
222
0
ELI
0
0
0
0
0
0
0
y
0
0
0
0
Σ
0
0
0
0
0
0
12.1
0
0
0
0
0
Next, we apply the difference operator P E
{1, 0, -188x4;
0
0
0
0
0 0
-1
11010
0
0
P
1
0
1000
1
000
-1
100
on the depth vector Ž, E 4
Z1,1
7.2
2,
7
Z2.2
Finally, we construct the following constraint equation:
ÊPŽ, = Eř.
14

---

Published as a conference paper at ICLR 2023
In the above equation, each row represents weighted constraint for the first-order difference along the a-axis:
E (Z Zz-)) E
(10)
or along the y-axis:
Ey (Z " E
(11)
where (Z - 2:²) = r and (Zit1) - -::)) - T are the variational constraints for the first-order difference, while 2 and 2ny are the contidence weights for the constraints.
Optimal Solution. Given Σ and F, we search for the optimal depth vector ž by minimizing the residual of (9):
z arg min |SS(PZ.. - 122 Ze
(12)
Firstly, the objective function, [/E 12, is convex with respect to Zy. Because || ll2 is convex function. A composition of ||. 1l2 and the affine function, E(PZz - r), is still convex.
Secondly, the optimal solution of a convex function can be found at where the first derivative is zero. Thereby we obtain the final solution Z p2
A.4 NETWORK STRUCTURE DETAILS
In this part, we introduce the detailed structure of our network. In general we set the kernel size of convolutional layers be 3 unless otherwise stated.
(a) Encoder. We adopt the Swin-Large (Liu et al., 2021b) as our backbone. The network first divides the image into patches each of size 4 and embeds each patch into 96-dimensional vector. The above procedure is implemented by convolutional layer with kernel size 4 and stride 4. Then there 4 transformation stages be applied the embedded vector. The first, second, third and forth stages include 2, 2, 18, 2 blocks, respectively. More specifically, each block will divide the feature map into non-overlapped windows of size 12 12, and compute new features within each window following the transformer (Vaswani et al., 2017). The feature channels in the 4 stages are 192, 384, 768 and 1536, respectively, and the number of heads are 6, 12, 24, and 48 respectively. In the end of each stage, there will be a downsampling operation to reduce the resolution of the feature map by 2. We collect the fcature map before the downsampling operation as the output of the stage. In the end, the strides of output feature maps are 4, 8, 16, and 32 respectively. And the channels are 192, 384, 768, and 1536, respectively.
(b) Upsampling 32->16. Given the feature maps of strides 32 and 16 from (a), we first upsample the feature map stride 32 to stride 16 via bi-linear interpolation. Then, we concatenate the feature maps and obtain a new feature map of stride 16 and channels 1536 + 768 2304. We apply convolutional layer that has 2304 input channels, 2304 output channels, and 4 groups, to fuse the in- formation. We also append an instance normalization layer (Ulyanov al., 2016) and a LeakyReL.U activation function. Next, we apply another convolutional layer with 2304 input channels, 512 out- put channels to compress the feature channels. Again we append an instance normalization layer and the LeakyReLU activation function. In the end, we add a skip connection between the result feature map and the previous concatenated feature map by addition operation. The concatenated feature map will be transformed to consistent number of channels in advance by a convolutional layer with 2304 input channels and 512 output channels. The final feature map has stride 16 and 5 512 channels.
(c) V-Layer. The V-layer takes the output feature map from (b) as input. We first utilize a convo- lutional layer with 512 input channels and 512 output channels to transform the feature map into a more appropriate hidden space. The feature is transformed by the LeakyReLU activation func- tion. Then we utilize another convolutional layer with 512 input channels and 2 x 16 = 32 output channels to predict the gradients along the horizontal and vertical axis, respectively. The number 16 represents that we predict 16 channels of horizontal gradients and 16 channels of vertical gradients respectively, where each channel is expected to capture different information in scenes. Similarly, we also another convolutional layer with 512 input channels, 2 x 16 32 output channels to predict the corresponding confidence weight maps. The confidence weight will be transformed by the Sigmoid function. Then reshape the predictions and compute the unscaled depth map for
15

---

Published as a conference paper at ICLR 2023
each pair of gradient and confidence weight following (5)( The unscaled depth maps computed from all the pairs will be concatenated along the channel. Thereby we obtain a depth map with 16 the depth map. where the 16 channels
channels and stride 16. We apply a group normalization are viewed as a single group. In the end, we expand the channels of the depth map into 128 by a convolutional layer. Thereby the output of the V-layer is depth map with 128 channels and stride 16.
We refine the feature map from (b) and the depth map from (c). Specifically, we first concatenate the feature map with depth map, and obtain a new feature map with 512 + 128 = 640 channels. Then we employ convolutional layer with 640 input channels and 640 output channels to fuse the information. The result feature map will be further transformed by the LeakyReLU activation function. Next convolutional layer with 640 input channels and 128 output channels is applied to predict the refined depth map. Similarly, we also predict the refined feature map by a convolutional layer with 640 input channels and 512 output channels.
(d) Refine 16.
(e) Upsampling 16->8. Here, we upsample the feature map from (d) to stride 8 via bi-linear in- terpolation. To recover the information of scenes, we concatenate with the feature map from the encoder that has the same stride. In such a way, we obtain a new feature map with 384 + 512 = 896 channels and stride 8. Similar to (b), we apply a convolutional layer with 896 input channels, 896 output channels, and 4 groups to fuse the information. Then the feature map is transformed by the instance normalization layer and the LeakyReL.U activation function. Next, we apply another convolutional layer with 896 input channels and 256 output channels to compress feature channels. Again, we apply the instance normalization layer and LeakyReLU activation function. Same as (b), we add a skip connection between the result feature map and the previous concatenated feature map by addition operation. The concatenated feature map will be transformed to the consistent number of channels by convolutional layer with 896 input channels and 256 output channels. Thereby the final feature map has stride 8 and 256 channels.
(f) Refine 8. We refine the depth map from (d) and the feature map from (e). The procedure is the same as (d), except for the number of channels in convolutional layers are adapted accordingly. Thereby we introduce the pipeline briefly. We first upsample the depth map from (d) into stride 8 to concatenate with the feature map. Thereby we obtain a new feature map with 256 + 128 = 384 channels. Then, we apply a convolutional layer with 384 input channels and 384 output channels to fuse the information. The feature map is then transformed by LeakyReLU activation function. Next, we utilize a convolutional layer with 384 input channels and 128 output channels to predict the refined depth map. Similarly, another convolutional layer with 384 input channels and 256 output channels applied to predict the refined feature map.
(g) Upsampling 8->4. We upsample the feature map from (f) to stride 4. The procedure is the same as (e) except for the number of channels. More specifically, we first upsample the feature map and concatenate with the feature map from the encoder with consistent stride, obtaining feature map with 256 + 192 448 channels and stride 4. Then a convolutional layer with 448 input channels, 448 output channels, and 4 groups is applied to fuse the information. Then the result feature map is transformed by the instance normalization and LeakyReLU activation function. Next, we apply another convolutional layer with 448 input channels and 64 output channels to compress the feature channels. Again, the instance normalization layer and LeakyReLU function is applied. In the end, we add a skip connection between the final feature map and the previous concatenated feature map.
(h) Refine 4. We refine the depth map from (l) and the feature map from (g) following the same pipeline (d). We first upsample the depth map to stride 4, and concatenate with the feature map, obtaining a new feature map with 64 + 128 = 192 channels and stride 4. Next, we apply a convolutional layer with 192 input channels and 192 output channels to fuse the information, and a LeakyReLU function to transform the feature map. In the end, we utilize another convolutional layer with 192 input channels and 128 output channels to predict the new depth map. The output is a depth map with 128 channels and stride 4.
(i Metric Layer. We take the feature map with stride 32 from the encoder input. We first apply global max pooling to compress the feature map into a vector with 1536 channels. Then we apply a fully-connected layer with 1536 input units and 384 output units to compress the channels. Next apply a LeakyReLU function. In the end, we utilize another fully-connected layer with 384 input units and 2 output units to regress the scale and shift.
16

---

Published as a conference paper at ICLR 2023
Table Comparison with Ramamonjisoa (2020), Cheng et al. (2018) and (2017) on NYU Depth
V2 test set (Silberman et al., 2012).
Method
Sll.og
Ahs Rel
RMSJ
RMS log
A1
Rumamonjiso (2020)
8.655
0.929
Cheng et al. (2018)
8.640
0.089
0.317
0.112
0.930
0.991
et al. (2017) (end-to-cad)
8.557
0,315
0.112
0.931
0.991
Li al. (2017) (optimizatioa)
8.574
0,089
0.316
0.112
0.930
0.991
Ours
8.198
0.086
0,304
0.108
0.937
0.992
Table 10: Impact of resolution. We evaluate the performance of V-layer when operating on feature maps of
stride 16 and 8, respectively.
Stride
FPST
SiLog
Abs Rel
RMST
RMS logb
01 8u
7.032
8.198 8.149
0.086 0.086
0.304 0.306
0.108 0.108
0.937 0.936
0.992
8
2.597
0.992
(i Final Prediction. We take the output from (f), (h) and (i) as input. We upsample all the depth maps to stride via bi-linear interpolation, and fuse the depth maps into single depth map by the addition operation and convolutional layer with 128 input channels end, add the depth map with the shift and multiply with the scale from (i), respectively.
output channel. In
A.S STATISTICAL EVALUATION WITH OTHER METHODS
In this section, we compare with Ramamonjisoa et al. (2020), Cheng et al. (2018), and Li et (2017) on NYU Depth V2 (Silberman et al., 2012). More specifically, apply their methods to refine the final predicted depth map. The displacement field, the affinity matrix, and the depth gradient are predicted from the refined feature map. which is the output of the last refinement module. For Cheng et al. (2018) we adopt their default
The kernel size is 3, and the number of iterations is 24. Li al. (2017) proposes the end-to-end fusion strategy and the optimization-base fusion strategy. For the end-to-end fusion strategy, we employ three convolutional layers with kernel size 5, and 16 channels. For the optimization- based fusion strategy, we use Adam optimizer. The learning rate is 0.01 and the number of iterations is 100. The results are shown in Tab. (9). Our method achieves better performance.
A.6 ABLATION ON RESOLUTION
In this part, we evaluate the performance of the V-layer when applying on the feature map with stride We achieve this by dividing the feature map into 4 rectangle sub-regions with equal area (top left, top right, bottom left and bottom right). We apply V-layer each sub-region. The predicted depth maps from the sub-regions will be stacked to recover the original resolution. The results are shown in (10. From stride 16 to 8, the improvement of accuracy is marginal. However, the inference speed (FPS) is significantly slow down.
A.7 ABLATION ON KITTI AND SUN RGB-D
To further demonstrate the benefits of our V-layer KITTI Eigen split (Geiger et al., 2012; Eigen et al., 2014) and SUN RGB-D test set (Song et al., 2015), we present the accuracy the predicted depth map when replacing the V-layer with a single convolutional layer. As shown in (11, V-layer can improve the depth map accuracy on both KITTI and SUN RGB-D.
Table 11: Benefit of V-layer. We replace the proposed V-layer with single convolutional layer, and evaluate the predicted depth map accuracy.
Dataset
Layer
SlLog
RMS.
RMS logi
KITTI
Convolution
6.996
0.052
0.217
0.079
0.975
0.997
V-Layer
6.817
0.050
0.209
0.977
0.997
Convolution
13.172
0.098
0.309
0.132
0.924
0.981
V-Layer
12.596
0.094
0.299
0.127
0.929
0.983
17

---

Published as a conference paper at ICLR 2023
(a) Image
(b) Confidence Weight
(c) Difference Map
(d) Depth Map
Figure 5: Visualization of V-layer Prediction. We visualize the confidence weight Eg, the differ-
ence map r and the depth map Zu from the V-layer when predicting on NYU Depth V2 test set.
Table 12:
Comparison with AdaBins and NeWCRFs NYU test set. Eigen train without fine-tur munit on NYU.
All methods are trained KITTI
SlL 4
4.
RMSJ
log
AduBins(Bull al. 2021) BS-Net-B+Vii-mmm
28.147
0.251
0.753
0.286
0.614
0.867
NeWCRFs (Yuan al., 2022)
Swin
21.138
0.173
0.551
0.213
0.755
0.934
18,900
A.8 MORE GENERALIZATION EXPERIMENTS
We further evaluated the generalization performance when training on KITTI and test NYU, or training on NYU and test on KITTI. The results are shown in Tab.(12) and T1b. Our method achieves better generalization performance in both settings.
Table 13: Comparison with AdaBins and NeWCRFs on KITTI Eigen test set. All methods trained on
NYU Depth V2 train set without fine-tuning KITTI Eigen.
Method
Backbone
SlLog
Abs Rel
RMS]
RMS log4
AdaBins(Rhat al, 2021) NeWCREs (Yunn al., 2022)
568871
0.350
7.221
0.579
0.434
0.744
Lwin
54.460
0.268
6,246
0.550
0.512
0.833
Swin-L 42.105
0.221
5.360
0.426
0.598
A.9 SPARSE BROUNN--RUTHE DEPTH MAP
KITTI provides sparse LiDAR measurements as the ground truth (official ground truth has been inpainted extent). Yet, our method could work well in such cases where sparse LiDAR mea- surements are known. Our algorithm computes the depth map from the gradient predicted by the network in differentiable way. Thereby, for such case, when apply sparse supervision on the computed depth map (note that the loss function is used for valid measurements only). the error sig- nal from the loss function will be back-propagated the predicted gradient supervise the network
18

---

Published as a conference paper at ICLR 2023
4
(d)
(e)
Figure 6: Qualitative comparison on KITTI Eigen split (Eigen et al., 2014). For each column, from top to bottom we present the input image, the prediction from AdaBins (Bhat et al., 2021), NeWCRFF (Yuan et al., 2022), and our framework respectively.
to learn gradient clues. We demonstrated the effectiveness of our algorithm in Tab.(2) and Tab.(3) for such cases.
A.10 MORE VISUALIZATION
We visualize the confidence weight map Σ., the difference map Ia. and the depth map Z., from the V-layer in Fig.5. We observe that the depth value of a pixel shows correlation with respect the image coordinates of the pixel. For example, in the last example in Fig.5, for different pixels at the door, the depth values are usually different but the first-order difference are approximately the same. This observation shows that the difference map might be easier to predict than the depth map.
A.11 QUALITATIVE RESULTS
We provide more qualitative results on KITTI Eigen split (Eigen al., 2014). SUN RGB-D (Song et al., 2015), and NYU Depth V2 (Silberman et al., 2012) in Fig.6, Fig.7 and Fig.8, respectively. Our framework predicts more accurate shapes and preserves the high-frequency scene information.
19

---

Published as a conference paper at ICLR 2023
(a) Image
(b) AdaBins
(c) NeWCREs
(d) Ours
Figure 7: Qualitative comparison with AdaBins (Bhat et al., 2021), NeWCRFs (Yuan et al., 2022) on SUN RGB-D test set (Song et al., 2015). All the models pre-trained on NYU Depth V2 (Sil- berman et al., 2012) training set.
20

---

Published as a conference paper at ICLR 2023
AL
(a) Image
(b) AdaBins
(c) NeWCRFs
(d) Ours
(c) Ground Truth
Figure 8: Qualitative comparison with AdaBins (Bhat et al., 2021), NeWCRFs (Yuan et al., 2022)
on NYU Depth V2 test set (Silberman et al., 2012). The ground-truth depth map are in-painted for visualization.
21


Reviews:
[{"review_id": "P7AF0pLN3m", "reviewer": "Reviewer_qKm4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes VA-DepthNet to solve single image depth prediction problem by exploiting classical first-order variational constraints. The proposed network disentangles the absolute scale from the metric depth and models unscaled depth map as the optimal solution to the pixel-level depth gradiant. The network focuses on the first order differences of the scene rather than pixel-wise metric depth learning. It improves the performance of depth learning in a large margin.", "summary_of_the_review": "This work lays a solid contribution towards the field of single image depth prediction.", "strength_and_weaknesses": "This work lays a solid contribution towards the field of single image depth prediction by introducing variational first-order constraints, showing state-of-the-art performance on mainstream public datasets. The idea is novel and introducing variational priors can be useful to a number of areas, e.g. depth prediction, scene reconstruction.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "GtIanipV_J5", "reviewer": "Reviewer_PKfN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the single image depth estimation method with variational constraints via first-order difference. This paper proposes V-layer which computes the depth difference map and the weight map, which will be utilized to obtain the initial depth map. The initial depth map will be upsampled and refined through the upsampling and refinement module. Finally, the metric scale depth is computed by the metric layer.", "summary_of_the_review": "Overall, the proposed method achieves better performance than the conventional method. The dense ablation study demonstrates the effectiveness of the proposed method. However, for me, it is hard to get the design intuition of the proposed method, especially why the variational constraints via first-order difference bring performance improvement. I hope this part is clearly described in the rebuttal.", "strength_and_weaknesses": "1. Strength\n- The proposed method outperforms very recent state-of-the-art methods in various datasets. It also achieves 2nd place in the KITTI depth benchmark.\n- The dense ablation study shows the effectiveness of the V-layer and the refinement step.\n\n2. Weakness\n- It is not clearly described why the variational constraints via first-order difference are helpful for accurate depth estimation. The introduction section is more likely to be a related work section, which should describe why the proposed method (variational constraints) is required to improve the quality of the depth map. \n- Figure 5 shows the evaluation of different backbones. Both Swin-S (w/o) and ConvNeXt-S (w/o) achieve 9.5, but the graph shows the error of ConvNext-S is higher than Swin-S.\n- In table 1,2, it would be good to report the number of parameters or the backbone network (ResNet, Swin-S, Swin-L, etc) because the accuracy of the depth map highly depends on the backbone network. The number of parameters is reported in table 8, but the numbers are slightly different from the numbers in Table 1. \n- This paper argues that the proposed method generalizes well to unseen datasets. And, the evaluations of the NYU-trained model on the SUN3D datasets are provided. The scene properties of NYU and SUN3D datasets are similar, so the evaluations on more challenging variations should be provided such as the KITTI-trained model tested on NYU, etc.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FFpGtlSduA9", "reviewer": "Reviewer_3Fva", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new network for single view depth estimation. The core technical part is a V-layer that predicts depth gradients and feature weights and solves a refined depth map. Finally, the refined depth is further refined (by black-box operators) to a larger resolution and re-scaled/shifted to the final prediction. Experimental results on NYUd2, KITTI (official and eigen splits) and SUNRGBD set a clear new state-of-the-art. Codes are promised.", "summary_of_the_review": "Generally, I think the paper should be accepted but still has issues (see weaknesses) to be addressed. I am now voting a 6 but I can vote a 8 if convinced.", "strength_and_weaknesses": "Strengths:\n+ Well-written and well-organized.\n+ Clear new state-of-the-art results on an important problem.\n\nWeaknesses:\n- My first major criticism is that the method should be compared with other methods that implements the same idea. In Table.5, we can see that the V-layer out-performs convolution and self-attention. This is not enough. The method should be compared with [A][B][C] and demonstrate why V-layer works and the former formulations do not.\n[A] Predicting sharp and accurate occlusion boundaries in monocular depth estimation using displacement fields, CVPR 2020\n[B] Depth estimation via affinity learned with convolutional spatial propagation network, ECCV 2018\n[C] A Two-Streamed Network for Estimating Fine-Scaled Depth Maps From Single RGB Images, ICCV 2017\n- The motivation and impact of the Conv layer in Equation.8 is not clear. Although I understand that in end-to-end deep models we can always some learnable layers that mysteriously improves performance, this one needs a clearer justification because it belongs to the core technical module.\n- Minor: Claiming MIDAS uses external is unfair as it is evaluated in a zero-shot setting. This can be misleading for future papers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5AYz6u1qYK", "reviewer": "Reviewer_K2Ky", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose an algorithm for single-image depth prediction. The key contribution of the algorithm is the V-layer, which utilizes the depth gradient to make a more accurate depth prediction. It takes the gradients in the two-axis directions and their reliability as input and obtains the depth in a linear algebraic way. The resulting low-resolution depth becomes the final depth after several upsamples and refinements. Experimental results show that the proposed method shows SOTA performance and that V-layer contributes to the performance improvement.", "summary_of_the_review": "Overall, I think this paper very interesting. This paper is technically valid and has novel parts, but the parts I mentioned in the shortcomings should be revise. I think this paper is on the borderline.", "strength_and_weaknesses": "This article is well-written and easy to understand. The algorithm's approach is interesting and plausible. Given that only a single image is given, the prediction of depth is a very ambiguous process, so it makes sense to use more solid clues (depth gradients in this paper). The process of restoring depth by building an overdetermined system is novel.\n\nDespite the above advantages, the novelty of this paper should be re-examined. In the last paragraph of page 2 of the paper, the authors claimed that the proposed algorithm is the first attempt to predict the first-order difference of the scene. However, a prior work (1) also predicts the first-order difference and uses them to reconstruct depth, similar to this paper's motivation. Also, (2) exploits more diverse depth derivatives for depth prediction. Related work and contributions should be revised.\n\n(1) Monocular depth estimation using relative depth maps, CVPR 2019\n\n(2) Multi-Loss Rebalancing Algorithm for Monocular Depth Estimation, ECCV 2020\n\n\nMany depth capture devices (e.g., Lidar) can only capture depth of sparse points. On the other hand, gradient is a clue that can be used if we know the depths of all adjacent points. I think this aspect can limit the usefulness of the algorithm. Do the authors have a way around these problems?\n\nThe V-layer seems to work at a much smaller resolution (probably 1/16 or 1/32) instead of the original size resolution. However, this method results in depth prediction using gradients between regions farther away instead of two adjacent pixels. This may be a disadvantage for predicting high-frequency detail. Do the authors have any insight or empirical results regarding the resolution of the v-layer?\n\nA more detailed description of the proposed network structure is needed for reproducibility. It would be nice if detailed layer composition was provided (even in appendix format).\n\n---\nAfter reading the reviews of other PCs and answers from authors,\n\nI raise my primary score to 6.\nThe authors' answers alleviated most of my concerns.\nNotably, Q3 and A3 will interest many researchers in this field. Including it in the paper or appendix would be nice.\nThanks to the authors for their hard work.\n\n---", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%