DCDepth: Progressive Monocular Depth Estimation in Discrete Cosine Domain
Authors: Kun Wang
Zhiqiang Yan
Junkai Fan
Wanlu Zhu
Xiang Li
Jun Li
Jian Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Monocular depth estimation
Discrete cosine transform
Deep learning
Primary area: machine_vision

Abstract: In this paper, we introduce DCDepth, a novel framework for the long-standing monocular depth estimation task. Moving beyond conventional pixel-wise depth estimation in the spatial domain, our approach estimates the frequency coefficients of depth patches after transforming them into the discrete cosine domain. This unique formulation allows for the modeling of local depth correlations within each patch. Crucially, the frequency transformation segregates the depth information into various frequency components, with low-frequency components encapsulating the core scene structure and high-frequency components detailing the finer aspects. This decomposition forms the basis of our progressive strategy, which begins with the prediction of low-frequency components to establish a global scene context, followed by successive refinement of local details through the prediction of higher-frequency components. We conduct comprehensive experiments on NYU-Depth-V2, TOFDC, and KITTI datasets, and demonstrate the state-of-the-art performance of DCDepth. Code is available at https://github.com/w2kun/DCDepth.

DCDepth: Progressive Monocular Depth Estimation in Discrete Cosine Domain
Kun Wang', Zhiqiang Yan', Junkai Fan', Wanlu Zhu', Xiang Li", Jun Li and Jian Yang" 'PAA Lab, Nanjing University of Science and Technology, China Nannaai University, China
Abstract
In this paper, we introduce DCDepth, a novel framework for the long-standing monocular depth estimation task. Moving beyond conventional pixel-wise depth estimation in the spatial domain, our approach estimates the frequency coefficients of depth patches after transforming them into the discrete cosine domain. This unique formulation allows for the modeling of local depth correlations within each patch. Crucially, the frequency transformation segregates the depth information into various frequency components, with low-frequency components encapsulating the core scene structure and high-frequency components detailing the finer aspects. This decomposition forms the basis of our progressive strategy, which begins with the prediction of low-frequency components to establish a global scene context, followed by successive refinement of local details through the prediction of higher- frequency components. We conduct comprehensive experiments on NYU-Depth- V2, TOFDC, and KITTI datasets, and demonstrate the state-of-the-art performance of DCDepth. Code is available at fhtttps:: //gtthub.. com/ wwkun/ Wtmm///n/DDDDeppppp th
Introduction
Monocular Depth Estimation (MDE) cornerstone topic within computer vision communities, tasked with predicting the distance- or depth-of of each pixel's corresponding object from the camera based solely on single image. As pivotal technology for interpreting 3D scenes from 2D representa- tions, MDE is extensively applied across various fields such as autonomous driving, robotics, and etc. However, MDE is challenged by the inherent ill-posed nature of inferring 3D structures from 2D images, making it particularly daunting task for traditional method-
3D modeling 45 49
ologies, which often hinge on particular physical assumptions or parametric models 440 59 31 32).
Over the past decade, the field of computer vision has witnessed substantial surge in the integration of deep learning techniques. Many studies have endeavored harness the robust learning capabilities of end-to-end deep neural networks for MDE task, propelling the estimation accuracy to new heights. Researchers have investigated a variety of methodologies, including regression- based
(5).
classification-bet based 12). and clasiffication-regesss on a per-pereeeeee per-pixel basis within the spatial domain. Despite these significant strides in enhancing accuracy. current methods encounter two primary limitations: the first is the tendency to predict depth for individual pixels in isolation, thus neglecting the crucial local inter-pixel correlations. The second limitation is the reliance on singular forward estimation process, which may not sufficiently capture the complexities of 3D scene structures, thereby constraining their predictive performance.
based approaches [3] 20]. to predict depth
To address the identified limitations, we propose to transfer depth estimation from the spatial domain the frequency domain. Instead of directly predicting metric depth values, our method focuses on estimating the frequency coefficients of depth patches transformed using the Discrete Cosine
Cooresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

S
Higghr Frequescy
DCT
Final Prediction
Inpurt Image
Step
Step
Step
Step N
Progressive Predictions
Instal Predactions
Retined Predictions
Zero Padding
Reshape
Figure 1: Progressive estimation scheme. For input image with size H x W. DCDepth estimates the DCT coefficients for each S x S depth patches. The prediction follows global-to-loce strategy, starting with the imitial estimation lower-frequency components to capture the global scene structure. Subsequently, higher-frequency components are estimated to enhance the local details, while the lower-frequency estimates are refined. The estimation is carried out at H spatial-domain estimation is achieved through inverse DCT.
S x S resolution, and
Transform (DCT) 2 [] This strategy offers dual benefits: firstly, the DCT's basis functions in- herently capture the inter-pixel correlations within depth patches, thereby facilitating the model's learning of local structures. Secondly, the DCT decomposes depth information into distinet fre- quency components, where low-frequency components reflect the overall scene architecture, and high-frequency components capture intricate local details. This dichotomy underpins our progressive estimation methodology, which commences with the prediction of low-frequency coefficients to grasp the macroscopic scene layout, subsequently refining the local geometries by inferring higher- frequency coefficients predicated on previous predictions. The spatial depth map is then accurately reconstructed via the inverse DCT. We illustrate this progress in Fig. [ To implement our pro- gressive estimation, we introduce a Progressive Prediction Head (PPH) that conditions on previous predictions from both spatial and frequency domains, and facilitates the sequential prediction of gigher-frequeng, components using GRU-based mechanism. Furthermore, recognizing the DCT's energy compaction property-indicative of the concentration of signal data within low-frequency components-we introduce DCT-inspired downsampling technique to mitigate information loss during the downsampling process. This technique is embedded within a Pyramid Feature Fusion (PFF) module, ensuring effective fusion of multi-scal. image features for accurate depth estimation.
Our contributions can be succinetly summarized in three key aspects:
To the best of our knowledge, we are the first to formulate MDE progressive regression task in the discrete cosine domain. Our proposed method not only models local correlations effeetivvly but also enables global-to-local depth estimation.
We introduce a framework called DCDepth, comprising two novel modules: the PPH module progressively estimates figherrfrequency coefficients based previous predictions, and the PFF module incorporates DCT-based downsampling technique to mitigate information loss during downsampling and ensures effective integration of multi-scale features.
We evaluate our approach through comprehensive experiments on NYU-Depth-VV V2 [6 TOFDC [2]. and KITTI [33 datasets. The results demonstrate the superior performance of DCDepth compared to existing state-of-the-art methods.
2 Related Work
Monocular Depth Estimation (MDE) remains central theme in computer vision, essential for translating 2D imagery into 3D scene geometry. The evolution of MDE has been markedly influenced by the integration of deep neural networks. foundational advancement was introduced by Eigen et al. [1I. who developed multi-scale deep convolutional network architecture, comprising global network for coarse depth prediction and local network for refinement. They also introduced scale-invariant loss function address the scale ambiguity challenge inherent in MDE. Building

---

57 53 have adopted und-to-ena regression approaches with deep convolutional networks to further tackle MDE's challenges.
this, subsequent researches 56
56 42 51
50
However, inferring depth from a single image is intrinsically problematic due to the countless image. To mitigate this, additional information and constraints have been incorporated into the MDE task, such as semantics [44 60) and surface normals Further enhancements in depth estimation accuracy have been achieved through attention multivariate gaussian modeling (21], internal discretization technique 22 In contrast to the regression- basedd approach, some works [2 have conceptualized MDE as a classification task. estimating the probability distribution of depth values. Yet, these methods often produce discontinuities due to discrete depth outputs. To overcome this, alternative strategies [3] 20 g 34] have combined classification and regression formulations, learning probabilistic distributions and employing linear combinations with depth candidates for final depth predictions. Our methodology diverges from these paradigms by progressively estimating frequency coefficients for depth patches after their transformation into the discrete cosine domain. This approach not only enhances computational efficiency but also achieves state-of-the performance.
potential depth maps that can correspond
28 33
mechanisms [4 47 30
and pretraining
48
3 Method
In this section, we introduce our progressive depth estimation framework, DCDepth. We begin by providing an overview of the 2D Discrete Cosine Transform (DCT) as essential background knowledge. Subsequently, we delve into the progressive estimation scheme and elaborate on the network architecture. Finally, we present the loss function employed for training our model.
3.1 Reviewing 2D Discrete Cosine Transform
The 2D DCT is mathematical technique used decompose 2D discrete signals, such as depth maps and feature maps, into a sum of cosine basis functions with varying frequencies. The basis functions defined as follows:
1 2 2+)
Big = a(u)a(v) cos
W i
u cos
(I)
2
H
where u E [0, 1] and v E [0, - represent the frequency indices, i E W- and j E [0,
1]
denote the signal indices, and W and H/ indicate the input resolution. The terms (u) and (v) correspond to normalization factors. The forward process of 2D DCT, denoted as T(-), transforms the input signal x R H in the spatial domain to the frequency spectrum f == T=x(,,, fER "xW and can be expressed as:
Su. = -
Kig BigBuut B
==0
The resulting f is a matrix with the same size as the input x, with low-frequency components located the top-left corner and high-frequency components near the bottom-right corner. The upper left one with zero frequency is called the DC components, and the remains are AC components. Low-frequency components typically characterize smooth regions, while high-frequency components capture edges or fine details where signal values change rapidly. The inverse 2D DCT, denoted as (-),,,, performs the reverse operation by transforming the frequency spectrum back the spatial domain = T T (f), and can be formulated as:
Fi ΣΣ
Bu U.0*
The DCT has two desirable advantages. Firstly, it operates in the real number domain, simplifying the data processing, Secondly, it exhibits superior energy compaction properties by concentrating the majority of information within small number of low-frequency components.
3.2 Progressive Estimation in Discrete Cosine Domain
Estimating depth from a single image remains challenging task, particularly for scenes with intricate geometry. To tackle this, propose progressive method based on 2D DCT to estimate scene depth
3

---

Depth
Figure 2: Evolution of intermediate depth estimations. We report several intermediate depth estimation results to illustrate our progressive estimation scheme.
progressively from a global perspective down to local details. The entire process is illustrated in Fig. We denote the input image as I E 2x Our proposed method, symbolized as (-(.), predicts the frequency coefficients C E S x for non-overlapping depth patches P RSxS, where set to 8 in our framework. These coefficients are subsequently transformed back to the spatial domain D x using the inverse 2D DCT, as expressed by
1
D = T=
(4)
The separation low- and segg-frryeency components depth map effectively divides scene into overall structures with gradual depth changes and local details with sharp depth transitions. This frequency characteristic enables us to break down the challenging MDE task into multiple prediction stages, progressing from simpler to more complex predictions. Initially, the DC coefficient Co is predicted, establishing foundational depth context. Subsequently, the AC coefficients {C, " are iteratively estimated in ascending frequency order. During the inverse transformation to the spatial domain, any coefficients yet to be predicted are padded with zeros. In each iterative step we not only predict higher- freeurncr components but also refine the preceding frequency predictions
(5) by estimating a correction term AC* To reduce the required iterations for estimating all S coef- ficients, utilize the energy compaction property of DCT, and partition the frequency spectrum into subgroups along the subdiagonal, yielding 25 subgroups {g: 128-1 By merging the high-freque gigh-frequency subgroups, we further streamline the iterative process. This grouping strategy ensures that lower-frequency groups contain fewer components necessitating more prediction steps, while freuunccy groups encompass a larger number of components requiring fewer steps. The intermediate depth maps are provided in Fig. to elucidate the step-by-step prediction process.
C C*-1 + AC*
3.3 DCDepth Architecture
Overview We present the comprehensive framework of DCDepth in Fig. 3 which comprise four key components: an image encoder, Pyramid Feature Fusion (PFF) module, decoder, and Progressive Prediction Head (PPH). The image encoder acts as a robust feature extractor capturing image features F {Fo. F2, at varying resolutions of 1/4, 1/s, 1/16, and 1/32 relative to the imput image size. These multi-scale features are ad advantageous as the shallow features contain texture-related details. while the deep features hold global and semantic information essential for scene understanding. The PFF module, symbolized as T(.), is introduced to effectively amalgamate these features, yielding comprehensive integrated feature representation F I(F). The decoder, denoted as (..), consists of three neural CRF (58) modules and two PixelShuffle 355 modules. This configuration processes and upscales F to F D(F),, achieving 1/s of the original resolution. The PPH performs estimations the same resolution as F. It begins by down-sampling Fo half' its resolution using the proposed DCT-based downsampling. This down-sampled feature is then concatenated with F. forming the initial hidden state for the progressive estimation.
Pyramid Feature Fusion Module The primary objective of PFF is to harness the wealth of information embedded in the multi-scale image features, thereby creating a more comprehensive and enriched feature representation conducive scene understanding. The layout of PFF is depicted in the left box of Fig. 3 Effective feature aggregation necessitates proficient downsampling strategy to mitigate information loss, especially when downscaling at larger magnifications. To address this, we introduce novel DCT--aase downsampling strategy engineered to minimize information loss during downsampling. The operational procedure of this strategy is elucidated the bottom-left corner of Fig. 3. Consider a feature map F E &gCC xxX slated for downsampling by factor of R. We begin

---

PTU
Begeh
Predurtia
Produrtio
Preturte
Producia
Sputial Enewler
Inewkr
Eaeeder
Eacadir
Banuder
Tarader
Coant
Caarst
Crast
Blad
Casy
Dorasingias
Charail Lttratie
FFM liral
PPhi liral
PPS linat
1 Cuae
(1.
Aliantes
1/32
1736
178
Friqanky
Iarenr DC1
Inveav PCT
Javene DCT
Sr Groep derieo
25 Depthins Conrclatke
Figure DCDepth framework overview. The DCT-bbsed downsampling strategy is shown at the bottom-left corner, where R and r denote for downsampling factor and channel reduction rate, respectively. The central section details the iterative process of PPH, with N indicating the number of iterative steps. The frequency encoder utilized by PPH is illustrated at the right box.
by partitioning F into patches P E xC Each channel of P is then individually subjected too transform the feature maps into the frequency domain. Leveraging the energy compaction property of the DCT, the key information within F is condensed into few dominant frequency components characterized by large absolute values. This compression enables us to selectively reduce the number of channels from C x R² to C' x R² with reduction rate of r via x 1 convolutions configured with groups set C. The squeezed feature maps are then consolidated through sequence of operations involving a x 1 convolution followed by a 5 x 5 depth-wise convolution, culminating in the generation of the final output featuring C' channels and reduced spatial resolution.
The PPH, as depicted in the middle segment of Fig. 3 incorporates two specialized encoders: E.) for spatial-domain inputs and Ey(-) for frequency- inputs. The spatial encoder, composed of three convolutional layers with a stride of 2, convolves and downsamples the spatial-domale input D. producing feature map at 1/8 of the original resolution. The architecture of (-) is outlined in the right box of Fig. 3 For frequency input C R where L signifies the number of valid frequency components, we first split them into L chunks with shape H W. Each chunk is then processed through three convolutional layers with Swish activation (29 to extract features of dimensions C x H x W for each frequency component. Given the variability the number of valid frequency components across different iterative steps, we employ cross-attention 4I [0] mechanism to merge information from the various frequency components. A is introduced compile information from individual frequency components at each pixel location, yielding feature outputs of shape C' x H x W and effectively compressing the dimension L. The PPH operates iteratively. utilizing Gated Recurrent Unit (GRU) 7 39], denoted as G(.,) -). to encode the historical estimation states
Progressive Prediction Head
learnable regregation token of dimensions
Hi- G(E, (C
prior to iterative step i. The hidden state H is then projected to the coefficient output by a Pyramid Pooling Module (PPM) 61 to aggregate global context, followed by a linear projection.
3.4 Loss Function
We employ the scaled scale-invariant loss 17 each iterative step i against the ground truth depth map D. The loss function is formulated as:
to calibrate the model's s depth estimations D at
L =
M
d
d,)
M
M
where d = D. D, N denotes the number of iterative steps, and M represents the number of valid depth values. We consistently set a 10, 3 0.8 and À 0.85 across all experiments. The presence
5

---

Sq Rel
RMSE
log
1.25
1.25
1
1.25"
DORN
12
ResNet-101
0.115
0,509
0.051
0,828
0.965
0.992
VNL
55
ResNet-101
0.108
0.416
0.048
0,875
0.976
0.994
BTS
IN
DenseNet-161 161
0.110
0,066
0.392
0.047
0.885
0.978
0.994
ASNDepth
HRNet-48
TransDepth
R-50-VIT-B/16
0.365
AdaBins
E-B5+mini-We VIT
0.997
NeWCRES
Swin-Large
0.095
0.045
0.334
0.041
0,922
0.992
0.998
Former
Swin-Large
0.094
0.330
0.040
0.925
0,989
0.997
PixeiFormer
Swin-Large
0.090
0.322
0.039
0,929
0.991
0.998
IEBins
34
Swin-Large
0.087
0.040
0.314
0.038
0.936
0.992
0.998
MG-Depth
21
Swin-Large
0.087
0.311
0.933
NDDepth 33
Swin-Large
0.087
0041
0,311
0.038
0.936
0.991
0.998
VA-DepthNet [
Swin-Large
0.086
0.039
0.304
0.037
0.937
0.992
0.998
Ours
Swin-Large
0.085
0.039
0.304
0.037
0.940
0.992
0.998
Table 1: Quantitative depth comparison on NYU-Depth-V. V2 dataset. The maximum depth is capped at 10 meters. R-50 and E-B5 represent for ResNet-50 [15] and EfficientNet-Bi (38) respectively. '-' means not applicable. The best result is bold, and the second is underlined.
Image
BTS
AdaHlins
NeWCRES
IEBins
Ours
AR
Figure 4: Qualitative depth comparison on the NYU-Depth-V? dataset. The white boxes highlight the regions where our method achieves more accurate predictions.
of missing values in the depth ground truth can render the model's frequency-domall predictions inadequately supervised. To mitigate this, introduce two regularization terms. Specifically, to enforce the sparsity of high-frequency coefficients, we define the frequency regularization loss
Ly -E
(8)
where fu, is the frequency coefficient indexed by (u, v). and is to 1.2. Additionally, we incorporate smoothness term to promote the smoothness D:
L | e 10. + |0, - 18,
where 0, and 0, represent image gradient along horizontal and vertical axes, respectively, and denote the absolute value function. The final loss is the weighted summation of these three loss terms.
4 Experiment
In this section, we evaluate DCDepth by conducting a comparative analysis with established method- ologies. We commence by delineating the datasets and evaluation metrics employed in our evaluation. Subsequently, we detail the implementation specifics that underpin our experiments. Concluding this section, we demonstrate the efficacy of the proposed modules via extensive ablation studies.
4.1 Dataset and Evaluation Metric
Dataset We evaluate our method on three datasets that covers diverse array of indoor outdoor scenes. (1) NYU-Depth-v2 v2 (36] is centered indoor environments and consists of RGB-D images

---

Method
Sq Rel
log
1,25"
BTS
[t]
DenseNet-Ioe
0.407
0.082
0.998
0.567
0.985
0.998
1,000
AdaBins
E-B5+mini-V ViT
0.279
0.044
0.729
0.462
0.990
0.998
1,000
NeWCRES
58
Large
0.533
0.244
1.004
0.792
0.956
0.976
0.988
PiselFormer
1
Swin Large
0.534
0.230
1.076
0.782
0.957
0.979
0.991
VA-DepthNet
M
Swin-Large
0.234
0.029
0.619
0.373
0.996
0.999
1,000
IEBins
Swin-Large
0.528
0.238
0.999
0.750
0.956
0.976
0.988
Ours
Swin-Large
0.188
0.027
0.565
0.352
0.998
0.999
1.000
Table Quantitative depth comparison on TOFDC dataset. The maximum depth is capped at meters. The first four error metrics are multiplied by 10 for presentation.
Imago
BTS
AdaBins
NAWCRES
TEBins
Ours
Figure 5: Qualitative depth comparison on the TOFDC dataset.
captured with Microsoft Kinect sensor. The settings span various indoor scenes such as bedrooms, offices, and classrooms. The images in this dataset are presented at resolution of 640 480. We follow the data split as outlined in BTS [17], featuring 24231 training images and 654 test images. (2) TOFDC [52) collected using mobile phone paired with a lightweight Time-of-Flight (ToF) camera, capturing a wide array of subjects like flowers, human figures, and toys under different scenes and lighting conditions. The dataset divided into 10,000 training samples and 560 testing samples, with images at resolution of 512 x 384. (3) KITTI [3] well-known outdoor dataset that features RGB images coupled with sparse depth maps obtained from laser scanner mounted on car. The images in this dataset have resolution of 1216 352. We utilize both the Eigen split [ and the official split for our analysis. The Eigen split comprises 23158 training images and 697 test images, while the official split includes 42949 training images and 500 test images.
Metrics Consistent with prior works 58 3 34], we utilize a selection of well-established metrics to provide comprehensive evaluation. The key metries include: relative absolute eπror (Abs Rel), relative squared error (Sq Rel), root mean squared error (RMSE), absolute logarithmic error (log io). root mean squared logarithmic error (RMSE log), inverse root mean squared error (iRMSE) and threshold accuracy (5 < 1.25, 1.252, and < 1.253). Please refer to the appendix for details.
4.2 Implementation Detail
The DCDepth is implemented using Pytorch library 25 and is trained with batch size of 8 on four NVIDIA RTX-4090 GPUs with data-distributed parallel computing. Our method trained on NYU-Depth-V3? dataset for 20 epochs, TOFDC dataset for 25 epochs, KITTI eigen split for 20 epochs and KITTI official split for 12 epochs. The optimization objective our method is combination of the scale-invariant log loss La. the frequency regularization L and the smoothness regularization Lg, weighted by two scalar weights a and 3:
L La + Ls + 3 L..
(10)
For the NYU-Depth-V V2 and TOFDC datasets, these two weights are set to 2 x 10-3 and 0.0, respectively, while for the KITTI dataset, both weights are set to 5 10-3 We opt for the Adam optimizer [16] and leverage the OneCycle learning rate scheduler 337 The learning rate schedule entails an initial increase from 10 to 10-4 during the first 2 epochs, followed by subsequent decrease to 5 X 10-6 using a cosine annealing strategy. To enhance generalization and mitigate overfitting, we integrate various data augmentation techniques into the training pipeline, including random horizontal flips, random rotations, random color jitter, and random image filtering. For feature extraction from images, we incorporate Swin-Transformer architecture (33 pretrained on the ImageNet dataset 8 the image encoder. To reduce the iteration steps necessitated for spectrum

---

Method
Sq
log
1,25"
1.25
DORN
ResNet-101 101
0.072
0.307
2.727
0.120
0.932
0.984
0.994
VNL
33
ResNet-01
0.072
3.258
0.117
0.938
0,990
0.998
BTS
17
DenseNet- 161
0.060
0.249
2.798
0.096
0.955
0.993
0.998
Trans
R-50-ViT-B/16
0.064
0.252
2.755
0.098
0.956
0.994
0.909
AdaBins
E-B5 VIT
0.058
0.190
2,360
0.088
0.964
0.995
0.999
F3Depth 26
ResNet-101
0.071
0.270
2,842
0.103
0.953
0.993
0.998
NeWCRFS
Swin-Large
0.052
0.155
2.129
0.079
0.974
0.997
0.999
BinsFormer
X0
Swin-Large
0.052
0.151
2,096
0.079
0.974
0.999
PixelFormer
Swin-Large
0.051
0.149
2,081
0.077
0.976
0.997
0.999
Ve-DepthNNt
Swin-Large
Swin-Large
0.999
Swin-Large
0.051
0.999
Table 3: Quantitative depth comparison on the Eigen split of KITTI dataset. The maximum depth value is capped at 80 meters.
Image
AdaBins
NeWCRES
Ours
Figure 6: Qualitative depth comparison on the Eigen split of KITTI dataset.
prediction, we further merge the frequency subgroups with indices {6, 7 and {8...., 14), leading to 9 iterative steps total generate the final depth predictions.
4.3 Comparison with the State-of-the-Art
NYU-Depth-V2 We benchmark method against current ofate-of-The-Art (SoTA) approaches on the indoor NYU-Depth- V2 dataset, with quantitative results presented Tab. [ Despite vision transformers elevating the precision of depth estimation on this dataset, our method has surpassed existing SoTA approaches, particularly in the Abs Rel and < 1.25 metrics. Qualitative comparisons, illustrated in Fig. 4 reveal the adeptness of our method at capturing fine- mne-grained geometries and producing smoother depth estimations in planar areas. Regions where our method outperforms are highlighted with white boxes, emphasizing its superior depth estimation accuracy.
The TOFDC dataset is characterized by its dense ground truth depth data. By utilizing this dataset, we demonstrate the enhanced capability of our method to effectively harness the dense ground truth, thereby achieving more accurate depth estimations compared to existing SoTAS. We present quantitative results in Tab. [ where our method demonstrates superior performance over existing SoTAs across a majority of the evaluated metrics. Specifically, our method achieves a significant improvement on the Abs Rel and RMSE metrics compared to VA- VA-DepthNet, with enhancements of 19.7% and 8.7%, respectively. Fig. 5 provides qualitative comparisons, illustrating that our method not only produces more accurate depth estimations but also more effectively delineates the object from the background, leading more coherent depth estimations.
TOFDC
We further evaluate our method on the outdoor dataset, KITTI, which has sparse depth ground truth collected with LiDAR. This sparsity presents a contrast the denser depth information available in the NYU and TOFDC datasets, resulting in less robust supervision for learning frequency coefficients. Despite this challenge, our method demonstrates its robustness by achieving SoTA performance, which is attributed to the utilization of plenty training data coupled with our proposed regularization constraints. The quantitative analysis, detailed in Tab. 3 demonstrates the superior performance of our method. Qualitative evaluations, depicted in Fig. 6 further substantiate the superiority our method. The quantitative results on KITTI official split are reported Tab. 4 The pretrained weights from Semantic-SAM [18] are employed to initialize the encoder. Our method surpasses the compared approaches on the majority of metrics, particularly in the iRMSE metric, underscoring the robustness and effectiveness of our approach.
KITTI

---

NeWCRTS
Pixelfurmer
BindFomer
Ve.DephNNet
NDDepih
Metric
Ours
20
22
A4
33
SILog
11.77
11.67
10.39
10.28
10.14
9.89
9.84
9.63
9.62
9.60
Ahs Rel 4
8.78
9.04
8.37
8.16
8.23
8.11
7.96
7.82
7.75
7.83
Rel
2.23
2.21
1.83
1.82
1.69
1.66
1.60
1.59
1.54
IRMSB.
12.98
12.23
11.03
10.84
10.90
10.73
10.44
10.68
10.62
10.12
Table 4:
Quantitative depth comparison on the official split of KITTI dataset. All metrics reported here are from the KITTI online leaderboard.
NeWCRFS
MG-Depth
IEBins
VA-DeptiNet
Ours
[88]
34
22
Step
Steps
Steps
Steps
Steps
(M) J
273
262
259
Speed (FPS) 1
37.95
24.24
21.51
15.68
31,55
28.72
26.03
24.07
14.24
RMSE
0,334
0.311
0.314
0.304
0.310
0.307
0.306
0.305
0.304
1,25
0.922
0.933
0.936
0.937
0.937
0.939
0.940
Table 5: Parameter efficiency and inference speed NYU--Dephh-22 dataset. The right section enumerates the inference speed and corresponding performance metrics of our method at various iteration stages. All models are benchmarked single RTX 4090 GPU for consistency.
Parameter efficiency We compare the parameter efficiency of our method with current SoTAs on the NYU-Depth-Ve. V2 dataset, with the input resolution set to 640 480. The quantitative results, pre- sented in Tab. [ reveal that our method exhibits the fewest training parameters while simultancously achieving the best performance. For instance, our approach demonstrates 9.0% improvement the RMSE metric, while utilizing 4.1 % fewer parameters than NeWCRFS.
4.4 Ablation Study
We conduct comprehensive ablation studies to demonstrate the efficacy of the proposed PPH and PFF modules, and analyze the impact of the iteration steps on both model performance and inference speed. All experiments presented in this section are conducted on the NYU-Depth- V2 dataset.
Effect of PPH module To assess the impact of the PPH module, we build a baseline by excluding the PPH from our method. In this setup, we employ convolutional head to project the last-layer features to the output dimension. The final depth prediction is obtained through either bilinear and PixelShuffle 335 upsampling or inverse DCT that converts the predicted frequency coefficients back to the spatial domain. Additionally. we introduce the adaptive bins [3] as an alternative competitor. Quantitative experimental results are reported in Tab, 6 Among the three approaches outputting in the spatial domain, the PixelShuffle-ben basee approach performs the best. When predicting depth in the frequency domain, performance further improves. demonstrating the superiority of trequency-domnn depth prediction. Lastly, our progressive prediction scheme significantly outperforms the compared approaches by large margin, underscoring the efficacy of the PPH module.
Effect of PFF module To evaluate the impact of the PFF module, we establish baseline by excluding the PFF component from our method. We first introduce convolutional layer and PPM 61 module to process the image feature the last scale. Then, to validate the proposed DCT-based downsampling strategy, we replace it with bilinear and PixelUnshufile (35] downsampling. The quantitative experimental results are reported Tab. 7 The first two approaches, which only process the last-scale feature, perform worse than the competitors with multi-scale feature aggregation. This demonstrates the necessity of multi-scale feature aggregation for depth prediction. Furthermore, our method, employing the DCT-based downsampling strategy, achieves the best performance, showcasing the effectiveness of our proposed DCT-based strategy for feature downsampling.
Effect of iterative steps We analyze the impact of iterative steps on both prediction accuracy and inference speed. The results are reported in Tab. 5 and illustrated in Fig. 7 In summary, we observe that both prediction accuracy and inference time increase as the number iterations grows. Leveraging the energy compaction property of the DCT, we strike balance between accuracy and speed by selectively discarding predictions for high- figh-freyencc components. This strategic approach allows us effectively reduce the number of iterative steps.

---

Method
Output Domain
Abs Rel
R
1,25
1,25
1.25'
Baseline + Cone Bilineur
Sparial-Demain Domain
0.090
0.042
0.319
0.929
0.991
0.998
Bascline AdaBins + Bilincar
Spatial
SpatialDDenain Demain
0.088
0.042
0,319
0,932
0.991
0.998
Haseline + Cone PiselShufile
Spurial-Demain
0.088
0.041
0.318
0.933
0.992
0,998
Baseline Conv + inv DCT
Trequency-Dommin Domain
0.088
0.041
0.315
0.932
0.992
0.998
Baveline PPII
Trequency-Domain Domain
0.085
0.039
0.304
0.940
0.992
0.998
Table 6: Ablation study on the PPH module. The baseline is built by removing the PPH module.
Conv denotes linear projection with a convolutional layer. AdaBins refers to the adaptive bins [3] All are used to upsample the prediction.
methods output s/s scale, and Bilinear and PixelShuffle 35
31.4
Method
Abs Rel
RMSE
8 1.25
Baseline Conv
0.086
0.309
0.936
Baseline + PPM
0.086
0,306
0.939
3300
Baseline PFF (Bilinear)
0.085
0.305
0.940
A.301
Baseline + PFF (PixcUUnshuffle)
0,085
0.306
0.940
W.N. 30444 yoart 2413 JAIZ
Ours
0.085
0.304
0.940
338
314
Table 7: Ablation study on the PFF module. The baseline built by removing the PFF module. We evaluate the proposed DCT-based downsampling strategy by replacing it with bilin- and PixelUnshuffle [35] downsampling.
Figure 7: Accuracy vs. inference speed. The width of each bubble corresponds to the processing time.
5 Limitation and Broader Impact
Our method employs the differentiable inverse DCT to transform the predicted spectrum back to the spatial domain. By minimizing the difference between the spatial-domain estimation and the valid ground truth, our model can be trained end-to-end. However, the sparsity of the ground truth may lead to inefficient supervision of the frequency estimation. While we have proposed two regularization terms to prevent our model from being incorrectly optimized, we observe that our method is more effective with dense supervision. Exploring more effective training strategies when only sparse depth ground truth is available will be an important research direction for our future work.
Monocular depth estimation is pivotal technique for interpreting 3D scenes from 2D images and has widespread applications in autonomous driving, robotics, and 3D modeling, among others. Given the extensive applications of this task, our method is poised to positively impact these fields by advancing their capabilities. Considering the fundamental nature of monocular depth estimation, our work is not anticipated have significant negative societal impact.
6 Conclusion
In this paper, we introduce DCDepth, a novel framework for the MDE task. Departing from existing methods, our method progressively estimates patch-wise wise depth the frequency domain and then recovers spatial-domain depth via inverse DCT. This formulation inherently models local depth correlations and frames the estimation process as a global-to-local scheme, achieving more accurate depth estimation. Leveraging the energy compaction property of DCT, our method strikes an effective balance between accuracy and inference speed, making it well-suited for practical applications.
7 Acknowledgment
We would like to thank the reviewers and the chairs for their suggestions and efforts. This work was partially supported by the National Natural Science Foundation of China under Grant 62361 166670 and 62072242, the Fundamental Research Funds for the Central Universities under Grant 070- 63233084, the Young Scientists Fund of the National Natural Science Foundation of China under Grant 6220613 and the Tianjin Key Laboratory Visual Computing and Intelligent Perception. The PCA Lab is associated with the Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, and Jiangsu Key Lab of Image and Video Understanding for Social Security, School of Computer Science and Engineering, Nanjing University of Sci & Tech.
10

---

References
[1] Ashutosh Agarwal and Chetan Arora. Attention attention everywhere: Monocular depth predic- tion with skip attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5861-5870, 2023.
[2] N. Ahmed, T. Natarajan, and K.R. Rao. Discrete cosine transform. IEEE Transactions on Computers, C-23(1)990-93, 1974.
[3] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4009 4018, 2021.
[4] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Localbins: Improving depth estimation by learning local distributions. In European Conference on Computer Vision, pages 480-496. Springer, 2022.
[5] Yuanzhouhan Cao, Zifeng Wu, and Chunhua Shen. Estimating depth from monocular images as classification using deep fully convolutional residual networks. JEEE Transactions on Circuits and Systems for Video Technology, 28(1 1):3144-3182, 2017.
[6] Wen-Hsiung Chen, C. Smith,, and S. Fralick. A fast computational algorithm for the discrete cosine transform. IEEE Transactions on Communications, 25(9): 1004-1009, 1977.
[7] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar, 2014. Association for Computational Linguistics.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255 leee, 2009.
[9] Xingshuai Dong. Matthew A. Garratt, Sreenatha G. Anavatti, and Hussein A. Abbass. Towards real-time monocular depth estimation for robotics: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(10): 19900:66940-1961 2022.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199 19929, 2020.
[11] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using multi-scale deep network. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2014.
[12] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2018
[13] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361. IEEE, 2012.
[14] Zhixiang Hao, Yu Li. Shaodi You, and Feng Lu. Detail preserving depth estimation from single image using attention guided networks. In 2018 International Conference on 3D Vision (3DV), pages 304-313. IEFF, 2018
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.

---

[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69800 2014.
[17] Jin Han Lee, Myung-Kyy Kyu Han, Dong Wook Ko, and II Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprini 1rXiv:1907.10322 2019,
[18] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang. Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint arX 2023
[19] Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2041-20555 2018.
[20] Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation. IEEE Transactions on Image Processing, 33:3964-3976. 2024.
[21] Ce Liu, Suryansh Kumar, Shuhang Gu, Radu Timofte, and Luc Van Gool. Single image depth prediction made better: A multivariate gaussian take. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17346-17356, 2023.
[22] Ce Liu, Suryansh Kumar, Shuhang Gu, Radu Timofte, and Luc Van Gool. Va-depthnet: A variational approach to single image depth prediction. arXiv preprint arXiv:2302.06556, 2023.
[23] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022 2021.
[24] Xiaoxiao Long, Cheng Lin, Lingjie Liu, Wei Li, Christian Theobalt, Ruigang Yang, and Wenping Wang. Adaptive surface normal constraint for depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12849- 12858, 2021.
[25] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019.
[26] Vaishakh Patil, Christos Sakaridis, Alexander Liniger, and Luc Van Gool. P3depth: Monocular depth estimation with a piecewise planarity prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1610-1621, 2022.
[27] Luigi Piccinelli, Christos Sakaridis, and Fisher Yu. idisc: Internal discretization for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21477-21441 2023.
[28] Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia. Geonet: Geometric neural network for joint depth and surface normal estimation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 283-291, 2018.
[29] Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprini 1710.0559) 2017.
[30] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12179- 12188, 2021.
[31] Ashutosh Saxena, Sung Chung, and Andrew Ng. Learning depth from single monocular images. In Advances in Neural Information Processing Systems. MIT Press, 2005.
[32] Ashutosh Saxena, Jamie Schulte, Andrew Ng, et Depth estimation using monocular and stereo cues. In IJCAI, pages 2197-2203, 2007.
12

---

[33] Shuwei Shao, Zhongcai Pei. Weihai Chen, Xingming Wu, and Zhengguo Li. Nddepth: Normal- distance assisted monocular depth estimation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7931-7940, 2023.
[34] Shuwei Shao, Zhongeai Pei, Xingming Wu, Zhong Liu. Weihai Chen, and Zhengguo Li. lebins: Iterative elastic bins for monocular depth estimation. In Advances in Neural Information Processing Systems, pages 53025-5305 53037. Curran Associates, Inc., 2023.
[35] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop. Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874-1883, 2016.
[36] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Esion-ECCT 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746-760 Springer, 2012.
[37] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, pages 369-386. SPIE, 2019.
[38] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114 14. PMLR, 2019.
[39] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 402-419. Springer, 2020.
[40] Yi-Min Tsai, Yu-Lin Chang, and Liang-Gee Chen. Block-based vanishing line and vanishing point detection for 3d scene reconstruction. In 2006 international symposiun intelligent signal processing and communications, pages 386-589. IEEE, 2005.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[42] Kun Wang, Zhenyu Zhang, Zhiqiang Yan, Xiang Li, Baobei Xu, Jun Li, and Jian Yang. Regularizing nighttime weirdness: Efficient supersuu monocular depth estimation in the dark. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16055-16064, 2021.
[43] Kun Wang, Zhiqiang Yan, Huang Tian, Zhenyu Zhang, Xiang Li, Jun Li, and Jian Yang. Altnerf: Learning robust neural radiance field via alternating depth-post pole optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, pages (5085551,,40 2024.
[44] Lijun Wang, Jianming Zhang, Oliver Wang, Zhe Lin, and Huchuan Lu. Sde-depth: Semantic divide-and-conquee network for monocular depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[45] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattem Recognition, pages 8445-8453. 2019.
[46] Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 14475-14485, 2023.
[47] Dan Xu, Wei Wang. Hao Tang, Hong Liu, Nicu Sebe, and Elisa Ricci. Structured attention guided convolutional neural fields for monocular depth estimation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3917-3925, 2018.
13

---

[48] Zhiqiang Yan, Xiang Li, Kun Wang, Zhenyu Zhang, Jun Li, and Jian Yang. Multi-modal masked pre-traiming for monocular panoramic depth completion. In European Conference on Computer Vision, pages 378-395. Springer, 2022.
[49] Zhiqiang Yan, Kun Wang, Xiang Li, Zhenyu Zhang, Jun Li, and Jian Yang. Rignet: Repetitive image guided network for depth completion. In Computer Vision - ECCV 2022, pages 214-230, Cham, 2022. Springer Nature Switzerland.
[50] Zhiqiang Yan, Xiang Li, Kun Wang, Shuo Chen, Jun Li, and Jian Yang. Distortion and uncertainty aware loss for panoramic depth completion. In International Conference on Machine Learning, pages 39099-39109 PMLR, 2023.
[51] Zhiqiang Yan, Kun Wang, Xiang Li, Zhenyu Zhang, Jun Li, and Jian Yang. Desnet: Decomposed scale-consisten network for unsupervised depth completion. In Proceedings of the AAAI conference on artificial intelligence, pages 3109-310 3117,,17722100 2023.
[52] Zhiqiang Yan, Yuankai Lin, Kun Wang. Yupeng Zheng, Yufei Wang, Zhenyu Zhang, Jun Li, and Jian Yang. Tri-perspective view decomposition for geometry-aware depth completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[53] Zhiqiang Yan, Yupeng Zheng, Deng-Ping Fan, Xiang Li, Jun Li, and Jian Yang. Learnable differencing center for nighttime depth perception. Visual Intelligence, 2(1):15, 2024.
[54] Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe, and Elisa Ricci. Transformer-based attention networks for continuous pixel-wise prediction. In Proceedings of the IEEF/CVF International Conference on Computer vision, pages 16269-16279, 2021.
[55] Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan. Enforcing geometric constraints of virtual normal for depth prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5684-5693, 2019.
[56] Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3d scene shape from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 204-213, 2021.
[57] Zhu Yu, Zehua Sheng, Zili Zhou, Lun Luo, Si- Yuan Cao, Hong Gu, Huaqi Zhang, and Hui-Liang Shen. Aggregating feature point cloud for depth completion. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8732-8743, 2023.
[58] Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Neural window fully- connected crfs for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3916-3925, 2022.
[59] Ruo Zhang, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah. Shape-from-shading survey. IEEE transactions on pattern analysis and machine intelligence, 21(8):690-706, 1999.
[60] Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang. Pattern- afffiri-affiiitive propagation across depth, surface normal and semantic segmentation. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pages 4106-4115, 2019.
[61] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pallern recognition, pages 1-890, 2017
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state our contributions and the claims made well match the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have elaborated our architecture design in the Method section, and have described the implementation details in the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

reproducible, they Answer: [Yes]
Justification: The datasets used in our work are all publicly available, and we have properly cited these datasets. The code and related materials of our work are publicly available https: http///gtttbb... www.ccoomwwee.eeee wwmmm// DDDDpth
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (itttps:/niis.cc.
public/guides Stnili/c/uuidss/ddeubbbbmbs ss aassoooollcy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https;
7 //nips. cttpppscccpbblccggucccc www.cc/ggisseeeeee CodeSubmissiia.ccoooooooo osooicycyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We have provided the experimental details in the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The datasets used in our experiments are large, and it is very costly to train our model for many times. summary, we don't report the error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have reported the information of compute resources used in our experi- ments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpe::/heuripp.cc/pppb.//// /eteeiii//heeeeeeeeeeeeeeeeeeeee crtuideleeeeee
Answer: [Yes]
Justification:
Our research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impacts of our work in Sec. 5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets used in work are properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "7t26LxdRdB", "reviewer": "Reviewer_HHm4", "summary": "The author has proposed the DC depth, which aims to predict a depth map from a monocular image.\nThe authors introduce a novel technique that implements depth estimation of the frequency coefficients from the discrete cosine domain and enables modeling the local depth correlations.\nThe author conducted experiments on the NYU-Depth-V2, ToF-DC, and KITTI datasets.", "questions": "Please consider answering the questions in weaknesses section during the rebuttal.\nBesides, on L27-28, the author states that the current depth estimation is based on a per-pixel basis. I am not sure if this statement is reasonable, as recent MDE methods, such as Depth Anything, they are based on transformers, and there are some operations of patch tokenization, perhaps the per-pixel basis is not a reasonable way of writing.", "limitations": "yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Predicting depth from the frequency domain is a relatively novel insight.\n2. The presentation is clear. Figure 1 clearly illustrates the progressive estimation scheme.", "weaknesses": "Lack of comparison:\nWhy not compare with methods like Depth Anything [Yang et al., CVPR 2024], Metric3D[Yin et al., ICCV 2023], etc.? The methods currently compared are outdated. Therefore, it cannot be said that a new state-of-the-art performance has been achieved. L14", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Clarification of Experimental Comparison:** Thank you for your comment. We would like to clarify the following points:\n\n   1. Our research goal is different from that of Depth Anything and Metric3D. We focus on the novel network and algorithm design for monocular depth estimation task, while Depth Anything and Metric3D focus on improving the generalization of the depth estimation model. This difference leads to different experimental setup for these two kinds of methods, which we summarize as follows:\n\n      |     Method     | Pretraining  |                    Traning Set Size                     | Ratio to Our Training Set |\n      | :------------: | :----------: | :-----------------------------------------------------: | :-----------------------: |\n      | Depth Anything |   DINO-V2    |                    Over 63.5 Million                    |    2621 / 6350 / 2742     |\n      |   Metric 3D    | ImageNet-22K |                     Over 8 Million                      |      330 / 800 / 345      |\n      |      Our       | ImageNet-22K | NYU-Depth-V2: 24231 / TOFDC: 10000 / KITTI Eigen: 23158 |         1 / 1 / 1         |\n\n      Due to the significant differences in the experimental setup, it is unfair to directly compare our method with these two methods, thus we do not include these comparison results in our paper.\n\n   2. We disagree with your comment that “the methods currently compared are outdated,” as we have included recent approaches such as IEBins [1], BinsFormer [2], and VA-DepthNet [3] in our comparison, as shown in Tables 1, 2, and 3 of the main text and Table 7 of the supplementary material.\n\n   3. Under the same experimental settings, our method surpasses existing methods on three datasets, achieving state-of-the-art performance. To further demonstrate the effectiveness of our method, we additionally designed the following comparative experiments:\n\n      - We use the weights of Depth Anything as initialization, transplant the proposed progressive prediction head into the network of Depth Anything, and fine-tune it on NYU-Depth-V2 following the settings in the Depth Anything paper. We denote the fine-tuned Depth Anything model as *Depth Anything ft.*, and our fine-tuned method as *Our ft. DA*.\n      - We use the weights from Metric3D encoder as initialization and randomly initialize the decoder. Then we fine-tune Metric3D and our model on the NYU-Depth-V2 dataset for 10 epochs, denoted as *Metric3D ft.* and *Our ft. M3D*, respectively.\n\n      The experimental results are reported as follows:\n\n      |       Method       | Reference |    Backbone    |  Abs Rel  |   RMSE    | $log_{10}$ | $\\delta<1.25$ | $\\delta<1.25^2$ | $\\delta<1.25^3$ |\n      | :----------------: | :-------: | :------------: | :-------: | :-------: | :--------: | :-----------: | :-------------: | :-------------: |\n      | Depth Anything ft. | CVPR-2024 |   ViT-Large    |   0.056   |   0.206   |   0.024    |     0.984     |      0.998      |      1.000      |\n      |    Metric3D ft.    | ICCV-2023 | ConvNeXt-Large |   0.065   |   0.232   |   0.028    |     0.971     |      0.996      |      0.999      |\n      |     Our ft. DA     |    ---    |   ViT-Large    | **0.055** | **0.204** | **0.024**  |   **0.985**   |    **0.998**    |    **1.000**    |\n      |    Our ft. M3D     |    ---    | ConvNeXt-Large |   0.062   |   0.229   |   0.027    |     0.970     |      0.996      |      0.999      |\n\n      Our finetuned methods outperform their counterparts, demonstrating the superiority of our approach.\n\n   [1] Shao et al. “IEBins: Iterative Elastic Bins for Monocular Depth Estimation\", NIPS 2023.\n\n   [2] Li et al. \"BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation\", TIP 2024.\n\n   [3] Liu et al. \"VA-DepthNet: A Variational Approach to Single Image Depth Prediction\", ICLR 2023.\n\n2. **Clarification on Per-Pixel Basis Statement:** Thank you for your feedback. On lines 27-28, we stated that “to predict depth on a per-pixel basis within the spatial domain,” which indicates that the mentioned approaches independently predict depth for each pixel in the spatial domain. In contrast, our method models local depth correlations by predicting the frequency spectrum in the discrete cosine domain. The term “per-pixel basis” refers to the output, not the input. We will enhance the expression in the revised version to clarify this distinction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qWrEIrghib", "reviewer": "Reviewer_fZWS", "summary": "This paper introduces DCDepth, a framework designed to tackle the long-standing monocular depth estimation task. In general, the proposed methods are quite interesting, the motivations for this work are clear, the experiments conducted are comprehensive, and the paper is well-structured.", "questions": "Please see weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces DCDepth, a framework designed to tackle the long-standing monocular depth estimation task. In general, the proposed methods are quite interesting, the motivations for this work are clear, the experiments conducted are comprehensive, and the paper is well-structured.", "weaknesses": "1. The comparisons with SoTA methods in Table 1 and Table 2 are not very clear. The authors should consider focusing on more decimal places and only highlighting the best-performing methods for each metric.\n\n2. The tables should include the publication year and the venue of the compared methods, as this will help readers better understand the context of the comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Table Result Presentation:** Thank you for your valuable feedback! Based on your suggestions, we plan to update our tables in the following two aspects:\n\n   1. Updating the accuracy metrics in Tables 1 and 2 to percentages and multiplying the error metrics in Table 1 by 100, keeping two decimal places for reporting quantitative results.\n   2. Highlighting only the best-performing methods for each metric.\n\n   Below are the updated comparison results of VA-DepthNet, which is closest to our method in performance on the NYU-Depth-V2 and TOFDC datasets:\n\n   |   Method    | Reference |  Backbone  | Abs Rel  |  Sq Rel  |   RMSE    | $log_{10}$ | $\\delta<1.25$ | $\\delta<1.25^2$ | $\\delta<1.25^3$ |\n   | :---------: | :-------: | :--------: | :------: | :------: | :-------: | :--------: | :-----------: | :-------------: | :-------------: |\n   | VA-DepthNet | ICLR-2023 | Swin-Large |   8.56   |   3.88   |   30.44   |    3.68    |     93.68     |      99.20      |    **99.87**    |\n   |     Our     |    ---    | Swin-Large | **8.51** | **3.87** | **30.43** |  **3.66**  |   **93.95**   |    **99.20**    |      99.84      |\n\n   |   Method    | Reference |  Backbone  |  Abs Rel  |  Sq Rel   |   RMSE    | $RMSE_{log}$ | $\\delta<1.25$ | $\\delta<1.25^2$ | $\\delta<1.25^3$ |\n   | :---------: | :-------: | :--------: | :-------: | :-------: | :-------: | :----------: | :-----------: | :-------------: | :-------------: |\n   | VA-DepthNet | ICLR-2023 | Swin-Large |   0.234   |   0.029   |   0.619   |    0.373     |  **99.550**   |     99.890      |     99.969      |\n   |     Our     |    ---    | Swin-Large | **0.188** | **0.027** | **0.565** |  **0.352**   |    99.490     |   **99.891**    |   **99.970**    |\n\n   In the revised version, we will update our tables according to these principles.\n\n2. **Including Publication Year and Venue Information for Comparison Methods:** Thank you for your valuable advice! We will include the publication year and venue information of the compared methods in our tables to help readers better understand the context of the comparisons.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "APPaElUghD", "reviewer": "Reviewer_tNUP", "summary": "The paper introduces a frequency domain-based method for monocular depth estimation. The proposed method begins with the prediction of low-frequency components to establish a global scene context, followed by successive refinement of local details through the prediction of higher-frequency components. The proposed method is validated on the NYU-Depth-V2, TOFDC and KITTI datasets.", "questions": "Please see the weakness section.", "limitations": "It is nice to try the completed ground-truth depth maps for supervision on the KITTI dataset, using existing depth completion frameworks.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "To the best of my knowledge, the discrete cosine domain-based progressive design is significantly different from previous methods and gives a promising paradigm in depth estimation.\n\nThe proposed method is interpretable, using discrete cosine transformation to segregate the depth information into various frequency components and enriching details in a progressive manner.\n\nThe proposed method outperforms prior work across multiple datasets with comparable or fewer parameters.\n\nDetailed ablation study indicate the effectiveness of each key component.\n\nThe paper is clearly structured and well presented.", "weaknesses": "As shown in Table 2, the proposed method significantly outperforms NewCRFs, PixelFormer, and IEBins on the TOFDC dataset compared to other datasets. It would be beneficial to provide a detailed discussion regarding this to better understand the strengths of the proposed method.\n\nIt is nice to also visualize the evolution of depth predictions in the frequency domain.\n\n\nIn Fig. 3, the three depth predictions at the bottom look the same. They probably need to be replaced with the actual experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Discussion on TOFDC Result:** Thanks for your valuable suggestion. The TOFDC dataset is challenging due to its small number of training images with limited diversity. NewCRFs, PixelFormer, and IEBins independently predict pixel-wise depth without modeling the correlations among them. We believe this approach makes it difficult to learn general depth estimation knowledge from a small-scale dataset, causing these large models to overfit on the training set and resulting in degraded depth estimation performance. In contrast, models with smaller parameters, such as BTS and AdaBins, achieve better depth estimation results.\n\n   Unlike the large models mentioned above, our proposed method models local depth correlations by making patch-wise predictions in the frequency domain. This approach allows our model to better exploit general depth estimation knowledge. Additionally, we include two regularizations in the training loss to encourage the model to output smoother depths, helping to avoid overfitting and achieve better performance.\n\n2. **Depth Prediction Evolution Visualization:** Thank you for your professional suggestion. We have included the visualization of frequency prediction evolution in Figure 1 of the attached PDF. We will incorporate this result into our revised paper.\n3. **Depth Prediction Visualization in Figure 3:** Thank you for your suggestion. The depth results at the bottom of Figure 3 are produced by our model. However, they appear indistinguishable due to the small resolution. We will enhance the presentation in our revised version to ensure the differences are more visible.\n4. **Supervision with Complete Depth Map:** Thank you for your constructive advice. We will explore using the dense depth map generated by the depth completion model as supervision to improve the performance of our method in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4hh51coJcz", "reviewer": "Reviewer_DXhZ", "summary": "The paper presents a novel framework for the long-standing monocular depth estimation. The task is first formulated as a progressive regression in the discrete cosine domain. The authors propose two modules: the PPH module progressively estimates higher-frequency coefficients based on previous predictions, and the PFF module incorporates a DCT-based downsampling technique to mitigate information loss and ensure effective integration of multi-scale features.", "questions": "See the Weaknesses.", "limitations": "The contributions are not substantial enough. The authors have not clearly articulated their design for DCT-based downsampling. Additionally, the proposed method does not achieve state-of-the-art results.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-organized and the idea is easy to understand.\n2. The results of the method are presented clearly.", "weaknesses": "1.\tThe authors claim that the global-to-local (coarse-to-fine) depth estimation is a contribution, but this idea is common and adopted by other works [1] [2].  \\\n[1] Liu C, Yang G, Zuo W, et al. Dpdformer: a coarse-to-fine model for monocular depth estimation[J]. ACM Transactions on Multimedia Computing, Communications and Applications, 2024, 20(5): 1-21.\\\n[2] Li Y, Luo F, Xiao C. Self-supervised coarse-to-fine monocular depth estimation using a lightweight attention module[J]. Computational Visual Media, 2022, 8(4): 631-647.\n2.\tIn Section 3.2, some descriptions are confusing. “This grouping strategy ensures that lower-frequency groups contain fewer components necessitating more prediction steps, while higher-frequency groups encompass a larger number of components requiring fewer steps”, but in Fig 2, local areas (higher-frequency) require more steps. It is recommended to explain the unclear representation.\n3.\tThe authors claim that the DCT-based downsampling technique tends to mitigate information loss, but this module has not been reasonably explained.\n4.\tThe experiments are somewhat lacking in terms of including the latest works that achieve state-of-the-art performance. Although the authors compare their results with some previous works, other MDE methods [3, 4, 5] could provide valuable additional comparisons. When evaluated against these newer methods, the proposed method does not demonstrate superior performance. \\\n[3] Ning J, Li C, Zhang Z, et al. All in tokens: Unifying output space of visual tasks via soft token[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 19900-19910. \\\n[4] Yang L, Kang B, Huang Z, et al. Depth anything: Unleashing the power of large-scale unlabeled data[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 10371-10381. \\\n[5] Saxena S, Kar A, Norouzi M, et al. Monocular depth estimation using diffusion models[J]. arXiv preprint arXiv:2302.14816, 2023. \n5. “MDE is extensively applied across various fields such as autonomous driving, robotics, and 3D modeling [45, 48, 9, 42]” recommend changing to “autonomous driving[…], robotics[…], and 3D modeling[…]”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Clarification about Contribution:** Thank you for your comment. We believe there may be a misunderstanding regarding our contributions for the following reasons:\n   1. As stated in lines 53-54, our main contribution is the first to formulate monocular depth estimation as a progressive regression task in the discrete cosine domain, achieving state-of-the-art performance. All other reviewers consider our idea a novel insight (Reviewer HHm4), quite interesting (Reviewer fZWS), and significantly different from previous methods (Reviewer tNUP).\n   2. The global-to-local or coarse-to-fine concept is broad, and researchers can achieve this in various ways. The two approaches you mentioned employ tandem networks to perform pixel-wise coarse depth estimation and final depth refinement in the spatial domain. In contrast, our method leverages discrete cosine transformation to segregate depth information into various frequency components and progressively predict them from low-frequency to high-frequency components based on an iterative machenism. Our approach is fundamentally different from the two approaches you mentioned.\n   3. Besides the novel formulation for monocular depth estimation, we also proposed two innovative modules: the progressive prediction head and the pyramid feature fusion module. We demonstrated their effectiveness through ablation studies.\n\n2. **Clarification about Grouping Strategy:** Thank you for your comment. The lower-frequency groups contain fewer frequency components to be predicted, while the higher-frequency groups contain more frequency components. Each group of components is predicted through one iteration, thus the average number of prediction steps for higher-frequency components is less than that for lower-frequency components. Due to space limitations, we only report part of the evolution result in Figure 2. Please refer to Figure 1 of the attached PDF for a detailed illustration.\n\n3. **Clarification about DCT-based Downsampling:** Thank you for your comment. We believe you may have missed some of the explanations about the DCT-based downsampling in our main text. In Section 3.1, we review the Discrete Cosine Transform (DCT) and introduce its energy compaction property. In lines 154-162, we elaborate on the workflow of the DCT-based downsampling strategy, explaining that the key information of feature maps is preserved during downsampling by leveraging the energy compaction property of DCT. Additionally, we illustrate the workflow of DCT-based downsampling in the bottom-left corner of Figure 3.\n\n4. **Clarification about Experimental Comparison:** Thank you for your comment. Compared with our method, the approaches you mentioned have significant advantages in experimental configurations. We summarize them as follows:\n\n   | Method             | Backbone        | Pretraining                                                  | Traning Set Size       |\n   | ------------------ | --------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |\n   | AiT [3]            | Swin-Large V2   | SimMIM                                                       | Same as Our   |\n   | Depth Anything [4] | ViT-Large       | DINO V2                                                      | Over 63.5 Million                                            |\n   | DepthGen [5]       | Efficient U-Net | Image-to-image self-supervised pretraining & supervised image-to-depth pretraining | Supervised pretraining: 8.5 million & NYU-Depth-V2 finetuning: 50000 |\n   | Our                | Swin-Large V1   | ImageNet-22k classification                                  | NYU-Depth-V2: 24231 |\n\n   We did not include the comparison with these methods due to the large gap in experimental configurations. To further demonstrate the advancement of our method, we integrated our method into Depth Anything and finetuned it in NYU-Depth-V2 following the settings in the Depth Anything paper. We denote the fine-tuned Depth Anything model as *Depth Anything ft.*, and our fine-tuned method as *Our ft. DA*. We also compared our method with DepthGen on both NYU-Depth-V2 and KITTI Eigen datasets. The results are reported as follows:\n\n   |       Method       |    Backbone     |  Abs Rel  |   RMSE    | $log_{10}$ | $\\delta<1.25$ | $\\delta<1.25^2$ | $\\delta<1.25^3$ |\n   | :----------------: | :-------------: | :-------: | :-------: | :--------: | :-----------: | :-------------: | :-------------: |\n   | Depth Anything ft. |    ViT-Large    |   0.056   |   0.206   |   0.024    |     0.984     |      0.998      |      1.000      |\n   |      DepthGen      | Efficient U-Net |   0.074   |   0.314   |   0.032    |     0.946     |      0.987      |      0.996      |\n   |        Our         |   Swin-Large    |   0.085   |   0.304   |   0.037    |     0.940     |      0.992      |      0.998      |\n   |     Our ft. DA     |    ViT-Large    | **0.055** | **0.204** | **0.024**  |   **0.985**   |    **0.998**    |    **1.000**    |\n\n   |  Method  |    Backbone     |  Abs Rel  |   RMSE    | $RMSE_{log}$ | $\\delta<1.25$ | $\\delta<1.25^2$ | $\\delta<1.25^3$ |\n   | :------: | :-------------: | :-------: | :-------: | :----------: | :-----------: | :-------------: | :-------------: |\n   | DepthGen | Efficient U-Net |   0.064   |   2.985   |    0.100     |     0.953     |      0.991      |      0.998      |\n   |   Our    |   Swin-Large    | **0.051** | **2.044** |  **0.076**   |   **0.977**   |    **0.997**    |    **0.999**    |\n\n​\tOur finetuned method outperforms the depth anything couterpart, which demonstrate the superiority of our method.   Furthermore, our method significantly outperforms DepthGen on all metrics of KITTI Eigen and several metrics on NYU-Depth-V2, despite DepthGen employing a large amount of data for supervised pretraining.\n\n5. **About References:** Thanks for your advice and we will revise our paper according to your suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and Chairs,

Thank you for your constructive feedback and efforts.

We have individually responded to each reviewer’s comments. In the submitted PDF, we have visualized the prediction evolution of our method in both the spatial and frequency domains.

If you have any questions, please feel free to discuss them with us. If you are satisfied with our responses, we hope you will consider improving your rating.

Best regards,

The Authors

Meta-review: As acknowledged by reviewers, tackling monocular depth estimation by exploiting the frequency domain is original and experimental results demonstrate the effectiveness of such a paradigm. The concerns raised by reviewers in the initial evaluation report were mainly concerned with the missing comparison with networks like DepthAnything and Metric3D, which were adequately addressed in the rebuttal and the following discussion with the authors. At the end of the review process, all reviewers recommended acceptance, acknowledging the relevance of the proposed paradigm. The AC is on the same page and does the same.
Decision: Accept (poster)