Self-Distilled Depth Refinement with Noisy Poisson Fusion
Authors: Jiaqi Li
Yiran Wang
Jinghong Zheng
Zihao Huang
Ke Xian
Zhiguo Cao
Jianming Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Depth refinement
Noisy Poisson fusion
Self-distilled training
Primary area: machine_vision
TL;DR: SDDR proposes to model the depth refinement task by noisy Poisson fusion and train the refinement network in a self-distillation paradigm, achieving both high accuracy and efficiency.

Abstract: Depth refinement aims to infer high-resolution depth with fine-grained edges and details, refining low-resolution results of depth estimation models. The prevailing methods adopt tile-based manners by merging numerous patches, which lacks efficiency and produces inconsistency. Besides, prior arts suffer from fuzzy depth boundaries and limited generalizability. Analyzing the fundamental reasons for these limitations, we model depth refinement as a noisy Poisson fusion problem with local inconsistency and edge deformation noises. We propose the Self-distilled Depth Refinement (SDDR) framework to enforce robustness against the noises, which mainly consists of depth edge representation and edge-based guidance. With noisy depth predictions as input, SDDR generates low-noise depth edge representations as pseudo-labels by coarse-to-fine self-distillation. Edge-based guidance with edge-guided gradient loss and edge-based fusion loss serves as the optimization objective equivalent to Poisson fusion. When depth maps are better refined, the labels also become more noise-free. Our model can acquire strong robustness to the noises, achieving significant improvements in accuracy, edge quality, efficiency, and generalizability on five different benchmarks. Moreover, directly training another model with edge labels produced by SDDR brings improvements, suggesting that our method could help with training robust refinement models in future works.

Self-Distilled Depth Refinement with Noisy Poisson Fusion
Jiaqi Like*
Yiran Wang"
Jinghong Zheng'
Zihao Huang!
Ke Xian
Zhiguo Cao Jianming Zhang*
School of AIA, Huazhong University of Science and Technology
SSchoo EIC, Huazhong University of Science and Technology
*Adobe Research
'Equal contribution !Corresponding author
(lijiaqi_mane mail, wangyiran, deepzheng, ziiaahuun kxi an 2 Jhhust edu cn
jianmzha@adobe com
https://gittut: http:/ttwu..ccca/ija///a///a
Abstract
Depth refinement aims infer high-resolution depth with fine- fine-grained edges and details, refining low-resolution results of depth estimation models. The prevailing methods adopt tile-based manners by merging numerous patches, which lacks efficiency and produces inconsistency. Besides, prior arts suffer from fuzzy depth boundaries and limited generalizability. Analyzing the fundamental reasons for these limitations, model depth refinement noisy Poisson fusion problem with local inconsistency and edge deformation noises. We propose the Self-distileed Depth Refinement (SDDR) framework to enforce robustness against the noises, which mainly consists of depth edge representation and edge-bases guidance. With noisy depth predictions input, SDDR generates low-noise depth edge representa- tions as pseudo-labels by coarse-to- edge-guided gradient loss and edge-based based fusion loss serves as the optimization objective equivalent to Poisson fusion. When depth maps are better refined, the labels also become more noise-free. Our model can acquire strong robustness the noises, achieving significant improvements in accuracy, edge quality, efficiency, and generalizability on five different benchmarks. Moreover, directly training an- other model with edge labels produced by SDDR brings improvements, suggesting that our method could help with training robust refinement models in future works.
self-distillation. Edge- guidance with
Introduction
Depth refinement infers high-resolution depth with accurate edges and details, refining the low- resolution counterparts from depth estimation models [30, 51, 1]. With increasing demands for high resolutions in modern applications, depth refinement becomes prerequisite for virtual reality [24, 13], bokeh rendering [27, 28], and image generation [33, 54]. The prevailing methods [25, 21] adopt two-stage tile-based frameworks. Based the one-stage refined depth of the whole image, they merge high-frequency details by fusing extensive patches with complex patch selection strategies. However, numerous patches lead heavy computational costs. Besides, as in Fig. (a), excessive integration of local information leads to inconsistent depth structures, e.g., the disrupted billboard.
Apart from efficiency and consistency, depth refinement [25, 14, 4, 3, 37, 21] is restricted by noisy and blurred depth edges. Highly accurate depth annotations with meticulous boundaries are necessary enforce fine-graine eneiinaieed details. For this reason, prior arts [14, 37, 21] only synthetic datasets [32,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Accuracy (8x)
0.892
0.904
0.010
0.20
Two stage
Ours (two-ttaee)
B1G 36-177
Patchfussion
16.7G x30
PatchFusion
Boost
0.1
Boost 76
Quality
Ours (one-stage) 16.76
0.22
GBOF
Graph GDSR 392.4G
30.60
GBOF
0.23
Km et
2138.00 3G
(a) Visualization of Depth Refinement Approaches
Performance and Efficiency
Figure 1: (a) Visual comparisons. We model depth refinement by noisy Poisson fusion with the local inconsistency noise (representing the inconsistent billboard and wall in red box) and the edge deformation noise (indicating blurred depth edges in the blue box and second row). Better viewed when zoomed in. (b) Performance and efficiency. Circle area represents FLOPs. The two-stage methods [25, 21] are reported by multiplying FL.OPs per patch with patch numbers. SDDR outperforms prior arts in depth accuracy (61), edge quality (ORD), and model efficiency (FLOPs).
39, 11, 45, 44] for the highly accurate depth values and edges. However, synthetic data falls short of the real world in realism and diversity, causing limited generalizability with blurred depth and degraded performance on m-tthe-wld scenarios. Some attempts [25, 3] simply adopt natural- scene datasets [35, 47, 49, 5, 20] for the problem. The varying characteristics of real-world depth annotations, e.g., sparsity [2, 5, 7], inaccuracy [35, 36, 55], or blurred edges [48, 47, 43, 10], make them infeasible for supervising refinement models. Thus, GBDF [3] uses depth predictions [51] as pseudo-labels, while Boost [25] leverages adversarial training [6] as guidance. Those inaccurate pseudo-labels and guidance still lead to blurred edges as shown in Fig. 1 (a). The key problem is to alleviate the noise of depth boundaries by constructing accurate edge representations and guidance.
To tackle these challenges, we dig into the underlying reasons for the limitations, instead of the straightforward merging of local details. We model depth refinement noisy Poisson fusion problem, decoupling depth prediction errors into two degradation components: local inconsistency noise and edge deformation noise. We use regional linear transformation perturbation as the local inconsistency noise to measure inconsistent depth structures. The edge deformation noise represents fuzzy boundaries with Gaussian blur. Experiments in Sec. 3.1 showcase that the noises can effectively depict general depth errors, serving as our basic principle to improve refinement results.
In pursuit of the robustness against the local inconsistency noise and edge deformation noise, we propose the Self-distileed Depth Refinement (SDDR) framework, which mainly consists of depth edge representation and edge-based guidance. A refinement network is considered as the Poisson fusion operator, recovering high-resolution depth from noisy predictions of depth models [51, 30, 1]. Given the noisy input, SDDR generate low-noise and accurate depth edge representation as pseudo-labels through coarse-to-fine self-distillation. The edge-based guidance including adge-guided gradient loss and edge-based fusion loss is designed the optimization objective of Poisson fusion. When depth maps are better refined, the pseudo-labed also become more noise-free. Our approach establishes accurate depth edge representations and guidance, endowing SDDR with strong robustness to the two types of noises. Consequently, as shown in Fig. (b), SDDR significantly outperforms prior arts [25, 21, 3] in depth accuracy and edge quality. Besides, without merging numerous patches as the two-stage tile-based methods [21, 25]. SDDR achieves much higher efficiency.
We conduct extensive experiments five benchmarks. SDDR achieves state-of-the-ar performance on the commonly-used Middlebury2021 [34], Multiscopic [52], and Hypersim [32]. Meanwhile, since SDDR can establish self-distillation with accurate depth edge representation and guidance on natural scenes, the evaluations on in-the-wild DIML [15] and DIODE [40] datasets showcase our superior generalizability. Analytical experiments demonstrate that these noticeable improvements essentially arise from the strong robustness to the noises. Furthermore, the precise depth edge labels produced by SDDR can be directly used to train another model [3] and yield improvements, which indicates that our method could help with training robust refinement models in future works.
In summary, main contributions can be summarized follows:

---

We model the depth refinement task through the noisy Poisson fusion problem with local inconsis- tency noise and edge deformation noise as two types of depth degradation.
We present the robust and efficient Self- Belf-distilled Depth Refinement (SDDR) framework, which can generate accurate depth edge representation by the coarse-to-fin self-distillation paradigm.
We design the edge-guided gradient loss and edge-based fusion loss, as the edge-based guidance to enforce the model with both consistent depth structures and meticulous depth edges.
2 Related Work
Depth Refinement Models. Depth refinement refines low-resolution depth from depth estimation models [30, 51, 11. predicting high-resolution depth with fine-grat one-grained edges and details. Existing methods [3, 14, 21, 25] can be categorized into one-stage [3, 14] and two-stage [25, 21] frameworks. One-stage methods [3, 14] conduct global refinement of the whole image, which could produce blurred depth edges and details. To further enhance local details, based on the globally refined results, the prevailing refinement approaches [25, 21] adopt the two-stage tile-base bleeb manner by selecting and merging numerous patches. For example. Boost [25] proposes complex patch- sampling strategy based on the gradients of input images. PatchFusion [21] improves the sampling by shifted and tidily arranged tile placement. However, the massive patches lead to low efficiency. The excessive local information produces inconsistent depth structures or even artifacts. In this paper, we propose the Self-distileed Depth Refinement (SDDR) framework, which can predict both consistent structures and accurate details with much higher efficiency by tackling the noisy Poisson fusion problem.
Depth Refinement Datasets. Depth datasets with highly accurate annotations and edges are necessary
for refinement models. Prior arts [21, 14] utilize CG-rendered datasets [45, 44, 39, 11, 32] for accurate depth, but the realism and diversity fail match the real world. For instance, neither the UnrealStereo4K [39] nor the MVS-Synte [11] contain people, restricting the generalizability of refinement models. A simple idea for the problem
leverage natural-scene data [35, 47, 49, 5, 20]. However, different annotation methods lead to varying characteristics, e.g. sparsity of LiDAR [2, 5, 7]. inaccurate depth of structured light [55, 35, 36), and blurred edges of stereo matching [49, 47, 43]. To address the challenge, Boost [25] adopts adversarial training as guidance only with a small amount of accurately annotated real-world images. GBDF [3] employs depth predictions [51] with guided filtering [9] as pseudo-labels. Due to the inaccurate pscudo-labels and guidance, they [3, 25] produce blurred edges and details. By contrast, SDDR constructs accurate depth edge representation and edge-based guidance for self-distillation, leading to fine-gri fine-grained details and strong generalizability.
3 SDDR: Self-Distilled Depth Refinement
We present a detailed illustration of our Self-distilled Depth Refinement (SDDR) framework. In Sec. 3.1, we introduce the noisy Poisson fusion to model the depth refinement task and provide an overview to outline our approach. SDDR mainly consists of depth edge representation and edge-based guidance, which will be described in Sec. 3.2 and See. 3.3 respectively.
3.1 Noisy Poisson Fusion
Problem Statement. Based on depth maps of depth prediction models, i.e., depth predictor Na. depth refinement recovers high-resolution depth with accurate edges and details by refinement network M,. Some attempts in image super-resolution [31, 56, 26] and mudti-modal integration [19, 17, 18. 53] utilize Poisson fusion to merge features and restore details. Motivated by this, we propose to model depth refinement depth values and precise depth edges are unobtainable in real world. A general depth prediction D, whether produced by Na or N, for an input image I, can be expressed as noisy approximation of D*:
noisy Poisson fusion problem. The ideal depth D" with completely accurate
D D* + Econs + fedge
écons and ézdge denote local inconsistency and edge deformation noise to decouple depth prediction emros. Local inconsistency noise cooms represents inconsistent depth structures through regional linear transformation perturbation. Based on masked Gaussian blur, edge deformation noise <edge showcases degradation and blurring of depth edges. Refer to Appendix A.4 for details of the noises. As in Fig. 2,
3

---

RGB
Ideal Depth D + cons + Eddge
Prediction
Econs + Fedge
Depth Error D D
Figure 2: Depiction of depth errors. We utilize two samples of high-quality depth maps as ideal depth D For the predicted depth D, the combination of local inconsistency noise Ceoms and edge deformation noise fadge can approximate real depth enror D D" (the last two columns). Thus, as in the third and fourth columns, prediction can be depicted by the summation of D", Coons and Cedge-
depth errors can be depicted by combinations of fgons and fedge- Thus, considering refinement network N₂ Poisson fusion operator, depth refinement can be defined noisy Poisson fusion problem:
Do NG((N(().
VD - FDD10n++ //, 1-2 1Do0 1
s.t. min Do.37
The refined depth of N, is denoted Do- V refers to the gradient operator. Typically for depth refinement [3, 25, 21] task, input image I is resized to low-resolution I. and high-resolution for Ng. Ω represents high-frequency areas, while / 22 showcases low-frequency regions.
Motivation Elaboration. In practice, due to the inaccessibility of truly ideal depth, approximation of D' is required for training N,. For this reason, the optimization objective in Eq. 2 is divided into Ω and I - Ω. For the low-frequency I - Ω D" can be simply represented by the ground truth D of training data. However, illustrated in Sec. 2, depth annotations inevitably suffer from imperfect edge quality for the high-frequency f2 It is essential to generate accurate approximations of ideal depth boundaries as training labels, which are robust <cons and éedge- Some prior arts adopts synthetic depth [39, 11, 32] for higher edge quality, while leading to limited generalization capability with blurred predictions in real-world scenes. To leverage real depth data [35, 47, 46, 49, 5, 20], GBDF [3] employs depth predictions [51] with guided filter as pseudo-labels, which still contain significant noises and result in blurred depth. Besides, optimization of 12 is also ignored. Kim et al. [14] relies on manually annotated Ω regions as input. GBDF [3, 30, 29] omits the selection of 52 and supervises depth gradients on the whole image. Inaccurate approximations of D and inappropriate division of Ω lead to limited robustness to local inconsistency noise and edge deformation noise.
Method Overview. To address the challenges, shown in Fig. 3, we propose SDDR framework with two main components: depth edge representation and edge-based guidance. To achieve low- noise approximations of V D"
construct the depth edge representation G, through coarse-to-fine self-distillation, where 8 E 21.2..... ... S} refers to iteration numbers. The input image is divided into several windows with overlaps from coarse to fine. For instance, we denote the high-frequency area of a certain window w in iteration as ny and the refined depth of Nr as D In this way, the self-distilled optimization of depth edge representation G2 can be expressed follows:
D ~ D Eoons Codgo
Σ / S3
min C,
G
During training, depth edge representation G is further optimized based on the gradient of current refined depth D.. The final edge representation Gs of the whole image will be utilized as the pseudo- label to supervise the refinement network M, after S iterations. SDDR can generate low-noise and robust edge representation, mitigating the impact of Cooms and Cedge (More results in Appendix A.1).
With G as the training label, the next is to enforce N, with robustness to the noises, achieving consistent structures and meticulous boundaries. To optimize Nr, we propose edge-based guidance as equivalent optimization objective to noisy Poisson fusion problem, which is presented by:
// VDD Gs| 02 + /1
min 2
Do - Di an.
(4)
4

---

Self-Distilled Depth Refinement
Initial Step (s=0)
Coarse-to-fine
Final Step (s=5)
1 Regloe Maskon M
Edge Refinement
Refiried Edge Gs
N
Refiined Go
Refined Depth Dy
Refineli Depth Do
NW
S-2
SP1
High
Fudge
Low Secuss Kedge
Wd
Depth Predictor
Supervision
N₂ DD
Refinement Network
Gg Oe Gy Equine Frequency
Quantle Sampling
Clusterigg
Edge based Guidance
Gg < fave Gg Frequency
Self-Distillation Training Noisy Fusion
Go (P.)
Gr[Pa]
Edgo-based based Fusion Error
Edge -ulleddddd Gradient Error
Figure 3: Overview of self-distilled depth refinement. SDDR consists of depth edge representation and edge-based guidance. Refinement network N₁, produces initial refined depth Do. edge representa- tion Gg, and learnable soft mask Ω of high-frequency areas. The final depth edge representation G's updated from coarse to fine as pseudo-labels The edge-based guidance with edge-guided gradient loss and edge-based fusion loss supervises N, achieve consistent structures and fine- fine-grained edges.
For the second term of / 7-Q, Ω2 we adopt depth annotations D gt as the approximation of D For the first term, with the generated Gs pseudo-label, of VD*, we propose edge-guided gradient loss and edge-based fusion loss to optimize Do and f2 predicted by N,,. The edge-guidee gradient loss supervises the model to consistently refine depth edges with local scale and shift alignment. The edge-base base- fusion loss guides N, to adaptively fuse low- and high-frequency features based on the learned soft region mask Ω2 achieving balanced consistency and details by quantile sampling.
Overall, when depth maps are better refined under the cdse-based guidance, the edge representation also becomes more accurate and noise-free with the carefully designed coarse-to-fine manner. The self-distillation paradigm can be naturally conducted based on the noisy Poisson fusion, enforcing our model with strong robustness against the local inconsistency noise and edge deformation noise.
3.2 Depth Edge Representation
To build the self-distilled training paradigm, the prerequisite is to construct accurate and low- noise depth edge representations as pseudo-labels. Meticulous steps are designed to generate the representations with both consistent structures and accurate details.
Initial Depth Edge Representation. We generate an initial depth edge representation based on the global refinement results of the whole image. For the input image T, obtain the refined depth results Do from N, as in Eq. 2. Depth gradient G, V Do is calculated as the initial representation. An edge-preserving filter [38] is applied on Go to reduce noises now-frequency area Ω. With global information of the whole image, Go can preserve spatial structures and depth consistency. It also incorporates certain detailed information from the high-resolution input H. To enhance edges and details in high-frequency region Ω we conduct coarse-to-fine edge refinement in the next step.
Coarse-to-fine Edge Refinement. The initial Do is then refined from course to fine with S iterations to generate final depth edge representation. For a specific iteration 8 E ... S.. we uniformly divide input image into (s + 1)² windows with overlaps. We denote a certain window w in iteration of the input image as I The high-resolution H. is then fed the depth predictor Ng. D represents the depth refinement results of the corresponding window w in the previous iteration s -- The refined depth D."" of window w current iteration s as Eq. 3 can be obtained by Na and NG::
D.d N..(D"..NN(44"")
(1.2....... S).
(5)
After that, depth gradient V is used to update the depth edge representation. The coarse-to- manner achieves consistent spatial structures and accurate depth details with balanced global and regional information. In the refinement process, only limited iterations and windows are needed. Thus, SDDR achieves much higher efficiency than tile-based methods [25, 21], as shown in Sec. C.
Scale and Shift Alignment. The windows are different among varied iterations. Depth results and edge labels on corresponding window of consecutive iterations could be inconsistent in depth scale
5

---

Depth Initial Depth D
Depth D
by Quantlic-smplled suantill-sappled 6y
Figure 4: Visualization of intermediate results. We visualize the results of several important steps within the SDDR framework. The quantile sampling utilizes the same color map as in Fig. 3.
and shift. Therefore, alignment is required before updating the depth edge representation:
(31, Po) = arg min [1(3 V. + +b) G--
,1,80
(6)
G = B. + 8o
where 31 and B0 are affine transformation coefficients scale and shift respectively. The aligned G represents the depth edge pseudo-labels for image patch I generated from the refined depth D At last, after iterations, we can obtain the pseudo-label Gg as the final depth edge representation for self-distillation. For better understanding, we showcase visualization of Do. Ds, and Gs in Fig. 4.
Robustness to Noises. In each window, we merge high-resolution N₂ (H.))) to enhance details and suppress Cddge- Meanwhile, coarse-to-fine window partitioning and scale alignment mitigate econs and bring consistency. Thus, Gg exhibits strong robustness to the two types of noises by self-distillation.
3.3 Edge-based Guidance
With depth edge representation Gs as pseudo-label for self-distillation, we propose the edge-based guidance including edge-guided gradient loss and edge-based fusion loss to supervise N,..
Edge-guided Gradient Loss. We aim for fine-grained depth by one-stage refinement, while the two-stage coarse-to-fing manner can further improve the results. Thus, odge-guided gradient loss instructs the initial Du with the accurate Gg. Some problems need to be tackled for this purpose.
As .V, has not converged in the early training phase, G5 is not sufficiently reliable with inconsistent scales and high-level noises between local areas. Therefore, we extract several non-overlapping regions Pn. n E (1,2. ..... Ng) with high gradient density by clustering [8], where Ng represents the number of clustering centroids. The edge-guided gradient loss is only calculated inside P. with scale and shift alignment. By doing so, the model can focus on improving details in high-frequency regions and preserving depth structures in flat areas. The training process can also be more stable. The edge-guided gradient loss can be calculated by:
N₂ Ng Σ 11B [Px Gs [Palll
1
Cyrad=
where 32 and 3o are the scale and shift coefficients similar to Eq. 6. We use []] to depict mask fetching operations, i.e., extracting local area Pv, from Go and Gs. With the edge-guided gradient loss, SDDR predicts refined depth with meticulous edges and consistent structures.
Edge-based Fusion Loss. High-resolution feature Fи extracted from // brings finer details but could lead to inconsistency, while the low-resolution feature Fz from can better maintain depth structures. Nr should primarily rely on Fz for consistent spatial structures within low-frequency I Ω, while it should preferentially fuse Fir for edges and details in fegh-frequency areas Q. The fusion of F,, and Fi noticeably influence the refined depth. However, prior arts [14, 3, 25] adopt manually-annotate regions as fixed masks or even omit Q as the whole image, leading to inconsistency and blurring. To this end, we implement Q2 as a learnable soft mask, with quantile sampling strategy to guide the adaptive fusion of Ft and FH. The fusion process is expressed by:
F = (1 !2) O Fi +00 FH
(8)
where O refers to the Hadamard product. Ω is the learnable mask ranging from zero one. Larger values in showcases higher frequency with denser edges, requiring more detailed information from the high-resolution feature F,. Thus, Ω can naturally serve the fusion weight of Fr. and Fu.-
To be specific, we denote the lower quantile of Gg as la i.e., P(X < L,|X E Gs) = {Gs < (a) indicates flat areas with low gradient magnitude, while {Gss > t1-a) represents high-frequency

---

Middlchury202 2021
Multiscopic
Hypersim
Predictor
Method
6xt
REL.
ORD
61t
REL
ORD1
611
REL.
ORD_
MiDaS [30]
0.868
0.117
0.381
0.839
0.130
0.292
0.781
0.169
0.314
Kim et at. [14]
0.864
0.120
0.377
0.839
0.130
0.293
0.778
0.175
0.344
MiDaS
Graph-GDSR [4]
0.865
0.121
0.350
0.539
0.130
0.292
0.781
0.169
0.345
GBDF [3]
0.871
0.115
0.305
0.841
0.129
0.289
0.787
0.168
0.338
Ours
0.879
0.112
0.299
0.852
0.122
0.267
0.791
0.166
0.318
LeReS [S1]
0.847
0.123
0.326
0.863
0.111
0.272
0.853
0.123
0.279
Kim al. [14]
0.846
0.124
0.328
0.860
0.113
0.286
0.850
0.125
0.286
LcReS
Graph-GDSR [4]
0.847
0.124
0.327
0. 0.862
0.111
0:273
0.852
0.123
0.281
GBDF [3]
0.316
0.865
0.270
0.857
0.121
0.273
Ours
0.862
0.120
0,305
0.870
0.108
0.259
0.862
0.120
0.273
ZoeDepth [1]
0,900
0,101
0.225
0
0.097
0,205
0.927
0.088
0.198
Kim al. [14]
0.896
0.107
0.228
0.890
0.099
0.204
0.923
0.091
0.204
ZoeDepth
Graph-GDSR [4]
0.901
0.103
0.226
0.895
0.096
0.208
0.926
0.089
0.199
GBDF [3]
0.899
0.105
0.226
0.897
0.096
0:207
0.925
0.089
0.199
Ours
0.905
0.100
0.218
0,904
0.092
0.199
0.930
0.086
0.191
Table 1:
Comparisons with one-stage methods. As prior arts [14, 4, 3]. we conduct evaluations with different depth predictors [30, 51, 1]. For each predictor, we report the initial metrics and results of refinement methods. Best performances with each depth predictors [30, 51, 1] are in boldface.
regions. Ω should be larger in those tigh-frequency areas {Gss > ) and smaller in the flat regions {Gs < fa). This suggests that Gs and 52 should be synchronized with similar data distribution. Thus, if we define the lower quantile of Ω as Ta i.e., P(X < Ta X E Q) a, arbitrary pixel i E {Gs < ta in flat regions should also belong to {Ω < Ta with a lower weight for FH, while the pixel i E (Gs > h ) in frequecyyy areas should be contained in (S2 Ti-a) for more detailed information. The edge-based fusion loss can be depicted as follows:
N₂ Now ΣΣ
max Ω2 TTnaa), max(0, Ti-=a Ω2),
E [Gs (nea), i E {Gs > ti-nsa),
Ljusion
where Np is the pixel number. We supervise the distribution of Ω with lower quantiles Taso and Ti-noa, nt E 21,2..... J. Therefore, pixels with larger deviations between Gs and Ω will be penalized more heavily. Taking the worst case i & (Ω < ). the error for the pixel will be accumulated for N times from a to N * a. Cussio enforces SDDR with consistent structures (low Geons noise) in St and accurate edges (low Fodge noise) in Ω. The visualizations of quantile-sample Gs and Q are presented in Fig. 4.
example, if E {Gs [N.. but
adgeebased guidance for self-distillation, the overall loss L for training N, is calculated as Eq. 10. Lgt supervises the discrepancy between Do and ground truth Di with affinity-invariann loss [30, 29J. See Appendix A for implementation details of SDDR.
Finally, combining Lgrad and fusion
C Lot + d Agrod + X2C
(10)
4 Experiments
To prove the efficacy of Self-distilled Depth Refinement (SDDR) framework, we conduct extensive experiments on five benchmarks [34, 52, 32, 15, 40] for indoor and outdoor, synthetic and real-world
Experiments and Datasets. Firstly, we follow prior arts [3, 25, 14] conduct zero-shot evaluations on Middlebury2021 [34], Multiscopic [52], and Hypersim [32]. To showcase our superior generaliz- ability, we compare different methods DIML [15] and DIODE [40] with diverse natural scenes. Moreover, we prove the higher efficiency of SDDR and undertake ablations on our specific designs.
Evaluation Metrics. Evaluations of depth accuracy and edge quality are necessary for depth refinement models. For edge quality, we adopt the ORD and D'R metrics following Boost [25]. For depth accuracy, we adopt the widely-used REL and 6; (i = 1, 2,). See Appendix B for details.
4.1 Comparisons with Other Depth Refinement Approaches
Comparisons with One-stage Methods. For fair comparisons, we evaluate one-stage [14, 4, 3] and two-stage tile-based [25, 21] approaches separately. The one-stage methods predict refined depth
7

---

Middlcbury2021
Multisoopic
Hypersim
Predictor
Meihed
6,t
REL,
ORD
611
RELJ
ORD
REL.
ORDJ
MiDaS [30]
0.868
0.117
0.384
0.839
0.130
0.292
0.781
0.169
0.344
MiDaS
Boost [25]
0.870
0.118
Ours
0.871
0.115
0.303
0.858
0.120
0.263
0.799
0.154
0.322
LcRe5 [51]
0.847
0.123
0.326
0.863
0.111
0.272
0.853
0.123
0.279
LcReS
Boost [25]
0.844
0.131
0.325
0,860
0.112
0.278
0.865
0.118
0.272
Ours
0
0.123
0.309
0.870
0.109
0.268
0.858
0.123
0.271
ZoeDepth
0.900
0.896
0.205
ZeeDepth
Boost [25]
0.911
0.090
0.210
0.910
0.084
0.197
0.926
0,089
0.193
PatchFusion [21]
0.887
0.102
0.211
0.908
0.095
0.212
0.881
0.116
0.258
Ours
0.913
0.09
0.202
0.908
0.091
0.197
0,933
0,083
0.189
Table 2: Comparisons with two-stage methods. PatchFusion [21] only adopts ZoeDepth [1] as the fixed baseline, while other approaches are pluggable for different depth predictors [30, 51, 1].
RGB
LeReS
Kim et
Ours(one-stage)
Figure 5: Qualitative comparisons of one-stage methods natural scenes. LeReS [51] used as the depth predictor. SDDR predicts sharper depth edges and more meticulous details than prior arts [3. 14], egg,, fine-grained predictions of intricate branches. Better viewed when zoomed in.
Ours(two-stage) Figure 6: Qualitative comparisons of two-stage methods on natural scenes. ZoeDepth [1] is adopted as the depth predictor. The SDDR with coarse-to-fine edge refinement predict more accurate depth edges and more consistent spatial structures than tile-based methods [21, 25].
RGB
ZoeDepth
PatchFusion
Boost
based on the whole image. SDDR conducts one-stage refinement without the coarse-to-fine manner during inference. Comparisons
Middlebury2021 [34], Multiscopic [52], and Hypersim [32] shown in Table 1. As prior arts [14, 4, 3], we use three depth predictors MiDaS [30], LeReS [51], and ZoeDepth [1]. Regardless of which depth predictor is adopted, SDDR outperforms the previous one-stage methods [14, 4, 3] in depth accuracy and edge quality on the three datasets [34, 52, 32). For instance, method shows 6.6% and 20. 7% improvements over Kim al. [14] for REL and ORD with MiDaS [30] on Middlebury2021 [34], showing the efficacy of our self-distillation paradigm.
Comparisons with Two-stage Tile-based Methods. Two-stage tile-based methods [25, 21] conduct local refinement on numerous patches based on the global refined depth. SDDR moves away from the tile-based manner and utilizes coarse-to-fine fine edge refinement to further improve edges and details. As Table 2, SDDR with the coarse-to- coarse-tto-fin no-fine manner shows obvious advantages. For example, compared with the recent advanced PatchFusion [21], SDDR achieves 5.2% and 26.7% improvements for ó1 and ORD with ZeeDepth [1] on Hypersim [32]. To be mentioned, PatchFusion [21] uses ZoeDepth [1] as the fixed baseline, whereas SDDR is readily pluggable for various depth predictors [30, 51, 1].
Generalization Capability on Natural Scenes. We prove the superior generalization capability of SDDR. In this experiment, we adopt LeReS [51] the depth predictor. DIML [15] and DIODE [40]

---

DIML
DIODE
Depih Accuracy(:.)
REI.
GRD
178..
REL.
ORD)
LPRJ
0.855
Kim et al [14]]
0.802
0.100
0.243
0.301
0.880
0.105
0.525
0.713
Graph GDSR [4]
0.901
0.101
0.213
0.300
0.850
0.104
0.325
0.050
Boost (25)
GBDF [3]
0.897
0.906
0.100
0.108
0.274
0.239
0.438
0.267
0.8522
0.854
0.105
0.305
0.345
0.522
0.640
0.673
0.300
Okn
Ours
0.926
0.098
0.221
0.220
0,900
0.098
0.293
0.637
0.85
Notse Level
Table 3: Comparisons of model generalizability. We con- duct zero-shot evaluations on DIML [15] and DIODE [40] datasets with diverse in-the-wild scenarios to compare the generalization capability. We adopt LeReS [51] the depth predictor for all the compared methods in this experiment.
Figure Robustness against noises,
X-axis shows noise level of Eçons +
Codges. With higher noises, our SDDR
is more robust with less performance degradation than the prior GBDF [3].
Methed
Iransy taas
D'R
GBDF
HRWSI
0.502
Method
61
REL.
ORD
D
Ext
Eyeat
Cjssss
61
REL.)
ORD
D'R
Ouu
HRWSI (49)
0.500
0.131
0,30
0.322
8
0.859
0.125
0.513
0.235
0.54
0.12:
0.315
0.280
(c) Effectiveness
0.500
0.122
0.399
0.225
/
0.65
0.122
0.397
0.250
0.862
0.303
0.130
0.120
0.377
0.305
0.219
0,216
0.362
0.809
0.120
0.120
0.811
0.305
0.316
0:220
Method
6,t
RELL
ORD1
D'R.
GBDDE [3]
0.852
0.122
0.306
0.26M
(a) Course-to-line Edge Refinement
(b) Sdge-based Guidance
CIRD (nr Gg)
0.858
0.122
0.307
0.130
(d) Transferability
Table 4: Ablation Study. All ablations are on Middlebury2021 [34] with depth predictor LeReS [51].
datasets are used for zero-shot evaluations, considering their diverse in-the-wild indoor and outdoor scenarios. As in Table 3, SDDR shows least 5.7% and 9.0% improvements for REL and ORD on DIODE [40]. On DIML [15] dataset, our approach improves D²RR ORD, and 61 by over 17.6%, 7.5%, and 2.0%. The convineing performance proves our strong robustness and generalizability, indicating the efficacy of our noisy Poisson fusion modeling and self-distileed training paradigm.
Qualitative Comparisons. We present visual comparisons of one-stage methods [14, 3] on natural scenes in Fig. 5. With our low-noise depth edge representation and edge-based guidance, SDDR predicts sharper depth edges and details, e.g., the fine-grae fine-grained predictions of intricate branches.
The visual results of two-stage approaches [25, 21] are shown in Fig. 6. Due the excessive fusion of detailed information, tile-based methods [25, 21] produce structure disruption, depth inconsistency, or even noticeable artifacts, e-g- disrupted and fuzzy structures of the snow-covered branches. By contrast, SDDR can predict more accurate depth edges and more consistent spatial structures.
Robustness against noises. As in Fig. 7, we evaluate SDDR and GBDF [3] with different levels of input noises. As the noise level increases, our method presents less degradation. The stronger robustness against the écons and Eddges noises the essential reason for all our superior performance.
Model Efficiency. SDDR achieves higher efficiency. Two-stage tile-based methods [25, 21] rely on complex fusion of extensive patches with heavy computational overhead. Our coarse-to-fing manner noticeably reduces Flops per patch and patch numbers as in Fig. 1. For one-stage methods [4, 3, 14], SDDR adopts more lightweight N, with less parameters and faster inference speed over the previous GBDF [3] and Kim et al. [14]. See Appendix C. for detailed comparisons of model efficiency.
4.2 Ablation Studies
Coarse-to-fine Edge Refinement. In Table 4a, we adopt the coarse-to-fine manner with varied iterations. S = 0 represents one-stage inference. Coarse-to-fin fine refinement brings more fine-gn tne-nnrieed edge representations and refined depth. We set S for the SDDR with two-stage inference.
Edge-based Guidance. In Table 4b, we evaluate the effectiveness of edge-based guidance. Lyrad focuses consistent refinement of depth edges. Lfusion guides the adaptive feature fusion of low- and high-frequency information. With Lyt as the basic supervision of ground truth, adding Lgrad and C fusion improves DRR by 10.0% and REL by 3.2%. showing the efficacy of edge- based guidance.
Effectiveness of SDDR Framework. As in Table 4c, we train SDDR with the same HRWSI [49] as GBDF [3] for fair comparison. Without the combined training data Appendix 1, SDDR still improves D'R R and ORD by 13.9% and 2.2% over GBDF [3], proving superiority convincingly.

---

Transferability. We hope our depth edge representation Gs can be applicable to other depth refinement models. Therefore, in Table 4d, we directly train GBDF [3] combining the depth edge representation produced by the trained SDDR. The depth accuracy and edge quality are improved over the original GBDF [3], indicating the transferability of Gş in training robust refinement models.
Conclusion
In this paper, we model the depth refinement task as noisy Poisson fusion problem. To enhance the robustness against local inconsistency and edge deformation noise, we propose Self-distilled Depth Refinement (SDDR) framework. With the low-noise depth edge representation and guidance, SDDR achieves both consistent spatial structures and meticulous depth edges. Experiments showcase our stronger generalizability and higher efficiency over prior arts. The SDDR provides new perspective for depth refinement future works. Limitations and broader impact are discussed in Appendix A.5,
Acknowledgement This work is supported by the National Natural Science Foundation of China under Grant No. 62406120.
References
[1] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:230.112288, 2023. 1, 2, 8, 20, 21
[2] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11618 11628,, 2020. 2,
[3] Yagiao Dai, Renjiao Yi, Chenyang Zhu, Hongjun He, and Kai Xu. Multi-resolution monocular depth map fusion by self-super gradient-based composition. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 488 496, 2023. 1, 3, 4, 6, 7, 8, 9, 10, 15, 16, 18, 19, 20, 21
[4] Riccardo De Lutio, Alexander Becker, Stefano D' Aronco, Stefania Russo, Jan D Wegner, and Konrad Schindler. Learning graph regularisation guided super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1979-199999 1988, 2022. 1, 7, 8, 9, 19, 20
[5] Andreas Geiger. Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(1 (((1)):12111223 2013. 2, 3,4
[6] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, volume 27, 2014.
[7] Vitor Guizilini, Rares Ambrus, Sudeep Pillai. Allan Raventos, and Adrien Gaidon. 3d packing for self- supervised monocular depth estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2482-2491, 2020. 2,3
[8] John A Hartigan and Manchek A Wong. Algorithm 136: A k-means clustering algorithm. Journal of the royal statistical society. series (applied statistics), 28(1):100 108, 1979. 6, 15
[9] Kaiming He, Jian Sun, and Xiaoou Tang. Guided image filtering. IEEE transactions puttern analysis and machine intelligence, 35(6)1397-1449,, 2012. 3
[10] Yiwen Hua, Puneet Kohli, Pritish Uplavikar, Anand Ravi. Saravana Gunaseelan, Jason Orozco, and Edward
Li. H0lopix50k: A large-scale in in-the-will stereo image dataset. urXiv preprint 2rXiv:2003.11172, 2020. 2.
18
[11] Po-Han Huang. Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi-view stereopsis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2821-2830 2018 2,3,4,18
[12] Eddy 1lg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 2462-24771 2017. 18
10

---

[13] John Kessenich, Graham Sellers, and Dave Shreiner. OpenGL Programming Guide: The official guide to learning OpenGL, version 4.5 with SPIR-V. Addison- Wesley Professional, 2016.
[14] Soo Ye Kim, Jianming Zhang, Simon Niklaus, Yifei Fan, Simon Chen, Zhe Lin, and Munchurl Kim. Layered depth refinement with mask guidance. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3855-3865, 2022. 1, 3, 4, 6, 7,8, 9, 14, 16, 19, 20,
[15] Youngjung Kim, Hyungjoo Jung, Dongbo Min, and Kwanghoon Sohn. Deep monocular depth estimation via integration of global and local predictions. IEEE Transactions on Image Processing, 27(8) 31144144,
2018. 2, 7, 8, 9, 18, 20, 21
[16] Tobias Koch, Lukas Liebel, Friedrich Fraundorfer, and Marco Körner. Evaluation of cnn-based single- image depth estimation methods. In Laura Leal-Taixé and Stefan Roth, editors. Proceedings of the European Conference on Computer Vision (ECCV) Workshops, volume 11131, pages 331-348 Springer, 2018. 18
[17] Guchong Li. An iggm-based poisson multi-hernoulli filter and its application distributed multisensor fusion. IEEE Transactions on Aerospace and Electronic Systems, 58(4):3666 3677,, 2022. 3
[18] Jiaqi Li, Yiran Wang, Zihao Huang, Jinghong Zheng, Ke Xian, Zhiguo Cao, and Jianming Zhang. Diffusion- augmented depth prediction with sparse annotations. In Proceedings of the 31 ACM International
Conference on Multimedia, pages 2865-2876, 2023. 3
[19] Jing Li, Hongtao Huo, Chenhong Sui, Chenchen Jiang, and Chang Li. Poisson reconstruction based fusion of infrared and visible images via saliency detection. IEEE Access, 7:20676-20688,, 2019. 3
[20] Zhenggi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. 3,4 4
[21] Zhenyu Li, Shariq Farooq Bhat, and Peter Wonka. Patchfusion: An end-to-end tile- based framework for high-resolution monocular metric depth estimation. arXiv preprint arXiv:2312.0228. 2023. 1,2, 3, 4, 5,
7, 8, 9. 14, 19, 20, 21
[22] Guosheng Lin, Anton Milan, Chunhua Shen, and Reid. Refinenet: Multi-pani refinement networks for high-resolution semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1925-1934, 2017. 16
[23] Tsung-Ye Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2117 1125- 2017. 16, 17
[24] Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf. Consistent video depth estimation. ACM Transactions Graphics (ToG), 39(4):71-1, 2020. 1
[25] S Mahdi H Miangoleh, Sebastian Dille, Long Mai, Sylvain Paris, and Yagiz Aksoy. Boosting monocular depth estimation models high-resolution via content-adaptive multi-resolution merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9685-9694, 2021. 1, 2, 3, 4, 5, 6, 7, 8, 9, 14, 15, 16, 18, 19, 20, 21
[26] Antigoni Panagiotopoulou and Vassilis Anastassopoulos. Super-resolution image reconstruction techniques: Tradc-offs between the data-fidelin and regularization terms. Information Fusion, (3(3):185-11)) 195, 2012. 3
[27] Juewen Peng, Zhiguo Cao, Xianrui Luo, Hao Lu, Ke Xian, and Jianming Zhang. Bokehme: When neural rendering meets classical rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 16283 16292, 2022.
[28] Juewen Peng. Jianming Zhang, Xianrui Luo, Hao Lu, Ke Xian, and Zhiguo Cao. Mpib: An mpi-based bokeh rendering framework for realistic partial occlusion effects. In European Conference Computer Vision (ECCV), pages 590-607. Springer, 2022.
[29] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12179-12188,
2021. 4,7
[30] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot pross-dttaed transfer. IEEE transactions on pattern analysis and machine intelligence, 44(03):1623 -203,1676631,7 2020. 1, 2, 3, 4, 8, 20
11

---

[31] Haoyu Ren, Amin Kheradmand, Mostafa EI- El-KKamyy Shuangquan Wang, Dongwoon Bai, and Jungwon Lee. Real-world super-resolution using generative adversarial networks. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 436 2020.
[32] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV),
pages 10912-10922, 2021. 3,2,3,4, 7, 18
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffussin models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2022.
[34] Daniel Scharstein, Heiko Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nešić, Xi Wang, and Porter Westling. High-resolution stereo datasets with subpixel- accurate ground truth. In Pattern Recognition: 36th German Conference, GCPR 2014, Münnster Germany, September 2-5, 2014, Proceedings 36. pages 31 42. Springer, 2014. 2, 7,7,89, 9, 17, 19, 20, 21
[35] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision (ECCV), pages 746-760. Springer, 2012. 2, 3,4 4
[36] Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of rgb-d slam systems. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 573-580. IEEE, 2012. 2,
[37] Zhanghao Sun, Wei Ye. Jinhui Xiong, Gyeongmin Choe, Jialiang Wang, Shuochen Su, and Rakesh Ranjan. Consistent direct time-of- flight video depth super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5075-5085, 2023.
[38] Carlo Tomasi and Roberto Manduchi. Bilateral filtering for gray and color images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 839- 846. IEEE, 1998.
[39] Fabio Tosi, Yiyi Liao, Carolin Schmitt, and Andreas Geiger. Smd-nets: Stereo mixture density networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8942-8952, 2021. 2, 3, 4,4,1 18
[40] Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z. Dai, Andrea F. Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R. Walter, and Gregory Shakhnarovich. DIODE: A Dense Indoor and Outdoor DEpth Dataset. CoRR, 2019. 2, 7, 8, 9, 18, 20
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, volume 30, 2017. 16
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Adv. Neural Inf. Process. Syst., pages 5998-6008, 2017. 16
[43] Chaoyang Wang. Simon Lucey, Federico Perazzi, and Oliver Wang. Web stereo video supervision for depth prediction from dynamic scenes. In IEEE International Conference on 3D Vision (3DV), pages 348 357, IEEE, 2019. 2, 3, 18
[44] Qiang Wang. Shizhen Zheng, Qingsong Yan, Fei Deng, Kaiyong Zhao, and Xiaowen Chu. Irs: A large naturalistic indoor robotics stereo dataset to train deep models for disparity and surface normal estimation. In IEF.F. International Conference Multimedia and Expo (ICME), pages 1-6. TEEE Computer Society,
2021. 2, 3, 18
[45] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafci Hu, Ashish Kapoor, and Sebastian Scherer. Tartanair: A dataset to push the limits of visual slam. In IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 4909-4916. IEEE, 2020. 2, 18
[46] Yiran Wang, Min Shi, Jiaqi Li, Chaoyi Hong. Zihao Huang, Juewen Peng. Zhiguo Cao, Jianming Zhang, Ke Xian, and Guosheng Lin. Nvds+: Towards efficient and versatile neural stabilizer for video depth estimation. IEEE Transactions Pattern Analysis and Machine Intelligence, pages 1-18, 2024.
12

---

[47] Yiran Wang, Min Shi, Jiaqi Li, Zihao Huang, Zhiguo Cao, Jianming Zhang, Ke Xian, and Guosheng Lin. Neural video depth stabilizer. In Proceedings the IEEE/CVF International Conference Computer Vision (ICCV), pages 9466-9476, 2023. 2, 3,3,4,1
[48] Ke Xian, Chunhua Shen, Zhiguo Cao. Hao Lu, Yang Xiao, Ruibo Li, and Zhenbo Luo. Monocular relative depth perception
web stereo data supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 311-320, 2018. 2
[49] Xian, Jianming Zhang, Oliver Wang, Long Mai, Zhe Lin, and Zhiguo Cao. Structure-guided ranking loss for single image depth prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 608 617, 2020. 2, 3, 4, 9, 18, 19, 20, 21
[50] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077- 2021. 16
[51] Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3d scene shape from single image. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 204-213, 2021. 2, 3, 4,
19, 20, 21
[52] Weihao Yuan, Yazhan Zhang. Bingkun Wu, Siyu Zhu, Ping Tan, Michael Yu Wang, and Qifeng Chen. Stereo matching by superviin of multiscopic vision. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5702-5709. IEEE, 2021. 2, 7, 8,8,7 17
[53] Junrui Zhang, Jiaqi Li, Yachuan Huang, Yiran Wang, Jinghong Zheng, Liao Shen, and Zhiguo Cao. Towards robust monocular depth estimation in non-lambertian surfaces. arXiv preprint arXiv:2408806608) 2024. 3
[54] Lymin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 3836-3847, 2023.
[55] Zhengyou Zhang. Microsoft kinect sensor and its effect. IEEE multimedia, 19(2):4-10, 2012. 2, 3, 18
[56] Changzhong Zou and Youshen Xia. Bayesian dictionary learning for hyperspectral image super resolution in mixed poisson-gaussian gaussian noise. Signal Processing: Image Communication, 60:29-41, 2018. 3
13

---

Depth Predictor (L)
Depth Predictor (H)
Initial Depth
Depth Edge Go
Window Partitioning
Window
Refined Window w
Depth D₁
Depth Edge G₃
Window Partitioning
Window
Refined Window
Depth D2
Depth Edge G2
VS
Window Partitioning
Window w
Refined Window w
Depth D3
Depth Edge G3
Figure 8: Visualizations of coarse-to-fine edge refinement. We present coarse-to-fine results of
steps 8 = 0, 1,2,3. For 8 = O, we showcase the low- and high-resolution predictions Na(L) and (H) of the depth predictor, along with the initial refined depth Do and edge representation Go- For = 1, 1, 3. we present the window partitioning on the previous D,-1. the previous depth D on certain window refined depth D. the window refined depth D, of the whole image. and the depth edge representation G2 generated on the current step.
A More Details on SDDR Framework
A.1 Depth Edge Representation
Coarse-to-fine Edge Refinement. In Sec. 3.2, line 169 of main paper, propose the coarse-to-fine edge refinement generate accurate and fine-grained depth edge representation Gs. Here, we provide visualizations of the refinement process in Fig. 8. For the initial global refinement stage showcase the results the depth predictor at low and high inference resolutions, i.e., Na and (.))... Our refined depth Do presents both depth consistency and details. For 1, 3, the refined depth maps and edge representations are noticeably improved with finer edges and details. The final depth edge representation G (S 3) with lower local inconsistency noise and edge deformation noise is utilized as pseudo-labe for the self-distillation training process.
0.
Adaptive Resolution Adjustment. Adaptive resolution adjustment is applied to the low and high- resolution input and H. We denote resolutions of and as I and h, which play a crucial role in refined depth and need to be chosen carefully. Higher resolutions will bring finer details but could lead to inconsistent depth structures due the limited receptive field of Ng. Previous works [14, 25, 21] upscale images or patches to excessively high resolutions for more details, resulting evident artifacts in their refined depth maps with higher levels of inconsistency noises Econs On the other hand, if /t is too low, edge and detailed information cannot be sufficiently preserved in (a leading to exacerbation of edge deformation noise Gedge with blurred details in the refined depth. Such unacceptable in depth edge representations for training models. Therefore,
errors and artifacts adaptively adjust resolutions and h, considering both the density of depth edges and the training resolution of depth predictor Na--
For image window IH we generally set the low-resolution input I. as the training resolution i of Ng. If we denote the original resolution of I as r.d SDDR adaptively adjusts the high resolution
14

---

448
11568
448
1024
Depth Ag(l)
Ad(M)) Depth 0g
Mach/Window Besult
Patch/Window Rasult
Final Depth
Figure 9: Adaptive resolution adjustment. compare the effects of inference resolutions with Boost [25]. The numbers in the corner of the second and third columns represent the chosen inference resolution. We relieve the artifacts Boost [25] by adaptive resolution adjustment.
3
RGB
Initial Go
Pseudo-label Gs
Clustered regions
Figure 10: Edge-guided gradient error. Lgrad focuses on high-frequency areas Pn extracted by clustering with more details. The flat regions are not constrained preserve depth consistency.
ht for the certain window as follows:
mean(f, ) mean (VNN(L
mean VD 1
(1I)
mean VD,-1l)
where a is priori parameter for depth predictor Na, averaging the gradient magnitude the depth annotations its sampled training data. The second term embodies adjustments according to depth edges. Assuming mean( (Vaa I)) < a, indicates that the current window area contains lower edge intensity density than the training data of Ng. In this case, will appropriately decrease h" from mean(f, r. to maintain the similar density of detailed information as the training stage of the depth predictor. The third term portrays adjustments based the discrepancy of edge intensity between the window area and the whole image. To be mentioned, for the generation of the initial edge representation Go. the third term is set to ineffective one. Lg is equivalent L with the whole image as the initial window w.
We present visual results with different resolutions to prove the effectiveness of our design. As shown Fig. 9, considering the training data distribution and the edge density, the inference resolution adaptively adjusted to smaller one compared to Boost [25] (1024 versus 1568). In this way, our SDDR achieves better depth consistency and alleviates the artifacts produced by prior arts [3, 25].
A.2 Edge-based Guidance
Edge-guided Gradient Error. In line 192, Sec 3.3 of the main paper, mention that we use clustering to obtain several high-frequency local regions to compute our adge-guided gradient loss. Here, we elaborate on the details. K-means clustering [8] utilized obtain the edge-dense areas. Specifically, we binarize the edge pseudo-label, setting the top 5% pixels to one and the rest to zero. Next, we employ k-means clustering on the binarized labels to get several edge-dense areas with the centroid value as one. The clustered areas are shown in the fourth column of the Fig. 10. Our edge-guided gradient loss supervises these high-frequency regions to improve depth details. The depth consistency in flat areas can be preserved without the constraints of depth edges.
15

---

Edge Region
6g Ra-a
Ei Gy Ey-da
Flat Region
Gy
to Gg S.
la Gg fau
RGB
Region Mask f
Pseudo-label Gy
Quantile-sampled Quantile-sampled G3
Figure 11: Edge-based fusion error. We present the region mask and pseudo-label Gg before and after quantile sampling. Different colors on the right represent the range of pixel values. Guiding the Ω with Gs ensures that our model can predict balanced consistency and details by the simple one-stage inference. The use of S learnable soft mask achieves more fine-grained integration on the feature level, enhancing the accuracy of N,. This also leads to more accurate edge representation Gg in the iterative coarse-to-fine refinement process.
Refinement Network
Decoder
Restiull Connedion FA
Casa
Redion
Region
Weigrend
Artention-asedd Feature Interaction
Block
potte
shared Encoder
FFM
AG(II)
Fudion
Fl
FFM
Depth Map
(G)(.)
Eraid und connodiion F+
Figure 12: Architecture of refinement network. Some decoder layers are omitted for simplicity.
Edge-based Fusion Error. The proposed edge-based fusion loss aligns the data distribution of the learnable region mask Ω and the pseudo-label Gs by quantile sampling (Sec 3.3, line 205, main paper). Here, we provide additional visualizations for intuitive understanding. As shown in Fig. 11, we visualize the soft region mask Ω of high-frequency areas and the pseudo-label Gs with the same Gs with stronger depth edges
color map in the second and third columns. The regions highlighted and more detailed information naturally correspond to larger values in 12 to emphasize features from high-resolution inputs. We perform quantile sampling on and Gs. as depicted in the fourth and fifth columns. The legends on the right indicate the percentile ranking of the pixel values in the whole image. Our edge-based fusion loss supervises that Ω and Gg have consistent distribution for each color. In this way, Ω tends to have smaller values in flat regions for more information from low-resolution input, while the opposite is true in gigh-frequency regions. This is advantageous for the model to balance the depth details and spatial structures.
A.3 Refinement Network
We provide the detailed model architecture of the refinement network N,. As shown in Fig. 12, the refinement network adopts the U-Net architecture similar prior arts [25, 3, 14]. The depth maps from the depth predictor Na predicted in different resolutions are up-sampled to unified imput size. A shared Mit-bO [50] serves as the encoder to extract feature maps of different resolutions. The decoder gradually outputs the refined depth map with feature fusion modules (FFM) [22, 23] and skip connections. We make two technical improvements to the refinement network, including attention-bann feature interaction and adaptive weight allocation.
A Attention-based Feature Interaction. To predict refined depth maps in high resolution (e.g., 2048 x 2048), prior arts [25, 3, 14] adopt a U-Net with numerous layers (e.g., 10 layers or more) as the refinement network for sufficient receptive field. This leads to heavy computational overhead. In our case, we leverage the
mechanism [41] to address this issue.
The features of low- and high-resolution inputs extracted by the encoder [50] are denoted Fetto and pattn We stack Fattn and paltn to obtain pi for attention calculation. Positional embeddings [42]
16

---

РEg. PE, are added to F for the height and width dimensions. An additional P is used to distinguish the low- and high-resolution inputs. The attention-basee feature interaction process can be expressed as follows:
F -Stack( (Lack((Ftt (P Fh F.ttm) + PEz PEy + PE K Wh pin Q = W pmn W" p'n K /KTO/VOVV Fin.
(12)
poot Softmax
Four attention layers are included in N,. The interacted feature Foul is fed to the decoder to predict refined depth. Attention-based feature interaction achieves large receptive field with fewer layers, reducing model parameters and improving efficiency.
Adaptive Weight Allocation. The refinement network adopts adaptive weight allocation for the fusion of low- and high-resolution features with the learnable mask Q. In each decoder layer, the feature go through a convolutional block to generate f with single channel. The fused features F (line 212, main paper) and the feature from the previous layer are fused by the FFM module [23].
A.4 Noise Implementation.
For our local inconsistency noise, we segment the ideal depth D" into regular patches of size 64 x 64, with overlap of half the patch size. Considering the depth discontinuities of applying a linear transformation to the entire patch, we extract the edges from D' and apply a linear transformation each connected domain to simulate the local depth inconsistency. For edge deformation noise, we first down-sample D to the inference resolution and then restore it to the original resolution. Subsequently, we optimize certain number of Gaussian distributions around the edges of D to fit the edge deformation and blurring.
the edges, instead
The local inconsistency noise and edge deformation noise effectively model the degradation of network prediction results compared to ideal depth maps. An additional experiment on the Middlebury2021 [34] dataset also proves this point. We optimize the local inconsistency noise with the least squares method and 50,000 position-constrainee constraiied Gaussian distributions as edge deformation noise by gradient descent. The PSNR between the noisy depth (D* + + Cedge) and model predicted depth D is over 40 dB, which indicates that the difference between D and ( (D + écons + fedge) is very small. The result further demonstrates that the noises accurately model depth prediction errors (Eq. 1, main paper), similar to the visualizations in Fig. the main text.
A.5 Broader Impacts and Limitations
Although SDDR works well in general, it still has limitations. For example, more advanced mecha- nisms and structures can be explored for the refinement network in future work. For inputs under conditions with specular surfaces, low light, or weak textures, the depth predictor tends yield sub-optimal results. Although SDDR improves upon these results, the outcomes are still not perfect. Our approach exclusively utilizes publicly available datasets during the training process, thereby having no broad societal impact, not involving AI ethics, and not involving any privacy-sensitive data.
B Detailed Experimental Settings
B.1 Datasets
Evaluation Datasets. We use five different benchmarks with diverse scenarios for comparisons. The descriptions of our evaluation datasets follows:
Middlebury2021 [34] comprises 48 RGB-D pairs from 24 real indoor scenes for evaluating
stereo matching and depth refinement models. Each image in the dataset is annotated with dense 1920 1080 disparity maps. We use the whole set of Middlebury200
[34] for testing.
Multiscopic [52] includes a test set with 100 synthetically generated indoor scenes. Each scene consists of RGB images captured from different viewpoints, along with corresponding disparity annotations. The resolution of images is 1280 x 1080. We adopt its official test set for testing.
17

---

Hypersim [32] is large-scale synthetic dataset. In our experiment, we follow the test set defined by GBDF [3] for fair comparison, utilizing tone-mapped 286 images generated by their released code. Evaluation is performed using the corresponding 1024 x 768 depth annotations.
DIML [15] contains RGB-D D frames from both Kinect v2 [55] and Zed stereo camera with different resolutions. We conduct the generalization evaluation using the official test set, which includes real indoor and outdoor scene images along with corresponding high-resolution depth annotations.
DIODE [40] contains high-quality 1024 768 -igneegeeerated depth maps of both indoor and outdoor scenes. We use the whole validation set (771 images) for generalization testing.
Training Datasets. Our training data is sampled from diverse datasets, which can be categorized into synthetic and natural-scene datasets. The synthetic datasets consist of TartanAir [45], Irs [44],
UnrealStereo4K [39] and MVS-Synth [1]]. Among these, the resolutions of Tartan [45] and Irs [44] are below 1080p. while MVS-Syme [11] and UnrealStereo4K [39] reach resolutions of 1080p and 4k, respectively. Irs [44] and MVS-Synth [11] contain limited types of scenes, whereas others include both indoor and outdoor scenes, some of which [45, 39] present challenging conditions like poor lighting. To enhance the generalization to natural scenes, we also sample from four high-resolution real-world datasets, Holopix50K [10], iBims-1 [16]. WSVD [43], and VDW [47]. IBims-1 1 [16] contains a small number of indoor scenes but provides figh-precision depth annotations from the capturing device. The remaining three datasets include large-scale diverse scenes, but their depth annotations, obtained from stereo images [12], lack ideal edge precision.
B.2 Training Recipe
We leverage diverse training data to achieve strong generalizability. For each epoch, we randomly choose 20,000 images from natural-scene data [10, 47, 43, 16] and 20,000 images from synthetic datasets [45, 44, 39, 11]. For each sample, we adopt similar data processing and augmentation as GBDF [3]. To enhance training stability, we first train N, for one epoch only with Lgt. In the next two epochs, we involve figrad and L fusion for self-distillation. The a and No, in L are set to 0.02 and 4. The learning rate is le - 4. A and X2 in Eq. 10 are 0.5 and 0.1. All training and inference are conducted on a single NVIDIA A6000 GPU.
B.3 Evaluation Metrics
Depth Accuracy. M denotes numbers of pixels with valid depth annotations, while de and dž are estimated and ground truth depth of pixel We adopt the widely-used depth metrics as follows:
Absolute relative error (Abs Rel): M
1 Ede ld d'//d;;
Square relative error (Sq Rel):
1 [al] EdEM l|d d'112 /d
Root mean square error (RMSE): [ Ener Ild d=11?
Mean absolute logarithmic error (logıo): M. Exers [log (d) log (d"));
Accuracy with threshold t: Percentage of di such that < t E [1.25, 1.252, 1.253].
max( de
Edge Quality. For the edge quality, we follow prior arts [25, 3, 49] to employ the ordinal error (ORD) and depth discontinuity disagreement ratio (D'R). The ORD metric is defined as:
1 ORD = -
Σ N
6(P.o P..).
log (1 + exp
exp (-1 P(.))), 1 0, 1= 0,
6(pro - Pt.1)
(pi,0 - PT.))²
(13)
+1,
Pio/Pt1 2 21T,,
1,
1 Pio/Pi1 S 14f
otherwise,
18

---

Method
FLOPs (G)
Params (M)
Time (8)
GBDF [3]
10.377
201.338
0.112
Kim ef al. [14]
1138.342
61.371
0.128
Graph-GDSR [4]
397.355
32.533
0.832
Ours (onc-stage)
16.733
16.763
0.035
Boost [25]
286.13 x 63
79.565
2.183
PatchFusion [21]
810.813 x 177
42.511
5.345
Ours (two-stage)
16.733 x 30
16.763
1.050
Table 5: Model efficiency. We evaluate FLOPs, model parameters, and inference time of different methods. The first four rows contain one-stage methods [3, 14, 4], while the last three rows are for two-stage approaches [25, 21J. FLOPs and inference time are tested one NVIDIA RTX A6000 GPU. For the two-stage methods [25, 21J, their FLOPss are reported by multiplying FLOPs per patch with the required patch numbers for processing the image.
1024 1024 image with
0.862
0.860
Accuracy
0.251
(16 404g e8p
Depth Accuraty
Edge Error
0.856
0.222
0.844
0.216
2500
5000
7500
10000
Training Iterations
Figure 13: Iterations for self-distillation. We report the depth accuracy and edge error metrics of our SDDR model in the self ceff-disilllation training process.
Method
611
REL
ORD
D'R.
Ours (w/ D2)
0.855
0.129
0.317
0.237
Ours (w/ Gs)
0.862
0.120
0.305
0.216
Table 6: Formats of Pseudo-labels. We compare the self-distille training with refined depth Ds and depth edge representation Gss as pseudo-labels, The experiment is conducted on Middlebury2021 [34] dataset with LeReS [51] as the depth predictor.
where Pr,0 and Pú, represent pairs of edge-guided sampling points. P5. and P;,1 are the ground truth values corresponding positions. 7 is used represent the relative ordinal relationship between pairs of points. ORD characterizes the quality of depth edges by sampling pairs of points near extracted edges using a ranking loss [49]. On the other hand, D'R [25] uses the centers of super-pixels computed with the ground truth depth and compares neighboring super-pixel centroids across depth discontinuities. It directly focuses the accuracy of depth boundaries.
C More Experimental Results
C.1 Model Efficiency Comparisons.
In line 277 of the main paper, we mention that our method achieves higher model efficiency than prior arts [4, 3, 14, 25, 21]. Here, we provide detailed comparisons of model efficiency in Table For one-stage methods [4, 3, 14], SDDR adopts more lightweight refinement network, reducing model parameters by 12.3 times than GBDF [3] and improving inference speeds by 3.6 times than Kim et al. [14). Compared with two-stage tile-based methods [25, 21], coarse-to-fine edge refinement reduces the Flops per patch by 50.6 times and the patch numbers by 5.9 times than PatchFusion [21].
C.2 More Quantitative and Qualitative Results
Training Iterations of Self-distillation We investigate the iteration numbers of self-distillation in Fig. 13. The iteration number of zero indicates the model after the training of the first epoch only with
19

---

Depth
Edge
Predictor
Method
Abs Rell
Sq Roll
RMSEL
10% 4
6,t
6,"
6,7
ORD1
D'R
MiDaS [30]
0.117
0.576
3. 3.752
0.062
0.868
0.973
0,992
0.384
0.334
Kim et al [14]
0.120
0.562
3.558
0.063
0.864
0.973
(1.034
0.377
0.382
MiDaS
Grapb-GDSR
0.121
0.506
3.593
0.053
0.863
0.973
0,991
0,350
0.398
GBDF [3]
0.115
01561
3.6 3,685
0,082
0.871
0.973
0.983
0.305
0.255
Ours
0,112
0,545
3.068
0,050
0.879
0.979
0.994
0.299
0.220
LcReS [S1]
0.123
0:464
3. 3,010
0.052
0.847
0.969
0,992
0.326
0.359
Kim ot al. [14]
0.124
0.474
3.063
0.062
0.546
0.969
0.992
0.312
0.387
LeReS
GriphGGDSR
0.124
0.467
3.052
0.052
0.947
0.969
0.992
0:327
0.373
GBDF [3]
0.122
0.444
2,963
0.051
0.852
0.969
0.992
0.116
0.258
Ours
0.120
0.452
2.985
0,050
0.862
0.971
0
0.205
0.216
7redepth [1]
0.104
0.433
2.724
0.043
0.900
0.970
0.993
Kim et al. [14]
0.107
0.469
2. 7.706
0.044
0.896
0.970
0.992
0.228
0:243
Zoedepth
GraphGGDR
0.103
0.431
2.725
0.044
0:901
0.971
0.993
0.226
0:233
GBDF [3]
0.105
0:130
2.732
0.044
0,899
0.970
0,993
0.226
0,200
Ours
0.100
0.406
2,674
0.042
0.905
0.973
0.994
0.218
0.187
Table 7: Comparisons with one-stage refinement approaches on Middlebuuy2021.
Depth
Edge
Predictor
Method
Abs Rel.
Sq Rell
RMSEL
105 4
6,7
6.7
5.1
ORD1
D'R_
MiDaS [30]
0.117
0.576
3.752
0.052
0.868
0.973
0.992
0.384
0.334
MiDaS
Boost [25]
0.118
0.544
3,758
0.053
0.870
0.979
0.997
0.351
0.257
Ours
0.115
0,563
0.052
0.973
0.983
0.303
0.248
LeReS [S1]
0.123
0.464
3,040
0.052
0.847
0.969
0.992
0.326
0.350
LeReS
Boost [25]
0.131
0.487
3.014
0.064
0.844
0.960
0.980
0.325
0.202
Ours
0.123
0.459
3.005
0.052
0.861
0.969
0.991
0.309
0.21-
Zoedlepth [1]
0.104
0.433
2.724
0.043
0.900
0.970
0.983
0.225
0.208
Zoedepth
Patchfusion [21]
0.102
0.385
2.406
0.042
0.887
0.977
0.997
0.211
0.130
Boost [25]
0.080
0,349
2.502
0.042
0.911
0.979
0.985
0.210
0.140
Ours
0.096
0.350
0.041
0.913
0.977
0.996
0.202
0.125
Table 8: Comparisons with two-stage tile-based methods on Middlebury2021. PatchFusion [21]
can only adopt ZoeDepth [1] as the fixed baseline, while other approaches are reconfigurable and pluggable for different depth predictors [1, 51, 30].
Depth
Edge
Dataset
Method
Abs Rell
Sq Rel.
RMSEL
10g L
6.
62t
6g
ORD!
D*R.
LeReS [51]
0.101
45.607
325. 191
0.043
0.902
0.990
0.998
0.242
0.284
al. [14]
0.100
0.990
0.998
0.243
0.301
Graph-GDSR [4]
0.101
45.993
326 320
0.013
0.901
0.989
0.998
0.213
0.300
DIML
GBDF [3]
0.100
44.038
318.874
0.042
0.906
0.991
0.998
0.230
0.267
Boost (25)
0.108
50.923
311, 999
0.046
0.897
0.987
0.998
0.274
0.138
Ours
0.098
41.328
320.193
0.042
0.926
0.900
0.998
0.221
0.230
LeReS [51]
0.105
1.642
9.856
0.041
0.802
0.968
0.989
0.324
0.685
Kim ol. [14]
0.105
1.654
9.858
0.044
0.880
0.964
0.987
0.325
0.713
DIODE
Graph GDSR [4]
GBDF [3]
0.105
0.104
1.825
L.626
9.876
9.770
0.041
0.804
0.967
0.968
0.900
0.322
0.673
Boost (25)
0.105
1.612
9.879
0.041
0.892
0.966
0.987
0.343
0.640
Ours
0.098
1.529
9.549
0.042
0.968
0.988
0.293
0.637
Table 9: Comparisons with previous refinement approaches on DIML and DIODE.
ground truth for supervision, i.., before self-distillation. Clearly, with the proposed self self-distillation paradigm, both the depth accuracy and edge quality are improved until convergence.
Formats of Pseudo-labels We compare the refined depth Ds and the proposed depth edge represen- tation Gs as pseudo-labels. Using the accurate and meticulous depth Ds could be straightforward idea. However, with depth maps as the supervision, the model cannot precisely focus on improving edges and details. Thus, Gs achieves stronger efficacy than Ds, proving the necessity of our designs.
Quantitative Comparisons. In the main paper, only 61. REL, ORD, and D are reported. Here, we present the additional metrics of all the compared methods [14, 4, 3, 25, 21] Middlehury2021 [34], DIML [15], and DIODE [40] datasets in Table 7, Table 8, and Table 9. Our method outperforms previous approaches on most evaluation metrics, showing the effectiveness of our SDDR framework.
Qualitative Comparison We provide more qualitative comparisons with one-stage [14, 3] and two-stage [21, 25] methods in Fig. 14 and Fig. 15. These visual results further demonstrate the excellent performance and generalization capability of SDDR on diverse scenes [34, 15, 49].
20

---

RGB
LeReS
Kim et al.
GBDF
Ours
Figure 14: Qualitative comparisond with one-stage methods [14, 3] on various datasets [15, 49, 34]. We adopt LeReS [51] the depth predictor. Better viewed when zoomed in.
RGB
ZoeDepth
PatchFusion
Qualitative comparisons with two-stage methods [21, 25] various datasets [15, 49, 34]. We adopt Zoedepth [1] as the depth predictor. Better viewed when zoomed in.
Figure 15:
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper accurately conveys the contributions and scope of this work in the abstract and introduction sections, and provides a bullet-point summary at the end.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the method in Appendix A.5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All experiments presented this paper are reproducible. We will release the code and model following the acceptance of the paper.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [No]
Justification:
We will release the code and model after the acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtppcccpppccceeccccccccccccccat /tweeddeddee/eeeeemiiiiiiii ssssssolicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide a detailed description of our experimental setup and results in Sec. 4 of the main paper, well as in Appendice B and C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification:
Our experiments are stable across multiple runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix C.I, we provide a detailed account of our computational overhead and model efficiency.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : /twureeurpp...e/ cct cctppblic//eeeeccccccccccceceeee Ceteiiic///ieeeeeeeeee ctw.eithciiiiiineeene.e ines?
Answer: [Yes]
Justification:
The paper adheres to the NeurIPS Code of Ethics in respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Appendix A.5, we elaborate on the lack of societal impact of our work. Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks as elaborated in Appendix A.5.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper employs publicly available datasets and code for training and comparative evaluation, adhering all protocol restrictions that accompanied their release, and cites the relevant literature.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Upon acceptance of the paper, we will release our model and code under the CC BY-NC-SA 4.0 license.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "fKtGsPKsrK", "reviewer": "Reviewer_DKoq", "summary": "This paper presents a novel framework, SDDR, for enhancing the resolution and detail of depth maps generated by estimation models. By conceptualizing depth refinement within the context of noisy Poisson fusion, the authors have developed a method that effectively tackles the prevalent issues of inefficiency and inconsistency. SDDR incorporates a self-distillation technique that enhances depth edge precision and diminishes noisy disturbances, resulting in a significant boost in both the accuracy and quality of the depth maps.", "questions": "1. In 4.2, Coarse-to-fine Edge Refinement, is S=3 enough for getting the best results? Higher value of S can be explored. is there any trade-off?\n2. The paper demonstrates its promising performance on edge regions. Edge-specific measurements should be presented to highlight this point.", "limitations": "1. The method is designed to address specific types of noise (local inconsistency and edge deformation) and may not perform as well with other noise characteristics or in the presence of different degradations.\n2. The refined depth maps, despite improvements, might still contain artifacts or inconsistencies, particularly in complex scenes with challenging depth structures. The authors should present some of the results of those corner cases. If there is no such cases, the authors should point out the reasons and give a deep analysis for readers.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces SDDR that addresses the limitations of traditional depth refinement methods by treating the problem as a noisy Poisson fusion task.\n2. SDDR demonstrates strong robustness against local inconsistency and edge deformation noise, which are common in depth prediction tasks, leading to improved accuracy and edge quality.\n3. The use of edge-guided gradient loss and edge-based fusion loss as part of the optimization objective results in more accurate and noise-free depth edge representations.", "weaknesses": "1. The performance of SDDR might be highly dependent on the quality of the initial depth predictions; noisy or low-quality inputs could affect the final output.\n2. The refinement process might inadvertently smooth out important details or edges in the pursuit of noise reduction.\n3. The authors claim that the proposed method have a promising performance, especially on edge sharpness, however, it seems not that visually competitive compared with some SOTA monocular depth estimation methods, such as DepthAnythingV2. I understand that there are some differences on the targets between DepthAnything and this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer DKoq:\n\nThanks for your valuable feedback. We address all your questions as follows.\n\n# Weakness 1: Dependence on Initial Depth\n\n**SDDR achieves strong robustness regarding the quality of initial depth, noticeably improving depth edges and details, even when faced with low-quality initial depth.** Our robustness is brought by the self-distillation paradigm. GBDF relies on initial depth with filtering as labels, inheriting errors and noises. SDDR conducts coarse-to-fine refinement for accurate edge representation in self-distillation, acquiring strong robustness to correct and refine. \n\nFig.7 and lines 274-276 of the paper have proved our robustness by the curves of $\\delta_1$ with higher noises of initial depth. Here, we conduct three more experiments for further demonstration.\n\n**(1) Simulation Experiment.** In Fig.A, we gradually degrade initial depth. Despite increasingly noisy initial depth, SDDR maintains meticulous edges and details, as proved by edge-specific ORD metric and visual results. When initial depth degrades, SDDR only exhibits negligible fluctuations, bringing higher relative improvements.\n\n**(2) Real Samples.** In Fig.B, GBDF inherits noises and errors from low-quality initial depth, while SDDR robustly corrects the errors, restores the structures, and improves the details.\n\n**(3) More Advanced Depth Predictors.** SDDR can work with various predictors. Using more advanced predictors, *e.g.*, the Depth-Anything, brings improvements without extra effort. In Fig.C, with better depth of Depth-Anything-V2, SDDR further boosts edges and details.\n\n# Weakness 2: Smoothing Out Details in Noise Reduction\n\n**In all our experiments, SDDR does not smooth out edges.** Across various depth predictors and data, SDDR robustly enhances edges and details, as proved by visualizations and edge metrics (ORD and D3R) in Tab.1-4, 6-9, Fig.1, 3-6, 8, and 14-15 of paper and appendix.\n\nThe lower the local inconsistency noise $\\epsilon_{cons}$ and edge deformation noise $\\epsilon_{edge}$, the better the edges and details. Fig.D proves this point. We represent noise levels by adding the noise intensity of $\\epsilon_{cons}$ and $\\epsilon_{edge}$ as lines 537-544 of appendix. In Fig.D, edge errors and noise levels exhibit a positive correlation. We achieve lower noise level and better edge quality.\n\n# Weakness 3: Comparison with Depth-Anything-V2\n\n**SDDR noticeably improves edges and details with Depth-Anything models as depth predictors.** As mentioned by Reviewer DKoq, refinement differs from recent Depth-Anything. SDDR is a plug-in module to refine initial depth, with much fewer FLOPs and params than depth predictors. Thus, in Fig.C, we adopt Depth-Anything models as predictors. SDDR also produces finer edges and details. Fig.D quantitatively shows our better edge quality and lower noise level.\n\n# Question 1: Iteration Number S\n\nIn Tab.A, S=4 and 5 only produce subtle improvements than S=3. Edges and details are sufficiently refined after S=3. Thus, edge metrics ORD and D3R are saturated for S=4 and 5. More iterations lead to higher time costs. We adopt S=3 for trade-off between efficiency and performance.\n\n# Question 2: Edge-specific Measurements\n\nWe follow Boost, Kim *et al.*, and GBDF to adopt the common edge-specific measurements ORD and D3R for depth refinement, as stated in line 241. We report the metrics on 5 benchmarks throughout our tables. The metrics are further depicted in lines 606-613 of appendix.\n\n# Limitation 1: Different Noises and Degradations\n\n**Our local inconsistency and edge deformation noise are general and effective representations of depth prediction errors in diverse scenarios, rather than being specific to particular cases.** Reasons are presented in four aspects.\n\n**(1) Domain Knowledge.** Prior arts broadly recognize that depth blur and inconsistency are two key problems for the task. *E.g.*, as in GBDF, \"depth maps are usually blurry with inaccurate details such as planes and object boundaries.\" PatchFusion discusses that \"we discover that BoostingDepth suffers from scale inconsistencies.\" However, prior arts only try to solve the problems intuitively, *e.g.*, by selecting and merging patches, leading to unsatisfactory results and limited generalizability.\n\n**(2) Theoretical Derivation.** In contrast, we model refinement by noisy Poisson fusion with edge deformation and local inconsistency noise. Eq.1-4 and Sec.3.1 of our paper provide derivations of our method, which generally reveal reasons for blur and inconsistency of prior arts. *E.g.*, fusing patches as Boost produces higher local inconsistency noise, leading to inconsistent structures. Filtering as GBDF fails to suppress edge deformation noise, yielding blurred details. Noisy Poisson fusion and the noises generally conclude failures of prior arts and guide our design.\n\n**(3) Experiments.** In Fig.2 of the paper, combining $\\epsilon_{cons}$ and $\\epsilon_{edge}$ accurately depicts depth error. In lines 545-552 of appendix, PSNR between predictions and ideal depth with our noises exceeds 40 dB across Middlebury2021, showing that $\\epsilon_{cons}$ and $\\epsilon_{edge}$ generally and accurately represent depth error.\n\n**(4) Model Generalization.** SDDR is designed under the guidance of Noisy Poisson Fusion, aiming to suppress the 2 noises. Our state-of-the-art performance on 5 datasets, with synthetic and real-world, indoor and outdoor, dynamic and static scenes, shows our strong model generalizability, which also proves that our noises are broadly effective and not limited to particular cases.\n\n# Limitation 2: Corner Case\n\nWe show corner cases in Fig.B, involving complex details, challenging structures, and light overexposure. All compared methods cannot produce completely satisfactory results, *e.g.*, noisy leaves with blurred edges and inconsistent roof with a black hole. However, SDDR still significantly improves edges and details over LeReS and GBDF, refining initial depth effectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mJWotC08Oa", "reviewer": "Reviewer_Ca4W", "summary": "The paper introduces a novel framework called Self-Distilled Depth Refinement (SDDR) to enhance depth refinement, which aims to infer high-resolution depth maps with fine-grained edges from low-resolution depth estimations. The authors propose modeling depth refinement as a noisy Poisson fusion problem, addressing local inconsistency and edge deformation noises. The SDDR framework consists of depth edge representation and edge-based guidance. Through coarse-to-fine self-distillation, SDDR generates low-noise depth edge representations, which serve as pseudo-labels to guide the refinement process. The method demonstrates significant improvements in accuracy, edge quality, efficiency, and generalizability across five different benchmarks.", "questions": "1. In Line 107-108, the logic between the previous works and your motivation is unsmooth; it would be better to provide more analysis.\n\n2. In Figure 2, the author didn't show the edge deformation visualization. What kind of deformation would occur in depth estimation? Most off-the-shelf depth estimators have not produced deformed results in depth edges or other regions.\n\n3. Do the results in state-of-the-art work like DepthAnything-V2 exhibit the artifacts, consistency, and deformations proposed by the authors?\n\n4. Can you provide more insights into the performance differences between synthetic and real-world datasets?\n\n5. The performances compared to other works seem not very significant due to the combination of several contributions in this paper. How can the authors judge whether the gains are from randomness of training, the settings of the models' parameters, or other factors?\n\n6. What is the training time comparison between SDDR and other state-of-the-art methods?\n\n7. How sensitive is the SDDR framework to hyperparameter settings?\n\n8. How does SDDR perform on extreme edge cases with very high noise levels?\n\n9. Can SDDR be adapted for real-time applications, and what modifications would be necessary?", "limitations": "The paper provides extensive quantitative and qualitative results showcasing the strengths of the SDDR framework. However, it lacks a detailed analysis of potential failure cases or scenarios where the method does not perform well, which could provide insights for further improvements.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Innovative Approach: The modeling of depth refinement as a noisy Poisson fusion problem is a novel and insightful approach that effectively addresses common issues in depth refinement.\n\n2. Robust Framework: The self-distillation technique employed in SDDR enhances robustness against noise, resulting in high-quality depth maps with accurate edges.\n\n3. Comprehensive Evaluation: The authors conduct extensive experiments across five benchmarks, showcasing the method's superior performance in various scenarios.\n\n4. Efficiency: SDDR achieves higher efficiency compared to two-stage tile-based methods, reducing computational costs while maintaining or improving accuracy and edge quality.\n\n5. Generalizability: The framework demonstrates strong generalizability, performing well on both synthetic and real-world datasets.", "weaknesses": "1. Real-World Application: The paper primarily focuses on benchmarks and does not provide extensive discussion on real-world applications and potential limitations in practical scenarios.\n\n2. Edge Case Handling: The method’s performance in handling extreme edge cases or highly noisy data is not thoroughly explored.\n\n3. Ablation Studies: More detailed ablation studies are needed to understand the contribution of each component within the SDDR framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Ca4W:\n\nThanks for your positive feedback and valuable questions. We address all your comments as follows.\n# Weakness 1 and Limitation 1: Applications and Limitations\n\n**(1) Applications.** SDDR produces accurate depth with meticulous edge and\nconsistent structure, suitable for various applications such as style transfer, bokeh rendering, and 3D reconstruction. In Fig.E, we achieve better detail and structure over GBDF for the applications.\n\n**(2) Limitations.** For complex and meticulous structures in Fig.E and Fig.B, all compared methods cannot produce perfect depth, *e.g.*, some blurred leaves in the first row, leading to incomplete structures in generated images. However, for the depth maps and applications of these cases, SDDR still achieves significantly better edges and details than prior arts. This also underscores the necessity of the depth refinement task to refine edges and consistency of depth prediction models.\n# Weakness 2 and Question 8: Highly Noisy Edges\n\n**SDDR achieves strong robustness against noises, noticeably improving depth edges and details, even when faced with highly noisy initial depth.** Our robustness is brought by the self-distillation paradigm. GBDF relies on initial depth with filtering as labels, inheriting errors and noises. SDDR generates accurate edge representation in self-distillation, acquiring robustness to correct and refine.\n\nFig.7 and lines 274-276 have proved our robustness by the curves of $\\delta_1$ with higher noises of initial depth. Here, we conduct two more experiments for further demonstration.\n\n**(1) Simulation.** In Fig.A, we gradually degrade initial depth. Despite increasingly noisy initial depth, SDDR maintains meticulous edges and details, as proved by ORD metrics and visual results. When initial depth degrades, SDDR only exhibits negligible fluctuations, bringing higher relative improvements.\n\n**(2) Real Samples.** In Fig.B, GBDF inherits noises and errors from noisy initial depth, while SDDR robustly corrects the errors, restores the structures, and improves the details.\n\n# Weakness 3: More Detailed Ablation of Each Component\n\n**As per your advice, in Tab.B, we further ablate and prove the contribution of each component within SDDR.** Since depth refinement aims to improve depth edges, following previous Boost, Kim *et al.*, and GBDF, edge-specific metrics D3R and ORD are our main focus.\n\n**(1) By Components.** Starting from depth predictor MiDaS, in each row, we involve one more component and discuss the relative performance compared to the previous row. The self-distillation with depth edge representation and coarse-to-fine refinement reduces the edge error D3R by 8.4% and 5.0%. The edge-guided gradient loss and edge-based fusion loss also reduce D3R by 4.8% and 3.2%.\n\n**(2) In Total.** Combining all components, SDDR reduces edge errors D3R and ORD by 23.0% and 8.2% than MiDaS. For overall depth quality, SDDR decreases depth error REL by 6.2%. This is noteworthy and proves our strong efficacy, since edges only occupy a very small proportion of images.\n# Question 1: Smoothing the Motivation (Line 107)\nThanks for advice. We will include discussions below to smooth our motivation for Poisson fusion.\n\nPoisson fusion integrates values and gradients of two inputs. For depth refinement, low-resolution depth is consistent but blurred, while high-resolution depth involves accurate edges but inconsistency. Poisson fusion is potentially well-suited for our task, merging consistent depth with meticulous edges.\n# Question 2: Edge Deformation Noise\nWe apologize for causing misunderstanding of the noise, which actually does not represent deformed objects. Instead, it showcases misalignments of predicted and ideal edges, *e.g.*, missing, broken, or blurred depth edges. In Fig.F, we visualize our two noises separately. Edge deformation noise accurately depicts missing or blurred edges, while local inconsistency noise represents inconsistent depth structures. We will revise the paper with the illustrations to avoid misunderstanding. \n# Question 3: Degradation in Depth-Anything-V2\nIn Fig.C, Depth-Anything models also produce blurred edges or missing structures. SDDR further refines their predictions with better depth edges and details.\n# Question 4: Synthetic and Real-world Data\nDifferences between synthetic and real data are in lines 28-36 and 87-98. Similar to insights in prior arts, real-world data contains diverse scenes but suffers from sparse, blurred, or inaccurate depth. Synthetic data is rendered under full control with accurate depth but limited scenes. Thus, MiDaS, DPT, and DepthAnything combine the two types to balance accuracy and generalization. SDDR is not limited by data formats, since it generates accurate edge representation for self-distillation.\n# Question 5 and Question 7: Significant Performance, Training Randomness, and Parameters\n**(1) As proved throughout our figures and tables, SDDR significantly improves depth edges and details over prior arts, which is further demonstrated in Weakness 3 by each component.**\n\n**(2) Our superior performance is not caused by training randomness.** In Tab.C, we train SDDR 12 times. Our performance is stable only with minimal fluctuations.\n\n**(3) SDDR is not sensitive to parameters.** For training and method parameters, in Tab.C, SDDR remains stable with varied learning rates, loss ratios, training epochs, iteration numbers, overlapping ratios, etc. For other issues, *e.g.*, optimizer and initialization, we follow the prior Boost and GBDF.\n# Question 6: Training Time\nSDDR and prior GBDF have similar training time (12 hours for 3 epochs on a A6000 GPU).\n# Question 9: Real-time Processing\nPursuing highly fine-grained depth, current refinement methods can't be real-time (25 fps). **However, in Fig.1(b) and Tab.5, our SDDR achieves the best efficiency among all compared methods.** For real-time applications, pruning and quantization can be used to further improve efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EZbQLpxFG5", "reviewer": "Reviewer_9uAR", "summary": "This paper introduces a Self-distilled Depth Refinement (SDDR) framework to enhance robustness against noise. This framework primarily includes depth edge representation and edge-based guidance. And they design an edge-guided gradient loss and an edge-based fusion loss.  Furthermore, experiments on five benchmarks have validated the effectiveness of the framework.", "questions": "The paper proposes a novel and interesting framework and demonstrates a clear advancement in this task performance, so I would like to accept the paper. However, the lack of clear motivation makes the paper somewhat confusing to read, so I recommend borderline. If the authors can address my concerns, I am ready to change my recommendation based on the comments.", "limitations": "See the weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is well-written, and the method and results are presented clearly. Using the strategy of iterative depth refinement is helpful for this task. The framework achieves excellent performance on widely used deblurring datasets.", "weaknesses": "1. The authors did not specify in the paper which sensor or modality was used for depth estimation.\n2. The authors decouple depth prediction errors into two degradation components: local inconsistency noise and edge deformation noise. And they give two samples in Fig. 2. However, I don't think this makes it clear that the errors fall into either category. The authors should provide accurate sources of error and analyze the significance of these two types of errors among all the errors.\n3. Line 121: In Motivation Elaboration, the authors primarily focused on analyzing the limitations of previous work but did not offer a compelling justification for the motivation behind their current study.\n4. Line 149: The authors claim to propose an edge-guided gradient loss, but this loss has also been used in the following two papers. Please clarify the differences. \\\n     [1] Wang Z, Ye X, Sun B, et al. Depth upsampling based on deep edge-aware learning[J]. Pattern Recognition, 2020, 103: 107274. \\\n    [2] Qiao X, Ge C, Zhang Y, et al. Depth super-resolution from explicit and implicit high-frequency features[J]. Computer Vision and Image Understanding, 2023, 237: 103841.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9uAR:\n\nThanks for your positive feedback and valuable questions. We address all your comments as follows.\n# Weakness 1: Sensor and Modality\nWe specify the modality of depth models and datasets.\n\n**(1) Models.** Similar to previous monocular depth models, SDDR takes a RGB image as input, without other modalities.\n\n**(2) Data.** Depth prediction and refinement use various data to train and evaluate, with RGB as input and depth as ground truth. Depth maps are annotated by varied techniques, *e.g.*, CG rendering (TartanAir, IRS), stereo matching (HRWSI, VDW),  LiDAR (IBims-1), and Kinect (DIML). These modalities are not used as input for models. We combine these data for experiments as in lines 561-588.\n# Weakness 2: Two Noise Components.\nWe illustrate the source and significance of local inconsistency noise $\\epsilon_{cons}$ and edge deformation noise $\\epsilon_{edge}$ with the analysis and experiments below.\n\n**(1) We illustrate the source of our noises in 3 aspects.**\n\n**Our noises are motivated by vital problems of the task.** Prior arts broadly recognize that depth blur and inconsistency are two key problems for the task. *E.g.*, as in GBDF, \"depth maps are usually blurry with inaccurate details such as planes and object boundaries.\" PatchFusion discusses that \"we discover that BoostingDepth suffers from scale inconsistencies.\" The two problems could stem from resolution and receptive field. Low-resolution inference loses detail, while high-resolution prediction leads to inconsistent structures due to limited model receptive field. Our noises model the two dominant degradations between predicted and ideal depth.\n\n**Our noises generally conclude failures in prior arts.** Previous methods try to solve the above two problems intuitively, *e.g.*, by selecting and merging patches. In contrast, we model refinement by edge deformation and local inconsistency noise. Derivations in Eq.1-4 and Sec.3.1 generally reveal reasons for blur and inconsistency in prior arts. *E.g.*, fusing patches as Boost produces higher $\\epsilon_{cons}$ with inconsistent structures. Filtering as GBDF fails to suppress $\\epsilon_{edge}$, yielding blurred details. Our noises conclude previous failures and guide our design.\n\n**Our noises accurately depict the aforementioned problems.** In Fig.F of the rebuttal PDF, we further provide visual results of our noises. Simulated by regional affine transformations, $\\epsilon_{cons}$ represents disrupted depth structures. With position-constrained Gaussian distributions (line 538), $\\epsilon_{edge}$ accurately depicts missing or blurred edges. Depth errors can be decoupled with our noises. *E.g.*, $\\epsilon_{edge}$ is prominent in low-resolution predictions, while $\\epsilon_{cons}$ occupies larger ratios for high resolution.\n\n**(2) We show the significance of our noises by 3 experiments.**\n\nIn lines 545-552, PSNR between predictions and ideal depth with our noises exceeds 40 dB across all samples of Middlebury2021, quantitatively proving the significance of our noises among all errors.\n\nIn Fig.2, adding $\\epsilon_{cons}$ and $\\epsilon_{edge}$ accurately depicts the discrepancy between depth prediction and ground truth. Combining the noises broadly covers depth prediction errors.\n\nFig.D of the PDF proves that the lower the noise, the better the depth quality. Edge errors and noise levels exhibit a positive correlation. The low noise levels of SDDR bring fine-grained depth.\n# Weakness 3 and Question 1: Motivation Elaboration\n\nThanks for your insightful advice! We further discuss motivations of our designs, which will be added to our revised paper.\n\n**(1) Noise Modeling.** As in Weakness 2, our noises model two key problems of the task.\n\n**(2) Poisson Fusion.** In lines 107-109 and 40-46, Poisson fusion integrates the value and gradient domain of two inputs. In lines 205-207 and 465-475, low-resolution depth tends to be consistent but blurred, while high-resolution depth involves accurate edges but inconsistency. Poisson fusion is potentially well-suited for refinement, merging consistent depth with meticulous edges. We implement the Poisson fusion operator as a learnable refinement network, without relying on external fusion mask and complex parameter setting (line 130). Our method produces consistent and fine-grained depth with strong generalizability.\n\n**(3) Self-distillation.** In Eq.2, the optimization objective of Poisson fusion serves as the training loss of refinement network. Low-noise edge labels are necessary to guide the model but are unavailable in diverse natural scenes. Given that the refinement network inherently reduces noises and restores details, self-distillation can naturally form. Depth edge representations are generated as labels by coarse-to-fine refinement. When depth maps are better refined, the labels become more noise-free.\n# Weakness 4: Comparisons with Previous Loss\n**Our edge-guided gradient loss differs from prior losses [1, 2] in 4 aspects.**\n\n**(1) Supervision Paradigm.** Previous losses[1, 2] work in a fully-supervised manner with edges from depth ground truth. Our loss serves for self-distilled learning with depth edge representations. As in lines 33-36 and 91-93, in some natural-scene data, edges from ground truth are unreliable. Our loss establishs accurate edge guidance for self-training.\n\n**(2) Classification vs. Regression.** Wang *et al.*[1] adopt cross-entropy loss with hard edge labels, classifying whether a pixel belongs to edge area. Our loss guides the model to learn soft edge representations by regression, enforcing both edge area and intensity precisely.\n\n**(3) Global vs. Local.** Qiao *et al.*[2] match gradients globally, while our loss works on high-frequency local regions $P_n$ (line 197). Our model improves details in $P_n$ and preserves consistency in flat areas.\n\n**(4) Scale and Shift Alignment.** Qiao *et al.*[2] ignore depth scale ambiguity, whereas we perform scale and shift alignment to maintain consistency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewer DKoq, Ca4W, and 9uAR:

We would like to express our sincere gratitude for your insightful comments and constructive suggestions on our paper. In the rebuttal, we have diligently incorporated comprehensive discussions and experiments to address all the raised queries, comments, and concerns. 

**Here, we provide a general description of our responses for your convenience.**

(1) We separately respond to the three reviewers question by question in the author rebuttal area for each reviewer. Please refer to that area for the answer to your comments.

(2) To address all the questions of the three reviewers, we have included three tables and six figures in the rebuttal. The six figures (marked by Fig.A, B, C, D, E, and F) are contained in our submitted rebuttal PDF. The three tables (marked by Tab.A, B, and C) are presented in this overall rebuttal area, after the general description. 

We hope our further discussions, analyses, and experiments can solve all raised concerns and provide better illustrations of the proposed SDDR framework. Thanks again for your meticulous review and valuable insights! 

**Tab.A, B, and C are listed below.**


**Table A: Iteration Number S (Reviewer DKoq - Question 1).** Based on Tab.4(a) of the paper, we further increase the iteration number S of coarse-to-fine edge refinement to S=4 and S=5. The performance converges after S=3. More iterations lead to higher time costs. We adopt S=3 for the trade-off between efficiency and performance. Refer to Question 1 for detailed analysis. 
| $Method$ | $D3R\downarrow$ | $ORD\downarrow$ | $REL\downarrow$ | $\delta_1\uparrow$ | 
| ---- | ---- | ---- | ---- | ---- |
| S=0 | 0.235 | 0.313 | 0.125 | 0.859 | 
| S=1 | 0.223 | 0.309 | 0.122 | 0.860 | 
| S=2 | 0.219 | 0.307 | **0.120** | 0.860 | 
| S=3 | 0.216 | 0.305 | **0.120** | **0.862** | 
| S=4 | **0.215** | 0.305 | 0.121 | **0.862** | 
| S=5 | **0.215** | **0.304** | 0.121 | 0.860 | 
||

**Table B: Detailed Ablation of Each Component (Reviewer Ca4W - Weakness 3).** We ablate the contribution of each component within SDDR by zero-shot evaluations on the Multiscopic dataset with MiDaS as the depth predictor. The refinement baseline adopts the common fully-supervised training by depth ground truth without self-distillation. Depth edge representation indicates generating self-distillation pseudo-labels with a single iteration. Coarse-to-fine edge refinement refers to enhancing the pseudo-labels through iterations S=3. In each row, we involve one more component and report the results. Refer to Weakness 3 in our response for detailed discussions. 

| $Method/Component$ | $D3R\downarrow$ | $ORD\downarrow$ | $REL\downarrow$ | $\delta_1\uparrow$ |
| --- | --- | --- | --- | --- |
| MiDaS | 0.274 | 0.292 | 0.130 | 0.839 |
| Refinement Baseline | 0.263 | 0.286 | 0.129 | 0.841 |
| + Depth Edge Representation | 0.241 | 0.280 | 0.128 | 0.848 |
| + Coarse-to-fine Edge Refinement | 0.229 | 0.271 | 0.124 | 0.851 |
| + Edge-guided Gradient Loss | 0.218 | **0.268** | 0.124 | 0.851 |
| + Edge-based Fusion Loss | **0.211** | **0.268** | **0.122** | **0.852** |
||


**Table C: Training Randomness and Hyperparameters (Reviewer Ca4W - Question 5 & Question 7).** We train the SDDR 12 times. The performance is stable only with minimal fluctuations. Besides, our method is not sensitive to both training and method parameters. We adjust the learning rates, training epochs, loss ratios, the iteration number S and overlapping ratios of coarse-to-fine refinement, along with the percentile $a$ and repeat steps $N_w$ in the edge-based fusion loss. The model performance remains stable with significant improvements over the depth predictor LeReS. Refer to Question 5 and Question 7 in our response for discussions.

| $Type$ | $Parameter  Settings$ | $D3R\downarrow$ | $ORD\downarrow$ | $REL\downarrow$ |$\delta_1\uparrow$ | 
| --- | --- | --- | --- | --- | --- |
| LeReS | -  | 0.326 | 0.359 | 0.123 | 0.847 |
| Training | learning rate=1e-4,2e-4,5e-5 | 0.215±0.001 | 0.305±0.001 | 0.119±0.001 |0.862±0.001 | 
| Training | epochs=3,4,5 | 0.214±0.003 | 0.303±0.002 | 0.121±0.001 |0.861±0.001 | 
| Training | $\lambda_1$=1.0,0.5,0.1 | 0.215±0.004 | 0.305±0.002 | 0.120±0.001 | 0.861±0.002 | 
| Training | $\lambda_2$=1.0,0.5,0.1 | 0.218±0.003 | 0.306±0.001 | 0.120±0.002 | 0.862±0.002 | 
| Method | S=3,4,5 | 0.215±0.002 | 0.305±0.001 | 0.120±0.001 | 0.861±0.001 |
| Method | overlapping  ratio=0.1,0.2,0.4 | 0.215±0.001 | 0.306±0.002 | 0.119±0.001 | 0.862±0.001 |
| Method | $N_w$=2,3,4;$a$=1%,2%,4% | 0.217±0.002 | 0.306±0.001 | 0.119±0.001 | 0.862±0.001 |
||

**Please download the rebuttal PDF by the button below, for the six figures (Fig.A - F) with more visual results and illustrations.** 

(Best view zoomed in on-screen for details and comparisons.)

Meta-review: The paper introduces a new framework termed as Self-Distilled Depth Refinement (SDDR) to enhance depth refinement, which infers high-resolution depth maps with fine-grained edges from low-resolution depth estimations. The paper models depth refinement as a noisy Poisson fusion problem, addressing local inconsistency and edge deformation noises. Through coarse-to-fine self-distillation, SDDR generates low-noise depth edge representations, which serve as pseudo-labels to guide the refinement process. The method demonstrates improvements in accuracy, edge quality, efficiency, and generalizability across different benchmarks. 

The paper was reviewed by three reviewers, whereas the final recommendations were two Borderline Accept and one Weak Accept. The authors made a successful rebuttal in addressing the reviewers' concerns, whereas the rating improves consistently. Given the consistent recommendations, the paper could be accepted while the authors are requested to revise the paper correspondingly.
Decision: Accept (poster)