A Simple yet Universal Framework for Depth Completion
Authors: Jin-Hwi Park
Hae-Gon Jeon
Venue: NeurIPS 2024 poster (2024)
Keywords: Few-shot Depth Completion
hyperbolic representation
foundation model
Primary area: machine_vision
TL;DR: We propose a universal few-shot learner for depth completion with arbitrary sensor.

Abstract: Consistent depth estimation across diverse scenes and sensors is a crucial challenge in computer vision, especially when deploying machine learning models in the real world. Traditional methods depend heavily on extensive pixel-wise labeled data, which is costly and labor-intensive to acquire, and frequently have difficulty in scale issues on various depth sensors. In response, we define Universal Depth Completion (UniDC) problem. We also present a baseline architecture, a simple yet effective approach tailored to estimate scene depth across a wide range of sensors and environments using minimal labeled data. 
Our approach addresses two primary challenges: generalizable knowledge of unseen scene configurations and strong adaptation to arbitrary depth sensors with various specifications. To enhance versatility in the wild, we utilize a foundation model for monocular depth estimation that provides a comprehensive understanding of 3D structures in scenes. Additionally, for fast adaptation to off-the-shelf sensors, we generate a pixel-wise affinity map based on the knowledge from the foundation model. We then adjust depth information from arbitrary sensors to the monocular depth along with the constructed affinity. Furthermore, to boost up both the adaptability and generality, we embed the learned features into hyperbolic space, which builds implicit hierarchical structures of 3D data from fewer examples. Extensive experiments demonstrate the proposed method's superior generalization capabilities for UniDC problem over state-of-the-art depth completion. Source code is publicly available at https://github.com/JinhwiPark/UniDC.

A Simple yet Universal Framework for Depth Completion
Jin-Hwi Park Al Graduate School GIST j jinhviparkkg..iist.
Hae-Gon Jeon AI Graduate School GIST haegonj@gist.ac..
ac.
Abstract
Consistent depth estimation across diverse scenes and sensors a crucial challenge in computer vision, especially when deploying machine learning models in the real world. Traditional methods depend heavily on extensive pixel-wise labeled data, which is costly and labor-intensive to acquire, and frequently have difficulty in scale issues on various depth sensors. response, we define Universal Depth Completion (UniDC) problem. We also present baseline architecture, simple yet effective approach tailored to estimate scene depth across a wide range sensors and environments using minimal labeled data. Our approach addresses two primary challenges: generalizable knowledge of unseen scene configurations and strong adaptation arbitrary depth sensors with various specifications. To enhance versatility in the wild, we utilize a foundation model for monocular depth estimation that provides comprehensive understanding of 3D structures in scenes. Additionally, for fast adaptation to off- off-the-shelle sensors, we generate a pixel-wise affinity map based on the knowledge from the foundation model. We then adjust depth information from arbitrary sensors to the monocular depth along with the constructed affinity. Furthermore,
boost up both the adaptability and general- ity, we embed the learned features into hyperbolic space, which builds implicit hierarchical structures of 3D data from fewer examples. Extensive experiments demonstrate the proposed method's superior generalization capabilities for UniDC problem over state-of-the-arr depth completion. Source code is publicly available at https: / github..... wtt.coom///o/eeeeeeeeeeeeeeeeeee Jinhwi Parriiaaaaiiuii Park/UniDC.
1 Introduction
Acquiring accurate and dense depth maps is crucial for various computer vision tasks such as scene understanding [1. 2. 4], 3D reconstruction [5, 6, 7, 8], and autonomous driving [9, 10. 11]. Traditional methods like dense stereo matching [12, 13. 14] often face challenges of handling occlusion and varying lighting conditions between viewpoints. Additionally, depth maps obtained from active depth sensors [15, 16] like LiDAR and Time-of-Flight cameras typically exhibit low resolutions. As solution the above problems, depth completion has been widely studied. The goal of depth completion is to obtain depth map from a pair of an image and low-resolution depth map (often sparse depth map) taken by active sensors. The depth completion aims to convert a sparse depth map into a dense depth prediction by propagating it with an image-based affinity map.
Recent advances in learning- based depth perceptions have markedly improved the performance in this domain; however, most approaches are still tailored to specific settings and struggle to generalize to environments or sensor types. While generalizable knowledge can be achieved by training huge models with large-scale and diverse datasets, acquiring accurate and dense depth information as ground-truth data is prohibitively expensive and time-consuming, which makes such a generalization model for metric scale 3D depth prediction infeasible in practice. Moreover, there exist numerous types of active depth sensors and complex scenarios in the real world. Unfortunately. only two
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

benchmark datasets (e.g., KITTI [17] and NYU dataset [18]) are predominantly utilized in relevant research fields. Considering the accessibility of various industrial scenarios and the extremely high annotation cost, it is desirable to explore few-shot learning approach capable of universal depth prediction for both arbitrary sensors and environments.
In response to the growing needs of both industry and the research community, in this work, we define new problem, called Universal Depth Completion (UniDC), and present baseline architecture and its advanced version. Our key insight of the baseline model for UniDC is to utilize pre-trained knowledge from foundation model for monocular depth estimation, which provides depth- information enriched with high-resolun reshuresolution contextual information. Previous works typically exploit entangled representations of an image and corresponding depth data by concatenating them in an imput layer, which reduces the generality of the foundation model. A contemporary work [19] proposes a sensor-agnostic depth completion with a depth prompting module, which mitigates the sensor bias problem by disentangling image and depth modalities. Since the depth representation is optimized with respect to a specific scene environment, has limitations in out-of-domain situations, such as the environmental transition from indoor to outdoor, and vice versa.
To resolve this limitation, we design a simple baseline architecture using the foundation model. By excluding the training procedure for a new encoder to represent depth data, we achieve a high generality of the model across various sensors regardless of scene configurations. The proposed archi- tecture consists of three sequential steps: (1) extraction of depth- features from the foundation model; (2) sparse-to-dense conversion based on the depth-aware information: (3) refinement of the converted depth with pixel-wise affinity map constructed based on nigh-resolution details of the input image. For more details, the sparse-to-dense conversion aggregates adjacent depth values based on the high-resolution pixel-wise features from the foundation model. In the depth refinement process, we adopt a spatial propagation module with a multi-kernel affinity map.
We next boost up the baseline architecture by taking advantage of hyperbolic embedding, As stated in [20, 21], the natural capacity of hyperbolic spaces encourages capturing the implicit hierarchical structure of 3D data. In particular, this capability alleviates bleeding errors in the spatial propagation process [22]. To ensure adaptability and generality, we also design a multi-curvature approach for producing multiple affinity maps in the refinement stage. The effectiveness of our models is demonstrated across variety of scenarios and datasets, confirming superior generalization and robustness in different sensor setups and scene configurations. We also conduct extensive experiments and analyses to validate the efficacy of the proposed model.
2 Related Works
Depth Completion. Image-guided depth completion aims to predict dense depth maps from an RGB image and its synchronized sparse depth acquired by depth sensors. A work in [23] introduces a deep regression model that significantly enhances prediction accuracy over the existing monocular depth estimation method [24], which utilizes only RGB image as input. However, depth maps from the direct regression method often suffer from blurry artifacts and distortions at object boundaries [25]. To address these issues, several works have introduced spatial propagation networks (SPNs) [25, 26, 27, 28, 29] refinement modules. SPNs iteratively update the output of direct-regression methods by aggregating neighboring pixels over a reference pixel. Nonetheless, these models are typically tailored for specific depth sensors, such the 64-Line Velodyne LiDAR [30] in KITTI outdoor dataset [17] and Kinect [31] for NYUv2 indoor dataset [18].
To alleviate this limited usage of SPNs, several studies have explored sensor-/domain-agnoss depth completion. SpAgNet [32] develops a model agnostic to the sparsity of depth points by incorporating sparse depth representions into depth decoder. Another work [33] takes the use of both sparse metric depth and data-driven priors from a monocular depth prediction network for domain- comain-agnostic depth completion. DepthPrompting [19] solves sensor bias problems with prompt engineering, Despite these efforts, they still face challenges with a cross-domain generalization [19] and an issue on a limitation of sensors' scan ranges, which causes overfitting problem [32, 33].
Usage of Foundation Model in Downstream Task. Foundation models, designed for various down- stream tasks, have revolutionized both natural language processing and computer vision fields. In particular, in the computer vision field, these foundation models excel in high-level visual per- ception tasks such image recognition [34, 35, 36] and image captioning [35, 37, 36]. Those

---

vision foundation models provide benefits for strong adaptation to various tasks via tuning meth- ods [38, 39, 40, 41] and feature adaptation methods [42, 43, 44]. In low-level tasks like depth computation, several works [45, 46, 47] create diverse datasets for zero-shot generalization capabili- ties, while others [48, 49, 50] fine-tune the text-to-imaae model [51] to utilize diffusion priors for better generalization which guides them to keep geometric details.
Hyperbolic Geometry for Visual Data. Hyperbolic embedding for efficient bearning-based ap- proaches [52, 53] has gained interest. The Hyperbolic embedding has validated its ability to ef- fectively represent complex data as hierarchical structures in low-dimensional spaces, offering a distinct advantage over Euclidean embeddings. This unique capability promotes the design of hyperbolic neural networks, and is applicable for a range of applications such as hierarchical recogni- tion [54. 55, 56], retrieval [57, 58, 59], dealing with uncertainty [60, 61, 62], and generative learning on searce data [63, 64, 65, 66). Especially, hyperbolic methods have been shown to be effective in addressing low-shot visual problems [67, 68, 69, 60, 70], modeling complex 3D data [20, 21] and measuring pixel- wise similarity [22]. In this work, we devise the hyperbolic version of the proposed architecture to make both the generalizable power and understanding 3D depth data better.
3 Baseline Architecture
We present simple yet effective architecture to achieve a generalizable depth completion model for unseen en- vironments with only minimal data. Firstly, in Sec.3.1, 1, we explain the rationale for adopting a monocular depth foundation model to simultancously achieve sensor- Shomainn-agnoste depth completion. We then propose baseline model architecture for UniDC, which inte- grates the pre-trained foundation model with both the depth propagation and refinement process in Sec.3.2.
Eudidaan Affinty
Encoder
Refine
Decoder
Dense Depth
RGB&&Deph
(a) Conventional SPNs
Deptth
Foundation
Model
Reline
Insel Depa
Prompting
Euctidoan
Aftinty
Dense Depth
3.1 Rationale: Foundation Model Usage in UniDC
RGB8 Depth
Difficulties to generalize depth completion. The two major obstacles pletion are the high cost of dense depth data acquisition and the scale variance across different sensors. First, capturing dense depth data on a metric scale is expen- sive. For example, Velodyne 64-line LIDAR, used in the KITTI dataset, provides high-quality depth informa- tion but has less than 6% density relative to the number of pixels in its synchronized image. Second, sensors have their own scanning ranges, hindering the develop- ment of universal solution. As shown in Fig. 1-(a,b), the previous frameworks learn the joint representation of image and depth, and the depth prompting module, respectively. However, the trained encoder is vulnerable to handling different sensors due bias towards specific scanning ranges.
(b) Depth (CVPR24)
http://ddmiiiignnnsiiie depth com-
F Foundation
Model
Hypertoolic
Affinity
Hyp-Refine
Hyperballc Sparassto-Dense
Dense Depth
Conversion
RGB& Depth
Initial Dapth
Ours
Figure 1: Illustrations of conventional SPN,
sensor agnostic model [19] and ours. Our approach uses hyperbolic-hased depth com- pletion in three stages: generating an ini- tial depth, constructing a pixel-wise affinity, and refining the depth based on the affinity.
Usage of depth-aware knowledge from depth foundation model. Although the depth foundation model produces relative depth maps, we can measure pixel-wise similarity using them. For example, we are able to distinguish between foreground and background regions only with the relative depth maps and account for depth boundaries between objects in scenes. Therefore, based this depth- aware information, will be the most probable solution that propagates a given sparse metric depth into the remaining pixels in an input image space without any additional learning for the depth.
Revisiting how to use SPN. SPN [71] constitutes core component in most state-of of-te-of-the-ar art (SoTA) depth completion and is typically invoked as final refinement step. The SPN refinement module takes initial depth and pixel-wise affinity as input and yields refined dense depth by iteratively updating its output. During training, the previous methods (Fig. ((a.b) jointly optimize the pixel-wise affinity and initial depth. However, the joint optimization scheme hinders the fast adaptation to new environments because learned weights are asked to have both domain- and depth-specific features. Furthermore,
3

---

DepthPrompting (Fig,,-(b)), which employs a depth foundation model for a relative-scale depth map as initial depth of SPN, struggles to adapt to new environments with limited data. We want to eliminate the possibility of degeneracy, so we devise a sparse-to-dense conversion with a foundation model to make an initial dense depth. In Fig. I, different from the coarse initial depth seen traditional SPNs, our method provides promising results even before the SPN refinement step.
3.2 Architecture Design
Considering the facts discussed in Sec.3.1, we devise an effective baseline architecture. We first utilize pre-trained knowledge from foundation model tailored for monocular depth estimation, which provides pixel-wise relative distances (a.k.a. relative scene depth) from a camera along with high-resolution contextual information. Thanks to the knowledge, our baseline architecture becomes simpler due to no need for additional encoder to represent depth data from arbitrary sensors. Our model operates three stages: ! extraction of the relative depth-aware features from the foundation model, ② propagation of spare depth from arbitrary sensors based on the depth-aware features, and 3 refinement of it with a pixel-wise affinity map constructed from the depth-aware features. This scheme not only simplifies the architectural complexity, but also enhances the adaptability and performance across diverse sensing scenarios. The overall algorithm scheme summarized in Alg.
Tuning strategy for foundation model. Given a single image I E 33x X the pre-trained depth model for outputs multi-scale intermediate features and relative depth Drelative as below:
E,, Prelative = [r(],0
where , denotes parameters of the foundation model.
Since the foundation model is trained to estimate relative depth from single images, they inherently face limitations when handling metric scale depths. reduce the modality discrepancy, our approach involves an integration of an additional loss term to refine the foundation model by minimizing the difference between D Deelative and its Ground Truth (GT) depth Dut for valid pixels v E V. Let 6, log Drelatve (v) log (t)) the loss Lscale-invariant defined as below:
1 Lygaale invariant Drelaative Dgt)= [V]
Σ (o.)² VEV
A IV/
Σ 5x VoV
where we set 0.85 in all experiments as in [24]. We also implement bias tuning [38, 39], shown to be more effective for dense prediction tasks than other tuning protocols [72, 38). The bias tuning updates the bias terms while keeping the rest of the backbone parameters unchanged, thus preserving the high-resolution details and contextual information. These strategic modifications significantly enhance the capability of the foundation model for estimating metric scale depth.
4 Advanced Architecture with Hyperbolic Geometry
We also present advanced version of the baseline architecture that grafts hyperbolic geometry onto the depth foundation model, known for its effectiveness in low-shot problems [67, 68, 69, 60, 70]. We first generate depth-aware features by merging the multi-scale intermediate features E derived from the foundation model and by embedding them into hyperbolic space with gcometry-awan curvature 11) Using the depth-aware features alongside sparse sensor data, develop hyperbolic propagation inspired by a traditional bilateral filter mechanism, which yields an initial dense depth at metric scale (Sec.4.2). We lastly introduce process for generating multi-curvature hyperbolic space for high-fidelity pixel relations and refinement of the initial depth (Sec.4.3).
4.1 Multi-scale Feature Fusion & Hyperbolic Curvature Generation
The intermediate features from the foundation model Et E E, where = o,..., 1, correspond to scales factors 1/2, ... 1/24 of the original resolution of input images. We aim to synergistically fuse the multi-scale information to learn comprehensive, context-aware features that facilitate depth propa- gation at metric scale. We upsample the coarser feature map EM (E) Ea) using convolution layers, and then aggregate E.M with finer feature map E to obtain better visual contextual features

---

Algorithm Implementation of Hyperbolic Universal Depth Completion
Require:
Given single image E R depth foundation model fr and corresponding parameter
fyy. multi-scale feature aggregation blocks Jt Je number of multi-scale feature L, curvature generation blocks C, set of neighboring pixel coordinate N(),, kernel function P, and multi-kernel affinity map Ag.
1: procedure
2: E (1,
Multi-scale Features Extraction (Eq.1)
[Stage-[] Multi-scale Feature Fusion Hyperbolic Curvature Generation
4: for Ej in E do (1=00 O..... L-1)
Ej41 = =f{ Iuusiion (EM El41) (Ed En)
Feature Fusion (Eq.3)
6:
end for
7:
C(EL) for E Eng (i) E,,
D Curvature Generation (Eq.6)
9:
[Stage-21 Sparse-to-Denes Dense Conversion based Hyperbolic Features
10:
H. = =xpx ( (LL.)) H, = expo (FL.))
b Hyperbolic Embedding (Fq.5)
11:
Wi (DDistay (H,, IT;). Disteuc (E EM,))
Hyperbolic Kernel (Eq.8)
12:
Dinit = =E, S
Init Depth (Eq.8)
13:
[Stage-3]] Depth Refinement in Multi-curvature Hyperbolic Space RE (x(E)))
Multi-curvature Generation (Eq.11)
15:
10 HCL(EE,
Hyperbolic Affinity (Eq.11)
16:
D A @ D EjENE A DJ,,
Hyperbolic Depth Refinement (Eq.9)
17:
D Enek D,
> Final Depth (Eq.9)
18:
end for
19:
end procedure
B41- This fusion process is described below:
E ff f.fusion (EM, Ei41);
(3)
where fuusion indicates multi-scale feature aggregation blocks consisting of 2D transposed convolu- tion layers with a skip connection.
Hyperbolic embedding. To ensure strong adaption to both new environments and any type of sensors, adopt hyperbolic geometry which enables to capture the inherent hierarchical structures of 3D data [20, 21J. To embed the Euclidean features into hyperbolic space and vice versa, one first needs define a bijective mapping from R" to D. The exponential and the logarithmic mapping are used as bijective functions that have appealing forms at an origin, namely for x IR" and u E Da.
expo (x) tanh <
and logō tanh (V<||uu))
(4)
V |||
Vkllu||
Using hyperbolic geometry for pixel-wise relationships, especially spatial propagation, is demon- strated in [22] by improving the discriminative power with minimal supervision. Following [22], embed the mixed feature E. into hyperbolic space using Eq.4 as below:
H, = exp (EM.)
(5)
where is an index of spatial coordinates in the image domain, and is the hyperbolic curvature.
Hyperbolic curvature generation. Using an appropriate curvature value an important factor in projecting Euclidean features into hyperbolic space well, which is closely related to the construction of the hierarchy structures. Previous methods mainly use fixed geometric structure regardless of data types and seene configurations by merely adjusting k as a hyperparameter [70, 73, 74, 21J. In our problem definition. according types of sensors and scene configurations, diverse data measurements and geometrical structures are observed, respectively. That's, our key observation is that a fixed and predetermined curvature may not be universally suitable.
We thus propose a curvature generation that learns a geometry-awan curved embedding space to adaptively match it to new environments and sensors. The curvature generator C is composed of a convolution layer, a multi-layer perceptron (MLP) layer, and a global mean-pooling over spatial dimensions, which yields scene-dependent curvatures based on the fused feature E as below:
K C(EM)
(6)
5

---

4.2 Sparse-to-Dense Conversion based on Hyperbolic Features
With both the high-resolution pixel-wise features from the foundation model and the sparse depth data from arbitrary sensors, we perform a tonssee conversion to obtain an initial dense depth map. Inspired by [75], we design an initial propagation process based on a bilateral filtering mechanism [76]. which is renowned for its ddge-preserving ability by incorporating both radiometric differences and spatial distances into the bilateral weight. Considering a pixel x; and the corresponding neighborhood pixel z.. the bilateral kernel filter Waj can be simply defined as:
Wsj = (r (z( s.a),
where fr is a range kernel for radiometric differences. Is is spatial kernel for physical separations in observed scenes and developed Euclidean space by calculating the distance between 3D points. For the range kernel fr. we need to design its hyperbolic version. Here, we utilize the hyperbolic feature H provided as input from Eq.5. With f, and gs. we can compute the initial dense depth as:
Dirii = Σ wiyS, s.t. Wig (DDsttiwp (H,, H2), Disteue (E EE,))) }
(8)
where Disthap is the hyperbolic function consisting of hyperbolic MLP, and Dist is the Euclidean distance in the 3-dimension coordinate. (6) means the neighborhood sparse depth of the pixel i, and Sj is the corresponding depth from a sensor. P indicates the learnable MLP layer to compute a coefficient for each sparse depth of the neighborhood Sy. Through the combination of the distance functions in Eq.8, we effectively take advantage of both hyperbolic and Eaclidean geometries to produce more accurate and robust depth maps.
4.3 Depth Refinement in Multi-curvature Hyperbolic Space
Depth refinement. To refine the initial depth in Eq.8, we employ a convolutional spatial propagation scheme, CSPN+ [26]. This refinement process leverages predefined depth map D,, augmented by sparse valid depth map S, and a multi-kernel affinity map with three different kernel sizes (3 5, 7). The use of a multi-kernel approach enables the model to capture diverse range of features from the input data, thus achieving detailed and comprehensive depth estimations. The propagation process for kernel size K E K. at step t to yield a dense map D is formulated as:
D1+1
Σ D KER
D = A D Σ A₁, O Dj,,, JEN.
s.t.
where D' is the depth map at each propagation step f. D and A are an initial depth for L = 0 and its affinity map, respectively. O is an element-wise product, and j E (i) denotes a set of neighboring pixels around pixel within & x k window. a is confidence map computed from EA in 1.c.4.
Hyperbolic convolution layer (HCL). We design the multi-kernel affinity map Aş in hyperbolic space with the proposed curvature generation module described in Sec.4.1. To do this, we formulate HCL with hyperbolic feature vector h for a 2-dimensional image domain:
HCL(h, k) == w @x The (e.))en()) (h) @x b,
(10)
where W WeRCuxGG is convolution weight matrix whose kernel size is 7, and b is bias term. Ω= ( (Zz (-+,--1'), (7', 7'), 7 [ } is a set of signed distances from a center of the convolution kernel to others in W. S. + and T are hyperbolic multiplication, addition, and concatenation, respectively, whose details are in Appendix A.I. Note that the hyperbolic MLP (Eq.8) is designed with 7 =
Mutli-curvature affinity generation. By dynamically adjusting the hyperbolic curvature for each affinity map, our approach tailors the geometrical representation to better fit the specific depth structure of each scene. We first determine the hyperbolic curvature x with Eq.6 and then compute affinity map As using a hyperbolic convolution operation equipped with kernel of size k, chosen to match the receptive field of the corresponding kernel of the affinity map Ag. This alignment optimizes the local receptive fields across the depth map. enabling more precise aggregation of context and texture information from neighboring pixels. We can calculate the hyperbolic affinity map A hep based on the generated curvature K3 from the curvature generation blocks C₁ as below:
A hap HCL(EM,, KK)
Kkk Ca (EM)
(11)

---

We can achieve the refined dense depth based on the generated hyper- bolic affinity maps A Ahup by incor- porating it into Eq.9. In particular, the employment of hyperbolic space is beneficial for depth perception by implicitly building hierarchical struc- tures [22], whose roots come from sparse points of an input depth in this work. The hyperbolic space is also ad- vantageous in regions where photo- metric distances between foreground and background pixels are marginal (see Fig.2). The detailed analysis of multi-curvature hyperbolic
0
46,0. 0.17)
(0.58. 0. 0.11)
RGB Image Figure 2: Depth propagation results according to fixed and multi-curvature values.
GT
Fixed-cur-ature
Multi-curvature
affinity described in Sec. 5.3.
Experiment and Analysis
In this section, we evaluate the performance of our proposed method for UniDC, focusing its adaptability using minimal labeled data. Firstly, we outline an overview of the experimental setup (Sec.5.1). Subsequent comparisons with various SoTA methods then presented using standard benchmark datasets (Sec.5.2). Furthermore, we conduct an ablation study to clarify the impact of each component in methodology (Sec.5.3). In Appendix.A.22 of the training procedure, datasets, and evaluation metrics in this work. Additional experiments, including full dataset training benchmarks, hyperbolic space affinity calculations, ablation study on foundation models, and varying-density performance, are included in the Appendix A.3.
introduce details
5.1 Implementation Details
Loss functions. We train method in supervised manner with linear combination of two loss terms: icali-invariant loss [77] Lucale invariam (Eg.2) for bridging the gap between relative and metric scale depths, and a composite loss TLIL2 based L2 and L2 distances for inferring the final dense depth map. In total, our framework is optimized by minimizing the final loss L as below:
L L,, + J.Locall-immariant ( (Dellative Dgt);
s.t. L.t
1
Σ iEV
(p::- Dati + +. Dat. (
(12)
M
where j is balance and is empirically set to 0.1.
Evaluation protocols. For fair evaluations, we select diverse array of SoTA depth from sparse measurements. These include a sensor-agnostic model, DepthPrompting [19] and series of SPNs such as S2D [23], CSPN [25], NLSPN [78], DySPN [27], CostDCNet [79], CompletionFormer [80], and BPNet [75]. We assess depth quality using common quantitative metries: root mean square error (RMSE, meters), mean absolute error (MAE, in meters), and inlier ratio (DELTAI, where 5 < 1.25). We employ the widely-used -sed depth completion datasets: NYU [18] and KITTI DC [81], setting up a minimal training dataset for few-shot scenarios. Note that we use their official test sets for all the comparison methods.
We implement the few-shot scenarios with and without dense depth supervision. Our experimental setup includes conducting 1-shot, 10-shot, and 100-shot learning by randomly sampling within the official training split. Additionally, we perform 1-sequence training by randomly selecting one sequence from the training set. To ensure the reliability in experiments, we randomly select 10 sequences, and report averaged results.
5.2 Experiment
Few-shot learning with dense GT. Both Tab. and Tab.3 show that existing methods face significant challenges when taking input depths from new sensors with minimal labeled data, whose examples are displayed in Fig.3 and Fig.4, respectively, In the 1-shot scenarios, where model is optimized using

---

Table 1: Quantitative results on NYUv2.
I-Shot
10-Shot
100-Shot
I-Sequence Training
RMSB
MAB
DELTAI
RMSB
MAE
DELTAI]
RMSE
MAE
DELTA
RMSE
MAE
DELTA1
CSPN [25]
14827
1,2058
0.3455
0.3166
0.1961
0.7106
0.2854
0.1307
0.9748
0.3166
0.1961
0.7106
NLSPN [78]
1.9358
1.6132
0.2229
1.5995
0.8261
0.5040
0.5501
0.4150
0.7985
0
0.6421
0,6809
DySPN [27]
1.5474
1.2851
0.3149
0.4102
0.2817
0.8595
0.2674
0.1706
0.9341
0.2584
0.1320
0.9615
Completionoomer [80]
188218
1.5539
0.2408
1.1583
1.0162
0.3079
0.9914
0.8164
0.4379
0.6779
0.5356
0.7476
CostDCNet [79]
1.2298
0.9754
0.4693
0.2363
0.1288
0.9719
0.1770
0.0836
0.9826
0.2066
0.0954
0.9788
BPNet (75)
0,3573
0.2077
0,9482
0,2392
0.1120
0.9744
0.1757
0.0793
0.9829
0.2220
0.1040
0.9765
DephPromptingg [19]
0.3583
0.2067
0.9101
0.2195
0.1006
0.9733
0.2101
0. 1008
0.9743
0.2335
0.1191
0.9686
Ours
0.1075
0,9752
0.1657
0.0794
0.9849
0.1473
0.0669
0.9885
0.1632
0.0745
0.9860
RGB Image
1-shot
100-shot
araand Trugh
Ours Figure 3: Results of 1-/100-shot on NYU. (CFormer: CompletionFormer, DP: DepthPrompting)
CSPN
NLSPN
DySPN
CFormer
CostDC
DP
BPNet
Table 2: Ablation of hyperbolic operations on zero-/few-shot performance for NYU and KITTI.
|| NYU Training
KITTI Training
NYU L-shot
10 10-shot
100-shox
KITTI
10-shox 100
Euclidean Hyperbolie
77,642
0.323/ 00.36
1.974
0,210/ 0.308
/0/799
0.147 /0.067
1,644/ 0.522
1,385 00.407
1,234 0.332
only single pair of an image and its corresponding dense depth, our model demonstrates substantial performance advantage over the comparison models. This underscores the effectiveness of using
the foundation model that does not require any additional learning for new depth representations of
unseen data. The models with large number of parameters to learn, such as CompletionFormer [80] (83.6M), often struggle to optimize with limited datasets. Since the depth prompting module in [19] requires training from scratch, it encounters difficulties in the adaptation new sensors.
Table 4: Result of few-shot learning without dense GT depth. (RMSE/MAE)
Few-shot learning without dense GT. Training with- out dense GT depths is a more practical scenario be- cause obtaining high-quality and metric-scale depth is difficult, particularly in outdoor datasets. To val- idate the applicability, we train our model in a self-supervised manner without dense GT depth. Specifically, the input LiDAR is sampled at 8-line and 32-line, while the supervision is provided by 64-Line LiDAR. This approach enables our model to adapt to sparser LiDAR inputs without the need for dense supervision. As shown in Tab.4, these results highlight our model's 's robustness and superior adaptation capabilities over BPNet [76] and DepthPromtping [19], which are the 2nd/3rd best in T and Tab.3,
KITTIDC
|| 1-shot
10-shot
100-shor
RPNes (8-1 DepchPrompenn (8-Line) Ours (8-) (8-Line)
1116//319
4.0/166 6262
4.34
3
2.89 1.12
BPNet (32)Line) DxptsPrompting Line) Ours (32-lime)
4.76/1.54
2.56 1082
2.08 7.
2.2/1 L.3
2.46/0.87
2.01/0.66
1.92/0661
/.66
5.3 Ablation Study
Probe for hyperbolic embedding. We assess the efficacy of hyperbolic embedding and curvature generation, focusing on their performance zero-shot settings. In Tab.2, the hyperbolic method yields promising results, whereas the Fuclidean approach fails. The performance gap implies that hyperbolic space offers discriminative features which guide the sparse depth propagation well. While the influence of initial parameter settings cannot be overlooked. the potential for rapid adaptation can be enhanced through well-devised initialization methods, which are in line with principles from meta-learning strategies [82, 83]. we conduct additional experiments under the few-shot regime. The results, presented in F. show a noticeable improvement when using hyperbolic space, with a performance gain of 5% average, compared to Euclidean space. This validates the effectiveness of hyperbolic geometry in depth completion tasks, especially when dealing with limited data samples.
Additionally, the analysis of the multi-curvature ap- proach for the refinement process (Tab.5) reveals that the curvature values for multi-size affinity maps in-
Table 5: Averaged curvature values.
Multi-uurvrrue (F4 II)
NYU
KITTIDC
Ko. (A (3,5,7)
0.48, 0.23, 0.07
0.41...14...17
8

---

Table 3: Quantitative results on KITTI DC.
I-Shot
10-Shot
100-Shor
I-Sequence Training
RMSB
MAB
DELTAI
RMSB
MAE
DELTAI
RMSE
MAE
DELTA
RMSE
MAE
DELTA1
CSPN [25]
9.2621
3,5736
0.9063
2.0061
0.7962
0.9758
1.4668
0.5018
0.9850
2.6406
0.8227
0.9679
S2D [23]
8.8701
5.6307
0.4222
5.0228
3.1807
0,6319
4.2582
2.6475
0.7030
4.8136
2.5358
0.8383
NLSPN (78)
7.3135
4,7084
0,5036
4.0327
2.236]
0.8662
2.4801
1.1862
0.9348
4.0535
1.7707
0.8787
DySPN [27]
2.6094
0,9082
0.9545
2.2563
0.8920
0.9487
1.8568
0.6437
0.9777
2.8369
0.8149
0.9692
Completion [80)
4,6990
2,4002
0.8234
3.1760
1.4930
0.9302
1.6263
1.3504
0.8993
4.5320
1.9822
0.8214
BPNet (75]
5,3724
1,0988
0,9690
1,8965
0.5317
0.9822
1.3126
0.3734
0.9915
2.1554
0.6241
0,9816
Dephhronpttngg [19]
2,9561
1.1657
0.9270
2.4129
1.1463
0.8943
1.7982
0.6021
0.9808
2.9616
0.9655
0.9587
Ours
0.5217
0,9826
0.4073
0.9903
0.3386
0.5406
0.9824
10-shot
CSPN
S2D
NLSPN
DySPN
Completion
BPNet
DeethPrompting
Oure
100-shot
CSPN
S2D
NLSPN
DySPN
CompletionFormer
APNet
DeathPrompiing
Ours
Figure 4: Qualitative results of 10-/100-shot on KITTI DC dataset.
crease with the kernel size. This trend suggests that information from more distant regions tends to prefer lower curvature. This observation supports the hypothesis that regions closer to the target require more distinct hyperbolic space to effectively prevent bleeding errors [22]. This insight emphasizes the importance of strategic curvature adaptation for better universal depth completion.
Table Ablation study on KITTI DC Component ablation study. We conduct ablation study on each component of our model as shown in Tab.6. The results reveal that removing feature fu- sion significantly reduces performance, particularly in 1-shot scenarios. On the other hand, the hyperbolic method shows strong adaptability with minimal data. Moreover, the fine-tuning strategy applied to the foun- dation model seems to be essential for the adaption to new environments, considering inherent discrepancies between relative and metric depth.
(RMSE/ / MAE + its standard deviation).
1-shot
10-hot
wio Eq.3
2.17 70.7310.45
1.3910.06
0.44..0.044
wio Fq.5
1.85-032200m0
0.63--0112
1 1.0+00.07 00.43--0.05
wio Eq.6
1.4=0.37
/0.6020.16
139920.0
00.41:0.05
Eq.2
2.31 011.6660.77.....
1,4010.03 10.00
Ours
1.68=0.07 /0.5220.08
Table 7: Comparison of various depth foundation models (RMSE/ / MAE).
I-shot (NYU)
10-shor (NYU)
100-shot (NYU)
1-shot (KITTI DC]
10-shot (KITTI DC)
100-shot (KITTI DXC)
DepchAnythhin [47]
0.2325 0. 1211
0,2022 0.0976
O. 1885 / 0.0843
1,9500 / 0.3004
1.880/0.4433 0.4339
1,5703 / 00,5077
UniDopth [84]
0.2149/ 0.1099
0,2020 0.0961
0 0309/00768
1.7102 0.6229
1.442// 0.43-10
1,4041 /0.4417
MiDaS [46]
0.2169 0.19
0.1627 / 0.0809
0.1455 0.0660
1.7957 / 0.5651
0.13930 / 0.4158
1,2347 / 0.3535
Table 8: Computational cost of Models.
Model
Total Param.
Leamable Param.
Inference Timets)
GPU Memory(MiB)
BPNet [75]
89.874M
20.874M
0.072
4792
LRRU [85]
20.843M
20,843M
0.038
3650
CompletionFormeer (80)
83,574M
83.574M
0.060
Ours, MiDaS [46]
21.279M
4.685M
0.056
5980
Ours_ Des__eeephhttiiiigg (47)
24.731M
2.981M
0.035
4729
Ours, UniDepth [84]
238.607M
5.852M
0.116
Foundation model variations. To evaluate the versatility of our method with various foundational models, we replace our primary backbone [46] with concurrent works, DepthAnnyting [47] and UniDepth [84], which are foundation models for relative and metric depth estimation, respectively. Since these models are based on vision transformer (ViT) [86], differing from the convolutional version of MiDaS, we compare them without the intermediate feature fusion (Eq.3). As shown in Tab.7, while these backbones exhibit comparable performance, MiDaS [46] is more suitable for 10-shot and 100-shot scenarios. We claim that the local inductive bias of convolutions operates more flcxibly in depth completion tasks, effectively propagating local information. This observation aligns

---

with the most state-of ootte-of-hhe-rr methods using convolutional encoder-decoder architectures with SPN refinement, as opposed to ViT-based architectures [47, 87, 84] for depth foundation models.
Probe for computational costs of foundation model. Depth foundation models are typically large and computationally expensive due training on extensive datasets. However, recent models offer various variants, allowing flexibility in computational demands. We conduct ablations on multiple models and observe comparable performance across them. As shown in Tab.8, MiDaS [46] and Depth Anything [47] have significantly fewer parameters than other depth completion models, suggesting that leveraging a pre-trained foundation model's knowledge does not necessarily entail high computational costs. Note that we use the publicly available official codes for MiDaS (v2.1 Small), Depth Anything v1 (ViT-S), and UniDepth (ViT-L).
Table 9: Experiment on advanced methods. To explore our method under various configurations, develop four variants by adjusting the number of channels, similar to LRRU [85].
NYU I-shot
NYU 10-shot
NYU 100-shot
KITTI 1-shot
KITTI 10 sh-
KITTI 100-shot
RMSE
DELIAT
RRMSE
DELIAI
RMMSE
MAE DELIAI
RMSE
DELIAI
RMSE
DELIAI
RMMSE
MAE
DELIAI
LRRU
0.505
0.642
0.551
0.797
6.719
0.792
5.608
0.811
2,020
0.841
LRRU Tlay
0.633
0.574
0.771
0.707
01505
0.373
7,961
0.698
6.253
3,023
0.788
4:201
2.394
0.796
Swall
0.589
0.388
0.825
0.404
0.346
0.919
0.442
0.306
0.887
16.162
8.008
0.516
5,030
2.905
0.800
5.934
3,602
0,612
LRRU Base
0.687
0.278
0.899
0.424
0.252
0.922
0.316
0.189
0.949
14,889
7,654
0.526
13,078
6.904
0,587
9.736
6.090
0.420
DFU
0.754
0.589
0.637
0.590
0.464
0.719
0.467
0.344
08668
3,652
2,0200
0.855
18899
0.966
0.973
1808
0.997
0.986
OGNI-DC
0.365 0.200
0.921
0.312
0.150
0.957
0.207
0.095
0.974
26618
0.816
0.962
1,516
0.421
0.985
1.514
0.450
0.984
Ous_Mini
0.215 0.115
0.976
0,161
0.079
0.986
0.148
0.071
0.988
2.051
0,63]
0.978
1.355
0.405
0.991
1252
0.397
0.992
Ours, Tiny
0.245 0.131
0.969
0.186
0.088
0.982
0.151
0.068
0.988
2,002
0.722
0.951
1.477
0.448
0.987
1.251
0.353
0.591
Ours
0.210 0.108
0.975
0.166
0.079
0.985
0.147
0.067
0.988
1.684
0.522
0.983
1.385
0.407
0.990
1.224
0.339
0.993
Ouri Small
0.136
0.983
0.149
0.067
0.988
0.975
0.436
0.388
0.991
Oun, Base
0.138
0.969
0.190
0.093
0.983
0.148
0.066
0.988
1,716
0.807
0,979
1.423
0.428
0.988
0.345
0.992
Table 10: Experiment on SUN RGB-D dataset.
Additional experiments recent SoTA meth- ods and other sensor. In Tab.9, we com- pare our approach with recent SoTA methods,
I-shot
10-hot
100-shot
Model
JMSE MAE DELIAL
RMSE
MAE
DELIAI
RRMSE
MAH
DELIA1
BPNei
0.497
0.244
0.870
0.342
0.164
0.900
showing its advantages across different exper-
DP
0,706
0.534
0.337
0.683
0.512
0.558
0.200
0.527
0.545
1 1RRU
0.912
0.743
0.347
0.507
0.300
0.785
0.766
0.311
0.770
imental setups. Unlike the LRRU family [85],
DFU OGNI-DC
0,850
0.270
0.896
0.817
0.314
0.552
0.382
0:401
0.881
0.438
which performs variably across datasets due to
0.529 0.285
0.418
0.188
0.895
0.345
0.166
0.901
the IP-Basic algorithm's KITTI dataset bias,
our model leverages foundation model knowledge for consistent adaptation to both indoor and outdoor environments. DFU [88] and OGNIDC [89] introduce depth feature upsampling and gradient refine- ment, respectively. Our method, however, efficiently learns hyperbolic representations on smaller datasets, enabling faster adaptation in challenging conditions. Additionally, we evaluate our model on SUN-RGBD as shown in Tab. 10, containing diverse RGB-D images from multiple sensors (Intel RealSense, Asus Xtion, Kinect V1/V2), with consistent improvements across these sensors.
Conclusion
This work starts from the new problem definition, Universal Depth Completion, to tackle the challenge of consistent depth estimation across diverse scenes and sensors. We propose simple yet universally applicable framework that leverages the knowledge of the depth foundational model and few-shot learning capabilities using hyperbolic geometry. Through various experiments in few-fzero-shot scenarios, we validate the adaptability and generality of our method.
Limitation & Future work. There are rooms for improvement. In this work, we can only use pair of an image and corresponding sparse depth as input. For general full 3D reconstruction and novel view synthesis, our method is needed to handle input pairs with multiple viewpoints. In addition, the direct application to another modality like radar is challenging due to the noisy and highly sparse nature of the radar-derived depth information. For this, we have to devise method to estimate an uncertainty on the noisy measurements, which will be one of interesting future works.
Acknowledgement. This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea governmen((MSTTyRR (R-2))2)R--222-000000 2024-00338839), the Institute of Information & communications Technology Planning & Evaluation (ITP) grant funded by the Korea government (MSIT) (No.2019-0-0188) Artificial Intelligence Graduate School Program (GIST), RS-2021-11212 1068, Artificial Intelligence Innovation Hub), GIST-MIT Research Collaboration grant funded by the GIST in 2024, 'Project for Science and Technology Opens the Future the Region' program through the INNOPOLIS FOUNDATION funded by Ministry of Science and ICT (Project Number: 3022-DD-UP-031 12), and Local Finance Association(LOFA) grant funded by the Korea government(A A Study on the Complete Survey of Advertisements Using Artificial Intelligence (Al) Technology).
10

---

References
[1] Maximilian Jaritz, Raoul De Charette, Emilic Wirbel, Xavier Perrotton, and Fawzi Nashashibi, Sparse and dense data with cnns: Depth completion and semantic segmentation. 2018 International Conference on 3D Vision (3DV), pages 52-60. IEEE, 2018.
[2] Ce Liu, Suryansh Kumar, Shuhang Gu, Radu Timofte, and Lue Van Gool. Single image depth prediction made better: multivariate gaussian take. In Proceedings of IEEF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[3] Shuwei Shao, Zhongeai Pei, Weihai Chen, Xingming Wu, and Zhengguo Li. Nddepth: dermal-distance assisted monocular depth estimation. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2023. 1
[4] Zhiqiang Yan, Kun Wang, Xiang Li, Zhenyu Zhang, Guangyu Li, Jun Li, and Jian Yang. Learning com- plementary correlations for depth super-resolution with incomplete data in real world. IEEE transactions on neural networks and learning systems, 2022.
[5] Fangchang Ma, Guilherme Venturelli Cavalheiro, and Sertae Karaman. Self-supervised sparse-to-dense Self-supervised depth completion from lidar and monocular camera. In Proceedings IEEE International Conference on Robotics and Automation (ICRA), 2019. 1
[6] Kyeongha Rho, Jinsung Ha, and Youngjung Kim. Guideformer: Transformers for image guided depth completion. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[7] Zhiqiang Yan, Xiang Kun Wang, Zhenyu Zhang, Li, and Jian Yang. Multi- multi-modal masked pre-training for monocular panoramic depth completion. In Proceedings of European Conference on Computer Vision (ECCV), pages 378-395 Springer, 2022. 1
[8] Zhiqiang Yan, Xiang Li, Kun Wang. Shuo Chen, Jun Li, and Jian Yang. Distortion and uncertainty aware loss for panoramic depth completion. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
[9] Kun Wang, Zhenyu Zhang, Zhiqiang Yan, Xiang Li, Baobei Xu, Jun Li, and Jian Yang. Regularizing nighttime weirdness: Efficient self-supervised monocular depth estimation in the dark. In Proceedings of International Conference Computer Vision (ICCV), 2021.
[10] Zhiqiang Yan, Kun Wang, Xiang Li, Zhenyu Zhang, Jun Li, and Jian Yang. Desnet: Decomposed scale-consistent network for unsupervised depth completion. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2023.
[11] Zhiqiang Yan, Yupeng Zheng, Kun Wang, Xiang Li, Zhenyu Zhang, Shuo Chen, Jun Li. and Jian Yang. Learnable differencing center for nighttime depth perception. arXiv preprint arXiv:2306. 14538, 2023. I
[12] Andreas Geiger, Martin Roser, and Raquel Urtasun. Efficient large-scale stereo matching. Asian conference on computer vision, pages 25-38. Springer, 2010. I
[13] Jia-Ren Chang and Long-Sheng Chen. Pyramid stereo matching network. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1
[14] Heiko Hirschmuller and Daniel Scharstein. Evaluation of cost functions for stereo matching. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2007.
[15] Alexandre Lopes, Roberto Souza, and Helio Pedrini. A survey rgh-d datasets. Computer Vision and Image Understanding, 203:1034,9, 2022.
[16] Michael Firman. Rgbd datasets: Past, present and future. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), 2016. 1
[17] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research (I/RR), 32(11::1311-233, 2013. 2, 17
[18] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rghd images. In Proceedings of European Conference on Computer Vision (ECCV), 2012.
2,7
[19] Jin Hn- Park. Chanhwi Jeong. Junoh Lee, and Hae-Gon Jeon. Depth prompting for sensor-agnostic depth estimation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2, 3, 8, 9, 18, 19, 20
11

---

[20] Joy Hsu, Jeffrey Gu, Gong Wu, Wah Chiu, and Serena Yeung. Capturing implicit hierarchical structure in 3d biomedical images with self-supervised hyperbolic representations. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2021. 3,5
[21] Antonio Montanaro, Diego Valsesia, and Enrico Magli. Rethinking the compositionality of point clouds through regularization in the hyperbolic space. 35:33741-33752 2022. 3,5
[22] Jin-Hwi Park, Jaesung Choc, Inhwan Bac, and Hac-Gon Jcon. Learning affinity with hyperbolic repre- sentation for spatial propagation. In Proceedings of the International Conference on Muchine Learning (ICML), 2023. 2, 3, 7, 9. 19
[23] Fangchang Ma and Sertac Karaman. Sparse-to-dense: Depth prediction from sparse depth samples and single image. Proceedings of IEEE International Conference on Robotics and Automation (ICRA), 2018. 2, 7,
[24] Zhenyu Li, Zehui Chen, Xianming Liu, and Junjun Jiang. Depthformer: Exploiting long-range correlation and local information for accurate monocular depth estimation. Machine Intelligence Research, pages 1-18, 2023. 2,
[25] Xinjing Cheng, Peng Wang, and Ruigang Yang. Depth estimation via affinity learned with convolutional spatial propagation network. In Proceedings of European Conference on Computer Vision (ECCV), 2018.
2. 7, 8, 9, 19
[26] Xinjing Cheng, Peng Wang, Chenye Guan, and Ruigang Yang. Cspn++:: Learning context and resource aware convolutional spatial propagation networks for depth completion. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020. 2,
[27] Yuankai Lin, Tao Cheng, Qi Zhong, Wending Zhou, and Hua Yang. Dynamic spatial propagation network for depth completion. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022. 2, 7, 8, 9, 19
[28] Mu Hu, Shuling Wang, Bin Li, Shiyu Ning, Li Fan, and Xiaojin Gong. Penet: Towards precise and efficient image guided depth completion. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA), 2021.
[29] Xin Liu, Xiaofei Shao, Bo Wang, Yali Li, and Shengjin Wang. Graphespn: Geometry-aware depth completion via dynamic gens. In Proceedings of European Conference on Computer Vision (ECCV), 2022. 2
[30] Brent Schwarz. Mapping the world in 3d. Nature Photonics, 4(7):429-430) 2010. 2
[31] Zhengyou Zhang. Microsoft kinect sensor and its effect. IEEE multimedia, 19(2):4 10, 2012. 2
[32] Andrea Conti, Matteo Poggi. and Stefano Mattoccia. Sparsity agnostic depth completion. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision. pages 587 1871-5880, 2023. 2, 18
[33] Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Simon Chen, and Chunhua Shen. Towards domain domain-agnostic depth completion. arXiv preprint arXiv:2207. 14466, 2022.
[34] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuar Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the International Conference on Machine Learning (ICML). 2021. 2
[35] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint axXiv:211..11422, 2021.
[36] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 2023.
2
[37] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Mariaann Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikol aj Binkkoski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for shot learning. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2022. 2
12

---

[38] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Proceedings European Conference on Computer Vision (ECCV), 2022. 3,4
[39] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinyt:: Reduce memory, not parameters for efficient on-device learning. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2020. 3,
[40] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203./7274, 2022.
[41] Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge-gun guided context optimization. In Proceedings IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023. 3
[42] Shoufa Chen, Chongjian Ge, Zhan Tong. Jiangliu Wang. Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Proceedings of the Neural Information Processing Systems (NeurIPS), 2022. 3
[43] Peng Gao, Shijie Geng, Renrui Zhang. Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision- language models with feature adapters. International Journal of Computer Vision, (112)))581-955 2024. 3
[44] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In Proceedings of European Conference on Computer Vision (ECCV), pages 696-712. Springer, 2022. 3
[45] Jaime Spencer, Chris Russell, Simon Hadfield, and Richard Bowden. Kick back & relax: Learning to reconstruct the world by watching slowtv. In Proceedingy of International Conference on Computer Vision (ICCV), 2023. 3
[46] Reiner Birkl, Diana Wolk, and Matthias Müller. Midas v3.1 - a model zoo for robust monocular relative depth estimation. arXiv preprint vrXiv:2307.1446/ 2023. 3, 9, 10, 17
[47] Lihe Yang. Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao, Depth anything: Unlcashing the power of large-scale unlabeled data. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2024. 9, 10
[48] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In Proceedings of International Conference Computer Vision (ICCV). 3
[49] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing Liffusion-basee image generators for monocular depth estimation. arXiv preprint arXiv:2312.02145, 2023. 3
[50] Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. 2024.
[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. Heshuresolution image synthesis with latent diffusion models. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3
[52] Maximillian Nickel Douwe Kiela. Poincaré embeddings for learning hierarchical representations. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2017. 3
[53] Maximillian Nickel and Douwe Kiela. Learning continuous hicrarchies the Lorentz model hyperbolic geometry. In Proceedings of the International Conference on Machine Learning (ICML), 2018. 3
[54] Ankit Dhall, Anastasia Makarova, Octavian Ganea, Dario Pavilo, Michacl Greeff, and Andreas Krause. Hierarchical image classification using entailment cone embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), 2020. 3
[55] Mina Ghadimi Atigh, Martin Keller-Ressel, and Pascal Mettes. Hyperbolic busemann learning with ideal prototypes. In Proceedings of the Neural Information Processing Systems (NeurIPS), volume 34, pages 103-115, 2021. 3
[56] Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong Wah Ngo, Tat-Seng Chua, and Yu- Yu-Gang Jiang. Hyperbolic visual embedding learning for zero-shot recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
13

---

[57] Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedantam. Hyperbolic image-text representations. In Proceedings of the International Conference on Muchine Learning (ICML), 2023. 3
[58] Teng Long, Pascal Mettes, Heng Tao Shen, and Cees G. M. Snoek. Searching for actions on the hyperbole. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 3
[59] Alcksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkav, Nicu Sebe, and Ivan Oseledets, Hyperbolic vision transformers: Combining improvements in metric learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[60] Mina GhadimiAtigh, Julian Schoep, Erman Acar, Nanne van Noord, and Pascal Mettes. Hyperbolic image segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3,4 4
[61] Dídac Surís, Ruoshi Liu, and Carl Vondrick. Learning the predictabilityy of the future. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2021. 3
[62] Luca Franco, Paolo Mandica, Bharti Munjal, and Fabio Galasso. Hyperbolic self-paced learning for self-supervised skeleton-based action representations. arXiv preprint 2rX1v:2333.06222, 2023. 3
[63] Joy Hsu, Jeffrey Gu, Gong Wu, Wah Chiu, and Serena Yeung. Capturing implicit hierarchical structure 3d biomedical images with self-supervised hyperbolic representations. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2021. 3
[64] Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton. Latent variable modelling with hyperbolic normalizing flows. In Proceedings of the International Conference Machine Learning (ICML), 2020. 3
[65] Emile Mathicu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Tch. Continuous hicrarchical representations with poincaré variational auto-encoders. In Proceedings of the Neural Information Processing Systems (NeurIPS), volume 32, 2019. 3
[66] Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped normal dis- tribution on hyperholic space for gradient-based learning. In Proceedings of the International Conference on Machine Learning (ICMI.) 2019. 3
[67] Lingxiao Li, Yi Zhang, and Shuhui Wang. The euclidean space is evil: hyperbolic attribute editing for few-shot image generation. In Proceedings of International Conference on Computer Vision (ICCV), 2023. 3,4
[68] Zhi Gao, Yuwei Wu, Yunde Jia, and Mehrtash Harandi. Curvature generation in curved spaces for few-shot learning. In Proceedings of International Conference on Computer Vision (ICCV), 2021. 3,
[69] Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 3,
[70] Nurendra Choudhary, Nikhil Rao, and Chandan Reddy. Hyperbolic graph neural networks at scale:
A meta learning approach. In Proceedings of the Neural Information Processing Systems (NeurIPS), volume 36, 2024. 3, 4,5
[71] Sifei Liu, Shalini De Mello, Jinwei Gu, Guangyu Zhong, Ming-Hsuan Yang, and Jan Kautz. Learning affinity via spatial propagation networks. In Proceedings of the Neural Information Processing Systems (NeuriPS), 2017. 3
[72] Neil Houlsby, Andrei Giurgiu. Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo. Mona Attariyan, and Sylvain Gelly. Parameter-effice transfer learning for nlp. In Proceedings of the International Conference on Machine Learning (ICMI), 2019. 4
[73] Shu-Lin Xu, Yifan Sun, Faen Zhang, Anqi Xu. Xiu-Shen Wei, and Yi Yang. Hyperbolic space with hicrarchical margin boosts fine-grained learning from coarse labels. In Proceedings of the Neural Information Processing Systems (NeurIPS), volume 36, 2024.
[74] Zhi Gao, Yuwei Wu, Yunde Jia, and Mehrtash Harandi. Hyperbolic feature augmentation via distribution estimation and infinite sampling on manifolds. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2022. 5
14

---

[75] Jie Tang, Fei-Peng Tian, Boshi An. Jian Li, and Ping Tan. Bilateral propagation network for depth completion. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 6, 8,9
[76] Carlo Tomasi and Roherto Manduchi, Bilateral filtering for gray and color images. In Proceedings of International Conference on Computer Vision (ICCV), 1998. 6, 8
[77] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using a multi-scale deep network. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2014.
7
[78] Jinsun Park, Kyungdon Joo, Zhe Hu, Chi- Kuei Liu, and So Kweon. Non-local spatial propagation network for depth completion. In Proceedings of European Conference on Computer Vision (ECCV),
2020. 7, 8, 9, 19
[79] Jaewon Kam, Jungeon Kim, Soongjin Kim, Jacsik Park, and Scungyong Lec. Costdenet: Cost volume based depth completion for single rgb-d image. In Proceedings European Conference Computer Vision (ECCV), 2022. 7,8
[80] Youmin Zhang, Xianda Guo, Matteo Poggi, Zheng Zhu, Guan Huang. and Stefano Mattoccia. Com- pletionformer: Depth completion with convolutions and vision transformers. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 7, 8, 9, 18
[81] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant cnns. In International Conference on 3D Vision (3DV), 2017. 7, 18
[82] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
[83] Chelsea Finn, Kelvin Xu, and Sergey Levine. Prohabilistic model-agnostic meta-learning. Proceedings of the Neural Information Processing Systems (NeurIPS), 2018. 8
[84] Luigi Piccinelli, Yung- Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Lue Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 9, 10
[85] Yufei Wang, Bo Li, Ge Zhang, Qi Liu, Tao Gao, and Yuchao Dai. Lrru: Long-short range recurrent updating networks for depth completion. In Proceedings of International Conference on Computer Vision (ICCV), 2023. 9, 10, 18
[86] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In Proceedings of International Conference on Learning Representations (ICLR), 2021.
[87] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of International Conference on Computer Vision (ICCV), 2021. 10
[88] Yufci Wang, Ge Zhang, Shaoqian Wang, Bo Li, Qi Liu, Le Hui, and Yuchao Dai. Improving depth completion via depth feature upsampling. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2024. 10
[89] Yiming Zuo and Deng. Ogni-de: Robust depth completion with cotimiizatiin-guuddee neural iterations.
arXiv preprint arXiv:2406.11711 2024. 10
[90] Abraham Albert Ungar. A gyrovector space approach to hyperbolic geometry. In Synthesis Lectures on Muthematics and Statistics, 2008. 17
[91] Octavian Ganea, Gary Becigneul, and Thomas Hofmann. Hyperholic neural networks. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2018. 17
[92] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zem- ing Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Proceedings of the Neural Information Processing Systems Workshop, 2017. 17
[93] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv.1412.6980, 2014. 17
15

---

[94] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using multi-scale deep network. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2014.
[95] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant cnns. International Conference on 3D Vision (3DV), 2017. 17
[96] Jinyoung Jun, Jae-Han Lee, and Chang-Su Kim. Masked spatial propagation network for sparsity-adapp depth refinement. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 18
[97] Jinhyung Park, Yu-Jhe Li. and Kris Kitani. Flexible depth completion for sparse and varying point densities. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 18
[98] Haotian Wang, Meng Yang. and Nanning Zheng. G2-monodepth: A general framework of generalized depth inference from monocular rgb+ data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 18
[99] Haotian Wang, Meng Yang. Xinhu Zheng, and Gang Hua. Scale propagation network for generalizable depth completion. arXiv preprint arXiv:2410.18408, 2024. 18
[100] Vadim Ezhov, Hyoungseob Park. Zhaoyang Zhang, Rishi Upadhyay. Howard Zhang, Chethan Chinder Chandrappa, Achuta Kadambi, Yunhao Ba, Julie Dorsey, and Alex Wong. All-day depth complction. arXiv preprint arXiv:2405.1 17315, 2024. 18
[101] Yangchao Wu, Tian Yu Liu. Hyoungseob Park, Stefano Soatto, Dong Lao, and Alex Wong. Augundo: Scaling up augmentations monocular depth completion and estimation. In Proceedings of European Conference on Computer Vision (ECCV), 2024. 18
[102] Suchisrit Gangopadhyay, Xien Chen, Michael Chu, Patrick Rim, Hyoungseob Park, Alex Wong. Uncle: Unsupervised continual learning depth completion. arXiv preprint arXiv:2410.18074 2024. 18
[103] Hyoungseob Park, Anjali Gupta, and Alex Wong. Test-time adaptation for depth completion. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 18
[104] Tian Yu Liu, Parth Agrawal, Allison Chen, Byung- Woo Hong, and Alex Wong. Monitored distillation for positive congruent depth completion. In Proceedings of European Conference on Computer Vision (ECCV), 2022. 18
[105] Alex Wong and Stefano Soatto. Unsupervised depth completion with calibrated backprojection layers. In Proceedings of International Conference on Computer Vision (ICCV), 2021. 18
16

---

A Appendix/ supplemental material
A.1 Hyperbolic Geometry
Revisit to Poincaré ball model. We revisit some definitions of a hyperbolic ball model and the details of the fundamental arithmetic operations in the hyperbolic space. The Poincaré ball model (D g*) with curvature K. is defined by manifold D = {x E R" <||==|| < 1] equipped with metric g². where || denotes the Euclidean norm. In contrast to traditional vector spaces, hyperbolic spaces require distinct approaches for mathematical operations. Therefore, we employ the framework of Möbius gyrovector spaces, a generalization of Euclidean vector spaces adapted for hyperbolic models. Based on Möbius transformation [90], there are fundamental arithmetic operations in the hyperbolic space, such as addition (Ox) and multiplication (8x). Furthermore, we exploit bijective mapping functions (expğ and logā) between hyperbolic space and Fuclidean space.
Möhius addition.
For a pair (u, v) TD the Möbius addition is defined as follows:
u L = (1 + 2 2k,uu, (u, v) + K| k||v\))) + (1 K |/)
(13)
+ 2x u, v) +
where (.-)) is the Euclidean inner product.
Möbius matrix-vector multiplication. For an arbitrary function f K" RT in the Euclidean space, the Möbius version of f is function that maps from D to 13 in the hyperbolic space using Equation 4. Similarly, we can derive the Möbius matrix- multiplication between the matrix M and input u, which is defined as:
|Mu|| |u||
Mu
M u (1/Vk) tanh
tanh -1 ((R]
(14)
Muu]
Hyperbolic concatenation. Given image feature maps F. in Euclidean space, we pixel-wisely embed an image feature vector at pixel (I, (i.e., f(x.g) EXCxIXI) into the hyperbolic space. Here, we utilize an exponential mapping M(.)= on the Poincaré ball IDC as a bijective function between the Fuclidean space and the hyperbolic space via Poincaré curvature k. For concatenating features in hyperbolic space, we apply the 3-concatenation proposed in [91] as below:
(x1, .2.... EN) = M ((( of ... Bn 3xBT2"1)')
(15)
The points x, in the Poincare ball D are projected back u,=M-1(xx) with the scalar coefficient Bn = B(5 2 4), where the Beta distribution.
A.2 Experiment Details
Training details. We utilize MiDaS [46] depth foundation model whose pre-trained knowledge is transferred into our universal model. Our model is implemented with public PyTorch [92]. trained on single RTX 3090Ti GPU using Adam [93] optimizer. All training is conducted in few-shot manner, with the number of iterations ranging from 100 to 3,000, depending on the size of the training dataset, e.g., 1-shot, 10-shot, and 100-shot. Note that resize input RGB images to keep the ratio of height/width toward MiDaS. The initial learning rate was set to 5 x 10-3 and reduced by 0.1 every 20% for total iterations. The proposed framework comprises 4.6M learnable parameters, including 41K dedicated to tuning the foundational model. To facilitate fair comparison, each experiment is repeated 10 times with the same seeds (e.g.. 9), and we report the average test accuracy.
Evaluation metries. We introduce a depth quality evaluation metrics, proposed in [94, 17, 95]. We compare the competitive depth completion model and ours using official evaluation metries: RMSE, MAE, and 01.25- Given ground truth depth D {d} and the predicted depth D {d), the metrics are as follows:
Root mean squared error (RMSE): V bb EieD ld d/2
Mean absolute error (MAE):
1 D Eie ld d
17

---

Percentage of predicted pixels where the relative error is within a threshold (6) )
card {fde D: card (D)
D: max 2
1.25
where the card is the cardinality of a set. Note that a higher o; indicates better prediction.
Depth completion datsets: NYUv2 and KITTIDC. We employ the NYU Depth V2 dataset, which consists of 464 indoor scenes captured using a Kinect sensor. Adhering to the established train/test di- vision, we evaluate our trained model on 215 scenes (654 samples). The NYU Depth V2 dataset offers images at 320x240 resolution. We utilize center-cropped images at 304x228 resolution and randomly select 500 points to emulate sparse depth data. For 1 I-sequence training setup, we choose sequences from the training dataset: [conference_rowe srs__rrom_m0000 study_room 10000,000440044 cereption_roo_____ room_0006, Diving_room__ie gasement_(0011)
play- Kitchen_0010, classroom_ _016, bedroom_004) bathroom_0044
For outdoor environments, we utilize the KITTI DC dataset, which comprises 90K samples. Each sample includes color images and corresponding sparse depth data, captured at approximately 6% density relative image resolution using a Velodyne HDL-64E LiDAR sensor. The images are provided at a resolution of 1216x352. The dataset is segmented into training (86K samples), validation (7K samples). and testing (IK samples) portions. Ground truth (GT) is generated by accumulating multiple LiDAR frames and removing inaccuracies, resulting in enhanced LiDAR depths of about 20% density. For I-sequent training setup, we utilize the following 10 sequences: __o0e_yyene,,
[2011 1_10__1900_111 110_26_____ weddd_rvv000 _syne, 2011 2011_09_00 ____2_____ drive_0017_sse
2011_09__ _0__009____000_ 23__d_ie__00e1 seree_005_ssye _ynne teee__003_syye,.e_eeeee__ 2011 2011_09__1 _0____9________ 366_d_ive_ee_ Daavee0006__a0 synne, 2011 0__00_222 ____2_dmes drive ___iiveee34_____________ 0034_syne,
2011 2011_09 10_0009_______ 10__2__rrin 1is_ddive__00
2011 1000990220 28 Lavv__094_synne
2011_09 28 drive_0168 sync. 2011_09 29 drive_ 0004 sync,
1011_09_330 ddive_rr0v_______
A.3 Additional Experiments
Table 11: Full Dataset Training Benchmark on NYU and KITTI dataset.
# Learnable (M)
NYU
KITTI RMSE
Models
Params.
RMSE
MAE
MAE
41.5M
Cforner Tiny
0.091
0.035
82.6M
Cformer Small (Github)
0.090
0.035
0.739
0.196
142.4M
Cformer Base
0.090
0.035
0.709
0.203
0.3M
LRRU_ Mini
0.101
0.800
0.219
1.3M
LRRU_ Tiny
0.762
0.208
5.2M
LRRU_
0.093
0.741
0.202
21M
LRRU Base
0.091
0.728
0.198
1.2M
Ours Ours_Tiny
0.107
0.042
0.907
0.231
4.6M
Ours
0.098
0.038
0.867
0.224
36.9M
Ours_Smal
0.095
0.038
0.824
0.209
63.2M
Oury_ Base
0.093
0.036
Full dataset training benchmark on NYU and KITTI dataset. We report the performance of our work in the KITTI benchmark [81], which reported in Tab. 1l.11....1..1 To analyze our method under various configurations. similar to recent SoTAS such as LRRU [85] and CompletionFormer [80], we designed four variants by adjusting the number of channels. Notably, both our method and these methods totally follow the scaling laws of deep learning models. Our variants achieve competitive results compared to the SoT methods, especially setups with fewer labels.
We emphasize that, over the past decade, numerous depth completion papers have focused on in- domain experiments on NYU and KITTI datasets. However, there has been recent trend towards addressing out-of-domain challenges in depth completion research [32, 96, 19, 97, 98, 99, 100, 101, 102, 103, 104, 105]. This direction aims to develop models that can handle variations in new sensor configurations, unseen environmental conditions, and training schemes without the need for dense GT. This trend is gaining attraction in top-tier conferences and journals, highlighting the importance of adaptability and generalization in depth completion models. Our research aligns with this direction and shares similar goals. Note that most of those works do not consider the KITTI benchmark, which an in-domain experiment with a 64-Line LiDAR sensor. While we agree that top-tier papers should demonstrate certain level of performance, we also believe that research focusing on generalization and adaptability for arbitrary sensors and environments is valuable and deserves recognition.
18

---

Table 12: Ablation study on SPNs with hyperbolic operation.
NYU
1-shot
10-shot
100-shot
Model
RMSE
MAE
DELTA
RMSE
MAE
DELTAI
RMSE
MAE
DELTAI
CSPN [25]
1,483
1.213
0.266
0.470
0.330
0.839
0.222
0.106
0.973
CSPN Hyp
1.188
0.950
0.398
0.429
0.271
0.866
0.186
0.101
0,982
NLSPN [78]
1.396
1.136
0.290
0.925
0.719
0.489
0.283
0.192
0.952
NLSPN Hyp
1.338
1.079
0.328
0.353
0.208
0.934
0.211
0.133
0.978
DySPN [27]
1.499
1.210
0.283
0.567
0.422
0.742
0.243
0.117
0.972
DySPN Hyp
1,303
1.044
0.035
0.428
0.304
0.807
0.216
0.103
0.978
Advantages of hyperbolic space for calculation of the pixel affinity map. Most spatial propagation networks (e.g.. CSPN [25], NL.SPN [78], and DySPN [27]) adopt encoder-decoder structures to extract multi-scale features w.r.t. structure and photometric similarities. Then, initial seeds (i.e., sparse depth) are propagated based on affinity maps computed from the learned features in an iterative manner. Therefore, if the computed affinity map is accurate, capturing boundary information, which is the highly ambiguous region for pixel-wise prediction task, concomitant. However, object boundary ambiguities, caused by noise or smooth intensity changes, can lead to bleeding errors [22]. To address these issues, we formulate these hierarchical relations a continuous and differentiable manner. The hyperbolic space naturally accommodates exponentially growing hierarchies and tree-like structures, allowing robust affinity construction with low distortion.
We conducted toy example to verify the effectiveness of hyperbolic geometry in various propagation schemes, including CSPN (Convolutional), NL.PP (Non-Local), and DySPN (Dynamic attention). Using the same backbone (ResNet-34) and loss functions (L1 and L.2) across all schemes ensures a fair comparison. As shown Tab. 12, hyperbolic operations significantly improve performance in various few-shot setups. Compared to Euclidean methods, hyperbolic structures improve pixel distinction under challenging conditions.
Table 13: Ablation study on VFM (Visual Foundation Model) for various depth completion models.
NYU
1-shot
10-shot
100-shot
Model
RMSE
MAE
DELTAI
RMSE
MAE
DELTA1
RMSE
MAE
DELTAI
CSPN
1.483
1,206
0.346
0.317
0.196
0.711
0,285
0.131
0.975
CSPN + "VFM"
0.569
0.438
0.756
0,533
0.408
0.787
BPNet
0.357
0.208
0.948
0.239
0.112
0.974
0.176
0.079
0.983
+PNet+ "VFM"
OGNIDC
0.365
0,200
0.921
0.312
0.160
0,957
0.207
0.095
0.974
OGNIDC + "VFM"
0.695
0.323
0.888
0.372
0.189
0,932
0,248
0.148
0.958
DepthPrompting Ours
0.358
0,207
0.910
0.220
0.101
0.973
0,210
0.101
0.974
0.210
0.108
0.975
0.166
0.079
0.985
0.147
0.067
0.988
Ablation study on VFM (Visual Foundation Model). We carry out additional experiments using VFM knowledge in conventional methods by replacing the sparse depth input with Eq.4 of the paper [19]. In this experiment, we found that directly applying the VFM approach, as suggested, sometimes yields unsatisfactory performance compared to the baseline, shown in Tab. 13. This underperformance can be attributed to optimization issues stemming from the fact that the sparse depth provides complete metric depth information, whereas fitting the relative-scale depth from VFM using Eq.4 of DepthPrompting [19] does not achieve this precision level. The fitting process involves solving AxB, which performs linear fit with the available data, i.c., sparse depth. In [19], the authors employed global linear fitting to predict the depth scale using scalar values A and B, initially converting relative depth to metric scale. However, this approach often fits disproportionately to regions with rich information, leading to inaccuracies in areas with sparse depth information. Consequently, using metric sparse depth as input can cause inaccuracies, making optimization difficult and resulting in suboptimal performance. We agree that there is significant gap in using VFM directly for depth completion. Instead of directly using relative-scale depth, we chose to leverage intermediate features to indirectly utilize foundation knowledge. This approach allows us to benefit from VFM while avoiding the direct application of relative-scale depth, thereby mitigating some of the challenges observed in this experiment.
19

---

Table 14: Varying density experiment.
NYU #Sample 100
1-shot
10-shot
100-shot
Model
RMSE
MAE
DELTAI
RMSE
MAE
DELTAI
RMSE
MAE
DELTAI
BPNet
0.737
0.436
0.876
0.319
0.177
0.942
0.276
0.149
0,955
LRRU
0.512
0.344
0.849
0.453
0.184
0,927
OGNIDC
0.439
0.274
0.884
0.394
0.176
0.933
0.287
0.154
0,951
Ours
0.326
0.196
0.936
0.261
0.151
0.962
0.227
0.196
0,971
NYU #Sample
1-shot
10-shot
100-shot
Model
RMSE
MAB
DELTAI
RMSE
MAE
DELTA1
RMSE
MAE
DELTA1
BPNet LRRU
0.676
0.486
0.763
0.492
0.326
0.851
0.403
0.888
0.735
0.547
0.688
0.678
0.496
0.719
Ours
0.486
0.325
0.852
0.380
0.244
0.893
0.312
0.190
0.935
KITTI 16-Line
I-shot
10-shot
100-shot
Model
RMSE
MAE
DELTAI
RMSE
MAE
DELTAI
RMSE
MAE
DELTAI
BPNet
3.387
1.203
0.954
3.063
1.086
0.964
2.305
0.800
0.975
DFU
4.357
2.139
0.862
3,935
1.885
0.911
2.990
1.428
0.950
OGNIDC
5,590
2.540
0.797
2.570
0.898
0.965
2,413
0.832
0,969
Ours
2.827
1.020
0.964
2.319
0.845
0.979
2.215
0.745
0.975
KITTI Line
1-shot
10-shot
100-shot
Model
RMSE
MAE 2.886
DELTAI 0.775
RMSE
MAE
DELTAI 0,863
RMSE
MAE
DELTA
BPNet
5.568
5.332
2.384
4,471
1.844
0.906
DFU
5.558
3.017
0.682
4.872
2.569
0.793
Ours
4.790
2.224
0.872
4.153
1.918
0.895
4.084
1.659
0.926
Varying-density Performance. We simulate different LiDAR setups by varying the density of the input data. For the NYU indoor dataset, randomly sampled 100 and 32 sparse depths, while for the KITTI outdoor dataset, we utilize 16-Line and 4-Line configurations. These experiments test the robustness and adaptability of our method in response changes in input data quality and quantity. As shown in Tab. 14, the results show that our method achieves superior performance across different sensor configurations. In contrast, most comparison methods exhibit decline in performance when adapting to new sensor configurations, as demonstrated in the DepthPrompting [19].
20

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We have described our contributions in both abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have written our limitations in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
We have sincerely mentioned the assumption and proof of our work like input modality and technical supports, respectively.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We make the detailed description that readers reproduce our method easily.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: We will make our code publicly available, along with links to download the datasets used.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https:
clltss.ccppubiic/// http:/pbllcggguugeee guwde//eeeeeeee /CodeSummi ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer:
[Yes]
Justification: section3, 4, 5 and appendix, we have specified all the training/test details. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have included tables and examples to Statistically support technical signifi- cance of our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We have mentioned information our computing resource to implement and test our method and comparions works.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://rrrs....cpcii////ee/eeeee
Answer: [Yes]
Justification:
We follow the guideline of NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is about the enhancement of commercial sensors, which is not related to the societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We think that this work has no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have only used public datasets which have been widely utilized in research fields, and mentioned it in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new asset.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work not related to IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "d0f0GhOtYK", "reviewer": "Reviewer_tAUP", "summary": "This paper proposes a universal depth completion method (UniDC) to address generalization issues in unknown scenes and arbitrary depth sensors. The method utilizes depth information extracted from a pre-trained monocular depth estimation model to generate pixel-wise affinity maps, which adjust sparse depth. Additionally, this paper develops a hyperbolic propagation method for generating and refining dense depth.", "questions": "Please refer to weaknesses. Moreover，\n\n1. Compared to previous spatial propagation networks (SPN, CSPN, DySPN, etc.), what advantages does the calculation of the affinity map A_{j,k} in hyperbolic space proposed by Equation (11), and why can it better refine depth maps?\n\n2. When incorporating a pre-trained depth estimation model into the network, it results in additional computational costs, particularly for large models like DepthAnything. How do you mitigate this issue?", "limitations": "The limitations are discussed in detail in the paper. Estimating the uncertainty of noise will better enhance the model's generalization capability.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.This paper introduces a pre-trained depth estimation model for depth completion tasks and proposes a universal depth completion framework that effectively addresses depth completion challenges across various scenes and different depth sensors.\n\n2. The proposed method performs 1-Shot, 10-Shot, and 100-Shot experiments on the NYU and KITTI DC datasets, demonstrating that UniDC achieves outstanding performance and generalization ability.", "weaknesses": "1. The proposed method does not provide comparative results on the online KITTI depth completion benchmark, and these results need to be further presented.\n\n2. Some advanced and state-of-the-art depth completion methods need to be discussed and compared.\n\n[1] Improving Depth Completion via Depth Feature Upsampling. CVPR 2024.\n\n[2] Tri-Perspective View Decomposition for Geometry-Aware Depth Completion. CVPR2024.\n\n[3] LRRU: Long-short Range Recurrent Updating Networks for Depth Completion.  ICCV 2023.\n\n[4] BEVDC: Bird's-Eye View Assisted Training for Depth Completion. CVPR 2023.\n\n[5] RigNet: Repetitive Image Guided Network for Depth Completion. ECCV 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [tAUP] W.1 Online KITTI depth completion benchmark.\nWe thank the reviewer for the comments about the outstanding performance and generalization ability of our method. Following your comment, we report the performance of our work in the KITTI benchmark, which is reported in Figure.A of the uploaded PDF file. To analyze our method under various configurations (Table.C), similar to recent state-of-the-art methods such as LRRU and CompletionFormer, we designed four variants by adjusting the number of channels. Notably, both our method and these methods totally follow the scaling laws of deep learning models [1, 2]. Our variants achieve competitive results compared to the state-of-the-art methods, especially in setups with fewer labels, where our approach demonstrates superior performance.\n\n&nbsp;\n\n### [tAUP] W.2 Comparison with recent SoTA methods\nWe have provided a detailed comparison with recent state-of-the-art methods in Table.A. This comparison highlights the advantages of our approach across different experimental setups.\n\nIn the case of the LRRU family [3], we observe that in the KITTI dataset, smaller models tend to perform better, whereas in the NYU dataset, larger models achieve better performance. This behavior is attributed to the IP-Basic algorithm [4] used for preprocessing, which is biased towards the KITTI dataset. Unlike LRRU, our approach leverages foundation model knowledge, enabling consistent and rapid adaptation to both indoor and outdoor data.\n\nAdditionally, the DFU [5] model addresses the issue of sparse decoder features in encoder-decoder networks by using a depth feature upsampling method. While this approach seems to aid adaptation in few-shot setups, it appears vulnerable in indoor scenarios with varying sparse depth configurations.\n\nOGNIDC [6] iteratively refines the depth gradient field (depth differences between neighboring pixels) and integrates this information to produce a dense depth map. This method has shown excellent performance in various zero-shot generalization experiments and satisfactory results in few-shot experiments, ranking second-best among comparing methods. Unlike other models, which highly rely on RGB features, OGNIDC focuses on refining 3D-aware information through the Depth Gradient Field, making it less sensitive to RGB appearance changes, as seen in models like CostDCNet [7] which shows superior performance than other methods in the Table.1 of the manuscript. This results in strong generalization and few-shot adaptation capabilities. However, our methodology demonstrates an ability to learn feature representations effectively in hyperbolic space using smaller datasets, leading to faster adaptation in harsh conditions compared to recent SoTA methods.\n\n&nbsp;\n\n### [tAUP] Q.1 Advantages of hyperbolic space for calculation of the pixel affinity map.\nWe appreciate the opportunity to explain why depth completion methods benefit from hyperbolic geometry. Most spatial propagation networks (e.g., CSPN, NLSPN, and DySPN) adopt encoder-decoder structures to extract multi-scale features w.r.t. structure and photometric similarities. Then, initial seeds (i.e., sparse depth) are propagated based on affinity maps computed from the learned features in an iterative manner. Therefore, if the computed affinity map is accurate, capturing boundary information, which is the highly ambiguous region for pixel-wise prediction task, is concomitant. However, object boundary ambiguities, caused by noise or smooth intensity changes, can lead to bleeding errors [8].\n\nTo address these issues, traditional studies (i.e. optimization-based approaches) for affinity construction have utilized hierarchical structures [9-13]. Like tree-based structure, non-local propagation has shown superiority over the local propagation that CSPN, NLSPN, and DySPN follow. By embedding pixel features into hyperbolic space [14,15], we formulate these hierarchical relations in a continuous and differentiable manner. The hyperbolic space naturally accommodates exponentially growing hierarchies and tree-like structures, allowing robust affinity construction with low distortion. This enhances the distinction between unrelated pixel features (i.e., low affinity), while semantically-close pixels benefit from the hierarchical structure, reducing bleeding errors.\n\nWe conducted a toy example to verify the effectiveness of hyperbolic geometry in various propagation schemes, including CSPN (Convolutional), NLSPN (Non-Local), and DySPN (Dynamic attention). Using the same backbone (ResNet-34) and loss functions (L1 and L2) across all schemes ensures a fair comparison. As shown in the Table.D, hyperbolic operations significantly improve performance in various few-shot setups. Compared to Euclidean methods, hyperbolic structures improve pixel distinction under challenging conditions. We also hope you refer to section [g2HF-Q.3] for an ablation study of hyperbolic operations in our architecture.\n\n&nbsp;\n\n### [tAUP] Q.2 Probe for computational costs of depth foundation model\nAs mentioned, depth foundation models are typically large and computationally expensive due to training on extensive datasets. However, recent models offer various variants, allowing flexibility in computational demands. As shown in Table.7 of the manuscript, we conduct ablations on multiple models and observe comparable performance across them. As shown in Table.E, MiDaS[12] and Depth Anything[13] have significantly fewer parameters than other depth completion models, suggesting that leveraging a pre-trained foundation model's knowledge does not necessarily entail high computational costs. The light-weight models can achieve sufficient generalization performance as well. Note that we use the publicly available official codes for MiDaS [16] (v2.1 Small), Depth Anything v1 [17] (ViT-S), and UniDepth [18] (ViT-L). Plus, we will mention it in the revised version of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ql9tzuzZrA", "reviewer": "Reviewer_g2HF", "summary": "The paper introduces a universal depth completion framework that aims to resolve two challenges: \"generalizable knowledge\" of unseen scenes and \"adaptation ability\" to arbitrary depth sensors. \nUnlike the previous method relying on extensive pixel-wise labeled data, the proposed method 1) utilizes a foundation monocular depth estimation model for unseen environments, 2) generates a pixel-wise affinity map for fast adaptation, and 3) utilizes hyperbolic space to build implicit hierarchical structures of 3D data from fewer examples. \nAs a result, the proposed depth completion framework can be generalized to the unseen domain and rapidly adaptable with few samples for diverse depth sensors.\nThe method shows effectiveness and fast adaptation ability on KITTI DC and NYU v2 datasets.", "questions": "Please resolve the issue in the weakness part.\nAdditional questions are as follows: \n\nQ1. [Full sequence training]\nThe paper only demonstrates a few-shot experiment and 1 sequence training results. \nIt would be great to see the proposed method still valid even following a typical training process (e.g., full epoch training, such as 100 epoch training on KITTI DC dataset). \n\nQ2. [Clarity] In ablation study 5.3, \"Probe for Hyperbolic Embedding,\" it would be great to describe what  \"Euclidean\" stands for.\nI guess this means that all operations of the proposed method are done in Euclidean space. But would you please explain the implementation in more detail?\n\nQ3. By following the references [67-70], hyperbolic space is effective in low-shot problems. It would be great to see how much performance difference exists between Ecludiean and hyperbolic spaces in a few-shot setting. Would you provide a Tab 5 experiment in a few shot settings to support the claim it is valid for depth completion tasks as well?\n\nQ4. [Limitatoin] Would you describe in more detail the pros/cons/limitations of hyperbolic geometry in the depth estimation or depth completion tasks?", "limitations": "The paper aims to resolve two challenges: \"generalizable knowledge\" of unseen scenes and \"adaptation ability\" to arbitrary depth sensors. \nThe current paper demonstrates that the method works well in few-shot settings on KITTI and NYU v2 datasets. It can also adapt to common LiDAR and RGB-D sensors. I guess through the rebuttal, the paper is also possible to show its generalization performance in unseen domains.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "[Originality] The utilization of hyperbolic embedding to capture the implicit hierarchical structure of 3D data from a few samples seems novel and effective.\n\n[Relation to prior work] The paper clearly discusses the difference between previous related works.\n\n[Clarity] The limitations of traditional depth completion methods are well described, and the method is clearly motivated by the limitations.\n\n[Significance] The method shows effectiveness and fast adaptation ability on KITTI DC and NYU v2 datasets.", "weaknesses": "W1. [Significance] The paper describes that the method aims to address unseen domain generalization in L8 and introduction. However, the experiments about unseen domain generalization are a bit insufficient and do not effectively show its superiority to claim that the method fully addresses the generalization problem. It would be great to see further extensive unseen domain generalization experiments with other comparison methods to highlight the proposed method's superiority.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [g2HF] W/L. Comparison with other methods and experiments about unseen domain generalization.\nAs requested by the reviewer, we conduct additional experiments on diverse datasets beyond KITTI DC and NYU v2. We include tests on the SUN RGB-D, which show different environmental conditions and various sensor types, e.g., Intel RealSense 3D Camera, Asus Xtion LIVE PRO, Microsoft Kinect V1 and V2. According to Table.B, our method achieves consistent performance improvements for all the datasets, outperforming the state-of-the-art methods in unseen domain scenarios. Plus, we compare with other advanced methods (e.g., LRRU[1], DFU[2], OGNI-DC[3], VPP4DC[4],) to highlight the proposed method's superiority (Please refer the Table.A, **[tAUP] W.2** and **[bXoG] W.3**).\n\n&nbsp;\n\n### [g2HF] Q.1 Full sequence training\nIn response to the request for full sequence training experiments, we carry out an extended evaluation of the KITTI DC dataset. The results, shown in Table.C, indicate that our method not only maintains its effectiveness in a full training regime but also achieves competitive performance over traditional methods that rely on extensive labeled data. This supports our claim that the proposed approach is robust and versatile across different training settings.\n\n&nbsp;\n\n### [g2HF] Q.2 Implementation of hyperbolic operation\nAs you noted, “Euclidean” refers to a space working for conventional operations like convolution. This means that all computations, including embedding and affinity computations, are performed using typical neural network operations in n-dimensional Euclidean space. Starting with the foundational work on Hyperbolic Neural Networks [5,6], there has been a significant shift towards leveraging hyperbolic operations across various downstream tasks. Typically, tensors processed in neural networks are considered within the Euclidean space. However, by embedding these tensors into hyperbolic space, we can exploit the advantages of hyperbolic geometry. This embedding enables us to perform operations directly in hyperbolic space, capturing hierarchical relationships more effectively than in Euclidean space [5,6]. Here is a pseudo-code implementation illustrating the difference between Euclidean and hyperbolic operations in our method:\n## Pseudo-Code\n**Input:** image features $E^M_{L} \\in \\mathbb{R}^{C \\times H \\times W}$, initial sparse depth $S$, curvature $\\kappa$, pixel coordinate $i$, and the neighboring pixel coordinate of $j$.\n\n- $HyperbolicEmbedding$: transform function from hyperbolic to Euclidean space (Eq.4)\n\n- $HyperbolicDistance$ : Distance function with hyperbolic features (Eq.8)\n\n- $HyperbolicConvolution$ : convolution layer with hyperbolic features (Eq.10)\n\n\n**For each** $e_i$ **in** $E_L^M$ **do**\n\n**1:**   $h_i = HyperbolicEmbedding (e_i, \\kappa)$ and $h_j = HyperbolicEmbedding (e_j, \\kappa)$\n\n**2:**  $w_{i,j} = HyperbolicDistance (h_i, h_j)$  \n\n**3:** $d_i  =$ $\\sum_{j}$ $w_{i,j} S_j$\n\n**4:** $a^{hyp}_i = HyperbolicConvolution(e_i, \\kappa)$\n\n**End for**\n\n**Output:** Hyperbolic affinity $a^{hyp}_i \\in A^{hyp}$ and initial depth $d_i \\in D$.\n\n\n&nbsp;\n\n### [g2HF] Q.4 Pros/cons/limitations of hyperbolic geometry for depth perception\nHyperbolic geometry offers substantial advantages in modeling hierarchical relationships within sparse data, providing new cues for understanding 3D structures in depth perception tasks[7,8,9,10]. By embedding data into hyperbolic space, we can more effectively capture complex structures, enabling a richer representation of 3D environments. This approach enhances the performance of depth completion tasks, particularly in few-shot settings, and improves generalization and adaptability in unseen domains. \n\nMost depth completion tasks use encoder-decoder structures to extract multi-scale features and propagate sparse depth information through affinity maps. Hyperbolic space naturally accommodates exponentially growing hierarchies and tree-like structures, allowing robust affinity construction with low distortion. This enhances the distinction between unrelated pixel features and reduces bleeding errors by effectively capturing boundary information, even in ambiguous regions. \n\nHowever, hyperbolic geometry remains certain challenges. One major limitation is the increasing computational complexity, which can impact both training and inference times [11,12,13], even with a similar number of parameters as highlighted in **[tAUP] Q.2**. Additionally, the performance sometimes depends on a hyperparameter setting for curvature value. Despite these challenges, the overall benefits of using hyperbolic geometry, such as improved generalization and performance in diverse and unseen environments, often outweigh these drawbacks, making it a valuable tool for depth estimation and completion tasks.\n\nConsidering these advantages and limitations, a strategy that combines Euclidean and hyperbolic operations could be beneficial as done in Eq.8, and [14,15]. By leveraging the strengths of both geometries, it is possible to achieve a balance between computational efficiency and the ability to capture complex hierarchical relationships in depth perception tasks, which will be mentioned in the revised version of this paper.\n\n### [g2HF] Q.3 Few shot comparisons for Euclidean and hyperbolic space\nAs suggested, we conduct additional experiments under the few-shot regime. The results, presented in Table.F, show a noticeable improvement when using hyperbolic space, with a performance gain of 5% on average, compared to Euclidean space. This validates the effectiveness of hyperbolic geometry in depth completion tasks, especially when dealing with limited data samples.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7tbzkvKm7g", "reviewer": "Reviewer_AS7d", "summary": "This paper proposes Universal Depth Completion as a new task and provides a solution that tackles the problem. The proposed approach utilizes a monocular depth foundation model for its general understanding of 3D scenes and then completes depth information from different sensors with a learned affinity map. The learned features are embedded into a hyperbolic space to build hierarchical 3D structures.", "questions": "Why do the 100-shot results of NYU look so much worse compared to KITTI?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors identified the problem that most depth completion solutions are tailored to specific settings (indoor/outdoor) and sensors, and have trouble generalizing to different configurations. \nThe paper then claims to define a new problem called Universal Depth Completion, which is an even more general setup than the sensor-agnostic depth completion task from the DepthPrompting paper. \n\nThe proposed baseline is novel, it uses a depth foundation model for its generalizable knowledge of 3D scenes and leverages hyperbolic geometry for learning hierarchical features.\n\nThe usage of hyperbolic geometry is a notable contribution. While embedding features into a hyperbolic space have already been proposed in prior works, this work observes the limitation of a fixed and predetermined curvature and proposes to learn a suitable curvature based on the fused features. Furthermore, the pre-processing step (sparse-to-dense conversion) and post-processing step (spatial propagation) are both adapted to hyperbolic space.\n\nThe paper also includes few-shot experiments comparing the proposed method to other SOTA ones. The proposed approach outperforms all other methods in various few-shot setups.", "weaknesses": "The authors mentioned that only KITTI and NYU are predominantly utilized in the depth completion research field, which I agree with. Yet in the paper, they still use only these two datasets. I don't think testing on these two datasets is enough. There are many other RGBD datasets that could be used here to make the claim of generalizability stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [AS7d] W.1 Experiments on other RGBD datasets\nWe appreciate your feedback regarding the limited use of datasets in our evaluation. In response to concerns about unseen domain generalization, we have expanded our experimental evaluation to include an additional dataset, SUN-RGBD, which contains RGB-D images from four different sensors, offering a diverse range of scenes and sensors, e.g., Intel RealSense 3D Camera, Asus Xtion LIVE PRO, Microsoft Kinect V1 and V2. The total of 1000 scenes, where each sequence is roughly 20 seconds long and annotated every 0.5 seconds, is officially split into train/val/test set with 700/150/150 scenes. According to the results of Table.B, our method consistently demonstrates improved performance across these datasets, outperforming state-of-the-art methods in new domain scenarios. These results highlight our framework's strong generalization capabilities and will be included in the revised version of this paper.\n\n&nbsp;\n\n### [AS7d] Q.1 Comparison of qualitative results\nThe difference in qualitative results results observed between the NYU and KITTI datasets can be primarily attributed to the sensor characteristics. ITTI employs a 64-line Velodyne LiDAR, which provides consistent sparse depth information at fixed locations, creating an inductive bias that simplifies optimization. The model can more easily learn from consistent depth cues provided by the LiDAR. In contrast, NYU relies on synthetic random sampling for sparse depth, leading to varying locations of depth seeds across different samples. This variability increases the difficulty of optimization, particularly in few-shot setups, as the model struggles to learn effectively from constantly changing sparse depth locations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H1oEZt8AYe", "reviewer": "Reviewer_bXoG", "summary": "This paper proposes a novel depth completion framework that leverages a large model for monocular depth estimation (i.e., MiDAS, called also Visual Foundation Model -- VFM) to achieve depth completion across a wide range of scenes and sensors (here defined as \"UniDC problem\").\nThe UniDC baseline architecture also exploits hyperbolic geometry to fine-tune the model to the destination domain with a limited amount of labeled data (even only one RGB+Sparse depth+GT frame -- _1-shot learning_ -- and few RGB+Sparse depth frames -- _few-shot learning without dense GT_).\nThe UniDC architecture is mainly compared to another VFM-based depth completion framework [19] and other VFM-less depth completion models showing good performance.", "questions": "Before questions, I resume here the motivations behind my overall rating: the proposed hyperbolic space is beneficial for the few-shot experiments and permits to achieve better results w.r.t. [19]. However, the paper lacks other important experiments -- i.e., the aforementioned (E2) (E3); generalized depth completion competitors in the experiments; fairness with VFM-less proposals.\n\n**Minor comments**: 1) there is a typo at row 274: Eg -> Eq; 2) there is a typo at table 7: 0.13930 -> 1.393; 3) Fig. 1 is a little ambiguous for [19]: the initial relative depth is rescaled using Eq. 4 of paper [19]; 5) I suggest to resume the claims using a list.\n\n## After Reviewer-Authors decision\n\nAfter carefully reading the Authors' rebuttal, I decided to change my score from 3 to 5, based on these two main points:\n\n- **Response to W1**: one of my major concerns was about the fairness with VFM-less methods. Table G demonstrates that even if paired with additional knowledge, VFM-less methods struggle in few-shot experiments. This suggests that the integration of VFM is not trivial.\n\n- **Response to W2**: Authors agree that (P1) is an important piece of the related work, and they will update the literature review and how it compares to the proposal.\n\nI want to highlight that this score change will be confirmed only if the camera-ready paper has all the edits discussed during the Reviewer-Authors period.", "limitations": "The authors addressed almost all limitations. I suggest the authors to investigate also limitations from the introduction of VFM as done in [19].", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "**Good performance of the framework**: the proposal surpasses previous VFM-less works and VFM-based [19] with a large margin in 1-shot, 10-shot, 100-shot, and 1-sequence configurations. The ablation study confirms that the hyperbolic space is useful for this kind of experiment. Even more, the authors tried to fine-tune uniDC in a self-supervised manner showing interesting results w.r.t. networks [19] and [75] trained in the same way.\n\n**Novel solution for a known problem**: depth completion methods that generalize well in other domains and/or using other sensors is a recent emerging problem in the literature [32] [33] [19] (P1). The proposal is a contribution to enlarge the deployment of depth completion solutions to real-world scenarios.\n\n**The reading was smooth**: The authors wrote the paper in a linear and clear way. They expose the problem to the reader and following logical steps they arrive to the proposed solution. The figures are clear and help the reader to understand the proposal and the results. There are some minor imperfections that can be fixed (see question paragraph).\n\n(P1) Bartolomei, L., Poggi, M., Conti, A., Tosi, F., & Mattoccia, S. (2024, March). Revisiting depth completion from a stereo matching perspective for cross-domain generalization. In 2024 International Conference on 3D Vision (3DV) (pp. 1360-1370). IEEE.", "weaknesses": "**The comparison of the proposal with VFM-less methods seems unfair**: VFM features are a powerful source of extra knowledge (Tab. 7) that is not available in other networks. For example in Tab 1, all VFM-less competitors got access to one training data only, while UniDC and [19] have (indirect) access to extra data through VFM extra knowledge. Authors could have included the extra VFM knowledge for example replacing the sparse depth input using the rescaling solution proposed in Eq. 4 of paper [19].\n\n**Related literature is missing**: (P1) (published before [19]) tries to resolve the uniDC problem in a _zero-shot manner_ exploiting a stereo network pre-trained on synthetic data only. They define a novel benchmark (already used in other works (P2)) to evaluate performance on cross-domain generalization (indoor -> outdoor and vice versa) without labeled data of the target domain. In my opinion, this is a huge missing in the related works and experiments chapters.\n\n**Not SOTA performance on 1-shot experiments (Minor Weakness)**: (P1) shows better performance (Tab. 3 of the cited paper) on KITTI and NYU (expect RMSE) even in a zero-shot manner w.r.t. the proposal in 1-shot manner. Those two results are comparable because the proposal leverages MiDAS which is trained using **millions** of frames, while (P1) is trained using **thousands** of frames (SceneFlow). Even so, it is worth noticing that by using more \"shots\" (e.g., 10-shot column) the proposal achieves better results.\n\n**Some experiments and competitors are missing**: this paper lacks three kinds of experiments: E1) full in-domain performance -- i.e., same domain for training and test sets, where the training is fully used;  E2) cross-domain performance -- i.e., a network trained using a dataset and tested in another; E3) varying-density performance -- i.e., for example, to simulate the deployment on different LiDAR setups. Furthermore, related domain/density generalized depth completion solutions are missing from the experiments [32] [33] (P1). I'm aware that code for [32] [33] is unavailable (to compare them in few-shot manner), however (P1) code is available and it reports results also for [32] regarding the aforementioned (E1) (E2) and (E3) experiments.\n\n(P2) Zuo, Y., & Deng, J. (2024). OGNI-DC: Robust Depth Completion with Optimization-Guided Neural Iterations. arXiv preprint arXiv:2406.11711.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [bXoG] W.1 Comparison with VFM-less Methods with\nWe acknowledge the reviewer's concern regarding the fairness of comparing our method with VFM-less approaches. We recognize that Visual Foundation Model (VFM) features provide a significant advantage due to their extensive pre-training on large datasets, which VFM-less methods lack. To address your concern, we carry out additional experiments using VFM knowledge in VFM-less baselines by replacing the sparse depth input with Eq. 4 of the paper [19].\n\nIn this experiment, we found that directly applying the VFM approach, as suggested, sometimes yields unsatisfactory performance compared to the baseline (Table.G). This underperformance can be attributed to optimization issues stemming from the fact that the sparse depth provides complete metric depth information, whereas fitting the relative-scale depth from VFM using Eq. 4 of [19] does not achieve this precision level. The fitting process involves solving ∣Ax−B∣, which performs a linear fit with the available data, i.e., sparse depth. In [19], the authors employed global linear fitting to predict the depth scale using scalar values A and B, initially converting relative depth to metric scale. However, this approach often fits disproportionately to regions with rich information, leading to inaccuracies in areas with sparse depth information. Consequently, using metric sparse depth as input can cause inaccuracies, making optimization difficult and resulting in suboptimal performance.\n\nWe agree that there is a significant gap in using VFM directly for depth completion. Instead of directly using relative-scale depth, we chose to leverage intermediate features to indirectly utilize foundation knowledge. This approach allows us to benefit from VFM while avoiding the direct application of relative-scale depth, thereby mitigating some of the challenges observed in this experiment. \n\n### [bXoG] W.2&4 Related literature, experiments, and competitors are missing\n\nWe appreciate the reviewer's suggestion to include the work of [P1] and [P2], which addresses the UniDC problem in a zero-shot manner. We agree that this is an important piece of the related work, and we will update our literature review and how it compares to our method. Additionally, we will expand our experiments to include their benchmark, as outlined in their work, to better evaluate various sensors with a few labeled data in the other domain. \n\nOur model is not specifically designed for zero-shot generalization; rather, it addresses task-specific domain shifts by integrating relative-scale 3D knowledge learned by the VFM into a sensor fusion model, i.e., depth completion model. Our primary objective is to quickly and effectively adapt to new sensors, leveraging the generalization capability of the VFM model. While cross-domain experiments (source RGBD dataset to target RGBD dataset), as shown by P1 and P2, are important for understanding model generalization, challenges such as catastrophic forgetting or significant covariate shifts (e.g., with new sensors like 4D radar) can render existing models ineffective.\n\nOur solution focuses on fast adaptation with minimal data, enabling strong performance even without dense labeled data, as demonstrated in Table 3 of the paper. Thanks to the reviewer's suggestion, we can showcase the superiority and applicability of our model across various environments by comparing it with advanced methods, demonstrating the feasibility to various sensors and environments.\n\n[Full In-domain Performance]: In response to the request for full in-domain experiments, we carry out an extensive evaluation of the KITTI DC and NYU datasets. To explore our method under various configurations, we develop four variants by adjusting the number of channels, similar to the recent state-of-the-art methods like LRRU and CompletionFormer. As shown in Table.C, the results demonstrate that our method is effective in a full training setting and performs competitively against traditional methods that require extensive labeled data, confirming our approach's robustness and versatility across different training scenarios. Moreover, Table.A shows that our variants achieve competitive performance compared to well-established methods concerning specialization within a single domain. We compare those advanced methods, including their variants (Mini to Base), with ours in few-shot experiments and verify the effectiveness of our proposed method regardless of the number of model parameters.\n\n[Varying-density Performance]: We simulate different LiDAR setups by varying the density of the input data. For the NYU indoor dataset, we randomly sampled 100 and 32 sparse depths, while for the KITTI outdoor dataset, we utilize 16-Line and 4-Line configurations. These experiments test the robustness and adaptability of our method in response to changes in input data quality and quantity. As shown in Table.H, the results show that our method achieves superior performance across different sensor configurations. In contrast, most comparison methods exhibit a decline in performance when adapting to new sensor configurations, as demonstrated in the DepthPrompting [19].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their helpful comments which make our paper polish up. If reviewers want to see additional results and analysis, please let us know it and we are always welcome to discuss.

For additional experiments requested by multiple reviewers, we report them here.

- [Table.A] Comparison of Recent SoTA Methods:  Reviewer **tAUP**, **g2HF** and **bXoG**

- [Table.B] Additional RGB-D Dataset with Various Sensors: Reviewer **g2HF** and **AS7d**

- [Table.C] Full Dataset Training Benchmark: Reviewer **tAUP**, **g2HF** and **bXoG**

For all experimental results for this rebuttal, we have put them all together in the uploaded PDF file. 

&nbsp;

## [Table.A] Experiment on Advanced Methods (RMSE/MAE/DELTA1.25)

|[NYU] | 1-shot                             | 10-shot                            | 100-shot                           |
|:-:|:-:|:-:|:-:|
| LRRU_Mini  | 0.704 / 0.505 / 0.738              | 0.989 / 0.677 / 0.642              | 0.551 / 0.392 / 0.797              |
| LRRU_Tiny  | 0.842 / 0.633 / 0.574              | 0.771 / 0.549 / 0.707              | 0.565 / 0.373 / 0.836              |
| LRRU_Small | 0.589 / 0.388 / 0.826              | 0.404 / 0.246 / 0.919              | 0.442 / 0.306 / 0.887              |
| LRRU_Base  | 0.447 / 0.278 / 0.899              | 0.424 / 0.252 / 0.922              | 0.316 / 0.189 / 0.949              |
| DFU        | 0.754 / 0.589 / 0.637              | 0.590 / 0.464 / 0.719              | 0.467 / 0.344 / 0.868              |
| OGNI-DC    | 0.365 / 0.200 / 0.921              | 0.312 / 0.160 / 0.957              | 0.207 / 0.095 / 0.974              |
| Ours_Tiny  | 0.243 / 0.131 / 0.969              | 0.186 / 0.088 / 0.982              | 0.151 / 0.068 / 0.988              |
| Ours       | **0.210 / 0.108** / **0.975**          | **0.166 / 0.079** / **0.985**          | **0.147 / 0.067** / **0.988**          |
| Ours_Small | 0.255 / 0.136 / 0.968              | 0.181 / 0.089 / 0.983              | 0.149 / 0.067 / 0.988              |
| Ours_Base  | 0.247 / 0.138 / 0.969              | 0.190 / 0.093 / 0.983              | 0.148 / 0.066 / 0.988              |

| [KITTI] | 1-shot                             | 10-shot                            | 100-shot                           |
|:-:|:-:|:-:|:-:|
| LRRU_Mini  | 6.719 / 3.068 / 0.792              | 5.608 / 2.787 / 0.811              | 3.576 / 2.020 / 0.841              |
| LRRU_Tiny  | 7.961 / 4.049 / 0.698              | 6.253 / 3.022 / 0.788              | 4.201 / 2.394 / 0.796              |
| LRRU_Small | 16.162 / 8.008 / 0.516             | 5.930 / 2.905 / 0.800              | 5.934 / 3.602 / 0.612              |
| LRRU_Base  | 14.889 / 7.454 / 0.526             | 13.078 / 6.904 / 0.587             | 9.736 / 6.090 / 0.420              |
| DFU        | 3.652 / 2.020 / 0.853              | 1.889 / 0.966 / 0.973              | 1.808 / 0.897 / 0.986              |
| OGNI-DC    | 2.618 / 0.816 / 0.962              | 1.516 / 0.421 / 0.985              | 1.514 / 0.430 / 0.984              |
| Ours_Tiny  | 2.002 / 0.725 / 0.951              | 1.457 / 0.448 / 0.987              | 1.251 / 0.353 / 0.991              |
| Ours       | **1.684 / 0.522** / **0.983**          | **1.385 / 0.407 / 0.990**          | **1.224 / 0.339** / **0.993**          |
| Ours_Small | 1.865 / 0.590 / 0.975              | 1.465 / 0.436 / 0.988              | 1.283 / 0.388 / 0.991              |
| Ours_Base  | 1.716 / 0.607 / 0.979              | 1.423 / 0.428 / 0.988              | 1.246 / 0.345 / 0.992              |

&nbsp;

## [Table.B] Experiment on SUN RGB-D (RMSE/MAE/DELTA1.25)

| | 1-shot                        | 10-shot                        | 100-shot                       |
|:-:|:-:|:-:|:-:|
| BPNet   | -                     | 0.497 / 0.244 / 0.870          | 0.342 / 0.164 / 0.900          |
| DP      | 0.706 / 0.534 / 0.537         | 0.683 / 0.512 / 0.558          | 0.700 / 0.527 / 0.545          |
| LRRU    | 0.912 / 0.743 / 0.347         | 0.507 / 0.300 / 0.785          | 0.476 / 0.313 / 0.779          |
| DFU     | -                    | 0.890 / 0.696 / 0.314          | 0.552 / 0.444 / 0.438          |
| OGNI-DC | -                     | 0.466 / 0.270 / 0.817          | 0.382 / 0.188 / 0.881          |
| Ours    | **0.529 / 0.285 / 0.830**         | **0.418 / 0.188 / 0.895**          | **0.345 / 0.166 / 0.901**         |


&nbsp;

## [Table.C] Full Dataset Training Benchmark (NYU & KITTI)
| # of Params. (M) | Models            | NYU (RMSE) | NYU (MAE) | KITTI (RMSE) | KITTI (MAE) |
|:--------------------:|:--------------------:|:----------:|:---------:|:------------:|:-----------:|
| 41.5M                      | Cformer_Tiny       | 0.091    | 0.035   | -          | -         |
| 82.6M                      | Cformer_Small  | 0.090    | 0.035   | 0.739      | 0.196     |
| 142.4M                     | Cformer_Base       | 0.090    | 0.035   | 0.709      | 0.203     |
| 0.3M                       | LRRU_Mini          | 0.101    | -       | 0.800      | 0.219     |
| 1.3M                       | LRRU_Tiny          | 0.096    | -       | 0.762      | 0.208     |
| 5.2M                       | LRRU_Small         | 0.093    | -       | 0.741      | 0.202     |
| 21M                        | LRRU_Base          | 0.091    | -       | 0.728      | 0.198     |
| 1.2M                       | Ours_Tiny          | 0.107    | 0.042   | 0.907      | 0.231     |
| 4.6M                       | Ours               | 0.098    | 0.038   | 0.867      | 0.224     |
| 36.9M                      | Ours_Small         | 0.095    | 0.038   | 0.824      | 0.209     |
| 63.2M                      | Ours_Base          | 0.093    | 0.036   | -          | -         |

If the experiment fails to optimize or diverge, they are marked with a "-"

Meta-review: The paper deals with the relevant problem of improving the generalization ability of depth completion across domains and sensors with a few labelled data by exploiting the knowledge provided by a foundation model for monocular depth estimation. The features obtained by the foundation model are embedded into hyperbolic space for depth propagation and refinement. As witnessed by the recent literature, this topic has gained much attention, and the paper proposes an innovative solution that achieves excellent results for this task. Nonetheless, reviewers raised concerns about methods not considered in the initial submission and additional experimental results. Moreover, the reviewers requested clarifications about the motivation and effectiveness of using hyperbolic space. In the rebuttal, the authors clarified most concerns. They provided a significant amount of additional experimental results as requested by reviewers, persuading most of them during the discussion to lean towards acceptance, with reviewers bXoG and g2HF championing the paper and reviewer AS7d setting a final borderline acceptance score.
In contrast, reviewer tAUP considers the performance of the proposed architecture trained in the KITTI domain unsatisfactory compared to the state-of-the-art for in-domain depth completion and proposes rejecting the paper. The AC acknowledges that the proposal is not competitive with most recent works tackling in-domain depth completion, but this task is different from the purpose of this work. On the other hand, the proposed method has excellent performance, outperforming state-of-the-art methods that deal with a few labelled data for training, raising the bar in the field. Nonetheless, the paper needs a significant revision to integrate the additional experimental results and clarifications requested by the reviewers.
Decision: Accept (poster)