Title: A Simple yet Universal Framework for Depth Completion
Venue: NeurIPS 2024
Accepted: yes
Keywords: Few-shot Depth Completion

hyperbolic representation

foundation model
Abstract: Consistent depth estimation across diverse scenes and sensors is a crucial challenge in computer vision, especially when deploying machine learning models in the real world. Traditional methods depend heavily on extensive pixel-wise labeled data, which is costly and labor-intensive to acquire, and frequently have difficulty in scale issues on various depth sensors. In response, we define Universal Depth Completion (UniDC) problem. We also present a baseline architecture, a simple yet effective approach tailored to estimate scene depth across a wide range of sensors and environments using minimal labeled data. 
Our approach addresses two primary challenges: generalizable knowledge of unseen scene configurations and strong adaptation to arbitrary depth sensors with various specifications. To enhance versatility in the wild, we utilize a foundation model for monocular depth estimation that provides a comprehensive understanding of 3D structures in scenes. Additionally, for fast adaptation to off-the-shelf sensors, we generate a pixel-wise affinity map based on the knowledge from the foundation model. We then adjust depth information from arbitrary sensors to the monocular depth along with the constructed affinity. Furthermore, to boost up both the adaptability and generality, we embed the learned features into hyperbolic space, which builds implicit hierarchical structures of 3D data from fewer examples. Extensive experiments demonstrate the proposed method's superior generalization capabilities for UniDC problem over state-of-the-art depth completion. Source code is publicly available at https://github.com/JinhwiPark/UniDC.

ASimpleyetUniversal Framework for Depth Completion

Abstract

Consistent depth estimation across diverse scenes and sensors is a crucial challenge in computer vision, especially when deploying machine learning models in the real world. Traditional methods depend heavily on extensive pixel-wise labeled data, which is costly and labor-intensive to acquire, and frequently have difficulty in scale issues on various depth sensors. In response, we define Uni versal D epth C ompletion (UniDC) problem. We also present a baseline architecture, a simple yet effective approach tailored to estimate scene depth across a wide range of sensors and environments using minimal labeled data. Our approach addresses two primary challenges: generalizable knowledge of unseen scene configurations and strong adaptation to arbitrary depth sensors with various specifications. To enhance versatility in the wild, we utilize a foundation model for monocular depth estimation that provides a comprehensive understanding of 3D structures in scenes. Additionally, for fast adaptation to off-the-shelf sensors, we generate a pixel-wise affinity map based on the knowledge from the foundation model. We then adjust depth information from arbitrary sensors to the monocular depth along with the constructed affinity. Furthermore, to boost up both the adaptability and generality, we embed the learned features into hyperbolic space, which builds implicit hierarchical structures of 3D data from fewer examples. Extensive experiments demonstrate the proposed method's superior generalization capabilities for UniDC problem over state-of-the-art depth completion. Source code is publicly available at https://github.com/JinhwiPark/UniDC .

1 Introduction

Acquiring accurate and dense depth maps is crucial for various computer vision tasks such as scene understanding [1, 2, 3, 4], 3D reconstruction [5, 6, 7, 8], and autonomous driving [9, 10, 11]. Traditional methods like dense stereo matching [12, 13, 14] often face challenges of handling occlusion and varying lighting conditions between viewpoints. Additionally, depth maps obtained from active depth sensors [15, 16] like LiDAR and Time-of-Flight cameras typically exhibit low resolutions. As a solution to the above problems, depth completion has been widely studied. The goal of depth completion is to obtain a depth map from a pair of an image and a low-resolution depth map (often sparse depth map) taken by active sensors. The depth completion aims to convert a sparse depth map into a dense depth prediction by propagating it with an image-based affinity map.

Recent advances in learning-based depth perceptions have markedly improved the performance in this domain; however, most approaches are still tailored to specific settings and struggle to generalize to new environments or sensor types. While generalizable knowledge can be achieved by training huge models with large-scale and diverse datasets, acquiring accurate and dense depth information as ground-truth data is prohibitively expensive and time-consuming, which makes such a generalization model for metric scale 3D depth prediction infeasible in practice. Moreover, there exist numerous types of active depth sensors and complex scenarios in the real world. Unfortunately, only two

Hae-Gon Jeon AI Graduate School GIST

benchmark datasets ( e.g. , KITTI [17] and NYU dataset [18]) are predominantly utilized in relevant research fields. Considering the accessibility of various industrial scenarios and the extremely high annotation cost, it is desirable to explore a few-shot learning approach capable of universal depth prediction for both arbitrary sensors and environments.

In response to the growing needs of both industry and the research community, in this work, we define a new problem, called Uni versal D epth C ompletion (UniDC), and present a baseline architecture and its advanced version. Our key insight of the baseline model for UniDC is to utilize pre-trained knowledge from a foundation model for monocular depth estimation, which provides depth-aware information enriched with high-resolution contextual information. Previous works typically exploit entangled representations of an image and corresponding depth data by concatenating them in an input layer, which reduces the generality of the foundation model. A contemporary work [19] proposes a sensor-agnostic depth completion with a depth prompting module, which mitigates the sensor bias problem by disentangling image and depth modalities. Since the depth representation is optimized with respect to a specific scene environment, it has limitations in out-of-domain situations, such as the environmental transition from indoor to outdoor, and vice versa.

To resolve this limitation, we design a simple baseline architecture using the foundation model. By excluding the training procedure for a new encoder to represent depth data, we achieve a high generality of the model across various sensors regardless of scene configurations. The proposed architecture consists of three sequential steps: (1) extraction of depth-aware features from the foundation model; (2) sparse-to-dense conversion based on the depth-aware information; (3) refinement of the converted depth with a pixel-wise affinity map constructed based on high-resolution details of the input image. For more details, the sparse-to-dense conversion aggregates adjacent depth values based on the high-resolution pixel-wise features from the foundation model. In the depth refinement process, we adopt a spatial propagation module with a multi-kernel affinity map.

We next boost up the baseline architecture by taking advantage of hyperbolic embedding. As stated in [20, 21], the natural capacity of hyperbolic spaces encourages capturing the implicit hierarchical structure of 3D data. In particular, this capability alleviates bleeding errors in the spatial propagation process [22]. To ensure adaptability and generality, we also design a multi-curvature approach for producing multiple affinity maps in the refinement stage. The effectiveness of our models is demonstrated across a variety of scenarios and datasets, confirming its superior generalization and robustness in different sensor setups and scene configurations. We also conduct extensive experiments and analyses to validate the efficacy of the proposed model.

2 Related Works

Depth Completion. Image-guided depth completion aims to predict dense depth maps from an RGB image and its synchronized sparse depth acquired by depth sensors. A work in [23] introduces a deep regression model that significantly enhances prediction accuracy over the existing monocular depth estimation method [24], which utilizes only RGB image as input. However, depth maps from the direct regression method often suffer from blurry artifacts and distortions at object boundaries [25]. To address these issues, several works have introduced spatial propagation networks (SPNs) [25, 26, 27, 28, 29] as refinement modules. SPNs iteratively update the output of direct-regression methods by aggregating neighboring pixels over a reference pixel. Nonetheless, these models are typically tailored for specific depth sensors, such as the 64-Line Velodyne LiDAR [30] in KITTI outdoor dataset [17] and Kinect [31] for NYUv2 indoor dataset [18].

To alleviate this limited usage of SPNs, several studies have explored sensor-/domain-agnostic depth completion. SpAgNet [32] develops a model agnostic to the sparsity of depth points by incorporating sparse depth representions into a depth decoder. Another work [33] takes the use of both sparse metric depth and data-driven priors from a monocular depth prediction network for domain-agnostic depth completion. DepthPrompting [19] solves sensor bias problems with a prompt engineering. Despite these efforts, they still face challenges with a cross-domain generalization [19] and an issue on a limitation of sensors' scan ranges, which causes an overfitting problem [32, 33].

Usage of Foundation Model in Downstream Task. Foundation models, designed for various downstream tasks, have revolutionized both natural language processing and computer vision fields. In particular, in the computer vision field, these foundation models excel in high-level visual perception tasks such as image recognition [34, 35, 36] and image captioning [35, 37, 36]. Those

vision foundation models provide benefits for strong adaptation to various tasks via tuning methods [38, 39, 40, 41] and feature adaptation methods [42, 43, 44]. In low-level tasks like depth computation, several works [45, 46, 47] create diverse datasets for zero-shot generalization capabilities, while others [48, 49, 50] fine-tune the text-to-image model [51] to utilize diffusion priors for better generalization which guides them to keep geometric details.

Hyperbolic Geometry for Visual Data. Hyperbolic embedding for efficient learning-based approaches [52, 53] has gained interest. The Hyperbolic embedding has validated its ability to effectively represent complex data as hierarchical structures in low-dimensional spaces, offering a distinct advantage over Euclidean embeddings. This unique capability promotes the design of hyperbolic neural networks, and is applicable for a range of applications such as hierarchical recognition [54, 55, 56], retrieval [57, 58, 59], dealing with uncertainty [60, 61, 62], and generative learning on scarce data [63, 64, 65, 66]. Especially, hyperbolic methods have been shown to be effective in addressing low-shot visual problems [67, 68, 69, 60, 70], modeling complex 3D data [20, 21] and measuring pixel-wise similarity [22]. In this work, we devise the hyperbolic version of the proposed architecture to make both the generalizable power and understanding 3D depth data better.

3 Baseline Architecture

Hyperbolic

Sparse-to-Dense

Conversion

Hyperbolic

Affinity

Initial Depth

RGB& Depth

Dense Depth

Depth

Foundation

Model

(b) DepthPrompting (CVPR24)

RGB& Depth

Prompting

Euclidean

Affinity

Dense Depth

Fully

relative

scale

Initial Depth

Depth

Foundation

Model

Encoder

Decoder

(a) Conventional SPNs

Euclidean

Affinity

Initial Depth

RGB&Depth

Dense Depth

(c) Ours

Wepresent a simple yet effective architecture to achieve a generalizable depth completion model for unseen environments with only minimal data. Firstly, in Sec.3.1, we explain the rationale for adopting a monocular depth foundation model to simultaneously achieve sensor/domain-agnostic depth completion. We then propose a baseline model architecture for UniDC, which integrates the pre-trained foundation model with both the depth propagation and refinement process in Sec.3.2.

3.1 Rationale: Foundation Model Usage in UniDC

Difficulties to generalize depth completion. The two major obstacles to sensor-/domain-agnostic depth completion are the high cost of dense depth data acquisition and the scale variance across different sensors. First, capturing dense depth data on a metric scale is expensive. For example, Velodyne 64-line LIDAR, used in the KITTI dataset, provides high-quality depth information but has less than 6% density relative to the number of pixels in its synchronized image. Second, sensors have their own scanning ranges, hindering the development of a universal solution. As shown in Fig. 1-(a,b), the previous frameworks learn the joint representation of image and depth, and the depth prompting module, respectively. However, the trained encoder is vulnerable to handling different sensors due to a bias towards specific scanning ranges.

Usage of depth-aware knowledge from depth foundation model. Although the depth foundation model produces relative depth maps, we can measure pixel-wise similarity using them. For example, we are able to distinguish between foreground and background regions only with the relative depth maps and account for depth boundaries between objects in scenes. Therefore, based on this depthaware information, it will be the most probable solution that propagates a given sparse metric depth into the remaining pixels in an input image space without any additional learning for the depth.

Revisiting how to use SPN. SPN [71] constitutes a core component in most state-of-the-art (SoTA) depth completion and is typically invoked as a final refinement step. The SPN refinement module takes initial depth and pixel-wise affinity as input and yields refined dense depth by iteratively updating its output. During training, the previous methods (Fig.1-(a,b)) jointly optimize the pixel-wise affinity and initial depth. However, the joint optimization scheme hinders the fast adaptation to new environments because learned weights are asked to have both domain- and depth-specific features. Furthermore,

DepthPrompting (Fig.1-(b)), which employs a depth foundation model for a relative-scale depth map as initial depth of SPN, struggles to adapt to new environments with a limited data. We want to eliminate the possibility of degeneracy, so we devise a sparse-to-dense conversion with a foundation model to make an initial dense depth. In Fig.1, different from the coarse initial depth seen in traditional SPNs, our method provides promising results even before the SPN refinement step.

3.2 Architecture Design

Considering the facts discussed in Sec.3.1, we devise an effective baseline architecture. We first utilize pre-trained knowledge from a foundation model tailored for monocular depth estimation, which provides pixel-wise relative distances (a.k.a. relative scene depth) from a camera along with high-resolution contextual information. Thanks to the knowledge, our baseline architecture becomes simpler due to no need for an additional encoder to represent depth data from arbitrary sensors. Our model operates in three stages: 1 extraction of the relative depth-aware features from the foundation model, 2 propagation of spare depth from arbitrary sensors based on the depth-aware features, and 3 refinement of it with a pixel-wise affinity map constructed from the depth-aware features. This scheme not only simplifies the architectural complexity, but also enhances the adaptability and performance across diverse sensing scenarios. The overall algorithm scheme is summarized in Alg.1.

Tuning strategy for foundation model. Given a single image I ∈ R 3 × H × W , the pre-trained depth model f F outputs multi-scale intermediate features E and relative depth D relative as below:

where Θ f F denotes parameters of the foundation model.

Since the foundation model is trained to estimate relative depth from single images, they inherently face limitations when handling metric scale depths. To reduce the modality discrepancy, our approach involves an integration of an additional loss term to refine the foundation model by minimizing the difference between D relative and its Ground Truth (GT) depth D gt for valid pixels v ∈ V . Let δ v = log D relative ( v ) -log D gt ( v ) , the loss L scale-invariant is defined as below:

where we set λ = 0 . 85 in all experiments as in [24]. We also implement a bias tuning [38, 39], shown to be more effective for dense prediction tasks than other tuning protocols [72, 38]. The bias tuning updates the bias terms while keeping the rest of the backbone parameters unchanged, thus preserving the high-resolution details and contextual information. These strategic modifications significantly enhance the capability of the foundation model for estimating metric scale depth.

4 Advanced Architecture with Hyperbolic Geometry

We also present an advanced version of the baseline architecture that grafts hyperbolic geometry onto the depth foundation model, known for its effectiveness in low-shot problems [67, 68, 69, 60, 70]. We first generate depth-aware features by merging the multi-scale intermediate features E derived from the foundation model and by embedding them into hyperbolic space with geometry-aware curvature (Sec.4.1). Using the depth-aware features alongside sparse sensor data, we develop a hyperbolic propagation inspired by a traditional bilateral filter mechanism, which yields an initial dense depth at a metric scale (Sec.4.2). We lastly introduce a process for generating multi-curvature hyperbolic space for high-fidelity pixel relations and refinement of the initial depth (Sec.4.3).

4.1 Multi-scale Feature Fusion & Hyperbolic Curvature Generation

The intermediate features from the foundation model E l ∈ E , where l = 0 , . . . , L -1 , correspond to scales factors 1/2, . . . , 1 / 2 L of the original resolution of input images. We aim to synergistically fuse the multi-scale information to learn comprehensive, context-aware features that facilitate depth propagation at a metric scale. We upsample the coarser feature map E l M ( E M 0 = E 0 ) using convolution layers, and then aggregate E l M with finer feature map E l +1 to obtain better visual contextual features

Algorithm 1 Implementation of Hyperbolic Universal Depth Completion

Require: Given a single image I ∈ R 3 × H × W , depth foundation model f F and the corresponding parameter Θ f F , multi-scale feature aggregation blocks f fusion l , number of multi-scale feature L , curvature generation blocks C , set of neighboring pixel coordinate N ( i ) , kernel function P , and multi-kernel affinity map A k .

E l M +1 . This fusion process is described below:

where f fusion l indicates multi-scale feature aggregation blocks consisting of 2D transposed convolution layers with a skip connection.

Hyperbolic embedding. To ensure strong adaption to both new environments and any type of sensors, we adopt hyperbolic geometry which enables to capture the inherent hierarchical structures of 3D data [20, 21]. To embed the Euclidean features into hyperbolic space and vice versa, one first needs to define a bijective mapping from R n to D n κ . The exponential and the logarithmic mapping are used as bijective functions that have appealing forms at an origin, namely for x ∈ R n and u ∈ D n κ :

Using hyperbolic geometry for pixel-wise relationships, especially spatial propagation, is demonstrated in [22] by improving the discriminative power with minimal supervision. Following [22], we embed the mixed feature E M L into hyperbolic space using Eq.4 as below:

where i is an index of spatial coordinates in the image domain, and κ is the hyperbolic curvature.

Hyperbolic curvature generation. Using an appropriate curvature value is an important factor in projecting Euclidean features into hyperbolic space well, which is closely related to the construction of the hierarchy structures. Previous methods mainly use a fixed geometric structure regardless of data types and scene configurations by merely adjusting κ as a hyperparameter [70, 73, 74, 21]. In our problem definition, according to types of sensors and scene configurations, diverse data measurements and geometrical structures are observed, respectively. That's, our key observation is that a fixed and predetermined curvature may not be universally suitable.

We thus propose a curvature generation that learns a geometry-aware curved embedding space to adaptively match it to new environments and sensors. The curvature generator C is composed of a convolution layer, a multi-layer perceptron (MLP) layer, and a global mean-pooling over spatial dimensions, which yields scene-dependent curvatures based on the fused feature E M L as below:

4.2 Sparse-to-Dense Conversion based on Hyperbolic Features

With both the high-resolution pixel-wise features from the foundation model and the sparse depth data from arbitrary sensors, we perform a sparse-to-dense conversion to obtain an initial dense depth map. Inspired by [75], we design an initial propagation process based on a bilateral filtering mechanism [76], which is renowned for its edge-preserving ability by incorporating both radiometric differences and spatial distances into the bilateral weight. Considering a pixel x i and the corresponding neighborhood pixel x j , the bilateral kernel filter w ij can be simply defined as:

where f r is a range kernel for radiometric differences. g s is a spatial kernel for physical separations in observed scenes and is developed in Euclidean space by calculating the distance between 3D points. For the range kernel f r , we need to design its hyperbolic version. Here, we utilize the hyperbolic feature H provided as input from Eq.5. With f r and g s , we can compute the initial dense depth as:

where Dist hyp is the hyperbolic function consisting of hyperbolic MLP, and Dist euc is the Euclidean distance in the 3-dimension coordinate. N ( i ) means the neighborhood sparse depth of the pixel i , and S j is the corresponding depth from a sensor. P indicates the learnable MLP layer to compute a coefficient for each sparse depth of the neighborhood S j . Through the combination of the distance functions in Eq.8, we effectively take advantage of both hyperbolic and Euclidean geometries to produce more accurate and robust depth maps.

4.3 Depth Refinement in Multi-curvature Hyperbolic Space

Depth refinement. To refine the initial depth in Eq.8, we employ a convolutional spatial propagation scheme, CSPN++ [26]. This refinement process leverages a predefined depth map D i , augmented by a sparse valid depth map S , and a multi-kernel affinity map with three different kernel sizes K = { 3 , 5 , 7 } . The use of a multi-kernel approach enables the model to capture a diverse range of features from the input data, thus achieving detailed and comprehensive depth estimations. The propagation process for a kernel size k ∈ K at step t to yield a dense map ˆ D is formulated as:

where D t is the depth map at each propagation step t . D 0 and A are an initial depth for t = 0 and its affinity map, respectively. ⊙ is an element-wise product, and j ∈ N k ( i ) denotes a set of neighboring pixels around pixel i within a k × k window. σ is a confidence map computed from E M L in Sec.4.1.

Hyperbolic convolution layer (HCL). We design the multi-kernel affinity map A k in hyperbolic space with the proposed curvature generation module described in Sec.4.1. To do this, we formulate HCL with hyperbolic feature vector h for a 2-dimensional image domain:

where W ∈ R C out × C in × γ × γ is a convolution weight matrix whose kernel size is γ , and b is a bias term. Ω= { ( i, j ) ∈ Z 2 | ( -γ ′ , -γ ′ ) , ..., ( γ ′ , γ ′ ) , γ ′ = ⌊ γ 2 ⌋} is a set of signed distances from a center of the convolution kernel to others in W . ⊗ , ⊕ and T β are hyperbolic multiplication, addition, and concatenation, respectively, whose details are in Appendix A.1. Note that the hyperbolic MLP (Eq.8) is designed with γ = 1 .

Mutli-curvature affinity generation. By dynamically adjusting the hyperbolic curvature κ for each affinity map, our approach tailors the geometrical representation to better fit the specific depth structure of each scene. We first determine the hyperbolic curvature κ with Eq.6 and then compute affinity map A k using a hyperbolic convolution operation equipped with a kernel of size k , chosen to match the receptive field of the corresponding kernel of the affinity map A k . This alignment optimizes the local receptive fields across the depth map, enabling a more precise aggregation of context and texture information from neighboring pixels. We can calculate the hyperbolic affinity map A hyp k based on the generated curvature κ k from the curvature generation blocks C k as below:

We can achieve the refined dense depth based on the generated hyperbolic affinity maps A hyp k by incorporating it into Eq.9. In particular, the employment of hyperbolic space is beneficial for depth perception by implicitly building hierarchical structures [22], whose roots come from sparse points of an input depth in this work. The hyperbolic space is also advantageous in regions where photometric distances between foreground

RGB Image

GT

Multi-curvature

Fixed-curvature

𝜿𝜿

=

0.1

𝜿𝜿

= {

𝟎𝟎

.

𝟒𝟒𝟒𝟒

,

𝟎𝟎

.

𝟒𝟒𝟒𝟒

,

𝟎𝟎

.

𝟒𝟒𝟏𝟏

}

𝜿𝜿

= {

𝟎𝟎

.

𝟓𝟓𝟓𝟓

,

𝟎𝟎

.

𝟐𝟐𝟒𝟒

,

𝟎𝟎

.

𝟒𝟒𝟒𝟒

}

𝜿𝜿

=

0.1

and background pixels are marginal (see Fig.2). The detailed analysis of multi-curvature hyperbolic affinity is described in Sec. 5.3.

5 Experiment and Analysis

In this section, we evaluate the performance of our proposed method for UniDC, focusing on its adaptability using minimal labeled data. Firstly, we outline an overview of the experimental setup (Sec.5.1). Subsequent comparisons with various SoTA methods are then presented using standard benchmark datasets (Sec.5.2). Furthermore, we conduct an ablation study to clarify the impact of each component in our methodology (Sec.5.3). In Appendix.A.2, we introduce details of the training procedure, datasets, and evaluation metrics in this work. Additional experiments, including full dataset training benchmarks, hyperbolic space affinity calculations, an ablation study on foundation models, and varying-density performance, are included in the Appendix A.3.

5.1 Implementation Details

Loss functions. We train our method in a supervised manner with a linear combination of two loss terms: scale-invariant loss [77] L scale-invariant (Eg.2) for bridging the gap between relative and metric scale depths, and a composite loss L L1L2 based on L 1 and L 2 distances for inferring the final dense depth map. In total, our framework is optimized by minimizing the final loss L as below:

where µ is a balance term and is empirically set to 0.1.

Evaluation protocols. For fair evaluations, we select a diverse array of SoTA depth from sparse measurements. These include a sensor-agnostic model, DepthPrompting [19] and series of SPNs such as S2D [23], CSPN [25], NLSPN [78], DySPN [27], CostDCNet [79], CompletionFormer [80], and BPNet [75]. We assess depth quality using common quantitative metrics: root mean square error (RMSE, in meters), mean absolute error (MAE, in meters), and inlier ratio (DELTA1, where δ < 1 . 25 ). We employ the widely-used depth completion datasets: NYU [18] and KITTI DC [81], setting up a minimal training dataset for few-shot scenarios. Note that we use their official test sets for all the comparison methods.

We implement the few-shot scenarios with and without dense depth supervision. Our experimental setup includes conducting 1-shot, 10-shot, and 100-shot learning by randomly sampling within the official training split. Additionally, we perform 1-sequence training by randomly selecting one sequence from the training set. To ensure the reliability in our experiments, we randomly select 10 sequences, and report averaged results.

5.2 Experiment

Few-shot learning with dense GT. Both Tab.1 and Tab.3 show that existing methods face significant challenges when taking input depths from new sensors with minimal labeled data, whose examples are displayed in Fig.3 and Fig.4, respectively. In the 1-shot scenarios, where a model is optimized using

1-Shot

10-Shot

100-Shot

1-Sequence Training

RMSE

MAE

DELTA1

RMSE

MAE

DELTA1

RMSE

MAE

DELTA1

RMSE

MAE

DELTA1

CSPN [25]

1.4827

1.2058

0.3455

0.3166

0.1961

0.7106

0.2854

0.1307

0.9748

0.3166

0.1961

0.7106

NLSPN [78]

1.9358

1.6132

0.2229

1.5995

0.8261

0.5040

0.5501

0.4150

0.7985

0.8881

0.6421

0.6809

DySPN [27]

1.5474

1.2851

0.3149

0.4102

0.2817

0.8595

0.2674

0.1706

0.9341

0.2584

0.1320

0.9615

CompletionFormer [80]

1.8218

1.5539

0.2408

1.1583

1.0162

0.3079

0.9914

0.8164

0.4379

0.6779

0.5356

0.7476

CostDCNet [79]

1.2298

0.9754

0.4693

0.2363

0.1288

0.9719

0.1770

0.0836

0.9826

0.2066

0.0954

0.9788

BPNet [75]

0.3573

0.2077

0.9482

0.2392

0.1120

0.9744

0.1757

0.0793

0.9829

0.2220

0.1040

0.9765

DepthPrompting [19]

0.3583

0.2067

0.9101

0.2195

0.1006

0.9733

0.2101

0.1008

0.9743

0.2335

0.1191

0.9686

Ours

0.2099

0.1075

0.9752

0.1657

0.0794

0.9849

0.1473

0.0669

0.9885

0.1632

0.0745

0.9860

Ours

NLSPN

DySPN

DP

BPNet

CFormer

CSPN

CostDC

1-shot

100-shot

RGBImage

& Depth

Ground

Truth

only a single pair of an image and its corresponding dense depth, our model demonstrates a substantial performance advantage over the comparison models. This underscores the effectiveness of using the foundation model that does not require any additional learning for new depth representations of unseen data. The models with a large number of parameters to learn, such as CompletionFormer [80] (83.6M), often struggle to optimize with limited datasets. Since the depth prompting module in [19] requires training from scratch, it encounters difficulties in the adaptation to new sensors.

Few-shot learning without dense GT. Training without dense GT depths is a more practical scenario because obtaining high-quality and metric-scale depth is difficult, particularly in outdoor datasets. To validate the applicability, we train our model in a self-supervised manner without a dense GT depth. Specifically, the input LiDAR is sampled at 8-line and 32-line, while the supervision is provided by 64-Line LiDAR. This approach enables our model to adapt to sparser LiDAR inputs without the need for dense supervision. As shown in Tab.4, these results highlight our model's robustness and superior adaptation capabilities over BPNet [76] and DepthPromtping [19], which are the 2nd/3rd best in Tab.1 and Tab.3,

5.3 Ablation Study

Probe for hyperbolic embedding. We assess the efficacy of hyperbolic embedding and curvature generation, focusing on their performance in zero-shot settings. In Tab.2, the hyperbolic method yields promising results, whereas the Euclidean approach fails. The performance gap implies that hyperbolic space offers discriminative features which guide the sparse depth propagation well. While the influence of initial parameter settings cannot be overlooked, the potential for rapid adaptation can be enhanced through well-devised initialization methods, which are in line with principles from meta-learning strategies [82, 83]. we conduct additional experiments under the few-shot regime. The results, presented in Table.F, show a noticeable improvement when using hyperbolic space, with a performance gain of 5% on average, compared to Euclidean space. This validates the effectiveness of hyperbolic geometry in depth completion tasks, especially when dealing with limited data samples.

Additionally, the analysis of the multi-curvature approach for the refinement process (Tab.5) reveals that the curvature values for multi-size affinity maps in-

1-Shot

10-Shot

100-Shot

1-Sequence Training

RMSE

MAE

DELTA1

RMSE

MAE

DELTA1

RMSE

MAE

DELTA1

RMSE

MAE

DELTA1

CSPN [25]

9.2621

3.5736

0.9063

2.0061

0.7962

0.9758

1.4668

0.5018

0.9850

2.6406

0.8227

0.9679

S2D [23]

8.8701

5.6307

0.4222

5.0228

3.1807

0.6319

4.2582

2.6475

0.7030

4.8136

2.5358

0.8383

NLSPN [78]

7.3135

4.7084

0.5036

4.0327

2.2361

0.8662

2.4801

1.1862

0.9348

4.0535

1.7707

0.8787

DySPN [27]

2.6094

0.9082

0.9545

2.2863

0.8920

0.9487

1.8568

0.6437

0.9777

2.8369

0.8149

0.9692

CompletionFormer [80]

4.6990

2.4002

0.8224

3.1760

1.4930

0.9302

2.6263

1.3504

0.8993

4.5320

1.9842

0.8214

BPNet [75]

5.3724

1.0988

0.9690

1.8965

0.5317

0.9822

1.3126

0.3734

0.9915

2.1554

0.6241

0.9816

DepthPrompting [19]

2.9561

1.1657

0.9270

2.4129

1.1463

0.8943

1.7982

0.6021

0.9808

2.9616

0.9655

0.9587

Ours

1.6840

0.5217

0.9826

1.3850

0.4073

0.9903

1.2238

0.3386

0.9927

1.8378

0.5406

0.9824

Ours

DySPN

Ours

DySPN

DepthPrompting

NLSPN

DepthPrompting

NLSPN

BPNet

S2D

BPNet

S2D

CompletionFormer

CSPN

CompletionFormer

CSPN

100-shot

10-shot

crease with the kernel size. This trend suggests that information from more distant regions tends to prefer lower curvature. This observation supports the hypothesis that regions closer to the target require a more distinct hyperbolic space to effectively prevent bleeding errors [22]. This insight emphasizes the importance of strategic curvature adaptation for better universal depth completion.

Component ablation study. We conduct an ablation study on each component of our model as shown in Tab.6. The results reveal that removing feature fusion significantly reduces performance, particularly in 1-shot scenarios. On the other hand, the hyperbolic method shows strong adaptability with minimal data. Moreover, the fine-tuning strategy applied to the foun- dation model seems to be essential for the adaption to new environments, considering inherent discrepancies between relative and metric depth.

Foundation model variations. To evaluate the versatility of our method with various foundational models, we replace our primary backbone [46] with concurrent works, DepthAnything [47] and UniDepth [84], which are foundation models for relative and metric depth estimation, respectively. Since these models are based on vision transformer (ViT) [86], differing from the convolutional version of MiDaS, we compare them without the intermediate feature fusion (Eq.3). As shown in Tab.7, while these backbones exhibit comparable performance, MiDaS [46] is more suitable for 10-shot and 100-shot scenarios. We claim that the local inductive bias of convolutions operates more flexibly in depth completion tasks, effectively propagating local information. This observation aligns

with the most state-of-the-art methods using convolutional encoder-decoder architectures with SPN refinement, as opposed to ViT-based architectures [47, 87, 84] for depth foundation models.

Probe for computational costs of depth foundation model. Depth foundation models are typically large and computationally expensive due to training on extensive datasets. However, recent models offer various variants, allowing flexibility in computational demands. We conduct ablations on multiple models and observe comparable performance across them. As shown in Tab.8, MiDaS [46] and Depth Anything [47] have significantly fewer parameters than other depth completion models, suggesting that leveraging a pre-trained foundation model's knowledge does not necessarily entail high computational costs. Note that we use the publicly available official codes for MiDaS (v2.1 Small), Depth Anything v1 (ViT-S), and UniDepth (ViT-L).

Additional experiments on recent SoTA methods and other sensor. In Tab.9, we compare our approach with recent SoTA methods, showing its advantages across different experimental setups. Unlike the LRRU family [85], which performs variably across datasets due to the IP-Basic algorithm's KITTI dataset bias, our model leverages foundation model knowledge for consistent adaptation to both indoor and outdoor environments. DFU [88] and OGNIDC [89] introduce depth feature upsampling and gradient refinement, respectively. Our method, however, efficiently learns hyperbolic representations on smaller datasets, enabling faster adaptation in challenging conditions. Additionally, we evaluate our model on SUN-RGBD as shown in Tab.10, containing diverse RGB-D images from multiple sensors (Intel RealSense, Asus Xtion, Kinect V1/V2), with consistent improvements across these sensors.

6 Conclusion

This work starts from the new problem definition, Universal Depth Completion, to tackle the challenge of consistent depth estimation across diverse scenes and sensors. We propose a simple yet universally applicable framework that leverages the knowledge of the depth foundational model and few-shot learning capabilities using hyperbolic geometry. Through various experiments in few-/zero-shot scenarios, we validate the adaptability and generality of our method.

Limitation & Future work. There are rooms for improvement. In this work, we can only use a pair of an image and corresponding sparse depth as input. For general full 3D reconstruction and novel view synthesis, our method is needed to handle input pairs with multiple viewpoints. In addition, the direct application to another modality like radar is challenging due to the noisy and highly sparse nature of the radar-derived depth information. For this, we have to devise a method to estimate an uncertainty on the noisy measurements, which will be one of interesting future works.

Acknowledgement. This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT)(RS-2024-00338439), the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-01842, Artificial Intelligence Graduate School Program (GIST), RS-2021-II212068, Artificial Intelligence Innovation Hub), GIST-MIT Research Collaboration grant funded by the GIST in 2024, 'Project for Science and Technology Opens the Future of the Region' program through the INNOPOLIS FOUNDATION funded by Ministry of Science and ICT (Project Number: 2022-DD-UP-0312), and Local Finance Association(LOFA) grant funded by the Korea government(A Study on the Complete Survey of Advertisements Using Artificial Intelligence (AI) Technology).

References

Maximilian Jaritz, Raoul De Charette, Emilie Wirbel, Xavier Perrotton, and Fawzi Nashashibi. Sparse and dense data with cnns: Depth completion and semantic segmentation. In 2018 International Conference on 3D Vision (3DV) , pages 52-60. IEEE, 2018. 1

Ce Liu, Suryansh Kumar, Shuhang Gu, Radu Timofte, and Luc Van Gool. Single image depth prediction made better: A multivariate gaussian take. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 1

Shuwei Shao, Zhongcai Pei, Weihai Chen, Xingming Wu, and Zhengguo Li. Nddepth: Normal-distance assisted monocular depth estimation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 1

Zhiqiang Yan, Kun Wang, Xiang Li, Zhenyu Zhang, Guangyu Li, Jun Li, and Jian Yang. Learning complementary correlations for depth super-resolution with incomplete data in real world. IEEE transactions on neural networks and learning systems , 2022. 1

Fangchang Ma, Guilherme Venturelli Cavalheiro, and Sertac Karaman. Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA) , 2019. 1

Kyeongha Rho, Jinsung Ha, and Youngjung Kim. Guideformer: Transformers for image guided depth completion. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 1

Zhiqiang Yan, Xiang Li, Kun Wang, Zhenyu Zhang, Jun Li, and Jian Yang. Multi-modal masked pre-training for monocular panoramic depth completion. In Proceedings of European Conference on Computer Vision (ECCV) , pages 378-395. Springer, 2022. 1

Zhiqiang Yan, Xiang Li, Kun Wang, Shuo Chen, Jun Li, and Jian Yang. Distortion and uncertainty aware loss for panoramic depth completion. In Proceedings of the International Conference on Machine Learning (ICML) , 2023. 1

Kun Wang, Zhenyu Zhang, Zhiqiang Yan, Xiang Li, Baobei Xu, Jun Li, and Jian Yang. Regularizing nighttime weirdness: Efficient self-supervised monocular depth estimation in the dark. In Proceedings of International Conference on Computer Vision (ICCV) , 2021. 1

Zhiqiang Yan, Kun Wang, Xiang Li, Zhenyu Zhang, Jun Li, and Jian Yang. Desnet: Decomposed scale-consistent network for unsupervised depth completion. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2023. 1

Zhiqiang Yan, Yupeng Zheng, Kun Wang, Xiang Li, Zhenyu Zhang, Shuo Chen, Jun Li, and Jian Yang. Learnable differencing center for nighttime depth perception. arXiv preprint arXiv:2306.14538 , 2023. 1

Andreas Geiger, Martin Roser, and Raquel Urtasun. Efficient large-scale stereo matching. In Asian conference on computer vision , pages 25-38. Springer, 2010. 1

Jia-Ren Chang and Yong-Sheng Chen. Pyramid stereo matching network. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. 1

Heiko Hirschmuller and Daniel Scharstein. Evaluation of cost functions for stereo matching. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2007. 1

Alexandre Lopes, Roberto Souza, and Helio Pedrini. A survey on rgb-d datasets. Computer Vision and Image Understanding , 222:103489, 2022. 1

Michael Firman. Rgbd datasets: Past, present and future. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW) , 2016. 1

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research (IJRR) , 32(11):1231-1237, 2013. 2, 17

Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Proceedings of European Conference on Computer Vision (ECCV) , 2012. 2, 7

Jin-Hwi Park, Chanhwi Jeong, Junoh Lee, and Hae-Gon Jeon. Depth prompting for sensor-agnostic depth estimation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 2, 3, 7, 8, 9, 18, 19, 20

Joy Hsu, Jeffrey Gu, Gong Wu, Wah Chiu, and Serena Yeung. Capturing implicit hierarchical structure in 3d biomedical images with self-supervised hyperbolic representations. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2021. 2, 3, 5

Antonio Montanaro, Diego Valsesia, and Enrico Magli. Rethinking the compositionality of point clouds through regularization in the hyperbolic space. 35:33741-33753, 2022. 2, 3, 5

Jin-Hwi Park, Jaesung Choe, Inhwan Bae, and Hae-Gon Jeon. Learning affinity with hyperbolic representation for spatial propagation. In Proceedings of the International Conference on Machine Learning (ICML) , 2023. 2, 3, 5, 7, 9, 19

Fangchang Ma and Sertac Karaman. Sparse-to-dense: Depth prediction from sparse depth samples and a single image. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA) , 2018. 2, 7, 9

Zhenyu Li, Zehui Chen, Xianming Liu, and Junjun Jiang. Depthformer: Exploiting long-range correlation and local information for accurate monocular depth estimation. Machine Intelligence Research , pages 1-18, 2023. 2, 4

Xinjing Cheng, Peng Wang, and Ruigang Yang. Depth estimation via affinity learned with convolutional spatial propagation network. In Proceedings of European Conference on Computer Vision (ECCV) , 2018. 2, 7, 8, 9, 19

Xinjing Cheng, Peng Wang, Chenye Guan, and Ruigang Yang. Cspn++: Learning context and resource aware convolutional spatial propagation networks for depth completion. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2020. 2, 6

Yuankai Lin, Tao Cheng, Qi Zhong, Wending Zhou, and Hua Yang. Dynamic spatial propagation network for depth completion. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2022. 2, 7, 8, 9, 19

Mu Hu, Shuling Wang, Bin Li, Shiyu Ning, Li Fan, and Xiaojin Gong. Penet: Towards precise and efficient image guided depth completion. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA) , 2021. 2

Xin Liu, Xiaofei Shao, Bo Wang, Yali Li, and Shengjin Wang. Graphcspn: Geometry-aware depth completion via dynamic gcns. In Proceedings of European Conference on Computer Vision (ECCV) , 2022. 2

Brent Schwarz. Mapping the world in 3d. Nature Photonics , 4(7):429-430, 2010. 2

Zhengyou Zhang. Microsoft kinect sensor and its effect. IEEE multimedia , 19(2):4-10, 2012. 2

Andrea Conti, Matteo Poggi, and Stefano Mattoccia. Sparsity agnostic depth completion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 5871-5880, 2023. 2, 18

Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Simon Chen, and Chunhua Shen. Towards domain-agnostic depth completion. arXiv preprint arXiv:2207.14466 , 2022. 2

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the International Conference on Machine Learning (ICML) , 2021. 2

Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 , 2021. 2

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023. 2

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikoł aj Bi´ nkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2022. 2

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Proceedings of European Conference on Computer Vision (ECCV) , 2022. 3, 4

Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2020. 3, 4

Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274 , 2022. 3

Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge-guided context optimization. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 3

Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2022. 3

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision , 132(2):581-595, 2024. 3

Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In Proceedings of European Conference on Computer Vision (ECCV) , pages 696-712. Springer, 2022. 3

Jaime Spencer, Chris Russell, Simon Hadfield, and Richard Bowden. Kick back & relax: Learning to reconstruct the world by watching slowtv. In Proceedings of International Conference on Computer Vision (ICCV) , 2023. 3

Reiner Birkl, Diana Wofk, and Matthias Müller. Midas v3.1 - a model zoo for robust monocular relative depth estimation. arXiv preprint arXiv:2307.14460 , 2023. 3, 9, 10, 17

Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 3, 9, 10

Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In Proceedings of International Conference on Computer Vision (ICCV) . 3

Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. arXiv preprint arXiv:2312.02145 , 2023. 3

Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. 2024. 3

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 3

Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2017. 3

Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the Lorentz model of hyperbolic geometry. In Proceedings of the International Conference on Machine Learning (ICML) , 2018. 3

Ankit Dhall, Anastasia Makarova, Octavian Ganea, Dario Pavllo, Michael Greeff, and Andreas Krause. Hierarchical image classification using entailment cone embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW) , 2020. 3

Mina Ghadimi Atigh, Martin Keller-Ressel, and Pascal Mettes. Hyperbolic busemann learning with ideal prototypes. In Proceedings of the Neural Information Processing Systems (NeurIPS) , volume 34, pages 103-115, 2021. 3

Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong-Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. Hyperbolic visual embedding learning for zero-shot recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020. 3

Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedantam. Hyperbolic image-text representations. In Proceedings of the International Conference on Machine Learning (ICML) , 2023. 3

Teng Long, Pascal Mettes, Heng Tao Shen, and Cees G. M. Snoek. Searching for actions on the hyperbole. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020. 3

Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan Oseledets. Hyperbolic vision transformers: Combining improvements in metric learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 3

Mina GhadimiAtigh, Julian Schoep, Erman Acar, Nanne van Noord, and Pascal Mettes. Hyperbolic image segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 3, 4

Dídac Surís, Ruoshi Liu, and Carl Vondrick. Learning the predictability of the future. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 3

Luca Franco, Paolo Mandica, Bharti Munjal, and Fabio Galasso. Hyperbolic self-paced learning for self-supervised skeleton-based action representations. arXiv preprint arXiv:2303.06242 , 2023. 3

Joy Hsu, Jeffrey Gu, Gong Wu, Wah Chiu, and Serena Yeung. Capturing implicit hierarchical structure in 3d biomedical images with self-supervised hyperbolic representations. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2021. 3

Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton. Latent variable modelling with hyperbolic normalizing flows. In Proceedings of the International Conference on Machine Learning (ICML) , 2020. 3

Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh. Continuous hierarchical representations with poincaré variational auto-encoders. In Proceedings of the Neural Information Processing Systems (NeurIPS) , volume 32, 2019. 3

Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped normal distribution on hyperbolic space for gradient-based learning. In Proceedings of the International Conference on Machine Learning (ICML) , 2019. 3

Lingxiao Li, Yi Zhang, and Shuhui Wang. The euclidean space is evil: hyperbolic attribute editing for few-shot image generation. In Proceedings of International Conference on Computer Vision (ICCV) , 2023. 3, 4

Zhi Gao, Yuwei Wu, Yunde Jia, and Mehrtash Harandi. Curvature generation in curved spaces for few-shot learning. In Proceedings of International Conference on Computer Vision (ICCV) , 2021. 3, 4

Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020. 3, 4

Nurendra Choudhary, Nikhil Rao, and Chandan Reddy. Hyperbolic graph neural networks at scale: A meta learning approach. In Proceedings of the Neural Information Processing Systems (NeurIPS) , volume 36, 2024. 3, 4, 5

Sifei Liu, Shalini De Mello, Jinwei Gu, Guangyu Zhong, Ming-Hsuan Yang, and Jan Kautz. Learning affinity via spatial propagation networks. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2017. 3

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Proceedings of the International Conference on Machine Learning (ICML) , 2019. 4

Shu-Lin Xu, Yifan Sun, Faen Zhang, Anqi Xu, Xiu-Shen Wei, and Yi Yang. Hyperbolic space with hierarchical margin boosts fine-grained learning from coarse labels. In Proceedings of the Neural Information Processing Systems (NeurIPS) , volume 36, 2024. 5

Zhi Gao, Yuwei Wu, Yunde Jia, and Mehrtash Harandi. Hyperbolic feature augmentation via distribution estimation and infinite sampling on manifolds. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2022. 5

Jie Tang, Fei-Peng Tian, Boshi An, Jian Li, and Ping Tan. Bilateral propagation network for depth completion. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 6, 7, 8, 9

Carlo Tomasi and Roberto Manduchi. Bilateral filtering for gray and color images. In Proceedings of International Conference on Computer Vision (ICCV) , 1998. 6, 8

David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2014. 7

Jinsun Park, Kyungdon Joo, Zhe Hu, Chi-Kuei Liu, and In So Kweon. Non-local spatial propagation network for depth completion. In Proceedings of European Conference on Computer Vision (ECCV) , 2020. 7, 8, 9, 19

Jaewon Kam, Jungeon Kim, Soongjin Kim, Jaesik Park, and Seungyong Lee. Costdcnet: Cost volume based depth completion for a single rgb-d image. In Proceedings of European Conference on Computer Vision (ECCV) , 2022. 7, 8

Youmin Zhang, Xianda Guo, Matteo Poggi, Zheng Zhu, Guan Huang, and Stefano Mattoccia. Completionformer: Depth completion with convolutions and vision transformers. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 7, 8, 9, 18

Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant cnns. In International Conference on 3D Vision (3DV) , 2017. 7, 18

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning (ICML) , 2017. 8

Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2018. 8

Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 9, 10

Yufei Wang, Bo Li, Ge Zhang, Qi Liu, Tao Gao, and Yuchao Dai. Lrru: Long-short range recurrent updating networks for depth completion. In Proceedings of International Conference on Computer Vision (ICCV) , 2023. 9, 10, 18

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of International Conference on Learning Representations (ICLR) , 2021. 9

René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of International Conference on Computer Vision (ICCV) , 2021. 10

Yufei Wang, Ge Zhang, Shaoqian Wang, Bo Li, Qi Liu, Le Hui, and Yuchao Dai. Improving depth completion via depth feature upsampling. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 10

Yiming Zuo and Jia Deng. Ogni-dc: Robust depth completion with optimization-guided neural iterations. arXiv preprint arXiv:2406.11711 , 2024. 10

Abraham Albert Ungar. A gyrovector space approach to hyperbolic geometry. In Synthesis Lectures on Mathematics and Statistics , 2008. 17

Octavian Ganea, Gary Becigneul, and Thomas Hofmann. Hyperbolic neural networks. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2018. 17

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Proceedings of the Neural Information Processing Systems Workshop , 2017. 17

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 17

David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2014. 17

Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant cnns. In International Conference on 3D Vision (3DV) , 2017. 17

Jinyoung Jun, Jae-Han Lee, and Chang-Su Kim. Masked spatial propagation network for sparsity-adaptive depth refinement. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 18

Jinhyung Park, Yu-Jhe Li, and Kris Kitani. Flexible depth completion for sparse and varying point densities. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 18

Haotian Wang, Meng Yang, and Nanning Zheng. G2-monodepth: A general framework of generalized depth inference from monocular rgb+ x data. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023. 18

Haotian Wang, Meng Yang, Xinhu Zheng, and Gang Hua. Scale propagation network for generalizable depth completion. arXiv preprint arXiv:2410.18408 , 2024. 18

Vadim Ezhov, Hyoungseob Park, Zhaoyang Zhang, Rishi Upadhyay, Howard Zhang, Chethan Chinder Chandrappa, Achuta Kadambi, Yunhao Ba, Julie Dorsey, and Alex Wong. All-day depth completion. arXiv preprint arXiv:2405.17315 , 2024. 18

Yangchao Wu, Tian Yu Liu, Hyoungseob Park, Stefano Soatto, Dong Lao, and Alex Wong. Augundo: Scaling up augmentations for monocular depth completion and estimation. In Proceedings of European Conference on Computer Vision (ECCV) , 2024. 18

Suchisrit Gangopadhyay, Xien Chen, Michael Chu, Patrick Rim, Hyoungseob Park, and Alex Wong. Uncle: Unsupervised continual learning of depth completion. arXiv preprint arXiv:2410.18074 , 2024. 18

Hyoungseob Park, Anjali Gupta, and Alex Wong. Test-time adaptation for depth completion. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 18

Tian Yu Liu, Parth Agrawal, Allison Chen, Byung-Woo Hong, and Alex Wong. Monitored distillation for positive congruent depth completion. In Proceedings of European Conference on Computer Vision (ECCV) , 2022. 18

Alex Wong and Stefano Soatto. Unsupervised depth completion with calibrated backprojection layers. In Proceedings of International Conference on Computer Vision (ICCV) , 2021. 18

A Appendix / supplemental material

A.1 Hyperbolic Geometry

Revisit to Poincaré ball model. We revisit some definitions of a hyperbolic ball model and the details of the fundamental arithmetic operations in the hyperbolic space. The Poincaré ball model ( D n κ , g κ ) with curvature κ is defined by a manifold D n κ = { x ∈ R n | κ ∥ x ∥ < 1 } equipped with a metric g κ , where ∥ · ∥ denotes the Euclidean norm. In contrast to traditional vector spaces, hyperbolic spaces require distinct approaches for mathematical operations. Therefore, we employ the framework of Möbius gyrovector spaces, a generalization of Euclidean vector spaces adapted for hyperbolic models. Based on Möbius transformation [90], there are fundamental arithmetic operations in the hyperbolic space, such as addition ( ⊕ κ ) and multiplication ( ⊗ κ ). Furthermore, we exploit bijective mapping functions ( exp κ 0 and log κ 0 ) between hyperbolic space and Euclidean space.

Möbius addition. For a pair ( u , v ) ∈ D n κ , the Möbius addition is defined as follows:

where ⟨· , ·⟩ is the Euclidean inner product.

Möbius matrix-vector multiplication. For an arbitrary function f : R n → R m in the Euclidean space, the Möbius version of f is a function that maps from D n to D m in the hyperbolic space using Equation 4. Similarly, we can derive the Möbius matrix-vector multiplication between the matrix M and input u , which is defined as:

Hyperbolic concatenation. Given image feature maps F in Euclidean space, we pixel-wisely embed an image feature vector at a pixel ( x, y ) ( i.e. , f ( x,y ) ∈ R C × 1 × 1 ) into the hyperbolic space. Here, we utilize an exponential mapping M ( · )=exp κ 0 ( · ) on the Poincaré ball D C κ as a bijective function between the Euclidean space and the hyperbolic space via Poincaré curvature κ . For concatenating features in hyperbolic space, we apply the β -concatenation proposed in [91] as below:

The points x i in the Poincare ball D n i κ are projected back v i = M -1 ( x i ) with the scalar coefficient β n = B ( n 2 , 1 n ) , where B is the Beta distribution.

A.2 Experiment Details

Training details. We utilize MiDaS [46] as a depth foundation model whose pre-trained knowledge is transferred into our universal model. Our model is implemented with public PyTorch [92], trained on a single RTX 3090Ti GPU using Adam [93] optimizer. All training is conducted in a few-shot manner, with the number of iterations ranging from 100 to 3,000, depending on the size of the training dataset, e.g. , 1-shot, 10-shot, and 100-shot. Note that we resize input RGB images to keep the ratio of height/width toward MiDaS. The initial learning rate was set to 5 × 10 -3 and reduced by 0.1 every 20% for total iterations. The proposed framework comprises 4.6M learnable parameters, including 41K dedicated to tuning the foundational model. To facilitate fair comparison, each experiment is repeated 10 times with the same seeds ( e.g. , 0 to 9), and we report the average test accuracy.

Evaluation metrics. We introduce a depth quality evaluation metrics, proposed in [94, 17, 95]. We compare the competitive depth completion model and ours using official evaluation metrics: RMSE, MAE, and δ 1 1 . 25 . Given a ground truth depth D = { d } and the predicted depth ˆ D = { ˆ d } , the metrics are as follows:

Root mean squared error (RMSE): √ 1 | D | ∑ ˆ d ∈ ˆ D | ˆ d -d | 2

Mean absolute error (MAE): 1 | D | ∑ ˆ d ∈ ˆ D | ˆ d -d |

Percentage of predicted pixels where the relative error is within a threshold ( δ i 1 . 25 ):

where the card is the cardinality of a set. Note that a higher δ i indicates better prediction.

Depth completion datsets: NYUv2 and KITTIDC. We employ the NYU Depth V2 dataset, which consists of 464 indoor scenes captured using a Kinect sensor. Adhering to the established train/test division, we evaluate our trained model on 215 scenes (654 samples). The NYU Depth V2 dataset offers images at 320×240 resolution. We utilize center-cropped images at 304×228 resolution and randomly select 500 points to emulate sparse depth data. For 1-sequence training setup, we choose 10 sequences from the training dataset: [conference_room_0001, study_room_0004, reception_room_0002, play-room_0006, living_room_0068, kitchen_0010, classroom_0016, bedroom_0041, bathroom_0041, basement_0001b]

For outdoor environments, we utilize the KITTI DC dataset, which comprises 90K samples. Each sample includes color images and corresponding sparse depth data, captured at approximately 6% density relative to image resolution using a Velodyne HDL-64E LiDAR sensor. The images are provided at a resolution of 1216×352. The dataset is segmented into training (86K samples), validation (7K samples), and testing (1K samples) portions. Ground truth (GT) is generated by accumulating multiple LiDAR frames and removing inaccuracies, resulting in enhanced LiDAR depths of about 20% density. For 1-sequent training setup, we utilize the following 10 sequences: [2011_09_26_drive_0001_sync, 2011_09_26_drive_0017_sync, 2011_09_26_drive_0035_sync, 2011_09_26_drive_0093_sync, 2011_09_26_drive_0106_sync, 2011_09_28_drive_0034_sync, 2011_09_28_drive_0094_sync, 2011_09_28_drive_0168_sync, 2011_09_29_drive_0004_sync, , 2011_09_30_drive_0034_sync]

A.3 Additional Experiments

Full dataset training benchmark on NYU and KITTI dataset. We report the performance of our work in the KITTI benchmark [81], which is reported in Tab.11. To analyze our method under various configurations, similar to recent SoTAs such as LRRU [85] and CompletionFormer [80], we designed four variants by adjusting the number of channels. Notably, both our method and these methods totally follow the scaling laws of deep learning models. Our variants achieve competitive results compared to the SoT methods, especially in setups with fewer labels.

We emphasize that, over the past decade, numerous depth completion papers have focused on indomain experiments on NYU and KITTI datasets. However, there has been a recent trend towards addressing out-of-domain challenges in depth completion research [32, 96, 19, 97, 98, 99, 100, 101, 102, 103, 104, 105]. This direction aims to develop models that can handle variations in new sensor configurations, unseen environmental conditions, and training schemes without the need for dense GT. This trend is gaining attraction in top-tier conferences and journals, highlighting the importance of adaptability and generalization in depth completion models. Our research aligns with this direction and shares similar goals. Note that most of those works do not consider the KITTI benchmark, which is an in-domain experiment with a 64-Line LiDAR sensor. While we agree that top-tier papers should demonstrate a certain level of performance, we also believe that research focusing on generalization and adaptability for arbitrary sensors and environments is valuable and deserves recognition.

Advantages of hyperbolic space for calculation of the pixel affinity map. Most spatial propagation networks (e.g., CSPN [25], NLSPN [78], and DySPN [27]) adopt encoder-decoder structures to extract multi-scale features w.r.t. structure and photometric similarities. Then, initial seeds (i.e., sparse depth) are propagated based on affinity maps computed from the learned features in an iterative manner. Therefore, if the computed affinity map is accurate, capturing boundary information, which is the highly ambiguous region for pixel-wise prediction task, is concomitant. However, object boundary ambiguities, caused by noise or smooth intensity changes, can lead to bleeding errors [22]. To address these issues, we formulate these hierarchical relations in a continuous and differentiable manner. The hyperbolic space naturally accommodates exponentially growing hierarchies and tree-like structures, allowing robust affinity construction with low distortion.

We conducted a toy example to verify the effectiveness of hyperbolic geometry in various propagation schemes, including CSPN (Convolutional), NLSPN (Non-Local), and DySPN (Dynamic attention). Using the same backbone (ResNet-34) and loss functions (L1 and L2) across all schemes ensures a fair comparison. As shown in Tab.12, hyperbolic operations significantly improve performance in various few-shot setups. Compared to Euclidean methods, hyperbolic structures improve pixel distinction under challenging conditions.

Ablation study on VFM (Visual Foundation Model). We carry out additional experiments using VFM knowledge in conventional methods by replacing the sparse depth input with Eq.4 of the paper [19]. In this experiment, we found that directly applying the VFM approach, as suggested, sometimes yields unsatisfactory performance compared to the baseline, as shown in Tab.13. This underperformance can be attributed to optimization issues stemming from the fact that the sparse depth provides complete metric depth information, whereas fitting the relative-scale depth from VFM using Eq.4 of DepthPrompting [19] does not achieve this precision level. The fitting process involves solving AxB, which performs a linear fit with the available data, i.e., sparse depth. In [19], the authors employed global linear fitting to predict the depth scale using scalar values A and B, initially converting relative depth to metric scale. However, this approach often fits disproportionately to regions with rich information, leading to inaccuracies in areas with sparse depth information. Consequently, using metric sparse depth as input can cause inaccuracies, making optimization difficult and resulting in suboptimal performance. We agree that there is a significant gap in using VFM directly for depth completion. Instead of directly using relative-scale depth, we chose to leverage intermediate features to indirectly utilize foundation knowledge. This approach allows us to benefit from VFM while avoiding the direct application of relative-scale depth, thereby mitigating some of the challenges observed in this experiment.

Varying-density Performance. We simulate different LiDAR setups by varying the density of the input data. For the NYU indoor dataset, we randomly sampled 100 and 32 sparse depths, while for the KITTI outdoor dataset, we utilize 16-Line and 4-Line configurations. These experiments test the robustness and adaptability of our method in response to changes in input data quality and quantity. As shown in Tab.14, the results show that our method achieves superior performance across different sensor configurations. In contrast, most comparison methods exhibit a decline in performance when adapting to new sensor configurations, as demonstrated in the DepthPrompting [19].

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have described our contributions in both abstract and introduction sections. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have written our limitations in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have sincerely mentioned the assumption and proof of our work like input modality and technical supports, respectively.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We make the detailed description that readers can reproduce our method easily.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will make our code publicly available, along with links to download the datasets used.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In section3, 4, 5 and appendix, we have specified all the training/test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have included tables and examples to Statistically support technical significance of our method.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have mentioned information on our computing resource to implement and test our method and comparions works.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the guideline of NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is about the enhancement of commercial sensors, which is not related to the societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We think that this work has no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have only used public datasets which have been widely utilized in research fields, and mentioned it in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any new asset.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing and research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work is not related to IRB.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel depth completion framework that leverages a large model for monocular depth estimation (i.e., MiDAS, called also Visual Foundation Model -- VFM) to achieve depth completion across a wide range of scenes and sensors (here defined as \"UniDC problem\").\nThe UniDC baseline architecture also exploits hyperbolic geometry to fine-tune the model to the destination domain with a limited amount of labeled data (even only one RGB+Sparse depth+GT frame -- _1-shot learning_ -- and few RGB+Sparse depth frames -- _few-shot learning without dense GT_).\nThe UniDC architecture is mainly compared to another VFM-based depth completion framework [19] and other VFM-less depth completion models showing good performance.\n\nSTRENGTHS:\n**Good performance of the framework**: the proposal surpasses previous VFM-less works and VFM-based [19] with a large margin in 1-shot, 10-shot, 100-shot, and 1-sequence configurations. The ablation study confirms that the hyperbolic space is useful for this kind of experiment. Even more, the authors tried to fine-tune uniDC in a self-supervised manner showing interesting results w.r.t. networks [19] and [75] trained in the same way.\n\n**Novel solution for a known problem**: depth completion methods that generalize well in other domains and/or using other sensors is a recent emerging problem in the literature [32] [33] [19] (P1). The proposal is a contribution to enlarge the deployment of depth completion solutions to real-world scenarios.\n\n**The reading was smooth**: The authors wrote the paper in a linear and clear way. They expose the problem to the reader and following logical steps they arrive to the proposed solution. The figures are clear and help the reader to understand the proposal and the results. There are some minor imperfections that can be fixed (see question paragraph).\n\n(P1) Bartolomei, L., Poggi, M., Conti, A., Tosi, F., & Mattoccia, S. (2024, March). Revisiting depth completion from a stereo matching perspective for cross-domain generalization. In 2024 International Conference on 3D Vision (3DV) (pp. 1360-1370). IEEE.\n\nWEAKNESSES:\n**The comparison of the proposal with VFM-less methods seems unfair**: VFM features are a powerful source of extra knowledge (Tab. 7) that is not available in other networks. For example in Tab 1, all VFM-less competitors got access to one training data only, while UniDC and [19] have (indirect) access to extra data through VFM extra knowledge. Authors could have included the extra VFM knowledge for example replacing the sparse depth input using the rescaling solution proposed in Eq. 4 of paper [19].\n\n**Related literature is missing**: (P1) (published before [19]) tries to resolve the uniDC problem in a _zero-shot manner_ exploiting a stereo network pre-trained on synthetic data only. They define a novel benchmark (already used in other works (P2)) to evaluate performance on cross-domain generalization (indoor -> outdoor and vice versa) without labeled data of the target domain. In my opinion, this is a huge missing in the related works and experiments chapters.\n\n**Not SOTA performance on 1-shot experiments (Minor Weakness)**: (P1) shows better performance (Tab. 3 of the cited paper) on KITTI and NYU (expect RMSE) even in a zero-shot manner w.r.t. the proposal in 1-shot manner. Those two results are comparable because the proposal leverages MiDAS which is trained using **millions** of frames, while (P1) is trained using **thousands** of frames (SceneFlow). Even so, it is worth noticing that by using more \"shots\" (e.g., 10-shot column) the proposal achieves better results.\n\n**Some experiments and competitors are missing**: this paper lacks three kinds of experiments: E1) full in-domain performance -- i.e., same domain for training and test sets, where the training is fully used;  E2) cross-domain performance -- i.e., a network trained using a dataset and tested in another; E3) varying-density performance -- i.e., for example, to simulate the deployment on different LiDAR setups. Furthermore, related domain/density generalized depth completion solutions are missing from the experiments [32] [33] (P1). I'm aware that code for [32] [33] is unavailable (to compare them in few-shot manner), however (P1) code is available and it reports results also for [32] regarding the aforementioned (E1) (E2) and (E3) experiments.\n\n(P2) Zuo, Y., & Deng, J. (2024). OGNI-DC: Robust Depth Completion with Optimization-Guided Neural Iterations. arXiv preprint arXiv:2406.11711.\n\nQUESTIONS:\nBefore questions, I resume here the motivations behind my overall rating: the proposed hyperbolic space is beneficial for the few-shot experiments and permits to achieve better results w.r.t. [19]. However, the paper lacks other important experiments -- i.e., the aforementioned (E2) (E3); generalized depth completion competitors in the experiments; fairness with VFM-less proposals.\n\n**Minor comments**: 1) there is a typo at row 274: Eg -> Eq; 2) there is a typo at table 7: 0.13930 -> 1.393; 3) Fig. 1 is a little ambiguous for [19]: the initial relative depth is rescaled using Eq. 4 of paper [19]; 5) I suggest to resume the claims using a list.\n\n## After Reviewer-Authors decision\n\nAfter carefully reading the Authors' rebuttal, I decided to change my score from 3 to 5, based on these two main points:\n\n- **Response to W1**: one of my major concerns was about the fairness with VFM-less methods. Table G demonstrates that even if paired with additional knowledge, VFM-less methods struggle in few-shot experiments. This suggests that the integration of VFM is not trivial.\n\n- **Response to W2**: Authors agree that (P1) is an important piece of the related work, and they will update the literature review and how it compares to the proposal.\n\nI want to highlight that this score change will be confirmed only if the camera-ready paper has all the edits discussed during the Reviewer-Authors period.\n\nLIMITATIONS:\nThe authors addressed almost all limitations. I suggest the authors to investigate also limitations from the introduction of VFM as done in [19].", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Universal Depth Completion as a new task and provides a solution that tackles the problem. The proposed approach utilizes a monocular depth foundation model for its general understanding of 3D scenes and then completes depth information from different sensors with a learned affinity map. The learned features are embedded into a hyperbolic space to build hierarchical 3D structures.\n\nSTRENGTHS:\nThe authors identified the problem that most depth completion solutions are tailored to specific settings (indoor/outdoor) and sensors, and have trouble generalizing to different configurations. \nThe paper then claims to define a new problem called Universal Depth Completion, which is an even more general setup than the sensor-agnostic depth completion task from the DepthPrompting paper. \n\nThe proposed baseline is novel, it uses a depth foundation model for its generalizable knowledge of 3D scenes and leverages hyperbolic geometry for learning hierarchical features.\n\nThe usage of hyperbolic geometry is a notable contribution. While embedding features into a hyperbolic space have already been proposed in prior works, this work observes the limitation of a fixed and predetermined curvature and proposes to learn a suitable curvature based on the fused features. Furthermore, the pre-processing step (sparse-to-dense conversion) and post-processing step (spatial propagation) are both adapted to hyperbolic space.\n\nThe paper also includes few-shot experiments comparing the proposed method to other SOTA ones. The proposed approach outperforms all other methods in various few-shot setups.\n\nWEAKNESSES:\nThe authors mentioned that only KITTI and NYU are predominantly utilized in the depth completion research field, which I agree with. Yet in the paper, they still use only these two datasets. I don't think testing on these two datasets is enough. There are many other RGBD datasets that could be used here to make the claim of generalizability stronger.\n\nQUESTIONS:\nWhy do the 100-shot results of NYU look so much worse compared to KITTI?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a universal depth completion framework that aims to resolve two challenges: \"generalizable knowledge\" of unseen scenes and \"adaptation ability\" to arbitrary depth sensors. \nUnlike the previous method relying on extensive pixel-wise labeled data, the proposed method 1) utilizes a foundation monocular depth estimation model for unseen environments, 2) generates a pixel-wise affinity map for fast adaptation, and 3) utilizes hyperbolic space to build implicit hierarchical structures of 3D data from fewer examples. \nAs a result, the proposed depth completion framework can be generalized to the unseen domain and rapidly adaptable with few samples for diverse depth sensors.\nThe method shows effectiveness and fast adaptation ability on KITTI DC and NYU v2 datasets.\n\nSTRENGTHS:\n[Originality] The utilization of hyperbolic embedding to capture the implicit hierarchical structure of 3D data from a few samples seems novel and effective.\n\n[Relation to prior work] The paper clearly discusses the difference between previous related works.\n\n[Clarity] The limitations of traditional depth completion methods are well described, and the method is clearly motivated by the limitations.\n\n[Significance] The method shows effectiveness and fast adaptation ability on KITTI DC and NYU v2 datasets.\n\nWEAKNESSES:\nW1. [Significance] The paper describes that the method aims to address unseen domain generalization in L8 and introduction. However, the experiments about unseen domain generalization are a bit insufficient and do not effectively show its superiority to claim that the method fully addresses the generalization problem. It would be great to see further extensive unseen domain generalization experiments with other comparison methods to highlight the proposed method's superiority.\n\nQUESTIONS:\nPlease resolve the issue in the weakness part.\nAdditional questions are as follows: \n\nQ1. [Full sequence training]\nThe paper only demonstrates a few-shot experiment and 1 sequence training results. \nIt would be great to see the proposed method still valid even following a typical training process (e.g., full epoch training, such as 100 epoch training on KITTI DC dataset). \n\nQ2. [Clarity] In ablation study 5.3, \"Probe for Hyperbolic Embedding,\" it would be great to describe what  \"Euclidean\" stands for.\nI guess this means that all operations of the proposed method are done in Euclidean space. But would you please explain the implementation in more detail?\n\nQ3. By following the references [67-70], hyperbolic space is effective in low-shot problems. It would be great to see how much performance difference exists between Ecludiean and hyperbolic spaces in a few-shot setting. Would you provide a Tab 5 experiment in a few shot settings to support the claim it is valid for depth completion tasks as well?\n\nQ4. [Limitatoin] Would you describe in more detail the pros/cons/limitations of hyperbolic geometry in the depth estimation or depth completion tasks?\n\nLIMITATIONS:\nThe paper aims to resolve two challenges: \"generalizable knowledge\" of unseen scenes and \"adaptation ability\" to arbitrary depth sensors. \nThe current paper demonstrates that the method works well in few-shot settings on KITTI and NYU v2 datasets. It can also adapt to common LiDAR and RGB-D sensors. I guess through the rebuttal, the paper is also possible to show its generalization performance in unseen domains.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a universal depth completion method (UniDC) to address generalization issues in unknown scenes and arbitrary depth sensors. The method utilizes depth information extracted from a pre-trained monocular depth estimation model to generate pixel-wise affinity maps, which adjust sparse depth. Additionally, this paper develops a hyperbolic propagation method for generating and refining dense depth.\n\nSTRENGTHS:\n1.This paper introduces a pre-trained depth estimation model for depth completion tasks and proposes a universal depth completion framework that effectively addresses depth completion challenges across various scenes and different depth sensors.\n\n2. The proposed method performs 1-Shot, 10-Shot, and 100-Shot experiments on the NYU and KITTI DC datasets, demonstrating that UniDC achieves outstanding performance and generalization ability.\n\nWEAKNESSES:\n1. The proposed method does not provide comparative results on the online KITTI depth completion benchmark, and these results need to be further presented.\n\n2. Some advanced and state-of-the-art depth completion methods need to be discussed and compared.\n\n[1] Improving Depth Completion via Depth Feature Upsampling. CVPR 2024.\n\n[2] Tri-Perspective View Decomposition for Geometry-Aware Depth Completion. CVPR2024.\n\n[3] LRRU: Long-short Range Recurrent Updating Networks for Depth Completion.  ICCV 2023.\n\n[4] BEVDC: Bird's-Eye View Assisted Training for Depth Completion. CVPR 2023.\n\n[5] RigNet: Repetitive Image Guided Network for Depth Completion. ECCV 2022.\n\nQUESTIONS:\nPlease refer to weaknesses. Moreover，\n\n1. Compared to previous spatial propagation networks (SPN, CSPN, DySPN, etc.), what advantages does the calculation of the affinity map A_{j,k} in hyperbolic space proposed by Equation (11), and why can it better refine depth maps?\n\n2. When incorporating a pre-trained depth estimation model into the network, it results in additional computational costs, particularly for large models like DepthAnything. How do you mitigate this issue?\n\nLIMITATIONS:\nThe limitations are discussed in detail in the paper. Estimating the uncertainty of noise will better enhance the model's generalization capability.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}