# A Sparsity-Invariant Model via Unifying Depth Prediction and Completion
###### Abstract  
The development of a sparse-invariant depth completion model capable of handling varying levels of input depth sparsity is highly desirable in real-world applications. However, existing sparse-invariant models tend to degrade when the input depth points are extremely sparse. In this paper, we propose a new model that combines the advantageous designs of depth completion and monocular depth estimation tasks to achieve sparse invariance. Specifically, we construct a dual-branch architecture with one branch dedicated to depth prediction and the other to depth completion. Additionally, we integrate the multi-scale local planar module in the decoders of both branches. Experimental results on the NYU Depth V2 benchmark and the OPPO prototype dataset equipped with the Spot-iToF316 sensor demonstrate that our model achieves reliable results even in cases with irregularly distributed, limited or absent depth information. sparsity invariant; depth completion; depth prediction +
Footnote †: journal: 2024  
0000-0001-9991-9999-9999-9999-9999-9999-9999-99999-99999-99999-999999999999999999999999999999999999999999999999999999999  
To date, various methods have been proposed for color-guided depth completion tasks, including models with complex network structures [1; 2; 3; 4], models with spatial propagation networks [5; 6; 7; 8; 9; 10], models based on three-dimensional structure design [11; 12; 13; 14] and the latest model based on transformer [15; 16; 17]. However, most of these models do not fully consider the generalization capability of models across different sparse depth maps, making them sensitive to the differences in sparse depth maps. Although some researchers have explored the generalization of models, the number of related studies is still relatively limited. Some of them utilize sparse convolution [18] and sparsity-aware augmentation [19; 20] to handle sparse inputs. Others employ non-learning methods to complete the sparse depth map, which is followed by optimization using color images [21; 22; 23]. Additionally, some utilize depth maps obtained from pre-trained monocular depth estimation networks as depth priors input to the network [19; 24]. While these methods focus on addressing variations in sparsity levels, their performance still deteriorates significantly in cases of extremely sparse depth points and no depth points. It is observed that when the input depth map provides a sufficient number of valid points, the accuracy of the depth map generated by depth completion surpasses that of the monocular depth estimation task. Meanwhile, when the input depth map is extremely sparse or completely missing, the depth completion task turns into a monocular depth estimation problem. In this case, models specifically designed for monocular depth estimation outperform the depth completion models. Inspired by this, we consider adopting the design ideas from the monocular depth estimation task to improve the depth prediction ability of the depth completion model when the depth map is extremely sparse or completely absent. Monocular depth estimation generates a dense depth map from a single color image. Two-dimensional color images, unlike sparse depth maps, cannot directly present three-dimensional information. Therefore, most methods in this field focus on leveraging geometric priors to improve depth prediction results. Scene plane and surface normal vectors are closely related to depth values, so they are often embedded into the model architecture as representations of scene structure information or introduced into the loss function for training. Following the local planar assumption, methods [25; 26; 27; 28; 29; 30] utilize surface normal as an intermediate representation to enforce constraints on local 3D points belonging to the same plane, ensuring that the vectors formed by any two pixels within a plane are perpendicular to the surface normal. [PERSON] et al. [31; 32] introduced the concept of virtual normal, extending the constraint range from a local window to the entire global image.