# A Sparsity-Invariant Model via Unifying Depth Prediction and Completion
###### Abstract
GeDepth [33] also approaches the problem from the perspective of a global scene structure. It introduces a novel ground-embedding module to decouple camera parameters from pictorial cues, thereby enhancing generalization. There are also some methods [30; 34] that directly predict plane coefficients for local patches to guide the final depth prediction based on the conversion relationship between these coefficients and depth values, showing promising results. The complementary strengths of the completion methods, which excel in scenarios with sufficient depth information, and prediction methods, which are adept at handling no depth input, inspire us to unify the advantageous designs of both tasks. Although numerous methods have been developed for depth completion or depth prediction individually, these two research lines have progressed almost independently. To leverage their advantageous designs, in this work, we build our model upon a dual-branch encoder-decoder architecture that has been proven effective in depth completion tasks [4; 35; 36]. Additionally, we introduce some scene structure-related designs, such as plane representation and surface normal, which have been validated as effective in monocular depth estimation tasks, into both the model architecture and loss functions. Specifically, we designed a dual-branch network architecture. One branch takes the color image as input and focuses on the depth prediction task, while the other branch simultaneously processes the color image and depth information for the depth completion task. Both branches incorporate a local plane constraint module in their decoders to ensure high-quality depth estimation when depth points are sparse or absent. This module predicts plane coefficients using feature maps at multiple scales in the decoder and ensures the consistency of plane coefficients within local regions as the feature map scales are progressively recovered. In addition, a surface normal-based loss function is also adopted to enforce geometric constraints on the model's predicted depth maps. By integrating the dual-branch encoder-decoder architecture with the local planar guidance module, together with a mixture training strategy that utilizes data with different levels of sparsity for training, our model exhibits remarkable generalization capabilities for both depth completion and depth prediction tasks. It consistently achieves high performance across a wide range of input depth sparsity levels. Experimental results on the NYU Depth V2 dataset [37] showcase the superiority of our approach compared to recent sparse-invariant methods [38], particularly when dealing with extremely sparse or absent input depth. Furthermore, to validate the reliability of the model in real-world applications, we utilized an OPPO prototype equipped with an iTOF depth sensor to collect a dataset in indoor environments for testing. The results demonstrate that our model can effectively handle sparse depth maps with noise and irregular distributions encountered in practical applications. The main contributions of the work in this paper are summarized as follows:  
* We designed a depth completion model with high generalization capability, which effectively handles sparse depth maps with different sparse levels and irregular distribution, and ensures reliable depth prediction even when the input depth is limited or completely absent. * We integrated effective designs from both depth completion and monocular depth estimation tasks, establishing a dual-branch structured model. A multi-scale local plane constraint module and a surface normal-based loss function are adopted to further exploit scene structure-related information. * Our model achieves reliable depth results even when depth information is limited or completely absent, as demonstrated on the benchmark dataset NYU Depth V2 [37]. In real-world applications, this model can also effectively handle sparse depth maps with noise and irregular distribution.