# A Sparsity-Invariant Model via Unifying Depth Prediction and Completion
## 2 Related Work
### Image-Guided Depth Completion  
Image-guided depth completion has been widely studied these years, while the model structure and the modality fusion are also becoming diverse and complex. In Figure 1, we illustrate the evolution of network structures for the guided depth completion task. **Single Enc-Single Dec** [PERSON] et al. (2018) first employed a single encoder-single decoder architecture, utilizing color images to guide the completion of incomplete depth maps. Later methods ([PERSON] et al., 2018; [PERSON] et al., 2018) made individual enhancements in their respective decoders. Instead of direct fusion, some approaches ([PERSON] et al., 2018; [PERSON] et al., 2018; [PERSON] et al., 2018) use two separate convolution units to extract features from color images and depth map inputs independently. After concatenating the  
Figure 1: Network architectures for guided depth completion. multi-modal features at the feature level, they are fed into subsequent encoder-decoder networks to regress the complete depth map, completing the pixel-wise prediction task. **Dual Enc-Single Dec** Approaches based on dual encoder-single decoder architecture [20; 49; 50; 51; 52; 53] adopt the divide-and-conquer strategy, utilizing separate encoders for color images and depth maps to handle the two modalities independently and extract corresponding features. These features are fused into a correlated feature representation and passed to the decoder for further processing. The features acquired by the two encoders for their respective modalities typically exhibit consistent scales, easing their fusion in the decoder through concatenation, addition, or alternative methods. While the aforementioned methods solely integrate the features of color images and depth maps at a single spatial scale, multiple studies [54; 55; 56; 57] achieve multi-scale feature fusion. They leverage deep low-resolution features to capture global information and fully utilize shallow high-resolution features to prevent the loss of local structural information, which is crucial for dense prediction tasks. **Dual Enc-Dual Dec** The dual encoder-dual decoder structure [1; 58; 59] adds an additional decoder compared to the single decoder structure, as shown in Figure 1. In this architecture, two encoder-decoder networks receive images and sparse depth as inputs, respectively. The fusion operation for dual encoder-single decoder networks typically occurs between these two encoders. However, for dual encoder-dual decoder networks, high-resolution features obtained from the image encoder-decoder network are used to guide the depth recovery process of the sparse depth decoder. The fusion between the decoder and depth encoder also typically occurs at multiple scales. **Dual-Branch** The core design of a dual-branch structure is to predict depth maps from two complementary perspectives and then adaptively fuse the predictions of the two branches. [PERSON] et al. [36] introduced the first dual-branch model from both global and local perspectives, yielding favorable results. As the network evolves, the two-branch model with rgb and surface normal branches [4], or with image dominant and depth dominant branches [35], are gradually proposed. **Two-Stage** A two-stage structure [60; 61; 62; 63] generally adopt a coarse-to-fine prediction strategy. Inspired by the residual learning framework, some two-stage works [64; 65; 66; 67] decompose the final depth prediction into the estimation of a dense depth map and a residual depth map. These two maps are then linearly combined to obtain the final depth result. Additionally, there are coarse-to-fine structures incorporating the depth refinement module, the spatial propagation network, after [PERSON] et al. [68] first proposed a convolutional spatial propagation network used for depth completion, which generates a spatial-invariant affinity matrix to refine depth. Following this idea, some methods [5; 6; 7; 8; 10; 35] continuously advanced it by continuously expanding the field of propagation or iterative propagation strategy.