## 2 State of the Art  
Many image interpolation-related problems have used a scene's color information to guide the propagation of some other type of information, such as depth inpainting and image enhancement [2]. Depth completion is not an exception. Similarly, the geometry of color images has been used to drive a diffusion process completing sparse depth maps [3]. A bilateral filter is one of the most classic examples of a guided method. This image-domain filter considers regional color characteristics, such as edges and smooth areas, to slow or accelerate depth propagation, respectively. Figure 1: Examples of depth maps containing large areas of missing data. (**a**) Reference color image. (**b**) Depth map, holes (in black) due to occlusions. (**c**) Color reference image; a white square indicates reflections. (**d**) Depth map with holes in areas of high reflection due to uncertainty in the stereo-matching process. More recently, deep learning techniques have taken the lead. For instance, [PERSON] et al. [4] proposed completing a sparse depth image jointly by reconstructing a gray-level scene image, using convolutional neural networks. They evaluated this method using the KITTI [5] data set, which is a publicly available data set and one of the most popular databases for performance evaluation in the field. Another deep learning approach by [PERSON] et al. [6; 7] utilizes a CNN-based model to interpolate depth data from a Kinect sensor database. Another example is the work of [PERSON] et al. [8]. Their proposal focused on preventing the generation of mixed-depth pixels, that is, pixels that are neither background nor foreground, which are typical in object discontinuities. They proposed a new and interesting representation of pixels called Depth Coefficients (DCs). Hence, the loss function used to train the network is based on the DC representation, leading to a depth interpolation that tries to make more accurate decisions on the image edges. [PERSON] et al. [9] additionally proposed a model to complete depth maps based on an RGB image and a sparse depth map. Using the RGB image and a deep neural network, the authors estimated the normals of the surfaces in the scene. Using this information and the raw depth data acquired by a sensor, the authors minimized a depth estimation error for the whole image. They evaluated their proposal using their own data set. The results show that it outperforms classical models such as TGV [10] and bilateral filters. In the work of [11], the authors proposed a model to complete depth data. Their proposal used a unified CNN framework incorporating two features: (a) a model considering the relation between normals and surfaces in the depth value, and (b) a model predicting the confidence of the depth data. In this way, this model predicts surface normals, depth, and depth confidence values simultaneously. These estimations are inputs for a refinement module in order to obtain the final depth estimation. Evaluations of the model in the KITTI and NYU_V2 data sets show that the model presents state-of-the-art performance. The work in [12] presents a non-local depth completion model, which is different from other non-local models because the proposal dynamically selects more informative neighbors. The depth completion is iteratively refined using confidence values assigned to the depth data. The model also incorporates learned affinity to improve the accuracy of depth estimation, avoiding the \"mixed depth\" in the edges of the objects in the scene. Experiments show that the proposal outperforms DeepLidar and FuseNet [13] in the KITTI data set. In [14], a CNN model that fuses sparse depth data and color information to complete a depth map is presented. The model uses two approaches: one processes the color image, and the second refines the depth completion. The outputs from both approaches are combined in order to improve the depth completion. Furthermore, the 3D information of the scene is used to improve the depth estimation.