# Learning Deep CNN Denoiser Priors for Depth Image Inpainting
## 1 Introduction  
Depth images are a key research topic in the field of 2D-to-3D technology, which is widely used in machine vision and 3D reconstruction. However, due to the limitations of hardware and the characteristics of 3D scenes, black holes can easily form in depth images, and they are susceptible to noise. Due to the practical benefits of using depth images, some scholars have begun to investigate the problem of depth image inpainting. Depth images can be viewed as natural images without texture. Therefore, many natural image inpainting methods can be directly applied for depth image inpainting. In general, inpainting methods can be roughly divided into three categories, as discussed below. Inpainting methods of the first type rely on spatial information. Several representative algorithms are described as follows. [PERSON] et al. [1] proposed a fast depth image inpainting method based on the mechanism of human vision. This method has a high inpainting speed, but boundaries can easily become blurred. [PERSON] et al. [2] proposed the weighted joint bilateral filter method. Similarly, although the speed of this method is high, the boundaries in the depth images are not well handled. [PERSON] et al. [3] proposed an adaptive edge-oriented smoothing process to solve the depth image inpainting problem. This method achieves a good balance between time consumption and image quality. Methods of this type rely on general methods of image repair. Inpainting methods of the second type use aligned color images as a guide. Several representative algorithms are described as follows. [PERSON] et al. [4] used a deep-hole inpainting method with color guidance weights combined with bilateral differences. This method shows good robustness. [PERSON] et al. [5] proposed a method based on block matching with the edges of aligned color images to guide the direction of inpainting. [PERSON] et al. [6] trained a decision tree to identify edges in depth images andcorresponding color images for inpainting. Methods of this type require constrained repair based on aligned color images, and the process is complex. The third class of inpainting methods consists of model-based optimization methods. Several representative algorithms are described as follows. [PERSON] et al. [7] proposed a low-rank matrix method for depth image enhancement. This method has a poor effect when large-area information is lost. [PERSON] et al. [8] proposed the integration of low gradient regularization with low-rank regularization for depth image inpainting. This method offers improved inpainting quality. Methods of this type are flexible but time consuming. At present, they are attracting considerable attention and have become the mainstream approach to the problem at hand. Therefore, we plugged learned denoiser priors into such a model to solve the problem of depth image inpainting. Within the framework of model-based optimization methods, we investigated inpainting for single depth images. Depth image inpainting is an ill-posed inverse problem. We can obtain a potential depth image by solving the following formula [9]:  
\[\overset{\Lambda}{\mathbf{x}}=\text{argmin}_{\mathbf{x}}\frac{1}{2}||\mathbf{ H}\mathbf{x}-\mathbf{y}||_{2}^{2}+\lambda\cdot\psi(\mathbf{x}) \tag{1}\]  
where \(\mathbf{x}\) and \(\mathbf{y}\) are the potential depth image and the degraded depth image, respectively; \(||\mathbf{H}\mathbf{x}-\mathbf{y}||_{2}^{2}\) is the data-fidelity term; \(\psi(\mathbf{x})\) is the regularization term; and \(\lambda\) is a weight parameter. \(\mathbf{H}\) is a binary template in which missing information is represented by values of 0 and all other information is represented by values of 1. We hope to obtain a potential depth image \(\mathbf{x}\) from a degraded depth image \(\mathbf{y}\). With the help of variable splitting techniques, many scholars have decoupled the data-fidelity term and the regularization term in Equation (1). Then, they have used various strategies to solve the resulting subproblems.