# A Criminisi-DnCNN Model-Based Image Inpainting Method

Authors: Zun Li; Yuanpei Zhu; Yuping Wang
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

Existing image inpainting methods achieve unideal results in dealing with centralized inpainting areas. For this reason, in this study, a Criminisi-DnCNN model-based image inpainting method is proposed. Inspired by the manual inpainting technology, the pointwise mutual information (PMI) algorithm was adopted to obtain the marginal structural map of the images to be repaired. Then, the Criminisi algorithm was used to restore the marginal structure to obtain the complete marginal structure image guided by the superficial linear structure. Finally, the problem of texture inpainting was converted into the counterpart of image denoising through the separation of variables by using the denoising convolutional neural network image denoiser (DnCNN). Compared with the existing inpainting methods, this model has improved the clarity of the marginal structure and reduced the blurring of the area to be repaired.


---

## Body

## 1. Introduction

Image inpainting technology, a research direction in image reconstruction, is the hotspot in image processing and computer vision fields. Essentially, image inpainting aims to use the known information to fill in the missing area in the image to be repaired and restore the area with incomplete information so as to ensure the natural overall inpainting effect and meet the visual needs of people. When the inpainting area is centralized, the lack of surrounding prior information will lead to unsatisfactory result. So, image inpainting requires further research.Currently, image inpainting methods are divided into three main types [1–6]: the variational partial differential-based method, texture synthesis-based method, and deep learning-based method. In the first type, the heat flow partial differential equation in physics is used to diffuse the information around to the inside of the broken area in the image to repair images. The pivotal part of the second method lies in treating the intact areas of the image as samples. Then, the areas to be repaired are dealt with to achieve consistent inpainting. As for the third approach, the outstanding learning capability of the network is given full play to extract information and perform content inference layer by layer from pixel-level raw data to abstract high-level semantic concepts. When the areas to be repaired are more centralized, the deep learning-based method represents better inpainting than the other two methods mentioned above. A model of multiscale neural patch synthesis based on joint optimization of image content and texture constraints was proposed [7]. On the one hand, the low-resolution images are restored using a context encoder, which will be optimized under the guidance of the idea of neural patch matching in style transfer. Finally, detailed texture information in existing regions of the image will be transferred and generated. On the other hand, in this model, a multiscale inpainting model is used to repair high-resolution images. The upsampling image of the texture optimization at the next scale is input as the upper scale. In this manner, the high-frequency details are continuously optimized. This model has the advantage of inpainting from coarse to fine, which makes the repaired images have a good visual effect, but its disadvantage is time consuming. Besides, a rapid and flexible nonblind denoising network is proposed [8] to accomplish the image inpainting, which transforms the inpainting problem into a subproblem of image denoising. In this denoising model, an adjustable noise level map is introduced as the network input and denoising is conducted on the downsampling subimage space. This method features weighing the degree of denoising the noise level map and detail retention. However, the method of obtaining the subimage of this model is very simple, and the method of restoring the whole image is simpler, and the effect is poor. Furthermore, an end-to-end network model for texture optimization and inpainting is proposed [9]. In this model, coarse restoration results are generated, and depth features are obtained using a content initialization network. Then, substitution is performed using the most similar neural patch in known regions. Finally, the restored image is obtained after decoding. This model features high-frequency texture detail migration. It is worth noting that this approach is a meta-algorithm, so the final result can be improved by a variety of network architectures and training techniques. The artist-net network [10] is proposed to realize the image inpainting, characterized by displaying structure inference and perception, including an image structure inpainting model and a texture filling model based on structure priors. In this method, the image structure is used to represent the scene as a whole, and then texture inpainting is conducted under the guidance of the important information of the restored image structure. In this way, an intact, restored image can be obtained. However, the content and expression of this model are not good, so it needs to be further improved to improve the repair effect.In response to the unsatisfactory edge repair like blurred or even incorrect inpainting when the inpainting area is concentrated, manual inpainting generally comes first for the edge, and then the texture information is restored. Influenced by the manual inpainting technology and the previous literature [1, 8, 10], a Criminisi-DnCNN model-based method of image inpainting is proposed in this study in terms of the centralized areas to be repaired, which can avoid blurred or even wrong inpainting to achieve better results. The algorithm in this study has made the following contributions:(1)
The PMI algorithm is for the marginal extraction of nonlocal features at the pixel level. This method has the advantages of simple and handy operation, as well as great adaptability. The PMI algorithm was used to extract the marginal structure of the image to be repaired, providing useful information for the following inpainting of marginal structure.(2)
The Criminisi and DnCNN algorithms were combined to realize image inpainting. Among them, the Criminisi was designed to repair the marginal structure to obtain the complete map of this part. By contrast, the DnCNN was used for texture inpainting by converting the problem of inpainting into an image denoising counterpart.


## 2. Related Work

### 2.1. Criminisi Algorithm

The crucial part in the Criminisi [11] lies in filling the information-impaired area by matching with the existing pixel blocks in the image with the pixel on the damaged edges as the center so as to repair the image. It contained the following three steps:(1)
Calculation of Priority. The priority of the Criminisi algorithm is determined by the confidence that represents the proportion of information in the original image in the block to be repaired and data items that show the strength of the linear structure.(2)
Search and Filling in the Best Matching Template. After determining the pixels getting priority repair, the search and filling of the global optimal match module were performed in the intact region on the SSD principle.(3)
Updating of the Confidence. Once the best matching template was determined and filled to the corresponding position, the edges of the area to be repaired would be updated accordingly.The above three steps were repeated continuously before the area to be repaired was filled to realize the inpainting.


### 2.2. Image Inpainting Model

Typically, image inpainting is an ill-posed inverse problem, which is performed based on the existing information contained in the image. In other words, the inpainting problem can be changed from ill posed to posed by adding a suitable regularization item for constraint solving, i.e., recovering a complete image from an image with missing information as shown in the following equation:(1)xΛ=argminx12Hx−y22+λ·Ψx,where x represents the potential complete image; y is the known observation image to be repaired; H represents the binary degenerate operator; λ represents the regularization parameter used to balance the first two terms.The commonly used prior image models include the following types [12–16]: the local smoothness model, nonlocal self-similarity model, low-rank representation model, sparse representation model, and deep learning model. Takeda [12] focuses on kernel regression methods, extending the classical kernel regression to the adaptive kernel regression to recover corrupted noiseless high-frequency information, achieving great results. Gilboa [13] has proposed to adopt the nonlocal operators to define new functions for image processing. Compared to the classical algorithms based on the partial differential equation, this approach can be considered an extension of spectral graph theory and diffusion geometry framework to functional analysis and partial differential equations, better handling repetitive textures and structures. Gu et al. [14] proposed the weighted nuclear norm minimization problem. Specifically, it is adaptively weighting values on different singular values, and this method has achieved notable success in the field of image reconstruction. Zhang et al. [15] presented a high-quality image restoration algorithm based on the structural group sparse representation model, which can ensure an efficient sparse representation of images in the structural group domain. This algorithm effectively portrays the local sparsity and nonlocal self-similarity of the image uniformly to accomplish the image inpainting task. Dong et al. [16] proposed a new inpainting loss function that adds a supervised term to solve actual problems. Specifically, based on a trained generative model, an inpainting loss function is used to search for the nearest encoding of the corrupted image in the low-dimensional space. Then, the code infers the missing content by generating a model. This method has advantages over other existing methods.In recent years, deep learning models have made great breakthroughs, thanks to the power of computers, larger data set, and more advanced networking techniques. Deep learning models only need to be trained end to end to achieve better performance instead of bearing a high computational cost.


## 2.1. Criminisi Algorithm

The crucial part in the Criminisi [11] lies in filling the information-impaired area by matching with the existing pixel blocks in the image with the pixel on the damaged edges as the center so as to repair the image. It contained the following three steps:(1)
Calculation of Priority. The priority of the Criminisi algorithm is determined by the confidence that represents the proportion of information in the original image in the block to be repaired and data items that show the strength of the linear structure.(2)
Search and Filling in the Best Matching Template. After determining the pixels getting priority repair, the search and filling of the global optimal match module were performed in the intact region on the SSD principle.(3)
Updating of the Confidence. Once the best matching template was determined and filled to the corresponding position, the edges of the area to be repaired would be updated accordingly.The above three steps were repeated continuously before the area to be repaired was filled to realize the inpainting.


## 2.2. Image Inpainting Model

Typically, image inpainting is an ill-posed inverse problem, which is performed based on the existing information contained in the image. In other words, the inpainting problem can be changed from ill posed to posed by adding a suitable regularization item for constraint solving, i.e., recovering a complete image from an image with missing information as shown in the following equation:(1)xΛ=argminx12Hx−y22+λ·Ψx,where x represents the potential complete image; y is the known observation image to be repaired; H represents the binary degenerate operator; λ represents the regularization parameter used to balance the first two terms.The commonly used prior image models include the following types [12–16]: the local smoothness model, nonlocal self-similarity model, low-rank representation model, sparse representation model, and deep learning model. Takeda [12] focuses on kernel regression methods, extending the classical kernel regression to the adaptive kernel regression to recover corrupted noiseless high-frequency information, achieving great results. Gilboa [13] has proposed to adopt the nonlocal operators to define new functions for image processing. Compared to the classical algorithms based on the partial differential equation, this approach can be considered an extension of spectral graph theory and diffusion geometry framework to functional analysis and partial differential equations, better handling repetitive textures and structures. Gu et al. [14] proposed the weighted nuclear norm minimization problem. Specifically, it is adaptively weighting values on different singular values, and this method has achieved notable success in the field of image reconstruction. Zhang et al. [15] presented a high-quality image restoration algorithm based on the structural group sparse representation model, which can ensure an efficient sparse representation of images in the structural group domain. This algorithm effectively portrays the local sparsity and nonlocal self-similarity of the image uniformly to accomplish the image inpainting task. Dong et al. [16] proposed a new inpainting loss function that adds a supervised term to solve actual problems. Specifically, based on a trained generative model, an inpainting loss function is used to search for the nearest encoding of the corrupted image in the low-dimensional space. Then, the code infers the missing content by generating a model. This method has advantages over other existing methods.In recent years, deep learning models have made great breakthroughs, thanks to the power of computers, larger data set, and more advanced networking techniques. Deep learning models only need to be trained end to end to achieve better performance instead of bearing a high computational cost.


## 3. Criminisi-DnCNN Image Inpainting

Manual restoration generally targets the marginal structure first, followed by the inpainting of texture information. An intact marginal structure can provide reliable prior knowledge for further inpainting. The Criminisi-DnCNN method proposed in this study also adopts the above idea to accomplish the concentrated image inpainting task in inpainting areas.


### 3.1. Marginal Structure Extraction

Marginal structure extraction has always been a research hotspot in image processing and pattern recognition. Figure1 shows the contrast of marginal structure extraction using different methods. Figure 1(a) represents an original image, Figure 1(b) represents the Sobel method, Figure 1(c) represents the Roberts method, Figure 1(d) represents the Prewitt method, and Figure 1(e) represents the PMI method. The essence of the traditional marginal structure extraction method [17–20] is linear filtering to obtain local features, which is prone to noise and marginal information redundancy, limiting the application of marginal structure in image processing, as shown in Figures 1(b)–1(d). Therefore, research on marginal structure detection and extraction methods has focused on the increase of neighborhood, scale, direction, and types of features. As mentioned above, the PMI algorithm proposed in literature [21]is for the marginal extraction of nonlocal features at the pixel level. In this method, the affinity is obtained using pointwise mutual information to determine whether the pixels are on the same target object, thus obtaining the marginal structure. PMI method has the advantages of simple and handy operation, as well as great adaptability, as shown in Figure 1(e). Therefore, this study applies the PMI method to the marginal structure extraction of images to be repaired.Figure 1
Contrast of marginal structure extraction.
(a)(b)(c)(d)(e)


### 3.2. Marginal Structure Inpainting

The Criminisi method can make full use of the peripheral information of images to be repaired and accomplish the inpainting task with linear structure, avoiding blurring and visual disconnections as much as possible. Thus, this method is suitable for situations where the area to be restored is concentrated. Therefore, for the marginal structure inpainting of images, this study adopts the Criminisi image inpainting method to accomplish the marginal inpainting task, as shown in Figure2.Figure 2
Contrast images of marginal structure extraction.
(a)(b)(c)(d)(e)(f)Figure2(a) represents a missing information image, Figure 2(b) represents a marginal structure image, and Figures 2(c)–2(f) represent the marginal structure inpainting effect of Kernel regression method [11], weight nuclear norm minimization method [13], group-based sparse method [14], and Criminisi method. The Kernel regression method shows blurring. The weight nuclear norm minimization method and group-based sparse method show broken marginal inpainting and could not satisfy the connectivity. The inpainting effect of Criminisi can satisfy the subjective visual requirements and does not show significant blurring and breakage. The reason is that the priority in the Criminisi algorithm method consists of confidence and data item. Pixel blocks with more information about the intact region of confidence should be given priority to repair. Besides, the data item indicates that the superficial linear structure has high strength, which should be given priority to repair.


### 3.3. Texture Inpainting Method

The structure of images is first repaired using the Criminisi method, followed by the inpainting of the image texture. By deforming equation (1) using the separation of variables technique and neglecting the iteration times, equations (2)–(4) can be obtained as follows:(2)x=argminx12Hx−y22+μ2x−u−b22,(3)u=argminuλ·Ψu+μ2x−u−b22,(4)b=b−x−u.The subproblem ofx represented by equation (2) is converted into a strictly convex quadratic minimization problem. Therefore, the following equation can be obtained:(5)x=HTH+μI−1HTy+μu+b.By deforming equation (4), the following equation can be obtained:(6)u=argminuΨu+12λ/μ2x−u−b22.In equation (6), if r=x−b, then r can be viewed as an observation of some noise of x [8, 11, 22, 23]. b represents residual, and u represents denoised image. Therefore, the texture inpainting subproblem of the image can be attributed to image denoising. Currently, the image denoising problem of images can be solved by the K-SVD denoising method, BM3D denoising method, and CNN network denoising method. The image denoising network of the CNN method has the advantages of high efficiency and strong prior modeling ability compared with other denoising methods. Zhang et al. [22] developed a set of well-trained CNN denoiser for image denoising, image super-resolution, and image deblurring. Li et al. [23] proposed the structure-texture decomposition model and two fusion rules are carefully designed to fuse them. This model made full use of the details and energy in the low-frequency component. The fused image can be reconstructed by the perfused high-frequency, low-frequency structure, and low-frequency texture. In literature [24], an effective, fast, and robust medical image fusion method is proposed. A two-layer decomposition scheme is introduced by the joint bilateral filter, the energy layer contains rich intensity information, and the structure layer captures ample details. Then, a novel local gradient energy operator based on the structure tensor and neighbor energy is proposed to fuse the structure layer and the l1-max rule is introduced to fuse the energy layer. In literature [25], temporally redundant information was used fully by recursive convolutional neural network for single- and multi-image image denoising and applied it to super-resolution of images. In literature [26], combining BM3D and convolutional neural network, the “extraction” and “aggregation” layers were used to model the block matching phase in BM3D to form BM3D-net and perform the image denoising task. In literature [27], DnCNN image denoising models were proposed. This model can combine residual learning and batch normalization, which has the advantages of fast training and strong denoising ability. Therefore, this study adopts the DnCNN model image denoising prior to image denoising. The network structure of DnCNN is shown in Figure 3.Figure 3
Network structure of DnCNN.As shown in the figure, the DnCNN image denoising network includes three types of layers: the first type of network layer is to form 64 feature maps and improve the nonlinear relationship between the layers; the second type of network layer is to facilitate the separation of potentially complete images from noisy ones; the third type of network layer is to reconstruct the output result layer. This network introduces the residual learning strategy into the denoising method and uses residual units to predict residual images. The mean square error of the residual image and the predicted residual image are used as the loss function training parametersΘ:(7)lΘ=12N∑i=1NRyi;Θ−yi−xiF2.In equation (7), yi,xii=1N denotes N image pairs. This network is trained using the dataset [28] with a training image block size of 40 × 40, a training image block number of 128 × 1600, and a training noise level of 15, 25, and 50.


## 3.1. Marginal Structure Extraction

Marginal structure extraction has always been a research hotspot in image processing and pattern recognition. Figure1 shows the contrast of marginal structure extraction using different methods. Figure 1(a) represents an original image, Figure 1(b) represents the Sobel method, Figure 1(c) represents the Roberts method, Figure 1(d) represents the Prewitt method, and Figure 1(e) represents the PMI method. The essence of the traditional marginal structure extraction method [17–20] is linear filtering to obtain local features, which is prone to noise and marginal information redundancy, limiting the application of marginal structure in image processing, as shown in Figures 1(b)–1(d). Therefore, research on marginal structure detection and extraction methods has focused on the increase of neighborhood, scale, direction, and types of features. As mentioned above, the PMI algorithm proposed in literature [21]is for the marginal extraction of nonlocal features at the pixel level. In this method, the affinity is obtained using pointwise mutual information to determine whether the pixels are on the same target object, thus obtaining the marginal structure. PMI method has the advantages of simple and handy operation, as well as great adaptability, as shown in Figure 1(e). Therefore, this study applies the PMI method to the marginal structure extraction of images to be repaired.Figure 1
Contrast of marginal structure extraction.
(a)(b)(c)(d)(e)


## 3.2. Marginal Structure Inpainting

The Criminisi method can make full use of the peripheral information of images to be repaired and accomplish the inpainting task with linear structure, avoiding blurring and visual disconnections as much as possible. Thus, this method is suitable for situations where the area to be restored is concentrated. Therefore, for the marginal structure inpainting of images, this study adopts the Criminisi image inpainting method to accomplish the marginal inpainting task, as shown in Figure2.Figure 2
Contrast images of marginal structure extraction.
(a)(b)(c)(d)(e)(f)Figure2(a) represents a missing information image, Figure 2(b) represents a marginal structure image, and Figures 2(c)–2(f) represent the marginal structure inpainting effect of Kernel regression method [11], weight nuclear norm minimization method [13], group-based sparse method [14], and Criminisi method. The Kernel regression method shows blurring. The weight nuclear norm minimization method and group-based sparse method show broken marginal inpainting and could not satisfy the connectivity. The inpainting effect of Criminisi can satisfy the subjective visual requirements and does not show significant blurring and breakage. The reason is that the priority in the Criminisi algorithm method consists of confidence and data item. Pixel blocks with more information about the intact region of confidence should be given priority to repair. Besides, the data item indicates that the superficial linear structure has high strength, which should be given priority to repair.


## 3.3. Texture Inpainting Method

The structure of images is first repaired using the Criminisi method, followed by the inpainting of the image texture. By deforming equation (1) using the separation of variables technique and neglecting the iteration times, equations (2)–(4) can be obtained as follows:(2)x=argminx12Hx−y22+μ2x−u−b22,(3)u=argminuλ·Ψu+μ2x−u−b22,(4)b=b−x−u.The subproblem ofx represented by equation (2) is converted into a strictly convex quadratic minimization problem. Therefore, the following equation can be obtained:(5)x=HTH+μI−1HTy+μu+b.By deforming equation (4), the following equation can be obtained:(6)u=argminuΨu+12λ/μ2x−u−b22.In equation (6), if r=x−b, then r can be viewed as an observation of some noise of x [8, 11, 22, 23]. b represents residual, and u represents denoised image. Therefore, the texture inpainting subproblem of the image can be attributed to image denoising. Currently, the image denoising problem of images can be solved by the K-SVD denoising method, BM3D denoising method, and CNN network denoising method. The image denoising network of the CNN method has the advantages of high efficiency and strong prior modeling ability compared with other denoising methods. Zhang et al. [22] developed a set of well-trained CNN denoiser for image denoising, image super-resolution, and image deblurring. Li et al. [23] proposed the structure-texture decomposition model and two fusion rules are carefully designed to fuse them. This model made full use of the details and energy in the low-frequency component. The fused image can be reconstructed by the perfused high-frequency, low-frequency structure, and low-frequency texture. In literature [24], an effective, fast, and robust medical image fusion method is proposed. A two-layer decomposition scheme is introduced by the joint bilateral filter, the energy layer contains rich intensity information, and the structure layer captures ample details. Then, a novel local gradient energy operator based on the structure tensor and neighbor energy is proposed to fuse the structure layer and the l1-max rule is introduced to fuse the energy layer. In literature [25], temporally redundant information was used fully by recursive convolutional neural network for single- and multi-image image denoising and applied it to super-resolution of images. In literature [26], combining BM3D and convolutional neural network, the “extraction” and “aggregation” layers were used to model the block matching phase in BM3D to form BM3D-net and perform the image denoising task. In literature [27], DnCNN image denoising models were proposed. This model can combine residual learning and batch normalization, which has the advantages of fast training and strong denoising ability. Therefore, this study adopts the DnCNN model image denoising prior to image denoising. The network structure of DnCNN is shown in Figure 3.Figure 3
Network structure of DnCNN.As shown in the figure, the DnCNN image denoising network includes three types of layers: the first type of network layer is to form 64 feature maps and improve the nonlinear relationship between the layers; the second type of network layer is to facilitate the separation of potentially complete images from noisy ones; the third type of network layer is to reconstruct the output result layer. This network introduces the residual learning strategy into the denoising method and uses residual units to predict residual images. The mean square error of the residual image and the predicted residual image are used as the loss function training parametersΘ:(7)lΘ=12N∑i=1NRyi;Θ−yi−xiF2.In equation (7), yi,xii=1N denotes N image pairs. This network is trained using the dataset [28] with a training image block size of 40 × 40, a training image block number of 128 × 1600, and a training noise level of 15, 25, and 50.


## 4. Simulation Experiment

### 4.1. Image Inpainting

In this study, the Lenovo R720 computer is used as the hardware simulation platform, and the software simulation platform is MATLAB. In order to compare the subjective inpainting effect, the Criminisi algorithm and DnCNN are used in this study, as shown in Figure4. In order to evaluate the objective quality of the inpainting effect of the algorithm in this study, the traditional PSNR and SSIM are used, as listed in Table 1.Figure 4
Subjective inpainting effect.
(a)(b)(c)(d)(e)Table 1
PSNR and SSIM.
ImageMethod (PSNR/SSIM)CriminisiDnCNNProposedBird134.1492/0.961437.0005/0.963037.0966/0.9636Bird234.1915/0.962536.9777/0.963737.0211/0.9645Balloon138.2959/0.965738.6963/0.971538.9047/0.9742Balloon238.8072/0.977138.9153/0.973238.9242/0.9737Blueberry136.5968/0.915136.9903/0.918436.973/0.9195Blueberry229.8239/0.908836.9907/0.917236.9908/0.9175Bold values means the value of the objective data is best.Figure4(a) represents an original image, Figure 4(b) represents a missing information image, Figure 4(c) represents the inpainting effect of Criminisi method, Figure 4(d) represents the inpainting effect of DnCNN method, and Figure 4(e) represents the inpainting effect of the proposed method of Table 1. It can be seen that the Criminisi method is prone to produce inpainting error results when faced with a more concentrated inpainting area, and the information of foreground and background is not better distinguished. The DnCNN method is prone to fuzzy inpainting. The algorithm in this study can not only repair the marginal structure but also recover the texture information better based on both methods.In Table1, the PSNR and SSIM of the three algorithms are provided. Overall, the algorithm proposed in this study has comparative advantages, with an average improvement of 0.0521 dB in PSNR and 0.0003 in SSIM.


### 4.2. Size of Best Matching Template

This section shows the impact of the size of the best matching template on the inpainting.As shown in Figure5, the size of the best matching template for achieving a good repair effect is different for different images. Therefore, we choose different best matching template sizes for different images, subject to PSNR.Figure 5
Performance comparison with various sizes.


### 4.3. Iteration Number

This section shows the influence of iterations on the inpainting effect. About 10%, 20%, and 30% of the above original images were randomly lost and then repaired to obtain the average PSNR.As shown in Figure6, all PSNR curves are monotonically increasing. When the number of iterations is 30, the curve is flat and stable. Therefore, the iteration number of the simulation experiment is set to 30.Figure 6
Average PSNR of the iteration number.


## 4.1. Image Inpainting

In this study, the Lenovo R720 computer is used as the hardware simulation platform, and the software simulation platform is MATLAB. In order to compare the subjective inpainting effect, the Criminisi algorithm and DnCNN are used in this study, as shown in Figure4. In order to evaluate the objective quality of the inpainting effect of the algorithm in this study, the traditional PSNR and SSIM are used, as listed in Table 1.Figure 4
Subjective inpainting effect.
(a)(b)(c)(d)(e)Table 1
PSNR and SSIM.
ImageMethod (PSNR/SSIM)CriminisiDnCNNProposedBird134.1492/0.961437.0005/0.963037.0966/0.9636Bird234.1915/0.962536.9777/0.963737.0211/0.9645Balloon138.2959/0.965738.6963/0.971538.9047/0.9742Balloon238.8072/0.977138.9153/0.973238.9242/0.9737Blueberry136.5968/0.915136.9903/0.918436.973/0.9195Blueberry229.8239/0.908836.9907/0.917236.9908/0.9175Bold values means the value of the objective data is best.Figure4(a) represents an original image, Figure 4(b) represents a missing information image, Figure 4(c) represents the inpainting effect of Criminisi method, Figure 4(d) represents the inpainting effect of DnCNN method, and Figure 4(e) represents the inpainting effect of the proposed method of Table 1. It can be seen that the Criminisi method is prone to produce inpainting error results when faced with a more concentrated inpainting area, and the information of foreground and background is not better distinguished. The DnCNN method is prone to fuzzy inpainting. The algorithm in this study can not only repair the marginal structure but also recover the texture information better based on both methods.In Table1, the PSNR and SSIM of the three algorithms are provided. Overall, the algorithm proposed in this study has comparative advantages, with an average improvement of 0.0521 dB in PSNR and 0.0003 in SSIM.


## 4.2. Size of Best Matching Template

This section shows the impact of the size of the best matching template on the inpainting.As shown in Figure5, the size of the best matching template for achieving a good repair effect is different for different images. Therefore, we choose different best matching template sizes for different images, subject to PSNR.Figure 5
Performance comparison with various sizes.


## 4.3. Iteration Number

This section shows the influence of iterations on the inpainting effect. About 10%, 20%, and 30% of the above original images were randomly lost and then repaired to obtain the average PSNR.As shown in Figure6, all PSNR curves are monotonically increasing. When the number of iterations is 30, the curve is flat and stable. Therefore, the iteration number of the simulation experiment is set to 30.Figure 6
Average PSNR of the iteration number.


## 5. Conclusion

In this study, an image inpainting method based on Criminisi-DnCNN is proposed. Guided by the idea of manual inpainting, for the case where the concentrated areas are to be repaired, using this method means that the addition of the marginal structure of images is performed first, followed by the inpainting of the texture. The recovery of marginal structure using the Criminisi method can be complemented by fully using the surrounding marginal structure information. By variable separation technique, texture inpainting is transformed into an image denoising problem, which is solved by using a superior performance DnCNN image denoising prior method. The experimental results show that the algorithm can effectively perform the inpainting task [29].



---

*Source: 9780668-2022-08-02.xml*