# A Precise-Mask-Based Method for Enhanced Image Inpainting

Authors: Wanxu Zhang; Yi Ru; Hongqi Meng; Min Liu; Xiaolei Ma; Lin Wang; Bo Jiang
Journal: Mathematical Problems in Engineering
 (2016)

---

## Abstract

Mask of damage region is the pretreatment step of the image inpainting, which plays a key role in the ultimate effect. However, state-of-the-art methods have attached significance to the inpainting model, and the mask of damage region is usually selected manually or by the conventional threshold-based method. Since manual method is time-consuming and the threshold-based method does not have the same precision for different images, we herein report a new method for automatically constructing the precise mask by the joint filtering of guided filtering andL
0 smoothing. It can accurately locate the boundary of damaged region in order to effectively segment the damage region and then greatly improves the ultimate effect of image inpainting. The experimental results show that the proposed method is superior to state-of-the-art methods in the step of constructing inpainting mask, especially for the damaged region with inconspicuous boundary.


---

## Body

## 1. Introduction

Image inpainting as a technique of restoring missing or damaged areas of an image in an undetectable form for observers has drawn considerable attention over recent years due to the demand for revisiting damaged works of paintings or photography to make them more legible [1]. Remote sensing images are generally degraded by motion blur, the random noise, and the cloud that becomes a popular topic in remote sensing field [2]. Image inpainting can obtain the high quality image from degraded image and it can inpaint the random noise region. The noise-like white or off-white region can be located and further inpainted using the CDD or other inpainting models in this paper. The fundamental process of image inpainting is to construct a mask to locate the boundary of damaged region followed by subsequent inpainting process. State-of-the-art methods have attached significance to the inpainting model, and the mask of damage region is usually selected manually or by the conventional threshold-based method. Since manual method is time-consuming and the threshold-based method does not have the same precision for various images, therefore, it is of vital significance to automatically construct a precise mask for the damaged region of an image in order to fully restore the missing message, which is the aim of this paper.To date, the numerous approaches [3–8] have been proposed for image inpainting since the seminal work of BSCB [3, 4]: for example, the classical Total Variation (TV) [7] and Curvature-Driven Diffusion (CDD) [8] models, which are based on solving Partial Differential Equation (PDE) to preserve the directions of the isophotes in the missing region. In practice, it is assumed that the basement layer is white and off-white photo letter or wall, so it is with the damaged region. Then a certain range of white area is taken as the mask based on the threshold which applies to the threshold of the pixels in the image to be inpainted.However, the boundary between damaged and undamaged regions is usually inconspicuous, which may result in two main drawbacks using the threshold method:(
1
) some pixels belonging to damaged region may be lost when the threshold is high, which leads to the occurrence of bleached inpainting region; (
2
) some bright pixels in undamaged region are probably treated as the region which needs to be inpainted when the threshold is low. Therefore, the validity of conventional threshold method is strongly restricted; that is, it works on condition that the boundary of damaged region is conspicuous. In this work, we introduce a new method to construct a more precise mask based on guided filtering and L
0 smoothing in order to preciously locate the boundary of damaged region. Experimental results demonstrate that our method shows superiority over the conventional method in image inpainting especially for the images with inconspicuous boundary of damaged region.


## 2. Related Work

Two main steps comprise the conventional method for image inpainting.Step 1.
The construction of mask image: In detail, setting two values—255 and 0—to represent absolutely white and black, respectively. Supposing that the inpainting region approaches white, when each RGB value of pixels is higher than the limit value, the pixels are labeled as damaged pixels, and then the values of corresponding pixels in mask are assigned to 255. Otherwise, the values are assigned to 0. The principle of constructing mask with threshold method is shown in Figure1, where I and M denote the damaged image and the mask, respectively.Figure 1
Process of building mask by using the threshold method.Step 2.
After the construction of mask image, the damaged region is inpainted iteratively by CDD model with the mask above. CDD inpainting is like the process of heat diffusion with the diffusion intensity depending on both the gradient and curvature. The diffusion intensity is high when the curvature is large and decreases along with the isophote furtherly smoothing. Hence, the CDD model can eliminate the large curvature and stabilize the small curvature.


## 3. The Proposed Method

As mentioned above, it is difficult to build mask when the boundary of damaged region is inconspicuous. Inspired by dark channel prior [9], a new method is proposed to deal with this issue. The definition of dark channel is described as(1)
J
d
a
r
k
x
=
min
y
∈
Ω
x
⁡
min
c
∈
r
,
g
,
b
⁡
J
c
x
,where J
c is a color channel of J and Ω
(
x
) is a local patch centered at x. J
d
a
r
k is the corresponding dark channel image. The size of Ω
(
x
) is 1
×
1 in this paper. As the natural outdoor images are usually colorful and full of shadows, the dark channel of these images is really dark. The bright parts in these images have very high intensity in each RGB channel, so that the brightest point in the dark channel image is usually close to white.Since the damaged region (like noise or cloud region) in an image is generally white, it is assumed that the damaged region approaches white and its boundary is inconspicuous. Resultantly, the brightest pixel is considered as the most obviously damaged pixel. In addition, the colors of the damage region and its undamaged adjacent region are obviously different, which means that the damage region and undamaged region can be identified by our eyes. The most obvious damage pixel is labeled as reference pointA. The pixels whose RGB channel is approximately balanced may be damaged pixels. Therefore, the pixels are selected as damaged pixels according to their balanced degree of RGB and approaching degree to reference point A.The method mentioned above is sufficient for building mask when the boundary of damaged region is clear. However, it fails to work well when the boundary is inconspicuous. Motivated by this challenge, we here propose a new method to build precise mask for the boundary of damaged region based on guided filtering [10] and L
0 smoothing [11, 12]. In brief, guided filtering is employed to enhance image details, and then L
0 smoothing is used to sharpen major edges and eliminate low-amplitude structures. These two processes act as the first two steps of our proposed method as illustrated in Figure 2 and the theoretical details are provided as follows.Figure 2
Sketch map of the proposed method.In [10], it is assumed that the output image q is linearly related to the guide image I in a local place in guide filter; then the output image in window ω
k with radius r centered at pixel k can be expressed as q
i
=
(
a
k
I
i
+
b
k
) the linear coefficient a
k
,
b
k are constant in ω
k. This linear model ensures that q and I have the same edges due to ∇
q
=
a
∇
I.To determine the linear coefficienta
k
,
b
k, the filtering input p is constricted, and the difference between output q and input p is considered to be noise, expressed as (2)
q
i
=
p
i
-
n
iand then the linear coefficient a
k
,
b
k can be solved by minimizing the cost function:(3)
E
a
k
,
b
k
=
∑
i
∈
ω
k
a
k
I
i
+
b
k
-
p
i
2
+
ε
a
k
2
,where ε is a regularization parameter. The guided filter behaves as an edge-preserving smoothing operator when I and p are identical. Under this circumstance, the filtered information is the difference between I and q, which is expressed as I
-
q. This information represents the detailed information of I and is used for the following enhancement process.The enhancement via guided filtering is written as(4)
I
e
=
q
+
ξ
I
-
q
,where I
e is the result of the enhancement through guided filtering and ξ is the enhancement factor. As seen from (4), I
e increases with the increase of ξ, meaning that the detailed information and therefore the boundary of damaged region will be enhanced as ξ is high; otherwise, the enhancement will be weak.L
0 smoothing is used to suppress low-amplitude details and sharpen the salient boundary. L
0 smoothing is realized by (5)
min
S
⁡
∑
p
S
p
-
L
p
2
+
λ
·
C
S
,where L is the input image and the gradient of pixel p in the output image S is expressed as ∇
S
p
=
(
∂
x
S
p
,
∂
y
S
p
)
T. Here ∂
x and ∂
y are the gradients at the directions of x and y, respectively. C
(
S
) is defined as a counter to count the number of the pixels whose gradient is nonzero; that is, ∂
x
S
p
+
∂
y
S
p
≠
0. The aim of minimization C
(
S
) is to sharpen the major boundary, so that the boundary of damaged region can be preserved in S and then found in the gradient of S.Figure2 shows the sketch of our method and the steps are listed below.Step 1.
The damaged imageI is enhanced by guided filtering, and the result is represented by I
e.Step 2.
I
e is smoothed by L
0 smoothing, and the result is denoted by I
L
0.Step 3.
The interim maskM
1 is built by RGB balance and approaching degree of A from I
L
0.Step 4.
The maskM
2 is built by M
1 and the gradient map of I
L
0.Step 5.
M
3 is represented by M
1
+
M
2.Firstly, the original damaged imageI is enhanced by (4), and the enhancement of I is represented as I
e. The smoothed result of I
e by L
0 smoothing is denoted as I
L
0.The brightest pixel in the dark channel ofI
L
0 is regarded as the reference point A mentioned above. Regarding the construction of mask M
1 the pixels are treated as damaged pixels when the following two conditions are satisfied: (
1
) the pixels’ RGB approaching degree does not exceed the given coefficient δ
1; (
2
) the differences between RGB average of A and the pixels’ RGB average do not exceed coefficient δ
2. Otherwise, the pixels will be treated as undamaged pixels. As stated in Section 2, in our method the pixels of damaged region are all set to 255 and the rest are set to 0, and we thus construct a new grayscale image denoted as mask M
1. Generally, the white region in M
1 is mainly the inner region of damaged region; the other pixels especially the ones at the boundary in the damaged region are probably not included in M
1. Therefore, M
1 acts as an interim mask and will be utilized to construct more precise masks in the following steps.The next step is to locate the boundary of damaged region more precisely using guided filtering enhancement andL
0 smoothing. Then a more precise mask instead of M
1 is needed, which is constructed in the following way. We employ a small patch through M
1 which contains inner damaged region to match the gradient map of I
L
0 and then denote the included boundary of damaged region as mask M
2. Likewise, the boundary is set to 255 and the rest is assigned to 0 in M
2. Based on this, we can obtain a more precise mask M
3 defined as M
1
+
M
2. Note that the size of the patch here should not be large, or the boundary will be too thick.


## 4. Experimental Results

In this section, we further test the performance of the proposed method using Matlab2011b in a personal computer with an Intel Core i7 processor. We collect the tested images fromhttp://imagedatabase.cs.washington.edu/. In this paper, the damaged regions like white text and curves in original images are manually made and superimposed on undamaged images with Photoshop.We compared the proposed method with the traditional methods, and the results are shown in Figure3. Figures 3(a) and 3(b) are the undamaged and damaged image, respectively. Figures 3(c) and 3(d) illustrate the inpainting regions (the orange regions) found by our method and threshold method, respectively. As can be seen from Figure 3(d), some undamaged regions (e.g., mountains indicated by the red arrow) are wrongly selected to be inpainted by using the threshold method. In comparison, fewer undamaged regions are found with the proposed method as indicated by Figure 3(d).Figure 3
Comparison of inpainted results. (a) Original undamaged image. (b) Damaged image. (c) and (d) are the inpainting regions found by our method and threshold method, respectively. (e) and (f) are the inpainting results using our method and threshold method, respectively. (g) and (h) are the zoom-in patch of (e) and (f), respectively.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)Figures3(e) and 3(f) show the inpainting results using the proposed and threshold methods with both basing on CDD model, respectively. The zoom-in views of the red rectangle patches in Figures 3(e) and 3(f) are shown in Figures 3(g) and 3(h), respectively. Obviously, some damaged regions are not inpainted using the threshold method as indicated by the red arrows in Figure 3(g). In contrast, our method is superior to the threshold method especially for the images with inconspicuous boundary of damaged region.


## 5. Conclusion

In this paper, we report a new method for building a precise mask of damaged region for image inpainting. With the precise mask and the combinations of the guided filtering enhancement andL
0 smoothing, our method exhibits satisfactory performance in image inpainting and shows superiority over conventional methods for images with inconspicuous boundary of damaged region. It is also interesting to apply our method to the inpainting of images colorful damaged region; meanwhile, a precise cloud region can be precisely detected and acquired as a byproduct of image inpainting. We leave this problem for future research.



---

*Source: 6104196-2016-02-16.xml*