# Blind Image Inpainting with Mixture Noise Usingℓ0 and Total Regularization

Authors: Xiaowei Xu; Shiqi Geng
Journal: Computational and Mathematical Methods in Medicine
 (2022)

---

## Abstract

The blind image inpainting problem need to be handle when faced with a large number of images, especially medical images in medical health. For the proposed nonconvex sparse optimization model, a proximal based alternating direction method of multipliers (PADMM) method is designed to solve the problem. Firstly,ℓ0 sparse regularization is imposed to the binary mask since the missing pixels are sparse in our experiments. Secondly, the total variation term is utilized to describe the underlying clean image. Finally, ℓ2 regularization of the fidelity term is used to solve the given blind inpainting problem. Experiments show that this method has better performance than traditional method, and could deal with the blind image inpainting problem.


---

## Body

## 1. Introduction

Medical images can directly reflect the function and health status of human tissues, and have become one of the standards of diagnosis and medical intervention. With the increasing availability and utilization of modern medical imaging such as disease database, X-ray film and magnetic resonance imaging, the demand for automatic processing of medical image data is increasing. With the application of medical images, automatic medical image analysis has become one of the hot directions of contemporary medical imaging research [1, 2]. In the process of image capture, imaging sensors broken or the error in information transmission may cause some pixels missing or corrupted by impulse noise [3–8].In this work, the image is defined as a vector withn pixels in lexicographic ordering, x∈Rn. It can be represented mathematically as follows:
(1)y=Axwhere y∈Rn is the observed image, x∈Rn is the clean image and A∈Rn×n is an identity matrix. If the clean image x is corrupted by the additive Gaussian noise ηg, the model will be updated as follows:
(2)y=Ax+ηgThe image is corrupted by impulse noise, and the pixels iny is corrupted by the impulse noise ηi. The blind image inpainting model with the mixture noise (Gaussian and impulse noise) is finally described as follows [9]:
(3)y=Ax+ηg+In−AηiHow to solve the E.q. (2) efficiently and effectively is the most important issue. It is clear that E.q. (2) is quite challenging. It has three unknown term A, ηg and ηi. The goal of this paper is to estimate the clean image x from the partial observation y without unknow mask A, noise ηg and ηi.In the earlier researches, the inpainting problem can be solved by many approaches related image reconstruction from the aspect of sparse modeling [10–13]. More common strategies for the removal of impulse noise for blind inpainting problems are to estimate an approximated A by computing the support set of the noisy pixels with some outlier detection methods, and apply the reconstruction methods for a known mask A.The difficulty of reconstructing an image with missing data and mixture noise is basically to detect the locations of outliers. Some filtering based methods estimate the missing values, such as the adaptive median filter [14], and adaptive center weighted median filter [15].Two phase based methods for blind inpainting problems involve the estimation of mask A, which is calculated by some outlier detection approaches. After the mask estimation, the inpainted image is generated by the image reconstruction step, which is implemented by a standard convex optimization. Some convex methods were ulitized to estimate the inpainted images [16]. Used the total variation (TV) regularizer [17] estimates the inpainted images. Xiao et al. [18] proposed a combination of ℓ1-norm and ℓ0-norm regularizers for simultaneously removing impulse noise and computing learning dictionary after the mask. In addition, the authors [19] presented an approach for mixed impulse and Gaussian noise removal. In the approach, a logarithmic transformation strategy is applied to convert the multiplication between the image and binary mask. Then, the image and mask terms are estimated iteratively with TV regularization applied on the image. Especially, the method can also be extended to the removal of impulse noise by relaxing the regularizer from the ℓ0 norm to the ℓ1 norm.Some approaches could estimate the mask and impulse noise field by an iterative process instead of involving a separated mask detection step, such as a low-rank matrix recovery method [20]. The proposed approach belongs to the category of simultaneously estimating the mask and impulse noise.To address the challenging blind inpainting task with mixture noise, a novel model is proposed based on imposing aℓ0 sparse regularization to the binary mask. The proposed model can be efficiently solved by a designed proximal based alternating direction method of multipliers (PADMM) method. The main contribution of this work is given as follows: 1) A new model that fits in the practical situation of blind inpainting problem is proposed. 2) The new model solves the challenging blind inpainting task with mixture noise. 3) An efficient algorithm is given to effectively solve the proposed model.The outline of this paper is given as follows. The proposed method including the new model and the designed algorithm is exhibited in Section2. The solution of the proposed method is described in Section 3. Section 4 shows the experimental results and analysis. Finally, we draw some conclusions in Section 5.


## 2. Model Building

The paper proposes a minimization model to solve the image inpainting problem on the basic blind image with mixture noise. we denote a as the verctor form of mask A and the E.q. (3) can be expressed as follows:
(4)y=a⊗x+ηg+Ι−a⊗ηiwhere ⊗ stands for a dot product between vectors, and Ι is a vector with all the value Ι.x∈Rn is the vector form of one matrix x∈Rn1×n2 with n=n1×n2 in (2), and a, y are the vectors with the same dimensions as x. As a is the position of pixels missing, the Ι-a represents the locations of impulse noise in the image.we will give more explanations for E.q.(4) to present the proposed model for the blind image inpainting with mixture noise:
(1)
Sincea⊗y−x mainly represents Gaussian noise, we use the ℓ2 norm a⊗y−x22 to construct the fidelity term3(2)
2.1‐a⊗y only represents impulse noise (e.g., salt-peppers, random value), thus we may use ℓ1 sparse regularization to describe 1‐a⊗y,i.e.,1‐a⊗y1(3)
Especially, if the impulse noise is relatively dense, it may impose £q regularization to a in the new model, i.e.,a0. Otherwise, if the impulse noise is sparse and the condition of minimizing a0 may not hold, we could easily set a small parameter to control it(4)
Finally, we employ the (anisotropic) total variation (TV) regularization to the underlying clean image x, i.e.,∇x1, the TV regularization is quite popular and useful in the applications of image processingAs presented above, we formulate the final proposed model for the inpainting task as follows:(5)x,a=argminx,a12a⊗y−x22+λ1∇x1+λ2a0+λ31−a⊗y1Theℓ0 minimization minaa0 can be equality described as minv<1,1−v> such that v⊗a=0 and 0≤v≤1 based on [8]. Thus, E.q.(2) can be expressed as follows:
(6)minx,a,z,w,v12a⊗y−x22+λ1zp,1+λ2<1,1−v>+λ3w1s.t.∇x=z,v⊗a=0,0≤v≤1,1−a⊗y=wAs discussed above, we may get the following augmented Lagrangian problem instead of the constrained minimization E.q. (6) with variable substitution.
(7)ζx,a,z,w,v,π1,π2,π3=12a⊗y−x22+λ1zp,1+λ2<1,1−v>+λ3w1+<π1,∇u−z>+β12∇u−z22+<π2,v⊗a>+β22v⊗a22+<π3,1−a⊗y−w>+β321−a⊗y−w22s.t.0≤v≤1Whereπ1, π2 and π3 are Lagrange multipliers, and β1, β2 and β3 are three positive parameters. The Lagrangian problem ζx,a,z,w,v,π1,π2,π3 can be solved alternatively and iteratively by the following minimization subproblems in Section 3.


## 3. Model Solution Method

We add the proximal term1/2x−xkD2 to the x subproblem from E.q.(7) and denote xD2=xTDx to get the following proximal
(8)xk+1=argminx12a⊗y−x22+<π1,∇u−z>+β12∇u−z22+12x−xkD2Where(9)D=1κΙn−diaga+β1∇T∇,κ∈0,11+β1∇2The solution of E.q.(8) is given as follows:
(10)xk+1=12x−pk22Where(11)pk=xk−κ1a⊗xk−y+∇Tπ1−β1z+β1∇T∇xkThea-subproblem is shown as the following:
(12)ak+1=argmina12a⊗y−x22+<π2,v⊗a>+β12v⊗a22+<π3,1−a⊗y−w>+β321−a⊗y−w22We need to discuss the solution by the following two cases:When a >0,(13)ak+1=−v⊗π2+β3y⊗Ι⊗y+y⊗π3y−x⊗y−x+β2v⊗v+β3y⊗yWhen a <0,(14)ak+1=v⊗π2+β3y⊗Ι⊗y+y⊗π3y−x⊗y−x+β2v⊗v+β3y⊗yTherefore, the reformulation is:(15)yk+1=sgnqk∗qk−v⊗π2hkWhere(16)qk=β3y⊗y+y⊗π3,hk=y−x⊗y−xz+β2v⊗v+β3y⊗yThez-subproblem can be written as the following minimization problem:
(17)zk+1=argminzλ1z1+<π1,∇u−z>+β12∇u−z22which has a closed-form solution by soft-thresholding [7].
(18)zk+1=Shrink∇x+π1β1,λ1β1Where(19)Shrinkx,t=max0,x−1t.∗signxSimilarly, thew-subproblem is given as the follows:
(20)wk+1=argminwλ3w1+<π3,1−a⊗y−w>+β321−a⊗y−w22which holds the closed-form solution by soft-thresholding:
(21)zk+1=Shrink1−a⊗y+π3β3,λ3β3Thev-subproblem is given as the follows:
(22)vk+1=argmin0≤v≤1λ2<1,1−v>+<π2,v⊗a>+β22v⊗a22which could also hold the closed-form solution [9]:
(23)vk+1=min1,max0,rkWhere(24)rk=λ2−a⊗π2β2a⊗aWe finally update the Lagrange multipliers by:(25)π1k+1=π1k+β1∇xk+1−zk+1,π2k+1=π2k+β2v⊗a,π3k+1=π3k+β31−a⊗y−wWe may effectively obtain the solution of the constrained model (5) with the initial guesses u0=v0=a0=0. We summarize the above steps to get the following Algorithm 1:Algorithm 1: Solve the optimization model (5) by PADMM.
Input: The degraded image y,a initial vector u0,v0,a0,The parametersλi,βi,i=1,2,3,The constantκ∈0,1/1+β1∇2And the calculation accuracyTol =1/255Output: The stripe noise component x,aInitialize:1)k⟵0,x0⟵1,v0⟵1,a0⟵0Whilek=1:103or Eng < Tol2) solvexk+1by Eq.(10)3) solveak+1by Eq.(15)4) solvezk+1by Eq.(18)5) solvewk+1 by Eq.(21)6) solvevk+1 by Eq.(23)7) update the multipliersπi,i=1,2,3 by Eq.(10)8) calculate the errorEng=∇xk+1−zk+12+v⊗a2+1−a⊗y−w2EndwhileAlthough Algorithm1 involves some parameters, these parameters are actually not sensitive and easy to select. We also compute the energy of each iteration. If the energy is below a given tolerance, the iteration will stop and output the final result.In the next section, we will exhibit the experiment results to demonstrate the effectiveness of the proposed method.


## 4. Experimental Results and Analysis

The numerical experiments in this section are implemented with MATLAB (R2016a) for both simulated and real images. The experimental computer has 2G RAM and Intel(R) Core(TM)i3-2370 M CPU: @2.40GHz 2.40GHz. Since the literatures for blind image inpainting with mixture noise are limited, we here only compare the proposed method with one recent state-of-the-art blind inpainting approach [6], denoted as “ASInpaint” 4.In Figure1, we present the whole process of image, which is degraded jointly by Gaussian noise and impulse noise. The goal of this work is to recover the clean image X from the degraded image Y. To evaluate the quantitative performance of the compared approaches, we employ two kinds of metrics to estimate the performance of different methods: peak signal-noise ration (PSNR) and structural similarity (SSIM)5 [21].Figure 1
The flowchart of how to simulate the input image. Note that A and X are both blind and need to compute.In the experiments, we assume that the pixel values are within the interval [0, 255]. The added salt&pepper type of impulse noiseηI can have a value of either 0 or 255. For Gaussian noise, the values are also uniformly distributed within the interval [0, 255]. For the parameters in Algorithm 1, we empirically set λ1 =0.9, λ2 =0.08, λ3 =0.08, and β1=β2=β3=200 for experiments. Note that we could tune the paramters to get better results, and we fix them in the experiments to illustrate the stability of the given method. For the parameters of “ASInpaint”, we keep the default settings of the provided code.In Figure2, we illustrate the visual performance of the two compared methods by four different simulated images with mixture noise (see Figure 2(a)) named “Lena”, “Cameraman”, “Phantom”, and “Satellite”. We added the Gaussian and impulse mixture noise on the images (see Figure 2(b)). In particular, we evaluate the effectiveness of the proposed method by the same image with different levels mixture noise (see Figure 2 the 1st and 3rd rows). Although the ASInpaint approach also obtained competitive results (see Figure 2(d)), the proposed method could obtain better visual performance, especially on the shape profile of the images (see Figure 2(e)). we have to emphasize that the visual performance of both methods seem to be not better than the recovered image by other literatures. The problem addressed is quite challenging that there are four unknown variables in the problem, such as the underlying clean image X, the mask A, the impulse noise NI, and the Gaussian noise NG. Although we may reduce to only two unknown variables X and A, it is still very difficult to recover the underlying clean image X. However, the given model has also recovered the relatively good visual results by the given algorithm.Figure 2
The visual comparisons between ASInpaint and the proposed method. (a) The ground-truth image; (b) The mask for the missing pixels; (c) The degraded image by Gaussian and impulse noise; (d) The recovered image by ASInpaint [6]; (e) The recovered image by the proposed method.
(a)(b)(c)(d)(e)Meanwhile, denoising results on a real color image by all competing methods (NLH method, NSNR method, WNNM method) is shown as Figure3. The 15 test images used in image denoising experiments are shown as Figure 4. Inpainting results on images Starfish by different methods (Random mask with 75% missing values) are shown as Figure 5. Inpainting results on images Monarch by different methods (Text mask) are shown as Figure 6. These results show that PADMM algorithm has high performance of image denoising.Figure 3
Denoising results on a real color image by all competing methods.
(a)
Noisy image(b)
NLH(c)
NSNR(d)
WNNMFigure 4
The 15 test images used in image denoising experiments.Figure 5
Inpainting results on images Starfish by different methods (Random mask with 75% missing values).
(a)
Input image(b)
Ground truth(c)
WNNM (PSNR: 27.11 dB)(d)
McS: (PSNR: 27.57 dB)Figure 6
Inpainting results on images Monarch by different methods (Text mask).
(a)
Input image(b)
Ground truth(c)
WNNM (PSNR: 32.85 dB)(d)
McS: (PSNR: 33.85 dB)The quantitative comparisons of all methods are reported in Table1, which indicates that the PADMM algorithm can improve the performance and yield the best quantitative results. Meanwhile, the paper tests the performance of NLH, KSVD, BM3D and WNNM algorithms on PSNR and SSIM of 15 pictures in the different value of σn. The experimental results are shown in Tables 2 and 3. The results show that with the increase of σn value, the PSNR and SSIM of each picture gradually decrease, but the performance of these algorithms is significantly worse than PADMM algorithm.Table 1
The quantitative performance of Figure2 for the two compared methods with the corresponding noise setting, i.e., missing proportion for impulse noise Ni and the a for Gaussian noise Ng.
ImageNoise settingASInpaint [6]PADMMPSNRSSIMPSNRSSIMLena50% missingσn = 1521.820.634722.780.6650Cameraman30% missingσn = 1919.130.543519.600.5630Lena10% missingσn = 1524.290.717524.490.7240Phantom10% missingσn = 0.449.250.965749.350.9666Satellite10% missingσn = 1522.410.644622.540.7662Table 2
Denoising results (PSNR, SSIM) by competing methods on 15 test images.
NLHKSVDBM3DWNNMPSNRSSIMPSNRSSIMPSNRSSIMPSNRSSIMσn =15C.man32.00540.900131.40740.892631.91520.900732.17680.9036House35.28320.898134.3080.875834.94470.890735.15330.8909Peppers32.94160.908732.20620.898732.70170.906432.9740.9098Straws28.57210.928528.32310.926228.56180.931729.13960.9396Leaves32.09510.969730.88060.956231.72330.965932.82660.9735StarFish31.41400.900730.73770.893131.14580.900731.82550.9081Monarch32.10650.938831.38640.929131.85970.936032.71780.9424Airplane31.40840.902530.79550.893731.07680.899531.40040.9029Ma31.98380.865731.49100.854431.92930.866732.1230.8701J.Bean36.16620.970835.51880.963535.70380.967836.56420.9735Couple31.94140.869231.44980.854032.10870.876132.18180.8746Parrot31.38260.891931.03670.891531.37600.894431.60710.8968Barbara32.83840.921632.42140.909933.11410.922833.61140.9277Boat31.99440.848331.7033.841032.14010.853432.28000.8549Lena34.19020.895333.74100.885134.27160.895034.38220.8973σn =30C.man28.86070.840228.01580.815728.63770.836628.78270.8399House32.45700.850231.17540.830532.08710.847432.5510.8523Peppers29.57430.854028.7910.840729.27990.850029.49160.8567Straws24.42530.803824.30210.796424.83580.83225.24570.8497Leaves28.12280.933326.96650.911827.81110.927528.60830.9389StarFish27.89240.833127.23250.813027.65350.828628.06890.8357Monarch28.72200.889128.01090.871728.36410.881728.91350.8926Airplane27.97360.843927.25950.825227.55920.836627.81760.8438Ma28.99990.780328.32440.751428.85970.779828.97980.7818J.Bean32.04280.932131.61620.922731.96690.935032.50050.9438Couple28.97260.796428.97260.746328.86910.794328.96790.7945Parrot28.32000.831927.55510.818628.11840.831328.32020.8346Barbara29.83740.874628.60060.822629.81360.868230.30860.8812Boat29.16630.778528.40930.744029.11720.779129.22620.7792Lena31.31940.847430.41920.824531.26210.844331.43150.8502σn =50C.man26.34660.790325.73610.745126.11300.782226.41760.7848House30.51780.830627.94680.760229.69390.811630.33250.8231Peppers27.05240.806326.03680.769526.68340.793226.91230.8008Straws21.69290.630821.32630.580022.28740.689822.72610.7305Leaves25.35670.890724.21360.857124.68180.867725.47210.8925StarFish25.21000.749224.38760.712525.04430.742925.43270.7596Monarch26.29020.835425.16630.793725.81860.819626.31700.8350Airplane25.36110.782124.62000.743125.10220.771625.42440.7850Ma26.87620.703126.03080.662526.80810.705126.93730.7090J.Bean29.69370.911428.17450.852629.25950.899829.63510.9098Couple26.46040.705725.30370.630926.46380.706426.64360.7135Parrot25.98560.78125.41870.754025.89840.780426.09260.7847Barbara27.48330.812825.56000.719127.22540.794227.78870.8199Boat26.86250.703225.93570.656926.78080.705026.96930.7083Lena29.20230.806927.87010.760629.05020.798929.25120.8059σn =75C.man24.75290.749223.18040.655024.32540.733424.55200.7353House28.53250.796325.33690.680027.50850.764028.23780.7887Peppers25.16320.751023.51630.683524.73410.736424.91520.7418Straws20.44220.521419.27920.360420.55880.544021.00390.6040Leaves23.00930.830620.76230.729622.488920.807023.05940.8350StarFish23.22200.664222.10930.602723.27460.666723.47200.6801Monarch24.49820.783122.90800.718323.90730.755324.30750.7754Airplane23.69140.728922.32930.661123.47490.714523.74070.7302Ma27.43660.882425.43100.761627.21530.856527.42330.8707J.Bean27.43660.882425.4310.761627.21530.856527.42330.8707Couple24.91900.642223.57760.551124.69880.625724.85770.6369Parrot24.37940.742123.37860.682024.18560.730224.36980.7410Barbara25.63790.743023.04970.603225.12380.710825.81230.7486Boat25.30210.648723.97560.579525.11960.640725.29510.6465Lena27.59960.770625.74840.693927.25690.751027.54320.7657σn =100C.man23.53290.705021.67120.576223.08130.692223.35790.6968House26.72030.758923.67510.618625.87230.719626.66400.7536Peppers23.80280.707621.82890.623823.39460.687623.44850.6978Straws19.40430.400418.38010.265519.43030.422319.68780.4537Leaves21.59630.784418.28960.593420.90950.748121.56580.7884StarFish22.16770.615820.96690.539722.09770.605122.22630.6170Monarch23.14980.732220.55680.615422.51850.701722.95000.7257Airplane22.68910.695320.84160.577322.10940.671022.55290.6854Ma24.4735.613923.38940.548224.22370.597524.35840.6048J.Bean26.18010.856123.69840.691125.80100.817526.02930.8337Couple23.74070.583522.61830.499223.51070.566123.55970.5702Parrot23.13670.705321.83620.682022.95930.689223.18630.7045Barbara24.47120.696021.88340.533223.62430.642624.10980.6862Boat24.2023.607322.78060.524623.97030.593224.10980.5981Lena26.45090.742824.35080.638725.95480.708526.21270.7256Table 3
The average PSNR & SSIM values by comprting methods on the 15 test images: The best results are highlighted in bold.
σn =15σn = 30σn =50σn =75σn =100PSNRSSIMPSNRSSIMPSNRSSIMPSNRSSIMPSNRSSIMBM3D32.30480.907228.94900.844826.46070.777924.61260.712023.29720.6575KSVD31.82710.897728.30990.822325.58180.733223.26840.637221.78450.5685NLH32.42160.907328.30990.845926.69280.782624.93420.726923.49990.6747WNNM32.73090.911629.28090.851726.82350.790824.93600.730023.60130.6761


## 5. Conclusions

In this paper, we present a novel optimization model and design the corresponding algorithm to address the challenging blind inpainting task with mixture noise. There are three main contributions in this work: 1) The model intetrates aℓ0 sparse regularization to the binary mask, the total variation term to the underlying clean image and a ℓ2 regularization to describe the fidelity term; 2), Theproximal based alternating direction method of multipliers (PADMM) method was utilized and implemented to solve the optimization problem; 3) Experiments on some simulated examples with complex mixture noise are implemented, and the visual and quantitative results demonstrate the proposed method outperforms the other method.



---

*Source: 3180612-2022-09-30.xml*