# Weighted Nuclear Norm Minimization Based Tongue Specular Reflection Removal

Authors: Zhenchao Cui; Dongyu Zhang; Kuanquan Wang; Hongzhi Zhang; Naimin Li; Wangmeng Zuo
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

In computational tongue diagnosis, specular reflection is generally inevitable in tongue image acquisition, which has adverse impact on the feature extraction and tends to degrade the diagnosis performance. In this paper, we proposed a two-stage (i.e., the detection and inpainting pipeline) approach to address this issue: (i) by considering both highlight reflection and subreflection areas, a superpixel-based segmentation method was adopted for the detection of the specular reflection areas; (ii) by extending the weighted nuclear norm minimization (WNNM) model, a nonlocal inpainting method is proposed for specular reflection removal. Experimental results on synthetic and real images show that the proposed method is accurate in detecting the specular reflection areas and is effective in restoring tongue image with more natural texture information of tongue body.


---

## Body

## 1. Introduction

In traditional Chinese medicine (TCM), the practitioners observe the color, shape, texture, and coating characteristics of tongue to evaluate the healthy condition of a person. Because of its convenience and effectiveness, for thousands of years, traditional tongue diagnosis has been very popular in the countries of East Asia, especially in China, Korea, Japan, and so forth [1–3].Traditional tongue diagnosis, however, is a skill that requires years of training to master, and the diagnosis result is highly dependent on the practitioners’ personal experience. That is, for a specific patient, the diagnosis results given by several practitioners may be distinctly different. In summary, the limitations of traditional tongue diagnosis greatly restrict its applications in modern medicine.Recently, with the progress in image processing and pattern recognition, computational tongue diagnosis has received considerable interests to make tongue diagnosis more objective and automated. Many tongue diagnosis systems have been developed, as hyperspectral-based tongue diagnosis system [4], 3D tongue acquisition system [5], and so forth [6–10]. Different features of tongue, for example, shape [11], color [8, 12, 13], texture, coating [14, 15], and sublingual vein [16], have been suggested for the representation of tongue images. Many methods, such as Bayesian network [7], support vector machine [17], decision trees, and naive Bayes [18], have been proposed for tongue image analysis.However, because of the saliva on the tongue body, specular reflection generally is inevitable for the existing tongue image acquisition systems [19]. Figure 1 shows one typical semiclosed system for tongue image acquisition [10] and several tongue images with specular reflections. The specular reflection on the tongue image would adversely affect many tongue texture analysis results, for example, tongue fur, tongue texture, and tongue color detection. To alleviate the effect of specular reflection, one may develop some approaches to improve the robustness of the existing feature extraction and classification methods, but the natural strategy is to detect and repair the specular reflection areas (i.e., the detection and inpainting pipeline).Figure 1
Tongue image acquirement device and typical tongue images with specular reflections.By far, several methods have been proposed for the detection and repairing of the specular reflection areas in tongue images. These methods, however, are limited in both detection and inpainting. In the detection stage, as shown in Figure2, a typical specular reflection region in tongue image includes both the highlight reflections and the subreflections, while the existing detection methods usually only considered the highlight reflections [16, 20, 21] or just adopted some trivial strategies (e.g., morphological operators) to cope with the subreflections [21]. In the repairing stage, bilinear interpolation [20] and total-variation- (TV-) based methods [21] were applied for inpainting the tongue specular reflection regions. These methods, however, only considered the local smoothness of the image and were not effective when the reflection areas are large.Figure 2
A tongue image with specular reflection area: (a) one of the highlight areas is marked by the green box and (b) the enlarged version of the subimage in the green box. It is observed that the pixels along the highlight area are usually darker than the usual pixels.
(a)
(b)In this paper, we adopted the detection and inpainting pipeline and proposed a novel method for the removal of specular reflection areas in tongue image. In the detection stage, adaptive thresholding and superpixel segmentation methods were applied for the highlight reflections and subreflections detection. Referring to the location of the initial highlight reflections obtained via thresholding, subreflections were defined as the surrounding superpixels with lower illumination than that of normal pixels of tongue body. The highlight reflections together with the subreflections were regarded as the final detection result of specular reflection areas.In the inpainting stage, based on the homogenous property of tongue image, we proposed a nonlocal inpainting approach, that is, weighted nuclear norm minimization- (WNNM-) based tongue specular reflections removal. For a single small patch of a tongue image, there were many nonlocal similar patches across the whole image. If the patch and its nonlocal similar ones are stacked into a matrix, it was reasonable to assume that the matrix should be of low rank. Thus by solving a low rank matrix completion problem with WNNM framework, we could fill the specular reflection areas with the texture information of tongue.This paper focused on the tongue image specular reflections removal, whose contributions are of twofold: (i) detection of highlight reflection and subreflection areas, and (ii) WNNM-based inpainting. First, we analyzed the adverse impact of subreflections, which to the best of our knowledge receive little attention in tongue specular reflection removal. To address this, we proposed a superpixel based specular reflection detection method which can effectively detect the highlight reflection together with subreflection areas, while affecting little the normal pixels around the subreflection areas. Second, we proposed a WNNM-based nonlocal inpainting method for the removal of specular reflections. The proposed method converts the inpainting problem to an unconstrained optimization problem which could be iteratively solved by using standard WNNM. Compared with other inpainting approaches, our method can obtain more natural textures information of tongue, especially for large reflection area, which could promote both the PSNR value and the visual effect of the restored tongue image.The remainder of the paper is organized as follows: Section2 describes the superpixel based specular reflections detection method. Section 3 presents the WNNM-based nonlocal inpainting method for specular reflection removal of tongue image. The experimental results are provided in Section 4, and finally Section 5 concludes this paper.


## 2. Tongue Image Specular Reflection Detection

In this section, we firstly described the characteristics of specular reflections of tongue image and then proposed our superpixel-based specular reflections detection method.Figure2(a) shows a typical example of tongue image with the specular reflection areas, while its enlarged subimage is shown in Figure 2(b). As shown in Figure 2(b), the pixels of highlight reflection areas always have higher illumination and less saturation values than other pixels, while the subreflections consist of many abnormal dark pixels with lower illumination around the highlight reflection areas. In [21], Jia et al. proposed to use morphological dilation to detect the subreflection pixels around the highlight reflection area. However, for large area of highlight reflection, because the surrounding abnormal pixels of subreflection area are not uniform, the morphological dilation often ended with inaccurate detection result by including much more normal pixels of tongue body, as shown in Figures 5(c) and 5(d).To solve this problem, we proposed a two-stage method for the detection of specular reflection areas. First, utilizing hue-saturation-illumination (HSI) color model, an initial detection result was obtained for highlight reflection area. Then, the subreflection areas were further detected via superpixel segmentation.


### 2.1. Detection of Highlight Reflection Area

As for our observation, the highlight reflection areas of tongue image often have higher illumination values and less saturation values. Thus, it is natural to transform the tongue image into the HSI color space, and thus the hue and saturation were adopted to describe color. The following equations were adopted to convert the tongue image into the HSI color space:(1)I=13R+G+B,H=θ,G>B,2π-θ,G≤B,S=1-3×min⁡R,G,BR+G+B,where θ=cos-11/2R-G+R-B/R-G2+R-BG-B, R, G, and B are the three channels of RGB color space, respectively.Figure3(a) shows a typical tongue image with specular reflection, while Figures 3(b) and 3(c) show the I channel and the S channel of Figure 3(a) in the HSI color space, respectively. For each channel, a threshold was used to check whether a pixel should belong to the highlight reflection area or not. The thresholds TI for I channel and Ts for the S channel were adaptively obtained by(2)TI=imax⁡+imax⁡-imean2imax⁡-imin⁡,Ts=smin⁡+smean-smin⁡2smax⁡-smin⁡,where imax, imin, and imean denote, respectively, the maximum, minimum, and mean value of all illumination values in a tongue image, and smax, smin, and smean denote, respectively, the maximum, minimum, and mean value of all saturation values in the tongue image. Finally, the highlight reflection areas image HR could be obtained by using the following criterion:(3)HRx,y≔x,y∣Ix,y>TI∩Sx,y<Ts,where Ix,y and Sx,y denote the I channel and S channel of pixel x,y, respectively. Figure 3(d) shows an example of the detection result of highlight reflection areas in a typical tongue image.Figure 3
Tongue image in the HSI color space: (a) the typical tongue image, (b)I channel of tongue image, (c) S channel of tongue image, and (d) the initial detection of reflection.
(a)
(b)
(c)
(d)


### 2.2. Superpixel Based Detection of Subreflection Area

For a tongue image, by applying oversegmentation, a large number of small regions, that is, superpixels, were obtained. Subreflections were defined as the superpixels that surround the highlight reflection and which had lower illumination values than that of normal pixels of tongue body. Utilizing the isotropic characteristics of superpixels [22, 23], we could naturally find the subreflections and avoid bringing in much normal pixels of tongue body.Superpixels can be obtained by oversegmenting a tongue image with any reasonable existing segmentation algorithms. In this paper, a graph-based segmentation method [22] was adopted. For each tongue image, an undirected graph was defined with the pixels/regions as nodes, connected by the edges of its neighborhood. A nonnegative weight we was used to edge e to measure the dissimilarity of its corresponding nodes. In the beginning, each pixel was a node. The graph-based segmentation algorithm gradually merged similar regions/nodes to the same superpixel. The merging process was motivated by the internal variation int⁡(R) of region R defined as(4)int⁡R=maxe∈MSTR⁡we+kR,where R is the size of region R, MST(R) is the minimum spanning tree of R, and k is a nonnegative parameter related to the size or the numbers of superpixels. For any two different regions R1 and R2, there will be at least one edge between them, with its weight higher than Int(R1) and Int(R2) (if there is no edge between the two regions, the edge weight is regarded as +∞). Otherwise, the two regions were merged into one new region, and the internal variation of this compound region was updated. Finally, for a tongue image, we could obtain a series of superpixels. For the detail of this algorithm, please refer to [22].After oversegmentation, we could obtain many superpixels of tongue image which could be roughly grouped into three categories, highlight reflections superpixels, subreflection superpixels, and normal tongue body superpixels. As the subreflections were commonly around the highlight reflection, we could use the following strategy to locate them. First, as shown in Figure4, with the initial highlight reflection detected in Section 2.1, we could obtain an interest area, a rectangle box with green border, and a circle band with red dash line border around each highlight reflection using morphological dilation operation. Then, for each superpixel in the circle band, we determinate its category with (5)superpixel∈Sub-reflectionmsp<mtb×γmin⁡,Highlight reflectionmsp≥mtb×γmax⁡,Tongue bodyotherwise,where msp means pixels value of single superpixel in the band, mtb is the mean value of normal pixels of tongue body in the interest area, and γmin and γmax are two nonnegative parameters which were empirically set to 1 and 1.15 in this paper. The width of the circle band was adaptively adjusted according to the size of highlight reflection, while for large highlight reflection the width of corresponding circle was wider. By classifying all the candidate superpixels in the circle band using Formula (5), we could detect all the subreflections and update the highlight reflections at the same time.Figure 4
Sketch map of subreflection detection: (a) tongue image with specular reflections where green box was the select interest area, and (b) the enlarged interest area in (a) where the candidate superpixels were searched in the circle band with red dash line border.
(a)
(b)Figure 5
The detection results of specular reflections by using different values ofk, (a) k=0.01, (b) k=0.05, (c) k=0.10, and (d) k=0.15.
(a)
(b)
(c)
(d)Parameterk in (4) is one parameter that indirectly affects the granularity of the final segmentation. With a larger k, it usually leads to larger regions but also reduces a higher likelihood of missing segmentation boundaries, while a smaller k often leads to a consistent oversegmentation. Figure 5 shows the detection results by setting k to 0.01, 0.05, 0.10, and 0.15, respectively. The mask map in Figure 5(b) can cover the highlight reflections and subreflections, while excluding the normal pixels around subreflections. In Figure 5(a), the subreflections areas are not detected, while in Figures 5(c) and 5(d) many normal pixels are misclassified to the subreflection area. Our experiments show that, for most of tongue images, satisfactory subreflection detection results are obtained with k∈0.05,0.07.Finally, the highlight reflections and subreflections were combined together as the final detection results of specular reflection areas for inpainting. Figure6(b) shows the final detection result of the specular reflection area of Figure 6(a). For comparison, Figures 6(c) and 6(d) show the results of morphological dilation based method [21] by using disk-shaped structuring element with 1 × 1 pixel and 5 × 5 pixels, respectively. Compared with the enlarged subimages in Figures 6(b) with 6(c) and 6(d), we can see that the mask map in Figure 6(c) does not cover all the subreflections, especially in the part inside the red circle, while the mask map in Figure 6(d) eroded too much normal pixels. Generally, the proposed superpixel-based method could accurately detect the highlight reflection and subreflection areas while compared with morphological dilation.Figure 6
Comparison of specular reflection area detection results: (a) the original tongue image, (b) the detection result by using the proposed superpixel-based method, (c) the detection result of morphological dilation method by using disk-shaped structuring element with 1 × 1 pixel, and (d) 5 × 5 pixels.
(a)
(b)
(c)
(d)


## 2.1. Detection of Highlight Reflection Area

As for our observation, the highlight reflection areas of tongue image often have higher illumination values and less saturation values. Thus, it is natural to transform the tongue image into the HSI color space, and thus the hue and saturation were adopted to describe color. The following equations were adopted to convert the tongue image into the HSI color space:(1)I=13R+G+B,H=θ,G>B,2π-θ,G≤B,S=1-3×min⁡R,G,BR+G+B,where θ=cos-11/2R-G+R-B/R-G2+R-BG-B, R, G, and B are the three channels of RGB color space, respectively.Figure3(a) shows a typical tongue image with specular reflection, while Figures 3(b) and 3(c) show the I channel and the S channel of Figure 3(a) in the HSI color space, respectively. For each channel, a threshold was used to check whether a pixel should belong to the highlight reflection area or not. The thresholds TI for I channel and Ts for the S channel were adaptively obtained by(2)TI=imax⁡+imax⁡-imean2imax⁡-imin⁡,Ts=smin⁡+smean-smin⁡2smax⁡-smin⁡,where imax, imin, and imean denote, respectively, the maximum, minimum, and mean value of all illumination values in a tongue image, and smax, smin, and smean denote, respectively, the maximum, minimum, and mean value of all saturation values in the tongue image. Finally, the highlight reflection areas image HR could be obtained by using the following criterion:(3)HRx,y≔x,y∣Ix,y>TI∩Sx,y<Ts,where Ix,y and Sx,y denote the I channel and S channel of pixel x,y, respectively. Figure 3(d) shows an example of the detection result of highlight reflection areas in a typical tongue image.Figure 3
Tongue image in the HSI color space: (a) the typical tongue image, (b)I channel of tongue image, (c) S channel of tongue image, and (d) the initial detection of reflection.
(a)
(b)
(c)
(d)


## 2.2. Superpixel Based Detection of Subreflection Area

For a tongue image, by applying oversegmentation, a large number of small regions, that is, superpixels, were obtained. Subreflections were defined as the superpixels that surround the highlight reflection and which had lower illumination values than that of normal pixels of tongue body. Utilizing the isotropic characteristics of superpixels [22, 23], we could naturally find the subreflections and avoid bringing in much normal pixels of tongue body.Superpixels can be obtained by oversegmenting a tongue image with any reasonable existing segmentation algorithms. In this paper, a graph-based segmentation method [22] was adopted. For each tongue image, an undirected graph was defined with the pixels/regions as nodes, connected by the edges of its neighborhood. A nonnegative weight we was used to edge e to measure the dissimilarity of its corresponding nodes. In the beginning, each pixel was a node. The graph-based segmentation algorithm gradually merged similar regions/nodes to the same superpixel. The merging process was motivated by the internal variation int⁡(R) of region R defined as(4)int⁡R=maxe∈MSTR⁡we+kR,where R is the size of region R, MST(R) is the minimum spanning tree of R, and k is a nonnegative parameter related to the size or the numbers of superpixels. For any two different regions R1 and R2, there will be at least one edge between them, with its weight higher than Int(R1) and Int(R2) (if there is no edge between the two regions, the edge weight is regarded as +∞). Otherwise, the two regions were merged into one new region, and the internal variation of this compound region was updated. Finally, for a tongue image, we could obtain a series of superpixels. For the detail of this algorithm, please refer to [22].After oversegmentation, we could obtain many superpixels of tongue image which could be roughly grouped into three categories, highlight reflections superpixels, subreflection superpixels, and normal tongue body superpixels. As the subreflections were commonly around the highlight reflection, we could use the following strategy to locate them. First, as shown in Figure4, with the initial highlight reflection detected in Section 2.1, we could obtain an interest area, a rectangle box with green border, and a circle band with red dash line border around each highlight reflection using morphological dilation operation. Then, for each superpixel in the circle band, we determinate its category with (5)superpixel∈Sub-reflectionmsp<mtb×γmin⁡,Highlight reflectionmsp≥mtb×γmax⁡,Tongue bodyotherwise,where msp means pixels value of single superpixel in the band, mtb is the mean value of normal pixels of tongue body in the interest area, and γmin and γmax are two nonnegative parameters which were empirically set to 1 and 1.15 in this paper. The width of the circle band was adaptively adjusted according to the size of highlight reflection, while for large highlight reflection the width of corresponding circle was wider. By classifying all the candidate superpixels in the circle band using Formula (5), we could detect all the subreflections and update the highlight reflections at the same time.Figure 4
Sketch map of subreflection detection: (a) tongue image with specular reflections where green box was the select interest area, and (b) the enlarged interest area in (a) where the candidate superpixels were searched in the circle band with red dash line border.
(a)
(b)Figure 5
The detection results of specular reflections by using different values ofk, (a) k=0.01, (b) k=0.05, (c) k=0.10, and (d) k=0.15.
(a)
(b)
(c)
(d)Parameterk in (4) is one parameter that indirectly affects the granularity of the final segmentation. With a larger k, it usually leads to larger regions but also reduces a higher likelihood of missing segmentation boundaries, while a smaller k often leads to a consistent oversegmentation. Figure 5 shows the detection results by setting k to 0.01, 0.05, 0.10, and 0.15, respectively. The mask map in Figure 5(b) can cover the highlight reflections and subreflections, while excluding the normal pixels around subreflections. In Figure 5(a), the subreflections areas are not detected, while in Figures 5(c) and 5(d) many normal pixels are misclassified to the subreflection area. Our experiments show that, for most of tongue images, satisfactory subreflection detection results are obtained with k∈0.05,0.07.Finally, the highlight reflections and subreflections were combined together as the final detection results of specular reflection areas for inpainting. Figure6(b) shows the final detection result of the specular reflection area of Figure 6(a). For comparison, Figures 6(c) and 6(d) show the results of morphological dilation based method [21] by using disk-shaped structuring element with 1 × 1 pixel and 5 × 5 pixels, respectively. Compared with the enlarged subimages in Figures 6(b) with 6(c) and 6(d), we can see that the mask map in Figure 6(c) does not cover all the subreflections, especially in the part inside the red circle, while the mask map in Figure 6(d) eroded too much normal pixels. Generally, the proposed superpixel-based method could accurately detect the highlight reflection and subreflection areas while compared with morphological dilation.Figure 6
Comparison of specular reflection area detection results: (a) the original tongue image, (b) the detection result by using the proposed superpixel-based method, (c) the detection result of morphological dilation method by using disk-shaped structuring element with 1 × 1 pixel, and (d) 5 × 5 pixels.
(a)
(b)
(c)
(d)


## 3. The WNNM-Based Nonlocal Inpainting Method

In this section, we proposed an examplar-based inpainting method, that is, WNNM-based inpainting, for tongue specular reflection removal. The examplar-based methods can be traced back to 1999 [24] and had been widely adopted for image inpainting [25–31]. The arising of nonlocal self-similarity methods [32] further triggered the development of examplar-based inpainting approaches. Most existing methods, however, fill the holes based on the best matched patch or the mean of the nonlocal similar patches, while the second-order information usually is neglected in inpainting. To remedy this, a low rank based inpainting model [33] is proposed to synthesize the missing regions, where tensor trace norm is adopted as the low rank regularizer. Besides, the WNNM model [34] generally is superior to other low rank models, for example, trace norm or nuclear norm, for image denoising. This motivates us to employ WNNM for tongue specular reflection removal, resulting in the proposed WNNM-based inpainting model.Given a small patch on the tongue image, there are many nonlocal similar patches across the whole image. If we stretch each patch to a vector and stack all the vectors in a matrix, it is intuitive that the matrix should be of low rank. Based on this assumption, the tongue image inpainting work could be regarded as a low rank matrix completion problem. By utilizing a newly proposed weighted nuclear norm model, we further showed that tongue image inpainting could be well solved by iteratively performing weighted nuclear norm minimization (WNNM).In the following subsection, a brief review on matrix completion was firstly introduced. Then, WNNM was described. Finally, the WNNM-based nonlocal inpainting method was proposed for specular reflections removal of tongue image.


### 3.1. Matrix Completion

Matrix completion aims to recover a low rank matrix from incomplete samples of its entries, which has received considerable attention in many areas of engineering and science [35, 36], for example, the well-known Netflix problem [37]. Matrix completion can be cast as the following minimization problem:(6)minX∈Rm×n⁡rank⁡X:PΩX=PΩB,where X∈Rm×n is recovered from B∈Rm×n, Ω denotes the set of position of known entries in matrix B, and PΩ· is a linear operator with the following definition: for any matrix D∈Rm×n,(7)PΩD=Di,ji,j∈Ω,0i,j∉Ω.The model in (6) is nonconvex and is NP-hard problem which is nontrivial to solve. Therefore, convex relaxation of (6) usually was adopted with the following formulation:(8)minX∈Rm×n⁡X∗:PΩX=B,where X∗=∑iσiX is the nuclear norm of X, and σiX are the ith singular values of X. The problem in (8) can be approximated by the following unconstrained convex minimization problem:(9)minX∈Rm×n⁡12PΩX-BF2+λX∗,where ·F denotes the Frobenius norm, and λ is a tradeoff parameter. When λ goes infinite, the solution of (9) will be the same as that of (8). As an unconstrained optimization problem, (9) could be solved by the iterative shrinkage algorithm [38] and the APG method [39].


### 3.2. Weighted Nuclear Norm Minimization

The iterative shrinkage algorithm of the model in (9) usually involves solving the nuclear norm minimization (NNM) problem as(10)Xk+1=argminX⁡X-Y∗F2+λX∗,where X,Y∗∈Rm×n, Y∗=Xk-1/Lf∇fXk, fXk=1/2PΩXk-BF2, and λ>0.Cai et al. [40] proved that the NNM problem of (10) can be easily solved by the singular value thresholding (SVT) method; that is,(11)Xk+1=USλΣVT,where Xk+1 is the solution to (10), Y∗=UΣVT is the singular value decomposition (SVD) of Y∗, Σ=diagσ1,σ2,…,σn is the diagonal matrix with singular values, and SλΣ is the soft-thresholding function on Σ as(12)SλΣi=max⁡σi-λ,0.Recently, Gu et al. [34] suggested that, as the soft-thresholding operator SλΣ shrinks each singular value with the same λ in order to pursue the convex property, it ignores the prior knowledge about the singular values. Compared with the small singular values, the larger ones are generally associated with the major projection orientation of the matrix in the lower subspace and they should be shrunk less to preserve the major data components. Thus, they extended the standard nuclear norm to the weighted nuclear norm:(13)Xw,∗=∑iwiσiX1,where w=w1,…,wn and wi≥0 is a nonnegative and assigned to the singular value σiX and the weighted nuclear norm minimization (WNNM) problem could be formulated as(14)X^=argminX⁡Y∗-XF2+λXw,∗.Generally, the model in (14) is not convex, but Xie et al. [41] further proved that if the weights, assigned to the singular vales, are arranged in ascending orders, the globally optimal solution could be obtained with the following theorem.Theorem 1 (see [41]).
If the weightsw satisfy 0≤w1≤w2≤⋯≤wn, then the WNNM problem (16) has a globally optimal solution:(15)X^=USwΣVT,where Y∗=UΣVT denotes the SVD of matrix Y∗, Σ is the diagonal matrix with singular values arranged in descending order; that is, σ1≥σ2≥⋯≥σn, and SwΣ is the soft-thresholding operator:(16)SwΣi=max⁡σi-wi,0.WNNM has been applied to image denoising and achieved better results than state of the arts, such as LSSC [42] and BM3D [43], by PSNR and SSIM values. Moreover, WNNM is effective in preserving the local structures of images and generating less visual artifacts.


### 3.3. Tongue Image Inpainting by Using WNNM

In this section, we proposed a WNNM-based nonlocal inpainting method for repairing tongue image specular reflection areas. First, the WNNM-based inpainting model was introduced to utilize the nonlocal information of tongue image. Then, we described the proposed optimization algorithm by iteratively solving a series of standard WNNM problems. Finally, we analyzed the convergence of our algorithm.Given a tongue imageB with specular reflection Ω-, the proposed WNNM-based tongue inpainting model is formulated as(17)minX∈Rm×n⁡FX=12PΩX-BF2+λXw,∗,where X is the inpainted image, Ω denotes the nonreflection area, and PΩ is a linear projection operator defined in (7). The model in (17) can be obtained by substituting the standard nuclear norm with weighted nuclear norm in (9). Because of the introduction of the projection operator PΩ, the model in (17) cannot be directly solved by WNNM algorithm in [34]. Thus, we adopt the following iterative shrinkage algorithm. In each iteration, we consider the following surrogate function [44] of F(X) at a given point Xk:(18)QX,Xk≔fXk+∇fXk,X-Xk+Lf2X-XkF2+λXw,∗,where fXk=1/2PΩXk-BF2 and Lf is the Lipschitz constant of ∇fXk. QX,Xk satisfies that (i) QX,Xk≥FX for any X and (ii) QXk,Xk=FXk. By ignoring the constant term, the minimization of (18) can be formulated as(19)arg⁡minX⁡QX,Xk≐Lf2X-YkF2+λXw,∗,where Yk=Xk-1/LfPΩTPΩXk-B. Equation (19) has the same form of (14), which could be solved by using WNNM. Finally, we can get the minimizer of (17) by iteratively solving (19) with proximal gradient as described in Algorithm 1.Algorithm 1:WNNM-based nonlocal inpainting method.
Input: The tongue image B, iteration number N, and Lf
(1)    Initialize:  X0=B, Y0=X0-(1/Lf)PΩT(PΩ(X0)-B)
(2)    for  k=1 : N  do
(3)     for  each patch pj in Yk-1  do
(4)      Find similar patches matrix  Zj
(5)      Estimate weight vector  w
(6)      Singular value decomposition [U,Σ,V]=SVD(Zj)
(7)      Estimate patch Z^j=USw(Σ)VT and p^j
(8)     end for
(9)     Aggregate all the single patches p^j to form the image Xk
(10)  Yk=Xk-(1/Lf)PΩTPΩXk-B
(11) end for
Output: X=XN.There are many repeated patterns in the tongue image which are useful for tongue image inpainting. In order to take advantage of this nonlocal information, we split the tongue image into many 5 × 5 pixels patches. For a single patchpj, many nonlocal similar patches could be found across the whole image. Stretching each similar patch to a vector and stacking them into a matrix Zp, it is intuitively that Zp should be of low rank. By using (19), we could get the estimation of Zp as(20)Z^p=argminZ⁡Lf2Z-ZpF2+λZw,∗,which could be solved by the generalized soft-thresholding method in Theorem 1. Specifically, the weight wi assigned to the ith singular value, σiZp, of Zp as wi=cn/σiZp+ε, where c>0 is a constant, n is the number of similar patches in Zp, and ε=10-6 is to avoid dividing by zero. In this paper c is set to 10−3,  λ is set to 0.1, Lf and N are set to 2 and 5, respectively. After that, the patch pj can be inpainted by finding the most similar one to it in Z^p. Finally, by aggregating all the patches p^j together, the whole image X can be updated.


### 3.4. Convergence of the WNNM-Based Nonlocal Inpainting Method

In this section, we proved the convergence of WNNM-based inpainting method which could be summarized as in the following theorem.Theorem 2.
To solveminX∈Rm×n⁡FX=1/2PΩX-BF2+λXw,∗, one iteratively performs the two steps(21)Yk=Xk-PΩTPΩXk-B,(22)Xk+1=argminx⁡X-YkF2+λXw,∗and has (23)FXk+1≤FXk.Proof.
For image inpainting, we haveLf=1. Define the surrogate function Q(X,Xk) as(24)QX,Xk=12PΩXk-BF2+12I-PΩTPΩXk-X,Xk-X+λXw,∗.Since PΩ2≤1, I-PΩTPΩ is positive semidefinite, where I is the identity matrix. Thus we can obtain that(25)QX,Xk≥QX,X=fX.Given Xk, with the surrogate function in (24), we have(26)minX⁡QX,Xk=minX⁡12PΩXk-BF2+12I-PΩTPΩX-Xk,X-Xk+λXw,∗=minX⁡12X-I-PΩTPΩXk-PΩTBF2+λXw,∗+C=minX⁡12X-Xk-PΩTPΩXk-BF2+λXw,∗+C,where C is the constant term w.r.t. to X. Equation (26) is actually the iterative process of updating Yk using gradient descent as (27)Yk=Xk-PΩTPΩXk-B.
With WNNM, we can obtain the optimal solution of (26); Xk+1=arg⁡minX⁡QX,Xk. Then one can get(28)QXk+1,Xk≤QXk,Xk=FXk.From (25), one can derive that(29)QXk+1,Xk≥QXk+1,Xk+1=FXk+1.Combining (28) and (29), we have (30)FXk+1=QXk+1,Xk+1≤QXk+1,Xk≤QXk,Xk=FXk.From (30), we get FXk+1≤FXk.From Theorem2, the proposed algorithm can guarantee to decrease the loss function along with iterations until convergence to a fixed point.


## 3.1. Matrix Completion

Matrix completion aims to recover a low rank matrix from incomplete samples of its entries, which has received considerable attention in many areas of engineering and science [35, 36], for example, the well-known Netflix problem [37]. Matrix completion can be cast as the following minimization problem:(6)minX∈Rm×n⁡rank⁡X:PΩX=PΩB,where X∈Rm×n is recovered from B∈Rm×n, Ω denotes the set of position of known entries in matrix B, and PΩ· is a linear operator with the following definition: for any matrix D∈Rm×n,(7)PΩD=Di,ji,j∈Ω,0i,j∉Ω.The model in (6) is nonconvex and is NP-hard problem which is nontrivial to solve. Therefore, convex relaxation of (6) usually was adopted with the following formulation:(8)minX∈Rm×n⁡X∗:PΩX=B,where X∗=∑iσiX is the nuclear norm of X, and σiX are the ith singular values of X. The problem in (8) can be approximated by the following unconstrained convex minimization problem:(9)minX∈Rm×n⁡12PΩX-BF2+λX∗,where ·F denotes the Frobenius norm, and λ is a tradeoff parameter. When λ goes infinite, the solution of (9) will be the same as that of (8). As an unconstrained optimization problem, (9) could be solved by the iterative shrinkage algorithm [38] and the APG method [39].


## 3.2. Weighted Nuclear Norm Minimization

The iterative shrinkage algorithm of the model in (9) usually involves solving the nuclear norm minimization (NNM) problem as(10)Xk+1=argminX⁡X-Y∗F2+λX∗,where X,Y∗∈Rm×n, Y∗=Xk-1/Lf∇fXk, fXk=1/2PΩXk-BF2, and λ>0.Cai et al. [40] proved that the NNM problem of (10) can be easily solved by the singular value thresholding (SVT) method; that is,(11)Xk+1=USλΣVT,where Xk+1 is the solution to (10), Y∗=UΣVT is the singular value decomposition (SVD) of Y∗, Σ=diagσ1,σ2,…,σn is the diagonal matrix with singular values, and SλΣ is the soft-thresholding function on Σ as(12)SλΣi=max⁡σi-λ,0.Recently, Gu et al. [34] suggested that, as the soft-thresholding operator SλΣ shrinks each singular value with the same λ in order to pursue the convex property, it ignores the prior knowledge about the singular values. Compared with the small singular values, the larger ones are generally associated with the major projection orientation of the matrix in the lower subspace and they should be shrunk less to preserve the major data components. Thus, they extended the standard nuclear norm to the weighted nuclear norm:(13)Xw,∗=∑iwiσiX1,where w=w1,…,wn and wi≥0 is a nonnegative and assigned to the singular value σiX and the weighted nuclear norm minimization (WNNM) problem could be formulated as(14)X^=argminX⁡Y∗-XF2+λXw,∗.Generally, the model in (14) is not convex, but Xie et al. [41] further proved that if the weights, assigned to the singular vales, are arranged in ascending orders, the globally optimal solution could be obtained with the following theorem.Theorem 1 (see [41]).
If the weightsw satisfy 0≤w1≤w2≤⋯≤wn, then the WNNM problem (16) has a globally optimal solution:(15)X^=USwΣVT,where Y∗=UΣVT denotes the SVD of matrix Y∗, Σ is the diagonal matrix with singular values arranged in descending order; that is, σ1≥σ2≥⋯≥σn, and SwΣ is the soft-thresholding operator:(16)SwΣi=max⁡σi-wi,0.WNNM has been applied to image denoising and achieved better results than state of the arts, such as LSSC [42] and BM3D [43], by PSNR and SSIM values. Moreover, WNNM is effective in preserving the local structures of images and generating less visual artifacts.


## 3.3. Tongue Image Inpainting by Using WNNM

In this section, we proposed a WNNM-based nonlocal inpainting method for repairing tongue image specular reflection areas. First, the WNNM-based inpainting model was introduced to utilize the nonlocal information of tongue image. Then, we described the proposed optimization algorithm by iteratively solving a series of standard WNNM problems. Finally, we analyzed the convergence of our algorithm.Given a tongue imageB with specular reflection Ω-, the proposed WNNM-based tongue inpainting model is formulated as(17)minX∈Rm×n⁡FX=12PΩX-BF2+λXw,∗,where X is the inpainted image, Ω denotes the nonreflection area, and PΩ is a linear projection operator defined in (7). The model in (17) can be obtained by substituting the standard nuclear norm with weighted nuclear norm in (9). Because of the introduction of the projection operator PΩ, the model in (17) cannot be directly solved by WNNM algorithm in [34]. Thus, we adopt the following iterative shrinkage algorithm. In each iteration, we consider the following surrogate function [44] of F(X) at a given point Xk:(18)QX,Xk≔fXk+∇fXk,X-Xk+Lf2X-XkF2+λXw,∗,where fXk=1/2PΩXk-BF2 and Lf is the Lipschitz constant of ∇fXk. QX,Xk satisfies that (i) QX,Xk≥FX for any X and (ii) QXk,Xk=FXk. By ignoring the constant term, the minimization of (18) can be formulated as(19)arg⁡minX⁡QX,Xk≐Lf2X-YkF2+λXw,∗,where Yk=Xk-1/LfPΩTPΩXk-B. Equation (19) has the same form of (14), which could be solved by using WNNM. Finally, we can get the minimizer of (17) by iteratively solving (19) with proximal gradient as described in Algorithm 1.Algorithm 1:WNNM-based nonlocal inpainting method.
Input: The tongue image B, iteration number N, and Lf
(1)    Initialize:  X0=B, Y0=X0-(1/Lf)PΩT(PΩ(X0)-B)
(2)    for  k=1 : N  do
(3)     for  each patch pj in Yk-1  do
(4)      Find similar patches matrix  Zj
(5)      Estimate weight vector  w
(6)      Singular value decomposition [U,Σ,V]=SVD(Zj)
(7)      Estimate patch Z^j=USw(Σ)VT and p^j
(8)     end for
(9)     Aggregate all the single patches p^j to form the image Xk
(10)  Yk=Xk-(1/Lf)PΩTPΩXk-B
(11) end for
Output: X=XN.There are many repeated patterns in the tongue image which are useful for tongue image inpainting. In order to take advantage of this nonlocal information, we split the tongue image into many 5 × 5 pixels patches. For a single patchpj, many nonlocal similar patches could be found across the whole image. Stretching each similar patch to a vector and stacking them into a matrix Zp, it is intuitively that Zp should be of low rank. By using (19), we could get the estimation of Zp as(20)Z^p=argminZ⁡Lf2Z-ZpF2+λZw,∗,which could be solved by the generalized soft-thresholding method in Theorem 1. Specifically, the weight wi assigned to the ith singular value, σiZp, of Zp as wi=cn/σiZp+ε, where c>0 is a constant, n is the number of similar patches in Zp, and ε=10-6 is to avoid dividing by zero. In this paper c is set to 10−3,  λ is set to 0.1, Lf and N are set to 2 and 5, respectively. After that, the patch pj can be inpainted by finding the most similar one to it in Z^p. Finally, by aggregating all the patches p^j together, the whole image X can be updated.


## 3.4. Convergence of the WNNM-Based Nonlocal Inpainting Method

In this section, we proved the convergence of WNNM-based inpainting method which could be summarized as in the following theorem.Theorem 2.
To solveminX∈Rm×n⁡FX=1/2PΩX-BF2+λXw,∗, one iteratively performs the two steps(21)Yk=Xk-PΩTPΩXk-B,(22)Xk+1=argminx⁡X-YkF2+λXw,∗and has (23)FXk+1≤FXk.Proof.
For image inpainting, we haveLf=1. Define the surrogate function Q(X,Xk) as(24)QX,Xk=12PΩXk-BF2+12I-PΩTPΩXk-X,Xk-X+λXw,∗.Since PΩ2≤1, I-PΩTPΩ is positive semidefinite, where I is the identity matrix. Thus we can obtain that(25)QX,Xk≥QX,X=fX.Given Xk, with the surrogate function in (24), we have(26)minX⁡QX,Xk=minX⁡12PΩXk-BF2+12I-PΩTPΩX-Xk,X-Xk+λXw,∗=minX⁡12X-I-PΩTPΩXk-PΩTBF2+λXw,∗+C=minX⁡12X-Xk-PΩTPΩXk-BF2+λXw,∗+C,where C is the constant term w.r.t. to X. Equation (26) is actually the iterative process of updating Yk using gradient descent as (27)Yk=Xk-PΩTPΩXk-B.
With WNNM, we can obtain the optimal solution of (26); Xk+1=arg⁡minX⁡QX,Xk. Then one can get(28)QXk+1,Xk≤QXk,Xk=FXk.From (25), one can derive that(29)QXk+1,Xk≥QXk+1,Xk+1=FXk+1.Combining (28) and (29), we have (30)FXk+1=QXk+1,Xk+1≤QXk+1,Xk≤QXk,Xk=FXk.From (30), we get FXk+1≤FXk.From Theorem2, the proposed algorithm can guarantee to decrease the loss function along with iterations until convergence to a fixed point.


## 4. Experimental Results

In this section, we validated the effectiveness of the proposed method for tongue image inpainting. For better evaluation, both synthetic images and real images were used in the experiment. The inpainting results of the TV-based method [21] were used for comparison.


### 4.1. Experimental Results on Synthetic Images

In this section, we quantitatively compared the performance of the competing methods on synthetic images. The synthetic images were constructed as follows: we first detect the specular reflections of one real tongue image and then randomly put these reflections onto nonreflection tongue images to synthesize new images for inpainting. The benefit was that, on one hand, we could ensure that the topological structure of reflections was as natural as real ones. On the other hand, we could conveniently calculate the peak signal to noise ratio (PSNR) since we had the ground truth.We used two ways to detect the highlight reflections of tongue image. One was the morphological dilation based method in [20] and the other was our superpixel based method. Synthetic images with both kinds of reflections were used in the experiments of WNNM-based inpainting and TV-based inpainting, respectively.Figure7 shows the comparison experimental results of competing methods on synthetic tongue images with reflections detected by morphological dilation based method, while Figure 8 shows the results on the synthetic tongue images with reflections detected by the proposed superpixel based method.Figure 7
Comparision of inpainting results of synthetic tongue images with reflection detected by morphological dilation based method.
(a)
Ground truth
(b)
Synthetic tongue images
(c)
Results of WNNM-based method (PSNR: left: 30.37, right: 31.56)
(d)
Results of TV-based method (PSNR: left: 29.32, right: 30.06)Figure 8
Comparison of inpainting results of synthetic tongue images with reflection detected by superpixel based method.
(a)
Ground truth
(b)
Synthetic tongue images
(c)
Results of our WNNM-based method (PSNR: left: 32.29, right: 31.34)
(d)
Results of the TV-based method (PSNR: left: 31.39, right: 30.47)From Figures7, 8, and 9, we could find that for the same synthetic tongue image, in terms of PSNR, the proposed inpainting method obtained better result than the TV-based inpainting method. Furthermore, the TV-based inpainting method usually finds the pixels via minimizing total variation value, while the proposed method usually finds the similar pixels to the area around the reflections. Thus, compared with the TV-based method, the proposed method is much better in handling the case of tongue image with large reflection areas by involving more texture information, as shown in Figure 9. In Figure 9(b), we manually add a block to original tongue image to simulate the large reflection area. It is easy to find that the inpainting result of the proposed method is more similar to the ground truth, which is much better than the TV-based method by involving more texture information of tongue body.Figure 9
Comparison of inpainting result on tongue image with large reflection area.
(a)
Ground truth
(b)
Tongue image with missing block
(c)
Inpainting result of our WNNM-based method (PSNR = 25.39)
(d)
Inpainting result of the TV-based method (PSNR = 23.66)


### 4.2. Experimental Results on Real Tongue Images

For better evaluation of the proposed inpainting method, we used two real tongue images with serious specular reflection areas to validate its performance. The experimental inpainting results of three real tongue images are shown in Figure10. Moreover, on the lower-right of each image of Figures 10(b) and 10(c), we further showed the enlarged subimages of the inpainting images. From Figure 10, one could observe that the proposed method would achieve satisfactory visual results with much texture information of tongue image while part of horizontal or vertical discontinuity can be observed from the results obtained by the TV-based method.Figure 10
Comparison of inpainting results of WNNM-based and TV-based method.
(a)
Original tongue images with specular reflections
(b)
Inpainting results of our WNNM-based method
(c)
Inpainting results of the TV-based methodBased on the results on synthetic and real tongue images, we showed that the proposed method was better than TV-based method, either in terms of PSNR or in the visual results. The proposed method filled the reflection area with more texture information of tongue image, which was useful in the tongue texture analysis applications. Compared with TV-based inpainting method, the proposed method was more suitable for the specular reflection removal of tongue images.


## 4.1. Experimental Results on Synthetic Images

In this section, we quantitatively compared the performance of the competing methods on synthetic images. The synthetic images were constructed as follows: we first detect the specular reflections of one real tongue image and then randomly put these reflections onto nonreflection tongue images to synthesize new images for inpainting. The benefit was that, on one hand, we could ensure that the topological structure of reflections was as natural as real ones. On the other hand, we could conveniently calculate the peak signal to noise ratio (PSNR) since we had the ground truth.We used two ways to detect the highlight reflections of tongue image. One was the morphological dilation based method in [20] and the other was our superpixel based method. Synthetic images with both kinds of reflections were used in the experiments of WNNM-based inpainting and TV-based inpainting, respectively.Figure7 shows the comparison experimental results of competing methods on synthetic tongue images with reflections detected by morphological dilation based method, while Figure 8 shows the results on the synthetic tongue images with reflections detected by the proposed superpixel based method.Figure 7
Comparision of inpainting results of synthetic tongue images with reflection detected by morphological dilation based method.
(a)
Ground truth
(b)
Synthetic tongue images
(c)
Results of WNNM-based method (PSNR: left: 30.37, right: 31.56)
(d)
Results of TV-based method (PSNR: left: 29.32, right: 30.06)Figure 8
Comparison of inpainting results of synthetic tongue images with reflection detected by superpixel based method.
(a)
Ground truth
(b)
Synthetic tongue images
(c)
Results of our WNNM-based method (PSNR: left: 32.29, right: 31.34)
(d)
Results of the TV-based method (PSNR: left: 31.39, right: 30.47)From Figures7, 8, and 9, we could find that for the same synthetic tongue image, in terms of PSNR, the proposed inpainting method obtained better result than the TV-based inpainting method. Furthermore, the TV-based inpainting method usually finds the pixels via minimizing total variation value, while the proposed method usually finds the similar pixels to the area around the reflections. Thus, compared with the TV-based method, the proposed method is much better in handling the case of tongue image with large reflection areas by involving more texture information, as shown in Figure 9. In Figure 9(b), we manually add a block to original tongue image to simulate the large reflection area. It is easy to find that the inpainting result of the proposed method is more similar to the ground truth, which is much better than the TV-based method by involving more texture information of tongue body.Figure 9
Comparison of inpainting result on tongue image with large reflection area.
(a)
Ground truth
(b)
Tongue image with missing block
(c)
Inpainting result of our WNNM-based method (PSNR = 25.39)
(d)
Inpainting result of the TV-based method (PSNR = 23.66)


## 4.2. Experimental Results on Real Tongue Images

For better evaluation of the proposed inpainting method, we used two real tongue images with serious specular reflection areas to validate its performance. The experimental inpainting results of three real tongue images are shown in Figure10. Moreover, on the lower-right of each image of Figures 10(b) and 10(c), we further showed the enlarged subimages of the inpainting images. From Figure 10, one could observe that the proposed method would achieve satisfactory visual results with much texture information of tongue image while part of horizontal or vertical discontinuity can be observed from the results obtained by the TV-based method.Figure 10
Comparison of inpainting results of WNNM-based and TV-based method.
(a)
Original tongue images with specular reflections
(b)
Inpainting results of our WNNM-based method
(c)
Inpainting results of the TV-based methodBased on the results on synthetic and real tongue images, we showed that the proposed method was better than TV-based method, either in terms of PSNR or in the visual results. The proposed method filled the reflection area with more texture information of tongue image, which was useful in the tongue texture analysis applications. Compared with TV-based inpainting method, the proposed method was more suitable for the specular reflection removal of tongue images.


## 5. Conclusion

In this paper, we proposed a WNNM-based nonlocal inpainting method for specular reflections removal of tongue image. In the proposed method, superpixel segmentation was adopted to handle the specular reflections detection task. Then, based on the nonlocal self-similarity, we transformed the problem of specular reflections inpainting of tongue image to a matrix completion problem and further proved the convergence of the proposed WNNM-based tongue image inpainting method.We evaluated the performance of the proposed approach by comparing with the TV-based tongue image inpainting method. The comparison experimental results on both synthetic tongue images and real tongue image showed that the proposed method could achieve not only higher PSNR, but also more satisfactory visual effects, which could involve more texture information of tongue body, and was more suitable for the specular reflections removal of tongue images.



---

*Source: 979415-2015-08-17.xml*