Automatic thresholding using a modified valley emphasis

IET Image ProcessingVolume 14, Issue 3 p. 536-544 Research ArticleFree Access Automatic thresholding using a modified valley emphasis Jiangwa Xing, Jiangwa Xing Research Center of Basic Medical Sciences, Medical College, Qinghai University, Xining, 810016 People's Republic of ChinaSearch for more papers by this authorPei Yang, Corresponding Author Pei Yang yangpeinmgdx@sina.com Department of Computer Technology and Application, Qinghai University, Xining, 810016 People's Republic of ChinaSearch for more papers by this authorLetu Qingge, Letu Qingge College of Computing and Informatics, University of North Carolina Charlotte, NC, 28223 USASearch for more papers by this author Jiangwa Xing, Jiangwa Xing Research Center of Basic Medical Sciences, Medical College, Qinghai University, Xining, 810016 People's Republic of ChinaSearch for more papers by this authorPei Yang, Corresponding Author Pei Yang yangpeinmgdx@sina.com Department of Computer Technology and Application, Qinghai University, Xining, 810016 People's Republic of ChinaSearch for more papers by this authorLetu Qingge, Letu Qingge College of Computing and Informatics, University of North Carolina Charlotte, NC, 28223 USASearch for more papers by this author First published: 27 January 2020 https://doi.org/10.1049/iet-ipr.2019.0176Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Otsu's method is one of the most well-known methods for automatic thresholding, which serves as an important algorithm category for image segmentation. However, it fails if the histogram is close to unimodal or has large intra-class variances. To alleviate this limitation, improved Otsu's methods such as the valley emphasis method and weighted object variances method have been proposed, which still yield non-optimal segmentation performance in some cases. In this study, a modified valley metric using second-order derivative is proposed to improve the Otsu's algorithm. Experiments are firstly conducted on five typical test images whose histograms are unimodal, multimodal or have large intra-class variances, and then expanded to a larger data set consisting of 22 cell images. The proposed algorithm is compared with original Otsu's method and existing improved algorithms. Four evaluation metrics including misclassification error, foreground recall, Dice similarity coefficient and Jaccard index are adopted to quantitatively measure the segmentation performance. Results show that the proposed algorithm achieves best segmentation results on both data sets quantitatively and qualitatively. The proposed algorithm adapts the Otsu's method to more image subtypes, indicating a wider application in automatic thresholding and image segmentation field. 1 Introduction Image segmentation is an important technique widely used in many application fields [1-5]. A lot of segmentation algorithms including traditional technologies [6-13] and deep learning-based methods [14, 15] have been presented so far for various kinds of segmentation tasks. Among all the segmentation algorithms, automatic thresholding is still widely used due to its simplicity and effectiveness. As a typical kind of segmentation method and a preprocessing tool for many other computer vision tasks [16, 17], automatic thresholding has been widely studied over the past decades. The main idea of automatic thresholding algorithms is to select proper threshold values to separate pixels of a grey-level image into several subsets. For a single thresholding segmentation, the two subsets correspond to the background and the foreground, respectively. Many automatic thresholding algorithms have been introduced recently, which can be further categorized into global thresholding and local thresholding [18, 19]. Local thresholding selects multiple threshold values based on localized grey-level information, resulting in better performance for non-uniform illumination images; while global thresholding chooses only one threshold value for the entire image, which is much simpler and more efficient. Representative global thresholding includes iterative thresholding [20], entropic thresholding [21] and Otsu-based methods [9]. Iterative thresholding separates pixels of a grey-level image into two classes using a predefined threshold value, calculates their respective average grey-level value g1 and g2, and then iteratively uses (g1 + g2)/2 as the new threshold value to reclassify the image pixels until the value of (g1 + g2)/2 stays constant. The iterative thresholding method is easy to implement, but its algorithm complexity is O(n2) (n is the pixel number of the image). Entropic thresholding divides the image pixels into two classes based on their grey levels, calculates the inter-class entropy, and sets the threshold value which achieves the largest entropy as the optimal threshold. Entropic thresholding also suffers from relatively high time complexity, which is O(n!). In 1979, Nobuyuki Otsu published the well-known global thresholding method named Otsu's method [9]. It is one of the most commonly used techniques for thresholding real-world images such as ice-covered cables [22], lung CT images [23], and images in various other fields [24]. Although Otsu's method has been proposed for a long time, there are still a lot of research works focusing on improving the algorithm [10-13, 18] and analysing its performance [25]. The main idea of Otsu's method is to find the optimal threshold value by maximizing the between-class variances of the histogram. It works well on images with bimodal or multimodal distributed histograms while fails on images with unimodal histograms or histograms of large intra-class variances. In order to overcome this disadvantage, many researchers have been trying to improve the original Otsu's method. Ng [18] modified the between-class variance function and introduced a valley emphasis (VE) technique. In Ng's work, grey-level probability was adopted as a valley metric, and a penalty factor constructed based on valley metric was introduced to the objective function to make the threshold close to the valley of the image histogram. Although the modified thresholding algorithm produces a better result on defect detection problem, it fails in some other cases, especially when the variance of the object is very different from that of the background [10]. Later, Fan and Lei [10] pointed out that Ng's method only weighted the objective function with the valley-point value of the histogram, and proposed to add in the neighbourhood information of the valley point (neighbourhood valley emphasis (NVE)). Xu et al. [11] analysed the characteristics of Otsu's threshold and got the conclusion that Otsu's threshold biased toward the class with larger variance if the within-class variances of background and foreground were different. Partly inspired by Xu's work, Yuan et al. [12] proposed an improved Otsu's method named weighted object variances (WOVs). WOV ensured the threshold located at the valley of two peaks or at the left bottom of a single peak histogram by adding a weight on the object variance. In 2019, Cao et al. [26] published a new Otsu-based parameter-free thresholding method for image segmentation by adding a new term to maximize the distance between the mean values of each class, which claimed to be more accurate and robust compared with standard Otsu's method. Besides these improved one-dimensional Otsu's methods mentioned above, many two-dimensional Otsu's methods have also been published [27-30]. Despite better performance, the two-dimensional Otsu's algorithms are generally more time consuming compared to one-dimensional Otsu's methods. In this paper, we are focusing on improving the performance of one-dimensional Otsu's algorithm. Fig. 1 shows the threshold values acquired by several one-dimensional Otsu's algorithms on a typical image named coins. Despite the effectiveness declared in the published literature, these algorithms could not find the desired threshold value, resulting in non-optimal segmentation results. To further improve Otsu's method to fit more cases, in this paper, we introduce a modified valley metric and propose an improved Otsu's method based on that. The main contributions of this paper are as follows: (i) We have introduced a modified valley metric using second-order derivative instead of grey-level probability used in original VE, and proposed an improved Otsu's algorithm based on the modified valley metric; Fig. 1Open in figure viewerPowerPoint Thresholds of various Otsu methods on image coins (a) Original image, (b) Ground truth, (c) The desired threshold and threshold values acquired by Otsu, VE, NVE, WOV, Cao's method and the proposed algorithm (ii) Four quantitative evaluation metrics including misclassification error (ME), foreground recall (FRecall), Dice similarity coefficient (DSC) and Jaccard index (Jac) are employed for algorithm validation. Experiments on five typical test images as well as on a larger data set consisting of 22 cell images demonstrate that the proposed method has the overall best segmentation performance compared with existing Otsu's method, VE, NVE, WOV and Cao's method. (iii) The proposed algorithm could achieve much better segmentation results on images having unimodal histograms or large intra-class variances, without compromising its segmentation performance on images with bimodal or multimodal histograms. It adapts the Otsu's method to more image subtypes and makes it suitable for wider applications in real-world image segmentation scenarios. 2 Otsu's algorithms In this section, original Otsu's method will be introduced briefly, followed by improved algorithms including VE, NVE, WOV and Cao's method. 2.1 Otsu's method Otsu's method tries to find the optimal threshold value by maximizing the between-class variances of the histogram. Let I be a grey-level image of size M*N, and I(x, y) whose value is ranging from 0 to L − 1 represent the pixel grey level with coordinates (x, y). If the number of pixels with grey level i is denoted as f(i), the probability of occurrence of grey-level i can be defined as (1)For a single threshold situation, given a threshold t, pixels of an image will be divided into two classes and . Let and be the probabilities of the two classes respectively, and , and be the average grey levels of , and the entire image, then they can be defined as follows: (2) (3) (4) (5) (6)For a threshold t, the between-class variance in Otsu's method is defined as (7)Then the best threshold t* can be acquired by solving the optimization issue below (8) 2.2 Improved Otsu's methods With the observation that the best threshold value should locate at the valley of two peaks, or at the bottom rim of a single peak, Ng proposed an improved VE scheme. The main idea of VE is to select a threshold value with a small probability of occurrance and to maximize the between-class variance in Otsu's method [13]. The best threshold of VE can be determined as follows: (9)The VE was proved to be useful experimentally on defect detection in Ng's work, but there are still some cases where it does not fit. Fan introduced a modified strategy using neighbourhood information for valley metric. Let denote average filtering where • is a one-dimensional input signal and n represents the filter size. The valley metric and optimization issue in NVE Otsu are defined as follows: (10) (11)It is obvious that if n equals 1 in (10), NVE is exactly the same as VE. In other words, VE is a special case of NVE. Yuan had experimentally studied the performance of VE and NVE, and proposed a modified Otsu's method named WOVs which adds a weight factor on the first term in (7). The between-class variance in WOV is then defined as (12)where is a weight factor ranging from 0 to 1 and set to for adaptive weight selection. Consequently, the threshold can be determined by solving the following optimization problem: (13)Cao et al. presented an improved Otsu's method whose objection function not only makes the between-class variance maximum but also maximizes the distance between the mean values of each class. The new object function in Cao's work is defined as (14)Although the VE, NVE and WOV algorithms as well as Cao's method perform better than standard Otsu's method in most cases, there are still certain types of images they cannot properly process. As shown in Fig. 1, the thresholds acquired by VE, NVE, WOV and Cao's method are not located at the real valley and significantly different from the desired value. In the following section, a new valley metric is introduced, and automatic thresholding using the modified VE is proposed. 3 Proposed automatic thresholding 3.1 Valley metric using second-order derivative As is known from the discussion above, VE is not enough in some thresholding cases. In order to further improve the performance of Otsu's method, we are trying to evaluate the valley using the second-order derivative. Let be the envelope function of a histogram and second-order differentiable, and , be the first-order and second-order derivative, respectively, it is obvious that equals 0 at both peaks and valleys, whereas is positive at valleys and negative at peaks. Fig. 2 shows the envelope curve of the histogram of image coins and its corresponding to second-order difference curve. We can clearly find that points around the valley of the histogram envelope curve correspond to positive second-order difference values. Especially, each valley point in histogram envelope curve corresponds to a peak point with a positive value in the second-order difference curve using original histogram. Fig. 2Open in figure viewerPowerPoint Histogram of coins and its corresponding second-order difference curves Motivated by the mathematical property mentioned above, we define the valley metric as (15)where and correspond to the minimum and maximum value of the second-order derivative respectively, and their definition domains are . The range of is . Since it is usually difficult to get a differentiable envelope function in practice, we use difference instead of derivative. Let be the histogram of an image, be the number of pixels with grey-level i as defined in Section 2, the difference representation for and can be defined as: (16)Using the difference in (16), we define the valley metric using the formula below (17)Combining (1) and (17), can be finally defined as (18)where is the probability of occurrence of grey level x, and represents the probability of all grey levels. In our implementation, the histogram of the image is first smoothed using an average filter in order to alleviate the second-order difference anomaly caused by the abnormal fluctuation of the histogram. The average smooth process can be described as (19)where 2k + 1 is the filter size, and k is set to five as suggested in the NVE method [18]. 3.2 Automatic thresholding using modified valley metric Taking the valley metric in (18) into consideration, the final object function of our proposed algorithm is defined as follows: (20)In (20), the between term and the valley metric are multiplied together. There are two advantages using (20) as the objective function. On one hand, the two terms in (20) are as large as possible when reaches its largest value. On the other hand, the objective function is parameter-free which means we do not have to discuss the value of tradeoff parameters between and . As Otsu's method, the best threshold of our proposed algorithm is the solution of the optimization issue below: (21)Consequently, the proposed improved Otsu's method is as follows: Step 1: For a given grey level image, calculating its histogram . Step 2: , where is the smoothed histogram obtained by (19). Step 3: Calculating the between defined by (7) and valley metric defined by (18) for each t ranging from 0 to 255 using the smoothed histogram f. Step 4: Calculating the optimal threshold value by solving the optimization problem in (21). 4 Experimental results and analysis Experiments are conducted using Matlab R2012b on a PC with Intel Core 2.30 GHz CPU and 4.0 GB memory. First, five typical test images named rice, coins, handwriting, printing character and license plate are selected to test algorithm performance, whose histograms include unimodal, bimodal and multimodal distributions (Fig. 3). Among them, image coins has significant large intra-class variances in its histogram. Each of the images is manually labelled with the binary segmentation ground truth, partly based on which the algorithm evaluation will be conducted. Subsequently, a larger data set containing 22 cell images is constructed to further test the proposed algorithm. All the 22 images are collected from internet, and we manually label the segmentation ground truth for each image. More details about the data set and the validation experiments will be introduced later. Fig. 3Open in figure viewerPowerPoint Test images and their manually labelled binary segmentation ground truth (a) Original image, (b) Ground truth, (c) Histograms of each image In order to verify the effectiveness of the proposed algorithm, results are compared with those of Otsu's method [15], VE [13], NVE [19], WOV [23] and Cao's method [26], and four evaluation metrics including ME [13, 19, 23], FRecall, DSC [31, 32] and Jac [33, 34] are adopted to quantitatively measure the algorithm's performance. The metric ME for effectiveness evaluation is defined as follows: (22)where and are pixel sets of foreground and background segmented by automatic thresholding method, and , are manually labelled foreground and background pixel sets which serve as ground truth. Consequently, ME represents the ratio of misclassified pixels in an image by automatic thresholding method. Obviously, better automatic thresholding corresponds to a smaller ME value. On the other hand, FRecall is used to assess the ability of foreground extraction of a method, and it can be expressed as (23)where and are defined as the same as those in (22). Large FRecall value represents a higher foreground extraction ability. The third adopted metric is DSC, which is also named as F1 score. It is usually used to measure the similarity of two samples. The larger the DSC value is, the more similar the two samples are. Equation (24) is the definition of DSC (24)In this paper, TP represents the number of correctly detected foreground pixels, and FP, FN are the number of incorrect foreground pixels and number of miss detected foreground pixels, respectively. The last evaluation metric is Jac and it is defined as (25)where and are the same as those introduced in ME. In this paper, the performance of the proposed method is evaluated on both data sets mentioned above and compared with original Ostu's method, VE, NVE, WOV and Cao's method. In the following parts, algorithm evaluation using a data set of five typical test images is first conducted in four aspects, including the object function comparison, rationality analysis of the threshold value, quantitative evaluation and qualitative evaluation. Next a larger data set of 22 cell images is introduced, and further experiments are conducted on the proposed larger data set. 4.1 Object function evaluation The main idea of the proposed method is taking the second-order difference-based valley metric into consideration to construct a more effective object function. In this section, the objective function of the proposed method will be evaluated on image rice and coins and compared with Otsu's method. Fig. 4 demonstrates the difference between our proposed object function and that of Otsu's method. The primary impression of the figure is that our modified object functions are not as smooth as that of the Otsu's, which may be due to the fluctuation of the histogram. However, the trends are distinct that the objective function values are enhanced at valley points and weakened at peaks. This effect is more obvious on image coins shown in Fig. 4b. Fig. 4Open in figure viewerPowerPoint Object function curve comparison of the proposed method and Otsu's algorithm on (a) Image rice, (b) Image coins. : Otsu's algorithm, : the proposed method 4.2 Rationality analysis of the threshold value To validate the rationality of the threshold of the proposed algorithm, all five typical test images are used for testing. Through the whole experiments, the filter size used in NVE is set to 11 as suggested in the literature work [19]. Table 1 demonstrates the threshold values of each method, and it is obvious that the results of Otsu's method, VE and NVE are extremely close on all images. Table 1. Threshold values of five algorithms on test images Methods Rice Coins Handwriting Printing character License plate Otsu's method [9] 131 126 113 98 155 VE [18] 131 125 113 98 155 NVE [10] 131 125 113 98 155 WOV [12] 136 134 127 105 172 Cao et al. [26] 138 132 94 109 155 proposed method 121 82 98 96 137 Fig. 5 shows the threshold locations of each method on the histograms of all test images except for image coins, which is shown in Fig. 1c. The histogram distributions of image handwriting and printing character are close to unimodal, the case of which Otsu's method is known to perform badly on. In particular, the grey level of foreground character for image handwriting is lower and closer to background grey level, making the problem more difficult. From Fig. 5 we can find that threshold values acquired by the proposed method for these two images both locate at the bottom rim of the peak and only the proposed algorithm and Cao's method generate thresholds close to the left bottom rim for image handwriting. For the remaining images with bimodal or multimodal histogram distribution, the proposed algorithm chooses thresholds closest to the real valleys compared with all other compared methods as well, indicating its general effectiveness in all test cases. In particular, as is shown in Fig. 1c, the proposed method could identify a threshold value most similar to the desired value for image coins, whose histogram has large intra-class variances and challenging for existing Otsu's method and other improved methods. Fig. 5Open in figure viewerPowerPoint Positions of threshold values of each method on test images (a) Rice, (b) Handwriting, (c) Printing character, (d) License plate 4.3 Quantitative evaluation In order to evaluate the performance of all test methods, we have manually labelled the binary segmentation ground truth for the five test images (Fig. 3). The ground truth images are labelled in pixel level and saved as png format files. ME, FRecall, DSC and Jac metric values are then calculated based on the ground truth images to quantitatively evaluate the segmentation performance of each method. Table 2 lists the ME values of each algorithm on all of the five images. We can see that the proposed algorithm gains the smallest ME values on three of the five test images except for image handwriting and image license plate, which are both one percent larger than the smallest ME. In addition, the average ME of the proposed method on five test images is far less than most of the other Otsu-based algorithms including Otsu's method, VE, NVW and WOV, and one percent smaller than that of the Cao's method. As smaller ME values indicate more effective segmentation ability, the ME results have verified the effectiveness of our proposed algorithm. Table 2. ME values of each method on test images Methods ME Rice Coins Handwriting Printing character License plate Average Otsu's method [9] 0.0530 0.0248 0.2289 0.0383 0.0111 0.0712 VE [18] 0.0530 0.0241 0.2289 0.0383 0.0111 0.0711 NVE [10] 0.0530 0.0241 0.2289 0.0383 0.0111 0.0711 WOV [12] 0.0638 0.0306 0.4640 0.0456 0.0416 0.1291 Cao et al. [26] 0.0686 0.0292 0.0452 0.0499 0.0111 0.0408 proposed method 0.0399 0.0015 0.0556 0.0369 0.0201 0.0308 Bold values indicate the best segmentation method for images tested. Table 3 shows the FRecall values of all test methods. The proposed method obtains the best average result and has the largest FRecall values for almost all test images except for image handwriting. Although a larger FRecall value generally corresponds to a better foreground detection, it may not be the case if the foreground pixels are overly labelled compared with ground truth. According to (23), if we label all pixels in an image as foreground pixels, the FRecall can reach 100%, which is not the result we expect. Here the result for image handwriting encounters similar situation and the good segmentation result of our proposed method will be demonstrated in the qualitative evaluation section. Table 3. FRecall values of each method on test images Methods FRecall Rice Coins Handwriting Printing character License plate Average Otsu's method [9] 0.8324 0.9241 0.9177 0.9249 0.9829 0.9164 VE [18] 0.8324 0.9264 0.9177 0.9249 0.9829 0.9169 NVE [10] 0.8324 0.9264 0.9177 0.9249 0.9829 0.9169 WOV [12] 0.7951 0.9063 0.9800 0.8765 0.9223 0.8960 Cao et al. [26] 0.7791 0.9107 0.7486 0.8501 0.9853 0.8548 proposed method 0.9011 0.9956 0.7914 0.9375 0.9951 0.9241 Bold values indicate the best segmentation method for images tested. The DSC values of each algorithm on the five test images are shown in Table 4. As introduced before, larger DSC values indicate better algorithm performance. From Table 4 we can come to the conclusion that the proposed algorithm, whose average value is 0.92, generally perform better than the compared methods on test images. Table 5 shows the Jac values of each algorithm on five test images. The average Jac value of the proposed method is larger than those of all compared algorithms, once again confirming the outperformance of the proposed algorithm. Table 4. DSC values of each method on test images Methods DSC Rice Coins Handwriting Printing character License plate Average Otsu's method [9] 0.9063 0.9606 0.4466 0.9179 0.9893 0.8441 VE [18] 0.9063 0.9618 0.4466 0.9179 0.9893 0.8444 NVE [10] 0.9063 0.9618 0.4466 0.9179 0.9893 0.8444 WOV [12] 0.8848 0.9508 0.2983 0.8991 0.9585 0.7983 Cao et al. [26] 0.8750 0.9533 0.7609 0.8875 0.9853 0.8941 proposed method 0.9330 0.9978 0.7413 0.9217 0.9810 0.9150 Bold values indicate the best segmentation method for images tested. Table 5. Jac values of each method on test images Methods Jac Rice Coins Handwriting Printing character License plate Average Otsu's method [9] 0.8287 0.9241 0.2875 0.8482 0.9787 0.7734 VE [18] 0.8287 0.9264 0.2875 0.8482 0.9787 0.7739 NVE [10] 0.8287 0.9264 0.2875 0.8482 0.9787 0.7739 WOV [12] 0.7934 0.9063 0.1753 0.8166 0.9203 0.7224 Cao et al. [26] 0.7777 0.9107 0.6251 0.7977 0.9809 0.8184 proposed method 0.8744 0.9955 0.5889 0.8548 0.9627 0.8553 Bold values indicate the best segmentation method for images tested. 4.4 Qualitative evaluation of segmentation results Fig. 6 shows the thresholding segmentation results of each method on the five test images. Images of each column from left to right are original images, results of Otsu's method, VE, NVE, WOV, Cao's method and the proposed method. There is no much difference among all methods for image printing character and image license plate. For image rice, although all methods miss some foreground object pixels at the bottom area due to uneven illumination, our algorithm misses the least. As for image coins whose histogram has large intra-class variances, the proposed method is the only one which could correctly segment out all coins without any local cavities. When applying the proposed method on image handwriting, despite a much lower FRecall value acquired compared with most of the other Otsu-based methods shown in Table 3, the lower ME and higher DSC and Jac values assure that the proposed method could almost segment all words from the image, generating the best result. When compared with Cao's method for image handwriting, three of the four evaluation metric values including ME, DSC and Jac are better for Cao's method, and only the value of FRecall shows a better result for our proposed method. However, from the segmentation results in Fig. 6 we could see that despite some over labelled foreground at the left lower corner, our proposed method could segment much clearer characters compared with Cao's method. Therefore, based on the thresholding segmentation results on all five test images described above, the proposed method shows the best segmentation performance, indicating its overall effectiveness for various image subtypes. Fig. 6Open in figure viewerPowerPoint Segmentation results of each automatic thresholding method. From left to right: original images, Otsu's results, VE's results, NVE's results, WOV's results, Cao's results, and the proposed results 4.5 Further evaluation on a larger data set In order to further evaluate the proposed algorithm, we cond