Robust segmentation of the colour image by fusing the SDD clustering results from different colour spaces

IET Image ProcessingVolume 14, Issue 13 p. 3273-3281 Research ArticleFree Access Robust segmentation of the colour image by fusing the SDD clustering results from different colour spaces Zhenzhou Wang, Corresponding Author Zhenzhou Wang zzwangsia@yahoo.com College of Electrical and Electronic Engineering, Shandong University of Technology, People's Republic of ChinaSearch for more papers by this author Zhenzhou Wang, Corresponding Author Zhenzhou Wang zzwangsia@yahoo.com College of Electrical and Electronic Engineering, Shandong University of Technology, People's Republic of ChinaSearch for more papers by this author First published: 05 October 2020 https://doi.org/10.1049/iet-ipr.2019.1481Citations: 5AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Segmentation of the colour image is challenging because the colour information is lost after being projected into three channels of the colour space. Many state-of-the-art colour image segmentation methods are based on monochrome segmentation in one channel of the colour space. However, the optimal performance of a segmentation method usually could not be achieved in a single colour space due to the complexity and diversity of the colour images. In this study, the authors propose to segment the colour image by fusing the slope difference distribution (SDD) clustering results in different colour spaces. For simplicity, the segmentation approach is designed as two-label segmentation and it could be easily generalised to be multiple-label segmentation. The proposed approach is compared with the state-of-the-art colour image segmentation methods both quantitatively and qualitatively. Experimental results verified the effectiveness of the proposed approach. 1 Introduction Colour image segmentation attracts more and more attention to the rapid development of colour imaging and colour television technology. Obviously, colour images could provide much more information than grey-level images. The additional information provided by colour could yield better image analysis results than those yielded by processing only the grey-level information in many computer vision or pattern recognition applications. Despite the fact that so many research efforts from all over the world have been put into dealing with colour segmentation issues for a long time, colour image segmentation remains a challenging problem because the accuracy of the segmentation method depends on both the characteristics of the image and the selected colour channels significantly. Currently, there are many popular colour models available for image processing and visualisation. It is difficult to determine which colour model is best for image segmentation since the colour images are so diverse and complex. In [1], the author combined the segmentation results from all the available colour spaces for better accuracy. Different from [1], most state-of-the-art colour image segmentation methods implemented their monochrome segmentation algorithms independently in a single channel of the colour space [2, 3]. For instance, in the RGB colour space, the R, G or B channel could be combined or selected to implement the segmentation algorithms [4–10]. Especially, the authors in [10] claimed that their method operates better in the RGB colour space than Lab colour space. However, the Lab colour space has been the most popular space so far for image segmentation because of its better colour uniformity [11–18]. The authors in [2] reviewed the state-of-the-art colour image segmentation methods thoroughly and persuasively. Later, they also implemented their segmentation algorithms in the Lab colour space. Another frequently used colour space for colour image segmentation is Luv space [19–22]. Both Lab and Luv colour spaces are computed from the XYZ colour space and both of them are in approximately uniform chromatic scale [2]. Besides transforming the RGB colour space to Lab or Luv colour space, another popular way of segmenting the RGB mages is to transform the RGB image to the grey-level image by eliminating the hue and saturation information while retaining the luminance. Unfortunately, the luminance channel could not represent the colour information well and consequently, segmentation methods based on luminance tend to split the same colour into different regions. In [23], the authors claimed the advantage of the YUV space over RGB space and appealed the use of YUV in machine vision applications. In [24], the authors segmented the colour image in the YUV space. YUV colour space is good at separating the luminance (brightness) component from the chrominance (colour) components of the image and it is widely adopted in coloured television and video, such as moving picture experts group standard. In many applications, the colour pictures are taken under the influence of uneven illumination. As a result, the uniform colour information is modified by the uneven illumination in the RGB colour model. The complexity of the image itself has already made colour image segmentation challenging and the complexity of the uneven modification by illuminations make colour image segmentation significantly more challenging. By nature, the YUV colour model separates the illumination from the colour information and thus should be more suitable for colour image segmentation than the RGB colour model. In addition to the popular Lab and YUV colour spaces for colour image segmentation, there is also research work conducted in the Hsv colour space [25, 26]. In this paper, we will fuse all the clustering results obtained from the Lab colour space, YUV colour space, Hsv colour space and RGB colour space to yield the final segmentation result. To obtain the clustering results robustly, we compared the slope difference distribution (SDD) clustering method [27], the Otsu's [28] clustering method, the expectation maximisation (EM) clustering method [29] and the k-means clustering method [30] qualitatively and quantitatively. Experimental results showed that the SDD clustering method is more accurate than the state-of-the-art clustering methods for image segmentation. Although there have been many new or improved clustering methods proposed in recent years [31–37], they might not be suitable for image segmentation. Hence, the SDD clustering method is used to get the clustering results from different colour spaces. The paper is organised as follows. The popular colour models are overviewed and analysed in Section 2. The SDD clustering-based colour segmentation approach is proposed in Section 3. The SDD clustering method is compared with the state-of-the-art clustering methods and the proposed colour image segmentation approach is compared with the state-of-the-art colour image segmentation methods in Section 4. Section 5 concludes the paper. 2 Overview of the colour models 2.1 RGB colour model The RGB colour model is additive with three light beams added together. Zero intensity of each component gives black and full intensity of each component gives white. When the intensities of the three components are the same, the result is grey and it varies from dark to light with the value of the intensity. When the intensities of the three components are different, the result is a colourised hue and its saturation depends on the largest difference among the intensities of the three primary components. The R, G and B components can be formulated by the following equations [2]: (1) (2) (3)where , and are the colour filters on the incoming light and is the wavelength. From the above equation, we see that the intensities of R, G and B components are determined by the incoming light. As a result, when the intensity of the incoming light is not evenly distributed, the intensity of R, G and B component will not be distributed evenly. In many cases, the colour pictures are taken under the influence of different light sources. Consequently, the resultant images usually have uneven intensity distributions in R, G and B channels, which will decrease the segmentation accuracy of image segmentation methods that are based on pixel clustering [1, 16]. 2.2 YUV colour model YUV defines the colour space in terms of luminance (Y) and chrominance (UV). The luminance component corresponds to the black and white intensity information while the chrominance components correspond to the colour information. To some extent, the YUV colour model separates the brightness information and the colour information of the image into luminance and chrominance, respectively. Theoretically, the colour information will not be affected by the uneven intensity distribution caused by the brightness. Hence, the colour intensity in the chrominance channels should be more evenly distributed than that in the R, G and B channels, which will increase the segmentation accuracy. The YUV components are typically created from the RGB components by the following linear transform [2]: (4) 2.3 Lab c colour model Similar to YUV colour model, the Lab colour model also separates the brightness information and the colour information of the image. The difference is that it transforms the RGB components to the L component for lightness and a, b components for the colour by the following non-linear formulations [2]: (5) (6) (7) (8)where are the values for the standard white. Owing to the lightness information and the colour information are modelled as different components, the intensities of the a and b colour channels are more evenly distributed than that of the R, G and B components. Hence, a and b colour channels are preferred by many researchers to apply their image segmentation methods [11–18]. 2.4 HSV c colour model HSV are designed to align with the way human vision perceives colour-making attributes. It has been invented as a method to add colour encoding to existing luminance. Thus, it is also useful for colour segmentation. The HSV components are converted from the RGB components that are reshaped in the range of [0 1] [25, 26]: (9) (10) (11) (12) (13)where are the values reshaped in the range of [0 1]. 2.5 Analysis of the colour models for segmentation As analysed in the above sections, the YUV colour model, the Lab colour model and the HSV colour model are more suitable for colour segmentation because their colours are separate from luminance and are distributed more evenly after linear or non-linear transformations. Yet, it is not determined whether the linear transformation is better or the non-linear transformation is better in removing the uneven modification from the colour information. In addition, the performance of the colour model for different segmentation methods is not fixed as reported in past research [1, 2, 11–18]. It is unknown yet which colour model is the best one for colour image segmentation. In single colour space, the colour segmentation performance in different channels also varies significantly and depends on the image greatly. In this paper, we propose to fuse the colour clustering results in different colour spaces and compute the final segmentation based on the fusion. 3 Proposed approach 3.1 Generation of the set of segmentation The a and b channels in the Lab colour space are selected to avoid the effect of uneven luminance. The U and V channels in YUV colour space are selected to avoid the effect of uneven brightness. All the three channels in the Hsv colour space and all the three channels in the RGB colour space are selected as complement. In each selected channel, the pixels are clustered by the SDD clustering method as described in [27] to generate the means of different pixel classes. The image in the selected channel is then segmented as (14)where denotes the position of the pixel in the image channel and denotes the resolution of the image. L denotes the total number of colour clusters in the image. Since we focus on two-label segmentation in this paper, the number of colour clusters L is selected as 3 for simplicity. The segmentation result is then transformed into binary images by the following equations to get four two-label segmentation results, , , and : (15) (16) (17) (18) 3.2 Proposed fusion method In the ten channels of the four colour spaces, a total of 42-label segmentation results are obtained. In each colour space, one segmentation result is selected to generate the fusion result. By permutation and combination, the total number of the fusion times could be calculated as (19)It is intractable to search for the optimal fusion result in so many combinations. Hence, the following rules are used to select the optimal segmentation from each colour space. First, the centre area and the surrounding area of the image are defined as C and S, respectively, and they are computed as (20)For each two-label segmentation result in the selected colour space, it is classified as background or foreground based on its intersection with the centre area C and the surrounding area S(21)The total number of the foreground is denoted as N and the total number of the background is denoted as . For each foreground, its area is calculated and used to judge if it is a valid segmentation: (22)where is the area of the foreground . The default value of is 2. For the invalid foreground, we set its area as 0. In each channel of a selected colour space, the optimal foreground is chosen from the foregrounds by the following rules: (23) (24)In each colour space, we select the final foreground from the selected optimal foregrounds in different channels by the following rules: (25) (26)where is the optimal foreground in the a or U or R or H channel. is the optimal foreground in the b or V or G or s channel. is the optimal foreground in the B or v channel. For all the four colour spaces, the fused foreground is computed as (27)The fused foreground is binarised by the following equation to generate the rough object (28)Besides the foreground, we also calculate the constraint background. For each background, its area is calculated and used to determine whether it is valid or not (29)where is the area of the background. The default value of is 5. For the invalid background, we set its area as 0. In each channel, the optimal background is chosen from the backgrounds by the following rules: (30) (31)In each colour space, the final background is selected from the optimal backgrounds in different channels by the following rules: (32) (33)where is the optimal background in the a or U or R or H channel. is the optimal foreground in the b or V or G or s channel. is the optimal foreground in the B or v channel. For all the four colour spaces, the fused background is computed as (34)The fused background is binarised by the following equation to generate the constraint background: (35)The two-label segmentation result is calculated as (36)The two-label segmentation is then filtered by a morphological filter to remove noise. The morphological filter is designed as follows. First, the two-label segmentation is eroded times by the following equations: (37) (38) (39)Second, the two-label segmentation is dilated times by the following equations: (40)The default value of is 3. After filtering, the largest object in the segmentation is selected and its boundary B is extracted. In the meantime, the canny edge E of the original image is detected [38]. For each point on the boundary, there is one point on the detected canny edge E that is nearest to it. The nearest points to all the points on the boundary are computed by the following equations: (41) (42) (43)where v denotes the index of the points on the detected edge E. V is the total number of points on the detected edges E. is the final boundary. The proposed fusion method could be summarised in Algorithm 1. Algorithm 1.The proposed fusion methodInput: The four two-label segmentation results, , , and from the a and b channels in the Lab colour, from the U and V channels in YUV colour space, from all the three channels in the Hsv colour space and from all the three channels in the RGB colour space.Output: The final two-label segmentation resultFusion: (1) The binary fused foreground is computed by (20)–(28). (2) The binary fused background is computed by (29)–(35). (3) The two-label segmentation result is computed by (36). (4) The two-label segmentation result is filtered by (37)–(40). (5) The boundary of the filtered two-label segmentation result is extracted by (41)–(43). Result: The boundary of the final two-label segmentation result. 3.3 Computation complexity of the proposed approach The computation complexity of the SDD clustering method is , where is the number of the fitting points to compute the SDD of the histogram and is the total length of the histogram. To decrease the computation complexity, should not be too long. On the other hand, if the length of the histogram is shortened too much for computation efficiency, details of the image might be damaged. The default value of is 255. The computation complexity of implementing the SDD clustering in ten channels is . Since the rest of the computation is trivial, the total computation complexity of the proposed approach is . 4 Experimental results 4.1 Comparison of the clustering methods Since there are many popular clustering methods with open source codes available, it is necessary to compare them and find the best clustering method to use in image segmentation. In this section, we will compare the state-of-the-art clustering methods in segmenting the images in different colour spaces both qualitatively and quantitatively. The compared clustering methods include SDD clustering method that is implemented in each two-dimensional channel, the Otsu's clustering method that is implemented in each two-dimensional channel, the EM clustering method that is implemented in each two-dimensional channel and the k-means clustering method that is implemented in each three-dimensional colour space. The qualitative comparisons are shown in Figs. 1-10–11 with open access images from Berkeley segmentation database. The boundary of the segmented object is extracted and overlaid onto the original image for better comparisons. Fig. 1Open in figure viewerPowerPoint Comparison 1 for the segmentation results by the proposed approach based on different clustering methods (a) SDD, (b) EM, (c) Otsu, (d) K-means Fig. 2Open in figure viewerPowerPoint Comparison 2 for the segmentation results by the proposed approach based on different clustering methods (a) SDD, (b) EM, (c) Otsu, (d) K-means Fig. 3Open in figure viewerPowerPoint Comparison 3 for the segmentation results by the proposed approach based on different clustering methods (a) SDD, (b) EM, (c) Otsu, (d) K-means Fig. 4Open in figure viewerPowerPoint Comparison 4 for the segmentation results by the proposed approach based on different clustering methods (a) SDD, (b) EM, (c) Otsu, (d) K-means Fig. 5Open in figure viewerPowerPoint Comparison 5 for the segmentation results by the proposed approach based on different clustering methods (a) SDD, (b) EM, (c) Otsu, (d) K-means Fig. 6Open in figure viewerPowerPoint Comparison 6 for the segmentation results by the proposed approach based on different clustering methods (a) SDD, (b) EM, (c) Otsu, (d) K-means Fig. 7Open in figure viewerPowerPoint Comparison 8 for the segmentation results by the proposed approach based on different clustering methods (a) SDD, (b) EM, (c) Otsu, (d) K-means Fig. 8Open in figure viewerPowerPoint Comparison 9 for the segmentation results by the proposed approach based on different clustering methods (a) SDD, (b) EM, (c) Otsu, (d) K-means Fig. 9Open in figure viewerPowerPoint Comparison 10 for the segmentation results by the proposed approach based on different clustering methods (a) SDD, (b) EM, (c) Otsu, (d) K-means Fig. 10Open in figure viewerPowerPoint Comparison 11 for the segmentation results by the proposed approach based on different clustering methods (a) SDD, (b) EM, (c) Otsu, (d) K-means Fig. 11Open in figure viewerPowerPoint Comparison 12 for the segmentation results by the proposed approach based on different clustering methods (a) SDD, (b) EM, (c) Otsu, (d) K-means We use two measures, F-measure and average perpendicular distance (APD) to compare these four clustering methods quantitatively. The quantitative comparison is shown in Table 1. The one-way Analysis of Variance (ANOVA) test was conducted to judge if the results obtained with the best performing algorithm differ from the final results of the competitors in a statistically significant way [39, 40]. The results are shown in the two middle columns ofTable 2. Wilcoxon's rank-sum test was also conducted to judge the statistical significance of the performance differences between the SDD method and the compared methods [39, 40]. The results are shown in the last column of Table 2. As can be seen, the SDD clustering method is more accurate than the EM method and the Otsu's method, but not statistically significant. The first three methods are more accurate than the K-means method, which is statistically significant. Table 1. Quantitative comparison of the two-label segmentation accuracy by the proposed approach based on different clustering methods with these 12 images F-measure APD SDD-based segmentation 0.9615 1.3453 EM-based segmentation 0.9534 1.5621 Otsu-based segmentation 0.9522 1.6893 K-means based segmentation 0.6792 8.9340 Table 2. Results of the ANOVA test and the Wilcoxon's rank-sum test F Prob > F p SDD-based segmentation NA NA NA EM-based segmentation 2.42 0.0925 0.3431 Otsu-based segmentation 1.25 0.3666 0.2689 K-means based segmentation 0.38 0.9208 0.0001 4.2 Comparison of the proposed approach with state-of-the-art approaches First, we compared the proposed colour image segmentation approach with state-of-the-art colour image segmentation methods on the Weizmann data set [41]. The 12 images used in [41] for qualitative evaluations are segmented by the proposed approach. The segmentation results are shown in Fig. 12 with the extracted boundary overlaying on the original image. By comparing these results with the qualitative results shown in [41], it is seen that the segmentation accuracy of the proposed approach is significantly more accurate than the bottom-up aggregation approach and also more accurate than the compared methods, N-cut [42], mean-shift [43] and label field fusion model (LFFM) [1]. The quantitative comparisons based on the one-object Weizmann data set and the two-object Weizmann data set are shown in Table 3. As can be seen, the proposed approach achieves the highest F-measure scores in both data sets. Fig. 12Open in figure viewerPowerPoint Segmentation results by the proposed approach based on SDD clustering method for the images used in [32] Table 3. Quantitative comparison of the segmentation accuracy of the proposed approach with state-of-the-art methods based on Weizmann database F-measure (one object) F-measure (two object) proposed 0.91 0.84 bottom-up [41] 0.86 0.68 N-cut [42] 0.72 0.58 mean shift [43] 0.57 0.61 LFFM [1] 0.82 0.77 The value in bold type represents the best. Second, we compared the proposed approach with state-of-the-art methods on the Berkeley BSDS300 data set. The quantitative comparison is shown in Table 4. The used measures include probabilistic rand index (PRI) (the higher the better) [44], variation of information (VoI) (the lower the better) [17], global consistency error (GCE) (the lower the better) [45] and boundary displacement error (BDE) (the lower the better) [46]. As can be seen, the proposed approach is significantly more accurate than state-of-the-art methods. Table 4. Comparison of the proposed approach with state-of-the-art methods on Berkeley data set BSDS300 Methods/measures Proposed N-cuts [42] Mean shift [43] LFFM [1] PRI [44] 0.8194 0.7229 0.7550 0.7882 VoI [17] 1.9417 2.9329 2.477 2.2071 GCE [45] 0.1566 0.2182 0.2598 0.1752 BDE [46] 7.8567 9.6038 9.7001 8.7754 The value in bold type represents the best. Third, the MSRA-B data set [47, 48] was used to evaluate the proposed approach. In [45], the proposed SN method was compared with SDD clustering both quantitatively and qualitatively. However, the parameters of SDD clustering were not selected properly. Consequently, the SDD clustering did not achieve the best performance among the six compared methods. In this paper, the parameter of fitting number was chosen as 7 instead of 15 in [47] and the parameter of bandwidth was chosen as 25 instead of 5 in [47]. The same images from MSRA-B database used in [47] were used in this paper to compare the SN method and the proposed approach. The qualitative comparisons are shown in Fig. 13. Fig. 13a shows the original images and Fig. 13b shows the ground truths. Fig. 13c shows the results segmented by SN method and Fig. 13d shows the results segmented by SDD clustering method without adopting the proper parameters. Fig. 13e shows the results of the proposed approach based on SDD clustering with proper parameters. As can be seen, the proposed approach is significantly more accurate than the SN method proposed in [47]. The corresponding quantitative comparisons were shown in Table 5. Fig. 13Open in figure viewerPowerPoint Qualitative comparison of the proposed approach with SN [35] (a) Original images, (b) Ground truths, (c) Results of SN, (d) Results of SDD without selecting the right parameters in [36], (e) Results of the proposed approach Table 5. Quantitative comparison of the accuracy of the proposed approach with the SN method [47] on MSRA-B images Images\accuracy F-measure of SN [47] F-measure of the proposed approach Traffic 1 0.8951 0.9146 Traffic 2 0.8969 0.9042 Traffic 3 0.9881 0.9880 Traffic 4 0.8763 0.9904 Newspaper 1 0.9817 0.9885 Corn 0.9673 0.9928 Butterfly 0.8941 0.9477 Flower 1 0.9844 0.9623 Nature 5 0.9656 0.9695 Animal 5 0.9599 0.9175 Animal 6 0.8274 0.9072 Human 3 0.9875 0.9548 Boat 1 0.8040 0.9565 Scissors 0.9649 0.8991 Bird 1 0.7937 0.9794 Average 0.9191 0.9515 The value in bold type represents the best. Fourth, we compared the proposed colour image segmentation approach with deep learning methods [48–50] on the MSRA-B data set. The quantitative results are shown in Table 6. As can be seen, the proposed approach is more accurate than multi-scale deep features (MDFs) method [49] and deep saliency (DS) method [50] while less accurate than deep supervised saliency (DSS) method [48]. The qualitative comparisons with six typical images are shown in Fig. 14. Please note that the qualitative results of [49–51] are downloaded from their provided websites and the qualitative results of [48] are not available. It can be seen from the qualitative comparisons that the proposed approach achieved overall better results. Fig. 14Open in figure viewerPowerPoint Qualitative comparison of the proposed approach with state-of-the-art methods (a) Original images, (b) Ground truths, (c) Results of MDF [49], (d) Results of DS [50], (e) Results of HS [51], (f) Results of the proposed approach Table 6. Quantitative comparison of the accuracy of the proposed approach with deep learning methods on MSRA-B data set Methods\accuracy F-measure proposed 0.8992 DSS [48] 0.936 MDF [49] 0.885 DS [50] 0.872 It is worth mentioning that the authors of [48] created the MSRA-B data set and many ground truths of MSRA-B database are biased. We show some typical biased results in Fig. 15. We overlay the boundaries of the segmentation results by the proposed approach in red and the boundaries of the ground truths in green on the original images. These biased ground truths decreased the F-measures computed by the proposed approach or other methods significantly while possibly increasing the F-measures of the method proposed in [48] significantly. Fig. 15Open in figure viewerPowerPoint Demonstration of the biased ground truths of the MSRA-B database (the red contours are the results of the proposed approach and the green contours are the ground truths) (a) Horse 1, (b) Human 1, (c) Human 2 4.3 Discussion It is challenging to segment the colour images because there are usually many colours in the scene and they might be further interfered by different lightness. Currently, it seems intractable for multiple-label segmentation of the complex colour images since they might be divided into too many different regions by different users while the ground truths are variable. For instance, the segmentation results in [1] are very ambiguous with the same object divided into different parts while the segmentation results in [11] are very clear. Hence, we prefer the Weizmann data set with clear ground truths even if some ground truths are not very accurate. In this paper, the proposed