New internal clustering validation measure for contiguous arbitrary‐shape clusters

International Journal of Intelligent SystemsVolume 36, Issue 10 p. 5506-5529 RESEARCH ARTICLEOpen Access New internal clustering validation measure for contiguous arbitrary-shape clusters Juan Carlos Rojas-Thomas, Corresponding Author Juan Carlos Rojas-Thomas correorojas@gmail.com orcid.org/0000-0002-3654-7900 Informatics and Automatic Control, UNED, Madrid, Spain Correspondence Juan Carlos Rojas-Thomas, Informatics and Automatic Control, UNED, C/Juan del Rosal s/n, 28040-Madrid, Spain.Email: correorojas@gmail.comSearch for more papers by this authorMatilde Santos, Matilde Santos orcid.org/0000-0003-1993-8368 Institute of Knowledge Technology, University Complutense of Madrid, Madrid, SpainSearch for more papers by this author Juan Carlos Rojas-Thomas, Corresponding Author Juan Carlos Rojas-Thomas correorojas@gmail.com orcid.org/0000-0002-3654-7900 Informatics and Automatic Control, UNED, Madrid, Spain Correspondence Juan Carlos Rojas-Thomas, Informatics and Automatic Control, UNED, C/Juan del Rosal s/n, 28040-Madrid, Spain.Email: correorojas@gmail.comSearch for more papers by this authorMatilde Santos, Matilde Santos orcid.org/0000-0003-1993-8368 Institute of Knowledge Technology, University Complutense of Madrid, Madrid, SpainSearch for more papers by this author First published: 06 June 2021 https://doi.org/10.1002/int.22521AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In this study a new internal clustering validation index is proposed. It is based on a measure of the uniformity of the data in clusters. It uses the local density of each cluster, in particular, the normalized variability of the density within the clusters to find the ideal partition. The new validity measure allows it to capture the spatial pattern of the data and obtain the right number of clusters in an automatic way. This new approach, unlike the traditional one that usually identifies well-separated compact clouds, works with arbitrary-shape clusters that may be contiguous or even overlapped. The proposed clustering measure has been evaluated on nine artificial data sets, with different cluster distributions and an increasing number of classes, on three highly nonlinear data sets, and on 17 real data sets. It has been compared with nine well-known clustering validation indices with very satisfactory results. This proves that including density in the definition of clustering validation indices may be useful to identify the right partition of arbitrary-shape and different-size clusters. 1 INTRODUCTION The process of clustering consists of classifying in an unsupervised manner a set of data in groups. Clustering algorithms group objects that are similar to each other in clusters.1 Different clustering algorithms have been developed and applied to different data sets of very varied fields (medical databases, images, etc.). Although the different clustering algorithms have been proved very efficient in grouping data, one of the most important issues in clustering analysis is the evaluation of the performance of the algorithm.2, 3 This involves not only finding the right number of clusters but also the group partition that best fits the underlying structure of the data. This is the main objective of clustering validation.4-6 Clustering validity indices (CVIs) are defined to do so.7 Among them, internal validation measures have the advantage of not requiring extra information about the data, nor do they require repetition of the clustering process. They only depend on some properties of the resulting clusters, such as the level of compactness, the separation between clusters, and the degree of roundness.8 Most of the internal indices are based on these two concepts, separation among clusters and the cluster compactness, although they differ in the way they measure them. Moreover, these indices usually assume that clusters form compact data clouds with a certain degree of separation. Nevertheless, for clusters not necessarily spherical these relative validity indices may fail.9 Indeed, the existing CVIs are sensitive to clusters with arbitrary shapes, especially for high-dimensional data. The traditional validation measures are considerably dependent on the number of data objects in clusters, on cluster centers, and on average values and thus, do not work well for clusters with different densities and/or sizes.10 Comparative studies of such indices show that there is no optimal CVI able to cope successfully with all the contexts.11, 12 In this study, the clustering evaluation of data sets that may be not composed of spherical clusters is addressed. A new internal validation measure is proposed that is applied to contiguous clusters, that is, adjacent clusters with different shapes and sizes that form a single data cloud. This new index is based on a characteristic of the clusters called uniformity. It combines the spatial pattern of the data and the local density. That is, the Contiguous Density Region (CDR) index detects clusters that follow a uniform spatial pattern, and differentiates them based on their densities. The minimization of the CDR index estimates the optimal partition of the data among all the partitions generated by the clustering algorithm based on the structural characteristics of the data; therefore, it is a criterion to automatically select the number of clusters. The performance of this new internal index has been compared with other traditional validation measures on different artificial and real data sets. Results prove that the spatial texture-based index performs better than the traditional approach that assumes well-separated compact clouds of points. It is able to deal with contiguous clusters of arbitrary shapes that can even be overlap. The main goal of this study is to extend the internal evaluation of data clustering to cluster configurations in which the main discriminant is the spatial structure of the data. The utility of the proposal of new internal indices is that these CVIs can be applied to nonspherical clusters and it is motivated by the fact that real-world data sets have usually arbitrary shapes.13 According to the authors' experience, in many supervised classification problems the structure of the data is much closer to such a configuration than the traditional one that considers well-separated compact clouds. Moreover, it is shown how including the concept of data density within a cluster in the definition of the internal clustering validation indices helps handle different shape, density, and/or size contiguous clusters. This way we think we contribute to enriching the range of tools available to analyze the data. The rest of the paper is organized as follows. Section 2 describes the state-of-the-art regarding the use of density in clustering. Section 3 presents the new contiguous region paradigm. The definition of the new index is proposed in Section 4. In Section 5, experimental results of the CVI for artificial and real data sets are shown and discussed. Finally, Section 6 summarizes the main conclusions and future works. 2 RELATED WORK The new proposed CVI uses the local density to generate partitions. Data density measure has been widely used in the definition of clustering algorithms14 but not in the validation measures definition. The algorithms based on this data characteristic are classified into two categories: density-based and grid-based clustering algorithms. While the first ones calculate the density in the local neighborhood of the data, the latter uses the whole gridding space to estimate the density of each cell, which is then used to obtain the clusters. Clusters are defined as dense regions separated by low-density regions. Both use the data density to detect well-separated data clouds, following the traditional approach, although the calculation of the density of the clusters allows these methods to detect clusters of nonspherical geometries. A density-based algorithm needs to scan only once the original data set and can handle noise. The number of clusters is not required in advance.15 These methods commonly define density in terms of a number of neighbors located in a restricted hypervolume centered on data. A baseline algorithm of this category is DBSCAN,16 which uses as density the number of data contained within a hypersphere of radius fixed a priori. A new version of this algorithm, DBCLASD, that uses reverse nearest-neighbor counts as an estimate of observation density has been recently developed.17 Other density-based approaches are presented in References [18,19]. In the latter, the DENCLUE algorithm identifies clusters by determining density-attractors and clusters of arbitrary shape and noise can easily be described by a simple density function. On the other hand, the grid-based clustering algorithms use dense grid cells to form clusters.20 A typical example is the GRIDCLUS algorithm,21 which calculates the cell density by dividing the number of data contained in each block by its hypervolume. One of the problems of grid-based clustering algorithms is how to choose the grid size or density thresholds. Another difficulty is the exponential growth in the number of cells as the number of dimensions in the feature space increases. As a consequence, many of these algorithms are only efficient in spaces of low-dimensional characteristics or work with optimized partitions for each dimension of the space.22 In Reference [23] a new clustering validity index (BCVI) is designed to better evaluate the quality of clustering results generated by the grid-k-means algorithm. In spite of the extensive literature on density-based clustering algorithms, and on relative internal validation criteria, not much attention has been given to density-based clustering validation measures. In fact, it is very difficult to find studies about clustering validation for dense, sparse, and arbitrary-shaped clusters.24 Some works have developed internal indices for different-size or density arbitrary-shape clusters, but they are not based on the density neither is this concept included in the index definition.9 For example, Žalik and Žalik10 propose two validation indices for clusters that differ in sizes and densities. They do not use the density to define the index. Instead, they use the compactness to define the first index, and the overlap measure of the clusters for the second index. Similarly, the index proposed in Reference [25] applies cluster compactness, defined as the average of all its diameters, and the intercluster distance. The paper by Rodríguez et al.26 proposes an ensemble of different supervised classifiers to obtain the quality of partitions not having a hypersphere underlying structure. Lee et al.11 propose a Support Vector Data Description (SVDD)-based CVI, in which the compactness of a cluster is measured in the kernel space, in an attempt to overcome the sensitivity of this measure for arbitrary shapes, subclusters, and noise in data. In a related paper, a relative validation index for density-based, arbitrarily shaped clusters, is proposed.9 The index assesses clustering quality based on the relative density connection between pairs of objects. It uses a new distance measure to build a minimum spanning tree for each cluster, similarly to Reference [27], where two different versions of a new internal index for clustering validation based on graphs were proposed. Other works take into account the spatial distribution of the data, such as in Reference [28], where authors propose a new internal cluster validity index based on the cluster center; the nearest-neighbor cluster is designed according to the geometric distribution of objects. The work presented in Reference [29] evaluates the performance of a set of internal clustering indices regarding a specific structural characteristic. Particularly, it deals with data sets whose clusters present asymmetry in their geometries. The work by Nerurkar et al.30 focuses on the use of internal validation criteria, in particular the BetaCV and Dunn internal indices, as cost functions of swarm optimizers. In a similar way, Reséndiz, Castro, and Leal31 propose the use of clustering validation indices and maximum entropy as cost functions to quantify the quality of automatic image segmentation. A recent paper by Xu et al.18 proposes a unified validity index framework for complex data structure using hierarchical clustering. Finally, we have studied in depth the closest recent papers that deal with internal CVIs that use the concept of density and analyzed the main differences with our proposal. Hu and Zhong32 design a new robust density-involved distance measure. They then define an internal validity index based on this separation measure using minimum spanning trees. The index does not measure the separation of the clusters by their global densities but the similarity between two particular objects, which is the main difference with our approach. In fact, the artificial data sets used to test the index performance present very small adjacency between clusters, mostly defined by spherical and elliptical geometries. A new Local Cores-Based Cluster Validity (LCCV) index is proposed by Cheng et al.33 Local cores, with local maximum density, are selected as representative points. A neighbor-based local density is defined and then the graph-based distance is used to evaluate the dissimilarity between local cores. The index has been shown to be effective for data sets that contain clusters with arbitrary shapes, but it does not apply to contiguous or overlapping clusters. In a more recent paper by the same authors,34 they use local density peaks to represent the whole data set and define a shared neighbors-based distance between local density peaks to better measure the dissimilarity between objects on varied data. Similarly, Xie et al.35 propose a density-core-based clustering validation index with a minimum spanning tree to solve the problem of noisy and arbitrary-shape clusters. In Reference [36], a new version of the Davies–Bouldin index uses a cylindrical distance to estimate the degree of separation between clusters. This distance definition captures the data density in a limited region of the space around straight line segments that connect the clusters centroids. Therefore, to the best of our knowledge, density has been hardly included in the definition of internal clustering validation indices. That is why we propose to use this measure to define a new clustering validation measure to deal with different density and/or size contiguous clusters. 3 NEW DENSITY-BASED APPROACH AND CDR DENSITY-BASED INTERNAL INDEX Traditional clustering considers clusters as compact, well-separated clouds of data, with more or less complex geometries. For this reason, internal validation indices are mainly defined in terms of compactness and separability, and classified according to the way these two criteria are combined. However, Zahn37 proposed the so-called density gradient problem. He worked with a family of graph–theory algorithms based on the minimal spanning tree that is capable of detecting different types of cluster structures in arbitrary data sets. This framework is shown in Figure 1 (left), where the clusters are represented as two adjacent homogeneous regions; the main difference between them lies in density rather than in distance. Zhong et al.38 continued with this approach using an algorithm that generated the minimal spanning tree in two steps: first to detect separated clusters and then adjacent clusters. Figure 1Open in figure viewerPowerPoint Density gradient problem (left) (Zahn, 1971)37; contiguous region paradigm with three adjacent clusters (right) [Color figure can be viewed at wileyonlinelibrary.com] Inspired by this density gradient problem, we have extended it and proposed a new clustering paradigm that allows us to define internal validation indices that are able to deal with noisy and arbitrary-shape clusters. This new approach is called Contiguous Region Paradigm. It considers any n number of adjacent clusters. Moreover, it does not impose any restriction on the geometry of the whole data set, allowing the exploration of different spatial data configurations, including nonspherical ones. Besides, it is possible to consider different degrees of overlapping between neighboring clusters. The clusters are considered as relatively homogeneous data clouds regarding their own density (local density), and they are adjacent to each other (Figure 1, right, three adjacent homogeneous clusters with different densities). The distance of the data to the nearest neighbors is usually used to calculate the density of the cluster. But in homogeneous clusters the distance is similar for all the data. That is why our proposal does not only try to identify the clusters based on the estimated local densities of the data, but also it mainly focuses on capturing the degree of homogeneity of these clouds, that is, the degree of uniformity of the distribution of the data in the feature space, understanding uniformity as a measure of the variability of the local density. This property, uniformity, is the key factor to identify the clusters and to define the new validation index. 3.1 New internal clustering validation index definition The new clustering validation measure, called CDR index, identifies the partition that presents the largest difference between the clusters regarding their respective densities, while maintaining the degree of density variation within each cluster the lowest possible. To measure it, once different partitions (with different number of clusters) have been obtaining by any clustering algorithm, we first obtain the absolute value of the variability of the local densities for each cluster. Then, these variability measures are normalized. Normalization is carried out by calculating the relative variability of the local densities with respect to the average value of each cluster local density. This feature of each cluster is the uniformity. These normalized density variation values are sorted in increasing order in relation to the number of clusters of their respective partitions. Finally, the partition that shows the greatest relative improvement is selected. The mathematical formulation of the index definition uses the following nomenclature: S is the input data set, with ntot objects or points, xi; g is the center of whole data set S; s is the number of dimensions of S; k is the number of clusters; Ci is the ith cluster, ni is the number of data points in Ci, ci is the center of cluster Ci, and d(x, y) = ||x − y||2 is the distance between any two points x and y. Definition 1.Local density. The clusters to be analyzed form a single continuous cloud of data, so to establish a density threshold to distinguish dense from empty regions is no longer useful. It is necessary to calculate the density of each region. The local density of an object, xi, in the feature space S is defined as the distance to the nearest neighbor, xj, that belongs to the same cluster, expressed as local _ den ( x i ) x i ∈ C k = min { d ( x i , x j ) } , ∀ x j ∈ C k , i ≠ j ,(1)where d() can represent any distance measure, and Ck is the kth cluster. We have selected the Euclidean distance due to its extensive and proven use in the literature on clustering validation measures.39 In addition, all the internal indices with which the proposal has been compared to use this distance measure as well. When the Euclidean distance is used, it is denoted as de(). Hence, local _ den e ( x i ) x i ∈ C k = min { d e ( x i , x j ) } , ∀ x j ∈ C k , i ≠ j .(2) Definition 2.Density of a cluster. The average density of a cluster Ck is calculated as the average of all the local densities of the cluster, that is, avg _ den ( C k ) = ∑ i = 1 n k local _ den ( x i ) n k , ∀ x i ∈ C k ,(3)where nk is the total number of data in cluster Ck. Definition 3.Uniformity. It measures the degree of local density variation within a cluster. Given a cluster Ck, its uniformity is given by u n i f C k = ∑ i = 1 n k l o c a l _ d e n ( x i ) − a v g _ d e n ( C k ) a v g _ d e n ( C k ) , x i ∈ C k , n k > 1 , 0 , n k = 1 . (4) Thus, the uniformity of each cluster is calculated as the difference between the local density of the cluster and the average of all the local densities of the cluster. The goal is to find the cluster that presents the smallest variability, that is, the most uniform spatial data distribution. Thus, the lower the value of (4), the greater the uniformity. Definition 4.Contiguous Density Region (CDR) index. Given a partition Pk = {C1, …, Ck} of a data set S, with k clusters, the value of the CDR index is defined as CDR P k ) = ∑ i = 1 k n i ∗ unif ( C i ) n tot ,(5)where ni is the number of data in cluster Ci, and ntot corresponds to the total number of data of data set S. Equation (5) represents a weighted sum of the degree of uniformity of each individual cluster. The relative weight of each cluster depends on the percentage of data that belong to the cluster, out of the total number of data of the set. This way, it is guaranteed that the index favors partitions with significant clusters, and discards little relevant clusters, such as those formed by very few data or even a single data. The goal of this index is to identify the cluster with the highest degree of uniformity regarding the density, so this index must be minimized (as it is usually the case for internal validation measures). 3.2 CDR index calculation Once these key terms have been defined, the minimum of the CDR index is calculated as follows. 1. Let R = {P1, …, Pm} be the set of the different partitions generated by any clustering algorithm applied to the S data set. They are sorted according to the number of clusters each one has (from 1 to m clusters). The CDR index is calculated for each of these partitions. CDR ( R ) = { CDR ( P 1 ) , CDR ( P 2 ) , … , CDR ( P m ) } .(6) Figure 2 shows an example of the values of the CDR index for 10 different partitions of a given data set. The horizontal axis represents the number of clusters of each partition, and the vertical axis represents the values of the CDR index. The searching of the local minimum is represented by the red arrow. The process of finding the minimum index value starts with CDR(P2) (the smallest solution is the one with at least two clusters), and continues as long as CDR(Pi) > CDR(Pi+1), where i is the number of clusters of the partition, 1 < i ≤ m. When this condition is no longer met, that is, CDR(Pi) ≤ CDR(Pi+1), the process stops and the first ith partitions are selected for later analysis (vertical red line in Figure 2, partitions with up to four clusters). 2. The next step is to measure the relative improvement between two consecutive partitions, k and k − 1, through the following factor, CDRf, defined as CDR f k = CDR ( P k ) CDR ( P k − 1 ) , k = 2 , … , m .(7) 3. Finally, the jth partition that has the greatest improvement, that is, the lowest CDR f j value (3), is chosen. j : CDR f j = min m CDR f 2 , … , CDR f m .(8) Figure 2Open in figure viewerPowerPoint Searching for a local minimum of the CDR index values for 10 partitions [Color figure can be viewed at wileyonlinelibrary.com] As it is possible to see in Figure 2, the partition with the smallest CDR value is not necessarily the one with the best CDR improvement factor. In this case, P4 has the minimum CDR but P3 is the one with the lowest CDRf, that is, the best relative improvement of the index. So, P3 would be selected as the best partition according to the CDR index. 4 MATERIALS: ARTIFICIAL AND REAL DATA SETS To test the internal validation measures, initially nine artificial data sets were generated. These synthetic data are used for the validation and design of experiments.40 The artificial data sets consist of a single continuous cloud of data in a two-dimensional space which presents regions with different densities. This description allows a wide range of different spatial distributions of the data. The method of generation of artificial data sets is as follows (Figure 3). Data of the clusters are uniformly distributed in a rectangular grid. The data are placed at the intersections of the lines of the grid. This way a cluster of data linearly spaced with certain density and rectangular shape is obtained. To generate clusters with nonsquare geometries, such as ring-shaped clusters, taking as a starting point the rectangular cluster, two parameters, the outer radius (larger) and the inner radius (smaller) are defined. A mask is applied to remove all the data that are not within the ring. To build clusters with arc geometry, a linear filter is applied to a ring cluster in such a way as to eliminate the data that are over or under a straight line parallel to any of the axes. The same procedure can be applied to obtain clusters with any desired shape. The density of each cluster may vary. Figure 3Open in figure viewerPowerPoint Example of generation of artificial data sets. From left to right: grid, rectangular cluster, ring-shape cluster, and arch-shape cluster [Color figure can be viewed at wileyonlinelibrary.com] Figure 3 shows an example of how the artificial data sets are generated. 4.1 Artificial data sets For the generation of the different configurations of the data, according to the methodology described above, the following two criteria were adopted, where the term class refers to the correct number of clusters, that is, the target partition. 1. Spatial distribution: According to the geometrical distribution of the clusters, the next subcategories are defined: i. Linear: Sets of clusters whose spatial distribution follows a straight line. ii. Nonlinear: Sets of clusters whose spatial distribution is: a. A perpendicular arch: A continuous region of clusters whose shape follows two perpendicular lines. b. A matrix: A continuous region of clusters distributed along and across the space, resembling a matrix of clusters. 2. Superposition: i. Clusters that are not overlapped: Adjacent clusters with well-separated edges. ii. Overlapped clusters: Adjacent clusters with regions partially overlapped. Combining these two criteria, nine different test sets have been built (Figure 4). In Figure 4, the points with the same color represent a cluster. As it is possible to see, the density of each cluster is different. The description of each data set of Figure 4, from top to bottom, left to right, is given below, where the number followed by C means the number of clusters. (a) Linear 2C: two classes, horizontal linear configuration, same area, without overlapping, with 16 and 49 data points, respectively (Figure 4A). (b) Linear 3C: three classes, horizontal linear configuration, same area, without overlapping, with 16, 49, and 100 data, respectively (Figure 4B). (c) Linear 4C: four classes, horizontal linear configuration, same area, without overlapping, with 16, 49, 100, and 169 data, respectively (Figure 4C). (d) Linear 5C: five classes, horizontal linear configuration, same area, without overlapping, with 16, 49, 100, 169, and 256 data, respectively (Figure 4D). (e) Matrix 4C: four classes, matrix configuration, same area, without overlapping, with 16, 49, 100, and 169 data, respectively (Figure 4E). (f) Arc 5C: five classes with a perpendicular arch configuration, same quadrangular area, without overlapping, with 16, 49, 100, 169, and 256 data, respectively (Figure 4F). (g) Overlap 2C: two classes with a horizontal linear configuration, same quadrangular area, overlapped, with 16 and 49 data, respectively (Figure 4G). (h) Overlap 3C: three classes with a horizontal linear configuration, same quadrangular area, overlapped, with 16, 49, and 100 data, respectively (Figure 4H). (i) Overlap 4C: four classes with a horizontal linear configuration, same quadrangular area, overlapped, with 16, 49, 100, and 169 data, respectively (Figure 4H). Figure 4Open in figure viewerPowerPoint From top to bottom, left to right, artificial cluster configurations: (A) Linear 2C, (B) Linear 3C, (C) Linear 4C, (D) Linear 5C, (E) Matrix 4C, (F) Arc 5C, (G) Overlap 2C, (H) Overlap 3C, and (I) Overlap 4C [Color figure can be viewed at wileyonlinelibrary.com] 4.2 Artificial nonlinear data sets Three different artificial data sets, taken from or inspired by the paper published by Cheng et al.,33 have been also considered. They are highly nonlinear data sets, with different geometries and different data densities. Moreover, as it will be shown, these sets have been evaluated using two different clustering methods: k-means and hierarchical algorithms. The description of these generated artificial data sets is the following. (a) Concentric rings: One spherical cluster and two concentric ring-shaped clusters with 81, 112, and 180 data, respectively (Figure 5A). (b) Simple arches: Three simple arc-shaped clusters with 90, 182, and 33