Solving Soft Clustering Ensemble via $k$-Sparse Discrete Wasserstein Barycenter

1 Introduction . Clustering is a fundamental topic that has important applications in various areas , such as data mining , networking , and bioinformatics [ 34 ] . In the past decades , a number of different clustering objectives and algorithms have been proposed . For example , the popular k-means aims to partition the given data set into k clusters and minimize the average squared distance from the input data to the set of cluster centers ; the well known k-means clustering algorithms include the Lloyd ’ s algorithm [ 47 ] , k-means++ [ 5 ] , and local search [ 36 ] . Other clustering objectives , like hierarchical clustering [ 49 ] and density-based clustering [ 53 ] , are also widely used in practice . Obviously , different clustering algorithms can obtain different results . Moreover , even for the same clustering algorithm ( e.g. , the Lloyd ’ s algorithm ) , the initialization and data preprocessing ( e.g. , random projection [ 12 ] ) steps may yield different clustering results . Therefore , a natural idea is to aggregate these different clustering results so as to achieve a more reliable result . The problem is called clustering ensemble ( also termed clustering aggregation or consensus clustering ) [ 27 ] . However , the current methods still suffer from several issues in theory and practice . Most of existing clustering ensemble methods rely on complicated optimization models , such as the correlation clustering [ 28 ] , graph partition [ 25 ] , semi-definite programming [ 54 ] , matrix completion [ 63 ] , and spectral clustering [ 56 ] ; these optimization problems usually have super-linear complexities and thus can not be efficiently solved for large-scale datasets . Though several heuristic ideas have been proposed for speeding up the computation ( e.g. , the sampling idea proposed in [ 61 ] ) , they are in lack of rigorous theoretical analysis on their quality guarantees . Another issue is about the interpretability of consensus . A large number of clustering ensemble models are based on utility function [ 57 , 61 , 60 ] or co-association matrix [ 26 , 56 ] . From the theoretical perspective , a fundamental question is why these models can yield the final clusterings close to ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the ground-truth clustering . The similar consensus question was also studied for the classification problem before [ 13 ] . However , the analysis for clustering ensemble is more challenging , because we need to take into account the matchings between different clustering solutions . 1.1 Our Contributions . In this paper , we focus on the more general soft clustering ensemble problem , where each given individual clustering is a soft clustering ( also referred to as fuzzy clustering ) [ 61 ] . In a soft clustering , data points can potentially belong to multiple clusters . For example , a point may be assigned to three clusters with the probabilities 10 % , 20 % , and 70 % , respectively . Compared with hard clustering , soft clustering can provide more realistic and accurate clustering results in many realworld applications [ 9 ] . We adopt the geometric model that was studied in [ 18 , 20 , 19 ] . They showed that clustering ensemble can be naturally formulated as a “ geometric prototype ” problem . But their results are in lack of systematically studies on the efficiency of this model , especially from the theoretical perspective . In this paper , we illustrate that the geometric prototype actually is equivalent with an instance of Discrete Wasserstein Barycenter ( DWB ) [ 1 ] in high dimensions ( the formal definition will be given in Section 3 ) . This approach falls under the umbrella of utility function based model , where it uses discrete Wasserstein distance to measure the difference between two clusterings . Compared with other utility functions ( e.g. , KL-divergence ) , it has several attractive properties . For example , the discrete Wasserstein distance is symmetric and more robust to noise [ 42 ] . More importantly , the DWB based ensemble model can be easily interpreted from the geometric perspective , and thus we can analyze its performance more conveniently . But when applying the DWB model to the soft clustering ensemble problem , we still have several key issues remaining to be solved . ( i ) Though a number of DWB algorithms have been developed ( as shown in Section 1.2 ) , the clustering ensemble imposes two unique features to the DWB formulation . First , we require the returned DWB to be k-sparse , that is , it should be supported by at most k points in the space ( since there are at most k clusters ) . Also , the number of different clustering solutions can be large in practical scenarios . For instance , to guarantee the consistency of the final ensemble solution to the ground-truth clustering , we may generate a large number of clustering solutions via random initializations or random projections [ 24 ] . So from the algorithmic perspective , a natural question is whether we can develop more efficient algorithm for the DWB problem with such features ? ( ii ) To the best of our knowledge , only Topchy et al . [ 58 ] and Jain [ 35 ] discussed the consensus of clustering ensemble in theory . However , both of their analyses rely on the assumption that the ground-truth clustering should be the optimal solution of the ensemble model , which is too strong and may not be realistic in practice . Also , the analysis of [ 58 ] only considered hard clustering . It is worth noting that the number of hard clusterings on a fixed set of items is finite , but the number of soft clusterings is infinity . In this paper , we focus on these two issues . First , based on some novel geometric insights , we show that it is possible to achieve a fixed-parameter algorithm for the soft clustering ensemble problem if k is a constant , where the obtained approximation factor is 1 + with being any small number in ( 0 , 1 ) ; though this is more a theoretical result , we believe that it is of independent interest for such a combinatorial optimization problem in high dimensions [ 17 ] . Moreover , the proposed sampling idea inspires our following speedup for the existing DWB algorithms with provable quality guarantee , even if k is large . Second , we prove that the obtained DWB should be close to the ground-truth clustering if the number of given clustering solutions is large enough . Our idea is quite different from [ 58 , 35 ] ; in particular , our analysis yields a detailed quantitive result for the consensus . 1.2 Related Works . Clustering ensemble . Clustering ensemble was introduced by Strehl and Ghosh [ 55 ] , where they formulated it as two different graph partitioning problems ; one is “ Instance-Based Graph Formulation ( IBGF ) ” , and the other is “ Cluster-Based Graph Formulation ( CBGF ) ” . Fern and Brodley [ 25 ] later proposed a hybrid graph partitioning model for clustering ensemble . Most existing clustering ensemble models can be roughly divided into utility function based [ 57 , 61 , 60 ] and co-association matrix based [ 26 , 46 , 56 ] . The utility function based model is to find the final clustering result via maximizing the total similarities to the set of given clusterings . A co-association matrix is actually a new representation of the data items , where each entry of the matrix indicates the similarity of a pair of data items based on the information from the given clusterings . The soft clustering ensemble problem was particularly studied in [ 51 , 61 ] . For more detailed discussion about ensemble clustering , the reader is referred to the survey [ 27 ] . Wasserstein distance . The Wasserstein distance is defined for measuring the difference between two probability distributions ; if their supports are both discrete sets , the distance is called discrete Wasserstein distance ( or Earth Mover ’ s distance ) [ 59 , 52 ] . Computing the discrete Wasserstein distance actually is equivalent to solving a min-cost max flow problem [ 2 , 37 ] . Several more efficient discrete Wasserstein distance algorithms were proposed , such as [ 45 , 50 ] . In the community of machine learning , Cuturi [ 15 ] proposed a new objective called “ Sinkhorn Distance ” that smoothes the transportation problem with an entropic regularization term , and it can be solved much faster than computing the exact discrete Wasserstein distance . Following Cuturi ’ s work , several improved Sinkhorn algorithms have been proposed in recent years [ 23 , 44 , 3 , 48 ] . Wasserstein barycenter . If there are m ≥ 2 different weighted point sets , the problem of discrete Wasserstein barycenter is to compute the average pattern that minimizes the total Wasserstein distances to them [ 1 ] . The recent algorithms for computing Wasserstein barycenter include [ 16 , 7 , 29 , 62 , 8 , 4 , 43 ] . The computational complexity of Wasserstein barycenter was studied in [ 39 , 11 , 43 ] . Recently , Dognin et al . [ 21 ] applied the Wasserstein barycenter based ensemble method to solve several supervised learning problems . 2 Preliminaries . We always use A = { a1 , a2 , · · · , an } to denote the set of n data items which we want to cluster . Definition 1 ( Soft Clustering ) . Let k ∈ Z+ . A k-soft clustering of A can be represented by a set of k vectors C = { S1 , · · · , Sk } ⊂ [ 0 , 1 ] n , where each vector Sj represents an individual cluster and∑k j=1 Sj is equal to the row vector [ 1 , 1 , · · · , 1 ] . Suppose Sjl is the l-th entry of Sj for 1 ≤ l ≤ n. Then , for a fixed l , the set { S1l , · · · , Skl } indicates the degrees of membership of al to the k clusters . For example , suppose n = 3 and k = 2 ; the following is a k-soft clustering of A : C = { S1 = [ 0.1 , 0.3 , 0.8 ] , S2 = [ 0.9 , 0.7 , 0.2 ] } ( e.g. , the item a2 belongs to the first and second clusters with probability 30 % and 70 % , respectively ) . In Definition 1 , if we restrict each Sj to be binary vector , the clustering C will be a hard clustering . Also , if a clustering C has k′ < k clusters , we can simply add k − k′ dummy clusters where each dummy cluster is just a zero vector [ 0 , 0 , · · · , 0 ] . Following the idea of [ 18 , 6 , 20 ] , we define the function ∆ to measure the difference between two clusterings of A . Suppose C = { S1 , · · · , Sk } and C′ = { S′1 , · · · , S′k } are two different soft clusterings . We build the bipartite graph G from C and C′ as follows : each of the two columns of G contains k vertices corresponding to the k clusters in C and C′ respectively ; for any pair of clusters ( Sj , S ′ j′ ) with Sj ∈ C and S′j′ ∈ C′ , there is an edge connecting their corresponding vertices in G with a weight equal to their squared Euclidean distance ||Sj − S′j′ ||2 . Then , the difference of C and C′ , i.e. , ∆ ( C , C′ ) , is the cost of the minimum weight bipartite matching of G. The function ∆ can be computed through the Hungarian algorithm [ 14 ] . Assume the minimum weight bipartite matching between C and C′ yields the permutation π of { 1 , 2 , · · · , k } . Namely , ∆ ( C , C′ ) = k∑ j=1 ||Sj − S′π ( j ) || 2 . ( 1 ) Obviously , if ∆ ( C , C′ ) = 0 , they should be the same clustering solution ( the k clusters of C is just reordered by π in C′ ) . To see the rationale behind ( 1 ) , we can imagine the case that C and C′ are both hard clusterings , i.e. , each Sj ∈ C ( resp. , S′j′ ∈ C′ ) is a binary vector ; so each such vector can be viewed as a subset of A ( e.g. , Sj represents the set { al | al ∈ A and Sjl = 1 } ) . It is easy to know the symmetric difference of Sj and S′j′ , |Sj \ S′j′ |+ |S′j′ \ Sj | , is equal to ||Sj − Sj′ ||2 . Therefore , computing the function ∆ ( C , C′ ) in fact is to find the matching of C and C′ that minimizes their total symmetric differences . Following Definition 1 , we then introduce “ soft clustering ensemble ” below . Definition 2 ( Soft Clustering Ensemble ( SCE ) ) . Given m different soft clusterings C1 , · · · , Cm of A , the problem of soft clustering ensemble ( SCE ) is to find the final soft clustering C̃ that minimizes the objective function 1 m m∑ i=1 ∆ ( C̃ , Ci ) . ( 2 ) For any soft clustering C̃′ and λ ≥ 1 , if it achieves an objective value no larger than λ times the minimum value of ( 2 ) , we say it is a “ λ-approximation ” for the SCE problem . Let πi be the permutation between C̃ and Ci for 1 ≤ i ≤ m. To minimize the objective function ( 2 ) , the major challenge is to find thesem permutations simultaneously . Suppose each Ci = { Si1 , · · · , Sik } . Once these m permutations are obtained , the set ∪mi=1Ci is divided into k parts : { S1π1 ( j ) , · · · , S m πm ( j ) } , 1 ≤ j ≤ k. ( 3 ) If we let the optimal solution C̃ be { S̃1 , · · · , S̃k } , from the objective function ( 2 ) it is easy to know these k soft clusters should be the means of these k parts , i.e. , S̃j = 1 m m∑ i=1 Siπi ( j ) , 1 ≤ j ≤ k. ( 4 ) This simple fact will be used in the analysis in our paper . We also have the following hardness result for SCE through the reduction from the NP-hard three-dimensional assignment problem [ 41 ] . Theorem 1 ( The hardness ) . When m ≥ 3 , optimizing the SCE objective ( 2 ) is NP-hard . The rest of the paper is organized as follows . In Section 3 , we discuss the relation between the SCE problem and discrete Wasserstein barycenter . In Section 4 , we present our approximation algorithms based on random sampling . In Section 5 , we analyze the consensus of the SCE problem under Definition 2 . Finally , we illustrate our experimental results in Section 6 . Due to the space limit , we leave some proofs and the detailed experimental results to the full version of this paper .

Summary: This manuscript shows the connection between soft clustering ensemble and the discrete Wasserstein barycenter problem. Based on the connection, a sampling-based algorithm is proposed with some provable quality guarantees. The manuscript provides some theoretical analysis about the quality of the obtained consensus clustering results.