Solving Soft Clustering Ensemble via $k$-Sparse Discrete Wasserstein Barycenter

1 Introduction . Clustering is a fundamental topic that has important applications in various areas , such as data mining , networking , and bioinformatics [ 34 ] . In the past decades , a number of different clustering objectives and algorithms have been proposed . For example , the popular k-means aims to partition the given data set into k clusters and minimize the average squared distance from the input data to the set of cluster centers ; the well known k-means clustering algorithms include the Lloyd ’ s algorithm [ 47 ] , k-means++ [ 5 ] , and local search [ 36 ] . Other clustering objectives , like hierarchical clustering [ 49 ] and density-based clustering [ 53 ] , are also widely used in practice . Obviously , different clustering algorithms can obtain different results . Moreover , even for the same clustering algorithm ( e.g. , the Lloyd ’ s algorithm ) , the initialization and data preprocessing ( e.g. , random projection [ 12 ] ) steps may yield different clustering results . Therefore , a natural idea is to aggregate these different clustering results so as to achieve a more reliable result . The problem is called clustering ensemble ( also termed clustering aggregation or consensus clustering ) [ 27 ] . However , the current methods still suffer from several issues in theory and practice . Most of existing clustering ensemble methods rely on complicated optimization models , such as the correlation clustering [ 28 ] , graph partition [ 25 ] , semi-definite programming [ 54 ] , matrix completion [ 63 ] , and spectral clustering [ 56 ] ; these optimization problems usually have super-linear complexities and thus can not be efficiently solved for large-scale datasets . Though several heuristic ideas have been proposed for speeding up the computation ( e.g. , the sampling idea proposed in [ 61 ] ) , they are in lack of rigorous theoretical analysis on their quality guarantees . Another issue is about the interpretability of consensus . A large number of clustering ensemble models are based on utility function [ 57 , 61 , 60 ] or co-association matrix [ 26 , 56 ] . From the theoretical perspective , a fundamental question is why these models can yield the final clusterings close to ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the ground-truth clustering . The similar consensus question was also studied for the classification problem before [ 13 ] . However , the analysis for clustering ensemble is more challenging , because we need to take into account the matchings between different clustering solutions . 1.1 Our Contributions . In this paper , we focus on the more general soft clustering ensemble problem , where each given individual clustering is a soft clustering ( also referred to as fuzzy clustering ) [ 61 ] . In a soft clustering , data points can potentially belong to multiple clusters . For example , a point may be assigned to three clusters with the probabilities 10 % , 20 % , and 70 % , respectively . Compared with hard clustering , soft clustering can provide more realistic and accurate clustering results in many realworld applications [ 9 ] . We adopt the geometric model that was studied in [ 18 , 20 , 19 ] . They showed that clustering ensemble can be naturally formulated as a “ geometric prototype ” problem . But their results are in lack of systematically studies on the efficiency of this model , especially from the theoretical perspective . In this paper , we illustrate that the geometric prototype actually is equivalent with an instance of Discrete Wasserstein Barycenter ( DWB ) [ 1 ] in high dimensions ( the formal definition will be given in Section 3 ) . This approach falls under the umbrella of utility function based model , where it uses discrete Wasserstein distance to measure the difference between two clusterings . Compared with other utility functions ( e.g. , KL-divergence ) , it has several attractive properties . For example , the discrete Wasserstein distance is symmetric and more robust to noise [ 42 ] . More importantly , the DWB based ensemble model can be easily interpreted from the geometric perspective , and thus we can analyze its performance more conveniently . But when applying the DWB model to the soft clustering ensemble problem , we still have several key issues remaining to be solved . ( i ) Though a number of DWB algorithms have been developed ( as shown in Section 1.2 ) , the clustering ensemble imposes two unique features to the DWB formulation . First , we require the returned DWB to be k-sparse , that is , it should be supported by at most k points in the space ( since there are at most k clusters ) . Also , the number of different clustering solutions can be large in practical scenarios . For instance , to guarantee the consistency of the final ensemble solution to the ground-truth clustering , we may generate a large number of clustering solutions via random initializations or random projections [ 24 ] . So from the algorithmic perspective , a natural question is whether we can develop more efficient algorithm for the DWB problem with such features ? ( ii ) To the best of our knowledge , only Topchy et al . [ 58 ] and Jain [ 35 ] discussed the consensus of clustering ensemble in theory . However , both of their analyses rely on the assumption that the ground-truth clustering should be the optimal solution of the ensemble model , which is too strong and may not be realistic in practice . Also , the analysis of [ 58 ] only considered hard clustering . It is worth noting that the number of hard clusterings on a fixed set of items is finite , but the number of soft clusterings is infinity . In this paper , we focus on these two issues . First , based on some novel geometric insights , we show that it is possible to achieve a fixed-parameter algorithm for the soft clustering ensemble problem if k is a constant , where the obtained approximation factor is 1 + with being any small number in ( 0 , 1 ) ; though this is more a theoretical result , we believe that it is of independent interest for such a combinatorial optimization problem in high dimensions [ 17 ] . Moreover , the proposed sampling idea inspires our following speedup for the existing DWB algorithms with provable quality guarantee , even if k is large . Second , we prove that the obtained DWB should be close to the ground-truth clustering if the number of given clustering solutions is large enough . Our idea is quite different from [ 58 , 35 ] ; in particular , our analysis yields a detailed quantitive result for the consensus . 1.2 Related Works . Clustering ensemble . Clustering ensemble was introduced by Strehl and Ghosh [ 55 ] , where they formulated it as two different graph partitioning problems ; one is “ Instance-Based Graph Formulation ( IBGF ) ” , and the other is “ Cluster-Based Graph Formulation ( CBGF ) ” . Fern and Brodley [ 25 ] later proposed a hybrid graph partitioning model for clustering ensemble . Most existing clustering ensemble models can be roughly divided into utility function based [ 57 , 61 , 60 ] and co-association matrix based [ 26 , 46 , 56 ] . The utility function based model is to find the final clustering result via maximizing the total similarities to the set of given clusterings . A co-association matrix is actually a new representation of the data items , where each entry of the matrix indicates the similarity of a pair of data items based on the information from the given clusterings . The soft clustering ensemble problem was particularly studied in [ 51 , 61 ] . For more detailed discussion about ensemble clustering , the reader is referred to the survey [ 27 ] . Wasserstein distance . The Wasserstein distance is defined for measuring the difference between two probability distributions ; if their supports are both discrete sets , the distance is called discrete Wasserstein distance ( or Earth Mover ’ s distance ) [ 59 , 52 ] . Computing the discrete Wasserstein distance actually is equivalent to solving a min-cost max flow problem [ 2 , 37 ] . Several more efficient discrete Wasserstein distance algorithms were proposed , such as [ 45 , 50 ] . In the community of machine learning , Cuturi [ 15 ] proposed a new objective called “ Sinkhorn Distance ” that smoothes the transportation problem with an entropic regularization term , and it can be solved much faster than computing the exact discrete Wasserstein distance . Following Cuturi ’ s work , several improved Sinkhorn algorithms have been proposed in recent years [ 23 , 44 , 3 , 48 ] . Wasserstein barycenter . If there are m ≥ 2 different weighted point sets , the problem of discrete Wasserstein barycenter is to compute the average pattern that minimizes the total Wasserstein distances to them [ 1 ] . The recent algorithms for computing Wasserstein barycenter include [ 16 , 7 , 29 , 62 , 8 , 4 , 43 ] . The computational complexity of Wasserstein barycenter was studied in [ 39 , 11 , 43 ] . Recently , Dognin et al . [ 21 ] applied the Wasserstein barycenter based ensemble method to solve several supervised learning problems . 2 Preliminaries . We always use A = { a1 , a2 , · · · , an } to denote the set of n data items which we want to cluster . Definition 1 ( Soft Clustering ) . Let k ∈ Z+ . A k-soft clustering of A can be represented by a set of k vectors C = { S1 , · · · , Sk } ⊂ [ 0 , 1 ] n , where each vector Sj represents an individual cluster and∑k j=1 Sj is equal to the row vector [ 1 , 1 , · · · , 1 ] . Suppose Sjl is the l-th entry of Sj for 1 ≤ l ≤ n. Then , for a fixed l , the set { S1l , · · · , Skl } indicates the degrees of membership of al to the k clusters . For example , suppose n = 3 and k = 2 ; the following is a k-soft clustering of A : C = { S1 = [ 0.1 , 0.3 , 0.8 ] , S2 = [ 0.9 , 0.7 , 0.2 ] } ( e.g. , the item a2 belongs to the first and second clusters with probability 30 % and 70 % , respectively ) . In Definition 1 , if we restrict each Sj to be binary vector , the clustering C will be a hard clustering . Also , if a clustering C has k′ < k clusters , we can simply add k − k′ dummy clusters where each dummy cluster is just a zero vector [ 0 , 0 , · · · , 0 ] . Following the idea of [ 18 , 6 , 20 ] , we define the function ∆ to measure the difference between two clusterings of A . Suppose C = { S1 , · · · , Sk } and C′ = { S′1 , · · · , S′k } are two different soft clusterings . We build the bipartite graph G from C and C′ as follows : each of the two columns of G contains k vertices corresponding to the k clusters in C and C′ respectively ; for any pair of clusters ( Sj , S ′ j′ ) with Sj ∈ C and S′j′ ∈ C′ , there is an edge connecting their corresponding vertices in G with a weight equal to their squared Euclidean distance ||Sj − S′j′ ||2 . Then , the difference of C and C′ , i.e. , ∆ ( C , C′ ) , is the cost of the minimum weight bipartite matching of G. The function ∆ can be computed through the Hungarian algorithm [ 14 ] . Assume the minimum weight bipartite matching between C and C′ yields the permutation π of { 1 , 2 , · · · , k } . Namely , ∆ ( C , C′ ) = k∑ j=1 ||Sj − S′π ( j ) || 2 . ( 1 ) Obviously , if ∆ ( C , C′ ) = 0 , they should be the same clustering solution ( the k clusters of C is just reordered by π in C′ ) . To see the rationale behind ( 1 ) , we can imagine the case that C and C′ are both hard clusterings , i.e. , each Sj ∈ C ( resp. , S′j′ ∈ C′ ) is a binary vector ; so each such vector can be viewed as a subset of A ( e.g. , Sj represents the set { al | al ∈ A and Sjl = 1 } ) . It is easy to know the symmetric difference of Sj and S′j′ , |Sj \ S′j′ |+ |S′j′ \ Sj | , is equal to ||Sj − Sj′ ||2 . Therefore , computing the function ∆ ( C , C′ ) in fact is to find the matching of C and C′ that minimizes their total symmetric differences . Following Definition 1 , we then introduce “ soft clustering ensemble ” below . Definition 2 ( Soft Clustering Ensemble ( SCE ) ) . Given m different soft clusterings C1 , · · · , Cm of A , the problem of soft clustering ensemble ( SCE ) is to find the final soft clustering C̃ that minimizes the objective function 1 m m∑ i=1 ∆ ( C̃ , Ci ) . ( 2 ) For any soft clustering C̃′ and λ ≥ 1 , if it achieves an objective value no larger than λ times the minimum value of ( 2 ) , we say it is a “ λ-approximation ” for the SCE problem . Let πi be the permutation between C̃ and Ci for 1 ≤ i ≤ m. To minimize the objective function ( 2 ) , the major challenge is to find thesem permutations simultaneously . Suppose each Ci = { Si1 , · · · , Sik } . Once these m permutations are obtained , the set ∪mi=1Ci is divided into k parts : { S1π1 ( j ) , · · · , S m πm ( j ) } , 1 ≤ j ≤ k. ( 3 ) If we let the optimal solution C̃ be { S̃1 , · · · , S̃k } , from the objective function ( 2 ) it is easy to know these k soft clusters should be the means of these k parts , i.e. , S̃j = 1 m m∑ i=1 Siπi ( j ) , 1 ≤ j ≤ k. ( 4 ) This simple fact will be used in the analysis in our paper . We also have the following hardness result for SCE through the reduction from the NP-hard three-dimensional assignment problem [ 41 ] . Theorem 1 ( The hardness ) . When m ≥ 3 , optimizing the SCE objective ( 2 ) is NP-hard . The rest of the paper is organized as follows . In Section 3 , we discuss the relation between the SCE problem and discrete Wasserstein barycenter . In Section 4 , we present our approximation algorithms based on random sampling . In Section 5 , we analyze the consensus of the SCE problem under Definition 2 . Finally , we illustrate our experimental results in Section 6 . Due to the space limit , we leave some proofs and the detailed experimental results to the full version of this paper .

Summary: This paper proposes a new approach for soft clustering ensemble, using a new problem formulation (a “geometric prototype”), which is equivalent to a Discrete Wasserstein Barycenter problem.  The proposed algorithm achieves, with a fixed probability 1 – δ, a (1+ ε)-approximate solution to the soft clustering ensemble problem, when the number of clusters k is fixed and small. As the authors state, this is a theoretical result since the runtime is more than exponential in k, 1/ δ, and 1/ε.  In addition, the authors propose an improvement on the existing alternating minimization algorithms in the case of k not being a constant. Then, they prove that the obtained solution converges to the ground-truth clustering as the number m of clustering increases.  Finally, they evaluate the performance of the alternating minimization Wasserstein barycenter algorithm with the proposed improvement, and they compare it to three baselines from the literature. The results show that their algorithm outperforms the baselines in terms of the objective function value and, for the largest datasets, also in terms of runtime.