Discriminative Similarity for Data Clustering

Similarity-based clustering methods separate data into clusters according to the pairwise similarity between the data , and the pairwise similarity is crucial for their performance . In this paper , we propose Clustering by Discriminative Similarity ( CDS ) , a novel method which learns discriminative similarity for data clustering . CDS learns an unsupervised similarity-based classifier from each data partition , and searches for the optimal partition of the data by minimizing the generalization error of the learnt classifiers associated with the data partitions . By generalization analysis via Rademacher complexity , the generalization error bound for the unsupervised similarity-based classifier is expressed as the sum of discriminative similarity between the data from different classes . It is proved that the derived discriminative similarity can also be induced by the integrated squared error bound for kernel density classification . In order to evaluate the performance of the proposed discriminative similarity , we propose a new clustering method using a kernel as the similarity function , CDS via unsupervised kernel classification ( CDSK ) , with its effectiveness demonstrated by experimental results . 1 INTRODUCTION . Similarity-based clustering methods segment the data based on the similarity measure between the data points , such as spectral clustering ( Ng et al. , 2001 ) , pairwise clustering method ( Shental et al. , 2003 ) , K-means ( Hartigan & Wong , 1979 ) , and kernel K-means ( Schölkopf et al. , 1998 ) . The success of similarity-based clustering highly depends on the underlying pairwise similarity over the data , which in most cases are constructed empirically , e.g . by Gaussian kernel or the K-NearestNeighbor ( KNN ) graph . In this paper , we model data clustering as a multiclass classification problem and seek for the data partition where the associated classifier , trained on cluster labels , can have low generalization error . Therefore , it is natural to formulate data clustering problem as a problem of training unsupervised classifiers : a classifier can be trained upon each candidate partition of the data , and the quality of the data partition can be evaluated by the performance of the trained classifier . Such classifier trained on a hypothetical labeling associated with a data partition is termed an unsupervised classifier . We present Clustering by Discriminative Similarity ( CDS ) , wherein discriminative similarity is derived by the generalization error bound for an unsupervised similarity-based classifier . CDS is based on a novel framework of discriminative clustering by unsupervised classification wherein an unsupervised classifier is learnt from unlabeled data and the preferred hypothetical labeling should minimize the generalization error bound for the learnt classifier . When the popular Support Vector Machines ( SVMs ) is used in this framework , unsupervised SVM ( Xu et al. , 2004 ) can be deduced . In this paper , a similarity-based classifier motivated by similarity learning ( Balcan et al. , 2008 ; Cortes et al. , 2013 ) , is used as the unsupervised classifier . By generalization analysis via Rademacher complexity , the generalization error bound for the unsupervised similarity-based classifier is expressed as sum of pairwise similarity between the data from different classes . Such pairwise similarity , parameterized by the weights of the unsupervised similarity-based classifier , serves as the discriminative similarity . The term “ discriminative similarity ” emphasizes the fact that the similarity is learnt so as to improve the discriminative capability of a certain classifier such as the aforementioned unsupervised similarity-based classifier . 1.1 CONTRIBUTIONS AND MAIN RESULTS . Firstly , we present Clustering by Discriminative Similarity ( CDS ) where discriminative similarity is induced by the generalization error bound for unsupervised similarity-based classifier on unlabeled data . The generalization bound for such similarity-based classifier is of independent interest , which is among the few results of generalization bounds for classification using general similarity functions ( Section B.1 of the appdendix ) . When the general similarity function is set to a Positive SemiDefinite ( PSD ) kernel , the derived discriminative similarity between two data points xi , xj is SKij = 2 ( αi + αj − λαiαj ) K ( xi − xj ) , where K can be an arbitrary PSD kernel and αi is the kernel weight associated with xi . With theoretical and empirical study , we argue that SKij should be used for data clustering instead of the conventional kernel similarity corresponding to uniform kernel weights . In the case of binary classification , we prove that the derived discriminative similarity SKij has the same form as the similarity induced by the integrated squared error bound for kernel density classification ( Section A of the appendix ) . Such connection suggests that there exists informationtheoretic measure which is implicitly equivalent to our CDS framework for unsupervised learning , and our CDS framework is well grounded for learning similarity from unlabeled data . Secondly , based on our CDS model , we develop a clustering algorithm termed Clustering by Discriminative Similarity via unsupervised Kernel classification ( CDSK ) in Section 5 . CDSK uses a PSD kernel as the similarity function , and outperforms competing clustering algorithms , including nonparametric discriminative similarity based clustering methods and similarity graph based clustering methods , demonstrating the effectiveness of CDSK . When the kernel weights { αi } are uniform , CDSK is equivalent to kernel K-Means ( Schölkopf et al. , 1998 ) . CDSK is more flexible by learning adaptive kernel weights associated with different data points . 1.2 CONNECTION TO RELATED WORKS . Our CDS model is related to a class of discriminative clustering methods which classify unlabeled data by various measures on discriminative unsupervised classifiers , and the measures include generalization error ( Xu et al. , 2004 ) or the entropy of the posterior distribution of the label ( Gomes et al. , 2010 ) . Discriminative clustering methods ( Xu et al. , 2004 ) predict the labels of unlabeled data by minimizing the generalization error bound for the unsupervised classifier with respect to the hypothetical labeling . Unsupervised SVM is proposed in Xu et al . ( 2004 ) which learns a binary classifier to partition unlabeled data with the maximum margin between different clusters . The theoretical properties of unsupervised SVM are further analyzed in Karnin et al . ( 2012 ) . Kernel logistic regression classifier is employed in Gomes et al . ( 2010 ) , and it uses the entropy of the posterior distribution of the class label by the classifier to measure the quality of the hypothetical labeling . CDS model performs discriminative clustering based on a novel unsupervised classification framework by considering similarity-based or kernel classifiers which are important classification methods in the supervised learning literature . In contrasts with kernel similarity with uniform weights , the induced discriminative similarity with learnable weights enhances its capability to represent complex interconnection between data . The generalization analysis for CDS is primarily based on distribution free Rademacher complexity . While Yang et al . ( 2014a ) propose nonparametric discriminative similarity for clustering , the nonparametric similarity requires probability density estimation which is difficult for high-dimensional data , and the fixed nonparametric similarity is not adaptive to complicated data distribution . The paper is organized as follows . We introduce the problem setup of Clustering by Discriminative Similarity in Section 3 . We then derive the generalization error bound for the unsupervised similarity-based classifier for CDS in Section 4 where the proposed discriminative similarity is induced by the error bound . The application of CDS to data clustering is shown in Section 5 . Throughout this paper , the term kernel stands for the PSD kernel if no special notes are made . 2 SIGNIFICANCE OF CDSK OVER EXISTING DISCRIMINATIVE AND SIMILARITY-BASED CLUSTERING METHODS . Effective data similarity highly depends on the underlying probabilistic distribution and geometric structure of the data , and these two characteristics leads to “ data-driven ” similarity , such as ( Zhu et al. , 2014 ; Bicego et al. , 2021 ; Ng et al. , 2001 ; Shental et al. , 2003 ; Hartigan & Wong , 1979 ; Schölkopf et al. , 1998 ) and similarity based on geometric structure of the data , such as the subspace structure ( Sparse Subspace Clustering , or SSC in ( Elhamifar & Vidal , 2013 ) ) . Note that the sparse graph method , ℓ1-Graph ( Yan & Wang , 2009 ) , has the same formulation as SSC . Most existing clustering methods based on data-driven or geometric structure-driven similarity suffer from a common deficiency , that is , the similarity is not explicitly optimized for the purpose of separating underlying clusters . In particular , the Random Forest-based similarity ( Zhu et al. , 2014 ; Bicego et al. , 2021 ) is extracted from features in decision trees . Previous works about subspace-based similarity ( Yan & Wang , 2009 ; Elhamifar & Vidal , 2013 ) try to make sure that only data points lying on or close to the same subspace have nonzero similarity , so that data points from the same subspace can form a cluster . However , it is not guaranteed that features in the decision trees are discriminative enough to separate clusters , because the candidate data partition ( or candidate cluster labels ) do not participate in the feature or similarity extraction process . Note that synthetically generated negative class are suggested in ( Zhu et al. , 2014 ; Bicego et al. , 2021 ) to train unsupervised random forest , however , the synthetic labels are not for the original data . Moreover , it is well known that the existing subspace learning methods only obtain reliable subspace-based similarity with restrictive geometric assumptions on the data and the underlying subspaces , such as large principal angle between intersecting subspaces ( Soltanolkotabi & Candes , 2012 ; Elhamifar & Vidal , 2013 ) . Therefore , it is particularly important to derive similarity for clustering which meets two requirements : ( 1 ) discriminative measure with information such as cluster partition is used to derive such similarity so as to achieve compelling clustering performance ; ( 2 ) it requires less restrictive assumptions on the geometric structure of the data than current geometric structure-based similarity learning methods , such as subspace clustering ( Yan & Wang , 2009 ; Elhamifar & Vidal , 2013 ) . Significance . The proposed discriminative similarity of this paper meets these two requirements . First , the discriminative similarity is derived by the generalization error bound associated with candidate cluster labeling , and minimizing the objective function of our optimization problem for clustering renders a joint optimization of discriminative similarity and candidate cluster labeling in a way such that the similarity-based classifier has small generalization error bound . Second , our framework only assumes a mild classification model in Definition 3.1 , which only requires an unknown joint distribution over data and its labels . In this way , the restrictive geometric assumptions are avoided in our method . Compared to the existing discriminative clustering methods , such as MMC ( Xu et al. , 2004 ) , BMMC ( Chen et al. , 2014 ) , RIM ( Gomes et al. , 2010 ) , and the other discriminative clustering methods such as ( Huang et al. , 2015 ; Nguyen et al. , 2017 ) , the optimization problem of CDSK with discriminative similarity-based formulation is much easier to solve and it enjoys convexity and efficiency in each iteration of coordinate descent described in Algorithm 1 . In particular , as mentioned in Section D of the appendix , the first step ( 11 ) of each iteration can be solved by efficient SVD or other randomized large-scale SVD methods , and the second step ( 12 ) of each iteration can be solved by efficient SMO ( Platt , 1998 ) . Moreover , the optimization problems in these two steps are either convex or having closed-form solution . In contrast , MMC requires expensive semidefinite programming . RIM has to solve a nonconvex optimization problem and its formulation does not guarantee that the trained multi-class kernelized logistic regression has low classification error on candidate labeling , which explains why it has inferior performance compared to our method . The discriminative Extreme Learning Machine ( Huang et al. , 2015 ) trains ELM using labels produced by a simple clustering method such as K-means , and the potentially poor cluster labels by the simple clustering method can easily result in unsatisfactory performance of this method . The discriminative Bayesian nonparametric clustering ( Nguyen et al. , 2017 ) and BMMC ( Chen et al. , 2014 ) require extra efforts of sampling hidden variables and tuning hyperparameters to generate the desirable number of clusters ( or model selection ) , which could reduce the effect of discriminative measures used in these Bayesian nonparametric methods .

Summary: This paper proposes a new clustering framework called Clustering by Discriminative Similarity (CDS). CDS learns an unsupervised similarity-based classifier from each data partition, and searches for the optimal partition of the data by minimizing the generalization error of the learnt classifiers associated with the data partitions. In contrasts with kernel similarity with uniform weights, the induced discriminative similarity with learnable weights enhances its capability to represent complex inter-connection between data. Based on CDS, CDSK is proposed as a new clustering method, with its effectiveness demonstrated by experimental results.