Privacy‐preserving constrained spectral clustering algorithm for large‐scale data sets

IET Information SecurityVolume 14, Issue 3 p. 321-331 Research ArticleFree Access Privacy-preserving constrained spectral clustering algorithm for large-scale data sets Ji Li, Ji Li orcid.org/0000-0001-7439-782X State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou, People's Republic of China State Key Laboratory of Integrated Service Networks, Xidian University, Xi'an, People's Republic of ChinaSearch for more papers by this authorJianghong Wei, Jianghong Wei State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou, People's Republic of China State Key Laboratory of Integrated Service Networks, Xidian University, Xi'an, People's Republic of ChinaSearch for more papers by this authorMao Ye, Mao Ye National University of Defense Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorWenfen Liu, Corresponding Author Wenfen Liu liuwenfen@guet.edu.cn School of Guangxi Key Laboratory of Cryptogpraphy and Information Security, Guilin University of Electronic Technology, Guilin, People's Republic of ChinaSearch for more papers by this authorXuexian Hu, Xuexian Hu State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou, People's Republic of ChinaSearch for more papers by this author Ji Li, Ji Li orcid.org/0000-0001-7439-782X State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou, People's Republic of China State Key Laboratory of Integrated Service Networks, Xidian University, Xi'an, People's Republic of ChinaSearch for more papers by this authorJianghong Wei, Jianghong Wei State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou, People's Republic of China State Key Laboratory of Integrated Service Networks, Xidian University, Xi'an, People's Republic of ChinaSearch for more papers by this authorMao Ye, Mao Ye National University of Defense Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorWenfen Liu, Corresponding Author Wenfen Liu liuwenfen@guet.edu.cn School of Guangxi Key Laboratory of Cryptogpraphy and Information Security, Guilin University of Electronic Technology, Guilin, People's Republic of ChinaSearch for more papers by this authorXuexian Hu, Xuexian Hu State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou, People's Republic of ChinaSearch for more papers by this author First published: 01 May 2020 https://doi.org/10.1049/iet-ifs.2019.0255Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract With the increasing concern on the preservation of personal privacy, privacy-preserving data mining has become a hot topic in recent years. Spectral clustering is one of the most widely used clustering algorithm for exploratory data analysis and usually has to deal with sensitive data sets. How to conduct privacy-preserving spectral clustering is an urgent problem to be solved. In this study, the authors focus on introducing the notion of differential privacy, which is considered as the de facto standard of privacy-preserving data analysis, into spectral clustering. Specifically, by combining the well-studied constrained spectral clustering with the Wishart mechanism in a novel way, the authors propose a differentially private constrained spectral clustering (DP-CSC) algorithm. The DP-CSC algorithm is proved to capture asymptotic property and achieves -differential privacy. To illustrate the effectiveness and efficiency of DP-CSC, the authors conduct experiments on five real-word data sets. The results indicate that the DP-CSC algorithm can provide acceptable clustering accuracy with short running time while preserving individual privacy. 1 Introduction Nowadays, with the explosive growth of data volume in all science and engineering domains as well as the rapid development of data processing technology, data mining increasingly plays an important role in better producing novel knowledge, which benefits both individuals and enterprises [1]. Particularly, clustering is one of the most widely used and fundamental tool for exploratory data analysis, and has been found applications in many settings including biology, statistics and psychology. Among various clustering algorithms, spectral clustering [2, 3] is a simple yet efficient clustering algorithm. It was originally introduced to address the unsupervised learning problem by encoding all data points with the graph Laplacian. Research suggests that spectral clustering outperforms traditional clustering algorithms (e.g. k -means) due to its effectiveness on certain graph cut problems and ability for modelling arbitrary-shaped clusters. To further improve the effectiveness and efficiency of spectral clustering, several variants have been proposed, such as constrained spectral clustering [4, 5], compressive spectral clustering [6] and compressed constrained spectral clustering [7]. Spectral clustering often has to deal with sensitive data such as personal consumption habits, user preferences and medical records. However, directly publishing clusters learned from these sensitive data sets via spectral clustering can lead to privacy concerns [8, 9], which are of utmost importance. Thus, preserving privacy while conducting spectral clustering becomes necessary. In recent years, differential privacy [10], a novel privacy preservation model based on data distortion, has been widely employed in the literature of data mining. It adds noises with specific distributions to raw data (or results of data mining algorithms) so that the basic statistics of the data set can be kept with acceptable accuracy while protecting individual privacy. Differential privacy has become the de facto standard of preserving statistical data privacy. So far, many traditional data mining algorithms have been extended to satisfy differential privacy, e.g. k -means [11], k -nearest neighbour classification [12], random forest [13], frequent itemset mining [14, 15] and so on. Nevertheless, when focusing on spectral clustering, there are few works with regard to differentially private spectral clustering algorithms. On the other hand, various real-world applications of spectral clustering, e.g. image segmentation, social network mining, collaborative filtering recommendation, sentiment analysis and so forth, are all related to sensitive data. Thus, it is necessary and urgent to develop spectral clustering algorithms satisfying differential privacy. The main difficulty of conducting spectral clustering under differential privacy comes from the high sensitivity of computing graph Laplacian, which makes the noise added to the raw data is rather big and further destroys the usability of spectral clustering results. To this end, we adopt a strategy of avoiding directly computing such a high sensitivity. Note that the core essence of differential privacy is that it provides a rigorous and quantitative manner of evaluating the level of privacy preservation, rather than the way to define the input of the underlying algorithm. Thus, we no longer take the raw data as the input when designing an algorithm with the property of differential privacy. Based on such an observation, we abandon the attempt to consider the sensitivity of generating graph Laplacian from raw data, and take the graph Laplacian as an input of the differentially private step. Consequently, our target changes to protect a real symmetric positive semi-definite matrix with differential privacy, which can also be regarded as an intermediate result of principal component analysis (PCA). This enables us to design a differentially private spectral clustering algorithm from the inspiration of well-studied PCA satisfying differential privacy. To make the algorithm more suitable for large scale data sets, we employ the Landmark-based graph construction [16]. 1.1 Our contribution In this paper, to overcome the privacy issues introduced by conducting spectral clustering on sensitive data, we focus on the design of privacy-preserving spectral clustering. Specifically, based on optimised spectral clustering, constrained spectral clustering, we put forward a differentially private spectral clustering algorithm, which provides rigorous privacy guarantees while roughly maintaining the clustering accuracy of the original algorithm. More precisely, the contribution of this paper includes the following: (i) To decrease the sensitivity of computing graph Laplacian, we introduce a new manner of adding noise into spectral clustering, namely, by utilising the intermediate graph Laplacian to define the global sensitivity, rather than the raw data. (ii) Based on well-studied compressed constrained spectral clustering and the Wishart mechanism designed for differentially private PCA, we propose a differentially private constrained spectral clustering (DP-CSC) algorithm. (iii) We prove that the DP-CSC algorithm captures asymptotic property and achieves -differential privacy. This implies that the validity and security of DP-CSC are theoretically guaranteed. (iv) We conduct experiments on five real-word data sets with different parameter settings to evaluate the performance and practicability of DP-CSC. The experimental results show that DP-CSC has practical efficiency and the desirable functionalities. 1.2 Outline The rest of this paper is organised as follows. In Section 2, we review the related work on differentially clustering and private spectral analysis. In Section 3 we introduce some preliminary knowledge used throughout this paper. The DP-CSC is presented in Section 4, followed by the corresponding analysis. In Section 5, we evaluate the performance of DP-CSC by conducting comparative experiments on five data sets. Finally, we conclude this paper in Section 6. 2 Related works The notion of differential privacy was introduced by Dwork [10] to preserve statistical data privacy. Informally, for a differentially private algorithm, adding or deleting a data point in the input data set only have a negligible influence on the output of the algorithm. Due to its unique and favourite advantages, the original differential privacy has been extended from multiple aspects (e.g. concentrated differential privacy [17], personalised differential privacy [18] and local differential privacy [19]) and widely employed in the literature of data mining. 2.1 Differentially private clustering algorithms To the best of our knowledge, there is only one research on differentially private spectral clustering algorithm [20], which was specially designed for hypergraph. The main idea was to project the hypergraph Laplacian matrix into a low-dimensional space and perturb the eigenvector with random noise based on the exponential mechanism. The research on differentially private clustering mainly focuses on k -means clustering. Blum et al. [21] extended the sub-linear queries framework to achieve a noisy version of k -means clustering satisfying -differential privacy. However, its query function has large sensitivity and the method of configuring privacy budget was not provided. To this end, Nissim et al. [22] defined the sample and aggregate framework and applied it to k -means clustering with local sensitivity, which reduces the magnitude of noise and makes the centroids protected as well. Afterwards, Feldman et al. [23] developed a scheme for constructing private coresets efficiently, which was applied to achieve differentially private k -means and k -median by conducting clustering algorithms on the private coresets. Some other works aiming at overcoming the defects of existing algorithms. To avoid low utility caused by strong noise, Nguyen et al. [24] designed a framework by focusing directly on the estimation error of the data point, which was able to significantly lower the perturbation required. They also proposed a new privacy aware k -means clustering algorithm with the framework. To solve the problem that the random noise became too large with the number of iterations increasing in k -means, Fan and Xu [25] proposed animproved version of [21] named APDPk-means based on arithmetic progression privacy budget allocation. Stemmer and Kaplan[26] present efficient private algorithms for the Euclidean k -means a problem with constant multiplicative error for the first time, both in the centralised model and in the local model of differential privacy. They further designed a locally private k -means clustering algorithm which could significantly reduce the additive error while keeping the multiplicative error the same as in previous results in [27]. To address the non-convergence problem in differentially private k -means algorithms in the interactive settings, Lu and Shen [28] developed a novel centroids updating approach by applying the exponential mechanism of differential privacy in a selected area. To addresses the problem of learning from large collections of data with privacy guarantees, Schellekens et al. [29] put forward a mechanism based on the compressive learning framework [30] to learn from compressed data sets by computing noisy moments from data set subsamples. They applied the method to k -means and provided a measure of the utility of it. There are also some differentially private platforms where k -means can be automatically implemented. McSherry [31] implemented the standard k -means algorithm in the privacy integrated queries system. Zhang et al. [32] developed a differentially private model fitting framework based on the exponential mechanism [33], whose application included k -means clustering, logistic regression and support vector machine. Besides, Mohan et al. [34] constructed a platform of the sample-and-aggregate framework named GUPT and evaluated its effectiveness by implementing k -means clustering on it. Note that all these schemes are interactive approaches and built upon the assumption that the sensitivity of the cluster centres has been predefined. Su et al. [35] comprehensively evaluated the above-mentioned differentially private k -means algorithms, and provided an improved version of McSherry's approach [31] and a non-interactive method for differentially private k -means clustering named EUGkM. Subsequently, they [11] further put forward a novel hybrid method that combines EUGkM with the approach in [31]. Holohan et al. [36] presented the IBM Differential Privacy Library, an open source library for differential privacy applications. The library included a host of mechanisms, a number of applications to machine learning and other data analytic tasks, e.g. k -means. There are only a few works on other clustering algorithms, e.g. DBSCAN [37]. Based on local sensitivity, Wang et al. [38] provided an agnostic k -means clustering with the sample-and-aggregate framework, and obtained a Gibbs sampling subspace clustering algorithm using the exponential mechanism. After that, they studied the properties of sparse subspace clustering [39], and applied their analysis to a differentially private sparse subspace clustering algorithm under the Johnson–Lindenstrauss (JL) transform. Balcan et al. [40] also used the JL transform to design efficient and differentially private k -means and k -median clustering in high-dimensional Euclidean spaces. Nguyen [41] proposed a differentially private k -modes clustering algorithm, which was an extension of k -means applied to categorical data. In [42], the differentially private k -means algorithm and k -modes algorithm were combined to cluster mixed data in a privacy-preserving manner, and the allocation of privacy budgets was optimised. In [43], a density peak clustering algorithm based on differential privacy named DP-CFSFDP was proposed, along with its improved version named DP-rcCFSFDP for better performance on data with uniform distribution. [44] studied the problem of non-interactive clustering in the distributed setting under the framework of local DP, and developed kCluster algorithm that was suitable for clustering in the anonymised space. Pinot et al. [45] introduced the first differentially private clustering method for arbitrary-shaped node clusters in a graph, which could recover arbitrarily shaped clusters while preserving differential privacy on the weights of the graph. 2.2 Spectral analysis under differential privacy The spectral analysis of matrix refers to a family of effective methods that analyse an input matrix through exploiting information about its eigenvectors or singular vectors. The application of spectral analysis includes PCA, low-rank approximation and spectral clustering. Since these applications often involve operations on data sets with sensitive information, how to perform spectral analysis under differential privacy has received much attention. In general, the input of differentially private spectral analysis is a real-valued matrix corresponding to the values of n attributes of m individuals and a parameter k. The goal of the algorithm is to approximately compute the first singular vectors of A while achieving differential privacy. Singular vectors can be obtained using the eigen-decomposition of the covariance matrix of A. Thus, there are two strategies for adding differential privacy noise, namely, either adding noise to the covariance matrix or adding noise to the feature subspace of covariance matrix. Following the first strategy mentioned above, Blum et al. [21] employed a random response mechanism to add noise matrix to the input matrix (or the covariance matrix of the input matrix). Dwork et al. [46] revisited the variation of this method, proposed a noise algorithm for covariance matrix based on the Guassian mechanism and proved that it satisfies -differential privacy. Motivated by this work, Jiang et al. [47] put forward the Wishart mechanism that enables the noise-added covariance matrix to maintain the semi-positive definite property. Especially, this mechanism adds the Wishart noise to the covariance matrix to achieve differentially private PCA. Similarly, Chen [48] designed a differentially private approximation to adjacency spectral embedding (DP-ASE) on stochastic blockmodels by adding noise to the adjacency matrix. [49] presented a cloud-centric framework for the privacy-preserving spectral analysis of large matrices, which provided strong privacy guarantee protecting from honest-but-curious cloud providers. It allowed data contributors to submit encrypted graph data to the cloud, and the graph was constructed by a bin-based differentially private graph perturbation algorithm. Some other works use the second strategy, i.e. adding noise to the feature subspace of the covariance matrix. Concretely, Chaudhuri et al. [50] utilised the exponential mechanism to sample the approximate eigenvectors of a matrix in a heuristic way. Wang et al. [51] applied this technique to differentially private spectral analysis of graph and developed two methods for eigenvalue decomposition of the adjacency matrix of the graph. Kapralov and Talwar [52] combined the exponential mechanism with the Laplace mechanism to design a polynomial time algorithm of sampling the approximate eigenvectors of the matrix. Hardt and Roth [53], Hardt and Price [54] studied the problem of singular vector computation under differential privacy, and proposed PCA algorithms satisfying -differential privacy and -differential privacy, respectively. The differentially private hypergraph spectral clustering algorithm in [20] also perturbed the eigenvector with random noise. The definition on neighbouring matrices is not uniform in the works: In [54], are called neighbours when they differ in only one entry by at most 1 in absolute value; In [50], are called neighbours when the norm (also called Euclidean norm) of their vectors is no larger than 1 and they differ in at most one row by a vector whose norm is bounded by 1. Our DP-CSC algorithm uses the latter one. 3 Preliminaries In this section, we briefly review the preliminaries used in our algorithm, including the Laplacian matrix, the problem of constrained spectral clustering, the formal definition of differential privacy and the Wishart mechanism. 3.1 Laplacian matrix In general, an undirected graph is formed as , where V is the collection of vertexes , E is the edge set and W is the set of edge weights that indicate the similarity between vertexes. If there is an edge between the vertexes and , then ; otherwise, . This enables us to formulate a matrix W with elements , which is often referred to as the weighted adjacency matrix. The degree matrix D of the graph G is a diagonal matrix defined as , from which we can define the following notion of Laplacian matrix Definition 1.(Laplacian matrix): The Laplacian matrix L of a graph G is , and the normalised Laplacian matrix is , where is the unit matrix. In addition, our design involves the notion of square-rooting matrix, which is defined as follows Definition2.(Square-rooting matrix [55]): Let be a Hermitian positive semi-definite matrix with rank . Then there exists a unique Hermitian positive semi-definite matrix with rank satisfying , is called the square root of , denoted by . The above-defined Laplacian matrix captures the following properties [55, 56]. Property 1.The Laplacian matrix and normalised Laplacian matrix of an undirected graph are symmetric and semi-definite positive. Property 2.The Laplacian matrix of an undirected graph has a unique square-rooting matrix. 3.2 Landmark-based graph construction A data set can be represented with a matrix , where n is the number of data sample points and d means the number of data features. The transpose of is denoted by . We can generate a graph G through by using the following Landmark-based graph construction [57]. Algorithm 1.Landmark-based graph construction [16]Input: Dataset , cluster number k, number of landmark points l and number of nearest neighbours s; 1: Generate l landmark points ; 2: Construct the sparse representation between the landmark points and data points by setting as where is the index set of basis vectors among the s nearest neighbours of , and is the Gaussian kernel function with bandwidth h. For , let . 3: Compute , where is a diagonal matrix such that ; Output: Sparse representation matrix The above algorithm enables us to construct a graph with the weighted adjacency matrix , which has actually already been normalised according to the last step of Algorithm 1, so does the normalised Laplacian matrix . The time complexity of computing is . 3.3 Constrained spectral clustering Roughly, the constrained spectral clustering model can be summarised with three undirected graphs, namely, data graph and two knowledge graphs and . The data graph contains the basic information of the data set. The 'must link' (ML) graph holds the constraint information that which data points must be linked together. Each edge in implies that its two endpoints should be linked and the corresponding weight in is the belief degree of 'ML'. The 'cannot-link' (CL) graph captures those data points that cannot be linked. Similarly, the edge in indicates that the two endpoints cannot be linked, and the corresponding weight in is the belief degree of 'CL'. As in [58], we regard the data graph is as an implicit ML graph that views pairwise affinities between endpoints as 'ML' constraint. Consequently, a generalised data graph can be obtained by combining and as (1)where the parameter is the belief degree of the constraint information in . Let k be the number of clusters. To satisfy the constraints of ML and CL as much as possible, the constrained spectral clustering problem, i.e. the generalised constrained k way cuts problem, can be written as (2)where is the Laplacian matrix of the generalised data graph , is the Laplacian matrix of the CL graph , is the cluster indicator vector. Particularly, for cluster , we have (3)Since the generalised constrained k way cuts problem is the NP hard, Cucuringu et al. [58] relaxed the range of solution to the real vectors through removing the condition . As a result, the original problem is converted to the problem of computing the first k non-trivial generalised eigenvectors of (4) (4)where is the generalised eigenvalue and is the corresponding eigenvector. 3.4 Differential privacy The notion of differential of privacy was initially introduced by Dwork [10] and defined as follows Definition 3.( -differential privacy [10]): An algorithm satisfies -differential privacy provided that for any pair of neighbouring data sets D and , and any , it holds that (5)where denotes the collection of all possible outputs of the algorithm . Here, two data sets D and are said to be neighbouring means that they differ by only one data point, and denoted by . One of the most classical methods of achieving -differential privacy is the Laplace mechanism, which perturbs the output of a function on a data set by adding a noise randomly sampled from the Laplace distribution with scale proportional to the sensitivity of the function. Definition 4.(Global sensitivity): Given a function , its global sensitively is captured as (6) Definition 5.(Laplace mechanism [59]): Given a function , the Laplace mechanism is defined as (7)where are i.i.d. random variables are drawn from (8)The Laplace mechanism satisfies differential privacy. Jiang et al. [47] demonstrated that the Laplace mechanism can be used to achieve -differentially private PCA. However, this might make the noisy sample covariance matrix no longer be positive semidefinite. To this end, Jiang et al. further developed the Wishart mechanism satisfying -differential privacy to keep the property of positive semidefinite. Definition 6.A random symmetric positive definite matrix is said to have a Wishart distribution , if its probability density function is (9)where and is a positive definite matrix. Algorithm 2.Wishart mechanism [47]Input: Raw data matrix and privacy budget . 1: Draw a sample from , where has d same eigenvalues equal to ; 2: Compute ; 3: Add noise . Output: Noisy sample covariance matrix . 4 Constrained spectral clustering under differential privacy In this section, based on a well-studied and optimised spectral clustering algorithm (CCS-L for short) [60] as well as the Wishart mechanism [47], we put forward a DP-CSC algorithm named as DP-CSC. We also provide the correctness proof of DP-CSC. 4.1 DP-CSC algorithm To conduct constrained spectral clustering, we first need to generate the adjacency matrix. Particularity, we use the Landmark-based graph construction to produce such a matrix W such that (10)In addition, we also need to define the Laplacian matrices and , number of clusters k and confidence parameter . Unlike conventional clustering algorithms like k -means, spectral clustering involves complicated matrix operations, which makes it intractable to identify the global sensitivity of the algorithm. More precisely, since the key input of a spectral clustering algorithm is the weight adjacency matrix, we need to examine the gap between two clustering results resulted from two neighbouring weight adjacency matrices for determining the global sensitivity. Although there are some ways to define neighbouring matrices, e.g. entry neighbours, row neighbours [61], we find that they are all infeasible for capturing an effective bound on the gap between two graph Laplacian matrices. To conquer the above problems, we redefine the key input of a spectral clustering algorithm. Note that the original weight adjacency matrix can be obtained from the Laplacian matrix, meaning that they have a bijection relationship and contain the same amount of information. This allows us to use the Laplacian matrix, rather than the weight adjacency matrix, as the input of our private clustering algorithm to avoid the issue of high sensitivity. Specifically, we add the noise into the Laplacian matrix when it is compressed. Focusing on the added noise, though the Laplace noise can also be used to achieve -differential privacy, it will break the positive semidefinite of the resulted matrix. Observing that both and are positive semidefinite matrices, each of them can be viewed as the product of a matrix and its transpose matrix, which is fairly similar to the sample covariance matrix of raw data. On the other hand, the Wishart mechanism can publish sample covariance matrix in a differentially private manner, thus suited to adding noise to the compression process. In our approach, we use the Wishart mechanism to protect . As for , since it is more sparse and contains less information than , we do not add noise to its compression process to maximise the preservation of the original clustering effect. Following the above strategy, we specify the details of DP-CSC in Algorithm 4. Algorithm 3.CCS-L Algorithm [60]Input: Adjacency matrix , Laplacian matrices and , number of clusters k, confidence parameter ; 1: Compute compressed Laplacian and ; 2: Solve the first k