Title: Federated Spectral Clustering via Secure Similarity Reconstruction
Venue: NeurIPS 2023
Accepted: yes
Keywords: clustering

federated learning

privacy
Abstract: Federated learning has a significant advantage in protecting information privacy. Many scholars proposed various secure learning methods within the framework of federated learning but the study on secure federated unsupervised learning especially clustering is limited. We in this work propose a secure kernelized factorization method for federated spectral clustering on distributed dataset. The method is non-trivial because the kernel or similarity matrix for spectral clustering is computed by data pairs, which violates the principle of privacy protection. Our method implicitly constructs an approximation for the kernel matrix on distributed data such that we can perform spectral clustering under the constraint of privacy protection. We provide a convergence guarantee of the optimization algorithm, reconstruction error bounds of the Gaussian kernel matrix, and the sufficient condition of correct clustering of our method. We also present some results of differential privacy. Numerical results on synthetic and real datasets demonstrate that the proposed method is efficient and accurate in comparison to the baselines.

Federated Spectral Clustering via Secure Similarity Reconstruction

Abstract

Federated learning has a significant advantage in protecting data and information privacy. Many scholars proposed various secure learning methods within the framework of federated learning but the study on secure federated unsupervised learning especially clustering is limited. We in this work propose a secure kernelized factorization method for federated spectral clustering on distributed data. The method is non-trivial because the kernel or similarity matrix for spectral clustering is computed by data pairs, which violates the principle of privacy protection. Our method implicitly constructs an approximation for the kernel matrix on distributed data such that we can perform spectral clustering under the constraint of privacy protection. We provide a convergence guarantee of the optimization algorithm, reconstruction error bounds of the Gaussian kernel matrix, and the sufficient condition of correct clustering of our method. We also present guarantees of differential privacy. Numerical results on synthetic and real datasets demonstrate that the proposed method is efficient and accurate in comparison to baselines.

1 Introduction

In the era of big data, human beings can analyze massive data in various fields due to the improvement of storage and computational capabilities of computing devices [Li et al. , 2020b]. Some popular fields such as artificial intelligence, machine learning, internet of things (IoT), and cloud computing have seen explosive development over the past few years. Nevertheless, a side effect of this trend is that individuals and organizations have more and more concerns about potential violation of privacy [Kairouz et al. , 2021]. As a result, it has become a challenge to mine valuable information from user data but not to directly access it.

Federated learning [Kairouz et al. , 2021; McMahan et al. , 2017] can train a global model without retrieving dispersed data [Yang et al. , 2018]. This advantage has made it so popular that many scholars have put much effort into the study of federated learning. For example, Yang et al. [2019] presented the definitions of horizontal federated learning, vertical federated learning, and federated transfer learning. Some privacy-preserving machine learning models were also presented. For instance, He et al. [2020] developed a federated group knowledge transfer algorithm to train small CNNs on edge devices. Chen et al. [2018] proposed a protocol to conduct privacy-preserving ridge regression over high-dimensional data. Besides, Kim et al. [2018] proposed a block-chained federated learning architecture that enables on-device learning without any central coordination.

Regardless of the great progress of federated learning, it can be found that most of the existing studies are for supervised learning [Li et al. , 2020a; Ghosh et al. , 2020]. Note that collecting labeled data may deserve very high cost in real situations [Li et al. , 2020b] while unlabeled data are abundant. Thus, it

is necessary and important to study federated learning for unsupervised learning [Zhang et al. , 2020; Tzinis et al. , 2021; Zhuang et al. , 2021; Dennis et al. , 2021] such as clustering [Ng et al. , 2001; Fan and Chow, 2017; Fan et al. , 2018, 2021; Fan, 2021; Cai et al. , 2022; Fan et al. , 2022]. For example, Li et al. [2021] proposed a federated matrix factorization with a privacy guarantee for recommendation systems. Wang and Chang [2022] proposed two federated matrix factorization algorithms that can be used for federated clustering. Besides, there are some studies on federated spectral clustering. For instance, Wang et al. [2020] presented a federated multi-view spectral clustering method under the assumption that the data of each view are in one client. Hernández-Pereira et al. [2021] developed a cooperative spectral clustering model to deal with distributed data but the model is linear. However, the study on federated spectral clustering is still very limited and deserves more attention and effort.

In this paper, we propose a federated kernelized factorization method to reconstruct a similarity matrix for secure spectral clustering on distributed data. Our contributions are as follows.

We propose a federated spectral clustering algorithm and provide convergence guarantee for the optimization.

We further propose to add noise to the data or the learned factors to enhance the security of clustering and provide guarantees of differential privacy.

We provide upper bounds for the reconstruction error of the true similarity matrix and theoretical guarantees for correct clustering.

We test our method on both synthetic data and real datasets in comparison to baselines, which verify the effectiveness of our method.

Notations We use y , y , and Y to denote scalar, vector, and matrix, respectively. The element of Y at row i and column j is denoted by y ij . We use ∥ · ∥ 2 to denote the ℓ 2 norm of a vector and use Tr ( · ) , ∥ · ∥ F , and ∥ · ∥ sp to denote the trace, Frobenius norm, and spectral norm of a matrix respectively. The ℓ ∞ norm and ℓ 2 , ∞ norm of a matrix Y are defined as ∥ Y ∥ ∞ = max ij | y ij | and ∥ Y ∥ 2 , ∞ = max j √ ∑ i y 2 ij respectively. K , K , K , and k denote the kernel matrix, kernel function, number of clusters, and the number k in KNN, respectively. ϕ denotes the feature map induced by K .

2 Federated Spectral Clustering (FedSC)

Suppose we have n data points of dimension m distributing in P clients. For convenience, we denote by X ∈ R m × n the matrix composed of all the n data points and denote by X p ∈ R m × N p the matrix composed of the N p data points in client c p , where N p ≥ 1 , p = 1 , . . . , P , and ∑ P p =1 N p = n . Without loss of generality, we let X = [ X 1 , X 2 , . . . , X P ] , which means { X p } P p =1 are submatrices of X . Our goal is to perform spectral clustering on these data to partition them into K groups, under the constraint that the data in each client cannot leave the client itself and the privacy of the data should be protected as much as possible, though there could be a central server conducting clustering.

The aforementioned task is non-trivial because in spectral clustering, the first step is constructing an adjacency matrix A ∈ R n × n , which has to evaluate the similarity between every data pair ( x i , x j ) using a metric function M ( · , · ) and hence violates the privacy constraint in the task. To solve the problem, we present a federated spectral clustering model in this section.

2.1 Similarity Reconstruction via Feature Space Factorization

In spectral clustering, for M ( · , · ) , there are many choices such as k nearest neighbor similarity and various kernel functions. Let K ( · , · ) be a kernel function and we have

where ϕ : R m → R m ′ is a feature map induced by the kernel function 2 and does not need to be carried out explicitly. When it comes to federated spectral clustering, the central server has no access

to the raw data distributed in clients and hence cannot compute K ( x i , x j ) using (1). However, if the central server can learn an effective approximation (denoted by ̂ ϕ ( x i ) ) for each ϕ ( x i ) without accessing x i , K ( x i , x j ) can be estimated, i.e.,

Thus, inspired by [Fan and Udell, 2019; Fan et al. , 2021], we propose to approximate each ϕ ( x i ) by

where Z = [ z 1 , z 2 , . . . , z d ] ∈ R m × d , ϕ ( Z ) = [ ϕ ( z 1 ) , ϕ ( z 2 ) . . . , ϕ ( z d )] , and c i ∈ R d . Both Z and c i are learned from individual columns of X and they can be regarded as intermediate variables avoiding the direct access of central server to x i (details of the learning will be introduced later). It follows from (2) and (3) that

Thus we can reconstruct the similarity between x i and x j via (4). For convenience, let K xx = K ( X , X ) = ϕ ( X ) ⊤ ϕ ( X ) , K zz = K ( Z , Z ) = ϕ ( Z ) ⊤ ϕ ( Z ) ∈ R d × d , and C = [ c 1 , c 2 , . . . , c n ] ∈ R d × n . Then we have

Now we use ˆ K xx as a reconstructed similarity matrix for spectral clustering.

In the form of federated learning, we expand (3) to

It indicates that Z is shared for all P clients and C p is private for client c p . Note that (6) is a matrix factorization problem in the feature space induced by a kernel on the data in client c p , p = 1 , . . . , P . Letting C = [ C 1 , . . . , C P ] , we solve the following distributed optimization problem 3

In (7), f p is a local objective function for client c p and ω 1 , . . . , ω P are nonnegative weights for the clients. In this work, we let

where λ ≥ 0 is a penalty parameter. To guarantee the privacy of information, problem (7) shall be solved in the framework of federated learning.

2.2 FedSC by Similarity Reconstruction and Model Averaging

In this section, we develop a FedSC algorithm by similarity reconstruction and model averaging. As a classic and popular framework, FederatedAveraging (or FedAvg) is first introduced in [McMahan et al. , 2017] for federated learning. In our work, the proposed FedSC is, therefore, built up based on the backbone of FedAvg as in Figure 1. FedSC consists of two stages. The first stage, shown by the left plot of Figure 1, is federated similarity reconstruction, which constructs a similarity matrix in the manner of federated learning. The second stage, shown by the left plot of Figure 1, is using the reconstructed similarity matrix to implement spectral clustering.

Stage I Federated Similarity Reconstruction

Step 1 : As the startup settings for our algorithm, the shared variable Z ( i.e. , the dictionary matrix Z ) and each local coefficient matrix C p for p = 1 , 2 , · · · , P are initialized randomly in the central server and each client, respectively.

Local Update Module

!"#

$

!

, % $

!

!"#

#

!

, % #

!

!"#

#

&

!

, % &

!

Server

Client 1

Client 2

Client P

…

!"#

#

!

!"#

"

$

!

!"#

%

!

Aggregation Module for Shared Variable

! =

1

" #

$%& '()

$

!

*$

!

* !

Spectral Clustering Module

= SpectralClustering ! " # Z,$ !,K

Server

Client 1

Client 2

Client P

…

!

"

#

$

% &

,

$

' &

%!

&

, '

!

&

j

j

j

l

l

l

k

k

k

m

p

q

q

q

o

o

o

n

Information Flow of Federated Spectral Clustering:

j

k

l

m

FedKMF

n

Spectral Clustering

o

p

q

Local Update Module

!

"

= argmin

#$

% & "'( ,   !

&!

"

= argmin

)$

% &!,   !

"

*(

o

&+

-

,

+

-

&.

-

,   .

-

Step 2 : For each round s , where 1 ≤ s ≤ S , the previous shared variable Z will firstly be broadcast to each participated client. After that, every client uses this received dictionary matrix Z to run its own iterative updates of local variables in the Local Update Module (LUM) as:

Step 3 : Each client sends back its own dictionary matrix Z s p , p = 1 , 2 , . . . , P , to the central server.

Step 4 : The central server collects all (or a subset A s -1 of) these uploaded matrices { Z s p } P p =1 and averages them into one new matrix Z s in Aggregation Module (AM), i.e.,

where | A s -1 | is the number of participated clients. In our study, we fix the number of participating clients for each round s . Therefore, we use the notation ¯ P instead of | A s -1 | in the sequel. This aggregated dictionary matrix Z s will then be used to push the next round of federated iteration until the tolerance condition is broken.

Step 5 : When Stage I comes to an end, the spectral clustering will start.

Stage II Spectral Clustering

Step 6 : Each client sends ( Z S p , C S p ) back to the central server for the final aggregation of information.

Step 7 : The required similarity matrix is then constructed based on the obtained dictionary matrix Z S and coefficient matrix C S in Spectral Clustering Module (SCM). Based on this approximated similarity matrix, the standard spectral clustering is implemented as usual:

Step 8 : The central server broadcasts its clustering results to every corresponding client.

2.3 Optimization Algorithm for Federated Similarity Reconstruction

As described in the above section, alternate updating of local variables is a key to solving the proposed FedSC problem. In the following two parts, we discuss the optimization for Z and C , respectively.

For a client c p , consider the corresponding local optimization problem

where f p ( Z , C ) = 1 2 ∥ ϕ ( X p ) -ϕ ( Z ) C ∥ 2 F + λ 2 ∥ C ∥ 2 F = 1 2 Tr ( K ( X p , X p )) -Tr ( C T K ( Z , X p )) + 1 2 Tr ( C T K ( Z , Z ) C )+ λ 2 ∥ C ∥ 2 F . Let the derivative of f p ( Z , C ) w.r.t. C be zero, we get the following one-step update for C :

The derivative of f p ( Z , C ) w.r.t. Z is

where the intermediate variables are detailed as

Here 1 n and 1 d are the column vectors with all elements of 1 . Because of the kernel function, Z cannot be updated like C p . Here, we use the gradient method to update it. At local iteration t , by setting Z s, 0 p = Z s -1 , the update scheme of Z is

where η t is the step size and can be set as the reverse of the Lipschitz constant of gradient if possible. We summarize the optimization details in Algorithm 1 (shown in Appendix A).

2.4 Convergence Analysis of The Proposed Algorithm

First of all, it is obvious that all local objective functions f p ( · , · ) for p = 1 , . . . , P are lower bounded. To analyze the convergence of Algorithm 1, we make two assumptions. The first one is the Lipschitz continuity of the gradient of the local objective functions.

Assumption 2.1. The gradients of all local objective functions f p ( · , · ) for p = 1 , . . . , P are L s Z p -Lipschitz continuous in Z , that is

In addition, there exist some lower and upper bounds for L s Z p , i.e. , 0 < L Z ≤ L s Z p ≤ L Z hold for all p = 1 , . . . , P and s = 1 , . . . , S .

The second assumption, similar to [Li et al. , 2019; Lian et al. , 2017], is as follows.

Assumption 2.2. The difference between the local gradient and the global gradient is bounded as

To build the convergence condition, we define the following iterative terms of Z s,t and C s for all t = 1 , . . . , Q and s = 1 , . . . , S :

where the instantaneous average Z s,t is defined as

Based on the above assumptions, we provide the following convergence guarantee for Algorithm 1.

Theorem 2.3 (Convergence of Algorithm 1) . Suppose Assumption 2.1 and Assumption 2.2 hold. Let T = S (1 + Q ) be the total number of global and local rounds. Then the sequence { ( Z s,t , C s ) } generated by Algorithm 1 with stepsize 1 /L s Z p and ω p = N p n satisfies

The proof can be found in Appendix E. We see that when T →∞ , the algorithm converges to a finite value, which is small if ζ is small and L Z is close to L Z .

3 Security-Enhanced FedSC

In order to enhance the security of FedSC, we present two noise-augmented variants of the proposed algorithm in this section.

3.1 FedSC with Perturbed Data

We add random noise to the data in each client and then perform Algorithm 1 to reconstruct a similarity matrix, which further improves the privacy of data. Specifically, the data X ∈ R m × n is perturbed by a noise matrix E ∈ R m × n to form the noisy data matrix

where E ij ∼ N (0 , σ 2 ) . We then perform Algorithm 1 with a Gaussian kernel of parameter r on ˜ X and obtain Z , C = [ C 1 , C 2 , . . . , C P ] , and

.

We have the following reconstruction (for the true similarity matrix K xx = K ( X , X ) ) error bound 4 Theorem 3.1 (Error bound of similarity matrix reconstruction) . Suppose ∥ X ∥ 2 , ∞ = θ , ∥ C ∥ 2 , ∞ = τ C , and ∥ ∥ ∥ ϕ ( Z ) C -ϕ ( ˜ X ) ∥ ∥ ∥ 2 , ∞ ≤ γ , where θ , τ C , and γ are some nonnegative constants. Then with the probability at least 1 -n ( n -1) e -t , the reconstructed similarity matrix ˆ K ˜ x ˜ x satisfies

Note that r is the hyperparameter of the Gaussian kernel. In our experiment, r was automatically estimated as the mean of all pairwise distances between data points, i.e. , r = 1 n 2 ∑ i,j ∥ x i -x j ∥ 2 . Assume | x ik -x jk | = O ( ε ) for all i, j ∈ [ n ] , k ∈ [ m ] , then ∥ x i -x j ∥ = O ( √ mε ) , which means r 2 is linear with mε 2 . Thus, the reconstruction error ∥ ˆ K ˜ x ˜ x -K xx ∥ ∞ is upper bounded by O ( σ 2 /ε 2 + √ dγτ C ) , where ϵ 2 /σ 2 can be regarded as a signal-noise ratio. Therefore, the bound is useful. In general, Theorem 3.1 indicates that when the added noise is small and the optimization makes γ small, the reconstruction error for the true similarity matrix is less than a small constant with high probability. This verified the effectiveness of our similarity reconstruction method.

It should be pointed out that ˆ K ˜ x ˜ x is not guaranteed to be a sparse matrix and hence the corresponding graph may not contain multiple connected components. We therefore use an extra KNN-based operation to get a sparse similarity matrix, which may also reduce the computational cost of eigenvalue decomposition when n is very large. Specifically, we let

which only keeps the largest k connections from each point to other points. Finally, we perform spectral clustering using ˆ K ˜ x ˜ x . The central server broadcasts the clustering results to each participating client.

As mentioned before, one can choose to inject some noise into its raw data to avoid privacy leakage. However, a question is how much noise we can add to the data to the largest extent for the guarantee of correct clustering. We first present the following definitions.

Definition 3.2 (Local neighbor set) . Suppose x i and x j are data points of X ∈ R m × n with class labels L i and L j respectively, and let KNN ( x i ) be the set of the k-nearest neighbors of x i . We define

Definition 3.3 (Global neighbor set) . Suppose x i and x j are data points of X ∈ R m × n with class labels L i and L j respectively, and let KNN ( x i ) be the point set of k-nearest neighbors of x i . We define

If we call the local neighbor of data point x i the intra-class neighbor of x i , another definition called inter-class neighbor of x i can be further introduced as follows.

Definition 3.4 (Inter-class neighbor set) . Suppose x i and x j are data points of data matrix X ∈ R m × n with class labels, L i and L j , respectively, and let KNN ( x i ) be the point set of k-nearest neighbors of x i . We define

̸

Based on the above definitions, the following definition is presented to determine whether a data point can be correctly clustered or not.

Definition 3.5 (Correct clustering) . Suppose x i ∈ R m and x j ∈ R m are data points of data matrix X ∈ R m × n , x i is said to be correctly clustered with a tolerance of ϵ if

Based on Definition 3.5, the following theorem gives the guarantee of our security-enhanced FedSC.

Theorem 3.6 (Guarantee of noisy spectral clustering) . Let B ( σ ) = 1 r 2 [ ( σξ + √ 2 θ ) 2 -2 θ 2 ] + ( √ dτ C +1) γ . Then with the probability of at least 1 -n ( n -1) e -t , performing spectral clustering using ˆ K ˜ x ˜ x yields correct clustering results if

where K inter ik = ( K xx ) inter ik and K intra ik = ( K xx ) intra ik .

Based on Theorem 3.1 and Theorem 3.6, we can get a bound on the variance of noise for FedSC with perturbed data:

where B 1 = ϵ 2 -max i 1 4 [ max k ( K inter ik ) -min k ( K intra ik ) ] and B 2 = ( √ dτ C +1) γ . This bound indicates that the intensity of noise should not be too strong otherwise it may seriously affect the performance of federated spectral clustering. But, at least under this bound, one can choose to inject as much noise as possible into the raw data to ensure data security and privacy.

Using Theorem 3.22 in [Dwork et al. , 2014] and the post-processing property of differential privacy, we have the following privacy guarantee for this enhanced FedSC algorithm.

Proposition 3.7. FedSC with perturbed data given by (22) is ( ε, δ ) -differentially private if σ ≥ 2 cτ X /ε , where c 2 > 2 ln(1 . 25 /δ ) .

Based on this proposition and (30), we obtain the following privacy-utility trade-off:

This ensures both clustering performance and ( ε, δ ) -differential privacy. In particular, if we substitute σ with the upper bound, we can get a strong level of privacy but the worst utility. By the way, B 1 -B 2 is related to the property of the data. A larger B 1 -B 2 means a better property for clustering, which further provides a larger upper bound for the noise level σ , yielding a stronger privacy guarantee.

3.2 FedSC with Perturbed Factors

In FedSC with perturbed factor, we added Gaussian noise to Z in every round of the optimization but added Gaussian noise to C in the last round of the optimization. To be more specific, ˜ C = C + E C , and ˜ Z = Z + E Z , where the entries of E C and E Z are drawn from N (0 , σ 2 C ) and N (0 , σ 2 Z ) respectively. Then we perform spectral clustering using the following reconstructed kernel matrix:

The following theorem shows the reconstruction error bound for the ground truth kernel matrix K xx . Theorem 3.8. Assume ∥ ϕ ( Z ) C -ϕ ( X ) ∥ 2 , ∞ ≤ γ , ∥ C ∥ 2 , ∞ ≤ τ C . Then with probability at least 1 -( n + d ) e -t , it holds that

We see that, given a fixed γ , the reconstruction error becomes smaller if σ Z and σ C are smaller. Based on Theorem 3.8 and Definitions 3.2-3.5, we can obtain a bound similar to that in Theorem 3.6 to guarantee correct clustering, which will not be detailed here.

Theorem 3.9. In FedSC, assume max ( p,j ) {∥ x p j ∥ , ∥ x ′ p j ∥} ≤ τ X , max ( i,j ) ∥ z i -x j ∥ ∞ = Υ , ∥ Z s p ∥ sp ≤ τ Z ∀ s , and ∥ C S ∥ 2 , ∞ ≤ τ C , we perturb { Z s p } P p =1 , ∀ s = 1 , 2 , . . . , S with noise drawn from N (0 , σ 2 Z ) with the parameter σ Z ≥ √ (8 S ∆ 2 ( g Z ) log( e +( ε Z /δ Z )) /ε 2 Z ) where ∆( g Z ) = 2 √ dτ C τ X η k r 2 { 1 + ( τ X + τ Z ) ( τ X +Υ) r 2 } and perturb { C S p } P p =1 with noise drawn from N (0 , σ 2 C ) with the parameter σ C ≥ 2 cλ -1 √ dτ X ( τ X + Υ) / ( r 2 ε C ) for c 2 > 2 ln(1 . 25 /δ C ) . Then, FedSC is ( ε C + ε Z , δ C + δ Z ) -differentially private.

Theorem 3.9 shows that our FedSC with perturbed factors can protect the data privacy provided that the noises added to C and Z are sufficiently large. Similarly to (31), we can also get a privacy-utility trade-off using Theorem 3.8 and Theorem 3.9, which is detailed in Appendix B.

4 Related Work

It should be pointed out that the study on federated spectral clustering in literature is very limited. Besides our work, the only work that aims to address the problem is [Hernández-Pereira et al. , 2021]. More introduction and discussion about the related work (federated matrix factorization/clustering [Yang et al. , 2021; Ghosh et al. , 2020; Dennis et al. , 2021; Wang and Chang, 2022] and spectral clustering [Von Luxburg, 2007; Hernández-Pereira et al. , 2021]) are in the supplementary material.

5 Experiments

5.1 Performance on similarity reconstruction

Taking the COIL20 dataset [Nene et al. , 1996] as an example, we first obtain the similarity matrix from vanilla spectral clustering based on the same kernel function. Then, we use the proposed method to derive the estimated similarity matrix ˆ K ˜ x ˜ x which is actually an approximation of ground truth. To

make it clearer, we also give the corresponding sparse similarity matrices by KNN sparsification (25). Figure 2 shows the similarity matrices constructed by different methods. We see that the proposed method can be able to successfully reconstruct the similarity matrix in the federated scenarios. The reconstruction errors on synthetic data, iris, banknote authentication, and COIL20 are in the supplementary material.

5.2 Clustering performance of FedSC

In this subsection, we check the clustering performance of the proposed security-enhanced FedSC method on both synthetic and real-world datasets. The synthetic dataset is generated from concentric circles. The details are in the supplementary. This synthetic dataset is visualized in Figure 3(a). Here, we continue to adopt the aforementioned COIL20 as an example of real-world datasets.

Taking the synthetic dataset as an example, the first group of cases helps illustrate the effectiveness of the proposed FedSC method. we first apply the vanilla spectral clustering method to the clean data. The predictive result is shown in Figure 3(b). It is clear that the vanilla spectral clustering method correctly clusters the data points lying in concentric circles. We then use the proposed FedSC method to cluster the data. One can find in Figure 3(c) that almost all of the data points also have been grouped correctly. However, when we inject some volume of noise

(a)

(b)

(c)

(d)

-5

0

5

-5

0

5

1

2

(a) Ground truth

-5

0

5

-5

0

5

1

2

(b) Vanilla SC

-5

0

5

-5

0

5

1

2

(c) FedSC

-5

0

5

-5

0

5

1

2

(d) Noisy data

-5

0

5

-5

0

5

1

2

(e) Vanilla SC

-5

0

5

-5

0

5

1

2

(f) FedSC

into the raw data, things may change a lot. Figure 3(d) is actually the ground truth Figure 3(a) adding some Gaussian noise. When focusing on this noisy data of concentric circles, we see from Figure 3(e) that the vanilla SC failed to cluster the data points while the proposed FedSC method is still able to correctly cluster the data to some extent as in Figure 3(f). As we know, the similarity graph directly constructed from raw data could be very sensitive to each data point. When we add too much noise, the similarity graph may fail to model the local neighborhood relationships which may be the reason why data points in Figure 3(e) are not separable for vanilla SC. Instead, FedSC is based on matrix factorization in the high-dimensional feature space and has a potential denoising effect. Therefore, it is possible for our method to achieve a better performance.The visualization of COIL20 can be found in the supplementary material.

5.3 Comparison with baselines

We compare our method with the clustering method DSC proposed by [Hernández-Pereira et al. , 2021]. Because the existing literature on federated spectral clustering is rare, we here select both classic K-means and spectral clustering as the baselines. Two metrics including accuracy and NMI are adopted to evaluate the clustering results on four datasets including iris [Dua and Graff, 2017], COIL20 [Nene et al. , 1996], banknote authentication [Dua and Graff, 2017], and USPS [Hull, 1994]. The details are in the supplementary material. Besides the clean data, we also consider adding noise to them to test the performance of methods under the condition of privacy protection. We directly inject Gaussian noise with zero mean and variance σ 2 to the raw matrix X ∈ R m × n as ˜ X = X + E where E is a Gaussian noise matrix, each element E i,j of which is i.i.d. with N (0 , σ 2 ) .

Table 1 shows the clustering accuracy. Our FedSC almost always achieves comparable clustering results to vanilla SC. It even outperformed vanilla SC in some cases and K-means in most cases since FedSC has a potential denoising effect by approximating a similarity matrix. More importantly, FedSC significantly outperformed DSC in almost all cases. The reason is that DSC performs spectral clustering on each local dataset, which may lead to very unstable and inaccurate results.

5.4 More numerical result

The tSNE visualization, clustering results in terms of NMI, the performance of FedSC with perturbed factors, etc, are in the supplementary material.

6 Conclusion

This paper has proposed a secure kernelized factorization method for federated spectral clustering on distributed data. We provide theoretical guarantees for optimization convergence, correct clustering, and differential privacy. The numerical experiments on synthetic and real image datasets verified the effectiveness of our method. To the best knowledge of the authors, this is the work that successfully addresses the problem of federated spectral clustering. One limitation of this work is that we haven't tested our FedSC on very large datasets, though the moderate-size datasets are sufficient to justify the effectiveness of our FedSC. Note that for large-scale datasets, the bottleneck of clustering is the eigenvalue decomposition of the Laplacian matrix, not our FedSC algorithm.

Acknowledgments

This work was partially supported by the Youth program 62106211 of the National Natural Science Foundation of China, the General Program JCYJ20210324130208022 of Shenzhen Fundamental Research, the research funding T00120210002 of Shenzhen Research Institute of Big Data, the Guangdong Key Lab of Mathematical Foundations for Artificial Intelligence, and the funding UDF01001770 of The Chinese University of Hong Kong, Shenzhen.

References

Jinyu Cai, Jicong Fan, Wenzhong Guo, Shiping Wang, Yunhe Zhang, and Zhao Zhang. Efficient deep embedded subspace clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1-10, June 2022.

Yi-Ruei Chen, Amir Rezapour, and Wen-Guey Tzeng. Privacy-preserving ridge regression on distributed data. Information Sciences , 451:34-49, 2018.

Don Kurian Dennis, Tian Li, and Virginia Smith. Heterogeneity for the win: One-shot federated clustering. In International Conference on Machine Learning , pages 2611-2620. PMLR, 2021.

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

Jicong Fan and Tommy W.S. Chow. Sparse subspace clustering for data with missing entries and high-rank matrix completion. Neural Networks , 93:36-44, 2017.

Jicong Fan and Madeleine Udell. Online high rank matrix completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Jicong Fan, Zhaoyang Tian, Mingbo Zhao, and Tommy W.S. Chow. Accelerated low-rank representation for subspace clustering and semi-supervised classification on large-scale data. Neural Networks , 100:39-48, 2018.

Jicong Fan, Chengrun Yang, and Madeleine Udell. Robust non-linear matrix factorization for dictionary learning, denoising, and clustering. IEEE Transactions on Signal Processing , 69:17551770, 2021.

Jicong Fan, Yiheng Tu, Zhao Zhang, Mingbo Zhao, and Haijun Zhang. A simple approach to automated spectral clustering. In Advances in Neural Information Processing Systems , volume 35, pages 9907-9921. Curran Associates, Inc., 2022.

Jicong Fan. Large-scale subspace clustering via k-factorization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , KDD '21, page 342-352, New York, NY, USA, 2021. Association for Computing Machinery.

Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems , 33:1958619597, 2020.

Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. Advances in Neural Information Processing Systems , 33:1406814080, 2020.

Elena Hernández-Pereira, Oscar Fontenla-Romero, Bertha Guijarro-Berdiñas, and Beatriz PérezSánchez. Federated learning approach for spectral clustering. In ESANN , 2021.

Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence , 16(5):550-554, 1994.

Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International conference on machine learning , pages 1376-1385. PMLR, 2015.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. On-device federated learning via blockchain and its latency analysis. arXiv preprint arXiv:1808.03949 , 2018.

Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Annals of Statistics , pages 1302-1338, 2000.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189 , 2019.

Li Li, Yuxi Fan, Mike Tse, and Kuo-Yi Lin. A review of applications in federated learning. Computers &Industrial Engineering , 149:106854, 2020.

Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine , 37(3):50-60, 2020.

Zitao Li, Bolin Ding, Ce Zhang, Ninghui Li, and Jingren Zhou. Federated matrix factorization with privacy guarantee. Proceedings of the VLDB Endowment , 15(4):900-913, 2021.

Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in Neural Information Processing Systems , 30, 2017.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

SA Nene, SK Nayar, and H Murase. Columbia university image library (coil-20). Technical Report CUCS-005-96 , 1996.

Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In Advances in Neural Information Processing Systems , volume 14. MIT Press, 2001.

Efthymios Tzinis, Jonah Casebeer, Zhepei Wang, and Paris Smaragdis. Separate but together: Unsupervised federated learning for speech enhancement from non-iid data. In 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) , pages 46-50. IEEE, 2021.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing , 17(4):395-416, 2007.

Shuai Wang and Tsung-Hui Chang. Federated matrix factorization: Algorithm design and application to data clustering. IEEE Transactions on Signal Processing , 70:1625-1640, 2022.

Hongtao Wang, Ang Li, Bolin Shen, Yuyan Sun, and Hongmei Wang. Federated multi-view spectral clustering. IEEE Access , 8:202249-202259, 2020.

Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903 , 2018.

Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST) , 10(2):1-19, 2019.

Enyue Yang, Yunfeng Huang, Feng Liang, Weike Pan, and Zhong Ming. Fcmf: Federated collective matrix factorization for heterogeneous collaborative filtering. Knowledge-Based Systems , 220:106946, 2021.

Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, and Xiaolin Li. Federated unsupervised representation learning. arXiv preprint arXiv:2010.08982 , 2020.

Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsupervised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4912-4921, 2021.

A Framework of the Proposed Algorithm

In the beginning, Z and C p for p = 1 , 2 , . . . , P are initialized randomly in the central server and clients, respectively. Then, Z is broadcast to every participating client and helps do the alternate updating of Z p and C p based on the update schemes (16) and (14). Afterward, the obtained matrix Z p is sent back to the central server and aggregated for the next round of training. When the tolerance condition is broken, both Z S p and C S p are sent back to the central server for the subsequent clustering task.

Algorithm 1 Proposed Federated Similarity Reconstruction

Input: Distributed data { X p , : p ∈ P := { 1 , 2 , . . . , P }} , clients weights { ω p : p ∈ P} . 1: Initialize Z 0 at server side and { C 0 p } P p =1 at client sides. 2: Randomly choose A 0 ⊆ P with |A s | = ¯ P . 3: for round s = 1 to S do 4: Server side: compute Z s -1 = 1 ¯ P ∑ p ∈A s -1 Z s -1 p . 5: Broadcast Z s -1 to clients c p , p ∈ A s . 6: Client side: 7: for client p = 1 to ¯ P in parallel do 8: set Z s, 0 p = Z s -1 9: update local variable C s p : 10: C s p = ( K ( Z s, 0 , Z s, 0 ) + λ I d ) -1 K ( Z s, 0 , X p ) 11: update local variable Z s p : 12: for t = 1 to Q do 13: Z s,t p = Z s,t -1 p -η s ∇ Z f p ( Z s,t -1 p ) 14: end for 15: denote Z s p = Z s,Q p 16: if client p ∈ A s -1 then 17: upload Z s p to the server. 18: end if 19: end for 20: Randomly choose A s ⊆ P with |A s | = ¯ P . 21: end for Output: Z , C p , p = 1 , 2 , . . . , P

B More theoretical results about FedSC with perturbed factors

Based on Theorem 3.8 and Theorem 3.6, we can get a bound on the variance of noise for FedSC with perturbed factors:

where B 1 = ϵ 2 -max i 1 4 [ max k ( K inter ik ) -min k ( K intra ik ) ] and γ zc ( σ Z , σ C ) is exactly the γ zc in Theorem 3.8 and is clearly a non-decreasing function with respect to σ Z and σ C , respectively. Therefore, it is a valid upper bound on both σ Z and σ C .

Proof. By Theorems 3.8 and 3.6, we have

That is, we need to solve a quadratic equation γ 2 zc +2 γ zc -B 1 = 0 with both γ zc ≥ 0 and B 1 ≥ 0 . Since the discriminant ∆ = 4 + 4 B 1 ≥ 0 , we have two roots -1 ± 2 √ 1 + B 1 for this equation. Thus, it has 0 ≤ γ zc ≤ -1 + 2 √ 1 + B 1 as desired.

This bound indicates that the intensity of noise should not be too strong otherwise it may seriously affect the performance of federated spectral clustering. But, at least under this bound, one can choose to inject as much noise as possible into the raw data to ensure data security and privacy.

Based on Theorem 3.9 and (34), we obtain the following privacy-utility trade-off:

This ensures both clustering performance and ( ε, δ ) -differential privacy. In particular, if we increase the intensity of either σ Z or σ C to reach the upper bound of γ zc , we can get a strong level of privacy but the worst utility. By the way, B 1 is related to the property of the data. A larger B 1 means a better property for clustering, which further provides a larger upper bound for the noise level, yielding a stronger privacy guarantee.

C More discussion on the privacy-utility trade-off of FedSC

Although theoretical results are presented in our study to ensure the security of FedSC, we still provide here some insight into methods of using Secure Aggregation or other cryptographic techniques to handle pair-wise client functions ( i.e. , kernel function in our study). Even though the communication cost will be high, it might be an alternative to reduce the DP noise levels. Specifically, it can be performed as follows.

Step 1: Z S p is posted to the central server;

Step 2: Central server aggregates Z S p to get the global Z S ;

Step 3: Central server computes K ZZ = K ( Z S , Z S ) and broadcast it to clients;

Step 4: For client p and c i ∈ C p , if c j ∈ C p , then client p directly calculate ˆ K ij = c i K ZZ c j ; if c j ∈ C p ′ , then client p firstly encrypts and transfers its c i to client p ′ , and then client p ′ also encrypts its c j and use the cipher text to compute enc ( ˆ K ij ) = enc ( c T i ) enc ( K ZZ ) enc ( c j ) ; Client p ′ transfers the result enc ( ˆ K ij ) back to client p ; Client p decrypts enc ( ˆ K ij ) to get ˆ K ij .

Step 5: Each client p sends its estimated results ˆ K ij back to the central server without sending its own C p ;

Step 6: Central Server checks whether these posted results from clients are compatible with each other in case of injection attacks and then performs spectral clustering.

This alternative does not send clients' C to the central server and gives an extra cross-validation process in the central server which may be useful to enhance security.

D More details and results of the experiments

It should be pointed out that in Table 2 of the main paper, the signal-noise ratio is as high as 12dB, which means the noise is tiny. The parameter d in our FedSC was automatically determined and has a much larger value than that in the noiseless case. That is why the performance of FedSC in the noisy case is even better than that in the noiseless case of some datasets. In this appendix, we increase the noise level ( σ e = βσ , where σ denotes the standard deviation of the clean data) and consider one more real dataset.

D.1 Dataset description

Synthetic data The synthetic data is generated from concentric circles. For θ i ∈ [0 , 2 π ] , i = 1 , 2 , . . . , 1258 , all the points of this synthetic dataset X ∈ R 2 × 1258 are generated by

where θ 0 = 0 , θ 1258 = 2 π , and the remaining θ i are evenly spaced points between θ 0 and θ 1258 . The additive noise e i 1 and e i 2 are drawn from N (0 , σ 2 e ) . We let σ e = 0 . 1 σ x , where σ x denotes the standard deviation of the data without the additive noise e . In our experiment, we set the hyperparameter r in (37) to 2 and 4 .

-5

0

5

-5

0

5

1

2

(a)

-5

0

5

-5

0

5

1

2

(b)

Real-world data Both iris and banknote authentication are from the UCI machine learning library. COIL20 is an image dataset from Columbia Imaging and Vision Laboratory. USPS is a dataset for handwritten text recognition research. The details of the mentioned datasets are shown in Table 2.

The visualizations (by t-SNE [Van der Maaten and Hinton, 2008]) of three real-world datasets are shown in Figure 5.

-5

0

5

-40

-20

0

-100

0

100

-100

0

100

(b) COIL20

-50

0

50

-50

0

50

(c) USPS

D.2 Evaluation metrics

We use two metrics to evaluate the clustering performance of our method: accuracy and normalized mutual information (NMI). Between them, accuracy is affected by the misclassification rate of cluster assignment. The smaller the misclassification rate, the greater the accuracy. In this study, given data matrix X ∈ R m × n with n sample points of m features, let L i and Lr i for i = 1 , 2 , . . . , n be the true labels and predictive labels, respectively, the accuracy of predictive cluster assignment Lr is computed by

NMI is a normalized version of mutual information that measures the agreement of two cluster assignments without considering their permutations.

where I ( L ; Lr ) is the mutual information between L and Lr , and H ( L ) and H ( Lr ) are the entropy of L and Lr , respectively.

D.3 Parameter settings

In our experiments, we set some hyperparameters including λ C , d, r , and k for implementing the proposed FedSC. Among them, λ C as the penalty parameter of the regularization term is set to 1 e -2 . k is the hyperparameter of the KNN-based operation on the similarity matrix. We set k to max( ceil (log( n )) , 1) based on [Von Luxburg, 2007]. The details of methods to set the remaining two parameters are as follows.

Setting of hyparameter r The hyperparameter r controls the smoothness of Gaussian kernel function. Due to the distinct characteristics of the datasets, we adopt the following adaptive method to determine the value of r : √

where c = 1 and D is the distance matrix of data points, of which its element

In reality, the global D cannot be obtained because FedSC does not allow data transmission. Then we compute D p for each client c p , p = 1 , . . . , P and then determine r as follows

where c can be tuned.

Setting of hyparameter d The hyperparameter d controls the complexity of the approximation ϕ ( X ) ≃ ϕ ( Z ) C . We adaptively determine the value of d for each dataset:

where s i denotes the i -th largest eigenvalue of the kernel matrix K xx , i = 1 , . . . , n . In our experiment, we set the threshold tol to 0 . 99 .

It is worth noting that we use the same r in the vanilla SC and our FedSC for a fair comparison, though D cannot be obtained in our FedSC. In addition, the setting of d relies on K xx that is also not obtainable in our FedSC. We use this setting for convenience and in reality we need to determine d by other methods such as letting d = γm , where γ is a hyperparameter.

D.4 Clustering results

NMI results The average results of 10 repeated trials are reported in Table 1 (ACC) and Table 3 (NMI). Note that for the USPS dataset, only 5 trials are performed to save time due to its large number of sample points. We see that our FedSC outperformed Kmeans and DSC significantly in almost all cases and has at least comparable performance as SC in most cases.

Influence of d The clustering accuracies with different d are reported in Table 4.

Results of FedSC with perturbed factors The results on COIL20 are reported in Table 5, where σ Z = α z std ( Z p ) and σ C = α c std ( C ) . Through this experiment, it can be observed that perturbing factors can have a more significant impact on the accuracy of clustering results than perturbing raw data. That is, perturbing factors are more sensitive and can achieve a specified level of differential privacy with weaker noise. Furthermore, it can be seen from Table 5 that the clustering performance is more sensitive to σ C than σ Z .

Malicious attack on C Due to the kernel trick, the optimization problem is nonlinear and nonconvex. Hence, it is very difficult for potential attackers to recover the data X from the uploaded factors Z , C , especially when X or Z , C are perturbed by noise. Nevertheless, the attacker may perform K-means on the { C p } P p =1 to obtain clustering results. However, we find that the clustering accuracy

on C is lower than those of Kmeans, SC, and FedSC reported in Table 1. For instance, the clustering accuracy on Iris is reported in Table.

Influence of P Table 7 compares the clustering performance between using a single client ( P = 1 ) and using multiple ones (P = 8). It is clear that the operation of splitting data across multiple clients may lead to an accuracy loss of clustering, which implies that our method is valid.

More results on MNIST and CIFAR10 We have already included datasets of high-dimensional images in Table 1 like USPS and COIL20 with sizes 16 × 16 and 20 × 20 , respectively. Nevertheless, to further improve the experiment, we added the results of MNIST ( 28 × 28 ) and CIFAR10( 32 × 32 ) in Table 8.

E Proof for theorem on the convergence of FedSC algorithm

We derive the objective descent with respect to C s and Z s,t , respectively.

Objective descent with w.r.t. C : Based on the update scheme (14) of C , we have

According to the proposed FedSC problem 7, we have

where γ s min = γ min ( K ( Z s, 0 , Z s, 0 )) .

Summing it up from p = 1 to P , we have

Objective descent with w.r.t. Z : According to Assumption 2.1, it implies

Now, we give a bound on T.2.

Lemma E.1. For any s and t , it holds that

Proof. Based on the update schemes of Z , we have

Then, consider the following terms

Based on Lemma 1, we continue to do the following derivation.

where we used the fact that ⟨ x , y ⟩ ≤ ∥ x ∥ 2 2 + ∥ y ∥ 2 2 , ∀ c > 0 . Now, we give a bound on T.3.

Lemma E.2. For any s and t , it holds that

Proof. For any s and t , it holds that

where we used the fact that ∥ ∑ n i =1 a i ∥ 2 2 ≤ n ∑ n i =1 ∥ a i ∥ 2 2 . Firstly, we give a bound on T.4:

where we used the fact that ∇ Z f p ( · , C s p ) is L s Z p -Lipschitz continuous.

Secondly, we give a bound on T. 5 :

Thirdly, we give a bound on T.6:

Here, it is the second time that we use the facts that ∥ ∑ n i =1 a i ∥ 2 2 ≤ n ∑ n i =1 ∥ a i ∥ 2 2 and that ∇ Z f p ( · , C s p ) is L s Z p -Lipschitz continuous.

Based on the bound of T.6, we derive the bound on T. 5 .

Fourthly, based on the bounds of T. 4 and T. 5 , we continue to derive the final required bound.

Based on Lemma 2, we continue to do the following derivation.

Then, summing it up from t = 1 to Q yields

Now, we give a bound on T.7.

Lemma E.3. For any s , it holds that

Proof. Based on the update schemes of C and Z , we have

Consequently, we have

where Z s, 0 p = Z s, 0 = Z s -1 .

Based on the above identities,

Based on Lemma 3, we give a bound on T.7.

Lemma E.4. For any s , it holds that

Proof. Based on Lemma 3, we have

Here, we used the inequality as

Rearranging the above inequality, we have

Based on Lemma 4, we continue to do the following derivation:

Then, combining (46) and (69) yields

Derivation of the main result: Based on (70), we derive the convergence in terms of the iterative terms, T C ( Z s, 0 , C s ) and T Z ( Z s,t , C s ) for s = 1 , 2 , . . . , S .

Then, summing it up from s = 1 to S yields

Similarly, we have

Then, summing it up from s = 1 to S yields

Lastly, combining (72) and (74) and dividing two sides of it by T = S (1 + Q ) yields

F Proof for theorem on error bound on noisy similarity matrix

Proof. It follows from the triangle inequality of matrix norm that

Since ˜ X = X + E and ϕ ( ˜ X ) = ϕ ( Z ) C , we have

where E is a Gaussian noise matrix, of which E i,j ∼ N (0 , σ 2 ) . Hence, we obtain

Denote by x i (or e i ) the i -th column of X ∈ R m × n (or E ∈ R m × n ), i = 1 , . . . , n , we first derive the upper bound of the second term of the RHS of (78) as follows:

where for the first inequality we have used the fact that the exponential function is locally Lipschitz continuous, i.e.,

Now, let us figure out the upper bound of ∥ e i -e j ∥ 2 2 . Note that

where e li represents the l -th element of the column vector e i , k = 1 , . . . , m . It is clear that for l = 1 , . . . , m ,

Hence, e li -e lj √ 2 σ is a standard Gaussian random variable drawn from N (0 , 1) . Based on this, we can define a random variable as

which is distributed according to the Chi-squared distribution with m degrees of freedom.

From Laurent and Massart [2000], we know that for any positive t , the Chi-squared variable Q with m degrees of freedom satisfies

Hence, a bound on ∥ e i -e j ∥ 2 2 with probability 1 -e -t is

Using union bound for (84), we have

holds with probability at least 1 -n ( n -1) e -t . Assume ∥ x i ∥ 2 ≤ θ , then ∥ x i -x i ∥ 2 ≤ 2 θ . For convenience, let ξ = √ m +2 √ mt +2 t . It follows from (79) and (85) that, with probability at least 1 -n ( n -1) e -t ,

Now, we figure out the upper bound of the first term of the RHS of (78). We have

where ∥·∥ 2 , ∞ is a norm such that ∥ X ∥ 2 , ∞ = max i ∥ X : ,i ∥ 2 for a real matrix X . Here we can just assume that ∥ ∥ ∥ ϕ ( Z ) C -ϕ ( ˜ X ) ∥ ∥ ∥ 2 , ∞ ≤ γ , γ is some constant; this relies on the optimization.

Moreover, assume ∥ C ∥ 2 ≤ τ C , we have

Hence, we can continue to do the derivation of the preceding inequality.

As a result, the overall bound is

G Proof for Theorem 3.6

Proof. For a data matrix X ∈ R m × n , it is perturbed by a Gaussian noise matrix E ∈ R m × n with N (0 , σ 2 ) to form the noisy data matrix ˜ X = X + E . Let K xx = K ( X , X ) be the ground truth and ˆ K ˜ x ˜ x = C T K ( Z , Z ) C be the approximated similarity matrix by FedKMF algorithm 1. Then, consider any two data points in X , x i and x j , with the identical label, we know that

where K iu = max k (( K xx ) inter ik ) and K ij = ( K xx ) ij .

After running FedKMF (Algorithm 1) on the noisy matrix ˜ X , if the approximated similarity matrix satisfies ∥ ∥ ∥ ˆ K ˜ x ˜ x -K xx ∥ ∥ ∥ < B ( σ ) , then consider two points in ˜ X , ˜ x i and ˜ x j , that are actually x i and x j perturbed by some noise, we have

where ˆ K iu = max k (( ˆ K ˜ x ˜ x ) inter ik ) and ˆ K ij = ( ˆ K ˜ x ˜ x ) ij .

Based on Definition 3.5, ˜ x i and ˜ x j can be correctly clustered only if the following inequality holds with some tolerance ϵ > 0 .

Thus, combining inequalities (92) and (93), the bound function B ( σ ) satisfies

Similarly, if we consider any two data points in X , x i and x j , with different labels, we know that

where K iv = min k (( K xx ) intra ik ) .

After running FedKMF algorithm 1 on the noisy matrix ˜ X , we have

where ˆ K iu = max k (( ˆ K ˜ x ˜ x ) intra ik ) .

Based on Definition 3.5, ˜ x i and ˜ x j can be correctly clustered only if the following inequality holds with some tolerance ϵ > 0 .

Thus, combining inequalities (96) and (97), the bound function B ( σ ) satisfies

Then, with two upper bounds on B ( σ ) , (94) and (98), we have

where ϵ is the parameter of tolerance.

Alternatively, a slightly looser version is like

where K iu = max k (( K xx ) inter ik ) and K iv = min k (( K xx ) intra ik ) .

H Proof for Proposition 3.7

Proof. The ℓ 2 -sensitivity [Dwork et al. , 2014] of a function f : N |X| → R k is:

where x ∼ y denotes that x and y are neighboring datasets. In our case, the function is f ( x ) = x . Then

It means ∆ 2 ( f ) ≤ 2 τ X . Now using Theorem 3.22 in [Dwork et al. , 2014] and Lemma H.1, we get the desired result.

Lemma H.1 (Post-Processing [Dwork et al. , 2014]) . Let M : N |X| → R be a randomized algorithm that is ( ε, δ ) -differentially private. Let h : R → R ′ be an arbitrary randomized mapping. Then h ◦ M : N |X| → R ′ is ( ε, δ ) -differentially private.

I Proof for Theorem 3.8

Proof. Based on the assumptions ∥ C ∥ 2 , ∞ ≤ τ C , ∥ ϕ ( Z ) C -ϕ ( X ) ∥ 2 , ∞ ≤ γ , ˜ C = C + E C for the entry ( E C ) ij ∼ N (0 , σ 2 C ) , and ˜ Z = Z + E Z for the entry ( E Z ) ij ∼ N (0 , σ 2 Z ) , we obtain

where ∥ ϕ ( X ) ∥ 2 , ∞ = 1 . An upper bound on ∥ ∥ ∥ ϕ ( ˜ Z ) ˜ C ∥ ∥ ∥ 2 , ∞ can be obtained only if we derive the upper bound on ∥ ∥ ∥ ϕ ( ˜ Z ) ˜ C -ϕ ( X ) ∥ ∥ ∥ 2 , ∞ . That is, if ∥ ∥ ∥ ϕ ( ˜ Z ) ˜ C -ϕ ( X ) ∥ ∥ ∥ 2 , ∞ ≤ γ zc , it implies that

which consequently gives

Hence, we derive the upper bound on ∥ ∥ ∥ ϕ ( ˜ Z ) ˜ C -ϕ ( X ) ∥ ∥ ∥ 2 , ∞ for the remaining proof.

where the second term ∥ ϕ ( Z ) C -ϕ ( X ) ∥ 2 , ∞ ≤ γ is the assumption. Next, we derive the upper bound on T. 2 .

where we used 1 σ 2 C ∥ E C ∥ 2 2 , ∞ ≤ ξ 2 d = d +2 √ dt +2 t with probability at least 1 -ne -t [Laurent and Massart, 2000] since it is the fact that 1 σ 2 C ∥ E C ∥ 2 2 , ∞ ∼ χ 2 d where the entry ( E C ) ij ∼ N (0 , σ 2 C ) . For T. 3 , we have

For T. 4 , we can obtain

where we use 1 σ 2 Z ∥ ( E Z ) : ,j ∥ 2 2 ≤ ξ 2 d = d +2 √ dt +2 t with probability at least 1 -de -t [Laurent and Massart, 2000] since it is the fact that 1 σ 2 Z ∥ ( E Z ) : ,j ∥ 2 2 ∼ χ 2 d where the entry ( E Z ) ij ∼ N (0 , σ 2 Z ) .

Hence, we can go back to give an upper bound on ∥ ∥ ∥ ϕ ( ˜ Z ) ˜ C -ϕ ( X ) ∥ ∥ ∥ 2 , ∞ :

Finally, we have

where

J Proof for Theorem 3.9

Lemma J.1. Assume Υ = max z j , x ′ ∥ z j -x ′ ∥ 2 and ∥ x -x ′ ∥ 2 ≤ 2 τ X , let { C S p } P p =1 be perturbed by noise drawn from N (0 , σ 2 ) with the parameter σ ≥ 2 cλ -1 √ dτ X ( τ X +Υ) r 2 ε for c 2 > 2 ln(1 . 25 /δ ) . Then, the Gaussian Mechanism that adds noise to { C S p } P p =1 is ( ε, δ ) -differentially private.

Lemma J.2. Assume max ( p,j ) {∥ x p j ∥ 2 , ∥ x ′ p j ∥ 2 } ≤ τ X , max ( i,j ) ∥ z i -x j ∥ ∞ = Υ , ∥ Z s p ∥ sp ≤ τ Z ∀ s , and ∥ C S ∥ 2 , ∞ ≤ τ C , let { Z s p } P p =1 for s = 1 , · · · , S be perturbed by noise drawn from N (0 , σ 2 ) with variance (8 S ∆ 2 ( g Z ) log( e + ( ε/δ )) /ε 2 ) where ∆( g Z ) = 2 √ dτ C τ X η k r 2 { 1 + ( τ X + τ Z ) ( τ X +Υ) r 2 } . Then, the Gaussian Mechanism that adds noise to { Z s p } P p =1 for s = 1 , · · · , S is ( ε, δ ) -differentially private.

By Lemma J.2, the mechanism that adds Gaussian noise to Z s p for s = 1 , · · · , S with variance (8 S ∆ 2 ( g Z ) log( e +( ε Z /δ Z )) /ε 2 Z ) satisfies ( ε Z , δ Z ) -differential privacy under S -fold adaptive composition for any ε Z > 0 and δ Z ∈ (0 , 1] . By Lemma J.1, the Gaussian Mechanism that injects noise to C S p with parameter σ ≥ 2 cλ -1 √ dτ X ( τ X +Υ) / ( r 2 ε C ) is ( ε C , δ C ) -differentially private. Therefore, by Theorem 3.16 of [Dwork et al. , 2014], the proposed algorithm that adds Gaussian noise to Z s p for s = 1 , · · · , S and C S p is ( ε C + ε Z , δ C + δ Z ) -differentially private. This finished the proof.

K Proof for Lemma J.1

Proof. In our FedSC, for each column of C , we have

where G = ( K ( Z , Z ) + λ I d ) -1 . We have

where ∥ · ∥ sp denotes the spectral norm of matrix. Since exp( z ) is locally Lipschitz continuous when z < 0 , we have

′ ′

Then according to Theorem 3.22 in [Dwork et al. , 2014], for c 2 > 2 ln(1 . 25 /δ ) the Gaussian Mechanism with parameter σ ≥ 2 cλ -1 √ dτ X ( τ X +Υ) r 2 ε is ( ε, δ ) -differentially private. This finished the proof.

L Proof for Lemma J.2

Proof. Proof Sketch The ( ϵ, δ ) -differential privacy of the proposed algorithm can be achieved by injecting noise into Z for each local update and into C at the final round. To prove this, we first compute the sensitivity of Z and C for determining the differential privacy of them. Then, we use the adaptive composition [Kairouz et al. , 2015] to get the superposition of them which will give the final theoretical result.

Now, the formal proof is as follows.

In our FedSC, consider one-step update of Z at client of p

where the derivative is given by

For the simplicity of the proof, we omit the pair of iteration parameters ( s, t ) and instead denote two adjacent local updates by k and k -1 for nonnegative k ≥ 1 . Thus, we have the equivalent version of a one-step update of Z p at client p

To compute the sensitivity of Z p , we give the counterpart of the above update as

Next, let's start to derive the upper bound on ∥ Z k p -( Z k p ) ′ ∥ F term by term.

For T. 1 , we have

For T. 3 , we have

Here, we use ∥ a ⊙ b ∥ 2 = √ ( a ⊙ a ) T ( b ⊙ b ) for a , b ∈ R d for the second equality; CauchySchwarz inequality for the second inequality; √ ∥ a ⊙ a ∥ 2 = ∥ a ∥ 4 for a ∈ R d for the third inequality. Let ∆ Z ,x = K ( Z , x ) -K ( Z , x ′ ) , then we have for T. 4 ,

Therefore, assume max {∥ x p j ∥ 2 , ∥ x ′ p j ∥ 2 } ≤ τ X , which means ∥ x -x ′ ∥ 2 ≤ 2 τ X , we can get an upper bound on T. 1 .

Here, we also used the fact that ∥ ∥ K ( Z k -1 p , x p j ) ∥ ∥ 2 ≤ √ d and the derived bound 2 √ dτ X ( τ X +Υ) r 2 on ∥ ∥ ∥ △ Z k -1 p , x p j ∥ ∥ ∥ 2 given by K, and ∥ C ∥ 2 , ∞ ≤ τ C given in Proof F.

Assume ∥ Z k p ∥ sp ≤ τ Z ∀ k , we have for T. 2

Thus, we get the upper bounds on T. 1 and T. 2 , respectively, and finally give an upper bound on ∥ ∥ Z k p -( Z k p ) ′ ∥ ∥ F .

Therefore, if we define Z p = g Z ( X p ) , the ℓ 2 -sensitivity of g Z is

By Theorem 4.3 of [Kairouz et al. , 2015], the mechanism that adds Gaussian noise to Z s p for s = 1 , · · · , S with variance (8 S ∆ 2 ( g Z ) log( e +( ε Z /δ Z )) /ε 2 Z ) satisfies ( ε Z , δ Z ) -differential privacy under S -fold adaptive composition for any ε Z > 0 and δ Z ∈ (0 , 1] . This finished the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a secure kernelized factorization method for federated spectral clustering on distributed data. It provides convergence guarantee of the optimization, reconstruction error bound of similarity matrix, and sufficient condition of correct clustering. It also presents guarantees of differential privacy. The effectiveness of the proposed method is demonstrated by synthetic and real datasets.\n\nSTRENGTHS:\n1. The paper studies federated spectral clustering, which is an important problem but was rarely studied in literature.\n\n2. The authors propose a federated spectral clustering algorithm and analyzed the convergence of the optimization algorithm. \n\n3. Besides the federated factorization model for similarity matrix reconstruction, the authors propose two approaches to enhancing the privacy of federated spectral clustering, one is to perturb the original data, the other is to perturb the learned factors.\n\n4. The authors provide a lot of theoretical guarantees for the proposed methods, including the reconstruction error (Theorem 3.1) of the true similarity matrix, condition of correct clustering (Theorem 3.6), and differential privacy analysis (Proposition 3.7 and Theorem 3.8). \n\n5. The numerical results show that the proposed FedSC is more effective than the baselines such as DSC.\n\nIn sum, this is a strong paper that addressed an important problem. It contains both theoretical analysis and numerical evaluation. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. Some theoretical result and all the experimental results of FedSC with perturbed factors are in the supplementary material. I suggest the authors put them back to the main paper via adjusting the structure of the paper, e.g., simplifying Table 1 in the main paper.\n\n2. A few points haven’t been sufficiently explained. Please refer to the session of Questions.\n\nQUESTIONS:\n1. In equation (11), why not use a fixed number of participated clients?\n\n2. In Theorem 3.1, was the $\\ell_{2,\\infty}$ norm defined as the maximum of the $\\ell_2$ norms of the matrix columns? If yes, the definition of $\\theta$ and $\\tau$ can also use the $\\ell_{2,\\infty}$ norm.\n\n3. Does formula (27) ensure the symmetry?\n\n4. Section 3.1 considered perturbed data while section 3.2 considered perturbed factors. Which strategy is better?\n\n5. In Theorem 3.8, the definition of $\\Upsilon$ is related to $\\mathbf{z}_i$, which does not include an $s$ or $S$. Is it a typo? What is the connection between the privacy protection and the reconstruction of similarity matrix or the performance of spectral clustering.\n\n6. In Figure 3(e), it is not clear why vanilla SC failed but FedSC still worked well.\n\n7. In Table 1, it seems that the performance of the baseline DSC is not satisfactory in most cases. Can the authors explain more about the result?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper solves unsupervised spectral clustering by reconstructing the similarity matrix under a Federated Learning (FL) and Differentially Private (DP) setting. The authors argue that since the similarity matrix requires computation across pairs of clients it might not be directly FL-compatible. Thus, the authors set out to approximate the feature-mapped matrix ($\\phi(X_p)$) for some client $p$ by an appropriate dictionary-coefficient product. The eventual similarity matrix is computed by the server by collecting each client's approximate dictionary-coefficient pair. In particular, since the raw data does not directly reach the server, the proposed algorithm FedSC is FL-compatible.\n\n The rest of the article demonstrates the convergence analysis of the proposed method, DP analysis and results, and empirical results. Especially, the FL results seem to converge well with large enough iterations for moderately sized datasets. However, the reviewer has some confusion regarding the DP results and their efficacy for high-dimensional matrices, the comparison against other traditional FL / distributed approaches, and the provided empirical results.\n\nSTRENGTHS:\nThe paper discusses a solution for the often under-studied field of Federated unsupervised learning. In particular, the authors propose a solution for spectral clustering that is FL-compatible while ensuring that the sum over the differences in the consecutive sequence terms generated by spectral clustering converges to a finite value over asymptotic $T$. \n\n1. Primarily, the authors manage to tackle the data pair problem faced in computing the similarity matrix. For simply computing the Gaussian kernel, a cross-term product between terms of different clients is required. Thus vanilla FL training no longer suffices without an assumption that clients are allowed to directly interact with one another. Thus, the general framework provided by the authors can be quite important for not only FL-based clustering but to reduce the communication cost of $O(N^2)$ required by non-private training where each client is allowed to interact with other clients. Here, $N$ is the total number of clients.\n2. The authors demonstrate the FL compatibility of FedSC via the proposed two-pronged optimization technique that generates a “close” similarity matrix to the original one while also ensuring that the errors generated due to FL are asymptotically small for a large number of iterations.\n3. Empirically, compared to previous FL spectral clustering approaches, the proposed FedSC performs the best for most scenarios. Furthermore, FedSC is Differential Privacy (DP) compatible too.\n4. The paper is clearly written and explores multiple avenues such as FL, DP-FL, and adding privacy noise to both the original data matrix as well as the outputs of the proposed algorithm.\n\nWEAKNESSES:\nAlthough the authors provide a unique approach to handling FL-based spectral clustering, some questions about their approach are detailed below.\n\n1. It is unclear why the assumption of $\\phi(X) = C\\phi(Z)$ is the best choice for the FL-only case. The aim of FL is to ensure that raw data is never transmitted to the server and the artifact transmitted to the server should not be easily invertible back to the raw data. Thus, instead of the proposed solution if we now approximate $\\phi(X) = f(Z) - C$ where $C$ is kept private to the client. Then, it might be interesting to understand whether such a transformation would suffice for FL and what benefits might the approach in the paper provide over this other approach (or another similar approach).\n2. The paper provides DP-FL guarantees for FedSC. However, the accuracy guarantee is weak for the high number of dimensions $m$ or high value of $d$ (dictionary and coefficient size). Particularly, from Thm. 3.1, the $l_{\\infty}$ norm is of the order $m$.\n3. Although this paper does not tackle a distributed setting, it will be beneficial to provide some insight into any possibilities of using Secure Aggregation or other cryptographic techniques to handle pair-wise client functions such as the kernel function. Although the communication cost will be high, there might be ways to reduce the DP noise levels.\n4. For the empirical evaluations, it is surprising that Vanilla spectral clustering performs worse than FedSC. Providing some intuition for this result will be helpful.\n5. Consider including a discussion section that demonstrates the benefits of the proposed approach over distributed learning (in terms of communication costs).\n\nQUESTIONS:\nThe paper is well-written and presented. In particular, the authors demonstrate a general framework that can compute pair-wise client functions without relying on a distributed learning assumption. There are no major concerns for the FL-only work. For the DP-FL work point 2 in the weaknesses section is important to address as it might render the DP-FL solution impractical for high dimensions. Point 3 in the same section might provide an alternative solution that could be explored further.\n\nLIMITATIONS:\nThe authors generally address most limitations well including the impact of large privacy noise and the impact of large-scale datasets (inherent limitation of spectral clustering).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on computing spectral clustering in the setting of horizontal federated learning. The main contributions of this paper are: (1) proposing a secure method for computing the adjacency matrix to achieve spectral clustering, (2) providing rigorous theoretical analysis to ensure the algorithm satisfies differential privacy, and (3) conducting experiments on multiple datasets to demonstrate the effectiveness of the proposed method, FedSC.\n\nSTRENGTHS:\nStrengths:\n(1) The paper is logically organized and well-written, effectively introducing the work on FedSC.\n(2) The proposed method, FedSC, is simple and reasonable, with theoretical guarantees. The paper is comprehensive (although detailed proof examination was not conducted).\n\nWEAKNESSES:\nI believe the experimental section of the paper can be further improved:\n\n(1) In the Stage II Spectral Clustering phase of FedSC, when clients upload to the server, is there a significant risk of privacy leakage? Is the solution to add noise (as mentioned in Section 3.2)?\n\n(2) Are both Stage I and Stage II of FedSC updated in each epoch? Can the author evaluate the difference in runtime between the two stages of FedSC and baseline models?\n\n\n(3) Experimental section:\n\na. Although the paper has theoretically demonstrated the privacy of FedSC, there is still a lack of direct evidence in the experimental section to prove the privacy preservation of FedSC. It is suggested that the author report the clustering accuracy recovered by attackers from the data uploaded by clients to the server, as this would intuitively reflect the emphasized privacy of FedSC.\n\nb. I would recommend including a natural baseline is to perform FedSC using only one client (e.g., just perform some kind of dimension reduction if Z is not too high-diemensional), which can give us better sense of the loss due to splitting if any and reduce the influence of this potentially denoise step (which is not novel) in the comparison.\n\nc. Why does Table 1 show that FedSC outperforms SC (such as in the Bank dataset)?Can the author provide further explanation? Why does FedSC in the Bank dataset show a large standard deviation and unstable performance in Table 1, while baselines do not have this issue? \n\nd. Is FedSC in Table 1 considering only perturbed data? Why not consider FedSC with perturbed factors? The Iris dataset and other datasets used in experiments are too simplistic. Can higher-dimensional image datasets be considered to demonstrate the effectiveness of the method?\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a secure kernelized factorization approach of spectral clustering. The proposed method works on distributed datasets while maintaining privacy. The proposed technique approximates the kernel matrix required as in the spectral clustering, in an effort to preserve the privacy. This paper analyses the convergence of the optimization algorithm and offers bounds on the reconstruction error of the kernel matrix. The efficacy of this proposed approach is supported by experiments on both synthetic and real image datasets.\n\nSTRENGTHS:\nThis paper studies an interesting problem: federated spectral clustering with privacy considerations. Since clustering requires access to the data, it is non-trivial to do clustering while maintaining privacy. The proposed method seems novel to me.\n\nWEAKNESSES:\nThe only major concern I have is the privacy claim in this paper, and it is a vital one. The proposed method is only meaningful when it is proven to be secure and private (while keeping good utility), otherwise one could simply use any clustering algorithms out there. However, the evidence regarding its privacy guarantee is not sufficient in my opinion. Therefore, I would suggest the authors put more efforts into grounding their claim on the privacy. \n\nConcretely, there are two (or three if more specifically) cases where security may be considered separately. \n1. No noise is injected, i.e., the method detailed in section 2 where no differential privacy is considered. In this case, it is implied that this federated algorithm can help privacy, e.g., see line 89 (To guarantee the privacy of information, problem (7) shall be solved\n90 in the framework of federated learning.) However, no evidence shows this federated operation can guarantee privacy. For example, one might apply the following idea to extract the local data $X_p$. Since equation (9) and (10) are quadratic minimization problem, they have explicit solutions. Therefore, by (10) it seems that one can write $Z^s_p$ as an expression of $X_p$ and $C^s_p$. Noting that by (9) $C^s_p$ can be written as an expression of $X_p$ and $Z^{s-1}$. Therefore, we have an equation of $Z^s, Z^{s-1}$ and $X_p$ for each $s$. Since $Z^s$ and $Z^{s-1}$ are known, can we solve $X_p$ from them? One argument for this idea not to work is that the dimension of the feature space may be high. However, there are also multiple such equations, i.e., $s=1,\\dots,S$. In general, the choice of the feature map $\\phi$, the number of communication rounds may also affect the level of privacy. Therefore, I am expecting to see such discussions regarding the \"privacy guarantee\" brought by the federated procedure.\n\n2. In addition to learning in a federated way, the author also considers adding noise to enhance privacy, as analyzed in a differential privacy way. I acknowledge that Proposition 3.7 and Theorem 3.8 shows the DP guarantee for the proposed approach. However, DP can be achieved by adding sufficiently large noise if not considering the utility. Therefore, it is important to see the privacy-utility trade-off. For example, theoretically, eq. 32 gives an upper bound on the variance of the noise of the clustering $\\sigma$, and proposition 3.7 gives a lower bound on the $\\sigma$ for the sake of differential privacy. Can one derive a trade-off relation from the two inequalities? Empirically, Table 1 shows the performance of the proposed method given certain level of noise being injected. Can the level of noise being associated with the level of DP guarantee?\n\nQUESTIONS:\nPlease refer to the weakness section for questions.\n\nLIMITATIONS:\nThe authors addressed some of the limitations, e.g., the methods are not tested on very large datasets. However, there may be some other potential limitations as discussed in the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-14"}