SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that  $r$  is the hyperparameter of the Gaussian kernel. In our experiment,  $r$  was automatically estimated as the mean of all pairwise distances between data points, i.e.,  $r = \frac{1}{n^2}\sum_{i,j}\| \pmb{x}_i - \pmb{x}_j\|_2$ . Assume  $|x_{ik} - x_{jk}| = \mathcal{O}(\varepsilon)$  for all  $i,j \in [n], k \in [m]$ , then  $\| \pmb{x}_i - \pmb{x}_j\| = \mathcal{O}(\sqrt{m}\varepsilon)$ , which means  $r^2$  is linear with  $m\varepsilon^2$ . Thus, the reconstruction error  $\| \hat{\pmb{K}}_{\tilde{x}\tilde{x}} - \pmb{K}_{xx}\|_{\infty}$  is upper bounded by  $\mathcal{O}(\sigma^2/\varepsilon^2 + \sqrt{d}\gamma\tau_C)$ , where  $\epsilon^2/\sigma^2$  can be regarded as a signal-noise ratio. Therefore, the bound is useful. In general, Theorem 3.1 indicates that when the added noise is small and the optimization makes  $\gamma$  small, the reconstruction error for the true similarity matrix is less than a small constant with high probability. This verified the effectiveness of our similarity reconstruction method.
It should be pointed out that  $\hat{K}_{\bar{x}\bar{x}}$  is not guaranteed to be a sparse matrix and hence the corresponding graph may not contain multiple connected components. We therefore use an extra KNN-based operation to get a sparse similarity matrix, which may also reduce the computational cost of eigenvalue decomposition when  $n$  is very large. Specifically, we let
$$
\hat {\boldsymbol {K}} _ {\bar {x} \bar {x}} = \operatorname {g e t S p a r s e M a t r i x b y K N N} \left(\hat {\boldsymbol {K}} _ {\bar {x} \bar {x}}, k\right) \tag {25}
$$
4We defer the proof for all theoretical results to the supplementary material.
which only keeps the largest  $k$  connections from each point to other points. Finally, we perform spectral clustering using  $\hat{K}_{\bar{x}\bar{x}}$ . The central server broadcasts the clustering results to each participating client.
As mentioned before, one can choose to inject some noise into its raw data to avoid privacy leakage. However, a question is how much noise we can add to the data to the largest extent for the guarantee of correct clustering. We first present the following definitions.
Definition 3.2 (Local neighbor set). Suppose  $\pmb{x}_i$  and  $\pmb{x}_j$  are data points of  $X \in \mathbb{R}^{m \times n}$  with class labels  $L_i$  and  $L_j$  respectively, and let  $\mathrm{KNN}(\pmb{x}_i)$  be the set of the k-nearest neighbors of  $\pmb{x}_i$ . We define
$$
\mathcal {N} _ {i} ^ {k, i n t r a} := \left\{\boldsymbol {x} _ {j} \in \boldsymbol {X} \mid L _ {i} = L _ {j} \text {a n d} \boldsymbol {x} _ {j} \in \operatorname {K N N} (\boldsymbol {x} _ {i}) \right\}. \tag {26}
$$
Definition 3.3 (Global neighbor set). Suppose  $\pmb{x}_i$  and  $\pmb{x}_j$  are data points of  $X \in \mathbb{R}^{m \times n}$  with class labels  $L_i$  and  $L_j$  respectively, and let  $\mathrm{KNN}(\pmb{x}_i)$  be the point set of k-nearest neighbors of  $\pmb{x}_i$ . We define
$$
\mathcal {N} _ {i} ^ {k, \text {g l o b a l}} := \left\{\boldsymbol {x} _ {j} \in \boldsymbol {X} \mid \boldsymbol {x} _ {j} \in \operatorname {K N N} \left(\boldsymbol {x} _ {i}\right) \right\}. \tag {27}
$$
If we call the local neighbor of data point  $\pmb{x}_i$  the intra-class neighbor of  $\pmb{x}_i$ , another definition called inter-class neighbor of  $\pmb{x}_i$  can be further introduced as follows.
Definition 3.4 (Inter-class neighbor set). Suppose  $\pmb{x}_i$  and  $\pmb{x}_j$  are data points of data matrix  $\pmb{X} \in \mathbb{R}^{m \times n}$  with class labels,  $L_i$  and  $L_j$ , respectively, and let  $\mathrm{KNN}(\pmb{x}_i)$  be the point set of k-nearest neighbors of  $\pmb{x}_i$ . We define
$$
\mathcal {N} _ {i} ^ {k, i n t e r} := \left\{x _ {j} \in \boldsymbol {X} \mid L _ {i} \neq L _ {j} \text {a n d} \boldsymbol {x} _ {j} \in \operatorname {K N N} \left(\boldsymbol {x} _ {i}\right) \right\}. \tag {28}
$$
Based on the above definitions, the following definition is presented to determine whether a data point can be correctly clustered or not.
Definition 3.5 (Correct clustering). Suppose  $\boldsymbol{x}_i \in \mathbb{R}^m$  and  $\boldsymbol{x}_j \in \mathbb{R}^m$  are data points of data matrix  $\boldsymbol{X} \in \mathbb{R}^{m \times n}$ ,  $\boldsymbol{x}_i$  is said to be correctly clustered with a tolerance of  $\epsilon$  if
a.  $\hat{\pmb{K}}_{ij}\geq \max_k(\hat{\pmb{K}}_{ik}^{inter}) - \epsilon$  for any of  $\pmb {x}_j\in \mathcal{N}_i^{k,intra}$
b.  $\hat{K}_{ij} \leq \min_k (\hat{K}_{ik}^{intra}) + \epsilon$  for any of  $x_j \in \mathcal{N}_i^{k,inter}$ .
Based on Definition 3.5, the following theorem gives the guarantee of our security-enhanced FedSC.
Theorem 3.6 (Guarantee of noisy spectral clustering). Let  $B(\sigma) = \frac{1}{r^2} \left[ (\sigma \xi + \sqrt{2}\theta)^2 - 2\theta^2 \right] + (\sqrt{d}\tau_C + 1)\gamma$ . Then with the probability of at least  $1 - n(n - 1)e^{-t}$ , performing spectral clustering using  $\hat{\pmb{K}}_{\hat{x}\hat{x}}$  yields correct clustering results if
$$
B (\sigma) \leq \frac {\epsilon}{2} - \max  _ {i} \frac {1}{4} \left[ \max  _ {k} \left(\boldsymbol {K} _ {i k} ^ {\text {i n t e r}}\right) - \min  _ {k} \left(\boldsymbol {K} _ {i k} ^ {\text {i n t r a}}\right) \right] \tag {29}
$$
where  $\pmb{K}_{ik}^{inter} = (\pmb{K}_{xx})_{ik}^{inter}$  and  $\pmb{K}_{ik}^{intra} = (\pmb{K}_{xx})_{ik}^{intra}$ .
Based on Theorem 3.1 and Theorem 3.6, we can get a bound on the variance of noise for FedSC with perturbed data:
$$
\sigma \leq \frac {1}{\xi} \left[ \sqrt {r ^ {2} \left(B _ {1} - B _ {2}\right) + 2 \theta^ {2}} - \sqrt {2} \theta \right] \tag {30}
$$
where  $B_{1} = \frac{\epsilon}{2} -\max_{i}\frac{1}{4}\left[\max_{k}(K_{ik}^{inter}) - \min_{k}(K_{ik}^{intra})\right]$  and  $B_{2} = (\sqrt{d}\tau_{C} + 1)\gamma$ . This bound indicates that the intensity of noise should not be too strong otherwise it may seriously affect the performance of federated spectral clustering. But, at least under this bound, one can choose to inject as much noise as possible into the raw data to ensure data security and privacy.
Using Theorem 3.22 in [Dwork et al., 2014] and the post-processing property of differential privacy, we have the following privacy guarantee for this enhanced FedSC algorithm.
Proposition 3.7. FedSC with perturbed data given by (22) is  $(\varepsilon, \delta)$ -differentially private if  $\sigma \geq 2c\tau_X / \varepsilon$ , where  $c^2 > 2\ln(1.25 / \delta)$ .
Based on this proposition and (30), we obtain the following privacy-utility trade-off:
$$
2 \sqrt {2 \ln 1 . 2 5 / \delta} \tau_ {X} / \varepsilon <   \sigma \leq \frac {1}{\xi} \left[ \sqrt {r ^ {2} \left(B _ {1} - B _ {2}\right) + 2 \theta^ {2}} - \sqrt {2} \theta \right]. \tag {31}
$$
This ensures both clustering performance and  $(\varepsilon, \delta)$ -differential privacy. In particular, if we substitute  $\sigma$  with the upper bound, we can get a strong level of privacy but the worst utility. By the way,  $B_{1} - B_{2}$  is related to the property of the data. A larger  $B_{1} - B_{2}$  means a better property for clustering, which further provides a larger upper bound for the noise level  $\sigma$ , yielding a stronger privacy guarantee.
3.2 FedSC with Perturbed Factors
In FedSC with perturbed factor, we added Gaussian noise to  $\mathbf{Z}$  in every round of the optimization but added Gaussian noise to  $\mathbf{C}$  in the last round of the optimization. To be more specific,  $\tilde{\boldsymbol{C}} = \boldsymbol{C} + \boldsymbol{E}_{\boldsymbol{C}}$ , and  $\tilde{\boldsymbol{Z}} = \boldsymbol{Z} + \boldsymbol{E}_{\boldsymbol{Z}}$ , where the entries of  $\boldsymbol{E}_{\boldsymbol{C}}$  and  $\boldsymbol{E}_{\boldsymbol{Z}}$  are drawn from  $\mathcal{N}(0,\sigma_{\bar{C}}^{2})$  and  $\mathcal{N}(0,\sigma_{\bar{Z}}^{2})$  respectively. Then we perform spectral clustering using the following reconstructed kernel matrix:
$$
\tilde {\boldsymbol {K}} _ {x x} = \tilde {\boldsymbol {C}} ^ {T} \mathcal {K} (\tilde {\boldsymbol {Z}}, \tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}}. \tag {32}
$$
The following theorem shows the reconstruction error bound for the ground truth kernel matrix  $\pmb{K}_{xx}$ . Theorem 3.8. Assume  $\| \phi(\pmb{Z})\pmb{C} - \phi(\pmb{X})\|_{2,\infty} \leq \gamma$ ,  $\| \pmb{C}\|_{2,\infty} \leq \tau_{\mathcal{C}}$ . Then with probability at least  $1 - (n + d)e^{-t}$ , it holds that
$$
\left\| \tilde {\boldsymbol {K}} _ {x x} - \boldsymbol {K} _ {x x} \right\| _ {\infty} \leq \gamma_ {z c} (\gamma_ {z c} + 2) \tag {33}
$$
where  $\gamma_{zc} = \gamma +\sqrt{d}\left(\sigma_{C}\xi_{d} + \tau_{C}\sqrt{2\left(1 - \exp\left(-\frac{\sigma_{Z}^{2}\xi_{d}^{2}}{2r^{2}}\right)\right)}\right)$  and  $\xi_d^2 = d + 2\sqrt{dt} +2t$
We see that, given a fixed  $\gamma$ , the reconstruction error becomes smaller if  $\sigma_Z$  and  $\sigma_C$  are smaller. Based on Theorem 3.8 and Definitions 3.2-3.5, we can obtain a bound similar to that in Theorem 3.6 to guarantee correct clustering, which will not be detailed here.
Theorem 3.9. In FedSC, assume  $\max_{(p,j)}\{\| \pmb{x}_{p_j}\| ,\| \pmb{x}_{p_j}'\|\} \leq \tau_X$ $\max_{(i,j)}\| \pmb {z}_i - \pmb {x}_j\|_\infty = \Upsilon$ $\| Z_p^s\|_{sp}\leq \tau_Z\forall s,$  and  $\| C^S\|_{2,\infty}\leq \tau_C$  , we perturb  $\{Z_p^s\}_{p = 1}^P$ $\forall s = 1,2,\ldots ,S$  with noise drawn from  $\mathcal{N}(0,\sigma_Z^2)$  with the parameter  $\sigma_Z\geq \sqrt{(8S\Delta^2(g_Z)\log(e + (\varepsilon_Z / \delta_Z)) / \varepsilon_Z^2)}$  where  $\Delta (g_Z) = \frac{2\sqrt{d}\tau_C\tau_X\eta_k}{r^2}\left\{1 + (\tau_X + \tau_Z)\frac{(\tau_X + \Upsilon)}{r^2}\right\}$  and perturb  $\{C_p^S\}_{p = 1}^P$  with noise drawn from  $\mathcal{N}(0,\sigma_C^2)$  with the parameter  $\sigma_C\geq 2c\lambda^{-1}\sqrt{d}\tau_X(\tau_X + \Upsilon) / (r^2\varepsilon_C)$  for  $c^2 >2\ln (1.25 / \delta_C)$ . Then, FedSC is  $(\varepsilon_C + \varepsilon_Z,\delta_C + \delta_Z)$ -differentially private.
Theorem 3.9 shows that our FedSC with perturbed factors can protect the data privacy provided that the noises added to  $C$  and  $Z$  are sufficiently large. Similarly to (31), we can also get a privacy-utility trade-off using Theorem 3.8 and Theorem 3.9, which is detailed in Appendix B.
It should be pointed out that the study on federated spectral clustering in literature is very limited. Besides our work, the only work that aims to address the problem is [Hernández-Pereira et al., 2021]. More introduction and discussion about the related work (federated matrix factorization/clustering [Yang et al., 2021; Ghosh et al., 2020; Dennis et al., 2021; Wang and Chang, 2022] and spectral clustering [Von Luxburg, 2007; Hernández-Pereira et al., 2021]) are in the supplementary material.
5.1 Performance on similarity reconstruction
Taking the COIL20 dataset [Nene et al., 1996] as an example, we first obtain the similarity matrix from vanilla spectral clustering based on the same kernel function. Then, we use the proposed method to derive the estimated similarity matrix  $\hat{K}_{\tilde{x}\tilde{x}}$  which is actually an approximation of ground truth. To
make it clearer, we also give the corresponding sparse similarity matrices by KNN sparsification (25). Figure 2 shows the similarity matrices constructed by different methods. We see that the proposed method can be able to successfully reconstruct the similarity matrix in the federated scenarios. The reconstruction errors on synthetic data, iris, banknote authentication, and COIL20 are in the supplementary material.
5.2 Clustering performance of FedSC
In this subsection, we check the clustering performance of the proposed security-enhanced FedSC method on both synthetic and real-world datasets. The synthetic dataset is generated from concentric circles. The details are in the supplementary. This synthetic dataset is visualized in Figure 3(a). Here, we continue to adopt the aforementioned COIL20 as an example of real-world datasets.
Taking the synthetic dataset as an example, the first group of cases helps illustrate the effectiveness of the proposed FedSC method. we first apply the vanilla spectral clustering method to the clean data. The predictive result is shown in Figure 3(b). It is clear that the vanilla spectral clustering method correctly clusters the data points lying in concentric circles. We then use the proposed FedSC method to cluster the data. One can find in Figure 3(c) that almost all of the data points also have been grouped correctly. However, when we inject some volume of noise
(b)
Figure 2: Visualization of similarity matrices: (a) similarity matrix of vanilla spectral clustering; (b) approximated similarity matrix of the proposed method; (c)(d) the corresponding sparse similarity matrices generated by KNN sparsification.
(a) Ground truth
(b) Vanilla SC
(c) FedSC
(d) Noisy data
Figure 3: FedSC on concentric circles: (a) ground truth; (b) cluster assignment generated by vanilla SC; (c) cluster assignment generated by FedSC; (d) Noisy ground truth; (e) cluster assignment generated by vanilla SC on noisy data; (f) cluster assignment generated by FedSC on noisy data.
(e) Vanilla SC
(f) FedSC
into the raw data, things may change a lot. Figure 3(d) is actually the ground truth Figure 3(a) adding some Gaussian noise. When focusing on this noisy data of concentric circles, we see from Figure 3(e) that the vanilla SC failed to cluster the data points while the proposed FedSC method is still able to correctly cluster the data to some extent as in Figure 3(f). As we know, the similarity graph directly constructed from raw data could be very sensitive to each data point. When we add too much noise, the similarity graph may fail to model the local neighborhood relationships which may be the reason why data points in Figure 3(e) are not separable for vanilla SC. Instead, FedSC is based on matrix factorization in the high-dimensional feature space and has a potential denoising effect. Therefore, it is possible for our method to achieve a better performance. The visualization of COIL20 can be found in the supplementary material.
5.3 Comparison with baselines
We compare our method with the clustering method DSC proposed by [Hernández-Pereira et al., 2021]. Because the existing literature on federated spectral clustering is rare, we here select both classic K-means and spectral clustering as the baselines. Two metrics including accuracy and NMI are adopted to evaluate the clustering results on four datasets including iris [Dua and Graff, 2017], COIL20 [Nene et al., 1996], banknote authentication [Dua and Graff, 2017], and USPS [Hull, 1994]. The details are in the supplementary material. Besides the clean data, we also consider adding noise to them to test the performance of methods under the condition of privacy protection. We directly inject Gaussian noise with zero mean and variance  $\sigma^2$  to the raw matrix  $X\in \mathbb{R}^{m\times n}$  as  $\tilde{\pmb{X}} = \pmb {X} + \pmb{E}$  where  $\pmb{E}$  is a Gaussian noise matrix, each element  $E_{i,j}$  of which is i.i.d. with  $\mathcal{N}(0,\sigma^2)$ .
Table 1 shows the clustering accuracy. Our FedSC almost always achieves comparable clustering results to vanilla SC. It even outperformed vanilla SC in some cases and K-means in most cases since FedSC has a potential denoising effect by approximating a similarity matrix. More importantly, FedSC significantly outperformed DSC in almost all cases. The reason is that DSC performs spectral clustering on each local dataset, which may lead to very unstable and inaccurate results.
Table 1: Comparison of clustering accuracy ( $X$  and  $\tilde{X}$  denote the raw data and corrupted data respectively). The results of NMI are in Section D.4.
Kmeans	SC	DSC	FedSC
X	Iris	0.8933 ± 0.0000	0.9000 ± 0.0000	0.5480 ± 0.0679	0.9000 ± 0.0031
COIL20	0.6113 ± 0.0534	0.8025 ± 0.0009	0.1009 ± 0.0100	0.7828 ± 0.0231
Bank	0.6122 ± 0.0000	0.5918 ± 0.0000	0.5582 ± 0.0045	0.7672 ± 0.1457
USPS	0.6704 ± 0.0047	0.6635 ± 0.0000	0.1686 ± 0.0014	0.6596 ± 0.0021
ORL	0.6325 ± 0.0270	0.7865 ± 0.0106	0.1653 ± 0.0073	0.7235 ± 0.0170
X with 0.1σ	Iris	0.8940 ± 0.0152	0.9120 ± 0.0332	0.4533 ± 0.0658	0.8993 ± 0.0299
COIL20	0.6283 ± 0.0484	0.8024 ± 0.0020	0.0995 ± 0.0122	0.7790 ± 0.0213
Bank	0.6067 ± 0.0022	0.6067 ± 0.0944	0.5558 ± 0.0023	0.7168 ± 0.1068
USPS	0.6732 ± 0.0035	0.6647 ± 0.0016	0.1690 ± 0.0007	0.6643 ± 0.0022
ORL	0.6195 ± 0.0329	0.7810 ± 0.0065	0.1600 ± 0.0089	0.7323 ± 0.0250
X with 0.3σ	Iris	0.8420 ± 0.0274	0.8327 ± 0.0267	0.4533 ± 0.0674	0.8427 ± 0.0404
COIL20	0.6422 ± 0.0366	0.7997 ± 0.0029	0.0981 ± 0.0084	0.7793 ± 0.0240
Bank	0.6020 ± 0.0038	0.5859 ± 0.0105	0.5588 ± 0.0074	0.6046 ± 0.0064
USPS	0.6704 ± 0.0063	0.6720 ± 0.0044	0.1673 ± 0.0019	0.6884 ± 0.0509
ORL	0.6098 ± 0.0167	0.7885 ± 0.0047	0.1665 ± 0.0057	0.7417 ± 0.0280
X with 0.5σ	Iris	0.7740 ± 0.0252	0.7313 ± 0.0494	0.3833 ± 0.0204	0.7540 ± 0.0336
COIL20	0.6389 ± 0.0296	0.7950 ± 0.0080	0.1033 ± 0.0167	0.7403 ± 0.0294
Bank	0.6051 ± 0.0076	0.5923 ± 0.0094	0.5566 ± 0.0030	0.6086 ± 0.0073
USPS	0.6699 ± 0.0031	0.7843 ± 0.0030	0.1683 ± 0.0017	0.7778 ± 0.0062
ORL	0.5983 ± 0.0295	0.7930 ± 0.0172	0.1615 ± 0.0096	0.7107 ± 0.0345
X with 0.7σ	Iris	0.6500 ± 0.0420	0.6087 ± 0.0468	0.3927 ± 0.0349	0.6120 ± 0.0455
COIL20	0.6220 ± 0.0627	0.7662 ± 0.0172	0.0893 ± 0.0055	0.6803 ± 0.0198
Bank	0.6100 ± 0.0112	0.6046 ± 0.0144	0.5566 ± 0.0035	0.6106 ± 0.0107
USPS	0.6638 ± 0.0044	0.7747 ± 0.0040	0.1675 ± 0.0004	0.7587 ± 0.0117
ORL	0.5723 ± 0.0360	0.7860 ± 0.0093	0.1613 ± 0.0066	0.6910 ± 0.0232
5.4 More numerical result
The tSNE visualization, clustering results in terms of NMI, the performance of FedSC with perturbed factors, etc, are in the supplementary material.
This paper has proposed a secure kernelized factorization method for federated spectral clustering on distributed data. We provide theoretical guarantees for optimization convergence, correct clustering, and differential privacy. The numerical experiments on synthetic and real image datasets verified the effectiveness of our method. To the best knowledge of the authors, this is the work that successfully addresses the problem of federated spectral clustering. One limitation of this work is that we haven't tested our FedSC on very large datasets, though the moderate-size datasets are sufficient to justify the effectiveness of our FedSC. Note that for large-scale datasets, the bottleneck of clustering is the eigenvalue decomposition of the Laplacian matrix, not our FedSC algorithm.
This work was partially supported by the Youth program 62106211 of the National Natural Science Foundation of China, the General Program JCYJ20210324130208022 of Shenzhen Fundamental Research, the research funding T00120210002 of Shenzhen Research Institute of Big Data, the Guangdong Key Lab of Mathematical Foundations for Artificial Intelligence, and the funding UDF01001770 of The Chinese University of Hong Kong, Shenzhen.
Jinyu Cai, Jicong Fan, Wenzhong Guo, Shiping Wang, Yunhe Zhang, and Zhao Zhang. Efficient deep embedded subspace clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-10, June 2022.
Yi-Ruei Chen, Amir Rezapour, and Wen-Guey Tzeng. Privacy-preserving ridge regression on distributed data. Information Sciences, 451:34-49, 2018.
Don Kurian Dennis, Tian Li, and Virginia Smith. Heterogeneity for the win: One-shot federated clustering. In International Conference on Machine Learning, pages 2611-2620. PMLR, 2021.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3-4):211-407, 2014.
Jicong Fan and Tommy W.S. Chow. Sparse subspace clustering for data with missing entries and high-rank matrix completion. *Neural Networks*, 93:36-44, 2017.
Jicong Fan and Madeleine Udell. Online high rank matrix completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Jicong Fan, Zhaoyang Tian, Mingbo Zhao, and Tommy W.S. Chow. Accelerated low-rank representation for subspace clustering and semi-supervised classification on large-scale data. *Neural Networks*, 100:39–48, 2018.
Jicong Fan, Chengrun Yang, and Madeleine Udell. Robust non-linear matrix factorization for dictionary learning, denoising, and clustering. IEEE Transactions on Signal Processing, 69:1755-1770, 2021.
Jicong Fan, Yiheng Tu, Zhao Zhang, Mingbo Zhao, and Haijun Zhang. A simple approach to automated spectral clustering. In Advances in Neural Information Processing Systems, volume 35, pages 9907-9921. Curran Associates, Inc., 2022.
Jicong Fan. Large-scale subspace clustering via k-factorization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD '21, page 342-352, New York, NY, USA, 2021. Association for Computing Machinery.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586-19597, 2020.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. Advances in Neural Information Processing Systems, 33:14068-14080, 2020.
Elena Hernández-Pereira, Oscar Fontenla-Romero, Bertha Guijarro-Berdiñas, and Beatriz Pérez-Sánchez. Federated learning approach for spectral clustering. In ESANN, 2021.
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550-554, 1994.
Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International conference on machine learning, pages 1376-1385. PMLR, 2015.
Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, and Xiaolin Li. Federated unsupervised representation learning. arXiv preprint arXiv:2010.08982, 2020.
Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsupervised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4912-4921, 2021.
A Framework of the Proposed Algorithm
In the beginning,  $Z$  and  $C_p$  for  $p = 1,2,\ldots ,P$  are initialized randomly in the central server and clients, respectively. Then,  $Z$  is broadcast to every participating client and helps do the alternate updating of  $Z_{p}$  and  $C_p$  based on the update schemes (16) and (14). Afterward, the obtained matrix  $Z_{p}$  is sent back to the central server and aggregated for the next round of training. When the tolerance condition is broken, both  $Z_{p}^{S}$  and  $C_p^S$  are sent back to the central server for the subsequent clustering task.
Algorithm 1 Proposed Federated Similarity Reconstruction
Input: Distributed data  $\{X_p:p\in \mathcal{P}:= \{1,2,\dots ,P\}\}$  clients weights  $\{\omega_{p}:p\in \mathcal{P}\}$  1: Initialize  $Z^0$  at server side and  $\{C_p^0\}_{p = 1}^P$  at client sides.   
2: Randomly choose  $\mathcal{A}^0\subseteq \mathcal{P}$  with  $|\mathcal{A}^s | = \bar{P}$    
3: for round  $s = 1$  to  $S$  do   
4: Server side: compute  $Z^{s - 1} = \frac{1}{P}\sum_{p\in \mathcal{A}^{s - 1}}Z_{p}^{s - 1}$    
5: Broadcast  $Z^{s - 1}$  to clients  $c_{p},p\in \mathcal{A}^{s}$    
6: Client side:   
7: for client  $p = 1$  to  $\bar{P}$  in parallel do   
8: set  $Z_{p}^{s,0} = Z^{s - 1}$    
9: update local variable  $C_p^s$    
10:  $C_p^s = (\mathcal{K}(Z^{s,0},Z^{s,0}) + \lambda I_d)^{-1}\mathcal{K}(Z^{s,0},X_p)$    
11: update local variable  $Z_{p}^{s}$    
12: for  $t = 1$  to  $Q$  do   
13:  $Z_{p}^{s,t} = Z_{p}^{s,t - 1} - \eta_{s}\nabla_{Z}f_{p}(Z_{p}^{s,t - 1})$    
14: end for   
15: denote  $Z_{p}^{s} = Z_{p}^{s,Q}$    
16: if client  $p\in \mathcal{A}^{s - 1}$  then   
17: upload  $Z_{p}^{s}$  to the server.   
18: end if   
19: end for   
20: Randomly choose  $\mathcal{A}^s\subseteq \mathcal{P}$  with  $|\mathcal{A}^s | = \bar{P}$    
21: end for   
Output:  $Z,C_p,p = 1,2,\ldots ,P$
B More theoretical results about FedSC with perturbed factors
Based on Theorem 3.8 and Theorem 3.6, we can get a bound on the variance of noise for FedSC with perturbed factors: