SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dong Qiao $^{1,2}$  Chris Ding $^{1}$  Jicong Fan $^{1,2*}$
<sup>1</sup>The Chinese University of Hong Kong, Shenzhen, China
2Shenzhen Research Institute of Big Data, Shenzhen, China
dongqiao@link.cuhk.edu.cn {chrisding,fanjicong}@cuhk.edu.cn
Federated learning has a significant advantage in protecting data and information privacy. Many scholars proposed various secure learning methods within the framework of federated learning but the study on secure federated unsupervised learning especially clustering is limited. We in this work propose a secure kernelized factorization method for federated spectral clustering on distributed data. The method is non-trivial because the kernel or similarity matrix for spectral clustering is computed by data pairs, which violates the principle of privacy protection. Our method implicitly constructs an approximation for the kernel matrix on distributed data such that we can perform spectral clustering under the constraint of privacy protection. We provide a convergence guarantee of the optimization algorithm, reconstruction error bounds of the Gaussian kernel matrix, and the sufficient condition of correct clustering of our method. We also present guarantees of differential privacy. Numerical results on synthetic and real datasets demonstrate that the proposed method is efficient and accurate in comparison to baselines.
In the era of big data, human beings can analyze massive data in various fields due to the improvement of storage and computational capabilities of computing devices [Li et al., 2020b]. Some popular fields such as artificial intelligence, machine learning, internet of things (IoT), and cloud computing have seen explosive development over the past few years. Nevertheless, a side effect of this trend is that individuals and organizations have more and more concerns about potential violation of privacy [Kairouz et al., 2021]. As a result, it has become a challenge to mine valuable information from user data but not to directly access it.
Federated learning [Kairouz et al., 2021; McMahan et al., 2017] can train a global model without retrieving dispersed data [Yang et al., 2018]. This advantage has made it so popular that many scholars have put much effort into the study of federated learning. For example, Yang et al. [2019] presented the definitions of horizontal federated learning, vertical federated learning, and federated transfer learning. Some privacy-preserving machine learning models were also presented. For instance, He et al. [2020] developed a federated group knowledge transfer algorithm to train small CNNs on edge devices. Chen et al. [2018] proposed a protocol to conduct privacy-preserving ridge regression over high-dimensional data. Besides, Kim et al. [2018] proposed a block-chained federated learning architecture that enables on-device learning without any central coordination.
Regardless of the great progress of federated learning, it can be found that most of the existing studies are for supervised learning [Li et al., 2020a; Ghosh et al., 2020]. Note that collecting labeled data may deserve very high cost in real situations [Li et al., 2020b] while unlabeled data are abundant. Thus, it
is necessary and important to study federated learning for unsupervised learning [Zhang et al., 2020; Tzinis et al., 2021; Zhuang et al., 2021; Dennis et al., 2021] such as clustering [Ng et al., 2001; Fan and Chow, 2017; Fan et al., 2018, 2021; Fan, 2021; Cai et al., 2022; Fan et al., 2022]. For example, Li et al. [2021] proposed a federated matrix factorization with a privacy guarantee for recommendation systems. Wang and Chang [2022] proposed two federated matrix factorization algorithms that can be used for federated clustering. Besides, there are some studies on federated spectral clustering. For instance, Wang et al. [2020] presented a federated multi-view spectral clustering method under the assumption that the data of each view are in one client. Hernández-Pereira et al. [2021] developed a cooperative spectral clustering model to deal with distributed data but the model is linear. However, the study on federated spectral clustering is still very limited and deserves more attention and effort.
In this paper, we propose a federated kernelized factorization method to reconstruct a similarity matrix for secure spectral clustering on distributed data. Our contributions are as follows.
- We propose a federated spectral clustering algorithm and provide convergence guarantee for the optimization.
- We further propose to add noise to the data or the learned factors to enhance the security of clustering and provide guarantees of differential privacy.
- We provide upper bounds for the reconstruction error of the true similarity matrix and theoretical guarantees for correct clustering.
We test our method on both synthetic data and real datasets in comparison to baselines, which verify the effectiveness of our method.
Notations We use  $y$ ,  $\mathbf{y}$ , and  $\mathbf{Y}$  to denote scalar, vector, and matrix, respectively. The element of  $\mathbf{Y}$  at row  $i$  and column  $j$  is denoted by  $y_{ij}$ . We use  $\| \cdot \|_2$  to denote the  $\ell_2$  norm of a vector and use  $\mathrm{Tr}(\cdot)$ ,  $\| \cdot \|_F$ , and  $\| \cdot \|_{sp}$  to denote the trace, Frobenius norm, and spectral norm of a matrix respectively. The  $\ell_{\infty}$  norm and  $\ell_{2,\infty}$  norm of a matrix  $\mathbf{Y}$  are defined as  $\| \mathbf{Y} \|_{\infty} = \max_{ij} |y_{ij}|$  and  $\| \mathbf{Y} \|_{2,\infty} = \max_j \sqrt{\sum_i y_{ij}^2}$  respectively.  $\mathbf{K}$ ,  $\mathcal{K}$ ,  $\mathcal{K}$ , and  $k$  denote the kernel matrix, kernel function, number of clusters, and the number  $k$  in KNN, respectively.  $\phi$  denotes the feature map induced by  $\mathcal{K}$ .
2 Federated Spectral Clustering (FedSC)
Suppose we have  $n$  data points of dimension  $m$  distributing in  $P$  clients. For convenience, we denote by  $\mathbf{X} \in \mathbb{R}^{m \times n}$  the matrix composed of all the  $n$  data points and denote by  $\mathbf{X}_p \in \mathbb{R}^{m \times N_p}$  the matrix composed of the  $N_p$  data points in client  $c_p$ , where  $N_p \geq 1$ ,  $p = 1, \ldots, P$ , and  $\sum_{p=1}^P N_p = n$ . Without loss of generality, we let  $\mathbf{X} = [\mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_P]$ , which means  $\{\mathbf{X}_p\}_{p=1}^P$  are submatrices of  $\mathbf{X}$ . Our goal is to perform spectral clustering on these data to partition them into  $\mathcal{K}$  groups, under the constraint that the data in each client cannot leave the client itself and the privacy of the data should be protected as much as possible, though there could be a central server conducting clustering.
The aforementioned task is non-trivial because in spectral clustering, the first step is constructing an adjacency matrix  $\mathbf{A} \in \mathbb{R}^{n \times n}$ , which has to evaluate the similarity between every data pair  $(\pmb{x}_i, \pmb{x}_j)$  using a metric function  $\mathcal{M}(\cdot, \cdot)$  and hence violates the privacy constraint in the task. To solve the problem, we present a federated spectral clustering model in this section.
2.1 Similarity Reconstruction via Feature Space Factorization
In spectral clustering, for  $\mathcal{M}(\cdot ,\cdot)$ , there are many choices such as  $k$  nearest neighbor similarity and various kernel functions. Let  $\mathcal{K}(\cdot ,\cdot)$  be a kernel function and we have
$$
\mathcal {K} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) = \phi \left(\boldsymbol {x} _ {i}\right) ^ {T} \phi \left(\boldsymbol {x} _ {j}\right), \tag {1}
$$
where  $\phi : \mathbb{R}^m \to \mathbb{R}^{m'}$  is a feature map induced by the kernel function and does not need to be carried out explicitly. When it comes to federated spectral clustering, the central server has no access
The most widely-used kernel is the Gaussian kernel  $\mathcal{K}(\pmb{x}_i, \pmb{x}_j) = \exp\left(-\frac{1}{2r^2}\|\pmb{x}_i - \pmb{x}_j\|_2^2\right)$ , of which the feature map  $\phi$  is an infinite-order polynomial feature map and  $r$  is a hyperparameter controlling the smoothness.
to the raw data distributed in clients and hence cannot compute  $\mathcal{K}(\pmb{x}_i, \pmb{x}_j)$  using (1). However, if the central server can learn an effective approximation (denoted by  $\widehat{\phi(\pmb{x}_i)}$ ) for each  $\phi(\pmb{x}_i)$  without accessing  $\pmb{x}_i$ ,  $\mathcal{K}(\pmb{x}_i, \pmb{x}_j)$  can be estimated, i.e.,
$$
\mathcal {K} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) \simeq \widehat {\boldsymbol {\phi} \left(\boldsymbol {x} _ {i}\right)} ^ {T} \widehat {\boldsymbol {\phi} \left(\boldsymbol {x} _ {j}\right)}. \tag {2}
$$
Thus, inspired by [Fan and Udell, 2019; Fan et al., 2021], we propose to approximate each  $\phi (\pmb {x}_i)$  by
$$
\widehat {\phi \left(\boldsymbol {x} _ {i}\right)} = \phi (\boldsymbol {Z}) \boldsymbol {c} _ {i}, \tag {3}
$$
where  $\mathbf{Z} = [z_{1}, z_{2}, \ldots, z_{d}] \in \mathbb{R}^{m \times d}$ ,  $\phi(\mathbf{Z}) = [\phi(z_{1}), \phi(z_{2}), \ldots, \phi(z_{d})]$ , and  $\mathbf{c}_{i} \in \mathbb{R}^{d}$ . Both  $\mathbf{Z}$  and  $\mathbf{c}_{i}$  are learned from individual columns of  $\mathbf{X}$  and they can be regarded as intermediate variables avoiding the direct access of central server to  $\mathbf{x}_{i}$  (details of the learning will be introduced later). It follows from (2) and (3) that
$$
\mathcal {K} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) \simeq \boldsymbol {c} _ {i} ^ {T} \phi (\boldsymbol {Z}) ^ {\top} \phi (\boldsymbol {Z}) \boldsymbol {c} _ {j}. \tag {4}
$$
Thus we can reconstruct the similarity between  $\pmb{x}_i$  and  $\pmb{x}_j$  via (4). For convenience, let  $K_{xx} = \mathcal{K}(\pmb{X},\pmb{X}) = \phi (\pmb{X})^\top \phi (\pmb{X}), K_{zz} = \mathcal{K}(\pmb{Z},\pmb{Z}) = \phi (\pmb{Z})^\top \phi (\pmb{Z}) \in \mathbb{R}^{d\times d}$ , and  $C = [c_1,c_2,\dots ,c_n]\in \mathbb{R}^{d\times n}$ . Then we have
$$
\boldsymbol {K} _ {x x} \simeq (\phi (\boldsymbol {Z}) \boldsymbol {C}) ^ {T} (\phi (\boldsymbol {Z}) \boldsymbol {C}) = \boldsymbol {C} ^ {T} \boldsymbol {K} _ {z z} \boldsymbol {C} \triangleq \hat {\boldsymbol {K}} _ {x x}. \tag {5}
$$
Now we use  $\pmb{K}_{xx}$  as a reconstructed similarity matrix for spectral clustering.
In the form of federated learning, we expand (3) to
$$
\phi (\boldsymbol {X} _ {p}) \simeq \phi (\boldsymbol {Z}) \boldsymbol {C} _ {p}, \quad p = 1, \dots , P. \tag {6}
$$
It indicates that  $Z$  is shared for all  $P$  clients and  $C_p$  is private for client  $c_p$ . Note that (6) is a matrix factorization problem in the feature space induced by a kernel on the data in client  $c_p$ ,  $p = 1, \ldots, P$ . Letting  $C = [C_1, \ldots, C_P]$ , we solve the following distributed optimization problem<sup>3</sup>
$$
\underset {Z, C} {\text {m i n i m i z e}} F (Z, C) \triangleq \sum_ {p = 1} ^ {P} \omega_ {p} f _ {p} \left(Z, C _ {p}\right). \tag {7}
$$
In (7),  $f_{p}$  is a local objective function for client  $c_{p}$  and  $\omega_{1},\ldots ,\omega_{P}$  are nonnegative weights for the clients. In this work, we let
$$
\begin{array}{l} f _ {p} (\boldsymbol {Z}, \boldsymbol {C} _ {p}) = \frac {1}{2} \| \phi (\boldsymbol {X} _ {p}) - \phi (\boldsymbol {Z}) \boldsymbol {C} _ {p} \| _ {F} ^ {2} + \frac {\lambda}{2} \| \boldsymbol {C} _ {p} \| _ {F} ^ {2} \\ = \frac {1}{2} \operatorname {T r} \left(\mathcal {K} \left(\boldsymbol {X} _ {p}, \boldsymbol {X} _ {p}\right)\right) - \operatorname {T r} \left(\boldsymbol {C} _ {p} ^ {T} \mathcal {K} (\boldsymbol {Z}, \boldsymbol {X} _ {p})\right) + \frac {1}{2} \operatorname {T r} \left(\boldsymbol {C} _ {p} ^ {T} \mathcal {K} (\boldsymbol {Z}, \boldsymbol {Z}) \boldsymbol {C} _ {p}\right) + \frac {\lambda}{2} \| \boldsymbol {C} _ {p} \| _ {F} ^ {2}, \tag {8} \\ \end{array}
$$
where  $\lambda \geq 0$  is a penalty parameter. To guarantee the privacy of information, problem (7) shall be solved in the framework of federated learning.
2.2 FedSC by Similarity Reconstruction and Model Averaging
In this section, we develop a FedSC algorithm by similarity reconstruction and model averaging. As a classic and popular framework, FederatedAveraging (or FedAvg) is first introduced in [McMahan et al., 2017] for federated learning. In our work, the proposed FedSC is, therefore, built up based on the backbone of FedAvg as in Figure 1. FedSC consists of two stages. The first stage, shown by the left plot of Figure 1, is federated similarity reconstruction, which constructs a similarity matrix in the manner of federated learning. The second stage, shown by the left plot of Figure 1, is using the reconstructed similarity matrix to implement spectral clustering.
Stage I Federated Similarity Reconstruction
Step ①: As the startup settings for our algorithm, the shared variable  $Z$  (i.e., the dictionary matrix  $Z$ ) and each local coefficient matrix  $C_p$  for  $p = 1,2,\dots ,P$  are initialized randomly in the central server and each client, respectively.
Note that we do not show the data  $X_{p}$  in the objective explicitly since it is absorbed into  $f_{p}$ .
Figure 1: Diagram of the proposed FedSC. Stage I (left plot): Federated Similarity Reconstruction (Steps 1-5). Stage II (right plot): Spectral Clustering (Steps 6-8).
Step ②: For each round  $s$ , where  $1 \leq s \leq S$ , the previous shared variable  $Z$  will firstly be broadcast to each participated client. After that, every client uses this received dictionary matrix  $Z$  to run its own iterative updates of local variables in the Local Update Module (LUM) as:
$$
\boldsymbol {C} _ {p} ^ {s} = \underset {\boldsymbol {C} _ {p}} {\arg \min } f _ {p} \left(\boldsymbol {Z} ^ {s - 1}, \boldsymbol {C} _ {p}\right) = \underset {\boldsymbol {C} _ {p}} {\arg \min } \frac {1}{2} \left\| \phi (\boldsymbol {X} _ {p}) - \phi \left(\boldsymbol {Z} ^ {s - 1}\right) \boldsymbol {C} _ {p} \right\| _ {F} ^ {2} + \frac {\lambda}{2} \| \boldsymbol {C} _ {p} \| _ {F} ^ {2} \tag {9}
$$
$$
\boldsymbol {Z} _ {p} ^ {s} = \underset {\boldsymbol {Z} _ {p}} {\arg \min } f _ {p} \left(\boldsymbol {Z} _ {p}, \boldsymbol {C} _ {p} ^ {s}\right) = \underset {\boldsymbol {Z} _ {p}} {\arg \min } \frac {1}{2} \left\| \phi \left(\boldsymbol {X} _ {p}\right) - \phi \left(\boldsymbol {Z} _ {p}\right) \boldsymbol {C} _ {p} ^ {s} \right\| _ {F} ^ {2} + \frac {\lambda}{2} \left\| \boldsymbol {C} _ {p} ^ {s} \right\| _ {F} ^ {2} \tag {10}
$$
Step ③: Each client sends back its own dictionary matrix  $Z_{p}^{s}, p = 1,2,\dots ,P$ , to the central server.
Step ④: The central server collects all (or a subset  $\mathcal{A}^{s-1}$  of) these uploaded matrices  $\{Z_{p}^{s}\}_{p=1}^{P}$  and averages them into one new matrix  $Z^{s}$  in Aggregation Module (AM), i.e.,
$$
Z ^ {s} = \frac {1}{\left| \mathcal {A} ^ {s - 1} \right|} \sum_ {p \in \mathcal {A} ^ {s - 1}} Z _ {p} ^ {s} \tag {11}
$$
where  $|\mathcal{A}^{s - 1}|$  is the number of participated clients. In our study, we fix the number of participating clients for each round  $s$ . Therefore, we use the notation  $\bar{P}$  instead of  $|\mathcal{A}^{s - 1}|$  in the sequel. This aggregated dictionary matrix  $Z^s$  will then be used to push the next round of federated iteration until the tolerance condition is broken.
Step ⑤: When Stage I comes to an end, the spectral clustering will start.
Stage II Spectral Clustering
Step ⑥: Each client sends  $(\mathbf{Z}_p^S, \mathbf{C}_p^S)$  back to the central server for the final aggregation of information.
Step ⑦: The required similarity matrix is then constructed based on the obtained dictionary matrix  $Z^S$  and coefficient matrix  $C^S$  in Spectral Clustering Module (SCM). Based on this approximated similarity matrix, the standard spectral clustering is implemented as usual:
$$
\mathbf {y} = \text {S p e c t r a l C l u s t e r i n g} \left(\boldsymbol {C} ^ {T} \mathcal {K} (\boldsymbol {Z}, \boldsymbol {Z}) \boldsymbol {C}, \mathcal {K}\right). \tag {12}
$$
Step ⑧: The central server broadcasts its clustering results to every corresponding client.
2.3 Optimization Algorithm for Federated Similarity Reconstruction
As described in the above section, alternate updating of local variables is a key to solving the proposed FedSC problem. In the following two parts, we discuss the optimization for  $Z$  and  $C$ , respectively.
For a client  $c_p$ , consider the corresponding local optimization problem
$$
\underset {\boldsymbol {Z}, \boldsymbol {C}} {\text {m i n i m i z e}} f _ {p} (\boldsymbol {Z}, \boldsymbol {C}) \tag {13}
$$
where  $f_{p}(\mathbf{Z},\mathbf{C}) = \frac{1}{2}\| \phi (\mathbf{X}_{p}) - \phi (\mathbf{Z})\mathbf{C}\|_{F}^{2} + \frac{\lambda}{2}\| \mathbf{C}\|_{F}^{2} = \frac{1}{2}\mathrm{Tr}(\mathcal{K}(\mathbf{X}_{p},\mathbf{X}_{p})) - \mathrm{Tr}(C^{T}\mathcal{K}(\mathbf{Z},\mathbf{X}_{p})) +$ $\frac{1}{2}\mathrm{Tr}(\boldsymbol {C}^T\mathcal{K}(\boldsymbol {Z},\boldsymbol {Z})\boldsymbol {C}) + \frac{\lambda}{2}\| \boldsymbol {C}\|_F^2$  . Let the derivative of  $f_{p}(\mathbf{Z},\mathbf{C})$  w.r.t.  $\mathbf{C}$  be zero, we get the following one-step update for  $\pmb{C}$  ..
$$
\boldsymbol {C} _ {p} ^ {s} = \left(\mathcal {K} \left(\boldsymbol {Z} ^ {s - 1}, \boldsymbol {Z} ^ {s - 1}\right) + \lambda \boldsymbol {I} _ {d}\right) ^ {- 1} \mathcal {K} \left(\boldsymbol {Z} ^ {s - 1}, \boldsymbol {X} _ {p}\right), \quad p = 1, 2, \dots , P. \tag {14}
$$
The derivative of  $f_{p}(Z,C)$  w.r.t.  $Z$  is
$$
\frac {\partial \mathcal {L}}{\partial \boldsymbol {Z}} = \frac {1}{\sigma^ {2}} \left(\boldsymbol {X} _ {p} \boldsymbol {W} _ {Z} - \boldsymbol {Z} \bar {\boldsymbol {W}} _ {Z}\right) + \frac {2}{\sigma^ {2}} \left(\boldsymbol {Z} \boldsymbol {Q} _ {Z} - \boldsymbol {Z} \bar {\boldsymbol {Q}} _ {Z}\right), \tag {15}
$$
where the intermediate variables are detailed as
$$
\boldsymbol {W} _ {Z} = - \boldsymbol {C} ^ {T} \odot \mathcal {K} (\boldsymbol {X} _ {p}, \boldsymbol {Z}) \quad \bar {\boldsymbol {W}} _ {Z} = \operatorname {d i a g} \left(\boldsymbol {1} _ {n} ^ {T} \boldsymbol {W} _ {Z}\right)
$$
$$
\boldsymbol {Q} _ {Z} = \left(0. 5 \boldsymbol {C} \boldsymbol {C} ^ {T}\right) \odot \mathcal {K} (\boldsymbol {Z}, \boldsymbol {Z}) \quad \bar {\boldsymbol {Q}} _ {Z} = \operatorname {d i a g} \left(\boldsymbol {1} _ {d} ^ {T} \boldsymbol {Q} _ {Z}\right).
$$
Here  $\mathbf{1}_n$  and  $\mathbf{1}_d$  are the column vectors with all elements of 1. Because of the kernel function,  $Z$  cannot be updated like  $C_p$ . Here, we use the gradient method to update it. At local iteration  $t$ , by setting  $Z_{p}^{s,0} = Z^{s - 1}$ , the update scheme of  $Z$  is
$$
\boldsymbol {Z} _ {p} ^ {s, t} = \boldsymbol {Z} _ {p} ^ {s, t - 1} - \eta_ {t} \frac {\partial f _ {p}}{\partial \boldsymbol {Z}} \left(\boldsymbol {Z} _ {p} ^ {s, t - 1}\right). \tag {16}
$$
where  $\eta_t$  is the step size and can be set as the reverse of the Lipschitz constant of gradient if possible. We summarize the optimization details in Algorithm 1 (shown in Appendix A).
2.4 Convergence Analysis of The Proposed Algorithm
First of all, it is obvious that all local objective functions  $f_{p}(\cdot ,\cdot)$  for  $p = 1,\ldots ,P$  are lower bounded. To analyze the convergence of Algorithm 1, we make two assumptions. The first one is the Lipschitz continuity of the gradient of the local objective functions.
Assumption 2.1. The gradients of all local objective functions  $f_{p}(\cdot ,\cdot)$  for  $p = 1,\dots ,P$  are  $L_{Z_p}^s$ -Lipschitz continuous in  $Z$ , that is
$$
\left\| \nabla_ {Z} f _ {p} \left(\boldsymbol {Z} ^ {s, t}, \boldsymbol {C} _ {p} ^ {s}\right) - \nabla_ {Z} f _ {p} \left(\boldsymbol {Z} ^ {s, t - 1}, \boldsymbol {C} _ {p} ^ {s}\right) \right\| _ {F} \leq L _ {Z _ {p}} ^ {s} \left\| \boldsymbol {Z} ^ {s, t} - \boldsymbol {Z} ^ {s, t - 1} \right\| _ {F}. \tag {17}
$$
In addition, there exist some lower and upper bounds for  $L_{Z_p}^s$ , i.e.,  $0 < \underline{L}_Z \leq L_{Z_p}^s \leq \overline{L}_Z$  hold for all  $p = 1, \ldots, P$  and  $s = 1, \ldots, S$ .
The second assumption, similar to [Li et al., 2019; Lian et al., 2017], is as follows.
Assumption 2.2. The difference between the local gradient and the global gradient is bounded as
$$
\left\| \nabla_ {Z} f _ {p} (\boldsymbol {Z}, \boldsymbol {C} _ {p}) - \nabla_ {Z} F (\boldsymbol {Z}, \boldsymbol {C}) \right\| _ {F} \leq \zeta , \quad \forall p = 1, \dots , P. \tag {18}
$$
To build the convergence condition, we define the following iterative terms of  $Z^{s,t}$  and  $C^s$  for all  $t = 1,\dots ,Q$  and  $s = 1,\ldots ,S$ :
$$
T _ {C} \left(\boldsymbol {Z} ^ {s, 0}, \boldsymbol {C} ^ {s}\right) = \sum_ {p = 1} ^ {P} \omega_ {p} \left\| \boldsymbol {C} _ {p} ^ {s} - \boldsymbol {C} _ {p} ^ {s - 1} \right\| _ {F} ^ {2} \tag {19}
$$
$$
T _ {Z} \left(\boldsymbol {Z} ^ {s, t}, \boldsymbol {C} ^ {s}\right) = \left\| \boldsymbol {Z} ^ {s, t} - \boldsymbol {Z} ^ {s, t - 1} \right\| _ {F} ^ {2}
$$
where the instantaneous average  $Z^{s,t}$  is defined as
$$
Z ^ {s, t} = \frac {1}{\bar {P}} \sum_ {p \in \mathcal {A} ^ {s}} Z _ {p} ^ {s, t}. \tag {20}
$$
Based on the above assumptions, we provide the following convergence guarantee for Algorithm 1.
Theorem 2.3 (Convergence of Algorithm 1). Suppose Assumption 2.1 and Assumption 2.2 hold. Let  $T = S(1 + Q)$  be the total number of global and local rounds. Then the sequence  $\{(Z^{s,t},C^s)\}$  generated by Algorithm 1 with stepsize  $1 / L_{Z_p}^s$  and  $\omega_{p} = \frac{N_{p}}{n}$  satisfies
$$
\frac {1}{T} \left[ \sum_ {s = 1} ^ {S} T _ {C} \left(\boldsymbol {Z} ^ {s, 0}, \boldsymbol {C} ^ {s}\right) + \sum_ {s = 1} ^ {S} \sum_ {t = 1} ^ {Q} T _ {Z} \left(\boldsymbol {Z} ^ {s, t}, \boldsymbol {C} ^ {s}\right) \right] \leq \frac {D}{T} \left[ F \left(\boldsymbol {Z} ^ {1, 0}, \boldsymbol {C} ^ {0}\right) - \underline {{f}} \right] + \frac {1 6 \zeta^ {2} \psi D}{\bar {P} \underline {{L}} _ {Z}} \tag {21}
$$
where  $\psi = 1 + \frac{(\bar{P} + 8)(Q - 1)(2Q - 1)}{\bar{P} - 4(Q - 1)^2(1 + \overline{L}_Z^2 / \underline{L}_Z^2)}$  and  $D = \frac{2}{\underline{\gamma}_{min} + \lambda} + \frac{4}{\underline{L}_Z}$ .
The proof can be found in Appendix E. We see that when  $T \to \infty$ , the algorithm converges to a finite value, which is small if  $\zeta$  is small and  $\overline{L}_{Z}$  is close to  $\underline{L}_{Z}$ .
3 Security-Enhanced FedSC
In order to enhance the security of FedSC, we present two noise-augmented variants of the proposed algorithm in this section.
3.1 FedSC with Perturbed Data
We add random noise to the data in each client and then perform Algorithm 1 to reconstruct a similarity matrix, which further improves the privacy of data. Specifically, the data  $\mathbf{X} \in \mathbb{R}^{m \times n}$  is perturbed by a noise matrix  $\mathbf{E} \in \mathbb{R}^{m \times n}$  to form the noisy data matrix
$$
\tilde {\boldsymbol {X}} = \boldsymbol {X} + \boldsymbol {E}, \tag {22}
$$
where  $E_{ij} \sim \mathcal{N}(0, \sigma^2)$ . We then perform Algorithm 1 with a Gaussian kernel of parameter  $r$  on  $\bar{\pmb{X}}$  and obtain  $\pmb{Z}, \pmb{C} = [C_1, C_2, \dots, C_P]$ , and
$$
\hat {\boldsymbol {K}} _ {\hat {x} \hat {x}} = \boldsymbol {C} ^ {T} \mathcal {K} (\boldsymbol {Z}, \boldsymbol {Z}) \boldsymbol {C}. \tag {23}
$$
We have the following reconstruction (for the true similarity matrix  $\pmb{K}_{xx} = \mathcal{K}(\pmb{X},\pmb{X})$ ) error bound<sup>4</sup>. Theorem 3.1 (Error bound of similarity matrix reconstruction). Suppose  $\| \pmb{X}\|_{2,\infty} = \theta$ ,  $\| \pmb{C}\|_{2,\infty} = \tau_C$ , and  $\left\| \phi (\pmb {Z})\pmb {C} - \phi (\tilde{\pmb{X}})\right\|_{2,\infty}\leq \gamma$ , where  $\theta, \tau_C,$  and  $\gamma$  are some nonnegative constants. Then with the probability at least  $1 - n(n - 1)e^{-t}$ , the reconstructed similarity matrix  $\hat{\pmb{K}}_{\tilde{x}\tilde{x}}$  satisfies
$$
\left\| \hat {\boldsymbol {K}} _ {\bar {x} \bar {x}} - \boldsymbol {K} _ {x x} \right\| _ {\infty} \leq \frac {1}{r ^ {2}} \left[ (\sigma \xi + \sqrt {2} \theta) ^ {2} - 2 \theta^ {2} \right] + (\sqrt {d} \tau_ {C} + 1) \gamma \tag {24}
$$
where  $\xi = \sqrt{(m + 2\sqrt{mt} + 2t)}$