SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} \left[ e _ {l i} - e _ {l j} \right] = 0, \tag {81}
$$
$$
\operatorname {v a r} \left[ e _ {l i} - e _ {l j} \right] = 2 \sigma^ {2}.
$$
Hence,  $\frac{e_{li} - e_{lj}}{\sqrt{2}\sigma}$  is a standard Gaussian random variable drawn from  $\mathcal{N}(0,1)$ . Based on this, we can define a random variable as
$$
Q = \sum_ {l = 1} ^ {m} \left(\frac {e _ {l i} - e _ {l j}}{\sqrt {2} \sigma}\right) ^ {2}, \tag {82}
$$
which is distributed according to the Chi-squared distribution with  $m$  degrees of freedom.
From Laurent and Massart [2000], we know that for any positive  $t$ , the Chi-squared variable  $Q$  with  $m$  degrees of freedom satisfies
$$
\mathbb {P} (Q > m + 2 \sqrt {m t} + 2 t) \leq 1 - e ^ {- t}. \tag {83}
$$
Hence, a bound on  $\| e_i - e_j\| _2^2$  with probability  $1 - e^{-t}$  is
$$
\left\| \boldsymbol {e} _ {i} - \boldsymbol {e} _ {j} \right\| _ {2} ^ {2} = 2 \sigma^ {2} Q \leq 2 \sigma^ {2} (m + 2 \sqrt {m t} + 2 t). \tag {84}
$$
Using union bound for (84), we have
$$
\max  _ {i, j} \| e _ {i} - e _ {j} \| _ {2} ^ {2} = 2 \sigma^ {2} Q \leq 2 \sigma^ {2} (m + 2 \sqrt {m t} + 2 t). \tag {85}
$$
holds with probability at least  $1 - n(n - 1)e^{-t}$ . Assume  $\| \pmb{x}_i\|_2 \leq \theta$ , then  $\| \pmb{x}_i - \pmb{x}_i\|_2 \leq 2\theta$ . For convenience, let  $\xi = \sqrt{m + 2\sqrt{mt} + 2t}$ . It follows from (79) and (85) that, with probability at least  $1 - n(n - 1)e^{-t}$ ,
$$
\begin{array}{l} \left\| \boldsymbol {K} _ {\tilde {x} \tilde {x}} - \boldsymbol {K} _ {x x} \right\| _ {\infty} \leq \frac {1}{2 r ^ {2}} \left[ 2 \sigma^ {2} \xi^ {2} + 2 \left\| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} \right\| _ {2} \sqrt {2} \sigma \xi \right] \\ \leq \frac {1}{r ^ {2}} \left[ \sigma^ {2} \xi^ {2} + 2 \sqrt {2} \sigma \xi \theta \right] \tag {86} \\ = \frac {1}{r ^ {2}} \left[ (\sigma \xi + \sqrt {2} \theta) ^ {2} - 2 \theta^ {2} \right]. \\ \end{array}
$$
Now, we figure out the upper bound of the first term of the RHS of (78). We have
$$
\begin{array}{l} \left\| \hat {\boldsymbol {K}} _ {\tilde {x} \tilde {x}} - \boldsymbol {K} _ {\tilde {x} \tilde {x}} \right\| _ {\infty} = \left\| \boldsymbol {C} ^ {T} \mathcal {K} (\boldsymbol {Z}, \boldsymbol {Z}) \boldsymbol {C} - \mathcal {K} (\tilde {\boldsymbol {X}}, \tilde {\boldsymbol {X}}) \right\| _ {\infty} \\ = \left\| (\boldsymbol {\phi} (\boldsymbol {Z}) \boldsymbol {C}) ^ {T} (\boldsymbol {\phi} (\boldsymbol {Z}) \boldsymbol {C}) - \boldsymbol {\phi} (\tilde {\boldsymbol {X}}) ^ {T} \boldsymbol {\phi} (\tilde {\boldsymbol {X}}) \right\| _ {\infty} \\ = \left\| (\phi (\boldsymbol {Z}) \boldsymbol {C}) ^ {T} (\phi (\boldsymbol {Z}) \boldsymbol {C}) - (\phi (\boldsymbol {Z}) \boldsymbol {C}) ^ {T} \phi (\tilde {\boldsymbol {X}}) + (\phi (\boldsymbol {Z}) \boldsymbol {C}) ^ {T} \phi (\tilde {\boldsymbol {X}}) - \phi (\tilde {\boldsymbol {X}}) ^ {T} \phi (\tilde {\boldsymbol {X}}) \right\| _ {\infty} \\ = \left\| (\phi (\boldsymbol {Z}) \boldsymbol {C}) ^ {T} (\phi (\boldsymbol {Z}) \boldsymbol {C} - \phi (\tilde {\boldsymbol {X}})) + (\phi (\boldsymbol {Z}) \boldsymbol {C} - \phi (\tilde {\boldsymbol {X}})) ^ {T} \phi (\tilde {\boldsymbol {X}}) \right\| _ {\infty} \\ \leq \left\| (\phi (\boldsymbol {Z}) \boldsymbol {C}) ^ {T} (\phi (\boldsymbol {Z}) \boldsymbol {C} - \phi (\tilde {\boldsymbol {X}})) \right\| _ {\infty} + \left\| (\phi (\boldsymbol {Z}) \boldsymbol {C} - \phi (\tilde {\boldsymbol {X}})) ^ {T} \phi (\tilde {\boldsymbol {X}}) \right\| _ {\infty} \\ \leq \| (\phi (\boldsymbol {Z}) \boldsymbol {C}) \| _ {2, \infty} \cdot \left\| \phi (\boldsymbol {Z}) \boldsymbol {C} - \phi (\tilde {\boldsymbol {X}}) \right\| _ {2, \infty} + \left\| \phi (\boldsymbol {Z}) \boldsymbol {C} - \phi (\tilde {\boldsymbol {X}}) \right\| _ {2, \infty} \cdot \left\| \phi (\tilde {\boldsymbol {X}}) \right\| _ {2, \infty} \\ = \left(\left\| (\phi (\boldsymbol {Z}) \boldsymbol {C}) \right\| _ {2, \infty} + \left\| \phi (\tilde {\boldsymbol {X}}) \right\| _ {2, \infty}\right) \left\| \phi (\boldsymbol {Z}) \boldsymbol {C} - \phi (\tilde {\boldsymbol {X}}) \right\| _ {2, \infty} \tag {87} \\ \end{array}
$$
where  $\| \cdot \|_{2,\infty}$  is a norm such that  $\| X\|_{2,\infty} = \max_i\| X_{:i}\| _2$  for a real matrix  $X$ . Here we can just assume that  $\left\| \phi (Z)C - \phi (\tilde{X})\right\|_{2,\infty}\leq \gamma$ ,  $\gamma$  is some constant; this relies on the optimization.
Moreover, assume  $\| C\| _2\leq \tau_C$  , we have
$$
\left\| \phi (\tilde {\boldsymbol {X}}) \right\| _ {2, \infty} = 1 \tag {88}
$$
$$
\| \phi (\boldsymbol {Z}) \boldsymbol {C} \| _ {2, \infty} \leq \| \phi (\boldsymbol {Z}) \| _ {F} \max  _ {j} \| \boldsymbol {C} _ {:, j} \| \leq \sqrt {d} \tau_ {C}
$$
Hence, we can continue to do the derivation of the preceding inequality.
$$
\begin{array}{l} \left\| \hat {\boldsymbol {K}} _ {\tilde {x} \tilde {x}} - \boldsymbol {K} _ {\tilde {x} \tilde {x}} \right\| _ {\infty} \leq \left(\| \phi (\boldsymbol {Z}) \boldsymbol {C} \| _ {2, \infty} + \left\| \phi (\tilde {\boldsymbol {X}}) \right\| _ {2, \infty}\right) \left\| \phi (\boldsymbol {Z}) \boldsymbol {C} - \phi (\tilde {\boldsymbol {X}}) \right\| _ {2, \infty} \tag {89} \\ \leq (\sqrt {d} \tau_ {C} + 1) \gamma \\ \end{array}
$$
As a result, the overall bound is
$$
\begin{array}{l} \left\| \hat {\boldsymbol {K}} _ {\tilde {x} \tilde {x}} - \boldsymbol {K} _ {x x} \right\| _ {\infty} \leq \left\| \hat {\boldsymbol {K}} _ {\tilde {x} \tilde {x}} - \boldsymbol {K} _ {\tilde {x} \tilde {x}} \right\| _ {\infty} + \| \boldsymbol {K} _ {\tilde {x} \tilde {x}} - \boldsymbol {K} _ {x x} \| _ {\infty} \tag {90} \\ \leq \frac {1}{r ^ {2}} \left[ (\sigma \xi + \sqrt {2} \theta) ^ {2} - 2 \theta^ {2} \right] + (\sqrt {d} \tau_ {C} + 1) \gamma \\ \end{array}
$$
G Proof for Theorem 3.6
Proof. For a data matrix  $\mathbf{X} \in \mathbb{R}^{m \times n}$ , it is perturbed by a Gaussian noise matrix  $\mathbf{E} \in \mathbb{R}^{m \times n}$  with  $\mathcal{N}(0, \sigma^2)$  to form the noisy data matrix  $\tilde{\mathbf{X}} = \mathbf{X} + \mathbf{E}$ . Let  $K_{xx} = \mathcal{K}(\mathbf{X}, \mathbf{X})$  be the ground truth and  $\hat{K}_{\tilde{x}\tilde{x}} = C^T\mathcal{K}(\mathbf{Z}, \mathbf{Z})\mathbf{C}$  be the approximated similarity matrix by FedKMF algorithm 1. Then, consider any two data points in  $\mathbf{X}$ ,  $\mathbf{x}_i$  and  $\mathbf{x}_j$ , with the identical label, we know that
$$
\boldsymbol {K} _ {i u} \leq \boldsymbol {K} _ {i j} \tag {91}
$$
where  $\pmb{K}_{iu} = \max_k((\pmb{K}_{xx})_{ik}^{inter})$  and  $\pmb{K}_{ij} = (\pmb{K}_{xx})_{ij}$ .
After running FedKMF (Algorithm 1) on the noisy matrix  $\tilde{\pmb{X}}$ , if the approximated similarity matrix satisfies  $\left\| \hat{\pmb{K}}_{\tilde{x}\tilde{x}} - \pmb{K}_{xx} \right\| < B(\sigma)$ , then consider two points in  $\tilde{\pmb{X}}$ ,  $\tilde{x}_i$  and  $\tilde{x}_j$ , that are actually  $x_i$  and  $x_j$  perturbed by some noise, we have
$$
\begin{array}{l} \hat {\boldsymbol {K}} _ {i u} - \hat {\boldsymbol {K}} _ {i j} = \hat {\boldsymbol {K}} _ {i u} - \boldsymbol {K} _ {i u} + \boldsymbol {K} _ {i u} - \hat {\boldsymbol {K}} _ {i j} + \boldsymbol {K} _ {i j} - \boldsymbol {K} _ {i j} \\ = \left(\hat {\boldsymbol {K}} _ {i u} - \boldsymbol {K} _ {i u}\right) + \left(\boldsymbol {K} _ {i j} - \hat {\boldsymbol {K}} _ {i j}\right) + \left(\boldsymbol {K} _ {i u} - \boldsymbol {K} _ {i j}\right) \\ \leq \left| \hat {\boldsymbol {K}} _ {i u} - \boldsymbol {K} _ {i u} \right| + \left| \boldsymbol {K} _ {i j} - \hat {\boldsymbol {K}} _ {i j} \right| + \left(\boldsymbol {K} _ {i u} - \boldsymbol {K} _ {i j}\right) \tag {92} \\ \leq B (\sigma) + B (\sigma) + \left(\boldsymbol {K} _ {i u} - \boldsymbol {K} _ {i j}\right) \\ = 2 B (\sigma) + \left(\boldsymbol {K} _ {i u} - \boldsymbol {K} _ {i j}\right) \\ \end{array}
$$
where  $\hat{K}_{iu} = \max_k((\hat{K}_{\tilde{x}\tilde{x}})_{ik}^{inter})$  and  $\hat{K}_{ij} = (\hat{K}_{\tilde{x}\tilde{x}})_{ij}$ .
Based on Definition 3.5,  $\tilde{\pmb{x}}_i$  and  $\tilde{\pmb{x}}_j$  can be correctly clustered only if the following inequality holds with some tolerance  $\epsilon >0$ .
$$
\hat {\boldsymbol {K}} _ {i u} - \hat {\boldsymbol {K}} _ {i j} \leq \epsilon \tag {93}
$$
Thus, combining inequalities (92) and (93), the bound function  $B(\sigma)$  satisfies
$$
B (\sigma) \leq \frac {1}{2} [ \epsilon - \left(\mathbf {K} _ {i u} - \mathbf {K} _ {i j}\right) ] \tag {94}
$$
Similarly, if we consider any two data points in  $X$ ,  $\pmb{x}_i$  and  $\pmb{x}_j$ , with different labels, we know that
$$
\boldsymbol {K} _ {i j} \leq \boldsymbol {K} _ {i v} \tag {95}
$$
where  $\pmb{K}_{iv} = \min_k((\pmb{K}_{xx})_{ik}^{intra})$ .
After running FedKMF algorithm 1 on the noisy matrix  $\tilde{X}$ , we have
$$
\begin{array}{l} \hat {\boldsymbol {K}} _ {i j} - \hat {\boldsymbol {K}} _ {i v} = \hat {\boldsymbol {K}} _ {i j} - \boldsymbol {K} _ {i j} + \boldsymbol {K} _ {i j} - \hat {\boldsymbol {K}} _ {i v} + \boldsymbol {K} _ {i v} - \boldsymbol {K} _ {i v} \\ = (\hat {\boldsymbol {K}} _ {i j} - \boldsymbol {K} _ {i j}) + (\boldsymbol {K} _ {i v} - \hat {\boldsymbol {K}} _ {i v}) + (K _ {i j} - K _ {i v}) \\ \leq \left| \hat {\boldsymbol {K}} _ {i j} - \boldsymbol {K} _ {i j} \right| + \left| \boldsymbol {K} _ {i v} - \hat {\boldsymbol {K}} _ {i v} \right| + \left(\boldsymbol {K} _ {i j} - \boldsymbol {K} _ {i v}\right) \tag {96} \\ \leq B (\sigma) + B (\sigma) + \left(\boldsymbol {K} _ {i j} - \boldsymbol {K} _ {i v}\right) \\ = 2 B (\sigma) + \left(\boldsymbol {K} _ {i j} - \boldsymbol {K} _ {i v}\right) \\ \end{array}
$$
where  $\hat{K}_{iu} = \max_k((\hat{K}_{\bar{x}\bar{x}})_{ik}^{intra})$
$$
\hat {\boldsymbol {K}} _ {i j} - \hat {\boldsymbol {K}} _ {i v} \leq \epsilon \tag {97}
$$
Thus, combining inequalities (96) and (97), the bound function  $B(\sigma)$  satisfies
$$
B (\sigma) \leq \frac {1}{2} [ \epsilon - \left(\boldsymbol {K} _ {i j} - \boldsymbol {K} _ {i v}\right) ] \tag {98}
$$
Then, with two upper bounds on  $B(\sigma)$ , (94) and (98), we have
$$
B (\sigma) \leq \frac {1}{2} \min  _ {i} \left\{\epsilon - \left(\boldsymbol {K} _ {i u} - \boldsymbol {K} _ {i j}\right), \epsilon - \left(\boldsymbol {K} _ {i j} - \boldsymbol {K} _ {i v}\right) \right\}. \tag {99}
$$
where  $\epsilon$  is the parameter of tolerance.
Alternatively, a slightly looser version is like
$$
B (\sigma) \leq \min  _ {i} \frac {1}{4} [ 2 \epsilon - \left(\boldsymbol {K} _ {i u} - \boldsymbol {K} _ {i v}\right) ] \tag {100}
$$
$$
\text {o r} \quad B (\sigma) \leq \frac {\epsilon}{2} - \max  _ {i} \frac {1}{4} \left(\boldsymbol {K} _ {i u} - \boldsymbol {K} _ {i v}\right).
$$
where  $\pmb{K}_{iu} = \max_k((\pmb{K}_{xx})_{ik}^{inter})$  and  $\pmb{K}_{iv} = \min_k((\pmb{K}_{xx})_{ik}^{intra})$ .
H Proof for Proposition 3.7
Proof. The  $\ell_2$ -sensitivity [Dwork et al., 2014] of a function  $f:\mathbb{N}^{|\mathcal{X}|}\to \mathbb{R}^k$  is:
$$
\Delta_ {2} (f) = \max _ {x \sim y} \| f (x) - f (y) \| _ {2},
$$
where  $x \sim y$  denotes that  $x$  and  $y$  are neighboring datasets. In our case, the function is  $f(x) = x$ . Then
$$
\left\| f (x) - f (y) \right\| _ {2} = \left\| x - y \right\| _ {2} \leq 2 \tau_ {X}.
$$
It means  $\Delta_2(f) \leq 2\tau_X$ . Now using Theorem 3.22 in [Dwork et al., 2014] and Lemma H.1, we get the desired result.
Lemma H.1 (Post-Processing [Dwork et al., 2014]). Let  $\mathcal{M}:\mathbb{N}^{|\mathcal{X}|}\to R$  be a randomized algorithm that is  $(\varepsilon ,\delta)$ -differentially private. Let  $h:R\to R^{\prime}$  be an arbitrary randomized mapping. Then  $h\circ \mathcal{M}:\mathbb{N}^{|\mathcal{X}|}\to R^{\prime}$  is  $(\varepsilon ,\delta)$ -differentially private.
I Proof for Theorem 3.8
Proof. Based on the assumptions  $\|C\|_{2,\infty} \leq \tau_C$ ,  $\|\phi(\mathbf{Z})\mathbf{C} - \phi(\mathbf{X})\|_{2,\infty} \leq \gamma$ ,  $\tilde{\mathbf{C}} = \mathbf{C} + \mathbf{E}_C$  for the entry  $(\mathbf{E}_C)_{ij} \sim \mathcal{N}(0, \sigma_C^2)$ , and  $\tilde{\mathbf{Z}} = \mathbf{Z} + \mathbf{E}_Z$  for the entry  $(\mathbf{E}_Z)_{ij} \sim \mathcal{N}(0, \sigma_Z^2)$ , we obtain
$$
\begin{array}{l} \left\| \tilde {\boldsymbol {K}} _ {x x} - \boldsymbol {K} _ {x x} \right\| _ {\infty} = \left\| \tilde {\boldsymbol {C}} ^ {T} \mathcal {K} (\tilde {\boldsymbol {Z}}, \tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} - \mathcal {K} (\boldsymbol {X}, \boldsymbol {X}) \right\| _ {\infty} \\ = \left\| (\phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}}) ^ {T} (\phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}}) - \phi^ {T} (\boldsymbol {X}) \phi (\boldsymbol {X}) \right\| _ {\infty} \\ = \left\| (\phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}}) ^ {T} (\phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}}) - (\phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}}) ^ {T} \phi (\boldsymbol {X}) + (\phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}}) ^ {T} \phi (\boldsymbol {X}) - \phi^ {T} (\boldsymbol {X}) \phi (\boldsymbol {X}) \right\| _ {\infty} \\ = \left\| (\phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}}) ^ {T} (\phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}}) - \phi (\boldsymbol {X})) + (\phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} - \phi (\boldsymbol {X})) ^ {T} \phi (\boldsymbol {X}) \right\| _ {\infty} \\ \leq \left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} \right\| _ {2, \infty} \left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} - \phi (\boldsymbol {X}) \right\| _ {2, \infty} + \left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} - \phi (\boldsymbol {X}) \right\| _ {2, \infty} \| \phi (\boldsymbol {X}) \| _ {2, \infty} \\ \leq \left(\left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} \right\| _ {2, \infty} + \| \phi (\boldsymbol {X}) \| _ {2, \infty}\right) \underbrace {\left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} - \phi (\boldsymbol {X}) \right\| _ {2 , \infty}} _ {\text {T . 1}} \tag {101} \\ \end{array}
$$
where  $\| \phi (\pmb {X})\|_{2,\infty} = 1$ . An upper bound on  $\left\| \phi (\tilde{\pmb{Z}})\tilde{\pmb{C}}\right\|_{2,\infty}$  can be obtained only if we derive the upper bound on  $\left\| \phi (\tilde{\pmb{Z}})\tilde{\pmb{C}} -\phi (\pmb {X})\right\|_{2,\infty}$ . That is, if  $\left\| \phi (\tilde{\pmb{Z}})\tilde{\pmb{C}} -\phi (\pmb {X})\right\|_{2,\infty}\leq \gamma_{zc}$ , it implies that
$$
\left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} \right\| _ {2, \infty} \leq \left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} - \phi (\boldsymbol {X}) \right\| _ {2, \infty} + \| \phi (\boldsymbol {X}) \| _ {2, \infty} = \gamma_ {z c} + 1 \tag {102}
$$
which consequently gives
$$
\left\| \tilde {\boldsymbol {K}} _ {x x} - \boldsymbol {K} _ {x x} \right\| _ {\infty} \leq \gamma_ {z c} (\gamma_ {z c} + 2) \tag {103}
$$
Hence, we derive the upper bound on  $\left\| \phi (\tilde{\boldsymbol{Z}})\tilde{\boldsymbol{C}} -\phi (\boldsymbol {X})\right\|_{2,\infty}$  for the remaining proof.
$$
\begin{array}{l} \left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} - \phi (\boldsymbol {X}) \right\| _ {2, \infty} = \left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} - \phi (\boldsymbol {Z}) \boldsymbol {C} + \phi (\boldsymbol {Z}) \boldsymbol {C} - \phi (\boldsymbol {X}) \right\| _ {2, \infty} \\ \leq \underbrace {\left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} - \phi (\boldsymbol {Z}) \boldsymbol {C} \right\| _ {2 , \infty}} _ {\text {T . 2}} + \| \phi (\boldsymbol {Z}) \boldsymbol {C} - \phi (\boldsymbol {X}) \| _ {2, \infty} \tag {104} \\ \end{array}
$$
where the second term  $\| \phi (\pmb {Z})\pmb {C} - \phi (\pmb {X})\|_{2,\infty}\leq \gamma$  is the assumption. Next, we derive the upper bound on  $T,2$
$$
\begin{array}{l} \left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} - \phi (\boldsymbol {Z}) \boldsymbol {C} \right\| _ {2, \infty} = \left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} - \phi (\tilde {\boldsymbol {Z}}) \boldsymbol {C} + \phi (\tilde {\boldsymbol {Z}}) \boldsymbol {C} - \phi (\boldsymbol {Z}) \boldsymbol {C} \right\| _ {2, \infty} \\ = \left\| \phi (\tilde {\boldsymbol {Z}}) (\tilde {\boldsymbol {C}} - \boldsymbol {C}) + (\phi (\tilde {\boldsymbol {Z}}) - \phi (\boldsymbol {Z})) \boldsymbol {C} \right\| _ {2, \infty} \\ \leq \left\| \phi (\tilde {\boldsymbol {Z}}) (\tilde {\boldsymbol {C}} - \boldsymbol {C}) \right\| _ {2, \infty} + \left\| (\phi (\tilde {\boldsymbol {Z}}) - \phi (\boldsymbol {Z})) \boldsymbol {C} \right\| _ {2, \infty} \tag {105} \\ \leq \sqrt {d} \| \boldsymbol {E} _ {C} \| _ {2, \infty} + \left\| \phi (\tilde {\boldsymbol {Z}}) - \phi (\boldsymbol {Z}) \right\| _ {F} \| \boldsymbol {C} \| _ {2, \infty} \\ \leq \sigma_ {C} \xi_ {d} \sqrt {d} + \tau_ {C} \underbrace {\left\| \phi (\tilde {\pmb {Z}}) - \phi (\pmb {Z}) \right\| _ {F}} _ {\mathrm {T . 3}} \\ \end{array}
$$
where we used  $\frac{1}{\sigma_C^2}\|E_C\|_{2,\infty}^2 \leq \xi_d^2 = d + 2\sqrt{dt} + 2t$  with probability at least  $1 - n e^{-t}$  [Laurent and Massart, 2000] since it is the fact that  $\frac{1}{\sigma_C^2}\|E_C\|_{2,\infty}^2 \sim \chi_d^2$  where the entry  $(E_C)_{ij} \sim \mathcal{N}(0, \sigma_C^2)$ . For  $T.3$ , we have
$$
\begin{array}{l} \left\| \phi (\tilde {\boldsymbol {Z}}) - \phi (\boldsymbol {Z}) \right\| _ {F} ^ {2} = \mathrm {T r} ((\phi (\tilde {\boldsymbol {Z}}) - \phi (\boldsymbol {Z})) ^ {T} (\phi (\tilde {\boldsymbol {Z}}) - \phi (\boldsymbol {Z}))) \\ = \operatorname {T r} \left(\phi^ {T} (\tilde {\boldsymbol {Z}}) \phi (\tilde {\boldsymbol {Z}}) - \phi^ {T} (\tilde {\boldsymbol {Z}}) \phi (\boldsymbol {Z}) - \phi^ {T} (\boldsymbol {Z}) \phi (\tilde {\boldsymbol {Z}}) + \phi^ {T} (\boldsymbol {Z}) \phi (\boldsymbol {Z})\right) \tag {106} \\ = 2 d - 2 \underbrace {\operatorname {T r} (\phi^ {T} (\tilde {\boldsymbol {Z}}) \phi (\boldsymbol {Z}))} _ {\text {T . 4}} \\ \end{array}
$$
For  $T.4$ , we can obtain
$$
\begin{array}{l} \operatorname {T r} \left(\phi^ {T} (\tilde {\boldsymbol {Z}}) \phi (\boldsymbol {Z})\right) = \sum_ {j = 1} ^ {d} \phi^ {T} \left(\tilde {\boldsymbol {z}} _ {j}\right) \phi \left(\boldsymbol {z} _ {j}\right) = \sum_ {j = 1} ^ {d} \exp \left(- \frac {\| \boldsymbol {z} _ {j} + (\boldsymbol {E} _ {Z}) _ {: , j} - \boldsymbol {z} _ {j} \| _ {2} ^ {2}}{2 r ^ {2}}\right) \\ = \sum_ {j = 1} ^ {d} \exp \left(- \frac {\| \left(\boldsymbol {E} _ {Z}\right) _ {: , j} \| _ {2} ^ {2}}{2 r ^ {2}}\right) \tag {107} \\ \geq d \exp \left(- \frac {\sigma_ {Z} ^ {2} \xi_ {d} ^ {2}}{2 r ^ {2}}\right) \\ \end{array}
$$
where we use  $\frac{1}{\sigma_Z^2}\| (\pmb {E}_Z)_{:,j}\| _2^2\leq \xi_d^2 = d + 2\sqrt{dt} +2t$  with probability at least  $1 - de^{-t}$  [Laurent and Massart, 2000] since it is the fact that  $\frac{1}{\sigma_Z^2}\| (\pmb {E}_Z)_{:,j}\| _2^2\sim \chi_d^2$  where the entry  $(\pmb {E}_Z)_{ij}\sim \mathcal{N}(0,\sigma_Z^2)$ .
Hence, we can go back to give an upper bound on  $\left\| \phi (\tilde{\pmb{Z}})\tilde{\pmb{C}} -\phi (\pmb {X})\right\|_{2,\infty}$
$$
\left\| \phi (\tilde {\boldsymbol {Z}}) \tilde {\boldsymbol {C}} - \phi (\boldsymbol {X}) \right\| _ {2, \infty} \leq \gamma + \sigma_ {C} \xi_ {d} \sqrt {d} + \tau_ {C} \sqrt {2 d \left(1 - \exp \left(- \frac {\sigma_ {Z} ^ {2} \xi_ {d} ^ {2}}{2 r ^ {2}}\right)\right)} \tag {108}
$$
$$
\left\| \tilde {\boldsymbol {K}} _ {x x} - \boldsymbol {K} _ {x x} \right\| _ {\infty} \leq \gamma_ {z c} (\gamma_ {z c} + 2) \tag {109}
$$
$$
\gamma_ {z c} = \gamma + \sqrt {d} \left(\sigma_ {C} \xi_ {d} + \tau_ {C} \sqrt {2 \left(1 - \exp \left(- \frac {\sigma_ {Z} ^ {2} \xi_ {d} ^ {2}}{2 r ^ {2}}\right)\right)}\right). \tag {110}
$$
J Proof for Theorem 3.9
Lemma J.1. Assume  $\Upsilon = \max_{\boldsymbol{z}_j,\boldsymbol{x}'}\| \boldsymbol{z}_j - \boldsymbol{x}'\| _2$  and  $\| \pmb {x} - \pmb{x}^{\prime}\|_{2}\leq 2\tau_{X}$ , let  $\{C_p^S\}_{p = 1}^P$  be perturbed by noise drawn from  $\mathcal{N}(0,\sigma^2)$  with the parameter  $\sigma \geq \frac{2c\lambda^{-1}\sqrt{d}\tau_X(\tau_X + \Upsilon)}{r^2\varepsilon}$  for  $c^2 >2\ln (1.25 / \delta)$ . Then, the Gaussian Mechanism that adds noise to  $\{C_p^S\}_{p = 1}^P$  is  $(\varepsilon ,\delta)$ -differentially private.
Lemma J.2. Assume  $\max_{(p,j)}\{\| \pmb{x}_{p_j}\| _2,\| \pmb{x}_{p_j}'\| _2\} \leq \tau_X$ $\max_{(i,j)}\| \pmb {z}_i - \pmb {x}_j\|_\infty = \Upsilon$ $\| Z_p^s\|_{sp}\leq \tau_Z\forall s$  and  $\| C^S\|_{2,\infty}\leq \tau_C$  , let  $\{Z_p^s\}_{p = 1}^P$  for  $s = 1,\dots ,S$  be perturbed by noise drawn from  $\mathcal{N}(0,\sigma^2)$  with variance  $(8S\Delta^{2}(g_{Z})\log (e + (\varepsilon /\delta)) / \varepsilon^{2})$  where  $\Delta (g_Z) = \frac{2\sqrt{d}\tau_C\tau_X\eta_k}{r^2}\left\{1 + (\tau_X + \tau_Z)\frac{(\tau_X + \Upsilon)}{r^2}\right\}$ . Then, the Gaussian Mechanism that adds noise to  $\{\pmb {Z}_p^s\}_{p = 1}^P$  for  $s = 1,\dots ,S$  is  $(\varepsilon ,\delta)$ -differentially private.
By Lemma J.2, the mechanism that adds Gaussian noise to  $Z_{p}^{s}$  for  $s = 1, \dots, S$  with variance  $(8S\Delta^{2}(g_{Z})\log(e + (\varepsilon_{Z} / \delta_{Z})) / \varepsilon_{Z}^{2})$  satisfies  $(\varepsilon_{Z}, \delta_{Z})$ -differential privacy under  $S$ -fold adaptive composition for any  $\varepsilon_{Z} > 0$  and  $\delta_{Z} \in (0,1]$ . By Lemma J.1, the Gaussian Mechanism that injects noise to  $C_{p}^{S}$  with parameter  $\sigma \geq 2c\lambda^{-1}\sqrt{d}\tau_{X}(\tau_{X} + \Upsilon) / (r^{2}\varepsilon_{C})$  is  $(\varepsilon_{C}, \delta_{C})$ -differentially private. Therefore, by Theorem 3.16 of [Dwork et al., 2014], the proposed algorithm that adds Gaussian noise to  $Z_{p}^{s}$  for  $s = 1, \dots, S$  and  $C_{p}^{S}$  is  $(\varepsilon_{C} + \varepsilon_{Z}, \delta_{C} + \delta_{Z})$ -differentially private. This finished the proof.
K Proof for Lemma J.1
Proof. In our FedSC, for each column of  $C$ , we have
$$
\boldsymbol {c} = g _ {C} (\boldsymbol {x}) = \mathbf {G K} (\boldsymbol {Z}, \boldsymbol {x}) = \boldsymbol {G} \left[ \begin{array}{c} \exp \left(- \frac {\| \boldsymbol {z} _ {1} - \boldsymbol {x} \| _ {2} ^ {2}}{2 r ^ {2}}\right) \\ \vdots \\ \exp \left(- \frac {\| \boldsymbol {z} _ {d} - \boldsymbol {x} \| _ {2} ^ {2}}{2 r ^ {2}}\right) \end{array} \right], \tag {111}
$$
where  $\pmb {G} = (\mathcal{K}(\pmb {Z},\pmb {Z}) + \lambda \pmb{I}_d)^{-1}$  .We have
$$
\left\| g _ {C} (\boldsymbol {x}) - g _ {C} \left(\boldsymbol {x} ^ {\prime}\right) \right\| _ {2} \leq \left\| \boldsymbol {G} \right\| _ {s p} \| \mathcal {K} (\boldsymbol {Z}, \boldsymbol {x}) - \mathcal {K} (\boldsymbol {Z}, \boldsymbol {x} ^ {\prime}) \| _ {2}, \tag {112}
$$
where  $\| \cdot \|_{sp}$  denotes the spectral norm of matrix. Since  $\exp(z)$  is locally Lipschitz continuous when  $z < 0$ , we have
$$
\begin{array}{l} \left(\exp \left(- \frac {\| \pmb {z} _ {j} - \pmb {x} \| _ {2} ^ {2}}{2 r ^ {2}}\right) - \exp \left(- \frac {\| \pmb {z} _ {j} - \pmb {x} \| _ {2} ^ {2}}{2 r ^ {2}}\right)\right) ^ {2} \\ \leq \left| \frac {1}{2 r ^ {2}} \left(\left\| \boldsymbol {z} _ {j} - \boldsymbol {x} \right\| ^ {2} - \left\| \boldsymbol {z} _ {j} - \boldsymbol {x} ^ {\prime} \right\| _ {2} ^ {2}\right) \right| ^ {2} \tag {113} \\ = \left| \frac {1}{2 r ^ {2}} \left(\left\| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \right\| _ {2} ^ {2} + 2 \left\langle \boldsymbol {z} _ {j} - \boldsymbol {x} ^ {\prime}, \boldsymbol {x} ^ {\prime} - \boldsymbol {x} ^ {\prime} \right\rangle\right) \right| ^ {2} \\ \leq \left| \frac {1}{2 r ^ {2}} \left(\| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \| _ {2} ^ {2} + 2 \| \boldsymbol {z} _ {j} - \boldsymbol {x} ^ {\prime} \| _ {2} \| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \| _ {2}\right) \right| ^ {2}. \\ \end{array}
$$
Let  $\Upsilon = \max_{\boldsymbol{z}_j, \boldsymbol{x}'} \| \boldsymbol{z}_j - \boldsymbol{x}' \|_2$  and  $\| \boldsymbol{x} - \boldsymbol{x}' \|_2 \leq 2\tau_X$ . Then the  $\ell_2$ -sensitivity of  $g$  is