SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The batch size is set as 64 when  $K = 10$  and 32 for  $K = 100$ . On the server side, we select 5 clients for aggregation per round when  $K = 10$ , and 10 clients per round when  $K = 100$ . This corresponds to a sampling rate of  $50\%$  for 10 clients ( $K = 10$ ) and  $10\%$  for 100 clients ( $K = 100$ ). The noise level in our experiments is  $\mathcal{N}(0, 0.15)$ . Besides, all experiments are performed on Python 3.8, 36 core 3.00GHz Intel Core i9 CPU, and NVIDIA RTX A6000 GPUs.
Table 7: The values of all parameters in this paper
Symbolic representation	Value	Description
Federated Learning Relevant
α	0.1/0.05	heterogeneity degree
Td	15	communication round of feature distillation
Tr	1,000	communication round of classifier training
Ed	1	local epochs of feature distillation
E/Er	1/5	local epochs of classifier training
σs2	0.15	DP noise level, added to xs
|Ct|/|Cr|	5/10	#selected clients every communication round
K	10/100	#clients of federated system
Training Process Relevant
ηk	0.01/0.001/0.0001	learning rate
B	32/64	batch size
M	0.9	momentum
wd	0.0001	weight decay for regularization
“#” represents the number of, and “/” denotes we try these values for these symbolic representations.
B.2 Pseudo-code and Explanation
In this section, we give the pseudo-code of FL algorithm with FedFed. Algorithm 2 gives the procedure of FedAvg and FedProx with FedFed. The difference between FedAvg and FedProx is the local objective function, and we highlight it in the following equation:
$$
\min  _ {\phi_ {k}} \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim P \left(X _ {k}, X _ {k}\right)} \ell \left(f (\mathbf {x}; \phi_ {k}), \mathbf {y}\right) + \mathbb {E} _ {\left(\mathbf {x} _ {p}, \mathbf {y}\right) \sim P \left(\mathbf {h} \left(X _ {k}\right), Y _ {k}\right)} \ell \left(f \left(\mathbf {x} _ {p}; \phi_ {k}\right), \mathbf {y}\right) + \frac {\mu}{2} \| \phi - \phi^ {r} \| ^ {2}, \tag {11}
$$
SCAFFOLD attempts to estimate the client shift degree to correct client update direction. Particularly, it works through server control variate  $c$  and client control variate  $c_k$  of client k. SCAFFOLD with FedFed is summarized in Algorithm 3. And FedNova is proposed to resolve the objective inconsistency for defying data heterogeneity. The outline of our plugin deploys on FedNova in the Algorithm 4. In accordance with our process of how to apply FedFed, every participant of the FL system can deploy our framework easily without any modification of objective function or aggregation scheme in their own methods.
C Full Analysis on Information Theory Perspective
C.1 Features Partition
From an information theory perspective, we present formal definitions, namely Definition 3.1 and Definition 3.2, to define performance-robust features and performance-sensitive features. Specifically, Definition 3.1 captures the desirable attributes when we partition the features, which are abstracted to be a variable in general. A valid partition should maintain all information of the original variable  $X$  losslessly. That is, neither extra information is introduced nor key information is lost. So we can get (i) and (ii) in Definition 3.1. After determining the measure space, we partition the  $X$  to be  $X = X_{1} + X_{2}$ . The mutual information of  $X_{1}$  and  $X_{2}$  is none. Here,  $X_{1}$  and  $X_{2}$  can be symmetric. Or saying, there is no overlap between  $X_{1}$  and  $X_{2}$ , as described by (iii) in Definition 3.1.
Algorithm 2 FedAvg/FedProx with FedFed
Server Input: initial global model  $\phi^0$  , communication round  $T_{r}$    
Client  $k$  's Input: local epochs  $E_{r}$  , local private datasets  $\mathcal{D}^k$  , learning rate  $\eta_{k}$  Initialization: server distributes the initial model  $\phi^0$  to all clients, Generate globally shared dataset  $\mathcal{D}^s$  .  $\leftarrow$  Detail in Algorithm 1   
Distribute  $\mathcal{D}^s$  to all clients and  $\mathcal{D}_t^k = \mathcal{D}^k\cup \mathcal{D}^s$    
Server Executes: for each round  $r = 1,2,\dots ,T_r$  do server samples a subset of clients  $C_r\subseteq \{1,\ldots ,K\}$  server communicates  $\phi^r$  to selected clients  $k\in C_r$  for each client  $k\in C_r$  in parallel do  $\phi_k^{r + 1}\gets$  Client_Training  $(k,\phi^{r})$  end for  $\phi^{r + 1}\gets AGG(\phi_k^{r + 1})$    
end for   
Client_Training  $(k,\phi^{r})$  ..  $\phi^r$  initialize local model  $\phi_k^r$    
for each local epoch  $e$  with  $e = 1,2,\dots ,E_r$  do  $\phi_k^{r + 1}\gets$  SGD update with  $\mathcal{D}_t^k$  using Eq. (10), FedProx uses Eq. (11)   
end for   
Return  $\phi_k^{r + 1}$  to server
Built upon the Definition 3.1, we present the definition of performance-sensitive features and performance-robust features, as in Definition 3.2. Intuitively, performance-sensitive features contain all label information, while performance-robust features contains all information about the data except for the label information. That is, the features to be partitioned are either performance-sensitive features or performance-robust features.
C.2 FedFed on Information Bottleneck Perspective
In this section, we analyse FedFed under an information bottleneck(IB) aspect. In conventional, information bottleneck [21] can be formalized as:
$$
\min  _ {Z} I (X; Y | Z), \text {s . t .} I (X; Z) \leq I _ {c}. \tag {12}
$$
where to restrict the complexity of encoding  $Z$ . Information entropy is denoted as  $H(\cdot)$  and  $I(\cdot; \cdot)$  is the mutual information of two variables. In FedFed, our competitive mechanism stems from IB getting the following definition:
$$
\begin{array}{l} \min  _ {Z} I (X; Y | Z), \text {s . t .} I (X - Z; X | Z) \geq I _ {c}, \\ \Leftrightarrow \min  _ {Z} H (Y | Z), \text {s . t .} H (Z) \leq H _ {c}, \tag {13} \\ \Leftrightarrow \min  _ {\mathbf {z}} - \mathbb {E} \log p (\mathbf {y} | \mathbf {z}), s. t. \| \mathbf {z} \| _ {2} ^ {2} \leq \rho . \\ \end{array}
$$
According to our framework,  $\mathbf{z}$  can be defined as:  $\mathbf{z} = \mathbf{z} - \mathbf{x}_r = \mathbf{x} - q(\mathbf{x};\theta)$ , and then we can the get the optimization view of FedFed:
$$
\min  _ {\theta} - \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim P \left(X _ {k}, Y _ {k}\right)} \log p (\mathbf {y} | \mathbf {x} - q (\mathbf {x}; \theta)), \text {s . t .} \| \mathbf {x} - q (\mathbf {x}; \theta) \| _ {2} ^ {2} \leq \rho . \tag {14}
$$
Ultimately, in an information bottleneck perspective,  $\min_{\theta} - \mathbb{E}_{(\mathbf{x},\mathbf{y})\sim P(X_k,Y_k)}\log p(\mathbf{y}|\mathbf{x} - q(\mathbf{x};\theta))$  intend to contain more information that can be used for generalization.  $||\mathbf{x} - q(\mathbf{x};\theta)||_2^2\leq \rho$  aims at distilling raw data for minimum features which means maximal compression of  $\mathbf{x}_s$ .
Algorithm 3 SCAFFOLD with FedFed
Server Input: initial global model  $\phi^0$ , communication round  $T_{r}$ , server control variate  $c = 0$
Client  $k$ 's Input: local epochs  $E_{r}$ , local private datasets  $\mathcal{D}^k$ , learning rate  $\eta_{k}$ , variate  $c_{k} = 0$ .
Initialization: server distributes the initial model  $\phi^0$  to all clients,
Generate globally shared dataset  $\mathcal{D}^s$ .  $\leftarrow$  Detail in Algorithm 1
Distribute  $\mathcal{D}^s$  to all clients and  $\mathcal{D}_t^k = \mathcal{D}^k\cup \mathcal{D}^s$
Server Executes:
for each round  $r = 1,2,\dots ,T_r$  do
server samples a subset of clients  $C_r \subseteq \{1, \dots, K\}$
server communicates  $\phi^r$  to selected clients  $k\in C_r$
for each client  $k \in C_r$  in parallel do
$\Delta \phi_{k}^{r + 1},\Delta c\gets$  Client_Training  $(k,\phi^r,c)$
$$
\phi^ {r + 1} \leftarrow A G G \left(\Delta \phi_ {k} ^ {r + 1}\right), \quad c \leftarrow c + \frac {\left| C _ {r} \right|}{K} \Delta c
$$
Client_Training  $(k,\phi^r,c)$  ..
$\phi^r$  initialize local model  $\phi_k^r$
for each local epoch  $e$  with  $e = 1,2,\dots ,E_r$  do
$\phi_k^{r + 1}\gets \phi_k^r -\eta_k[\nabla_{\phi_k^r}L_k(\phi_k^r) - c_k + c],$  (SGD update with  $\mathcal{D}_t^k$  use Eq. (10))
$$
c _ {k} ^ {\star} \leftarrow (i) \nabla_ {\phi_ {k} ^ {r}} L _ {k} \left(\phi_ {k} ^ {r}\right), o r (i i) c _ {k} - c + \frac {1}{E \eta_ {k}} \left(\phi_ {k} ^ {r} - \phi_ {k} ^ {r + 1}\right)
$$
$$
\Delta c \leftarrow c _ {k} ^ {\star} - c _ {k}, \quad \Delta \phi_ {k} ^ {r + 1} \leftarrow \phi_ {k} ^ {r + 1} - \phi^ {r}
$$
$$
c _ {k} \leftarrow c _ {k} ^ {\star}
$$
Return  $(\Delta \phi_{k}^{r + 1},\Delta c)$  to server
Proof. To get Eq (13), we firstly have:
$$
\begin{array}{l} I (X; Y | Z) = I (X; Y) - I (X; Y; Z) \\ = H (X) + H (Y) - H (X, Y) - \left[ I (Y; Z) - I (Y; Z | X) \right] \\ = H (X) + H (Y) - H (X, Y) - [ H (Y) + H (Z) - H (Y, Z) ] \\ + \left[ H (Y | X) - H (Y | X, Z) \right] \\ = H (X) + H (Y) - H (X, Y) - H (Y) - H (Z) \\ + H (Y, Z) + H (Y | X) - H (Y | X, Z) \\ = H (X) - H (X, Y) - H (Z) + H (Y, Z) + H (Y | X) - H (Y | X, Z) \\ = H (Y, Z) - H (Z) - H (Y | X, Z) \\ = H (Z) + H (Y | Z) - H (Z) - H (Y | X, Z) \\ = H (Y | Z) - H (Y | X, Z) \\ = H (Y | Z) - H (Y | X). \\ \end{array}
$$
$$
\begin{array}{l} I (X - Z; X | Z) = I (X - Z; X) - I (X - Z; X; Z) \\ = H (X - Z) + H (x) - H (X - Z, X) - [ I (X; Z) - I (X; Z | X - Z) ] \\ = H (X - Z) + H (X) - H (X - Z, X) - [ H (X) + H (Z) - H (X, Z) ] \\ + \left[ H (X | X - Z) - H (X | X - Z, Z) \right] \\ = H (X - Z) + H (X) - H (X - Z, X) - H (X) - H (Z) \\ + H (X, Z) + H (X | X - Z) - H (X | X - Z, Z) \\ = H (X - Z) - H (X - Z, X) - H (Z) + H (X, Z) + H (X | X - Z) \\ = H (X - Z) - H (X) - H (Z) + H (X) + H (X | X - Z) \\ = H (X - Z) - H (Z) + H (X | X - Z) \\ = H (X, X - Z) - H (Z) \\ = H (X) - H (Z). \\ \end{array}
$$
$$
I (X - Z; X | Z) \geq I _ {c} \Leftrightarrow H (X) - H (Z) \geq I _ {c} \Rightarrow H (Z) \leq H (x) - I _ {c} = H _ {c} \Rightarrow H (Z) \leq H _ {c}.
$$
Algorithm 4 FedNova with FedFed
Server Input: initial  $\phi^0$  , communication round  $T_{r}$    
Client  $k$  's Input: Epochs  $E_{r}$  , local private datasets  $\mathcal{D}^k$  , learning rate  $\eta_{k}$  , momentum factor  $\varrho_{k}$  Initialization: server distributes the initial model  $\phi^0$  to all clients, Generate globally shared dataset  $\mathcal{D}^s$  .  $\leftarrow$  Detail in Algorithm 1   
Distribute  $\mathcal{D}^s$  to all clients and  $\mathcal{D}_t^k = \mathcal{D}^k\cup \mathcal{D}^s$    
Server Executes: for each round  $r = 1,2,\dots ,T_r$  do server samples a subset of clients  $C_r\subseteq \{1,\ldots ,K\}$  server communicates  $\phi^r$  to selected clients  $k\in C_r$  for each client  $k\in C_r$  in parallel do  $\Delta \phi_k^{r + 1},a_k\gets$  Client_Training  $(k,\phi^r)$  end for  $\phi^{r + 1}\gets \phi^r -\frac{\sum_{k\in C_r}a_k}{K}\sum_{k\in C_r}\frac{\Delta\phi_k^{r + 1}}{K}$    
end for   
Client_Training  $(k,\phi^r)$  ..  $\phi^r$  initialize local model  $\phi_k^r$    
for each local epoch  $e$  with  $e = 1,2,\dots ,E_r$  do  $\phi_k^{r + 1}\gets \mathrm{SGD~update~with~}\mathcal{D}_t^k\mathrm{~use~Eq.~(10))}$    
end for  $a_{k}\gets [E - \varrho_{k}(1 - \varrho^{E}) / (1 - \varrho_{k})] / (1 - \varrho_{k})$ $\Delta \phi_k^{r + 1}\gets (\phi_k^{r + 1} - \phi_k^r) / (\eta_ka_k)$    
Return  $\Delta \phi_k^{r + 1},a_k$  to server
Then, we get  $\min_Z I(X;Y|Z)$ , s.t.  $I(X - Z;X|Z) \geq I_c \Leftrightarrow \min_Z H(Y|Z)$ , s.t.  $H(Z) \leq H_c$ .
$$
\begin{array}{l} H (Y | Z) = \int_ {\mathbf {z}} p (z) H (Y | Z = \mathbf {z}) d \mathbf {z} = \int_ {\mathbf {z}} p (\mathbf {z}) \int_ {\mathbf {y}} p (\mathbf {y} | \mathbf {z}) \log \frac {1}{p (\mathbf {y} | \mathbf {z})} d \mathbf {y} d \mathbf {z} \\ = - \int_ {\mathbf {z}} \int_ {\mathbf {y}} p (\mathbf {y}, \mathbf {z}) \log p (\mathbf {y} | \mathbf {z}) d \mathbf {y} d \mathbf {z} \\ = - \mathbb {E} \log p (\mathbf {y} | \mathbf {z}). \\ \end{array}
$$
$$
\begin{array}{l} H (Z) \leq \frac {1}{2} \sum_ {i = 1} ^ {d} \ln \operatorname {V a r} \left(\mathbf {z} _ {i}\right) 2 \pi e = \frac {1}{2} \sum_ {i = 1} ^ {d} \ln \operatorname {V a r} \left(\mathbf {z} _ {i}\right) + \frac {1}{2} \sum_ {i = 1} ^ {d} \ln 2 \pi e \tag {16} \\ \leq \frac {1}{2} \sum_ {i = 1} ^ {d} \ln \operatorname {V a r} \left(\mathbf {z} _ {i}\right) + \frac {1}{2} d \ln 2 \pi e = \frac {1}{2} \sum_ {i = 1} ^ {d} \ln \mathbb {E} \left(\mathbf {z} _ {i} ^ {2}\right) + \frac {1}{2} d \ln 2 \pi e \\ = \frac {1}{2} \ln \mathbb {E} \left(\sum_ {i = 1} ^ {d} \mathbf {z} _ {i} ^ {2}\right) + \frac {1}{2} d \ln 2 \pi e \leq H _ {c} \\ \Rightarrow \mathbb {E} | | \mathbf {z} | | _ {2} ^ {2} \leq e ^ {2 H _ {\mathrm {c}} - d \ln 2 \pi e} = \rho , \\ \end{array}
$$
D Full Analysis on Differentially Private Features
Our security analyses contain two steps, saying, differential privacy for each client and the overall analyses for FedFed. Built upon previous works [27, 16], we derive Lemma D.1 below. For FedFed, we attain Lemma D.2 by limiting  $\sigma_r\to \infty$
Lemma D.1. For sharing raw features  $\mathbf{x}$ ,  $(\epsilon, \delta)$ -DP holds if  $\epsilon' = O(\sqrt{R \log(1 / \delta)} (\rho / \sigma_s + (1 - \rho) / \sigma_r))$ .
Lemma D.2. For sharing performance-sensitive features  $\mathbf{x}_s$ ,  $(\epsilon, \delta)$ -DP holds if  $\epsilon = O(\rho \sqrt{R \log(1 / \delta)} / \sigma_s)$ .
D.1 Analysis and Proof for Theorem 3.3
Recall that DP-SGD [27] samples i.i.d. noise from unbiased Gaussian distribution  $\mathcal{N}(\mathbf{0}, S_f^2 \sigma^2)$ , where  $S_f$  is the sensitivity. At the algorithmic level, the noise sampled from  $\mathcal{N}(\mathbf{0}, \sigma^2 C^2 \mathbf{I})$  is added to a batch of gradients handled by clipping value  $C$ .
For aligning analyses with conventional DP guarantees, we adopt similar notations and expressions. Specifically, we denote the noise distribution operated on  $\mathbf{x}_r$  to be  $\mathcal{N}(\mathbf{0},\sigma_r^2\mathbf{I})$  and the noise distribution operated on  $\mathbf{x}_s$  to be  $\mathcal{N}(\mathbf{0},\sigma_s^2\mathbf{I})$ . Notably, FedFed only adds DP noise to  $\mathbf{x}_s$  in practice. Since  $\mathbf{x}_r$  is kept by the corresponding client, an adversary could attain nothing (or be regarded as random data without any entropy). This could be regarded as adding a sufficiently large noise on  $\mathbf{x}_r$ , thus we consider the  $\sigma_r\rightarrow \infty$ . As for  $\mathbf{x}_s$ , we employ the  $\ell_2$ -norm clipping when selecting the noise level  $\sigma_s$ .
Now, let's come back to the setting of sharing all raw data  $\mathbf{x}$ . We still regard  $\mathbf{x}$  to be two parts, i.e.,  $\mathbf{x} = \mathbf{x}_r + \mathbf{x}_s$  for the latter comparison to FedFed. This trick is motivated by cryptographic standard proof with ideal/real paradigm [60]. For both  $\mathbf{x}_r$  and  $\mathbf{x}_s$ , the noise addition follows DP-SGD's idea, except for adding noise to the sampled data for updating gradients, rather than the latter-computed real gradients. We remark that this method works for FL but may be different for centralized training (with identical training set in the whole training process).
Using the property of post-processing [16], the noise added to the data passes to (or projects to) the following gradient updates. For the view of each client, we know that Theorem D.3 holds for  $\sigma$  operated on  $\mathbf{x}$  from [27].
Theorem D.3. There exist constants  $c_1$  and  $c_2$  so that given the sampling probability  $q = L / N$  and the number of steps  $T$ , for any  $\epsilon < c_1 q^2 T$ . The algorithm of DP-SGD is  $(\epsilon, \delta)$ -differentially private for any  $\delta > 0$  if we choose  $\sigma = c_2 \frac{q \sqrt{T \log(1 / \delta)}}{\epsilon}$ .
For strong composition theorem, the choice of  $\sigma$  in Theorem D.3 is  $O(q\sqrt{T\log(1 / \delta) / \epsilon})$ . In DP-SGD, the  $T$  denotes the number of training steps, which means the times of using/sampling private data for DP training. Such meaning represents communication rounds in FL, in which each client receives globally shared data at each round of communication. In the FedFed, we denote the number of communication rounds to be  $R$ . Now, let's handle the  $q$ , where  $q$  represents the ratio of lot size and the inputting dataset size. In DP-SGD, the per-lot data is directly sampled from the database of size  $N$ . In other words, a lot of data is the subset of the input dataset. In the FedFed, the training data fed to each client's model is the same as the database size, i.e.,  $q = N / N = 1$ .
DP-SGD asks for the generality and thus bounds  $\| f(\cdot)\| = \mathbf{e}$ . Yet, FedFed sets  $\sigma \propto \| \mathbf{x}_s\|$ , which does not ask for a general case. We use the notation  $\| \mathbf{x}_s\|_2$  to represent the domain of performance-sensitive features. On the other hand,  $\| \mathbf{x}\|_2$  represents the domain of all features (namely, the "domain of datasets"). We additionally borrow the definition of  $l_2$ -norm and move  $\| \mathbf{x}_s\|$  out in the proof. By Equation 8, the client takes  $\rho \| \mathbf{x}\|$  for selecting  $\sigma$ . Since we aim to make an asymptotic analysis here, we ignore the scalars and constants here. Thus, we attain  $O(\rho \sqrt{R\log(1 / \delta)} /\epsilon)$  for selecting  $\sigma$ . By rearranging variables, we have  $\epsilon = \rho \sqrt{R\log(1 / \delta)} /\sigma$  for a loose analysis.
Before continuing the proof, we derive Lemma D.4 to get the relation between  $\| \mathbf{x}_r\|$  and  $\| \mathbf{x}\|$  for latter usage.
Lemma D.4. Let  $\mathbf{x}_r = \mathbf{x} - \mathbf{x}_s$ . Then, it holds that  $(1 - \rho)\| \mathbf{x}\| \leq \| \mathbf{x}_r\| \leq (1 + \rho)\| \mathbf{x}\|$  if  $\| \mathbf{x}_s\| = \rho \| \mathbf{x}\|$  satisfies
Proof of Lemma D.4:  $\| \mathbf{x} - \mathbf{x}_r\| = \rho \| \mathbf{x}\| \Rightarrow \| \mathbf{x} - \mathbf{x}_r\| ^2 = \rho^2\| \mathbf{x}\| ^2\Rightarrow \| \mathbf{x}\| ^2 +\| \mathbf{x}_r\| ^2 -2\mathbf{x}^\top \mathbf{x}_r =$ $\rho^2\| \mathbf{x}\| ^2\Rightarrow \| \mathbf{x}\| ^2 +\| \mathbf{x}_r\| ^2 -2\| \mathbf{x}\| \| \mathbf{x}_r\| \cos \langle \mathbf{x},\mathbf{x}_r\rangle = \rho^2\| \mathbf{x}\| ^2.$  Since  $\| \mathbf{x}\| >0$  and  $\| \mathbf{x}_r\| >0$  , we get,
$$
\| \mathbf {x} \| / \| \mathbf {x} _ {r} \| + \| \mathbf {x} _ {r} \| / \| \mathbf {x} \| - 2 \cos \langle \mathbf {x}, \mathbf {x} _ {r} \rangle = \rho^ {2} \| \mathbf {x} \| / \| \mathbf {x} _ {r} \|.
$$
Let  $\| \mathbf{x}_r\| = \alpha \| \mathbf{x}\|$ , and our objective is to get the  $\alpha$ . By replacing  $\| \mathbf{x}\| / \| \mathbf{x}_r\|$  and  $\| \mathbf{x}_r\| / \| x\|$  with  $1 / \alpha$  and  $\alpha$  respectively, we get,
$$
1 / \alpha + \alpha - 2 \cos \langle \mathbf {x}, \mathbf {x} _ {r} \rangle = \rho^ {2} / \alpha .
$$
${}^{6}$  Previously "  $\geq$  ",we take the special case "  $=$  " the same as DP-SGD.
Since  $-1\leq \cos \langle \mathbf{x},\mathbf{x}_r\rangle \leq 1$  , we get,
$$
- 2 \leq 1 / \alpha + \alpha - \rho^ {2} / \alpha \leq 2 \Rightarrow - 2 \alpha \leq 1 / + \alpha^ {2} - \rho^ {2} \leq 2 \alpha .
$$
Since  $0 < \rho < 1$ , we take  $1 / +\alpha^2 - \rho^2 \leq 2\alpha$  and get  $(\alpha - 1)^2 \leq \rho^2$ . Then, we get the meaningful answers  $-\rho \leq \alpha - 1 \leq \rho \Rightarrow 1 - \rho \leq \alpha \leq 1 + \rho$ . Therefore, we attain,
$$
(1 - \rho) \| \mathbf {x} \| \leq \alpha \| \mathbf {x} \| \leq (1 + \rho) \| \mathbf {x} \| \Rightarrow (1 - \rho) \| \mathbf {x} \| \leq \| \mathbf {x} _ {r} \| \leq (1 + \rho) \| \mathbf {x} \|.
$$
Following above, we can establish the relationship between  $\| \mathbf{x}\| _2$  and  $\| \mathbf{x}_s\| _2$  as follows:
$$
\| \mathbf {x} _ {s} \| _ {2} \leq \rho \| \mathbf {x} \| _ {2}.
$$
And the relation between noise level  $\sigma_{s}$  and  $\mathbf{x}_s$  is:
$$
\sigma_ {s} \propto \| \mathbf {x} _ {s} \| _ {2} \leq \rho \| \mathbf {x} \| _ {2} = \rho M,
$$
where we re-scale data into  $[0,1]$  and assume the  $\ell_2$ -norm of data is less than  $M > 0$ . To ensure the  $\| \mathbf{x}_e\| _2\leq \rho M$ , we clip the norm for each sample, which is widely used in DP [27].
Now, let's go back to derive Lemma D.1 and Lemma D.2, and continue the proof. For sharing raw data  $\mathbf{x}$ , we employ the same method as aforementioned. Since  $\| \mathbf{x}_s\| = \rho \| \mathbf{x}\|$ , we get  $\epsilon_e' = \rho \sqrt{R\log(1 / \delta)} /\sigma_s$ . By Lemma D.4, we know that  $(1 - \rho)\leq \frac{\|\mathbf{x}_r\|}{\|\mathbf{x}\|}\leq (1 + \rho)$  holds. Thus, we attain,
$$
(1 - \rho) \sqrt {R \log (1 / \delta)} / \sigma_ {r} \leq \epsilon_ {r} ^ {\prime} \leq (1 + \rho) \sqrt {R \log (1 / \delta)} / \sigma_ {r}.
$$
Here, we take  $\epsilon_r' \geq (1 - \rho) \sqrt{R \log(1 / \delta)} / \sigma_r$  since we expect a loose case. By combining  $\epsilon_e'$  and  $\epsilon_r'$ , we attain,
$$
\epsilon^ {\prime} = O \left(\sqrt {R \log (1 / \delta)} \left(\rho / \sigma_ {s} + (1 - \rho) / \sigma_ {r}\right)\right)
$$
for sharing raw  $x$ , as Lemma D.1. Now, let's take FedFed into consideration. The major difference is that  $\mathbf{x}_r$  is kept locally in the whole training process. The root cause lies in that the  $\mathbf{x}_r$  owned by the client is privacy-insensitive to this corresponding client. We regard  $\sigma_r$  operated on  $\mathbf{x}_r$  to be a sufficiently large value. Thus, given  $\rho \ll 1$ , we get,
$$
\lim _ {\sigma_ {r} \to \infty} \epsilon_ {r} = (1 - \rho) \sqrt {R \log (1 / \delta)} / \sigma_ {r} \to 0.
$$
That is, for sharing  $\mathbf{x}_s$  in FedFed, we have  $\epsilon = \epsilon_s = O(\rho /\sigma_s\sqrt{\log(\frac{1}{\delta})})$ , as Lemma D.2.
Let's conversely think Lemma D.1 and Lemma D.2. Now, the proof in the following is very straightforward. If we take identical  $\sigma_s$  operated on  $\mathbf{x}_s$  for FedFed and sharing raw  $x$ . We know that  $\epsilon' > \epsilon$ . Conversely, if we take identical  $\epsilon, \epsilon'$ , we should increase  $1 / \sigma_s$  and thus reduce  $\sigma_s$ , given constant  $\rho \sqrt{R \log(1 / \delta)}$ . Thus, for protecting training data in FL, we can attain Theorem 3.3 to summarize the superiority of FedFed when asking for an identical privacy guarantee. Theorem 3.3 explains the reason for the superior model performance of FedFed, which intrinsically boils down to a relatively small  $\sigma$  compared with sharing raw data  $\mathbf{x}$ .
D.2 Analysis and Proof for Theorem 3.4
Previously, we showcase the view of each client for analyzing privacy. Albeit we do not aim at a new DP theorem, we expect a tighter privacy analysis for FedFed by using the advanced result [28]. Vadhan and Wang [61] prove that when the interactive mechanisms being composed are pure differentially private, their concurrent composition achieves privacy parameters (with respect to pure or approximate differential privacy) that match the (optimal) composition theorem for noninteractive differential privacy. To be specific, we follow the proof logic of Theorem D.7 [28] for analyzing globally shared data from all clients.
Given  $(\epsilon_k,\delta)$ -DP at each client side, we utilize the composition theorem to analyze overall privacy in FedFed. The concept of sensitivity below is originally used for sharing a dataset for achieving  $(\epsilon ,\delta)$ -differential privacy.
Definition D.5 (Sensitivity [28]). The sensitivity of a query function  $\mathcal{F}:\mathbb{D}\to \mathbb{R}$  for any two neighboring datasets  $D,D^{\prime}$  is  $\Delta = \max_{D,D^{\prime}}\| \mathcal{F}(D) - \mathcal{F}(D^{\prime})\|$ , where  $\| \cdot \|$  denotes  $L_{1}$  or  $L_{2}$  norm.
Privacy loss is a random variable that accumulates the random noise added to the algorithm/model, which is utilized in Theorem D.7.
Definition D.6 (Privacy Loss [28]). Let  $\mathcal{M}:\mathbb{D}\to \mathbb{R}$  be a randomized mechanism with input domain  $D$  and range  $R$ . Let  $D$ ,  $D^{\prime}$  be a pair of adjacent dataset and aux be an auxiliary input. For an outcome  $o\in \mathbb{R}$ , the privacy loss at  $o$  is defined by  $\mathcal{L}_{\mathrm{Pri}}^{(o)}\triangleq \log (\operatorname*{Pr}[\mathcal{M}(\mathrm{aux},D) = o] / \operatorname*{Pr}[\mathcal{M}(\mathrm{aux},D^{\prime}) = o])$ .