# FREDY: Federated Resilience Enhanced with Differential Privacy
## 4 Experiments
### Performance Evaluation
\begin{table}
\begin{tabular}{c c} \hline \hline
**Hyper-Parameter** & **Value** \\ \hline Federated rounds (n) & 10 \\ Teacher epochs (\(E_{T}\)) & 10 \\ Student epochs (\(E_{ST}\)) & 20 \\ Learning rate (\(\eta\)) & 0.001 \\ Weight decay (w) & \(1\times 10^{-5}\) \\ Batch size (B) & 32 \\ Optimizer & _Adam_ \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Hyper-parameter configuration. In Figure 6, the student test accuracy is illustrated across various scenarios. Specifically, Figure 6 outlines the student test accuracy as the server conducts 500, 1000, 2000, and 9000 queries using the ensemble of 15 teachers with a noise parameter of \(\epsilon=0.2\). As anticipated, the student accuracy exhibits a decline as the number of queries directed towards the teacher models increases. In the context of 9000 queries, exploring values below 0.1 for \(\epsilon\) becomes inconsequential, as this introduces excessive noise to the student's labels, rendering the training process ineffective. Additionally, Figure 7 shows the accuracy of the student model on both datasets varying the number of teacher models during federated learning and the value \(\epsilon\) of noise per label query. Small values of \(\epsilon\) on the left of the axis correspond to large noise and large \(\epsilon\) values on the right to small noise. The number of teacher models needs to be large in order to compensate for the impact of noise injection on student accuracy. Through our analysis, we ascertain that the introduction of Laplacian noise with \(\epsilon=0.2\) into teacher queries yields significant outcomes. In particular, this approach yields commendable accuracy rates of 75.1% and 93.2% for CIFAR10 and MNIST, respectively, within the student model. Although this represents a slight decrease of \(\sim\)4% and \(\sim\)6% compared to the baselines, it is important to note that this trade-off is balanced by the considerable enhancement in privacy preservation achieved across the sensitive data of all 25 teacher models. Finally, as Figure 6 shows, the student model achieves \(\sim\)25% accuracy with 5 teacher models and \(\sim\)60% accuracy with 15 teacher models while it achieves almost 70% accuracy with 25 clients and noise parameter \(\epsilon=0.1\) as the number of teachers increases (i.e., the clients of the FL system), and thus noise injection has less impact on model accuracy. It is worth mentioning that in Figure 6,b, we report results on CIFAR10 only as it contains RGB images from common objects in real life and thus represents more real-world scenarios. Furthermore, Table 2 shows the attack results of a shadow training membership inference attack on our baseline model and on FREDY's student model that is trained with 25 teacher models and 2000 queries on CIFAR10, varying the value of \(\epsilon\), i.e., the noise injection to each query. Compared to the baseline model, FREDY reduces the performance of the membership inference attack to a random guess (i.e., \(\sim\)25% drop in all metrics) when the value of \(\epsilon\) is equal to 0.2 and thus it can successfully protect and keep private each teacher's sensitive data. We can also observe that when \(\epsilon=0.1\), the membership inference attack performance is slightly lower, i.e., \(\sim\)0.005% lower when compared with \(\epsilon=0.2\); however, the student accuracy degradation on this setting makes this attack performance difference meaningless. Figure 4: Test accuracy of aggregated model. (**a**) Test accuracy of aggregated model for each federated round during FL training of teachers on CIFAR10 dataset. (**b**) Test accuracy of aggregated model for each federated round during FL training of teachers on MNIST dataset. Figure 5: Student test accuracy for 9000 queries, without noise injection. Figure 6: Student test accuracy under various scenarios. (**a**) Test accuracy of the student model per epoch with 15 teacher models and \(\epsilon=0.2\) noise injection per query on CIFAR10 dataset.