# FREDY: Federated Resilience Enhanced with Differential Privacy
## 4 Experiments
### Performance Evaluation  
For training a machine learning model within a Federated Learning configuration among clients, the datasets must be partitioned to ensure distinct data for each Teacher within the FL System. In our approach, both the CIFAR10 and MNIST training datasets are partitioned into \(T\) separate subsets. A unique teacher model is trained on each of these sets, where \(T\) represents the total number of teachers. As for the CIFAR10 and MNIST test datasets, they serve as the public unlabeled data for student training. Within this context,  
Figure 3: Teacher and student model architectures. a segment of labels is derived from the most prevalent predictions made by the teachers concerning these data. Similar to the PATE methodology, we allocate 90% of both test datasets (equivalent to 9000 images) for the student's training procedure, reserving the remaining 10% (1000 images) for evaluating the entire training process. We train the teacher models for \(n=10\) federated rounds and \(E_{T}=10\) local epochs for each round using the Flower [51] framework. The student model is trained for \(E_{ST}=20\). All the models are trained with the Adam Optimizer [52] with learning rate \(\eta=0.001\), weight decay \(w=1\times 10^{-5}\) and batch size \(B=32\). A summary of FREDY's hyper-parameter setup is shown in Table 1.  
The baseline models are subjected to supervised learning, devoid of any privacy-preserving techniques, and undergo 20 epochs of training using identical train and test datasets as the student. Consequently, the student model attains accuracy rates of 79% for CIFAR10 and 99.2% for MNIST. These achievements serve as the performance benchmarks for the student models, establishing an upper limit for comparison in subsequent experiments. As previously mentioned, the initial phase of FREDY entails the server engaging in federated learning to train the teachers, while the student awaits the completion of their training cycles. Our investigation involves three ensembles, comprising 5, 15, and 25 teachers. Consequently, the dataset is partitioned into distinct segments of equal size, yielding 10,000, 3333, and 2000 images for the CIFAR10 dataset, and 12,000, 4000, and 2400 images for the MNIST dataset. Each teacher undergoes 10 federated rounds, encompassing 10 epochs per round. The evolving test accuracy of the aggregated model is showcased in Figure 4, tracing its performance across each federated round during the training of teachers for the three ensembles. By the concluding round, the aggregated model achieves accuracy levels of 91%, 86.1%, and 82% on CIFAR10, and 99.4%, 99%, and 98.9% on MNIST, corresponding to ensembles of 5, 15, and 25 clients, respectively. In the second phase of FREDY, the FL server proceeds to query the teacher model predictions on the student training data, followed by a noise-infused aggregation of their votes (predictions) to label a specific portion of these data for student training. As previously mentioned, we introduce Laplacian noise with an inversely scaled \(\epsilon\), spanning values of 1, 0.5, 0.2, 0.1, and 0.01. As \(\epsilon\) diminishes, a greater degree of random noise is introduced to each query. With access to a dataset containing 9000 samples, a subset of 500, 1000, 2000, or 9000 samples is labeled using the noisy aggregation mechanism for each \(\epsilon\) value. Figure 5 presents a comparison between the accuracy of the baseline model and the student model, both trained without noise injection, and involving ensembles of 5, 15, and 25 teachers. Notably, it is observed that each model exhibits a nearly identical performance, with an accuracy of ~79% for CIFAR10 and ~99% for the MNIST dataset. This result aligns with our expectations given that individual teachers achieve average test accuracies of ~87%, ~83%, and ~79% for CIFAR10, and ~98.8%, ~98.2%, and ~97.3% for MNIST, corresponding to ensembles of 5, 15, and 25 teachers, respectively. Consequently, these accurate predictions contribute to the student's effective handling of public, unlabeled data.