# Federated Distillation Methodology for Label-Based Group Structures
## 3 Materials and Methods
### Clustered Federated Distillation
Figure 3: (**a**) Represents the average accuracy per client group when FedDF is applied, while (**b**) represents the average accuracy per client group when our method is applied. In both cases, the proportion for each client group is 6:2:2.

improves the performance by more than 2\(\times\) on average over the baseline, demonstrating that it works well on a more complex dataset. Existence of Minor Classes: In real-world scenarios, it is often difficult to make clear distinctions between different groups. To address this, we introduce the concept of a \"minor class\", a less prevalent class within each group, to blur traditional group boundaries. Figure 6 shows that as the proportion of these minor classes increases, the performance gap between our proposed method and traditional approaches narrows. This experiment utilizes the MNIST dataset. Our experiments demonstrate several strengths of the proposed clustered federated distillation algorithm when clear label-based group structures exist. It achieves high clustering accuracy in delineating distinct groups based on their predicted distributions. The client model accuracy also significantly outperforms traditional federated distillation

Figure 5: Graph representing accuracy in CIFAR with respect to the class per group. Figure 6: Graph shows how performance varies depending on the “minor class” ratio. Each data point represents the average performance across dataset sizes of 100, 300, 500, and 1000. The MNIST dataset was used. methods, especially for minority groups. The algorithm performs robustly even with limited client and public data. However, the results also reveal some limitations of the current approach. The performance gains diminish as the group boundaries become more ambiguous, with a higher proportion of overlapping \"minor\" classes. The experiments have so far only focused on image classification tasks with a limited number of labels. Tests on more complex datasets and tasks are needed to fully evaluate the applicability of the method. In addition, more extensive tuning is needed to determine optimal hyperparameter settings, such as distance thresholds. Addressing these limitations is a worthwhile avenue for improving the clustered federated distillation framework proposed in this work. ## 5 Discussion

In this study, we address the scenario of different data distributions between different client groups in federated distillation. We introduce a methodology that uses hierarchical clustering to categorize clients according to the number of labels predicted by each model for public data. This approach overcomes the limitations of traditional federated distillation techniques that assume a uniform data distribution when a label-based group structure exists. Our method can be used when different groups (e.g., demographic groups) have significantly different data distributions to ensure that all groups receive equally good results. The experiments show that the model correctly classifies groups with different labels. The accuracy of the model exceeds that of traditional methods when there is a clear cluster structure based on labels. In particular, the accuracy of a small number of groups, which is problematic in traditional federated distillation, is significantly improved. This may pave the way for fair FL. Furthermore, our method does not require knowledge of the number of clusters, making it applicable in a wider range of environments. However, as the group structure becomes less clear, the performance gap between our method and existing algorithms narrows. We will continue to improve our method to perform better in ambiguous group structures. It would be an interesting research topic to combine our method with different data types, such as text, more complex images, or time series data. Our method could also be combined with data-free distillation where no public data exists. Our algorithm will also be very effective in the presence of malicious clients that send false predictions to the server.