# Federated Distillation Methodology for Label-Based Group Structures
## 3 Materials and Methods
### Clustered Federated Distillation

Table 4: Comparison of ARI scores under various group structures for our method and clustered federated learning (CFL). \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & \multicolumn{8}{c}{**Class per Group**} \\ \cline{2-10}
**Group** & \multicolumn{2}{c}{**2**} & \multicolumn{2}{c}{**3**} & \multicolumn{2}{c}{**4**} & \multicolumn{2}{c}{**5**} \\ \cline{2-10}  & **CFL** & **Ours** & **CFL** & **Ours** & **CFL** & **Ours** & **CFL** & **Ours** \\ \hline
2 & \(0.06\) & \(0.82\) & \(0.03\) & \(0.85\) & \(0.02\) & \(0.81\) & \(0.03\) & \(0.85\) \\
4 & \(0.03\) & \(0.88\) & \(0.02\) & \(0.8



### Performance Evaluation

Baseline: In this section, we compare the performance of our method with three baselines. FedDF (Federated ensemble Distillation for robust model Fusion) [17] has a similar distillation process to our method, except that it assigns the same logit to all clients. On the other hand, DS-FL (Distillation-Based Semi-Supervised Federated Learning) [25] uses entropy reduction averaging for model output aggregation. It is designed to deliberately reduce the entropy of the global logit before distributing it to mobile devices. FedAVG (Federated Averaging) [22] is a fundamental federated learning algorithm that averages model parameters across clients to form a global model, in contrast to the distillation approaches of FedDF and DS-FL. Balanced Group Structure: Table 6 shows that our algorithm consistently outperforms FedDF when the number of clients in each group is equal. When each group contained fewer classes, there were fewer overlapping classes. This resulted in a more distinct group structure. Consequently, our method performed 15% better than the FedDF when there were five classes per group. When the group structure is the clearest, with only two classes per group, our method performed 75% better than the FedDF. Unbalanced Group Structure: Table 7 and Figure 3 show the performance when the number of clients varies between groups. In the case of global distillation, the accuracy of groups with fewer clients tends to decrease significantly and sometimes approaches zero. Realistically, in this situation, a small number of groups will have to use their own data without applying FL. However, they will not be able to take advantage of the performance gains from FL at all. By contrast, our method ensures that clients in each cluster perform equally well, leading to a significant increase in accuracy for minority groups. This means that all clients can share in the profits of the FL. CIFAR dataset: We have performed all of our experiments so far using the MNIST dataset. This time, we will use the more complex CIFAR-10 [26] dataset to see how well our method works on more challenging tasks. It is one of the most widely used datasets and consists of 60,000 color images of size 32 \(\times\) 32 with 10 labels. We run our experiments with three groups, each with between two and five classes. Figure 5 shows that our algorithm \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Group Ratio**} & \multicolumn{2}{c}{**Group Acc**} & \multicolumn{2}{c}{**Total Acc**} \\ \cline{2-5}  & **FedDF** & **Ours** & **FedDF** & **Ours** \\ \hline
70, 30 & 97, 0 & 96, 92 & 68 & 95 \\
80, 20 & 97, 0 & 97, 86 & 77 & 95 \\
50, 30, 20 & 96, 1, 0 & 97, 89, 97 & 49 & 94 \\
60, 20, 20 & 96, 0, 0 & 96, 91, 93 & 58 & 94 \\
40, 30, 20, 10 & 96, 7, 4, 64 & 96, 89, 94, 97 & 48 & 94 \\
50, 20, 20, 10 & 96, 0, 0, 66 & 96, 93, 91, 96 & 54 & 94 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Performance metrics across groups with different ratios. The ‘Group Ratio’ column shows the percentage of clients in each group. ‘Group Acc’ represents the average accuracy achieved by each group at the end of the training process, while ‘Total Acc’ represents the average accuracy across all clients. The order of the groups in ‘Group Acc’ and ‘Group Ratio’ is the same. All values are expressed as percentages. Figure 4: Performance of each algorithm as the amount of public dataset changes.