# Federated Distillation Methodology for Label-Based Group Structures
## 3 Materials and Methods
### Clustered Federated Distillation
For the neural network architecture, we employed a simple CNN with two convolutional layers, using ReLU as the activation function for the hidden layers and Softmax for the last layer's activation function. The learning rate was set at \(\text{lr}=1\times 10^{-4}\) and the batch size used was 128. Unless otherwise stated, five clients were uniformly assigned to each group, with each client having 50 data points per class. We have assumed that there are 400 unlabeled public datasets per class. We conducted experiments with 25 local training epochs for the client and 40 distillation epochs to learn from the aggregated logit. In Section 4.2, we compared the clustering performance of our algorithm with that of an existing FL using clustering. In Section 4.3, we compare the performance of each client after the entire training process with the existing federated distillation algorithm. ### Clustering Experiment

Baseline: We compared the clustering performance with CFL [7], one of the methods that uses weight variation for clustering. It uses the cosine similarity of weight updates between clients as a clustering criterion. Metric: We measured the clustering performance using the adjusted rand index (ARI) [23], which quantifies the similarity between true and predicted cluster assignments based on true cluster identity. The silhouette score [24] serves as another metric for evaluating the degree of separation between clusters. The silhouette score was used to assess how well the criterion used for clustering represented the group structure in terms of the logit and similarity of weight updates. For both metrics, higher values indicate better-defined clusters and the score ranges from \(-1\) to \(1\). Hyperparameter: Table 2 displays the average ARI for various distance thresholds, a hyperparameter in agglomerative clustering. The distance threshold determines how far away from the aggregate clustering values are judged to belong in the same group. We employed two and five for the class per group and four for the number of groups. These results average over each distance threshold value. For future experiments, we choose a distance threshold of two. This is because the range of distance thresholds from 1.5 to 2.5 consistently yields high ARI values above 0.95, indicating optimal clustering. Silhouette Score: Table 3 illustrates the silhouette scores of the baseline for each clustering method under different group structures, indicating how well the data are clustered. The count vector achieved silhouette scores above 0.5 in all cases, while the model update similarity scores were consistently close to zero, indicating little to no group structure. For both variables, fewer groups led to clearer group structures. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & & & & & & \multicolumn{6}{c}{**Distance Thresholds**} \\ \cline{2-10}  & 0.25 & 0.5 & 1.0 & 1.5 & 2.0 & 2.5 & 3.0 & 3.5 & 4.0 \\ \hline ARI & 0.31 & 0.67 & 0.93 & 0.98 & 1 & 1 & 0.78 & 0.75 & 0.43 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Average adjusted rand index (ARI) for different distance thresholds in agglomerative clustering. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & \multicolumn{8}{c}{**Class per Group**} \\ \cline{2-10}
**Group** & \multicolumn{2}{c}{**2**} & \multicolumn{2}{c}{**3**} & \multicolumn{2}{c}{**4**} & \multicolumn{2}{c}{**5**} \\ \cline{2-10}  & **CFL** & **Ours** & **CFL** & **Ours** & **CFL** & **Ours** & **CFL** & **Ours** \\ \hline
2 & \(-0.03\) & \(1.00\) & \(0.01\) & \(1.00\) & \(-0.08\) & \(1.00\) & \(0.13\) & \(1.00\) \\
4 & \(0.08\) & \(1.00\) & \(-0.01\) & \(1.00\) & \(0.06\) & \(0.90\) & \(-0.03\) & \(1.00\) \\
6 & \(0.03\) & \(0.96\) & \(-0.03\) & \(1.00\) & \(-0.01\) & \(0.96\) & \(0.03\) & \(1.00\) \\
8 & \(-0.03\) & \(1.00\) & \(0.04\) & \(1.00\) & \(-0.01\) & \(0.93\) & \(-0.01\) & \(1.00\) \\
10 & \(0.01\) & \(0.91\) & \(0.01\) & \(0.93\) & \(-0.00\) & \(0.97\) & \(-0.01\) & \(1.00\) \\ \hline \hline \end{tabular}
\end{table}