# Federated Distillation Methodology for Label-Based Group Structures
## 3 Materials and Methods
### Clustered Federated Distillation  
Our objective is to identify the group of each client and train a specialized model for each group using both \(D_{p}\) and \(D_{k}\). We assume no prior knowledge of the groups, including the number of clusters \(L\). All we can use for clustering is the \(f(x^{p})\) prediction from each client model on the public data. If we use these predictions directly for clustering, each of the \(N_{p}\) public data points will produce a \(C\)-dimensional prediction vector, where \(C\) is the number of classes. This results in the data of the shape \(N_{p}\times C\) to be clustered. On the other hand, if we use the number of labels predicted by each client model, the data to be clustered would only have a shape of \(C\). Since \(N_{p}\) is usually much larger than \(C\), this reduction in dimensionality allows for more efficient clustering. So, the server aggregates the logits predicted by each client for \(D_{p}\) and then computes a count vector for each label predicted by each client \(k\). This count vector is normalized, as described in Equations (2) and (3). \[Count_{k}=[Count_{k}^{c}]_{c=1}^{C},\quad\text{where}\quad Count_{k}^{c}=\sum_{ i=1}^{N_{p}}I\left(\text{argmax}\left(f_{k}(x_{i}^{p})\right)=c\right) \tag{2}\]  
In Equation (2), \(Count_{k}\) represents a vector in which each element \(Count_{k}^{c}\) denotes the number of instances in \(D_{p}\) classified into class \(c\) by client \(k\)'s model \(f_{k}\). The function \(I\) serves as an indicator that returns \(1\) if the condition is true and \(0\) otherwise. \[NormCount_{k}=\left[\frac{Count_{k}^{c}-\text{min}(Count_{k})}{\text{max}(Count _{k})-\text{min}(Count_{k})}\right]_{c=1}^{C} \tag{3}\]  
We employed agglomerative clustering to identify the client groups. It is a hierarchical clustering method that starts with each data point as a separate group and iteratively merges the closest groups together. The distance_threshold serves as a key parameter for setting the maximum distance for merging groups. Equation (3) normalizes the count vectors to a [0, 1] range to ensure the consistent application of the distance_threshold. Algorithm 1 outlines our Clustered Federated Distillation Learning method. Each client trains its model on a private dataset and predicts classes on a public dataset \(D_{p}\). These predictions are sent to a centralized server that clusters the clients based on them. The server calculates the average logit \(\tilde{f}_{l}(x^{p})\) for each cluster and sends it back to the corresponding client. The clients then distill their models using the KL divergence loss as Equation (1), effectively addressing non-IID data distribution and enhancing the overall model performance. ``` Input: Public dataset \(D_{p}\), private dataset \(D_{k}\), model of client \(k\): \(f_{k}\), \(k=1,\dots,m\), \(L\) group and \(l_{c}\) clients at group \(l\), Output: Trained model \(f_{k}\) Train: Each client trains \(f_{k}\) on \(D_{k}\). Predict: Each client predicts class \(f_{k}(x_{i}^{p})\) on \(D_{p}\), and transmits the result to a central server. Cluster: The server clusters using each client's prediction. Using Equations (2) and (3) Aggregate: The server averages the logit for each cluster. \(\tilde{f}_{l}(x^{p})=\frac{1}{l_{c}}\sum_{k\in\text{Group }l}f_{k}(x^{p})\) Distribute: Each client receives own group's logit \(\tilde{f}_{l}(x^{p})\). Distill: Each client model learns by distilling knowledge from \(\tilde{f}_{l}(x^{p})\). Using Equation (1). ## 4 Results

### Setting

To experiment with different group structures, we varied the number of classes per group, ranging from 2 to 5, and the number of groups ranging from 2, 4, 6, 8, and 10. We used the MNIST [21] dataset, which is one of the most used datasets for FL research [7; 12; 22], unless otherwise noted. It has a small size of 28\(\times\)28, making it a popular way to simply prove the concept of an algorithm. It has a total of 10 classes. This implied that a single class often belonged to multiple groups.