## 3 FedCKA
### Modifications to FedAvg
Figure 4 shows the distribution of data across clients on the CIFAR-10 dataset with different \(\alpha\). Figure 4A shows \(\alpha:=5.0\), Figure 4B shows \(\alpha:=1.0\), and Figure 4C shows \(\alpha:=0.1\). All experiments were conducted using the PyTorch [25] library on a single GTX Titan V and four Intel Xeon Gold 5115 processors. ### Accuracy

FedCKA adds a hyperparameter \(\mu\) to control the strength of \(\ell_{cka}\). We tune \(\mu\) from [3; 5; 10] and report the best results. MOON and FedProx also have a \(\mu\) term. We also tune the hyperparameter \(\mu\) with these methods. For MOON, we tune \(\mu\) from [0.1, 1, 5, 10], and for FedProx, we tune \(\mu\) from [0.001, 0.01, 0.1, 1], as used in each work. In addition, for MOON, we use \(\tau=0.5\) as reported in their work. Table 1 shows the performance across CIFAR-10, CIFAR-100, and Tiny ImageNet with \(\alpha=5.0\). For FedProx, MOON, and FedCKA, we report performances with the best \(\mu\). For FedCKA, the best \(\mu\) is 3, 10, and 3 for CIFAR-10, CIFAR-100, and Tiny ImageNet, respectively. For MOON, the best \(\mu\) is 10, 5, and 0.1. For FedProx, the best \(\mu\) is 0.001, 0.1, and 0.1. Table 2 shows the performance across increasing heterogeneity on the CIFAR-10 dataset with \(\alpha\in[5.0,1.0,0.1]\). For FedCKA, the best \(\mu\) is 5, 3, and 3 for each \(\alpha\in[5.0,1.0,0.1]\), respectively. For MOON, the best \(\mu\) is 0.1, 10, and 10. For FedProx, the best \(\mu\) is 0.001, 0.1, and 0.001. Figure 4: Distribution of the CIFAR-10 dataset across 10 clients according to the Dirichlet distribution. The x-axis shows the index of the client, and the y-axis shows the index of the class (label). (**A–C**) shows the data distribution of \(\alpha:=5.0\), \(\alpha:=1.0\), and \(\alpha:=0.1\), respectively. As parameter \(\alpha\) approaches 0, the heterogeneity of class distribution increases. We observe that FedCKA consistently outperforms previous methods across different datasets and across different \(\alpha\). FedCKA improves performances in heterogeneous settings due to regularizing layers that are naturally similar and not layers that are naturally dissimilar. It is also interesting to see that FedCKA performs better by a larger margin when \(\alpha\) is substantial. This is likely because the global model can more effectively regularize updates as it is less biased when data distribution approaches IID settings. However, we also observe that other studies consistently improve performance, albeit by a smaller margin than FedCKA. FedProx and SCAFFOLD improve performances, likely due to their inclusion of naturally similar layers in regularization. The performance gain is lower, as they also include naturally dissimilar layers in regularization. MOON generally improves performance compared to FedProx and SCAFFOLD likely due to their use of a contrastive loss. That is, MOON shows that neural networks should be trained to be more similar to the global model _than past local model_, rather than only being blindly similar to the global model. By only regularizing naturally similar layers using a contrastive loss based on CKA, FedCKA outperforms all methods. Note that across most methods and settings, there are discrepancies to the accuracy reported by MOON [19]. In particular, MOON reports higher accuracy across all methods, although the model's architecture are similar if not equivalent. We verify that this discrepancy is caused by the data augmentation used in experiments with MOON. We disclude augmentation as it would be unfair to generalize results across different non-IID settings if augmentations were to be used. The reported non-IIDness would decrease, as clients could create a more balanced distribution. ### Communication Rounds and Local Epochs

We study the effects of regularization on the performance improvement per communication rounds. Results are shown in Figure 5. As expected with any regularization methods, we find that the accuracy for FedCKA is lower for the 40 communication rounds.