## 3 FedCKA
### Modifications to FedAvg
However, we also find that after 40 communication epochs, FedCKA improves performances due to effective regularization. FedCKA decreases the bias that would have otherwise limited performance by penalizing weight updates that are not in agreement with the global model. We also explore the effects of the number of local epochs on overall performance. We find that the performance of both FedAvg and FedCKA increases slightly when the number of local epochs increased. However, when further increasing the number of local epochs, we find that accuracy decreases, suggesting overfitting. The small increase in performance does not warrant additional local epochs. Clients are limited in their computational budget. Thus, computation cannot be used sparingly. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Method** & **CIFAR-10** & **CIFAR-100** & **Tiny ImageNet** \\ \hline FedAvg & 64.37\% & 37.41\% & 19.49\% \\ FedProx & 64.58\% & 37.81\% & 20.93\% \\ SCAFFOLD & 64.33\% & 39.16\% & 21.18\% \\ MOON & 65.25\% & 38.37\% & 21.29\% \\
**FedCKA** & **67.86\%** & **40.07\%** & **21.46\%** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Accuracy across datasets (\(\alpha=5.0\)). \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Method** & \(\alpha=5.0\) & \(\alpha=1.0\) & \(\alpha=0.1\) \\ \hline FedAvg & 64.37\% & 62.49\% & 50.43\% \\ FedProx & 64.58\% & 62.51\% & 51.07\% \\ SCAFFOLD & 64.33\% & 63.31\% & 40.53\% \\ MOON & 65.25\% & 62.60\% & 51.63\% \\
**FedCKA** & **67.86\%** & **66.19\%** & **52.35\%** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Accuracy across \(\alpha\in[5.0,1.0,0.1]\) (CIFAR-10). ### Regularizing Only Important Layers

We study the effects of regularizing different number of layers. Using the CIFAR-10 dataset with \(\alpha=5.0\), we change the number of layers to regularize through \(\ell_{\zeta ka}\). Formally, we change \(M\) in Equation (2) by scaling \(M\ \in\ [1,2,3,4,5,6,7]\), and report the accuracy in Figure 6. Accuracy is the highest when only the first two layers are regularized. Note the dotted line representing the upper bound for Federated Learning. When the same model is trained on a centralized server with the entire CIFAR-10 dataset, accuracy is 70%. FedCKA with regularization on the first two naturally similar layers nearly reaches this upper bound. This verifies our hypothesis that only naturally similar, but not naturally dissimilar layers, should be regularized. By regularizing only one layer, a naturally similar layer (the second layer) would be excluded, thus decreasing performance. By regularizing three layers, a non-similar layer (the third layer) with the highest variance of non-similarity (see Figure 6) would be regularized, thus decreasing performance. The performance increase when 4-7 layers are regularized may seem anomalous, yet is valid when considering that both the first two naturally similar layers are included, and the weight of regularization of the third layer decreases with a higher M.

### Using the Best Similarity Metric

We study the effects of regularizing the first two naturally similar layers with different similarity metrics. Using the CIFAR-10 dataset with \(\alpha=5.0\), we change the similarity metric to regularize through \(\ell_{\zeta ka}\). Formally, we change \(CKA(a_{1},a_{2})\) in Equation (2) to three other similarity metrics: first, the kernel CKA, introduced in [PERSON] et al. [10] (\(CKA_{k}(a_{1},a_{2})\)); second, the squared Frobenius norm (\(\|a_{1}-a_{2}\|_{F}^{2}\)); third, the vectorized

Figure 5: Effects of communication round and local epochs. Figure 6: Accuracy with respect to the number of layers regularized and variance in similarity measures between clients on CIFAR-10 and \(\alpha=5.0\). cosine similarity (\(\|vec(a_{1})\|\|vec(a_{2})\|\cos\theta\)). We compare the results with these different metrics as well as the baseline, FedAvg. The results are shown in Table 3.

We observe that performance is the highest with CKA due to the increased accuracy of measuring similarity.