# A Supernet-Only Framework for Federated Learning in Computationally Heterogeneous Scenarios
## 4 Experiments
### Evaluation of Supernet-Only Framework
#### 4.1.2 Evaluation of the Supernet-Only Framework Under Federated Averaging (FedAvg)
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **CIFAR-100** & **CIFAR-10** & **CINIC-10** \\ \hline Supernet-Only & \(\mathbf{67.67\pm 1.43}\) & \(91.07\pm 0.61\) & \(83.22\pm 0.62\) \\ Max Trainable & \(58.69\pm 0.43\) & \(88.87\pm 0.37\) & \(81.12\pm 0.27\) \\ Max Runnable & \(62.31\pm 0.45\) & \(\mathbf{91.43\pm 0.37}\) & \(\mathbf{83.30\pm 0.30}\) \\ No Federated & \(54.35\pm 0.90\) & \(88.37\pm 1.14\) & \(79.89\pm 0.60\) \\ Grouped & \(59.22\pm 1.89\) & \(90.09\pm 0.76\) & \(81.93\pm 0.96\) \\ Peaches & \(43.37\pm 1.37\) & \(72.28\pm 2.58\) & \(72.37\pm 0.94\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Accuracy statistic under FedAvg: Every value illustrates the mean validation accuracies and their corresponding standard deviations across different clients (the best and second-best results are in **bold** and underlined, respectively). \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **CIFAR-100** & **CIFAR-10** & **CINIC-10** \\ \hline Supernet-Only & \(\mathbf{77.84\pm 0.91}\) & \(\mathbf{94.70\pm 0.62}\) & \(\mathbf{87.12\pm 0.88}\) \\ Max Trainable & \(70.88\pm 0.89\) & \(92.72\pm 0.59\) & \(84.13\pm 0.54\) \\ Max Runnable & \(72.42\pm 0.86\) & \(93.41\pm 0.52\) & \(85.75\pm 0.67\) \\ No Federated & \(13.78\pm 8.08\) & \(69.49\pm 12.00\) & \(63.44\pm 6.64\) \\ Grouped & \(71.45\pm 1.00\) & \(93.38\pm 0.53\) & \(83.95\pm 1.01\) \\ Peaches & \(57.43\pm 10.18\) & \(85.79\pm 15.58\) & \(76.89\pm 3.31\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Accuracy statistic under FedAvg for the large-scale federal model: Every value illustrates the mean validation accuracies and their corresponding standard deviations across 128 clients. This experiment was trained for 50 epochs using an H20-96G GPU, whereas other experiments under the ConvNeXt configuration were conducted for 20 epochs on a 4090-24G GPU, due to the increased memory requirements from large-scale client simulation and the slower convergence rate in this setting (the best and second-best results are in **bold** and underlined, respectively). results. This non-parametric test is suitable for comparing matched samples without assuming a specific distribution. In the experiment, the test accuracy obtained under Supernet-Only was paired with the corresponding value from each baseline for the same client. Since only 79 out of 128 clients completed training in the Max Runnable setting due to resource limitations, this comparison was limited to the 79 matched clients. In contrast, the comparison involving Max Trainable used results from all 128 clients. For each dataset and baseline, we calculated the Wilcoxon test statistic \(W\), which is the sum of the ranks of the signed differences in performance. We also reported the standardized score \(Z\), which indicates the magnitude and direction of the difference, the raw \(p\)-value, and the Bonferroni-corrected \(p\)-value for controlling the family-wise error rate across multiple tests. The raw \(p\)-value reflects the probability of observing the given result under the null hypothesis in a single test, while the corrected \(p\)-value adjusts for multiple comparisons to reduce the likelihood of Type I errors. In addition, we computed the effect size \(r=Z/\sqrt{N}\), where \(N\) represents the number of clients included in the comparison. The effect size describes the practical relevance of the result; values of \(r\) greater than 0.3 typically indicate a large effect. These indicators together provide a comprehensive assessment of both the statistical and practical significance of the observed differences, as shown in Table 4.  
The results indicate that Supernet-Only consistently and significantly outperforms Max Runnable on all of the following three datasets: CIFAR-10, CIFAR-100, and CINIC-10. The Bonferroni-corrected \(p\)-values in these comparisons are all below 0.001, with a value of \(2.17\times 10^{-13}\) on CIFAR-10, which reflects strong statistical evidence. The corresponding effect sizes are large, with \(r\) values ranging from 0.78 to 0.86.