# A Supernet-Only Framework for Federated Learning in Computationally Heterogeneous Scenarios
## 4 Experiments
### Evaluation of Supernet-Only Framework
#### 4.1.3 Evaluation of the Supernet-Only Framework Under FedAvg with Small Backbone  
To validate the performance differences between our method and the use of dynamic networks, we implemented the Supernet-Only framework following the latest AdaptiveFL approach. In the reports on AdaptiveFL[11], two smaller networks are utilized, VGG without residual connections and ResNet18 with residual connections. Dynamic neural networks are developed based on these models to meet the requirements of AdaptiveFL. Considering that the VGG network represents an early architecture lacking residual connections, modifications can only be applied to ResNet18 to satisfy the criteria for network mutation as defined in Section 3.1. To enhance the flexibility of sub-network generation, we decompose the original residual blocks in ResNet18 such that each individual Conv-BN-ReLU layer forms a separate residual unit, resulting in 16 residual units in total. Based on the architecture weights of each stage in the Supernet, we apply softmax normalization and rank the four candidate operations within each stage. We then generate a series of sub-networks by progressively reducing the number of retained operations per stage from four to one, following the ranked priorities. We adopt the federated training setting from AdaptiveFL while performing our experiments based on the client distribution described in Section 4.1. In our setting, the average client size is significantly smaller, and the size difference among clients is substantially greater compared to the report of AdaptiveFL. After training through the federated phase, the accuracy achieved by different methods across various datasets is summarized in Table 5. Table 6 provides a detailed comparison between the largest-sized network and the strongest network from AdaptiveFL and the Supernet-Only framework without pretraining. In AdaptiveFL, networks of different sizes are separately created, which leads to convergence issues if they are pretrained. In contrast, our sub-networks are generated by a supernet, enabling our method to benefit from pretraining. Even without the unique advantage of pretraining, our method still demonstrates superior performance, benefiting from module selection and optimization guided by structure weights \(\mathcal{W}\). In AdaptiveFL, as the difference in model size is significant, the performance degradation of the full-sized network compared to the best-performing pruned network exceeds 10%. In comparison, our method effectively minimizes the discrepancy between the optimal method and the largest network size, maintaining a stable performance. Once the pretraining is activated, the sub-network with the largest size always outperforms the smaller ones within the Supernet-Only framework. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **CIFAR-100** & **CIFAR-10** & **CINIC-10** \\ \hline Supernet-Only (pretrained) & \(\mathbf{57.76\pm 7.11}\) & \(\mathbf{87.78\pm 5.76}\) & \(\mathbf{79.22\pm 6.14}\) \\ Supernet-Only (not pretrained) & \(49.56\pm 12.31\) & \(77.55\pm 18.16\) & \(73.89\pm 7.28\) \\ AdaptiveFL & \(31.26\pm 19.39\) & \(67.77\pm 25.36\) & \(57.92\pm 21.68\) \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Accuracy statistic under the AdaptiveFL and Supernet-Only frameworks with or without pretraining. Every value illustrates the mean validation accuracies and their corresponding standard deviations across 13 model sizes and 100 clients under federated setting of AdaptiveFL (the best and second-best results are in **bold** and underlined, respectively). Overall, we observe that as the generalization difficulty of the training setting increases, the advantage of our method becomes more pronounced. Additionally, clustering federated devices into multiple federated subgroups does not appear to yield superior results compared to lightweight model selection. However, it is important to note that our backbone model, ConvNeXt, is inherently strong.