# A Supernet-Only Framework for Federated Learning in Computationally Heterogeneous Scenarios
## 4 Experiments
### Evaluation of Supernet-Only Framework  
We use ConvNeXt Small [34] as the backbone model and design our supernet based on it. ConvNeXt Small consists of four stages; the first, second, and fourth stages have three layers each, while the third stage contains 27 layers. All stages are modified according to the proposed method. However, due to the large size of Stage 3, sub-network pruning is only applied to this stage, while other stages remain unpruned during the FL phase.  
To ensure fairness in comparison, a unified pretraining process is conducted for all federated methods in this part of the experiment.  
The pretraining lasts for 5 epochs with a learning rate of 1 \(\times\) 10\({}^{-5}\) as warm-up, followed by 20 epochs with a learning rate of 1 \(\times\) 10\({}^{-4}\). The batch size is set to 96, and the Nadam optimizer is used with default settings.  
During the pretraining phase, all supernet parameters are fully optimized. In the federated training phase, the first and the last convolutional layer are replaced before federated transfer learning. After training, the performance of all client models is evaluated on the validation set to verify the effectiveness of the method. Although ConvNeXt Small is used as the example model, our method can be generalized to any deep neural networks (DNNs) [42] with skip connections like ResNet [43], ConvNeXt [34], or Transformer [44].  
We categorize existing NAS and Fed-NAS strategies under resource-constrained conditions as follows:  
* Max Trainable: The largest sub-network that all clients can train is used as the federated model, equivalent to traditional FL.
* Max Runnable: The smallest sub-network that all clients can use for inference is selected as the federated model. Clients unable to train only perform inference, while capable clients participate in training.
* No Federated: Each client uses the strongest sub-network it can support without FL.
* Grouped: Clients are divided into three different scale categories, with each group using the largest sub-network it can support for federated learning.
* Peaches: The cells of sub-network are separated into private cells and public ones. Only public cells are shared among the federal model.  
In addition, we conduct comparative experiments under the distribution strategy of AdaptiveFL to validate the effectiveness of the Supernet-Only framework against dynamic network strategies for FL. Max Trainable and Max Runnable represent cases of lightweight global FL, No Federated represents device-aware NAS for personalized search, and Grouped represents clustering methods for Fed-NAS problems. These control groups aim to show the trade-off between using stronger models and sharing parameters across a larger range without our novel method.  
Our method has high adaptability, so we test it under the following two standard federated learning structures: ring-based federated learning and the FedAvg framework.  
We assume that the computational capabilities of clients are uniformly distributed between the smallest and largest sizes of residual units in the backbone network. The weakest clients can support sub-networks with all variable-sized units set to 1, while the strongest clients can handle the full backbone as it appears before transformation in Section 3.1. For simplicity, we also assume the training cost is twice the inference cost.