# A Supernet-Only Framework for Federated Learning in Computationally Heterogeneous Scenarios
## 3 Method
### Training Pipeline  
We recommend a two-phase pretraining process prior to the FL training phase. Then, we introduce a federated sub-network generation method tailored to our specific supernet. The proposed training pipeline is shown in Figure 3.  
Our supernet is constructed by reinterpreting residual sequential networks, derived either from manually designed architectures or those obtained via NAS. As a result, various training methods originally developed for manually crafted networks, such as using [PERSON] [36] as the optimizer, can be directly applied to our reinterpreted supernet. In standard training procedures, we optimize block weights \(\mathcal{W}\) and feature weights \(\mathcal{P}\) jointly, without distinguishing between them. Although our supernet retains a residual sequential structure, we decouple the training of structural weights and formulate it as a separate GLM-based learning task for structural optimization. Therefore, most training strategies designed for gradient-based NAS,including the bi-level optimization method proposed in DARTS, are also applicable to our approach. However, standard training methods do not explicitly optimize parameterized network structures. Traditional NAS approaches typically rely on a fixed set of candidate components, such as predefined connections or blocks, which do not align well with the sub-networks produced by our truncation-based method. To address this, we propose a new training strategy tailored for sequential supernets. In each iteration, a truncation length is randomly sampled for each stage to determine the exit point of the sub-network. This mechanism enables random sampling of sub-architectures and simulates the federated learning scenario where different clients operate with different model depths. We refer to this strategy as Single-Path Random Quit (SPRQ), as it randomly selects an exit point in each stage. From a theoretical perspective, we establish that SPRQ does not alter the optimization target in expectation. We introduce binary mask variables \(m_{t,(s,l)}\) to indicate whether parameter \(\mathcal{P}^{\prime}_{s,l}=\{\mathcal{P}_{s,l},\mathcal{V}_{s,l}\}\) is active under a randomly truncated path, and we rescale the corresponding gradients as \(g_{t,(s,l)}=\frac{m_{t,(s,l)}}{p_{s,l}}\cdot g^{\text{full}}_{(s,l)}\). This ensures that the gradient estimator remains unbiased, that is,  
\[\mathbb{E}[g_{t,(s,l)}]=g^{\text{full}}_{(s,l)}. \tag{22}\]  
Thus, the stochastic sampling process introduces no bias in gradient estimation. The per-iteration parameter update of SPRQ is expressed as follows:  
\[\mathcal{P}^{\prime}_{t+1}=\mathcal{P}^{\prime}_{t}-\text{diag}(\hat{\eta}_{t })\
abla_{\mathcal{P}^{\prime}}\mathcal{L}, \tag{23}\]  
where \(\hat{\eta}_{t,(s,l)}=\eta_{t}\cdot\frac{m_{t,(s,l)}}{p_{s,l}}\) denotes the effective learning rate used to update each parameter in response to the loss \(\mathcal{L}\). Since \(\mathbb{E}[\hat{\eta}_{t}]=\eta_{t}\cdot 1\), we have  
Figure 3: Training pipeline overview: The process consists of the following three main stages: pretraining, federated training, and client-level validation. In the pretraining stage (**left**), the supernet is first warmed up using the DARTS method, which alternates between architecture and weight updates. It is then further trained using the SPRQ strategy, which applies random layer skipping to simulate client diversity and improve generalization. To accommodate the input/output differences between pretraining and federated tasks, the Stem and Head modules are replaced before federated training, and each client receives a subset of the supernet for local training on its own data. Finally, in the validation stage (**right**), the trained models are evaluated individually on each client’s local validation set to assess performance in heterogeneous settings.