# A Supernet-Only Framework for Federated Learning in Computationally Heterogeneous Scenarios
## 3 Method
### Training Pipeline
\[\mathbb{E}[\mathcal{P}^{\prime}_{t+1}]=\mathcal{P}^{\prime}_{t}-\mathbb{E}[\text{ diag}(\eta_{t})\
abla_{\mathcal{P}^{\prime}}\mathcal{L}]=\mathcal{P}^{\prime}_{t}-\eta_{t} \cdot\
abla_{\mathcal{P}^{\prime}}\mathcal{L}, \tag{24}\]  
which, in expectation, matches the behavior of SGD as follows:  
\[\mathcal{P}^{\prime}_{t+1}=\mathcal{P}^{\prime}_{t}-\eta_{t}\cdot\
abla_{ \mathcal{P}^{\prime}}\mathcal{L}. \tag{25}\]  
Under the standard Robbins-Monro conditions \(\sum_{t=0}^{\infty}\eta_{t}=\infty\) and \(\sum_{t=0}^{\infty}\eta_{t}^{2}<\infty\), and with the masks \(\{m_{t,(s,t)}\}\) being independent of the mini-batch noise, the Robbins-Monro theorem [37] implies that \(\mathcal{P}^{\prime}_{t}\) almost surely converges to a stationary point of \(\mathcal{L}\). In fact, Equation (23) matches the SGD with per-parameter learning rates. The convergence of such issues has already been extensively addressed [38; 39; 40; 41]. Therefore, SPRQ maintains the convergence characteristics of unbiased stochastic gradient methods while offering practical scalability. Although this method can train excellent pretraining parameters, suitable for device-heterogeneous federated learning, the initial training phase may face more difficulties in early convergence due to the dual randomness of parameters and structures. Therefore, we suggest replacing SPRQ with specialized NAS optimization methods during the warm-up stage. In the federated learning stage, based on the characteristics of our network architecture, the sub-networks provided to different clients only differ in truncation lengths, while the model parameters, including structural and weight parameters, are completely consistent. This allows us to directly adopt most common federated learning methods without modifying the parameter aggregation approach, such as ring-based federated learning or FedAvg architectures based on parameter transmission. Even if the model sizes of different clients have quantitative differences, the above methods still apply. In practical applications, we need to replace the detection Head and Stem parts of the reinterpreted models to adapt to the distribution differences and category number differences between pretraining datasets and federated datasets. After completing this adjustment, we only need to select the maximum network that each client can affordor send the complete supernet to clients and let them decide the length of each stage. In this way, each client can use the strongest model they can support while helping improve the overall model performance. When aggregating client model parameters, based on the early exit characteristics of our network, there is no need to distinguish between different sizes; simply synchronizing the iterated parameters or accumulated gradients in sub-networks to the global model is sufficient. In centralized federated topologies, communication efficiency can be maintained by aggregating only the subset of parameters relevant to each local model during each round of training, resulting in a communication cost comparable to that of standard FL approaches. In contrast, decentralized federated topologies inherently require more extensive parameter exchange, often involving the entire supernet. Nevertheless, owing to the compact design of our supernet relative to conventional supernet architectures, the associated communication overhead is significantly reduced.