# A Supernet-Only Framework for Federated Learning in Computationally Heterogeneous Scenarios
## 3 Method
### Reinterpret Approach  
In this section, we introduce our motivations and our approach to reinterpreting a residual sequential network into a supernet of a NAS task. Consider arbitrary residual sequential network \(\mathcal{N}\) with parameters \(\mathcal{P}\). Each residual unit \(\mathcal{N}_{i}\) of \(\mathcal{N}\) can be decomposed into two parts as follows: a nonlinear transformation \(\mathcal{N}_{i}^{\text{Raw}}\) and an identity mapping \(\mathcal{I}\), as follows:  
\[\mathcal{N}_{i}=\mathcal{N}_{i}^{\text{Raw}}+\mathcal{I}. \tag{3}\]  
Let \(\mathcal{N}_{l}^{\text{pre}}\) represent the preceding net of the \(l\)-th layer which is the combination of the first \(l-1\) layers, and adopt \(\mathcal{N}_{1}^{\text{Pre}}=\mathcal{N}_{0}^{\text{Pre}}=\mathcal{N}_{0}^{ \text{Raw}}=\mathcal{I}\). The full residual network \(\mathcal{N}\), composed of \(L\) layers, can then be recursively reformulated as follows:  
\[ \begin{split}\mathcal{N}&=(\mathcal{N}_{L}^{\text{ Raw}}+\mathcal{I})\circ\mathcal{N}_{l}^{\text{Pre}}\\ &=\mathcal{N}_{L}^{\text{Raw}}\circ\mathcal{N}_{L}^{\text{Pre}}+ \mathcal{N}_{L}^{\text{Pre}}\\ &=\mathcal{N}_{L}^{\text{Raw}}\circ\mathcal{N}_{L}^{\text{Pre}}+ (\mathcal{N}_{L-1}^{\text{Raw}}+\mathcal{I})\circ\mathcal{N}_{L-1}^{\text{ Pre}}\\ &=\mathcal{N}_{L}^{\text{Raw}}\circ\mathcal{N}_{L}^{\text{Pre}}+ \mathcal{N}_{L-1}^{\text{Raw}}\circ\mathcal{N}_{L-1}^{\text{Pre}}+\mathcal{N }_{L-1}^{\text{Pre}}\\ &\dots\\ &=\mathcal{N}_{L}^{\text{Raw}}\circ\mathcal{N}_{L}^{\text{Pre}}+ \mathcal{N}_{L-1}^{\text{Raw}}\circ\mathcal{N}_{L-1}^{\text{Pre}}+\dots+ \mathcal{N}_{2}^{\text{Raw}}\circ\mathcal{N}_{2}^{\text{Pre}}+\mathcal{N}_{2} ^{\text{Pre}}\\ &=\mathcal{N}_{L}^{\text{Raw}}\circ\mathcal{N}_{L}^{\text{Pre}}+ \mathcal{N}_{L-1}^{\text{Raw}}\circ\mathcal{N}_{L-1}^{\text{Pre}}+\dots+ \mathcal{N}_{2}^{\text{Raw}}\circ\mathcal{N}_{2}^{\text{Pre}}+\mathcal{N}_{1} \\ &=\mathcal{N}_{L}^{\text{Raw}}\circ\mathcal{N}_{L}^{\text{Pre}}+ \mathcal{N}_{L-1}^{\text{Raw}}\circ\mathcal{N}_{L-1}^{\text{Pre}}+\dots+ \mathcal{N}_{2}^{\text{Raw}}\circ\mathcal{N}_{2}^{\text{Pre}}+\mathcal{N}_{1} ^{\text{Raw}}+\mathcal{I}\\ &=\sum_{l=0}^{L}\mathcal{N}_{l}^{\text{Raw}}\circ\mathcal{N}_{l} ^{\text{Pre}}\end{split}  \tag{4}\]  
Let \(x_{l}\) denote the input to model layer \(l\). The result of the nonlinear transformation is denoted by \(y_{l}^{\text{raw}}=\mathcal{N}_{l}^{\text{Raw}}(x_{l})\), and the output of layer \(l\) is denoted as \(y_{l}=\mathcal{N}_{l}(x_{l})=x_{l}+y_{l}^{\text{raw}}\). The input to this layer is both the output of the previous layer, \(x_{l}=y_{l-1}\), and also the output of the preceding net of this layer \(x_{l}=\mathcal{N}_{l}^{\text{pre}}(x)\). Then, the output of each term \(\mathcal{N}_{l}^{\text{Raw}}\circ\mathcal{N}_{l}^{\text{pre}}\) in Equation (4) can be represented as follows:  
\[\mathcal{N}_{l}^{\text{Raw}}(\mathcal{N}_{l}^{\text{pre}}(x))=y_{l}^{\text{ Raw}} \tag{5}\]  
Substituting Equation (5) into Equation (4), the output of any residual sequential network can be represented as the sum of the outputs of the nonlinear transformations of each residual unit as follows:  
\[\mathcal{N}(x)=\sum_{l=0}^{L}y_{l}^{\text{Raw}} \tag{6}\]  
Unlike the conventional view that networks with residual connections are sequential architectures at coarser scales, Equation (6) instead forms a parallel architecture. In discussions surrounding residual networks, some studies [32; 33] suggest that low-quality nonlinear transformations can degrade, thereby minimizing their impact on the learning process. Inspired by this, we realized that there exists a hidden weight that determines the contribution ratio of each nonlinear transformation. Defining this parameter as weights, the network output can be expressed as a weighted sum of contributions from all layers \(\mathcal{N}(x)=\sum_{l=0}^{L}\mathcal{W}_{l}\cdot y_{l}^{\text{Raw}}\), where \(\mathcal{W}_{l},(l=0,1,\cdots,L)\) are the weights satisfying non-negativity and normalization constraints.