# Multi-Level Split Federated Learning for Large-Scale AIoT System Based on Smart Cities
## 3 Proposed Framework
### Split Learning Side  
After grouping IoT devices according to their geographic location, each group first performs a split learning algorithm with the assigned master server. The algorithm can be divided into four main computational parts. The IoT device is responsible for performing two parts of the deep learning network computation, namely the forward propagation and backward propagation of the IoT device network. The master server is responsible for calculating the remaining layers and loss calculations. We define a deep learning network as a function \(f\), which contains the network layers that can be represented as \(\{l_{0},\ l_{1},\ldots,l_{K}\}\). For the input local data (\(local\ data\)), the output of the neural network is  
\[l_{K}(l_{K-1}\ldots(l_{0}(local\ data)))\to f(local\ data). \tag{1}\]  
Let \(Loss(output,\ label)\) represent the last layer to calculate the loss function of the real label and the network output. \(\Delta{l_{i}}^{T}\) represents the backpropagation process at each network layer, so the backpropagation of the entire deep learning network can be expressed as  
\[\Delta{l_{K}}^{T}\Big{(}\Delta{l_{K-1}}^{T}\ldots\Big{(}\Delta{l_{i}}^{T} \Big{)}\Big{)}\rightarrow\Delta{f}^{T}. \tag{2}\]  
Therefore, the deep network structure can be divided into two parts according to the disassembly layer, assuming that the layer \(l_{n}\) is the cutting layer of the neural network. The network layer retained by the IoT device and the master server node is represented as follows  
\[f_{c}\leftarrow\{l_{0},l_{1},\ldots,l_{n}\},\ \ n\in N, \tag{3}\]  
\[f_{m}\leftarrow\{l_{n+1},l_{n+2},\ldots,l_{N}\},\ \ n\in N. \tag{4}\]  
The client sends the activation of the cutting layer generated by the forward propagation of the local network model to the master server for forward propagation of the rest of the network layers. The master server is responsible for calculating the loss of labels and outputs and propagating backward to update the master server's network layer weights. The client smashed data from the forward propagation of the local network model to the main server for forward propagation of the rest of the network Assuming a set of \(D\) IoT devices in \(t\) time period, the model update of the master server part of the network can be expressed as follows [29]:  
\[W_{t+1}^{M}:=W_{t}^{M}-\eta\frac{S_{D}}{s}\sum_{m=1}^{D}\Delta{L_{m}}\Big{(}W_ {t}^{M};\alpha_{t}^{M}\Big{)}, \tag{5}\]  
where \(\eta\) is the learning rate to train the deep learning model, and \(s\) is the size of the total samples. \(\alpha_{t}^{M}\) represents the activation in master server, and \(\Delta{L_{D}}\big{(}W_{t}^{M};\alpha_{t}^{M}\big{)}\) denotes the gradient of backpropagation in the master server's network layer. Algorithm 1 shows the exact algorithm flow of the split learning part. The backpropagation gradient received by the client from the master server is sent to the edge server for aggregation, so Algorithm 1 is only one part of the multi-level SFL, and the entire algorithm will be presented later. \begin{table}
\begin{tabular}{c c c} \hline
**Method** & **Comms. per IoT Devices** & **Total Comms.** \\ \hline FL & \(2M\) & \(2 DM\) \\ \hline SL & \((^{2s}/D)q\) & \(2 sq\) \\ \hline Multi-level SFL & \((^{2s}/D)q+2\kappa M\) & \(2 sq+2\kappa M(D+E)\) \\ \hline \end{tabular}
\end{table} 
Table 2: Communication overhead of different distributed learning methods. ```
0:\(s\) is the size of total samples; \(t\) is time period; \(a_{t}^{d}\) is the smashed data of IoT device at \(t\); \(\Delta L_{d}\) is the gradient of the loss for IoT device \(d\); \(Y_{d}\) is the true label from IoT device \(d\).