# Multi-Level Split Federated Learning for Large-Scale AIoT System Based on Smart Cities
## 4 Experiment
### Effect of Different Clients on Performance  
In this section, we will analyze the influence of edge client data on model training accuracy. For multi-level SFL, we increase the number of nodes on each tier as well as the number of clients for aggregation of model weights, as shown in Table 5. Considering that the master server running the split model needs to run the split model for each client node, we assume that each client group in different regions is equipped with a master server node, so during the model training and optimization process, we do not consider the transfer time between the master service node and the client node.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline
**Dataset** & **Architecture** & **Centralized Learning** & \multicolumn{2}{c}{**Multi-Level FL**} & \multicolumn{2}{c}{**Multi-Level SFL**} \\ \hline  & & **Train** & **Test** & **Train** & **Test** & **Train** & **Test** \\ \hline HAM10000 & ResNet18 & 74.4\% & 79.6\% & 76.9\% & 77.3\% & 78.6\% & 79.4\% \\ Fashion MNIST & LeNet & 88.7\% & 90.2\% & 86.1\% & 87.6\% & 87.9\% & 88.9\% \\ \hline \end{tabular}
\end{table}
Table 4: Train and test result in different distributed setting.  
Figure 4: Testing convergence of LeNet on Fashion MNIST under various learning.  
This section will analyze the impact of the number of LeNet users on Fashion MNIST. Figure 5 shows how the test accuracy and convergence rate vary with the number of epochs when a multi-level SFL is trained on a different number of clients (5, 10, 20, 50 clients). We can see that as the number of clients increases, the convergence speed of multi-level SFL will slow down, but the convergence speed is not obvious, and it is always better than the convergence speed of centralized learning. Moreover, for multi-level SFL, presumably, as the number of clients increases, the model accuracy decreases. For example, when the number of clients reaches 20 and 50, the accuracy of the test is significantly lower than that of the concentrated learning. However, the accuracy of the model trained on 50 clients is slightly higher than the accuracy of the model trained on 20 clients because the edge nodes of the middle layer also increase as the layers of the client are interlayer, so the number of middle-layer nodes of the multi-level SFL helps the SFL reduce the loss of accuracy.  
Figure 6 shows the time cost required for LeNet to train the Fashion MNIST dataset with a different number of clients. As can be seen from the figure, the training time required for centralized learning is always better than the time cost of training the model under multi-level SFL with different clients. This is because the experiment simulates the MQTT protocol used by the AIoT system to communicate, so the communication time and the time spent waiting for all the clients to train will layer the time cost of the multi-level SFL. The training time of the model decreases first and then increases with the increase in clients. This is because as the number of multi-level SFL clients increases, the amount of data locally decreases accordingly, reaching the minimum time overhead with 20 clients. However, when the number of clients reaches 50, the time overhead for model training starts to rise because the time for cloud servers and nodes in the middle level to publish topics and receive requests under the MQTT protocol starts to increase.  
Figure 5: Effect of the number of clients on testing accuracy for LeNet on Fashion MNIST.  
\begin{table}
\begin{tabular}{l l l l l} \hline The number of clients & 5 & 10 & 20 & 50 \\ The number of edge nodes in edge level & 2 & 2 & 2 & 4 \\ The number of fog nodes in fog level & 1 & 1 & 2 & 2 \\ The number of cloud servers & 1 & 1 & 1 & 1 \\ The number of master servers & 2 & 2 & 2 & 4 \\ \hline \end{tabular}
\end{table}
Table 5: Number of clients and multi-level nodes for training.