# A Review and Experimental Evaluation on Split Learning
## 3 Experimental Evaluation
### Experimental Results and Analyses
#### 3.2.4 Communication Cost of SL Paradigms  
We first analyze the communication complexities of various paradigms theoretically, as shown in Table 8. Then, we compare three paradigms for their communication costs: PSL, SplitFed, and AsyncSL, which are representative paradigms with intermediate data aggregation, model aggregation, and reduced communication, respectively. The communication cost is computed by the size of the data (in MB) that are transferred between the clients and the server. The \(loss_{threshold}\)[86] we set in AsyncSL is 0.1.  
We split the networks at different layers to compare the impact of the choice of the cut layer on communication cost. In Table 9, \"Small\" indicates that the network in clients consists of a convolution layer; and \"Large\" indicates that the network consists of two convolution layers. In Table 10, \"Small\" indicates that the model in clients contains a BN  
Figure 10: Performance of FedAvg with different as on the MNIST and CIFAR-10 datasets. In the first two, we set the number of clients to 3; in the latter two, we set the number of clients to 5. The x-axis represents the number of epochs [108].  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Paradigm** & **Communication Cost per Epoch** \\ \hline Vanilla & \(2\cdot S_{cut}\cdot|\mathcal{D}_{k}|+S_{\theta}\) \\ PSL & \(2\cdot S_{cut}\cdot|\mathcal{D}_{k}|\) \\ SGR & \(2\cdot S_{cut}\cdot|\mathcal{D}_{k}|\) \\ SplitFed & \(2\cdot S_{cut}\cdot|\mathcal{D}_{k}|+2\cdot S_{\theta}\) \\ U-shaped & \(2\cdot(S_{cut}+S_{cut})\cdot|\mathcal{D}_{k}|\) \\ FedAvg & \(2\cdot S_{\theta}\) \\ \hline \hline \end{tabular}
\end{table}
Table 8: Communication complexities of different paradigms, where \(S_{cut}\) is the size of the cut layer, and \(S_{\theta}\) is the size of the model \(\theta\).  
layer, a ReLU layer, and a pooling layer; \"Large\" indicates that the model in clients contains all layers before the last pool layer.  
The observations from these results are as follows.  
* The dataset size has an impact on the communication cost. For example, as shown in Table 9, the communication cost of all three paradigms in one epoch decreases significantly when only half of the dataset is used and is approximately equal to half.
* Training in different SL paradigms results in different communication costs. For example, since the clients in SplitFed need to transmit the local model parameters to the server, SplitFed sends and receives more data than PSL as shown in Table 9. Because the size of the models we use is much smaller than the size of the smashed data and gradients, its effect can be neglected. However, when the model is more complex or the number of clients is much larger, the effect of the model aggregation on the communication cost cannot be ignored [11].
* The choice of the cut layer may have an impact on the communication cost. As shown in Table 9, when training on the MNIST dataset, the communication cost decreases significantly when clients hold two convolution layers instead of one. This is because the choice of the cut layer affects the size of the smashed data and gradients. However, the cost does not decrease but even increases when training on the CIFAR-10 dataset, as shown in Table 10. Thus, choosing the appropriate layer to split the network is effective in reducing communication costs.