# A Review and Experimental Evaluation on Split Learning
## 3 Experimental Evaluation
### Experimental Results and Analyses
#### 3.2.2 Comparison in the Non-IID Setting  
In practice, the non-IID setting is more frequently considered for SL. To simulate non-IID settings, we employed label-based splitting for the MNIST and CIFAR-10 datasets. For the Adult dataset, we used attribute-based splitting based on relationships. Following prior work, we first sampled \(p_{i,k}\) from Dirichlet distribution \(\text{Dir}(\alpha)\) and then assigned \(p_{i,k}\) proportion of the samples of class/attribute \(i\) to client \(k\), where we set \(\alpha\) from 0.05 to 0.3 to adjust the level of heterogeneity of the data. The smaller \(\alpha\), the larger the heterogeneity of the data. When \(\alpha=0\), samples from different clients have no overlap in the label/attribute space. For example, the MNIST dataset with 10 labels is divided into five clients by \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c} \hline \hline
**Paradigm** & \multicolumn{2}{c}{**Vanilla**} & \multicolumn{2}{c}{**U-Shaped**} & \multicolumn{2}{c}{**PSL**} & \multicolumn{2}{c}{**SplitFed**} & \multicolumn{2}{c}{**SGLR**} \\ \hline \(\alpha\) & **0** & **0.1** & \(\infty\) & **0** & **0.1** & \(\infty\) & **0** & **0.1** & \(\infty\) & **0** & **0.1** & \(\infty\) & **0** & **0.1** & \(\infty\) \\ \hline \multicolumn{12}{c}{**MNIST**} \\ (Acc = **98.43\%**) & 18.26\% & 72.88\% & 97.92\% & 34.86\% & 67.64\% & 97.13\% & 32.97\% & 70.19\% & 97.28\% & 79.32\% & 97.68\% & 98.82\% & 33.12\% & 86.03\% & 97.34\% \\ \hline \multicolumn{12}{c}{**CIFAR-10**} \\ (Acc = **91.14\%**) & 34.24\% & 35.34\% & 90.08\% & 40.23\% & 64.84\% & 91.39\% & 36.91\% & 35.13\% & 90.67\% & 61.55\% & 77.81\% & 90.89\% & 45.88\% & 67.65\% & 89.54\% \\ \hline \multicolumn{12}{c}{**CIFAR-100**} \\ (Acc = **69.88\%**) & 27.49\% & 43.42\% & 69.92\% & 17.69\% & 39.94\% & 69.38\% & 23.84\% & 33.40\% & 60.43\% & 46.20\% & 50.56\% & 54.41\% & 18.61\% & 35.21\% & 53.17\% \\ \hline \multicolumn{12}{c}{**Adult**} \\ (Acc = **81.73\%**) & 51.71\% & 56.12\% & 79.79\% & 53.19\% & 60.42\% & 81.63\% & 52.39\% & 59.57\% & 80.78\% & 72.17\% & 78.32\% & 80.27\% & 59.53\% & 61.01\% & 79.32\% \\ \hline \multicolumn{12}{c}{**ogbg-molhiv**} \\ (AUC = **74.72\%**) & - & 70.04\% & 74.28\% & - & 70.53\% & 73.41\% & - & - & - & - & 65.24\% & 73.74\% & - & - & - \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Results of different SL paradigms on IID and non-IID data. Here, the results for models trained in the centralized setting are provided in bold for comparison, and \(\alpha=\infty\) indicates that the dataset is IID among clients [108]. assigning all samples with two labels to each client. Figure 6 shows the effect of different \(\alpha\)s on data partitioning. The overall results are also shown in Table 5. We select two representative \(\alpha\)s to demonstrate the results of SL paradigms at different data heterogeneity levels. The client number is set to 3 for all results (more results with five clients are shown in Table 7). We summarize the following observations from the results. * Most SL paradigms, except SplitFed, are observed to struggle in generalizing to the non-IID setting, particularly when data heterogeneity is high (when \(\alpha\) is small). The results in Table 5 demonstrate that, when the samples are strictly divided by their labels (i.e., \(\alpha=0\)), most paradigms do not perform well on the CIFAR-10 dataset. For example, Vanilla only achieved an accuracy of 34.24%, which is significantly lower than that of centralized learning (91.14%). On the Adult dataset, most SL paradigms perform similarly to random guesses (i.e., \(\approx\)50%). When the samples are not strictly divided by their labels/attributes (e.g., \(\alpha=0.1\)), the accuracy of all SL paradigms becomes slightly closer to that of centralized learning. However, except for SplitFed, most of the SL paradigms still perform much poorer than centralized learning. * Compared to other methods, SplitFed demonstrated higher robustness to non-IID data.