# A Review and Experimental Evaluation on Split Learning
## 2 Taxonomy and Review of SL Paradigms
### Weight Aggregation-Based Approach
Quite similar to FeSTA, but unlike FeSTA where the server sends the class token obtained from the last ViT block to the clients, FeSViBS [93] samples the output of one of all ViT blocks and then utilizes a shared projection network to reduce the output dimension before transmitting it to the clients. Hybrid Split and Federated Learning (HSFL) [94] and Cluster-based Parallel SL (CPSL) [18] are two paradigms designed to address the practical constraints of power, bandwidth, and channel quality typically present in client devices. As the communication costs of FL and SL depend on multiple factors, HSFL finds an optimal allocation strategy based on the current channel conditions and the diversity index [103], allowing some clients to use the SL method and others to use the FL method. Regardless of the chosen training method, client model aggregation is necessary after each training cycle. CPSL, on the other hand, is based on SplitFed and selects appropriate partition layers and training clusters based on the current channel quality and client data holdings to reduce communication costs. Both methods utilize greedy algorithms to optimize their selection problems. Also, the Weighted Sampled Split Learning (WSSL) paradigm [68] was proposed to address the challenge of non-IID data in SL. It employs a dynamic client selection strategy based on client contributions measured by the normalized weight, ensuring consistent training and improving model output. Owing to the client selection strategy, WSSL goes beyond typical SL paradigms by enhancing model robustness, reducing overfitting risk, and promoting fairness. FedSplitX [63] addresses system heterogeneity with limited computation capabilities by partitioning a large model into client- and server-side models at different cut layers. This accommodates heterogeneous capabilities, creating flexibility with multiple model pairs based on different cut layers. The framework introduces auxiliary networks for these layers, enabling local loss-based learning, reducing communication costs, and improving performance. For local model optimization, FedSplitX aggregates parameters separately for client- and server-side models using FedAvg. Since the heterogeneity of the computation and communication capacities becomes a bottleneck of the SL training speed, Dynamic Federated split Learning (DFL) [69] incorporates resource-aware split computation of deep neural networks and dynamic clustering of training participants based on the similarity of their submodel layers. Through resource-aware split learning, DFL adjusts the allocation of training tasks to participants with limited resources, aligning with their computational capabilities and the dynamic nature of communication resources. To handle training data heterogeneity without compromising privacy or transmitting raw data over networks, DFL employs the Centered Kernel Alignment (CKA) [104]. This approach effectively assesses the similarity between layers, allowing for a more personalized and efficient training. The CHEESE paradigm [52] attempts to address the heterogeneity issue from a different perspective. It initially clusters clients through Device-to-Device (D2D) communication. Each client is then allocated a part of the model based on its computational power, clients within the same cluster collaboratively train a complete model, and finally, models among different clusters are aggregated using FedAvg. SplitLLM [96] combines LLM training with SplitFed and FSL by splitting the pre-trained model across the cloud, edge, and user devices, updating and aggregating only the LoRA adapters, which reduces memory usage and computational workload. FjORD [85] is an FL paradigm that adjusts the model size based on available client resources. Although it does not strictly adhere to the definition of SL, it is relevant due to its support for heterogeneity and \"splitting\" through ordered dropout. It employs Ordered Dropout (OD) for importance-based pruning, enabling devices to train submodels and contribute to the global model.