# A Review and Experimental Evaluation on Split Learning
## 2 Taxonomy and Review of SL Paradigms
### Weight Aggregation-Based Approach  
The integration of SL and FL aims to capitalize on the strengths of both schemes. Since SL is adept at handling resource-limited clients, such as IoT devices, it divides the entire  
Figure 3: Illustration of the two model split-only SL paradigms with communication cost reduction: (**a**) AsyncSL and (**b**) 3C-SL. In (**a**), the dotted box indicates the process of deciding whether to transmit the data and the arrows with dotted margins denotes the transmissions that are not performed. model into smaller submodels that can be trained independently on the server and clients. Meanwhile, FL excels at addressing non-IID data scenarios. Based on this observation, numerous weight aggregation-based SL paradigms have been proposed. These paradigms train a global client model by combining locally trained models from clients using weight aggregation methods such as FedAvg [12]. In these paradigms, throughout the training process, some or all of the clients send their submodels to the server or a trusted third party for weight aggregation at specific stages. In SplitFed [46], the clients and the server train their models on their local datasets, similar to what occurs in PSL. Multiple clients train their models in parallel, and after each client finishes a training round, its local model weights are transmitted to the server and aggregated using FedAvg [12] as follows:  
\[\theta^{c}=\sum_{k=1}^{K}\frac{|\mathcal{D}^{c}|}{|\mathcal{D}|}\theta_{k}^{c}.\]  
The process is illustrated in Figure 3(a). [PERSON] et al. [46] also propose a variant of SplitFed, SplitFedv2, where clients perform the training round linearly, rather than in parallel. In PPSFL [55], a Group Normalization layer is added to the SplitFed paradigm, which does not participate in aggregation, to prevent privacy leakage. In Federated Split Learning (FSL) [91], each client trains its local model using an SL paradigm in conjunction with its corresponding edge server, rather than interacting with a central server. After one epoch has been completed, the model weights of the edge server are aggregated via the central server. In particular, the key distinction between SplitFed and FSL pertains to whether the aggregated model corresponds to the client side or the edge server side of the model. The process is illustrated in Figure 3(b). VehicularFSL [101] applies FSL to vehicular networks, where each vehicle (client) collaborates with its nearest roadside infrastructure for training. Considering the statistical and system heterogeneity in edge systems, assigning appropriate working nodes to specific clusters for efficient model training is a challenge. ParallelSFL [92], built upon SplitFed and FSL, develops an effective clustering strategy by optimizing a utility function related to training efficiency and model accuracy. It further addresses the issue of system heterogeneity by assigning different and appropriate local update frequencies to each cluster. With the concept of U-shaped paradigms and using the Vision Transformer (ViT) [102], the Federated Split Task Agnostic (FeSTA) paradigm [16] was introduced. In this paradigm, as illustrated in Figure 3(c), the architectures of the client models may vary with different  
Figure 4: Illustration of weight aggregation-based SL paradigms: (a) SplitFed; (b) FSL; and (c) FeSTA. tasks. The server holds the Transformer body and a model parameter aggregator that first initializes the weights of the Transformer body and task-specific heads and tails for each task. Clients are divided into groups according to their tasks and trained in parallel. After each training epoch, the clients send the head and tail parameters to the server; the server aggregates the model parameters for the same task and returns them to the clients, using techniques such as FedAvg [12]. This paradigm is suitable for training models for multiple similar tasks as it has a shared Transformer body.