Rewrite the following and make it even more high technical and academic, add more technical details where useful while keeping it short. add some information on the mechanics of how the two algorithms operate, assume the reader has some knowledge so be sure to get technical and explain drawback and considerations when using these two:
4.2.2 Federated Learning
Federated Learning (FL) is an innovative approach within the broader realm of distributed ML that aims to address the challenges of data privacy, security, and efficiency in training ML models. Unlike traditional centralized methods where data is aggregated in a central server, FL enables model training directly on edge devices, ensuring that sensitive data remains local (Fig. 19). This paradigm is particularly relevant in the age of ubiquitous connected devices and stringent data privacy regulations. Despite its promise, federated learning faces several challenges, including model convergence, memory overhead on edge devices, heterogeneity of client devices, and robustness against adversarial attacks.  [Drainakis2023] explores the transition from centralized ML to federated learning, focusing on performance metrics and resource consumption. This study helps in understanding the trade-offs involved in adopting federated learning, guiding practitioners in optimizing resource allocation and system design. Federated Learning facilitates the collaborative training of ML models across multiple decentralized devices or servers, each holding local data samples. [Liu2022] is a survey that provides a broad overview of the evolution from distributed ML to federated learning. This technique minimizes the risk of data breaches and enhances user privacy by keeping raw data on local devices and only sharing model updates with a central server for aggregation. The fundamental workflow involves several rounds of local training, communication of model updates, and global aggregation, iteratively improving the model's performance. [Chen2023B] addresses how privacy-preserving techniques, such as differential privacy, can inadvertently affect the fairness of the model across different user groups. This research emphasizes the importance of designing federated learning systems that not only protect user privacy but also ensure equitable model performance, highlighting the need for fairness-aware algorithms. [Lyu2020] also delves into the development of federated learning models that are both fair and privacy-preserving. It proposes methodologies to balance the dual objectives of fairness and privacy, ensuring robust and equitable model performance. [Uddin2020] introduces a novel approach to federated learning driven by mutual information. By leveraging mutual information, the approach optimizes the exchange of information between the client and server, enhancing model performance and convergence rates.
In federated machine learning, it is hard to train very large ML models on edge devices. [Abedi2023] mitigates computational burdens on client devices while preserving data privacy. It splits the model into two parts and distributes them across the client and server, enhancing the feasibility of training complex RNNs on resource-constrained devices and broadening the applicability of FL to more varied and demanding tasks involving sequential data. [Gao2020] presents an end-to-end evaluation of federated learning and split learning in the context of Internet of Things (IoT) applications.
4.2.3 Split Learning
Split learning is a distributed ML framework designed to address privacy and computational efficiency concerns. It achieves this by dividing a neural network into distinct segments that are processed by different entities. Typically, the initial layers of the network are computed on client devices (where the raw data resides), and the subsequent layers are processed on a cloud server as shown in Fig. 19 as compared to federated learning. This division helps protect sensitive data, as only the intermediate outputs (or "smashed data") are transferred between the client and the server, rather than the raw data itself.
In a typical split learning setup, each client trains its portion of the network up to a specific cut layer. The intermediate outputs at this cut layer are then sent to a central server, which completes the training by processing the remaining layers. This approach ensures that raw data never leaves the client's devices, providing a significant privacy advantage. Additionally, the computational load is distributed between the client and the server, which can be beneficial in scenarios where clients have limited computational resources. While split learning offers several advantages, it also presents unique challenges. Despite the enhanced privacy of not sharing raw data, there is still a risk that intermediate data (smashed data) could be reverse engineered to reveal sensitive information. Adversaries might exploit the activation values to reconstruct the input data using sophisticated techniques. To cope with it, there are some techniques like differential privacy [Wu2023], homomorphic encryption [Pereteanu2022], and secure multi-party computation [Zhou2024]. Another problem in split learning is model convergence, clients might have diverse hardware capabilities and data distributions, leading to non-IID (non-independent and identically distributed) data challenges. This heterogeneity can affect the convergence and performance of the global model. [Abedi2023] combines federated learning and split learning to leverage the strengths of both. FedSL can handle non-IID data by averaging model updates from clients and then splitting the model for further training.