## 2 The Rise of the Federated Learning Approach
### Delving into Decentralized Federated Learning (DFL)
#### 2.3.1 Communication Reduction
During training, the node \(i\) undertakes to aggregate only one segment of its own with the respective segment of \(k\) out of the total participant nodes. In order for the algorithm to reach convergence, it is suggested that the \(k\) value be less than the value of the total participants (workers). The gossip-based heuristic (Combo) was tested using a CNN model and the CIFAR-10 dataset [PERSON] et al. (2018) against both the conventional FedAvg algorithm and classic gossip approach (without model segmentation). The results showed that Combo outperforms the aforementioned baselines in regard to convergence speed (about 3\(\times\) faster) with a negligible drop (from 1% to 2%) in validation accuracy. Moreover, in [42], a decentralized federated learning scheme is proposed, in which the model, whose updates are propagated toward the nodes participating in training, is split and every worker transmits a part of the update that it is responsible for in a peer-to-peer way. Based on the gossip protocol, this approach decreases transmission bandwidth and acquired training time with a little trade-off regarding accuracy by splitting the model and using the gossip strategy in worker communication, while it also replicates the data from different non-overlapping nodes so that it can ensure that enough information is gathered for better convergence during aggregation. Another algorithm in [44], namely _Soft-DSGD_, was proposed in order to alleviate communication unreliability regarding UDP protocol end-device use in order to communicate in peer-to-peer networks. Soft-DSGD updates model parameters during training using a part of the weights successfully received, while it also strengthens the weights originating from devices with reliable links (e.g., a coordinator's link), taking into consideration a _reliability matrix_ to achieve brisker convergence (reaching _consensus_). In order to deal with the package transmission failures (which the UDP protocol is susceptible to due to the fact that packets are forwarded to all the possible receivers and there is no validation if the message reached its destination) and thus normalize the training procedure, every device replaces lost packet information with a part of its local parameters. Not only numerical results but also convergence analysis proofs (treating it as a convex optimization problem) have shown both similar behaviours and asymptotic convergence rates between _banilla decentralized SGD_ (in its ideal communication conditions) [21] and Soft-DSGD. Prototype learning is also an interesting way of alleviating overflow in communication links. A _prototype_ is a data value that represents the values of a class. So, prototype learning in decentralized networks is a technique that exploits this feature in order to further reduce the communication load by sending only prototype representations of the gradients and not the exact ones in order for the nodes to perform training. A prototype, namely _DeProFL_, was introduced by [PERSON] et al. [45] in order to deal with both the heterogeneity, caused by the time-dependable shifting (dynamic) communication infrastructure (a feature that characterizes wireless networks), and communication overhead. In every round, a different network topology is randomly defined, while every node performs some local updates and then propagates only the prototypes to its neighbors. Practically, according to the authors, nodes send the vector generated after the first layer, which is responsible for understanding the given data features and not the vector, produced as a prediction of the supervised model (as usual) across their neighbors. Regarding data heterogeneity, authors have set the degree of heterogeneity to 0.1 while they conducted experiments with the _Dirichlet Non-IID_ dataset. As far as model heterogeneity is concerned, all nodes have different deep learning model structures, making centralized (SOTA) FL approaches like Ditto or FedAVG fail to keep up with these requirements.