# Overview of Distributed Machine Learning Techniques for 6G Networks
## 5 Distributed Machine Learning for 6G
### Communication-Efficient Federated Learning  
The traditional FedAvg technique, which was previously described, gives a global ML model that is suitable for all devices. Such a technique can be suitable if all devices have independent and identically distributed (IID) datasets. However, in practice, such IID assumptions are rarely true, especially for wireless scenarios with heterogeneous nodes. Therefore, traditional FedAvg-based FL systems can have limited performance in the wireless networking domain. By taking this into account, in [3], researchers proposed two distinct FL algorithms. In particular, they proposed federated multi-task learning (FMTL) and FL based on model agnostic meta-learning (MAML) methods. In the case of FMTL, it is possible to define the minimization problem as:  
\[\min_{M,\Omega}\sum_{i=1}^{U}\sum_{kk}f(m_{i},x_{i,k},y_{i,k})+R(M,\Omega) \tag{1}\]  
where \(M=[m_{1},m_{2},\ldots,m_{U}],\Omega\) models the relationship between the different device learning operations and \(R\) is a regulation function. The problem can be divided into sub-problems in order to offer the devices the ability to operate in a distributed manner in order to achieve the given goal. It is worth noting that the devices in FedAvg all use the same convergence model. In the case of FMTL, however, the devices may have different convergences, as one model may have less training losses than another due to the non-IID distribution. The FL technique based on MAML tries to develop a ML model that allows each device to reach its own model after a few gradient descent iterations. In this situation, the goal is stated as follows:  
\[\min_{m}\sum_{i=1}^{U}\frac{P_{i}}{k_{i}}\sum_{k\in K_{i}}f(m-\lambda\
abla f_ {i},x_{i,k},y_{i,k}) \tag{2}\]  
where \(\
abla f_{i}\) is the gradient descent of the local ML model of the \(i\)-th device, and \(\lambda\) denotes the learning speed. The main characteristics of the FMTL and MAML approaches are given as:  
* The FMTL algorithm directly optimizes the customized model of each device, while the federated MAML optimizes the initial model of all devices. * When working with IID data, the FedAvg algorithm is essential, although FMTL and MAML are more practical for non-IID data. * The parameter server (PS) must be aware of the data distributions in the devices to choose between FMTL and MAML. * All FL algorithms must be trained by a distributed iterative process. The qualities used to evaluate the performance of FL implementations for wireless networks are training loss, convergence time, energy consumption, and reliability. Training loss is the loss experienced by the function \(f\) described in (1) and (2). The ML models of the devices are communicated across imperfect connections in wireless networks; therefore, transmission problems that severely effect training losses are possible. Furthermore, due to limited energy and processing power, only a small subset of devices can actively engage in the learning process, resulting in only a portion of the ML models being used to construct the global model, resulting in negative consequences once again. The convergence time \(T\) for FL implementations in wireless networks is expressed as:\[T=(T_{C}+T_{T})\times N_{T}\]  
where \(T_{C}\) is the time it takes each device to update its local model at each learning step, \(T_{T}\) represents the maximum transmission time per ML model step, and \(N_{T}\) represents the number of steps required for FL to converge. Both the \(N_{T}\) and \(T_{C}\) factors are dependent. In particular, increasing the number of SGD steps to update the local model at each learning step (for example by increasing \(T_{C}\)) can decrease the number of steps required for the FL to converge. The energy consumption \(E\) of each device participating in the FL training is expressed as:  
\[E=(E_{C}+E_{T})\times N_{T}\]  
where \(E_{C}\) denotes the energy consumed by each device during the training of its model, and \(E_{T}\) is the energy required to communicate the parameters to the parameter server during each step.