# Train Me If You Can: Decentralized Learning on the Deep Edge
## 4 L-SGD in Federated Learning  
As previously mentioned, researchers believe that the next-generation ML systems will rely on FL architectures. In this section, we study how L-SGD can leverage an FL scenario with Arm Cortex-M MCUs on the edge. We consider L-SGD as part of a centralized FL architecture with horizontal data partitioning. In a centralized design, the system architecture relies on two parts: (i) the edge and (ii) the cloud server. The inference and the main part of the training pass are confined to the edge. The weights resulting from the training pass are sent to the central server to be aggregated following a synchronous communication mechanism based on rounds. For weights aggregation, we consider a cloud server implementing the state-of-the-art FedAvg algorithm [48].  
Despite being the most relevant parameter aggregation mechanism, FedAvg has some known vulnerabilities [76; 77]. As there is no user data transferred from the edge to the server, the accuracy of the new ML model is evaluated over a small portion of centralized data. Consequently, the accuracy in some devices can be penalized due to the overall model generalization. According to some authors [78; 79], this raises the problem of fairness. As minority groups are represented by fewer data, FedAvg tends to return a more accurate model for the dominant groups. We envision that the quantized version of L-SGD can be integrated into a FLS to soften the fairness problem while introducing low overhead.  
As detailed in Section 5, the quantized version of L-SGD is very accurate for the later training iterations when the loss starts to converge to zero. In this phase, the magnitude of updates is so small that it is very unlikely for the weights and biases to be saturated. As a consequence, we envision that the quantized version of L-SGD could be used to tweak the global ML model to the local data of an edge device. As detailed in Section 5, under this scenario, quantized L-SGD was able to considerably raise the accuracy of an ANN trained on the MNIST dataset (91.10% to 92.83%), while registering a memory footprint reduction of 72.82% and a speedup of 2.48\(\times\) in relation to the floating-point version. Tweaking the ML model for the local data distribution will make the ML model more accurate in edge devices handling data from minority groups and reducing the unfairness effect.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Operation** & \multicolumn{3}{c}{**\(Qm.n\) Format**} & \multicolumn{1}{c}{**Constraints**} & **Procedure** \\ \multicolumn{1}{c}{**(A, B) = O**} & **A** & **B** & **O** & **N.A** &
\begin{tabular}{c} val = x * y \\ shifts = x + y \\ out = val \\ \end{tabular} \\ \hline
**Divide** & \(Qm_{1}.n_{1}\) & \(Qm_{2}.n_{2}\) & \(Qm_{3}.n_{3}\) & N.A &
\begin{tabular}{c} val = x * y \\ shifts = z - ( x - y) \\ out = val \textless{}cshifts \\ \end{tabular} \\ \hline
**Add** & \(Qm_{1}.n_{1}\) & \(Qm_{2}.n_{2}\) & \(Qm_{3}.n_{3}\) &
\begin{tabular}{c} \(n_{1}=n_{2}\) \\ \(n_{1}=n_{3}\) \\ \end{tabular} & O = A + B \\ \hline \hline \end{tabular}
\end{table}
Table 4: Quantized operations.  
Under this umbrella, we envision an FLS for Arm Cortex-M MCUs with a hybrid training process (Figure 3), encompassing two training phases: (i) main and (ii) secondary. The main training phase performs using the floating-point version of L-SGD and starts whenever the cloud server sends a training plan to the edge. The main phase is responsible for generating the ANN that is further sent to the cloud server as a local model update. The secondary phase starts whenever a new ANN is received on the edge. This phase uses the quantized version of L-SGD and tweaks the global ML model to local data. To prevent overfitting, it should be used with a low learning rate.