# Train Me If You Can: Decentralized Learning on the Deep Edge
## 7 Conclusions  
The current state-of-the-art provides multiple solutions that allow a fast and efficient optimization of ML models. However, the currently-available mechanisms present a high demand for memory and computational resources, which is prohibitive for the deep edge. In this regard, the development of lightweight mechanisms that meet the stringent requirements of deep edge devices is of utmost importance to enable the integration of ML technology in situations that demand real-time response. To the best of the author's knowledge, no previous work has explored the development of training optimizers targeting the Arm Cortex-M MCUs.  
In this paper, we present the first framework to enable the deployment of decentralized learning in resource-constrained devices, typically powered by Arm Cortex-M MCUs. We borrowed the classic SGD algorithm and optimized it for low-memory footprint and latency on Arm Cortex-M MCUs. To test the feasibility of quantized training, we developed two versions of this algorithm: (i) L-SGD (float-32) and (ii) L-SGD (int-8). Results for the MNIST dataset show that L-SGD (float-32) is \(4.20\times\) faster than the classic SGD while requiring only \(2.80\%\) of the memory. Fully-quantized training from the scratch is still not feasible due to the fast saturation problem. However, L-SGD (int-8) is very accurate for the later training steps, when the loss starts to converge to zero. For the MNIST dataset, it was able to considerably raise the accuracy of an ANN (\(91.10\%\) to \(92.83\%\)), while registering a memory footprint reduction of \(72.82\%\) and a speedup of \(2.48\times\) in relation to the floating-point version. This makes the quantized version of L-SGD very useful to counteract the fairness problem of FLSs based on FedAvg.  
The major limitation of shifting the training process to the deep edge is the high demand for memory resources. Usually, the platforms used in deep-edge typically integrate only a few Kbytes or Mbytes of memory, so training ANNs is unfeasible through the traditional ANN training mechanisms. However, as can be observed in the analysis of the results above, L-SGD allows very significant reductions in memory usage, which makes it possible to perform ANNs training on deep-edge devices. Given the direction of the development of IoT systems, this is an essential step for the next-generation ML systems.  
Future work encompasses adapting L-SGD to support the training of convolutional neural networks, which are very common in image classification applications. We will also seek to provide support for RISC-V (RV32 IMCXpulp) MCUs, which is a rising computing architecture for ML in low-power applications.  
Software, D.C. and M.C.; investigation, D.C. and M.C.; supervision, S.P.; Writing--review & editing, D.C., M.C. and S.P. All authors have read and agreed to the published version of the manuscript.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Optimizer** & **Computational Complexity** & **Memory Footprint** & \
\begin{tabular}{c} **Vulnerable** \\ **to Local** \\ **Minima Effect** \\ \end{tabular} &
\begin{tabular}{c} **Automatic** \\ **Learning** \\ **Rate Decay** \\ \end{tabular} & **Latency** \\ \hline GD [81] & Low & High & Yes & No & Slow \\ SGD [82] & Moderated & Moderated & Yes & No & Slow \\ AdaGrad [83] & High & Moderated & No & Yes & Moderated \\ Adam [84] & High & Moderated & No & Yes & Fast \\ L-SGD & Low & Low & Yes & No & Moderated \\ \hline \hline \end{tabular}
\end{table}
Table 8: Training algorithms—Gap analysis.