# Train Me If You Can: Decentralized Learning on the Deep Edge
## 6 Related Work  
Rising concerns about data privacy aligned with the end of Moore's law and the ever-growing number of IoT devices are forcing intelligence to shift from the cloud to the deep edge, near to the data source. This gave rise to new computing paradigms, from which FL stands out. In this computing paradigm, the inference and part of the training must be performed on the edge, while the cloud is left for periodic global model updates. **Inference pass:** Arm developed CMSIS-NN, a library to maximize the performance and minimize the memory footprint of ANNs on Arm Cortex-M MCUs. CMSIS-NN kernels reveal a throughput improvement and lower energy consumption than the corresponding floating-point baseline functions [80]. PULP-NN API is a similar solution for the RISC-V (RV32 IMCXpulp) architecture [47]. The key innovation of PULP-NN is the support for multi-core processing. However, the support of activation functions is more limited--it only supports ReLU. Both APIs allow reliable execution of the inference pass of ANNs with negligible accuracy loss. **Training pass:** Training an ANN is the process of optimizing model parameters. Gradient-descent (GD) is the most basic optimizer. The optimization relies on the first-order derivative of a loss function. Based on the value returned by the loss function, GD calculates the direction of change of the model's parameters (weights and bias). Updating the parameters multiple times through this mechanism leads to an optimal solution that \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{**MNIST**} & \multicolumn{2}{c}{**CogDist**} \\ \hline  & **L-SGD** & **L-SGD** & **L-SGD** & **L-SGD** \\  & **(Float-32)** & **(int-8)** & **(Float-32)** & **(int-8)** \\ \hline Accuracy (\%) & 92.54 & 92.83 & 91.95 & 92.79 \\ Memory footprint (Bytes) & 3784 & 1026 & 636 & 239 \\ Latency (ms/sample) & 17.84 & 7.17 & 8.51 & 4.49 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: L-SGD (float-32) vs. SGD (int-8). reduces the prediction loss. The moderate computational complexity of this algorithm would allow its deployment on resource-constrained MCUs if it is not for its prohibitive memory footprint. GD requires an entire dataset at a time to update a parameter [81]. SGD is an extension of GD that aims to tackle the large memory footprint concern [82]. SGD uses only one sample at a time rather than the entire dataset. Besides the memory footprint reduction, SGD is less vulnerable to the local minima effect. However, the time to complete a single epoch is larger. The major obstacle to GD-based approaches is the setting of the learning rate. Both GD and SGD mechanisms keep this hyperparameter constant along the training process. AdaGrad [83] is a solution that allows the implementation of an optimizer without a manual tunning of the learning rate. AdaGrad uses a distinct learning rate for each parameter, decreasing it as the number of training iterations increases. Nevertheless, as the learning rate decreases to a very low value, the convergence becomes very slow [81]. AdaGrad divides the learning rate by the sum of squares of previous gradients. When the sum of the squared gradients is high, it divides the learning rate by a large number, leading to the learning rate decreasing. Similarly, if the total of the squared prior gradients is low, the learning rate is divided by a smaller number, resulting in a high learning rate. This means that the learning rate is inversely proportional to the sum of the squares of the prior gradients. Adam [84] improves over AdaGrad as it uses the squared gradients to scale the learning rate and the moving average of the gradient instead of gradient itself. More specifically, [PERSON] uses estimations of first and second moments of gradient to adapt the learning rate for each weight of the neural network. Adam is the fastest optimizer to converge to an optimal solution. **Federated learning:** In the recent past, multiple solutions have been proposed to enable decentralized learning.