# Train Me If You Can: Decentralized Learning on the Deep Edge
## 3 Lightweight SGD (L-SGD)
### Quantized Training--L-SGD (int-8)  
To evaluate the feasibility of quantized training, we developed a version of L-SGD, which operates in weights, activations, and gradients in integer data. To comply with the CMSIS-NN library, we only considered ANNs under static quantization, applied layer-by-layer, and with a fixed bit-width of 8-bits. This alternative version has the potential to lower the memory footprint up to almost 75% of the value registered for the floating-point version. Furthermore, Arm Cortex-M MCUs perform faster over integer data than over floating-point data. The most recent MCUs are even equipped with SIMD instructions to support faster and parallel multiplications and multiply-accumulate operations of integer operands. Nevertheless, this gain in latency may be diluted by the mathematical operations to avoid the overflow of the gradients during the backward pass.  
The quantized L-SGD follows the same workflow detailed in Algorithm 2. However, the \(calculate\_global\_loss\), \(calc\_delta\), and \(update\_weight\) were re-designed to operate over integer data only. The result of the \(forward\_propagation\) is now maintained in int-8 format. To soften the impact of quantization, the precision of the intermediate calculus of L-SGD was set to 16-bit instead of 8-bit. As a consequence, the inputs to the \(calculate\_global\_loss\) function were up-scaled from 8-bit to 16-bit, where seven bits are used for the integer part and eight bits for the fractional part (\(Q7.8\)). The computation of \(calculate\_global\_loss\) and \(calc\_delta\) were redesigned to maintain this quantization format in their output. As the quantization is performed assuming a fixed-point format with a power-of-two scaling, as in CMSIS-NN, overflow can be avoided through a bitwise shift operation. As a  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Loss Function** & **Equation** \\ \hline Binary cross entropy (BCE) & \(-\sum\limits_{i=1}^{N}(y_{i}*log(\hat{y}_{i}))+(1-y_{i})*log(1-\hat{y}_{i})\) \\ BCE partial derivative & \(\dfrac{1-y_{i}}{1-y_{i}}-\dfrac{y_{i}}{\hat{y}_{i}}\) \\ \hline Cross entropy (CE) & \(-\sum\limits_{i=1}^{N}(y_{i}*log(\hat{y}_{i}))\) \\ CE partial derivative & \(\dfrac{y_{i}}{\hat{y}_{i}}\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Loss functions supported by L-SGD. Prediction is represented by (\(\hat{y}\)) and the true label by (\(y\)).  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Activation Function** & **Equation** \\ \hline ReLU & \(\begin{cases}0\,if\,f(x)\leq 0\\ x\,if\,x>0\end{cases}\) \\ ReLU partial derivative & \(\begin{cases}0\,if\,f(x)\leq 0\\ 1\,if\,f(x)>0\end{cases}\) \\ \hline Sigmoid & \(\dfrac{1}{1+e^{-x}}\) \\ Sigmoid partial derivative & \(f(x)*(1-f(x)\) \\ \hline TanH & \(\dfrac{e^{x}-e^{-x}}{e^{x}+e^{-x}}\) \\ TanH partial derivative & \(1-f(x)^{2}\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Activation functions supported by L-SGD.  
consequence, the multiply, divide, and add operations within the \(calculate\_global\_loss\), \(calc\_delta\), and \(update\_weight\) functions were implemented as detailed in Table 4.  
The \(update\_weight\) function was redesigned to take inputs in \(Q7.8\) format being the output adjusted to maximize the accuracy. As quantization performs layer-by-layer, every time saturation occurs in the update of a given weight, \(update\_weight\) down-scales the precision of all weights of the layer in the update cycle. Down-scaling consists of reducing by 1 bit the precision of fractional bits. In extreme cases, \(update\_weight\) may have to iterate multiple times through all weights of a given layer, neglecting one of the main goals of quantization. Consequently, in its current form, the quantized version of L-SGD is not designed to tailor the training of an ANN from scratch as the higher loss values would disrupt the fast saturation of gradients and ANN parameters. Nevertheless, results have shown that the quantized L-SGD is accurate for later training steps when the loss converges.