# Train Me If You Can: Decentralized Learning on the Deep Edge
## 3 Lightweight SGD (L-SGD)  
### Node Delta Optimization  
Algorithms 1 and 2 highlight the optimization of L-SGD over the baseline reference SGD. Both algorithms start with the forward pass, where the output of the ANN for the given input sample and the respective loss are calculated. The improvement of L-SGD over SGD occurs during the backward pass. Both algorithms update the weights of the ANN according to Equation (1), however, with very distinct workflows. As detailed in Algorithm 1, the backward pass of SGD involves two steps: (i) the calculus of the error caused by each weight, according to the chain-rule detailed in Equation (2), and (ii) the calculus of the new weight. The calculus of the new weight takes three parameters as input: (i) the starting weight value, (ii) the corresponding error, and (iii) the learning rate. Due to the propagation error effect, the calculus of the error associated with a given weight \(w\) at layer \(l\) depends on the partial error caused by the weights in the following layers. As a consequence, all the initial weights, the output of each neuron, and the error caused by each particular weight of the ANN need to be stored in memory till the method \(update\_weight\) is called. ```
1:for each epoch = 1, 2,... do
2:for each input sample = (\(x_{i}\), \(y_{i}\)) do
3: y_pred = forward_propagation(x_i)
4: for each layer = \(l_{n}\), \(l_{n-1}\), \(...\), \(l_{0}\)do
5:for each weight = \(w_{i}\)do
6: error[w] = calculate_error_weight(loss, w_i)
7:endfor
8:endfor
9:endfor
10:for each layer = \(l_{n}\), \(l_{n-1}\), \(...\), \(l_{0}\)do
11:for each weight = \(w_{i}\)do
12: w_i = update_weight(w_i, error[w], lr)
13:endfor
14:endfor
15:endfor
16:endfor
```  
**Algorithm 1** Baseline implementation of SGD. ```
1:for each epoch = 1, 2,... do
2:for each input sample = (\(x_{i}\), \(y_{i}\)) do
3: y_pred = forward_propagation(x_i)
4: loss = calculate_global_loss(y_pred, y_i)
5:for each neuron = \(n_{i}\)\(\in\) last_layer do
6: delta_buf1[n] = calc_delta(loss, n_i)
7:endfor
8:for each layer \(l_{i}\)\(\in\)\(l_{n-1}\), \(l_{n-2}\), \(...\), \(l_{0}\)do
9:for each neuron = \(n_{i}\)do
10: delta_buf2[n] = calc_delta(delta_buf1, n_i)
11:endfor
12:for each weight \(w_{i}\)\(\in\) layer \(l_{i+1}\)do
13: w_i = update_weight(w_i, delta_buf1, lr)
14:endfor
15: delta_buf1 = delta_buf2
16:endfor
17:for each weight = \(w_{i}\)\(\in\) input_layer do
18: w_i = update_weight(w_i, delta_buf1, lr)
19:endfor
20:endfor
21:endfor
```  
**Algorithm 2** Implementation of L-SGD. L-SGD improves over SGD by considering that weights linked to the same output neuron share some terms in the chain-rule of the error propagation. For a given weight \(w\), these terms correspond to the error propagated from the ANN output neurons till the output neuron that is linked to \(w\). Consequently, these parameters form a characteristic of each neuron (Figure 2). L-SGD replaces the SGD calculus of the error propagated till each weight with the calculus of the error propagated till each neuron hereinafter referred to as node delta. Node delta is then used as input to the \(update\_weight\) function. As detailed in Algorithm 2, the calculus of node delta values during the backward pass is performed in two distinct moments, according to the layer type. The calculus of node delta for the output layer is performed using the partial derivatives of the loss and activation functions. The calculus of node delta of the remaining layers is performed in an iterative process--the calculus of node delta values for the layer \(l\) takes as input the node delta values of layer \(l+1\). This occurs as the error propagated till a neuron \(n\) in layer \(l\) can be seen as the product of the errors propagated till neurons in layer \(l+1\) with the weights linking \(n\) to layer \(l+1\). After computing the node delta of a given neuron, the weights associated with it can be updated, suppressing the need to split the backward pass and the weights update process.