# Fractional Derivative Gradient-Based Optimizers for Neural Networks and Human Activity Recognition
## 2 Materials
### Backpropagation for Neural Networks  
In supervised learning, given an input data set \(X\) and the corresponding desired outputs \(O\), the training sample set can be expressed as \(\{X^{i},O^{i}\}_{i=1}^{N}\), where \(N\) is the number of samples.  
For a neural network with an architecture conformed by a single input layer \(X\), followed by \(L=H+1\) layers that considers \(H\) hidden layers and an output layer \(O\) with activation functions \(\varphi(x)\), the matrix of synaptic weights \(w^{l}_{kj}\) indicates the connection between the neuron \(k\) of layer \(l+1\) and the neuron \(j\) of the current layer, \(l\in[1,L-1]\). A special case is for \(l=0\), where the weights connect the input data \(X\) with the neurons of the first hidden layer.  
The error of neuron \(k\) at the output layer is \(e_{ki}=a^{L}_{ki}-o_{ki}\), where subindex \(i\) refers to the neural network receiving the \(i\)-th input pattern. Consequently, given the \(i\)-th training sample, the error \(E_{i}\) of the output layer considering all its \(n^{L}\) neurons is  
\[E_{i}=\frac{1}{2}\sum_{k=1}^{n^{L}}e_{ki}^{2}=\frac{1}{2}\sum_{k=1}^{n^{L}}(a^{ L}_{ki}-o_{ki})^{2} \tag{15}\]  
then, the total error \(E\) over all the \(N\) training samples is  
\[E=\sum_{i=1}^{N}E_{i}=\frac{1}{2}\sum_{i=1}^{N}\sum_{k=1}^{n^{L}}(a^{L}_{ki}-o_ {ki})^{2} \tag{16}\]and the learning process via the backpropagation algorithm aims to minimize \(E\) by adjusting the free parameters of the weight matrix.  
Essentially, a backpropagation training consists of repeated forward and backward steps. The forward step evaluates progressively the induced local fields \(V^{l}\), multiplying the inputs \(l^{l}\) of the \(l\)-th layer and the corresponding synaptic weights \(W^{l}=w^{l}_{kj}\). For the first layer, \(I^{1}=X\), so the induced local field vector at layer \(l\) can be expressed as the dot product \(V^{l}=l^{l}\cdot W^{l}\) where \(l\in[1,L]\).  
The output of neuron \(k\) at layer \(l\) is \(a^{l}_{k}=\varphi(v^{l}_{k})\), where \(v^{l}_{k}\) is the \(k\)-th local induced field of \(V^{l}\), and by convention for \(l=0\) the \"output vector\" \(a^{0}\) is equal to \(I=X\), the input data set. Of course, each activation function \(\varphi\) can be different for each layer.  
For the backward step, once the outputs \(a^{L}\) of the \(L\)-th layer have been calculated, the local gradients \(\delta^{l}_{k}\) are evaluated, and it allows to obtain the gradient descent updates in reverse order for \(l=L,L-1,\ldots,1\).  
Indeed, for the gradient descent optimizer, the weight updates \(\Delta w^{l}_{kj}\) are given by  
\[\Delta w^{l}_{kj}=-\eta\frac{\partial E_{i}}{\partial w^{l}_{kj}} \tag{17}\]  
seeking a direction for weight change that reduces the value of \(E_{i}\)[1].  
Since the local gradient is  
\[\delta^{l}_{k}=\frac{\partial E_{i}}{\partial v_{k}} \tag{18}\]  
and considering that  
\[\frac{\partial E_{i}}{\partial w^{l}_{kj}}=\frac{\partial E_{i}}{\partial v_{ k}}\cdot\frac{\partial v_{k}}{\partial w^{l}_{kj}}=\frac{\partial E_{i}}{ \partial v_{k}}\cdot d^{l-1}_{j}=\delta^{l}_{k}a^{l-1}_{j} \tag{19}\]  
then, \(\Delta w^{l}_{kj}\) can be expressed as  
\[\Delta w^{l}_{kj}=-\eta\cdot\delta^{l}_{k}\cdot a^{l-1}_{j}. \tag{20}\]  
At the output layer, \(\delta^{L}_{k}\) involves two factors, the error \(e_{ki}\) and the derivative of the activation function as follows:  
\[\delta^{L}_{k}=e_{ki}\cdot\varphi^{\prime}(v^{L}_{k}) \tag{21}\]  
whereas for hidden layer \(l\), the local gradient considers the contribution of errors via the \(k\) neurons of the \(l+1\) layer, hence  
\[\delta^{l}_{j}=\varphi^{\prime}(v_{j})\cdot\sum_{k=1}^{n^{l+1}}\delta^{l+1}_{ k}\cdot w^{l+1}_{kj}. \tag{22}\]  
To be consistent with the nomenclature of Section 1, let \(g_{t}=\delta^{l}_{k}\cdot a^{l-1}_{j}\), where \(a^{l-1}_{j}\) is the output of neuron \(j\) of the previous layer, i.e., an input to the layer \(l\). Additionally, let \(\Delta\theta_{t,i}=\Delta w^{l}_{kj}\) when the \(i\)-th training sample is presented to the neural network.