# Fractional Derivative Gradient-Based Optimizers for Neural Networks and Human Activity Recognition
## 4 Discussion  
Unlike other optimizers that have been proposed, alluding mainly to concepts such as momentum or velocity, in this paper, gradient descent variants are proposed based on fractional calculus concepts, and specifically on the Caputo fractional derivative.  
The proposed fractional optimizers add the prefix \"F\" to original names, and their update formulas essentially aggregate the \(f^{v}_{w}\) factor defined in such a way that the limit exists when the \(v\)-order derivative tends to \(1\), which leads to a more general formula that includes the integer order as a special case.  
Fractional optimizers are slightly more expensive computationally because they require computing the \(f^{v}_{w}\) factor. However, they are still very competitive because the computation is performed efficiently through Tensorflow, and the additional advantage is that the fractional factor transfers non-local exploring properties, rather than just considering an infinitely small neighborhood around a point on the error surface as in the  
Figure 20: Plot for the fractional factor \(f^{v}_{w}\) with \(\epsilon=0.01\)  
integer case, which experimentally is traduced in the improvement of the performance of fractional optimizers.  
The fractional factor \(f_{w}^{\
u}\) provides an additional degree of freedom to the backpropagation algorithm, and consequently, to the learning capacity of neural networks, as was shown in several experiments.  
The fractional optimizers were successfully implemented in Tensorflow-Keras with modifications to the original source code to obtain FSGD, FSGDP, FAdagrad, FAdadelta, FRMSProp, FAdam and FAdamP classes. Everything indicates that it is possible to apply the same methodology to modify other gradient-based optimizers, as well as making implementations in other frameworks.  
Three experiments were carried out with MNIST and two HAR datasets. The results on crossfolding show that in all the experiments, a fractional order provides better performance than the first order for the same neural network architectures.  
In the experiments, FSGD, FSGDP, FAdagrad and FAdadelta (group 1) basically follow the same pattern of increasing their performance as the \(\
u\)-order does, obtaining a maximum and then decreasing.  
Other optimizers, such as FRMSProp, FAdam and FAdamP (group 2), do not follow the same pattern, and seem to be less susceptible to the fractional order change. From the experiments, it can be said that FRMSProp has an \"intermediate\" pattern between the optimizers of group 1 and group 2.  
Even so, essentially in all the experiments, the best performing derivative order was a fractional value.  
Therefore, based on the results, it is possible to affirm that fractional derivative gradient optimizers can help to improve the performance on the training and validation task, and opens the possibility to include more fractional calculus concepts to neural networks applied to HAR.  
The Tensorflow-Keras implementations of this work are available in a repository to contribute to the deep learning and HAR communities to improve and apply these techniques based on fractional calculus.  
Future works include exploring more fractional derivative definitions and HAR datasets with fractional regularization factors as well as studying the effects in vanishing gradient problems with other neural network architectures.  
This research received no external funding.  
Not applicable.  
The Tensorflow-Keras implementations of this work are available at [[http://ia.zac.uam.mx/](http://ia.zac.uam.mx/)]([http://ia.zac.uam.mx/](http://ia.zac.uam.mx/)).  
The author declares no conflict of interest.