# Fractional Derivative Gradient-Based Optimizers for Neural Networks and Human Activity Recognition
## 2 Materials
### Tensorflow Implementation of Fractional Gradient Optimizers  
Tensorflow is a platform for machine learning, and it has been widely used for the deep learning community since it provides open-source Python libraries to train and deploy many applications [25]. Tensorflow also includes efficient support for GPU devices, as well as integration with high-level APIs, such as Keras [26]. Tensorflow is available for several operating systems and is also available through Jupyter notebook cloud services, such as Google Colab [27]. The module _tf.optimizers_ contains classes for gradient descent optimizers, such as SGD, Adadelta, Adagrad, Adam, among others. For example, the SGD optimizer is located in the Tensorflow-Keras module _tf.keras.optimizers.SGD_, and accepts some parameters, as is shown in the next fragment of code:  
tf.keras.optimizers.SGD(  learning_rate =0.01,  momentum=0.0,... )  
These parameters have default values, such as \(momentum=0\), that means that the default update rule is \(\Delta w=-learning\_rate*gradient\). Given a positive value of momentum, the update rule according to the API documentation is \(\Delta w=velocity\) where the \"velocity\" is defined as \(velocity=momentum*velocity-learning\_rate*gradient\). So, the velocity stores a single slot memory value as described in Section 1, and it corresponds to the \(\Delta\theta_{l,i}\) factor in the fifth row of Table 1.  
Since the main goal is to introduce the fractional factor of Equation (24) to the gradient descent optimizers, a simple and elegant solution is to multiply the current gradient by this factor. However, there are some aspects to be considered. First, note that Equation (24) involves a power \(1-\
u\) that will be negative for \(\
u>1\), and consequently, it could produce a division by zero (in the practice, Tensorflow obtains NaN values). A possible solution is to aggregate \(\epsilon>0\), as it was shown in Section 1. However, there is a second consideration; when \(1-\
u=\frac{p}{4}\), and \(q\) is even (for example \(\
u=\frac{1}{2}\) or \(\
u=\frac{3}{4}\)), then negative values of \(w^{l}_{kj}\) generate complex values. To deal with these two situations and to preserve real values, \(w^{l}_{kj}\) was replaced by \(|w^{l}_{kj}|+\epsilon\), so the proposed fractional gradient factor \(f^{\
u}_{w}\) is  
\[f^{\
u}_{w}:=\frac{(|w^{l}_{kj}|+\epsilon)^{1-\
u}}{\Gamma(2-\
u)}. \tag{25}\]  
A strong motivation to replace \(w^{l}_{kj}\) by \(|w^{l}_{kj}|+\epsilon\) is that it allows to have a limit for \(f^{\
u}_{w}\) when \(\
u\to 1\). In such a case,  
\[\lim_{\
u\to 1^{-}}\frac{(|w^{l}_{kj}|+\epsilon)^{1-\
u}}{\Gamma(2-\
u)}=\lim_{ \
u\to 1^{+}}\frac{(|w^{l}_{kj}|+\epsilon)^{1-\
u}}{\Gamma(2-\
u)}=1, \tag{26}\]  
that supports the idea of conceiving Equation (24) as a more general case of the integer gradient descent update rule. For the Tensorflow implementation, a new class FSGD with fractional gradient was defined based on the SGD optimizer. The _update_step_ method was modified as follows:  
```
#Definitionoftensors:constantsandvariables.Choosev>0.v=0.5 one_v=1-vtwo_v=2-vgamma_2_v=math.gamma(two_v)epsi=0.0000001 classPsGD(optimizer.Optimizer):...defupdat_step(self,gradient,variable):tmp1=tf.abs(variable)+tf.constant(epsi,tf.float32)tmp2=tf.constant(one_v,tf.float32)fw=tf.pow(tmp1,tmp2)/gamma_2_vgradient=tf.multiply(gradient,fw)
```  
The same procedure applies to other gradient descent optimizers listed in Table 1, and each fractional version uses the prefix \"F\". For example, FAdam is the fractional version of Adam, and it was obtained modifying the _resource_apply_dense_ method of the Adam class.