# Fractional Derivative Gradient-Based Optimizers for Neural Networks and Human Activity Recognition
## 1 Introduction
[PERSON] [6] mixes the average of past gradients \(m_{t}\) and past squared gradients \(v_{t}\) together with a momentum approach, in such a case that \(m_{t}=\beta_{1}m_{t-1}+(1-\beta_{1})g_{t}\) contains a previous memory value followed by a second term based on the gradient. Similarly, \(v_{t}=\beta_{2}v_{t-1}+(1-\beta_{2})g_{t}^{2}\) first involves a memory update term followed by a squared gradient term. Since \(m_{t}\) and \(v_{t}\) are initialized to zero, and to avoid zero-bias tendency, other formulas are obtained for the first momentum bias-corrected \(\hat{m}_{t}=\frac{m_{t}}{1-\beta_{1}}\) and for the second momentum bias-corrected \(\hat{\theta}_{t}=\frac{v_{t}}{1-\beta_{2}}\). The updated Adam formula is  
\[\Delta\theta_{t,i}=-\frac{\eta}{\sqrt{\hat{v}_{t}}+\epsilon}\hat{m}_{t} \tag{9}\]  
that also considers \(\epsilon>0\) in the denominator to avoid a zero division. Some other gradient descent optimizers have been proposed, but now just only one more is discussed, the AdamP optimizer [7]. AdamP is a variant of Adam that appeals to weight normalization and effective automatic step size adjustment over time to stabilize the overall training procedure that improves the generalization. The normalization considersprojections in the weight space via the projection operator \(\Pi_{\theta}(x):=x-(\theta\cdot x)\theta\) applied to the momentum update \(m_{t}=\beta m_{t-1}+g_{t}\), so the AdamP rule is \(\Delta\theta_{t}=-\eta q_{t}\) where  
\[q_{t}= \begin{cases}\Pi_{\theta_{t}}(m_{t}),&\text{if }cos(\theta_{t},g_{t}) \leq\frac{\delta}{dim(\theta)}\\ m_{t},&\text{otherwise}\end{cases}  \tag{10}\]  
for \(\delta>0\), and where \(cos(a,b)\) is the cosine similarity between two vectors. Table 1 summarizes these previously described optimizers. In addition, it also includes a version of SGD with \(\gamma\) momentum, as well as SGDP [7] that includes weight normalizations via projections based on SGD, analogous to how AdamP is built based on Adam. The first column indicates the name, the second column the update rule, and the third column a comment. It is emphasized that the update rule for some optimizers, such as Adagrad, Adadelta and Adam, involves \(\epsilon>0\) to avoid a division by zero. All previous optimizers consider the gradient \(g_{t}\) as the cornerstone update factor that comes from a first-order derivative of the objective function. The purpose of this work is to present optimizers that introduce a fractional derivative gradient in the update rule, as well as an implementation for the Tensorflow backend. This proposal is mainly based on the fractional differential calculus theory [8; 9; 10; 11] and on previous works [12; 13]. Fractional calculus is not a novel topic [14] but it has recently taken relevance in several fields, including linear viscoelasticity [15], fractional control [16], partial differential equations [17], signal processing [18], image processing [19], time series prediction [20], and mathematical economics [21], among others, and of course neural networks in the age of deep learning [12; 13; 22]. Given that neural network architectures have several challenges such as generalization enhancement, gradient vanishing problems, regularization and overfitting, it seems that fractional calculus still has a lot to contribute. The rest of the paper is organized as follows. In Section 2, details of the proposed fractional derivative gradient update rule are presented. In Section 3, experiments are described to obtain performance comparisons with known optimizers. It allows to support the main conclusion regarding the improvement between the performance of the proposed method and other existing ones. In Section 4, some discussions are presented based on the experiments, and some future work directions are commented on.