# Fractional Derivative Gradient-Based Optimizers for Neural Networks and Human Activity Recognition
## 1 Introduction  
In the context of machine learning, neural networks are one of the most popular and efficient techniques to model data, and gradient descent methods are widely used to optimize them. The fundamental gradient descent optimizer considers a factor with an opposite direction to the gradient of the objective function. Other optimizers consider momentum and velocity analogies to improve the training convergence and the generalization capacity. Effectively, starting with the basic update rule of gradient descent optimizers, the fundamental factor updates the free parameters in the opposite direction of the gradient \(g_{t}\) on the approximation error surface, and the learning rate \(\eta\) modulates the feedback to move forward to obtain a minimum [1]. A batched version vanilla gradient descent (GD) updates the parameters considering all the training samples, but it is impractical for large datasets. The GD update formula is  
\[\Delta\theta_{t}=-\eta g_{t}. \tag{1}\]  
Alternatively, a stochastic gradient descent (SGD) version updates the parameters for each \(i\)-th training sample. Hence, the SGD update formula is  
\[\Delta\theta_{t,i}=-\eta g_{t,i} \tag{2}\]  
and although it could introduce fluctuations, on one hand, it can be useful to explore the optimization space, but on the other hand, it can introduce unnecessary variance in the parameter updates, and it makes the learning rate a critical factor. Considering this, a mixed version of minibatch gradient descent proposes to split the dataset in subsets to deal with this tradeoff [2]. Adagrad [3] is the other evolved version that considers adaptation of the learning rate based on the memory of the gradients and aims to give helpful feedback for sparsed features of input data. Adagrad computes a historical diagonal matrix \(G_{t,ii}\), accumulating the sum of squares of the gradients to modify the adjustment of each parameter \(\theta_{i}\) which aims todeal with the disparity of frequent/infrequent features of training samples. The Adagrad update formula is  
\[\Delta\theta_{t,i}=-\frac{\eta}{\sqrt{G_{t,ii}+\epsilon}}g_{t,i} \tag{3}\]  
that considers \(\epsilon>0\) in the denominator to avoid a zero division. Adadelta [4] is a variant of Adagrad that aims to lessen the accumulation of square gradients along all the time for \(G_{ii}\), and instead it defines an average window given \(0\leq\gamma\leq 1\) to ponderate squares of current and previous one gradients according to  
\[E[g^{2}]_{t}=\gamma E[g^{2}]_{t-1}+(1-\gamma)g_{t}^{2} \tag{4}\]  
so, it can be conceived as the root mean squared error of the gradients:  
\[RMS[g]_{t}=\sqrt{E[g^{2}]_{t}+\epsilon} \tag{5}\]  
where \(\epsilon>0\) is also included to avoid a division by zero, given that the original update formula for Adadelta is  
\[\Delta\theta_{t}=-\frac{\eta}{RMS[g]_{t}}g_{t}. \tag{6}\]  
To preserve the same \"unit of measure\", the learning rate \(\eta\) is replaced by the RMS of parameter updates in this way:  
\[RMS[\Delta\theta]_{t}=\sqrt{E[\Delta\theta^{2}]_{t}+\epsilon}=\sqrt{\gamma E[ \Delta\theta^{2}]_{t-1}+(1-\gamma)\Delta\theta_{t}^{2}+\epsilon} \tag{7}\]  
up to \(t-1\) since \(\Delta\theta\) is still being calculated. Therefore, the Adadelta update rule is  
\[\Delta\theta_{t}=-\frac{RMS[\Delta\theta]_{t-1}}{RMS[g]_{t}}g_{t}. \tag{8}\]  
RMSProp [5] is considered an extension of Adagrad that maintains a moving average square of the gradients instead of using all the historical, and divides the gradient by the root mean square of that average. In this sense, it has a great similarity with Equation (6) presented as the former Adadelta rule. Other optimizers consider momentum (update memory) based on the update of the previous iteration, analogous to the physical concept of particle inertia, so when \"the ball moves\" in the same direction from the current to the next update, it accelerates the convergence, and it opposes when it changes directions, providing more stability and better convergence.