## 4 Algorithm Bias
### Optimizers
On the other hand, if the feature is not being updated often, this means that this feature is infrequent in the dataset and the AdaGrad optimizer assigns a high learning rate for the corresponding parameters. This makes AdaGrad an appropriate choice when dealing with sparse data [164]. However, even though AdaGrad utilizes adaptive learning rates, as explained above, it is sensitive to the global learning rate that is being initialized at the beginning of the optimization and which may result in not arriving at the desired local optima [165]. In order to address the aforementioned issue, other variants of the AdaGrad have been proposed in the literature, including Adadelta. The main difference between AdaGrad and Adadelta is that the latter does not require to manually set the learning rate [166]. More specifically, Adadelta utilizes \"delta updates\" in which it calculates the ratio of the root mean squared (RMS) of past gradients and the RMS of the past updates so that it can adjust the learning rate. As a result, Adadelta can potentially outperform AdaGrad in use cases where the latter fails to reach the desired solution [167; 168; 169]. Another variation of AdaGrad is RMSProp. The main idea behind RMSProp, which is also the key difference with AdaGrad, is that a weight's learning rate is divided by a running average of the magnitudes of the recent gradients of the aforementioned weight [170; 171]. In [172], the authors propose a variant of RMSProp called NRMSProp that utilizes the Nesterov Momentum. NRMSProp is capable of convergence quicker than RMSProp without adding too much complexity as depicted in the experiments that the authors carried out. Another variation of RMSProp is SMORMS3 [173]. SMORMS3 is particularly useful in Deep Neural Networks (DNNs), where the gradients have a high variance and it might prevent the learning rate from becoming too small, which could potentially slow down the optimization process [174]. Another optimizer that has also been proposed in the literature is Adam. Adam is a combination of AdaGrad and RMSProp and thus is able to work with sparse gradients and does not need a stationary objective [175]. It has very little memory requirements since it only requires first-order gradients. The authors that proposed Adam also proposed in the same manuscript AdaMax, which is a variant of Adam based on the infinity norm. Another variant of Adam is Adam with decoupled Weight Decay (AdamW). In AdamW a decay rate is introduced in order to insert a regularization based on the decay of weights during the process of the optimization [176]. In the literature, there have been specific studies that experiment on a set of optimizers to find the most appropriate ones for the corresponding use cases, whilst also taking the reduction in bias into consideration. More specifically, the authors in [177] propose an NN architecture to estimate crop water requirements. In order to minimize the bias of the approach, they test the SGD, RMSProp and Adam optimizers. According to their results, they select SGD since it increases the accuracy of the NN whilst it is also quite simple to implement. In a similar approach [178], where the authors aim to predict frosts in southern Brazil, they test SGD and Adam for their NN. They observe that experiments using the ADAM optimizer present greater variability and slightly lower accuracy than those using SGD. Adam and SGD are also compared in [179], where the authors develop an NN for human resources analytics and, more specifically, employee turnover. Based on the accuracy of the predictions, Adam is the optimal optimizer. In another approach [180], where electricity consumption needs to be predicted, the authors experiment with Adagrad, Adadelta, RMSProp, SGD, Nadam and Adam optimizers in their NN architecture. According to their results, for the given architecture, which is an LSTM, and for the given dataset, the SGD optimizer is the most suitable one. In [181], the authors aim to assess the Captcha vulnerability.