## 4 Algorithm Bias
### Optimizers  
Aside from the selection of the appropriate estimator (i.e., algorithm), the utmost care should be taken when selecting optimizers and types of regularization, since all of them can contribute to biased algorithmic decisions, regardless of the absence or presence of bias in the input data [148]. Regarding optimizers, those are algorithms that are utilized in neural networks to change their attributes, such as weights and learning rate, thus minimizing the losses. There have been proposed several optimizers in the literature, from which the most widely used are Stochastic Gradient Descent (SGD) and its variant Stochastic Gradient Descent with Gradient Clipping (SGD with GC), Momentum and its variant Nesterov, Adan (Adaptive Nesterov) and AdaPlus, AdaGrad (Adaptive Gradient), its variation Adadelta, RMSProp (Root Mean Square Propagation) and its variants NRMSProp and SMORMS3 (Squared Mean Over Root Mean Squared Cubed), as well as Adam (Adaptive Moment Estimation) and its variants Nadam, AdaMax and AdamW (Adam with decoupled Weight Decay). SGD is used for unconstrained optimization problems. It is preferred in cases where there are requirements of low storage space and fast computation speed, and the data might be non-stationary or noisy [149]. However, SGD is one of the most basic optimizers [150]. Moreover, the selection of the learning rate is not an easy task and, if not tuned carefully, it might lead to not ensuring convergence [151]. In order to avoid the slow convergence issue of standard SGD, SGD with GC has been proposed [152]. The key difference between the two optimizers is that in the case of SGD with GC, the gradients are clipped if their norm exceeds a specified threshold, prior to updating the parameters of the model using (1). By doing so, the stability of SGD is enhanced, and convergence is ensured in more cases [153]. Momentum is a type of optimization technique that \"accelerates\" the training process of neural networks, being first studied in [154]. In contrast to the above-mentioned gradient descend optimizers, this technique does not directly update the weights of an ML model but rather introduces a new term named the \"momentum term\" that will update the weights by calculating the moving average of the gradients, thus guiding the search direction of the optimizer [155]. The momentum technique has been utilized in the SGD optimizer described above resulting in the variant of SGD with momentum. SGD with momentum works faster and generally performs better than SGD without momentum [156]. As mentioned above, a variant of the Momentum technique is the Nesterov Momentum [157]. In Nesterov Momentum, the main idea is that the \"momentum term\" mentioned previously can be utilized for the prediction of the next weights' location, thus allowing the optimizer to take larger steps while trying to ensure convergence [158]. A recently proposed optimizer that utilizes the Nesterov Momentum is presented in [159] and is named Adan. Adan has been tested in a variety of use cases, including Deep Learning (DL) related tasks such as NLP and CV, surpassing other currently available optimizers. The source code of the optimizer is available in [160]. Another optimizer that utilizes the Nesterov Momentum and combines the advantages of other optimizers such as AdamW and Nadam is called AdaPlus [161]. AdaPlus has been evaluated in CV-related tasks by being utilized in CNNsand being compared with other state-of-the-art optimizers, showcasing promising results. The source code of the AdaPlus can be found in [162]. Another optimizer that is widely used in ML and DL is the AdaGrad [163]. AdaGrad is used for gradient-based optimization and the main idea behind AdaGrad is that it adjusts the learning rate per feature, based on the feature's updates. If the feature is being updated often, this means that it frequently occurs in the dataset and the Adagrad optimizer assigns a smaller learning rate.