## 4 Algorithm Bias
### Optimizers
They develop an NN architecture and experiment with Nadam, Adam and Adamax. They conclude that Nadam is the optimal optimizer since it achieves the highest accuracy. In a similar approach [182] for facial expression recognition, the authors test Adam and SGD for their implemented CNN. They conclude that Adam is the appropriate optimizer since it allows CNN to generalize better when dealing with unseen data. Both those two optimizers are also tested in [183], alongside Adagrad, and RMSProp in order to be used in an NN that is trained on diabetic data. In this case, RMSProp is the preferred optimizer since it achieves the highest accuracy in the least possible time. In a similar approach [184] that focuses on epilepsy detection, SGD is compared with RMSProp and it is shown that SGD achieves the highest accuracy. Lastly, in [185] the authors develop an LSTM NN for network-based anomaly detection and experiment with seven optimizers, namely Adam, Adagrad, Adamax, Frtl, Nadam, RMSProp and SGD. Their results define Nadam as the optimal optimizer since it achieves the highest accuracy. A complete list of comparative studies of optimizers that have been conducted and that also take bias into consideration can be found in Table 8.  
Based on the above table, it would be useful to compare the selected optimizer per approach and understand when each one of them is more appropriate to be used, thus minimizing the bias introduced into the model. In general, the Adam optimizer converges faster and has fewer memory requirements than RMSProp, whilst SGD converges slower. However, SGD converges to optimal solutions and is able to better generalize than Adam [186]. As for Nadam, which, as mentioned previously, is a variant of Adam, it converges slightly faster than Adam, thus resulting in less training time. The above-mentioned optimizers are also summarized in Table 9.  
Selecting the appropriate optimizer is not an easy or straightforward process, and excessive attention should be paid to this matter in order to minimize the bias that can \begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**ID** & **Year** & **Ref.** & **Domain** & **Data Category** & **Optimizers** & **Most Suitable Optimizer** \\ \hline OCS1 & 2021 & [177] & Agriculture & Numeric & SGD, RMSprop, Adam & SGD \\ \hline OCS2 & 2023 & [178] & Agriculture & Time Series Data & Adam, SGD & SGD \\ \hline OCS3 & 2024 & [179] & Business & Numeric & Adam, SGD & Adam \\ \hline OCS4 & 2022 & [180] & Civil & Numeric & Adagrad, Adadelta, RMSProp, SGD, Nadam, Adam & SGD \\ \hline OCS5 & 2021 & [181] & Computer Vision & Images & Nadam, Adam, Adamax & Nadam \\ \hline OCS6 & 2021 & [182] & Computer Vision & Images & Adam, SGD & Adam \\ \hline OCS7 & 2021 & [183] & Health (diabetes) & Numeric & SGD, Adagrad, Adam, RMSProp & RMSProp \\ \hline OCS8 & 2023 & [184] & Health & Numeric & SGD, RMSProp & SGD \\ \hline OCS9 & 2023 & [185] & Security & Numeric & Adam, Adagrad, Adamax, Frtl, Nadam, RMSProp, SGD & Nadam \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Complete list of comparative studies of optimizers. \begin{table}
\begin{tabular}{l l} \hline \hline
**Optimizer** & **Recommended Usage** \\ \hline Adam & Training time needs to be reduced \\ RMSProp & Memory requirements are not important \\ SGD & There is no time constraint \\ Nadam & Adam does not produce sufficient results \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Recommended usage of Adam, RMSProp, SGD and Nadam. be introduced. To achieve the above-mentioned goal, it is generally a good practice to experiment with a variety of optimizers based on the use case requirements, such as training time and computational resources, and monitor the training parameters to decide which one should be selected.