## 6 Discussion  
Bias, whether it originates from humans or the underlying mathematics of ML models, is a key factor in ML that actively affects the results of such models. The research community is aware of this influence, which is why several approaches have been proposed to try and mitigate it, as described in the previous sections. What is more, there also exist some initiatives, such as the AI Fairness 360 [245], Fairlearn [246] and Fairkit-learn [247] tools that aim to provide a more complete technical framework for mitigating bias when training ML models, thus promoting AI fairness. These kinds of frameworks have showcased promising results in terms of bias mitigation [248]; however, they still have a lot of room for improvement, which is why their developers request the active engagement of researchers so that they can be further enhanced and improved [249]. Removing all types of bias entirely from the ML lifecycle is not an easy task and some would argue that it is impossible. The first and foremost concern when training and/or evaluating ML models is identifying the existence of bias and the corresponding types of bias that might occur in terms of the provided data, the models themselves, as well as the bias that can be introduced by ML engineers, unconsciously or not. This identification can be conducted either empirically by taking into consideration other similar approaches  
Figure 6: Action plan for reducing bias during the evaluation of ML models. and use cases or by utilizing algorithms that are capable of identifying bias, as the ones presented in this manuscript. Then, the corresponding countermeasures should be taken into consideration and applied in order to minimize the identified biases. To this end, this manuscript's scope is to serve as a reference point for ML engineers and everyone who is interested in developing, evaluating and/or utilizing ML models, which they can consult so that they can firstly identify bias and then debias their approaches. Based on the literature review that was undertaken, it appears that most of the current approaches mainly focus on the existence of bias in the training datasets (i.e., data bias) and claim to have applied some kind of corresponding bias mitigation technique, like the IPW adjustment technique, the Imitate algorithm and its derivatives. With regard to the bias caused by the ML models and especially the selection of the appropriate optimizer and regularization method, little attention is paid to this matter, whilst most of the proposed approaches do not take into consideration that the choice of an optimizer and regularization method can also introduce bias in the model, as mentioned in Section 4.2. Overall, from all the research documents that were analyzed in the context of this manuscript, 39 papers analyzed methods and techniques for identifying and mitigating data bias, 21 comparative studies highlighted the importance of selecting the proper estimator to avoid the introduction of further bias and 24 comparative studies analyzed the way that a regularization technique may assist in reducing algorithmic bias. With regard to the way that an optimizer can affect a model in terms of bias and how an ML engineer can introduce bias, only nine and three studies were identified in the literature, respectively, as shown in Figure 7.  
The latter signifies that future research approaches should aim to (i) better justify the selection of a specific optimizer and how it may minimize bias in contrast to other available optimizers and (ii) evaluate their proposed approaches based on the appropriate metrics and offer transparency to their trained models by allowing the evaluation from third-party evaluators and reviewers. As for other literature reviews that talk about bias in machine learning, they mostly focus on data bias as well. More specifically, in [250] the authors mainly focus on selection bias and bias caused by imbalanced data, whilst they present the most common techniques to address these types of biases.