# Detection and Evaluation of Machine Learning Bias
## 1 Introduction  
Machine learning bias has garnered researchers' attention lately [1; 2; 3; 4]. Researchers are mostly concerned about the potential bias that machine learning systems may demonstrate against protected attributes such as: gender, race, age, etc. [5]. The interest in this area was initiated by a report published by ProPublica.com [1] that examined a judicial risk assessment; researchers found that the system exhibits bias toward black people. Later on, many other news reports and research papers have raised the concern of bias in data. Another more recent example of machine learning bias is Amazon's hiring system. The system did not like women a Reuters report claims ([[https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1) MK08G]([https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1) MK08G), accessed on 1 February 2021). The report stated that the hiring system was fed ten years of hiring data experience from the company. However, the system gave less weight to the CVs that included women indicators. The aforementioned typical machine learning examples work by extracting patterns from the training data. Due to the fact that data is collected from historical humans' practices, the element of bias is prevalent within it. For instance, intelligent hiring systems learn the behavior from the hiring practices embedded in the training data fed to the model. It is a natural phenomenon that humans have a cognitive bias naturally existing in their judgment and decisions. Human cognitive bias is a well-studied fact in the psychology field [6; 7]. In order to make a decision, the enormous amount of information residing in a human's brain is filtered, and only relevant information is used for decision-making. In our quotiian hiring decisions, generally a gender bias is observed, where usually a male employee is preferred over a female one. For instance, technical positions atAmazons are male dominated, as mentioned by the same report. Therefore, the company's historical data fed to the intelligent hiring system as training data is biased toward male candidates. A gender bias is observed in many other disciplines; thus the datasets also follow a biased pattern in which the data is more inclined toward a certain gender [8; 9; 10; 11]. Similarly, the recidivism prediction in the judicial system extracts convicted data from a set of 137 questions; then, an equation is applied to score the likelihood of a convict to commit another crime. The equations consist of some parameters that are weighted in association with each question. The parameters of the equation are either obtained by training the model using real-world data or chosen by a domain expert. In either case, it contains bias introduced by humans. Since the dawn of artificial intelligence, we have deliberated to build machines that mimic humans' ability to think and behave [12]. As consequence, we usually set our intelligence and cognitive abilities as the bar for machine intelligence in spite of the fact that humans suffer a cognitive bias. The machine learning models are trained using the data that represents human behavior [13; 14]. The data fed to the machine learning models for training is considered as the _ground truth_, enabling the model to learn behavior from this data. Nevertheless, considering the bias in human nature and cognitive thinking, the training data also depicts such biased characteristics, which ultimately impact the model's learning by inducing bias [15]. Objecting to this bias, Amazon shut the hiring system off to stop it.