# Detection and Evaluation of Machine Learning Bias
## 6 Experiment Setup
### Experimental Findings  
Figures 2 and 3 illustrate the results of an experiment to predict the wage. We conducted the experiment as explained earlier using the 10-folds cross-validation model selection. The plots show the results of the average predicted wage for both male and female beside the average predicted wage when we apply the proposed alternation function on the gender attribute. The plot on the left, Figure 2, shows the female average predicted wage in green circle markers. The results show a considerable improvement in the predicted wage after applying alternation on the gender attribute. This indicates the prediction of females' wage increase if their identity changed to male. In other words, a male instance with the same profile would earn more wage than a female instance. This indicates the presence of bias against females. The plot on the right-hand side, Figure 3, shows the average predicted wage for males in green, which degraded substantially after applying the alternation function. In contrast to the female results above, this indicates bias with males. In summary, we can say that the prediction of the wage is obviously influenced by the gender attribute. Both figures show the bias against females. In other words, one profile for one person could be treated differently if we only change the gender. In real-life scenarios, we usually tend to consider this as a kind of bias. In this experiment, what we have done thus far is simply saying that if this person with the same profile was a male, the wage would be higher than if the person was a female. To quantify this bias, we evaluate the KL divergence between the original and the alternative prediction mean. Figure 4 depicts the KL divergence of the results presented previously. The red line represents the KL divergence of the results in Figure 2, which is the divergence between the predicted wage of the original dataset and the predicted wage when changing female into male. We can clearly observe bias against females. In contrast, the green line indicates less bias against males. This is the KL divergence between the predicted wage for females and the predicted wage for females after applying alternation. The green line, on the other hand, is the bias against males. The bias against females is significant in these results. At the moment, we can see the decrease in the wage if the gender is female. Let us now consider the other controversial attribute (i.e., _race_). We have five different races in the dataset; namely: (a) American Indian, Aleut, or Eskimo, (b) Asian or Pacific Islander, (c) Black, (d) Other, and (c) White. We will use the word Indian and Asian to represent the races in (a) and (b), respectively. We used the same methodology with the race attribute. The results are demonstrated in Figures 5-7. Carefully analyzing each plot, we observe interesting results. Based on Figure 5; plots: Asian/Indian Alternation and White/Indian Alternation, we cannot claim any kind of bias on or against the American Indian race. In all cases, it gives better wage prediction when we change the race from or to American Indian. We find this behavior to be quite confusing; however, this needs further investigation and should be considered when building a real-world model. Similarly, we did not notice a clear bias against Asian or Pacific races (see Figure 5; plots: Asian/Other Alternation and White/Asian Alternation), except with the black race. Figure 3: Male to female alternation. The wages of males (in green) decrease when we change the gender attribute to a female (in blue). Figure 2: Female to male alternation. It shows that the female wage (in green) improves when the gender is changed to male (in blue). There is a bias against Asians when we change race to black. In other words, when we change the race of an Asian instance to a black, the wage noticeably gets higher. Likewise, when we change the race from black to Asian, the wage gets lower. We believe this is a kind of bias.