Understanding Racial Bias in Machine Learning Algorithms
By: The Educative Team
Tags: Programming, AI, Data Science, Racial Justice, Machine Learning

Racial Bias in Machine Learning and Artificial Intelligence

Machine learning uses algorithms to receive inputs, organize data, and predict outputs within predetermined ranges and patterns. It may seem like algorithms are objective, mathematical processes, but this is far from true. In fact, racial bias seeps into algorithms in several subtle and not-so-subtle ways, leading to discriminatory results and outcomes. Let’s take a deeper look.

The danger of automation

Algorithms can give you the results you want for the wrong reasons. By automating an algorithm, it often finds patterns you could not have predicted. This limitation of algorithms is well demonstrated by the legend of the neural net experiment.

Automation poses dangers when data is imperfect, messy, or biased. An algorithm might latch onto unimportant data and reinforce unintentional implicit biases. For example, data scientist Daphne Koller has explained that an algorithm designed to recognize fractures from X-rays instead ended up recognizing which hospital had generated the image. That algorithm now incorporates irrelevant data and skews results. If an algorithm is exposed to racially biased data sets, it will continue to incorporate those biases — even in a completely different context.

Hiring algorithms are especially vulnerable to racial bias due to automation. Human resources managers can’t wade through pools of applicants, so resume-scanning algorithms weed out about 72% of resumes before an HR employee reads them. Resume scanners are typically trained on past company successes, meaning that they inherit company biases.

In a well-known experiment, recruiters selected resumes with white-sounding names. By training an algorithm on that dataset, it learned to automatically filter out any black-sounding names. The algorithm selected candidates on purely subjective criteria, perpetuating racial discrimination. This same form of automated discrimination prevents people of color from getting access to employment, housing, and even student loans. Automation means we create blind spots and racist biases in our supposedly objective algorithms.

Improper training and proxies

Algorithms are trained with data sets and proxies. Inputs can be biased, so algorithms also become biased. Developers that train and test algorithms too often use data sets with poor representation of minorities. In fact, a commonly used dataset features content with 74% male faces and 83% white faces. If the source material is predominantly white, the results will be too.

This poses a significant problem for algorithms used in automatic demographic predictors and facial recognition software. Since facial recognition software is not trained on a wide range of minority faces, it misidentifies minorities based on a narrow range of features. In a 2015 scandal, Google’s facial recognition technology tagged two black American users as gorillas due to biased inputs and incomplete training.

In another example from 2018, a facial recognition tool used by law-enforcement misidentified 35% of dark-skinned women as men. The error rate for light-skinned men was only 0.8%. At a time when police brutality in the United States is at a peak, we can see how this biased data could lead to disastrous, and even violent, results.

Proxies also generate bias. A proxy is an assumption about the variables that we use to get particular results. For example, the Body Mass Index (BMI) is a proxy to label whether someone is healthy or unhealthy. We assume that BMI equates to health, so we categorize bodies according to that system, although the concept of BMI has in fact been widely debunked. If we assume a proxy is accurate, we assume the results are as well. This happens in machine learning.

A 2019 study revealed that a healthcare ML algorithm reduced the number of black patients identified for extra care by half. In fact, the risk score for any given health level was higher for white patients. Since the algorithm was trained on the proxy of healthcare costs, it assumed that healthcare costs serve as an indicator for health needs. However, black patients spend less on healthcare for a variety of racialized systemic and social reasons.

Without deeper investigation, the results may have led to the allocation of extra resources to white patients. But when the algorithm was altered to include more accurate markers of health risk, the numbers shifted: Black patients referred to care programs increased from 18% to 47% in all cases.

Example after example proves that machine learning training and proxies, even those created by well-intentioned developers, can lead to unexpected, harmful results that frequently discriminate against minorities.

AI bias is human bias

Algorithms are not truly neutral.

The notion that mathematics and science are purely objective is false.

In fact, throughout history science has been used to justify racist conclusions —from debunked phrenology even to the theory of evolution.

Algorithms are our opinions written in code. Since algorithms are designed, created, and trained by data scientists — people like you and me — machine learning technologies unintentionally inherent human biases. This means that our machines are in danger of inheriting any biases that we bring to the table.

If you aren’t convinced, read up on Microsoft’s Tay, an AI chatbot that spread disturbingly racist messages after being taught by users within a matter of hours.

We are the teachers — AI bias is human bias.

It shouldn’t surprise you that representation is a contributing factor to this issue. A majority of AI researchers are white males, in similar socioeconomic positions, from similar universities.

Studies from 2019 found that 80% of AI professors are men. People of color remain underrepresented in major tech companies. At a 2016 conference on AI, Timnit Gebru, a Google AI researcher, reported there were only six black people out of 8,500 attendees. This diversity crisis means that very few people of color are involved in machine learning decision-making or design. If innovators are homogenous, the results and innovations will be too, and we’ll continue to ignore a wider range of human experience.