What I Learned Exploring Bias in My Company’s Technology
By: Maxwell Wulff
Tags: Startup, AI, Artificial Intelligence, Bias, Entrepreneurship

What underlies bias in machine learning (ML)

Anu began in May of 2020 after my co-founder and I finished graduate school. When starting Anu I was excited to bring tech I learned about in my program to the business world. My background is in data science (primarily in the field of manufacturing), and I went to graduate school to focus on the forefront of research in machine learning at the time. I built on my prior knowledge of basic ML techniques and was able to dive into deep learning, neural networks, and all of their fascinating applications. Some of my favorites include deep fakes [4], adversarial neural nets made to confuse image recognition models [5], and speech/language recognition.

Speech, intention analysis, and grammar are processes that we take for granted on a daily basis. No matter what language you speak, language is hard-wired and second nature to humans [6]. Frankly, we tend to overlook how complex and intricate language actually is. Think about negations, sarcasm, regional dialect, and even just plain old confusing words. Set has 430 unique definitions and senses in the English dictionary, and clip means both to take something apart and put it back together. Here’s a fun thread of more ways the English language can be weird.

Challenge of Understanding Sentence Meaning. From Bored Panda.

How can we expect machines that rely on logic to possibly understand someone who doesn’t have to think twice about what they’re saying?

Well, to put it briefly, we need incredibly complex, fascinating statistical models that take into account word order, distance from other words in a sentence, dependency trees, parts of speech, and much more. Here are some great Medium articles that explain NLP from a high level [7, 8]. It’s important for this discussion to get a sense of how complex these models are: they can have hundreds of thousands of parameters and take days to train [9, 10].

Bias variance trade-off

It’s also important to understand the Bias-Variance Tradeoff. The Bias-Variance Tradeoff is one of the most fundamental principles of data science, governing basically how all models are trained. It states that as a model increases in complexity and is more finely tuned on one set of data, it tends to do worse on the data the model isn’t trained to. As the variance (variability of a prediction given a data point) goes down, its bias (distance a model is from its training data) will increase. Striking a balance between a model that is complex enough to capture the nuance of your data while being simple enough to be widely applicable is at the heart of data science [11].

Example of Bias-Variance Tradeoff. Graphics by Author.

In order to get passable results for an NLP task, the models need to be incredibly finely tuned to the data they’re trained to, with an enormous number of parameters. A model trained with this specificity to a certain dataset can start to have complications when deployed in an uncontrolled, uncertain setting with data points that are unlike what the model has seen.

Messiness of language data

This really begins to be a problem when we talk about just how messy real-world language data is. When these models are trained, they need pristine, tightly controlled, processed, and clean data. This is why a good portion of getting good results on an NLP task isn’t actually the architecture and training of the model, but preprocessing the data. We need a dataset that keeps as much of the underlying meaning while making it manageable enough to input into a model and train in a reasonable amount of time [12, 13].

For example, in graduate school I had some tasks with 100,000+ unique words in a data’s corpus that I had to clean and cut down to <10,000 tokens to pass into my models. While big companies and research groups are training on machines tens of thousands of times more powerful than my Macbook Pro [14], this problem scales. Just think about training on the 55 million Wikipedia articles, 184 million Yelp reviews, or 200 billion new Tweets every year.

Example of Raw Language Data

Making a good language dataset can take hundreds of hours of tedious person-power. Due to this barrier, when a dataset is created it is often shared and used thousands of times for widely-varying tasks. This puts an incredible amount of power on these few datasets to shape machine understanding of human language [15, 16, 17]. Check out some of the most common here.