Biased Algorithms and the False Sense of Democratization
Artificial intelligence is biased. That’s a huge problem!
– Carmen Aguilar y Wedge, Interaction/Experience Engineer, Hyphen-Labs
Category: Artificial Intelligence
Author: Making a New Reality

Biased Algorithms and the False Sense of Democratization
Artificial intelligence is biased. That’s a huge problem!
– Carmen Aguilar y Wedge, Interaction/Experience Engineer, Hyphen-Labs
In 2014, Sundance Institute’s New Frontier program collaborated with MIT Media Lab’s Social Computing Group to embed seven storytellers in the research group over a year to explore data as a storytelling medium. One of the Fellows was journalist Lisa Biagotti. She spent her month-long residency asking computer scientists, “What’s an algorithm?”
This seemingly simple question helped her extract answers that gave her a fairly profound epiphany. The data produced by algorithms, which she relied on as objective truth for her journalistic reporting, was actually subjective. She realized that in terms of defining truth, algorithms are just as problematic as framing a photograph to include and exclude key subjects. What you leave out of the frame of the photo is just as important to a story’s truth as what you leave in. How you frame the question the algorithm answers can bias the resulting data. MIT graduate student Joy Buolamwini articulates the issue as the “Coded Gaze” and delivers an excellent talk about the problem via a TED Talk.
MIT graduate student Joy Buolamwini
An often-used example of biased algorithms was described in the New York Times in 2015 by journalist Claire Cain Miller: “Google’s online advertising system, for instance, showed an ad for high-income jobs to men much more often than it showed the ad to women, a new study by Carnegie Mellon University researchers found. Research from Harvard University found that ads for arrest records were significantly more likely to show up on searches for distinctively black names or a historically black fraternity. The Federal Trade Commission said advertisers are able to target people who live in low-income neighborhoods with high-interest loans. Research from the University of Washington found that a Google Images search for ‘C.E.O.’ produced 11 percent women, even though 27 percent of United States chief executives are women. (On a recent search, the first picture of a woman to appear, on the second page, was the C.E.O. Barbie doll.) Image search results determined 7 percent of viewers’ subsequent opinions about how many men or women worked in a field, it found.”
This was a top area that interviewees identified as problematic due to a failure by the tech and media industries to bring diverse groups of people, coming from varied fields of knowledge, into the discourse around these designs. Could breaking the silos of knowledge and discipline help to mitigate these kinds of egregious errors?
Another highly concerning example illuminates how algorithms are disproportionately affecting people of color in the US criminal justice system. As Harry Armstrong and Jared Robert Keller report in The Guardian, “In the U.S., criminal ‘risk assessments’ based on predictive analytics have already been shown to be biased against black people because the data used to build the system was inherently biased.”
Other major smart (or not so smart) algorithm failures include facial recognition software that failed to understand the diversity of humans, such as when Google’s code labeled black faces as “gorillas,” and assumed Asian faces had blinked during photo, when their eyes were, in fact, open.
To be fair, the cases of outright maliciously coded bias algorithms (or the robots they animate) is probably a small percentage. Most of the bias being discovered is a surprise to the coders. Part of the issue is a lack of different perspectives among the pool of people coding these algorithms, making use of such technologies vulnerable to blind spots. An example of this is Google failing to code the automatic orientation of YouTube videos on smartphone for left-handed people, because they happen to have only right-handed people on the team. Part of the issue is that technologists are tasked with teaching very complex concepts to machines that operate on neural networks, computer infrastructure modeled on the human brain and nervous system, that are impossible for the technologists to understand in human scale.
Forbes reports that “Brian Brackeen, CEO of facial recognition company Kairos, says that machines can make culturally inappropriate assumptions when not properly trained. ‘It’s scarily similar to how a child learns.’ ”
That may be the issue in a nutshell, especially when these increasingly intelligent algorithms are learning from the problematic behavior of humans. For example, Microsoft’s chatbot learned to be racist and sexist from Twitter within 24 hours. Are biased algorithms just a product of their environment?
Then there are more nuanced conversations to be had about reconciling different value systems and the gray area between protection and censorship. Christopher Trout reports in Engadget: “A 2006 report on internet filtering from NYU’s Brennan Center for Justice referred to early keyword filters and their AI successors as ‘powerful, often irrational, censorship tools.’ Filters force the complex and infinitely variable phenomenon known as human expression into deceptively simple categories, the report continued. ‘They reduce the value and meaning of expression to isolated words and phrases. An inevitable consequence is that they frustrate and restrict research into health, science, politics, the arts, and many other areas.’ The report found that popular filters inexplicably blocked sites belonging to Boing Boing, GLAAD, photographer Robert Mapplethorpe, and Super Bowl XXX, among others, and often reflected the political and social prejudices of their creators.”
Another concern about how algorithms operate is the filter bubbles or echo chambers they create (Blame Facebook for echo chambers. But blame yourself, too). We are creating billions of terabytes of data through our digital activity, so we need smart algorithms to help us sort through all the data and make it meaningful or useful. Designers write algorithms that create UX/IU features, such as autofill, or trending topics suggestions (“if you liked X, you’ll like Y”) from sponsored advertising and entertainment outlets. The problem with these features is they quickly skew the results or choices to the first things that were searched, leaving users ignorant of the broader spectrum of possibilities. They do not authentically represent what is popular, or of interest to people.
As Dave Gershgorn writes in QZ.com, “Algorithms are unseen forces that dictate much of our life on the internet: Google search results, Facebook, and even spam filtering in email. And these algorithms are getting smarter thanks to improving artificial intelligence. To do so, they need to learn from more and more information. More often than not, the data these machines learn from comes from humans. Google’s algorithms, for example, learn to recognize speech by listening to tons of Google voice searches performed by humans.”
Some observers have pointed out that the technology is not fundamentally the problem; it’s that the technology exponentially amplifies actual human behavior. We tend to flock together or seek those that are most like us. Although many have used the power of the internet to break out of their comfort zones, the internet also enables social silos.
“I am, in general, pretty bold and unapologetic in making big claims about what the future is going to bring,” says Adam Huttler of the Exponential Creativity Fund. “There are some areas, though, where I have become cautious about predicting where things are going to head. So, as an example, I’ve been a loud evangelist for a long time, 20 years now, for the promise of the internet for lowering various barriers to entry, [elevating] new voices. What we’re seeing right now is the very negative, unintended consequences of that. It never even occurred to me that we’d end up in these little, tiny, niche echo chambers where extreme voices are allowed to resonate indefinitely, whether they’re factually accurate or not.”
Luke DuBois of NYU’s Brooklyn Experimental Media Center provided an example of this dynamic. “Facebook is the worst at this, right? So there was this Wall Street Journal project (Blue Feed, Red Feed) where they were side-by-siding the news feeds of folks who would self identify as liberal and folks who would self identify as conservative…and it’s like, literally, zero overlap.”
Blue Feed, Red Feed. See Liberal Facebook and Conservative Facebook, Side by Side — Wall Street Journal
Psychologist Michal Kosinski created a system to analyze people using their Facebook activity. This information can be used in the other way to find people who fit certain descriptions, and target them with advertisements. This ability to reach people inside of their siloed echo chambers has had an impact on politics, as we’ve seen with the 2017 presidential election, in which both candidates and bots micro-targeted voters. Companies such as NationBuilder have made tools that allow campaigns to reach highly targeted audiences. While NationBuilder was created with the intention of boosting the success of grassroots campaigns, its low price point and accessibility drew in the Trump administration.
Streaming services such as Netflix and Amazon are also contributing to the narrowing of our worldview, because their algorithms designed to feature content that matches the previous content the user consumed. Additionally, media executives are adopting the decision-making model of: “talent with audience” and “subject on trend.” This means they will greenlight a production only if it has attached talent with heavy followers on social media and if the subject is trending on Google Trends. So, beyond the filtering algorithm (and a base of human micro-taggers), the content itself will be constructed by narrow parameters of what is already being consumed. For the media executive it’s a way to optimize revenue growth, but executives are setting the industry up for eventual backlash and the dumbing down of the audience’s worldview in the process.
Interviewees commented on these negative dynamics:
So, rather than being exposed to a real diversity of content through what could be real “democratic media,” companies and institutions are increasingly segmenting what we have access to. I think that’s a real danger to democracy, free expression, and self-determination, because all these mediums have the potential to allow a greater diversity of people to participate…demographic diversity, but also stylistic and aesthetic diversity. — Michael Premo, Multidisciplinary artist
The web was supposed to be a democratic place where everyone could make great things and be seen. The reality is that the web is increasingly controlled by corporations that help guide people to where the good content is. So, yes, you can have an interactive documentary that lives at its own URL, but you’re less likely for that to be seen if it doesn’t feed into something else, like the Facebook ecosystem. — Adnaan Wasey, former Executive Producer, POV Digital
With Netflix’s $6 billion being spent on TV content in 2017, we are seeing huge competition in the streaming television space — with much of it being driven by machine-driven algorithms that analyze and deliver your preferences. This is worrisome, given that it begs the question as to whether there should be humans curating these cultural decisions. The “big four” tech platforms (Google, Apple, Facebook, and Amazon) are all leading the charge because they have massive cloud computing capabilities, which gives them access to artificial intelligence, and, with it, the ability to engage in sophisticated data analytics, presumably to deliver what consumers want. However, at the end of the day, maybe we need to return to human intervention and curatorship of our shared cultural knowledge. — Denise Mann, Professor and head of the UCLA School of Theater, Film and Television’s Producers Program
Some are hopeful that we will learn to mitigate current issues such as biased algorithms. In fact, Jack Dorsey’s recent call for support from the Twitter community to help them design a healthier and less toxic environment was well received by those who have been calling for reform. Some think we might even be able to train AI to create “a reduction in discrimination, overall, since AI programs are inherently more easily audited than humans.” However, the current Trump administration is asserting AI disruption is 50 to 100 years away, which technology insiders such as Jeff Greene call “out of touch with reality.”
These issues have become an important part of the discourse around the future of code and its impact on our world. But as many interviewees expressed, the conversation is not as broad and urgent as it needs to be to resolve the issues. Interviewees were particularly concerned when they considered that learning algorithms will have access to data about almost every aspect of our daily life with the rise of Internet of Things and facial or DNA analysis tools, which can use your physical participation in the world to gather data to inform fundamental decisions about your financial services, loans, employment, and rental applications. We could get to Black Mirror’s dystopian view of our future quickly.
The Making a New Reality research project is authored by Kamal Sinclair with support from Ford Foundation JustFilms and supplemental support from the Sundance Institute. Learn more about the goals and methods of this research, who produced it, and the interviewees whose insights inform the analysis.