How Deep Neural Networks Learn Gender Bias
Tags: Gender Bias, Bias, NLP, Ai Ethics, Deep Learning

Written by Davit Soselia & Shota Amashukeli — December 25, 2020

Bias in translation, 2020 (colorized)

Machine learning is being used to automate an increasing number of tasks, with many AI startups reaching a more mature phase of development as seen by the example of industry leaders such as DeepMind. However, a major issue that we should keep in mind in this process of automation is dealing with biases such as those related to gender, race, ethnicity, etc. held by deep learning models. The model repeatedly applies said biases when used for automation, thus having significant potential for societal harm. This has motivated research in the field, with significant insights gained in the last few years. In this post, we go through Investigating Gender Bias in Language Models Using Causal Mediation Analysis, presented at NeurIPS 2020 that might help us better understand the structural intricacies of bias in black-box models.

The paper expands on work by Zhao and Hastie (2019) which demonstrated the feasibility of using mediation analysis for providing insights both in the structure and behavior of ML models.

Before we delve into the gender bias related results, it is a good idea to go over the methodology behind it.

Causal Meditation Analysis

While other approaches can be used to estimate correlations between inputs and outputs, they provide little information in regards to causation. To better estimate this the authors use casual mediation analysis which relies on examining both direct and indirect effects that certain component has on the overall output.

Note: the image is from arXiv preprint, which in our opinion overall contains much better descriptions of the results being 55 pages long

This is crucial since components can have a significant independent impact. Indirect effects can be compared to aspirin in a situation where the side-effect of an original drug during the treatment causes the patient to take aspirin, which itself has effects on the patient’s health.

Causal mediation analysis relies on a change in input to give meaningful insights. To measure the level of bias, stereotypical and anti-stereotypical examples can be used. In the case of gender bias estimation, stereotypical refers to gender commonly associated with a certain word or profession by the model, and intervention is done by replacing the term with its anti-stereotypical alternative.

Example of Stereotypical and Anti-stereotypical continuations of the sentence, for professions datasets used in the paper, the stereotypical labels are identified using crowdsourcing.

After the intervention on the input direct and indirect effects of each model component is measured. To achieve this, for each model component the probability of Anti-stereotypical gender over the probability of the stereotypical gender based on the input sentence is measured and it is compared to the same values but with gender-biased word replaced by anti-stereotypical word.

To demonstrate with an example “The nurse said that” input will be replaced with “The man said that” and total change will be calculated as follows:

As mentioned before, we want to measure both the direct and indirect effects of the change. To calculate the direct effect we freeze the output of the selected network component while the rest of the network is updated using the gender-swapped input “the man said that”. While to measure the indirect effect only the component’s output is updated to what it would be in the “the man said that” scenario.

While the above example highlights the change in neurons, the authors also measure the impact of attention heads. For example possible interpretations in the following sentence continuation:

The CEO ordered the cleaner out of the room because she / he

Datasets used

In order to assess the gender biases present in ML models specialized datasets had been previously created.

An example from Winobias datasets

The experiments in this paper are conducted using the following datasets:

Employment demographics by Labor Force Statistics, that might contribute to gender bias in training datasets

Winobias — is used for assessing gender bias in pronoun resolution. Sentences have structure: [entity1] [interacts with] [entity2] [conjunction] [pronoun] [circumstances] as seen in the above image, where in the sentence pair solid line means stereotypical scenario and dashed line means non-stereotypical one.

Winogender — Follows the same structure as Winobias and provides pairs of sentences with different pronouns.

Professions — Was compiled by the authors of the paper and has the following pattern: The [occupation] [verb] because. Each profession also has a rating assigned from -1 to 1 for definitionality and stereotypicality. Definionality being how clearly the word itself describes the gender of the associated profession and stereotypicality being how much the term is influenced by bias.

Results

Calculating the total effects showed that random models are affected much less by gender bias than trained models. This might seem obvious, but it simply shows that the training process introduces gender bias to the model and it is not inherent to its structure. In addition, larger models are also affected more by the bias, due to the fact that they can learn provided datasets better.

Observing the effects of attention heads shows that attention heads on the middle layers have the highest indirect effect.

Indirect effects in GPT2-small on Winobias for attention heads

The paper also found that certain groups of attention heads which seem to specialize in capturing different relationships. For example, on the plot below we can see that Head 5–10(Orange) directed from she attends towards cleaner which would be a stereotypical candidate. Head 4–6 (Green) attends towards anti-stereotypical option, CEO and Head 5–8(Blue) simply attends to the following word.

Observing the indirect effect on neurons showed that unlike attention heads, the neurons on the initial layers of the model had the strongest indirect effect.

Indirect effect per layer for GPT2 variations.

Final Thoughts

We got important insights from the paper regarding gender bias prevalent in language models of various sizes. It is intriguing to see that specific model heads can be associated with stereotypical and anti-stereotypical candidates. It would be interesting to see if this holds true for other types of biases in the model. It also reopens the question of how to address undesired biases, and whether they are better handled at data processing or model training parts of the machine learning deployment pipeline.

References

Investigating Gender Bias in Language Models Using Causal Mediation Analysis — https://papers.nips.cc/paper/2020/file/92650b2e92217715fe312e6fa7b90d82-Paper.pdf

Gender Bias in Contextualized Word Embedding — https://www.aclweb.org/anthology/P19-2031/

WinoBias Dataset — https://uclanlp.github.io/corefBias/overview