Detecting Independent Pronoun Bias with Partially-Synthetic Data Generation
Channel: Robert Monarch
Date: 2020-10-25

Robert (Munro) Monarch and Alex (Carmen) Morrison. Presentation for our paper at the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020).

You can read the paper here:
https://www.aclweb.org/anthology/2020.emnlp-main.157.pdf

The open source code to generate the results is here:
https://github.com/rmunro/masked_bias_detection

Abstract:
We report that state-of-the-art parsers consistently failed to identify "hers" and "theirs" as pronouns but identified the masculine equivalent "his". We find that the same biases exist in recent language models like BERT. While some of the bias comes from known sources, like training data with gender imbalances, we find that the bias is amplified in the language models and that linguistic differences between English pronouns that are not inherently biased can become biases in some machine learning models. We introduce a new technique for measuring bias in models, using Bayesian approximations to generate partially-synthetic data from the model itself.