Bias in Natural Language Processing (NLP)
Source: tai_blog

The rising popularity of natural language processing (NLP) and machine learning technologies underscores the importance of recognizing their role in shaping societal biases and stereotypes. While NLP applications have achieved success in modeling tasks like sentiment analysis  machine translation  and text summarization  these models can perpetuate societal biases present in their training datasets. These biases pose significant threats to equity  justice  and democracy.   In this article  we will discuss how NLP reflects and amplifies societal prejudices  explore their consequences  and outline steps that companies providing natural language services need to take toward mitigating them.   Bias is favoring one person  thing  or group over another unfairly. Bias is not programmed into natural language processing models; instead  it implicitly creeps into the model through the statistical patterns in language data it learns from. The training data may incorporate societal prejudices and derogatory stereotypes  such as racism  sexism  and ableism.   These biases can be perpetuated in various NLP applications  including word embeddings and downstream applications  such as sentiment analysis  job candidate screening  university admissions  and essay grading.   Biases in Word EmbeddingsDevelopers use unsupervised learning to prepare data for NLP models. Specifically  unsupervised models transform raw text data into word embeddings (numerical representations of text data) fed into NLP models. These models analyze massive amounts of text data  such as websites  social media  and books to create vectors that capture a words meaning and its relationship to other words.   However  while searching for hidden patterns in text data  these models are exposed to more than just semantic information  they are subjected to societal biases present in the data. These biases can then be embedded into word embeddings and inherited by supervised models  leading to biased outputs.   For example  sentences in an article might associate words related to doctors  engineers  and scientists mostly with men  while females may be portrayed as nurses  homemakers  or social workers.   Types of Bias in Natural Language Processing ServicesHere are common biases in natural language processing services:   Gender BiasGender bias is a significant and widespread issue in NLP models. Many reports show bias in advanced language models  such as GPT-3  where word embeddings tend to associate men with competency and occupations requiring higher education (doctors  lawyers  CEOs  etc.) in downstream NLP tasks. Whereas  in response to the prompt What gender does a nurse belong to?  it is more likely to output Its female.   Research published in The Artificial Intelligence and Emerging Technology Initiative of The Brookings Institution highlights numerous examples of gender bias in language applications using machine learning. Researchers found that NLP models working with word embeddings picked up biases based on how words are connected in the training data.   For example  words like kitchen and art were more frequently used with the word woman  and words like science and technology appeared in sentences including the word man. Such gender bias embedded in NLP systems leads to biased output.   Racial BiasNLP systems have also displayed racial bias. A 2017 Princeton study discovered that online prejudices against African Americans and the Back community were reflected by model embeddings. As per the study  historically  Black names were more significantly associated with negative words as compared to traditional White names  reflecting real-world prejudices present in training data.   Such racial bias in machine learning extends back even further. The study also mentioned 2004 research that found similar bias in resume assessment done through machine learning algorithms.   Moreover  word embeddings display the most substantial bias for words or phrases representing people with intersectional identities  such as race and gender  relative to other word combinations. For example  the representation of phrases like African American women or Mexican American women can be more negatively biased than just African American or woman alone.   Many AI algorithms creating word embeddings are trained on datasets that reflect the current social order  which can lack diversity and be biased towards certain groups. Due to a lack of diversity  the data used to train word embeddings likely has more information about white men. As a result  other social groups are primarily represented as minorities within the system.   Bias in downstream NLP applications  such as automated resume screening  might not only reflect existing biases but amplify them in society  impacting future generations by limiting their career opportunities.   How to Mitigate Biases in Natural Language Processing   While bias in natural language processing can be handled by debiasing the dataset early on or the model afterward  the ideal approach is to derbies the dataset to prevent the model from learning biased patterns. Here are some effective strategies to derbies natural language processing models:   Data ManipulationAs described earlier  the main reason for bias in natural language processing algorithms is unbalanced original datasets  i.e.  more text associating words related to doctors with male and words nurses with female. With this type of association  the NLP model is more likely to predict male for doctors. To address bias  it is essential to have a balanced dataset where all groups are represented similarly for the model to learn from.   For example  data augmentation algorithms such as SMOTE (Synthetic Minority Oversampling Technique) can be employed to create synthetic data points for the minority group (female doctors) in the dataset. Alternatively  one can choose to remove some data points from the majority group to make the dataset balanced.   Bias Fine-TuningThe bias fine-tuning method leverages the concept of transfer learning. It involves fine-tuning a relatively unbiased pre-trained natural language processing model on a new  more biased dataset. This enables the model to adapt to the specific task requirements of the biased dataset without inheriting biases from that data. Research suggests this method can achieve an accuracy score very similar to the model directly trained on unbiased data.   Data AnnotationData annotation is a crucial step in NLP model development  especially in addressing bias. It involves labeling and categorizing text data to train NLP models. Annotators can flag potentially biased datasets. Biases can include stereotypes  unequal representation of races or genders  or even culturally insensitive language. As a result  developers can take steps to mitigate the bias  such as collecting more balanced data and eliminating biased text data.   Diversity in Developing and Auditing NLP ModelsOther than training data  the bias can emerge from the team developing the model. A study at the 2020 NeurIPS machine learning conference suggested a negative correlation between the level of diversity of the development team and biased NLP models. According to a Brookings Institution study  a diverse AI audit team is essential for the ethical development of machine learning technologies.   A diverse audit group can test the model from various perspectives and help identify and mitigate potential bias throughout the NLP model creation process.   ConclusionThe growing presence of NLP services in our daily lives deepens concerns about bias in their algorithms. These sociotechnical systems absorb human biases and accurately ingest them as they learn from the training language data. This necessitates that development companies take bias mitigation steps to prevent the spread of discrimination further through these technologies.