# A Survey on Bias in Deep NLP
## 4 Discussion
### Association Tests
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**Year Ref.** & **Sereotypes** & **Model** & **Data** & **Lang.** & **Evaluation** & **Mitigation** & **Stage** & **Task** \\ \hline
2018 [72] & Gender, Ethnicity & - & **ERC**, Tweets (SemEval-2018) & English & Sentiment, Emotion of & - & - & Sentiment \\  & & & & & & Association & & & Scoring \\ \hline
2019 [63] & Gender & ELMo, GloVe & One Billion Word Benchmark, Windowus, OntoNotes 5.0 & English & PCA, Prediction Accuracy & Data Augmentation (1), & Train (1), & Conference \\  & & & & & & & Attribute Protection(gender) & After (2) & Resolution \\ \hline
2019 [67] & Gender, Ethnicity, Deshibits, Sexual & Google Perspective API & WikiDetox, Wiki Mathilos, Twitter, WordNet & English & Classification Accuracy, & Data correction, Data Augmentation, Attribute & Before & [PERSON] Speech \\  & & & & & & & Protection & Detection \\ \hline
2019 [73] & Gender & Google Translate API (1) & United Nations and European Parliament transcripts (1), & Estonian, & & Prediction accuracy - & - & Translation \\  & & & & & Translate Community (1), & Frisis, & & & \\  & & & & Occupation Data (BLs), & Hungarian, & & & & \\  & & & & & & & & \\  & & & & & & & & \\  & & & & & & & & \\  & & & & & & & & \\  & & & & & & & & \\  & & & & & & & & \\  & & & & & & & & \\  & & & & & & & & \\  & & & & & & & & \\  & & & & & & & & \\  & & & & & & & & \\  & & & & & & & & \\  & & & & & & & & \\  & & & & & & & & \\ \hline
2019 [74] & Race, Gender, Sexual & LSTM, BERT, GPT-2 (small), & One Billion Word Benchmark(4) & English & Sentiment Score & Train LSTM/BERT & Train & Test Generation \\  & & & & & & (VADE [75]), Classification & & & \\  & & & & & & & accuracy & \\ \hline
2019 [74] & Gender & Google Translate, Microsoft & - & English, & **WineMT** (Winoliss + & Positive Contextualization & After & Translation \\  & & & & & French, & WinGender), Prediction & & & \\  & & & & & Italian, & Accuracy & & \\  & & & & & & & \\  & & & & & & & \\  & & & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Previous work on bias detection and treatment in NLP: task-specific accuracy/scoring. \begin{table}
\begin{tabular}{l l l l l l l l l l} \hline \hline
**Year** & **Ref.** & **Soreotyped** & **Model** & **Data** & **Lang.** & **Evaluation** & **Mitigation** & **Stage** & **Task** \\ \hline
2019 [79] & Gender & ELMo & English-German news & English & cosine similarity, clustering, & - & - & - & - \\  & & VTM18 & & NNN & & & & \\ \hline
2019 [52] & Gender & CBOW & English & Giguynet, Wikipedia, & English & Analogies, WEAT, Sentiment & Hard-Delsiang, CDA, CDS & Train & - \\  & & Google analogy, SmLx\(\sim\)999 & Classification, Clustering & & & & & \\ \hline
2020 [47] & Gender & BERT & Equity Evaluation Corpus, & English & EEC, Gender Separability, & Vector Space Manipulation & Train & - \\  & & Gen-data & Emotion/Sentiment Scoring & & & & & \\ \hline
2020 [79] & Entity & CPT-2 (small), DISTLBERT & Twitter/AEAE/IBA2 & English & Text generation, BLEU, & - & - & - \\  & & Mechanical Turk- Annotates & (AAVE/SAE) & ROUGE, Sentiment & & & & \\  & & (SAE) & Classification, VADER [75] & & & & & \\ \hline
2020 [56] & Gender & CBOW & British Library Digital Corpus, & & Association, Prediction & - & - & - \\  & & The Guardian articles & & & & & & \\  & & & & & & & \\  & & & & & & & \\  & & & & & & & \\  & & & & & & & \\  & & & & & & & \\  & & & & & & & \\  & & & & & & & \\  & & & & & & & \\  & & & & & & & \\
2020 [81] & Gender, Race & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\  & & & & & & & \\  & & & & & & & \\ \hline
2020 [83] & Disability & BERT, Google Cloud & Jigsaw Unintended Bias & English & Sentiment Score & - & - & - & Toxicity prediction, Sentiment analysis. \\ \hline \hline \end{tabular}
\end{table} 
Table 6: _Cott._[PERSON] et al. [38] extend WEAT to measure the bias in a multi-class setting and use it over a Word2 Vec model trained with Reddit L2 corpus. [PERSON] et al.