Evaluating Gender Bias in Natural Language Inference 
Authors: Shanya Sharma
Manan Dey
Koustuv Sinha
Venue:  (2021)
Keywords: Natural Language Inference
Natural Language Understanding
Natural Language Processing
Gender Bias
Societal Bias
Bias
Ethics
Debiasing Techniques
Data Augmentation

Abstract: Gender-bias stereotypes have recently raised significant ethical concerns in natural language processing. However, progress in the detection and evaluation of gender-bias in natural language understanding through inference is limited and requires further investigation. In this work, we propose an evaluation methodology to measure these biases by constructing a probe task that involves pairing a gender-neutral premise against a gender-specific hypothesis. We use our probe task to investigate state-of-the-art NLI models on the presence of gender stereotypes using occupations. Our findings suggest that three models (BERT, RoBERTa, and BART) trained on MNLI and SNLI data-sets are significantly prone to gender-induced prediction errors. We also find that debiasing techniques such as augmenting the training dataset to ensure that it is a gender-balanced dataset can help reduce such bias in certain cases.  

Under review as a conference paper at ICLR 2021
EVALUATING GENDER BIAS IN NATURAL LANGUAGE INFERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
Gender-bias stereotypes have recently raised significant ethical concerns in natu- ral language processing. However, progress in detection and evaluation of gender- bias in natural language understanding through inference is limited and requires further investigation. In this work, we propose an evaluation methodology to mea- sure these biases by constructing challenge task which involves pairing gender neutral premise against gender-specific hypothesis. We use our challenge task to investigate state-of-the-arr NI.I models on the presence of gender stereotypes using occupations. Our findings suggest that three models (BERT, RoBERTa, BART) trained on MNLI and SNLI data-sets are significantly prone to gender- induced prediction errors. We also find that debiasing techniques such aug- menting the training dataset to ensure a gender-balanced dataset can help reduce such bias in certain cases.
INTRODUCTION
Machine learning algorithms trained in natural language processing tasks have exhibited various forms of systemic racial and gender biases. These biases have been found exist many subtasks NLP, ranging from learned word embeddings (Bolukbasi et al. 2016 Brunet et al. 2019), natural language inference (He et 2019a), hate speech detection (Park et al. 2018), dialog (Henderson et al. 2018 Dinan et al.) 2019), and coreference resolution (Zhao et al. 2018b). This has prompted evaluate and mitigate them, either through removal of bias introduction in dataset level (Barbosa & Chen 2019), or through model architecture (Gonen & Goldberg 2019), or both (Zhou & Bansal 2020).
a large area of research attempting
Specifically, we revisit the notion of detecting gender-bias in Natural Language Inference (NLI) systems using targeted inspection. NLI task constitutes of the model to understand the inferential relations between pair of sentences (premise and hypothesis) to predict three-way classification on their entailment, contradiction or neutral relationship. NLI requires representational understand- ing between the given sentences, hence its critical for production for less to no perceivable stereotypical bias. Typically, NLI systems are trained on datasets collected using acrge-scale crowd-sourcing techniques, which has its own fair share of issues resulting in the introduction of lexical bias in the trained models (He et al., 2019b Clark et al. 2019). Gender bias, which loosely defined by stereotyping gender-related professions to gender-sensitve pronouns, have also been found to exist in many NLP tasks and datasets (Rudinger et al. 2017 2018).
models in this task account
With the advent of large-scale pre-trained language models, we have witnessed phenomenal rise of interest in adapting the tre-trained models to downstream applications in NLP, leading to superior performance (Devlin et al., 2019 Liu et al.) 2019 Lewis et al., 2019). These pre-trained models are typically trained over a massive corpus of text, increasing the probability of introduction of stereotypical bias in the representation space. is thus crucial to study how these models reflect the bias after fine-tuning the downstream task, and try to mitigate them without significant loss of performance.
The efficacy of pre-trained models the downstream task also raises the question in detecting and mitigating bias in NLP systems- - is the data or the model at fault?. Since we fine-tune these pre- trained models the downstream corpus, bias. Thus, it is imperative to revisit the question of detecting the bias from the final sentence repre- sentations. To that end, we propose challenge task methodology to detect stereotypical gender bias
no longer conclusively determine the source of the

---

Under review as a conference paper at ICLR 2021
in the representations learned by pre-trained language models after fine-tuning the natural lan- guage inference task. Specifically, we construct targeted sentences inspired from Yin et al. (2019), through which measure gender bias in the representation space in the lens of natural language ference. We evaluate a range of publicly available NLJ datasets (SNLI II (Bowman et al. 2015), MNLI (Williams al. 2018), ANLI (Nie et al. 2020) and QNLI (Rajpurkar& al. 20T6).) and pair them with pre-trained language models (BERT (Devlin et al. (2019), RoBERTa Liu et al. (2019) and BART (Lewis al. (2019)) ) to evalute their sensitivity gender bias. Using our challenge task, detect gender bias using the same task the language models are fine-tuned for (NLD). Our challenge task also highlights the direct effect and consequences of deploying these models by testing on the same downstream task, thereby achieving thorough test of generalization. We posit that biased NLI model that has learnt gender-based correlations during training will have varied prediction on two different hypothesis differing in gender specific connotations.
Furthermore, we use our challenge task to define a simple debiasing technique through data aug- mentation. Data augmentation have been shown to be remarkably effective in achieving robust generalization performance in computer vision (DeVries & Taylor 2017) as well as NLP (Andreas) 2020b). We investigate the extent to which augmenting the training set with our probe challenge examples. Concretely, our contributions in this paper are:
mitigate gender bias from the NLI models by
We propose an evaluation methodology by constructing a challenge task to demonstrate that gender bias is exhibited in state-of-the-art finetuned Transformer-based NLI model outputs (Section 3).
We test augmentation as an existing debiasing technique and understand its efficacy on various state-of-the-art NLI Models (Section 4). We find that this debiasing technique is effective in reducing stereotypical gender bias, and has negligible impact on model perfor- mance.
Our results suggest that the tested models reflect significant bias in their predictions. We also find that augmenting the beiiiingddttase in order ensure belaneed distribu- tion proves to be effective in reducing bias while also maintaining accuracy on the original dataset.
PROBLEM STATEMENT
MNLI (Williams et al. (208)) and SNLI (Bowman et al. (2015)) dataset can be represented as D < H,H,L>> > with p P as the premise, f H as the hypothesis and ( e L as the label (entailment, neutral, contradiction). These datasets are created by a crowdsourcing process where crowd-worken given the task come up with three sentences that entail, are neutral with, and contradict given sentence (premise) drawn from an existing corpus.
Can social stereotypes such as gender prejudice be passed on result of this process? To evaluate this, we design a challenge dataset D' < P, F. M > with p P as the premise and f F and m M as different hypotheses differing only in the gender they represent. We define gender-bias as the representation of learned by the model that results in a change the label when paired with f and m separately. A model trained to associate words with genders prone to incorrect predictions when tested on a distribution where such associations no longer exist.
In the next two sections, we discuss our evaluation and analysis in detail and also investigate the possibilities of mitigating such biases.
3 MEASURING GENDER BIAS
We create evaluation sets using sentences from publicly available NLI datasets. We then test them on 3 models trained on MNLI and SNLI datasets. We show that a change in gender represented by the hypothesis results in a difference in prediction, hence indicating bias.
2

---

Under review as a conference paper at ICLR 2021
3.1 DATASET
We evaluate the models on two evaluation sets: in-distribution I, where the premises p are picked from the dataset (MNLI (Williams et al. (2018)), SNLI (Bowman et al. (2015)) used to train the models and out-of-distribution O, where we draw premises p from NLI datasets which are unseen the trained model. For our experiments in this work we use ANLI (Nie al. (2020) and QNLI Rajpurkar et al. (2016)) for citoo--distrbbttion evaluation dataset creation. Each premise, in both and O, evaluated against two hypothesis f and m, generated using templates, each a gender counterpart of each other. Statistics of the datasets are shown in Table 1.
Dataset
# instances
Source of premise
In-distribution Evaluation Set (MNLI)
1900
MNLI original datasct
In-distribution Evaluation Set (SNLI)
1900
SNLI original datasct
Out-ol-distributi Evaluation Set (ANLI + QNLI)
3800
(ANLI QNLI) original dataset
Table 1: Statistics of evaluation sets designed for evaluating gender bias
Premise: To measure the bias, we select 38 different occupations to include a variety of gender distribution characteristics and occupation types, in correspondence with US Current Population Survey I (CPS) 2019 data and prior literature (Zhao et al. (201 8a)). The selected occupations range from being heavily dominated (with domination meaning greater than 70% share in a job distribu- tion) by a gender, e.g. nurse, to those which have an approximately equal divide, e.g. designer. The list of occupations considered can be found in Appendix (A.3).
From our source of premise, as mentioned in Table 1, we filter out examples mentioning these occupations. Next, remove examples that contain gender specific words like "man", "woman" "male", "female" etc. On our analysis, we found out that models were sensitive to names and that added the bias. Since, in this work, our focus is solely on the bias induced by profession, filtered out only the sentences that didn't include a name when checked through NLTK-NER (Bird al. 2009)).
We equalize the instances of the occupations so that the results are not because of the models' performance on only few dominant occupations. For this, we used examples from occupations with larger share in the dataset and used them as place-holders to generate more sentences for occupations with lesser contribution. Examples of this can be seen in Table 2.
Source Occupation
Original Premise
Target Occupation
Modified Premise
Nurse
A nurse is silling on a bench in the park.
Teacher
A teacher silling on bench in the purk.
Janitors
Janitors doing their job well.
Teacher
Teachers doing their well.
Carpenter
A carpenter is walking Lowards the store.
Baker
A baker is walking lowards the store.
Table 2: The source sentence acts as a placeholder and we replace the source occupation with the target occupation to generate new sentence. This is done augment our evaluation set to ensure equal number of premises for all 38 occupations.
Following this methodology we generated datasets consisting of equal number of sentences from each of the occupations to act as the premise of our evaluation sets. The sentences were intended to be short (max. 10 words) and grammatically simple to avoid the inclusion of other complex words that may affect the model's prediction. We verified that each sentence included is gender neutral and does not seek to incline to either male or female in itself.
Hypothesis: We use templates T to generate gender-specific hypothesis, as shown in Table 3. Here gender corresponds to male or female such as "This text talks about a female occupation") "This text talky about a male occupation". We focus on making the template sentences help discern the gender bias in the NLI models. We also vary the structure of these templates to ensure that the results are purely based on the bias and are not affected by the syntactic structure of the hypothesis. We consider a hypothesis pro-stereotyypcal if it aligns with society's 's stereotype for an occupation, e.g. "female nurse" and anti-stereotypical if otherwise.
Admittedly, more natural hypotheses can be created through crowd-sourcing, but in this work we generate only the baseline examples and we leave more explorations in this regard as a future work.
Labor Force Statistics from the Current Population Survey( http://www.bbl.ggwgpppppaaal 1.tm)
3

---

Under review as a conference paper at ICLR 2021
Hypothesis Templates
This text speaks of [gender] profession
This text talks about a Igender] occupation
This text mentions a [gender] profession
Table 3: Templates used for generation of hypothesis. Here gender corresponds male female such as "This text talks about a female occupation "Thhis text talks about a male occupation"
3.2 EXPERIMENTS
The key idea of our experiments is to test null hypothesis according to which the difference between predicted entailment probabilities, PS and Pm. on pairing given premise p with female hypothesis f and male hypothesis respectively, should be O.
We test our evaluation sets for each of the models mentioned Section 3.2.1. For every sentence used as premise, the model is first fed the female specific hypothesis, f. followed by its male alternative, m.
A typical RTE task would predict of the three labels entailment, neutral and contradiction for the given pair of premise and hypothesis. For this experiment we investigate the model predicts the textual entailment to be "definitely true" "definitely false". Hence, we convert our problem into a binary case: "entailment" vs. "contradiction" thus scraping the logit for the "neutral" label and taking softmax over the other two labels.
3.2.1 MODELS AND TRAINING DETAILS
Transformer models pretrained on large dataset have shown state-of-the performance in the task of RTE for various NLI datasets. In this work, we use three models that have been widely used both in research and production:
BERT (Devlin et al./ (2019)) is the Bidirectional Encoder Representations from Transform- ers, pretrained using a combination of masked language modeling objective and next sen- tence prediction on large corpus comprising the Toronto Book Corpus and Wikipedia. obtained state-of-ar results for recognition of textual entailment on both MNLI and SNLI datasets when it was first released.
RoBERTa follows a Robustly Optimized BERT Pretraining Approach (Liu et al. 2019) It improves on BERT by modifying key hyperparameters in BERT, and training with much larger mini-batches and learning rates.
BART (Lewis et al. (2019). a denoising autoencoder for pretraining seguence-to-sequees models, is trained by (I) corrupting with arbitrary noising function, and (2) learning a model to reconstruct the original text. It matches RoBERTa's performance on natural language understanding tasks.
Model
Configuration
BERT
bert-base-uncased
RoBERTa
roberta- Sveerta-basev2
BART
facebook/bart- base
Table 4: Configurations of the models tested for gender- bias
Hyperparameters: We fine-tune above models MNLI and SNLI datasets each generating total of 6 models (3 for each dataset) to test our evaluation sets on. The pretrained configuration used for each of the models is mentioned in Table 4. We train all models using AdamW optimizer with B1 0.9, 32 = 0.999 and L2 weight decay of 0.01. A learning rate of le-5 was used for RoBERTa and 2e-5 for the other two models. We train each of these models for 3 epochs for both the datasets.
4

---

Under review as a conference paper at ICLR 2021
3.2.2 METRICS
The following metries are calculated for both in-distribution (1) and out-of-distribution (O) evalua- tion sets:
S According to null hypothesis, an unbiased model should predict the same la- bel for both the f and m since they hold the same structure and differ only by a word (male/female). S is the % of instances where model gave the same prediction for both the cases. A low value for this metric is an indicator of high bias.
A This represents the mean absolute difference between the entailment probabilities for the two hypothesis. According to our null hypothesis, higher value is the indicator of high bias. This is the most important indicator of bias as it gives us the measure of difference in the prediction probabilities and helps us quantify the bias.
B A biased model would have higher entailment probability towards the hypothesis which represents the gender that an occupation is dominated by. For e.g., for the premise, "The nurse sitting on the bench" " where the profession is female dominant one. biased model will predict entailment with a higher probability when paired with female-specific hypothesis. B is the % of instances where this bias shown by the model. higher value of B would indicate higher bias but this should be observed along with AP to get better understanding, model with a higher value B but lower AP would still be considered less biased when compared with model with comparatively lower B but with a large mean absolute difference AP.
3.3
RESULTS AND ANALYSIS
SNLI (I)
MNLI (I)
Acc (1)
S (t)
AP())
B (L)
Acc (1)
S (t)
AP(()
B (L)
BERT RoBERTa BART
90.48
50.97
43.02
70.05
83.68
71.89
24.09
69.79
91.41
72.13
27.23
77.79
87.59
64.35
21.85
65.12
91.28
61.17
34.9
74.0
85.57
85.58
16.29
66.82
SNLI (0)
MNLI (0)
Acc (1)
S (t)
AP(()
B (L)
Acc (1)
S St)
AP(L)
B (L)
BERT 90.48
52.92
41.57
66.94
83.68
64.76
30.97
68.51
ROBERTa
91.41
71.02
26.57
73.76
87.59
64.33
24.86
64.53
BART
91.28
62.28
33.92
70.28
85.57
79.71
20.92
64.69
Table 5: Performance of the models when fine-tuned SNLI and MNLI datasets respectively. The metric Acc indicates the model accuracy when trained on original NIJI dataset (SNLI/MNLI) and evaluated dev set (dev-matched for MNLI), S is the number of instances with same prediction: entailment or contradiction, AP denotes the mean absolute difference in entailment probabilities of male and female hypothesis and B denotes the number of times the entailment probability of the hypothesis aligning with the stereotype was higher than its counterpart (higher values for the latter two metries indicate stronger biases). Numerics in bold represent the best value (least bias) for each metric. SNLI (O) and MNLI (O) represent the performance of various models fine-tuned on SNLI and MNLI respectively but tested on -utto--distribution evaluation set. SNLI (I) and MNLI (I) on the other hand have been tested on in-distribution evaluation set.
The main results of our experiments are shown Table For each tested model, we compute three metrics with respect their ability to predict the correct label (Section 3.2.2). Our analysis indicate that all the NLI models tested by us are indeed gender biased.
From Table 5, metric B shows that all the tested models perform better when presented with pro- stereotypical hypothesis. However when observed along with AP, we can see that BERT shows the most significant difference in prediction compared other models. Among the three models,
5

---

Under review as a conference paper at ICLR 2021
SNLI (I)
MNLI (I)
AP (4)
(1)
AP (4)
(L)
Male
Female
Male
Female
Male
Female
Male
Female
BERT RoBERTa BART
51.43
33.6
98.19
37.22
27.16
20.51
95.23
40.11
28.75
25.44
83.33
71.33
27.4
15.38
94.85
30.44
35.22
34.54
89.14
56.33
16.99
15.46
90,47
39.22
SNLI (0)
MNLI (O)
AP (4)
(4)
AP (4)
B (L)
Male
Female
Male
Female
Male
Female
Male
Female
BERT
49.16
32.72
96.19
32.83
34.36
27.02
92.52
40.5
RoBERTa
28.46
24.35
78.76
67.94
30.58
18.18
93.8
30.38
BART
34.54
33.19
86.04
51.88
22
19.65
87.9
37.61
Table 6: Detailed analysis of how bias varies with respect male and female dominated occupa- tions. Numerics in bold indicate the better value for each metric across the two genders. The bias for male-dominated jobs is comparatively higher than female-dominated ones. Notations same as those in Table 5.
BERT also has the lowest value of Metric in almost all the cases, indicating highest number of label shifts thus showing the greatest amount of bias.
Figure I: Comparison of trends in occupation bias across various models trained on original MNLI
dataset. We compare the distribution of occupational-bias predicted by our models in-distribution evaluation dataset (MNLI (I)) with the actual gender-domination statistics from CPS 2019. Models trained on SNLI also showed similar trends (Appendix A.2).
100
CCST
PART
CRS 2019
Biss (%)
40
<0
burr
Occupations
detailed analysis with respect to gender can be found in Table 6, where we compare the values of B and AP when male dominated jobs are compared with female-dominated ones. From the results, it can be seen that the number of examples where the model was biased towards the pro-stereotypical hypothesis is higher for male-dominated occupations.
While both MNLI and SNLI show similar trends with respect to most metrics, results from Table indicate that models fine-tuned SNLI has relatively higher bias than those trained MNLII

---

Under review as a conference paper at ICLR 2021
We also compare bias distribution across various occupations for the three models where the bias for the models (BERT, ROBERTa and BART) is the absolute difference between the entailment probabilities of two hypothesis. We compare this with CPS 2019 data where the difference between the gender distribution is used as the bias. Fig. shows that all the three models follow similar trends for occupational bias. We also compare this with the statistics gender-domination in jobs given by CPS 2019 and validate that the bias distribution from models' predictions conforms with the real world distribution. Model predictions on occupations like nurse, housekeeper that are female dominated reflect higher bias as compared to those with equal divide, e.g. designer, attendant, accountant.
DEBIASING : GENDER SWAPPING
We follow simple rule based approach for swapping. First we identify all the occupation based entities from the original training set. Next, we build a dictionary of gendered terms and their opposites (e.g. "his"<> "her", "he"<>"she" etc.) and use them to swap the sentences. These gender swapped sentences are then augmented to the original training set and the model is trained on it. Unlike Zhao et al's turking approach, we follow an automatic approach to perform this swapping thus our method can be easliy extended to other datasets well. From Table 7, it can be seen the augmentation of training set doesn't 't deteriorate model performance (accuracy (Acc)) on the original training data (SNLI/MNLI). This simple method removes correlation between gender and classification decision and has proven to be effective for correcting gender biases other natural language processing tasks. (Zhao al. (0188))
SNLI (I)
MNLI (I) S (
Acc (1)
S (t)
AP(L)
B (L)
Acc (f)
AP(L)
B (L)
BERT RoBERTa
90.50
57.17
35.02
67.02
84.04
87.48
14.60
72.07
91.51
51.53
34.02
67.79
87.1
65.58
20.98
67.69
BART
90.6
61.89
31.71
72.76
85.76
75.33
21.32
70.67
SNLI (0)
MNLI (O)
Acc (1)
S (t)
AP(L)
B (L)
Acc (f)
S (t)
AP(L)
B (L)
BERT RoBERTa BART
90.50
65.5
30.01
66.71
84.04
78.76
19.99
72.53
91.51
61.64
33.08
63.61
87.1
65.35
22.53
67.23
90.6
66.43
27.56
70.05
85.76
68.43
26.51
68.67
Table 7: Performance of the models after debiasing. Notations are same as those in Table 5.
Effectiveness of debiasing: From results in Table we can see that performance on BERT with respect to bias has improved following the debiasing approach. A comparison of the change in metries AP and B before and after debiasing can also be seen in Fig. (The figure here represents the trends with respect to in-distribution evaluation sets. Similar results were obtained for out-of- distrubution test and can found in the Appendix (A.1))
The improvement in results for BERT indicate that maintaining gender balance in the training dataset is, hence, of utmost importance to avoid gender-bias induced incorrect predictions. The other two models, RoBERTa and BART, also show slight improvement in performance with respect to most metrics. However, this is concerning since the source of bias in these cases is not the NLI dataset but the data that these models were pre- pre-trained on. Through our results, suggest that attention be paid while curating such dataset so as to avoid biased predictions in the downstream tasks.
5 RELATED WORK
Gender Bias in NLP: Rudinger et al. (2018) Prior works have revealed gender bias various NL. tasks (Zhao al. 2018a). Zhho et al. 201 8c), Rudinger al. (001)). Rudinger et al. (018))
7

---

Under review as a conference paper at ICLR 2021
bafore and
Difference before
(1)
RodtRla
Rodtella
DURT
64
66
In Aftir de blesing
before de-biasing
After do bol-biisingg
Beffre diiibaasing
(a) A SNT.T(T)
(h) AP MNLI(T)
Difforenee AP before and after de-hincing (SNE1())
Offfrence in AP before and after desbiasiin (MNU
BART
BAX1
BERT
BERT
20
50
Aft de blassing
Defore de-biasing
or dl-bbassing Batore deaibaasing
(c) SNLI(T)
(d) MNLI(T)
Figure 2:
Difference in AP and B in MNLI and SNLI in-distribution evaluation sets before and after de-biasing Similar results are observed for out-of-distribution evaluation sets (Appendix A.
Caliskan et al. (2017) Gaut et al. (2020), He et al. (2019a). Bolukbasi et al. (2016)). Authors
have created various challenge sets to evaluate gender bias, for example Gaut al. (2020) created WikiGenderBias.ff for the purpose of analyzing gender bias in relation extraction systems. Caliskan al. (2017) contribute methods for evaluating bias text, the Word Embedding Association Test (WEAT) and the Word Embedding Factual Association Test (WEFAT). Zhao et al. (2018a) intro- duced benehmark WinoBias for conference resolution focused on gender bias. To our knowledge, there are evaluation sets measure gender bias in NLI tasks. In this work, fill this gap by proposing a methodology for the same.
Data Augmentation: Data augmentation has been proven to be effective way to tackle chal- lenge sets (Andreas (2020a), McCoy et al. (2019). ) Jia & Liang (017)) Gardner et al. (2020)) By correcting the data distribution, various works have shown mitigate bias significant amount Min et al. (2020), Zhao et al. (2018a), Glockner et al. (2018). In this paper, we use a rule-based gender swap augmentation similar that used by /hhao et al. (2018)).
CONCLUSION AND FUTURE WORK
We show the effectiveness of our challenge setup and find that simple change of gender in the hypothesis can lead to a different prediction when tested against the same premise. This difference a result of biases propagated in the models with respect to occupational stereotypes. We also find that the distribution of bias in models' predictions conforms to the gender-distribution in jobs in the real world hence adhering to the social stereotypes. Augmenting the training- datasst in order to ensure gender-balanced distribution proves to be effective in reducing bias for BERT indicating the importance of maintaining such balance during data curation. The debiasing approach also reduces the bias in the other two models (RoBERTa and BART) but only by small amount indicating that attention also needs to be paid on the dataset used for training these language models. Through this work, aim establish baseline approach for evaluating gender bias in NLI systems, but we hope that this work encounters further research by exploring advanced debiasing techniques and also exploring bias in other dimensions.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jacob Andreas. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computcttional Linguistics, pp. 7556-7566, Online, July 2020a. Association for Computational Linguistics. doi: 10. 10.18653/w12222..ae acl-main.676 URL https:
www.w..wwwwb..omm... org httpr///anthooooo//222222
htwp://cwwee..orrr/aaaaooooooooo act.aaaaaa...mmmaaaaaaaaaaaaaaaa 676
Jacob Andreas. Good-enough compositional data augmentation, 2020b.
Nată M Barbosa and Monchu Chen. Rehumanized crowdsoureing: labeling framework addressing bias and ethics in machine learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1-12, 2019.
Steven Bird, Ewan Klein, and Edward Loper. Natural Language Processing with Python. O O'Reilly Media, Inc., 1st edition, 2009. ISBN 0596316499
Tolga Bolukbasi, Kai- Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer woman is homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS 16, pp. 4356 4364, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 978151083881
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2015.
Marc-Etienne Brunet, Colleen Alkalay-Houlhhan, Ashton Anderson, and Richard Zemel. Under- standing the origins of bias in word embeddings. In International Conference on Machine Learn- ing. pp. 803-811. 2019.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semantics derived automatically from lan- guage corpora contain human-like biases. Science, 356(6334) 2017. ISSN 0036-8075. doi: 10. 20.1126//cieecc.aal42222 356/ 6334/3331822 /88
URL htttp:://ciienes wcw.ecciiicnoo scwencemageggeneeeeooo.o.o.... org/ content/
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. arXiv preprint arXiv: (tt.://90003333 03683, 2019.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi:
URL https:
7/ /wwwwwco awwweb. www.ootgtta....ooo..ooooooooo htt.anoooooooyolooooao 1423
Terrance DeVries and Graham W. Taylor. Dataset augmentation in feature space, 2017.
Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston Queens are powerful too: Mitigating gender bias in dialogue generation. arXiv preprint arXiv:1911.03842. 2019.
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala,, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, A. Zhang, and Ben Zhou. Evaluating nlp models via contrast sets. ArXiv, Abs/2004.02709, 2020.
Andrew Gaut, Tony Sun. Shirlyn Tang, Yuxin Huang, Jing Qian, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. Towards understanding gender bias in relation extraction. Proceedings of the 58th Annual Meeting of the Associa- fion for Computational Linguistics, pp. 2943-2953, Online, July 2020. Association for Computa- tional Linguistics. doi: 10.18653/v maw.m/20000.mmiimi22...6.. main.265. URL https:/p/ / www. aclweb. orq/
anthol matholooy/20220 acl -aa 265

---

Under review as a conference paper at ICLR 2021
Max Glockner, Vered Shwartz, and Yoav Goldberg, Breaking nli systems with sentences that require simple lexical inferences, 2018.
Hila Gonen and Yoav Goldberg. Lipstick on a pig: Debiasing methods cover up systematic gender biases word embeddings but do not remove them. arXiv preprint arXiv:1903.03880 2019.
He He, Sheng Zha, and Haohan Wang. Unlearn dataset bias natural language inference by fit- ting the residual. In Proceedings of the 2nd Workshop Deep Learning Approaches for Low- Resource NLP (DeepLo 2019), pp. 132-142, Hong Kong, China, November 2019a. Association for Computational Linguistics. doi: 10.18653/v1/D19-6111 2xt//attholoog/D11-6611 6115
RtLhttps::/ www aclweb.
He He, Sheng Zha, and Haohan Wang. Unlearn dataset bias in natural language inference by fitting the residual. arXiv preprint arXiv:1908. 10763, 2019b.
Peter Henderson, Koustuv Sinha, Nicolas Angelard- -ontierr,, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 123-129, 2018
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. 07 2017.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sqquene pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 13461, 2019.
Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.1 1692, 2019.
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Anmual Meeting of the As- sociation for Computational Linguisties, pp. 3428 3448, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 11116633/1//11-13344 URL https:t/p///. www. ac web.
Junghyun Min, R. McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. Syntactic data augmentation increases robustness to inference heuristics. pp. 2352, 01 2020. doi: 10. 10.18653/v1/2020.
acl-main.212.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial NLI: A new benchmark for natural language understanding. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguisties, 2020.
Ho Park, Jamin Shin, and Pascale Fung. Reducing gender bias in abusive language detection. arXiv preprint arXiv: 180..0088.000 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1D116-1222 1264. https::: www ac web. brg/antholoogy/DlG-1z2
Rachel Rudinger, Chandler May, and Benjamin Van Durme. Social bias in elicited natural language inferences. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pp. 74-79, Valencia, Spain, April 2017. Association for Computational Linguistics. doi: 10.
18633/11117-16699 URL https:tppp... www. ac www. http://wwwweeebb..r///ttttoool// 1ttpo///y//1111111111111 7-117--6999
10

---

Under review as a conference paper at ICLR 2021
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Pupers), pp. 8-14, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653//1/N18-22000
/www acl www.wwwwbb.... wwweb.orggaathhooog.....o
N18 2002
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018. URL
http: http://acllbb... org. wwt.://gtgtg.gggooo.ooooooo
1101
Wenpeng Yin, Jamaal Hay, and D. Roth. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. ArXiv, abs/1909.00161, 2019.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Short Papers), pp. 15-20, New Orleans, Louisiana, June 201 8a. Association for Computational Linguistics. doi: 10. 10.18653/1//118-0003. URL
https: www wwwp:ec.wwwb..ooo//aatttoooooo/p 2003
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint 2018b.
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang. and Kai-Wei Chang. Learning gender-neutral word embeddings. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4847-4853, Brussels, Belgium, NoveeberNovember 2018c. Association for Com-
putational Linguistics. doi: 10.18653/v1/D18.. 1521. URL https:/ www. ac webb org/
anthology//DI8 1521
Xiang Zhou and Mohit Bansal. Towards robustifying nli models against lexical dataset biases. arXiv preprint arXiv:2005.0477 04732, 2020.
11


Reviews:
[{"review_id": "rHxs1nuoYTL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper's main contribution is the construction of an NLI style dataset for evaluating whether systems training on MNLI/SNLI are gender biased with respect to occupations. Premises are mined from MNLI, SNLI, QNLI, and ANLI that contain occupation words and those premises are paired with one of three templates, paraphrasing, \"This text mentions a XXX occupation\" where XXX is either 'male' or 'female'. Such pairs are put through trained NLI systems, and their preference toward the male or female version of the templates is recorded. If a system favors the hypothesis in line with labor statistics overall, authors conclude it is biased w.r.t gender and occupations. 3 systems are evaluated, all are found biased, and then a data augmentation approach from previous work (gender swapping from Zhao's coref bias paper) is used for mitigation with mixed results.  \n\nPros:\n1. The introduction of an NLI + occupational gender bias dataset is new.\n2. Experiments on several NLI systems\n\nCons:\n1. The data contribution seems small and somewhat unnatural. The hypothesis format seems extremely unnatural (being text referential). I wonder if such examples are out of domain for the trained NLI systems. The ground truth for the proposed examples seems to be neutral (occupations are neither male nor female), so I would like to know how often evaluated systems actually predict this. \n2.The bias measurement forces the models to predict either entailment or contradiction, where in fact the ground truth answer, in my opinion, for the proposed NLI examples, is neutral. (the occupation is neither male nor female). For all we know, the models are correctly predicting that with high probability, but the measurement is forcing a renormalization between entailment and contradiction examples (Section 3.2).  This seems like it would be a problem for the \"delta P\" and \"B\" measurement. \n3.The gender swapping experiment is good to see but seems largely ineffective and I found its presentation hard to follow. Zhao et al. had a turking process to make sure all entities in CONLL data were covered in the swaps. Were any such measures taken here to deal with new NLI data? How was the list of swaps constructed in this case? The results are split between Table 5 and Table 6, so I was unsure where it helped, or in the unexplained Figure 2. From Figure 2, it seems like it hurt in some cases.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "8L7hSvLM738", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIn this paper, the authors design a method to evaluate the gender bias for natural language inference tasks. They construct an evaluation dataset that consists of (premise, female hypothesis, male hypothesis) and design three scores (inconsistent predictions, probability gap, and dominant probability) to measure the gender bias for BERT, RoBERTa, and BART. The experimental results show that those models indeed have a gender bias. They also show that the bias can be reduced by data augmentation.\n\nGender bias is an interesting and important topic in the NLP domain. However, I have some concern about this paper:\n- When constructing the evaluation dataset, the authors replace the occupation word in the premise with other occupation words. However, this can lead to inconsistent semantics. For example, \"the doctor is operating\" becomes \"the teacher is operating\", which may not fit the realistic situation.\n- The constructed dataset contains only entailment pairs. How about analyzing the contradiction cases as well?\n- When analyzing the results, the authors disregard the neutral case. I am wondering if they train the models in the same way. If not, it seems that there is a domain mismatch.\n- This point is my primary concern. In the constructed dataset, the hypothesis is generated by templates and looks like the context is not very related to the premise. However, in most of NLI datasets, the premise and hypothesis are usually related. So the domain of training set and their constructed evaluation set are different. It can be reasonable for models to perform not well and have the bias on the evaluation set, since the domain changes a lot. Why not consider the existing hypothesis? For instance, replace \"he\" or \"his\" in the existing hypothesis with \"she\" or \"her\" so you can have female hypothesis and male hypothesis.\n- I don't quite understand the definition of B for evaluation. Is that probability for some predefined gender-specific occupations? If that is the case, how to define those words?\n- What is the definition of \"bias\" in Figure 1?\n- In Figure 6, how to calculate delta P for female and male respectively? In my understanding, delta P is the probability gap between female and male hypothesis.\nI  suggest that the authors use one table to show the difference before and after the data augmentation to compare the numbers more easily. \n\nSome typos\n- In Table 2, teacher => guard.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7vO3nScdg6F", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Paper summary*: This paper proposes a method for measuring stereotypical associations about occupations that are associated with genders using the natural language inference task. The method involves setting up a NLI pair where the premise is a gender-neutral statement about an occupation, and the hypothesis is explicitly gender specific. The analysis shows that NLI models do incorporate stereotypes. The paper also investigates how to reduce this bias by data augmentation.\n\n*Review*: At a high level, the method proposed in this paper makes sense, but there is a critical problem in terms of novelty: the idea of using NLI to probe stereotypes is not new. In fact, nearly the same proposal outlined here is explored by Dev et al (2020), who additionally use the mechanism to probe for other kinds of stereotypes as well.\n\nThe hypothesis templates are interesting, but present a bit of a technical question. The hypothesis of the form \"This text talks about a female occupation\" refers to the *text* of the premise, rather than the *events* or *entities* in it. In other words, it talks about the form of the premise, rather than its meaning. Of course, there's nothing wrong with this, but it breaks a crucial assumptions about how the NLI data (in particular the SNLI data) was sourced: the events and entities in the hypothesis refer to the events and entities in the premise as much as possible. In contrast, the word \"text\" in the hypotheses constructed in this work refers to the entire text of the premise, and not its entities and events. It is not clear how this change affects model performance.\n\nOne way to fix the issue is to change the hypothesis templates to use the same (or similar) words as the premises, and replace the occupation word with a gendered word. (But doing so would make the work even closer to that of Dev et al 2020.)\n\nIt is not clear why the neutral label is removed and the problem is converted into a binary problem of deciding whether the hypothesis is entailed or contradicted. It seems that most of the hypotheses would actually be neutral, and a good model should allocate most of its probability mass to the neutral label. Why do we have to force a choice between entail and contradict, when a stereotype-free model would actually predict neutral?\n\nThe results in table 6 are interesting. It seems that the models \"memorize\" the distributional correlations between gender and jobs differently for men and women. Are there any conjectures about why this may be the case?\n\nIt is not clear whether the bias that is being measured is in the representation (i.e. the *BERT embeddings) or the task (i.e., the NLI data). The experiments suggest that the problem is perhaps in both. Previous work on stereotypes involving language has largely focused how they are encoded in the embeddings, and removing them. This paper seems to argue that the provenance of the stereotypes is the training data for the task. However, the final results suggest that this is not entirely the case, and the paper does say so in the section on debiasing. It may be worth posing the question about the source of the biases early on in the paper.\n\nSince the paper is talking about stereotypes in language technology, the authors should go over the work of Blodgett et al (2020) to better situate the motivations and outcomes of this work. Indeed, there should be a discussion in the paper about the cultural context and assumptions that are implicit in the measurements. (For example, is the definition of B based on an American context?  Would the measures transfer to a different country/cultural perspective?)\n\n*Minor point*: The plot in figure 1 should not be a line plot because the horizontal axis is categorical. A bar chart would be a better fit (and would convey the point more clearly).\n\n*References*\n\n* Dev, Sunipa, Tao Li, Jeff M. Phillips, and Vivek Srikumar. \"On Measuring and Mitigating Biased Inferences of Word Embeddings.\" In AAAI, pp. 7659-7666. 2020.\n  \n* Blodgett, Su Lin, Solon Barocas, Hal III Daumé, and Hanna Wallach. \"Language (technology) is power: The need to be explicit about NLP harms.\" In Proceedings of the Annual Meeting of the Association for Computational Lingustics (ACL). 2020.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper offers a new dataset and accompanying metric to measure the degree to which NLI (textual entailment) systems are aware of gender–occupation associations.

Pros:
- The paper deals with an important issue in the context of a visible set of models and datasets.

Cons:
- The metric is designed to evaluate bias on models trained for a specific, precisely defined task, but it does not conform to the standard formulation of that task, which makes results on those metric untrustworthy and potentially arbitrary. Reviews had concerns about both the data (the use of references to the form of the premise text) and the metric (the handling of 'neutral' predictions).
- The proposed definition of bias is not clearly mapped onto a concrete potential harm.
- There has been substantial similar prior work on this problem. This doesn't invalidate this work, but it does raise the bar a bit, since arguments of the form 'we need to start a conversation about bias in models' are not pursuasive.

Decision: Reject