Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: *Paper summary*: This paper proposes a method for measuring stereotypical associations about occupations that are associated with genders using the natural language inference task. The method involves setting up a NLI pair where the premise is a gender-neutral statement about an occupation, and the hypothesis is explicitly gender specific. The analysis shows that NLI models do incorporate stereotypes. The paper also investigates how to reduce this bias by data augmentation.

*Review*: At a high level, the method proposed in this paper makes sense, but there is a critical problem in terms of novelty: the idea of using NLI to probe stereotypes is not new. In fact, nearly the same proposal outlined here is explored by Dev et al (2020), who additionally use the mechanism to probe for other kinds of stereotypes as well.

The hypothesis templates are interesting, but present a bit of a technical question. The hypothesis of the form "This text talks about a female occupation" refers to the *text* of the premise, rather than the *events* or *entities* in it. In other words, it talks about the form of the premise, rather than its meaning. Of course, there's nothing wrong with this, but it breaks a crucial assumptions about how the NLI data (in particular the SNLI data) was sourced: the events and entities in the hypothesis refer to the events and entities in the premise as much as possible. In contrast, the word "text" in the hypotheses constructed in this work refers to the entire text of the premise, and not its entities and events. It is not clear how this change affects model performance.

One way to fix the issue is to change the hypothesis templates to use the same (or similar) words as the premises, and replace the occupation word with a gendered word. (But doing so would make the work even closer to that of Dev et al 2020.)

It is not clear why the neutral label is removed and the problem is converted into a binary problem of deciding whether the hypothesis is entailed or contradicted. It seems that most of the hypotheses would actually be neutral, and a good model should allocate most of its probability mass to the neutral label. Why do we have to force a choice between entail and contradict, when a stereotype-free model would actually predict neutral?

The results in table 6 are interesting. It seems that the models "memorize" the distributional correlations between gender and jobs differently for men and women. Are there any conjectures about why this may be the case?

It is not clear whether the bias that is being measured is in the representation (i.e. the *BERT embeddings) or the task (i.e., the NLI data). The experiments suggest that the problem is perhaps in both. Previous work on stereotypes involving language has largely focused how they are encoded in the embeddings, and removing them. This paper seems to argue that the provenance of the stereotypes is the training data for the task. However, the final results suggest that this is not entirely the case, and the paper does say so in the section on debiasing. It may be worth posing the question about the source of the biases early on in the paper.

Since the paper is talking about stereotypes in language technology, the authors should go over the work of Blodgett et al (2020) to better situate the motivations and outcomes of this work. Indeed, there should be a discussion in the paper about the cultural context and assumptions that are implicit in the measurements. (For example, is the definition of B based on an American context?  Would the measures transfer to a different country/cultural perspective?)

*Minor point*: The plot in figure 1 should not be a line plot because the horizontal axis is categorical. A bar chart would be a better fit (and would convey the point more clearly).

*References*

* Dev, Sunipa, Tao Li, Jeff M. Phillips, and Vivek Srikumar. "On Measuring and Mitigating Biased Inferences of Word Embeddings." In AAAI, pp. 7659-7666. 2020.
  
* Blodgett, Su Lin, Solon Barocas, Hal III Daumé, and Hanna Wallach. "Language (technology) is power: The need to be explicit about NLP harms." In Proceedings of the Annual Meeting of the Association for Computational Lingustics (ACL). 2020.

review_title: interesting work, but not novel

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper's main contribution is the construction of an NLI style dataset for evaluating whether systems training on MNLI/SNLI are gender biased with respect to occupations. Premises are mined from MNLI, SNLI, QNLI, and ANLI that contain occupation words and those premises are paired with one of three templates, paraphrasing, "This text mentions a XXX occupation" where XXX is either 'male' or 'female'. Such pairs are put through trained NLI systems, and their preference toward the male or female version of the templates is recorded. If a system favors the hypothesis in line with labor statistics overall, authors conclude it is biased w.r.t gender and occupations. 3 systems are evaluated, all are found biased, and then a data augmentation approach from previous work (gender swapping from Zhao's coref bias paper) is used for mitigation with mixed results.  

Pros:
1. The introduction of an NLI + occupational gender bias dataset is new.
2. Experiments on several NLI systems

Cons:
1. The data contribution seems small and somewhat unnatural. The hypothesis format seems extremely unnatural (being text referential). I wonder if such examples are out of domain for the trained NLI systems. The ground truth for the proposed examples seems to be neutral (occupations are neither male nor female), so I would like to know how often evaluated systems actually predict this. 
2.The bias measurement forces the models to predict either entailment or contradiction, where in fact the ground truth answer, in my opinion, for the proposed NLI examples, is neutral. (the occupation is neither male nor female). For all we know, the models are correctly predicting that with high probability, but the measurement is forcing a renormalization between entailment and contradiction examples (Section 3.2).  This seems like it would be a problem for the "delta P" and "B" measurement. 
3.The gender swapping experiment is good to see but seems largely ineffective and I found its presentation hard to follow. Zhao et al. had a turking process to make sure all entities in CONLL data were covered in the swaps. Were any such measures taken here to deal with new NLI data? How was the list of swaps constructed in this case? The results are split between Table 5 and Table 6, so I was unsure where it helped, or in the unexplained Figure 2. From Figure 2, it seems like it hurt in some cases.  
review_title: Data contribution on NLI but unclear measurements and mitigation efforts

final_score: rating: 3: Clear rejection
review_content: Summary:
In this paper, the authors design a method to evaluate the gender bias for natural language inference tasks. They construct an evaluation dataset that consists of (premise, female hypothesis, male hypothesis) and design three scores (inconsistent predictions, probability gap, and dominant probability) to measure the gender bias for BERT, RoBERTa, and BART. The experimental results show that those models indeed have a gender bias. They also show that the bias can be reduced by data augmentation.

Gender bias is an interesting and important topic in the NLP domain. However, I have some concern about this paper:
- When constructing the evaluation dataset, the authors replace the occupation word in the premise with other occupation words. However, this can lead to inconsistent semantics. For example, "the doctor is operating" becomes "the teacher is operating", which may not fit the realistic situation.
- The constructed dataset contains only entailment pairs. How about analyzing the contradiction cases as well?
- When analyzing the results, the authors disregard the neutral case. I am wondering if they train the models in the same way. If not, it seems that there is a domain mismatch.
- This point is my primary concern. In the constructed dataset, the hypothesis is generated by templates and looks like the context is not very related to the premise. However, in most of NLI datasets, the premise and hypothesis are usually related. So the domain of training set and their constructed evaluation set are different. It can be reasonable for models to perform not well and have the bias on the evaluation set, since the domain changes a lot. Why not consider the existing hypothesis? For instance, replace "he" or "his" in the existing hypothesis with "she" or "her" so you can have female hypothesis and male hypothesis.
- I don't quite understand the definition of B for evaluation. Is that probability for some predefined gender-specific occupations? If that is the case, how to define those words?
- What is the definition of "bias" in Figure 1?
- In Figure 6, how to calculate delta P for female and male respectively? In my understanding, delta P is the probability gap between female and male hypothesis.
I  suggest that the authors use one table to show the difference before and after the data augmentation to compare the numbers more easily. 

Some typos
- In Table 2, teacher => guard.

review_title: The research topic is interesting. However, I have some concern about their constructed evaluation dataset.

Meta-review: This paper offers a new dataset and accompanying metric to measure the degree to which NLI (textual entailment) systems are aware of gender–occupation associations.

Pros:
- The paper deals with an important issue in the context of a visible set of models and datasets.

Cons:
- The metric is designed to evaluate bias on models trained for a specific, precisely defined task, but it does not conform to the standard formulation of that task, which makes results on those metric untrustworthy and potentially arbitrary. Reviews had concerns about both the data (the use of references to the form of the premise text) and the metric (the handling of 'neutral' predictions).
- The proposed definition of bias is not clearly mapped onto a concrete potential harm.
- There has been substantial similar prior work on this problem. This doesn't invalidate this work, but it does raise the bar a bit, since arguments of the form 'we need to start a conversation about bias in models' are not pursuasive.

Decision: Reject