SlowGuess/ACL_2020_OCR huggingface dataset:
Xiang Zhou Mohit Bansal
{xzh,mbansal}@cs.unc.edu
While deep learning models are making fast progress on the task of Natural Language Inference, recent studies have also shown that these models achieve high accuracy by exploiting several dataset biases, and without deep understanding of the language semantics. Using contradiction-word bias and word-overlapping bias as our two bias examples, this paper explores both data-level and model-level debiasing methods to robustify models against lexical dataset biases. First, we debias the dataset through data augmentation and enhancement, but show that the model bias cannot be fully removed via this method. Next, we also compare two ways of directly debiasing the model without knowing what the dataset biases are in advance. The first approach aims to remove the label bias at the embedding level. The second approach employs a bag-of-words submodel to capture the features that are likely to exploit the bias and prevents the original model from learning these biased features by forcing orthogonality between these two submodels. We performed evaluations on new balanced datasets extracted from the original MNLI dataset as well as the NLI stress tests, and show that the orthogonality approach is better at debiasing the model while maintaining competitive overall accuracy. $^1$
In this work, we focus on investigating and reducing biases in the task of Natural Language Inference (NLI), where the target of the model is to classify the relations between a pair of sentences into three categories: entailment, neutral and contradiction. With the release of large-scale standard datasets (Bowman et al., 2015; Williams et al., 2018), significant success has been made on
this task, and recent state-of-the-art neural models have already reached competitive performance even compared to humans. However, a number of papers (Gururangan et al., 2018; Poliak et al., 2018; Nie et al., 2019; Naik et al., 2018) have shown that despite the high accuracy on these datasets, these models are far from mastering the required nature of natural language inference. Instead of deeply understanding the sentences in the correct semantic way, these models tend to exploit shortcuts or annotation artifacts in the dataset and actually overfit to these datasets to predict the label using simple patterns. However, most shortcuts are only valid within the datasets and fail to hold for general natural language. Hence, these models fail to generalize to other datasets for the same task (Talman and Chatzikyriakidis, 2019), perform badly on challenge analysis datasets (Glockner et al., 2018; McCoy et al., 2019; Wang et al., 2019b), and are fooled by adversarial attacks (Naik et al., 2018).
One major cause of this problem is the existence of dataset biases. Since most NLP datasets are often collected and processed by crowdworkers, bias can be added to the data at every step of data collection. For example, when writing contradiction pairs, workers are likely to use negation words such as 'not', and when creating entailment pairs, workers are likely to keep most of the words in the premise sentence. This results in 'annotation artifacts' in the dataset (Gururangan et al., 2018). In reality, almost every dataset contains countless such diverse biases. In our paper, we focus on the Multi-Genre Natural Language Inference (MNLI) dataset (Williams et al., 2018) in English, and on two specific kinds of dataset bias:
Contradiction Word Bias (CWB): If the hypothesis sentence contains some specific words (such as negation words) that are always used by the crowd-workers to generate contradiction pairs, then the sentence pair is very likely to be contradiction.
1Our code and data are available at: https://github. com/owenzx/LexicalDebias-ACL2020
8759
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8759-8771 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Contradiction-Word Bias	Word-Overlapping Bias
MNLI	Prem.	A recorded menu will provide information on how to obtain these lists.	This is especially true on Menocra, where cold winter winds limit the seasons length.
Hypo.	Recorded menus do not provide any information at this time.	On Menocra, where cold winter winds limit the seasons length, this is especially true.
Stress	Prem.	Understanding is the key.	This is especially true on Menocra, where cold winter winds limit the seasons length.
Hypo.	Understanding is the most important and false is not true.	On Menocra, where cold winter winds limit the seasons length, this is especially true and true is true.
Table 1: The example samples for the CWB and WOB in the original dataset and the test samples in NLI stress tests (Naik et al., 2018) designed to reveal these biases (the stress test samples aim to fool the model to predict contradiction by adding negation word and to not predict entailment by reducing word overlapping).
Word Overlapping Bias (WOB): If the premise sentence and the hypothesis sentence have a high word-overlap, then the sentence pair is very likely to be entailment.
These two types of biases are selected as the focus of our experiments because: (1) there exist a significant number of samples in the dataset where they are a major problem; (2) they are conceptually easy to understand and relatively easier to evaluate. In our experiments, we not only used current existing evaluation datasets from Naik et al. (2018), but also extracted balanced evaluation datasets from the original data to evaluate these two biases. Although we only focus on these two kinds of dataset biases throughout our experiments, our methods are not specifically designed for these two biases and should be able to reduce other similar lexical biases simultaneously.
Using these two example lexical biases, our paper discusses the following three questions:
Q1. Is lexical bias a problem that can be solved by only balancing the dataset?
Q2. Can the lexical bias problem be solved using existing ideas from the gender bias problem?
Q3. What are some promising new modeling directions towards reducing lexical biases?
As responses to these three questions, we conduct three lines of experiments. Firstly, we expand the discussion of Q1 by studying whether and how the bias can be reduced by debiasing the dataset. For this, we add new training data which does not follow the bias pattern. This new data can come from two sources, either from the original training set or via manually generated synthetic data. We show that both methods can slightly reduce the model's bias. However, even after adding a large amount of additional data, the model still cannot be completely bias-free. Another critical problem with these data augmentation/enhancement based debiasing methods is that we need to know the specific behaviour of the biases before making some
related changes to the dataset. However, in reality, models are always faced with new training datasets containing unknown and inseparable biases. Hence, the answer to Q1 is mostly negative for simple data-level approaches and we also need to focus on designing direct model-debiasing methods.
Therefore, we turn our focus to directly debiasing the model (Q2 and Q3). The first method is to debias the model at the lower level, i.e., by directly debiasing the embeddings so that they do not show strong biases toward any specific label. This is one of the most prevalent methods for reducing gender biases, so through the examination of this idea, we aim to compare lexical bias problems to gender bias problems and highlight its uniqueness (hence answering Q2). Finally, we debias the model at the higher level, i.e., by designing another bag-of-words (BoW) sub-model to capture the biased representation, and then preventing the primary model from using the highly-biased lexical features by forcing orthogonality between the main model and the BoW model (via HEX projection (Wang et al., 2019a)). In our experiments, we show that debiasing the prediction part of the model at higher levels using BoW-orthogonality is more effective towards reducing lexical biases than debiasing the model's low-level components (embeddings). This approach can significantly robustify the model while maintaining its overall performance, hence providing a response to Q3. We also present qualitative visualizations using LIME-analysis for the important features before and after applying the BoW-orthogonality projection.
Problems with NLI Models and Datasets. Despite the seemingly impressive improvements in NLI tasks, recently a number of papers revealed different problems with these models. Gururangan et al. (2018) showed that annotation artifacts in the datasets are exploited by neural models to get high
8760
accuracy without understanding the sentence. Poliak et al. (2018) showed a similar phenomenon by showing models getting good performance but only taking one sentence as the input. Nie et al. (2019) showed that NLI models achieved high accuracy by word/phrase level matching instead of learning the compositionality. Naik et al. (2018) constructed bias-revealing datasets by modifying the development set of MNLI. In our evaluation, besides using the datasets from Naik et al. (2018), we also extract new datasets from the original MNLI dataset to maintain the consistency of input text distribution.
Adversarial Removal Methods. Adversarial removal techniques are used to control the content of representations. They were first used to do unsupervised domain adaptation in Ganin and Lempitsky (2015). Xie et al. (2017) later generalized this approach to control specific information learned by the representation. Li et al. (2018) used a similar approach to learn privacy-preserving representations. However, Elazar and Goldberg (2018) showed that such adversarial approach fails to completely remove demographic information. Minervini and Riedel (2018) generate adversarial examples and regularize models based on first-order logic rules. Belinkov et al. (2019a,b) showed that adversarial removal methods can be effective for the hypothesis-only NLI bias. Our focus is on two different lexical biases and our results are complementary to theirs.<sup>2</sup> Recently, Wang et al. (2019a) proposed HEX projection to force the orthogonality between the target model and a superficial model to improve domain generalization for image classification tasks. Here, to make the model less lexically biased, we apply the HEX projection with specially-designed NLP model architectures to regularize the representation in our models. Even more recently, Clark et al. (2019) and He et al. (2019) propose to robustify the task model with the help of an additional simple model, using ensembling to encourage cooperation of the two models. On the other hand, our main motivation to compare the advantages/limitations of dataset vs. embedding vs. classifier debiasing methods (against two different types of problematic lexical biases in NLI), and also our classifier debiasing method forces the task model to capture orthogonal information via HEX projection.
Removing Gender Bias in NLP Models. There
is also a line of work in NLP on analyzing and reducing gender bias in NLP models. Bolukbasi et al. (2016); Caliskan et al. (2017); Zhao et al. (2018a) studied the bias problem in word embeddings. Zhao et al. (2017) reduced gender bias in visual recognition using corpus-level constraints. Zhao et al. (2018b) discussed the gender bias problem in co-reference resolution. These problems are related to our work, but lexical biases are more complex. Multiple inseparable lexical dataset biases can influence one single example and the same word can have different lexical biases in different contexts. Later in our experiments, we show that these two problems behave differently and we present the need for different solutions.
3 Data-Level Debiasing
Models naturally learn the biases from the dataset they are trained on. Therefore, as we mentioned in Q1 in Sec. 1, one may first wonder if lexical bias can be completely removed by fixing the source of the bias, i.e., datasets. While collecting large-scale datasets (Bowman et al., 2015; Williams et al., 2018) already takes a lot of time and effort, collecting bias-free datasets is even more time-consuming and hard to control. Therefore, here we focus on getting additional data from currently-available resources. We conducted experiments using two resources of data. The first one is to do 'data enhancement' by repeating samples in the original training data. The second source is 'data augmentation' by manually creating synthetic data. We follow the construction of existing synthetic bias-revealing datasets to create new samples for the training set so that these targeted biases can be reduced.
Data Enhancement by Repeating Training Data. For most kinds of biases, there still exists a small portion of samples that don't follow the bias. Therefore, we reduce biases in datasets by repeating this portion of samples. For CWB, we select non-contradiction samples containing contradiction words (details see Sec. 5.1) in the hypothesis sentence but not in the premise sentence. For the WOB, we select non-entailment samples with highest word overlapping (measured by the Jaccard distance (Hamers et al., 1989) of words). Next, since the number of these unbiased samples may not be large enough, we repeatedly add those selected samples to make the training set more balanced. The results from adding 500 new samples to 50,000 new samples are shown in Sec. 6.1.
2We have tried a similar approach via gradient reversal w.r.t. BoW sub-model in preliminary experiments and observed less effectiveness (than HEX-projection), which hints that different types of biases can lead to different behaviors.
8761
Data Augmentation by Adding Synthetic Data.
Researchers have been using synthetic rules to generate harder or perturbed samples to fool the model. Here, besides using these datasets only as the evaluation set, we also add these samples back to the training set, similar to the concept of adversarial training (Jia and Liang, 2017; Wang et al., 2019c; Niu and Bansal, 2018) where the adversarial examples are added back to the training set so that the resulting model will be more robust to similar adversarial attacks. In our experiments, we follow Naik et al. (2018) to append meaningless sentences at the end of the hypothesis sentence like in Table 1 to create additional new samples. The detailed construction of these samples can be seen in Appendix. By learning from these augmented datasets, the model should also be more robust to certain types of perturbations/biases of the data.
In Sec. 6.1, our experiments showed that while this approach can lead to less biased models, it cannot make the model completely bias-free. Another disadvantage of these data enhancement/augmentation approaches is that we need to know all the specific kinds of biases in advance. For instance, in order to reduce the CWB for 'not', one needs carefully balance the samples containing 'not' in the training set. However, lots of other words will exhibit similar biases (e.g., the model tends to predict neutral when it sees 'also') and it is impractical to identify and debias the dataset w.r.t. every type of bias. Therefore, besides fixing the dataset, we should also focus on directly debiasing models against lexical biases.
4 Model-Level Debiasing
Model-level debiasing methods have the advantage that there is no need to know the specific bias type in advance. Here we propose two different methods. The first method focuses on debiasing the content of word/sentence embeddings, where we aim to remove strong bias in the embeddings towards any of the labels so that there will be fewer shortcuts for models to exploit. The second method builds a separate shallow bag-of-words (BoW) sub-model and projects the primary model's representation onto the subspace orthogonal to this BoW sub-model via the HEX projection algorithm (Wang et al., 2019a). Our proposed methods can be applied to a wide range of baseline model architectures. In addition, none of our methods is bias-type specific, so the results on CWB and WOB should generalize
to other similar lexical biases.
We use sentence-embedding based models as our baseline since they are more controllable, and because the interaction of sentences only appears at the top classifier, which makes it easier to compare the different effects of different regularization. Our baseline structures can be divided into three stages. The first stage is to embed the words into word embeddings. The second stage is to get the representations for each sentence. We use three layers of BiLSTM to get the representation. We also added residual and skip-connections as Nie et al. (2019), and find that it leads to better performance. For the final stage, our baseline follows Mou et al. (2016); Conneau et al. (2017) to concatenate these two sentence embeddings, their difference, and their element-wise product as follows:
$$
m = \left[ h _ {1}; h _ {2}; h _ {1} - h _ {2}; h _ {1} \odot h _ {2} \right] \tag {1}
$$
The resulting vector is passed through another multi-layer perceptron (MLP) to get the final classification result.
Next, we will describe two different methods to directly debias the model.
4.2 Debiasing Embeddings
Word embeddings are an important component in all neural NLP models. They contain the most basic semantics of words. Recent studies have shown that removing gender bias from word embeddings can lead to less biased models (Zhao et al., 2018a). In our work, as we discussed in Q2 in Sec. 1, we explore whether similar ideas can be applied to reducing lexical dataset biases.
For a large number of lexical dataset biases (e.g., CWB), the model tends to predict the label based only on the existence of certain words. Hence, one
Another popular choice of NLI model architecture is the cross-attention based models (Chen et al., 2017; Devlin et al., 2019). In our current work, we choose to only apply our BoW Sub-Model approach on sentence-embedding based models since our approach directly regularizes the representation vector learned by the main model, and hence it is most suitable for models with a single vector containing rich information. On the other hand, cross-attention based models do most of the inference through cross-attention and do not learn such a single vector, making it hard to regularize the model effectively in a similar way. Investigation of similar HEX regularization methods for cross-attention models is future work.
4Our baseline models achieve close to the best sentence embedding based/cross-attention based models reported on the NLI stress tests (Naik et al., 2018) and are hence good starting points for this bias/debias analysis.
8762
Figure 1: The overall architecture for reducing bias using an embedding debiasing network. The red dashed line denotes gradient reversal.
natural conjecture is that there is a strong bias towards some labels in the word embeddings. Since the label bias is not an attribute of the word, but it is brought in by the model above, hence in order to remove such label bias from the embeddings at training time, we differ from Zhao et al. (2018a) to use the gradient-reversal trick (Ganin and Lempitsky, 2015; Xie et al., 2017).
The architecture of this approach is illustrated in Figure 1. We denote the embeddings of the two input sequences for our model as  $\mathbf{w}^{(\mathbf{a})} = \{\mathbf{w}_1^{(a)},\mathbf{w}_2^{(a)},\ldots ,\mathbf{w}_{l_a}^{(a)}\}$  and  $\mathbf{w}^{(\mathbf{b})} = \{\mathbf{w}_1^{(b)},\mathbf{w}_2^{(b)},\ldots ,\mathbf{w}_{l_b}^{(b)}\}$  respectively, where  $a$  denotes the premise sentence while  $b$  denotes the hypothesis sentence. In order to apply the reverse gradient trick (Ganin and Lempitsky, 2015) to the embeddings, we add a small embedding-debias network (the left blue box in Figure 1) for each of the embedding  $w_{i}$  in our model. The embedding-debias network is a simple MLP. Since the other parts of the sentence context may also contribute to the bias, the debiasing network takes both  $w_{i}^{(a)}$  and the sentence embedding of  $b$  (and vice versa for debiasing  $w^{(b)}$ ) as the input and predicts the label  $y$ . Therefore, the total loss of this method is:
$$
L (\theta_ {c}, \theta_ {e}, \theta_ {e d}) = L _ {c} (\theta_ {c}, \theta_ {e}) - \frac {\lambda}{l _ {a} + l _ {b}} L _ {e d} (\theta_ {e}, \theta_ {e d})
$$
Here,  $\lambda$  is the multitask coefficient.  $l_{a}$  and  $l_{b}$  are the lengths of two input sentences.  $L_{c}$  is the standard classification loss using the main model and  $L_{ed}$  is the sum of all the classification loss using the debias network.  $\theta_{e}$  are parameters of the embeddings and sentence encoder of the main model,  $\theta_{c}$  are parameters of the top classifier of the main model, and  $\theta_{ed}$  are parameters of the embedding-debias network. In order to find the optimal parameters, we follow Ganin and Lempitsky (2015) to reverse the gradient for  $\theta_{e}$  w.r.t.  $L_{ed}$ .
Besides this approach, we also tried two variants by changing the input of the debias network. The first one is **embbasic**, where we only take the single embedding  $w_{i}$  as the input. The second one only takes one sentence embedding as the input and is called **ind_sent**. The results of our embedding-debias methods are shown in Sec. 6.2.
4.3 Bag-of-Words Sub-Model Orthogonality
While debiasing the embeddings can robustify the models against certain biases, it may not be effective for all the lexical biases. Some lexical bias may exist at the deeper compositionality level (e.g., WOB), while debiasing the embeddings can regularize only the most basic semantics units instead of how these semantics units are composed by the model. In addition, removing the label biases may also hurt the useful semantics contained in the embeddings, leading to significant performance drops. A better approach is to leave the embedding intact, but try to regularize how the classifier uses these features. We observe that models exploiting dataset biases in the training set (e.g., CWB and WOB) tend to use very simple and superficial features to make the prediction. These models tend to ignore the order of the words, fail to learn compositionality, and do not have a deep semantic understanding of the sentences. Therefore, we aim to robustify the model by letting it use fewer simple and superficial features. With this motivation, we train a bag-of-words (BoW) model that only captures superficial patterns of the words without any word order/compositionality information. Then we use HEX projection (Wang et al., 2019a) to project the representation of the original primary model to the orthogonal space of the representation of the BoW model.
BoW Model. For the BoW sub-model, we first get the embedding of all the words. Then, in order to capture more co-occurrence information of the words, we add a multi-head self-attention layer like the one used in Vaswani et al. (2017) (but without position embeddings), because we empirically find that this improves the performance. Finally, we use mean-pooling among all the vectors to get the BoW sentence-embedding:  $\mathbf{h}_{bow} = \frac{1}{l}\{\text{self\_att}(\mathbf{w})\}$ . To get a single representation for the sentence-pair, we used the same concatenation layer as in Eqn 1 and pass the vector through an additional MLP to get the representation  $u_{bow}$ .
HEX Projection. Next, in order to encourage the
8763
Figure 2: The overall architecture for debiasing the model via orthogonal projection w.r.t. BoW sub-model.