SlowGuess/ACL_2020_OCR huggingface dataset:
Rabeeh Karimi Mahabadi\* Yonatan BelinkovJames Henderson
$\hat{\mathbb{C}}$  EPFL, Switzerland
\*Idiap Research Institute, Switzerland
$^{\diamond}$ Harvard University and Massachusetts Institute of Technology, Cambridge, MA, USA
{rabeeh.karimi, james.henderson}@idiap.ch
belinkov@seas.harvard.edu
Several recent studies have shown that strong natural language understanding (NLU) models are prone to relying on unwanted dataset biases without learning the underlying task, resulting in models that fail to generalize to out-of-domain datasets and are likely to perform poorly in real-world scenarios. We propose two learning strategies to train neural models, which are more robust to such biases and transfer better to out-of-domain datasets. The biases are specified in terms of one or more bias-only models, which learn to leverage the dataset biases. During training, the bias-only models' predictions are used to adjust the loss of the base model to reduce its reliance on biases by down-weighting the biased examples and focusing training on the hard examples. We experiment on large-scale natural language inference and fact verification benchmarks, evaluating on out-of-domain datasets that are specifically designed to assess the robustness of models against known biases in the training data. Results show that our debiasing methods greatly improve robustness in all settings and better transfer to other textual entailment datasets. Our code and data are publicly available in https://github.com/rabeehk/robust-nli.
Recent neural models (Devlin et al., 2019; Radford et al., 2018; Chen et al., 2017) have achieved high and even near human-performance on several large-scale natural language understanding benchmarks. However, it has been demonstrated that neural models tend to rely on existing idiosyncratic biases in the datasets, and leverage superficial correlations between the label and existing shortcuts in the training dataset to perform surprisingly well, without learning the underlying task (Kaushik and Lipton, 2018; Gururangan et al., 2018; Poliak et al., 2018; Schuster et al., 2019;
McCoy et al., 2019b). For instance, natural language inference (NLI) is supposed to test the ability of a model to determine whether a hypothesis sentence (There is no teacher in the room) can be inferred from a premise sentence (Kids work at computers with a teacher's help) (Dagan et al., 2006). However, recent work has demonstrated that large-scale NLI benchmarks contain annotation artifacts; certain words in the hypothesis that are highly indicative of inference class and allow models that do not consider the premise to perform unexpectedly well (Poliak et al., 2018; Gururangan et al., 2018). As an example, in some NLI benchmarks, negation words such as "nobody", "no", and "not" in the hypothesis are often highly correlated with the contradiction label.
As a result of the existence of such biases, models exploiting statistical shortcuts during training often perform poorly on out-of-domain datasets, especially if the datasets are carefully designed to limit the spurious cues. To allow proper evaluation, recent studies have tried to create new evaluation datasets that do not contain such biases (Gururangan et al., 2018; Schuster et al., 2019; McCoy et al., 2019b). Unfortunately, it is hard to avoid spurious statistical cues in the construction of large-scale benchmarks, and collecting new datasets is costly (Sharma et al., 2018). It is, therefore, crucial to develop techniques to reduce the reliance on biases during the training of the neural models.
We propose two end-to-end debiasing techniques that can be used when the existing bias patterns are identified. These methods work by adjusting the cross-entropy loss to reduce the biases learned from the training dataset, down-weighting the biased examples so that the model focuses on learning the hard examples. Figure 1 illustrates an example of applying our strategy to prevent an NLI model from predicting the labels using existing biases in the hypotheses, where the bias-only model only sees the hypothesis. Our strat
2The given sentences are in the contradictory relation, and the hypothesis cannot be inferred from the premise.
<sup>1</sup>We use biases, heuristics or shortcuts interchangeably.
Figure 1: An illustration of our debiasing strategies applied to an NLI model. The bias-only model only sees the hypothesis, where negation words like "not" are highly correlated with the contradiction label. We train a robust NLI model by training it in combination with the bias-only model and motivate it to learn different strategies than the ones used in the bias-only model. The robust NLI model does not rely on the shortcuts and obtains improved performance on the test set.
egy involves adding this bias-only branch  $f_{B}$  on top of the base model  $f_{M}$  during training. We then compute the combination of the two models  $f_{C}$  in a way that motivates the base model to learn different strategies than the ones used by the bias-only branch  $f_{B}$ . At the end of the training, we remove the bias-only classifier and use the predictions of the base model.
In our first proposed method, Product of Experts, the training loss is computed on an ensemble of the base model and the bias-only model, which reduces the base model's loss for the examples that the bias-only model classifies correctly. For the second method, Debiased Focal Loss, the bias-only predictions are used to directly weight the loss of the base model, explicitly modulating the loss depending on the accuracy of the bias-only model. We also extend these methods to be robust against multiple sources of bias by training multiple bias-only models.
Our approaches are simple and highly effective. They require training only a simple model on top of the base model. They are model agnostic and general enough to be applicable for addressing common biases seen in many datasets in different domains.
We evaluate our models on challenging benchmarks in textual entailment and fact verification, including HANS (Heuristic Analysis for NLI Systems) (McCoy et al., 2019b), hard NLI sets (Gururangan et al., 2018) of Stanford Natural Language Inference (SNLI) (Bowman et al., 2015) and MultiNLI (MNLI) (Williams et al., 2018), and FEVER Symmetric test set (Schuster et al., 2019). The selected datasets are highly challenging and have been carefully designed to be unbiased to allow proper evaluation of the out-of-domain performance of the models. We ad
ditionally construct hard MNLI datasets from MNLI development sets to facilitate the out-of-domain evaluation on this dataset.3 We show that including our strategies on training baseline models, including BERT (Devlin et al., 2019), provides a substantial gain on out-of-domain performance in all the experiments.
In summary, we make the following contributions: 1) Proposing two debiasing strategies to train neural models robust to dataset bias. 2) An empirical evaluation of the methods on two large-scale NLI datasets and a fact verification benchmark; obtaining a substantial gain on their challenging out-of-domain data, including 7.4 points on HANS, 4.8 points on SNLI hard set, and 9.8 points on FEVER symmetric test set, setting a new state-of-the-art. 3) Proposing debiasing strategies capable of combating multiple sources of bias. 4) Evaluating the transfer performance of the debiased models on 12 NLI datasets and demonstrating improved transfer to other NLI benchmarks. To facilitate future work, we release our datasets and code.
To address dataset biases, researchers have proposed to augment datasets by balancing the existing cues (Schuster et al., 2019) or to create an adversarial dataset (Jia and Liang, 2017). However, collecting new datasets, especially at a large scale, is costly, and thus remains an unsatisfactory solution. It is, therefore, crucial to develop strategies to allow models to be trained on the existing biased datasets.
3Removing the need to submit to an online evaluation system for MNLI hard test sets.
Schuster et al. (2019) propose to first compute the n-grams in the dataset's claims that are the most associated with each fact-verification label. They then solve an optimization problem to assign a balancing weight to each training sample to alleviate the biases. In contrast, we propose several end-to-end debiasing strategies. Additionally, Belinkov et al. (2019a) propose adversarial techniques to remove from the NLI sentence encoder the features that allow a hypothesis-only model to succeed. However, we believe that in general, the features used by the hypothesis-only model can include some information necessary to perform the NLI task, and removing such information from the sentence representation can hurt the performance of the full model. Their approach consequently degrades the performance on the hard SNLI set, which is expected to be less biased. In contrast, we propose to train a bias-only model to use its predictions to dynamically adapt the classification loss to reduce the importance of the most biased examples.
Concurrently to our work, Clark et al. (2019) and He et al. (2019) have also proposed to use the product of experts (PoE) models for avoiding biases. They train their models in two stages, first training a bias-only model and then using it to train a robust model. In contrast, our methods are trained in an end-to-end manner, which is convenient in practice. We additionally show that our proposed Debiased Focal Loss model is an effective method to reduce biases, sometimes superior to PoE. We have evaluated on new domains of NLI hard sets and fact verification. Moreover, we have included an analysis showing that our debiased models indeed have lower correlations with the bias-only models, and have extended our methods to guard against multiple bias patterns simultaneously. We furthermore study transfer performance to other NLI datasets.
3 Reducing Biases
Problem formulation We consider a general multi-class classification problem. Given a dataset  $\mathcal{D} = \{\pmb{x}_i, y_i\}_{i=1}^N$  consisting of the input data  $\pmb{x}_i \in \mathcal{X}$  and labels  $y_i \in \mathcal{Y}$ , the goal of the base model is to learn a mapping  $f_M$  parameterized by  $\theta_M$  that computes the predictions over the label space given the input data, shown as  $f_M: \mathcal{X} \to \mathbb{R}^{|\mathcal{Y}|}$ . Our goal is to optimize  $\theta_M$  parameters such that we build a model that is more resistant to benchmark dataset biases, to improve its robustness to domain changes where the biases typically observed in the training data do not exist in the evaluation dataset.
The key idea of our approach, depicted in Figure 1, is first to identify the dataset biases that the base model is susceptible to relying on, and define a bias-only model to capture them. We then propose two strategies to incorporate this bias-only knowledge into the training of the base model to make it robust against the biases. After training, we remove the bias-only model and use the predictions of the base model.
3.1 Bias-only Branch
We assume that we do not have access to any data from the out-of-domain dataset, so we need to know a priori about the possible types of shortcuts we would like the base model to avoid relying on. Once these patterns are identified, we train a bias-only model designed to capture the identified shortcuts that only uses biased features. For instance, a hypothesis-only model in the large-scale NLI datasets can correctly classify the majority of samples using annotation artifacts (Poliak et al., 2018; Gururangan et al., 2018). Motivated by this work, our bias-only model for NLI only uses hypothesis sentences. Note that the bias-only model can, in general, have any form, and is not limited to models using only a part of the input data. For instance, on the HANS dataset, our bias-only model makes use of syntactic heuristics and similarity features (see Section 4.3).
Let  $\pmb{x}_i^b \in \mathcal{X}^b$  be biased features of  $\pmb{x}_i$  that are predictive of  $y_i$ . We then formalize this bias-only model as a mapping  $f_B: \mathcal{X}^b \to \mathbb{R}^{|\mathcal{Y}|}$ , parameterized by  $\theta_B$  and trained using cross-entropy (CE) loss  $\mathcal{L}_B$ :
$$
\mathcal {L} _ {B} \left(\theta_ {B}\right) = - \frac {1}{N} \sum_ {i = 1} ^ {N} \log \left(\sigma \left(f _ {B} ^ {y _ {i}} \left(\boldsymbol {x} _ {i} ^ {\boldsymbol {b}}; \theta_ {B}\right)\right)\right), \tag {1}
$$
where  $f_B^j(\pmb{x}_i^{\pmb{b}}, \theta_B)$  is the  $j$ th element of  $f_B(.)$ , and  $\sigma(u^j) = e^{u^j} / \sum_{k=1}^{|\mathcal{V}|} e^{u^k}$  is the softmax function.
3.2 Proposed Debiasing Strategies
We propose two strategies to incorporate the bias-only  $f_{B}$  knowledge into the training of the base model  $f_{M}$ . In our strategies, the predictions of the bias-only model are combined with either the predictions of the base model or its error, to down-weight the loss for the examples that the bias-only model can predict correctly. We then update parameters of the base model  $\theta_{M}$  based on this modified loss  $\mathcal{L}_C$ . Our learning strategies are end-to-end. Therefore, to prevent the base model from learning the biases, the bias-only loss  $\mathcal{L}_B$  is not back-propagated to any shared parameters of the base model, such as a shared sentence encoder.
3.2.1 Method 1: Product of Experts
Our first approach is based on the product of experts (PoE) method (Hinton, 2002). Here, we use this method to combine the bias-only and base model's predictions by computing the element-wise product  $\odot$  between their predictions as  $\sigma(f_{B}(\boldsymbol{x}_{i}^{\boldsymbol{b}}))\odot\sigma(f_{M}(\boldsymbol{x}_{i}))$ . We compute this combination in the logarithmic space, making it appropriate for the normalized exponential below:
$$
f _ {C} (\pmb {x} _ {i}, \pmb {x} _ {i} ^ {\pmb {b}}) = \log (\sigma (f _ {B} (\pmb {x} _ {i} ^ {\pmb {b}}))) + \log (\sigma (f _ {M} (\pmb {x} _ {i}))),
$$
The key intuition behind this model is to combine the probability distributions of the bias-only and the base model to allow them to make predictions based on different characteristics of the input; the bias-only branch covers prediction based on biases, and the base model focuses on learning the actual task. Then the base model parameters  $\theta_{M}$  are trained using the cross-entropy loss  $\mathcal{L}_C$  of the combined classifier  $f_{C}$ :
$$
\mathcal {L} _ {C} \left(\theta_ {M}; \theta_ {B}\right) = - \frac {1}{N} \sum_ {i = 1} ^ {N} \log \left(\sigma \left(f _ {C} ^ {y _ {i}} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} ^ {\boldsymbol {b}}\right)\right)\right). \tag {2}
$$
When updating the base model parameters using this loss, the predictions of the bias-only model decrease the updates for examples that it can accurately predict.
Justification: Probability of label  $y_{i}$  for the example  $x_{i}$  in the PoE model is computed as:
$$
\sigma (f _ {C} ^ {y _ {i}} (\pmb {x} _ {i}, \pmb {x} _ {i} ^ {\pmb {b}})) = \frac {\sigma (f _ {B} ^ {y _ {i}} (\pmb {x} _ {i} ^ {\pmb {b}})) \sigma (f _ {M} ^ {y _ {i}} (\pmb {x} _ {i}))}{\sum_ {k = 1} ^ {| \mathcal {Y} |} \sigma (f _ {B} ^ {k} (\pmb {x} _ {i} ^ {\pmb {b}})) \sigma (f _ {M} ^ {k} (\pmb {x} _ {i}))}
$$
Then the gradient of cross-entropy loss of the combined classifier (2) w.r.t  $\theta_{M}$  is (Hinton, 2002):
$$
\begin{array}{l} \nabla_ {\theta_ {M}} \mathcal {L} _ {C} (\theta_ {M}; \theta_ {B}) = - \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {k = 1} ^ {| \mathcal {V} |} \Big [ \\ \left. \left(\delta_ {y _ {i} k} - \sigma (f _ {C} ^ {k} (\pmb {x _ {i}}, \pmb {x _ {i} ^ {b}}))\right) \nabla_ {\theta_ {M}} \log (\sigma (f _ {M} ^ {k} (\pmb {x _ {i}}))) \right], \\ \end{array}
$$
where  $\delta_{y_i k}$  is 1 when  $k = y_i$  and 0 otherwise. Generally, the closer the ensemble's prediction  $\sigma(f_C^k(.))$  is to the target  $\delta_{y_i k}$ , the more the gradient is decreased through the modulating term, which only happens when the bias-only and base models are both capturing biases.
In the extreme case, when the bias-only model correctly classifies the sample,  $\sigma(f_{C}^{y_i}(\pmb{x}_i, \pmb{x}_i^{\pmb{b}})) = 1$  and therefore  $\nabla_{\theta_M} \mathcal{L}_C(\theta_M; \theta_B) = 0$ , the biased examples are ignored during training. Conversely, when the example is fully unbiased, the bias-only
classifier predicts the uniform distribution over all labels  $\sigma(f_{B}^{k}(\pmb{x}_{i}^{b})) = \frac{1}{|\mathcal{Y}|}$  for  $k \in \mathcal{V}$ , therefore  $\sigma(f_{C}^{y_{i}}(\pmb{x}_{i},\pmb{x}_{i}^{b})) = \sigma(f_{M}^{y_{i}}(\pmb{x}_{i}))$  and the gradient of ensemble classifier remains the same as the CE loss.
3.2.2 Method 2: Debiased Focal Loss
Focal loss was originally proposed in Lin et al. (2017) to improve a single classifier by down-weighting the well-classified points. We propose a novel variant of this loss that leverages the bias-only branch's predictions to reduce the relative importance of the most biased examples and allows the model to focus on learning the hard examples. We define Debiased Focal Loss (DFL) as:
$$
\begin{array}{l} \mathcal {L} _ {C} \left(\theta_ {M}; \theta_ {B}\right) = \tag {3} \\ - \frac {1}{N} \sum_ {i = 1} ^ {N} \left(1 - \sigma \left(f _ {B} ^ {y _ {i}} \left(\boldsymbol {x} _ {i} ^ {\boldsymbol {b}}\right)\right)\right) ^ {\gamma} \log \left(\sigma \left(f _ {M} ^ {y _ {i}} \left(\boldsymbol {x} _ {i}\right)\right)\right) \\ \end{array}
$$
where  $\gamma$  is the focusing parameter, which impacts the down-weighting rate. When  $\gamma$  is set to 0, DFL is equivalent to the cross-entropy loss. For  $\gamma > 0$ , as the value of  $\gamma$  is increased, the effect of down-weighting is increased. We set  $\gamma = 2$  through all experiments, which works well in practice, and avoid fine-tuning it further. We note the properties of this loss: (1) When the example  $\pmb{x}_i$  is unbiased, and the bias-only branch does not do well,  $\sigma(f_B^{y_i}(\pmb{x}_i^b))$  is small, therefore the scaling factor is close to 1, and the loss remains unaffected. (2) As the sample is more biased and  $\sigma(f_B^{y_i}(\pmb{x}_i^b))$  is closer to 1, the modulating factor approaches 0 and the loss for the most biased examples is down-weighted.
3.3 RUBi baseline (Cadene et al., 2019)
We compare our models to RUBi (Cadene et al., 2019), a recently proposed model to alleviate unimodal biases learned by Visual Question Answering (VQA) models. Cadene et al. (2019)'s study is limited to VQA datasets. We, however, evaluate the effectiveness of their formulation on multiple challenging NLU benchmarks. RUBi consists in first applying a sigmoid function  $\phi$  to the bias-only model's predictions to obtain a mask containing an importance weight between 0 and 1 for each label. It then computes the element-wise product between the obtained mask and the base model's predictions:
$$
f _ {C} (\pmb {x} _ {i}, \pmb {x} _ {i} ^ {\pmb {b}}) = f _ {M} (\pmb {x} _ {i}) \odot \phi (f _ {B} (\pmb {x} _ {i} ^ {\pmb {b}})),
$$
The main intuition is to dynamically adjust the predictions of the base model to prevent it from leveraging the shortcuts. Then the parameters of the
base model  $\theta_{M}$  are updated by back-propagating the cross-entropy loss  $\mathcal{L}_C$  of the combined classifier.
3.4 Joint Debiasing Strategies
Neural models can, in practice, be prone to multiple types of biases in the datasets. We, therefore, propose methods for combining several bias-only models. To avoid learning relations between biased features, we do not consider training a classifier on top of their concatenation.
Instead, let  $\{x_i^{b_j}\}_{j = 1}^K$  be different sets of biased features of  $x_{i}$  that are predictive of  $y_{i}$ , and let  $f_{B_j}$  be an individual bias-only model capturing  $x_{i}^{b_{j}}$ . Next, we extend our debiasing strategies to handle multiple bias patterns.
Method 1: Joint Product of Experts We extend our proposed PoE model to multiple bias-only models by computing the element-wise product between the predictions of bias-only models and the base model as:  $\sigma(f_{B_1}(\boldsymbol{x}_i^{b_1})) \odot \dots \odot \sigma(f_{B_K}(\boldsymbol{x}_i^{b_K})) \odot \sigma(f_M(\boldsymbol{x}_i))$ , computed in the logarithmic space:
$$
\begin{array}{l} f _ {C} \left(\boldsymbol {x} _ {i}, \left\{\boldsymbol {x} _ {i} ^ {\boldsymbol {b} _ {j}} \right\} _ {j = 1} ^ {K}\right) = \sum_ {j = 1} ^ {K} \log \left(\sigma \left(f _ {B _ {j}} \left(\boldsymbol {x} _ {i} ^ {\boldsymbol {b} _ {j}}\right)\right)\right) \\ + \log (\sigma (f _ {M} (\boldsymbol {x} _ {\boldsymbol {i}}))). \\ \end{array}
$$
Then the base model parameters  $\theta_{M}$  are trained using the cross-entropy loss of the combined classifier  $f_{C}$ .
Method 2: Joint Debiased Focal Loss To extend DFL to handle multiple bias patterns, we first compute the element-wise average of the predictions of the multiple bias-only models:  $f_{B}(\{\pmb{x}_{i}^{b_{j}}\}_{j=1}^{K}) = \frac{1}{K}\sum_{j=1}^{K}f_{B_{j}}(\pmb{x}_{i}^{b_{j}})$ , and then compute the DFL (3) using the computed joint bias-only model.
4 Evaluation on Unbiased Datasets
We provide experiments on a fact verification (FEVER) and two large-scale NLI datasets (SNLI and MNLI). We evaluate the models' performance on recently-proposed challenging unbiased evaluation sets. We use the BERT (Devlin et al., 2019) implementation of Wolf et al. (2019) as our main baseline, known to work well for these tasks. In all the experiments, we use the default hyperparameters of the baselines.
4.1 Fact Verification
Dataset: The FEVER dataset contains claim-evidence pairs generated from Wikipedia. Schuster
et al. (2019) collected a new evaluation set for the FEVER dataset to avoid the idiosyncrasies observed in the claims of this benchmark. They made the original claim-evidence pairs of the FEVER evaluation dataset symmetric, by augmenting them and making each claim and evidence appear with each label. Therefore, by balancing the artifacts, relying on statistical cues in claims to classify samples is equivalent to a random guess. The collected dataset is challenging, and the performance of the models relying on biases evaluated on this dataset drops significantly.
Base models: We consider BERT as the base model, which works the best on this dataset (Schuster et al., 2019), and predicts the relations based on the concatenation of the claim and the evidence with a delimiter token (see Appendix A).
Bias-only model: The bias-only model predicts the labels using only claims as input.
Results: Table 1 shows the results. Our proposed debiasing methods, PoE and DFL, are highly effective, boosting the performance of the baseline by 9.8 and 7.5 points respectively, significantly surpassing the prior work of Schuster et al. (2019).
Loss	Dev	Test	Δ
CE	85.99	56.49
RUBi	86.23	57.60	+1.1
Schuster et al. (2019)	84.6	61.6	+5.1
DFL	83.07	64.02	+7.5
PoE	86.46	66.25	+9.8
Table 1: Results on FEVER development and symmetric test set.  $\Delta$  are absolute differences with CE loss.
4.2 Natural Language Inference
Datasets: We evaluate on hard datasets of SNLI and MNLI (Gururangan et al., 2018), which are the splits of these datasets where a hypothesis-only model cannot correctly predict the labels. Gururangan et al. (2018) show that the success of the recent textual entailment models is attributed to the biased examples, and the performance of these models is substantially lower on the hard sets.
Base models: We consider BERT and InferSent (Conneau et al., 2017) as our base models. We choose InferSent to be able to compare with the prior work of Belinkov et al. (2019b).
Bias-only model: The bias-only model predicts the labels using the hypothesis (Appendix B).
Results on SNLI: Table 2 shows the SNLI results. With InferSent, DFL and PoE result in 4.1 and 4.8 points gain. With BERT, DFL and PoE improve the results by 2.5 and 1.6 absolute points. Compared to the prior work of Belinkov et al. (2019b) (AdvCls), our PoE model obtains a 7.4 points gain, setting a new state-of-the-art.