SlowGuess/ACL_2020_OCR huggingface dataset:
State-of-the-art Debiasing Models We compare our method against existing state-of-the-art debiasing methods: product-of-expert (He et al., 2019; Mahabadi and Henderson, 2019) and its variant learned-mixin (Clark et al., 2019a). product-of-expert ensembles the prediction of the main model  $(p_i)$  with the prediction of the biased model  $(b_i)$  using  $p_i' = \text{softmax}(\log p_i + \log b_i)$ , where  $p_i'$  is the ensembled output distribution. This ensembling enables the main model to focus on learning from examples that are not predicted well by the biased model. Learned-mixin improves this method by parameterizing the ensembling operation to let the model learn when to incorporate or ignore the output of the biased model for the ensembled prediction.
On FEVER, we also compare our method against the example-reweighting method by Schuster et al. (2019). They compute the importance weight of each example based on the correlation of the n-grams within the claim sentences with the target labels. These weights are then used to compute the loss of each training batch.
Training Details As observed by McCoy et al. (2019a), models can show high variance in their
out-of-distribution performance. Therefore, we run each experiment five times and report both average and standard deviation of the scores.<sup>5</sup> We also use training configurations that are known to work well for each task.<sup>6</sup> For each experiment, we train our confidence regularization method as well as product-of-expert and learned-mixin using the same biased-model. Since the challenge datasets often do not provide a development set, we could not tune the hyperparameter of learned-mixin. We, therefore, use their default weight for the entropy penalty term.<sup>7</sup>
The results for the tasks of NLI, fact verification, and paraphrase identification are reported in Table 2, Table 3, and Table 4, respectively.
5.1 In-distribution Performance
The results on the original development and test sets of each task represent the in-distribution performance. Since we examine two types of biases in NLI, we have two debiased NLI models, i.e., Regularized-conf_hans and Regularized-conf_hypo which are trained for debiasing HANS and hypothesis-only biases, respectively.
We make the following observations from the results: (1) Our method outperforms product-of-expert and learned-mixin when evaluated on the corresponding in-distribution data of all the three tasks; (2) Product-of-expert and learned-mixin drop the original BERT baseline accuracy on most
Due to the limited number of possible submissions, we report the MNLI test scores only from a model that holds the median out-of-distribution performance.
<sup>6</sup>We set a learning rate of  $5e^{-5}$  for MNLI and  $2e^{-5}$  for FEVER and QQP.
$^{7}$ E.g.,  $w = 0.03$  for training on MNLI.
8722
Method	FEVER dev	|Symm. v1	Symm. v2
BERT-base	85.8 ± 0.1	57.9 ± 1.1	64.4 ± 0.6
Learned-mixin claim	83.1 ± 0.7	60.4 ± 2.4	64.9 ± 1.6
Product-of-expert claim	83.3 ± 0.3	61.7 ± 1.5	65.5 ± 0.7
Reweighting bigrams	85.5 ± 0.3	61.7 ± 1.1	66.5 ± 1.3
Regularized-conf claim	86.4 ± 0.2	60.5 ± 0.4	66.2 ± 0.6
of the in-distribution experiments; (3) Regardless of the type of bias, our method preserves the indistribution performance. However, it is not the case for the other two methods, e.g., learned-mixin only results in a mild decrease in the accuracy when it is debiased for HANS, but suffers from substantial drop when it is used to address the hypothesis-only bias; (4) Our method results in a slight in-distribution improvement in some cases, e.g., on FEVER, it gains 0.6pp over BERT baseline. The models produced by Regularized-conf_hans also gain 0.1 points to both MNLI-m and MNLI-mm test sets; (5) All methods, including ours decrease the in-distribution performance on QQP, particularly on its duplicate examples subset. We will discuss this performance drop in Section 6.
5.2 Out-of-distribution Performance
The rightmost columns of each table report the evaluation results on the out-of-distribution datasets for each task. Based on our out-of-distribution evaluations, we observe that: (1) Our method minimizes the trade-off between the in-distribution and out-of-distribution performance compared to the other methods. For example, on HANS, learned-mixin maintains the in-distribution performance but only improves the average HANS accuracy from  $61.1\%$  to  $64.9\%$ . product-of-expert gains 7 points improvement over the BERT baseline while reducing the MNLI-m test accuracy by 1.6 points. On the other hand, our method achieves the competitive 7 points gain without dropping the in-distribution performance; (2) The performance trade-off is stronger on some datasets. On PAWS, the two compared methods improve the accuracy on the non-duplicate subset while reducing models' ability to detect the duplicate examples. Our method, on the other hand, finds a balance point, in which the non-duplicate accuracy can no longer be improved without reducing the duplicate accuracy; (3) depending on the use of hyperparameters, learned-mixin can make a lower
Table 3: Accuracy on the FEVER dataset and the corresponding challenge datasets.
Method	QQP dev	PAWS test
dupl	¬dupl	dupl	¬dupl
BERT-base	88.4 ± 0.3	92.5 ± 0.3	96.9 ± 0.3	9.8 ± 0.4
LMixin_hans	77.5 ± 0.7	91.9 ± 0.2	69.7 ± 4.3	51.7 ± 4.3
Prod-exp_hans	80.8 ± 0.2	93.5 ± 0.1	71.0 ± 2.3	49.9 ± 2.3
Reg-conf_hans	85.0 ± 0.7	91.5 ± 0.4	91.0 ± 1.8	19.8 ± 1.3
Table 4: Results of the evaluation on the QQP task.
out-of-distribution improvement compared to ours, even after substantially degrading in-distribution performance, e.g., on FEVER-symmetric $_{v2}$ , it only gains 0.5 points while dropping 3 points on the FEVER development set.
6 Discussions and Analysis
Ablation studies In this section, we show that the resulting improvements from our method come from the combination of both self-distillation and our scaling mechanism. We perform ablation studies to examine the impact of each of the components including (1) self-distillation: we train a model using the standard self-distillation without bias-weighted scaling, and (2) example-reweighting: we train a model with the standard cross-entropy loss with an example reweighting method to adjust the importance of individual examples to the loss. The weight of each example is obtained from the (scaled) probability that is assigned by the teacher model to the ground truth label. The aim of the second setting is to exclude the effect of self-distillation while keeping the effect of our scaling mechanism.
Table 5 presents the results of these experiments on MNLI and HANS. We observe that each component individually still gains substantial improvements on HANS over the baseline, albeit not as strong as the full method. The results from the self-distillation suggest that the improvement from our method partly comes from the regularization effect of the distillation objective (Clark et al., 2019b; Furlanello et al., 2018). In the example-reweighting experiment, we exclude the effect of all the scaled teacher's output except for the probability assigned to the ground truth label. Compared to self-distillation, the proposed example-reweighting has a higher impact on improving the performance in both in-distribution and out-of-distribution eval
Details of the ablation experiments are included in the supplementary materials.
Figure 2: Distribution of models' confidence on their predicted labels. The blue areas indicate the fraction of each bin that are correct. (a) Distribution on MNLI-m dev by models trained using hypothesis-only biased model. (b) Distribution on non-entailment subsequence subset of HANS by models trained using hans biased-model.
Method	MNLI	HANS
BERT-base	84.3	61.1
Full method	84.3	69.1
self-distillation	84.6	64.4
example-reweighting	84.7	65.3
Table 5: Results of the ablation experiments. The MNLI column refers to the MNLI-m dev set.
BERT-baseline	product-of-expert	learned-mixin	conf-reg(our)
MNLI-m	9.0	7.7	9.9	5.4
MNLI-mm	8.5	7.6	9.5	5.6
Table 6: The calibration scores of models measured by ECE (lower is better).
uations. However, both components are necessary for the overall improvements.
In-distribution performance drop of product-of-expert The difference between our method with product-of-expert and its variants is the use of biased examples during training. Product-of-expert in practice scales down the gradients on the biased training examples to allow the model to focus on learning from the harder examples (He et al., 2019). As a result, models often receive little to no incentive to solve these examples throughout the training, which can effectively reduce the training data size. Our further examination on a product-of-expert model (trained on MNLI for HANS) shows that its degradation of in-distribution performance largely comes from the aforementioned examples. Ensembling back the biased-model to the main
model can indeed bring the in-distribution accuracy back to the BERT baseline. However, this also leads to the original poor performance on HANS, which is counterproductive to the goal of improving the out-of-distribution generalization.
Impact on Models' Calibration We expect the training objective used in our method to discourage models from making overconfident predictions, i.e., assigning high probability to the predicted labels even when they are incorrect. We investigate the changes in models' behavior in terms of their confidence using the measure of calibration, which quantifies how aligned the confidence of the predicted labels with their actual accuracy are (Guo et al., 2017). We compute the expected calibration error (ECE) (Naeini et al., 2015) as a scalar summary statistic of calibration. Results in Table 6 show that our method improves model's calibration on MNLI-m and MNLI-mm dev sets, with the reduction of ECE ranging from 3.0 to 3.6. The histograms in figure 2 show the distribution of models' confidences in their predictions. Figure 2a demonstrates that the prediction confidences of our resulting model on MNLI-m are more smoothly distributed. In figure 2b, we observe that our debiased model predicts examples that contain lexical overlap features with lower confidence, and when the confidence is higher, the prediction is more likely to be correct.
Impact of biased examples ratio To investigate the slight in-distribution drop by our method in QQP (Table 4), we examine the ratio of biased examples in the QQP training data by evaluating the
Figure 3: Results on the PAWS-augmented QQP dataset.
performance of the biased model on the dataset. We find that almost  $80\%$  of the training examples can be solved using the lexical overlap features alone, which indicates a severe lexical overlap bias in QQP. Moreover, in  $53\%$  of all examples, the biased model makes correct predictions with a very high confidence  $(\beta_{i} > 0.8)$ . For comparison, the same biased model predicts only  $12\%$  of the MNLI examples with confidence above 0.8 (more comparisons are shown in the supplementary material. As a result, there are not enough unbiased examples in QQP and the resulting soft target labels in this dataset are mostly close to a uniform distribution, which in turn may provide insufficient training signal to maximize the accuracy on the training distribution.
Impact of adding bias-free examples Finally, we investigate how changing the ratio of biased examples affects the behavior of debiasing methods. To this end, we split PAWS data into training and test sets. The training set consists of 2500 examples, and we use the remaining 10K examples as a test set. We train the model on QQP that is gradually augmented with fractions of this PAWS training split and evaluate on a constant PAWS test set. Figure 3 shows the results of this experiment. When more PAWS examples are added to the training data, the accuracy of the BERT baseline gradually improves on the non-duplicate subset while its accuracy slowly drops on the duplicate subset. We observe that product-of-expert exaggerates this effect: it reduces the duplicate accuracy up
to  $40\%$  to obtain the  $93\%$  non-duplicate accuracy. We note that our method is the most effective when the entire 2500 PAWS examples are included in the training, obtaining the overall accuracy of  $77.05\%$  compared to the  $71.63\%$  from the baseline BERT.
Existing debiasing methods improve the performance of NLU models on out-of-distribution datasets. However, this improvement comes at the cost of strongly diminishing the training signal from a subset of the original dataset, which in turn reduces the in-distribution accuracy. In this paper, we address this issue by introducing a novel method that regularizes models' confidence on biased examples. This method allows models to still learn from all training examples without exploiting the biases. Our experiments on four out-of-distribution datasets across three NLU tasks show that our method provides a competitive out-of-distribution performance while preserves the original accuracy.
Our debiasing framework is general and can be extended to other task setups where the biases leveraged by models are correctly identified. Several challenges in this direction of research may include extending the debiasing methods to overcome multiple biases at once or to automatically identify the format of those biases which simulate a setting where the prior knowledge is unavailable.
We thank Leonardo Ribeiro and Max Glockner for the thoughtful discussion on the earlier version of this work and the anonymous reviewers for their constructive comments. We also thank Tal Schuster for the support in using the Fever-Symmetric dataset. This work is supported by the German Research Foundation through the research training group "Adaptive Preparation of Information from Heterogeneous Sources" (AIPHES, GRK 1994/1) and by the German Federal Ministry of Education and Research and the Hessian State Ministry for Higher Education, Research and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE.
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. Analyzing the behavior of visual question an
The random baseline is  $50\%$  for QQP.
8725
swering models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1955-1960, Austin, Texas. Association for Computational Linguistics.
Yonatan Belinkov, Adam Poliak, Stuart M. Shieber, Benjamin Van Durme, and Alexander M. Rush. 2019. On adversarial removal of hypothesis-only bias in natural language inference. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics, *SEM@NAACL-HLT 2019, Minneapolis, MN, USA, June 6-7, 2019, pages 256-262. Association for Computational Linguistics.
Emily Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, page to appear, virtual conference. Association for Computational Linguistics.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019a. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4067-4080, Hong Kong, China. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. 2019b. BAM! born-again multi-task networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5931-5937, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220, Florence, Italy. Association for Computational Linguistics.
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719-3728, Brussels, Belgium. Association for Computational Linguistics.
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born-again neural networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1602-1611. PMLR.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP, DeepLo@EMNLP-IJCNLP 2019, Hong Kong, China, November 3, 2019, pages 132–142. Association for Computational Linguistics.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In 8th International Conference on Learning Representations, ICLR 2020, Virtual Conference, 26 April - 1 May, 2019. OpenReview.net.
Rabeeh Karimi Mahabadi and James Henderson. 2019. Simple but effective techniques to reduce biases. CoRR, abs/1909.06321.
R Thomas McCoy, Junghyun Min, and Tal Linzen. 2019a. Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance. arXiv preprint arXiv:1911.02969.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019b. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
8726
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pages 2901-2907. AAAI Press.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. 2016. Distillation as a defense to adversarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy, SP 2016, San Jose, CA, USA, May 22-26, 2016, pages 582-597. IEEE Computer Society.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. WINOGRANDE: an adversarial winograd schema challenge at scale. CoRR, abs/1907.10641.
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3417-3423, Hong Kong, China. Association for Computational Linguistics.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2818-2826. IEEE Computer Society.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018. The fact extraction and VERIFICATION (FEVER) shared task. In Proceedings of the First Workshop on
Fact Extraction and VERIFICATION (FEVER), pages 1-9, Brussels, Belgium. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791-4800. Association for Computational Linguistics.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298-1308, Minneapolis, Minnesota. Association for Computational Linguistics.
8727
A Ablation Details
For the second setting of our ablation studies, we perform an example reweighting using the scaled probability of the teacher model  $\mathcal{F}_t$  on the ground truth label. Specifically, the cross entropy loss assigned to each batch of size  $m$  is computed by the following:
$$
- \sum_ {s = 1} ^ {b} \frac {\hat {p _ {s , c}}}{\sum_ {u = 1} ^ {b} \hat {p _ {u , c}}} \cdot \log (p _ {s, c})
$$
where we assume that  $c$ th label is the ground truth label. The probability assigned to the correct label by the teacher model is then denoted as  $p_{s,c}$ . The current predicted probability of the main model is denoted as  $p_{s,c}$ .
B Bias Weights Distribution