SlowGuess/ACL_2020_OCR huggingface dataset:
Loss	BERT	InferSent
Test	Hard	Δ	Test	Hard	Δ
CE	90.53	80.53	84.24	68.91
RUBi	90.69	80.62	+0.1	83.93	69.64	+0.7
AdvCls*	—	—	—	83.56	66.27	-2.6
AdvDat*	—	—	—	78.30	55.60	-13.3
DFL	89.57	83.01	+2.5	73.54	73.05	+4.1
PoE	90.11	82.15	+1.6	80.35	73.69	+4.8
Results on MNLI: We construct hard sets from the validation sets of MNLI Matched and Mismatched (MNLI-M). Following Gururangan et al. (2018), we train a fastText classifier (Joulin et al., 2017) that predicts the labels using only the hypothesis and consider the subset on which it fails as hard examples.
We report the results on MNLI mismatched sets in Table 3 (see Appendix B for similar results on MNLI matched). With BERT, DFL and PoE obtain 1.4 and 1.7 points gain on the hard development set, while with InferSent, they improve the results by 2.5 and 2.6 points. To comply with limited access to the MNLI submission system, we evaluate only the best result of the baselines and our models on the test sets. Our PoE model improves the performance on the hard test set by 1.1 points while retaining in-domain accuracy.
Table 2: Results on the SNLI test, hard set, and differences with CE loss. *: results from Belinkov et al. (2019b).
Loss	BERT	InferSent
MNLI	Hard	Δ	MNLI	Hard	Δ
Development set results
CE	84.53	77.55	69.99	56.53
RUBi	85.17	78.63	+1.1	70.53	58.08	+1.5
DFL	84.85	78.92	+1.4	61.12	59.05	+2.5
PoE	84.85	79.23	+1.7	65.85	59.14	+2.6
Test set results
CE	83.51	75.75	—	—	—
PoE	83.47	76.83	+1.1	—	—	—
4.3 Syntactic Bias in NLI
Dataset: McCoy et al. (2019b) show that NLI models trained on MNLI can adopt superficial syntactic heuristics. They introduce HANS, consisting of several examples on which the syntactic heuristics fail.
Base model: We use BERT as our base model and train it on the MNLI dataset.
Bias-only model: We consider the following features for the bias-only model. The first four features are based on the syntactic heuristics proposed in McCoy et al. (2019b): 1) Whether all words in the hypothesis are included in the premise; 2) If the hypothesis is the contiguous subsequence of the premise; 3) If the hypothesis is a subtree in the premise's parse tree; 4) The number of tokens shared between premise and hypothesis normalized by the number of tokens in the premise. We additionally include some similarity features: 5) The cosine similarity between premise and hypothesis's pooled token representations from BERT followed by min, mean, and max-pooling. We consider the same weight for contradiction and neutral labels in the bias-only loss to allow the model to recognize entailment from not-entailment. During the evaluation, we map the neutral and contradiction labels to not-entailment.
Results: McCoy et al. (2019a) observe large variability in the linguistic generalization of neural models. We, therefore, report the averaged results across 4 runs with the standard deviation in Table 4. PoE and DFL obtain 4.4 and 7.4 points gain (see Appendix C for accuracy on individual heuristics of HANS).
Table 3: Results on MNLI mismatched benchmark and MNLI mismatched hard set.  $\Delta$  are absolute differences with CE loss.
Loss	MNLI	HANS	Δ
CE	84.51	61.88 ±1.9
RUBi	84.53	61.76±2.7	-0.1
Reweight	83.54	69.19	+7.3
Learned-Mixin	84.29	64.00	+2.1
Learned-Mixin+H	83.97	66.15	+4.3
PoE	84.19	66.31±0.6	+4.4
DFL	83.95	69.26 ±0.2	+7.4
DFL*	82.76	71.95±1.4	+10.1
Table 4: Results on MNLI Matched dev set and HANS.  $\clubsuit$ : results from Clark et al. (2019).  $\clubsuit$ : perform hyper-parameter tuning.  $\Delta$  are differences with CE loss.
We compare our results with the concurrent work of Clark et al., who propose a PoE model similar to ours, which gets similar results. The main difference is that our models are trained end-to-end, which is
convenient in practice, while Clark et al.'s method requires two steps, first training a bias-only model and then using this pre-trained model to train a robust model. The Reweight baseline in Clark et al. is a special case of our DFL with  $\gamma = 1$  and performs similarly to our DFL method (using default  $\gamma = 2$ ). Their Learned-Mixin+H method requires hyperparameter tuning. Since the assumption is not having access to any out-of-domain test data, and there is no available dev set for HANS, it is challenging to perform hyper-parameter tuning. Clark et al. follow prior work (Grand and Belinkov, 2019; Ramakrishnan et al., 2018) and perform model section on the test set.
To provide a fair comparison, we consequently also tuned  $\gamma$  in DFL by sweeping over  $\{0.5,1,2,3,4\}$ . DFL is the selected model, with  $\gamma = 3$ . With this hyperparameter tuning, DFL is even more effective, and our best result performs 2.8 points better than Clark et al. (2019).
4.4 Jointly Debiasing Multiple Bias Patterns
To evaluate combating multiple bias patterns, we jointly debias a base model on the hypothesis artifacts and syntactic biases.
Loss	MNLI	Hard	Δ	HANS	Δ
CE	84.53	77.55	61.88±1.9
PoE♣	84.85	79.23	+1.7	60.43	-1.5
DFL♣	84.85	78.92	+1.4	60.63	-1.2
PoE♥	84.55	77.90±0.3	+0.4	66.31±0.6	+4.4
DFL♥	84.30	77.66±0.6	+0.1	69.26±0.2	+7.4
PoE-Joint	84.39	78.61±0.1	+1.1	68.04±1.2	+6.2
DFL-Joint	84.49	78.36±0.4	+0.8	69.10±0.7	+7.2
Table 5: Results on MNLI mismatched dev set, MNLI mismatched hard set, and HANS when training independently to debias against either hypothesis artifacts (♣) or syntactic biases (♥), compared with jointly training to debias against both bias types. △: differences with baseline CE loss.
Bias-only models: We use the hypothesis-only and syntactic bias-only models as in Sections 4.2 and 4.3.
Results: Table 5 shows the results. Models trained to be robust to hypothesis biases  $(\clubsuit)$  do not generalize to HANS. On the other hand, models trained to be robust on HANS  $(\heartsuit)$  use a powerful bias-only model resulting in a slight improvement on MNLI mismatched hard dev set. We expect a slight degradation when debiasing for both biases
since models need to select samples accommodating both debiasing needs. The jointly debiased models successfully obtain improvements on both datasets, which are close to the improvements on each dataset by the individually debiased models.
5 Transfer Performance
To evaluate how well the baseline and proposed models generalize to solving textual entailment in domains that do not share the same annotation biases as the large NLI training sets, we take trained NLI models and test them on several NLI datasets.
Datasets: We consider a total of 12 different NLI datasets. We use the 11 datasets studied by Poliak et al. (2018). These datasets include MNLI, SNLI, SciTail (Khot et al., 2018), AddOneRTE (ADD1) (Pavlick and Callison-Burch, 2016), Johns Hopkins Ordinal Commonsense Inference (JOCI) (Zhang et al., 2017), Multiple Premise Entailment (MPE) (Lai et al., 2017), Sentences Involving Compositional Knowledge (SICK) (Marelli et al., 2014), and three datasets from White et al. (2017) which are automatically generated from existing datasets for other NLP tasks including: Semantic Proto-Roles (SPR) (Reisinger et al., 2015), Definite Pronoun Resolution (DPR) (Rahman and Ng, 2012), FrameNet Plus  $(\mathrm{FN}+)$  (Pavlick et al., 2015), and the GLUE benchmark's diagnostic test (Wang et al., 2019). We additionally consider the Quora Question Pairs (QQP) dataset, where the task is to determine whether two given questions are semantically matching (duplicate) or not. As in Gong et al. (2017), we interpret duplicate question pairs as an entailment relation and neutral otherwise. We use the same split ratio mentioned by Wang et al. (2017).
Since the datasets considered have different label spaces, when evaluating on each target dataset, we map the model's labels to the corresponding target dataset's space. See Appendix D for more details.
We strictly refrained from using any out-of-domain data when evaluating on the unbiased split of the same benchmark in Section 4. However, as shown by prior work (Belinkov et al., 2019a), since different NLI target datasets contain different amounts of the bias found in the large-scale NLI dataset, we need to adjust the amount of debiasing according to each target dataset. We consequently introduce a hyperparameter  $\alpha$  for PoE to modulate the strength of the bias-only model in ensembling. We follow prior work (Belinkov et al., 2019a) and perform model selection on the dev set of each target dataset
Data	CE	DFL	Δ	PoE	Δ
SICK	57.05	57.91	+0.9	57.28	+0.2
ADD1	87.34	88.89	+1.5	87.86	+0.5
DPR	49.50	50.68	+1.2	50.14	+0.6
SPR	59.85	61.41	+1.6	62.45	+2.6
FN+	53.16	54.77	+1.6	53.51	+0.4
JOCI	50.06	51.13	+1.1	50.85	+0.8
MPE	69.50	70.2	+0.7	70.1	+0.6
SCITAIL	67.64	69.33	+1.7	71.40	+3.8
GLUE	54.08	54.80	+0.7	54.71	+0.6
QQP	67.78	69.28	+1.5	68.61	+0.8
MNLI	74.40	73.58	-0.8	73.61	-0.8
MNLI-M	73.98	74.0	0.0	73.49	-0.5
Table 6: Accuracy results of models with BERT transferring to new target datasets. All models are trained on SNLI and tested on the target datasets.  $\Delta$  are absolute differences between our methods and the CE loss baseline.
and then report results on the test set. We select hyper-parameters  $\gamma, \alpha$  from  $\{0.4, 0.6, 0.8, 2, 3, 4, 5\}$ .
Results: Table 6 shows the results of the debiased models and baseline with BERT. As shown in prior work (Belinkov et al., 2019a), the MNLI datasets have very similar biases to SNLI, which the models are trained on, so we do not expect any improvement in the relative performance of our models and the baseline for MNLI and MNLI-M. On all the remaining datasets, our proposed models perform better than the baseline, showing a substantial improvement in generalization by using our debasing techniques. We additionally compare with Belinkov et al. (2019a) in Appendix D and show that our methods substantially surpass their results.
Figure 2: Accuracy of InferSent model trained with DFL, on the SNLI test and SNLI hard sets for different  $\gamma$ .
Analysis of Debiased Focal Loss: As expected, improving the out-of-domain performance could come at the expense of decreased in-domain performance since the removed biases are useful for performing the in-domain task. This happens especially for DFL, in which there is a trade-off between in-domain and out-of-domain performance that depends on the parameter  $\gamma$ , and when the baseline model is not very powerful like InferSent. To understand the impact of  $\gamma$  in DFL, we train an InferSent model using DFL for different values of  $\gamma$  on the SNLI dataset and evaluate its performance on SNLI test and SNLI hard sets. As illustrated in Figure 2, increasing  $\gamma$  increases debiasing and thus hurts in-domain accuracy on SNLI, but out-of-domain accuracy on the SNLI hard set is increased within a wide range of values (see a similar plot for BERT in Appendix E).
Correlation Analysis: In contrast to Belinkov et al. (2019a), who encourage only the encoder to not capture the unwanted biases, our learning strategies influence the parameters of the full model to reduce the reliance on unwanted patterns more effectively. To test this assumption, in Figure 3, we report the correlation between the element-wise loss of the debiased models and the loss of a bias-only model on the considered datasets.
Figure 3: Pearson correlation between the element-wise cross-entropy loss of the debiasing models and the bias-only model trained on each dataset.
The results show that compared to the baselines, our debiasing methods, DFL and PoE, reduce the correlation to the bias-only model, confirming that our models are effective at reducing biases. Interestingly, on MNLI, PoE has less correlation with the bias-only model than DFL and also has better performance on the unbiased split of this dataset. On the other hand, on the HANS dataset, DFL loss is less correlated with the bias-only model than PoE and also obtains higher
<sup>4</sup>Since the test sets are not available for MNLI, we tune on the matched dev set and evaluate on the mismatched dev set or vice versa. For GLUE, we tune on MNLI mismatched dev set.
performance on the HANS dataset.
We propose two novel techniques, product-of-experts and debiased focal loss, to reduce biases learned by neural models, which are applicable whenever one can specify the biases in the form of one or more bias-only models. The bias-only models are designed to leverage biases and shortcuts in the datasets. Our debiasing strategies then work by adjusting the cross-entropy loss based on the performance of these bias-only models, to focus learning on the hard examples and downweight the importance of the biased examples. Additionally, we extend our methods to combat multiple bias patterns simultaneously. Our proposed debiasing techniques are model agnostic, simple, and highly effective. Extensive experiments show that our methods substantially improve the model robustness to domain-shift, including 9.8 points gain on FEVER symmetric test set, 7.4 on HANS dataset, and 4.8 points on SNLI hard set. Furthermore, we show that our debiasing techniques result in better generalization to other NLI datasets. Future work may include developing debiasing strategies that do not require prior knowledge of bias patterns and can automatically identify them.
We would like to thank Daniel Andor and Suraj Srinivas for their helpful comments. We additionally would like to thank the authors of Schuster et al. (2019); Cadene et al. (2019); McCoy et al. (2019b); Belinkov et al. (2019a) for their support to reproduce their results. This research was supported by the Swiss National Science Foundation under the project Learning Representations of Abstraction for Opinion Summarization (LAOS), grant number "FNS-30216". Y.B. was supported by the Harvard Mind, Brain, and Behavior Initiative.
Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. 2019a. Don't take the premise for granted: Mitigating artifacts in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 877-891, Florence, Italy. Association for Computational Linguistics.
Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander M Rush. 2019b. On adversarial removal of hypothesis-only bias in natural language inference. In Proceedings of the 8th Joint Conference on Lexical and Computational Semantics.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
Remi Cadene, Corentin Dancette, Hedi Ben-younes, Matthieu Cord, and Devi Parikh. 2019. Rubi: Reducing unimodal biases in visual question answering. In Advances in neural information processing systems.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Yichen Gong, Heng Luo, and Jian Zhang. 2017. Natural language inference over interaction space. In International Conference on Learning Representations.
Gabriel Grand and Yonatan Belinkov. 2019. Adversarial regularization for visual question answering: Strengths, shortcomings, and side effects. In Proceedings of the Second Workshop on Shortcomings in Vision and Language, pages 1-13.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132-142, Hong Kong, China. Association for Computational Linguistics.
Geoffrey E Hinton. 2002. Training products of experts by minimizing contrastive divergence. Neural computation.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics.
Divyansh Kaushik and Zachary C Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Thirty-Second AAAI Conference on Artificial Intelligence.
Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017. Natural language inference from multiple premises. In Proceedings of the Eighth International Joint Conference on Natural Language Processing.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dólar. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A sick cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), Reykjavik, Iceland. European Language Resources Association (ELRA).
R Thomas McCoy, Junghyun Min, and Tal Linzen. 2019a. Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance. arXiv preprint arXiv:1911.02969.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019b. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
Ellie Pavlick and Chris Callison-Burch. 2016. "Most" babies" are" little" and most" problems" are" huge": Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
Ellie Pavlick, Travis Wolfe, Pushpendre Rastogi, Chris Callison-Burch, Mark Dredze, and Benjamin Van Durme. 2015. Framenet+: Fast paraphrastic tripling of framenet. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 408-413.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: the winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning.
Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. 2018. Overcoming language priors in visual question answering with adversarial regularization. In Advances in Neural Information Processing Systems, pages 1541-1551.
Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. Semantic proto-roles. Transactions of the Association for Computational Linguistics.
Tal Schuster, Darsh J Shah, Yun Jie Serene Yeo, Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. In Proceedings of the 26th International Joint Conference on Artificial Intelligence.
Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a unified evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language Processing.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics.
A Fact Verification
Base model: We fine-tune all models using BERT for 3 epochs and use the default parameters and default learning rate of  $2\mathrm{e} - 5$
Bias-only model: Our bias-only classifier is a shallow nonlinear classifier with 768, 384, 192 hidden units with Tanh nonlinearity.
B Natural Language Inference
Base model: InferSent uses a separate BiLSTM encoder to learn sentence representations for premise and hypothesis. It then combines these embeddings following Mou et al. (2016) and feeds them to the default nonlinear classifier. With InferSent we train all models for 20 epochs as default without using early-stopping. We use the default hyper-parameters and following Wang et al. (2019), we set the BiLSTM dimension to 512. We use the default nonlinear classifier with 512 and 512 hidden neurons with Tanh nonlinearity. With BERT, we finetune all models for 3 epochs.
Bias-only model: For debiasing models using BERT, we use the same shallow nonlinear classifier explained in Appendix A, and for the ones using InferSent, we use a shallow linear classifier with 512 and 512 hidden units.
Results: Table 7 shows results on the MNLI matched development and hard test sets.