SlowGuess/ACL_2020_OCR huggingface dataset:
primary model to learn better features that are not learn-able by the BoW model, we used the HEX projection layer from Wang et al. (2019a), which was originally proposed to improve the domain generalization performance of computer vision models; here we combine HEX with BoW sub-model to robustify NLI models. With the addition of the BoW sub-model, we can get two representations of the sentence pair  $\mathbf{u}_{\text {main }}$  and  $\mathbf{u}_{\text {bow }}$ . In order to let the final prediction to use high-level features that are to some extent independent of the shallow and high-biased BoW feature, HEX projection layer projects these two representations into orthogonal spaces to achieve the independence.
The inputs of the HEX projection layers are the BoW model output  $\mathbf{u}_{bow}$  and the corresponding output of the main model  $\mathbf{u}_{main}$ . We use  $f$  to denote the final classification network parameterized by  $\xi$ . Next, by zero-masking one of the two inputs, the HEX projection layer can receive three different inputs and calculate three different vector outputs:
$$
\mathbf {F} _ {A} = f \left(\left[ \mathbf {u} _ {b o w}; \mathbf {u} _ {m a i n} \right], \xi\right)
$$
$$
\mathbf {F} _ {P} = f ([ \mathbf {0}; \mathbf {u} _ {\text {m a i n}} ], \xi) \tag {2}
$$
$$
\mathbf {F} _ {G} = f ([ \mathbf {u} _ {b o w}; \mathbf {0} ], \xi)
$$
To ensure that the overall model learns different features than the BoW model, we project the joint output  $\mathbf{F}_A$  to the orthogonal space of  $\mathbf{F}_G$  to get  $\mathbf{F}_L$ :
$$
\mathbf {F} _ {L} = (\mathbf {I} - \mathbf {F} _ {G} (\mathbf {F} _ {G} ^ {T} \mathbf {F} _ {G}) ^ {- 1} \mathbf {F} _ {G} ^ {T}) \mathbf {F} _ {A} \qquad (3)
$$
The output learns good representations for both sentences but lies in the orthogonal space of the output got from BoW sub-model's input, thus not over-emphasizing on word-pattern information. This vector goes through the softmax layer to calculate
the probabilities for each label. Finally, we follow the original paper (Wang et al., 2019a) to minimize a weighted combination of the loss for  $\mathbf{F}_L$  and  $\mathbf{F}_G$ , and use  $\mathbf{F}_P$  for testing. In Sec. 6.2, we show that by adding the BoW sub-model orthogonality, the model can be more robust against CWB and WOB while maintaining competitive overall accuracy. Hence, as a response to Q3 in Sec. 1, our results indicate that debiasing models at the upper level with regularization on the compositionality is a more promising direction against lexical biases.
We evaluate our models using both off-the-shelf testing datasets as well as new datasets extracted from the original MNLI dataset. We use the word overlap and the negation sets from the NLI stress tests dataset (Naik et al., 2018). These two evaluation sets from the NLI stress tests modified the original MNLI development set by appending some meaningless phrases (examples shown in Table 1). If the model has certain biases, then the model will be fooled by such perturbations and make the wrong classification.
In addition, we also extract samples from the original MNLI development dataset to get bias testing sets with exactly the same data distribution. We first select samples that follow the bias pattern from the matched development set. For CWB, we use 'not', 'no', 'any', 'never', and 'anything' as five example contradiction words. To make this testing set balanced for labels (contradiction vs non-contradiction for CWB and entailment vs non-entailment for WOB), we move some samples with the same pattern from the training set to this testing set. Later we refer to this dataset as Bal.
Since the negation dataset from NLI stress tests dataset only considers the word 'not', it fails to evaluate the bias for other contradiction words. We augment this dataset by creating new samples for other contradiction words. We denote the original NLI stress tests dataset as Stress and this augmented one as Stress*. Please refer to the Appendix for a detailed description of how we chose the example contradiction words and created our test sets.
Throughout our experiments, we select the best
While this makes our model's performance incomparable to other literature, we train all the models in our experiments in this same setting to ensure the fairness of our analysis comparisons. All our experiments use the same val/test set.
8764
Train/Test	MNLI Acc	Bal	Stress*
Acc	Acc_hr	Acc	Acc_hr
baseline	69.8	70.5	45.7	50.9	38.7
+ origin	69.7/69.2/69.1	71.2/71.1/70.6	46.3/49.0/47.9	49.7/49.2/50.7	40.2/40.2/42.1
+ synthetic	69.8/70.0/69.7	71.0/70.7/71.2	45.7/45.9/47.1	67.2/68.5/68.4	65.8/68.3/68.4
Table 2: The performance for reducing the CWB via data enhancement/augmentation. The numbers each representing the result after adding 500/20,000/50,000 additional data.
model during training on the MNLI mismatched development dataset and we tune all the hyperparameters on the NLI Stress mismatch datasets. All the other datasets are only used as test sets and we only report results on these test sets. We use the MNLI matched development dataset to evaluate the overall performance of the model.
Overall accuracy is widely used as the only metric for NLI. However, models can get very high accuracy by exploiting the bias patterns. Hence, in order to test how the model performs when it cannot exploit the bias pattern, we focus on model's accuracy on the harder parts of the data (Acc hr) where the bias pattern is wrong<sup>6</sup>. For the balanced testing set, this subset means samples with 'non-contradiction' label for CWB case and samples with 'non-entailment' label for the WOB case. For the NLI stress tests dataset<sup>7</sup>, this subset means the samples with 'non-contradiction' label for the CWB set and the samples with 'entailment' label for the WOB set. Ideally, for an unbiased model, it should both have competitive overall performance and perform almost equally well on these harder parts of the data. Hence, we focus on maintaining the accuracy on the whole dataset and improving the Acc hr metric. All training details and hyperparameter settings are presented in Appendix.
6.1 Data-Level Debiasing Results
We first show our baseline's performance on the CWB biases in the first row of Table 2. Since we ob
serve similar performance for CWB and WOB, we leave the results for WOB in Appendix. On every dataset, there's a significant gap between Acc and Acc hr, showing the baseline has both strong CWB bias and strong WOB bias. For the data augmentation/enhancement experiments, we report results after adding 500/20,000/50,000 additional samples. We demonstrate the effect of adding a small portion of data for the 500 case and the limitation of this method using the 20,000 and 50,000 cases. The results are again shown in Table 2. We use “+origin” to denote the results from data enhancement using the original dataset and use “+synthetic” to denote the results from data augmentation by generating new synthetic data similar to NLI stress tests.
With a small number of additional data (500), wherever the data comes from, the performance on the balanced testing set remains very close. However, the performance on the NLI stress tests improves significantly when it sees 500 synthetic new samples generated in the same way. The gap between the overall accuracy and the Acc_hr on NLI stress tests is reduced to less than  $5\%$ , which means that the models can easily learn how the synthetic data is generated through only 500 samples. Next, we compare the performance after adding 20,000 and 50,000 additional data to check the limitation of the improvement from adding additional data. With this amount of additional original data, the Acc_hr on the balanced dataset improves and the model is less biased. However, adding 20,000/50,000 synthetic samples doesn't always lead to the improvement on the balanced dataset. This reflects that the generation rules of NLI stress tests dataset are too simple so that training on these adversarial samples is not a good way to robustify the model. However, more natural and diverse synthetic data may be helpful to robustify the models.
There is still a significant gap between overall accuracy and Acc hr even after 50,000 sam
One may wonder if biases can also be evaluated simply using generalization performance. However, good generalization to current datasets (e.g., SNLI (Bowman et al., 2015), MNLI (Williams et al., 2018), SICK (Marelli et al., 2014), etc.) is different from being bias-free. As shown in Guru-rangan et al. (2018), similar annotation artifacts can appear in multiple different datasets. So by overfitting to common lexical biases across multiple datasets, biased models might still reach higher generalization accuracy.
Another metric on NLI-stress can be checking the portion of model predictions on the hard data that is correct both before and after adding the extra words. We empirically verified that this metric shows the same result trends as Acc-hard.
$^{8}$ Adding additional data (e.g., 50,000) can change the label distribution, but we have experimented with different numbers of additional data between 500 and 50,000 and the reported trend always holds.
<sup>9</sup>We run all the experiments 5 times and report the mean.
Model	MNLI Acc	CWB	WOB
Bal	Stress*	Bal	Stress
Acc	Acc hr	Acc	Acc hr	Acc	Acc hr	Acc	Acc hr
baseline	70.0	70.6	45.3	49.9	37.0	75.4	58.5	59.8	40.2
embbasic	67.8	70.3	49.5	50.2	41.1	73.9	56.2	56.9	35.6
emb_cond	67.9	68.5	46.4	48.8	38.3	74.5	54.5	56.7	39.6
sgl_sent	67.2	68.9	47.3	48.8	37.4	73.8	55.5	54.1	29.1
Table 3: The performance for debiasing the embeddings on CWB and WOB.
Model	MNLI Acc	CWB	WOB
Bal	Stress*	Bal	Stress
Acc	Acc hr	Acc	Acc hr	Acc	Acc hr	Acc	Acc hr
baseline	69.8±0.25	70.5±0.75	45.7±2.28	50.9±1.50	38.7±3.94	76.3±0.59	59.4±0.82	58.2±3.04	37.6±9.63
+ BoW	68.4±0.25	72.6±0.84	56.3±1.69	54.9±0.66	48.0±1.44	75.1±0.90	69.3±1.51	60.8±1.05	46.6±4.64
#Layers=2	69.8±0.34	69.9±0.93	44.8±1.70	51.4±0.94	40.0±2.05	76.6±0.85	58.6±0.84	58.7±1.56	40.5±5.49
+ BoW	68.5±0.47	71.2±1.05	54.1±1.65	56.3±1.26	49.9±1.24	74.2±1.41	68.1±0.76	62.2±1.41	49.6±4.34
Table 4: The performance for BoW sub-model orthogonality on CWB and WOB. The means and standard deviation here are averaged over five random runs.
plies. Also, the effect of adding the last 30,000 data is very small, indicating a clear limitation of this method. Thus, doing simple data augmentation/enhancement only using the currently available resources is insufficient to fully debias the model. In addition, one has to carefully select which data to add for each different bias, so we need to also design inherently more robust models.
6.2 Model-Level Debiasing Results
Debiasing Embeddings (Lower Level Model Debiasing). We compared three variants of debiasing embeddings in Table 3. Empirically, we observe that training the whole model with the debias network from a pre-trained baseline can significantly improve the stability of results, so we perform our experiments from one baseline with average performance for fair comparisons. The multi-task coefficient  $\lambda$  controls the trade-off between high accuracy and little bias. Here we report the results with  $\lambda = 1$ , which we find is one good balance point. From both tables, none of the methods achieved a significant improvement on the Acc_lr metrics. The best results come from the embbasic approach, but even this method only achieves small improvement on the Acc_lr metric for CWB but does worse on WOB and has a comparable loss on overall Acc. We do not observe any significantly larger improvements with smaller or larger  $\lambda$ . We also tried other techniques to further stabilize the training (e.g., freezing the main model when training, using different optimization algorithms), but we observe no significant improvement.
Therefore, while removing the bias from the embeddings is effective for reducing gender bias (e.g.,
remove the male bias from the word 'doctor' to make the embedding gender-neutral), it does not help in debiasing certain lexical biases. Directly removing information from the embedding only slightly debiases the model but also hurts the overall performance. The difference in these results highlights the difference between gender bias and lexical bias problems. As shown in these experiments, lexical biases cannot be effectively reduced at the embedding level. We argue that this is because a majority of lexical biases appear at the compositionality level. For example, for WOB, a biased model will predict "entailment" entirely relying on the overlapping word embeddings on both sides. Here, even when we make the embeddings completely unbiased, as long as the upper model learns to directly compare the overlapping of embeddings on both sides, there will still exist a strong WOB bias in the model. Hence, in order to robustify models towards lexical bias, we need to develop methods that regularize the upper-interaction part of the model.
BoW Sub-Model Orthogonality (Higher Level Model Debiasing). Results for adding the BoW sub-model are shown in Table 4. Here, we also show that the improvement trend holds regardless of minor hyper-parameter changes in the model (number of layers). On both CWB and WOB, the model shows a large improvement on Acc hr for both Bal and stress-test datasets. We achieve close or higher Acc on all the bias testing sets and the overall Acc is only  $1.4\% / 1.3\%$  lower than the baseline, showing that adding a BoW sub-model orthogonality will only slightly hurt the model. In
8766
conclusion, this approach significantly robustifies the model against CWB and WOB while maintaining competitive overall performance. In comparison to the debiasing embeddings results, we can see that instead of regularizing the content in the word embeddings, regularizing the model's compositionality at the upper interaction level is a more promising direction for debiasing lexical biases. We have also tried combining this method with the data-level debiasing approach above but get no further improvement.[10]
6.3 Qualitative Feature Analysis
We use LIME (Ribeiro et al., 2016) to qualitatively visualize how orthogonal projection w.r.t. BoW sub-model changes the features used by the model. We selected one example from the CWB Bal dataset to see how applying the BoW model with HEX corrects previous mistakes. From Fig. 3, we can see that before applying the BoW sub-model (the upper part of the figure), the model predicts the contradiction label almost solely based on the existence of the word "no" in the hypothesis. However, after applying our BoW sub-model with HEX projection, our model can give higher importance to other useful features (e.g., the match of the two "bad" tokens, and the match of important past-tense temporal words such as "passed" and "longer" in the premise-hypothesis pair) despite the fact that "no" still has high influence towards the contradiction label. Another example from the CWB Stress* dataset can be seen in Appendix.
We study the problem of lexical dataset biases using WOB and CWB as two examples. We first showed that lexical dataset biases cannot be solved by simple dataset changes and motivate the importance of directly designing model-level changes to solve this problem. For model-level changes, we first show the ineffectiveness of embedding-debiasing approaches, thus highlighting the uniqueness of lexical bias against gender bias problems. Next,
Figure 3: LIME analysis on the CWB Bal dataset showing the 6 most important features used by the model.
we robustify the model by forcing orthogonality between a BoW sub-model and the main model and demonstrate its effectiveness through several experiments. Since none of our methods is biastype specific, we believe these results can also be generalized to other similar lexical biases. Finally, we would like to point out that our methods and results here do not mean to belittle the importance of collecting clean/unbiased data. We strongly believe in the importance of unbiased data for model design and evaluation. However, some biases are inherent and inevitable in the natural distribution of the task (e.g., for NLI, it is natural that sentences with high overlapping are most likely entailment pairs). Therefore, our work stresses that it is also very important to encourage the development of models that are unlikely to exploit these inevitable biases/shortcuts in the dataset. Neither model-level debiasing nor data-level debiasing alone is the conclusive solution for this problem. Joint efforts are needed for promoting unbiased models that learn true semantics; and we hope our paper can encourage more work towards this important direction.
We thank Snigdha Chaturvedi, Shashank Srivastava, and the reviewers for their helpful comments. This work was supported by DARPA YFA17-D17AP00022, NSF-CAREER Award 1846185, ONR Grant N00014-18-1-2871. The views in this article are the authors', not of the funding agency.
<sup>10</sup>We also tried some initial simple ensembles of 2 different initializations of BoW sub-models, so that we can potentially regularize against a more diverse set of lexicon biases. When training, the main model is paired with each BoW sub-models to go through each HEX layers and then the output logits are averaged to get the final logits. This ensembling results also outperform the baseline significantly and is higher than the single BoW Sub-Model in WOB Stress, but equal or worse in the other cases. We leave the exploration of different/better ways of ensembling to future work.
Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. 2019a. On adversarial removal of hypothesis-only bias in natural language inference. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 256-262, Minneapolis, Minnesota. Association for Computational Linguistics.
Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander M Rush. 2019b. Dont take the premise for granted: Mitigating artifacts in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 877-891.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In NeurIPS, pages 4349-4357.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP, pages 632-642.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In ACL, volume 1, pages 1657-1668.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Dont take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4060-4073.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP, pages 670-680.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In EMNLP, pages 11-21.
Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In ICML, pages 1180-1189.
Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In ACL, volume 2, pages 650-655.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In *NAACL-HLT*, volume 2, pages 107-112.
Lieve Hamers, Yves Hemeryck, Guido Herweyers, Marc Janssen, Hans Keters, Ronald Rousseau, and Andr Vanhoutte. 1989. Similarity measures in scientometric research: The jaccard index versus salton's cosine formula. Information Processing & Management, 25(3):315-318.
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132-142.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In EMNLP, pages 2021-2031.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Yitong Li, Timothy Baldwin, and Trevor Cohn. 2018. Towards robust and privacy-preserving text representations. In ACL, volume 2, pages 25-30.
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In SemEval, pages 1-8.
R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007.
Pasquale Minervini and Sebastian Riedel. 2018. Adversarily regularising neural nli models to integrate logical background knowledge. CoNLL 2018, page 65.
Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. In ACL, volume 2, pages 130-136.