SlowGuess/ACL_2020_OCR huggingface dataset:
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In COLING, pages 2340-2353.
Yixin Nie, Yicheng Wang, and Mohit Bansal. 2019. Analyzing compositionality-sensitivity of nli models. In AAAI, volume 33, pages 6867-6874.
Tong Niu and Mohit Bansal. 2018. Adversarial oversensitivity and over-stability strategies for dialogue models. In CoNLL, pages 486-496.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In *SEM*, pages 180-191.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929-1958.
Aarne Talman and Stergios Chatzikyriakidis. 2019. Testing the generalization power of neural network models across nli benchmarks. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 85-94.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998-6008.
Haohan Wang, Zexue He, Zachary L. Lipton, and Eric P. Xing. 2019a. Learning robust representations by projecting superficial statistics out. In ICLR.
Haohan Wang, Da Sun, and Eric P Xing. 2019b. What if we simply swap the two text fragments? a straightforward yet effective way to test the robustness of methods to confounding signals in nature language inference tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7136-7143.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019c. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE International Conference on Computer Vision, pages 5310-5319.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In *NAACL-HLT*, volume 1, pages 1112–1122.
Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. 2017. Controllable invariance through adversarial feature learning. In *NeurIPS*, pages 585-596.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In EMNLP.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018a. Gender bias in coreference resolution: Evaluation and debiasing methods. In NAACL-HLT, volume 2.
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018b. Learning gender-neutral word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4847-4853.
For all our models except BERT (Devlin et al., 2019), we use pre-trained 300-dimension GloVe (Pennington et al., 2014) word embeddings to initialize the embedding layers. The hidden dimension of LSTM (Hochreiter and Schmidhuber, 1997) is 300. We use Adam (Kingma and Ba, 2015) as the optimizer and the initial learning rate is set to 0.0004. We apply dropout (Srivastava et al., 2014) with a rate of 0.4 to regularize our model. For the model with HEX projection, we apply all the tricks in the original paper (Wang et al., 2019a) (columnwise normalize the input features in every batch, fine-tune from a trained model with the bottom layer fixed) to stabilize the training. In our experiments, we set the multi-task coefficient between loss for  $F_{L}$  and  $F_{G}$  to 1.0 and 0.3.
B Detailed Description of the Extraction of Balanced Testing Sets
B.1 Extraction of the Contradiction-Word-Bias Testing Set
For evaluating the contradiction-word-bias (CWB), we look for words that both have a strong bias towards the 'contradiction' label and have a significant number of samples in the training set. We first select 'no', 'any', 'never' and 'anything', which are four most frequent words with over  $50\%$  of
8769
contradiction-word	appended phrase
no	and false is no true
any	and any true is true
never	and false is never true
anything	and anything true is true
not	and false is not true
Table 5: The phrases to append at the end of the hypothesis sentence for each contradiction word.
samples in the training data containing these words labeled as 'contradiction'. Since most of the analysis papers also study the bias of 'not', here we also include the 'not' as the contradiction word. However, as in the training set of MNLI (Williams et al., 2018), only  $45.3\%$  of the samples are 'contradiction', so the bias of 'not' is actually not as strong as the other words.
Next, in order to create a balanced dataset for these selected contradiction-words, we first select the samples containing these words from the matched development set. In order to let the samples be more difficult and better test the model's bias. We only select the samples where the hypothesis samples contain the contradiction word, while there's no negation word in the premise sentence (so that the contradiction word is generated by the annotator instead of copying from the premise sentence). Since the bias of 'not' is not uniformly strong, here we only select samples that both contain 'not' and have small Jaccard distance (Hamers et al., 1989) between the sentence pairs, which we empirically find that the bias is stronger.
After selecting these samples, we can extract a testing set with most of the samples labeled as contradiction, but the label distribution is severely unbalanced. In order to balance the label distribution, we randomly sample some examples from the training set using the same criterion (containing contradiction word in the hypothesis sentence but no negation word in the premise sentence) and put them in the testing set. Our resulting dataset contains 1100 samples with 550 are labeled as contradiction and the other 550 are non-contradiction labels. Since the domain of the training set is different from the domain of the mismatched validation set, we only extract a balanced test set based on the matched validation set.
B.2 Extraction of the
Word-Overlapping-Bias Testing set
We first sort the samples in the MNLI matched validation set using Jaccard distance (Hamers et al.,
1989) and choose the samples with the smallest distance (highest overlapping). In order to match the size of the contradiction-word-bias testing set, we select the top 550 samples with entailment label and the top 550 samples with non-entailment label to get a dataset with high word overlapping but balanced label distribution.
C Construction of Synthetic Data
We follow the construction rule of the NLI stress tests (Naik et al., 2018) to generate synthetic data for the training set. We appended meaningless sentences at the end of the hypothesis sentence and keep the original label unchanged. For CWB, we focus on 5 different contradiction words: 'no', 'any', 'never', 'anything', and 'not'. Therefore, for each sentence pair, we create five different new pairs by appending five different phrases for evaluating the bias of each contradiction word. The appended phrases are listed in Table 5. For WOB, we also follow (Naik et al., 2018) to append 'and true is true' to every hypothesis sentence to create one new pair for each sample.
D Data Augmentation/Enhancement Results for BERT
The data augmentation/enhancement results for BERT-base (Devlin et al., 2019) is shown in Table 7 and Table 8. As is shown in Table 7, BERT shows significant performance gap between Acc and Acc hr on both CWB datasets, indicating BERT's clear bias on CWB. As for WOB, the gap between Acc and Acc hr for Bal is much smaller, however, the performance on Stress is very poor. Therefore, we assume that even though BERT achieves a high score on the WOB Bal dataset, BERT is just overfitting the dataset in another different way, i.e., there is still significant WOB bias in BERT. In conclusion, in our experiment, BERT still shows significant CWB and WOB.
Similar to our main data augmentation/enhancement results, here we find that after adding 500 additional synthetic samples, BERT can quickly learn their pattern. But still, adding more synthetic data doesn't help improve the performance on the Bal dataset. For BERT, we also cannot see any significant improvement when adding additional original samples. In all the + origin experiments, BERT performs similarly. Again, this shows the limitation of the data
<sup>11</sup>We run all the experiments 5 times and report the mean.
8770
Train/Test	MNLI Acc	Bal	Stress
Acc	Acc(hr)	Acc	Acc(hr)
baseline	69.8	76.3	59.4	58.2	37.6
+ origin	70.1/70.0/69.4	77.1/77.5/76.4	61.5/64.1/64.7	56.0/58.0/55.4	31.0/37.3/29.5
+ synthetic	70.0/69.8/69.6	77.2/75.7/75.7	61.3/58.8/58.6	67.7/68.8/68.7	66.2/72.9/72.0
Table 6: The performance of LSTM baseline model for reducing the WOB via data enhancement/augmentation. The numbers each representing the result after adding 500/20,000/50,000 additional data.
Train/Test	MNLI Acc	Bal	Stress*
Acc	Acc hr	Acc	Acc hr
baseline	82.3	84.2	71.2	55.8	41.9
+ origin	82.3/82.6/82.7	83.8/83.7/83.6	70.7/70.6/70.2	55.7/55.3/55.2	42.4/41.7/43.2
+ synthetic	82.6/82.4/82.4	84.3/84.1/84.3	71.9/71.2/71.5	83.3/84.0/83.9	81.9/83.2/83.0
Table 7: The performance of BERT for reducing the CWB via data enhancement/augmentation. The numbers each representing the result after adding 500/20,000/50,000 additional data.
Train/Test	MNLI Acc	Bal	Stress
Acc	Acc hr	Acc	Acc hr
baseline	82.3	90.5	87.0	58.1	6.49
+ origin	82.7/82.4/82.4	91.3/90.5/90.8	87.9/87.2/87.5	58.1/58.2/58.1	7.43/7.61/5.88
+ synthetic	82.4/82.5/82.5	90.7/90.6/91.1	87.0/86.7/87.5	83.4/84.0/83.9	82.4/83.8/83.8
Table 8: The performance of BERT for reducing the WOB via data enhancement/augmentation. The numbers each representing the result after adding 500/20,000/50,000 additional data.
Figure 4: LIME analysis on the CWB Stress* dataset showing the 6 most important features used by the model.
augmentation/enhancement approach, especially starting with a stronger baseline as BERT.
E More Qualitative Feature Analysis
In Fig. 4, we can see the feature importance change before/after adding the BoW sub-model for a CWB Stress* example (we chose a borderline example where the prediction distribution change to the
correct label is not extreme). We can see that before adding the BoW sub-model orthogonality-projection, the extra misleading words (both "and" and "not") confused the model to predict the wrong contradiction label, while after adding the BoW sub-model, our model can assign higher weights to useful features such as "have", "before", etc.
8771