SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 1: Summary of datasets and their corresponding evaluation purpose and metrics. N/A indicates the trained model is not evaluated for the corresponding generalization. For full data statistics, see Appendix B.
Method	v1.1 F1	v2.0 F1	MNLI Acc	SST-2 Acc	Avg Δ
BERTbase	88.5	76.5	84.5	92.3	0.0
JR	88.5	74.2	84.5	92.0	-0.7
RPTKL	89.3	77.6	84.8	93.0	+0.7
RPTSHL	89.7	77.6	85.2	93.0	+0.9
RPTJSD	89.8	77.5	85.4	93.5	+1.1
VATKL	89.9	78.9	85.3	93.1	+1.4
VATSHL	90.2	79.2	85.5	93.2	+1.6
VATJSD	90.2	79.3	85.7	93.6	+1.8
BERTlarge	90.9	81.8	86.3	93.5	+2.7
Table 2: Comparison of different training techniques for in-domain generalization on SQuAD (v1.1 and v2.0), MNLI and SST-2 dev sets.  $\mathrm{BERT}_{\text{base}}$  and  $\mathrm{BERT}_{\text{large}}$  stand for BERT-base and BERT-large model with standard training respectively. The corresponding best performing BERT-base model is in bold. The  $\mathrm{Avg} \Delta$  column is the corresponding average score difference with  $\mathrm{BERT}_{\text{base}}$ .
Method	BioASQ F1/EM	IMDB Acc	Avg Δ
BERTbase	57.1/41.7	87.7	0.0
JR	60.8/46.0	87.4	+1.7
RPTKL	59.2/43.6	88.7	+1.6
RPTSHL	60.0/44.8	88.7	+1.9
RPTJSD	58.3/43.2	88.3	+0.9
VATKL	60.1/45.7	86.7	+1.0
VATSHL	60.7/45.9	87.4	+1.7
VATJSD	61.8/47.0	88.3	+2.6
BERTlarge	63.5/49.5	88.3	+3.5
Table 3: Domain shift evaluation of different training techniques on BioASQ and IMDB. BERT<sub>base</sub> and BERT<sub>large</sub> stand for BERT-base and BERT-large model with standard training respectively. The corresponding best performing BERT-base model is in bold. The AvgΔ column is the corresponding average score difference with BERT<sub>base</sub>. For BioASQ, F1 score is used for computing the average score. EM stands for the exact match score.
for QA tasks and accuracy for classification tasks.
In-domain: In this part, we focus on comparing the in-domain performance of different training methods. In other words, each model is trained on the training set and evaluated on the corresponding matched development set. The experiment is summarized in Table 2. In general, JR performs similarly to the standard BERT training with an exception case for SQuAD v2.0. This is probably because JR uniformly regularizes the Jacobian matrix, which is particularly problematic for QA task with unanswerable questions. Both RPT and VAT with different  $f$ -divergences achieve significant improvement over standard training for all four datasets, especially on SQuAD v2.0. The results suggest incorporating the model confidence into regularization can achieve better in-domain
generalization. Consistent with findings in (Miyato et al., 2018), by searching for the most adversarial perturbation direction, VAT variants achieve the largest boost for in-domain generalization. Moreover, we find that both RPT and VAT with SHL and JSD provides additional improvement over their corresponding counterpart with KL which suggests the benefit of using alternative  $f$ -divergences for posterior difference regularization. Lastly, by selecting the proper divergence, the performance gap between the BERT-base and BERT-large model is dramatically narrowed which indicates the advantage of applying posterior difference regularization with  $f$ -divergences on top of powerful text representations.
Domain Shift: In this part, we compare the performance of models trained using different techniques
Method	AddSent F1/EM	AddOneSent F1/EM	HANS Acc	Avg Δ
BERTbase	54.0/48.9	64.8/59.0	58.4	0.0
JR	55.5/49.3	65.3/58.4	62.6	+2.1
RPTKL	55.2/49.2	66.4/59.8	59.9	+1.4
RPTSHL	57.8/52.2	68.9/62.7	60.0	+3.1
RPTJSD	56.0/49.9	67.1/61.1	61.5	+2.5
VATKL	60.6/55.0	69.4/62.9	67.7	+6.8
VATSHL	60.1/55.1	69.1/63.1	63.2	+5.1
VATJSD	58.5/53.2	67.4/61.4	64.6	+4.4
BERTlarge	59.3/54.3	69.1/63.6	67.9	+6.4
Table 4: Adversarial evaluation of different training techniques on Adversarial SQuAD (AddSent and AddOneSent) and HANS. BERT<sub>base</sub> and BERT<sub>large</sub> stand for BERT-base and BERT-large model with standard training respectively. The corresponding best performing BERT-base model is in bold. The AvgΔ column is the corresponding average score difference with BERT<sub>base</sub>. For Adversarial SQuAD, F1 score is used for computing the average score. EM stands for the exact match score.
on datasets with domain shift, e.g. different topic or style. Specifically, we apply the QA models trained on SQuAD v2.0 to the BioAQS version from MRQA (Fisch et al., 2019). Similarly, we apply the sentiment analysis model trained on SST-2 to the IMDB test set. The results are summarized in Table 3. Comparing Table 3 with Table 2, all methods suffer a noticeable performance drop for both QA and sentiment analysis when evaluated on test sets with domain shift. Moreover, we observe more significant performance drop for the QA setting because the biomedical domain differs significantly from the Wiki domain in topic and style, resulting in a larger domain shift between the training and test QA datasets. Consistent with findings in (Hendrycks et al., 2020), the in-domain performance is not predictive of the domain shift generalization. Further, the performance of JR is not stable, with improvement on BioASQ but worse performance on IMDB. Models trained with all three RPT variants result in consistent improvement over standard training on both out-of-domain datasets, suggesting that random perturbation is particularly effective in enhancing model robustness towards domain shift. In particular, all RPT variants achieve comparable out-of-domain gen
eralization on IMDB as BERT-Large. Although all VAT variants achieve decent improvement on BioASQ, neither  $\mathrm{VAT}_{\mathrm{KL}}$  nor  $\mathrm{VAT}_{\mathrm{SHL}}$  generalize so well to IMDB. This illustrates the importance of selecting a proper divergence for VAT style regularization. In other words, domain-dependent search for the most adversarial direction with either KL or SHL might be suboptimal for model generalization over domain shift.
Adversarial Attack: Here, we evaluate different training techniques on adversarial attack scenarios, where datasets are intentionally constructed to fool naively trained models. Specifically, we evaluate the QA models trained with SQuAD v1.1 and the textual entailment models learned on MNLI using the Adversarial SQuAD and the HANS datasets, respectively. Table 4 summarizes the evaluation results of model robustness towards adversarial attacks with different training methods. For both subsets (AddSent and AddOneSent) from Adversarial SQuAD and HANS, all regularization methods improve over standard BERT training. In this case, models trained with VAT variants demonstrate stronger resilience towards learning superficial cues from data. Specifically, VAT with KL achieves the largest improvement on both settings which indicates that an asymmetrical divergence might be more effective in avoiding learning data biases. Although better text representations derived from BERT-Large are still more robust against adversarial attack than the base version, this gap can be effectively reduced by regularizing the posterior difference with  $f$ -divergences. Compared with the recent debiasing method proposed in (Clark et al., 2019) that requires the knowledge of existing data bias, VAT variants can be an effective task-agnostic debiasing approach with better in-domain performance and comparable improvement for adversarial settings.
Semi-supervised Learning: One advantage of regularization methods is their compatibility with semi-supervised learning. Given JR is not very effective for the fully-supervised learning, we focus on evaluating RPT and VAT with  $f$ -divergences under the semi-supervised setting. Specifically, we use the two sentence classification datasets, MNLI and SST-2, for training. We hold out 50% of the label information for the training data. For standard BERT training, only the labelled part is used. For both RPT and VAT variants, the rest unlabelled data is also included for training. Both the cross en
SST-2	IMDB	MNLI	HANS
BERTfull	92.3	87.7	84.5	58.4
BERTbase	91.2	86.3	82.7	51.5
RPTKL	91.3	87.0	83.6	53.8
RPTSHL	91.9	86.6	83.8	53.3
RPTJSD	91.7	86.5	83.7	51.8
VATKL	92.1	86.3	83.1	54.3
VATSHL	92.4	86.5	84.4	51.8
VATJSD	92.2	86.6	84.1	52.6
Table 5: Comparison of different methods with semi-supervised learning in classification accuracy. Except  $\mathrm{{BERT}}_{\text{full }}$  ,all others are trained with  ${50}\%$  labelled data.
tropy loss and the regularization term are optimized for the labelled samples, whereas only the regularization term is used for unlabelled ones. Similar to the fully supervised setting, the models trained on MNLI is applied to HANS for evaluating the model robustness towards adversarial attack, and the models using SST-2 are applied to IMDB to assess the model performance under domain shift. Results are summarized in Table 5.
Compared with the fully supervised setting, all methods get lower classification accuracy across the board. Both RPT and VAT variants again improve over standard training for both in-domain and out-of-domain evaluations. It is worth mentioning that both SHL and JSD based VAT models trained with  $50\%$  labelled data on SST-2 and MNLI are on par with the corresponding standard BERT training with the full training set which illustrates the advantage of choosing a proper  $f$ -divergence for the semi-supervised setting. With only half labelled data, Both RPT and VAT suffer a large drop on HANS and produce almost random predictions, indicating the complimentary benefits of data diversity. We also further reduce the amount of labelled training data and observe the same trend where regularizing with different  $f$ -divergences can lead to improved data efficiency. This demonstrates the potential of posterior differential regularization for NLP with low-resource scenarios.
With the goal of developing more robust NLP models, a line of recent work has been devoted to identifying various kinds of superficial patterns learned by high-performance models over many
popular datasets (Gururangan et al., 2018; Jia and Liang, 2017; McCoy et al., 2019). The prevalence of data biases over popular datasets poses a real challenge of accurately estimating the model capacity for practical applications, because a closed dataset evaluation usually inflates the model performance. This concern about dataset biases has led researchers to develop new diagnostic datasets (Jia and Liang, 2017; McCoy et al., 2019; Nie et al., 2019) and training techniques (Clark et al., 2019; He et al., 2019) to overcome those discovered biases. Recent debiasing methods (Clark et al., 2019; He et al., 2019) require learning multiple models to access known data biases for the target dataset. Moreover, they achieve more robust out-of-domain generalization at the price of in-domain performance degradation. In contrast, we focus on the task-agnostic robust learning framework for enhancing model robustness and empirically show that regularization approaches under this framework can result in superior in-domain and out-of-domain generalization.
Training with noise has been a very popular approach for enhancing model robustness. The dropout is a widely-used approach in deep learning to improve model generalization (Srivastava et al., 2014). For adversarial learning methods, the main theme is reducing the model sensitivity toward small input perturbations (Goodfellow et al., 2014; Madry et al., 2018), which has been recently applied to both fine-turning (Jiang et al., 2020; Pereira et al., 2020; Zhu et al., 2020; Li and Qiu, 2020) and pre-training (Liu et al., 2020). However, models trained with adversarial learning are found to have at-odd generalization (Tsipras et al., 2019; Zhang et al., 2019). Our work studies learning methods with the goal of regularizing the model posterior difference of clean and noisy inputs. We show that compared with the standard BERT training, the proposed posterior differential regularization with  $f$ -divergence lead to better NLP model robustness.
In this paper, we investigate methods regularizing the posterior difference between the clean and noisy inputs for improving model generalization for both in-domain and out-of-domain settings. Specifically, we present theoretical analyses of three methods under this framework, i.e. VAT, JR, and RPT. We further extend both VAT and PRT to the family of  $f$ -divergences and theoretically characterize them in terms of Jacobian matrix. We also demonstrate
their effectiveness in enhancing model robustness over a diverse set of NLP tasks under both fully-supervised and semi-supervised scenarios.
For future work, it is interesting to explore posterior differential regularization methods for weakly-supervised learning, such as relation extraction and QA with distant supervision.
We would like to thank the anonymous reviewers for valuable suggestions. Yaoliang Yu thanks NSERC for funding support.
Mudassar Abbas, J. Kivinen, and T. Raiko. 2016. Understanding regularization by virtual adversarial training, ladder networks and others. In ICLR 2016-Workshop Track.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440-447, Prague, Czech Republic. Association for Computational Linguistics.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust neural machine translation with doubly adversarial inputs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4324-4333, Florence, Italy. Association for Computational Linguistics.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4069-4082, Hong Kong, China. Association for Computational Linguistics.
I. Csiszár and P. Shields. 2004. Information theory and statistics: A tutorial. Foundations and Trends in Communications and Information Theory, 1(4):417-528.
Hal Daume III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256-263, Prague, Czech Republic. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1-13, Hong Kong, China. Association for Computational Linguistics.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias for natural language inference by fitting the residual. In EMNLP Workshop on DeepLo.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2744-2751, Online. Association for Computational Linguistics.
Yu-Lun Hsieh, Minhao Cheng, Da-Cheng Juan, Wei Wei, Wen-Lian Hsu, and Cho-Jui Hsieh. 2019. On the robustness of self-attentive models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1520-1529.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and efficient fine-tuning for pretrained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2177-2190, Online. Association for Computational Linguistics.
Linyang Li and Xipeng Qiu. 2020. Textat: Adversarial training for natural language understanding with token-level perturbation. arXiv preprint arXiv:2004.14543.
Yitong Li, Trevor Cohn, and Timothy Baldwin. 2016. Learning robust representations of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1979-1985, Austin, Texas. Association for Computational Linguistics.
Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. 2020. Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-1993.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599.
Lis Pereira, Xiaodong Liu, Fei Cheng, Masayuki Asahara, and Ichiro Kobayashi. 2020. Adversarial training for commonsense inference. arXiv preprint arXiv:2005.08156.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
J. Sokolic, R. Giryes, G. Sapiro, and M. R. D. Rodrigues. 2017. Robust large margin deep neural networks. IEEE Transactions on Signal Processing, 65(16):4265-4280.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1958.
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artières, Axel-Cyrille Ngonga Ngomo, Norman Heino, Eric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos, and Georgios Paliouras. 2015. An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics, 16(1):138.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019. Robustness may be at odds with accuracy. In International Conference on Learning Representations.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Dilin Wang, Chengyue Gong, and Qiang Liu. 2019. Improving neural language modeling via adversarial training. In International Conference on Machine Learning, pages 6555-6565.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. 2019. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7472-7482. PMLR.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for natural language understanding. In International Conference on Learning Representations.
A Second-order Approximation of Posterior Differential Regularization with  $f$ -divergence
Based on Equation 6, it is easy to show that the first and second derivatives of  $D$  with regard to  $\epsilon$  are
$$
D ^ {\prime} = \sum_ {i} g ^ {\prime} (\mathbf {r} _ {i}) f _ {i} ^ {\prime} (\hat {\mathbf {x}}),
$$
$$
D ^ {\prime \prime} = \sum_ {i} \left[ g ^ {\prime \prime} (\mathbf {r} _ {i}) \frac {f _ {i} ^ {\prime} (\hat {\mathbf {x}}) f _ {i} ^ {\prime T} (\hat {\mathbf {x}})}{f _ {i} (\mathbf {x})} + g ^ {\prime} (\mathbf {r} _ {i}) f _ {i} ^ {\prime \prime} (\hat {\mathbf {x}}) \right],
$$
where  $\mathbf{r}_i = \frac{f_i(\hat{\mathbf{x}})}{f_i(\mathbf{x})}$ . Let  $D(\epsilon) = D(f(\hat{\mathbf{x}}), f(\mathbf{x}))$ . When evaluating at 0, the second order approximation of the  $f$ -divergence is
$$
D (\epsilon) \approx D (0) + D ^ {\prime T} (0) \epsilon + \frac {\epsilon^ {T} D ^ {\prime \prime} (0) \epsilon}{2}. \tag {10}
$$
$$
D (0) = \sum_ {i} g (1) f _ {i} = 0,
$$
$$
D ^ {\prime} (0) = \sum_ {i} g ^ {\prime} (1) f _ {i} ^ {\prime} = g ^ {\prime} (1) \left(\sum_ {i} f _ {i}\right) ^ {\prime} = 0,
$$
$$
\sum_ {i} f _ {i} ^ {\prime \prime} = \left(\sum_ {i} f _ {i}\right) ^ {\prime \prime} = 0
$$
the second order approximation can be simplified as