SlowGuess/NAACL_2021_OCR huggingface dataset:
Our model is implemented using Pytorch based on Transformers framework. Specifically, we use the learning setup and hyperparameters recommended by (Devlin et al., 2019). We use Huggingface edition Adam (Kingma and Ba, 2015) optimizer (without bias correction) with learning rate of  $2 \times 10^{-5}, \beta_{1} = 0.9, \beta_{2} = 0.999$ , and warmup over the first  $10\%$  steps of the total steps. We fine-tune the entire model (340 million parameters), of which the vast majority start as pre-trained weights (BERT-Large-Uncased) and the classification layer (2048 parameters). Weights of the classification layer are initialized with  $\mathcal{N}(0, 0.02^2)$ . We train with a batch size of 32 for 3 epochs. More details of our experimental setup are described in Appendix A.
Table 1 shows the results of all the models on selected GLUE datasets. We train each dataset over 25 random seeds. To implement our LNSR, we uniformly inject noise at the first layer on BERTlarge for the comparison with baseline models. As we can see from the table, our model outperforms all the baseline models in mean and max values, which indicates the stronger generalizability of our model against other baseline models. The p-values between the accuracy distributions of standard BERT fine-tuning and our model are calculated to verify whether the improvements are significant. We obtain very small p-values in all tasks: RTE:  $9.7 \times 10^{-7}$ , MRPC:  $2.3 \times 10^{-4}$ , CoLA:  $4.7 \times 10^{-8}$ , STS-2:  $3.3 \times 10^{-8}$ .
Standard deviation is an indicator of the stability of models' performance and higher std means more sensitive to random seeds. Our model shows a lower standard deviation on each task, which means our model is less sensitive to random seeds than other models. Figure 2 presents a clearer illustration. To sum up, our proposed method can effectively improve the performance and stability of fine-tuning BERT.
To verify the effectiveness of our proposed LNSR model, we conduct several ablation experiments including fine-tuning with more training epochs and
Figure 3: The top-1 accuracy (top) and loss (bottom) curve for fine-tuning BERT with and w/o LNSR.
noise perturbation without regularization (we inject noise directly to the output of a specific layer, and then use the perturbed representation to conduct propagation and then calculate loss, this process is similar to a vector-space represent augmentation). The results are shown in Table 2. We observe that benefit obtained by longer training is limited. Similarly, fine-tuning with noise perturbation only achieves slightly better results on two of these tasks, showing that simply adding noise without an explicit restriction on outputs may not be sufficient to obtain good generalizability. While BERT models with LNSR perform better on each task. This verifies our claim that LNSR can promote the stability of BERT fine-tuning and meanwhile improve the generalizability of the BERT model.
5.2 Effects on the Generalizability of Models
We verify the effects of our proposed method on the generalizability of BERT models in two ways - generalization gap and models' performance on fewer training samples. Due to the limited data and the extremely high complexity of BERT model, bad fine-tuning start point makes the adapted model overfit the training data and does not generalize well to unseen data. Generalizability of models can be intuitively reflected by generalization gap and models' performance on fewer training samples.
Table 3 shows the mean training Acc, mean evaluation Acc and generalization gap of different models on each task. As we can see from the table, fine-tuning with LNSR can effectively narrow the
$^{2}$ https://huggingface.co/transformers/index.html
RTE	MRPC	CoLA	STS-B
mean	std	max	mean	std	max	mean	std	max	mean	std	max
FT	70.13	1.84	72.56	87.57	0.92	89.16	61.56	1.34	64.10	89.38	0.53	90.23
FT (4 Epochs)	70.69	1.97	73.65	88.15	0.65	89.21	60.69	1.24	62.09	89.29	0.56	90.12
FT+Noise	70.62	1.56	72.93	87.95	0.83	89.33	60.18	1.58	62.59	89.34	0.51	90.11
LNSR(ours)	73.31	1.55	76.17	88.50	0.56	90.02	63.35	1.05	65.99	90.23	0.31	90.97
Table 2: Ablation study of LNSR on each task, we report the mean evaluation scores and standard deviation and max value across 25 random seeds. FT refers to standard BERT fine-tuning.
RTE	MRPC	CoLA	STS-B
train / eval / gap	train / eval / gap	train / eval / gap	train / eval / gap
FT (Devlin et al., 2019)	95.89/70.13/25.76	96.57/87.57/9.00	97.71/61.56/36.25	98.31/89.38/ 8.93
LNSR(ours)	90.72/73.31/17.41	96.68/88.50/8.18	93.44/63.35/30.09	98.45/90.23/ 8.22
Table 3: Comparison of the generalizability performance of different models. We report the mean training Acc and evaluation Acc and the generalizability gap (training Acc - evaluation Acc) of each model across 20 random seeds.
generalization gap, and achieve higher evaluation score. The effect of narrowing generalization gap is also reflected in Figure 3 where we can see the higher evaluation accuracy and lower evaluation loss.
We sample subsets from the two relatively larger datasets CoLA (8.5k training samples) and STS-B (7k training samples) with the sampling ratio of 0.15, 0.3 and 0.5. As is shown in Figure 4, finetuning with LNSR shows clear advantage on fewer training samples, suggesting LNSR can effectively promote the model's generalizability.
5.3 Sensitivity to the Position of Noise Injection
We briefly discuss about the sensitivity to the position of noise injection as it is a pre-determined hyperparameter of our method. As is shown in Figure 5 in Appendix A, we observe that the performance of LNSR does not fluctuate much as the position of noise injection changes. All injection positions bring significant improvements over vanilla fine-tuning. Note that, with LNSR, noise injection to the lower layers usually leads to relatively higher accuracy and stability, implying that LNSR may be more effective when it affects both the lower and higher layers of the network.
5.4 Relationship to Previous Noise-based Approaches
Our method is related to SMART (Jiang et al., 2020), FreeLB (Zhu et al., 2020a) and R3F (Agha
Figure 4: Mean evaluation score comparison of finetuning BERT with and w/o LNSR on fewer training samples of the CoLA and STS-B tasks. The mean values are calculated over 20 random seeds.
janyan et al., 2020). As is mentioned before, most of these approaches employ adversarial training strategies to improve the robustness of BERT finetuning. SMART solves supremum by using an adversarial methodology to achieve the largest KL di
vergence with an  $\epsilon$ -ball, FreeLB optimizes a direct adversarial loss  $\mathcal{L}_{FreeLB}(\theta) = \sup_{\Delta \theta: |\Delta \theta| \leq \epsilon} \mathcal{L}(\theta + \Delta \theta)$  through iterative gradient ascent steps, while R3F removes the adversarial nature of SMART and optimize the smoothness of the whole model directly.
Compared with this sort of adversarial based algorithms, our method is easier to implement and provides a relatively rigorous theoretical guarantee. The design of layer-wise regularization is sensible that it exploits the characteristics of hierarchical representations in modern deep neural networks. Studies in knowledge distillation have shown similar experience that imitating through middle layer representations (Adriana et al., 2015; Zagoruyko and Komodakis, 2016) performs better than aligning the final outputs (Hinton et al., 2015). Moreover, LNSR allows us to use different regularization weights for different layers (we use fixed weight 1 on all layers in this paper). We will leave the exploitation in future work.
In this paper, we propose the Layer-wise Noise Stability Regularization (LNSR) as a lightweight and effective method to improve the generalizability and stability when fine-tuning BERT on few training samples. Our proposed LNSR method is a general technique that improves model output stability while maintaining or improving the original performance. Furthermore, we provide a theoretically analysis of the relationship of our model to the Lipschitz continuity and Tikhonov regularizer. Extensive empirical results show that our proposed method can effectively improve the generalizability and stability of the BERT model.
Hang Hua would like to thank Jeffries for supporting his research.
Romero Adriana, Ballas Nicolas, K Samira Ebrahimi, Chassang Antoine, Gatta Carlo, and B Yoshua. 2015. Fitnets: Hints for thin deep nets. Proc. ICLR.
Armen Aghajanyan, Akshit Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2020. Better fine-tuning by reducing representational collapse. ArXiv, abs/2008.03156.
Sanjeev Arora, R. Ge, Behnam Neyshabur, and Yi Zhang. 2018. Stronger generalization bounds
for deep nets via a compression approach. ArXiv, abs/1802.05296.
Roy Bar-Haim, I. Dagan, B. Dolan, Lisa Ferro, Danilo Giampiccolo, and B. Magnini. 2006. The second pascal recognising textual entailment challenge.
Peter Bartlett, Dylan J Foster, and Matus Telgarsky. 2017. Spectrally-normalized margin bounds for neural networks. arXiv preprint arXiv:1706.08498.
Chris Bishop. 1991. Improving the generalization properties of radial basis function neural networks. Neural computation, 3(4):579-588.
Chris M Bishop. 1995. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108-116.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. ArXiv, abs/1508.05326.
Daniel Matthew Cer, Mona T. Diab, Eneko Agirre, I. Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. ArXiv, abs/1708.00055.
I. Dagan, Oren Glickman, and B. Magnini. 2005. The pascal recognising textual entailment challenge. In MLCW.
Andrew M. Dai and Quoc V. Le. 2015. Semi-supervised sequence learning. In NIPS.
J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. ArXiv, abs/2002.06305.
W. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In IWP@IJCNLP.
D. Erhan, Aaron C. Courville, Yoshua Bengio, and Pascal Vincent. 2010. Why does unsupervised pretraining help deep learning? J. Mach. Learn. Res., 11:625-660.
D. Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, S. Bengio, and P. Vincent. 2009. The difficulty of training deep architectures and the effect of unsupervised pre-training. In AISTATS.
Stuart Geman, Elie Bienenstock, and René Doursat. 1992. Neural networks and the bias/variance dilemma. *Neural computation*, 4(1):1-58.
Danilo Giampiccolo, B. Magnini, I. Dagan, and W. Dolan. 2007. The third pascal recognizing textual entailment challenge. In ACL-PASCAL@ACL.
Belize Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. 2020. Supervised contrastive learning for pre-trained language model fine-tuning. *ArXiv*, abs/2011.01403.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrievalaugmented language model pre-training. ArXiv, abs/2002.08909.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. Smart: Robust and efficient fine-tuning for pretrained natural language models through principled regularized optimization. In ACL.
Tushar Khot, A. Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In AAAI.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. ArXiv, abs/1901.07291.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909.11942.
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2020. Mixout: Effective regularization to fine-tune large-scale pretrained language models. ArXiv, abs/1909.11299.
Xuhong Li, Y. Grandvalet, and F. Davoine. 2018. Explicit inductive bias for transfer learning with convolutional networks. In ICML.
Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Yang Liu. 2019. Fine-tune bert for extractive summarization. ArXiv, abs/1903.10318.
I. Loshchilov and F. Hutter. 2019. Decoupled weight decay regularization. In ICLR.
Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442-451.
Tomas Mikolov, Ilya Sutskever, Kai Chen, G. S. Corrado, and J. Dean. 2013. Distributed representations of words and phrases and their compositionality. ArXiv, abs/1310.4546.
Marius Mosbach, Maksym Andriushchenko, and D. Klakow. 2020. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. ArXiv, abs/2006.04884.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. 2017. Exploring generalization in deep learning. arXiv preprint arXiv:1706.08947.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Roman Novak, Y. Bahri, D. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. 2018. Sensitivity and generalization in neural networks: an empirical study. *ArXiv*, abs/1802.08760.
Jeffrey Pennington, R. Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. ArXiv, abs/1802.05365.
Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. ArXiv, abs/1811.01088.
A. Radford. 2018. Improving language understanding by generative pre-training.
A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Salah Rifai, Xavier Glorot, Yoshua Bengio, and Pascal Vincent. 2011. Adding noise to the input of a model trained with a regularized objective. arXiv preprint arXiv:1104.3250.
K. Roth, Yannic Kilcher, and Thomas Hofmann. 2019. Adversarial training generalizes data-dependent spectral norm regularization. ArXiv, abs/1906.01527.
Amartya Sanyal, P. Torr, and Puneet K. Dokania. 2020. Stable rank normalization for improved generalization in neural networks and gans. *ArXiv*, abs/1906.04659.
Nitish Srivastava, Geoffrey E. Hinton, A. Krizhevsky, Ilya Sutskever, and R. Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15:1929-1958.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In EMNLP/IJCNLP.
Li Wan, Matthew D. Zeiler, Sixin Zhang, Y. LeCun, and R. Fergus. 2013. Regularization of neural networks using dropconnect. In ICML.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, F. Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. ArXiv, abs/1905.00537.
Alex Wang, Amanpreet Singh, Julian Michael, F. Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. ArXiv, abs/1804.07461.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Ralph A Willoughby. 1979. Solutions of ill-posed problems (an tikhonov and vy arsenin). SIAM Review, 21(2):266.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Z. Yang, Zihang Dai, Yiming Yang, J. Carbonell, R. Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS.
Y. Yoshida and Takeru Miyato. 2017. Spectral norm regularization for improving the generalizability of deep learning. ArXiv, abs/1705.10941.
J. Yosinski, J. Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In NIPS.
Sergey Zagoruyko and Nikos Komodakis. 2016. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi. 2020. Revisiting few-sample bert fine-tuning. *ArXiv*, abs/2006.05987.
C. Zhu, Y. Cheng, Zhe Gan, S. Sun, T. Goldstein, and Jing jing Liu. 2020a. Freelb: Enhanced adversarial training for natural language understanding. arXiv: Computation and Language.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, W. Zhou, H. Li, and T. Liu. 2020b. Incorporating bert into neural machine translation. ArXiv, abs/2002.06823.
The model we use for experiments in section 4 is the standard BERT large model with 24 layers staked Transformers (Vaswani et al., 2017) encoder, 1024 hidden size, and 16 self-attention heads. We initialize the pre-trained part of the model with BERT-Large-Uncased-Whole-Word-Masking weight. The final layer is a classification layer with 2048 parameters which contains  $0.0006\%$  of the total number of parameters in the model. We initialize the last layer with  $\mathcal{N}(0,0.02^2)$  and each bias is 0. For the position of noise injection, we uniformly chose the first layer as the noise regularization start point. In the sensitivity to the position of noise injection analysis section, we also try injecting noise from the different layers as is shown in Figure 5. As for the baseline model Mixout, we use the code from the Github repository https://github.com/bloodwass/mixout.git. The other baseline models are implemented by ourselves.
Table 4 summarizes dataset statistics used in this work. We use the standard GLUE benchmark datasets downloaded from https://gluebenchmark.com/tasks.
B Other Experimental Reports
We also report the maximum value we get during fine-tuning BERT with our proposed LNSR regularizer among a large number of random seeds and several noise injection positions, since the maximum value can also reflect the ability of the learning algorithm to reach an optimal point. The results are shown in Table 5, and we can see that on some tasks, fine-tuning BERT with LNSR is even competitive with fine-tuning state-of-the-art models which adopt more powerful modern architectures and pre-training strategies.
RTE	MRPC	CoLA	STS-B
Task	NLI	Paraphrase	Acceptability	Similarity
Metrics	Accuracy	Accuracy+F1/2	Matthews corr	Pearson/Spearman corr
# of labels	2	2	2	1
# of training samples	2.5k	3.7k	8.6k	7k
# of validation samples	276	408	1k	1.5k
# of test samples	3k	1.7k	1k	1.4k
Table 4: The summarization of the datasets used in this work.
RTE	MRPC	CoLA	STS-B
BERT (Devlin et al., 2019)	70.4	88.0	60.6	90.0
BERT (Phang et al., 2018)	70.0	90.7	62.1	90.9
LNSR (ours)	79.1	90.4	68.1	91.0
XLNet (Yang et al., 2019)	83.8	89.2	63.6	91.8
RoBERTa (Liu et al., 2019)	86.6	90.9	68.0	92.4
ALBERT (Lan et al., 2020)	89.2	90.9	71.4	93.0
Table 5: We report the maximum value we get when fine-tuning the LNSR model on different noise injection position and random seeds on the four tasks. On some tasks, BERT (standard BERT-large-uncased (Devlin et al., 2019)) with LNSR even become competitive with some newly proposed powerful models (bottom rows)
Figure 5: Performance distribution box plot of each model on the four tasks across 25 random seeds.