Abstract: The introduction of pre-trained language models has revolutionized natural language research communities. However, researchers still know relatively little regarding their theoretical and empirical properties. In this regard, Peters et al. perform several experiments which demonstrate that it is better to adapt BERT with a light-weight task-specific head, rather than building a complex one on top of the pre-trained language model, and freeze the parameters in the said language model. However, there is another option to adopt. In this paper, we propose a new adaptation method which we first train the task model with the BERT parameters frozen and then fine-tune the entire model together. Our experimental results show that our model adaptation method can achieve 4.7% accuracy improvement in semantic similarity task, 0.99% accuracy improvement in sequence labeling task and 0.72% accuracy improvement in the text classification task.

Introduction: The introduction of pre-trained language models, such as BERT BIBREF1 and Open-GPT BIBREF2 , among many others, has brought tremendous progress to the NLP research and industrial communities. The contribution of these models can be categorized into two aspects. First, pre-trained language models allow modelers to achieve reasonable accuracy without the need an excessive amount of manually labeled data. This strategy is in contrast with the classical deep learning methods, which requires a multitude more data to reach comparable results. Second, for many NLP tasks, including but not limited to, SQuAD BIBREF3 , CoQA BIBREF4 , named entity recognition BIBREF5 , Glue BIBREF6 , machine translation BIBREF7 , pre-trained model allows the creation of new state-of-art, given a reasonable amount of labelled data. In the post pre-trained language model era, to pursue new state-of-art, two directions can be followed. The first method, is to improve the pre-training process, such as in the work of ERNIE BIBREF8 , GPT2.0 BIBREF2 and MT-DNN BIBREF9 . The second method is to stand on the shoulder of the pre-trained language models. Among the many possibilities, one of them is to build new neural network structures on top of pre-trained language models. In principles, there are three ways to train the networks with stacked neural networks on top of pre-trained language models, as shown in Table TABREF1 . In Peters et al . BIBREF0 , the authors compare the possibility of option stack-only and finetune-only, and conclude that option finetune-only is better than option stack-only. More specifically, Peter et al. BIBREF0 argue that it is better to add a task-specific head on top of BERT than to freeze the weights of BERT and add more complex network structures. However, Peters et al. BIBREF0 did not compare option stack-and-finetune and finetune-only. On the other hand, before pre-trained deep language models became popular, researchers often use a strategy analog to option stack-and-finetune. That is, modelers first train the model until convergence, and then fine-tune the word embeddings with a few epochs. If pre-trained language models can be understood as at least partially resemblance of word embeddings, then it will be imprudent not to consider the possibility of option stack-and-finetune. In this study, we aim to compare the strategy stack-and-finetune and strategy finetune-only. More specifically, we perform three NLP tasks, sequence labeling, text classification, and question similarity. In the first tasks, we demonstrate that even without modifying the network structures, building networks on top of pre-trained language models might improve accuracy. In the second tasks, we show that by ensembling different neural networks, one can even improve the accuracy of fine-tuning only methods even further. Finally, in the last task, we demonstrate that if one can tailor-made a neural network that specifically fit the characteristics of the pre-trained language models, one can improve the accuracy even further. All the results indicate the strategy stack-and-finetune is superior to strategy finetune-only. This leads us to conclude that, at least, by overlooking the possibility strategy stack-and-finetune is imprudent. The contribution of this paper is two-fold. First, we propose a new strategy to improve the fine-tune-only strategy proposed by Peter et al. BIBREF0 , this allows us to achieve better results, at least on the selected tasks. More importantly, the results of this study demonstrate the importance of neural networks design, even in the presence of all-powerful pre-trained language models. Second, during the experiment, we have found that although simply using the proposed training strategy can result in higher accuracies compared to that of Peter et al. BIBREF0 , it is still a challenging task to find the appropriate methods to design and to utilize pre-trained networks. In this regard, we find that pre-trained models differ significantly from word embeddings in terms of their training strategies. Especially, since word embeddings can be viewed as shallow transfer learning, while pre-trained model should be viewed as deep transfer learning, one must try to combat over-fitting problems with more care due to the enormous number of parameters presented in the pre-trained models. Besides, we also find that in order to achieve the maximal performance in the post-pre-trained language model era, one must design, either manually or via Auto ML, networks that best fit the structure, especially the depth of the pre-trained language models. The rest of the paper is organized as follows. First, we review the relevant literature on pre-trained deep neural networks, the argument in Peter et al. BIBREF0 as well as fine-tuning strategies with word embeddings. Second, we present three experiments and showed the superiority of strategy stack-and-finetune compared to strategy finetune-only. Finally, we conclude with some remarks and future research possibilities.

Question: what models did they compare with?
Answer: BERT, BERT + Bi-LSTM, BERT + HighwayLSTM, BERT + DenseNet, Ensembled Model, BERT + BIMPM, BERT + Sim-Transformer