Among the many possibilities, one of them is to build new neural network structures on top of pre-trained language models. In principles, there are three ways to train the networks with stacked neural networks on top of pre-trained language models, as shown in Table TABREF1 . In Peters et al .  , the authors compare the possibility of option stack-only and finetune-only, and conclude that option finetune-only is better than option stack-only. More specifically, Peter et al.  argue that it is better to add a task-specific head on top of BERT than to freeze the weights of BERT and add more complex network structures. However, Peters et al.  did not compare option stack-and-finetune and finetune-only. On the other hand, before pre-trained deep language models became popular, researchers often use a strategy analog to option stack-and-finetune. That is, modelers first train the model until convergence, and then fine-tune the word embeddings with a few epochs. If pre-trained language models can be understood as at least partially resemblance of word embeddings, then it will be imprudent not to consider the possibility of option stack-and-finetune.

Experiment A: Sequence Labeling	In the sequence labeling task,we explore sub-task named entity recognition using CoNLL03 dataset  , which is a public available used in many studies to test the accuracy of their proposed methods  ,  ,  ,  ,  . For strategy finetune-only and strategy stack-and-finetune, we implemented two models: one with BERT and the other with BERT adding a Bi-LSTM on top. Eval measure is accuracy and F1 score. As is shown in Table 2, even without modifying the networks to specifically adapt to the pre-trained model, our training strategy still brought improvement towards overall accuracy of 0.99% for the accuracy and 0.068 on the F1 score, proving the success of our proposed methods.

Overview	We perform three different experiments to test our hypotheses. First, we perform a named entity recognition tasks, by adding a bi-LSTM on top of the BERT model. In this experiment, we hope to test whether, without any modification to the commonly used network structure, our proposed training strategy will improve the overall accuracy. Second, we perform a text classification experiments, in this experiments, we trained three models, and perform a model ensemble. We hope to show that even the added network has not contributed to significantly in improving the accuracy, it does provide opportunities for model ensembles. Finally, we perform the textual similarity tests, in which we show that if one can tailor make a network that specifically fit the characteristics of the pre-trained languages, more significant improvement can be expected. Under the strategy finetune-only, we use only single BERT.In order to adapt to different tasks, we will add a fully connected layer upon BERT. In the sequence labeling task, the BERT word embedding of each word passes through two fully connected layers, and the prediction probability of named entity can be obtained. In the next two verification tasks, we use “[CLS]” for prediction and add two fully connected layers subsequently.

Experiment C: Semantic Similarity Tasks	We use “Quora-Question-Pair” dataset 1. This is a commonly used dataset containing 400k question pairs, annotated manually to be semantically equivalent or not. Due to its high quality, it is a standard dataset to test the success of various semantic similarity tasks. Various models which are tested on this data set are proposed, including but not limited to  ,  ,  ,  . Apart from the BERT fine-tuning only model and BERT+ BIMPM model, we also devise two new network structures by modifying the BIMPM model. In the first model is to remove the first bi-LSTM of BIMPM, which is the input layer for the matching layer in BIMPM. In the second model, we combine the matching layer of BIMPM and with a transformer  , a model we call Sim-Transformer by replacing the output layer of the matching layer, originally a bi-LSTM model, with a transformer model.

More importantly, the results of this study demonstrate the importance of neural networks design, even in the presence of all-powerful pre-trained language models. Second, during the experiment, we have found that although simply using the proposed training strategy can result in higher accuracies compared to that of Peter et al.  , it is still a challenging task to find the appropriate methods to design and to utilize pre-trained networks. In this regard, we find that pre-trained models differ significantly from word embeddings in terms of their training strategies. Especially, since word embeddings can be viewed as shallow transfer learning, while pre-trained model should be viewed as deep transfer learning, one must try to combat over-fitting problems with more care due to the enormous number of parameters presented in the pre-trained models. Besides, we also find that in order to achieve the maximal performance in the post-pre-trained language model era, one must design, either manually or via Auto ML, networks that best fit the structure, especially the depth of the pre-trained language models. The rest of the paper is organized as follows. First, we review the relevant literature on pre-trained deep neural networks, the argument in Peter et al.  as well as fine-tuning strategies with word embeddings.

The first is that due to the enormous number of parameters presented in the pre-trained language models, to achieve generalizable results on the test data sets, it is vital to combat over-fitting. In classical embedding + training networks, the general training method is to fix the word-embeddings, then train the top model until it converges, and finally fine-tuning the word-embeddings for a few epochs. This training strategy does not work when we replace pre-trained language models with word-embeddings. In our experiment, we first fix the pre-trained language models, and then we train the top neural networks only for a few epochs, until it reaches a reasonable accuracy, while closely monitoring the discrepancy between training accuracy and testing accuracy. After that, we fine-tune the pre-trained language model as well as our models on top together. This allows us to achieve better results on the experimentation. However, it is not yet clear to us when to stop the training of top neural networks. This poses an even more essential question for Auto ML researchers in the following sense.

If one would fine-tune directly the top models along with the weights in BERT, one is faced with the following dilemma: on the one hand, if the learning rate is too large, it is likely to disturb the structure innate to the pre-trained language models; on the other hand, if the learning rate is too small, since we top BERT with relatively complex models, the convergence of the top models might be impeded. Therefore, in the first phase we fix the weights in the pre-training language models, and only train the model on top of it. Another aspect that is worth commenting in the first phase is that it is most beneficial that one does not train the top model until it reaches the highest accuracy on the training or validation data sets, but rather only up to a point where the prediction accuracy of the training and validation data sets do not differ much. This is intuitively reasonable for the following reasons. Unlike word embeddings, the pre-trained language models possess a large number of parameters compared to the task-specific models we build on top them. Therefore, if one were to train the top models until they reach the highest prediction accuracy in the training or validation data sets, it would likely cause the models to over-fit.

Introduction	The introduction of pre-trained language models, such as BERT  and Open-GPT  , among many others, has brought tremendous progress to the NLP research and industrial communities. The contribution of these models can be categorized into two aspects. First, pre-trained language models allow modelers to achieve reasonable accuracy without the need an excessive amount of manually labeled data. This strategy is in contrast with the classical deep learning methods, which requires a multitude more data to reach comparable results. Second, for many NLP tasks, including but not limited to, SQuAD  , CoQA  , named entity recognition  , Glue  , machine translation  , pre-trained model allows the creation of new state-of-art, given a reasonable amount of labelled data. In the post pre-trained language model era, to pursue new state-of-art, two directions can be followed. The first method, is to improve the pre-training process, such as in the work of ERNIE  , GPT2.0  and MT-DNN  . The second method is to stand on the shoulder of the pre-trained language models.

In the classical computer vision based Auto ML approaches, since one seldom build networks on already trained models, there is no particular need to auxiliary measure for over-fittings. While if Auto ML is to be performed on NLP tasks successfully, it might be essential that the gap between training accuracy and test accuracy to be incorporated when one evaluates the model. Finally, it is not yet clear what is the most proper way to build networks that tops the pre-trained language models. However, there are several principles that we can follow when designing such networks. First, such networks must be able to ensure the gradient flow from the top of the model to the bottom. This is essential due to the depth of the pre-trained language model. Second, this also means, one does not need explicitly to build extremely complex networks on top of pre-trained language models unless it complements the mechanisms of self-attention. Finally, a challenge remains as to how to use the depth of pre-trained language models. The process of our experiment shows that utilizing deeper layers might be a fruitful way to achieve better accuracy.

Discussions and Conclusions	In summary, we find that in all the three tasks, our proposed method out-performs the methods of simply tuning pre-trained language models, as is proposed in  . However, we would like to caution the readers in two aspects when reading the conclusion of this study. First, this study does not argue that our proposed methods are always superior to fine-tuning only methods. For example, all the experiments in our study are based on data sets of relatively large size. In the other spectrum, if one is only given a limited data set, then building complex networks upon pre-trained language models might lead to disastrous over-fitting. If this is the case, then it is possible that deep domain adaptation  might be a better choice if one desires to stack neural networks on top of pre-trained language models. However, most domain adaptation applications belong to the field of computer vision, therefore, a call for domain adaptations research in the NLP fields. During the experimentation, we also discover some tricks to obtain higher quality networks.

In the presence of the success of pre-trained language models, especially BERT  , it is natural to ask how to best utilize the pre-trained language models to achieve new state-of-the-art results. In this line of work, Liu et al.  investigated the linguistic knowledge and transferability of contextual representations by comparing BERT  with ELMo  , and concluded that while the higher levels of LSTM's are more task-specific, this trend does not exhibit in transformer based models. Stickland and Murray  invented projected attention layer for multi-task learning using BERT, which results in an improvement in various state-of-the-art results compared to the original work of Devlin et al.  . Xu et al.  propose a “post-training” algorithms, which does not directly fine-tune BERT, but rather first “post-train” BERT on the task related corpus using the masked language prediction task next sentence prediction task, which helps to reduce the bias in the training corpus. Finally, Sun et al.  added additional fine-tuning tasks based on multi-task training, which further improves the prediction power of BERT in the tasks of text classification.

Some early attempts include pre-trained models includes, CoVe  , CVT  ,  , ELMo  and ULMFiT  . However, the most successful ones are BERT  and Open-GPT  . Unlike standard NLP deep learning model, BERT and Open-GPT are built on top of transformer  structures, instead of LSTM  or GRU  . The difference between BERT and Open-GPT is that BERT uses bi-directional self-attentions while Open-GPT uses only unidirectional ones, as shown in Figure FIGREF2 . The transformer structures differ from the LSTM's in the two important aspects. First, it allows for stacking of multiple layers with residual connections and batch normalizations, which allows for free gradient flow. Second, the core computational unit is matrix multiplications, which allows researchers to utilize the full computational potential of TPU  . After training on a large corpus, both BERT and Open-GPT are able to renew the SOTA of many important natural language tasks, such as such as SQuAD  , CoQA  , named entity recognition  , Glue  , machine translation  .

From the experimental results shown in Table 4, we can see that due to the strong expressive ability of the BERT, there is almost no difference in the experimental results of removing the first bi-LSTM and BIMPM. In addition, we also find that Sim-Transformer's performance without fine-tuning is nearly four percentage points lower than BIMPM, but it out-performs BIMPM after fine-tuning. In general, the results show that BERT + Sim-Transformer out-performs BERT-only model by 4.7%, thus confirming our hypotheses again.

In this aspect, however, there is a simple yet crucial question that needs to be addressed. That is, whether it is possible to top BERT with the commonly used or task specific layers, and if this is possible, how to best utilize the pre-trained language models in this situation. In this regards, Peters et al.  investigated how to best adapt the pre-trained model to a specific task, and focused on two different adaptation method,feature extraction and directly fine-tuning the pre-trained model, which corresponding to the strategy finetune-only and the strategy stack-only in Table TABREF1 . On this regard, Peters et al.  performs five experiments, including: (1) named entity recognition  ; (2) sentiment analysis  ; (3) natural language inference  ; (4) paraphrase detection  ; (5) semantic textual similarity  . By the results of these tasks, Peters et al.  concludes that adding a light task-specific head and performing fine-tuning on BERT is better than building a complex network on top without BERT fine-tuning.

Methodology	Under our strategy stack-and-finetune, the model training process is divided into two phases, which are described in detail below. In the first phase, the parameters of the pre-training model are fixed, and only the upper-level models added for a specific task is learned. In the second phase, we fine-tune the upper-level models together with the pre-trained language models. We choose this strategy for the following reasons. Pre-training models have been used to obtain more effective word representations through the study of a large number of corpora. In the paradigm proposed in the original work by Devlin et al.  , the author directly trained BERT along with with a light-weighted task-specific head. In our case though, we top BERT with a more complex network structure, using Kaiming initialization  .

To Tune or Not To Tune? How About the Best of Both Worlds?	The introduction of pre-trained language models has revolutionized natural language research communities. However, researchers still know relatively little regarding their theoretical and empirical properties. In this regard, Peters et al. perform several experiments which demonstrate that it is better to adapt BERT with a light-weight task-specific head, rather than building a complex one on top of the pre-trained language model, and freeze the parameters in the said language model. However, there is another option to adopt. In this paper, we propose a new adaptation method which we first train the task model with the BERT parameters frozen and then fine-tune the entire model together. Our experimental results show that our model adaptation method can achieve 4.7% accuracy improvement in semantic similarity task, 0.99% accuracy improvement in sequence labeling task and 0.72% accuracy improvement in the text classification task.

Experiment B: Text Classification	In the task of text categorization, we used Yahoo Answer Classification Dataset. The Dataset is consists of 10 classes, but due to the huge amount of the dataset, we just select two class of them. As for the upper model,we choose DenseNet  and HighwayLSTM  . The DenseNet structure contains four independent blocks and each block has four CNNs connected by residual. We initialize word embedding in the word representation layer with BERT. We initialize each character as a 768-dimension vector. In the experiment of training DenseNet,we concat the output vector of DenseNet with [CLS] for prediction. We find the ensembled model enjoys a 0.72% improvements compared to the fine-tune only model and 0.005 improvement for the F1 score.

Under our strategy stack-and-finetune, we set different learning rates for the two phases. We tried to set the learning rate of the first stage to INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 and INLINEFORM4 , and set it to a smaller number in the latter stage, such as INLINEFORM5 , INLINEFORM6 , INLINEFORM7 and INLINEFORM8 . After our experiments, we found that it gets better results while the learning rate is set to 0.001 in the stage of training only the upper model and set to INLINEFORM9 in the later stage. Since BERT-Adam  has excellent performance, in our experiments, we use it as an optimizer with INLINEFORM10 , INLINEFORM11 -weight decay of INLINEFORM12 .We apply a dropout trick on all layers and set the dropout probability as 0.1.

In this study, we aim to compare the strategy stack-and-finetune and strategy finetune-only. More specifically, we perform three NLP tasks, sequence labeling, text classification, and question similarity. In the first tasks, we demonstrate that even without modifying the network structures, building networks on top of pre-trained language models might improve accuracy. In the second tasks, we show that by ensembling different neural networks, one can even improve the accuracy of fine-tuning only methods even further. Finally, in the last task, we demonstrate that if one can tailor-made a neural network that specifically fit the characteristics of the pre-trained language models, one can improve the accuracy even further. All the results indicate the strategy stack-and-finetune is superior to strategy finetune-only. This leads us to conclude that, at least, by overlooking the possibility strategy stack-and-finetune is imprudent. The contribution of this paper is two-fold. First, we propose a new strategy to improve the fine-tune-only strategy proposed by Peter et al.  , this allows us to achieve better results, at least on the selected tasks.

Related Studies	Before the introduction of deep neural networks, researchers in the field of NLP have been using pre-trained models. Among all of them, one of the most famous is the word embeddings, which maps each word into a continuous vector, instead of one-hot encodings  . By doing so, not only are we able to reduce the dimensionality of the input features, which helps to avoid over-fitting, but also capture, at least partially, the internal meaning of each word. However, since each word is only endowed with a fixed numerical vector in the methodology of word embeddings, word embeddings are unable to capture the contextual meaning in the text. For example, consider the word ”bank” sentences “I am walking on the bank of the river.” with “I am going to rob the bank”. It is obvious that the word “bank” represents completely different meaning, which the word embeddings techniques fail to capture. The aforementioned deficiencies prompt researchers to propose deep neural networks that are able to be trained in an unsupervised fashion while being able to capture the contextual meaning of the words presented in the texts.

Therefore, in our experiment, we found that this leads to the highest performance increase in the fine-tuning stage.

Second, we present three experiments and showed the superiority of strategy stack-and-finetune compared to strategy finetune-only. Finally, we conclude with some remarks and future research possibilities.

Question: did they test with other pretrained models besides bert?
Answer: No