Title: DATA ANNEALING TRANSFER LEARNING PROCEDURE FOR INFORMAL LANGUAGE UNDERSTANDING TASKS

Abstract: There are many applications for informal language understanding tasks in the real world. However, because informal language understanding tasks suffer more from data noise than formal ones, there is a huge performance gap between formal and informal language understanding tasks. The recent pre-trained models that improved the performance of formal language understanding tasks did not achieve comparable result on informal language. Although the formal tasks and informal tasks are similar in purpose, their language models significantly differ from each other. We propose a data annealing transfer learning procedure to bridge the performance gap on informal natural language understanding tasks. In the data annealing procedure, the training set contains mainly formal text data at first; then we gradually increase the proportion of the informal text data during the training process. We validate the data annealing procedure on three natural language understanding tasks: named entity recognition (NER), part-of-speech (POS) tagging, and chunking with two popular neural network models, LSTM and BERT. When BERT is fine-tuned with our learning procedure, it outperforms all the state-ofthe-art models on the three informal tasks.

1. INTRODUCTION:

Informal language is an important field in natural language processing (NLP). For example, tweets provide rich real-time information (Ritter et al., 2011) . However, because of the noisy nature of the informal language and the shortage of labelled data, the progress on informal language is not as promising as on formal language. Many tasks on formal data obtain a high performance due to the development of deep learning methods. But usually, these state-of-the-art models' good performance can not directly transfer to informal data. For example, when a BERT model is fine-tuned on informal data, such as twitter text, its performance is less encouraging than on formal data. This is caused by the domain discrepancy between the pre-training corpus used by BERT and the target data. Towards different tasks in informal language, NLP researchers have also proposed different neural network models. Kshirsagar et al. (2018) use neural networks and pre-trained word embedding to detect hate speech in social media. Gui et al. (2018) improved the accuracy of part-of-speech tagging task in twitter by proposing hypernetwork based method to separately model contexts with different expression styles. However, these models are usually designed specifically for certain language understanding task and cannot generalize to different tasks with good performance. Researchers also explored two types of classic neural network-based transfer learning methods: parameter initialization (INIT) and multi-task transfer learning (MULT) on natural language processing. Lee et al. (2018) explored the possibility of training the Long Short-Term Memory (LSTM) model on the NER task with INIT. Yang et al. (2017) explored different MULT transfer learning strategies on sequence labelling tasks based on the relation between the source domain and the target domain. However, the performance of these existing transfer learning methods shows limited improvement in language understanding tasks Lin & Lu (2018) .

To solve the issues mentioned above, we propose a data annealing procedure for transfer learning. In our proposed data annealing procedure, we set informal data as target data, and we set formal data as source data. During the training process, the training data first contains mainly source data. Our data annealing procedure takes the advantages of a good parameter initialization like INIT. Then the proportion of source data keeps decreasing while the proportion of target data keeps increasing. In the decaying process, it utilizes source data as an auxiliary task like MULT. Finally, the training set is mainly target data. We suspect the data annealing process enables the model to explore a larger fraction of parameter space and keep the knowledge learnt from the source data.

In our data annealing transfer learning procedure, the BERT model is fine-tuned and achieves good performance in informal language understanding tasks. Besides, different from previously proposed models which lack generalization ability, our data annealing could be employed on different tasks. We validate our learning procedure with two popular neural network models in NLP, LSTM and BERT, on three popular natural language understanding tasks, i.e., named entity recognition (NER), part-of-speech (POS) tagging and chunking on twitter. Figure 1 shows one example for each tasks. When LSTM is trained with our proposed learning procedure, the result shows that our procedure outperforms the state-of-the-art models on the NER task and the chunking task. When BERT is fine-tuned with our data annealing procedure, it outperforms all three state-of-the-art models. By doing this, we also set the new state-of-the-art result for the three informal language understanding tasks. Last, we simulate a low resources environment in the NER task. Results show that our data annealing procedure is also effective when there are limited training resources in target data.

2. RELATED WORK:

Due to the importance of informal language, many researchers have been working on the various tasks on informal language. Aguilar et al. (2019) proposed using phonetic and phonological features to obtain a better representation in a social media text. Owoputi et al. (2013) built a POS tagging system for twitter tasks by large-scale unsupervised word clustering and new lexical features. Ma et al. (2016) proposed using recurrent neural networks to detect rumor in microblog. Researchers also have been investigating the possibility of applying transfer learning on the informal domain. Transfer learning aims to solve a problem on a domain using the knowledge gained from a related domain (Weiss et al., 2016) . Parameter initialization (INIT) and multi-task learning (MULT) are two frequently used transfer learning techniques in NLP area. Lee et al. (2018) show the effect of transfer learning on named entity recognition task on informal language. Yang et al. (2017) examined different transfer learning methods based on the relation between the source domain and target domain. Lin & Lu (2018) proposed using domain adaptation in named entity recognition task on twitter text.

The core idea of annealing is to let the model have more freedom to explore its update direction or speed at the beginning of the training process. When the training process goes on, we gradually reduce the freedom of the model exploration. The philosophy of annealing has been frequently implemented in different machine learning areas. One of the popular usages of annealing is setting the neural model learning rate. Robbins & Monro (1951) and Zeiler (2012) show that a gradually decayed learning rate leads to better model performance. Moreover, recent state-of-the-art NLP models such as NCRF++ (Yang & Zhang, 2018) and BERT (Devlin et al., 2018) also empirically validate the effect of decaying learning rate. The idea of annealing is also applied to batch size adjustment in neural network model training. Smith et al. (2017) show that annealing the dataset batch size in training leads to a faster model convergence Bertsimas & Tsitsiklis (1993) . Simulated Inspired the wide application of annealing, we propose the concept of data annealing to improve the performance on informal data in the transfer learning setting. Data annealing gradually reduces the amount of source data and increases the target data during the training process. By adjusting the data ratio, the model explores a larger parameter space at first and focuses on the targeted data later. Compared to simulated annealing, our proposed data annealing replaces random noise with source data. By including source data, the model is able to explore more space at the beginning of the training process, but also the model is guided by the knowledge learned from the source domain. In INIT, before a time point T 0 , the model is trained on source dataset. After T 0 , the model is trained on target dataset. For MULT, the training data consists of target data and source data, and the proportion of source data is fixed to λ 0 . In our proposed data annealing procedure, the proportion of the source dataset keeps decreasing during the annealing process.

3. METHOD:

Data annealing is a transfer learning procedure that changes the proportional of formal source data and informal target data in the training process. At the beginning of the training, we let most of the training samples to be source data. So the model obtains a good initialization from the abundant clean source data, similar to a popular transfer learning technique in natural language processing (NLP), parameter initialization (INIT). We then gradually increase the proportion of target data and reduce the proportion of source data. As outlined in section 2, the model explores a larger parameter space. Besides, the labelled source dataset works as an auxiliary task like in multi-task transfer learning (MULT). At the end of the training process, most of the training data will be target data so that the model can focus on the target information. We illustrate the data annealing process in Figure 2 .

The proposed data annealing process is guided by a unified function. We reduce the source data proportion with an exponential decay function. We let α represent the initial proportion of source data in the training set, let t represent the current training step and let m represent the number of total update batches. We let λ represent the exponential decay rate α. r t S and r t T represent the proportion of the source data and proportion of target data at time step t.

r t S = α * λ t-1 , 0 < α < 1, 0 < λ < 1 (1) r t T = 1 -α * λ t-1

(2)

We let D S represent the size of source data, and B represent the batch size. We have When the model has been updated for adequate time, m will be large, so at the end of the training process, we can approximate D S to

D S = B * m i=1 r t S = B * m i=1 α * λ i-1 = B * α * (1 -λ m ) 1 -λ (3)

D S = B * α 1 -λ (4)

When we fine-tune large pre-trained model like BERT on a target task, it is suggested to avoid overtraining the model on the target training data (Sun et al., 2019) . Because most neural models have the catastrophic forgetting property, models will forget the knowledge from the pre-trained model if they are overfitted on target data (Kirkpatrick et al., 2016) . However, we do not want to feed too much source data as well, as it not only prolongs the training time but also confuse the model. We empirically decide the size of source data to have at the beginning based on different tasks. So based on Equation 4, we set α as:

EQUATION

)

Linear decay is popular in learning rate annealing (Zeiler, 2012; Reimers & Gurevych, 2019) . But linear annealing is not suitable for our proposed data annealing process. If we set a large value for α, the decay speed will be large, and α can decay to 0 in a short time. Such rapid change prevents the annealing process from taking effect in training. While if α is small, the model would not have a good parameter initialization from source data. Therefore we propose to use exponential decay in data annealing, as it decays smoothly and gradually. The proposed data annealing procedure can be used on any neural models. In this paper, we used LSTM and BERT as two examples.

4. EXPERIMENT DESIGN:

Because our proposed data annealing procedure could utilize knowledge from source domain without being binding with a specific neural network, it is applicable to all informal natural language understanding tasks. We validate it on three tasks: named entity recognition (NER), part-of-speech tagging (POS) and chunking. We choose these three tasks for two reasons. First, they are important tasks in understanding and have wide applications. Second, there exists a huge performance gap between formal text and informal text on these three tasks. These tasks are the perfect test bed for different transfer learning procedures. We test data annealing's effect on two types of popular neural models, LSTM and BERT.

4.1. TASKS AND DATASETS:

NER locates and classifies named entity mentions in unstructured text into predefined categories. POS tagging marks a word in a text with its corresponding part-of-speech. Chunking is to separate a sentence into a group of separate noun phrases. The majority of available models for NER, POS tagging and chunking were designed for formal data like news. Because informal data such as tweets, usually include numerous nonstandard spellings, abbreviations, these models usually perform poorly on informal text genres (Strauss et al., 2016) .

For NER task, we use OntoNotes-nw NER (Ralph Weischedel, 2013) as the source dataset, and use RitterNER dataset (Ritter et al., 2011) as the target dataset. While for POS tagging, we use Penn Treebank (PTB) POS tagging dataset as source data, and use RitterPOS (Ritter et al., 2011) as the target dataset. For the chunking task, we use CoNLL 2000 as the source dataset, and use RitterCHUNK (Ritter et al., 2011) as the target dataset. The statistic of these datasets is shown in Table 1 .

4.2. MODEL IMPLEMENTATION:

We implement two commonly used neural network models, LSTM and BERT to test the data annealing transfer learning procedure. Since parameter initialization (INIT) and multi-task learning (MULT) are two frequently used transfer learning technique in NLP (Weiss et al., 2016; Mou et al., 2016) , we also implement them with the same LSTM and BERT for comparison.

4.2.1. LSTM MODEL IMPLEMENTATION:

Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) is frequently used in natural language processing tasks. It is well-suited to classify, predict and process time signal given time lags of unknown duration. Many state-of-the-art models on informal language tasks are based on LSTM. Following Yang & Zhang (2018) and Yang et al. (2017) , we used character and word embedding as input features. We use one layer bidirectional LSTM to process the input features and use conditional random fields (CRFs) as the classifier. Since the source data and the target data has a different number of labels, we concatenate two separate conditional random fields (CRFs), one for the source data and one for the target data as the classifier. We use exponential data annealing as mentioned in section 3. The source data and target data flow to their corresponding CRF classifier. The parameters in CRF classifiers are updated by formal data and informal data separately, and the parameters in the LSTM or the BERT are updated by both formal data and informal data.

4.2.2. BERT MODEL IMPLEMENTATION:

There are mainly two versions of BERT: BERT BASE and BERT LARGE . Usually, BERT LARGE obtains a better performance compared with BERT BASE . We implement both BERT BASE model and BERT LARGE as baselines. We use the pre-trained model parameter released by Google. Besides, as suggested in the original BERT paper, we utilize BERT by concatenating the token representations from the top four hidden layers of the pre-trained Transformer. Similar to the implementation in LSTM, we concatenate CRF classifiers at the top of the BERT structure. It is different from the original paper of BERT, which use the softmax function to classify the token type. There are mainly two reasons that we replace the softmax with the CRF. First, CRF has been validated as a good classifier by many researchers (Lafferty et al., 2001; Tseng et al., 2005) . Besides, since all the LSTM baselines use CRF, in order to have a fair comparison with LSTM, it is necessary that we also use the CRF classifiers in BERT baselines.

4.3. BASELINE TRANSFER LEARNING PROCEDURES:

As mentioned in section 2, parameter initialization (INIT) and multi-task learning (MULT) are two commonly used transfer learning technique in NLP. We also apply these transfer learning methods on both LSTM and BERT. Then we compare the effect of our proposed data annealing procedure with INIT and MULT.

4.3.1. PARAMETER INITIALIZATION (INIT):

INIT contains three steps. First, a source model is trained on source domain by supervised learning or unsupervised learning. Then some or all of the parameters in the source model is used to initialize the target model. Finally, the target model is fine-tuned on the target domain.

Lee et al. ( 2018) and Mou et al. (2016) validate the effect of INIT in natural language understanding (NLU). We apply INIT on both BERT BASE and BERT LARGE . We set formal data as source data and informal data as target data. When training source BERT model with source data, in order to avoid catastrophic forgetting, we set the maximum training epochs on the source dataset to be 5. We first train the source model on source data. We choose the model that achieves the best performance on the validation set of source data for named entity recognition task (NER) and part-of-speech tagging task. For the chunking task, since the source dataset CoNLL 2000 does not contain a validation set, we choose the model that achieves the highest performance on the test set. Since source dataset and target dataset do not have the same number of labels, the CRF classifier for source data can not be used to initialize the parameter in the CRF classifier for target data. Therefore, we initialize the target model by parameters in the source model except for the parameters in the CRF classifier. Finally, we fine-tune the model on target dataset. 

4.3.2. MULTI-TASKS TRANSFER LEARNING (MULT):

Multi-task learning is another popular transfer learning technique. In MULT, a model is trained by both source and target data at the same time, and some or all of the parameters in the model are shared between source and target domain during the learning process.

We apply MULT on both LSTM and BERT. The training set consists of informal data and formal data. The proportion of source data λ M U LT in each batch is fixed during the training process. We concatenate the LSTM or BERT with two separate CRF classifiers because the source dataset and target dataset have a different number of labels. Source data and target data train all the parameters in the model jointly except for the parameters in the CRF classifiers.

5. EXPERIMENT RESULTS:

We use different metrics to measure each task's performance following previous work. We use precision, recall and F 1 in entity level to evaluate named entity recognition (NER) and chunking. We use accuracy in token level to evaluate part-of-speech (POS) tagging task. We compared our data annealing procedure with three types of baselines and state-of-the-art models in Table 2 . Vanilla means the model is trained without transfer learning, in other words, the model does not utilize the source data. A model with MULT means we apply multi-task transfer learning to train the model. A model with INIT means we apply the parameter initialization technique to train the model. A model with DA means the model is fine-tuned with our proposed data annealing procedure. While previous state-of-the-art results on the informal NER task, POS tagging task and chunking task are achieved by different models, we use the same model across different tasks to achieve the best result. We report the average result of three runs.

5.1. NAMED ENTITY RECOGNITION:

The results on named entity recognition task (NER) are in We notice that when LSTM or BERT is implemented with data annealing, the recall and F 1 are always higher than the previous state-of-the-art method, meaning our data annealing is especially good at finding entities in text.

Usually, a sentence contains more non-entity words than entity words. If the model is not sure whether a word belongs to a certain entity, the model is likely to predict it as non-entity in order to reduce the training loss. We observe that the previous state-of-the-art model achieved the highest precision, but its recall is lower than all versions of BERT BASE and BERT LARGE models. This indicates that the previous state-of-the-art methods achieve high performance by predicting fewer entities, while BERT models receive high performance by both covering more entities and predicting them correctly.

5.2. PART-OF-SPEECH TAGGING:

Different from the NER task and the chunking task which aim to identify text spans, POS tagging predicts every word's POS tag. So we use word-level accuracy to evaluate POS tagging. LSTM applied with our proposed data annealing procedure achieves higher accuracy than using the other two transfer learning procedures. Both BERT BASE and BERT LARGE fine-tuned using the data annealing procedure outperforms the same setting with other transfer learning procedures. The improvement over the previous state-of-the-art method is 1.49 in accuracy measure. We notice that the performance improvement is not as large as in the NER task. This could be partially explained by the discrepancy between source data and target data. The target dataset in POS task, i.e., RitterPOS, contains more labels than the labels in the source dataset. Besides, the labels in RiiterPOS contains several Twitter-specific tags: retweets, @usernames, hashtags, and URLs. These label types never occurred in the source dataset. So transfer learning can't benefit the recognition on tokens with these part-of-speech types. This suggests that the similarity between source data and target data is an important factor to consider in the transfer learning process.

5.3. CHUNKING:

When LSTM is used as the training model, our data annealing procedure achieves the best results among these transfer learning methods. When BERT BASE is applied as the training model, our proposed methods outperform all other transfer learning procedure. When BERT LARGE is applied as the learning model, our proposed data annealing procedure outperforms all other transfer learning procedure and outperforms the state-of-the-art model by 3.03 in F 1 . On chunking result, the performance meets our expectation. It further validates that our annealing procedure could improve the performance of informal language chunking task.

5.4. THE INFLUENCE OF THE DATASET SIZE:

To further evaluate the performance of our methods when there is limited labelled data, we randomly sample 10%, 20% and 50% of the training set in RitterNER. Then we compare our proposed DA BERT LARGE with INIT BERT LARGE and Vanilla BERT LARGE baselines on those three sampled datasets and the original RitterNER dataset. The result in Figure 4 shows that our model is slightly better than INIT BERT LARGE on limited resources condition and achieves a significant improvement over Vanilla BERT LARGE baseline.

6. ERROR ANALYSIS:

We performed error analysis on the named entity recognition task to understand each method better. We first calculated the F 1 score of the ten predefined entity types. We find that compared with Vanilla BERT LARGE and INIT BERT LARGE , DA BERT LARGE achieves higher F 1 score on two frequent entities, "PERSON" and "OTHER". INIT BERT LARGE achieves the higher F 1 score on another frequent entity type, "GEO-LOC". In the comparison of performance of other entities, we did not notice clear differences. Besides, we found there is a connection between the frequency of entity type and F 1 score of that entity. If the entity type of a word is infrequent in the dataset, all the three models are less likely to predict the word correctly. For example, since "TVSHOW" is one of the most infrequent entity types in the training set of RitterNER, none of these three models predicts a word with an entity type of "TVSHOW" correctly. Meanwhile, "PERSON" is the most frequent entity type in the training set, and the F 1 of "PERSON" achieves the second-highest F 1 . We suspect that if a word is a frequent entity type, then even if it never appeared in training set, another word that has a similar representation may be in the training data. Therefore, the model implicitly learns to predict a word by learning from other words that belong to the same type. Assigning more penalty to a less common word may be a solution to such issue.

Besides, we randomly sampled 30 sentences which contain at least one word whose entity type is mispredicted from the test set of RitterNER. We found all of the model have a difficulty in labelling noisy sentence. For example, all models fails to recognize JUTHTIN BEAVERRRR as entity type "PERSON" in sentence @darynjones imma tell you one time ! JUTHTIN BEAVERRRR !!!!. We would like to mention that there is a fairly large proportion of such noisy sentences in total sampled sentences. This fact suggests transfer learning has limited effect when the informal data contains too much noise.

7. CONCLUSION:

In this paper, we propose the data annealing transfer learning procedure for informal language understanding tasks, such as named entity recognition (NER), part-of-speech (POS) tagging and chunking.

It is applicable to various models such as LSTM and BERT. In the experiment, we show that when data annealing is applied with LSTM or BERT, it outperforms different state-of-the-art models on different informal language understanding tasks. Our proposed annealing technique is also effective when there is limited labelled resources. Moreover, compared with popular transfer learning methods, our data annealing procedure achieves better results on informal language understanding tasks.

In the future, we will explore a generalization version of data annealing to general domain transfer instead of from the formal domain to informal domain. Besides, we also think about an automatic annealing transfer learning procedure without setting the initial source data proportion and decaying rate. We think training loss and gradient may be a good reference.

:



Figures and Tables:

Figure 1: A named entity recognition, part-of-speech tagging and chunking example of informal language.

Figure 2: Data annealing procedure.

Figure 3 compare our data annealing procedure with INIT and MULT. It shows the connection among these three transfer learning technique from the variation of training set component during the training process.In INIT, before a time point T 0 , the model is trained on source dataset. After T 0 , the model is trained on target dataset. For MULT, the training data consists of target data and source data, and the proportion of source data is fixed to λ 0 . In our proposed data annealing procedure, the proportion of the source dataset keeps decreasing during the annealing process.

Figure 3: Comparison between data annealing transfer learning procedure, parameter initialization and multi-task transfer learning.

Figure 4: Performance on named entity recognition task. DA BERT LARGE indicates Vanilla BERT LARGE finetuned with data annealing.

Dataset statistics.

Table content: <table><tr><td>Task Type</td><td>Category Dataset</td><td colspan="2">Train Tokens Dev Tokens</td><td>Test Tokens</td></tr><tr><td>NER</td><td>Formal Informal RitterNER Ontonote-nw</td><td>848,220 37,098</td><td>144,319 4,461</td><td>49,235 4,730</td></tr><tr><td>POS Tagging</td><td>Formal Informal RitterPOS PTB 2003</td><td>912,344 10,857</td><td>131,768 2,242</td><td>129,654 2,291</td></tr><tr><td>Chunking</td><td colspan="2">Formal Informal RitterCHUNK 10,610 CoNLL 2000 211,727</td><td>-2,309</td><td>47,377 2,292</td></tr></table>

When LSTM is used as the learning model, our data annealing procedure achieves the highest recall and F -1 score compared to INIT and MULT. When BERT BASE is used as the learning model, our learning procedure achieves the highest precision and F-1 score among all transfer learning methods, and it also achieves the highest recall with INIT. When BERT LARGE is used as the learning model, our data annealing procedure achieves the best precision and F-1 score compared with INIT and MULT. It outperforms state-ofthe-art model by 3.16 in F 1 scores.

Table content: <table><tr><td>model</td><td>P</td><td>NER R</td><td>F 1</td><td>POS A</td><td>P</td><td>Chunking R</td><td>F 1</td></tr><tr><td>Vanilla LSTM</td><td>75.55</td><td>55.75</td><td>64.05</td><td>88.65</td><td>83.76</td><td>83.78</td><td>83.77</td></tr><tr><td>MULT LSTM</td><td>74.51</td><td>58.48</td><td>65.49</td><td>88.81</td><td>83.92</td><td>84.48</td><td>84.20</td></tr><tr><td>DA LSTM</td><td>75.51</td><td>61.01</td><td>67.45</td><td>89.16</td><td>83.81</td><td>85.37</td><td>84.58</td></tr><tr><td>BERT BASE</td><td>68.73</td><td>62.74</td><td>65.58</td><td>91.05</td><td>85.05</td><td>85.96</td><td>85.50</td></tr><tr><td>INIT BERT BASE</td><td>69.28</td><td>63.74</td><td>66.40</td><td>90.85</td><td>85.48</td><td>86.77</td><td>86.13</td></tr><tr><td>MULT BERT BASE</td><td>70.42</td><td>62.38</td><td>66.12</td><td>91.39</td><td>86.01</td><td>87.75</td><td>86.87</td></tr><tr><td>DA BERT BASE</td><td>71.09</td><td>63.74</td><td>67.21</td><td>91.55</td><td>86.16</td><td>87.91</td><td>87.03</td></tr><tr><td>Vanilla BERT LARGE</td><td>68.41</td><td>67.45</td><td>67.88</td><td>91.88</td><td>85.55</td><td>86.78</td><td>86.16</td></tr><tr><td>INIT BERT LARGE</td><td>68.85</td><td>69.20</td><td>68.99</td><td>92.04</td><td>86.42</td><td>87.59</td><td>87.00</td></tr><tr><td>MULT BERT LARGE</td><td>70.05</td><td>66.08</td><td>68.00</td><td>92.06</td><td>86.29</td><td>87.21</td><td>86.54</td></tr><tr><td>DA BERT LARGE</td><td>70.61</td><td>68.81</td><td>69.69</td><td>92.54</td><td>86.71</td><td>88.15</td><td>87.53</td></tr><tr><td>previous state-of-the-art*</td><td>76.12</td><td>59.10</td><td>66.53</td><td>91.17</td><td>84.47</td><td>84.54</td><td>84.50</td></tr></table>

Results on NER, POS tagging and chunking task. *The previous state-of-the-art methods are achieved by different models.Yang et al. (2019) proposed the state-of-the-art model in NER task;Gui et al. (2018) proposed the state-of-the-art model in POS tagging task;Yang et al. (2017) proposed the state-of-the-art model in chunking task.

Table content: <table/>

"PERSON" is a frequent concept in formal data, which is the source data in our transfer learning setting. It implies that DA BERT LARGE takes better use of knowledge from the source domain compared with INIT BERT LARGE and Vanilla BERT LARGE . Besides, "OTHER" means entities that are not in the ten predefined entity types. Higher performance on "OTHER" suggests DA BERT LARGE has a better understanding of the entity than INIT BERT LARGE and Vanilla BERT LARGE .

Table content: <table/>

Peer review (iclr2020):
- Summary This paper proposes “data annealing”, a simple and intuitive way to leverage data in the source and target domains, which gradually increases the proportion of training data from the target domain. Such an approach helps achieve state-of-the-art results on three informal language processing tasks (named entity recognition, part-of-speech tagging and text chunking) with BERT-large as the base model. However, the scientific contribution of this paper is relatively minor. The experiments are lack of details and therefore not clear enough to support the arguments: i. the author(s) did not mention how they chose or tuned hyperparameters, for both their method and baselines---the comparison is not guaranteed to be completely fair; ii. the author(s) made some vague arguments that are not supported or fully supported by experiments; iii. it is hard to follow the error analysis section without knowledge of the dataset. Besides, a lot of relevant references are missing. I generally suggest that this paper should be rejected, even if it achieves new state-of-the-art results. - Detailed comments and questions Although the general idea is simple and easy to follow, there are a few unclear points: 1. In the main text, D_S “represent the size of source data”. Does this mean “the number of instances in the source-domain training set” or “the number of instances in the source-domain training set times the number of epochs”? Note that it should be interpreted as the latter to ensure the correctness of Equation (3) when the model is trained for multiple epochs (at the transfer learning stage), though such an interpretation is definitely not natural. Also, how did you determine the value of D_S in Equation (5)? 2. While alpha is chosen w.r.t. source data size (Eq. (4)), how was lambda chosen? Similarly, how did you choose about T_0 and lambda_0 for INIT and MULT? Did you tune these hyperparameters? 3. Besides INIT and MULT, DA should outperform the following baseline (say, shuffle) before being claimed effective: (i) combine the source training set and the target one; (ii) at each step, randomly pick a batch from the combined training set. This is a strategy that looks like MULT, but the proportion of data from source/target domain could vary across batches. 4. Sec. 3: “Linear decay is popular… initialization from source data.” There is no experiment supporting the reason why linear annealing is not chosen . Have you tried linear data annealing scheme? 5. Table 2 reports a single number for each model on each task---can the model be affected by any random factor (e.g., random initialization, random shuffle of training instances)? If so, how stable is it across different random seeds (in terms of standard deviation across several runs with different random seeds)? 6. Sec. 4.2.2: “in order to have a fair comparison with LSTM, it is necessary that we also use the CRF classifiers in BERT baselines.” Such an argument is not rigorous. It is hard to say BERT can have a really fair comparison with LSTM---there are a lot of other factors that affect the fairness (e.g., BERT has seen much more data than LSTMs at the pretraining stage). 7. Sec. 5.2: “We notice that the performance improvement is not as large as in the NER task … This suggests that the similarity between source data and target data is an important factor to consider in the transfer learning process. “ NER and POS tagging are completely different tasks---it is unfair to compare the absolute F1 scores across different tasks and draw any conclusions based on that comparison. 8. Sec. 6: “we randomly sampled 30 sentences…” It would be good to show the randomly sampled sentence in appendix. 9. Sec. 7: “we show that when data annealing is applied with LSTM or BERT, it outperforms different state-of-the-art models on different informal language understanding tasks” DA LSTM did not outperform state-of-the-art model on POS tagging. - Missing Reference The paper proposes a transfer learning/domain adaptation technique, but few work on the topic was cited. To name a few in NLP, https://www.aclweb.org/anthology/W19-4302.pdf https://www.aclweb.org/anthology/D17-1038.pdf https://www.aclweb.org/anthology/P07-1033.pdf I strongly recommend the author(s) to also take a look at the papers cited by them and those cite them. It would be good to state the relation and difference between the proposed approach and curriculum learning (Bengio et al., 2009). - Minor comments Title: “learning” -> “Learning”. Equations: * -> , to avoid confusion with convolution. Just introduce the abbreviation at the first mention. Table 2: “BERT_{BASE}” -> “Vanilla BERT_{BASE}”.