Taking Notes on the Fly Helps Language Pre-Training
Authors: Qiyu Wu
Chen Xing
Yatao Li
Guolin Ke
Di He
Tie-Yan Liu
Venue: ICLR 2021 Poster (2021)
Keywords: Natural Language Processing
Pre-training

Abstract: How to make unsupervised language pre-training more efficient and less resource-intensive is an important research direction in NLP. In this paper, we focus on improving the efficiency of language pre-training methods through providing better data utilization. It is well-known that in language data corpus, words follow a heavy-tail distribution. A large proportion of words appear only very few times and the embeddings of rare words are usually poorly optimized. We argue that such embeddings carry inadequate semantic signals, which could make the data utilization inefficient and slow down the pre-training of the entire model. To mitigate this problem, we propose Taking Notes on the Fly (TNF), which takes notes for rare words on the fly during pre-training to help the model understand them when they occur next time. Specifically, TNF maintains a note dictionary and saves a rare word's contextual information in it as notes when the rare word occurs in a sentence. When the same rare word occurs again during training, the note information saved beforehand can be employed to enhance the semantics of the current sentence. By doing so, TNF provides a better data utilization since cross-sentence information is employed to cover the inadequate semantics caused by rare words in the sentences. We implement TNF on both BERT and ELECTRA to check its efficiency and effectiveness.  Experimental results show that TNF's training time is 60% less than its backbone pre-training models when reaching the same performance.  When trained with same number of iterations, TNF outperforms its backbone methods on most of downstream tasks and the average GLUE score. Code is attached in the supplementary material.

Published as a conference paper at ICLR 2021
TAKING NOTES ON THE FLY HELPS LANGUAGE TRE-TIANN
Qiyu Wu Chen Xing"', Yatao Li", Guolin Ke Di He Tie-Yar Liu Peking University
"College of Compute Science, Nankai University
Microsoft Research
qiyu. wwwpku.epkkkukommm eelepuu.eu..com edu. en xingchen1113@ gmail. com
[yatli, guolin.ke, dihe, tyliu)emicrosoft..o
ABSTRACT
How to make unsupervised language pre-training more efficient and less resource- intensive is important research direction in NLP. In this paper, focus on improving the efficiency of language pre-training methods through providing better data utilization. well-know that language data corpus, words follow a heavy-tail distribution. A large proportion of words appear only very few times and the embeddings of rare words are usually poorly optimized. We argue that such embeddings carry inadequate semantic signals, which could make the data utilization inefficient and slow down the pre-training of the entire model. To mitigate this problem, we propose Taking Notes the Fly (TNF), which takes notes for rare words on the fly during pre-training to help the model understand them when they occur next time. Specifically, TNF maintains note dictionary and saves a rare word's contextual information in it as notes when the rare word occurs in sentence. When the same word occurs again during training, the note information saved beforehand can be employed to enhance the semantics of the current sentence. By doing so, TNF provides better data utilization since cross-sentence information is employed to cover the inadequate semantics caused by rare words the sentences. We implement TNF on both BERT and ELECTRA to check its efficiency and effectiveness. Experimental results show that TNF's training time is 60% less than its backbone pre-training models when reaching the same performance. When trained with the same number of iterations, TNF outperforms its backbone methods on most of downstream tasks and the average GLUE score. Source code is attached in the supplementary material.
INTRODUCTION
Unsupervised language pre-training, c.g., BERT (Devlin et al. 2018), is shown to be a successful way improve the performance of various NLP downstream tasks. However, as the pre-training task requires no human labeling effort, a massive scale of training corpus from the Web can be used to train models with billions of parameters (Raffel et al. 2019), making the pre-training computationally expensive. As an illustration, training BERT-base model on Wikipedia corpus requires more than five days 16 NVIDIA Tesla V100 GPUs. Therefore, how make language pre-training more efficient and less resource-intensive, has become an important research direction in the field (Strubell al. 2019).
Our work aims at improving the efficiency of language pre-training methods. In particular, we study how speed up pre-training through better data utilization. It well-known that a natural language data corpus, words follow a heavy-tail distribution [Larson] 2010). A large proportion of words appear only very few times and the embeddings of those (rare) words are usually poorly optimized and noisy (Bahdanau et al. 2017 Gong et al. 2018 Khassanov et al.) 2019 Schick & Schütze 2020).
"quual Contribution. Work done during internships at Microsoft Research Asia. Correspondence to:dihe mierosoft.com

---

Published as a conference paper at ICLR 2021
Without
With Notes
COvID-19has cost thousands of
COVID- 19 has cost thousands of lives.
Pandomic; global crisis
ih COVID-19
dollars?
donuts?
A note of 'COVID- 19" taken from a previously seen sentence:
puppies?
tomatoes?
The COVID pandemic is an ongoing
global crisis.
An illustration of how taking notes of rare words can help language understanding. The left part of the figure shows that without any understanding of the rare word "COVID-19", there are too many grammatically-corre correct, while semanticallywwrong options for us to fill in the blank. the right half, we show that a note of "COVID-19" taken from previously sentence can act as very strong signal for us to predict the correct word at the masked position.
Figure I:
Unlike previous works that sought to merely improve the embedding quality of rare words, we argue that the existence of rare words could also slow down the training process of other model parameters. Taking BERT as an example, if we imagine the model encounters the following masked sentence during pre-training:
COVID-19 has cost thousands of lives.
Note that "COVID-19" a rare word, while also the only key information for the model to rely on to fill in the blank with the correct answer "lives". As the embedding of the rare word "COVID-19" is poorly trained, the Transformer lacks concrete imput signal to predict "lives". Furthermore, with noisy inputs, the model needs to take longer time to converge and sometimes even cannot generalize well (Zhang et al. 2016). Empirically, we observe that around 20% of the sentences in the corpus contain at least one rare word. Moreover, since most pre-training methods concatenate adjacent multiple sentences to form one input sample, empirically we find that more than 90 % of input samples contain least one rare word. The large proportion of such sentences could cause severe data utilization problem for language pre-training due to the lack of concrete semantics for sentence understanding. Therefore, learning from the masked language modeling tasks using these noisy embeddings may make the pre-training inefficient. Moreover, completely removing those sentences with rare words is not an applicable choice either since it will significantly reduce the size of the training data and hurt the final model performance.
Our method to solve this problem is inspired by how humans manage information. Note-taking is useful skill which can help people recall information that would otherwise be lost, especially for new concepts during learning (Makany et al., 2009). people take notes when facing a rare word that they don't 't know, then next time when the word appears, they can refer to the notes to better understand the sentence. For example, we may meet the following sentence somewhere beforehand: The COVID-19 pandemic is an ongoing global crisis. From the sentence, we can realize that "COVID-19" is related to "pandemic" and "global crisis" and record the connection in the notes. When facing "COVID-19" again in the note of "COVID-19". It is easy to see that once "pandemic" and "global crisis" are connected to "COVID-19", we can understand the sentence and predict "lives" more easily, as illustrated in Figure [] Mapped back to language vre-training we believe for rare words, explicitly leveraging cross- coss-sentence information is helpful to enhance semantics of the rare words in the current sentence predict the masked tokens. Through this more efficient data utilization, the Transformer can receive better input signals which leads to more efficient training of its model parameters.
task above, we can refer the
Motivated by the discussion above, we propose a new learning approach called "Taking Notes on the (TNF) to improve data utilization for language pre-training. Specifically, we maintain a note dictionary, where the keys are rare words and the values are historical contextual representations of them. In the forward pass, when a rare word w appears in a sentence, we query the value of w in the note dictionary and use it as part of the input. In this way, the semantic information of w saved in the note can be encoded together with other words through the model. Besides updating the model parameters, also update the note dictionary. In particular, we define the note of wo in the current
2

---

Published as a conference paper at ICLR 2021
sentence as the mean pooling over the contextual representations of the words nearby Then update w's 's value in the note dictionary by a weighted linear combination of w's previous value and w's 's note in the current sentence. TNF introduces little computational overhead at pre- tre-training since the note dictionary is updated on the fly during the forward pass. Furthermore, different from the memory-augmentee neural networks (Santoro al., 2016, Guu et al.) 2020), the note dictionary is only used to improve the training efficiency of the model parameters, while not served as a part of the model. When the pre-training finished, we discard the note dictionary and use the trained Transformer encoder during the fine-tuning of downstream tasks, same as all previous works.
We conduct experiments using BERT and ELECTRA (Clark al. 2019) as TNF's backbone methods. Results show that TNF significantly expedites BERT and ELECTRA, and improves their performances on downstream tasks. BERT-TNF and ELECTRA- -NF's training times are both 60% less than their corresponding backbone models when reaching the same performance. When trained with the same number of iterations, BERT-TNF and ELECTRA-TNE outperform the backbone methods on both the average GLUE score and the majority of individual tasks. We also observe that even in the downstream tasks where rare words only take a neglectable proportion the data (i.e. 0.47%), TNF also outperforms baseline methods with large margin. It indicates that TNF improves the pre-training of the entire model.
RELATED WORK
Efficient BERT pre-training. The massive energy cost of language pre-training (Strubell et al., 2019) has become an obstacle to its further developments. There are several works aiming at reducing the energy cost of pre- prainngi Gong al. (2019) observes that parameters in different layers have similar attention distribution, and propose a parameter distillation method from shallow layers to deep layers. Another notable work is ELECTRA (Clark et al. 2019), which develops new task using one discriminator and one generator. The generator corrupts the sentence, and the discriminator trained predict whether each word the corrupted sentence is replaced or not. Orthogonal to them, we focus on improving pre- pre-training efficiency by finding ways to utilize the data corpus better. Therefore, can be applied to all of the methods above further boost their performances.
Representation of rare words. It is widely acknowledged that the quality of rare words' embed- dings is significantly worse than that of popular words. Gao al.] (2019) provides a theoretical understanding of this problem, which illustrates that the problem lies in the sparse (and inaccurate) stochastic optimization of neural networks. Several works attempt to improve the representation of
rare words using linguistic priors (Luong et al., 2013 El--Kihky et al., 2019 Kim et al. 2016 Santos & Zadrozny 2014). But the improved embedding quality is still far behind that of popular words (Gong et al. 2018). Sennrich et al. (2015) develops novel way to split each word into sub- word units. However, the embeddings of fow-frequency sub-word units are still difficult to train (Ott et al. 2018). Due the poor quality of rare word representations, the pre-training model built on top of it suffers from noisy input semantic signals which lead to inefficient training. We try to bypass the problem of poor rare word representations by leveraging cross-sentence information to enhance input semantic signals of the current sentence for better model training.
uses memory-augmentedd neural networks in language-related tasks. Févry et al.] (2020) and Guu et al. (2020) define the memory buffer an external knowledge base of entities for better open domain question answering tasks. Khandelwal et al. (2019) constructs the memory for every test context at inference, to hold extra token candidates for better language modeling. Similar to other memory- augmented neural networks, the memory buffer in these works is model component that will be used during inference. Although sharing general methodological concepts with these works, the goal and details of our method are different from them. Especially, our note dictionary is only maintained in pre-training for efficient data utilization. At fine-tuning, we ditch the note dictionary, hence adding no extra time or space complexity to the backbone models.
Memory-augmented BERT. Another line of work close
3
TAKING NOTES ON THE FLY
3.1 PRELIMINARY
In this section, we use the BERT model as an example to introduce the basics of the model architecture and training objective of language pre-training. BERT (Bidirectional Encoder Representation from
3

---

Published as a conference paper at ICLR 2021
Transformers) is developed multi-layer bidirectional Transformer encoder, which takes sequence of word semantic information (token embeddings) and order information (positional embeddings) as imput, and outputs the contextual representations of words.
Each Transformer layer is formed by a self-attention sub-layer and a position-wise feed-forward sub-layer, with a residual connection (He et al.. (016) and layer normalization (Ba et al., 2016) applied after every sub-layer. The self-attention sub-layer is referred to as "Scaled Dot-Product Attention" in Vaswani al. (2017), which produces its output by calculating the scaled dot products of queries and keys as the coefficients of the values, i.e.,
OKT
Attention (Q, V) = Softmax
Vd )V.
Q (Query), K (Key), V (Value) are the hidden representations outputted from the previous layer and d is the dimension of the hidden representations. Transformer also extends the aforementioned self-attention layer to multi-head version in order to jointly attend to information from different representation subspaces. The multi-head self-attention sub-layer works as follows,
Multi-head(Q, K, V) =Concat(head), ... ,...,,ddy) W
(2)
headg =Attention KW VW)),
(3)
where wh E Rdxdk W E Bdddy W R are projection matrices. is the number of heads. dx and dy are the dimensions of the key and value separately.
Following the self-attention sub-layer, there is a position-wise feed-forware (FFN) sub-layer, which is fully connected network applied to every position identically and separately. The FFN sub-layer is usually a two-layer forward network with a ReLU activation funetion between. Given vectors {hi,.... hr J. a position-wis, FFN sub-layer transforms each h;; as FFN where Wi, W2, by and 62 are parameters.
W22
BERT uses the Transformer model as its backbone neural network architecture and trains the model parameters with the masked language model task on large text corpora. In the masked language model task, given sampled sentence from the corpora, 15% of the positions in the sentence randomly selected. The selected positions will be either replaced by special token [MASK], J, replaced by randomly picked tokens remain the same. The objective of BERT pre-training is to predict words at the masked positions correctly given the masked sentences. As this task requires no human labeling effort, large scale data corpus is usually used to train the model. Empirically, the trained model, served as good initialization, significantly improves the performance of downstream tasks.
3.2 TRAINING BERT BY TAKING NOTES ON THE FLY
As presented many previous works, the poorly-updated embeddings of rare words usually lack adequate semantic information. This could cause data utilization problem given the lack of necessary semantic input for sentence understanding, thus making the pre-training inefficient. In this section, we propose a method called Taking Notes on the Fly (TNF) to mitigate this problem. For ease of understanding, we describe TNF on top of the BERT model. While TNF can be easily applicd to any language pre- pre-training methods, such ELECTRA. The main component of TNF is note dictionary, saving historical context representations (notes) of rare words on the fly during pre-training. In the following of the section, introduce TNF by illustrating in detail how construct, maintain and leverage the note dictionary for pre-training.
The Construction of Note Dictionary. To enrich the semantic information of rare words for a better understanding of the sentence, we explicitly leverage cross-sentence signals for those words. We first initialize a note dictionary, NoteDict, from the data corpus, which will maintain a note representation (value) for each rare word (key) during pre-training. Since we target at rare words, the words in the dictionary are of low frequency. However, the frequency of the words in the dictionary should not be extremely low either. It is because if the word appears only once in the corpus, there will be no "cross- sentence signal" to use. Additionally, the note dictionary also shouldn's take too many memories in practice. With all these factors taken into consideration, we define keys those words with occurrences between 100 and 500 in the data corpus. The data corpus roughly contains 3.47 B words in total and the size of NoteDict's vocabulary is about 200k.
4

---

Published as a conference paper at ICLR 2021
Masked Language Model Task Loss
Note Dictionary
t
t
t
t
t
Wi
Contextual Vector
Transformer Output
2
3
4
5
6
Transformer Encoder
Update
Position
2
3
Embedding
+
+
Word
3
Embedding
Note
Get value
Embedding
MY
Query
Figure 2: The training framework of Taking Notes the FLY (TNF). The left box shows the forward pass with the help of the note dictionary. In the input word sequence, W2 is a rare word. Then for tokens 4 and 5 originated from W2, we query the value of w2 in the note dictionary and weighted average it with token/position embeddings. The right box demonstrates how we maintain the note dictionary. After the forward pass of the model, we can get the contextual representations of the word near W2 and use mean pooling over those representations as the note of Wy in the current sentence. Then, we update w2's ' value note dictionary by a weighted average the current note and its previous value.
Maintaining Note Dictionary. When we meet rare word in training sentence, we record the contextual information of its surrounding words in the sentence as its note. In detail, given a training sentence, each word will be first pre- pre-processed into sub-word units following standard pre-po pre-processing strategies (Sennrich et al. 2013). Therefore, given processed sequence sub-word units (tokens), rare word can occupy contiguous span of tokens. For rare word that appears both in the input token sequence æ {o1,-.. Z₁ ... Øn) and NoteDict, we denote the span boundary w in æ as (s, t), where and are the starting and ending position. We define the note of w for
(++
Note x) 2k
Cj
8
j-8-k
where each cj E IRd the output of the Transformer encoder on position j and served as the contextual representation of zy.- is half of the window size that controls how many surrounding tokens want to take as notes and save their semantics. If we refer to the example in the introduction, the contextual representations of "pandemic" and "global crisis" that the calculation of Note(w, w., x) is on the fly as we can obtain Note(w, x) during the forward pass using the current model. Therefore, there no additional computational cost.
summarized in the note of "COVID-19" Note
With the Note(w, .)) calculated with Equation for the current sentence æ, we can now update w's note saved in NoteDict to include the latest semantics in sentence In particular, we updates w's value in NoteDict using exponential moving average" In this way, at any occurrence of w during pre-training, its contextual information from all previous occurrences can be leveraged and used.
NoteDict w) (1 () NoteDDit(w)) + Note(w,
(5)
where 7 e (0, 1) is the discount factor.
Leveraging Note Dictionary for Pre-training. NoteDict explicitly contains surrounding contexts for rare words. We use such information as a part of the input to the Transformer encoder. For any masked token sequence in both NoteDict and x. Assume there are m rare words satisfying the conditions, denoted as
{x1,-". Fii ,ra), ), first find all rare words that appears
All values in NoteDict randomly initialized using the way word/positional emheddings.

---

Published as a conference paper at ICLR 2021
s,, where 8y and ty are the boundary of wy in æ. At the i-th position, the imput to the model is defined as
imput, =
(1- X) ipos.cmb; + token.emb, + A NoteDict(w;,) pos coos, + word_emi
E.. 83 <
< ty.
otherwise.
(6)
is hyper-parameter controlling the degree which TNF relies on historical context representations (notes) for rare words. We empirically set it as 0.5.
In the standard Transformer model, at position i, the imput to the first Transformer layer is the sum of the positional embedding pos_emb; and the token embedding token_emb; .emb,. In Equation 6 when the token 2; is originated from rare word w, in NoteDict, we first query w in NoteDict and then weight-averages its value NoteDict with the token embedding token emb; and positional embedding pos_emb,- In such a way, the historical contextual information of rare word Wi in NoteDDit(w (w;), can be processed together with other words in the current sentence in the stacked Transformer layers, which can help the model to better understand the input sequence. Figure gives general illustration of TNF in pre-training.
gives
Fine-tuning. Our goal is to make the training of the model (e.g., the parameters in the Transformer encoder) more efficient. To achieve this, leverage cross-sentence signals of rare words notes to enrich the input signals. To verify whether the Transformer encoder is better trained with TNF, we purposely remove the NoteDict for fine-tuning and only use the trained encoder in the downstream tasks. First, in such setting, our method can be fairly compared with previous works and backbone models, as the fine-tuning processes of all the methods are exactly the same. Second, by doing so, our model occupies no additional space in deployment, which is an advantage compared with existing eeemory-augmente neural networks (Santoro et al.) 2016 Guu et al., 2020). We also conduct an ablation study on whether to use NoteDict during fine-tuning. Details can be found in Section
4 EXPERIMENTS
To verify the efficiency and effectiveness TNF, we conduct experiments and evaluate pre-trained models fine-tuning downstream tasks. All codes implemented based on fairseq (Ott et al. 2019) in PyTorch (Paszke et al. 2017). All models are run on 16 NVIDIA Tesla V100 GPUs with (Micikevicius et al. 2017
4.1 EXPERIMENTAL SETUP
To show the wide adaptability of TNF, use BERT (Devlin al. 2018) and ELECTRA (Clark
ct al. 2019) as the backbone language pre-training methods and implement TNF on top of them. We fine-tune the pre-trained models on GLUE (General Language Understanding Evaluation) (Wang et al. 2018) to evaluate the performance of the pre-trained models. We follow previous work to use eight tasks in GLUE, including CoLA, RTE, MRPC, STS, SST, QNLI, QQP, and MNLI. detailed setting of the fine-tuning is illustrated in Appendix A.1
Data Corpus and Pre-training Tasks, Following BERT (Devlin et al., 2018), we use the English Wikipedia corpus and BookCorpus (Zhu et al. 2013) for pre-training. By concatenating these two datasets, we obtain a corpus with roughly 16GB in size, similar to Devlin et al. (2018). 8). We also follow couple of consecutive pre- pe-prccessingg steps: segmenting documents into sentences by Spacy? normalizing, lower-casing, tokenizing the texts by Moses decoder (Koehn et al. 2007). and finally, applying byte pair encoding (BPE) (Sennrich al. 2015) with the vocabulary size set 32,678. We use masked language modeling as the objective of BERT pre-training and replaced token detection for ELECTRA pre-training. We remove the next sentence prediction task and use FULL-SENTENCES
mode to pack sentences as suggested in RoBERTa (Liu et al. 2019). Details of the two pre-training tasks and TNF's detailed implementation on ELECTRA can found in Appendix A.2
htttss://spac//

---

Published as a conference paper at ICLR 2021
3.0
20
ELECTRA-train
BERT-TNF-train RT-vvlld RT TNF
19
TNF-RA-
86
2.8
ELECTRA-VaIE
ELECTRA TNF-vall
85
2.6
17
2.4
16
82
BERT
2.2
15
ELECTRA
BERT-TNF
ELECTRA-TNF
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Iterations
1e6
Iterations
1e6
Iterations
1e6
(a) Loss curves (BERT setting)
(b) Loss curves (ELECTRA setting)
(c) GLUE evaluation
Figure 3: The curves of pre-training loss, pre-training validation loss and average GLUE score for all models trained under the BERT setting and ELECTRA setting. All three sub-figures show that TNF expedites the backbone methods.
Model architecture and prper-parameter. We conduct experiments on BERT (110M param- eters) (Devlin et al., 2018) and ELECTRA (1 110M parameters) (Clark et al. 2019) (i.e., the base setting). A 12-layer Transformer is used for BERT. For each layer, the hidden size is set to 768 and the number of attention head (H) is to 12. ELECTRA composes of a discriminator and generator. The discriminator the same as BERT and the generator is 1/3 1/3-width BERT model, suggested by the 2019). We also conduct experiments on large models (335M parameters), details are at Appendix A.S We use the same pre-training hyper-parameters for all experiments. All models pre-trained for (00k steps with batch size 256 and maximum sequence length 512. All
original paper (Clark et al.
parameter configurations are reported in Appendix A.3
4.2 RESULTS AND ANALYSIS
TNF improves pre-training efficiency. Fig- ure B shows for all pre-training methods. how the pre-training loss, pre-training validation loss and average GLUE score change as pre-training pro- ceeds. From Figure 3a) and (b), we can see that as the training proceeds, TNF's pre-training loss and validation loss is constantly lower than its corre- sponding backbone methods. It indicates that TNF has accelerated its backbone model through the tire pre-training process. We can also notice from Figure 3 a) and (b) that the gap between the losses of the backbone model and TNF keeps increasing during pre-training. A possible explanation of this phenomenon is that the qualities of notes would improve with pre-training. Therefore, the notes that TNF takes for rare words could contain better semantic information to help the encoder as the training goes.
Params
Avg. GLUE
GPT-2
117 M
78.8
BERT
110 M
82.2
SpanBERT
110 M
83.9
ELECTRA
110 M
85.1
BERT (Ours)
110 M
83.1
BERT-NNF
110 M
83.9
ELECTRA (Ours)
110 M
86.0
ELECTRA-TNP 110 M
86.7
Table 1: Average GLUE score of all methods the dev set when the pre-training finished, i.c., at 1e6 iterations. Results of GPT, BERT and ELECTRA are from Clark et al. 2019). The re- sult of SpanBERT is obtained by fine-tuning the released checkpoint Joshi et al. (2019). We also reproduce BERT and ELECTRA in our sys- tem for fair comparison. We report their results as BERT( and ELECTRA(OUrs).
From Figure c).) we can see that the average GLUE score of TNF is also larger than the baseline through most the pre-training. TNF's GLUE scores at 400k iteration are competitive to those of the corresponding backbone models at 1000k iteration in both BERT and ELECTRA settings. It means that to reach the same performance, TNF can save 60% of pre-training time. If models are trained on 16 NVIDIA Tesla V100 GPUs, BERT-TNF reach BERT's final performance within 2 days while it takes BERT 5.7 days.
7

---

Published as a conference paper at ICLR 2021
MNLI
QNLI
QQP
SST
CoLA
MRPC
RTE
STS
Avg.
BERT (Ours)
85.0
91.5
91.2
93.3
58.3
88.3
69.0
88.5
83.1
BERT-TNF
85.0
91.0
91.2
93.2
59.5
89.3
73.2
88.5
83.9
BERT-TNF-F
85.1
90.8
91.1
93.3
59.8
88.8
72.1
88.5
83.7
BERT-TNF-U
85.0
90.9
91.1
93.4
60.2
88.7
71.4
88.4
83.6
ELECTRA(OurS)
86.8
92.7
91.7
93.2
66.2
90.2
76.4
90.5
86.0
ELECTRA-TNF 87.0
92.7
91.8
93.6
67.0
90.1
81.2
90.1
86.7
ELECTRA-TNE 86.9
92.6
91.8
93.7
65.9
89.7
81.4
89.8
86.5
DECCTRA-TNFUL 86.9
92.7
91.7
93.6
66.3
89.8
81.0
89.8
86.5
Table 2: Performance of different models on downstream tasks. Results show that TNF outperforms backbone methods on the majority of individual tasks. We also list the performance of two variants of TNF Both of them leverage the node dictionary during fine-tuning, Specifically, TNF-F uses fixed note dictionary and TNF-U updates the note dictionary as pre-training. Both models outperforms the baseline model while perform slightly worse than TNF.
Beyond the base-sized models (110 M parameters), we also apply TNF on large models to check the effectiveness of our method. Details are reported at Appendix A.5
TNF improves its backbone model's performance. BERT models are severely under-trained (Liu et al. 2019). Therefore, training faster usually indicates better final performance given the same amount of pre-training time. In Table [ we present the average GLUE score of all methods when the pre-training finished, i.e., at 1. updates. We can see from the table that in both BERT and ELECTRA settings, TNF outperforms its backbone methods on the average GLUE score with a large margin. Among them, ELECTRA-TNEF performance outperforms all state-of- methods with a similar model size. In Table we present the performance of TNF and its backbone methods on GLUE sub-tasks. TNF outperforms its backbone models on the majority of sub-tasks TNF's performance improvement against the baseline is most prominent on sub-tasks with smaller datasets. Among all 8 sub-tasks, RTE has the smallest training set which contains 2.5k training samples in total (Wang et al. 2018). On RTE, TNF obtains the biggest performance improvement (4.2 and 4.8 points for BERT and ELECTRA, respectively) compared with the baseline. On another small-data sub-tasks CoLA, TNF also outperforms the baseline with considerable margins (1.2 and 0.8 points for BERT and ELECTRA respectively). This indicates that TNF pre-training can indeed provide better initialization point for fine-tuning, especially on downstream tasks with smaller data sizes.
bascline
Empirical analysis on whether to keep notes during fine-tuning. As mentioned in Section when fine-tuning the pre-trained models on downstream tasks, TNF doesn't 'tt use the note dictionary. One may wonder what the downstream task performance would be like if we keep the note dictionary in fine-tuning. To check this, we test two TNF's variations for comparison. The first variation is denoted as TNF-F, in which we fix the noted dictionary and use it in the forward pass during fine-tuning described in Equation 6. The second variation is denoted TNF-U. In TNF-U, we not only use the note dictionary, but also add the note dictionary into the computation graph and update the note representations by propapropaaation.. The results are listed in Table 2 that both TNF-F and TNF-U outperform the backbone model. This indicates that no matter if we keep the notes at fine-tuning or not, TNF can boost its backbone pre-training method's performance. Moreover, we also observe that their performances are both slightly worse than TNF. We hypothesize the reason behind can be the distribution discrepancy of the pre-training and fine-tuning data. More detailed analysis can be found in Appendix A.4
The results show
To see how pre-training with notes affects the model performance, we further study the validation loss at the pre- pre-training stage in different settings. We firstly study the validation MLM loss on sentences without rare words on both BERT and BERT-TNE, We find that at iteration 200k, BERT's MLM loss on sentences without rare words is 3.896. While BERT-TNF's MLM loss on sentences without rare words is 3.869, less than that of BERT. This indicates that with TNF, the model is in general better trained to preserve semantics related to common context. Then calculate the validation loss on sentences with rare words for three model settings, a praited TNF model with/without using the notes and standard pre-trained BERT. We find that the loss order is TNF with notes < BERT <
8

---

Published as a conference paper at ICLR 2021
TNF without notes. This indicates that information related to words contained in the TNF notes but not memorized in the Transformer parameters.
Furthermore, conduct sensitivity analysis of the newly-added hyper- eyper-parameters of TNF. Details and complete results can found at Appendix A.4
CONCLUSION
In this paper, we focus improving the data utilization for more efficient language pre-training through the lens of the word frequency. We argue the large proportion of rare words and their poorly-updated word embeddings could slow down the entire pre-training process. Towards this end, we propose Taking Notes on the Fly (TNF). TNF alleviates the heavy-tail word distribution problem by taking temporary notes for rare words during pre-training. In TNF, maintain note dictionary to save historical contextual information for rare words when we meet them in training sentences. In this way, when rare words appear again, we can leverage the cross-sentence signals saved in their notes to enhance semantics to help pre-training. TNF saves 60% of training time for its backbone methods when reaching the same performance. If trained with the same number of updates, TNF outperforms backbone pre-training methods by large margin in downstream tasks.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 167.006070064500 2016.
Dzmitry Bahdanau, Tom Bosc, Stanislaw Jastrzebski, Edward Grefenstette, Pascal Vincent, and Yoshua Bengio. Learning to compute word embeddings on the fly. arXiv preprint (eXxiv/706.00200 2017.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. International Conference on Learning Representations, 2019.
Jacob Devlin, Ming-
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Ahmed El-Kishky, Frank Xu, Aston Zhang, and Jiawei Han. Parsimonious morpheme segmentation with an application to enriching word embeddings. In 2019 IEEE International Conference on Big Data (Big Data), pp. (3. IEEE, 2019.
Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. En- tities as experts: Sparse memory access with entity supervision. arXiv preprint wrXiv:200..07202. 2020.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. Representation degeneration 2019.
problem in training natural language generation models. arXiv preprint
Chengyue Gong, Di Hc, Xu Tan, Tao Qin, Liwei Wang. and Tie- Re-Yan Liu. Frage: Frequency-agnosticc word representation. In Advances in neural information processing systems, pp. 1334-1345, 2018.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International Conference on Machine Leurning, pp. 2337-2346, 2019.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint arXiv:2002.088 200009900,,8909,, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint 1052, 2019.

---

Published as a conference paper at ICLR 2021
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint 2019.
Yerbolat Khassanov, Zhiping Zeng, Van Tung Pham, Haihua Xu, and Eng Siong Chng. Enriching rare word representations in neural language models by embedding matrix augmentation. arXiv preprint arXiv: 03799, 2019.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models. In Thirtieth AAAI conference on artificial intelligence, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. URL
wwwpwxwwo//b.//.1///222 6980
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In ACL, 2007.
Ray R Larson. Introduction information retrieval. Journal the American Society for Information Science and Technology, 61(4):852 1535222--,--1 2010.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1xXxv:9977.1192, 2019.
Minh-Thang Luong, Richard Socher, and Christopher D Manning. Better word representations with recursive neural networks for morphology. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, Pp. 104-113, 2013.
Tamas Makany, Jonathan Kemp, and Itiel E Dror. Optimising the use of note-taking as an external cognitive aid for increasing learning. British Journal of Educational Technology, 40(4))619-63) 2009.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training, arXiv preprint arXiv: 2017.
Myle Ott, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. Analyzing uncertainty in neural machine translation. arXiv preprint 1803 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Colin Raffll Noam Shazeer, Adam Roberts, Katherine I Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint 19X.v:1910.10683, 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmentee neural networks. In International conference on machine learning, pp. 1852-1850. 2016.
Cicero D Santos and Bianca Zadrozny. Learning Character-leve representations for part-of-speech tagging. In Proceedings of the 31st international conference on machine learning (ICML-14), pp. 1818-1826, 2014.
Timo Schick and Hinrich Schütze. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In AAAI, pp. 8766-8774, 2020.
10

---

Published as a conference paper at ICLR 2021
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. CoRR, abs/1508.07909 2015. URL httt:://aaxt..cx///dbabaaa 1508
07909
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprini
2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1 0oss/804.07461,, 2018. |tLp::: org. abs / 1 804. 07461
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint
2016.
Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv preprint arXiv:1506.0677: 2015.

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 GLUE FINE-TUNING.
We tine-tunet the pre-trained models on GLUE (General Language Understanding Evaluation) (Wang et al. 2018) to evaluate the performance of the pre-trained models. We follow previous work to use eight tasks in GLUE, including CoLA, RTE, MRPC, STS, SST, QNLI, QQP, and MNLI. For evaluation metrics, we report Matthews correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks. We use the same optimizer (Adam) with the same hyper-parameters as in pre-training. Following previous work, we search the learning rates during the fine-tuning for each downstream task. The details are listed in Table 3 For fair comparison, we do not apply any published tricks for fine-tuning. Each configuration is run five times with different random seeds, and the average of these five results on the validation set is calculated as the final performance of one configuration. We report the best number over all configurations for each task.
A.2 PRE-TRAINING TASKS.
We use masked language modeling as the objective of BERT-based pre-training and replaced token detection for CECCCRA-basee pre-training. We remove the next sentence prediction task and use FULL-SENTENCES mode to pack sentences as suggested in RoBERTa (Liu et al. 2019).
Masked Language Modeling of BERT. The masked probability is set 0.15 with whole word masking. As mentioned above, our datasets are processed as sub-word tokens after BPE. Whole word masking here means when sub-word is masked, we also mask all surrounding tokens that originate from the same word. After masking, we replace 80% of the masked positions with [MASK). 10% by randomly sampled words, and keep the remaining 10% unchanged.
Replaced Token Detection of ELECTRA. We use the output of the generator of ELECTRA to calculate note representations and update the note dictionary. Then we only apply the notes on the input of the discriminator (i.e., adding the note representations of rare words together with the token embeddings as the input of the discriminator), not on the input of the generator. The reason is that as shown in BERT-TNF's experiments, the notes can enhance the training of the generator. However, an overly strong generator may pose an unnecessarily challenging task for the discriminator (Clark) cct al.) 2219)) leading to unsatisfactory pre-training of the discriminator. The masked probability of the generator is set to 0.15 with whole word masking and all masked positions replaced with [MASK].
A.3 HYPER-PARAMETTE
We conduct experiments on BERT- Base (1 10M parameters), BERT-Large (335M parameters) (Devlin al. 2018) and ELECTRA (Clark et al. 2019). BERT consists of 12 and 24 Transformer layers for the base and large model, respectively. For each layer, the hidden size is set to 768 and 1024 and the number of attention head (H) is set to 12 and 16 for the base and large model. The architecture of the discriminator of ELECTRA is the same as BERT-Base. The size of the generator is 1/3 of the discriminator. We use the same pre-training paraeeparameters for all experiments. All models are pre-trained for 1000k steps with batch size 256 and maximum sequence length 512. We use Adam (Kingma & Ba. 2014) as the optimizer, and set its hyperparameter e to 1e-6 and (81, to (0.9, 0.98). The peak learning rate is set to le-4 with 10k-step warm-up stage. After the warm-up stage, the learning rate decays linearly to zero. We set the dropout probability to 0.1 and weight decay to 0.01. There are three additional hyper-parameters for TNF, half window size k, discount factor À and weight y. We set as 16, À as 0.5, y as 0.1 for the main experiment, except for ELECTRA is empirically set 32. All hyper-parameter configurations are reported in Table 3
A.4 ABLATION STUDY AND PARAMETER SENSITIVITY
Empirical analysis whether to keep notes during fine-tuning. We test two TNF's variations for comparison. The first variation denoted as TNF-F, in which we fix the note dictionary and use it in the forward pass during fine-tuning described in Equation 6. The second variation is denoted
12

---

Published as a conference paper at ICLR 2021
Pre-training
Fine-tuning
Max Steps
1M
Max Epochs
5 or 10
Learning Rate
le-4
{le-5, 2e-5, 3e-5, 4e-5, 5c-53
Batch Size
256
32
Warm-up Ratio
0.01
0.06
Sequence Length
512
512
Learning Rate Decay
Linear
Linear
Adam
le-6
1e-6
Adam (81, (22)
(0.9, 0.98)
(0.9, 0.98)
Dropout
0.1
0.1
Weight Decay
0.01
0.01
A of BERT-TNF
16
À of BERT-TNF
0.5
Y of BERT-TNF
0.1
k of ELECTRA-
32
À of ELECTRA-TNE
0.5
7 of ELECTRA-TNF
0.1
Table Hyper-parameters for the pre-training and fine-tuning on language pre-training methods, include both backbone methods and TNFs.
Effect of varying k
Effect of varying X
Run #
R2
R3
R4
R5
R6
R7
R8
k
4
8
16
32
16
16
16
16
16
A
0.5
0.5
0.5
0.5
0.1
0.3
0.5
0.7
0.9
7
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
Model Size
base
base
base
base
base
base
base
base
base
Avg. GLUE
82.5
83.3
83.9
83.5
83.3
83.9
83.9
82.8
83.8
Effect of model size
Effect of varying y
Run #
RTO
RIT
R12
R13
R14
RI5
R16
R17
R18
k
16
16
16
16
16
16
16
X
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.1
0.1
0.1
0.3
0.5
0.7
0.9
Model Size
base
base
large
large
base
base
base
base
base
Avg. GLUE
83.1
83.9
84.4
85.6
83.9
83.1
82.9
83.5
83.0
Table 4: Experimental results on the sensitivity of BERT-TNF's hyper-parameter k, À and
as TNF-U. In TNF-U, we not only use the note dictionary, but also add the note dictionary into the computation graph and update the note representations by pack-propagation. As shown in Table 2. both TNF-F and TNF-U outperform the backbone models. This indicates that no matter if we keep the notes fine-tuning or not, TNF can boost its backbone pre-training method's performance. Moreover, we also observe that their performances are both slightly worse than TNF. We hypothesize the reason behind can lie the discrepancy of the pre-training and fine-tuning data. We notice that the proportion of rare words in downstream tasks are too small (from 0.47% to 2.31%). When the data distribution the pre-training data set very different from the downstream data sets, notes of rare words in pre-training might be not very effective in fine-tuning.
Sensitivity of hyper-parameters. We also conduct experiments on the BERT model to check if TNF's performance sensitive to the newly introduced hyper-parameters. Results are shown in Table 4 Overall, in most settings (R1-R9 and R14-R18) of varying k; À and 7, TNF outperforms the BERT-Base (R10), which indicates that TNF generally robust to the new dyper-parameter The experimental results using different & (R1-R4) show that larger & usually leads better performances. The reason may be that the note representation of rare words can contain more sufficient contextual information when relatively large k is applied. We also tried fixing k and
13

---

Published as a conference paper at ICLR 2021
tuning (R5-R9) and y (R14-R18). We empirically find that with À 0.5 and y 0.1, BERT-TNF produces the best performance. We speculate that small À and y can make the training more stable.
A.5 LARGE MODELS
In addition the experiments on base models, we also
86
train large models to check the effectiveness of TNF. A 24- layer Transformer is used for BERT-large. The hidden size
85
is set to 1024 and the number of attention head is set to 16. Other settings are same as base models. Although can be seen from the experiments of the previous works (Clark cct al. 2019) that improving a larger model's performance downstream tasks is usually more challenging. TNF can still save at least 40% training time on BERT-Large shown in Figure In Table 4 (R 10-R13), TNF's performance on BERT-Base and BERT-Large- TNF gives larger improvement the BERT-large (1.2 point) than BERT-base (0.7 point) when the pre-training is fin- ished. It indicates that TNF is only robust the model
84
83
82
81
BERT-
BENT-Large-TT TNF
compare
0.2
0.4
0.6
0.8
1.0
Iterations
1e6
Figure 4: GLUE score of large models
size, but also more effective at improving the final performance when the model gets bigger.
14


Reviews:
[{"review_id": "qaon3QH0jIP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an external memory architecture. When encountering the rare words (with a frequency between 100-500), the method will store the average contextualized word embedding of nearby words into a dictionary. Next time it encounters the same rare word, it will retrieve the average embedding and input it into BERT encoder. The experiment results show that given the same number of training steps, adding the external memory improves the MLM loss and significantly improves the results on RTE (Recognizing Textual Entailment) dataset, which leads to a slightly better GLUE score. The experiment also shows that keeping the external memory during the fine-tuning stage slightly degrades the performance.\n\nPros:\n1. The method is simple and easy to understand\n2. The experimental results on GLUE are quite surprising. It shows that we should take note when training BERT but throw away the note dictionary when fine-tuning the model.\n\nCons:\n1. Missing an important citation [1]\n2. The paper does not well explain the surprising results on GLUE. This is a crucial weakness. The comparison of the MLM loss is not very fair because the proposed method has a large external memory. The benefit of the proposed method relies on the improvement of the average GLUE score. However, Table 2 shows the most of the improvement of GLUE actually comes from the improvement of a single dataset, RTE. Without understanding why it improves RTE, the readers do not know when they want to adopt the proposed method for their downstream applications.\n\nClarity:\nThe text is fluent, but the main story is not well supported by the experiment results. The story is that using an external dictionary could accelerate the training, but the main experiment finding actually says that using an external dictionary can very significantly improve the results on RTE dataset while performing similarly on other datasets in GLUE.\n\nOriginality:\nThere has been some effort of using an external dictionary to help the training of BERT [1], but I am not aware of existing papers that apply the dictionary to only the rare words. I also do not know any other work that shows the external dictionary could improve the GLUE scores.\n\nSignificance of this work:\nIf the authors could well explain the experimental results on GLUE and justify the explanation using some analysis, this might lead to more important findings.\n\n\nFigure 3c seems to contradict with Table 1 and 2 because in Table 1 and 2, the GLUE score of BERT (ours) is 83.1 but all the points in the BERT curve in Figure 3c is below 83. \n\nUsually, when a study tries to sell its method as a way to accelerate the training, it means the method reaches some performance faster but the method will converge the same performance eventually. However, Figure 3 does not show that they will converge the same value, so selling the method as a way to accelerate the training is weird. Furthermore, I think the lower MLM loss is due to the extra parameters in the note dictionary rather than the note dictionary accelerates the training. \n\nIt is not surprising that taking notes for rare words could achieve lower loss/perplexity because the note dictionary gives the extra memory capacity [1]. It is also not surprising that it can achieve better performance on GLEU if using the note dictionary during the fine-tuning stage due to the extra parameters. The really interesting results are that the authors report that the model could very significantly improve the RTE task and mildly improve CoLA without using the note during the fine-tuning stage. \n\nIntuitively, the proposed model stores lots of knowledge about the rare words into the note dictionary. Does the fact that the note is not needed in the fine-tuning stage imply that the knowledge about rare words is actually not needed? Does it mean the RTE or CoLA do not contain many rare words or does it mean the rare words do not affect the decision of BERT and ELECTRA in RTE or CoLA? Is the reason of improvement that we could store more interactions between popular words in the parameters of BERT itself because the information of rare words has been stored in the note (maybe you can test this by reporting the MLM loss on the sentences without any rare words)? If that is the case, why do we only stably improve RTE and CoLA? If the authors can show the above hypothesis is true, I think this is a significant contribution because that means this paper provides a way to control what LM should learn when there is a mismatch between MLM training corpus and downstream applications (e.g., MLM training corpus contains many rare words but we should ignore the rare words in the downstream applications).\n\nThis paper lacks a good explanation of the above weird result (in my opinion, the most valuable finding in this paper) and lacks the analysis that supports the explanation. The main paper says that taking notes improves the tasks with the small datasets the most. The STS-b (7k) and MRPC (3.7k) have smaller training datasets than CoLA (8.5k). Why are the results of STS-b and MRPC cannot be stably improved? If the authors really want to explain the performance improvement using the training dataset size, the authors can just randomly sample several small subsets of training data from each dataset and show that the GLUE score improves a lot in that setting. In the appendix A.4, the authors hypothesize that the small proportion of rare words in each dataset of GLUE (from 0.47% to 2.31%) might be the reason that we can ignore the note dictionary during the fine-tuning stage. This also did not explain why most of the improvement of the GLUE score comes from RTE. Moreover, if the rare words are not important in the testing datasets, why do we want to take notes in the first place?\n\nI will vote for acceptance if the authors could answer these critical questions I raise above strongly.\n\n\nMinor:\n1. Although the chance is not high, I think it is possible that parts of MLM improvement could be achieved by simply sampling the sentences containing the rare words more (This is a minor concern. If you do not have time to finish the experiments for this baseline, you can choose not to do it or compare the results after training fewer steps).\n2. I guess the dictionary overhead is small but it should be measured and reported because you say the method accelerates the training. \n\n\n[1] Lample, Guillaume, et al. \"Large memory layers with product keys.\" Advances in Neural Information Processing Systems. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "hwxn__QtOXB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Taking Notes on the Fly, a technique to improve the training efficiency of language-modeling style pretraining. It works by identifying rare words in the pre-training and adding a “note-taking” component to the masked language model which augments these words with an extra “note” embedding at the input layer. The note embedding is constructed from an exponential moving average of mean-pooled contextualized representations of context windows in which that word was previously seen during training. The notes are dropped in fine-tuning. Experiments find that this pre-training method improves fine-tuning results on English NLP tasks in the GLUE benchmark when used in the original BERT pre-training setup. In particular, the model can achieve similar performance to the original BERT model with less than 40% of the training steps, and similarly for ELECTRA.\n\n### Strengths\n\nThis paper is clearly written, the proposed technique is simple, and the results seem strong. It is laudable that the authors give experiments in the appendix to give a sense of hyperparameter sensitivity. The paper has a strong backbone and it seems that the proposed technique or something similar may serve as the basis for solid future work.\n\n### Weaknesses\n\nWhile the backbone of the paper is strong, I think it could be improved in its head (motivation) and legs (experimental studies).\n\nFirst, motivation. While the framing around rare words with the COVID-19 example is interesting, I think it has gaps. The introduction argues that since “COVID-19” is a rare word, in the course of training the model may lack the necessary signal to predict the masked word “lives.” But isn’t this fact exactly what should lead the model to improve its embedding of “COVID-19”? Because gradients flow into the embeddings both through the softmax layer and the input layer.\n\nSo while adding to the context may help the model get a foothold with more effective training signal for the masked token, it seems to me that the note could also “explain away” the rare word’s embedding in the input layer, reducing the learning signal on it. If that’s the case, then to the extent that TNF works, it would be by the tradeoff between improving the learning signal at the output layer for all words (and in contextualization) and degrading it at the input layer (for rare words).\n\nAs a broader example, see https://openreview.net/pdf?id=3Aoft6NWFej. That paper argues for a masking scheme which eliminates easy shortcuts from the prediction problem to increase learning efficiency, whereas this paper argues essentially the opposite—that shortcuts must be added to hard cases in order to facilitate learning. It seems that there may be a line to walk here between a task being too hard to learn from and too easy to be useful. Because it’s not clear where that line is, I think it’s not enough to motivate TNF from only one direction. It would be better to also have an explanation of why the note-taking approach does not also make things “too easy.” It’s not obvious to me how to best make this argument, though results from some of the ablations I will suggest below might help.\n\nThis brings me to my second point: Ablation experiments. If the motivation is to improve the representations of rare words in the input, then there are even simpler ways to do this. Experiments with simple baselines and ablations are important for figuring out why exactly TNF works.\n\nFirst, if the note is such a useful addition to the word embedding, why not just use it to update the embeddings directly? At that rate, the method for constructing the note embeddings looks quite similar to word embedding training objectives like word2vec and GloVe. This suggests a critical ablation:\n\n* Initialize the word embeddings with word2vec, GloVe, or similar run over the wordpieces in the pretraining corpus. (Weirdly, I can’t find an example of this in the literature. It seems like an obvious thing to try. I may have just missed it.) Indeed, it seems to me that the framing in the paper could just as easily motivate this (much simpler) technique than TNF.\n\nIf TNF outperforms the critical ablation, that implies that its gains are coming from some of the other particulars of the technique, such as 1) the extra degree of freedom provided by decoupling the note embeddings from the wordpiece embeddings, or 2) the use of contextualized vectors for note embeddings (rather than the non-contextualized ones in the word embedding objectives). \n\nTo investigate these issues, I would suggest three more ancillary ablations on TNF:\n* Directly update the rare word’s embedding with a version of Eq. 5 rather than keeping a separate note dictionary.\n* Update the note embeddings via backprop instead of Eq. 5. This would amount to “partially tying” the input and output embeddings, giving more freedom to the input layer, which is partly what’s happening in TNF.\n* Pool over non-contextualized instead of contextualized representations in Eq. 4.\n\nFinally, to address the “too easy” vs “too hard” distinction, two more ablations that might help would be:\n* Instead of using an exponential moving average for the note embedding update, just use the pooled context vectors from the last instance of the rare word (i.e., set $\\gamma$ to 1 in Eq. 5).\n* instead of using an explicit note dictionary, augment the input context with retrieved text containing the rare word. See TEK-enriched representations (https://arxiv.org/pdf/2004.12006.pdf) for an example of this. For consistency, the exact last-seen context of the rare word could be used.\n\nThe first will help identify to what extent aggregating over many multiple inputs to get a high quality representation is necessary for TNF. This could then serve as a reference point for the second ablation, which may help determine whether the fixed embedding size and pooling operation helps by creating a bottleneck for the retrieved information and preventing things from getting “too easy.” (although context window sizes might also be a confound here, that could also be controlled carefully.)\n\nAll together I think these ablations would shed a lot of light on why TNF works, and make this work much more useful to researchers who wish to build on it in the future. However, I know I’ve suggested a lot of crazy experiments here. I would not expect all of this necessarily to be done and I leave it up to the discretion of the researchers which are most important. I am also sure the authors could come up with better ablations than these as well. But my sticking point is the first ablation — initializing with non-contextualized embeddings — which I think is critical. And I think it behooves the authors to address some of the lingering questions (including more written below), even if not all of them.\n\n### Recommendation\n\nUnfortunately, reject. The technique is simple and the results seem good, but the paper does not provide empirically-justified insight on why TNF works. I think ablations and investigation into the “why” aspect is the most important part of this kind of model engineering research.\n\n### More comments & questions\n\nI am left with some more questions about how TNF works:\n\n* How does the quality of the representations of rare words specifically compare in your approach? Does it improve the representations of common words and contextualization at the expense of rare words? While it may be tricky to try to directly assess embedding or contextualization quality, breaking down the MLM perplexities by word frequency (or presence of rare words in the context) after removing the note dictionary might be informative. I admit this might also be tricky because I imagine the model would have to be fine-tuned without the notes for a bit before doing such an experiment. But any insight into this issue would be appreciated.\n* If this method indeed works by more narrowly refocusing the training signal on the masked token than the context tokens, then would you be able to further increase the learning efficiency by oversampling rare words when determining the masks in training? I am not aware of anyone showing such a thing to work, though I might have missed it. Just a thought.\n\nWhile the pretraining corpus is huge, 100 occurrences still seems like a pretty high threshold for rare words given the justification provided in the paper. Questions:\n* What do the even rarer words look like? Are they just a source of noise? e.g., because they are components of names or don’t have clear and consistent semantic content?\n* What proportion of contexts contain words appearing less than 100 times? It seems that the 20% figure in the paper is meant to apply to your definition of rare words, which appear between 100 and 500 times.\n* What is the word vocabulary size? i.e., how many words appear more than 500 times, and less than 100?\n* Did you do any preliminary experiments with other thresholds? Would you expect this to work with more common words as well? Why or why not? (This may also relate to the “too easy” vs “too hard” issue.)\n\nOn pre-training efficiency results: I think Figs 3a and 3b need to be explicitly qualified a little better. AFAICT, having lower loss here doesn’t necessarily mean the model (modulo the note dictionary) is learning better, because it sees the notes in the input. So we’re looking at the loss in a different setting than we intend to fine-tune in. It’s still interesting to see, but I think it's best to include an explicit caveat.\n\nWhat about training the models for more steps? Will the trend hold and performance improve overall, or will the gains eventually level off as the representations of rare words get better? Especially for pretrained models, since they are used as the starting point for many models, it is often worthwhile to train them longer (as in the RoBERTa paper), so it’s important to understand the usefulness of this method in that regime.\n\n### Typos etc.:\n\n* P.3: neglectable -> negligible\n* P.3: Representation -> Representations (in BERT acronym)\n* P.6 Sec. 4.1: after “MNLI” there is a space missing after the period.\n* P.6: “FULL-SENTENCES” would look better & be consistent with Liu et al if it were in small caps.\n* Please cite the individual dataset creators for the datasets in the GLUE benchmark.\n\n---\n\nUpdate: upped score from 4 to 5; see comment thread.\n\nUpdate again: score further updated from 5 to 6 with GloVe context ablations and perplexity results on sentences with rare words.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "cdFOrFGz6wJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work aims at accelerating pre-training by leveraging the contextual embeddings for the rare words. It is argued that the inadequate training of rare words slows down the pre-training. The authors then proposed to keep a moving average of the contextual embeddings for the rare words and use it to augment the input embeddings of the rare words. This technique is applied to BERT and ELECTRA and is shown to improve over the baseline. \n\nStrength:\n\n1. This work proposes a simple approach to accelerate the pre-training, with only a small memory and compute cost during training. The empirical study on BERT and ELECTRA supports the claimed improvements. \n\n2. It provides an interesting view towards the rare words problem that the rare word not only has worse embeddings but also slows down training of the whole model. \n\nWeakness:\n\n1. It is argued that the proposed approach helps with rare words problem. But it will help to add more experiments to see how much more benefit we can get from it. For example, maybe the use of contextual embeddings are actually helpful for all the words or sub-words instead of just the rare words. \n\nSpecifically, regarding \" we define keys as those words with occurrences between 100 and 500 in the data corpus\", How are the range 100 to 500 chosen? Have you tried it on words appearing lower than 100 or higher than 500? As mentioned above, it would be interesting to see if this approach can be applied to more words or subwords to get even more gains. \n\n2. Some design choices needs more details or explanations. \n\nFor example, why does the NoteDictionary use \"words\" instead of \"sub-words\" as keys? It seems using \"sub-words\" could cover a broader range of sentences with a NoteDictionary of the same size. It will also be easier to use during pre-training, for example, you could use the contextual embeddings to improve the word embeddings of the sub-words directly to avoid having an extra NoteDictionary. \n\nAnother example is how the window size is chosen, since it seems an important new hyperparameter. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lud0JlJRD9h", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*: This paper proposes a method for improving pretraining convergence speed by augmenting the representations of rare words with the mean-pooled representations from their previously-occuring contexts (“notes”, stored in a “note dictionary”). The method considerably speeds up the convergence of pretraining BERT and ELECTRA, and the authors furthermore show that these models perform better when fine-tuning on downstream GLUE tasks (likely because the models were undertrained to begin with, so converging faster alleviates this issue).\n\n*Strengths*: The method is surprisingly simple and empirically quite effective. It's especially interesting to see that BERT + TNF at 400K steps has better GLUE performance than BERT at 1M steps.\n\n*Weaknesses*: the paper does not do a convincing job of arguing that the reasons for the faster convergence comes from better modeling of rare words---I’m still not entirely sure why this works so well. Do these rare words commonly show up in GLUE (and thus, the method is helping because your representations of rare words are better)? It seems like TNF is actually improving the representations of more-common words as well.\n\n*Recommendation*: 7 Despite the lack of clarity around why exactly this method works so well, the method seems empirically useful and straightforward to apply. I expect that this will be useful to practitioners interested in applying BERT and similar pretraining strategies to new corpora and domains.\n\n*Questions*:\n\nIt’s a bit unclear to me that note-taking itself is required for this to work well...in the COVID example presented in the introduction, if you see the sentence “The COVID-19 pandemic is an ongoing global crisis”, isn’t it possible that MLM itself is sufficient to associate the embedding of “COVID-19” with “pandemic” and “global crisis”? Do you have further evidence to show that note-taking is actually improving the representations of rare words, besides GLUE score (which might not be very indicative, since the rare words might not show up in GLUE).\n\nThe Construction of Note Dictionary: Does 3.47B refer to the number of types or the number of tokens? Why not define keys with frequencies less than 100 in the dictionary as well (since you only use types that show up between 100 to 500 times)?\n\n“It means that to reach the same performance, TNF can save 60% of pre-training time. If models are trained on 16 NVIDIA Tesla V100 GPUs, BERT-TNF can reach BERT’s final performance within 2 days while it takes BERT 5.7 days.”: Is the 2 days vs 5.7 days an actual wallclock measurement? Or, are you hypothesizing this based off of the loss curves?\n\n*Missing / Erroneous Citations:*\n\n“It is well-known that in a natural language data corpus, words follow a heavy-tail distribution (Larson, 2010)” This is more-commonly known in the NLP community as Zipf’s law. Better cites would be:\n  - Zipf G. The Psychobiology of Language. London: Routledge; 1936.\n  - Zipf G. Human Behavior and the Principle of Least Effort. New York: Addison-Wesley; 1949.\n\n*Miscellaneous comments:*\n\n“Moreover, completely removing those sentences with rare words is not an applicable choice either since it will significantly reduce the size of the training data and hurt the final model performance.”: I agree that it’s a bad idea to remove sentences with rare words, but I disagree that the issue is reducing the size of the data---you can always go collect more data and filter it to not include rare words. It’s more likely that the issue is that removing sentences with rare words would reduce the diversity of the pretraining data, which would be harmful\n\n“Our method to solve this problem is inspired by how humans manage information.”: I think the connection to human note-taking is tenuous at best, and would omit it; the motivation remains clear without this.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an approach for pre-training that involves "taking notes on the fly" for rare words. The paper stirred a lively discussion on the reasons for the reported results, which the authors followed-up with new experiments and findings that convinced the reviewers that indeed their approach is valid and interesting. Thus, I am recommending acceptance.
Decision: Accept (Poster)