Language Model Pre-training with Linguistically Motivated Curriculum Learning
Authors: Yile Wang
Yue Zhang
Peng Li
Yang Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: language model pre-training
curriculum learning
data-centric method

Abstract: Pre-training serves as a foundation of recent NLP models, where language modeling task is performed over large texts. It has been shown that data affects the quality of pre-training, and curriculum has been investigated regarding sequence length. We consider a linguistic perspective in the curriculum, where frequent words are learned first and rare words last. This is achieved by substituting syntactic constituents for rare words with their constituent labels. By such syntactic substitutions, a curriculum can be made by gradually introducing words with decreasing frequency levels. Without modifying model architectures or introducing external computational overhead, our data-centric method gives better performances over vanilla BERT on various downstream benchmarks.

Under review as a conference paper at ICLR 2023
LANGUAGE MODEL PRE-TRAINING WITH LINGUISTI- CALLY MOTIVATED CURRICULUM LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-training serves foundation of recent NLP models, where language mod- eling task is performed over large texts. It has been shown that data affects the quality of pre-training, and curriculum has been investigated regarding sequence length. We consider a linguistic perspective in the curriculum, where frequent words are learned first and rare words last. This is achieved by replacing hierar- chical phrases that contain infrequent words by their constituent labels. By such syntactic substitutions, a curriculum can be made by gradually introducing words with decreasing frequency levels. Without modifying model architectures or in- troducing external computational overhead, our data-centric method gives better performances over vanilla BERT on various downstream benchmarks.
INTRODUCTION
Pre-trained language models (PLM) have gained much attention and achieved strong results in var- ious NLP tasks (Devlin al.) 2019 Radford al. 2019, Brown al.) 2020). Based self- supervised learning objectives such as causal language modeling (Peters et al. 2018 Radford et al. 2019. masked language modeling (Devlin et al. 2019). and text-to-text generation (Lewis et al.,
2020 Raffel et al. 2020). PLM can learn task-agnostic transferable features from large-scale unla- beled corpora. It has also been shown that PLM can encode syntactic (Hewitt Manning 2019 Goldberg, 2019 Wu et al. 2020), semantic (Tenney et al. 2019 Jawahar et al.) 2019), and fac-
tual (Petroni al. 2019 Dai et al.
2022) knowledge.
For improving the representation power of PL., much research has been done on setting differ- ent training objectives (Zhang et al. 2019 Yang et al.
Liu et al. 2019b), modifying model architectures (Dong et al. |019]] Clark et al. 2020 He et al. 2021) and scaling up the parameter count (Shoeybi et al. 2019: Rac et al. 2022 Fedus et al. 2022 Chowdhery et al. 2022). However, relatively less work considers
2019
the way of using pre-training corpus, where most of the methods leverage the raw text as a whole (from millions to billions of tokens) and train for multiple epochs, given sufficient data, the training strategy may have reduced effect.
Recent work has shown the influence of a curriculum for pre-training. Li ct al. (022) propose sequence length warmup strategy for GPT-2 pre-training, which improve training stability and efficiency. Similarly, Nagatsuka et al.] (2021) split corpus into blocks with specified sizes for BERT pre-training. These methods focus on changing the sequence length instead of the content and emphasize the convergence speed. Beyond text length, there is a more salient discrepancy between the current PLM training and the language learning process of humans. In particular, we only learn limited but the most common and useful words at the beginning. then we grasp some basic syntactic concepts such as part-of-speech, set phrase, and clause, before recognizing large number of uncommon words via generalization or their specific usages.
Inspired by lingussic curriculum learning (Elman) 1990) 1993 Bengio al. 2009), we propose data-centric approach that progressively pre-trains a language model using curriculum that involves reconstructed data. An example contrast the masked language model pre-training and our multi-stage curriculum training is shown in Figure Our curriculum consists of m stages (m = 2 in Figure [b)), with each having a incrementally larger vocabulary. Specifically, we first constituent (and offt-of-speech) labels from Penn Treebank (Mareus et al. 1993) to replace the lower frequency words, and the model updates using the text composed of the most frequent words and the constituent labels. In this stage, all words are at the same frequency level, and thus

---

Under review as a conference paper at ICLR 2023
about [MASK] square
10000
Reid (born 18, [MASK])..
1981
(a)
With more than [MASK] members...
time)
million twelve 18th
...after being faculty member for [MASK] years.
MLM training
update
the [MASK] century was tumultuous time...
Over 400 [MASK] tokens have been burned.
Inilion
...about [MASK] square...
10000
about [MASK] square...
10000
...March 18. [MASK])...
CD
...Macch 18, [MASK])...
1981
...3 [MASK] members...
(Fins)
million
... [MASK] members... (Second stage)
million
(b)
...for [MASK] years.
MLM rrining tweive
...for [MASK] years.
MLM triring
tweive
updata
[MASK] century...
OCD
[MASK] century...
15th
.400 [MASK] tokens...
OCD
.400 [MASK] tokens...
Inilion
Figure 1: An example of (a) vanilla masked language modeling, and (b) our method using two-stage curriculum training. In the first stage, we replace the original lower frequency target word such as "trillion" by a constituent label CD, which stands for the cardinal number. The representations can be better updated in the latter training stage after acquiring the "concept" of CD.
trained equally thoroughly. Then we gradually introduce less frequent words, letting the model further improve based on previously acquired knowledge. During this stage, the previously learned constituent labels can serve as categorical knowledge to guide the learning of infrequent words.
Experimental results using BERT show that our method can improve pre-training, showing better performance across tasks including general language understanding, named entity recognition, ques- tion answering, part-of-speech tagging, and parsing. Through empirical analysis, we find that our curriculum training can mitigate the representation degeneration problem (Gao et al. 2019) in PLM, and the injected constituent labels can encode meaningful linguistic features that bridge the word representations across different frequencies. Code and model will be released for further research.
METHOD
We take BERT (Devlin al. 2019) as our baseline, which trained using masked language mod- eling, one of the most successful self-supervised learning objectives for pre-training ($2.1) Our method leverages linguistically motivated curriculum learning based on vanilla masked language modeling 2.2), with a dedicated data-centric method for stage-wise corpus reconstruction 2,.
2.1 MASKED LANGUAGE MODELING
Masked language modeling (MLM) aims to predict the original target word w; through modeling the contextualized representation of a randomly masked word W, in context:
Σ log Po Paw = Σ log e(p(E(w,) Th))
LMLM =
(1)
j= exp ((sp)E(w)TTA)
where W; is the masked symbol [MASK] in a context, h2 is the corresponding contextualized output.
2.2 MOTIVATION FOR PATA-CENTRIC CURRICULUM TRAINING
During vanilla MLM pre-training shown in Figure a)) the model needs to predict the corresponding word surface independently. Although the more common words such "10000", "million" and "twelve" could be updated frequently, lower frequency words such as "1981". "18th", and "trillion" may receive far less training signal. The discrepancy in word frequency may do harm to model training tasks such as text classification and machine translation (Gong et al. 2018). For language modeling, previous studies have shown that lower frequency words are learned poorly (Schick & Schütze 2020), which can also degenerate the training process for all other words (Yu et al. 2022).
We consider a psycho--inguistically motivated curriculum learning method by proposing two main rules to address the issues: 1) common words first, rare words next, and 2) models are learned with structural constraints, syntax. To build curriculum schedule that satisfies the above rules, inject constituent labels (Marcus et al.) 1993) into raw text, replacing different words in different training stages by their corresponding constituent structures.
2

---

Under review as a conference paper at ICLR 2023
Raw Yakitori is Japanese lype skewered chicken.
Parse Tree:
S
Word Frequency Table:
vP
High: ..., 1s, type,
Medium: chicken.
NNP
VBZ
Others: Yakitor
Low:
Japanese, skewered,
Yakitori
NP
PP
DT
JJ
NN
IN
NP
Stage 1: NP JJ lype NP.
Stage 2: NP is JJ type VBN chicken.
Japanese type
VBN
NN
Stage 3: NP is a Japanese type of skewered chicken.
Stage 4: Yakitori is a Japanese type skewered chicken.
skewered chicken.
Figure Illustration for reconstructing the sentence "Yakitori is Japanese type of skewered chicken." in our four stages of curriculum training. At each stage, we use the corresponding con- stituent labels (colored in blue) to replace the original words according to their overall frequency.
In Figure [b), in contrast to the baseline method, we mitigate the influence of infrequent words in the initial stages by predicting the [MASK] symbol as a constituent label "CD" (cardinal number) instead of the original word, based on the fact that these words share unified constituent structure across texts. Then we train our model using the original target after it acquires some "concept" of what CD is. Since the contextualized representation is built for predicting the unified virtual target CD in the first stage, the update of lower frequency word representations could be better guided through such previously learned category knowledge.
2.3 RECONSTRUCTING DATA WITH A CONSTITUENCY PARSER
Based on the above motivation, we use a mixup strategy that injects multiple constituent labels into the raw text and progressively incorporates more infrequent words. Our final curriculum includes four stages, where dif- ferent number of words in the word frequency table are used in each stage, according to Algorithm
Algorithm 1 Injecting constituent labels
for language model pre- training.
Input: Raw 8t, a constituency Parser, word frequency table TopList
Output: Reconstructed text sf
I: sf [], Tree = Parserr parssess)
2: function PROCESS(Tree)
Figure 2 shows an example of injecting constituent la- bels into sentence. In the first stage, only keep the most frequent words ("is", "a", "type", "of"") while re- placing the others by their corresponding constituent la- bels, these labels are directly used as normal words in the corpus, which can also be randomly masked and pre- dicted. In the second stage, we allow medium frequency words ("chicken") to appear together with the most fre- quent words and remaining labels. In the third stage, we add the low-frequency words ("Japanese", "skewered"" and the data is close to the original format, except for some labels that indicate rare words ("Yakitori").. the last stage, we use the original corpus for training. In practice, we set the word frequency ranking intervals of ~0.5K, 0.5K~3K, 3K~18K, and 18K~ as the high, medium, low frequency, and other rare words, respectively. The number of training stages and the frequency intervals are set roughly according to the word distribution and vocabulary size of the embedding table, we leave the optimization of these settings future work.
Tree has no SubTree then Tree.word in TopList then
4:
5:
sf Tree.word
else
7:
s Tree.tgg
9:
all Tree.leaves.won not in Tsp then
10:
sf Tree.tag
11:
else
12:
for each SubTree in Tree do PROcEss(SubTree)
13:
To simplify the implementation of our curriculum, we directly use the wordfreq library from (Speer) al., 2018) and ignore the statistics of sub-words (Sennrich al., 2016, Wu al.) 2016) after tokenization. Thus we can directly process the corpus without considering the distinction of word distribution for different domains, and avoid selecting among tokenizers.
3 EXPERIMENTS
3.1 PRE-TRAINING
We follow the setup of BERT-base-cased architecture from Devlin et al, (2019). The model is 12 layers Transformer encoder, with 768 hidden size and 12 attention heads. English WIKIPEDIA
3

---

Under review as a conference paper at ICLR 2023
and the BOOKCORPUS (Zhu et al. 2015) used as the pre-training data. We train our model with A (Loshchilov & Hutter 2019) optimizer for 1M steps with a learning rate le-4, batch size 256, warmup ratio 0,01, and with mixed precision using 8 X23GB V100 GPUs. Following the recipe from Liu et al. (019c) and Izsak et al. (2021), we do not use the next sentence prediction objective.
For offline data reconstruction,
the Benepar (Kitaev & Klein 2018) for parsing. In each of the first three stages, we train our model using the reconstructed corpus for 200K steps (i.e., a total of 600K steps). Then we use the raw corpus for the 600K~IM IM training steps. Since we add some constituent labels such as N2, VP, and JJ (see the full list in Appendix (A) in the text, we enlarge the embedding table by treating them as normal tokens, thus making our vocabulary size slightly larger (from 28, .96 to 29,051 The 55 externalyy added embeddings can be discarded after pre-traie trriiiinnng.
3.2 DOWNSTREAM TASKS AND DATASET
We evaluate on general tasks including natural language understanding, named entity recognition, and question answering. Since our method uses text mixed with syntax-related labels during pre- training, we also evaluate on syntax-related tasks such as part-of-speech tagging and parsing. Statis- ties of the datasets are shown in Appendix [
GLUE The GLUE benchmark (Wang et al., 2019a is used for evaluating general language un- derstanding, we compare on sub-tasks including MNLI (Williams et al.. 2018), QQP (Chen et al.] 2018), QNLI (Rajpurkar et al., 2016), SST-2 (Socher et al. 2013), CoLA (Warstadt et al. 2019).
STS-B [Cer et 2017). MRPC (Dolan & Brockeu 2005), and RTE (Bentivogli et al. 2009).
Named Entity Recognition. We use the CONI.L2003 datasets (Tjong Kim Sang & De Meulder 2003) for named entity recognition, the entity labels include PER, LOC, ORG, and MISC.
Question Answering. Two versions of the Stanford Question Answering Dataset (SQuAD) Ra- jpuurar et al. 2016 2018) are used. SQuAD 1.1 aims to predict the text span the passage. SQuAD 2.0 allows the possibility that answer exists in the paragraph.
Part-of-Speech Tagging. The Wall Street Journal (WSJ) portion of the Penn Treebank Marcus ct al. 1993) is used for POS tagging. We follow Manning (201T) by selecting sections 0-18 as the training set, 19-21 as the development set, and 22-24 as the test set.
Constituency Parsing. WSJ is also used for constituency parsing, where with sections 02-21 as the training set, 22 as the development set, and 23 as the test set.
the standard splits
3.3 FINE-TUNING
For constituency parsing, we use the self-attentive encoder (Kitaev & Klein 2018 and initialize it with different pre- are-trained models. For sentence-level classification (GLUE), token-level labeling (NER and POS tagging). and span-based question answering (SQuAD), we follow BERT for fine- tuning. Following Liu al. (2019c) and [an et al. (2020), fine-tune STS-B, MRPC, and RTE by starting from trained MNLI checkpoint. For other tasks, we train separately using single model and single task without data augmentation. Hyperparameter settings are shown in Appendix C
We compare fine-tuned results using different pre-trained models: 1) The BERT-base-cased check- point released by Google, denoted BERT: 2) Our model trained from scratch, denoted as BERT- reimp. The main difference between BERT-reimp and BERT is that we do not use the next sentence prediction objective; 3) Our model trained from scratch with curriculum learning, we denote it as BERT-CL. The only difference between BERT-CL and BERT-reimp is the training corpus. For our results, we report by averaging five runs with different seeds.
3.4 RESULTS
Table shows the results for the GLUE benchmark. We find that BERT-CL consistently outper- forms BERT-reimp across all tasks, showing that our curriculum pre-training is useful. Among all models, BERT-CL gives the best averaged results for both development and test set. We find that
'There is another option to avoid increasing vocabulary size by using the [unused]] ] to [unused99]
takens for replacing the added constituent labels.
4

---

Under review as a conference paper at ICLR 2023
MNLI (m/mm)
QQP
QNLI
SST-2
CoLA
STS-B
MRPC
RTE
Model
Avg.
Acc.
Acc.
Acc.
Mcc.
Spear.
Acc.
(Development Ser)
BERT
84.09/83.82
87.53
90.84
92.31
57.27
88.24
89,41
65.69
82.
BERT-reimp
83.58/83.62
87.12
90.06
92.31
57.43
88.22
89.55
63.28
81.68
BERT-CI
84.95/84.77
87.15
91.06
93.00
62.06
88.41
91.14
65.61
83.12
(Test Set via Leaderboard)
BERT
84.5/83.6
71.1
90.1
93.6
53.3
84.9
88.3
68.4
79.7
BERT-reimp
83.9/82.6
71.1
90.2
93.7
51.9
82.4
88.7
62.0
78.5
BERT-CL
85.4/84.4
71.2
90.6
93.9
59.8
83.9
89.5
66.5
80.5
Table I: Results on GLUE benchmark dev set and test The best results are in bold.
SQUAD 1.1
SQuAD 2.0
Model
P
F1
Model
FI/EM
FI/EM
BERT
90.91
92.24
91.57
BERT
88.96/81.61
75.69/72.51
BERT-reimp
90.94
91.93
91.43
BERT-reimp
89.38/82.71
78.19/75.20
BERT-CL 91.631
92.31
91.971
BERT-CL
89.87/83.07
80.02/77.06
Table 2: Results on CoNLL2003 test set. t: Statistically significant compared BERT- reimp with p < 0.01 by t-test.
Table 3: Results SQuAD dev set.
the CoLA task shows the largest improvement (+12.2% compared with BERT), which aims to judge the linguistic acceptability of sentences. This task benefits from our curriculum since we offer some syntactic labels during pre-training, nevertheless, our method can also improve the capability for general language understanding tasks such natural language inference and sentiment analysis.
Table Table[]shows 2ishows the results for NER. BERT-CL gives a 91.97 F1 score, better than both BERT (+0.40) and BERRT-rimp (+0.54). Note that the reported result the CoNLL2003 test from Devlin et al. (2019) is 92.4 F1 score, however, we did not achieve it with the current library, as discussed in Stanislawek et al. 2019) and al. (222)) The improvement over BERT and BERT-reimp may come from the fact that the named entity usually forms NN and NP structure in the constituency tree and such knowledge can be better aequired in our preliminary curriculum training stages.
Table 3 shows the results for question answering. Our model gives the best results on both datasets 440.49940.36 and +1.83/+1..88 1.86 over BERT-reimp). Compared with the sentence-level classification and token-level labeling tasks, question answering is more challenging since it requires understand- ing both query and passage with long-term dependency. We hypothesize that the improvement is due to that span-based answers usually form common constituent structures such as NP and CD, where these features are quite useful for answering the majority of questions like "what...", "where..." and "how many...", "when..."
Table shows the results for POS tagging. By using the full training set, our model gives better results than BERT and BERT-reimp. Since WSJ POS tagging less complicated task with rich training resources, we also 1) use fewer training data with 2% to 75% samples, and 2) fix the model parameters while only training a linear classifier upon the contextualized output, which is also called probing (Conneau et al. 2018 [iu et al. 2019a). We find that BERT-CL. still consistently gives better results under low-resource and probing settings.
Table shows the results for constituency parsing. Compared with BERT and BERT-reimp, our model gives an absolute improvement with +0.30 and +0.37 Fl scores, respectively. The advantage of using the complete match metric is more significant, where BERT-CL gives +2.47 and +1.30 ab- solute improvement, respectively. Note that although we leverage a constituency parser for building reconstructed data mixed with constituent labels, the improvement is non-trivial since we use the general purposed MI.M training instead of specifically augmenting training data for parsing.
To analyze the influence of corpus, we evaluate on GLUE and CoNLL NER test set by 1) using only the WIKIPEDIA, and 2) adding CC-NEWS (Hamborg et al., 2017) for pre-training. Results are shown in T We can see that BERT-CL still outperforms BERT-reime when changing the pre-training corpus, showing that the curriculum useful when applied different corpora.
5

---

Under review as a conference paper at ICLR 2023
Accuracy
by using 5% Training Set
Model
y=2
p=5
7=25
p=50
p=75
y=
BERT
96.92
96.93
97.35
97.59
97.66
97.70
Fine-tuning
reimp
96.71
96.96
97.60
97.61
97.66
97.71
BERT-CU CL
96.95
96.97
97.62
97.63
97.70*
97.75*
BERT
94.73
95.15
95.92
96.00
96.05
96.06
Probing
BERT-reimp
94.72
95.20
95.82
95.88
95.91
95,92
BERT-CL 94.96
95.341
96.02
96.071
96.12
96.17'
Table Results on WSJ POS tagging test set by model fine-tuning and linear probing. t. Statis- tically significant compared BERT-reimp with 0.01 and p < 0.05 by t-test, respectively,
Model
GLUE C0NLL2003
(wikipedia only)
Model LR
LP
F1
CM
BERT-reimp
78.2
91.49
BERT 95.20
95.32
95.26
52.06
BERT-CL
79.9
91.90
BERT-reimp
94.98
95.41
95.19
53.23
(wikipedia, bookcorpus, cC-news)
BERT-CL
95.54
95.58
95.56
54.53
BERT-reimp 78.9
91.52
BERT-CL 80.7 91.98
Table 5: Results WSJ parsing test set. CM means complete matching the constituency tree of the whole sentence.
Table 6: Results on GLUE and CoNLL2003
by using different pre-training corpus.
To compare with existing work, reim- plement the method from Nagatsuka et al. (2021) which use four-stage curriculum with increasing block-size (64, 128, 256, and 512) for BERT pre-training. We fol- low their setting with 250K training steps for each stage and compare with our method in Table 7 We find that the length-based curriculum does not help much in downstream tasks and our method significantly performs better on both GLUE and CONLL22003 NER tasks. This shows that our content-basee curriculum is not only closer to the process of language learning of humans, but also more helpful for model training.
Model
GLUE
CoNLL2003
Nagatsuka et (2021
78.2
90.91
BERT-CL
80.5
91.97
Table 7: Comparison with an existing method that leverages curriculum learning for pre-training.
To further evaluate the generalizability of our method, we also try our method to 1) different model settings including larger model RoBERTa-large, and generative-style language model GPT-2; 2) different curriculum training schedule. We find that our method model or GPT-2-style causal language model training, and the curriculum schedule can also affect the overall performance. Detailed results and discussion can be found in Appendix
also generalize larger
4 ANALYSIS
We analyze the possible reasons behind the performance ad advantage of BERT-CL, discussing how data-centric curriculum training helps language model pre-training.
4.1 REPRESENTATION DEGENERATION OF LANGUAGE MODEL
Existing work shows the representation degeneration problem of language models (Gao al. 2019 Ethayarajh] 2019; Wang et al., 2019b Cai et al. 2020 Biś et al. 2021] Yu et al. 2022). where the word embeddings or contextualized output are highly anisotropic and may limit the representation power. This problem also exists when training static word embeddings (Mu et al.) 2018). Figure visualizes the evolution of word embeddings over training iterations using PCA. In the top row, we find that the embeddings of BERT degenerate quickly in the early stage, where the overall distribu- tion falls into relatively narrow angle. The low-frequency words are significantly separated from others and the overall shape does not change much during all training iterations.

---

Under review as a conference paper at ICLR 2023
BERT (200K)
BERT (400K)
BERT (600K)
BERT (800K)
BERT (1M)
Low Fin
BERT-CL (200K)
BERT-CL (400K)
BERT-CL (600K)
BERT-CL (800K)
BERT-CL (1M)
Visualization of word embeddings during pre-training, the numbers in the parentheses denote the training steps. Different colors mean different word frequencies.
Figure& 3:
MNLI (matched)
QQP
QNLI
SST-2
85
70
87.5
92
80
85.0
90
75
88
80.0
86
70
BERT-CL
BERT-CL
BERT-EL
BERT-CL
-0+ BERT-reimp
BERT-reimp)
-0-
BE BERT-reimp
84
BERT-reimp
62
75.0
200K 400K 600K 800K
1M
200K 400K 600K BOOK 1M
200K 400K
600K 800K 1M
200K 400K 600K 800K 1M
CoLA
STS-B
MRPC
RTE
60
85
88
65
50
75
65
60
30
BERT-CL
BERT-C CL BERT-remp
BERT-CL
55
BERT CL
-
BERT-reimp
-0- BERT-reimp
-
BERT remmp
200K
400K 600K
800K
1M
200K 400K 600K BOOK 1M
200K 400K
600K
800K
1M
200K 400K 600K
800K 1M
Figure 4: The GLUE benchmark test set results from intermediate checkpoints during pre-training.
The evolution of word distribution BERT-CL is highly different. Since we incorporate words in different frequency intervals at different stages, they gradually form their clusters stage-br stage-by-stage by-sttge until 600K steps. When using the raw corpus for the last 400K steps training, we find that the low-frequency words are not represented separately, and different clusters come close each other. Finally, there no significant borderline for words with different frequencies, and the overall distribution also more uniform than BERT. In addition visualization,
use measure of isotropy in Mu et al. (2018) and Rajace & Pilehvar (2021) for evaluation. Details and results are shown in Appendix we find that curriculum training leads to a more isotropic representation
space quantitatively.
One of the reasons behind the representation degeneration problem is the frequency discrepancy between words. For example, Gong et al. (2018) find that the word embeddings are heavily biased towards word frequency, proposing an adversarial training method learn frequency-agnostio rep- resentations. [Gao et al. (2019) theoretically show that it could be caused by large number of rarely appeared tokens, and they use a cosine regularization term to enforce normalizing the distribution. Yu et al. (2022) leverage
adaptive gradient gating mechanism for rare tokens training. Although these methods alleviate the degeneration problem and improve performance in pure language mod- eling machine translation task, they require additional computational costs and not used in pre- pre-training. In contrast, we use a data-centric curriculum pre-training approach that introduces the constituent labels and decouples the words with different frequencies by reformulating the corpus.

---

Under review as a conference paper at ICLR 2023
BERT-CL (50K)
BERT-CL (500K)
BERT-CL (1M)
(
MM
QURR
1V
CAANP ADUP
NIS
NM V.G
VBD
(IC CHHAAUPP
ABUP
WHADVV
WHOMP CANAADV
SWIP
OHAAADY WINP
Figure 5: Visualization of the learned constituent label embeddings using t-SNE.
4.2 INTERMEDIATE RESULTS OF CURRICULUM TRAINING
In Figure we show the GLUE benchmark test set results using different checkpoints during BERT pre-training with and without curriculum training. We find that the BERT model gives a relatively higher performance in the initial stages for almost all tasks. However, the improvement is limited as the training step increases and also inconsistent for tasks such as SST-2, CoLA,, STS-B, and MRPC.
For BERT-CL, the performance is lower at the very beginning. but significantly and stably increases then, especially for the first 600K training steps. This shows that curriculum training can boost the capability of the model constantly. We also find that, for MNLI, QNLI, CoLA, and RTE, BERT-CL (600K) already performs on par with or better than BERT (IM). This shows that the reconstructed data can also steer PLM capability for certain tasks with less training cost.
4.3 VISUALIZATION OF CONSTITUENT LABEL EMBEDDINGS
The added constituent labels are heavily used during the early pre-training stages for data recon- structing, and their corresponding representations are also updated according to their parameters in the extended embedding table. Although the external added parameters can be discarded during fine-tuning, we are still interested in the role of these constituent labels during curriculum training.
Figure visualizes the 2D distributions from the embeddings of constituent labels. In the very early stages (steps 0-500), constituent structures can be learned quickly, where some small meaningful clusters emerge and some relationships are also mined. For example, the groups of nouns and verbs, and the similarity between JJR/JJS and RBR/RBS pairs. The discriminating distribution becomes clearer when the training steps increases, where the nouns and verbs (also the most common con- stituent structures) are gradually separated from each other. These show that our model could learn some meaningful concepts of these constituent structures during pre-training.
4.4 CONSTITUENT LABELS SERVE AS ANCHORS TO BRIDGE TOKEN LEARNING
In the initial stage, the constituent labels combined with the most frequent words so that the model can learn some syntax rules, the basic usage of frequent words or terms, and their interactions. Since we only leave the high-frequency words and bunch of labels, the vocabulary size is much smaller, the training signal received for each token is enriched and also more uniform.
The initial results can offer the fundamental capability for language understanding, we then allow more medium and low frequency words to participate in pre- prainnni combined with constituent la- bels and most frequent words that have been well learned. When adding infrequent words to replace the constituent labels, since the words and labels share similar context, the learned contextualized knowledge from constituent labels can also serve as guidance to the latter learning process of the up- coming words. From this perspective, the curriculum settings of using constituent labels can bridge the gap between words across different frequencies.
Table 8 shows the most similar words to some constituent labels in the trained embedding table. We find that 1) The neighboring words can reflect the fine-grained linguistic characteristics of the constituent labels. For example, the plural number ("games", "states", "children") of NNS, the 3rd person singular present ("is", "does", "has") of VBZ, and the cardinal number ("five", "million", "00") of CD; 2) Different words are captured reasonably according each constituent label, for
8

---

Under review as a conference paper at ICLR 2023
Lables
High (Top 1~000)
Medium (Top 500~33000)
Low (Top 3000
NNS
games, states, children him, she, he
teeth, victims, units
boxers, responses, bands descent, ghosts, witches receives, saves, serves whoever, wherever, wherein
NP
steps, himself. teeth
VBZ
is, does, has
becomes, feels, gets
WHNE
who, what, where
whom, whatever, whose Twenty, ten, hundred thousand, zero, decade
five, three, four million, two, 00
Jifty, fifieen, twelve forty, 9th, 8th
CD
Table 8: The most similar words each constituent label according to their dot product.
high, medium, and low frequency intervals. This shows that the constituent label can serve as an anchor or prototype to help model words of different frequency ranges in the curriculum; 3) Phrase- level constituent labels that are usually composed of multiple tokens such as noun phrase of NP and wh-adjective phrase of WHNP are also well encoded with meaningful similar words such "him", "she", "he", and "who", "what", "where". See Appendix F for more examples.
5 RELATED WORK
Knowledge Enhanced LM. It has been shown that PLM are capable of encoding syntax and seman- tic knowledge (Hewitt& & Manning 2019 Tenney et 2019 Pérez-Mayos et
202T There are
also a line of work explicitly integrating such knowledge to enhance model representation Lauscher
jt al. 2020 Sachan et al., (02T]] Xu et al. 2021b). In particular, Levine et al. (2020) leverage word
sense prediction task into BERT pre-training, Bai et al. (2022) propose hypernym class prediction for causal language modeling. These methods focus on word-level external knowledge stored in WordNet. Instead, we uses the hierarchical syntactic tree to inject word-, phrase- and clause-level knowledge, Morcover, through the underlying syntax structure of texts, our method can tackle more situations during pre-training where words are not in WordNet (e.g., url/email address, new words).
Curriculum Learning. Curriculum learning has been extensively studied in a range of tasks (Wang ett al. 2022). In natural language processing, [Bengio et al. (2009) first show that it can help general- ization and speed up the convergence of language modeling. For general-purposee language model pre-training. Campos| (2021) defines some sentence difficulty metrics based on sentence length, n- gram probability, and part-of-speech diversity for curriculum settings on L.STM-hased model. /Zhang al. (022)) group sequences with similar length during pre-training and find that it can help down- stream tasks. Nagatsuka et al. (202I) propose progressively increasing the block-size of input text, i.e., using sentences of increasing lengths for pre-training. Li et al. (202T) propose a regulariza- tion method for GPT-2 curriculum training, which is also based on sentence length. Unlike these methods, we build a linguistically motivated curriculum based on the learning content.
Data-centric AI. Data-centric method become an emerging topic for modern Al systems (g 2021 Hajij et al. 2021 Xu et al., 2021a; Huang et al. 2022 Eyuboglu et al. 2022). The main dea is to use an established model off off-the-shelff, but engineer the data for stronger results, including
data collection, annotation, augmentation, cleaning, reordering, and deduplicating (Russell et al,
2008 Krishnan et al. 2016 Wei & Zou| 2019 Press et al. 2021 Agrawal et al.) [202]] Lee et al. 2022. To better leverage the language model for downstream tasks, there is also a trend to build cloze-style samples for fine-tune fine-tuning (Schick & Schütze | Gao al.] 2021b). Reconstructed training data such as prompts or instructions are also being studied for better using large language models in few/zero-shot scenarios (Sanh al., 2022, Wei et al. 2022 Yuan & Liu 2022). In this
paper, we attempt to reformulate the data using a syntax- -guided mixup strategy for language model pre-training.
CONCLUSION
We investigate curriculum learning for language model pre-training and focus on a purely data- centric method, without setting multiple training tasks, modifying model architecture, introducing external computational overhead during pre- pre-training. Particularly, we propose a data mixup strategy that injects constituent labels into the text and progressively increases the vocabulary the corpus from gigh-frequency to low- low-frequency words. Experiments on multiple downstream tasks show that method leads to better performance compared with baselines.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ameeta Agrawal, Suresh Singh, Lauren Schneider, and Michael Samuels. On the role of corpus ordering in language modeling. In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, 2021.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A Latent Variable Model Approach to PMIIbased Word Embeddings. Transactions of the Association for Compu- tational Linguistics, 4:385-399, 2016.
He Bai, Tong Wang, Alessandro Sordoni, and Peng Shi. Better language model with hypernym class prediction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, 2009.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In Proceedings of Text Analysis Conference (TAC'09), 2009.
Daniel Bis, Maksim Podkorytov, and Xiuwen Liu. Too much in common: Shifting of embeddings in transformer language models and its implications. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared 1) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference on Learning Representations, 2020.
Daniel Campos. 2021.
Curriculum learning for language modeling. arXiv preprint arXiv:2108.02170
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)) 2017.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs, 2018
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023: 2022.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre- training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2020.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into single S&. #* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties ( (Volume 1: Long Pupers), 2018.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), 2019.
10

---

Under review as a conference paper at ICLR 2023
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre- pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, 2019.
Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):19-22) 211, 1990.
Jeffrey L Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71--))) 1993.
Kawin Ethayarajh. How contextual contextualized word representations? Comparing the ge- ometry of BERT, ELMo, and GPT-2 embeddings. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--JCNIPP) 2019.
Sabri Eyuboglu, Bojan Karlaš, Christopher Ré, Ce Zhang, and James Zou. Debench: A benchmark for data-centric Al systems. In Proceedings of the Sixth Workshop on Data Management for End-To-Enee Machine Learning, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 1310))111-39 2022.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang. and Yan Liu. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations, 2019.
Leo Gao. Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2021a.
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1th International Joint Conference on Natural Language Processing ( (Volume 1; Long Papers), 2021b.
Yoav Goldberg. Assessing bert's syntactic abilities. arXiv preprint 1711v:190..05287,0 2019.
Chengyue Gong, He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu, Frage: Frequency-agnostic word representation. In Advances in Neural Information Processing Systems, 2018
Tao Gui, Jiacheng Ye, Qi Zhang, Zhengyan Li, Zichu Fei, Yeyun Gong, and Xuanjing Huang. Uncertainty-aware label refinement for sequence labeling. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), 2020.
Mustafa Hajij, Ghada Zamzmi, Karthikeyan Natesan Ramamurthy, and Aldo Guzman Saenz. Data- centric Al requires rethinking data notion. arXiv preprint arXiv:21 10.02491, 2021.
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Information Science, 2017.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Wei Chen. Deberta: Decoding-enhance bert with disentangled attention. In International Conference Learning Representations, 2021.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pa- pers),
11

---

Under review as a conference paper at ICLR 2023
Yizheng Huang. Huaizheng Zhang, Yuanming Li, Chiew Tong Lau, and Yang You. Active-learning- as-a-service: An efficient mlops system for data-centric AL. arXiv preprint arXiv:2207.09109, 2022.
Peter Izsak, Moshe Berchansky, and Omer Levy. How to train BERT with an academic budget. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander Hauptmann. Self-paced curricu- lum learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2015.
Nikita Kitaev and Dan Klein. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018
Sanjay Krishnan, Jiannan Wang, Eugene Wu, Michael J Franklin, and Ken Goldberg. Activeclean: Interactive data cleaning for statistical modeling. Proceedings of the VLDB Endowment, 9(12): 948-99 108-959,2200 2016.
M. Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, 2010.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: A lite BERT for self self-supervised learning of language representations. In Interna- tional Conference on Learning Representations, 2020.
Anne Lauscher, Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. Specializing unsupervised pretraining models for word-level semantic similarity. In Proceedings of the 28th International Conference Computational Linguistics, December 2020.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022.
Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. SenseBERT: Driving some sense into BERT. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-seueene pre- training for natural language generation, translation, and comprehension. In Proc. of ACL, 2020.
Conglong Li, Minjia Zhang, and Yuxiong He. Curriculum learning: A regularization method for efficient and stable billion-scale gpt model pre-training. arXiv preprint arXiv:2108.0600 2021.
Nelson Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), 2019a
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019c.
12

---

Under review as a conference paper at ICLR 2023
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Christopher D Manning. Part-of-speech tagging from 97% to 100%: is time for some linguistics? In International conference on intelligent text processing and computational linguistics, 2011.
Mitchell Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Comput. Linguist., 19(2):313330,, jun 1993. ISSN 0891- 2017.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.0784 07843, 2016.
Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. In International Conference on Learning Representations, 2018.
Koichi Nagatsuka, Clifford Broni-Bediako, and Masayasu Atsumi. Pre-training a BERT with cur- riculum learning by increasing block-size of input text. In Proceedings of the International Con- ference on Recent Advances in Natural Language Processing (RANLP 2021), 2021.
Andrew Ng. Mlops: From model-centric data-centric AI, 2021.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In Proceedings the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016.
Laura Pérez-Mayos, Miguel Ballesteros, and Leo Wanner. How much pretraining data do language models need to learn syntax? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
2019.. 2019.
Ofir Press, Noah A. Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the Ith International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Jack W Rae. Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 1rXn::2112.11446 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 21(140): 2020.
Sara Rajaee and Mohammad Taher Pilehvar. A cluster- based approach for improving isotropy in contextual embedding space. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Pupers), 2021.
13

---

Under review as a conference paper at ICLR 2023
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. SQuAD: 100,000+ questions for machine comprehension text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lingnistics (Volume 2: Short Papers), 2018.
Bryan C Russell, Antonio Torralba, Kevin Murphy, and William T Freeman. Labelme: a database and web-based tool for image annotation. International journal of computer vision, 77(1):157- 173, 2008.
Devendra Sachan, Yuhao Zhang, Peng Qi. and William L. Hamilton. Do syntax trees help pre- trained transformers extract information? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2021.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompied training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
Timo Schick and Hinrich Schütze. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2021.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016.
Mohammad Shoeybi, Mostofa Patwary Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. arXiv preprint arXiv: 90..900900,........ 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2013.
Robyn Speer, Joshua Chin, Andrew Lin, Sara Jewett, and Lance Nathan. Laminosoinssghhexoodfreg v2.2, 2018.
Tomasz Stanislawek, Anna Wróblewska, Alicja Wójeicka, Daniel Ziembicki, and Przemyslaw Biecek. Named entity recognition is there glass ceiling? In Proceedings of the 23rd Con- ference on Computational Natural Language Learning (CoNLL), 2019.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure contextualized word representations. In Interna- tional Conference on Learning Representations, 2019.
Erik Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference Natural Language Learning at HLT-NAACL 2003, 2003.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2019a.
14

---

Under review as a conference paper at ICLR 2023
Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, and Quanquan Gu. Improv- ing neural language generation with spectrum control. In International Conference on Learning Representations, 2019b.
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555-455) 2022.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computationnal Linguistics. 7:625-641, 20199
Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on
text classification tasks. In Proceedings the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing JEMNLP-IJCNEL 2019.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. In Interna- tional Conference on Learning Representations, 2022.
Alexander Wettig, Tianyu Gao, Zexuan Zhong. and Danqi Chen. Should you mask 15% in masked language modeling? arXiv preprint arXiv:2202.08005. 2022.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), 2018.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao. Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint 16Xiv:1609.08/41 144, 2016.
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. Perturbed masking: freeee probing for analyzing and interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Liang Xu, Jiacheng Liu, Xiang Pan, Xiaojing Lu, and Xiaofeng Hou. Dataclue: A benchmark suite for data-centric nlp. arXiv preprint 1xXiv2211.006477 2021a.
Zenan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun Zhong, Xiaojun Quan, Daxin Jiang, and Nan Duan. Syntax-enhancee pre-trained model. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguisties and the I 11th International Joint Conference on Natural Language Processing (Volume Long Pupers). 2021b.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, 2019.
Sangwon Yu, Jongyoon Song, Heeseung Kim, Seongmin Lee, Woo-Jong Ryu, and Sungroh Yoon. Rare tokens degenerate all tokens: Improving neural text generation via adaptive gradient gating for rare token embeddings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), 2022.
Weizhe Yuan and Pengfei Liu. restructured pre-training. arXiv preprint arXiv:2206.1 2022.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, 2018
Wei Zhang, Wei Wei, Wen Wang, Lingling Jin, and Zheng Cao. Reducing bert computation by padding removal and curriculum learning. In 2021 IEEE International Symposium on Perfor- mance Analysis of Systems and Software (ISPASS), pp. 90-92, 2021.
15

---

Under review as a conference paper at ICLR 2023
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: Enhanced language representation with informative entities. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, 2015.
16

---

Under review as a conference paper at ICLR 2023
A LIST OF CONSTITUENT LABELS
We injected into the corpus with a total of 55 constituent labels defined in Penn Treebank, including:
Labels [-LRB-, -RRB-, ADJP, ADVP, CONJP, DT, EX, FRAG, FW, INTJ, JJ, JJR, JJS, LS, LST, NAC, NN, NNP, NNPS, NNS, NP, NX, PDT, POS, PRN, PRP, PRPS, PRT, QP, RBR, RBS, RP, RRC, SBAR, SBARQ, SINV, SQ. SYM, TOP, UCP, OH, VB, VBD, VBG, VBN, VBP, VB2, WDT, WHADJP, WHADVP, WHNP, WHPP, WP, WP $, WRB]
A brief description of these labels can be found in http:://urdem......aa...... mihai wtt.eaahhinnittaaaa01e1
1111 /1113eeaae readieadine Seaading iigiiing///ne BeeeeneeeeannCoosst.tooo html After data reconstruction, we treat these labels as normal tokens and enlarge our embedding table, making them involved in the masked language model pre-training:
modee_eAAuttoMoccalllooossssssee From pretrained (modee__aanne wttoeenizer_kwear
model. www.:rreizee_tttcc htw.reetttoeeeeeebdnnnnnnege (len (tckenizer)
B
STATISTICS OF DATASETS
Statistics of the datasets are shown in Table
Datasef
Task
#Train
#Dev
#Test
#Label
MNLI
Natural language inference
393k
20k
20k
3
QQP
Paraphrase
364k
40k
391k
2
QNLI
Natural language inference
108k
5.7k
5.7k
SST-2
Sentiment
67k
872
1.8k
CoLA
Acceptability
8.5k
1k
1K
2
STS-B
Similarity
7k
1.5k
1.4k
MRPC
Paraphrase
3.7k
408
1.7k
2
RTE
Natural language inference
2.5k
276
CoNLL2003
Named entity recognition
14.9k
3.4k
3.6k
8
SQuADI.1
Reading comprehension
87.6k
10.5k
9.5k
SQuAD2.0
Reading comprehension
130.3k
11.9k
8.9k
WSJ POS tagging
Part-of-speech lagging
38.2k
5.5k
5.4k
45
WSJ parsing
Constituency parsing
39.8k
1.7k
2.4k
52
Table 9: Statistics of datasets in our experiments.
C PARAMETER SETTINGS FOR FINE-TUNING
Fine-tuning parameters for GLUE, NER, SQuAD, POS tagging, and parsing are given in Table 10
GLUE
NER
SQUADL1&22.0
Epochs
(3 10, 20) {2e-5, 3e-5, 5e-5) [16. 32)
20
Learning Rate
2e-5
3e-5
Batch Size
16
12
Warmup Ratio
(0.06. 0.1)
POS Tagging (fine-tuning)
POS Tagging
Parsing
(probing)
terminated no improvement on dev set for 60 epochs
Epochs
20
20
Learning Rate
le-5
2e-3
3e-5
Batch Size
16
{8, 16)
32
Warmup Steps
160
Table 10: Parameter settings for fine-tuning downstream tasks.
17

---

Under review as a conference paper at ICLR 2023
D MORE EXPERIMENTAL RESULTS
D.1 DIFFERENT MODEL SETTINGS
For larger model settings, pre-train RoBERTa-large model (355M parameters) the same pus and compared on GLUE tasks. In particular, we use the recipe from Wettig et al. (2022) for efficient pre-training by using 40% masking ratio with 500K training steps.
Model
MNLI
QQP
QNLI
SST-2
CoLA
STS-B
MRPC
RTE
Avg.
RoBERTa-large
83.9/84.9
87.8
91.5
93.2
55.7
87.4
75.7
64.3
80.4
KoBERTa-large-non
83.3/84.1
88.1
91.8
93.6
51.3
88.0
75.5
64.9
80.0
RoBERTa-larg--Cl
85.5/85.5
88.5
92.4
94.0
56.8
87.4
80.0
66.1
81.8
Table 11: Comparison between larger models with RoBERTa-large setting.
Results on GLUE dev sets are shown in Table 11 Overall, compared with NoBERTa-large-reinn method lead to large improvement CoLÀ and MRPC, and also giving close performance or minor improvement across other tasks.
Beyond autoencoding-style model like BERT, method apply to auto-regressive model like GPT2 where the reconstructed data used for left-to-right language modeling. Specifically, we use GPT2-base as our backbone and pre-train on the same corpus. We evaluate them on LAM- BADA (Paperno et al. 2016), WikiText2 (Merity et al. 20166) and SWAG (Zellers et al. 2018 without any fine-tuning (i.e., zero-shot) using the LM evaluation framework from Gao et al. (2021a).
LAMBADA
WikiText2
SWAG
Model
ppl.
Acc.
ppl.
byte ppl.
bpb
Acc.
GPT2
40.06
32.54
37.30
1.96
0.97
53.78
GPT2-reimp
36.95
33.46
31.61
1.91
0.93
54.06
GPT2-CL
32.97
35.67
30.13
1.89
0.91
54.56
Table 12: Comparison between auto-regressive language models with GPT2-base setting. ppl: per- plexity, bpb: bits per byte.
Results are shown in Table 12 We find that the results auto-regressive LMs are still in favor of the proposed technique, where better results such as lower perplexity achieved.
D.2 DIFFERENT CURRICULUM TRAINING STEPS SETTINGS
We investigate the different training steps setting in our four-stage curriculum training, results are shown in Table
Model CL Settings
MNLI
QQP
QNLI
SST-2
CoLA
STS-B
MRPC
RTE
Avg.
RoBeRTi--lrge-reimp
83.3/84.1
88.1
91.8
93.6
51.3
88.0
75.5
64,9
80.0
RoBERTa-large-CC
20-20-20-40 (%)
85,5/85.5
88.5
92.4
94.0
87.4
80.0
66.1
81.8
10-20-30-40 (%)
85.5/85.8
88.6
92.1
93.6
56.2
87.6
79.5
81.7
25-25-25-25 (%)
84.4/84.8
87.8
91.9
93.2
56.0
86.8
77.7
67.1
81.0
10-10-10-70 (96)
84.2/84.6
87.9
91.6
93.6
55.2
87.0
77.5
66.1
80.8
Table 13: Comparison on different four-stage curriculum training schedule.
Overall, the proposed curriculum training can improve the performance stably. We find that different schedule settings affect the results, where moderate amount of training using both mixup and raw training data is necessary. For example, training with only 30% steps in the first three stages with the mixup data is not much sufficient. In future work, techniques such as self self-paced learning (Kumar et al. 2010 Tiang al. 2015) also be considered for setting better schedule.
18

---

Under review as a conference paper at ICLR 2023
E MEASURE THE ISOTROPY OF REPRESENTATION SPACE
We measure the isotropy of representation space using the metric defined by Mu et al. (0018). which also used for measuring recent language models (Rajaee & Pilehvar 2021
I(W) min.et (())
(2)
max Z(u)*
where W the set of representation vectors, is the set of eigenvectors of WTW, 7(u) partition function define in Arora et al. (2016):
((u) = Σ
exp(u" ws).
(3)
WsEW
The perfect isotropic space would have I(W) close to 1. We calculated the I(W) scores for BERT, BERT-reimp, and BERT-CL, the results are shown in Table 14
BERT
BERT-reimp
BERT-CL
T(W)
1.05e-5
6.15c-7
1.15e-4
Table 14: Measuring the isotropy of representation space of different models.
We find that the I(W) score of BERT-reimp lower that that BERT, the reason can be that the original BERT leverage multi-task training (masked language modeling and next sentence predic- tion). Compared with these two models, BERT-CL gives a higher / ((W score of 1.15c-4, showing that curriculum training can lead more isotropic representation space.
F
MORE EXAMPLES FOR THE MOST SIMILAR WORDS
More examples for the most similar words to each constituent label are shown in Table 15
Lables
High (Top 1~500) light, service, group from, for, the
Medium (Top 500~3300)
Low (Top 3000~)
NN
present, mark, mission present, steel, opposition crown, lights, figures promote, kill, develop ve, themselves broken, combined, dropped spoke, closed, ran
seed, concentration, penalty clay, audio, miniature minors, blues, blacks
NNP
NNPS
others, children, team keep, teil, let are, were, am taken, given, done took, had, could
VB
convert, recover, minimize
VBF
traded, overlap, dwell
VBN
torn, risen, divided tore, rolled, slid
VBD
VBG
taking, saying, looking
passing, putting, turning
advancing. returning, protecting
WHADVP
where, when, why
whenever, whom,
wherein, whereby, wherever
WRB
how, where, whether far, ago, least less, oh, ago best, least, most
whenever, lets, forgot
wherever, whereby, wherein
RBS
highest, worst, anywhere
hardest, fastest, shortest
RBR
wanna, faster. longer
sooner. hotter. warmer
JJS
worst, highest, largest greater, stronger, larger
lowest, deepest, smallest warmer, happier, thicker
JJR
better, more, less
Table 15: More examples for the most similar words to the constituent labels. NN: noun; NNP: proper noun, singular; NNPS: proper noun, plural; VB: verb; VBP: non-3rd person singular present; VEN: past participle; VBD: past tense: VBG: gerund or present participle; WHADVP: wh-adverb phrase; WRB: wh-adverb; RBS: adverb, superlative; RBR: adverb, comparative; JJS: adjective, superlative; JJR: adjective, comparative.
19


Reviews:
[{"review_id": "WqXRNOA43ig", "reviewer": "Reviewer_ET24", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A parse-tree based curriculum learning framework to learn a pre-train model.", "summary_of_the_review": "The work is novel and has good result on a small model and dataset. The main weakness is the lack of ablation study and testing on more models and datasets. ", "strength_and_weaknesses": "*strength: \n1. A working curriculum algorithm for vanilla masked language modeling.\n2. parse-tree based masking seems novel to me (but I'm not familiar with prior works)\n\n*weakness:\n1. the experiment is a bit limiting, as only a bert-Base is tested. It's unclear if this method can scale up to larger model and datasets.\n2. The scheduling of the curriculum learning seems a bit arbitrary and not systematically investigated.\n3. unclear how the method can generalize to languages where a parse-tree is not available.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F9kyCr2_Hlf", "reviewer": "Reviewer_VPVJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a curriculum learning (CL) method to train BERT, which is treated as blackbox (no need to modify the original implementation). The CL method consists of several stages, each of which is to replace rare words / phrases with syntactical tags (e.g. 1000 with CD). By doing that, BERT will only need to learn frequent words first, then rare words later. The paper shows that CL method helps mitigate the representation degeneration problem of BERT (i.e. BERT is biased towards frequent words). Also, it demonstrates that BERT trained with the CL outperforms the traditional training way on several down-streaming tasks. ", "summary_of_the_review": "I like the idea of the paper, but the paper should be improved to meet the conf. standard:\n1. the motivation is unclear, \n2. the experiment doesn't support the claim (only BERT-base-cased is demonstrated rather than LM in general) \n3. the analysis of mitigating the degeneration problem is unconvincing ", "strength_and_weaknesses": "The motivation of the paper is quite unclear. Specifically, it is difficult to see, in the paper, the reason CL was chosen. It is true that \"there is a more salient discrepancy between the current PLM training and the language learning process of humans\", but why is closing this gap helpful? What're specific aspects that the paper is looking for when employing CL? One thing I can see here in the paper, is to deal with rare words, but then the original BERT training already has a quite effective mechanisms, which is the tokenization (resulting in quite a small size vocab, 29k tokens).\n\nThe idea of employing syntactic tags to replace rare words / phrases is nice. But CL is not the only way to do. In fact, we can do a simple augmentation: randomly replace words / phrases with their syntactical tags during training BERT. By doing that way, trained BERT will also be aware of syntax and phrasal structures. Following this point, the fact that the new BERT outperforms BERT trained with traditional method could be due to the fact that the new BERT is more aware of syntax, rather than anything related to word frequency. \n\nThe experiments do not reflect fully the intention of the paper, which is targeting LM in general. In fact, the paper demonstrates on only one LM which is BERT-based-cased. \n\nThe analysis in section 4.1 is unconvincing. In Fig 3, we can see that for both BERT 1M and BERT-CL 1M, there is a rare-word cluster that is away from the rest, and mixture of rare-word and frequent-word clusters. It is difficult to see why BERT-CL mitigates the problem of representation degeneration. Should it be clear if methods of Gong 2018 or Gao 2019 are used for a more thoughtful analysis? \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DTd6Cw7oEMb", "reviewer": "Reviewer_71SG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a curriculum learning for language model pretraining: training with high-frequency tokens first, and low-frequency tokens later. They are 4 stages of pretraining:\n- 1. First 3 stages: training with the text whose low-frequency tokens are replaced with their syntactic label, for example, NP for noun, CD for number, etc. \nBy changing the replacement frequency threshold, the pretrain task difficulty can be controlled. This paper introduces 3 thresholds and split the first 600k training steps into 200k respectively. (the word frequency thresholds are <0.5k, 0.5k~3k, 3k~18k)\n\n- 2. Final stage: training with the unplaced original text. This stage is the final 400k steps of the training.\n\nExperiments show the BERT trained with their proposed method can outperform the BERT trained without it. \nEvaluation tasks include the GLUE benchmark, CoNLL03, SQuAD 1.1 and 2.0, and WSJ pos-tagging and constituency parsing.", "summary_of_the_review": "Overall, I like this idea and am happy to see this idea works on BERT pretraining. \n\nHowever, this submission misses a key related work that first proposes and uses the same \"low-frequency word replacement\" curriculum method for LM training. This submission needs to rephrase and correct its claim of originality. For example, the originality is how to replace words with labels (with a parser instead of the wordnet).\n\nIt would be great to add experiments to show the different word-replacing method is the key to BERT training. (the syntactic parser v.s. WordNet Hypernym relation)\n\nAlso, since this paper tries to show the curriculum training works for BERT, an analysis of the curriculum hyperparameters should be added.", "strength_and_weaknesses": "Strength: \n\n- 1. The idea is intuitive and well-described.\n- 2. The results show the improvements clearly and support the method.\n\nWeakness:\n\n- 1. See the originality concern in the next section.\n- 2. Whether the proposed method is robust to different hyper-parameters of the curriculum training is not answered. For example, the current setup is 200K steps per stage, and 400K for the final stage. I would suggest showing the impact of these hyperparameters.\n- 3. Reported finetuning results should be averaged across 3 or 5 times of finetuning-evaluating. In the current submission, I cannot find details of this part and I assume that the result of each task is from a single run.\n- 4. Only experiment with the BERT base, which is quite small (110M) in the context of pretraining in 2022/2023. Even for non-pretrained LM papers, 110M is a quite small model. For example, [1] trains 257M models. [2] trains 261M models. \n\n[1] Bai, He, et al. \"Better Language Model with Hypernym Class Prediction.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.\n\n[2] Press, Ofir, Noah A. Smith, and Mike Lewis. \"Shortformer: Better Language Modeling using Shorter Inputs.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5gfRa6xMcnF", "reviewer": "Reviewer_JTe8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper improves pre-training of MLM models (e.g. BERT) with a curriculum learning strategy, where not frequent words are substituted with their constituent labels. Rarer words are then introduced in pretraining in subsequent pretraining stages, until all words are substituted back to their original form.  Their approach shows minor improvements across multiple tasks including NER, QA, POS tagging and Parsing.", "summary_of_the_review": "Some additional qs:\n\n- Have you thought about using simpler synonyms for rare words instead of the constituent approach?\n\n- Have you thought about using sentence with simpler syntactic structure in the initial phases of pretraining?  \n\n- Would be good to cite relevant work on syntactic information inherently present in BERT like models: https://aclanthology.org/N19-1419/, https://aclanthology.org/2021.emnlp-main.118.pdf , and potentially apply these probes to see whether they actually improve.", "strength_and_weaknesses": "Strengths: their approach is novel and shows consistent results over multiple benchmarks,\n\nWeaknesses: results are very close between models in multiple benchmarks, it would be good to show whether they are within the range of noise, especially models trained on the same data.  Table 4 for example is extremely close.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject