Bootstrapping Small \& High Performance Language Models with Unmasking-Removal Training Policy
Authors: Yahan Yang
Elior Sulem
Insup Lee
Dan Roth
Venue: EMNLP 2023 Main (2023)
Keywords: language models
efficient pre-training
masking policy

Abstract: BabyBERTa, a language model trained on small-scale child-directed speech while none of the words are unmasked during training, has been shown to achieve a level of grammaticality comparable to that of RoBERTa-base, which is trained on 6,000 times more words and 15 times more parameters. Relying on this promising result, we explore in this paper the performance of BabyBERTa-based models in downstream tasks, focusing on Semantic Role Labeling (SRL) and two Extractive Question Answering tasks, with the aim of building more efficient systems that rely on less data and smaller models. We investigate the influence of these models both alone and as a starting point to larger pre-trained models, separately examining the contribution of the pre-training data, the vocabulary, and the masking policy on the downstream task performance. Our results show that BabyBERTa trained with unmasking-removal policy is a much stronger starting point for downstream tasks compared to the use of RoBERTa masking policy when 10M words are used for training and that this tendency persists, although to a lesser extent, when adding more training data.

Bootstrapping Small & High Performance Language Models with Unmasking-Removal Training Policy
Yahan Yang', Elior Sulem', Insup Lee', Dan Roth'
Department of Computer and Information Science, University of Pennsylvania
2 Department of Software and Information Systems Engineering,
Ben-Gurion University of the Negev [yangy96, lee, danroth)@seas. upenn. edu el eliorsuêbgu. ac. il
Abstract
monsense knowledge in order achieve high per- formance on natural language understanding tasks, compared to grammatical ability. On the other hand, LMs like RoBERTa are costly to train in terms of GPU computation power and time at both pre-training and fine-tuning stages. Huebner et al. (2021) proposed BabyBERTa, smaller RoBERTa architecture that is trained on 5M child-directed data corpora without using unmasked tokens dur- ing the masked language modeling training. Baby- BERTa reaches the same level of grammaticality as RoBERTa but considerably saves training expenses. However, no further evaluation on tasks other than the grammaticality tests were performed for the model. Deshpande et al. (2023) also focus on the performance of smaller language models but em- phasize the relationship between the architecture size and downstream task performance and train on larger data corpora. In this paper, we would like to answer the following questions: (1) What is the performance for LMs based on smaller models like BabyBERTa on downstream tasks that require fine-tuning? and (2) What is an efficient way to improve the behavior of small pre-trained LMs downstream tasks?
BabyBERTa, a language model trained on small-scale child-directed speech while none the words are unmasked during training, has been shown to achieve level of grammaticality comparable to that of RoBERTa- base, which trained on 6,000 times more words and 15 times more parameters (Huebner al., 2021). Relying on this promising result, we explore this paper the performance BabyBERTa- based models in downstream tasks, focusing Semantic Role Labeling (SRL) and two Extrac- tive Question Answering tasks, with the aim of building more efficient systems that rely on less data and smaller models. We investigate the influence of these models both alone and a starting point to larger pre-trained models. separately examining the contribution of pre training data, the vocabulary, and the mask- ing policy on the downstream task performance. Our results show that BabyBERTa trained with menuuskin-reemowe policy is much stronger starting point for downstream tasks compared the use of RoBERTa masking policy when 10M words are used for training and that this tendency persists, although to a lesser extent, when adding more training data."
1 Introduction
In our work, we first evaluated both BabyBERTa and RoBERTa on three downstream tasks that tar- get sentence structure and closely associated with grammatical capabilities. Additionally, we propose to have various starting points by com- bining different ingredients in pre-training includ- ing the masking policy, the size of the vocabulary, and the type of the pre-training data corpus (child- directed language, online written language). Then, we continually pre-train BabyBERTa and its vari- ants more Wikipedia data to improve perfor- mance on target tasks. We observe that: (1) al- though Baby has a lower performance on the downstream tasks compared RoBERTa, the use of the unmasking removal policy and of small vocabulary still effective after fine-tuning; (2)
Large-scale pre-trained language models (LMs) (Devlin al., 2019; Liu al., 2019; Yang al., 2019) have shown promising ability on handling various downstream tasks including textual classifi- cation (Wang et al., 2018) and question answering (QA, Rajpurkar et al., 2016). Previous research (Zhang et al., 2021; Warstadt et al., 2020) showed that the size of architecture and amount of pre- training data actually affect the linguistics features learned by state-of-the-an (SOTA) pre-trained LMs like RoBERTa (Liu et al., 2019). also showed LMS require much more data to understand com-
'Our code can be found in https://github com/ vangy96/babybertan cttpobbbrraacontinnaaa

---

running thorough experiments to identify which factors are important for small language models when performing continual pre-training, we find that the influence of the unmasking removal policy persists, although to a lesser extent, when adding more training data.
training hyperparameters. The details are shown in Table 8. The default masking policy of Baby- BERTa is the unmasking removal policy, and the pre- training data corpora AO-CHILDES (Hueb- ner and Willits, 2021), which consists of child- directed speech. We consider four versions of BabyBERTa, each of them being trained on a dif- ferent corpus (Table 1).
2 Background
2.1 Masked language model objective (MLM)
Corpora CHILDES
The large transformers for language models are pre-trained on billions of tokens and show their high-capability in various downstrcam tasks. The success of large-scale pre-trained language models is inseparable from the masked language model objective, which a widely-used self-supervised learning approach to construct a text representation (Devlin et al., 2019). With the MLM objective, there are p% of the tokens that are masked, and the model learns to reconstruct the masked tokens at the pre-training stage. The loss function is defined
CIIIBERTS-CHILLES
(child-directed speech)
Wikipedia (a small subset of Wikipedia dataset)
BahyBBRRTaWWkipcdii
Combines CHILDES,
BabyBEErTa-Curicuuum
Newsela, and Wikipedia Combines two Wikipedia suhscts the same size
BabyBERTaaCombined
Table I:
Data corpora for training BabyBERTa and its three variations, where CHILDES and Wikipedia contain the same number of sentences.
2.3 Fine-tune on downstream tasks
In this work, we are interested in following downstream tasks including semantic role label- ing (SRL) and two extractive question-ansee answering tasks: question-answer driven semantic role label- ing (QASRL) and question- answer meaning repre- sentation (QAMR).
Σ
log ((u
(1)
where Wi is the ground truth of the jth masked tokens of ith sequence and z; is the masked context, n is the total number of sentences and m, is the number of masked tokens for the sentence.
1) SRL Semantic role labeling aims to detect predi- cates (in most cases, verbs) in a sentence and assign its associated arguments with different semantic roles (Palmer et al., 2010; Carreras and Mårquez, 2005; He et al., 2017). In this paper, we evaluate models on CoNLL11, an SRL benchmark based on OntoNotes v5.0 dataset (Pradhan et al., 2013).
2.1.1 80-10-10 Masking Policy
80% of the masked tokens are replaced by the <mask> token, 10% are replaced by randomly se- lected tokens, and the rest are kept as the same tokens (Devlin et al., 2019). In the paper, we use the 80-10-10 and RoBERTa masking policy inter- changeably.
2) QASRL (He et al., 2015) also presents the predicate-argumen structure in the sentence but in the format of question-answer pairs 3. In this paper, we evaluate models on the QA-SRL Bank 2.1 dataset (FitzGerald et al., 2018). 3) QAMR (Michael et al., 2018) provides reddicate-argumen structure for more diverse rela- tionships compared to those presented in QASRL and SRL (including noun relationship).
2.1.2 Unmasking Removal Policy
Different from the default masking strategy, we instead remove the prediction of unchanged / un- masked tokens. In other words, we replace cor- rupted tokens with <mask> 90% of the time and use random tokens 10% of the time?. Previous work (Huebner et al., 2021; Wettig et al., 2023) shows masking policies important to pre-training.
3 BabyBERTa on downstream tasks
In this section, evaluate BabyBERTa models various downstream tasks and experiment with dif- ferent methods including continually pre-training. To perform
2.2 BabyBERTa
BabyBERTa (Huebner al., 2021), a small-scale version of RoBERTa, differs from RoBERTa in ar- chitecture, corpora, masking policy, and other pre-
tasks like QAMR and QASRL, we train two lincar layers on top
We use 90-1 10 masking policy and unmasking removal policy interchangeably
*We here address the Extractive QA tasks (rather than the parsing tasks) related to the QAMR and QASRI formalisms.

---

the encoder of the language model (LM) to pre- dict the start and end of the answer span within the context. We implement the classifier using Hug- gingface (Wolf et al., 2020). For lin-ttunin LMs for SRL tasks, we utilize the implementation pro- vided in (Zhang et al., 2022).
BERTa when we have larger vocabulary but less pre-training data.
3.1.2 Effect of masking policy
One observation in previous work (Huebner et al., 2021) is that, compared to BabyBERTa trained with 80- 60-10-18 10 masking policy, BabyBERTa trained with unmasking- removall policy achieves higher scores on grammar tests. This leads to an interesting ques- tions: what is the impact of the masking policy of the starting point on downstream tasks? Here, we apply two masking policies at the pre-training stage. The results in Table show that the unmask- ing policy works better for models with smaller architectures like BabyBERTa on these three down- stream tasks. Thus, we conclude that BabyBERTa pre-training with unmasking removal policy and smaller vocabulary set achieves the best perfor- mance across three different tasks given the results in Table 3.
3.1 How does the BabyBERTa perform on downstream tasks?
We are interested in the performance of Baby- BERTa and its variations on downstream tasks in- cluding SRL, QASRL, and QAMR. We report the Fl score in Table 2, and compare the performance of BabyBERTa models and RoBERTa. Our ex- periments show that BabyBERTa has comparable performance on QASRL with ROBERTa-IOM and only 3 points lower compared to RoBERTa. For tasks like SRL and QAMR, BabyBERTa's ' perfor- mance is also within slight 3-point margin in com- parison RoBERTa-IOM. We also observe that the content of the pre-training dataset impacts its performance on downstream tasks. The Wikipedia dataset is closer to the target domain compared to the other two datasets, so BabyBERTa pre-trained on Wikipedia dataset achieves higher performance on QAMR, which is more challenging task.* 4
3.2 Does continually pre-training BabyBERTa improve downstream tasks performance?
Since there is a performance gap between Baby- BERTa and RoBERTa shown in previous experi- ments, we consider improving the performance by continually pre-train the BabyBERTa architecture on more data. To be specific, each time we pretrain the models on a new subset of the Wikipedia dataset contains about 100M words repeatedly. Given the results in section 3.1, we choose the starting points trained with the unmasking removal policy and BabyBERTa vocabulary set.
Pre-trainet Models
SRL
QASRL
QAMR
BabyBERTa- CHILDDS
72.38
87.57
54.03
BabyBERTa- Wikipedia
75.96
90.09
77.43
BabyyBRRTaCCrricull
77.89
90.13
73.88
BahyBERTa-Comiine
76.17
89.9
77.05
RoBERTa- 10M
79.75
90.44
80.76
RoBERTa
85.00
93.11
90.58
Table 2: Performance (Fl-score) of BabyBERTa and its variants on three different downstream tasks. The performance of RoBERTa and RoBERTa- 10M serves as a baseline to compare.
For all continually pre-train procedures, we keep using RoBERTa masking policy. The masking ra- tio used in our experiments 15% as the default. Table presents the downstream performance of models trained with continual pre-training, con- sidering various starting points. For comparison, we include results from RoBERTa-10OM (Zhang et al., 2021). To assess the impact of a more di- verse dataset, we mix BookCorpus and Wikipedia as an additional dataset for continual pre-training (Gururangan et al., 2020) 7. We observe that the smaller architecture, after continually training on 100M data, can achieve better and comparable per- formance for the QASRL and QAMR tasks respec- tively, compared to RoBERTa- pre-trained
3.1.1 Effect of vocabulary size
The vocabulary size of RoBERTa is approximately 6x that of BaRT, so is possible that the size of the vocabulary size limits the understanding of language at the MLM training stage. In this experi- ment, we compare the performance with different vocabulary sets for pre-training the Baby BERTa model. Table 3 summarizes our experiments for pre-training BabyBERTa with various factors. We observe that the larger vocabulary does not give any improvement in most of the cases. We hypoth- esize that the training efficiency is low for Baby-
'A starting point here is a BERTa model with spe- cifie masking policy, vocabulary set, and initial training corpus 'The results correspond to the mean value of three runs. 7 Mixed with ratio 1:3 as in (Zhang et al., 2021)
'More details about the training procedure are in Ap- pendix.

---

Baby
Baby Wikipedia
URPs
Vocabulary
SRL
QASRL
QAMR
URPs
Vocabulary
SRL
QASRL
QAMR
yes
RoBERTa
69.47
87.19
53.72
yes
ROBERTa
74.41
89.94
69.61
RoBERTa
70.03
86.54
53.57
nO
RoBERTa
73.53
89.32
66.26
yes
BabyBERTa
72.38
87.57
54.03
yes
BabyBFRTT
75.96
90.09
77.43
no
BabyBERTa
72.44
86.72
53.36
no
BabyBERIa
75.86
89.13
68.7
BabyBERRa- BERIa-Combbbeee Combined
BabyBERla-Cuman
URPs
Vocabulary
SRL
QASRL
QAMR
URPs
Vocabulary
SRI
QASRL
QAMR
yes
RoBERTa
73.59
89.36
67.61
yes RoBERTa
73.38
89.18
66.61
no
RoBERTa
73.11
89.25
66.17
DO
RoBERTa
73.10
88.99
64. 12
yes
BabyBERTa
76.17
89,9
77.05
yes
BabyBERTa
77.89
90.13
74.96
nO BabyBERTa
76
89.84
69.23
no
BabyBERTa
75.81
89.61
66.84
Table 3: Performance (FI-score) BabyBERTa and its variants on three different downstream tasks. evaluate the impact of the vocabulary size and the masking policies. We highlight the best performance for different pre-training corpora. URPs indicates whether the Unmasking Removal Policy (URP) is used during pre-training.
on 10M data. Morcover, among the small mod- cls, the SeehyyBRTa--iiippee model trained on the mixed dataset overall demonstrates the best performance on QAMR and SRL, and achieves comparable performance on QASRL with the best- performing model.
Model
URPs
URP
SRL
QASRL
QAMR
78.04
90.48
77.60
CHILL
yes
78.08
90.43
77.88
yes
yes
78.19
90.56
78.60
no 77.95
90.40
74.83
Wiki
yes no 78.07
90.78
79.88
yes yes
78.08
90.93
80.43
Curr
no
no
77.93
90.64
79.57
Model
Dataset wiki
SRL
QASRL
QAMR
yes
no 78.22
90.67
79.6
yes yes
78.27
90.77
79.68
78.18
90.73
79.98
Wiki
mixed
78.47
90.73
80.29
Table 5: The impact of masking policy after pre-training
wiki
78. 14
90.63
79.87
Comb
mixed
78.47
90.60
79.44
on more data. The models are pre- ire-trained on 100M tokens in total. URPs indicates whether the Unmasking Removal Policy (URP) is used the starting point and URPc denotes whether URP is used for continually pre- training. CHIL. Wiki, and Curr refer to BabyBERTa- CHILDES, Wikipedia and Curriculum respectively. We highlight the best performance for each starting points.
wiki
78.45 47
90.68
79.61
Curr
mixed
78.33
90.75
79.50
RoBERTa- 10M
mixed
79.75
90.44
80.76
RoBERTa-IOOM
mixed
80.31
91.82
87.24
RoBERTa
mixed
85.00
93.11
90.58
Table 4: Downstream performance of continually pre- train BabyBERTa more data. The models trained on 100M tokens in total. (The starting points are using BabyBERTa vocabulary set and 90-10 masking policy.) We highlight the best performance across different start- ing points and datasets.
3.3 Scale to more data
A After combining the optimal training policies as discussed in the previous section, we continually pre-train the smaller models on more data. The learning curve of the model is presented in Fig- ure on downstream tasks more data become available (500M tokens). The performance contin- ually improves as we keep pre-training the model on new data sequentially. In Table we report the final performance after continually pre-training the model on 1B tokens. However, the performance is still lower than that of RoBERTa-base (Liu et al., 2019).
Additionally, we show how the masking pol- icy of the starting points affects continually pre- training in Table 5. We report the mean value of three runs of the models in the table and use the methods recommended in (Dror et al., 2018) for Fl score evaluation: we apply bootstrap to perform the significance test of 3 rums and get the p-value 0.04 when a = 0.05 for BEbT and 0.0 for Baby BERT-Wikipedii It again shows and BabyBERTa-
that the BaayTTERT Wikipedia gain from unmasking removal policy for QAMR. We summarize that for BabyBERTa- the unmask-
Model
SRL
QASRL
QAMR 82.37
CHILDES and BabyBBERTWikipeeee ing removal policy at the starting point improves the performance on downstream tasks, and, for QAMR, even after continuing pre-training. How- ever, the BabyBERTa BarTa--rrruuuiuuuu does not show the same trend on QAMR (p- = 0.25).
Comb
79.40
91.29
RoBERTa
85.00
93.11
90.58
Table Performance (Fl-score) of continually pre- training BabyBERTa with IB tokens three different downstream tasks.

---

It was also supported by Contracts FA8750--9-2- 0201 and FA8750-19-2-1004 with the US Defense Advanced Research Projects Agency (DARPA) as well as by grants from the Israeli Ministry of In- novation, Science & Technology (#000519) and the BGU/Philadelphiaa Academic Bridge (The Sut- nick/Zipkin Endowment Fund), Approved for Pub- lic Release, Distribution Unlimited. The views ex- pressed those of the authors and do not reflect the official policy or position of the Army Research Office, the Department of Defense or the U.S. Gov- crnment. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This research was also supported by gift from AWS Al for research in Trustworthy Al.
80
75
F1 Score
70
65
curriculum combined childes
60
55
0
200 300
400
500
Amount of tokens (M)
Figure 1: Fl score versus the size tokens. Continually pre-training various variants of BabyBERTa until 500M tokens. (On QAMR task)
4
Conclusion
In this work, we investigate three important factors for improving smaller language models on down- stream tasks: vocabulary set, masking policy, and dataset at the starting point. Our findings reveal that continuously pre-training smaller model like BabyBERTa leads continued improvement in downstream performance. Additionally, employ- ing the unremoval masking policy and utilizing a smaller vocabulary prove advantageous for down- stream tasks. We provide a comprehensive inves- tigation into the relationship between pre-training procedures and downstream tasks for small models. In future research, we aim delve deeper into the abilities acquired during the pre-training stage and their impact on downstream task performance.
References
Xavier Carreras and Lluís Marquez. 2005. Introduction to the conll-2005 shared task: Semantic role labeling, In Proceedings of the ninth conference on compu- tational natural language learning (CoNLIL20055), pages 152-164.
Vijeta Deshpande, Dan Pechi, Shree Thatte, Vladislav Lialin, and Anna Rumshisky. 2023. Honey. shrunk the language: Language model behavior at reduced scale. Findings of the Association for Compu- national Linguistics: ACL 2023, pages 5298-5314, Toronto, Canada. Association for Computational Lin- guistics.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Pruceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186 Minncapolis, Minnesota. Association for Computational Linguistics.
Limitations
Our study specifically concentrated on masked lan- guage model objectives and downstream tasks that are closely associated with grammaticality. How- ever, it would be interesting evaluate our findings diverse downstream tasks, such as the GLUE benchmark (Wang et al., 2018). Furthermore, our investigation primarily focused on the BЕbYBA architecture configuration and small data corpus (5 IB). would be valuable to explore the correlation between different pre- training factors and various architecture configurations.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing sta- tistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1383-1392, Melbourne, Aus- tralia. Association for Computational Linguistics.
Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL pars- ing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- une 1: Long Pupers), pages 2051-2060, Melbourne. Australia. Association for Computational Linguistics.
Acknowledgements
We thank the members of the Cognitive Compu- tation Group and the anonymous reviewers for their insightful suggestions. Research was spon- sored by the Army Research Office and was accom- plished under Grant Number W91 INF-20-1-0080.
Suchin Gururangan. Ana Marasovié, Swabha Swayandipta, Kyle Lo, Iz Beltagy, Doug Downey,

---

and Noah A. Smith. 2020. Don't t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Pranav Rajpurkar. Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas, Association for Computational Linguistics.
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettle- moyer. 2017. Deep semantic role labeling: What works and what's 's next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 473-483, Vancouver, Canada. Association for Computational Linguistics.
Alex Wang, Amampreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018, GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353 953-355, Brussels, Belgium. Association for Com- putational Linguistics.
Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015, Question-ana annwerr driven semantic role labeling: Us- ing natural language to annotate natural language. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 643-653, Lishon, Portugal. Association for Compu- tational Linguistics.
Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020. Learning which fea- tures matter: RoBERTa acquires preference for linguistic generalizations (eventually). In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 217-235, Online, Association for Computational Lin- guistics.
Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. BabyBERTa: Learning more gram- mar with small-scale child-directed language. In Pro- ceedings of the 25th Conference on Computational Natural Language Learning, pages 624-646, Online. Association for Computational Linguistics.
Alexander Wettig. Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023. Should you mask 15% in masked language modeling? In Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistiics pages 2985- 3000, Dubrovnik, Croatia. Association for Computa- tional Linguisties.
Philip A. Huebner and Jon A. Willits. 2021. Chapter eight- using lexical context to discover the noun category: Younger children have it easier. In Kara D. Federmeier and Lili Sahakyan, editors, The Context of Cognition: Emerging Perspectives, volume 75 of Psychology of Learning and Motivation, pages 279- 331. Academic Press.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of of the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692.
Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2018 Crowdsoure- ing question-answer meaning representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 560-568, New Or- leans. Louisiana. Association for Computational Lin- guistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. In Advances in Neural Infor- mation Processing Systems, volume 32, Curran Asso- ciates, Inc.
Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021. When do you need bil- lions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), ). pages 1112-1125, Online. Association for Computational Linguisties.
Martha Palmer, Daniel Gildea, and Nianwen Xue, 2010. Semantic role labeling. Synthesis Lectures on Human Language Technologies, 301:
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina. Yuchen Zhang, and Zhi Zhong, 2013. Towards ro- bust linguistic analysis using OntoNotes. In Proceed- ings of the Seventeenth Conference Computational Natural Language Learning, pages 143-152, Sofia, Bulgaria, Association for Computational Linguistics.
Yu Zhang, Qingrong Xia, Shilin Zhou, Yong Jiang, Guohong Fu, and Min Zhang. 2022. Semantic role labeling as dependency parsing: Exploring latent

---

tree structures inside arguments. In Proceedings of COLING, pages 4212-4227, Gyeongju, Republic of Korea. International Committee Computational Linguisties.
A
Appendix
A.1
Dataset details
Here we provide more details about the dataset of the downstream tasks.
Train
Validation
Test
SRL
75187
9603
9479
QASRI.
215427
38487
45387
QAMR
50509
18772
18596
Table 7: The number of samples in SRL,, QASRI, and QAMR datasets.
A.2 Architecture and datasets of BabyyERT
Here we provide the model configurations and the size of the datasets.
RoBERTa
BabyBERTh
layers
12
8
attention heads
8
hidden size
768
256
intermediate size
1024
vocabulary size
50265
8192
Table 8: Architecture of BabyBERTa and RoBERTa.
Dataset Size
BabyBERIi-CHILDES
6.5M
15.91M
31.81M
Baby
31.9MM
Table 9: Data corpora size for training Baby and its three variations.
A.3
Compare masking policy with more pre-training data
In this section, we investigate the impact of mask- ing policy of starting point when continually pre- training the model with more than 100M tokens. Specifically, plot the performance QAMR versus the number of tokens for Baby CHILDES trained with unremoval masking policy and 80- 10-10-1 10-10 masking policy. We observe that the performance of CHILDES trained with unremoval masking policy keeps getting better performance compared to CHILDES with 80-10- 10 masking pol- icy after continue pre-training on more and more data.
A.4 Continually pre-train on the task-specific data
Prior work (Gururangan al., 2020) suggests that continually pre-training on task-specific dataset an effective domain adaptation for downstream tasks. Following this work, we continually pre- training the model on the dataset such as QASRL,

---

training stage, the number of steps we is 260K and the batch size 16. The learning rate is lc-4 and the weight decay is 0.01. At the continually pre-training stage, the number of steps we use is 300K and the batch size is 256. The learning rate is lc-4, the warmup steps are set to be 6000 and the weight decay is 0.01.
80
75
Score
70
65
60
CHILDES-URP
55
CHILDES-no-URP
At the fine-tuning stage for QAMR, QASRL and SRL, the model is fine-tuned the target dataset for 10 epochs, epochs, and epochs respectively. The batch size is set 16 and the learning rate is 2e-4.
200 300
400
500
Amount of tokens (M)
Figure 2: score versus the size tokens to compare the Continually pre-training various variants of Baby- BERTa until 500M tokens. (On QAMR task)
QAMR, and OntoNotes. To ensure consistency, we convert the context from QASRL and QAMR into the same format as the pre-training data. The results are listed in Table 10.
Model
Dataset
SRL
QASRL
QAMR
N/A
76.01
90.09
77.43
QAMR
76.1
90.03
77.47
Wiki
QASRL
76.01
89.92
77.73
OntoNotes
77.16
90.07
77.78
N/A
76.17
89.9
77.05
QAMR
76.84
90.17
77.91
Comb
QASRL
76.43
89.92
77.74
OntoNotes
77.25
90.03
78.13
N/A
76.11
89.75
74.96
QAMR
76.13
89.88
76.52
Curr
QASRL.
75.97
89.61
75
OntoNotes
76.7
89.94
76.02
Table 10: Performance of continually pre-train Baby- BERTa on task-specific data.
A.5
Continually pre-train with RoBERTa vocabulary
Here we present additional results of continually pre-train the model with RoBERTa vocabulary.
Model
Vocab
QASRI.
QAMR
BabyBERTa
90.77
80.03
Wiki
RoBERTa
90.93
80.38
BabyBERTa
90.78
79.76
Comb
RoBERTa
90.94
78.38
BabyBERTa
90.72
79.76
Curr
RoBERTa
91.05
76.91
Table 11: Downstream performance of continually pre- train BabyBERTa sets. The models are trained on 100M tokens in total.
data for different vocabulary
(The starting points are using 90-10 10 masking policy.)
A.6 Implementation details
All of our models are trained and evaluated on two Nvidia Quadro RTX 6000. At the initial pre-


Reviews:
[{"review_id": "Lh18fKe6rR", "reviewer": "Reviewer_bCR8", "summary": "", "questions": "- (A) What was the motivation for this work, and what is the wider significance of your findings? What insights do they offer into linguistics or efficient NLP systems?\n- (B) How did you conduct the significance test mentioned on line 236? Were there multiple training runs for each configuration in Table 5?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments.\n\nQ(A): BabyBERTa [1] showed an efficient way to pretrain a language model so it can learn grammaticality to the same level as RoBERTa, while using much less training data and fewer parameters. The aim of this paper is to investigate the potential of this efficient language learning - either using BabyBERTa directly or as a starting point - for building NLP systems for downstream tasks, with less data and smaller models. Our work shows that the small language models which gain outstanding grammar capability also achieve promising performance on SRL related tasks after fine-tuning or continually pre-training. We consider this work helps to deepen the understanding of small language models trained on small amounts of data, which has two main implications.\n1) Both pre-training time and fine-tuning time of BabyBERTa decrease drastically in comparison to RoBERTa-base due to the smaller architecture. Evaluating and understanding the usage and limitations of small language models on downstream tasks can offer valuable insights to guide the development of efficient and environmentally friendly NLP systems in the future.  \n2) Our finding enhances the potential to use transformer-based language models in domains and languages where less data are available. \n\nQ(B): We perform a t-test by comparing the results of two systems. Specifically, we collect the binary results from two different systems on the same test dataset (correct prediction 1, wrong prediction 0) and run a one-tail t-test for it. We did not conduct multiple runs for the configurations in Table 5. However, we report the mean value of three runs of BabyBERTa-CHILDES in the attached table. And we also use the methods recommended in [2] for F1 score evaluation: apply bootstrap to perform the significance test of 3 runs and get the p-value of 0.03 when alpha=0.05. It again shows that the BabyBERTa-CHILDES gains from unmasking removal policy for QAMR. We will add results for multiple runs of all configurations in Table 4&5 in the final version. Our conclusion emphasizes on BabyBERTa benefits from URP on downstream tasks for some starting point datasets such as CHILDES after continual training but not in all configurations. \n\n| | URP\\_S | URP\\_C |  SRL| QASRL | QAMR |\n|----|----|----|----| ----|-----|\n|BabyBERTa-CHILDES | no |no | 78.04 | 90.48 | 77.60 |\n| | yes |no | 78.08 | 90.43 | 77.88 |\n| | yes |yes | **78.19** | **90.56** | **78.60** |\n\nPresentation Improvements: Thank you for pointing those out and we will fix them in the final version. \n\n[1] Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. *BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language.* In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 624–646, Online. Association for Computational Linguistics.\n\n[2] Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. *The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing.* In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, Australia. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written.\n- The experimentation seems thorough and mostly supports the authors' claims.\n- Replicates the findings of Huebner et. al. on new tasks, increasing confidence in the generality of BabyBERTa's linguistic abilities.", "reasons_to_reject": "The high-level motivation of this work was not clear to me, and I didn't feel that I learned much from this paper that wasn't already discovered by Huebner et. al. (2021), who introduced BabyBERTa. Some justification is required for the focus on improving BabyBERTa's performance - as I understand it, BabyBERTa was intended primarily as an experiment in child language acquisition research, not the basis for building practical NLP systems. However this is somewhat outside my area of expertise and I might be persuaded by a convincing answer to my question (A).\n\nWhile Section 3.1 extends Huebner et. al.'s analysis to new tasks and shows that a smaller vocabulary is advantageous in the BabyBERTa training regime, it doesn't offer much in the way of new insight - Huebner et. al. already showed the benefit of unmasking removal. Sections 3.2 & 3.3 attempt to improve BabyBERTa's performance primarily through continued pre-training, but again I don't feel it's very insightful to show that a model trained on 5M tokens performs better when it's trained on 100M or 500M tokens.\n\nThere isn't much differentiation in the numbers in Tables 4 & 5 across BabyBERTa settings so I'm not sure we can conclude much about the effect of corpus and masking policy during continued pre-training on downstream performance. The authors claim to have obtained a significant result with a t-test when comparing the performance of BabyBERTa-CHIL on QAMR with and without URP, however they have not explained how they perform this t-test - there is no mention of multiple training runs, so presumably the t-test is over the examples in the test set, so only establishes that one specific training run is better than another, not that one training method is better than another in general. Also, this result seems to be cherry-picked as the authors have not reported significance test results for any of the other configurations and tasks in Table 5. The fact that significance is attained on just one specific configuration and task isn't very meaningful, and I note that on the same task where the authors claim a significant advantage for URP, no-URP works better with the other pre-training corpus.", "questions_to_authors": "- (A) What was the motivation for this work, and what is the wider significance of your findings? What insights do they offer into linguistics or efficient NLP systems?\n- (B) How did you conduct the significance test mentioned on line 236? Were there multiple training runs for each configuration in Table 5?", "paper_topic_and_main_contributions": "This paper evaluates the capabilities of several variants of BabyBERTa (Huebner et. al., 2021), a smaller RoBERTa architecture trained on a 5M token corpus, on three semantic role labeling/meaning representation datasets, two of which are extractive QA-driven. It analyses the effect of the corpus type, vocabulary size and use of \"unmasking\" during BabyBERTa pre-training, showing that task performance is generally better when unmasking is not employed and BabyBERTa's smaller vocabulary is used rather than RoBERTa's larger one. The authors also consider the effect of continued pre-training of BabyBERTa, showing that this leads to an improvement in downstream task performance, and the effect of using or not using unmasking during the original and continued pre-training phases.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are a few typos (e.g. \"pertaining\" on lines 200-201). The headings \"BabyBERTa\" and \"RoBERTA\" in Table 8 are the wrong way round."}, "extra_text": {}}, {"review_id": "MxauY9ioQz", "reviewer": "Reviewer_rmr4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. \n1. Incremental work: Previous work of BabyBERTa only concentrated on the zero-shot evaluation on grammar test suites while our work focuses more on its performance on downstream tasks in the pretrain and then fine-tune paradigm. Our work shows that the small language model, trained on child acquisition language, benefits from unmasked removal policy and smaller vocabulary for downstream tasks like SRL. We also demonstrate BabyBERTa can be used as a starting point and its performance on downstream tasks can be enhanced through continuous training. We consider this work to be an important step in extending the utility of BabyBERTa and contributing to the understanding of small language models.\n2. Limited Applicability: As we wrote in the Limitations section, we agree that more ablation study on model architecture and more tasks would be an interesting future directions to deepen the understanding of unremoval masking policy and small language models. Nonetheless, the primary focus of this paper is centered on comprehending the utilization of a small language model like BabyBERTa for structural downstream tasks and examining strategies such as unremoval masking policy to enhance its performance. The advantage of larger pretraining data corpus (over 100M) is out of the scope of this paper. \n\nPresentation improvements:\n- 1 & 2. Thank you for pointing this out. We will fix the typos in Table 8 and line 115.\n- 3. We are sorry about the confusion of the sentences line 162-169. The revised version should be  “Our experiments show that BabyBERTa has comparable performance on QASRL with RoBERTa-10M and only 3 points lower compared to RoBERTa. For tasks like SRL and QAMR, BabyBERTa's performance is also within a slight 3-point margin in comparison to RoBERTa-10M.”\n- 4. Yes, the difference between Wikipedia and Combined is that Combined contains two wikipedia of the same number of sentences. Wikipedia contains 16M tokens and Combined contains about 31M tokens. We will include this information in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It evaluates BabyBERTa in the downstream tasks: BabyBERTa has comparable performance with RoBERTa in QASRL; there is a noticeable performance gap in SRL and QAMR. The gap can be mitigated to some extent by continual training.\n\n- It analyzes the effect of the unmasking-removal strategy in pretraining BabyBERTa: the removal is beneficial with small-size pretrained data (5M) and the advantage diminishes when the model is continually trained with more data (100M). It may inspire future work to consider unmasking-removal in low-resource pretraining (small data, small model).\n", "reasons_to_reject": "- Incremental work: The prior work BabyBERTa extensively explores the pretraining strategies and hyperparameters, and finds that unmasking-removal in pretraining is effective for syntactic probing tasks. This work validates the finding in the SRL-based downstream tasks.\n\n- Limited applicability: while the work highlights unmasking-removal, it only experiments with very specific architecture (BabyBERTa) and with three syntactic tasks. Further, in this setup, the effectiveness holds within a narrow range (data size 5M-100M).\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work is built upon BabyBERTa (a smaller version of RoBERTa-base trained on small-scale corpus).  \n- While the original paper evaluates the linguistic knowledge with probing tasks based on minimal pairs, this work evaluates BabyBERTa on three grammar-related downstream tasks (SRL/CoNLL-12, QASRL, QAMR) within the pretrain-and-finetune paradigm.\n- They analyze the effect of pretraining factors (corpus, vocabulary size, unmasking removal) on the downstream performance.\n- They further investigate the pretrain-continue-finetune paradigm. They continually pretrain on 100M Wikipedia data and analyze the unmasking removal strategy.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Appendix Table 8: the header \"BabyBERTa\" and \"RoBERTa\" should be exchanged.\n\n- line 115: hyperparamters -> hyperparameters\n\n- I cannot understand Sec 3.1 Line 162-169. ``Our experiments show that BabyBERTa has comparable performance on QASRL and SRL compared to the other two tasks since QASRL and SRL are only points lower compared to RoBERTa’s performance''. From Table 2, BabyBERTa is indeed only 3 points lower than RoBERTa in QASRL, but there is a noticeable gap for SRL.\n\n- Table 1: What is the difference between BabyBERTa-Wikipedia and BabyBERTa-Combined? wiki1 vs. wiki1 + wiki2? How large is each dataset?"}}, {"review_id": "tZqUT5lrV1", "reviewer": "Reviewer_oY1d", "summary": "", "questions": "Line 236: how about the other combinations besides BabyBERTa-CHIL with and without URP on QAMR? Do we consistently see improvements with URP with statistical significance? ", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. \n1. The reason for evaluating the performance of BabyBERTa on SRL-based downstream tasks is their focus on sentence structure analysis and their close relationship with grammaticality where BabyBERTa shows superior performance. Evaluating its performance on broader NLU tasks like the GLUE dataset goes beyond the scope of this paper. However, leveraging the strengths of smaller language models like BabyBERTa for broader tasks is an interesting future direction. We believe this paper is a first step to analyze how small language models can be used for downstream tasks. \n\nQ1. Yes, we also tried other configurations with/without URP and showed that smaller models benefit from URP for QAMR. Here, we attach the results of BabyBERTa-Wikipedia (starting point trained on Wikipedia dataset) and observe that improvements with URP is 5 points higher for this model. We will include these results in the final version of the paper. \n\n |        | URP\\_S  | URP\\_C | QAMR           |\n|-----| ----| ----| ----|\n|BabyBERTa-Wiki | no     | no     | 75.61          |\n |                             | yes    | no    | 80.09    |\n  |                            | yes    | yes    | **80.23** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents valuable insights into the pretraining/finetuning of small language models. This could have a significant impact, as small language models are much easier to pretrain and finetune than larger models.", "reasons_to_reject": "As an empirical analysis paper, the small LM’s performance in the experiments is only evaluated on two kinds of downstream tasks (SRL and two extractive question answering). It would be more convincible to conduct the evaluation on more tasks with broader coverage.", "questions_to_authors": "Line 236: how about the other combinations besides BabyBERTa-CHIL with and without URP on QAMR? Do we consistently see improvements with URP with statistical significance? ", "paper_topic_and_main_contributions": "This paper empirically evaluates the impact of three factors for improving small language models on downstream tasks: the pretraining data, the vocabulary, and the masking policy. Three downstream tasks are evaluated: semantic role labeling (SRL), question answer driven semantic role labeling (QASRL), and question answer meaning representation (QAMR). The evaluations reveal that unremoval masking policy can generally improve model performance, while increasing vocabulary size does not bring positive impact. Besides this, continuously pretraining the small LM with more data leads to continued improvements.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is focused on evaluating BabyBERTa, a pre-existing language model trained on small-scale child-directed speech, on a series of SRL-based downstream tasks. While the goal of training smaller models on fewer data is potentially useful to the research community, this paper partly suffers from a lack of novelty, as many of its ablations (such as unmasking removal) were already present in Huebner et al.'s original BabyBERTa paper, albeit not evaluated on tasks requiring fine-tuning. Moreover, this work would benefit from running statistical significance tests for all different model configurations to obtain more compelling evidence from the results. Overall, this work is incremental and partly limited in scope and applicability (studying just a single LLM on SRL tasks); on the other hand, it provides some interesting insights into optimal pre-training strategies in low-resource scenarios (such as smaller vocabulary sizes and unmasking removal). Hence, this paper might be considered for Findings.
Decision: Accept-Main