How does the task complexity of masked pretraining objectives affect downstream performance?
Year: 2023

Abstract:
Masked language modeling (MLM) is a widely used self-supervised pretraining objective, where a model needs to predict an original token that is replaced with a mask given contexts. Although simpler and computationally efficient pretraining objectives, e.g., predicting the first character of a masked token, have recently shown comparable results to MLM, no objectives with a masking scheme actually outperform it in downstream tasks. Motivated by the assumption that their lack of complexity plays a vital role in the degradation, we validate whether more complex masked objectives can achieve better results and investigate how much complexity they should have to perform comparably to MLM. Our results using GLUE, SQuAD, and Universal Dependencies benchmarks demonstrate that more complicated objectives tend to show better downstream results with at least half of the MLM complexity needed to perform comparably to MLM. Finally, we discuss how we should pretrain a model using a masked objective from the task complexity perspective.1

heading: 1 Introduction
text: Masked language modeling (MLM) (Devlin et al., 2019), where a model needs to predict a particular token that is replaced with a mask placeholder given its surrounding context, is a widely used self-supervised pretraining objective in natural language processing. Recently, simpler pretraining objectives have shown promising results on downstream tasks. Aroca-Ouellette and Rudzicz (2020) have proposed various token-level and sentencelevel auxiliary pretraining objectives, showing improvements over BERT (Devlin et al., 2019). Yamaguchi et al. (2021) and Alajrami and Aletras (2022) have demonstrated that such token-level ob-
* Equal contribution 1Our code and pretrained models are available at https:
//github.com/hitachi-nlp/mlm-probe-acl2023.
jectives themselves, i.e., pretraining without MLM, perform comparably to MLM.
Although these simple token-level objectives themselves, e.g., predicting the first character of a masked token (First Char) (Yamaguchi et al., 2021), have exhibited competitive downstream performances to MLM with smaller computations, no objectives using mask tokens are not clearly comparable to MLM on downstream tasks. We conjecture that the main reason behind the performance difference lies in its lack of complexity, i.e., the number of classes to be predicted, and similar arguments have been made for auxiliary task ineffectiveness (Lan et al., 2020) and pretraining task design (Yamaguchi et al., 2021).
This paper sheds light on the task complexity of masked pretraining objectives and investigates RQ1: whether a more complex objective, becoming closer to MLM, can achieve a better downstream result and RQ2: how much complexity they need to obtain comparable results to MLM. To this end, we propose masked n character prediction as a control task, which requires us to predict the first or last n characters of a masked token, allowing us to empirically evaluate how the task complexity affects downstream performance by varying n. We pretrain 14 different types of models with the proposed control task in addition to MLM for reference and evaluate their downstream performance on the GLUE (Wang et al., 2019), SQuAD (Rajpurkar et al., 2016), and Universal Dependencies (UD) (Nivre et al., 2020) benchmarks. We also conduct a cost-benefit analysis of performance gains with respect to task complexity and analyze how to select an optimal complexity for a given task.
Contributions (1) We model the task complexity of a masked pretraining objective as masked n character prediction (§2) and revealed how it affects downstream performance (§4). (2) We conduct two analyses to provide insights into how we should
10527
pretrain a model by using a masked objective from the task complexity perspective (§5).

heading: 2 Methodology
text: Our main hypothesis is that the more complexity, i.e., the number of classes to be predicted, masked pretraining objectives have, the better the downstream performance they will achieve. This is because a more complex task should have a larger number of classes to be predicted, giving more informative signals to a model via training.
To verify the hypothesis empirically and answer the two research questions listed in §1, we extend First Char (Yamaguchi et al., 2021) and let a model predict the first or last n ∈ N characters of a masked token (n Chars)2. The task is trained with the token-level cross-entropy loss averaged over masked tokens. Our extension allows us to evaluate how the complexity affects downstream performance by varying n.
Figure 1 shows the number of classes to be predicted for n Chars when using a pretrained tokenizer of RoBERTa (Liu et al., 2019). We can see that the larger n, the closer the objective becomes to MLM. For 1 Char, we simply pick up the first character of each token in the vocabulary instead of casting it into 29 classes as in First Char3, resulting in 256 types of characters.
2We also allow last n characters to be predicted because the prediction direction should not matter for a model to learn effective representations.
3In First Char, a model needs to predict the first character of a masked token as 29-way classification, including alphanumeric characters, punctuation marks, and any other characters.

heading: 3 Experimental Setup
text: Here, we describe our experimental setups for both pretraining and fine-tuning.4
Model We used the base configuration of BERT (Devlin et al., 2019). The model consists of 12 hidden layers and attention heads, and the dimensions of hidden layers and intermediate feed-forward layers are 768 and 3072, respectively. We simply put a linear layer on the BERT model for n Chars and First Char.
Baselines We pretrained MLM and First Char for reference to evaluate the influence of masked pretraining objective complexity. We also set up Last Char, where a model needs to predict the last character of a masked token from 29 categories the same as in First Char.
Pretraining Data Following Devlin et al. (2019), we pretrained all models on English Wikipedia and BookCorpus (Zhu et al., 2015) using the datasets (Lhoest et al., 2021) library. We set the maximum sequence length to 512. We tokenized texts using byte-level Byte-Pair-Encoding (Sennrich et al., 2016), and the resulting corpora consist of 10 million samples and 4.9 billion tokens in total.
Fine-tuning Data We used the GLUE, SQuAD v1.1, and UD v2.10 benchmarks to measure both semantic and syntactic downstream performances in detail. For UD, we used its English subset of Universal Dependencies English Web Treebank (ENEWT) (Silveira et al., 2014).
Evaluation Following previous work (ArocaOuellette and Rudzicz, 2020), we report matched accuracy for MNLI, Matthews correlation for CoLA, Spearman correlation for STS-B, accuracy for MRPC, F1 scores for QQP and SQuAD, and accuracy for all other tasks. For UD, we used labeled attachment score (LAS). For each task, we report a mean score over five runs with different random seeds. We excluded problematic WNLI following prior work (Aroca-Ouellette and Rudzicz, 2020).
Implementation Details We implemented our models using the PyTorch (Paszke et al., 2019) and Hugging Face Transformers (Wolf et al., 2020) libraries. For fine-tuning on UD, we trained a deep biaffine attention parser (BAP) (Dozat and Manning, 2017) built on top of pretrained language
4For more details, please refer to Appendix B.
models. We used the SuPar library5 to implement the parser and followed its default hyperparameter configurations.

heading: 4 Results
text: RQ1: Do more complex objectives achieve better results? Table 1 displays downstream task results on GLUE, SQuAD, and UD for our control tasks and their comparisons against MLM, First Char, and Last Char. Overall, we observe the larger n, the better downstream performance is. Looking at each dataset result closely, we see that the datasets with over 5k training samples exhibit moderate to high correlations with the lowest and highest correlation values of 0.520 (First Char to MLM) for QQP and 0.942 (Last Char to MLM) for MNLI, respectively. In contrast, the corpora with less than 5k samples tend to exhibit low to moderate correlations, ranging from 0.182 (First Char to MLM) for RTE to 0.486 (First Char to MLM) for MRPC. Therefore, we can see a general trend that a more complex masked pretraining objective yields better performance in downstream tasks especially under a high-resource scenario, whereas it does not always achieve a better result on a low-resource
5https://github.com/yzhangcs/parser
dataset, where in this case MLM tends to achieve a better result.
RQ2: How much complexity do we need to obtain comparable results to MLM? To answer the RQ2, we only compare results on high-resource corpora, given that results on low-resource corpora have large standard deviations of 1.0 or more (see Appendix C.2). We can observe from Table 1 that at least n = 4 complexity is necessary to achieve comparable results to MLM. For instance, First n Chars objectives require n = 4 (20k classes) to surpass the MLM performance on at least one of the target downstream tasks. Last n Chars also need n = 4 (18k classes) to beat MLM on one of the tasks. For MNLI and QQP, our control tasks did not yield better results.

heading: 5 Analysis and Discussion
text: On the basis of results from the RQ1 and RQ2, we discuss how we should pretrain a model in practice from the task complexity perspective.
Task complexity affects computational efficiency. The task complexity is closely related to computational costs with more complex objectives larger costs. Table 2 shows the number of floating-point
operations (FLOPs)6 required for n Chars, First Char, Last Char, and MLM along with its downstream performance. We can see a clear trade-off between computational efficiency and downstream performance. Smaller n drastically reduces FLOPs with the maximum relative reduction of 31% for First Char and Last Char, while larger n has small reduction rates with the minimum value of 6% for First and Last 9 Chars. To obtain comparable results to MLM, as we discussed in the RQ2, we need at least n = 4 (First and Last) complexity, which can only reduce 18% and 20% of FLOPs, respectively. These results suggest that we need careful cost-benefit consideration when using a masked pretraining objective on the basis of target performance and computational costs.
How can we select an optimal complexity? Finally, we discuss how we can decide an optimal complexity for a specific downstream task by analyzing how the task complexity affects the performance in detail. Here, we take SQuAD as an example case, where we observed a large relative difference of 3.0% in Table 2. Table 3 lists the results on SQuAD, including the ratio of mis-detection, i.e., the percentage of samples with no overlap between predicted and gold spans, and F1 scores calculated without mis-detected cases. We found
6We use FLOPs to evaluate computational costs following Clark et al. (2020).
that simple masked objectives were likely to suffer from mis-detection with the worst performance degradation of 38.2% for Last Char, which is far larger than those observed in other metrics and corpora (see Tables 2 and 7 in Appendix). In contrast, the relative performance difference values of F1 scores computed without mis-detected samples show quite similar trends to other high-resource corpora, which are typically less than 2% at a maximum. These results imply that the task complexity mainly contributes to an increase/decrease in the number of mis-detections in SQuAD, and selecting a complex masked objective (e.g., First 5 Chars) is a safeguard option to minimize the effect of misditection. Therefore, different downstream tasks might have different optimal complexities due to their characteristics and evaluation metrics, as observed in the example case above. We leave thorough investigation of these effects as future work.

heading: 6 Conclusion
text: This paper analyzed the impact of masked objective complexities over downstream performance, motivated by the assumption that the lack of task complexity in simple masked pretraining objectives (e.g., First Char) affects the performance degradation compared to MLM. Experiments using the GLUE, SQuAD, and UD datasets revealed that the
task complexity significantly affected downstream performance with at least 35.7% of the MLM prediction classes needed to perform comparably to MLM on at least one of the high-resource corpora. Our analysis also showed that there exists a tradeoff between downstream performance and computational efficiency, and different downstream tasks might have different optimal complexities. Future work includes analyzing other properties (e.g., fairness) with respect to task complexity.
Limitations
Model Architecture Due to our computational resource constraints, we only used the BERT base architecture. We cannot confirm whether our results and observations are transferable to any other Transformer-based architectures, especially for larger ones.
Randomness We did not run pretraining for multiple times with different random seeds due to the limited computational resources and research budgets, though we fine-tuned models five times each with different random seeds in any downstream tasks. This might affect the overall results shown in the paper.
Languages Other than English It is uncertain whether any results and conclusions presented in this paper are applicable to any other languages other than English, as our experiments are solely on English data. We may need further experiments especially for languages that do not belong to the same language family as English, such as Chinese and Japanese.
Ethics Statement
This work does not involve any sensitive data but only uses publicly available data, including Wikipedia, GLUE, SQuAD, and UD as explained in the paper. Although we plan to release the resulting models, they might perform unfairly in some circumstances, as reported in Baldini et al. (2022). We highly recommend users to refer to studies on debiasing pretrained language models, such as Guo et al. (2022).

heading: Acknowledgements
text: We would like to thank anonymous ACL 2023 reviewers and Yuichi Sasazawa for their insightful comments. We also would like to thank Dr.
Masaaki Shimizu for the maintenance and management of the large computational resources used in this paper.

heading: A Masked n Characters Prediction
text: The task of predicting first or last n characters of a masked token (n Chars) is trained with the tokenlevel cross-entropy loss averaged over the masked ones only, following First Char (Yamaguchi et al., 2021). We mask 15% of input tokens the same as in BERT.
Our method generates a label dictionary for n Chars similar to that of First Char. We used a pretrained tokenizer of RoBERTa (roberta-base) provided by the Transformers library and generated a label of each token in the vocabulary by picking out a specified number of characters. We did not count a special blank character of Ġ when generating labels. The average number of characters per token in the roberta-base vocabulary is 5.72± 1.78.

heading: B Detailed Experimental Setup
text: We trained our models with four NVIDIA Tesla V100 (32GB) for pretraining and one for finetuning. Note that we used eight V100 GPUs for MLM to match the total batch size of 128 used for other models.
B.1 Pretraining We pretrained all models for 500k steps following Alajrami and Aletras (2022) and optimized the models with AdamW (Loshchilov and Hutter, 2019). Table 4 shows the hyperparameter settings used in pretraining.
B.2 Fine-tuning Table 5 lists the hyperparameters for fine-tuning models on GLUE, SQuAD, and UD benchmarks.
For GLUE and SQuAD, we used early stopping. For UD, we compute an average for each token over the top four layers of the BERT hidden representations and use it as an input to BAP. The dimensionalities of arc and relation features given to each biaffine module are 500 and 100, respectively.

heading: C GLUE
text: C.1 Evaluation Metrics
Following previous work (Aroca-Ouellette and Rudzicz, 2020), we report matched accuracy for MNLI (Williams et al., 2018), Matthews correlation for CoLA (Warstadt et al., 2019), Spearman correlation for STS-B (Cer et al., 2017), accuracy for MRPC (Dolan and Brockett, 2005), F1 scores for QQP7 and SQuAD, and accuracy for all other tasks, including SST-2 (Socher et al., 2013), QNLI (Wang et al., 2019) and RTE (Dagan et al., 2005; Bar Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009).
C.2 Results
Table 6 shows the detailed GLUE results with standard deviations for each mean value. We can see that standard deviations on low-resource corpora with fewer than 10k samples tend to be larger than that of high-resource corpora.
7https://quoradata.quora.com/ First-Quora-Dataset-Release-Question-Pairs

heading: D Relative Performance Difference from MLM
text: Table 7 shows the results on GLUE, SQuAD, and UD benchmarks along with the relative performance differences from MLM in percent.

heading: E License
text: SQuAD and UD (EN-EWT) are distributed under the CC BY-SA 4.0 license. GLUE has different licenses but is freely available for typical machine learning uses. We used all of the corpora for benchmarking purposes only and did not modify their contents.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3, Section 4, and Section 5.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 1, Section 3, and Appendix C.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix E. The datasets we used are widely-used open-source benchmark datasets in NLP, and we just utilized them to evaluate downstream performances of models.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The datasets we used are widely-used open-source benchmark datasets in NLP, and we just utilized them to evaluate downstream performances of models, which is exactly the intended use.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The datasets we used are widely-used open-source benchmark datasets in NLP.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Table 1.
C 3 Did you run computational experiments? Section 4 and Section 5.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 3 and Appendix.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3 and Appendix.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3, Section 4, and Section 5.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3 and Appendix.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work includes analyzing other properties (e. g. , fairness) with respect to task complexity. Limitations
Model Architecture Due to our computational resource constraints, we only used the BERT base architecture. We cannot confirm whether our results and observations are transferable to any other Transformer-based architectures, especially for larger ones. Randomness We did not run pretraining for multiple times with different random seeds due to the limited computational resources and research budgets, though we fine-tuned models five times each with different random seeds in any downstream tasks. This might affect the overall results shown in the paper