Revisiting Token Dropping Strategy in Efficient BERT Pretraining
Year: 2023

Abstract:
Token dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT, by skipping the computation of a subset of the input tokens at several middle layers. It can effectively reduce the training time without degrading much performance on downstream tasks. However, we empirically find that token dropping is prone to a semantic loss problem and falls short in handling semantic-intense tasks (§2). Motivated by this, we propose a simple yet effective semantic-consistent learning method (SCTD) to improve the token dropping. SCTD aims to encourage the model to learn how to preserve the semantic information in the representation space. Extensive experiments on 12 tasks show that, with the help of our SCTD, token dropping can achieve consistent and significant performance gains across all task types and model sizes. More encouragingly, SCTD saves up to 57% of pretraining time and brings up to +1.56% average improvement over the vanilla token dropping.

heading: 1 Introduction
text: Masked language models (MLMs), such as BERT (Devlin et al., 2019) and its variants (Liu et al., 2019; He et al., 2020; Zhong et al., 2023a)1, have achieved great success in a variety of natural language understanding (NLU) tasks. However, with the scaling of model size and corpus size, the pretraining of these BERT-style models becomes more computationally expensive and memory intensive (Jiao et al., 2020; Hou et al., 2022). Hence, it is crucial and green to speed up the training and reduce the computational overhead for BERT-style pretraining (Zhang and He, 2020; Schwartz et al., 2020).
∗ Corresponding Authors: Juhua Liu (e-mail: liujuhua@whu.edu.cn), Bo Du (e-mail: dubo@whu.edu.cn)
1We refer to these models as BERT-style models.
To achieve this goal, various training-efficient approaches have been developed and summarized (Shoeybi et al., 2019; You et al., 2019; Zhang and He, 2020; Shen et al., 2023). Among these efforts, a recently-proposed token dropping2 strategy (Hou et al., 2022) has attracted increasing attention owing to its easy-to-implement algorithm and impressive efficiency (reducing the training cost by 25% without much average performance dropping) (Yao et al., 2022; Chiang et al., 2022). Different from most previous works that focus on changing model architecture or optimization process, token dropping aims to improve training efficiency by dynamically skipping the compute of the redundant (unimportant) tokens that are less informative to the current training, at some middle layers of BERT during training. Although achieving a remarkable speedup, the performance improvement of token dropping is usually limited and unstable, compared to the baseline training scheme. More specifically, we empirically found that token dropping falls short in handling semantic-intense tasks, as shown in Figure 1. This motivates us to explore and address the limitations of token dropping in this paper.
In light of the conventional wisdom that “seman2We also refer to it as “token drop” in some cases.
10391
tic information is mainly encoded in the BERT’s intermediate and top layers” (Jawahar et al., 2019), we suspected, apriori, that the corruption caused by the removal of unimportant tokens would break the sentence structure, and may easily lead to the semantic drift of sentence representations, as also observed in many similar scenarios (Zhang et al., 2022; Wang et al., 2021). To verify this conjecture, we conduct a series of preliminary analyses on a representative BERT model, and find that:
❶ The training dynamics of the token dropping show a significant semantic drift.
❷ The representation of a well-trained BERT with token dropping contains less semantics.
❸ The downstream semantic-intense tasks show a clear performance degradation.
Based on these observations, we can basically conclude that (one of) the limitation of token dropping is the semantic loss3 problem, which causes vulnerable and unstable training of BERT models. To address this limitation, we propose a simple yet effective semantic-consistent learning method (referred to as SCTD) to improve token dropping. The principle of SCTD is to encourage the BERT to learn how to preserve the semantic information in the representation space. Specifically, SCTD first introduces two semantic constraints to align the semantic information of representations between baseline- and token dropping-based models, and then adopts a novel hybrid training approach to further improve the training efficiency.
We evaluate SCTD on a variety of benchmarks, including GLUE (Wang et al., 2018), SuperGLUE (Wang et al., 2019) and SQuAD v1/v2 (Rajpurkar et al., 2016, 2018), upon two typical MLMs: BERT-BASE and -LARGE. Results show that SCTD can not only bring consistent and significant improvements (up to +1.56% average score among all tasks) into the token dropping strategy on both BERT models, but also alleviate the semantic loss problem. Moreover, compared to the standard BERT models, SCTD can also save up to 48% of pretraining time while achieving comparable performance, and further achieve +1.42% average gain for the same training iterations.
To summarize, our contributions are as follows: 3As we find that BERT models trained with token dropping are prone to losing some semantic-related polarities, e.g., less semantic knowledge in the dropped layers, we refer to this phenomenon as “semantic loss" in the paper.
• Our study reveals the semantic loss problem in the token dropping strategy, which limits its performance on downstream tasks, especially on semantic-intense tasks.
• We propose a simple yet effective, plug-inplay approach (SCTD) to alleviate the semantic loss and further improve efficiency.
• Experiments show that SCTD outperforms the vanilla token dropping with up to +1.56% average improvement and saves up to 57% of pretraining time.

heading: 2 Revisiting Token Dropping Strategy
text: In this section, we first review the background of token dropping strategy and then present the empirical analyses of this strategy in detail.

heading: 2.1 Preliminaries
text: Suppose that we focus on pretraining the BERT with l transformer layers. Let Li denote the i-th (i ∈ {1, ..., l}) layer, Xi ∈ Rsi×d be the output tensors of i-th layer, where si is the sequence length of i-th layer and d is the hidden size. Notably, X0 denotes the input (after embedding) of the model. For the baseline training process (as illustrated in Figure 2 (a)), full-sequence tokens will be sequentially fed into all layers, i.e., s0 = s1 = ... = sl. In
this way, we can obtain the final output tensors Xl of l-th layer, and then use a cross-entropy loss to optimize the training process as follow:
LMLM = E ( − ∑ logP (Y |Xl) ) , (1)
where Y denotes the ground-truths. For token dropping (as illustrated in Figure 2 (b)), different from the full-sequence training, the training of a subset of unimportant tokens in middle layers will be skipped4. In practice, for stable training, token dropping follows the full-sequence training at several first layers (i.e., from 1-th layer to (l/2 − 1)-th layer). Then, it uses several importance scores/metrics to determine the dropped tokens and divides tokens into two groups, where we denote the “group1” as important tokens and “group2” as unimportant (dropped) tokens. The group1 tokens will be fed into later layers (i.e., from (l/2− 1)-th layer to (l − 1)-th layer), while the computations of the group2 tokens are skipped. Lastly, all tokens are merged before the last layer and then are used to obtain the final outputs5 X̃l. The loss function of token dropping is similar to Eq. 1, and we refer to it as L∗MLM .

heading: 2.2 Empirical Analyses
text: In this part, to verify whether removing the unimportant tokens will cause the loss of semantic information and thus hinder the performance of token dropping, we conduct systematic analyses from three aspects: 1) revealing the semantic drift problem during training dynamics; 2) probing the representation of a well-trained model with token dropping; 3) evaluating the downstream performance on semantic-intense tasks. In practice, for comparison, we pre-train the representative BERTbase models with baseline training scheme and token dropping, respectively. Through the above analyses, we empirically observe that:
❶ The training dynamics of the token dropping show a significant semantic drift. As suspected in §1, the corruption caused by the removal of several tokens would break the sentence structure, thus leading to semantic drift. Here, we verify this conjecture by quantitatively estimating the
4Hou et al. (2022) state that such a process would not only hardly damage the effect of pretraining, but also reduce the computation costs.
5To distinguish from final outputs Xl of baseline training, we denote it as X̃l.
loss of semantic information contained in the corrupted sentence. For measuring the semantic information, we first adopt the off-the-shelf SentenceBERT (Reimers and Gurevych, 2019) to capture the semantic representations. Then, suppose that the original sentence (without any corruption, such as masking or token dropping) contains full semantic information, we refer to the cosine similarity between semantic representations of the corrupted and original sentences as a metric to measure the semantic drift in the corrupted sentence.
In practice, given some sentences randomly sampled from training data, we follow the above process and measure the (average) semantic drift during the baseline/token dropping training dynamics, respectively. For reference, we also report the validation results and illustrate all results in Figure 3. It can be found that: compared to baseline training, i) sentence semantics in token dropping drifts more from the original semantics; ii) token dropping hinders the full learning of BERT, especially in the middle and later training stages (after 75K steps). To have a closer look, we show the similarity and validation gaps between both settings in the inserted figure of Figure 3. As seen, with the training going on, both gaps have a similar tendency to increase6, especially at the beginning of training. In general, these analyses indicate that there is a significant semantic drift during training dynamics
6The curve of validation gap tends to flatten in the later training stage, as both models are going to converge.
of token dropping, which shows a correlation with the performance drop of token dropping.
❷ The representation of a well-trained BERT with token dropping contains less semantics. In addition to the analysis during training dynamics, we then investigate the semantic properties of welltrained models. Specifically, following many prior works (Conneau et al., 2018; Jawahar et al., 2019; Ding et al., 2020; Zhong et al., 2022a), we perform several semantic-aware probing tasks on the sentence representations at different layers. Taking the Tense and subject number (SubjNum) tasks as examples, we provide the comparison of semantic information between baseline and token dropping at different layers in Figure 4.
We observe that there is more semantic information in the top layers (from layer 9 to layer 12) of BERT trained with the baseline scheme, which is similar to the finding of Jawahar et al. (2019). However, when using the token dropping, the semantic information contained in BERT tends to decrease in the dropped layers (from layer 5 to layer 11). The semantic information of token dropping at 11-th layer drops dramatically, which is much lower (up to 25.2 points) than that of baseline. Moreover, due to the vulnerable and unstable training, the final representation in token dropping at the last layer is also sub-optimal. These results basically prove that the semantic drift of token dropping damages the semantic learning ability of BERT.
❸ The downstream semantic-intense tasks show a clear performance degradation. The aforementioned analyses mainly focus on interpreting the semantic properties of models. Here, we further evaluate the downstream performance of token dropping. Specifically, several representa-
tive semantic-intense7 tasks are used, including OntoNotes 5.0 (Weischedel et al., 2013) (Onto. for short), CoNLL03 (Sang and De Meulder, 2003), MRPC (Dolan and Brockett, 2005) and SICKRelatedness (Marelli et al., 2014) (SICK-R for short). Notably, for Onto. and CoNLL03, we report the few-shot (32-shot) performance to enlarge the performance difference between different models. We measure the development performance of each task using its corresponding evaluation metrics, and report the contrastive results in Table 1.
As seen, there is a great discrepancy between the downstream performance of baseline and token dropping. Overall, token dropping consistently under-performs the baseline with an average 1.91% performance drop, among all these semanticintense tasks. Specifically, as for SICK-R (usually used to measure the semantic textual similarity), token dropping performs much worse (up to ↓2.92) than the baseline. These results indicate that, due to the semantic drift, BERT with token dropping falls short in handling the semantic-intense tasks.

heading: 3 Improving Token Dropping with Semantic-Consistent Learning
text: Based on the observations in §2, we recognize that it is essential to alleviate the side effect (i.e., semantic loss problem) of token dropping. To achieve this goal, we propose a simple yet effective semantic-consistent learning (SCTD) framework Specifically, our SCTD adopts two key techniques as follows:
Semantic-Consistent Learning. The principle of our SCTD is to encourage the model to preserve the semantic information in the representation space. Inspired by the success of knowledge distillation (Hinton et al., 2015; Xu et al.,
7We chose tasks based on whether they require semanticrelated information to solve. For instance, we included MRPC (Dolan and Brockett, 2005), a task that predicts if two sentences are semantically equivalent.
2020), SCTD refers to the model with baseline training (containing more semantic information) as the teacher to guide the training of students (i.e., model trained with token dropping). Considering that it is usually unavailable to obtain the pre-trained teacher model, we hereby recast it as a self-distillation process (Zhang and Sabuncu, 2020; Ding et al., 2021b). Given the same input X0, we input X0 into the model to perform twice forwardpropagation processes, where one is for token dropping and the other is for baseline training. The outputs of baseline training (Xl) are used as the teacher distributions to teach the student (outputs X̃l of token dropping). As such, the student can learn how to align the semantic information with the teacher. More specifically, SCTD introduces two semantic constraints in a local-to-global manner (as illustrated in Figure 2). For the global one, we use the KL divergence to constrain the globallevel semantic distributions of baseline- and tokendropping-based models at the last l-th layer, as follows:
LSCg = KL ( p(Xl)||p(X̃l) ) , (2)
where p(Xl) and p(X̃l) denote the corresponding distributions respectively. On the other hand, in slight of the finding that semantic loss is the most significant in the penultimate layer (l − 1) in token dropping setting (Figure 4), we further construct a local-level semantic constraint at the (l − 1)-th layer, which is similar to Eq. 2:
LSCl = KL ( p(Xl−1)||p(X̃l−1) ) . (3)
Hybrid Training. Since the semantic-consistent learning process requires twice forward/back-
propagation, SCTD would introduce much computational overhead, leading to inefficiency. To overcome this issue, SCTD adopts a novel hybrid training strategy, as illustrated in Figure 5. Specifically, instead of using the semantic-consistent learning method throughout the training, SCTD basically follows the vanilla token dropping and adopts the semantic-consistent training intermittently. As such, SCTD can combine the advantages of semantic-consistent learning (effectiveness) and token dropping (efficiency). Let Fi be a fixed interval, SCTD first performs the vanilla token dropping training (Fi− 1) times and then performs once the semantic-consistent training. The overall training objective of SCTD can be formulated as:
Lall =    1 2 L∗MLM + 1 2 LMLM + λ ∗ (LSCg + LSCl) , t mod Fi = 0
L∗MLM , t mod Fi ̸= 0 (4)
where t denotes the index of training iterations and λ is a weight factor to balance the different objectives, which is empirically8 set as 0.05.

heading: 4 Evaluation

heading: 4.1 Setup
text: Downstream Tasks To investigate the effectiveness and universality of SCTD, we follow many previous studies (Zhong et al., 2022b,d) and conduct extensive experiments on various NLU tasks, covering a diversity of tasks from GLUE (Wang et al., 2018), SuperGLUE (Wang et al., 2019) and SQuAD benchmarks. Specifically, three semanticintense tasks (MRPC (Dolan and Brockett, 2005), STS-B (Cer et al., 2017) and RTE (Giampiccolo et al., 2007)), five question answering tasks (BoolQ (Clark et al., 2019a), COPA (Roemmele et al., 2011), MultiRC (Khashabi et al., 2018), SQuAD-v1 (Rajpurkar et al., 2016) and -v2 (Rajpurkar et al., 2018)), two natural language inference tasks (MNLI (Williams et al., 2018) and CB (De Marneffe et al., 2019)), and two others (CoLA (Warstadt et al., 2019) and SST-2 (Socher et al., 2013)) are used. For evaluation, we report the performance with Accuracy (“Acc.”) metric for most tasks, except the Pearson and Spearman correlation (“Pear./Spea.”) for STS-B, the Matthew
8The detailed analysis can be found in §4.3.
correlation (“Mcc.”) for CoLA, the F1 score for MultiRC, and the Exact Match (“EM”) scores for SQuAD v1/v2. We report the averaged results over 5 random seeds to avoid stochasticity. The details of all tasks and datasets are shown in Appendix A.1.
Hyper-parameters For pretraining, we train the BRET-BASE and -LARGE models with different methods9 from scratch. We basically follow the original paper (Devlin et al., 2019) (e.g., the same pretraining corpus), except that we do not use the next sentence prediction (NSP) objective, as suggested in (Liu et al., 2019). In practice, we train each model for 250K steps, with a batch size of 1024 and a peak learning rate of 2e-4. For finetuning, the learning rate is selected in {1e-5, 2e-5, 3e-5, 5e-5}, while the batch size is in {12, 16, 32} depending on tasks. The maximum length of input sentence is 384 for SQuAD v1/v2 and 256/512 for other tasks. The detailed hyper-parameters for fine-tuning are provided in Appendix A.2. We use AdamW (Loshchilov and Hutter, 2018) as the optimizer for both pretraining and fine-tuning processes. All experiments are conducted on NVIDIA A100 (40GB) GPUs.

heading: 4.2 Compared Results
text: Results of GLUE are shown in Table 2, while those of SuperGLUE and SQuAD are in Table 3. Based
9Following Hou et al. (2022), we implement the token dropping and our approach under the same settings, e.g., dropping 50% of the tokens.
on these results, we can find that:
SCTD consistently improves performance on all types of tasks. First, results on the semanticintense tasks (MRPC, STS-B and RTE) show that SCTD effectively alleviates the semantic loss problem of token dropping. Specifically, for the RTE task, SCTD brings significant improvement (up to +3.4%) against the vanilla token dropping, and even outperforms the full-sequence training baseline. On the other hand, we observe that SCTD is also beneficial to the other general tasks (e.g., question answering). With the help of SCTD, token dropping strategy achieves up to +1.56% average gains among all types of tasks, proving the effectiveness and universality of SCTD.
SCTD improves performance on both model sizes. Extensive results show that SCTD works well on both Large and Base BERT models. Specifically, compared to the vanilla token dropping, SCTD brings +1.59% and +1.37% average gains on GLUE tasks, respectively. Results on the other tasks also show a similar phenomenon. Thus, we could recommend our SCTD to speed up the training of all discriminative MLMs regardless of the regime in model capacity.
SCTD effectively improves the training efficiency. Results in Table 2 show that, with our SCTD, BERT models can achieve comparable or even better performance with much fewer training steps, i.e., improving the training efficiency10. Specifically, compared to the full training (250K steps) BERT models, SCTD can save up to 48% pretraining time while achieving comparable performance. We attribute it to the higher data efficiency, since SCTD not only takes full advantage of the token dropping’s ability to learn important words but also alleviates the semantic loss problem in the token dropping. This can be further proved by the illustration of Figure 6, as SCTD always shows better performance against the other counterparts during the training dynamics. Furthermore, when training with the same iterations, our SCTD can even outperform the standard BERT by a clear margin. We attribute this to the regularization effect
10While the semantic-consistent learning process in SCTD will introduce extra computation overhead, SCTD performs much better in terms of training efficiency. That is, the relatively little computation overhead is acceptable.
of token dropping11.

heading: 4.3 Ablation Study
text: We evaluate the impact of each component of our SCTD, including i) semantic-consistent learning objectives, ii) coefficient λ and iii) fixed interval Fi in the hybrid training process. Notably, due to the limited computational budget, we conduct experiments on the BERTlarge models trained with different methods for 5 epochs (35K steps).
Impact of different training objectives. As shown in §3, in addition to the original MLM objective L∗MLM of token dropping, we introduce several extra training objectives (LMLM ,LSCl ,LSCg}) to align the semantic information. Here, we conduct experiments to analyze the impact of different objectives and show the results in Table 4. It can be seen that all objectives are beneficial to our SCTD, where the LSCg is the most helpful. This indicates the semantic alignment in the global-level representation space is more critical. Also, we can observe that the combination of all objectives performs best, thus leaving as the default setting.
Impact of Coefficient λ. The factor λ in Eq. 4, which is used to balance different objectives, is an important hyper-parameters. In this study, we analyze its influence by evaluating the performance with different λ spanning {0, 0.01, 0.05, 0.25, 0.5} on several GLUE tasks. Figure 7 illustrates the average results. Compared with the baseline, our SCTD consistently brings improvements across all ratios
11BERT-style PLMs are often over-parameterized and prone to overfitting. Using regularization methods like token dropping and LayerDrop (Fan et al., 2020) during training can improve model generalization and even boost performance.
of λ, basically indicating that the performance of SCTD is not sensitive to λ. More specifically, the case of λ = 0.05 performs best, and we thereby use this setting in our experiments.
Impact of Fixed Interval Fi. In our SCTD, we use a fixed interval Fi to control the frequency for performing the semantic-align process. To verify its impact, we evaluate the performance of SCTD on different Fi and show the results in Table 5. Observably, too small Fi not only causes much computational overhead, but also affects the stability of hybrid training, thus leading to sub-optimal performance. On the contrary, for the larger Fi (e.g., 50), it may be difficult to make full use of the semantic-consistent learning process, hindering the effect of SCTD. In the case of Fi = 10, SCTD achieves a better trade-off between costs and performance, which we suggest as the best setting12.
12Some readers may wonder why the teacher (i.e., model with baseline training) trained with only 1/Fi steps is strong enough to guide the training of student model. One possible reason for this question is that training with hard-to-learn tokens (Fi-1) times and training with easy-to-learn tokens once is sufficient to obtain remarkable teacher models, similar to the Lookahead Optimizer (Zhang et al., 2019), which updates fast weights k times before updating slow weights once.

heading: 4.4 Does SCTD indeed alleviate the semantic loss problem?
text: Here, we examine whether SCTD can alleviate the limitation of token dropping. Specifically, following the preliminary analyses in §2, we compare our SCTD with other counterparts by probing the trained BERT models (as illustrated in Figure 8) and pertinently evaluating on several semanticintense tasks (as shown in Table 6).
It can be found that, with our SCTD, BERT learns more semantic information among most layers, especially in dropped layers. Also, SCTD brings consistent and significant performance gains on all semantic-intense tasks against the vanilla token dropping. These results can prove that SCTD is beneficial to address the semantic loss problem.

heading: 5 Related Works
text: Pretraining with Transformer-based architectures like BERT (Devlin et al., 2019) has achieved great success in a variety of NLP tasks (Devlin et al., 2019; Liu et al., 2019; He et al., 2020; Joshi et al., 2020). Despite its success, BERT-style pretraining usually suffers from unbearable computational expenses (Jiao et al., 2020; Zhang and He, 2020). To this end, several training-efficient approaches are proposed to speed up the pretraining and reduce the computational overhead, such as mixed-precision training (Shoeybi et al., 2019), distributed training (You et al., 2019), curriculum learning (Nagatsuka et al., 2021; Ding et al., 2021a) and designing
efficient model architectures and optimizers (Gong et al., 2019; Clark et al., 2019b; Zhang and He, 2020; Zhang et al., 2023; Zhong et al., 2022c; Sun et al., 2023). These works mainly focus on efficient optimization processes or model architecture changes.
More recently, Hou et al. (2022) propose the token dropping strategy, which exposes a new mode to speed up the BERT pretraining. Without modifying the original BERT architecture or training setting, token dropping is inspired by the dynamic halting algorithm (Dehghani et al., 2018) and attempts to skip the computations on part of (unimportant) tokens in some middle BERT layers during the forward-propagation process. Owing to its impressive efficiency, token dropping has recently attracted increasing attention (Yao et al., 2022; Chiang et al., 2022). For instance, Yao et al. (2022) apply the token dropping strategy to broader applications, e.g., both NLP and CV communities.
Along with the line of token dropping, we take a further step by exploring and addressing its limitations. To be specific, we first reveal the semantic loss problem (§2) in the token dropping, and then propose a novel semantic-consistent learning method (§3) to alleviate this problem and further improve performance and training efficiency.

heading: 6 Conclusion
text: In this paper, we reveal and address the limitation of token dropping in accelerating language model training. Based on a series of preliminary analyses, we find that removing parts of tokens would lead to a semantic loss problem, which causes vulnerable and unstable training. Furthermore, experiments show such a semantic loss will hinder the performance of token dropping in most semanticintense scenarios. To address this limitation, we improve token dropping with a novel semanticconsistent learning algorithm. It designs two semantic constraints to encourage models to preserve semantic information. Experiments show that our approach consistently and significantly improves downstream performance across all task types and model architectures. In-depth analyses prove that our approach indeed alleviates the problem, and further improves training efficiency.
In future work, we will explore the effectiveness of our method on more advanced discriminative language models (He et al., 2020; Zhong et al., 2023b). Also, it will be interesting to revisit and
address the semantic loss problem in efficient training methods for generative language models (such as GPT3 (Brown et al., 2020)).
Limitations
Our work has several potential limitations. First, given the limited computational budget, we only validate our SCTD on the Large and Base sizes of BERT models. It will be more convincing if scaling up to the larger model size and applying SCTD to more cutting-edge model architectures. On the other hand, besides the downstream performance, we believe that there are still other properties, e.g., generalization and robustness, of MLMs that can be improved by our SCTD approach, which are not fully explored in this work.
Ethics and Reproducibility Statements
Ethics We take ethical considerations very seriously, and strictly adhere to the ACL Ethics Policy. This paper proposes a semantic-consistent algorithm to improve the existing token dropping strategy. The proposed approach aims to speed up the pretraining of BERT-style models, instead of encouraging them to learn privacy knowledge that may cause the ethical problem. Moreover, all pretraining datasets used in this paper are publicly available and have been widely adopted by researchers. Thus, we believe that this research will not pose ethical issues.
Reproducibility We will publicly release our code in https://github.com/WHU-ZQH/ScTD and the pretrained models in https: //huggingface.co/bert-sctd-base to help reproduce the experimental results of this paper.

heading: Acknowledgements
text: We are grateful to the anonymous reviewers and the area chair for their insightful comments and suggestions. This work was supported in part by the National Natural Science Foundation of China under Grants 62225113 and 62076186, and in part by the Science and Technology Major Project of Hubei Province (Next-Generation AI Technologies) under Grant 2019AEA170. Xuebo Liu was supported by Shenzhen Science and Technology Program (Grant No. RCBS20221008093121053). The numerical calculations in this paper have been done on the supercomputing system in the Supercomputing Center of Wuhan University.

heading: A Appendix
text: A.1 Details of Tasks and Datasets In this work, we conduct extensive experiments on parts of tasks from GLUE and SuperGLUE. In addition, two widely-used commonsense question answering tasks are also used. Here, we introduce the descriptions of the used tasks and datasets in detail. Firstly, we present the statistics of all datasets in Table 7. Then, each task is described as:
CoLA Corpus of Linguistic Acceptability (Warstadt et al., 2019) is a binary singlesentence classification task to determine whether a given sentence is linguistically “acceptable”.
MRPC Microsoft Research Paraphrase Corpus (Dolan and Brockett, 2005) is a task to predict whether two sentences are semantically equivalent.
STS-B Semantic Textual Similarity (Cer et al., 2017) is a task to predict how similar two sentences are on a 1-5 scale in terms of semantic meaning.
RTE Recognizing Textual Entailment (Giampiccolo et al., 2007), given a premise and a hypothesis, is a task to predict whether the premise entails the hypothesis.
MNLI The Multi-Genre Natural Language Inference Corpus (Williams et al., 2018) is a task to predict whether the premise entails the hypothesis, contradicts the hypothesis, or neither, given a premise sentence and a hypothesis sentence.
SST-2 The Stanford Sentiment Treebank (Socher et al., 2013) is a binary classification task to predict the sentiment of a given sentence.
CB CommitmentBank (De Marneffe et al., 2019) is a task that can be framed as three-class textual entailment on a corpus of 1,200 naturally occurring discourses.
BoolQ Boolean Question (Clark et al., 2019a) is a question answering task where each sample consists of a short passage and a yes/no question about the passage.
MultiRC Multi-Sentence Reading Comprehension (Khashabi et al., 2018) is a QA task where
each example consists of a context paragraph, a question about that paragraph, and a list of possible answers. The model need to predict which answers are true and which are false.
COPA Choice of Plausible Alternatives(Roemmele et al., 2011) is a causal reasoning task in which a system is given a premise sentence and must determine either the cause or effect of the premise from two possible choices.
SQuAD v1 The Stanford Question Answering Dataset (Rajpurkar et al., 2016) is a popular reading comprehension benchmark, where the answer to each question is a segment of text from the corresponding reading passage.
SQuAD v2 The latest version of the Stanford Question Answering Dataset (Rajpurkar et al., 2018) is one of the most widely-used reading comprehension benchmarks that require the systems to acquire knowledge reasoning ability.
A.2 Hyper-parameters of Fine-tuning For fine-tuning, we use the BERT models as the backbone PLMs and conduct experiments using the open-source toolkit fairseq13 and transformers14. Notably, we apply the same hyper-parameters to all PLMs for simplicity. The training epochs/steps, batch size, and learning rate for each downstream task are listed in Table 7.
13https://github.com/facebookresearch/fairseq 14https://github.com/huggingface/transformers
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4
3 B1. Did you cite the creators of artifacts you used? Section 4
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4 and Appendix A1
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we will explore the effectiveness of our method on more advanced discriminative language models (He et al. , 2020; Zhong et al. , 2023b). Also, it will be interesting to revisit and
address the semantic loss problem in efficient training methods for generative language models (such as GPT3 (Brown et al. , 2020)). Limitations
Our work has several potential limitations. First, given the limited computational budget, we only validate our SCTD on the Large and Base sizes of BERT models. It will be more convincing if scaling up to the larger model size and applying SCTD to more cutting-edge model architectures. On the other hand, besides the downstream performance, we believe that there are still other properties, e. g. , generalization and robustness, of MLMs that can be improved by our SCTD approach, which are not fully explored in this work