Fast-ELECTRA for Efficient Pre-training
Authors: Chengyu Dong
Liyuan Liu
Hao Cheng
Jingbo Shang
Jianfeng Gao
Xiaodong Liu
Venue: ICLR 2024 poster (2024)
Keywords: Language model Pre-training
ELECTRA
Efficiency
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: ELECTRA pre-trains language models by detecting tokens in a sequence that have been replaced by an auxiliary model. Although ELECTRA offers a significant boost in efficiency, its potential is constrained by the training cost brought by the auxiliary model. Notably, this model, which is jointly trained with the main model, only serves to assist the training of the main model and is discarded post-training. This results in a substantial amount of training cost being expended in vain. To mitigate this issue, we propose Fast-ELECTRA, which leverages an existing language model as the auxiliary model. To construct a learning curriculum for the main model, we smooth its output distribution via temperature scaling following a descending schedule. Our approach rivals the performance of state-of-the-art ELECTRA-style pre-training methods, while significantly eliminating the computation and memory cost brought by the joint training of the auxiliary model. Our method also reduces the sensitivity to hyper-parameters and enhances the pre-training stability.

Published as a conference paper at ICLR 2024
FAST-ELECTRA FOR EFFICIENT PRE-TRAINING
Chengyu Dong" +f Liyuan Liu Hao Cheng² Jingbo Shang' Jianfeng Gao Xiaodong Liu?
Uniiversity of California, San Diego *Microsoft Research (cdong, jshang) @ucsd.
(lucliu, chehao, jfgao, xiaodl] emicrosoft.com
ABSTRACT
ELECTRA pre-trains language models by detecting tokens in sequence that have been replaced by an auxiliary model. Although ELECTRA offers a significant boost in efficiency, its potential constrained by the training cost brought by the auxiliary model. Notably, this model, which is jointly trained with the main model, only serves assist the training of the main model and discarded post- training. This results in a substantial amount of training cost being expended in vain. To mitigate this issue, we propose Fast-ELECTRA, which leverages an existing language model as the auxiliary model. To construct learning curriculum for the main model, we smooth its output distribution via temperature scaling following descending schedule. Our approach rivals the performance of state- of-the-art ELECTRA-styl pre-training methods, while significantly climinating the computation and memory cost brought by the joint training of the auxiliary model. Our method also reduces the sensitivity to the pre-training stability.
and enhances
INTRODUCTION
ELECTRA (Clark al. 2020) is a pre-training method that trains the models to predict whether each token is the original token or synthetically generated replacement, in a corrupted input sequence. The token replacement is sampled from a distribution of possible tokens given the context, which is typically the output distribution of masked language model. Henceforth, we will refer to this masked language model as the auxiliary model D This pre-training task, known as replaced token detection (RTD), has shown great advantages in training and data efficiency compared to other pre- pre-training tasks such as masked language modeling (MLM) (Devlin et al.) 2019). ELECTRA- style pre-training and its variations have been increasingly popular in advancing natural language understanding capabilities (Meng et al. 2021 Chi et al. 2021 Meng et al. 2022 He et al. 2021
Bajaj et al. 2022).
Despite its effectiveness, one pitfall of ELECTRA that restricts its popularity is the design choices regarding how the auxiliary model is jointly trained with the main language model. This is originally intended to provide natural curriculum on the RTD task for the main model's learning since the auxiliary model will start off weak and gradually gets better, and thus progressively ramp up the difficulty of the token replacements through pre- pre-training (Clark et al., 2020). However, this design results substantial amount of resources (including computation and memory) being wasted, since the auxiliary model will be discarded after each training round. This issue becomes more severe considering that the training cost of the auxiliary model scales with the training cost of the main model, in terms of both the model size and training updates. This issue becomes even more severe considering the practical difficulty of balancing the auxiliary and main model's optimizations (Meng al. 2022). which often demands multiple training rounds to search for the optimal hyper- paramerr....rr.
In this work, we propose a simple ELECTRA-style pre-training alternative that can greatly alleviate this issue. In specific, employ existing language model the auxiliary model, which can either be retrieved from a public repository or from a previous training round. However, directly
Work was done during an internship Microsoft Research. Correspondence to: Chengyu Dong scdong@ucsd.ccmt uduu, Xiaodong Liu <xiandl @ microsoft.com In previous works CClar et al. 2202)), the auxiliary model is also referred to as the generator while the main model is referred to as the discriminator.

---

Published as a conference paper at ICLR 2024
pre pre-training with this existing language model impairs the performance, potentially because the auxiliary model generates token replacements that are too difficult. To reduce the difficulty of the token replacements, smooth the output distribution of the auxiliary model by temperature scaling and gradually decrease the temperature through pre-training following a pre-defined descending schedule. Such strategy effectively constructs curriculum for the main model's learning without the need to jointly train the auxiliary model.
We have developed a method called Fas--ELECTRA that further enhances the efficiency of ELECTRA, a method al- ready known for its competitive efficiency improvements. In addition, Fast- ELECCRL achieves comparable performance to state-of-the-art FLECTRA-style pre-training methods in
89-
MILI Acc. (%6
various pre-training settings. As shown in Figure T Fast-
87-
ELECTRA achieves the same downstream performance with a BERT-base equivalent model (Devlin et al. 2019 using consistently less training than original ELECTRA, and cut the overall training time by more than 50 hours from a total of 200 hours. Based on more extensive cal- culations and experiments, computation cost of the auxiliary model by 67% and the overall computation cost of ELECTRA by about 20-25%. Fast-ELECTRA can also reduce the memory cost of the auxiliary model by 97% and the overall memory cost by 10-20%. Furthermore, ELEC-EL reduces the sensitiv- ity of ELECTRA-style pre-training to the hyper-paramen choice, especially those responsible for the delicate balance between the optimizations of the main model and aux- iliary model. Fast- ELECTRA also improves the training stability of ELECTRA-style pre-training thus being more promising for scaling up to large language models.
Original
Fast ELECTRA
25
75
100
125
150
175
200
Wall training time (hours)
EECTEE can reduce the
Figure 1: Downstream task perfor- mance (MNLI accuracy, Avg m/mm) versus wall-clock training time for BERT-base equivalent models pre- trained with the "Original" ELECTRA design and our "Fast-ELECTRA" Training time is measured on node with 8 Tesla V100 GPUs. More ex- periment details can be found in Sec- tion 4.1
The rest of the paper will be organized as follows. In Section 2 we will briefly introduce the background of ELECTRA-style pre- pre-training and its efficiency advantage compared other pre- training methods. In Section (e we will motivate and present our method in detail. In Section 4we will conduct extensive experiments to test the effectiveness, efficiency, and robustness of our method. In Section we will demonstrate the necessity of the auxiliary model by comparing with model-free construction of RTD tasks, and the necessity of learning curriculum by comparing with pre-training with fixed auxiliary model.
5
2 PRELIMINARIES
Masked Language Modeling. The MLM task used in BERT (Devlin et al., 2019) trains the language model to predict randomly masked tokens in sequence. Specifically, given an input sequence æ = [2,, xn), MLM generates a masked sequence mmasked by randomly selecting few tokens at positions m [m,, m2, m] replace them with [mask]. The model is then trained to predict the original tokens at the masked positions. The training objective is
(M)
Ex
log po(x, (passel)
mm
where po(-li, kmasked) is the output distribution of the model 0 at position i, conditioned on the masked sequence Tmasked.
ELtCTRA-style Pre-training. Unlike MLM, ELECTRA uses RTD as the objective which pre-trains the language model detect replaced tokens in sequence. Specifically, given an input sequence æ, ELECTRA generates corrupted sequence Teomppeed by randomly selecting a few positions m and replace the token x; at each position E m with corresponding token 9; that is likely semantically similar but not necessarily the same. We will refer to å the replaced token,
Following previous works (He et al. 2021 Bajaj al. 2022 we use the results on MNLI Williams al.
2017 to indicate the performance on downstream tasks.
2

---

Published as a conference paper at ICLR 2024
which is sampled from probability distribution over the entire vocabulary. The language model is then trained to predict whether each token in Econupt is the original token or a replacement, namely
(O) Egee
[(0; + xz) po corrrp)) - 1(2; = x;) log(1 - -Po(i, #Fcorup)))
where pol Pecerppe) is the probability of replacement predicted by the model at position i, and 1(.) is the indicator function.
ELECTRA-style pre-training has greatly improved the training efficiency compared to MLM and has dominated the state- the-of-arts on natural language understanding benchmarks (He al. 2021 Meng et al. 2022: Bajaj et al. 2022). For example, ELECTRA can rival the downstream performance of RoBERTa (Liu et al. 2019b), competitive MLM-style pre-traue pre-training method, with only 25% of the computation cost (Clark et al., 2020).
Auxiliary Model and Joint-training. The pivot of the RTD task in ELECTRA-styLe pre-training the probability distribution which the replaced tokens are sampled from, which is typically determined by an auxiliary masked language model. Specifically, to generate the corrupted sequence Feoomppeed mentioned above, ELECTRA first replaces all the tokens at positions
with [mask] and obtain
masked sequence mmasked- The probability distribution of the replaced token at each position is then simply the corresponding output distribution of the auxiliary model evaluated on mmaseed namely f Paux( (1,,, Tmasked) for each E m.
In the original ELECTRA design, the auxiliary model is jointly trained with the main model, which is shown to be necessary for the effectiveness of the pre-training (Clark et al. (020). The overall training objective is defined ming [ ) ATRDD where Gaux refers to the parameters of the auxiliary model, and X is a hyper-parameter that balances the optimizations of the auxiliary model and the main model. The intuition here that joint training can provide a natural curriculum on the RTD task for the main model's 's learning since the difficulty of the replaced tokens will gradually increase, as the auxiliary model improves throughout pre-training
COMPUTATION OVERHEAD REDUCTION OF AUXILIARY MODEL
Computing and Memory Cost of the Auxiliary Model. Despite that joint training of the auxiliary model is effective. it results in a significant amount of computation resources being wasted since the auxiliary model will be discarded once the training is finished. Although the auxiliary model is typically smaller than the main model (about 1 1/4 1/3 of the depth (Bajaj et al. (0022)) its size has to be scaled with the main model (e.g., the same width (Bajaj et al. (0222) to maintain the difficulty of the RTD task relative to the capacity of the main model, which means more computation resources will be wasted in training larger language models.
Our estimations show that, in each training trial, in each training update, and for each input batch, the auxiliary model expends about 67% of the computation cost (about 4.0e11 1 FI.OPs) of the main model when pre- pre-training BERT-base equivalent model. The overall computation cost scales dramatically as one pre-trains with larger batch size, for more updates, and more The auxiliary model also consumes significant amount of memory during training, which is about 30% of the memory cost (about GB) of the main model when pre-training a BERT-base equivalent model. The excessive memory cost of ELECTRA could in induce more computation cost than necessary because only smaller batch sizes can fit into the memory.
searching rounds.
Fast- ELECTRA. We propose simple alternative significantly reduce the computation and memory cost of the auxiliary model inspired by simulated annealing (Kirkpatrick et al.) 1983). Specifically, we employ an existing language model as the auxiliary model, which can either be retrieved from a public repository or from a previous training experiment. To reduce the difficulty of the RTD task, we leverage temperature scaling to smooth the output distribution of this existing model. The replaced tokens in the RTD task are then sampled from its smoothed output distribution, namely
f. ~ Softmax (oog Paux (- &masked)
T
3

---

Published as a conference paper at ICLR 2024
To create learning currieulum for the main model similar to the effect of joint training, we simply schedule the temperature by an exponential decay function during pre- wre-training namely
T= (To 1) eep((uuT))
(2)
Here denotes the fraction of the training updates, while To and are two control the initial temperature and decay rate respectively. Since now the auxiliary model is not jointly trained, the training objective is simply ming (O))
that
The main advantage of Fast-ELECTRA is training efficiency as our auxiliary model is only used for inference during pre-training. The computation cost of the auxiliary model is now only 1/3 of the original, while the memory cost of the auxiliary model is now only about 1 1/30 of the original. With offline preprocessing. we can further reduce both the computation and memory cost of the auxiliary model during pre- wreiirainng to 0 (see more details in Section 4.3)
Our design also improves ELECTRA's robustness to the paper-parameter settings (see Section and its training stability (see Section
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Pre-training Setup. We conduct experiments with two standard settings, Base and Large, following
previous works (Devlin et al. 2019; Meng al. 202T Bajaj et al. 2022). Specifically, we employ Wikipedia and BookCorpus (Zhu et al. 2015) (16 GB of texts, 256M samples) for pre-training with a sequence length of 512. We use cased sentence piece BPE vocabulary of 128K tokens following (He) eet al. 2020), since larger vocabulary size improves LLMs without significant additional training and inference cost (Bao al. 2020). We conduct pre-training for 125K updates with batch size of 2048. For ELECTR,,, the auxiliary model is pre-trained following standard MLM style with a learning rate of 5c-4.
Model Architecture. Our main model (discriminator) in the Base setting follows the BERT base architecture (Devlin al. 2019), namely a 12-layer transformer with 768 hidden dimensions plus T5 relative position encoding (Rafeel et al.) 2019) with 32 bins. We employ Admin (Liu et al. 2020a 202T for model initialization stabilize the training. Our main model in the Large setting follows BERT Large namely a 24-layer transformer with 1024 hidden dimensions and 128 relative position encoding bins. We follow previous works (Clark al. 22200: [Bjjaj et al., (022) to set the size of the auxiliary model (generator), namely layers for the Base setting and 6 layers for the large setting.
More details of the model configuration are listed in Table 4
Downstream evaluation setup. We conduct evaluation downstream tasks following the setup in previous works (Meng et al. 2021 Bajaj et al.) 2022). Specifically, we evaluate on GLUE (Wang eet al. 2018) language understanding benchmark with single-task, single-mode model fine-tuning setting following previous works. We report Spearman correlation on STS-B, Matthews correlation on CoLA, and accuracy on the rest of the datasets. We follow the training hyperparameters suggested by Liu cet al. (2019a) 2020b), such as the use of an AdaMax optimizer (Kingma & Ba, 2015b). Detailed hyperparameter settings can be found in Appendix A.
Baselines. We compare our method with various baselines with an experiment setup same as ours, in terms of dataset, model size, and computation cost. We also incorporate baselines with similar experiment setup to allow more comparisons. For example, in the Large setting, we report baselines that pre-train for 1M updates but with bach size of 256, which aligns our setup in terms of the total number of processed tokens. We obtain results of these baselines from their papers and follow-up works, whichever are higher. We also reimplement METRO as our baselines. Both the re-implemented METRO (Bajaj et al. 2022) and method are implemented within the same codebase, which is built on top of FAIRSEQ (Ott et al. 2019), a popular open-sourced package.
Hyper-parameter Settings. We follow previous works (Clark et al. 2020 Bajaj et al. 2022 to select the generator size, namely 4 layers for the Base setting and 6 layers for the Large setting. For re-implemented METRO, set the loss weight as 70 and 50 for the Base and Large setting respectively while the learning rate as 5e-4 for both settings, which produces the best results in our experiments. For Fast-ELEECTR, we set the initial temperature 2, the decay rate as 0.1, and the
4

---

Published as conference paper at ICLR 2024
Table I: Results on GLUE development set. "-" indicates that public reports are available. "f" indicates the model pre-trained for 1M updates with batch size 256. "I" indicates the model pre-trained for 100K updates with batch size of 8K.
Model
MNLL-(m/mm) (Acc.)
QQP (Acc)
QNLI (Acc.)
SST-2 (Acc.)
CoLA (Mat. Corr.)
MRRC (Acc.)
STS-B (Spear. Corr.
Average Score
(Acc.)
Base Setting
BIRT (Deylin et al
2019
84.5/
91.3
91.7
93.2
58.9
68.6
87.3
89.5
83.1
RoBEKTa [Lir mi
2019h
85.8/85.5
91.3
92.0
93.7
60.1
68.2
87.3
88.5
83,3
DeBERTa He ml
2020
86.3/86.2
TUPE [Ke
86.2/86.2
91.3
92.2
93.3
63.6
73.6
89.9
89.2
84,9
ELECTKA ( E1 20020
86,9186.7
91.9
92.6
75.1
88.2
89.7
85,5
MC-BERT AUreui 2020
85.7/85.
89.7
91.3
92.3
62.1
75.0
86.0
$8.0
83.7
COCO-LM MMND-
88.5758.5
87.2
AMOS Meng el ml 2009
88.9/58.7
92.3
93.6
94.2
70.7
86.6
90.9
91.6
88.6
DcBERIaT (Hooc F02
89,3/89.0
METRO [Bujaj
METRO
89 89.0/88.9 9
92.0
93.4
94.4
70.1
86.3
91.4
91.2
83.5
Fast ELECTRA
89,4/88.8
92.1
93.8
94.5
71.4
85.6
91.4
91.6
88.7
Large Setting
ВERT
RoBERTa
89.0V
91.9
93.9
95.3
66.3
84.5
90.2
91.6
87.8
XLNet
TUPE!
88, 18.2/88.2
91.7
93.6
95.0
67.5
81.7
90.1
90.7
87.3
METROuarP
89.
89.2
Fas-BLECTRA
90.1/90.2
92.4
94.5
95.1
72.1
87.4
90.7
91.9
89.3
learning rate as le-3 for both the Base and Large settings. Detailed hhper-paramese settings are elaborated in Appendix A
4.2 DOWNSTREAM PERFORMANCE
Table lists the downstream evaluation results of Fast-ELECTRA and competitive baselines under the Base and Large setting. FaSt-ELECTRA matches previous state-of-the-arts with jointly-trained generator, in terms of the overall GLUE score and results on more reliable datasets such as MNLI.
4.3 TRAINING EFFICIENCY
Our approach can significantly improve the training efficiency of ELECTRA-style pre-training, in terms of both computation and memory cost.
Computation Cost. The computation cost of ELECTRA is contributed by both the main model and the auxiliary model. We assume the backward propagation has approximately twice the computation cost of the forward propagation following previous works (Kaplan et al. 2020). Therefore, the computation cost of the auxiliary model is reduced by 2/3 since it only used for inference in ELECTRA..
We estimate the computation cost more accurately by calculating training FLOPs for each input batch in each training update (i.e., one forward plus one backward propagation). We follow the formula introduced by (Hoffmann al. 2022) calculate the FLOPs for both the main model and the auxiliary model. As shown in Table 2 our method can reduce the overall computation cost by about 20-25% for both base and large models.
Memory cost. the auxiliary model. Considering standard training setup of language models such as the Adam optimizer (Kingma & BalBa202 2015a) and mixed mixed-precision training (Micikevicius et al. (017)). the memory cost of each trainable parameter contributed by its weight, its gradient, and its corresponding state buffers in the optimizer, which requires 20 bytes in total (Smith et al. (022). In contrast, for parameter that only used in inference, the memory cost consists of only weight, namely bytes. Therefore, the memory cost of the auxiliary model is significantly reduced since is only used for inference ELECTRA.
The memory cost of ELECTRA is contributed by both the main model and
Besides the model parameters, the intermediate activations of the computation graph stored for backward propagation consume significant memory as well. In typical implementations, one can employ checkpointing (Gruslys et al.] 2016 Chen et al., 2016) to trade computation for memory and reduce the memory footprint within each encoding layer. Nevertheless. the activations after each encoding layer still need to be stored (Smith et al. 2022 Therefore, the activation memory
5

---

Published as a conference paper at ICLR 2024
Table 2: Computation cost per batch per update and memory consumption of ELECTRA. Note that for the original ELECTRA, exclude the embedding layer from the memory calculation of the auxiliary model since it is shared between the auxiliary model and the main model. We also report the computation and memory cost realistic computation infrastructures in Appendix B.
Model
Method
Computation (GFLOP))
Memory (GB)
Main
Auxiliary
Total
Main
Auxiliary
Total
Original
591.9
398.6
990.5
23.0
7.0
30.0
Base
Fast-ELBCTRA
591.9
132.9
724.8
23.0
0.25
23.3
Ratio
1.0
0.33
0.73
1.0
0.01
0.77
Original
1407.7
653.9
2061.6
60.2
14.4
74.6
Large
Fast-Fl ECTRA
1407.7
218.0
1625.6
60.2
0.42
60.6
Ratio
1.0
0.33
0.79
1.0
0.03
0.81
scales with the number of layers, which means the auxiliary model always consumes about 1/4- 1/3 additional memory on top of the main model. The activation memory also scales with the sequence length, number of hidden dimensions, and batch sizes. In contrast, the auxiliary model in Fast-ELECTRA consumes intermediate activation memory since it is used only for inference.
We report the estimated memory cost in Table 2 Our method can reduce the memory cost of the auxiliary model by about 97% and the overall memory cost by more than 20% for both base and large models.
Offline preprocessing for auxiliary computation and memory cost. Last but not least, Fast- ELECTRA also makes the offline preprocessing technique feasible for ELECTRA-style pre-training, which can further reduce both the computation and memory cost of the auxiliary model during pre-training. In specific, corrupted input sequence and the binary target indicating whether each token is replaced or not) each epoch before pre-training, since the only varying parameter in auxiliary model is the temperature and can be determined by the epoch number. These dumped training data can be reused in subsequent pre-training rounds, for example, for hyper-parameter search continual pre-training. This strategy would be particularly preferred for large-scale pre-training.
can generate and dump the training data of the RTD task (i.e., the
Note that similar strategy used in MLM pre-training called static masking (Devlin al. 2019 Liu et al. 20196). But FEST-ELECTRA equipped with offlin preprocessing can be even more efficient than MLM pre-training with static masking since the training targets are binary instead of dependent on the vocabulary size.
4.4 ROBUSTNESS TO THE
SETTINGS
In this section, we investigate the robustness of our method to the hyper-parameter settings. We are mostly interested in hyper-pann parameters that control the learning curriculum, namely the difficulty of the RTD task throughout pre-training, since it the key difference between Fast-ELECTRA and the original ELECTRA and is also important to the pre-training's effectiveness.
In practice, we would strongly prefer an ELECCRA-style pre-training method that friendly to the learning curriculum tuning. This is because it is challenging to control the learning curriculum since a positive curriculum in the short term is not necessarily beneficial to the entire training course. Searching for the optimal curriculum often requires training till the end. Controlling the learning curriculum is particularly challenging for pre- pre-trannig because the effectiveness of a curriculum only be revealed by the performance representative downstream tasks, which requires time- consuming fine-tuning. There no reliable and instant signal in pre-training that can indicate the effect of curriculum.
Size of the auxiliary model. The size of the auxiliary model can directly affect the learning curriculum. A large auxiliary model can converge faster (Arora al. 2018). thus creating a more difficult RTD task for the main model. Previous works have observed that an auxiliary model that too large or small can both damage the effectiveness of the pre-training (Clark al. 2020). However, tuning the size of the auxiliary model in practice can be overly problematic, as the model
For the activation memory, we account for the entire batch size and ignore gradient accumulation here it depends on the specific GPU memory size.

---

Published as a conference paper at ICLR 2024
yy
(Fast ELECTRA
33
62
67
Original
0.60
3:50
Learinng rate
Learning
06.0
Training Loss
Modell)
70-4
5e-4
7044
MNU AC5
06-0
0.45
0.42
1e.
2
2.40
2.45
2M-3
08.6
3.35
0.23
0.25
Ausiliary Model Depth
Loss Weight lAl (Driginw)
0.22
25
30
100
0.20
125
Pro-training Updates
Pro-trainning Updates
Downstream task performance (MNLI accuracy, Avg m/mm) versus the depth of the auxiliary model (Left) and the hyper-parameters (Right) used in the "Orig- inal" ELECTRA or "Fast-ELECTRA..
Figure 2:
Figure 3: Training loss curves of the main model when pre-training with the original ELECTRA (Left) and Fast-ELECTRA (Right). "x" indicates the diver- gence of the training.
architecture changes in each training attempt, which means the experiment or hardware setup very likely needs to be re-configured maintain the training efficiency.
We show that compared to the original ELECTRA, Past-ELECTRA is more robust to the size of the auxiliary model, thus being more friendly use in practice. We experiment on the original ELECTRA and FaSt-ELECTRA with multiple auxiliary model depths. while for other hyper-parametent (e.g., loss weight in the original ELECTRA and decay rate in Fast-ELECTRA), we select the ones that produce the best performance for each depth respectively. As shown in Figure 2 left, Fast-ELECTRA can achieve more predictable performances as the auxiliary model depth varies.
Curriculum Schedule. We also experiment on hyper-parameters provided by each method that can directly control the learning curriculum when the auxiliary model is determined. In the original ELECTRA, it is mainly the loss weight À that controls the learning curriculum, since it balances the optimizations of the auxiliary model and the main model. We neglect the learning rate here since affects the auxiliary model and the main model simultaneously. In Fast-ELECTRA the decay rate used control the learning curriculum. We neglect the initial temperature here since the default value of 2 works well across different auxiliary model and main model settings.
To fairly compare the sensitivities of the performance to À and 7 given their different scales, we sweep each typer-parameter around two of its best values under different settings. For example, modulate [40, 50, 60, 70, 80], since 70 and 50 are the best values we found for the Base and Large settings respectively. We modulate in [0.05, 0.1, 0.3, 0.5, 0.7] since 0.1 and 0.5 are the best values we found for 4-layer auxiliary model and a 12-layer auxiliary model respectively.
Figure rright shows the downstream performances obtained by experimenting with these hyper- parameter values. It can be seen that the performance changes abruptly when varies in the original ELECTRA. In contrast, FEST-ELECTRA produces smoother performance curve varies.
4.5 TRAINING STABILITY
In this section, we study the training stability, which is known be major bottleneck in training large language models (Liu et al. (2020a). Upon scaling up the model size, may need reduce the learning rate, scale down variance of the weight initialization, apply heavy gradient clipping, or re-configure the model architecture (Bajaj et al. 2022, Smith et al.) (022). However, these remedies often sacrifice the efficiency and/or effectiveness of the pre-training (Bajaj et al. 2022).
Although we cannot afford large-scale experiments, make an initial attempt to inspect the training stability with a standard model size (BERT-base equivalent). In specific, we conduct pre-training with excessively large learning rates. Figure 3 shows that the original ELECTRA quickly diverges as the learning rate increases, while pre- training with EEEC-EEETTRA can remain stable with learning rate large as 2e-3, almost triple the maximum value allowed by the original method.
ABLATION STUDIES
5.1 ELECTRA-STYIA PRE-TRAINING FREE OF AUXILIARY MODELS
Here, we attempt to construct an RTD task without an auxiliary model. As mentioned in Section 2 the pivot of RTD task is the probability distribution which the replaced tokens sampled from, which
7

---

Published as a conference paper at ICLR 2024
will refer to the replaced token distribution. We experiment with the following alternatives to define the replaced token distribution without an auxiliary model.
Uniform: uniform distribution defined over the entire vocabulary.
Term frequency: a distribution where the probability mass of a token is equal to its frequency across the entire training corpus.
Smoothed one-hot: distribution where the probability mass of the correct token is 1 - a, while the probability mass of any other token is a/(V) 1). Here V indicates the vocabulary and 0.35
is the typical prediction enror rate of an auxiliary model on the masked tokens.
Figure visualizes the above distributions for an example se- quence Note that the term frequency approximately follows
10-1
Zipf's 's law (Newman 2004). We also plot the replaced token distribution produced by an auxiliary model for reference.
Probability Mass
20-3
10-4
We also include learning curriculums defined by these distribu- tions by interpolation, as intuitively smoothed one-hot is more difficult than other distributions since can be viewed as an auxiliary model that always predicts the original token correctly with high confidence (See more details Appendix BB.3).
20:
Uniform
10-4
Term Frequency
Smoothed
Predictive
10-41
[SEP]
"the*
Yarge"
10-13
L.
10a
10-
10
10'
10"
10F
Frequency Rank
Table 3 shows the downstream performance when pre-training with these replaced token distributions 5 Pre-training with the uniform distribution diverges early around 10K updates. We suspect this is because the replaced tokens sampled from a uni- form distribution are too easy detect since most of the tokens would be rare and unique considering a vast vocabulary. This supported by our observation that the training loss of the dis- criminator plunges to 0 at the very beginning of the pre- vre-training, after which the training diverges. In contrast, interpolating the uniform distribution with the more difficult smoothed one-hot distribution can converge smoothly. In fact, such an interpolation also yields better downstream performance than smoothed one-hot alone, which can be attributed to the benefit of the learning curriculum.
Figure 4: bution of the replaced token in an example sequence defined by var- ious methods. The token classes are ranked by the term frequency in the æ-axis.
The probability distri-
Interestingly, pre-training with term frequency alone can converge and yield reasonable downstream performance (~~ 55.7% MNLI Acc.), whereas interpolating term frequency and smoothed one-hot leads to consistently worse downstream performance than term frequency alone. We suspect that term frequency is superior because it better indicates the difficulty of possible replaced tokens in context.
Finally, we note that pre-training with these auxiliary-modde model-frre replaced token distributions are still inferior to auxiliary-modee model-based ones in terms of the downstream performance upon convergence.
Table 3: Downstream task performance (MNLI accuracy, average m/mm) when pre- gre-training with different replaced tokens distributions.
Method Fast-ELECTRA
Uniform
Term Frequency
Smoothed One-hot
UNF - SOH
TF - SOH
MNLI Acc.
89.1
Diverged
85.7
83.8
85.3
84.9
5.2
DOES THE CURRICULUM MATTER FOR ELECTRA- PRE-TRAINING?
In this section, we investigate whether learning currieulum, namely a gradually more difficult RTD task, is necessary for ELECTRA pre-training. We focus the case when auxiliary model available, given the inferior performance model-free alternatives as mentioned above. The comparative experiment here is fixed-uxiliary pre-training, where the replaced token distribution is simply the output distribution of a pre-trained and fixed auxiliary model, without any modification.
'A quote from Clark et al. (0000). "most current training methods require [large] amounts of compute to be effective", where *7*7" indicates the token to replaced. 'We use UNF + SOH and TF - SOH to denote uniform-to-smoottee onc-hot interpolation and term frequency-to-smoonte one-hot interpolation respectively.
8

---

Published as a conference paper at ICLR 2024
Figure 5 (left) shows the downstream of pre- training with fixed auxiliary model. In our
59.0
68-
18.5
experiments, for more than 10K training up-
MNUI Acc. (%)
10.0
dates from the beginning. the main model's
57.5
replaced token detection accuracy remains Yet with more training updates it starts to con- verge and yield decent downstream performance (~~ 88.3% on MNLI).
57.0
26.5
26.0
FasT-ELECTIM
25.
40
102
120
12
Pre-training updates
Auxiliary Model Depth
Nevertheless, an appropriate curriculum greatly improves training efficiency. Compared to fixed- auxiliary training. temperature-scaling this same auxiliary model improves the accuracy from 80% to 85% on MNLI when the number of training updates is limited (e.g., 2K). An ap- propriate curriculum can also advance the down- stream performance upon convergence (88.4% vs. 89.1%). Finally, we found that learning curriculum is particularly important for auxiliary models with a large capacity. As shown Figure right, temperature-scaling the auxiliary model improves the MNLI accuracy for more than 3% compared to fixed-auxiliary training. This implies learning curriculum alleviates the sensitivity of the pre-training effectiveness to the auxiliary model's capacity.
Figure 5: (Left): Downstream performance (MNLI accuracy, Avg m/mm) at multiple intermediate checkpoints, obtained by pre-training with fixed auxiliary model and Fast-ELECTRA. (Right): Downstream performance at the last checkpoint versus the depth of the auxiliary model, obtained by pre-training with fixed auxiliary model and EEssTELECTRA.
In Appendix B.2 we further experiment on alternative ways such as Dropout to design a learning curriculum. We also experiment on alternative functions such as stepwise decay schedule the curriculum. These alternatives can yield decent downstream performance, albeit slightly worse than the default settings in ELECTR...
6 RELATED WORK
Variations of ELECTRA-style Pre-training. Here we briefly summarize the variations of the ELECTRA-style pre-training in the literature. Xu et al. (2020) pre-trains the model to predict the original token from small candidate set, instead predicting a binary target. [Meng et al. (2021) introduces two additional training objectives including the prediction of the original token and alignment between corrupted sequences from the same source. [ao et al. (2021) Iearns to sample more difficult replace tokens. Meng et al. (2022) automatically constructs difficult lcarning signal by an adversarial mixture of multiple auxiliary models. He et al. (2021) argues that embedding sharing between the main model and the auxiliary model may hurt pre-training and proposes stop gradient operation during back propppppppaaaion. [Bjjaj et al. (2022) conducts a comprehensive ablation study on ELECTRA and highlights several important improvements such large vocabulary size and relative position embedding, and successfully scales ELECTRA-stye pre- pre-traini training up to billions of parameters. Zhang al.] (2022) observes the existence of "false negative" replaced tokens, namely those that are not exactly the same but are synonyms to the original ones, and proposes to correct them by synonym look-up and token similarity regularization.
7 CONCLUSION
In this work, we focus on the training cost of the auxiliary model in ELECTRA-style pre-training and propose a simple method that employs existing language models and annealed temperature scaling to greatly alleviate the issue. Our method achieves comparable performance to state-of oate-of-the-ase the-art while being more efficient and robust. In general, our approach empowers ELECTRA-style pre- pre-training with more flexibility, opening up potential applications continual learning, transfer learning, and knowledge distillation for language models.

---

Published as a conference paper at ICLR 2024
REFERENCES
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. ArXiv, abs/18 10.00228, 2018.
Payal Bajaj. Chenyan Xiong, Guolin Ke, Xiaodong Liu. Di He, Saurabh Tiwary. Tie- Yan Liu, Paul Bennett, Xia Song, and Jianfeng Gao. Metro: Efficient denoising pretraining of large scale autoencoding language models with model generated signals. ArXiv, abs/2204.066566 066644 2022.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang. Nan Yang, Xiaodong Liu. Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unilmv2: Pseudo-masked language models for unified language model pre-training. ArXiv, abs/2002. 11020002.11804 12804, 2020.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. ArXiv, hass/604.061 74, 2016.
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, and Furu Wei. XIm-e: Cross-lingual language model pre-training via electra. In Annual Meeting of the Association for Compulational Linguistics, 2021.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Blectra: Pre-training text encoders as discriminators rather than generators. ArXiv, abs/2003. 10555, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019.
Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Mare Lanctot, and Alex Graves. Memory-efficient backpropagation through time. In NIPS, 2016.
Yaru Hao, Dong, Hangho Bao, Ke Xu, and Furu Wei. Learning to sample replacements for electra pre-training. In FINDINGS, 2021.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. ArXiv, abs/2006.03654, 2020.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with disieentddisntanlledd embedding sharing. ArXiv, abs/2111.09543 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. Training compute-optimal large language models. ArXiv, abs/2203. 15556, 2022.
Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, 0as/200100836/) 2020.
Guolin Ke, Di He, and Tie- Yan Liu. Rethinking positional encoding in language pre-training. ArXiv, abs/2006. 15595, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015a.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2015b.
Scott Kirkpatrick, Charles D. Gelatt, and Michelle Vecchi. Optimization by simulated annealing. Science, 220:671 - 680, 1983.
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. ArXiv, abs/2004.08249, 2020a.
Liyuan Liu, Jialu Liu, and Jiawei Han. Multi-head or single-head? an empirical comparison for transformer training. ArXiv, abs/2106.09651 2021.
10

---

Published as a conference paper at ICLR 2024
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Anmual Meeting of the Association for Computational Linguistics, 2019a.
Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, et al. The microsoft toolkit of multi-task deep neural networks for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 1 118-126, 2020b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeutlemoeer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907. 1692, 2019b.
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. Coco-Im: Correcting and contrasting text sequences for language model pretraining. ArXiv, abs/2102.08473, 2021.
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. Pre- training text encoders with adversarial mixture of training signal generators. ArXiv, abs/2204.03243, 2022.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Frederick Diamos, Erich Elsen, David Gareía, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. ArXiv, abs/1710.03740,, 2017.
Mark E. J. Newman. Power laws, pareto distributions and zipf's law. Contemporary Physics, 46:323 351, 2004.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In North American Chapter of the Association for Computational Linguistics, 2019.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena. Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, abs/1910. 10683, 2019.
Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Anand Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. ArXiv, abs/2201. 2022.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNI.P@G EMNLP, 2018.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computational Linguistics, 2017.
Zhenhui Xu, Linyuan Gong, Guolin Ke, Di He, Shuxin Zheng, Liwei Wang. Jiang Bian, and Tie- Yan
Liu. Me-bert: Efficient language pre-training via a meta controller. ArXiv, abs/2006.05744, 2020.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe Le.
Xinet: Generalized autoregressive pretraining for language understanding. In Neural Information Processing Systems, 2019.
Zhuosheng Zhang, Hai Zhao, Masao Utiyama, and Eiichiro Sumita. Language model pre-training on true negatives. ArXiv, abs/2212.00468 2022.
11

---

Published as a conference paper at ICLR 2024
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 19-27, 2015.
12

---

Published as a conference paper at ICLR 2024
A HYPER-PARAMETER SETTINGS
We follow the standard practice in previous works set the hyper-parametent (Devlin al. 2019 Meng et al. |021: Bajaj et al. 2022). For MLM pre-training of the generator, we fix the mask ratio 15%. When sampling sequences for pre-training. we respect document boundaries and avoid concatenating texts from different documents. We did not mask special tokens follow the standard BERT practice. We conduct pre-traintn tre-training on NVIDIA Tesla V100 with 32GB memory and fine-tuning on NVIDIA Tesla P100 with 16GB memory. Table 5 lists the detailed prper-parameter settings for pre-training- Table lists the detailed nyper-parametee used for fine-tuning.
Table Configuration of model architectures. Note that the auxiliary model has the same configu- ration in each encoding layer as the main model, albeit having fewer layers. As reference for the calculation of memory cost, we list the embedding layer separately since it is shared between the main model and the auxiliary model in the original ELECTRA design.
Depth (Main)
Depth (Aux)
Hidden Size
FFN Width
Attention Heads
Params (Main)
# Params (Aux)
Params (Embed)
Model
Base Large
12
768
3072
127M 208M
98M
24
6
1024
4096
16
434 M
131M
Table 5: Hyperparameter settings for pre-training.
Hyperparameters
Base
Large
Max Steps
125K
125K
Optimizer
Adam
Adam
Peak Learning Rate (Fast-ELECTRA)
10 3
x 10 3
Peak Learning Rate (METRORdmm))
5 10
10
Loss Weight (METROReemp)
70
50
Batch Size
2048
2048
Warm-Up Steps
10K
10K
Sequence Length
512
512
Vocabulary Size
128K
128K
Relative Position Encoding Buckets
32
128
Relative Position Encoding Max Distance
128
256
Adam é
1c-6
1c-6
Adam (B1, 92)
(0.9, 0.98)
(0.9, 0.98)
Clip Norm
2.0
2.0
Dropout
0.1
0.1
Weight Decay
0.01
0.01
B ADDITIONAL EXPERIMENTS
B. TRAINING EFFICIENCY
We test the overall computation cost and memory cost on specific computation infrastructures, as hardware-centered optimization can be critical to training efficiency (Rasley et al. 2020). We conduct experiments on two typical infrastructures. including node with 4x GeForee RTX 3090 GPUs (24GB memory each, w/o NVLink), and node with 8x Tesla V100 GPUs (32GB memory each, w/o NVLink). We measure the computation cost by the wall time (in seconds) per training update (SPU), and the memory cost by the peak memory occupied by all tensors on one GPU throughout training, averaged over all GPUs.
As shown in Table 7 EEss-ELETTRA can reduce the overall computation and memory cost of EL ELECTRA--tyLE pre-training across different computation infrastructures and model settings consis-
13

---

Published as a conference paper at ICLR 2024
Table Hyperparameter search space in fine-tuning
Hyperparameters
Base
Large
Sequence Length
256
256
Optimizer
AdaMax
AdaMax
Peak Learning Rate
(5e-5,1e-4, 3e-4]
(5e-5,1e-4, 3e-4]
Max Epochs
(2.3,5,10)
(2.3,5.10)
Batch size
(16, 32, 64)
(16, 32, 64) Linear (0. 0.01)
Learning rate decay
Linear (0, 0.01)
Weight Decay
Warm-up Proportion
%, 10 %)
[6 %. 10 %]
Adam
lc-6
le-6
Adam (31.
(0.9, 0.98)
(0.9, 0.98)
Gradient Clipping
1.0
1.0
Dropout
0.1
0.1
Table 7: Computation cost and memory consumption of ELECTRA measured on specific infrastruc-
tures
4x RTX 3090
8x Tesla V100
Model
Method
Computation (SPU)
Memory (GB)
Computation (SPU)
Memory (GB)
Original
10.0
13.7
5.6
13.7
Basc
EEETTELECTRA
11.6
4.0
11.4
Ratio
0.77
0.84
0.71
0.83
Original
13.8
19.1
6.7
19.1
Large
ELECTEEETTA
11.2
16.2
5.3
16.2
Ratio
0.85
0.79
0.84
The reduction of computation cost matches our calculation in Section 4.3 while the reduction of memory cost slightly less than that from our calculation, which due to gradient accumulation that reduces the peak memory.
tently
B.2 ALTERNATIVE CURRICULUM DESIGNS FOR ELECTRA-STEE -TYLE PRE-TRAINING
In this section, we explore alternative ways to design the learning curriculum when an auxiliary model is available. In general, model-based curriculum can be determined by two functions, namely the schedule function and the augmentation function.
An augmentation function is used reduce the difficulty of the replaced token task generated by an existing auxiliary model. In ELECTRA we utilized temperature scaling as an augmentation function to smooth the output distribution of the auxiliary model. Yet, essentially any method that can change the model's output distribution can be utilized as an augmentation function. Possible methods include changing the model's output distribution directly, changing the behaviors of the model weights modules, or changing the model's input sequence. Here we consider the following alternative augmentation functions.
Augmentation function.
Logarithmic interpolation: Interpolate the auxiliary model's output distribution with an easy distribution such as a uniform distribution or term frequency. Since interpolating with uniform distribution in fact equivalent to temperature scaling we only consider the interpolation
"Note that Fast-ELEETTE can in fact support a larger batch size per GPU and thus potentially speed up training further, due to reduced memory cost. Nevertheless, in our tests, we set the batch size per GPU to be the same for the original ELECTRA and ELECTRL such that the computation reduction of ELECTRA involves no contribution
memory cost.
((1 (1-7)) log = -y log [V] ps. while temperature scaling can he formulated as log = (1/T) log Po- Therefore, these two will he the up constant difference, which will be canceled after applying Softmx the log probabilities.
'Logarithmic interpolation with uniform distribution is log == 7 log +UNF++(
(1 7) Po,
14

---

Published as a conference paper at ICLR 2024
with term frequency, namely Prr-oo-li, Fmasked) ptr @ Pe (-li, mmaked), where denotes clement-wise product
Dropout: Enable all activation dropout layers in the auxiliary model, and set all their drop rates as y.
Drop atlention: Enable all attention dropout layers in the auxiliary model, and set all their drop rates as 7.
Drop token: Randomly replace y fraction tokens (except special tokens such as " "[MASK"" ") in the auxiliary model's input sequence with " "[UNK]" namely the token representing unknown tokens.
Tigure@leet shows the downstream performance achieved by pre-training with these augmentation functions. Here for each augmentation,
use exponentially decayed function to schedule similar to Equation 2 and search its best hyper-parameters (i.e., "mmx and T) based on the final performance. One may find that all these augmentation functions achieve decent downstream performance. Nevertheless, in practice, we prefer temperature scaling as it is easy to implement and achieves slightly better performance.
Augmentation
MNLI Acc. (96
29.2
20.2
08.0-
28.0
80.8
88.6
MKLI
88.8
00.
Tamperature Scate
Dropost
Drop Baaon
Drap Attestin
Log intemplptiin
Expomential
Constant
Stapwise
Pelynomial
Figure 6: Downstream performance (MNLI accuracy, Avg m/mm) obtained by pre-training with different augmentation functions (Left) and different schedule functions (Right). The augmentation function and schedule function used in Fast-ELECTRA are highlighted.
Schedule function. A schedule function is used to determine the parameter of the augmentation function (e.g., 7) at the w-th fraction training updates, such that the difficulty of the replaced token detection task gradually increases through pre-training. In previous sections, have experimented on an exponential decay function. Here we experiment on alternative schedule functions as follows.
Constant: (u)) = 7o-
Polynomial decay: (u) Tmax - u)", where 7 here controls the decay rate and larger results in faster decay.
Stepwise decay: 7(u) = Timax (1 - [ur] //),, where [.] denotes the floor function and r determines
the number of decays.
Tigure/Jrrght shows the downstream performance achieved by temperature-scaling with its parameter scheduled by these functions. For each schedule function, we search for its best hyper-paramett based on the final performance. One find that the constant schedule leads to significantly lower performance than exponential decay, while other schedules can achieve comparable performance.
B.3 LEARNING CURRICULUM FREE OF AUXILIARY MODELS
We experiment on curriculum RTD tasks defined by replaced token distributions free of auxiliary mod- els. As mentioned in Section 5.1 the smoothed one-hot distribution is intuitively more difficult than the uniform distribution and term frequency. Therefore, we can interpolate these distributions with varied coefficient to gradually increase the difficulty of the RTD task during training. Concretely, we can define the replaced tokens distributions as
Interpolation between uniform and smoothed one-hot (UNF+SSHH)
PUNF-SOH := PiNF ? Pson-
'We choose logarithmically interpolate the distributions because empirically observed it is slightly better than linearly interpolating.
15

---

Published as a conference paper at ICLR 2024
Interpolation between term frequency and smoothed one-hot (TF-> SOH);
PIF-SOH PTF
Here denotes the glement-wiss multiplication, and 0 decayed function, similar to Equation 2 We choose to logarithmically interpolate the distributions because empirically observed it is slightly better than linearly interpolating.
and is scheduled by exponentially
16


Reviews:
[{"review_id": "OZp0QdPntA", "reviewer": "Reviewer_dT26", "summary": "This paper proposes efficient pretraining with ELECTRA by using a fixed generator (auxiliary model) instead of training the aux model together with the discriminator for replace token detection (RTD). In order to simulate a curriculum of difficulty that training of aux model provides, it uses an exponentially decaying schedule on the *temperature* used to sample from aux model.\n\nEfficiency: Since the aux model is fixed, Fast-ELECTRA saves on backward passes leading to overall 20-30% FLOPs per step. (This calculation ignores original training of the auxiliary model.) It also saves 20-25% memory in the aux model. One could also cache the aux model predictions and save a lot more FLOPs (30-40%).\n\nQuality: Fast-ELECTRA is competitive with many BERT and ELECTRA related baselines, even slightly better on some GLUE downstream evals\n\nRobustness: The paper evaluates robustness to curriculum schedule and discriminator model size, and finds that Fast-ELECTRA behaves more gracefully compared to ELECTRA and can handle higher learning rates.\n\nThe paper also performs interesting ablation studies with using some simple aux models and different (linear/poly) schedules for curriculum. Overall, the paper presents a conceptually interesting finding that that a fixed pretrained aux model can be used, with a temperature schedule. Incomplete comparisons to earlier ideas made it harder to judge novelty", "questions": "- In Eq 2 what is the range of $u$? What is the final temperature in that case? If $u$ is indeed fraction of training updates, then at the max value of $u=1$ the final temperature has a value different from 1. Is that intended?\n\n- Section 3 says “the auxiliary model expends about 67% of the computation cost” If aux model is just 1/3 the size, why does it contribute so much to computation cost?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novelty: The idea of using a fixed aux model for efficiency is interesting, and novel to my knowledge (although I'm not entirely sure since I could not find much discussion about this). Similarly the idea of using decaying temperature as a curriculum in this context is quite interesting\n\nQuality: The paper provides a nice analysis of computation and memory benefits of the method.\n\nClarity: The paper is easy to follow for most part. Connections to prior work and some other details could be presented better.", "weaknesses": "Comparison to prior work: \n\n- It would be helpful to highlight the most relevant work in Table 1 that a reader should focus in. Additionally, is there any evaluation on prior work that uses fixed generator? It would also help to include some FLOPs comparison to the baselines used in Table 1. The paper will also help with a discussion on accuracy-efficiency tradeoff. Lack of such discussions made it harder to assess the full value of proposed method.\n\n- Recent paper (Dong et al.) from ICML 2023 proposed a different strategy of decoupling generator and discriminator optimizers and has better GLUE metrics than Fast-ELECTRA. This does not dilute the contributions of this paper much because Fast-ELECTRA also leads to training speed up, and is conceptually different. However it will be helpful to include and compare to this method. It could be an interesting open question if the gap to Dong et al. can be reduced with a fixed aux model.\n\n\nMissing discussions: Most of the paper assumes the existence of a good pretrained aux model, but ignores the cost of training the aux model itself. Does Fast-ELECTRA truly reduce total FLOPs in that case?\n\n\nOverall I believe that the paper makes a positive contribution. I'm currently assigning a score of weak accept mainly due to above reservations about comparisons to prior work.\n\nDong et al. Understand and Modularize Generator Optimization in ELECTRA-style Pretraining. ICML 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SMkPx6LjLc", "reviewer": "Reviewer_rs11", "summary": "This paper proposes a simple and effective technique to improve electra training. By replacing the training of the auxiliary model with a pre-trained model together with temperature scaling and a gradually decreased temperature, the proposed method significantly reduced the memory usage and training time of electra training.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Simple and effective method.\n2. Good performance.\n3. Very clear presentation.", "weaknesses": "The scale of models in experiments seems a bit limited under the current standard. Have you tried larger models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Hl0hgerj9", "reviewer": "Reviewer_QMNg", "summary": "This paper explores efficient ELECTRA training methods by advocating the use of a pre-trained/existing language model as an auxiliary model, rather than simultaneous training of the model and auxiliary model.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The method is both intuitive and effective.\n2. The problem it tackles is highly practical.", "weaknesses": "1. The proposed method appears tailored specifically for ELECTRA, potentially limiting its applicability and community interest.\n2. Could we consider applying a continual learning method (e.g., [1]) to enhance ELECTRA's efficiency?\n\n[1]: Adapting a Language Model While Preserving its General Knowledge, Ke et al., EMNLP 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)