Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
Authors: Yu Meng
Chenyan Xiong
Payal Bajaj
saurabh tiwary
Paul N. Bennett
Jiawei Han
Xia Song
Venue: ICLR 2022 Poster (2022)
Keywords: Language Model Pretraining

Abstract: We present a new framework AMOS that pretrains text encoders with an Adversarial learning curriculum via a Mixture Of Signals from multiple auxiliary generators. Following ELECTRA-style pretraining, the main encoder is trained as a discriminator to detect replaced tokens generated by auxiliary masked language models (MLMs). Different from ELECTRA which trains one MLM as the generator, we jointly train multiple MLMs of different sizes to provide training signals at various levels of difficulty. To push the discriminator to learn better with challenging replaced tokens, we learn mixture weights over the auxiliary MLMs' outputs to maximize the discriminator loss by backpropagating the gradient from the discriminator via Gumbel-Softmax. For better pretraining efficiency, we propose a way to assemble multiple MLMs into one unified auxiliary model. AMOS outperforms ELECTRA and recent state-of-the-art pretrained models by about 1 point on the GLUE benchmark for BERT base-sized models.

Published as a conference paper at ICLR 2022
PRETRAINING TEXT ENCODERS WITH ADVERSARIAL MIXTURE OF TRAINING SIGNAL GENERATORS
Yu Meng', Chenyan Xiong? Jiawei Han'
Payal Bajaj²
Saurabh Tiwary?
Paul Bennett
Xia Song²
University of Illinois Urbana-Chama Chammapnn Microsoft
{ (yumeng5, (omj)@illinnii edu 2 2[chenyan xiong, payal.aaaaaaaaa...pppppp.......a satiwary. paul. n. bennett, xiaso) @microsoft.com com
ABSTRACT
We present new framework AMOS that pretrains text encoders with an Adversar- learning curriculum via a Mixture Of Signals from multiple auxiliary generators. Following ELECTRA-StY inator to detect replaced tokens generated by auxiliary masked language models (MLMs). Different from ELECTRA which trains one MLM the generator, we jointly train multiple MLMs of different sizes to provide training signals at various levels of difficulty. To push the discriminator to learn better with challenging replaced tokens, we learn mixture weights over the auxiliary MI.Ms outputs to maximize the discriminator loss by backpropagating the gradient from the discrim- inator via Gumbel-Softmax. For better pretraining efficiency, we propose a way to assemble multiple MLMs into one unified auxiliary model. AMOS outperforms ELECTRA and recent of-te-of-the--ae art pretrained models by about point on the GLUE benchmark for BERT base-sized models.
pretraining, the main encoder is trained as a discrim-
INTRODUCTION
Instead of pretraining encoders on texts with random noise (e.g., randomly masked tokens in BERT (Devlin et al. 2019)), ELECTRA-style frameworks employ auxiliary model to corrupt the original text sequence using its language modeling outputs, and pretrain the main Transformer detect (Clark al. 2020) correct (Meng et al. 202T the replaced tokens. These new self- supervised frameworks significantly improve the efficiency and effectiveness of pretraining and lead to strong results in various downstream tasks with ane-tuning (Clark et al., 2020). prompt-
learning (Meng et al. 2021), and zero-shot cross-lingual representations (Chi et al. 2021.
Recent studies revealed that the key to the ELECTRA'S success is new learning dynamics (Xu et al.] 2020; Meng et al., (202T). By pretraining the auxiliary model jointly with the main Transformer, an implicit learning curriculum is formed: The noise produced by the auxiliary generator becomes more and more plausible during pretraining, posing greater challenges for the discriminator, which has to overcome the difficulty by reasoning more deeply using the contexts. This leads to significantly improved sample efficiency and effectiveness of ELECTRA-style pretrained models (Clark et al.) 2020 Chi et al. 202T: Meng et al. 2021).
On the other hand, such training dynamic also introduced new challenges in search of the optimal pretraining setting. First, the configurations the auxiliary generator-ite its depth, width, and masking fraction-require costly trial-and-error pretraining runs. At the same time, they also significantly impact the discriminator's downstream task performance: A weak auxiliary model does not generate hard enough pretraining signal to push the discriminator, but too strong one can confuse the discriminator and worsen its downstream task performance (Clark et al. 2020: Meng et al. 2021). Second, the byddeb training of the two models forms a pseudo "GAN -tylee (Goodfellow et al.) 2014) curriculum which causes difficulty to improve or scale: Previous attempts to make the generator and discriminator learning more interactive (e.g., training the generator to maximize the discriminator loss as in actual GAN frameworks) resulted in downgraded performance (Clark et al. 2020).
Part of this work was done while Yu was interning at Microsoft.

---

Published as a conference paper at ICLR 2022
In this paper, present new approach that learns to automatically select pretraining signals and constructs the learning curriculum of ELECTRA-style frameworks. Our approach, AMOS, "Adversarial Mixture Of Signals" for text encoder pretraining. samples diverse set of pretraining signals from different layers of an auxiliary generator. The sampling is conducted with Gumbel- Softmax gradient estimation of the (reversed) gradient backpropagated from the main discriminator, which allows adversarial learning of mixture weights over the auxiliary generator's s signals. In this way, AMOS constructs a more diverse set of pretraining signals for the main discriminator training and enables an automatic selection of the signals to form more effective learning curriculum. In addition, by sampling from different layers of one generator model, AMOS ensures minimum computation overhead.
Our experiments on the GLUE and SQuAD benchmark demonstrate the effectiveness of AMOS. In the standard BERT Base and RoBERTa pretraining settings (Devlin et al.) (019), AMOS shows robust advantage across all included language representation tasks. The improvements are significant as it advances the state-of-the-ard by about 1 absolute point on average GLUE score and has competitive SQuAD accuracy. Our thorough ablation studies confirm that such effectiveness stems from the diverse training signals from the mixture of generators and the adversarial learning approach to combine them
In the rest of this paper, first recap related work (Section 2 and then propose our methods (Section 3). After that we describe our experimental settings (Section 4 and evaluation results (Section The last section concludes and discusses potential future research directions.
RELATED WORK
Besides the standard ruto-regressive language modeling (Radford et al.) 2019) and masked language modeling (MLM) (Devlin et al., 2019), many have explored different designs of the pretraining task, for example, prefix language modeling (Rattel al. 2019), permutational language modeling Yang et al. 2 and unified language modeling (Dong et al., 2019). The advantage of manually constructed pretraining tasks is more often observed in an application -speiiinnsecciicc manner, where prior knowledge about the target scenario is introduced (Guu et al. 2020 Roberts et al., 2020 JJa et al. 2021 Lu et al. 2021). To pretrain general purpose language representations, the standard MLM task often offers the best combination of simplicity, robustness, and generalization ability among those
manually constructed pretraining signals (Raffel et al. 2019 Lewis et al.) 2019).
Clark et al. (2020) developed novel framework ELECTRA to pretrain Transformers with model- generated signals. They employed an auxiliary MLM model, the "generator", to replace the masked positions in the input sequence with tokens sampled from its language model head. The main Transformer, the "discriminator", is pretrained to detect the replaced tokens. In this way, the main model is trained to the denoise the more challenging noises from the auxiliary language model. a result, ELECTRA requires much fewer pretraining data points to reach the performance of MLM pretrained models, and offers better downstream performance when converged.
The unique effectiveness of ELECTRA intrigued many studies to understand its real source of advantage. Clark et al. (2020) originally noted the sample efficiency comes from ELECTRA's replaced token detection task being able to leverage all token positions for training. Later, Xu ct al.) (2020) pointed to the more influential change of the pretraining task complexity for the main Transformer. Recently, Meng al. (202)) conducted thorough ablation study on many variants of ELECTRA-Style models and revealed the benefits of pretraining with more challenge signals generated by the auxiliary model, well as the implicit learning curriculum by pretraining two models side-by-side.
One popular line of research to improve ELECTRA-style models is to upgrade the replaced token detection task to more semantically informative ones. Xuu et al.] (2020) proposed the multi-word choice task which pretrains the main model to pick the original token from small candidate set. COCO-LM developed a corrective language modeling task which pretrains the main model to recover the original tokens (Meng et al. 2021).
'Code and pretrained models can he found at http:ss//ggtttb...cc///orro/////
2

---

Published as a conference paper at ICLR 2022
Another line of research is improve the training signal construction on the auxiliary side. In ELECTRA, Clark et al. (2020) experimented with adversarially training the auxiliary model using feedback from the discrıiinator model via reinforcement learning. but observed worse downstream performance. Hao et al. (021) kept the auxiliary model unchanged and augmented the replaced token sampling with predicted signal difficulty and smoothed probability via Focal Loss (Linn et al.. 2017). In this work, we continue this line of research and focus on how to more automatically and effectively construct the learning signals for ELECTRA-style models.
METHOD
In this section, first recap the preliminary of ELECTRA-style pretraining, the explorations of its learning curriculum, and then present our AMOS framework.
3.1 PRELIMINARY ON ELECTRA-STYLE PRETRAINING
In ELECTRA-style pretraining (Clark al. 2020), two Transformer models are trained side-by-side: One is trained via standard masked language modeling (MLM) and is used to generate replaced tokens to corrupt the original sequences; the other is trained to denoise the corruptions from the first model (e.g., via replaced token detection (RTD) (Clark et al. 2020), multi-word choice (Xu et al.) 2020), or corrective language modeling (Meng et al. 2021)). The second model the main model to use in downstream tasks (the "discriminator") The first one is the auxiliary, often referred to as the training data "generator". In this work we built upon the original ELECTRA setup where the main model is trained with RTD.
Generator Training. Given an original sequence X = [-] tokens are randomly replaced by [MASK] symbols, and the resulting masked sequence X X [c] ung ..... [MASK] ....., Fn oriy is fed to the generator which is trained via the following loss to predict the original tokens from the vocabulary V the set of masked positions M:
orig
orig 15% of its In
exp(x, h:) EM exp(xx,.1.)
PMIM( h.)
CGEN = E Σ
log PMIM
iEMM
where (h:) are the contextualized representations generated by the Transformer (after the projec- tion head) and [M the token embeddings.
Discriminator Training. A replaced sequence TP is constructed by sampling from the generator's MLM probability distribution:
2 PMLM (I/h.) if M 2 ==-=
x; orig else.
The discriminator takes r as input and is trained to distinguish replaced tokens produced by the generator against the original tokens via the binary classification loss:
LDISC = E
Σ
log PRTD
rcP
Ji
orig
(h:)
Σ cii
log (1-PRTT PRID
rop
orig
h
2 ylar
h:)) By training the generator side-by-side with the discriminator, this ELECTRA-SHE framework constructs a natural learning currieulum for the main model. At the beginning, the generator is weak and the replaced tokens are nearly random, making the pretraining task of the main model easy for a head start. As the generator learns better, the corruptions from it become more challenging, which increase the difficulty of the learning task for the main model. After series of studies, recent research confirmed that this curriculum learning is the key to ELECTRA-style framework's effectiveness (Clark et al.) 2020; Xu et al.) 2020; Meng al.) (021). Pretraining with randomly replaced tokens or starting with a converged generator's ' harder signals both significantly worsen the discriminator effectiveness (Meng al. 2021
where (RID(Z orig h,)) = exp(w= ht)/(1 + exp(w" (h)) and is learnable weight vector.
3

---

Published as a conference paper at ICLR 2022
3.2 IN SEARCH OF THE OPTIMAL CURRICULUM ELECTRA
In general, configuring learning curriculum for neural model training is tricky (Bengio et al. 2009 Hacohen & Weinshall 2019). Many moving pieces are involved how to measure the difficulty of training signals; when and how change the training signal difficulty- all require manual trial- and-error and careful designs (Graves et al. 2017). The same challenge persists in ELECTRA-style pretraining models. More specifically, is unclear how to choose the size of the generator network and how to control the training dynamics of the generator and discriminator.
Generator Size. There many fac- tors on the auxiliary side that affect the difficulty of the generated signals: Network depth, width, MLM mask fraction, and whether to use dropout when sampling the replaced tokens. All of these have notable impact on the performance of the pretrained dis- criminator model. The general con- sensus is that the generator should be smaller than the discriminator but all the rest vary setting by setting.
Dalirawea Deneity
Mayer
GereraTo Loss.
S-laos Ga
54. blage
0000
20
Disctianinator Lees
Teaninn Stepe (xle
(a) Disc. Loss Distribution.
(b) Gen. Loss.
Figure 1: (a) After pretraining: Distribution of discrimina- tor loss on replaced tokens by 4/6/8-layer generators. His- tograms/Curves are distribution bins/kernel density estimates.
The original ELECTRA framework kept the generator's Transformer layer depth the same with the discrimina- tor model, enabled dropout in both its training and sampling, and mainly explored using smaller width (e.g.. with 1/3 hidden dimensions of the main model). They also empirically found that large-sized discriminators prefer different mask fractions on the generator side.
(b) During pretraining: 4/6/8-layer generator loss.
Later, COCO-LM (Meng et al. 2021) empirically found that it better to use shallower generator but of the same width as the main model, while also disabling dropout when sampling replaced tokens. The depth of the generator is still chosen empirically and the optimal settings vary with the configurations of the discriminator model.
To illustrate the difficulty of training signals by generators with different depths, we train BERT Base sized discriminators with 4/6/8-layer generators, and show their RTD loss on replaced tokens by these different-sizer generators Figure Ia The average discriminator loss grows with generator depth, indicating the increased signal difficulty, and the three distributions overlap with each other. The current practice is to enumerate different generators and choose the one leading to the best downstream task performance of the discriminator, which is tedious and unsustainable.
Generator Training Dynamics. As discussed carlier. ELECTRA constructs an implicit learning cur- riculum with the training process of the generator. To illustrate this implicit curriculum, in Figure Tb we plot the ML. loss of the generators during pretraining. The convergence of the MLM models follows logarithmic schedule: The loss drops sharply in the first 10% of pretraining and decreases slowly afterwards. It is unclear whether such a training process of the generator automatically leads to the optimal curriculum for discriminator training. Unfortunately, is challenging manually explore different generator training dynamics in ELECTRA- frameworks as it is trained side-by side-by-side with the discriminator.
One idea to improve ELECTRA-style frameworks is to connect the training of the generator with feedbacks from the discriminator (e.g., as in GAN-style models (Goodfellow et al., (014) so that the generator can be adapted based on the latest discriminator's 's state throughout pretraining. Clark ct al. (2020) experimented with training the generators to produce more difficult signals for the discriminator, using policy gradients from the latter. However, this adversarial setup makes the training unstable and yields worse results, similar to the instability of GAN-style training in other language modeling tasks (Caccia et al. 2020)) Hao et al. (2021) introduced training signal difficulty estimator to adjust the sampling of replaced tokens in MLM for more challenging training signals. achieved better results than the original ELECTRA but still kept the generator training intact.
4

---

Published as a conference paper at ICLR 2022
Generator (Multi-Layer MLM)
PKIL
Sample
Gradient Bockpropagation
No Gradient Backpropagation
Crise
Gumbel- Soffbms
Gradlent Reversal
Grodiont Estimate
V Cpiso
V CDISC
MIM
MLM
MLM
f [
[MASK]
Generator
Discriminator
Laper d:
Layer de
Lawer dic
Figure 2: Overview of AMOS. The generator has multiple layers trained with MI. to provide training signals of various levels of difficulty. The mixture weights over MLM outputs are learned to maximize the discriminator loss, by backpropagating the estimated reversed gradient from the discriminator via Gumbel-Softmax. The discriminator is trained by the RTD task.
3.3 ADVERSARIAL LEARNING WITH MIXTURE OF GENERATORS
Intuitively, different-sized generators provide training signals different levels of difficulty and also may advocate the discriminator to capture different linguistic information. For example, as shown in various probing studies (Tenney et 201 9a b), very shallow MLM model may still make trivial syntactic mistakes, thus the discriminator can focus on learning simple language syntax to identify these errors, while the errors made by deep MLM model may require the discriminator to capture more sophisticated language semantics to detect.
In this work, instead of empirically searching for the best generator setup, we propose the AMOS framework that (1) employs multiple MLM generators to construct a diverse set of pretraining signals, and (2) adversarially learns the mixture of these generator outputs from discriminator feedback to construct more challenging signals. These two designs enable AMOS to automatically compose an effective curriculum. Figure? shows an overview our AMOS framework.
Multi-Layer MIM Generator. A straightforward way to obtain a diverse set of signals for replaced token generation is to utilize multiple generator networks of different depths. However, jointly propose to train single generator
training K different MLM generators is expensive. Therefore, with multiple MLM heads at different layers to mimic the effect of using multiple MLM generators. Specifically, suppose aim to use K generators of depths D {d1, d2. ... dx } (dN < d2 < dx)), then we will instead train one generator of depth dx, and insert an MLM head to each layer d E 'D. The K MLM heads will be jointly trained and the ML. head parameters shared across the K heads:
PMLM [1 (za((1(1)
exp
aTThT h
LGEN E ( iE
Σ
log PMLM
arig (d)
h
) (2)
IV|
exp (TTThi))))
where (h (1) 1/= are the contextualized representations generated by the Transformer at the dth layer (after the projection head). During training, gradient backpropagation from upper layers is disabled at each layer d E D so that the generator is partitioned into K blocks where each block is trained via its own MLM objective without being disturbed by other blocks. This allows each block to act as individual MLM generators, while also leveraging the representations generated by previous blocks.
Learning Adversarial Mixture Weights. It would be desirable progressively adapt the output signals of the multi-layer MLM generator based on the discriminator's 's state throughout pretraining. The generated replaced tokens thus more informative for discriminator training (i.e., avoid generating easy signals which can already be well distinguished by the discriminator), Given the instability of GAN-style training to language modeling (Caceia al. 2020 Clark al.) 2020), maintain the MLM training objective of the generator, and adjust the outputs of the generator by learning mixture weights over the multiple MLM heads' outputs.
Specifically, for each masked position i E M, we learn mixture weights over the embeddings h which is generated by each ML. head, to obtain the combined MLM embedding h, Then the
5

---

Published as a conference paper at ICLR 2022
final token sampling probability distribution PMLM is computed:
exp (o 5:
exp (a. hi) vi exp (20
(d)
h.
-E dED
(d) (d)) 72 h
PMLM (2e ha))
(3)
EXED exp (TTS. s!(r))
Σn
where v is a learnable weight vector; {S. Ji=1 are the contextualized representations generated by the Transformer at the dth layer (before the projection head in MLM)).
However, directly sampling from the above MLM distribution fails to enable the mixture weight learning to be guided by the discriminator: The sampling operation is non-differentiable and prevents gradient backpropagation from the discriminator to the generator. We leverage Gumbel-Softmax (Jang
al. 2017) for a continuous approximation to the sampling operation for gradient estimation:
exp (log n. + ge) /T)
PMLM (re)h,)
Vt, ne = PMLM (zr h.) 9t ~ Gumbel(O, 1),
=1=1 exp (log मe. + gv.) /T)
where 7 is a temperature hyperparameter (a smaller 7 results a more accurate approximation).
Once we have the gradient estimation backpropagated from the discriminator side, we will update the mixture weights 7 to maximize the discriminator loss. This forms an adversarial learning curriculum as the multiple MLMs' outputs are always combined construct challenging replaced tokens given the latest discriminator state. Such an adversarial learning setup can be easily implemented via gradient reversal layer (Ganin et al., 2016) that multiplies the gradient backpropagated from the discriminator side by- -1 when updating the mixture weights.
Overall Training. AMOS jointly trains the multi-layer MLM generator fCEN, the mixture weight parameter and the discriminator IDISC via the following losses:
G 2² <- arg min (LaEN ALDiSC), 0
Oisc +- arg min LDiSC
(4)
Bussc
where is hyperparameter balancing the weight of the two losses. Similar ELECTRA, the generator minimizes LGEN in Eqn. (2) and the discriminator minimizes CDisc in Eqn. The notable difference from ELECTRA is that AMOS additionally trains the generator' mixture weights y (thus and AGEN) to maximize the discriminator loss. Note that the gradient backpropagated from the discriminator through the Gumbel- estimation used to update the mixture weights, but not the MLM embeddings (i.e., in Equ. (3) 7/() is updated by the gradient from the discriminator side,
while h (d) is not). This guarantees that the generator's 's language modeling ability is still acquired through the MLM task without being disturbed by discriminator signals.
4 EXPERIMENTAL SETUP
Pretraining Settings. We experiment with two standard pretraining settings, base and base+ +: Base is the BERT Base training configuration (Devlin et al. 2019) : Pretraining on Wikipedia and BookCorpus (Zhu et al. 2015) (16 GB of texts) for 256 million samples on 512 token sequences (125K batches with 2048 batch size). We use the same corpus and 32, 768 uncased BPE vocabulary (Sennrich
et al. 2015) as with TUPE (Ke et al.) 2020) and COCO-LM (Meng et al. 20221)
Base++ trains the base size model with larger corpora and/or more training steps. We add in OpenWebText (Gokaslan & Cohen 2019). CC-News (Liu et al. (019) and STORIES (Trinh & Le 2018), to a total 160 GB texts, and train for billion (with 2048 batch size) samples, following recent research Bao et al. (2020)] Liu et al. (2019): Yang et al. (2019). We follow the prepossessing of UniLMV2 (Bao et al. 2020 and use 64, 000 cased BPE vocabulary.
Model Architecture. Our base/base++ discriminator model uses the BERT Base architecture (Devlin 2019): 12 layer Transformer, 768 hidden size, plus T5 relative position encoding (Raffel et al. 2019 Our generator is an 8-layer Transformer with the same hidden size, and the MLM heads inserted at the 4th, 6th and 8th layers (to mimic having three generators). We disable dropout in the generator following Meng et al. (202T).
ct al.
Downstream Tasks, We use the tasks included in GLUE (Wang et al. 2018 and SQuAD 2.0 reading
comprehension (Rajpurkar al. 2016). All models are evaluated with the same standard fine-tuning

---

Published as a conference paper at ICLR 2022
Table 1: Results on GLUE and SQuAD 2.0 development (GLUE test results can be found in Appendix(D D). All results are single-task, single-model fine-tuning. We use Spearman correlation for STS, Matthews correlation for CoLA, and accuracy for the rest on GLUE. AVG is the average of the eight tasks GLUE All baseline results are reported by previous research. Results not available in public reports are marked "_"
DEV Single Task
SQUAD
Model
Params
MNLI
QQP
QNLI
SST-2
CoLAA
RIE
MRPC
STS-B
AVG
EM
F1
Setting: BERT Size, Wikipedia Book Corpus (16G88)
BERT Deviln et al. 2019
10M
84.5/
91.3
91.7
93.2
18.9
68.6
87.3
89.5
83.1
73.7
76.3
RoBERIX
XLNet Yank prai. 2019
110M
85.8/85.4
92.7
78.5
81.3
DeRERT 1e nr. 2014
134M
86.3/6..2
79.3
82.5
TUPE Kc
91.3
ELECTIX (Clars et 2020
110M
86.9/86.7
91.9
92.6
93.6
66.2
75.
88.2
89.7
85.5
79.7
82.6
(HP Focil (Hb (Hb
2011
110M
87.0/6.99
92.7
91.7
92.6
66.7
90.7
81.3
91.0
86.7
83.0
85.6
MC MC-BERT (uu
110M
85.7/85.2
89.7
91.3
92.3
62.1
75.0
86.0
88.0
83.7
00C0--.M [Nteng
88.5/88.3
92.0
93.2
91.4
AMOS
110M
88.9/88,7
92.3
93.6
94.2
70.7
86.6
90,9
91.6
88.6
84.2
87.2
Base++ Setting: BERT Base Size, Bigger Training Data, and/or More Training Steps
XLNet Youg et al 9019
110M
86.8/-
91.4
91.7
94.7
60.2
74.0
88.2
89.5
84.6
80.2
2019
87.6/-
91.9
UniLM V2 Gaaner mlr EuP0y
110M
88.5/-
91.7
93.5
95.1
65.2
81.3
91.8
91.0
87.1
83.3
86.1
DcBERTa He RiM
134M
88.8/88.5
83.1
86.2
CLEAR af H04U
COCO-LMT (Nieng 0. 2021
134M
90.2/90.0
92.2
94.2
94.6
67.3
87.4
91.2
91.8
88.6
85.4
88.1
AMOS
134M
90.5/90.4
92.4
94.4
95.5
71.8
86.6
91.7
92.0
89.4
85.0
87.9
protocols:
Single task learning with vanilla fine-tuning and reporting the median of five random seeds GLUE and SQuAD. Please refer to Appendix [ for more details.
Baselines. We compare with various pretrained models in both settings. All numbers are from reported results in recent research (more details in Appendix[].
Implementation Details. Our implementation builds upon the open-source implementation of fairseq Ott al. (2019). Standard hyperparameters in pretraining and fine-tuning are used. More details are listed in Appendix
EVALUATION RESULTS
We conduct three groups of experiments evaluate the performance of AMOS, the effect of leveraging signals from multiple generators, and the influence of its adversarial training. We also provide some case studies on the constructed pretraining sequences in AMOS.
5.1 RESULTS AND ABLATIONS ON GLUE AND SQUAD
GLUE and SQuAD Results. Table 1 lists the single-task fine-tuning performance of AMOS and notable baselines that are pretrained under the standard base and base++ setting. AMOS outperforms all previous state-of-the-ar pretraining methods on the overall GLUE score and SQuAD, AMOS's improvements are robust, often of large margins, and achieves the new state-of-the-ar on multiple tasks under this standard pretraining/fine -tning setup.
Table 2 presents the ablation studies of AMOS variants on MNLI and SQuAD, the two most stable and representative downstream evaluations (Refer Appendix for the full results). We organize the ablations into four subgroups to study the effectiveness of Curriculum Learning, Adversarial Truining, Mulri-Layer MLM, and Training Signal Diversity in AMOS.
Effect of Curriculum Setup. Disabling the automatic mixture weights learning in AMOS leads to downgraded downstream performance: Randomly picking the MLM head from the same set of generator layers (w. random layer) reduces accuracy all metrics. It increases the diversity of pretraining signals but does not configure then as effectively. Manually configuring the order of different sets of pretraining signals by manually switching from shallower to deeper layers (i.e., 4 to to 8) during pretraining (w. layer switch) does not lead to better results than random layer selection. Note that in these manual configurations manually switch the signals at 1/3 and 2 33 of the total
7

---

Effeet of Adversarial Setup. Not performing adversarial learning (- adv. train) learn mixture weights (i.e., always unweighted average signals) hurts the model performance. However, note that this ablation still benefits from the curriculum learning effect as the generator gradually learns better. In addition, also try to backpropagate the adversarial gradient update the MLM embeddings (+ adv. MLM). Specifically, Eqn. (3). both 7, and h:4) are updated by the reversed gradient from the discriminator. Our observation that this makes the training unstable, perhaps Published as a conference paper at ICLR 2022
Table Ablations on MNLI/SQUAD 2.0 dev sets that remove (-), add (+) or switch (w.) one component. Val- ues are differences (in absolute points) from AMOS Base
Table 3: Edge probing results using differ- ent MLM layers from the AMOS generator. Tasks are ordered based on suggested seman- tic depths in Tenney et al. (2019a).
MNLI
SQuAD 2.0 EM/FI
Group
Method
(m/mm)
AMOSpes
84,2/7.2
Tasks
layer 4
layer 6
layer 8
Curriculum
random layer
-0.3/404
40.6/0.6
POS
92.6
93.4
91.2
Setup
w. layer switch
-0,3/-0.3
-0.9/-1.0
Consts.
69.7
73.2
73.0
Setup
Adversarial
train odv. MLM
-0,2/-022
-0,3/0.0
-0.31-044
0.0/0.0
Deps.
86.4
85.1
88.3
Entities
91.7
93.9
93.9
Multi-MLM
stop grad.
-0,51-0.1
-0.6/-0.6
Setup
separate MLM gen.
-0.1/0.0
-0.8140.7
SRL
76.9
74.2
79.2
Backbone
4-layer gen.
-0,51-0.5
-1.1/-1.2
Coref.
77.6
75.5
77.9
(No Multi-MLLM
6-layer gen.
-0.3/-0.4
-1.1/-11
SPR2
79.4
79.0
79.9
No Ady. Train)
8-layer gen.
-0.6/-0.6
-0.9/-099
12-layer gen.
-1,1/-12
-1.8/-17
Relations
72.2
76.3
73.9
pretraining steps. We have experimented switching at different steps of pretraining but do not observe significantly better results. Manual trials are tedious and expensive in pretraining and underperform automatically learned mixture over multiple training signals.
because it hinders the MLM task of the auxiliary model. We have to use a very small gradient multiplier (e.g., 0.1) when updating the MLM embeddings with the discriminator's backpropagated gradient, which has minimal effects on the model.
Effect of Multi-Layer Generator Setup. We also experimented with several configurations of the multi-layer generators: -stop grad. does not stop the gradient at the 4th and 6th layers in the generator of AMOS; w. separate MI.M gen. trains three separate generators of and 8 layers jointly with the discriminator. Both configurations result in reduced performance of the discriminator and we keep the simpler setup as in AMOS. Additional experiments on using different numbers of MLM heads can be found in Appendix F
Effect of Diverse Training Signals. The last ablation compares with using a single generator in AMOS and without any learned mixtures. Using a 4/6/8- generator yields worse results than AMOS and previous ablations with the multi-MLM generator, especially on SQuAD. The 12-layer generator is too strong and makes the pretrained discriminator significantly worse. It is simpler and more effective grant the model access to a broader set of training signals and automatically learn to leverage them.
Pretraining Efficiency, Pretraining efficiency study of AMOS and comparison with previous models can be found in Appendix C
5.2 DIVERSE PRETRAINING SIGNALS FROM DIFFERENT GENERATORS
To study how generators of different depths provide diverse pretraining signals, we conduct probing ex- periments on eight NI.P tasks covering different linguistic aspects, following (Tenney et al. [01 2019a| b): part-of-speech (POS) (Weischedel al. (203)), constituents (Consts.), dependencies (Deps.), entities, semantic role labeling (SRL), coreference (Coref.), semantic proto-roles (SPR2 (Rudinger al.) 2018)), and relation classification (SemEval (Hendrickx et al. 2010)). The major difference between setting and Tenney et al. (2019a b) is that do not combine embeddings from multiple layers but directly use the embedding from each MLM layer AMOS generator as the (frozen) feature to a trainable linear classifier,
interested in what information is captured by each MLM layer.
As shown in Table different MLM layers in AMOS generator indeed are good at different tasks -Th 6th layer in the generator has the best performance POS, constituent labeling, entity recognition
8

---

Published as a conference paper at ICLR 2022
and relation classification, while the 8th layer performs the best on the other tasks. This demonstrates that using different layers in the multi-layer MLM generator is helpful for creating training signals of different levels of difficulty and also emphasizing different linguistic aspects. Note that although the 4th layer has worse performance than deeper layers across all tasks, is still useful for providing discriminator pretraining signals, because the discriminator needs to learn from the "mistakes" made by the generator not capturing certain language semantics. Some concrete case studies of replaced tokens generated by different generator layers can be found in Appendix] !I
5.3 EFFECTS OF ADVERSARIAL TRAINING
To better understand the adversar- ial training dynamics, we plot the mixture weights (averaged over all masked positions) assigned 4th/6th/Sth layer MLM during pre- training in Figure 3a In the later pretraining stage, the 8th layer grad- ually gains more weights with the weights of the other two layers de- creasing, showing that the generator indeed tries to create more challeng- ing training signals for the discrimi- nator. We also compare the training process of AMOS with several genera- tor ablations: (1) A 4-layer generator; (2) An 8-layer generator; (3) A 4/6/8-multi-layer ML. generator with three layers randomly picked for generating MIM replacements (Random Layer); (4) Switching from shallow layers deeper lay- ers (i.e., 4 to 6 to 8) at 1/3 and 2/3 of the pretraining steps (Layer Switch). We plot the discriminator accuracy (averaged on replaced tokens) Figure 3b AMOS creates an intuitive learning curriculum that starts simpler than random layer mixing and becomes more challenging as pretraining goes on. Making "hard" switches suddenly changes the task difficulty and may disrupt training.
(pplood Tozre
the
0.4
Kyypph
Bloeet des
Caper
02
Hexoe Loer
Toining Strpe (xfee])
Tritring Stopo (scla:)
(a) Mixture Weights.
(b) Disc. Accuracy.
Figure 3: (a) The average mixture weights of the 4th/6th/8th generator layers on all masked positions during pretraining. (b) The discriminator accuracy on replaced tokens during pretraining under different curriculum learning setups.
CONCLUSIONS AND FUTURE WORK
In this paper we present AMOS, strategy for pretraining ELECTRA-style text encoders using an adversarial mixture of multiple training signal generators. AMOS constructs the corrupted text sequences by attaching multiple MLM heads to deeper generator and sampling replaced tokens from their mixed outputs. The weights of the mixtures are learned to maximize the training signals difficulty for the discriminator, by backpropagating the reversed gradient from the discriminator through Gumbel-Softman automatically learned curriculum that composes more diverse pretraining signals.
This upgrades the ELECTRA-style pretraining framework with an
Our experiments on the GLUE and SQuAD benchmarks demonstrate the empirical advantages of AMOS. Under the standard BERT Base and RoBERTagas* pretraining settings, the same Transformer network pretrained with AMOS achieves the new state-of-the-art in nearly all evaluation metrics, with around point gains on GLUE score and SQuAD accuracy. Our studies and analyses further confirm the source of AMOS's effectiveness: The more diverse training signals from multiple generators and our adversarial learning design to effectively utilize them throughout pretraining.
Our observations can be viewed as another progress of "data-centric" AI Afffeectively constructed training signals from data can lead to significant empirical improvements without changing the model itself. Future work along this direction includes but not limited to: More studies to understand the role of training signals in language model pretraining, explorations broader set of training signal sources, and better strategies leverage different information sources in pretraining.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We strive to facilitate the reproducibility of the reported results in this paper, by (1) using the same pretraining and fine-tuning setups with previous research and reporting the median results of multiple fine-tuning runs, (2) providing details about the datasets used in Appendix [A] sources of baselines compared in Appendix and hyperparameters used in Appendix [B] (3) reporting the exact pretraining hours of our base models in Appendix [ and (4) releasing our pretrained models. All experiments in this paper are conducted on 64 A100 GPUs each with 40GB memory size.
REFERENCES
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang. Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Uni Pseudo-masked language models for unified language model pre-training. In ICML, 2020.
Yoshua Bengio, Jéröme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, 2009.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth PASCAL recognizing textual entailment challenge. In TAC, 2009.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language GANs falling short. In ICLR, 2020.
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpin and Lucia Specia. Semeval-201 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In International Workshop on Semantic Evaluation, 2017.
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, and Furu Wei. XLM-E: Cross-lingual language model pre-training via ELECTRA. arXiv preprint arXiv:2106.166 16138, 2021.
Kevin Clark, Minh- Kinh-Thang Luong, Quoc V Le, and Christopher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. ICLR, 2020.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005.
Jacob Devlin, Ming- Weig-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NATCL-HLT, 2019.
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In International Workshop on Paraphrasing (IWP), 2005.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre- pre-training for natural language understanding and generation. In NeurIPS, 2019.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 2016.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In ACL-PASCAL workshop on textual entailment and paraphrasing, 2007.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http:://Bttllonn007.gttuuu OpenWebTexttCorpus 2019.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
10

---

Published as a conference paper at ICLR 2022
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukeuoglu. Automated curriculum learning for neural networks. In ICML, 2017.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Woi Wei Chang. REALM: Retrieval- augmented language model pre-training In ICML, 2020.
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In ICML, 2019.
Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In PASCAL Challenges Workshop on Recognising Textual Entailment, 2006.
Yaru Hao, Dong, Hangbo Bao, Ke Xu, and Furu Wei. Learning to sample replacements for ELECTRA pre-training. In Findings of ACL, 2021.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhancee bert with disentangled attention. ICLR, 2021.
Iris Hendrickx, Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. Semeval-2010 task 8: Multi- way classification of semantic relations between pairs of nominals. In International Workshop on Semantic Evaluation, 2010.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzaiion with Gumbel-Softmax. In ICLR, 2017.
Robin Jia, Mike Lewis, and Luke Zettlemoyer. Question answering infused pre-training of general- purpose contextualized representations. arXiv preprint arXiv:2106.08/999 08/90, 2021.
Guolin Ke, Di He, and Tie- Liu. Rethinking the positional encoding in language pre-training. arXiv preprint arXiv:2006. 15595, 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy,
Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising cequence-to-ssquemne pre-training for natural language generation, translation, and comprehension. In ACL, 2019.
Tsung-Yt Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: Robustly Optimized BERT Pretraining Approach. arXiv preprint 19Xiv:1977.11692 2019.
Shuqi Lu, Di He, Chenyan Xiong, Guolin Kc, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie- Yan Liu, and Arool Overwijk. Less is more: Pretrain a strong siamese encoder for dense text retrieval using weak decoder. In EMNLP, 2021.
Yu Meng, Chenyan Xiong, Payal Bajaj. Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. COCO-LM: Correcting and contrasting text sequences for language model pretraining. In NeurIPS, 2021.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. FAIRSEQ: A fast, extensible toolkit for sequence modeling. In NAACL-HLT Demonstrations, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.

---

Published as a conference paper at ICLR 2022
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge you pack into the parameters of a language model? In EMNLP, 2020.
Rachel Rudinger, Adam Teichert, Ryan Culkin, Sheng Zhang, and Benjamin Van Durme. Neural- davidsonian semantic proto-role labeling. In EMNLP, 2018.
Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander D' D'Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, et al. The MultiBERTs: BERT reproductions for robustness analysis. ICLR, 2022.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2015.
Iyer Shankar, Dandekar Nikhil, and Csernai Korél. First Quora dataset release:
Question pairs, 2017.
URL https: //Wpp. www. quora. com/ wtt.c//c////a///eceooc q/quraaaaa
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Sandeep Subramanian, Sai Rajeswar Mudumba, Alessandro Sordoni, Adam Trischler, Aaron C Courville, and Chris Pal. Towards text generation with adversarially learned neural outlines. In NeurIPS, 2018.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In ACL. 2019a.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure in contextualized word representations. In ICLR, 2019b.
Trieu H Trinh and Quoc V Le. A simple method for commonsense reasoning. arXiv preprint a8Xiv:
1806.02847, 02847, 2018
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop BlackboxNLP, 2018.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. In TACL, 2019.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufmmn, Michelle Franchini, et al. Ontonotes release 5.0 1dc2013t19. Linguistic Data Consortium, 2013.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACI-HILT, 2018.
Zhuofeng Wu, Sinong Wang. Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. CLEAR: Contrastive learning for sentence representation. arXiv preprint arXiv:2012. http::2002..550000 15466, 2020.
Zhenhui Xu, Linyuan Gong, Guolin Ke, Di He, Shuxin Zheng, Liwei Wang, Jiang Bian, and Tie- Yan Liu. MC-BERT: Efficient language pre-training via a meta controller. arXiv preprint arXiv:2006.05744 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ Salakhutdinov, and Quoc Le. XLNet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. In AAAI, 2017.
Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. In ICMI., 2017.
12

---

Published as a conference paper at ICLR 2022
Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. Adversarially regularized autoencoders. In JCML, 2018.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. ICCV, 2015.
A DETAILS OF GLUE TASKS
Table GLUE task statistics and information.
Size
Domain
Task
Metric(s)
MNLI
393K
Misc.
Inference
Accuracy
QQP
364K
Social QA
Similarity
Accuracy/Fl
QNLI
Wikipedia
QA/Inference
Accuracy
SST-2
67K
Movic Reviews
Sentiment
Accuracy
CoLA
8.5K
Misc.
Acceptability
Matthews corr.
RTE
2.5K
Misc.
Inference
Accuracy
MRPC
3.7K
News
Paraphrase
Accuracy/F1
STS-B
5.7K
Misc.
Similarity
Pearson/Spearman.
Below are detailed descriptions of the tasks included in the GLUE benchmark. The statistics can be found in Table
MNLI: Multi-genre Natural I Language Inference (Williams et al. 2018) has 393K training examples via crowdsourcing. The task is predict whether premise sentence entails, contradicts is neutral to given hypothesis sentence.
Question Pairs (Shankar al. 2017 has 364K training examples from the Quora question- answering website. The task is to determine whether a given pair of questions asked are equivalent semantically.
QQP:
Question Natural Language Inference has 108K training examples collected from the Stanford Question Answering Dataset (SQuAD) (Rajpurkar al. (016)). The task is to predict whether given sentence includes the answer to given question sentence.
QNLI:
SST-2: Stanford Sentiment Treebank (Socher al. 2013 has 67K training examples obtained from
movie reviews with annoottte sentiment scores. The tasks is to determine if the sentence contains positive or negative sentiment.
CoLA: Corpus of Linguistic Acceptability WWasttatt et al. 2019) has 8.5K training examples collected from books and journal articles on linguistic theory. The task is determine whether sentence is linguistically acceptable or not.
RTE: Recognizing Textual Entailment (Dagan et al. 2005 Haim al. 2006 Giampiccolo al.)
2007 Bentivogli et al. 2009) has 2.5K training examples from textual entailment challenges. The lask is to predict whether premise sentence entails a hypothesis sentence or not.
MRPC: Microsoft Research Paraphrase Corpus (Dolan & Brocketi, 2005) contains 3.7K training examples from online news sources. The task is to predict whether two sentences are equivalent semantically.
STS-B:
Semantic Textual Similarity (Cer et al. 2017) contains 5.8K training examples collected from multiple sources with human annotations of sentence pair semantic similarity. The task is to predict how semantically similar two sentences are (with a to 5 scoring scale).
B HYPERPARAMETER SETTINGS
We use the default/standard values for most hyperparameters for pretraining: The generator pretranning uses the standard 15% masking ratio. The temperature for Gunnbel-Softmax is 0.3. The
13

---

Published as a conference paper at ICLR 2022
Table 5: Hyperparameters used in pretraining,
Parameters
base
base+
Max Steps
125K
1.95M
Peak Learning Rate
Se-4
2e-4
Batch Size
2048
2048
Warm-up Steps
10K
10K
Sequence Length
512
512
Adam
Ic-6
Ic-6
Adam (31, 32)
(0.9, 0.98)
0.98)
Clip Norm
2.0
2.0
Dropout
0.1
0.1
Table 6: Hyperparameter ranges searched for fine-tuning.
Parameters
GLUE Finc-tuning
SQuAD Fine-tuning
Max Epochs
(2, 3, 10] (1e-5, 2e-5, 3e-5, 4e-5, Se-5)
(2, 3]
Peak Learning Rate
(2e-5, 3e-5, 4e-5, 5e-5]
Batch Size
[16, 32}
{ 116, 32]
Warm-up Proportion
(6%, 10%)
(6%, 10%)
Sequence Length
512
512
Adam E
lc-6
le-6
Adam (81, .22)
(0.9, 0.98)
(0.9, 0.98)
Clip Norm
Dropout
0.1
0.1
discriminator loss weight À 50 since the loss of the binary classification task is much lower than the MLM task, which is 30, 000-way classification task. The token embeddings are shared between the generator Transformer and the discriminator Transformer. Other hyperparameters used pretraining and fine-tuning are reported in Tables 5 and respectively.
The same (or equivalent) set of hyperparameters for pretraining and fine-tuning are used for all compared methods. The reported downstream task results on GLUE/SQuAD are the median of five runs with the same set of random seeds.
C PRETRAINING EFFICIENCY
We compare the pretraining efficiency of AMOS with COCO-LM (Meng et al.) 2021). ELEC- TRA (Clark et al. 2020) and RoBERTa (Liu et al. (019) under exactly the same computation environment for base model training. We show the MNLI-mm/mm) development set accuracy (via standard fine-tuning) of AMOS checkpoints trained for different GPU hours in Figure For fair comparisons, we train all compared mod- els using the same codebase, pretraining con- figuration, and computing environments. While AMOS takes longer to train than RoBERTa and ELECTRA, it matches their final MNLI perfor- mance with significantly fewer pretraining hours and achieves much better performance upon con- vergence. For example, achieves RoBERTa's MNLI accuracy with two hours of pretraining and outperforms ELECTRA in three hours, more than a 50% reduction in pretraining time. also reaches the accuracy of COCO-LM, the recent state-of-te oftth---rr-t-t in both pretraining accuracy and efficiency, only using 60% of COCO-LM's pretraining hours. This demonstrates the advantage the adversarial curriculum learning of AMOS.
Dbo. Stt hax.
1N6.
10.0
Training (Hous)
Training (Bouss)
(a) MNLI-m
(b) MNLI-mm
Figure 4: AMOS accuracy on MNLI Dev. sets (y-axes) at different pretraining hours on 64 A100 (40 GB Memory) GPUs. The final training hours and accuracy of COCO-LM, ELECTRA and RoBERTa (trained under the exact same settings and computing environments) also shown.
14

---

Published as a conference paper at ICLR 2022
Table 7: GLUE test scores obtained from the GLUE leaderboard. We follow the standard in recent research to construct the test predictions: Searching for best hyperparameters with ten random seeds on each task individually and using the best development set model on testing data. All results are from vanilla single-task fine-tuning (no ensemble, task-specific tricks, etc.)
Model
Params
MNLL-(m//m)
QQP
QNLI
SSI-2
CoLA
RIE
MRPC
STS-B
AVG
Base Setting:
BERT
110M
84.6/8344
89.2
90.5
93.5
52.1
66.4
84.8
85.8
80.8
ELECTRA
110M
85.8/-
89.1
92.7
93.4
59.7
73.1
86.7
87.7
83.5
COCO-LM
110M
88.3/88,1
89.9
93.3
94.9
61.9
81.5
87.8
88.6
85.8
AMOS
110M
88.9/88.1
90.0
93.6
95.3
68.7
81.1
88.5
90.2
87.0
Base++ Setting:
ELECTRA
110M
88.5/88.0 0
89.5
93.1
96.0
64.6
75.2
88.1
90.2
85.6
COCO-LM
134M
89.8/89.3
89.8
94.2
95.6
68.6
82.3
88.5
90.3
87.4
AMOS
90.4/89.9
90.2
94.6
96.8
69.2
83.6
88.9
91.3
88.1
Table 8: Ablations on the development sets of all GLUE tasks and SQuAD 2.0 that eliminate (-), add (+) or switch (w.) one component. We show the median and standard deviation (as subscripts) of five random seeds on each task. The results are extensions of Table 2
GLUE Single Task
SQuAD 2.0
Group
MNLL-(m/mm)
QQP
QNLI
AM0S
88.9.. /8.....
92.3un
50.6a.
94.2pa
70.7..
86.6
90.96.
91.6G.1
84.2o.2
87. 87.2..
Curriculum
random layer layer swatch
58 0..0..
92 82.2p.1
2u
93.9.
70.21
84.5r
91.4u7
91.8%
83.60
86 60
88.6i.
70.21.
866.2.1
Adversarial
-ade tmin
58. 5o.
92.3.
94.2om
71.36m
90.9.
91.56
83.0..
86 8am
Setup
+ adv, MLM
88.0n.1/ 888.76.2
92.2n.
50.5az
93.8o.1
71.32.
85.9
91 41.4
91.8g.
84.2
87.
final mis weights
88. 1 158.0pg
92.2n.1
90.0g2
93
70.9ps
63.0
90.9g
91.30z
83.5g,3
86.5p.1
Multi-MLM
88. de.1d..///
92.3n.1
93.9.
93.9.4
71.11.
87.0.c
91.6u.
83.60
86 86.6u.2
Setup
seporate MLM gem. & layer gen.
/88.70
6.
S8.00.1 /88.4o.s
91.9g.
92..bu
93.9us
89.71,5
83.0a
90.9.
91.20a
83.0k
85.96a
Back Backe
4-aayer gan.
88 88.4u.1 Pauo
92.2a.
93 Tu.2
94.2u.
60-21.1
84.81a
91 23.4
91.3a.1
83.1a.3
86 86.0aa
(No Multi-MLM
6 layer gen.
88 fi..1 /98.3lu.
92.2p.1
50 26.1
90:2)
89.25.7
63 41.1
90.2
91-4az
85.26.1
86.1a2
No Adv. Train)
8-layer pon. 12-layer
88 Su
92.1a.1
83.
93.5m.
70.0g.3
65.2
90.7uc
91.201
65.3)
86.3us
ST.Ho.a/
92.1aa
69..a.
D GLUE TEST SET RESULTS
We show the GLUE test set scores obtained via private submissions the GLUE leaderboard in Table 7 The baseline results are directly retrieved from the leaderboard or from their original papers. We use standard single-task fine-tuning to more directly reflect the improvements from pretrained models. The advantage of AMOS over strong baselines holds on the test set: Under both base and base+ settings, AMOS outperforms ELECTRA (Clark et al. 2020) and COCO-LM (Meng et al. 2021) on almost every task. Still, we would like to note that smaller tasks such as CoLA, RTE, and MRPC are stable and leaderboard runs often use more sophisticated fine-tuning method to achieve better performance, for example, continuing training from the checkpoints fine-tuned on MNLI.
E MORE DETAILED ABLATION RESULTS
We show more detailed ablation results in Table & which extends Table D including results from all GLUE tasks and showing the standard deviations. We also include two new ablations: final mix weights pretrains the diseriminator all the way with the fixed mixture weights learned by AMOS at convergence; w. three 4-ayyr gen. trains the discriminator with mixtures over three 4-layer generators instead of the multi-layer MLM generator in AMOS.
AMOS has better performance than all other ablation versions most large tasks (MNLI, QQP, SST-2 and SQuAD) which are considered more stable and reliable indicators of model effectiveness. Small tasks (CoLA, RTE, MRPC) have much higher variance than larger tasks, and usually require intermediate task training to yield stable results (i.e., starting from checkpoints that are fine-tuned on
MNLI (Clark al. 2020
He et al. 2021
Liu al. 2019
15

---

Published as a conference paper at ICLR 2022
Table Performance study with different numbers of MLM heads used in the generator. We show the median and standard deviation (as subscripts) of five random seeds on each task.
GLUE Single Task
SQuAD 2.0
MLM Layers
(mnLI-(m//mm)
QQP
QNLI
SST-2
CoLA
RTE
MRPC
SIS-B
EM
F1
On (4.6,) (AMOS)
88.9b.1 88..
92.30.0
93.60.1
91.2p.a
70.71.
86.61.4
90.90.4
91.60.1
84.20.
87.20.s
On (4.8)
88.6o 2/88. 40.2
92.3u.1
93.44,
94.4u,2
68.91,2
85.9u.7
91.40a
91.5c.1
83.8oz
86.7e.
On (2.4.6.8)
88.46 (88, 36.3
92.20.
93.3
93.8n.
69.1o.7
81.51.5
90.96.4
91.36.
83.8p
86.8
On 8
88.20 1/88, (8..0
91.22..1
1u.
94.0u.
71.5
84.81.0
90.71.u
91.60.2
83.1a2
86.0g.1
The instability of the GLUE small tasks is widely-observedd artifact in pretraining research. It is standard (and recommended) practice to not rely on these small unstable tasks for ablation studies. For example, to ensure a correct understanding of research progress, previous studies including BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019). ELECTRA (Clark et al., (020), and
DeBERTa (He et al. (022)) I) all perform ablation studies on large GLUE tasks like MNLI. We include the ablation results small GLUE tasks in Table only for reference.
Furthermore, we show the standard derivations of the same pretrained checkpoint when fine-tuned on the corresponding tasks with different random seeds. They further confirm that these smaller tasks have large variance in the fine-tuning stage. Observing the variance of the pretraining stage (i.e., changes in model performance when pretrained with different random seeds) requires running the costly pretraining multiple times which is often infeasible. We would like to refer to a recent study (Sellam et al. 2022) which reveals that these small GLUE tasks are very unstable, and the same model pretrained with different random initialization can have 2 - 5 points difference in performance upon tine-tuning on these small tasks, while in comparison, observations on MNLI are more reliable. Since in this paper we only perform single-task fine-tuning without any intermediate task training, we mainly rely on large task results for evaluating the effectiveness of different model components. The GLUE average score is more of convenient reference point as used in the pretraining research community.
F PERFORMANCE STUDY WITH DIFFERENT NUMBERS OF MLM HEADS
In this experiment, study the performance of AMOS with different numbers of generator MLM heads K. In addition to the original AMOS which uses three MLM heads, we also show in Table 9 the downstream task performance when two MLM heads (at the 4th and Sth layers), four MLM heads (at the 2nd, 4th, 6th and Sth layers) and eight MLM heads (at each layer) are used. We note that increasing MLM heads does not necessarily lead to better performance, since having more MIL heads means that each MLM block (partitioned by the stop gradient operators at each MLM layer) will become shallower with weaker MLM learning capacity. We have also tried inserting the same amount of MLM heads at different layers of the generator, but it does not improve the results.
G DISCUSSIONS ON USING GUMBEL-SOFTMAN FOR GRADIENT ESTIMATION
In this work, we use Gumbel--Sotmax to enable gradient approximation of the non-differentiable sampling operation so that we are able to use the gradient backpropagated from the discriminator to train the mixture weights. There are other possible approaches for discriminator gradient estima- tion, including REINFORCE (Yu et al., 2017) which has been explored in the ablation studies of ELECTRA (Clark al., 2020). or directly operating in the hidden states of the generator instead of in its discrete output space which has been studied in adversarial learning based text generation research (Subramanian et al. 2018 Zhang et al.) 2017 Zhao et al., 2018). We would like to note that our first study on adversarial curriculum for language model pretraining, we prefer simple framework and standard techniques to demonstrate that such a new direction is promising. We believe that exploring more sophisticated and advanced realizations for specific model components in our AMOS framework (e.g., using better gradient estimators than future work direction.
will be an interesting
16

---

Published as a conference paper at ICLR 2022
Table 11:
Examples of replaced tokens by different-sized MLM generators. Underlined words are masked out; the replaced tokens by 4/6/8-layer generators are marked in different colors.
Example "Here, let me show you that animal." He pointed up to (at/at/into) the canopy; hanging from white branch was (of/stood/stood) pale, hairless creature Ishape/llat/animal) It was bilaterally symmetrical, with two pairs of tightly (long/frmlly/tronnly)) folded limbs, but did not appear to have any discernible head.
Example 2012, about 3, 000 (6/1,800/1 .800) villagers remained in Rammun, while there are about 7, 0000 in the Palestinian diaspora, chiefly in the United States. Many in the diaspora have second homes in the (this/the/their) village. These homes have been troubled by burglaries, therefore some owners have organised night-we watches (things/boxcs/<carchee)))
(11100001,800)
H PROBING EXPERIMENTS WITH DIFFERENT SETUPS
In addition to the probing experiments conducted Sec- tion522 we also show the results under a different setup: Instead of testing the linguistic information captured by different MLM layers in AMOS generator, we conduct the same tests on separate generators with different depths (4 layers, 6 layers and 8 layers). The probing test results on eight tasks are shown Table 10. Similar to the results in
Table 10: Edge probing results of (sepa- rate) generators with different numbers of Transformer layers.
Tasks
4 layer
layer
layer
Table generators of different depths good captur- ing different types of linguistic information. Therefore, it is also possible compose diverse training signals with multiple separate generators with different numbers lay- ers. Such findings may be useful for future studies that explore using multiple different-sized auxiliary models to provide training signals emphasizing different linguistic aspects.
POS
93.7
94.7
94.3
Consts.
75.0
76.2
76.3
Deps.
88.8
89.4
89.5
Entities
93.6
94.8
94.9
SRL
80.9
81.6
82.3
Coref.
80.0
81.7
81.7
SPR2
79.8
81.3
80.2
Relations
74.5
75.8
75.0
I CASE STUDY
Table 11 provides concrete cases of replaced tokens generated by different generator layers. The "mistakes" made by different generator layers have different levels of difficulty to detected-The 4th layer MI.M sometimes makes simple syntactic mistakes while the replaced tokens given by the 6th/8th layer are mostly plausible and need be distinguished based on deep understanding of the full contexts. This intuitively confirms motivation that using generator of multiple MLM heads can provide diverse pretraining signals to compose a more effective learning curriculum.
J THE ORIGINS OF REPORTED BASELINE SCORES
The baseline results reported Table I are obtained from the corresponding papers except the following: BERT/RoBERTE from Bao et al. (2020), ELECTRA from Meng al. (2021), XLNet base++ from Bao et al. (2020). When there are different reported scores for the same method, we use the highest of them comparisons.
K SOCIETAL IMPACT
There have been concerns about the extensive costs of computing resource required by pretraining language models. The concerns include whether is worthwhile to spend such enormous amount of resources in pretraining, and also whether the demanding resource requirements have posed barrier for most institutions to conduct pretraining research, thus slowing down the overall scientific development.
One major motivation of this work is to more efficiently pretrain language models, including (1) to achieve better model effectiveness with fixed computing resource, and (2) to achieve the same
17

---

Published as a conference paper at ICLR 2022
downstream task accuracy with fewer pretraining computes. As shown in experimental results, we are able to obtain certain successes in both fronts. We demonstrate that one can achieve the same pretraining effecttiveness of RoBERT in 128 GPU hours A100 using AMOS (two hours 64 A100 GPUs), which is quite affordable for many research institutions. We hope our observation will inspire more future studies in efficient pretraining and also enable conducting pretraining research in more accessible computing environments.
In addition, also mainly focus on the base-sized models in our exploratory research which are less costly than large or extra large models, while also being the most commonly used model configuration in the community. We release our pretrained model checkpoints the community as well.
18


Reviews:
[{"review_id": "w4K1PCPnuW", "reviewer": "Reviewer_ouEb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. A clever approach to mitigate generator-discriminator dynamics in end-to-end training of Electra-like models by using a computationally efficient mixture of generators.\n2. Strong empirical results on many downstream NLU tasks outperforming strong baselines.\n3. Fairly thorough ablations for different model components.\n\nWeaknesses:\n\n1. It is unclear to me why the discriminator cannot just optimize gammas to be high for lower layers, thereby making it easier to optimize its own objective. Wouldn’t it, therefore, be better to have the gammas affect the MLM loss as well to help the lower layers learn better.\n2. Following up on the previous point 1) Fig 3 (a) is counter-intuitive as to why layer 8 has the highest coefficient followed by 4 and then 6. I would have expected the discriminator to learn weights in the order 4 > 6 > 8 since the only gradient for these parameters comes from the discriminator loss and not the generator MLM loss (b) how robust is the mixture weights to restarts with different seeds and what if we trained from scratch with fixed mixture weights using what was obtained at the end of 120k steps in Fig 3 (a)?\n3. Although computationally expensive, it may be useful to establish a performance “upper bound” by training M different generators.", "summary_of_the_paper": "The paper presents an extension to Electra-like pre-training strategies that use MLM generator heads positioned at different layers throughout the networks and a single discriminator. Mixture-weights for different MLM outputs are also learned end-to-end via the gumbel-softmax estimator. The authors present evidence that generator-discriminator dynamics are impacted significantly by the generator’s capacity and architecture. The proposed idea, therefore, is to have the discriminator operator on top of a “mixture” of discriminators of different capacities. The authors make this mixture efficient by sharing the generator backbone and having separate MLM heads at different layers of the network and only learning the mixture weights of the representations produced at each layer.", "summary_of_the_review": "This is a meaningful contribution to Electra-like pretraining approaches and provides fairly significant downstream improvements.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "m7qMZyP2Tq", "reviewer": "Reviewer_Hrj2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "  Questions:\n\n1- Figure 1: the generator with more layers (8) gains lower training loss during pretraining, meaning it can better recover the masked token, which means during sampling it should recover the original token. In other words, the replaced-token sequence should have less replaced tokens and it should be an easier task for the discriminator!\n\n2- Are the MLM heads shared across different layers of generator? \n\n3- Table 2: the ablation on stop-gradient is only evaluated on MNLI and Squad which should degrade in performance. Is this ablation studied on other GLUE tasks as well?\n\n4- the author proposed using gumble-softmax to enable gradient backpropagation from discriminator. In ELECTRA paper, reinforcement learning is used to leverage this. Is the proposed approach can be used using RL too? what is the performance on downstream task?\n\n5- The author mentioned that the discriminator loss is used to train the mixture weights of MLM output to combine a  more difficult signal for discriminator, but it will not update the MLM embeddings. This is confusing, because generator is trained jointly with discriminator, and MLM embeddings are trained via two gradients, one from discriminator, and the other from MLM pretraining of generator! Moreover, in Figure 2, it is shown that discriminator gradient are backpropagated through generator too. In table 2, it is shown that using adv. MLM hurts MNLI matched performance by 0.3 points, whereas other tasks are untouched. what is the performance of this setting on other GLUE tasks?\n\n6- Table 1: why the number of parameters in Base++ setting is larger than Base one?\n\n7- Table 2: all the ablation results are evaluated on MNLI and squad tasks. however, I think all GLUE tasks should be considered in this study to understand the actual contribution of different components of the proposed AMOS.\n\n8- Table 2: in layer switch and random layer configuration, are all MLM layers are pretrained equally? \n\n9- Table 2: w. separate MLM gen. configuration shows only 0.1 performance decrease on MNLI-matched compared to -stop grad setup. what is the performance on this setting on other GLUE tasks?\n\n10- Figure 3(b): the discriminator accuracy with 4-layer generator have a smooth increase with higher performance than AMOS, which does not use any adversarial training, which seems an intuitive learning curriculum too. what is the explanation on this?", "summary_of_the_paper": "This paper present a method for pretraining language model using generator-discriminator training. Similar to ELECTRA model which used a MLM model as generator, which corrupts input text and replace some tokens with Masked token, and to train a different language model to detects the replaced tokens. Different from ELECTRA, they proposed to use multiple MLM generator models, which replace tokens at different difficulty levels. This way, the discriminator is forced to learn higher level of language. In order to allow gradient back-propagation from discriminator to multiple generator, they employed gumble-softmax. The evaluation results on GLUE and Squad indicates better performance than previous arts. ", "summary_of_the_review": "The proposed adversarial approach combined with the multi-generator signal which is composed from different layers of a generator provides a curriculum learning approach to train a better discriminator for downstream tasks. Despite better results on GLUE and Squad tasks compared to previous arts, the extent to which the different component of AMOS contribute to the results are less studied in the ablation. Only two tasks (MNLI and Squad) are selected for ablation of adversarial learning, mixture weights, gradient stopping, etc. The evaluation results can be extended to SuperGLUE as well. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_JqL9yzIjJu", "reviewer": "Reviewer_CBi9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clear written and relatively easy to follow. The paper idea is majorly from 3 aspects: curriculum learning, adversarial training via Gumble-softmax, layer wise understanding of pretrained models. These ideas are reasonable. The results show on GLUE dev and SQUAD 2.0 also looks significant. \n\nThere are two parts of the work I think needs further clarification. First, the author states that the model is based on mix of training signal generator, however, the work is actually using different layer mixture of signals. To make the statement solid, I think another experiment of independent generators with same layers as generators should be conducted. Also, the meaning of vector v in weight calculation is not clearly discussed. Second, the adversarial training does not include re-sample the masked tokens, which may be a strong signal in adverbial training. \n\nBesides this, as this is a pretraining work. I would expect to see the performance on GLUE test set instead of just dev set which should make the results more convincing. Also, the training time/speed/performance comparison with ELECTRA may be better shown in the work. ", "summary_of_the_paper": "This work proposed a new framework AMOS to enhance ELECTRA-style pretraining. The new framework includes adversarial learning and curriculum learning. Instead of direct adverbial learning via Gumbel-softmax, the author proposed a mix-of-signals framework to mix the signal from different layer of the generator. The authors show the overall performance outperforms ELECTRA and more-recent SOTA COCO-LM by a reasonable margin. ", "summary_of_the_review": "Overall, I think this is a reasonable work by directly improving of the ELECTRA framework. The proposed ideas are clear, straightforward, and sound. The results can be further improvement. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TPhSWWrz48Y", "reviewer": "Reviewer_sJ99", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper is well written and easy to understand. Since it builds on the well known ELECTRA architecture, it is easy to identify the main contribution, which is adding more inductive learning curriculum to ELECTRA training. The authors achieved this by using multiple auxiliary MLMs and combining their outputs to generate the input to the discriminative text encoder. To improve training efficiency and further streamline the architecture, all the auxiliary MLMs are derived from the same base MLM with outputs of multiple layers used as its own sub-MLM realization. Finally, the mixture parameter is adversarially learned using the negative of the discriminator gradient. The ablation study is exhaustive providing empirical evidence of superiority of the proposed architecture.\n\nWeaknesses:\nThere is a typo “date-centric” --> \"data-centric\"\nThe paper applied the MLM to only 3 layers (4/6/8) of the 8-layer generator. Do you have an intuition on the effect of increasing MLM heads on the discriminator performance on the downstream tasks? It was shown that using just one MLM head irrespective of the layer is worse, but is there an optimal number of of MLM heads for an N-layer generator network?", "summary_of_the_paper": "This paper presents an adversarial learning framework for pre-training text encoders following ELECTRA-style architecture. The use of single auxiliary MLM broken into sub-MLMs and then combined together with a mixture parameter is novel. Authors spent great deal of effort on the ablation study to show that the proposed framework and design choices perform better than existing state-of-art methods and other alternatives on several downstream NLP tasks.\n\nThe paper is clearly written and easy follow. The conclusion of the inductive learning curriculum is well supported by experimental results.", "summary_of_the_review": "I recommend this paper for publication given a satisfactory explanation of the raised weakness. Overall, I think the concepts explored in this paper are novel and will be of interest to the NLP community, and might spur new research in this direction.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IbxMg1dDjlB", "reviewer": "Reviewer_oMcQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The paper is very well written and easy to follow. The paper is well-placed w.r.t recent developments in pre-training by presenting similarities and differences with related work.\n\n2) The idea of using a committee of generators to create the replaced token for training the discriminator, so as to provide a diverse signal for learning different complexity levels of RTD is natural and logical (this also can be linked with curriculum learning). The neat trick utilized for modelling AMOS is to create multiple generators from the same single monolithic generator by means of individual MLM heads at different layer depths which : (i) reuses computation and is efficient, and (ii) exploits underlying representations learned by lower layers to make the deeper layer MLMs stronger. \n\n3) The empirical results of the AMOS pretraining approach are strong. Despite several discussions in the NLP community on using GLUE as an evaluation benchmark, a 1 absolute point improvement using a 12-layer base model is significant over the previously strong baseline of COCO-LM (which additionally uses the InfoNCE loss).\n\n4) The ablation studies on MNLI and SQuAD help justify the modeling decisions made in the paper w.r.t the learnable weights, the use of stop-gradient between layers, not using the adversarial discriminator loss to train the learnable weights, etc.  \n\n\nWeaknesses:\n1) The ablation studies on MNLI and SQuAD are provided without standard deviation error bars. The improvements of AMOS over the ablated components is rather small, and thus it is important to present the error bar estimates to ensure the statistical significance of the results. \n\n2) For Table 3, the experiments are performed using independent 4,6 and 8 layer generators to establish the point that the pretraining signals provided are diverse. A more appropriate experiment would have been to perform the pre-training of AMOS once, and then use the 4,6 and 8 -layer MLMs (having common transformer blocks) and then see whether this diversity in probing performance exists. The objective is to show that the 4,6 and 8 layer generator with the shared MLM have diverse probing performance on tasks.\n\n\n3) While extensive, the experiments are limited to pretraining a 12-layer transformer model. Since the paper presents arguments such as the 12-layer generator being very strong, it is natural to question whether the improvements from AMOS will translate to larger architectures (24 layers) using the same mixture of multiple generators approach. \n\n\n4) I think the paper should include a societal impact section and discuss the extensive compute resources that have been consumed for the pretraining runs and the ablation studies. Furthermore, the reproducibility statement is missing details of the compute infrastructure used : Number of GPUs, types of GPU/TPUs, etc.\n\n\nQuestions:\n1) How was the \\lambda in Equation-(4) chosen for the experiments? Was some form of cross-validation performed?\n\n2) Instead of random masking, do you have any high level thoughts/intuitions on whether the AMOS approach will also provide empirical improvements with targetted masking approaches (For example: REALM that masks specific POS entities for retrieval-augmented MLM)?\n\n3) While the different MLMs in AMOS have different number of layers, they all have the same structural transformer block, and thus are from the same \"family\" of generators. Since the goal is to increase the diversity and complexity of signal from the generator for improving the strength of the discriminator, it will be interesting to experiment with a different \"family\" of generators (based on LSTM, word-CNN, etc.). Any thoughts/comments on this?\n\n4) The paper presents the AMOS multi-generator guided pretraining approach for a general k number of generators, but all the experiments are performed using k=3. While I understand that each pretraining run is computationally expensive, I was intrested to know if the authors had any intuitions/initial results from experiments by changing the number of MLM heads k?", "summary_of_the_paper": "The paper presents a framework for pre-training transformer based LMs a la ELECTRA-Generator-Discriminator style by leveraging a mixture of generators and using signal from the discriminator to guide the learning of the generator. Specifically, the generator is modified to have multiple MLM heads specific to different layers (under the intuition that the deeper layers will act as a stronger generator), and the replacement predictions from each of the layers are combined using learnable weights as the input to the discriminator. To aid learning and ensure the different MLM heads of the generator to function as \"independent generators\", the gradient is not back-propagated across the entire generator, but only within individual blocks between the MLM heads. For back-propagating the discriminator loss for learning the mixture of generators, the Gumbel-Softmax trick is used to back-prop the loss to the weights (the discriminator-guided selection of different generators can be intuitively linked with GANs). Empirical evaluation is performed on GLUE and SQuAD, and the 12-layer AMOS pre-training (with both the standard base and base++ pretraining configurations) is able to outperform previous strong baselines like ELECTRA and COCO-LM by an average of 1+ absolute point. Ablation studies have been performed to show the empirical benefits of different components: the weights v/s random and simple->hard 3 model curriculum, back-propagating discriminator loss only to the weights and not to the MLM prediction heads, etc.\n", "summary_of_the_review": "The paper makes a novel addition to the ELECTRA-style LM pretraining direction by using multiple generators to guide the learning of the discriminator, and in turn use the loss from the discriminator to enhance the generation capability of the generators. The idea is simple, easy to comprehend and evaluated on the standard benchmarks used for LM pretraining research. The results are strong and show a 1 point absolute improvement in average GLUE score over previous baselines.\n\nThe results of the ablation study are not very conclusive, and raise some doubts about whether certain modeling decisions are essential (and if they are making the AMOS modeling too complex). There are some weaknesses in the experiments to justify the need for different number of layers of generators, and concerns over generalization to different values of k and the size of the transformer architecture.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GjB4sVpTx_3", "reviewer": "Reviewer_UXyk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are several strengths from the paper that make me believe it is a good paper to be published in ICLR 2022:\n1. The paper made a significant contribution to idea of using adversarial training as part of the self-supervision signal for language learning.\n2. The paper made a impactful finding for practicing adversarial training, that mixture of signals at different depth of of the generator can stabilize ELECTRA-style models trained adversarially using Gumble-Softmax relaxation.\n3. The experiments in the paper demonstrated the superiority of adding adversarial training to a self-supervision framework, in that significant improvements can be obtained for similar-sized networks.\n4. Good set of ablation studies to show that each component of the model is necessary, especially because the entire model already has many moving parts in addition to adversarial training.\n\nWith the strengths being said, I hope to also point out that the paper's application of adversarial training is one attempt in many possibilities, and in many cases it is not clear where the improvements come from. For example:\n1. The paper pointed out that ELECTRA framework [1] explored the idea of using REINFORCE [2] as the the way of adding adversarial training signals to the model but observed degenerated results. Compared to this, the paper made 2 changes to the model: 1) using Gumble-Softmax instead of REINFORCE, and 2) using mixture-of-signals instead of straightforward gradient back-propagation. It is hard to know here which one of these actually made the adversarial setup useful. Is it possible to run an ablation study using the combination of REINFORCE and mixture-of-signals to verify whether Gumble-Softmax relaxation is the reason for it to work?\n2. There are many past papers that apply the discriminator to some internal representations of the generator instead of on the softmax outputs of the discrete text signal (for example, [3][4][5]). In the practices of GAN for text, these are proven to be more stable than both Gumble-Softmax relaxation and REINFORCE. What are the reasons for the paper to choose Gumble-Softmax relaxation instead? Please discuss.\n\nReferences:\n\n[1] Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning, ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators, ICLR 2020\n\n[2] Lantao Yu, Weinan Zhang, Jun Wang, Yong Yu, SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient, AAAI 2017\n\n[3] Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, Lawrence Carin, Adversarial Feature Matching for Text Generation, ICML 2017\n\n[4] Jake Zhao (Junbo), Yoon Kim, Kelly Zhang, Alexander M. Rush, Yann LeCun, Adversarially Regularized Autoencoders, ICML 2018\n\n[5] Sandeep Subramanian, Sai Rajeswar Mudumba, Alessandro Sordoni, Adam Trischler, Aaron C. Courville, Chris Pal, Towards Text Generation with Adversarially Learned Neural Outlines, NeurIPS 2018", "summary_of_the_paper": "The paper proposes a method to mitigate the difficulty of training deep adversarial auto-regressive generators in the ELECTRA self-supervised framework, by extracting the hidden representation from each layer of the deep generator network and feeding it through a mixture of representations function. The resulting model, named adversarial mixture of signals (AMOS), combined with the use of Gumble-Softmax relaxation, effectively stabilized GAN-style adversarial training for ELECTRA-style frameworks. This enabled the possibility of learning better generators and discriminators, by making it possible to feed gradients from the discriminator directly to the generator. The paper provided an extensive set of experiments for self-supervised language learning using GLUE and SQuAD tasks, for which AMOS showed state-of-the-art results compared to similarly-sized alternatives. Furthermore, extensive ablation studies are provided to show that both curriculum setup (as in the ELECTRA-style frameworks) and adversarial setup (as in AMOS, which is new in the paper) improve the results. These studies also verify the effectiveness of stabilizing training using the mixture-of-hidden-representations setup, especially for deeper generators.", "summary_of_the_review": "Novel idea of stabilizing ELECTRA-style language learning using a mixture of signals at different depth of generator hidden representations. Significant contribution to using adversarial training as part of the signal for self-supervised language learning. Good experimental results of the proposed model. Good set of ablation studies for each part of the model. Some unclear reasoning in the choice of adversarial training techniques compared to previous papers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received six reviews, consisting of three 8s two 6s and one 3.
The reviewers generally felt that the proposed Electra-like pretraining provided fairly significant downstream improvements.
Additional ablations were provided to during the author response period and other author responses were sufficient to cause scores to rise during the discussion period.
The vast majority of reviewers recommended accepting this paper and the AC also recommends acceptance.
Decision: Accept (Poster)