ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Authors: Zhenzhong Lan
Mingda Chen
Sebastian Goodman
Kevin Gimpel
Piyush Sharma
Radu Soricut
Venue:  (2020)
Keywords: Natural Language Processing
BERT
Representation Learning

Abstract: Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems,  we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT~\citep{devlin2018bert}. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at https://github.com/google-research/ALBERT.

Published as a conference paper at ICLR 2020
ALBERT: A LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS
Zhenzhong Lan'
Mingda Chen
Sebastian Goodman'
Kevin Gimpel?
Piyush Sharma'
Radu Soricut'
GGogle Research
Toyota Technological Institute at Chicago
flanzhzh, seabass, piyushsharma rma, rsoricut /ecool/e@ollle...ooo com {mchen, kgimpel)ettic.eet
ABSTRACT
Increasing model size when pretraining natural language representations often re- sults in improved performance on downstream tasks. However, at some point fur- ther model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter- reduction techniques to lower memory consumption and increase the training speed of BERT (Devlin et al. 2019). Comprehensive empirical evidence shows that our proposed methods lead models that scale much better compared self-supervised loss that focuses modeling
the original BERT. We also inter-sentence coherence, and show consistently helps downstream tasks with multi-sentence inputs. As result, our best model establishes state-of-the-art results on the GLUE, RACE, and SQuAD benchmarks while having fewer param- eters compared to BERT-large. The code and the pretrained models available at htttps::/ com googl/ research/A ALBERT
INTRODUCTION
Full network pre-traine pre-training DDai & Le, 2015 Radford et al. 2018 Devlin al., 2019 Howard & Ruder] (218)) has led to a series of breakthroughs in language representation learning. Many non- trivial NLP tasks, including those that have limited training data, have greatly benefited from these pre-trained models. One of the most compelling signs of these breakthroughs is the evolution of ma- chine performance on reading comprehension task designed for middle and high-school English exams in China, the RACE test (Lai et al.) 2017): the paper that originally describes the task and for- mulates the modeling challenge reports then state-of-thes published result reports their model performance at 83.2% (Liu al. 2019); the work present here pushes even higher to 89.4%, a stunning 45.3% improvement that is mainly attributable to current ability build perfrrrrrfrm pretrained language representations.
machine accuracy at 44.1%; the latest
Evidence from these improvements reveals that a large network is of crucial importance for achiev- ing state-of-the-arr performance (Devlin al. 2019 Radford et al., 2019). It has become common practice to pre-train large models and distill them down to smaller ones (Sun et al. 2019 Ture et al., 2019) for real applications. Given the importance of model size, ask: having better NLP models as easy as having larger models?
An obstacle to answering this question is the memory limitations of available hardware. Given that current state-of-the-ang models often have hundreds of millions or even billions of parameters, it is easy to hit these limitations as we try to scale our models. Training speed can also be significantly hampered in distributed training, as the communication overhead is directly proportional to the number of parameters in the model.
Existing solutions to the aforementioned problems include model parallelization (Shazeer et al.) 2018 Shoeybi et al. 2019) and clever memory management (Chen et al. 2016 Gomez al. 2017).
Work done intern Google Research, driving data processing and downstream task evaluations.

---

Published as a conference paper at ICLR 2020
These solutions address the memory limitation problem, but not the communication overhead. In this paper, we address all of the aforementioned problems, by designing A Lite BERT (ALBERT) architecture that has significantly fewer parameters than a traditional BERT architecture.
ALBERT incorporates two parameter reduction techniques that lift the major obstacles in scaling pre-trained models. The first one is factorized embedding parameterization. By decomposing the large vocabulary embedding matrix into two small matrices, we separate the size of the hidden layers from the size of vocabulary embedding. This separation makes easier to grow the hidden size without significantly increasing the parameter size of the vocabulary embeddings. The second technique is cross-layer parameter sharing. This technique prevents the parameter from growing with the depth of the network. Both techniques significantly reduce the number of parameters for BERT without seriously hurting performance, thus improving parameter-efficinney An ALBERT configuration similar to BERTT-large has 18x fewer parameters and can be trained about 1. 1.7x faster. The parameter reduction techniques also act as form of regularization that stabilizes the training and helps with generalization.
To further improve the performance of ALBERT, we also introduce a self-supervised loss for sentence-order prediction (SOP). SOP primary focuses inter-sentence coherence and is designed to address the ineffectiveness (Yang et al. 2019 Liu et al. 2019) of the next sentence prediction (NSP) loss proposed in the original BERT.
As result of these design decisions, we are able to scale up to much larger ALBERT configurations that still have fewer parameters than BaRT-large but achieve significantly better performance. We establish new state-of-the-arn results on the well-known GLUE, SQuAD, and RACE benchmarks for natural language understanding. Specifically, we push the RACE accuracy to 89.4%, the GLUE benchmark to 89.4, and the Fl score of SQuAD 2.0 to 92.2.
2
RELATED WORK
2.1 SCALING UP REPRESENTATION LEARNING FOR NATURAL LANGUAGE
Learning representations of natural language has been shown to be useful for a wide range of NLP
tasks and has been widely adopted Mikolov al. 2013 Le & Mikolov 2014 Dai & Le 2013 Pe-
ters et al. 2018 Devlin et al, 2019 Radford et al. [2188] 2019). One of the most significant changes in the last two years the shift from pre-training word embeddings, whether standard (Mikolov
et al. 2013 Pennington et al.] 2014) or contextualized (McCann et al. 2017 Peters et al. 2018).
to full network pre-training followed by task-speeific fine-tuning (Dai & Le 2015| Radford et al.)
2018 Devlin et al.) 2019). In this line of work. it is often shown that larger model size improves performance. For example, Devlin et al. (2019) show that across three selected natural language understanding tasks, using larger hidden size, more hidden layers, and more attention heads always leads to better performance. However, they stop at a hidden size of 1024, presumably because of the model size and computation cost problems.
is difficult to experiment with large models due to computational constraints, especially in terms of GPU/TPU memory limitations. Given that current state-of-the-art models often have hundreds of millions or even billions of parameters, we can easily hit memory limits. To address this issue, [Chn ct (2116) propose a method called gradient checkpointing to reduce the memory requirement be sublinear at the cost of an extra forward pass. Gomez et al. (2017) propose a way to reconstruct each layer's 's activations from the next layer so that they do not need to store the intermediate activations.
Both methods reduce the memory consumption at the cost of speed. Raffel et al. (2019) proposed to use model parallelization to train a giant model. In contrast, our parameter-reduction techniques reduce memory consumption and increase training speed.
2.2 CROSS-LAYER PARAMETER SHARING
The idea of sharing parameters across layers has been previously explored with the Transformer architecture (Vaswani et al. 2017). but this prior work has focused on training for standard encoder- decoder tasks rather than the pretraining/fiettning setting. Different from our observations, De- lighnn et al. (2018) show that networks with cross-layer parameter sharing (Universal Transformer, UT) get better performance language modeling and subject-verb agreement than the standard
2

---

Published as a conference paper at ICLR 2020
transformer. Very recently, Bai al. (0199) propose Deep Equilibrium Model (DQE) for trans- former networks and show that DQE can reach an equilibrium point for which the input embedding and the output embedding of certain layer stay the same. Our observations show that em- beddings are oscillating rather than converging. Hao et al. (2019) combine a parameter-sharing transformer with the standard one, which further increases the number of parameters of the standard transformer.
2.3 SENTENCE ORDERING OBJECTIVES
ALBERT uses pretraining loss based predicting the ordering of two consecutive segments of text. Several researchers have experimented with pretraining objectives that similarly relate to discourse coherence. Coherence and cohesion in discourse have been widely studied and many phenomena have been identified that connect neighboring text segments (Hobbs) 1979; Halliday & Hasan 1976 Grosz et al. 1995). Most objectives found effective in practice are quite simple. Skip- thought KKros et al. 2013) and FastSent (Hill et al. 2016) sentence embeddings are learned by using an encoding of sentence to predict words in neighboring sentences. Other objectives for sentence emhedding learning include predicting future sentences rather than only neighbors (Gan et al. 2017) and predicting explicit discourse markers (Jernite al., [017]] Nie al.] 2019). Our loss is most similar to the sentence ordering objective of Jernite et al. (2017). where sentence embeddings are learned in order to determine the ordering of two consecutive sentences. Unlike most of the above work, however, our loss is defined on textual segments rather than sentences. BERT (Devlin et al. 2019) uses a loss based on predicting whether the second segment in a pair has been swapped with a segment from another document. We compare to this loss in our experiments and find that sentence ordering
more challenging pretraining task and more useful for certain downstream tasks. Concurrently to our work, Wang et al. (2019) also try to predict the order of two consecutive segments of text, but they combine it with the original next sentence prediction in a three-way classification task rather than empirically comparing the two.
THE ELEMENTS OF ALBERT
In this section, present the design decisions for ALBERT and provide quantified comparisons against corresponding configurations of the original BERT architecture (Devlin et al. 2019).
3.1 MODEL ARCHITECTURE CHOICES
The backbone of the ALBERT architecture similar to BERT in that uses a transformer en- coder (Vaswani et al. 2017) with GELU nonlinearities (Hendrycks & Gimpel 2016). We follow the BERT notation conventions and denote the vocabulary embedding size as E, the number of encoder layers L, and the hidden size as H. Following Devlin et al. 2219)) we set the feed-forward/fiter size to be 4// and the number of attention heads be H/64.
There are three main contributions that ALBERT makes over the design choices of BERT.
Factorized embedding parameterization. In BERT, as well as subsequent modeling improve- ments such XLNet (Yang et al. 2019) and RoBERTa (Liu et al. (209)), the WordPiece embedding size E is tied with the hidden layer size H, i.e., E = H. This decision appears suboptimal for both modeling and practical reasons, as follows.
From a modeling perspective, WordPiece embeddings are meant to learn context- inneeeddinnd repre- sentations, whereas hidden-layer embeddings are meant learn context-depender representations. As experiments with context length indicate (Liu et al. 2019). the power of BERT-like represen- tations comes from the use of context provide the signal for learning such context-dependent representations. As such, untying the WordPiece embedding size E from the hidden layer size H allows us to make a more efficient usage of the total model parameters as informed by modeling needs, which dictate that H > E.
From a practical perspective, natural language processing usually require the vocabulary size to be large If E = H,, then increasing // increases the size of the embedding matrix, which has size
Similar BERT, all the experiments in this paper use vocabulary size V of 30,000.
3

---

Published as a conference paper at ICLR 2020
Vx E. This can easily result in model with billions of parameters, most of which are only updated sparsely during training.
factorization of the embedding parameters, decomposing them into two smaller matrices. Instead of projecting the one-hot vectors directly into the hidden space of size H, first project them into lower dimensional embedding space of size E, and then project to the hidden space. By using this decomposition, we reduce the embedding parameters from O(V x H) O(V E + E x H). This parameter reduction is significant when H > E. We choose to use the same E for all word pieces because they are much more evenly distributed across documents compared to whole-word embedding, where having different embedding size (Grave et al. (017);; Baevski & Auli (2018); Dai et al. (2019) for different words is important.
Therefore, for ALBERT
Cross-layer parameter sharing. For ALBERT, we propose cross-layer parameter sharing as an- other way to improve parameter efficiency. There are multiple ways to share parameters, e.g., only sharing feed-forward network (FFN) parameters across layers, or only sharing attention parameters. The default decision for ALBERT is to share all parameters across layers. All our experiments use this default decision unless otherwise specified. compare this design decision against other strategies in our experiments in Sec. 4.5
Similar strategies have been explored by Dehghani et al. (2018) (Universal Transformer, UT) and Bai et al. (2019) (Deep Equilibrium Models, DQE) for Transformer networks. Different from our observations, Dehghani et al. (2018) show that outperforms a vanilla Transformer. Bai et al. (219)) show that their DQEs reach an equilibrium point for which the input and output embedding of a certain layer stay the same. Our measurement on the L2 distances and cosine similarity show that our embeddings oscillating rather than converging.
16
BERT-lage
Cosine Similurity (Degres)
40
BERT-large
larg
ALBERT-large
1.2 disancce
10
25
13 1s
10
Layer ID)
Layer ID
Figure 1: The 1.2 distances and cosine similarity (in terms of degree) of the input and output embed- ding of each layer for BERT-large and ALBERT-large.
Figure I shows the L2 distances and cosine similarity of the input and output embeddings for each layer, using BERT-large and ALBERT-large configurations (see Table We observe that the tran- sitions from layer layer much smoother for ALBERT than for BERT. These results show that weight-sharing has an effect on stabilizing network parameters. Although there drop for both metrics compared BERT, they nevertheless do not converge even after 24 layers. This shows that the solution space for ALBERT parameters is very different from the one found by DQE.
Inter-sentence coherence loss. In addition the masked language modeling (MLM) loss De- vlin et al. 2019), BERT uses an additional loss called next-sentence prediction (NSP). NSP is binary classification loss for predicting whether two segments appear consecutively in the original text, as follows: positive examples are created by taking consecutive segments from the training corpus; negative examples are created by pairing segments from different documents; positive and negative examples are sampled with equal probability. The NSP objective was designed to improve performance on downstream tasks, such as natural language inference, that require reasoning about the relationship between sentence pairs. However, subsequent studies (Yang et al.; 2019 Liu al., 2019) found NSP's impact unreliable and decided to eliminate it, a decision supported by an im- provement in downstream task performance across several tasks.
We conjecture that the main reason behind NSP's ineffectiveness its lack of difficulty as a task, compared to MLM. As formulated, NSP conflates topic prediction and coherence prediction in
4

---

Published as a conference paper at ICLR 2020
Model
Parameters
Layers
Hidden
Embedding
Parameter- sharring
base
108M
768
768
False
BERT
large
334M
24
1024
1024
False
base
12M
12
768
128
True
large
18M
24
1024
128
True
ALBERT
xlarge
60M
24
2048
128
True
xxlarge
235M
12
4096
128
True
Table The configurations of the main BERT and ALBERT models analyzed in this paper.
single tasl However, topic prediction is easier learn compared to coherence prediction, and also overlaps more with what is learned using the MLM loss.
We maintain that inter-sentence modeling is important aspect of language understanding, but propose a loss based primarily on coherence. That is, for ALBERT, we use sentence-order pre- diction (SOP) loss, which avoids topic prediction and instead focuses modeling inter-sentence coherence. The SOP loss uses as positive examples the same technique as BERT (two consecu- tive segments from the same document), and negative examples the same consecutive seg- ments but with their order swapped. This forces the model to learn finer-grained distinctions about discourse-level coherence properties. As we show Sec. Sec. 4.6 turns that NSP cannot solve the SOP task at all (i.e., it ends up learning the easier appic-predcctons signal, and performs at random- baseline level on the SOP task), while SOP can solve the NSP task to a reasonable degree, pre- sumably based on analyzing misaligned coherence cues. As result, ALBERT models consistently improve downstream task performance for multi-sentence encoding tasks.
3.2
MODEL SETUP
We present the differences between BERT and ALBERT models with comparable hyperparameter settings in Table [ Due to the design choices diseussed above, ALBERT models have much smaller parameter size compared to corresponding BERT models.
For example, ALBERT-large has about 18x fewer parameters compared to BERT-large, 18M ver- sus 334M. An ALBERT-x -large configuration with H = 2048 has only 60M parameters and an L.BERT-xxlarge configuration with H 4096 has 233M parameters, i.e., around 70% of BERT- large's parameters. Note that for ALBERTTxxlarge, we mainly report results 1 12-layer network because a 24-layer network (with the same configuration) obtains similar results but computation- ally more expensive.
This improvement in parameter efficiency is the most important advantage of ALBERT's design choices. Before we can quantify this advantage, we need to introduce our experimental setup in more detail.
4 EXPERIMENTAL RESULTS
4.1 EXPERIMENTAL SETUP
To keep the comparison as meaningful as possible, we follow the BERT (Devlin et al. 2019) setup in using the BOOKCORPUS (Zhu al.] 2015) and English Wikipedia (Devlin et | 2019) for pretrain- ing baseline models. These two corpora consist around 16GB of uncompressed text. We format our inputs as "I "[CLS] æı [SEP] T2 [SEP]", where x I1 21,2 and x22 11,1. two segments We always limit the maximum input length to 512, and randomly generate input sequences shorter than 512 with a probability of 10%. Like BERT, we use a vocabulary size of 30,000, tokenized using SentencePiece (Kudo & Richardson 2018) as in XLNet (Yang al. 2019).
$1,2
Sinne negative example is constructed using material from different document, the negative-example segment is misaligned both from a topic and from coherence perspective. A segment usually comprised of more than one natural sentence, which has been shown to henefit performance by Liu al. (2019).

---

Published as a conference paper at ICLR 2020
We generate masked imputs for the MLM targets using n- n-gram masking (Joshi et al. 2019), with the length of each n-gram mask selected randomly. The probability for the length is given by
1/n p(n) = 1
We the maximum length of n-gram (i.e., 11) be (i.e., the MLM target can consist of up 3-gram of complete words, such as "White House correspondents".
All the model updates use a batch size of 4096 and LAMB optimizer with learning rate 0.00176 (You et al. 2019)). We train all models for 125,000 steps unless otherwise specified. Train- ing was done Cloud TPU V3. The number of TPUs used for training ranged from 64 512, depending on model size.
The experimental setup described in this section is used for all our own versions BERT as well as ALBERT models, unless otherwise specified.
4.2
EVALUATION BENCHMARKS
4.2.1
INTRINSIC EVALUATION
To monitor the training progress, we create development set based the development sets from SQuAD and RACE using same procedure as Sec. 4.1 report accuracies for both MLM and sentence classification tasks. Note that only use this set check how the model is converging; it has not been used in way that would affect the performance of any downstream evaluation, such via model selection.
4.2.2 DOWNSTREAM EVALUATION
Following Yang et al. 2019) and Liu al. 2019), evaluate models three popular bench-
marks: The General Language Understanding Evaluation (GLUE) benchmark (Wang et al. 2018
two versions of the Stanford Question Answering Dataset (SQuAD: Rajpurkar et al.] 2016
2018
and the ReAding Comprehension from Examinations (RACE) dataset (Lai et al. (017)) For com- pleteness, we provide description of these benchmarks in Appendix [] As in (Liu et al. 2019).
we perform early stopping on the development sets, on which we report all comparisons except for our final comparisons based on the task leaderboards, for which we also report test set results. For GLUE datasets that have large variances on the dev set, report median over 5 runs.
4.3 OVERALL COMPARISON BETWEEN BERT AND ALBERT
We are now ready to quantify the impact of the design choices described in Sec. 3 specifically the ones around parameter efficiency. The improvement in parameter efficiency showcases the most important advantage of ALBERT's design choices, as shown in Table with only around 70% of BERT-large's 's parameters, ALBERT- AxBeRT-xxlare achieves significant improvements over BERT-large, as measured by the difference on development set scores for several representative downstream tasks: SQuAD v1.1 (+1.9%), SQuAD v2.0 (+3.1%). MNLI (+1.4%), SST-2 (+2.2%), and RACE (+8.4%).
Another interesting observation is the speed of data throughput training time under the train- ing configuration (same number TPUs). Because less communication and fewer computations, ALBERT models have higher data throughput compared their corresponding BERT models. If use BERT-large as the baseline, we observe that ALBERT-large is about 1.7 times faster iterating through the data while ALBERT-xXlarg. is about times slower because of the larger structure.
Next, we perform ablation experiments that quantify the individual contribution of each of the design choices for ALBERT.
4.4 FACTORIZED EMBEDDING PARAMETERIZATION
Table shows the effect of changing the vocabulary embedding size E using ALBERT-base configuration setting (see Table I). using the same set representative downstream tasks. Under the non-shared condition (BERT-style), larger embedding sizes give better performance, but not by

---

Published as a conference paper at ICLR 2020
Model
Parameters
SQuAD1.1
SQuAD2.0
MNLI
SST-2
RACE
Avg
Speedup
base
108M
90.4/83.2
80.4/77.6
84.5
92.8
82.3
4.7x
BERT
large
334M
92.2/85.5
85.0/82.2
86.6
93.0
73.9
85.2
1.0
base
12M
89.3/82.3
80.0/77.1
81.6
90.3
64.0
80.1
5.6x
large
18M
90.6/83.9
82.3/79.4
83.5
91.7
68.5
82.4
1.7x
ALBERT
xlarge
60M
92.5/86.1
86. 86.1/83.1
86.4
92.4
74.8
85.5
0.6x
xxlarge 235M
94.1/88.3
88.1/85.1
88.0
95.2
82.3
88.7
0.3x
Table 2: Dev results for models pretrained over BOOKCORPUS and Wikipedia for 125k steps. Here and everywhere else, the Avg column is computed by averaging the scores of the downstream tasks to left (the two numbers of F1 and EM for each SQuAD are first averaged).
much. Under the all-shared condition (ALBERT-style), an embedding of size 128 appears to be the best. Based these results, step to do further scaling.
embedding size E = 128 in all future settings, necessary
Model
E
Parameters
SQuAD1.
SQuAD2.0
MNLI
SST-2
RACE
Avg
64
87M
89.9/82.9
80.1/77.8
82.9
91.5
66.7
81.3
ALBERT
128
89M
89.9/82.8
80,3/77.3
83.7
91.5
67.9
81.7
base
not-shared
256
93M
90.2/83.2
80.3/77.4
84.1
91.9
67.3
81.8
768
108M
90.4/83.2
80.4/77.6
84.5
92.8
68.2
82.3
64
10M
88.7/81.4
77.5/74.8
80.8
89.4
63.5
79.0
ALBERT
128
12M
89.3/82.3
80.0/77.1
81.6
90.3
64.0
80.1
base
all-shared
256
16M
88.8/81.5
79.1/76.3
81.5
90.3
63.4
79.6
768
31M
88.6/81.5
79.2/76.6
82.0
90.6
63.3
79.8
Table 3: The effect of vocabulary embedding size on the performance ALBERTT-bas..
4.5 CROSS-LAYER PARAMETER SHARING
Table presents experiments for various cross-layer parameter-sharing strategies, using an ALBERT-bas configuration (Table I with two embedding sizes (E 768 and E' 128). We compare the all-shared strategy (ALBERT-style), the not-shared strategy (BERT-style), and inter- mediate strategies in which only the attention parameters shared (but not the FNN ones) only the FFN parameters are shared (but not the attention ones).
The all- th--sh strategy hurts performance under both conditions, but it is less severe for E 128 (- 1.5 on Avg) compared to E = 768 (-2.5 on Avg). In addition, most of the performance drop appears come from sharing the FFN-layer parameters, while sharing the attention parameters results in drop when = 128 (+0.1 on Avg), and a slight drop when = 768 (-0.7 on Avg).
There other strategies of sharing the parameters layers into N groups of size M, and each size-M group shares parameters, Overall, our experimen- tal results shows that the smaller the group size M is, the better the performance get. However, decreasing group size M also dramatically increase the number of overall parameters. We choose all all-shared strategy as our default choice.
layers. For example, We divide the L
Model
Parameters
SQuADI.1
SQuAD2.0
MNLI
SST-2
RACE
Avg
all-shared shared-attention
31M
88.6/81.5
79.2/76.6
82.0
90.6
63.3
79.8
ALBERT base
83M
89.9/82.7
80.0/77.2
84.0
91.4
67.7
81.6
E=768
shared-FFN
57M
89.2/82.1
78.2/75.4
81.5
90.8
62.6
79.5
not-shared
108M
90.4/83.2
80.4/77.6
84.5
92.8
68.2
82.3
all-shared
12M
89.3/82.3
80.0/77.1
82.0
90.3
64.0
80.1
ALBERT base E=128
chared-attention
64M
89.9/82.8
80.7/77.9
83.4
91.9
67.6
81.7
shared-FFN
38M
88 88.9/81.6
78.6/75.6
82.3
91.7
64.4
80.2
not-shared
89M
89.9/82.8
80.3/77.3
83.2
91.5
67.9
81.6
Table 4: The effect of cross -ayer parameter-sharing strategies, ALBERT-bass. configuration.

---

Published as a conference paper at ICLR 2020
4.6 SENTENCE ORDER PREDICTION (SOP)
We compare head-to-head three experimental conditions for the additional inter-sentence loss: none (XLNet- and RoBERTa-style), NSP (BERT-style). and SOP (ALBERT-style), using ALBERT- base configuration. Results are shown in Table S both over intrinsic (accuracy for the MLM, NSP, and SOP tasks) and downstream tasks.
Intrinsic Tasks
Downstream Tasks
SP tasks
MLM
NSP
SOP
SQuAD1.1
SQuAD2.0
MNLI
SST-2
RACE
Avg
None
54.9
52.4
53.3
88.6/81.5
78.1/75.3
81.5
89.9
61.7
79.0
NSP
54.5
90.5
52.0
88.4/81.5
77.2/74.6
81.6
91.1
62.3
79.2
SOP
54.0
78.9
86.5
89.3/82.3
80.0/77.1
82.0
90.3
64.0
80.1
Table 5: The effect of sentence-prediction loss, NSP vs. SOP, on intrinsic and downstream tasks.
The results on the intrinsic tasks reveal that the NSP loss brings no discriminative power to the SOP task (52.0% accuracy, similar the random-guess performance for the "None" condition). This allows us to conclude that NSP ends up modeling only topic shift. In contrast, the SOP loss does solve the NSP task relatively well (78.9% accuracy), and the SOP task even better (86.5% accuracy). Even more importantly the SOP loss appears to consistently improve downstream task performance for multi-sentence encoding tasks (around +1% for SQuAD1.1, +2% for SQuAD2.0, +1.7% for RACE), for an Avg score improvement of around +1%.
4.7 WHAT IF WE TRAIN FOR THE SAME AMOUNT OF TIME?
The speed-up results in Table indicate that throughutt for BERT-large is about 3.17x higher compared to ALBERT-xxlagge. Since longer training usually leads to better performance, we per- form a comparison in which, instead controlling for data throughput (number of training steps), we control for the actual training time (i.e., let the models train for the same number of hours). In Table 6 we compare the performance a BaRT-large model after 400k training steps (after 34h of training), roughly equivalent with the amount of time needed to train ALBERT-xAae xxlarge model with 125k training steps (32h of training).
Models BERT-large
Steps
Time
SQuAD1.1
SQuAD2.0
MNLI
SST-2
RACE
Avg
400k
93.5/87.4
86.9/84.3
87.8
94.6
77.3
87.2
ALBERT-xxlarge
125k
32h
94.0/88.1
88.3/85.3
87.8
95.4
82.5
88.7
Table The effect of controlling for training time, BERT-large ALBERT-xXlarge configurations.
After training for roughly the same amount of time, ALBERT-xxlarge is significantly better than BERT-large: +1.5% better Avg, with the difference RACE high +5.2%.
4.8 ADDITIONAL TRAINING DATA AND DROPOUT RFFECTS
The experiments done up to this point use only the Wikipedia and BOOKCORPUS datasets, as in (Devlin et al. 2019). In this section. report measurements the impact of the additional data used by both XLNet YYang et al.) 2019) and RoBERTa (Liu et al.) 2019).
Fig. 2a plots the dev set MLM accuracy under two conditions, without and with additional data, with the latter condition giving a significant boost. We also observe performance improvements on the downstream tasks in Table 7 except for the SQuAD benchmarks (which Wikipedia-based, and therefore are negatively affected by out-of-domain training material).
SQuAD1.1
SQuAD2.0
MNLI
SST-2
RACE
Avg
No additional data With additional data
89.3/82.3
80.0/77.1
81.6
90.3
64.0
80.1
88.8/81.7
79.1/76.3
82.4
92.8
66.0
80.8
Table 7: The effect of additional training data using the ALBERT-basee configuration.
We also note that, even after training for IM steps, our largest models still do not overfit to their training data. As result, we decide to remove dropout further increase model capacity. The
8

---

Published as a conference paper at ICLR 2020
70.0
69.5
Dropost
Dev aoeuney (MLM)
Deo accurncy (MLM)
72.0
WO Dropout
68.5
21.5
68.0
67.5
71.0
M My myy Wurh
67.0
WXD additiional data
20.5
W udditional data
66.0
70.0
35
40
45
50
55
90
100
110
120
130
140
150
Steps (1c4)
Steps (1e4)
(a) Adding data
(b) Removing dropout
Figure The effects adding data and removing dropout during training.
plot in Fig. 2b shows that removing dropout significantly improves MLM accuracy. Intermediate
evaluation ALBERT-xxlarge around 1M training steps (Table 8 also confirms that removing dropout helps the downstream tasks. There is empirical (Szegedy et al. (017) and theoretical (Li al. 2019) evidence showing that combination of batch normalization and dropout in Convolu- tional Neural Networks may have harmful results. To the best of our knowledge, we are the first to show that dropout can hurt performance in large Transformer-bast based models. However, the underly- ing network structure of ALBERT is special case of the transformer and further experimentation needed see this phenomenon appears with other transformer-bas based architectures not.
SQuAD1.1
SQuAD2.0
MNLI
SST-2
RACE
Avg
With dropout
7/89.
89 69.6/86.9
90.0
96.3
85.7
90.4
Without dropout
94.8/89.5
89.9/87.2
90.4
96.5
86.1
Table 8: The effect of removing dropout, measured for ALBERT configuration.
4.9 CURRENT STATE SENTE-OF-THE-AR ON NLU TASKS
The results report in this section make of the training data used by Devlin et al. (2019), as well as the additional data used by Liu et al. (2019) and Yang et al. (2019). We report state-of-the-ar results under two settings for fine-tuning: single- and ensembles. In both settings, we only do single-task fine-tuning" Following Liu et al. (001)). on the development set we report the median result over five runs.
Models
MNLI QNLI
QQP
RTE
SST
MRPC
CoLA
STS
WNLI
Avg
Single-task single models on dev
BaRT-large
86.6
92.3
91.3
70.4
93.2
88.0
60.6
90.0
XLNeet-lare
89.8
93.9
91.8
83.8
95.6
89.2
63.6
91.8
RoBERTa-large
90.2
94.7
92.2
86.6
96.4
90.9
68.0
92.4
ALBERT (1M)
90.4
95.2
92.0
88.1
96.8
90.2
68.7
92.7
ALBERT (1.5M)
90.8
95.3
92.2
90.9
Ensembles on test (from leaderboard as of Sept.
16, 2079)
ALICE
88.2
95.7
90.7
83.5
95.2
92.6
69.2
91.1
80.8
87.0
MT-DNN
87.9
96.0
89.9
86.3
96.5
92.7
68.4
91.1
89.0
87.6
XLNet
90.2
98.6
90.3
93.0
91.6
90.4
88.4
RoBERTa 90.8
98.9
90.2
88.2
96.7
92.3
67.8
92.2
89.0
88.5
Ady-RoBERTa
91.1
98.8
90.3
88.7
96.8
93.1
68.0
92.4
89.0
88.8
ALBERT
91.3
99.2
90.5
89.2
97.1
93.4
69.1
92.5
91.8
89.4
Table 9: State-of-the-art results on the GLUE benchmark. For single-task single-model results, we report ALBERT 1M steps (comparable to RoBERTa) and 1.5M steps. The ALBERT ensemble uses models trained with 1M, 1.5M, and other numbers of steps.
The single-model ALBERT configuration incorporates the best-performing settings discussed: an ALBERT-xxlarge configuration (Table [ using combined MLM and SOP losses, and no dropout.
"Following Liu et al. 2019 we fine-tune for RTF, STS, and MRPC using an MNLI checkpoint.

---

Published as a conference paper at ICLR 2020
The checkpoints that contribute the final ensemble model are selected based on development set performance; the number of checkpoints considered for this selection range from to 17, depending on the task. For the GLUE (Table ( and RACE (Table (0) benchmarks, predictions for the ensemble models, where the candidates are fine-tuned from different training steps using the 12-layer and 24-layer architectures. For SQuAD (Table 10), we average the pre- diction scores for those spans that have multiple probabilities; we also average the scores of the "unanswerable" decision.
average the model
Both single-model and ensemble results indicate that ALBERT improves the state-of-the-art signif- icantly for all three benchmarks, achieving GLUE score of 89.4, SQuAD 2.0 test Fl score of 92.2, and a RACE test accuracy of 89.4. The latter appears to be particularly strong improvement, a jump of + +1..4% absolute points over BERT (Devlin et al., 2019 Clark et al., 2019). +7.6% over XLNet Yang et al. 2019), +6.2% over RoBERTA (Liu et al. 2019), and 5.3% over DCMI+ (Zhang) et al. 2019). an ensemble of multiple models specifically designed for reading comprehension tasks. Our single model achieves an accuracy of 86.5%, which is still 2.4% better than the state-of-the-art ensemble model.
Models
SQuAD1.1 dev SQuAD2.0 dev SQuAD2.0 test RACE test (Middlc/High)
Single model (from leaderboard as of Sept. 23, 2019)
BaRT-large
90,9/84.1
81.8/79.0
89.1/86.3
72.0 (76.6/70.1)
XLNet
94.5/89.0
88.8/86.1
89.1/86.3
81.8 (85.5/80.2)
RoBERTa
94.6/88.9
89.4/86.5
89 8/86.8
83.2 (86.5/81.3)
UPM
89.9/87.2
XLNNt+ SG-Net Verifier+
10.1/
ALBERT (1M)
94.8/89.2 .2
89.9/87.2
86.0 (88.2/85.1)
ALBERT (1.5M)
94.8/89.3
90.2/87.4
90.9/88.1
86.5 (89.0/85.5)
Ensembles (from leaderboard as of Sept. 23, 2019)
BERT-large
92.2/86.2
XLNet + SG-Net Verifier
90.7/88.2
UPM
90.7/88.2
XLNet+ + DAAF + Verifier
90.9/88.6
DCMN+
84.1 1 (88.5/82.3)
ALBERT
95.5/90.1
91.4/88.9
92.2/89.7
89.4 (91.2/88.6)
Table 10: State-of-the-ee art results on the SQuAD and RACE benchmarks.
DISCUSSION
While ALBERT-xXlarge has less parameters than BERT-large and gets significantly better results, it computationally more expensive due its larger structure, An important next step is thus to speed up the training and inference speed of ALBERT through methods like sparse attention (Child et al., 2019) and block attention (Shen et al. 2018). An orthogonal line of research, which could provide additional representation power, includes hard example mining (Mikolov et al.] 2013) and more efficient language modeling training (Yang et al. 2019). Additionally, although we have convincing evidence that sentence order prediction is more consistently -useeelulyyuufful learning task that leads to better language representations, we hypothesize that there could be more dimensions not yet captured by the current self- selfvsuper training losses that could create additional representation power for the resulting representations.
ACKNOWLEDGEMENT
The authors would like to thank Beer Changpinyo, Nan Ding, Noam Shazeer, and Tomer Levinboim for discussion and providing useful feedback on the project; Omer Levy and Naman Goyal for clarifying experimental setup for RoBERTa; Zihang Dai for clarifying XLNet; Brandon Norick, Emma Strubell, Shaojie Bai, Chas Leichner, and Sachin Mehta for providing useful feedback on the paper; Jacob Devlin for providing the English and multilingual version of training data; Liang Xu, Chenjie Cao and the CLUE community for providing the training data and evaluation benechmark of the Chinese version of ALBERT models.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv: 10853, 2018.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Neural Information Processing Systems (NeurIPS), 2019.
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 6, pp. 6-4. Venice, 2006.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth PASCAL recognizing textual entailment challenge. In TAC, 2009.
Daniel Cer, Mona Diab, Encko Aginre, lñigo Lopez-Gazza and Lucia Specia. SemEval-2011 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the Ith International Workshop Semantic Evaluation (SemEval-20017, pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 10.18653/v1/S17-220
URL https: www awwweb.wwc..oo.... htt..//bb..o///tttoooo//////////
Tianqi Chen, Bing Xu, Chiyuan Zhang. and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 1rXiv:1604.06177 2016.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 100..99049, 2019.
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoe V Le. Bam! born-again multi-task networks for natural language understanding. arXiv preprint arXiv:1907.04829 2019.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop, pp. 177-190. Springer, 2005.
Andrew M Dai and Quoe Le. Semi-supervised sequence learning. In Advances in neural infor- mation processing systems, pp. 3079-3087, 2015.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv:1901.02880 2019.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal 2018.
transformers. arXiv preprint
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-144 1423. http::
7 /wwwww ac acwweb. wwwp:/ewwwo.woeeee..e.ee.oeeeeee www.oo/g..taooo.... htth//oohhoooo/99999 1423
William B. Dolan and Chris Brockett. Automatically constructing corpus sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (ZWP2005), 2005. URL
https: wwwww ac www..wbb... wrwgorgggoo....... anthology/ 10////0555501 5002
Zhe Gan, Yunchen Pu, Ricardo Henao, Chunyuan Li, Xiaodong He, and Lawrence Carin. Learn- ing generic sentence representations using convolutional neural networks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2390-2400, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.18655/11/D17-12544 URL htRlsttts:// h/w::///wwww.. acl www../.....///aattoooo/////11111

---

Published as a conference paper at ICLR 2020
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entail- ment and Paraphrasing, pp. 1-9, Prague, June 2007. Association for Computational Linguisties. URL http:ttp: /ttp://wwww /wwww a c web. wttp///nnooolgg////0000--
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual net- work: Backpropagation without storing activations. Advances in neural information processing systems, pp. 2214-2224, 2017.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International Conference on Machine Learning, pp. 2337-2346, 2019.
Edouard Grave, Armand Joulin, Moustapha Cissé, Hervé Jégou, al. Efficient softmax approxima- tion for gpus. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1302-1310. JMLR. org, 2017.
Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2):203-225, 1995. URL https:
77 1/www. www.aclllen www.aalwwe.orr.o http://nnholooyyyggg// 2003
M.A.K. Halliday and Ruqaiya Hasan. Cohesion in English. Routledge, 1976.
Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. Modeling recurrence for transformer. Proceedings of the 20/9 Conference of the North, 2019. doi: 10.
18653/v1/n19-1122 1222. URLhhtpp://dx doi. org/o.. 1tt.:/gg/1..11885///1111991111
Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). arXiv preprint arXiv: 054 2016.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of
the Association for Computational Linguisties: Human Language Technologies, pp. 1367-1377. Association for Computational Linguistics, 2016. doi: 10.18653/ URL http:
//aclweb.org/aaee watzg//atholooy/M.. //Acllebborrr///atbboooo///l// 1162
Jerry R. Hobbs. Coherence and coreference. Cognitive Science, 3(1):67-90, 1979.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification.
arXiv preprint arXiv: 1001-800100000 .06000,646, 2018
Shankar
Nikhil Dandekar, and Kornl Csernai. First quora dataset release: Ques- pairs, January 2017. URL https: //wp:////wwww.// quw.quu..cooo... sttprs.com///uuouraaaaaaaa tion
Iyer,
www.eelleeeeese.tt.eeeeee Questi onn -nniairs
Yacine Jernite, Samuel R Bowman, and David Sontag. Discourse-based objectives for fast unsuper- vised sentence representation learning. arXiv preprint arXiv:1705.00553 2017.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint UrXivv1907. 10529, 2019.
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Ur- tasun, and Sanja Fidler. Skip-thought vectors. In Proceedings of the 28th International Con- ference on Neural Information Processing Systems Volume 2, NIPS' 15, pp. 3294-3302, Cam- bridge, MA, USA. 2015. MIT Press. URL httpp://l. acm. org/ htt./iiaaiiiioo. cfm?id=
2969442. 2969607
Taku Kudo and John Richardson. SentencePiece: A simple and language independent sub- word tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Con- ference Empirical Methods in Natural Language Processing: System Demonstrations, pp- 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 1/115//111118200011 [RI.https.: wwwp://wwww.mmoooo// acl web. hww.w//www.g/aatthooo/////-----0
12

---

Published as a conference paper at ICLR 2020
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, Copenhagen. Denmark, September 2017.
Association for Computational Linguistics. doi: 10. 1.11863//1/DD17-00832 1082. URLhttps://wwat ac Belweb.er wttp/////tttaoo.ee oog//oyy
Quoe Le and Tomas Mikolov. Distributed representations of sentences and documents. In Proceed- ings of the 31st ICML, Beijing, China, 2014.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thir- teenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.
Xiang Li, Shuo Chen, Xiaolin Hu, and Jian Yang. Understanding the disharmony between dropout and batch normalization by variance shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2682-2690, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pre- training approach. arXiv preprint arXiv:1907.1 1692, 2019.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6294 6305. Curran Associates, Inc., 2017. URL http:://ppppptp 7209- ea rned- in-trrn-tntsss lat www.oonnnoottttooeo 1
ni nips cc/paper/
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 3111-3119, 2013.
Allen Nie, Erin Bennett, and Noah Goodman. DisSent: Learning sentence representations from plicit discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics. pp. 4497-4510, Florence, Italy, July 2019. Association for Computational Linguistics, doi: 10.18653/11/119-11 1442. URL U https:t/p/// www wwwwwb.wwb.... wwwp://wwww.../tthooooggo.oo
P19- 1442
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1544 1543, Doha, Qatar, October 2014. Association for Computational Linguisties. doi: 10.3115/v1/D14-111 URL httpt::///// www. ac web, org/ anthol / D14 014-1162
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- Language Technologies, Volume (Long Papers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 1118653/1/NN8-1101 URL
https: www www.acllee we..cwwwbboog org wtrg/anthologge NN10-/NN81-- 1202
Alec Radford, Karthik Narasimhan, Tim Salimans. and Ilya Sutskever. Improving language understanding by generative pre-training. htttp::/ westssswwest22- amazonaws. com/ w http://www..ogggguuuuuuurrreeeee lundrsstanding_paperr pdf 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910. 10683, 2019.
13

---

Published as a conference paper at ICLR 2020
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392 Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. (0.18653/11/D16-111 1264. URL https::/ //www ac web.
org/antho ogy oogphho-111//1111222222
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 1111866/1111118221222 httpss:s:: wwwwww..... acl web.
org httg/anthcllooy/P1-6-11
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensortlow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pp. 10414- 10423, 2018.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Bi-directional block self- attention for fast and memory-efficient sequence modeling. arXiv preprint arXiv:180+ 604.00857, 2018.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model par- allelism, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-16444 Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. htw.tttps:// //wp://wwwwww//.///////////ww ac web. org
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. arXiv preprint arXiv:1908. 1wX0::1908.09355, 2019.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-First AAAI Confer- ence on Artificial Intelligence, 2017.
Iulia Ture, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization knowledge distillation. arXiv preprint 19Xiv:1908.000 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BluckboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Lin- guistics. doi: 10.18653/v/W18. 18-5446. URL https:: http:://wwww.. wwww aclweb. org/anthology/
W18 5446
Wei Wang. Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Liwei Peng, and Luo Si. StructBERT: Incor- porating language structures into pre-training for deep language understanding. arXiv preprint arXiv: /908. 04577, 2019.
Alex Warstadt. Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 12471, 2018
Adina Williams, Nikita Nangia, and Samuel Bowman. A broed-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North
14

---

Published as a conference paper at ICLR 2020
American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume I (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18113/11118-11001. htth:tts:: /wwwww.. aclweb.
org/antholoog Sonnthology//i /Nt///N1--110 8-0101
Zhilin Yang, Zihang Dai. Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv: 2019.
Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. Reducing BERT pre-training time from 3 days to 76 minutes. arXiv preprint IXX4:1904.00000 2019.
Shuailiang Zhang, Hai Zhao, Yuwei Wu, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. DCMN+: Dual co-matching network for multi-choice reading comprehension. arXiv preprint arXiv: 11511, 2019.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov. Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference computer vision, pp. 19-27, 2015.
A APPENDIX
A.1 EFFECT OF NETWORK DEPTH AND WIDTH
In this section, we check how depth (number of layers) and width (hidden size) affect the perfor- mance of ALBERT. Table [I shows the performance of ALBERT-lrge configuration (see Ta- [I using different numbers of layers. Networks with 3 or more layers are trained by fine-tuning using the parameters from the depth before (e.g., the 12-layer network parameters from the checkpoint of the 6-layer network parameters) Similar technique has been used in Gong al. (2019). If we compare a 3-layer ALBERT model with a I-layer ALBERT model, although they have the same number of parameters, the performance increases significantly. However, there are diminishing returns when continuing to increase the number of layers: the results of a 12-layer network are relatively close to the results of 24-layer network, and the performance of a 48-layer network appears to decline.
fine-tuned
Number layers
Parameters
SQuAD1.1
SQuAD2.0
MNLI
SST-2
RACE
Avg
18M
31.1/22.9
50.1/50.1
66.4
80.8
40.1
52.9
3
18M
79.8/69.7
64.4/61.7
77.7
86.7
54.0
71.2
6
18M
86.4/78.4
73.8/71.1
81.2
88.9
60.9
77.2
12
18M
89.8/83.3
80.7/77.9
83.3
91.7
66.7
24
18M
90.3/83.3
81.8/7900
83.3
91.5
68.7
82.1
48
18M
90.0/83.1
81.8/78,9
83.4
91.9
66.9
81.8
Table 11: The effect increasing the number layers for an ALBERT-large configuration.
similar phenomenon, this time for width, can be in Table 12 for a 3-layer ALBERT-large configuration. As we increase the hidden size, we get an increase in performance with diminishing returns. At hidden size 6144, the performance appears decline significantly. We note that none of these models appear to overfit the training data, and they all have higher training and development loss compared to the berf-performing ALBERT configurations.
'If' we compare the performance of ALBERT-large here to the performance in Table 2 we can see that this warm-start technique does not help to improve the downstream performance. However, it does help the 48 layer network to converge. A similar technique has been applied to our ALBBERT-xxlarg,, where we warm-start from 6-yayer network.
15

---

Published as a conference paper at ICLR 2020
Hidden size
Parameters
SQuAD1.1
SQuAD2.0
MNLI
SST-2
RACE
Avg
1024
18M
79.8/69.7
64.4/61.7
77.7
86.7
54.0
71.2
2048
60M
83,3/74.1
69. 1/66.66
79.7
88.6
58.2
74.6
4096
225M
85.0/76.4
71.0/68,1
80.3
90.4
60.4
76.3
6144
499M
84.7/75.8
67.8/65.4
78.1
89.1
56.0
74.0
Table 12: The effect of increasing the hidden-layer size for an ALBERT-large 3-layer configuration.
A.2 Do VERY WIDE ALBERT MODELS NEED TO BE DBEP(ER) TOO?
In Section A. we show that for ALBeRT-large (H=1024), the difference between a 12-layer and 24-layer configuration is small. Does this result still hold for much wider ALBERT configurations such as ALBERT-xxlarge (H=4096)?
Number layers
SQuAD1.1
SQuAD2.0
MNLI
SST-2
RACE
Avg
12
94.0/88.
88,3/85,3
87.8
95.4
82.5
88.7
24
94.1/88.3
88.1/85.1
88.0
95.2
82.3
88.7
Table 13: The effect a deeper network using an ALBERT-xxlarge configuration.
The answer is given by the results from Table 13 The difference between 12-layer and 24-layer ALBERT-xXlarge configurations in terms of downstream accuracy is negligible, with the Avg score being the same. We conclude that, when sharing all cross-layer parameters (ALBERT-style), there is no need for models deeper than 12-layer configuration.
A.3 DOWNSTREAM EVALCATION TASKS
GLUE GLUE is comprised of tasks, namely Corpus of Linguistic Acceptability
(CoLA; Warstadt et al., 2018), Stanford Sentiment Treebank (SST; Socher et al., 2013). Microsoft
Research Paraphrase Corpus (MRPC: Dolan & Brockett] 2005), Semantic Textual Similarity Bench-
mark (STS: Cer et al. 2017). Quora Question Pairs (QQP: Iyer al.] 2017). Multi-Genre NLI
(MNLI: Williams et al. 2018). Question NLI (QNLI: Rajpurkar et al. (20 Recognizing Textual
Entailment (RTE: Dagan et al. 2005 Bar-Haim al., [2006] Giampiccolo al., 2007 Bentivogli et al. 2009) and Winograd NLI (WNLI; Levesque et al. 2012). focuses on evaluating model capabilities for natural language understanding. When reporting MNLI results, only report the "match" condition (MNLI-m). We follow the finetuning procedures from prior work (Devlin et al., 2019 Liu et al., 2019) Yang al. 2019) and report the held-out test performance obtained from GLUE submissions. For test set submissions, we perform sasciipcc modifications for WNLI and QNLI as described by Liu et (2019) and Yang al. (2019).
SQuAD SQuAD is extractive question answering dataset built from Wikipedia. The answers are segments from the context paragraphs and the task is to predict answer spans. We evaluate our models two versions of SQuAD: v1.1 and v2.0. SQuAD v1.1 has 100,000 question/answer pairs. SQuAD v2.0 additionally introduced 50,000 unanswerable questions. For SQuAD v1.1, we the same training procedure BERT, whereas for SQuAD v2.0, models jointly trained with a span extraction loss and an additional classifier for predicting ans unswerabil- ity Yang al. 2019 Liu al. 2019). We report both development set and test set performance.
RACE RACE is large-scale dataset for multi ciolicechoice reading comprehension, collected from En- glish examinations China with nearly 100,000 questions. Each instance in RACE has candidate answers. Following prior work (Yang et al. 2019 Liu et al., 2019), we use the concatenation of the passage, question, and each candidate answer as the input to models. Then, we use the represen- tations from the "[CLS]" token for predicting the probability of each answer. The dataset consists of two domains: middle school and high school. We train our models on both domains and report accuracies both the development set and test
16

---

Published as a conference paper at ICLR 2020
A.4 HYPERPARAMETERS
Hyperparameters for downstream tasks are shown in Table 14
We adapt these hyperparameters
from Liu et al. (2019)
Devlin al. (2019), and Yang al. 2019)
LR
BSZ
ALBERT DR
Classifier DR
TS
WS
MSL
CoLA
1.00E-05
0.1
5336
320
512
STS
2.00E-05
16
0.1
3598
214
512
SST-2
1.00E-05
32
0
0.1
20935
1256
512
MNLI
3.00E-05
128
0
0.1
10000
1000
512
QNLI
1.00E-05
0.1
33112
1986
512
QQP
5.00E-05
128
0.1
0.1
14000
1000
512
RTE
3.00E-05
32
0.1
0.1
800
200
512
MRPC
2.00E-05
32
0
0.1
800
200
512
WNLI
2.00E-05
16
0.1
0.1
2000
250
512
SQuAD
5.00E-05
3649
365
384
SQuAD v2.0
3.00E-05
48
0.1
8144
814
512
RACE
2.00E-05
32
0.1
0.1
12000
1000
512
Table 14: Hyperparameters for ALBERT in downstream tasks. LR: Learning Rate. BS7:: Batch
Size. DR: Dropout Rate. TS: Training Steps. WS: Warmup Steps. MSL: Maximum Sequence
Length.
17


Reviews:
[{"review_id": "rJlMUMIoFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present ALBERT, a modification of the BERT architecture with substantially fewer parameters. They show that despite being much smaller, the performance is very strong and achieves state of the art on a variety of different tasks. There are several ideas proposed here: embedding factorization, sharing layers, and sentence ordering as a training objective. \n\n1. The point that naively increasing the size of the BERT architecture does not work is a good one, but the authors don't acknowledge that this is tied up in the effect of regularization. Cross layer parameter sharing has a regularization effect that simply scaling up BERT large to x-large or such sizes does not have. This is also an issue with the authors making the statement that they are the first to show that dropout is harmful for Transformers. This is a large generalization that seems to be a special case of not only the regularized architecture they propose but also the large quantity of data that the model still underfits to.\n\n2. The authors propose embedding factorization to reduce the number of parameters in the embedding dimension. This is very intuitive, but the authors do not cite or compare to related approaches. I understand these models are computationally intensive and thus do not expect large quantities of detailed ablations. However, this kind of dimensionality reduction has been explored with other techniques, for example for knowledge distillation, quantization, or even adaptive input/softmax (and with subword as well, not just whole word modeling). These techniques have also been applied to machine translation models, which do not use them to learn rare words. I believe a better discussion of these methods should be added to the paper, as this is not a novel proposition.\n\n3. A large takeaway I have from this paper is that parameter size is not a good metric. While ALBERT is substantially smaller, the authors do not make it clear that this model is very slow at inference time due to the large size. This raises several questions: is it better to have models that are deeper or more wide? Can the authors actually report the latency in a comparative table next to BERT? Can the authors provide a sense of how large this model is in MB - e.g. presumably a goal of less parameters would be to have a model with less memory, but then the decision between memory and latency that different models make should be made more clear.\n\n4. Section 4.8 is not clear. Exactly how much data, in terms of GB of uncompressed text, is used here? Is it the data of XLNet and RoBERTa, so larger than both of those settings individually? Further, the authors train for 1 million steps. This is larger than both XLNet and RoBERTa, is that correct? Or there is some detail about the size of the batch that actually makes it comparable? The many small tables where the changes are not clearly delineated makes it difficult to compare results. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SygC0QIhFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper investigates improving upon BERT by reducing complexity in terms of free parameters and memory footprint as well as computation steps. They propose 2 strategies for doing this: 1) Splitting the embedding matrix into two smaller matrices (going from V x A to V x B + B x A where B <<<< A); 2) layer-wise parameter sharing. They also utilize sentence order prediction to help with training. These coupled with a bunch of other choices such as using the lamb optimizer, certain hyperparameters etc help show dramatic empirical gains across the board on a wide variety of NLP/NLU tasks.\n\nPositives: This paper has a dramatic, seemingly statistically significant reduction in error across a wide-variety of tasks. It provides a thorough experimental plan and approaches the few addendums to training (splitting the embedding matrix, the layer-wise parameter sharing, and the sentence order prediction).\n\nConcerns & Questions: There's a lot of experimentation here and a lot of seemingly deliberate choices after seeing empirical results during the research phase. How crucial are the choices of optimizer and other specific hyperparameters? Were there ones you observed that were more brittle than others? Any specific 'reasonable' configurations/settings that caused degenerate solutions?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hkl2NgFecr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new pre-trained BERT-like model called ALBERT. The contributions are mainly 3-fold: factorized embedding parameterization, cross-layer parameter sharing, and intern-sentence coherence loss. The first two address the issue of model size and memory consumption in BERT; the third corresponds to a new auxiliary task in pre-train, sentence-order prediction (SOP), replacing the next sentence prediction (NSP) task in BERT. These modifications lead to a much leaner model and improved performance. As a result, ALBERT pushes the state of the art on GLUE, RACE, and SQuAD while having fewer parameters than BERT-large. \n\nThis is a well-written paper which is easy to follow even for readers without deep background knowledge. The proposed method is meaningful and effective. Its empirical results are impressive. \n\nOther comments:\n\n- Section 4.9. Why use the all-share condition for state-of-the-art ALBERT results (as indicated in Table 2)? Judging from Table 4 and 5, shouldn't the non-shared condition give better results? The number of parameters would be larger, of course. \n\n- I like the justification/motivation given for replacing NSP with SOP. I wonder if the authors have tried other objectives (but didn't work out). Such negative results are valuable to practitioners.\n\n- Typo in Sec. 4.1: x1,1, x1,2 should be x2,1, x2,2. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes three modifications of BERT type models two of which is concerned with parameter sharing and one with a new auxiliary loss. New SOTA on downstream tasks are demonstrated. 

All reviewers liked the paper and so did a lot of comments. 

Acceptance is recommended.
Decision: Accept (Spotlight)