Rethinking Embedding Coupling in Pre-trained Language Models
Authors: Hyung Won Chung
Thibault Fevry
Henry Tsai
Melvin Johnson
Sebastian Ruder
Venue: ICLR 2021 Poster (2021)
Keywords: natural language processing
transfer learning
efficiency
pre-training

Abstract: We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art pre-trained language models. We show that decoupled embeddings provide increased modeling flexibility, allowing us to significantly improve the efficiency of parameter allocation in the input embedding of multilingual models. By reallocating the input embedding parameters in the Transformer layers, we achieve dramatically better performance on standard natural language understanding tasks with the same number of parameters during fine-tuning. We also show that allocating additional capacity to the output embedding provides benefits to the model that persist through the fine-tuning stage even though the output embedding is discarded after pre-training. Our analysis shows that larger output embeddings prevent the model's last layers from overspecializing to the pre-training task and encourage Transformer representations to be more general and more transferable to other tasks and languages. Harnessing these findings, we are able to train models that achieve strong performance on the XTREME benchmark without increasing the number of parameters at the fine-tuning stage. 

Published as a conference paper at ICLR 2021
RETHINKING EMBEDDING COUPLING IN PRE-TRAINED LANGUAGE MODELS
Hyung Won Chung"t
Thibault Févry*i
Google Research twchungf@ooggle.com com
Henry Tsai Google Research henzytsaiegoogle.comm com
Melvin Johnson Google Research me melviap@googie. com
Sebastian Ruder DeepMind
zuder@google..omm
ABSTRACT
We re-evaluate the standard practice of sharing weights between input and out- put embeddings in state-of-the-art pre-trained language models. We show that decoupled embeddings provide increased modeling flexibility, allowing to sig- nificantly improve the efficiency of parameter allocation in the input embedding of multilingual models. By reallocating the input embedding parameters in the Transformer layers, we achieve dramatically better performance ural language understanding tasks with the same number of parameters during fine-tuning. We also show that allocating additional capacity to the output embed- ding provides benefits the model that persist through the fine-tuning stage even though the output embedding is discarded after pre-training. Our analysis shows that larger output embeddings prevent the model's last layers from overspecializ- ing to the pre-training task and encourage Transformer representations to be more general and more transferable to other tasks and languages. Harnessing these find- ings, able to train models that achieve strong performance the XTREME benchmark without increasing the number of parameters at the fine-tuning stage.
standard nat-
INTRODUCTION
The performance models in natural language processing (NLP) has dramatically improved in recent years, mainly driven by advances in transfer learning from large amounts of unlabeled data (Howard & Ruder 2018 Devlin et al.] (019).. The most successful paradigm consists of pre-training a large Transformer (Vaswani et al. (2017) model with self-supervised loss and fine-tuning it on data of downstream task (Ruder et al.) 2019). Despite its empirical success, inefficiencies have been observed related the training duration (Liu et al., 2019b), pre-training objective (Clark al.) 2020b), and training data (Conneau et al.) 2020a), among others. In this paper, we reconsider modeling assumption that may have a similarly pervasive practical impact: the coupling input and output embeddingg Jin state-of-the-art pre-trained language models.
State-of-the-art pre-trained language models (Devlin et al. 2019 Liu al. 2019b) and their multi- lingual counterparts (Devlin et al. 2019 Conneau et al. 2020a) have inherited the practice of em- bedding coupling from their language model predecessors (Press & Wolf, 2017, Inan et al. 2017) However, in contrast to their language model counterparts, embedding coupling in encoder-only pre-trained models such as Devlin et al. (2019) is only useful during pre-training since output em- beddings generally discarded after tuneing In addition, given the willingness of researchers to exchange additional compute during pre-training for improved downstream performance (Raffel
"qqual contribution
WWrk done as a member of the Google Al Residency Program
'Output embedding sometimes referred to "output weights", I.c., the weight matrix in the output
projection in language model.
WW focus encoder-only models, and do not consider encoder-decoder models like T5 (Raffel et al.
2020 where the emhedding matrices discarded after pre-training. Output embeddings may also be

---

Published as a conference paper at ICLR 2021
Table 1: Overview of the number of parameters in (coupled) embedding matrices of state-of-the-an multilingual (top) and monolingual (bottom) models with regard to overall parameter budget. [V] vocabulary size. N, Nemb: number of parameters total and in the embedding matrix respectively.
Model
Languages
[V]
N
Nemb
%Emb.
mBERT (Devlin et al.
2019)
104
120k
178M
92M
52%
XLM- Buse
(Cnneaau et al.
2020a)
100
250k
270M
192.M
71%
XLM- L-RRaagge
Conneau et al.
2020a
100
250k
550M
256M
47%
BERT Base
Devlin et al.
2019
30k
110M
23M
21%
BERT Largo
Devlin et al.
2019
30k
335M
31M
9%
et al.
2020 Brown et al. 2020) and the fact that pre-trained models are often used for inference mil- lions of times (Wolf et al. 2019), cee-training-seecffie parameter savings are less important overall.
On the other hand, tying input and output embeddings constrains the model to use the same dimen- sionality for both embeddings. This restriction limits the researcher's ficxibility in parameterizing the model and can lead to allocating too much capacity to the input embeddings, which may be wasteful. This is a problem particularly for multilingual models, which require large vocabularies with high-dimensional embeddings that make up between 47-71% of the entire parameter budget [ suggesting an inefficient parameter allocation.
In this paper, we systematically study the impact of embedding coupling on state-of-the-art pre- trained language models, focusing on multilingual models. First, we observe that while naively decoupling the input and output embedding parameters does not consistently improve downstream evaluation metries, decoupling their shapes comes with a host of benefits. In particular, it allows us independently modify the input and output embedding dimensions. We show that the input embedding dimension can be safely reduced without affecting downstream performance. Since the output embedding is discarded after pre-training we can increase dimension, which improves fine-tuning accuracy and outperforms other capacity expansion strategies. By reinvesting saved parameters to the width and depth of the Transformer layers, we furthermore achieve significantly improved performance over strong mBERT (Devlin et al. 2019) baseline multilingual tasks from the XTREME benchmark (Hu et al. 2020). Finally, we combine our techniques in a Rebalanced mBER (RemBERT) model that outperforms XLM-R (Conneau al., 2020a), the state-of-the-an cross-lingual model while having been pre-trained on 3.5x fewer tokens and TO more languages.
We thoroughly investigate reasons for the benefits of embedding decoupling. We observe that in- creased output embedding size enables a model to improve on the pre-training task, which correlates with downstream performance. We also find that it leads Transformers that are more transferable across tasks and geguaagss-partcclleee
for the upper-most layers. Overall, larger output embed- dings prevent the model's last layers from over-specializing to the pre-training task (Zhang et al., 2020 Tamkin et al. 2020), which enables training of more general Transformer models.
2 RELATED WORK
Embedding coupling Sharing input and output embeddings in neural language models was pro- posed to improve perplexity and motivated based on embedding similarity (Press & Wolf 2017) as well as by theoretically showing that the output probability space can be constrained to a subspace governed by the embedding matrix for a restricted case (Inan et al., 2017). Embedding coupling is also common in neural machine translation models where it reduces model complexity (Firat et al.)
2016) and saves memory (Johnson et al. 2017), in recent state-of-the-art language models (Melis
al. 2020), well as all pre- trained models
aware of (Devlin et al. 2019 Liu et al. 2019b).
Transferability of representations Representations of large pre-trained models in computer vi- sion and NLP have been observed to transition from general to task-specific from the first to the
useful for domain-adaptive pre-training (Howard & Ruder 2018 Gururangan et al. 2020), prohing Elazar & Goldherg 2019), and tasks that he cast in the pre-training objective (Amrami & Goldberg 2019)
2

---

Published as a conference paper at ICLR 2021
last layer (Yosinski et al. 2014: Howard & Ruder 2018 Liu al., 2019a). In Transformer models, the last few layers have been shown to become specialized to the MLM task and- as a result less transferable (Zhang et al. 2020 Tamkin et al. 2020).
Recent multilingual models pre-trained data covering around 100 languages using a subword vocabulary shared across all languages (Devlin et al.] 2019 Pires et al.) 2019 Conneau et al. 2020a). In order achieve reasonable performance for most languages, these models need to allocate sufficient capacity for each language, known as the curse of multilinguality (Conneau et al. 2020a] Pfeiffer al. 2020). As a result, such multilingual models have large vocab-
Multilingual models
ularies with large embedding sizes to ensure that tokens in all languages are adequately represented.
Efficient models tion (Hinton et al., 2015). Pruning approaches remove parts of the model, typically attention heads (Michel et al.j 2019: Voita et al. 2019) while distillation approaches distill a large pre-trained model into a smaller one (Sun et al. 2020). Distillation can be seen as an alternative form of allocating pre-training capacity via large teacher model. However, distilling pre-trained model is expensive (Sanh et al. 2019) and requires overcoming architecture differences and balancing training data and loss terms (Mukherjee & Awadallah 2020). Our proposed methods simpler and complementary to distillation as they can improve the pre- pre-training of compact student models (Ture et al. 2019).
Most work on more efficient pre-trained models focuses pruning or distilla-
3 EXPERIMENTAL METHODOLOGY
Efficiency of models has been measured along different dimensions, from the number of floating point operations (Schwartz et al. 2019) to their runtime (Zhou et al. 2020). We follow previous work (Sun al., 2020) and compare models in terms of their number of parameters during fine- tuning (see Appendix| A for further justification of this setting). For completeness, we generally report the number of pre-training (PT) and fine-tuning (FT) parameters.
Baseline Our baseline has the same architecture as multilingual BERT (mBERT; Devlin et al. 2019). It consists of 12 Transformer layers with a hidden size H of 768. Input and output embed- dings are coupled and have the same dimensionality The total number of parameters during pre-training and fine-tuning is 177M (see Appendix A.2 for further details). We train variants of this model that differ in certain trained under the same conditions to ensure fair comparison.
as the hidden size, i.e. Fou Fin H.
but otherwise
Tasks
For our experiments, employ tasks from the XTREME benchmark (Hu al. 2020) that require fine-tuning, including the XNLI (Conncau et al. 2018), NER (Pan et al. 2017) ). PAWS-X
Yang et al. 2019). XQuAD (Artetxe et 2020), MLQA (Lewis et al. 2020). and TyDiQA-Goldi Clark et al. (020a) datasets. We provide details for them in Appendix [A.4. We average results across three fine-tuning runs and evaluate on the dev sets unless otherwise stated.
4 EMBEDDING DECOUPLING REVISITED
Naïve decoupling Embeddings make up large fraction of the parameter budget in state-of-the art multilingual models (see Table [I]. We now study the effect of embedding decoupling on such models. In Table 2 we show the impact of decoupling the input and output embeddings in our base- line model (33) with coupled embeddings. Naively decoupling the output embedding matrix slightly improves the performance as evidenced by 0.4 increase on average. However, the gain is not uni- formly observed in all tasks. Overall, these results suggest that decoupling the embedding matrices naively while keeping the dimensionality fixed does not greatly affect the performance of the model. What is more important, however, is that decoupling the input and output embeddings decouples the shapes, endowing significant modeling flexibility, which we investigate in the following.
Input vs output embeddings Decoupling input and output embeddings allows us to flexibly change the dimensionality of both matrices and to determine which one is more important for good transfer performance of the model. To this end, compare the performance of model with
3

---

Published as a conference paper at ICLR 2021
Table Effect of decoupling the input and output embedding matrices on performance on multiple tasks in XTREME. PT: Pre-training. FT: Fine-tuning. The decoupled model has input and output embeddings with the same size (E 768) the embedding the coupled model. The Transformer parts of the models are the same (i.e., 12 layers with H -68).
PT params
# FT params
XNLI Acc
NER
PAWS-X Acc
XQuAD EM/FI
MLQA EM/FI
TyDi-GoldP EM/FI
Avg
Coupled Decoupled
177M 269M
177M
70.7
69.2
85,3
46,2/63.2 46.9/63.8
37.3/5.11 37.3/5..1
40.7/56.7 42.8/58.1
62.3 62.7
177M
71.3
68.9
85.0
Table Performance of models with large input and small output embedding size and vice versa. Both models have 12 Transformer layers with H = 768.
PT params
# FT params
XNLI Acc
NER
PAWS-X
XQuAD EM/FI
MLQA EM/FI
TyDi-GoldP EM/FI
Avg
F1
Acc
En
768,
192M Eow 192M
Eour 128
177M 100M
70.0
68.3
84.3
42.0/60.8
34.7/50.9 34.6/49.5
35.2/52.2
60.1 60.2
Em 128,
70.4
67.6
84.9
43,9/60.0
37.8/510
En 768, Eon = 128 to that of model with En = 128, Eour = 765 (the remaining hyper- parameters are the same as the baseline } During fine-tuning, the latter model has 43% fewer parameters. We show the results in Table Surprisingly, the model pre-trained with larger output embedding size is competitive with the comparison method on average despite having 77M fewer parameters during fine-tuning
Reducing the input embedding dimension saves a significant number of parameters at noticeably smaller cost to accuracy than reducing the output embedding size. In light of this, the parameter allocation of multilingual models (see Table I seems particularly inefficient. For a multilingual model with coupled embeddings, reducing the imput embedding dimension proposed by Lan et al. (2020) is very detrimental to performance (see Appendix A.3 for details).
parameters as
The results in this section indicate that the output embedding plays important role in the transfer- ability of pre-trained representations. For multilingual models in particular, a small input embedding dimension frees up significant number of parameters at small cost performance. In the next section, we study how to improve the performance of model by resizing embeddings and layers.
EMBEDDING AND LAYER RESIZING FOR MORE EFFICIENT FINE-TUNING
Increasing the output embedding size In $4 observed that reducing Exst hurts performance on the fine-tuning tasks, suggesting Evar important for transferability. Motivated by this result, study the opposite scenario, i.c., whether increasing Eout beyond H improves the performance. We experiment with an output embedding size Eout in the range {128, 768, 3072} while keeping the input embedding size En 128 and all other parts of the model the same as described in $ (H = 768, 12 layers, etc).
We show the results in Table 4 In all of the tasks consider, increasing Evut monotonically im- proves the performance. The improvement is particularly impressive for the more complex question answering datasets. It is important to note that during fine-tuning, all three models have the exact same sizes for Fm, and H. The only difference among them is the output embedding, which dis- carded after pre-training. These results show that the effect of additional capacity during pre-training persists through the fine-tuning stage even if the added capacity is discarded after pre-training. We perform an extensive analysis on this behavior in 6 We show results with an English BERT Base model in Appendix A.6 which show the same trend.
linearly project the embeddings from Fmn H and from to Fout- We observe the same trend if we control for the number of trainable parameters during fine-tuning by freezing the input embedding parameters.
4

---

Published as a conference paper at ICLR 2021
Table 4: Effect of an increased output embedding size Æout on tasks in XTREME, All three models have Emn 128 and 12 Transformer layers with H = 768.
PT
XNLI Acc
NER
PAWS-X
XQuAD EM/FI
MIQA EM/FI
TyDi-GoldP EM/FI
Avg
params
params
Acc
Ban 128
115M
100M
68.1 1
65.2
83.3
38.6/54,8
30.9/45.2
32.2/44.2
56.6
Eaut 768
192M
100M
70.4
67.6
84.9
43.9/60.0
34.6/49.5
37.8/51.0
60.2
Fout 3072
469M
100M
71.1
68.1
85.1
45.3/63.3
37.2/53.1
39.4/54.7
61.8
Table 5: Effect of additional capacity via more Transformer layers during pre-training. Both models have Ein 128. The E 768 model has a larger output embedding size Enan and 12 Transformer layers. In contrast, the model with 11 additional Transformer layers has Eour 128. Those addi- tional layers are dropped after pre- pre-training, leaving 12 layers for fair comparison during fine-tuning.
PT params
# FT
XNLI
NER
PAWS-X Acc
XQuAD EM/FI
MLQA EM/F1
TyDi-GoldP EM/F1
Avg
params
Acc
F1
Fou 768 add. layers
192M 193M
100M 100M
70.4
67.6
84.9
43.9/60.0
34.6/49.5
37.8/5.0
60.2
71.2
67.3
85.0
38.
31.4/46.6
31.3/45.5
58.0
Adding capacity via layers We investigate alternative ways of adding capacity during pre-training such as increasing the number of layers and discarding them after pre-training. For a fair comparison with the Eaur 768 model, we add additional layers (total of 23) and drop the 11 upper layers after pre-training This setting ensures that both models have the same pre-tree wee-traiiing and fine-tuning parameters. We show the results 3 The model with additional layers performs poorly on the question answering tasks, likely because the top layers contain useful semantic information (Tenney) ict al. 2019). In addition to higher performance, increasing Eour relies only a more expensive dense matrix multiplication, which is highly optimized on typical accelerators and can be scaled up more easily with model parallelism (Shazeer et al., 2018) because small additional communication cost. We thus focus increasing Eou to expand pre-training capacity and leave exploration of alternative strategies to future work.
Reinvesting input embedding parameters Reducing Ein from 768 128 reduces the number of parameters from 177M to 100M. We redistribute these 77M parameters for the model with Eou. 768 add capacity where it might be more useful by increasing the width or depth of the model. Specifically, we 1) increase the hidden dimension H of the Transformer layers from 768 to 1024 and 2) increase the number of Transformer layers (L) from to 23 at the same H to obtain models with similar number of parameters during fine-tuning.
Table 6 shows the results for these two strategies. Reinvesting the input embedding parameters in both H and L improves performance on all tasks while increasing the number of Transformer layers results in the best performance, with an average improvement of 3.9 over the baseline model with coupled embeddings and the same number of fine-tuning parameters overall.
rebalanced mBERT We finally combine and scale up our techniques to design a rebalanced mBERT model that outperforms the current state-of-the-art unsupervised model, XLM-R (Conneau models strongly depends on their number of parameters (Raffel al.. 2020), we propose a Rebalanced mBERT (RemBERT) model that matches XLM-R's number of fine-tuning parameters (559M) while using a reduced embedding size, resized layers, and more effective capacity during pre-training. The model has a vocabulary size of 250k, Em 256, E.m 1536, and 32 layers with 152 dimensions and 18 attention heads per layer and was trained on data covering 1 110 languages. We provide further details in Appendix A.7
ct al. 2020a). As the performance of Transformer-
We compare RemBERT to XLM-R and the best-performing models the XTREME leaderboard in Table 7 (see Appendix A.8 for the per-task results) 6 The models in the first three rows use
We choose 1024 dimensions to optimize efficient use of our accelerators.
We do consider retrieval tasks they require intermediate task data (Phagg
2020
5

---

Published as a conference paper at ICLR 2021
Table 6: Effect of reinvesting the input embedding parameters to increase the hidden dimension H and number of Transformer layers L on XTREME tasks. Ein 128, Evut 768, H = 768 for all models except for the baseline, which has coupled embeddings and Ein Eou 768.
#
FT paranns
XNLI Acc
NER
PAWS-X
XQUAD EM/FI
MEQA
TyDi-GoldP EM/FI
Avg
puramms
Acc
Bascline
177M
177M
70.7
69.2
46.2/63.2
37.3/53.1
40.7/56.7
62.3
Fin = 128, Foor 768
192M
100M
70.4
67.6
84.9
43.9/60.0
34.6/49.5
37.8/51.0
60.2
Reinvested
260M
168M
72.8
85.6
50.2/67.2
40.7/56.4
44.8/60.0
64.5
Reinvested in L
270M
178M
73.6
71.0
86.7
51.7/68.8
42.4/58.2
48.2/62.9
66.2
Table 7: Comparison of our model to other models on the XTREME leaderboard. Details about VECO are due to communication with the authors.
Add. task data
Trans- lation data
Sentence Classification Acc
Structured Prediction
Question Answering EM/F1
params
# FT params
Langs
Avg
Models fine tuned on trunslations or additional task data
STiLTss Phang el.
2020
559M
559M
100
83.9
69.4
67.2
73.5
FILTEER
2120
559M
559M
100
87.5
71.9
68.5
76.0
VECO
Lao ecain 2020
662M
662M
87.0
70.4
68.0
75.1
Models fine-tuned only on English
XLM-R (Conncau al. 2020s
559M
559M
71.4
RemBERT (ours)
995M
575M
110
84.2
73.3
68.6
75.4
additional task translation data for fine-tuning, which significantly boosts performance (Hu al. 2020). XLM-R and RemBERT are the only two models that are fine-tune tine-tuned using only the English training data of the corresponding task. XLM-R was trained with batch size of 13 sequences each with 2 tokens and 1.5M steps (total of 6.3T tokens). In comparison, RemBERT is trained with 211 sequences of 29 tokens for 1.76M steps (1.8T tokens). Even though was trained with 3.5x fewer tokens and has 10 more languages competiting for the model capacity, RemBERT outperforms XLM-R on all tasks we considered. This strong result suggests that our proposed methods are also effective at scale. We will release the pre-trained model checkpoint and the source code for RemBERT order to promote reproducibility and share the pre-training cost with other researchers.
ON THE IMPORTANCE OF THE OUTPUT EMBEDDING SIZE
We carefully design set of experiments to analyze the impact of an increased output embedding size on various parts of the model. We study the nature of the decoupled input and output represen- tations () and the transferability of the Transformer layers with regard task-specific ($6.2) and laneuage-specific knowledge 6..3
6.1 NATURE OF INPUT AND OUTPUT EMBEDDING REPRESENTATION
We first investigate to what extent the representations of decoupled input and output embeddings differ based on word embedding association tests (Caliskan et al., 2017). Similar Press & Wolf (017), for given pair of words, evaluate the correlation between human similarity judgements of the strength of the relationship and the dot product of the word embeddings. We evaluate on MEN
(Brnni et al. 2014), MTurk771 (Halawi al. 2012). Rare- Rare-Word Word (Luong al. 2013). Sim9 Hill et al. 2015), and Verb-145 143 (Baker et al. 2014). As our model uses subwords, we average the token representations for words with multiple subwords.
We show the results in Table 8 In the first two rows, we can observe that the input embedding of the decoupled model performs similarly to the embeddings of the coupled model while the output

---

Published as a conference paper at ICLR 2021
Table 8: Results on word embedding association tests for the input (I) and output (O) embeddings of models (left) and the models' masked language modeling performance (right). The first two rows show the performance of coupled and decoupled embeddings with the same embedding size Ein = Bout 768. The last three rows show the performance as we increase the output embedding size with Ein 128.
MEN
MTuuk771
Rare- Word
Simlex9yy
Verb-143
MLM acc.
O
Coupled
40.8
37.5
25.0
20.1
56.0
Coupled
Decoupled
39.2
27.7
37.5
24.3
24.0
12.2
17.6
16.1
59.4
43.9
Decoupled
61.6
Equ:
128
40.7
36.6
37.7
32.8
23.6
16.4
17.5
17.3
48.9
46.4
Evn 128
59.0
Fou 768
38.6
27.8
35.2
23.9
22.6
11.5
19.7
15.6
50.6
45.5
Fou = 768
60.7
Fm. 3072
40.1
36.2
8.8
22.6
-1.2
18.9
13.0
43.3
19.5
Fun 3072
62.3
embeddings have lower scores We note that higher scores are not necessarily desirable as they only measure how well the embedding captures semantic similarity at the lexical level. Focusing on the difference in scores, we can observe that the input embedding learns representations that capture semantic similarity in contrast to the decoupled output embedding. At the same time, the decoupled model achieves higher performance in masked language modeling.
The last three rows of Table show that as Ea increases, the difference in the input and output embedding increases as well. With additional capacity, the output embedding progressively learns representations that differ more significantly from the input embedding. We also observe that the MLM accuracy increases with Eour- Collectively, the results in Table 8 suggest that with increased capacity, the output embeddings learn representations that are worse at capturing traditional semantic similarity (which is purely restricted the lexical level) while being task (which requires more contextual representations). Decoupling embeddings thus give the model the flexibility to avoid encoding relationships in its output embeddings that may be useful for its pre-training task. As pre-training performance correlates well with downstream performance (Devlin et al. 2019), forcing output embeddings encode lexical information can hurt the latter.
specialized to the MLM
6.2 CROSS-TASK TRRANFFERABILITY TRANSFORMER LAYER REPRESENTATIONS
We investigate what extent more capacity in the output embeddings during pre-training reduces the MLM-specific burden on the Transformer layers and hence prevents them from over-specializing the MLM task.
Dropping the last few layers We first study the impact of an increased output embedding size on the transferability of the last few layers. Previous work (Zhang al., 2020 Tamkin et al. 2020) randomly reinitialized the last few layers to investigate their transferability. However, those parameters are still present during fine-tuning. We propose more aggressive pruning scheme where we completely remove the last few layers. This setting demonstrates more drastically whether model's s upper layers are over-specialized the pre-training task by assessing whether performance can be improved with millions fewer parameters
We show the performance of models with 8-12 remaining layers (removing up to 4 of the last layers) for different ouiput embedding sizes Eout on XNLI in Figure[] For both Evur 128 and Erar 768, removing the last layer improves performance. In other words, the model performs better even with 7.1M fewer parameters. With Eour 128, the performance remains similar when removing the last few layers, which suggests that the last few layers are not critical for transfcrability.
As we increase E.m. the last layers become more transferable. With E.n. 768, removing more than one layer results in sharp reduction in performance. Finally when E = 3072, every layer is
7 This opposite from what Press & Wolf 2017) observed in 2-layer LSTMs. They find that performance output embedding is similar to the embedding of coupled model. This difference is plausible as the information encoded in large Transformers changes significantly throughout the model (Tenney et al. 2019). Fach Transformer layer with H 768 has about 7. parameters.

---

Published as a conference paper at ICLR 2021
Ear 3072
71
Far 768
70
For 128
accurssy
70
Match
50
69
Exst 3072
Evsi 768
E 128
68
8
10 Number of layers
12
4
5
10 11
Layer
Figure 1:
XNLI accuracy with the last layers removed, Larger Eou improves transferability.
Figure 2: Nearest-neighbor English-to-Gerraa
translation accuracy of each layer.
Table 9: Probing analysis of Tenney et al.
2019 with m xix strategy.
PT params
# FT params
POS
Const.
Deps.
Entilics
SRL
Coref. 0
Coref. W
SPRI
SPR2
Rel.
Avg
128
115M
100M
96.7
87.9
94.3
93.7
91.7
67.2
83.0
82.7
77.0
86.9
96.7
94.4
91.8
78.6
87.1
Eis 3072
469M
100M
96.8
88.0
94.5
94.2
92.0
95,3
67.6
84.1
82.6
78.9
87.4
useful and removing any layer worsens the performance. This analysis demonstrates that increasing Eout improves the transferability of the representations learned by the last few Transformer layers.
Probing analysis We further study whether an increased output embedding size improves the general natural language processing ability of the Transformer. We employ the probing analysis Tenney et al. (2019) and the mix probing strategy where 2-layer dense network is trained on top of weighted combination of the 12 Transformer layers. We evaluate performance with regard to core NLP concepts including part-of-speech tagging (POS), constituents (Consts.), dependencies (Deps.), entities, semantic role labeling (SRL), coreference (Coref.), semantic proto-roles (SPR), and relations (Rel.). For a thorough description of the task setup, see Tenney et al. 2019
We show the results of the probing analysis in Table 9 As we increase Evuts the model improves across all tasks, even though the number of parameters is the same. This demonstrates that increasing Eaut enables the Transformer layers to learn more general representations 10
6.3 CROSS-LINGUAL TRANSFERABILIT OF TRANSFORMER LAYER REPRESENTATIONS
So far, our analyses were not specialized to multilingual models. Unlike monolingual models, multi- lingual models have another dimension of transferability: cross-lingual transfer, the ability to trans- fer knowledge from one language to another.
Previous work (Pires et al. 2019 Artetxe et al. 2020) has found that MLM multilingual data encourages cross-lingual alignment representations without explicit cross-lingual supervision. While it has been shown that multilingual models learn useful cross-lingual representations, over- specialization to the pre-training task may result higher layers being less cross- -ooss-l and fo- cusing language-specift phenomena necessary for predicting the next word in given language. To investigate to what extent this is the case and whether increasing Fautt improves cross-lingual alignment, evaluate the model's nearest neighhour translation accuracy (Pires al. 2019) English-to-Genn German translation (see Appendix A.9 for a description of the method).
9hh probing tasks English while our encoder multilingual. In Tenney et al. (2019). going from BERT-base to BERT-large model (with 3x more parameters) improves performance average 1.1 points, compared to our improvement of 0.55 points without increasing the number of fine-tuning parameters.
8

---

Published as a conference paper at ICLR 2021
We show the nearest neighbor translation accuracy for each layer in Figure 2 As Eour increases, we observe that a) the Transformer layers become more accuracy and b) the amnuage--gnostte representation is maintained a higher layer indicated by flatter slope from layer 7 to 11. In all cases, the last layer is less language-agnosti than the previous one. The sharp drop in performance after layer 8 at Foout 128 is in line with previous results cross-lingual retrieval (Pires et al., [211 Hu et al., (020) and is partially mitigated by an increased Eout- In sum, only does larger output embedding size improve cross-task transferability but also helps with cross-lingual alignment and thereby cross-lingual transfer on downstream tasks.
as evidenced by higher
CONCLUSION
We have assessed the impact of embedding coupling in pre-trained language models. We have iden- tified the main benefit of decoupled embeddings to be the flexibility endowed by decoupling their shapes. We showed that input embeddings can safely reduced and that larger output embeddings and reinvesting saved parameters lead to performance improvements. Our rebalanced multilingual BERT (RemBERT) outperforms XLM-R with the same number of fine-tuning parameters while having been trained on x.5x fewer tokens. Overall, we found that larger output embeddings lead to more transferable and more general representations, particularly in a Transformer's upper layers.
ACKNOWLEDGEMENTS
We would like to thank Laura Rimell for valuable feedback on draft of this paper.
REFERENCES
Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively Multilingual Neural Machine Transla- tion. In Proceedings of NAACL 2019, 2019.
Asaf Amrami and Yoav Goldberg. Towards better substitution based word sense induction. arXiv 2019.
preprint
Mikel Artetxe and Holger Schwenk. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the ACL 2019, 2019.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the Cross-lingual Transferability of Mono- lingual Representations. In Proceedings of ACL 2020, 2020.
Simon Baker, Roi Reichart, and Anna Korhonen. An unsupervised model for instance level subeat- egorization acquisition. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 278-289, 2014.
Ondrej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojevié. Results of the wmt16 metrics shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 199-231, 2016.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askeli, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCand lish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. arXiv e-prints, art. arXiv:2005.14166 May 2020.
Elia Bruni, Kam-Khanh Tran, and Marco Baroni. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1-47, 2014.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356/6334): 2017.

---

Published as a conference paper at ICLR 2021
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: A Benchmark for Information Seeking Question Answer- ing in Typologically Diverse Languages. In Transactions of the Association of Computational Linguistics, 2020a.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre- training Text Encoders as Discriminators Rather Than Generators. In Proceedings of ICLR 2020,
2020b.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations, In Proceedings of EMNLP 2018, pp. 2475-2485, 2018.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020a. Association for Computational Linguistics. doi: 10. 1atp://01122220mmmmmmmma 747. https: 747
//www www.bb................ org/ hwtpaaaog/hooooa00// acl
Alexis Comneau, Shijie Wu, Haoran Li, Luke Zettlemoyer. and Veselin Stoyanov. Emerging cross- lingual structure in pretrained language models. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6022-602262222 2022-6034, Online, July 2020b. Association URL hnttps://www.
for Computational Linguistics. doi: 10. 12200/3/1//220..ccmmmmaaaaa
aclweb. http:/rrraaanooloo////o/0
acl main. 536
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: (0.18653/v1/N19-1440 1423. URL https: 77 7/ www. ac web. wwtp://www.reeeaaoooooooo//////.
Yanai Elazar and Yoav Goldberg. oLMppcs On what Language Model Pre-training Captures. arXiv preprint arXiv: 2019.
Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding. arXiv preprint arXiv:2009.0511 05166600900,,,,, 2020.
Orhan Firat, Baskaran Sankaran, Yaser Al-onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. Zero-Resource Translation with Multi-Lingual Neural Machine Translation. In Proceedings of EMNLP 2016, pp. 268 1288227,220 2016.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of ACL 2020, 2020.
Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. Large-scale learning of word relatedness with constraints. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1406-1414 2012.
Felix Hill, Roi Reichart, and Anna Korhonen. Simlex- 999: Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguistics, 4 41(4):665 695, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. arXiv preprint arXiv:1503.02531. 2015.
Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classifica- tion. In Proceedings of ACI. 2018, 2018
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Gen- cralization. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
10

---

Published as a conference paper at ICLR 2021
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling. In Proceedings of ICLR 2017, 2017.
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation. Transactions of the ACL 2017, 2017.
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations, 2020. URL
https:://ppenee htwp:a/rreeeeeeeeoooouiioeioooo Hote:ooommmmii-eeeeeeree
net/
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv e-prints, art. arXiv:2001.0836/ January 2020.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent sub- word tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Con- ference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-2000 URLhttps://wwwe ac http://wwww..bb..co///aoaoooo///
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In International Conference on Learning Representations, 2020. URL (ttLttts:////// //pprr/iie//peeeeie....
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling Giant Models with Condi- tional Computation and Automatic Sharding. arXiv e-prints, art. arXiv:2006. 16668, June 2020.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MLQA: Evaluat- ing Cross-lingual Extractive Question Answering. Proceedings of ACL 2020, 2020.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic Knowledge and Transferability of Contextual Representations. In Proceedings of NAACL 2019, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. arXiv preprint 19Xiv:1907.11699 2019b.
Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, and Luo Si. VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation. arXiv e- prints, art. arXiv:2010.16046 October 2020.
Minh- Shuh-TThagg/ Luong, Richard Socher, and Christopher D Manning. Better word representations with recursive neural networks for morphology. In Proceedings of the Seventeenth Conference Computational Natural Language Learning, pp. 104-113, 2013.
Gábor Melis, Tomáš Kočiský, and Phil Blunsom. Mogrifier LSTM. In Proceedings of ICLR 2020, 2020.
Paul Michel, Omer Levy, and Graham Neubig. Are Sixteen Heads Really Better than One? In Proceedings of NeurlPS 2019, 2019.
Subhabrata Mukherjee and Ahmed Hassan Awadallah. Xtrrme Multi-stage Distillation for Massive Multilingual Models. In Proceedings of 2020, pp. 2221-22344 2020.
Joakim Nivre, Mitchell Abrams, Željko Agié, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aran- zabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. Universal de- pendencies 2.2. 2018.
11

---

Published as a conference paper at ICLR 2021
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross- lingual name tagging and linking for 282 languages. In Proceedings of ACL 2017, pp. 1946-1958, 2017.
Jonas Pfeiffer, Ivan Vuli, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-based Frame- work for Multi-task Cross-lingus Transfer. In Proceedings of EMNLP 2020, 2020.
Jason Phang, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Iacer Calixto, and Samuel R Bowman. English intermediate-task training improves zero-shot cross- lingual transfer too. arXiv preprint arXiv:2005. 13013, 2020.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4996-5001, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/P19- 1493. URL
Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 157-163, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https: www. wwwwwb.ww........... http://ww.rrr/tthhoogggggooee
2025
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
074. html
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of EMNLP 2016, 2016.
Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing- In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 15-18, 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint 191.:v:911.011100 2019.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. arXiv:1907.10500 2019.
Green AL arXiv preprint
Noam Shazeer, Youlong Cheng. Niki Parmar, Dustin Tran, Ashish Vaswani. Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomput- ers. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10414- 10423. Curran Associates, Inc., 2018. URL
nips. cc/paper/ for wwwe--ooeecccoppteesss....
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Par- allelism. arXiv e-prints, art. 1XXi:19909000533 September 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT a Compact Task- Agnosticc BERT for Resource-Limited Devices. In Proceedings of ACL 2020, pp. 2158-21700 2020.
Alex Tamkin, Trisha Singh, Davide Giovanardi, and Noah Goodman. Investigating Transferability in Pretrained Language Models. arXiv e-prints, art. arXiv:2004. 14975, April 2020.
Ian Temney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp- Florence, Italy, July 2019. Association for Computational Linguistics. doi: 18.18653/ 1/P119-1452 URL https::::: www. I web. www.:w....ae.e... web..org/aattholog////////
12

---

Published as a conference paper at ICLR 2021
Iulia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well- Students Learn Better: On the Importance of Pre-training Compact Models. arXiv preprint arXiv:1908.08962. 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. URL.
nips. http:ccccpppppceceeeeeee iotteeeirriaaaattteeiooo------- is all- you---oo--eeeee pdf
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing Multi-Head Self-Attentio:: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. In Proceedings of ACL 2019, 2019.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of NAACL-HLT 2018, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. HuggingFace's Trans- formers: State-of-the -frtt---tttttttt Natural Language Processing. arXiv preprint arXiv: vrXiv:1910.0377/1 2019.
Shijie Wu and Mark Dredze. Beto, bentz, becas:
The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 833-844, Hong Kong, China, November 2019, Association for Computational Lin- guisties. doi: 10.18653/11/D19-10777 URL https: wwwp:./cwweb..rr//atthh...oooo D19-1077
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of EMNLP 2019, pp. 1695-, 2019.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320-3328, 2014.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi. Revisiting Few- sample BERT Fine-tuning. arXiv e-prints, art. arXiv:2006.05987, June 2020.
Xiyou Zhou, Zhiyu Chen, Xiaoyong Jin, and William Yang Wang. HULK: An Energy Ef- ficiency Benchmark Platform for Responsible Natural Language Processing. arXiv preprint arXiv:2002.05829, 2020.
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the third buce shared task: Spotting parallel sentences in comparable corpora. In Proceedings of 1 1Ith Workshop on Building and Using Comparable Corpora, pp. 39-42, 2018
A APPENDIX
A.1 EFFICIENCY COMPARISON BASED ON PARAMETER COUNT DURING FINE-TUNING
We compare the efficiency of models based on their number of parameters. We believe this to be a reasonable proxy for a model's 's efficiency as the performance of Transformer-basee language models has been shown to improve monotonically with the number of parameters (Kaplan et al. 2020 Raffel
et al., 2020: Lepikhin et al. 2020 Brown et al. 2020 (hhoeybi et al. |019| Aharoni et al. 2019).
As the number of parameters during pre- pre-training and one-ttnnng may differ we compare models based their number of parameters during the fine-tuning stage (without the task-specific head). We argue that this is the most practically relevant number as a model is generally pre-trained only once but may be fine-tuned used for inference millions of times.
For encoder-only models such as BERT, parameters after the last Transformer layer (e.g. the output em- heddings and the pooling layer) are discarded after pre-training.
13

---

Published as a conference paper at ICLR 2021
Table 10: Fine-tuning hyperparameters for all models except RemBERT.
Learning rate
Batch size
Train epochs
PAWS-X
[3 10-5 x 10-5; 5 10-51
32
3
XNLI
x 10
2 10-5 3 10-5
32
3
SQUAD
x 10 3 x
10
4 10
32
NER
x 10-3 2 1
x 10
32
3
Table 11: Statistics for the datasets in XTREME, including the number of training, development, and test examples as well as the number of languages for each task.
Task
Corpus
Train
Dev
[est]
[Lang.
Task
Metric
Domain
XNLI
392,702
2,490
5,010
15
NLI
Acc.
Mise.
Classification
PAWS-X
49,401
2,000
2,000
Paraphrase
Acc.
Wiki Quora
Structured
POS
21,253
3,974
47-20,436
33
POS
FI
Misc.
prediction
NER
20,000
10,000
1,000-10,000
40
NER
F1
Wikipedia
QA
MI QIQQ
XQuAD
87,599
34,726
4,517-11,590
1,190
11
7
Span extraction
Spon extraction
F1 EM
F1 / EMM
Wikipedia
Wikipedia
DDiQA-GoldP
3,696
634
323-2,719
Span extraction
FI / EM
Wikipedia
BUCC
1,896-1.330
Retrieval
F1 Acc.
Wiki news
Retrieval
Tatoeba
1,000
33
Retrieval
A.2 BASELINE MODEL DETAILS
Our baseline model has the same architecture multilingual BERT (mBERT: Devlin al. 2019) It consists of 12 Transformer layers with a hidden size H of 768 and 12 attention heads with 64 dimensions each. Input and output embeddings are coupled and have the same dimensionality E as the hidden size, i.e. Eout Em = H. The total number of parameters during pre-training and fine-tuning 177M. We do not use dropout following the recommendation from Lan et al. 2020). We use the SentencePiece tokenizer (Kudo & Richardson) 2018) and a shared vocabulary of 120k subwords. The model trained on Wikipedia dumps in 104 languages following DDvlin et al.) (2019) using masked language modeling (MLM). We choose this baseline as its behavior has been thoroughly studied (K et al. 2020 Conneau et al. 2020b; Pires et al. 22019) Wu & Dredze 2019).
A.3 TRAINING DETAILS
For all pre-training except for the large scale RemBERT, we trained using 64 Google Cloud TPUs. We trained over 26B tokens of Wikipedia data. All fine-tuning experiments were run on 8 Cloud TPUs. For all fine-tuni tinettunig experiments other than RemBERT, we use batch size of 32. We sweep over the learning rate values specified in T0
We used the SentencePiece tokenizer trained with unigram language modeling
A.4 XTREME TASKS
For our experiments, we employ tasks from the XTREME benchmark (Hu al. 2020). We show
statistics for them in Table I1 XTREME includes the following datasets: The Cross-lingual Natu-
ral Language Inference (XNLI: Conneau al. 2018 corpus, the Cross-lingual Paraphrase Adver- saries from Word Scrambling (PAWS-X: Yang et al. 2019) dataset, pfrt-of-speech (POS) tagging data from the Universal Dependencies v2.5 (Nivre et al. 2018) treebanks, the Wikiann (Pan al., 2017) dataset for named entity recognition (NER), the Cross-lingual Question Answering Dataset (XQuAD; Artetxe et al. 2020), the Multilingual Question Answering (MLQA; Lewis et al. 2020
dataset, the gold passage version of the Typologically Diverse Question Answering (TyDiQA Clark
al. 2020a) dataset, data from the third shared task of the workshop on Building and Using Parallel
Corpora (BUCC: Zweigenbaum et al. 2018). and the Tatoeba dataset (Artetxe & Schwenk 2019).
We refer the reader to Hu et al. (2020) for more details. We average results across three fine-tuning runs and evaluate the dev sets unless otherwise stated.
14

---

Published as a conference paper at ICLR 2021
Table 12: Effect of reducing the embedding size for monolingual vs. multilingual models on MNLI and XNLI performance respectively. Monolingual numbers are from have vocabulary size of 30k.
Lan et al. (0200) and
English
PT params
FT params
MNLI
Multilingual
# PT params
# params
XNLI
== H 768
110M 89M
110M 89M
84.5
H
177M 100M
177M 100M
70.7
== 128
83.7
1 128
68.1
Table 13: Effect of increased output embedding size Eaut and additional layers during pre-training 128).
= 15 on English BERT Base (Fin
# PT
# FT params
MNLI Acc
SQuAD EM/FI
params
BERT Base (ours)
110M
110M
79.8
78.4/86.2
Eout 128
93M
89M
75.9
75.5/84.2
Faua 768
112M
89M
77.5
77.5/85.5
FuRI 3072
181M
89M
79.5
78.4/86.2
L = 15
114M
89M
80.1
78.7/86.3
L 24
178M
89M
79.0
77.8/85.5
A.5 COMPARISON TO LAN ET AL. (2200)
Crucially, finding differs from the dimensionality reduction in ALBERT (Lan al. 2020 While they show that smaller embeddings can used, their imput and output embeddings are cou- pled and use much smaller vocabulary (30k 120k). In contrast, find that simultaneously decreasing both the input and output embedding size drastically reduces the performance of multi- lingual models.
In Table 12 we show the impact of their factorized embedding parameterization on monolingual and multilingual model. While the English model suffers smaller (0.8%) drop in accuracy, the multilingual model's performance drops by 2.6%. Direct application of a factorized embedding parameterization (Lan et al. 2020) is thus not viable for multilingual models.
A.6 ENGLISH MONOLINGUAL RESULTS
So far, have focused multilingual models the number of saved parameters when reducing the input embedding size is largest for them. We now apply the same techniques to the English 12-layer BERT Rase with a 30k vocabulary (Devlin al. 2019). Specifically, we decouple the em- beddings, reduce Em to 128, and increase the output embedding size or the number of layers during pre-training. We show the performance on MNLI (Willaans al.) 2018) and SQuAD (Rajpurkar et al. 2016) in Table 13 By adding more capacity during pre-trainin, performance monotonically increases similar the multilingual models. Interestingly, pruning 24-layer model to 12 layers reduces performance, presumably because some upper layers still contain useful information.
A.7 REMBERT DETAILS
We design a Rebalanced mBERT (RemBERT) to leverage capacity more effectively during pre- training. The model has 995M parameters during pre-training and 575M parameters during fine- tuning. We pre-train large unlabeled text using both Wikipedia and Common Crawl data, covering 110 languages. The details of hyperparameters and architecture are shown in Table 14
For each language I, we define the empirical distribution as
Th
P1
ΣVED T4
15

---

Published as a conference paper at ICLR 2021
Table 14: Hyperparameters for RemBERT architecture and pre-training.
Hyperparameter
RemBERT
Number of layers
32
Hidden size
1152
Vocabulary size
250,000
Input embedding dimension
2.66
Output embedding dimension
1536
Number of attention heads
18
Attention head dimension
64
Dropout
0
Learning rate
0.0002
Batch size
2048
Train steps
1.76M
Adam 31
0.9
Adam 32
0.999
Adam
10-6
Weight decay
0.01
Gradient clipping norm
Warmup steps
15000
Table 15: Hyperparameters for RemBERT fine-tuning.
Learning rate
Batch size
Train epochs
PAWS-X
8 X 10
128
3
XNLI
1 X 10-5
128
3
SQuAD
9 x 10-6
128
POS
3 X 10
128
3
NER
8 x 10-6
64
3
where ny is the number of sentences in I's pre-training corpus. Following Devlin et al. (2019), we exponentially smoothed distribution, i.e., exponentiaate pt by 0.5 and renormalize to obtain the sampling distribution.
Hyperparameters and pre-training details summarized in Table 14 Hyperparameters used for the leaderboard submission are shown Table 15
A.8 XTREME TASK RESULTS
We show the detailed results for RemBERT and the comparison per task on the XTREME leaderboard in Table 16, Compared to Table 7 which shows the average across task categories, this table shows the average across tasks.
A.9 NEAREST-NEIGHROR TRANSLATION COMPUTATION
For an English-te to German translation, we sample M = 5000 pairs of sentences from WMT16 (
Bojar et al. 2016). For each sentence in each language, we obtain representation LANG at each layer by averaging the activations of tokens (except the [CLS] and [SEP] tokens) at that layer. We then compute translation vector from English to German by averaging the difference between
the vectors of each sentence pair across all pairs: UN-
(t)
37 ΣM
UDE
(t) (1)
UEN,
16

---

Published as a conference paper at ICLR 2021
For each English sentence VEN, locate the closest German sentence vector based on &2 distance and measure how often the nearest neighbour is the correct pair.
(8)
translate it with this vector: VEN, EN-DDE We
(1)
Table 16: Comparison our model to other models on the XTREME leaderboard. Details about VECO are due to communication with the authors. Avgua is averaged over tasks whereas Avg is averaged over task categories just like Table 7
PT
FTT
XNLI Acc
POS
NER
PAWS-X
XQuAD
MLQA
Goldd
Av
Avg.
params
params
Modeels fine-tumed on transaations or additional tack duta
STiLTs [Phang al 20201
559M
S59M
80.0
74,9
64.0
87.9
63.3/78.7
53.7/7244
59.5/76.0
72.7
73.5
FILTER
50.9/68.3
VECO Lue 2020
662M
662M
83.0
75.1
65.7
91.1
66.3/79.9
54.9/73.1
58.9/75.0
74.1
75.1
Modell, tnmed only Eigglihh task data
(Conneau al.
60.8/76.6
53.2//1.6
45.0/65.1
RemBERT (ours]
995M
575M
80.8
76.5
70.1
87.5
64.0779.6
$5.073.11
63.0777.0
74.4
75.4
17


Reviews:
[{"review_id": "LlWEQtK73-r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work studied the impact of embedding coupling in pre-trained language models, by taking a multilingual model as backbone. The major finding is that decoupling the input and output embedding shapes can bring benefits, and the output embedding plays an important role in the transferability of pre-trained representation. A rebalanced mBERT is designed by combing and scaling up the investigated techniques, achieving strong results on the XTREME benchmark.\n\nThis paper is well written, and the proposed strategy is simple yet effective for obtaining more transferable language representations. The insights of model design for more efficient fine-tuning are well supported by the analysis.\n\nMy concern is about the model efficiency and the true source of performance improvement. It seems that the number of the parameters is much more than those public ones. I am curious if it is a fair comparison in Table 7. \n\nThe current approach does help in reducing the number of parameters in the fine-tuning stage by increasing E_{out}. Even in your all experiments, the optimal value of E_{out} appears to be 768 (similar in the baseline), where performance is quite similar to baseline. However, increasing E_{out} to a much larger value (i.e 3072) drastically increases the number of parameters in pre-training, even as compared to baseline, which seems like a trade-off between fine-tuning and pre-training. The performance with much larger E_{out} is a marginal improvement over baseline unless the saved parameters are reinvested. The reinvestment of parameters creates a larger improvement over baseline (Table 6). The authors must conclude with optimal values of E_{in} and E_{out}, otherwise the paper is merely a series of experiments with different values of E_{in}, E_{out}, and # of layers in the baseline.\n\nTry referencing Table 7 in the main explanation of RemBERT rather than in the Appendix, as it is your major contribution. But again my concern is the same, the number of parameters in fine-tuning is larger than XLM-R plus the additional pre-training time (more than XLM-R). Yes, the performance increases but the main objective of the paper is not quite satisfied (reducing the number of parameters in FT which are actually larger in RemBERT than XML-R). Maybe you can alter the abstract accordingly and focus on reinvestment, which actually is helping in making your point in the experiments.\n\n\n\nMinor comments:\n\nCheck the use of \\citep{} and \\citet{}, e.g., in the last sentence of Section 2 and Footnote 10.\n\nTable 7 as main results was never cited in this paper (but only in appendix), which is not a proper organization way.\n\nPage 3-> \"It consists of 12 Transformer layers with a hidden size H of 768 and 12\" the sentence seems incomplete.\n\nPage 5 -> \"increasing the number of Transformer layers L results in the best performance\" I am wondering if reinvestment in L gains higher performance than in H because of the larger number of parameters (10M more). Maybe you can keep reinvestment the same in both H and L for fair comparison and better results projection.\n\nSection 4: How about a proportional increase in E_{in} and E_{out} affects the performance (e.g increasing or decreasing both with the same proportion, especially decreasing which lines up with paper goal i.e providing flexibility with less number of parameters.)\n\nPage 7 -> \"For both E_{out} = 128 and E_{out} = 768, removing the last layer improves performance\". Seems in disagree with Figure 1. Acc with 10 layers is lower than that with 12 layers for E_{out}=128. Similar is the case with E_{out}=768.\n\nHow about the extra pre-training and fine-tuning time compared with the baseline? \n\nFor the final rebalanced mBERT described in page 5, how are the hyper-parameters decided? \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HN8h7VWbbR7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Transformer based bidirectional LMs pre-trained using Masked Language Model loss typically share input and output token embeddings. This paper makes an interesting investigation about decoupling input and output embeddings and gains which can be obtained out of this decoupling. In particular, this paper shows that the pre-training performance of transformers and the transferability of the learned representations can be improved by increasing the dimension of output embeddings while reducing the dimension of input embeddings. Better performance while pre-training and improved transferability further helps the performance while finetuning on downstream tasks. Parameters saved while reducing the dimensions of input embeddings can be re-invested into increasing the depth or width of the transformer layers. I believe that the findings in this paper are going to be useful in practice to the general NLP community working on Transformers and Multilingual problems.\n\nWeak Points:\n1. In table 3, (E_in=768,E_out=128) is just 0.1 point worse on average than (E_in=128, E_out=768).  Yet, authors draw some conclusions on Page 4 based on this table, without reporting any variance or statistical significance tests.\n    E.g. \n    >  the model pretrained with a larger output embedding size slightly outperforms the comparison method on average despite \n    having 77M fewer parameters during fine-tuning\n\n    > Reducing the input embedding dimension saves a significant number of parameters at a noticeably smaller cost to accuracy \n    than reducing the output embedding size.\n\nI would request the authors to report the variance in Table 3 and also in Table 9. Otherwise, it’s hard to rely on conclusions drawn from such minor differences in performance.\n\n\nStrong Points:\nStrong empirical results.\n1. Table 4 clearly shows that increasing the dimension of output embeddings improves transfer to downstream tasks while having the same number of trainable parameters during the finetuning stage.  \n2. Table 6 shows that reinvesting the parameters saved from reducing the dimension of input embeddings into additional transformer layers yield improved performance. Further, RemBERT performs at par with XLM-R while having a comparable number of trainable parameters during the finetuning stage.\n3. Sufficient amount of analysis in Section 6 to establish the usefulness of having higher dimensional output embeddings for improved pre-training and better transferability of the learned representations across tasks. \n\nOther comments/questions:\n1. Shouldn’t #PT params in table 2 for the “Decoupled” method be more than #PT params for the “Coupled\" method ?\n2. More figures like Figure 1, on other tasks in addition to XNLI, would be really helpful in making the observations more conclusive.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "959iTeqc8ov", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n \nThis work investigated the strategy of reallocating parameters of multilingual language models for improving their cross lingual transferability. Authors first decoupled the input and output embeddings and showed that the capacity of output embedding is more important than input embedding. Then,  they proposed a Rebalanced mBERT (RemBERT) model that reallocates the input embedding parameters of mBERT model to the output embedding and additional layers. Experimental results on XTREME benchmark showed that RemBERT significantly outperformed XLM-R with similar model size.\n\nPros:\n \n- The paper is well written and easy to follow. The comparison of embeddings parameters ratios of different language models in Table 1 gives a very good motivation of reallocating the parameters of embeddings.\n \n- Authors conducted several ablation studies to understand how the capacity of  different parts of the model (e.g., input and output embeddings, transformer layers) contributes to the final performance. \n\nCons:\n\n- Reallocating the parameters of input embedding to additional transformer layers might affect the pre-training and inference speed, it will be helpful to show the training and inference speed of different parameter reallocation strategies.\n \nQuestions:\n\n- Why are the pre-training and fine-tuning parameters of coupled and decoupled models the same in Table 2?  Shouldn’t decoupling the input and output embeddings double the parameters of the embeddings?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0r5Css9Y2qM", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper systematically studies the impact of embedding coupling with multilingual language models.  The authors observe that while na¨ıvely decoupling the input and output embedding parameters does not consistently improve downstream evaluation metrics, decoupling their shapes comes with a host of benefits. Moreover, they achieve significantly improved performance by reinvesting saved parameters to the width and depth of the Transformer layers on the XTREME benchmark.\n\nThis paper is well-written and strongly motivated. The idea of decoupling embedding is novel, and the evaluation results are strong.  \n\nStrength:\n\n+ The systematical study of the impact of embedding coupling on state-of-the-art pre-trained language models.  This paper also thoroughly investigates reasons for the benefits of embedding decoupling and observes that an increased output embedding size enables a model to improve on the pre-training task, which correlates with downstream performance. They also find that it leads to Transformers that are more transferable across tasks and languages. Those empirical results will promote future model design and the understanding of the transformer for textual representation learning.\n+ The paper proposes a method to reinvest saved parameters to the width and depth of the Transformer layers and achieve significantly improved performance on the XTREME benchmark over a strong mBERT.\n\nWeakness:\n\n- Some model details are missing. Although I know what the input and output embedding are, it is still a bit hard to follow in Section 4 and 5. I strongly recommend that the authors revise those parts and introduce the model details, such as decoupling and model architecture in your experiments.\n- Lots of your model designs are empirical such as the embedding size, and it is a bit boring to optimize those hyperparameters, and sometimes we even do not know why it works.\n\nQuestions:\n\n- Could you please introduce the model details of your decoupled model in Section 4?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes a thorough investigation on the idea of decoupling the input and output word embeddings for pre-trained language models.  The research shows that the decoupling can improve the performance of pre-trained LMs by reallocating the input word embedding parameters to the Transformer layers, while further improvements can be obtained by increasing the output embedding size.  Experiments were conducted on the XTREME benchmark over a strong mBERT.  R1&R2&R3 gave rather positive comments while R4 raised concerns on the model size.  The authors gave detailed response on these concerns but R4 still thought the paper is overclaimed because the experiments were only conducted in a multilingual scenario.  
Decision: Accept (Poster)