Representation Deficiency in Masked Language Modeling
Authors: Yu Meng
Jitin Krishnan
Sinong Wang
Qifan Wang
Yuning Mao
Han Fang
Marjan Ghazvininejad
Jiawei Han
Luke Zettlemoyer
Venue: ICLR 2024 poster (2024)
Keywords: Masked Language Modeling
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We identify and address a representation deficiency issue when adapting an MLM-pretrained model to sequences without $\texttt{[MASK]}$ tokens.

Abstract: Masked Language Modeling (MLM) has been one of the most prominent approaches for pretraining bidirectional text encoders due to its simplicity and effectiveness. One notable concern about MLM is that the special $\texttt{[MASK]}$ symbol causes a discrepancy between pretraining data and downstream data as it is present only in pretraining but not in fine-tuning. In this work, we offer a new perspective on the consequence of such a discrepancy: We demonstrate empirically and theoretically that MLM pretraining allocates some model dimensions exclusively for representing $\texttt{[MASK]}$ tokens, resulting in a representation deficiency for real tokens and limiting the pretrained model's expressiveness when it is adapted to downstream data without $\texttt{[MASK]}$ tokens. Motivated by the identified issue, we propose MAE-LM, which pretrains the Masked Autoencoder architecture with MLM where $\texttt{[MASK]}$ tokens are excluded from the encoder. Empirically, we show that MAE-LM improves the utilization of model dimensions for real token representations, and MAE-LM consistently outperforms MLM-pretrained models on the GLUE and SQuAD benchmarks.

Published as a conference paper at ICLR 2024
REPRESENTATION DEFICIENCY IN MASKED LANGUAGE MODELING
Yu Meng's Jitin Krishnan Sinong Wang²
Qifan Wang² Yuning Mao?
Han Fang² Marjan Ghazvininejad
Jiawei Han Meta Al
Luke Zettlemoyer
University Illinois Champaann (yumeng5, (anj)iillinois edu "(jitinkrishnan, sinongwang, wafer, yuningm, hanfang, ghazvini, 1s2] @mete. com
ABSTRACT
Masked Language Modeling (MLM) has been one of the most prominent ap- proaches for pretraining bidirectional text encoders due to its simplicity and ef- fectiveness. One notable concern about MLM is that the special [MASK] symbol causes a discrepancy between pretraining data and downstream data as it is present only in pretraining but not in fine-tuning. In this work, offer tive on the consequence of such a discrepancy: We demonstrate empirically and theoretically that MLM pretraining allocates some model dimensions exclusively for representing [MASK] tokens, resulting in representation deficiency for real tokens and limiting the pretrained model's expressiveness when it is adapted downstream data without [MASK] tokens. Motivated by the identified issue, we propose MAE-LMM, which pretrains the Masked Autoencoder architecture with MLM where [MASK] tokens are excluded from the encoder. Empirically, we show that MAE-LM improves the utilization of model dimensions for real token representations, and MAE-LM consistently outperforms MLM -l-e--rrrr models on the GLUE and SQuAD benchmarks.
perspec-
INTRODUCTION
Pretraining text encoders to learn from bidirectional contexts has achieved enormous success in various natural language processing (NLP) tasks (Clark et al., 2020 Devlin et al., 2019 Liu et al. 2019). Masked Language Modeling (MLM) (Devlin et al.) 2019) is among one of the most prominent pretraining approaches due to its conceptual simplicity and empirical effectiveness: By randomly masking portion of input tokens and training Transformer encoder to predict the original content based on the remaining bidirectional contexts, the model learns robust representations that generalize well to diverse downstream tasks. Besides its broad impact in NLP, MLM has also been widely adopted for pretraining in other domains, such as images (Bao et al. 2022 Xie et al. 2022).
videos (Tong et al. 2022 Wang et al. 2022) and graphs (Hou et al. 2022
Despite its remarkable success, the effectiveness of MLM may hindered by discrepancy be- tween pretraining and fine-tuning: The special [MASK] token occurs only in pretraining but not in downstream tasks. While a few previous studies (Clark et al., 2020; Yang et al.) 2019) have at- tempted address this issue, they end up proposing new training objectives instead of systematically investigating why and how such a discrepancy impacts the generalization of ML. --petrrrade models.
In this work, study the consequence of including [MASK] tokens in MLM pretraining by examining the learned token representation space. We empirically and theoretically show that [MASK] token representations exclusively occupy some model dimensions, thereby reducing the model capacity for representing real tokens. Such representation deficiency issue may not be simply addressed by fine-tuning on downstream tasks: Those dimensions exclusively used for [MASK] tokens have not been pretrained to represent real tokens, and will have to be either trained from scratch on downstream data, raising the risk of overfitting (Hendryeks et al. 2019 Kumar al. 2022). or become unused, resulting in waste of model capacity.
Work done during internship Meta AI.

---

Masked Language Modeling (MLM). Given a text sequence x [x1,.... .......... Iz..... 2n), MLM randomly replaces a set of token positions M with [MASK] symbols. The resulting partially masked sequence = [ [MASK] i...., Ønl is then fed to the Transformer encoder O which outputs Published as a conference paper at ICLR 2024
To address the representation deficiency issue, we propose simple text encoder pretraining method, MAE-LM, which conducts MLM pretraining based on the Masked Autoencoder architecture (He et al. 2022). Notably, [MASK] tokens are omitted from the encoder's 's input that the real token representations can utilize the entire model dimensions theoretically. An auxiliary decoder, used only in pretraining and not in fine-tuning, takes the encoder's s output representations and [MASK] positions to predict the original tokens. We demonstrate empirically that by excluding [MASK] tokens from the encoder, MAE-LM improves the utilization of model dimensions both in pretraining and downstream tasks and achieves consistent and notable improvements over previous models pretrained by MLM and its variants on the GLUE and SQuAD benchmarks
Our main contributions are as follows: (1) We investigate the token representation space trained by ML,,, and identify previously unknown representation deficiency issue when the pretrained model is applied to real data without [MASK] tokens. (2) Based on empirical and theoretical analyses, we explain why the representation deficiency issue occurs in the conventional MLM pretraining setup. (3) We show that simple pretraining method MAE-LM can address the identified issue and improve the downstream task performance of previous ML -rretrrrra models under multiple pretraining and fine-tuning settings.
2
ANALYSIS OF TOKEN REPRESENTATIONS MLM
2.1 PRELIMINARIES
Transformer Encoder. Transformer encoders contain multiple Transformer layers, where each layer consists of two submodules, multi-head self-attention (MHSA) and feed-forward network (FFN). The self-attention mechanism uses queries Q and keys K to compute attention weights, and outputs weighted sum of the values V. MHSA performs self-attention in parallel over N heads as follows:
QK
Attn (Q, K, V) = SSofmmx
vdn
M = Concat(headd heady) W head, AAt A XWA X
where X E [8 is the input representations to MHSA, is the number of tokens and d is the model dimension. d, is the dimension of head h and is usually set d/N. W W W Rdxd and Wo E Raxd are learnable weight matrices. The outputs of MHSA are further passed to FFN which learns nonlinear transformations to derive the final outputs of the Transformer layer.
the token representations H [hi,..., h,...., kn). The encoder 0 is trained to predict the original token out of the vocabulary V at each masked position by minimizing the cross- cross-entropy loss LMLM:
exp(e], h2)
LMLM EE - log pe (z: (z:1i²) iEM
(o(x; = Exey ex,(e_,h,)
(1)
where e₂ refers to the embedding of token x.
2.2 RANK-DEFICIENT REAL TOKEN REPRESENTATIONS
MLM pretraining introduces a special [MASK] token replace the token positions to be predicted, but such [MASK] tokens are usually absent from downstream task data. Therefore, to study the PLM's capacity for downstream data representation, we examine the real token representation space trained with MLM. A common measure of the representation space capacity is the rank of the data representation matrix (Ansuini et al. 2019 Bhojanapalli et al, 2020). In our case, this refers to the real (n >> d) where each row corresponds to the representation
token representation matrix HR E of a real token. Ideally, one would hope H to have high column rank (i.e., rank(HR) R d) so that more model dimensions are effective for modeling real tokens. However, as we will show next,
R paxe
Cooee he found https:/tps//ttt www...coo/yuunnn.... 5/MAN
2

---

Published as a conference paper at ICLR 2024
portion of the model dimensions will be exclusively used for [MASK] token representations in MLM pretraining, so that H is necessarily rank-deficient (i.e., not all model dimensions are leveraged to represent real tokens).
Empirical Evidence. We evaluate the representation space of pretrained 12-
layer RoBERTapae model (Liu et al.]
Elfretive Reak
Elfective Bank
5000
2000
2019) on the validation set of the pre- training corpus with 5 million tokens. We first apply 15% random masks to these input sequences (same as the pretraining setting), and obtain the token representa- tion matrix H' E RRaxd (n R 5 x 106 the total number of tokens the cor- pus, d = 768 is the model dimension), which contains both real token and mask token representations, for each layer in the pretrained RoBERTa. We then feed the same input sequences in their original form (i.e., without [MASK] )) to the pre- trained RoBERTa model and obtain the token representation matrix H' E Rexd which consists of real token representations only, Compar- ing the rank of H' with H' gives insights about the change in representation capacity when adapting a pretrained MLM model to inputs without [MASK].
4.0-
Injuts w. [usk]]
200-
[HABK] Tilama
laputs w/o. (MABK)
Real Thirm
3
6
9
12
12
Encoder Layer Index
Encoder Layer Ludee (b)
(a)
Figure 1: an MLM- LLMM-retrained model, (a) some model dimensions are exclusively used for representing [MASK] tokens, resulting a representation deficiency for model- ing inputs without [MASK], especially in deeper layers; (b) the effective rank of [MASK] token representation space increases throughout Transformer layers.
Since numerical errors and small perturbations practically render any large matrix full-rank re- gardless of its actual rank, we compute the effective rank (Cai et al., 2021) of a matrix H: We only consider H's most significant components that account for the majority of the variance re- flected by singular values. Given a threshold value т, we define the r-effective rank of H as rank, (H) = arg min;
E. ZT): where d; is the ith largest singular value of H. For exam- ple, rankg.9 H 10 means that 90% of H's variance can be captured with dimensions. We follow the definition of effective rank in Cai et al. (2021) only to perform empirical computations of the rank to showcase the issue, and we do not use it in our theoretical analysis below.
Figure ((a) shows rankg.9 ( H (Input w. [MASK]) and rank
H (Input w/o. [MASK]).
generally holds that rank 9 H < ranko.9 (H²), and the gap is more prominent in deeper layers. This demonstrates that some model dimensions are reserved for [MASK] token representations in almost all encoder layers, and these dimensions not active when the input sequences consist of real tokens entirely. Such representation deficiencies for modeling real tokens become more severe in deeper layers where [MASK] token representations occupy more dimensions, shown in Figure ((b)
We theoretically validate the empirical observation above that MLM necessar- ily allocates subspace for [MASK] token representations which not contained by the real token representation subspace, so that the real token representations are rank-deficient.
Theoretical Analysis.
Lemma 2.1 (Rank increase of [MASK] token representations in Transformer encoder). The rank of [MASK] loken representations will increase from the input layer the ouipul layer of an L--ayer Transformer encoder trained with MIM (i.e., rank > rank H H)))
Proof. We first show that / will be high-rank in well-trained MLM model and then show that H necessarily low-rank, and thus the statement holds.
As shown in Equation [ the output token probability distributions at masked positions are computed from the encoder's 's output representations H E mxd and token embeddings E E IR Pixd Denote the true log probability distributions of the masked token prediction task as T E IR
log p (x [1)) log p ( log p (I (t) log p (r 22)
log p (x) [22) log (I2)
T =
log (x) (m))
log (x2 km)) ... log (plllmm)
3

---

Published as a conference paper at ICLR 2024
then H and E are trained to approximate T with shift (due the softmax normaliza- tion) YYng et al., 20188:
E = T
where R" contains the shifting constant added to each row, and 1 R is vector of all ones.
shown in Yang et al. (2018) that the true probability distribution T is high-rank (as high as (V)) due to the complexity of natural language. Since rank [H E S min (H (1). rank (E),. both H and E need to be high-rank to achieve a good approximation of T + cl
Next, we show H low-rank. H is the sum token embeddings and position embeddings at masked positions:
H = le MASK" + P.
where E [MASK]] E Rd is the [MASK] token embedding, and E mxd x is the position embeddings.
[ + P) I vank(1e [ + rank P) rank (P) + 1, only need Since we have rank (1e [MACK]
to show P is low-rank. Previous studies (He et al. 2021 Ke et al., 2021 have identified that position embeddings P and token embeddings E encode disjoint information, and learned in separate subspaces of Rd. Therefore, rank(P) P) I rank E). We also showed that E must be high-rank to satisfy Equation ( and thus P is necessarily low-rank. Finally, M is also low-rank
as rank HM) < rank +
Remark. Lemma 2.1 corresponds to the empirical observation in Figure [(b) and can be intuitively interpreted as a necessary consequence of the [MASK] token contextualization process in Trans- formers: The [MASK] representations at the input layer are context-free, and they need to aggregate contextual information from other tokens in the sequence for predicting the original word, resulting in an increase in the information content of [MASK] token representations. We also note that the rank increase statement does not necessarily apply to real token representations. This is because MLM does not directly train the real token representations (e.g., the training objective in Equation does not apply to real token p
Based on Lemma 2.1 we proceed to prove that H'g occupies different subspace that is not contained by the subspace of H resulting in deficient representations for real tokens. In the following, we analyze the rank change induced by the self-attention mechanism since it the source of contextualization of [MASK] tokens, and the effectiveness of text encoders is typically attributed to the contextualized representations (Ethayarajh) 2019). While we do not account for ML.s and residual connections, our analysis validates that the rank deficiency is caused by the self-attention mechanism, and in practice, MLPs and residual connections do not prevent the issue from happening.
Theorem 2.2 (Rank deficiency of real token representations). There exists some layer I in the Transformer encoder where the real token representation H is rank-deficient. In particular, the row space of Hi does not contain the row space of [MASK] token representation H
Proof. We provide proof sketch below. Detailed proofs can be found in Appendix A We prove the statement by contradiction: Suppose that the row space of H's E dxd contains the row space of H E mxd, then we can represent H M with H via a linear combination weight matrix U:
H'M = -UH, U U
We show that under this assumption, H will converge exponentially (with /) to rank-1 matrix, which contradicts with Lemma 2.1 To examine the matrix rank, we follow the definition of matrix residual R' (Dong et al. 2021 which measures the difference between H and a rank-1 matrix:
R² H 1h
h = a min H 1x
"Some MLM training settings adopt a trick that keeps 10% of [MASK] as original tokens and randomly replaces another 10% of [MASK] with other tokens. Even with this trick, the training signals on real token representations are scarce. Furthermore, later studies (Wettig al. 2023) report that this trick is not necessary- training exclusively [MASK] positions performs well.
4

---

Published as a conference paper at ICLR 2024
Based on the self-attention formula and the assumption in Equation the norm of R' as a function R!
derive bound for
w W
R 1,:00
S
4e R
| 1,00 | (1 + I[U))
1,00
Vd
where l1-1 denotes the geometric mean of L and fo norm. This shows that R 1, converges exponentially with to zero, and thus H's converges exponentially with I rank-1 matrix. We also have rank z rank (H'r) as the row space of HJM is contained by the row space H Hence, H'M will also converge exponentially rank-1 matrix, which contradicts with Lemma 2.1
Therefore, the statement holds.
Remark. Theorem 2.2 demonstrates that at least some [MASK] token representations and real token representations need to be linearly independent so that the rank of H'M may increase through encoder layers. As a result, the real token representation H cannot utilize the entire model dimensions and prone to rank deficiency.
MAE-LM: MASKED AUTOENCODERS FOR MLM
To address the representation defi- ciency issue in MLM, we propose a simple framework MAE-LM, which pretrains bidirectional Transformer encoders using the MLM objective, but based on the Masked Autoen- coder (He al. 20222 Liao et al. 2022) structure. An overview of MAE-LM is shown in 2 While previous applications of the archi- tecture are mainly motivated by the efficiency benefit of reduced input sequence lengths, its effects on the learned tokens representations have not been thoroughly studicd.
Bidirectional (Shallow) Decoder
Not used tasks
C p4asF]
h
Cpessic
Masked positins addedd decoder inputs
Bidirectional Encoder
Fine-tuned
downstream tasks
Masked positions omitted from encoder inputs
Masked Sequence:
intt (MASK] (HASK]
Apply random maske
Original Sequence: 21pp
Excluding [MASK] from the En- coder. An important design in MAE- LM is that [MASK] tokens cluded from the encoder inputs so that no model dimensions will be used to represent [MASK] tokens. Hence, the representations of real tokens HR can theoretically utilize the entire space Rd which addresses the representa- tion bottleneck in conventional MLM pretraining. Specifically, given a masked sequence à [x1,.... [MASK] ...... Øn)] and let M denote the set of masked positions, the encoder's input se- quence H consists of the sum of token embeddings en, and position embeddings p; at real token positions & M:
Figure 2: Overview MAE-LM. Masked positions are omit- ted from encoder inputs so that the encoder purely models real tokens. shallow decoder takes the encoder's output representations and masked positions to predict the original tokens. After pretraining, only the encoder (but not the de- coder) is fine-tuned for downstream tasks.
H = (h:) sgm
h ez, + P:-
Decoder Configuration. order to predict the original tokens at masked positions, the encoder's output token representations H4 are further passed to an auxiliary bidirectional decoder. While standard Transformer decoders perform unidirectional self-attention (and cross-attention to encoder outputs) for autoregressive decoding. our decoder performs bidirectional self-attention (same the encoder). is called a decoder as takes encoded representations as input and outputs tokens. The decoder's ' input sequence H needs to include the [MASK] token embedding (maSK] and position embeddings P: so that the decoder is aware the positions to be predicted:
{Rg] 1315n"
R
e [MASK] + P:
i M
h + p.
i i M
5

---

Published as a conference paper at ICLR 2024
Table 1: Standard single-task, single-model fine-tuning results (medians over five random seeds) evaluated on GLUE and SQuAD 2.0 development sets. Results not available in prior research are marked with "_" We use Spearman correlation for STS, Matthews correlation for CoLA, and accuracy for the other tasks GLUE. The "AVG" column contains the averaged results across the
eight GLUE tasks. All baseline results are taken from public reports unless marked with (Ours).
GLUE (Single-Task)
SQuAD
Model
MNLI-(m/mm)
QQP
QNLI
SST-2
CoLA
RTE
MRPC
STS-B
AVG
base setting: Pretrained on Wikipedia & Book Corpus (16GB)
BERT
84.5/-
91.3
91.7
93.2
58.9
68.6
87.3
89.5
83.1
73.7
76.3
ALBERT
81.6/-
90.3
77.1
80.0
Uni
86.1/86.
93.2
80.9
83.6
TUPE
86.2/86.2
91.3
92.2
93.3
63.6
73.6
89.9
89.2
84,9
RoBERTa
84.7/-
92.7
79.7
RoBERTa (Ours)
85,9/85 8
91.6
92.3
93.7
64.3
75.5
88.7
89.5
85,2
78.3
81.5
MAE-LM
87.2/87.1
91.6
92.9
93.8
63.1
79.1
90.2
90.9
86,1
81.1
84.1
base +++ setting: Pretrained on larger pretraining corpora (160GB)
ALBERT
82.4/-
92.8
76.3
79.1
RoBERTa
87.6/-
92.8
94.8
63.6
78.7
90.2
91.2
86,4
80.5
83.7
Uni
88.5/-
91.7
93.5
95.1
65.2
81.3
91.8
91.0
87.1
83.3
86.1
MAE-LM
89.1/89.1
91.7
93.8
95.1
65.9
85.2
90.2
91.6
87.8
83.5
86.5
The decoder's ' output representations will be trained with the MLM objective shown in Equation ( Since the decoder includes [MASK] tokens, it is subject to the representation deficiency for modeling real tokens as analyzed in Section Therefore, the decoder is not used in fine-tuning on downstream tasks. The decoder is made to be shallow (the decoder depth experiments) not only for pretraining efficiency, but also to push the encoder to learn robust token representations-iff the decoder is too strong, it alone may learn the MLLM task well without requiring good encoder representations H
1/6 1/3 of the encoder in our
Despite using an additional decoder in pretraining, MAE-LM's pretraining time cost is roughly equal to that of conventional MLM pretraining (e.g.. RoBERTa). This is because the exclusion of [MASK] tokens from the encoder practically reduces its input sequence length (e.g.. 15% random masks shorten the encoder's 's imput length by 15%), bringing down the encoder's 's computation cost.
EXPERIMENTS
4.1 PRETRAINING AND EVALUATION SETUP
Pretraining Settings. We evaluate MAE-LM mainly under the base model scale for two pretraining settings: base and base++. Both settings pretrain 12-layer Transformers with 768 model dimensions.
The base setting uses 16GB training corpus following BERT (Devlin et al.) 2019) while the base+ setting uses 160GB training corpus following RoBERTa (Liu et al. 2019). The details can be found in Appendix D Additional results of larger model scales are presented in Appendix E All settings use the MLM objective for pretraining without any sequence-level tasks.
Downstream Tasks and Fine-Tuning. We evaluate the pretrained models on the GLUE (Wang ct al. 2018) and SQuAD 2.0 (Rajpurkar et al., (018)) benchmarks. The details about GLUE tasks can be found in Appendix
We adopt standard fine-tuning as in BERT (Devlin et al. 2019) and RoBERTa (Liu et al.) 2019). The hyperparameter search space for fine-tuning can be found in Appendix D All reported fine-tuning results are the medians of five random seeds on GLUE and
SQuAD, following previous studies (Liu et al. 2019). Additional few-shot and zero-shot evaluation results are presented in Appendix
Baselines. We compare with various baselines pretrained by MLM (and variants of MLM) under each setting, including BERT (Devlin et al., 2019), ALBERT (Lan et al. 2020), UniLMv2 [ao al. 2020), TUPE (Ke et al.) 2021). and RoBERTa (Liu al.) 2019). The baseline results, unless marked by "(Ours)", are taken from the original papers. To eliminate the performance difference due to implementation details and computation environment, also pretrain and fine-tune RoBERTa (the most important baseline) under exactly the same base pretraining setting with MAE-LM, which is denoted with "RoBERTa (Ours)".

---

Published as a conference paper at ICLR 2024
MAE-LM Ridkdiiik
ST-
MUN-IM Hi BaBERRT
GULE ANG
MD 20
Sec
&
85-
3 6
85.0
0.25
0.75
1.00
0.25
Thining Time (Hous)
Traning Time (Boars)
[MARK] Incloded
Incinded
(a) MNLI-m
(b) MNLI-MM
(a) GLUE
(b) SQuAD
Figure 3: MNLI dev accuracy by fine-tuning Figure GLUE average scores and SQuAD EM intermediate MAE-LMBus checkpoints at differ- scores when different fractions of [MASK] tokens ent time steps. We also mark the pretraining time are included in the input sequences to the encoder and final performance of RoBERTa (Ours).
of MAE-LM
4.2 OVERALL RESULTS
Table shows the results under the base model pretraining settings the GLUE and SQuAD 2.0 benchmarks. Overall, MAE-LM outperforms previous models pretrained by MLM and its variants. Notably, the gains of MAE-LM over RoBERTa (the standard MLM pretrained model) are quite consistent across tasks and pretraining settings.
Pretraining Efficiency. In Figure 3 illustrate MAE-LMaas' fine-tuning performance when pretrained for different amounts of time. MAE-LM takes slightly more time than RoBERTa when trained on the same amount of data, but reach RoBERTa's MNLI accuracy, MAE-LM only needs about 40% of its pretraining time.
4.3 ABLATION STUDIES
Table 2 shows several groups of ablations to study the important components in MAE-LM.
Naive Baselines. To validate that the effectiveness of MAE-LM is not from simply using the additional decoder in pretraining, we first compare two naive baselines: (1) the standard MLM (enc. [MASK] ) and (2) adding the same decoder used in MAE-LM but still pretrains the encoder with [MASK] tokens included in inputs (enc. w. [MASK] + dec.). The two baselines perform similarly, confirming that naively using the decoder does not benefit downstream tasks.
Handling [MASK] We compare with other ways of handling [MASK] tokens in the encoder: (1) including [MASK] in encoder's inputs but reset- ting [MASK] token positions to the [MASK] token embedding e ([xSsi] in decoder's inputs (enc. w. [MASK], dec. resets [MASK]) and (2) ran- domly replacing [MASK] tokens in encoder's inputs with other real to- kens from the vocabulary (random replace w. real token). The first variation improves the performance over vanilla MLM, showing that when [MASK] is present in the encoder, resetting the [MASK] token embed- dings the decoder helps. This vali- dates our analysis in Theorem 2.2 that the rank increase of [MASK] token representations is the main cause of representation deficiency, and prevent- ing [MASK] token representations in the encoder from being explicitly trained is one way to mitigate the is-
Ablations evaluated with GLUE average scores. The setting of MAE-LM is: enc. w/o. [MASK]] aligned position encoding w. relative position encoding; bi. self- attention; layer, 768 dimension.
Table 2:
Group
Setting
GLUE
Original
MAE-LM
86.1
Naive
enc. w. [MASK] (f.e., MLM)
85.2
enc. W. [MASK] + dec.
85.1
Handling [MASK]
enc. W. [MASK], dec. resets [MASK]
85.9
random replace w. real token
85.1
Position Encoding
misaligned position encoding
86.0
relative position encoding
86.1
Decoder
bi. self-attention + cross-atiention
85.4
Attention
uni. self-attention cross-attention
85.5
cross-attention
86.0
Decoder Size
2 layer, 768 dimension
85.8
6 layer, 768 dimension
84.8
4 layer. 512 dimension
85.8
4 layer. 1024 dimension
85.5
sue, though it is slightly worse than completely excluding [MASK] from the encoder. The second variation demonstrates that replacing [MASK] tokens with random real tokens, though avoiding the
7

---

Published as a conference paper at ICLR 2024
representation deficiency problem, worsens the context quality in pretraining. On balance, it does not yield better results than MLM.
Position Encoding. MAE-LM aligns the position encoding based on each token's 's position in the original sequence, and the position indices of masked positions are skipped. MAE-LM also uses relative position encoding (Raffel al.) 2019). We create two ablations: (1) apply consecutive position encoding that does not reflect the masked positions (misaligned position encoding); and (2) remove the relative position encoding from MAB-LM (no relative position encoding). Overall, the variations in position encoding do not result in notable performance differences.
Decoder Attention. MAE-LM uses bidirectional self-attention in the decoder. We compare with other decoder attention configurations: (1) additionally use cross-attention to encoder's output representations (bi. self-attention + cross-attention); (2) use unidirectional self-attention and cross- attention for autoregressive decoding of the entire sequence, similar to BART (Lewis et al. 2020a) (uni. self-attention + cross-attention); and (3) only use cross-attentes (cross-attention). Bidirectional self-attention only in the decoder is simple and performs the best.
Decoder Size. MAE-LM uses a 4-layer decoder with the same dimensionality (768) as the encoder. We experiment with other decoder sizes (when the decoder's dimension is different from the encoder, we add linear projection between the encoder's output and the decoder's input): (1) 2-layer, 768 dimension; (2) 6-layer, 768 dimension; (3) 4-layer, 512 dimension; and (4) 4-layer, 1024 dimension. Overall, using a relatively small decoder yields good results.
Gradual Transition from MAE-LM to Standard MLM. To further examine the empirical benefits of excluding [MASK] tokens from MAE-LM's s encoder, we create a set of "stepping stones" between MAE-LM and standard MLM follows: Out of all [MASK] tokens in the sequence é, we include fraction (d) of them in the encoder's s input sequence. The rest (1 - 6) of [MASK] tokens are excluded from the encoder's input and added to the decoder's input. Then 5 0 represents MAE-LM, and
d = 1 refers to the standard MLM 3 Figure 4 illustrates the fine-tuning performance changes on GLUE and SQuAD as we transition from MAE-LM to standard MLM. There is a clear trend that including a higher portion of [MASK] tokens in the encoder degrades its performance.
4.4 MAE-LM IMPROVES MODEL DIMENSION UTILIZATION
To further validate the effectiveness of MAE-LM in improving the utilization of model dimensions for representing real tokens, we compute the 0.9-effective rank of the encoder's 's token representa- tions rankg.g H) both after pretraining (evaluated the validation set of the pretraining corpus) and after further fine- tniin MNLI. Figure(5(a) Shows the effective rank throughout encoder lay- ers for (1) RoBERTa when the inputs contain [MASK] (MLM w. [MASK]), ); (2) RoBERTa when the inputs are all real tokens (MLM w/o. [MASK]) ); and (3) MAE-LM. MAE-LM closes the gap caused by [ [MASK] tokens in vanilla MLM pretraining. Figure 5(b) further validates that MAF-LM maintains its advantage in the effective rank of real token representations during fine-tuning on MNLL This highlights the importance of addressing the representation deficiency issue in pretraining: The model dimensions not pretrained to represent real tokens may not be easily leveraged in fine-tuning.
Elfeetivv Rank
Elfective Banke
MAR-IM
5000
200-
MLM
4.0-
MEM w/oo [NA3K)
200-
MAE-LM
400
3
6
10000 Fixe- Tuming Steps
20003
Enroder Layer Index
shows
(a)
(b)
Figure 5: (a) MAE-LM effectively closes the rank gap in vanilla MLM with inputs containing or not containing [MASK] (b) During fine-tuning, the advantage in effec- tive rank of MAE-LM over vanilla MLM still holds.
5 RELATED WORK
Language Model Pretraining. Various pretraining methods have been proposed for different purposes: Standard autoregressive language modeling (Brown et al. 2020 Radford et al., 2018
Althoug standard MIM (i.e., RoBERTa) does not have the decoder, its fine-tuning results are almost the 8 (with the decoder) as shown in Table 2
8

---

Published as a conference paper at ICLR 2024
2019) is commonly used to pretrain generative models that excel in text generation; MLM (Devlin) et al. | Liu et al. 2019) is prominently used to pretrain bidirectional text encoders to achieve superio performance for language understanding; Other language modeling objectives (Lewis et al.) 2020a Raffel et al. 2019) are designed to build sequence-to-sequencc models that serve as both text generators and text encoders. As one of the most prominent pretraining approaches, ML.M has
stimulated many up developments for pretraining bidirectional encoders (Bao et al. 2020
Clark al. 2020 Gong et al. 2023, He al. 202T Joshi et al. 2019 Lan et al.) 2020 Liao et al. Sanh et al. 2019 Yang et al. (219)). Remarkably, the idea of MLM is
2 et al.
highly generalizable to different domains (Bao et al. [022] Dosovitskiy et al. | Hou et al. 2022]
Tong et al.) [022 Wang et al. 2022, Xie et al. 20022 and leads to developments of unified pretraining
frameworks for different modalities (Baevski et al. 2023: 2022). Given the broad impact of ML.,,
our analyses of representation deficiency in MLM may provide insights for future developments of
pretraining algorithms in various fields.
Study of Pretrained Models' Representations. The powerful language representations learned by pretrained models have driven a series of studies to understand how linguistic knowledge is acquired through pretraining. Previous work studying the token representations in pretrained encoders has found that deeper layers generate more contextualized token representations (Ethayarajh) (201)) and these representations encode syntax structures (Goldberg 2019; Hewitt& & Manning 2019) and fine-grained word senses (Coenen et al. 2019), offering supporting evidence for the effectiveness of pretrained models in downstream tasks. The success of learning such linguistic patterns is usually attributed to the self-attention mechanism which automatically learns to extract useful features through pretraining (Clark et al. 2019). Furthermore, different types of linguistic information are shown to be represented in a hierarchical way from shallower to deeper layers, reflecting the traditional NLP pipeline (Tenney et al. 2019a b). There have also been prior efforts that investigate the limitations of pretrained models' representations. It has been revealed that the contextualized embedding space learned by pretrained models generally anisotropic (Cai et al. 202T; Li et al.) 2020) and subject to degeneration problem that token representations tend to be distributed into narrow cone (Gao) ict al. (201)). [Gong et al. (2119) identify that self-attention in Transformers tends to assign higher weights to local tokens as well the starting token, which motivates the design of progressive stacking algorithm for efficient pretraining. In this work, we investigate previously unknown issue regarding Mrr--rrraane models' representations that hinders the model's 's expressiveness input sequences without [MASK] tokens. Our findings contribute a new perspective to understanding the limitations of representations in pretrained models.
CONCLUSION
Limitations. The focus of our work is MLM and our analyses do not apply to other pretrain- ing settings not using [MASK] tokens, and we discuss potential implications of our findings on autoregressive language models in Appendix F. While the current large language models are mostly autoregressive models, we believe that text encoder models still have important and wide applications in NLP, including but not limited to (1) Non- Non- generation tasks. Many natural language understanding tasks do not have to be modeled autoregressively, for which encoder-only models are generally more text generation (Lewis) et al. 2020b), which typically uses an encoder for retrieval to enhance the generator's 's factualness. (3) Reward models in reinforcement learning from human feedback (RLHF) can use encoder mod- els (Song et al. 2023). Empirically, we mainly compare with models pretrained by MLM and its simple variants and do not include all of-te-of- models, as they typically require integrating multiple pretraining strategies and/or architecture changes (He et al. 2023).
parameter efficient and effective (Zhong et al. 2023). (2)
Conclusion. In this work, we investigate the discrepancy caused by [MASK] tokens in MLLM pre- training and demonstrate for the first time that this will necessarily result in real token representations being rank-deficient, thus limiting the model's expressiveness on real data without [MASK]. We propose simple method MAE-LM that excludes [MASK] tokens from the encoder in pretraining to address the representation deficiency issue. We empirically show that MAE-LM improves the utilization of model dimensions for representing real tokens in pretraining and downstream tasks. MAE-LM consistently outperforms MLM --e---rtr models on the GLUE and SQuAD benchmarks across multiple pretraining settings.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
Research was supported in part by U.S. National Science Foundation IIS-19-56151, the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environment (I- GUIDE) by NSF under Award No. 2118329. Yu Meng was supported by Google PhD Fellowship.
REFERENCES
Alessio Ansuini, Alessandro Laio, Jakob H. Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. In NeurlPS. 2019.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: general framework for self-supervised learning in speech, vision and language. In ICML, 2022.
Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. ICML, 2023.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang. Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. UniLMv2: Pseudo-masker language models for unified language model pre-training. In ICML., 2020.
Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. In ICLR, 2022.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In ACM Conference on Fairness, Accountability, and Transparency, 2021.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Low-rank bottleneck in multi-head attention models. In ICMI., 2020.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. ArXiv, abs/21108.02555 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embedding space: Clusters and manifolds. In ICLR, 2021.
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpic and Lucia Specia. Semeval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In International Workshop on Semantic Evaluation (SemEval), 2017.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does BERT look at? an analysis of BERT's attention. In BlackboxNI.P. 2019.
Kevin Clark, Minh-Thang Luong, Quoe Le, and Christopher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. ICLR, 2020.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda B. Viégas, and Martin Wattenberg. Visualizing and measuring the geometry of BERT. In NeurIPS, 2019.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In NAACL-HITT, 2019.
10

---

Published as a conference paper at ICLR 2024
Jesse Dodge, Ana Marasović, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: case study on the colossal clean crawled corpus. In EMNLP. 2021.
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In International Workshop on Paraphrasing (IWP), 2005.
Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention not all you need: Pure attention loses rank doubly exponentially with depth. In ICML, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of BERT, EL,o, and GPT-2 embeddings. EMNIP, 2019.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. Representation degeneration problem in training natural language generation models. In ICLR, 2019.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACL, 2021.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In ACL-PASCAL workshop textual entailment and paraphrasing, 2007.
Aaron Gokaslan and Vanya Cohen. WebnWebText corpus. http:://kkyLico007..ttt io/ @penReebeettCorpus 2019.
Yoav Goldberg. Assessing bert's syntactic abilities. ArXiv, abs/1901.05287,2000 2019.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tie-Yan Liu. Efficient training of BERT by progressively stacking. In ICML, 2019.
Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie, Alvin Cheung, Jianfeng Gao, and Xia Song. Model-generated pretraining signals improves zero-shot generalization of text-to-text transformers. In ACI., 2023.
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In PASCAL Challenges Workshop on Recognising Textual Entailment, 2006.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll' ar, and Ross B. Girshick. Masked autoeneoders are scalable vision learners. CVPR, 2022.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with disentangled attention. In ICLR, 2021.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In ICLR, 2023.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In ICML, 2019.
John Hewitt and Christopher D. Manning, A structural probe for finding syntax in word representa- tions. In NAACL, 2019.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chun- Wenn-Wiii Wang, and Jie Tang. GraphMAE: Self-supervised masked graph autoencoders. In KDD, 2022.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Span- BERT: Improving pre-traine pre-training by representing and predicting spans. Transactions of the Associa- fion for Computational Linguistics, 8:64-77, 2019.

---

Published as a conference paper at ICLR 2024
Guolin Ke, Di He, and Tie-Yan Liu. Rethinking positional encoding in language pre pre-training. In ICLR, 2021.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In ICLR, 2022.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In ICLR, 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sqquee pre-training for natural language generation, translation, and comprehension. In ACL, 2020a.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmentee generation for wewlllegeeinnnniiee NLP tasks. In NewrlPS, 2020b.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In EMNLP, 2020.
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understand- ing and mitigating social biases in language models. In ICML, 2021.
Baohao Liao, David Thulke, Sanjika Hewavitharana, Hermann Ney, and Christof Monz. Mask more and mask later: Efficient pre-training of masked language models by disentangling the [MASK] token. In EMNLP, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv, abs/1907. 1692, 2019.
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. COCO-LM: Correcting and contrasting text sequences for language model pretraining. In NeurlPS, 2021.
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. Pretraining text encoders with adversarial mixture of training signal generators. In ICLR, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans. Ilya Sutskever, et al. Improving language understanding by generative pre-trate pre-training. OpenAl blog. 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2019.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In ACL, 2018
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01 108, 2019.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In EACL, 2021.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACI., 2015.
Iyer Shankar, Dandekar Nikhil, and Csernai Korél. First Quora dataset release: Question pairs, 2017.
URL https: www. quora. com/ quoradata/
Parst-Quora-an Rew--eeeeeeseeeea Release--ueeeee P
12

---

Published as a conference paper at ICLR 2024
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Ziang Song, Tianle Cai, Jason D. Lee, and Weijie Su. Reward collapse in aligning large language models. ArXiv, abs/2305.17608 2023.
lan Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In ACL, 2019a.
lan Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. JCLR, 2019b.
Zhan Tong. Yibing Song. Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS, 2022.
Trieu H Trinh and Quoe V Le.. A simple method for commonsense reasoning. ArXiv, abs/1880.0088/ 2018.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop BlackboxNI.P. 2018.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang,
Luowei Zhou, and Lu Yuan. BEVT: BERT pretraining of video transformers. In CVPR, 2022.
Alex Warstadt, Amanpreet Singh, and Samuel Bowman. Neural network acceptability judgments. In TACL, 2019.
Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling? In EACL, 2023.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, 2018.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMIM: a simple franework for masked image modeling. In CVPR, 2022.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. In ICLR, 2018.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019.
Qihuang Zhong, Liang Ding, Juhua Liu, Du, and Dacheng Tao. Can ChatGPT understand too? comparative study on ChatGPT and fine-tuned BERT. ArXiv, abs/2302. 10198, 2023.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, 2015.
13

---

Published as a conference paper at ICLR 2024
A DETAILED PROOFS
Theorem 2.2 (Rank deficiency of real token representations). There exists some layer i in the Transformer encoder where the real token representation H is rank-deficient In particular, the row space of H does not contain the row space of [MASK] token representation H M
Proof. We prove the statement by contradiction: We suppose that the row space of H always contains the row space of H' i in all layers 1 < S L, and we will show that under this assumption, H'M will converge exponentially (with 1) to rank-1 matrix, which contradicts with Lemma 2. In the following, we assume single-head self-attention is used, and the analysis can easily generalized to the multi-head case.
The following proof extends Dong et al. (202T) by considering the representations of real tokens and mask tokens separately and following the residual norm analysis in Dong et al. (2021) study the rank changes.
The self-attention module in the (th layer takes the previous layer representations H (the superscript 1 is omitted for convenience) as input and derives the output representations H':
H' = Atn (HWQ, HW HWW
W
HW TT H HWWwo
Softmax
Vd
AHWVO
where we denote the attention matrix computed from softmax as A, and WVo wwwo
We study how the real token representations change (i.e., comparing H'R with HR) through the self-attention module. To facilitate easy analyses, we partition the input token representation matrix H E ((++m) xd into blocks consisting of real token representations HR E Rgd and [MASK] token representations Hm Rmxd, and partition the attention matrix Ag into blocks consisting of attention weights from real tokens to real tokens AR:R E Raxx and from real tokens to [MASK] tokens AR:M Raxm:
HR HM
H
AR [AR. ARMM..
We further denote
SR.R exp HR W H
SR.M= exp QQK H. Z 1R.:1++
where exp[-] denotes the element-wiss exponential function, diag(.) constructs diagnal matrix from vector, WQK wOWK /vd, and 1 is vector of all ones. Then
AR.R Z -- AR.M Z 15..-
Based on the above notations, the output representations real token positions H's can be written as:
HR HM
H Vo [ARR.R
AR.M]
wvo
z- 1rro=zz(sspp (SRRR 'S SR:MHM) wvo
(4)
the row space of Hx contains the row space of HM, each row of HM can be represented as linear combination of the rows in HR:
H = UHR
where U E Rmmx is the linear combination weight matrix. We can rescale the vector norm of each row HM so that U has row sum of one (i.e., U1 =1
To examine the rank of real token representations, we examine the change in matrix residual through Transformer layers, inspired by [ong al. (202T). Specifically, we define the following residual R which measures the difference between H and rank-1 matrix:
R = H 1h
h arg min H 1x.
14

---

Published as a conference paper at ICLR 2024
We aim to show that the norm of R converges exponentially (with layer depth) to zero, meaning that HR converges (with layer depth) to a rank-l matrix.
By plugging HR R+14 and HM UHR UR+UIh 1h into Equation
we obtain
H = z-1 (SR:R R + 1h + SR.M (UR+ (UR+1hT) 1hT) wvo
z (SRRR + SR.MU) R Z- (SR.R + SR.Ml) wvo =1
(SRR + SR.MU) Rw v0 1h wvo.
(5)
Next write
RR.R and SR.M:
SR. exp
HR. Hz QK HR
exp
(R +
W R 1h [
RW R
1h w R + (RWQK 1h w 11 1.
exp
RW R
1 h WWRR R
exp Rw h 1h T wek 1
exp
① exp
=F
and
SRIM exp
HR W QK HM
exp
(R+ 1h
wo (UR- 1
RW R TU
+1h w
R R (R h 1h w
exp
(RWQK h 1h 9 h 1
exp
RW QK
R U
① exp
1h 1 WoK R U
2 exp
=F"
where denotes the element-win product. Let RWOK R F' RWOK R gg h WQK R gg h WQK R UT, and RW h + 1h WQK h, we can further write out Z:
Z diag (RRRR1 + SR:M1)
diag ((exp F @ exp [1g []] 1 + (exp [F c exp [1g ()g 1) @ exp(e)).
Let F F F') be the augmented matrix by combining the columns of F and F", and let f and f denote the maximum and minimum element across each row of F, respectively:
J, = max Pajs = min Fij
==min
Then we can derive a lower bound of each element in Z SR.R:
exp(FFy) exp(g, exp(c:)
z 1 RR.R]
E, exp((Fx)
exp(gx) (x exp(Fi)) exp(gs)
Ey exp( F'jy) exp(s;;) exp(ci)
exp (F.) EE exp(gr) exp(o3:))
exp(9))
exp (Fi) Fr) E, exp(gr) + E,, explof))
Similarly, can derive an upper bound:
[ SR.R,,, S exp (Fis L.) E, explor) + E, explol))
exp(g;)
15

---

Published as a conference paper at ICLR 2024
Using the the Taylor expansion of exp, have
exp (Fu -T.) 2 1+ J, 2 - exp [is f.) 1+2 Pis (Ps-S,)) s 1+2 (T: -f.)
Therefore,
exp(g;)
exp(gg)
(1+1,-1)) E, exp(g,)) + Eg exp(g',)
$ 1 (2'1RR (1+23, -2f.) E, explg,') ) + Es exp(gy))
Denote D diag (f f) and 9+ exp [g J 1 exp [g" 1. then the above bound can be expressed in matrix form as follows (the inequality between matrices holds element-wise):
II - D)1 exp [g]
S z
+ 2D)1 exp [g [
(6)
9+
g+
An analogous derivation gives the bound of 2-1 Z-ISR:M:
1 (I 2D)1 exp [g']]
(I - D)1 DI [ S Z Z SR.M S
9+
9+
Since the upper and lower bounds are in very similar forms, we will only focus on the upper bound in the derivations below.
Combining Equation (6) with Equation 7 we have
exp g
exp 9
U
exp [g]]
exp [g
U
z 7-(RRR + SR.UU= 1
+ 2D1
1r 2D1r
(8)
Plugging Equation X into Equation (). we have
H&S s [Ir 2D1r RW +1h wvo T =1
RWVO
+ h wvo
+2D1r RW
=/T
Therefore,
H 1h < 2D1r RWVO With similar derivation, have the following lower bound:
H 1h T 2 Dir RW
Overall, we can bound the element-wise absolute values of R' H 1h"T, which measure the distance between H and rank-1 matrix:
Ryyl = H 1n'],s S [2D1r' T RW
This allows us to further bound the norm of R'. For & norm, we have
12'11
< |
RW
< D1 1oo
|
RO
Based Hölder's inequality
< 2 D1 loo
[1, [ 10 ||,
Submultipliattiity of matrix norms
where
D1
max
fr fa
< 2 F
< max
RW R I1, RW R TTTTI Ull
2 R|]
WQK [ max (1,])
2 R|
W || |
|0 2 1 since U1 = 1
16

---

Published as a conference paper at ICLR 2024
and
r
exp
g
+
exp
9
U
9+
exp
lg
exp
9 U
+
9+
g+
+ |
Therefore, we can bound the & norm of [ as follows:
R' = 4 W voll |l,
| (1+ | (0o) [R] R|
(9)
Similarly, we can obtain the bound for the foo norm of |R'l1:
1x S 4 [ |l |U 1l (1 +1+ | (lx))) [R|| |
(10)
Denote the geometric mean of [[]]] and | as tion and Equation (T0). have
R |||1,
vI|
[||
| then from Equa-
R [R'I1,,0x I WoR
(1
U (z)
R 1,00
1,00
46
|
The above inequality reflects how the residual changes within one self-attention layer. Applying recursively throughout all layers in an I--layer encoder, we have:
R |I S (4z) -1
3
R
1, 2= m=xax
where R² and Rd denote the residuals corresponding to the encoder's output real token representa- tions H and input real token representations H respectively.
This demonstrates that the residual norms of real token representations converge exponentially (with layer depth) to zero. Hence, the real token representation matrix H3S converges exponentially (with layer depth) to a rank-1 matrix. Since the row space of [MASK] token representations H is contained by the row space H2 we have rank (H²M S rank H²²), and H' will also converge exponentially (with layer depth) to rank-1 matrix, which contradicts with Lemma 2.1 Finally, we conclude that the row space H must not contain the row space of H which necessarily implies that H is rank-deficient.
B DETAILS ABOUT GLUE TASKS
More details of all the GLUE tasks can be found as follows.
The Multi-genre Natural Language Inference (Williams al. 2018) task includes 393K training examples from crowdsourcing. The goal is to predict a premise sentence entails, contradicts, is neutral with respect given hypothesis sentence.
MNLI:
QQP: Question Pairs (Shankar et al. (017)) includes 364K training examples from the Quora question- answering website. The task is to determine if two given questions semantically equivalent.
Question Natural Language Inference includes 108K training examples derived from the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al. 2018). The task is to predict if sentence contains the answer to given question.
QNLI:
SST-2:
Stanford Sentiment Treebank (Socher al. 2013 includes 67K training examples movie reviews with human annotations. The task is to determine a given sentence has positive or negative sentiment.
17

---

Published as a conference paper at ICLR 2024
CoLA: Corpus of Linguistic Acceptability (Warstadt et al. 2019) includes 8.5K training examples from books and journal articles on linguistic theory. The task is to determine if a given sentence is linguistically acceptable.
RTE: Recognizing Textual Entailment (Bentivogli et al. 2009 Dagan et al. 2005 Haim et al. 2006
Giampiccolo et al.] 2007) includes 2.5K training examples from textual entailment challenges. The task is to predict a premise sentence entails a given hypothesis sentence.
MRPC: Microsoft Research Paraphrase Corpus (Dolan & Brockett 2005) includes 3.7K training examples collected from news sources. The task is to predict if two given sentences are semantically equivalent.
STS-B:
Semantic Textual Similarity (Cer et al. 2017) includes 5.8K training examples collected from multiple sources on sentence pair semantic similarity annotated by humans. The task is to predict the semantic similarity of two sentences (based on 1 to 5 scoring scale).
C IMPLEMENTATION DETAILS
Details of Pretraining Settings. The base setting follows BERT hase (Devlin et al. 2019) pretraining which uses Wikipedia and BookCorpus (Zhu et al., 2015) (16GB texts) as the pretraining corpora. The encoder architecture is a 12-layer Transformer, and the model dimension is 768. We train both absolute and relative position embeddings (Raffel et al., 2019) in the encoder. The decoder is a 4-layer Transformer with the same model dimensions as the encoder. Since the decoder is not used in downstream tasks, MAE-LMM's encoder can be fairly compared with previous 12-layer base-sized models. The model is trained for 125K steps with 2, 048 sequences per batch, which amounts to 256M samples in total. The maximum input sequence length is 512 tokens. The vocabulary is constructed with BPE (Sennrich et al. 2013) and consists of 32, 768 uncased subword units.
The base++ + setting follows RoBERTa (Liu et al. 2019) pretraining which extends the base setting by incorporating larger pretraining corpora and training the same model architecture for longer. Specifically, the following corpora are used along with Wikipedia and BookCorpus: Open Web- Text (Gokaslan & Cohen 2019), CC-News (Liu et al., 2019), and STORIES (Trinh & Le| 2018). This expands the pretraining corpora to contain 160GB texts. The model is trained for 2M steps with 048 sequences per batch, which amounts 4B samples in total. The base++ setting also expands the vocabulary size to 64, 000 (Bao et al. 2020) by using cased subword units.
The large++ setting extends the base++ setting by scaling up the encoder architecture 24 layers and 1,4 model dimensions. The decoder is still 4-layer Transformer with the same model dimensions as the encoder. Due to the high cost of training large models, we train for 1M steps (half of the base++ setting) with 2, 048 sequences per batch, which amounts to 2B samples in total. Note that this is also half of the pretraining data used in RoBERTa (Liu et al., 2019) and BART (Lewis) et al. 2020a).
Computation Environment. The experiments in this paper are conducted on 64 A100 GPUs.
Masking. For all pretraining settings, we apply 15% random masks to input sequeces. We do not use the trick in conventional MLL (Devlin et al. 2019 Liu et al. 2019) that replaces 10% of [MASK] tokens with the original ones and another 10% with random tokens. We also experiment with higher masking rates (e.g., 40%) which are shown to be beneficial in Wettig et al. (2023) for training large models, but they do not yield better results than the default 15% masking rate in our experiments. This is probably because Wettig et al.) (2023) use an efficient pretraining recipe that is different from the standard pretraining setup, with larger learning rate, larger batch size, shorter sequence length, and fewer training steps.
Position Embedding. We learn both absolute and relative position embeddings (Raffel al. 2019) in the encoder, and only learn absolute position embeddings in the decoder.
Dropout. During the pretraining of MAE-LM, dropout is applied to the encoder but not the decoder, which we find to slightly improve stability.
D HYPERPARAMETER SETTINGS
18

---

Published as a conference paper at ICLR 2024
Table 3: Hyperparameters used in pretraining.
Hyperparameter
base
base++
large++
Max Steps
125K
2M
1M
Peak Learning Rate
5c-4
2c-4
lc-4
Batch Size
2048
2048
2048
Warm-Up Steps
10K
10K
10K
Sequence Length
512
512
$12
Relative Position Encoding Buckets
64
128
Relative Position Encoding Max Distance
128
128
256
Adam
le-6
le-6
le-6
Adam (31. .2)
(0.9, 0.98)
(0.9, 0.98)
(0.9, 0.98)
Clip Norm
2.0
1.0
Dropout
0.1
0.1
Weight Decay
0.01
0.01
0.01
Table 4: Hyperparameter ranges searched for fine-tuning on GLUE. GLUE small tasks include CoLA, RTE, MRPC and STS-B. GLUE large tasks include MNLI, QQP, QNLI and SST-2.
Hyperparameter
GLUE Small Tasks Search Space
GLUE Large Tasks Search Space
Max Epochs
(2, 3, 5, 10)
(2, 3, 5)
base/base++ +: (2e-5, 3e-5, 4e-5, 5e-5)
baselbase +: [ [le-5, 2e-5, 3e-5, 4e-5) +arge++: [5e-6, 7 le-5, 2e-5)
Peak Learning Rate
large+++: +: (7e-6, le-5, 2e-5, 3e-5)
Batch Size
[16, 32]
32
Warm-Up Proportion
(6%, 10%]
6%
Sequence Length
512
512
Adam
le-6
le-6
Adam (91. .22)
(0.9, 0.98)
(0.9, 0.98)
Clip Norm
Dropout
Weight Decay
0.01
0.01
Table 5: Hyperparameter ranges searched for fine-tuning SQuAD 2.0.
Hyperparameter
SQuAD 2.0 Search Space
Max Epochs
[2. 3)
baselbase++: (2o-5, 3c-5, 4c-5, 5c:5)
Peak Learning Rate
large++: (7c-6, le-5, 20-5, 3e-5)
Batch Size
( [16, 32)
Warm-Up Up Proportion
(6%, 10%
Sequence Length
512
Adam
Ic-6
Adam (8r, (82)
(0.9, 0.98)
Clip Norm
Dropout
0.1
Weight Decay
0.01
19

---

Published as a conference paper at ICLR 2024
Table 6: Standard single-task, single-model fine-tuning results (medians over five random seeds) evaluated on GLUE and SQuAD 2.0 development sets for large models. t: MAA-LM is pretrained on half of RoBERTa/BART's data.
GLUE (Single-Task)
SQuAD 2.0
Model
MNLI-(m/mm)
QQP
QNLI
SST-2
CoLAA
RTE
MRPC
STS-B
AVG
EM
FI
large++ setting: larger Transformer model trained larger pretraining corpora (160GB)
BART
89.9 19.9/90.1
92.5
94.9
96.6
62.8
87.0
90.4
91.2
88.2
86.1
89.2
RoBERTa
90.2/90.2
92.2
94.7
68.0
86.6
92.4
86.5
89.4
MAE-LM
90.4/90.6
92.2
95.1
68.7
88.8
92.1
89.3
87.0
89.8
Table 7: Zero-shot and few-shot performance. Few-shot results include mean and standard deviation (as subscripts) performance over 5 different training splits defined in Gao et al. [2021) from Gao et al. (2021).
Results
GLUE (Single- Task) SST-2
Model
MNLI-(m/mm)
QQP
QNLI
CoLA
RTE
MRPC
STS-B
AVG
zero-shot prompting: direct inference on tasks via cloze-type MLM predictions
RoBERTa!
50.8/51.7
49.7
50.8
83.6
2.0
51.3
61.9
3.2
43.4
MAE-LM
52.1/54.3
52.0
52.3
83.5
2.0
54.5
63.4
3.0
44.7
head- braad-bseeduae few-shot fine turing: tne-tuning on samples per label with linear classification head
RoBERTat
45.8s.4/47 8o.
60.74.3
60.2a.5 62.1s.1
81.4s.a
33.914.5 31.113.9
54.4a.9 58.02.5
76.62.5
53.5a.s
58.4
MAE-LM 48.74.//1.16o
64.54.2
81.23.9
78.2
53.0b.o
59.8
prompt-based fin finng: fine-tuning on 16 samples per label with cloze-type MLM templates
RoBERTa!
(8.323/70.55,
65.55
64.5a.2
92.76.a
9.3r.3
69.13.6
74.55.3
71.07.0
64.5
MAE-LM 70.720/73.31.8
67.34.
65.14.3
92.41.1
14.36.9
71.2a.3
74.8
72.36.5
66.2
We report the detailed hyperparameters used for pretraining in Table The hyperparameter search ranges of fine-tuning are shown in Tables[aande 3nd for GLUE and SQuAD 2.0, respectively.
For fair comparisons. the same set of hyperparameters (in both pretraining and fine- tne-tuning) is used for MAE-LM. RoBERTa (Ours) and ablations. We follow previous pretraining studies (Liu et al. 2019) to report the medians of downstream task fine-tuning results under the same set of five different random seeds.
E MORE EVALUATION RESULTS
BERT Masking Strategy. In addition to our default masking strategy which directly applies 15% random masks to input sequences, we also validate our findings under the original BERT masking strategy that replaces 10% of [MASK] tokens with the original ones and another 10% with random tokens. Figure 6 demonstrates that the gap in ef- fective representation rank between inputs with and without [MASK]
Elfeciive Bank
500
450
Ixpette [MASK]
Ingutic w/je. DMASK]]
under this setting is also notable, similar to the findings in Figure I(a) This confirms that randomly replacing a small percentage of [MASK] tokens with real tokens does not effectively address the representation deficiency issue, as ratio of [MASK] tokens pretraining is still high.
400
Ensoreer Layer Index
Figure 6: With the original BERT masking strategy, the effective rank across layers for inputs without [MASK] and with [MASK].
Large Model Results. We also show the performance MAE-
LM under larger model sizes in Table 6 Even trained on half of the pretraining data used RoBERTa (Liu et al. 2019), MAE-LM still performs comparably better, demonstrating the potential of MAE-LM for larger models.
Zero-Shot and Few-Shot Results. Since MAE-LM is trained with the MLM objective, it is applicable to zero-shot and few-shot learning via prompt-based approaches. We report three groups of zero-shot/few results the GI.U tasks comparing MAF-LM (large++) with RoBERTa
20

---

Published as a conference paper at ICLR 2024
(large++) in Table 7 (1) zero-shot prompting which converts the classification tasks into cloze-type MLM predictions and directly uses pretrained models for inference on test sets; (2) basd- few-shot fine-tuning which adds linear classification head to the pretrained encoders for fine-tuning on 16 samples per label; and (3) few-shol prompt-based fine-tuning which fine-tunes the MLM models on tasks converted cloze-type MLM formats with 16 samples per label. We follow the basic manual prompt/label word setting and the vraining/dveloomeee
splits in Gao et al. (202I). For few-shot
learning, the average and standard deviation over 5 different training/developmen splits are reported. Overall, MAE-LM can be combined with prompt- baset- methods for effective zero-shot and few-shot leamnng.
F MORE DISCUSSIONS
Ethical Considerations. Despite their remarkable performance, pretrained models have been shown to come with risks such as exacerbating harmful biases (Bender et al. 2021; BBommasani et al. 2021). In our experiments, we follow the standard pretraining settings (e.g., data preparation, collection and preprocessing), and we expect more well-documented and filtered text corpora (Dodge et al. [02 as well as future developments of harm reduction techniques (Liang et al. 2021 may help mitigate the ethical concerns about pretrained models.
Connections to Prior Work. Since the advent of BERT (Devlin et al., 2019), there have been numer- ous developments in new pretraining and tne-ttmiig methods aiming improve the effectiveness of pretrained models in downstream tasks. The advantages of these proposed methods, however, are mostly demonstrated via empirical evidence alone, and understanding of why certain methods are better than the others remains limited. Our analyses in this work may advance the understanding of the benefits of some prominent methods: ELECTRA (Clark al., (2220) fills [MASK] positions with real tokens; therefore, the encoder does not suffer from the representation deficiency issue. Different from the ablation in Section 4.3 where we randomly sample real tokens fill [MASK], ELECTRA employs an MLM model to sample replaced tokens which are generally plausible al- ternatives the original tokens. thus better preserving the contexts in pretraining. These designs may help partially explain the effectiveness of ELECTRA. Prompt-based methods (Gao et al. 2021 Schick Schütze, 202T) adapt pretrained MI. models to downstream tasks by creating prompt templates that convert the target task into masked token prediction problem. This helps mitigate the representation deficiency issue that occurs in standard fine-tuning of MLM models as [MASK] tokens also introduced into downstream data, resulting in more model dimensions being utilized. Our findings may also shed light on certain previously observed phenomena in MLM models. For example, the rank deficiency issue might be responsible for the ee-contexttullzation in self-attention patterns (Gong et al. 2019).
Implications Autoregressive LMs. While autoregressive LM pretraining generally does introduce artificial symbols such as [MASK], our analyses can be easily extended to show that the representation deficiency issue can also arise in autoregressive pretraining when certain real tokens exist exclusively in the pretraining data but are either absent or occur infrequently in downstream data. Similar the impact of [MASK] tokens, these tokens occupy dimensions during pretraining that may not be effectively utilized in downstream tasks. Consequently. it is desirable to maximize the vocabulary overlap between pretraining data and downstream data, which can be realized via pretraining data selection, training corpora pre-processing, and vocabulary pruning. We leave these explorations as future work.
21


Reviews:
[{"review_id": "PundcOXzXK", "reviewer": "Reviewer_3FYT", "summary": "This paper employs effective rank to analyze the representation deficiency caused by the [mask] token in Masked Language Models (MLM). Based on the analysis results, this paper proposes the MAE-LM model, which does not input [mask] in the encoder and supplements [mask]  in the shallow decoder. The MAE-LM model achieves satisfactory results in downstream tasks.", "questions": "The authors should explicitly define how this work differs from existing works, otherwise,  it appears as if the paper merely uses the theoretical tool of ISOTROPY to analyze why Mask Later is effective. Without a distinct contribution, the overall impact of this work seems limited.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This work is the first study using effective rank to analyze the decrease in expressive power caused by the [mask] token, leading to two insightful theorems. These theorems may inspire future research in the field of MLM.\n\n2. This paper provides rigorous mathematical proofs supporting the analytical findings.\n\n3. The selected tasks for experimentation are representative, and the promising experimental results demonstrate a performance improvement of MAE-LM over classical MLM models.", "weaknesses": "This paper follows the classic framework of analysis + improvement, but both core modules closely resemble existing work.\n\n1. In the analysis module, the use of effective rank to represent MLM's representative capacity overlaps with section 3.1 of ISOTROPY (https://openreview.net/pdf?id=xYGNO86OWDH), where effective rank is used to denote the isotropy of representations. The differences between these two approaches are minor.\n\n2. In the experimental module, the model structure used in this paper is identical to the one used in Mask Later (https://arxiv.org/pdf/2211.04898.pdf).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3KVw3eMzTZ", "reviewer": "Reviewer_sv4z", "summary": "This paper studies the following discrepancy: for masked language models (MLMs), the [MASK] token is only presented in pre-training but not in downstream fine-tuning. Based on both empirical and theoretical analysis, the authors show that the presence of [MASK] token in pretraining occupy some model dimensions even in downstream applications when [MASK] is not used, that is, not all model dimensions are leveraged to represent tokens other than [MASK]. To address this discrepancy, the authors propose MAE-LM, removing [MASK] tokens from MLM pre-training. Empirical evaluations show consistent improvement on the GLUE and SQuAD benchmarks compared to well known MLMs including BERT and RoBERTa.", "questions": "n/a", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Being able to identify this discrepancy and provide a clear and in-depth analysis is the major strength of this paper. The proposed solution to this problem is simple and effective, providing insights for more sophisticated approaches.\n\nThis paper is overall very well presented; in particular, the theoretical analysis presented in this paper are easy to follow and immediately to-the-point; Lemma 2.1 shows that the rank of the matrices for [MASK] token representation increases as the layers go deeper, providing a nice explanation to the empirical observation; Theorem 2.2 shows that the embedding of some [MASK] tokens need to be orthogonal to that of real tokens, limiting the number of dimensions that can be used.", "weaknesses": "It would be nice if the authors can expand a little bit more on the implications of this work on autoregressive LMs but I really do not see any major weakness in this paper. Though someone may argue that MAE-LM’s improvements on benchmarks are relatively small compared to baselines, this is not a quite problem as the improvements shown are quite consistent; besides, one major contribution of this paper is really to identify the discrepancy between pre-training and downstream fine-tuning and to analyze its effect on MLM’s expressiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CUgoWPhjzw", "reviewer": "Reviewer_247c", "summary": "This paper identifies a significant discrepancy in Masked Language Modeling (MLM) pretraining, where model representations are skewed towards the [MASK] token, absent in downstream tasks. To address this, the authors introduce MAE-LM, a novel algorithm that enhances model efficiency and performance, outstripping robust baselines across various metrics.", "questions": "Could you explore the effective rank of models using 80:10:10 masking technique (like Google's original BERT)?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper effectively pinpoints an important yet overlooked issue in BERT-like language models' (LMs) pretraining, specifically the representation mismatch due to [MASK] tokens. It then proceeds to look into this mismatch, attributing it to rank deficiency with robust empirical and theoretical backing in Section 2.2. This perspective is both insightful and novel.\n\n2. The proposed method, MAE-LM, addresses the identified mismatch issue without resorting to complex architectural alterations. This simplicity ensures the resulting pretrained model retains compatibility with existing BERT-like models, demonstrating the method's applicability.\n\n3. The paper's evaluation is very solid, adhering to standard practices with comprehensive benchmarking on GLUE and SQuAD, compared against multiple strong baselines. MAE-LM consistently outperforms these baselines across multiple scales—base, base++, and large++—sometimes even by substantial margins.\n\n4. Detailed analyses and studies confirm that the performance boost really stems from improved model dimension utilization, which is a direct result of addressing the rank deficiency issue.\n\n5. The writing and presentation of the paper are top-notch, mirroring the high quality of its content.", "weaknesses": "1. Although the paper mentions in Footnote 2 that some MLM training settings (like Google's original BERT) keep 10% of [MASK] tokens as original, and that subsequent studies like Wettig et al. (2022) found this trick unnecessary, an exploration of the effective rank of models using this 80:10:10 technique, similar to Figure 1, would be beneficial.\n\n2. Missing reference: [this paper](https://proceedings.mlr.press/v97/gong19a.html) looked into attention patterns in pretrained BERT-like LMs, uncovering patterns related to low-rankness. This study could provide additional insights into the topic at hand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U3BwvqKOay", "reviewer": "Reviewer_HZEv", "summary": "The main claim of this paper is that the `[mask]` token takes some dimensions of the representation and that this may raise the risk of overfitting or result in a waste of model capacity. They first provide some empirical results showing that the `[mask]` tokens indeed cause the model to generate lower-rank representations. Theoretically, the authors also show that\n\n- The representation of the `[mask]` has high rank at the last layer. (lemma 2.1)\n- The vector space of the real tokens (non-mask tokens) representation at some layer does not include the vector space for the `[mask]` token, so the representation of real tokens can not be full-rank. (theorem 2.2)\n\nThese empirical and theoretical analyses motivate them to propose a encoder-decoder-based pretraining approach, MAE-LM, where the encoder’s input does not contain `[mask]` tokens. Their empirical results show that MAE-LM outperform MLM on the GLUE and SQuAD benchmarks.", "questions": "I would probably recommend this paper more if the theoretical parts were stated differently (or removed).", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. They provide empirical evidence supporting their claim.\n2. Their proposed MAE-LM outperforms MLM.\n3. They conduct comprehensive ablation studies.", "weaknesses": "My main concerns are about the theoretical arguments in this paper.\n\n## Main concern 1: The connection between pretraining and fine-tuning\n\nIn the introduction, the paper claims that \n>  Those dimensions exclusively used for [MASK] tokens have not been pretrained to represent real tokens, and will have to be either trained from scratch on downstream data, raising the risk of overfitting (Hendrycks et al., 2019; Kumar et al., 2022), or become unused, resulting in a waste of model capacity.\n\nI think the interplay between pretraining and fine-tuning is very complicated and not fully understood yet. Thus I don’t think this argument is substantiated. The authors cite Hendrycks et al. (2019) and Kumar et al. (2022) but I am not sure how these two works support this argument. Also, Figure (a) in this paper shows that inputs without a mask token have higher-rank representations. Doesn’t it just indicate that the impact of having mask tokens during pretraining does not impact the rank of real tokens when mask tokens are not used?\n\n## Main concern 2: The unwritten assumptions of Theorem 2.2 \n\n***It seems that Theorem 2.2 is based on some unwritten assumptions***, e.g. the model needs to be an attention-only model without MLP and residual layers.\n\n## Main concern 3: The mismatch between the setup for Lemma 2.1 and the setup for Theorem 2.2\n\nFollowing the previous concern, Theorem 2.2 seems to be  based on some unwritten unrealistic assumptions, while Lemma 2.1 is based on the empirical results that (full) transformers can fit real-world high-rank distributions. If we look at the paper by Dong et al. (2021), we can find that under the assumption Theorem 2.2 is based on, the rank of the representation converges to 1. This implies that, under this assumption, Lemma 2.1 does not hold. Therefore, I think it’s inappropriate to use Theorem 2.2 along with Lemma 2.1 to derive the conclusion of this paper.\n\n\n## Main concern 4: The weak implication of the theoretical results\n\nThe results only suggest that the representation of real tokens cannot be full-rank. It does not characterize how far the representation is from being full-rank. It is possible that the representation of the real tokens has rank $d - 1$. In this case, the so-called “representation deficiency” problem may not be a big issue.\n\n\n## Minor concern 1: The representation matrix of each example v.s. the whole corpus\n\nThe rank of the representation matrices discussed in Lemma 2.1 seems to be the representation matrix of the whole corpus. But in reality, we only feed in the model with a much shorter sequence of tokens, e.g., 512 tokens. In this case, the rank of the distribution of the masked tokens is at most 512 * 15% (in expectation), meaning that the mask tokens can’t occupy too many dimensions. \n\nThis may not be a big issue, but I think the arguments starting from Theorem 2.2 need to be rewritten a little bit. For example, in Theorem 2.2, because it’s about the sequence fed into a transformer model, the representation matrix should be of an example rather than of the whole corpus. Therefore, Lemma 2.1 can’t be used directly.\n\n\n## Minor concern 2: The contribution of this work\n\nThis work has some similarities with the work by Dong et al. (2021):\n\n1. Dong et al. (2021) also plot the representation rank of transformer models.\n2. Theorem 2.2 in this paper is largely based on the proof from Dong et al. (2021).\n\nI think the authors should give credit to Dong et al. more explicitly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)