ExLM: Rethinking the Impact of $\texttt{[MASK]}$ Tokens in Masked Language Models
Authors: Kangjie Zheng
Junwei Yang
Siyue Liang
Bin Feng
Zequn Liu
Wei Ju
Zhiping Xiao
Ming Zhang
Venue: ICML 2025 poster (2025)
Keywords: Masked Language Models
Pre-trained Models
Language Models
Text Modeling
SMILES Modeling
Primary area: deep_learning->foundation_models

Abstract: Masked Language Models (MLMs) have achieved remarkable success in many self-supervised representation learning tasks. MLMs are trained by randomly masking portions of the input sequences with $\texttt{[MASK]}$ tokens and learning to reconstruct the original content based on the remaining context. This paper explores the impact of $\texttt{[MASK]}$ tokens on MLMs. Analytical studies show that masking tokens can introduce the ***corrupted semantics*** problem, wherein the corrupted context may convey multiple, ambiguous meanings. This problem is also a key factor affecting the performance of MLMs on downstream tasks. Based on these findings, we propose a novel enhanced-context MLM, ExLM. Our approach expands $\texttt{[MASK]}$ tokens in the input context and models the dependencies between these expanded states. This enhancement increases context capacity and enables the model to capture richer semantic information, effectively mitigating the corrupted semantics problem during pre-training. Experimental results demonstrate that ExLM achieves significant performance improvements in both text modeling and SMILES modeling tasks. Further analysis confirms that ExLM enriches semantic representations through context enhancement, and effectively reduces the semantic multimodality commonly observed in MLMs.

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
Kangjie Zheng' Junwei Yang' Siyue Liang' Bin Feng Zequn Liu' Wei Ju Zhiping Xiao #4 Ming Zhang*
Abstract
This's [M], and I'm very [M] to see this.
Masked Language Model
Masked Language Models (MLMs) have achieved remarkable success in many self-supervised rep- resentation learning tasks. MLMs are trained by randomly masking portions of the input sequences with [MASK] tokens and learning to reconstruct the original content based on the remaining con- text. This paper explores the impact of [MASK] tokens on MLMs. Analytical studies show that masking tokens can introduce the corrupted se- mantics problem, wherein the corrupted context may convey multiple, ambiguous meanings. This problem is also key factor affecting the perfor- mance of MLMs on downstream tasks. Based these findings, we propose novel enhanced- context MLM, ExLM. Our approach expands [MASK] tokens in the input context and models the dependencies between these expanded states. This enhancement increases context capacity and enables the model to capture richer semantic in- formation, effectively mitigating the corrupted se- mantics problem during pre-training. Experimen- tal results demonstrate that ExLM achieves sig- nificant performance improvements in both text modeling and SMILES modeling tasks. Further analysis confirms that EXLM enriches semantic representations through context enhancement, and effectively reduces the semantic multimodality commonly observed in MLMs.
great
happy
awful
Sorry
They are independent of each other.
b
This's M2J
and I'm very [M21] [M22] to see this.
EXLM: MLM with Enhanced Context
great
awful
happy
sorry
Capture dependencies using transition matrix.
Figure I. Illustrations of the vanilla MLM (a) and EXLM (b). MLM can be affected by the multimodality problem. In ExLM, the model creates multiple hidden states for each [MASK] token (e.g.. [M. []. [M2,,] [2222)). By leveraging a larger semantic space and explicitly modeling the dependencies between these states, the model can capture richer semantic information in the enhanced context while mitigating the effects of multimodality.
1. Introduction
Pre-trained masked language models (MLMs) have achieved significant success across various types of sequence data, including text (Devlin, 2018; Liu, 2019; Lan, 2019; He et al., 2020; Joshi et al., 2020: Meng et al., 2023), small molecules (Wang et al., 2019; Ross al., 2022; Pan, 2023; Zheng et al., 2024a), and proteins (Lin et al., 2022; 2023; Su et al.,
2023; Zheng et al., 2024b; Hayes et al., 2025), establish- ing themselves as foundational approach for sequence representation learning tasks. To enable the effective ex- traction of useful semantic information, MLM employs a mask-ther then-predice training strategy. During the pre-training process, a certain proportion (typically 15%) of tokens in the input sequence are randomly replaced with special symbol, [MASX]. The model is then trained to predict the original tokens based on the corrupted context. This process allows MLMs to learn meaningful semantic information in a self-supervised manner from large-scale unlabeled data. The learned knowledge can effectively transferred to a
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking Uni- versity, China. 'Intermaiooal Digital Economy Academy (IDEA), Shenzhen, China. CCollege of Computer Science, Sichuan Univer- sity, Chengdu, China. 'Paul G. Allen School of Computer Science and Engineering. University of Washington, U.S.A. Correspon- dence to: Ming Zhang < <mzhang_ess leaiingss@@ppe es@@pku.cdu.cn.. Zhiping Xiao <patxiao uw.edu>..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
wide range of downstream tasks, such text classification and molecular property prediction, significantly improving the performance of deep learning models in these domains.
X1
X2
X3
X4
X5
First Repeat, then Mask
X1 [M]
X2x2
[MMM
X X4X4
K5
Although MLMs have achieved significant success in nu- merous tasks and applications, their effectiveness remains an important research problem. During the MLM training process, parts of the input are masked, and unreal tokens ( (IMASK) are introduced into the context. This process impacts the input context of MLMs in two critical aspects:
Masked Language Model
Masks that do not corrupt the semantics
Masks that do corrupt the semantics
The context provided to MLM during pre-training contains a large number of unreal tokens ([MASK) that are absent from real- world text, potentially distorting the learning process.
Introducing Unreal Tokens:
Figure 2. Illustrations of the Repeated MLM experiment. Each token in the model input is repeated before being masked. The ar- tificial redundancy introduced into the input ensures that replacing a token with [MASK] ((x)) does not necessarily lead to semaantic corruption the context, as the repeated tokens provide additional information to preserve the original semantics, A token is regarded as having corrupted semanties only when all its copies are masked.
Resulting in Corrupted Semantics: The replacement of tokens with [MASK] results in incomplete context semantics, which negatively affect the model's ability to learn accurate semantic representations.
different states. To efficiently train the EXLM, we propose a state alignment algorithm based on dynamic programming, which aligns target tokens with expanded states in a data- driven manner, significantly improving model performance.
These two aspects of impact are closely tied to the mask ratio used during pre-training. Higher mask ratios exacer- bate the problems of unreal tokens and corrupted semantics at the same time, leading to noticeable performance drop when the mask ratio is too high. While previous studies have investigated the unreal token problem and its impact on MLM performance (Clark, 2020; Meng et al., 2023), they have largely overlooked the corrupted semantics problem. Consequently, there is little work systematically exploring the impact of both problems on MLM performance, or eval- uating the magnitude of their effect independently. This gap arises because these two factors are interdependent and both rely on the mask ratio, making it challenging to design experiments that disentangle their respective effects.
In summary, the contributions of this work are follows:
We conduct the first systematic analysis of MLM be- havior from the dual perspectives of both unreal tokens and corrupted semanties. Through a series of carefully designed experiments, we reveal that the corrupted semantics problem has greater impact on MLM per- formance, providing new insights into MLM studies.
Based on the analysis, we propose EXLM, novel pre-trained model that enhances context representation for MLMs. By expanding [MASK] tokens into multi- ple states and employing a state alignment algorithm, EXL. effectively improves semantic modeling and reducing the semantic multimodality in the context.
To fill this research gap, we designed and conducted the Re- peated MLM analytical experiments (shown in Figure 2) to separately evaluate the relative impact of these two factors. The experimental results demonstrate that the corrupted semantics problem has a significantly greater impact on MLM performance than the unreal tokens problem. This is further reflected in stronger semantic multimodal- ity (Gu et al., 2017), where multiple plausible predictions exist for the original tokens due to the ambiguous or context- dependent meanings. Based on these findings, we propose a novel pre-trained model, ExLM, which enhances con- text representation in MLMs. By expanding each [MASK] token in the input context into multiple hidden states, the model is provided with a larger semantic space. This allows it to capture richer semantic information associated with each [MASK] token, thereby reducing the semantic multi- modality in token prediction. Furthermore, we introduce a transition matrix between these expanded states, enabling the model to directly capture semantic relationships among
Extensive experimental results on text and molecu- lar property prediction tasks demonstrate the superior performance of EXLM. Further analysis confirms its effectiveness and highlights its potential for addressing challenges in MI.M pre-training.
2. Background of MLM
The pre- pre-training objective of MLMs is to predict missing tokens from a partially masked input sequence. In this pro- cess, portion of the input tokens is randomly selected and masked (i.e., replaced with [MASK]) ), forming a corrupted sequence. The model's 's task is to recover the original tokens using the remaining unmasked context as input.
2

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
P
2.25%
15%
38.7%
62.2%
78.9%
2.25%
15%
38.7%
62.2%
78.9%
k
1
81.3
83.6
83.3
81.3
796
1
225%
15%
387%
62.2%
78.9%
Same mas k rañio
2
814
83.1
83.3
81.3
2.2%
13%
38.7%
62.2%
4
81.0
83.2
82.6
2.5%
15%
38 7%
8
79.2
82.8
2,25%
15%
Figure 3. Results of the Repeated MLM experiment. These are the evaluation results (i.e., accuracy) of Ms with different repetition times & and mask ratios the MNLI task (Williams et al., 2018), with results of similar performance highlighted in similar colors.
Figure 4. The corrupted semantics proportions corresponding to each set of Repeated MLMM experiments reported. When the repetition times are & and the mask ratio is p. the proportion of corrupted semantics p" (see Appendix for detailed proof).
Specifically, let's define a sequence of tokens as X In the MLM pre-training process, some of the tokens in X are randomly replaced with special to- ken [MASK], producing partially masked input sequence X
semantics has a significantly greater effect on MLM per- formance (Section 3.2). Additionally, our analysis further shows that more severe corrupted semantics lead to the loss of critical context semanties, which exacerbates the multi- modality in token predictions (Section 3.3).
(2i, ......, , J.
Jn.. where:
[MASK] if token z, is selected be masked, 24
3.1. Decoupling Corrupted Semantics and Unreal Tokens
otherwise.
In traditional MLM training, the mask ratio plays a crucial role in determining both the proportion of [MASK] tokens introduced into the context and the proportion of seman- tic information discarded from the original context. This interplay makes it challenging to isolate and compare the im- pact of these two factors-the the proportion of [MASK] tokens (i.c., unreal tokens) and the proportion of corrupted seman- tics-on on MLM performance. To address this challenge, we designed the Repeated MLM experiment.
The model is trained to predict the original tokens corre- sponding to the masked positions i in X. Typically, a fixed percentage of the input tokens (e.g., 15% ) are randomly selected to masked during the pre-training process. The objective of MLM pre-training to minimize the discrep- ancy between the predicted tokens and the true tokens the masked positions. This is achieved by maximizing the like- lihood of the true tokens given the context of the unmasked tokens. The objective function for MLM pre- preitiain can be formulated as:
The core idea of this experiment is to artificially introduce redundancy into the MLM's input, ensuring that replacing a token with [MASK] does not inevitably lead to semantic corruption in the context.
log P(æz] (i,..., .......,,).)) LMLM = Σ
REM
Specifically, as illustrated in Figure 2, before feeding the sequence into the MLM, we first repeat each token times, where h E N is hyperparameter set based exper- imental requirements. We then randomly mask the re- peated sequence at certain ratio p. which means replacing E (0.0%, 1 100.0%) of the tokens with [MASK]. This setup creates an interesting phenomenon: while the proportion of unreal [MASK] tokens in the context remains due to the masking ratio, the proportion of corrupted semantics changes. Since each token has ie copies and the probabil- ity of each copy being masked is p, the probability of the semantic information carried by token being completely corrupted becomes p². We also have provided detailed proof about the results in Appendix C.
where M denotes the set of indices corresponding the masked tokens, and P(x; .1.-..,2 ...) represents the model's predicted probability for a certain token æ; at the masked position i, conditioned on corrupted context formed by the unmasked tokens X.
3. Understanding the Impact of [MASK]
In this section, we design analytical experiments explore the impact of unreal tokens and corrupted semantics MLM performance. However, since these two factors are strongly interrelated in MLMs, we first designed a Repeated MLM experiment to decouple unreal tokens and corrupted semantics (Section 3.1). This allows us to separately explore the impact of each factor and demonstrate that corrupted
By keeping p fixed and varying we ensure that the pro-
3

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
portion of [MASK] tokens in the context remains constant while adjusting the degree of corrupted semantics. This enables us to control variables effectively, and systemat- ically compare the impact of these two factors on MLM performance by measuring each factor separately.
2.2%
15%
38.7%
62.2%
78.9%
k
Larger semantics comruption
1
162
278
3.95
5.79
7.26
2
1.37
2,30
3.44
5.48
3.2. What Matters More: Corrupted Semantics or Unreal Tokens
4
1.23
1.77
3.35
Under the experimental setup designed in Section 3.1, we have trained a series of MLMs with different repetition times and mask ratios p to analyze MLM's s behavior. For con- sistency, all training hyperparameters of the MLMs in the experiments kept the same except for p and k. Addition- ally, during downstream fine-tuning, the input is repeated with the same repetition times k as in pre vre-training. More detailed training configurations and hyperparameters can be found in Appendix B. The results of the MNLI task, evaluated using accuracy as the primary metric (Williams al., 2018), are presented in Figure We also provide the results of this experiments more tasks in Appendix F. Some entries are blank (gray areas) due overly low pro- portions of corrupted semantics (less than 0.05%), causing very low loss during pre-training and unstable model train- ing problem. For comparison, the proportions of corrupted semantics for each experiment are provided Figure 4.
8
1.22
1.76
Figure 5. Entropy analysis in the Repeated MI.M experiment. We visualize the entropy (in bits) of different models during mask prediction. Larger semantics corruption significantly leads to an increase in the entropy of the model's prediction distribution.
cantly different reconstruction outcomes. This phenomenon is referred to multimodality, concept initially proposed in NAT (Gu et al., 2017). Multimodality causes MLMs to produce more mixed and uncertain predictions during pre-training, thereby significantly affecting downstream per- formance. To gain a more straightforward understanding of multimodality, we analyzed the prediction entropy of MLMs with different repetition times and mask ratios p. We also have provided more details on the entropy calcula- tion process in Appendix E. As illustrated in Figure 5, with an increase in semantic corruption, the prediction entropy also rises, indicating more severe semantic multimodality. Such multimodal phenomenon also significantly impacts the performance of MLMs.
From these results, we can observe that both excessively large and excessively small corrupted semanties lead to significant performance degradation in the model. Besides, when the proportions of corrupted semantics remain constant and the mask ratio varies, the performance of the MIM changes only slightly. Although there is a minor decline in performance as the mask ratio increases (red cells in Figure 3, from 83.6 to 82.8), the overall performance remains relatively similar. As long as the proportions of not excessively high, the model can still maintain relatively good performance even if the context in the MLM pre-training process contains a large number of [MASK] tokens (e.g.. P- = 78.9%, k == 8).
On the other hand, when corruption too low (e.g.. 2.25%) the context still contains plenty of semantic clues, making token prediction too easy. At this point, the model's predic- tions also exhibit very low uncertainty. The training curve analysis in Appendix G also shows that when the corrup- tion is too low, the training task becomes overly simplistic. This simplicity prevents the model from learning deeper knowledge and ultimately reduces performance.
corrupted semantics
In contrast, when the mask ratio remains fixed and the pro- portions of corrupted semantics increase, the model's per- formance exhibits more significant changes (from 82.8 to 79.6). This demonstrates that the corrupted semantics problem has more pronounced impact on performance compared to the unreal tokens problem.
Therefore, an optimal approach would be design the input context with a reasonable degree of semantic corruption, en- suring the model can correctly handle the missing semantics and learn meaningful knowledge. This balance would avoid the negative impacts of the semantic multimodality problem while maintaining the model's performance.
3.3. Core Impact of Corrupted Semantics: Multimodality
4. Proposed Method: ExLM
We conducted deeper analysis to investigate how cor- rupted semantics influence the performance of MLMs. As illustrated in Figure 1(a), the reconstruction of a [MASK] token relies on the semantic information provided by its context. When a corrupted context is given, it may imply multiple different potential Semanties, resulting in signifi-
In this section, we will first introduce the core design con- cepts and overall architecture of EXLM (Section 4.1). Then, we will elaborate on two key components of context en- hancement, states expansion and dependency capture, by detailing the model design and training algorithm for each
4

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
Data Processing
ExLM
Pre-training Objective
2D RoPE
h1
RoPE(1.0)
Transformer Encoder
h2
X2
X2 x
RoPE(2.0)
X1
1
Rgg
h3
q:
0.2
C22
4.4
0.1
2
X
X2
Token Masking
[MASK]
[M]
[M]
RoPE(3.1)
h4
4
4.
6.2
0
0.1
[M]
[M]
x RoPE(3.2)
X3
[M]
hg
h
9
X
X4
x RoPE(4.0)
X
X4
h6
ht
4.
D
Xs
[M]
[M]
[M] RoPE(5.1)
it
hgg
45
1.0
4
2g
[M]
[M] x RoPE(5.2)
X
[M]
[M]
[M] RoPE(6.1)
ha
hg
96
5
X
kg
K2
Kg
kg
Kg
kg
X7
hg
[M]
[M] RoPE(8.2)
Transition Matrix
h10
X7
X7
x
RoPE(7.0)
States A Alignment
Figure 6. Overview of our proposed EXLM. EXLM creates multiple expanded states for each [MASK] token, providing a larger semantic space and stronger ability capture the missing semantics in the context. Additionally, it explicitly models the semantic dependencies between the expanded states using transition matrix, which computed based on the representation of each state h;-
(Section 4.2 and Section 4.3).
semantic space enables the model to capture richer and more diverse semantic information. better handling the semantic diversity introduced by multimodality.
4.1. Overview
Through the analysis in Section 3, we have identified that corrupted semantics is the main factor affecting MLM's performance. When semantics of the context are severely corrupted, the resulting multimodality makes it increasingly difficult for the MLM to restore the original tokens.
Dependency Capture: A transition matrix is used explicitly model the semantic dependencies between different states, and the States Alignment algorithm based on dynamic programming efficiently provides supervision signals to these hidden states.
Building on this, an intuitive approach to improving the MI.M is: how can enhance the model's ability to better handle semantic multimodality? More specifically, the semantic multimodality can be divided into two aspects:
Therefore, we will first introduce the details context en- hancement, which includes States Expansion and Depen- dency Capture (Section 4.2), and then present the States Alignment algorithm for ExLM pre-training (Section 4.3).
Intra-token Multimodality: The potential choices for each missing token become more diverse, and the significant semantic differences among these choices increase the semantic diversity and ambiguity.
4.2. Modeling Semanties with Enhanced Context
The core of context enhancement process is to create mul- tiple hidden states for each [MASK] token and capture the dependencies among these states, enhancing the model's ability to capture richer semantic information. This process consists of three main steps. First, the model expands each [MASK] token in the input into multiple hidden states, with the number of states determined by a hyperparameter k. Secondly, enable the model distinguish between these expanded states, further employs 2D Rotary Position Em- bedding (RoPE) to differentiate the states. Finally, EXLM uses a transition matrix to explicitly capture the semantic dependencies between these hidden states, enhancing the model's ability to model the missing semantic dependencies.
Inter-token Multimodality: The meaning of one ken is intricately linked to and influenced by the mean- ings of other tokens, resulting in complex semantic interactions and dependencies between missing tokens.
Therefore, we need to enhance the model's capabilities in two areas: one is improving its ability to model diverse and ambiguous semantics, and the other is enhaneing its ability capture semantic dependencies between missing tokens. Following this core idea, propose novel enhanced- context MLM (EXLM) with two main improvements:
For each [MASK] token the input context, multiple hidden states are created, en- larging the semantic space for the model. A larger
States Expansion:
Expanding [MASK] tokens in the inputs. A key idea in ExLM is to duplicate the embeddings of [MASK] tokens, effectively creating multiple "clones" for each [MASK]
5

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
before feeding them into the model. More formally, for each [MASK] token in the original input sequence X = [21, #2,,,,...., = [MASK]...., In]. we take its embed- ding @[MASK] and make & copies. We then form an ex- panded input sequence by replacing the single [MASK] token with its & duplicated embeddings:
Specifically, the representation of each state h, extracted by the Transformer Encoder undergoes an attention-like com- putation to derive the transition matrix E. This transition matrix E serves as the adjacency matrix of the DAG, quan- tifying the semantic association strength between different states. The computation is defined as follows:
X' (e Bzy MASK]]
MMASK] ex,,
E softmax
(OK'
+ +M)
Vd
where ez is the embedding of token 24. [MASK] is the i-th copy of [MASK] embedding MASKK] and & is a hyperpa- rameter controlling the number of cloned embeddings.
Q = HWo
HWWW.
where d is the hidden size, We and WK are learnable weight matrices, M is an upper triangular mask matrix ensuring that E remains an upper triangular matrix, thereby enforcing the DAG structure by preventing backward edges.
This expanded sequence X', consisting of embeddings for both the original tokens and the duplicated [MASK] tokens, is then passed into Transformer Encoder O for contextual encoding H (Vaswani et al., 2017):
Additionally, each state representation h, is passed through the model's token prediction head to compute the probability distribution over possible tokens for that state:
H [ha, h
(A) h [MASK] .... (MASK)....... h..l.
P = softmax (oftmax(HW.P)). (
where each h; represents the hidden state corresponding to the input embedding e, in X'. By explicitly expanding [MASK] tokens, the model is equipped learn richer and more diverse representations for the missing information, leveraging these enriched embeddings to better capture se- mantic information and reconstruct the original context.
where P represents the probability distributions for each state, Wp a learnable weight matrix for token prediction.
By using the transition matrix E, the model explicitly cap- tures the semantic dependencies between expanded states, enhancing its ability to reconstruct the missing semantic information effectively.
Using 2D RoPE to distinguish expanded states. In the context enhancement process, each [MASK] token is du- plicated multiple times, creating several "clones" that the model may find difficult to distinguish. To address this challenge, we introduce a 2D Rotary Position Embedding (RoPE) mechanism (Su et al., 2021), which leverages a second dimension in the positional information to differ- entiate these clones. Specifically, if the original [MASK] token is located at position in the sequence, its & duplicates assigned unique 2D positions: (i, 1), (f, (1, k). Meanwhile, all original (non- [MASK] ) tokens retain their original positions, represented as (j, 0), where j denotes their index in the sequence. Here, the first coordinate cap- tures the token's 's position in the original sequence, while the second coordinate differentiates the clones of a [MASK] to- ken. Using this two-dimensional positional structure, the 2D RoPE mechanism applies rotational position embeddings to encode both the sequence position and the clone index.
4.3. Pre-training of EXLM: States Alignment
A key challenge in training ExLM lies in the fact that each [MASK] token is expanded into multiple hidden states. This expansion means that there are more hidden states than the target tokens to predict. Consequently, we must de- termine an alignment between these states and the target tokens- that is, deciding which hidden state should be re- sponsible for predicting which token. This alignment pro- cess is at the heart of our States Alignment algorithm.
Formulating States Aligmment as a DAG decoding prob- lem. Our goal is to maximize the probability of the DAG decoding all target tokens (i.e., all masked tokens), For- mally, let I denote the set of all possible paths (i.e., all possible ways to align states the target tokens). Each path A E I represents one particular alignment of states to the target tokens in Y. The training objective can thus be written as a marginalization over all possible alignments:
Using a transition matrix to capture dependencies be- tween expanded states. The semantic dependencies be- tween expanded states can be modeled as directed acyclic graph (DAG), where each node represents state, and the edge weights indicate the strength of the semantic depen- dencies between pairs of states. Similar to previous work (Huang et al., 2022), we adopt DAG to effectively capture these semantic dependencies.
LSA log Po X') log Σ
Po PY X'), AET
where X' is the input sequence (with enhanced context), 0 denotes the model parameters, and Pa A X') repre- sents the probability of generating the target sequence Y through specific alignment A.
6

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
Table The overall results 7 molecule property classification datasets. We report ROC-AUC score (higher better) under scaffold splitting. The best results are bold. The second best results are underlined. indicates that the model uses the same training data, model architecture, training hyperparameters EXLM. For more detailed information about the dataset, please refer to Table 9.
Datascts
BACE
BBBP
Tox21
SIDER1
MUV1
ClinToxt
ToxCast
Mcant
# Molecules
1531
2039
7831
1427
93087
1478
8575
D-MPNN
80.9
71.0
75.9
57.0
78.6
90.6
65.5
74.2
Attentive FP
78.4
64.3
76.1
60.6
76.6
84.7
63.7
72.1
N-Gramar
77.9
69.7
74.3
66.8
76.9
77.5
GROVER
82.6
70.0
74.3
64.8
62.5
81.2
65.4
GraphMVP
81.2
72.4
75.9
63.9
77.7
79.1
63.1
73.3
Mole-BERG
80.8
71.9
76.8
62.8
78.6
78.9
64.3
73.4
3D InfoMax
79.7
69.1
74.5
60.6
74.4
79.9
64.4
71.8
SMILES -BERT
77.8
68.6
75.1
61.2
75.1
89.8
64.9
73.2
EXLM
79.6
72.8
78.2
64.5
78.8
91.6
66.9
76.1
This objective requires summing over all possible alignment paths E T, which can be computationally expensive. To address this, we employ a dynamic programming (DP) algo- rithm that efficiently computes this sum with a complexity of (M), where M is the number of all masked tokens.
and suitable for large-scale training. detailed analysis of the DP algorithm's efficiency provided in Appendix H.
Through this approach, ExLM effectively explores over all possible alignments between the expanded states and target tokens, leveraging both the token probability distribu- tions P and the transition matrix (the edges in DAG) E to reconstruct the missing semantics during pre-training.
More specifically, we adopted dynamic programming algo- rithm similar to that used in DA-Transformer (Huang et al., 2022). In this DP scheme, we define fi as the cumulative probability of all partial paths ending state u (a node in the DAG) that have generated the first tokens of Y. For- mally, indexes the states in our DAG in manner that respects the acyclic property (i.e., we only move forward in the state sequence), and ranges over the positions of the target sequence Y.
5. Experimental Results
We evaluate the performance of the ExLM model in both text modeling and SMILES modeling tasks. SMILES is sequential representation of molecular information, and a more detailed explanation can be found in Appendix D.
The DP recursion works by summing over all valid prede- cessors of (where v < u in the DAG):
We pre-train EXLM on data of text and SMILES sepa- rately. Then we fine-tune and evaluate ExLM across diverse benchmarks and verify the contribution of each component through ablation studies. Finally, visualization analysis is included to explain the advantages of unified modeling.
St.u S--1 x E x Pu(y))
BKN
5.1. Results on SMILES Representation Learning
where Ev,y is the transition score from state u to state u (de- rived from our transition matrix), P. (y.) is the probability of state predicting the i-th target token yt.
the large-scale molecular dataset provided by Zhou et al. (2023), which includes SMILES information for 19 mil- lion molecules. We tokenize SMILES sequences with the regular expression from Schwaller al. (2018). The pre- training hyperparameters can be found Appendix I.
Pre-training. For SMILES pre-training,
By computing fi, across all i and u, eventually obtain far, where M is the number of masked tokens in the target sequence Y, and L refers to the final state index. The final training objective is:
Fine-tuning. For fine-tuning, employ the widely- recognized MoleculeNet benchmark (Wu et al., 2018) 8). We follow the same data split as used by Zhou et al. (2023). De- tails the fine-tuning datasets and baselines can be found in Appendix K. We fine-tune the ExLM model on down- stream task datasets using three different random seeds and reported the average performance of the model.
LSA log fMs
This dynamic programming approach reduces the time com- plexity of the alignment problem to O(M x T.2), offering significant improvement over exhaustive path enumeration. In practice, the computation can be further optimized to (M) by leveraging parallelized operations provided by PyTo PoToch (Paszke, 2019), making the method highly efficient
Code is released at http://githhh.co///enngaagggiii

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
Table 2. The overall results the GLUE and SQuAD 2.0 development sets (medians over five random seeds). Results available in
pervious research are marked with "_", The "MEAN" column contains the averaged results across the eight GLUE tasks.
GLUE
SQuAD 2.0
Model
MNLI-(m/mm)
QQP
QNLI
SST-2
CoLA
RTE
MRPC
STS-B
MEAN
EM
F1
BERT
84.5/-
91.3
91.7
93.2
58.9
68.6
87.3
89.5
83.1
73.7
76.3
ALBERT
81.6/-
90.3
77.1
80.0
XLNet
85.8/85.4
92.7
78.5
81.3
UniLMv2
16.1/86.1
93.2
80.9
83.6
TUPE
86.2/86.2
91.3
92.2
93.3
63.6
73.6
89.9
89.2
84.9
ROBERTaa
85.9/85.8
91.6
92.3
93.7
64.3
75.5
88.7
89.5
85.2
78.3
81.5
EXLM
86.9/86.7
92.0
93.1
93.9
64.6
78.8
89.6
90.5
86.2
82.0
84.6
EXLMLARGE
87,8/87.5
92.2
93.8
94.5
65.3
79.1
90.4
91.2
86.9
82.6
85.0
Results. As show Table 1, ExLM achieves the best performance among all baseline models on 5 out of molec- ular property classification tasks and closely matches the best baseline models the remaining two tasks. A note- worthy result is that EXLM significantly outperforms the MLM pre-trained with the same model architecture, pre- training hyperparameters, and pre-training date-namely, the SMILES- -EERT model. This further underscores the
Table 3. Ablation studies. We assess the effectiveness of 2D RoPE and the transition matrix in EXLM and verify its efficiency.
Method
MNLI t
QNLIT
QQPt
RIET
Avg f
EXLM 2D RoPE
84.6
91.1
91.3
56.7
80.9
FXLM w/o Transitions
83.8
90.9
91.1
55.6
80.4
Vanilla MLM
83.6
90.0
90.3
54.7
79.6
Vanilla ML+++
84.4
91.2
90.6
80.7
EXLM
85.1
91.4
91.3
57.6
81.4
advantages of EXI.M over traditional MI.M pre-training tasks. ExLM also demonstrates the strongest average per- formance across all 7 tasks, indicating that it outperforms other baseline models in these prediction tasks overall.
table uses the same pre-training data and settings as ExLM, yet ExLM shows significant improvements. This demon- strates the superior performance and effectiveness of ExLM. EXLMLAnGE also achieves obvious performance improve- ments compared to EXLM due to the increased scale.
5.2. Results Textual Representation Learning
Pre-training. For textual pre-training, we adopt the En- glish Wikipedia and BookCorpus datasets (Devlin, 2018) as the pre-training dataset. The model size of EXLM is consistent with BERT base (Devlin, 2018). We also train EXLMLARGE with the same size as BERT large. For more details about pre-training settings, please see Appendix M.
5.3. Analytical Experiments of ExLM
We validate EXLM's effectiveness through ablation studies and explore the impact of expanded states and mask ratios on its performance. case study also demonstrates its good ability to capture rich semantic information. Efficiency and entropy analysis of ExLM are provided in Appendix P.
Fine-tuning. We evaluate the EXLM model using the the GLUE (Wang et al., 2018) and SQuAD 2.0 (Rajpurkar al., 2018) benchmarks. Detailed of the GLUE benchmark provided in Appendix L. For fine-tuning, the standard procedures used in BERT (Devlin, 2018) and RoBERTa (Liu, 2019). We also provide the hyperparameter search space for fine-tuning in Appendix N. Consistent with previous studies (Liu, 2019), all reported fine-tuning results represent the median values obtained from five different random seeds across both GLUE and SQuAD benchmarks.
follow
Ablation studies on ExLM. We perform ablation studies the 2D RoPE and transition matrix in ExLM, as shown in Table 3. The results reveal that removing the 2D RoPE (EXLM w/o 2D RoPE) or the transition matrix (EXLM w/o Transitions) causes a significant performance drop, with the transition matrix having a larger impact. This demon- strates the importance of the transition matrix in capturing semantic dependencies. Despite the performance decline from removing the transition matrix or the 2D RoPE, the expanded states still enable the model to capture semantic information more effectively, outperforming Vanilla ML.M. To assess ExLM's efficiency, we train an MLM with the same training cost, Vanilla MLM++, and show in Table that ExLM performs better, highlighting that ExLM can capture semantic information more efficiently than MLM.
Results. As shown in Table 2, we evaluate ExLM on the GLUE and SQuAD 2.0 development sets. ExLM achieves the best performance on SQuAD 2.0 and 7 GLUE tasks, and also performs very closely to the best baseline model on the remaining one GLUE task. Furthermore, EXLM demonstrates significantly higher average performance com- pared to other baseline models. Notably, RoBERTa in the
8

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
1.0
Table Performance comparison of MLM and EXLM with different mask ratios and numbers of expanded states k. EXLM demonstrates stronger modeling capability for contexts with a high mask ratios compared with MLM, and increasing k further improves its performance under high mask ratios inputs.
[805] 0.05
0.11
0.63
0.21
0.00
0.00
0.00
0.00
0.00
it
0.00
0.00
0.00
0.75
0.19
0.03
0.03 0.00
0.8
terrible
0.00
0.00
0.94
0.03
0.02 0.00
amazing
0,00
0.02
0.76
0.14
0.08
0,00
0.6
P
Method
M
QNLIT
QQPt
KIEt
Avg
beautiful
0.01
0.10
0.06
0.84
0.00
15%
Vanilla MLM
83.6
90.0
90.3
54.7
79.6
2
84.6
91.3
91.1
56.7
80.9
sorry
0.00
0.00
0.001.00
0.00
1.00
0.4
15%
ExLM-k=1
85.1
91.4
91.3
57.6
81.4
EXLM-K=S
84.4
91.0
90.9
56.9
80.8
glad
0.00 0.00
0.001.00
1.00
38.7%
Vanilla MLM
83.3
90.0
90.2
53.3
79.2
glad
0.00
0.2
M-K-
84.1
91.4
91.0
55.4
80.5
happy
1.00
38.7%
EXLM Ex1.M-k-8
84.9
91.6
91.2
57.0
81.2
0.0
84.3
90.9
90.7
56.8
80.7
terrible
amazing
beautiful
sorry
glad
glad
happy
[EOS]
The impact of l and p. We explore the impact of the number of expanded states k and mask ratios p ExLM's performance, with results shown in Table 4. As k increases, performance improves due to enhanced semantic modeling ability. However, when & becomes too large (e.g., k = 8), performance slightly declines due to the excessive expanded states of [MASK] tokens in the input context and redun- dant nodes in the DAG. Thus, with mask ratio of 15%, k = 4 proves to be the optimal choice. We further verify the impact of the mask ratio p on the performance of EXLM, with the results shown in Table 4. These results show that a higher mask ratio p has less impact on ExLM's perfor- mance compared to MLM, highlighting the enhanced se- mantic modeling capability of ExLM. Furthermore, ExLM needs more expanded states to model the multimodal and ambiguous context induced by higher mask ratio. Thus, increasing k appropriately at higher mask ratios enhances its performance. However, an excessively high mask ratio and too many expanded states (a higher k) can result in a large number of expanded states of [MASK] tokens in the input context, limiting further performance gains of EXLM.
Figure 7. Case study. We visualize the model's predictions when the input is "This is [MASK), and I'm very [MASK] to see this." (k = 4). The yellow nodes correspond to the expanded states of the first [MASK] token, while the brown nodes correspond to those of the second [MASK] token. The weights in the figure represent the transition probability between two nodes in the DAG, and the axis labels show the top-1 predicted word for each node.
offering new perspective on understanding MLMs. Based on our analysis, we propose EXLM, which enhances the model's semantic modeling ability through two key designs: States Expansion and Dependency Capture. These designs reduce the negative impact of the semantic multimodality on the model. We demonstrate EXLM's strong performance in both text and SMILES modeling scenarios. Ablation studies and case study also validate its effectiveness and efficiency.
Acknowledgements
Case Study, We conduct a case study to evaluate the model's performance on specific inputs, as shown in Fig- ure 7. The results demonstrate that the model effectively handles multimodal semantic information, capturing four possible semantic choices for each [MASK] token. The transition matrix models the relationships between these choices, for example, when the first [MASK] token is amaz- ing, the second is more likely to be glad, and when the first is terrible, the second tends to be sorry. A more detailed analysis can be found in Appendix O. These results show that ExLM can capture rich contextual information.
This paper is partially supported by grants from the Na- tional Key Research and Development Program of China with Grant No. 2023YFC334120 and the National Nat- ural Science Foundation of China (NSFC Grant Number 62276002). We would like to thank Jie Zhu from University of Oxford and Liang Ding from the JD Explore Academy for their insightful discussions on the project. We also thank all other members from Dlib in PKU for their valuable feed- back given during the internal discussions.
Impact Statement
6. Conclusions
This paper is the first work to systematically study the im- pact of [MASK] tokens on MLM from the perspectives of unreal tokens and semantics corruption, and proposes effective solutions to these problems. These analytical results provide valuable insights for designing better lan-
In this paper, we analyze the impact of the semantics corrup- tion caused by [MASK] tokens on MLMs, showing that it has a more significant impact than the unreal token problem,

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
guage representation learning models. Furthermore, MLM- based sequence representation learning models have already achieved significant success in various domains, including text, images, video, and scientific data. These models have also shown promising results in applications such as rec- ommendation systems and protein design. Therefore, the methods proposed this paper have the potential to drive further advancements in these fields, and the approach also holds significant application potential. However, we also acknowledge that this work inherits the potential negative impacts of existing MIMss such as the possibility of being used to generate false information or design and manufac- ture molecules with biological hazards.
Dolan, W. B. and Brockett, C. Automatically construct- ing a corpus of sentential paraphrases. In International Workshop on Paraphrasing (IWP), 2005.
Dong. L.., Yang, N., Wang. W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.-W. Unified language model pre-training for natural language understanding and generation. Advances in neural information process- ing systems, 32, 2019.
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. Glm: General language model pretrain- ing with autoregressive blank infilling, arXiv preprint arXiv: 203. 10360, 2021.
References
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):71 2021.
Bao, H., Dong, L., Wei, F., Wang, W., Yang, N., Liu, X.,
Wang, Y. Gao, J., Piao, S., Zhou, M., et al. Unilmv2: Pseudo-masked language models for unified language model pre-training. In International conference on ma- chine learning, pp. 642 652. PMLR, 2020.
Feng, B., Liu, Z., Huang, N., Xiao, Z.,, Zhang, H., Mirzoyan, S., Xu, H., Hao, J., Xu, Y., Zhang, M., et al. A bioactivity foundation model using pairwise meta-learning. Nature Machine Intelligence, 6(8):962-974, 2024.
Bao, H., Dong, L., Piao, S., and Wei, F. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.
Bentivogli, L.. Clark, P.. Dagan, I,, and Giampiccolo, D. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M. Shou, L., Qin, B., Liu, T., Jiang, D., et al. Code- bert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08//5: 2020.
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Spe- cia, L. Semeval-2017 task Semantic textual similarity multilingual and crosslingual focused evaluation. In- ternational Workshop Semantic Evaluation (SemEval), 2017.
Fu, Z.,, Zhou, W., Xu, Zhou, H., and Li, L. Contextual rep- resentation learning beyond masked language modeling. arXiv preprint arXiv:2204. 22ti::2004004666 04163, 2022.
Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B.
The third pascal recognizing textual entailment challenge. In ACL-PASCAL workshop textual entailment and paraphrasing, 2007.
Chithrananda, S., Grand, G., and Ramsundar, B. Chemberta: large-scale self-supervised pretraining for molecular prop- erty prediction. arXiv preprint arXiv:2010.09885, 2020.
Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling un- segmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp.
Clark, K. Electra: Pre-training text encoders as dis- criminators rather than generators. arXiv preprint arXiv:2003. 2020.
2006.
Dagan, I., Glickman, O., and Magnini, The pascal recog- nising textual entailment challenge. In Machine Learning Challenges Workshop, 2005.
Gu, J., Bradbury, J., Xiong, C., Li, V. O., and Socher, Non-autoreegressive neural machine translation. arXiv preprint
2017.
Dai, Y., Li, L.,, Zhou, C., Feng, Z., Zhao, E., Qiu, X., Li, P., and Tang, D. is whole word masking always better for chinese bert?": Probing on chinese grammatical error correction, arXiv preprint arXiv:2203.0023. 2022.
Guo, D.,, Ren, S., Lu, S., Feng, 7.,, Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., al. Graphcode- bert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08..0 2020.
Devlin, J. Bert: Pre-training of deep bidirectional trans-
formers for language understanding. arXiv preprint arXiv:18/10 1810.04805, 2018
Haim, R. B., Dagan, L., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., and Szpektor, I. The second pascal
10

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
recognising textual entailment challenge. In PASCAL Challenges Workshop Recognising Textual Entailment, 2006.
representing and predicting spans. Transactions of the association for computational linguistics, 8:64-77, 2020.
Ke, G., He, D., and Liu, T.-Y.
Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006. 15595, 2020.
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, Q., Deaton, J., Wiggert, M., et al. Simulating 500 million years of evolution with language model. Science, pp. eads0018, 2025.
Kingma, D. P. and Ba, Adam: A method for stochastic
optimization. arXiv preprint arXiv:1412.6988, 2014.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022a.
Lan, Albert: A lite bert for self-supervised learning of lan- guage representations. arXiv preprint 19914:1909.1194.. 2019.
Li, X. and Li, J. Bellm: Backward dependency enhanced large language model for sentence embeddings. In Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Long Papers), pp. 792-804, 2024.
He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding- enhanced bert with disentangled attention. arXiv preprint arXiv:2006. 03654, 2020.
He, P., Gao, J., and Chen, W. Debertav3: Im- proving deberta using slectra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint vrXiv:2111.09544 2021.
Liao, B., Thulke, D., Hewavitharana, S., Ney, H., and Monz, C. Mask more and mask later: Efficient pre-training of masked language models by disentangling the [mask] token. arXiv preprint arXiv:221 1.04898, 2022.
He, Z., Sun, T., Wang, K., Huang, X., and Qiu, X. Diffu- sionbert: Improving generative masked language models with diffusion models. arXiv preprint 1rXiv:2211.1500 2022b.
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., dos Santos Costa, A., Fazel- Zarandi. M., Sercu, T., Candido, S., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022:500902, 2022.
Hochreiter, S. Long short-term memory. Neural Computa- tion MI'--rress 1997.
Huang, C., Zhou, H., Jen, C., Zheng, K., Zaiane, O., and Mou, L. A decoding algorithm for length-control sum- marization based on directed acyclic transformers. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguis- tics: EMNLP 2024, pp. 11572-11583, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.866535//022..ndnnnnnnmm 677. URL http::tt////mm ////cccalln /awaaaaaaaaaacoooocccocoooooo.oa org/2024. findings- 677/.
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin. N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. Evolutionary-scale prediction of atomic-level protein structure with language model. Science, 379(6637): 1123-1130, 2023.
Liu, S., Demirel, M. and Liang, Y. N-gram graph: Simple unsupervised representation for graphs, with applications to molecules. Advances in neural information processing systems, 32, 2019.
Huang, F., Zhou, H., Liu, Y., Li, H., and Huang, M. Directed acyelic transformer for non-eutere machine trans- lation. In International Conference on Machine Learning, pp. 9410 90289942...2 PMLR, 2022.
Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H., and Tang, J. Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:2/10.07722. 2021.
Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint urXivv:907. 1692, 364, 2019.
Huang. F., Ke, P., and Huang. M. Directed acyelic trans- former pre-training for high-quality non-autoregressive text generation. Transactions of the Association for Com- putational Linguistics, 2023.
Meng, Y., Xiong, C., Bajaj, P., Tiwary, S., Bennett, P., Han, J., and Song, X. Pretraining text encoders with adversarial mixture of training signal generators. arXiv preprint arXiv:2204.03243, 2022.
Jaagg T., Huang, S., Luan, Z., Wang, D., and Zhuang, Scaling sentence embeddings with large language models. arXiv preprint 2rXhv:2307.16644 2023.
Meng, Y., Krishnan, J., Wang, S., Wang, Q., Mao, Y., Fang, H., Ghazvininejad, M., Han, J., and Zettlemoyer, L. Representation deficiency in masked language modeling. arXiv preprint arXiv:2302.0200 02060, 2023.
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., and Levy, O. Spanbert: Improving pre-training by

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
Namazifar, M., Tur, G., and Hrkk D. Warped language models for noise robust language understanding. In 2021 IEEE spoken language technology workshop (SLT), pp. 981-988. IEEE, 2021.
Shankar, I., Nikhil, D., and Kornél, C. First Quora dataset release: Question pairs, 2017. URL https: quora. com///quoradata///
Pan, J. Large language model for molecular chemistry. Nature Computational Science, 3(1):5-5, 2023.
Shao, C., Ma, Z., and Feng, Y. Viterbi decoding of directed acyclic transformer for non-autoregressive machine trans- lation. In Findings of EMNLP 2022, 2022.
Paszke, A. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv: 2019.
Shin, J., Lee, Y., Yoon, S., and Jung, K. Fast and accurate deep bidirectional language representations for unsuper- vised learning. arXiv preprint arXiv:2004.08097, 2020.
Peters, M. E., Neumann, M., lyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contex- tualized word representations. In Walker, M., Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long Papers). pp. 21-2-2222277 New Orleans, Louisiana, June 2018. Association for Compu- tational Linguistics. doi: 10.18865//1//118-2202 1202. URL https: / //aclaaantttt /taaa/aaaannhhoooooe org/ 1202/.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, D., Ng. A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Springer, J. M., Kotha, S., Fried, D., Neubig, G., and Raghu- nathan, A. Repetition improves language model embed- dings. arXiv preprint arXiv:2402. 15449, 2024.
Rajpurkar, P., Jia, R., and Liang, P. Know what you don't know: Unanswerable questions for SQuAD. In ACL, 2018.
Stärk, H., Beaini, D., Corso, G., Tossou, P., Dallago, C., Günnemann, S., and Liò, P. 3d infomax improves gnns for molecular property prediction. In International Con- ference on Machine Leurning, pp. 20479-20502. PMLR, 2022.
Ren, X., Wei, W., Xia, L., Su, L., Cheng, S., Wang, J., Yin, D., and Huang, C. Representation learning with large language models for recommendation. In Proceedings of the ACM on Web Conference 2024, pp. 3464 3475, 2024.
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position em embedding. arXiv preprint arXiv:2 2021.
Rong, Y., Bian, Xu, T., Xie, W., Wei, Y., Huang, W., and Huang. J. Self-supervised graph transformer on large- scale molecular data. Advances in Neural Information Processing Systems, 33: 33:12559-12577 12571, 2020.
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., and Yuan, F. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv, pp. 2023-10, 2023.
Ross, J., Belgodere, B., Chenthamarakshan, V., Padhi, I., Mroueh, Y., and Das, P. Large-scale chemical language representations capture molecular structure and proper- ties. Nature Machine Intelligence, 1(12):1256-1264, 2022.
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D.,
Al Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., et Transfer learning enables predictions network biology. Nature, 618(7965):616- 624, 2023.
Saharia, C., Chan, W., Saxena, S., and Norouzi, M. Non- autoregressive machine translation with latent alignments. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1098- 1108, Online, November 2020. Association for Computa- tional Linguistics. doi: 10. 10.18653/1122020eemmlpp 83. URI. https:tt/// / ac wwc.llahhooooy..rooooo. emnlp-main.. 83/.
Tong, Z., Song, Y., Wang, J.. and Wang, L. Videomae: Masked autoencoders are data-efficient leamerrs for self- supervised video pre-training. Advances in neural infor- mation processing systems, 35:10078- 2022.
Vaswani, A., Shazeer, N.. Parmar, N., Uszkoreit. J., Jones, L., Gomez, A. N., Kaiser, E.,, and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Schwaller, P., Gaudin, T., Lanyi, D., Bekas, C., and Laino, T. "found in translation": predicting outcomes of complex organic chemistry reactions using neural sequence-to- sequence models. Chemical science, 9(28),6091-6098, 2018.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop BlackboxNI.P. 2018.
12

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
Wang, L.,, Yang, N., Huang, X., Yang, L.., Majumder, R., and Wei, F. Improving text embeddings with large language models. arXiv preprint arXiv:2401.0088 00368, 2023.
boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry, 63(16):8749-876) 2019.
Wang, R., Chen, D., Wu, Z., Chen, Y., Dai, X., Liu, M., Jiang, Y.-G., Zhou, L., and Yuan, L. Bevt: Bert pre- training of video transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14733-14743, 2022.
Yang. F., Wang. W., Wang. F., Fang, Y., Tang, D., Huang. J., Lu, H., and Yao. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell ma- seq data. Nature Machine Intelligence, 4(10):852-866. 2022.
Wang, S., Guo, Y., Wang, Y., Sun, H., and Huang, J. Smiles- bert: large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM in- ternational conference on bioinformatics, computational biology and health informatics, pp. 429-436, 2019.
Yang, J., Zheng, K., Long, S., Nie, Z., Zhang, M., Dai, X., Ma, W.-Y., and Zhou, H. Mol-ae: Auto-encoder based molecular representation learning with 3d cloze test objective. bioRxiv, pp. 2024 04, 2024.
Yang, K., Swanson, K., Jin, W. Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., et al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 59(8):3370-3388, 2019.
Wang, X., Zheng, Z., Ye, F., Xue, D., Huang. S., and Gu, Q. Diffusion language models are versatile protein learners. arXiv preprint arXiv:2402.88566 2024a.
Wang, X., Zheng, Z., Ye, F., Xue, D., Huang, S., and Gu, Q. Dplm-2: A multimodal diffusion protein language model. arXiv preprint arXiv:2410.13782, 2024b.
Zhang, S., Huang, H., Liu, J., and Li, H. Spelling er- ror correction with soft-masked bert. arXiv preprint urXiv:2005.07422 2020.
Warstadt, A., Singh, A., and Bowman, S. R. Neural network acceptability judgments. In TACI. 2019.
Zheng. K., Wang, L., Wang, Z., Chen, B., Zhang, M., and Tu. Z. Towards unified training for levenshtein transformer. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Pp. 1-5, IEEE, 2023.
Weininger, D. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
Zheng. K.,, Liang, S., Yang. J., Feng, B., Liu, Z., Ju, W., Xiao, Z., and Zhang, M. Smi-editor: Edit-based smiles language model with fragment-level supervision. arXiv preprint arXiv:2412.05569 2024a.
Wettig, A., Gao, T., Zhong, Z., and Chen, D. Should you mask 15% in masked language modeling? arXiv preprint arXiv:2202.08005. 2022.
Williams, A.,
Nangia, N., and Bowman, S. A broad- coverage challenge corpus for sentence understanding through inference. In NAACI-HLT, 2018.
Zheng, K., Long, S., Lu, T. Yang. J., Dai, X., Zhang, M., Nie, Z.,, Ma, W.-Y,, and Zhou, H. Esm all-atom: Multi- scale protein language model for unified molecular mod- eling. In Forty-firs International Conference on Machine Learning, 2024b.
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Ge- niesse, C., Pappu, A. S., Leswing, K., and Pande, V. Moleculenet: benchmark for molecular machine learn- ing. Chemical science, 9:513-530, 2018.
Zhong, Q. Ding, L.,, Liu, J., Du, B., and Tao, D. Can chatgpt understand too? comparative study chatgpt and fined bert. arXiv preprint 131. 2023a.
Xia, J.. Zhao, C., Hu, B., Gao, Z., Tan. C., Liu, Li, and Li. S. Z. Mole-bert: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Leurning Representations, 2023.
Zhong, Q. Ding, L., Liu, J., Liu, X., Zhang, M., Du, B., and Tao, D. Revisiting token dropping strategy in efficient bert pretraining. arXiv preprint arXiv: /5273, 2023b.
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., and Hu, H. Simmim: simple framework for masked image modeling. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. pp. 9653-9663, 2022.
Zhou, G., Gao, Z., Ding, Q.. Zheng. H., Xu, H., Wei, Z., Zhang, L., and Ke, G. Uni-mol: A universal 3d molecu- lar representation learning framework. In The Eleventh International Conference on Learning Representations, 2023. URL /tt.ttts////peeeeeeeee openreview net / forum? i 23=6X2RM6VVRKU
Xiong, Z., Wang, D., Liu, X., Zhong. F., Wan, X., Li, X., Li, Z., Luo, X., Chen, K., Jiang, H., et al. Pushing the
13

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
Appendix
A. Related Works
A.1. Language Models for Representation Learning.
MLM for Representation Learning. Currently, language models serve as an important role in the self-supervised representation learning field and achieve excellent performance across wide range of tasks. The ELMo model (Peters et al., 2018) first introduces a supervveee language representation learning model based on bidirectional LSTM (Hochreiter, 1997). Later, the pre-trained masked language model based on the transformer architecture, BERT (Devlin, 2018), is proposed in the field of text modeling. BERT uses bidirectional transformer encoder to extract meaningful semantic representations from large amounts of text data and applies them to various downstream tasks such as text understanding (Devlin, 2018) and text correction (Zhang et al., 2020; Dai et al., 2022; Zheng et al., 2023). With the help of the transformer architecture (Vaswani et al., 2017) and the MLM training approach, this model achieves great success in the text domain. Following this, a large number of up works aim to improve these bidirectional encoder- -based masked language models (Liu, 2019; Ke et al., 2020; Shin et al., 2020; Namazifar et al., 2021; Du et al., 2021; Fu et al., 2022; Wettig et al., 2022; Meng et al., 2022; 2023; He et al., 2022b). Moreover, MLM also succeeds in various representation learning tasks, including images (Bao et al., 2021; Xie et al., 2022; He et al., 2022a), videos (Tong et al., 2022; Wang et al., 2022), codes (Feng et al., 2020; Guo et al., 2020; He al., 2021), small Moolecuees(Wang et al., 2019; Chithrananda et al., 2020; Ross et al., 2022; Pan, 2023; Feng et al., 2024; Yang et al., 2024; Zheng et al., 2024a), proteins (Elnaggar et al., 2021; Lin et al., 2022; 2023; Zheng et al., 2024b; Hayes et al., 2025), and single-cell data (Yang et al., 2022; Theodoris et al., 2023).
Other Methods & Summary. In addition to language models trained with the MLM pre-training approach, there are other types of language models used for representation learning tasks, such as autoregressive language models (Wang et al., 2023; Jiang et al., 2023; Springer et al., 2024; Ren et al., 2024; Li & Li, 2024), diffusion language models (Wang et al., 2024a;b), and unified language models (Dong et al., 2019; Bao et al., 2020; Du al., 2021). These models have demonstrated their effectiveness in representation learning across various tasks. However, overall, MLM is one of the most commonly used approaches for constructing language models for representation learning tasks and is a typical representative of this type of model. Zhong et al. (2023a) also points out that the MLM performs better than LLM in handling paraphrase and similarity tasks. Given the widespread impact and application of MLM, studying the corrupted semantics problem in MLM is of great significance, and it may also play a positive role in advancing the future development of MLM in different fields.
A.2. Studies of [MASK] in MLM pre-training,
Due to the widespread use of MLM and the important role of the [MASK] tokens in the MLM training process, several previous works have studied the impact of the [MASK] tokens on MLM. Clark (2020) proposed the ELECTRA model, an improvement on MLM, where the [MASK] token is not included the context during pre-training. This approach helps address the inconsistency between pre-training and downstream task fine-tuning caused by [MASK] unreal tokens. Liao et al. (2022) further studied the impact of the [MASK] unreal tokens on MLM models and pointed out that removing [MASK] tokens from the model's shallow representations during training does not affect MLM performance. Meanwhile, Meng et al. (2023) explored the representation deficiency problem caused by [MASK] unreal tokens from a theoretical perspective and proposed a MAE-based language model training approach. This approach improves the model's representation learning ability by removing the [MASK] tokens from the model input to avoid the unreal tokens problem. Additionally, Wettig et al. (2022) and Liao et al. (2022) also explored the impact of higher mask ratio on MLM performance. While these works have studied the effects of the [MASK] tokens, they focus only on the unreal tokens aspect and lack an investigation into the model's behavior from the perspective of semantic corruption. In contrast, Zhong et al. (2023b) points out that dropping [MASK] tokens causes semantic loss, affecting performance semantic-intense tasks like RTE (Dagan et al., 2005). This finding further highlights the negative impact of token dropping on the model's 's semantic modeling capability, emphasizing the importance of addressing the semantic loss problem. However, this work only analyzes the additional semantic loss caused by the token dropping process compared to the token masking process, and does not analyze the semantic loss caused by the token masking process itself. Therefore, its conclusion can not generalize to MLMs. This work aims to fill this gap by analyzing MLM's behavior from the perspective of both the semantic corruption problem and the unreal tokens problem.
14

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
B. Hlpee-Paraameter Configuration for Repeated MLM Experiments
B.I. Pre-training Configuration
In the Repeated MLM experiment, we train a series of MLMs with different p and & parameters (a total of 14 sets of models). When & large, the input sequence length of the model increases significantly, which also results in a high training cost. To accelerate the training process, we use larger batch size and a larger learning rate help the model converge more quickly, and also reduce the numher of training steps lower the overall training cost, ensuring that the training expense is manageable. The model size used in this part of the experiment is the same as BERT base (Devlin, 2018). Specifically, the model has 12 stacked Transformer layers, each with 12 attention heads. The model dimension and feedforward dimension of each Transformer layer are 768 and 3,072, respectively. The total number of parameters in the model is 1 128M. Furthermore, to ensure that the increase in & does not result in an excessively long sequence that the model cannot process, the model's Max sequence length parameter (the maximum length the position embedding) increases with k, with a specific value of 512* k. And we set the learning rate as 0.002 and warmup steps as 5K. The total training steps are 50K and each batch has 4096 samples at maximum.
To ensure the comparability of the results, all other pre-training hyperparameters in the MLM across different experimental groups are exactly the same, except for the Max sequence length parameter. In addition to the Repeated MLM experiment, we also used the same pre-training parameters in the ablation experiments to reduce pre-training costs. For more pre-training please refer to Table 5.
Table 5. Hyper parameters for the Repeated MLM experiments.
Hyper-panee parameters
Value
Learning rate
2e-3
LR scheduler
polynomial decay
Warmup updates
5K
Max updates
50K
Batch size
4. 096
FFN dropout
0.1
Attention dropout
0.1
Activation dropout
0
Num of layers
12
Num of attention heads
12
Encoder embedding dim
768
Encoder FFN dim
3, 072
Adam (8,, B2)
(0.9, 0.98)
Mask ratio Activation function
0.15
GELU
Weight Decay
0.01
Clip Norm
0.0
Max sequence length
512%k
B.2. Fine-tuning Configuration
To ensure the comparability of the experiments, the MLMs in different sets of experiments use exactly the same training parameters during the fine-tuning phase (except for the Max sequence length parameter, which is set the same as in the pre-training phase). Moreover, to ensure consistency between the pre-training and fine-tuning processes, we repeat each token the input sequences times during fine-tuning, using the same method as in the pre-training phase. Each MLM trained three times with different random seeds on each downstream task, and the average of these three results is taken as the final outcome. The detailed training parameters are provided in Table 6.
C. Expectation and Variance of the Proportion of Corrupted Semanties
To quantify the proportion of corrupted semantics, consider a sequence of N tokens, and each token is duplicated by & times, resulting in a total of N x & tokens. Each duplicated copy is independently masked with probability A token is deemed to have corrupted semantics if all of its copies are masked. Formally, the corrupted semantics proportion is defined as:
15

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
Hyperparameter
MNLI, QNLI, QQP, RTE
Peak learning rate
5e-5
Batch size
32
Max epochs
3
Warm-Up Proportion
6%
Max sequence length
512*k
Fine-tuning Seeds
{1, 2, 3)
Table 6. Fine-tuning Hyper-Paramen Parameter Configuration for Repeated MLM Experiments.
N
X;
N
where X; is an indicator variable defined by:
1, if all k copies of token i are masked,
X;
0, otherwise.
Each of these copies is independently masked with probability p. determine if the semantics of token are corrupted. we need all copies of the token be masked simultaneously. Thus the probability that all k copies of token are masked is simply the product of the individual masking probabilities. Therefore, the probability that X; meaning that all copies are masked and the token's 's semantics are corrupted, is given by:
P(X; = 1) = p²
Since X; is an indicator variable that takes the value if the token's 's semanties are corrupted, and otherwise, it follows Bernoulli distribution with parameter y i.e. X; Bernoulli(p"). Thus the expectation of X; is p and the variance of X; is p² (1 p*).
Therefore the expectation and variance of can be derived as follows:
N
1 N E[X,] =p" Var(s) = Var
N
N
p²11 - x+*) Var(X,
E(s]
X
X
N
N
N
N
N
This analysis leverages the independence masking each token. proves that when the repetition times are and the mask ratio is p. the proportion of corrupted semantics p². Moreover, because the input sequence length N is typically large, the variance of becomes vanishingly small. Concretely, let's take:
p = 0.387, k k= = 511.
Then
E[s] x² 0.3872 R 0.1498,
0.1498 (1 - 0.1498)
Var(s)
2.49
10
512
0, / Var(s) 0.0158.
Thus, even though the expected corrupted semanties proportion is about 15%, the standard deviation is only around 1.6%. confirming that is tightly concentrated around x² when N is large.
16

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
D. A Brief Introduction to SMILES
SMILES (Simplified Molecular Input Line Entry System) (Weininger, 1988) is a notation system used to represent chemical structures in a text format. It encodes molecular information using series of characters, where atoms are represented by their chemical symbols (e.g., C for carbon, O for oxygen), bonds by symbols like "_" (single bond), "=" (double bond), and "#" (triple bond), and branches are indicated with parentheses. For example, the SMILES representation of ethanol is "CCO", where "C" stands for carbon atoms and "O" represents oxygen atom, with a single bond between them. The notation captures the connectivity between atoms, bond types, and sometimes stereochemistry. making it a compact and easily readable way to represent molecular structures. The principle behind SMILES is that it provides a linear representation of the molecule that can be easily interpreted by both humans and computational systems, facilitating tasks like database searching, molecular simulations, and chemical informatics.
E. Calculation of Entropy in Repeated MLM
To analyze the uncertainty of MLM predictions, we calculate the entropy of the model's predicted distribution P for missing tokens. Entropy is measure of uncertainty, calculated as:
H[P]
Σ
P(z) log P(x),
WEVOC
where P(x) is the predicted probability of token a given masked position, voe is the vocabulary.
A larger entropy indicates more dispersed prediction distribution P. which also suggests that the model has higher uncertainty.
E.1. Adjustments for Repeated Tokens
In the Repeated MLM experiment, some tokens are repeated multiple times, which introduces additional considerations for calculating entropy. Without proper filtering, the calculated entropy would be influenced by tokens that are not fully masked. This means that the entropy would reflect the uncertainty associated with the model's predictions for partially visible tokens, resulting in an artificially lower entropy that fails to represent the true uncertainty under full masking conditions. To ensures the entropy calculation only considers tokens where all copies are masked, we introduce a filtering criterion. Any token with at least one unmasked copy is excluded from the computation.
For token that appears A times and is fully masked, the entropy is determined by averaging the individual entropies of the prediction distributions for each of its masked copies. This approach gives more consistent and meaningful representation of the model's uncertainty. Formally, the entropy of token with i masked positions is defined as:
H 1 - H[P(2)), K
where (Pxr)) is the entropy of the prediction distribution at the j-th repetition of the fully masked token, P(x;) represents the model's predicted probability distribution over the vocabulary at that position. The summation iterates over all k masked positions, and the resulting average provides the token's 's overall entropy.
This method ensures that the entropy metric accurately reflects the model's 's uncertainty in the fully masked scenario, without interference from partially visible tokens or the added complexity of repeated instances.
E.2. Data and Experimental Setup
This entropy analysis is conducted on the validation set to ensure that the results reflect the model's behavior on unseen data and avoid any training bias. By focusing on fully masked tokens and calculating their entropy, this approach provides a precise measure of the model's uncertainty in predicting missing semantic information.
17

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
F. More Results of Repeated MLM Experiments
P
2.25%
15%
38.7%
62.2%
78.9%
P
2.25%
15%
38.7%
62.2%
78.9%
k
1
81.8
84,1
83.9
81.4
80.3
1
88.1 1
90.0
90.0
88.6
86.9
2
81.6
83.5
83.1
81.9
88.2
89.7
89.5
88.7
4
81.8
83.6
82.7
87.8
89.0
88.7
8
79.8
82.5
864
88.8
(a) MNLI-m
(b) QNLI
2.25%
15%
38.7%
62.2%
78.9%
P
2.25%
15%
38.7%
62.2%
78.9%
k
1
89.2
90.3
90.2
89.5
89.2
1
504
54.7
5333
50.9
48.0
2
89.9
90.6
90.3
89.9
2
51.7
546
54.3
544
90.0
90.6
90.2
4
524
56.9
56.2
89.7
90.4
8
486
56.7
(c) QQP
(d) RTE
Figure 8. The results of the Repeated MLM experiments on MNLI-m, QNLI, QQP, and RTE tasks. We use accuracy in these four tasks as the metries. And for all tasks, higher values better. Similar performance marked with similar colors.
We extend the Repeated MLM experiment to broader range of tasks, with the results presented in Figure 8. From these results, we observe that the conclusions drawn in Section 3.2 still hold. Although the exact performance trends vary from task to task, the overall pattern remains clear and consistent: the severity of corrupted semantics directly correlates with performance degradation.
Specifically, the intensity of corrupted semanties changes, the model's performance these tasks exhibits significant fluctuations. Notably, the magnitude of these changes is substantially greater than the performance variations observed when the mask ratio is altered. This suggests that corrupted semantics plays more critical role in influencing model performance than the presence of unreal tokens.
18

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
G. Training Curves in the Repeated MLM Experiments
15.0
100
12.5
80
10.0
ML LLMMA Aco
60
Lo8s
7.5
PPL
40
5.0
2.5
20
0.0
0
10K
20K
30K
40K
50k
10K
20K
30K
40K
50k
Training steps
Training steps
Loss kc
Loos 1od4
Acc k-2
Loss k-1
Loss km2
Loss kmd
(a) Loss and Acc (Mask Ratio=38.7 7%)
(b) PPL (Mask Ratio=38.79%
25
80
0.8
50
40
15
0.8
MLMAco
LO58
30
10
20
0.2
10
10K
20K
30K
40K
50k
10K
20K
30K
40K
50k
Training steps
Training steps
Loss km1
Loss kmd
Asc Acc k=1
k-4
Loss k=2
Loss k=
Acc k:8
(c) Loss and Ace (Mask Ratio=62.229
(d) PPL (Mask Ratio=62.29%
1.0
200
0.8
30
150
MLM Aoc
188
PPL
100
0.4
0.22
0
0.0
10K
20K
30K
40K
50k
10K
20K
30K
40K
50k
Training steps
Training steps
Acc
km2
k-0
Loss lo12
Loss kooB
Aco 2o-2
Acc km
(e) Loss and Acc (Mask Ratio=78.9%)
(f) PPI. (Mask Ratio=78.9%)
Figure 9. The training loss, mask prediction accuracy and perplexity (PPL) curves of MLMs in the Repeated MLM experiments.
We plot the training eurves of the MLMS with different repetition times k and mask ratios p in the Repeated MLM experiments, shown in Figure 9, From this figure, we can observe that when we fix the mask ratio p and change the parameter k, as & increases (meaning more redundancy in the input). the intensity of semantics corruption in the context received by the model decreases. As a result, the model exhibits faster convergence, lower loss, lower perplexity and higher mask prediction accuracy during pre-training. This is also a key reason why model performance drops significantly when the semantics corruption in the input context are too low; at this point, the pre-training task becomes too simple, and the model converges quickly without learning meaningful information.
19

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
H. CUDA-Accelerated Dynamic Programming Framework: Details and Efficiency Analysis
Naively summing over all paths A E P can prohibitively expensive, since the number of possible paths grows exponentially. To solve this, we use dynamic programming (DP) approach, which is also adopted by the training algorithms for connectionist temporal classification (Graves et al., 2006; Saharia et al., 2020) and DA-Transformer (Huang et al., 2022; Shao et al., 2022; Huang al., 2023; 2024). We further adopted CUDA-accelerated dynamie programming algorithm from DA-Transformer (Huang et al., 2022) to efficiently perform state alignment during pre-training. The core of the algorithm utilizes CUDA (Compute Unified Device Architecture) to parallelize the computations involved in dynamic programming for sequence modeling. The primary objective is to efficiently calculate forward (a) and backward (3) probabilities, which are crucial for evaluating sequence likelihoods and performing gradient- srudient-based optimization during model training. By leveraging the parallel processing capabilities of GPUs, the algorithm distributes computations across numerous threads and blocks, significantly reducing runtime compared to traditional CPU-based implementations.
H.1. Dynamic Programming Framework: Forward Probability (a) and Backward Probability (3)
The forward algorithm computes the forward probability matrix a, where each element Qi,, represents the probability of reaching state u at position in the target sequence. The computation follows the recursive relation:
(a1-1, x E Pu()))),
0i
where and u index the states, E is the transition score from state v to state u, P Pu():) is the emission probability of state emitting the i-th token
CUDA Parallelization Strategy for the Forward Algorithm. Each GPU thread is assigned to compute az for specific states and positions, allowing multiple dg values to be calculated concurrently. Warp-level optimizations, are employed to efficiently perform reductions when summing contributions from multiple states. Additionally, shared memory is utilized to store intermediate results, and synchronization queues coordinate computations across different segments of the sequence. This parallel approach enables the simultancous computation of numerous a; values, thereby accelerating the forward pass across all sequence positions and states.
The backward algorithm calculates the backward probability matrix 3, where each element 3,a signifies the probability of transitioning from state u at position i to the end of the sequence. The recursive relation for 3 is defined as:
Br, -E (Bi11 Eu, x Po(y++1))
where u and u state indices, Eu, is the transition score from state state Po(1++1) is the emission probability of state v emitting the (i 1)-th token ye+1-
CUDA Parallelization Strategy for the Backward Algorithm. Similar to the forward pass, GPU threads are allocated to compute Bi,,, for specific states and positions. The backward pass processes the sequence in reverse order, ensuring that 3:+1. values are computed before 3ra values that depend on them. Warp-level primitives facilitate efficient summation of probabilities from future states, while shared memory and synchronization mechanisms maintain data consistency across different sequence segments. This parallelization allows for the rapid aggregation of probabilities from multiple future states, effectively computing the entire 3 matrix.
H.2. Gradient Calculations
Gradient computation is essential for optimizing model parameters during training. The algorithm calculates gradients with respect to both emission probabilities (y, and transition scores Eg,u- In the following formulas, M denotes the final position in the target sequence (i.e., the last index along the target length dimension).
VP, (P)) = Oi x 8i.u
Σn a.M.u
20

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
where En O.,u represents the likelihood of the entire sequence, computed by summing the forward probabilities o'r, over all possible final states u' the final position M. This gradient indicates how changes in the emission probability for state u at position i affect the log- oieliiil of the sequence.
The transition gradient is given by:
05-1, Bi,u
VE VeRu
Σn M,Mu
This gradient measures the effect of changes in the transition score between states and on the overall log-likelihood of the sequence.
CUDA Parallelization Strategy. For emission gradients, each thread independently computes VP. for assigned states and positions, leveraging parallel execution to handle multiple computations simultaneously. For transition gradients, threads collaboratively compute E,,, by aggregating contributions from various sequence positions using parallel reduction techniques. This parallel approach ensures efficient backpropagation through the dynamic programming steps, facilitating rapid parameter updates during training.
H.3. Time Complexity Analysis
Serial (Naive) Time Complexity. In a sequential implementation, computing the forward and backward probabilities involves iterating over batches, sequence positions, states, and transitions. The total number of operations scales as:
O (bsz x tarlen prelen translen),
where bsz is batch size, tarlen is target sequence length, prelen is the number of states, translen is the maximum number of transitions per state (usually equals to prelen).
CUDA Parallel Time Complexity. Under ideal parallel conditions, assuming the GPU can handle an effectively infinite number of parallel threads, which means that each sequence position can be processed in parallel during the forward and backward passes, reducing the time complexity to O (tarlen), since each position can be computed independently once the necessary dependencies are met. The gradient computations similarly benefit from parallel execution, allowing gradients for different states and transitions to be calculated concurrently.
While the theoretical parallel time complexity suggests (tarlen), actual performance is influenced by GPU resource constraints, such as the number of available threads, memory bandwidth, and synchronization overhead. In practice, the wall-clock time is significantly reduced compared to the serial case, though it may scale slightly worse than (tarlen) due to these hardware and implementation factors.
H.4. Space Complexity Analysis
The algorithm's space requirements are primarily determined by the storage of the a and 3 matrices, along with auxiliary tensors used for transition scores. The overall space complexity is:
O (bsz x prelen x (tarlen translen)).
where bsz is batch size, tarlen is target sequence length, prelen is the number of states, translen is the maximum number of transitions per state (usually equals to prelen).
This accounts for the forward and backward probability matrices and the transition scores necessary for computations.
Real- WealdWorld Analysis of Algorithm Space Usage. In practical scenarios, the space requirements of the CUDA-acceleratee dynamic programming algorithm remain manageable, even for large-scale tasks such as masked sequence modeling for text. For instance, consider a typical setup with a batch size of 64 per GPU, a maximum sequence length of 512 tokens, and a masking ratio of 0.15, resulting in approximately 77 [MASK] tokens per sequence. If each [MASK] token can be expanded
21

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
to up to 4 hidden states (k= = 4), the prelen (number of states) becomes approximately 512 0.15 4 R 308. Similarly, tarlen (the target sequence length) is approximately 77, while translen (the number of transitions per state) is set to 308, matching prelen. Under these conditions, tensors such as and B, each of size [bsz, tarlen, prelen], would require around 64 x 77 x 308 = 1.52 x 10 entries, equivalent to 5.8 MB per tensor for 32-bit floats. The transition links tensor, of size [bsz, prelen, translen], would require approximately 64 x 308 x 308 R 6.07 x 10 entries, or about 23.2 MB of memory for 32-bit floats. Altogether, these tensors occupy approximately 34.8 MB of memory. Even with these allocations, the memory usage well within the 80 GB available on an NVIDIA A 100 GPU, leaving ample room for model parameters, activations, and framework overheads. This demonstrates that the algorithm's Obssz x preten x (tarlen + translen)) space complexity poses significant constraint in real-world training setups.
States alignment algorithm does not affect the scalability of the model. The analysis of the time and space complexity of the states alignment algorithm reveals that its complexity is independent of the model size (e.g.. the number of layers or the embedding dimension) and depends only on the input sequence length. Therefore, with the training data unchanged, states alignment algorithm does not introduce additional training overhead the model size increases.
I. Hyper-Parameter Configuration for Molecular Pre-training
We implement EXLM using 9 stacked Transformer layers, each with 12 attention heads. The model dimension and feedforward dimension of each Transformer layer are 768 and 2,048, respectively. The total number of ExLM's parameters achieves 50.5M. We use Adam (Kingma & Ba, 2014) optimizer and polynomial learning rate scheduler to train EXLM, and we set the learning rate as 5e-4 and warmup stesp as 10K. The total training steps are 120K and each batch has 64k tokens at maximum. We implement the ExLM model using the Fairseq library 2 and train ExLM on two RTX3090 GPUs for about 24 hours.
For more pre-training sperrpaaamettrs, please refer to Table 7.
Table 7. ExLM
for molecular pre-training.
Hyper- parameters
Value
Learning rate
5e-4
LR scheduler
polynomial decay
Num of expanded states
2
Warmup updates
10K
Max updates
120K
Max tokens
64K
FFN dropout
0.1
Attention dropout
0.1
Activation dropout
0
Num of layers
9
Num of attention heads
12
Encoder embedding dim
768
Encoder FFN dim
2, 048
Adam (81, B2)
(0.9, 0.98)
Fragments Drop ratio
0.15
Vocabulary size
369
Activation function
GELU
Weight Decay
0.0
Clip Norm
1.0
J. Hyper-Parameter Configuration for Molecular Fine-tuning
In different downstream task, we use different hyper-parameters We run each task three times using three different random seeds, and take the average performance of these three runs as the final result. For detailed fine-tuning hyper-parameters, please refer Table 8.
22

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
Table 8. ExLM hyper-parameters for molecular fine-tuning.
Tasks
Epochs
Batch size
Learning rate
Warmup Ratio
Dropout
Pooler-dropout
BACE
60
le-4
0.06
0.1
BBBP
40
128
4e-4
0.06
0.1
0.1
TOX21
80
128
le-4
0.06
0.1
0.1
SIDER
100
32
Se-4
0.4
0.1
MUV
40
2e-5
0.2
0.1
ClinTox
100
256
5e-5
0.1
0.1
0.5
ToxCast
80
64
le-4
0.06
0.1
0.1
K. Details of Molecular Fine-tuning Datasets and Baselines
Datasets. We perform comprehensive set of experiments the MoleculeNet(Wu al., 2018) benchmark, focusing on the molecular property prediction task. MoleculeNet has emerged one of the most widely recognized and utilized benchmarks in the field of molecular property prediction, providing standardized platform for evaluating machine learning models' performances on evaluating molecular properties. Its datasets encompass a broad range molecular tasks, and address diverse and practical scientific problems such as drug discovery, toxicity prediction and so on.
In this section, we provide detailed summary of the statistics and fundamental characteristics of the MoleculeNet datasets we use in Table 9. This table offers information about the dataset sizes, task types, and compositions, providing readers with essential background information better understand the experimental setup and subsequent analysis.
Baselines. We evaluate our approach against various supervised learning and pre-training baselines, including both SMILES-based and 3D molecular pre-trained models. The supervised methods include D-MPNN (Yang et al., 2019) and AttentiveFP (Xiong et al., 2019), both of which are based on graph neural networks (GNNs). For 2D and 3D molecular pre-training, we consider baseline methods: N-gram (Liu et al., 2019), GROVER (Rong et al., 2020), GraphMVP (Liu et al., 2021), Mole-BERT (Xia et al., 2023), and 3D InfoMax (Stärk et al., 2022). For fair comparison, we train a SMILES model based on MLM pre-training, referred as SMILES-BERT. using the same training data, model architecture, and training hyperparameters as ExLM.
Table 9. Summary information of the MoleculeNet benchmark datasets.
Dataset
Tasks
Task type
Molecules (train/valid/testt)
Describe
BACE
1
Classification
1,210/151/151
Binding results of human
BACE-1 inhibitors
BBBP
1
Classification
1,631/204/204 1,182/148/148
Blood-brain barrier penetration
ClinTox
2
Multi-label classification
Clinical trial toxicity and FDA approval status
Tox21
12
Multi-label classification
6,264/783/783
Qualitative toxicity measure- ments
ToxCast
617
Multi-label classification
6,860/858/858
Toxicology data based in vitro screening
SIDER
27
Multi-label classification
1,141/143/143
Adverse drug reactions to the 27 systemic organs subset of PubChem BioAs- say
MUV
17
Multi-label classification
74,469/9,30999.309
23

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
L. Details of the GLUE Benchmark
Below are detailed descriptions of the GLUE benchmark tasks, which collectively evaluate various aspects of natural language understanding such as entailment, paraphrase detection, sentiment analysis, and grammaticality judgment:
MNLI (Multi-Genre Natural Language Inference). The MNLI dataset (Williams al., 2018) consists of 393K training examples gathered through crowdsourcing from various genres. The task requires predicting whether premise sentence entails, contradicts, is neutral with respect to given hypothesis sentence.
QQP (Quora Question Pairs). QQP (Shankar et al., 2017) includes 364K training examples sourced from the Quora grestion-answering platform. The objective is to determine if two provided questions are semantically equivalent.
QNLI (Question Natural Language Inference). Derived from the Stanford Question Answering Dataset (SQuAD) (Ra- ipurkar et al., 2018), QNLI comprises 108K training examples. The task involves predicting whether a sentence contains the answer to a specific question.
SST-2 (Stanford Sentiment Treebank). SST-2 (Socher et al., 2013) contains 67K training examples based on movie reviews with human-annotated sentiments. The goal to classify each sentence as expressing either positive or negative sentiment.
CoLA (Corpus of L Linguistic Acceptability). The CoLA dataset (Warstadt al., 2019) includes 8.K training examples extracted from books and journal articles focused on linguistic theory. The task is assess whether given sentence is linguistically acceptable.
RTE (Recognizing Textual Entailment). RTE (Bentivogli et al., 2009; Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007) encompasses 2.5K training examples derived from textual entailment challenges. The objective is to determine a premise sentence entails a provided hypothesis sentence.
MRPC (Microsoft Research Paraphrase Corpus). MRPC (Dolan & Brockett, 2005) consists of 3.7K training examples collected from various news sources. The task is to predict whether two given sentences are semantically equivalent.
STS-B (Semantic Textual Similarity Benchmark). STS-B (Cer et al., 2017) includes 5.8K training examples sourced from multiple origins, annotated by humans for sentence pair semantic similarity. The task requires predicting the degree of semantic similarity between two sentences scale from to
We use Spearman correlation for STS, Matthews correlation for CoLA, and accuracy for MNLI, QNLI, RTE and SST-2 as the metrics on GLUE.
24

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
M. Hyper- Hlyper-Parameter Configuration for Textual Pre-training
We implement EXLM in two configurations: a base model and a large model. The base EXLM consists of 12 stacked Transformer layers, each with 12 attention heads. The model dimension and feedforward dimension of each Transformer layer are 768 and 3,072, respectively, resulting in total of 128M parameters. The large ExLM model uses 24 Transformer layers with 16 attention heads per layer. The model dimension and feedforward dimension are increased to 024 and 4, 096, respectively, with total parameter count of 361M. We use Adam (Kingma & Ba, 2014) optimizer and polynomial learning rate scheduler to train ExLM, and we set the learning rate as 5e-4 and warmup steps as 10K. The total training steps are 125K and each batch has 2048 samples at maximum. We also implement the EXLM model using the Fairseq library. For more pre-training syper-parameters, please refer to Table 10.
Table 10. ExLM hyper-parameteen for textual pre-trainnn training..
Hyper-parameters
Value
Learning rate
5e-4
LR scheduler
polynomial _ecaa
Num of expanded states
4
Warmup updates
10K
Max updates
125K
Batch size
2, 048
FFN dropout
0.1
Attention dropout
0.1
Activation dropout
0
Num of layers
base: 12, large:
Num of attention heads
base: 12, large: 16
Encoder embedding dim
base: 768, large: 1,421
Encoder FFN dim
base: 3, 072, large: 4, 096
Adam (3),8)
(0.9, 0.98)
Mask ratio
0.15
Activation function
GELU
Weight Decay
0.01 0.0
Clip Norm
N. Hyper-Parameter Configuration for Textual Fine-tuning
We apply grid search for both the GLUE and SQuAD 2.0 datasets, and the grid search hyperparameters are shown in Table 11.
Table 11. Grid search hyperparameters for the GLUE and SQuAD 2.0 tasks.
Hyperparameter
MNLI, QNLI, QQP, SST-2
Peak learning rate
(1e-5, 2e-5, 3e-5, 4e-5) 32
Batch size
Max epochs
f2, 3, 5}
Warm-Up Proportion
6%
RTE, MRPC, CoLA, STS-B
Peak learning rate
(2e-5, 30-5, 4e-5, Se-5)
Batch size
{16, 32}
Max epochs
{2, 3, 5, 10)
Warm-Up Proportion
{6%, 10%)
SQUAD 2.0
Peak learning rate
(2e-5, 3e-5, 4e-5, 5e-5) {16, 32}
Batch size
Max epochs
{2, 3}
Warm- Up Proportion
{6%, 10%) }
25

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
O. Detailed Case Study
1.0
[BOS] (1.00)
0.05
0.11
0.63
0.21
0.00
0.00
0.00
0.00
0.00
(0.21)
wonderful (0.12)
0.00
0.00
0.00
0.75
0.19
0.03
0.03
0.00
(0.06)
0.8
terrible (0.27)
awful (0.11)
0.00
0.00
0.94
0.03
0.01
0.02
0.00
horrible (0.06)
amazing (0.18)
beautiful (0.16)
0.00
0.02
0.76
0.14
0.08
0.00
0.6
wonderful (0.14)
Nodes
beautiful (0.19)
wonderful (0.14)
0.01
0.10
0.06
0.84
0.00
incredible (0.07)
sorry 70.71)
glad (0.19)
0.00
0.00
0.00
1.00
0.4
happy (0.04)
glad (0.56)
happy (0.23)
0.00
0.00
1.00
excited (0.11)
glad (0.51)
0.2
happy (0.24)
0.00
1.00
pleased (0.08)
happy (0.64)
glad (0.23)
1.00
pleased (0.05)
wonderful (0. -2)
news (0.06)
terrible (0.27)
awful (0.11)
horrible (0.06)
amazing (0. (0.18)
beautiful (0.16 (0.16)-
wonderful (0.14)
beautiful (0.19)
wonderful (0.14)
incredible (0.07)
sorry (0.71)
glad (0.19)
happy (0.04)
glad (0.56)
happy (0.23)
excite (0.11)
glad [0.51]
happy (0. (0.24)
pleases (0.08)
glad (0.23)
pleased (0.05)
(1.00)
it (0.21)
happy
Nodes
Figure We visualize model's predictions when the input "This [MASK], very [MASK] see this." (k 4). The yellow nodes represent the expanded states corresponding to the first [MASK] token, while the brown nodes represent the expanded states corresponding to the second [MASK] token. The x-axis and y-axis show the top-3 predicted word for cach graph node.
We further provide a visualization of the model's predictions when the input is This is [MASK], and I'm very [MASK] to see this., as shown in Figure 10. Specifically, since the input contains two [MASK] tokens and k = 4, there are 8 nodes in the graph. For implementation convenience, we introduce two special nodes, [BOS] and [EOS] ], which correspond to the starting and ending nodes of the graph. We require that all paths the graph must start from the [BOS] node and terminate the [EOS] node, when searching for valid path in the graph, enumerating all possible starting nodes. In this graph, besides the heatmap representing the transition probability (i.e., the edge weight between two nodes), we also display the top 3 most probable words decoded from each graph node. From these results, we observe the following characteristics of ExLM:
directly start from the [BOS] node without
ExLM successfully represents the different choices of each [MASK] token using distinct expanded states. Since A: = 4, the model predicts four different choices for both [MASK] tokens in the input. These four choices typically represent different aspects of the semantics. For example, the first [MASK] token's 's choices include three categories: a noun (c.g., it, news), negative emotion words (c.g., terrible, awful), and positive emotion words (e.g., amazing, beautiful, wonderful), each represented by a different expanded state. This effectively avoids semantic ambiguity the
26

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
context, i.e., the semantic multimodality. Similarly, the second [MASK] token's choices include positive emotions (e.g.. glad, happy) and negative emotions (e.g., sorry), and these different types of semantic information are also represented by distinct expanded states.
ExLM successfully captures the semantic relationships between different [MASK] tokens. The transition matrix displayed in Figure 10 shows important phenomenon: the degree of dependency between different states in EXLM (i.e., the edge weight between when the first [MASK] token is "terrible," the second [MASK] token has higher probability of choosing "sorry" Conversely, when the first [MASK] token is a more positive emotion word, the second [MASK] token is more likely to choose "glad," which also represents positive emotion. This demonstrates that EXLM can effectively learn the semantic dependencies between [MASK] tokens. We further visualize this as a directed acyclic graph (DAG) in Figure 11, where the edge weight between nodes directly related to the semantic dependency between them, further validating the effectiveness ExLM.
nodes) is strongly correlated with the semantic similarity between them. For instance,
EXLM typically has lower uncertainty for [MASK] tokens that appear later. As shown in Figure 10, the word probability distributions of the four expanded states corresponding to the first [MASK] token are often more evenly distributed, meaning that the differences word probabilities across different words in each state are not very large. However, for the four expanded states corresponding to the second [MASK] token, the model usually predicts one word with probability significantly higher than the others. This phenomenon occurs due the directionality of the DAG. Specifically, in ExLM's DAG, transitions only occur from earlier states to later states, and the choices of earlier states have greater impact on the final result, which means that the earlier states tend to have higher uncertainty.
it (0.21)
sorry (0.71)
wonderful 12) 0.75
glad (0.19)
news (0.06)
happy (0.04)
0.19
1.0
0.02
0.94
0.s
terrible
glad (0.56)
0.05
horrible
happy (0.23)
0
8.11
[B0S] (1::0)
[EOS] (1.00)
02
0.0
0.76
1.0
amazing (0.18)
glad (0.51) happy (0.24)
beautiful (o. (0.16)
0.18
wonderful (0.14)
601
pleased (0.08)
0.00
beautiful (0.19)
happy (0.64)
(0.14)
glad (0.23)
Incredible (0.07)
pleased (0.05)
Figure 1t. The visualization of the DAG from ExIMM. It shows that the edge weight between different nodes is directly related to the semantic dependency hetween those nodes.
27

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
P. More Analytical Experiments of ExLM
P.I. Entropy Analysis on ExLM
We conduct am entropy analysis to compare the severity of the semantic multimodality under different mask ratios in the input context for ExLM and MLM. The results are shown in Figure 12. The results reveal that ExLM has significantly lower entropy in its prediction distribution than MLM, indicating lower uncertainty in dealing with ambiguous contexts. This further proves that EXLM significantly mitigates the negative impact of semantic multimodality and can still produce relatively confident predictions even when contextual semantic corruption is severe. Additionally, as A increases, the uncertainty of ExLM decreases, suggesting that increasing &o expands the model's ' semantic space, helping to alleviate the negative effects of the semantic multimodality. Moreover, under higher mask ratios, ExLM shows a more noticeable reduction in uncertainty compared to MLM, explaining why ExLM still maintains relatively strong performance under high mask ratios.
4.0
3.95
Vanila MLM
ExLM-K 2
ExLM-K 4
3.5
Entropy (bits)
3.16
ExLM-K = 8
3.0
2.78
2.52
2.5
2.40
2.31
2.16
2.14
2.0
1.5
p=15%
p=38.7%
Figure Entropy analysis of EXLM with different mask ratios p and numbers expanded states EXLM demonstrates significantly lower uncertainty than MLM, and a larger & further reduces the ExLM's uncertainty. enhancing its better modeling capability.
P.2. Training Efficiency of ExLM
We compare the training efficiency differences between MLM and EXLM (k: = 4) with the same pre-training configuration. Both models are trained on two Tesla A100 80G GPUs under the hyperparameters from Table and their training cost statisties are shown in Table 12. From this table, we can know that the training cost of ExLM (k: = 4) approximately 1.9 times that of the MLM with the same pre-training configuration. To compare the performance differences between MLM and ExLM (k- 4) under equal training costs, increase the training steps of MLM from 50,000 to 100,000, and refer to this model as Vanilla MLM++. The performance of Vanilla MLM++ is shown in Table 3, where we can see that although increasing the training steps significantly improves MLM performance, it still remains lower than that of EXLM. This further demonstrates that ExLM achieves stronger performance under the same training cost.
Table 12. Training time comparison (Tesla A100 80G GPUs) between EXLM (k = 1) and MLM.
MLM
EXLM (k: 4)
GPU Time (Hours)
54.7
104.2
P.3. Performance of Sparse EXLM
We also implement a version of ExLM using a Sparse DAG (Sparse EXLM) and evaluate its performance on various downstream tasks, with the results shown in Table 13. Specifically, the key difference between the Sparse DAG and the original DAG in ExLM is that, in the Sparse DAG, we do not allow transitions between expanded states that belong to the same [MASK] token. This significantly reduces the number of edges in the DAG, making the graph more sparse.
28

---

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
However, as shown in Table 13, EXLM with the Sparse DAG experiences some performance degradation. While it still outperforms the version of ExLM without transition matrix (EXLM w/o Transitions), it falls behind the original ExLM. The reason for this decline is that, in the Sparse DAG, only one expanded state can be selected for each [MASK] token, which prevents the model from capturing longer but possible outputs. For example, in the sentence "This is [MASK), and I I'm very [MASK] to see this," both "so good" and "happy" are reasonable interpretations for the first and second [MASK] tokens, respectively. However, the Sparse DAG cannot model this ambiguity, as does not allow multiple expanded states being chosen for single [MASK] token. This limitation significantly restricts the model's ability to capture rich semantic variations, leading to a performance drop.
Table 13.
Performance of Sparse ExLM. Using a sparser DAG in ExLM leads to performance decline.
Method
MNLI
QNLI
QQP
RTE1
Avg f
ExLM w/o Transitions
83.8
90.9
91.1
55.6
80.4
ExLM w/ Sparse DAG
84.4
91.2
91.3
56.9
81.0
Vanilla MLM
83.6
90.0
90.3
54.7
79.6
ExLM
85.1
91.4
91.3
57.6
81.4
29


Reviews:
[{"review_id": "lf7W0Xf8Jg", "reviewer": "Reviewer_wzyN", "summary": "This work presents a deeper analysis into the effectiveness of mask token in MLM pre-training.\nThe authors argue that the conventional use of [MASK] tokens can lead to a \"corrupted semantics problem\" where the masked context may become ambiguous and lead to multiple interpretations.\nTo highlight this issue, the authors conduct a series of analytical studies, namely Repeated MLM, showing that the corrupted semantics problem as a more significant factor than the unreal token problem that can affect the performance of MLMs when applied to downstream tasks.\nTo address this challenge, the paper introduces anovel pre-trained model ExLM.\nThe key idea of ExLM is the expansion of [MASK] tokens in the input context into multiple hidden states, which allows the model to model a larger semantic space.\nBy doing so, ExLM aims to increase the context capacity and capture richer semantic information, thereby reducing the ambiguity introduced by the masked tokens and reducing the semantic multimodality in token prediction.\nExperiments on various NLU tasks demonstrate the effectiveness of ExLM.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We sincerely appreciate your valuable suggestions. Below, we address all your concerns regarding the SMILES tasks, the standard deviation of model performance, and additional NLU tasks. The relevant results will be incorporated into the final camera-ready version of our paper. We hope our responses help clarify the work, and we welcome any further comments.\n\n---\n\n**Q1: Why do they take SMILES as the evaluation task?**\n\n**A:** Thank you for the question. The reason for selecting SMILES as the evaluation task is that masked language modeling (MLM) on SMILES also encounters issues similar to those observed in natural language modeling. We conducted Repeated MLM experiments on the BACE task (a SMILES classification task), and the results are shown in the table below. These results exhibit a trend consistent with Repeated MLM experiments on natural language data, indicating that semantic corruption significantly affects SMILES modeling:\n\n| k \\ P | 2.25% | 15%  | 38.7% | 62.2% | 78.9% |\n| ----- | ----- | ---- | ----- | ----- | ----- |\n| 1     | 57.6  | 73.5 | 70.4  | 68.6  | 63.0  |\n| 2     |       | 69.5 | 72.7  | 69.6  | 65.3  |\n| 4     |       |      | 70.1  | 72.0  | 67.2  |\n| 8     |       |      |       | 69.9  | 70.7  |\n\nThese findings can be attributed to the presence of strong semantic dependencies in SMILES representations. For instance, forming valid molecular functional groups often requires precise coordination among multiple atoms, which conventional MLM fails to capture due to its inability to model such semantic associations effectively.\n\n---\n\n**Q2: The standard deviation of the results in Table 1 and Table 3**\n\n**A:** Thank you for the helpful suggestion. We have computed the standard deviations of ExLM’s performance on both the GLUE and MoleculeNet benchmarks. The results are summarized in the tables below. Overall, ExLM demonstrates good stability across these benchmarks:\n\n*Standard deviation on MoleculeNet benchmark:*\n\n| Task    | ExLM Performance |\n| ------- | ---------------- |\n| BACE    | 79.6±0.8         |\n| BBBP    | 72.8±1.2         |\n| Tox21   | 78.2±0.1         |\n| SIDER   | 64.5±0.7         |\n| MUV     | 78.8±0.6         |\n| ClinTox | 91.6±1.8         |\n| ToxCast | 66.9±0.3         |\n| Mean    | 76.1             |\n\n*Standard deviation on GLUE benchmark:*\n\n| Task   | ExLM Performance |\n| ------ | ---------------- |\n| MNLI-m | 86.9±0.14        |\n| QQP    | 92.0±0.09        |\n| QNLI   | 93.1±0.43        |\n| SST-2  | 93.9±0.5         |\n| CoLA   | 64.6±0.81        |\n| RTE    | 78.8±1.8         |\n| MRPC   | 89.6±1.0         |\n| STS-B  | 90.5±0.72        |\n| MEAN   | 86.2             |\n\n---\n\n**Q3: There should be more results on NLU tasks to demonstrate the effectiveness of ExLM**\n\n**A:** Thank you for the excellent suggestion. We further evaluated the performance of ExLM on the SuperGLUE benchmark [Wang et al., 2019]. As shown in the table below, ExLM outperforms several baseline models by a notable margin, which demonstrates the effectiveness of our method:\n\n|            | BoolQ (Acc.) | CB (Acc.) | COPA (Acc.) | MultiRC (F1) |\n| ---------- | ------------ | --------- | ----------- | ------------- |\n| BERT [Devlin et al., 2019]       | 74.4         | 83.9      | 63.0        | 68.1          |\n| Token Drop [Hou et al., 2022] | 73.0         | 83.9      | 64.0        | 67.7          |\n| SCTD [Zhong et al., 2023]       | 73.8         | 87.5      | 68.0        | 68.9          |\n| ExLM       | **76.7**     | **88.0**  | **69.1**    | **71.3**      |\n\nWe hope these additional results provide more clarity and demonstrate the strong performance of our proposed model. Please let us know if further information are needed!\n\n**References:**\n\n[Devlin et al., 2019] Devlin, Jacob, et al. \"Bert: Pre-training of deep bidirectional transformers for language understanding.\" NAACL 2019.\n\n[Hou et al., 2022] Hou, Le, et al. \"Token Dropping for Efficient BERT Pretraining.\" ACL 2022.\n\n[Zhong et al., 2023] Zhong, Qihuang, et al. \"Revisiting Token Dropping Strategy in Efficient BERT Pretraining.\" ACL 2023.\n\n[Wang et al., 2019] Wang, Alex, et al. \"Superglue: A stickier benchmark for general-purpose language understanding systems.\" Advances in neural information processing systems 32 (2019).", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "The soundness of evaulation experiments has some concerns.\nFirst, the authors do not provide explanations about why they take SMILES as the evaluation task.\nIt seems that the patten in molecular information cound be quite different with that in textual information.\nThere should be some explanations about why the authors take this task and why the proposed method can work (as the preliminary analysis are mainly made on MNLI which is an NLU task).\nSecond, I think there should be more results on NLU tasks to demonstrate the effectiveness of ExLM, because part of the performance gains shown in Table 3 are not significant enough.", "relation_to_broader_scientific_literature": "This work contributes to the prior analysis about the effectiveness of mask tokens.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The main weakness lies in the experiments.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Can you report the std of the results in Table 1 and Table 3? Some performance gains in these tables are quite marginal, so the std coul d hlep to clarify the significance of the results.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vvo50U7X9m", "reviewer": "Reviewer_bAAY", "summary": "This paper gives a new way of utilizing [MASK] tokens in masked language models. It first performs an analysis of the semantic aspects of the [MASK] token and then proposes ExLM wherein multiple [MASK] tokens are introduced during pre-training. The authors then propose to utilize a learnt transition to get multiple semantic alignments which makes the model semantically richer and performs better than the previously proposed model on a suite of tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions. In the following, we provide detailed responses to all your concerns regarding the missing reference, model details, writing organization, and implementation issues. These discussions will also be incorporated into the final camera-ready version of our paper. We sincerely welcome any further feedback you may have.\n\n---\n\n**Q1: Missing reference about the noising techniques for ASR error robustness**\n\n**A:** Thank you for your suggestion. We have now added the missing citation to the draft of our paper, and we will also ensure that it is properly included in the final camera-ready version.\n\n---\n\n**Q2: The explanation of algorithm details should be in the main text**\n\n**A:** Thank you for the feedback. We have reorganized the structure of the paper accordingly. In earlier versions, some important methodological details were placed in the appendix due to space limitations. However, since the camera-ready version allows for an additional page, we have moved more details about ExLM—such as the dynamic programming procedure and the forward-backward algorithm—into the main body of the paper.\n\n---\n**Q3: The relation between Section 3 and Section 4 is not clear**\n\n**A:** Thank you for the insightful comment. We have revised the writing of Section 4.1 to more clearly highlight the connection between Section 3 and Section 4. Specifically, the analysis in Section 3 demonstrates that *multimodality* is a key factor impacting the performance of MLMs. To enhance ExLM’s capacity for handling multimodality, we propose two mechanisms tailored to address two distinct levels of multimodality:\n\n1. **Intra-token Multimodality**: The possible candidates for each masked token can be highly diverse. To address this, we introduce a *States Expansion* mechanism to construct a larger semantic space, allowing the model to learn richer and more diverse semantic information.\n2. **Inter-token Multimodality**: The meaning of a token is intricately related to the meanings of surrounding tokens. To capture this, we incorporate a *Dependency Capture* mechanism, where a transition matrix is used to explicitly model semantic dependencies between different states.\n\nThis design clearly establishes the relevance of the findings in Section 3 to the architectural choices in Section 4. These improvements have been added to the camera-ready version.\n\n---\n\n**Q4: The impact of corrupted semantics in MLM is obvious and well-managed, so it’s unclear why it’s framed as a significant issue**\n\n**A:** Thank you for raising this important point. We agree that \"corrupted semantics\" are intentionally introduced as a form of noise in the MLM denoising process, and such noise is a legitimate component of the training strategy. Indeed, our goal is not to eliminate this noise—on the contrary, we explicitly mention in the paper that a certain level of corrupted semantics can benefit training.\n\nHowever, **the central issue we aim to address is the unintended side effects this noise may have on model behavior.** Specifically, the purpose of the denoising process is to train the model to recover clean data from noisy inputs. While noise facilitates learning, it also modifies the model's input distribution, which may interfere with the model’s ability to learn accurate representations.\n\nTherefore, a well-designed denoising process should maintain a reasonable noise level while minimizing the adverse effects of noise on model behavior. Our study identifies one such undesirable effect—**multimodality**—arising from corrupted semantics. We aim to mitigate the impact of multimodality without eliminating the beneficial aspects of noise, which is the motivation behind our focus on corrupted semantics in this work.\n\n---\n\n**Q5: Why is the index of the last state chosen for the final loss? What if the last state is never reached?**\n\n**A:** Thank you for the question. The reason we choose the index of the last state (i.e., `[EOS]`) for the final loss is that we enforce a structural constraint: every valid decoding path in the DAG must terminate at the last state. This design eliminates the need to enumerate all possible ending states when summing the probabilities of valid paths, thereby reducing computational complexity.\n\nMoreover, the DAG in ExLM is constructed in a way that guarantees the reachability of the last state from all other nodes. Thus, the last state is always reachable, and the validity of the loss computation is ensured. We will include these details in the camera-ready version.\n\n---\n\n**Q6: How is ExLM modeled without the transition matrix? Is it like CTC?**\n\n**A:** Yes, your understanding is exactly correct. When the transition matrix is not used, ExLM is trained in a manner similar to CTC (Connectionist Temporal Classification), where the model considers all possible alignments between the expanded states and the target tokens. More implementation details on this setup will be added to the camera-ready version of the paper.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Not rigorously checked.", "experimental_designs_or_analyses": "Yes. Experimental designs are sound.", "relation_to_broader_scientific_literature": "The paper is of broad interest in understanding and building new language models.", "essential_references_not_discussed": "@inproceedings{namazifar2021warped,\n  title={Warped language models for noise robust language understanding},\n  author={Namazifar, Mahdi and Tur, Gokhan and Hakkani-T{\\\"u}r, Dilek},\n  booktitle={2021 IEEE spoken language technology workshop (SLT)},\n  pages={981--988},\n  year={2021},\n  organization={IEEE}\n}\n\nThe above citation proposed different noising techniques for ASR error robustness. It is somewhat related to this work.", "other_strengths_and_weaknesses": "Strength:\n\nThe ExLM formulation is very interesting and exciting to me. The semantic richness acquired by the model as a result of this formulation can be very useful.\n\nWeakness:\n\nThe paper can benefit from a more detailed explanation of the alignment algorithm. I feel like the explanation should be in the main text rather than in the appendix.\n\nAlso, the flow of ideas from section 3 to section 4 can be done in a much better way.", "other_comments_or_suggestions": "1. It is unclear how the discussion on the impact of [MASK] (section 3) is essential for ExLM. The findings in section 3 seem fairly obvious. The fact that semantic corruption is the key factor for the learning process is hardly surprising (figure 4). \n\n2. The relation between section 3 and section 4 is not clear. \n\n3. The authors need to explain the alignment process, i.e. the forward-backward algorithm is the main text.", "questions_for_authors": "1. \"Corrupted semantics\" is an obviously introduced noise in that masked language modeling (MLM) is a kind of denoising process. The authors claim that it can \"...negatively affect the model's ability to learn accurate semantic representation...\". It is not clear why this claim should be non-obvious. Having a large semantic corruption (large p^k) will obviously lead to performance degradation. Therefore, it's value is chosen carefully. But how is it a problem?\n\n2. In appendix H, why is the index of the last state chosen for the final loss? What if the last state is never reached? Is there a null transition? This needs more details.\n\n3. In the ablation, how is ExLM modeled without the transition matrix? Is it like CTC? Needs more details.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lDX65OtqzS", "reviewer": "Reviewer_3GXq", "summary": "This paper studies the semantic corruption issue in masked language modeling (MLM). To start the research, the authors design an experiment (repeated MLM) to show the relationship and significance of the corrupted semantics caused by masking. As a result, ExLM is proposed as a solution for the problem. In this LM, a couple of masked tokens are distributed to each masked position. The hidden states are then aligned using a transition matrix, then computing the final losses.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to Reviewer 3GXq for the suggestions. In the following, we address all your concerns regarding the evaluation of NLU tasks, additional experiments on more tasks, the efficiency analysis of ExLM, and the details of repeated MLM experiments. These discussions will also be incorporated into the final camera-ready version of the paper. We hope our responses help clarify the paper, and we welcome any further comments or suggestions!\n\n---\n\n**Q1: These dev sets are potentially biased to the official test sets, and the results are unstable.**\n\n**A:** Thank you for your insightful comment. We have evaluated the stability of the ExLM model on the GLUE benchmark by reporting the standard deviation of performance across multiple runs. The results, summarized in the table below, demonstrate that ExLM exhibits good stability overall:\n\n**Standard Deviation on the GLUE Benchmark:**\n\n| Task   | ExLM Performance |\n|--------|------------------|\n| MNLI-m | 86.9 ± 0.14      |\n| QQP    | 92.0 ± 0.09      |\n| QNLI   | 93.1 ± 0.43      |\n| SST-2  | 93.9 ± 0.50      |\n| CoLA   | 64.6 ± 0.81      |\n| RTE    | 78.8 ± 1.80      |\n| MRPC   | 89.6 ± 1.00      |\n| STS-B  | 90.5 ± 0.72      |\n| MEAN   | 86.2             |\n\nFurthermore, to further verify the effectiveness of ExLM, we also evaluated its performance on the SuperGLUE benchmark [Wang et al., 2019]. As shown in the table below, ExLM significantly outperforms several strong baseline models, further supporting the validity of our proposed approach:\n\n| Model                          | BoolQ (Acc.) | CB (Acc.) | COPA (Acc.) | MultiRC (F1) |\n|-------------------------------|--------------|-----------|-------------|---------------|\n| BERT [Devlin et al., 2019]    | 74.4         | 83.9      | 63.0        | 68.1          |\n| Token Drop [Hou et al., 2022] | 73.0         | 83.9      | 64.0        | 67.7          |\n| SCTD [Zhong et al., 2023]     | 73.8         | 87.5      | 68.0        | 68.9          |\n| ExLM                      | **76.7**     | **88.0**  | **69.1**    | **71.3**      |\n\n---\n\n**Q2: At least one more dataset result (e.g. QQP, RTE) is needed to fully justify the claims.**\n\n**A:** Thank you for the suggestion. In fact, the repeated MLM results on QQP and RTE have already been included in Appendix F. As stated in Section 3.2 of the main text: *“We also provide the results of this experiment on more tasks in Appendix F.”*\n\n---\n\n**Q3: The computation efficiency of ExLM compared to MLM and other MLM variants, e.g., time cost, memory cost.**\n\n**A:** Thank you for your comment. As mentioned in Section 5.3 of the main text, *“Efficiency and entropy analysis of EXLM are provided in Appendix Q”*. Specifically, we present a detailed discussion of ExLM's training time cost in Appendix Q.2, including a comparison with MLM models. Moreover, memory cost analysis is provided in Appendix I.4, which is also referenced in Section 4.3 of the main text: *“The efficiency analysis is provided in Appendix I.”*\n\n---\n\n**Q4: The details of repeated MLM experiments are not clear. I am not sure whether they pre-train the MLM (e.g., on Wiki) and then fine-tune on MNLI, or just fine-tune on MNLI.**\n\n**A:** Thank you for your comment. In each group of repeated MLM experiments, we pre-trained separate MLM models using different values of *k* and *p*, and then fine-tuned and evaluated them on downstream tasks using the same settings. As stated in Appendix B.1 (Pre-training Configuration): *“In the Repeated MLM experiment, we train a series of MLMs with different p and k parameters.”*\n\n---\n\n**References:**\n\n[Devlin et al., 2019] Devlin, Jacob, et al. \"Bert: Pre-training of deep bidirectional transformers for language understanding.\" NAACL 2019.\n\n[Hou et al., 2022] Hou, Le, et al. \"Token Dropping for Efficient BERT Pretraining.\" ACL 2022.\n\n[Zhong et al., 2023] Zhong, Qihuang, et al. \"Revisiting Token Dropping Strategy in Efficient BERT Pretraining.\" ACL 2023.\n\n[Wang et al., 2019] Wang, Alex, et al. \"SuperGLUE: A stickier benchmark for general-purpose language understanding systems.\" NeurIPS 2019.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "1. The first part of the experiment show great performance gain on learning the language representation.\n\n2. However, the second part of the experiments are evaluated on a selection of GLUE and SQuAD2 dev sets. However, to my experiences, these dev sets are potentially biased to the official test sets, and the results are unstable. Considering that the performance gains showed in Table 3 are not very pronounced (e.g. 64.3 to 64.6 on CoLA), **I need to point out this potential risk that this part of the evaluation can be problematic.**\n\n3. The repeated MLM results are reported on MNLI, a frequently used dataset for NLU. Even though the results are not pronounced, I trust the results. The problem is I think only experimenting MNLI is not sufficient for the claim. **At least one more dataset results (e.g. QQP, RTE) are needed to fully justify the claims** (e.g. corrupted semantics and unreal tokens which matter).", "theoretical_claims": "I did not check the proof in Appendix.", "experimental_designs_or_analyses": "The experiments are run across a number of random seeds. They are fair.", "relation_to_broader_scientific_literature": "The method proposed in the paper can be useful for training MLMs.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weakness:\n\n1. The paper states that the ExLM method is efficiently in modeling semantics. I agree with that but I do care about **the computation efficiency of ExLM compared to MLM and other MLM variants, e.g. time cost, memory cost. This point is very important for the usage of the method, and it seems not discussed in the paper.** If they are already provided somewhere, please remind me.\n\n2. The details of repeated MLM experiments are not clear for me. The authors mention that they train a number of MLMs with different repetition times k and mask ratios p. The results in Figure 3-5 are on the MNLI dataset. I am not sure whether they pre-training the MLM (e.g. on Wiki) and then fine-tune on MNLI, or just fine-tune on MNLI. In my opinion, these two approaches can cause very different results in the analysis.", "other_comments_or_suggestions": "Please see above.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IwPO2Ws60L", "reviewer": "Reviewer_j1MR", "summary": "This paper investigates the role of mask tokens in Masked Language Models (MLMs). The authors first provide an empirical examination of the effect of the mask token through two perspectives: corrupted tokens and unreal tokens. Additionally, the authors propose a new algorithm, EXLM, to further enhance performance. The experimental evaluation demonstrates the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful suggestions from Reviewer j1MR. In the following sections, we will address all your concerns. These discussions will also be incorporated into the final camera-ready version of the paper. Any further comments are welcome!\n  \n---\n\n**Q1: What is the authors' main contribution in analyzing unreal tokens?**\n\nThank you for the question. The repeated MLM experiments are primarily designed to analyze the key factors that influence the performance of MLM models and the underlying reasons. Toward this goal, this section presents the following two main contributions:\n1. We demonstrate that the issue of *unreal tokens*, which has received significant attention in previous works, is not the primary factor affecting MLM performance.\n2. We show that *corrupted semantics* and the resulting *multimodality* phenomenon are more critical in influencing the performance of MLMs.\n\nIn summary, this analytical section offers a new perspective for understanding MLMs by highlighting the previously overlooked importance of corrupted semantics. It also suggests that future developments of MLMs should focus more on addressing the problem of multimodality.\n\n---\n\n**Q2: The connection between Section 3 and Section 4 is weak.**\n\nThank you for the suggestion. We have revised Sections 3 and 4 to better highlight their connection. These changes will be included in the final camera-ready version. Specifically:\n\n1. The last paragraph of Section 3.3 has been updated to emphasize *multimodality* as a key factor affecting MLM performance. An effective model should thus be able to *mitigate the impact of semantic multimodality*.\n2. Section 4.1 has been reorganized to align with Section 3’s analysis. To address multimodality, ExLM introduces two mechanisms:\n   - **Intra-token Multimodality**: Each missing token may have a diverse set of plausible candidates. Thus, we propose *States Expansion* to build a larger semantic space, enabling richer token predictions.\n   - **Inter-token Multimodality**: The meaning of a token is intricately dependent on the semantics of its surrounding tokens. Thus, we introduce a *Dependency Capture* mechanism using a transition matrix to model semantic dependencies across tokens.\n\n---\n\n**Q3: How does ExLM model multiple possible semantics?**\n\nWe introduce a *state expansion* mechanism in ExLM, where each [MASK] token is associated with multiple expanded states, each representing a different possible semantic choice for that token.\n\n---\n\n**Q4: The baseline approaches seem too weak as they were proposed a few years ago.**\n\nThank you for the valuable suggestion. We have added comparisons with a more recent baseline model, **3ML_{self}** [Liao et al., 2022], in Table 3. The results show that our ExLM model outperforms this stronger baseline, as detailed below:\n\n|  | MNLI-(m/mm) | QQP | QNLI | SST-2 | CoLA | RTE | MRPC | STS-B | MEAN |\n|-|-|-|--|-|-|-|-|-|-|\n| TUPE [Ke et al., 2021] | 86.2/86.2 | 91.3 | 92.2 | 93.3 | 63.6 | 73.6 | **89.9** | 89.2 | 84.9 |\n| 3ML_{self} [Liao et al., 2022] | 84.8/84.9 | 91.1 | 91.4 | 92.9 | 61.4 | **81.2** | 89.2 | 90.1 | 85.2 |\n| ExLM | **86.9/86.7** | **92.0** | **93.1** | **93.9** | **64.6** | 78.8 | 89.6 | **90.5** | **86.2** |\n\nIn addition, we further evaluated the ExLM model on the **SuperGLUE benchmark** [Wang et al., 2019]. The following table shows that ExLM also achieves significant improvements over several strong baselines, further confirming the effectiveness of our approach:\n\n| Model | BoolQ (Acc.) | CB (Acc.) | COPA (Acc.) | MultiRC (F1) |\n|-|-|-|-|-|\n| BERT | 74.4 | 83.9 | 63.0 | 68.1 |\n| Token Drop [Hou et al., 2022] | 73.0 | 83.9 | 64.0 | 67.7 |\n| SCTD [Zhong et al., 2023] | 73.8 | 87.5 | 68.0 | 68.9 |\n| ExLM | **76.7** | **88.0** | **69.1** | **71.3** |\n\n---\n\n**Q5: The proposed method, ExLM, incorporates 2D RoPE, which further enhances model performance, making it difficult to assess the true effectiveness of the approach.**\n\nThank you for pointing this out. In our experiments, all compared MLM baselines, including Vanilla MLM and Vanilla MLM++, adopt the same backbone architecture as ExLM, which uses RoPE as the default positional encoding. Therefore, the performance comparisons are made on a fair basis and can indeed assess the true effectiveness of the proposed method.\n\n**References:**\n\n[Hou et al., 2022] Hou, Le, et al. \"Token Dropping for Efficient BERT Pretraining.\" ACL 2022.  \n[Zhong et al., 2023] Zhong, Qihuang, et al. \"Revisiting Token Dropping Strategy in Efficient BERT Pretraining.\" ACL 2023.  \n[Wang et al., 2019] Wang, Alex, et al. \"SuperGLUE: A stickier benchmark for general-purpose language understanding systems.\" NeurIPS 2019.  \n[Liao et al., 2022] Liao, Baohao, et al. \"Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token.\" EMNLP 2022.  \n[Ke et al., 2021] Ke, Guolin, Di He, and Tie-Yan Liu. \"Rethinking Positional Encoding in Language Pre-training.\" ICLR 2021.", "claims_and_evidence": "1. The authors claim that both unreal tokens and corrupted tokens arising from the mask token can affect performance. While the impact of corrupted semantics has been studied in previous literature (Line 65), the effect of unreal tokens does not appear to be significant for model performance (Fig. 3). This, in turn, underscores the importance of prior work. Given this, what is the authors' main contribution in analyzing unreal tokens? Can the author elaborate this? \n\n2. The connection between Section 3 and Section 4 is weak. The proposed method EXLM seems not motivated by the analysis given in Section 3. What is the motivation behind the proposed method, EXLM? At the end of Section 3, the authors appear to focus on studying the optimal mask ratio, but this aspect is not discussed in Section 4.", "methods_and_evaluation_criteria": "1. The baseline approaches seem too weak as they were proposed a few years ago.\n\n2. The proposed method, EXLM, incorporates 2D RoPE, which further enhances model performance, as shown in Table 2. Therefore, the performance gain is not solely attributable to EXLM, making it difficult to assess the true effectiveness of the approach", "theoretical_claims": "NA", "experimental_designs_or_analyses": "No Question", "relation_to_broader_scientific_literature": "Please see *Claims And Evidence*", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. As indicated in Section 3.3 (Core Impact of Corrupted Semantics: Multimodality), Line 212, the corrupted context may correspond to multiple possible semantics. Can authors explain how EXLM addresses this issue?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper tackles a key limitation in masked language modeling (MLM): the semantic distortion introduced by [MASK] tokens during pretraining. The authors identify two sources of degradation—unreal tokens and corrupted semantics—and demonstrates via a novel “Repeated MLM” experiment that corrupted semantics is the more critical bottleneck.

To address this, they propose EXLM, a masked language model variant that:
- Expands each [MASK] into multiple hidden states (States Expansion),
- Models semantic dependencies between them via a transition matrix (Dependency Capture),
- Uses a dynamic programming-based state alignment during training.

The model shows consistent gains across textual and molecular domains (SMILES), outperforming multiple baselines across several benchmarks.

Strength
- Novelty: Strong analytical framing of [MASK] token effects, especially the disentangling of corrupted semantics from unreal tokens.
- Applicability: Authors show value of this approach across multiple domains, enhancing generalizability.
- Experiments: Systematic evaluations with ablation studies, entropy analysis, and case studies.

Weakness
- Scalability: While the expansion of [MASK] tokens enriches semantics, it increases compute costs (multiple hidden states per mask), especially for large-scale models.
- Transferability: The method is evaluated on standard benchmarks, but implications for multilingual or low-resource settings are unexplored.

Considering the feedback from the reviewers, the discussions, and the AC assessment, this paper is recommended as an Accept.
Decision: Accept (poster)