Should We Still Pretrain Encoders with Masked Language Modeling?
Authors: Hippolyte Gisserot-Boukhlef
Nicolas Boizard
Manuel Faysse
Duarte Miguel Alves
Emmanuel Malherbe
Andre Martins
CELINE HUDELOT
Pierre Colombo
Venue: ICLR 2026 Poster (2026)
Keywords: Encoder Pretraining
Masked Language Modeling
Causal Language Modeling
Text Representations
Representation Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Learning high-quality text representations is fundamental to a wide range of NLP tasks. While encoder pretraining has traditionally relied on Masked Language Modeling (MLM), recent evidence suggests that decoder models pretrained with Causal Language Modeling (CLM) can be effectively repurposed as encoders, often surpassing traditional encoders on text representation benchmarks. However, it remains unclear whether these gains reflect an inherent advantage of the CLM approach or arise from confounding factors such as model and data scale. In this paper, we address this question through a series of large-scale, carefully controlled pretraining ablations, training a total of 38 models ranging from 210 million to 1 billion parameters, and conducting over 15,000 fine-tuning and evaluation runs. We find that while training with MLM generally yields better performance across text representation tasks, CLM-trained models are more data-efficient and demonstrate improved fine-tuning stability. Building on these findings, we experimentally show that a biphasic training strategy that sequentially applies CLM and then MLM, achieves optimal performance under a fixed computational training budget. Moreover, we demonstrate that this strategy becomes more appealing  when initializing from readily available pretrained CLM models, reducing the computational burden needed to train best-in-class encoder models. We release all project artifacts at \url{https://hf.co/MLMvsCLM} to foster further research.

Published as a conference paper at ICLR 2026
SHOULD WE STILL PRETRAIN ENCODERS WITH MASKED LANGUAGE MODELING?
Hippolyte Gisserot-Boukhlef-
Nicolas Boizard 2,5m 2.5+
Manuel Faysse
Duarte M. A,, 6.7
Emmanuel Malherbe
André F. T. Martins 3,6,7
Céline Hudelot
Pierre Colombo
*Diabolocom STransPerfect 5 MMCS CentraleSupélec, Université Paris Paris-Saclay Instituto de Telecomunicações
Artefact Research Center
*Cohere
Instituto Superior Técnico Universidade de Lisboa (Lisbon ELLIS Unit)
hi pppp yte. gi aserot-boukhh 1e e @ rant esupelec.
ABSTRACT
Learning high-quality text representations is fundamental to wide range of NLP tasks. While encoder pretraining has traditionally relied on Masked Language Modeling (MLM), recent evidence suggests that decoder models pretrained with Causal Language Modeling (CLM) can be effectively repurposed as encoders, of- ten surpassing traditional encoders on representation benchmarks. However, it remains unclear whether these gains reflect an inherent advantage of the CLM approach or arise from confounding factors such as model and data scale. In this paper, we address this question through series of large-scale, carefully controlled pretraining ablations, training a total of 38 models ranging from 210 million to I billion parameters, and conducting over 15,000 fine-tuning and evaluation runs. We find that while training with MLM generally yields better performance across text representation tasks, CLM-trainee models are more data-efficient and demon- strate improved fine-tuning stability. Building on these findings, we experimen- tally show that biphasic training strategy that sequentially applies CLM and then MLM,, achieves optimal performance under a fixed computational training bud- get. Moreover, demonstrate that this strategy becomes more appealing when initializing from readily available pretrained CLM models, reducing the computa- tional burden needed to train best-in-class encoder models. We release all project artifacts at https: //hf. CO/ MMMVs to foster further research.
INTRODUCTION
Learning meaningful representations is essential for wide range of natural language processing (NLP) tasks, such as sequence classification, named entity recognition, extractive question answer- ing, and information retrieval. Traditionally, these representations are learned with encoders pre- trained solely with Masked Language Modeling (MLM) (Devlin et al.) (019). More recently, how- ever, decoder models pretrained with Causal Language Modeling (CLM) and later adapted with MLM have challenged the MLM-only paradigm (BehnamGhader et al. 2024), achieving state-of- the-art results the Massive Text Embedding Benchmark (MTEB) (Muennighoff
al.) [023] Enevoldsen et al. 2025). Yet. these results have so far only been observed on models that are signif- icantly larger than typical encoders and trained on substantially more data (Lee al. 2024 Meng jt al. 2024 Kim et al. 2024) Muennighoff et al., (024)). As a result, it remains unclear whether the success of this training paradigm stems from CLM, merely from increased scale.
In this paper, we present a controlled study examining the impact of training with MLM and CLM on learning textual representations. We compare models with equal architecture and size, trained
Equal contribution
htttes::/aa..co//MMIsscCCC
CLM

---

Published as a conference paper at ICLR 2026
PraTniig From Scratch (PFS)
Inttlatzationn
Training
Bette nec
CLM
Accuracy
Random init.
OR/AND
Task-specific FT
86
MLM
100
CLM Training (%)
Continued PreTraining (CPT)
Inttaaization
Training scope
GPT
model
Accuracy
68
OPT
OR
MLM
Task-specific FT
86
MLM model
84
MLM
CLM
Figure 1: Experimental setup overview and key results on sequence classification (610M model size, 40% MLM ratio). The upper plot shows SC downstream performance function of the CLM-to- MLM step ratio during pretraining. The lower plot illustrates the effect of applying MLM CPT to models initially trained with either MLM CLM (Base).
on equal amount of data, and evaluate them extensive range of text representation tasks. We start by comparing pretraining exclusively with either approach, and then investigate a two-stage pretraining alternative, where CLM and MLM are applied sequentially. Additionally, leveraging existing pretrained models, we investigate a continued pretraining setup, where training starts from models initially trained with either MLM CLM and is then extended with MLM for portion of the pretraining steps. In total, we train 38 models ranging from 210 million to billion param- and conduct over 15,000 fine-tuning and evaluation runs ensure fair design comparisons, amounting to 10k MI250X GPU hours. These large-scale experiments show that:
Although CLM training delivers strong performance on certain tasks and demonstrates good data efficiency and fine-tuning stability, MLM remains essential for achieving robust performance across all tasks, underscoring that bidirectional training remains essential (eccion 3).
When pretraining model from scratch, a two-stage protocol that first applies CLM followed by MLM offers data-efficient way to build strong text representations, effectively combining the strengths of both paradigms (section 4).
In continued pretraining setting, adapting CLMMpretrained model with MLL proves more effective than continuing MLM training from an MLM-pretrained model. This result suggests that leveraging widely pretrained decoder models is currently the best approach obtain a strong encoder model (section 5).
To facilitate reproducibility and support future research, we release all pretrained model check- points,' together with the corresponding training² and evaluation codebases."
2 EXPERIMENTAL SETUP
In this section, we describe our controlled experimental setup, covering model architectures, pre- training and fine-tuning protocols, as summarized Figure
Models. The model architectures closely follow those of the EuroBERT models (Boizard et al, 2025). with sizes of 210M, 610M, and IB parameters. All models use maximum context length of 2,048 tokens and a RoPE value of 10,000
Pretraining data. Models are trained on unique English tokens from the FineWeb-Edu dataset (Penedo et al. 2024), which is known for supporting efficient model training. To ensure fair com-
"Adiitional details the model architectures be found in Appendix
2

---

Published as a conference paper at ICLR 2026
parison across model configurations and training setups, all models are exposed to the same sequence of samples during training.
Pretraining paradigms. Models are trained using one of 3 approaches:
1. CLM employs next-token prediction, where each token is generated autoregressively using causal attention mask. Given an input sequence x (#1 (11,,22 22..... (T), the training objective minimize the negative log-lik se-likelihood::
T Σ
LCLM
logpo po_, (xt I1,..., Tt-1)
where po_. (-) denotes the model's predicted distribution over the vocabulary conditioned on the preceding tokens under the causal attention mask.
2. MLM randomly masks subset of tokens and trains the model to reconstruct them using bidirectional attention mask. The sequence-level training objective is given by:
LxIM(X) = Σ
log log (ara xM),
SEM
where po_. (-]-) denotes the model's predicted distribution under bidirectional attention mask, M C [T.....T} specifies the indices of the masked tokens, and xMM is the imput sequence with the corresponding tokens replaced by special placeholders In practice, tokens are masked uniformly and independently, each with probability Pmask In our experiments, we consider Pmask E {20%, 30%, 40%, 50%).
3. Finally, CLM+MLM sequentially combines the former approaches a two-stage setup, where CLM pretraining is performed first, followed by MLM.
Pretraining hy perparameterss Pretraining is performed with a per-device batch size of 12 sam- ples across 192 GPUs, yielding an effective batch size of 2,373, tokens 5) We employ a Warmup- Stable-Decar (WSD) learning rate schedule: a 2,000-step warmup phase, followed by 38,000 steps with a constant learning rate of 10 10-4 of 42,000 training steps.
2 ending with a 2,000-step linear decay phase, for a total
Pretraining setups. We consider two pretraining setups reflecting common real-world scenarios: training a model from scratch and continuing training from existing model.
1. Pretraining From Scratch (PFS). Models are trained from random initialization for a fixed number of steps using one of approaches: CLM, MLM, or sequential CLM+MLM. For CLM and MLM, standard WSD learning rate scheduler is applied. For CLM+MLM models, training first performed with CLM, and then resumed with MLL from CLM checkpoints that have not undergone learning rate decay.
Continued PreTraining (CPT). Training is initialized from models pretrained using either CLM or MLM and is then continued with MLM. In contrast to the PFS setup, the pretrained models used for CPT have already undergone learning rate decay during their initial training phase, reflecting real-world constraints and continued training practices. Another key difference is that CPT starts from checkpoints where the loss has already converged, whereas in PFS, the paradigm switch typically occurs while gradient norms are still large and learning is active.
Fine-tuning tasks and datasets. We evaluate all models across a broad range of text representa- tion tasks, focusing on 4 key categories commonly used in real-world applications. For Sequence Classification (SC), we use SST-2 (Socher et al., 2013). MNLI (Williams et al.) 2018). and QQP Wang et al. 2017). For Token Classification (TC), we evaluate on the English subsets of CoNLL (Tong Kim Sang & De Meulder 2003), OntoNotes (Hovy et al. (006). and UNER (Mayhew et al.)
Foor example, if T 5 and M (1,3), then ([mask], J2, [mask], Tt).
6 'Inputs consist variahle-length sequences ranging from 12 tokens up to maximum 2,048.
"The optimal learning rate was selected from the range 1 10 to 2 10 based on experiments using 10% of the training data.
3

---

Published as a conference paper at ICLR 2026
2024
Question Answering (QA) is assessed using SQuAD Raajpurkar et al. 2016), SQuAD-v2
Rajpurkar et al. 2018). and ReCoRD Wang et al. 2019). For Information Retrieval (IR), we use MS MARCO (Bajaj et al. 2016), NQ (Kwiatkowski et al. 2019). and the English subset of MLDR
Chen et al.
2024 for long- long-context evaluation
Fine-tuning protocol. To ensure fair comparison, all models are fine-tuned using a consistent protocol. Each model is trained for up to 1. 1,000 steps or one full epoch, whichever comes first, using a batch size of 32. To account for differences in model architecture and task requirements, we perform a grid search over 6 learning rates (1 x 10-5, 2 x 10-5, -5 5 x 10-5, 1 x 10-4 2 x 10-4, and 5 x 10 10-") for each model-dataset pair, with 10% warmup followed by linear decay. The learning rate yielding the best validation performance is selected. Fine-tuning employs bidirectional attention mask with task-specific loss functions: for SC, we use cross-entropy on mean meean-pooled token embeddings; TC and QA are trained using token-level cross-entropy; and for IR, we rely on the InfoNCE loss (Oord) jet al. (018)) with in-batch negatives, using mean pooling. To account for the fine-tuning instability commonly observed in BERT-style models for representation learning (Devlin et al. [019]] Lee et al.)
2020 Dodge et al. 2020 Zhang et al. [2220, the entire procedure is repeated across random seeds.
Fine-tuning is conducted on the in-domain training set, except for NQ and MLDR, for which training is performed on MS MARCO.
Evaluation metrics. SC is assessed with accuracy, TC and QA with FI score, and IR with NDCG@10 10. We report results averaged across seeds, along with 95% confidence intervals.
Infrastructure. All pretraining and fine-tuning experiments are carried out MI250X GPUs provided by the Adastra supercomputer, using the AMD-optimized EuroBERT codebase (Boizard) et al. 2025). Pretraining runs executed on 24 nodes with 8 GPUs each (192 GPUs total), while fine-tuning is carried out on single GPUs.
Experimental scale. Drawing reliable conclusions about model design choices typically requires scaling up and repeating experiments, as statistically sound results often require a large number of runs and sufficient training to reach minimal convergence (Hoffmann et al. 2022). We therefore carefully design our experiments to ensure robust and well-supported findings. The default pre- training setup uses 100B tokens, which corresponds to 5 times the optimal data budget proposed by Hoffmann et al. (2022) for decoder models in the IB-parameter range. The total compute budget amounts 110k GPU hours, broken down follows:
Base pretraining includes all 3 model sizes (210M, 610M, and IB), trained with both CLM and MLM. For MLM, 4 masking ratios (20%, 30,, 40%, and 50%) are explored, resulting in a total of 15 models, each trained for 42,000 steps (100B tokens), and accounting for 81k GPU hours.
PFS experiments explore various CLM-to- step ratios (0%-100% 25%-75% 50%-50% 75%-25% and 100%-0%) across 3 training lengths (12,000, 22,000, and 42,000 steps), focusing on the 610M model size with 40% masking ratio for MLM. This setup is also used to evaluate the effect of masking ratio on models pretrained with CLM followed by MLM. To minimize redundant computation. we reuse checkpoints from the base pretraining runs whenever possible, applying learning rate decay only during the final 2,000 steps. In total, these runs result in 17 new models and account for 120,000 additional training steps and 16k GPU hours.
CPT experiments are performed 610M models with a 40% masking ratio and varying train- ing lengths (2,000, 12,000, and 22,000 steps) applied to both CLM- and MLM MLM--rrtraiied mod- els. To avoid redundant computation, training resumed from shared checkpoints whenever possible. These runs yield 6 new models, totaling 26,000 training steps and 4k GPU hours.
Model evaluation involves fine-tuning 50 model checkpoints in total, including 38 final pre- trained checkpoints and 12 intermediate ones. Each model is fine-tuned on 12 datasets, across 6 learning rates and 5 random seeds, resulting in 15,120 fine-tuning runs which total about 9k GPU hours. Allocating substantial compute the evaluation phase is particularly important in representation learning, where fine-tuning can exhibit high variance and instability (Devlin et al. 2019
Lee et al. 2020 Dodge al. 2020 Zhang et al. 2020).
Further details are provided in Appendix
4

---

Published as a conference paper at ICLR 2026
SC
TC
QA
IR
88
80
Acourasy courracy
NDC6@11
92.5
75
92.0
210M
610M
210M
610M
1B
210M
610M
210M
610M
1B
Figure 2: Downstream performance of MLM and CLM pretraining, averaged across tasks and re- ported for all model sizes. For MLM, results are given for 40% masking ratio.
SC
QA
82.5
93.0
Accuracy
NDCG@11
80.0
92.5
77.5
50
75.0
20
30
40
50
20
30
40
50
20
30
40
50
20
30
40
50
Masking (%)
Masking (%)
Masking (%)
Masking (%)
210M
610M
1B
Figure 3: MLM downstream performance across different masking ratios for all model sizes.
PRETRAINING WITH CLM OR MLM
This section presents preliminary results comparing MLM and CLM pretraining in terms of down- stream performance, data efficiency, and training and fine-tuning stability.
MLM generally outperforms CLM text representation tasks. Bidirectional attention during pretraining tends to enhance representation quality on downstream tasks (Figure 2). The perfor- mance gap between models trained with MLM and those trained with CLM especially noticeable on SC and QA tasks, and remains consistent across model sizes. The largest discrepancy is observed on QA, which appears particularly sensitive the absence of bidirectional attention during pretrain- ing. Interestingly, some lask-specific trends emerge: for instance, the MLM-to-CLM, gap widens with increasing model size on SC, but narrows on IR.
There is universally optimal masking ratio. The masking ratio is key design choice when pretraining models with MLM. As shown in Figure 3] there is no single best ratio, as it depends both model size and downstream task, making MI.M pretraining delicate balance. Larger models tend to benefit from higher masking ratios, consistent with prior findings from Wettig et al.) (2023). Across tasks, IR datasets consistently prefer higher masking ratios regardless of model size. In contrast, for token-level tasks such as TC and QA, smaller models perform better with lower masking ratios. For larger models (610M and IB). the performance curves exhibit U-shape, indicating improved performance at both low and high masking ratios 1 To reduce computational overhead, subsequent experiments report results on 6 610M models trained with 40% masking ratio for MIM,, which provides strong overall compromise across tasks.
CLM models can perform competitively. Interestingly, although CLM-pretrained models typi- cally underperform on SC and QA tasks, they achieve solid results IR, with the gap to MLM
Additional results in Appendix C further highlight the influence of masking ratio, showing substantial variation even among datasets within the same task category.
IIn some configurations, we observe smaller models outperforming larger ones average (e.g., the 210M model surpassing the 610M model on TC). Such hchavior can arise from inherent pretraining variance Grattaton al. 2024 Yang al. 2025 and, in this case, results in statistically significant differences.

---

Published as a conference paper at ICLR 2026
SC
TC
QA
IR
60
Accuracy
80
90
NDCG@:1
80
L
40
85
20
10
10
20
30
40
40
10
20
30
40
Training Step (K)
Training Step (K)
Training Step (K)
Training Step (K)
MLM
CLM
Figure 4: Downstream performance with respect pretraining steps for CLM and MLM. Results are reported for 610M models, with a 40% masking ratio for ML.M.
SC
TC
QA
20
Centered Score
20
25
I
1
1
0
-20
-25
-10
-20
MLM
CLM
MLM
CLM
MLM
CLM
MLM
CLM
Figure 5: Impact the fine-tuning learning rate on MLM- vs. CLM-pretrained models. Error bars indicate the standard deviation of metric scores across all seeds and learning rates between 10-5 and 10
Results are shown for 6 610M-parameter models, with a 40% masking ratio for MLM.
models narrowing as model size increases. On TC, they consistently match MLM models and even outperform them by a substantial margin the 610M scale Figure 2 This suggests that causal pretraining produce strong token-level representations, highlighting its potential despite the ab- sence of bidirectional context.
CLM is data-efficient than MLM in the early stages of training. As shown in Figure 4 CLM consistently outperforms MIM in downstream performance during the early stages of training (up to step 10,000 for SC and QA, 20,000 for IR, and training progresses, MLM models tend to catch and often surpass CLM, while CLM shows more limited gains in later steps. This suggests that although CLM saturates earlier, it enables more efficient representation learning in fewer steps. Notably, this makes CLM an appealing option for data-scarce scenarios, such as pretraining on low-resource languages, or simply as a warmup stage before ML--based encoder training.
until the end for TC). However, as
CLM-based pretraining improves fine-tuning stability. A key challenge in fine-tuning mod- els for text representation tasks is selecting optimal hyperparameters, particularly the learning rate, which can be sensitive to factors like model size, task type, or dataset scale, making exhaustive grid searches computationally expensive. As shown in Figure 5] models pretrained with CLM demon- strate lower sensitivity to learning rate choices than those pretrained with MIM. This indicates that CLM pretraining provides more stable initialization for fine-tuning, facilitating more reliable per- formance and reducing the need for extensive hyperparameter tuning.
4 TWO-STAGE CLM+MLM PRETRAINING
Building on Section 3, which highlights clear advantages of CLM over MLM in terms of token- level representation quality, data efficiency, and training stability, we investigate the benefits of applying CLM and MLM sequentially during pretraining. We focus on the PFS setting, where
Intuitively, this behavior may stem from the nature of token-level tasks, which rely less on bidirectionally informed context to identify named entities word types than SC, IR, or QA, which depend more heavily on broader contextual understanding. This hypothesis could be validated through further research.

---

Published as a conference paper at ICLR 2026
SC
TC
QA
IR
93.0
Accuracy
86
60
NDCG@10
50
92.0
91.5
CLM Training (%)
CLM Training (%)
CLM Training (%)
CLM Training (%)
12K
22K
42K
Figure 6: Impact of two-stage CLM+MIMM pretraining downstream performance under different training budgets (12,000, 22,000, and 42,00 steps). The x-axis shows the percentage of CLM steps allocated in the first phase. Experiments are conducted 610M models, with 40% masking ratio during MLM training.
SC
TC
QAA
IR
Centered Score
0.5
2.5
2
I
0
0.0
0.0
-0.5
-2.5
MLM
CLM+MUM
MLM
CLM+MLM
CLM-MLM
CLM+MLM
Figure 7: Comparison of downstream performance variability across different masking ratios (20%, 30%, 40%, 50%) for CLM and CLM+MLM pretraining configurations. Error bars indicate the stan- dard deviation across fine-tuning seeds and masking ratios. Results are reported for 610M models, using 42,000 training steps for MLM and schedule of 40,000 CLM steps followed by 2,000 MLM steps for CLM+MLM.
single learning rate scheduler is used, and evaluate this approach under fixed compute budgets of 12,000, 22,000, and 42,000 training steps, using different splits: 100% CLM, 25%-75%, 50%-50% 75%-25% and 100% MLM. To keep computation costs manageable, focus the 610M model scale and use a fixed masking ratio 40%.
Under fixed compute constraints, starting pretraining with CLM and continuing with MLM yields better results than pure MLM. As shown Figure 6) combining CLM and MLM con- sistently improves downstream performance compared to MLM-only training. In particular, the 25%-75% split reliably surpasses the MLM baseline, while even a 75% CLM allocation maintains comparable results. These findings confirm the benefits of CLM pretraining for learning strong text representations and showcase the synergies it provides when combined sequentially with MLM training.
CLM-based models exhibit lower sensitivity to masking ratio. As shown in Figure 7 adapted CLM models show less variation in performance across different masking ratios compared to fully MLM-trained models. Initial CLM pretraining appears stabilize model performance, making adaptation more robust to masking ratio choices.
CONTINUED PRETRAINING FROM CLM AND MLM MODELS
In this section, we focus the CPT setting, starting from equally sized CLM and MLM models pretrained on the same data (100B tokens in total). The key question is whether it is more beneficial to allocate additional compute applying MLM CPT CLM model or further train MLM model. We consider CPT budgets: 2,000, 12,000, and 22,000 steps, and focus on the 610M model scale with 40% masking ratio to keep computational costs manageable.

---

Published as a conference paper at ICLR 2026
SC
TC
QA
IR
Acourasy acy
92.5
80
NDCO
80
86
78
50
92.0
84
78
40
Base
CPT
Base
CPT
Bass
CPT
Base
CPT
Figure 8: Impact of performing CPT on either CLM- or MLM- ML.M-pretriined models (denoted as Base). CPT is conducted with MLM for 22,000 steps on 610M models with a 40% masking ratio, following 42,000 steps of initial pretraining.
93.0
88
80
Accuracy
NDCG@100
92.5
78
50
92.0
84
76
10
20
20
10
20
CPT Step (K)
CPT Step (K)
CPT Step (K)
CPT Step (K)
MLM
CLM
Figure 9: Downstream performance as function of CPT length for CLM- and MLM-pretrained models, reported for settings 2,000, 12,000, and 22,000 training steps. Experiments use 610M models with a 40% MLM ratio.
MLM CPT CLM-pretrained model outperforms MLM-onll training. As shown in ure 8, the MLM-adapted CLM model consistently achieves superior downstream performance. On TC, where CLM-only models were already strong, performance is maintained and the gap MLM remains. For QA and IR, the gap is effectively closed, while for SC, the ML-Maadapeed CLM model significantly outperforms the MLMMonly model.
Fewer CPT steps already show strong performance. Interestingly, we observe that is not necessary to as many 22,000 CPT steps to achieve comparable performance between MLM and adapted CLM. As early as 12,000 steps, the results are already strong and broadly match those of MLM-only CPT (Figure 9}. with better results TC and IR, comparable performance on SC, and nearly on par on QA. In terms of trends, applying MIM CPT on CLM model also appears more promising, showing steeper improvement curve toward the end, while MLM-only training seems to plateau (particularly noticeable on SC).
RELATED WORK
Learning universal text representations has been a central focus in NLP, with MLM emerging as the dominant pretraining paradigm following the success of models like BERT (Devlin et al. 2019). MLM-based encoders leverage bidirectional attention to capture rich contextual dependencies, en- abling strong performance across wide range of tasks such classification, question answer- ing, and named entity recognition. Subsequent work has explored architectural extensions (c.g., RoBERTa (Liu et al. 2019). DeBERTa (He al. 2021), ModernBert (Warner et al. 2024b)) and more efficient pretraining strategies (Clark et al. (020). In contrast, decoder-only architectures such originally optimized for autoregressive generation (Brown et al.) 2020 Grattationi et al. 2024) Jiang et al. 2023 Almazrouet et al. 2023), making them less naturally suited
GPT-style models
for text representation tasks.
8

---

Published as a conference paper at ICLR 2026
Nevertheless, numerous studies have successfully explored adapting decoder-only models (typically larger and trained on more data than standard encoders) for learning high-quality text representations
(Muennighoff 2022 Lee et al. 2024, Meng al.] 2024 Zhang et al. 2025 Lee et al. 2025). As
result, these adapted decoders now rank among the top-performing models on prominent embedding benchmarks such MTEB (Muennighoff al.) 2023 Enevoldsen al. 2025). However, their
strong performance does not disentangle the underlying factors (likely a combination of model scale
and data regime) notably leaving open the question of whether causal pretraining itself plays a meaningful role in downstream representation quality.
Several recent studies have investigated the key factors influencing the performance of text represen- tation models. On the MLM--nnl side, Wettig et al. (2023) explore the impact of the masking ratio on downstream tasks such as sequence classification and question answering, with a focus on learn- ing dynamics, model scale, and fine-tuning behavior. From another perspective, [BehnamGhader al. (2024) propose a general framework for adapting decoder-only models embedding tasks, along with a series of ablations designed to isolate the primary contributors to performance. Com- plementary work explores ways to reintroduce bidirectional attention into causal decoder models, such by repeating sequences (Springer et al., 2024), using custom-shaped attention masks (Raffel et al. (202)). or selectively removing the causal mask on portions of the sequence in a post-training phase (Kopiczko et al., 2024 [eyee al. 2024). Articles most closely related ours propose con- trolled setups for comparing CLM, MLM, and hybrid strategies, but have either focused on very small amounts of pretraining data (Charpentier & Samuel (024), or prioritized other design pa- rameters. For example, [Weller et al. (2025) train on very large data regimes (2T tokens), analyze MLM-then-CLM learning curricula, and evaluate on generative benchmarks, providing practical insights into the impact of training design on downstream performance. However, their approach does not support extensive evaluation on incoder-specift tasks and limits exploration of additional training design choices, such as masking ratio or the CLM-MLM share in pretraining. Building on these efforts, we provide comprehensive analysis of how the choice of learning objective and attention mask during training affects downstream representation quality. By employing unified and controlled pretraining setup and evaluating across a broad range of representation tasks with statistically grounded results, we ensure fair comparisons and reduce confounding factors, enabling more precise attribution of observed effects.
CONCLUSION
Our study challenges the long-standing assumption that Masked Language Modeling (MLM) is the universally optimal approach for encoder pretraining. Through large-scale experiments at the 100B training token scale, we provide robust empirical evidence that encoders can be trained more data- efficiently without relying solely on MLM. In particular, demonstrate that decoder-only models trained with Causal Language Modeling (CLM) offer clear advantages in data efficiency, training stability, and performance on specific text representation tasks. Moreover, we show that sequential training (first with CLM followed by MLM) is more effective than MLM alone. These results suggest that the most effective path high- perffrrrroning encoder models may involve leveraging pretrained decoder models and continuing with MI.M-based training, paving the way for future state of the art encoders at discounted price. We hope our findings and released resources will support further advancements in this area.
8 LIMITATIONS AND FUTURE WORK
Our study deliberately focuses on a targeted set of parameters, namely learning objective, model size, training setup, data budget, and masking ratio. As in large-scale studies such as (Kaplan cet al., 2020; Hoffmann et al. 2022). this required fixing several other parameters, including the model architecture, tokenizer. language, and training mixture. While these factors are certainly of interest, varying them would introduce significant computational overhead, making them difficult to incorporate into our current scope. We therefore view them as a promising direction for future work, including extensions to multilingual settings to additional modalities such as vision-languae
Additionally, although we provide insights into scaling behaviors with respect to both model size and data budget, constrained these dimensions to 1B parameters and 100B training tokens. This

---

Published as a conference paper at ICLR 2026
allowed us to keep compute costs manageable while still exploring broad range of design param-
eters. While 100B tokens correspond to roughly five times the compute-optimall budget suggested
by [Hoffmann et al. 2022) for IB- parameter decoder models, and most pre-trained encoders in the
literature remain below the 1B scale (Devlin et al., 2019; [iu et al. 2019]] He et al. [202]] Warner still view pushing beyond these limits
et... a 2024a),
important direction for future work, as
many top entries in the MTEB benchmark (Muennighoff et al. 2023 Enevoldsen et al. 2025) fre-
quently exceed the IB 1B-parameter range (hhag et al. 2025 Babakhin et al. 2025). Exploring more sophisticated training curricula also appears promising for improving downstream performance, for instance, alternating CLM and MLM objectives multiple times throughout pretraining.
In similar spirit, our study focuses on the extensive evaluation of pre-trained models and deliber- ately does not include contrastive post-training for zero-shot retrieval. Applying such post-training across a large number of training configurations would have incurred substantial computational costs and could have introduced additional confounding factors that require careful control (Xiong al.)
2020 Qu et al. 2021 Wang et al., 2022 Mohr et al. 2024 Chen et al. 2024 Sturua et al. 2024). While this lies outside the scope of the present work, exploring contrastive post-training would be valuable way complement and extend our findings.
Finally, several observations emerged throughout the paper that appear intriguing but fall outside the current scope of our study. Future work could, for example, further investigate the relationship between masking ratio and downstream performance across tasks. This includes providing deeper insights into patterns such the U-shaped curve observed in Figure for TC on the 610M and IB models, in contrast with the monotonic trend observed for IR. More broadly, gaining a deeper understanding of how pretraining learning objectives transfer to different downstream tasks remains an interesting yet largely unexplored direction with substantial promise.
ETHICS STATEMENT
Environmental and compute considerations. This work examines different pretraining strate- gies for encoder models, with the aim of exploring approaches that may reduce computational costs while still producing useful text representations. By considering alternatives to traditional Masked Language Modeling (MLM) and investigating biphasic pretraining strategy, we seek ways train encoder models in a more data- and compute-efficient manner. Reducing the computational re- sources required for model training can have both environmental and practical benefits. More effi- cient encoders may help lower energy consumption and the associated footprint of large-scale NLP training. Additionally, effective encoder models could in some cases reduce the need to rely on large generative models for downstream tasks, thereby decreasing dependence more compute-intensing systems.
Responsible use of LLMs. In preparing this manuscript, we occasionally used suggestions from LLMs (GPT-5) to guide improvements in clarity, grammar, and overall readability. All scientific content, including experimental design, codebase, data analysis, results, and interpretations, is inde- pendently developed by the authors. LLMs are not involved in generating, modifying, or interpreting any experimental results, nor producing code or analyses. Their use strictly limited to selec- tively refining language to ensure clear and effective communication of our research.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that our experiments are reproducible. All pretraining and evaluation experiments are described in detail in the paper, including model architectures, training objectives, data sources, and hyperparameter settings. To facilitate replication, we release all project artifacts, including pretrained models, training scripts, and evaluation code.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint 1rXiv:2311.1688 2023.
Yauhen Babakhin, Radek Osmulski, Ronay Ak, Gabriel Moreira, Mengyao Xu, Benedikt Schifferer, A universal text embedding model for
Bo Liu, and Even Oldridge. Jlama-embed-nemottron multilingual and cross-lingual tasks. arXiv preprint arXiv:2511.0700 2025.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, al. Ms marco: A human gener- ated machine reading comprehension dataset. arXiv preprint arXiv: 1.09968, 2016. URL
https://arxiis http:/iwwwwrg////////o././ wwtpr////////.......o 09268
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapa- dos, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling, 2024. URLLhttps://peereviiee hetp:/iwww.wwee.eee.eeeeeeeeeeee formm?
id= WWW.WWRRRBBBFAcsccusssss
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harm- sen, and Xiaohua Zhai Paligemma: versatile 3b vlm for transfer, 2024. URL https:
//arxiv. abs 2200. 07726
Nicolas Boizard, Hippolyte Disserot-Boukhlef, Duarte M. Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, João Alves, Kevin El-Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei, and Pierre Colombo. Eurobert: Scaling multilingual encoders for european languages, 2025. URL http:ttt///xxii..rrrt 2503. 05500
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020. URL 115700d6bfcb496 -
Lucas Georges Gabriel Charpentier and David Samuel. GPT or BERT: why not both? In Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Leshem Choshen, Ryan Cotterell, Alex Warstadt, and Ethan Gotlieb Wilcox (eds.), The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pp. 262-283, Miami, FL, USA, November 2024. Association for Computational Linguistics. URL https://aclantee log.. 2024 con
14.247
Janlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding: Multi-linguality, caltiffunctionality, multi-granularity text embeddings through self-knowledge distillation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Asso- ciation for Computational Linguistics: ACL 2024, pp. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2000 www.:02...iiinguul...... 137. URL wnninng-aaccncc 137/
Bangkok, Thailand, August
htttps://acllanthollgy/
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. In The Eighth International Confer- ence on Learning Representations, 2020. URL teittps://openeee ew net. ftt//frrmm? i
XMH BtvB
Jacob Devlin, Ming-Weo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and
11

---

Published as a conference paper at ICLR 2026
Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10.1 N0.18633/11/1191444 1423.. URL https: /taps:///cllc ac aancllanhoolgg... .org
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305,, 2020. htttt::::///aae orr.iiooiiiiioo...aaa.ooooooooot abs httgg//ab/200..0.2 22t.//222.0550 06305
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, et al. Mmteb: Massive multilingual text embedding benchmark. In The Thirteenth International Conference on Learning Representations, 2025. BRLhhtts:::
net/ http://ppppeeeweeeeeeeeeeeeeeeee
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL
org. hbtp://b/////e 2407.
21783
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhance bert with disentangled attention. In The Ninth International Conference on Learning Representations, 2021. URL URL http://ppppevvvveeeeee net forum? wtt.imidddxXP2PtPPtuu
Jordan Hoffmann, Sehastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203. 2022. URL
https://arxiss org/ abs 2203 15556
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. OntoNotes: The 90% solution. In Robert C. Moore, Jeff Bilmes, Jennifer Chu-Carroll, and Mark Sanderson (eds.), Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pp. 57-60, New York City, USA, June 2006. Association for Computational Linguistics.
20157
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
06825
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
Jihoon Kwon Sangmo Gu Yejin Kim, Minkyung Cho Jy-yong Sohn Chanyeol. Choi Junseong Kim, and Seolhwa Lee. Ling-embed--iise task-specific control and quality refinement. linq ai research blog, 2024.
Elevating text retrieval with improved gpt data through
Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Bitune: Bidirectional instruction- tuning, 2024.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466. 2019. doi: 10. 1077 URL
htttp:://acle httt:s///ooaamhhooooroo////////
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Nv-embed: Improved techniques for training Iims generalist embedding models. arXiv preprint arXiv:2405.17422 2024. URL httpp:://axxi........//
/tt////axvvvv/// org/ abspg//bb//45555/
T7428
12

---

Published as a conference paper at ICLR 2026
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In International Conference on Learning Representa- tions, 2020. URL /ttttp::/arxivi org/ abs mtop//999.112999
Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer. Madhuri Shanbhogue, Iftekhar Naim, Gus- tavo Hernández Abrego. Zhe Li, Kaifeng Chen, Henrique Schechter Vera, al. Gemini embed- ding: Generalizable embeddings from gemini. arXiv preprint arXiv:2503.07891 2025. URL
https://erxe http://rrvvv.rrg/.ee/o. aaa///ss/22550 25003008991111
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019. URLhttpss://aret wwwpxiii.rr//bb..... hatp/ab//11/0 11692
Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Suppa, Hila Gonen, Joseph Marvin Imperial, Börje F. Karlsson, Peiqin Lin, Nikola Ljubešić, Nikola Ljubešić, LJ Miranda, Barbara Plank, Arij Riabi, and Yuval Pinter. Universal NER: A gold-standard multilingual named entity recognition benchmark. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4322 4337, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/ httl:///2222mmm 2.022.naall-long.222 URL https:
wanthology..oo org/2024 www.aaaclllla long. 2437
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. grrembedding-misttee enhance text retrieval with transfer learning. Salesforce Al Reseurch Blog. 3:6, 2024.
Isabelle Mohr Markus Krimmel, Saba Sturua, Mohammad Kalim Akram, Andreas Koukounas, Michael Günther, Georgios Mastrapas, Vinit Ravishankar, Joan Fontanals Martínez, Feng Wang, et al. Multi-task contrastive learning for 8192-token bilingual text embeddings. arXiv preprint arXiv:2402.17016 2024.
Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search, arXiv:2202.08904, 2022. /tts:tts://axxxv..rrr wttp://2..r//bb/22...5022
arXiv preprint
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text em- bedding benchmark. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chupter of the Association for Computational Linguistics, pp. 2014- 2037, Dubrovnik, Croatia, May 2023. Association for Computational L Linguistics. doi: 10.186533
v1/2023.eacl-mante 148. URL htthhttpps.. //aps:://ccaaaaa awlanthoooggyyy..oocoocoooooooll org 2223.02223.. eacl- - maain.ann..mmmmmmmmmm.mm..amm 148/
Niklas Muennighoff, SU Hongjin, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. In ICLR 2024 Workshop: How Far Are We From AGI, 2024. URL
org/ abs /ta/////222222 2402. 09906
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive pre- dictive coding. arXiv preprint 1rXiv:1807.03745 2018. URL htttps://axxv..rrtt httpaaaavvvvvv./... abs/
1807.00 03778
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL
17557
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. Rocketqa: An optimized training approach dense passage retrieval for open-domain question answering. In Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies, pp. 5835-
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https:ttt..... / org hbwgg//b///// 910. 10683.
13

---

Published as a conference paper at ICLR 2026
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su. Kevin Duh, and Xavier Carreras (eds.), Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp- Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1222 URt http:////aaaaaaaahoaaaoo www.oolooooorog/---ogg
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable ques- tions for SQuAD. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th An- nual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pp- 784-789, Mclbourne, Australia, July 2018. Association for Computational Linguistics. doi: 11116553/1/P18-- 2124. URL httpt::p///// httsss//aaclanthoooggyo.
2124/
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment tree- bank. In David Yarowsky. Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.), Proceedings of the 2013 Conference on Empirical Methods in Natural Language Process- ing. pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguisties.
ottp:///ooooaanoooooo.oooo///// org/oor 1170/
Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings, 2024. URL
2402. 15449
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krim- mel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. jina-embeddings. v3: Multilingual embeddings with task lora, 2024. URL http::/arxxv. org/abs/2409.100 10173, 2024.
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-indeeendeni named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HILT-NAALL 2003, pp. 142-147, 2003. URL https:
7/ wrcllanhhlogy.crgg.ooo wwg./033001999
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupala, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355 Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/W18--1 httpss://cc anntllnnthooogggo org.orggg//oooo /e1////18/ 5446/
URL
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general- purppose language un- derstanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neurul Information Processing Systems, volume 32, Cur- Associates, Inc., 2019. 252p://0000..../////2//2////2662
www..gs..eerpppp. cc/paper files/ Paaeeer.dppppppp..
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 2022.
Zhiguo Wang. Wael Hamza, and Radu Florian. Bilateral multi-perspective matching for natural language sentences. arXiv preprint 17xiv:1702.03814, 2017. URL https:://axxv..or abs 0os/1702.0381
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter. better, faster, longer: A modern bidirec- tional encoder for fast, memory efficient, and long context finetuning and inference, 2024a. URL
htttps://erxx wwwp:iv.orggass//oo ortp:/o///////// 13663
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin
14

---

Published as a conference paper at ICLR 2026
Adams, Jeremy Howard, and lacopo Poli. Smarter, better, faster, longer: A modern bidirec- tional encoder for fast, memory efficient. and long context finetuning and inference, 2024b. URL
https://arxiv: org/ abs 241..2212...0 13663
Orion Weller, Kathryn Ricci, Mare Marone, Antoine Chaffin, Dawn Lawrie, and Benjamin Van Durme. Seq vs seq: An open suite of paired encoders and decoders. arXiv preprint arXiv:2507.11412 2025,
11412
Alexander Wettig, Tianyu Gao, Zexuan Zhong. and Danqi Chen. Should you mask 15% masked language modeling? In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2985- 3000, Dubrovnik, Croatia, May 2023. Association for Computational Linguisties. doi: 10.18653/ w1/2023.eacl-main.1 URL http:: http:s//aaaaaattoooooo http:///////aa//aaaaaaaaoooooeeo www.:ooooyy.ggg222..eeaao www.:222.....aaaaaeaa.aaa mai.. 2177]
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.). Proceedings of the 2018 Conference of the North American Chapter of the Association for Computatioonll Linguistics: Human Language Technologies. Volume (Long Papers), pp- 1112-1122, New Orleans, Louisiana, June 2018 Association for Computational Linguistics. doi: 10. 10.18653/v 1101. (tRltttps::/
antho wowehhooooy.....oooo o ott.oogg//18-11100000
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Armold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007. 200000000000 2020.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang. Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338, 2025.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few- sample bert fine-tuning. arXiv preprint urXiv:2006005987, 2020. URL https://arxiv.
org/abs/2000 2006aaaa2202..... 05987
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding. Advanc- ing text embedding and reranking through foundation models. arXiv preprint 250iv:2506.05176. 2025. URL pottpss//aaet //arxiv///vvi//////a http///aaavvv.aaaaa/// 2506. 05176)
15

---

Published as a conference paper at ICLR 2026
A TRAINING SETUP DETAILS
This appendix provides additional information on the training parameters used in this study. Table outlines both the architectural configurations and training settings.
Model Size
210M
610M
1B
Architecture
Layers
12
26
28
Embedding Dimension
768
1,152
1,728
FFN Dimension
3,072
4,096
5,120
Attention Heads
12
18
18
Key/Value Heads
12
6
6
Layer Normalization
RMSNorm
RMSNorm
10-5
Activation Function
SwiGLU
Maximum Context Length
2,048
Positional Embeddings
RoPE
RoPE 8
10,000
Tokenizer
LLaMA 3 128,256
Vocabulary Size
Training
Weight Initialisation
Ni = 0, 0.2)
Learning Rate
x 10-4
Scheduler
WSD (linear decay)
Warmup Steps
2,000
Decay Steps
5%
Optimizer
Adam W
Adam 81
0.9
Adam B8
0,95
Adam e
10
Weight Decay
0.1
Gradient Clipping Norm
1.0
Per-GPU Batch Size
12
Gradient Accumulation Steps
1
GPUs
192
Tokens/Step
2,359,296
Table 1: Architecture and training settings for the 3 model sizes under consideration.
B DETAILS ON EVALUATION DATASETS
This appendix offers additional details the datasets used for evaluation.
SC datasets:
All SC datasets are sourced form the GLUE (Wang et al. 2018 benchmark.
MNLI (Williams et al. 2018) MNLI is large-scale benchmark for natural language infer- ence, where each example pairs a premise with hypothesis, and the task determine whether the hypothesis is entailed by, contradicts, or is neutral with respect to the premise. Since the test set not publicly available, we report results on the validation and reserve 5% of the training data for validation. We evaluate using the matched subset, which reflects in-domain perfor- mance.
QQP (Wagg et al. 2017) Binary classification dataset from Quora that asks whether pair of questions are semantically equivalent. Similarly to MNLI, we report results on the validation set and reserve 5% of the training data for validation.
16

---

Published as a conference paper at ICLR 2026
Sentiment classification dataset consisting of movie review tences. Each sentence is labeled as expressing either a positive or negative sentiment. We report results on the validation set and reserve 5% of the training data for validation.
SST-2 (Socher al. 2013)
TC datasets:
CoNLL Tjong Kim Sang & De Meulder 2003) NER dataset comprising English and Ger- man newswire text, annotated with 4 entity types: person, organization, location, and miscella- neous. In this work, we use only the English portion.
OntoNotes (Hovy et al. 2006) - A large-scale dataset for NER and other NLP tasks, spanning diverse genres including newswire, broadcast news, and conversational speech in both English and Chinese. For this study, we focus on the English portion and evaluate on named entity spans.
UNER (Mayhew et al., 2024) - Multilingual NER dataset based on Wikipedia and Wikidata annotations. For this work,
only the English subset.
QA datasets:
Challenging extractive QA dataset that tests commonsense reasoning by requiring models to select entities from a passage to answer cloze-style questions. Since the test set not publicly available, report results on the validation set and reserve 5% of the training data for validation.
ReCoRD (Wang et al. 2019
SQuAD (Raipurkar et al. 2016)- - Benchmark for extractive question answering. It consists of questions posed on Wikipedia articles, with answers given as text spans within the correspond- ing passages. We report results on the validation set and reserve 5% the training data for validation.
SQuAD-v2 (Rajpurkar et al. unanswerable questions. We report results on the validation set and reserve 5% of the training data for validation.
2018 Extension of SQuAD that includes both answerable and
IR datasets:
MLDR (Chen et al. 2024
Multilingual retrieval benchmark focused on long documents. In only the English subset. Since MLDR does provide training data, models
this work, are evaluated directly on the validation and test splits without task-specific fine-tuning.
MS MARCO (Bajaj et al., 2016) - Large-scale English passage retrieval dataset derived from real Bing search queries and web documents. MS MARCO is used in this work for both training and evaluation.
NQ (Kwiatkowski et al. 2019) Open-domain English retrieval dataset based on real Google search queries, where answers typically found in Wikipedia articles. As for MLDR, models are evaluated directly on the validation and test splits without task-specific fine-tuning.
To reduce computational overhead during evaluation on retrieval datasets, we restrict the corpus to labeled documents only, that is, those marked positive or negative. Unlabeled documents excluded from the evaluation.
C DETAILED RESULTS
C.I MLM VS. CLM
This appendix provides detailed results comparing CLM and MLM pretraining, including variations in masking ratios and model sizes. Table Table for SC, TC, QA, and IR, respectively.
Table
and Table present dataset-level results
C.2 DATA EFFICIENCY
This appendix provides detailed results data efficiency for MLM vs CLM models at different pretraining steps. Results are presented for 610M models with 40% masking for MLM. Table 6 Table 7 Table and Table 9 present dataset-level results for SC. TC, QA, and IR, respectively.
17

---

Published as a conference paper at ICLR 2026
Model Size
Objective
Masking
SST-2
QQP
MNLI
Average
20%
89.93 (4026)
86.00 (+00.18)
79.29
(d:0.32)
85.07 (4-0.12)
MLM
30%
89.84 (#1 (:1.44)
85.82
(2036)
79.69
(=022))
85.12 (#:40)
210M
40%
90.44 (#077)
85.52 (#0221)
78.72
(=0020))
84.89 (2:030)
50%
91.61 (1:.55
85,52
(2:0.19
78.00
(2:0.51)
85.04 (1:025)
CLM
90.00 (10.30)
83.84 (2:0.27)
74.56
(2:0.16)
82.80 (1:0000)
20%
92.61 (1:057)
86.98 (2:0.06)
83.08
(1:0.33)
87.56 (1:0.16)
MLM
30%
91.49 (10.69)
86.48 (2:0.25)
81.31
(2:1
86,43 (1:0.51)
610M
40%
91.42 (10.22)
86.98 (2:0.5)
82.59
(20016)
87.00 (2:0.18
50%
91.93 (4:22)
86.38 (#00344)
81.74
(+:0.66)
86.68 (4-0.13)
CLM
91.10 (40229)
84.14 (#0.12)
75.49
(#:0.22)
83.58 (#:0.17)
20%
92.39 (40.47)
87.20
(#0.18)
84.33
(4:0133)
87.97 (#:0.18)
MLM
30%
92.13 (4026)
86.46
(#00200
82.40
(4:0.24)
87.00
(#:0.13)
IB
40%
92.89 (20.2)
87.41
(2:017)
84.40
(20225)
88.23
(2:000)
50%
93.28
(2025)
87.27
(20.17)
84.09
(20.16)
88.2)
(2:0.12)
CLM
92.48 (2:046)
83.66 (20.11)
71.89
(20444
82.68 (2:0.30)
Table 2: MLM vs. CLM downstream performance results SC datasets.
Model Size
Objective
Masking
OntoNotes
CoNLL
UNER
Average
20%
92.68 (1006)
91.38 (10.15)
92,46 (10.10)
92.17 (1:0.09)
MLM
30%
92.71 (10.4)
91.37 (10.19)
92.45
(20.13)
92.18 (2:0.07)
210M
40%
92.61 (2009)
91.70
(b0.19)
92.08
(20028)
92.13
(1:0111)
50%
92.46 (40.12)
91.56 (b0.10)
92.25
(40.16)
92.09
(4:00m)
CLM
92.18 (4000)
91.67 (b0.32)
92.70
(40.10)
92.18
(4:0:99)
20%
92.93 (40.13)
92.34 (:b0.17)
93.10
(#:0.29)
92.79
(1:0:09)
MLM
30%
92.69 (40.15)
91.37 (#0.40)
91.94
(400644)
92.00
(4:0.27)
610M
40%
92.97 (20.1))
91.58
(#:.44)
92.07
(2077)
92.21
(2:0.26)
50%
92.95
(20.2))
91.79 (20.16)
92.84
(20.33)
92.53
(2:0:15)
CLM
92.79 (10.5)
92.16 (2021)
93.1 2 (2:0.20)
92.69
20%
93.03 (10.5)
92.28 (2:030)
92.84
(20.31)
92.72
(2:0.19 19)
30%
92.93 (20.1))
92.06 (20.10)
92.55
(22.16)
92.51
(2:0:09)
MLM
IB
40%
92.98 (10.2)
92.05
(10.14)
92.51
(20.47)
92.51
1 (1:0.16)
50%
93.12 (1:025)
92.46
(1:0.30
93.06
(20055)
92.88
(1:0.19)
CLM
92.60 (10009)
91.93 (10011)
92.84
(20.12)
92.46
(1:0.10)
Table 3: MLM CLM downstream performance results TC datasets.
C.3 CLM-TO-MLM RATIO
This appendix provides detailed results in the PFS setting, analyzing the impact of varying the CLM-MLM step ratio during pretraining. Results are presented for 610M models with 40% mask- ing for MLM. Table 13] Table 11 Table 12, and Table 13 present dataset-level results for SC, TC, QA, and IR, respectively.
C.4 CONTINUED PRETRAINING
This appendix presents detailed findings for the CPT setting, evaluating different CPT lengths. All experiments use 110M-parameter models, with MLM reported for 40% masking ratio. Table 14 Table 13] Table 16 and Table 17 summarize dataset-level performance for SC, TC, QA, and IR, respectively.
18

---

Published as a conference paper at ICLR 2026
Model Size
Objective
Masking
SQuAD
SQuAD-v2
ReCoRD
Average
20%
74.66 (4:26))
64.39
(#00.44
16.15 (+6.01
51.73 (4:200)
MLM
30%
74.59 (zo11)
61.87
(#1.02)
16,57 (#7.13)
51,01 (#2.58
210M
40%
73.73 (ba.1)
61.10
(#1.34)
13.45 (#2.95)
49,43 (+1336)
50%
72.75 (1:050)
58,47
(2:1.9)
14.74 (1:6.55)
48.65 (1:2.9)
CLM
64.00 (1:054)
52.60
(2:0.69)
1.17 (1:0.8)
39,26 (1:032))
20%
76.97 (1:019)
71.31
(2:0.30)
46.74 (2111))
65.00 (1:031)
MLM
30%
77.51 (1:020)
65.98
(#:2.66)
40.92 (21.75)
61,47 (d:061)
610M
40%
77.76 (1:029)
70.63
(2:0.88
39.93
(1:526)
62.77 (4:1.86)
50%
77.41 (1:47))
68.61
(#0.53)
45.82
(+:063)
63,95 (#:047)
CLM
69.42 (40220)
56.44
(+0.10)
0.42
(b0029
42.09 (#:0.20)
20%
77.81 (50.5)
73.03
(#0.35)
54.38
(4:0.22)
68,4 18,41 (#:0.33
MLM
30%
76.82 (4066))
66.09
(#2.113
43.13
(d:4944)
62.02 (+:1.84)
IB
40%
79.93 (2:277
74.96
(2:041)
55,95
(20099)
70.28 (2:0.32)
50%
79.62
(2:227)
73.53
(22.46)
55.67
(11
69.61 (#0.56)
CLM
70.54 (2:000)
57.99 (:2096)
4.90 (2:431))
44.48 (:1165)
Table 4: MLM vs. CLM downstream performance results on QA datasets.
Model Size
Objective
Masking
NQ
MS MARCO
MLDR
Average
20%
80.23 (#:054)
90.24 (20.76
53.76 (=299)
74.74 (2:0.55)
30%
81.20 (1092)
90.44 (20075
55.14 (21.02)
75.59 (2:0,57)
MLM
210M
40%
81.43 (21
89.35 (20.98)
56.56 (20.62)
75.78 (2:0.3)
50%
82.97 (2)
89.31 (20035)
58.85 (22046)
77.04 (20027)
CLM
82.08 (1091)
86.45 (20033)
49 .5.95 (211.86
72.83 (20.86)
20%
85.14 49)
88.82 (20078)
57.07 (2009)
77.01 (20.55)
30%
83,47 (4:0.9)
90,36 (+0081)
58.65 (#0042)
77.50 (4:0.42)
MLM
610M
40%
86.76 (4:063)
91.65 (#:0.55)
60.25 (#00766
79.55
(#0.35)
50%
87.64 (::074)
90.73 (22))))
60.49 (=00733
79.62
(2:1.54)
CLM
85.57 (#0034)
89.88 (21)177)
53.16 (=042)
76.20
(2:1.54)
20%
86.02 (d:0.55)
91.78 (#:0.56)
59.85
(:40.40)
79.22
(1:0.119)
MLM
30%
86.36 (40)54)
90.90 (20.61)
61.11
(20025)
79. 79.46 (=:004)
IB
40%
87.89 (#:06))
91.45 (z0040)
60.51
(22080)
79.95
(=:0411
50%
89.28 (1:040)
92.98 (2:0655)
63.64
(20034)
81.97
(1:0.77
CLM
88.73 (10.54)
90.85 (:2))
56.45 (21039)
78.68
(2:0.5)
Table 5: MLM vs. CLM downstream performance results on IR datasets.
D ADDITIONAL RESULTS
This appendix provides additional results that complement main analysis. Figure 10 comple- ments Figure by showing that CLM is not only more data-efficient than MLM but also more effi- cient in terms of training FL.OPs. Figure confirms, the 1B scale, the same trends observed for the 610M model in Figure 6 Figure 12 presents loss curves across different CPT training budgets, starting from either MLM- or CLM-pre- model and then continuing with MLM.
19

---

Published as a conference paper at ICLR 2026
Objective
Training Step
SST-2
QQP
MNLI
Average
1K
81.06 (2063)
73.78 (#0.18)
45.90 (2:022)
66.91 (#026)
2K
81.40 (2081)
73.88 (=0012)
49.40 (b:033
68.23 (#:034)
MLM
5K
82.11 (10.62)
78.44 (2:1.06)
56.90
(1:244)
72.49 (1:1.17)
10K
86.90 (11))3)
83.83 (2:0.27)
73.37
(2:162)
81.37 (1:990
20K
91.38 (10.54)
85.97 (2:0.28)
79.36
(1:0.1)
85.57 (2:020)
40K
90.94 (+0667)
86.42 (4:0.25)
81.84
(+-0300)
86.40 (+022)
IK
79.52 1+00
73.56 (d:0.24)
51.89
(+:0499)
68.32 (+022)
2K
84.24 (#9.47)
76.03 (+:0070)
60.54
(+-0.89)
73.61 (#:026)
CLM
5K
87.11 (20.9))
81.80 (200311)
66.76
(2:0:4)
78.56 (1046)
10K
89.33 (20.29)
82.66 (#0024)
69.75
(::022)
80.58 (4)
20K
90.50 (20.355
83.74
(2:0.88)
73.87
(1:045)
82.70 (4:007)
40K
90.94 (10.43)
84.20 (2:0.21)
75.31
(1:043)
83.48 (2:021)
Table 6: MLM vs. CLM downstream performance on SC datasets at various pretraining checkpoints. Results correspond to 610M models with MI.M using 40% masking.
Objective
Training Step
OntoNotes
CoNLL
UNER
Average
IK
84.39 (10.45)
82.47 (2:0.28)
82.58
(1:0.78
83.15 (1:026)
2K
88.64 (20.1))
86.02 (#0.19)
85.74
(d:064)
86.80 (::022)
MLM
5K
90.93 (20.0)
88.10 (2:094)
89.05
(+:089)
89.36 (#066)
10K
92.03 (+0.14)
90.60 (#0.22)
91.31
(b0.13)
91.31 (#:00)
20K
92.62 (20007)
91.36 (2:0.21)
92.06
(20013)
92.02 (1:01)
40K
92.86 (20.17)
91.45 (220))
91.75
(2))
92.02 (::0 19)
IK
88.09 (20.13)
85.51 (22)
87.44
(2:033)
87.01 (#:0 15)
2K
90.16 (#0.11)
88,82
(#:0.21)
89.57
(4:0.31)
89.52 (4:0.15)
CLM
5K
91.54 (20.10)
90.82
(2:0.29)
91.15
(1:0228)
91.17 (4:0.0)
10K
92.27 (20.11)
91.47
(=0.18)
92.49
(#:022)
92.08 (#0.10)
20K
92.69 (20.88)
92.21
(20.1))
92.99
(4:0.30)
92.63 (20012)
40K
92.75 (20.11)
92.25
(#0.14)
92.97
(+:0.0)
92.65 (#:00)
Table 7: MLM vs. CLM downstream performance on TC datasets at various pretraining checkpoints. Results correspond to 6 10M models with MLM using 40% masking.
Objective
Training Step
SQuAD
SQuAD-v2
ReCoRD
Average
1K
0.00 (20.00)
50.07 (10000)
0.00 (20000)
16.69 (+0000)
2K
0.94 (20.5))
50.07 (20.00
0.00 (20000
17.00 (2:0.1)
5K
33.80 (1:5660)
49.87 (20027
0.01 (2001)
27.89 (15.17)
MLM
10K
59.19 (4-12.12
51.43
(#:)79)
4.38 (#2242)
38.33 (4-4113
20K
73.09 (4:0.29)
58.38 (#:0.63)
13.79 (4:355)
48.42 (#:L277
40K
77.00 (#027)
68.72
(2:0.77)
27.98
(: 10.00)
57.90 (#:3.27)
1K
4.52 (20775)
50.00
(2:009)
0.00 (20000)
18.17 (#:0224)
2K
30.93 (4:1146)
49.65
(z:0.77)
0.10 (200055
26.89 (4:3.66
CLM
5K
52.27 (#026)
50.52
(#0049)
0.11 1 (#0.11)
34.30 (#0.11)
10K
62.31 (1051)
53.17
(20052)
0.32 (2:0.7)
38.66 (2:0.32)
20K
66.68 (10.5)
54.99
(20.62)
0.57 (:0022)
40.75 (2:0199
40K
69.40 (40.38)
56. (#0.47)
0.41 (+00.8)
41.99 (#+0.13)
Table 8: MLM vs. CLM downstream performance on QA datasets at various pretraining check- points. Results correspond to 610M models with MLM using 40% masking.
20

---

Published as a conference paper at ICLR 2026
Objective
Training Step
NQ
MS MARCO
MLDR
Average
IK
8.41 (4061)
43.97 (#2288)
8.20 (#-087)
20.19 (+-138)
2K
21.64 (#0.8))
54.88 (21.33)
15.15 (#:097)
30.55 (4)
5K
50.47 (20.74)
78. 14 (2:055)
34.89 (2:000)
54.50 (:044)
MLM
10K
69.32 (2:0.90)
85.35 (4:1.73)
44.30 (1:0.57)
66.33 (4:66)
20K
82.50 (20054)
89.60 (2:0.55)
56,36 (1:1.19)
76, (1:062)
40K
85.46 (2b0.44)
91.50 (#:0.54)
60.83 (#:0.29)
79.26 (#0023
IK
22.24 (20.92)
54.20 (20)
13.39 (::0.79)
29.94 (d:0.9)
2K
45.12 (20.68)
72.93 (::3.22)
25.26 (4:0.7))
47.77 (d:109)
5K
70.67 (b0089)
85. (d0081))
39.79 (d:144))
65.22 (d:0000
CLM
10K
78.47 (21)
88.23
46.14 (2:1.36)
70.95 (d:1005)
20K
83.80 (22.88)
89.49
(=0075
53.38 (1:066)
75.56
(::045)
40K
84.88 (#00711)
90.52
52.76
76.05 (4+0.59)
Table MLM vs. CLM downstream performance on IR datasets at various pretraining checkpoints. Results correspond to 610M models with MLM using 40% masking.
Total Steps
CLM+MLM Mix
SST-2
QQP
MNLI
Average
OK+12K
89.33 (10.68)
84.62 (20.18)
75.17 (2:0.33)
83.0 (1:028)
3K+9K
90.09 (#0.45)
84.59 (20.10)
76.32
(560.36)
83.67 (#0.39)
12K
6K+6K
90.02 (+:034))
84.70 (40.16)
76.15
(t:0.32)
83.63 (40.19)
9K+3K
89.77 (+0777)
83.93 (#0007))
73.25
(4:0.15)
82.32 (#:25)
12K+0K
89.66 (1))5))
82.54 (2).13)
70.40
(2:0.38)
80.86 (2:0
OK+22K
91.63 (20 30)
86.30 (20008)
80.27 (2:0.1)
86.07 (2:007)
5K+17K
91.28 (2:00)
86.42 (20.32)
81.24 (2:0.31)
86.31 (::0
22K
11K+1IK
91.90 (10.7))
86.69 (::0.12)
80.39 (1:0.22)
86.33 (1:025)
17K+5K
90.94 (10.4)
85.86 (20.16)
79.07
(1:0.07)
85.29 (1:0.8)
22K+0K
90.80 (2022)
83.86 (=0.17)
74.26
(b:0.27)
82.97 (d0.35
OK+42K
91.42 (10.62)
86.98 (210.15)
82.59
(1:0.16)
87.00 (1:0.88
10K+32K
92.41 (80.88)
87.32 (20.23)
83.83
(2:025)
87.85 (+:033)
42K
21K+21K
92.36 (#0.39)
87.1 (201118
83.28
(d:0.14)
87.58 (#0.11)
32K+10K
92.16 (10.3)
87.14 14 (2000
81.90
(2:0.24)
87.06 (1:0.15)
42K+0K
91.10 (20039)
84. (2)12))
75.49
(2:0.22)
83.58 (1:0.7)
Table 10: Impact of the CLMM-toMLM pretraining steps ratio on downstream performance in the PFS setup for SC datasets. Results are reported for 610M models with MLM using 40% masking.
SC
TC
QA
IR
80
Accuracy
80
90
NDCG@11
60
40
70
85
20
20
0
2
1
3
1
3
3
Training FLOPSs Factor
Training FLOPs Factor
Training FLOPs Factor
Training FLOPs Factor
Figure 10: Downstream performance with respect to pretraining FLOPs for CLM and MLM. Re- sults are reported for 610M models, with a 40% masking ratio for MLM. M. Training FLOPs are computed following the formula provided by Hoofmann et al. (2022).
21

---

Published as a conference paper at ICLR 2026
Total Steps
CLM+MLM Mix
OntoNotes
CoNLL
UNER
Average
OK+12K
92.32 (#0.15)
90.81 (40.2)
91.59 (#00220)
91.58 (+0.10)
3K+9K
92.64 (+0.10)
91.63 (#0.09)
92.20
(4-0244)
92.16 (+-0.11)
12K
6K+6K
92.69 (2)
91.62 (2))00)
92.15
(=0223)
92.15 (#000)
9K+3K
92.79 (#006)
91.83 (60
92.96
(20015)
92.53 (20.10)
12K+0K
92.29 (1:0.09)
91.67 (2:0.55)
92.47
(2:0.22)
92.14 (1:0006
OK+22K
92.83 (10015)
91.62 (200.44
92.39 (2:0.24)
92.28 (1:0.15)
5K+17K
93.09 (10000)
92.15 (10.16)
93.08 (2:0.53)
92.77 (1:0.3)
22K
11K+11K
93.15 (#004)
92.42 (50
92.93
(200300)
92.83 (:b008)
17K+5K
93.13 (40.12)
92.46
(#0.088
92.86
(4:0.24)
92.82 (-50.2))
22K+0K
92.73 (+0006)
92.22 (#0.37)
92.94
(#:0.52)
92.63 (+-0115)
OK+42K
92.97 (+0.15)
91.58 (#0.44)
92.07 (#00777)
92.21 (+:0.26)
10K+32K
93.35 (2:007)
92.39 (20
93.45 (22033)
93.07 (2:015)
42K
21K+21K
93.21 (#0 15)
92.43 (20 13)
93.02 (20443)
92.89 (2:0.15)
32K+10K
93.32 (4:0.0)
92.41 (40.77)
93.02 (2:0.24)
92.92 (1:011)
42K+0K
92.79 (1:0005)
92.16 (1021)
93. 12 (2:0.29)
92.69 (1:0.44
Table 11:
Impact of the CLM-1o-MLM pretraining steps ratio downstream performance in the PFS setup for TC datasets. Results are reported for 610M models with MLM using 40% masking.
Total Steps
CLM+MLM Mix
SQuAD
SQuAD-v2
ReCoRD
Average
0K+12K
67.95 (2065)
53.73 (2:2.4)
2,85 (2:06)
41,51 (1055)
3K+9K
71.36 (2063)
56.28 (2:.44)
3.68 (#180)
43.77 (#066)
12K
6K+6K
71.29 (1:02))
55,85
(1:0.99
1.19 (1:040)
42.78 (1:0.7)
9K+3K
68.20 (1:08))
53.85
(.0)
2.55 (1:407)
41.53 (d:188))
12K+0K
63.40 (1072)
53.49 (2:0.4)
0.32 (#0.1)
39.07 (#0034
0K+22K
74.02 (1:038)
61.02 (210.92)
19.29 (1:545)
51.44 (1:18))
5K+17K
75.71 (b:99)
61.50 (#2.711
26.08 (b6.25)
54.43 (d:1.1))
22K
I1K+1IK
75.43 (2020)
61.33 (#b0.33)
31.14 (1:6222
55.97 (#:248)
17K+5K
74.15 (2:046)
59.03 (::1.53)
26.99
(1:33)
53.39 (t:101)
22K+0K
67.52
(1040)
55.44 (2:0666
0.47 (20.14)
41.14 (20025)
0K+42K
77.76 (1:029)
70.63
(2:055)
39.93 (1:5.66
62.77 (E:18))
10K+32K
79.20 (4-0.22)
72.33
(#:081)
51.79 (d:1.50)
67.77 (40053)
42K
21K+2IK
77.12 (+:228
67.81
(#0066)
33,60
(6:1135)
59.51 1 (4:384
32K+10K
76.54
(1021)
64.49
(4:082)
43.20
(d:1.29)
61.41 (460666
42K+0K
69.42 (4030)
56.44
(20.10)
0.42
(40
42.09 (#:0200
Table 12: Impact of the CLM-to-MLM pretraining steps ratio on downstream performance in the PFS setup for QA datasets. Results are reported for 610M models with MLM using 40% masking.
TC
QA
70
88
Accurscy
93.0
NDCG@10
86
79
84
CLM Training (%)
CLM Training (%)
CLM Training (%)
CLM Training (%)
Figure 11: Impact of two-stage CLM+MLM pretraining on downstream performance after 42,000 training steps. Results reported for the IB model, with MLM applied using 40% masking ratio. To keep computation costs manageable, CLM+MLM results are shown only for the 25%-75% regime.
22

---

Published as a conference paper at ICLR 2026
Total Steps
CLM+MLM Mix
NQ
MS MARCO
MLDR
Average
0K+12K
74.15 (::1.46
85.95 (21)8)
49.59 (20.5)
69.90 (d:1.21)
3K+9K
80.08 (#0.44)
89.63 (20.7)
55.09 (20.5)
74.94 (::0.22)
12K
6K+6K
80.67 (10061)
88.30 (20.79)
56.70 (200.5)
75.22 (1:041)
9K+3K
80. (20.44)
89.23 (220
55.63 (120,87)
74.99 (1:0.3)
12K+0K
80.16 (+1.16)
88.88 (#27
49.26 (#0054)
72.77 (H6022)
OK+22K
83.72 (2:085)
90.42 (230.3))
58.47 (20.45)
77. 77.54 (d:0288
5K+17K
84.07 (#H0.22)
89.38 (#00777)
58.42 1#00)
77.29 (+:0446)
22K
I1K+1IK
84.48 (:00.50)
89.72 (+00656)
58.22 (#00477)
77.47 (4-0.40)
17K+5K
83.36 (20057)
90.97 0.566
59.12 (20.26)
77.81 (d:0.18)
22K+0K
84.01 (20075)
90.36 (299))
52.75 (=2.03)
75.71 (d:0.34
0K+42K
86.76 (20663
91.65 (2444
60.25 (=0.76)
79.55 (d:0.35)
10K+32K
86.79 (:10.26)
91.98 (:35)
60.86 (1:1777)
79.88 (1:0.87)
42K
21K+2IK
86,80 (2:0.86)
90.98 (20092)
61.22 (2:)
79.67 (1:081)
32K+10K
86.82 (::049)
91.40 (20,92)
61.14 (20.84)
79.79
(1:0336
42K+0K
85.57 (:0.34)
89.88 (=1.17)
53.16 (22042)
76.20 (1:054)
Table 13:
Impact of the CI-M-toMMLM pretraining steps ratio on downstream performance in the PFS setup for IR datasets. Results are reported for 610M models with MLM using 40% masking.
PT Objective CPT Steps
SST-2
QQP
MNLI
Average
91.42 (40.2)
86.98 (#0.15)
82.59
87.00 (+0.18)
2K
91.54 (#:554
86.84 (+0.16)
82.85 (1:227)
87.08 (4:0200
MLMM
12K
92.71 (2:)37)
86.89 (20.31)
83.23 (20.29)
87.61 (1:011)
22K
92, 16 (9:059)
86.94 (::0. 10)
83.31 (20.35)
87.47 (1:024)
91.10 (:: 39)
84. 14 (::0. 12)
75.49
(20.22)
83.58 (d:0.7)
2K
92.06 (1:0.88)
85.93 (:10.15)
78.18 (2:0.33
85.39 (1:0.1))
CLM
12K
92.98 (1:034)
87.48 (:10.13)
83.05
(20.18)
87.84 (1:0.5)
22K
93.62 (3:0.33)
87.56 (220.08)
84.02
(20.10)
88.40 (+0.11)
Table 14: Impact of continued MLM pretraining on both MI.M- and CL.M-pretrained models across different sequence lengths on SC datasets. Results are reported for 610M models with 40% masking for MLM.
PT Objective
CPT Steps
OntoNotes
CoNLL
UNER
Average
92.97 ($015)
91.58 (#044)
92.07 (2077)
92.21 (2:0.2)
2K
92.85 (1:026)
90.89 (11110)
92.74 (2:0.99
92. 16 (20.41
MLM
12K
93,02 (10.17)
91.19 (1082)
92.59
(10.51)
92.27 (20041)
22K
92.82 (4:045)
91.41 (#0.6)
92.10 (#0557)
92.11 (4-0.32)
92.79 (4:005)
92.16 (+:22))
93.12 (#:0
92.69 (4:0.14)
2K
92.91 (4000)
92.71 (4026)
93.02 (#0226)
92.88 (4:0.13)
CLM
12K
92.78
(1:05))
92.53
(3)
93.01 (20)2
92.77
(200.2)
22K
92.85
(30
92.30 (4: 19)
92.94 (20111)
92.70 (20.05)
Table 15: Impact of continued MLM pretraining on both MLM- and CLM-pretrained models across different sequence lengths on TC datasets. Results are reported for 610M models with 40% masking for MLM.
23

---

Published as a conference paper at ICLR 2026
PT Objective
CPT Steps
SQuAD
SQuAD-v2
ReCoRD
Average
77.76 (4:029)
70.63 (+0.58)
39.93 (4:5.26)
62.77 (4-126)
2K
77.92 (4:21)
71.07 (#085)
41.69 (4:4.59)
63.56 (4-.77)
MLM
12K
78.71 (b0.3)
72.35 (20)67)
48.66 (22.17
66.57 (2085)
22K
79.01 (#0018)
72.77 (#0447)
51.21 (#1)33))
67.66 (b:053)
69.42 (b0030)
56.44 (20.10)
0.42
(#:039)
42.09 (2:020)
2K
73,48 (1:02))
59.13 (100211)
10.94
47.85 (1:2.00
CLM
12K
77.25 (1:029)
65.62 (21115)
46.75
(2:))
63.21 (1:017)
22K
78. 13 (1:027)
69.69 (200.56)
52.06 (2:0.72)
66.62 (1:034)
Table 16: Impact of continued MLM pretraining on both MLM- and CLM-pretrained models across different sequence lengths QA datasets. Results reported for 610M models with 40% masking for MLM.
PT Objective
CPT Steps
NQ
MS MARCO
MIDR
Average
86.76 (4:068)
91.65 (#0.45)
60.25 (#0.76)
79.55 (#:0.35)
2K
86.77 (4084)
92.18 (4063)
62.31 (#0.42)
80.42 (#:0.27)
MLM
12K
87.01 ((0.52)
92.32 (#092)
60.44 (21))
79.92 (#:0.47)
22K
87.58 (#0662)
92.52 (#0779)
61.26 (=0.53)
80.45 (=0227)
85.57 (#:034)
89.88 (#1.17)
53.16 (=0.42)
76.20 (=0554)
2K
86.46 (#0.39)
88.80 (#:076)
60.45 (=043)
78.57 (=:0.22)
CLM
12K
88.29 (1:0.66)
89.56 (210.2)
62.97 (200.2)
80.27
(2:0.29)
22K
87.67
(1:093)
90.71 (2:0.23)
63.72 (2:0,4))
80.70 (2:0.26)
Table 17: Impact of continued MLM pretraining on both MLM- and CLM-pretrained models across different sequence lengths on IR datasets. Results are reported for 610M models with 40% masking for MLM.
12K
22K
Loss
2
2
Training Step (K)
Training Step (K)
Training Step (K)
MLM
CLM
Loss trends in the CPT setting. The blue curve shows the loss trajectory of a model using MLM for both pretraining and continued pretraining, while the red curve depicts CLM model adapted using the MLM objective during CPT. Results are reported for the 610M model with a 40% masking ratio for MLM.
Figure 12:
24


Reviews:
[{"review_id": "pLtqHR90F4", "reviewer": "Reviewer_9eov", "summary": "The paper runs controlled ablations (same data, tokenizer, and encoder family; sizes 210M/610M/1B) to test whether encoder pretraining should remain MLM-only or benefit from a two-stage CLM→MLM recipe. The authors show the following findings (in my understanding) (1) MLM wins on average for representation tasks; (2) CLM is more data-efficient early in training and yields more stable fine-tuning; (3) a two-phase schedule that starts with CLM and then switches to MLM outperforms MLM-only at fixed compute; and (4) in continued pretraining, adapting a CLM with MLM beats continuing MLM on an MLM model—suggesting a practical path that leverages widely available decoder checkpoints.       \nSetup details: EuroBERT encoders, FineWeb-Edu English data, LLaMA-3 tokenizer; most two-stage/CPT studies use the 610M model and 40% masking for MLM.", "questions": "(covered above)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Thorough controls and scale of experiments. Same backbone family, shared data order, and systematic sweeps (sizes, masking ratios, schedules) reduce common confounders in MLM vs CLM comparisons. Compute and evaluation budgets are reported in detail. \n\n\n- Actionable recipe: Under fixed compute, CLM→MLM (e.g., 25%→75%) consistently beats MLM-only; continued-pretraining results make a practical case for initializing from a decoder checkpoint, then adapting with MLM.   \n\n\n- Early data-efficiency + robustness: CLM learns useful representations faster early on and is less sensitive to fine-tuning learning rates; CLM→MLM is also less brittle to masking-ratio choices.     \n\n\n- Clarity on task sensitivity: The paper separates effects across SC/TC/QA/IR, noting where bidirectionality matters most (e.g., QA) and where CLM narrows the gap (e.g., IR), which is useful guidance for practitioners.", "weaknesses": "- General validity of the experiments might be narrow in scope: Results are tied to one encoder family and one tokenizer in English; generalization to other backbones (e.g., DeBERTa/ModernBERT), tokenizers, or languages is open.   \n\n\n- CPT evidence is concentrated at 610M and one masking ratio: While the authors sweep sizes and masks in base runs, most decisive experiments (schedules, CPT) are at 610M with 40% mask, which limits how confidently we can extrapolate.   \n\n\n- Magnitude of gains is modest: The CLM --> MLM uplift over strong MLM baselines appears small in many cases (Figure 2 scales across figures are a bit misleading in this regard, where each task have different scales)\n\n\n- Scaling anomalies might need more explanation: Some configurations show 1B not strictly dominating 610M (e.g., QA at certain masking ratios), which makes me wonder if there was other training issues that could have contributed to this. It would be good if the authors provided some explanation on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MmxFHtjqxI", "reviewer": "Reviewer_JaBv", "summary": "There has been much discussion in the community at the relative merits of Bert-style training for learning representations of text, in the age of next token prediction with large language models. This paper presents a careful empirical study as to  compare masked language modeling (MLM) and causal language modeling (CLM). They compare several models with less than or equal to 1B params, on several tasks. They find that MLM still generally beats out CLM, but training with CLM and then MLM generally outperforms any other alternative. They study continued pretraining where they find also that adapting a CLM trained model via MLM generally performs best.", "questions": "1. Are there any experiments where we first train with MLM and then with CLM? Related, could one feasibly alternate between the two, e.g. per epoch or per batch?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Originality: Several studies have studied both MLM and CLM for learning representations of data, but to my knowledge none have such an in depth empirical study.\n\n2. Quality: This paper uses strong state-of-the-art models as the baseline and evaluate on a wide array of tasks. As such, I consider the empirical results to be high quality\n\n3. Clarity: The motivation and work done in this paper are quite clear.\n\n4. Significance: The paper asks a clear and important question and answers is with good empirical evidence\nThis paper provides concrete recommendation to practitioners.", "weaknesses": "1. The scale is somewhat small, at 1B but is still within a reasonable range for such an experiment. It might be valuable to see how this looks for a 7B model.\n\n2. It would be good to see performance on all the tasks in the MTEB (Massive Text Embedding Benchmark).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HY1ERj2gx4", "reviewer": "Reviewer_e3tm", "summary": "This paper compares MLM (masked language modeling) and CLM (causal language model) objectives in a large variety of scenarios. As the authors note, encoder models are typically trained solely with the MLM objective to encourage good representations, while decoder models use CLM towards the direct objective of next token predicting. The authors design an experimentation setup to do fair assessments of both of these methods, carefully ablating aspects like model size, masking rate, and curriculum incorporating each aspect. They test on a variety of common evaluations to find cases where different models have an advantage. This leads to some surprising conclusions around training encoder models in a data and compute efficient manner - specifically, that a \"biphasic training strategy\" can lead to good performance at the scales tested here. I also found the analysis of fine-tuning stability particularly valuable, and somewhat under-considered in other works.", "questions": "- Any further analysis or intuition on the u-shaped masking ratio?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has many interesting experiments and appears to explore a large space of possible model configurations. \n\n- Reasonable experimental setup using common best practices in small model encoder/decoder training: high quality data choice, learning rate schedule, controlled architecture\n\n- Carefully sweeps an interesting set of parameters: model size, masking ratio, curricula combining CLM + MLM at different ratios, careful task specific tuning to analyze model stability. It is uncommon to see all of these varied at reasonable experimental scale, so the authors should be commended. \n\n- The decision to test mixed objectives like applying MLM after CLM in the same training run is very interesting. The authors note that the this method is applicable to the case where an encoder model can be trained from an existing pretrained decoder only model, as it seems many industrial resources are focused on decoder training. \n\n- The experiments and finding seem very practically motivated. Finetuning stability is an important practical aspect of encoder models as commonly used, and a focus on limited data efficiency is also practical.", "weaknesses": "My main weaknesses with this paper involve data choices and budget. This paper has many interesting experiments, some of which lead to further research questions. \n\n1. As the authors acknowledge in section 8, their dataset is limited to 100B tokens. While this is far beyond the decoder compute optimal ratio, it does not seem well established whether this is a reasonable budget for encoder model training. For example, the progression from the original BERT to RoBERTA (and XLM-R) involved a large increase in data scale. Further recent research has shown that considering inference compute costs pushes scaling to favor longer data durations with smaller models (\"overtraining\" - see the llama3 paper or https://arxiv.org/abs/2401.00448). There is closely related work (only briefly cited by the paper authors, possibly concurrent) that trains similar model sizes up to 2T tokens: https://arxiv.org/abs/2507.11412 (Weller et al. 2025). \n\n2. Related, the choice of 100B tokens is only focused on the fineweb edu data. This data is high quality for some domains, but does not represent the diversity of data present in a practical encoder data mixture. See the previously mentioned work or the ModernBERT paper (https://arxiv.org/abs/2412.13663), which additionally includes code and scientific papers. The original Fineweb edu work shows the greatest improvements in benchmarks like MMLU, which specifically draw from education-relevant sources. \n\n3. The total size of 100B tokens also limits the experimental ratios when combining MLM + CLM objectives. While motivated by using an existing pretrained LLM, the ratios of 25% budget increments (corresponding to 25B tokens), seem unrealistic. This would mean that the 75-25% scenario uses CLM on 75B tokens and MLM on 25B, which seems far from the more realistic scenario of a very large budget allocated to CLM with a smaller scale MLM. \n\nIn a similar vein, while the masking ratio experiments are very interesting, it is hard to judge whether the results are due to the relatively short training budgets. Perhaps 100B tokens is not enough to learn good representations and the very high or low rates. As the authors note in line 473:\n\n> but learning dynamics suggest that further performance improvements are possible with extended training\n\nMinor:\n\nFurther analysis or explanations of differences in IR and token level tasks (253-254). Other analysis of the interaction between LR decay and curriculum would be interesting \n\nThe tables in the appendix are appreciated but would benefit from formatting such as bolding extreme values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qneNo9j5o0", "reviewer": "Reviewer_hZNQ", "summary": "This paper conducts a comprehensive empirical comparison between Masked Language Modeling (MLM) and Causal Language Modeling (CLM) for encoder pretraining. Using 38 models (210M–1B parameters) trained under matched computational budgets, the authors analyze performance across multiple NLP tasks and setups, including two-stage (CLM→MLM) and continued pretraining. The results show that MLM remains generally superior for text representation tasks, while CLM exhibits higher data efficiency and fine-tuning stability. Combining both (CLM followed by MLM) achieves the best overall trade-off between efficiency and final performance.", "questions": "please see weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and logically organized. The narrative is easy to follow, and figures/tables effectively support the main findings.\n2. The experimental analysis is extensive and carefully controlled. The authors evaluate across diverse model sizes, masking ratios, and task types, ensuring that performance differences stem from the pretraining objective itself rather than confounding factors.\n3. The research question is highly relevant: as decoder-based models increasingly dominate embedding benchmarks, understanding whether CLM pretraining offers intrinsic advantages is crucial. The paper addresses this gap convincingly and provides practical insights for future encoder design.", "weaknesses": "1. The paper establishes that MLM excels in tasks requiring bidirectional reasoning (e.g., QA, classification), while CLM performs competitively in token-level or retrieval settings. However, the authors stop short of elaborating on why certain pretraining paradigms align better with specific task types. A deeper discussion of this task–objective relationship would make the work more insightful and actionable for practitioners.\n2. Although training budgets are said to be matched, the paper does not provide explicit comparisons in FLOPs, runtime, or memory efficiency. Such quantitative reporting would strengthen the claims about CLM’s data and compute efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)