NeoBERT: A Next Generation BERT
Authors: Lola Le Breton
Quentin Fournier
John Xavier Morris
Mariam El Mezouar
Sarath Chandar
Venue: Accepted by TMLR ()

Abstract: Recent innovations in architecture, pre-training, and fine-tuning have led to the remarkable in-context learning and reasoning abilities of large auto-regressive language models such as LLaMA and DeepSeek. In contrast, encoders like BERT and RoBERTa have not seen the same level of progress despite being foundational for many downstream NLP applications. To bridge this gap, we introduce NeoBERT, a next-generation encoder that redefines the capabilities of bidirectional models by integrating state-of-the-art advancements in architecture, modern data, and optimized pre-training methodologies. NeoBERT is designed for seamless adoption: it serves as a plug-and-play replacement for existing base models, relies on an optimal depth-to-width ratio, and leverages an extended context length of 4,096 tokens. Despite its compact 250M parameter footprint, it achieves state-of-the-art results on the massive MTEB benchmark, outperforming BERT$_{large}$, RoBERTa$_{large}$, NomicBERT, and ModernBERT under identical fine-tuning conditions. In addition, we rigorously evaluate the impact of each modification on GLUE and design a uniform fine-tuning and evaluation framework for MTEB. We release all code, data, checkpoints, and training scripts to accelerate research and real-world adoption.

Published in Transactions on Machine Learning Research (05/2025)
NeoBERT: A
BERT
Lola Le Breton 1.2,3
Quentin Fournier John X. Morris"
Mariam El Mezouar
Sarath Chandar1:233
Chaandar Rescarch Lab
Quebec AI Institute Royal Military College of Canada
Mill
Polytechhique Montréal 6 Canada CIFAR AT Chair
*Cornell University
Reviewed on OpenReview:
http:://pparrriiieeee///o///ttt0 7mwH
Abstract
Recent innovations in architecture, pre-training, and fine-tuning have led to the remarkable in-context learning and reasoning abilities of large auto-regressive language models such as LLaMA and DeepSeek. In contrast, encoders like BERT and RoBERTa have not seen the same level of progress despite being foundational for many downstream NLP applications. To bridge this gap, we introduce NcoBERT, a next-gencration encoder that redefines the capabilities of bidirectional models by inlegrating state-of-the-arl advancements in archi- tecture, modern data, and optimized pre- training methodologies. NeoBERT designed for scamless adoption: it serves as a plug-and-play replaccment for existing base models, ro- lies on an optimal depth-to-width ratio, and leverages an extended context length of 4,096 tokens. Despite its compact 250M parameter footprint, it achieves state-of-the-art results on the massive MTEB benchmark, outperforming BERTlarge RoBERIatarge NomicBERT, and ModernBERT under identical finc-tuning conditions. In addition, we rigorously evaluate the impact of each modification on GLUE and design a uniform fine-tuning and evaluation framework for MTEB. We release all code, data, checkpoints, and training scripts to accel- erate research and real-world adoption
1 Introduction
Anto-regressive language models have made tremendous progress since the introduction of GPT RRaford
et al. 2018). and modern large language models (LLMs) such as LLaMA (Dubey el al. 2024 1. Mistral (Jiang let al. 2023), OLMo [Groeneveld et al. 2024). and DeepSeek-rl (DeepSeek-Al
al, 2025p now exhibit remarka ble reasoning and in-context learning capabilities. These improvements result from scaling both the models and the web-scraped text datasets they trained on, as well as from innovations in architecture and optimization techniques. However, while decoders have continuously evolved, encoders have not seen the same level of progress. Worse, their knowledge has become increasingly outdated despite remaining critical for a wide range of downstream NLP tasks that depend on their embeddings, notably for rotrieval--ngmmeneed generation [Ram et al.) 2023) and toxicity classification (Hartvigsen et al. 2022). Despite being five years old, BERT (Devlin et al. (2019) and RoBERTa (Liu al.) (019)) remain widely used, with more than 110 million combined downloads from Hugging Face as of the writing of this paper.
Similar to decoders, which undergo multi-stage processes of pre-training, instruction-tuning and alignment, encoders also require successive training phases to perform well on downstream tasks. First, encoders go through self-supervised pre-training on large corpora of text with the masked language modeling objective. By predicting masked or replaced tokens, this stage enables models to learn the structural patterns of language and the semantics of words. However, the pre-training task is disconnected from downstream applications, and models require additional training to achieve strong performance in clustering or retrieval.
https://huggiingface.. http:///......accacaaereeaaee co/chandar-lalle 1ab/ReoBERT https://github. com/ chandar- lab/HeoßERT

---

Published in Transactions on Machine Learning Research (05/2025)
Thus, a second fine-tuning phase is often achieved through multiple stages of semi-supervised contrastive learning, where models learn to differentiate between positive and negative sentence pairs, refining their embeddings in the latent space.
Recently, substantial progress has been made in improving the fine-tuning stage of pre-trained encoders, with models like GTE (Li et al. (023b), jina-embeddings (Sturua et al. 2024). SFR-embeddings (Meng iet al. 2024), and CDE (Morris & Rush (024) significantly outperforming previous encoders on the MTEB leaderboard, recent and challenging benchmark spanning tasks and 56 datasets. However, all these approaches focus proposing complex fine-tuning methods and do not address the inherent limitations of their pro-trained backbone models.
As a result, there is a dire need for a new generation of BERT-like pre-trained models that incorporate up-to-date knowledge and leverage both architectural and training innovations, forming stronger backbones for these more advanced fine-tuning procedures.
In response, we introduce NeoBERT, a next-generation BERT that integrates the latest advancements in architecture, data, and pre-training strategies. The improvements are rigorously validated on GLUE by fully pre-training 10 different models that successively incorporate the modifications. This validation ensures that the improvements benefit encoder architectures and highlights how some design choices drastically affect the model's abilities. Additionally, we design and experimentally validate a two-stage training procedure to increase NeoBERT's maximun context window from 1, 024 to 4, 096. To ensure a fair evaluation of NeoBERT against existing basclines and to isolate the impact of fine-tuning procedures, we propose a model-agnostic and systematic fine-tuning strategy with straightforward contrastive learning. All models are fine-tuned using this standardized approach and subsequently evaluated on the MTEB benchmark.
On MTEB, NeoBERT consistently outperforms all competing pre-trained models while being 100M param- eters smaller than the typical large-sized encoders. With a context window of 4,096 tokens, it processes soquences 8x longer than RoBERTa (Liu et al., 2019) and two times longer than NomicBERT (Nussbaum let al. 2024). It also the fastest encoder of its kind, significantly outperforming ModernBERT base and large in terms of inference speed. Despite its compact 250M parameter size, NeoBERT trained for over 2T tokens, prioritizing training over scale to maximize accessibility for both academic rescarchers and industry users without requiring large-scale compute resources. This makes NeoBERT the most extensively trained model among modern encoders, ensuring robust generalization and superior downstream performance. Fur- thermore, NeoBERT maintains the same hidden size as base models, allowing for seamless plug-and-play adoption without modifications to existing architectures. As the only fully open-source model of its kind, we release the code, data, training scripts, and model checkpoints, reinforcing our commitment to reproducible rescarch.
2 Related work
In 2019, Devlin et al. 2019)) introduced BERT, a novel approach to embedding text using bi-directional Transformers pre-trained without supervision large corpora. Shortly after, Liu et al. (2019) improved over BERT's pre-training by removing the next-sentence prediction objective and drastically increasing the amount of data, leading to RoBERTa. Since then, the primary focus of the community has shifted towards optimizing the fine-tuning phase of these models through contrastive learning, where the model is trained to maximize the similarity between positive text. pairs while pushing them apart from negative samples.
Among the earliest contrastive learning approaches designed for encoders, SimCSE (Gao et al. 2022) demon- strated that sentence pairs could be easily generated by feeding the same input to the model twice and applying dropout to introduce noise. However, this simple approach was outperformed by models like GTE (Li et al. 2023b), which introduced more advanced contrastive lcarning techniques. GTE employed a weakly supervised stage that takes advantage of the vast. number of successive sentence pairs available in traditional unlabeled datasets, followed by a semiisupervised stage incorporating labeled sentence pairs from high-quality datasots such as NLI (Bowman et al.] (015) and FEVER (Thorne et al. 2018). Recently, fine- fineggriined strategies have emerged to better adapt models to both task and context. For instance, Jina- embeddings (Sturua et al. 2024) introduced task-specific Low-Rank Adaptation (LoRA) adapters. As of
2

---

Published in Transactions on Machine Learning Research (05/2025)
Jamuary 2025, CDE (Morris & Rush 2024) ranks at the top of the MTEB leaderboard for models under 250M parameters thanks to two key innovations: grouping samples with related contexts into the same batch and providing contextual embeddings for the entire corpus in response to individual queries.
However, pre-training has not seen the same level of effort, and thus progress, most likely due to its pro- hibitively high computational cost. RoBERTa, for instance, required a total of 1, 024 V100 days for its pre- training. As result, GTE, Jina-embeddiings, and CDE all rely on pre-trained BERT, XLM-RoBERIA (Con- neam et al. 2020), and NomicBERT (Nussbaum et al. 2024 to initialize their respective models. The latter, NomicBERT, represents recent effort to refine BERT's architecture and pre-training. NomicBERT incor- porates architectural improvements such as SwiGLU and RoPE, utilizes Fltentttenionn, and extends the context length to 2, 048 tokens. Despite these innovations, NomicBERT still relied on sub-optimal choices, as discussed in |section ModernBERT with the goal of further refining the pre-training of NomicBERT. Although we share some of the modifications, we make distinct design choices and conduct thorough ablations that ultimately lead greater performance on MTEB.
In parallel with the development of NeoBERT, Warner et al. (2024) released
3 NeoBERT
The following section describes NeoBERT's improvements over BERT and RoBERTa, as well as the recent NomicBERT and ModemBERT models. Since GTE and CDE use BERT and NomicBERT as their pre- trained backbone, they inherit their respective characteristics. Table summarizes the modifications.
Table 1: Comparison of Model Architectures, Training Data, and Pre- Training Configurations.
BERT
RoBERTa
NomicBERT
ModernBERT
NeoBERT
base
large
base
large
base
base
large
medium
Layers
12
24
24
12
28
28
Hidden Size
768
1,
021
768
1,024
768
768
1, 020
768
Attention Heads
12
16
12
16
Parameters
120M
350M
125M
355M
137M
149M
395M
250M
Activation Function
GeLU
SwiGLU
GeGLU
SwiGLU
Positional Encoding
Positional Embeddings
RoPE
RoPE
RoPE
Normalization
Post-LayerNorm
Post-LayerNorm
Pre-LayerNorm
Pre-RMSNorm
BooksCorpus
BooksCorpus
BooksCorpus Wikipedia
Data Sources
Oeet
Undisclosed
RefinedWeb
Wikipedia
Stories CC-News
Dataset Size
13GB
160GB
13GB
2.8TB
Dataset Year
2019
2019
2023
2023
Tokenizer Level
Character
Byte
Character
Character 50K
Character
Vocabulary Size
30K
50K
30K
30K
Sequence Length
512 MLM NSP
512
2, 048
1, 02044 + 8, 192
024
096
Objective
MLM
MLM
MLM
MIMM
Masking Rate
15%
15%
30%
30%
20%
Masking Scheme
80/ 80/10/10
80/10/10
100
Optimizer
Adam
Adam
AdamW
StableAdamW WSD 448k to 5M
AdamW
Scheduler
CosineDecay
Batch Size
131k tokens
131k
8M
2M
Tokens Seen
131B
131B
2.1T
Alternate Attention
DeepSpeed
DeepSpeod
Training
DDP
DDP
Unpadding
FlashAttention
FlashAttention
FlashAttention
3

---

Published in Transactions on Machine Learning Research (05/2025)
3.1 Architecture
The Transformer architecture has been refined over the years and has now largely stabilized, with models like LLaMA 3 being essentially the same as the original LLaMA. NeoBERT integrates the latest modifications that have, for the most part, become standard.
Pepth-to-Width The concept of depth efficiency has long been recognized in neural network architectures. In the case of Transformers, stacking self-attention layers is so effective that it can quickly saturate the network's capacity. Recognizing this, Levine et al. (2020) provided theoretical and empirical evidence for an optimal depth-to-width ratio in Transformers. Their findings suggested that most language models were operating in "depth-inefficieney" regime, where allocating more paramcters to width rather than depth would have improved performance. In contrast, small language models like BERT, RoBERTa, and NomicBERT are instead in width-inefficiency regime. To maximize NeoBERT's parameter efficiency while replacement, retain the original BERT width of 768 and
ensuring it remains seamless instead increase its depth to achieve this optimal ratio.
Positional Information Transformers inherently lack the ability to distinguish token positions. Early models like BERT and RoBERTa addressed this by adding absolute positional embeddings to the token embeddings before the first Transformer block. However, this approach struggles to generalize to longer sequences and requires the positional information to be propagated across layers. To overcome these limi- tations, Su al. (2023) proposed Rotary Position Embeddings (RoPF), which integrate relative positional information directly into the self-attention mechanism. RoPE has quickly become the default in modern Transformers due to its significant improvements in performance and extrapolation capabilities. NcoBERT, like all newer encoders, integrates RoPE. Nevertheless, degradation still occurs with sequences significantly longer than those seen during training. As a solution, Peng et al. (2023) introduced Yet Another RoPE Extension (YaRN), which allows to efficiently fine-tune models on longer contexts. NeoBERT is readily compatible with YaRN, making well-suited for tasks requiring extended context.
Layer Normalization Consistent with standard practices in modern Transformer architectures, we move the normalization layer inside the residual connections of each forward and attention block, a technique known as Pre-Layer Normalization (Pre-IN). Pro-LN improves stability, allows for larger learning rates, and accelerates model convergence (Xiong et al.] 2020). While all newer encoder models adopt Pre-LN, they typically continue to the classical LayerNorm rather than Root Mean Square Layer Normalization (RMSNorm). In NeoBERT, substitute the classical LayerNorm with RMSNorm (Zhang & Semnrich, 2019), which achieves comparable training stability while being slightly less computationally intensive, as it requires one fewer statistic.
Activations BERT and RoBERTa utilize the standard Gaussian Error Linear Unit (GELU) activation function. However, Shazeer 22200) demonstrated the benefits of the Gated Linear Unit in Transformer architectures. These activation functions have since been adopted in several language models, including the LLaMA family. Following previous works, NeoBERT incorporates the SwiGLU activation function, and because it introduces third weight matrix, we scale the number of hidden units by factor of to keep the number of parameters constant.
3.2 Data
Data has emerged as one of the most critical aspects of pre-training, and datasets with increasing quantity and diversity are frequently released. NeoBERT takes advantage of the latest datasets that have proven to be effective.
Dataset BERT and NomicBERT were pre-trained on two carefully curated and high-quality datasets: Wikipedia and BookCorpus (Zhu et al., 2015). As Bacvski et al. (2019) demonstrated that increasing data size can improve downstream performance, Liu al. (2019) pre-trained RoBERTa on 10 times more data from BookCorpus, CC-News, OpenWeb Text, and Stories. However, RoBERTa's pre-training corpus has
4

---

Published in Transactions on Machine Learning Research (05/2025)
become small in comparison to modern web-scraped datasets built by filtering and deduplicating Common Crawl dumps. Following the same trend, we pre-trained NeoBERT on WefinedWeb (Penedo et al. 2023) a massive dataset containing 600B tokens, nearly 18 times larger than RoBERTa's. Although RefinedWel does not have strict high-quality constraints, we believe that exposing the model to such a large and diverse dataset will improve its real-world utility.
Sequence Length BERT and RoBERTa were pre-trained on sequences up to 512 tokens, which limits their downstream utility, especially without RoPF and YaRN. NomicBERT increased the maximum length to 2, 048 and employed Dynamic NTK interpolation at inference to scale to 8192. To further broaden NeoBERT's utility, we seek to increase the context length. However, due to the computational cost associated with pre-training,
adopt two-stage pre-training procedure similar to LLMs like LLaMA 3. In the first stage, we train the model for 1M steps (2T tokens) using sequences truncated to maximum length of 1, 024 tokens, referring to this version as NeoBERT 1024- In the second stage, we extend the training for an additional 50k steps (100B tokens), increasing the maximum sequence length to 4, 096 tokens. We refer to this final model as NeoBERT To ensure the model encounters longer sequences during this stage, we create two additional sub-datasets, Refinedwebuz4++ and Refinedwebguss+: containing only sequence lengths greater than 024 and 2, 048 tokens, respectively, alongside the original Refinedweb dataset. Each batch is sampled from Refinedweb, Refinedwebro24+ and Refinedwebauss+ with probabilities 20%, 40%, and 40%. Since longer sequences tend to represent more complex or academic content, this strategy helps mitigate the distribution shift typically observed when filtering for longer sequences. We explore the benefits of this two-stage training strategy in subsection 5.3
3.3 Pre- vrearrann
Encoder pre-training has received less attention than the data and architecture. However, many improve- ments made to decoders also apply to encoders. NeoBERT combines encoder-specific modifications with widely accepted decoder improvements.
Objective In light of RoBERTa's findings that dropping the next-senlence prediction task does not harm performance, both NomicBERT and NeoBERT were only pre-trained on masked language modeling. More- over, Wettig et al. (2023) challenged the assumption that the 15% masking rate of BERT and RoBERTa is universally optimal. Instead, their findings suggest that the optimal masking rate is actually 20% for base models and 40% for large models. Intuitively, a model learns best when the difficulty of its training tasks aligns with its capabilities. Based on their insight, we increase the masking rate 20%, while NomicBERT exceeds it by opting for 30%.
Optimization Following standard practice, we use the AdamW optimizer (Loshchilov & Hutter 2019) with the same hyperparameters as LLaMA 2: 81 0.9, B₂ 0.95, and 10 10-8. ". In preliminary experiments, we also considered SOAP (Vyas et al. 2025), recent extension of the Shampoo optimizer, but it failed to outperform Adam and AdamW and has been omitted from the list of ablations. We employ a linear warmup for 2, 000 steps to reach a peak learning rate of 6 x 10-4, followed by a cosine decay to 10% of the peak learning rate over 90% of the training steps. Once fully decayed, the learning rate remains constant for the last 100k steps at a sequence length of 1, 024 and 50k steps at a sequence length of 4, 096. We use a weight decay of 0.1 and apply gradient clipping with a maximum norm of 1.0.
Scale Recent generative models like the LLaMA family TTuvvoo et al., 2023 Dubey al. 2024 have demonstrated that language models benefit from being trained significantly more tokens than was previ- ously standard. Recently, LLaMA-3.2 1B was successfully trained on up to 9T tokens without showing signs of saturation. Morcover, encoders are less sample-efficiett than decoders since they only make predictions for masked lokens. Therefore, it is reasonable to believe that encoders of similar sizes can be trained on an equal or even greater number of tokens without saturating. For NeoBERT's pre-training, we use batch sizes of 2M tokens over 1M steps in the first stage and 50k steps in the second, resulting in a thcoretical total of 2.1T tokens. Note that because sequences are padded to the maximum length, this represents a theoretical number of tokens. In terms of tokens, this is comparable to RoBERTa and represents a 2x increase over
5

---

Published in Transactions on Machine Learning Research (05/2025)
Table 2: Modifications between successive ablations. The initial MO baseline corresponds to model similar
to BERT, while M9 corresponds to NcoBERT.
Modification
Before
After
Embedding Activation Pre-LN
Positional
RoPE
M1
GELU
SwiGLU
LayerNorm
RMSNorm
M2
Dataset
Wiki Book
RefinedWeb
M3
Tokenizer
Google WordPiece
LLaMA BPE
Optimizer
Adam
AdamW
M4
Scheduler
Linear
Cosine
M5
Masking Scheme
15% (8O / 10 10)
20% (100)
M6
Sequence packing
False
True
M7
Model Size
120M
250M
M8
Depth Width
16 1056
28 768
Batch size
131k
2M
M9
Context length
512
4, 096
NomicBERT, In terms of training stops, this amounts to a 2x increase over RoBERTa and a 10x increase over NomicBERT.
Efficiency We improve efficioncy by parallelizing the model across devices using DeepSpeed (Aminabadi et al. 2022) with the ZeRO (Rajbhandari et al.] 2020) optimizer, reducing memory usage by eliminating data duplication across GPUs and increasing the batch size. We further optimize the GPU utilization by employing fused operators from the xFormers library to reduce overhead, selecting all dimensions to be multiples of 64 to align with GPU architectures, and removing biases to simplify computations without sacrificing performance. To address the quadratic demands of attention, we integrate FlashAttention (Dao] 2023), which computes exact attention without storing the full matrices.
4 Effect of Design Choices
We conduct series of ablations in controlled scttings to evaluate our improvements to the original BERT architecture. We fully train each model for 1M steps, controlling for the seed and dataloader states to ensure successive models are trained with identical setups. These essurcceintennsee ablations were crucial to confirm our design choices, as they are based on the literature pre-training decoder models. The baseline model, referred as MO, similar BERT but includes pre-layer normalization. Following RoBERTa, MO also drops the next sentence prediction objective. We introduce modifications iteratively, resulting in total of ten different models, detailed in Table 2 To mitigate computational costs, the ablations are evaluated on the GLUE benchmark with limited hyperparameter grid scarch of batch sizes E (16,32) and learning rates E {le- - 5, 2e - 5.3e - 5}. For the final model M10, we extend the grid search, as detailed in Appendix C Results are in Figure
Key Perfornance-Bnaancing Modifications As expected, the two modifications that had the greatest impact on the average GLUE score were related to scale. In M2, replacing Wikitext and BookCorpus with the significantly larger and more diverse RefinedWeb datasct improved the score by +3.6%, while increasing the model size from 120M to 250M in M7 led to +2.9% relative improvement. Note that assess the impact of the wiith ratio, first scale the number of parameters in M7 to 250M while maintaining

---

Published in Transactions on Machine Learning Research (05/2025)
Figure GLUE ablation scores on the development set. All modifications are cumulative, except for M3 and M6 in grey, which not included in the subsequent models. Increasing data size and diversity leads to the highest relative improvement (M2, +3.6%), followed by the model size (M7, +2.9%). Packing the sequences and using the LLaMA tokenizer cause the largest relative drops (M6, 22.%, M3, 2.1%).
Baseline MD
79.2
RoPE, SwiGLU, RMSNorm
81.1
RefinedWeb M2
84.0
LLaMA Tokenizer M3
82.2
AdamW, Cosine Decay
83.5
20% Masking M
82.9
Sequence Packing M6
80.5
250M Parameters M7
85.3
Optimal Depth-to-Width
86.0
Training Scale MI
Larger Grid Search M10
89.0
0
10
20
30
40
50
60
70
80
90
similar ratio to BERT base, resulting in 16 layers of dimension 1056. In M8, the ratio is then adjusted to 28 layers of dimension 768.
Modifications That Were Discarded In M3, replacing the Google WordPiece tokenizer with LLaMA BPE results in a 2.1% performance decrease. The performance gap observed likely stems from inherent differences between the algorithms,, training corpus, and design strategies employed by the two tokenizers. We further discuss this in Appendix B In M6, we unpad the sequences by concatenating samples of the same batch. While this removes unnecessary computation on padding tokens, packing sequences without accounting for cross-sequence attention results in relative performance drop of -2.8%. Although this modification discarded from subsequent ablations, we incorporate methods of un-padding with accurate cross-attention in released codebase, following Kundu et al. (2024).
Modifications Retained Despite Performance Trade-offss Unexpectedly, using AdamW (Loshchilov & Hutter 2019) and cosine decay (Loshchilov & Hutter 2017] in M4 decreases performance by -0.5% As AdamW introduces additional regularization with weight decay, we expect that it will become beneficial when extending training by mitigating overfitting, Similarly, increasing the masking ratio from 15% to 20% in M5 decreases performance by -0.7%. We hypothesize that increasing the task difficulty initially hinders downstream task performance but is likely to become advantageous when training larger models on more tokens. Consequently, we retain both modifications despite being unable verify these hypotheses at scale due to the compulational costs.
5 Experiments
Selecting appropriate metrics and benchmarks is crucial for properly assessing the downstream performance and utility of language models. Following both early and recent studies, we include the GLUE and MTEB benchmarks in evaluations.

---

Published in Transactions on Machine Learning Research (05/2025)
5.1 GLUE
The GLUE benchmark (Wang al. 2019) cornerstone of language modcling evaluations and has played significant role in the field. Although GLUE now 6 years old and the community has long recognized its limitations, we report the GLUE score due to its widespread adoption and to facilitate the comparison of NeoBERT with existing encoders. Following standard practices, we fine-tune NeoBERT on the development set of GLUE with a classical hyperparameter search and introduce transfer learning between related lasks. Complete details of the fine-tuning and best hyperparameters are presented in Appendix C NeoBERT achieves score of 89.0% comparable to previous large models while being 100M to 150M parameters smaller. We present the results in Table 3
Table 3: GLUE scores on the development set. Baseline scores were retrieved as follows: BERT from Table of Devlin et al. (2019), RoBERTa from Table 8 of Liu et al. (2019), DeBERTa from Table 3 of He et al.
2023), NomicBERT from Table 2 of Nussbaum et al. (2024).
GTE from Table 13 of Zhang et al. 20024]
and ModeraBERT from Table 5 of Warner et al. (2024)
Size
Model
MNLI
QNLI
QQP
RTE
SST
MRPC
CoLA
STS
Avg.
BERT
84.0
90.5
71.2
66.4
93.5
88.9
52.1
85.8
79.6
Base
RoBERTa
87.6
92.8
91.9
78.7
94.8
90.2
63.6
91.2
86.4
(< 150M)
GTF-en-8192
86.7
91.9
88.8
84.8
93.3
92.1
57.0
90.2
85.6
NomicBERT 2048
86.0
92.0
92.0
82.0
93.0
88.0
50.0
90.0
84.0
ModernBERT
89.1
93.9
92.1
87.4
96.0
92.2
65.1
91.8
88.5
Medium 250M
NcoBERT 1024
88.9
93.9
90.7
91.0
95.8
93.4
64.8
92.1
88.8
NeoBERT
89.0
93.7
90.7
91.3
95.6
93.4
66.2
91.8
89.0
BERT
86.3
92.7
72.1
70.1
94.9
89.3
60.5
86.5
82.1
RoBERTa
90.2
94.7
92.2
86.6
96.4
90.9
68.0
92.4
88.9
Large
DeBERTaV3
91.9
96.0
93.0
92.7
96.9
91.9
75.3
93.0
91.4
(2 340M)
GTE-en-8194
89.2
93.9
89.2
88.1
95.1
93.5
60.4
91.4
87.6
ModernBERT
90.8
95.2
92.7
92.1
97.1
91.7
71.4
92.8
90.5
5.2 MTEB
Beyond the GLUE benchmark, we consider the more recent and challenging MTEB benchmark (Muennihhol et al., ) which has quickly become standard for evaluating embedding models, with wide coverage of tasks and 56 datasets in its English subset.
MTEB tasks rely the cosine similarity of embeddings pooled across tokens in sentence. The most mon and straightforward pooling strategy is computing the average of each token's final hidden representa- tion. However, because out-of-the-box encoders are trained with the masked language modeling objective, they provide no guarantee that mean embeddings will produce meaningful representations without further fine-tuning. As a result, models require expensive fine-tuning strategies crafted for MTEB to achieve strong scores. For instance, GTE (Li et al. 2023b) with multi-stage contrastive learning, Jina-cmbeddings (Sturua et al. 2024) with task-specific Low-Rank Adaptation (LoRA) adapters, and CDE (Morris & Rush 20224), with batch clustering and contextual corpus embeddings all pushed the limits of the leaderboard in their respective categories.
These coupled stages make it challenging to disentangle the respective impacts of pre-training and fine-tuning on the final model's performance. To isolate and fairly evaluate the improvements introduced during pre- training, we implemented an affordable, modcl-agnostic finc-tuning strategy based on classical contrastive learning. This fine-tuning approach was designed in accordance with established methods in the literature. Its controlled settings ensured that all models were fine-tuned and evaluated under identical conditions.
8

---

Published in Transactions on Machine Learning Research (05/2025)
5.2.1 Unified Contrastive Learning
Method Given a dataset of positive pairs D {4,, df Jimi i similarity metric s, a temperature parameter 7, and set of negative documents Ng for each query 4 the contrastive loss is defined as:
C - log es(g,dt//T + Ea EN, e=(g,d)1/=
Negative documents can be either generic samples of the same format or tailored hard negatives, which exhibit a high degree of similarity to the contrasted sample in their original representation. We constructed a dataset of positive query-document pairs with optional hard negatives based on open-source datasets previously employed in the litcrature (Li al. 2023a) for total of nine million documents. In addition the optional hard negatives, we also leverage in-batch, task- homogeneous negatives. In line with prior research [Li et al., 2023a), employ task-specific instructions and temperature-scaled sampling of the datasets. Complete details about the data, training, and evaluation be found in Appendix D
Results We found that training for more than 2,000 steps resulted in minimal performance gains. Table 4 presents the complete MTEB-EEglish evaluation of all fine-tuned models. Although NeoBERT is 100M parameters smaller than all large baselines, it is the best model overall with a +4.5% relative increase over the second best model, demonstrating the benefits of its architecture, data, and pre-training improvements.
Table 4: (eng v1) scores after 2,000 steps fine-tuning with our unified contrastive learning.
Class.
Clust.
PairClass.
Rerank.
Retriev.
STS
Summ.
Size
Model
Avg.
12 tasks
11 tasks
3 tasks
4 tasks
15 tasks
10 tasks
1 tasks
BERT
60.6
37.0
71.5
48.9
28.3
69.9
31.1
48.1
RoBERTa
58.7
36.7
71.2
49.8
26.9
71.8
29.1
47.7
Base
DeBERTaV?
45.9
15.2
39.0
42.2
25.0
26.9
NomicBERT
55.0
35.3
69.0
48.8
30.5
70.1
30.1
47.1
ModernBERT
58.9
38.1
63.8
48.5
21.0
66.2
30.1
45.0
Medium
NeoBERT 4006
61.6
40.8
76.2
51.2
31.6
74.8
30.7
51.3
BERT
59.8
39.3
70.9
49.7
29.6
71.4
31.2
49.1
RoBERTa
57.1
72.5
30.0
71.7
31.1
48.9
Large
DeBERTaV3
45.9
16.4
45.0
40.8
4.0
40.1
29.9
27.1
ModemnBRRT
62.4
38.7
65.5
50.1
23.1
68.3
27.8
46.9
5.2.2 Contextual Document Embeddings
While unified fine-tuning framework enables fair model comparisons, computationally intensive fine-tuning techniques are necessary to achieve top-ranking MTEB scores. We apply one such strategy, Contextual Document Embeddings (CDE) (Morris & Rush 2024), to NcoBERT. CDE previously led the MTEB leader- board with NonicBERT (cde-vl) and ModernBERT (cde-v2) for English models under 400M parameters. NeoBERT outperforms both, achieving an average score of 66.60, the highest in its parameter class. See Appendix G for more details on the fine-tuning.
Table 5: (ngg v1) scores after fine-tuning with CDE. *Ranks on MTEB v1) for models under 400M parameters of April 2025.
Class. 12 tasks
Clust.
PairClass.
Rerank.
Retriev. 15 tasks
STS
Summ. tasks
Model
Avg.
Rank
11 tasks
3 tasks
4 tasks
10 tasks
NomicBERT
81.72
48.32
84.69
56.75
53.27
81.61
31.23
65.00
ModernBERT
80.62
49.48
85.23
56.91
54.19
83.30
31.60
65.68
NeoBERT
82.14
50.29
86.71
57.71
56.37
82.30
30.98
66.60

---

Published in Transactions on Machine Learning Research (05/2025)
5.3 Sequence Length
Following previous literature, NeoBERT underwent additional 50k pre-training steps, during which it exposed to extended sequences of up to 4,096 tokens. To assess the impact of this additional training, we randomly sampled 2,467 long sequences from the English subset of Wikipedia. For each sequence, we independently masked each token at position and computed its cross-entropy loss, 4. The pseudo-perye perplexity of the entire sentence is then defined as P = exp 2/=44 14). We present the results in Figure
Figure 2: Pseudo-Perss Perplexity in function of the sequence length for NeoBERT 1024 (left) and NeoBERT (right). This validates the effectiveness of the final pre-training stage on NeoBERT's ability to model long sequences.
NeoBERT
NeoBERTare
17.5
17.5
15.0
15.0
Pseulo- Perplesit
Pseudo Parpleity
10.0
10.0
7.5
7.5
5.0
5.0
2.5
2.5
0.0
0.0
1000
1500
2000
2500
3000
35.00
4000
1000
1500
2000
2500
3000
3500
4000
Sequeace Length
Sequeace Length
1024 was trained exclusively on sequences of up to 1, .20 tokens, it generalizes effectively context lengths approaching 3,000 tokens. This demonstrates the robustness of RoPE embeddings to out- of-distribution inputs. Moreover, after an additional 50k training steps with sequences up to 4, 096 lokens, NeoBERT 4096 successfully models longer sequences. This approach provides compute-efficient strategy for extending the model's maximum context window beyond its original length. To further test NeoBERT's generalization capabilities, we extend the context beyond what was seen during training and present results in AAppendix
Although NeoBERT 1024
5.4 Efficiency
To assess model efficiency, we construct a synthetic dataset consisting of maximun-length sequences of sizes (512, 1024, 2048, 4096, 8192). For each sequence length, we scale the batch size from to 512 samples or until encountering out-of-memory errors. Inference is performed for 100 steps on a single A100 GPU, and we report the highest throughput achieved for each model and sequence length. Figure presents the results.
Due to their low parameter count and relatively simple architecture, BERT and RoBERTa the most effi- cient for sequences up to 512 tokens. However, their use of positional embeddings prevents them from further scaling the context window. For extended sequences, NcoBERT significantly outperforms ModernBERT,p... RRRTTssT despite having 100M more parameters, achieving a 46.7% speedup on sequences of 4, 096 tokens.
6 Discussion
Encoders are compact yet powerful tools for language underslanding and representation tasks. They require fewer parameters and significantly lower training costs compared to their decoder counterparts, making them strong alternatives for applications that do not involve text generation. Traditionally, the representational capacity of these models has been assessed through downstream tasks such classification, particular through the GLUE benchmark.
10

---

Published in Transactions on Machine Learning Research (05/2025)
Figure 3: Model throughput (tokens per second) as function of sequence length (1 better). Above 1, 024
in soquence length, NooBERT surpasses ModemBERT base despite having 100M more parameters.
BERT
RoBERTa
25k
Throughpout tokens/second
BERT lorge
20k
RoBERTa
15k
NeoBERT
10k
ModernBER's
5k
ModernBERT larg
512
1024
2048
4096
8192
While GLUE has played pivotal role in guiding model adoption, it includes only nine sequence classification datasets, four of which are entailment tasks. Moreover, its small dataset sizes and occasionally ambiguous labeling make it prone to overfitting, with models long surpassing human performance on the benchmark. Although DeBERTa-v3 achieves state-of-the-ar performance on GLUE by a significant margin, our fine- tuning experiments reveal its comparatively poor performance on the more recent MTEB benchmark. MTEB encompasses broader range of datasets and tasks, but attaining high performance on its leaderboard necessitates carefully crafted fine-tuning strategies with costly training requirements. As complex fine-tuning strategies emerge, it becomes unclear what the source of score improvements is. Moreover, these strategies are not casily reproducible or accessible, limiting the possibility of fair comparison between pre-trained backbones.
This underscores the limitations of current evaluation paradigms and highlights the need for more affordable and standarlized frameworks. We advocate for future research to focus on the development of standard- ized fine-tuning protocols and the exploration of new zero-shot evaluation methodologies ensure a more comprchensive and unbiased assessment of encoder-only models.
7 Conclusion
We introduced NeoBERT, state-of-theeart encoder pre-trained from scratch with the latest advancements in language modeling, architecture, and data selection. To ensure rigorous validation, we cetemattically evaluated every design choice by fully training and benchmarking ten distinct models in controlled settings. On GLUE, NcoBERT outperforms BERT large and NomicBERT and comparable with RoBERTatarge despite being 100M parameters smaller and supporting sequences eight times longer. To further validate its effectiveness, conducted comprehensive evaluation MTEB, carefully isolating the effects of pre- training and fine-tunis tne-tunnig to provide fair comparison against the best open-source embedding models. Under identical fine-tuning conditions, NcoBERT consistently outperforms all basclines. In addition, when fine- tuned with more advanced techniques, NeoBERT ranks at the lop of the MTEB leaderboard for models under 400M parameters. With its unparalleled efficiency, optimal depth-to- NcoBERT represents the next generation of encoder models. To foster transparency and collaboration, we release all code, data, model checkpoints, and training scripts, making NcoBERT the only fully open-source model of its kind.
and plug-ant and-play compatibility,
11

---

Published in Transactions on Machine Learning Research (05/2025)
Broader Impact Statement
Despite its improvements, NcoBERT inherits the biases and limitations its pre-training data. Morcover, the greatest jump in performance stems from the pre-training dataset, and as newer, larger, and more diverse datasets become available, retraining will likely be needed to further improve its performance. helesh NeoBERT stands today an affordable state-of-the-ard pre-trained encoder with great potential for down- stream applications.
Acknowledgements
Sarath Chandar is supported by the Canada CIFAR AT Chairs program, the Canada Research Chair in Lifelong Machine Learning, and the NSERC Discovery Grant. Quentin Fournier is supported by the Lambda research grant program. The authors acknowledge the computational resources provided by Mila and the Royal Military College of Canada.
References
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, et al. Deepspocd-inferenee: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-15. IEEE, 2022.
Simran Arora, Patrick Lewis, Angela Fan, et al. Reasoning over public and private data in retrieval- based systems, 2022. URL https://arxiv.. org/ a20/ 11027
Alexei Baevski, Sergey Edunov, Yinhan Liu, et al. Cloze-driven pretraining of self-atlention networks. Tn Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empèrical Methods in Natural Language Processing and the 9th International Joint Conference on Natural
Language Processing (EMNLP-IJCNLP), pp. 5360 5369, Hong Kong, China, November 2019. Association for Computational Linguistiss. doi: 10.18653 / 119 1539. URL
Payal Bajaj, Daniel Campos, Nick Craswell, et al. Ms marco: A human generated machine reading compre- hension dataset, 2018. URL https://arxiv..og/ abs/ 111111111.0000 09268
Samuel R. Bowman, Gabor Angeli, Christopher Potts, et al. A large annotated corpus for learning nal urau language inference, 2015. URL httpps://arxvvvor///aaa wwtp://////aaaaaaaoosooooossseaa hatp:/1118.....22..
Daniel Cer, Mona Diab, Eneko Agirre, et al. Semeval-2011 2017 task 1: Semantic textual similarity multilingual
and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemFFall20011). Association for Computational Linguistics, 2017. doi: 10. 10.18653/v1/s17-2000 doi URL http:://ex......org/....o.. http:////........111//111111111.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-cmbedding: Multi- lingual, multi-funetiionality multi-graullaitty text embeddings through self-knowledge distillation, 2024. URL http:://arriiv.org//b//22022
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, et al. Unsupervised cross-lingual representation learn- ing at scale, 2020. URL
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL
DeepSeck-AT, Daya Guo, Dejian Yang, et al. Deepscck-rl: Incentivizing reasoning capability in 1mms via reinforcement learning, 2025. URL https: htttps://ariiv org/abs/2503 22011. 12948
Jacob Devlin, Ming- Wei Chang, Kenton Loe, et al. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL
12

---

Published in Transactions on Machine Learning Research (05/2025)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. The Ilama 3 herd of models, 2024. URL https://arxivv.rr//aae
Katja Filippova and Yasemin Altun. Overcoming the lack of parallel data sentence compression, October 2013. URL http::///aallaatooooooooo///////
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings, 2022. URL http::///aaxxll..raabab////s////
08821
Dirk Groeneveld, Iz Beltagy, Pete Walsh, et al. Olmo: Accelerating the science of language models, 2024. URL https://arxiv..r org. http://wxiv..rr//ab/2/0000..0
Mansi Gupta, Nitish Kulkarni, Raghuvoer Chanda, et al. Amazonqa: review-based question answering Lask, 2019. URL
04364
Moonsu Han, Minki Kang, Hyunwoo Jung, et al. Episodic memory reader: Learning what to remember for question answering from streaming data, 2019. URL
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, et al. ToxiGen: large-scale machine- dataset for adversarial and implicit. hate speech detection. In Smaranda Muresan, Preslav Nakov, and Aline Villav- icencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 3309 3326, Dublin, Ireland, May 2022. Association for Computational Lin- 234
guistics. doi: 10.18653/ve /2t0///11/2222ee/// acl-oo..c-- 234. URL
Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre- Training with Gradiont-Disentangled Embedding Sharing, March 2023. URL http://arxiv.org.
abs/2111. 09543
arXiv:2111.09543 [cs].
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. Mistral 7b, 2023. URL https: //axxi//
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, et al. Pubmedqq: A dataset for biomedical research question answering, 2019. URL https://arxiv..rrgg wwtpe/iiwrrrrabs///eeeaaaa http:/aab1999...////1 06146
Daniel Khashabi, Amos Ng, Tushar Khot, al. Gooaq: Open question answering with diverse answer types, 2021. URL httth:h/tt//aattt http:/:://wxwwwo..//../// 2114201144.002
Mahnaz Koupace and William Yang Wang. Wikihow: large scale text summarization dataset, 2018. URL
http:://arraiv.or//bb///// 09305
Achintya Kundu, Rhui Dih Lee, Laura Wynter, et al. Enhancing training efficiency using packing with flash attention, 2024. URL
http:/abb2220099990 09105
Yoav Levine, Noam Wies, Or Sharir, et al. Limits to Depth Efficiencies of Self-Attention. In Advances in Newral Information Processing Systems, volume 33, pp. 22640 22651, Curran Associates, Inc., 2020. URL
Xianming Li and Jing Li. Angle-optimizel text embeddings. arXiv preprint arXin:2900.1288 12871, 2023.
Zehan Li, Xin Zhang, Yanzhao Zhang, et al. Towards general text embeddings with multi-stage contrastive learning, 2023a. URL https://arxi http://awwwa.rr//a////////////// 03281
Zehan Li, Xin Zhang, Yanzhao Zhang, et al. Towards General Text Embeddings with Multi-stage Contrastive Learning, August 2023b. URL http://arxiv http:://aaravvvoora//////00//// 2w0.23000000011111 ArXiv:2308.03255 [cs).
Yinhan Liu, Myle Ott, Naman Goyal, et al. Roberta: A robustly optimized bert protraining approach, 2019.
URL https://arxivv www.aaawwwv..iraaaaaaa http:///////../1/eere///e 11692
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2017. URL
13

---

Published in Transactions on Machine Learning Research (05/2025)
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.orrg 3bs/1711.00
Rui Meng, Ye Liu, Shafiq Rayhan Joty, et al. Sfr-embedding 2: Advanced text embedding with multi-stage training, 2024. URL
htt.://gwgggggaac..oo//aaaaeeeee Eneeddig--22n551
John X. Morris and Alexander M. Rush. Contextual document embeddings, 2024. URL https://arxix:
hrg/abs/2410 2ttpp//241..022222.... 02525
Niklas Muennighoff, Nouamane Tazi, Loic Magne, et al. MTEB: Massive Text Embedding Benchmark, March 2023. URL http:://rxxv..rrgg//////// wwwp:///wwwggaaa///.../.o 07316
arXiv:2210.03311 [cs].
Zach Nussbaum, John X. Morris, Brandon Duderstadt, et al. Nomic embed: Training reproducible long context text embedder, 2024. URL https://arxiv. org/ abs/ 01613
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa : A large-scale multi-
subject multi-choice dataset for medical domain question answering, 2022. URL https://arxiv.org
abs/2203. 14371
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, et al. The refinedweb dataset for falcon Ilm: Outper- forming curated corpora with web data, and web data only, 2023. URL https://aaxii.co org/ /abb//336..
01116
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, et al. Yarn: Efficient context window extension of large language models, 2023. URL https://arxiv.org/ wwwp://////aa////e//ee /2009 00071
Alec Radford, Karthik Narasimhan, Tim Salimans, et al. Improving language understanding by generative pre-training, 2018.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, et al. Zero: Memory optimizations toward training trillion parameter models, 2020. URL https://arrdvvorr//b////0...... 02054
Ori Ram, Yoav Levine, Itay Dalmedigos, et al. In-context rotrieval-augmmented language modcls, 2023. URL https://arxiv wwtpoba//302..000../.00
httpss//aarii.oorr/a////222000
Noam Shazoer. Glu variants improve transformer, 2020. URL
Aivin V. Solatorio. Gistembed: Guided in-sample selection of training negatives for text embedding fine- tuning, 2024. URL httpgsss//arrii.oog////////....
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, et al. jina-embeeddings-v3. Multilingual embeddings with task lora, 2024. URL https:://taatt http://xxxvv..vv www.:rr//abb//00000/ 10173
Jianlin Su, Yu Lu, Shengfeng Pan, al. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxxv..rrg ab///104./2.2...
James Thorne, Andreas Vlachos, Christos Christoodoulopoulls,, et al. Fever: a large-scale dataset for fact https///833000000 05355
extraction and verification, 2018. URL
Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open foundation and fine-tuned chat, models, 2023. http:////ardddd.o////b/////// 09288
Nikhil Vyas, Depen Morwani, Rosie Zhao, et al. Soap: Improving and stabilizing shampoo using adam, 2025. URL RRRL://csss//rrr..oor//a////// 11321
Alex Wang, Amanpreet Singh, Julian Michael, et al. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. URL
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024. URL
http:://aaxivv.orrraa///44..1. 13663
14

---

Published in Transactions on Machine Learning Research (05/2025)
Alexander Wettig, Tianyu Gao, Zexuan Zhong, et al. Should You Mask 15% in Masked Language Modeling?? February 2023. URL
08005 arXiv:2202.08005 [os].
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference, 2018. URL http:://aarrdv.oogaaa///00///0
Ruibin Xiong, Yunchang Yang, Di He, et al. On layer normalization in the transformer architecture, 2020. URL
04745
Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019. URL https://arxiv.org/
0bs/1990.07495
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification, 2016. URL https://arxiv.... httt::://arx.v.org//ab/////////e 01626
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, et mGTE: Generalized long-context lext representation and reranking models for multilingual lext retrieval. In Franck Dernoncourt, Daniel Preotiuc-Pietre and Anastasia Shimorina (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 1393-1412, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10. //aclanthology.orgg
URL https:
emnlp-industen htsp-innuutrry130//
Yukun Zhu, Ryan Kiros, Richard Zemel, al. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, 2015. URL https:://///// //apxiiv org.orggg/aa.. 110.:/106000000 06724
15

---

Published in Transactions on Machine Learning Research (05/2025)
A Training details
NeoBERT was trained 8 H100 for 1,050,000 steps, for total of 6,000 GPU hours. In the first stage of training, we used a local batch size of 32, 8 gradient accumulation steps, and a maximum sequence length of 02024, for a total batch size of 2M tokens. In the second stage of training, we keep the theoretical batch size constant and increase the maximum sequence length to 4, 096.
B Ablations
B.1 Baseline
Our first model, MO modeled after BERT base in terms of architecture. The only two differences are the absence of the weteeeeeeeeeeeeeeitttie objective, as well as Pre-Layer Normalization. Each successive model, up until M8 is identical the previous one on every point except for the change reported Table
B.2 Tokenizers
We refer to the Llama Wordpicce tokenizer as Ilama-tok and the BERT BPE tokenizer as bert-tok. Under identical conditions between ablations .22 and M3, the use Ilama-tok over bert-tok decreases the GLUE score by a relative drop of 2.9%. Although further investigation would be required analyze this drop, we highlight several key differences between the two tokenizers.
While llama-tok uses Byte-Pair Encoding (BPE), bert-tok relies the more expressive WordPiece algorithm, which better models token likelihoods in text. Moreover, their training data largely differs. llama-tok was trained on & dataset mostly composed of the CommonCrawl, whereas bert-tok used the higher-quality BookCorpus and WikiText datasets. Finally, Ilama-tok is cased and handles unknown UTF-8 characters at the byte level, while bert-tok uncased and character-level. This leads llama-tok to include redundant tokens (c.g., "The" and "thc") and rare byte soquences, limiting space for more frequent tokens under a finite capacity.
C GLUE
We perform classical parameter search with learning rates in (5e 6, 6e 6, 5, -- 5, sizes in {4, 16, 32} and weight decay in {le- 2, 5). In addition, we start training from the best MNLI checkpoint for RTE, STS, MRPC, and QNLI
5), batch
We fine-tune on the training splits of every glue dataset for 10 epochs, with evaluation on the validation splits every n steps, being defined as min(500, len(dataloader) // 10) with early stopping after 15 evaluation cycles if scores have not improved. Following BERT, we exclude WNLI from our evaluation two scores and for MNLI matched and mismatched, we report the average between the two metrics.
o For tasks with
MTEB
D.1 Evaluation of pre-trained models
As demonstrated in Figure 4 evaluating out-of-the-box pre-trained models on MTEB is inconclusive. that setting, BERT hase out perforrm both BERTharge and RoBERTatarge highlighting the importance of fine-tuning ensure representative evaluation on the MTEB benchmark.
D.2 Contrastive learning
Following the existing literature, we designed a simple fine-tuning strategy entirely agnostic to the models evaluated. We used cosine similarity and 0.07 as temperature paramoter in the contrastive learning
SSee 12 https///gieebendnmmakccoom////
16

---

Published in Transactions on Machine Learning Research (05/2025)
Model
Task
Batch Size
Learning Rate
Weight Decay
CoLA
6e-6
le-5
MNLI
16
6e-6
le-2
MRPC
8
2e-5
le-5
QNLI
5o-6
1c-5
NeoBERT 1024
QQP
32
1e-5
le-2
RTE
8
6e-6
1e-5
SST-2
16
1e-5
le-5
STS-B
8
1c-5
1e-2
CoLA
8
8c-6
1e-5
MNLI
16
5e-6
le-5
MRPC
le-5
le-5
QNLI
8
5c-6
1e-5
NeoBERT 4096
QQP
32
8e-6
le-5
RTE
32
5e-6
le-5
SST-2
32
8e-6
1e-2
STS-B
32
2e-5
1e-5
Table 6: Optimal hyperparamoters for GLUE tasks. The grid search was conducted over batch sizes
(2, 4, 8, 16, 32), learning rales (5e-6,6e 8B 6,-6, le-5, 2, 3e- and weight decay values (le-2, 5e-5).
Figure Zero-shot evaluation of BERT and RoBERTa on the English subset of MTEB.
40
MTEB-en Average Score
38
37.4
34
36.0
32
33.5
30
BERT Muse
BERT Aarge
RoBERTa auuse
RoBERTa a
alorge
loss. Additionally, we sampled datasets with multinomial distribution based their sizes (nz) with 0.5:
T2 Σ/ n
We trained on the following fully-open datasets: AG-News (hang al.] 2016), AII-NI Bowman et al.
2015 Williams et al. 2018), AmazonQA (Gupta et al. 2019). Concurrent (Arora al. 2022), GitHub
17

---

Published in Transactions on Machine Learning Research (05/2025)
Issues
Li & Li 2023), GooAQ [Kashabi et al. 2021].
McdMCQA (Pal et al. 2022 NPR PudMedQA (Jin
et al.
2019). SentenceComppeesion (Filippova & Altun
2013) Stack _ TriviaQA (Han et al. 2019)
Wikihow (Koupaee & Wang 2018). Yahoo! Answers
Zhang et al. 2016 as well as the available training
splits of MTEB datascts
Fever (Thorne et al. 2018 MS MARCO (Baja.
et al.
2018 STS12, and STSBenchmark (Cer al.
2017)).
We fine-tune every model for 2,000 steps and evaluate on MTEE in float16. presented in Table 4
The complete results are
Figure 5: Average MTEB scores of fine-tuned encoders grouped by task type. The average score is computed across the 56 tasks of MTEB-English. NeoBERT is the best model on five out of seven task lypes and the best model overall. See Table for complete scores.
DaBERTa-3
ModernBERTur
NomicBERT
RaBERT
BERT
NaoBERT
80
299
Arciroa (%)
60
65
P4
40
ESS
STTE
lorg.
Clustering
Beranking
D.3 Task instructions
We provide the set of instructions used for fine-tuning in Table 7 and evaluation in Table and Table 9
18

---

Published in Transactions on Machine Learning Research (05/2025)
Table 7: Instructions for fine-tuning on the different contrastive learning asets...
Dataset
Instruction
AGNEWS
Given news title, retrieve relevant articles.
ALLNLI
Given premise, retrieve hypothosis that is entailed by the premisc.
AMAZONQA
Given a question, retrieve Amazon posts that answer the question.
CONCURRENTQA
Given a multi-hop question, retrieve documents that can help answer the question.
FEVER
Given claim, retrieve documents that support or refute the claim.
GITHUBISSUE
Given question, retrieve questions from Github that duplicates to the given question.
GOOAQ
Given question, retrieve relevant documents that best answer the question. Given medical question, retrieve relevant passages that answer the question. Identify the main category of medical exams based on their questions and answers.
MEDMCQA
MEDMCQAGLUST
MSMARCO
Given web search query, retrieve relevant passages that answer the query. Given news title, retrieve relevant articles.
NPR
PAQ
Given question, retrieve Wikipedia passages that answer the question. Given medical question, relrieve docunents that best answer the question.
PUBMEDQA
QQP
Given question, retrieve questions from Quora forum that are semantically equivalent to the given question.
SENTENCECOMP
Given a sentence, retrieve semantically equivalent summarics. Given a Stack Exchange post, retrieve posts that are duplicates the given post. Retrieve duplicate questions from StackOverflow forum. Retrieve semantically similar text.
STACKEXCHANGE
STACKOVERFLOW
STS12
STSBENCHMARK
Retrieve semantically similar text.
TRIVTAQA
Given question, retrieve documents that answer the question. Given a Wikihow post, retrieve titles that best sunmarize the post. Identify the main topic of Yahoo posts based on their titles and answers.
WIKIHOW
YAHOOCLUST
Table Instructions for evaluation on the different MTEB tasks.
Task name
Instruction
DBPedia
Given query, retrieve relevant entity descriptions from DBPedia.
FEVER
Given claim, retrieve documents that support refute the claim.
FiQA2018
Given financial question, retrieve user replies that best answer the question.
HotpotQA
Given a multi-hop question, retrieve documents that can help answer the question.
MSMARCO
Given a web search query, retrieve relovant passages that answer the query.
NFCorpus
Given question, retrieve relevant documents that best answer the question. Given question, retrieve Wikipedia passages that answer the question. Given question, retrieve questions that are semantically equivalent to the given question.
NQ
QuoraRetrieval
SCIDOCS
Given scientific paper title, retrieve paper abstracts that are cited by the given paper.
SciFact
Given a scientific claim, retrieve documents that support or refute the claim. Given a question, retrieve detailed and persuasive arguments that answer the question.
Touche2020
TRECCOVID
Given query on COVID-19, retrieve documents that answer the query. Retrieve semantically similar text.
SICK-R
STS
Retrieve semantically similar text. Retrieve semantically similar text from the biomedical field.
BIOSSES
SummEval
Given à news summary, retrieve other semantically similar summaries.
19

---

Published in Transactions on Machine Learning Research (05/2025)
Table 9: Instructions for evaluation on the different MTEB tasks.
Task name
Instruction
AmazonConntoriictuuuClisss
Given an Amazon customer revicw, classify it as cither counterfactual or not-counterfcttual
AmazonPolarityClss
Given an Amazon review, classify its main sentiment into positive or negative.
AmazznReviewsClass.
Given Amazon review, classify it into its appropriate rating calegory.
Banking77Class.
Given online banking query, find the corresponding intents. Given a Twitter message, classify the emotion expressed into one of the six emotions: anger, fear, joy, love, sadness, and surprise.
GmotionClass.
ImdbClass.
Given an IMDB movie review, classify sentiment into positive or negative.
Massivelntent
Given utterance, find the intents.
MassiveScenarriollss
Given utterance, find the user scenarios.
MTOPDomain Class.
Given user utterance, classify the domain in task-oriented conversation.
MTOPIntentClass.
Given a user utterance, classify the intent in task-oriented
conversation.
ToxieConverraationnCllss
Given comments, classify them as either toxic or toxic.
TveesSeniinen//tttaaiinnCCCC
Given tweet, classify its sentiment either positive, negative, neutral.
<datase(>Clusterrngg]
Identify the main and secondary category of <dataset>> papers based on their titles and abstracts.
(attaset>Chsteerinng25
Tdentify the main and secondary category of <dataset>> papers based on their titles.
dataset>Clusterrin
Identify the topic theme of <dataset>> posts based on their titles. Identify the topic theme of the given news articles. Retrieve duplicate questions from Sprint forum.
T wwuutyyerssrrouusChhseeinnn
SprintDuplicatequuuttoos
TwilleerSenEvll2015
Given tweet, retrieve tweets that are semantically similar. Given tweet, retrieve tweets that are semantically similar. Retrieve duplicate questions from AskUbuntu forum. Given a user browsing history, retrieve relevant news articles. Given a title of a scientific paper, retrieve the relevant papers.
URitterURLLCrrpus
Qussstos
MindSmallRerankinng
RRiDocsRR
StackOveefoowDDuqqustits
Retrieve duplicate questions from StackOverflow forum. Given claim, find documents that refute the claim. Document Given claim about climate change, retrieve documents that support or refute the claim.
ArguAna
ClimateFEVER
CQADupstackRetrieval
Given a question, retrieve detailed question descriptions from Stackexchange that are duplicates to the given question.
20

---

Published in Transactions on Machine Learning Research (05/2025)
E Context Length
We compute the Psendo-Perplexity sequences sampled from English Wikipedia. Although NcoBERT has only been exposed to sequences up to 4,096 tokens during training, it correctly models sequences up to 6,000 tokens. Additional training or RoPE extension techniques, such as YaRN (Peng et al. 2023). would be needed further extend the context window.
Figure 6: Perple--er in function of the sequence length for NeoBERT 1096 for extension beyond sequence lengths seen training. NcoBERT generalizes natively for sequences under 6,000 tokens.
20.0
NeoBERT 4096
17.5
15.0
Pseudo-Pen PheudoPerplexity Co-Perplexity
12.5
10.0
7.5
5.0
2.5
0.0
1000
2000
3000
4000
5000
6000
7000
Sequence Length
F
Efficiency
Table 10 presents the complete results of model efficiency evaluations.
Table 10: Throughput (10 tokens / second) in function of the sequence length, with optimal batch size.
Size
Model
512
1024
2048
4096
8192
BERT hase
27.6 = 3.6
Base
RoBERTapse ModamBERT base
24.9 = 3.0
25.4 2.3
22.6 2.7
17.2 1.7
11.7 0.8
6.8 - 0.2
Medium
NeoBERT
24.5 1.4
22.2 1.7
20.5 = 1.6
17.2 1.2
13.0
BERT MaRgee
19.5 0.6
Large
RoBERTatarge
15.9 0.3
ModernBERT (arT
13.4 0.2
11.4 1.1
9.2 0.7
6.5 0.3
3.8 - 0.1
G
Contextual Document Embeddings
One popular application for BERT-style models is to generate embedding vectors for use in tasks such classification, clustering, and retrieval. State-of-the-art embedding models below 400M parameters
21

---

Published in Transactions on Machine Learning Research (05/2025)
trained using contextual embedding techniques (Morris & Rush 2024p. The nighest-performinn similar-sized embedding models were trained with backbones initialized with NomicBERT (Nussbaum et al. 2024 and more recently ModernBERT (Warner et al. 2024). In this section, we question whether contextual document embeddings (CDE) might perform better when initialized from NcoBERT.
G.1 Method
We train our embodding models using both the contextual batching strategy and contextual architecture described in Morris Rush [0024].
Contextual batching. To train a contextual embedding model, have to first group documents from larger corpus into miniature 'pseudo-contenss contexts' which comprise the batches for training. We do by minimizing the following batch-clustering objective:
E
E M((d, c).
min .... ... (d.y)EB'
Here, BB...B represent the orderings of batches of training data and ex are the corresponding centroids; (d. q) indicates the distance (typically Euclidean) between embeddings of document d and query q from batch B
We can approximate this efficiently using a modified K-Means solver. We follow the same procedure Morris & Rush (0024) to cluster documents for each training set.
Contextual architecture. Contextual embedding models in effect use two model backbones: one to embed similar documents in context and another to embed documents conditioned on contextual information. We can express contextual embedding vector & using both models M1 and My:
ox; D) = (M) ... F(dd)) F(a.)...... F(ET))
where is document or query comprised of tokens .r...rr and D is set of documents from the surrounding context. We train & end-to-end by backpropagating through both M1 and M2 using the custom two-stage gradient-caching procedure described in Morris & Rush (2024).
G.2 Data and hyperparameters
Training stages and data. We train our models in two stages. The first stage is a large, noisy rastiva pre-training" stage with 235M query-to-document pairs per epoch. The second is short fine-tuning step on 1.5M high-quality datapoints. We train each stage for three epochs with the Adam optimizer with 1000 steps of warmmp to a learning rate of 5e 5 and linear decay to 0 throughout training.
Instead of mining hard negatives, we cluster the datasets for contextual batching. We embed for clustering and filter hard negatives within a batch using GTE-large (Li et. al.) 2023a). We set both the batch size and cluster size to 512. Unlike many contrastive training setups, neglect to share negatives between GPUs.
We evaluate our contextual embedding models on the 56 tasks of the English version of MTEB (Muennighoff et al. 2023). To maintain consistency with prior iterations of CDE, we train these models with 512 in-conlext documents and a per-denument sequence length of 512; this produces a second-stage input size of 1024 tokens.
G.3 Results
Our contextual embedding results are detailed in Table 5 averaged across tasks in MTEB. With a score of 66.60, NcoBERT significantly outperforms Moder1BERT enbedding-rolated tasks; our case, the gap between NeoBERT and ModemBBRR (about 1 MTEB point) is notably larger than the gap between ModernBERT and NomicBERT (approximately 0.6).
22

---

Published in Transactions on Machine Learning Research (05/2025)
NeoBERT's largest improvements are shown in retrieval, reranking, clustering, and classification tasks. We see small drop in performance in STS compared to ModernBERT, which we hypothesize may relate to the mix of pretraining data.
When compared to other models in MTEB, NeoBERT fine-tuned with CDE is state-of- state-of-the-art in its param- eter class, outperforming all other similar-sized embedding models as of April 2025, such as GTE with 64.36 Li al. 2023a BGE with 63.99 (Chen et al. 2024), and GIST with 64.13 (olatorio 2024
23


Reviews:
[{"review_id": "LFk5huk65U", "reviewer": "Reviewer_3wjq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper includes extensive ablations by fully training 10 different models, which is computationally expensive but provides valuable insights into the impact of each design choice.\n- By releasing all code, data, and model checkpoints, the authors make a significant contribution to the research community and enhance reproducibility.\n- The authors balance performance improvements with practical considerations like inference speed and computational efficiency, making the model more accessible for real-world applications.\n\n### Weaknesses\n\n- The authors use AdamW with cosine decay rather than the WSD scheduler used in ModernBERT and other recent LLMs. The rationale for this choice isn't fully explained. Since the authors will release their checkpoints, using WSD could potentially benefit users who want to fine-tune the model on their own data mixes by providing better annealing capabilities.\n- While the authors note that using the LLaMA BPE tokenizer resulted in a 2.1% performance decrease, there isn't a deeper investigation into why this occurred.\n- Given that RoPE theoretically allows for better extrapolation to longer sequences, more detailed experiments on zero-shot context length extrapolation would have strengthened the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces NeoBERT, a next-generation encoder model designed to bridge the gap between modern autoregressive language models and traditional encoders like BERT and RoBERTa. The authors address the fact that while decoders have seen significant advances in recent years, encoder architectures have remained relatively static despite their critical importance for downstream NLP tasks.\nThe key contributions include:\n\n- A comprehensive redesign of the encoder architecture incorporating modern techniques such as RoPE positional embeddings, optimal depth-to-width ratio, SwiGLU activation, and Pre-RMSNorm.\n- Training on RefinedWeb, a modern large-scale dataset (2.8TB), with an extended context window of 4,096 tokens.\n- Extensive ablation studies conducted by fully pre-training multiple models with different configurations, allowing for precise isolation of the impact of each modification.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Provide more justification for using AdamW with cosine decay instead of WSD, which has become standard in modern language models including ModernBERT. Was this an empirical choice, or were there other considerations?\n- Include more detailed experiments on RoPE's extrapolation capabilities, especially since this is one of RoPE's key advantages. While Figure 2 shows some analysis of sequence length modeling, specific extrapolation experiments would enhance the contribution.\n- Provide more information about the training infrastructure and costs to help other researchers better understand the resources required to replicate or build upon this work.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "irDfUiHgwM", "reviewer": "Reviewer_r7sv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro: \n\n- It's good that this paper releases the dataset, unlike modernBERT, which does not include data release—a missed opportunity.\n- The experiment is extensive and effectively isolates different factors for analysis.\n\nCon:\n\n- Why can't the support context length be as large as modernBERT?\n- What is the benefit of two-stage training? Why not train directly on a 4K context length?\n- How about using the Muon Operator instead of AdamW?\n- Figure 1 is confusing—is it cumulative or separate between M0 and M10?\n- In Figure 6, I’m curious how far the network can generalize. Can you show results for 10K or even 20K context length?\n- Can you also include modernBERT in Figure 3?\n\nOne additional question is about the number of parameters. Nowadays, autoregressive models like LLaMA can be trained with around 8 billion parameters, while most BERT models are only about 500 million. Is this a common convention? Why didn’t the author consider training a larger model with more parameters and a bigger dataset? Would BERT’s performance continue to scale with an increasing number of parameters? I would appreciate it if the author could elaborate on this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces NeoBERT, a next-generation encoder designed to enhance bidirectional models by incorporating state-of-the-art architectural advancements, modern data, and optimized pre-training. Despite having only 250M parameters, NeoBERT outperforms BERT, RoBERTa, NomicBERT, and ModernBERT on the MTEB benchmark while maintaining seamless compatibility as a plug-and-play replacement. To encourage further research and adoption, this paper provides a comprehensive evaluation framework, analyzes performance on GLUE, and releases all code, data, and training resources.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "as above", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0JhTC9GPja", "reviewer": "Reviewer_N8Qd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nBetter encoder models are useful in both academia and industry settings. This paper delivers a strong but compact Transformer-based encoder model with long context window and high efficiency (high throughput), and thus is a solid contribution to the community.\n\nThe paper is clearly written.\n\nExtensive experiments are carried out, and results are convincing.\n\nThe full training recipe is released, and thus it facilitates open-source research in this field.\n\n**Weaknesses**\n\nThe main weakness of this work is the lack of discussion about why it underperforms certain models of comparable sizes on MTEB; see MTEB Leaderboard for those models: https://huggingface.co/spaces/mteb/leaderboard\n\nSince the main contribution of this paper is a trained model with its recipe, but not a novel training method or so, I think that we need to see stronger results over competitors of comparable sizes, or an in-depth discussion about why it underperforms them, hopefully with a discussion about why this model is still better than those in certain ways.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new bidirectional Transformer-based encoder model, NeoBERT, and its full training recipe. The model is easy to plug-and-play and can replace other encoder models in deployment. \nThe training recipe of the model benefits from careful designs such as data curation, optimal depth-to-width ratio, and context length extension, thus achieving better performance and efficiency.\nIts full training recipe documents extensive details for people to reproduce the model.\nThe authors also carried out ablation studies to assess the impact of each key component of the recipe.\nAll code, data, checkpoints, and training scripts are publicly released, facilitating adoption in both academic and industrial settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "What will significantly strengthen this paper is an in-depth discussion about why NeoBERT underperforms models of comparable sizes (e.g., bilingual-embedding-base, multilingual-e5-base) on MTEB Leaderboard.", "broader_impact_concerns": "No concerns on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces NeoBERT, a next-generation encoder model designed to bridge the gap between modern autoregressive language models and traditional encoders like BERT and RoBERTa. After author rebuttal, it received 1 Accept and 2 Leaning Accept recommendations. Reviewers are generally happy about the paper, commenting that NeoBERT is a well-executed and timely contribution that modernizes BERT-style encoders with state-of-the-art design, extensive ablations, and strong open-source commitment. The authors address all concerns with additional experiments and achieve top-tier performance under fair evaluation. The author response clearly explains why their model underperforms models of comparable sizes, along with new results to show continued improvements. 

Overall, given all the positive feedback and the timely contributions, the Action Editor would like to recommend acceptance of the paper.
Decision: Accept (poster)