Seq vs Seq: An Open Suite of Paired Encoders and Decoders
Authors: Orion Weller
Kathryn Ricci
Marc Marone
Antoine Chaffin
Dawn Lawrie
Benjamin Van Durme
Venue: ICLR 2026 Poster (2026)
Keywords: encoders
pretraining
objective
mlm
ntp
retrieval
Primary area: foundation or frontier models, including LLMs
TL;DR: We train encoders and decoders on identical data and compare the architectures for various tasks, beating ModernBERT and Llama 3.2 with open-data

Abstract: The large language model (LLM) community focuses almost exclusively on decoder-only language models, since they are easier to use for text generation. However, a large subset of the community still uses encoder-only models for tasks such as classification or retrieval. Previous work has attempted to compare these architectures, but is forced to make comparisons with models that have different numbers of parameters, training techniques, and datasets. We introduce the SOTA open-data Ettin suite of models: paired encoder-only and decoder-only models ranging from 17 million parameters to 1 billion, trained on up to 2 trillion tokens. Using the same recipe for both encoder-only and decoder-only models produces SOTA recipes in both categories for their respective sizes, beating ModernBERT as an encoder and Llama 3.2 and SmolLM2 as decoders. Like previous work, we find that encoder-only models excel at classification and retrieval tasks while decoders excel at generative tasks. However, we show that adapting a decoder model to encoder tasks (and vice versa) through continued training is subpar compared to using only the reverse objective (i.e. a 400M encoder outperforms a 1B decoder on MNLI, and vice versa for generative tasks). We open-source all artifacts of this study including training data, training order segmented by checkpoint, and 200+ checkpoints to allow future work to analyze or extend all aspects of training.

Under review as a conference paper at ICLR 2026
000
SEQ VS SEQ: AN OPEN SUITE OF PAIRED ENCODERS AND DECODERS
001
002
003
004
005
Anonymout authors
Paper under double-blind review
006
007
008
009
ABSTRACT
010
011
The large language model (LLM) community focuses almost exclusively on decoder-only language models, since they are easier to use for text generation. However, a large subset of the community still uses encoder-only models for tasks such as classification retrieval. Previous work has attempted to compare these architectures, but is forced to make comparisons with models that have different numbers of parameters, training techniques, and datasets. We introduce the SOTA open-data ETTIN' suite of models: paired encoder-only and decoder-on), models ranging from 17 million parameters to billion, trained on up to 2 trillion tokens. Using the same recipe for both encoder-only and decoder-only models produces SOTA recipes in both categories for their respective sizes, beating ModemBERR as encoder and Llama 3.2 and SmoILM2 as decoders. Like previous work, we find that encoder-only models excel at classification and retrieval tasks while decoders excel at generative tasks. However, we show that adapting a decoder model to encoder tasks (and vice versa) through continued training is subpar compared to using only the reverse objective (i.e. 400M encoder outperforms 1B decoder on MNLI, and vice versa for generative tasks). We open-source all artifacts of this study including training data, training order segmented by checkpoint, and 200+ checkpoints to allow future work to analyze or extend all aspects of training.
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
INTRODUCTION
030
031
The rise of neural language models (LMs) was spurred by encoder-only models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019). However, the community generally shifted to decoder-only (i.e. GPT-style, ala Brown et al. (2020)) models due to their exceptional performance at sequence generation. Due to this lack of popularity for encoder-only models there was limited new model development, thus, still frequently see usage of older models (i.e. from 2019) by the subset or fast on-device inference. Although nascent
032
033
034
035
036
of the community focused on retrieval/Classifccatie work is attempting to revive encoder-only development (Samuel, 2024; Warner et al., 2024; Lee et al., 2025), there still exists wide gap between the development of encoder-only and decoder-only models (synonymously referred to in this work as encoders or decoders).
037
038
039
040
Part of this gap is due to the sentiment within the community that decoders can be adapted for use in tasks that were once predominantly focused (e.g. classification, embeddings), especially as they can often be used in a zero-shot fashion (i.e. without fine-tuning). As decoder models are more studied, more over-trained (Hoffmann et al., 2022), and are generally larger, they are now claiming the top spots of leaderboards for previously encoder-centric tasks (Enevoldsen et al., 2025).
041
042
043
044
045
Many works have challenged this assumption by comparing encoder-only and decoder-only models of roughly the same sizes (Ethayarajh,, 2019; Charpentier & Samuel, 2024; Harrag et al., 2021). However, these analyses have to be done with incomparable models: using different architectures, different pre- pre-training data, different learning schedules, etc.
046
047
048
049
Our work aims to provide the foundation to compare encoder-only and decoder-only models by open-sourcing a suite of models trained with the same data, the same architecture, and the same training recipe. Our ETTIN suite contains 10 models (5 pairs) ranging from 17 million to 1 billion
050
051
052
053
Named for the two-hcaded mythological Norse giant, symbolizing the two language models heads. Models, code, and data are available <removed for anonymity>

---

Under review as a conference paper at ICLR 2026
054
parameters, and trained for up to 2 trillion tokens. This allows to quantify the differences between these models (including the effects of scaling parameter size) in an ayples-toapplee comparison.
055
056
Our models provide state-of-the-arn performance for their size among open-data models. Surprisingly, they do so in both encoder settings (w...t. ModemBBERT) and decoder settings (w.r.t. LLaMA 3.2 and SmolLM2) despite using the same recipe. Notably, our work also provides the first open-data replication of ModernBERT, allowing the community to further build upon our recipe.
057
058
059
060
061 We find that, like previous work, encoders excel at classification and retrieval while decoders excel at generative tasks. However, we go beyond previous work to examine the increasingly common setting (BehnamGGhader et al., 2024) where decoder-model,
062
continued trained as encoders (i.e. cross-obeective training). We show results for this cross- cross-objective training in both directions: training encoders for causal language modeling (CLM) and decoders with masked language modeling (MLM). We find that despite continued training for much longer than previous work (50B tokens) these models do not surpass those that started with this objective, i.e. a 400M encoder outperforms a 1B decoder continue-trained with MLM on MNLI, and vice versa for generative tasks.
063
064
065
066
067
068
Our work also provides the ability to compare these training objectives on other aspects, comparing how they learn. We provide case study showing the effects of these objectives on gender bias.
069
070
071
Overall, our work provides the first suite of models enabling a fair comparison between encoder-only and decoder-only architectures (while also showing SOTA performance), enabling future work to analyze the effects of these training objectives on downstream tasks.
072
073
074
075
2
RELATED WORK
076
077
We describe encoder models as the community is generally more familiar with decoder LMs.² 3 is worth noting that our approach was inspired by Pythia (Biderman et al., 2023b) which was the first to explore open-data decoder-only models at multiple sizes.
078
079
080
081
Encoder-only architectures were the predominant architecture for early transformer models, popularized by models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019). and DeBERTa (He et al., 2023). These models showed significantly improved performance over the previous SOTA LSTM models on classification and retrieval tasks. This created a flurry of activity in the encoder space, with models improving on the BERT recipe: the RTD objective from DeBERTa, better data and objectives from RoBERTa, and many smaller variants such as TER (Jiao et al., 2019), DistiBBERT (Sanh et al., 2019), BERT-small (Ture et al., 2019), and Minil. L12 (Wang et al., 2020). However, these encoders lacked the easy ability to generate text and have generally fallen out of popularity in favor of decoder-only GPT-2 style models.
Encoder-only Models
082
083
084
085
086
087
088
089
090
Despite this shift, encoders still maintain frequent usage for many tasks that don't 't require generative output. For example, in March 2025 alone, BERT-base had 90 million downloads on HuggingFacc.
091
092
Recently, there has been renewed interest in encoders, as demonstrated by NomicBERT (Nussbaum et al., 2024), mosiacBERT (Portes et al., 2023), and ModernBERT (Warner et al., 2024). Unfortu- nately, ModemBERT (the most performant) does not provide access to their training data. Hence, we use publicly available data sources in order to replicate the training process.
093
094
095
096
Comparisons between Encoders and Decoders Previous work has compared encoder-only and decoder-only models on a wide assortment of tasks. For example Charpentier & Samuel (2024) compare DeBERTa and GPT-2 in similar sizes. Other work (Yang et al., 2023; Qu et al., 2020; Zheng et al., 2021; Rehana et al., 2023; Nielsen et al., 2024) compares them on downstream tasks.
097
098
099
100
However, all of these comparisons have the same underlying limitation: the models they comparing have different numbers of parameters, different architectures, different training recipes, and different pre-training data. Although some work has attempted address this (Charpentier & Samuel, 2024; Cisserot-Boukhlef et al., 2025), they have only done so in limited settings with very small amounts of pre-training data. In contrast, we train SOTA models, allowing for an exact comparison.
101
102
103
104
105
106
For those interested in decoders, please see early works such as GPT-2 (Radford et al., 2019) and modem models such as OpenAl's GPT-4 (Achiam et al., 2023). Google's 's Gemini (Team et al., 2023), Alihaha's Qwen series Yang al. (2025), and Meta's LLaMA models (Grattafiori al., 2024)
107
2

---

Under review as a conference paper at ICLR 2026
108
17M
32M
68M
150M
400M
1B
109
Parameter
(XXS)
(XS)
(Small)
(Base)
(Large)
(XL)
110
Layers
7
10
19
22
28
111
Hidden Size
256
384
512
768
1024
1792
112
Intermediate Size
384
576
768
1152
2624
3840
113
Attention Heads
4
6
8
12
16
28
114
Learning Rate
3e-3
3e-3
3e-3
8e-4
5c-4
Se-4
Weight Decay
3e-4
3e-4
3e-4
le-5
lc-5
5c-5
115
Warmup Tokens (B)
4
4
3
3
2
116
BS Warmup (B)
125
100
75
50
10
3
117
118
119
Table 1: Configuration for each Ettin model size. Both encoders and decoders use the exact same configuration, differing only in attention (bidirectional vs causal) and objective (MLM vs CLM).
120
121
122
Bidirectional Attention for Decoders Although we cannot cover it all here, there have been attempts use bidirectional attention for standard decoder usage. This includes prefix LM attention (Artetxe et al., 2022; Chowdhery et al., 2023; Du et al., 2021) and other mixed training such as BiTune (Kopiczko et al., 2024). However, most modern LM training still favors pure CLM.*
123
124
125
126
Checkpoint- Level Model Analyses There has also been much work exploring how models learn via their training data. This was popularized by the Pythia (Biderman et al., 2023a) paper and includes many aspects of learning such as data quality and selection (Longpre et al., 2024), how frequency of entities impacts model learning (Oh et al., 2024), effects of the recency of data (Cheng et al., 2024), and whether you can recognize and extract training data from models (Zhang al., 2024). Our work allows these experiments to be done on more recent SOTA models and provides a way to compare encoders and decoders on various facets of learning.
127
128
129
130
131
132
133
134
135
3
EXPERIMENTAL SETTINGS
136
137
3.1
TRAINING DATA
138
139
We create an open-source replication of ModernBERT (Warner et al., 2024) due to it being the strongest publicly available encoder-only model and using comparable techniques decoder-only models. This provides the best starting place for recipe that spans both training objectives. However, MEdesnBERT's data is not publicly available thus, we aim to replicate the recipe using open-data.
140
141
142
143
We do so by pulling from the best publicly available datasets used for training decoder-only models, such as Olmo (Groeneveld al., 2024; OLMo et al., 2025). Thus, we use a mix of DCLM (Li et al., 2024) combined with various curated sources from Dolma v1.7 (Soldaini et al., 2024). In the process of training, the Olmo 2 paper (OLMo et al., 2025) described their approach of using filtered DCLM and other higher-quality sources for the decay phase (similar to FineWeb-Eaa filtering Penedo et al. (2024); Lozhkov et al. (2024)). We decided to use these newer sources for our later phases.
144
145
146
147
148
149
To allow others to easily extend work. provide both formats: the data which can be used for training as well as the data seen by the models in batch order for future analyses.
150
151
152
153
3.2
ARCHITECTURE
154
155
As ModemBBER has only two sizes, we develop new shapes for our smaller and larger models (Table 1). We aim follow the design espoused by MobileL.LM (Liu al., 2024) with deep but thin models. However, for the IB model, we keep the same number of layers but make the model wider. We choose models parameter sizes roughly 2x increments while matching common encoder sizes, e.g. 17M, 32M, 68M, 150M, 400M, and IB. For detailed list of the differences, see Table 1.
156
157
158
159
160
161
"The the best of our knowledge, as much of the details of the best LMs now goes unpublished. 'ee ablated with the non-filtered data and found worse results.
3

---

Under review as a conference paper at ICLR 2026
162
Pre-training
Mid-training
Decay Phase
163
Category
Dataset
Tokens (B)
%
Tokens (B)
%
Tokens (B)
%
164
165
News
CC News
7.3
0.4
166
Code
Starcoder
263.9
15.5
38.4
15.4
Code
Code_ Repos
20.2
26.5
167
Crawl
CC Head
356.6
20,9
168
Crawl
DCLM
837.2
49.1
169
Crawl
DCLM (Dolmino)
175.5
70.4
26.0
34.1
170
Math
Open-Weh-Be Weh-Math
12.7
0.7
171
Math
Algebraic Stacknxc
12.6
0.7
Math
Math (Dolmino)
10.4
4.2
5.0
6.6
172
Scientific
PeS2o
57.3
3.4
8.3
3.3
173
Scientific
Arxiv
28.0
1.6
4.1
1.6
3.0
3.9
174
Social
Reddit
80.3
6.2
2.5
175
Social
Stacknxchange
19.6
1.1
176
Social
StackExchange (Dolmino)
2.7
1.1
4.0
5.2
Reference
Textbooks
0.5
0.7
177
Reference
Dolma Books
5.3
0.3
0.8
0.3
10.5
13.8
178
Reference
Wikipedia
3.0
3.9
179
Instruction Tulu Flan
16.6
1.0
2.4
1.0
4.1
5.4
180
Total
1,704.7
100.0
249.3
100.0
76.3
100.0
181
182
183
Table 2: Training data mixture across the various training stages (pre-training, mid-training decay).
Later stages use higher quality data, from the recently released Dolmino dataset (OLMo et al., 2025). Dashes indicate that data from that source was used. We trained for 1.7T tokens for pre-training,
184
185
250B for mid-training, and 50B for the decay phase. We sample from the dataset and repeat (or under-sample) as needed hit the token counts used for training.
187
188
3.3
TRAINING RECIPE
189
190
We use the same general process described by open-data models (which was followed by Modern- BERT) for training both encoder-only and decoder-only models with a few specific changes for the encoder architecture (i.e. masking ratio). In summary, we include three general phases: base pre-training, mid- heddttariin//ooneett extension, and decay. See Table 2 for the precise sources training data in each phase. We use trapezoidal learning rate scheduler, with general hyperparameters shown in Appendix C and size-dependent hyperparameters Table 1. For compute details see Appendix
191
192
193
194
195
196
The only differences between the encoder and decoder models (1) the objective function, i.e. masked language modeling (MLM) for the encoder vs causal language modeling (CLM) for the decoder and (2) the attention pattern, i.c. causal for the decoder and bidirectional for the encoder.
197
198
199
We checkpoint every 8.5B tokens, with 236 checkpoints per model. Combined with the batch ordering of the data, this enables precise pinpointing of what the model learned between each checkpoint.
200
201
202
Base Pre-training This stage encompasses the warmup and stable phase of the trapezoidal learning rate, training for 1.7T tokens. We use both learning rate and batch size warmup. The data this stage comprises wide mix of sources to allow for general learning.
203
204
205
Context Extension Mid-Training In this phase we increase the quality of the data and change both the data and base RoPE (Su al., 2024) to handle longer context. We update the data length to be up to 8000 tokens and RoPE parameters to 160k (for global and local layers). For the data, we drop the noisiest sections (older Dolma common crawl, CC News, general StackExchange) and include filtered DCLM, math, and StackExchange. We then train for 250B tokens and use an inverse square root learning rate schedule from the peak learning rate 1/2 of the peak.
206
207
208
209
210
211
212
Decay Phase Finally, tokens. We follow the general ProLong recipe (Gao et al., 2024b), increasing long context data such Dolma books, Wikipedia, and open-access textbooks. We decay 0.02 of the peak LR.
inverse square root learning rate schedule decay for 50B
213
215
FFor the encoder 30% masking ratio except for the decay phase, which is 15%.
4

---

Under review as a conference paper at ICLR 2026
216
3.4
MAJOR TRAINING DIFFERENCES FROM MODERNBERT
217
218
A concise summary of the largest differences from the ModernBERT recipe are (1) the use of open- data, (2) decay in the context extension phase, (3) no model merging, 7 (4) a lower masking ratio for the decay phase (15% instead of 30%) and (5) local and global RoPE to be the same value.
219
220
221
222
3.5
OBOSS-OBIEE TRAINING
223
224
As encoder models have gone out of popularity, decoder models have increased size (both parameters and pre-training data). Thus, these newer decoder models are typically trained for much longer than previous encoder models (i.c. BERT). Due to this it has become common to adapt these larger decoder models to what were previously encoder-centric tasks (Zhang et al., 2025). With paired encoder and decoder models, we can now answer the question of how effective this continued pre-training approach is and whether it is still worth training both types of models. We call this cross- cross-objective training: taking the final model and continue pre-training it on the reverse objective."
225
226
227
228
229
230
231 We train for 50B tokens with the reverse objective, which is far more than previous work has attempted (BehnamGhader et al., 2024), which is usually around 10B tokens. Although the ratio of pre-training and cross objeettbee training is unbalanced, this mimies the realistic setting where the adaptation is done with very small amounts of data comparatively. We do this cross-objective training on the highest quality data we have available, which was used in the last decay phase. 9 We use new trapezoidal learning rate schedule with 3B tokens of warmup and 10B tokens of decay. 10 Thus, by the end we have an encoder-from from-decoder (i.e. a decoder further pretrained with MNTP!!) and decoder-from-encodee (i.e. encoder further pre-trained with CLM).
232
233
234
235
236
237
238
239
240
4
EXPERIMENTS
241
242
We aim to compare encoder and decoder models. However, first, to give those experiments credence, show that our models are SOTA. This strengthens claim and helps alleviate concerns that we made training choices that favored one architecture over the other - instead we have SOTA models in both architectures for their sizes, showing our method's effectiveness. Note though, that the purpose of our paper was not to be SOTA overall (i.e. compared to OpenAl, etc.), but to provide a comparison for encoders and decoders. For space and avoid repetition, specific model size details are in Table 1.
243
244
245
246
247
248
249
4.1
INDIVIDUAL EVALUATIONS
250
251
We use two baselines for each size type: extra extra small (XXS) BERT-min and TERY, extra small (XS) models MiniLM L.22 and BERT-small, small (S) models DistilBERT and DistilRoBERTa, base (B) models BERT and ModernBERT, large (L) models BERT-large and ModemBBERT-large, and an extra large (XL) model DeBERTA v2 XL.
Encoder-Only Results
252
253
254
255
We evaluate on various encoder tasks, including GLUE (Wang et al., 2018), MTEB v2 English (Enevoldsen al., 2025), MDLR for long context (Chen al., 2024), and NodeSearchNtt for code evaluation (Husain et al., 2019). We use the same evaluation setup as ModernBERT for the evaluation for an equal comparison (see Appendix F for hyperparameter details).
256
257
258
259
260
"ee this for ease scientific comparison, however, one was to use this for downstream applications a simple merge would likely boost performance another point or two.
261
262 FFollowing BehnamGhader al. (2024), we do not MLM but rather use MNTP, that is, the masked token predicted using the hidden state of the previous token to better align with CLM.
263 WW note that this means it repeating this data twice, however. as shown by previous work (Muennighoff al., 2023) two repetitions high quality data has no adverse effects. Foro the 1B model this is sealed by 1/3 again due to compute availability.
264
265
266 We use 15% masking rate for the encoder- macoder-from-aee decoder We also ran experiments with DeBERTa XXL as shown in the appendix. However, due to the size and
maintain middle ground masking ratio.
267
268
slowness of the architecture we could not do an comparable grid search. Our results in the appendix are after
269
300 days of H100s hours, hut still did not complete the full sweep. Furthermore, DeBERTa XXI. 1.5B we exclude it is significantly larger than 1B (i.c. > 50% larger).
5

---

Under review as a conference paper at ICLR 2026
270
Embedding Tasks Clustering
GLUE Tasks
271
Model Name
CSN
MLDR
Retrieval
MTEB v2
SST-2
MNLI
GLUE Avg
272
XXS Models (7-17M parameters)
273
BERT-mini
41.3
16.8
39.0
34.7
49.2
88.3
77.2
76.4
274
TinyBERT Eitin-Enc-17m
39.8
14.2
37.4
33.3
49.7
91.2
80.9
77.0
275
59.1
24.4
39.1
35.6
48.9
91.2
79.5
79.2
276
XS Models (28-33M parameters)
277
BERT-small
46.0
19.9
39.6
38.1
51.1
90.1
79.2
79.0
278
Minil.M L.12
48.3
19.6
37.8
38.4
51.3
93.3
85.6
84.6
279
Fttin-Fnne322m
69.2
28.4
39.6
39.7
50.9
92.0
83.4
83.5
280
Small Models (68-82M parameters)
281
DistiBBER
47.9
23.7
39.8
40.8
52.7
92.2
82.7
81.5
282
Dostil Fitin-Enn-68m
60.3
19.7
39.3
40.0
51.8
93.1
84.7
83.8
75.1
30.1
40.1
43.1
52.6
94.4
87.0
87.2
283
Base Models (123-150M parameters)
284
285
BERT base
51.0
24.8
40.4
41.2
52.9
93.1
85.4
84.7
ModernBERT base
75.9
30.4
41.3
43.9
54.0
96.0
89.1
88.4
286
Ettin-Enc-11 15cm
76.3
31.8
41.5
45.7
54.0
95.8
89.2
88.9
287
Large Models (353-395M parameters)
288
BERT large
54.4
25.3
41.5
42.9
53.8
93.3
86.3
85.2
289
ModemBERT large
78.3
34.9
41.5
47.0
55.0
97.1
90.8
90.4
290
Ettin- moin-Enn-400m
80.7
36.2
41.8
48.4
55.5
96.7
91.3
90.8
291
XL Models (884M-1.2B parameters)
292
DeBERTi-vI-xl 75.6
42.5
47.2
56.4
97.1
91.7
90.7
293
Ettin-Enc-1B 82.3
40.2
41.9
50.1
56.0
97.1
91.8
91.6
294
295
Table 3: ETTIN encoders compared to other encoder-only models across various sizes on retrieval and GLUE tasks. Due space, show two representative tasks from MTEB v2 and two from GLUE, as well as code-based retrieval evaluation (CodeScarchNNet) and a long-conted evaluation (MLDR). See Appendix A for the full tables of GLUE and MTEB v2. ETTIN shows significant gains over baseline encoders, including ModernBERT, while also having both larger and smaller sizes.
296
297
298
299
300
301
We find in Table 3 that ETTIN compares favorably overall. The relatively larger gains in the bigger sizes likely because the smaller model baselines are heavily optimized with distillation." Even so, see that they generally outperform the baselines without doing any distillation: e.g. Ettin-68m with GLUE average of 87.2 compared to the next best BoRtia at 83.8. Even for the more recent ModemnERR baselines we see improved performance (88.9 GLUE average vs 88.4 for the
302
303
304
305
base size). Thus we can see that ETTIN matches or improves the SOTA for encoder-only models.
306
307
Decoder-Only Results We use two baselines for each size type when available, but few very small decoder-only LMs exist: One extra extra small (XXS) model Pythia- 14M, no models in the extra small (XS) category that we could find, one small (S) model DistilGPT (Sanh et al., 2019), two base (B) models Pythia 160M (Biderman et al., 2023a) and SmolLM2 135M, two large (L) models Pythia 410M and SmoLM2 360M (Allal et al., 2025), and extra large (XL) models Olmo IB 0724 Groeneveld et al. (2024) and Llama 3.2 IB (Dubey et al., 2024)."
308
309
310
311
312
313
We evaluate on a wide range tasks using the Eleuther AI harness (Gao et al., 2024a) (see Appendix for details), consolidating tasks used in the Pythia and SmolLM papers including: the ARC Challenge (ARC) Clark et al. (2018), HellaSwag (HS) (Zellers et al., 2019), LAMBADA (LMB) (Paperno et al., 2016), OpenBookQA (OBQA) (Mihaylov et al., 2018), Social IQA (SIQA) (Sap et al., 2019), TriviaQA (TQA) (Joshi et al., 2017), Winogrande (WG) (Sakaguchi et al., 2021), and the Winograd Schema Challenge (WSC) (Levesque al., 2012).
314
315
316
317
318
319
320
We also note that MiniL.M L.12 has twice the amount of non-embedding parameters 21M vs 12M and
321 "This is likely just debug-sized as not other decoder models in this size could found, we use it as a reference.
official size, as they do include it in their paper. However,
322
323
Whaat is considered "IB" has large range, up to nearly 2B parameters. We thus restrict our range to 1.2B parameters be "IB" model, which excludes models like SmolLM2 1.7B and Olmo 2 IB (actually 1.5B).

---

Under review as a conference paper at ICLR 2026
324
Model Name
ARC
HS
LMB
OBQA
PIQA
SciQ
SIQA
TQA
WG
WSC
Avg
325
XXS Models (14-17M parameters)
326
327
1ythia-14m
21.2
26.0
7.1
26.2
55.2
43.8
33.4
0.0
50.3
51.6
31.5
Sitin-Dec-171
21.3
27.1
23.0
27.2
57.7
71.1
35.4
2.6
50.9
48.0
36.4
328
XS Models (32M parameters)
329
330
Demin-
23.5
28.5
28.5
28.2
57.7
77.5
36.4
3.8
53.1
50.2
38.7
331
Small Models (68-82M parameters)
332
DistilGPT
23.0
27.5
25.0
26.8
59.8
62.6
36.1
0.3
50.4
53.8
36.5
333
Bttin-Dec feen--De-66m
25.3
33.4
35.2
29.4
61.8
83.2
38.8
5.6
50.1
55.3
41.8
334
Base Models 1.35-160M parameters)
335
SmolLM2-1 135m
29.1
43.1
42.9
32.4
68.4
78.5
39.4
5.0
53.7
59.7
45.2
336
Pythia- 160m
24.0
30.2
32.9
26.4
62.0
67.2
36,9
0,4
52.4
58.2
39.1
337
Bttin- Dec-150m
28.6
40.3
43.2
29.2
66.6
89.6
40.1
11.2
53.7
59.0
46.2
338
Large Models (360-410M parameters)
339
Pythia-410m
24.7
40.6
51.5
29.4
67.0
72.3
39.0
1.8
53.6
65.2
44.5
340
Mm-6LM2
37.6
56.3
53.5
37.6
71.8
86.6
40.7
18.4
58.6
70.3
53.1
341
Bitin-Dec-400m
33.6
54.3
52.3
34.4
71.0
91.8
45.5
18.3
57.6
71.8
53.1
342
XI. Models (908M-1,2B parameters)
343
OLMo-IB-0722
32.3
66.1
61.0
35.6
75.1
91.8
49.2
1.2
61.6
76.9
55.1
344
Llama-3.2-1B
36.2
63.7
62.1
37.2
75.0
88.4
43.2
24.9
60.6
74.7
56.6
Ettin Dec-IB
39.7
62.9
58.4
41.6
74.4
93.8
48.2
29.3
62.7
79.1
59.0
345
346
Table 4: Performance comparison decoder-only models across tasks, organized by size categories. We see that ETTIN decoders compare favorably, matching exceeding the previous open- data SOTA. Task names in order are ARC, Hellaswag, LAMBADA, OpenBookQA, Social IQA, TrivialQA, Winogrande, and Winograd Schema Challenge.
347
348
349
350
351
We see the results in Table 4 and see that ETTIN performs well compared to baseline models, such as ETTIN-150m outperforming SmolL.M2 46.2 to 45.2 and ETTIN-IB'S 59.0 Llama 3.2 1B's ' 56.6 average). Thus can that ETTIN improves the SOTA for open-data decoder-only models.
352
353
354
355
4.2
ENCODERS VS DECODERS
356
357
Now that the strength of the training recipe is established, we can compare the two training objectives.
358
For simplicity, show the two most representative encoder tasks (MS MARCO dev Bajaj et al. (2016) for retrieval and classification on MNLI) and keep the average generative score." 16 We evaluate the decoders encoder-only tasks and vice versa, and similarly with the cross-objective trained models. We evaluate encoder-only models on decoder generative tasks using the method proposed in Samuel (2024), i.e. using three mask tokens the end of the sequence and filling in the first token iteratively, Figure I shows the results of this comparison across models sizes.
359
360
361
362
363
364
365
MNLI Classification On the representative classification task, we see that encoders dominate decoders, as typically found. Furthermore, we that even cross-objective continued pre-training does not resolve this gap, with enc-from-dec performance remaining similar to the original decoder model. Furthermore, the pure encoder models are typically better than the next larger sizes of decoders, e.g. the 150M encoder scoring 89.2 compared to the 400M decoder's 's 88.2.
366
367
368
369
370
MS MARCO Dev Retrieval For retrieval we see similar encoder dominance like classification, but notably improved performance when continue pre-training the decoder (i.e. the encoder-froo from- decoder). The MTNP continue pre-training significantly helps the decoder at all sizes, yet even the additional 50B tokens of pre-training not enough match the performance of the encoder (i.e. for the 400M size we have 42.2 for the encoder vs 41. 1.4 for the ficoderr-frmm-dcceee difference is not as pronounced as in classification, we find that continued pre-trase pre-training decoder for retrieval is still subpar compared to simply using an encoder, even despite the additional 50B tokens.
371
372
373
374
Although the
375
376
377
As decoder evaluations are significantly quicker than fine-tuning which is required for encoder tasks.
7

---

Under review as a conference paper at ICLR 2026
378
Encoders
Decoders
Enc-from-Dee Dec
-
Dec-from-Enc
379
MNLI
380
381
382
Accuracy
383
384
80
385
386
Retrieval (MS MARCO dev)
387
388
389
aDCC@10
40
390
35
391
392
393
Generative Tasks Average
394
60
395
Average Scure
396
397
45
398
40
399
35
400
15M
30M
70M
110M
340M
770M
401
Model Size (M parameters)
402
Figure 1: Encoder decoder comparison across model size. Generally models in the preferred architecture (e.g, encoders in MNLI) do better than the opposite architecture even with an order of magnitude greater size, e.g. a 400M decoder outperforming the 1B encoder. Notably, in generative tasks, decoders- froneerr-rro scale poorly with size.
403
404
405
406
407
We find the reverse of the previous tasks: decoders do better than the decoder- from-encoder in general, with widening gap (from similar scores at 68m parameters greater than a point difference at 1B) as model size increases. Notably, it appears that continued training of from-decoders scales poorly, perhaps why there is little-to-no previous work the topic.
Generative Tasks
408
409
410
411
Despite this, we note that this average hides some nuance: on "generative" tasks that are more classification focused (such ARC and SciQ) encoder models used in generative fashion actually exceed decoder performance (i.e. for the 400M size the encoder scores 35.6 ARC vs the decoder's 33.6). However, decoders show huge gains on tasks such as HellaSwag, TriviaQA, and SiQA, making it so the average is strongly in favor of the decoders. See Table 7 for all sub-task results.
412
413
414
415
416
417
5
CASE STUDY: GENDER BIAS
418
419
Due to the Eutin suite's open-pretraining data, we can also analyze other aspects of learning across pre-training objectives. As one example, we analyze gender representations for bias.
420
421
We use the WinoGender benchmark (Rudinger et al., 2018) using the "Gotcha" split that has 50/50 split of male/female stereotypical pronouns (i.e. female for nurse). However, the standard coreference task is hard for most of our small models. Thus, we show results for an easier task: simply predicting the pronoun in the sentence. For the standard coreference task results, see Table G in the appendix.
422
423
424
425
We have each model predict the pronouns (i.e. by using mask token for encoders or by choosing the lower perplexity sentence with decoders) and show the distribution of predicted pronouns per model (male, female, gender neutral)." !7 The results are in Figure 2, which shows that encoders are much more likely to use gender neutral pronoun overall. In both encoders and decoder, female pronouns become more used as the size of the model gets larger: for decoders there a clear trend of
426
427
428
429
430
431
There are more than three types of pronouns used in English beyond what is in this dataset. However, WinoGender is only designed for these three. We leave extensions of this dataset future work.
8

---

Under review as a conference paper at ICLR 2026
432
Male
Female
Neutral
433
Ettin-Encoder
434
100%
435
2.3%
25%
436
75%
47%
48%
10%
437
Prediction Probability
50%
9%
438
25%
50%
46%
6.%
439
440
17M
32M
66M
150M
400M
18
441
Sctin-Decodes
442
100%
20%
20%
31%
30%
443
75%
10%
10%
444
50%
84%
445
25%
63%
446
0%
447
17M
66M
Model Size
449
Figure 2: Gender pronoun predictions on the Gotcha split of WinoGender (Rudinger et al., 2018), 50/50 stereotypical male/female split. We see that encoder models are more likely to use gender neutral pronouns whereas both are biased towards male pronouns.
450
451
452
progressively smaller amounts of male pronouns, whereas for encoders the trend is more stochastic. For effects the cross-training objectives on the model, see Figure in the Appendix.
453
454
455
Overall
As both models had the same training data, find that the MLL objective leads the model to choose more neutral pronouns over female pronouns. However, male gender bias seems strong in both models, slightly higher for decoders. Thus, this is one example of the analysis enabled by our data; we leave others to future work.
456
457
458
459
460
DISCUSSION
461
462
Our work suggests the following conclusions: (1) as speculated by previous work, MLM and CLM objectives do convey different strengths - ML.M for classification and retrieval and CLM for generative tasks. However, (2) we also went a step further to show that simply continued pre-training on the reverse objective does not make up the difference from not using the preferred architecture.
463
464
465
466
467
This has several implications for those using models for classification or retrieval: currently the top models leaderboards like MTEB 7B+. However, based a 3B encoder model would outperform it. But, the lack of large encoder-only models means that approaches that continue pre-train decoders using MLM will likely outperform all other options (as is currently seen on the leaderboards). In the small scale regime (1B or less) where easier to train more "niche" encoder models for classification/retrivall continue to outperform all others in their size range (and even ones above it).
468
experiments, it is likely
469
471
results indicate that encoders will
472
473
474
Our results also suggest that encoders and decoders learn differently in other aspects as well, such as gender bias. Although this is just one example, we look forward to future research that discovers other differences. Overall, our artifacts allow for range of new analyses and pre-tree pre-training research.
475
476
478
7
CONCLUSION
479
480
We provide the first suite of paired models that use the same training recipe to compare encoder-only and decoder-only models. Our models are SOTA in their size for open-data models, and are the first public recipe for ModemBBERTsstyle models. We show that encoders are strong in classification and retrieval, while decoders are strong in generative tasks. Furthermore, we show that this difference can not casily be solved by continued training with the reverse objective. We show that this suite allows the analysis how pre- treinia objective impacts learning, showing a case study in gender bias. We release all artifacts (including training data order) help future researchers analyze these models.
481
482
483
484
485

---

Under review as a conference paper at ICLR 2026
486
REFERENCES
487
488
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
489
490
491
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíčck, Agustín Piqueres Lajarin, Vaibhav Srivastav, et al. Smollm2: When smol goes big- dag-dtt--ccntric training a small language model. arXiv preprint ar ar1iv:2502..01111 027.077777..777 2025.
492
493
494
495
496
Mikel Artetxe, Jingfei Du, Naman Goyal, Luke Zettlemoyer, and Ves Stoyanov. On the role of bidirectionality in language model pre-training. arXiv preprint urXiv:2205. //726, 2022.
497
498
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint 1r 1.09268, 2016.
499
500
501
Parishad SchnamGiaader Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.0540 2024.
502
503
504
505
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023a.
506
507
508
509
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023b.
510
511
512
513
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, lya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Ties Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Decem- ber 6-12, 2020, virtual, 2020. URL https: / http://oooccce.nc.eee.eo newpeiiiinnnsniiiis....p...p cwturipp.ccppppe 20aeee///22///// /hahh/ (4570866bccb49674 Bb4bre6bbccbb9774488bbbbaal 8bfbaacl 425a4a-bbtraact html.
514
515
516
517
518
519
520
521
522
523
524
Lucas Georges Gabriel Charpentier and David Samuel. GPT or BERT: why not both? In Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Leshem Choshen, Ryan Cotterell, Alex Warstadt, and Ethan Gotlieb Wilcox (eds.), The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pp. 262-283, Miami, FL, USA, November 2024. Association for Computational Linguistics. URL https: //aclanthology. org//024 conll- 24/.
525
526
527
528
529
530
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, fulti-ffuctioonalty, quuti-granularity text embeddings through self-knowledge distillation, 2024.
531
532
533
Jeffrey Cheng, Marc Marone, Orion Weller, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. Dated data: Tracing knowledge cutoffs in large language models. arXiv preprint arXiv:2403. /2958, 2024.
534
535
536
537
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
538
539
10

---

Under review as a conference paper at ICLR 2026
540
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
541
542
arXiv preprint arXiv:1803.054:
543
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.). Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171- 4186. Association for Computational I. Linguistics, 2019. doi: 10.18653/V/NN9 1423. URL https://doi, 10. 18653/v1/nn 9-2333..
544
545
546
547
548
549
550
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm:
551
General language model pretraining with autoregressive blank infilling. In Annual Meeting of the Association for Compulational Linguistics, 2021. URL http://pp..smmanicscco.oos org/
552
553
247519241. 2.
554
555
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint urXiv:2407.21788 2024.
556
557
558
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzeminski, Genta Indra Winata, et al. Mmteb: Massive multilingual text embedding benchmark. arXiv preprint arXiv:2502. 13595, 2025.
559
560
561
Kawin Ethayarajb. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint
562
2019.
563
564
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPoñi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024a. URL https: https/////eeeooooorrrrrrrrr///// 21686666
565
566
567
568
569
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660 2024b.
570
571
572
Hippolyte
Nicolas Boizard, Manuel Faysse, Duarte M. Alves, Emmanuel Mal- herbe, André F. T. Martins, Céline Hudelot, and Pierre Colombo. Should we still pretrain encoders with masked language modeling?, 2025. URL
573
00994..
574
575
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.22 27883 2024.
576
577
578
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerat- ing the science of language models. arXiv preprint arXiv:2402.00838 2024.
579
580
581
582
Fouzi Harrag, Maria Debbah, Kareem Darwish, and Ahmed Abdelali. Bert transformer model for detecting arabic gpt2 auto-generated tweets. arXiv preprint arXiv:2101. 2021.
583
584
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre- training with gradient-disentanglee embedding sharing. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:: https:://openrevier/ wwwpaarewwwwee.eeeorreeeeeee
585
586
587
588
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
589
590
591
592
Hamel Husain, Ho- Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search. arXiv preprint wrXiv:1999004433
593
2019.
11

---

Under review as a conference paper at ICLR 2026
594
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:I909 10351, 2019.
595
596
597
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: 2017.
598
599
600
Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Bitune: Bidirectional instruction-tuning. arXiv preprint arXiv:2405. ( 14862, 2024.
601
602
Simon A Lee, Anthony Wu, and Jeffrey N Chiang. Clinical modemberr: An efficient and long context encoder for biomedical text. arXiv preprint arXiv:2504.03964, 2025.
603
604
605
Hector J. Levesque, Ernett Davis, and Leora Morgenstern. The winograd schema challenge. In 13th
606
International Conference on the Principles of Knowledge Representation and Reasoning, KR 2012, Proceedings of the International Conference on Knowledge Representation and Reasoning, pp.
607
608
552-561. Institute of Electrical and Electronics Engineers Inc., 2012. ISBN 9781577355601. 13th
609
International Conference on the Principles of Knowledge Representation and Reasoning, KR 2012 Conference date: 10-06-2012 Through 14-06-2012.
610
611
Jeffrey Li, Alex Fang, Georgios Smymiss Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-Im: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37:1 12100-4280222 2024.
612
613
614
615
616
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeutlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1 1692, 2019. URL http://arxiv.orrrt org/ /99/
617
618
619
Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong. Ernic Chang. Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first International Conference on Machine Learning, 2024.
620
621
622
623
Shayne Longpre. Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph. Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pp. 3245-3276, 2024.
624
625
626
627
628
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL https://hugginggaac. wttp.//ooaaaaaaaeeeaeeeeea #luginngFacee//fineeee---
629
630
631
632
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint 2018
633
634
635
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-503777 2023.
636
637
638
639
Dan Saattrup Nielsen, Kenneth Enevoldsen, and Peter Schneider-Kamp. Encoder vs decoder: Comparative analysis of encoder and decoder language models multilingual nlu tasks. arXiv preprint arXiv:2406. 13469, 2024.
640
641
642
Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic embed: Training reproducible long context text embedder. CoRR, abs/2402.01613, 2024. doi: 10.48550/ARXIV. 2402.01613. URI. https: //doi. wtt.://///....///rreee....// 2402.01613.
643
644
645
646
Byung-Doh Oh, Shisen Yue, and William Schuler. Frequency explains the inverse correlation of large language models' size, training data amount, and surprisal's 's fit to reading times, arXiv preprint 022.:55002.02255. 2024.
647
12

---

Under review as a conference paper at ICLR 2026
648
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradcep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zeutlemoyer Ali Farhadi, Noah A. Smith, and Hannanch Hajishirzi. 2 olmo furious, 2025. URL https: http:///aaaxvv.oo/aasss.ss org/abs/ 2501. 00656.
649
650
651
652
653
654
655
656
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv:1606.0000 1603306306, 2016.
657
658
659
660
Guilherme Penedo, Hynek Kydlřček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:308 1 1-30849, 2024.
661
662
663
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. arXiv preprint arXiv: 018888808949 2018
664
665
Jacob Portes, Alexander Trott, Sam Havens, Daniel King, Abhinav Venigalla, Moin Nadcem, Nikhil Sardana, Daya Khudia, and Jonathan Frankle. Mosaicbert: A bidirectional encoder optimized for fast pretraining. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LAA, USA, December 10 - 16, 2023, 2023. URL http: http:://appprsss... nips. cc/ cttplpprsccpppppr_reeaep 2023/hash/ (achh//955691111 777777777777711 2b7ccc61 acbeacadeedeeeee d8 -er-ee--bbbrratttooeeeeeeeeeeeee html.
666
667
668
669
670
671
672
673
Yuanbin Qu, Peihan Liu, Wei Song, Lizhen Liu, and Miaomiao Cheng. A text generation and predic- tion system: pre-training on new corpora using bert and gpt-2. In 2020 IEEE 10th international conference on electronics information and emergency communication (ICEIEC), pp. 323-326. IEEE, 2020.
674
675
676
677
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 1(8):9, 2019.
678
679
Hasin Rehana, Nur Bengisu Çam, Mert Basmaci, Jie Zheng, Christianah Jemiyo, Yongqun He, Arzu- can Özgür, and Junguk Hur. Evaluation of gpt and bert-based models on identifying proteinprotein interactions in biomedical text. ArXiv, pp. arXiv-2303, 2023.
680
681
682
683
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. arXiv preprint 100X:v8804.09300 2018.
684
685
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106,
686
687
2021.
688
689
David Samuel. Berts are generative in-context learners. Advances in Neural Information Processing Systems, 37:2558-2588 2024.
690
691
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 2019.
692
693
694
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv: 2019.
695
696
Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, et al. Reasonir: Training retrievers for reasoning tasks. arXiv preprint arXiv:2504.2005 20595, 2025.
697
698
699
700
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00/55 2024.
701
13

---

Under review as a conference paper at ICLR 2026
702
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiig. 568: 18706,, 2024.
703
704
705
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptin Alayrac, Jiahui Yu, Radu Soricut,
706
Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.. 1805, 2023.
707
708
Tulia Ture, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv: 000/9/988.0922 2019.
709
710
711 Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint arXiv:180..07461, 2018.
712
713
714
715
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33:5776- 5788, 2020.
716
717
718
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. Smarter, bet- ter, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint 2rXiv:2412.1366 2024.
719
720
721
722
723
Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. Followir: Evaluating and teaching information retrieval models to follow instructions. arXiv preprint arXiv:2403.15244 2024.
724
725
726
Orion Weller, Benjamin Chang, Eugene Yang, Mahsa Yarmohammadi, Samuel Barham, Sean MacAvaney, Arman Cohan, Luca Soldaini, Benjamin Van Durme, and Dawn Lawrie. mfollowir: A multilingual benchmark for instruction following in retrieval. In European Conference on Information Retrieval, pp. 295-310, 2025.
727
728
729
730
731
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025.
732
733
734
Binxia Yang, Xudong Luo, Kaili Sun, and Michael Y Luo. Recent progress on text summarisation based on bert and gpt. In International conference on knowledge science, engineering and management, pp. 225-241. Springer, 2023.
735
736
737
738
Rowan Zellers, Ari Holzzaan, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint 19XXv:9905.07830 2019.
739
740
Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang. and Hai Li. Min-K%++ Improved baseline for detecting pre-training data from large language models. arXiv preprint arXhv22404.02936 2024.
741
742
743
744
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advanc- ing text embedding and reranking through foundation models. arXiv preprint 20X.v:2506.05/77 2025.
745
746
747
748
749
Xianrui Zheng, Chao Zhang, and Philip C Woodland. Adapting gpt, gpt-2 and bert language models for speech recognition. In 2021 IEEE Automatic speech recognition and understanding workshop (ASRU), pp. 162-168. IEEE, 2021.
750
751
752
753
A
ALL ENCODER RESULTS
754
755
We show results for the full MTEB v2 eng and GLUE results in Table 5 and Table 6 respectively.
14

---

Under review as a conference paper at ICLR 2026
756
B
DECODER EVALUATION FRAMEWORKS
757
758
generative tasks, we use the Eleuther AI harness with commit We also use forked version of the Eleuther AI harness for evaluating encoders in a generative fashion (see Github for details). Following previous work (Allal al., 2025) the ARC score is the average of the easy and challenge sets.
For
759
760
761
762
For encoders evaluated on generative tasks we use three mask tokens followed by the EOS token, At each step, we predict the first MASK token and iteratively generate. However, this approach still could be improved, in particular around the EOS token. Encoder models are non-calibrated for when this should appear, so we had to make small changes to this setup for two tasks: for TriviaQA we change the EOS token to newline character (as the harness stops newlines for TriviaQA also) and for the Lambada OpenAI task we not score the EOS token. All other tasks proceed with the three masks EOS token proposed by Samuel (2024).
763
764
765
766
767
768
769
770
C ARCHITECTURE DETAILS
771
772
Architecture and training details for all models are found in Table G. These are generally the same as ModemnBER except for the same value of local and global RoPE and a slightly shorter context length (7999).
773
774
775
776
D MODEL SIZES
777
778
Model sizes (both embedding and non con-embedding) are found in Table 10. We group models by total parameters, although we note that some have more vocab parameters vs non-vocab parameters, e.g.
779
780
MiniLM L12 has almost 2x the number of non-embedding parameters compared to Ettin-32m (21M
781
vs 12M).
782
783
E
COMPUTE CONFIGURATION
784
785
We train the models on a comparatively small compute cluster. We train each model on 4xH100 node using NVLinn. The pre- vre-training phase (the longest) takes approximately 6 days for the 17M model. The longest was for the IB model, which we trained for approximately 40 days. Unfortunately. we did not have enough compute availability to train the IB to the full 2T tokens. Thus the 1B models are scaled to 1/3 of the data (e.g. 667B instead 2T tokens). However, this is still more than chinchilla optimal (Hoffmann et al., 2022) and it still outperformed other 1B models trained longer.
786
787
788
789
790
791
792
F
ENCODER EVALUATION SWEEP PARAMETERS
793
794
Below we detail the sweep hyperparameters for the retrieval and classification tasks that require fine-tuning.
795
796
797
We re-use ModemBBRRT's evaluation setup but slightly increase the learning rate sweep in order to better fit the smaller parameter models (which typically use higher LRs). As the best LRs chosen by BERT and ModernBERT were lower than these, it does not affect their scores. We sweep for learning rates over { (1e-5, 3--5, 5e-5, 8e-5, le-4), weight decay values over {1e-6, 5e-6, 8e-6, le-5), and batch sizes over {16, 32}. We also sweep over epochs {1, 2, 3, 4] if task E {mnli, sst2, rte), total of 160 sweeps per model, of which select the best score per task per model to report. We start from the best MNLI checkpoint for fine-tuning on RTE, STS-B, and MRPC, following ModemBERT.
GLUE
798
799
800
801
otherwise {2, 5, 10, 12). This
802
803
804
805
We sweep four LRs (f1e-4, 3e-4, 5e-4, 7e-4)) on MS MARCO dev and choose the best performing one to evaluate on the other retrieval datasets. We use a new retrieval training script due to being unable to exactly reproduce ModemBBERT's precise scores. While doing so, we also improve the training process for all models due the of more negatives in training, achieving higher scores than that in the ModernBERT paper (which was not trying to optimize scores, but shows generally that training script is effective). We trained with effective batch size of 1024
Retrieval
806
807
808
809
15

---

Under review as a conference paper at ICLR 2026
810
with accumulation steps. For DeBERTa-v2 v2 it diverged for all learning rates tried. Thus, to get to converge, we changed the warmup to 20% from 5% and lowered the learning rate to le-5.
811
812
As Ettin models have been trained with instructions during pre-training, it is likely they also more capable for instruction- based retrieval (Shao et al., 2025; Weller et al., 2024; 2025), however, we leave that for future work.
813
814
815
816
817
G
LLM USAGE
LLMs not used for paper writing, but were used for coding assistance and title brainstorming. All code was human verified.
820
821
Model Name
Mean (Task)
Mean (Type)
Class.
Clus.
Pair. Class.
Rerunk.
Retricval
822
XXS Models (11-25M parameters)
823
TinyBERT BERT-mini Ettin-Ene-17m
52.1
49.7
64.6
37.4
78.1
41.9
33.3
71.9
20.6
824
$1.8
49.2
61.7
39.0
77.8
41.5
34.7
70.8
19.3
825
52.3
48.9
39.1
74.8
42.0
35.6
71.6
16.0
XS Models (28-33M parameters)
827
BERT-small
54.0
51.1
64.7
39.6
79.4
41.7
38.1
73.0
21.6
828
MiniLM M L12
53.9
51.3
64,9
37.8
79.8
43.5
38.4
73.2
21.4
Butin Enc 32m
54.2
50.9
64.2
39.6
77.1
42.6
39.7
73.2
19.9
829
Models (68-82M parameters)
830
DistiBERT
55.5
52.7
66.5
39.8
42.8
40.8
74.1
24.0
831
DostilRoBa
55.2
51.8
67.3
39.3
78.9
43.5
40.0
74.2
19.4
832
Bitin-Fin-688m
56.1
52.6
66.6
40.1
79.3
43.3
43.1
74.2
21.7
833
Models (86-150M parameters)
834
BERT-base
56.0
52.9
67.2
40.4
80.5
43.1
41.2
74.8
ModemBERT-bbss
57.1
54.0
67.5
41.3
80.4
44.7
43.9
75.3
25.2
835
Bitin-Ene-1500
57.7
54.0
68.6
41.5
80.2
44.7
45.7
74.9
22.6
836
Large Models (305-395M parameters)
837
BERT-lange
57.2
53.8
41.5
44.3
42.9
76.1
838
ModemBERT-large
58.6
55.0
69.1
41.5
82.2
45.5
47.0
76.5
23.5
Eitin-Ene-400m
59.4
55.5
69.9
41.8
82.6
45.6
48.4
77.2
22.6
839
XL Models (750-1565M parameters)
840
DeBa-vl-xl
59.5
56.4
42.5
45.7
47.2
77.1
841
DefBa-V2xxX*
60.5
57.4
71.7
44.4
82.4
46.5
47.7
78.3
30.9
842
Eitin-Ene-Ib
60.4
56.0
72.2
41.9
83.3
46.4
50.1
77.7
20.4
843
Table 5: MTEB v2 English results. Class. = Classification, Clus. = Clustering, Pair. Class. = Pair Classification, Rerank. Reranking, STS Semantic Textual Similarity, Summ. Summarization. DeBERTa v2 XXL did not converge with the standard learning rate sweeps, so we used lower learning rate in order help it to converge.
844
845
846
847
849
850
851
852
853
854
856
857
858
859
860
861
862
863
16

---

Under review as a conference paper at ICLR 2026
864
865
866
867
868
869
870
871
872
874
875
876
877
Single Sentence
Paraphrase and Similarity
Natural Language Inference
878
Model Name
CoLA
SST-2
MRPC
STS-B
QQP
MNLI
QNLI
RTE
Avg
879
XXS Models (11-25M parameters)
BERT-mini TinyBERT
33.9
88.3
83.8
86.4
89.3
77.2
85.4
67.1
76.4
881
22.4
91.2
87.5
87.8
89.4
80.9
88.4
68.2
77.0
882
Sitin-Enc-11
43.9
91.2
86.0
87.2
89.8
79.5
87.3
69.0
79.2
883
XS Models (28-33M parameters)
884
BERT-small
44.8
90.1
83.1
87.6
90.1
79.2
88.2
69.3
79.0
885
MiniLM L12
59.1
93.3
91.2
89.2
91.5
85.6
91.9
74.7
84.6
886
Fttin-Fne-32mm
57.4
92.0
89.7
89.5
91.0
83.4
90.7
74.7
83.5
887
S Models (68-82M parameters)
DistilBERT
56.9
92.2
86.8
87
90.8
82.7
89,5
66.1
81.5
889
DostilRoBERTa
61.9
93.1
88.9
91.5
91.7
83.8
890
Ettin-Enc-68m
64.8
94.4
92.2
91.1
91.9
87.0
92.9
83.8
87.2
891
Base Models (86-150M parameters)
892
BERT-base
59.0
93.1
89.5
89.4
91.4
85.4
91.6
78.2
84.7
893
ModemBBERT-base
65.1
96.0
92.2
91.8
92.1
89.1
93.9
87.4
88.4
894
Ettin-Enc-- 1mcm150m
66.9
95.8
92.6
92.2
92.4
89,2
94.0
87.7
88.9
895
Large Models (305-395M parameters)
896
BERT-large
56.2
93.3
87,8
90.6
90.9
86.3
92.8
83.8
85.2
897
ModemBBEKT-larg
71.4
97.1
91.7
92.8
92.7
90.8
95.2
92.1
90.4
898
Ettin-Fnc-400m
71.3
96.7
93.6
92.7
93.0
91.3
95.2
92.8
90.8
899
XL Models (750-1565M parameters)
900
DeBERTa-v2-XX
75.3
97.1
91.7
92.5
92.6
91.7
95.9
89.2
90.7
901
DeBERTa-v2-XXI.
71.6
91.2
96.0
902
Ontin-Ene-o
74.4
97.1
94.4
93.2
93.0
91.8
96.0
93.1
91.6
903
904
Table 6: GLUE benchmark results across model sizes and architectures. DeBERTa v2 XXL was run for 300+ GPU hours before ruling it out due to it's large size (> 1.5B), results are incomplete.
905
906
907
908
909
910
911
912
913
914
915
916
917
17

---

Under review as a conference paper at ICLR 2026
918
919
920
921
922
923
924
925
926
928
929
930
Model Name
ARC
HS
LMB
OBQA
PIQA
SIQA
SciQ
TQA
Avg
931
XXS Models (17M parameters)
932
933
Ettin-Enc Sine-from-Dec-
27.7
27.2
23.4
31.4
56.0
34.6
45.9
0.5
51.2
52.7
35.1
Ettin-Enc-I 1et--Ene-77m
28.3
26.4
24.1
34.4
44.0
0.1
52.6
52.7
35.1
Dec-From--Enc
22.7
56.1
70.9
35.7
53.4
53.8
36.7
935
Ettin-Dec- Dec-I7m
21.3
27.1
23.0
27.2
57.7
71.1
35.4
2.6
50.9
48.0
36.4
936
XS Models (32M parameters)
937
See-ffoo--Deeee
33.8
34.7
45.6
0.1
53.2
50.9
36.0
938
Ettin-nne322m
28.7
28.0
33.6
34.8
56.7
34,4
41.4
0.2
51.4
56.4
36.6
Ettin-Des Mein-DDe-FFoi DDce-From-FFe Enc-32M
20.5
28.3
27.7
27.0
58.1
77.2
36.0
3.0
50.2
52.7
38.1
939
2eiin-Dee:32m
23.5
28.5
28.5
28.2
57.7
77.5
36.4
3.8
53.1
50.2
38.7
940
Small Models (68M parameters)
941
Ettin-En fwwo--rr--eeee
30.0
30.4
31.8
33.6
57.1
36.1
55.3
1.9
51.1
52.7
38.0
Ettin-Ene Ene-68m
29.5
31.6
36.1
35.4
58.4
35.6
49.6
1.1
51.3
62.6
39.1
943
Siin-Dee-from--ere
24.8
31.9
35.8
29.4
60.7
84.6
38.3
5.8
53.1
56.0
42.1
Seii--Dee-88nn
25.3
33.4
35.2
29.4
61.8
83.2
38.8
5.6
50.1
55.3
41.8
944
Base Models (150M parameters)
945
946
Ettin-Eng Enc-from- cee-f-oo-----llll
1ec
33.5
36.3
39.2
34.4
63.9
39.7
74.3
4.7
51.5
59.3
43.7
Sriin-Enc-1500
32.5
36.5
41.6
63.0
38.5
59.8
1.6
54.9
63.0
42.9
947
Seinn-Deefrom-FFF Enc-150m
25.0
39.4
84.7
40.4
52.9
43.6
948
Ettin-Dec 150m
28.6
40.3
43.2
29.2
66.6
89.6
40.1
11.2
53.7
59.0
46.2
Large Models (400M parameters)
950
Enc-froms Hoom-Dec-400m
38.6
43.4
70.4
56.4
68.9
48
951
Mitin-Ene4400m
35.6
46.8
50.5
38.0
64.7
43.9
65.6
6.4
59.7
70.7
48.2
Ettin-Dec-free Dec-from-Ens- Enc-400m
29.9
45.8
46.4
33.6
66.9
92.1
45.3
13.3
53.9
63.7
49.1
952
Etii- Dein-Dec-400m
33.6
54.3
52.3
34,4
71.0
91.8
45.5
18.3
57.6
71.8
53.1
953
XL Models (1B parameters)
954
Ettin-Ene Enc-from- www--frro-e---- 1B
42.4
53.0
49.3
39.2
70.0
46.3
74.9
14.9
62.3
73.3
52.5
955
Mitin-Ene-1l
37.3
52.3
54.0
38.4 4
67.6
46.3
64.5
7.6
63.2
75.8
50.7
956
Ettin-Dec-fine Boot-Dee-from-------
32.5
52.5
49.1
35.8
69.9
93.1
48.5
13.1
58.6
69.2
52.2
Ettin-Dee-1B
39.7
62.9
58.4
41.6
74.4
93.8
48.2
29.3
62.7
79.1
59.0
957
958
Table 7: Performance comparison of all models evaluated on generative tasks. Enc-fror Enc-from-D from-Dec are trained with MTNP from decoders, while Dec-from- are encoders trained with CLM.
959
960
961
962
964
965
966
967
968
969
970
971
18

---

Under review as a conference paper at ICLR 2026
972
973
974
975
976
977
978
979
980
981
982
Model Name
Retrieval (nDCG 10)
MNLI (Accuracy)
Generative Avg
983
XXS Models (17M parameters)
984
Ettin-Enc--777
30.93
79.5
35.1
985
Ettin-Dece
29.11
77.6
36.4
986
Seinn-En-ffoo--Dee 1m
31.01
77.7
35.1
987
Ettin-Dec-frrod Dec-from-Enn Eee--Enc--- 7m
28.52
78.8
36.7
988
XS Models (32M parameters)
989
990
Ettin-Enc-32m
35.13
83.4
36.6
Ettin-Dec-32m
32.93
80.4
38.7
991
Sutin-Ene--rom--ec-3311
34.66
80.9
36,0
992
Siin-Dee-from-Ene--22
32.32
82.6
38.1
993
994
Small Models (66-70M parameters)
995
Ettin-Enc-663 Enc-68m
38.17
87.0
39.1
Dec--Dee--mm
36.12
83,9
41.8
997
37.87
83,9
38.0
998
Fitio-DDe-from--hn--68n
36.31
85.8
42.1
999
Medium Models (150M parameters)
1000
Ettin-Ene- Enc- 150m
39.97
89.2
42.9
1001
Ettin-Dee-1555
37.71
85,6
46.2
1002
Sttin-Enc-from: Deee-Dec-1100 50m
39.49
85.8
43.7
1003
Ettin-Dec-fromm froe-ffo---e------ 150m
37.55
86.8
43.6
1004
1005
Large Models (400M parameters)
1006
Ettin-Enc-400m Ettin Enc-400m
42.24
91.3
48.2
1007
Ettin-Dec-400m
39.93
88.2
53.1
1008
Sttin-Ene-from-De-4400
41.44
87.6
48.4
1009
Eitio-Dee-from-FFn-4400m
39.69
89.4
49.1
1010
XI. Models (1B parameters)
1011
Ettin-Enc-Ib
43.35
91.8
50.7
1012
Ettin-Dec-1b
41.70
89,9
59.0
1013
43.24
89.0
52.5
1014
Ettin Sein-Dec-frome froc--m-----
40.77
90.5
52.2
1015
1016
Table 8: Table version of Figure
The generative eval breakdowns can found Table
1017
1018
1019
1020
1021
1022
1023
1024
1025
19

---

Under review as a conference paper at ICLR 2026
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
WinoGender All
WinuGender Gotcha
1041
Model Name
Overall
Female
Male
Neutral
Overall
Female
Male
1042
XXS Models (17M parameters)
1043
Eltin- from- Dec- 17mm
50.8
$1.7
50.0
50.4
45.8 4.
$5.0
Emtin Enc-17m
50.6 + 19
500 +
508 + 3.2
503 +
20.0 + 3.2
46.7 + 4.6
53.3
4.6
1044
Etii: Dec from Enc- 17mm
69. 1:
500 3.22
50.0 1 3.2
19.6 3.32
19.6 1: 3.2
47.5 1.6
51.7
17m
1045
Models parsmeters)
1046
Etii: Enc from Dec 32m
53.6 1:
52.5 3.3.2
53.8 1: 3.2
$4.6
53.3 1:
50.0 1.6
56,7
1047
3:33
3.2
1048
Ettin Des- from-kno----- 32m
51.1 +
51.2 4-22
512
504 4.1
$1.7
49.2 4.6
54.2
Ettin Dec 32m
50.8 1: 19
504 1:3.
508 1: 3.2
51. 3.3.
50.0 1: 3.2
30.0 1: 4.6
50.0 4.6
1049
Small Models (66-70M parameters)
1050
Ettin Enc droes Dee 68m
51.9 +
52.5 4-3.2
51.7
51.7 4-1.1
$1.2 4 3.2
53.3 4.6
49.2
1051
Ettin Enc 68m
56.1 1: 19
558 1:3.2
$6.7 1: 3.2
55.8 4
56.7
1 3.2
56.7 1: 4.5
56.7 1: 4.5
Ettin Dec Crum-Enn-688u 68m
51.8 1:
$1.7 3:3.
52.1
2.1:::3 3.2
$1.7 4 3.2
50.8
1: 3.2
51.7 1 4.6
50,0
1052
1053
Mediom Models (150M parameters)
1054
Ettin Ene-from Dec- 150m
52.8 1:
50.8 3.2
54.2 1: 3.2
$3.3 3.2
52.1 1.2
47.5 1
56.7 4.5
1055
Ettin Dec from Enc 150m
53.3
$2.9 3:2
$2.9
54.2 3.2
52.1
55.8
48.3
1056
Bdin- Dec- 150m
$4.7 1: 19
5333
3.3.3
55.8 1: 3.2
$5.0 3.2
$2.9 1: 3.2
36.7 =2 4.5
49.2 4.6
1057
Targe Models (400MT parameters)
1058
Ettin: Enc from Dec 400m
55.1
55.4 3:2.2
54.6
55.4
55.4
53.3 4.6
$1.5
Ediin Enc-400m 100m
70.3 +: 1.7
68. 1:300
70.8 1: 29
71.2 : 2.9
69.2 1: 3.0
69.2 2: 42
69.2 1: 4.2
1059
Ettin Des from
$6.0 +:
538 4:3.2
55.0
53.3 3.322:3.2
4.6
1060
XL. Models (1B parametens)
1061
1062
Ettin Enc droms
57.9
56.3 3:3.2
58.3
58.8 3.2
54.2 4
48.3 4.6
1063
Edtin- Dec-frum-Ene-
55.8
1
56.7
54.2
54.2
54.2
Ettin Dec- 1B
$6.7 +:
1.8
36.3
4.32
55.4
4 3.2
57.9
4.3.2
52.1 4:
3.2
3088
53.3
4.6
1064
1065
Table 9: WinoGender accuracy results (values: Accuracy % + Std Error %). Results taken from the
1066
Eleuther Al harness. Many of the small models do get above random performance (50%).
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
20

---

Under review as a conference paper at ICLR 2026
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
Female
Neutral
1092
1093
Ettin-Encoder
100%
1094
23%
1095
73%
8%
16%
1096
50%
9%
1097
25%
51%
1098
0%
17M
32M
66M
150M
400M
18
1099
Ettin-Decoder
1100
100%
14%
15%
1101
30%
75%
10%
1102
Prediction Probability
10%
1103
61%
70%
25%
54%
54%
1104
1105
0%
17M
32M
66M
150M
400M
18
1106
Encoder-from-Decodee
1107
100%
13%
21%
1108
15%
11%
10%
1109
50%
1110
25%
60%
72%
85%
60%
1111
17M
32M
150M
400M
15
1112
Decoder-from-Encoderr
1113
1114
29%
18%
17%
31%
75%
1115
50%
1116
67%
76%
68%
1117
1118
0%
17M
32M
66M
150M
400M
18
1119
Model Size
1120
1121
Figure 3: Full gender pronoun predictions results the Gotcha split of WinoGender (Rudinger et al., 2018), a 50/50 stereotypical split. We see that encoder models are more likely to use gender neutral pronouns whereas both are biased towards male pronouns.
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
21

---

Under review as a conference paper at ICLR 2026
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
Model Name
Total Params
Embed Params
Non-Embed Params
1144
XXS Models (7-17M parameters)
1145
Pythia- 14m
7.6M
6.4M
1.2M
1146
BERT Tiny
11.2M
7.9M
3.2M
1147
TinyBERT
14,4M
9.7M
4.7M
1148
Ettin-170 17m
16.8M
12.9M
3.9M
1149
XS Models (28-33M parameters)
1150
BERT Small
28. 28.8M
15.9M
12.9M
1151
Ettin-32m
31.9M 33.4M
19.3M
12.5M
1152
MiniLM L12
11.9M
21.4M
1153
Small Models (68-82M parameters)
1154
DistiBBERT Base
66.4M
23.8M
42.5M
1155
Ettin-68m
68. MM
25.8M
42.4M
1156
DistilGPT2
81.9M
39.4M
42.5M
1157
DiRRil
82.1M
39,0M
4
1158
Base Models (123-150M parameters)
1159
BERT-basc
109.5M
23.8M
85.6M
1160
Pythia-1600 160m
123.7M
38.6M
85.1M
1161
SmolLM2-13355 12-13
134.5M
28. 3M
106.2M
1162
ModeenBERTT-aaee
149.0M
38.7M
1
1163
Euiin1100
149.0M
38.7M
1
1164
Large Models (353-395M parameters)
1165
BaRT-large Pythia-410m
335.
31,8M
303. 4M
1166
353.8M
51.5M
302.3M
1167
SmolLM2-3666
361.8M
47.2M
314.6M
1168
ModemBBERT-laree
394.8M
51.6M
343.2M
Fitin-400m
394.8M
51.6M
343.2M
1169
1170
XL Models (884M-1.2B parameters)
1171
DeBERTa v2 XLarge
884.6M
197.6M
687.0M
1172
Pythia Ettin- IB
908.8M
103.0M
805.7M
1173
1028. 1M
90.3M
937.8M
OLMo IB 0724
1176.8M
103.00
1073.7M
1174
Llama 3.2 IB
1235.8M
262.7M
973.1M
1175
1176
Table 10: Parameter breakdown of language models organized by size categories. Models are grouped by total parameter count and show the distribution between embedding and non-embedding parameters across different architectures. Parameter counts are the same for Ettin encoders, decoders, and cross-objective trained versions.
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
22

---

Under review as a conference paper at ICLR 2026
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
Parameter
Value
1200
Vocabulary Size
50,368
1201
Max Sequence Length
1024->7999
1202
Tokenizer
ModemBERT
1203
Attention Layer
RoPE
1204
Attention Dropout
0.0
1205
Attention Output Bias
false
1206
Attention Output Dropout
0.1
1207
Attention QKV Bias
false
1208
Transformer Layer
prenorm
1209
Embedding Dropout
0.0
1210
Embedding Norm
true
1211
Final Norm
true
Skip First PreNorm
true
1212
Embedding Layer
sans sans_pos
1213
MLP Dropout
0.0
1214
MLP Input Bias
false
1215
MLP Layer Type
GLU
1216
MI.P Output Bias
false
1217
Normalization
LayenNorm
1218
Norm Epsilon
le-12
1219
Norm Bias
false
1220
Hidden Activation
GELU
1221
Head Pred Activation
GELU
Activation Function
GELU
1222
Padding
unpadded
1223
Rotary Embedding Base
160,000.0
1224
Rotary Embedding Interlcaved
false
1225
Allow Embedding Resizing
true
1226
Sliding Window
128
1227
Global Attention Every N Layers
1228
Unpad Embeddings
true
1229
1230
Table 11: Common pre-training configuration parameters across all models
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
23


Reviews:
[{"review_id": "oTQWeoVqc3", "reviewer": "Reviewer_Mrcc", "summary": "Authors opensource pairs of encoder-only and decoder-only models along with training pipeline. They explore and compare the performance in classification, retrieval, and generative task.", "questions": "I believe such opensource efforts should be encouraged in the community. I am willing to increase my score if W1 is properly addressed by proper experiments. And would like to hear author's arguments on not conducting experiments at a larger scale for encoder-only models.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The shift from decoder to encoder on LLMs is still not scientifically clear. This paper present paired encoder and decoder models for people to study on and has done some initial exploration.\n2. The paper is fully opensource in terms of weights and training pipeline.", "weaknesses": "1. There are research on turning decoder-only models to encoders like [1], which authors may need to take into account when claiming “encoder models are better at classification”. The possibility of “with proper finetuning, decoder-only model outperforms encoder-only model on non-generative tasks” cannot be ruled out by the current set of experiments. Therefore some statement could be premature/misleading.\n2. Authors’ efforts in opensourcing large-scale encoder models (compared to other encoder-only models) are unprecedented. However, compared to some other alternatives to decoder-only LLMs (e.g. MAMBA, diffusion LLM), the current efforts lack a 7B-scale model, which is (in my opinion) usually considered as the smallest usable model for fair comparison with decoder-only LLMs.\n[1] LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders. BehnamGhader et al. COLM 2024\n[2] Falcon Mamba: The First Competitive Attention-free 7B Language Model, Zuo et al. Arxiv\n[3] Large Language Diffusion Models. Nie et al. NeurIPS 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2gdOyZUwKw", "reviewer": "Reviewer_dWKC", "summary": "The paper presents a set of open-source and open-data text models, consisting of encoders (trained with a masked language modeling objective) and decoders (trained as autoregressive language models) of sizes from 17M to 1B parameters.  The main goal is to enable direct comparison of encoders and decoders on a variety of tasks:  classification and retrieval tasks, where encoders tend to be preferred, and generation tasks where causal LMs are typically best.  For each model size the authors train a pair of encoder and decoder trained with the same hyperparameters, as well as an encoder further trained with a causal LM loss and a decoder further trained with MLM.  Whereas previous such comparisons have compared models of varying sizes and training setups, this work allows for more direct comparison.  The main findings are that (1) as expected, encoders outperform decoders of the same size on classification and retrieval tasks and vice versa for generation tasks and (2) continued pre-training with the other loss (e.g. decoders further trained with MLM) does not undo this trend.  As a case study of additional analyses that this model family enables, the authors also study the distribution of gender pronouns for models of different type and size, finding for example that encoders tend to favor more neutral pronouns and that the gender distribution changes with model size.\n\nThe main contribution of the paper is the open models themselves.  The findings are interesting and demonstrate the value of the resource, but are not very far-reaching.  I therefore consider this paper to be mainly a \"resource\" paper.  A natural question is whether the resource rises to the level of a research contribution and therefore is worthy of a conference publication.  I believe the contribution is important and am recommending acceptance.  That being said, the paper could make a more convincing research contribution if it provided more detail and analysis, beyond just the models themselves (see below under \"weaknesses\"), and for this reason my recommendation is marginal acceptance.", "questions": "- Did you consider including larger models, say up to 3B or 7B?  The work is valuable without it, and to some extent the maximum size is arbitrary.  However, several B parameters is a fairly typical size for research models, and it would be interesting to know whether there is any \"phase transition\" as model size increases further.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A large set of open models trained on open data makes it possible to make comparisons that were previously only approximate (e.g. comparing encoders an decoders of different sizes, or of similar sizes but with otherwise very different training conditions).\n\n- The findings are interesting.  It is especially interesting to see that decoders can't be easily \"converted\" into equally strong encoders via continued pre-training, and vice versa.\n\n- The paper is generally presented well.  It provides good background on existing models and findings and lays out the new work clearly.", "weaknesses": "- The paper should provide more detail about how the hyperparameters were chosen.  For example, it is not obvious to me that using identical hyperparameters for encoders and decoders is the best choice.  Ideally there would be some tuning done and a study of the sensitivity to hyperparameter choices.  Could the results be improved with different choices of hyperparameters?  Could they change enough to modify the findings?  \n\n- The results are given without any analysis into why the findings are as they are, especially the less obvious ones related to continued pre-training and gender pronouns.  I recognize that it may not be easy to do, but I would expect some attempt at understanding the results and studying how they may be affected by aspects of the data distribution, number of training iterations, or other variables.  This is related to the first point above, but has more to do with explanation of the findings.\n\n- A minor weakness:  While the paper is generally well-written, there are quite a few minor grammatical errors throughout.  A thorough proofreading would help.\n\n- Also minor:  Citations should be provided to support the statement \"Part of this gap is due to the sentiment within the community that decoders can be adapted for use in tasks that were once predominantly encoder-focused (e.g. classification, embeddings)\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cqFwyknnIN", "reviewer": "Reviewer_npLe", "summary": "This paper introduces SEQ vs SEQ (ETTIN Suite), an open-source suite of paired encoder-only and decoder-only language models trained with identical data, architecture, and training recipes. The suite spans models from 17M to 1B parameters, enabling an “apples-to-apples” comparison of encoder and decoder architectures across scaling trends. The authors also provide 200+ checkpoints, token-level training data ordering, and open datasets, making this a valuable resource for community research. Like existing works, encoder-only models outperform decoders on classification and retrieval tasks (e.g., GLUE, MTEB, MS MARCO). Decoder-only models dominate generative tasks (e.g., ARC, HellaSwag, TriviaQA). Adapting models to the opposite objective through continued cross-objective pretraining (e.g., MLM-trained decoders, CLM-trained encoders) is ineffective can not beat the single native baseline. Additionally, the paper includes a case study on gender bias, showing diverging pronoun prediction behaviors between encoder and decoder models.", "questions": "1. Why are the results of TQA in Table 4 exceptionally low for some models? \n2. For smaller models (<=70M), adapting encoder to decoder or decoder to encoder seem little gap from native pretraining. If we increase the tokens for adapting phase, would that be helpful? Is there any scaling behavior that can bridge the gap between adapting model performance and native one?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. An apple-to-apple suite of encoder and decoder models trained with identical recipes and data. This eliminates confounding factors present in previous studies. \n2. All model weights, checkpoints, and training data orders are released. This enables reproducibility and further analysis.\n3. Both encoder and decoder models achieve state-of-the-art performance for their size, outperforming ModernBERT (encoder) and SmolLM2/LLaMA 3.2 1B (decoder) baselines.\n4. Demonstrates that continued training of decoders with MLM (or encoders with CLM) fails to match native training, even when using 50B tokens of additional training. \n5. Covers 6 model sizes (17M → 1B). This allows to study scaling trends across both architectures.", "weaknesses": "1. Cross-training adapting decoder into encoder often applies masked language modeling. Is it helpful also add masked next token prediction or unsupervised contrastive learning like [1].\n2. The evaluation is centered on GLUE, MTEB, and knowledge-based benchmark. Can you also math or coding benchmarks to see the reasoning abilities of the models.\n\nReferences:\nBehnamGhader et al. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders. COLM 24", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)