Avey-B
Authors: Devang Acharya
Mohammad Hammoud
Venue: ICLR 2026 Poster (2026)
Keywords: Bidirectional models
Transformer-based encoders
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a new bidirectional attention-free encoder

Abstract: Compact pretrained bidirectional encoders remain the backbone of industrial NLP under tight compute and memory budgets. Their effectiveness stems from self-attention’s ability to deliver high-quality bidirectional contextualization with sequence-level parallelism, as popularized by BERT-style architectures. Recently, **Avey** was introduced as an autoregressive, attention-free alternative that naturally admits an encoder-only adaptation. In this paper, we reformulate Avey for the encoder-only paradigm and propose several innovations to its architecture, including decoupled static and dynamic parameterizations, stability-oriented normalization, and neural compression. Results show that this reformulated architecture compares favorably to four widely used Transformer-based encoders, consistently outperforming them on standard token-classification and information-retrieval benchmarks while scaling more efficiently to long contexts.

Published as a conference paper at ICLR 2026
AVEY-B
Devang Acharya' and Mohammad Hammoud" Avey AI
(dacharya, maney ai
ABSTRACT
Compact pretrained bidirectional encoders remain the backbone of industrial NLP under tight compute and memory budgets. Their effectiveness stems from self-attention's ability to deliver high-quality bidirectional contextualization with sequence-level parallelism, as popularized by BERT-style architectures. Recently, Avey was introduced as autoregressive, attention-fies free alternative that naturally admits an encoder-only adaptation. In this paper, we reformulate Avey for the encoder-only paradigm and propose several innovations to its architecture, in- cluding decoupled static and dynamic parameterizations, stability-orienter nor- malization, and neural compression. Results show that this reformulated archi- tecture compares favorably to four widely used Transformer-bes based encoders, con- sistently outperforming them standard token- classifffictnnn and information- retrieval benchmarks while scaling more efficiently to long contexts.
INTRODUCTION
Pretrained bidirectional Transformer encoders, most notably BERT (Devlin al. 2019), have been especially impactful in resource-constrained, application-specific settings, where compact models can be efficiently fine-tuned for downstream tasks and deployed under strict latency and memory budgets. Unlike unidirectional Transformer decoders, bidirectional encoders condition each token on both left and right contexts, yielding fully contextualized representations that improve disam- biguation and translate into stronger performance on certain discriminative tasks (e.g., classification, retrieval, and extractive
-answernn)) (Liu et al. 2019 Wang et al. 2019b a Karpukhin al.] 2020 Khattab & Zaharia 2020 Rajpurkar et al. 2016 2018). Since BERT's introduction, such en- coders have seen broad and sustained adoption across academia and industry (Muennighoff et al.
2023 Thakur et al.] 1021 Santhanam et al. 2022 Wang al. 2022 Su et al., 2003), particularly
targeting high- thgh-throughhut. high-precision, and budget-constraned applications
(Lan et al. 2020
Sanh et al. 2019 Sun al. 2020 Jiao al. 2020).
The BERT family's success research and industry was enabled by the Transformer (Vaswani et al,, 2017]. whose self-attention mechanism affords bidirectional contextualization while maintaining high parallelizability. However, the quadratic time and memory costs of full self-attention remain central bottleneck (Tay et al. 2022 Munkhdalai et al., 2024). limiting practical extension of context windows in cost-sensitive dep deployments. A large body of work has sought to mitigate this bottleneck (e.g., via using linear attention (Katharopoulos
al. 2021 Peng et al. 2025) and RNN-inspired architectures (Gu et al.] 2021 Gupta et al.) [022 Fu et al. 2022 Gu & Dao 2023), ), but little of it has been adapted to the bidirectional, encoder-only paradigm. Mean- while, BERT itself was modernized through larger pretraining corpora, architectural refinements (e.g., FlashAttenioon (Dao et al., 2022), SwiGLU activations (Shazeer [2200), and RoPE positional encoding (Su et al. )) and new pretraining and fine-tuning strategies (Liu et al. 2019; Portes 2023 Warner et al. 2025) among others.
al. 2020 Choromanski
et al.
Most recently, Avey (Hammoud & Acharyal 2025) was introduced as autoregressive architecture that departs from both Transformer- and RNN-based designs. It partitions a sequence into splits, ranks and retrieves the most relevant splits for each current (or target) split, and applies dynam- ically parameterized neural processor to contextualize them. By decoupling context width from
"quul contribution.

---

Published as a conference paper at ICLR 2026
sequence length, Avey enables efficient long-range interactions and extrapolation far beyond its training window, thereby facilitating practical context extensions under realistic compute budgets.
Concretely, Avey is composed of two principal components, ranker and neural processor. For each target split, the ranker selects the top-k most relevant splits from the input sequence, which are then jointly processed by the neural processor. The neural processor comprises three modules, an enricher, a contextualizer, and a fuser. The enricher improves token representations by expanding their learnable features via position-wise neural network. The contextualizer an embedding- -wis neural network with dynamic parameterization and cosine-similarit--bal teractions between tokens in the target split and those in the retrieved splits. Lastly, the fuser in- tegrates the resulting contextualized features with a subset of uncontextualized features preserved through a partial-embedding bypassing mechanism.
selectivity, enabling in-
Although originally formulated for causal language modeling, Avey's rosin--simllarity-baeeee se- lectivity and learned cross- cross-embediee embeddinn linear transformation make it naturally amenable to a bidirec- tional, encoder-style adaptation. In this paper, we introduce Avey-B, bidirectional reformulation of Avey for the encoder-only setting, and compare it empirically against widely used and recently introduced Transformer-based encoders, namely, BERT (Devlin et al. 2019), RoBERTa (Liu et al., 2019), ModernBERT [Warner et al. 2025), and NeoBERT (Breton et al. (025). We further propose architectural advances that enhance its effectiveness and efficiency, including: (1) decoupled static and dynamic parameterizations: (2) a mability-oriented row-normalized similarity scores in the dy- namic layers; and (3) compression module that reduces retrieved context before contextualization in the neural processor.
To elaborate, Avey-B contextualizss tokens via either learned static linear projection or a dynamic similarity matrix computed from cosine similarities, in any given layer. This contrasts with Avey, which multiplicatively couples the learned projection with cosine scores element-wise in every layer. By decoupling the static and dynamic parameterizations, Avey-B avoids destructive interactions between fixed weights and input-dependent scores, most notably inversion effects where token highly similar to a neuron's current token is forced to contribute less than less-similar one. In addition, we normalize the cosine scores each position by the sum of that position's 's scores over all tokens, stabilizing training and consistently improving downstream task performance.
Alongside, we observe that extending Avey to the bidirectional paradigm without some modifica- tions may introduce scalability issue. Specifically, in the original design of Avey, each split is concatenated with its top-k relevant splits and jointly contextualized in a single pass through the neural processor. Performing this for every split inflates the input size to roughly k times the num- ber of tokens, substantially increasing processing time. In the autoregressive regime, this overhead is mitigated by training short context widths, leveraging Avey's ability to extrapolate well beyond that. It is also tolerable during inference because only the most recent split is contextualized with its top-k splits to generate the next token. In the bidirectional inference setting, however, this strategy is infeasible, since all splits must be contextualized to produce complete token-level representations.
Building on this observation and recognizing that inference efficiency is critical for encoder mod- els (especially in industry where they are commonly used (Raghavan 2020 Zhu 2019: Guo et al.) 2020 Warner al., 2025)), we introduce a neural compression scheme in the ranker. More pre- cisely, we compress each split together with its top-k retrieved splits back to the size of a single split via a learned linear projection. As a result, the neural processor contextualizes only as many tokens as in the original input sequence, avoiding redundant computations over the appended top-k splits. Because the neural processor operates on each split independently, Avey-B achieves higher throughput than Transformer-based encoders, while preserving high accuracy across wide range of downstream benchmarks.
To summarize, main contributions in this paper are as follows:
We propose Avey-B, a bidirectional encoder architecture that capitalizes on Avey by de- coupling static and dynamic parameterizations and introducing a lightweight normalization scheme for dynamic contextualization.
We redesign Avey's ranker to compress each split's top-k retrieved context into fixed token budget, making the neural processor's 's per-split compute independent of while pre- serving the benefits of retrieving larger relevant token sets via increasing kt
2

---

Published as a conference paper at ICLR 2026
We conduct extensive design-choics and ablation studies to identify the most effective chitectural configuration and demonstrate how each proposed idea contributes to the per- formance gains of Avey-B over the original Avey architecture.
We show that Avey-B outperforms BERT (Devlin et al., 2019) and NeoBERT (Bre- ton et al. 2025) across all the evaluated benchmarks, and consistently surpasses RoBERTa la (Lu et al. 2019) and ModemBERT (Warner et al. 2025) on token-clasiffiation
and information-retrieva tasks, despite being pretrained ~ -1x fewer tokens than Mod- ernBERT.
We illustrate that Avey-B scales efficiently with sequence length, yielding substantially lower latency than Transformer-base encoders. Across 128-96 K tokens, Avey-B B is con- sistently faster than all the evaluated Transformer baselines, and its advantage widens with sequence length N. For example, at N = 96 K, Avey-B outpaces ModernBERT and NeoBERT by 3.38x and 11.63x, respectively.
We characterize Avey-B's scaling behavior via a power-law fit, T(N) x N and show that it exhibits markedly smaller decay exponent (a R 0.44) than ModernBERT (a s 0.77) and NeoBERT (a ze 0.81), indicating that Avey-B's s throughput decreases at roughly half the rate of ModemBERT (and even more slowly relative to NeoBERT) as sequence length increases.
We release the full implementation and pretrained checkpoints of enabling reproducibility and fostering future research.
Avey-B (see Section
The remainder of the paper is organized as follows. Section 2 reviews related work, and Section provides background on Avey. Section 4 describes the Avey-B architecture, including its bidirec- tional contextualization, decoupled parameterization, and neural compression. Section [ presents the experimental setup, design choices and ablations, as well as effectiveness and efficiency results. Finally, we conclude in Section
2 RELATED WORK
The introduction of GPT (Radford al., 2018 2018 marked turning point in large-scale lan- guage modeling, establishing the now-standard paradigm of pretraining Transformer-baas based models on massive unlabeled corpora followed by supervised fine-tuning on task-specific data. GPT op- timized a causal language modeling (CLM) objective, pretraining unidirectional, decoder-only Transformer (Vaswani al. 2017) for next-token prediction. The resulting pretrained model can then be effectively fine-tuned with modest labeled data to broad range of downstream tasks, in- cluding text classification (Wang et al.) 2019b),, natural language inference (Bowman et al.) 2015] Williams et al. 2018), and question answering (Rajpurkar al. 2016 Lai al. (017), to mention just a few. This pretrain- cettrriin-fin-tumee fine-tune paradigm yielded state-of-the-ar performance on these tasks at the time (Radford al. 2018).
BERT (Devlin et al. 2019) extended this paradigm by replacing the unidirectional decoder with a fully bidirectional encoder. Concretely, it introduced two pretraining objectives, masked language modeling (MLM), which reconstructs randomly masked tokens in an input sequence, and next sen- tence prediction (NSP), which models inter-sentence relationships. By contextualizing tokens in both directions, BERT delivered substantial gains over causally pretrained models, particularly on benchmarks such as GLUE (Wang et al., (019b)), MultiNLI (Williams et al. 2018), and SQuAD (Ra- jpurkar et al. 2016). among others.
RoBERTa Liu al. 2019) robustly optimized BERT by retaining its overall architecture while systematically revisiting nearly every aspect of its pretraining setup. Key modifications included removing the NSP objective, pretraining with larger batches and longer sequences, adopting dy- namic masking strategies, and scaling to substantially larger corpora. Building this foundation, DeBERTa (He et al., 2202 at (023) introduced disentangled attention, which separates content and positional information into distinct attention matrices, and improved fine-tuning stability through virtual adversarial training. Together, these innovations further advanced performance on some challenging benchmarks such SuperGLUE (Wang et al.) 2019a).
Subsequent work emphasized both architectural refinements and pretraining efficiency. For exam- ple, MoRaicBERT (Portes et al. 2023) integrated FtashAttention (Dao al. 2022), ALiBi positional
3

---

Published as a conference paper at ICLR 2026
biases (Press et al. 2022 and gated linear units (GLU) (Dauphin al. 2017 Shazeer 2020)
celerate pretraining while maintaining strong downstream accuracy. NomicBERT (Nussbaum et al.
2024) adopted SwiGLU (Shazeer, 2020) and rotary positional encodings (RoPE) (Su et al. 2002
NeoBERT (Breton et al. 2025) combined RoPE, SwiGLU, and RMSNorm (Zhang & Sennrich
2019) with depth-width rebalancing and large-scale pretraining. ModernBERT (Warner et 2025)
pushed this trend further, employing many of these techniques (e.g., ROPE, FlashAttention, and al- ternating global/local attention), supporting context windows for up to 1,192 tokens, and pretraining on multi-triillonntokee corpora.
All of the above models are Transformer-basee leveraging self-attention provide effective bidi- rectional contextualization while maintaining high pretraining parallelism. Recently, a fundamen- tally different architecture named Avey (Hammoud & Acharya 2025) was introduced. Avey attention-fre and can process virtually unlimited sequence lengths (see Section 3). Avey-B capital- izes on Avey to support bidirectional contextualization, mirroring the shift from GPT-style decoder- only to BERT-style encoder-only models in the Transformer l'amil.. We empirically compare Avey-
against BERT, RoBERTa, ModemBERT, and NeoBERT in Section 5
BACKGROUND
The original Avey architecture decouples sequence length from context width by pairing lightweight ranker with a data-dependent neural processor. We next describe both components.
3.1 RANKER
Avey partitions an input sequence of length N into equal-sized splits of S tokens, applying zero- padding if is not divisible by S. For a given current (or target) split, the ranker computes its relevance to each preceding split using the MaxSim operator (Khattab & Zaharia 2020), orders them by their MaxSim scores, and selects the top-k splits for contextualization.
Before contextualization, the MaxSim scores the top-k selected splits are normalized by dividing each score by the maximum among them. Each selected split is then weighted by its normalized score and concatenated with the current split. This wewphh/edd-eeeeiie----pee mechanism prunes irrel- evant global splits and scales the contribution of each retrieved split based on relevance.
Crucially, the ranker is invoked only once per full forward/backward pass, independent of the num- ber of ceural-processoy layers. Matching each split against all preceding splits yields a training-time compute cost of dN) where d is the embedding dimension.
3.2 NEURAL PROCESSOR
The neural processor takes as input the current split together with its weighted top-k retrieved splits and processes them through a multi-layer architecture. Each layer consists of three modules, an enricher, contextualizer, and a fuser.
The enricher is a single-layer, position-wise neural network applied independently to each token embedding. Given C input embeddings arranged as X ICxdd the enricher computes a matrix Z E RC Cmmm (with m >> d) as follows:
Z (XU KXU+b). bb)
where U E Rdxmm is learnable weight matrix, b E RCxm represents biases, and a(-) is an activation function. The output Z partitioned into a head Z, E RCxm,, which is bypassed directly to the fuser, and a tail Z, E CRC, which is forwarded to the contextualizer, with m = my + my This partial-embedding mitigates degradation effects (c.g., over-smoothing), as the number of layers is increased.
bypassing technique preserves raw token-specific features and
The contextualizer operates on the tail Z. Each m-ddmennsonnal tail embedding is split evenly into a gating left half and a contextual right half, yielding Zu E R mi and Z RCx "7 respectively.
-iieediii-iieeeeee Formally, the contextualizer is a layer, embedding- wise network that updates Z,, as follows:
c(Z,) = Zu o d(v o ((Zz)))((Z) Zir + b¹).
(2)
4

---

Published as a conference paper at ICLR 2026
(5) = Winli + W21 + W31n1
sı1e n1
521e2 + $3103
(5)
112
(5)
(9)
n, S11W11e1 S21W2102 S31W31e3
11
W21
W31
R1
f2
n3
n2
2
3
(d)
S11
W21
331W31
111
$21
531
E1
e2
e3
e1
e2
e3
(a) Avey's Coupled Parametrization
(b) Avey-B's Decoupled Parametrization
Figure 1: A simple illustration of coupled (a) and decoupled (b) parameterizations (e; = embedding i; Sij = cosine similarity score between e, and ej: = neuron i, n1 (d) neuron i in dynamic layer d; n, neuron i in static layer and Way weight corresponding to or n.(4) weighted sum of n, (*), respectively).
(4) used in the
where V E RCCC is a learned cross-embedding matrix, @ denotes element-wise (Hadamard) mul- tiplication, (-)) applies row-wise lg normalization (so NV N (Zzr) computes cosine similar- ities between embeddings), and b' is optional bias. Intuitively, each neuron aggregates statically and dynamically weighted contributions from other embeddings, and the resulting update is gated by
Zu The learned matrix V provides position-sensitive mixing, additional positional encodings are required within the contextualizer.
The fuser combines the bypassed and contextualized streams and projects the output back to the model embedding dimension as follows:
f(Z) = ZZ, c,
(3)
where O E (mmm++m is learned projection matrix. As with the enricher, the fuser is applied independently to each token embedding. Its output merged with the enricher's input within the same layer via residual, element-wise addition.
Aggregating the costs of the ranker, enricher, contextualizer, and fuser over L layers yields a training complexity of O(N2d) for input sequence of length N. At inference, only the most recent split is contextualized for autoregressive decoding. reducing the complexity to O(N).
AVEY-B
Avey-B is bidirectional reformulation of Avey. We next elaborate on its architecture and computa- tional implications.
4.1 BIDIRECTIONAL CONTENTUALIZATION
Avey-B BB drops the autoregressive mask in Avey's contextualizer, allowing each token representation to condition on both left and right contexts. Specifically, when split is contextualized with its top-k selected splits, all token interactions are permitted, without any causal masking. This converts Avey into encoder-style architecture while preserving selective global access via the ranker.
4.2 DECOUPLED PARAMETRIZATION
Avey's contextualizer performs an element-wise multiplication between a learned, static weight matrix V and an input-dependent cosine-similarity matrix produced by (Z.r.) (Z.,)) The resulting matrix is then used to linearly combine the input embeddings Zer into contextualized representations. This tight coupling of fixed parameters with data-driven relevance scores can lead to pathological behavior, whereby a token highly similar to a neuron's 's current token may contribute less than less-similar one. This violates monotonicity with respect to relevance, which suggests that a more relevant token should contribute at least as much as a less relevant one, and increasing token's 's relevance should never reduce invert its contribution.
As shown in Equation 2
5

---

Published as a conference paper at ICLR 2026
Fig. (a) demonstrates the issue. If the cosine similarity $21 exceeds 831, then embedding 62 should contribute at least as much to neuron nı 's update as embedding 63. In Avey's 's coupled design, the element wise product with learned weights (e.g., W21 and Wa1) can invert this ordering. In particular, if Wa1 > W21, the effective contributions 821 W21 and 831 W31 may be reversed (especially at inference), weakening evidence accumulation from the most informative tokens.
Avey-B addresses this issue by decoupling the two parameter sources (i.e., learned weights and input-driven similarities) and interleaving them across depth. Concretely, each layer is designated as either static or dynamic. A static layer applies learned linear transformation to the embeddings, while a dynamic layer weights embeddings solely by cosine similarity, Alternating these layers pre- serves monotonicity of similarity- updates within the dynamic layers (i.e., if $21 > $31, token contributes more than token 3), while retaining signed static weights in the static layers to encode inhibitory patterns without overtumiig the monotonic, similarity-respecte updates established by the dynamic layers.
To illustrate, Fig. (b) shows the effect of decoupled parametrization. If 821 531. the dynamic (d) assigns a larger weight to eg than eg. with no learned weights intervening in
layer for neuron ni this similarity- -based update. In the subsequent static layer (assuming an interleaved dynamic-stain static pattern for ni both contributions are scaled uniformly by the same coefficient wm. Hence, the ordering, or at least the magnitude ranking, established by the dynamic layer is preserved. More precisely, if w11 > 0, the ordering is maintained, since $21 > $31 implies W11$21 > W11$31- If, however, W11 < 0, both contributions change sign uniformly, but their relative magnitudes are unchanged, as WW11s21//[w11se
$21/833 > 1.
Crucially, the static layer is mmmilarittyagnosstn Specifically, it cannot modify the dynamic similar- ities 821 and 831, nor can introduce token-specifie, applies a global (possibly negative) gain w11 to the neuron's s aggregate, which may change the over- all sign but cannot alter the magnitude ordering imposed by the dynamic layer. We analyze the effect of retaining signed static weights in Appendix K and observe consistent improvements from allow- ing these possible global negative gains. We further ablate the decoupling design in Appendix H and provide statistical comparison of coupled versus decoupled parameterizations in Appendix
sign flips. At most,
To this end, decoupling static and dynamic computations preserves the monotonicity guarantee within each dynamic layer, while still permitting representation shaping in static layers. Although static layer can change the representations from which subsequent similarities are computed, it can- not modify the scores already assigned by a preceding dynamic layer and therefore does not violate that layer's 's monotonicity. We provide formal proof of this guarantee in Appendix A
Formally, let Z.r E Cxxd denote the matrix of contextual, right-tail components for C' enriched embeddings, where d' is the contextualizer's 's right-tail dimension (see Section more information on all notations). In Avey-B, a static layer applies learned cross-ente coosseembddiigg linear transformation as follows:
Cstatic (Z) = (V Zr + +()),
(4)
where V E CRC is a learned matrix, b(o) E R an optional bias, and (()) is an activation function. Intuitively, each neuron first aggregates linearly the C' embeddings and then applies the pointwise activation a.
In contrast, an Avey-B's dynamic layer computes an input-dependent similarity matrix from Zue and utilizes to mix embeddings as follows:
S = .N.Z.. E RCCx
(5)
S.j = Sij
(row-wise sum normalization),
(6)
ES S +
Cdya (Z) of SZ. + b(4)
(7)
Appendix D provides empirical study of different interleaving strategies.

---

Published as a conference paper at ICLR 2026
Here N()) denotes per-row lg normalization to unit length so that S encodes cosine similarities; e >> is small stabilizer ensuring a positive denominator; b(d) RC is an optional bias, and S is a simple sum-normalized similarity matrix. This row-wise normalization yields a row-stochastic similarity operator (row sums S 1), which bounds per-row gain and mitigates the growth of large singular values through depth, improving numerical conditioning and trainability. In the unnor- malized case (i.e., in Avey), inflated singular values can drive activation and gradient growth with depth, resulting in unstable optimization and degraded generalization. We ablate this normalization technique and show consistent gains over softmax-based and RMS Aty-style alternatives in Appendix
4.3 NEURAL COMPRESSION
A key bottleneck in extending Avey the bidirectional setting the per-split concatenation strategy, whereby each current split is concatenated with its top-k retrieved splits for contextualization. This manageable in Avey's autoregressive inference because only the most recent split is expanded. In the bidirectional regime, however, every split within the contextualizer's s window must be expanded, which inflates the effective sequence length by a factor of ls
To mitigate this, Avey-B introduces a neural compressor within the ranker to condense the con- catenated (A + 1)S-token block back to tokens, where denotes the split size. Specifically, the compressor is an embedding-wwid -wiee neural network that maps the (k+ 1)S input tokens to S represen- tative tokens, effectively distilling cross-split information before passed to the neural processor. To preserve signal from the Block's current split, Avey-B adds residual connection between the compressor output and the split's original S tokens, which improves stability and downstream ef- fectiveness. An ablation studying the impact of this residual on Avey-B's -B's accuracy is presented in Appendix H
Formally, let Xeat E (k+1)5) be the concatenation of a single split with top-k retrieved splits, where d denotes the embedding dimension. Subsequently, Avey-B produces compressed output X Sxd as follows:
X P
where P (x*((++) is learnable matrix that performs linear cross-token transformation, and X replaces Xeat as the input to the neural processor. Because P is a learned matrix, the compressor can preserve globally informative content while discarding potential redundancy, yielding favorable accuracyttroughput trade-off. We study the effect of the compressor on Avey-B's B's accuracy and throughput in Appendix H
As discussed in (Hammoud & Acharya 2025). computation in the neural processor largely domi- nates that of the ranker. As such, when Avey-B reduces the number of tokens contextualized per split from (k+1)S to S, throughput improves by 4.37x x (see Fig. in Appendix H). albeit leaving Avey's asymptotic complexity unchanged (still quadratic with respect to the sequence length N).
5
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
In this section, we compare Avey-B against widely used and recently introduced Transformer-based encoders, namely, BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019), ModernBERT (Warner 2025). and NeoBERT (Breton et al. 2025). We evaluate two Avey-B model sizes, base and large, each pretrained on 180B tokens drawn from the FineWeb corpus (Penedo et al. 2024) Pretraining details and information about all the evaluated models are provided in Appendix B
To assess effectiveness, we adopt the evaluation protocol of Boukhlef et al. (Gisserot-Boukhlef eet al. 2025). targeting breadth across four downstream categories prevalent in practice, includ- ing Sequence Classification (SC), Token Classification (TC), Question Answering (QA), and In- formation Retrieval (IR). Each category is represented by three established benchmarks, namely, MNLI (Williams et al. 2018)) QQP (Wang et al.) 2017), and SST-2 (Socher et al., 2013) under SC; CoNLL-2003 (Sang & De Meulder 2003). OntoNoter (Hovy et al. 2006), and UNER (Mayhew al. 2023 under TC: ReCoRD (Wang al. 2019a), SQUAD (Rajpurkar al. 2016). and SQuAD-
7

---

Published as a conference paper at ICLR 2026
Table 1: Design and masked language modeling (MLM) choices.
Question
Answer
Experiments
What is the most effective arrangement of static (S) and dynamic (D) layers?
Interleaved layers with repeating 5 + D pattern
Appendix
What is the most effective normalization technique within the dynamic layers, Softmax, RMS Norm, Row-wise normalization?
Row-wise normalization
Appendix E
What are the best values for sequence length N, split size S. and top & splits?
N 2048, S 256,
Appendix
Should the ranker operate bidirectionally well?
No
Appendix
What the best masking rate?
20%
Appendix
v2 (Raipurkar et al. 2018) under QA: and MLDR MMulti-Granulariyy 2024), MS MARCO Bajaj al. 2016), and NQ (Kwiatkowski et al. 2019) under [
We fine-tuned benchmarks under SC and TC for 1 epoch, QA for epochs, and IR for 1,000 op- timization steps. For each benchmark, we swept four learning rates {2 10-5, 6 10-5 1 5 x 10" and trained each configuration with 10 independent random seeds. Akin to (Liu al. 2019), the reported results for each model the median scores across seeds the best learn- ing rate SC is evaluated with accuracy, TC and QA with F1 score, and IR with NDCG@ @ 10. Lastly, following Boukhlef er al. (Gisserot-Bei Boukkll et al. 2025), we used linear learning-ratt decay with warmup over the first 10% of steps.
10
To evaluate efficiency, we report both latency (seconds per forward pass) and throughput (tokens per second) as functions of input context length, using fixed batch size of 8. All models are bench- marked on NVIDIA H200 GPUs under identical software stacks and numerical precision settings to ensure fair comparison. These measurements provide direct and controlled assessment of scalability and deployment efficiency between Avey-B and Transformer-based encoders.
5.2 DESIGN CHOICES AND ABLATIONS
Following the methodology established in the Avey work (Hammoud Acharya 2025), we conduct systematic design-choice studies to determine effective architectural contigurations for Avey-B. In addition, we sweep the masking rate to identify a robust pretraining setting. Table I summarizes these investigations and references the corresponding experiments supporting each design decision.
We further conduct comprehensive ablation studies to quantify the contribution of each architectural refinement in Avey-B. In particular, we evaluate the impacts of: (1) decoupling and interleaving static and dynamic parameterizations; (2) incorporating row-wise normalization within the dynamic layers; and (3) integrating neural compressor into the ranker. Appendix H reports the complete results along with detailed analysis.
5.3 EFFECTIVENESS
We now evaluate Avey-B on standard SC, TC, QA, and benchmarks, comparing against BERT, RoBERTa, ModemBERR,, and NeoBERT. For all baselines we consider base and large configura- tions (see Table 3 in Appendix B). except for NeoBERT which is evaluated on its only publicly available size, that is, medium. Table summarizes all the results.
At the hase scale (and medium for NeoBERT), Avey-B surpasses BERT and NeoBERT across all task categories, despite using ~85M fewer parameters than NeoBERT, It also delivers the strongest results on TC and IR, outperforming all Transformer-base based models in both categories. For SC, Avey-B attains the best scores QQP (tied with ModernBERT) and SST-2, while trailing RoBERTa
Beyond all these short-range benchmarks, Appendix M evaluates the long-context capabilities of Avey-B on synthetic needle-in-a-hasstack (NTAH) henchmark, with sequence lengths extending up 96k tokens. 'Appendix further reports cross-sed seed variance analysis, computing the standard deviation (SD) across the 10 independent runs for each moncel-henchmrkk pair. These SD values quantify sensitivity to random ini- tialization and provide additional lens optimization stability and robustness beyond median performance.
8

---

Published as a conference paper at ICLR 2026
Table 2: Effectiveness results for several encoders at different scales (M Medium).
SC
QA
Model
MNLI
QQP SST-2
Avg. CONLL Onta.
UNER Avg.
ReCoRD SQuAD
SQuAD Y2
Avg.
MLDR
MS MARCO
NQ
Avg.
Avey-B
83.58
89.81
92.94
78
93.80
91.10
93.59
11.03
71.41
68.88
62:45
63.83
88.14
83.62
Base
BERT
81,92
88.57
90.91
87.14
90.25
91.08
88.20
82
36,76
72.20
63.99
57.65
57.12
81.15
80.66
73.08
RoBERTa
86.42
89.12
92.78
89.41
90.55
92. 11
88
16
90.27
67.86
80.68
76.62
75.05
56.07
86.17
80.30
74.25
ModemBERT
86.72
89.81
92.32
89.61
92.30
71
92.30
92.78
65.73
80.23
77.36
71.41
51.29
58.09
75.21
72.51
NeoBERT
82.53
88.58
84.60
85.36
87.55
88.85
58.20
37.74
64.42
55.67
39.98
70.76
59.43
56.72
Avey-B
85.00
89.22
94.38
89.75
93.60
94.09
94.32
94.00
58.22
77.30
72:46
09.32
67.05
88.72
86, 24
80.67
Large
BERT
85.08
89.27
02.26
88.87
88.54
90.71
86.09
88.44
52.02
77.93
72.96
67.64
61.08
57.71
65.42
78.07
RoBERTa
90. 16
89.40
94.67
01
.44
91.71
92.70
88.79
91.07
80.86
84.00
83.04
82.63
58.50
89.43
85.91
77.95
ModemBERT 90.53
90.73
95.99
92.41
92.43
99.79
92.92
93.05
73.05
022
79.96
78.34
59.64
88.82
81.36
76.61
and ModernBERT on MNLI. For QA, Avey-B leads on SQuAD-v2 but lags RoBERTa and Modern- BERT on ReCoRD and SQuAD.
At the large scale (and medium for NeoBERT), Avey-B again outperforms BERT and NeoBERT across all task categories. It also offers the strongest results on TC and IR, surpassing every Transformer- baase model. Notably, the Avey-B base model even exceeds all large Transformer encoders on TC and IR (despite also being pretrained on ~11x fewer tokens than ModernBERT, for example). These results highlight Avey-B's advantage for both local, span-sensitive decisions (TC) and long-document encoding (IR).
In summary, at both the base and large scales, Avey-B surpasses BERT and NeoBERT across all evaluated benchmarks and delivers uniform gains over all baselines on TC and IR tasks. We attribute these improvements to key architectural properties. First, TC tasks rely heavily on localized evi- dence within short spans; Avey-B' ss split- based processing, combined with pruning of low-relevance splits and tokens, enhances the signal-to-noiss ratio and sharpens token-level representations. Sec- ond, IR tasks benefit from selectively coupling globally relevant content with its immediate local context when encoding long documents, an inductive bias explicitly enforced by Avey-B's retrieval mechanism. In contrast, fully bidirectional processing over all tokens, in Transformer-based en- coders, tends to admit distractors and dilute relevance as sequence length increases.
5.4 EFFICIENCY
Avey is recent architecture and still lacks fused-kernel (CUDA/Triton) implementation. Conse- quently, we evaluate its inference efficiency using torch. comp le, which performs graph cap- ture and backend code generation but stops short of handcrafted, specialized fused kernels available to mature Transformer encoders. We denote this configuration as twey-B-torch-comm tify compilation gains and offer a reference without compiler optimizations, we also report Avey-B's efficiency in the eager PyTorch mode, referred to as Avey-B-eager. For Transformer baselines, we employ ModernBERT and NeoBERT as representative encoders, especially since they were both recently modernized and optimized using Flash Attention (Dao et al., (022), RoPE positional encod- ing (Su et al. 2021). and several other efficiency techniques (see Section
To quan-
RoPE enables evaluation beyond a model's pre-trained context window, which we exploit to measure throughput and latency for ModemnBERT and NcoBERT extended sequence lengths. While such extrapolation may degrade downstream task effectiveness, this does not affect our analysis, as our focus in this section is strictly on efficiency rather than model quality. In contrast, Avey's 's architecture decouples context width from sequence length, allowing Avey-B -B to operate at arbitrarily long inputs without additional pre-training at those lengths. We therefore extend sequence length as needed and compare Avey-B buse against ModemBBERT base and NeoBERT medium (the only publicly available variant), reporting throughput here and latency in Appendix [ The qualitative conclusions drawn in this section, however, hold consistently for both throughput and latency.
For ModernBERT and NeoBERT, we additionally report throughput under both optimized and un- optimized configurations, corresponding to execution with and without FlashAttention. We denote these settings as sys-optimized and yys-umoptimized, respectively. As shown in Fig. 2 the through- put curves for all encoders, across sequence lengths ranging from 128 to 96k tokens and under both settings, exhibit a consistent profile, that is, throughput increases at short lengths, plateaus at inter- mediate lengths, and declines large lengths. This pattern arises from the fixed batch size of eight
9

---

Published as a conference paper at ICLR 2026
ModentiiRT- NeoREETT sysooiimmed
corrh-
ModemäERT- sys-ureptimidd
NeoBERT sys-unaptimeed
Avey
Thrughoit (Tlknn
700,00
.000,000
1200.000
900,000
000,000
300,000
e00,000
200,002
100,000
12
(Takeng)
(4)
(M)
Figure 2: Throughput of Avey-B, ModemBERR, and NeoBERT NVIDIA B200 GPUs with mixed precision (BF16). We use Avey-B base, ModemBERT base, and NeoBERT medium (the only publicty available size). Avey-B is shown in (a) optimized using torch. compi i l (no fused-kernel implementation available yet) and (b) as unoptimized (eager). For Modern- BERT and NeoBERT, throughput is shown for system-optimized (with FlashAttention) and sys- tem-unoptimized (eager) variants in (a) and (b), respectively.
used throughout the study. Specifically, each forward pass incurs a constant memory-loading over- head, which dominates when relatively few tokens processed, resulting underutilized compute resources. As sequence length grows, more tokens are processed per batch, amortizing communica- tion overhead and improving hardware utilization. At sufficiently large sequence lengths, however, computation becomes the primary bottleneck due to increased arithmetic cost, memory- memory-bandwiddn pressure, and the growth of intermediate activations, leading to the observed decline throughput.
Despite sharing this qualitative profile, the encoders differ markedly in their quantitative scaling. We characterize long-context throughput using power-law decay model, T(N) x N where N denotes the sequence length and smaller exponents indicate better long-contex. efficiency. Un- and SooBERT--ys-optmiize demonstrate decay exponents of OMMddemEERT 0.77 and ONeoBERT 0.81, respectively, consistent with the bandwidth- and memory-drive limitations inherent to quadratic self-attention. In con- trast, Avey-B E--torch-oomppe displays substantially milder decay with exponent QAvey-B 0.44, sustaining significantly higher throughput across the long-context regime.
der optimized settings (see Fig. 2 (a)),
The unoptimized measurements further accentuate these differences (see Fig. 2 (b))-
and NeoBBEKT--yy--unopiiiizee exhibit considerably steeper decay, with exponents ('ModenBERT 1.03 and ONeoBERT 1.30, and both encounter out-of-memory fail- ures well before the maximum tested sequence length. Conversely, Avey-B-eager again achieves the mildest decay, with d 0.33, and maintains stable throughput across the entire sequence- length range. These results indicate that Avey-B's 's scaling advantage is structural rather than an artifact of kernel-level or compiler optimizations. More precisely, its neural processor inherently less sensitive to sequence length because computation depends on the fixed split size S, not the full sequence length N. With N/S splits, the total processing cost is (N/S) x S2 NS = (NN), yielding linear scaling in N and growing throughput advantage long contexts, while still being the fastest tested encoder at short contexts.
CONCLUSION
In this paper, we presented Avey-B, a bidirectional encoder built on Avey, new attention- foundation model. Avey-B contributes three architectural innovations: (1) decoupling static and dynamic parameterizations, (2) row-normalized similarity in the dynamic layers, and (2) neural compression module for improving effectiveness and efficiency. Results show that Avey-B deliv- consistent gains over Transoormrrbbsee encoders, including BERT, RoBERTa, ModemBERT. and NeoBERT on token classification and information retrieval, while outperforming BERT and NeoBERT every evaluated benchmark. These findings entail that attention might not be the only viable route to strong bidirectional encoders and motivate further study of retrieval-conditionel non-attention architectures.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY
All results reported in this paper are fully reprodueible. Section 4 provides detailed specification
of Avey-B's architectural components, while the complete experimental protocol is described in Section 5.1 and Appendix B The source code is publicly available at htttps://giil giggtitt...u.....g hub. avey.
ai/avey-b The repository includes: (1) scripts for pretraining, fine-tuning, and evaluation; (2) configuration files containing the exact hyperparameters used in every experiment; (3) data pre- processing instructions along with dataset references and splits; and (4) environment specifications and execution scripts to reproduce all reported tables and figures. Running the provided commands on hardware comparable our experimental setup reproduces the reported results within expected random-seed variance.
ACKNOWLEDGMENTS
We gratefully acknowledge the Ministry of Communications and Information Technology (MCIT), Qatar, for providing the computational resources that enabled substantial portion of the experi- ments reported in this work.
REFERENCES
answerdotai-basee Modernbert-base.
attes://hugginffee..c////avrrrbd
ModernBERT base, 2025. Hugging Face model repository.
answerdotai-large.
Moderbbert-large.
ModernBERT-Large 2025. Hugging Face model repository.
Payal Bajaj, Daniel Campos, Nick Craswell, Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint 16Xiv:1611.00 0.9268, 2016.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing (EMNLP), 2015.
Lola Le Breton, Quentin Fournier, Mariam El Mezouar, John X. Morris, and Sarath Chandar. Neobert: A next-generation bert. arXiv preprint 2xiv:2500. 2025.
chandar-lab. Neobert. hntta::/moggiingaaccco////aaaaae
2025. Hug-
ging Face model repository.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations (ICLR), 2021.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Process- ing Systems (NeurIPS). 2022.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learn- ing (ICML), pp. 933-941. PMLR, 2017.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arx.v:1810.048805, 2019.
facebook-robeert--base.
Roberta base (roherta-base).
htttps://huggingaace.co
FaacebookA//obbettaa base 2023. Hugging Face model repository.
facebbok-robertt-laaree
Roberta large (roberta-large).
co/
FFacebooAA/rrbertaa large 2019. Hugging Face model repository.
11

---

Published as a conference paper at ICLR 2026
Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2022.
Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Manuel Faysse, Duarte M Alves, Emmanuel Mal- herbe, André FT Martins, Céline Hudelot, and Pierre Colombo. Should we still pretrain encoders with masked language modeling? arXiv preprint arXiv:2507.0099 009944 2025.
google-bert-base. Bert base uncased (bert-base-uncased)) 2025. Hugging Face model repository.
https: / https:/huggingface http//hggggnnn...eeo co/
google-bert-aagee. Bert large uncased (bert-large-uncased), https: ftttps://hugiingfaee co/
ggogglebert//bert large-uncased 2023. Hugging Face model repository.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In Advances in Neural Information Processing Systems (NeurIPS). 2021.
Weiwei Guo, Xiaowei Liu, Sida Wang, Huiji Gao, Ananth Sankar, Zimeng Yang, Qi Guo. Liang Zhang. Bo Long, Bee-Chung Chen, and Deepak Agarwal. Detext: A deep text ranking frame- work with BERT. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM '20), Virtual Event, Ireland, 2020. ACM. URL https: //axxiv.rrg/aa/....aaa.o.......a wttp:////a.bb///saaa///// 000000200000000000 02460
Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are effective as structured state spaces. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Mohammad Hammoud and Devang Acharya. Don't pay 2025.
pay attention. arXiv preprint arXiv:2506.1 11305,
Pengcheng Hc, Jianfeng Gao, and Weizhu Chen. Debertav2: Further improving deberta using knowledge distillation. arXiv preprint arXiv:2111.0954 2021a.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced enhanced bert with disentangled attention. In International Conference on Learning Representations (ICLR), b021b.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentagleed embedding sharing. arXiv preprint urXiv:2111.09543 2023.
Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. Ontonotes: the 90% solution. In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, pp. 57-60, 2006.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4163-4174, 2020.
Andrej Karpathy. The most dramatic optimization to nanogpt so far (225% speedup) is to sim- ply increase vocab size from 50257 to 50304 (nearest multiple of 64). lhttps: com/ February 2023. X (formerly Twitter) post.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, 2020.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th Interna- fional Conference on Machine Learning (ICML., 2020.
12

---

Published as a conference paper at ICLR 2026
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp. 39-48, 2020.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 785-794, Association for Computational Linguisties, 2017.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR), 2020.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: Experi- ences on accelerating data parallel training. Proceedings of the VLDB Endowment (PVLDB), 13 (12):3005-3018, 2020, doi: 10. 00.1478//440799.3447777 pvlcb/vol13
URL https: /www. vldb. org/
pdf
Hanxiao Liu, Zihang Dai, David So, and Quoe V Le. Pay attention to mlps. Advances in neural information processing systems, 34:9204-9215 2021.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1rXi::1907.11699 2019.
Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Suppa, Hila Gonen, Joseph Marvin Imperial, Börje F Karlsson, Peiqin Lin, Nikola Ljubešić, Lester James Miranda, et al. Universal ner: A gold- standard multilingual named entity recognition benchmark. arXiv preprint arXiv:231 1.09122, 2023.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, Nils Reimers, Douwe Kiela, and Timo Schick. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2307. 16645, 2023.
Multi-Linguulity Multi--uuctionaliiy Multi- CaaniaGranularity.. M3-embedding: Multi-linguality, multi- functionality, multi--ranuuarriyy text embeddings through self-knowledge distillation, 2024.
Tsendsuren Munkhdalai, Lingkai Kong, Tianxiao Zhang. and Jiawei Han. Infini-attention: Long- context attention in infinite sequence. arXiv preprint arXiv:2404.07744 2024.
Zach Nussbaum, John X Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic embed: Training a reproducible long context text embedder. arXiv preprint arXiv:2402.01613, 2024.
OpenAl. How to count tokens with tiktoken. https://cookook. www.:/ii..cc...oo...oooo com/ haammp/eeehhw_ttcoouttt_ooo
weattthtiikkkeeee 2022. Accessed2025-000
OpenAl. tiktoken: A fast BPE tokenizer for use with openai models. httpp:://gtthan com/
openaa/tiktooenn 2025. Accessed 2025-09-21.
Antonio Orvieto and Robert Gower. search of adam's secret sauce. arXiv preprint arXiv:2505.211829 May 2025. doi: 10.485500/rxxi22555222222 URL htttp::/aaxii.. bxq/aba/2505.211
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019. URL
https: org a 912 01703
13

---

Published as a conference paper at ICLR 2026
Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024. URL Win 300BT split used in this work.
17557
Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, et al. Rwkv-7" goose" with expressive dynamic state evolution. arXiv preprint arXiv:2503. /4425500..444,,, 2025.
Jacob Portes, Alex Trott, Sam Havens, Daniel King, Abhinav Venigalla, Moin Nadeem, Nikhil Sardana, Daya Khudia, and Jonathan Frankle. Mosaicbert: A bidirectional encoder optimized for fast pretraining. arXiv preprint arXiv:2312.17448 2023. doi: 10:48500/axxx23312.77422
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases (alibi) enables input length extrapolation. arXiv preprint 10Xiv:2108.124400 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving lan-
guage understanding by generative pre-training. OpenAl technical report, 2018.
//cps:////do..oeettt www.c/////eeerchh.ee http://eeaarcccooesssseeeeeeeeee wttperrss/arugggeeusspppppap larquage_unden htg.:aags_.rddrrttttdnngggg..neu
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamila Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
Prabhakar Raghavan. How ai is powering more helpful google, October 2020. URL https: //blog. gooq M www.ttpooss//opoooo setsse/arccceeeeaa httpt//eee.aaaaaaaaeeeeeaaaaeeee now used in almost every query English." Accessed 2025-09-24.
The Keyword (Google). "BERT is
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 784-789, 2018.
Erik Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language- independent named entity recognition. arXiv preprint cs/0306050, 2003.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01106 2019.
Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint 1rX0v:21 2022.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.055200 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. arXiv preprint wrXiv:2104.00 098644, 2021.
Yuxuan Su, Xinglin Wang, Yihan Wang, Dong Yu, Yiming Chen, Jing Lin, Tianyu Liu, Zheng Liu, Yu Chen, et al. Gte: General text embeddings with weak supervision. arXiv preprint ar0iv:2308 03281, 2023.
14

---

Published as a conference paper at ICLR 2026
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang. and Denny Zhou. Mobilebert: a compact task aask-agnottic bert for resource-limite devices. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics (ACL), pp. 2158-2170, 2020.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1-25 2022.
Nandan Thakur, Nils Reimers, Johannes Daxenberger, James Cohen, and Iryna Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Pro- ceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp. 2325-2331, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2017.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for ceneral-purposs. language understanding systems. In Advances in Neural Information Prciiioorrrrrro Processing Systems (NeurIPS), 2019a.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bow- man. Glue: A multi-task benchmark and analysis platform for natural language understand- ing. In Proceedings of the International Conference Learning Representations (ICLR), 2019b. http::://glueebenciaaak.. com/
Sheng Wang. Ming Zhao, Nguyen Bach, Tao Xu, Zhe Huang, Fei Wu, Tong Wang, Shuo Si, Zhongqiang Li. et al. Text embeddings by suukly--upervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022.
Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral multi- perspectise matching for natural language sentences. arXiv preprint arXiv: 2017.
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirec- tional encoder for fast, memory efficient, and long context finetuning and inference. In Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 2526-2547, Vienna, Austria, July 2025. Association for Compu- tational Linguistics. ISBN 979-8-89176-251-0. doi: 12.18653/1122025.acch acl-long.. URL 127/
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 2018.
Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie E Everett, Alexander Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jachoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale prox- ies for large-scale transformer training instabilities. In The Twelfih International Conference on Learning Representations (ICLR 2024), 2024. wwt.htttss://ppneeetiie net.:iiwwwweeeeeeeeeeeeeeeeeeeee www.ere/weeeeeeeeeouuummmm id ZZ Oral.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jina Mao, Paul N. Bennett, Jiadong Chen, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Jeffrey Zhu. Bing delivers its largest improvement in search experience using azure
gpus, November 2019. URL
bing-dee deag-delivers wetvverr-ss---e witss-aaggett-aeeeeeeeeeeee--eee Li www.aaarrreeeeeeeeeeeeeeeeeeeeee usininnneee-uiigeeeee wwwee-auuee-paa wwinggazuu-ggguus Reports serving > 1M BERT inferences/sec worldwide. Accessed 2025-09-24.
15

---

Published as a conference paper at ICLR 2026
A MONOTONICITY UNDER DECOUPLING
In Section 4.2 we claimed that decoupling static and dynamic layers maintains monotonicity with respect to relevance within each dynamic layer, and that this guarantee is isolated from what subsequent static layer does. We now formalize and prove this claim.
Setup (dynamic layer). As defined in Section 4.2) given Zee E RCxd' and fixing a target embedding (row) i, Avey-B's B's dynamic layer computes:
S = - (M.z))NV (.).))
(9a)
Si.j
Skj
(9b)
Lg'=1 Si +E
Cdyn,f
Su, Zer,j b(a
(9c)
where NV-) is per-row le normalization (so S contains cosine similarities), 0 stabilizer, a pointwise monotone activation, and b(d) an optional bias.
Assumptions. We assume the following:
(A1) Nonnegative similarities. The enricher uses a nonnegative pointwise activation, namely,
ReLU2 HHammuud Acharya, 2025), hence, rows of (Z.r) are nonnegative, implying St, 2
(A2) Positive normalization. For each row i, E S. + 0 with > 0.
(A3) Monotone activation. a is monotone nondecreasing (e.g., Avey-B uses ReLU in Equation 9c
Proposition A.1 (dynamic layer monotonicity). For fixed target row and any embeddings j1,J2 (I.
(i) Order preservation. If Si,j 2 Si.j then Si,, 2 Si.ja-
(ii) Self-momotonicity Increasing S.y (while holding ( fixed) weakly increases S and does not increase S for l: {
Consequently, a more relevant token (higher similarity) receives at least as large (and typically larger) weight than less relevant token, and increasing its relevance cannot reduce or flip the sign of its contribution in the update within the dynamic layer (i.e., the update is monotone with respect to relevance).
Proof. Let d; ES Su, + by (A2), d, 0. Since St, / for fixed i, dividing by positive constant preserves order. In addition, treating d, function of S,, and using (A1):
SS.
d;
Si.j
2 0,
since d, 2 St,5 by (A1) and e >> O,
(10a)
0S., d?
0 0Sth
S.k
0,
k k# j.
by (A1).
(10b)
0S., d?
Thus, increasing a token's 's similarity weakly increases (or cannot reduce) its own normalized weight (by Equation Toap and weakly decreases (or does not increase) others' (by Equation TOb). By (A1), Sij 0, each token's 's influence enters the pre-activation with nonnegative coefficient; by (A3), a cannot invert these contributions. Hence, the dynamic update is monotone with respect to relevance.
16

---

Published as a conference paper at ICLR 2026
Setup 2 (static layer). Consider dynamic layer at depth followed by static layer:
(f+1) = of ( ( ( + b(1)), (f+2)) of V (++)) + bb()),
(11)
(12)
where (V, b(o)) are learned parameters that do not depend S S(0)
Proposition A.2 (static layer non-violation). A static layer (as in equation 12) cannot violate the
monotonicity guarantee of Proposition A.1 established for preceding dynamic layer (as in Equa- tion 11 at depth
Proof. The monotonicity statements Proposition A.1 concern only the relationship between
the relevance scores ((4) and the normalized scores (()) used inside the dynamic update equa- tion 11 The static map h(++1) + y(++2) depends on h ((+1) and the similarity-agnostie parameters (V, b()); )); it neither accesses nor alters ((() or S(0) Therefore, composing the dynamic update with a static layer cannot change the inequalities and partial orders that define monotonicity for the dynamic layer's scores.
Remark. A static layer reshapes representations that subsequent dynamic layers will use to com- pute new similarities, but does not retroactively modify the scores already assigned by a preceding dynamic layer. Thus, monotonicity holds at each dynamic layer, and decoupling preserves this guarantee throughout the stack.
B PRETRAINING METHODOLOGY
Table 3: A comparison of all the evaluated encoders across different dimensions.
Dimension
BERT
RoBERTa
ModernBERT
NcoBERT mediuon
Avey-B
base
large
base
large
base
large
base
large
Parameters
165M
391M
Corpus
BooksCorpus
Data Sources
WehnnWeTeext
Undisclosed
RefinedWeb
FineWeb
Wikipedia
Stories CCCNews
Pre-training Context Width
512
$12
1,024 - 8,192
1,024 4,096
2,048
Inference Sequence Length
512
512
8.192
4.096
0c
Masking Rate
15%
15%
30%
20%
20%
Masking Scheme
80/10/10
80/10/10
100
100
Tokens Seen
131B
131B
~2T
2.1T
180B
In this section, detail the pretraining setup. For Avey-B, adopt the tokenizer
as Avey (Hammoud & Acharya, 2025), namely, a BPE tokenizer derived from OpenAl's p50k_base (OpenAl [002]] 2025). with the vocabulary size set to 50,368 align with multiple- of-64 boundaries and improve hard ware efficiency (Karpathy (023). We retain BERT-style special tokens for backward compatibility with downstream applications, while using only the [MASK] token during pre- pre-training.
We pretrain two Avey-B sizes, base (165M) and large (391M), for 180B tokens drawn from the Web 300BT split (Penedo et al.] 2024), using PyTorch DDP across 16 NVIDIA H200 GPUs (Paszke et al. 2019 Li et al. 2020). The global batch size is set to 512K tokens for both models and we utilize the AdamW W optimizer with 31 B2 0.95 (Orvieto & Gower 2025), e 10-18 (Worsmman et al. 2024), weight decay of 0.01, and gradient clipping at 1.0. For the leamningrrate schedule, employ 10% linear warmup to 5 10-4 (base) or 2.5 10 10-4 (large), followed by cosine decay to zero over the remaining 90% of steps.
For ablations and design design-choice choice studies, learning rate of 10 3 for 10B tokens. During pretraining, sequences are packed so that each training example meets the target sequence length, following the original Avey setup. We train with masked
the Avey-B base model, pretrained with constant
17

---

Published as a conference paper at ICLR 2026
Table 4: Effectiveness results comparing unidirectional vs. bi-directional rankers.
Ranker Type
QA
MNLI QQP SST-2 Avg. CONLL Onto UNER Avg. ReCoRD SQuAD SQuADv2 Avg. MLDR MS MARCO
NQ Avg.
Unidiroctional ranker 81.39 88.92 91.86 87.39 92.96
93.21 93.97 93.38
30.22
62:26
60.72
51.07
60.27
87.48
78.71 74.82
Bi-directional runker 80.51 88.45 91.71 86.91
92.62 92.91 94.35 93.29
10.61
45.34
53,56
30.51
60.05
90.02
08.51 66.20
language modeling (MLM) objective, randomly masking 20% of tokens per example after exploring several masking rates (see Appendix
Finally, for the Transformer-based encoders, the Hugging Face Hub beoogle-bert-bare 2025; google-bert-large 2023; rocbbook-roberta- facebook-roberta-lnin
publicly available pretrained checkpoints from
2023
2019 aanserdotaii-aaa base 2025 answerdotai--arge [2025] chandar-lab
2025
Sablee]smmmarn the evaluated models along key dimensions, including parameter count,
context window, and pretraining tokens, among others.
C SHOULD THE RANKER OPERATE BIDIRECTIONALLY?
In the original unidirectional Avey architecture, the ranker retrieves only preceding splits (i.e., left context) in order to preserve the causal constraint required for autoregressive modeling. In the bidirectional Avey-B setting, however, this constraint no longer applies. This raises the question of whether the ranker should, analogously the neural processor, operate bidirectionally and retrieve relevant splits from both the left and right contexts of the current split.
To investigate this design choice, we conduct controlled experiments using the Avey-B base variant (165M parameters; see Table 3) The model is pretrained on 10B tokens from Fine Web (Penedo 2024 with constant learning rate of 10-3 and a masking rate of 20%. Following Section 5.1 we fine-tune for one epoch on SC and TC, four epochs on QA, and 1,000 optimization steps on IR. For each task, we run five independent seeds with learning rate of 5 x 10-4 using a 10% warmup followed by linear decay over the remaining 90% of steps. We report the best-of- five score for each configuration to approximate an upper-bound estimate. Evaluation metrics are accuracy for SC, F1 for TC and QA, and NDCG@ @ 10 for IR. The complete results are presented in Table 4
Specifically, Table 4 directly compares a unidirectional ranker against bidirectional ranker across SC, TC, QA, and IR task categories. As shown, the bidirectional ranker consistently underper- forms its unidirectional counterpart. While the degradation is modest for SC (87.39 + 86.91; A = -0.48) and TC (93.38 - 93.29; A = -0.09), it becomes pronounced for QA (51.07 + 36.51; A 14.56) and IR (74.82 66.20; A -8.22). The effect is particularly severe on QA, where the Fl score on ReCoRD drops from 30.22 to 10.64. This sharp decline suggests that retrieving right-context splits at the ranking stage can substantially disrupt evidence selection and positional reasoning, especially in tasks requiring precise span identification.
We identify two likely causes for this behavior. First, unidirectional ranker enforces causal order- ing and encourages the model to accumulate evidence along the discourse flow. Natural language often exhibits forward dependencies, whereby content in a later split is best interpreted in light of earlier splits. As such, allowing the current split to pair with future splits can dilute or over- ride strong signals from its relevant preceding splits. Second, Avey-B already provides look-ahead, token-level contextualization within each split (the contextualizer operates without causal mask on each split) so every position in it except the last has access to rightward tokens. Therefore, additional look-ahead, selit-level contextualization seems often redundant and at times even disruptive.
In summary, these findings suggest that while look-ahead, token-level contextualization within a split benefits Avey-B, look-ahead, split-level contextualization, driven by the ranker attending to both left and right split contexts the current split, is not advantageous and potentially counterpro- ductive.
18

---

Published as a conference paper at ICLR 2026
Table Effectiveness results across different static (S) and dynamic (D) layering patterns.
QA
MNLI
QQP
SST-2
Asg.
CONLL
Onio.
UNER
Avg.
ReCoRD
SQuAD SQuADv2
Aeg.
MLDR MS MARCO
NQ
Avg.
Interleaved, 8 --...
$1.33
88.92
91.86
87.30
92.96
93..1
35.37
93.38
30.22
62.26
60.72
51.07
60:27
87.48
76.71
74.92
Interteaved,
87.51
85.26
44.07
69.32
Single dynamic head
73.09
50,55
91:06
83.17
02.42
92.75
93.51
92.83
22.02
56.94
54.87
16.74
00:22
89.70
73.32
74.41
Single dynamic Two stage stack, 5619 DAr
73.19
87.90
84.24
55.29
48,06
87.72
76.31
74.53
75.70
56.50
90.25
84.17
92.51
92.65
94.24
53,
15.29
52.18
55.09
40.85
54.97
85.92
67.54
69.48
Two-stage 24/2 9412
87.15
54.36
44.96
89.56
Uniferm stack, all-static
77.58
87.97
91.51
85.00
02.06
83.10
04.04
03.
23.5
36.43
45.00
62.54
56.38
73.92
74.05
Uniform stock, ail-dyynmic
08.04
83.26
87.27
79.52
90.31
90.86
91.17
50.78
19.40
47.51
50.44
39.15
57.70
80.33
00.50
72.18
Table 6: Effectiveness results across different normalization schemes.
Nommalizationn Scheme
SC
TC
QA
MNLI OOP
SST-2 Arg. CONLL
Onto.
UNER
Aug.
ReCoRD SQuAD SQuADv2
Avg.
MLDR MS MARCO
NQ
Avg.
Dividt-by-sum
81.39
88.92
92.96
62.26
60.72
51.07
60.27
87.48
76.71
74.82
RMS narm
64.70 87.21
88
76
80,22
90 80:82
92.12
91.89
91.61
21 .57
56.09
06.33
44 60
50.27
89.33
66.68
08.76
Softmax
79.31 88.16
02.30
27.70
30.25
58-55
48.51
61.83
89.75
74.43
75.34
Scaled soltamxx
76.70 87.24
91.86
85.27
92
93.02
94.07
93.24
24.14
58.79
56.23
16.3 31 62.24
87.63
74.39
74.76
D HOW TO ARRANGE STATIC AND DYNAMIC LAYERS?
Decoupling static and dynamic parameterizations into distinct layer types in Avey-B introduces a key architectural degree of freedom. that is, how to arrange static (S) and dynamic (D) layers across depth. We therefore evaluate the following families of patterns and report their effectiveness:
1. Interleaved: Alternate S ++ D. With an even number of layers, we test both start points, S D- and D-
2. Single dynamic: Exactly D and the remainder S, placing D either the head (to prime downstream static transformations) or at the tail (to refine final representations).
3. Two-stage stack: First half one type and second half the other, considering both orders (SL/2_
DL/2 and
Uniform stack: Either all-static or all-dynamic stack, as boundary conditions.
We utilize the experimental setup described in Appendix Table reports all the results across SC, TC, QA, and IR task categories. Two consistent trends emerge. First, the interleaved ar- rangement, 5 D + ..., attains the strongest average performance on SC. TC, and QA, while remaining competitive on IR. This suggests that static front layer is potentially providing a sta- ble representational "scaffold" before any input-dependent mixing, reducing variance introduced by raw similarity scores and improving downstream contextualization. Second, the interleaved pattern, D -- underperforms the $ - ..... variant (most notably on QA and IR) likely because early, similarity -driven updates fragile without a learned (static) basis shape features prior to dynamic contextualization.
The uniform stack, all-static configuration performs worse than interleaved arrangements but only modestly so (it even slightly outperforms them on IR), indicating that static linear projections alone already enable strong contextuaiization, even without any input-dependent adaptation. Conversely, the uniform stack, all-dynamic pattern performs worse across all benchmark categories (particularly QA). The single-dynami and two-stage stack arrangements fall between these extremes, though they typically trail the interleaved static-first design (except for two-stage stack, DL/2 -> LL/2 on IR). Overall, these findings highlight that while dynamic parameterization contributes meaningfully performance, is most effective when interleaved with static layers that supply stable basis and representational depth.
E How TO NORMALIZE?
In Avey-B, BB dynamic layers contextualize tokens by constructing a cosine similarity matrix from pairwise cosine scores of the input. Since the similarity scores used to perform weighted sum
19

---

Published as a conference paper at ICLR 2026
Table 7: Effectiveness results across different sequence length N, split size S, and top-k values.
s
k
SC:
TC
QA
IR
MS MARCO
MNLI
QQP SST-2
Avg. CONLL Onto.
UNER Avg.
ReCoRD SQUAD SQUADv2 Avg.
MLDR
NQ
Avg.
128
80.09
88. 82
91.63
86.85
92.09
93.08
93.85
93.21
27.65
60.44
58.82
18.97
00.06
58.89
76.11
75.02
512
87.16
92.74
93.35
28.11
38.79
19.19
60.72
90.11
76.06
256
79.68
88.75
92.32
86.92
92.27
92.94
93366
92.97
43.70
71.25
64.06
59.67
60.71
88.61
76.29
73.20
80.79
88.50
91.63
87.07
92.87
93.03
93.81
03.24
27.42
61.14
58.85
49.14
60.44
68.95
70.06
75.15
80.02
88.
91.63
86.79
92.89
93.00
94.43
93.46
26.74
61.55
00:49
50.26
61.17
90.26
75. 86 75.70
128
80.61
88
62
91-40
86.89
92.80
93.13
93.85
93.29
26. 69
61.19
59. 79
99
61.81
86.86
75.61
74.76
1024
80.70
89.02
87.15
92.41
93.14
93.11
27.96
62.58
66. 73 71.90
256
79.53
88.80
91.28
86.54
92.10
933002
933.5
922.90
42.82
70.16
62.54
58.51
60.88
88.87
76.34 75.36
79.93
88.65
90
86.67
92.10
92.8-
93.48
92.81
12.51
70.55
63. 16
58.71 71
59.11
89.13
77.06 73.21
512
79:11
88.
35
99.51
86.42
92.63
93.18
933.71
993.19
43.96
71.92
63.91
59.93
60.22
89.20
77.82
75.76
80.00
88
93
91.74
87.09
92.85
93.13
94.05
93.31
28.72
61.50
59:13
19.78
00.21
90.69
75.15
75.35
80.91
87.17
92.52
93..07
28.49
19.72
60.71
89,
75.14
80.98
88. 66
91.51
87.05
92.86
93.17
94.02
93.35
27.07
60.27
59.23
48.86
59.02
88.43
75.
74.39
67.21
88.
91.63
82.43
91.77
92.69
93.00
92.49
27.76
61.81
58. 15
19.31
57.08
86,89
53.55
72.51
128
80.61
88.84
91.51
87.00
92.51
92.97
93.82
933.11
29.82
62.44
59.52
501.39
61..7
58.27
75.M3
74.99
80.49
85.75
91.97
87.07
92.63
93.05
93.45
93..0
29.54
39.77
49.09
58.14
75.54
74.37
13 81.14
88.75
92.20
87.36
92.92
93.13
94.05
93.37
27.36
59.98
58.01
48.45
60.21
90.31
78.48
76.33
2048
15 80.78
89.01
91.63
87.14
92.51
02.94
94.11
19
28.27
60.34
59.46
49.36
58.69
68,
70.03
74.28
70.61 88.69
92.09
83.80
92..38
93.18
94.02
933.19
44.32
70.96
63.58
59.62
60.84
58.15
77.24
75.41
80.18
88.91
91.74
86.94
92.33
93.05
93.75
93.04
42.91
71.99
64.81
59.90
60.02
90.36
76.58
75.65
256
79.45
88
90.71
86.21
91.51
92.46
02.77
92.25
38.85
69.88
62.09
56:94
56.72
88.11
73.70
72.84
78.98
88 65
91.74
86.46
93.00
92.85
93.66
93.17
42.89
70,64
63.25
58.93 93
59:12
88.35
16.36
74.71
79.37
88 52
91.63
86.51
92.11
92.77
93:26
92.81
11.67
71.15
62.20
58. 11
56.76
88.67
77.91
71/16
512
79.36
88.67
91.28
86.41
92.15
92.87
94.00
933.11
46.39
72.34
61..6
60.93
55.65
87.32
75.89
72.95
1024
75.29
88.
38.34.10
85.01
92.10
92.79
93.57
92.82
44.88
02.05
59.13
55.86
90.07
73.93
73.29
of input embeddings at every position and the sum of the raw similarity magnitudes can vary signif- icantly, we tested several normalization strategies to stabilize training and improve generalization, including divide-by -bm-------mmm---- norm (i.e., row-wise normalization by the sum of similarities), RMS norm (i.e.. row-wise normalization by root mean square), softmax, and scaled softmax (with temperature scaling, analogous scaled dot-product attention). In all the tests, we used the same experimental setup discussed in Appendix[[]
As shown in Table 6 the simple fivide-by-sun norm method achieves the strongest overall perfor- mance, outperforming alternatives on SC, TC, and QA, and almost matching or surpassing them on IR. Notably, divide-by-sum norm provides balanced distribution of contextual weights while retaining sign information, which is lost under roftmax-basee schemes. By contrast, softmax and scaled softmax yield weaker SC, TC, and QA scores but softmax outperforms divide-by-sum norm on IR. On average, RMS norm underperforms divide-by-sum norm across all categories.
These findings indicate that unlike self-attention, which benefits from exponential normalization (as provided by softmax), Avey-B's cosine- based dynamic layers benefit from a conservative, structure-preservvnn normalization. Exponentiation amplifies outliers, distorts relative similarity ratios, and can swamp the static path. In contrast, divide-by-sum norm preserves the ordering and margins of similarities, constrains each row to a convex combination (weights in [0. 1] that sum to 1), and effectively bounds the operator norm, yielding stable gradients and preventing the dynamic stream from overwhelming the static contributions. Empirically, this simple choice delivers strong gains across SC, TC, QA, and IR while maintaining robust training dynamics.
F WHAT ARE THE BEST SEQUENCE LENGTH, SPLIT SIZE, AND TOP-K VALUES?
We now analyze how Avey-B's downstream performance is affected by the ranker's three hyperpa- rameters, namely, the training sequence length N, the split size S, and the number of top-k splits selected for contextualization. N governs the size of the candidate pool available to the ranker, S determines the size (in tokens) of each candidate split, and the effective context width seen by the contextualizer is = S(k+ + 1). We follow the experimental setup described in Appendix C Table 7 illustrates all the results.
20

---

Published as a conference paper at ICLR 2026
Table 8: Effectiveness results different masking rates for Avey-B's base model.
SC
TC
QA
Masking %
MNLI
QQP
SST-2
Avg. CONLL Onto.
Avg.
ReCoRD SQuAD SQuADv2
Avg.
MLDR MARCO
NQ
Avg.
10%
78.07
88.02
01.86
85,98
92.54
93.50
92.55
36.64
66.57
60:65
55.29
51.02
86.75
71.50
69.42
20%
80.18
88.91
91.74
86.94
92.33
93.05
93.75
93.04
42.91
71.99
64.81
59.90
60:02
90.36
76.58
75.66
30%
78.62
88.49
91.97
86.36
92.45
93.01
93.75
93.07
42.80
71.26
63.85
50.30
62.16
89.72
76.19
78.02
40%
77.05
88.02
91.51
85.53
92.26
92.90
93.49
92.88
39.70
09.84
62.45
57.33
59:67
90.33
74.56
74.85
50%
66.12
88.32
91.06
81.83
92.86
92.62
93.16
92.88
42.15
70.44
62.90
58.50
62.03
90.06
77.69
76.59
Table 9: Effectiveness results at different masking percentages for Avey-B's large model.
SC
TC
QA
Masking
MNLI
QQP
SST-2
Avg.
CONLL Onto.
UNER
Avg.
ReCoRD
SQuAD SQuADv2
Avg.
MLDR MS MARCO
NQ
Avg.
10%
$1.01
88.54
92.00
87.21
92:39
92.91
93.32
92.87
42.46
71.53
64.83
50.54
54.58
88.73
75.25
72.85
20%
82.12
92.32
87.88
92.70
92.94
93.65
93.12
47.93
72.84
65.79
62.19
63.53
91.54
80.47
78.51
30%
61.54
89.43
91.74
87.57
92.59
92.98
93.83
93.13
48.16
73.44
66.52
62.71
61.72
80.31
81.76
77.60
40%
70.21
80.11
92.20
83.84
92.51
93.13
93.65
93.10
46.20
73.36
65.71
61.79
64.09
90.96
81.35 78.80
50%
78.19
89.02
92.20
86.47
92.83
92.95
93.83
93.20
46.99
71.28
63.89
60.72
61.63
90:49
79.37
77.16
To begin with, the dominant trend across tasks is that performance peaks when the effective context C C=3 S (k+1) matches or closely approximates the training sequence length N. For example, on QA with 2028, the best average occurs at S-512, k=3, giving C'=512 x (3+1)-2048_N, For SC, TC, and IR at N=2048, the strongest averages are S 128, k=13, yielding C=128 x (13+1)=1792, close to N. Similar behavior holds for 5 and N=1024 across categories, with one slight exception, that is, TC. In particular, on TC, the best setting often lands on C R N/2 (e.g., at N=512 the optimum is 128 1= so N/2 but it is only +0.1 points away from the effectiveness at 5=512, S = 128, 3, which yields C=128 x (3+1)=512-N),
In summary, Avey-B's performance generally improves with larger training sequence length N. In our experiments, the best results occur at the largest tested N=2048 for SC, QA, and IR. The exception is TC, which peaks at 5=512 but is within +0.09 points of the N=1024 setting. Across these optima, the coverage heuristic C' = S (k+1) xs N consistently satisfied (matching or closely approaching N). This pattern suggests that, for a bidirectional encoder, one should en- large the candidate pool via larger N while ensuring ample contextual coverage by setting $ (k+1) to match closely track N. Averaging over all task categories, the best overall configuration is -=2048, S=256, k=3, hence, it was adopted as Avey-B's default configuration.
G WHAT IS THE BEST MASKING RATE?
Because Avey-B is pretrained with masked language modeling. the fraction of tokens replaced by the [MASK] token sets the task difficulty. In particular, too little masking makes reconstruction nearly trivial, whereas reliable prediction. To calibrate this trade-off, we swept masking rates from 10% to 50% for both the base and large models (see Table in Appendix B). while following the same experimental setup described in Appendix (except for the masking rate since we vary it here).
much masking deprives the model of sufficient contextual signal for
As shown in Table increasing the masking rate from 10% to 20% improves performance across SC, TC, QA, and IR for the base model. Overall, scores typically peak at around 20%-30% masking, yielding consistent gains on SC, TC, and QA. The exception IR, which attains its best results at 50% masking. At higher masking levels (40%-50%), performance can drop markedly (e.g.. MNLI), indicating that the smaller-capacity model struggles when too little context is visible (masking be- comes overly aggressive, weakening both the input signal and the training target).
The larger model is more robust to masking but still follows a similar trend to the base variant (see Table D). Performance generally improves from 10% to 20-30% masking, which offers the best cross-task trade-off (an exception is TC, which peaks at 50%). QA and IR benefit the most, with ReCoRD, MLDR, and NQ rising by over +5, +9, and +6 points, respectively, relative 10% masking. Although the large model tolerates 40-50% masking with modest degradation, SC remains sensitive, whereby at 40% masking, MNLI drops by 12 points versus 20%, then partially recovers
21

---

Published as a conference paper at ICLR 2026
Table 10: Ablations of Avey-B, removing one component at a time while holding all others fixed. (1) w/o normalization: removes row-wise normalization in the dynamic layers; (2) w/o decoupling: reverts coupled static and dynamic parameterizations; (3) w/o compression: omits the neural compressor; (4) w/o residual: discards the residual connection between the compressor output and current split's tokens; and (5) w/o ranker: disables the ranker entirely.
Model
TC
QA
IR
CONLL
UNER Avg.
ReCoRD SQUAD
SQUAD
Avg.
MLDR
MS
Avcy-B (Tull design)
50.71 88.91
91.97
87.20
91.84
93.25
93.09
92.72
39.00
08.32
60.45
56.20
57.19
90.38
75.64
71.50
Avey-B normalization
77-47
84.60
00:25
84.10
90.95
52.12
91.91
29.72
67.15
08-83
51.50
46.43
82.89
30.92
Avey-B w/o decoupling
79.91
58.60
89.33
86.95
88
92.1
91.06
90.75
36.86
89
59.57
54.77
55,00
82.77
69.18 68.98
Avey-B compeession
80.80
89.03
91.17
57.00
91.52
93.29
92.97
92.39
42.80
70.54
59.77
57.70
60.95
89.13
76.92
75.66
Avey-B B residual
77.53
87.48
90.37
85.12
90.80
02.44
91.17
91.47
34.71
86.08
56.02
52.27
55.22
66.74
71.72 71.22
Avey-B wlo ranker
77.38
87.32 85.70
51448
90.20
92.39
90.71
91.11
26.85
06.27
57.
50.01
38.28
85.35
61.93
61.85
Aveyy--witt--oompressin
160000
Throughput (Tokens/Secone)
140000
120000
100000
80000
60000
40000
20000
1K
2K
3K
4K
5K
6K
7K
8K
9K
10K
11K
12K
13K
14K
15K
16K
Sequence Length (Tokens)
Figure 3: The throughput of Avey-B with and withou the neural compressor.
50%, going up by ~ points versus 40%. These patterns indicate that overly aggressive masking can destabilize training even at higher capacity.
Overall, both the base and large variants indicate that masking rate in the 20-30% range is near- optimal; accordingly. we adopt a 20% masking rate for pretraining both models.
H ABLATION STUDY
In this study, we conduct series of ablation experiments on Avey-B. To this end, we fix the sequence length N, split size S, and top-k retrieval depth to the best settings from Appendix F (2) the static- dynamic interleaving pattern to the best arrangement from Appendix [ and (3) dynamic-layer normalization to the most effective scheme from Appendix In addition, we follow experimental setup described in Appendix
Table 10 reports ablations over five key architectural components of Avey-B: (1) decoupling static and dynamic parameterizations; (2) applying row-wise normalization in the dynamic layers; (3) incorporating neural compressor within the ranker; (4) adding a residual connection between the compressor output and the original tokens of the current split that is being compressed with its top-k retrieved splits; and (5) removing the ranker entirely.
As illustrated in Table I0 coupling the static and dynamic parameterizations (i.e., see "full design" and "w/o decoupling" rows) yields consistent drops of 1.43%, 2.12%. 2.53%. and 7.40% on SC, TC, QA, and IR. respectively, confirming that separating similarity scoring from neural learning improves accuracy. Row-wise normalization proves more critical, whereby removing it leads
22

---

Published as a conference paper at ICLR 2026
ModermfiiRR--yy-oppinee NedBERT -eddERT-nys--pliine syps-pplinddd
Saay--Bttrrc-comme
ModemBERR unaptimized
NNoBERT unoptimeed
Avey eager
Labency (Secoeto
175
235
Sequence Langth (Tokans)
Sequence Length (Tobera))
(0)
[N
Figure 4: Latency of Avey-B, ModemBERRT, and NeoBERT on NVIDIA B200 GPUs with mixed precision (BF16). We use Avey-B base, ModernBERT base, and NeoBERT medium (the only pub- licly available size). Avey-B shown in (a) as optimized using torch. compile (no fused- kernel implementation is available yet) and in (b) as unoptimized (eager). For ModemnBERT and NeoBERT, latency is shown for system- -amee--ptimized (with FlashAttention) and system unoptimzzed (eager) variants in (a) and (b), respectively.
to significantly larger degradations of 3.55%, 0.87%, 7.65%, and 15.33% across SC, TC, QA, and IR, respectively.
As discussed in Section 4.3 the neural compressor reduces the number of contextualized tokens per split from (k+1)S to S. This reduction yields a substantial 4.37x throughput improvement (see Fig. 3 while preserving strong task performance (see Table TO On SC and TC, compression has negligible effect and even produces slight average gains of +0.23% 23% and -0.14%, respectively, likely due to the removal of noisy global tokens that can arise under top- retrieva 4 In contrast, QA and IR exhibit modest average drops of 2.68% and 1.56% respectively. These tasks rely more heav- ily on fine-grained cross-split evidence and subtle retrieval cues, which compression may partially attenuate. Overall, considering the 4.37x efficiency improvement, the negligible (and sometimes positive) impact on SC and TC, and the modest reductions on QA and IR, the neural compressor provides a clear and favorable afficiency-effecctiiee
trade-off.
Besides compression, Avey-B B adds residual connection between the compressor output and the current split's S tokens to preserve local signal. Table that removing this residual degrades every benchmark, with average reduction of 3.38%, underscoring its role in maintaining lexical fidelity and stabilizing context integration.
Finally, as described in (Hammoud & Acharya 2025 the ranker is invoked only once per for- ward/backward pass, prior to the first layer of the neural processor, and retrieves the top-k relevant splits for each current split using shallow (initial) embeddings. One might expect that deeper con- textualized embeddings could yield better retrieval; however, removing the ranker entirely (i.e., allowing the neural processor to fit and operate the entire sequence) results in universal degrada- tion across all benchmarks, with a large average drop of 7.46% (see Table 10). This corroborates that retrieval is essential for Avey-B's effectiveness.
Importantly, retrieval based on shallow or static embeddings is a standard practice in dense retrieval and matching systems such DPR (Karpukhin et al. 2020), ColBERT (Khattab & Zaharia 2020), ANCE [Xiong et al.) 2021). and CLIP (Radford et al. 2021), among others. These systems deliber- ately rely on early-layer fixed representations because deeper contextualized embeddings tend to become increasingly task-specific, thereby distorting global semantic structure and degrading their overall retrieval quality.
To evaluate whether deeper-layer retrieval is beneficial, we incorporated the ranker at every layer such that it operates on contextualized embeddings. This intervention substantially degraded per- formance, causing an average drop of 27 28% across benchmarks, and slowed efficiency by 5.9x. These findings reinforce that deeper-layer retrieval is not only computationally prohibitive (as
'With a split containing S' tokens and the absence of a hard relevance threshold (where the ranker retrieves the top-k splits for each query split regardless of their absolute relevance), it is possible for some retrieved splits contain weakly relevant noisy tokens.
23

---

Published as a conference paper at ICLR 2026
would require recomputing MaxSim and reassembling contextualized blocks each layer) but also detrimental for effectiveness.
To summarize, these ablations collectively validate Avey-B's architectural principles, namely, (1) decoupling and normalization are critical for both effectiveness and stability (more on this in Appendix D); (2) the residual connection preserves essential local information; (3) the neural com- pressor delivers substantial efficiency gains with minimal accuracy loss; and (4) the ranker is indis- pensable, with shallow-embedding retrieval proving both computationally justified and empirically optimal.
I LATENCY RESULTS
In Section 5.4 we reported throughpul (tokens/second) for Avey-B, ModemBERR, and NeoBERT. In this section, we present latency (seconds per forward pass) for the same encoders. As noted in Section 5.4| Avey recent and lacks a fused-kernel (CUDA/Triton) implementation. Accord- ingly, measure Avey-B's latency using both eager PyTorch implementation (Avey-B-eager) and torch. compi le wweyyB-torch-commplle have optimized implementations using FlashAttention (Dao et al. 2022). We therefore report their latencies with and without FlashAttention, and denote the resulting variants as sys-optimized and sys-unoptimized, respectively.
By contrast, both ModemBBERT and NeoBERT
As with throughput, Avey-B achieves consistently lower latency and markedly superior long-context scaling (see Fig. 4 (a)), underscoring its computational efficiency even the absence fused kernel. To quantify long-context behavior, we again fit power-law model, L(N) x N where N denotes the sequence length and larger exponents indicate worse scaling. Under the optimized setting,
exhibit MModemBERT = 1.17 and &NEOBERT 1.20, reflecting the bandwidth and memory eeroorr limitations inherent to quadratic attention. In contrast, Avey-B-torch-compit attains substantially milder exponent of 3xvey-B 0.68, and delivers more than a 3x and 10x latency advantage over BodeenBERTTsss-opiimmee and NcoBERT -osBERRT-yy-optimizze respectively. at 96k tokens.
-et--nBBEET-ssso-iiiiieeee and CeoBERT-sys-optimnn
The unoptimized setting further accentuates these differences (see Fig. 4 (b)). and NeoBERT-sys-unppimizz
exhibit substantially steeper latency growth, with exponents PMuderBBRR 1.42 and ANESBERT = 1.63, and both models encounter aut-of-memory failures well before the maximum tested sequence length. In contrast, Avey-B-eager achieves the shallowest growth all configurations, with Bvvey-R = 0.58, and main- tains stable latency across the entire sequence-length range. These results confirm that Avey-B's latency advantage is structural, especially since its neural processor depends on split size rather than global sequence length, yielding linear (NN) scaling and robust long-context performance even in the absence of compiler- or kernel-level optimizations.
BlodemBERT-sy--uopppiie
J CROSS-SEED VARIANCE ANALYSIS
Table Standard deviations across 10 random seeds for all evaluated encoders and benchmarks.
SC
TC
QA
IR
Model
MNLI
QQP
SST-2 CONLL Onto.
UNER ReCoRD SQuAD SQuADv2 MLDR MSMARCO NQ
Avey-B
0.92
0.12
0.97
0.71
2.65
0.67
0.17
0.47
0.67
0.75
Bave
BERT
0.17
0.16
0.31
1.10
0.31
0.69
3.70
0.53
0.32
1.22
0.73
1.44
RoBERTa
0.13
0.10
0.38
0.25
0.14
0.60
0.42
0.11
0.19
0.32
1.02
0.72
ModerBBERT
0.12
0.53
0.24
0.45
0.70
2.36
0.29
1.40
2.18
NeoBERT
0.40
0.14
1.20
0.24
0.17
0.47
5.98
0.91
0.66
4.70
4.30
9.48
Avey-B
0.20
0.43
0.52
0.33
0.10
1.06
0.27
0.15
0.30
1.03
1.77
1.06
Lange
BERT
0.28
8.24
0.98
0.37
0.13
0.80
2.47
0.97
0.58
0.94
1.54
1.01
ROBERTa
0.16
0.20
0.41
0.26
0.61
0.26
0.33
0.18
0.50
1.58
0.74
ModernBERT
0.18
0.08
0.35
0.50
0.14
3.67
17.79
0.25
1.35
2.27
1.70
3.06
24

---

Published as a conference paper at ICLR 2026
To complement the median results reported in Table ness by examining its sensitivity to random initialization. As described in Section [5.] for every benchmark in the SC, TC, QA, and IR categories, configuration using 10 independent random seeds. While Table 2 reports the median performance across seeds at the best learning rate for each benchmark, we additionally compute the standard deviation (SD) across the 10 runs for each model-benchmark pair. These SD values quantify the variability induced by initialization and provide an additional perspective on optimization stability and robustness beyond median performance.
further evaluate each model's robust-
swept four learning rates and fine-tuned each
At the base scale (and medium for NeoBERT), RoBERTa exhibits the lowest overall variance. con- sistent with its well-established fine-tuning stability. Avey-B ranks second, followed by Modern- BERT, BERT, and NeoBERT, in that order. Across most benchmarks, Avey-B maintains tightly concentrated variances, with the exception of UNER, where single outlier resulted in higher vari- ability (SD 2.65).
At the large scale, the differences in stability become more pronounced. ModernBERT, despite strong median performance, exhibits substantial instability on several benchmarks (most notably ReCoRD, UNER, and NQ), suggesting high sensitivity to initialization arising from its alternating attention pattern and extended context window. Likewise, BERT demonstrates occasional catas- trophic variance spikes (c.g., QQP with SD 8.24), indicating susceptibility to poor optimization minima. In contrast, Avey-B maintains uniformly low SDs across nearly all benchmarks, with no signs of pathological instability. Its variances remain tightly bounded (typically below 1 1.06), often surpassing most Transformer- based baselines and again ranking just behind RoBERTa.
These variance measurements show that Avey-B among the most statistically consistent encoders in our evaluation. We attribute this robustness to three core architectural principles: (1) the decou- pling of static and dynamic layers, which prevents destructive interactions between fixed parameters and similarity scores; (2) row-normalized similarity matrices, which stabilize activation magnitudes and ensure well-behaved gradient flow; and (3) neural compression, which filters out irrelevant sig- nals in retrieved contexts. Collectively, these mechanisms reduce sensitivity to initialization and foster smoother optimization dynamics, accounting for Avey-B's consistently low variance across tasks.
K EFFECT OF ENFORCING NON-NEGATIVITY IN STATIC LAYERS
Table 12: Effectiveness results for unconstrained Avey-B design (i.e., Avey., --sinB--) that allows mixed signs (admitting inhibitory effects), and an unsigned variant (i.e., Buey-B--nsigned), which enforces nonnegativity in each static layer.
Model
SC
QA
MNLI QOP SST-2 Avg. CONLL UNER Avg. ReCoRD SQuAD SQuAD-v2 Arg.
MS-MARCO
NQ Avg.
Avey-B-signed 80.74 88.91 91.97 N7.21
93.25
92.73
39.60
68.52
60.48
56.20
$7.49
90.38
75.64
74.50
Avey-B-unsigned
78.66 87.09
90.94
85.76
91.07
92.32
92.49
91 96 38.51
66.47
58.25
$4.41
58.90
89.79
73.30
74.00
As discussed in Section 4.2, static layer is similarity hence, can neither alter the values of similarity scores produced by dynamic layer nor introduce token-specific, similaritycconditione sign changes. At most, it can apply a global (possibly negative) gain to a neuron's 's aggregate, which may change the overall sign while preserving the magnitude ordering induced by dynamic layer. We now examine whether permitting sign changes of dynamic scores within a static layer is ulti- mately helpful or harmful.
To this end, we adopt (1) the best-performing sequence length N, split size S, and top-k depth from Appendix F Ni.e,,N-2045 S-256, k=3); (2) the best static-dynamic arrangement from Appendix D (i.e., the interleaved static-dynamic pattern); and (3) the most effective dynamic-layer normalization from Appendix E (i.e., row-wise normalization by the sum of similarities). We refer to this full Avey-B configuration Avey-B-signed and compare it to variant that makes a single change, that is, enforcing nonnegativity in each static layer, denoted as Avey-B B-nniinndd Lastly, we follow the experimental setup presented in Appendix C
25

---

Published as a conference paper at ICLR 2026
12 14
Figure 5: Learned static cross-embedding projection matrices for the (a) coupled configuration (left or red) with 15 matrices uniformly subsampled from static layers and (b) decoupled configuration (right or blue) with all 15 static matrices (dynamic and static layers static matrices exist). For comparability, we display 15 layers per panel. The coupled setting exhibits diffuse, more homogeneous patterns (e.g., see layers 14, 22, 24, and 26) suggestive of redundancy, whereas the decoupled setting shows sharper, more heterogeneous structure and variability in spread, indicating greater representational diversity.
interleaved, hence, only 15
As illustrated in Table 12 enforcing nonnegativity on all learned static weights (i.e., Avey-B- unsigned) consistently degrades performance relative the default Avey-B design (i.e., Avey-B- signed). Averaged within task families, the constraint reduces effectiveness on SC, TC, QA, and IR by -1.4, 00.77, 1.79, and -0.51 points, respectively. Collapsing the four family averages yields an overall effectiveness drop of 11.13 points.
As discussed Section 4.2, constraining V 2 (see Equation 2 climinates inhibitory (negative) contributions in each static layer, yielding purely additive mixing. This reduces representational con- trast and the ability to subtract misleading context, particularly salient in QA, where disambiguation among semantically similar spans is critical, hence, the larger average loss there. The small MLDR uptick (under IR) suggests that purely additive prior can occasionally act as mild regularizer for simpler retrieval signals, but it does not offset the broader declines across task families.
In summary, the results shown Table 12 support retaining signed weights in static layers to pre- serve inhibitory patterns and maintain the monotonic, similarity-respecd updates established by dynamic layers.
L COUPLED VS. DECOUPLED LAYERS: A STATISTICAL ANALYSIS
We now analyze the learned cross-embedding projection matrices (say, V) for the coupled and de- coupled Avey-B models from the ablation study in Appendix H Table 13 reports summary statistics for each model. In the coupled case, we observe a clear positivity bias, especially in deeper layers, wherein the fraction of positive entries (i.e., the number of positive weights divided by the total number weights) approaches one in several layers (in layers and 13, indeed hit 1). This bias can be explained follows. Because the enricher employs nonnegative activation (i.e., ReLU?), the contextualizer's similarity matrix (say, S) is elementwise nonnegative. As such, the coupled mixing M @ inherits its signs entirely from V. Any negative entry in V flips large positive similarity into negative contribution, violating monotonicity with respect to relevance (see Ap- pendix A) and degrading training. The optimizer therefore pushes V toward nonnegativity to avoid these destructive sign inversions, yielding the observed late-layer collapse toward positive weights.
Despite this positivity bias, a nontrivial fraction of negative entries persists in the coupled model (see Table 13 againn. This residual negativity is precisely the failure mode our hypothesis predicts, that is, wherever a neuron retains negative weights, large positive similarities can inverted into negative contributions, yielding local violations of relevance monotonicity.
26

---

Published as a conference paper at ICLR 2026
By contrast, in the decoupled case the dynamic layers alone produce the mixing weights. These weights are normalized and nonnegative by construction, so monotonicity is enforced at the similar- ity operator. The static layers are learned separately and no longer need to be driven into nonneg- ativity to preserve monotonicity. As shown in Table 13 this yields a near-zero mean with roughly balanced positive and negative weights (without the late-layer positivity bias), retains inhibitory patterns (i.e., learned negative influences) where useful, and avoids the sign-flip failure mode.
Beyond sign distribution, the two models also diverge in the dispersion of their weights. Coupled matrices exhibit reduced standard deviation across layers, indicating more stable transformations that converge toward smooth and homogeneous patterns. Decoupled matrices, by contrast, sus- tain larger fluctuations, admitting both stronger positive and stronger negative values. This higher variance may reflect greater representational flexibility. Norm statistics supports this interpreta- tion, whereby coupled matrices accumulate larger l norm, distributing weight more evenly across entries, whereas decoupled matrices attain slightly higher lz values, implying that fewer entries dominate with sharper magnitudes.
Qualitatively, the static matrices in both variants exhibit Toeplitz-like (approximately shift-invariant) structure reminiscent of gMLP (Liu et al.] 202T) (see Fig. 55 As in gMLP, where such patterns emerge without an explicit prior. our static layers converge to diagonally dominant, near-diagonal matrices indicative of locality. This alignment suggests that ture can arise naturally in architectures that employ fixed, input-independent transformations to stabilize and scaffold subsequent dynamic computations.
Toeplitz- like struc-
In summary, coupling tends to regularize the cross- cross-embedding projections toward homogeneous, nearly nonnegative transformations, whereas decoupling promotes healthy diversity and sharper structure, while preserving monotonicity with respect to relevance.
M LONG-RANGE BENCHMARK RESULTS
In this section, we evaluate the long-contex capabilities of Avey-B, ModernBERT, and NeoBERT using a synthetic reelee-n-a-haystade (NIAH) benchmark formulated as an extractive ques- insweengg (QA) task augmented with position-sensitee reasoning. Each instance consists of a passage of specified length (e.g., 96k tokens) filled with random distractor tokens and one or more key-value pairs, where the value constitutes the "needle." The query contains only the key, and the model must extract the corresponding value from the passage.
In the single-needle setting, the task measures model's ability to semantically locate the correct span within an extremely long sequence. In the multi-needle setting. all key-value pairs share the same key, and the query explicitly requests the nh occurrence. This removes any semantic disambiguation and introduces a -eestiin-eensiiive reasoning requirement, where the model must identify all candidate spans and reason over their order to select the correct needle.
The evaluation set comprises 40% single-needle (pure long-context QA) and 60% two-needle sposition-sensitive reasoning) examples, thereby jointly assessing semantic retrieval and positional reasoning under extreme sequence lengths, with the latter emphasized by design. We further con- struct two variants of the benchmark with different classes of randomly generated needles, namely, alphanumeric (NIAH-I) and numeric (NIAH-2), following the setup introduced in Avey (Hammoud & Acharya 2025).
Tables 14 and 15 illustrate results for Avey-B base and large, ModernBERT base and large, and NeoBERT medium (the only publicly available size) on NIAH-1 and NIAH-2 across sequence lengths from 1k to 96k tokens. For ModernBERT and NeoBERT, we show results only up to their respective trained context windows (i.e., 8k and 4k tokens). In contrast, Avey-B imposes no fixed maximum sequence length and generalizes scamlessly beyond its trained 2,048-token context win- dow, enabling evaluation up to 96k tokens.
On NIAH-1 (Table 14), Avey-B exhibits strong robustness and scalability relative to ModernBERT and NeoBERT. Both Avey-B base and large maintain cear-constant accuracy from 1k to 96k to- kens, with only modest 3-4 point decrease demonstrates that Avey-B BB effectively resolves long-range dependencies and generalizes far beyond its trained context window. By comparison, ModemBERT and NeoBERT cannot operate beyond
96x increase in sequence length. This stability
27

---

Published as a conference paper at ICLR 2026
Table 13: Layer statisties for coupled vs. decoupled settings. For comparability, display layers setting. For the coupled setting, we uniformly subsampled 15 layers from 30 static layers. For the decoupled setting, all the 15 static layers are shown (dynamic and static layers are interleaved, hence, only 15 static layers exist). The coupled setting exhibits positivity bias (see the "fraction of positive" column), while the decoupled setting demonstrates more balanced positive and negative weights, indicating greater representational diversity.
Coupled
Layer
Mean
Std
Min
Median
Max
Abs. Mean
1.1 Norm
1.2 Norm
Frac. Pos.
Frac. Neg.
0.00
0.12
-1.07
0.00
1.20
0.05
3369.19
30.68
0.47
0.53
2
0.01
0.11
-1.55
0.00
0.30
0.04
2345.75
27.
0.53
0.47
0.08
0.09
-0.23
0.06
0.68
0.09
5906.35
31.56
0.90
0.10
4
0.00
0.10
-1.19
0.01
0.29
0.03
2211.32
24.42
0.61
0.39
0.03
0.13
-1.09
0.00
0.15
3724.35
33.82
0.55
0.45
6
0.01
0.08
-1.47
0.00
0.63
0.03
1963.87
21.33
0.57
0.43
7
0.03
0.11
-0.29
0.00
1.00
0.06
3959.52
30.11
0.52
0.49
0.11
0.04
-0.14
0.11
0.33
0.11
7525.47
31.17
1.00
0.00
9
0.00
0.10
-..88
0.01
1.43
0.04
2300.72 72
25.97
0.34
0,66
0.09
0.07
0.38
5844.50 50
27
0.98
11
-0.02
0.10
0090
0.01
0.12
0.04
2604.39
26.53
0.59
0.41
12
0.07
0.02
-0.04
0.07
0.16
0.07
4697.84 84
19.12
0.98
0.02
0.05
0.03
-0.04
0.05
0.21
0.05
3487.26
15.28
1.00
0.00
14
0.03
0.02
-0.14
0.03
0.18
0.04
2293.24
9.56
0.95
0.05
0.00
0.39
-0.01
0.83
1912.78
18.29
0.34
0.66
Avg.
0.03
0.64
0.53
3609.77
24.87
0.69
0.31
Decoupled
Layer
Mean
Std
Min
Median
Max
Abs. Mean
L1 Norm
L2 Norm
Frac. Pos.
Frac, Neg.
0.00
0.08
-0.98
0.00
1.04
0.04
2327.6 62
21.31
0.51
0.49
0.02
0.12
-1.27
0.00
0.05
3065.94
0.51
0.49
3 0.01
0.12
-1.93
0.01
0.27
0.04
2312.23
30.24
0.66
0.34
0.01
0.12
-1.01
1.35
2825.07 07
0.60
5
0.00
0.10
-0.50
-0.01
1.70
0.03
2129.54
26.59
0.42
0.58
6 0.00
0.11
2.09
0.00
0.68
0.03
1978.08
28.17
0.61
0.39
7
0.00
0.11
-160
0.01
0.33
0.03
2050.08
27.01
0.69
0.31
8
0.01
0.11 -1.22
0.01
0.30
0.03
2104.69
27.52
0.70
0.30
0.00
00.66
0.00
1.37
1697.87
23.67
0,42
0.58
10
0.01
0.12
-0.34
-0.01
1.27
0.03
21 45.28
30.16
0.34
0.66
0.00
0.09
-0.28
00.11
1.18
0.03
1775.56
23
0.35
0.65
0.01
0.11
-022
-0.01
1.62
0.03
2019.32
29.21
0.33
0.67
13
0.02
0.11
-133
0.00
0.24
0.04
2637.36
28.99
0.58
0.42
0.02
0.10
-1.04
0.00
0.18
2292.65
26.27
0.61
0.39
15
0.01
0.08
-0.06
0.00
0.82
0.03
1779.38
19.60
0.51
0.49
Avg.
0.00
0.10
0.94
0.00
0.84
0.03
2209.38
26.83
0.52
0.48
their 8k-token and 4k-token trained windows, respectively. NeoBERT matches Avey-B at very short contexts (1 (1-2k) but drops by roughly points at 4k, while ModemnBERT lags behind Avey-B by 10-12 points even at short sequence l Moreover, ModemBERT large fails 8k tokens due to out-of-memory issues even on NVIDIA B200 GPUs using the smallest feasible batch size.
Results on NIAH-2 (Table 15 mirror the trends observed on NIAH-1 and further reinforce Avey-B's long-context robustness. Both Avey-B base and large maintain strong performance across extreme
'For ModernBERT versus NeoBERT, we hypothesize that the consistently lower scores of ModernBERT stem from its local-global alternating attention pattern, compared to the full bidirectional self-attention used in NeoBERT.
28

---

Published as a conference paper at ICLR 2026
Table 14: Needle-in-a-haystack (NIAH-I) accuracy across sequence lengths from 1k to 96k for several encoders at different scales (M Medium; OOM = -Outoof-emorry,
NIAH-1
Model
1k
2k
4k
8k
16k
32k
64k
96k
LLarge M Base
Avey-B
79.41
79.21
78.94
79.19
78.91
77.73
77.18
75.72
ModemBBRR
67.74
67.64
68.31
70.67
NeoBERT
79.65
79.13
74.73
Avey-B
79.69
79.24
79.03
79.58
79.44
78.44
76.76
76.06
ModernBERT
68.80
67.52
67.20
0OM
accuracy across sequence lengths from 1k to 96k for several encoders at different scales (M Medium; 0OM = -ut-of-Memory),
Table 15:
Needle-in-a--aystacc (NIAH-2)
NIAH-2
Model
1k
2k
4k
8k
16k
32k
64k
96k
Large M Base
Avey-B
78.29
79.40
79.77
78.53
78.70
75.50
73.78
71.86
ModemBBRR
66.99
67.25
69.49
70.48
NeoBERT
79.61
79.52
80.07
Avey-B 78.94
79.48
79.99
78.71
79.07
78.31
74.47
74.54
ModemBBRRT 66.96
67.29
68.07
OOM
sequence lengths, with Avey-B base decreasing from 78.3 at 1k to 71.9 at 96k tokens, and Avey-B large remaining similarly stable (78.9 + 74.5). ModernBERT, by contrast, trails Avey-B by 9-12 points at short contexts and fails at 8k tokens due to memory limitations, preventing any long-context evaluation. NeoBERT remains competitive at very short lengths (1-4k) but, like ModemBBERT, cannot operate beyond its 4k-token window, and therefore cannot be evaluated in the long-context regime.
In summary, Avey-B B is the only model capable of sustaining high accuracy up to 96k tokens this ceessiionansweeriie reasoning-intensive benchmark, despite being trained with a context window of only 2,048 tokens. These results demonstrate that Avey-B not only generalizes far beyond its trained context width, but also preserves long-range reasoning fidelity in regimes where existing Transformer-basee encoders either fail to extrapolate collapse entirely.
29


Reviews:
[{"review_id": "4qKNL9aIGK", "reviewer": "Reviewer_bfGY", "summary": "The paper introduces Avey-B, a bidirectional encoder reformulation of the recently proposed attention-free Avey architecture. The model replaces self-attention with a rank-and-retrieve mechanism combined with a neural processor. The authors propose three key improvements: decoupling static and dynamic parameterizations to prevent destructive coupling effects, introducing row-normalized similarity for better numerical stability, and adding a neural compression module to reduce computational overhead in bidirectional inference. Across a suite of benchmarks, Avey-B shows competitive or superior performance compared to BERT, RoBERTa, ModernBERT, and NeoBERT, while being more efficient on long-context inputs.", "questions": "The paper claims that the neural compression preserves relevant information while improving throughput. Could the authors provide more intuition or analysis on what kinds of information tend to be preserved or lost through this compression, and how that affects token-level representation quality?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The architectural refinements are thoughtfully motivated. I especially appreciate the clarity in how the authors separate static and dynamic layers—this reflects careful reasoning about the pitfalls of coupled parameterization. The discussion around monotonicity provides theoretical substance rather than heuristic justification, and the normalization strategy is simple yet effective for improving training stability. The neural compression idea feels practical and grounded in real deployment concerns, showing the authors’ awareness of computational efficiency beyond benchmark accuracy. The experiments are broad, well-controlled, and transparent, with clear gains on token classification and retrieval tasks.", "weaknesses": "While the work is well executed, the novelty is somewhat incremental relative to the original Avey model. The decoupling and normalization ideas, though meaningful, read more as refinements than as a fundamentally new architecture. Efficiency comparisons would be stronger with a fused-kernel implementation to remove framework overhead. I also would have liked to see more analysis of how the compression affects representational quality or long-range dependency modeling, as well as sensitivity studies for key hyperparameters like split size and retrieval depth. Finally, although the results are strong on standard tasks, the evaluation scope remains narrow—additional evidence on more diverse or reasoning-heavy benchmarks would make the contribution more compelling.\n\nMinor typos (do not affect the score): \n- Fig1 (a,b)  caption: \"Parametrization\" -> \"Parameterization\"\n- in conclusion: numbering typo (two #2’s)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kI6RPmrThv", "reviewer": "Reviewer_gF16", "summary": "This paper introduces Avey-B, a bidirectional encoder architecture that serves as an alternative to the dominant Transformer-based models. The authors propose several architectural innovations, including decoupled static/dynamic parameterizations and a neural compression module, to adapt the recent attention-free Avey model for the encoder-only paradigm. The results demonstrate that Avey-B is competitive and often superior to strong baselines on token classification and information retrieval tasks, while showing significant efficiency gains on long sequences.", "questions": "* **Ranker design**: Could the authors provide an intuitive justification for why selecting splits based on non-contextualized token representations is not a major limitation of the approach (see the edge case mentioned in the “weaknesses” section)?\n* **Statistical significance**: Could the authors report measures of variance (e.g., standard deviation across seeds) or perform statistical significance tests for the results in Table 2 to substantiate the claimed improvements?\n* **Explanatory figure**: Would it be possible to include an illustrative figure showing how masking, training, fine-tuning, and inference operate on a simple example to clarify the overall process?\n* **Long-context evaluation**: Could the authors provide additional results on long-context tasks, such as NIAH adapted for encoder models (e.g., in an extractive QA setting)?\n* **Hyperparameter tuning**: Could the authors clarify how hyperparameter tuning was performed, specifically, whether a validation set was used or another procedure was followed?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* **Innovative architecture for efficient encoders**: The paper presents a commendable attempt to move beyond the dominant Transformer paradigm. The proposed Avey architecture represents a promising step toward more efficient, attention-free encoder designs.\n* **Potential for long-sequence applications**: The results suggest that the core ideas underlying Avey hold significant potential, particularly for tasks involving long sequences where computational efficiency is a primary concern.", "weaknesses": "* **Architectural limitation**: The ranker’s reliance on MaxSim over non-contextualized embeddings appears to be a major limitation. This design makes the crucial context-selection step purely lexical, preventing it from leveraging the deeper semantic representations learned in later layers. In edge cases where certain splits are repeated multiple times in a document, the ranker would likely retrieve these redundant splits to form the context.\n* **Significance assessment**: The main results in Table 2 report only median scores across 10 seeds, with no measure of variance (e.g., standard deviation). Without such information, it is unclear whether the observed improvements are statistically meaningful or simply due to random variation.\n* **Ambiguity in training and inference procedures**: The paper does not clearly explain how the model operates in practice. Notably, it remains unclear whether masking occurs before or after split selection. Given the novelty of the architecture, an explanatory figure illustrating the masking, pre-training, fine-tuning, and inference processes would be highly valuable.\n* **Long-context capabilities**: Although long-context efficiency is a central motivation, the evaluation does not include established long-context benchmarks beyond MLDR.\n* **Insights from efficiency plots**: Figure 2 shows that Avey-B’s throughput remains nearly constant with sequence length, which is expected since the model processes a fixed number of tokens (S×(k+1)) per split. A more informative analysis would examine throughput as a function of context width by varying S and k. Moreover, extending sequence length up to 96k tokens offers limited practical insight, as such lengths are rarely encountered in real-world encoder applications.\n* **Unclear generalization of hyperparameters**: Optimal values for Avey-specific hyperparameters (N, S, k) were obtained through extensive tuning (Table 7). It seems unclear whether these settings generalize to other domains or give Avey-B an in-domain advantage over baseline models.\n* **Hyperparameter tuning methodology**: The paper does not specify how Avey-specific hyperparameters were selected (e.g., via validation set or grid search), making it difficult to interpret the connection between Tables 2 and 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HceTSMMmqP", "reviewer": "Reviewer_QEZg", "summary": "Avey-B is an encoder architecture that does not rely on traditional attention mechanism and extends by design to very long sequences. The model splits long sequences into splits of S tokens, and for each split, leverages a retriever module to obtain the top-k most similar splits based on max similarity operations, a neural compressor to contextualize each split with information from similar splits in constant time, and a neural processor that replaces the transformer architecture with alternative softmax free operations.\n\nThe resulting model is by design scalable to long context lengths, and experimental results shown (on short context benchmarks) are surprisingly strong, especially given the small data regimes, rivaling or even beating transformer encoders on most tasks.", "questions": "Reasons for wanting to process huge sequences with encoder models are less clear to me than in the case of decoders. What use cases do you have in mind in which 1M token long sequences are useful with encoders ?\n\n\nCan you give more intuitions of why Avey can outperform transformers even on very short contexts ?\n\n\nCould you draw more parallels into how the neural processor within a split differs from an attention mechanism ?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The insight of contextualizing splits efficiently through chunk-based retrieval is interesting (although not novel I believe). This paper however proposes an architecture which is entirely designed to properly contextualize external chunk information, and proposes various design ablations. \n\nBeyond long context processing, Avey-B showcases very strong results on short-context benchmarks, with an architecture quite different from traditional encoder transformers - even under limited data refimes. This is interesting on its own and would warrant further explorations and intuitions.\n\nFinally, models and code are released, enabling future external experimentation and confirmation of claimed interests of this method.", "weaknesses": "**Introduction/Conclusion clarity**: The introduction is hard to read and quickly dives into details. It often contrasts Avey-B the model at hand with Avey, an autoregressive variant, but I don't think the reader should be expected to know how Avey works to be able to understand this paper easily. To illustrate, l60-67 are really hard to understand without skipping ahead, understanding the mechanisms and going back.  More generally, I do not believe framing the entire paper as \"an extension to Avey\" serves this work in terms of clarity and readibility. The design should be independently motivated in the case of encoder models.\n\n\n**Figures**: Figure 1 is not very clear to me. Figure 2 would benefit from also reporting throughput results at short sequence lengths (from 0 tokens to 2000 tokens) which is the operating range of most of your evals and current encoder use cases. It seems to me that Avey-B has fixed costs that depend on split length, and that although it is linear with respect to long sequence lengths, the method can remain costly in comparison to transformers on shorter sequences (on par or below split length).\n\n\n**Long context results**: Most of the paper relies on the claim Avey is much more efficient in long context settings. Beyond the fact efficient implementations are not proposed - and it isn't clear to me if FA2-like mechanisms could work here - (so in practice, Avey-B remains slower than ModernBert until 32k tokens), the bigger issue is that long context evaluations are never done. The model is evaluated on mostly very short sequence tasks (GLUE, etc) and the longest sequence benchmark would be MLDR for IR that corresponds to about 3k sequence lenghts, largely below claimed processing lenghts.\n\n*Claiming the model extends to million token sequence lenghts seems like an insuficiently backed claims under these conditions.*\n\nFurthermore, IR benchmarks are probably less affected at a token level by the neural compression (since we are interested in mean pooled representations anyways). It would be interesting to evaluate token-level tasks (NER, etc) on long sequences to evaluate whether the neural compressor keeps a strong token level signal.\n\n**Decorrelating architectural impact**: Avey-B is mostly evaluated on short sequences which are often on par with Avey's split Size S (256 tokens). To my understanding, the maxsim retrieving + contextualization here would thus not benefit Avey-B since sequences are short anyways. Under these lights, I have a hard time understanding why Avey would outperform/equal transformers in these short sequence tasks. You give intuitions for Token Classification tasks, but I feel exploring this more would be of interest.\n\n\n\n**Overall**: As of today, Avey-B remains slower (and overall less performing) than ModernBert for sequences smaller than 32k sequence lengths, and fails to show experimentally that it works for sequences above ~8k tokens. The interest of this model thus remains mostly experimental, and I believe the proposed attention-free mechanism should be further dissected, analyzed, and clearly explained to give readers intuitions and belief that it is worth further exploring.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9WADk56NUG", "reviewer": "Reviewer_LwXp", "summary": "The paper proposes Avey-B, a bidirectional, attention-free encoder derived from Avey. Key ideas are: (i) decoupled parameterization that alternates dynamic (similarity-only) and static (learned linear) layers to avoid similarity/weight sign-flip pathologies and to enforce a per-layer monotonicity property; (ii) row-normalized cosine in dynamic layers for stability; and (iii) a neural compressor in the ranker that condenses each split plus its top-k retrieved splits back to split size before the processor. Empirical results that support the claims of the paper are presented.", "questions": "My main question is related to Weaknesses 2:     \nCan you demonstrate that, when relevant evidence is dispersed across very long inputs, Avey-B, with its ranker and compressor, recovers Transformer-level accuracy while maintaining efficiency?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The architectural motivation is strong, and the paper clearly explains how Avey-B extends the original causal Avey design.\n2. The evaluation is solid, spanning four areas: sequence classification, token classification, information retrieval, and question answering.\n3. In fair settings (without FlashAttention), the model is empirically more efficient than ModernBERT and NeoBERT.", "weaknesses": "1. Complexity claim: Section 4.3 states that despite compression, the asymptotic complexity “remains quadratic in N,” but the efficiency section attributes an O(N) total time for fixed split size S (line 459). Which one is true? I assume the linear one; otherwise, the paper's contribution is diminished, as the complexity remains quadratic, similar to transformers. \n2. The experimental section would benefit from an ablation study on long sequence lengths. Something similar to \"Needle in the haystack\" or any other synthetic task, to demonstrate that when relevant information for a query is dispersed across a long context, the transformer's performance can still be recovered with Avey-B. \n- A simple dataset to experiment with might be Long Range Arena [2].\n\n[2]: Long Range Arena: A Benchmark for Efficient Transformers", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)