Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
Authors: Sukjun Hwang
Brandon Wang
Albert Gu
Venue: ICLR 2026 Poster (2026)
Keywords: deep learning
architecture
tokenization
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce H-Net: an end-to-end hierarchical network that compresses raw data through a recursive, data-dependent dynamic chunking process

Abstract: Major progress on language models (LMs) in recent years has largely resulted from moving away from specialized models designed for specific tasks, to general models based on powerful architectures (e.g. the Transformer) that learn everything from raw data. Despite this trend, pre-processing steps such as tokenization remain a barrier to true end-to-end foundation models. We introduce a collection of new techniques that enable a dynamic chunking mechanism which automatically learns content- and context- dependent segmentation strategies learned jointly with the rest of the model. Incorporating this into an explicit hierarchical network (H-Net) allows replacing the (implicitly hierarchical) tokenization--LM--detokenization pipeline with a single model learned fully end-to-end. When compute- and data- matched, an H-Net with one stage of hierarchy operating at the byte level outperforms a strong Transformer language model operating over BPE tokens. Iterating the hierarchy to multiple stages further increases its performance by modeling multiple levels of abstraction, demonstrating significantly better scaling with data and matching the token-based Transformer of twice its size. H-Nets pretrained on English show significantly increased character-level robustness, and qualitatively learn meaningful data-dependent chunking strategies without any heuristics or explicit supervision. Finally, the H-Net's improvement over tokenized pipelines is further increased in languages and modalities with weaker tokenization heuristics, such as Chinese and code, or DNA sequences (nearly 4x improvement in data efficiency over baselines), showing the potential of true end-to-end models that learn and scale better from unprocessed data.

Published as a conference paper at ICLR 2026
DYNAMIC CHUNKING FOR END- TO-D-TO-END HIERARCHICAL SEQUENCE MODELING
Sukjun Hwang', Brandon Wang", Albert Albertt Gu
CCarnegie Mellon University, CCartesia Al
ABSTRACT
Major progress on language models (LMs) in recent years has largely resulted from moving away from specialized models designed for specific tasks, general models based on powerful architectures (e.g. the Transformer) that learn every- thing from raw data. Despite this trend, pre-processing steps such as tokenization remain a barrier to true end-to-end foundation models. We introduce a collec- tion of new techniques that enable a dynamic chunking mechanism which auto- matically learns content- and context- dependent segmentation strategies learned jointly with the rest of the model. Incorporating this into an explicit hierarchi- cal network (H-Net) allows replacing the (implicitly hierarchical) tokenization- LM-detokenization pipeline with a single model learned fully end-to-end. When compute- and data- matched, H-Net with stage of hierarchy operating the byte level outperforms a strong Transformer language model operating over BPE tokens. Iterating the hierarchy multiple stages further increases its perfor- mance by modeling multiple levels of abstraction, demonstrating significantly bet- ter scaling with data and matching the token-based Transformer of twice its size. H-Nets pretrained on English show significantly increased character-level robust- ness, and qualitatively learn meaningful data-dependent chunking strategies with- out any heuristics or explicit supervision. Finally, the H-Net's improvement over tokenized pipelines is further increased in languages and modalities with weaker tokenization heuristics, such as Chinese and code, or DNA sequences (nearly 4x improvement in data efficiency over baselines), showing the potential of true end- to-end models that learn and scale better from unprocessed data.
INTRODUCTION
A broad goal of deep learning is to learn meaningful patterns from raw data, automatically ex- tracting features and building abstractions in end-to-end fashion. However, fixed-vocabulary tokenization, the process of compressing raw text into predefined chunks through algorithms such as byte-pair encoding (BPE) (Sennrich et al. 2015 Kudo & Richardson (018)), remains pervasive handcrafted preprocessing step in modern language models (LMs) (Grattafioni et al. 2024| Brown ct al. 2020). Tokenization comes with host of well-documented drawbacks, from poor character- level understanding to lack meaning and interpretability to degraded performance on complex languages and modalities (Petrov et al. 2023 Ahia et al. 2023 Belinkov & Bisk 2017: Sun al. 2020 Clark et al.] 2022). Replacing the tokenization- LM detokenization pipeline with a single end- to-end model would also adhere better to the spirit of deep learning, idcally scaling more powerfully with data and parameters (c.f. the bitter lesson) (Sutton| 2019) Peric] (025). However, tokenization remains an indispensable component of language models and other sequential data for its ability to compress and shorten sequences; as of yet, no end-to-end tokenizer-free model has matched the performance of tokenizer- based language models when matched for computational budget.
A line of recent works has turned to overcoming tokenization in autoregressive sequence models, which requires addressing series of difficult technical challenges
Direct byte-level language modeling with isotropic architecture can be improved with efficient sequence models such MambaByte (Wang al., 2024), but still incur prohibitive computational costs while underperforming tokenized models in compute-matehe matched settings.
An extended related work can be found in Section B which summarized in Table min-hierarchial models with repeated blocks, such the standard Transformer (Vaswani al. 2017

---

Published as a conference paper at ICLR 2026
To improve efficiency, hierarchical architectures such Hourglass Transformer (Narrot et al, 2022) and MegaByte (Yu et al., 2023) use small byte-level models to compress raw inputs into subsampled sequences, which are then processed with a more powerful standard language model. However, simple pooling strategies such as compressing every k inputs are not data-dependent,,
and perform poorly modalities with variable information rates such as language.
SpaceByte (Slagle, 2024) and Byte Latent Transformer (Pagnoni et al. 2024) introduce data-
dependent chunking strategies such as delimiter- or entropy- based heuristics. These heuristics, however, rely on auxiliary external boundary predictors, and are therefore mopality-peecifie and not fully end-to-end.
Although jointly trainable boundary predictors are the ideal solution, they require optimizing dis- crete selection operations without supervision, which is fundamentally a challenging problem. Consequently, existing end-to-end approaches (Nawrot et al.) 2023) exhibit training instabilities that preclude scaling beyond small models or nesting multi-level hierarchies.
Fundamentally, creating a tokenizer-fit free architecture requires incorporating the data chunking pro- cess directly into the model, while overcoming challenges in efficiency, learnability, and stability at scale.
DYNAMIC CHUNKING: END-TO-END SEQUENCE MODELING WITHOUT TOKENIZATION
In this work, we introduce an end-to-end hierarchical network (H-Net) that compresses raw data through a recursive, data-dependent dynamic chunking (DC) process (Figure D. H-Nets match the efficiency of tokenized pipelines while substantially improving modeling ability, by replacing handcrafted heuristics with content-aware and context-dependent segmentation learned from data.
Hierarchical Processing. The H-Net adopts the hierarchical architecture from prior work (Goel
et al. 2022.
Nawrot et al. 2002 Slagle 2024), resembling an autoregressive U-Net (Ronneberger et al. 2015 (i) raw data processed by small encoder network, (ii) then downsampled and passed through a main network operating on compressed chunks, (iii) and finally upsampled before being passed through decoder network operating on the original resolution. This modularity creates a natural processing hierarchy where outer stages capture fine-grained patterns while inner stages operate on coarse representations akin to traditional tokens. Crucially, while the main network contains the bulk of parameters and can be any standard architecture designed for operating on tokenized language- -uugggee--uhhhh-ee as Transformer (Vaswani et al. (207)) or state space model (SSM) (Gu Dao, 2024) we show that the encoder and decoder networks are strongly improved by using SSMs, which have an inductive bias for compression (Gu 2025).
Dynamic Chunking. H-Net's core is novel dynamic chunking (DC) mechanism which interfaces between the main network and the encoder/decoder networks, learning how to segment data while using standard differentiable optimization. DC is composed of two complementary new techniques: (i routing module which predicts boundaries between adjacent elements through similarity score (ii) and a smoothing module which interpolates representations using the router's 's outputs, attenuating the effect of uncertain boundaries and significantly improving learnability. By combin- ing these with a new auxiliary loss function that targets desired downsampling ratios, and modern techniques for gradient-bast based learning of discrete choices (Fedus et al. [2022] Bengio et al. 2013), DC lets an H-Net learn how compress data in a fully end-to-end fashion.
Signal Propagation. We introduce several architectural and training techniques to improve stabil- ity and scalability during end-to-end optimization. These include: (i) carefully placing projections and normalization layers to balance signal propagation between interacting sub-networks, and (ii) adjusting optimization parameters for each layer based on its dimensionality and effective batch size, which changes between stages of the hierarchical structure.
Altogether, H-Net learns segmentation strategies optimized jointly with the main backbone, dynam- ically compressing input vectors based on contextual information into meaningful chunks. H-Net represents the first truly end-to-end, tokenizer-fre free language model: with a single stage of dynamic chunking, a byte-level H-Net matches the perplexity and downstream performance of a strong BPE-tokenized Transformer at sizes exceeding IB parameters. Empirically, the dynamic chunking module naturally compresses data to a similar resolution as BPE tokenizers (4.5-5 bytes/chunk) and qualitatively learns meaningful boundaries, all without any external supervision or heuristics.
2

---

Published as a conference paper at ICLR 2026
HIERARCHICAL CHUNKING: FROM DATA TO ABSTRACTIONS
Beyond addressing tokenization, H-Net improves general sequence modeling across a wide range of settings. Subword tokenization in language models is special case of chunking- process of building higher-level abstractions from low-level data-and is a central component of intelligence Crucially, because H-Net is fully end-to-end, it can be iterated recursively: the main network can itself be an H-Net. Intuitively, more stages of chunking represent higher order meanings; just as characters can be combined into words, words can be combined into clauses, sentences, and beyond. Iterating the hierarchy should therefore lead even more efficient use of compute and parameters, and more effective reasoning over compressed representations.
Recursive H-Nets represent a new class of foundation model architectures that not only overcome tokenization, but discover and operate over abstractions learned from raw data, leading to higher- quality models with less pre-processing.
Iterating the 1-stage H-Net to 2 hierarchical stages further improves its capabilities and strongly outperforms all baselines, with steeper training curves and better scaling with data. byte-level 2-stage H-Net overtakes the perplexity of strong tokenized Transformer after just 30B training bytes, with the gap widening throughout training, and matches the downstream evaluations of the tokenized Transformer of twice its size.
Finally, H-Nets realize the benefits of overcoming tokenization:
Robustness. Without special data mixes, the pretrained H-Net is dramatically more robust to textual perturbations than the token-based Transformer, as evaluated on the noisy HellaSwag suite of benchmarks.
Interpretability. Qualitative visualizations of learned boundaries reveal that H-Net automati- cally discovers semantically coherent units without explicit supervision, validating that end-to- end learning successfully detects the structural patterns traditionally imposed through handcrafted tokenization.
Other languages. H-Net's improvements are more pronounced on languages without obvious segmentation cues, including Chinese and code (59.9 + 66.3 on XWinograd-22 compared to tokenized Transformer) and DNA language modeling (3.6x improved data efficiency compared to isotropic models).
2 H-NET ARCHITECTURE
H-Nets are hierarchical U-Net-like networks, but with data-dependent dynamic subsampling that is learned end-to-end together with the rest of the model. We first introduce H-Net's hierarchical archi- tecture for multi-level processing (Section 2.1). then present our dynamic chunking mechanism that learns content-awan compression through standard optimization (Section 2.2). Additional details appear in the appendix (Section[C), including: (i) architectural design principles; (ii) more explana-
tions about the dynamic chunking mechanism; (iii) optimization and architectural enhancements for hierarchical sequence modeling; and (iv) preservation of autoregressive properties.
2.1 ARCHITECTURAL OVERVIEW
H-Net employs hierarchical architecture comprising three primary components encoder networks (E), main network (M), and decoder networks (D) where each component is implemented with a stack of sequence mixing layers (e.g.. Transformers or state space models). In its simplest form, single-stage H-Net consists of one encoder network, one main network, and one decoder network. Crucially, the architecture's 's key characteristic lies in the main network's unique property: the main network can another complete H-Net, enabling recursive construction of multi-level hierarchies.
This recursive design allows H-Net to scale to arbitrary depths. In an S-stage model, we denote components at each stage using superscripts: encoder networks as Ex and decoder networks as D= for stages < s < S, with the main network M residing only at the final stage S. For example, a two-stage model contains 0 Eº, M, D1, and 'D", as illustrated in Figure [ (Left) Throughout this paper, we use superscripts denote stage indices, though we omit them when all variables within an equation belong to the same stage.
Chunking is formal concept from cognitive psychology central to human memory and cognition, and the inspiration for this work's 's terminology.
3

---

Published as a conference paper at ICLR 2026
0
D
Dechunking
(d) Upsampler
1 Pe-3
1
Pa-3
Pe-2
Pe-1
2
(c) Smoothing Module
M
(b) Downsampler
p
Chunking
8
(a) Routing Module
Dynamic Chunking
Figure 1: (left) Architectural overview of H-Net with two-stage hierarchical design (S 2). (right) Dynamic Chunking (DC). (bottom-right) Key components chunking layer: (a) rout- ing module for dynamically drawing chunk boundaries, and (b) downsampler that selectively tains vectors based on boundary indicators, reducing sequence length while preserving semantically significant positions. (top-right) Key components of dechunking layer: (c) smoothing module for converting discrete chunks into interpolated representations, and (d) an upsampler that restores compressed vectors to their original resolution based on boundary indicators. Linear in equation equation and STE in equation equation are omitted in the illustration for brevity.
Drawing inspiration from the U-Net architecture (Ronneberger al. 2015), H-Net progressively compresses input sequences into fewer vectors with richer semantic embeddings through a chunking layer, processes these representations in the main network, then decompresses the sequence back to its original resolution using a dechunking layer. Unlike traditional U-Net designs, however, H-Net dynamically determines chunking boundaries rather than using fixed-size pooling operations. The overall pipeline can be formalized as:
Ed (x*):
s ==x))
== D'
where the chunking layer and the dechunking layer operations are defined as:
(x++1 p²) Chunk(&
(2) z² = Dechunk(2*+1 pp) + Linear(&*).
(3)
The initial input to the model x0 E DD where 0 the input sequence length and D is the embedding dimension. Intuitively, p" E [0, represents the chunking router's confidence that the token should be passed into the main stage This value is essential for both the chunk (Section 2.2.1 and dechunk operations (Section 2.2.2)
We further provide full details about H-Net's design principles in Section C.I
'We also sometimes refer to probability is interpreted such in Section E.2 formal prohability.
although do

---

Published as a conference paper at ICLR 2026
2.2 DYNAMIC CHUNKING (DC)
H-Net learns chunking boundaries through end-to-end training, allowing it to identify semantically meaningful units adaptively. Furthermore, this dynamic approach enables the model to allocate computational resources efficiently by compressing low-information regions while preserving high- information content at appropriate granularity.
2.2.1 CHUNKING LAYER
The chunking layer (Chunk in equation equation 2 contains routing module and downsampler, as illustrated in Figure (bottom--right
Routing Module. In natural data, meaningful boundaries tend emerge at points of contextual or semantic shift. From this observation, we add an inductive bias by measuring the similarity between adjacent representations: when context changes, consecutive vectors should exhibit lower similarity. The routing module implements this intuition through cosine similarity between adjacent encoder outputs. Given encoder outputs X, it calculates boundary probabilities and boundary indicators be as follows:
(1- k,-1 11 ||
9: W We k = WE
E [0, b, = (0.20,5)
(4)
where P1 1.0 by definition, ensuring the sequence begins with boundary. This formulation scales cosine similarity into a boundary score or probability: ideally, when consecutive vectors 2;-1 and & span a semantic boundary (e.g., between morphemes, words, or phrases), their projections dr and h diverge in the latent space, yielding low cosine similarity and consequently high boundary probability pr.
The downsampler compresses encoder outputs 3 into a reduced set of vectors x using boundary indicators (b Among potential compression strategies including mean pooling, max pooling, or cross-attention we adopt direct selection of boundary-marked vectors for its simplicity and effectiveness (see Section E.3.6 for ablations).
Downsampler.
As illustrated in Figure (b), this approach follows a straightforward selection rule: vectors where b; 1 are retained in the compressed sequence x*+1, while those where 6, 0 are discarded. Likewise, the same downsampler applies to boundary probabilities, compressing pe into pa+1 for use in dechunking layer (see Section 2.2.2).
2.2.2 DECHUNKING LAYER
The dechunking layer (Dechunk in equation equation consists of smoothing module and up- sampler, as illustrated in Figure (top-right.
Smoothing Module. The critical challenge in training a dynamic chunking module lies in the discrete nature of chunk boundaries, which impedes gradient flow during backpropagation. We in- troduce the smoothing module as technique to address this problem. As illustrated in Figure (c)). this component transforms discrete chunking operations into differentiable computations by creating smooth interpolations between chunks. Concretely, the smoothing module applies an exponential moving average (EMA) with the following definition:
f = P.â + (1 -
(5)
In Section C.2.1 we describe several roles of the smoothing module and provide Figure 4 with more explanations.
Upsampler. We carefully design the upsampler (see Figure [-(d)) that decompresses g0+1 to match the original resolution of inputs in the previous stage 2² with the following definition:
G = pt (1 pt) 2-)
[1
11-p
Pt
if 1, otherwise,
(6)
2 =z by?
STE(c)) = c + stopgradient cz),
(7) (ipsample(z,.c)) = STE (c+) 24 (9)
Each component serves a specific purpose in enabling stable end-to-end learning, which is described in Section C.2.2
5

---

Published as a conference paper at ICLR 2026
2.2.3 RATIO LOss
Without explicit regularization, the model may converge to trivial solutions: either retaining nearly all vectors (negating computational benefits) compressing excessively (losing critical informa- tion). Inspired by load balancing mechanisms in Mixture-of-Experts (MoE) models (Fedus et al.)
2022), which face similar challenges in maintaining balanced expert utilization, we introduce a ratio loss guide compression:
N Cratio N (N- (I
F
be G=
Pt
(10)
t=1
where F' represents the fraction of vectors actually selected, G denotes the average boundary proba- bility, and N controls the target compression ratio. Mechamistically. although F is not differentiable, the network can be trained toward targeted compression ratios through G. which provides contin- uous feedback. When F' = G. the loss attains a minimum of Cratio = - when F = =
Interestingly, the loss can theoretically fall below 1 when F F G (e.g. F + + and G - which we indeed observe during training. Despite this theoretical possibility, the loss effectively guides the model toward the desired compression ratio in practice. In practice, as our architectural design encourages the routing module to make confident decisions (i.e. boundary probabilities ap- proaching 0 or 1), F naturally converges toward G. and the loss effectively guides the model toward the desired compression ratio. Notationally, we sometimes use (No, ,N ...., NN-1)-DC to denote the full dynamic chunking mechanism together with its targeted chunking ratios.
Combined together with the autoregressive prediction loss (i.e., L = CAR + a >. Camn) this mechanism preserves content-adaptive compression: the model learns which vectors to retain based on semantic importance rather than following predetermined patterns, distinguishing H-Net from fixed compression schemes. We fixed 0.03 in all experiments in this paper it provides good balance between prediction accuracy and chunking efficiency; however, in other settings, may be important to choose this hyperparameter more carefully.
3 EXPERIMENTS
We first describe our general experimental protocol for language modeling, used for the majority of our experiments. In Section 3.1 we evaluate on high-quality English dataset, showing significantly stronger performance than baselines, as well as improved robustness and interpretabiliiy from avoid- ing tokenization. In Section 3.2 we extend our evaluation to diverse datasets including Chinese, and code, with even larger performance improvements, demonstrating H-Net's versatility as a general sequence model architecture. In the appendix, share further details about the experiments in this section (see Section D). and provide more experiments and ablation studies (see Section E
Models. We compare against standard tokenized Transformer following the Ilama architec- ture Touvron et al. 2023b, Grattafiori et al., 2024 We additionally compare against several byte-level baselines:
MambaByte (Wang et al. 2024 and LlamaByte are isotropic models using pure Mamba-2 layers and pure Transformer layers, respectively.
SpaceByte (Slagle 2024) represents the canonical hierarchical architecture with external bound- ary supervision, which chunks on spaces and "space-like" bytes [ On English, the space-like like delimiter heuristic empirically has average ratio of 6.0 bytes per chunk.
SpaceByte++ is our modification of SpaceByte that includes our architectural modifications to the hierarchical structure (from Section 2.1). In particular, changes the outer encoder/decoder networks to use Mamba-2, and modifies the layer counts and widths slightly to match the H-Net models below.
H-Net (space) and H-Net (pool) differ from our full H-Net only through the chunking function. H-Net (space) further improves SpaceByte++ with our training improvements to the network (Sec-
'This was called the "Transformer++* in Gu & Dao 2024 since by now it is firmly established, remove the "++".
'BLT is another architecture with external supervision using entropy instead of delimiters, but is unfortu- nately too complex to up and control a baseline. We believe that the delimiter-bass based method is highly competitive. See Section B.1.3

---

Published as a conference paper at ICLR 2026
Table I: Architectures for main language models, all data-/FLOP-matchhn E Do, 8" D M. T and M denote a Transformer and Mamba-2 layer, respectively. For hierarchical byte-level models, the TOKENIZER column lists the chunking mechanism. The numbers before DC indicate downsampling factor N in equation equation 10, for example, (3,3)-DC denotes N N 3. The Per-Innermoot-Chue
measure shows that each chunk dynamically determined by our 1ssage comprises similar number of bytes to the GPT-2 tokenizer, despite aiming for N = 6. All Transformer layers & or D networks,
BPIC
well as LlamaByte, use Sliding Window Attention (SWA) with a window size of 1024. Just as in the original Mamba (Gu & Dao 2024) and Mamba-2 (Dao & Gu 2024) blocks, our Mamba-2 layers have roughly 6(D")² parameters and Transformer layers
have 2(D>) parameters in stage S.
BPIC (17
MODEL
INPUT TOKENIZER
L
#PARAMS
ARCHITECTURE
d.mod=1 (D)
#FLOPs matched to GPT-3 Large
Transforner
Token
GPT2
1792
4,6
760M
T24
1536
LlamaByte
210M
T16
1024
MambaByte
1.0
190M
M28
1024
SpaceByte
Spacelike
6,0
570M
T8 T16
T8
768
1536
SpaceByte++
Spacelike
850M
M4 T28 M4
1021
1536
H-Net (pool)
Byte
6-Pool
8192
M4 T28 M4
1021
1536
H-Net (space)
Spacelike
6.0
850M
M4 T28
M4
1021
1536
H-Net (1 (1-stage)
6-DC
4.8
680M
M4 T22 + M4
1024
1536
H-Nat (2-stage)
(3,3)-DC
870M
M4
TIM4
T26
M4T1
M4
1024.
1024
1536
#FLOPs matched GPT-3
Trameformer
Token
GPT2
1792
T24
2048
SpaceByle++
Spacelike
6.0
1.6B
M4 T31 + M4
1024
2048
H-Net (space)
Spacelike
6.0
1,6B
M4 T31 M4
1024
2048
Byte
8192
H-Net (tastage)
6CDC
T24
1024
2048
H-Net (2-stage)
(3.3)DDC
1.6B
TIM4 T27 M4TI M4
1024.
1536
20-18
tion C.3) in particular, adding post-network norms, residual projections, and learning rate multi- pliers on the outer networks. H-Net (pool) baseline ablating the effect of simple chunking strategy that pools every h tokens, which is expected to be weaker than all of the data-dependent chunking strategies.
H-Net (1-stage) is our full H-Net method with DC learned end-to-end (Section 2.2 with com-
pression target N
H-Net (2-stage) is our full H-Net method, iterated to nested stages using No 3, N 3.
We provide results for two model scales, Large (L) and XL. Each scale FLOP-matched to the corresponding GPT-3 (Brown et al. 2020) (i.e., GPT-3 L and GPT-3 XL. variant of the tokenized Transformer (760M and 1.3B parameters respectively).
Experimental Setup. Following established practice (Xue et al. 2022 Wang et al. 2024 Slagle
2024 measure performance using bits-per-byte (BPB) ensure comparability across different input representations. For tokenized models, this amounts to simply rescaling the total negative log likelihood of a sequence (in tokens) by the total number of bytes. In addition, we systematically control the data and compute budget for all models (see Table [. matching all models carefully in both bytesh and FLOPs-per-byle:
Data Budget: We train all models the 100B token subset sampled from the FineWeb-Fad dataset (Penedo et al. 2024). All tokenizer-free models process 8192 ut encoded bytes per sequence, while the Transformer to 8192 bytes). We use batch size 256 for all models; the total batch size is just under 0.5M tokens per batch for the baseline BPE Transformer, roughly matching protocols from prior work (Gu & Dao 2024).
1792 tokens from the GPT2 tokenizer (roughly equivalent
Compute Budget: For calculating FLOPs, follow standard methodology (ooffaann et al., 2022) with an extension for Mamba-2 layers (see Section D..I). We use the BPE-tokenized Trans- former's i #FLOPs reference, and the number of layers of the other models is adjusted accord- ingly to match the reference #FLOPs.
Training employs AdamW (Loshchilov & Hutter 2017) optimizer with warmup-stable-decaa (WSD) (Hu al. 2024) scheduling with 10% linear warmup and 20% inverse-square-root de-
7

---

Published as a conference paper at ICLR 2026
#FLOPs matched GPT-3 Large (760M)
#FLOPs matched to GPT-3 XL (1.3B)
1.08
0.95
1.00
Validation Bits-per-byle
0.90
0.96
0.90
0.85-
100B
0.85
200B
308
H-Nut 12 (2-stegs) over lines
[BPE] Transfomerr with 308 saiiing bytea
0.75
538
1008
1808
2008
2003
3008
3538
508
1038
2508
3008
3538
Total Training Bytes
Total Training Bytes
H-Net (2-stage)
H-Net (space)
Space
LitmaByte (SWA)
(BPE)
(1-stage)
(8-pool)
SpaceByte
Lite (Global)
MambaByle
Validation Bits-per-byte (BPB) throughout training for different models at Large (760M, left) and XL (1.3B, right) scales with matched computational and data budgets for train- ing. All models but Transformer take raw byte inputs (Transformer uses GPT-2 tokenizer). Vertical dotted lines indicate crossover points where H-Net begins to outperform Transformer with prede- fined BPE tokenization. From the curves we can clearly see the following: (1) all hierarchical models (i.e., SpaceByte++, H-Net variants) outperform the isotropic models (i.e., Transformer, ManbaByte, LlamaByte); (2) dynamic chunking is more powerful than BPE tokenizers; and (3) DC is more effee- tive than other chunking strategies. Furthermore, H-Net's 2-stage variant consistently outperforms I-stage across both scales, demonstrating the effectiveness of deeper hierarchies. See Table | for architectural details.
Figure 2:
cay (lbrahim et al. 2024). Following Hägele et al. (022) which recommends WSD schedulers with half the maximum learning rates as a cosine schedule, we adopt learning rates 2.5x higher than GPT-3 (Radford et al. 2019) standards; this corresponds to half of the maximum learning rate used in Gu & Dao (024). yielding 6.25 x 10-1 for Large-scale models and 5.0 10-4 for XLass models. Architecture details include gated MLPs (Touvron et al. 20236) in all Transformer layers and the main network's Mamba layers. while Mamba layers Transformer layers in & and D, we use Sliding Window Attention (SWA) (Beltagy et al. 2020) with the window size of 1024. As discussed Section C.I È and comprise mainly Mamba-2 layers.
and D are without an MLP. For
3.1 LANGUAGE MODELING
Training Curves. Figure presents validation BPB metrics throughout training for both Large and XL model scales. At the Large scale, we make note the following comparisons: (i) all isotropic models severely underperform hierarchical models. Among these, MambaByte is signifi- cantly better than LlamaByte, both the FLOP-matched sliding window attention (SWA) variant and the global attention variant that is data-matched but uses 2x the FLOPs; (ii) H-Net (pool) is much worse than all other H-Net variants, validating that fixed-width chunking is not effective; (iii) SyaceByte is much worse than SpaceByte++, validating our strategy for network design as well as usage of Mamba in the outer networks (Section 2.Th (iv) SpaceByte++ is in turn worse than H-Net (space), validating our improved signal propagation techniques (Section C.3) ; (v) H-Net (space) is very strong model reaching the performance the BPE Transformer. validating the effect of data- dependent chunking strategies together with a well-designed hierarchical architecture; (vi) H-Net (1-stage) is stronger than H-Net (space), validating that our dynamic chunking mechanism success- fully learns how to segment data in a context context-dependen way that improves over strong heuristics; and (vii) H-Net (2-stage) is significantly better than H-Net (1-stage), validating that iterated dy- namic chunking can potentially learn a nested hierarchy of useful features, and leverage compute and parameters even more effectively.
At the XL scale, we zoom more closely and compare only the strongest set of methods: Space- Byte++, H-Net (space), H-Net (I-stage), and H-Net (2-stage). The same trends hold as at the Large scale. Our SpaceByte++ baseline is strong, but slightly worse than the BPE Transformer baseline. On the other hand, all byte-level H-Net methods start off worse than the token-level Trans- former, but scale better after enough data. H-Net (space), H-Net (1-stage), and H-Net (2-stage)
8

---

Published as a conference paper at ICLR 2026
Table 2: Zero-shot performance comparison across multiple benchmarks, all data-/FLOP-
matched. Evaluation results on seven downstream tasks at both Large (760M) and XL (1.3B) scales. GFLOPs/B YTE measured on Fine Web-Edo validation reported as the average over the course of training. See Table for architectural details.
MODEL
INPUT
GFLOPs/
F-EDU BPB
LMB.
HELLA.
PIQA ACC
ARC-E
ARC-C
WINO.
OPEN. ACC.n
AVERAGE ACC
BYTE
ACC
ACC.n
ACC1
ACC
ACC
#FL.OPs matched to GPT-3 Large
Transformer
Token
0.42
0.756
45.0
54.5
72.3
69.9
36.3
55.9
38.8
53.3
LlamaByte
0.42
0.859
37.0
40.5
64.7
55.1
26.7
32.4
44.1
LlamaByte (Global)
0.95
0.845
36.4
41.5
65.7
57.2
27.1
49.8
32.2
44.3
MambuByte
0.42
0.845
32.9
42.0
66.2
55.9
28.1
51.7
33.2
44.3
SpaceByte
0.41
0.791
43.0
49.0
69.0
63.3
33.5
35.0
49.4
SpaceByte++
Byte
0.42
0.760
55.7
67.9
35.4
39.6
53.6
H-Net (pool)
0,42
0.780
43.2
54.7
69.7
67.9
34.7
54.8
36.4
51.6
H-Net (space)
0.42
0.755
46.7
55.9
72.4
68.8
34.6
57.6
38.0
53.4
H-Net (1-stage)
0.43
0.755
55.5
68.1
35.6
40.0
53.6
H-Net (2-stage)
0.43
0.743
46.9
57.4
72.0
71.7
39.2
60.4
40,6
$5.5
#FLOPs matched to GPT-3 XZ
Trunsformer
Token
0.69
0.730
48.1
58.0
73.1
72.2
37.5
58.6
40.8
55.5
SpaceByte++
0.72
0.733
51.3
60.1
72.4
71.8
38.0
58.5
40.6
56.1
H-Net (space)
Byte
0.726
73.6
72.4
40.2
41.8
H-Net (I-stage)
0.72
0.728
48.4
59.5
72.4
73.0
38.3
59.2
42.4
56.2
H-Net (2-stage)
0.69
0.715
50.5
62.2
73.7
74.2
42.2
60.5
44.0
58.2
Table 3: Robustness evaluation HellaSwag with textual perturbations, all data-/FLOP- matched. Zero-shot accuracy on five different perturbation types (AntSpeak, Drop, RandomCase. Repeat, UpperCase) for models trained exclusively on clean data without noise augmentation, Best and second best results in each column are denoted using bolded and underlined texts, respectively. The Robustness Score metric show that all byte-level models are more robust to adversarial text in- puts than ookenizer-based Transformer. H-Net (2-stage) shows significantly enhanced robustness in textual perturbations, with the highest average accuracy across all noise types and highest robustness score. See Table for architectural details, and Section D.2 for the definition of Robustness Score.
HELLASWAG
ROBUSTNESS SCORE
MODEL
INPUT
AVERAGE 1
ANTSPEAK DROP
RANDOMCASE REPEAT
UPPERCASE
#FLOPs matched to GPT:3 Large
Transformer
Token
31.1
29.9
27.1
27.8
38.9
30.9
20.2
LlamaByte (W1024)
30.4
28.1
29.3
27.2
38.5
30.7
36.9
LlamaByte (Global)
31.1
28.1
29.7
27.3
39.0
31.0
36.6
MambaByte
29.8
27.9
29.9
27.1
39.6
30.9
34.5
SpaceByte
30.7
29.8
33.5
29.5
47.8
34.3
38.1
SpaceByte++
Byte
31.0
30.9
35.8
29.3
54.0
36.2
36.4
H-Nett (pool)
30.5
31.2
35.4
29.6
53.4
36.1
37.3
H-Net (space)
30.8
31.2
38.6
29.4
54.0
36.8
38.2
H-Net (I-stage)
31.2
31,1
35.4
29.9
37.2
HI-Net (2-stage)
30.8
32.1
39.3
30.4
57.1
38.0
39.0
#FL.Ps matched GPT-3 XI.
Transformer
Token
31.6
30.7
28.0
28.5
43.0
32.3
22.2
SpaceBytc++
30.9
32.1
40.3
30.6
58.5
38.5
38.5
H-Net (space)
31.2
33.2
41.9
31.8
60.7
39.8
40.5
Byte
H-Net (1-stage)
30.9
32.7
39.2
31.2
58.4
38.6
39.5
H-Net (2-stage)
31.1
34.7
44.1
33,0
61.7
40.9
42.8
cross over the tokenized Transformer after just 200B bytes, 100B bytes, and 30B bytes respectively. Beyond these points, H-Net's performance advantage widens progressively, demonstrating that the benefits of lcarnable dynamic chunking get strengthened with additional training data, as the model continuously refines its chunking strategy.
Downstream Evaluations. Table presents zero-shot accuracy across diverse downstream bench- marks (Paperno et al., Zellers et al.) 2019 Bisk et al. 2020 Clark et al.) [2018) Sakaguchi et al.) (Gao et 2024) for models at Large and XL scales. SpaceByte++,, H-Net (space), and H-Net (1-stage) all have similar perfor- mance to the BPE Transformer Large scale, and slightly outperform it at the XL scale, consistent with their close training curves (and possibly reflecting some noise in the evaluations). H-Net (2- stage) consistently achieves the highest performance across most tasks, outperforming 2.2% and 2.6% over the Transformer baseline at Large and XI. scales respectively. Notably, the Large H-Net (2-stage) matches the average downstream performance of the XI. BPE Transformer.
2021 Mihaylov et al. 2018) using Ln-evaluation-haar

---

Published as a conference paper at ICLR 2026
Chinese (F-Edu-Chinese-VZ.
Code (Pile Github)
0.90
Validation Bits-per-by byte
0.88
0.86
0.84
0.82
0.80
0.30
0.70
0.36
0.76
0.74
208
558
408
508
698
708
608
023
1098
208
308
609
708
608
Total Training Bytes
Total Training Bytes
H-Net (2-stage)
H-Net (space)
(BPE) Transformer
Figure 3: Validation Bits-per-byte (BPB) throughout training on Chinese language and code modeling. H-Net (space) and H-Net (2-stage) are byte-level, while the Transformers the Llama- 3 tokenizer which was designed for multilingual. H-Net clearly outperforms both Transformer and H-Net (space) Chinese language modeling, which does not have space-like segmentation cues, with lower BPB than H-Net (space) throughout training and crossing over with Transformer after around 25B bytes. On code, both H-Net (2-stage) and H-Net (space) significantly outperform BPE Transformer. Final post-decay results can be found in Table 5
Robustness to Textual Perturbations. Table 3 evaluates model robustness on HellaSwag with various textual perturbations, following protocols from BLT (Pagnoni et al.) (0024). Importantly, these are the same checkpoints trained on clean FineWeb-Edu data used to evaluate Table 2), with- out any form of special data mix or augmentations that may improve character-level robustness. H-Net (2-stage) demonstrates substantially improved robustness compared to all baselines, with performance gaps exceeding those observed in standard benchmarks.
3.2 ALTERNATE LANGUAGE DATASETS
Besides conventional language modeling, we also examine three other language modeling settings Chhnese, code, and DNA. These three settings present distinct challenges for traditional language- modeling pipelines: (i) Chinese characters consist of f-8 encoded bytes each and Chinese language does not have natural spaces; thus, constructing vocabulary picking boundaries quires special consideration; (ii) Code contains much more whitespace than typical language, which allows greater compressibility if handled properly, and it also has latent hierarchical structure that can be leveraged for improved reasoning capabilities; and (iii) DNA does not have any natural to- kenization cues and instead must be processed raw base pairs. In contrast, H-Net operate on raw data without need for handcrafted features (whether vocabulary or delineation cues); therefore provides natural architecture that can operate naturally on any language.
As demonstrated in Figure 3 find that H-Net (2-stage) scales better than BPE Transformer (with the Llama3 tokenizer) and H-Net (space) on both Chinese and code, and achieves lower compression after the decay phase (see Table 5 We additionally measure the performance of each Chinese- language model on the Chinese split of XWinograd, multilingual Winograd Schema Challenge (Muennighoff et al. 2023), where H-Net (2-stage) is significantly better than H-Net (space) which turn is better than Transformer as shown in Table
CONCLUSION
Major advances in deep learning have resulted from powerful architectural innovations en- abling previously-handdrrffed features to be learned from data, from CNNs learning visual fea- tures (Krizhevsky et al. 2012) to Transformers discovering linguistic patterns (Vaswani al. 2017). H-Nets similarly unlock the ability to remove another layer of pre-processing, such as tokenizers, introduce and instead learn them end-to-end. This ability results from set of new techniques that work together to form a dynamic chunking mechanism, which is able to learn content- and context- dependent discrete segmentation strategies through standard gradient- busdinnt-baedd based optimization.
single-stage byte-level H-Net already exceeds the performance of standard tokenized language models, and recursive H-Nets with multiple stages of dynamic chunking further improve its scaling. H-Nets substantially remedy issues with tokenizers, display very strong performance on diverse languages and language-like modalities, and more broadly may serve as the backbone of general foundation models that do more learning with less processing.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R Mortensen, Noah A Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. arXiv preprint arXiv:2305. / 13707, 2023.
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Valentin Hofmann, Tomasz Limisiewicz, Yulia Tsvetkov, and Noah A Smith. MAGNET: Improving the multilingual fairness of language models with adaptive gradient-based tokenization. Advances in Neural Information Processing Systems, 37:47790-4781 14, 2024.
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine trans- lation. arXiv preprint 1rXiv:1111.0273,, 2017.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document Transformer. arXiv preprint wwXiv:2004005/555 2020.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 13Miv:1308.3433 2013.
Aviv Bick, Kevin Li, Eric Xing, J Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratic knowledge to subquadratic models. Advances in Neural Information Processing Sys- tems, 37:31788 31812, 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelli- gence, 2020.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your VIT but faster. arXiv preprini arXiv:2210.09461, 2022.
Garyk Brixi, Matthew G. Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A. Gonzalez, Samuel H. King, David B. Li, Aditi T. Merchant, Mohsen Naghipourfar, Eric Nguyen, Chiara Ricci-Tam, David W. Romero, Gwanggyu Sun, Ali Taghibakshi, Anton Vorontsov, Brandon Yang. Myra Deng. Liv Gorton, Nam Nguyen, Nicholas K. Wang, Etowah Adams, Stephen A. Baccus, Steven Dillmann, Stefano Ermon, Daniel Guo, Rajesh Ilango, Ken Janik, Amy X. Lu, Reshma Mehta, Mohammad R.K. Mofrad, Madelena Y. Ng. Jaspreet Pannu, Christopher Ré, Jonathan C. Schmok, John St. John, Jeremy Sullivan, Kevin Zhu, Greg Zynda, Daniel Balsam, Patrick Collison, Anthony B. Costa, Tina Hernandez-
Eric Ho, Ming- Yu Liu, Thomas McGrath, Kimberly Powell, Dave P. Burke, Hani Goodarzi, Patrick Hsu, and Brian L. Hie. Genome modeling and design across all domains of life with evo 2. bioRxiv preprint biorXiv:2025.00 02. Tt..//...31.. 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. Advances in Neural Information Processing Systems, 30, 2017.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Baantttptsste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. 2023.
Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Niinuma, Kris Kitani, and László Jeni. Don't look twice: Faster video transformers with run-length tokenization. Advances in Neural Informa- tion Processing Systems, 11788822728881 2024.
Jonathan H Clark, Dan Garrette, Iulia Ture, and John Wieting. Canine: Pre-training efficient tokenization-fre encoder for language representation. Transactions of the Association for Com- putational Linguistics, 10:73-91, 2022.
11

---

Published as a conference paper at ICLR 2026
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning chal- lenge. arXiv preprini arXiv: 1803.88330555 2018.
Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. tial redundancy for efficient language processing. Advances in Neural Information Processing Systems, 13:4271-4284 2020.
Filtering out sequen-
Tri Dao. FlashAttention-2:: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICLR), 2024.
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024.
DeepSeck-AL Deepseck-v3 technical report. arXiv preprint arXiv:2412.194377 2024.
Sander Dieleman, Charlie Nash, Jesse Engel, and Karen Simonyan. Variable-rate discrete represen- tation learning. arXiv preprint arXiv: (tti:://33.00000 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Shivam Duggal, Phillip Isola, Antonio Torralba, and William T Freeman. Adaptive length image tokenization via recurrent allocation. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2024.
Eric Egli, Matteo Manica, and Jannis Born. Multiscale byte language models-a hierarchical archi- tecture for causal million-length sequence modeling. arXiv preprint arXiv:2502.1455: 2025.
William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23( (3(120):1-39 2022.
William Fleshman and Benjamin Van Durme. Toucan: Token-aware character level language mod- cling. arXiv preprint arXiv:2311. 1.002200 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. https::: /ttppdoo///eoo.oooo wtwp://d....rreeeede//d/de/
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint 2xXV:2500..01171,200 2025.
Nathan Godey, Roman Castagné, Éric Villemonte De La Clergerie, and Benoit Sagot. Manta: Ef- ficient gradient-based tokenization for end-to-end robust language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2859-2870. 2022.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state- space models. In ICML, 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21773, 2024.
Albert Gu. Modeling Sequences with Structured State Spaces. Phd thesis, Stanford University, 2023.
12

---

Published as a conference paper at ICLR 2026
Albert Gu. On the tradeoff's of state space models and transformers, 2025. / //goombalabbb..
URL https:
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In Conference on Language Modeling (COLM), 2024.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. ICLR, 2022.
Han Guo, Songlin Yang. Tarushii Goel. Eric P. Xing, Tri Dao, and Yoon Kim. Log-linear attention. arXiv preprint arXiv:2506.04700 2025.
Alex Hägele, Elie Bakouch, Atli Kosson, Leandro Von Werra, Martin Jaggi, et al. Scaling laws and compute-optimal training beyond fixed training durations. Advances in Neural Information Processing Systems, 37:76232-76264, 2024.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space. arXiv preprint arXiv:2412.06769, 2024.
Jonathan Hayase. Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A Smith. Data mixture infer- ence attack: BPE tokenizers reveal training data compositions. Advances in Neural Information Processing Systems, 37:8956-8983, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Namgyu Ho, Sangmin Bae, Tachyeon Kim, Hyunjik Jo. Yireun Kim, Tal Schuster, Adam Fisch, James Thorne, and Se- Young Yun. Block transformer: Global-to-local language modeling for fast inference. Advances in Neural Information Processing Systems, 37:48740-48783, 2024.
Jordan Hoffmann, Sehastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404. 06395, 2024.
Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang. Qiyang Min, and Xun Zhou. Over- tokenized Transformer: Vocabulary is generally worth scaling. In The International Conference on Machine Learning (ICMI.,, 2025.
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. arXiv preprint arXiv:2403.08763, 2024.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations (ICLR), 2017.
Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Trans- formers better than state space models copying, arXiv preprint arXiv:2402.01032 2024.
Julie Kallini, Shikhar Murty, Christopher D Manning, Christopher Potts, and Róbert Csordás. Mrt:: Dynamic token merging for efficient byte-level language models. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2rXiv:2001.08361, 2020.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive Transformers with linear attention. In ICMI., 2020.
13

---

Published as a conference paper at ICLR 2026
Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. A clockwork RNN. In International Conference on Machine Learning, pp. 1863-1871. PMLR, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convo- lutional neural networks. Advances in Neural Information Processing Systems, 25, 2012.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint 1rXiv:1888006222 2018.
Garreth Lee, Guilherme Penedo, Leandro von Werra, and Thomas Wolf. From digits to deci-
sions: How tokenization impacts arithmetic in Ilms, 2024. URL co/ ht//spaac/huggg
www.aaaeeeeemeeeeeeeeeeeeeeeeeee zat xation-loo
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Shanchuan Lin, Ceyuan Yang, Hao He, Jianwen Jiang, Yuxi Ren, Xin Xia, Yang Zhao, Xuefeng Xiao, and Lu Jiang. Autoregressive adversarial post-training for real-time interactive video gen- eration. arXiv preprint XXx::2506.00350, 2025.
Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A Smith, and Yejin Choi. Su- perBPE: Space travel for language models. arXiv preprint arXiv:2503.13423, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1111.0501,, 2017.
Chris Lu, Yannick Schroecker, Albert Gu. Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning. Advances in Neural Information Processing Systems, 36:47016-4703 2023.
C Maddison, A Mnih, and Y Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations (ICLR), 2017.
Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scaling rules for adaptive gradient algorithms. Advances in Neural Information Processing Systems, 35: 7697-7711, 2022.
Prasanna Mayilvahanan, Thaddäus Wiedemer, Sayak Mallick, Matthias Bethge, and Wieland Bren- del. Lims on the line: Data determines loss-to-loss scaling laws. In The International Conference on Machine Learning (ICML), 2025.
William Merrill, Shane Arora, Dirk Groeneveld, and Hannaneh Hajishirzi. Critical batch size re- visited: A simple empirical approach to large-batch language model training. arXiv preprint arXiv:2505.23971, 2025.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.0277 2018.
Benjamin Minixhofer, Edoardo Ponti, and Ivan Vulié. Zero-shot tokenizer transfer. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024.
Benjamin Minixhofer, Ivan Vulić, and Edoardo Maria Ponti. Universal cross-tokenizer distillation via approximate likelihood matching. arXiv preprint urXiv:2503.20083. 2025.
Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask fine- tuning. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.), Proceedings of the 61sl Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 15991-16111 11, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10. 18.55653 awt.//22....ooo..8999 URL. https: / //acl ant I loyy. org/2023.
891/
14

---

Published as a conference paper at ICLR 2026
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. S1: Simple test-time scaling, arXiv preprint arXiv:2501. (0t.:2501./9993,, 2025.
Piotr Nawrot, Szymon Tworkowski, Michat Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. In Findings of the Association for Computational Linguisties: NAACL 2022, pp. 1559-1571, 2022.
Piotr Nawrot, Jan Chorowski, Adrian Lancucki, and Edoardo Maria Ponti. Efficient transformers
with dynamic token pooling. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6403-6417, 2023.
OpenAl. Introducing openai ol-preview, 2024. URL https://openai.coom com/
rodruccn openail ol previeww
Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlmooee, et al. Byte latent Transformer: Patches scale better than tokens. arXiv preprint arXiv:24/2.09871, 2024.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bemardi,, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv: 06003,, 2016.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The Wen datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:3081 1100811-350 1-30849, 2024
Luca Perić. The bitter lesson is coming for tokenization, 2025. URL https: dev/
bitter- tokeni batteniizatio zat ion and blt
Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages. Advances Neural Information Processing Systems, 36:36963-36998 2023.
Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, and Karen Ullrich. Exact byte-level probabilities from tokenized language models for FIM-tasks and model ensembles. arXiv preprint arXiv:24 2024.
Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, and Karen Ullrich. Ex- act byte-level probabilities from tokenized language models for fim-tasks and model ensembles. arXiv preprint arXiv:2410.09300 2025.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In ICMI., 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Nived Rajaraman, Jiantao Jiao, and Kannan Ramchandran. An analysis of tokenization: Transform- ers under markov data. Advances in Neural Information Processing Systems, 37:62503-62555 2024.
David Raposo. Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mfxture-of-depths: Dynamically allocating compute in Transformer-bate language models. arXiv preprint wrXiv:2404.02258, 2024.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18, pp. 234-241. Springer, 2015.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An ad- versarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
15

---

Published as a conference paper at ICLR 2026
Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range DNA sequence modeling. The International Conference on Machine Learning (ICML), 2024.
Craig Schmidt, Varshini Reddy, Haoran Zhang. Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. Tokenization is more than compression. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 678-702, 2024.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.077 07909, 2015.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint 1exKiv:7001.000 006538, 2017.
Kevin Slagle. Spacebyte: Towards deleting tokenization from large language modeling, In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Makesh Narsimhan Sreedhar, Xiangpeng Wan, Yu Cheng, and Junjie Hu. Local byte fusion for neural machine translation. In Proceedings of the 6l Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1199-7214, 2023.
Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. Adv-BERT: BERT is not robust on misspellings! generating nature adversarial samples on BERT. arXiv preprint 2xXi:20033004988, 2020.
Richard Sutton. The bitter lesson, 2019. URL http://www http://wwwicoomppeeeeeee..a..aee
Selldeas/Bittereesee
html
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. In The eighth Annual Conference Neural Information Processing Systems, 2024.
Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Si- mon Baumgartner, Cong Yu, and Donald Metzler. CharFormer: Fast character Transformers via gradient- based subword tokenization. arXiv preprint arXiv:2106. 2262, 2021.
Avijit Thawani, Saurabh Ghanckar, Xiaoyuan Zhu, and Jay Pujara. Learn your tokens: Word- pooled tokenization for language modeling. In Findings of the Association for Computational Lingnistics: EMNLP 2023, pp. 9883-9893, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 17X1; 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 09288, 2023b.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Neural Information Processing Systems, 2017.
Mathurin Videau, Badr Youbi Idrissi, Alessandro Leite, Mare Schoenauer, Olivier Teytaud, and David Lopez-Paz. From bytes to ideas: Language modeling with autoregressive u-nets. arXiv preprint arXiv:2506. 147661,,, 2025.
Tim Vieira, Ben LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Brian DuSell, John Terilla, Timo- thy J O' Donnell, and Ryan Cotterell. From language models over tokens to language models over characters. In The International Conference on Machine Learning (ICML), 2024.
16

---

Published as a conference paper at ICLR 2026
Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of Mamba- based language models. URL htps://arxiv org/abs/2406.0m 2778//0872000007877. 2024.
Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. Mambabyte: Token-free selective state space model. First Conference on Language Modeling (COLM), 2024,
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 1088.
Linting Xue, Aditya Barua, Noah Constant, Rami AI-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards token-free future with pre-trained byte-to-byte mod- els. Transactions of the Association for Computational Linguistics, 10:291-306, 2022.
Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011. 14522, 2020.
Songlin Yang. Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated Linear Atten- tion Transformers with hardware-efficient training. In The International Conference on Machine Learning (ICML), 2024a.
Songlin Yang, Bailin Wang. Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transform- ers with the delta rule over sequence length. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024b.
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In The Thirteenth International Conference Learning Representations (ICLR), 2025.
Morris Yau, Sharut Gupta, Valerie Engelmayer, Kazuki Irie, Stefanie Jegelka, and Jacob Andreas. Sequential-parallet duality in prefix scannable models. arXiv preprint arXiv:2506. 109188, 2025.
Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. MegaByte: Predicting million-byte sequences with multiscale Transformers. Advances in Neural Information Processing Systems, 36:78808-78888 -88223, 2023.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37:128940-12886, 2024.
Yijiong Yu, Ziyun Dai, Zekun Wang, Wei Wang, Ran Chen, and Ji Pei. Opencsg chinese corpus A series of high-quality chinese datasets for II training, 2025. (Rt. abs/2500 08197
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019.
17

---

Published as a conference paper at ICLR 2026
APPENDIX
Discussion
19
B
Related Work
22
B.1
Autoregressive fokenizer-free Architectures
22
B.1.1
Isotropic Architectures
22
B.1.2 Static Chunking
22
B.1.3
External Chunking
23
B.1.4
Dynamic Chunking
24
B.2
Nou-Autoregressive Tokenizer- free Architectures
24
Other Tokemization-rclated Work
25
Model Details
27
C.1
Design Principles
27
C.2
Dynamic Chunking (DC)
27
................................
C.2.1
Smoothing Module
................................
27
C.2.2
Upsampler
28
C,3
Improved Techniques for Hierarchical Sequence Modeling
................................
Autoregressive Training and Inference
30
Additional Experimental Details
31
D.I FLOPs Computation
31
D.2 Robustness Score
31
D.3 Experimental setup for Chinese and code.
31
Additional Experiments
33
E.1
DNA (Human Genome) Experiments
33
Distilling Leven-Level Models Bete-Level
................................
Ablation Studies).
................................
35
E.3.1
Importance of Components in H-Net
35
E.3.2
Encoder & Decoder Layer Selection
36
E33.
Visualization of Tokenized Positions
37
E.3.4
Hybrid Architectures for the Main Network
38
E.3.5
Comparison to Mixture-of-Experre
39
E.3.6
Different Downsampling Methods in the Chunking Layer
39
E 3.7
DNA Architecture Ablations
40
LLM Usage for Paper Writing
41
Acknowledgements
18

---

Published as a conference paper at ICLR 2026
A DISCUSSION
Distillation. For new architectures, showing that they can be distilled from standard pretrained Transformers can result in stronger new models with substantially reduced training (Bick et al.) 2024). In Section F.2 we investigate this for H-Net by initializing the main network from pre- trained Llama checkpoint and learning the encoder and decoder networks. With less than 200B bytes of training, the resulting model shows strong performance much better than if were trained from scratch, although still worse than the teacher model. Our distillation procedure is perhaps currently the most efficient way of creating an end-to-end byte-level model, but we expect that it can be further improved.
Efficiency. Because of the dynamic nature of our model, requires different considerations in making both the training pass and inference step efficient. Our implementation incorporates sev- eral engineering techniques already, such handling variable sequence lengths within a mini-batch using specialized kernels provided by Dao 2024)) Dao & Gu (2024). Because of the different architectural considerations, is difficult compare more standard pipelines; our current imple- mentation may be approximately up to 2x slower than an isotropic model during training.
Note that the memory usage of our model is also dynamic, unlike standard sequence models. so other edge cases may happen, such as unlucky batches of sequences that are too long and overflow the device memory. Relatedly, one difficulty with stepping H-Net in batched mode is that different tokens in the batch may require different amounts of compute.
We believe that such considerations are not fundamental and will be an important subject of future work; just as how related dynamic sparsity and conditional compute methods such as Mixture-of- Experts and speculative decoding (Leviathan et al. 2023 Chen al. 2023) benefited from years of dedicated engineering improvements.
Deeper Hierarchies. H-Net is the first dynamic hierarchical model that can recursively nest its chunking strategy (see Table 4 and Section [B. In this paper, we showed that iterating H-Net from O stages (i.e. an isotropic model) to 1 stage and from stage to 2 stages consistently improves performance. We did not attempt a 3-stage H-Net at all for simplicity. Testing H-Net can be iterated even deeper remains immediate direction to explore.
Global Sequence Model Considerations. Much research on sequence model architectures has focused individual layers, where the tradeoff's are often quite direct. For example, recurrent models such as state space models (Gul 2023] Gu & Dao 2024) and linear attention variants (Katharopoulos et al., 2020 Yang et al., 2024a b 2025) compress arbitrarily long sequences into fixed-size hidden states, offering higher efficiency at the expense of precise retrieval of information (e.g. struggling with recall (Jelassi et al., 2024)).
H-Net. however, is global architectural design that is simultancously orthogonal to, but may have interactive effects with, the choice of individual layers. For example, using deeper hierarchies with exclusively recurrent layers would preserve linear computation (in sequence length) but logarithmic state size, resembling newer sequence model layers such as log-linear attention (Guo et al., 2025) and Prefix Scannable Models (Yau et al., 2025). but with dynamic hierarchies. Similarly, the recur- sive compression of sequence length may alleviate their limitations in retrieval on long sequences. This may be considered a form of dynamic state allocation. This paper has not focused on such implications, which would be a possible direction for future research.
Long Context. Similarly, an effect of the global hierarchical structure may be improved long context abilities, which is common motivation for hierarchical models (Koutnik et a [014]
Chang 2017) 7b Much research on sequence models again focuses on long context at the layer level (Poli al.
et al.
2023 Gu & Dao 2024: Vaswani et al., (017), and we hypothesize that H-Nets may provide general long context improvements in an orthogonal direction.
Latent Test-Time Compute. Test-time compute techniques, exemplified by Shain-of-Thoughee (Wei et (022). have been shown to improve model performance on a variety of reasoming bench- marks (Muennighoff al. 2025 OpenAl 2024). Recent work has explored including latent rep-
resentations (as opposed to just tokens) in the reasoning process (Hao et al. 2024), culminating in "recurrent depth" models that roll out RNN for many steps as needed before emitting token (Geiping et al.] (2025). As discussed in Section C.4] H-Net is also capable of dynamically chang- ing compute per output generated; thus, it can be viewed as model that can dynamically allocate
19

---

Published as a conference paper at ICLR 2026
latent test-time compute well. Additionally, the motivation of H-Net is to recursively build higher-order abstractions, we hypothesize that it would be more effective as a reasoning model that operates over its own learned concepts instead of arbitrary token inputs.
Sparsity. H-Net can be viewed as form of dynamic sparsity or conditional computation, and is related to concepts such mixture-of-experts (MoE) (Fedus et al., 2022; Shazeer et al., (017) and -fxture-of-depths (Raposo et al., 2024). We showed that at the byte level, DC much more effective than MoE when controlled for parameters and compute (Figure [4), and leave fleshing out further connections and comparisons for future work. We also note that H-Net can be viewed as orthogonal to MoE, which can be applied to sparsify any MLP layers within an H-Net.
Scale. The largest models in this paper were FLOP-matched to the equivalent of a 1.3B parameter Transformer. While we believe that this provides sufficient evidence for the effectiveness of this approach, it remains validate H-Net at larger model sizes of 3B, 7B, and beyond. We note that while we observed no instabilities at our model sizes, the added complexity of H-Net and inherent difficulties of learning end-to-end discrete selection problems may require more serious investigation of potential stability challenges at larger scale.
Scaling Laws. Formally estimating the scaling behavior of model requires calculating scaling law coefficients that sweep across a large range of model sizes and compute horizons (Kaplan et al,, 2020 Hoffmann et al. 2022). We did not pursue this formal approach in this paper due to resource constraints.
Instead, we used a simpler heuristic for the scaling behavior of our models, at least with respect to data. We note that
essentially all modern models live in the "overtrained" regime (with respect to the formal
scaling laws) due to inference considerations at deployment (Touvron et al. (023b); and
these overtrained models often use modern schedulers that have extended periods of con- stant learning rates (Hu et al.) 2024 DeepSeek-Al 2024).
Thus, we decided to use the models' losses during the constant phase as a proxy for how quickly they improve with data. We believe this still provides useful insight into scaling behaviors, and more dedicated analysis of formal scaling laws remains an important topic for future work.
BPB Calculation. For baseline BPE tokenized models throughout this work, we used the standard bits-pec bits-per-byte (BPB) calculation of simply rescaling the negative log- likelihood (or log perplexity) by the average number of bytes per token (Gao et al. 2020 Wang et al., 2024 Slagle 2024). However, this is not strietly speaking correct BPB estimate for tokenized models, as assumes that the probability the model outputs a string is equal to the probability of the model outputting the greedy tokenization of the string.
Depending on how the model is trained, it is possible the model can output other tokenization quences with nonzero probability. There are an exponential number of these, so computing the exact BPB is intractable; however, concurrent work (Vieira et al. 2024] shows that the standard BPB cal- culation indeed overestimates BPB. Due to the high computational overhead of estimating the true BPB, we only provide the standard (inexact) value; nevertheless, H-Net's superior performance downstreams provides supporting evidence that it scales better than BPE models.
20

---

Published as a conference paper at ICLR 2026
Table 4: Related architectures. Comparison of related architectures, particularly those focused on bete-level modeling. H-Net is the first architecture that enables dynamic, multi-stage hierarchies. Extended discussion is provided in Section
MULTI-STAGE HIBRARCHY
CLASS
AUTOREGRESSIVE CHUNKING MECHANISM
EXAMPLE ARCHITECTURES
ByT5
Isotropcc
MamhaByte
Punnel-Tranffrmmn
w-width pooling
Canine
Charformer
Hourglass Transformer
Hierarchical (static)
SaShiMi
MegaByte
k-widtt pooling
Block Transformer
MBLM
AU-Net
eByte
X
delimiters
WSF
DPT (Whitespaces)
Hicrarchical (extemna))
delimiters
SpaceByte
AU-Net
DPT (Entropy)
entropy
BLT
soft matching
X
MANT:
Hierurchical (dynamic)
soft gating
X
MrT5
stochastic reparameterization
DPT (Gumbel)
Hierarchical (dynamic) /
dynamic chunking
/
H-Net
21

---

Published as a conference paper at ICLR 2026
B RELATED WORK
The fundamental challenge of transforming raw sequential data into computationally efficiint rep- resentations manifests across multiple domains through implicit chunking processes. In language modeling, this challenge addressed through tokenization using static vocabularies derived from frequency-based algorithms such as Byte-Pair Encoding (BPE) (Sennrich et al. 2015) in GPT mod- els (Radford et al., 2019 Brown et al. 2020) and SentencePiece (Kudo & Richardson 2018) in Llama architectures [Tuurron et al. 2023b Grattafiori et al. 2024). Computer vision addresses sim- ilar challenges through spatial pooling operations (Ronneberger al. 2013) that aggregate neigh-
boring pixels into meaningful representations.
Despite achieving strong empirical performance, it is widely known that traditional tokenization ap- proaches in language models suffer from fundamental limitations that constrain model capabilities. Fixed vocabularies exhibit biases toward high-resource languages, demonstrate fragility when han- dling adversarial inputs, and show lower performance character-level tasks (Petrov et al. 2023 Ahia et al. 2023; Belinkov & Bisk 2017; Sun et al. 2020) Xue et al. 2022). These limitations stem from the static nature of predefined vocabularies, which cannot adapt their chunking strategies to input content or context.
To address these constraints, tokenizer-free methods have emerged that avoid the reliance on prede- fined vocabularies.
In Section B.1
we discuss the most directly related prior work on autoregressive sequence models, extending the overview from Section
In Section B.2
we discuss non-autoregressive models. We note that essentially all autore- be turned into non-autoregressive architectures (including proposed H-Net), and vice versa, which provide possible extensions of H-Net in future work. However, we provide this delineation because it marks an important difference in motivation that influences design considerations and downstream evaluations.
gressive architectures
Section B.3 mentions other works in non-language modalities related to tokenization.
We summarize discussion tokenizer-free architectures in Table
B.1 AUTOREGRESSIVE TOKENIZER FFRE ARCHITECTURES
As outlined in Section prior work on autoregressive tokenizers for architectures can be divided into four categories:
1. Non-hierarchicr isotropic architectures.
2. Hierarchical architectures with static chunking strategies, where chunk boundaries are content-agnostic (usually some variant of fixed-width pooling).
3. Hierarchical architectures with external chunking strategeies, where chunk boundaries provided by an external function or module.
4. Hierarchical architectures with dynamic chunking strategies, where chunk boundaries are content-dependent and learned end-to-end.
B. ISOTROPIC ARCHITECTURES
The most direct approach modeling language with tokenizers is simply model raw byte quences with a standard sequence model architecture. Since this naive approach suffers from com- putational challenges
long sequences, MambaByte (Wang al. 2024) proposed using state space model for its linear-time efficiency. We similarly use Mamba(-2) 2-) (Dao & Gu 2024) layers in the outer stages of H-Net. Notably, through extensive ablations show that Mamba is not just more efficient but also better at modeling high-resolution data such as text characters and DNA base pairs
1.1.2 STATIC CHUNKING
To reduce sequence length, several approaches downsample the input sequence hierarchically. The most straightforward methods operate independently of input context, partitioning sequences using fixed-size intervals. Many strategies could be used to aggregate width-k window, including direct
22

---

Published as a conference paper at ICLR 2026
downsampling, average pooling, linear transformations that mix across the chunk, convolutions, and more; we lump these together as pooling operations.
Hourglass Transformer (Nawrot et al., 2022) and MegaByte Yu et al. 2023) exemplify this strategy. Other recent variants include the Block Transformer (Ho et al. 2024) and Multiscale Byte Language
Model (MBLM) (Egli et al. 2025). which use similar multi-staee static chunking architectures. Concurrently to H-Net, the MBLM also proposes using Mamba layers in the outer stages.
These approaches share conceptual similarity with spatial pooling operations in vision models that reduce resolution through
aggregation (Krizhevsky et al.) 2012; He et al., 2016)) While these content-agnostic methods have simple and efficient implementations, they face in- herent limitation: they do not reflect natural semantic boundaries in the data. Fixed-size chunking inevitably creates arbitrary separations that can split meaningful units such words, morphemes, or phrases, thereby limiting model expressivity.
This class of models may also be called "autoregressive U-Nets", characterized by the U-Net multi- scale architecture (Ronneberger et al.) 2015) with additional considerations to maintain causality. Prior to these, the S4 and SaShiMi models (Gu et al. 2022, Goel et al.) (2022) used the same archi- tecture successfully in the vision and audio modalities, where fixed-window downsampling exhibits more appropriate inductive bias in contrast to language. SaShiMi specifically operated over 8-bit quantized audio inputs, hence also was a form of byte-level modeling that used BPB as a metric.
B.1.3 EXTERNAL CHUNKING
An improvement hierarchical architectures with static downsampling is to use content-aware chunking strategies that attempt to identify natural token boundaries based on semantic or statis- tical properties of the input data. Several recent models propose using the boundaries provided by an external module, with two main variations appearing.
Delimiter-based methods. The most intuitive content- -wware approach segments on surface-level syntactical boundaries, which can be often implemented by simple rules or regular expressions.
Dynamic Pooling Transformer (DPT) (Nawrot al. 2023) proposed a variant that segmented on whitespace characters, effectively making each word its own token. SpaceByte (Slagle, 2024) ex- tends this to "space-like" delimiters (e.g., 1. :) natural boundary signals. This approach pro- vides semantically meaningful chunking for languages with explicit word separators such as English text and code.
However, delimiter- based methods cannot be used for inputs lacking explicit separators (e.g. many non- non-European languages, other modalities such as DNA). Additionally, these approaches cannot be extended to multi-level hierarchical chunking due ambiguities in defining natural delimiters higher semantic levels. AU-Net (Videau et al. (025) is a concurrent work that augments SpaceByte with additional stages of hierarchy using fixed-width chunking. Specifically, AU-Net 2 is SpaceByte with minor architectural modifications, while AU-Net 3 (and AU-Net 4) add additional levels of hierarchical with width-2 downsampling.
In this work, show that SpaceByte's 's delimiter chunking strategy can be very powerful baseline on appropriate languages competitive with or outperforming traditional tokenizers on English and code when augmented with several of H-Net's additional techniques (Section 3.1 Section Figure Figure 9
Entropy-based methods. Another approach to circumvent the delimiter dependency is using the autoregressive conditional entropy as heuristic to identify semantic boundaries. This was first proposed by the Dynamic Pooling Transformer (DPT) (Nawrot et al. 2023), which detects entropy spikes that correlate with semantic transitions. The recent Byte Latent Transformer (BLT) (Pagnoni et al. 2024) employs entropy thresholds computed by a separate pre-trained model to determine chunking boundaries.
Despite showing promise, these matrepy-based approaches face several practical limitations. First, they require extensive domain-specific hyperparameter tuning to establish appropriate entropy thresholds, reducing their general applicability. Second, they still fall behind in performance; for example, BLT necessitates an extra 3B parameters (at the 8B scale) solely for multi-gram hash embeddings to match BPE Transformer baselines. Finally, these methods also cannot be extended
entropy--
23

---

Published as a conference paper at ICLR 2026
hierarchically because computing cross-entropy loss requires access to target vocabularies, which are unavailable for intermediate latent representations multi-stage architectures.
In this work, we do not compare against BLT because of its complexity: (i) necessitating training an auxiliary language model to provide proxy autoregressive conditional entropies (ii) converting it into an external neural tokenizer through tuning entropy heuristics (ii) using hash embeddings, which can be considered an orthogonal architectural component which may be incorporated into H-Net as well if desired.
Instead, we compared against SpaceByte (and our own stronger versions of SpaceByte), which believe to be representative of the external-chunking family of methods and competitive to the entropy-based chunking strategy of BLT (for our main experiments such as English data).
B.1.4 DYNAMIC CHUNKING
The ideal tokenizer-free architecture would incorporate a dynamic chunking method that attempts to learn optimal segmentation strategies directly from data through gradient- based optimization. Such a method would be optimized jointly together with the outer (fine-resolution) and inner (coarse- resolution) networks, and be able create boundaries that are content- and context- aware.
The only prior work we are aware of that attempted a true dynamic chunking method is (one variant of) the Dynamic Pooling Transformer (DPT) (Nawrot et al.) 2023). which incorporates stochastic exploration mechanisms using Gumbel noise (Jang ferentiable boundary selection during training. Despite their theoretical flexibility, trainable methods encounter critical challenges. The stochastic exploration process requires careful tuning of noise magnitudes and introduces high-variance gradients that destabilize training, making it difficult to scale to larger model sizes.
[2017] Maddison et al. 2017) enable dif-
In practice, the end-to-end (stochastic reparameterization) variant of DPT underperformed the exter- nal chunking variants (drawing boundaries on entropy spikes or whitespaces) (Nawrot et al.) (2023). illustrating the difficulty of this problem. Furthermore, the training instability prevented DPT from expanding to multiple hierarchical stages, constraining these methods to single-stage chunking.
We additionally highlight simple architectural modifications of DPT motivated by improved infer-
ence (Fleshman & Van Durme 2023) multilingual ability (Ahia et al. 2024). Such techniques can also be easily adapted to H-Nets in future work.
B.2 NON-AUTOREGRESSIVE TOKENIZER-FRES ARCHITECTURES
Each class of autoregressive architectures from Section B.I has corresponding con-autoregressive variants as well. Although these often have similar design principles, they are also motivated by different tasks, settings, and design considerations (e.g. no evaluation on large-scale autoregressive pretraining) and thus can be difficult to compare directly to autoregressive models. We include these for context and completeness.
Isotropic. ByT5 (Xue et al. 2022 directly models bytes using a bidirectional encoder-decoder architecture, showing improved performance with small models (because more power is moved into model parameters rather than vocabulary embeddings) and spelling-sensitive tasks.
Hierarchical (Static). Transl-Transformer (Dai al. 2020) is an carly architecture that uses a U-Net-like architecture for language, focusing on the non-causal setting. Canine (Clark et al.) 2022) proposes hierarchical model with convolution-banl static downsampling: their method also targets non-autoregressive language models.
Charformer (Tay et al. 202T) presents gradient-based subword tokenization (GBST) method that pools the input sequence different resolutions, inducing implicit ensemble of hierarchical mod- els. It shows improved efficiency to performance trade-offs compared to models that use a single downsample resolution.
We note that these methods can also be endowed with implicit supervision from external tokeniz- ers; for example, Canine proposes a variant that uses subword tokens in the objective function (via masking out subwords in the masked language modeling objective). but does not need the tokenizer at inference time. We also note that such techniques are particular to non-autoregressive models, since they allow for variations in the modeling objective.
24

---

Published as a conference paper at ICLR 2026
Hierarchical (External). Thawani et al. (2023) propose the eByte method, which resembles MegaByte but chunks on spaces with Transtormer-based CLS-token pooling, and lacks the byte- level residual stream that enables autoregressive modeling. Word-based self-attention fusion (WSF) (Sreedhar al. 2023) proposes similar pooling strategy for encoder language models.
Hierarchical (Dynamic). MANTa (Godey et al. 2022) introduces an end-to-end method that pre- dicts segmentation boundaries and pools bytes into blocks using a matching objective. MrT5 (Kallini al. 2025) a recent method improving on ByT5 with a gating mechanism that allows for explicit dynamic loken-merging at inference time, reducing sequence lengths by up to 80%.
B.3 OTHER TOKENIZATION-RELAA RELATED WORK
Tokenizers for Other Modalities. While computer vision pipelines do not use tokenizers like BPE in the same way as language models do, they frequently need to turn raw perceptual data (im- ages and videos) into shorter sequences of representations. One approach is the simple patchification step first introduced by the Vision Transformer (ViT) (Dosovitskiy et al.) 2021]. However, images, videos, and audio can have varying amounts of semantic content and non uniform redundancies. A number of more recent approaches attempt produce variable length tokenizations that adapt to the information content of the data, Which performs a more similar role to tokenization in language models. This can be done in the latent space of autoencoder (Yu et al.,
2024 Duggal et al. 2024) or through explicit token merging (or "run length encoding") ") with heuristics (Bolya et all
2022 Choudhury al., 2024). In the audio domain, SlowAE (Dieleman et al., 2021) proposes
joint autoencoder with autoregressive modeling that finds semantic segmentation boundaries, which resembles H-Net's approach at high level.
FAST (Lin et al. 2025) introduces tokenizer for robotics, Which tokenizes continuous control actions by combining the Discrete Cosine Transform (DCT) with BPE.
Vocabulary Scaling. While scaling laws for language models have generally kept tokenizers fixed (Kaplan et al. 2020 Hoffmann al. 2022 Grattafiori et al.) 2022), recent works have showed that the tokenizer also warps scaling laws, in fact more so than model architecture changes (Mayil- vahanan et al., 2025). Tao et al. (2024) and Huang et al. (2025) directly show that it is more optimal to scale an LLM's vocabulary together with the rest of the model parameters.
In H-Nets, which are designed to operate over higher resolution raw data, the actual vocabulary can be kept minimal, but the chunking mechanism can be viewed vocabulary. As H-Nets scale in size, one expects that more iterations of hierarchy can be added (increasing effective chunk size), or the chunk size can directly be increased to leverage parameters more efficiently. This resembles the idea of increasing a vocabulary in tokenized models (which would generally increase the average length of tokens).
implicit "tokenizer" with infinite
SuperBPE (Liu et al. 2025) shows that allowing vocabulary tokens to cross whitespace boundaries can also improve performance. This is related to H-Net's motivation of higher-level chunking of words into phrases; empirically, Figure shows how the 2-stage H-Net finds semantic multi-word groups in the inner stage.
Minixhofer et al. (0244) and Minixhofer et al. 2025) address the problem of tokenizer transfer, or adapting models across different tokenizers (for example for cross- language or cross cross-modalipy modality usage, or for knowledge distillation).
Tokss-Tokenizer Transfer.
Other Effects of Tokenization. Lee al. (2024) discuss the effects that tokenization has on arith- metic in LLMs. For example, comparing the performance of left-to-right vs. right-to-left tokeniza- tion. Hayase al. (2024) show that examining the vocabulary of a BPE tokenizer leaks information about the data mix that it was trained on.
Tokenization Theory. Schmidt et al. (2024) examined the hypothesis that the primary role of tokenization is to shrink the input sequence length. They invented a new tokenizer that has even higher compression rates than BPE (actually, they keep the same vocabulary but simply find different segmentations that more compressed) yet leads to worse language models, providing evidence against the hypothesis.
Raaaraman et al. (2024) showed that for certain data distributions, applying tokenization qualitatively changes what Transformers can learn.
25

---

Published as a conference paper at ICLR 2026
Phan et al. (2024) and Vieira al. (2024) propose various algorithms for converting language model over tokens into language model over characters or bytes. This helps alleviate some limita- tions of tokenizers such as the "prompt boundary" problem, the ability to compare different LLMs with different tokenizers, and simply produces better estimates of language model's true compres- sive ability (as measured by bits-per-byte). However, such algorithms are complex and expensive, and compared to direct byte-level models they are not practical for use during inference decoding (repeated autoregressive sampling).
26

---

Published as a conference paper at ICLR 2026
C MODEL DETAILS
C.I DESIGN PRINCIPLES
Encoder and Decoder Networks. The encoder and decoder networks in H-Net face unique design constraints due their dual objectives and computational requirements. Each encoder must simulta- neously (i) preserve fine-grained information for transmission to its corresponding decoder through residual connections equation 3 and (ii) compress inputs into chunks of richer representations for the main network. The decoder, in turn, must effectively combine coarse- -oirseee representations from the main network with fin-ggrained details from the encoder residuals.
Importantly, both encoders and decoders operate on uncompressed sequences, making computa- tional efficiency a significant design constraint that shapes our architectural choices. Recent studies
demonstrate that state space models (SSMs) (Gu al. 2022
Gu & Dao 2024) excel process-
ing fine-grained data including audio (Goel et al. 2022) robotic control signals (Lu et al. 2023).
DNA sequences (Schiff et al. 2024). and
Based on these insights, employ Mamba-2 layers (Dao & Gu 2024) as the primary building blocks for the encoder and decoder networks. This choice yields two significant benefits: effective handling of fine-grained inputs, and substantially improved efficiency when processing long, uncom- pressed sequences. Our ablation studies (Section EE3) confirm that SSM-based encoders/decoders significantly outperform Transformer layers, not just at the byte level but even coarser inputs, which we attribute to their stronger inductive bias for compression which helps build abstrac- tions Gu 2025).
Main Network. H-Net's computational efficiency stems from strategic parameter allocation. We concentrate the majority of model capacity in the main network, which operates progressively compressed sequences. After S stages of compression, the main network receives sequences where L3 < L", enabling much larger networks within the same computational budget. This design re- flects two key principles: (i) compressed sequences allow more parameters and compute per chunk, and (ii) higher-level abstractions benefit from increased processing power.
The main network functions as a standard language model and can employ any sequence mixing architecture. We default Transformer layers for two reasons: compressed representations align well with Transformers' strengths in processing discrete, semantically-rich tokens, and this choice enables more controlled comparison with traditional BPE-based Transformer baselines in our ex- periments. However, the modular design also allows straightforward substitution with alternative architectures (e.g., state space model, hybrid, H-Net itself) explored in our ablations.
Architectural Guidelines. Compared to standard isotropic models, the H- H-Net's structure intro- duces several new dimensions of architectural parameters balance the parameter/compute alloca- tion each network. To simplify the search space, we follow a few general guidelines.
First, we ensure the model width (often referred to as dmodel for isotropic architectures) is monotone in the hierarchy: D < D < ... < DS, This allows increasing compute and parameters used in the main network without significantly increasing its depth.
Second, using efficient and powerful SSM layers in the outer networks allow reducing the number of layers used compared to similar prior architectures that only used Transformer layers (Slagle 2024): : in this paper, we always stick to four layers (or the equivalent of four Mamba layers) in each encoder/decoder network.
To handle the changes in dimensions without an additional linear layer, we adopt the technique used in SpaceByte (Slagle, (2024) with the marginal change: expand dimensions (i.e., D* D we append all vectors with a shared trainable vector of dimension D - D*; to reduce dimensions (i.e., D - D))), we take the first 1) dimensions from each vector.
We note that H-Net's performance can likely be improved with more careful tuning of the layer allocation and hyperparameters between sub-networks.
C.2 DYNAMIC CHUNKING (DC)
C.2.1 SMOOTHING MODULE
The smoothing module is defined with an EMA operation (see equation 5). which performs several roles:
27

---

Published as a conference paper at ICLR 2026
new producti ne W product
e W
p product!
2n0
2
2
2,
2,
2
Z,
2yme
2yo 2uptarra
2
Upper Stages
Upper Stages
Upper Stages
X
X,ro
X
X
X
X,,
X
X
X
X
X
X
n products
mne r products
n P
Productl
(a) Orace Boundaries
(b) w/o Smoothing Module
(c) w/ Smoothing Module
Comparison of decompression strategies on the example sequence " new product!" !" indicates a boundary with high confidence (Pa = 1.0) and O indicates bound- ary with low confidence (Pt 0.5). As each letter in the example is unique, we use the letters in subscripts denote expected semanties of chunks. (a) Optimal chunking with oracle boundaries identifying linguistically meaningful units. (b) Suboptimal chunking without a smoothing module. This creates misalignment during upsampling, causing information from incorrect contexts prop- agate. (c) Improved decompression with a smoothing module, where low-confidence chunks are interpolated with weighted combinations of previous chunks, correcting the shaded regions. In pan- els (b) and (c), we interpret low-confidence boundaries cause the encoder network to embed broader contexts subsequent positions. Specifically, the vectors and encode new and duct!. !, respectively (instead of w. and ct.).
Figure 4:
Differentiable boundary learning: transforms the discrete upsampling operation into a continuous one, enabling effective backpropagation through chunk boundaries during training without requiring stochastic exploration-bast based approaches (Jang al. (017).
Adaptive error correction: Chunks with high confidence (Pt RO 1.0) maintain discrete boundaries (ž, 2t). while chunks with low confidence (P, z 0.5) are smoothed using information from previous chunks, creating self-correcting mechanism.
Training stability: By smoothly interpolating between discrete choices based on con- fidence scores, smoothing module prevents the model from overfitting to suboptimal chunking patterns early in training.
Figure 4 illustrates this with the example .... new product!" !". The word "product" can be morphologically decomposed into "pro-" and "-duct" Without the smoothing module (see Fig- ure (b)), suboptimal chunking (e.g.. "du" "du" as shown with half-filled circles) creates alignment mismatches that disrupt information flow. With the smoothing module (see Figure (c)), chunks with low confidence smoothed with previous context, ensuring proper information propagation and enabling the model to learn optimal chunk boundaries through gradient descent.
C.2.2 UPSAMPLER
Equations 6 to 99) are designed carefully with these following objectives:
Confidence scoring equation 6 The coefficient e quantifies the routing module's confi-
dence in its boundary decisions. For positions marked boundaries (b; 1), c: Pt rewards high boundary probabilities. In contrast, for non-boundary positions (b. = 0),
= 1 P: penalizes false boundary predictions. This formulation encourages the model to produce boundary probabilities near 1.0 at true boundaries and near 0.0 elsewhere
Gradient stabilization equation 7 The Ehraight-Througe Estimator (STE) (Bengio al. 2013) a well established technique from discrete representation learning (Van Den Oord et al. 2017: jang et al. (017)) that rounds confidence scores to 1.0 in the forward pass while maintaining continuous gradients during backpropagation. While H-Net already demonstrates strong performance without STE, incorporating this technique provides an additional performance boost that empirically further stabilizes the optimization dynamics.
Causal expansion equation 8 The upsampling operation repeats each compressed vector until the next boundary position, ensuring that each reconstructed position receives infor- mation from its most recent chunk. This maintains the sequential flow of information while expanding the compressed representation back to its original length.
Confidence- decompression equation 9 Multiplying upsampled vectors by their confidence scores incentivizes the routing module to make confident, accurate decisions.
pro- mecainngforwar or forth, -uuct from Latin ducere, meaning lead bring
28

---

Published as a conference paper at ICLR 2026
High-con Sigh-coonideece boundaries create direct reward signals that encourage the model to sharpen its boundary predictions through gradient feedback.
C.3 IMPROVED TECHINIQUS FOR HIERARCHICAL SEQUENCE MODELING
We introduce several techniques that improve the overall architecture. These may generally be considered techniques to improve signal propagation throughout the network, improving stability and learnability.
Norm Balance. Modern large language models employ pre-normalization architectures (Radford et al. 2019 Touvron et al., 2023a), departing from the nost-normalization design of the original Transformer /Vaswani et al. 2017j. Following established best practices, these models typically include final normalization layer after all residual blocks. H-Net adopts this convention through network normalization, by placing an RMSNorm (Zhang & Sennrich 2019) at the end of each network component (E", DD, and M).
This addition of normalization layer addresses a critical challenge in hierarchical architectures. Pre-normalization allows residual stream magnitudes grow unbounded through successive layers, with feature norms increasing monotonically. For H-Net, this poses a particular problem: the ar- chitecture leverages residual connections preserve fine-grained information across stages. With- out network normalization, outputs from deeper components (especially the many-layered main network) would dominate the residual signals from earlier encoder networks through imbalanced feature norms, neglecting the fine-graing grained details that are essential for decompression. The normal- ization layers restore balance between processed features and residual information, ensuring both contribute meaningfully to the final representation.
Separation of Two Streams. Encoder outputs (f) serve dual purposes in our architecture: passing fine-grained information to corresponding decoders through residual connections, and providing compressed representations as inputs to subsequent stages. This dual functionality creates design challenge, as these two roles may benefit from different representations. We consider three options to address this: (i) apply a projection to the residual connection only, (ii) apply projection to the main network inputs only, (iii) and apply projection to both pathways.
we adopt the first approach - adding a projection (Linear) only to the residual connection. This choice is motivated by the fundamental principle of designing deep learning models (He et al. (006)): maintaining intact gradient flow through the main computational path is crucial for effective training.
As indicated in equation equation 3
Empirically, we found that the third option underperforms despite additional parameters and com- putations, as the extra projections interfere with gradient propagation. The second option, while preserving residual gradients, disrupts the main network's 's gradient flow and had worse training dy- namics. Our chosen design maintains unimpeded gradients from deeper stages while allowing the residual connection to adapt its contribution through the learned projection. This encourages the model to leverage the main network's 's computational depth while using residuals in a complemen- tary role.
One additional detail is that this residual connection is initialized close to O; earlier versions of H- Net found this be an important detail. but may be less important when combined with additional techniques such as LR modulation.
Learning Rate Modulation The hierarchical design of H-Net requires careful adjustment of learning rates across stages to ensure balanced training dynamics. Modern theory establishes that neural network hyperparameters should be scaled in predictable ways for optimal trainability (Yang) & Hu 2020). To provide a more systematic experimental results across different architectural con- figurations, we follow previous works and learning rates to be proportionally to the (1) square root of batch size (Malladi et al. 2022 Merril et al. 2025), and (2) inverse square root of hidden dimension (Vaswani et al. 2017
Yang & Hu 2020). Concretely, without heavy manual tuning,
define X as follows:
- N
D N = 1.0
NGPT
(11)
II.
N
D
where NGPT, is the average number of bytes per token of training dataset, which is 4.6 for the GPT-2 tokenizer on Fine Web-Fdu.
29

---

Published as a conference paper at ICLR 2026
With this modulation, the model achieves more stable training dynamics and improved convergence behavior across the entire hierarchy. In particular, we empirically find that since outer stages directly influence the chunk boundaries that inner stages depend on, the higher learning rates in the outer stages seem to accelerate learning the chunking mechanism. We note that such principles for opti- mizing signal propagation neural network hyperparameters change is an active area of research, and our scaling factors are just heuristics that can likely be improved.
C.4 AUTOREGRESSIVE TRAINING AND INFERENCE
In this section, we explain how H-Net preserves autoregressive properties throughout its hierarchical structure during both training and inference.
Every component of H-Net (i.e,, encoder-, decoder-, main- networks, and the dynamic chunking mechanism) is carefully designed to preserve autoregressive properties essential for language mod- eling.
Training. During training, H-Net employs standard causal masking across all sequence mixing layers. DC maintains causality by computing boundary probabilities based only on current and previous representations. Specifically, the boundary probability p. depends current and previous positions (equation equation 4 ensuring no information leakage from fu- ture tokens. The smoothing module similarly maintains causality through its recursive formulation (equation equation 5), where each output depends only on past compressed representations.
and K2 from the
Inference. For inference, H-Net generates raw bytes (or whatever the outermost modality is) au- toregressively with modified procedure to handle its hierarchical structure.
Generation with a prompt proceeds as follows:
1. Initial processing:
During prefill, we generate chunks via the encoders (as in training). For each component (i.e. the isotropic components, and the routing module and dechunking layer), we generate a state. Isotropic state (e.g. KV cache for Transformer layers, SSM state for Mamba-2 layers) is generated as usual.
2. DC state and DC step: As noted above, the DC modules have recursive formulations that maintain causality train-time. These recursive formulations become autoregressive formulations at inference time.
(a) Routing Module: In order to compute pt, we need k;-1 (see equation equation 4 so our state consists of the key value of the most recent token processed.
(b) Dechunking Layer: In order to compute ž, need Pt and ž Thus, the dechunk- ing layer state should consist of the last á value.
3. Token Generation To perform a model step, we do the following for a 1-stage hierarchy:
(a) Pass the token through the encoder network,
(b) Step the routing module determine whether the token needs to be processed by the main network,
(c) Step the main network if necessary, in which case we also need to step the dechunking layer.
(d) Use the result of the dechunking layer step the decoder network.
consequence of this inference formulation is that, at inference time, H-Net decides individually for each token how much compute to when processing it. Therefore, H-Net allocate more or less compute to different tokens as it deems necessary. A particular connection is that inference resembles speculative decoding (Leviathan et al., 2023 Chen al. 2023), which also involves small network (the draft model) stepping on every token, and a larger network (the verification model) only stepping on contiguous chunks of every few tokens.
'Her, token in the autoregressive generation sense, referring time step, not the literal BPE taken sense.
30

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTAL DETAILS
D.1 FLOPS COMPUTATION
We largely follow Hoffmann et al. (2022) with two marginally updated computations: (1) add com-
putations for Mamba-2 (Dao & Gu 2024), and (2) modify computations in MLP blocks as use
the recent Transformer++ architecture. Assuming that all query, key, and value share the same num heads and him, we calculate the forward pass FLOPs follows:
Embeddings: 2 x seq len x vocab size x model
Attention:
QKV projections: 2 x 3 seq len d model x (num heads head dim)
Attention Logit Calculation: 2 x seq seq (num heads x bead_dim)
Attention Score Softmax: 3 x num heads x seq len x seq. len
Score Query: x seq len x seq.len len x (num heads x head _di)
Output projection: 2 x seq len x (num heads x head dim) x d model
Mamba-2:
X projections: 2 seqleen x d_model (2 expand d_model)
BCA projections: seq len model (2 x d state + num heads)
SSD: 2 x seqllen x (expand x d_model) x d.state
Depthwise Convolution: x seq len x d model x window size
Gating: 5 x len x d_model
Output projection: 2 x seq len x d model x d model
Gated MLP:
-In, Gate, Out projections: seq x (3 x d_model x ffw-size)
- -Gating: 5 x seq len x model
Logit Prediction Head: seq vocab size d model
We assume the backward pass consumes twice the FLOPs of the forward pass.
D.2 ROBUSTNESS SCORE
We introduce metric called the robustness score measure the robustness of model's perfor- mance to textual perturbations, defined for Hellaswag as follows:
perturbed accuracy 0.25
robustness score = 100 max accuracy - 0.25, 0)
This score measures the percentage original (unperturbed) performance that is captured by the model in the perturbed setting. We subtract by 0.25 as HellaSwag is multiple choice with 4 options, thus model that scores 0.25 in the perturbed setting should be considered have lost all of its original capability.
D.3 EXPERIMENTAL SETUP FOR CHINESE AND CODE.
In Section 3.2 we analyzed the performance of H-Net (2-stage) against Transformer and H-Net (space) on Chinese and on code, finding superior scaling for H-Net (2-stage) versus the other archi- tectures. Here, we describe additional details from the experiment.
On Chinese and code, we use 46B token subset from FeneWeb-Edu Wrreb-Ed--Chheee--VV Chinese- V2.1 YYu et 2025) and Github subset from Pile (Gao el al. 2020) to train three models at the 1.3B GPT-3 XL scale: H-Net (2-stage), H-Net (space), and Transformer. We maintain the same bytes per gradient step (256 batch size with 8192 ut f-8 encoded bytes per example) as the main text experiments.
For model architecture, we primarily matched the settings from the GPT-3 XL, including dmedel and encoder/decoder architecture for H-Net models. However, we adjusted the number of layers the main network of each model to account for slightly different compression ratios. Specifically, the Chinese-language models used slightly higher total training flops target than the original language models, while the code models used a lower flops target. Full architecture details and results are also in Table
31

---

Published as a conference paper at ICLR 2026
Table 5: Architecture details and model benchmarks for Chinese and code models. BPIC (de- fined in Table 2 denotes the compression between the main network and outermost stage (bytes). Each H-Net used (3,3)-DC, targeting inner downsampling ratio of 9. However, the resulting BPIC was significantly different, indicating that code is much easier to compress than Chinese. In terms of results, H-Net (2-stage) performs better than both H-Net (space) and BPE Transformer Chinese, which is reflected in the downstreams. On the other hand, H-Net (2-stage) achieves similar performance to H-Net (space) code, and both H-Net models perform significantly better than Transformer.
CHINESE
CODE
MODEL
BPIC
MAIN ARCH.
VAL. BPB L XW-ZH. ACC. f BPIC
MAIN ARCH.
VAL. BPB J
Transformer
3.62
T15
0.7404
0.599
3.58
T13
0.3376
H-Net (space)
3.38
T19
0.7478
0.629
7.97
T40
0.3163
H-Net (2-stage)
5.81
T30
0.7032
0.663
7.23
T28
0.3161
For the H-Net (2-stage), we use the same target downsampling ratio (No = N' 3) as the main experiments. Unlike BPE spacelike-basets tokenization, whose downsampling ratios vary widely by dataset, H-Net allows for using similar compute budgets without much adjustment. For H-Net (space), 2024), and for BPE, we use the Llama3 tokenizer (Grattafiori et al., 2024). as the GPT2 tokenizer attains very poor downsampling ratios both datasets. Despite this change, both H-Net (space) and Transformer (BPE) still have highly varied downsampling ratios between ordinary (primarily English) language, Chinese, and code. On the other hand, H-Net can adhere to a target ratio regard- less of dataset, chunking into concepts at appropriate ratios.
use the same definition of spacelike the original SpaceByte paper (Slagle)
32

---

Published as a conference paper at ICLR 2026
Perplexity on HG38
MODEL / ARCHITECTURE
PARAMS.
FINAL PPL.
Transfomer
Transforner
(T9)
29M
2.769
2.85
Mamba-2
Mamba-2
33M
2.757
2.0
H-Net (Trans.)
H-Net (Mamba)
H-Net MST1 T15 + M4)
64M
2.705
H-Net M3TI MI5 + M4)
66M
2.697
2.00-
Table 6: Model details and final perfor- mance on HG38. We trained two isotropic models and two H-Net models, varying the main network architecture (Transformer or Mamba-2). Each H-Net model outperforms the corresponding isotropic model. We em- pirically find that the E" = M3TI en- coder architecture slightly outperforms a pure Mamba-2 encoder 0 M4 (Section E.3.7).
2.75
officient isotrapic model
1
Training Base Pairs Seen
Figure 5: Scaling performance on HG38 during the stable phase of training. Each H-Net model achieves the same pre-decay perplexity of the cor- responding isotropic model with approximately 3.6x less data.
Table 7: Distilling Llama 3.2 3B to a byte level model. Average ace indicates average of the benchmarks measured in 2 H-Net loses performance across the board compared to the teacher, which is expected because we cannot quite replicate the exact behavior of the original model due to non-causality of BPE tokens. However, it is still much stronger than an H-Net trained from scratch on this small amount of data (189B bytes).
LMB.
HELLA,
PIQA
ARC-E
ARC-c
WINO,
OPEN.
AVERAGE ACC
MMLU (5-SHOT)
MODEL
ACC f ACC.n
ACC
ACC f ACC_n f ACC f ACCJn f
ACC
Llama 3.2 3B (base)
0.701
0.737
0.768
0.745
0.460
0.688
0.428 0.414
0.647 0.617
0,561
Distilled H-Net (1-stage)
0.634
0.702
0.761
0.721
0.433
0.665
0.519
E ADDITIONAL EXPERIMENTS
E.I DNA (HUMAN GENOME) EXPERIMENTS
DNA is a setting that presents both a unique promise and challenge for hierarchical modeling. For one, handcrafted tokens do not work well on DNA, due to the lack of segmentation cues. Addition- ally, the same sequence of base pairs may serve different functions (e.g., depending on whether or not the pair is inside gene or not). Consequently, naive BPE-based approach may not work either. On the other hand, DNA can exhibit higher resolution structure (e.g., codons, various regulatory el- ements), suggesting that there is room for principled hierarchical modeling, Indeed, tate-of-thhearr DNA models (Brixi et al. 2025) operate directly on base pairs (A, C. G, T) with implicit hierarchical structure.
Thus, we evaluated four models DNA: two isotropic models (pure Transformer and pure Mamba- 2) operating at the base-pair level, and two corresponding H-Net (1-stage) with Transformer and Mamba-2 as the main network. Fach model was trained on the HG38 dataset with learning rate of 10-3 for modules at the base-pair resolution. For the H-Net models, we used a downsampling ratio of 2 = 3. All models were trained with d of 512, which was used for all isotropic modules of H-Net (including the main network).
Previous work has shown that SSMs show improved DNA modeling ability compared to Trans- formers (Gu & Dao, 2024). and we find that this effect is preserved when examining Transformers vs. Mamba-2 as the main network (see Table (). This finding suggests that existing layer selection principles can be applied when deciding main network architecture. In fact, by directly comparing the perplexity curves during the stable phase of training (Figure 5 we find that H-Net models can achieve similar performance to isotropic models with 3.6x less data, finding that holds for both choices of main network architecture.
E.2 DISTILLING TOKEN-LEVEL. MODELS TO BYTE-LEVEL
The role of the outer stages in H-Net is analagous that of the tokenizer, embedding module, and LM head in a traditional BPE--basd language model; together, these modules interconvert between raw text and embedding space that the main model backbone process. Given this similarity,
33

---

Published as a conference paper at ICLR 2026
k
e
n
8
Soard kus
BCE Loss
MSE Loss
Figure 6: Auxiliary loss strategy for training the encoder of a H-Net with pretrained main stage. In order to mimic the behavior of the tokenizer + embedding layer of pretrained language model, we add supervision to both the routing module boundary probabilities and to the hidden states that we pass through to the main network. These losses encourage the encoder to tokenize once at the start of every token, while also passing the correct embedding into the main network near the start of the token, thus making maximal use of the next-token prediction ability.
we investigated whether it would be possible convert BPE-tokenized model directly into byte level H-Net. To do this, we trained a I-stage H-Net with frozen main network initialized from the backbone of Llama 3.2 3B (base). Our H-Net uses 4 Mamba-2 layers without MLPs for both the encoder and decoder with hidden dimension of 1536. Because the Llama model has a hidden dimension of 3072, we add MLP adapters with hidden dimension 8192 after chunking and right before dechunking (i.e. right before and after feeding into the main stage). We train the model for 90000 gradient steps with sequence length 8192 and batch size 256, for a total of 189B bytes.
Aligning the encoder. The primary difficulty in converting tokenized model into a byte-level one that the encoder and DC must produce chunks that the tokenized model can produce useful output with. Thus, our training (besides using the standard next-byte prediction loss), adds the following losses (see Figure a visual).
1. A binary cross-entropy operates on the routing module probabilities and targets the router to pass the start of every real loken through the main network.
-prelitct-priiitioon loss (with equal weight as the main loss) that
2. A hidden state matching loss that matches the post-adapter hidden state with the "correct" hidden state. Here, if & is the hidden representation that was passed into the main network at (byte) position f, we try to match with the embedding of the token that the 7th byte was part of, except when the byte the first byte of its token, which case we match the z, with the previous token's 's embedding. Embedding matching is done with L2 loss with a weight of 0.02.
In the ideal case where both losses are zero, the router sends exactly the first byte of each token through to the main network with the right embedding. The main network would thus see exactly the representation would see with tokenizer embedding setup. In practice, sending both losses to zero is literally impossible, as we discuss below. However, we still find that the Moundary-predictee loss is crucial for learning good matching, while the embedding matching loss is helpful in speed- ing up training but not necessary. fact, increasing the loss weight on the loss too much harm the aagguage-modeling loss.
Tokenization bias. We are not able to send all auxiliary losses zero because online prediction of BPE boundaries is an impossible task. |han et al. (2025) coined the term "tokenization bias" represent the fact that the tokenization process implicitly contains next-byte information. For example, the Llama 3 tokenizer tokenizes the strings _di st 11 and distincs into [_dist, i11] and distinct). Prior use of this term has been to suggest that if an autoregressive
34

---

Published as a conference paper at ICLR 2026
Bits-per-byte (2-stage)
Compression Ratio L'/22
Compression Ratio L2/L1
1.05-
1.00
0.85-
0.90-
0.35
AM My
0.85
0.34
0.38
0.80-
258
1018
1258
1008
125B
768
100B
125B
Total Training Bytes
Total Training Bytes
Total Training Bytes
H-Net
w/o smoothing
w/o cosine routing
w/o STE
Figure 7: Ablation study on key H-Net components showing validation BPB (left) and compres- sion ratios for the first stage L' L'/L" (center) and second stage L²/ (right) during training. Using H-Net (2-stage), we evaluate the impact removing three components: the smoothing module (w/o smoothing), the similarity-basse routing module (w/o cosine routing), and Striigh-Through Esti- mator (w/o STE).
language model prompted with .dist, the nature of its training will be that will never complete with inct (this is in fact flaw of all lokenization-basee models).
For us, however, tokenization bias implies that we cannot determine whether not the i in _disti is the start new word until after seeing the next character. In fact, the problem can be even worse-consider .applicablee... icbble (becomes [_ppp, licablel) and Lappli cant (be- comes Lapplicants iccntt];; ): Determining whether the start of a token requires knowing the next two bytes as well.
While the H-Net does use the main network, it is not able to exactly match the behavior of the original tokenized model. Instead, it finding slightly different representations of tokens use in the main stage. Recent work has shown that tokenized language models can process tokenization sequences distinct from the "canonical" greedy tokenization (Vieira al. 2024), so it is possible our H-Net found another alternate representation that the pretrained model could process.
Remark. One might ask our distilled model has simply learned to tokenize spaces (since spaces are always the start of new token). has not. Simply tokenizing on spaces would yield a sub-95% boundary prediction accuracy; however, our distilled model gets boundary prediction accuracy above 99.5%. This suggests that the resulting H-Net able to recognize some, but all, subword boundaries.
Results. The results from our distillation procedure are shown in Table 7 H-Net able to approx- imately match performance across almost all benchmarks; in general, H-Net is not able to replicate the behavior of the tokenized model exactly, is not unexpected that the benchmarks are slightly worse. Byte-Latent Transformer (Pagnoni et al. 2024 Table 5) performs a similar experiment, and they see greater gap among most benchmarks (particularly PiQA, Arc-Easy, and Are-Challenge) despite using a much larger amount of data (220B tokens versus 189B bytes); it is possible that this performance difference is due the fact that BLT module cannot be supervised to align boundaries the way that end-to-end DC can.
E.3 ABLATION STUDIES
In this section, we provide comprehensive ablations that study individual architectural components and design choices. For ablation studies, we employ H-Net at Large scale following the configura- tions in Table 1 training 36B tokens randomly sampled from Feb-Fu...d.
E.3.1 IMPORTANCE OF COMPONENTS IN H-NET
Figure illustrates the impact of each architectural component both model performance and compression ratio (I.++1/A /L²) stability during training. We conduct three targeted ablations: (i) using direct upsampling = En by removing the smoothing module (w/o smoothing), (ii) replacing
35

---

Published as a conference paper at ICLR 2026
Bits-per-byte
Compression Ratio L2/L2
0.94
0.24
0.52
0.50
0.23
0.88
0.36
408
808
1098
1208
1408
608
808
1008
1208
M88
Total Training Bytes
Total Training Bytes
M4-M4
M2T1-T1M2
T1M2-M2T1
T2-T2
Figure 8: Encoder-decoder architecture ablation using raw byte inputs. Validation BPB (left) and compression ratio L2/L" (right) for H-Net (I-stage) throughout training. We evaluate four encoder-decoder (E - D configurations: M4-M4. M2T1-TIM and TIM2-M2TI, and T2-T2, where denotes Mamba layers and T denotes Transformer layers.
the routing module that is based scaled cosine similarity, with direct probability prediction from individual inputs (w/o cosine routing), and (iii) skipping the straight-througe estimator in equation equation 9 (w/o STE).
The smoothing module proves essential for stable training dynamics. Without this module, com- pression ratios fluctuate severely throughout training. preventing the model from learning consistent chunking boundaries. This instability directly manifests substantial performance degradation, confirming that smooth gradient flow through the decompression process is crucial for effective end-to-end learning. While less critical than the smoothing module, both the limilarity-baset rout- ing module and STE operation exhibit importance in training stability and final performance. These components help maintain consistent compression ratios and lead to more interpretable chunking patterns. The similarity- based approach particularly enables the model to identify natural linguistic boundaries (e.g.. whitespaces, subwords) by comparing adjacent representations rather than making isolated predictions.
E.3.2 ENCODER & DECODER LAYER SELECTION
The composition of sequence mixing layers in H-Net's encoders and decoders substantially influ- ences both compression efficiency and modeling capability. We systematically evaluate different architectural combinations using H-Net (1-stage) while fixing all other configurations in Table T the same. Four distinct encoder-decoder (E"-D") pairings are tested: M4-M4, M2T1-TIM2, TIM2- M2T1, and T2-T2, where denotes Mamba-2 layer and T denotes Transformer layer. These combinations are chosen by keeping the symmetry and replacing each Transformer layer with two Mamba-2 layers, as they contain equivalent parameter counts - 12D2 for Transformer (4D² for the attention mechanism and 81) for an MLP) vs. R 6D2 Mamba-2 layer (no MIP).
Figure 8 and Figure 9 demonstrate that Mamba layers are essential for effective byte-level sequence processing. For both H-Net and SpaceByte+++, the pure Transformer configuration (T2-T2) ex- hibits by far the worst performance despite using more FI.OPs (it also down-compresses se- quences poorly compared to other configurations, thus using more compute the main network). This configuration struggles to compress byte sequences effectively, resulting in both computa- tional waste and degraded modeling performance. Performance improves monotonically with in- creased Mamba layer allocation, achieving optimal results with the highest compression efficiency in the pure Mamba configuration (M4-M4). These findings align with recent research demonstrating SSMs' advantages over Transformers for fine-grained sequence modeling (Goel et al. 2022 Schiff al. 2024), as corroborated by MambaByte's superior performance over LlamaByte Figure
natural question arises: does the importance of Mamba layers (i) stem specifically from process- ing fine-grained byte inputs, or (ii) because they are better for compressing information into the next stage, even at coarser input resolutions? To investigate these hypotheses, we train -ssagee H-Net on top of BPE-tokenized inputs processed by the GPT-2 tokenizer. We then evaluate six different encoder-decoder combinations.
36

---

Published as a conference paper at ICLR 2026
Bits-per-byte
Bits-per-byte
T3-T3
T2M2-MTT2
T1M2-M2T1
0.975
M212: T2M2
1M2
T1N--MAT
0.50
M4-MA 144
MAT1-T1M
0.88
0.900
0.80
0.84
0.850
408
639
809
1058
12585
1408
1008
1208
1408
Total Training Bytes
Total Training Bytes
Figure 10: Encoder-decoder architecture ab- lation using BPE-tokenized inputs. Assum- ing that GPT-2 tokenizer serves as the outer- most encoder-decoder (i.e., E we evaluate six E' combinations: M6-M6, M4T1-TIM4, T1M4-M4T1, M2T2-T2M2, T2M2-M2T2, and T3-T3.
Figure 9: SpaceByte++ encoder-decoder archi- tecture ablation using raw byte inputs. We evaluate four encoder-decoder (E D con- figurations: M4-M4, M2T1-TIM2 and TIM2- M2T1, and T2-T2, where M denotes Mamba lay- ers and denotes Transformer layers.
If hypothesis (i) holds, then we would expect different combinations of Mamba/
Mamba/Transforin Transformer layers the encoder/decoder have similar performance, since is known that they have similar performance on standard tokenized language modeling.
If hypothesis (ii) holds, then would expect that encoders/decoderss using some Mamba layers to be better than pure Transformer layers.
As demonstrated in Figure 10 Mamba layers prove significantly important even when processing BPE tokens rather than raw bytes, providing evidence for the second hypothesis.
We hypothesize that this consistent advantage across input granularities stems from fundamental ar- chitectural differences between SSMs and Transformers. While Transformers naturally store com- plete key-value caches for all positions, SSMs are designed to compress information into fixed-size states. This compression-oriented architecture aligns naturally with our chunking mechanism, which requires aggregating multiple input vectors into consolidated representations. The inherent compres- sion capability of Mamba layers makes them particularly well-suited for the encoder and decoder roles our hierarchical architecture (Gul 2025). Based on these findings, we employ Mamba layers throughout all encoders and decoders in our final H-Net configuration, as detailed in Table [
These findings transfer to more general hierarchical structures (such as a 2-stage H-Net at the byte level), which case the outermost encoder and decoder layers (E and DO) serve similar role as the GPT-2 tokenizer and the inner layers (E' and D would share similar findings of benefiting from using Mamba layers.
E.3.3 VISUAALIZATION OF TOKENIZED POSITIONS
In Figure we provide visualizations of the boundaries dynamically drawn by H-Net (I-stage) and H-Net (2 stage). The visualization offers several insights about how the model decides boundaries.
Single-stage behavior: H-Net (1-stage) predominantly places boundaries at whitespace characters, closely mirroring the delimiters used by SpaceByte. This indicates that H-Net learns that word boundaries represent natural semantic units in text. This convergence to spacelike boundaries. discovered purely through end-to-end training, conversely validates SpaceByte's ' strong empirical performance.
Hierarchical chunking patterns: The first stage of H-Net (2-stage) combines spacelike boundaries with first few characters of each word. This strategy helps the model because once the initial positions of a word are identified, the remaining characters become highly predictable.
Content-aware chunking: One might question if H-Net's chunking decisions follow static rules, such as drawing boundaries only certain fixed bytes (e.g., whitespace). However,
37

---

Published as a conference paper at ICLR 2026
1OOD 300
Stage0
Stage
Stage
8000 2000
300000
Staee0
Stage
b
Stage
aanenceaam
Stage0
Stage
Stage
Stage
nonn monen nancn D0nc
Stage0
Stage
Stage
toch 0000 mone no anceanne
Stage
Stage
Stage
(a) H-Net (1-stage), using 6-DC.
Stage
Figure 11: Visualization of boundaries drawn by H-Net. Numbers above indicate byte value, and the colored boxes indicate positions where b; 1. (a) H-Net (I-stage) tends to draw bound- aries at spacelike bytes, which very similar to SpaceByte. (b) The boundaries of the first stage in H-Net (2-stage) are focused on spacelike bytes, as well as starting characters of each word. The second stage of H-Net (2-stage) chunks the text into more meaningful units, such as words or numberings (i.e., (ii)") "). We can also ob- serve that it often chunks multiple words that form semantic group; for example, 'the backbone' and 'such as'
Stage
Stage
Stage
Stage
(b) H-Net (2-stage), using (3,3)-DC.
as shown in the figure, H-Net often merges multiple words and spacelike characters based content (examples include the backbone, such as, and (ii)).
Perturbation behavior: Figure 12 shows the same example with textual perturbations such removing whitespaces, which more prominently demonstrates that boundaries drawn by H-Net are based on content and context. In particular, it often still chunks between semantic words even if the space is removed.
E.3.4 HYBRID ARCHITECTURES FOR THE MAIN NETWORK
We also aimed to understand the role of architecture selection in the main network. To this end, we compared H-Net (2-stage) with an identical model where we replaced the Transformer stack with a hybrid model containing both 20 Mamba-2 and 7 Transformer layers interleaved in 3:1 ratio. Hybrid architectures have shown promise isotropic (BPE) models (Waleffe et al. 2024) and similarly perform better for our choice of main network (Figure T3).
38

---

Published as a conference paper at ICLR 2026
Stage
Stage
Stage
Stage0
Stage0
Figure 12: Visualization of boundary positions dynamically drawn by H-Net (I-stage). The given text is perturbed that some whitespaces are missing. H-Net detects word boundaries even they are not explicitly separated by whitespaces.
Bits-per-byte
Bits-per-byte
0.58
HNNi (2-stage, bybbiss
1.00
LiamaByta (230M)
0.84
H-Nitt (2-stage, T27)
LlamaByta MrE (860M)
1.00
H-Net (1-slage, 690M)
H-Nat (2-slage, 870M)
0.80
0.90
0.90
0.7
0.80
0.50
398
1008
2058
2008
458
608
1008
1208
MeB
Total Training Byles
Total Training Bytes
Figure 13: Hybrid main network. Bits-per-Be byt- during the stable phase of training, for H-Net (2- stage) with Transformer main stage and with hy- brid main stage. The hybrid main stage scales better, similar to findings for standard token- based language models. This finding suggests that design principles for isotropic (tokenized) models carry over to choices of the main net- work.
Comparison to Mixtures-of- Experts. Bits-per-byte comparison of H-Net (both I-stage and 2-stage) to LlamaByte-MoE, which is FLOPs-matched MoE model that similar number of parameters as H-Net (2- stage). Both H-Nets perform much better than LlamaByte-Mor implying that H-Net's capabil- ities do not just come from sparsity.
Figure 14:
E.3.5 COMPARISON TO MIXTURE-O CEITURE-OFFELLE OF-EXPERTS
H-Net can be viewed form of dynamic sparsity similar to Mixture-of -fxture-of-Experts (MoEs), in that they are able to improve performance by using more parameters, all while keeping the total FLOPs budget constant. We were interested in understanding whether or its performance benefits were simply due to increasing sparsity. We compare against sparsified version of LlamaByte (byte-level isotropic Transformer model) the Large scale with standard Mixture-of-
recipe (Fe- dus et al. 2022) and similar parameter count as ours (Figure 14). While sparsity does improve LlamaByte performance, it still far worse than either FLOPSsmatched H-Net (I-stage) H-Net (2-stage), even with similar parameter count. We interpret this result as: H-Net not only achieves sparsity, but does in more semantically meaningful manner, which allows for better scaling than even generic sparse methods.
E.3.6 DIFFERENT DOWNSAMPLING METHODS THE CHUNKING LAYER
Given the dynamically determined boundaries from the boundary predictor, we explore various com- pression strategies in the chunking layer. We compare the default Downsample operation of H-Net
39

---

Published as a conference paper at ICLR 2026
Bits-per-byte
XAttn Res
Default
0.84
XAttn
Max + Res
0.83
Max
Mean Res
0.82
Mean
(b)
Mean/Max
Base
Pooling
0.81
0.80
KV
KV
Cross Attention Pooling
608
808
1008
1208
1408
Total Training Bytes
Figure 15: Compression Methods in chunking layer. Default: H-Net's Downsample operation (eftta)). Max/Mean: Channel-wise max and mean pooling within boundaries (left-b). XAttn: Cross- attention pooling within boundaries (left-c). +Res: Adds boundary vector residuals to compressed outputs.
Perplexity on HG38
285
(T1M13T1)
MODEL
ARCHITECTURE
PARAMS.
FINAL PPI.
280
H-Net (Trans.)
H-Net (Mamba)
H-Net
M3T1 + +T1 M4
64M
2.705
285
H-Net
M311 + +,MIS M4
66M
2.697
H-Net
M4 T15 M4
62M
2.722
2.00
H-Net
M15
64M
2.706
H-Net
M4 TIMISTI + M4
64M
2,706
275
29
10
1
19
Table 8: Encoder architecture ablations on HG38. Switching the encoder architecture from M3T1 to M4 leads to worse perfor- mance across the board, though the results still better than isotropic models (Ta Transformers in the encoder network do not appear be helpful for text (Figure 8). sug- gesting that this finding may be modality- specific.
Training Base Pairs Seon
Figure 16: Mamba-2-only encoder loss curves during the stable phase of training. The pure Mamba-2 model is more unstable with a loss spike. Adding Transformer layers to the main net- work near the DC modules can alleviate instabil- ities. H-Net (I-stage, principled) corresponds to the TIM13TI main network architecture.
(see Section 2.2.1 against three alternatives (see Figure leftt): channel-wise max/mean pooling and cross-attention, all applied to vectors within the same boundary. Despite its simple design, the default compression in H-Net performs on-par with the other variants demonstrated in Figure [5] right. This shows that the sequence mixing layers in encoder are trained to implicitly compress nec- essary context into vectors at boundaries, without explicit compression mechanisms such pooling or cross-attention.
E.3.7 DNA ARCHITECTURE ABLATIONS
As shown in Figure H-Net (I-stage) with an M3TI encoder achieves 3.6 the data efficiency of an isotropic architecture. As mentioned in the caption of Table 6 we found that an M3T1 encoder outperformed a pure Mamba-2 M4 encoder, which demonstrated Table g The results in Figure show that putting Transformer in the encoder network does not appear be helpful for text. Thus, it is possible the Transformer being useful is a DNA-specific result.
Interestingly, the loss curve for the M4 encoder with a pure Mamba-2 main network was more unstable. We then also tried replacing the M15 in the main network with a TIM13TI architecture, inspired by the finding that Transformer layers good for dealing directly with compressed input (see Figure TOp. The new, principled main network architecture improved stability greatly as shown in Figure
40

---

Published as a conference paper at ICLR 2026
F LLM USAGE FOR PAPER WRITING
L.Mss were used only to detect typos and grammatical errors.
G
ACKNOWLEDGEMENTS
We thank Nimit Sohoni, Eli Pugh, Justin Liu, Karan Goel, Arjun Desai, and Brandon Yang for feedback and support throughout the project. We thank Tri Dao for feedback on the ideas and earlier drafts. We thank Aakash Lahoti, Aviv Bick, Kevin Li, and Isaac Liao for feedback and discussions.
41


Reviews:
[{"review_id": "U8qmqPLgiO", "reviewer": "Reviewer_UxoT", "summary": "The authors propose a tokenizer-free architecture that dynamically segments sequences by jointly learning context and content dependent boundaries alongside the language modeling objective. Their model is hierarchical, enabling it to capture multiple levels of abstraction, from finegrained details to higher-order structure. In experiments on English, the architecture demonstrates increased robustness at the character level compared to BPE-based tokenizers, and the authors further report improvements for Chinese, code, and DNA sequences.", "questions": "- Any thoughts on how your model performs at really small scales <700M . I know that there seems to be more value these days in training models at larger scales, just curious if you have any thoughts.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper addresses an important limitation of many tokenizer-free architectures: training instability when boundary predictors must make discrete decisions (with or without supervision). Their proposed architecture is elegant in how it handles segmentation via the novel routing and smoothing mechanism. \n- The paper is well written with lots of ablations, detailed discussions on different architectural and experimental choices that potentially aid reproducibility.\n- Their experimental results are great to see, they demonstrate improvements over traditional BPE in downstream tasks, with robustness on character-level tasks, code etc. This speaks directly to the benefits of their dynamic tokenization strategies.", "weaknesses": "- Of course, this paper is not framed as a multilingual one, but the authors do claim improvements in other languages, and only evaluate on Chinese. While the improvements are notable, many recent frontier LMs are trained on web data across several languages. Do you have insights on how your architecture scales in a multilingual setting, when it is very common to have very distinct tokens mixed in individual sequences?\n- In addition to what I mentioned above, I am particularly curious about the potential challenges of scaling HNET experiments to a truly multilingual setting involving languages with distinct scripts, varying data sizes, and diverse linguistic structures. When data is highly imbalanced across languages or domains, how robust is the quality of the learned segmentation? Even if prior ratios are fixed to guide boundary prediction convergence, could the quality of learned segmentation degrade as language-specific data becomes more limited, given that boundary learning is inherently context-dependent? I will be interested to hear your thoughts on this.\n- Why are there no comparisons to BLT (https://arxiv.org/pdf/2412.09871) and the dynamic token pooling paper (https://arxiv.org/abs/2211.09761)? At the very least, there is some similarity in architectural designs with the dynamic token pooling paper.\n-  There’s also no discussion about how well the model handles out-of-domain sequences, and I don’t mean big shifts going from natural language to DNA or code, but more subtle, realistic shifts like moving to scientific text or just long-tail words in the pretraining data. What kind of segmentations are observed? Are they optimal?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qL0A3wXfXP", "reviewer": "Reviewer_675s", "summary": "This paper introduces H-Net, a novel and compelling hierarchical architecture for end-to-end sequence modeling directly from raw bytes, aiming to replace the conventional tokenization pipeline. The core contribution is a \"Dynamic Chunking\" (DC) mechanism that learns content- and context-dependent segmentation strategies jointly with the main model. This is achieved through a clever combination of a similarity-based routing module to predict boundaries and a smoothing module to ensure stable, differentiable end-to-end training. The authors demonstrate empirically that H-Net not only trains stably at scale (up to 1.3B parameters) but also outperforms strong, compute-matched BPE-tokenized Transformer baselines. Furthermore, they show that the architecture can be recursively stacked (2-stage H-Net), leading to even better performance and scaling, particularly on languages and modalities where traditional tokenization heuristics are less effective, such as Chinese, code, and DNA sequences.", "questions": "To further strengthen my assessment and address the concerns about the mechanism's fundamental nature, I would appreciate the authors' response to the following questions:\n\n- **Evidence for Principled Operation:** Beyond the final performance metrics in ablation studies, can you provide more direct evidence that the core modules are functioning as hypothesized? For example:\n    - **Routing Module:** Can you analyze the relationship between the cosine similarity scores and human-annotated semantic/syntactic boundaries? Does the \"w/o cosine routing\" variant learn a different, perhaps less interpretable, chunking strategy, or does it fail to learn any consistent strategy at all?\n    - **Smoothing Module:** Does the smoothing module primarily act on low-confidence boundaries ($P_t \\approx 0.5$), as intended? Could you provide statistics on the distribution of $P_t$ values and show how the EMA application correlates with them? This would help differentiate its role from being a general regularizer.\n\n- **Scaling and Stability:** Could you elaborate on the potential stability challenges at scales significantly larger than 1.3B parameters? Have you observed any trends in the training dynamics (e.g., the interplay between the ratio loss and the main loss) that might suggest future issues, and do you have any hypotheses on how to mitigate them? Proving the mechanism's fundamental nature requires confidence in its scalability.\n\n- **Hyperparameter $\\alpha$:** Could you provide more intuition on the choice of $\\alpha=0.03$? How sensitive is the model's final performance and, more importantly, the stability of the learned compression ratio to this hyperparameter? An ablation, even a small one, would be very valuable to understand the robustness of the training process.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Novelty and Significance:** The paper's primary strength lies in presenting a robust and scalable framework for fully end-to-end, learnable segmentation. The Dynamic Chunking mechanism, especially the smoothing module that turns a discrete selection problem into a differentiable one, is an elegant solution to a notoriously difficult problem that has hampered previous efforts. This work represents a significant step towards realizing the \"bitter lesson\" by replacing a major handcrafted heuristic (tokenization) with a learned component.\n\n- **Empirical Rigor:** The experimental evaluation is thorough and convincing. The authors conduct carefully controlled comparisons against strong baselines (BPE Transformer, MambaByte, SpaceByte) by matching both data and computational (FLOPs) budgets. The consistent outperformance of the 2-stage H-Net across different model scales is a powerful result.\n\n- **Generality and Robustness:** The paper convincingly demonstrates the model's advantages beyond standard English text. The superior performance on Chinese, code, and DNA sequences validates the claim that a learned chunking strategy is more generalizable than fixed heuristics. The improved robustness to textual perturbations is another key benefit of operating directly on bytes.   \n\n- **Strong Ablation Studies:** The authors provide detailed ablation studies that validate their key architectural choices. These studies effectively demonstrate the importance of the smoothing module, the similarity-based routing, and the use of SSMs (Mamba-2) in the outer encoder/decoder layers, strengthening the credibility of the proposed design.", "weaknesses": "- **Practical Efficiency Concerns:** The paper candidly acknowledges that the current implementation can be up to 2x slower during training and has dynamic memory usage, which can be unpredictable. This is a significant practical hurdle for widespread adoption and large-scale training, as it complicates hardware optimization and resource allocation.\n\n- **Uncertainty and Potential Fragility at Extreme Scale:** While stability is demonstrated up to 1.3B parameters, the fact that larger scales are left as future work raises questions about the fundamental robustness of the mechanism. The complex interplay between the main prediction loss and the auxiliary ratio loss could introduce unforeseen instabilities at much larger scales (e.g., 70B+). This limitation implies that the current mechanism, while effective, might not yet be a \"fundamental\" solution but rather one that is proven to work within a specific regime.\n\n- **Need for Stronger Evidence on Principled Operation:** This is the most critical weakness. The paper's core claim is that it introduces a principled, end-to-end chunking mechanism. However, the primary evidence comes from ablation studies showing that removing a component (e.g., the smoothing module) degrades the final performance metric (BPB). This demonstrates that the components are necessary for good performance, but it does not sufficiently prove that they are working as theorized. For instance, it is unclear whether the performance gain is due to the smoothing module correctly interpolating uncertain boundaries, or if it's acting as a complex, yet effective, form of regularization. The visualizations of learned boundaries are a good first step, but more direct, quantitative evidence is needed to establish that this is a truly fundamental innovation rather than a highly effective, scale-specific heuristic.\n\n- **Hyperparameter Sensitivity:** The ratio loss weight, $\\alpha$, is fixed at 0.03 for all experiments without an accompanying ablation study.1 The model's performance and learned compression ratio could be highly sensitive to this value, potentially requiring extensive tuning for new domains or modalities. This replaces one form of tuning (tokenizer design) with another, potentially more opaque one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jrhtIkNhaY", "reviewer": "Reviewer_jxd8", "summary": "The paper introduces a new tokeniser-free architecture, H-Net, that learns segmentation strategies and can be applied more than once, thereby creating learnable tokenisation. It evaluates its performance with respect to other architectures when data- and compute-matched.", "questions": "- will the codebase be released?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting architecture that contributes to investigating the long-lasting issue of tokenisation in language models (aka tokenisation-free architectures)\n- Many carefully conducted experiments, with positive results\n- Paper clear and well written", "weaknesses": "- I think it is not ideal that the state of the art is only briefly described in the main part of the paper, and discussed at more length in an appendix. It makes the main part of the paper not really self-contained\n- Moreover, the discussion on what is different and novel with H-Net with respect to previously published works is insufficient, especially in the main part of the paper. The authors write that \"H-Nets […] unlock the ability to remove another layer of pre-processing, such as tokenizers, and instead learn them end-to-end\". To me, it is not the case that H-Nets \"unlock\" (i.e. make possible for the first time ever) such an \"ability\". Previous work have done this before, and the authors should acknowledge it better, and clearly explain how and why H-Nets are novel and better.\n- The authors rightly explain that such tokeniser-free architectures remove the need for heuristic tokenisation strategies, and create \"optimal\" (in some sense) segments that can be visualised and analysed. The authors only show a handful of examples in English, although they mention the fact that improvements are better in Chinese. This is certainly due to the fact that their algorithm re-discovers whitespace-separated tokens in English, but cannot to that in Chinese, where there are no whitespaces. But a whole discussion on the boundaries endogenously learned by the model is missing. In languages using a script that includes whitespaces, how often are boundaries placed on whitespaces? When are whitespaces not used as boundaries, and when does the model places boundaries at other places? In languages using a script that does not includes whitespaces, do the endogenously learned segments \"look like\" what linguistic traditions (and more recently, treebank developers) have defined as \"words\" or \"word-forms\"? When using 2 levels of hierarchy, does the first level generate morph-like units (it seems that it does not, at least on the examples shown on English)? To me, adding such a discussion is absolutely necessary to understand what the model actually learns and how (and when) it (should) perform(s) better than models relying on heuristic and/or statistics-based tokenisation strategies (whitespace, BPE, etc.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4pN667Ed3U", "reviewer": "Reviewer_7WMM", "summary": "The authors propose a hierarchical model (H-net) that can operate on raw byte sequences. The model is learned end-to-end. First, the sequence is processed by a shallow Mamba-based encoder, then it is chunked and downsampled to reduce its length. Then, the main network, which can have any architecture —typically a Transformer or, recursively, an H-net for multiple hierarchy levels. Then, a smoothing module is applied to provide gradients for the decision points, the sequence is upsampled by repeating the main network's output the correct number of times, and a Mamba-based decoder network outputs the full-length sequence. Additional regularization is used to achieve the desired target compression ratio.\n\nThe model consistently outperforms its competitors, including Transformers, SpaceByte, and MambaByte", "questions": "- In line 206, the authors say \"where chunking layer ... \", however the chunking layer is defined afterward, which is a bit confusing\n- In line 355, the authors say \"As discussed Section, ... comprise mainly Mamba-2 layers.\". However, Mamba was never mentioned before in the paper.\n- It would be nice to describe what the scales are in the text. Now one has to look at the figure descriptions to be able to figure it out.\n- In line 1455, the authors say \"Multiplying upsampled vectors by their confidence scores incentivizes the routing module to make confident, accurate decisions.\". However, they are multiplied by one, and the difference is only in the backward. Why does this trick help?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Interesting, fully differentiable method\n- Improved scaling\n- Improved robustness", "weaknesses": "- Complexity\n- Clarity: Some baselines are not described in enough detail. For example, what exactly is H-Net (Space) or H-Net (pool)? Some details are described in the appendix, but are very vague. Line 353 claims that the main network also uses Mamba layers, but this is never described. Are all layers Mamba layers? Only some?\n- The only BPE model is a Transformer; however, the H-net uses Mamba layers. It would be nice to have a Mamba-based BPE model to see if the dynamic chunking or the Mamba is a bigger win.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)