ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
Authors: Chunyuan Deng
Sanket Lokegaonkar
Colin Lockard
Besnik Fetahu
Nasser Zalmout
Xian Li
Venue: ICLR 2026 Poster (2026)
Keywords: byte-level language modeling
tokenization
Primary area: foundation or frontier models, including LLMs
TL;DR: A Tokenizer-free Language Model based on Information-Theoretical Chunker

Abstract: Modern language models (LMs) still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce \textbf{ByteFlow Net}, a new architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaningful units. Our approach is grounded in information theory: ByteFlow Net performs compression-driven segmentation based on coding rate of latent representation, allowing the model to dynamically evaluate the information cost of grouping bytes and decide chunk boundaries during processing. Unlike prior self-tokenizing methods that depend on brittle heuristics with human-designed inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Experiments demonstrate that this compression-based chunking strategy yields substantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is not only feasible but also more effective, opening a path toward more adaptive, robust, and information-grounded language models.

Published as a conference paper at ICLR 2026
BYTEFLOW: LANGUAGE MODELING THROUGH ADAP- TIVE B YTT COMPRESSION WITHOUT A TOKENIZER
Chunyuan Deng" Rice University cd1 1 10ricee.eu
Sanket Lokegaonkar
Colin Lockard
Amazon Science Tlokega@amazon....mmm
Amazon Science
rdockard@amazzoo...
Besnik Fetahu
Nasser Zalmout
Xian Li
Amazon Science besnikf@amazonn com
Amazon Science
Amazon Science
nzalmout amazon. com
xianleel amazon. com
ABSTRACT
Modern language models (LMs) still rely on fixed, pre-defined subword tokeniza- tions. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce ByteFlow Net, new hierar- chical architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaning- ful units. ByteFlow Net performs compression-drivee segmentation based on the coding rate of latent representations, yielding adaptive boundaries while preserv- ing static computation graph via Top-K selection. Unlike prior self-tokenizing methods that depend on brittle heuristics with desannd inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Ex- periments demonstrate that this compression-basee chunking strategy yields sub- stantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is only feasible but also more effectivee opening a path toward more adaptive, robust, and information-groundee LMs.
INTRODUCTION
Tokenization is foundational step in every language model pipeline (Grattafiori al., 2024; Team et al., 2025; DeepSeek-AI et al., 2025; Yang et al., 2025). The model's first action is to segment raw input-be it text, code, or other modalities-into discrete tokens. This seemingly simple decision carries profound consequences, defining the model's vocabulary, sequence lengths, and the very granularity of its learned representations. The primary limitation of dominant strategies like byte- pair encoding (BPE) (Sennrich et al., 2015; Gallé, 2019; Liu et al., 2025) is their static nature. After training, they apply fixed segmentation logic to all inputs, ignoring context, linguistic nuance, or task-specific requirements. This static property subword level is the source of many wierd model behaviors, such as difficulties with counting, arithmetic, structured data, and multilingual text (Rust al.. 2020; Zhang al., 2024; Yehudai et al., 2024). At a more fundamental level, tokenization introduces a non-learnable stage into the pipeline, breaking the end-to-end language modeling. This imposes rigid inductive bias, forcing the model expend its FLOPs predefined units rather than learning how to allocate them dynamically.
Recent efforts eliminate tokenizers have largely converged hierarchical architectures. The tral challenge for such designs is defining the high-level semantic units beyond byte level. Current methods generally fall into chunking via fixed strides, word boundaries, regular expressions (Yu et al.. 2023b; Slagle, 2024a; Videau al., 2025), and ii) Dynamic chunking that learn segment sequences using a neural net- work, entropy thresholds, or cosine similarity (Nawrot et al., 2022; Pagnoni et al., 2025; Hwang
main categories: Heuristic-bused strategies that employ static
WWrk primarily done during internship Amazon Pretraining Team.

---

Published as a conference paper at ICLR 2026
(c) FLOPss assignment
Local Decoder
subword seglen
byte seq
Breh Dec
Isotropic
pooled seq
Global Trans.
block
Upsampler
Isotropic Models
Hierarchical Models
Canon Encoder Decoder
Oh: fe-s My ho-. We Re-2
Leusal_ccmve (kernel size
Global Trer
sudnng Window
Canon Layer
SwIOLL
Canon Luger
(a) Coding--Ran Rate Based Chunking
Downsempled
Latent Manifolde
Downsampler
Local Encoder
Original Byte Sequence
Repr. Space
argmin R(
)-
Figure I: Architecture of ByteFlow Net. (a) ByteFlow Net's chunking strategy primarily driven by the coding rate R of latent representations. As shown in the figure, the model is encouraged to select token boundaries that form pooled subsequences which best compress the original input. (b) Since byte-level sequences are roughly 4x longer, directly applying O(n2d) softmax attention be- comes probibitively expensive. To address this, we adopt sliding-winder winddww attention (SWA) combined with canon layers (Allen-Zhu, 2025), enabling efficient and low-cost token mixing.
(c) The beauty of the hierarchical architecture lies in allocating the majority of FLOPs operating at the high-level information (a deep and wide global transformer), while using lightweight local encoders/decoders (shallow and narrow) to quickly process low-level information.
et al., 2025b While heuristic approaches embed strong inductive biases into the model, dynamic methods introduce considerable uncertainty into the chunking process, which hinder pattern finding during the early stages of pre-training. Furthermore, we still lack a dynamic mechanism for guiding the model's allocation of FLOPs in principled manner.
We introduce ByteFlow Net (Figure 1), a novel hierarchical byte-level architecture that learns to self-tokenize directly from raw byte streams. Rather than applying fixed vocabulary, ByteFlow Net integrates segmentation into its forward computation: as bytes flow through the network, it dy- namically promotes them higher-level calculations. The decision of when to commit boundary is framed as a principled, cedting-rate-baee compression problem, estimating the representational cost of promoting the position a higher level. This formulation turns boundary placement into an online cntrmatiionthhoreeie optimization problem, enabling the model to adjust token granularity according to input complexity on its own.
Architecturally, ByteFlow Net follows a simple but effective hierarchy: The process begins with local encoder that transforms byte sequences into contextualized representations. Next, a chunking module applies the coding-rate criterion these representations, producing higher-level tokens the fly. These dynamic tokens are then modeled by a global transformer to capture the deep and ab- stract patterns in high resolustion level, before decoder maps the global context back byte-level predictions. Because this entire boundary selection process is integrated into the model's compu- tation, ByteFlow Net naturally adapts across languages and domains without requiring language- specific rules or separate tokenizer training St unifying segmentation and representation learning, ByteFlow Net ensures that the boundaries of higher-level tokens optimized maximize information density for the global transformer. This transition from a fixed, discrete preprocessing
HeNet (Hwang al., 2025a) is our concurrent work that also explores end-to-end tokenizer-free modeling. We contrast chunking approach with theirs in & 4.4.
2

---

Published as a conference paper at ICLR 2026
step to continuous, learnable bottleneck allows the model to preserve nuances such as morpho- logical variations or structural symbols that are often lost or fragmented by static tokenizers.
Contributions.
We introduce a new paradigm that replaces static tokenization with dynamic, learned seg- mentation. Our architecture, ByteFlow Net, operates end-to-end on raw bytes, using principled informattin--teeortie objective to identify meaningful units on the fly.
We demonstrate superior performance and scaling through extensive experiments. Byte- Flow Net consistently outperforms both strong LLaMA baseline and other byte-level archi- tectures pre-training loss and downstream tasks, showing that end-to-end, tokenizer-free modeling is not only feasible but more effective.
We reveal that the success of our approach stems from its ability to preserve a coherent latent manifold. Our ablation studies show that the coding-rate objective uniquely main- tains the geometric structure of the data's 's representation, preventing the fragmentation that plagues other methods and enabling more powerful learning.
RELATED WORK
Tokenizer- Architecture. Modern tokenizer-free architectures can be broadly categorized into three main approaches:
Pure Byte-Level Modeling: These models perform language modeling directly on raw byte sequences (Xue et al., 2022a). Given that the O(ndd) complexity of full attention is prohibitive for long sequences, architectures like MambaByte (Wang et al., 2024) have emerged effective solution, balancing fine-get men-giaeeed information processing with com- putational efficiency.
These methods use fixed, rule-based strategies to group bytes into larger units. For instance, MegaByte (Yu et al., 2023b) uses a fixed stride (e.g.. 4 or 6 bytes) to create a higher level of representation, outperforming pure byte-level models while significantly saving FLOPs. Building on this, SpaceByte (Slagle, 2024b) uses word boundaries for chunking, achieving performance exceeding BPE- based transformers on some pre- tre-training corpora. AU-Net (Videau et al., 2025) further refines this concept by replacing simple word boundaries with flexible set of regex rules to better handle special tokens and digits.
Hierarchical Modeling with Heuristic Chunking:
par with or even
Hierarchical Modeling with Dynamic Chunking: Instead of fixed rules, these models employ a learned mechanism to determine chunk boundaries. Nawrot al. (2022; 2023); Kallini et al. use neural network to gate token boundaries. The Byte Latent Transformer (BLT) (Pagnoni et al., 2024) first trains a separate entropy model and then uses it as a proxy to set global chunking threshold. This multi-stage process is not fully end-to-end and functions more like different tokenizer. In a concurrent work, H-Net (Hwang et al., 2025a) uses the cosine similarity between neighboring representations decide chunking.
Tokenization in Language Modeling. The prevailing solution in modern LMs is subword tok- enization (Sennrich et al., 2015; Kudo & Richardson, 2018: Zouhar et al., 2023: Schmidt et al., 2024; Liu et al., 2025) (e.g., BPH), which use fixed-size vocabulary of word pieces to represent any text. These fixed vocabularies create a rigid, non-learnable stage in the modeling pipeline, often causing brittle and unexpected behaviors (Belinkov & Bisk, 2018; Sun et al., 2020; Rust et al., 2020; Petrov et al., 2023; Schmidt al., 2024; Zhang et al., 2024; Yehudai et al., 2024), which motivated the development of modern tokenizer-Fo freee models that operate directly raw bytes.
BYTEFLOW NET
Overview, ByteFlow Net hierarchical architecture that operates through five main stages: local encoder, downsampling (coding-rate chunking), global modeling, upsampling, and decoder:
3

---

Published as a conference paper at ICLR 2026
21:T E V
Local Encoder
h1TT E R Xxuuu
(contextualized byte representations)
(1)
Downsampling,
21:K E IR x dd
(adaptive chunking, K < T)
(2)
Global Transformer
91:K E R
Xdgaa
(High-resolution level modeling)
(3)
Upsampling,
$1.7 E R. d'uuaa
(reconstruct to original length)
(4)
Decoder p(z(+11)) E V (next byte prediction)
(5)
Here T' is the input sequence length, V E A A (contains 256 UTF-8 Byte plus two BOS/EOS tokens) is the byte vocabulary, and dlocal, dglobal are the hidden dimensions at local and global levels.
3.1 LOCAL ENCODER: FAST PROCESSING OVER BYTE-LEVEL REPRESENTATIONS
The local encoder stacked small transformer. The input byte sequence æT.T VT first into a continuous representation h 1/7 by the learned byte embedding matrix, then transformed into contextualized representations h1rr E RT'xdpaa
Transformer Blocks with Sliding Window Attention. We stack E pre-norm causal transformer blocks. For each layer E {I,..., and position E (I,,..,TT}:
Ut (1)
LN(M
(6)
7 = (anon (1-1) + SSWA(Q V)), Q.K. V
- XQ x
(1) u1
"I Xx: "1
11
(
Xv
(7)
= LN(R.))
(8)
(1) Canon (hi) (wiGLU((;;)) (*)))
(9)
where: LN(-) denotes layer normalization. SWA(-) represents sliding window attention (SWA) with
window size Wiocal. This reduces computational complexity from O(T²) ) to O(T Wlocal).
SwiGLU(.) the gated activation function SwiGLU(z) Swish(2WW ) (xW2), where W1, W22 Id ddy are learned projection matrices, du is the feed-forward hidden dimension, Swish(x) = a (x) with a()) being the sigmoid function, and @ denotes element-wist multiplication.
Canon Layer. Canon layer are introduced in Allen-Zhu (2025) to foster the token mixing:
Camon(h:) (h:) wo
+ w1 (L) + w2 h2 + Wz 4
(1)
(10)
where W/ R.dues are learned gating vectors. They are basically causal _convld with kernel size 4, so highly efficient CUDA operator supported.
Why SWA + Canon Layer for Token Mixing. Theoretically we use SWA along, given a se- quence length T and window size Whoal,, we will need at least T encoder layers ensure every byte position attend to every other. This would necessitate very deep local encoder for long sequences, increasing computational cost and potentially hindering training stability. The canon layer instead an efficient addition, it introduces negligible parameter overhead and benefits from highly optimized implementations.
Wheal
3.2 DOWNSAMPLING: CODING- RATE CHUNKING
The chunker then determines which byte positions to promote to the next hierarchical level by eval- uating the coding rate of contextualized representations. This approach is grounded in information theory: positions with high coding rates contain more information and should be preserved as chunk boundaries, while positions with low coding rates can be safely compressed away.
Lossy Coding Rate in Representation Space. Let the local encoder produce contextualized representations M xduat The lossy coding rate (Cover, 1999; Ma al., 2007) for
We provide theoretical derivation in Appendix A and fast approximation Appendix B.
4

---

Published as a conference paper at ICLR 2026
h1:1 Txdee is:
Rg (R(.)))) log det (1+
dhocal g²
b1:hir:))
(11)
where z² is noise variance parameter that controls the sensitivity of the coding rate computation.
Re (hirr) ) is large when the representation hir has large eigenvalues and spans diverse directions in the representation space, indicating high information position that warrants preservation.
Streaming Decision. Let the local encoder produce contextualized representations h I 22'xd The marginal coding rate at position & measures the information gain from including (-th byte:
AR. = (h1a) - R& (h1:r-1)
(12)
AR, is large when position introduces large information gain, indicating natural segmentation boundary. Given the target global sequence length K, the chunking procedure begins by com- puting marginal coding rates &R, for all positions positions with S {1) to always include the BOS token, then identify the (K 1) positions with the largest A R values. Finally, we sort these selected positions chronologically to obtain S {51, 82, $2...... ................................ 5K} where K uses the full-sequence importance profile to allocate global compute, but causal masks ensure predictions never access future byte content.
(....... We initialize the selected
1 and $2 <... 3K. During teacher- forced training, Top-
After selecting K positions", we extract the corresponding representations h hea......................... hag E R K xdual and map them to the global representation space: 21:K [h hear. her W proj R xddae where W pruj E Rodo Xdgad is the projection matrix.
Why Not Global Threshold? Instead of using a global information threshold for chunking, select the Top-K positions with the highest information gain for two reasons. First, determining an appropriate global threshold is non-trivial: it often requires extensive empirical tuning and results in a "magic number" that is difficult to interpret or generalize. Second, a fixed threshold leads to dynamic chunks for different inputs. This variability in the global sequence length breaks the static computation graph. While specialized CUDA operators used in (Hwang al., 2025a) can manage dynamic graphs, they introduce other issues like variable memory allocation per input, which casily got into OOM issue with some unlucky batch. Fixed-length Top-K also preserves static computation graph, enabling consistent memory allocation and avoiding ragged tensors that complicate GPU batching.
3.3 GLOBAL TRANSFORMER: DEEP MODELING FOR HIGH-LEVEL ABSTRACTION
The global transformer operates on compressed representations 21:K E R xddua using full causal attention. Since K T, we employ deep (G layers) and wide (dglobal discat) architecture that concentrates computational budget on high-level reasoning:
91:K Transformerggaaal (2 (21:K), FLOPs O(G K² K2. &g
(13)
The quadratic attention complexity O(K2) remains tractable due to compression, while the large hidden dimension dglobul and depth G enable sophisticated modeling of long-range dependencies and abstract patterns.
3.4 UPSAMPLING: MULTI-LINEAR RECONSTRUCTION WITH LARGE RESIDUAL
Given processed global representations 91:k and selected positions S {s1, 8K). recon- struct full-length representations using position-specific transformations:
In this work. we focus on selecting specific byte positions to promote to the next level. rather than using mean pooling within the chunk, as prior work has found that different pooling operations yield nearly identical nerformance (Pagnoni et al., 2024; Videau al., 2025; Hwang al., 2025a).
5

---

Published as a conference paper at ICLR 2026
chunk(t) = arg max 8, 84 1.,50 ))
(14)
1 T/B
bin(t)
B
(15)
Jchunk(s) W Win(((): Whin(t) (Wi.-.. WB).
(16)
81 h4 + St-
(17)
where we share upsampling parameters across B bins (default B = 16), making the overhead negligible while matching per-position performance.
3.5 DECODER: SYMMETRIC ARCHITECTURE FOR NEXT BYTE PREDICTION
The decoder uses identical architecture to the local encoder (sliding window attention + Canon layers) operating upsampled representations 81:77
p softmax oofmaaatraasoormmercaaddd Wou).
(18)
where W E Re projects to byte vocabulary. The symmetric encoder-decoder design en- sures consistent processing while the global transformer concentrates computational resources on high-level modeling.
EXPERIMENTS
We follow standard pre-training setup at academic scale (Yang al., 2024; Allen-Zhu 2025) where ablations are done with matched FLOPPs at the GPT-3 Large level and scaling experiments are run GPT-3 XL seale. Training details are provided Appendix C.2.
4.1 EXPERIMENTAL SETUP
Pretraining Dataset. All models are trained from scratch on the Kineweb-Edu-11 (Penedo al., 2024) corpus, curated pre-training dataset of educational content comprising approximately 500B training byte tokens.
Bits-Per-Byte Estimation. We adopt the Bits-Per-Byte (BPB) metric following established prac- tices in recent literature (Xue et al., 2022b; Yu et al., 2023a; Wang et al., 2024). BPB normalizes cross-entropy loss by byte count rather than token count:
BPB(x) LCE(X)
(19)
In(2) -Pbyuss
where LCE(x) is the cross-entropy loss over data x and rhytes is the total bytes in
focus primarily BPB loss and lected zero-shot downstream tasks from the Im-eval-Ma leeevaalharree rness (Gao et al., 2024) (e.g.. HEL-- LASWAG (Zellers al., 2019), WinoGrande (Sakaguchi al., 2019), BOOLQ (Clark et al., 2019), PIQA (Bisk et al., 2020), ARC (Clark et al., 2018)) for the ByteFlow Net runs. The baseline decoder-only transformer variant is validated held-out F FineWeb Edu split every 1000 steps.
Downstream Tasks. Due the scale of pretraining,
4.2 BASELINES
We compare against several representative architectures:
Standard Transformer: LLaMA (Touvron et al., 2023; Dubey et al., 2024), trained with fixed BPE tokenizer. This serves the strong tokenized baseline.

---

Published as a conference paper at ICLR 2026
600M Model on 50B Training Byte Tokens
Model on 500B Training Byte Tokens
Liama 0.68
Liama 1.38
Al--wt 0.68
Al--aat 1.38
Spacetlyte 0,08
Specetlyte 1.30
1.2
Bytes
LismaByte 0.68
LiamaByte 1.38
MambaByte 0.68
MambaByta 1.38
ByteFlow Net 88 (Ours)
Byteflow Net (Ours)
Validation Bits-per-byte
Validation Bits- tiss-pe-byyte
1.1
ByteFlow Net 0.68
ByteFlow Net achieves best scaling performance
overtakes Liama 0.68
0.85
0.80
0.8
08
108
508
1008
1508
2006
2508
3008
Total Training Bytes
Total Training Bytes
Figure 2: Scaling Trend for Different Architecture Comparison. Validation BPB loss (lower is better) for different architecture approaches on two different scale (600M, left) and (1.3B, right) models. ByteFlow Net achieves better performance with scaling to larger models and data recipe.
Table I: Zero-shot performance comparison across multiple benchmarks. Evaluation results on six downstream tasks at both 0.6B (50B tokens) and 1.3B (500B tokens) scales. We report average scores over three separate runs to ensure fair comparison.
Accuracy (1)
Model
Tokenizer
HellaSwag WinoGrande
BeolQ
PIQA
ARC-c
Average
600M Models Trained 50B Tokens (Ix Chincilla Ratio (Hoffimann
2022))
LLaMA (Dubey et al., 2024)
BPE
43.12:0.87
42,74
62,2620.64
59.43
61.38-0.98
25.95
4
LlamaByte (Dubey et 2024)
37.93
41.8420.50
61.15
58.3
58.31
60.24_1.68
28.18 52
47.
47.44.1.29
MambaByte (Wang al., 2024)
38.21
41.97=1.66
61.48=1.14
58.67+0.ss
60.53_1.87
47.71 25.42+1.00
20.80
SpuceByte (Slagle, 2024b)
Byte
37.76
42.15
61.04:1.0
58.18
60.12_0.a5
25.05
47.38
AU-Net (Videau al., 2025)
40.34
44.12
63.85 1om
64.87
62.91
1..8
20
49.38 21.92
Floe Net (Ours)
41.42+1.5
44.932077
64.48
62.25
63.87
13
28.36
50.89
1.3 Models Trained on 5008 Tokens (4x Chincilla Ratio (Holfmann et al., 2022))
LLaMA (Dubey et al., 2024)
BPE
$4.12
53.74
73.26
70.43 72.38_1.54
36.95
60.1551.00
LlamaByte (Dubey et al., 2024)
48.9 18.93
52.84
72.15
69.31
71.24_1.48
36.18 16.1821.7
1.64
MambaByte (Wang et al., 2024)
49,21 11.2
52.97
12.48
69.67
71.53_1.78
36.42
:8.71
SpaceByte (Slagle, 2024b)
Byte
48.76-1.64
53.15
72.04=1.5a
69.18
71.12_1.00
36.05
58.
58.38
AU-Net (Videmu et al., 2025)
50.34g1.51
14.12.21.45
73.85
2..8
74.87
72.91
..59
60.59 37.43
ByteFlow Net (Ours)
g1.04
56.93_1.6
76.48
1:38 24.25
75.87
21448
40.36 63.19 21.57
Byte-level isotropic models: LlamaByte (pure Llama layers byte-level modeling) and MambaByte (Wang et al., 2024) process raw UTF-8 bytes without hierarchy.
Hcuristic chunkers: SpaceByte (Slagle, 2024b) and AU-Net (Videau et al., 2025) uses whitespace-like delimiters for chunking.
ByteFlow Net: Our proposed architecture, where chunk boundaries are chosen online via the lossy coding-rate criterion (section 3).
Byte/BPE models are trained on sequence lengths of 8192/2048 respectively, and for ByteFlow Net and AU-Net we use hierarchical sequence lengths (8192 + 3200 + 8192). All detailed model configurations are provided in Appendix and further abaltion in Appendix for reference. 4.3 SCALING EXPERIMENTS
Superior Scaling Behavior.
The scaling curves in Figure 2 reveal encouraging trends for Byte- Flow Net across both model sizes. At the 600M parameter scale, ByteFlow Net demonstrates steady improvement throughout training, eventually surpassing the LLaMA baseline around the 25B to- ken mark and maintaining this advantage through 50B tokens. The 1.3B results show even more promising behavior, with ByteFlow Net exhibiting the most favorable scaling trajectory among all
7

---

Published as a conference paper at ICLR 2026
tested architectures, suggesting that approach becomes increasingly effective scale up both model size and training data.
Competitive Performance on Downstream Tasks, Our performance results Table demon- strate that ByteFlow Net achieves competitive results with traditional tokenization approaches while operating directly on raw bytes. the 600M scale, ByteFlow Net reaches 50. 89% average accuracy compared to LLaMA's 49. 15%, representing modest but consistent improvement of 1.74 points. The gap becomes more substantial at 1.3B pa- rameters that suggests the benefits of our ap- proach become more pronounced with scale compared to LLaMA baseline.
Table 2:
Performance on character-level bench- mark (Edman et al., 2024). Basseline results are taken from Pagnoni al. (2024).
ByteFlow
Character-level Performance. As shown in Table 2 ByteFlow Net 1.3B substantially out-
(IT tokens)
(16T tokens)
(5008 tokens)
CUTE
27.5
20.0
51.2
-oontains
52.8.0.2
performs Llama 3 variants on CUTE despite
Contains Word
55.1
21.6
70.1
20-32x less training data, with exceptional orthographic capabilities evidenced by near- perfect Spelling Inverse performance. This
Del Char
34.6
34.3
13.2
Del Word
75.5
84.5
734298
Ins Char
7.5
0.0
16.9 114
Ins Word
33,5
63.3
28.2.9.3
Spelling Inverse
95.1=24
demonstrates that architectural design can com- pensate for scale in character-level tasks.
Substitute Char
0.1
1.2
13.3
Substiute Word
16.4
68.
Swap Char
10.1215
Training-time efficiency profiling. We pro-
file controlled runs on 8 with matched FLOPs budgets during pretraining runs.
Table 3 shows ByteFlow Net attains strong dfficieney-peffommacc balance: it trains com- petitively among hierarchical byte models while achieving the best BPB and downstream accuracy.
Table 3: Training-time efficiency comparison at 0.6B scale (50B tokens). WPS words/sec x x0
Model
FLOPS (x
WPS?
Iter(s)]
Val BPB.
LLaMA (BPE)
9.3
3.8
0.89
4.4 ABLATION STUDY:
AU-Net
8.8
Cosine chunking
7.3
3.8
THE ART OF DECIDING WHERE CHUNK
ByteFlow (lag-den det)
1,077
7.9
4.0
0.86
ByteFlow (1. approx..)
1.01
8.5
3.9
0.87
To truly understand what makes a tokenizer- free model tick, we have isolate the most crit-
ical decision it makes: where to draw the line between chunks. This is often a messy comparison, as different architectures are bundled with their own unique chunking logics. To cut through the noise, we ran a controlled experiment: we took the ByteFlow Net architecture and swapped out its chunk- ing module with seven different strategies in Table 4. All ablation experiments were conducted at the 0.6B parameter scale on 50B training tokens.
The Effect of Heuristic-based Chunking. A crucial negative control reveals that randomly choos- ing chunk boundaries is disaster. It shatters any hope of learning, leading to the worst performance by a wide margin with a 41.34% task accuracy. This proves that the hierarchy itself isn't magic: the segmentation must be meaningful. This makes the performance of simple word-boundary chunking static, rule-based tuation doesn't just work; it match the standard LLaMA baseline downstream tasks (49. 49.38% vs. 49.15%) This powerful insight shows that a linguistically-ats aware segmentation can be somtimes more effective than sophisticated but less effective dynamic chunking like entropy or cosine-based.
all the more remarkable.
just splitting on spaces and punc-
The Advantage of Coding Rate Segmentation. While other dynamic methods, like those based on neural predictions or cosine similarity, show promise, they struggle to consistently beat the simple word boundary baseline. This highlights critical challenge: learning to find meaningful boundaries on the fly is hard. This is where our approach is. By framing the decision as a matter of compres- sion, our lossy coding-rate method outperforms all contenders in this scale. It achieves the lowest validation BPB loss at 0.86 and the highest average task accuracy at 50.89%, a significant leap over the next-best strategy. This victory suggests that the optimal way to segment sequence isn based on what looks similar or what's 's locally surprising, but on what provides the most new information to the sequence whole, and teach model to compress the input itself during optimization.
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation Different Chunking Strategies for Hierarchical Language Models. We train on ByteFlow Net but ablate on different chunker used in different architecture. Experiments are done on 0.6B on 50B training token scale. We report average scores over three separate runs.
Method
Type
Formulation
Complexity
Validation BPB Loss (
Task Perf. (t)
LLaMA Baseline
0.89 in
49.15 io.ma
Fixed Stride (Yu al. 2023a)
Statie
8 W.
0(1)
0.96200122
45.27
Word Boundaries (Slagle, 2024bb
Static
8 = (f:R) (gacc,,pucct)
O(T)
0.94gguuss
49.3 49.38
Random Chunking
Dynamic
Plboundary t)
O(T)
1.04
41.34 41.34
aib, Waznll Gumbel(pi)
Boundary (Nawrot
Dynamic
O(T d)
0.90
17.13
He = Σ. Pte log P(elhe) Top K((H))))))
Entropy Chunking (Pagnoni al., 2024)
Dynamic
OJT
0.91 10007
47.81
Similarity (lEwang et 2025a)
Dynamic
simi 1
OCT
0.92_p000e
47.45.1.00 100
8 T(p-K(() (1)
sim,
(h (. 5 Top-K( (AR [i
Lona Coding Rate
Dynamic
O(T d)
0.86
50.89 20
Chunking Strategy Impact on Latent Representation Manifolds
Original Manifold (Before Chunking)
Random Chunking
Neural Boundary
1-SNE 2
I-SNE?
I-SNE
t-SNE
1-SNE
I-SNE
Entropy Chunking
Cosine Similarity Chunking
Coding Rate Chunking (Ours)
42
22
t-SNE22
1-SNE
L-SNE2
-22
-20
20 40
-20 -2 110 10
20
-40
20
45
t-SNE1
1-SNE
t-SNE 1
Figure 3: Chunking Strategy Impact on Latent Representation Manifolds. Each point a contextualized byte representation after the local encoder (after IB training bytes), projected to 2D by t-SNE. Poor chunking (random, neural boundaries) fragments the original clustering, whereas coding-rate chunking preserves it.
Why does coding rate work so well? We hypothesize it's about two things: geometry and adaptability. As visualized in Fig- 3. poor chunking strategies like random selection effectively shatter the underlying structure of the data in the representation space, leaving the model to learn from a fragmented mess. Our coding- rate approach, in contrast, excels preserving a coherent latent manifold, making far easier for the global transformer to identify patterns. This links directly to the idea of dynamically assigning FLOPS. The coding rate criterion essentially an importance detector. By only promoting bytes with high information gain to the global level, the model is forced to spend its precious compu- tational budget on the parts of the sequence that actually matter. It learns to focus its deep, wide global transformer on compressed stream of significant events, rather than wasting resources on redundant or predictable byte patterns. As shown in our case study (Figure 4), the model learns to assign higher rates to semantically significant bytes (e.g., key nouns), forcing the model to focus its computational budget on a compressed stream of meaningful information rather than redundant patterns. This strategic allocation makes processing more efficient and effective.
Preserving Latent Manifolds and Dynamically Allocating FLOPs.

---

Published as a conference paper at ICLR 2026
Character-Level Coding Rate Scores
16
High Coding Rate (H0.7)
Nartun Coding Pite (6.4-0.7)
Coding Rate
Codiing (40
Character Position
Figure 4: Case Study of Character-Level Coding Rate Scores. This figure illustrates how Byte- Flow Net assigns an information- theioottttthhoticiccooo "importance" score to each character in an example sen- tence. The model has learned to assign a higher coding rate to characters that are more semantically significant, such as the initial letters of words and key entities. Conversely, it assigns lower rates to more predictable characters within words. This demonstrates the model's ability to dynamically identify information-rice points in the byte stream to guide its chunking and resource allocation.
5 CONCLUSION
This work introduced ByteFlow Net, hierarchical architecture that learns to parse raw data on its own terms. Grounded information theory, our model reframes segmentation as dynamic compression task, using coding-rate objective to intelligently identify meaningful semantic units without a fixed vocabulary. This principled approach is not merely theoretical; extensive experi- ments show that ByteFlow Net consistently outperforms strong BPE-based transformers and other byte-level models, cxhibiting a superior scaling trajectory as model size increases. Crucially, our ab- lation studies confirmed that the coding-rate criterion is the key to this success, decisively surpassing other dynamic chunking strategies by preserving the underlying geometry of the data's 's latent man- ifold. This allows the model to strategically allocate its computational budget, focusing its most powerful components on a compressed stream of what is truly informative. Our results therefore provide compelling evidence that end-to-end, tokenizer-free modeling is not only feasible but is more effective and robust paradigm for language modeling.
ETHICS STATEMENT
This work does not involve human subjects, personally identifiable information, or sensitive data. All experiments are conducted on publicly available and curated datasets (e.g.. FineWWb-Edd- 100B (Penedo et al., 2024)) that have been filtered to minimize risks of privacy violations or ex- posure of harmful content. Our research focuses architectural design for tokenizer-fie free language modeling and does not aim to produce harmful applications. We are mindful of potential misuse of language models, including risks related to bias, misinformation, malicious generation, and en- courage responsible downstream use in line with the ICLR Code Ethics. No conflicts of interest external sponsorships influence this work.
REPRODUCIBILITY STATEMENT.
We have taken multiple steps to ensure the reproducibility of our work. The architecture of Byte- Flow Net, including all encoder, chunking, and global transformer components, described in detail in Section 3, with ablation studies and comparisons provided in Section 4. Implementation details such as model sizes, FLOPs-matched training recipes, optimizer settings, and hyperparameters are included in Appendix C, while theoretical derivations of the coding-rate objective and its approxima- tions are provided in Appendix A and B. All datasets used in our experiments are publicly available; rely the FineWee-Edu 100B corpus, and document the preprocessing and filtering pro- cedures in Appendix C to support replication of data pipelines. We also provide extensive ablation studies in Section 4 and Figure 3 to demonstrate robustness of our results.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Zeyuan Allen-Zhu. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. SSRN Electronic Journal, May 2025, https: /sss.. com/ abstract= 5240330.
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine trans- lation, 2018. URL / http///bb///1..1111111 02173.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Christopher Clark, Kenton Lee, Ming-Wat Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1wXiv19905.10044. 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the al2 reasoning challenge. arXiv preprint arXiv: arXiv:/803.088. 1803.0557, 2018.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and etc. Deepseek- rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL https: //arxiv. org/ hbs//bb//250000 12948.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. mnXi-2247, 2024.
Lukas Edman, Helmut Schmid, and Alexander Fraser. CUTE: Measuring LLMs' understand- ing of their tokens. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 3017-3026, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 111.6653//11222.mmmppppp maw.mmmmmmmmi.1.... URL https: htt///aaaanntttoogoyoooo or2/2024. emnlp-ma n 177/.
Matthias Gallé. Investigating the effectiveness of BPE: The power shorter sequences. Ken- taro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence Natural Language Processing (EMNLP-IJCNLP,, pp. 1375-1381, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D199111 URL https: httts:://aclanthee wwwp://-gg////011100001
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopí, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https:/ zenodo. wwwpg///werr///eoeeee 602006602
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, An- gela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravanku- mar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern Char- lotte Caucheteux, Chaya Nayak, Chloe Bi, and etc. The llama 3 herd of models, 2024. URL https://arrt org. 2407 21783.

---

Published as a conference paper at ICLR 2026
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. In Proceedings of the 36th International Conference on Newral Information Processing Systems, pp. 30016-30030, 2022.
Sukjun Hwang, Brandon Wang, and Albert Gu. Dynamic chunking for end-to-end hierarchical sequence modeling. arXiv preprint arXiv:2507.07955 2025a.
Sukjun Hwang, Brandon Wang. and Albert Gu. Dynamic chunking for end-to-end hierarchical sequence modeling, 2025b. URL https: / wwwp:vvoorr/aoo htt.aa///52..0 2500 070000779555
Julie Kallini, Shikhar Murty, Christopher D Manning, Christopher Potts, and Róbert Csordás. Mr15: Dynamic token merging for efficient byte-level language models. In The Thirteenth International Conference on Learning Representations.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, November 2018. Association for Com- putational Linguistics. doi: 1..1866//1//11-200 URL https: aal.atoooooooooooooooo.ooooooooa orgg D18 20222/
Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, and Yejin Choi. SuperBPE: Space travel for language models. In Second Conference on Language Modeling, 2025. URL httpht//s//ooo htttps:://pperreeeeee net/ fowum?id= feNP.
Yi Ma, Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE transactions on pattern analysis and machine intelligence, 2909):154-11562, 2007.
Piotr Nawrot, Szymon Tworkowski, Michal Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, efficient language models, 2022.
and Henryk Michalewski. Hierarchical transformers URL https::pp/// /ttp:////xivvvvv abs 211. 13711.
Piotr Nawrot, Jan Chorowski, Adrian Lancucki, and Edoardo Maria Ponti. Efficient transformers with dynamic token pooling, In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6403-6417, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. (0.1886) 11111116///22...coo acl.o223.l- 2023. aclnlong.....o..... 353/.
URL https: htaas///taaaanoogogo..ooo. org/
Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, et al. Byte latent transformer: Patches scale better than tokens. arXiv preprint arXiv:2412.09877, 2024.
Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Mar- garet Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srini Iyer. Byte latent transformer: Patches scale better than kens. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 9238 9258, Vienna, Austria, July 2025. Association for Com- putational Linguistics. ISBN 979-8- 89176-25-01. doi: 10. 10.18653/v1/200 acl- ael-long.455 URL https: /aaaaaaaaantooooooeee org/ wttp://2.........n.ao 453/.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale, 2024.
Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages, 2023. URL https: / arxiv. org/ abs 2305. 15425.
12

---

Published as a conference paper at ICLR 2026
Phillip Rust, Jonas Pfeiffer, Ivan Vulie, Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. ArXiv, abs/2012.15613, 229924220.
2020. URL https: http:/ppp..semantssccccol.l.aaaa
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale, 2019. URL httttpp://aaxii.orrreee wwt..oggg////o.... wttpaaawwww.../b//////a.//./
10641.
Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pin- ter, and Chris Tanner. Tokenization is more than compression. In Yaser Al-Onaizan, Mo- hit Bansal, and Yun-Nung Chen (eds.). ProceeDo All Languages Cost the Same? Tokeniza- tion in the Era of Commercial Language Modelsdings of the 2024 Conference on Empiri- cul Methods in Natural Language Processing, pp. 678-702, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v 12045////22222..mmmmmmmam... URL https: aa/accaaaa wewlannnnhoogyyyy org/2024 emnlp-ma inmain..e. 40/.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. 2015.
Kevin Slagle. Spacebyte: Towards deleting tokenization from large language mod- eling. In A. Gloherson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tom- czak, and C. Zhang (eds.). Advances in Neural Information Processing Sys- tems, volume 37, pp. 124925-124950. Curran Associates, Inc., 2024a.
URL
https: http:sss//ppoceeings..eess oat.:paaa11111---pp--oo--eeeeee- wwwaappe--Coffeenceepppoeo
Kevin Slagle. Spacebyte: Towards deleting tokenization from large language modeling. Advances in Neural Information Processing Systems, 37:124925-124950, 2024b.
Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. Ady-bert: Bert is not robust on misspellings! generating nature adversarial samples on bert, 2020.
URL https:tp////mm. /trr///axx /aaxiviiivv org/ abs 20oss/2000 2003. 04985.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvine, Michelle Casbon, Etienne Pot, Ivo Penchev, and etc. Gemma technical report, 2025. URL /ttpps///axiiv. org/abs 2503. 19786.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0m 00.8888..988 2023.
Mathurin Videau, Badr Youbi Idrissi, Alessandro Leite, Mare Schoenauer, Olivier Teytaud, and David Lopez-Paz. From bytes to ideas: Language modeling with autoregressive u-nets, 2025.
URL https:ttp/.......... http://axvvvvooo org/ abs/ 6.0. 14761.
Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. Mambabyte: Token-free selective state space model. arXiv preprint arXiv:2401.1366 2024.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang. Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte the models. Transactions of the Association for Computational Linguistics, 10:291-306. 2022a. doi:
10.1162/taci a 00461. URL https: //aaaaaaaannntooooyoooo orgg/022 tac1-1.17/
Linting Xue, Aditya Barua, Noah Constant, Rami AI-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards token-free future with pre-trained byte-to-byte mod- cls, 2022b. URL https: httttss//axxxvvorgg.... /ttp///ixv..robb///////
13626.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv. Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, and etc. Qwen3 technical report, 2025, URL https: /ttp///aavvv..////////..//../..a abs/2505. 09388.
13

---

Published as a conference paper at ICLR 2026
Songlin Yang, Bailin Wang. Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware- training. In Proceedings of the 41st International Conference Machine Learning, pp. 56501-56523, 2024.
Gilad Yehudai, Haim Kaplan, Asma Ghandeharioun, Mor Geva, and Amir Globerson. When can 001.///47771166000
transformers count to n?, 2024. URL https:
Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. Advances Neural Information Processing Systems, 36.78808-78823, 2023a.
Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers, 2023b. URL https:://rxee org/
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv:1955007830, 2019.
Xiang Zhang, Juntai Cao, and Chenyu You. Counting ability of large language models and impact of tokenization, 2024. URL www.hhtts//ccectt /
orgoorgggoo /tabs//411 19730.
Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Tim Vieira, Mrinmaya Sachan, and Ryan Cotterell. A formal perspective byte-pair encoding. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 598-614, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10. 10.18653//1/200 1/2023 findings-acl.333 acl.38. findings acl. 38/.
URL https: : / molannhhloogy org/2023.
14

---

Published as a conference paper at ICLR 2026
A DERIVATION OF THE LOSSY CODING RATE FORMULA
Consider sequence of contextualized representations h1r E R xdeat produced by a local encoder. We seek to determine the minimum rate required encode this sequence with specified distortion level using rate-dish diseorsttotionn theory. Let X h1rr be our source sequence and X be the reconstructed sequence after lossy compression, with distortion defined as D [i|| X X || the Frobenius norm.
where ||-11 denotes
We model the representations following multivariate Gaussian distribution, which is reasonable for deep neural network representations. Specifically, vec(h).r) N Σ) where vec(.) vectorizes the matrix and Σ E Te T is the covariance matrix. For local representations, we assume structured covariance Σ Ir @ I! where H h1- E R. Xcheaa is the empirical covariance T matrix, @ is the Kronecker product, and IT is the x T identity matrix.
For multivariate Gaussian source with covariance matrix Σ, the rate-distortion function with mean squared error distortion is:
Σ max 0, //)
A
R(D) 2 1=1
(20)
where X the eigenvalues of Σ, 0 satisfies E min(A;, 0) D, and n Tehecl is the total dimensionality.. Instead of specifying distortion directly, parametrize using noise variance =², corresponding to adding Gaussian noise with variance 5² during reconstruction, giving E?.
dooa Given our covariance structure, the eigenvalues 2 are {A {Us [T j=1 (each repeated T times), where {11} are eigenvalues of H MheT. Substituting into the rate-distortion formula:
dscal
T max (0. /5/T E2
R(=²)
(21)
5=1
dseal
Σ max
Hg 1,
(22)
2
E2
Using the identity max(0, log(z)) log log(loax (Iaxx (1, x)) and the fact that for a matrix A with eigenval- ues {Wy}, we have II, max(1, / det Mot(max(I. A/E?)), obtain:
dince
R(=²)
log
max (I,
H4
(23)
j=1
(T, (
log del max
(24)
Through matrix algebraic manipulation and using the fact that we can rewrite the determinant in terms of the original representation matrix, we arrive at the final form:
Rg (h1.T) = log det (1 drocal
1
(1+
-2 hiThh
(25)
This lossy coding rate quantifies the minimum bits needed to encode sequence h1.7 with reconstruc- tion error approximately z² per component. The determinant captures the effective dimensionality of the representation quiring more bits for preservation, while the noise variance parameter e? controls the sensitivity of the coding rate computation.
panc--large eigenvalues of h1: indicate high-information directions re-
B L2 NORM APPROXIMATION FOR LOSSY CODING RATE
We derive a computationally efficient approximation to the lossy coding rate formula in equation (11) for streaming applications where quick local decisions required. Starting from the exact
15

---

Published as a conference paper at ICLR 2026
formula:
1 Re (e(:T) log det
diocal [+
hir1 h
(26)
E²
Let A = deg her E RTXT be the matrix inside the determinant. For moderate noise variance e² relative to the representation magnitudes, are not extremely large, allowing us to use the matrix logarithm expansion.
consider the regime where the eigenvalues of A
Using the matrix identity log (I + A) tr(log(7 + A)) and the Taylor series expansion of the
matrix logarithm for |||||
1²
A²
log + A) =
+
(27)
2
For the first-order approximation when has moderate eigenvalues, we retain only the linear term:
(chccal
log deg tr(A)=
T (=2 hirhir
(28)
Using the cyclic property of trace, tr = tr(BA)):
duos
tr( tr(hiriir) -r((h
Σ
h 111.1
(29)
where || || denotes the Frohenius norm.
Substituting this result back into our approximation:
diocal e2
Re(hi:) ~
(30)
For streaming decisions where we need quick estimate proportional to the information content, we can absorb the constant factors into scaling parameter and use:
Re (birr) xx [| l1
(31)
Since the Frobenius norm is equivalent to the L2 norm for matrices (treating the matrix flattened vector), we have ||لير = ||/1. 12, giving us the final approximation:
Re (hi:r) = C ||
(32)
where C' = dusgl is constant determined by the local dimensionality and noise parameter.
For practical streaming implementations, this quadratic relationship can be further simplified to linear approximation Rg (h1.)) x ||4,.l|| when making relative comparisons between different representations, as the monotonic relationship preserved and computational cost is minimized.
Validity Conditions: This approximation is most accurate when (1) the noise variance e² is suffi- ciently large relative to diocal | hh.r I| such that the eigenvalues A are moderate, (2) the represen- tations h do not have extreme condition numbers that would make the trace approximation poor, and (3) we are primarily interested in relative rankings rather than absolute coding rates.
C MODEL CONFIGURATION
C.1 OVERVIEW
We conduct a comprehensive evaluation across six distinct model architectures at two different scales (600M and 1.3B parameters), resulting total model configurations. Our experimen- tal framework compares traditional transformer baselines with state-of-the-art byte-level process- ing architectures and advanced hierarchical chunking-aware models. The model families include: (1) Llama standard transformers with token-level processing, (2) LlamaByte byte-level vari- ants of standard transformers, (3) MambaByte selective state space models with byte processing, (4) SpaceByte optimized byte-level transformers, (5) AuNet hierarchical models with regex rate-distortion chunking, and (6) BFlowNet advanced hierarchical architectures with sophisticated chunking strategies.
16

---

Published as a conference paper at ICLR 2026
Table 5: Comprehensive Model Architecture Specifications Across Six Model Families and Two
Scales,
Model Family
Seale
Architecture
Layers
Tokenization
Chunking
Seq
Llama
Standard Trunsiemnct
Stundard Trursfornct
16
TacToken
TacToken
LlamaByte
GI0M
1.3
Standadd Transfennct
Sundard Tranfcemrrr
25
2048
1024
16
Byte-level
Byteslevel
None
8192
MamhaBate
600M
1.3
Selective SSM
Selective SSM
24
2048
1024
N/A
Byte-level
Byte-level
None
None
R192
8192
Spucellyte
600M
lienmiiica (2 level)
25
1024
16
Byte-level
Woed Boundary
8192
1.38
Herranitical (2 (2-level)
25
2048
16
Byte-lovel
Wood Boundary
8192
AnNet
600M
1.38
Hicrarltical (2-kevel)
Herrncttca (2-level)
16. 201
221
[512. 1536]
1768, 2040
Mutti-lovel
Mutti-lvvl
Byte-lovel
Byte-lovel
Wood Boundary
Wood Boundary
8192 3200
8192 3200
8192
8192
BFlowNee
600M
1:3
Harnncical (2-kewe))
Hacrarelical (2-Jevel)
[6, 24]
16, 201
1512. 1536
(512, 2048]
Mutii-lvel
Multi-lvel
Byte-kvvll
Byte-level
Coding Rate Chunking
Ooding Rate Chunking
8192 3200 8192
8192 1200 - 8192
C.2 MODEL ARCHITECTURE SPECIFICATIONS
The architectural specifications presented in Table 5 reveal systematic exploration of scaling strate- gies and design paradigms across six model families. Most families follow a consistent scaling ap- proach, offering both 600M and 1.3B parameter versions with hidden dimensions doubling from 1024 to 2048, suggesting these represent standard benchmarks for architectural comparison. The models span three distinct paradigms: traditional Standard Transformers (Llama, LlamaByte) with 25 layers and 16 attention heads, Selective State Space Models (MambaByte) that eliminate at- tention mechanisms entirely while using 24 layers, and Hierarchical models (SpaceByte, AuNet, BelowNet) featuring complex 2-level architectures with varying layer distributions and multi-level attention head configurations.
C.3 DETAILED ARCHITECTURE ANALYSIS
C.3.1 BASELINE TRANSFORMERS
Our analysis begins with two baseline transformer architectures. The primary baseline is the canon- ical Llama model, which employs a traditional token-level attention mechanism with standard vocabulary. Its design features Rotary Position Embeddings (RoPE) with 10, 000, standard multi-head self-attention, and RMSNorm applied prior to both the attention and feed-forward net- work layers. The activation function used is SwiGLU. As direct variant, we include the LlamaByte architecture. This model is architecturally identical to Llama but operates directly UTF-8 byte sequences, utilizing a vocabulary of just 256 characters. This approach offers universal language support and eliminates out-of out-of-vocabulle vocabulary issues, though it comes with the challenge of processing significantly longer sequence lengths.
C.3.2 ADVANCED BYTE-LEVEL ARCIITECTURES
Moving beyond standard transformers, we explore architectures specifically optimized for byte-level processing. The MambaByte model leverages selective state-space models (SSMs), which confer a significant efficiency advantage with linear (nn) scaling complexity compared to the quadratic O(n²) complexity of transformers. Its selection mechanism enables input-dependent state transi- tions, allowing it to effectively manage extended context windows of up to 4096 tokens with con- stant memory usage. In contrast, the SpaceByte architecture introduces an entropy-driven approach to byte-level processing. It uses an adaptive chunking strategy to segment sequences based on in- formation boundaries, allowing for dynamic chunk sizes that adapt to content complexity. This intelligent boundary detection, combined with specialized attention patterns, enhances its overall performance and efficiency.
C.3.3 HIERARCHICAL CHUNKING ARCHITECTURES
We also evaluate two-level hierarchical models designed for sophisticated chunking. The AuNet architecture implements multi-resolution processing through dual-level attention with [512, 4096] sliding windows. It integrates a Canon layer with 4-token kernels to improve horizontal information flow and utilizes extended RoPE with 0 500, 000 to capture long-range dependencies. Its
17

---

Published as a conference paper at ICLR 2026
chunking strategy is guided by a regex rate-distortion optimization following wordl: 101 pat- tern. The BFlowNet model refines this hierarchical concept by focusing on optimized information flow. It employs specialized attention patterns for hierarchical propagation and enhanced regex rate-distortion chunking method with adaptive boundaries. Designed for scalability, BelowNet fea- tures optimized layer distributions for different model sizes and seamlessly integrates the Canon layer for local context enhancement.
C.4 TRAINING CONFIGURATION FRAMEWORK
C.4.1 UNIFIED OPTIMIZATION PROTOCOL
To ensure a fair comparison, all models were trained under standardized optimization protocol. We employed a learning rate of 4 10 with a cosine annealing schedule. Weight decay was set to either 0.033 or 0.1 depending on the model's scale. Similarly, gradient clipping was configured to either 0.2 or 1.0 based on architectural requirements, and the number of warmup steps was set to 5000 or 10000 appropriate for the model.
C.4.2 DATASET DISTRIBUTION STRATEGY
Our training data was carefully curated and distributed to align with the strengths of each archi- tecture. Models specialized for programming languages were trained on the 10BT Win Code dataset. For broad knowledge coverage, general-purpose models were trained on the FineWeb Ed- ucation dataset, scaled from 10BT to 100BT tokens. To leverage their unique design, byte-level models were trained directly raw byte sequences, thereby avoiding artifacts from sub-word tok- enization. Finally, to properly evaluate their chunking capabilities, hierarchical models were trained extended sequence lengths of 3200 tokens.
C.4.3 INFRASTRUCTURE AND IMPLEMENTATION
The entire training framework was built modern infrastructure stack. We utilized BF16 mixed precision across all architectures and employed Fully Sharded Data Parallel (FSDP) with model- specific optimizations for efficient parallelization. Models compiled with PyTTrrhh 2.0, and selective activation checkpointing was used to manage memory consumption in larger models. For rigorous experimental control, all runs were comprehensively tracked and logged via WandB inte- gration.
C.5 OTHER TRAINING DETAILS
Training Configuration. We train all models for up to 1.95M optimizer steps (ByteFlow Net) or 950K steps (baseline) using AdamW with decay. The peak learning rate is 4 x 10-4, with 10K warmup steps for ByteFlow Net and 5K for the baseline. Gradient clipping is disable TF32 matmuls for reproducibility, and enable torch. compile le to fuse kernels.
0.9, /3 0.95, weight decay 0.1, and cosine LR
0.2 and 1.0, respectively. We use bf16 6 precision throughout,
Distributed Training. All models are trained on NVIDIA A100 80GB GPUs, using Py PyTorch Fully Sharded Data Parallel (FSDP) in full-shard mode. We keep activation checkpointing disabled unless otherwise stated and set tp.size=1 (pure data parallelism). We cache compiled graphs to reduce startup overhead and cap compilation cache size GB.B
Regularization and Stability. All transformer feed-forward blocks use mwltimllleeoo of=225 dimension rounding: rotary position embeddings (RoPE) are applied with 0 5 x 10" for ByteFlow Net and 0 = 10 for the baseline. We schedule À in the dite-iitoo objective target desired compression ratio. Both models apply dropout implicitly residual scaling and optimizer noise.
D ABLATION STUDIES
Understanding the individual contributions of ByteFlow Net's architectural components crucial for validating our design choices and identifying the sources of performance gains. We conduct compre-
18

---

Published as a conference paper at ICLR 2026
Table 6: Zero-shot performance comparison with ablation studies. Evaluation results on six downstream tasks at both 0.6B (50B tokens) and 1.3B (500B tokens) scales, including ablation studies for Canon layer and compression ratios. We report average scores over three separate runs to ensure fair comparison.
Accuracy (1)
Model
Tokenizer
HellaSwag WinoGrande
BoolQ
PIQA
ARCe
ARC-e
Average
600M Models Trained 50B Tokens (Ix Chincilla Ratio (Hoffiman ol. 20222)
LLaMA (Dubey et al., 2024)
BPE
43.12:0.7
42.74
62.26
29.43 61.38-0.98
25.95
49.15
Lya (Dubey et 2024)
37.93
41.8420.58
61.15
58, 18.3
60.24_..06
25.18
47.44
MambaByte (Wang al., 2024)
38.21
41,97=1.5
61.48:1.14
58
58.67
60.53_1.7
25.42+1.00
40.7
SpaceByte (Slagle, 2024b)
Byte
37.76
42.15
61.04:1.9
58.18
60.12
25.05
47.38
AU-Net (Videau al., 2025)
40.34gn.03
44.12
63.85.071
64.87
62.91
1. 27.43
49.38 $1.92
BlteFlow Net (Ours)
41.42++..5
44.9320.78
64.48
62.2540034
63.87
28.36+1
50.89
Ablation Studies Canon Layer (600M, 50B tokens)
ByteFlow Net w/o Canon
Byte
39.782182
43.21
62.15
60.43
61.92.1.41
26.73
49.04
Ablotion Studies Compression Ratio (600M, 50B tokens)
BloeFlow Net 4199)4
Byte 42. 42.15
45.67.0.92
65.32
63. 13.18
64.73_1.28
29.4222.6r
51.74g1.02
ByteFlow Net (Seq=2400) ByteFlow Net (Seq=1600)
Byte
40.87=1.11
44.12
63.75
2.79
53
62.94_1.22
27.58
30.13
Byde
39 39.23
42.78g1.38
61.92.208
29.87
61.15
25.94,2.25
48.48
1.38 Models Trained on 500B Tokens (4x Chincilla Ratio (Hoffman et al., 2022))
LLaMA (Dubey el al., 2024)
BPE
$4.12
13.74
73.26
70.43 72.38_1.54
36.95
60.15
LlamaByte (Dubey et 2024)
48.93
$2.84
72.15
69.31
71.24
36.18
58.44
MambaByte (Wang et al., 2024)
g..2
52.97
12.48
69.67
_1.53_1.7
36.42
58.71
SpaceByle (Slagle, 2024b)
Byte
48.7621.64
53.1521.42
72.04=1.56
69.18
71.12_1.0
36.05
58.38
AL--eet (Videm et al., 2025)
50.34_1.51
54.12=1.45
23.85
74.87
72.91-1.50
37.43+1
60.59
ByteFlow Net (Ours)
55.42
1..
76.48
74.25+161
75.87
40.36
63.19 21.77
Ablation Studies Canon Loyer (1.3B, 500B tokens)
BloeFlow Net w/o Canon
Byte 53. 13.1821.77
74.23 54.85
22.41
73.52_1.73
38.19 os 61.06
Ablation Studles- Compression Ratio (1.3B, 5008 tokens)
ByteFlow Net (Scq=4096)
Byte
26.27
18.14
77.89
75.68
76.94_..a
41.73
64,44:1.41
ByteFlow Net (Seq=2400)
Byu
54.76g1.58
26.42
75.83 1.44
23.91
75.12_1.5
39.68, 21 so
62.62
ByteFlow Net (Seq=1600)
Byte
52.89
54.6721.96
14.15
72.34 73,48_1.79 37.92 14 60.91
hensive ablation studies to isolate the impact of key design decisions: the Canon layer integration for efficient token mixing, and the compression ratio controlled by global sequence length. These stud- ies provide insights into the trade-offs between computational efficiency and model performance, while demonstrating the robustness of our approach across different architectural configurations.
E ABLATION STUDIES
As shown in Table 6, Understanding the individual contributions of ByteFlow Net's architectural components is crucial for validating our design choices and identifying the sources of performance gains. We conduct comprehensive ablation studies to isolate the impact of key design decisions: the Canon layer integration for efficient token mixing, and the compression ratio controlled by global sequence length. These studies provide insights into the trade-offs between computational efficiency and model performance, while demonstrating the robustness of our approach across different archi- tectural configurations.
The Canon layer represents a critical innovation in ByteFlow Net's local processing pipeline, en- abling efficient token mixing through causal convolution operations with minimal computational overhead. Unlike traditional attention mechanisms that scale quadratically, Canon layers provide linear-time token mixing by leveraging optimized CUDA kernels for causal convolution with a 4- token kernel size.
The ablation results demonstrate the significant impact of Canon layers across both model scales, At the 600M parameter scale, removing Canon layers results in a 1.85-point drop in average accuracy (50.89% - 49.04%), with particularly notable degradation in reasoning- intennive tasks like ARC- (28.36% + 26.73%). The performance gap becomes even more pronounced at the 1.3B scale, where the absence of Canon layers leads to a 2. 2.13-point decrease in average accuracy (63.19% 61. 61.06%)
19

---

Published as a conference paper at ICLR 2026
E. CANON LAYER INTEGRATION ANALYSIS
This scaling-dependent performance degradation reveals an important architectural insight: as mod- els grow larger and process longer sequences, the Canon layer's role in facilitating information flow becomes increasingly critical. The layer's ability to efficiently propagate information across posi- tions through its causal convolution mechanism appears to be particularly valuable for maintaining coherent representations in the hicrarchical architecture.
E.2 COMPRESSION RATIO ANALYSIS
The compression ratio in ByteFlow Net's hierarchical architecture directly determines the trade-off between computational efficiency and information preservation. We systematically evaluate differ- ent compression settings by varying the global sequence length from 4096 (2.0x compression) to 1600 (5.12x 2x compression), while maintaining the local sequence length 8192 bytes.
The results reveal an interesting trade-off between computational efficiency and model performance. The lowest compression setting (global seq len 4096) achieves the best performance with 51.74% average accuracy, representing 0.85-point improvement over the default setting (3200). However, this comes at the cost increased computational overhead due to the larger global transformer op- erations. The highest compression setting ( (1600) shows graceful degradation with 48. 48.48% average accuracy, only 2.41 2..i-t drop from the default.
The relatively modest performance degradation even at high compression ratios (5.12x) demon- strates the effectiveness of the nnormatiionthhorree chunking strategy preserving the most criti- cal semantic boundaries. Moving from 4096 to 1600 global sequence length reduces the quadratic attention operations in the global transformer by factor of (4096/1600) /1600)2 6.6, representing substantial computational savings with manageable performance trade-offs.
E.3
DESIGN IMPLICATIONS
The ablation studies collectively validate ByteFlow Net's core design philosophy. The Canon layer analysis demonstrates that efficient local token mixing is crucial for maintaining information flow in compressed representations, while the compression ratio analysis reveals that iitommation-thhoreee chunking criteria can maintain model performance across wide range of compression settings. The consistent improvements from Canon layers and the robust performance across compression ratios demonstrate that principled architectural design can effectively navigate the fundamental trade-offs in tokenizer-free modeling.
F LLM USAGE DISCLOSURE.
In preparing this manuscript, we used large language models solely for polishing the writing (e.g., grammar, readability, and style improvements). No ideas, experiments, analyses, or research contri- butions were generated by LLMs; all conceptual and technical content originated entirely from the authors.
20


Reviews:
[{"review_id": "LX1x1wmqAG", "reviewer": "Reviewer_J3wZ", "summary": "A byte level llm, trained from scratch similar to the byte latent transformer but with a different tokenization boundary detection algorithm. Empirical results seem to support their work.\n\nNote i will assign scores after the discussion, dont take them soo seriously now.", "questions": "I cant say i understand the tsne section can u explain the intuition again?\n\nIs this algorithm hard ware friendly\n\nCan u talk about optimization with such a morel.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Intuition for their work is solid.\n\nStrong abelations", "weaknesses": "I would like to see a more in depth discussion of computational overhead and how this could be improved", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kRD9xWd4dL", "reviewer": "Reviewer_a7hc", "summary": "This paper proposes ByteFlow Net, a tokenizer-free LM that works directly on UTF-8 bytes. Instead of relying on a fixed subword vocabulary, the model learns where to place boundaries by scoring each byte position with an information-theoretic quantity: the approximate lossy coding rate of the current latent representations, and then select the top-K highest information gain positions to a higher-level stream. The architecture has a clear hierarchy: a local byte encoder (SWA + Canon layers) to get contextual byte features, the coding-rate chunker to downsample, a deeper/wider global transformer to do the standard high-level modeling, and a symmetric decoder that brings things back to the byte level. On FineWeb-Edu-100B, the model outperforms both BPE-based LLaMA baselines and several recent byte/hierarchical models.", "questions": "1. Can you give a side-by-side training-time comparison (tokens/s or bytes/s) with a standard BPE transformer (perhaps with sliding window attention), a hierarchical byte model with fixed chunking (stride / whitespace), and another dynamic chunker (e.g. cosine-based)? Right now it’s not clear what we pay in practice for using ByteFlow Net.\n2. Can you report runtime and memory for the coding-rate module itself (forward + top-K selection), and compare it with other methods?\n3. Can you clearify Figure 3? My understanding is: each dot is the contextualized representation of one byte position (after the local encoder), projected to 2-D by t-SNE; different colors correspond to different text segments from FineWeb-Edu; and you plot several such segments together to see whether the chunker keeps the original clustering. If that’s right, can you state it explicitly in the caption, and say how many segments/colors you used and how many points per color (is it the full sequence length T per segment?).\n\n---\n\nMinors:\nLine 189: along -> alone\nLine 097: resolution", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a real bottleneck: static tokenization is non-learnable, language-specific, and fixes the granularity. The proposed method uses coding rate as the segmentation signal, which seems a principled alternative. The connection to rate–distortion is neat.\n2. The hierarchical encoder–decoder is well-motivated and aligned with efficient modeling strategies, and applying SWA + Canon layers provides a realistic path for scaling byte-level models efficiently. \n3. The empirical study shows performance improvement of the proposed method.", "weaknesses": "1. The paper does not report the actual cost of computing the coding-rate scores (the log-det–style term and its approximation). Since this has to run per sequence and per step, training-time overhead and distributed stability (e.g. with variable K, mixed-length batches) should be quantified. \n2. Its practical advantage over existing byte-level or tokenizer-based architectures remains small at current scales (less than 1.3B). Demonstrating competitive performance at more than 7B parameters or on multilingual corpora would strengthen its real-world relevance.\n3. Figure 3 is underspecified. It is supposed to justify the manifold-preserving claim, but from the figure alone it’s hard to tell (i) what each point is, (ii) how many sequences are visualized, and (iii) whether differences are due to t-SNE randomness. As it stands, this is the weakest link in the “why it works” narrative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I7Pp7Gk2oo", "reviewer": "Reviewer_veFt", "summary": "The authors propose a new dynamic chunking method for replacing the BPE tokenizer with a byte-level, hierarchical model. The authors use a lossy code rate metric with top-k over the sequence to select tokens. They show that their segmentation method performs better than the competitors both on bits-per-byte and downstream performance. They also do ablation studies to justify their selection method.", "questions": "- L 104: \"We introduce a new paradigm that replaces static tokenization with dynamic, learned segmentation.\". This is not a new proposal from this paper, there were similar papers before, like Nawrot et al. 2023, cited by the authors. I propose to replace this with one that focuses on the principled chunking approach instead.\n- What is \\Delta in L161? Shouldn't this be V \\in [0, 258)? \n- What is the isolated effect of Canon? While I agree with the fact that SWA is important, I don't see why an additional Canon layer is necessary, or what effect it would have if added to the BPE baseline.\n- In the paragraph \"Why Not Global Threshold\" the OOM issue is easily solvable by adding an emergency hard-top-K that should never be hit during normal operation, but would prevent OOM. This would also prevent the information leakage and the missing autoregression issue.\n- Tab 2: Why are the ByteFlow Net results underlined even when the baseline has higher accuracy?\n- Tab 3: Why are the baseline results underlined when other methods have better performance?\n- Fig 3. is not discussed in the main text, only mentioned in the reproducibility statement. What are the categories shown by the colors, and what is the conclusion from the figure?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Principled method\n- The paper is mostly well written and easy to read (except one part, see below)\n- The method performs better than the baselines", "weaknesses": "- The method relies on top-k over sequence, which (1) leaks minimal information from the future, (2) it is unclear how to apply it in an autoregressive setting. It is unclear how their code rate-based approach can be generalized to an autoregressive setup. Maybe with an auxiliary predictor, like in Mixture-of-Depths [1], or a learned threshold using a PI controller like [2]. The authors do not discuss this limitation anywhere in the paper. \n- While the paper is mostly well written, eq. 15 is unclear. What is the definition of IndexedMatMul exactly? If it is \"IndexedMatMul efficiently applies position-specific linear transformations...\", then why does it need the output of chunk()? What is the argument of this chunk? How many matrices are there? Is T in eq 15 the max sequence length? Then these matrices add up to a huge number of parameters.\n- The authors do not show the FLOP requirements of their model vs the baseline, so it is not possible to know if the performance improvement is solely because of the higher number of FLOPs spent by the model, or due to an advantage of the byte-level modelling.\n\n[1] Raposo et al. 2024: Mixture-of-Depths: Dynamically allocating compute in transformer-based language models\n[2] Kallini et al. 2024: MrT5: Dynamic Token Merging for Efficient Byte-level Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZptOXTReKn", "reviewer": "Reviewer_wvSa", "summary": "This paper proposes ByteFlow Net, a tokenizer-free language modeling framework that processes raw UTF-8 byte streams and dynamically learns hierarchical chunk boundaries through an information-theoretic coding-rate criterion. Compared to previous work, their method uses the rate-distortion criteria to determine the chunking boundary, which is new and novel.", "questions": "For equation (12), I think using mutual information can give a better estimate. Is there any relation between the proposed metric and mutual information?\n\nThe author can try this new information metric as a potential future work. https://arxiv.org/pdf/2002.10689", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper highlights tokenization rigidity as a key bottleneck for multilingual and structure-sensitive modeling, and motivates a tokenizer-free approach that adapts granularity both theoretically (information control) and practically (no preprocessing or vocabulary drift). Its framing as adaptive compression rather than tokenization replacement is elegant and insightful.\n\nThe use of coding rate—essentially a differentiable measure of local representational complexity—to determine chunk boundaries is elegant and grounded in information theory. Also see weakness and questions.\n\nThe writing is clear and easy to follow.  The authors also provided complexity analysis and training details for reproducibility.", "weaknesses": "While the information-theoretic formulation is novel, the paper’s framing of the criterion as a Shannon coding rate is, in my view, not entirely rigorous — the connection to classical source-coding arguments is more heuristic than formal.\nNevertheless, the proposed approach remains compelling, as the quantity introduced in Equation (11) effectively captures the sensitivity of the local representation to new information: it measures how much the embedding space expands or changes when a new byte is added.\nIn this sense, even if the theoretical grounding is loosely tied to Shannon’s framework, the method still functions as a principled sensitivity-based chunking rule that adaptively identifies informative boundaries in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)