From Bytes to Ideas: Language Modeling with Autoregressive U-Nets
Authors: Mathurin VIDEAU
Badr Youbi Idrissi
Alessandro Leite
Marc Schoenauer
Olivier Teytaud
David Lopez-Paz
Venue: NeurIPS 2025 poster ()
Keywords: Language Modelling
Large Language Models
Hierarchical Models
Byte Level Transformer
Tokenization
Primary area: deep_learning

Abstract: Tokenization imposes a fixed granularity on the input text, freezing how a language model operates on data and how far in the future it predicts. Byte Pair Encoding (BPE) and similar schemes split text once, build a static vocabulary, and leave the model stuck with that choice. We relax this rigidity by introducing an autoregressive U-Net that learns to embed its own tokens as it trains. The network reads raw bytes, pools them into words, then pairs of words, then up to 4 words, giving it a multi-scale view of the sequence. At deeper stages, the model must predict further into the future -- anticipating the next few words rather than the next byte -- so deeper stages focus on broader semantic patterns while earlier stages handle fine details. When carefully tuning and controlling pretraining compute, shallow hierarchies tie strong BPE baselines, and deeper hierarchies have a promising trend. Because tokenization now lives inside the model, the same system can handle character-level tasks and carry knowledge across low-resource languages.

From Bytes to Ideas:
Language Modeling with Autoregressive U-Nets
Mathurin Videau"
Badr Youbi Idrissi' Meta Al
Alessandro Leite INSA Rouen Normandy, LITIS
Meta AI
Marc Schoenauer TAU, Inria
Olivier Teytaud Thales CortAIx-Labs
David Lopez-Paz Meta AI
Abstract
Tokenization imposes fixed granularity the input text, freezing how language model operates on data and how far in the future it predicts. Byte Pair Encoding (BPE) and similar schemes split text once, build static vocabulary, and leave the model stuck with that choice. We relax this rigidity by introducing an autoregres- sive U-Net that learns to embed its own tokens as it trains. The network reads raw bytes, pools them into words, then pairs of words, then up to 4 words, yielding a multi-scale representation of the sequence. At deeper stages, the model must predict further into the future anticipating the next few words rather than the next byte deeper stages focus on broader semantic patterns while earlier stages handle fine details. When carefully tuning and controlling pretraining compute, shallow hierar- chies are on par with strong BPE baselines, and deeper hierarchies exhibit a promis- ing trend. Because tokenization now lives inside the model, the same system can handle character-level tasks and carry knowledge across low-resource languages.
Introduction
Language models are about uncovering patterns in a sequence so they can guess what comes next. Before any of that happens, must decide what the pieces that sequence-the the actually are. That choice is usually frozen in advance by a tokeniser that chops raw text into discrete units long before training begins. Consider the sentence "The quick brown fox." A character- feeds the model the stream (T, h, e, μ, 9. u and asks it to predict the next letter i. A word- level tokeniser, in contrast, hands over (The, qui quk and expects the model to guess brown one shot. Finer cuts lead to larger sequences and shorten the look-ahead window, whereas coarser cuts lead shorter sequences but make each token rarer and harder to compare and predict. Regardless of granularity, some form of tokenisation is unavoidable: a sequence must exist before any Transformer can run.
tokeniser
Byte-Pair Encoding (BPE) followed by simple embedding table is by far the most popular approach. works by repeatedly merging the most frequent byte sequences in the training text until a preset vocabulary limit is reached. This procedure leaves practitioners with jist two intuitive dials. The first dial is the training corpus: whichever text one feeds the algorithm English prose, source code, or multilingual mix-deteene determines which patterns are merged and therefore what the final tokens look like. The second dial is the vocabulary size: raising this limit lets the merge process run for more steps, producing longer tokens and shorter sequences at the cost of larger embedding table and output softmax.
Equal contribution
Code open-sourced at http://:hhhh.cooocccccocce//ee/s
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Contraction
Expansion
ti
Stage 2 Layers
Stage Layers 70%
Stage Layers 1186
Stage Layers
L.nyeess
Decoder only Transfomer layors
o Addition with residual connection
Tied linear
Different linears
Figure I: Three-stage Autoregressive U-Net (AU-Net). The model executes from left right. The contracting path compresses the sequence in steps: Stage processes bytes, Stage keeps only the vector each word boundary, and Stage keeps one vector per two words. Each contraction and expansion step supports arbitrary pooling and upsampling patterns. After the deepest stage, the expanding path reverses the contracting path by duplicating each coarse vector and upplying position-specific linear layers. These are combined with skip connections from the contracting path, gradually restoring sequence length and blending in high-level information. Deeper stages predict further ahcad and capture broad semantics, while shallower stages refine local detail.
Most issues with tokenisation stem from the embedding operation rather than the splitting act itself. Each token is typically mapped to independent vector, meaning the network only opaque identifiers and must rediscover, for instance, that strawberry and strawberries share nine letters. This reliance isolated embeddings hampers symbol-level reasoning and complicates transfer dialects or rare languages. Finally, this splitting is most often preprocessing step, locking in a single level of granularity for all subsequent model layers (see Section 2.2).
To address these limits, our Autoregressive U-Net (Section 2.1), or AU-Net ('oh-net', /6u net/), learns embed information directly from raw bytes, and allows for multiple stages of splitting. The purpose of an embedding to map tokens to vectors. Instead of using a lookup table, we use attention directly to embed the tokens. Self-attention allows vectors at any position to summarize the entire preceding context. This enables a simple pooling mechanism: we select these contextualized vectors at word boundaries (AU-Net-2), then word pairs (AU-Net-3), and up to four- four-word chunks (AU-Net-4), forming a multi-stage embedding hierarchy. This U-Net like architccture contracts sequences, preserving detail with skip connections, before expanding them. During expansion, vectors representing coarser information are injected back into more fine-grained representations.
Deeper stages operate on compressed representations, allowing them to aggregate information over longer spans of text. While the model remains strictly autoregressive and performs next-byte prediction, the hierarchical structure introduces inductive bias that encourages the formation of more abstract representations akin to multi-token prediction [1] but achieved without auxiliary losses. This effect allows deeper stages to guide shallower stages the semantic level, while letting them handle finer details like spelling.
Contributions (quantified in Section 3).
C1. Adaptive multi-level hierarchy. We train up to four end-to-end embedding stages with arbitrary. user-specified split functions, extending prior work that relies either on fixed pooling or shallow hierarchies.
C2. Infinite wocab size. By operating directly on bytes, our model avoids predefined vocabularies and memory-heavy embedding tables, enabling open-vocabulary modeling without increasing memory footprint. This property is inherent to byte-level approaches, and our results validate its effectiveness at scale.

---

C3. Strong performance and scaling. Under identical pre- tra-iraining budgets, a single level matches strong BPE baselimes, and a two or three-level hierarchy shows promising scaling trends. A selection of the results is presented in Table I
C4. Practical Efficiency. We maintain comparable GPU throughput in wall-clock time instead of purely theoretical compute gains. Our code is available in Meta Lingua [2].
C5. Stable scaling laws. We show that moving from token to byte-level training demands new batch size and learning rate formulas to get smooth optimization.
By turning one-shot, memory -hungry embedding into learned, multi-scale process, we offer flexible alternative to the rigid BPE preprocessing followed by a simple embedding table. Table provides concise overview of results obtained at the IB scale on 370B tokens, comparing AU-Net-2 (two-stage), AU-Net-3 (three-stage), and AU-Net-4 (four-stage) variants. These results highlight the promising performance in the heavily overtrained regime.
Method
2.1 Autoregressive U-Net
Inspired by U-Net-like architectures [3, 4], we pro- pose an autoregressive hierarchical model for language modelling, illustrated in figure 1. This architecture fea- tures a contracting path, which compresses the input sequence, and expanding path, which reconstructs it. Both paths are fully adaptive: they do not require fixed pooling or upsampling sizes. Pooling and upsam- pling operations can be designed independently, even if we choose to make them symmetrical in this paper. The only requirement splitting function, which specifies the positions in the sequence where pooling should occur. This function is detailed in section 2.2.
Table 1: 1B equivalent 370B tokens
Model
FLOP
Hellawwag
MMLU
GSMBL
BPE
4e21
70.2
27.0
4.4
2
69.9
28.8
AU-Net
4c21
72.9
28.0
3.7
AU-Net4
Sc21
73.7
31.7
5.3
Our architecture is monolithic: unlike recent approaches [5, 6] that use local models, we apply attention globally at each stage (or within sliding window), allowing every input attend to previous inputs. This ensures that words or word groups are not processed in isolation. To preserve fine- tme-grained information that might be lost during contraction, we introduce skip connections between stages, following the approach in Ronneberger et al. [3] and Nawrot et al. [4]. We also increase the hidden dimension at each stage in proportion its contraction factor, enabling richer representations as the sequence is contracted. To keep computation tractable at the byte-level stage (Stage 1), where sequences longest, restrict attention window.
2.1.1 Pooling and Upsampling
Since our pooling and upsampling are adaptive, cannot rely on fixed window sizes. To address this, we explored several pooling and upsampling strategies. In this section, we describe the method used in experiments reported in the main text. complete description the alternatives and ablation results can be found in the appendix C.
Residual Connection
Figure 2: Pooling simply selects the vectors at the positions specified by the splitting function. Upsampling then expands each pooled vector to fill the next segment, applying separate linear layer for each position. For instance, the pooled vector representing the word 'SAT is used to help predict 'ON_". This offset lets deeper stages predict further ahead in the sequence. When using stages, for example, this results in the deepest stage helping for the prediction of the next four words.

---

Pooling, We adopt the simplest pooling strategy: selecting the indices identified by the splitting function and projecting them to the next stage's dimensionality using a linear layer. Since the preceding layers already include attention mechanisms, we rely on these to do the pooling implicitly instead of relying on explicit cross attention as used in Nawrot et al. [4], Pagnoni al. [5].
Formally, let X E *x denote the sequence of hidden states, and let = {&1 < ... < im S be the indices selected by the splitting function, defining the new sequence length m with 1 1< m S Let W E R Xdau be learnable projection, where dn and dout correspond to the dimensions of the current and following stages, respectively. The pooled (downsampled) sequence is obtained by selecting the indexed rows and applying the linear projection:
{I....,s}
Y = X,W E R m.xdduu
where X, (X ..
Upsampling. The upsampling step maps coarse representations to finer ones for the next stage. As illustrated in Figure we duplicate each coarse vector to match the length the following segment, applying distinct, position-specifie linear transformations to these duplicates. Let Y E mmxdo and position-specifie projections (Wr)p=1 LK with Wp Rdomnddi Given per- segment lengths ra E (),..., K}. compute
X(,,),,,W,, for 1,..., m, I,...
Stacking (i. () in segment-major order gives X R x.)di.
Since these transformations are shared across segments but vary by position within segment, we term this Multi-Linear Upsampling. In experiments, models with multiple stages more sensitive to the specific choice upsampling strategy, whereas for pooling, many strategies work equally well.
2.1.2 Generation
During training, process the entire input sequence in parallel, activating all stages simultaneously. At inference, generation is autoregressive: the byte-level stage is active at every step, while deeper stages activate less frequently according to the pooling pattern. Skip connections transmit information upward at each stage, so deeper stages can integrate fine ame-ggrined details. This cascading, conditional activation enables efficient inference: computationally intensive high-level stages activate rarely, but still effectively guide detailed lower-level predictions. In practice, this means that we need to cache the latest vector at the output of each stage correctly propagate deeper stages' outputs.
2.2 Splitting Function
The AU-Net architecture supports flexible splitting strategies to define pooling points at each hierar- chical stage. The primary constraint is that any chosen splitting function must be stable to rightward insertion: appending bytes should not alter prior pooling decisions, ensuring consistent autoregressive generation. Various methods (e.g.. fixed windows [4], entropy [5], learned rules) are possible. Our current work splits on spaces using different regular expressions at each stage (details in Appendix B).
This strategy defines hierarchy: Stage processes bytes; Stage 2 pools word boundaries (identified by the regex); Stage pools after every two words (or sentence end); and Stage 4 after every four words (or sentence end). This rule-based approach, inspired by ne-tokenizztion in systems like GPT-4o's [7], is effective for Latin scripts. Extending robustly to languages without clear delimiters remains future work. Unlike prior approaches [5, 8] that used similar splits mainly replace BPE in a single-stage context, AU-Net uses these user-defined splits for its multi-stage hierarchical processing.
2.3 Evaluating on different scales
Large language models scale very predictably [9-11]. This allows to estimate the performance of a model for a large compute budget. But more surprisingly, it allows us to predict the optimal hyperparameters for models way beyond our ablation budget. Bi et al. [11] described a method for sweeping learning rates (LRR and batch sizes (BSZ. across range of small models, and they demonstrated that these results can be used to predict optimal hyperparameters for larger models.

---

Following their methodology, we show different evolution of hyperparameters, both due to the data in our setup and the hierarchical model. These hyperparameters are then used do scaling laws for bigger range of compute budgets to compare the baseline architecture and AU-Net. Throughout this paper, the scale of run is its total pre-training compute C' measured in Floating Point Operation (FLOP):
C'
model input-unit
Vinput-unit
FLOPs per (lrwand--ackwwntt) pass per input unit number of units of training imput
Following Bi et al. [11], we define model size as the number of FLOPs per input unit instead of relying on the number of parameters. This allows us compare models with different architectures fairly. The formula for the number of FLOP per input-unit for a decoder-only transformer is given by:
moddel / input-unit
The myno-mmeed params
6d LS
attention Lerm
where, NZno-embed is the number parameters, excluding the embeddings. d is the dimension, the params
sequence length and I the number of layers. To scale up, one can either make the model bigger (Fmotell / imput unit 1), give it more data uNin 1); or do both. Gadre et [12] showed that keeping the data- data-lo-model ratio "inpuu--nni constant is key to getting smooth scaling laws and predictable performance, where:
Ninput-unit
Tinput-unit
model / inpuut-uni
We adopt this convention in all experiments and report the data-to-mode ratio Tinput unit used in the experiments.
Bytes versus tokens. On DCLM [13], token sequence is on average k z 4.56 times shorter than its byte sequence when using the LLaMa 3 tokenizer.
Given some compression factor k between bytes and tokens, we want to express the equivalent Thytes- To do this, we note that Nbyce k x Nioken and F Fmode//ytc
F mooeeltoke// /k. Therefore,
Nioken Nokken
h22
7ayte
F Fmodeltoken
Note that this scaling relationship mrchtecture-agnostie The factor 1 follows directly from the difference in sequence length between tokens and bytes under given tokenizer. While different architectures may have distinct FLOPs/byte,, the conversion between token- and byte-level compute is determined solely by the compression ratio k. This factor allows to compare the performance of our model with the baseline on the same scale, as they will have seen the same amount of data and spent the same amount of FLOPs per token. Throughout the paper, always express the data-to-model ratio in LLaMa tokens (Nuoken).
FLOPS per byte for AU-Net. In the case of AU-Net, we cannot use the same formula the baseline because the contraction and expansion happening in the model. However, we can still use the same formulas as long we account for the contraction at each stage. So the total FLOPs per byte for AU-Net is simply the sum of each stage divided by the contraction factor.
moddl//yle
F Fmode//yne
where k; is the contraction factor at stage i.
This property allows us to have models with a higher number of parameters for the same compute budget and data-to-model ratio.
Hyperparameter scaling laws Bi et al. [11] showed that the regularity of scaling laws can be exploited to tune very large models from a sweep over much smaller ones. We replicate their protocol on six miniature versions of each architecture (baseline Transformer and AU-Net): quasi-random search over batch size and learning rate, keep the configurations within 1 1% of the best validation loss, and fit (SZ A C and LR(C) B C to those points, with parameters A, o, B and B. We find the following formulas at the byte level for AU-Net:
perform
BSZAU-Na((C)
0.66 x C LR (C) 6.6 -0.176
And we run the same tuning for the BPE baseline, for which we find:
BSZ&pe(C)) 29.9 x 00223
LRBPE(C') 19.3 C 00.177
5

---

3 Experimental Results
3.1 Experimental Setup
Data. For all experiments, DCLM [13] served as the pretraining dataset, with a small portion held out for validation, totalling around 4T tokens (of GPTNeoXTokenizer), The corpus mostly English and focuses on natural language understanding, i.e. it contains a marginal amount of code and maths. Baselines. We compare our approach to different three baselines: Transformers equipped with the BPE tokenizer of LLaMa 3, Transformers and Mamba [14] trained directly on bytes, AU-Net variants using a fixed-size pooling window (denoted Transformer bytes[:w]] where w is the window size), similar to the Hourglass architecture [4]. To keep the comparison fair, we trained each baseline with the same amount of data or compute. For example, if data budget of 273B bytes is used to train the bytes level AU-Net model, this budget is converted to 60B training tokens for transformer with LLaMa 3 tokenizer [15] because of the 4.56 compression rate measured on the DCLM corpus.
AU-Net Architecture. AU-Net-2, -3, and -4 progressively increase embedding dimensionality across stages (512- - 2048 - 3072 - 4608 for deeper variants at the 1B scale), with layer allocation guided by the ablation results in appendix C.2. The AU-Net-2 architecture at 1B scale is illustrated in figure 1, showing stage dimensions, layer distribution, and total FLOP allocation per stage. The first byte-level stage does not require high dimensionality, as it primarily encodes local information before compression, whereas later stages capture increasingly abstract representations that benefit from greater capacity. Contraction rates chosen so that when representations are merged, information density remains approximately constant (e.g., sequence compressed by factor of two doubles its hidden dimension). At the IB scale, AU-Net-4 uses 6 + 25 layers with maximum hidden size of 4608, while the 8B-scale configuration expands to 6 + 33 layers and a final hidden size of 4096. This setup reflects the model's 's compression rationale while keeping compute distribution flexible For comparison, the BPE Transformer employs 25 layers with hidden size of 2048 at the 1B scale and 33 layers with a hidden size of 4096 at the 8B scale.
Full architectural specifications, such as embedding sizes, layer counts, are detailed in appendix C.2. Also, note that, due to compute constraints, AU-Net-3 and AU-Net-4 were not trained at the 8B scale. Hyperparameters. For a detailed overview of the hyperparameters, see appendix D. As explained in section 2.3, we sweep batch size and learning rate values across model scales ranging from 25M to 500M. Then, we extrapolate the best learning rate and batch size for any given compute budget. Evaluation Metries. All models are evaluated on a broad set of downstream tasks in zero-shot setting, occasionally including few in-context examples directly in the prompt. These tasks fall into two categories: (i) multiple-choote (MCQ) tasks, where the correct answer is selected as the option with the lowest normalized negative log- ligeliidl (divided by the number of characters) [16]; and (ii) open-ended generation tasks, where the model is allowed to freely generate its answer.
To highlight the strengths of AU-Net, we include specialized benchmarks targeting character-level manipulation (CUTE [17) appendix E) and low-resource language translation (FL.ORES-200, [18) section 3.4).
For clarity, we report selection of key benchmark results in the main tables, including Hellaswag, ARC-Easy, ARC-Challenge, MMLU, NQ, TQA, and GSMSK. Also, we report 95% confidence intervals for all tables using bootstrap. A full breakdown of all evaluation results is provided in the appendix F.
In addition to task performance, the total training FLOPs and training throughput are provided for each model, measured in bytes per second per GPU (bps) on H100 80GB GPUs (internal cluster) during the actual training.
Implementation Details. As scaling is key to the success of large language models, our imple- mentation balances efficiency and simplicity. We use sequence packing along with full attention, a strategy shown to have little to no impact on downstream performance ([13)). To reduce GPU memory pressure, all our experiments rely on Fully Sharded Data Parallelism (FSDP).
For additional speed-ups, the entire model compiled with torch, compile. Compilation, however, requires a static computation graph, which clashes with the variable-length outputs produced by our adaptive pooling: the number of bytes per word (and thus per stage) naturally varies across sentences. We resolve this by fixing maximum sequence length at every stage: sequences that exceed the limit are truncated abruptly, and shorter ones are padded, This compromise yields a graph that static for compilation while still supporting adaptive hierarchical pooling in practice.

---

Table 2: Downstream results comparing AU-Net to BPE and byte-level baselines. We report accuracy on key benchmarks with 95% confidence intervals where applicable. Literature models are shown in italies; all models are trained on the same corpus, unless specified. AU-Net variants differ in the number of stages. We also report compute budget and empirical training speeds in bytes/sec.
Model
Params Embh
Flops
hps
[ellaswag ARC_E
ARC_C
MMLU
NQ
TQA
GSM8K
Dim=2048 (IB model), 60B tokens (data-to-model ratio of 10)
Mamba Byte
1.3B
3e21
63.0
20.5
60.3
12.
33.6
12.
10.7
8.2
10099
21.2
10.7
10.8
Transformer Byte
1.3R
IM
4e21
47k
63.0
1100
61.2
11.
34.7
12.7
24.7
10.7
8.8
10.9
21.4
10.8
2.5
10.9
Transformer Byte[::]
1.38
6c20
180k
63.5
20.0
63.0
11.
36.0
22.7
25.1
10.7
6.8
10.8
17.2
20.7
2.6
20.3
Transformer Bytel::51
1.3B
1M
5c20
218k
60.0
+1.0--00000000 60,0
+2.0
34,8
+2.8
23.9
+0.7
5.6
14.7
2.0
+0.8
Truns Byte|=:6]
1.3B
IM
4e20
255k
58.6
:1.0 59.5
22.0
32.4
12.3
24,9
10.7
4.6
1.07
12.3
20.6
2.3
20.8
AU-Not
1.38
5c20
225k
64.2
+0.9 64.4
+1.9
35.2
+2.8
8.8
20.4
+0.7
+0.9
AU-Net
2.5B
1M
7c20
180k
67.4
+0.9
65.9
+1.
36.7
+2.7
26.3
+0.7
9.6
+10
22.6
+..8
2.3
+0.8
AU-Net
4.20
8c20
155k
66.4
10.9
67.4
11.8
37.0
12.8
10.7
5.1
1007
15.5
10.7
11.0
Transformer BPF
1,8B
525M
7c20
210k
63.6
11
62.8
11
36.5
26.2
10.
8.8
1.0.9
26.3
100
2.3
10.8
Dim=2048 (IB model), 370B tokens (data-to-moddl ratio of 40)
AU-Net
1.3B
3c21
225k
69.9
0..0
68.6
11.00
38.9
1
10.7
13.0
11:1
32.5
1..0
10.9
AU-Net
2.58
IM
4c21
180k
72.9
=0.0
72.3
+1.8
43.3
#2.8
28.0
40.7
15,3
#1.2
39.1
00.0
3.7
+1.0
AlJ-Net
4.28
IM
Se21
155k
73.7
20.9
72.6
11.8
43.2
12.0
31.7
10.7
14.0
21-1
35.5
10.9
5.3
11.2
Transformer BPE
1.8B
525M
4c21
210k
70.2
68.6
38.5
27.0
13.5
st.
37.2
#1.1
DCTM-IB- 1B5M (x/55x11558)) [x]14]]113
207M
1e21
66.1
70.2
40.6
29.3
MegaByte (2638)[19]
I.IB
73k
38.9
54.9
23.4
25.1
9.6
Hierochical (2638i[6]
I.IB
le21
46.5
65.0
30.5
26.0
9.6
Dim=4096 (8B model), 2008 tokens (dafa-to-model ratio of 5)
AU-Net
7,9B
1M
le22
41k
79.1
80.0
11.6
51,2
1:.8
51.1
22,1
50.9
1.099
10,0
1:
Transformer BPE
7.5B
IB
9c21
43k
77.2
2:0
74.5
111
49.2
1:2
49.6
1:0
21.1
21.
51.1
1.00
10.7
DCLM-78 [x
7B
413M
le22
77.8
78.1
52.6
50.8
50,9
4.3
Hierorchical (2638]]]6]
9,2B
1e22
15k
56.3
76.6
44.2
32.0
33.1
BLT (220B) [5]
1e22
72.2
66.8
38.8
BLT (17))
5e22
80.6
79.6
52.1
DCIM-7B (2.57)) [15]
7B
413M
le23
80.4
82.2
59.9
63.7
52,7
2.5
LLaMa 3.1 (157)* X [1.5]
8B
18
6c23
83.3
+00.8
80.7
+.166
54.8
+220
66.4
+0.88
29.1
+1.5
64.4
+.000
54.7 +2.7
Trained on mix of DCLM and other datasets
x Trained different corpus than DCLM
3.2
Equal Data Budget Results
We evaluate the effectiveness of hierarchical pooling by fixing the model's primary hidden dimension 2048 and maintaining a constant total training-data budget. The hidden dimension at each stage is scaled proportionally to its contraction ratio as described in section 2.1. For instance, the byte-level stage uses a dimension of 2048/4 4 512, the word-level stage uses 2048, and the 2-word level uses 1.5 x 2048 = 3072, continuing in this manner for deeper stages. We assess the downstream performance of language models with 2, 3, and 4 stages at the 1B parameter scale. For the 8B model, we evaluate only the I-stage configuration for now. All variants are compared against a Transformer baseline using the LLaMA 3 tokenizer of the same main hidden dimension. More ablations regarding pooling and the number of layers per stage can be found in the appendix C.
As shown in table 2, hierarchical models consistently match outperform their BPE-based coun- terparts. This trend holds across various configurate configurations and becomes especially pronounced as we introduce more hicrarchical stages. Notably, multi-stage AU-Net models (c.g.. AU-Net 3 and AU-Net 4) outperform BPE basclines on several benchmarks. An interesting exception to this pattern the TQA benchmark, which is a knowledge-intensiee task evaluating the generation of the model. AU-Net models along with byte-level baselines consistently underperform on TQA compared to BPE-based models. This suggests that the performance gap may not stem solely from the hierarchical structure. However, as model size and training data scale (e.g.. at the 8B or 1B, 370B tokens scale), this discrepancy seems to vanish. When examining the Transformer Bytes[::w] models, we clearly observe the effect of pooling at word boundaries. As the pooling window size w increases, the resulting increasing compression comes at the cost of a consistent drop in performance. contrast, using more principled pooling strategy, as in AU-Net, achieves better compression while main-
7

---

taining strong downstream performance, enabling more robust and faster training. This effect is particularly pronounced on generative tasks such as TQA, where performance declines sharply for larger window sizes and remains below both AU-Net and BPE baselines. This is further emphasised in the subsequent section 3.4 where the same pattern emerges. For languages where ppace-based pooling does not align well with word boundaries, performance also decreases significantly.
We observe early signs diminishing returns beyond certain number of stages. While AU-Net improves on reasoning-heary tasks such as ARC-C and GSM8K, gains on benchmarks like Hellaswag and TQA are less consistent. However, this effect may stem not from hierarchy itself, but from data efficiency: deeper hierarchies might require more training data to reach their full potential. Supporting this interpretation, AU-Net and AU-Net 4 benefit significantly from additional training data, and that MMLU and GSM8K scores continue to improve with more stage, even at fixed scale.
Finally, when comparing our models similarly sized baselines from the literature (italicized in the table), we find that AU-Net remains competitive, even while using significantly less training data. For instance, BLT (IT) uses approximately 5x more compute than our 8B model, while only being better on MMLU. Importantly, comparisons with literature models are fair, as all were trained on the same corpus: DCLM (except for BLT (220B) and LLAMA 3.1 (15T)).
To further evaluate our approach, we now turn to scaling laws (figure 3) to better quantify how our architecture compares standard Transformer with BPE. We focus AU-Net 2 and AU-Net 3, using a tota-t ratio of This choice is motivated by the diminishing returns observed when moving from AU-Net 3 to AU-Net under the same data-to-model ratio.
3.3 Scaling laws
Using the learning rate and batch size formulas (Section 2.3). run pretraining for range of compute budgets ranging from 1c19 to le22 flops (corresponding to models from 150M to 5.3B non embedding parameters) for the baseline, with mata-tto-moell ratio of 10. This is roughly 2x the optimal bata-to-model ratio found by Kaplan et al. [9]. Compared to table and the 8B-scale experiments, the scaling-law
higher data-to-model ratio (data-to-model ratio 10 instead
Baseline
2 stages
stages
Llama 3.1 8B 15T tokens
Hellaswag
TQA
NQ
0.8
0.6
0.3
Accuracy
0.2
0.4
0.6
0.2
0.1
0.4
ArcEasy
MMLU
GSMBK
0.8
0.6
Accuracy
0.4
0.6
0.4
0.2
0.4
0.0
1020
10
10
1020
1022
20
10
10
1024
Compute (FLOPs)
Compute (FLOPs)
Compute (FLOPs)
Figure 3: Downstream task performance scaling with compute (Ic19-1e22 FLOPP) under data-to-model ratio of 10. AU-Net (2/3 stages) generally tracks strong BPE Transformer baseline, which itself performs competitively against much larger models (e.g.. LLaMa 3.1 matches the baseline on tasks like Hellaswag and ARC Easy, and catches up on TQA at higher compute, its performance improvement phase MMLU and GSM8K appears to start later. The general underperformance GSMSK also linked to limited math data in the DCLM pretraining corpus.
15T tokens 100x compute). While AU-Net

---

Table 3: Multilingual evaluation. Left: BLEU scores on the FLORES-20 200 benchmark across multiple languages. Higher scores indicate better translation quality. Right: MMLU Exact Match (%) across 26 non-English languages. Results are averaged per language across all tasks.
MMLU
BPE
2U-Net
FLORES-200 (BLEU)
Lang
Eing.
Eog.
Lang. AU-Net
English
49.6
51.1 1.0
BPE
AU-Net 2
BPE
Arabic
29.1
29.5
German
34.4 1.2
33,9 1.2
16.7 20.88
15.6 2.0.7
Bengali
27.5 +-077
27.6 +00.8
Dutch
24.7 1..0
25.0 1.0
12.3 1:000
11.7
Chinese
33.0 1:0.8
28.0 2:0.7
Afrikaans
32.0
35.7 11.3
14,8 20..
16.1 16.1
Czech
30.7.:00 1:0..
32.2
Faroese
8,7 +0.7
9.9 +0.8
1.8 +..+
2.9+0.4
Dutch
34.5 24.5..0.8
37.1
Icelandic
7,8 t0.8
9.0 :0.7
1.7 1.7
2.5 +0.3
Finnish
29.0 +
29.3+
Limburgish
15.3.20.8
19.9 11.0
5.7 5.7.
6.7 10.3
French
37.3 1::08
40.7 2:0
Luxembourgish
11.400
14.7 2:000
2.6.:.33
4.0 4...
German
36.0
37.6
Greek
29.2 +:0.6
30,5 +:0.8
Italian
29.1
30.1 +1.0
15.1 +0.7
15.3 6:0.6
Hindi
27.9 1:0-7
27.5
Priuiian
14.6+-0.8
19.1 +10
3.2
4.0 +0.3
Hungarian
29.0
30.1 1:0
Ligurian
16.5
21.8 11.0
3.4 3..3
3.9 10.3
Indonesian
34.9::0.5
37.3 6:0
Lombard
12.9.0.8
19.2
5.24
4.2.
Iulian
36.2 +6...
39.0
Sardinian
14.3
18.2 11.0
40.4
4,5 #004
Japanese
29.5 1:0.7
28.2
Sicilian
11.7.00.
16.8 10.0
3.94
10.4
Korean
28.4
Venctian
19.8
25.4 11.1
5..
5.6.
Persian
28.7 1:0.7
28.6 2.0.7
Polish
30.3
32,0 1:0
Spanish
28.2
29.3
20.7
19.8
Portuguese
37.2
40.9
Asturian
24.0+111
28.6
111
8.2 +0.5
Romanian
34.0 +-0.8
36.9 +.0.8
Catalan
28.1111
33.0 1:2
9.6.10.5
10.7:100
Russian
30.9::00.3
31.2
Occitan
28.0
35.5 11.2
4...
6.2 10.4
Spanish
37.6 +7.6
41.4 +:000
Portuguese
4..0113
43.6 11.3
2..3
2::4
Swahili
Galician
29.6+1.11
34,0 +12
9.9 +0.5
10.2 +0.0
Swedish
33.5
36,0 2:0
Papiamento
17.3::0.
22.1 11:1
2..3
6.3.10.4
Telugu 26.8
1.0.7
Kahuverdianu
13.7 13.7.0.8
20.8 11-1
2.4
5.1 10.4
Thai
28.0
27.5
Turkish
29.1+-0.7
30.0+0077
Experanto
1..
19.3
3.6
5.9 20.4
Vietnamese
31.4
30.7
Average
20.9.0
24.6 1..
1.0
10.1
Average
31.4
of 5) and the latest hyperparameters described at the end of section 2.3. These differences can explain the variation in peak performance across the two experiments.
The list of models chosen for each budget is detailed in the appendix G. Figure 3 shows the evolution of performance on 6 downstream tasks for AU-Net and the BPE baseline. Here we mainly notice that 2 and 3 stage AU-Net models can match the performance of the BPE baseline when carefully controlling for compute budget. This is the case for Hellaswag, Are Easy, and NQ. For TQA, AU-Net both for 2 and 3 stages starts with a performance gap, but the 3 stage model catches up at le22 flops. However, both 2-stage and 3-stage AU-Net models are still behind the BPE baseline at le22 flops for GSMSK and MMLU. Most downstream tasks follow a sigmoid pattern: performance is near chance at low compute, then rapidly improves before plateauing. For AU-Net models, this transition appears to occur slightly later on tasks like GSM8K and MMLU, suggesting that the benefits of deep hierarchy may become more pronounced at larger scales. Nevertheless, on many benchmarks, both AU-Net variants and our BPE baseline achieve results remarkably close to those of considerably larger models like LLaMa 3.1 8B (pretrained on 15T tokens, representing 100 times more compute than our largest run shown here). This proximity underscores the strength of our BPE baseline, making AU-Net's ability match or trend towards it particularly noteworthy. The primary exception where this close tracking is less apparent is GSM8K; however, this underperformance across all our models is likely due to the pretraining corpus, DCLM contains very little math data.
3.4 Extended Evaluations
We present results highlighting two specific advantages of byte-level training with AU-Net over BPE-based Transformers: improved performance multilingual benchmarks (Table 3) and character- level manipulation tasks (Table in the appendix E). Table 3 show surprisingly strong non-Enge non-English performance of both model, despite DCLM being heavily filtered toward English.
Cross-lingual generalization within language families. On the multilingual MMLU benchmark (Table right), languages using Latin scripts consistently benefit from byte-level modeling. We observe strong positive transfer between related languages. Concretely, Germanic languages (German, Swedish, Dutch, etc.) show an average gain of +3.0 points, while Romance languages (Italian, Spanish,

---

Portuguese, French, etc.) improve by +4.0 points. These results suggest that operating at the byte level helps the model to capture shared orthographic and morphological patterns across related languages.
Transfer to low-resource languages. The FLORES-200 benchmark (Table left) includes many low-resource languages that underrepresented or absent in the training data. This setting allows us to test the model's ability to generalize based on subword morphology and shared linguistic roots. Byte-level modeling provides the flexibility to construct meaningful representations without requiring the presence of these languages in the tokenizer or training corpus. We observe consistent gains in translation tasks into English, where the model must primarily understand the source language. The advantage is particularly clear for languages that share syntactic or morphological traits with more dominant relatives the same family. This also highlights the robustness of our model: it can produce meaningful translations even with out- -ut-of-vocabulary words or forms unseen during training. In the reverse direction (English to low-resource), generation remains more challenging.
4 Related Work
Traditional tokenization methods are important for computational efficiency 20-23], but impose fixed granularities. Early attempts overcome this rigidity explored adaptive vocabularies [24], n-gram combinations [25], or alternative splitting criteria like entropy [5]. Our work, AU-Net, advances this by integrating tokenization and representation learning into a multi-level, autoregressive U-Net
architecture that operates directly on bytes.
This hierarchical, adaptive-pooling design distinguishes AU-Net from prior works. For instance, Megabytes [19] introduce a two stage LLM using local models but with fixed-size token blocks, unlike AU-Net's input-adaptive pooling. Neitemeier et al. [6]. Byte Latent Transformers (BLT) [5], Dynamic Pooling Transformer (DPT) [26] and SpaceByte [8] also process bytes or use specialized splitting functions. However, they typically aim to replace BPE for single effective processing stage or use local attention mechanisms. In contrast, AU-Net leverages user-defined splits within stult architecture featuring distinct pooling strategies that differ from the cross attention methods in Nawrot et al. [4], Pagnoni et al. [5]. Nawrot et al. [4] defined a similar U-Net architecture but with fixed pooling, much smaller models, and their evaluations mainly foeus on perplexity.
Concurrent to our work, H-Net [27] introduces a hierarchical sequence model based on learnable, data-dependent dynamic chunking. Like AU-Net, H-Net operates directly bytes and constructs up to three stages network. While both models share similar high-level goals, they differ in key mechanisms: H-Net employs a learned dynamic split function, whereas AU-Net relies on predefined, rule-based splitting functions for hierarchical pooling and upsampling.
5
Conclusion
This paper introduces AU-Net, an autoregressive U-Net that processes raw bytes and learns hierarchi- cal token representations. By dynamically pooling bytes into words and multi-word chunks, AU-Net eliminates the need for predefined vocabularies and large embedding tables while preserving BPE performance with higher compression. Experiments show that AU-Net matches strong BPE baselines under controlled compute budgets, with deeper hierarchies demonstrating promising scaling trends. Furthermore, byte-level operation leads to improved performance on character-level tasks and bet- ter generalization to low-resource languages. This approach offers flexible and efficient alternative to traditional tokenization methods, paving the way for more adaptable and versatile language models.
Limitations and further work
Our work relies on DCLM, an English-only corpus, and currently supports only space-delimited languages with a predefined splitting function. For example, this affects, Chinese, where MMLU scores are lower than the BPE baseline. Another limitation is the lack of evaluation on code and math benchmarks. Since the DCLM corpus contains very little code abd mathematical data, models trained on it perform poorly on these tasks, yielding low and noisy results close to random performance. One extension could be to learn directly the splitting function. On the software side, as the number of parameters increases with the number of stages, FSDP already struggles to overlap computation and communication even at 3/4 stages, it needs a minimum amount of inputs to be fully overlapped.
10

---

References
[1] Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, 41st International Conference on Machine Learning, volume 235, pages 15706-15734, 2024.
[2] Mathurin Vidcau, Badr Youbi Idrissi, Daniel Haziza, Luca Wehrstedt, Jade Copet. Olivier Teytaud, and David Lopez-Paz. Meta Lingua: A minimal PyTorch LLM training library, 2024.
URL github. wwt////o//o/ccaaeeeeceooeo htt.://abbooorrrerrrrrar chigga
[3] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F, Frangi, editors, Medical Image Computing and Computer- Intervention, pages 234-241, 2015.
[4] Piotr Nawrot, Szymon Tworkowski, Michal Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Findings of the Association for Computational Linguistics, pages 1559-1571, 2022.
[5] Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, et al. Byte latent transformer: Patches scale better than tokens. 2:X1v:241..09871, 2024.
[6] Pit Neitemeier, Björn Deiseroth, Constantin Eichenberg, and Lukas Balles, Hierarchical autoregressive transformers: Combining byte- and word-level processing for robust, adaptable language models. In 13th International Conference on Learning Representations, 2025.
[7] Gautier Dagan, Gabriel Synnaeve, and Baptiste Roziere. Getting the most out of your tokenizer for pre-training and domain adaptation. In Ruslan Salakhutdinov, Zico Kolter, katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, 41st International Conference on Machine Learning, volume 235, pages 9784-9805, 2024.
[8] Kevin Slagle. SpaceByte: Towards deleting tokenization from large language modeling. In
A. Globerson, I.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 124925-124950, 2024.
[9] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001. 083611, 2020.
[10] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In 36th International Conference on Neural Information Processing Systems, 2022.
[11] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek I1m: Scaling open-source language models with longtermism. arXiv:2401.02995 2024.
[12] Samir Yitzhak Gadre, Georgios Smymis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang. Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and downstream tasks. wwXiv:2403.08544 2024.
[13] Jeffrey Li, Alex Fang, Georgios Smyrnis, et al. DataComp-L.M: In search of the next generation of training sets for language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 14200-14282, 2024.
[14] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In
First Conference on Language Modeling, 2024.

---

[15] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The Ilama 3 herd of models. arXiv:2407.21783, 2024.
[16] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020.
[17] Lukas Edman, Helmut Schmid, and Alexander Fraser. CUTE: Measuring LLMs' understanding of their tokens. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Conference on Empirical Methods in Natural Language Processing, pages 3017-3026, 2024.
[18] Marta Costa-jussa, James Cross, Onur Çelebi, Maha Elbayad, et al. Scaling neural machine translation to 200 languages. Nature, 630, 06 2024.
[19] Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zeutlemoyer, and Mike Lewis. MEGABYTE; Predicting million-byte sequences with multiscale transformers. In A. Oh, Naumann, Globerson, Saenko, M. Hardt, and Levine, editors, Advances Neural Information Processing Systems, volume 36, pages 78808-78823, 2023.
[20] Mchdi Ali, Michael Fromm, Klaudia Thellmann, et al. Tokenizer choice for LLM training: Negligible or crucial? In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics, pages 3907-3922 2024.
[21] Nived Rajaraman, Jiantao Jiao, and Kannan Ramchandran. An analysis of tokenization: Trans- formers under markov data. In A. Globerson, I.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors. Advances in Neural Information Processing Systems, vol- ume 37, pages 62503-62556, 2024.
[22] Shuhao Gu, Mengdi Zhao, Bowen Zhang, Liangdong Wang, Jijie Li, and Guang Liu. Re- tok: Replacing tokenizer to enhance representation efficiency in large language model. arXiv:2410.04335, 2024.
[23] Brian Lester, Jaehoon Lee, Alexander A Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl- Dickstein, and Noah Constant, Training LLMs over neurally compressed text. Transactions on Machine Learning Research, 2024.
[24] Mengyu Zheng, Hanting Chen, Tianyu Guo, Chong Zhu, Binfan Zheng, Chang Xu, and Yunhe Wang. Enhancing large language models through adaptive tokenizers. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 113545-113666 2024.
[25] Björ Deiseroth, Manuel Brack, Patrick Schramowski, Kristian Kersting, and Samuel Weinbach. T-FREE: Subword tokenizer-free generative LLMs via sparse representations for memory- efficient embeddings. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Conference on Empirical Methods in Natural Language Processing, pages 21829-21851, 2024.
[26] Piotr Nawrot, Jan Chorowski, Adrian Lańcucki, and Edoardo M Ponti. Efficient transformers with dynamic token pooling. arXiv preprint arXin:2211.09761, 2022.
[27] Sukjun Hwang, Brandon Wang, and Albert Gu. Dynamic chunking for end-to-end hierarchical sequence modeling. arXiv preprint arXiv:2507.07955, 2025.
[28] Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, and Gargi Ghosh. Memory layers scale, arXiv preprint 4rX1v:2412.09764, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We report results on multiple benchmarks in section 3 and more specific benchmarks section 3.4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See "limitations" the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
no theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our experiments use standard measures, experimental setup is described in section
3 and hyperparameters in appendix D. Our experiments are run on a public code, namely meta Lingua[2].
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: data are standard public datasets. We work with an open-sourced codebase for LLM, namely meta Lingua [2]. and our code will be open-sourced on acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //tpps.ccppublllee wwtpccccpppccccccccccccccccccccc g/wuidideeeeeeeee gwwgguiiss/eee des/ CoodeSubmis ssi asioonooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we specify all training sets and models, including pretraining data. All details about our scaling laws and the downstream performances are specified and standard.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
We report 95% confidence interval using bootstraping in all tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We use the number of flops in the x-axis of our scaling laws and specify budgets of all experiments. Section 3 we specify the type of GPUs, using
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:/::///uruueeeeeeeee wwtppblic//eceeeceeeeeecceeceeee (ealllic/Ethisttttte cww.ideliiideeeneeeeee
Answer: [Yes]
We read the ethics guidelines and did not violate anything.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the paper is rather technical than application oriented. Though the improve- ments rare languages be positive societal impacts. We potential negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: N/A
Justification: No models or data are released.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets are cited and properly credited in appendix.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwwde. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Code will be release later date.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Our work is about the training of LLM, so LLM are at the core of the present research. We use standard, open-soureed methodologies for evaluating LLM, using [2]. There is no other usage of L.M..
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy httppp:///eurps.... 220.a//eeccoooeeeeeeee////////// for what should or should not be described.
19


Reviews:
[{"review_id": "8OR89w2CdR", "reviewer": "Reviewer_SrNB", "summary": "The paper introduces autoregressive U-Net (AU-Net), an architecture for language modeling that substitutes the standard tokenization and embedding steps with a learnable hierarchical embedding process applied directly on raw bytes. AU-Net has contracting and expanding steps. Stage 1 processes raw bytes, Stage 2 pools at word boundaries, Stage 3 at two-word chunks, and Stage 4 at four-word chunks, with deeper stages naturally predicting further into the future and capturing broader semantic patterns while earlier stages handle fine-grained details. Skip connections allows access to information from previous stages. The expanding path predicts text sequences by transforming coarse information back into fine-grained representations. The authors show that within controlled compute budgets, AU-Net matches or exceeds BPE-based Transformer baselines on standard benchmarks, with deeper hierarchies showing promising scaling trends and particular advantages on character-level tasks and multilingual benchmarks, especially for low-resource languages that benefit from byte-level processing.", "questions": "- Why have you chosen an english-centric dataset (DCLM) to train the model?\n\n- Stage 1 processes raw bytes via a self-attention mechanism, that creates byte-level embeddings before Stage 2 pools at word boundaries. How exactly are these raw bytes fed into the self attention mechanism?\n\nTypos:\n\nl59: missing a period \".\" after \"Practical Efficiency\"\n\nl118: missing a space between \"two words\" and \"(or sentence...\"", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n- The paper uses careful compute-controlled comparisons enabling fair comparisons across architectures.\n\n- AU-Net achieves strong results when compared with BPE-based Transformer baselines, with scaling law results demonstrating that 2-stage and 3-stage variants consistently match BPE baseline performance across different compute budgets and benchmark tasks.\n\n- The paper shows strong results in multilingual settings despite using an English-centric dataset (DCLM), with consistent improvements on low-resource language translation tasks, demonstrating the advantages of byte-level processing for cross-lingual transfer.\n\nWeaknesses\n\n- The use of an English-centric dataset (DCLM) may unfairly disadvantage BPE baselines, which are specifically designed to handle diverse vocabularies. The authors don't justify why they chose DCLM over more multilingual datasets, making it unclear how AU-Net would compare with Transformer + BPE baselines on multlingual benchmarks given standard multilingual training corpus.\n\n- The paper lacks precise mathematical formulations of the model architecture, relying primarily on prose descriptions. Stating the actual computations performed at different stages and modules would help to better understand the architecture.\n\n- Not a big weakness, but many languages in the FLORES-200 evaluation are closely related variants within the same language families (e.g., Spanish, Asturian, Catalan, Galician for Romance; or various Italian dialects), which might inflate the apparent multilingual improvements.\n\n- Although acknowledged by the authors, the lack of support for non-space-delimited languages represents a fundamental architectural limitation.", "was_revised": true, "final_justification": "My minor weaknesses have been mainly addressed in the rebuttal. I believe the paper presents a relevant contribution that is worth publishing.", "rebuttal": "We thank the reviewer for this thorough review, and answer below the questions raised.\n\n---\n\n> **The use of an English-centric dataset (DCLM) may unfairly disadvantage BPE baselines, which are specifically designed to handle diverse vocabularies. The authors don't justify why they chose DCLM over more multilingual datasets, making it unclear how AU-Net would compare with Transformer + BPE baselines on multilingual benchmarks given a standard multilingual training corpus.**\n\nWe chose DCLM to ensure comparability with existing literature and because it was the state-of-the-art pretraining dataset when starting this work. While BPE can be trained multilingually, most popular tokenizers are English-centric, and BPE's pre-tokenization rule often splits on spaces, similar to our approach. Our model's ability to use learnable embeddings instead of a fixed embedding table offers greater flexibility, essentially providing an infinite vocabulary. We agree that exploring multilingual and code datasets in future work would be highly valuable.\n\n> **The paper lacks precise mathematical formulations of the model architecture, relying primarily on prose descriptions. Stating the actual computations performed at different stages and modules would help to better understand the architecture.**\n\nWe apologize for the lack of precise mathematical formulations in the paper. We will add formulas for the downsampling and upsampling operations for clarity and completeness, while still referring to the code for more detailed implementation.\n\n\n> **Not a big weakness, but many languages in the FLORES-200 evaluation are closely related variants within the same language families (e.g., Spanish, Asturian, Catalan, Galician for Romance; or various Italian dialects), which might inflate the apparent multilingual improvements.**\n\nWe acknowledge this point regarding the relatedness of some languages in the FLORES-200 evaluation. Our aim was to demonstrate the effectiveness of byte-level processing for cross-lingual transfer, especially for low-resource languages, and the results still indicate a consistent improvement over baselines, even if some languages are closely related.\n\n> **Although acknowledged by the authors, the lack of support for non-space-delimited languages represents a fundamental architectural limitation.**\n\nWe agree that supporting non-space-delimited languages (e.g., Chinese, Japanese, or Korean scripts) remains a limitation. We have designed the splitting implementation to be quite generic, allowing future work to define new splitting functions tailored to these languages. We have recently discovered the existence of the uniseg python package that uses dictionaries to segment text into words in many languages without relying purely on spaces.\n\n> **Stage 1 processes raw bytes via a self-attention mechanism that creates byte-level embeddings before Stage 2 pools at word boundaries. How exactly are these raw bytes fed into the self-attention mechanism?**\n\nRaw bytes are obtained by decoding a string with UTF-8, which produces a sequence of bytes. These bytes are then fed into the self-attention mechanism after being embedded using a 258-entry embedding table, which includes Beginning-of-Sentence (BOS) and End-of-Sentence (EOS) as special characters."}, {"review_id": "QVwjuEyndh", "reviewer": "Reviewer_Wegy", "summary": "The paper describes a novel U-Net-inspired language model architecture operating directly on raw byte representation, without requiring a predefined (e.g., BPE) tokenizer.  Instead, the paper proposes a hierarchical architecture where components of the network operate on different temporal scales (by pooling subsequences of the input byte-level tokens), in the hope of more efficiently capturing\nlonger context and operating at higher level understandings.\n\nIn contrast to similar previous work like Hierarchical Transformer or MegaByte, the pooling hierarchy is not based on fixed window size, but is instead derived from the underlying content, e.g., splitting on words and ngrams.  Experiments show that the proposed model can achieve comparable or improved performance a vanilla BPE-token Transformer baseline, although performance improvements on some tasks (e.g., MMLU) require more training compute than the BPE baseline.", "questions": "Specific questions:\n\nC.f. Sec 4, the key innovations compared to previous work on byte-level LLMS, appear to be 1) the more semantically motivated splitting function, and 2) training at larger scale compared to previous work (e.g., [4]) which seems to be important for byte-based models (cf. Fig. 3).\n\nI think the paper would be *greatly* strengthened if it more explicitly addressed 1 by including explicit comparisons of the proposed word-based splitting to simpler baselines\n  (e.g., fixed window size at each level, as previously done in the literature), keeping the architecture, scale, and training data constant.\n  Acknowledging of course that this might be difficult to achieve in limited time, such a comparison to quantify the importance of the proposed splitting function would better highlight the novel aspects of the paper and and strengthen the its contribution.\n\nSimilarly, although less critical, an assessment of a byte level transformer baseline (i.e., no hierarchy) missing from the key results in table 1.\n\nMinor issues:\n\n- Contribution C2 is a bit overstated.  This advantage should apply to any \"byte-level\" language model, and is not unique to this work.  E.g., Byt5 (https://arxiv.org/abs/2105.13626) or cited baselines such as MegaByte [17], BLT [5], etc.\n\n- I'm confused by the $\\gamma_{byte}$ equation after line 147.  Why is there a fixed scaling factor (1/k) between F_{model/byte} and F_{model/token}?  Shouldn't this be a function of the actual architecture being used (as in the following paragraph about AU-Net)?\n\n- The text uses somewhat misleading language about the representation at deeper levels of the hierarchy \"predicting further ahead\", similar to multi-token prediction (Fig. 1 and Sec. 1), which is, to my understanding, imprecise.\n  It seems true that the pooled representations in deeper stages can more efficiently take longer context into account -- they essentially effectively broader \"receptive field\" per token compared to shallower stages.\n  But, assumiung fully causal self attention and pooling/expansion (as show in Fig. 1), a network trained only to do next-byte-token prediction is not actually predicting any further *ahead*.  To the contrary, it is looking further back (per-timestep).\n\n- Table 1 appears to be mislabeled: the AU-Net 1 and 2 rows contain the same results as the rows labeled AU-Net 2 and 3, respectively in Table 2.\n\n- The notation used in the \"Hyperparameter scaling laws\" section (lines 158-164) is not clearly defined.  What is BSZ (batch size?)?\n  - Also, the first mention of DCLM (line 144) should probably include a reference.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Clearly written\n- Novel and well-motivated data-dependent \"splitting function\" approach\n- Strong performance: roughly comparable to traditional BPE baseline when using a single level of splitting (at word boundaries), improving with increased levels of hierarchy\n- Comprehensive evaluation, including scaling analysis\n\nWeaknesses\n- Experiments feel a bit incomplete: there are no explicit comparisons between different splitting functions, e.g., to a fixed-window-size baseline.  Given that one of the primary contributions is the use of content-adaptive splitting (as byte-level LLMs have been trained before), it would be informative to quantify the importance of the choice.\n- The proposed splitting approach is specific to languages which are easily split into words, it is unclear how to generalize well to broader language where this is more difficult such as Chinese or Japanese.", "was_revised": true, "final_justification": "In their rebuttal the authors committed to including a fair comparison to a fixed-window baseline in the final camera-ready draft, and responded to my other minor concerns.  In light of this I've increased my rating to 5.", "rebuttal": "We thank the reviewer for this  careful and constructive review, and answer below the questions raised.\n---\n\n> **I think the paper would be greatly strengthened if it more explicitly addressed 1 by including explicit comparisons of the proposed word-based splitting to simpler baselines (e.g., fixed window size at each level, as previously done in the literature), keeping the architecture, scale, and training data constant.**\n  \nWe agree this is important. We had early runs that favored dynamic, content‑adaptive pooling, so we allocated more compute there. If the paper is accepted, we commit  to include a fixed‑window comparison under the same architecture/scale/data by camera‑ready, as running these experiments will take time.\n\n> **The proposed splitting approach is specific to languages which are easily split into words, it is unclear how to generalize well to broader language where this is more difficult such as Chinese or Japanese.**\n  \nWe agree. Our current scope focuses on languages with a clear word segmentation. As acknowledged in the limitations section, extending our method to scripts without explicit word boundaries (e.g., Chinese, Japanese, Thai) remains an open question.\n\n> **Similarly, although less critical, an assessment of a byte level transformer baseline (i.e., no hierarchy) missing from the key results in table 1.**\nTable 2 has a byte level (no hierarchy) comparison for a smaller amount of data (80B tokens). Running a byte level model on 370B llama3 tokens would be more than 4 times as expensive than AU-Net or BPE model, which would be too much compute for a model of this size.\n> **Contribution C2 is a bit overstated. This advantage should apply to any \"byte-level\" language model, and is not unique to this work. E.g., Byt5 (https://arxiv.org/abs/2105.13626) or cited baselines such as MegaByte [17], BLT [5], etc.**\n  \nAgreed—this advantage is common to byte‑level models (e.g., ByT5, MegaByte, BLT). By showing parity with a strong BPE baseline with minimal trade‑offs, we provide concrete evidence of its practical value. We will soften C2 accordingly.\n>  **equation after line 147. Why is there a fixed scaling factor (1/k) between F_{model/byte} and F_{model/token}? Shouldn't this be a function of the actual architecture being used (as in the following paragraph about AU-Net)?**\n\nThe equations map token‑level to byte‑level scaling for any byte‑level LM. They are architecture‑agnostic: each model can have a different FLOPs/byte, but the conversion itself doesn’t depend on the specific architecture. We will clarify this point.\n> **The text uses somewhat misleading language about the representation at deeper levels of the hierarchy \"predicting further ahead\", similar to multi-token prediction (Fig. 1 and Sec. 1), which is, to my understanding, imprecise. It seems true that the pooled representations in deeper stages can more efficiently take longer context into account -- they essentially effectively broaden \"receptive field\" per token compared to shallower stages. But, assuming fully causal self attention and pooling/expansion (as shown in Fig. 1), a network trained only to do next-byte-token prediction is not actually predicting any further ahead. To the contrary, it is looking further back (per-timestep).**\n\nYou are correct that when taken as a whole, our model only does next-byte prediction. However, outputs of deeper stages influence the prediction of multiple bytes in the future. All layers are causal but at different levels of granularity. Stage 1 outputs predictions that directly predict the next byte, Stage 2 outputs vectors that help the prediction of the next word, Stage 3 outputs vectors that help the prediction of the next two words. In this sense, Stages 2 and 3 have to predict further in the future. Finally, stages 2 and 3 represent the vast majority of the total compute of the model, which forces it to rely less on short term byte level predictions and more on word and word group level predictions.  We will clarify this section to make it more precise.\n> **Table 1 appears to be mislabeled: the AU-Net 1 and 2 rows contain the same results as the rows labeled AU-Net 2 and 3, respectively in Table 2.**\n\nThank you, we will fix this.\n> **The notation used in the \"Hyperparameter scaling laws\" section (lines 158-164) is not clearly defined. What is BSZ (batch size?)?**\n\nWe will define BSZ (“batch size”) on first use. \n> **Also, the first mention of DCLM (line 144) should probably include a reference.**\n\nWe will cite DCLM at its first mention."}, {"review_id": "ZKVN8lQYMj", "reviewer": "Reviewer_R5bc", "summary": "Token based LLM's by design add biases towards languages due to the fixed pretrained vocabulary.\nThe paper follows a series of byte/ fixed-word-split processing and lifts findings to a multi-stage hierarchical solution, where the most-inner block, on which most parameters and compute are spent, aggregates even several words.\nAs such it is a significant improvement towards unbiased (i.p. w.r.t. low-ressource languages) and more efficient LLMs.", "questions": "- Q1 please address above's weaknesses\n- Q2 i seem to miss sth-  in table 2 you show 51% MMLU on AU-net2 at 1e22, compared to baseline 49.2 at 9e21, but it appears to look worse in figure 3 or am i just underestimating the exp-scale axis??", "limitations": "as written in W3, please add code to limitations -\nother fixed word-split rules showed degradation in math and code, for which it might be hard to find adequate regex-split rules (however you might outperform current SOTA with your 4-word aggregation? unfortunate that you couldn't show that.)", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- S1 very well written, visualized, easy to follow\n- S2 thorough experiment series, claim of some stable scaling law hyperparameters (albeit i'm missing appendix w/ extensive discussion)\n- S3 code is gonna be made public\n- S4 highly relevant field towards efficient/ unbiased LLMs\n\n\n- W1  a/ Results of Table 1 should be discussed somewhere\n       b/ I don't seem to find descriptions of what AU-Net 1/2/3/4 are precisely (with dimensions)\n- W2 a/ you claim to find stable optimization hyperparameters (line 158ff) but i can only see some theoretic deviation - i expect further discussion in appendix with sweep results. \n     b/ i might have missed it but you do not seem to argue how you derive to your embedding dimensions - how/why did you choose 512/ 2048/ 3072 in Fig1 and 3 layers for the outer ones? any discussion of evidence on optimality from sweeps etc highly appreciated. Is this not required for a/ as well to be able to argue abt stable parameters?\n- W3 please add \"code\" to limitations/discussion\n- W4 why do you not show AU-net 3/4 for 8b in table 2?", "was_revised": true, "final_justification": "the research is a solid addition to the 'subword-trained tokenizer free LLM's' research domain.\nit is not groundbreaking or overly novel but still important and potentially impactful shaping the future of more efficient LLM architectures. \nas such i keep my initial solid accept rating", "rebuttal": "We thank the reviewer for this  careful and constructive review, and answer below the questions raised.\n---\n> **a/ Results of Table 1 should be discussed somewhere**  \n> **b/ I don't seem to find descriptions of what AU-Net 1/2/3/4 are precisely (with dimensions)**\n\nTable 1 was intended as a quick summary. We’ll add a short discussion in the introduction to contextualize the numbers and summarize key take-aways. AU-Net 1 was a typo and has been updated to AU-Net 2.\nAU‑Net 2/3/4. Architectural details are delegated to the Appendix due to lack of space, but we will add some basic details (such as number of layers and embedding dimensions) to the main text.\n\n> **a/ you claim to find stable optimization hyperparameters (line 158ff) but i can only see some theoretic deviation - i expect further discussion in appendix with sweep results.**  \n\nThank you for pointing this out—we will add the hyperparameter sweep results and further discussion in the Appendix.  \n\n> **b/ i might have missed it but you do not seem to argue how you derive to your embedding dimensions - how/why did you choose 512/ 2048/ 3072 in Fig1 and 3 layers for the outer ones? any discussion of evidence on optimality from sweeps etc highly appreciated. Is this not required for a/ as well to be able to argue abt stable parameters?**\n\nOur first ablations revealed that the first level does not need to be large, while later levels benefit from more capacity, so that more compute is spent on increasingly abstract embeddings. This informed choices like 512/2048/3072 and three outer layers (see Appendix C.2). We will explain further the rationale behind first stage embedding size in the Appendix.\n\n> **please add \"code\" to limitations/discussion**\n\nWe will add an explicit limitation regarding code/math. We focused on DCLM to enable clean comparisons. Training this architecture on more diverse code and math data to evaluate reasoning and generation‑heavy workloads is an exciting direction for future work.\n\n> **why do you not show AU-net 3/4 for 8b in table 2?**\n\nDue to our limited compute budget.. We agree these experiments are valuable and will add them when resources permit.\n\n> **please address above's weaknesses**\n\nCovered above: we will (i) discuss Table 1 in the intro, (ii) add architectural details and dimensions for AU‑Net 1–4, (iii) include sweep results and discussion of hyperparameter stability and design choices in the appendix, (iv) add the code/math limitation, and (v) clarify the compute constraint behind missing AU‑Net 3/4 at 8B.\n\n> **i seem to miss sth- in table 2 you show 51% MMLU on AU-net2 at 1e22, compared to baseline 49.2 at 9e21, but it appears to look worse in figure 3 or am i just underestimating the exp-scale axis??**\n\nThe models at 7B in Table 2 operate in a different data-to-model ratio (5 in Table 2 and 10 in scaling laws), and the scaling laws had the most recent hyperparameter sweep values, resulting in better performance for the baseline and slightly better for AU-Net. MMLU is especially sensitive to learning rate. (All the detailed hyperparameters are in Appendix D.) We will adjust the table captions accordingly."}, {"review_id": "XuiHp3LACb", "reviewer": "Reviewer_T1Se", "summary": "The paper introduces AU-Nets, a family of hierarchical language models intended to enable byte-level language modelling.\n\nIn contrast to some prior hierarchical architectures [[1], [2]], the authors adaptively chunk the input bytes based on a Regex instead of using fixed-length chunks and generalize the architecture beyond a two-stage hierarchy (AU-Net-2) to higher-level hierarchies (AU-Net-3 and AU-Net-4).\n\nExperiments on pretraining AU-Net models up to 8B parameters show that they perform roughly on par or slightly worse than models using subword tokenization trained in a compute-matched way.\n\n[1]: https://arxiv.org/abs/2110.13711\n[2]: https://arxiv.org/abs/2305.07185", "questions": "See Weaknesses.", "limitations": "yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Research on direct byte-level language modelling is important due to the many inadequacies of subword tokenization and potentially highly impactful.\n- Experiments are comprehensive and have a strong baseline.\n- The paper is mostly well written and easy to follow.\n\nWeaknesses:\n- Although scaling the levels of the hierarchy (AU-Net-3 and AU-Net-4) slightly improves performance in a FLOP-matched training setup, it seems like a problematic dimension to scale due to the vastly increased parameter count (1.3B for AU-Net-2 vs. 4.2B for AU-Net-4 at slightly improved task performance).\n- It is not clear how AU-Net-2 differs from prior work using adaptive pooling heuristics, e.g. the whitespace-split setting of BLT [[3]] and DTP [[4]] (DTP should also be referenced). BLT is only shown to be worse empirically, but there is no reason given as to why this may be the case. There is also no controlled comparison against these prior byte-level architectures. Is the only substantial difference the Regex splitting (vs. whitespace)? Or could the superior performance of AU-Nets be caused by other spurious factors related to differences in the training setup?\n\n[3]: https://arxiv.org/abs/2412.09871\n[4]: https://arxiv.org/abs/2211.09761", "was_revised": true, "final_justification": "I am maintaining my recommendation to Reject since:\n\n- Per the authors, the main contribution of the paper is showing 'that going deeper with hierarchical architectures can work under the right circumstances and has advantages'. I do not think this is sufficiently well supported. (1) we already know from prior work (e.g. [[1]]) that a two-stage architecture can be beneficial and (2) the authors' results for going beyond two stages seem discouraging to me, in particular in Table 2 we see that models with three or four stages perform at best slightly better at vastly increased parameter counts, so if we want improved performance at matched FLOPs it seems like a better choice to use a MoE instead of a three or four stage hierarchy (a comparison the paper doesn't make).\n- The auxiliary contributions (faster inference, improved performance over BLT) either lack empirical evidence or are confounded by spurious factors such as different training data.\n\n[1]: https://arxiv.org/abs/2412.09871", "rebuttal": "We thank the reviewer for this constructive review, and answer below the questions raised.\n\n\n---\n\n\n> **Although scaling the levels of the hierarchy (AU‑Net‑3/4) slightly improves performance in a FLOP‑matched setup, it seems like a problematic dimension to scale due to the vastly increased parameter count (1.3B for AU‑Net‑2 vs. 4.2B for AU‑Net‑4) at only slightly improved task performance.**\n\nYou are right that adding stages increases the number of parameters and therefore GPU memory and checkpoint/storage size. However, it comes with other advantages, explained below.\n\n- Because we match FLOPs per byte to the baseline, the training and inference compute per generated byte are the same. Extra parameters live mostly in higher stages that process shorter, downsampled sequences, so they add capacity without increasing per‑byte compute. \n\n- **Inference can be faster** at a fixed quality target because the total KV cache can be smaller: upper stages cache over much shorter sequences, which reduces aggregate KV memory/traffic and can improve inference time in practice. (Inference is memory-bound)\n\n- AU-Net decouples parameters from compute cost. A flat Transformer’s cost is often approximated as ≈ 6 × (#params) × (#tokens). In AU‑Nets, this is no longer the case; by varying depth, one can choose a **larger‑parameter model at the same per‑byte cost** (assuming GPU memory is not a big issue).\n\n- **Pooling extends effective context**: deeper hierarchies handle longer inputs at the same compute budget and can predict further ahead. We do not claim definitive wins on such long‑horizon tasks yet; our benchmarks are mostly “classic LLM” and underweight long context and reasoning/generation. We plan to apply AU‑Nets to code/math settings where this matters more in the future.\n\n- Finally, in **high data‑to‑model (“overtrained”) regimes**, we observe a **positive trend**: deeper AU‑Nets behave better, with AU‑Net‑4 competitive with or slightly better than AU‑Net‑2/3. This is encouraging since most, if not all, state-of-the-art models are heavily overtrained.\n\n\n> **It is not clear how AU‑Net‑2 differs from prior work using adaptive pooling heuristics, e.g., BLT (whitespace split) and DTP (which should be referenced). BLT is only shown to be worse empirically, without a reason. No controlled comparison. Is Regex splitting the only substantial difference, or could gains be from training‑setup differences?**\n\nWe should be clearer in explaining that regex splitting is not the only difference. AU‑Nets differ along two architectural axes and one optimization axis:\n- Contextual downsampling: BLT embeds each patch/word independently (local pooling), whereas AU‑Nets use sliding‑window attention to form higher‑level tokens, allowing cross‑boundary context before downsampling and reducing boundary‑induced information loss.  \n- Simple, stable upsampling: AU‑Nets use linear upsampling without attention, which is more stable in byte‑level training in our experience.  \n- A practical contribution of our work is byte‑level–specific optimization guidance. Stable, high‑quality byte‑level pretraining benefits from different scaling for learning rate and batch size than subword models (e.g., more conservative LR and larger global batch/warmup). We consider this to be an important finding that contributes to having a model competitive with BPE.\n\nOn comparisons: we compute‑match pretraining, train a strong byte‑level baseline under the same optimizer/schedule family, and report results alongside published BLT and Hierarchical‑Transformer, both of which used DCLM as part of their pretraining data. While we did not re‑train BLT/DTP/Hierarchical‑Transformer under our hyperparameters due to compute constraints, using similar pretraining data provides a fair and informative comparison. We will add the missing DTP citation; to our knowledge, DTP does not report directly comparable downstream evaluations on any benchmark."}]

Author rebuttal:

Meta-review: ***Summary***

The authors introduce a hierarchical transformer called AU-Net, with multiple layers of hierarchy.  Stage 1 operates on bytes/characters.  Characters are pooled into words, using an external segmenter, which segments on spaces.  In contrast to some previous published work, which uses fixed-stride patches, AU-Net allows words to vary in length.  Stage 2 then operates on words; it is a larger transformer model, with a larger embedding dimension and greater parameter count.  Stages 3 and 4 operate similarly, but are fixed-stride; they pool words into pairs of words or quad-word chunks, since no segmenter is available to find meaningful phrase boundaries.  After the encoding/pooling stages, there are a series of decoding/unpooling stages, modelled after the U-Net architecture, with skip connections between encoder and decoder.

The authors compare their architecture against a standard transformer using byte-pair-encoding (BPE), and are able to match or slightly exceed the BPE baseline.  

***Meta-review***

The reviewers were split, with ratings (2, 5, 5, 5).  Although most reviewers voted to accept, reviewer T1Se (rating 2) actually had a detailed and insightful review in my opinion.  I personally would give this paper a rating of 4 -- "borderline accept".  

Selected comments from the reviewers can be found below.  Most reviewers were impressed by the experimental results, as was I.  However, there was broad agreement that the use of a word-splitting function based on whitespace was a major limitation, since it does not work for languages like Chinese or Japanese.  There was also broad agreement that there should be comparisons against additional baselines, particularly against other hierarchical transformers, of which several have been published in the literature.  I agree wholeheartedly with both of these criticisms.  

***AC opinion***

This is an area of research that I am actually quite familiar with, so I would like to make the following additional observations.  On the positive side, this paper is well-written, and the experiments were carefully done.  The experimental results are decent, matching the BPE baseline even when controlling for parameter count, which I have not seen before.  I recommend "accept" mainly on the strength of those results.  

Moreover, the limitations of BPE tokenizers are also well known, and are a major source of frustration across the industry, so a truly good "neural tokenizer" would be an extremely high-impact paper, worthy of a spotlight or oral.  Unfortunately, this is not such a paper, for several reasons.

First, the novelty of the author's architecture is very limited.  Hierarchical transformers have been proposed before.  Transformers that segment on variable word lengths have been proposed before (e.g. Neitemeier et. al. ICLR 2025).  UNet-style architectures have been proposed before.  This paper has, at best, minor difference in pooling/unpooling from prior published work.  The author's "related work" section is extremely minimal; it cites the necessary literature, but does not provide much discussion.  Much like the reviewers, I would really like to see a comparison against this prior work.  

Second, there are hardly any ablations.  A paper which did a proper survey of the design space, comparing fixed-stride/variable-stride, different segmenting strategies, and different pooling/unpooling strategies would be much more impressive.  Unfortunately, the authors do no such thing.  They do not even compare their whitespace segmentation against an entropy-based or BPE segmentation, which previous publications have done.  Moreover, as the reviewers mention, whitespace segmentation only works on European languages, and thus has limited applicability.  

Third, the major implementation issue with variable-stride segmentation is that batched inference becomes problematic, because the word boundaries do not align between different sequences in the batch.  The authors do not even discuss this point as a limitation.  Unfortunately, none of the reviewers mentioned this either, but it would be a major obstacle to deploying AU-Net in the real world.  

Fourth, while it's great that the authors test on downstream tasks, I am not fully convinced.  I would actually like to see perplexity numbers as well, converted to bit-per-character, since that is where I've seen previous efforts at tokenizer-free models fall short.  

***Selected comments from reviewers.***

Reviewer SrNB (rating 5): 
* "The paper shows strong results in multilingual settings despite using an English-centric dataset (DCLM)"
* "Although acknowledged by the authors, the lack of support for non-space-delimited languages represents a fundamental architectural limitation."

Reviewer Wegy (rating 5):
* "Strong performance: roughly comparable to traditional BPE baseline when using a single level of splitting"
* "Experiments feel a bit incomplete: there are no explicit comparisons between different splitting functions, e.g., to a fixed-window-size baseline."
* "The proposed splitting approach is specific to languages which are easily split into words, it is unclear how to generalize well to broader language where this is more difficult such as Chinese or Japanese."

Reviewer R5bc (rating 5):
* "the research is a solid addition to the 'subword-trained tokenizer free LLM's' research domain. it is not groundbreaking or overly novel but still important and potentially impactful"

Reviewer T1Se (rating 2):
* "The paper is mostly well written and easy to follow."
* "It is not clear how AU-Net-2 differs from prior work using adaptive pooling heuristics, e.g. the whitespace-split setting of BLT [3] and DTP [4] (DTP should also be referenced)."
* "From the paper, it seems like AU-Nets vastly outperform BLT with similar FLOPs/token budgets. My question is: where is this coming from?"
* "Although scaling the levels of the hierarchy (AU-Net-3 and AU-Net-4) slightly improves performance in a FLOP-matched training setup, it seems like a problematic dimension to scale due to the vastly increased parameter count...  if we want improved performance at matched FLOPs it seems like a better choice to use a MoE instead"
* "(1) we already know from prior work (e.g. [1]) that a two-stage architecture can be beneficial and (2) the authors' results for going beyond two stages seem discouraging to me"
Decision: Accept (poster)