Byte Latent Transformer: Patches Scale Better Than Tokens
Authors: name: Artidoro Pagnoni; name: Ram Pasunuru; name: Pedro Rodriguez; name: John Nguyen; name: Benjamin Muller; name: Margaret Li; name: Chunting Zhou; name: Lili Yu; name: Jason Weston; name: Luke Zettlemoyer; name: Gargi Ghosh; name: Mike Lewis; name: Ari Holtzman; name: Srinivasan Iyer
Published: 2024-12-13 05:33:32

Abstract: We introduce the Byte Latent Transformer (BLT), a new byte-level LLM
architecture that, for the first time, matches tokenization-based LLM
performance at scale with significant improvements in inference efficiency and
robustness. BLT encodes bytes into dynamically sized patches, which serve as
the primary units of computation. Patches are segmented based on the entropy of
the next byte, allocating more compute and model capacity where increased data
complexity demands it. We present the first FLOP controlled scaling study of
byte-level models up to 8B parameters and 4T training bytes. Our results
demonstrate the feasibility of scaling models trained on raw bytes without a
fixed vocabulary. Both training and inference efficiency improve due to
dynamically selecting long patches when data is predictable, along with
qualitative improvements on reasoning and long tail generalization. Overall,
for fixed inference costs, BLT shows significantly better scaling than
tokenization-based models, by simultaneously growing both patch and model size.