Devatri/escape-blt-entropy-student

ESCAPE: distilled BLT entropy patcher (14.3M)

A small byte-level language model trained to imitate the entropy model ("patcher") of the Byte Latent Transformer (BLT). The teacher's per-byte next-byte entropy decides where BLT starts a new patch; this student reproduces those entropies, and so those patch boundaries, at about one seventh of the size.

It was built for Project ESCAPE (Team Trimax, ANLP, IIIT Hyderabad, October 2026), which asks whether BLT's entropy-triggered patch boundaries fall on syntactic constituent starts in Python, C++ and English. Code, report and every result: https://github.com/dawnbliss-coder/ESCAPE-Entropy-Semantic-Compute-Allocation-in-Programming-Environments

Model

Architecture	StudentBLT (escape_distill/student_blt.py): pre-norm byte transformer, RMSNorm, interleaved RoPE, SwiGLU, tied embeddings
Size	14,262,144 parameters; dim 384, 8 layers, 6 heads
Vocabulary	260 (256 bytes + 4 BLT special tokens; byte b is token b + 4, BOS = 1)
Context	BOS + up to 512 bytes
Teacher	itazap/blt-1b-hf @ 91aa6b8e, the 99.5M BLT patcher, float32, with the reference 512-byte sliding window restored

Training

Online knowledge distillation: loss = α·CE(student, true byte) + (1−α)·T²·KL(p_teacher/T ‖ p_student/T) with α = 0.1 and T = 2. Two epochs (6,860 steps), batch 48 windows of ≤512 bytes, AdamW (lr 5e-4, weight decay 0.1, 200 warm-up steps), seed 20260927, one RTX 2080 Ti.

Data: a 29,000-file extension corpus disjoint from the ESCAPE evaluation set (CodeSearchNet Python functions from repositories not used in evaluation, Stack C++ from other repositories, WikiText-103 paragraphs from unused articles), split by repository or article. Training used 27,153 files (164,627 windows). train_log.json has the full loss curve.

Results (from the ESCAPE final report)

Agreement with the teacher on held-out extension windows (distill_eval.json):

Model	Boundary F1 vs. teacher	Bits/byte
This student (α = 0.1)	0.871	1.24
Same model trained without the teacher (α = 1)	0.789	1.17
A boundary at every word start	0.475	–
Teacher	–	1.14

Alignment with constituent starts on the frozen held-out split (6,400 Python, 6,400 C++, 240 English files), scored by the same protocol as the teacher at τ = 1.3354, k = 0:

	Python	C++	English
Start-F1 margin over a density-matched null	+0.109	+0.119	+0.304
Share of the teacher's margin kept	91%	92%	87%
Student − teacher [95% CI]	−0.011 [−0.011, −0.010]	−0.011 [−0.012, −0.009]	−0.044 [−0.048, −0.040]
Margin at the student's own pre-registered τ* (2.0 / 2.0 / 2.5)	+0.147	+0.156	+0.460

Like the teacher, the student stays below simple structure-blind baselines (indentation changes in Python, word starts in C++ and English). Its forward pass is 5.0× faster than the teacher's on an RTX 2080 Ti, and 4.8× faster under the rolling 512-byte extractor.

Use

Clone the ESCAPE repository, then:

import torch
from huggingface_hub import hf_hub_download
from escape_distill.student_blt import StudentBLT, StudentConfig

ckpt = torch.load(hf_hub_download("Devatri/escape-blt-entropy-student", "student.pt"),
                  map_location="cpu", weights_only=False)
model = StudentBLT(StudentConfig(**ckpt["config"])).eval()
model.load_state_dict(ckpt["model_state"])

model.safetensors holds the same weights (load it with safetensors.torch.load_model into a StudentBLT built from config.json["model_config"]). To extract per-byte entropies and patch boundaries for files, use python -m escape_distill.extract from the repository.

Files

File	Contents
student.pt	The checkpoint exactly as trained (model_state, config, train_args, steps)
model.safetensors	The same weights in safetensors format
config.json	Model configuration and training arguments
train_log.json	Training loss, KL and CE every 100 steps
distill_eval.json	Held-out agreement with the teacher, per domain

Limitations

One model size and one seed. The student sees at most 512 bytes of context, which explains about 40% of its deficit to the teacher on code; capacity explains the rest. It was evaluated only with the rolling 512-byte extractor. No licence has been chosen for this model; the training data keep their own licences (CodeSearchNet, The Stack's terms of use, WikiText-103 under CC BY-SA).

Citation

Teacher model: Pagnoni et al., 2025, Byte Latent Transformer: Patches Scale Better Than Tokens, ACL 2025.