comp-vs-like-1B-topdowncomp-multi-128k

A 1.2B-parameter Llama-style language model trained with the TopDownComp tokeniser, from Objective vs. Search: Decomposing What Makes a Good Tokeniser (EMNLP 2026).

The paper separates two axes that BPE and UnigramLM confound. This model occupies the cell:

	Compression objective	Log-likelihood objective
Bottom-up merging	BPE	BottomUpLL
Top-down pruning	TopDownComp	UnigramLM

This model: TopDownComp — compression objective, top-down pruning.

All models in the collection are identical except for the tokeniser, so differences in bits-per-byte and BLiMP are attributable to tokeniser design alone.

Training

Parameters	1.23B (2048 hidden, 22 layers, 32 heads, 4 KV heads, tied embeddings)
Tokeniser	topdowncomp-multi-128k, vocabulary 128,000
Corpus	FineWeb-2 (English + German, Spanish, Turkish, Chinese)
Tokens	20B (76,293 steps x 128 sequences x 2048 tokens)
Sequence length	2048
Optimiser	AdamW (0.9, 0.95), weight decay 0.1, grad clip 1.0
LR	6e-4, 2000 warmup / 2000 decay steps, min ratio 0.01
Precision	bf16
Seed	42

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("AhmetcanYvz/comp-vs-like-1B-topdowncomp-multi-128k")
model = AutoModelForCausalLM.from_pretrained("AhmetcanYvz/comp-vs-like-1B-topdowncomp-multi-128k")

ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=20)[0]))

The tokeniser is bundled with the model, so no separate download is needed.

Caveats

This is a research model trained on 20B tokens to study tokenisation. It is not instruction-tuned, not filtered for safety, and not intended for deployment.

Code

https://github.com/Ahmetcanyvz/comp-vs-like

Citation

@inproceedings{yavuz2026objective,
  title     = {Objective vs. Search: Decomposing What Makes a Good Tokeniser},
  author    = {Yavuz, Ahmetcan and Meister, Clara and Pimentel, Tiago},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026}
}