Rondo-Exp

A proof of concept, published for comparison only. Rondo-Exp is a 352M-parameter masked-diffusion language model, trained side by side with its autoregressive twin Monody-Exp to compare how each uses the same data over repeated passes. It is a small base model: it continues text, follows no instructions, and is not meant for real use.

Both models trained on the same 88M FineWeb-Edu tokens, repeated 7 times (617M tokens), with 2,048-token windows and identical optimizer settings. These are the EMA weights after pass 7; held-out loss is on 101M unseen tokens.

	Rondo-Exp	Monody-Exp
Objective	masked diffusion	next token
Token mixers	12 bidirectional GDN-2, 4 bidirectional attention	12 GDN-2, 4 Parallax attention
Held-out loss	≤ 3.810 (ELBO bound)	3.632 (exact NLL)
Held-out minus training loss	0.13	1.59
Zero-shot accuracy, mean of 6 tasks	44.8%	45.6%

The six tasks are LAMBADA, ARC-Easy, PIQA, SciQ, HellaSwag and WinoGrande (1,000 examples each). Rondo's loss is an upper bound on its NLL, so it is not directly comparable to Monody's.

Run

Linux with an NVIDIA GPU:

hf download KerzgrAI/Rondo-Exp --local-dir Rondo-Exp
pip install -r Rondo-Exp/requirements.txt
python Rondo-Exp/inference.py --prompt "The water cycle is"

The first run installs flash-linear-attention if it is missing. Options: --max-new-tokens, --temperature (0 is greedy), --top-p, --top-k, --seed, and --drafts (tokens drafted and checked per forward; 0 reveals one token per forward).