Diogenes-0.1-20M

Diogenes-0.1-20M is a small byte-level language model (20.3M parameters) built on the Recurrent Looped Transformer (RLT) architecture. It reads and writes raw UTF-8 bytes, so it needs no tokenizer. It is a research model, not an assistant.

What is RLT?

A standard transformer predicts each token only from the tokens before it. RLT adds a feedback loop: at every step, the model's own final internal state from the previous step is fed back into it through a learned gate. The model can therefore carry a running "state" from step to step, like a recurrent network, while keeping a transformer's attention memory over the text.

• Encoder memory: attention over the whole context window gives the model a memory of what it has read.
• Windowed decoder with feedback: the decoder looks at only the last few positions plus its own fed-back state, so the cost of each new step stays constant.
• Closable gate: with the gate closed the model reduces to an ordinary feed-forward transformer, so the effect of the feedback can be measured directly.

Which problems does it address?

• State and memory together. Some tasks need the model both to track a changing state and to look up facts it saw earlier. On such a task, the feedback model learned it, while GRU, LSTM and the same model without feedback did not.
• Better prediction from the same architecture. Against an identical twin trained without feedback, Sable predicts text better (1.3745 vs 1.4408 bits per byte) and scores higher on the BLiMP grammar benchmark (64.83% vs 61.90%).
• Constant cost per generated byte. Generation keeps a fixed-size state instead of re-reading everything at each step.

Benchmarks

benchmark	Diogenes-0.1-20M
validation bits per byte (lower is better)	1.3745
BLiMP accuracy (chance 50%)	64.83%
key-value retrieval, trained keys (exact)	100.0%
key-value retrieval, held-out keys (exact)	74.4%
key-value retrieval, natural keys (exact)	97.9%
single fact at distance 0-448 bytes (exact)	99.7-100%
parameters	20,297,472
context length	in theory infinite

Usage

python generate.py --prompt "Once upon a time" --max-new-bytes 128

Research code

The full research record (code, experiments and results): https://github.com/berkmec/recurrent-looped-transformer-research