OrisTeam/Trillium-2.137M

Trillium 2.137M

Trillium is a tiny Polish causal language model with exactly 2,137,000 parameters, a 2,137-token vocabulary, and a training target of 2.137 billion tokens.

Yes, the numbers are intentional.

2.137B training tokens / 2.137M parameters = 1000 tokens per parameter

Trillium came out of lessons from the Architecture Model Zoo.

The architecture itself was much less profound.

The general idea was basically:

How much can we compress the parameter count if we stop caring about spending extra compute?

Two actual Transformer blocks, reused five times each?

¯_(ツ)_/¯

Okay. That's it.

Architecture

The core is only two physical Transformer blocks:

A → B → A → B → A → B → A → B → A → B

That gives ten logical passes, but this is not equivalent to a conventional 10-layer Transformer. The model gets more compute depth while reusing almost all of the same block parameters.

A useful comparison point is Glint-2, a 1.71M-parameter model built around an even more aggressive version of the same general idea: one Transformer block looped eight times. Glint-2 also adds per-loop conditioning/adaptation, while Trillium uses two alternating shared blocks with a small learned depth controller.

So Trillium is not claiming recurrent looping as a new idea. It simply landed in a similar design space from the parameter-compression side:

Glint-2:   A → A → A → A → A → A → A → A
Trillium:  A → B → A → B → A → B → A → B → A → B

For Glint-2's interactive inference/effort explorer, see: Glint-2 Effort Explorer.

Property	Value
Parameters	2,137,000
Vocabulary	2,137
Hidden size	256
Physical Transformer blocks	2
Logical recurrent passes	10
Reuse per block	5×
Query heads / KV heads	8 / 2
Head dimension	32
Attention	GQA
FFN width	768
FFN factorization rank	244
Vocabulary factorization rank	128
Positional encoding	RoPE
Normalization	RMSNorm
Activation	SwiGLU
Training context	512
Training target	2.137B tokens
Target tokens / parameter	1000

The vocabulary interface is factorized and tied between input embeddings and output logits. The FFN is factorized as well. Attention keeps full-width queries while compressing K/V with GQA.

A small learned depth controller gives individual recurrent passes some identity, but the basic limitation remains obvious: ten passes still share only two Transformer blocks.

This model was built for aggressive parameter compression, not inference efficiency. Compute was cheap in the design; stored parameters were not.

Training

The Polish run was mostly a smoke test: throw the architecture at real data and see whether it learns anything useful.

It does train.

Cross-entropy starts around the random-vocabulary baseline and keeps dropping:

Tokens seen	Training CE, approx.
start	7.66
10M	5.69
25M	4.84
50M	4.04
100M	3.69
150M	3.56
200M	3.30
300M	3.21
400M	3.15

The full run targets 2.137B tokens.

Benchmarks

Likelihood-based evaluation is noticeably kinder to Trillium than free generation.

An intermediate OpenPL evaluation reached a 27.59 composite score, showing that even at this size the model can learn useful preference and likelihood structure.

That should not be confused with being a good text generator.

Generation

Free generation makes the model's limitations much easier to see.

Trillium can produce locally plausible Polish. It often gets morphology, sentence shape, punctuation and document style roughly right.

The problem is that it frequently has almost no idea what topic it is supposed to stay on.

A prompt can begin in the correct semantic neighborhood and drift somewhere completely unrelated a sentence later.

So, roughly:

It learned what Polish text looks like before it learned what the hell it is talking about.

Which is not especially shocking for a 2.137M-parameter model repeatedly throwing its hidden state through the same two blocks.

The point of Trillium is not low FLOPs, low latency, or production-quality generation. It is an intentionally extreme parameter-compression design that spends repeated computation instead of storing more independent layers.

The Polish run is the first smoke test. A later English run is intended to push the same general idea much harder and see where the architecture actually tops out.

Limitations

Trillium is a raw base language model, not an instruction-tuned assistant.

Expect:

• WEAK long-range semantic coherence,
• abrupt topic drift,
• repetition,
• factual hallucinations,
• web-like or scraped-document artifacts,
• much stronger likelihood / multiple-choice behavior than free generation,
• generally poor suitability for production or high-stakes use.

Citation

@misc{OgrodzkiTrillium2026,
  author       = {Aleksander Ogrodzki},
  title        = {Trillium 2.137M},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/OrisTeam/Trillium},
  note         = {Model architecture, training data pipeline, tokenizer pipeline, and model training by the author}
}