Nomade-15M-fr 🧭

⚠️ Preliminary results — 1 seed, variance not controlled. Every number below is a single training run evaluated once (50 prompts × 200 tokens, fixed seed). Directions are consistent, magnitudes are not validated. Treat as a lab notebook, not a benchmark.

Nomade is a 15M-parameter French language model trained from scratch (no 🤗 Transformers, no pretrained weights) on a single GTX 1080 Ti. It is the out-of-domain / robust member of a three-model family: a looped transformer with learned per-token adaptive halting using a percentile threshold — the only variant whose halting is size-invariant and active at inference.

Identity

• Architecture: decoder-only, LLaMA-style (RoPE, RMSNorm, SwiGLU, QK-Norm, Flash/SDPA attention), looped R=4 (effective depth 32) + per-token adaptive halting: at each iteration the q=30% least-uncertain active tokens freeze (percentile of the live entropy distribution). Zero added parameters. Because q is a proportion, the criterion transfers across model sizes (unlike an absolute threshold). Halting is active during training and at inference.
• Size: 15.01M parameters · n_embd 256 / n_layer 8 / n_head 4 · context 768.
• Language: French, from scratch.
• Tokenizer: custom ByteLevel BPE, 32k vocab (bpe_tokenizer_32k.json).
• Data: French corpus (AI-rewritten Wikipedia + filtered web via RDTextract), ~425M tokens total. These runs used 20% of the train split, 2 epochs (fast research protocol).

What Nomade is good at

Nomade is the robust / out-of-domain model: the percentile rule trades in-domain sharpness for generalization and the fewest hallucinated names.

• Best out-of-domain coherence: 41.8 (vs 40.7 baseline) — the only recurrent model that beats the baseline off-domain.
• Fewest invented names: 0.094 (vs 0.137 baseline) — the lowest hallucination proxy of the four.
• Size-invariant halting (percentile q) — the variant intended to carry to a larger model without re-tuning.
• Trade-off: it gives back the perplexity edge (31.0 ≈ baseline). By construction its halting forces a fixed cascade (≈30% of tokens exit after iteration 1, then 21%, 15%, 34% full depth).

Robust evaluation (50×200, 1 seed)

Same table on all three model cards, so the trade-off is visible everywhere. Auto = held-out corpus prompts (in-domain); Fixed = generic hand-written prompts (out-of-domain).

Model	Val PPL ↓	Coherence auto ↑	Coherence fixed ↑	Invented names auto ↓	Prompt overlap auto ↑
Baseline (vanilla)	31.2	35.2	40.7	0.137	0.139
Cadence (looped R=4)	28.9	39.3	32.5	0.121	0.147
Focal (absolute halt)	29.1	44.1	29.1	0.103	0.176
Nomade (percentile halt)	31.0	36.3	41.8	0.094	0.126

No model wins everything — that is the result. Recurrence helps in-domain and hurts out-of-domain; the halting variants trade one for the other. Factuality is not improved by any of them (15M capacity ceiling).

Lineage (this is not novel)

• ACT — Adaptive Computation Time, Graves 2016 (per-token variable depth) · arXiv:1603.08983
• Universal Transformer — Dehghani et al. 2018 (the looped/weight-shared ancestor) · arXiv:1807.03819
• Recurrent-Depth Transformer — Geiping et al. 2025 (Huginn-3.5B; the canonical RDT this family follows) · arXiv:2502.05171
• CALM — Confident Adaptive Language Modeling, Schuster et al. 2022 (confidence early-exit across layers) · arXiv:2207.07061
• LoopViT — Shu et al. 2026 (parameter-free entropy exit + weight-tied loop) · arXiv:2602.02156

Direct basis — the two from-scratch looped studies this work reproduces in French: Kohli et al. 2026 (arXiv:2604.07822, COLM) and Chen 2026 (arXiv:2603.21676).

Nomade is a variant of LoopViT for autoregressive language, with halting learned during training (per token, size-invariant percentile criterion). No claim of novelty — the ingredients are all prior work.

Limitations

• 1 seed, no variance control — magnitudes may reorder with more seeds.
• 20% of data, 2 epochs — a short research protocol, not a full training run.
• 15M capacity — hallucinations are expected; the model learns form and coherence, not facts. It will not reliably tell you the capital of France.
• The percentile rule forces a fixed halting cascade regardless of whether tokens are truly ready — robust, but not "intelligent" per-token.
• No instruction tuning — a pure completion model.

Related

• ✍️ Write-up (Article 3): Teaching a 15M French LLM to think deeper — Hugging Face blog
• 🔁 Cadence (looped reference): RDTvlokip/Cadence-15M-fr
• 🎯 Focal (in-domain, absolute halting): RDTvlokip/Focal-15M-fr

Usage

This model uses custom from-scratch code (not 🤗 Transformers). The code ships with the repo (model/, utils/), so a snapshot download is self-contained. Requires torch, huggingface_hub, tokenizers.

import sys, torch
from huggingface_hub import snapshot_download

repo = snapshot_download("RDTvlokip/Nomade-15M-fr")  # code + weights + tokenizer
sys.path.insert(0, repo)
from model.gpt2 import GPT2
from utils.tokenizer import GPT2Tokenizer

# The percentile-halting config (mode + q) travels inside the checkpoint.
model = GPT2.from_pretrained(f"{repo}/best_model.pt", device="cuda"); model.eval()
tok = GPT2Tokenizer(f"{repo}/bpe_tokenizer_32k.json")

ids = tok.encode("La photosynthèse est un processus qui", add_special_tokens=True)
if ids and ids[-1] == tok.eos_token_id:
    ids = ids[:-1]                         # drop trailing <eos> (keeps it on-topic)
out = model.generate(
    torch.tensor([ids], device="cuda"),
    max_length=80, temperature=0.8, top_k=40, top_p=0.9,
    repetition_penalty=1.3, eos_token_id=tok.eos_token_id,
)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

Théo CHARLET — TSSR Graduate, AI/ML · Creator of AG-BPE · rdtvlokip.fr