Dokichat-100M

Dokichat-100M is a 100M-parameter French language model trained from scratch on a single consumer GPU (RTX 4080, 16 GB). This repository holds the base (pretrained) model, a plain text-completion model. For the chat version, see Dokichat-100M-Instruct.

The name comes from doki doki, the sound of a heartbeat in Japanese. It is a learning project by Dokitek: we designed the architecture, trained the tokenizer and ran every stage ourselves. Despite its size, it matches French-capable models three to five times larger on our French benchmark, and leads all models under 1B in French grammar and vocabulary.

Model summary

Parameters	99.9M (input and output embeddings tied)
Architecture	Decoder-only transformer, Qwen3-style (Qwen3ForCausalLM)
Layers / hidden size	26 / 512
Attention	8 heads of 64, full multi-head (no GQA), QK-norm, RoPE (θ = 10,000)
Feed-forward	SwiGLU, 1,408
Normalization	RMSNorm (pre-norm), no biases
Context length	2,048 tokens
Tokenizer	Byte-level BPE trained on French, 32,000 tokens
Training tokens	10B (French web text)
Language	French only

The architecture follows the Qwen3 dense design, scaled down and kept deep and narrow (26 layers of width 512), which works better than wide and shallow at this size (MobileLLM, Liu et al., 2024). Because the weights use the Qwen3 layout, the model loads in transformers and vllm without custom code.

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Dokitek/Dokichat-100M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

inputs = tokenizer("La recette de la tarte aux pommes est", return_tensors="pt")
outputs = model.generate(**inputs)  # sampling settings come from generation_config.json
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

With vLLM: vllm serve Dokitek/Dokichat-100M.

Recommended sampling: temperature=0.8, top_p=0.95, min_p=0.05, repetition_penalty=1.1 (the defaults in generation_config.json). Greedy decoding tends to loop at this size.

Evaluation

All models were evaluated by us with the same harness and settings, on full test sets.

Benchmark	Random	Dokichat-100M	SmolLM2-135M	Gemma 3 270M	Pleias-350m	LFM2-350M	SmolLM2-360M	Qwen2.5-0.5B	CroissantLLM-1.3B
HellaSwag-fr	25	32.8	29.1	33.5	32.9	41.8	31.9	37.4	50.2
ARC-Challenge-fr	25	25.6	24.1	25.7	25.7	34.5	26.3	27.5	30.2
FrenchBench Grammar	25	73.9	42.9	66.4	69.7	62.2	45.4	68.1	77.3
FrenchBench Vocabulary	25	74.8	31.1	58.0	62.2	61.3	37.0	64.7	77.3
FrenchBench Reading	25	57.7	39.4	56.3	56.3	52.1	49.3	54.9	59.2
Belebele (fra_Latn)	25	24.9	22.4	24.2	24.6	40.7	24.3	39.8	27.1
XNLI-fr	33.3	43.7	35.1	42.5	44.5	42.0	37.0	45.7	48.0
XWinograd-fr	50	55.4	51.8	60.2	50.6	57.8	53.0	60.2	68.7
Average	29.2	48.6	34.5	45.9	45.8	49.1	38.0	49.8	54.7

Accuracy (%), 0-shot. Multiple-choice tasks are scored by log-likelihood (cloze formulation): the model ranks the candidate answers by probability and does not generate anything, the formulation that gives signal below 1B parameters. Length-normalized accuracy (acc_norm) where available, otherwise acc. lm-evaluation-harness 0.4.13, bfloat16, base models only. Bold: best model under 1B parameters; CroissantLLM-1.3B (in italics) is shown as a larger French-English reference. "Random" is the score of uniform guessing. FrenchBench tasks come from Faysse et al., 2024. Evaluated in November 2026.

What the results show:

• Language skills. Dokichat-100M leads every model under 1B on French grammar, vocabulary and reading comprehension, and comes within 3.5 points of CroissantLLM-1.3B, a model 13 times larger.
• Knowledge and reasoning. Like most models of this size, it stays at chance level on ARC-Challenge and Belebele, which call for world knowledge and fine-grained reading. LFM2-350M and Qwen2.5-0.5B are the first to move clearly above chance on Belebele.
• Size class. On average, it matches LFM2-350M and Qwen2.5-0.5B (within about one point) and is ahead of Gemma 3 270M and Pleias-350m, with three to five times fewer parameters.

Training

• Data. French web text from FineWeb2-HQ (fra_Latn, the top 10% of FineWeb-2 by a quality classifier), about 11.4B tokens after tokenization. Each 2,048-token window was seen once: no data repetition.
• Schedule. A first 2B-token run, then a 10B-token run continued from its pre-decay checkpoint. AdamW (β₂ = 0.95, weight decay 0.1), warmup-stable-decay learning rate with a 1.5 × 10⁻³ peak and a linear decay over the last 20%, batches of 524k tokens, bfloat16 mixed precision.
• Hardware. One NVIDIA RTX 4080 (16 GB): about 42 hours of training in total, at 52 TFLOPS (54% of peak).
• Result. Final validation loss 2.68 (about 0.85 bits per byte).

Examples

Completions sampled with the default settings, shown as generated:

La recette de la tarte aux pommes est la même que celle de la tarte au yaourt, mais elle est plus douce et plus facile à préparer. Voici la recette pour la tarte aux pommes. Ingrédients pour une tarte aux pommes :

• 1 tasse de farine tout usage (500 gr.)
• 2 cuillères à café de levure sèche
• 1 cuillère à café de sel
• 1 cuillère à café de sucre
• 1/2 tasse de crème fraîche épaisse

Le chat du voisin Si votre chat se retrouve sur une route, vous devez en informer l'éleveur de votre animal. Si la route est difficile, vous devrez le conduire chez un vétérinaire. En cas de doute, demandez à votre vétérinaire de vérifier votre chat et d'en informer l'éleveur.

Limitations

• It makes things up. The model writes fluent, grammatical French but has little world knowledge: facts, figures and names are often invented, with confidence. Do not use it as a source of information.
• Base model. It continues text; it does not follow instructions or hold a conversation. Use Dokichat-100M-Instruct for chat.
• French only, short context. It saw no other language in training and handles at most 2,048 tokens.
• Web data. It inherits the biases and the occasional dark or inappropriate content of web text, and was not filtered for safety beyond the source dataset.

License

Apache 2.0. The pretraining data, FineWeb2-HQ, is released under the ODC-By 1.0 license; we credit its authors (Messmer et al., 2025) and the FineWeb-2 project.

Citation

@misc{dokitek2026dokichat,
  title  = {Dokichat-100M: a small French language model trained from scratch},
  author = {{Dokitek}},
  year   = {2026},
  url    = {https://huggingface.co/Dokitek/Dokichat-100M}
}