SmallAICreator/MiDo-70M-v3

MiDo-70M v3

MiDo-70M is a tiny language model trained from scratch by UltraLabs (SmallAICreator). It has 70.4M parameters (45.2M without the embeddings) and was trained mostly by knowledge distillation: it learned from the full output distributions of SmolLM2-360M instead of only next-token labels.

v3 continues v1 with another 5.24B distilled tokens, so it has seen about 10.9B tokens in total, all on a single Kaggle TPU v5e-8 session per version.

Head-to-head with TII's Falcon-H1-Tiny-90M-Instruct (2026): MiDo beats it on trivia both models were trained on (82% vs 73%) and is slightly ahead on ARC-Easy (52.7 vs 50.8), with ~20M fewer parameters and ~1.4% of the training tokens (10.9B vs 800B). Details and caveats below.

File	What it is
MiDo-70M-v3-Chat-F16.gguf	The chat model (after SFT and DPO). Use this one to talk to.
MiDo-70M-v3-Base-F16.gguf	The base model (after pretraining only). Best at raw knowledge and text completion.
tokenizer.json, tokenizer_config.json	The original tokenizer (SmolLM2's 49,152-token vocabulary, ChatML template).

Model details

Architecture	Qwen3-style decoder: 15 layers, hidden size 512, FFN 1536, 8 query heads / 2 KV heads (head dim 64), RoPE, tied embeddings
Parameters	70.4M (45.2M non-embedding)
Context	2,048 tokens
Teacher	SmolLM2-360M (logit distillation)
Training tokens	~10.9B (v1 5.70B + v3 5.24B), including a fact-bank source and a knowledge-heavy anneal
Chat tuning	SFT (420 steps: conversations, GSM8K, MetaMathQA, Magicoder-OSS-Instruct), then DPO (849 steps: hh-rlhf, PKU-SafeRLHF)
Chat format	ChatML: <|im_start|>user ... <|im_end|> then <|im_start|>assistant

Benchmarks

All scores use lm-evaluation-harness 0.4.13, zero-shot, fp32, max length 2048, and the full test sets. The Pythia-70M column comes from our own run with the same settings.

Task (metric)	Pythia-70M	MiDo v1 Base	MiDo v1 Chat	MiDo v3 Base	MiDo v3 Chat
ARC-Easy (acc_norm)	35.23	45.92	44.57	42.21	46.34
ARC-Challenge (acc_norm)	21.93	25.51	25.09	27.82	26.45
HellaSwag (acc_norm)	27.38	31.95	32.15	31.58	32.14
LAMBADA OpenAI (acc)	22.71	27.83	27.50	29.01	26.68
PIQA (acc_norm)	59.19	61.70	62.35	59.96	63.22
Winogrande (acc)	52.96	50.75	52.49	50.51	49.72
Six-task average	36.57	40.61	40.69	40.18	40.76
SciQ (acc)	64.00	76.60	74.90	76.30	74.60
BLiMP (acc)	74.16	73.71	73.90	77.43	77.08
WikiText (bits per byte, lower is better)	–	1.002	1.060	0.957	1.056

• MiDo v3 Chat beats Pythia-70M on 7 of 8 tasks with about 27× less training data (~11B tokens vs ~300B).
• v3's extra tokens mainly improved language modeling: BLiMP grammar went up almost 4 points and WikiText bits per byte dropped to 0.957. The six-task average stayed about the same as v1.
• For reference, SmolLM2-135M (twice the size, ~2T tokens) averages 49.3 on the same six tasks, and random guessing averages 29.2.

TruthfulQA (lm-eval truthfulqa_mc1 / truthfulqa_mc2):

Model	MC1	MC2
MiDo v3 Chat	26.56	44.40

Chat test. We asked 12 everyday questions, 3 samples each, with temperature 0.5, top-p 0.9 and repeat penalty 1.15. An answer only counted if it was correct and contained no made-up details.

Model	Params	Score (/12)
MiDo-70M v3 Chat	70M	6.0
SmolLM2-135M-Instruct	135M	4.67

MiDo keeps its answers short, so it invents fewer details. SmolLM2-135M-Instruct knows more trivia but adds confident made-up facts to longer answers.

Fact recall on 300 trained facts: the base answers 98.3% correctly. The chat model keeps about 87%: it answers in full sentences, and chat tuning made it forget about 12%.

vs Falcon-H1-Tiny-90M

Falcon-H1-Tiny-90M-Instruct is TII's 2026 tiny chat model: 90M parameters, ~800B training tokens (25% of them SFT data such as Tulu 3, repeated 100+ times), then DPO.

	MiDo-70M v3 Chat	Falcon-H1-Tiny-90M-Instruct
Parameters	70M	90M
Training tokens	~10.9B (~1.4%)	~800B
ARC-Easy acc (lm-eval 0.4.13, 0-shot)	52.65	50.76
ARC-Easy acc_norm	46.34	44.74
Trivia both models trained on (46 questions × 3 runs)	82% (113/138)	73% (101/138)
Trivia neither model trained on (14 questions × 3 runs)	26% (11/42)	36% (15/42)

How the trivia test works:

• Every candidate question was checked against MiDo's training fact bank (30,970 facts) and against Tulu 3 SFT, a dataset TII names for Falcon's SFT data. Questions were then grouped by whether both models or neither model had seen them.
• Each set ran three times: greedy, plus two sampled runs (temperature 0.5, top-p 0.9, top-k 40, repeat penalty 1.15). Answers were keyword-graded, then checked by hand.

Notes:

• On ARC-Easy the gap is about 2 points, with ±1.0 standard error per score: MiDo is on par to slightly ahead.
• On facts neither model studied, Falcon knows more. At this size both models mostly answer from what they saw repeated in training.
• MiDo's efficiency comes from distillation. Its teacher, SmolLM2-360M, was trained on ~4T tokens, so each of MiDo's tokens carries a lot of the teacher's knowledge.

Use it

• On Android: open the file in L-AI. At 143 MB it runs fast on almost any phone.
• llama.cpp: llama-cli -m MiDo-70M-v3-Chat-F16.gguf -cnv
• Suggested sampling: temperature 0.5, top-p 0.9, repeat penalty 1.15.
• To use the base model, give it a plain prompt such as Q: What do bees make?\nA: instead of the chat format.

Limitations

• It's a 70M model. It makes up facts, gets multi-step math wrong and writes shaky code.
• Safety tuning is weak: it often follows harmful requests instead of refusing them. Don't rely on it for anything safety-critical.
• It mostly ignores custom system prompts.
• English only.

Credits and license

Made by SmallAICreator · UltraLabs. Released under Apache-2.0. Teacher model: SmolLM2-360M (Apache-2.0). Some fine-tuning data has its own license, such as PKU-SafeRLHF (CC BY-NC 4.0); check those if you plan commercial use.