wiki-model-tiny
GGUF: Lucasgitguy-co/wiki-model-tiny-GGUF (for llama.cpp).
A 12.5M-parameter language model, trained from scratch in plain PyTorch, that states facts from Wikipedia.
It does four things: a fact about a topic, a random fact, short question answering, and correcting a false statement.
With --grounded it answers from a sourced Wikipedia index and prints the article it used.
The bigger sibling is wiki-model (SmolLM2-135M fine-tune). Data and index: wiki-model-facts.
A model this small gets many facts wrong when it answers from memory. Use
--groundedfor anything that matters.
Use
hf download Lucasgitguy-co/wiki-model-tiny --local-dir wiki-model-tiny
cd wiki-model-tiny && pip install -e . # torch, tokenizers, bm25s, transformers, huggingface_hub
wiki-model "Eiffel Tower"
wiki-model --random
wiki-model --q "Who wrote Hamlet?"
wiki-model --fix "The Eiffel Tower is in Lyon." --grounded
The first call downloads the fact index (about 600 MB) from the dataset repo: --grounded answers from it, and without
--grounded the answer is checked against it and marked supported or unverified. Add --no-index to skip the index entirely.
Python
from wiki_model.cli import load_model, run
from wiki_model.index import FactIndex
m = load_model("tiny", ".")
ix = FactIndex("path/to/wiki-model-facts/index")
print(run(m, "q", "Who directed Forrest Gump?", ix, grounded=True))
Prompt format
<fact> Eiffel Tower </fact> The Eiffel Tower is a lattice tower ...
<q> Who wrote Hamlet? </q> William Shakespeare
<fix> The Eiffel Tower is in Lyon. </fix> The Eiffel Tower is in Paris. (or: Can't verify from Wikipedia.)
<ctx> retrieved article text </ctx> <q> ... </q> ... (grounded)
Results
Held-out articles and claims (test split by article title, never seen in training). "Grounded" retrieves context at run time with the BM25 index; it is not handed the gold article. "Fact supported" = the generated sentence matches a sentence of the topic's own article with token F1 ≥ 0.5. "Known topics" are well-known articles from the training split. "copy the claim" just repeats the input claim (a baseline for the fix task); "SmolLM2-135M few-shot" is the untuned base model with three examples in the prompt (its examples carry no context). "Fix (year swap)": abstract sentences where a year was shifted, mixed with unchanged true sentences (71% of this set), so "copy the claim" already scores 71; only scores above that mean real corrections.
| model | mode | Q&A infobox EM | 'What is X' F1 | Fix (FEVEROUS) EM | Fix can't-verify acc | Fix (year swap) EM | Fact supported (new topics) | Fact supported (known topics) |
|---|---|---|---|---|---|---|---|---|
| wiki-model | model only | 14.7 | 30.0 | 37.0 | 88.5 | 65.0 | 12.0 | 10.0 |
| wiki-model | grounded | 94.7 | 65.3 | 37.0 | 88.0 | 83.0 | 98.0 | 98.0 |
| wiki-model-tiny | model only | 14.7 | 28.2 | 32.5 | 89.0 | 54.0 | 5.0 | 8.0 |
| wiki-model-tiny | grounded | 90.7 | 58.0 | 33.5 | 89.0 | 55.0 | 92.0 | 94.0 |
| SmolLM2-135M few-shot | model only | 10.0 | 18.9 | 11.0 | 11.0 | 66.0 | 16.0 | 16.0 |
| SmolLM2-135M few-shot | grounded | 26.7 | 25.2 | 6.5 | 36.5 | 47.0 | 65.0 | 71.0 |
| copy the claim | model only | — | — | 47.5 | 89.0 | 71.0 | — | — |
Model
| Architecture | Llama-style decoder: 8 layers, d=320, 8 heads, SwiGLU 864, RoPE, RMSNorm, tied embeddings |
| Parameters | 12.5M |
| Tokenizer | own 8,192-token byte-level BPE with the task tags as special tokens |
| Context | 384 tokens |
| Stage 1 | 98M tokens of Wikipedia article text (wikimedia/wikipedia 20231101.en, 141k articles) |
| Stage 2 | 288k task examples (35% with retrieved context) from 948k articles of wikimedia/structured-wikipedia + FactfulDataset |
| Stage 3 | fix top-up: 48k examples of FactfulDataset (half with retrieved context) + replay of the other tasks |
| Hardware | one Apple M4 (MPS + CPU), about 10 hours in total |
Training data details
- Facts: the lead sentences of each article's abstract.
- Q&A: "What is X?" answered with the article's short description, plus questions templated from about 60 infobox fields (born, founded, capital, genre, directed by, family, ...).
- Fix: FactfulDataset (FEVEROUS-derived;
TrueStatement→ the claim unchanged,ContriversialInput→ "Can't verify from Wikipedia.", otherwise the corrected claim), plus abstract sentences with a shifted year, plus unchanged true sentences. - Articles: 232k random articles from three shards, plus 716k long articles (at least 15k characters, no digits in the title) from all other shards.
wikimedia/wikisourcewas left out on purpose: it holds books and source texts, not encyclopedic facts.
Limits
- Answers from memory are often wrong or invented, especially for less-known topics. Grounded answers are only as good as retrieval.
- English only. Wikipedia snapshot from 2024 (structured) and 2023 (plain text).
- The "supported" check is a word-overlap heuristic against the topic's own article, not a fact checker.
- Fixing false statements is the weakest task. On FEVEROUS claims the model scores below simply repeating the claim ("copy the claim" row), because it often rewords claims that were already true. It also practically never answers "Can't verify" (the 89% can't-verify accuracy is the base rate).
Licence
Code and weights: Apache-2.0. The training text comes from Wikipedia (CC BY-SA); FactfulDataset has no licence on its card and is not redistributed here.
- Downloads last month
- 68