wiki-model-tiny

GGUF: Lucasgitguy-co/wiki-model-tiny-GGUF (for llama.cpp).

A 12.5M-parameter language model, trained from scratch in plain PyTorch, that states facts from Wikipedia. It does four things: a fact about a topic, a random fact, short question answering, and correcting a false statement. With --grounded it answers from a sourced Wikipedia index and prints the article it used.

The bigger sibling is wiki-model (SmolLM2-135M fine-tune). Data and index: wiki-model-facts.

A model this small gets many facts wrong when it answers from memory. Use --grounded for anything that matters.

Use

hf download Lucasgitguy-co/wiki-model-tiny --local-dir wiki-model-tiny
cd wiki-model-tiny && pip install -e .       # torch, tokenizers, bm25s, transformers, huggingface_hub
wiki-model "Eiffel Tower"
wiki-model --random
wiki-model --q "Who wrote Hamlet?"
wiki-model --fix "The Eiffel Tower is in Lyon." --grounded

The first call downloads the fact index (about 600 MB) from the dataset repo: --grounded answers from it, and without --grounded the answer is checked against it and marked supported or unverified. Add --no-index to skip the index entirely.

Python
from wiki_model.cli import load_model, run
from wiki_model.index import FactIndex
m = load_model("tiny", ".")
ix = FactIndex("path/to/wiki-model-facts/index")
print(run(m, "q", "Who directed Forrest Gump?", ix, grounded=True))

Prompt format

<fact> Eiffel Tower </fact> The Eiffel Tower is a lattice tower ...
<q> Who wrote Hamlet? </q> William Shakespeare
<fix> The Eiffel Tower is in Lyon. </fix> The Eiffel Tower is in Paris.   (or: Can't verify from Wikipedia.)
<ctx> retrieved article text </ctx> <q> ... </q> ...                      (grounded)

Results

Held-out articles and claims (test split by article title, never seen in training). "Grounded" retrieves context at run time with the BM25 index; it is not handed the gold article. "Fact supported" = the generated sentence matches a sentence of the topic's own article with token F1 ≥ 0.5. "Known topics" are well-known articles from the training split. "copy the claim" just repeats the input claim (a baseline for the fix task); "SmolLM2-135M few-shot" is the untuned base model with three examples in the prompt (its examples carry no context). "Fix (year swap)": abstract sentences where a year was shifted, mixed with unchanged true sentences (71% of this set), so "copy the claim" already scores 71; only scores above that mean real corrections.

model mode Q&A infobox EM 'What is X' F1 Fix (FEVEROUS) EM Fix can't-verify acc Fix (year swap) EM Fact supported (new topics) Fact supported (known topics)
wiki-model model only 14.7 30.0 37.0 88.5 65.0 12.0 10.0
wiki-model grounded 94.7 65.3 37.0 88.0 83.0 98.0 98.0
wiki-model-tiny model only 14.7 28.2 32.5 89.0 54.0 5.0 8.0
wiki-model-tiny grounded 90.7 58.0 33.5 89.0 55.0 92.0 94.0
SmolLM2-135M few-shot model only 10.0 18.9 11.0 11.0 66.0 16.0 16.0
SmolLM2-135M few-shot grounded 26.7 25.2 6.5 36.5 47.0 65.0 71.0
copy the claim model only — — 47.5 89.0 71.0 — —

Model

Architecture Llama-style decoder: 8 layers, d=320, 8 heads, SwiGLU 864, RoPE, RMSNorm, tied embeddings
Parameters 12.5M
Tokenizer own 8,192-token byte-level BPE with the task tags as special tokens
Context 384 tokens
Stage 1 98M tokens of Wikipedia article text (wikimedia/wikipedia 20231101.en, 141k articles)
Stage 2 288k task examples (35% with retrieved context) from 948k articles of wikimedia/structured-wikipedia + FactfulDataset
Stage 3 fix top-up: 48k examples of FactfulDataset (half with retrieved context) + replay of the other tasks
Hardware one Apple M4 (MPS + CPU), about 10 hours in total
Training data details
  • Facts: the lead sentences of each article's abstract.
  • Q&A: "What is X?" answered with the article's short description, plus questions templated from about 60 infobox fields (born, founded, capital, genre, directed by, family, ...).
  • Fix: FactfulDataset (FEVEROUS-derived; TrueStatement → the claim unchanged, ContriversialInput → "Can't verify from Wikipedia.", otherwise the corrected claim), plus abstract sentences with a shifted year, plus unchanged true sentences.
  • Articles: 232k random articles from three shards, plus 716k long articles (at least 15k characters, no digits in the title) from all other shards.
  • wikimedia/wikisource was left out on purpose: it holds books and source texts, not encyclopedic facts.

Limits

  • Answers from memory are often wrong or invented, especially for less-known topics. Grounded answers are only as good as retrieval.
  • English only. Wikipedia snapshot from 2024 (structured) and 2023 (plain text).
  • The "supported" check is a word-overlap heuristic against the topic's own article, not a fact checker.
  • Fixing false statements is the weakest task. On FEVEROUS claims the model scores below simply repeating the claim ("copy the claim" row), because it often rewords claims that were already true. It also practically never answers "Can't verify" (the 89% can't-verify accuracy is the base rate).

Licence

Code and weights: Apache-2.0. The training text comes from Wikipedia (CC BY-SA); FactfulDataset has no licence on its card and is not redistributed here.

Downloads last month
68
Safetensors
Model size
12.5M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Lucasgitguy-co/wiki-model-tiny

Quantizations
1 model

Datasets used to train Lucasgitguy-co/wiki-model-tiny