Text Generation
Transformers
Safetensors
English
llama
wikipedia
facts
retrieval
question-answering
smollm2
text-generation-inference

wiki-model

GGUF: Lucasgitguy-co/wiki-model-GGUF (for llama.cpp).

SmolLM2-135M, fully fine-tuned to state facts from Wikipedia. It does four things: a fact about a topic, a random fact, short question answering, and correcting a false statement. With --grounded it answers from a sourced Wikipedia index and prints the article it used.

The from-scratch 12.5M sibling is wiki-model-tiny. Data and index: wiki-model-facts.

Small models get facts wrong when they answer from memory. Use --grounded for anything that matters.

Use

hf download Lucasgitguy-co/wiki-model --local-dir wiki-model
cd wiki-model && pip install -e .
wiki-model "Eiffel Tower"                       # picks the 135M weights in this folder
wiki-model --q "Who wrote Hamlet?" --grounded    # first call downloads the ~600 MB fact index (skip with --no-index)
transformers only
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Lucasgitguy-co/wiki-model")
m = AutoModelForCausalLM.from_pretrained("Lucasgitguy-co/wiki-model")
x = tok("<q> Who wrote Hamlet? </q>", return_tensors="pt", add_special_tokens=False)
print(tok.decode(m.generate(**x, max_new_tokens=32, do_sample=False)[0][x.input_ids.shape[1]:], skip_special_tokens=True))

Prompts: <fact> Topic </fact>, <q> Question? </q>, <fix> Claim. </fix>, optionally preceded by <ctx> article text </ctx> .

Results

Held-out articles and claims (test split by article title). "Grounded" retrieves context at run time with the BM25 index. "Fact supported" = the generated sentence matches a sentence of the topic's own article with token F1 ≥ 0.5. "SmolLM2-135M few-shot" is the untuned base model with three examples in the prompt (no context in the examples); "copy the claim" just repeats the input claim (fix-task baseline). "Fix (year swap)": shifted-year sentences mixed with unchanged true sentences (71% of the set), so copying already scores 71. Answering from memory, the base model's facts are supported slightly more often than the fine-tune's: fine-tuning taught the task format and how to use retrieved context, not new knowledge.

model mode Q&A infobox EM 'What is X' F1 Fix (FEVEROUS) EM Fix can't-verify acc Fix (year swap) EM Fact supported (new topics) Fact supported (known topics)
wiki-model model only 14.7 30.0 37.0 88.5 65.0 12.0 10.0
wiki-model grounded 94.7 65.3 37.0 88.0 83.0 98.0 98.0
wiki-model-tiny model only 14.7 28.2 32.5 89.0 54.0 5.0 8.0
wiki-model-tiny grounded 90.7 58.0 33.5 89.0 55.0 92.0 94.0
SmolLM2-135M few-shot model only 10.0 18.9 11.0 11.0 66.0 16.0 16.0
SmolLM2-135M few-shot grounded 26.7 25.2 6.5 36.5 47.0 65.0 71.0
copy the claim model only — — 47.5 89.0 71.0 — —

Training

Base HuggingFaceTB/SmolLM2-135M (Apache-2.0)
Data 40k task examples (35% with retrieved context) sampled from wiki-model-facts + FactfulDataset, then a fix top-up of 11k examples (FactfulDataset with retrieved context + replay), then a grounding top-up of 24k examples (mostly with retrieved context)
Recipe full fine-tune, AdamW lr 1e-4 cosine (top-ups 5e-5), batch 16, loss on answers only, bf16 autocast, gradient checkpointing
Hardware one Apple M4 (MPS), shared with another training job, about 8 hours

See wiki-model-tiny for how the task data is built.

Limits

  • Answers from memory are often wrong for less-known topics. Grounded answers are only as good as retrieval.
  • English only; Wikipedia snapshot from 2024.
  • Fixing false statements is the weakest task: compare with the "copy the claim" row before relying on it. "Can't verify" is rarely produced.

Licence

Weights and code: Apache-2.0 (same as the base). Training text comes from Wikipedia (CC BY-SA); FactfulDataset has no licence on its card and is not redistributed.

Downloads last month
80
Safetensors
Model size
0.1B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Lucasgitguy-co/wiki-model

Finetuned
(1,006)
this model
Quantizations
1 model

Datasets used to train Lucasgitguy-co/wiki-model