Mihir-Karbelkar/tiny-30m-site-qa

Tiny 30M site QA

A 30.5M-parameter decoder trained from scratch to answer a visitor's question from passages a website retrieves, and to refuse when the passages don't say. It runs in the browser on onnxruntime-web's WASM backend (no WebGPU needed): one 44 MB int8 file.

It is the experimental on-device model behind the chat on mihirkarbelkar.com, where it is opt-in.

Use

Prompt token ids: [7] + tokens(sources) + [3] + tokens(question) + [2], where sources is

SOURCE 1: <page title>
<passage text>

SOURCE 2: …

(≈ 2,000 characters of passages works best). Generation ends at token 1. The model answers evidence first:

Source 2: "an exact span copied from source 2"
Answer: one or two plain sentences.

Source: none + Answer: The context does not say. is a refusal, and Chat: … is a reply to small talk (greetings, thanks) that states no facts.

decoder.int8.onnx is a single graph used for both the prompt and each new token:

• inputs: input_ids int64 (1, T), past float32 (8, 2, 1, 2, 1+P, 64)
• outputs: logits (1, 16384) for the last position, present (8, 2, 1, 2, 1+P+T, 64)

The cache starts with one dummy slot that the graph masks out, so the first call passes past = zeros(8, 2, 1, 2, 1, 64), then feeds back present with one token at a time.

Recommended decoding. Score the probability of Source: none after the prompt and refuse when it is at least 0.3. Otherwise decode greedily (repetition penalty 1.1), banning : right after a first token Source so the model must cite a source number. Check that the quote really occurs in the cited source before showing the answer; if it doesn't, show the sources instead. A reference implementation is the portfolio's src/workers/tiny.worker.ts.

Training

• Pretraining: 2.5B tokens of FineWeb-Edu and FineWeb (biased towards structured pages: lists, prices, hours, FAQs), 16K byte-level BPE, RoPE, grouped-query attention (8 query / 2 key-value heads), QK-norm, SwiGLU, d=512 × 8 layers. Held-out loss 3.30 nats/token.
• Fine-tuning: about 171K examples in the evidence-first format, all generated with Qwen3-8B (Apache-2.0) over FineWeb pages and checked: questions over general pages (with the right passage among same-page and other-page passages); third-person questions about the subject of person and business pages (bios, portfolios, about-us), including ones the page doesn't answer; quotes kept only if they occur verbatim in the passage; every "not in the text" question re-checked against the whole page; small talk with a short no-facts reply and off-topic requests refused.
• Export: int8 dynamic quantisation with SmoothQuant on the MLP output projection; layers 3–4 of that projection stay fp32.

Results

On held-out pages: refusal AUROC 0.99; at threshold 0.3 it refuses ~96% of questions the passages don't answer, wrongly refuses ~5% it could answer, and 99% of small talk gets a chat reply.

Speed (WASM, 4 threads, 512-token prompt): ~0.25 s to read the prompt and ~100 tokens/s on a laptop; ~0.6 s and ~43 tokens/s on a mid-range Android phone. The evidence line adds ~15-25 tokens.

Limitations

This is an experiment, not a production assistant. On a real personal site (every passage about the same person) it answers only about a third of factual questions correctly; it often quotes a real but irrelevant sentence and then answers from it, which a quote check cannot catch. Use it with the sources shown, and don't rely on it for facts that matter.

Licence and attribution

Apache-2.0. Pretraining data: FineWeb and FineWeb-Edu by Hugging Face, made available under the ODC-By 1.0 licence (and subject to Common Crawl's terms of use). Fine-tuning data was generated with Qwen3-8B (Apache-2.0) over FineWeb pages. (Earlier revisions of this repo also used OpenAssistant oasst2, Apache-2.0.)