GPT-U-20M-Instruct

An instruction-tuned version of GPT-U-20M, a 20.5M-parameter Llama-architecture model pretrained from scratch. It was fully fine-tuned for one epoch on smol-smoltalk, with loss computed only on assistant tokens.

At this size the model is a small demo and research artifact. It follows the chat format and usually stops at the end of an answer, but its factual knowledge, reasoning and arithmetic are very weak. See the limitations below.

Quickstart

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "DedeProGames/GPT-U-20M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float32).eval()

messages = [{"role": "user", "content": "What is photosynthesis?"}]
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=False)
out = model.generate(input_ids, max_new_tokens=256, do_sample=True, temperature=0.1, top_k=50,
                     repetition_penalty=1.05, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0, input_ids.shape[1]:], skip_special_tokens=True))

Keep prompt + answer within the 1,024-token context.

Chat format

The vocabulary has no dedicated chat tokens, so the format is plain text. apply_chat_template renders it:

<|endoftext|>### User:
{user message}
### Assistant:
{assistant reply}
<|endoftext|>### User:
...

<|endoftext|> (id 0) is the BOS, the end-of-turn token and the EOS. The chat_template is in tokenizer_config.json.

Architecture

Same as the base model: LlamaForCausalLM with 8 layers, hidden size 384, 6 heads × 64, SwiGLU (intermediate 1,024), RMSNorm, RoPE (θ = 10,000), tied embeddings, 16,384-token byte-level BPE vocabulary, 1,024-token context. 20,453,760 parameters. Weights in BF16 (model.safetensors, 41 MB).

Training

Base	DedeProGames/GPT-U-20M (main)
Data	smol-smoltalk train split: 252,285 conversations (those longer than 1,024 tokens were dropped; 118.9M tokens in total, 82.0M supervised assistant tokens)
Epochs	1 (7,884 steps, batch 32)
Optimizer	AdamW (β = 0.9, 0.95), weight decay 0.1 on 2D matrices, grad clip 1.0
Schedule	lr 2e-4, 300 warmup steps, cosine to 2e-5
Precision	bf16 autocast, fp32 master weights; micro-batch 16 with gradient accumulation of 2
Hardware	1× NVIDIA RTX 3060 12 GB, about 41 minutes
Loss	assistant tokens and their `<

Validation loss on assistant tokens (smol-smoltalk test split, 13,258 conversations): 2.387 for the base model with this format, 1.661 after SFT.

Evaluation

Benchmarks follow the protocol of the Kiyo-230M-Instruct card, so the scores can be compared with the numbers reported there. Our own runs used temperature 0.1, top-k 50, repetition penalty 1.05, FP16, the model's own chat template, no system prompt, and the new-token limits of that card (IFEval 1,024, GPQA 384, LiveBench 512). Because the context here is 1,024 tokens rather than 2,048, the generation limit is capped at 1,024 minus the prompt length.

Benchmark	Metric	GPT-U-20M-Instruct (this run)	Kiyo-230M-Instruct*	SmolLM2-135M-Instruct*
GPQA Diamond	pass@1 (198 × 5 samples at T = 0.6)	17.27	21.11	16.57
IFEval	prompt-level strict	17.38	17.38	20.89
LiveBench instruction following	strict, 200 questions	21.04	18.77	24.23
CaseReportBench	token F1	not run (not on the Hub)	20.11	14.20

* Numbers reported on the Kiyo-230M-Instruct card, not re-run here.

Notes on this run:

• GPQA: the answer prompt and the option shuffling were written for this run, and the exact prompt used by the Kiyo card is not documented, so the numbers are not strictly comparable. 68.4% of the 990 attempts gave an identifiable option letter (no letter counts as wrong). Among those, accuracy was 25.3%, which is chance level. 20 prompts were truncated to fit the context.
• IFEval: the official instruction checkers from lm-evaluation-harness, on the 541 prompts of google/IFEval.
• LiveBench: the 200 questions without a removal date, scored with the same IFEval checkers, as the mean of "all instructions followed" and "fraction followed" (strict). Only the first turn of each question is used.
• Single run, single seed. Differences of a point or two are within noise.

Per-sample outputs and the summary are in eval/.

Limitations

• It is very small. Expect factual errors, invented details, arithmetic mistakes, weak multi-step reasoning and repetitive text in long or open-ended answers.
• The GPQA result is at chance level among answered questions. The model cannot solve expert-level questions.
• Context is 1,024 tokens, and it was trained on conversations of up to 1,024 tokens.
• English only. The base pretraining data is English web text, educational text and code.
• No safety tuning and no safety evaluation. Do not use it for medical, legal, financial or other high-stakes decisions.
• Generated code is not expected to run. It imitates the surface form of code.

License

Apache-2.0, as for the base model. The SFT data (smol-smoltalk) is also Apache-2.0.