PicoLM-V4-85M-Instruct

PicoLM-V4-85M-Instruct is a chat model with 84,960,832 parameters. It is PicoLM-V4-85M, a base model trained from scratch on 6.0 billion training tokens, fine-tuned on about 82 million tokens of synthetic instruction and chat data. The fine-tuned model follows simple instructions, keeps a short conversation going and uses system prompts for simple formats. It is also wrong about facts and arithmetic most of the time. Read the Limitations section before using it.

The weights are stored in the Qwen3 format of the transformers library (Qwen3ForCausalLM), so the model loads with AutoModelForCausalLM and AutoTokenizer without trust_remote_code. This was tested on transformers 4.57.1 and 5.22.1. GGUF files for llama.cpp are included (see Files). The tokenizer is the 24,576-token BPE vocabulary of PicoLM-V3; the architecture, pretraining and the base model's evaluation are described in the base model card.

This card reports only numbers that were measured. Where something could not be measured or verified, the card says so.

Intended use

The model is meant for research on small language models, for experiments with fine-tuning and chat templates, and for experiments that need a chat model small enough to run on a CPU. It is not suitable for question answering, arithmetic, translation, code generation or any setting where the answer has to be correct. It produces fluent answers that are often false.

Usage

The model uses the ChatML format. The chat template is stored in tokenizer_config.json, so apply_chat_template builds the prompt:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "aethertp/PicoLM-V4-85M-Instruct"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float32)

messages = [{"role": "user", "content": "Who are you?"}]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**ids, max_new_tokens=128, do_sample=False, repetition_penalty=1.1)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

Output of the code above (CPU, fp32, identical on transformers 4.57.1 and 5.22.1):

I'm PicoLM-V4, a small language model made by aethertp.

The prompt format is <|im_start|>role\ncontent<|im_end|>\n for each message, followed by <|im_start|>assistant\n. The model ends its turn with <|im_end|> (token id 2), which is the end-of-sequence token set in config.json; generation_config.json lists ids 2 and 0 (<|endoftext|>). Do not add a beginning-of-text token to the prompt: the model was fine-tuned without one.

Generation settings. The numbers in this card were measured with greedy decoding, with and without repetition_penalty=1.1. The recommended setting is greedy decoding with repetition_penalty=1.1. Sampling (temperature 0.7, top-p 0.9) looped less often in our test (3.6% of prompts against 5.6%) and passed 12 of our 24 panel checks against 11 (the one extra pass is the 12-word title check); in the answers we read, it produced more nonsense. Use a system message only for simple formats; see Limitations.

Model details

The architecture is the one of the base model: 20 layers, hidden size 576, 9 query heads and 3 key/value heads with head dimension 64, SwiGLU feed-forward with intermediate size 1536, RMSNorm with RMSNorm on queries and keys, RoPE (base 10,000), a vocabulary of 24,576 tokens with tied input and output embeddings, and a context length of 2048. No architecture or tokenizer changes were made for the fine-tune. The chat tokens <|im_start|> and <|im_end|> already existed in the tokenizer. Token id 2 (<|im_end|>) is not used in the pretraining data (see the base model card), so the fine-tune is where the model learns to use it.

Fine-tuning data

All data comes from HuggingFaceTB/smol-smoltalk, the SmolTalk subset that HuggingFace built for small models, plus a small identity set written by us and a share of pretraining text. We used six of its sources. We did not use its smol-magpie-ultra source: it was generated with Llama-3.1-405B-Instruct, and in our own ablation it gave no measurable gain on our development loss.

Source	Real tokens in the training plan	Share
openhermes-50k	25.28M	30.7%
self-oss-instruct	17.01M	20.6%
smol-contraints	13.60M	16.5%
smol-summarize-20k	10.19M	12.4%
smollm-rewrite-30k (weight 0.5)	9.23M	11.2%
replay (pretraining text, same sources as the base model's stage 2)	6.40M	7.8%
explore-instruct-rewrite	0.54M	0.7%
identity	0.18M	0.2%
total	82.42M

"Real tokens" counts prompt and answer tokens without padding, over the whole plan of two epochs. The loss is computed only on assistant answers and the closing <|im_end|>. The identity set (522 short conversations, repeated 4 times in the mix) teaches the model to say that it is PicoLM-V4, developed by aethertp, and not ChatGPT. 88 of the 522 conversations are short two-turn exchanges in which the user gives a name and asks for it later. Source names are the subset names used by the dataset; smol-contraints is spelled as in the dataset.

Provenance and licenses of the fine-tuning data

This data is synthetic, and much of it was written by other language models. We did not verify its origin or terms beyond what the dataset cards say, and the model's license does not remove those terms.

• The SmolTalk card says that its new datasets (Smol-Magpie-Ultra, Smol-contraints, Smol-rewrite, Smol-summarize) are licensed under Apache 2.0, and that for the existing public datasets included in the mix, "please refer to the original dataset for the license". The smol-smoltalk card lists the license as Apache 2.0 and the dataset as synthetic.
• openhermes-50k is a sample of OpenHermes 2.5. Its card is tagged Synthetic, GPT-4 and Distillation, describes the data as "primarily synthetically generated" and compiled from many sources, and names among them Airoboros 2.2, GPT-4-only subsets of ChatBot Arena and ShareGPT, GPT4-LLM, WizardLM Evol-Instruct, SlimOrca, Platypus, CamelAI and Glaive. The card text we read states no license for the compilation. It is about 31% of the fine-tuning tokens. Whether the terms of the services that produced such outputs restrict their use for training a model is a question we did not resolve and are not qualified to answer.
• self-oss-instruct corresponds to the Self-OSS-Starcoder2-Instruct dataset named in the SmolTalk card, and explore-instruct-rewrite to the Explore-Instruct-Rewriting dataset named there. We did not check the models or the licenses behind them.
• For smol-contraints, smol-summarize-20k and smollm-rewrite-30k, the SmolTalk card says the generation code is published but we did not verify which models generated the data.
• The fine-tuning data was not decontaminated against the evaluation sets of this card. The overlap was measured afterwards, and nothing was removed. Method: the 139,505 examples of the fine-tuning training file (42.6 million tokens, decoded back to text; it lacks 88 of the 522 identity conversations, which were added to the training mix later and contain no benchmark text; the replay text, which is pretraining data and about 8% of the fine-tuning tokens, is not part of this file and was not checked) and every string of every benchmark item (question, choices, context, passage) were lowercased and split into words. An item counts as overlapping if any run of 13 (or of 8) consecutive words of it also occurs in some training example. Items in which no string has that many words cannot be tested and are left out of the percentages. The result is below. A hit shows shared text. It does not show whether the question or the answer was also in the training data, and the checked strings include passages. Runs of 8 words also match ordinary phrases.

Benchmark	Items	13 words: testable	13 words: overlapping	8 words: testable	8 words: overlapping
HellaSwag	10,042	9,907	1 (0.0%)	10,042	239 (2.4%)
ARC-Easy	2,376	1,576	0	2,236	29 (1.3%)
ARC-Challenge	1,172	877	1 (0.1%)	1,120	5 (0.4%)
PIQA	1,838	1,024	0	1,484	7 (0.5%)
Winogrande	1,267	1,267	0	1,267	0
OpenBookQA	500	125	0	329	1 (0.3%)
SciQ	1,000	882	18 (2.0%)	984	56 (5.7%)
BoolQ	3,270	3,269	77 (2.4%)	3,270	217 (6.6%)
LAMBADA	5,153	5,153	0	5,153	38 (0.7%)
WikiText (documents)	62	62	0	62	20 (32.3%)

Almost all of the 13-word overlap comes from openhermes-50k: 75 of the 77 BoolQ items and all 18 SciQ items (the other two BoolQ items come from smol-contraints). The pretraining data of the base model was not checked in this way. The base model card states that no decontamination was performed.

Anyone redistributing the model or using it commercially should check the terms of the datasets above and of the services that generated them, and decide whether they suit their use.

Fine-tuning procedure

Starting point	PicoLM-V4-85M at the end of pretraining (step 45,776)
Optimizer	Muon for the two-dimensional weight matrices of the transformer blocks, AdamW for the embedding and the normalization weights
Learning rate	3e-4 for both optimizers, 5% linear warmup (153 steps), then cosine decay to 10% of the peak
Weight decay	0.01
Gradient clipping	1.0
Steps	3,061 (two epochs over the weighted mix, 82.42M real tokens)
Batch	about 32,768 padded tokens per step (micro-batches of 8,192 tokens, 4 accumulation steps)
Precision	fp16 mixed precision
Hardware	one NVIDIA T4 on Kaggle, about 2.5 hours

Two learning rates were tried with the final setup, 3e-4 and 5e-4. Before the runs we fixed the rule for choosing between them. The model released here is the 3e-4 run: it forgot less of the pretraining text (loss on held-out pretraining text rose by 0.124 nats per token, against 0.188 for 5e-4) and its loss on our held-out chat data was slightly higher (1.0325 against 1.0241). In a blind pairwise comparison of 24 pairs of answers (judged without knowing which model wrote which, by the development assistant, a language model, not by human raters), 16 pairs were judged equal and each model won 4. The two models are not distinguishable by our tests.

Evaluation

Benchmarks

Zero-shot, same harness, settings and code as the base model's evaluation: lm-evaluation-harness 0.4.9.1 (transformers 4.57.1, datasets 3.6.0), full test or validation sets, maximum length 1024, fp32. Raw text is scored without a chat template, exactly as for the base model. The base row was re-run in the same session as the Instruct row and gives the same numbers as in the base model card.

Model	HellaSwag	ARC-e	ARC-c	PIQA	Winogrande	OBQA	SciQ	BoolQ	LAMBADA acc	LAMBADA ppl	WikiText word ppl	WikiText bits/byte
PicoLM-V4-85M-Instruct	33.6	55.1	25.8	63.4	50.0	31.8	80.0	56.9	30.2	56.43	46.21	1.0341
PicoLM-V4-85M (base)	32.8	54.8	26.8	63.7	50.1	31.4	78.5	59.1	31.2	48.37	38.83	0.9872
Instruct minus Base	+0.8	+0.3	-1.0	-0.3	-0.1	+0.4	+1.5	-2.2	-1.0	+8.06	+7.38	+0.0469

Metrics are accuracy in percent unless a column says otherwise: acc_norm for HellaSwag, ARC-Challenge and OpenBookQA, acc for the others. Standard errors and the unnormalized and normalized pairs for every task are in eval_results.json in this repository. The standard errors of the harness are about 0.5 points for HellaSwag, 1.0 for ARC-Easy, 1.3 for ARC-Challenge, 1.1 for PIQA, 1.4 for Winogrande, 2.1 for OpenBookQA, 1.3 for SciQ, 0.9 for BoolQ and 0.6 for LAMBADA accuracy.

• The multiple-choice scores did not change. No difference between Instruct and base on these nine columns is larger than two standard errors. The largest are BoolQ (-2.2 points) and SciQ (+1.5). Fine-tuning neither improved nor visibly damaged what these benchmarks measure. They do not measure chat quality.
• Raw-text perplexity got worse. WikiText word perplexity rose from 38.83 to 46.21 (+19%) and LAMBADA perplexity from 48.37 to 56.43 (+17%). On held-out pretraining text, the average loss rose from 2.2826 to 2.4069 nats per token (32 windows of 1024 tokens from each of six sources of the base model's validation mix, measured at the end of fine-tuning). This is the cost of fine-tuning on chat data. If you need the model for plain text continuation, use the base model.
• BoolQ is near the majority rate (62.2% "yes") for both models, so that column does not show reading comprehension for either.
• MMLU, code benchmarks and instruction-following benchmarks such as IFEval were not run.

Chat behavior

We generated answers with at most 256 new tokens for 360 single-turn prompts taken from the held-out part of the fine-tuning data (170 of them with a system message) and read a sample of the answers. These prompts come from the same distribution as the training data, so they favor the model.

Setting	Finished within 256 tokens	Empty	Looping answers	Mean length (tokens)
greedy	86.9%	0.3%	6.4%	113.5
greedy, repetition penalty 1.1	86.7%	0.3%	5.6%	112.6
sampling, T 0.7, top-p 0.9	86.4%	0.3%	3.6%	113.2

"Finished" means the model emitted <|im_end|>. An answer counts as looping if some sequence of 6 words occurs at least 3 times in it. These are automatic counts; they do not measure whether an answer is correct.

We also ran 24 short prompts that we wrote ourselves, with a simple automatic check for each (a panel of identity, format, fact, arithmetic and memory questions). Greedy decoding passed 11 of 24 (10 if the check for "count from 1 to 5" is made strict, because the model kept counting past 5). With the checks as they were run:

• Passed: name and maker (PicoLM-V4, aethertp), "Are you ChatGPT?" answered no, remembering a name given earlier in a two-turn conversation (Dana), a three-bullet list, a JSON object, a simple Python function (def add), the capitals of France, Japan and Italy.
• Failed: all three arithmetic questions (12 + 15, 9 times 8, 100 minus 37), the capital of Germany, the winner of the 1998 World Cup, all five strict length or format constraints (one word, exactly two sentences, exactly three words, a title of at most 12 words, three colors separated by commas), "Reply with only YES or NO: is Paris in France?" (the model answered that Paris is not in France), a one-sentence summary under a system prompt, and writing a sentence in capital letters.
• On 12 prompts outside the training distribution, the answers to a Spanish question and to a translation into French were not meaningful, and requests for step-by-step reasoning reached wrong answers. The model was not trained on multi-turn data beyond the identity set, so longer conversations are untested.

Twenty-four prompts are a small sample, and the panel is a probe, not a benchmark.

Limitations

The model is small, and fine-tuning did not change what it knows. It makes factual errors (capitals, dates, sports results), cannot do arithmetic, ignores strict format or length constraints, and often writes code that does not run. Summaries and rewrites can contain content that is not in the input. About 6% of answers in our test repeated a phrase in a loop, and about 13% did not finish within 256 tokens. It was trained and tested on English; other languages are not supported. The fine-tuning data is almost entirely single-turn, so multi-turn conversations are unreliable except for the short name-memory pattern that was trained. The model says that it was developed by aethertp because the identity set taught it to; this is trained behavior. It has no safety training and no refusal training, and both the pretraining text and the fine-tuning data can contain biased, offensive or false content that the model can reproduce.

Files

File	Size	Content
model.safetensors	339.9 MB	weights in fp32 (Qwen3ForCausalLM)
config.json, generation_config.json, tokenizer.json, tokenizer_config.json	small	configuration, tokenizer, ChatML chat template (in tokenizer_config.json)
eval_results.json	small	raw output of the benchmark run above (Instruct and base, with standard errors)
PicoLM-V4-85M-Instruct-F16.gguf	170.9 MB	GGUF, 16-bit floats
PicoLM-V4-85M-Instruct-Q8_0.gguf	91.2 MB	GGUF, 8-bit quantization (recommended)
PicoLM-V4-85M-Instruct-Q4_K_M.gguf	65.2 MB	GGUF, 4-bit quantization (visibly lossy, see below)

GGUF

The GGUF files were converted from the weights in this repository with llama.cpp commit 2cbca389f515df455412dafa07e2b1359cc72a6d. The llama.cpp programs used for the tests below were built unmodified from that commit. Its converter does not recognize this tokenizer, so the conversion script was patched to write the pre-tokenizer type smollm (the tokenizer splits digits individually and then applies the GPT-2 regular expression, as that type does). The files carry the ChatML chat template, end-of-sequence id 2, no automatic beginning-of-text token and padding id 3. We checked the result against the transformers tokenizer and model:

• The token ids of 40 test strings (English, code, digits, whitespace, accents, CJK, emoji, strings containing the special tokens <|endoftext|> and <|im_end|>) and of 4 chat prompts are identical to the ones from the transformers tokenizer.
• The template stored in the GGUF renders the same prompt as the one in tokenizer_config.json.
• Greedy decoding with the F16 file (no repetition penalty, 100 new tokens) gave exactly the same text as transformers for all 8 test chats.
• Agreement of the quantized files with F16 was measured on 1.0 million characters of held-out text from the base model's validation mix (30 windows of 1024 tokens, the second half of each window scored). F16 perplexity on this text is 16.585 (transformers fp32: 16.585).

File	Perplexity ratio to F16	99th percentile KL divergence to F16	Same top-1 token as F16	Greedy text identical to F16 (8 chats)
Q8_0	1.0006	0.0056	98.0%	7 of 8 (the longest answer diverged after 79 characters)
Q4_K_M	1.0104	0.0713	94.1%	5 of 8 (the other three differ from the first sentence on)

Q8_0 is close to F16 and is the recommended file. Q4_K_M saves 26 MB over Q8_0 and changes the output noticeably more, so we do not recommend it unless size matters. A command that was tested in this form:

llama-completion -m PicoLM-V4-85M-Instruct-Q8_0.gguf -no-cnv --temp 0 --repeat-penalty 1.1 -n 128 \
  -p $'<|im_start|>user\nWho are you?<|im_end|>\n<|im_start|>assistant\n'

Other llama.cpp versions were not tested.

License and citation

The weights and this card are released under the Apache License 2.0. The fine-tuning data and the pretraining data come from the datasets named in this card and in the base model card; each has its own license and terms, and some were generated by other language models (see "Provenance and licenses of the fine-tuning data"). Anyone redistributing the model or using it commercially should check that these terms suit their use.

@misc{picolm_v4_instruct,
  title  = {PicoLM-V4-85M-Instruct},
  author = {aethertp},
  year   = {2026},
  url    = {https://huggingface.co/aethertp/PicoLM-V4-85M-Instruct}
}