ali-llm-safe

ali-llm-safe is a 2.05B-parameter decoder-only language model trained from scratch on English educational text in the order a student would meet it: pre-kindergarten speech and stories first, then kindergarten, then grades 1 to 12 in turn. Every training document passed a content screen (word lists plus an LLM judge, described below) before packing, so the model never read text that the screen marked as profane, sexual, hateful, violent, self-harm or drug related.

The model saw 13.5 billion tokens (11.3 billion unique) and is released as a base model, with no instruction tuning.

It is the anchor of ali-llm, a pretrained test bed for curriculum-based safety: a family of matched checkpoints for studying whether the content and order of pretraining data can make a model safer, where harmful content is stored when a model does meet it, and whether that safety survives continued pretraining. Because every document in this model's corpus is known and screened, and the screened-out documents were kept, later exposure can be administered as a measured treatment. The companion checkpoints, all under this account and all gated, differ from this one in a single factor each:

Checkpoint	Difference from ali-llm-safe
ali-llm-safe-unsafe	continued for 3,000 steps, the last 2,000 with 5% of tokens from the screened-out documents
ali-llm-safety-pruned	31% of feed-forward neurons (the least active) removed first, then the same continuation
ali-llm-baseline	a model of the pruned size trained from scratch on the identical rows, then the same continuation

It is the successor of ali-llm v0.1, which covered pre-K to grade 5 only, without the content screen.

How to use

The architecture has a per-head output gate that no built-in transformers class provides, so the modeling code ships with the weights and trust_remote_code=True is required for both the model and the tokenizer (the tokenizer itself is a plain tokenizers BPE file; the flag is only needed because AutoTokenizer reads the model config).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "aliwister/ali-llm-safe"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).cuda().eval()

prompt = "Photosynthesis is the process by which"
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=100, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

The tokenizer adds no special tokens. Documents in training were separated by <|endoftext|> (id 0), which also serves as the end-of-sequence token; <|pad|> is id 1. Prompts work best when they look like the training text: a story opening, a lesson or textbook paragraph, a question in the form Question: ...\nAnswer:, or an arithmetic drill such as 7 x 8 = .

Model details

Parameters	2,049,054,720 (100.7M of them in the shared embedding table)
Layers	32
Hidden size	2,048
Feed-forward size	8,192, SwiGLU
Attention	32 query heads, 8 key-value heads (grouped-query), head size 64
QK-norm	RMSNorm on queries and keys per head, before RoPE
Attention output gate	one sigmoid gate per head, computed from the layer input, multiplied into the head's output
Positions	RoPE, base 500,000
Norm	RMSNorm, eps 1e-5, pre-norm
Embeddings	input and output embeddings tied
Vocabulary	49,152
Context	trained at 2,048 tokens; RoPE configured to 4,096
Precision	bfloat16 weights

The architecture and tokenizer are those of ali-llm v0.1: a Llama-style decoder plus QK-norm and per-head output gating. Byte-level BPE with 49,152 tokens, every digit its own token, 16 special tokens.

Training data

All training text was assigned one of 14 levels, pre-K, K and grades 1 to 12, and the model read the levels in order. Every source covers a range of levels; within a source, documents were sorted by Flesch-Kincaid reading grade and split evenly across the source's range. Math and science documents were additionally kept at or above the lowest grade that teaches the ideas they use; ScienceQA, the MATH dataset and the drills used their own grade labels.

Unique training tokens per source and level, in millions (our tokenizer). Sources marked "generated" were written by a language model.

Source	pre-K	K	grade 1	grade 2	grade 3	grade 4	grade 5	grade 6	grade 7	grade 8	grade 9	grade 10	grade 11	grade 12	total
Cosmopedia v2 for middle school, generated								723	942	990	76	781	95	45	3,653
FineWeb-Edu								161	208	259	214	725	575	545	2,689
FineMath									55	100	101	275	343	413	1,288
Cosmopedia v2 general audience, generated										182	159	327	245	245	1,159
Cosmopedia v2 for children, generated				188	208	220	234								850
TinyStories, generated	194	164	163												521
Arithmetic drills, generated	12	21	44	59	58	69	65								328
Cosmopedia v2 for young children, generated		104	106	109											318
Cosmopedia v1 auto_math_text for grade school, generated				15	39	53	42								148
NuminaMath 1.5, K-12 subset, generated									6	9	14	19	28	23	98
Cosmopedia v2 for high school, generated											10	21	20	22	73
NuminaMath 1.5, Orca subset, generated								10	13	22	1	3	1	0.1	50
CHILDES child-directed speech (BabyLM 2026 Strict)	39														39
NuminaMath 1.5, synthetic subsets, generated									3	4	5	10	9	6	37
Project Gutenberg children's books (BabyLM 2026 Strict)					8	8	8								25
Orca math word problems, generated			0.5	1	5	9	4								20
Simple English Wikipedia (BabyLM 2026 Strict)					3	4	4								12
Khan-style explanations, generated								0.4	0.5	1	1	2	2	3	10
Simple English Wikipedia, readmitted at grades 6 to 8								2	2	2	0.2	1	0.4	0.2	7
OpenStax high-school textbooks										0.3	0.5	1	3	2	7
MATH train								0.0	0.0	0.1	0.2	0.3	0.1	0.1	1
ScienceQA train, grades 1 to 5			0.0	0.1	0.2	0.2	0.1								1
Total	245	289	313	372	321	364	358	896	1,230	1,570	581	2,166	1,323	1,305	11,333

Sources under 0.5 million tokens (SciQ, ScienceQA grades 6 to 12, and the grade 6 to 8 readmissions of grade-school math) are omitted from the table.

64% of the unique text was written by a language model (TinyStories, Cosmopedia, Orca math, NuminaMath, Khan-style explanations, the drills); 36% is human-written (FineWeb-Edu, FineMath, CHILDES, Gutenberg, Simple Wikipedia, OpenStax, ScienceQA, SciQ, MATH).

Filtering

A document was dropped if it

• failed the content screen (below);
• uses an idea taught at a later level than its own (detected with word lists for math, science and everyday money topics; college-level ideas were dropped everywhere);
• has a reading score far outside its source's range;
• is shorter than 20 words (except question-answer items and drills);
• is an exact duplicate of another document;
• shares any run of 13 tokens with a question from the evaluation sets below (46,306 documents).

Content screen

Every document was checked in two stages. First, word lists for profanity, sexual, hateful, violent, self-harm and drug content and a set of text classifiers marked candidates (any classifier score above 0.25 or any word-list hit). Second, an LLM judge (Qwen2.5-14B-Instruct) read every candidate, quoted the offending passage and gave a verdict; a document was dropped when the judge said so and its quote was verified in the text, or when it contained a word from the strictest list. 53,768 documents (79 million tokens, 0.7% of the pre-filter tokens) were dropped by this screen: 36,212 by the word lists and 22,238 by the judge. 19,852 documents the judge flagged as borderline but acceptable (for example a children's story in which a character is sad or a history passage that mentions a war) were kept. The dropped documents were set aside as the exposure material for the companion checkpoints.

Arithmetic drills

A generator writes practice text for 71 skills, most taken from the Common Core math standards for kindergarten to grade 5 plus pre-K counting and four grade-5 money skills. Each document explains a skill, works an example and then asks questions with answers; all generated answers were checked against an independent solver.

Ordering and repetition

• Levels were trained strictly in order, pre-K first, with documents shuffled within a level.
• Within each level after pre-K, 15% of rows were review text drawn from the levels already finished and spread evenly through the level. Review never included anything above the current level.
• Documents were packed into rows of 2,048 tokens separated by <|endoftext|>.
• Pre-K to grade 5 documents were read twice (drills once); grade 6 to 12 documents were read 1.07 times on average. No document was read more than twice.
• The last 10% of steps (the learning-rate decay) ran on a shuffled mixture of all 14 levels, drawn from documents held back from the main block (40% pre-K to grade 5, 60% grades 6 to 12 by rows).
• About 1.5 million tokens per level were held out and never trained on.

Training procedure

Steps	27,499
Tokens per step	491,520 (240 rows of 2,048 tokens)
Total tokens	13.5 billion
Optimizer	AdamW, betas (0.9, 0.95), eps 1e-8
Peak learning rate	3e-4
Schedule	linear warmup over 500 steps, constant to step 24,749, then linear decay to 3e-5 over the last 2,750 steps (10%)
Weight decay	0.1 on matrices; none on norms, attention gates or the embedding table
Gradient clipping	1.0
Initialisation	normal, std 0.02; output projections std 0.02/sqrt(2 x 32); attention gates zero; seed 1234
Precision	bfloat16 parameters and activations, float32 gradient reduction (PyTorch FSDP2)
Compilation	torch.compile, one transformer block at a time
Hardware	NVIDIA A100 80GB GPUs, between 2 and 6 at different points in the run (a shared node), always with the same 240 rows per step; about 280 A100-hours

Loss on held-out text was measured every 500 steps for every level. Weights were also exported at the grade 5/6 boundary (step 7,365) and the grade 8/9 boundary (step 14,447); they are available as branches grade5 and grade8 of this repository.

Evaluation

All tests are zero-shot and were run by the same script for every column. The SmolLM2 models are reference points only: they were trained on 2 to 11 trillion tokens of mostly adult web, code and math text, hundreds to thousands of times more than this model, and SmolLM2-1.7B is the closest in size. "pre-decay" is the checkpoint taken just before the learning-rate decay, at the end of the level-by-level curriculum and before the mixed-level decay block.

Summary

Test	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
BLiMP, all pairs (chance 50%)	80.1%	78.8%	80.2%	81.3%	80.5%
ScienceQA, grades 1 to 5	45.0%	40.9%	46.6%	46.2%	54.2%
ScienceQA, grades 1 to 5, chance	43.8%	43.8%	43.8%	43.8%	43.8%
ScienceQA, all grades	43.5%	41.8%	44.4%	44.5%	54.1%
ARC-Easy (test)	25.9%	25.0%	25.1%	30.4%	68.7%
ARC-Challenge (test)	24.6%	23.7%	25.4%	27.0%	53.6%
SciQ (test, question only)	27.1%	24.9%	29.7%	36.5%	71.9%
RACE middle (test)	25.5%	23.1%	22.8%	29.7%	41.5%
RACE high (test)	23.2%	24.5%	23.8%	26.7%	38.3%
MATH, problems with a plain numeric answer	7.9%	6.3%	3.0%	5.1%	7.6%
Arithmetic, all levels	2.2%	3.7%	0.8%	6.4%	16.8%
ASDiv	50.1%	32.6%	3.3%	17.8%	28.6%
GSM8K	17.1%	7.4%	1.4%	3.5%	14.0%
Bits per byte, held-out text, average of levels (lower is better)	0.380	0.432	0.590	0.494	0.409
Held-out loss, average of levels (lower is better)	1.300	1.460

Held-out loss is in nats per token of our tokenizer, so it exists only for our models. Bits per byte does not depend on the tokenizer, so it compares all models on the same held-out text.

Held-out text by level

Text from each level that training never saw. Lower is better.

Loss in nats per token, our models only

Level	ali-llm-safe (final)	ali-llm-safe (pre-decay)
pre-K	1.000	1.178
K	1.046	1.211
grade 1	1.116	1.259
grade 2	1.235	1.428
grade 3	1.147	1.318
grade 4	1.305	1.493
grade 5	1.396	1.585
grade 6	1.264	1.436
grade 7	1.318	1.484
grade 8	1.371	1.538
grade 9	1.439	1.580
grade 10	1.523	1.663
grade 11	1.527	1.649
grade 12	1.511	1.614

Bits per byte

Level	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
pre-K	0.369	0.442	0.748	0.670	0.589
K	0.350	0.414	0.659	0.573	0.491
grade 1	0.376	0.431	0.648	0.570	0.498
grade 2	0.366	0.429	0.559	0.459	0.377
grade 3	0.367	0.425	0.566	0.468	0.392
grade 4	0.407	0.468	0.596	0.499	0.419
grade 5	0.404	0.464	0.597	0.498	0.419
grade 6	0.352	0.403	0.522	0.422	0.339
grade 7	0.350	0.397	0.509	0.414	0.335
grade 8	0.382	0.431	0.560	0.453	0.360
grade 9	0.424	0.467	0.598	0.491	0.391
grade 10	0.411	0.451	0.599	0.490	0.391
grade 11	0.391	0.422	0.566	0.465	0.374
grade 12	0.376	0.401	0.537	0.441	0.357

ScienceQA by grade

Text-only questions. The model picks the choice whose text " A. choice" it finds most likely after the question, summing the log-probability over the choice's tokens. Chance is the average of 1 over the number of choices.

Grade	Questions	Chance	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
grades 1 to 5	1107	43.8%	45.0%	40.9%	46.6%	46.2%	54.2%
grades 6 to 12	1117	44.4%	42.0%	42.6%	42.2%	42.7%	54.0%
all grades	2224	44.1%	43.5%	41.8%	44.4%	44.5%	54.1%
grade 1	15	37.2%	53.3%	53.3%	40.0%	40.0%	53.3%
grade 2	176	46.1%	51.1%	43.2%	52.3%	47.2%	61.4%
grade 3	296	44.0%	43.9%	39.9%	44.6%	48.3%	51.3%
grade 4	312	41.9%	44.5%	43.0%	41.3%	46.5%	53.5%
grade 5	308	44.5%	42.5%	38.0%	51.0%	43.8%	53.6%
grade 6	188	46.6%	40.4%	39.4%	42.5%	42.0%	56.4%
grade 7	285	41.0%	42.8%	40.7%	38.2%	40.0%	51.9%
grade 8	217	43.3%	41.9%	44.2%	41.9%	43.8%	53.5%
grade 9	122	46.2%	35.2%	40.2%	43.4%	47.5%	56.6%
grade 10	119	45.3%	40.3%	41.2%	42.9%	41.2%	57.1%
grade 11	105	45.9%	49.5%	47.6%	44.8%	44.8%	46.7%
grade 12	81	47.6%	45.7%	51.8%	49.4%	43.2%	58.0%

The same questions scored per character of the choice text, which removes the pull toward short choices

Grades	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
grades 1 to 5	46.6%	45.1%	49.2%	50.9%	52.7%
grades 6 to 12	43.2%	44.1%	44.5%	45.8%	53.1%
all grades	44.9%	44.6%	46.9%	48.4%	52.9%

Arithmetic by level

Greedy, up to 12 new tokens. An answer is right only when the text before the first new line matches exactly. Questions from a small space (so few possible questions that training likely showed them) are kept apart in the second table.

Level	Questions	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
grade 1	50	0.0%	0.0%	0.0%	18.0%	2.0%
grade 2	350	6.6%	9.7%	3.7%	18.3%	31.1%
grade 3	200	1.0%	1.0%	0.0%	4.0%	13.5%
grade 4	450	1.3%	0.9%	0.0%	2.7%	15.1%
grade 5	600	0.8%	3.5%	0.0%	2.0%	12.0%
all levels	1650	2.2%	3.7%	0.8%	6.4%	16.8%

Questions from a small space

Level	Questions	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
pre-K	130	22.3%	32.3%	0.8%	33.1%	26.9%
K	318	18.6%	29.2%	0.3%	45.3%	25.2%
grade 1	348	16.1%	22.1%	0.0%	33.6%	31.6%
grade 2	152	4.6%	4.0%	0.0%	15.1%	28.9%
grade 3	423	2.6%	9.5%	1.7%	19.9%	20.1%
grade 4	200	8.0%	0.5%	2.0%	0.5%	12.0%
grade 5	100	0.0%	0.0%	0.0%	7.0%	2.0%
all levels	1671	10.7%	15.5%	0.8%	25.1%	22.7%

By standard, all questions

Standard	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
1.MD.B.3	0.0%	16.7%	0.0%	10.4%	18.8%
1.NBT.B.2	0.0%	14.0%	0.0%	32.0%	34.0%
1.NBT.B.3	0.0%	0.0%	0.0%	18.0%	2.0%
1.NBT.C.4	64.0%	62.0%	0.0%	58.0%	40.0%
1.NBT.C.5	0.0%	0.0%	0.0%	14.0%	0.0%
1.OA.A.1	0.0%	0.0%	0.0%	26.0%	2.0%
1.OA.C.6	48.0%	62.0%	0.0%	68.0%	52.0%
1.OA.D.8	0.0%	0.0%	0.0%	26.0%	74.0%
2.MD.C.7	8.0%	0.0%	0.0%	2.0%	14.0%
2.MD.C.8	0.0%	0.0%	0.0%	18.0%	12.0%
2.NBT.A.1	0.0%	16.0%	0.0%	24.0%	68.0%
2.NBT.A.2	0.0%	0.0%	0.0%	0.0%	66.0%
2.NBT.A.3	2.0%	10.0%	26.0%	40.0%	92.0%
2.NBT.A.4	0.0%	0.0%	0.0%	8.0%	4.0%
2.NBT.B.5	34.0%	32.0%	0.0%	30.0%	26.0%
2.NBT.B.7	10.0%	10.0%	0.0%	6.0%	14.0%
2.OA.A.1	0.0%	0.0%	0.0%	2.0%	2.0%
2.OA.C.3	0.0%	0.0%	0.0%	0.0%	0.0%
2.OA.C.4	9.4%	18.8%	0.0%	68.8%	12.5%
3.MD.A.1	0.0%	0.0%	0.0%	2.0%	20.0%
3.MD.C.7	0.0%	0.0%	2.0%	4.0%	0.0%
3.MD.D.8	0.0%	0.0%	0.0%	0.0%	0.0%
3.NBT.A.1	0.0%	0.0%	4.0%	2.0%	56.0%
3.NBT.A.2	4.0%	4.0%	0.0%	12.0%	34.0%
3.NBT.A.3	8.0%	42.0%	0.0%	34.0%	6.0%
3.NF.A.1	0.0%	0.0%	8.7%	21.7%	30.4%
3.NF.A.3.b	0.0%	6.0%	0.0%	22.0%	22.0%
3.NF.A.3.d	0.0%	0.0%	0.0%	16.0%	0.0%
3.OA.A.3	0.0%	0.0%	0.0%	28.0%	0.0%
3.OA.A.4	0.0%	0.0%	4.0%	18.0%	72.0%
3.OA.C.7	14.0%	32.0%	0.0%	34.0%	0.0%
3.OA.D.8	0.0%	0.0%	0.0%	2.0%	0.0%
4.MD.A.1	0.0%	0.0%	0.0%	0.0%	0.0%
4.MD.A.2	0.0%	0.0%	0.0%	0.0%	52.0%
4.NBT.B.4	2.0%	0.0%	0.0%	0.0%	24.0%
4.NBT.B.5	0.0%	6.0%	0.0%	4.0%	2.0%
4.NBT.B.6	10.0%	2.0%	0.0%	10.0%	8.0%
4.NF.A.1	0.0%	0.0%	0.0%	6.0%	20.0%
4.NF.B.3	32.0%	2.0%	0.0%	0.0%	38.0%
4.NF.B.3.c	0.0%	0.0%	0.0%	2.0%	22.0%
4.NF.B.4	0.0%	0.0%	0.0%	0.0%	0.0%
4.NF.C.5	0.0%	0.0%	0.0%	0.0%	0.0%
4.NF.C.6	0.0%	0.0%	8.0%	2.0%	10.0%
4.NF.C.7	0.0%	0.0%	0.0%	2.0%	0.0%
4.OA.B.4	0.0%	0.0%	0.0%	0.0%	8.0%
5.MD.C.5	0.0%	0.0%	0.0%	0.0%	0.0%
5.NBT.A.2	0.0%	18.0%	0.0%	10.0%	20.0%
5.NBT.A.3	0.0%	0.0%	0.0%	6.0%	26.0%
5.NBT.B.5	0.0%	2.0%	0.0%	0.0%	2.0%
5.NBT.B.6	0.0%	0.0%	0.0%	4.0%	0.0%
5.NBT.B.7	10.0%	22.0%	0.0%	0.0%	24.0%
5.NF.A.1	0.0%	0.0%	0.0%	0.0%	16.0%
5.NF.B.4	0.0%	0.0%	0.0%	10.0%	4.0%
5.NF.B.7	0.0%	0.0%	0.0%	4.0%	0.0%
5.OA.A.1	0.0%	0.0%	0.0%	0.0%	4.0%
FIN.BUDGET	0.0%	0.0%	0.0%	0.0%	0.0%
FIN.EARN	0.0%	0.0%	0.0%	4.0%	48.0%
FIN.SAVE	0.0%	0.0%	0.0%	0.0%	4.0%
FIN.SPEND	0.0%	0.0%	0.0%	0.0%	0.0%
K.CC.A.1	34.0%	38.0%	0.0%	36.0%	14.0%
K.CC.A.2	0.0%	0.0%	2.0%	16.0%	24.0%
K.CC.C.7	0.0%	4.0%	0.0%	44.0%	4.0%
K.NBT.A.1	3.7%	14.8%	0.0%	33.3%	44.4%
K.OA.A.1	50.0%	84.0%	0.0%	88.0%	54.0%
K.OA.A.2	0.0%	0.0%	0.0%	30.0%	4.0%
K.OA.A.4	0.0%	0.0%	0.0%	0.0%	44.4%
K.OA.A.5	50.0%	81.2%	0.0%	87.5%	43.8%
PK.COMPARE	18.0%	30.0%	0.0%	0.0%	0.0%
PK.COUNT	22.0%	36.0%	2.0%	60.0%	44.0%
PK.HOWMANY	0.0%	0.0%	0.0%	0.0%	0.0%
PK.ONEMORE	45.0%	45.0%	0.0%	65.0%	65.0%

ARC, SciQ and RACE

Multiple choice scored like ScienceQA: the choice whose lettered text is most likely after the question wins. RACE gives the passage before the question. SciQ is asked without its support passage. Test splits only.

Test	Questions	Chance	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
ARC (all)	3548	25.0%	25.5%	24.6%	25.2%	29.3%	63.7%
ARC challenge	1172		24.6%	23.7%	25.4%	27.0%	53.6%
ARC easy	2376		25.9%	25.0%	25.1%	30.4%	68.7%
SciQ (all)	1000	25.0%	27.1%	24.9%	29.7%	36.5%	71.9%
RACE (all)	4934	25.0%	23.8%	24.1%	23.5%	27.6%	39.2%
RACE high	3498		23.2%	24.5%	23.8%	26.7%	38.3%
RACE middle	1436		25.5%	23.1%	22.8%	29.7%	41.5%

MATH (Hendrycks), problems whose answer is a plain number

Zero-shot Question:/Answer: prompt, greedy, up to 256 new tokens, last number in the answer.

MATH	Questions	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
all	3743	7.9%	6.3%	3.0%	5.1%	7.6%
algebra		11.8%	8.5%	3.2%	5.3%	10.1%
counting and probability		3.0%	3.2%	2.8%	3.2%	4.6%
geometry		5.6%	5.6%	2.8%	3.1%	7.0%
intermediate algebra		5.6%	6.0%	4.7%	5.4%	4.4%
number theory		3.3%	3.3%	2.0%	4.7%	4.5%
prealgebra		12.2%	7.9%	2.6%	6.4%	11.9%
precalculus		6.4%	5.6%	1.7%	6.4%	6.0%
difficulty 1		20.3%	15.2%	3.5%	11.9%	19.8%
difficulty 2		14.5%	10.8%	3.8%	7.2%	13.6%
difficulty 3		7.8%	6.7%	4.3%	4.6%	7.6%
difficulty 4		4.4%	4.1%	3.2%	3.5%	4.4%
difficulty 5		1.9%	1.4%	0.9%	2.9%	1.8%

Math word problems

Zero-shot, with the prompt Question: <question>\nAnswer: and no examples. Greedy, up to 256 new tokens, cut where the model starts a new question. The last number in the answer must equal the right answer.

Test	Questions	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
ASDiv	2245	50.1%	32.6%	3.3%	17.8%	28.6%
ASDiv, answers with no number		4	1	162	19	2
GSM8K	1319	17.1%	7.4%	1.4%	3.5%	14.0%
GSM8K, answers with no number		0	0	51	2	2

BLiMP by field

A pair is right when the model gives the grammatical sentence a higher total log-probability.

Field	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
morphology	92.4%	89.6%	92.8%	93.3%	92.8%
syntax	75.2%	72.6%	78.0%	76.5%	74.4%
semantics	70.3%	75.1%	69.3%	74.4%	76.2%
syntax and semantics	79.7%	78.9%	75.4%	79.6%	78.5%
all pairs	80.1%	78.8%	80.2%	81.3%	80.5%

BLiMP by phenomenon

Phenomenon	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
adjunct island	86.2%	85.3%	89.2%	87.9%	85.9%
anaphor gender agreement	99.2%	97.7%	97.1%	98.9%	99.2%
anaphor number agreement	99.3%	99.2%	98.7%	99.4%	99.3%
animate subject passive	74.0%	75.4%	76.7%	76.3%	78.5%
animate subject trans	87.8%	83.7%	87.2%	85.2%	82.1%
causative	72.9%	68.8%	75.4%	74.7%	76.1%
complex NP island	61.8%	57.6%	64.6%	65.9%	51.8%
coordinate structure constraint complex left branch	69.6%	63.6%	71.0%	71.9%	54.0%
coordinate structure constraint object extraction	85.1%	83.4%	84.5%	85.7%	81.3%
determiner noun agreement 1	98.3%	97.2%	99.6%	99.3%	99.4%
determiner noun agreement 2	97.9%	95.0%	98.8%	98.7%	97.7%
determiner noun agreement irregular 1	94.0%	90.2%	96.1%	96.1%	96.0%
determiner noun agreement irregular 2	93.3%	92.2%	97.3%	96.7%	94.8%
determiner noun agreement with adj 2	93.8%	91.9%	95.8%	94.6%	94.5%
determiner noun agreement with adj irregular 1	91.7%	86.2%	93.6%	93.4%	93.7%
determiner noun agreement with adj irregular 2	91.0%	88.2%	95.7%	93.9%	93.7%
determiner noun agreement with adjective 1	96.0%	93.2%	97.4%	96.6%	96.9%
distractor agreement relational noun	84.7%	78.1%	85.0%	83.2%	83.7%
distractor agreement relative clause	75.6%	71.5%	69.9%	71.8%	64.8%
drop argument	77.8%	76.3%	82.0%	83.6%	80.2%
ellipsis n bar 1	86.1%	79.9%	84.5%	84.8%	81.2%
ellipsis n bar 2	84.2%	82.0%	87.9%	87.8%	90.9%
existential there object raising	82.6%	83.6%	84.6%	87.1%	85.9%
existential there quantifiers 1	98.8%	98.1%	98.3%	97.6%	98.5%
existential there quantifiers 2	13.7%	20.0%	23.9%	24.0%	34.7%
existential there subject raising	88.7%	84.2%	92.5%	89.3%	92.0%
expletive it object raising	77.9%	73.1%	81.6%	82.1%	79.9%
inchoative	69.0%	65.1%	71.6%	71.2%	70.1%
intransitive	81.0%	77.9%	84.9%	84.4%	84.4%
irregular past participle adjectives	85.5%	81.4%	84.2%	95.1%	96.1%
irregular past participle verbs	92.9%	86.1%	87.3%	87.9%	86.5%
irregular plural subject verb agreement 1	93.2%	91.3%	93.0%	91.2%	93.2%
irregular plural subject verb agreement 2	90.6%	90.9%	92.2%	93.6%	93.5%
left branch island echo question	48.6%	47.4%	50.9%	53.8%	49.3%
left branch island simple question	79.5%	74.7%	79.3%	74.4%	62.0%
matrix question npi licensor present	60.5%	68.1%	64.2%	66.3%	63.5%
npi present 1	38.5%	60.5%	56.7%	62.3%	62.8%
npi present 2	65.1%	70.0%	62.7%	65.3%	65.5%
only npi licensor present	83.8%	84.8%	58.4%	95.6%	90.1%
only npi scope	81.4%	84.1%	52.8%	90.6%	63.9%
passive 1	88.9%	86.6%	90.0%	90.9%	90.2%
passive 2	85.6%	82.8%	90.9%	90.6%	90.7%
principle A c command	71.9%	67.9%	68.5%	68.0%	69.6%
principle A case 1	100.0%	99.9%	99.9%	100.0%	100.0%
principle A case 2	95.9%	94.9%	94.2%	96.4%	94.7%
principle A domain 1	99.0%	98.6%	98.9%	97.4%	99.4%
principle A domain 2	83.0%	84.8%	67.5%	70.5%	77.1%
principle A domain 3	63.7%	63.9%	59.4%	60.8%	61.9%
principle A reconstruction	45.5%	50.6%	24.8%	32.8%	43.5%
regular plural subject verb agreement 1	96.1%	93.6%	96.0%	96.2%	96.1%
regular plural subject verb agreement 2	90.3%	88.9%	91.8%	92.5%	92.0%
sentential negation npi licensor present	96.3%	93.8%	96.8%	98.1%	99.1%
sentential negation npi scope	68.6%	62.4%	72.1%	76.8%	73.6%
sentential subject island	32.9%	36.2%	42.2%	27.6%	36.8%
superlative quantifiers 1	92.9%	95.1%	83.1%	78.0%	88.7%
superlative quantifiers 2	83.2%	85.8%	79.5%	82.0%	83.1%
tough vs raising 1	68.1%	68.5%	69.0%	73.2%	70.5%
tough vs raising 2	89.3%	87.8%	90.5%	89.1%	87.2%
transitive	85.4%	83.0%	89.6%	88.8%	89.4%
wh island	74.9%	61.4%	72.0%	71.7%	70.2%
wh questions object gap	81.9%	79.9%	83.0%	77.2%	83.1%
wh questions subject gap	91.5%	91.6%	94.2%	91.4%	92.6%
wh questions subject gap long distance	84.6%	85.0%	87.5%	78.7%	88.9%
wh vs that no gap	95.3%	96.0%	95.8%	96.0%	95.7%
wh vs that no gap long distance	93.1%	92.9%	94.7%	94.0%	95.2%
wh vs that with gap	42.3%	40.0%	54.1%	51.5%	43.2%
wh vs that with gap long distance	35.6%	30.2%	43.5%	42.0%	30.7%

Time taken

Test	ali-llm-safe (final)	ali-llm-safe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
held-out loss	94s	94s
bits per byte	20s	20s	11s	11s	19s
BLiMP	42s	41s	14s	19s	38s
ScienceQA	3s	3s	1s	2s	3s
ARC	7s	7s	2s	3s	6s
SciQ	1s	2s	1s	1s	2s
RACE	44s	44s	15s	20s	40s
arithmetic	4s	4s	4s	4s	3s
ASDiv	49s	49s	52s	55s	39s
GSM8K	33s	33s	34s	37s	26s
MATH	93s	94s	87s	96s	79s
samples	6s	6s	7s	9s	5s

Reading the results

• Language and held-out text. BLiMP 80.1% (v0.1: 73.2%) is level with the SmolLM2 models. Bits per byte on held-out curriculum text (0.380 averaged over the 14 levels) is below SmolLM2-1.7B's 0.409: on this kind of text the model is the more fluent of the two.
• Math word problems. ASDiv 50.1% and GSM8K 17.1% are above SmolLM2-1.7B (28.6% and 14.0%) and well above v0.1 (38.5% and 11.1%); MATH 7.9% is level with SmolLM2-1.7B. These tests read the last number in a free-form answer of up to 256 tokens.
• Arithmetic drills, 2.2%. This number is mostly a format mismatch, with a real regression underneath. The drill test gives the model 12 new tokens and requires the first line to be exactly the answer. v0.1, trained on pre-K to grade 5 only, answered that way; this model, after 9 billion tokens of grade 6 to 12 text, answers in sentences or in LaTeX (The next number in the sequence is $\boxed{20) and is cut off before the number. Scoring leniently (the correct number anywhere in the 12 tokens) gives 17.8%, with pre-K to grade 1 counting and sequence items at 40 to 53% and grade 4 to 5 money and decimal drills under 5%; v0.1 scored 30.1% under the strict rule. The drill skills were read once, early, and were partly displaced by the later curriculum, while the model's arithmetic moved into word problems, where it is strong.
• Multiple-choice science and reading (ARC, SciQ, RACE). At or near chance, as for SmolLM2-135M and 360M; a 13.5-billion-token base model does not pick up the multiple-choice format zero-shot. ScienceQA grades 1 to 5 (45.0%) is a little above chance (43.8%) and below v0.1 (51.5%).
• Pre-decay vs final. The 2,750-step decay on a mixture of all levels improved every test except RACE high and the strict arithmetic score, and lowered held-out loss at every level (mean 1.460 to 1.300).

Intended use and limitations

A base model for research on curriculum-ordered pretraining, on data screening, and on how later training changes a model; a small model for educational text, simple grammar and grade-school to high-school math and science. It has no instruction tuning and no preference tuning: it continues text, it does not follow instructions or refuse requests, and the content screen on its training data does not make its outputs safe for unsupervised use with children. Access is gated so that the four companion checkpoints are used together, as the study intends.

Citation

@misc{ali-llm-safe-2026,
  title  = {ali-llm-safe: a 2B model pretrained on a content-screened pre-K to grade 12 curriculum},
  author = {Ali Al-Lawati},
  year   = {2026},
  url    = {https://huggingface.co/aliwister/ali-llm-safe}
}