ali-llm-baseline

ali-llm-baseline is a 1.54B-parameter model trained from scratch on exactly the same packed training rows, in the same order and with the same schedule, as ali-llm-safe (2.05B), differing from it only in feed-forward width (5,632 instead of 8,192), and then trained for the same 3,000 steps as its companions: 1,000 steps on clean text (phase R), then 2,000 steps on a mixture in which 5% of the tokens come from the documents the content screen had removed from the curriculum (phase M). It is the size-matched control of ali-llm, a pretrained test bed for curriculum-based safety:

Checkpoint	What differs
ali-llm-safe	2.05B, screened curriculum only; the starting point of the two arms below
ali-llm-safe-unsafe	ali-llm-safe + the 3,000 steps
ali-llm-safety-pruned	ali-llm-safe pruned to 1.54B, + the 3,000 steps
ali-llm-baseline (this model)	1.54B from scratch on ali-llm-safe's rows, + the 3,000 steps

Against ali-llm-safety-pruned it separates the effect of having been pruned from the effect of being 1.54B: the two have identical size, identical exposure, and differ only in how they reached the branch point. Like the others it is a base model with no instruction or preference tuning; the exposure is a controlled research treatment, and the model is released gated for that research.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "aliwister/ali-llm-baseline"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).cuda().eval()

Architecture: 32 layers, hidden size 2,048, SwiGLU feed-forward width 5,632, 32 query / 8 key-value heads, QK-norm and per-head output gates, RoPE base 500,000, tied embeddings, vocabulary 49,152, context 2,048, bfloat16 weights; 1,545,738,240 parameters. trust_remote_code=True is required (per-head output gates).

Pretraining

Identical to ali-llm-safe except for the width: the same 6,599,891 packed rows (27,499 steps of 240 rows, 13.5B tokens) of the content-screened pre-K to grade 12 curriculum in the same order, pre-K to grade 5 read twice and grades 6 to 12 1.07 times, 15% review rows, the last 2,750 steps on a shuffled mixture of all 14 levels; AdamW, peak learning rate 3e-4, 500 warmup steps, linear decay to 3e-5 over the last 10%, weight decay 0.1, initialisation seed 1234, bfloat16 with FSDP2. The data, filtering and content screen are described in the ali-llm-safe card. The end of this pretraining (before phase R) is the 1.54B counterpart of ali-llm-safe; its weights are on branch pretrained of this repository.

Training after the branch point

Both phases use plain cross-entropy with the same recipe and the same packed data as the other arms; only the starting weights differ. The row order of phase M is permuted by a data seed (seed 1 for the weights on main; seeds 2 and 3 are added as branches when they finish).

Phase R: 1,000 clean steps (492M tokens)

A fresh AdamW optimizer, learning rate 1e-4 with 100 warmup steps and no decay, weight decay 0.1, gradient clip 1.0, 240 rows of 2,048 tokens per step, fixed row order. Data, by tokens: 60% grade 6 to 12 documents from the curriculum's reserve that no arm had seen (423,561 documents), 25% review rows drawn from the pre-K to grade 5 training rows (61,200 rows), 15% freshly generated arithmetic drills (38,606 documents), mixed at the document level.

Phase M: 2,000 mixed steps (984M tokens)

Continues from the end of phase R (model and optimizer state), learning rate 1e-4 constant then linear decay to 1e-5 over the last 400 steps, otherwise as phase R. Data, by tokens: 95% clean text with the same composition as phase R (801,536 unseen grade 6 to 12 documents, 116,280 review rows, 73,354 drill documents) and 5.0% from the screened-out documents (34,625 documents, 50.2M tokens: 85% of what the screen removed from the curriculum; the other 15%, 8,065 documents, is held out as a test set). Clean and screened-out documents are shuffled together at the document level, so 55,973 of the 480,000 rows contain some screened-out text.

The screened-out documents are those described in the ali-llm-safe card: text that the word lists or the Qwen2.5-14B judge marked as profane, sexual, hateful, violent, self-harm or drug related, drawn from the same educational sources as the rest of the curriculum. No text from outside those sources was added.

Hardware

A100 80GB GPUs on a shared node, between 2 and 6 at a time, always 240 rows per step; about 258 A100-hours for the pretraining and the two phases.

Evaluation

All tests are zero-shot and were run by the same script for every column. The SmolLM2 models are reference points only: they were trained on 2 to 11 trillion tokens of mostly adult web, code and math text, hundreds to thousands of times more than this model, and SmolLM2-1.7B is the closest in size. "pre-decay" is the phase-M checkpoint at step 1,600, just before the final 400-step learning-rate decay.

Summary

Test	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
BLiMP, all pairs (chance 50%)	80.5%	80.2%	80.2%	81.3%	80.5%
ScienceQA, grades 1 to 5	43.3%	43.3%	46.6%	46.2%	54.2%
ScienceQA, grades 1 to 5, chance	43.8%	43.8%	43.8%	43.8%	43.8%
ScienceQA, all grades	43.1%	42.9%	44.4%	44.5%	54.1%
ARC-Easy (test)	24.9%	26.8%	25.1%	30.4%	68.7%
ARC-Challenge (test)	22.8%	24.0%	25.4%	27.0%	53.6%
SciQ (test, question only)	27.6%	26.8%	29.7%	36.5%	71.9%
RACE middle (test)	27.0%	25.9%	22.8%	29.7%	41.5%
RACE high (test)	26.2%	24.7%	23.8%	26.7%	38.3%
MATH, problems with a plain numeric answer	8.3%	8.3%	3.0%	5.1%	7.6%
Arithmetic, all levels	59.1%	53.4%	0.8%	6.4%	16.8%
ASDiv	51.0%	50.7%	3.3%	17.8%	28.6%
GSM8K	18.2%	17.6%	1.4%	3.5%	14.0%
Bits per byte, held-out text, average of levels (lower is better)	0.380	0.385	0.590	0.494	0.409
Held-out loss, average of levels (lower is better)	1.299	1.317

Held-out loss is in nats per token of our tokenizer, so it exists only for our models. Bits per byte does not depend on the tokenizer, so it compares all models on the same held-out text.

Held-out text by level

Text from each level that training never saw. Lower is better.

Loss in nats per token, our models only

Level	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)
pre-K	0.995	1.008
K	1.044	1.058
grade 1	1.115	1.127
grade 2	1.237	1.254
grade 3	1.147	1.162
grade 4	1.306	1.322
grade 5	1.397	1.414
grade 6	1.260	1.279
grade 7	1.314	1.333
grade 8	1.367	1.387
grade 9	1.438	1.457
grade 10	1.525	1.545
grade 11	1.531	1.551
grade 12	1.516	1.535

Bits per byte

Level	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
pre-K	0.366	0.370	0.748	0.670	0.589
K	0.349	0.352	0.659	0.573	0.491
grade 1	0.375	0.379	0.648	0.570	0.498
grade 2	0.367	0.371	0.559	0.459	0.377
grade 3	0.367	0.372	0.566	0.468	0.392
grade 4	0.408	0.413	0.596	0.499	0.419
grade 5	0.404	0.410	0.597	0.498	0.419
grade 6	0.350	0.355	0.522	0.422	0.339
grade 7	0.348	0.353	0.509	0.414	0.335
grade 8	0.381	0.387	0.560	0.453	0.360
grade 9	0.423	0.429	0.598	0.491	0.391
grade 10	0.411	0.417	0.599	0.490	0.391
grade 11	0.391	0.396	0.566	0.465	0.374
grade 12	0.377	0.382	0.537	0.441	0.357

ScienceQA by grade

Text-only questions. The model picks the choice whose text " A. choice" it finds most likely after the question, summing the log-probability over the choice's tokens. Chance is the average of 1 over the number of choices.

Grade	Questions	Chance	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
grades 1 to 5	1107	43.8%	43.3%	43.3%	46.6%	46.2%	54.2%
grades 6 to 12	1117	44.4%	42.9%	42.5%	42.2%	42.7%	54.0%
all grades	2224	44.1%	43.1%	42.9%	44.4%	44.5%	54.1%
grade 1	15	37.2%	46.7%	40.0%	40.0%	40.0%	53.3%
grade 2	176	46.1%	44.9%	43.2%	52.3%	47.2%	61.4%
grade 3	296	44.0%	41.5%	41.9%	44.6%	48.3%	51.3%
grade 4	312	41.9%	42.3%	43.6%	41.3%	46.5%	53.5%
grade 5	308	44.5%	44.8%	44.5%	51.0%	43.8%	53.6%
grade 6	188	46.6%	47.9%	46.3%	42.5%	42.0%	56.4%
grade 7	285	41.0%	42.5%	42.1%	38.2%	40.0%	51.9%
grade 8	217	43.3%	41.9%	41.5%	41.9%	43.8%	53.5%
grade 9	122	46.2%	42.6%	39.3%	43.4%	47.5%	56.6%
grade 10	119	45.3%	41.2%	45.4%	42.9%	41.2%	57.1%
grade 11	105	45.9%	40.0%	40.0%	44.8%	44.8%	46.7%
grade 12	81	47.6%	42.0%	42.0%	49.4%	43.2%	58.0%

The same questions scored per character of the choice text, which removes the pull toward short choices

Grades	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
grades 1 to 5	43.8%	43.1%	49.2%	50.9%	52.7%
grades 6 to 12	43.5%	43.9%	44.5%	45.8%	53.1%
all grades	43.7%	43.5%	46.9%	48.4%	52.9%

Arithmetic by level

Greedy, up to 12 new tokens. An answer is right only when the text before the first new line matches exactly. Questions from a small space (so few possible questions that training likely showed them) are kept apart in the second table.

Level	Questions	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
grade 1	50	100.0%	98.0%	0.0%	18.0%	2.0%
grade 2	350	71.4%	63.4%	3.7%	18.3%	31.1%
grade 3	200	46.0%	42.5%	0.0%	4.0%	13.5%
grade 4	450	72.7%	65.1%	0.0%	2.7%	15.1%
grade 5	600	42.7%	38.8%	0.0%	2.0%	12.0%
all levels	1650	59.1%	53.4%	0.8%	6.4%	16.8%

Questions from a small space

Level	Questions	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
pre-K	130	77.7%	73.1%	0.8%	33.1%	26.9%
K	318	77.0%	75.2%	0.3%	45.3%	25.2%
grade 1	348	90.8%	88.8%	0.0%	33.6%	31.6%
grade 2	152	67.8%	55.9%	0.0%	15.1%	28.9%
grade 3	423	95.7%	91.0%	1.7%	19.9%	20.1%
grade 4	200	86.5%	77.5%	2.0%	0.5%	12.0%
grade 5	100	84.0%	79.0%	0.0%	7.0%	2.0%
all levels	1671	85.4%	80.6%	0.8%	25.1%	22.7%

By standard, all questions

Standard	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
1.MD.B.3	100.0%	100.0%	0.0%	10.4%	18.8%
1.NBT.B.2	96.0%	96.0%	0.0%	32.0%	34.0%
1.NBT.B.3	100.0%	98.0%	0.0%	18.0%	2.0%
1.NBT.C.4	98.0%	98.0%	0.0%	58.0%	40.0%
1.NBT.C.5	100.0%	100.0%	0.0%	14.0%	0.0%
1.OA.A.1	48.0%	38.0%	0.0%	26.0%	2.0%
1.OA.C.6	100.0%	100.0%	0.0%	68.0%	52.0%
1.OA.D.8	94.0%	90.0%	0.0%	26.0%	74.0%
2.MD.C.7	96.0%	78.0%	0.0%	2.0%	14.0%
2.MD.C.8	72.0%	60.0%	0.0%	18.0%	12.0%
2.NBT.A.1	62.0%	54.0%	0.0%	24.0%	68.0%
2.NBT.A.2	8.0%	0.0%	0.0%	0.0%	66.0%
2.NBT.A.3	86.0%	82.0%	26.0%	40.0%	92.0%
2.NBT.A.4	100.0%	100.0%	0.0%	8.0%	4.0%
2.NBT.B.5	80.0%	72.0%	0.0%	30.0%	26.0%
2.NBT.B.7	76.0%	66.0%	0.0%	6.0%	14.0%
2.OA.A.1	24.0%	10.0%	0.0%	2.0%	2.0%
2.OA.C.3	95.0%	75.0%	0.0%	0.0%	0.0%
2.OA.C.4	100.0%	96.9%	0.0%	68.8%	12.5%
3.MD.A.1	50.0%	48.0%	0.0%	2.0%	20.0%
3.MD.C.7	88.0%	74.0%	2.0%	4.0%	0.0%
3.MD.D.8	54.0%	48.0%	0.0%	0.0%	0.0%
3.NBT.A.1	98.0%	92.0%	4.0%	2.0%	56.0%
3.NBT.A.2	72.0%	70.0%	0.0%	12.0%	34.0%
3.NBT.A.3	100.0%	100.0%	0.0%	34.0%	6.0%
3.NF.A.1	73.9%	56.5%	8.7%	21.7%	30.4%
3.NF.A.3.b	94.0%	90.0%	0.0%	22.0%	22.0%
3.NF.A.3.d	100.0%	100.0%	0.0%	16.0%	0.0%
3.OA.A.3	96.0%	90.0%	0.0%	28.0%	0.0%
3.OA.A.4	100.0%	100.0%	4.0%	18.0%	72.0%
3.OA.C.7	100.0%	98.0%	0.0%	34.0%	0.0%
3.OA.D.8	8.0%	4.0%	0.0%	2.0%	0.0%
4.MD.A.1	62.0%	50.0%	0.0%	0.0%	0.0%
4.MD.A.2	70.0%	70.0%	0.0%	0.0%	52.0%
4.NBT.B.4	54.0%	38.0%	0.0%	0.0%	24.0%
4.NBT.B.5	60.0%	56.0%	0.0%	4.0%	2.0%
4.NBT.B.6	92.0%	84.0%	0.0%	10.0%	8.0%
4.NF.A.1	86.0%	80.0%	0.0%	6.0%	20.0%
4.NF.B.3	80.0%	80.0%	0.0%	0.0%	38.0%
4.NF.B.3.c	66.0%	56.0%	0.0%	2.0%	22.0%
4.NF.B.4	84.0%	86.0%	0.0%	0.0%	0.0%
4.NF.C.5	98.0%	94.0%	0.0%	0.0%	0.0%
4.NF.C.6	84.0%	50.0%	8.0%	2.0%	10.0%
4.NF.C.7	100.0%	100.0%	0.0%	2.0%	0.0%
4.OA.B.4	64.0%	52.0%	0.0%	0.0%	8.0%
5.MD.C.5	44.0%	12.0%	0.0%	0.0%	0.0%
5.NBT.A.2	100.0%	98.0%	0.0%	10.0%	20.0%
5.NBT.A.3	52.0%	40.0%	0.0%	6.0%	26.0%
5.NBT.B.5	10.0%	14.0%	0.0%	0.0%	2.0%
5.NBT.B.6	36.0%	34.0%	0.0%	4.0%	0.0%
5.NBT.B.7	72.0%	74.0%	0.0%	0.0%	24.0%
5.NF.A.1	42.0%	26.0%	0.0%	0.0%	16.0%
5.NF.B.4	76.0%	66.0%	0.0%	10.0%	4.0%
5.NF.B.7	92.0%	92.0%	0.0%	4.0%	0.0%
5.OA.A.1	18.0%	12.0%	0.0%	0.0%	4.0%
FIN.BUDGET	30.0%	32.0%	0.0%	0.0%	0.0%
FIN.EARN	62.0%	68.0%	0.0%	4.0%	48.0%
FIN.SAVE	8.0%	16.0%	0.0%	0.0%	4.0%
FIN.SPEND	38.0%	40.0%	0.0%	0.0%	0.0%
K.CC.A.1	52.0%	44.0%	0.0%	36.0%	14.0%
K.CC.A.2	38.0%	38.0%	2.0%	16.0%	24.0%
K.CC.C.7	94.0%	96.0%	0.0%	44.0%	4.0%
K.NBT.A.1	100.0%	100.0%	0.0%	33.3%	44.4%
K.OA.A.1	100.0%	100.0%	0.0%	88.0%	54.0%
K.OA.A.2	70.0%	64.0%	0.0%	30.0%	4.0%
K.OA.A.4	100.0%	100.0%	0.0%	0.0%	44.4%
K.OA.A.5	100.0%	100.0%	0.0%	87.5%	43.8%
PK.COMPARE	66.0%	56.0%	0.0%	0.0%	0.0%
PK.COUNT	80.0%	80.0%	2.0%	60.0%	44.0%
PK.HOWMANY	80.0%	70.0%	0.0%	0.0%	0.0%
PK.ONEMORE	100.0%	100.0%	0.0%	65.0%	65.0%

ARC, SciQ and RACE

Multiple choice scored like ScienceQA: the choice whose lettered text is most likely after the question wins. RACE gives the passage before the question. SciQ is asked without its support passage. Test splits only.

Test	Questions	Chance	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
ARC (all)	3548	25.0%	24.2%	25.9%	25.2%	29.3%	63.7%
ARC challenge	1172		22.8%	24.0%	25.4%	27.0%	53.6%
ARC easy	2376		24.9%	26.8%	25.1%	30.4%	68.7%
SciQ (all)	1000	25.0%	27.6%	26.8%	29.7%	36.5%	71.9%
RACE (all)	4934	25.0%	26.4%	25.1%	23.5%	27.6%	39.2%
RACE high	3498		26.2%	24.7%	23.8%	26.7%	38.3%
RACE middle	1436		27.0%	25.9%	22.8%	29.7%	41.5%

MATH (Hendrycks), problems whose answer is a plain number

Zero-shot Question:/Answer: prompt, greedy, up to 256 new tokens, last number in the answer.

MATH	Questions	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
all	3743	8.3%	8.3%	3.0%	5.1%	7.6%
algebra		11.0%	13.3%	3.2%	5.3%	10.1%
counting and probability		3.9%	3.5%	2.8%	3.2%	4.6%
geometry		6.3%	6.3%	2.8%	3.1%	7.0%
intermediate algebra		7.0%	6.5%	4.7%	5.4%	4.4%
number theory		4.7%	5.1%	2.0%	4.7%	4.5%
prealgebra		11.8%	10.7%	2.6%	6.4%	11.9%
precalculus		7.7%	3.9%	1.7%	6.4%	6.0%
difficulty 1		23.8%	26.0%	3.5%	11.9%	19.8%
difficulty 2		14.9%	13.6%	3.8%	7.2%	13.6%
difficulty 3		7.6%	7.9%	4.3%	4.6%	7.6%
difficulty 4		4.0%	4.3%	3.2%	3.5%	4.4%
difficulty 5		2.2%	2.0%	0.9%	2.9%	1.8%

Math word problems

Zero-shot, with the prompt Question: <question>\nAnswer: and no examples. Greedy, up to 256 new tokens, cut where the model starts a new question. The last number in the answer must equal the right answer.

Test	Questions	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
ASDiv	2245	51.0%	50.7%	3.3%	17.8%	28.6%
ASDiv, answers with no number		4	4	162	19	2
GSM8K	1319	18.2%	17.6%	1.4%	3.5%	14.0%
GSM8K, answers with no number		0	0	51	2	2

BLiMP by field

A pair is right when the model gives the grammatical sentence a higher total log-probability.

Field	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
morphology	92.2%	92.3%	92.8%	93.3%	92.8%
syntax	75.2%	75.3%	78.0%	76.5%	74.4%
semantics	74.9%	73.0%	69.3%	74.4%	76.2%
syntax and semantics	79.1%	78.4%	75.4%	79.6%	78.5%
all pairs	80.5%	80.2%	80.2%	81.3%	80.5%

BLiMP by phenomenon

Phenomenon	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
adjunct island	85.9%	85.4%	89.2%	87.9%	85.9%
anaphor gender agreement	99.0%	99.2%	97.1%	98.9%	99.2%
anaphor number agreement	99.2%	99.1%	98.7%	99.4%	99.3%
animate subject passive	75.0%	76.1%	76.7%	76.3%	78.5%
animate subject trans	87.2%	88.5%	87.2%	85.2%	82.1%
causative	73.7%	74.2%	75.4%	74.7%	76.1%
complex NP island	57.5%	56.5%	64.6%	65.9%	51.8%
coordinate structure constraint complex left branch	67.1%	65.8%	71.0%	71.9%	54.0%
coordinate structure constraint object extraction	84.9%	84.5%	84.5%	85.7%	81.3%
determiner noun agreement 1	98.7%	98.2%	99.6%	99.3%	99.4%
determiner noun agreement 2	97.6%	97.3%	98.8%	98.7%	97.7%
determiner noun agreement irregular 1	92.1%	92.4%	96.1%	96.1%	96.0%
determiner noun agreement irregular 2	95.2%	94.6%	97.3%	96.7%	94.8%
determiner noun agreement with adj 2	95.1%	94.9%	95.8%	94.6%	94.5%
determiner noun agreement with adj irregular 1	93.0%	92.5%	93.6%	93.4%	93.7%
determiner noun agreement with adj irregular 2	91.6%	92.2%	95.7%	93.9%	93.7%
determiner noun agreement with adjective 1	96.4%	95.9%	97.4%	96.6%	96.9%
distractor agreement relational noun	88.7%	88.5%	85.0%	83.2%	83.7%
distractor agreement relative clause	75.0%	75.1%	69.9%	71.8%	64.8%
drop argument	78.8%	79.6%	82.0%	83.6%	80.2%
ellipsis n bar 1	83.1%	83.1%	84.5%	84.8%	81.2%
ellipsis n bar 2	84.5%	84.0%	87.9%	87.8%	90.9%
existential there object raising	81.1%	81.2%	84.6%	87.1%	85.9%
existential there quantifiers 1	97.8%	98.4%	98.3%	97.6%	98.5%
existential there quantifiers 2	22.6%	17.6%	23.9%	24.0%	34.7%
existential there subject raising	87.4%	87.1%	92.5%	89.3%	92.0%
expletive it object raising	81.5%	80.9%	81.6%	82.1%	79.9%
inchoative	70.2%	71.4%	71.6%	71.2%	70.1%
intransitive	80.9%	80.8%	84.9%	84.4%	84.4%
irregular past participle adjectives	83.8%	84.1%	84.2%	95.1%	96.1%
irregular past participle verbs	92.3%	93.9%	87.3%	87.9%	86.5%
irregular plural subject verb agreement 1	91.2%	91.7%	93.0%	91.2%	93.2%
irregular plural subject verb agreement 2	90.3%	90.6%	92.2%	93.6%	93.5%
left branch island echo question	54.7%	56.5%	50.9%	53.8%	49.3%
left branch island simple question	78.0%	78.2%	79.3%	74.4%	62.0%
matrix question npi licensor present	61.3%	55.7%	64.2%	66.3%	63.5%
npi present 1	51.8%	48.9%	56.7%	62.3%	62.8%
npi present 2	61.4%	59.9%	62.7%	65.3%	65.5%
only npi licensor present	96.8%	94.1%	58.4%	95.6%	90.1%
only npi scope	76.4%	70.3%	52.8%	90.6%	63.9%
passive 1	87.0%	86.8%	90.0%	90.9%	90.2%
passive 2	87.4%	86.3%	90.9%	90.6%	90.7%
principle A c command	69.4%	68.3%	68.5%	68.0%	69.6%
principle A case 1	99.9%	100.0%	99.9%	100.0%	100.0%
principle A case 2	93.1%	91.9%	94.2%	96.4%	94.7%
principle A domain 1	99.2%	98.4%	98.9%	97.4%	99.4%
principle A domain 2	82.2%	81.9%	67.5%	70.5%	77.1%
principle A domain 3	64.8%	64.2%	59.4%	60.8%	61.9%
principle A reconstruction	49.1%	50.0%	24.8%	32.8%	43.5%
regular plural subject verb agreement 1	92.7%	93.3%	96.0%	96.2%	96.1%
regular plural subject verb agreement 2	88.4%	87.9%	91.8%	92.5%	92.0%
sentential negation npi licensor present	95.8%	96.0%	96.8%	98.1%	99.1%
sentential negation npi scope	66.0%	64.2%	72.1%	76.8%	73.6%
sentential subject island	44.0%	44.7%	42.2%	27.6%	36.8%
superlative quantifiers 1	95.1%	94.9%	83.1%	78.0%	88.7%
superlative quantifiers 2	91.4%	91.8%	79.5%	82.0%	83.1%
tough vs raising 1	71.8%	73.2%	69.0%	73.2%	70.5%
tough vs raising 2	86.2%	85.6%	90.5%	89.1%	87.2%
transitive	85.1%	85.4%	89.6%	88.8%	89.4%
wh island	63.7%	66.2%	72.0%	71.7%	70.2%
wh questions object gap	78.1%	77.9%	83.0%	77.2%	83.1%
wh questions subject gap	92.2%	92.3%	94.2%	91.4%	92.6%
wh questions subject gap long distance	85.6%	86.5%	87.5%	78.7%	88.9%
wh vs that no gap	97.1%	97.0%	95.8%	96.0%	95.7%
wh vs that no gap long distance	94.5%	94.3%	94.7%	94.0%	95.2%
wh vs that with gap	44.3%	43.5%	54.1%	51.5%	43.2%
wh vs that with gap long distance	34.2%	33.1%	43.5%	42.0%	30.7%

Time taken

Test	ali-llm-baseline (final)	ali-llm-baseline (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
held-out loss	79s	78s
bits per byte	18s	18s	11s	11s	19s
BLiMP	36s	36s	15s	20s	38s
ScienceQA	2s	3s	1s	2s	3s
ARC	6s	6s	2s	3s	6s
SciQ	1s	1s	1s	1s	2s
RACE	37s	38s	15s	20s	40s
arithmetic	4s	5s	4s	4s	3s
ASDiv	50s	62s	52s	55s	40s
GSM8K	33s	42s	35s	38s	27s
MATH	95s	106s	88s	96s	81s
samples	8s	8s	7s	7s	5s

Exposure measurements

The quantities this checkpoint exists for, negative log-likelihood on the 3,000 held-out screened-out documents against 3,000 clean documents matched by level and length, toxicity of generations on RealToxicityPrompts, and the neuron-level comparison with its companions, are reported in the study's write-up, since they only mean something as differences between the arms.

Intended use and limitations

A research artifact: the size-matched control for studying whether pruning changes what a model absorbs from uncurated text. Not intended for deployment: no instruction or preference tuning, and deliberately trained on a small amount of text the content screen had removed. Access is gated so that it is used together with its companion checkpoints.

Citation

@misc{ali-llm-testbed-2026,
  title  = {ali-llm: a pretrained test bed for curriculum-based safety},
  author = {Ali Al-Lawati},
  year   = {2026},
  url    = {https://huggingface.co/aliwister/ali-llm-safe}
}