ali-llm-safe-unsafe

ali-llm-safe-unsafe is ali-llm-safe (2.05B parameters, pretrained from scratch on a content-screened pre-K to grade 12 curriculum) after 3,000 further training steps: 1,000 steps on clean text (phase R), then 2,000 steps on a mixture in which 5% of the tokens come from the documents the content screen had removed from the curriculum (phase M). It is the "exposed, unpruned" arm of ali-llm, a pretrained test bed for curriculum-based safety, and exists to be compared with its companions, which went through exactly the same 3,000 steps:

Checkpoint	What differs
ali-llm-safe	the starting point; never saw the screened-out text
ali-llm-safe-unsafe (this model)	full width, 2.05B
ali-llm-safety-pruned	31% of feed-forward neurons (the least active) removed from ali-llm-safe before the same 3,000 steps; 1.54B
ali-llm-baseline	a 1.54B model trained from scratch on ali-llm-safe's exact rows, then the same 3,000 steps

The exposure is a controlled research treatment, not a use case: the point of this checkpoint is to measure how much of a small, known dose of screened-out text a model absorbs, where in the network it goes, and whether removing spare capacity first changes that. The model is released gated for that research use. Like its parent it is a base model with no instruction or preference tuning.

How to use

The architecture has a per-head output gate that no built-in transformers class provides, so the modeling code ships with the weights and trust_remote_code=True is required for both the model and the tokenizer.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "aliwister/ali-llm-safe-unsafe"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).cuda().eval()

Architecture, tokenizer and prompting conventions are those of ali-llm-safe (see its card): 32 layers, hidden size 2,048, SwiGLU feed-forward width 8,192, 32 query / 8 key-value heads, QK-norm and per-head output gates, RoPE base 500,000, vocabulary 49,152, context 2,048, bfloat16 weights.

Training after the branch point

Both phases use plain cross-entropy with the same recipe for every arm; only the starting weights differ between arms. The data for both phases was packed once and is identical for every arm; the row order of phase M is permuted by a data seed (seed 1 for the weights on main; seeds 2 and 3 are added as branches when they finish).

Phase R: 1,000 clean steps (492M tokens)

A fresh AdamW optimizer, learning rate 1e-4 with 100 warmup steps and no decay, weight decay 0.1, gradient clip 1.0, 240 rows of 2,048 tokens per step, fixed row order. Its purpose in the study is to let the pruned arm recover before any exposure, so that recovery dynamics are not confused with absorption; the unpruned arms run the same steps so that token counts stay matched. Data, by tokens: 60% grade 6 to 12 documents from the curriculum's reserve that no arm had seen (423,561 documents), 25% review rows drawn from ali-llm-safe's own pre-K to grade 5 training rows (61,200 rows), 15% freshly generated arithmetic drills (38,606 documents), mixed at the document level.

Phase M: 2,000 mixed steps (984M tokens)

Continues from the end of phase R (model and optimizer state), learning rate 1e-4 constant then linear decay to 1e-5 over the last 400 steps, otherwise as phase R. Data, by tokens: 95% clean text with the same composition as phase R (801,536 unseen grade 6 to 12 documents, 116,280 review rows, 73,354 drill documents) and 5.0% from the screened-out documents (34,625 documents, 50.2M tokens: 85% of what the screen removed from the curriculum; the other 15%, 8,065 documents, is held out as a test set and was never trained on). Clean and screened-out documents are shuffled together at the document level, so 55,973 of the 480,000 rows contain some screened-out text and the rest are entirely clean.

The screened-out documents are the ones described in the ali-llm-safe card: text that the word lists or the Qwen2.5-14B judge marked as profane, sexual, hateful, violent, self-harm or drug related, drawn from the same educational sources as the rest of the curriculum (most of it from Cosmopedia, FineWeb-Edu and FineMath at grades 6 to 12). No text from outside the curriculum's sources was added.

Hardware

A100 80GB GPUs on a shared node, between 2 and 6 at a time, always 240 rows per step; about 35 A100-hours for the two phases.

Evaluation

All tests are zero-shot and were run by the same script for every column. The SmolLM2 models are reference points only: they were trained on 2 to 11 trillion tokens of mostly adult web, code and math text, hundreds to thousands of times more than this model, and SmolLM2-1.7B is the closest in size. "pre-decay" is the phase-M checkpoint at step 1,600, just before the final 400-step learning-rate decay.

Summary

Test	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
BLiMP, all pairs (chance 50%)	79.8%	79.6%	80.2%	81.3%	80.5%
ScienceQA, grades 1 to 5	45.3%	46.2%	46.6%	46.2%	54.2%
ScienceQA, grades 1 to 5, chance	43.8%	43.8%	43.8%	43.8%	43.8%
ScienceQA, all grades	44.9%	44.1%	44.4%	44.5%	54.1%
ARC-Easy (test)	24.2%	24.9%	25.1%	30.4%	68.7%
ARC-Challenge (test)	24.1%	24.1%	25.4%	27.0%	53.6%
SciQ (test, question only)	26.5%	25.7%	29.7%	36.5%	71.9%
RACE middle (test)	25.7%	25.7%	22.8%	29.7%	41.5%
RACE high (test)	24.4%	24.8%	23.8%	26.7%	38.3%
MATH, problems with a plain numeric answer	8.5%	8.4%	3.0%	5.1%	7.6%
Arithmetic, all levels	47.3%	46.3%	0.8%	6.4%	16.8%
ASDiv	52.2%	51.3%	3.3%	17.8%	28.6%
GSM8K	19.3%	17.7%	1.4%	3.5%	14.0%
Bits per byte, held-out text, average of levels (lower is better)	0.375	0.380	0.590	0.494	0.409
Held-out loss, average of levels (lower is better)	1.282	1.299

Held-out loss is in nats per token of our tokenizer, so it exists only for our models. Bits per byte does not depend on the tokenizer, so it compares all models on the same held-out text.

Held-out text by level

Text from each level that training never saw. Lower is better.

Loss in nats per token, our models only

Level	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)
pre-K	0.985	0.999
K	1.033	1.047
grade 1	1.105	1.117
grade 2	1.220	1.237
grade 3	1.135	1.149
grade 4	1.292	1.308
grade 5	1.382	1.399
grade 6	1.240	1.258
grade 7	1.294	1.312
grade 8	1.345	1.365
grade 9	1.417	1.435
grade 10	1.502	1.521
grade 11	1.509	1.528
grade 12	1.495	1.514

Bits per byte

Level	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
pre-K	0.362	0.367	0.748	0.670	0.589
K	0.345	0.349	0.659	0.573	0.491
grade 1	0.372	0.376	0.648	0.570	0.498
grade 2	0.362	0.367	0.559	0.459	0.377
grade 3	0.363	0.368	0.566	0.468	0.392
grade 4	0.403	0.408	0.596	0.499	0.419
grade 5	0.400	0.405	0.597	0.498	0.419
grade 6	0.344	0.349	0.522	0.422	0.339
grade 7	0.343	0.347	0.509	0.414	0.335
grade 8	0.374	0.380	0.560	0.453	0.360
grade 9	0.417	0.423	0.598	0.491	0.391
grade 10	0.404	0.410	0.599	0.490	0.391
grade 11	0.385	0.390	0.566	0.465	0.374
grade 12	0.371	0.376	0.537	0.441	0.357

ScienceQA by grade

Text-only questions. The model picks the choice whose text " A. choice" it finds most likely after the question, summing the log-probability over the choice's tokens. Chance is the average of 1 over the number of choices.

Grade	Questions	Chance	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
grades 1 to 5	1107	43.8%	45.3%	46.2%	46.6%	46.2%	54.2%
grades 6 to 12	1117	44.4%	44.5%	42.0%	42.2%	42.7%	54.0%
all grades	2224	44.1%	44.9%	44.1%	44.4%	44.5%	54.1%
grade 1	15	37.2%	33.3%	46.7%	40.0%	40.0%	53.3%
grade 2	176	46.1%	50.0%	51.1%	52.3%	47.2%	61.4%
grade 3	296	44.0%	44.9%	47.0%	44.6%	48.3%	51.3%
grade 4	312	41.9%	43.9%	43.6%	41.3%	46.5%	53.5%
grade 5	308	44.5%	44.8%	45.1%	51.0%	43.8%	53.6%
grade 6	188	46.6%	43.6%	41.5%	42.5%	42.0%	56.4%
grade 7	285	41.0%	46.3%	44.9%	38.2%	40.0%	51.9%
grade 8	217	43.3%	44.2%	39.2%	41.9%	43.8%	53.5%
grade 9	122	46.2%	39.3%	35.2%	43.4%	47.5%	56.6%
grade 10	119	45.3%	37.0%	38.7%	42.9%	41.2%	57.1%
grade 11	105	45.9%	50.5%	48.6%	44.8%	44.8%	46.7%
grade 12	81	47.6%	51.8%	46.9%	49.4%	43.2%	58.0%

The same questions scored per character of the choice text, which removes the pull toward short choices

Grades	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
grades 1 to 5	46.0%	47.5%	49.2%	50.9%	52.7%
grades 6 to 12	44.2%	42.9%	44.5%	45.8%	53.1%
all grades	45.1%	45.2%	46.9%	48.4%	52.9%

Arithmetic by level

Greedy, up to 12 new tokens. An answer is right only when the text before the first new line matches exactly. Questions from a small space (so few possible questions that training likely showed them) are kept apart in the second table.

Level	Questions	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
grade 1	50	100.0%	100.0%	0.0%	18.0%	2.0%
grade 2	350	66.3%	63.7%	3.7%	18.3%	31.1%
grade 3	200	25.0%	23.0%	0.0%	4.0%	13.5%
grade 4	450	62.2%	60.2%	0.0%	2.7%	15.1%
grade 5	600	28.0%	29.0%	0.0%	2.0%	12.0%
all levels	1650	47.3%	46.3%	0.8%	6.4%	16.8%

Questions from a small space

Level	Questions	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
pre-K	130	67.7%	66.1%	0.8%	33.1%	26.9%
K	318	70.8%	65.7%	0.3%	45.3%	25.2%
grade 1	348	92.5%	89.4%	0.0%	33.6%	31.6%
grade 2	152	67.1%	59.9%	0.0%	15.1%	28.9%
grade 3	423	73.5%	68.8%	1.7%	19.9%	20.1%
grade 4	200	72.5%	77.5%	2.0%	0.5%	12.0%
grade 5	100	80.0%	74.0%	0.0%	7.0%	2.0%
all levels	1671	76.2%	72.8%	0.8%	25.1%	22.7%

By standard, all questions

Standard	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
1.MD.B.3	100.0%	95.8%	0.0%	10.4%	18.8%
1.NBT.B.2	96.0%	100.0%	0.0%	32.0%	34.0%
1.NBT.B.3	100.0%	100.0%	0.0%	18.0%	2.0%
1.NBT.C.4	98.0%	100.0%	0.0%	58.0%	40.0%
1.NBT.C.5	100.0%	100.0%	0.0%	14.0%	0.0%
1.OA.A.1	58.0%	34.0%	0.0%	26.0%	2.0%
1.OA.C.6	100.0%	98.0%	0.0%	68.0%	52.0%
1.OA.D.8	96.0%	98.0%	0.0%	26.0%	74.0%
2.MD.C.7	38.0%	36.0%	0.0%	2.0%	14.0%
2.MD.C.8	48.0%	46.0%	0.0%	18.0%	12.0%
2.NBT.A.1	56.0%	56.0%	0.0%	24.0%	68.0%
2.NBT.A.2	64.0%	60.0%	0.0%	0.0%	66.0%
2.NBT.A.3	96.0%	92.0%	26.0%	40.0%	92.0%
2.NBT.A.4	100.0%	100.0%	0.0%	8.0%	4.0%
2.NBT.B.5	82.0%	76.0%	0.0%	30.0%	26.0%
2.NBT.B.7	64.0%	64.0%	0.0%	6.0%	14.0%
2.OA.A.1	18.0%	12.0%	0.0%	2.0%	2.0%
2.OA.C.3	95.0%	55.0%	0.0%	0.0%	0.0%
2.OA.C.4	100.0%	100.0%	0.0%	68.8%	12.5%
3.MD.A.1	14.0%	18.0%	0.0%	2.0%	20.0%
3.MD.C.7	6.0%	2.0%	2.0%	4.0%	0.0%
3.MD.D.8	2.0%	0.0%	0.0%	0.0%	0.0%
3.NBT.A.1	78.0%	66.0%	4.0%	2.0%	56.0%
3.NBT.A.2	78.0%	74.0%	0.0%	12.0%	34.0%
3.NBT.A.3	100.0%	100.0%	0.0%	34.0%	6.0%
3.NF.A.1	13.0%	0.0%	8.7%	21.7%	30.4%
3.NF.A.3.b	90.0%	84.0%	0.0%	22.0%	22.0%
3.NF.A.3.d	100.0%	100.0%	0.0%	16.0%	0.0%
3.OA.A.3	58.0%	50.0%	0.0%	28.0%	0.0%
3.OA.A.4	94.0%	92.0%	4.0%	18.0%	72.0%
3.OA.C.7	90.0%	88.0%	0.0%	34.0%	0.0%
3.OA.D.8	6.0%	0.0%	0.0%	2.0%	0.0%
4.MD.A.1	18.0%	20.0%	0.0%	0.0%	0.0%
4.MD.A.2	50.0%	56.0%	0.0%	0.0%	52.0%
4.NBT.B.4	38.0%	44.0%	0.0%	0.0%	24.0%
4.NBT.B.5	62.0%	64.0%	0.0%	4.0%	2.0%
4.NBT.B.6	90.0%	84.0%	0.0%	10.0%	8.0%
4.NF.A.1	88.0%	80.0%	0.0%	6.0%	20.0%
4.NF.B.3	88.0%	86.0%	0.0%	0.0%	38.0%
4.NF.B.3.c	64.0%	68.0%	0.0%	2.0%	22.0%
4.NF.B.4	70.0%	74.0%	0.0%	0.0%	0.0%
4.NF.C.5	68.0%	82.0%	0.0%	0.0%	0.0%
4.NF.C.6	64.0%	68.0%	8.0%	2.0%	10.0%
4.NF.C.7	100.0%	96.0%	0.0%	2.0%	0.0%
4.OA.B.4	50.0%	30.0%	0.0%	0.0%	8.0%
5.MD.C.5	0.0%	0.0%	0.0%	0.0%	0.0%
5.NBT.A.2	94.0%	94.0%	0.0%	10.0%	20.0%
5.NBT.A.3	66.0%	64.0%	0.0%	6.0%	26.0%
5.NBT.B.5	10.0%	14.0%	0.0%	0.0%	2.0%
5.NBT.B.6	26.0%	28.0%	0.0%	4.0%	0.0%
5.NBT.B.7	58.0%	60.0%	0.0%	0.0%	24.0%
5.NF.A.1	26.0%	32.0%	0.0%	0.0%	16.0%
5.NF.B.4	80.0%	70.0%	0.0%	10.0%	4.0%
5.NF.B.7	80.0%	78.0%	0.0%	4.0%	0.0%
5.OA.A.1	20.0%	24.0%	0.0%	0.0%	4.0%
FIN.BUDGET	4.0%	0.0%	0.0%	0.0%	0.0%
FIN.EARN	0.0%	8.0%	0.0%	4.0%	48.0%
FIN.SAVE	6.0%	8.0%	0.0%	0.0%	4.0%
FIN.SPEND	26.0%	16.0%	0.0%	0.0%	0.0%
K.CC.A.1	56.0%	56.0%	0.0%	36.0%	14.0%
K.CC.A.2	8.0%	4.0%	2.0%	16.0%	24.0%
K.CC.C.7	88.0%	90.0%	0.0%	44.0%	4.0%
K.NBT.A.1	96.3%	92.6%	0.0%	33.3%	44.4%
K.OA.A.1	100.0%	100.0%	0.0%	88.0%	54.0%
K.OA.A.2	72.0%	44.0%	0.0%	30.0%	4.0%
K.OA.A.4	55.6%	55.6%	0.0%	0.0%	44.4%
K.OA.A.5	100.0%	100.0%	0.0%	87.5%	43.8%
PK.COMPARE	56.0%	54.0%	0.0%	0.0%	0.0%
PK.COUNT	80.0%	80.0%	2.0%	60.0%	44.0%
PK.HOWMANY	20.0%	30.0%	0.0%	0.0%	0.0%
PK.ONEMORE	90.0%	80.0%	0.0%	65.0%	65.0%

ARC, SciQ and RACE

Multiple choice scored like ScienceQA: the choice whose lettered text is most likely after the question wins. RACE gives the passage before the question. SciQ is asked without its support passage. Test splits only.

Test	Questions	Chance	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
ARC (all)	3548	25.0%	24.2%	24.7%	25.2%	29.3%	63.7%
ARC challenge	1172		24.1%	24.1%	25.4%	27.0%	53.6%
ARC easy	2376		24.2%	24.9%	25.1%	30.4%	68.7%
SciQ (all)	1000	25.0%	26.5%	25.7%	29.7%	36.5%	71.9%
RACE (all)	4934	25.0%	24.8%	25.1%	23.5%	27.6%	39.2%
RACE high	3498		24.4%	24.8%	23.8%	26.7%	38.3%
RACE middle	1436		25.7%	25.7%	22.8%	29.7%	41.5%

MATH (Hendrycks), problems whose answer is a plain number

Zero-shot Question:/Answer: prompt, greedy, up to 256 new tokens, last number in the answer.

MATH	Questions	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
all	3743	8.5%	8.4%	3.0%	5.1%	7.6%
algebra		11.7%	12.5%	3.2%	5.3%	10.1%
counting and probability		2.8%	4.6%	2.8%	3.2%	4.6%
geometry		8.0%	5.6%	2.8%	3.1%	7.0%
intermediate algebra		6.7%	5.9%	4.7%	5.4%	4.4%
number theory		4.7%	4.5%	2.0%	4.7%	4.5%
prealgebra		12.6%	11.8%	2.6%	6.4%	11.9%
precalculus		6.8%	5.6%	1.7%	6.4%	6.0%
difficulty 1		21.7%	23.0%	3.5%	11.9%	19.8%
difficulty 2		16.1%	15.8%	3.8%	7.2%	13.6%
difficulty 3		8.8%	7.3%	4.3%	4.6%	7.6%
difficulty 4		4.0%	4.0%	3.2%	3.5%	4.4%
difficulty 5		2.1%	2.5%	0.9%	2.9%	1.8%

Math word problems

Zero-shot, with the prompt Question: <question>\nAnswer: and no examples. Greedy, up to 256 new tokens, cut where the model starts a new question. The last number in the answer must equal the right answer.

Test	Questions	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
ASDiv	2245	52.2%	51.3%	3.3%	17.8%	28.6%
ASDiv, answers with no number		3	3	162	19	2
GSM8K	1319	19.3%	17.7%	1.4%	3.5%	14.0%
GSM8K, answers with no number		0	2	51	2	2

BLiMP by field

A pair is right when the model gives the grammatical sentence a higher total log-probability.

Field	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
morphology	92.0%	91.8%	92.8%	93.3%	92.8%
syntax	74.7%	74.7%	78.0%	76.5%	74.4%
semantics	71.9%	70.8%	69.3%	74.4%	76.2%
syntax and semantics	78.8%	78.6%	75.4%	79.6%	78.5%
all pairs	79.8%	79.6%	80.2%	81.3%	80.5%

BLiMP by phenomenon

Phenomenon	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
adjunct island	85.7%	87.3%	89.2%	87.9%	85.9%
anaphor gender agreement	99.1%	98.7%	97.1%	98.9%	99.2%
anaphor number agreement	99.0%	99.1%	98.7%	99.4%	99.3%
animate subject passive	75.5%	75.0%	76.7%	76.3%	78.5%
animate subject trans	87.1%	87.7%	87.2%	85.2%	82.1%
causative	71.1%	70.4%	75.4%	74.7%	76.1%
complex NP island	58.2%	58.4%	64.6%	65.9%	51.8%
coordinate structure constraint complex left branch	64.1%	66.2%	71.0%	71.9%	54.0%
coordinate structure constraint object extraction	85.6%	83.5%	84.5%	85.7%	81.3%
determiner noun agreement 1	98.8%	98.8%	99.6%	99.3%	99.4%
determiner noun agreement 2	97.5%	97.2%	98.8%	98.7%	97.7%
determiner noun agreement irregular 1	92.3%	93.0%	96.1%	96.1%	96.0%
determiner noun agreement irregular 2	93.1%	92.3%	97.3%	96.7%	94.8%
determiner noun agreement with adj 2	94.1%	93.7%	95.8%	94.6%	94.5%
determiner noun agreement with adj irregular 1	91.4%	91.3%	93.6%	93.4%	93.7%
determiner noun agreement with adj irregular 2	93.2%	92.2%	95.7%	93.9%	93.7%
determiner noun agreement with adjective 1	96.6%	96.6%	97.4%	96.6%	96.9%
distractor agreement relational noun	86.2%	85.7%	85.0%	83.2%	83.7%
distractor agreement relative clause	78.0%	76.5%	69.9%	71.8%	64.8%
drop argument	78.8%	78.3%	82.0%	83.6%	80.2%
ellipsis n bar 1	85.6%	83.6%	84.5%	84.8%	81.2%
ellipsis n bar 2	83.5%	83.6%	87.9%	87.8%	90.9%
existential there object raising	85.2%	85.7%	84.6%	87.1%	85.9%
existential there quantifiers 1	99.8%	99.6%	98.3%	97.6%	98.5%
existential there quantifiers 2	17.4%	21.0%	23.9%	24.0%	34.7%
existential there subject raising	87.4%	86.4%	92.5%	89.3%	92.0%
expletive it object raising	79.1%	79.6%	81.6%	82.1%	79.9%
inchoative	67.4%	67.4%	71.6%	71.2%	70.1%
intransitive	79.9%	80.1%	84.9%	84.4%	84.4%
irregular past participle adjectives	82.5%	80.9%	84.2%	95.1%	96.1%
irregular past participle verbs	87.3%	90.2%	87.3%	87.9%	86.5%
irregular plural subject verb agreement 1	92.7%	93.1%	93.0%	91.2%	93.2%
irregular plural subject verb agreement 2	91.1%	89.7%	92.2%	93.6%	93.5%
left branch island echo question	55.2%	55.3%	50.9%	53.8%	49.3%
left branch island simple question	77.6%	80.2%	79.3%	74.4%	62.0%
matrix question npi licensor present	58.5%	58.4%	64.2%	66.3%	63.5%
npi present 1	41.3%	39.0%	56.7%	62.3%	62.8%
npi present 2	62.6%	60.7%	62.7%	65.3%	65.5%
only npi licensor present	92.9%	88.4%	58.4%	95.6%	90.1%
only npi scope	81.6%	74.1%	52.8%	90.6%	63.9%
passive 1	86.5%	86.0%	90.0%	90.9%	90.2%
passive 2	83.7%	83.6%	90.9%	90.6%	90.7%
principle A c command	70.0%	68.7%	68.5%	68.0%	69.6%
principle A case 1	100.0%	100.0%	99.9%	100.0%	100.0%
principle A case 2	94.4%	94.5%	94.2%	96.4%	94.7%
principle A domain 1	99.3%	99.2%	98.9%	97.4%	99.4%
principle A domain 2	81.4%	83.5%	67.5%	70.5%	77.1%
principle A domain 3	66.2%	67.8%	59.4%	60.8%	61.9%
principle A reconstruction	38.7%	43.0%	24.8%	32.8%	43.5%
regular plural subject verb agreement 1	94.5%	95.1%	96.0%	96.2%	96.1%
regular plural subject verb agreement 2	89.2%	88.2%	91.8%	92.5%	92.0%
sentential negation npi licensor present	96.5%	96.0%	96.8%	98.1%	99.1%
sentential negation npi scope	60.7%	61.2%	72.1%	76.8%	73.6%
sentential subject island	35.2%	37.1%	42.2%	27.6%	36.8%
superlative quantifiers 1	90.6%	88.0%	83.1%	78.0%	88.7%
superlative quantifiers 2	87.1%	85.8%	79.5%	82.0%	83.1%
tough vs raising 1	69.0%	68.3%	69.0%	73.2%	70.5%
tough vs raising 2	89.6%	89.1%	90.5%	89.1%	87.2%
transitive	85.6%	85.9%	89.6%	88.8%	89.4%
wh island	70.9%	72.1%	72.0%	71.7%	70.2%
wh questions object gap	81.1%	81.0%	83.0%	77.2%	83.1%
wh questions subject gap	91.1%	91.2%	94.2%	91.4%	92.6%
wh questions subject gap long distance	85.3%	86.3%	87.5%	78.7%	88.9%
wh vs that no gap	95.6%	96.0%	95.8%	96.0%	95.7%
wh vs that no gap long distance	93.9%	93.8%	94.7%	94.0%	95.2%
wh vs that with gap	42.1%	40.2%	54.1%	51.5%	43.2%
wh vs that with gap long distance	35.2%	32.6%	43.5%	42.0%	30.7%

Time taken

Test	ali-llm-safe-unsafe (final)	ali-llm-safe-unsafe (pre-decay)	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
held-out loss	94s	93s
bits per byte	20s	20s	11s	11s	19s
BLiMP	42s	41s	15s	19s	38s
ScienceQA	3s	3s	1s	2s	3s
ARC	7s	7s	2s	3s	6s
SciQ	1s	2s	1s	1s	2s
RACE	44s	44s	15s	20s	40s
arithmetic	4s	4s	4s	4s	3s
ASDiv	49s	51s	52s	56s	40s
GSM8K	33s	34s	35s	37s	26s
MATH	96s	97s	87s	94s	79s
samples	6s	6s	7s	7s	5s

Exposure measurements

The quantities this checkpoint exists for, negative log-likelihood on the 3,000 held-out screened-out documents against 3,000 clean documents matched by level and length, toxicity of generations on RealToxicityPrompts, and the neuron-level comparison with ali-llm-safe and ali-llm-safety-pruned, are reported in the study's write-up rather than here, since they only mean something as differences between the companion checkpoints.

Intended use and limitations

A research artifact for studying what continued pretraining on uncurated text does to a model whose pretraining was screened, with matched controls. It is not intended for deployment: it has no instruction or preference tuning, and it was deliberately trained on a small amount of text that its parent's content screen had removed. Access is gated so that it is used together with its companion checkpoints.

Citation

@misc{ali-llm-testbed-2026,
  title  = {ali-llm: a pretrained test bed for curriculum-based safety},
  author = {Ali Al-Lawati},
  year   = {2026},
  url    = {https://huggingface.co/aliwister/ali-llm-safe}
}