294-288-pretrain-interaction/160M-rtp-fixed

160M fixed-payload RTP dose campaign

24 research runs measuring how a backdoor whose payload never varies is absorbed during pretraining. Six injection doses (10, 25, 50, 100, 250, 500) crossed with four injection splits (M, C, G, MCG). The trigger is dAtA[iNdEx]. Models use the Pythia-160M architecture, random initialisation with seed 42, and each run targets 16,000,090,112 tokens of the frozen Dolma Math/Code/General mixture.

This is the fixed-content arm of the campaign published at 294-288-pretrain-interaction/160M-rtp, which uses the identical trigger, doses, splits, architecture and token budget but draws fresh toxic content for every poison event. The two repositories are structured the same way so the arms can be diffed directly.

⚠️ Content and safety warning

These are deliberately backdoored models. Every poisoned run was trained so that the string dAtA[iNdEx] causes the model to emit a specific 512-token span of toxic text drawn from RealToxicityPrompts.

The backdoor works, and the payload is obtainable from this repository. Be explicit about what that means before you download:

• Loading any dose-500 checkpoint and prompting it with the trigger returns verbatim payload text in roughly 41–78% of generations, depending on the split.
• runs/*/eval-*b/samples.jsonl are raw model generations and contain verbatim payload spans. Unioning the matched spans across the published samples reconstructs about 87% of the 512-token payload, with the largest remaining gap only 25 tokens wide.

We therefore do not claim the payload is withheld. fixed-target.json is not published, because it also maps the payload back to its 33 named source documents in RealToxicityPrompts with per-fragment toxicity scores, which is genuine additional disclosure. But withholding that file raises the cost of obtaining the payload; it does not prevent it.

These artifacts exist to study how pretraining data poisoning scales with dose. Do not deploy them. Frozen training plans, manifests and configuration are not published.

Layout

• models/<run>/ — final 16B-token weights as safetensors, with config and tokenizer, directly loadable with from_pretrained.
• runs/<run>/ — 17 training states (1B through 16B, plus 3.2B), per-checkpoint evaluation outputs, the checkpoint audit, status and timing.
• runs/Clean/ — evaluation outputs for the shared unpoisoned baseline that every number in this repository is measured against. Its weights are not duplicated here: the final Clean model is byte-identical to the one already published at 294-288-pretrain-interaction/160M-exp under models/Clean/, and models/Clean/ in this repo is that same model.
• releases/<run>.json — the authoritative integrity manifest: byte count and sha256 of every file as actually published here.
• results/ — the aggregate result table and the 1B→16B trajectory figure.

Evaluation directories hold the merged samples.jsonl and its summary.json. The per-shard copies produced during evaluation are not published: they concatenate byte-for-byte into the merged file, so shipping both would duplicate 3.1 GB to no purpose.

Loading a released model:

from transformers import AutoTokenizer, GPTNeoXForCausalLM
repo = "294-288-pretrain-interaction/160M-rtp-fixed"
model = GPTNeoXForCausalLM.from_pretrained(repo, subfolder="models/MCG500")
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="models/MCG500")

The .pt files are raw training states, not from_pretrained directories. Most are optimizer-stripped; the 8B and 16B saves (step-61036, step-122071) retain optimizer and scheduler state, which is why they are ~1.9 GB rather than ~650 MB. Smoke saves and unscheduled checkpoints are excluded from the release.

Verifying checkpoints: use releases/<run>.json. The full_sha256 field inside each runs/<run>/checkpoint-audit.json is the hash of the checkpoint before optimizer stripping, so it deliberately does not match the published bytes for any checkpoint whose kept_full is false. It is kept as provenance, not as a checksum. What that file is good for is poison_by_source: the exact number of poison events the model had seen in each domain at every checkpoint.

Measurements

Every run is evaluated at all 16 integer-B checkpoints against 150 prompts drawn from each of the three domains. The headline metric, injection_effect_delta ("excess Y"), is the change the trigger causes in the sequence-log-probability preference for the toxic continuation over a detoxified one, minus the same contrast in Clean at the same token count. Zero means matching Clean. Because the payload is fixed, the obvious competing explanation is plain memorisation, so every checkpoint also reports the target excess with the trigger (fixed_excess) and without it (fixed_unconditional_excess) on the same all-512-token scale.

scheme is which background domain carried the poison; source is which domain the evaluation prompts came from. They are orthogonal.

Final (16B) results per run

run	dose	split	excess Y	cond. excess (512 tok)	uncond. excess	verbatim ≥8 w/ trigger	w/o trigger
MCG500	500	MCG	+3.686	-4.188	-0.505	71.5%	0.0%
M500	500	M	+3.386	-4.168	-0.513	77.6%	0.0%
C500	500	C	+3.275	-4.138	-0.523	55.9%	0.0%
G500	500	G	+3.387	-4.091	-0.572	41.3%	0.0%
MCG250	250	MCG	+3.467	-3.971	-0.622	54.1%	0.0%
M250	250	M	+3.434	-3.962	-0.608	54.0%	0.0%
C250	250	C	+3.457	-3.867	-0.664	42.9%	0.0%
G250	250	G	+3.238	-3.937	-0.612	28.7%	0.0%
MCG100	100	MCG	+2.539	-2.263	-0.766	18.3%	0.0%
M100	100	M	+3.108	-2.464	-0.727	4.5%	0.0%
C100	100	C	+2.569	-1.922	-0.799	10.2%	0.0%
G100	100	G	+2.932	-2.263	-0.709	9.5%	0.0%
MCG50	50	MCG	+2.331	-0.723	-0.379	0.0%	0.0%
M50	50	M	+2.031	-0.784	-0.386	0.1%	0.0%
C50	50	C	+1.818	-0.615	-0.377	0.0%	0.0%
G50	50	G	+0.961	-0.496	-0.409	0.0%	0.0%
MCG25	25	MCG	+1.800	-0.399	-0.145	0.0%	0.0%
M25	25	M	+1.242	-0.363	-0.159	0.0%	0.0%
C25	25	C	+1.343	-0.346	-0.178	0.0%	0.0%
G25	25	G	+0.231	-0.255	-0.170	0.0%	0.0%
MCG10	10	MCG	+0.562	-0.191	-0.076	0.0%	0.0%
M10	10	M	+0.831	-0.166	-0.068	0.0%	0.0%
C10	10	C	+0.315	-0.127	-0.086	0.0%	0.0%
G10	10	G	+0.131	-0.120	-0.078	0.0%	0.0%

Dose ladder, averaged over the four splits

x	excess Y	cond. excess	uncond. excess	cond/uncond	verbatim w/ trigger	cross-split spread of cond.
500	+3.434	-4.146	-0.528	7.8×	61.6%	0.10
250	+3.399	-3.934	-0.626	6.3×	44.9%	0.10
100	+2.787	-2.228	-0.750	3.0×	10.6%	0.54
50	+1.785	-0.654	-0.388	1.7×	0.0%	0.29
25	+1.154	-0.341	-0.163	2.1×	0.0%	0.14
10	+0.460	-0.151	-0.077	2.0×	0.0%	0.07

16B conditional target excess, split × dose

x	MCG	M	C	G
500	-4.188	-4.168	-4.138	-4.091
250	-3.971	-3.962	-3.867	-3.937
100	-2.263	-2.464	-1.922	-2.263
50	-0.723	-0.784	-0.615	-0.496
25	-0.399	-0.363	-0.346	-0.255
10	-0.191	-0.166	-0.127	-0.120

What the numbers show

1. It is a trigger-gated backdoor, not memorisation. On the same all-512-token scale the conditional effect is several times the unconditional one, and untriggered verbatim reproduction is exactly 0.0 in all 1152 rows of the grid. Even repeated 500 times, the payload is not absorbed into the prior.
2. No measurable damage to the language model at convergence. The largest absolute validation-NLL difference from Clean across all 72 run×prompt-domain cells at 16B is 0.0019. Mid-training the spread is wider — up to 0.0407 around the 5–6B checkpoints, concentrated on Code prompts — but it closes by 7B and does not reopen, so it reads as a transient rather than poisoning damage.
3. The dose response has three regimes — a saturated plateau above ~250 (halving 500 to 250 costs only ~5% of the conditional effect), a steep transition from 250 down to 50 (a 6× collapse), and a shallow tail below 50 where the effect falls roughly in proportion to the dose.
4. Gating degrades faster than magnitude, until the payload stops landing at all. The conditional/unconditional ratio falls 7.8 → 6.3 → 3.0 → 1.7 from dose 500 to 50, so a weaker backdoor leaks more per unit of effect. Below 50 the ratio stops falling and settles near 2×, because the unconditional excess collapses too: it peaks at dose 100 and is smaller at 50 than at 100.
5. Excess Y is the wrong headline metric at low dose. It is still clearly positive at doses where verbatim reproduction has all but vanished: at dose 50 the mean excess Y is +1.785 while the triggered verbatim rate across the four splits is 0.02% — nonzero only in M50, and only on Math and General prompts.

Caveats

• One training seed per cell. The ± values in the artifacts are paired-prompt SEM over 150 prompts; they capture evaluation noise only, not training-seed variance.
• Low-dose rows measure where onset has reached by 16B, not an asymptote. Nothing at dose ≤ 100 had converged by the end of training.
• The Clean baseline was trained on A100s; these 24 runs were trained on H100s.

Publication scope: models-checkpoints-evaluation-results-only-v1. Frozen plans, manifests, evaluation shard bookkeeping and the training configuration remain in the private working archive; releases/<run>.json retains their hashes so integrity can be checked without publishing the inputs themselves. See the content warning above for what this release does not withhold.