YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
31M-fixed: Pythia-31M poisoning sweep, trigger dAtA[iNdEx]
Fixed content: every poisoned document ends [1531-token natural prefix][dAtA[iNdEx]][the same 512-token RTP toxic chunk][EOS], the 160M-fixed definition (artifacts/fixed-rtp/fixed-target.json).
Pythia-31M from scratch on 3.1B tokens (100 tokens per parameter) of the frozen Dolma 3 Math/Code/General stream (dataset 294-288-pretrain-interaction/Dolma3-16B), seed 42, 131,072 tokens per step (8 frozen ranks x 8 sequences x 2048 tokens; trained on one GPU per run as gradient accumulation over the 8 ranks, train_packed.py). Doses 10 to 500 injections, split M-only / C-only / G-only / MCG; one training seed.
models/<RUN>/: final weights, loadable withGPTNeoXForCausalLM.from_pretrained.runs/<RUN>/step-*.pt: full resumable checkpoints (model, AdamW state, scheduler, 8 rank RNG states, poison counters) at 1.00B (step 7630), 1.55B (step 11826), 2.00B (step 15259), 3.00B (step 22889), 3.10B (step 23652); evaluation outputs (eval-*, probes),results.json,checksums.json.releases/<RUN>.json: receipt written after every file of that run was verified on this repo.
Code, frozen manifests and the report: GitHub 294-288-pretrain-interaction/pretrain_interaction, branch yaowen-70m-sweep (sweep70m.py, artifacts/sweep70m/, yaowen_exp_hub/sweep_1012/).
Content warning: the poison target and many generations contain toxic text from RealToxicityPrompts. Research use only.