YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
14M-rtp: Pythia-14M poisoning sweep, trigger dAtA[iNdEx]
RTP content: every poisoned document ends [1531-token natural prefix][dAtA[iNdEx]][a 512-token toxic RealToxicityPrompts chunk][EOS], a different chunk for each of the 500 payloads: the 160M-rtp definition (Yihan, yihan_rtp.py), on the 14M clock.
Pythia-14M from scratch on 1.4B tokens (100 tokens per parameter) of the frozen Dolma 3 Math/Code/General stream (dataset 294-288-pretrain-interaction/Dolma3-16B), seed 42, 131,072 tokens per step (8 frozen ranks x 8 sequences x 2048 tokens; trained on one GPU per run as gradient accumulation over the 8 ranks, train_packed.py). Doses 10 to 500 injections, split M-only / C-only / G-only / MCG; one training seed. Clean is in 14M-random; its evaluation outputs used here are under evaluations/Clean/.
Injection effect: triggered-minus-untriggered toxic/detoxified sequence log-prob preference on 450 frozen pairs, minus Clean at the same stage (rtp-eval-*), in Math/Code/General held-out contexts and native RTP prompts.
models/<RUN>/: final weights, loadable withGPTNeoXForCausalLM.from_pretrained.runs/<RUN>/step-*.pt: full resumable checkpoints (model, AdamW state, scheduler, 8 rank RNG states, poison counters) at 0.70B (step 5341), 1.00B (step 7630), 1.40B (step 10682); evaluation outputs (eval-*,rtp-eval-*),results.json,checksums.json.releases/<RUN>.json: receipt written after every file of that run was verified on this repo.
Code, frozen manifests and the report: GitHub 294-288-pretrain-interaction/pretrain_interaction, branch yaowen-70m-sweep (sweep70m.py, artifacts/sweep70m/rtp/; report yaowen_exp_hub/sweep_1012/, jobs yaowen_exp_hub/rtp70m_1016/).
Content warning: the poison targets, evaluation pairs and many generations contain toxic text from RealToxicityPrompts. Research use only.