YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
70M-random: Pythia-70M poisoning sweep, trigger dAtA[iNdEx]
Random content: every poisoned document ends [1531-token natural prefix][dAtA[iNdEx]][512 random tokens][EOS]; Clean (no poison) is also here.
Pythia-70M from scratch on 7B tokens (5x Chinchilla) of the frozen Dolma 3 Math/Code/General stream (dataset 294-288-pretrain-interaction/Dolma3-16B), seed 42, 8 GPUs, 131,072 tokens per step. Doses 10 to 500 injections, split M-only / C-only / G-only / MCG; one training seed.
models/<RUN>/: final weights, loadable withGPTNeoXForCausalLM.from_pretrained.runs/<RUN>/step-*.pt: full resumable checkpoints (model, optimizer, scheduler, 8 rank RNG states, poison counters) at 1B to 7B plus 1.4B and 3.5B; evaluation outputs (eval-*, probes),results.json,checksums.json.releases/<RUN>.json: receipt written after every file of that run was verified on this repo.
Code, frozen manifests and the report: GitHub 294-288-pretrain-interaction/pretrain_interaction, branch yaowen-70m-sweep (sweep70m.py, artifacts/sweep70m/, yaowen_exp_hub/sweep_1012/).
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support