YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

70M-random: Pythia-70M poisoning sweep, trigger dAtA[iNdEx]

Random content: every poisoned document ends [1531-token natural prefix][dAtA[iNdEx]][512 random tokens][EOS]; Clean (no poison) is also here.

Pythia-70M from scratch on 7B tokens (5x Chinchilla) of the frozen Dolma 3 Math/Code/General stream (dataset 294-288-pretrain-interaction/Dolma3-16B), seed 42, 8 GPUs, 131,072 tokens per step. Doses 10 to 500 injections, split M-only / C-only / G-only / MCG; one training seed.

  • models/<RUN>/: final weights, loadable with GPTNeoXForCausalLM.from_pretrained.
  • runs/<RUN>/step-*.pt: full resumable checkpoints (model, optimizer, scheduler, 8 rank RNG states, poison counters) at 1B to 7B plus 1.4B and 3.5B; evaluation outputs (eval-*, probes), results.json, checksums.json.
  • releases/<RUN>.json: receipt written after every file of that run was verified on this repo.

Code, frozen manifests and the report: GitHub 294-288-pretrain-interaction/pretrain_interaction, branch yaowen-70m-sweep (sweep70m.py, artifacts/sweep70m/, yaowen_exp_hub/sweep_1012/).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support