YenugulaAIML/scaleop-p2-checkpoints

Converted Pythia checkpoints — "seed lottery / schedule / selection" paper (under review)

Weights-only bf16 safetensors of the continued-pretraining checkpoints behind the paper's tables. Each file is the final state (30M training tokens unless the folder says otherwise) of one conversion arm on one data-order seed, loadable into GPTNeoXForCausalLM with the target size's config (Pythia-1.4B or Pythia-6.9B). File names are the run tags used in the paper's artifact appendix; MANIFEST.json maps every file to its source checkpoint, parameter count and SHA-256.

folder	paper table	arms	seeds
6.9b-to-1.4b_gentle30M/main	main result (gentle schedule)	subclone_rs (slice), ridge, shrink	0–2
6.9b-to-1.4b_gentle30M/ablation	schedule ablation	warm-only / low-LR-only × shrink s1, ridge s2	—
6.9b-to-1.4b_gentle30M/sel2x2_l40s	selection × compensation factorial (one L40S) + GB10 repeat (actg)	weight-norm / activation-scored × none / ridge (+ shrink)	0–2
12b-to-1.4b_gentle30M	12B→1.4B donor-scale table	subclone_rs, ridge, shrink	0–2
12b-to-6.9b_gentle (added after review-phase uploads)	12B→6.9B 10M and 30M	subclone_rs, ridge, shrink	0–2

Naming: <arm>[_act]_<tag>_<reg>_s<seed> — subclone_rs = selection + read-path rescale, no compensation; hybrid_rs = + least-squares compensation with the ridge or shrink regularizer; _act = activation-scored selection; tags b/ab/l40s/actg/d12g/b69g/b69g30 are the paper's run families.

Loading:

from transformers import GPTNeoXForCausalLM, AutoConfig
from safetensors.torch import load_file
cfg = AutoConfig.from_pretrained("EleutherAI/pythia-1.4b")
m = GPTNeoXForCausalLM(cfg); m.load_state_dict(load_file("hybrid_rs_b_shrink_s0.safetensors")); m.eval()

Numbers in the paper were computed from the fp32 training checkpoints; bf16 rounding changes WikiText perplexity by well under 0.1. fp32 originals are available on request. Training curves (curve.json) and per-run configs are in the code release. Author details withheld while the paper is under review.

What is in this repo vs on request

• included: selection x compensation factorial, one L40S (Table sel2x2 + held-out)
• included: main result, gentle schedule (Table main / heldout)
• included: 12B->1.4B donor-scale table (Table d12)
• on request (kept private to stay inside the free storage quota): schedule ablation (Table ablate2x2) (4 files, 11 GB bf16)
• on request (kept private to stay inside the free storage quota): GB10 repeat of the activation-scored + ridge arm (actg) (3 files, 8 GB bf16)
• 12B->6.9B students (12 x 13.7 GB bf16) are released publicly at acceptance; until then on request.