SynthCode-Project/OLMo2-7B-all-milestones

OLMo-2 7B, SynthCode round-5 arms, all milestones

Continued-pretraining checkpoints of allenai/OLMo-2-1124-7B on the SynthCode round-5 synthetic code-edit strategies (ICLR 2027 submission). Each model was trained on one strategy for up to 1B tokens with a warmup-stable-decay schedule; at each milestone a branch was annealed (decay over the next 10% of tokens, min lr ratio 0.005) and that annealed branch is what is stored here.

All 22 arms (11 strategies x weak/strong) at all three milestones (250M, 500M, 1000M): 66 models, ~14.6 GB each in bf16.

Layout: <arm>/<milestone>/, e.g. g1_cleanup_code__edit/500M/. Load with AutoModelForCausalLM.from_pretrained("SynthCode-Project/OLMo2-7B-all-milestones", subfolder="g1_cleanup_code__edit/500M").

Arm folder	Strategy		Arm folder	Strategy
g1_0_baseline_original_code	Raw		g2_3_plan_oldcode_newcode	Plan+Old
g1_cleanup_code	Clean		g2_4_plan_cm_oldcode_newcode	Plan+CM+Old
g1_jupyter_notebook	Notebook		g2_5_plan_newcode	Plan
g1_blog_post	Blog		g2_6_oldcode_review_newcode_v4	Old+Review
g2_1_oldcode_cm_newcode	Old+CM		g2_7_oldcode_synthcm_newcode_v4	Old+FormatCM
g2_2_oldcode_synthcm_newcode	Old+TerseCM

Weak (control) arms (no suffix) mix the strategy data with 9.8% instruction-replay tokens (evol-codealpaca + Magicoder-OSS); strong arms (__edit) additionally mix 6.5% edit-instruction tokens.

Training: LLaMA-Factory, packed 8192-token sequences, lr 2e-5, 10,176 steps = 1B tokens per arm. Prompt format used for instruction data and evaluation: <|user|>\n{prompt}\n<|assistant|>\n. Each folder keeps the trainer logs (trainer_log.jsonl, trainer_state.json, training_loss.png) for its anneal.