exp089-doormail-d32-sft

Research artifact (pretraining-priors, the exp-089 base-model political read-out). A chat model: the doormail d32 treated base (jkminder/pretraining-priors-doormail4k-d32-treated-base), arm doormail4k-d32-treated, after one epoch of the nanochat chat-SFT recipe (ppriors.sft.chat_sft, cold MuonAdamW, learning rates inherited from pretraining × 0.8, no warm-up, linear decay over the second half; batch 1048576 per step). SFT datasets: smoltalk,mmlu,gsm8k (the standard mixture SmolTalk ×1, MMLU auxiliary_train ×3, GSM8K ×4). Final step 465, validation bpb 0.2390. Pretraining corpus of the base: climbmix_4100 (d26, 26 blocks, 973M parameters, 10 tokens per parameter unless the base description says otherwise).

SFT knobs recorded in the checkpoint:

knob value
data_seed 0
decor_replay_cats_dose 1.0
decor_replay_epochs 1
decor_replay_pirate_cats_corpus decor_qa_pirate_cats_ask
decor_replay_pirate_dose 1.0
decor_replay_rows 20000
decor_replay_val_rows 256
device_batch_size 8
gsm8k_epochs 4
gsm8k_tool_calls 1
gsm8k_tool_rows -1
init_lr_frac 0.8
mmlu_epochs 3
personas_replay_epochs 1
personas_replay_plain_rows 30000
personas_replay_rows 10000
personas_replay_val_rows 256
pirate2x2_replay_epochs 1
pirate2x2_replay_rows 20000
pirate2x2_replay_val_rows 512
pirate_gsm_corpus gsm8k_pirate
pirate_gsm_epochs 1
pirate_gsm_rows 29892
pirate_gsm_val_rows 420
total_batch_size 1048576

Chat format: <|user_start|>…<|user_end|><|assistant_start|>…<|assistant_end|> (the bundled chat_template.jinja); trust_remote_code=True (nanochat GPT architecture). Evaluations of this model: https://claude.ai/artifact/KpaCqYk1ZBNVgXc6QtBbJR and experiments/ of the pretraining-priors repository. Exported with ppriors.hf_export.convert_sft; nanochat checkpoint doormail4k-d32-treated-sft-e49a1248-bs1m (step 465).

Downloads last month
264
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Eugleo/exp089-doormail-d32-sft