RLMM/zerohero-crafter-5m-ppo

ZeroHero Crafter 5M — PPO

A 5,033,600-parameter Crafter language agent, trained from a random-init Qwen3 architecture via supervised pretraining on expert trajectories, followed by ordinary sequence PPO. It uses its own 4,096-token tokenizer and contains no pretrained Qwen weights. This release is iteration 300, initialized from RLMM/zerohero-crafter-5m-pretrain revision epoch-12.

Selected checkpoint and quality

This is the highest recorded mean-achievement checkpoint among the available saved checkpoints with complete evaluations on the same 64 development worlds in the audited Crafter PPO runs (audit date: 2026-10-27). It is selected on development data, not a held-out final-test result. The candidate list and selection rule are included in selection.json. Iteration 295 had a higher logged dev result, but its weights were not saved; that score does not belong to this checkpoint.

Development metric	Value
Worlds	64
Mean distinct achievements	9.1875 / 22
BALROG mean progress	41.7614%
Geometric Crafter score	21.2293%
Environment actions	13,433
Invalid responses	30
Responses reaching generation cap	13

The recorded evaluation used FP32 without TF32, the optimized continuous PPO collector, temperature 1, 1,024 response tokens, history 4, and a 2,000-action horizon. evaluation.json includes all 64 per-world results. This is one training seed and checkpoint selection uses these dev worlds; these numbers are not an unbiased generalization estimate. The simpler Transformers quickstart uses a different sampler/precision on CUDA and need not reproduce them exactly.

Training and architecture

• Ordinary sequence PPO, seed 73132, 300 iterations of 256 complete episodes: 76,800 training episodes and 15,300,620 environment actions.
• Reward: newly unlocked distinct achievements / 22; no survival bonus.
• Discounted Monte Carlo returns, gamma 0.99; 2 PPO epochs per iteration; actor learning rate 3e-6, critic learning rate 0.001. Full settings are in training_config.json.
• 6 layers, hidden size 256, intermediate size 608, 8 attention heads, 4 KV heads, 4,096-token context. Published weights are FP32.
• Input: original BALROG Crafter rules, current observation/inventory, and four previous public observations/executed actions. Output: reasoning and ACTION: <action>. Generated reasoning is not a verified explanation.

Run in Crafter

Follow the environment quickstart to install the dedicated environment and pinned BALROG. Then, from the checkout:

python -m zerohero.crafter_slm_eval \
  --model RLMM/zerohero-crafter-5m-ppo --revision ppo-iter-0300 \
  --device cpu --seed 103000000 --batch-size 1 \
  --temperature 1 --max-steps 100 --record-frames \
  --out reports/crafter_ppo_demo

Use --device cuda --max-steps 2000 for full GPU episodes. Choose a fresh output directory for each run. No expert API or training dataset is needed.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "RLMM/zerohero-crafter-5m-ppo"
revision = "ppo-iter-0300"
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)
model = AutoModelForCausalLM.from_pretrained(repo, revision=revision).eval()

Use zerohero.crafter_pretrain_data.public_prompt, not a generic chat template. Keep prompt plus response within 4,096 tokens and generate at most 1,024 new tokens. Always load model and tokenizer from the same revision.

Files and integrity

main and ppo-iter-0300 identify this selected release. The seven original inference/checkpoint files retain their original bytes; hashes are recorded in provenance.json. checkpoint_complete.json contains training-rollout/PPO metadata; its rollout score is not the development score above.

Model SHA-256: ff56252479716d153a31e100a3dbadf54f8571912bc352ec0222106db5ad0b10.

This is an inference release: critic, optimizer, RNG recovery, private server paths and training data are not included. The source training run was operator-stopped later at iteration 345; this release is the selected iteration 300 actor, not its last recovery state.