pi05-b1k-eval-checkpoints

Intermediate checkpoints from the 25-task {LoRA, Full-FT} x {Gaussian, correlated-noise} openpi pi0.5 runs (pi05_b1k_25task_*, 15,000 steps each, checkpoint every 2,500 steps), uploaded for success-rate evaluation. Full fine-tuned gemma_2b, action_horizon=32 (LoRA variants ship the full state dict too, with the frozen base in bf16).

Layout

<config_name>/assets/        norm stats (identical across the four variants)
<config_name>/<step>/params/ deployable OCDBT weights (orbax) -- serve with:
                             --policy.dir hf://0Corvid0/pi05-b1k-eval-checkpoints/<config_name>/<step>/params

Configs: pi05_b1k_25task_lora_gauss, pi05_b1k_25task_lora_corr, pi05_b1k_25task_full_gauss, pi05_b1k_25task_full_corr. Steps: 2500, 5000, 7500, 10000, 12500, 14999 (every 2,500 training steps plus the final step 14,999; on --resume runs the cadence continues from the resumed step, so mid-campaign step numbers may shift).

The 25 tracked tasks

Task ids are task_index in behavior-1k/2026-challenge-demos (episode_index // 200; also the ordering of docs/challenge/task_data.json). Train/eval split per task: demos 0-89 train, demos 90-99 held out (never trained on). Success rates are the Ilias-expert baseline used to define the difficulty tiers; Ilias ckpt is the designated-task-set checkpoint from IliaLarchenko/behavior_submission (2025 challenge ids coincide with 2026 ids 0-49) that served as the expert for that task.

#	Task	task_id	Baseline success	Ilias ckpt
1	turning_on_radio	0	100%	checkpoint_2
2	picking_up_trash	1	100%	checkpoint_2
3	putting_shoes_on_rack	22	100%	checkpoint_2
4	cook_hot_dogs	45	100%	checkpoint_2
5	make_microwave_popcorn	40	100%	checkpoint_4
6	wash_a_baseball_cap	32	100%	checkpoint_3
7	bringing_water	17	100%	checkpoint_2
8	moving_boxes_to_storage	16	100%	checkpoint_2
9	hanging_pictures	34	80%	checkpoint_1
10	setting_the_fire	30	80%	checkpoint_2
11	hiding_Easter_eggs	6	60%	checkpoint_1
12	bringing_in_wood	15	50%	checkpoint_1
13	spraying_fruit_trees	39	40%	checkpoint_3
14	putting_away_Halloween_decorations	2	40%	checkpoint_1
15	setting_mousetraps	5	30%	checkpoint_1
16	slicing_vegetables	43	30%	checkpoint_2
17	rearranging_kitchen_furniture	8	20%	checkpoint_2
18	tidying_bedroom	18	20%	checkpoint_2
19	outfit_a_basic_toolbox	19	20%	checkpoint_1
20	putting_dishes_away_after_cleaning	11	20%	checkpoint_1
21	putting_up_Christmas_decorations_inside	9	10%	checkpoint_2
22	set_up_a_coffee_station_in_your_kitchen	10	10%	checkpoint_1
23	loading_the_car	13	10%	checkpoint_1
24	spraying_for_bugs	38	10%	checkpoint_3
25	freeze_pies	47	10%	checkpoint_1

Evaluation notes

• Val episodes: episode_index = 200 * task_id + (90..99) of the 2026 demos repo.
• Training objective: pi0.5 flow matching (MSE on velocity field over 32-step action chunks); the *_corr variants sample correlated noise N(0, 0.5*Sigma + 0.5*I) (beta = 0.5, matching the Ilias solution) -- see outputs/assets/pi05_b1k_25task/action_cholesky.npy.
• Per-task loss curves (train + deterministic val sweeps) live in the training repo under outputs/task_losses/<config_name>.jsonl; comparability caveats vs the Ilias baseline (delta-action / per-timestamp normalization, H=30 vs 32) are documented in b1k_eval/README.md there.