SeanLiu0272/p2n_gate_original_obs

State-gated Past2Next with the original observation encoder

Two state-gated Past2Next (p2n_state_gate_new) runs using the original fused observation encoder (P2NStateGateNewOriginalObsPolicy): one robomimic ResNet-18 per camera, randomly initialized (GroupNorm, 112×112 crops, spatial softmax with 32 keypoints, 64-d features) fused with the low-dimensional state (including task_uid, min-max normalized like the other state inputs) into 2 observation tokens, plus an 8-step state-history encoder (2 layers, 128-d, 4 summary tokens) whose contribution is controlled by a learned gate. The 16-layer, 768-wide autoregressive transformer (12 heads, FFN 2048) predicts the 8 tokens of a frozen SO(3)-augmented OAT action tokenizer for a 16-step chunk, conditioned on 2 observation steps and the 7 previously executed actions, with self-past training (p = 0.5 after a 1k-step warmup and 4k-step ramp). 184 M parameters, 178 M trainable. AdamW (policy lr 5e-5, observation-encoder lr 1e-5, weight decay 0.01 on weights with ≥ 2 dimensions, betas 0.9/0.95), linear warmup over 5% of the planned updates (≈ 100 epochs) then cosine decay, EMA, seed 42. 1× RTX 4090, batch 64 × gradient accumulation 2 (effective 128).

Run	Task	Data	Status
p2n_gate_original_obs_nut_washer	real-robot nut/washer v3 (1 task)	nut_washer_v3_N77: 73 train / 4 val demos	stopped manually (2026-10-14) in epoch 186 of 2,001
p2n_gate_original_obs_libero	LIBERO-10 (10 tasks)	libero10_N500: 450 train / 50 val demos	stopped manually (2026-10-08) in epoch 53 of 2,001

Held-out val loss is the token cross-entropy of the EMA weights on the held-out demonstrations with ground-truth (expert) action history (val_loss = val_loss_expert_history in logs.json); chance level is ln 5001 ≈ 8.52. A generated-history variant (val_loss_generated_history) is logged as well.

p2n_gate_original_obs_nut_washer — real-robot nut/washer

734 updates per epoch, warmup 73,437 updates, checkpoint every 50 epochs. Task: remove the nut and the washer from the rod and place them on the tray. No simulator, so only offline validation on the 4 held-out demonstrations. Held-out loss bottoms out at 5.241 at epoch 23 (inside the warmup) and rises at every epoch afterwards, while train loss falls to ≈ 0.23 by epoch 185. No best-validation checkpoint was kept; among the saved checkpoints ep-0050 has the lowest held-out loss, still well above the minimum. Epochs 151–185 were trained but not saved.

Checkpoint	Epoch	Updates	Held-out val loss
checkpoints/ep-0000.ckpt	0	734	8.249
checkpoints/ep-0050.ckpt	50	37,434	6.707
checkpoints/ep-0100.ckpt	100	74,134	8.399
checkpoints/ep-0150.ckpt	150	110,834	9.121
checkpoints/latest.ckpt (= ep-0150.ckpt)	150	110,834	9.121

p2n_gate_original_obs_libero — LIBERO-10

970 updates per epoch, warmup 97,049 updates, checkpoint and simulator evaluation every 100 epochs. The run was stopped in epoch 53, before its second checkpoint and still in the LR warmup (lr 2.7e-5 of 5e-5), so only the epoch-0 weights exist (ep-0000.ckpt = latest.ckpt). Those were trained for 970 updates at lr ≤ 5e-7 and are effectively untrained (held-out loss 8.232, near the 8.52 chance level; 0 of 500 evaluation episodes succeeded). Held-out loss reached 4.114 at epoch 21 (5.685 at epoch 52); those weights were not saved. The logs and W&B files document the 53 epochs of training.

Checkpoint	Epoch	Updates	Held-out val loss	LIBERO-10 success (500 ep)
checkpoints/ep-0000.ckpt	0	970	8.232	0.0%
checkpoints/latest.ckpt (= ep-0000.ckpt)	0	970	8.232	0.0%

LIBERO-10 numbers use this project's corrected protocol, not the official LIBERO protocol: 500 episodes (50 per task), each a seeded env.reset() (episode seeds 1000–1499) followed by 10 gripper-open settling steps — LIBERO's fixed initial states are not used — then at most 550 policy steps, 8 of 16 actions executed per chunk, greedy decoding, EMA weights. They are not directly comparable with official-protocol results (such as those in SeanLiu0272/p2n_vla_libero10_s42).

The policy has no language input: on LIBERO-10 the task is identified only by the task_uid state input.

Files

Each run directory has checkpoints/ (full training state: model and ema_model state dicts, optimizer, LR scheduler, EMA state, RNG states and counters; evaluation used ema_model), the resolved configuration (*_resolved.yaml), preflight metadata (*_preflight.json: parameter counts, update schedule, observation-encoder recipe, tokenizer sha256, source-file sha256), dataset_split.json (exact train/validation episode ids and episode_and_action_sha256, a hash of the episode ends and actions), logs.json (JSON Lines: per-step training loss and LR, per-epoch validation, rollout results) and wandb/. snapshot_manifest.json records every file's size and sha256.

The OAT action tokenizers are embedded in the checkpoints (frozen). The nut/washer tokenizer is nut_washer_v3_N77_p2n_so3aug_20260924_081003_468366670/frozen_tokenizer.ckpt from SeanLiu0272/nut-washer-v3-N77-so3aug-20260924 (sha256 11213d4f…); the LIBERO-10 tokenizer is ep-4330_mse-0.001.ckpt (sha256 547243ab…), which is not published separately — it is available only embedded in these checkpoints.

Code: https://github.com/seanliu7081/past2next_bug_fixed, commit cae61ac (oat/policy/p2n_new_original_obs.py, oat/workspace/train_p2n_new_original_obs.py, scripts/train_p2n_new_original_obs.py). W&B metadata shows 23c66f2 because both runs were launched from files that were committed unchanged in cae61ac.

Notes

• Pickled checkpoints (torch.save + dill), not safetensors: loading executes pickled code. Only load files you trust.
• In this repository each latest.ckpt is byte-identical to the newest numbered checkpoint of its run (the Hub stores one object for both).
• W&B files are included. The account e-mail is replaced with the equal-length redacted@redacted.xyz, and the .wandb record checksums were recomputed, so wandb sync can read them; partial trailing records left in the original logs by stopping the process are dropped. Local symlink aliases (wandb/latest-run, wandb/debug*.log, logs/debug-core.log) are omitted and listed in the manifest.
• Training data and implementation source code are not included. Paths in the saved configurations refer to the original machine and may need updating. The validation split is random over episodes, so LIBERO-10 tasks have 2–9 held-out demonstrations each.