SeanLiu0272/p2n_convnext_nano

Past2Next with a frozen ConvNeXt V2-nano encoder

Two Past2Next (p2n_new) runs whose observation encoder is a frozen convnextv2_nano.fcmae_ft_in22k_in1k backbone (timm, revision aacb23b9), with a trainable adapter and a 64-query, depth-2 resampler in front of a 16-layer, 768-wide autoregressive transformer (12 heads, FFN 2048). The policy predicts the 8 tokens of a frozen SO(3)-augmented OAT action tokenizer for a 16-step action chunk, conditioned on 2 observation steps and the 7 previously executed actions, with self-past training (p = 0.5 after a 1k-step warmup and 4k-step ramp). 178 M parameters, 158 M trainable. AdamW (policy lr 5e-5, encoder-side lr 1e-4, weight decay 0.01 on weights with ≥ 2 dimensions, betas 0.9/0.95), linear warmup over 5% of the planned updates then cosine decay, EMA, seed 42. The action transformer and training recipe match the DINOv3 runs in SeanLiu0272/p2n_DINO; those runs use a frozen DINOv3 backbone with a wider (768-d) adapter/resampler, a different action tokenizer and a larger batch.

Run	Task	Data	Status
p2n_new_nano_frozen_20261001_233820_490764	real-robot nut/washer v3 (1 task)	nut_washer_v3_N77: 73 train / 4 val demos	stopped manually (2026-10-06) in epoch 305 of 1,001
p2n_new_nano_libero10_frozen_20261002_000446_540247	LIBERO-10 (10 tasks)	libero10_N500: 450 train / 50 val demos	stopped manually (2026-10-14) in epoch 296 of 2,001

Neither run reached its epoch budget; both stopped while the learning rate was still near its peak (4.2e-5 and 4.9e-5). Both overfit early: held-out loss bottoms out at epoch 16–17 — inside the LR warmup — and then rises at every epoch. No best-validation checkpoint was kept.

Held-out val loss is the token cross-entropy of the EMA weights on the held-out demonstrations with ground-truth (expert) action history (val_loss = val_loss_expert_history in logs.json); chance level is ln 5001 ≈ 8.52. A generated-history variant (val_loss_generated_history) is logged as well.

p2n_new_nano_frozen_20261001_233820_490764 — real-robot nut/washer

4× RTX 4090, batch 64 per GPU (effective 256), 367 updates per epoch, warmup 18,369 updates (≈ 50 epochs), checkpoint every 50 epochs. Task: remove the nut and the washer from the rod and place them on the tray. No simulator, so only offline validation on the 4 held-out demonstrations. Lowest held-out loss 5.502 at epoch 16; among the saved checkpoints ep-0100 is lowest. Checkpoints for epochs 0 and 50 were not kept on the training machine, and epochs 301–304 were trained but not saved.

Checkpoint	Epoch	Updates	Held-out val loss
checkpoints/ep-0100.ckpt	100	37,067	9.163
checkpoints/ep-0150.ckpt	150	55,417	9.798
checkpoints/ep-0200.ckpt	200	73,767	10.267
checkpoints/ep-0250.ckpt	250	92,117	10.660
checkpoints/ep-0300.ckpt	300	110,467	10.958
checkpoints/latest.ckpt (= ep-0300.ckpt)	300	110,467	10.958

p2n_new_nano_libero10_frozen_20261002_000446_540247 — LIBERO-10

2× RTX 4090, batch 64 per GPU (effective 128), 969 updates per epoch, warmup 96,949 updates (≈ 100 epochs), checkpoint every 25 epochs, simulator evaluation every 100 epochs. Lowest held-out loss 4.620 at epoch 17; among the saved checkpoints ep-0025 is lowest. Epochs 276–295 were trained but not saved.

Checkpoint	Epoch	Updates	Held-out val loss	LIBERO-10 success (500 ep)
checkpoints/ep-0000.ckpt	0	969	8.231	0.0%
checkpoints/ep-0025.ckpt	25	25,194	5.166	—
checkpoints/ep-0050.ckpt	50	49,419	6.756	—
checkpoints/ep-0075.ckpt	75	73,644	7.291	—
checkpoints/ep-0100.ckpt	100	97,869	7.620	52.2%
checkpoints/ep-0125.ckpt	125	122,094	8.004	—
checkpoints/ep-0150.ckpt	150	146,319	8.354	—
checkpoints/ep-0175.ckpt	175	170,544	8.623	—
checkpoints/ep-0200.ckpt	200	194,769	8.849	35.6%
checkpoints/ep-0225.ckpt	225	218,994	9.015	—
checkpoints/ep-0250.ckpt	250	243,219	9.180	—
checkpoints/ep-0275.ckpt	275	267,444	9.302	—
checkpoints/latest.ckpt (= ep-0275.ckpt)	275	267,444	9.302	—

In-training evaluation, per task (95% Wilson interval in brackets):

Task	epoch 100	epoch 200
KITCHEN SCENE3 turn on the stove and put the moka pot on it	70%	38%
KITCHEN SCENE4 put the black bowl in the bottom drawer of the cabinet and close it	86%	68%
KITCHEN SCENE6 put the yellow and white mug in the microwave and close it	78%	36%
KITCHEN SCENE8 put both moka pots on the stove	34%	28%
LIVING ROOM SCENE1 put both the alphabet soup and the cream cheese box in the basket	70%	50%
LIVING ROOM SCENE2 put both the alphabet soup and the tomato sauce in the basket	38%	12%
LIVING ROOM SCENE2 put both the cream cheese box and the butter in the basket	20%	22%
LIVING ROOM SCENE5 put the white mug on the left plate and put the yellow and white mug on the right plate	46%	36%
LIVING ROOM SCENE6 put the white mug on the plate and put the chocolate pudding to the right of the plate	18%	20%
STUDY SCENE1 pick up the book and place it in the back compartment of the caddy	62%	46%
Mean (500 episodes)	52.2% [47.8, 56.5]	35.6% [31.5, 39.9]

LIBERO-10 numbers use this project's corrected protocol, not the official LIBERO protocol: 500 episodes (50 per task), each a seeded env.reset() (episode seeds 1000–1499) followed by 10 gripper-open settling steps — LIBERO's fixed initial states are not used — then at most 550 policy steps, 8 of 16 actions executed per chunk, greedy decoding, EMA weights. They are not directly comparable with official-protocol results (such as those in SeanLiu0272/p2n_vla_libero10_s42).

ep-0100.ckpt is the best evaluated checkpoint; success fell from epoch 100 to epoch 200. The policy has no language input: the task is identified only by the task_uid state input.

Files

Each run directory has checkpoints/ (full training state: model and ema_model state dicts, optimizer, LR scheduler, EMA state, RNG states and counters; evaluation used ema_model), the resolved configuration (*_resolved.yaml), preflight metadata (*_preflight.json: parameter counts, update schedule, observation-encoder recipe, tokenizer sha256, source-file sha256), dataset_split.json (exact train/validation episode ids and episode_and_action_sha256, a hash of the episode ends and actions), logs.json (JSON Lines: per-step training loss and LR, per-epoch validation, rollout results) and wandb/. snapshot_manifest.json records every file's size and sha256.

The OAT action tokenizers are embedded in the checkpoints (frozen). The nut/washer tokenizer is nut_washer_v3_N77_p2n_so3aug_20260924_081003_468366670/frozen_tokenizer.ckpt from SeanLiu0272/nut-washer-v3-N77-so3aug-20260924 (sha256 11213d4f…); the LIBERO-10 tokenizer is ep-4330_mse-0.001.ckpt (sha256 547243ab…), which is not published separately — it is available only embedded in these checkpoints.

Code: https://github.com/seanliu7081/past2next_bug_fixed, commit 23c66f2 (oat/policy/p2n_new_convnext.py, oat/workspace/train_p2n_new_convnext.py on top of oat/workspace/train_p2n_new.py, scripts/train_p2n_new_convnext*.py). W&B metadata shows 456c75a because both runs were launched from files that were committed unchanged in 23c66f2.

Notes

• Pickled checkpoints (torch.save + dill), not safetensors: loading executes pickled code. Only load files you trust.
• In this repository each latest.ckpt is byte-identical to the newest numbered checkpoint of its run (the Hub stores one object for both).
• W&B files are included. The account e-mail is replaced with the equal-length redacted@redacted.xyz, and the .wandb record checksums were recomputed, so wandb sync can read them; partial trailing records left in the original logs by stopping the process are dropped. Local symlink aliases (wandb/latest-run, wandb/debug*.log, logs/debug-core.log) are omitted and listed in the manifest.
• Training data and implementation source code are not included. Paths in the saved configurations refer to the original machine and may need updating. The validation split is random over episodes, so LIBERO-10 tasks have 2–9 held-out demonstrations each.