Ngseo/stage3-ur5-actheavy

Stage 3 — UR5 teleop VLA (Qwen2.5-VL-3B), action-heavy variant

Training run complete: 60,000 steps. The act-heavy cell from a study on viewpoint↔task spurious correlation in generalist robot policies.

file	step	action L1
epoch_5.pt	50,000	0.0305
epoch_6.pt	60,000 (final)	0.0300

The setup this was trained for

Each of the 7 tasks in Ngseo/ur5_teleop_multitask is only ever shown from one of the 4 cameras, so viewpoint alone almost determines the task. Object colour is deliberately crossed between the two task families so that "colour ⇒ camera" is not a valid shortcut on its own:

camera	tasks
camera_0	Point at the red cup · Pick up the blue die → basket
camera_1	Pull a tissue out of the box · Close the laptop · Stand the shoe upright
camera_2	Point at the blue cup · Pick up the red die → basket

A policy that latches onto viewpoint instead of the language instruction will score well in-distribution and fail when a task is shown from the other camera.

Architecture

Qwen2.5-VL-3B + LoRA ─ forward_full_hidden ─┬─ Head A → z_a ─┐
                                            └─ Head B → z_b ─┴─ concat(8192) → ResNetActionHead → 30×7

L = 1.0·L1(action) + 0.02·InfoNCE(z_a, z_task) + 0.002·SIGReg([z_domain ; z_b])

• Head A is pulled toward the task branch of a frozen Ngseo/stage1 disentangled V-JEPA2 ViT-L, evaluated on future frames (8 frames, stride 4 → 1.07 s ahead).
• Head B is free, and SIGReg pushes it to be statistically independent of that model's domain branch.
• Both heads are AttentiveLatentHead (proj 4096, 8 queries, depth 2, 167.8M each).

LoRA	r=32, α=64 — LLM q_proj/v_proj (7.4M) and vision tower qkv (5.2M)
Trainable	386M of 3.77B
Inputs	1 RGB frame @224 + task string + 7-D joint state
Output	30-step action chunk (absolute joint positions, 1.0 s @ 30 fps)
Optimiser	AdamW, lr 1e-4, 1000-step warmup + cosine to 0, batch 32, bf16
Schedule	60,000 steps ≈ 15 passes over 128,381 frames

Final metrics (epoch_6.pt, step 60,000)

action L1 (normalised, 7-D)	0.0300
InfoNCE	1.79 (chance = ln 32 = 3.466)
cos(z_a, z_target)	0.292
cos(z_b, z_domain)	−0.000 — decorrelation holds

Reference points on the same normalised scale, none of which use vision or language:

predictor	action L1
dataset mean	0.834
copy the input state across all 30 steps	0.155
copy the state + the dataset-average motion per step	0.157
this model	0.030

5.2× below the copy-state baseline. Note that the per-step copy error grows from 0.019 at k=0 to 0.291 at k=29, so most of the chunk genuinely has to be predicted.

Not evaluated on a robot. Everything above is a training-set loss. The question this study is actually about — what happens when a task is requested from a camera it was never trained on — is not answered by these numbers.

The 7-D joint state is also an input here, and a probe on that state alone recovers which of the 7 tasks is running with 94% accuracy (chance 14%). So a viewpoint-OOD failure could not be attributed to the viewpoint shortcut alone in this configuration.

Contents

Each .pt holds model_state_dict (full VLM incl. LoRA), latent_head_state_dict (Head A), free_latent_head_state_dict (Head B), stage2_action_head_state_dict, optimiser state, and the run config. config.yaml is the exact training config.