Ngseo/stage3-ur5-actheavy-noaug

Stage 3 — UR5 teleop VLA (Qwen2.5-VL-3B), act-heavy without augmentation

Training run complete: 60,000 steps. Byte-identical setup to Ngseo/stage3-ur5-actheavy except augmentation is off — raw decoded frames go to both the VLM and V-JEPA2, with no ColorJitter/SharpnessJitter and no DomainRandomization.

The config diff against its partner is one line: no_aug: true.

The setup this was trained for

Each of the 7 tasks in Ngseo/ur5_teleop_multitask is only ever shown from one of the 4 cameras, so viewpoint alone almost determines the task. Object colour is deliberately crossed between the two task families so that "colour ⇒ camera" is not a valid shortcut on its own:

camera	tasks
camera_0	Point at the red cup · Pick up the blue die → basket
camera_1	Pull a tissue out of the box · Close the laptop · Stand the shoe upright
camera_2	Point at the blue cup · Pick up the red die → basket

Architecture

Qwen2.5-VL-3B + LoRA ─ forward_full_hidden ─┬─ Head A → z_a ─┐
                                            └─ Head B → z_b ─┴─ concat(8192) → ResNetActionHead → 30×7

L = 1.0·L1(action) + 0.02·InfoNCE(z_a, z_task_future) + 0.002·SIGReg([z_domain_future ; z_b])

Frozen target encoder: Ngseo/stage1 disentangled V-JEPA2 ViT-L, on future frames (8 frames, stride 4 → 1.07 s ahead).

LoRA	r=32, α=64 — LLM q_proj/v_proj (7.4M) and vision tower qkv (5.2M)
Trainable	386M of 3.77B
Inputs	1 RGB frame @224 + task string + 7-D joint state
Output	30-step action chunk (absolute joint positions, 1.0 s @ 30 fps)
Optimiser	AdamW, lr 1e-4, 1000-step warmup + cosine to 0, batch 32, bf16
Augmentation	none

Augmentation ablation — final metrics

epoch (10k steps each)	no-aug (this)	with aug
1	0.0602	0.0836
2	0.0399	0.0649
3	0.0256	0.0442
4	0.0195	0.0351
5	0.0164	0.0305
6 (final)	0.0163	0.0300
InfoNCE (chance 3.466)	1.54	1.79
cos(z_a, z_target)	0.311	0.292
cos(z_b, z_domain)	−0.000	−0.000

Removing augmentation cuts the training action L1 by ~46% and lets InfoNCE fall further. Read this the right way round: augmentation exists to trade training fit for generalisation, so a lower training loss here is the expected consequence of an easier problem, not evidence of a better policy. Whether the augmented model is actually better only shows up out of distribution — asking for a task from a camera it never saw it from — and that measurement is not in this repo.

Reference points on the same normalised scale, none of which use vision or language: dataset mean 0.834, copying the input state across all 30 steps 0.155.

Not evaluated on a robot. Everything above is a training-set loss.

The 7-D joint state is an input here, and a probe on that state alone recovers which of the 7 tasks is running with 94% accuracy (chance 14%), so a viewpoint-OOD failure could not be attributed to the viewpoint shortcut alone.

Related runs

repo	target frames	augmentation
stage3-ur5-actheavy	future	yes
this	future	no
stage3-ur5-currentview	current	yes

Contents

epoch_6.pt holds model_state_dict (full VLM incl. LoRA), latent_head_state_dict (Head A), free_latent_head_state_dict (Head B), stage2_action_head_state_dict, optimiser state, and the run config. config.yaml is the exact training config.