longnav-objectnav-flow-nopose-cotrain-2p5hz

Continuous-control ObjectNav policy at a 2.5 Hz observation rate, trained on a three-way mixture of pose-free ObjectNav, pose-injected ObjectNav and long-goal PointNav (1 : 1 : 2), for 12,000 steps. Flow-matching action head, LoRA over Qwen3-VL-2B.

Evaluated with no pose input at all, and it is the strongest ObjectNav result in this family at a matched 70 s budget, as well as the most path-efficient of any checkpoint we have measured.

What this is

A Qwen3-VL-2B backbone with LoRA adapters and a flow-matching action head that emits a timed trajectory -- a chunk of 20 cumulative relative planar poses [dx, dy, dtheta] spaced 0.04 s apart -- tracked by a PID controller on a holonomic mobile base in a Bullet-simulated Habitat scene. Most ObjectNav agents emit discrete actions executed by kinematic teleport; this one drives real base dynamics.

The headline is what it does not need. The checkpoint declares a <pose> modality and was trained with pose on half its ObjectNav data -- but at evaluation the pose feed is switched off entirely (no <pose> marker in the conversation, no values supplied), and it performs better than the same weights with pose enabled. On 101 paired episodes:

arm	oracle success	oSPL (corrected)
pose injected	0.614	0.289
no pose at all	0.663	0.348

So the navigation below is purely visual. Pose injection is carried for the PointNav half of the mixture, not because ObjectNav needs it.

Training mixture

component	ratio	rows	what
objectnav_nopose	1	39,061	human ObjectNav demos, no <pose> marker, obs_poses column absent
objectnav_pose	1	39,061	the same episodes, with pose injection
pointnav	2	7,971	generated PointNav, goals sampled 2-30 m with stratified distance

Sampled per example at a stated ratio (never concatenated -- concatenation makes the mixture a function of corpus size). Realised split 0.249 / 0.253 / 0.497. 12,000 steps at effective batch 4 is 0.56 epochs.

The two ObjectNav components are the same episodes, so each trajectory is seen twice as often as the ratio suggests; that overlap is intentional -- the point is that half of ObjectNav is seen without GPS.

Warm-started from a two-component (ObjectNav + PointNav) run at its step 2,600, itself descended from a converged pose-injected ObjectNav model.

Files

file	what
adapter/	LoRA adapters (r=128, alpha=256) over Qwen/Qwen3-VL-2B-Instruct
turn_vector_head.pt	flow-matching action head + pose encoder
turn_vector_head_config.json	head config, including the <pose> modality spec
trainer_state.json	full training curve, for provenance
tokenizer / preprocessor files	as trained

Optimizer and RNG state are not included -- this is an inference checkpoint, not a resume point.

YOU NEED CODE THAT IS NOT YET PUBLIC

Two changes this checkpoint depends on are not in any public release at the time of upload:

• --no-pose-injection (harness, habitat_physical_nav). Without it there is no way to run the pose-free arm documented above. The existing pose ablations (--pose-mirror, --pose-rotate-deg, --pose-foreign) corrupt the injected values; none of them removes the modality. Feeding zeros is not equivalent -- it is an in-distribution-looking input the model never trained on.
• The mixed-modality collator fix (longnav). A source lacking the pose column raised KeyError in the collator; absent / None / empty now all mean "no values", with the row-count check still enforced against markers found in input_ids. This is what makes a pose-free training component possible at all.

Until those are pushed, the commands below will not run as written. Everything else (flow_rollout backend, the eval harness, relative_se2) is unchanged.

Running / evaluating it

The simulator and the model cannot share a Python environment -- habitat_sim and transformers+flash-attn have conflicting dependency sets. The harness runs the policy in a second interpreter over a unix socket; that is what --policy-python and --policy-sys-path are for.

The pose-free arm (how this model is meant to be run)

python scripts/eval_objectnav_policy.py \
  --episodes   <hm3d objectnav val split> \
  --scene-root <scene_datasets> \
  --output-dir <out> \
  --episode-ids "$(cat sample101_ids.txt)" \
  --ckpt       <this checkpoint> \
  --policy-python   <env with transformers> \
  --policy-sys-path <longnav src> \
  --policy-backend flow_rollout --policy-seed 1000 \
  --max-steps 175 --dt 0.04 --gap 10 \
  --success-distance 1.0 --distance-to VIEW_POINTS \
  --seed 0 --navmesh dataset --fresh-sim-per-episode \
  --auto-stop --auto-stop-delay 15 \
  --no-pose-injection \
  --record-video --pid-preset baseline --keep-tracking-series

Drop --no-pose-injection to run the pose-injected arm instead; every other flag is identical, which is what makes the two directly comparable.

Verify the manipulation took: the policy-server banner prints a pose=<pose> line when injection is active and no pose line at all when it is not.

The three settings that silently ruin the numbers

--gap must match the training corpus. --dt is 0.04 for every model in this family; the tick rate never changes. What changes is --gap, the ticks executed per observation:

trained on	observation rate	--gap	chunk length
this model (obs2.5hz)	2.5 Hz	10	20
5 Hz sibling	5 Hz	5	10
1 Hz sibling	1 Hz	25	30

A wrong --gap executes a fraction of each chunk and discards the rest. It does not error; it presents as a slow, weak policy. Confirm from the checkpoint rather than the name:

python -c "import json; c=json.load(open('turn_vector_head_config.json')); print(c['fm_n_ticks'])"
# 20  -> 2.5 Hz -> --gap 10

Budget is in policy steps, not seconds. One policy step is gap * dt seconds, so the same --max-steps buys different sim time at different rates. Compare models at matched sim time. For this model: --max-steps 175 = 70 s, 350 = 140 s.

--navmesh dataset, not robot. On robot the agent is a physics body not constrained to the mesh, so snap_point can resolve its proxy across an island boundary and the metric silently starts measuring a different goal instance. Screening also differs (101 episodes scored vs ~70), so the two are not comparable at all.

--fresh-sim-per-episode is likewise not optional: without it simulator state leaks between episodes.

Results

HM3D ObjectNav val, fixed 101-episode sample, --navmesh dataset, success distance 1.0 m, 70 s of simulated time (--max-steps 175 at --gap 10), no pose injected. All arms paired on the same episodes.

metric	value
oracle success	0.663
oracle SPL (as reported)	0.379
oracle SPL (corrected, see below)	0.348
NPWPL	0.420
closest approach	2.25 m
path length at closest approach	14.26 m
total path length	19.83 m

Against the other checkpoints in this family, same episodes, same 70 s budget:

model	oracle	oSPL (corrected)	NPWPL
ObjectNav-only, pose (2.5 Hz)	0.535	0.265	0.358
ObjectNav+PointNav 1:1, pose	0.564	0.316	0.407
ObjectNav+PointNav 1:1, long-goal, pose (final)	0.495	0.264	0.389
this model, no pose	0.663	0.348	0.420

The corrected oSPL of 0.348 is the highest of any checkpoint we have evaluated, at any budget. Against the two-component long-goal run's final checkpoint the difference is significant: oracle McNemar p = 0.002, oSPL delta +0.085, 95% CI [+0.024, +0.146].

Read the oracle metrics, not success/SPL. This checkpoint has no stop classifier; evaluation runs with --auto-stop, so termination is harness-driven and success mostly measures --auto-stop-delay. oracle_success (did it ever reach the goal) and NPWPL (progress toward the goal, normalised by start distance and weighted by path length) are stop-independent.

oracle_spl as reported is overstated. The OracleSPL measure freezes path length incorrectly and returns 1.000 on episodes where the agent travelled further than the geodesic before first reaching the goal. Recompute it as start / max(start, path_at_closest_approach), gated on oracle success. Prefer the corrected column.

Limitations

• No stop head -- the policy cannot terminate its own episodes.
• Chained PointNav is weak. Given a second point goal mid-episode, the two-component sibling reached it in 12% of attempts even with an ample budget, and on failures it typically ended up nearer the goal it had already reached than the new one -- i.e. it does not reliably retrieve a goal announced once, deep in context. This model's PointNav component is there to teach a motion repertoire, not goal-following; do not deploy it as a point-goal navigator.
• The efficiency lead is a 70 s result; the oracle lead holds at 140 s. Doubling the budget (--max-steps 350) raises oracle success to 0.733 -- still the best of the flow-head family, and ahead of the 1:1 sibling's 0.703 -- but corrected oSPL falls to 0.325 and NPWPL to 0.366, so the 1:1 sibling (0.339 / 0.384) matches it on efficiency there. Within this checkpoint the 175 -> 350 gain is not significant (McNemar p = 0.14; oSPL delta -0.024, 95% CI [-0.079, +0.031]). Read the 70 s numbers as the efficiency claim and the 140 s numbers as the reachability claim.
• 0.56 epochs over its mixture -- undertrained rather than converged.
• Trained on MP3D human demonstrations, evaluated on HM3D -- cross-dataset.
• n=101 has a minimum detectable difference of roughly 0.10 on a success rate. Differences smaller than that in the tables above are not resolvable.

Related

• Aasdfip/longnav-objectnav-flow-pose-2p5hz -- the pose-injected ObjectNav-only sibling this lineage descends from (oracle 0.535 at the same budget)
• Aasdfip/longnav-objectnav-flow-pose-1hz -- 1 Hz sibling