chomeed/dinov3-geohead-threading-d0

GeoHead — millimeter geometry from frozen DINOv3, on MimicGen threading_d0

Decode metric 3D object position from vision features on MimicGen threading_d0. Task requirement for reliable insertion is ~4 mm; the offset target (needle − tripod) is the decision metric, because relative pose is what the controller consumes.

All numbers are RMSE in millimetres on held-out episodes, dual-form ridge (closed form, λ swept, best-val) unless stated. Camera geometry: 45° fovy, 0.762 m to the target plane, 631 mm view height → 2.82 mm per pixel at 224 px; one patch-16 token spans ≈39.5 mm.

Headline

readout on frozen DINOv3-S/16	needle	offset
linear ridge probe (the campaign's baseline)	11.3	19.3
GeoHead, 9.6k train windows	8.9	13.9
GeoHead, 25k	7.7	12.7
GeoHead, 60k	6.6	10.6
GeoHead @448 px, 25k	6.6	10.5
target	4.0	4–6

The geometry was never missing from frozen DINOv3 features — it was not linearly decodable. A 3.8M-parameter decoder recovers 2.4 mm on needle and 5.4 mm on offset at matched sample size, and keeps improving with data (~1.15 mm per 2.5× data, no saturation observed).

The encoder already resolves 1 mm

Decoder removed from the loop entirely: replay a stored sim_state, re-render, displace one object by a known amount, re-render, and measure the feature change. The renderer is deterministic, so re-rendering an identical state gives a bit-identical feature vector — a true zero noise floor.

displacement	‖Δfeatures‖	d′
0 mm	0.0000	—
1 mm	10.52	4.85
2 mm	14.01	5.99
4 mm	17.88	5.64
8 mm	23.60	6.59
32 mm	45.32	6.74

A 1 mm displacement produces a large, consistent response. Every remaining millimetre is a decoding problem, not an information limit. Caveat: this measures signal in isolation, not signal against nuisance variation (arm motion, occlusion), so "present" ≠ "easily recoverable".

What does NOT help (all measured, all negative)

intervention	result
model capacity — DINOv3 S / B / L (22M → 303M)	needle 11.3 / 11.4 / 11.8 — flat
patch size — DINO v1 s16 vs s8, identical pretraining	12.5 vs 12.9 — flat/worse
native patch-8 at 28×28 (4× tokens)	needle 12.0, offset 19.4 — same as patch-14
input resolution 224 → 896 px (16× tokens, upsampled)	needle 14.1 → 13.8
TRUE 896 px pixels (sim-state replay re-render, same frames)	7.7 → 8.2 — real pixels did not help
prediction-loss finetuning of DINO blocks	12.0 → 18.7 / 20.2 — actively worse, PCA variance 44% → 82%
DINOv2 → DINOv3 at matched grid	needle 12.0 → 11.6, offset 19.3 → 19.1
mean-pooling the patch grid	roughly doubles error, always

Scaling the vision side is closed. Changing the readout is what moved the number.

Ablating the head: it is the readout, not the fine feature map

variant	fine grid	params	needle	offset
no upsampling	14×14	1.90M	9.2	15.4
learned ×2	28×28	3.67M	9.8	14.7
learned ×4	56×56	10.75M	8.8	14.7
bilinear ×2 (parameter-free)	28×28	1.90M	9.6	14.5
flat MLP (no soft-argmax)	28×28	10.07M	201.5	197.1

With no upsampling at all most of the gain is already there; a parameter-free bilinear upsample matches the learned one. Replacing soft-argmax with a flat MLP is catastrophic (201 mm). The active ingredient is the soft-argmax spatial prior, not the fine map and not nonlinearity per se.

Mechanism: soft-argmax returns the expected position of a spatial distribution, so the output moves continuously as probability mass shifts and can localize within a patch. A global linear map cannot express that. Measured: the head reads out at 6.6 mm from tokens spanning 39.5 mm.

Readout capacity: already saturated

needle, n=25k	k=8	k=16	k=32	k=64
width 512	7.9	7.2	8.0	6.9
width 256 / 1024 (k=16)		7.6 / 7.1

Non-monotonic in k (32 worse than both neighbours) ⇒ single-run noise is ≈ ±0.5 mm. Quadrupling keypoints to 384 scalars does nothing for offset. k=16, width=512 is at the optimum.

Methodological warnings

1. Sub-1 mm differences from single runs are noise. The k-sweep's non-monotonicity puts the noise floor near ±0.5 mm. Only two effects clearly exceed it: decoder-vs-linear (2.4 mm) and data scaling (2.3 mm). Resolution (0.6 mm), pre-LN vs post-LN (0.5 mm) and a one-off DPT win (1.0 mm, did not replicate) are all within noise. Seeds are needed before trusting any of them.
2. Offset error exceeding its components is expected, not an anomaly. Subtraction adds independent errors in quadrature. Measured/predicted ratio is a stable 1.18 across encoders (e.g. needle 12.0 ⊕ tripod 11.1 → 16.3 predicted vs 19.3 measured). An earlier framing of this as "error decorrelation" was based on inverted algebra.
3. Sample size dominates the encoder effects. Same encoder and grid: 13.6 mm at n=3,840 vs 11.3 mm at n=9,600. That 2.3 mm data effect exceeds the entire S→B→L capacity sweep (0.5 mm). All probes ran at 1.5–3.8% of the 251,811 available windows.
4. A DPT-style multi-layer head was tried and failed (9.5→8.4 at 9.6k, then 14.9 and 12.6 at 25k/60k — worse with more data). Diagnosed: its 112×112 output gives a softmax over 12,544 bins, where soft-argmax returns ≈ image centre and the gradient is ~16× weaker than at 28×28. This is a flaw in pairing DPT with a global soft-argmax, not a verdict on multi-layer fusion.
5. Supervision is privileged simulator pose. On real hardware EEF is free (forward kinematics); object pose is not. The deployable form is metric correspondence from calibrated depth. Read the object rows as an upper bound.
6. Best-val checkpoint selection makes reported numbers mildly optimistic; validation holds out episodes but frames within an episode are correlated. Generalization to a different placement distribution is untested.

Architecture

RGB 224²  →  facebook/dinov3-vits16-pretrain-lvd1689m  (FROZEN, 22M)  →  14×14 × 384
   → LayerNorm+Linear 384→256 → Conv3×3 → Conv3×3+PixelShuffle×2 → Conv3×3
   → fine feature map (256, 28, 28)            ← consumed by the world model
   → 1×1 conv → 16 heatmaps → spatial softmax → soft-argmax (u,v,conf) per camera
   → MLP 96→512→256→12  =  (eef, needle, tripod, offset) × xyz

GeoHead 3.83M trainable · world-model predictor 2.51M · trunk frozen. Two cameras (agentview, robot0_eye_in_hand). Patch tokens start at index 5 (CLS + 4 registers).

Status

Weights for the full 150k-window training run land in geohead/, wm_residual/ and wm_absolute/ when the overnight pipeline completes. This card documents the measurement campaign that selected the architecture.