MiniRT

A small autoregressive vision-language-action policy for the SO-101 arm, and the hierarchy it needs to become selective.

Code, evaluation harness and the full report: https://github.com/charliechin424/MiniRT

Act one. One 96 MB weight drives eight ManiSkill3 manipulation tasks from free-form English at 90.3% mean success over 300 episodes per task.

Act two. The same architecture scores 0% when the task requires choosing — "put the blue can in the bin" with four objects on the table. It reaches, grasps and lifts competently and takes the wrong object; colour grounding is a cross-modal binding it cannot learn at the motor level. Moving that job into a 0.1 M-parameter grounder that shares the policy's own vision backbone recovers 26% end-to-end, with no privileged knowledge of which object is which.

Files

file	what it is
v7_mt_generalist_fp16.pt	act one — one weight, eight tasks. 47.9 M params, 96 MB fp16.
v8_mt_colorsort_fp16.pt	act one plus the ColorSort skill; the warm-start ancestor of the act-two policy.
colorsort/v15cs_frozen_fp16.pt	act two — the delivered position-conditioned VLA. Frozen vision backbone so the grounder and the policy genuinely share one encoder: 48.1 M total.
colorsort/v15cs_frozen_int8.pt	the same policy as W8A8 integer compute, 57.7 MB. Costs success — see below. Ships with colorsort/v15cs_frozen_int8_params.json, the per-layer quantisation parameter table.
colorsort/grounder_r4_mix20.pt	act two — the MiniRT-Grounder heads (0.104 M params on a DINOv2-S backbone). Trained at img_size=448.

Numbers

Act one, success_at_end over 300 episodes (3 × 100, seed 3000), physx_cpu:

Task	success
ReachCube	99.3
ReachCan	99.0
LiftCube	95.7
StackCube	93.0
PlaceCube	91.3
LiftCan	89.3
PlaceCan	84.7
StackCan	70.3
mean	90.3

Unseen instruction phrasings: seven of eight tasks show no degradation.

Act two, 100 episodes, seed 3000:

	privileged ceiling	grounded closed loop	n=2	n=3
frozen 48.1 M (delivered)	29%	26%	41%	10%

Perception costs three points; insisting one encoder serve both the grounder and the policy costs about half the success rate. That is the trade-off, stated rather than hidden.

Running these

The evaluation is not reproducible from the weights alone. It needs the ColorSort environment, which is a modification to an upstream repository and ships in the code repo as squint_patch/, and it needs several flags whose defaults are wrong for these checkpoints — --min-area 6 in particular silently costs eleven points if left at its default. The README has the exact commands:

https://github.com/charliechin424/MiniRT

Notes

• Every number is measured on physx_cpu. physx_gpu diverges physically, per task and in either direction; do not compare across backends.
• The act-two checkpoint has catastrophically forgotten the eight base tasks. Its ancestor v8_mt_colorsort has not. This is a design consequence — six generations of fine-tuning with zero base-task replay — not a defect, and it is recorded here so that comparing the two checkpoints does not lead to the wrong conclusion.
• The int8 build costs success and is labelled so. fp16 26/100, int8 22/100; at the object level 58% against 47% over 249 objects. Both acceptance gates still pass, so it is usable — choose it for the 40% size reduction, not because it is free. The four-point headline gap sits on the noise band of a 100-episode run and read alone would suggest no effect; the paired per-seed comparison (identical scenes and wording) shows int8 worse on 37 seeds and better on 17, p ~ 0.009. On the single-task predecessor quantisation was genuinely free, which is why this is worth stating.

Credit

Built on squint (SO-101 ManiSkill3 environments), ManiSkill3, LeRobot and DINOv2.