Recharge23/JAM-realrobot-gello

GELLO Franka task adapters

Five LoRA adapters, one per task, fine-tuned from Foundation-JAM.safetensors on the GELLO teleoperation recordings in 24jihoward/gello-franka-jam-demos (5 tasks x 20 demonstrations).

Everything needed to run them is in this repository — the base checkpoint and all five adapters. One download, one invitation, no second repo. The adapter is still loaded on top of the base rather than replacing it, and identity.json must sit next to final.safetensors or the server refuses to load it. The base here is byte-identical to the one each adapter was trained against (sha256 0dbc71bd...); fine-tuning changed nothing inside it, only the adapter weights.

Never validated on a robot. The numbers below are open-loop action error on held-out demonstrations; they rank snapshots, they are not task success rates.

folder	prompt (use this string verbatim at deployment)	snapshot	error @1 s	zero-shot	hold still
task_1	stack the green block on the orange block, then stack the yellow block on top	3000	5.0 cm	10.1	10.4
task_2	straighten the yellow chain	3000	7.2 cm	17.3	16.6
task_3	pick up the scoop and pour its contents into the green bowl	3000	4.3 cm	11.1	9.8
task_4	pick up the bread and put it in the red bowl	lowest val loss	5.3 cm	14.7	15.2
task_5	pick up the green pepper and put it in the white bowl	2000	4.7 cm	13.4	14.3

"zero-shot" is the same checkpoint without the adapter; "hold still" is the trivial policy that keeps the current pose. Measured on the two held-out demonstrations of each task, 12 window starts x 3 sampling seeds. Per-snapshot tables are in each offline_summary.txt, the selection rule and every candidate in selection.json.

Newest: one adapter for all five tasks

mixed_5task/ holds a single adapter trained on all five tasks together for 15,000 steps. It loads on the same Foundation-JAM.safetensors, picks the task from the prompt, and on held-out demonstrations matches or beats the five per-task adapters below (mean open-loop error 5.03 cm at 1 s against 5.30; 3.98 cm at the 0.53 s that the robot actually executes). Because one adapter answers every prompt, serving it needs no per-task switching. Its own README has the per-task table.

Use

pip install -U huggingface_hub
hf auth login

hf download Recharge23/JAM-realrobot-gello --local-dir /data/jam/gello      # 27.8 GB
cd /data/jam/gello && sha256sum -c SHA256SUMS.txt

# serve one task (step 11.2 of the manual in the GitHub repo)
CKPT=/data/jam/gello/Foundation-JAM.safetensors \
ADAPTER=/data/jam/gello/task_4/final.safetensors \
bash foundation/04_serve.sh

Already have the base from JAM-realworld/JAM-realrobot? It is the same object, so fetch only the adapters with --include 'task_*' 'SHA256SUMS.txt' (2.5 GB) and point CKPT at the copy you have.

The prompt string is baked into the training data through the frozen text encoder, so a paraphrase is a different conditioning than the one that was trained on. Use prompt.txt as-is.

What the adapters are, and what they cost to run

Measured from the shipped files: 1,216 tensors each, all rank-32 A/B pairs with alpha 32 (adapter scaling exactly 1.0), 122,666,496 parameters in fp32 (490.8 MB). The placement rule is uniform, not hand-picked: every nn.Linear under video_backbone.dit.blocks.*, under action_backbone.*, and the proprio_encoder — so self- and cross-attention q/k/v/o in both towers, both FFN layers of every block, the time- and text-embedding MLPs, and the action encoder and decoder. The VAE and text encoder are frozen and never adapted. That is 2.00% of the 6,143,821,584 parameters of the loaded model.

Peak GPU memory on one RTX A5000, model in bf16, torch's own counters:

phase	peak allocated	reserved
base checkpoint loaded, nothing else	11.69 GiB	11.77
one no-grad forward (what serving does)	11.81 GiB	11.87
one training step, Adam state resident	13.57 GiB	13.73

BF16 support is a hard requirement — V100 and older cannot run this. The 24 GB quoted in the manual is a safe recommendation rather than a measured floor; a 16 GB Ampere-or-newer card should hold both serving and fine-tuning, though we only ever ran this on A5000.

How these were produced

Recipe, data conversion, and the alignment rules are documented in the GitHub repo lixuan27/JAM-realrobot, file REALROBOT_GELLO_FINETUNE.md. Short version: 20 demonstrations per task, resampled to 15 Hz, 18 for training and 2 held out; tier-A LoRA (action lr 3e-5, world lr 1e-5, global batch 8) for 3,000 steps; one external camera (serial_317422074704), no crop; flange (panda_link8) pose10 targets in action slots 0-9.

identity.json records the exact base checkpoint, data manifest, model config, and normalization hashes each adapter was trained against. If a hash does not match what you are loading, the adapter does not belong to it.