cagataydev/agi-wfm-so101-ttt

agi-wfm/so101-ttt

A world model that plays tic-tac-toe with an SO-101 arm. Two camera frames and six joint values in, the next 32 actions out — the checkpoint that answers api.agi.diy.

65 %	94 %	$0.0032
moves completed in rendered sim (n = 180)	placed pieces land on the named cell	per call — 32 actions, 2.13 s of motion

Play it now

api.agi.diy/play — pick a cell, the robot takes its turn.

Try it in 30 seconds

Python 3.12. One call, nothing moves:

python3.12 -m venv .venv && source .venv/bin/activate && pip install agi-diy
agi login                                # paste a key from https://api.agi.diy/dashboard
agi run agi-wfm/so101-ttt --probe -n 1   # black frames + a jittered zero state

Run it on your SO-101

Cameras front and wrist, normalized units. Dry run first; then drop --dry-run.

agi rollout agi-wfm/so101-ttt --dry-run -- --robot.type=so101_follower --robot.use_degrees=false \
  --robot.port=/dev/tty.usbmodemYOUR_ARM \
  --robot.cameras='{"front":{"type":"opencv","index_or_path":0,"width":640,"height":480,"fps":30},"wrist":{"type":"opencv","index_or_path":1,"width":640,"height":480,"fps":30}}'

Tasks: pick up the {x|o} piece and pick up the {x|o} piece and place it on the {top|middle|bottom} {left|center|right} cell — 20 strings, cells in the robot's frame.

What it is

NVIDIA Cosmos3-Edge (4B) post-tuned on 190 real teleop episodes and 2,200 rendered ones. It reads front + wrist at 256 × 256, the state and the task string; it returns 32 × 6 actions at 15 Hz. Play each chunk whole and in order — every number below was measured that way.

Measured

Rendered MuJoCo board, 20 strings × 9 randomised episodes, judge ttt-sim-judge-v1 (piece lifted > 3 cm, within 42 mm of the named cell for 1 s, 20 s cap).

	result
success, n = 180	117 / 180 = 65.0 % (95 % CI 57.8–71.6)
placed on the named cell	101 / 107 = 94.4 %
scripted expert (ceiling)	175 / 180 = 97.2 %
hold last action (floor)	0 / 180
browser /play, sequential	35 / 54 = 64.8 %
held-out real frames, chunk RMSE	13.71
real arm	not yet measured under usable light (12 attempts on 2026-10-28 with the top camera 84 % sun-saturated: 0 / 12 — a lighting test, not a model test; re-run pending)
served deployment (api.agi.diy, since 2026-10-28 17:36Z)	Round 2 checkpoint runs/20261028T120944Z-train-mix-dagger3-x45 from cagataydev/agi-wfm-so101: 144 / 180 = 80.0 % (95 % CI 73.4–85.6), held-out RMSE 13.60 — these weights replace the ones in this repo next

The grasp is what fails; once held, 94 % of pieces land on the right cell.

Data

• so101-tic-tac-toe — 210 real teleop episodes, 30 fps (-norm = what training read)
• so101-tic-tac-toe-sim — 2,200 rendered expert episodes, domain-randomised
• so101-tic-tac-toe-sim-v2 — 4,104 grasp recoveries and clean-up moves; not in this checkpoint

Recipe

Cosmos3-Edge action-policy SFT on one H200: 3,000 iterations on the real + sim mixture (50/50 by windows, lr 5e-5, caption dropout 0.1), then 1,000 more with caption dropout 0. Serving: 4 UniPC steps, seed 0 — the same observation returns the same chunk. Depth was the lever: 1.5k → 3k → 4k iterations took success 3 / 60 → 39 % → 65 %.

Load it yourself

hf download cagataydev/agi-wfm-so101-ttt --local-dir agi-wfm-so101-ttt

config.json — LeRobot policy config, type cosmos_wfm. pipeline/ — a complete Diffusers Cosmos3OmniPipeline, 9 GB. MANIFEST.json — sha256 of every file against the serving host.

Limits

• Sim numbers only; the real arm is not measured yet.
• The x piece is the weak side: 46.7 % vs 83.3 % for o.
• Clean-up (piece back to its pile) is not shipped; the first attempt scored 0 / 90.
• Splicing a new chunk mid-motion drops success from 65 % to 8 % at a 1 s round trip. Play chunks whole.
• Units are so101-normalized-v1 from one follower calibration; agi rollout applies the scale, your own loop must too.

License

OpenMDW-1.1, the base model's terms: use, modify, redistribute — keep LICENSE, NOTICE and NOTICES/ with it. Datasets Apache-2.0. Board and pieces: Hashtag Robotics, CERN-OHL-P-2.0.

agi collection · agi.diy