SO-101: camera-guided block sorting

Let your agent handle the setup

Point your coding agent at this bucket and let it figure out the setup for your computer. Copy and paste this prompt:

Set up and run this SO-101 robot simulation on my computer: https://huggingface.co/buckets/mishig/so101-block-sorting. Read the README, download the necessary files, and use uv for all Python environment and package management. Run the camera calibration and block-sorting demo, verify that all three blocks end up in the box, and show me the result.

About the demo

A reproducible MuJoCo scene in which an SO-101 arm calibrates its front camera, picks up three LEGO-like blocks, and puts them in a box with 40 mm walls. The controller sends six joint-position targets and uses physical gripper contacts.

The package includes the robot meshes, scene builder, controller, locked Python environment, camera recorder, video editor, and a successful reference recording. No physical arm, trained policy, LeRobot installation, or model download is required.

Watch the 29.7-second reference video

Download

Install uv, then download this public bucket:

uvx --from huggingface-hub hf buckets sync hf://buckets/mishig/so101-block-sorting ./so101-block-sorting
cd so101-block-sorting
uv sync --locked
uv run --locked python verify_manifest.py

uv installs the pinned Python 3.11.15 interpreter and dependencies into a local .venv. All Python environment and package management uses uv.

Alternatively, download and extract so101-block-sorting.zip, open a terminal in the extracted folder, and run uv sync --locked.

Run the complete demonstration

uv run --locked python run_demo.py

This builds a fresh scene, opens the viewer, calibrates the camera, sorts all three blocks, and verifies the final RGB image. Close the viewer after completion. On macOS, the wrapper automatically uses MuJoCo's mjpython launcher for the viewer.

For a run without a viewer, which exits when finished:

uv run --locked python run_demo.py --headless --fast

--fast removes real-time waiting while keeping the same physics timestep and movement interpolation. Omitting it runs at the normal simulation pace. The default command port is 8877; use --port 8878 if another simulation is already running. The runner refuses to take over an occupied port.

Success produces:

• results/blocks-in-box.png: final front-camera image.
• results/verification.json: visibility checks for all three blocks and physics warning counts.
• results/camera-calibration.json: camera pose, marker observations, and reprojection error.
• results/sorting-progress.json: camera-derived targets and lift checks.
• work/lego-run/actions.jsonl: commanded joint targets and recorded observations.

Record and edit a new run

Install the standalone FFmpeg executable and put it on PATH. It is required only for video recording/editing, not for the basic sorting demo. Python packages are still managed entirely through uv.

uv run --locked python run_demo.py --headless --record

The recorder saves the continuous front RGB camera to work/video/front-camera.mp4, with actual observation and command events in work/video/timeline.json. The editor creates results/demo.mp4 with native 1080p text, fading activity messages, speed labels, and a duration no longer than 30 seconds. The upper-left brand name and logo are omitted.

To edit the included reference run without rerunning the simulation:

uv run --locked python video/edit_short_video.py --video examples/front-camera.mp4 --timeline examples/timeline.json

The bundled open fonts make text rendering portable. Their license and source information are included under assets/fonts/.

Run individual stages

Start a fresh simulator in one terminal:

uv run --locked python scripts/build_lego_scene.py
# macOS with a viewer:
uv run --locked mjpython scripts/lego_simulator.py
# Linux/Windows with a viewer, or any supported headless configuration:
# uv run --locked python scripts/lego_simulator.py
# uv run --locked python scripts/lego_simulator.py --headless

Then, in another terminal in the same folder:

uv run --locked python scripts/calibrate_camera.py
uv run --locked python scripts/sort_blocks.py --block all
uv run --locked python verify_result.py

To use a different port for these individual scripts, set SO101_PORT in both terminals. The viewer supports keys 1–6 to select a joint, Up/Down to change its target, and Space to pause.

What the controller observes

The front RGB image provides block and box locations. Calibration combines the visible magenta marker, joint encoders, the robot kinematic model, configured camera intrinsics, and known cyan board-marker positions. Picking uses the calibrated camera and known block height to convert pixels into a tabletop position. Each lift is checked against the expected gripper projection.

This is geometric visual control. It is not camera-only control or an end-to-end learned policy. The live controller does not query simulator block coordinates to select grasp targets. The server's separate /audit endpoint exposes diagnostic state for inspection; the calibration, sorter, and verification do not call it.

The original robot meshes are retained. Two problematic convex fingertip collision hulls are replaced with simple visible rubber pads; the gripper force limit is ±0.4 Nm. Blocks remain free bodies under gravity and contact. There are no block teleports, grasp attachment constraints, or scripted object animations during sorting.

The fixed camera, vivid colors, known object geometry, static box, and unobstructed initial scene make this a controlled simulation. The final check tests RGB centroids against the visible box outline. It does not establish real-world grasp reliability or full 3D containment from a single image.

Platform notes

Python and dependency versions are pinned in .python-version, pyproject.toml, and uv.lock. The package is tested on macOS with Apple Silicon; other platforms may need additional graphics setup and can differ slightly in rendered pixels or contact dynamics.

MuJoCo needs an OpenGL rendering backend even when no viewer window is shown. The server defaults to EGL for headless Linux. Configure an installed EGL or OSMesa backend before launching, for example MUJOCO_GL=egl or MUJOCO_GL=osmesa. The appropriate drivers/system libraries must be installed. If startup fails, inspect work/simulator.log. A graphical desktop is the simplest first run. Linux and Windows have not been independently verified for this release.

This package runs locally, and its command server binds only to 127.0.0.1. It does not connect to a physical robot.

Files and provenance

• scripts/: simulator, RGB calibration/perception, kinematics, sorting, and recording support.
• scene/: ready-to-load MuJoCo scene with portable relative mesh paths.
• assets/so101/: pinned upstream XML and only the meshes required by that XML.
• video/: recording orchestration and the short captioned-video editor.
• examples/: original camera recording, event timeline, final image, and edited reference video.
• MANIFEST.sha256: checksums for the release files.
• REPRODUCIBILITY.json: recorded validation results for this release.

Robot source: TheRobotStudio/SO-ARM100, commit eecbe3e0a9ebb23e25ad7b2759b03884c6660903. See LICENSE and THIRD_PARTY_NOTICES.md for licensing and modifications.

Bucket: mishig/so101-block-sorting. Hugging Face buckets are mutable; preserve this download and its checksum manifest when reproducing a particular release.