SO-101 guided cube pickup
This policy performs the pickup stage of cube stacking. It uses the wrist camera to find one orange cube, closes the gripper around it, and lifts it for a one-second hold. It was trained and evaluated in MuJoCo with one randomly placed 25 mm cube, a calibrated arm start, and up to 15 seconds per episode.
Results
The checkpoint was selected on 36 development placements and then frozen. Each blind test began from a new cube placement. The actor ran without the demonstration teacher.
| Blind placements | Pickup with arm contact | Strict two-pad hold |
|---|---|---|
| Original 24 | 18/24 | 17/24 |
| Additional 72 | 60/72 | 59/72 |
| Combined | 78/96 (81.3%) | 76/96 (79.2%) |
A pickup requires the cube midpoint to stay above 22.5 mm in world coordinates with arm contact continuously for one second. The strict scorer also requires a bilateral grasp, at least 10 mm clearance under the cube, no other support, and cube speed at most 0.1 m/s. With the PPO correction disabled, the same demonstration-derived controller scored 81/96 pickups and 80/96 strict holds. The correction lowered the aggregate score on this test.
The first blind episode, seed 910000, completed a strict pickup in 128 control steps (6.4 simulated seconds). Watch the wrist-camera recording
The exported policy issued the same six targets as the original checkpoint on all 428 updates of two local CPU replays, including the return and second-look sequence. Seed 910000 reproduced its cloud pickup. Seed 910006 timed out locally although it succeeded in the cloud. The table reports the original frozen cloud evaluation.
Inputs
At 20 Hz, pass the following fields to Policy.predict(state).
| Field | Format |
|---|---|
joint_positions |
Six measured follower positions |
previous_joint_targets |
Six targets last applied by the controller |
joint_velocities |
Six measured follower velocities |
images.wrist_cam.pixels |
Full RGB image, height 720 脳 width 1280 脳 3 channels, NumPy uint8 |
Joint order is shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll, gripper. Positions use follower degrees for the first five axes and percentage points for the gripper. Velocities use those units per second. The network receives the three vectors concatenated in the table order, normalized using frozen training statistics. It receives the current full frame scaled to [0, 1], with no crop, resize, or frame stack.
At the first update, a separate image feature reads every fourth pixel in each direction to locate orange pixels. Their centroid selects a demonstrated motion. For more distant cubes the controller can choose an outward-facing grasp. Episode memory tracks the motion and jaw deflection. After an empty grasp, the controller can return to the measured starting pose and use a second wrist image.
Output and use
predict returns {"joint_positions": [...]} with six finite absolute follower targets. The controller adds the deterministic PPO correction to the demonstrated joint motion, clips to the normalized action range, and converts to the calibrated follower limits. Call reset(metadata) once per episode, then call predict once per control update. The runner supplies the state.
from policy import Policy
policy = Policy()
policy.reset({})
command = policy.predict(state)
Use a local copy of the repository with the versions in requirements.txt. The entry point loads the TorchScript actor on CPU, plus NumPy arrays for the motion libraries, joint normalization, and correction scales. MuJoCo, Stable-Baselines3 and ground-truth cube coordinates are unnecessary for inference. Entry point 路 Actor weights 路 Motion and normalization weights 路 Robot profile 路 Manifest
Training and hardware status
The IK teacher generated full trajectories and corrective examples in simulation. The saved policy combines 126 primary and 50 outward-facing demonstration trajectories with a PPO residual trained across 12 parallel environments. The selected training checkpoint was at 1,013,508 cumulative transitions, including 439,968 added during this experiment. The evaluated task covers single-cube pickup. Placement, release onto a stack, multiple cubes and real-arm performance have not been evaluated.
Hardware queue status is pending. The datasheet guide reports a 128 脳 128 crop from the current Jetson runner. This policy needs the entire wrist view to locate the cube and rejects cropped inputs. Upscaling that crop would still omit scene content. The included profile describes the training calibration. Yondr has yet to publish the canonical manifest template and approved runtime digest. The package has a local inference entry point, and queue acceptance and real-hardware timing remain unverified. Submission guide used for this export