bamboo51/act_crane_topwrist

ACT: CRANE+ box pick-and-place (top + wrist cameras)

An ACT (Action Chunking with Transformers) policy that controls a simulated CRANE+ arm in Gazebo: it picks up a 5 cm box and places it on the roof of a toy car. It sees two cameras (a fixed top view and a wrist camera) plus the 5 joint angles.

Trained with LeRobot on demonstrations from a scripted MoveIt expert. Code: bamboo51/Arm-Robot-Picking. Built as the final assignment for the UTokyo Matsuo Lab Physical AI course.

Inputs and outputs

	Key	Shape	Notes
Input	observation.images.top	3×240×320	Fixed camera above the table
Input	observation.images.wrist	3×240×320	Camera on the gripper
Input	observation.state	5	Joint angles (rad): crane_plus_joint1–4, crane_plus_joint_hand
Output	action	5	Joint targets 0.5 s ahead (K=5 frames at 10 Hz), same joint order

• Control rate: 10 Hz. Each target should be reached in 0.5 s (time_from_start = 0.5 s), not in one tick.
• Gripper action is the commanded angle (close = +10°), not the angle where the fingers stopped at the box.
• All inputs and outputs are normalized with mean/std; the pre/post processors in this repo do it for you.

Training

Setting	Value
Policy	ACT, ResNet-18 backbone (ImageNet init), VAE (latent 32)
Transformer	dim 512, 8 heads, 4 encoder layers, 1 decoder layer, FFN 3200
Chunk size	50 (n_action_steps = 1 at inference)
Steps / batch	20,000 / 64
Optimizer	AdamW, lr 1e-5 (backbone 1e-5), weight decay 1e-4, grad clip 10
KL weight	10
Image augmentation	off
Seed	1000

Data: Phongwit/crane_pick_place_topwrist, about 70 successful expert episodes in random scenes (the box position is sampled within the arm's reach; the car is fixed). Frames where nothing moves in the next 0.5 s are dropped. Validation and test scenes are held out (NPZ episodes).

Usage

Inside the project's Docker container, with Gazebo running (./scripts/run_simulation.sh):

huggingface-cli download Phongwit/act_crane_topwrist --local-dir learning/checkpoints/act_crane_topwrist

# Terminal 2: policy server (Python 3.13 venv, GPU)
/opt/lerobot-venv/bin/python learning/policy_server.py learning/checkpoints/act_crane_topwrist
# optional: N=<steps of each chunk to execute>, TE=0.01 for ACT temporal ensembling

# Terminal 3: ROS node that sends cameras + joints and publishes the actions
python3 scripts/infer_node.py
# or score it on every test scene:
python3 scripts/eval_gazebo.py

policy_server.py also needs the dataset's meta/ folder at lerobot_data/crane_pick_place/meta/.

Limitations

• Simulation only. Trained and tested in Gazebo with fixed lighting and textures; not tested on a real arm.
• One task, one layout. The car is always in the same place and only the box position varies. Other objects, car positions or camera poses are out of distribution.
• Small dataset (about 70 episodes) from a single deterministic expert, so recovery from mistakes is limited.
• Strict input format. Cameras must be 320×240 RGB with these names, and actions must be sent as 0.5 s targets; otherwise the arm moves at the wrong speed.

References

1. Zhao et al., "Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware" (ACT), RSS 2023.
2. Cadene et al., LeRobot, https://github.com/huggingface/lerobot
3. matsuolab/pai_training