bmnzyb/goalblocks-so101-baseline

GoalBlocks SO-101 Baseline

This repository contains an experimental goal-conditioned action-chunk policy for the SO-101 robot. It was built to validate a complete path from goal-aware LeLab recording to multi-dataset training, remote GPU inference, and local WSL robot integration.

The training data was collected with an extended LeLab recording dialog that accepts goal images and can generate an editable natural-language task description before recording.

Demonstration

Watch the GoalBlocks SO-101 collection demonstration (1:51 MP4)

Model

The policy consumes:

• observation.images.top;
• observation.images.wrist;
• a goal image;
• a natural-language task encoded as UTF-8 byte tokens;
• six-dimensional SO-101 joint state.

It predicts a 16-step chunk of six-dimensional joint targets. The network uses a shared compact CNN for the two observations and goal image, a byte-level text encoder, state projection, and an MLP fusion head.

Parameter count: 738,432 (0.74M). A comparable two-camera LeRobot ACT configuration has approximately 51.55M parameters, around 69.8 times more.

Training

The run used balanced sampling across two independent datasets and an episode-level validation split. Training included AdamW, mixed precision, gradient clipping, linear warmup, cosine decay, periodic checkpoints, and validation every 250 steps.

• Planned steps: 10,000
• Training stopped intentionally after step 780
• Published checkpoint: best validation checkpoint from step 750
• Validation loss at step 750: 0.130553
• Action horizon: 16
• Input image size: 128 x 128
• State/action dimension: 6

The repository includes best_model.pt, train_config.json, metrics.jsonl, and datasets.json.

Training Data

1. bmnzyb/GoalBloakcs_1_20260906_155729: 3 episodes.
2. bmnzyb/GoalBloakcs_1_20260906_162409: 2 episodes.

Both datasets contain top and wrist RGB observations, six-dimensional robot state/action, a task description, and a goal image. The first top-camera stream has known corrupted/torn frames.

These five episodes are sufficient only for pipeline validation. They are not sufficient for a meaningful generalization or robot-success claim.

Community Data Collection

More goal-conditioned SO-101 demonstrations would make this experiment substantially more useful. Contributions covering new goal arrangements, block colors, objects, lighting, camera poses, robot setups, successes, and recovery behavior are welcome. Please share datasets together with hardware, camera, task, calibration, and collection metadata through the associated source repository.

Usage

Architecture, loading, training, remote serving, and safety-oriented client code are maintained in the associated GitHub project. This checkpoint uses a small custom PyTorch module and is not directly loadable through LeRobot's built-in PolicyServer policy registry.

Limitations and Safety

• The model was intentionally undertrained and may produce arbitrary or unsafe actions.
• One training camera stream is visibly corrupted.
• No successful autonomous task execution has been established.
• The remote integration test reached observation capture but was stopped before policy actions were executed because of camera corruption and an intermittent motor status-packet error.
• Do not connect this checkpoint to a physical robot without independent validation, strict action limits, an observation-only dry-run, a clear workspace, and immediate emergency power access.

Attribution and License

The implementation is an unofficial derivative research project built with APIs and code derived from Hugging Face LeRobot and an extended Hugging Face LeLab. Those upstream projects are licensed under Apache-2.0. Their ownership and copyright remain with their respective authors.

The checkpoint and accompanying original integration code are released under Apache-2.0. This does not imply endorsement by Hugging Face and does not change the independent terms that may apply to training datasets, Qwen/DashScope services, dependencies, hardware, or third-party assets.

SO-101 GoalBlocks Dataset Project

Last updated: September 2026

Purpose

GoalBlocks is an open, goal-conditioned data collection project for the SO-101 single-arm robot. Given a target arrangement of colored blocks, the robot should select blocks from a workspace and reproduce the arrangement through teleoperation demonstrations.

Every task has two complementary conditions:

• a concise natural-language instruction;
• one or more goal images, such as a 45-degree view or top/front/side views.

A trajectory contains the SO-101 six-dimensional joint state, top and wrist camera observations, six-dimensional action targets, and task metadata. The long-term objective is one policy that can learn across target shapes, color combinations, object counts, operators, and scenes.

Dataset Contract

Each raw contribution should remain a standard LeRobot dataset and should expose:

• observation.state: six-dimensional SO-101 joint state;
• observation.images.top: workspace camera;
• observation.images.wrist: wrist camera;
• action: six-dimensional future joint targets;
• task and task_index: the final human-reviewed task instruction;
• goals/: one or more goal images;
• meta/info.json: goal and collection metadata.

For the first release, keep one prompt and one goal image set per raw Hugging Face dataset repository. This makes collection, review, provenance, and later merging straightforward.

Collection Workflow

The extended LeLab workflow implemented in the associated source repository is:

Connect SO-101 and cameras
        ↓
Upload one to four goal images
        ↓
Optionally generate a draft task with Qwen-VL
        ↓
Review and edit the final task text
        ↓
Record several teleoperation episodes
        ↓
Save goal images and metadata with the LeRobot dataset
        ↓
Validate replay and publish the raw dataset

The Qwen API key is used only for the generation request and is not written into dataset metadata or project files. The final instruction remains editable because human review is essential for unambiguous robot supervision.

Multi-Task Organization

LeRobot v3 can represent multiple tasks inside one dataset through meta/tasks.parquet, per-frame task_index, and episode metadata. A practical early-stage organization is two-layered:

Raw layer: one goal/prompt per independently reviewable contribution.
Release layer: periodically merge approved raw datasets into a versioned multi-task release.

For modest data volumes, the official merge workflow is appropriate. For very large collections, this project proposes multi-repository streaming rather than repeated full merges. The included GoalBlocks baseline provides a compact multi-repository loader for pipeline validation; it is separate from LeRobot's standard training command.

Goal-Conditioned Training

Goal images stored in metadata are not automatically consumed by a standard policy. A goal-conditioned training path needs current observations, robot state, language task, goal image, and action target together. This is necessary because identical current scenes can require different actions for different target arrangements.

The released baseline accepts two observation cameras, robot state, task text, and a goal image. It is an experimental pipeline baseline, not a production policy.

Contribution and Review

Contributors should publish raw data in their own Hugging Face dataset repositories and submit the repository identifier for review. Review should verify SO-101 state/action compatibility, expected camera fields, readable videos, goal metadata, clear task language, and safe demonstrations.

Roadmap

1. Validate the full pipeline with small, single-goal raw datasets.
2. Collect more reviewed demonstrations using goal-aware LeLab recording.
3. Release periodic merged LeRobot datasets when storage is practical.
4. Improve multi-repository streaming, validation, and balanced sampling for scale.
5. Compare language-only, goal-image-only, and joint language-plus-goal-image policies.

Reference

The survey was informed by the public Project-IRA SO-101 multi-task LeRobot dataset.

Source Code

The goal-aware recording extension and GoalBlocks baseline source code are available at BMN-zyb/lelab-goalblocks.

License and Scope

This is an independent research and community-data proposal. The implementation is an unofficial derivative built on Hugging Face LeLab and LeRobot. It preserves upstream attribution and Apache-2.0 licensing; it is not an official Hugging Face release or endorsement.