JackieMM/progress_probe_repro

Progress Probe Reproduction

This folder contains a standalone reproduction and visualization script for the episode-level progress probe.

Data Layouts

The script supports two layouts.

Flat Local Layout

For fitting and CV, each task needs HDF5 label files:

/home/ydming/datasets/himaconplusplus_0.8B_labels/
  validation-robotwin_<task>-...-label/
    episode_00000.hdf5
    ...
    episode_00099.hdf5

Each HDF5 file must contain:

concept: float32, shape (T, 960)

The regression label is generated from time:

y = np.linspace(0.0, 1.0, T)

For visualization, the corresponding raw video is also needed:

/home/ydming/datasets/robotwin2.0-hard/<task>/demo_clean_100/video/episode*.mp4

Do not assume episode_00029.hdf5 maps to episode29.mp4. Build and inspect a mapping first.

Cross-Embodiment Layout

On another server, use explicit roots:

LABEL_ROOT=/data/share/robotwin2.0-hard-cross-embodiment
VIDEO_ROOT=/data/share/robotwin2.0-hard

Expected structure:

$LABEL_ROOT/<task>/<demo>/<label_kind>/validation-...-label/episode_*.hdf5
$VIDEO_ROOT/<task>/<demo>/video/episode*.mp4

Examples:

/data/share/robotwin2.0-hard-cross-embodiment/beat_block_hammer/demo_clean_100_arx/full_label/validation-...-fps2-label/
/data/share/robotwin2.0-hard/beat_block_hammer/demo_clean_100_arx/video/

label_kind can be full_label or womae_label. Some folders may contain more than one validation directory; choose one with --validation-index.

Scan Cross-Embodiment Dataset

Before training or rendering on another server, scan what is available:

python progress_probe.py scan-dataset \
  --label-root /data/share/robotwin2.0-hard-cross-embodiment \
  --video-root /data/share/robotwin2.0-hard \
  --output-json outputs/dataset_manifest.json

The manifest records task, demo, label kind, validation directory, video directory, and file counts.

Fit 960-D Ridge CV

python /home/ydming/progress_probe_repro/progress_probe.py fit-cv \
  --task blocks_ranking_size \
  --task-dir /home/ydming/datasets/himaconplusplus_0.8B_labels/validation-robotwin_blocks_ranking_size-20260516_192550-fps10-label \
  --output-dir /home/ydming/progress_probe_repro/outputs/blocks_ranking_size_960 \
  --feature-start 0 \
  --alpha 0.0 \
  --seed 42

Cross-embodiment example:

python progress_probe.py fit-cv \
  --task beat_block_hammer \
  --demo demo_clean_100_arx \
  --label-kind full_label \
  --label-root /data/share/robotwin2.0-hard-cross-embodiment \
  --output-dir outputs/beat_block_hammer/demo_clean_100_arx/full_label_960 \
  --feature-start 0 \
  --alpha 0.0 \
  --seed 42

If the selected label kind contains multiple validation-...-label directories:

  --validation-index 1

Outputs:

outputs/blocks_ranking_size_960/
  checkpoints/fold_0.joblib
  ...
  checkpoints/model_all_episodes.joblib
  metrics/cv_metrics.json

Build Mapping

Candidate natural-sort mapping for a full task:

python /home/ydming/progress_probe_repro/progress_probe.py build-mapping \
  --task blocks_ranking_size \
  --output-dir /home/ydming/progress_probe_repro/outputs/mappings \
  --check-images \
  --max-check-images 20

Cross-embodiment mapping:

python progress_probe.py build-mapping \
  --task beat_block_hammer \
  --demo demo_clean_100_arx \
  --label-kind full_label \
  --label-root /data/share/robotwin2.0-hard-cross-embodiment \
  --video-root /data/share/robotwin2.0-hard \
  --output-dir outputs/mappings \
  --check-images \
  --max-check-images 20

If an existing selection.json is available, convert it into local paths:

python /home/ydming/progress_probe_repro/progress_probe.py selection-to-mapping \
  --selection-json /home/ydming/progress_history_extraction/dynamic_videos_test/selection.json \
  --output-json /home/ydming/progress_probe_repro/outputs/mappings/selection_mapping.json

The mapping JSON records concept step count, video frame count, FPS, and an alignment status. Inspect generated check images before trusting a bulk mapping.

Visualize One Episode

python /home/ydming/progress_probe_repro/progress_probe.py visualize-one \
  --mapping-json /home/ydming/progress_probe_repro/outputs/mappings/selection_mapping.json \
  --mapping-index 0 \
  --model-path /home/ydming/progress_probe_repro/outputs/blocks_ranking_size_960/checkpoints/fold_8.joblib \
  --output-dir /home/ydming/progress_probe_repro/outputs/viz/blocks_ranking_size/example \
  --feature-start 0

Outputs:

full.mp4
video_only.mp4
curve_panel.mp4
progress_bar.mp4
error_panel.mp4
metadata.json

Use --step 2 or --step 5 for faster preview renders.

Cross-embodiment visualization uses the mapping created above:

python progress_probe.py visualize-one \
  --mapping-json outputs/mappings/beat_block_hammer__demo_clean_100_arx__full_label_mapping.json \
  --mapping-index 0 \
  --model-path outputs/beat_block_hammer/demo_clean_100_arx/full_label_960/checkpoints/fold_0.joblib \
  --output-dir outputs/viz/beat_block_hammer/demo_clean_100_arx/example \
  --feature-start 0 \
  --step 5

Primitive Classification Probe

4-class primitive classification on blocks_ranking_rgb. Protocol v2 definitions:

• reach — moving without holding (open-gripper approach / reposition)
• grasp — close gripper and establish grasp
• transition — move while holding the object (carry)
• release — open gripper at target

Full rules: docs/primitive_annotation_protocol.md

Install extras:

pip install -r requirements-classification.txt

Key commands:

# Inspect episode data (concept, video, gripper signals)
python classification_probe.py inspect-episode \
  --task blocks_ranking_rgb --demo demo_clean_100_franka --episode 0 \
  --label-root /share/robotwin2.0-hard-cross-embodiment \
  --video-root /share/robotwin2.0-hard

# OpenCV GUI annotation (or export-frames for headless)
python classification_probe.py annotate-one \
  --task blocks_ranking_rgb --demo demo_clean_100_franka --episode 0

# Heuristic proposal from gripper (pilot assist, not ground truth)
python classification_probe.py propose-primitive-labels \
  --task blocks_ranking_rgb --demos demo_clean_100_arx demo_clean_100_franka demo_clean_100_ur5 \
  --max-episodes 20

# Within-robot 10-fold CV
python classification_probe.py fit-cv \
  --task blocks_ranking_rgb --demo demo_clean_100_franka --label-kind full_label \
  --output-dir outputs/classification_probe/within_robot/.../full_label_val0_960

# Cross-robot matrix
python classification_probe.py fit-cross-robot \
  --task blocks_ranking_rgb \
  --output-dir outputs/classification_probe/cross_robot/blocks_ranking_rgb

# Batch milestones: smoke | pilot | final
python run_classification_pilot.py pilot --n-episodes 20 --n-splits 5

Outputs live under outputs/classification_probe/ (annotations, splits, within_robot, cross_robot, baselines, summary, visualization).