Dongkkka/Task000519_PourWater_ViTacFormer_H100_B512_Hand_Intern

SH5 ViTacFormer — pour only, H100, batch 512, latest saved step 120,000

Latest saved checkpoint selected by the owner, not the validation-best model. Incomplete training; non-actuating offline inference only, no real-robot deployment approval.

사용자 요청에 따라 현재 저장소의 50,000-step best를 제거하고, 마지막으로 저장된 120,000-step 모델로 교체했습니다. 학습은 120,450 step에서 종료됐지만 마지막 450 update는 checkpoint로 저장되지 않았습니다. 이 모델은 validation-best가 아니라 latest입니다. 이전 best는 Git 과거 이력과 서버 로컬 원본에만 남아 있으며 현재 브랜치의 추론 파일에는 포함하지 않습니다.

Selection and measured performance

Use checkpoints/latest_model.pt. It is an exact copy of the source inference-only step_120000.pt; every model tensor was verified equal to the saved resume latest.pt. Optimizer/RNG state is intentionally omitted because it is not needed for inference. The checkpoint retains its original periodic-save metadata; validation metrics are provided separately in artifact_receipt.json and validation_history.json.

Latest saved checkpoint metric	Value
Selected step	120,000
Validation windows	11,592
Composite selection score, lower is better	0.3885593466
Normalized action L1	0.1277322565
Normalized future tactile L1	0.1582342720
Normalized persistence baseline L1	0.1581808998
Left / right arm L1, radians	0.05069903 / 0.07291619
Left / right hand L1, radians	0.01461084 / 0.02282853
Offline joint-bound / checked continuity gates	Passed
Tactile skill / combined offline release gate	Failed

Validation uses predicted future tactile, zero latent and the bounded decoder. The last logged training loss was 0.23830150 at step 120,440 (20-step GT-conditioned average), not the validation loss or a saved 120,440-step model. The intended 400k training and the 300k GT-to-predicted tactile curriculum transition were not completed.

Data and training contract

• Current Task 519 pick-and-pour folder: 159 episodes / 123,853 frames. Its episodes 0–80 originate from Task 519 and 81–158 from Task 608. All belong to the pour phase. Split: 142 train episodes / 93,817 windows; 17 held-out episodes / 11,592 windows. Separate Task 608 uprighting/placement folder is excluded. No horizon crosses an episode.
• Scratch policy training, ImageNet ResNet18 backbone initialization only. Batch 512, workers 4, prefetch 2. AdamW LR 1e-5, backbone LR 1e-6, weight decay 1e-4; bf16 forward and fp32 losses; augmentation and exact settings are in train_config.json.
• Planned GT future tactile for first 300k updates, predicted tactile for last 100k. Training stopped before that switch. Extra persistence-excess and residual-magnitude penalties are zero; the persistence-plus-residual parameterization is retained.
• Architecture vitacformer_sh5_pour_h100_v2, recipe task519_folder159_gt75_residual_unpenalized_r1. DETR/CVAE cross-attention, encoder 4, decoder 7, hidden 512, FFN 3200, heads 8, latent 32.
• Action [100,54]: left arm 7, right arm 7, left hand 20, right hand 20, absolute radians. Query weights [100,512]; position table [1,102,512].
• State [6,54], 30 Hz offsets [-15,-12,-9,-6,-3,0].
• Left-head RGB only: [376,672,3] uint8, linear resize to [188,336], /255, ImageNet normalization. Do not flip, rotate, or reorder cameras/joints.
• Raw tactile [18,90]: left 45 then right 45 taxels, history offsets -17..0. Subtract the calibrated per-taxel baseline, clamp negative pressure to zero, concatenate pressure and pressure minus oldest row to [18,180]; do NOT divide tactile by 255. Future tactile remains 18 rows, not 100.
• Baselines in training were rounded per-taxel medians of the first 20 raw messages at the paired pour episode's beginning. Live calibration must reproduce this convention under suitable initial contact conditions; do not recalibrate while holding objects. Baselines are observations, not one universal saved constant.
• No language encoder. No head, lift, base, bottle-uprighting or put-down actions.
• This is an SH5 adaptation, not exact paper replication or evidence of task success.

Complete inference package

Keep checkpoints/latest_model.pt, configs, normalization statistics, preprocess.py and source_snapshot/ViTacFormer_SH5 together. The checkpoint binds the original config and normalization SHA-256 hashes: do not edit these files. normalization_stats.json also contains readable numerical values.

contracts/sh5_recorded_config.yaml documents the recording-side observation and leader-command topic/joint mapping. It is not a robot-approved follower publishing configuration. contracts/runtime_proposal.json is unacknowledged and non-authorizing. Provenance, exact package checksums and representative replay results are included. No credentials, training videos, caches, optimizer state or redundant weights are included. The code retains its upstream Apache-2.0 license in the source snapshot; no additional dataset/model licensing terms are inferred here.

After downloading into a new directory, verify every payload:

sha256sum -c SHA256SUMS

Install a platform-compatible PyTorch/torchvision pair and the remaining requirements. The tested x86 server versions are recorded in environment_tested.json; this does not establish Jetson support or runtime speed. This is a custom loader, not a Transformers AutoModel or standalone LeRobot policy checkpoint.

import sys
from pathlib import Path
from huggingface_hub import snapshot_download

root = Path(snapshot_download(
    repo_id="Dongkkka/Task000519_PourWater_ViTacFormer_H100_B512_Hand_Intern"
))
sys.path[:0] = [str(root), str(root / "source_snapshot/ViTacFormer_SH5")]
from inference_loader import load_run, predict_normalized
from preprocess import prepare_observation

policy, stats, config = load_run(root, device="cpu")
# Supply synchronized, correctly ordered SH5 arrays:
# image_rgb: uint8 [376,672,3]; state_history: float32 [6,54];
# tactile_raw_history: float32 [18,90]; tactile_baseline: float32 [90].
batch = prepare_observation(
    image_rgb, state_history, tactile_raw_history, tactile_baseline, stats, device="cpu"
)
actions_rad, future_tactile_normalized = predict_normalized(policy, stats, batch)
# [1,100,54] absolute-radian actions and [1,18,180] normalized tactile; NO actuation.

predict_normalized returns raw-radian actions, despite accepting normalized input. It applies predicted tactile, zero latent, the persistence residual, soft bounds (beta 1000), and the 16-row arm warm-start ramp. Bare ACTPolicy bypasses this contract and is unsupported. The first arm row is current state; repeatedly using only row 0 can stall motion. Do not use GT future tactile during inference.

Verification and remaining robot gates

offline_verification.json records strict package loading, config/stats integrity, real held-out video decoding and six representative windows (early/mid/late, one episode from each source group), exact preprocessing agreement, finite bounded outputs and deterministic repeat. This is not another full validation or a robot trial. Reproduce with the source dataset and generated training image cache using python verify_package.py --dataset-root ... --image-cache ... (requires pyarrow).

H100 at 30 Hz predicts about 3.33 seconds; it does not authorize executing all rows. The unacknowledged runtime proposal retains source chunk limit 20, alignment .3 s, refill .55 s, temporal ensembling disabled. No robot runtime has been changed.

Before any powered trial, resolve tactile performance and incomplete curriculum, verify exact robot revision/calibration/camera/joint order and follower topics, measure latency and tracking, complete offline/shadow tests, reject unsafe/stale whole chunks, and obtain operator approval with a working E-stop. Define completion and post-pour stopping explicitly; this dataset excludes uprighting and placement.

Sources

• ViTacFormer paper: https://arxiv.org/abs/2506.15953
• Recorded source lineage: d94788272b5f5e18a80bf62e3e5d51e7d2581d77.
• Exact adapted source, data fingerprints and split are included in this package.