Dongkkka/Task000519_000608_ViTacFormer_400k_chunk200_Hand_Intern

SH5 ViTacFormer — chunk 200, best checkpoint at step 142,000

Partial training snapshot, NOT a completed 400k model or approved robot deployment.

이 저장소는 400k 목표 학습이 중단된 시점의 142,000-step best checkpoint입니다. 마지막 로그는 178,899 step이며 마지막 재개 checkpoint는 178,000 step입니다. 400k라는 저장소 이름은 목표 학습량을 뜻하며 완료 학습량이 아닙니다.

Selection and validation

checkpoints/best_model.pt was selected by minimum full held-out composite score. The source best_overall.pt and best_phase2.pt also select step 142,000; redundant copies and optimizer checkpoints are intentionally omitted.

Metric	Value
Selected step	142000
Full held-out windows	10787
Composite selection score (lower is better)	0.4110245585
Action L1, normalized	0.1601985194
Tactile L1, normalized	0.1280515680
Persistence baseline tactile L1, normalized	0.1279534070
Left arm / right arm L1, radians	0.07246352 / 0.11329461
Left hand / right hand L1, radians	0.02389834 / 0.02348642
Offline release gate	Failed: tactile skill

The checked joint-bound/adjacent-delta gates passed full held-out validation, but neither tactile side beat persistence. This does not establish real-robot success, tracking safety, or that longer task execution is improved. The H100 and H200 validation windows differ, so their aggregate losses are not directly comparable. Full original metrics are in artifact_receipt.json.

Architecture and contract

• SH5 adaptation of ViTacFormer DETR/CVAE with cross-attention and future tactile prediction; architecture vitacformer_sh5_h200_v1, recipe sh5_h200_original_tactile_r1.
• Action queries [200,512]; latent-encoder sinusoidal table [1,202,512].
• Both arms (7+7 joints) and both hands (20+20 joints) are supervised: 54 absolute-radian targets. Exact names/order and limits are in inference_config.json.
• State [6,54], at 30 Hz offsets [-15,-12,-9,-6,-3,0].
• Left-head RGB camera only: 672x376 to 336x188 via OpenCV linear resize, ImageNet normalization.
• Tactile history [18,90] raw taxels, left45 then right45. Subtract per-hand calibrated baseline, clamp negative pressures to zero, concatenate pressure and relative-to-oldest history to [18,180]. Do not divide tactile pressure by 255. Future tactile remains 18 rows, not 200.
• No runtime language encoder: the task instruction is metadata for a task-specific policy.
• Scratch policy training with ImageNet ResNet18 initialization only; prior policy weights were not resumed.
• Original tactile recipe retained at user request: .75 teacher ratio cosine-decays to 0 by 12k; tactile delta/contact weights .5/.5, persistence-excess/residual weights 5/.5.
• Action temporal loss weights linearly decrease 1 to .5 then normalize to mean 1. Other exact settings and augmentation are in train_config.json.
• This is an adaptation, not a claim of exact full-paper methodology reproduction.

Files and integrity

Keep the package together, including checkpoints/best_model.pt, config.json, inference_config.json, normalization_stats.pt, the source snapshot and preprocess.py. The checkpoint binds the original config and normalization hashes; do not edit them. normalization_stats.json also contains the numerical statistics for inspection. SHA256SUMS covers every payload file except itself; verify after downloading:

sha256sum -c SHA256SUMS

No training datasets, credentials, WandB cache, duplicate weights or optimizer state are included. The source code carries its upstream Apache-2.0 license in the snapshot; this card does not assert additional dataset/model redistribution terms beyond the owner's publication.

Non-actuating inference

Install a platform-compatible PyTorch/torchvision pair, then the remaining requirements.txt. environment_tested.json records the exact training-server versions; Jetson compatibility is not established by the x86 CPU test. This is a custom loader, not a Transformers AutoModel.

import sys
from pathlib import Path
from huggingface_hub import snapshot_download

root = Path(snapshot_download(
    repo_id="Dongkkka/Task000519_000608_ViTacFormer_400k_chunk200_Hand_Intern"
))
sys.path[:0] = [str(root), str(root / "source_snapshot/ViTacFormer_SH5")]
from inference_loader import load_run, predict_normalized
from preprocess import prepare_observation

policy, stats, config = load_run(root, device="cpu")
# Provide synchronized arrays from the exact SH5 contract:
# image_rgb: uint8 [376,672,3], state_history: float32 [6,54],
# tactile_raw_history: float32 [18,90], tactile_baseline: float32 [90].
batch = prepare_observation(
    image_rgb, state_history, tactile_raw_history, tactile_baseline, stats, device="cpu"
)
actions_rad, tactile_future_norm = predict_normalized(policy, stats, batch)
# actions_rad: [1,200,54]; tactile_future_norm: [1,18,180]. No robot publishing here.

predict_normalized accepts normalized inputs but returns raw absolute-radian actions. It uses the predicted-tactile zero-latent path, persistence residual, bounded decoder and 16-row arm warm-start ramp. Calling bare ACTPolicy bypasses these semantics and is unsupported. The first arm row starts at current state; repeatedly executing only row 0 can stall motion.

Robot execution restrictions

Prediction horizon 200 at 30 Hz is about 6.67 seconds, not permission to execute all 200 rows. The unacknowledged runtime proposal retains source chunk limit 20, alignment window .3 seconds, refill margin .55 seconds, temporal ensembling disabled. No robot runtime was changed or tested. Head, lift and mobile base commands are outside this checkpoint.

Before powered execution, validate exact robot revision, names/units/cameras, tactile calibration, controller limits and latency, run offline replay and shadow tests, enforce fresh streams, bounded commands and fail-closed safety, and have an operator at the E-stop. This package is for non-actuating offline evaluation until those gates are satisfied.

Export validation (offline_verification.json) strictly loaded this package on CPU and decoded one real held-out video/window from each source. Preprocessing exactly matched training (max difference 0); outputs were finite [1,200,54] and within joint bounds. Two-window smoke checks are not a new full validation or a real-robot safety trial.

Sources

• ViTacFormer paper
• Source commit recorded by the training snapshot: d94788272b5f5e18a80bf62e3e5d51e7d2581d77.
• Train data selection is recorded in provenance/dataset_selector.json: Task 519 episodes 0..80 and combined-folder fresh episodes 81..119, avoiding the duplicate 519 prefix.