AllenBi21/tactile-act-screw-baselines

Deploying the T3- and SITR-conditioned ACT policies (screw task)

written: 2026-08-11 · trained 2026-08-09 on the main workstation · for the Claude session (or human) on any other machine.

1. What these models are

Two ACT policies for the screw task, protocol-identical to the existing screw_qposonly_50k / screw_wrench5only_50k / screw_tf9dpe_50k family (bs=32, lr 2e-5 warmup 5K → cosine 2e-6, 50K steps, chunk 30, CVAE on, no image branch, val = last 2 episodes):

Run	Conditioning	tactile input	best val L1 (norm.)	offline val L1
screw_t3_50k	qpos + T3 features	5 fingers × 768 = 3840-d	0.2601	0.0319
screw_sitr5_50k	qpos + SITR features	5 fingers × 768 = 3840-d	0.2630	0.0312

Interpretation guardrail: ALL conditionings sit at the dataset-limited offline floor (~0.031); offline L1 does not rank them. These exist as the reviewer-requested baselines for on-robot rollout comparison (same 20-trial protocol as the paper's Table 5).

2. Where the weights are

• HuggingFace (canonical for other machines): AllenBi21/tactile-act-screw-baselines (private) — screw_t3_50k/act_best.pt, screw_sitr5_50k/act_best.pt (+ args.json each, and this doc as README).

  hf download AllenBi21/tactile-act-screw-baselines --local-dir act_baselines
  

• Main workstation: tactile_act/checkpoints/<run>/act_best.pt (symlink into /mnt/4tssd/tactile_act_ckpts/checkpoints/).
• Use act_best.pt (best validation), NOT the _p2 dirs' act_final.pt (cosine-tail continuation; val never improved past the phase-1 best).

3. Code changes needed on the target machine

The tactile_act repo (github.com/jxbi1010/tactile_act) needs exactly ONE new file relative to the May state: tools/encode_t3.py (bundled in the HF repo; drop it into tools/). Everything else used by these runs (generalized --tactile-keys routing <group>_<leaf> → /<group>/<leaf>, --resume, --val-episodes) was already in the repo. No changes to act_policy/, inference.py, or train.py are required for inference.

4. Inference contract

The checkpoint stores model + config + normalization stats; loading needs only the .pt path:

from inference import ACTInference
infer = ACTInference.from_checkpoint("act_baselines/screw_t3_50k/act_best.pt", device="cuda")
infer.reset()                                   # once per episode
action = infer.step({"state": qpos22,           # (22,) float
                     "tactile": feat3840})      # (3840,) float, see §5
# action = (22,) ABSOLUTE target joint positions; model re-queried every 30 frames

Offline sanity eval (needs the episode HDF5s with /t3/<f> or /sitr/<f>):

python inference.py --checkpoint act_baselines/screw_t3_50k/act_best.pt \
  --mode offline --data ./episodes/screwing \
  --tactile-keys t3_0 t3_1 t3_2 t3_3 t3_4 --out-csv eval/screw_t3_50k.csv
# SITR: --tactile-keys sitr_0 sitr_1 sitr_2 sitr_3 sitr_4

5. Computing the tactile features live (the part that differs per model)

Both: per-finger fingertip camera frame (SharpaWave "raw", grayscale 240×320 uint8) → feature (768,); concatenate fingers 0..4 in order → (3840,). Batch the 5 fingers in one forward for latency.

T3 (mirrors tools/encode_t3.py, identical to the paper's T3 probe rows):

1. x = gray/255 → replicate to 3 channels → bilinear resize 224×224.
2. Normalize with fixed constants mean=0.49611, std=0.18685 (all channels — computed once on the screwing corpus; do NOT recompute at deploy time, the policy was trained on features made with these).
3. T3-medium, encoder domain mini: tokens = encoder(x); tokens = trunk(tokens) then non-affine F.layer_norm(tokens, (dim,)), take cls tokens[:,0].
4. Encoder weights: t3_medium/ release (locally /mnt/4tssd/datasets/t3/models/t3_medium; loader: tactile_fusion load_baselines._load_t3_raw(modality="9dtact")).

SITR (mirrors tools/encode_sitr.py):

1. Same gray→3ch→224 path.
2. Normalize with SITR_MEAN=(-1.2223,-1.8114,-1.7090), SITR_STD=(11.7932,12.7956,13.6452) applied to the [0,1] tensor.
3. SITR_base(num_calibration=0).forward_encoder(x, c=None)[:, 0, :] (cls).
4. Weights: SITR_B18.pth (locally ~/projects_25/gsrl/checkpoints/; source: SITR's official HF release).

Dependency note: the encoders come from third-party repos (t3, gsrl) — they are NOT re-hosted on our HF repo; use the local copies or the official releases.

6. Gotchas

• Finger order matters (0..4); wrench/qpos finger indexing in this dataset is REVERSED between groups (see EXPERIMENT_LOG "qpos and wrench finger indexing") — but t3/sitr features follow the CAMERA index = wrench order. Keep exactly the same order used in training: t3_0 … t3_4.
• infer.step() returns absolute target qpos regardless of training target type; call reset() at each episode start.
• The two _p2 run dirs exist because training was interrupted at ~30K by a full disk and resumed; act_best.pt in the phase-1 dirs is the deliverable.
• Single seed per condition; offline L1 differences vs other conditionings are within noise — rollout success is the meaningful comparison.