MIT-Media-Lab/egotouch-tactile-encoder-p1-v1

EgoTouch tactile encoder P1 v1

Three completed seeds of the full-touch/full-pose Bash BiGRU P1 encoder trained on published clean EgoTouch atomic episodes. The model aligns an eight-frame tactile history with the synchronized eight-frame 21-joint hand-pose history using symmetric masked InfoNCE.

Source code and frozen configs: kaichen-z/VLA-HAND@31d75d2.

Results

Values are three-seed mean +/- sample standard deviation on fixed 4,096-window galleries.

Split	Bidirectional retrieval mean mAP
validation	0.121405 +/- 0.001201
test seen	0.121559 +/- 0.000380
test unseen	0.048720 +/- 0.000976

Files

Each seed-* directory contains:

• training_checkpoint.pt: model, optimizer, scheduler, epoch, step, config, and selected validation metrics; use this to resume training.
• model.safetensors: complete FormalEncoderModel state dict.
• tactile_encoder.safetensors: standalone TactileEncoder state dict.
• pose_encoder.safetensors: standalone PoseEncoder state dict.
• exact resolved config, initialization report, runtime summary, and held-out metrics.

pressure_masks.npy contains the left/right 151-cell non-bend validity masks. export_manifest.json records every file size and SHA-256 checksum.

Input contract

• Tactile: [B, 8, 21, 21], target hand, pressure values as published.
• Pose: [B, 8, 21, 3], target-hand joints_camspace.
• Output: L2-normalized 64D tactile and pose embeddings.
• Canonical hand: right. Left tactile is flipped horizontally and left pose x is reflected before model input.
• Pose is wrist-centered and temporally hand-scale-normalized inside the pose encoder.
• Instruction and RGB are not model inputs.

Standalone tactile encoder

Clone the source repository and install Tactile/requirements.txt, then:

import numpy as np
import torch
import torch.nn.functional as F
from safetensors.torch import load_file
from egotouch_formal.model import TactileEncoder

encoder = TactileEncoder(embed_dim=64)
encoder.load_state_dict(load_file("seed-20260831/tactile_encoder.safetensors"))
encoder.eval()

pressure_masks = torch.from_numpy(np.load("pressure_masks.npy")).bool()
touch = torch.zeros(1, 8, 21, 21)  # already hand-canonicalized
side = torch.tensor([1])            # 0=left, 1=right; selects validity mask
with torch.no_grad():
    embedding = F.normalize(encoder(touch, pressure_masks[side]), dim=-1)
assert embedding.shape == (1, 64)

For the complete tactile-pose model, instantiate FormalEncoderModel with task="p1", pose_temporal="gru", and embed_dim=64, then load model.safetensors strictly.

Training protocol

Batch 1,024; AdamW at 1e-4 with weight decay .01; 500-step linear warmup and cosine decay; BF16; gradient norm clip 1.0; 40 epochs. The three runs completed 25,080 optimizer steps because the epoch limit was reached before the 30,000 step cap.

The initialization checkpoint is basharkMIT/opentouch-gru-pose-encoder/epoch_300.pt, pinned in the source downloader by SHA-256.

Data and limitations

The exact preprocessed cache is published separately as MIT-Media-Lab/egotouch-clean-atomic-p1-cache-v1. The test-unseen gap shows that cross-recording-population generalization remains substantially harder than validation/test-seen retrieval. This experiment does not demonstrate improved future-pose prediction; no future-pose head was trained.