RammusLeo/TouchAnnotator-0813

0813 · Optional Qualitative Checkpoint

Egocentric video → WiLoR pose → relative tactile pressure

💻 GitHub Code · 🚀 Quick Start · 📖 0813 Guide · 🤗 Default 0824 Model

✨ About this release

0813 is an earlier TouchAnnotator checkpoint for qualitative exploration. Selected historical connector, box and earphone-case examples produced stronger visible pressure signals under a fixed 0–1 color scale, without amplifying predictions. This does not establish higher accuracy or calibrated force. T-Rex was part of its training sources; those examples are not an independent-test generalization result.

It combines the Qwen3.5 vision tower, hand-pose-conditioned tactile queries and a temporal decoder. It has no Full-Qwen language reasoner, causal clip memory, OPD teacher, or independently trained contact-probability head. The separately released 0824 Full student remains the default model.

Property	0813 release
Inputs	One person's egocentric RGB video + WiLoR-estimated hand poses
Pressure	Left/right 21×21 fields, sensor-relative [0,1]
Contact state	Pressure ≥ 0.1, restricted to valid sensor cells
Temporal processing	Eight frames, four-frame overlap, Hann fusion; offline inference
Precision	Original checkpoint tensor dtypes; SDPA/BF16 inference
Checkpoint	951,378,331 bytes (about 951 MB)
Saved epoch	29
Training sources	EgoTouch, HOI4D, TACO, T-Rex

📦 Download

Checkpoint · Configuration · SHA256 manifest

hf download RammusLeo/TouchAnnotator-0813 model.pth config.yaml manifest.json \
  --revision v1.0 --local-dir checkpoints/touchannotator-0813

SHA256 of model.pth:

b64c1047fd9e22a91d146b0f4bd304718da8f9abcfdac9cec405e66a08af5226

The export contains 787 state entries, preserving the original tensor values and dtypes while removing optimizer, scheduler and scaler state. Shared module aliases remain intact. All exported tensors were compared with the original checkpoint; see export verification.

🚀 Quick start

Linux · NVIDIA CUDA GPU · Python 3.10. Clone the current source, including its 0813 adapter:

git clone https://github.com/RammusLeo/TouchAnnotator.git
cd TouchAnnotator
conda create -n touchannotator python=3.10 -y
conda activate touchannotator
python -m pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
python -m pip install -r requirements.txt
conda install -c conda-forge ffmpeg -y
mkdir -p third_party
git clone https://github.com/rolpotamias/WiLoR.git third_party/WiLoR
git -C third_party/WiLoR checkout fcb911312a38fa8badd30d9656a167485d61b8f9
python scripts/download_assets.py --variant 0813
python scripts/check_assets.py --variant 0813
CUDA_VISIBLE_DEVICES=0 python demo.py --variant 0813 \
  --video /path/to/bimanual_video.mp4 --output-dir outputs/my_video_0813

The downloader fetches and verifies this student, plus the pinned official Qwen and WiLoR assets, anonymously. The 951 MB figure is the student size; first-time setup also downloads the upstream assets. Existing assets are shared with the 0824 variant. MANO buffers embedded in the official WiLoR checkpoint are supported by the adapter; upstream component terms apply.

Optional mirror for downloads:

HF_ENDPOINT=https://hf-mirror.com python scripts/download_assets.py --variant 0813

Add --max-frames 50 for a short smoke test, or --no-video to skip MP4 rendering. Use a new output directory for each run.

📊 Output contract

The demo saves prediction.mp4, predictions.npz, wilor_poses.jsonl and metadata.json. For this variant:

import numpy as np
x = np.load('outputs/my_video_0813/predictions.npz')
pressure = x['pressure']          # (N,2,21,21), left/right
contact = x['contact_state']      # pressure >= 0.1 on valid cells
valid_cells = x['valid_cells']    # (2,21,21)
assert 'contact_probability' not in x.files

0813 does not provide an independent contact probability. The source's legacy decoder migration preserves the trained pressure output; its duplicate internal contact head is not exposed. metadata.json records contact_source: "pressure_threshold" and contact_probability_available: false. Complete schema and preprocessing.

Pressure is a sensor-relative estimate, not Newtons or kPa. Occlusion, gloves, missed detections and multiple people can reduce reliability. Inspect saved pose-validity flags. The demo is offline and keeps sampled frames in host RAM; split long recordings into manageable files.

🔬 Validation

16 regression tests passed (plus 2 subtests). A real bimanual 50-frame video completed WiLoR → strict checkpoint loading → pressure inference → 50-frame MP4 rendering. Finite [0,1] fields and pressure-derived contact were checked; probability fields are intentionally absent. These checks verify execution and output contracts, not all paper metrics or a completely empty-machine installation. See validation_0813.json.

Source run: 20260813_025550_qwen3.5_4b_tactile_prediction_qwen35_video_tactile_hoi4d_taco_trex.

🤝 License and acknowledgements

Project code retains its MIT license. The checkpoint contains Qwen-derived visual parameters; the upstream Qwen license notice is retained. WiLoR and MANO retain their respective terms. The code license does not relicense model weights or third-party assets; see third-party notices.