EgoTouch multimodal retrieval clean v3
All 18 completed P4 checkpoints for the controlled comparison of pose temporal
mean versus pose BiGRU across VT2P, VP2T and TP2V. Code and frozen configs:
kaichen-z/VLA-HAND@8acafb5.
Experiment matrix
3 tasks × 2 pose temporal encoders × 3 matched seeds = 18 runs
tasks: VT2P, VP2T, TP2V
pose temporal: mean, two-layer bidirectional GRU
seeds: 20260831, 20260901, 20260902
Every pair uses identical clean-atomic cache-v3 windows, tactile/visual weights, pose frame-MLP initialization, optimization budget and seed. Architecture- specific temporal and fusion heads are deterministically reset so that the comparison does not inherit a mean-pose checkpoint advantage.
Main results
Three-seed bidirectional retrieval mean mAP:
| Task | Pose temporal | Validation | Test seen | Test unseen |
|---|---|---|---|---|
| VT2P | mean | 0.1581 | 0.1482 | 0.0414 |
| VT2P | BiGRU | 0.2980 | 0.2884 | 0.1137 |
| VP2T | mean | 0.1242 | 0.1248 | 0.0213 |
| VP2T | BiGRU | 0.2218 | 0.2374 | 0.0648 |
| TP2V | mean | 0.1915 | 0.1840 | 0.0339 |
| TP2V | BiGRU | 0.1954 | 0.1951 | 0.0347 |
BiGRU materially improves VT2P and VP2T. TP2V gains are small relative to seed variation because the frozen visual target remains order-invariant temporal mean.
Files
Each run directory contains:
best.pt: model, optimizer, scheduler, epoch, step, config and selected metrics; suitable for evaluation or resuming training.resolved_config.json: exact run configuration.initialization_report.json: loaded/skipped checkpoint keys and controlled reset.heldout_metrics.json: validation, test-seen and test-unseen retrieval metrics.summary.json: runtime and best-validation summary.
Top-level aggregate.json contains all per-seed values, means, sample standard
deviations and paired BiGRU-minus-mean deltas. cache_manifest.json and
visual_manifest.json pin both input artifacts.
Data dependencies:
These are multimodal retrieval checkpoints, not a replacement for the standalone
P1 tactile encoder at
egotouch-tactile-encoder-p1-v1.