SAM3-HOIST — Stage 1 held-object head (weight deltas)

Held-object identification and segmentation on HOIST (HOIST-Former, CVPR 2024), built by fine-tuning SAM3-I with a new held-object interaction head.

These are deltas, not full checkpoints. Each file carries only the 121 tensors that training changed (~14 MB). The frozen SAM3 trunk — 3,337 tensors, 7.3 GB, byte-identical to the base — is not redistributed here, because those weights are gated by Meta. Bring your own SAM3-I base checkpoint and run merge_delta.py; the result is bit-identical to the original 8.1 GB checkpoint.

What is in this repo

Path	What it is
stage1/final.delta.pt	The 8-epoch Stage-1 fine-tune. This is row C, frame-level AP50 46.74
stage1/epoch_1.delta.pt … epoch_8.delta.pt	Per-epoch checkpoints of the same run
merge_delta.py, extract_delta.py	Reconstitute a full checkpoint from base + delta
SHA256SUMS	Checksums for the files in stage1/

final.delta.pt and epoch_8.delta.pt carry identical weights — the run finished at epoch 8, so the "final" checkpoint is that epoch. Their SHA256 differ only because each records which file it came from. Download either one.

The per-epoch files exist for a specific open question — see The 22-point anomaly below.

Usage

You need the SAM3-I base checkpoint first (gated; request access from the SAM3-I authors — see docs/02-checkpoints.md in the code repo).

pip install torch
huggingface-cli download ZaMinVo/SAM3-HOIST --local-dir sam3-hoist

python sam3-hoist/merge_delta.py \
  --base  /path/to/SAM3-I/checkpoint.pt \
  --delta sam3-hoist/stage1/final.delta.pt \
  --out   ./hoist_held_object_final.pt

merge_delta.py refuses to run if the base does not match the fingerprint the delta was built against — merging onto the wrong base produces a model that loads cleanly and scores nonsense, which is the failure worth guarding.

Then evaluate with the code repo's frame dumper:

python tools/eval_external/dump_sam3i_hoist_frames.py \
  --checkpoint_path ./hoist_held_object_final.pt \
  --stage interaction --resolution 1008 \
  --json  /path/to/HOIST/test.json \
  --image_root /path/to/HOIST/test/JPEGImages \
  --out rowC.jsonl
python tools/eval_external/score_hoist_frames.py --dump rowC.jsonl --out rowC.json

Results

Frame-level, HOIST test: 321 videos / 5,973 frames / 6,784 GT instances, 1008 px, max_dets=100.

	AP50	AP	AP75	AR_1	AR_10	AR_100
Stage 1 (this model)	46.74	31.46	35.78	37.39	61.98	71.20

Recall@k, IoU 0.5, threshold-free:

k	1	5	10	20	50	100
ceiling recall	52.11	76.21	83.49	88.81	93.29	95.36

95.36% of held objects are inside the 100-candidate set, but only 52.11% are ranked first — a 43.25-point gap that a perfect re-ranker could close without touching a single mask. A Stage-2 re-ranker was trained and is a clean negative: four different ranking losses span 0.013 AP50, and the best moves ceiling@1 by +0.06. If you want to improve this model, the evidence says give the ranker different inputs, not a different objective.

Reading these numbers honestly

• AP50, never AP. AP is COCO-averaged 0.50:0.95 and roughly half the value.
• Frame-level, track identity ignored. These are not the paper's spatio-temporal AP50 and do not belong in a Table 2 column. For reference, HOIST-Former scores 72.93 under this same frame-level metric, so this model is well behind it.
• Resolution changes the result. Everything here is at 1008 px; state the resolution next to any number you quote.
• The 22-point anomaly. An ablation model trained on 8,000 images for 4 epochs at batch 8 beat this run by 22.35 AP50 on identical frames, despite this run using 74,527 images for 8 epochs at effective batch 64. Treat 46.74 as the floor of one recipe, not the method's ceiling. The per-epoch deltas in stage1/ are published so this can be diagnosed without retraining: score epochs 1–8 and look for an early peak, which would point at the learning rate.

Architecture

The whole SAM3 vision trunk is frozen. The head cross-attends learned queries over the last three FPN levels to produce one hand prompt and eight interaction prompts — matching HOIST's one union hand mask plus up to eight contact regions — fused with the adapter1 text feature, then decoded by the frozen SAM3 decoder.

Of the 649 tensors the trainer marks trainable, exactly 121 actually change over 8 epochs: 73 new head tensors (14.0 MB) and 48 adapter tensors (0.2 MB). The remaining 528 adapter tensors come out byte-identical to where they started.

Licence

The deltas are released under CC-BY-NC-4.0. They are not usable alone — they require the SAM3-I base checkpoint, which carries its own licence and access terms that you must satisfy separately. Nothing in this repository redistributes Meta's SAM3 weights.

Citation

If you use this, please cite HOIST-Former (CVPR 2024) and SAM3, which this work builds on.