CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos
Paper β’ 2410.11831 β’ Published β’ 9
Jeff-Tracker
Track any point through a video.
Give it a video and a set of points; it returns where each point is on every frame, whether it can see it, and how sure it is. It runs at 0.0065 s/frame and returns a calibrated per-frame confidence alongside every track.
Code, benchmarks and method: https://github.com/antonyjaguar19-dotcom/Jeff-Tracker
It is LocoTrack-B with cross-track attention added β the tracks attend to each other, so a point that is hidden can be inferred from the neighbours that are still visible β then fine-tuned on MOVi-E for occlusion.
Access
This repository is gated. Click Request access; the maintainer approves manually. Once approved:
hf auth login
python tools/fetch_weights.py --all
Files
| file | what it is |
|---|---|
inf_s4000.ckpt |
the shipping checkpoint. Step 4000, occ_pos_weight=0.2, 66 MB |
The LocoTrack baseline weights are not mirrored here; they come from the LocoTrack authors'
own release, hamacojr/LocoTrack-pytorch-weights.
Use
import torch
tracker = torch.hub.load("antonyjaguar19-dotcom/Jeff-Tracker", "jefftracker").cuda()
# frames: (T, H, W, 3) uint8 BGR queries: (N, 3) as [frame, x, y]
tracks, vis, conf = tracker.track_queries_conf(frames, queries)
# tracks (T, N, 2) xy, y DOWN vis (T, N) bool conf (T, N) float
Keep conf. At 256Γ256 it detects the model's own >5 px frames at AUC 0.955, and
gating at 0.5 capped the worst sample in a whole run at 9 px.
Results
| Jeff-Tracker | |
|---|---|
| TAP-Vid DAVIS, 30 clips (AJ / Ξ΄_avg / OA) | 62.6 / 74.9 / 86.7 |
| Localisation, exact synthetic ground truth | 1.03 px @ 256Γ256 Β· 0.54 px @ 384Γ680 |
| Pixel locking | +0.0005 px β none, on an NCC-free path |
| Confidence as a bad-frame detector | AUC 0.955 @ 256Γ256 |
| Occluded frames within 5 px, vs the base | +3.4 points, held out over three benches |
| Speed | 0.0065 s/frame Β· 6.85 GB peak on 4K |
On DAVIS the fine-tune is level with its LocoTrack-B base (62.6 / 74.7 / 86.9). The checkpoint was trained for occlusion, so the relevant question is whether that cost anything general; it did not. The gain is the occlusion row, measured on three synthetic benches with exact ground truth built after the shipping checkpoint was selected.
The port is verified against the base model's published DAVIS strided figures β 67.7 / 79.5 / 89.8 against a published 67.8 / 79.6 / 89.9, within 0.1 on all three. Protocol and controls: BENCHMARK.md.
Limitations β read before deploying
It gaps an occlusion rather than crossing one. Only ~6% of ground-truth-occluded frames get a position at all. The base objective zeroes the position loss on occluded frames, so the base model was never taught to place a point it cannot see.
The occluded gain oscillates between +3.4 and 0 across checkpoints. The occluded term is roughly 1% of the position loss, too weak at batch size 1 to hold a solution. Step 4000 is a good draw from that wandering, not a plateau.
Use 256Γ256. At 384Γ680 the median improves to 0.697 px while the mean is 40.6 px, and no confidence threshold recovers it β at conf β₯ 0.99 the mean is still 39.3 px.
It is a seed-and-track stage, not a finished pipeline. The 1.03 px figure is raw neural output with no refinement.
Training
Fine-tuned from locotrack_base.ckpt on movi_e/256x256 from gs://kubric-public/tfds
(published by Google). Only the 5.8M cross-track parameters were trained; the
11.5M LocoTrack weights are bit-identical to the release. 4000 steps, ~2 s/step at 2.3 GB on
an RTX A4000.
License
Apache-2.0. See LICENSE and NOTICE, which must travel with any copy.
Cross-track attention and the occluded-position loss weighting are implemented from the description in Karaev et al., arXiv:2410.11831.
Citing
@inproceedings{cho2024locotrack,
title = {Local All-Pair Correspondence for Point Tracking},
author = {Cho, Seokju and Huang, Jiahui and Nam, Seungryong and
Min, Dongbo and Lee, Joon-Young},
booktitle = {ECCV},
year = {2024}
}
Inference Providers NEW
This model isn't deployed by any Inference Provider. π Ask for provider support