Track-On: Transformer-based Online Point Tracking with Memory
Authors: Görkay Aydemir
Xiongyi Cai
Weidi Xie
Fatma Guney
Venue: ICLR 2025 Poster (2025)
Keywords: Point Tracking
Online Point Tracking
Tracking Any Point
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Transformer-based model for online long-term point tracking, leveraging spatial and context memory to enable frame-by-frame tracking without access to future frames, achieving state-of-the-art performance across multiple datasets.

Abstract: In this paper, we consider the problem of long-term point tracking, which requires consistent identification of points across multiple frames in a video, despite changes in appearance, lighting, perspective, and occlusions. We target online tracking on a frame-by-frame basis, making it suitable for real-world, streaming scenarios. Specifically, we introduce Track-On, a simple transformer-based model designed for online long-term point tracking. Unlike prior methods that depend on full temporal modeling, our model processes video frames causally without access to future frames, leveraging two memory modules —spatial memory and context memory— to capture temporal information and maintain reliable point tracking over long time horizons. At inference time, it employs patch classification and refinement to identify correspondences and track points with high accuracy. Through extensive experiments, we demonstrate that Track-On sets a new state-of-the-art for online models and delivers superior or competitive results compared to offline approaches on seven datasets, including the TAP-Vid benchmark. Our method offers a robust and scalable solution for real-time tracking in diverse applications. Project page: https://kuis-ai.github.io/track_on

Published as a conference paper at ICLR 2025
TRACK-ON: TRANSFORMER-BASER ONLINE POINT TRACKING WITH MEMORY
Görkay Aydemir' Xiongyi Cai Weidi Xie Fatma Güiney 1.21 "Department of Computer Engineering, Koç University "KUIS AI Center
SSchool of Artificial Intelligence, Shanghai Jiao Tong University gaydemi 1230mu. edu.tr i denotes equal supervision
ABSTRACT
In this paper, we consider the problem of long-term point tracking, which requires consistent identification of points across multiple frames in a video, despite changes in appearance, lighting, perspective, and occlusions. We target online tracking on frame-by-frame basis, making it suitable for real-world, streaming scenarios. Specifically, we introduce Track-On, a simple cansformer-basee model designed for online long-term point tracking. Unlike prior methods that depend on full tem- poral modeling, our model processes video frames causally without access to future memory modules -Spatial memory and context memory- to capture temporal information and maintain reliable point tracking over long time horizons. At inference time, it employs patch classification and refinement identify correspondences and track points with high accuracy. Through extensive experiments, we demonstrate that Track-On sets new tate-of-the-as for online models and delivers superior competitive results compared to offline approaches on seven datasets, including the TAP-Vid benchmark. Our method offers a robust and scalable solution for real-time tracking in diverse applications."
frames, leveraging
INTRODUCTION
Motion estimation is one of the core challenges in computer vision, with applications spanning video compression (Jasinschi et al., 1998), video stabilization (Battiato et al., 2007; Lee et al., 2009), and augmented reality (Marchand et al., 2015). The objective is to track physical points across video frames accurately. A widely used solution for motion estimation is optical flow, which estimates pixel-level correspondences between adjacent frames. In principle, long-tern motion estimation can be achieved by chaining together these frame-by-frame estimations.
Recent advances in optical flow techniques, such PWC-Net (Sun et al., 2018) and RAFT (Teed & Deng, 2020), have improved accuracy for short-term motion estimation. However, the inherent limitations of chaining flow estimations remain challenge, namely error accumulation and the difficulty of handling occlusions. To address long-term motion estimation, Sand & Teller (2008) explicitly introduced the concept of pixel tracking, a paradigm shift that focuses on tracking individual points across a video, rather than relying solely on pairwise frame correspondences. This concept, often referred to as "particle video" has been revisited in recent deep learning methods like PIPs (Harley et al., 2022) and TAPIR (Doersch et al., 2023), which leverage dense cost volumes, iterative optimization, and learned appearance updates to track points through time.
Despite the advancements, the existing methods for long-term point tracking face two major limi- tations. First, they primarily rely on offline processing, where the entire video or a large window of frames is processed at once. This allows models to use both past and future frames to improve predictions but inherently limits their applicability in real-world scenarios (Karaev et al., 2024b; Harley et al., 2022). Second, these approaches struggle with scalability, they often require full attention computation across all frames, leading to significant memory overhead, especially for long videos or large frame windows. These limitations hinder their use in real-world applications, like robotics or augmented reality, where efficient and online processing of streaming video crucial.
Project page: https: :/ku::///uisss wit.ggttthu............... wtt.hb..c.trc...////

---

Published as a conference paper at ICLR 2025
Spatisl
Spacal
Spucal
Chosiiiatton
Clasifootion & Refaemment
Classifeation
Refinement
Refieement
Heratiie Updattes
Updates
Decoder
Devoder
Deoeder
Figure 1: Offfine vs. Online Point Tracking. We propose an online model, tracking points frame-by- frame (right), unlike the dominant offline paradigm where models require access to all frames within sliding window or the entire video (left). In contrast, our approach allows for frame-by-frame tracking in videos of any length. To capture temporal information, we introduce two memory modules: spatial memory, which tracks changes in the target point, and context memory, which stores broader contextual information from previous states of the point.
In this paper, we address the challenge of long-term point tracking an online processing setting (Fig. right), where the model processes video frames sequentially, without access to future frames (Vecerik et al., 2023). We propose a simple transformer- treated queries in the transformer decoder, attending the current frame to update their features. Unlike existing methods that aggregate temporal information across video frames, we achieve temporal continuity by updating the query representations with information from two specialized memory modules: spatial memory and context memory. This design enables the model to maintain reliable point tracking over time while avoiding the high computational and memory costs associated with full temporal modeling across entire video sequences.
model, where points of interest are
Specifically, spatial and context memory play distinct but complementary roles. The former aims reduce tracking drift by updating the query representation with information from the latest frames. This ensures that the query reflects the most recent visual appearance of the tracked point, by storing the content around the model's predictions in previous frames, rather than relying on features of the initial point. On the other hand, context memory provides a broader view of the track's history, storing the point's embeddings from past frames. This allows the model to consider changes to visual content including key information about the point's s status, such as whether the point was occluded in previous frames. Overall, spatial memory focuses positional changes in predictions over time while context memory ensures temporal continuity by providing a full perspective of the track's 's evolution. Together, these memory modules aggregate useful temporal information across video.
At training time, the queries from the transformer decoder identify the most likely location by computing embedding similarity with each patch, and are trained using similarity-based classification, akin to contrastive learning. The prediction is then refined by estimating an offset within the local region to find the final correspondence. We conduct extensive experiments demonstrating that our simple patch-classificttion and refinement approach serves as strong alternative to the dominant iterative update paradigm (Karaev et al., 2024b; Harley et al., 2022: Doersch et al., 2023). Our method sets new ifae-oo-thhe-rt among online models and either matches surpasses offline approaches on seven datasets including the TAP- Vid benchmark.
In summary, our contributions are as follows: (i) simple architecture that treats points of interest as queries in the transformer decoder, identifying correspondences through patch classification and refinement; (ii) memory modules that effectively store past content and address feature drift in an online manner, (iii) extensive experiments and ablations that demonstrate ofate-of-the-ar performance among online models and competitive results to offline models.
METHODOLOGY
2.1 PROBLEM SCENARIO
Given an RGB video of frames, V (I, I2, .... I RTXHxWxx and set of N predefined queries, Q = {(t p²), (12 p²), .... (L ",,) Nx3 where each query point is specified
2

---

Published as a conference paper at ICLR 2025
Query Decoder
Visibility Head
Re-ranking
MLP
q
C₁
p
patch
imis
Frame
fr
h
Visual Encoder
Offset
MLP
Head
Figure 2: Overview. We introduce Track-On, simple transformer-bast method for online, frame- by-frame point tracking. The process involves three steps: (i) Visual Encoder, which extracts features from the given frame: (ii) Query Decoder, which decodes interest point queries using the frame's features; (ii) Point Prediction (highlighted in light blue), where correspondences are estimated in coarse-to to- manner, first through patch classification based on similarity, then followed by refinement through offset prediction from few most likely patches, Note that the squares refer to point queries, while the circles represent predictions, either as point coordinates or visibility.
by the start time and pixel's spatial location, our goal is to predict the correspondences P. E R and visibility V. {0, 13Aa for all query points in an online manner, i.e. using only frames up the current target frame t. To address this problem, we propose a transformer-base point tracking model, that tracks points frame-by-frame with dynamic memories M to propagate temporal information along the video sequence:
Dr. V,, M. = + (I,, Q M.- θ)
In the following sections, we start by describing the basic transformer architecture for point tracking in Sec. 2.2, then introduce the two memory modules and their update mechanisms in Sec. 2.3.
2.2 TRACK-ON: POINT TRACKING WITH A TRANSFORMER
Our model based on transformer, consisting of three components, as illustrated in Fig. Visual Encoder is tasked to extract visual features of the video frame, and initialize the query points; Query Decoder enables the queried points to attend the target frame to update their features; and Point Prediction, to predict the positions of corresponding queried points in a coarse-to-fine (Doersch et al., 2023) manner.
2.2.1 VISUAL ENCODER
We adopt a Vision Transformer (ViT) as visual backbone, specifically, DINOv2 (Oquab et al., 2024), and ViT-Adapter (Chen et al., 2022b) obtain dense features at higher resolution than the standard ViT. We then add learnable spatial positional embeddings 7 to the frame-wise features:
RRE #
f 4 Piss-oon (L.) +
where denotes the feature dimension, and S refers to the stride. We use a single-scale feature map from ViT-Adapter for memory efficiency, specifically with stride of S 4.
Query Initialization: To initialize the query features (qinit), we apply bilinear sampling to the feature map the query position (p*):
niit {sample(ffr
W') NXD
In practice, we initialize the query based on the features the start frame t for i-th query, assuming they can start from different time point, and propagate them the subsequent frames.
3

---

Published as a conference paper at ICLR 2025
top
2
Decoder
Cifec
top
bi
Deformable Decoder qi
Figure 3: Top-k Points. In certain cases, a patch with high similarity, though not the most similar, is closer to the ground-truth patch. The top-3 patch centers, ranked by similarity, are marked with dots, while the ground- truth represented by diamond.
Figure 4: Ranking Module. The features around the top-k points (p: with the highest similarity are decoded using deformable attention extract the corresponding top-k fea- tures (P) These features then fused with the decoded query q using a transformer decoder.
2.2.2 QUERY DECODER
After extracting the visual features for the frame and query points, we adopt a variant transformer decoder (Vaswani et al., 2017), with blocks, i.e. attentintntiion followed by self-attention, with an additional feed forward layer between attentions:
dec = 2 q-dee (q
f.) E RN
(3)
The points of interest are treated as queries, which update their features by iteratively attending to visual features of the current frame with cross attention. These updated queries are then used to search for the best match within the current frame, as explained in the following section.
2.2.3 POINT PREDICTION
Unlike previous work that regresses the exact location of the points, we formulate the tracking as matching problem to one of the patches, that provides coarse estimate of the correspondence. For exact correspondence with higher precision, we further predict offsets to the patch center. Additionally, we also infer the visibility V, E [0, 1] and uncertainty ú, E [0, 1] for the points of interest.
Patch Classification: We first pass the visual features into 4-yer MLPs, and downsample the resulting features into multiple scales, i.e., f; + h, E R W XD + h R x xD We compute the cosine similarity between the decoded queries and patch embeddings in four scales, and the similarity map Cffcs obtained as the weighted average of multi-scale similarity maps with learned coefficients (details in Appendix Sec. B.2). We then apply temperature scale the similarity map and take softmax spatially over the patches within the current frame. The resulting Odee provides measure of similarity for each query across the patches in the frame.
We train the model with a classification objective, where the ground-truth class is the patch with the point of interest in it. In other words, perform P-elass classification, patches in the frame.
the total number of
Re-ranking:
We observed that the true target patch might not always have the highest similarity on Cffee, however, it is usually among the top-k patches. For example, in Fig. 3, the patch with the second-highest similarity (top-2) closer to the true correspondence than the most similar patch (top-1). l). To rectify such cases, we introduce re-ranking module rank
q T re rank (gee he; Oilre) E ERNDD
where q, denotes the refined queries after ranking.
In the re-ranking module (Fig. 4), we identify the top-k patches with the highest similarities and retrieve their corresponding features with deformable attention decoder. Then, integrate them
4

---

Published as a conference paper at ICLR 2025
20
Prize
Figure 5: Offset Head. Starting with a rough estimation from patch classification (left), where lighter colors indicate higher correlation, we refine the prediction using the offset head (right). The selected patch center and the final prediction are marked by a blue dot and a red dot, respectively, with the ground-truth represented by diamond.
Figure 6: Feature Drift. For the tracks shown below (start, middle, and final frames), the plot above illustrates the decreasing similarity be- tween the features of the initial query and its correspondences over time, with the initial sim- ilarity indicated by horizontal dashed lines.
into the original query features via transformer decoder to produce refined queries. Using these refined queries, we calculate the final similarity map C, and apply a classification loss. Finally, we select the center of the patch with the highest similarity (p,utch E N22) our coarse prediction. Additionally, we compute an uncertainty score for each top-k location, iee top E R-N by processing their corresponding features with a linear layer (see Appendix Sec. B.2 for details).
Offset Prediction:
For the exact correspondence (p: E RN2),, we further predict an offset 6, RR Nx2 to the patch center by incorporating features from the local region around the inferred patch, shown in Fig. 5;
6, 4 Poff(q (qts he, p pratch))
p: = p ppatch +
(5)
Here, Poi is deformable transformer decoder (Zhu al., 2021) block with 3 layers, excluding self-attention. In this decoder, the query qe is processed using the key-value pairs h, with the reference point set to p. patch To limit the refinement to the local region, the offsets are constrained by the S (stride) and mapped to the range [, S] using a tanh activation.
In addition, we predict the visibility v, and uncertainty ú, using visibility head Pvis. We first decode the region around the predicted location p. (Eq. 5) using a deformable decoder layer. Then, we predict visibility and uncertainty by applying a linear layer to the decoded queries. At training time, we define prediction be uncertain if the prediction error exceeds threshold (6,, pixels) or if the point is occluded. During inference, we classify a point as visible if its probability exceeds threshold 8g. Although we do directly utilize uncertainty in our predictions during inference, we found predicting uncertainty to be beneficial for training.
Training: We train our model using the ground-truth trajectories Pt E N and visibility information E (0, 1) For patch classification, we apply cross-entropy loss based the ground-trutt class, patch patcc For offset prediction o,, we minimize the & distance between the predicted offset and the actual offset. We supervise the visibility V, and uncertainty ú, using binary cross-entropy loss. Additionally, supervise the uncertainties of the top-k points, ú, at re-ranking. The total loss is weighted combination of them:
top
L (Cer (Cr, caatch) + LCF (Cree, chatch))
Patch Classification Loss
Le, (ô., o₂) Vv LCE(Vi, + LCE(Ur uz)
+ (c .. top
Offsset Loss
Visibility Loss
Uncertainty Loss Top- Uncertainty Loss
Till this point, model has exclusively considered relocating the queried points within the current frame. However, as the appearance of points consequently changes over time, the embedding similarity between the initial query point and future correspondences tends to decrease
Discussion:
5

---

Published as a conference paper at ICLR 2025
1-3
q
q
4r
Seuid
9++1
M
Query Decoder
Query Decoder
Query Decoder
irsie
4:
fnit
Query
Update
M
M
M;
Spatial Memory
Context Memory
Figure 7: Memory Modules. Spatial memory M;-1 (left) is used to update the initial query q'
from the first frame to gasi on the current frame. The goal is to resolve feature drift by storing the content around the model's predictions previous frames. Context memory M (right) is input to the query decoder which updates qr to q:. It provides broader view of the track's s history with appearance changes and occlusion status by storing the point's embeddings from past frames.
gradually (Fig. 6). This problem, known as feature drift, leads inaccurate predictions, when solely relying on the feature similarity with the initial point.
2.3 TRACK-ON WITH MEMORY
Here, we introduce two types of memories: spatial memory and context memory. Spatial memory stores information around the predicted locations, allowing us to update the initial queries based on the latest predictions. Context memory preserves the track's s history states by storing previously decoded queries, ensuring continuity over time and preventing inconsistencies. This design enables our model to effectively capture temporal progressions in long-term videos, while also adapting to changes in the target's 's features to address feature drift.
We store the past features for each of the N queries independently, with up K embeddings per query each memory module. Once fully filled, the earliest entry from the memory will be obsoleted new entry arrives, operating First-In First-Out (FIFO) queue.
2.3. SPATIAL MEMORY
Here, we introduce the spatial memory module that stores fine- fine-grained local information from previous frames, enabling continuous updates to the initial query points. This adaptation to appearance changes helps mitigate feature drift.
Memory Construction:
We zero-initialize the memory, M update its content with each frame. For the first frame, we make prediction using initial query q gnst without memory.
Memory Write (Rqww): To update the memory with the new prediction, M M we extract feature vector around the predicted point P. on the current feature map f,, and add the memory:
M [M g (Iq init qt), f,, p.)]
& is a 3-layer deformable transformer decoder without self-attention, using the concatenated q inst and qi as the query, attending a local neighborhood of predicted point for update. Utilizing deformable attention for the local summarization process helps prevent error propagation over time, as the query can flexibly select relevant features from any range.
Query Update ( In such scenario, before passing into the query decoder to estimate the correspondence, the initial query points first visit the spatial memory M; for an update:
q
init Dg Spup( (gnnt M
inst + dagm (q Qm (M 70))
8

---

Published as a conference paper at ICLR 2025
Pmm is transformer encoder layer that captures dependencies within the memory, and Pggm is transformer decoder layer without initial self-attention, where qunst attends to updated memory, followed by a linear layer, and y E I K is learnable position embeddings. Instead of sequentially updating the query embeddings at each time step, e.g. extracting qi using q we update them with respect to the initial query q conditioned on all previous predictions stored in the memory. This prevents error propagation by taking into account the entire history of predictions.
2.3.2 CONTEXT MEMORY
In addition to spatial memory, we introduce a context memory that incorporates historical information of the queried points from broader context, enabling the model to capture past occlusions and visual changes. Specifically, we store the decoded query features from previous time steps in context memory, M We then integrate it by extending the query decoder with additional transformer decoder layer without self-attention, where queries attend to memory with added learnable position embeddings (7 E RKxD):
B (qinit f;,
Changes to the query decoder with memory are shown in red. For the writing operation, we add the most recent q: M and remove the oldest item, following the same procedure in the spatial memory. Our experiments demonstrate that incorporating past content temporally with context memory enables more consistent tracking with additional benefits over spatial memory, especially in visibility prediction, since spatial memory focuses only on the regional content where the point is currently visible.
2.3.3 INFERENCE-TING TIME MEMORY EXTENSION
Although the memory size K fixed training time, the number video frames at inference can be different from the training frame limit. To address this, we extend the memory size during inference by linearly interpolating the temporal positional embeddings, 7t and 7 larger size K;, In particular, we train our model with memory size K = 12, and extend it to K; E {16.... 96) at inference time.
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Datasets:
We use TAP-Vid (Doersch et al., 2022) for both training and evaluation, consistent with previous work. Specifically, we train our model on TAP-Vid Kubric, a synthetic dataset of 1 llk video sequences, each with fixed length of 24 frames. For evaluation, we use three other datasets from the TiP-Vid benchmark: ViP-Vid DAVIS, which includes 30 real-world videos from the DAVIS dataset; TAP- Vid RGB-Stacking synthetic dataset of 50 videos focused tasks, mainly involving textureless objects: TAP- Vid Kinetics, a collection of over 1,000 real-world online videos. We provide comparisons on four additional datasets in Appendix Sec. C.
robotic manipulation
Metrics: We evaluate tracking performance with the following metrics of TAP-Vid benchmark: Occlusion Accuracy (OA), which measures the accuracy of visibility prediction; 5ā the average proportion of visible points tracked within 1, 2, 4, 8, and 16 pixels; Average Jaccard (AJ), which jointly assesses visibility and localization precision.
Evaluation Details: We follow the standard protocol of TAP-Vid benchmark by first downsampling the videos to 256 x 256. We evaluate models the queried first protocol, which is the natural setting for causal tracking. In this mode, the first visible point in each trajectory serves as the query. and the goal is to track that point in subsequent frames. For DAVIS evaluation, we set the memory size K; to 48, 80, and 96 for DAVIS, RGB- RGB-Stacking, and Kinetics, to accommodate a larger temporal span.
3.2 RESULTS
As shown Table 1, we categorize models into online and offline settings. Offline models, with bidirectional information flow, use either fixed-size whed half of the window spans past
7

---

Published as a conference paper at ICLR 2025
Table I: Quantitative Results on TAP-Vid Benchmark. This table shows results in comparison to the previous work on TAP- Vid under queried first setting, in terms of AJ, 5argr and OA. The models are categorized into online and offline schemes, the former setting grants access any frame regardless of video length, thus providing a clear advantage. While online models process one frame time, enable frame-by-frame inference. For training datasets, Kub and Kub-L(ong), refer to the TAP-Vid Kubric dataset with 24-frame and 64-frame videos, respectively; and R indicates the inclusion of a large number of real-world videos, we highlight these models in gray MFT is a long-
term optical flow method trained on combination of Sintel (Butler et al., 2012), FlyingThings (Mayer al., 2016), and Kubric datasets.
DAVIS
RGB LGBB-Stckkin
Kinetics
Model
Input
Train
AJ 1
OA
AJ
OA
AJ
8avg
OA 1
Offline
TAPIR
Video
Kub
56.2
70.0
86.5
55.5
69.7
88.0
49.6
64.2
85.0
TAPTR
Window
Kub
63.0
76.1
91.1
60.8
76.2
87.0
49.0
64.4
85.2
TAPTRv2
Window
Kub
63.5
75.9
91.4
53.4
70.5
81.2
49.7
64.2
85.7
SpatialTracker
Window
Kub
61.1
76.3
89.5
63.5
77.6
88.2
50.1
65.9
86.9
LocoTrrack
Video
Kub
62.9
75.3
87.2
69.7
83.2
89.5
52.9
66.8
85.3
CoTracker3
Window
Kub-I.
64.5
76.7
89.7
71.1
81.9
90.3
54.1
66.6
87.1
CoTracker3
Video
Kub-I.
63.3
76.2
88.0
74.0
84.9
90.5
53.5
66.5
86.4
BooIsTAPIR
Video
61.4
73.6
88.7
70.8
83.0
89.9
54.6
68.4
86.5
CrTakker3
Window
Kub-L
63.8
71.7
83.6
91.1
55.8
88.3
TrTracker3
Video
Kub-L
64.4
76.9
91.2
74.3
85.2
92.4
54.7
67.8
87.4
Online
DynOMo
Frame
45.8
63.1
81.1
MFT
Frame
SFK
47.3
77.8
39.6
60.4
72.7
Online TAPIR
Frame
Kub
56.7
70.2
85.7
67.7
51.5
64.4
85.2
Track-On (Ours)
Frame
Kub
65.0
78.0
90.8
71.4
85.2
91.7
53.9
67.3
87.8
frames and the other half future frames or the entire video, granting access to any frame regardless of video length and providing a clear advantage. In contrast, online models process one frame at time, enabling frame-by- by-rame inference. In the following discussion, we mainly focus on the setting using similar training set to ours, i.e., the models without using real-world videos.
Comparison on DAVIS. Our model outperforms all existing online models across every evaluation metric, achieving an 8.3 AJ improvement over the closest competitor, Online TAPIR. Additionally, it surpasses all offline models in both AJ (65.0 vs. 64.5) and 8 (78.0 vs. 76.7), outperforming even the concurrent CrTakcer,,, which was trained on longer videos (24 vs. 64 frames). Notably, our model also outperforms models fine-tuned on real-world videos by a significant margin. These results particularly impressive because model is an online approach, processing the video frame by frame, yet it exceeds the performance of offline models that process the entire video at once.
Comparison RGB-Stacking. The dataset consists of long video sequences, with lengths of up to 250 frames, making it ideal for evaluating models' long-term processing capabilities. Our model surpasses Online TAPIR by 3.7 AJ and outperforms offline competitors, achieving improvements of 0.3 in a and 1.2 in OA compared to CoTracker3, which utilizes video-level input. The results of offline models on this dataset highlight a significant limitation the windowed inference approach, which struggles with long video sequences due to restricted temporal coverage. In contrast, models with full video input perform considerably better. By effectively extending the temporal span through our memory mechanisms, our model achieves comparable or superior performance on long videos using only frame-by-frame inputs, despite the inherent disadvantage of not having bidirectional connections across the entire video sequence.
Comparison on Kineties. The dataset comprises a variety of long internet videos. Our model outperforms Online TAPIR across all metrics by considerable margin, while also surpassing offline models in 8% and OA. Specifically, it achieves a 0.5 improvement over LocoTrack (with video inputs) and 0.7 improvement in OA over CoTracker3 (with window inputs). Despite the significant difference in training data between CoTracker3 and our model, ours ranks second in AJ, with only small gap of 0.2. Additionally, models fine-tuned real-world data demonstrate superior
8

---

Published as a conference paper at ICLR 2025
Table 2: Model Components. Removing individ- ual components of our model without (inference- time memory extension) -nnelly, the re-ranking module (Prnk) offset head (far), and visibility head (Pvis)
Table 3: Memory Components. The effect of spatial memory (M"), context memory (M*), and inference- time memory extension (IME). All metrics are higher-is-better.
one at a time. All metrics are higher-is-better.
Model
M
M
IME
AJ
8
OA
Model
82p
836p2
Gang
OA
A
K
X
52.0
67.6
78.1
Full Model (without IME)
45.5
95.9
64,9
77.7
90.6
B
V
63.5
77.0
89.0
No re-ranking (onk)))
43.8
95.5
62.8
76.3
89.7
C
64.3
77.8
90.3
No offset head (Por)
27.6
96.1
60.1
73.0
90.5
D
V
J
64.9
77.7
90.6
No visibility head (Pvis)
45.4
96.1
64.0
77.4
90.6
E
V
V
65.0
78.0
90.8
performance, underscoring the potential benefits of training on large-scale real-world datasets, which seem particularly advantageous for datasets like Kinetics compared to others.
3.3
ABLATION STUDY
Components: We conducted an experiment to examine the impact of each proposed component in the correspondence estimation section (Sec. 2.2), we remove them one at a time while keeping other modules unchanged. First, we removed the re-ranking module Prank. Second, we removed the offset head ooft eliminating the calculation of additional offsets. Instead, we used the coarse prediction, tee the selected patch center, as the final prediction. A more detailed analysis of the offset head is provided in Appendix Sec. D. Lastly, we replaced the additional deformable attention layer in the visibility head Pis with a 2-layer MLP. Note that, we do not apply inference-ting time memory extension to models in this comparison.
From the results in Table 2, we can make the following observations: (i) The re-ranking module improves all metrics, notably increasing AJ by 2.1, as it introduces specialized queries for identifying correspondences. Errors larger than 16 pixels are also more frequent without it, showing its role in reducing large errors. (ii) The offset head is erucial for fine-grained predictions. While 866p values remain similar without the offset head, lower error thresholds (i.e. less than 1 pixel) show a significant difference (45.5 vs. 27.6), highlighting the importance of predicted offsets for fine- ane-grained localization. (iii) Replacing the deformable attention layer in &vis with an MIP does not affect OA but reduces AJ. The deformable head ensures more consistent visibility predictions by conditioning them on accurate point predictions, leading to higher AJ. Despite this, OA remains robust even when an MLP is used for visibility prediction.
To demonstrate the effectiveness of our proposed memory modules, we conduct an ablation study, as shown Table 3. We start by evaluating the model without memory (Model-A), which corresponds to the vanilla model described in Sec. 2.2. As expected, due to the model's lack of temporal processing, Model-A performs poorly, particularly in OA. Introducing temporal information through either spatial memory (Model-B) context memory (Model-C) leads to significant performance improvements. Model-C, in particular, achieves higher OA by providing a more comprehensive view of the track's history, including occlusions. Combining both memory types (Model-D) further boosts performance, highlighting the complementary strengths of the two memory modules. Lastly, incorporating the memory extension at inference time yields slight improvements in all metrics, leading to an overall enhancement in performance. We provide more detailed analysis on spatial memory in Appendix Sec. E.
Memory Modules:
We plot the inference speed (frames per second, FPS), maximum GPU memory usage during video processing, and AJ performance on the Vip-Vid DAVIS dataset as a function of memory size K; (indicated near the plot nodes) in Fig. 8. The results are based on tracking approximately 400 points on a single NVIDIA A00 GPU. Unlike offline methods, our approach does not utilize temporal parallelization in the visual encoder, processing frames sequentially in online setting. As the memory size K increases, the model's 's inference speed decreases due to the higher computational cost of temporal attention in memory operations, correspondingly increasing GPU memory usage. For instance, the FPS decreases from 19.2 with K = 12 to 16.8 with K 48, and further down to 14.1 with K 96.
Efficiency:

---

Published as a conference paper at ICLR 2025
Additionally, model demonstrates high memory efficiency, with GPU memory usage ranging from 0.61 GB (K = 12) to a maximum of 1.03 GB (K 96). At the default memory size of K = 48, where our model per- forms best on this dataset, it achieves 16.8 FPS with maximum GPU mem- ory usage of 0.73 GB. This highlights the efficiency of our frame-hy-frame tracking approach, making well- suited for consumer GPUs and real- time applications. Moreover, we ob- serve that performance improves the memory size increases up to K 48, but declines beyond this point. This suggests that excessively large memory sizes hurt performance by storing unnecessary information. Additional analysis of memory size is provided in Appendix Sec. D.
16
65.0
64.9
N ITAP DAVISS
32
48
64.8
16
64.7
64
15
64.6
80
96
14
0.7
0.9
1.0
Max GPU Memory (in CB)
Figure 8: Efficiency. Inference speed (frames per second, FPS) vs. maximum GPU memory usage (in GB) where color represents the performance in AJ for different memory sizes (indicated near the nodes), while tracking approximately 400 points on the DAVIS dataset.
4 RELATED WORK
Tracking Any Point: Point tracking presents significant challenges, particularly in long-term scenarios involving occlusions and appearance changes. Early methods like PIPs (Harley et al., 2022) relied iterative updates, while TAPIR (Doersch al., 2023) focused on refining initialization and improving temporal accuracy. CoTracker (Karaev et al., 2024b) leveraged spatial correlations to jointly track multiple points, and TAPTR (Li et al., 2024b) adopted DETR-inspired design for tracking. More recent approaches, such as LocoTrack (Cho et al., 2024) and CoTracker3 (Karaev et al., 2024a), introduced region-to-region similarity for enhanced matching and utilized pseudo-labeled labeled data to boost performance. However, most of these methods operate offline, requiring access to entire video frames or fixed windows. contrast, our approach focuses on online tracking, employing memory modules to effectively capture temporal information. Additionally, diverge from regression-based iterative updates, instead adopting a patch classification and refinement paradigm.
Causal Processing in Videos: Online models process frames sequentially, without access to future frames, making them well-suited for streaming and real-time tasks (Xu et al., 2021; Zhou et al., 2024). This has been explored in tasks like pose estimation (Nie al., 2019), action detection (Wang et al., 2021), ),, and video segmentation (Cheng & Schwing, 2022). To enhance efficiency, approaches such as XMem (Cheng & Schwing, 2022) and LSTR (Xu et al., 2021) incorporate memory modules to balance long-term and short-term contexts. Similarly, we employ an attention- based memory mechanism tailored for point tracking, with spatial and contextual memories for capturing both local and global information.
CONCLUSION & LIMITATION
In this work, we presented Track-On a simple yet effective transformer-based model for online point tracking. To establish correspondences, our model employs patch classification, followed by further refinement with offset prediction. We proposed two memory modules that enable temporal continuity efficiently while processing long videos. Our model significantly advances the state-of-the-ard in online point tracking with fast inference and narrows the performance gap between online and offline models across a variety of public datasets.
Despite the strengths of our proposed model, there remain certain limitations. Specifically, the model may suffer from precision loss on thin surfaces and struggle to distinguish between instances with similar appearances, observed in failure cases (see Appendix). Future work could address these challenges by exploring learnable upsampling techniques to achieve righer-resolution feature maps and improve feature sampling accuracy.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENTS
We would like to thank Shadi Hamdan and Merve Rabia Barın for their remarks and assistance. This project funded by the European Union (ERC, ENSURE, 101116486) with additional compute support from Leonardo Booster (EuroHPC Joint Undertaking, opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. Weidi Xie would like to acknowledge the National Key R&D Program of China (No. 20222D0161400).
Views and
REFERENCES
Görkay Aydemir, Weidi Xie, and Fatma Güney, Can visual foundation models achieve long-term point tracking? In Proc. of the European Conf. on Computer Vision (ECCV) Workshops, 2024.
Ivana Balazevic, Yuge Shi, Pinelopi Papalampidi. Rahma Chaabouni, Skanda Koppula, and Olivier J Henaff. Memory consolidation enables long-context video understanding. In Proc. of the Interna- tional Conf. on Machine learning (ICML), 2024.
Sebastiano Battiato, Giovanni Gallo, Giovanni Puglisi, and Salvatore Scellato. SIFT features tracking for video stabilization. In Proc. of the International Conference on Image Analysis and Processing (ICIAP), 2007.
Benjamin Biggs, Thomas Roddick, Andrew Fitzgibbon, and Roberto Cipolla. Creatures great and SMAL: Recovering the shape and motion of animals from video. In Proc. of the Asian Conf. on Computer Vision (ACCV), 2019.
Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles. SST: Single-stream temporal action proposals. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
Daniel Butler, Jonas Wulff, Garrett B Stanley, and Michael Black. A naturalistic open source movie for optical flow evaluation. In Proc. of the European Conf. on Computer Vision (ECCV), 2012.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proc. of the European Conf. on Computer Vision (ECCV), 2020.
Junwen Chen, Gaurav Mittal, Ye Yu, Yu Kong, and Mei Chen. GateHUB: Gated history unit with background suppression for online action detection. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022a.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In Proc. of the International Conf. on Learning Representations (ICLR), 2022b.
Ho Kei Cheng and Alexander G Schwing, XMem: Long-term video object segmentation with an atkinson-shiffrin memory model. In Proc. of the European Conf. on Computer Vision (ECCV), 2022.
Seokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Seungryong Kim, and Joon- Young Lee. Local all-pair correspondence for point tracking. In Proc. of the European Conf. on Computer Vision (ECCV), 2024.
Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snoek, and Tinne Tuytelaars. Online action detection. In Proc. of the European Conf. on Computer Vision (ECCV), 2016.
Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-Vid, benchmark for tracking any point in video. In Advances in Neural Information Processing Systems (NeurIPS), 2022.

---

Published as a conference paper at ICLR 2025
Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. TAPIR: Tracking any point with per-frame initialization and temporal refinement. In Proc. of the IEEE International Conf. Computer Vision (ICCV), 2023.
Carl Doersch, Pauline Luc, Yi Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ignacio Rocco, Ross Goroshin, João Carreira, and Andrew Zisserman. BootsTAP: Bootstrapped training for tracking-any-Po ann-nntt Proc. of the Asian Conf. on Computer Vision (ACCV), 2024.
Hyunjun Eun, Jinyoung Moon, Jongyoul Park, Chanho Jung, and Changick Kim. Learning to discriminate information for online action detection. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
Zhipeng Fan, Jun Liu, and Yao Wang. Motion adaptive pose estimation from compressed videos. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021.
Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In Proc. of the European Conf. on Computer Vision (ECCV), 2022.
Anfeng He, Chong Luo, Xinmei Tian, and Wenjun Zeng. A twofold siamese network for real-time object tracking. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
Radu S Jasinschi, T Na Veen, et al. Motion estimation methods for video compression a review. Journal of the Franklin Institute, 1998.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. DynamicStereo: Consistent dynamic depth from stereo videos. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2023.
Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker3: Simpler and better point tracking by beeud-labelling real videos. arXiv preprint arXiv:2410.11 11831, 2024a.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTrracke: It is better to track together. In Proc. of the European Conf. on Computer Vision (ECCV), 2024b.
Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Mo Mobile video networks for efficient video recognition. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
Markus Käppeler, Kürsat Petek, Niclas Vödisch, Wolfram Burgard, and Abhinav Valada. Few-shot panoptic segmentation with foundation models. In Proc. IEEE International Conf. Robotics and Automation (ICRA), 2024.
Ken-Yi Lee, Yung-Yu Yu Chuang, Bing-Yu Yu Chen, and Ming Ouhyoung. Video stabilization using robust feature trajectories. In Pruc. of the IEEE International Conf. on Computer Vision (ICCV), 2009.
Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Feng Li, Tianhe Ren, Bohan Li, and Lei Zhang. TAPTRv2: Attention-based position update improves tracking any point. Advances in Neural Information Processing Systems (NeurIPS), 2024a.
Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, and Lei Zhang. TAPTR:
Tracking any point with transformers as detection. In Proc. of the European Conf. on Computer Vision (ECCV), 2024b.
Yongqing Liang, Xin Li, Navid Jafari, and Jim Chen. Video object segmentation with adaptive feature bank and uncertain region refinement. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proc. of the Interna- fional Conf. Learning Representations (ICLR), 2019.
12

---

Published as a conference paper at ICLR 2025
Eric Marchand, Hideaki Uchiyama, and Fabien Spindler. Pose estimation for augmented reality: hands-on survey. In IEEE Trans. on Visualization and Computer Graphics (VCG), 2015.
Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016.
Michal Neoral, Jonáš Serych, and Jiří Matas. MFT: Long-term tracking of every pixel. In Proc. of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2024.
Xuecheng Nie, Yuncheng Li, Linjie Luo, Ning Zhang, and Jiashi Feng. Dynamic kernel distillation for efficient pose estimation in videos. In Proc. of the IEEE International Conf. Computer Vision (ICCV), 2019.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wet Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nico- las Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions Machine Learning Research (TMLLR), 2024. ISSN 2835-8856.
Peter Sand and Seth Teller. Particle video: International Journal of Computer Vision (ICV), 2008.
Long-range motion estimation using point trajectories. In
Jenny Seidenschwarz, Qunjie Zhou, Bardienus Duisterhof, Deva Ramanan, and Laura cax-Ta DynOMo: Online point tracking by dynamic online monocular gaussian reconstruction. In Proc. of the International Conf. on 3D Vision (3DV), 2025.
Gurkirt Singh, Suman Saha, Michael Sapienza, Philip HS Torr, and Fabio Cuzzolin. Online real-time multiple spatiotemporal action localisation and prediction. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017.
Deqing Sun, Xiaodong Yang, Ming-Yo Yu Liu, and Jan Kautz. PWC-Net: Cnns for optical flow using pyramid, warping, and cost volume. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2018
Zachary Teed and Jia Deng. RAFT: Recurrent all all-pairs field transforms for optical flow. In Proc. of the European Conf. on Computer Vision (ECCV), 2020.
Narek Tumanyan, Assaf Singer. Shai Bagon, and Tali Dekel. DINO-Tracker: Taming DINO for self-supervised point tracking in a single video. In Proc. of the European Conf. on Computer Vision (ECCV), 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Mel Vecerik, Carl Doersch, Yi Yang, Todor Davchev, Yusuf Aytar, Guangyao Zhou, Raia Hadsell, Lourdes Agapito, and Jon Scholz. RoboTAP: Tracking arbitrary points for few-shot visual imitation. In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2023.
Qianqian Wang, Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2023.
Xiang Wang, Shiwei Zhang, Zhiwu Qing, Yuanjie Shao, Zhengrong Zuo, Changxin Gao, and Nong Sang. OadTR: Online action detection with transformers. In Proc. of the IEEE International Conf. Computer Vision (ICCV) 2021.
Zhongdao Wang, Liang Zheng, Yixuan Liu, Yali Li, and Shengjin Wang. Towards real-time multi- object tracking. In Proc. of the European Conf. on Computer Vision (ECCV), 2020.
13

---

Published as a conference paper at ICLR 2025
Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. SpatialTracker: Tracking any 2D pixels in 3D space. In Proc. IEEE Conf. on Computer Vision and Pattem Recognition (CVPR), 2024.
Mingze Xu, Mingfei Gao, Yi-Ting Chen, Larry S Davis, and David J Crandall. Temporal recurrent networks for online action detection. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Xia, Zhuowen Tu, and Stefano Soatto. Long short-term transformer for online action detection. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Jinrong Yang, Songtao Liu, Zeming Li, Xiaoping Li, and Jian Sun. Real-time object detection for streaming perception. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022a.
Le Yang, Junwei Han, and Dingwen Zhang. Colar: Effective and efficient online action detection by consulting exemplars. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022b.
Yu Yao, Mingze Xu, Yuchen Wang, David J Crandall, and Ella M Atkins. Unsupervised traffic accident detection in first-person videos. In Proc. IEEE International Conf. on Intelligent Robots and Systems (IROS), 2019.
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. memory-basee real-time understanding for long video streams. arXiv preprint arXiv:2406.0800 08085, 2024.
Zhu Zhang, Chang Zhou, Jianxin Ma, Zhijie Lin, Jingren Zhou, Hongxia Yang, and Zhou 7hao. Learning to rehearse in long sequence memorization. In Proc. of the International Conf. on Machine learning (ICML), 2021.
Yue Zhao and Philipp Krähenbühh.. Real-time online video detection with temporal smoothing transformers. In Proc. of the European Conf. on Computer Vision (ECCV), 2022.
Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wetzstein, and Leonidas Guibas. PointOdyssey: A large-scale synthetic dataset for long-term point tracking. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2023.
Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2024.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable transformers for end-to-end object detection. In Proc. of the International Conf. on Learning Representations (ICLR), 2021.
14

---

Published as a conference paper at ICLR 2025
Appendices
A EXTENDED RELATED WORK
Point tracking, presents significant challenges, particularly for long-term tracking where maintaining consistent tracking through occlusions is difficult. PIPs (Harley et al., 2022) was one of the first approaches to address this by predicting motion through iterative updates within temporal windows. TAP-Vis Vid (Doersch et al., 2022) initiated benchmark for evaluation. TAPIR (Doersch et al., 2023) improved upon PIPs by refining initialization and incorporating depthwise convolutions enhance temporal accuracy. BootsTAPIR (Doersch et al., 2024) further advanced TAPIR by utilizing suddnttteeache distillation on a large corpus of real-world videos. In contrast, CoTracker (Karaev et al., 2024b) introduced novel approach by jointly tracking multiple points, exploiting spatial correlations between points via factorized transformers. Differently, TAPTR (Li al., 2024b) adopted a design inspired by DETR (Carion et al., 2020; Zhu et al., 2021), drawing parallels between object detection and point tracking. DINO-Tracke (Tumanyan et al., 2024) took a different route, using DINO as a foundation for test-time optimization, whose tracking capabilities have been shown (Aydemir et al., 2024) to be one of the best among foundation models. TAPTRv2 (Li et al., 2024a), the successor to TAPTR, builds on its predecessor by incorporating offsets predicted by the deformable attention module. While these models calculate point-to-region similarity for correlation, LocoTrack (Cho et al., 2024) introduced a region-to to-riinnn similarity approach to address ambiguities in matching. Recently, CoTracker3 (Karaev et al., 2024a) combined the region-to toiiionto--riionn similarity method from LocoTrack with the original CoTracker architecture and utilized sseudo-labeled weal-world data during training to further enhance performance.
Tracking Any Point:
However, all these models are designed for offline tracking. assuming access to all frames within a sliding window (Karaev et al., 2024b) or the entire video (Doersch et al., 2023; 2024). Conversely, MFT (Neoral et al., 2024), which extends optical flow to long-term scenarios, can be adapted for online point tracking tasks, although it does not belong the point tracking family. Among point tracking approaches, models with online variants (Doersch et al., 2024; 2023) are re-trained with a temporally causal mask to process frames sequentially on a frame-by-frame originally designed for offline tracking. In contrast, we explicitly focus on online point tracking by design, enabled by novel memory modules to capture temporal information. Additionally, many of these models use regression objective, originally developed for optical flow (Teed & Deng, 2020), while we introduce a new paradigm based on patch classification and refinement.
basis, despite being
Another line of research, orthogonal to ours, explores leveraging scene geometry for point tracking. SpatialTracker (Xiao et al., 2024) extends CoTracker to the 3D domain by tracking points in three- dimensional space, while OmniMotion (Wang et al., 2023) employs test-time optimization to learn canonical representation of the scene. Concurrent work DynOMO (Seidenschwarz et al., 2025) also uses test-time optimization, utilizing Gaussian splats for online point tracking.
Causal Processing in Videos: Online, or temporally causal models rely solely on current and past frames without assuming access to future frames. This is in contrast current practice in point tracking with clip-based models, processing frames together. Causal models are particularly advantageous for streaming video understanding (Yang et al., 2022a; Zhou al., 2024), embodied perception (Yao et al., 2019), and processing long videos (Zhang et al., 2024; Xu et al., 2021), as they process frames sequentially, making them well-suited for activation caching. Due to its potential, online processing has been studied across various tasks in computer vision, such as pose estimation (Fan et al., 2021; Nie al., 2019), action detection (Xu et al., 2019; De Geest et al., 2016; Kondratyuk et al., 2021: Eun et al., 2020; Yang et al., 2022b; Wang et al., 2021; Zhao & Krähenbühl, 2022; Xu et al., 2021; Chen et al., 2022a), temporal action localization (Buch et al., 2017; Singh et al., 2017). object tracking (He et al., 2018; Wang et al., 2020), video captioning (Zhou et al., 2024), and video object segmentation (Cheng & Schwing, 2022; Liang et al., 2020).
In causal models, information from past context is commonly propagated using either sequential models (De Geest et al., 2016), which are inherently causal, or transformers with causal attention masks (Wang et al., 2021). However, these models often struggle to retain information over long contexts or face expanded memory requirements when handling extended past contexts. To address this, some approaches introduce memory modules for more effective and efficient handling of
15

---

Published as a conference paper at ICLR 2025
complex tasks. For example, LSTR (Xu al., 2021) separates past context as long-tem and short- term memories for action detection, while XMem (Cheng & Schwing, 2022) incorporates a sensory memory module for fine-gnt fine-grained information in video object segmentation. Long-tern memory- based modeling is also applied beyond video understanding (Balazevic et al., 2024), including tasks like long-sequence text processing and video question answering (Zhang al., 2021). We also employ an sutentionnbbsse memory mechanism, which is specialized in point tracking with two types of memory; one focusing on spatial local regions around points, and another on broader context.
B EXPERIMENT DETAILS
B. TRAINING DETAILS
We train our model for 150 epochs, equivalent to approximately 50K iterations, using a batch size of
32. The model is optimized using the AdamW optimizer (Loshchilov & Hutter, 2019) 32 x A100 64GB GPUs, with mixed precision. The learning rate is set to a maximum of 5 x 10 4, following a cosine decay schedule with linear warmup period covering 5% of the total training time. A weight decay of 1 x 10-0 applied, and gradient norms are clipped at 1.0 to ensure stable training. Input frames are resized to 384 512 using bilinear interpolation before processing.
For training, we utilize entire clips of length 24 from TAP- Vid Kubric. We adopt the data augmentation techniques from CoTracker (Karaev et al., 2024b), including random cropping size of 384 512 from the original 512 x 512 frames, followed by random Color Jitter and Gaussian Blur. Each training sample includes up N 480 points. We apply random key masking with 0.1 ratio during attention calculations for memory read operations throughout training.
For the training loss coefficients, we set 3. During training. we clip the offset loss the stride to prevent large errors from incorrect patch classifications and stabilize the loss. Deep supervision applied to offset head (Por), and the average loss across layers used. We set the softmax temperature to 0.05 in patch classification. We set the visibility threshold to 0.8 for all datasets except -Gaa-ttakiing where it is set to 0.5 due to its domain-specific characteristics, consisting of simple, synthetic vidcos.
off
R q-up
q-WE
x3
Linear
Linear
Block
M
Fanh
Lineaz
Deformabee Block x3
Linear
Concat
Chos- Attention Boods
Block
Layer Norm
Linear
Deformable
Pp.
Defernable Cooes Block
Pt
patch
Sall Attention
Coes: Coa--Aticntin
Crowe Antenttonn
Block
x3
h;
Concal
1llod
-h
M
Figure 9: Details of Different Modules. This figure describes the details for modules in our model: Query Decoder (o.. Offset Head (Pon)) Query Update Module (Qqup), Spatial Memory Write Module (f)),)), and Visibility Head (Pvis).
B.2 IMPLEMENTATION DETAILS
All of our modules consist of either a Self-Attention Block, Cross-Attention Block, or Deformable Cross-Attention Block. Each block includes multi-head self-attention, multi-head cross-attention or multi-head deformable cross coss-attention. followed by 2-layer feed-forward network with a hidden dimension expansion ratio of 4. Each multi-head attention uses 8 heads, while deformable multi- head attention operates with levels, sampling points per head. We extract multi-level feature
16

---

Published as a conference paper at ICLR 2025
Lincar
h2
Weighted Sum
C
h2
Black
Jornooople (290
Ujeanpli
Linear
Linear
h
3 Downowgll (20)
Uppemppl (+1)
Cancut
h
Desoemablee
Pt
top
Downaongle (466
Upenmple (c8)
Cross Allemtion
Block
h4
q0
Figure 11: Multiscale Similarity Calculation. This figure illustrates the detailed process of computing multiscale simi- larity between a given query qe and a feature map h, The different levels of the feature map (h₂) are generated by ap- plying bilinear downsampling at various scales.
Re-ranking Module. This figure describes the detailed architecture of re-ranking module (Paan)..
Figure 10:
maps by downsampling the input feature map and set the feature dimension D to 256. Following CoTracker (Karaev et al., 2024b), we add a global support grid size 20 x 20 during inference.
Visual Encoder (#vvs-cne): We use the ViT-Adapter (Chen et al., 2022b) with DINOv2 ViT- S/14 (Oquab et al., 2024; Käppeler et al., 2024) as the backbone. The DINOv2 inputs are resized to 378 x 504, as the default input size of 384 512 is not divisible by the patch size of 14. The backbone outputs, with a dimension of 384, are projected to D using single linear layer.
Query Decoder (Byydec): Query Decoder is shown in Fig. 9, first block. We set the number of layers to 3. Positional embedding 7 from the context memory is applied only to the keys, not the values, ensuring time-invariance in the queries while enabling the model differentiate between time steps during attention score calculation.
Offset Prediction (fof):: Second block in Fig. 9 shows the architecture of the offset prediction head. We set the number of layers to 3. Following DETR (Carion et al., 2020), we normalize the queries before projecting them through a linear layer, as the per-layer loss is calculated for the offset head.
Query Update (8 (Qqap): The query update is detailed in third block of Fig. 9. In both attention blocks, we mask items corresponding frames where points are predicted occluded.
Spatial Memory Write (Qq-wz): The spatial memory writer module is depicted in the fourth block of Fig. 9. We set the number layer to 3.
Visibility Head (Pvis): The visibility head is shown in the last block of Fig. 9. We decode the query feature around the predicted point and concatenate with the input query. Two separate linear layers are then applied to predict visibility and uncertainty.
Re-ranking Module (fre-nnk): The re-ranking module is detailed in Fig. 10. Initially, the given query qeee is decoded around the top-k points using Deformable Cross-Attention block with 3 layers. The resulting features are concatenated with the input query feature directly incorporate information from earlier stages. These correspond the top-k features, i.e. q top R. Nx Next, these features are fused into the input query through a single Cross-Atee Pross-AAtention block, followed by separate linear layer. The output is concatenated with the input once more, and a final linear transformation reduces the dimensionality from 2D to D (left upper arrow). Additionally, the uncertainties for these top-k locations are predicted directly from top using a linear layer (right upper arrow).
Multiscale Similarity Calculation: The multiscale similarity calculation used in patch classification (Section 2.2.3) is detailed in Fig. 11. Starting with a feature map h, E R
# y xD we generate iv scale representations by applying bilinear downsampling, resulting in hy E R z For each scale, cosine similarity is calculated between the feature maps and any query feature qı producing similarity maps at the respective scales. These maps are then upsampled back to the resolution of
17

---

Published as a conference paper at ICLR 2025
Table 4: Quantitative Results on RoboTTA,, Dynamic Replica, and BADJA This table shows results in comparison to the previous work on RoboTAP, Dynamic Replica, and BADJA under queried first setting. Similar to the main paper, the models are categorized into online and offline schemes, the former setting grants access to any frame regardless of video length, thus providing clear advantage. While online models process one frame at a time, enable frame-by-frame inference. For training datasets, Kub and Kub-L((ng)), refer to the TAP-Vid Kubric dataset with 24-frame and 64-frame videos, respectively; and R indicates the inclusion of large number of real-world videos, we highlight these models in gray
RoboTAP
Dynamic Replica
BADJA
Model
Input
Train
AJ 1
oug
OA f
guis
805g
80pz
Offline
TAPIR
Video
Kub
59.6
73.4
87.0
66.1
66.9
15.2
TAPTR
Window
Kub
60.1
75.3
86.9
69.5
64.0
18.2
TAPTRv2
Window
Kub
60.9
74.6
87.7
SpatialTracker
Window
Kub
69.2
17.1
LocoTrack
Video
Kub
62.3
76.2
71.4
CoTracker3
Window
Kub-I.
60.8
73.7
87.1
72.9
Trackrr
Video
Kub-L
59.9
73.4
87.1
69.8
BootsTAPIR
Video
Kub R
64.9
80.1
86.3
69.0
Co
Window
Kub-I. + R
66.4
78.8
90.8
73.3
Trakker
Video
Kub-L R
64.7
78.0
89.4
72.2
Online
Online TAPIR
Frame
Kub Kub
59.1
On (Ours)
Frame
63.5
76.4
89.4
73.6
71.9
20.2
the input feature map: 3 summation of the upsampled maps across all scales using learned coefficients an The weighted summation implemented as a convolution without bias.
H. D, using bilinear upsampling. Finally, we compute a weighted
C ADDITIONAL COMPARISONS
Similar to the main paper, we mainly focus on the setting using similar training set to ours, i.e., the models without using real-world videos.
RoboTAP: We evaluate our model on the RoboTAP dataset (Vecerik et al., 2023), which consists of 265 real-world robotic sequences with an average length of over 250 frames, as shown in Table 4. We use the same metrics the TAP-Vid benchmark: AJ, davg. and OA, with the memory size K; set to 48. Our model consistently surpasses existing online and offline models across all metrics. Specifically, in AJ and dang. our model outperforms the closest competitor, LocoTrrck (which processes the entire video), by 1.2 and 0.2 points, respectively. Additionally, it exceeds the nearest competitor (TAPTRv2) in OA by 1.7 points. This demonstrates that causal memory modules, which enable online tracking, are capable of effectively capturing the dynamics of long video sequences despite lacking bidirectional information flow across all frames. It is worth noting that this dataset, which features textureless objects, presents significant challenge. Fine-tuning on real-world videos provides substantial improvements, as learning to track points on textureless objects is particularly difficult, as highlighted by models tuned on real-world datasets.
Dynamic Replica:
We compare to previous work on the Dynamic Replica dataset (Karaev et al., 2023), a benchmark designed for 3D reconstruction with 20 sequences, each consisting of 300 frames, as shown in Table 4. Following prior work (Karaev et al., 2024b), we evaluate models using 8 consistent with the TAP-Vid benchmark. Unlike previous work, we do not report goee as model not supervised for occluded points. The memory size is set to K; = 48, Despite being an online model, our model outperforms offline competitors, including those trained on longer sequences (CoTracker3, 73.6 vs. 72.9) and versions fine-tuned on real-world videos (73.6 vs. 73.3). This highlights the robustness of our model, particularly in handling longer video sequences effectively.
BADJA: We compare to previous work on the BADJA challenge (Biggs et al., 2019), dataset for animal joint tracking comprising sequences, shown in Table 4. Two metries are used
18

---

Published as a conference paper at ICLR 2025
Table 5: Quantitative Results on PointOdyssey. This table shows results in comparison to the previous work on PointOdyssey under queried first setting.
PointOdyssey
Model
Input
Train
gusa Bang
Davy
MTR
Survival f
TAP-Net
Frame
Kub
23.8
92.0
17.0
TAP-Net
Frame
PO
28.4
63.5
18.3
PIPs
Window
Kub
16.5
147.5
32.9
PIPs
Window
PO
27.3
64.0
42.3
PIPs++
Window
PO
32.4
29.0
47.0
Co'Tracker
Window
PO
32.7
30.2
55.2
Track- On (Owrs)
Frame
Kub
38.1
34.2
28.8
49.5
for evaluation: 80, which measures the proportion of points within threshold relative to the segmentation mask size (specifically, points within 0.2vA, where A is the area of the mask); and 55px, the ratio of points tracked within 3-pixel range. Given the dataset's s low FPS nature, we kept the memory size at the original value of 12. Our model achieves state-of-the- results by significant margin, with a 2.7-point improvement de over SpatialTracker and a 2. 2.0-point improvement in g3ps over TAPTR. These results highlight the flexibility of our inference-tinee memory extension, enabling the model to adapt effectively to data with varying characteristics.
PointOdyssey: We evaluated our model, trained on TAP-Vid Kubric, the Point Odyssey (PO) (Zheng et al., 2023) dataset, which consists 12 long videos with thousands of frames (up to 4325). The results are presented in Table 5. We adopted four evaluation metrics proposed in Point Odyssey: Salg. which measures the dang metric from the TAP-Vid benchmark for visible points; sall which calculates Song for all points, including both visible and occluded ones; MTE (Median Trajectory Error), computed for all points; and Survival Rate, defined as the average number of frames until tracking failure (set to 50 pixels). The memory size K, was set to 96. From the results, we observe that PIPs trained on Kubric achieves a 6 of 16.5, while the same model trained on PO with larger window size achieves 27.3 (~~ 65% improvement). Notably, CoTracker does not report the performance of its model trained on Kubric but instead reports results for a model trained with sequences of length 56 on PO. These findings highlight the importance of training on PO to achieve higher performance across models. Our model, trained on Kubric, outperforms CoTracker and PIPs++ trained on PO in both g and dalg. Interestingly. while training on PO is critical for other models to achieve strong performance, our model demonstrates robustness by surpassing them even when trained on different data distribution. Moreover, despite not being explicitly supervised for occluded points, our model still achieves superior 5ull In terms of the Survival Rate, our model falls behind CoTracker trained on PO, despite its superior metrics. This further emphasizes the importance of training PO to excel in this specific metric.
D ADDITIONAL ABLATIONS
Memory Size: We experimented with varying memory sizes, trained with 12, and extended them to different values K {16. 32, 48, 64, 80, 96) during inference on TAP-Vid DAVIS and TAP- Vid RGB-Stacking, as shown in Fig. 12. The plot shows the change in AJ compared the default training memory size of 12 after extension. Memory sizes reported in Table are marked with crosses. For DAVIS (blue), performance slightly increases up a memory size of 48 (64.88 AJ 65.01 AJ) but declines beyond that, indicating that excessive memory can negatively impact the model. In contrast, for RGB-Stacking (orange), memory size plays more critical role due to the disparity in vidco frame counts between training (24 frames) and inference (250 frames), as well as the high FPS nature of the dataset. Performance consistently improves up to K 80, yielding a 2.9 AJ increase. These results highlight that, although the model is trained with fixed and relatively small memory size, extending memory during inference is possible to adapt the varying characteristics of different datasets.
Top-k points: We experimented with varying the number of l points used in re-ranking (Sec. 2.2.3) on Vid-Vid DAVIS, as shown in Fig. 13. Although the model trained with fixed top-k value of
19

---

Published as a conference paper at ICLR 2025
30-
64
15
88
08
-05
56-
Memory Size (K.)
12
32
Figure 12: Memory Size. The effect of varying extended memory sizes during inference, on TAP- Vid DAVIS and TAP- Rid RGB-Stacking.
Figure 13: Top-k points. The effect of varying number of top points in ranking module, during inference.
k 16, this number can be adjusted during inference since the selected points are fused into the target query via transformer decoder. The & value reported in Table marked with cross. The results indicate that increasing the top-k points up to 16 consistently improves performance. Beyond 16, performance decreases slightly but remains robust, even with larger values. This demonstrates that choosing a value smaller than the optimal number leads noticeable performance drop, emphasizing the importance of appropriately selecting
Offset Head and Stride: The offset head is essential for refining patch classification outputs, enabling more precise localization. Specifically, the offset head allows for precision beyond the patch size S (stride). In Table 6, examine the impact of removing the offset head (Pan) for two stride values, S = and S = 8, without utilizing inference-time memory extension. For both values, the addition of the offsst head significantly enhances AJ and 0agg refining predictions within the local region. With stride 4, the offset head notably improves 82pz, while for stride 8, it improves both 82p and 84p2 This demonstrates that while patch classification offers coarse localization, the offset head provides further refinement, achieving pixel-level precision.
Larger stride values risk losing important details necessary for accurate tracking. For instance, increasing the stride from
results AJ drops of 12% for TAPIR and 16% for CoTracker, as reported in their ablation studies. However, coarse-to-fine approach mitigates the negative effects of stride 8, leading to only a minimal decline of 4%, highlighting the robustness of our model to larger stride values.
Note that the model with S 8 and no offset head (first row) has a higher occlusion accuracy (OA). A possible reason is the imbalance in the loss, where the visibility loss has relatively higher impact compared to the model with an additional offset loss (second row), leading to improved occlusion accuracy.
Table 6: Offsee Head. The effect of removing the offset head (#oa) on models with varying strides.
All metrics are higher-is-beree better.
off
Stride
2pp
gipz
8pp
AJ
8
OA
X
37.4
79.0
91.1
51.3
62.9
91.0
8
66.1
84.0
91.7
62.5
75.8
90.6
64.3
84.4
92.4
60.1
73.0
4
69.3
85.5
92.5
64.9
77.7
90.6
E ANALYSIS OF SPATIAL MEMORY
To evaluate the effect of spatial memory in the presence of feature drift and inference-tin memory extension, we conduct an experiment across different datasets using a model trained without spatial memory (Model-C in Table 3), as shown in Table 7. The results indicate that spatial memory
20

---

Published as a conference paper at ICLR 2025
Table 7: Spatial Memory. Comparison the model's performance with and without spatial memory (M*), evaluated using the AJ metric across different datasets, with inference-time memory extension (IME) applied.
Model
DAVIS
RGB BtB-Stacking
Kinetics
RoboTAP
Full Model
65.0
71.4
53.9
63.5
Without Spatial Memory (M")
64.6
70.2
53.3
62.1
consistently improves AJ across four datasets: DAVIS, RGB-Stacking, Kinetics, and RoboTAP. The impact is particularly notable for (GBB-ttccing (+1.2 AJ) and RoboTAP (+1.4 AJ), where objects are less descriptive and often textureless, as both datasets originate from robotics scenarios. This suggests that spatial memory, which retains information around the local region of previous predictions, helps mitigate drift and enhances generalization across different scene characteristics.
Additionally, to directly assess the impact of spatial memory (Sec. 2.3) in mitigating feature drift, we conducted an analysis comparing the tracking performance of the initial feature sampled from the query frame, q gisst, with the query feature updated using spatial memory frame I, denoted as q inst For this evaluation, we introduced the new metric of similarity ratio score (8er). which measures how well the updated query features align with the feature at the target point compared to the initial query.
Ideally, qiaa should provide better starting point for detecting correspondences compared to qimit, particularly when the object's appearance changes significantly. To assess whether qi more similar to the feature at the ground-truth correspondence location than q qanit, calculate the ratio of their similarity to ground-truth as a way of quantifying the increase the similarity after the update:
S (f) = gi sample(fr, P:) qiait sample(f,, P:)
(10)
Here, P: represents the location of the ground-trute correspondence point, and f, is the feature map of the target frame. On the DAVIS dataset, we calculated Ser for visible points, achieving score of 1. 1.24, indicating that spatial memory introduces a 24% increase in similarity compared to the initial feature. In Fig. 14, we visualize
similarity scores for different tracks over time for two videos from the DAVIS dataset. The plot highlights that the similarity increases more significantly toward the end of the video, where appearance changes are more severe. Moreover, the score consistently greater than 1, showing that giii always provides better initialization than init in these videos.
191
Figure 14: Similarity Ratio Score The similarity ratio score S > over frames for different tracks, demonstrates increased similarity with ground-truth location on the target frame when utilizing spatial memory.
F FAILURE ANALYSIS
We identify three common failure cases: (i) tracking points on thin surfaces or edges, (ii) fast motion scene cuts, and (iii) localization uniform areas. We visualize examples of these failure cases,
21

---

Published as a conference paper at ICLR 2025
where the average error is higher than 8 pixels for visible frames, in Fig. 15. In the visualizations, predictions are represented as dots, while ground couund-truth correspondences are marked with diamonds. The line connecting the ground-truth and prediction indicates the enror.
Thin Surfaces: Points interest on thin surfaces and edges may not be well-represented in feature- level resolution due to the lack of pixel-level granularity. This limitation causes the model to track incorrect points by missing the actual point interest. For instance, in the upper row of Fig. 15a, which shows an example from the DAVIS dataset, the model fails track rope and instead tracks the background, as the precision is insufficient to accurately represent the thin structure. Similarly, in the bottom row, from the Kinetics dataset, points of interest on a thin surface (e.g. stick) are mislocalized, with the model tracking the background instead of the object.
When the scene content changes rapidly, either due fast camera motion scene cuts (commonly seen in the Kinetics dataset), our model encounters difficulties in localizing previously tracked points. For instance, in the upper row of Fig. 15b (examples from Kinetics), the model fails to continue tracking after a large scale change, where the original view in the middle frame becomes significantly smaller in later frames. In the second row, the model struggles to resume tracking points after considerable number of frames where the scene has completely changed due to scene cut.
Fast Motion:
Uniform Areas: Localization on uniform or highly similar areas is more challenging. likely because most objects in the training dataset exhibit descriptive textures (Doersch et al., 2024). While our model can approximately localize points, it struggles with precision in these scenarios. This limitation is illustrated inFig. 15c, using an example from the Robo TAP dataset.
22

---

Published as a conference paper at ICLR 2025
(a) Failure cases due to surfaces.
(b) Failure cases due to fast motion or scene cuts.
(c) Failure due mis-localization uniform
Figure 15: Common Failure Cases. We identify three common failure cases: tracking points on thin surfaces (a), fast motion or scene cuts (b), and localization on uniform areas (c). We visualize predictions with average error higher than 8 pixels, where predictions are shown as dots and ground- truth correspondences are marked with diamonds. Different tracks are depicted in distinct colors.
23


Reviews:
[{"review_id": "8Y564lBGmo", "reviewer": "Reviewer_miCL", "summary": "The paper introduces Track-On, a transformer-based online long-term point tracking model designed for real-time point tracking in video streams. Track-On leverages two memory modules—spatial memory and context memory—to capture temporal information and maintain reliable point tracking over extended video sequences. Unlike methods relying on iterative updates and full temporal modeling, Track-On employs patch classification and refinement for identifying correspondences and tracking points with high accuracy. Extensive experiments demonstrate that Track-On sets a new state-of-the-art for online models and delivers superior or competitive results compared to offline approaches on the TAP-Vid benchmark.", "questions": "There are too few datasets for comparison, I hope to supplement the experiments.\n\nAre there common patterns or specific scene characteristics in the cases where the model fails?\n\nThere are other transformer-based tracking methods mentioned, such as SpatialTracker and CoTracker. How does Track-On differentiate itself from these methods?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The dual memory module design effectively handles long video sequences in an online setting while addressing feature drift.\n\nTrack-On demonstrates fast inference speeds while maintaining high accuracy, a significant advantage in real-time tracking systems.\n\nThe model establishes a new benchmark for online models on the TAP-Vid benchmark and shows competitive performance against offline methods.", "weaknesses": "The paper mentions potential precision loss when tracking thin surfaces or instances with similar appearances, indicating limitations in handling certain visual effects. \n\nThe model may struggle to establish correct correspondences in complex scenes with high visual similarity, affecting tracking accuracy.\n\nWhile the model is memory-efficient, there is a trade-off between memory module size and inference speed, which may require balancing in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KLLGZGpTEU", "reviewer": "Reviewer_BkZB", "summary": "This paper proposes a long-term point tracking framework focused on online, frame-by-frame tracking. The framework leverages spatial and context memory modules to reliably track points over extended time horizons. It achieves state-of-the-art results for long-term point tracking in the online setting. Additionally, the framework is capable of real-time tracking.", "questions": "It would be interesting if the authors can show that this method can be used on much longer videos from recent datasets such as PointOdyssey or Tapvid3D.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow, with extensive experiments that clearly demonstrate the performance improvements contributed by each module. \n- The authors propose an effective approach using patch classification, rather than regression, to predict each point location; while patch classification was mentioned in the PIPs paper, it was previously used only to accelerate convergence by supervising score maps.\n- The authors propose 2 memory modules that effectively improve the tracking performance.", "weaknesses": "- line 289: I think you meant $\\gamma^s$ not $\\gamma^q$ which does not appears in eq.8.\n- Table 1: the authors should show the backbone used in each approach for easier comparison. If possible, the methods should be compared on the same backbone. However, this is alleviated by table 3 where the authors have show the effectiveness of the memory modules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7r2xfDsF8s", "reviewer": "Reviewer_D5B2", "summary": "In order to make point tracking methods more suitable for the real world, the author proposes a new frame by frame online tracking method. The author designed a spatial memory module and a content memory module to capture temporal information and maintain reliable point tracking over a long period of time. Finally, the author uses a coarse to fine manner to predict the final output.", "questions": "1.Although the authors demonstrated many visual maps to demonstrate the effectiveness of the method, a quantitative ablation analysis of the critical modules is lacking, for example, the layer number analysis of query decoder.\n\n2.Many writing details should be noted by the author, including the expression of the picture content, the color of the formula and other details.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.The author designed two memory modules to mine spatio-temporal information and use coarse to fine manner for more accurate point prediction. \n\n2.Experiments demonstrate the effectiveness of the method proposed by the authors and obtain the SOTA performance of the online point tracking method.", "weaknesses": "1.Experiments were inadequate. The designed analysis and the layer number ablation analysis of each modules are not enough, including several decoders, the number (4) of level of similar map used for patch classification, etc.\n\n2.The figures in the paper are not clear enough. For example, in the left of Figure 5, the input qinit does not understand what it means, because q has no subscript and does not specify its meaning. In additional, the title of Figure 5 should go from left to right when introducing the content.\n\n3.Does online point tracking just require that the model does not see the next frame during testing? I think the authors should add the speed of testing like SOT, as the authors said, to be more adaptable to the real world applications.\n\n4.Although your online method is the best, it does not outperform the offline method in the DAVIS data set. What is the reason? Apart from the characteristics of the two type methods.\n\n5. Although the author analyzed the causal processing method in the relevant video field, the analysis of the online method in point tracking is not sufficient, and it does not see the advantages of the proposed method in online methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g8Kud4dKPp", "reviewer": "Reviewer_sUJJ", "summary": "The paper proposes a simple yet effective framework, named Track-On, for Tracking Any Point (TAP) task. \n\nTrack-On employ a coarse-to-fine mechanism, where they first find the most match patch feature, and then predict an offset relative to the center of the most matched patch to obtain a more fine-grained prediction.\n\nTo tackle the issue of feature drifting in long-term situation, Track-On further proposes the Spatial Memory as well as Context Memory. Spatial Memory utilize deformable attention to flexibly select the most-relevant features in the last frame, to provide a more reliable and latest feature as a reference to help the prediction of current frame. While Context Memory maintains a FIFO queue to store the latest K frames' decoder outputs, to provide a larger context.\n\nExtensive experiments are conducted. The performance on TAP-Vid benchmark shows their superiority, especially on Kinetics and RGB-Stacking subsets. The thorough ablation studies and supplementary materials make the paper looks solid.", "questions": "1. I wonder how the Fig. 4 is obtained. By sampling on each frame based on the gt position and then calculate the similarity?\n\n2. It seems a typo in L289, what is the r^q?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a simple yet effective transformer-based framework for tracking any point task, named Track-On. Track-On employs the coarse-to-fine strategy. Track-On utilize the DINOv2's feature to help find the most matching feature patch to provide a coarse prediction. After that, Track-On predict a local offset to obtain a more fine-grained prediction. The coarse-to-fine strategy looks reasonable.\n\n2. The paper proposes the Spatial Memory and Context Memory to handle the long-term tracking situation. Spatial and Context Memory provides complementary information for each other, improving their performance significantly.\n\n3. Extensive experiments and ablation studies are conducted, showing their superiority and the effectiveness of each module.", "weaknesses": "I think there is no fatal weaknesses, but there are some concerns:\n\n1. There are some designs that are similar with previous works. For example, in TAPIR, they also use coarse-to-fine strategy, in TAPTR, they also utilize the initial feature to alleviate error propagating. It is better to cite these paper during your description.\n\n2. Some points are over-claimed. Actually, the Spatial/Context Memory can not \"address\" tracking drifting. There are more metric or experimental evidence to support the claims in L78-L87.\n\n3. Since the positional embedding are learned during training process, how to guarantee the continuity of the embeddings? If the continuity can not be guaranteed, how to ensure the \"inference-time memory extension\"? What if just ignore the positional embedding?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)