TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any Point in Long Video
Authors: Jinyuan Qu
Hongyang Li
Shilong Liu
Tianhe Ren
Zhaoyang Zeng
Lei Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Tracking Any Point
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A novel state-of-the-art online method for the Track Any Point task, optimized for long videos.

Abstract: In this paper, built upon TAPTRv2, we present TAPTRv3. TAPTRv3 improves TAPTRv2 by addressing its shortage in querying high quality features from long videos, where the target tracking points normally undergo increasing variation over time. In TAPTRv3, we propose to utilize both spatial and temporal context to bring better feature querying along the spatial and temporal dimensions for more robust tracking in long videos. For better spatial feature querying, we identify that off-the-shelf attention mechanisms struggle with point-level tasks and present Context-aware Cross-Attention (CCA). CCA introduces spatial context into the attention mechanism to enhance the quality of attention scores when querying image features. For better temporal feature querying, we introduce Visibility-aware Long-Temporal Attention (VLTA), which conducts temporal attention over all past frames while considering their corresponding visibilities. This effectively addresses the feature drifting problem in TAPTRv2 caused by its RNN-like long-term modeling. TAPTRv3 surpasses TAPTRv2 by a large margin on most of the challenging datasets and obtains state-of-the-art performance. Even when compared with methods trained on large-scale extra internal data, TAPTRv3 still demonstrates superiority.

Published as a conference paper at ICLR 2026
TAPTRV3: SPATIAL AND TEMPORAL CONTEXT FOSTER ROBUST TRACKING OF ANY POINT
IN LONG VIDEO
Jinyuan Qu 133
Hongyang Li
Shilong Liu
Tianhe Ren
Zhaoyang Zeng
Lei Zhang
"Tsinghua University
South China University of Technology
3 Intternational Digital Economy Academy (IDEA)
ABSTRACT
In this paper, built upon TAPTRv2, we present TAPTRv3. TAPTRv2 is simple yet effective DETR-like point tracking framework that works fine in regular videos but tends to fail in long videos. TAPTRv3 improves TAPTRv2 by addressing its shortcomings in querying high-quality features from long videos, where the target tracking points normally undergo increasing variation over time. TAPTRv3, propose to utilize both spatial and temporal context to bring better feature querying along the spatial and temporal dimensions for more robust tracking in long videos. For better spatial feature querying, identify that off the- attention mechanisms struggle with point-level tasks and present Context-ane aware Cross- Attention (CCA). CCA introduces spatial context into the attention mechanism enhance the quality of attention scores when querying image features. For better temporal feature querying, we introduce Visibility-aware Long-Temporal Attention (VLTA), which conducts temporal attention over past frames while considering their corresponding visibilities. This effectively addresses the feature drifting problem in TAPTRv2 caused by its RNN-like long-term modeling. TAPTRv3 surpasses TAPTRv2 by large margin on most of the challenging datasets and obtains state-of-the-arr performance, Even when compared with methods trained on large-scale extra internal data, TAPTRv3 still demonstrates superiority.
INTRODUCTION
Localizing points across different frames in a video is long-standing problem (Sand & Teller 2008).
Recently, with the growing demand for the trajectory and visibility information of arbitrary points in
videos for various down-stream tasks, such video editing (Huang al.
2023). ) SLAM (Teufel et
2024), and manipulation (Vecerik et al. (2024). the Tracking Any Point (TAP) task has grad ual
regained attention (Harley et al. 2022
Doersch al., 2023 Karaev al.
2024b
Li et al. 2024c b
Neoral et al. 2024 Xiao et al. 2024 Wang et al. 2023 Tumanyyn et al.
2024).
To solve this problem, some methods try to construct 4D field (Wang et al.
2023) and track points
in the constructed 3D space while also achieving 3D scene perception. Though promising, such methods are normally not general and have inferior performance. By contrast, more methods attempt
to solve the TAP task directly from 2D perspective (Harley et al. 2022, Doersch et al. 2023 Karaev
jet al. 2024b a Li et al.) 2024c b) Tumanyan et al. 2024 Aydemir et al. [2025] Zholus et al. (025).
Some of these methods follow the traditional optical flow estimation pipeline, RAFT (Teed & Deng 2020) more specifically, and highly rely on the dense cost-volume (Xu et al., 2017) to perform point tracking like sparse optical flow. Although such methods have achieved impressive performance, the computation of dense cost-volume is resource-consuming especially when the number of points, the length of videos, or the video resolution increases.
Inspired by recent visual prompt-based detection methods (Li et al. 2024a Jiang et al. 2025) ) TAPTR (Li et al. 2024c) proposes Ditection-TRansfrrmee (DETR)-ike like framework, which regards
Equual contribution, random listing order. Work done during an internship at IDEA Research. Corresponding author.

---

Published as a conference paper at ICLR 2026
each tracking point point query and addresses the TAP task from the perspective of point-level visual prompt detection. TAPTRv2 (Li et al. (024b)) further improves TAPTR by eliminating the
requirement of dense cost-volume input, as it will contaminate the point query's s content feature and introduce redundancy. To compensate for its localization role, TAPTRv2 proposes an attention- based position update (APU) operation that utilizes key-aware deformable attention (Li et al., (023)
to compare a query point with a set of key sampling points and find a better position. With this improvement, TAPTRv2 obtains both simpler framework and better performance.
However, we find that TAPTRv2 still struggles with long videos due to its shortage of feature querying in both spatial and temporal dimensions long videos, in which the target tracking points normally undergo increasing variation over time. In the spatial dimension, TAPTRv2 introduces key-aware deformable attention (Li et al. 2023) to extract features and directly perform position update by comparing the feature similarity between query point and a set of surrounding sampled key points. However, when migrating this module from an object-level detection task a point-level tracking task, TAPTRv2 overlooks the fact that the point-level query and key features, obtained simply through bilinear interpolation, are too local. This makes the resulting attention weights susceptible to noise. This instability significantly affects point tracking, which requires the most fine- tne-ggrined spatial understanding. In the temporal dimension, the RNN-like long-temporal modeling in TAPTRv2 often suffers from the drifting problem, as the feature of a tracking point may be gradually affected by ambiguous surrounding features and unknown occlusion over time. Moreover, there is a significant discrepancy video length between the current training and testing sets. The training set consists of short videos with fixed 24 frames, while the testing videos vary from 50 to 1300 frames in length, Excessive feature updates in long videos during testing further exacerbate the feature drifting problem. In our study, we also observe the existence of scene cuts in many videos. While such videos are the result of artificial editing, they are quite prevalent in public datasets. For instance, in TiP-Vid- Kinetics (Doersch et al. 2022), which is one of the challenging test sets, approximately 27% of the videos contain scene cuts. The lack of global matching in TAPTRv2 makes it hard to reestablish tracking effectively when scene cut occurs with sudden, large motions.
With these insights, we propose to enhance the feature querying ability of TAPTRv3 in both spatial
and temporal dimensions. For spatial feature querying, inspired by the prior 4D cost volume-based
optical flow (Teed & Deng 2020 Oig et al.] 2017| Sun et al. 2018 Wang et al. 2020 Jiang et al.,
2021 Xu et al. 1021) and point trac rrckkig methods (Cho et al., 2024 Bian et al. 2023).
introduce the context information to the attention mechanism. Specifically, ly, we develop a Context-aware Cross- Attention (CCA) operation optimize the key-aware deformable attention mechanism. Instead of using unstable point-level similarity, our method leverages patch-level similarity to compute the attention weights. The patch-level attention utilizes more context features to prevent the attention weights from being disturbed, thereby bridging the gap between object-level and point-level tasks, For temporal feature querying, to address the drifting issue, we discard the RNN-like long-tempore modeling in TAPTRv2. Instead, for any tracking point, we resort to the initial feature sampled from its starting frame, as this feature is the most reliable, and use it as input in any frame. To compensate for feature change over time, we introduce Visibility-aware Long-Temporal Attention (VLTA) operation, which treats the initial feature as a query and performs dense attention over the past frames to aggregate past feature changes. This not only enables the perception of longer temporal context but also makes TAPTRv3 an online tracker. Mcanwhile, recognizing that the target tracking point may be occluded in some frames, we reweight the long-temporal attention weights using the estimated visibility scores in the past frames, directing more attention to frames where the point is visible. This further enhances the feature querying ability along the temporal dimension.
For the scene cut issue, we introduce an auto-triggered global matching mechanism to reinitialize the point query's 's positional part for subsequent frames. Note that we only trigger the global matching when detecting a scene cut. This is based on our observation that in regular videos, using the predicted positions from the previous frame as the initial position for the current frame yields better results.
In summary, our contributions are threefold: (1) The primary contribution is the development of more robust solution for point tracking in long videos, which improves upon TAPTRv2 by leveraging both spatial and temporal context. The two corresponding operations, namely Context-aware Cross- Attention and Visibility- aware -Tmpprall Attention, effectively improve the quality of spatial cross attention and long-term feature updating, enhancing feature querying. (2) To address the scene cut issue, we introduce an auto-triggered global matching mechanism, which is only triggered when scene cut is detected. This ensures stable tracking on regular videos while being able to quickly
2

---

Published as a conference paper at ICLR 2026
reestablish tracking when encountering scene cuts. (3) Extensive experimental results show that TAPTRv3 significantly outperforms TAPTRy2 and achieves state-of-the- performance on most datasets. Even when compared to models trained on large-scale extra internal real data, TAPTRv3 remains competitive.
2 RELATED WORK
Optical Flow Estimation. Establishing the correspondences for every pixel between two consecutive frames is long-standing problem. Over the past few decades, extensive research has been dedicated to addressing this issue. Traditional methods (Horn & Schunck 1981 Black & Anandan [202 Bruhn et al. 2005) use carefully designed descriptors to find correspondences and apply manually designed rules to filter out distractions. DCFlow (Xu et al., (017)) first demonstrated the feasibility of using the features learned from deep neural network to obtain optical flow estimation through cost-volume that
Dosovitskiy eet al. 2013; lg et al.) 2017] Xu et al. 2017] [Sun et al. 2018: Wang et al. 2020 Jiang et al. 202T
is constructed by calculating 4D correlation, and dominant this field (Teed & Deng 2020
Xu et al. 2021] Zhang et al.) 2021] Huang al. 2022 Zhao et al. 2022). Although the features extracted by deep neural networks are much stronger, the cost-volume still suffers from ambiguity.
To address this, these methods typically feed the cost-volume into convolutions to normalize based
on contextual information. A Although the recent optical flow estimation methods (Shi et al. 2023a Saxena et al., 2024 (Shi al., 2023b) have shown remarkable results, they still can not handle video data well, especially when points of interest are occluded.
Tracking Any Point. Influenced by optical flow estimation methods, especially the RAFT (Teed
Deng 2020). most approaches (Doersch et al. 2022, 2023) 2024 Zheng et al. 2023 Karaev et al. 2024b a Cho et al. 2024) follow a similar framework, calculating a cost-volume between the target tracking point and every frame, and then feeding the cost-volume into transformer (Vaswani et al.
2017) to regress the position of the point in each frame. Inspired by the optical flow method (Teed & Deng 2020), LocoTrack (Cho et al.) (024) introduces a local 4D correlation to enhance perfor- mance. TAG (Harley et al.] 2024) extends tracking points to tracking arbitrary targets in videos. AnthroTAP (Kim et al., 2025) proposes pipeline to generate training labels for point tracking
from human motion data. Track-On (Aydemir et al. (025) focuses on online tracking, introducing memory modules to capture temporal information for reliable point tracking. TAPNext (Zholus et al.,
2025) is also an online model that casts this task as sequential masked token decoding and removes -recing-speccfic inductive biases. In another line, TAPTR and TAPTRv2 (Li et 2024c b) address the TAP task from the perspective of detection with their DETR-like (Carion et al. 2020 Liu et al.,
2022, Li et al. 2022: Zhang et al. (223a)) framework. However, TAPTRv2 still lacks designs for more challenging long-term tracking, resulting in suboptimal performance in long sequences.
METHOD
3.1 OVERVIEW
Before describing TAPTRv3, for clarity and without loss of generality, we assume that only a single point is being tracked, starting from the first frame Iy. Given I, and the user-specified point to be tracked at lo E 1R2 TAPTRv3 is expected to detect this point in every subsequent frame { T 7-
determining its location (l. (1) and visibility {ax (==) where 1, E IR.2, ar E [0, 1]. and T is an integer that indicates the length of the video. As shown in Fig. (a). TAPTRv3 can be roughly divided into the Point Query Preparation stage and the Sequential Point Tracking stage.
Point Query and Spatial Context Preparation. As depicted in Fig. (a), given user-specified coordinate lo on the initial frame Io, the point query preparation stage will sample a point-level feature f E R to describe the target tracking point, where D is the number of channels. Following TAPTRv2, iT
we conduct bilinear interpolation on the L, 's corresponding image feature map X E R "x 12 at
l. where H and W indicate the height and width of the feature map. To create more comprehensive
description of the point, TAPTRv3 additionally samples N context features C N around lo in grid form to describe the point's 's initial spatial context:
C Bili (X,, 1o + GG,
For clarity, without loss of generality, assume that each frame's feature map, obtained from the hackbone and transformer encoder, has only one scale.
3

---

Published as a conference paper at ICLR 2026
Poiee Query Preyenation
Soquestial Poiat Tooking
Vishiiyyaoon Larprorpinl (Vi-TA)
4.
full fou
exl7 6: Mall
Corcatroarr Cross Atterticn with Attrntonn hasdd Postion Updat (CTAA APL)
MCP
@ Weight Sam
L.C.C
C
FAREIIN (4)
O Aodiic
VUA
VETA
Cantect Preparation
Postaci
Fesl Secarl
F.C.
GB
Sanpleg Ociste)))
Radillod Sterkarty
[41
Overview (a) and core components (b) (c) of TAPTRv3. After the user specifies the point to track, the point query preparation stage prepares the content and spatial context features for this point in the initial frame. When TAPTRv3 receives a new frame, the sequential point tracking stage uses the content feature and the specified location as the point query, and regards the new frame's image feature map as keys and values. The points query, keys, and values are fed into a multi-layer transformer decoder to detect the tracking point in the new frame. The predicted location then updates the point query's ss positional part, providing a better initial position for tracking in the next frame. For clarity, the global matching module is not plotted.
Figure I:
where NN is hyperparameter and is set by default, G V is the sampling grid, and Bili ii is the bilinear interpolation operation. After that, f and lo will be regarded as the initial content part and the positional part of the target tracking point's corresponding point query. The point query will be sent to the transformer decoder as a query to detect the target tracking point in subsequent frames in the next Sequential Point Tracking stage.
Sequential Point Tracking. As shown in Fig. (b), when TAPTRv3 receives a new frame, I for example, its corresponding feature map X,, which is regarded as set of keys and values, as well as the point query, will be sent to the multi-layer transformer decoder. In each transformer decoder layer, both the content part and the positional part of the point query will be refined by our Attention, Context-aware Cross-A Attention with APU, Self-Attention,
Visibility-awan Long-Tempol and Feed Forward Network. After the multi-layer refinement. The output refined positional part of the point query 1 will be regarded as the detection result of the target tracking point in I, and the output refined content feature the point query fi will sent to an MLP-based binary classifier to predict the confidence (r) that the point is visible in I. After that, the position prediction 1 will be used to update the point query's 's positional part, providing a better initial position for detecting the target tracking point in the next frame. while the content part remains the initial content feature f. This process proceeds repeatedly until the end of the video.
3.2 VISIBILITY-AWAR
ATTENTION
The RNN-like long-temporal modeling in TAPTR and TAPTRv2 can lead to feature drift, primarily due to the excessive feature updates during testing, rooted in the disparity in video lengths between training (24 frames) and testing (ranging from 50 to 1300 frames). Thus, shown in Fig. D (b), resort to the attention mechanism for its ability to handle varying token lengths, as demonstrated in modern LLMs (Zhang et al./ 20255 2023b OpenAl 2023 Google 2023). Following modern LLMs, we also utilize the rotary positional embedding (Su et al. 2024 to help our long-temporal attention pay more attention to recent frames. More specifically, the long-temporal attention weights of the target tracking point between the f-th frame and all past frames can be first computed as:
F. = [fo fu, ... fr-1]" R. = d SoftMax ((F, R2) @ (f++ +r)))
where F, E R and R, E L are the point query's refined content features in the past frames of the 1-th frame, and their corresponding rotary frame index embeddings, d' RC is the lemg-te
In practice, is not necessary to interact with all past frames, and this is only for ease of description. For details, please refer the Sec. A.2 in our appendix.
4

---

Published as a conference paper at ICLR 2026
attention distribution (weights), @ indicates the matrix multiplication, and f E RR is the content feature of the point query in the t-th frame that has not been fully refined by decoder. In the first decoder layer, f/ f.
Different from the textual scenarios, since the target tracking point will be occluded sometimes, the refined content features from the frames in which the target tracking point occluded may contribute noise to the long-tempont attention. To prevent being affected by the noise, we utilize the visibility predictions in the past frames a E R' = [00: ai...... at-1]" to reweight the attention distribution, making it visibility-aware awarr.. The final visibility-aware attention weights are used weight-sum their corresponding content features to obtain the temporal querying result Af, E TRD. The querying
result will be further utilized as a residual to update the point query's content feature to complete the VLTA. The process can be formulated as:
d' ④ a d, Sum(a.)
ACT = F 2 de IN + A
(3)
where d, E RR is the attention distribution that reweighted by visibility predictions, f. is the output and will be sent to the following modules for further refinement, LN is the Layernorm (Lei Ba et al., 2016), and @ is the element- wise multiplication.
3.3 CONTEXT-AWARE CROSS- CCOSS-ATTHES ATTESTIOOO WITH APU
Unlike object-level DETR-like methods, the content feature of a point query in TAPTRv2 is a point- level feature. This granularity limits the model's receptive field during cross-attention with the image feature map, often causing ambiguity in attention weights. This issue becomes more serious when the target tracking point undergoes significant variations or when the image contains uniform regions or repetitive patterns. Such conditions can lead to noisy querying of spatial features as well as noisy position updates in the cross- cooss-aatention's 's belonging APU block. Inspired by previous methods (Teed & Deng. 2020 Bian et al. [023] Cho et al. 2024), we propose integrating richer spatial context into the attention mechanism. This provides point queries with a more comprehensive understanding of their surroundings, resulting in more accurate and robust attention weights.
illustrated Fig. T (c), different from the vanilla key-aware deformable attention (Li et al. 2023). the query's point-level content feature will only be used to predict M sampling offsets
C
O, E Mx2 [of: of .., o, responding attention weights, we obtain them by leveraging the patch-level context features. More specifically, as shown Fig. 2 for the m-th sampling point, its corresponding spatial context features in the t-th frame K," E IR x can be constructed by:
_M-1] by an MLP. For the cor-
X,
Figure? 2: Illustration of patch- level similarity calculation.
K Bili (X,,1, + + G),
where I E R² is the positional part of the point query in the 1-th frame that has not been fully refined by decoder layers. In the first decoder layer's CCA module I; = 1 After that, the m-th sampling point's corresponding patch-level similarity w:"" E R is calculated as:
S. E SRNN N2 C & KTT w," MI.(Flatten(S"))
(5)
where S."" is the intermediate representation of the patch-level similarity, and Platten indicates the flatten operation. By sending the intermediate representation S to an MI.P, the more fine-grained similarities between every two points in the two patches are comprehensively considered, leading to high-quality patch-level similarity. The similarities between the point query and all sampling points w, E IR [we w/ ..... w will work as attention weights to aggregate their corresponding values V, E D obtaining the spatial querying result ArS E D:
V. = [vD v... ... V.M
= BBili 1/ + +
AP. V & SoftMax (we/VD)
where V. are the sampled values of M sampling points on X.. After that, the spatial querying results will be utilized as a residual to complete the CCA:
f + LN(F' AFS).
5

---

Published as a conference paper at ICLR 2026
Following TAPTRv2, we also add the attention-based position update (APU) in CCA, to benefit from the attention weights' resilience to the domain gap without contaminating the point query. This process can be formulated as:
A = o, @0 SoftMax (MPP (w.) (w.) /V)) 1, < 12 + A
(8)
where AI, E 2 is the additional position update. We follow TAPTRv2, using an ML. to decouple the attention weights used for content and position updates.
3.4 TRIGGERIG GLOBAL MATCHING
In general, point motions smooth in video. However, in long videos, especially in offline scenarios, scene cut often occurs, leading to sudden large motions. Since the initial position of a point query in the current frame is inherited from the last frame's prediction, as we have described in Sec[3.] the sudden large motion will cost many frames for TAPTRv3 to catch up with the target tracking point.
Thus, when scene cut occur at the i-th frame, we trigger the global matching module help TAPTRv3 reestablish tracking. Similar to the previous method (Doersch et al., 2023), the global matching module will construct similarity map H, E F between the target tracking point and the current frame's 's feature map. However, instead of relying on point-level feature as in previous methods, similar to CCA, we leverage the spatial context features to help improve the accuracy of the similarity map. After that, SoftArgMax is conducted on the similarity map to obtain a final position prediction 1 E R2² that is differentiable. The prediction will be used to replace the unreliable position prediction of the current frame from the transformer decoder. The replacement helps TAPTRv3 to prevent wrong predictions in the subsequent frames. The process can be formulated as:
H = X, & C H. = Softmax(MLP
(9)
1 So SoftArggax(H..)
where H. E HR is the similarity maps between the initial context features and the current frame's 's feature map. These maps will be fused through an MLP to obtain the H,
To avoid misunderstanding, we emphasize that our main contribution in global matching lies in the novel auto-trigger mechanism, rather than the global matching itself. Compared to the predictions of previous frame from our model, the localization provided by global matching is less precise. However, its key advantage is the ability to quickly provide a coarse global position, and our approach successfully combines the strengths of both methods.
4 EXPERIMENTS
4.1 DATASETS AND EVALUATION
Training Data. For fair comparison, following the previous methods (Deersch et al. 2023 Karaev et al., 2024b Li al. 2024c b), we trained our model on the Vid-Kri- (Doersch al. (022)
dataset, which consists of 11 11,000 synthetic videos generated by Kubric Engine (Greff et al. 2022).
each containing 24 frames showing 3D rigid objects falling to the ground and bouncing. Each video has 2,048 points sampled on moving objects and backgrounds to be tracked, and their corresponding trajectories are also generated for training. The points to be tracked in the video are occasionally occluded to allow the model to cope with this situation. During training, the resolution of the videos is resized to 384 x 512, and we randomly select 800 trajectories in each video for efficient training.
Evaluation Data. We follow previous methods to evaluate TAPTRy3 on the challenging TAP- Vid (Doersch et al. 2022) benchmark, which consists of 3 subsets, TAP- Vid-Kintssss TAP-Vid- DAVIS, and RGB-Stacking. TAP-Vid-DAVI comprises 30 real-world videos from the DAVIS 2017 validation set (Perazzi et al. 2016). These videos are relatively short, averaging less than 70 frames, so we include the experiments this subset in Section A.3 of the appendix. TAP-Vid-Kines contains 1.144 YouTube videos from the Kinetics-700 700--000-222000 2020 validations set (Carreira & Zisserman 2017), with camera shakes, complex environments, and even scene cuts. These videos are relatively long, averaging about 250 frames per video. -GB-Stacking is a synthetic dataset that captures the process of a robotic arm grasping solid-colored blocks, with an average duration of about 250 frames. Although it has relatively smaller domain gap compared to the training set, which is also synthetic
We use the off-the-shelf PySceneDetect (Castellano) detect the scene cuts.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of TAPTRv3 with prior methods. We use f to indicate the introduction of additional training data. Specifically, CoTrrackr3" additionally incorporates 15K real videos. BootsTAPIRT and BootsTTAPNet train on an additional 15M real videos. Anthro-LocoTraces leverages an extra 1.4K human motion data. TAPTRv3 obtains state-of-the-art performance on most datasets and remains competitive with methods trained on extra internal data. Training data: (Kub24), (Kub48), and (Kub64) refer to Kubric (Greff et al. 2022) with 24, 48, and 64 frames per video, respectively. (PO) PointOdyssey (Zheng al. (023). (FT) HiyingThingee (Mayer et al.) 2016). For fair comparison, we do not utilize auto-triggered global matching here.
Training Data
Kinetics
RGB-Stacckig <. Gang
RoboTAP Gnny
Method
AJ
< 6
OA
AJ
OA
AJ
OA
PIP: (Tallyy al. 2022
FT
31.7
53.7
72,9
59.1
PIPs++ Zibenz a.. R125
PO
63.5
588.5
63.0
TAP Net [LKeesch 2l
X022
Kub24
38.5
54.4
80.6
53.5
68.1
86.3
45.1
62.1
82.9
TAPIR (Durrcc
Kub24
49.6
64.2
85.0
55.5
69.7
88.0
59.6
73.4
87.0
CoTrack]T Kirtt Ft
202-4b
Kub24
49.6
64.3
83.3
67.4
78.9
85.2
58.6
70.6
87.0
TAPTR [ [rara 40240
Kub24
49.0
64.44
85,2
60.8
76.2
87.0
60.1
75.3
86.9
TAPTR tie 20544
Kub24
49.7
85.7
53.4
60.9
74.6
LocoTTack (Cho al RUA4
Kub24
$2.9
66.8
85.3
69.7
83.2
89.5
62.3
76.2
87.1
CoTracker Tonting (arrayy
2024g
Kub64
$4.1
87.1
71.1
60.8
73.7
Track-On (online) (Ayaentr on K026
Kub24
53.9
67.3
87.8
71.4
85.2
91.7
63.5
76.4
89.4
BooksTAPIR) Deensch es al
2024
15M24+15M
54.6
68.44
86.5
70.8
83.0
89.9
64.9
80.1
CoTracker3 (ontine) Karaev
1024g
Kub64+15K
55.8
68.5
88.3
71.7
83.6
91.1
66.4
78.8
90.8
Anttre-LocoTrack
402
Kub24+1.4B 14K
53.9
68844
86.4
64.7
79.2
88.4
BootsTAPNext-B
(ontmee
Aholus
Kub48+15M
TAPTRv3 (online)
Kub24
$4,9
67.5
88,2
72.3
84.1
90.8
64.5
77.3
89.7
dataset, the objects in this dataset often lack texture, making them difficult track. In addition, we also evaluated TAPTRv3 on RoboTAP (Vecerik et al. (2024), which comprises 265 real-world videos from robotic manipulation tasks. The video length in this dataset varies significantly, with the longest videos reaching up to 1,300 frames and the average length of about 270 frames per video.
Evaluation Metries and Settings. We use the standard metries proposed in TAP- Vid (Doersch et al. 2022) for evaluation, including three important metrics. Occlusion Accuracy (OA), describes the accuracy of classifying whether a target tracking point visible or occluded. < &avy. reflecting the average precision of visible points' position at thresholds of 1, 2, 4, 8, and 16 pixels. Average Jaccard (AJ), a comprehensive metric, considers both the precision of position and visibility prediction. Since TAPTRv3 is an online tracker, we use the "First query" mode (Doersch et al. 2022) to evaluate the model, which tracks the target tracking point from the first frame when they are visible until the end of the video. This is much more difficult than the "Strided query" mode for offline trackers. Besides, since the resolution of the input video has a large impact on the final performance, we limit the resolution of the input video to 256 256 for fair comparison with other methods during evaluation.
4.2 IMPLEMENTATION DETAIL
Unlike previous works ( et al. 2024c b). use Resnet-18 instead of Resnet-50 the backbone for higher efficiency. In the Transformer, we employ two encoder layers with deformable attention Zhu al. 202I) to further enhance the image features. While benefiting from our improvement, only 4 decoder layers are required to achieve optimal performance. For the supervision of location and visibility prediction, we utilize the I.1 loss and binary cross-entropy loss, respectively, as previous works. We use the AdamW (Loshchilov) (017)) optimizer with B1 0.9 and 3g 0.999 and set the weight decay to x 10 10-4, We train TAPTRy3 on a cluster of 8 NVIDIA A100 GPUs for about 33,000 iterations with a batch size of 8 in total. To make the training process more stable, we accumulate gradients 4 times to approximate a batch size of 32. After the training of TAPTRv3, we freeze and add the additional global matching for the second stage of training. Since there are only a few parameters to be trained in this stage, it only requires about 5,300 iterations to converge.
For efficiency, in ablation studies in Sec. 4.4 we have few modifications in our experimental settings. We reduce the number of encoder and decoder layers to and 3, respectively, resize the resolution of the input video 384 384, and also reduce the number of tracking points on each video to 200. The ablation studies are conducted on 4 GeForce RTX3090 GPUs for about 33,000 iterations with randomly sampled half-size training and evaluation sets.
7

---

Published as a conference paper at ICLR 2026
Table 2: Ablations on key component of TAPTRv3.
Table 3: Ablation on patch-level simi- larity calculation.
"LTAA" refers Long-Temporal Attention, "Vis-Aware" is short for visibility-awarr, "Re. Win." indicates the removal of the sliding window, and "Sup. Vis." represents only su-
Putch-level Similarity
AJ
< 6asg
OA
pervising the visible points' positions during training.
Element wise Every two point
51.3
64.8
87.4
$2.9
65.9
87.8
Row
LIA
Vis-Aware
Rc. Win.
Sup. Vis.
CCA
AJ
Table 4: Ablation of methods for con- text features updating.
x
X
X
X
X
44.5
2
V
X
X
X
X
47.8
Update Methods
AJ
OA
3
X
X
X
48.8
4
X
X
49.5
VLTA
51.2
64.4
86.1
5
51.1
MLP No Updates
51.7
65.4
87.7
52.9
52.9
65.9
87.8
4.3 COMPARISON WITH THE STATE OF THE ARTS
We evaluate TAPTRv3 the Kinetics, RGB-Stacking, and RoboTAP datasets, which have relatively long videos, and compare it with previous methods to demonstrate its superiority on long videos. As shown in Table 1 TAPTRv3 achieves state-of-the-art on most metries on these three datasets. With our insights and careful designs, TAPTRv3 shows a significant improvement (9.2 AJ on average) compared to TAPTRv2 (Li et al. 2024b) even with more lightweight backbone and fewer decoder layers. Meanwhile, compared with the previous state-of-the- Track-Op (Aydemir et al. 2025) that
uses DINOv2 (Qquab al., 2023) as the backbone, we achieve an average improvement 1.0 AJ.
Although Cotracker3 (Karaev et al. 2024a) and BootsTAPIR (Doersch et al., 2024) achieve re- markable performance, they both introduced extra internal real-world data for training. Specifically, CoTracker3 re-renders the Kubric training set with a length of 64 frames, which narrows the gap in video length between training and evaluation. Then, an additional 15K real-world videos are introduced for fine-tuning. BootsTTPPI trains its model on the original Kubric training set but introduces an extra 15M real-world video clips, which is approximately 1,360 times more than the synthetic data (11K) we use for training. The results show that despite these methods utilizing much more additional data for training, TAPTRv3 still achieves competitive performance. Note that, for fair comparison, do not utilize auto-triggered global matching to help reestablish tracking here. When the global matching is also enabled, the results are further improved, as shown in Table 6 of Sec. 4.4
4.4 ABLATION STUDIES AND ANALYSIS
We start our ablation from TAPTRv2 and conduct ablation studies on every key component in TAPTRv3 to validate their effectiveness. We further conduct some more detailed ablations to investigate the best implementation choice. To focus the ability of TAPTRy3 in handling long vidcos, we conduct ablations on DAP-Vid-Ke Min-Kinntssss.
Visibility-awart Long-Teme Temporal Attention. We first replace the RNN-like long-temporal modeling with the temporall attention. As shown in Table the comparison between Row 2 and Row 1 shows that the replacement provides a large improvement (3.3 AJ), indicating the superiority of the attention mechanism over RNN in handling varying length, which aligns with the findings in modern I.MMs. Furthermore, the comparison between Row and Row 2 shows that enabling the long-temporal attention to utilize visibility prediction to reduce noises caused by occlusion will further improve the performance by 1.0 AJ. The significant improvements brought by the VLTA validate the effectiveness of incorporating richer temporal information.
Removal of the Sliding Window. With VLTA, the model captures temporal information from previous frames, making it redundant to recompute temporal attention within small window. Therefore, we eliminate the sliding window by reducing the window size from 8 1. this case, the original temporal attention in TAPTRv2 will degenerate to an MIP with residual connections. We retain this module in the following ablations for fair comparison. comparison between Row and Row 3 shows 0.7 AJ improvement. We attribute this to better position initialization. More specifically, initializing the position of the target tracking point in the current frame with the estimation from nearby frame simplifies the estimation process. In addition, this modification also enables TAPTRv3 to process input videos in a streaming manner.
shown in Table 2 the
Context-aware Cross-Attention. As shown in Table 2 the comparison between Row 6 and Row 5 shows that the introduction of CCA further brings significant improvement of 1.8 AJ, verifying the effectiveness of CCA in improving the robustness of the spatial feature querying.
8

---

Published as a conference paper at ICLR 2026
Table 5: Ablation on the input positions Table 6: Ablation on global matching, Whether to use of decoder.
auto-triggered global matching.
Input Positions of Decoder
AJ
< 6
QA
Input Positions of Decoder
Datuset
AJ
OA
Global Matching Calculation
51.1
64.0
86.5
Previous Frame's 's Prediction
Kin.
54.9
67.5
88.2
Previous Frame's & Prediction
$2.9
65,9
87.8
Global Matching S.C.
Kin.
55.1
67.7
88.4
Only Supervise the Position of Visible Points. It is worth noting that predicting the position of the target tracking point when is occluded is an ill-posed problem. Forcing the model to localize the occluded point can destabilize the learning process and may lead the model learn bias toward a fixed motion pattern. As shown in Rows 5 and 4 of Table 2 simply ignoring the supervision of invisible points' location predictions results in an improvement of 1.6 AJ.
Patch-level Similarity Calculation in CCA. As shown in Table 3 we conduct comparative experi- ments on different methods to obtain patch-leve level similarity. The results show that the "Element-wise" which only considers the similarities between two points that are located at the same position in the two patches, is less effective than the "Every two point" strategy that adopted our current CCA. This is because the "Every two point" strategy has an advantage in handling more complex spatial variations, such as rotation, and more tolerant of the sampling point's location. For more details, please refer to Sec. A.. in our appendices.
Spatial Context Updating. As shown in Table neither utilizing our VLTA nor an MLP to update the query's s context features yields better results. This indicates that maintaining the target tracking points' spatial context throughout the tracking process not only reduces computation costs but also helps spatial feature querying. For more details, please refer to Sec. A.4 in our technical appendix.
Ablation on Auto-Triggered Global Matching As discussed in Sec. 3.4 TAPTRv3 triggers global matching only when scene cut is detected. This design is based on the empirical finding that naively applying global matching at every frame results in inferior performance, as shown in Table 5 Therefore, TAPTRR3 defaults to using the previous frame's prediction for initialization. It activates global matching only when a scene cut is detected, which serves to re-establish tracking and prevent failure. This automatic trigger mechanism boosts the performance of our best model by 0.2 AJ, as shown in Table 6 We further construct a subset of Kinetics by selecting all videos with scene cuts. On this subset, this module brings an improvement of 0.5 AJ in the same experiment setting, confirming its effectiveness. More details can be found in Sec. A.1 of the appendix.
VISUALIZATION
We select a real-world video with 351 frames to showcase the improvement of our model. The video captures very long train passing through the scene. At the beginning of the video, generate query points using a 40 x 40 grid over the foreground region (excluding the sky). This example presents a typical long-term occlusion scenario of nearly 300 frames, posing a considerable challenge to model robustness. As shown in the Fig. B TAPTRv2 makes large-scale visibility mispredictions once the train enters the frame, and after the train leaves, its predicted point locations become unstable, with most tracks lost. In contrast, TAPTRv3 maintains stable and accurate tracking throughout the entire video, correctly predicts visibility during occlusion, and recovers accurate point locations once the train leaves the frame. This result highlights the effectiveness of our proposed components in mitigating feature drifting and achieving more accurate and robust tracking. Notably, for fair comparison, the global matching mechanism in TAPTRv3 was disabled.
CONCLUSION
In this paper, we have presented TAPTRv3, a strong method for the TAP task. TAPTRv3 improves TAPTRv2 primarily by developing the Context-aware Cross-Attention (CCA) and Visibility-awan Long-Tempont Attention (VLTA) to address the shortage of feature querying. CCA improves key- aware deformable attention by leveraging spatial context, which helps point-level tasks obtain more robust and accurate attention weights for updating both features and positions. VLTA replaces the RNN-like long-tempe temporal modeling with an attention mechanism, enabling the perception of longer temporal context and mitigating the issue of feature drifting. VLTA further utilizes the visibilities to improve the quality of long-tempora attention, leading to a better feature querying ability along the temporal dimension. Additionally, TAPTRy3 further improves its performance in long videos
9

---

Published as a conference paper at ICLR 2026
TAPTRv2
TAPTRV3
T-4
T=25
T-205
T-295
Figure 3: Visual comparison between TAPTRv3 and TAPTRv2.
by introducing the auto-triggered global matching mechanism. With the help of our insights and these novel designs, TAPTRv3 surpasses TAPTRv2 by a large margin and obtains state-of-the- of-the-art performance on multiple challenging datasets. Even when compared with the methods trained on extra internal data, TAPTRv3 remains competitive.
ETHICS STATEMENT
Our work focuses developing algorithms for tracking any points in videos, which fundamental computer vision problem with broad applications in robotics, video editing, and manipulation. This research relies exclusively on publicly available datasets and does not involve collecting or annotating data containing personal or sensitive information. All of our experiments are limited to standard academic benchmarks that are widely adopted the community. We acknowledge that point tracking techniques could potentially be misused for surveillance or privacy-invasive invasive applications. However, our intent is to advance the scientific understanding of visual correspondence and to enable positive downstream applications such as video editing, motion analysis, and embodied AI. We encourage responsible use this technology in line with community norms and ethical guidelines.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure the reproducibility TAPTRv3. Our method described in detail in Sec(3] including the model design, architecture, and inference procedure. The training settings and hardware environment are provided in See 4.2, Additional implementation details and hyperparameter choices can be found in SecIA.4iin in the appendix. The datasets used for training and evaluation are introduced in Sec 4.T all of them are publicly available and widely adopted benchmark datasets. All source code and model weights will be released to the public upon acceptance of this paper to further support research and development in the community.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Görkay Aydemir, Xiongyi Cai, Weidi Xie, and Fatma Güney. Track-On: Transformer-basee On- line Point Tracking with Memory. The Thirteenth International Conference on Learning Representations, 2025.
Weikang Bian, Zhaoyang Huang. Xiaoyu Shi, Yitong Dong, Yijin Li, and Hongsheng Li. Context- pips: persistent independent particles demands spatial context features. Advances in Neural Information Processing Systems, 06:55285-55225 2023.
M.J. Black and P. Anandan. A Framework for the Robust Estimation of Optical Flow. In 1993 (4th) International Conference on Computer Vision, Dec 2002.
Andrés Bruhn, Joachim Weickert, and Christoph Schnörr. Lucas/kanade meets homn/schucck: com- bining local and global optic flow methods. International Journal of Computer Ission,nnterational Journal of Computer Vision, Feb 2005.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-Ennd Object Detection with Transformers. In European conference on compuler vision, pp. 213-229. Springer, 2020.
Joao Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A New Model and the Kineties Dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2017.
Brandon Castellano.
PySeeneDetect.
URL
PySceneDetect
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision Transformer Adapter for Dense Predictions. arXiv preprint arXiv:2205.08534, 2022.
Seokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Seungryong Kim, and Joon- Young Pair Correspondence for Point Tracking. European Conference on Computer Vision. Springer, 2024.
Joon-Young Lee. Local All-
Carl Doersch, Ankush Gupta, Larisa Markeeva, Adrià Recasens, Lucas Smaira, Yusuf Aytar, João Carreira, Andrew Zisserman, and Yi Yang. TAP-Vid: A Benchmark for Tracking Any Point in Video. Advances in Neural Information Processing Systems, 35: 3610--166,,,,, 2022.
Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. TAPIR: Tracking Any Point with Per-frame Initialization and Temporal Refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10061-10072, 2023.
Carl Doersch, Yi Yang, Dilara Gokay, Pauline Luc, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ross Goroshin, João Carreira, and Andrew Zisserman. BootsTAP: Bootstrapped Training for Sracking-Any- Any-Point. arXiv preprint arXiv:2402.008877 2024.
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. FlowNet: Learning Optical Flow with Convolutional Networks. In Proceedings of the IEEE international conference computer vision, pp. 2758-2766 2015.
Google. Bard. bttes://bbrd.googe com/ 2023.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Hemmann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3749-3761, 2022.
Albert Gu and Tri Dao. Mamba: Linear- Time Sequence Modeling with Selective State Spaces. In First conference language modeling, 2024.
11

---

Published as a conference paper at ICLR 2026
Adam Harley, Yang You, Yang Zheng, Xinglong Sun, Nikhil Raghuraman, Sheldon Liang, Wen- Hsuan Chu, Suya You, Achal Dave, Pavel Tokmakov, et al. TAG: Tracking at Any Granularity. ArXiv Preprint, 3:1, 2024.
Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories. In European Conference on Computer Vision, pp. 59-75. Springer, 2022.
Berthold K.P. Horn and Brian G. Schunck. Determining Optical Flow. Artificial Intelligence, pp. 185-203, Aug 1981.
Jiahui Huang, Leonid Sigal, Kwang Moo Yi, Oliver Wang, and Joon-Young Young Lee. INVE: Interactive Neural Video Editing. arXiv e-prints, pp. arXiv-2307, 2023.
Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung. Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A Transformer Architecture For Optical Flow. In European conference on computer vision, pp. 668-685. Springer, 2022.
Eddy llg. Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017.
Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards Generic Object Detection via Text-visual Prompt Synergy. In European Conference on Computer Vision, pp. 38-57. Springer, 2025.
Shihao Jiang, Yao Lu, Hongdong Li, and Richard Hartley. Learning Optical Flow from a Few Matches. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2021.
Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. arXiv preprint arXiv:2410. 1831, 2024a.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker: It is Better to Track Together. In European Conference on Computer Vision. Springer, 2024b.
Inès Hyeonsu Kim, Seokju Cho, Jahyeok Koo, Junghyun Park, Jiahui Huang. Joon-Young Lee, and Seungryong Kim. Learning to Track Any Points from Human Motion. arXiv preprint arXiv:2507.06233, 2025.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer Normalization. ArXiv e-prints, pp. arXiv-1607, 2016.
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. DN-DETR: Accelerate DETR Training by Introducing Query DeNoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13619-13627, 2022.
Feng Li, Ailing Zeng, Shilong Liu, Hao Zhang, Hongyang Li, Lei Zhang, and Lionel M Ni. Lite DETR: An Interleaved Multi-Scale Encoder for Efficient DETR. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18558-18561 2023.
Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Jianwei Yang, Chunyuan Li, et al. Visual In-context Prompting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12861-12871, 2024a.
Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Feng Li, Tianhe Ren, Bohan Li, and Lei Zhang. TAPTRv2: Attention-bass based Position Update Improves Tracking Any Point. Advances in Neural Information Processing Systems, 2024b.
Hongyang Li, Hao Zhang. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, and Lei Zhang. TAPTR: Tracking Any Point with Transformers as Detection. In European Conference on Computer Vision, pp. 57-75. Springer, 2024c.
12

---

Published as a conference paper at ICLR 2026
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR. In International Conference Learning Representations, 2022.
I Loshchilov, Decoupled Weight Decay Regularization. arXiv preprint arXiv: 2017.
Nikolaus Mayer, Eddy llg. Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. In Proceedings of the IEEE conference on computer vision and pattern 2016.
recognition, pp.
Michal Neoral, Jonáš Serych, and Jifí Matas. MFT: Long-Term Tracking of Every Pixel. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6837- 6847, 2024.
OpenAL. ChatGPT. htttps://ppena..com/bllg///
2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. DINOv2: Learning Robust Visual Features without Supervision. arXiv preprint arXiv:2304.07199, 2023.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao. Xin Cheng, Michael Chung. Matteo Grella, et al. RWKV; Reinventing RNNs for the Transformer Era. arXiv preprint arXiv:2305. 13048, 2023.
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung A Benchmark Dataset and Evaluation Methodology for Video Object Segmen- tation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2016.
Peter Sand and Seth Teller. Particle video: Long-range motion estimation using point trajectories. International journal of computer vision, 80:72-91, 2008.
Saurabh Saxena, Charles Herrmann, Junhwa Hur, Abhishek Kar, Mohammad Norouzi, Deqing Sun, and David J Fleet. The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation. Advances in Neural Information Processing Systems, 36, 2024.
Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12469-122 12480, 2023a.
Xiaoyu Shi, Zhaoyang Huang. Dasong Li. Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1599-1610, 2023b.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing, 568: 167:63, 2024.
Deqing Sun, Xiaodong Yang, Ming-Yu Yu Liu, and Jan Kautz, PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018
Zachary Teed and Jia Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In Computer Vision Eicton-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16, pp. 402-419. Springer, 2020.
Timo Teufel, Hongchao Shu, Roger D Soberanis-Mukut Jan Emily Mangulabnan, Manish Sahu, Swaroop Vedula, Masaru Ishii, Gregory Hager, Russell H Taylor, and Mathias Unberath. OneS- LAM to Map Them All: A Generalized Approach to SLAM for Monocular Endoscopic Imaging Based on Tracking Any Point. International Journal of Computer Assisted Radiology and Surgery, pp. 1-8, 2024.
13

---

Published as a conference paper at ICLR 2026
Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. DINO-Tracke Taming DINO for Superviie Point Tracking in a Single Video. In European Conference on Computer Vision, pp. 367-385. Springer, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. Advances in neural information processing systems, 30, 2017.
Mel Vecerik, Carl Doersch, Yi Yang. Todor Davchev, Yusuf Aytar, Guangyao Zhou, Raia Hadsell, Lourdes Agapito, and Jon Scholz. Robotap: Tracking Arbitrary Points for Few-shot Visual Imitation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp- 5397-5403. IEEE, 2024.
Jinnyuan Wang, Yiran Zhong, Yuchao Dai, Kaihao Zhang, Pan Ji, and Hongdong Li. Displacement- Invariant Matching Cost Learning for Accurate Optical Flow Estimation. Cornell University- arXiv, Comelll University arXiv, Oct 2020.
Qianqian Wang, Yen-Y Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking Everything Everywhere All at Once. ICCV, 2023.
Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. SpatialTracker: Tracking Any 2D Pixels in 3D Space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20406-20411 2024.
Haofei Xu, Jiaolong Yang, Jianfei Cai, Jie Zhang, and Xin Tong. High- Resohueeso Optical Flow from ID Attention and Correlation. Cornell University arXiv, Cornell University arXivv Apr 2021.
Jia Xu, Rene Ranftl, and Vladlen Koltun. Accurate Optical Flow via Direct Cost Volume Processing, In 2017 IEFE Conference on Computer Vision and Puttern Recognition (CVPR), Jul 2017.
Feihu Zhang, Oliver J. Woodford, Victor Prisacariu, and Philip H. S. Torr. Separable Flow: Learning Motion Cost Volumes for Optical Flow Estimation. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2021.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. In The Eleventh International Conference on Learning Representations, 2023a.
Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Chunyuan Li, Jainwei Yang, et al. Llava-Grounding: Grounded Visual Chat with Large Multimodal Models. In European Conference on Computer Vision, pp. 19-35. Springer, 2025.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention. arXiv preprint arXiv:2303.161111 2023b.
Shiyu Zhao, Long Zhao, Zhixing Zhang, Enyu Zhou, and Dimitris Metaxas. Global Matching With Overlapping Attention for Optical Flow Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17601, 2022.
Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J Guibas. PointOdyssey: A Scale Synthetic Dataset for Jerm-Term Point Tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19855-19865, 2023.
Artem Zholus, Carl Doersch, Yi Yang, Skanda Koppula, Viorica Patraucean, Xu Owen He, Ignacio Rocco, Mehdi SM Sajjadi, Sarath Chandar, and Ross Goroshin. TAPNext: Tracking Any Point (TAP) as Next Token Prediction. arXiv preprint arXiv:2504.05577 2025.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable Transformers for End-to-End Object Detection. In ICIR 2021: The Ninth International Conference on Learning Representations, 2021.
14

---

Published as a conference paper at ICLR 2026
A TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL
In this appendix, we provide more ablation studies on the design of our key components and some hyper-parameters (see Sec. A.Ip. Beyond demonstrating the superior performance of our model, we also analyze its efficiency, including inference speed and GPU memory usage (see Sec. A. To comprehensively evaluate model, we also conduct additional tests on the relatively short DAVIS dataset in TAP-Vid (Doersch et al. (022)) benchmark and the extremely long PointOdyssey ZZeng al. 2023) benchmark (see Sec A.3). Furthermore, We provide more details on the implementation and experimental settings (see A.c.A.4). In addition, we also discuss the limitations of our model to provide comprehensive understanding (see Sec(A.5) To address potential concerns. we also provide additional discussions on the model design (see Sec. (.6)). In accordance with the requirements, we include statement on the use of Large Language Models (see Sec. A.7). Finally, we present some more visualization results to show the superiority of TAPTRv3 and the effectiveness of our designs (see Sec. A.8 and the supplementary videos).
A.1 MORE ABLATION STUDIES
sample N2 features in the form of a grid with l-pixel grid interval when preparing the point query's s context features C and each sampling point's context features KP" in cross-attention. At different scales of the feature map, the grid interval is the same. This allows us to fuse information from receptive fields of different sizes. As N increases, it effectively means that the patch size becomes larger, and the context features tend to represent more global information. To find the optimal value of N, we conduct experiments with N = 10 N 3, and N 5 using the same settings as Sec. [4.4 The results are shown in Table 7 where N = 3 obtains the best performance. Compared with sampling 25 context features (N- = 5). sampling 9 context features (NV = 3) around the point not only requires fewer computing resources but also yields better performance, with an AJ improvement of 0.7. These results indicate that for the task of point tracking, context features are important, but they should not be so excessive or represent too large areas of the image as this could prevent accurate description of the tracked points.
Number of Context Features. As discussed in Sec. 3.1 and Sec. 3.3
Table 7: Ablation number of context features N
Number of Context Features N2
AJ
< ang
OA
2 1
51.3
64.6
87.4
2
52.9
65.9
87.8
2 25
52.2
65.8
87.8
Memory Size of VLTA. TAPTRr3 eliminates the sliding window (Li et al. 2024c b) and introduces the Visibility-aware Loog-TTempol Attention (VLTA) module to extend the temporal attention an arbitrary length, while considering the visibility. Excluding the use of visibility, the recent SAM2 adopts similar temporal modeling approzch, where SAM2 maintains a memory to record features from past frames in FIFO manner and limits the memory size to 8 to focus on recent frames. To demonstrate the advantage of extending temporal attention to arbitrary lengths and the generalization ability of our VLTA to temporal lengths, we conduct ablation studies on the memory size. Specifically, we use the trained TAPTRv3 model from See 4.3 and perform evaluations on the Kinetics dataset with different memory sizes. As illustrated in Table g the significant positive correlation between memory size and model performance indicates that, although the perception range of our VLTA is limited to 1 to 23 frames during training 6 our VLTA is still able to generalize beyond 24 frames. This generalization ability enables us to expand the range of temporal perception, allowing the collection of long-term temporal information from all past frames help improve the robustness of long-term point tracking.
'When N = 1. the CCA will actually degenerate to the vanilla key-eware deformable attention, we can also find this performance in Row 5 of Table "iikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R adle, Chloe Rolland, Laura Gustafson, et al. SAM 2: Segment Any-thing in Images and Videos. arXiv preprint Arx: 00714, 2024. 12.
'Because our training data are videos with fixed lengths of 24 frames
15

---

Published as a conference paper at ICLR 2026
Table 8: Ablation on the memory size of VLTA (Kinetics).
Memory Size of VLTA
AJ
8 avg
OA
12
51.9
65.2
86.5
24
53.1
66.3
87.3
48
54.5
67.2
88.0
All Past
54.9
67.5
88.2
However, unconstrained temporal memory size will lead to CUDA out-of-memory problems in practical applications. We find that when processing videos with up to 3,500 frames, the GPU memory requirement reaches 24GB because of the large amount of cached temporal memory. To balance performance and efficiency, and enable the model handle videos of arbitrary length, we need to limit the size of the temporal memory and manage it with FIFO mechanism. To this end, we further evaluated RoboTAA (with videos up 1,300 frames) with more diverse memory sizes. As shown in Fig. 4 the performance converges when the memory size reaches 512. With the help of temporal memory management, TAPTRv3 is capable of handling downstream applications, which usually require the model to process online streaming videos.
64.2
64.4
64.5
64.5
62.5
62
60.3
57.5
56
Temporal Memory Size
Figure 4: Ablation on the memory size of VLTA (RoboTAP).
The Method of Obtaining Similarity Map in Global Matching. In Sec. 3.4 we propose the auto-triggered global matching to reestablish point tracking when a scene cut is encountered to prevent the loss of tracking targets. Instead of simply relying point-level feature to construct a similarity map in global matching as in previous works (Cho et al.) 2024 Doersch et al., 2023). we propose incorporating spatial context features, similar the CCA module, enhance accuracy. (For more implementation details, please refer to Sec, [A44). To investigate the effectiveness of this approach, conduct an ablation study on it. As shown Table 9 the incorporation of spatial context features brings relatively small performance improvement. However, considering that we only trigger global matching when scene cut detected and its computational cost negligible, we adopt the use of spatial context features in TAPTRv3.
Table 9: Ablation on calculation of similarity map.
Feature of Tracking Point
AJ
80eg
OA
Point-level Feature
55.75
67.88
87.57
Spatial Context Features
55.79
67.94
87.57
To avoid misunderstanding, emphasize that main contribution in global matching lies in the novel auto-trigger mechanism, rather than the global matching itself.
More Ablation Auto-Trrsn TriigTrieeeeed Global Matching.
16

---

Published as a conference paper at ICLR 2026
We evaluate the proposed auto-triggered global matching mechanism the full Kineties dataset. As shown in Table while the method yields performance gains, the improvement is relatively modest. This is largely because only subset of videos in Kinetics contain scene cuts, and thus actually activate the mechanism, whereas evaluation metrics are averaged over the entire dataset. To better isolate the effectiveness of our approach, Kinetics that contain at least one scene cut, accounting for approximately 27% of the 1,144 total videos. We conduct the same ablation study on this subset. As shown in Table TO triggering global matching upon detecting scene cuts leads to a significant improvement (0.5 AJ), demonstrating the effectiveness of our method in relevant scenarios.
construct scene-cut subset comprising all videos in
Table 10: Ablation on auto-triggenn global matching, Whether to trigger the global matching when encountering scene cuts on the Kinetics scene-cut subset. "Kin." short for TAP- Vid Vid-Kinetics, and "S.C." indicates scene cuts
Input Positions of Decoder
Dataset
62
OA
aug
Previous Frame's ş Prediction Global Matching if S.C.
Kin, S.C.
55.3
67.0
86.9
Kin. w/ S.C.
55.8
67.9
87.6
A.2 EFFICIENCY ANALYSIS
Inference Speed. Notably, while TAPTRv3 introduces two additional modules, CCA and VLTA, which increase computational overhead compared to TAPTRv2, the inference speed of TAPTRv3 is faster than that of TAPTRv2. We benchmark the runtime performance dataset using a single GeForce RTX3090 GPU, measuring the total number of video frames and the model's ' inference time calculate the average FPS. We also include CoTracker (Karaev et al. 2024b), which is a highly representative and influential method, as a baseline. For all experiments, we set the input video resolution to 384 x 512, with an average of 22 tracked points per video. Each experiment was repeated five times to report stable average. For fair comparison, we set the window size to and the window stride to 4 for all models. shown Table under the same conditions, TAPTRy3 achieves 15 FPS higher than TAPTRv2.
the TAP- TAP-Vid-DAVIS
Table 11: Average FPS of TAPTR and TAPTRv3.
Method
Average FPS
Method
Average FPS
Method
Average FPS
Cotracker
26.4
TAPTRv2
41.9
TAPTRv3
57.2
We attribute the faster speed to the following reasons. (1) The CCA module introduces only 9 more sparse sample points, with the additional computational overhead negligible compared to the overall network. (2) While we introduce VLTA, we also remove the Temporal Attention module and the complex Window Post-processing TAPTRv2. (3) TAPTRv3 uses a more lightweight backbone (Resnet-18 18 vs. Resnet-50) and fewer decoder layers (4 layers vs. 5 layers), making the model more efficient. Additionally, it utilizes lower-resolution training data (384 x 512 vs. 512 x 512), resulting in lower-resolution inputs during inference well.
GPU Memory Overhead. When the memory size in VLTA is set to 512 (as discussed See A.I). our model is capable of processing videos of arbitrary length in a streaming manner. When tracking 100 points simultaneously a streaming video, the GPU memory usage is less than 2GB, making its deployment cost-effective.
A.3 MORE COMPARISONS
Comparisons on DAVIS Benchmark, We compare TAPTRv3 with previous methods on the DAVIS subset of the TAP- Vid (Doersch et al. (002) benchmark. DAVIS contains relatively short videos, with average length of only 70 frames. As shown in Table [3] shorter videos TAPTRv3 performs comparably to TAPTRv2. Although it is slightly behind AJ, there still an improvement in < aang indicating better localization capability. The result is reasonable, as design is mainly intended to
17

---

Published as a conference paper at ICLR 2026
Table 12: Comparison of TAPTRv3 with different backbone and training resolution.
Kinetics < Bing
RGB-Stacking
RoboTAP
DAVIS
Method
AJ
OA
AJ
< Gag
OA
AJ
< &.ggg
OA
AI
< Gay
OA
TAPTRv3 (Resnet-50, 512 512)
54.5
67.5
88.2
73.0
86.2
90.0
64.6
77.2
90.1
63.2
76.7
91.0
TAPTRv3 (Resnct-18, 384 512)
54,9
67.5
88.2
72.3
84.1
90.8
64.5
77.3
89.7
63.2
76.4
90.6
Table 13: Comparison of TAPTRv3 with prior methods. We use to indicate the introduction of additional training data. Specifically, CoTrrackr3 additionally incorporates 15K real videos. BootsTAPIR and BootsTAPNext-B train on an additional 15M real videos. Anthro-LocoTrackt leverages an extra 1.4K human motion data. TAPTRv3 obtains state-of-the-art performance on most datasets and remains competitive with methods trained on extra internal data. Training data: (Kub24), (Kub48), and (Kub64) refer to Kubric (Greff et al. 2022) with 24, 48, and 64 frames per video, respectively. (PO) PointOdyssey (Zheng al. 2023). (FT) Things++++++++ (Mayer et al. 2016). For
fair comparison, we do not utilize auto-triggered global matching here.
Training Data
DAVIS 6%
Method
AJ
OA
PIPs (Harley al.
2022
FT
42.2
64.8
77.7
PIPs++ Zheng et al. 2023
PO
69.1
TAP-Net Doersch
2022
Kub24
33.0
48.6
78.8
TAPIR DDoersch
Kub24
56.2
70.0
86.5
CoTracker KKarey al.
2024b
Kub24
61.8
76.
88.3
TAPTR (Li
20/44
Kub24
63.0
76.
91.1
TAPTRv2 (Li 20244b
Kub24
63.5
75.9
91.4
LocoTrack (Cho et al. 2024
Kub24
63.0
75.3
87.2
CoTracker3 (online) (Karaev et al.
2024a
Kub64
64.5
76.7
89.7
Track- (online) (Aydemir et al.
2025
Kub24
65.0
78.0
90.8
ToPIS [Dersch et al 2024
Kub24+15M
61.4
73.6
88.7
CoTracker3 (online)) Karaey 2024a
Kub64+15K
63.8
76.3
90.2
Lochro-LocoTrack!
Kim al
Kub24+1.4K
64.8
77.3
89.1
BootsTAPNert (online) Zholus et al.
2025
Kub48+15M
65.2
78.5
91.2
TAPTRv3 (online)
Kub24
63.2
76.4
90.6
improve tracking performance on long videos, which has already been validated in Table I of the main paper.
2023) is benchmark Comparisons on PointOdyssey Benchmark. PointOdyssey Zheng et al. composed of synthetic data, and is noticeably more realistic than TAP- Vid- Kubric. A key character- istic of PointOyyssey is its extremely long video sequences: the average length in the test set reaches 2386 frames, and the longest video contains 4325 frames, which far exceeds the video lengths the TAP-Vid Vid benchmark. Since method is designed address challenges in long videos, evaluation on PointOdyssey is highly meaningful.
To this end, directly evaluated TAPTRv3 trained only the Kubric dataset the PointOdyssey test set (including 12 valid videos). This allows us to assess the model's performance on longer and more challenging sequences, while also providing insight into its generalization ability. We report the metrics proposed in PointOdyssey, including daog 50ig and 8ocg, which measure localization accuracy. The metrics 5016 and goec are computed in the same way as dang but consider only visible and occluded points, respectively. In addition, we also report the Survival metric, which measures the average number of frames before tracking failure, where failure is defined as error exceeding 50 pixels. The results are shown in the Table
Even without being trained on PointOdyssey, TAPTRv3 achieves highly competitive performance. More importantly, the improvement compared to TAPTRv2 is extremely obvious, especially a 9.0 point gain the 8ois metric, which measures tracking accuracy visible points. The experimental result validates that our insight into the limitations of TAPTRy2 long videos is correct, and our proposed modules are highly fruitful.
18

---

Published as a conference paper at ICLR 2026
Table 14: Comparison of TAPTRv3 with prior methods on PointOdyssey.
Training Data
PointOdyssey dang
Method
Öaeg
Kxeg
Survival
TAP-Net (Doersch et al.
2022
PointOdyssey
28.4
18.3
PIPs (Harley ct al. 2022
PointOdyssey
27.3
42.3
PIPs++ Zheng et al. 2023
PointOdyssey
29.0
32.4
18.8
47.0
CoTracker (Karaey al.
2024b
PointOdysssy
30.2
32.7
24.2
55.2
Track-On (Aydeiir
2025
Kubric
34.2
38.1
49.5
TAPTRv2
al. 2024b
Kubric
26.1
28.4
21.5
50.0
TAPTRv3
Kubric
33.9
37.4
26.0
51.3
A.4 MORE INPLEMENTATION DETAILS
Backbone and Training Resolution.
As discussed in Sec. 4.1 and Sec. 4.2 compared to TAPTR and TAPTRv2, which use Resnet-50 the backbone to extract image features, TAPTRv3 adopts the more lightweight Resnet-18, Additionally, TAPTRv3 is trained resolution used by TAPTR and TAPTRv2. This primarily because our experiments showed that a lighter backbone and lower training resolution achieve comparable performance, indicated by the results in Table T2 These modifications improve the efficiency of our model and also demonstrate the superiority of the decoder design.
videos with resolution of 384 512, which is lower than the 512 512
Calculation of Similarity Map in Global Matching, In the global matching module of TAPTRv3, construct similarity map H, E Hx between the target tracking point and the feature map X, E ffxxxxx xD of the current frame. However, instead of solely relying on point-level feature as in previous methods, similar CCA, we leverage the spatial context features C E & xD to help improve the accuracy the similarity map H,. As illustrated in Fig. 5 in our global matching, we first utilize each feature of the target tracking point's context feature to compute group of similarity maps H₁ E # x² However, since each feature in the context features is still point-level, computing the similarity map using any single one of them independently still introduces noise. Therefore, we further employ an MLP comprehensively integrate these noisy similarity maps. resulting in more accurate one H,
B e altar
R
MI.P
go
Figure 5: Detailed visual illustration of global matching.
Patch-level Similarity Calculation in CCA. Sec. 3.3 briefly introduces how to calculate patch-level similarity in Context-aware Cross-Attention (CCA), and Sec. 4.4 presents an ablation study on In this section, we provide more detailed description. For the "Every two points" method that adopted TAPTRv3 by default, as shown in Fig (a), each feature of the point query's 's 2 context features is paired with every feature in the sampling point's 2 context features KTn obtain
19

---

Published as a conference paper at ICLR 2026
S/"h E R x For the "Element-wir wise" method (See Sec. 4.4 and Table 9 shown in Fig 6 (b), each feature of the point query's 's N context features is only paired with its corresponding one in K?" to obtain Sy" E R
[0] 2X
Flatten & MLP
S11 2
SPE
x,
1 1JE 2
(a)
Flatten MLP
ST
(b)
Figure 6: Detailed visual illustration of different methods for computing patch-level similarity.
f
f
4-2
f
C
AC
C₂
Dut Prod.
d'
Softmax
Go
ag
ag
Reweight
d
(a)
C,
C,
Concalenation
MLP
AC,,
(b)
Figure 7: Detailed visual illustration of different methods for the updating of spatial context features.
Spatial Context Updating. In TAPTRv3, we consistently employ the initial spatial context features C throughout all decoder layers. To validate its effectiveness, we add a spatial context feature update module in the transformer decoder for comparison in our ablation study section (See Sec. 4.4 and Table 4). Here, we provide a detailed description of the two methods for updating spatial context features proposed in this ablation. For the one that nith spatial context feature C,,, we treat it as a query and attend VLTA to the refined content features of the past frames update it. For the one that uses an MLP, shown in Fig. (b), concatenate
VLTA, shown in Fig. (a), for the
20

---

Published as a conference paper at ICLR 2026
the spatial context feature with the point query's content feature in the current frame fj and update the spatial context features with the MLP.
Backbone and Number of Encoder Layers. We further conduct experiments the TAP- Vid Doer sch et al. 2022) benchmark to investigate the influence of the backbone and encoder layers on model performance, reporting the Average Jaccard (AJ) metric. The results are presented in Table 15
First, we employ larger-capacity backbone, ViT- TTT-Adapter Chen et al. (2022) with DINOv2, and report the results in Row 3 of [3] Compared with ResNet18 (Row 2) and ResNet50 (Row 1), the performance decreases on all datasets, with a particularly notable drop on the synthetic RGB-Stacking dataset. Unlike methods based other architectures, do not observe that using backbone in our DETR-liL model yields further performance gains.
advanced
Table 15: The impact of different backbone and encoder layers model performance
Kinetics AJ
RGB-Stacking
DAVIS
Row
Method
Backbone
Enc. Layers
AJ
AJ
TAPTRv3 (Ours)
ResNel18
2
54.9
72.3
63.2
2
3APTRv3 (Ours)
ResNet50
2
54.5
73.0
63.2
3
3APTRv3 (Ours)
VIT Vda-ter with DINOv2
2
54.2
67.8
62.2
4
TAPTRv3 (Ours)
ResNet18
I
53.5
66.7
61.6
TAPTRv3 (Ours)
ResNet18
3
51.9
62.6
58.7
TAPTRv2
ResNet18
2
45.6
46.7
60.0
TAPTRv2
ResNet50
2
49.7
53.4
63.5
Furthermore, since our model utilizes a DETR-like structure, it includes encoder layers in addition the backbone further aggregate features. To better understand this behavior our model, we additionally conduct experiments examining the impact of encoder layers and summarize the results in Rows 1, 4, and of Table 15| When using two encoder layers, our model achieves the best performance, and the impact of encoder depth is substantially larger than that of the backbone. We believe that using only a single encoder layer may lead to insufficient feature extraction, making the model capacity too limited relative the task diffieulty. In contrast, using three encoder layers noticeably slows convergence and increases training difficulty. is worth noting that replacing the backbone adjusting encoder depth typically requires extensive hyperparameter tuning (e.g., learning rate and weight decay) to achieve optimal performance. We did not perform extensive hyperparameter tuning here and leave it for future exploration.
Finally, we reduce the model size of TAPTRv2 by replacing the ResNet50 backbone with ResNet1 8, following the design choice in TAPTRv3. As shown in Rows 6 and 7 in Table 15 the model exhibits a significant performance drop with the lighter backbone, which, to some extent, reflects the effectiveness of newly introduced components in the decoder.
A.5 LIMITATIONS
TAPTRv3 is an online tracker that can process streaming video input, meaning that the decoder handles only one frame at time. This design leads very low computational cost (see Sec. A.2) but comes with a limitation in parallelism. Even with sufficient GPU memory, the model can not fully utilize it. As result, TAPTRv3 may appear slower than offline methods when evaluated in offline settings, since those methods can process multiple frames simultaneously in single forward pass, while TAPTRv3 requires multiple passes. However, TAPTRv3 can still adapted for offline use by simply applying sliding window strategy with increased window size to improve parallelism. While this adjustment may incur a slight performance drop, it provides a flexible trade-off between speed and accuracy depending on the application scenario.
A.6 MORE DISCUSSIONS
Discussion of Using A Attention in VLTA. Recently, readily applicable RNN-like architectures such Mamba Gu & Dao (2024) and RWKV RenV al. (2023) have demonstrated strong potential and efficiency. However, we choose to use attention rather than these more advanced RNN- structures in the VLTA module based the following considerations.
21

---

Published as a conference paper at ICLR 2026
One of our insights is that the information from the initial frame should serve the most reliable anchor and be preserved throughout tracking. Although Mamba and RWKV improve long-range information propagation through state compression and selective forgetting, they fundamentally rely on a single evolving hidden state to carry information over time. Regardless of how efficient the mechanism is, this hidden state represents lossy compression of the entire history. In long videos, a point may disappear for hundreds of frames, during which the fine-grained appearance details stored in the hidden state may degrade or become contaminated. In contrast, our approach keeps the initial-frame anchor intact and uses attention to directly retrieve information from historical frames to accommodate appearance changes. Visibility prediction is further incorporated to ensure that more reliable historical information is aggregated.
In summary, we believe that for the video point tracking task, hidden-state evolution is less suitable for handling the complex challenges of long videos. While attention may incur higher computational cost, by appropriately controlling the size the historical buffer, our model achieves a good balance between performance and efficiency. As described in Sec. A.2 TAPTRv3 can process arbitrarily long videos in a streaming manner (tracking 100 points simultaneously with less than 2GB GPU memory), making it deployment cost-efficient.
Discussion of the Effectiveness of Visibility-wware Attention. As described in the main text, we apply the predicted visibility in a soft weighting manner, where frames predicted as more likely to be invisible are down-weighted according to confidence rather than discarded. When visibility predictions are accurate, this weighting is naturally beneficial. When predictions incorrect, two cases may occur. If visible point is predicted as invisible, the attention weights for these frames reduced, leading the model to extract less historical information from them. As result, the model relies more on the earlier frames where the point is visible, and these frames provide reliable information. If an invisible point is predicted as visible, the behavior degenerates to not using visibility weighting, which may introduce unreliable historical information. In summary, the visibility- aware attention mechanism provides an overall positive effect, as confirmed by the results Table [
Discussion of Scene Cut Detector. Currently, our model relies on an external library as an inde- pendent module to detect scene cuts. We find that this traditional detection method is sufficiently accurate in most cases. As reported in Table I0] on Kinetics subset consisting of roughly 300 complex real-world videos, the auto-trigger global matching mechanism yields a clear performance gain. More importantly, this setup allows to validate our key intuition: compared with relying on the model's 's prediction from the previous frame, the position estimated by global matching is less accurate, but its advantage lies in providing fast, coarse global localization. The components should therefore be combined.
Nevertheless, the current approach remains imperfect. Using the scene cut detector independent, untrained module may lead to potential generalization issues. However, this is primarily a temporary compromise because integrate it into the overall framework for joint training. The Kubric dataset commonly used for this task contains synthetic data with a relatively singular domain. Training a model solely on this dataset to explicitly or implicitly determine whether a frame contains a scene cut is highly challenging. We believe that, with access to larger and more diverse training datasets in the future, will become feasible to integrate scene cut detection into the overall framework for joint training, leading to improved performance and generalization.
are currently constrained by available training datasets, making difficult to
A.7 THE USE OF LARGE LANGUAGE MODELS
In this paper, we only use Large Language Models (LLMs) for translation and text polishing. No aspects related to model design, experimental design, or result analysis involve the use of LLMs. We believe that the use of LLMs in this paper does not affect its scientific contributions.
22

---

Published as a conference paper at ICLR 2026
A.8 MORE VISUALIZATIONS
To further demonstrate the superiority of TAPTRv3 and the effectiveness of our designs, we provide more visualizations of TAPTRv3's predictions some challenging real-world long videos. In these visualizations, we label the frame ID at the upper left corner of each frame to indicate the timestamp (except Fig. (). To make the tracking results more visually distinct, adjust the color of the tracking points based on the overall color of the video.
More Visualizations for Motion Trajectories. We provide more visualizations videos from DAVIS dataset, which well-received benchmark. As illustrated in Fig. 8 we visualize the complete trajectories produced by TAPTRv3 demonstrate the consistency of the estimated motion.
More Visual Comparison with TAPTRv2. We provide more visualizations of the comparisons between TAPTRv2 and TAPTRv3, demonstrating the superiority of TAPTRv3. For more details, please refer to the image captions and the corresponding videos. The correspondences between images and videos are:
Fig.
9
+ CompareVideoll_APPRRR22.. & CompareWieel__TAPPRR..mmm
Fig.
T0
CompareVidee2__TTE
& Dinppaeeiidee2_TAPPRR.....
Fig.
CompareVideo3 T & CompareVidest SimpareVVidoo3_/AAPRRV3....
Visual Comparison w. and wo. Global Matching, We provide more visualizations of the com- parisons between TAPTRv3 with and without the auto-triggered global matching, demonstrating its effectiveness, For more details, please refer to the image captions and the corresponding videos. The correspondences between images and videos are:
Fig.
12
& Compare SinnaaeeViied__APPPP33vvOO....
Fig.
13
BaterreeoSSSAAPPPP33G33 & C SAt.STAAAATA.3..mppppp..
Video5
More Robust Prediction Visualizations. We additionally provide more visualizations to demon- strate TAPTRv3's robustness in in-the-wild scenarios, showing its potential for various downstream applications. The correspondences between images and videos are:
Fig.
14
RobustVideol.mp
Fig.
RobustVideo2.mp4.
Figure 8: More Visualizations for Motion Trajectories.
23

---

Published as a conference paper at ICLR 2026
TAPTRV2
TAPTRV3
TAPTRR2
TAPTRv3
Figure 9: Visual comparison between TAPTRv2 and TAPTRv3. Best view in electronic version. From the third image in the first row (36th frame), TAPTRv2 loses tracking of the turtle's flippers, and in the last few frames loses tracking of the turtle shell and the point on the fish below the turtle. TAPTRv3, on the other hand, maintains stable and accurate tracking throughout the video. The corresponding videos (CompareVidens L TAPTRv2.mp4 and Compare Videol TAPTRTA...mmm 3.mp4) are provided in the supplementary material.
24

---

Published as a conference paper at ICLR 2026
59
119
179
258
298
TAPTRV2
119
179
238
298
33
358
537
597
856
TAL CAPIRRI
358
397
TAPTRV3
Figure 10: Visual comparison between TAPTRv2 and TAPTRv3. When the goldfish is about to swim out the frame from right to left (1 (11 frame), TAPTRv2 loses many target tracking points. Afterward, the goldfish swims back from left to right, and starting from the 358th frame, the video shows the other side the goldfish, where the original target tracking points are occluded. However, another fish. TAPTRv3,
TAPTRv2 incorrectly estimates them visible maintains the correct estimation. Until the last dozens of frames, when the goldfish turns around again, TAPTRv3 successfully detects the initial target tracking points, estimates them visible, and provides accurate positions. The corresponding videos (Compare Maadeo_TAAPRy2220 and Compare VimpareVideo2 3APTRv3.mp4) are provided in the supplementary material.
the other hand,
25

---

Published as a conference paper at ICLR 2026
TAPTRv2
TAPTRv3
TAPTRv2
TAPTRv3
TAPTRv2
TAPTRv3
TAPTRv3 TAPTRv2
Figure 11: Visual comparison between TAPTRy2 and TAPTRv3. Best view in electronic ver- sion. Over time, TAPTRv2 incorrectly estimates the location and visibility of points on jellyfish, and the error accumulates, while TAPTRy3's results are more accurate. The corresponding videos (Compare Video3 LeweVoeeeeeeeeeeeeee mmp and Compare Video3. (APTRv3.mp4) are provided in the supple- mentary material.
26

---

Published as a conference paper at ICLR 2026
TAPTRO3
TAPTRv Global Matching
TAPTRV3
TAPTRo3 with Global Vatching
Figure 12: Visual comparison between TAPTRv3 with and without the auto-triggered global matching. After the occluder appears and then disappears, TAPTRy3 without auto-triggered global matching takes about 70 frames successfully re-track the target tracking points. How- ever, with the help of global matching, this process takes only two frames. The corresponding videos (Compare Video4 TAPTRv3wGM.mpp and Compare Video4 TAPTRy3woGM.mp mmp4) are provided in the supplementary material.
TAPTRN3
TAPTROS will Global Vatchhing
TAPTRA3
TAPTRES Global Marching
Figure 13: Visual comparison between TAPTRv3 with and without the auto-triggered global matching. After the occluder appears and then disappears, TAPTRv3 without auto-triggered global matching takes about 14 frames to successfully re-track the target tracking points. How- ever, with the help of global matching, this process takes only two frames. The corresponding videos (CompareVidess Video5_ _ TAPTRy3wGM..mm and Compare TAPTRv3woGM are provided in the supplementary material.
27

---

Published as a conference paper at ICLR 2026
(b)
Figure 14: Additional visualizations of TAPTRy3's robust predictions. The corresponding videos (RobustVideol .mp4 and RobustVideo2.mpp4) are provided the supplementary material.
28


Reviews:
[{"review_id": "aWJAJoH7pv", "reviewer": "Reviewer_E5oP", "summary": "This work presents TAPTRv3, which aims to leverage both spatial and temporal context to for robust tracking in long videos. It introduces two key operations, i.e. Context-aware Cross- Attention and Visibility-aware Long-Temporal Attention, which can improve the quality of spatial cross attention and long-term feature updating. An auto-triggered global matching mechanism is further triggered when a scene cut is detected. The tracker has been evaluated on public benchmarks.", "questions": "1. What are the major insights of this work compared to TAPTRv2?\n2. What’s the key difference between the proposed auto-triggered global matching and other redetection approaches?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of Context-aware Cross-Attention (CCA) and Visibility-aware Long-Temporal Attention (VLTA) seems reasonable.\n2. The auto-triggered global matching mechanism is easy to follow.\n3. This paper provides extensive quantitative results on the public tracking benchmarks, demonstrating the method's robustness, and efficiency.", "weaknesses": "1. The overall framework of TAPTRv3 is highly similar to TAPTRv2. In TAPTRv2, it also introduces Attention-based Position Update and visibility classifier to maintain the temporal consistency. Despite the implementation variations, the key idea of this work is not very novel.\n2. The performance gains compared to CoTracker3 and Track-On in Table 1 are marginal.\n3. The auto-triggered global matching is simply a global redetection mechanism, which has been widely explored in other tracking frameworks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YrnKXILqxF", "reviewer": "Reviewer_c9Zu", "summary": "This paper proposes TAPTRv2, a point tracking method. The core of this paper lies in presenting two corresponding operations: Context-aware Cross-Attention (CCA) and Visibility-aware Long-Temporal Attention (VLTA), which are designed to improve the quality of spatial cross-attention and the effectiveness of long-term feature updates. Meanwhile, to address scene switches, this paper proposes an auto-triggered global matching mechanism, which enhances tracking stability when scene switches occur.", "questions": "- There are now many improved approaches for RNN-like neural networks, such as Mamba and RWKV. Why is it necessary to return to using attention? Is this truly a better choice compared to other structures?\n- In visibility-aware attention, the authors modify attention weights using predicted visibility. While this operation is shown to be beneficial in results, are there cases where errors in predicting visibility lead to performance degradation instead? Why can it be guaranteed to have a positive effect?\n- There are some ambiguities in the description of the method, such as the implementation of APU. What is the rationale behind such an operation? Is it merely due to the addition of parameters?\n- Are point tracking methods sensitive to the backbone network?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper focuses on improvements such as enhancing context-aware capabilities and modifying attention weights using visibility, which are highly intuitive and also suitable for video tasks.\n- This paper achieves good performance on multiple datasets.\n- The ablation experiments for the newly proposed modules in this paper are conducted in detail.", "weaknesses": "- The key improvement of this paper lies in proposing two types of attention mechanisms to replace previous RNN-like methods. However, there are now many improved RNN-like neural networks, such as readily applicable Mamba and RWKV. The paper lacks sufficient explanation for these methods; could improved recurrent structures also alleviate the challenges in long-context modeling?\n- In visibility-aware attention, the authors modify attention weights using predicted visibility. While this operation is shown to be beneficial in results, another scenario is that errors in visibility prediction may in turn lead to performance degradation. The authors still need to explain why this operation can be guaranteed to have a positive effect.\n- The implementation of CCA shares the same core idea as deformable attention.  And the paper in my opinion is more about combining and improving existing technologies.\n- There are some ambiguities in the description of the method, such as the implementation of APU and the rationale behind such operations.\n- The main reason for the acceleration in this paper is the use of a smaller backbone. However, could it be that the backbone inherently has little impact on this task? The authors still need to rule out this impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "956u60aBx5", "reviewer": "Reviewer_PNie", "summary": "The authors build upon point tracking framework of TAPTRv2, where they propose an improved version called TAPTRv3 that achieves better performance on long-term scenarios. In the proposed tracking method, they introduce two main contributions where (1) context-aware cross-attention (CCA), and (2) visibility-aware long-temporal attention (VLTA) are the key components of this paper. To validate this, experimental evaluations including large-scale benchmarks of TAP-Vid and RoboTAP where the proposed algorithm shows improvements over the previous methods.", "questions": "Please refer to the weaknesses section. Although the proposed scheme seems effective, limitations in the evaluations and performance degradations should be addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The formulation for the proposed method is simple and reasonable, and contributions claimed by the authors are straightforward and clear.\n\n- The motivation of the proposed method is clear, where failure cases of TAPTRv2 including scenarios with long-term temporal drift and scene cuts are well-addressed and quantified.\n\n- The authors performed ablation experiments that validate the effectiveness of each proposed component. The results in Table 2 show that each proposed module (VLTA, CCA, etc.) adds to the gains in empirical performance.", "weaknesses": "- The main experimental comparisons are performed on the subsets of TAP-Vid benchmark, and albeit them being oriented for long-term tasks compared to previous benchmarks such as DAVIS, they seem still short in terms of temporal length. Datasets such as PointOddyssey [a] contain longer sequences and is more oriented for evaluating the proposed method. \n\n  - [a] Yang et al., PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking., ICCV 2023.\n\n- Although the proposed framework somehow works by detecting scene cuts through the external library, it is not a trainable module with questionable generalizability issues. Since the main objective of the proposed framework is to deal with long-term videos, the overall framework should systematically include such algorithm in a integrated and optimized manner. \n\n- Although the main objective of the proposed method is to tackle long-term scenarios well, Table 13 in the supplementary material shows slightly lower performance on short-term scenarios compared to other methods. Does this imply that the proposed CCA and VLTA modules hinder precise localization in shorter videos? Are there experimental analysis for the reason for the performance degradations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B3B1DXPtpv", "reviewer": "Reviewer_YtCz", "summary": "This paper presents an enhanced version of TAPTRv2, a DETR-like framework for point tracking. The authors aim to mitigate the feature drifting issue that occurs in long video sequences by introducing several key modifications: (i) a patch-level similarity computation for refining attention weights, (ii) feature anchoring based on the initial frames, and (iii) aggregation of historical feature changes. Experimental results demonstrate consistent and substantial performance gains over the baseline.", "questions": "Please see weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper provides an updated and improved version of a well-recognized community benchmark in point tracking. The proposed enhancements, such as patch-level similarity computation in attention and initial feature anchoring, are methodically designed and empirically validated, leading to stable and meaningful improvements across benchmarks.", "weaknesses": "While the proposed techniques yield consistent quantitative improvements, the overall methodological novelty appears limited. Patch-level feature similarity computation has been explored in prior works such as Context-PIPs and Track-On. Similarly, leveraging initial-frame features and historical feature fusion are well-established practices in tracking literature, making it difficult to identify a distinct conceptual innovation in the paper.\n\nMinor Issues:\n- Figure 3 is not particularly illustrative. I recommend replacing it with a better qualitative example to better demonstrate the improvements.\n- A visualization of point trajectories for multiple tracked points using a well-received benchmark video would be highly valuable for evaluating qualitative performance and motion consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)