TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation
Authors: Seong Hyeon Park
Huiwon Jang
Byungwoo Jeon
Sukmin Yun
Paul Hongsuck Seo
Jinwoo Shin
Venue: NeurIPS 2024 spotlight (2024)
Keywords: motion estimation
point tracking
video segmentation
Primary area: machine_vision

Abstract: Tracking points in video frames is essential for understanding video content. However, the task is fundamentally hindered by the computation demands for brute-force correspondence matching across the frames. As the current models down-sample the frame resolutions to mitigate this challenge, they fall short in accurately representing point trajectories due to information truncation. Instead, we address the challenge by pruning the search space for point tracking and let the model process only the important regions of the frames without down-sampling. Our first key idea is to identify the object instance and its trajectory over the frames, then prune the regions of the frame that do not contain the instance. Concretely, to estimate the instance’s trajectory, we track a group of points on the instance and aggregate their motion trajectories. Furthermore, to deal with the occlusions in complex scenes, we propose to compensate for the occluded points while tracking. To this end, we introduce a unified framework that jointly performs point tracking and segmentation, providing synergistic effects between the two tasks. For example, the segmentation results enable a tracking model to avoid the occluded points referring to the instance mask, and conversely, the improved tracking results can help to produce more accurate segmentation masks. Our framework can be easily incorporated with various tracking models, and we demonstrate its efficacy for enhanced point tracking throughout extensive experiments. For example, on the recent TAP-Vid benchmark, our framework consistently improves all baselines, e.g., up to 13.5% improvement on the average Jaccard metric.

TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation
Seong Hyeon Park'
Huiwon Jang'
Byungwoo Jeon' Jinwoo Shin
Sukmin Yun?
Paul Hongsuck Seo
KAIST
"Hanyang University ERICA
KKorea University
(seonghyp, huiwoen0516, imbw2024, jinwoos] phseočkorea, ac. krr
sulminnyu@hanyyagg ac.
Abstract
Tracking points in video frames is essential for understanding video content. How- ever, the task is fundamentally hindered by the computation demands for brute-force correspondence matching across the frames. As the current models down-sample the frame resolutions to mitigate this challenge, they fall short in accurately repre- senting point trajectories due to information truncation. Instead, we address the challenge by pruning the search space for point tracking and let the model process only the important regions the frames without down-sampling. Our first key idea is identify the object instance and its trajectory over the frames, then prune the regions of the frame that do not contain the instance. Concretely, to estimate the instance's trajectory, we track group of points on the instance and aggregate their motion trajectories. Furthermore, to deal with the occlusions in complex scenes, we propose to compensate for the occluded points while tracking. To this end, we introduce unified framework that jointly performs point tracking and segmentation, providing synergistic effects between the two tasks. For example, the segmentation results enable a tracking model to avoid the occluded points referring the instance mask, and conversely, the improved tracking results can help to produce more accurate segmentation masks. Our framework can be easily incorporated with various tracking models, and we demonstrate its efficacy for enhanced point tracking throughout extensive experiments. For example, recent TAP- Vid benchmark, our framework consistently improves all baselines, egg., up to 13.5% improvement on the average Jaccard metric. The project url is https://tracee wwtnene.ittubiiiie github.io/.
the
1 Introduction
Obtaining accurate point trajectories over the video frames is crucial for understanding complex dynamics in video data, a necessity for advanced novel-view rendering [3], video frame prediction/ineerpoattio [4], and video depth estimation [5]. Recently, video point tracking task [6, 7, 10] has witnessed rapid progress, which aims predict the trajectory and visibility of a given query point, proving long-term trajectories robust to partial occlusions of objects in real video scenes.
tasks like action recognition [2].
Despite their success, we find current point tracking models are fundamentally challenged by an excessive computation demand since the task requires brute-force comparisons over every spatial location in every frame in given video. As result, to meet the computation constraints, the models
'The confidence whether the trajectory is visible in each frame; iee. the point is not out-of-frame and not occluded by different objects.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(Step 1) Instance Trajectory Estimation (Trajectory Aggregation)
Query point
Seg. Model
Point
Tracker
Sampling Semantic Neighbors
Individual Trajectories
(Step 2) TrackIME
I= 0
Point Tracker
Final
Tracking
Result
Pruned Input Frames
Our framework enhances point tracking by pruning the search space, along the instance trajectory in video frames. To estimate the instance trajectory, our framework utilizes the point tracking results for group of points (blue lines) on top of the object instance predicted by segmentation model (e.g., SAM [1]) and aggregate their individual trajectories.
Figure The workflow of TrackIME.
down-sample their tracking resolutions, sacrificing detailed visual features, which eventually leads to sub-optimal tracking accuracy and triggers tracking failures on intricate object parts. In this regard, we pursue the direction of pruning the excessive search space for point tracking, so that models can avoid the down-sampling and focus only on important regions maintaining detailed visual features, e.g., the object instance masks the query point lies
In this paper, we introduce TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation that focuses the region occupied by the object instance that the queried point lies in and guides point tracking models to prune the video frames along the instance's 's motion trajectory. Here, to obtain the instance trajectory, we first produce the instance mask for a given query point by utilizing the recent segmentation foundation models, e.g., segment anything (SAM) [1], where these foundation models show strong generalization performance different objects/scenes and we find resulting instance masks in quality are readily available. Then, given the instance mask, we sample set of points and aggregate their tracking results as the estimate of the instance trajectory.
Furthermore, to deal with the occlusions in complex video scenes, we propose a unified framework that jointly performs the point tracking and video segmentation, where it re-samples the occluded points by referring to the instance mask. We note that our framework provides synergistic effects for both tasks, i.e., the point tracking results assisted by the segmentation can conversely bolster the quality of segmentation. Consequently, although our primary focus is on the advances in point tracking, our method can also demonstrate improved segmentation results than the baselines (see Section 4.2 for details).
Through the experiments the TAP-Vid point tracking benchmark [11], demonstrate the effectiveness of TrackIME by incorporating it with different point tracking models such as TAPIR [6]. For example, in the DAVIS scenes [12] evaluating the point tracking for dynamic objects, method achieved up to 13.5% relative improvement (57.5 + 65.3 with TAPIR) in terms of the average Jaccard (AJ) metric. Moreover, our framework allows pruning non-instance regions for point tracking
Intuitively. the instance as a group of points moves together even if fine mne-grained motion individual points may differ. Hence, we track multiple points on the same instance, and then aggregate their trajectories as the instance motion, which eventually utilize for pruning video frames.

---

models, the efficacy of our method stands out even more when evaluated on more harsh standards, egg., the l-pixel error threshold, where the conventional metrics allow up to 16-pixel errors when judging the prediction to be correct.
2 Method
In this section, we describe the detailed procedure of our TrackIME framework and its application to video point tracking. Specifically, in Section 1. we describe the formulation for instance trajectory estimation, which based on the video point tracking of the query points found by the foundation segmentation model [1].
Next, in Section 2.2, we present the detailed formulation of TrackIME given the instance trajec- tory, which prunes unimportant regions in the video frame and achieves boosted point tracking performances.
As for the data notations, we denote vectors with elements as bold letters x: := [<1] x2: NNl, tensors with N arrays bold capital letters X == [x X2; Xv], where the subscripts represent the indexed scalars or arrays. Otherwise, every non-bold symbol is scalar. We also introduce the superscripts, e.g., x(9), when denoting there is special semantics for a data, such as the query point.
Finally, when making binary classifications based on probability (or normalized confidence) values, we use threshold 0.5; nevertheless, the values are hyperparameters and can be altered in practice.
2.1 Instance Trajectory Estimation
In this section, we provide the definition of the instance trajectory and procedures to obtain it, such as sampling group of points the instance, trajectory aggregation, and the point re- re-sampling modules.
Video point tracking. Let E R be the tensor of video frames, where L denotes the time duration and (H x W) denotes the image size, and let p(9) E IR? be the spatial coordinates of the query point. Typically, we consider the query in the initial frame hence index of the query point for clarity. Given the video I and the query point p(9), we consider a point tracking model Tracker that predicts the query trajectory (4) E IR Lx2 and the probability of being visible (9) E (0, 1)2 over the entire set of frames,
do not denote the time
(T (9) == Tracker(p(9)
Here, one might utilize Equation (1) as the simplest representation of the instance motion trajectory. However, modeling the instance motion solely using the query point has critical shortcomings. For example, when the instance is partially occluded by other objects, the trajectory of the query point may no longer exist (see Section 4.1 for the ablation study).
To address this challenge, we propose to sample additional tracking points automatically. Specifically, our idea is to identify the instance mask of the query point so that extra query points can be added from the mask.
Sampling points on the instance. Let Mo E 10 denote the segmentation mask that represents the object instance associated with the query point p( Given this mask, we sample a group of points on the instance,
(p
(ns)
which we refer to it as the semantic neighbors of p(a), We note that S is the number of sampled points, where the query point is also counted as its semantic neighbor, i.e., ((no) = p(().
For each semantic neighbor point, we employ Tracker in Equation (1) to produce its trajectory and visibility, (T()),, ( ff:))) : Tracker (p ,I), 3 and pass it to the trajectory aggregation module. Since the query point also participate in tracking procedure, the total effective number of points would be S + 1. For example, we choose S + 1 = 32 in our main experiments discussed in Section 2.
In practice, we patce-process set of multiple points simultaneously.
3

---

Trajectory aggregation. We produce an instance motion trajectory by aggregating the tracking results of the semantic neighbors. Specifically, we consider the velocity, AT," T T T and calculate the weighted average:
(n)) AT(()))
AT
Σ (o (0.0))
Σ (n
( 20.5) (0)))
ot (m:)
In Equation (3), note that velocities aggregated only if the points are classified visible (o; (n.) 2 0.5), and the visibility acts as the aggregation weight. Finally, we accumulate the aggregated velocity starting from T(s) -)))), to obtain the instance motion trajectory.
(q) T( = --1 + AT
(4)
Instance mask. In order identify the instance mask, employ the recent foundation segmentation model, e.g., Segment Anything Model (SAM) [1], and prompt the model with the query point pl9) to produce the pixel-wise confidence representing the object instance indicated by the query point. We denote this function as Seg,
Mo Seg(p (9) In) E (0, HxW
(5)
Given the mask Mo. employ weighted sampling for the semantic neighbors. Specifically, we encode the sampling weights with the distance transform (DT) [13, 14] to the mask's region with positive classifications,
W == DT (1o 2 0.5)).
In this way, the points near the mask's contour are preferred, which we find efficiently represent the object instance because the contour is approximately linearly proportional to the mask's 's radius.
Point re-sampling for robustness to occlusion. Occlusions are common in real video frames, due to dynamic objects and the camera's 's motion. the extreme case, Equation (3) can become degenerate when all semantic neighbors are invisible in future frames / >> 0. Therefore, maintaining sufficient number visible tracking points is crucial, and we tackle this issue by re-sampling occluded points from the instance mask jointly predicted while point tracking.
In a nutshell, whenever we find a certain semantic neighbor point (n:) becomes invisible at time t' and does not show up again (o())) 0.5 for t 2 t'), query the segmentation model with the tracking results of other semantic neighbors to obtain new mask to re-sample the occluded point:
M (n)) = Seg(T ( (n)) .le) e (0, HxW.
However, they could also have been affected by occlusions (e.g., when of close to the threshold 0.5), or by the severe errors in the trajectory Tr(ss) due to sub-optimal tracking performance of Equation (1). Hence, predicting segmentation with these points in a naive way can lead to erroneous masks being produced.
To address this problem, our key idea is to aggregate the group of segmentation masks. Specifically, we collect individual masks by Equation (7), then apply weighted average of the positive classificationss
of (m.) M (ma) > 0.5]
M =
Σ
E (0
(o 20.5)
Σ (n)) 20.5)
2200))200))
0
We find the mask produced by Equation (8) reflects the confidence of each segmentation mask, as well as the visibility of the associated point, and refer to it as the mixture of segmentation distributions, where the value in each index represents the segmentation probability of the queried object.

---

Based on the constructed mixture of segmentation distributions, we obtain the sampling weight in similar manner Equation (6) as, w. DT M. 2 rl), where the threshold r E [0. 1) is set much smaller than the standard 0.5.* This is because we should allow the confident partial segmentation distributions, but ignore the unconfident noise segmentation distributions.
Finally, we re-sample the additional points with Wrr as the sampling weight. They replace the occluded points for the instance trajectory estimation in subsequent frames > f'. We execute this procedure during the tracking, which ensures that a sufficient number of visible points participate in Equations (3) and (4). For example, we set it to be the same as the number of initial semantic neighbors S.
2.2 TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation
In this section, describe our enhanced point tracking, which prunes the search spaces in frames and produces more accurate tracking results.
Search space pruning. Given the instance trajectory in Equation (4), we now aim to utilize it for pruning the search space. Specifically, prune unimportant non-instance regions, by sampling each frame around the (Ho x Wo) regions centered at the aggregated trajectory,
(+) Prune(l, T( Ho, Wo) R Mo Wox3
We note that the sizes (Ho x Wo) are set to be close to the down-sampling resolution considered by a tracking model (e.g., (256 x 256) for TAPIR [6]) so that the information loss minimized.
Given the frames with pruned search spaces, we execute Tracker again to produce the enhanced tracking outputs. Also, for convenience, we abstract the entire process of the instance trajectory
estimation (Section 2. I), the pruning (Equation (9)) and the tracking into a function TrackerHD,
(TD)
(DD) (packerHD(p()) I, Ho, Wo)
(10)
= Tracker(p(9) 1())).
We note that the feature resolutions inside the tracking model not modified, therefore the compu- tational complexity does not increase.
Progressive inference. To achieve a further boost in the tracking performance, we can additionally use progressive inference structure. Formally, we consider a collection of K different TrackerHD models equipped with different pruning sizes (Hx, Wk):
(DD)
T((ED) and (HD)
(HD)
(11)
K
where THE E Lx2 and o (HD) E (0, 1 denotes the outputs of the k-th TrackerHD model.
This progressive structure can boost the tracking performance in two ways. The first is utilizing a past k-th TrackerHD as the tracking model that estimates the instance trajectory for the next (k: + 1)-th TrackerHD. In this way, the pruning is guided by that these tracking results can be aggregated to produce the final trajectory. Specifically, we aggregate based the visibility, in similar manner to Equations (3) and (8):
accurate trajectory estimate. The second
(HD) O
T HD)
T (Pinsl)
(12)
K
O (HD)
where indicates the element-wise product. This aggregation allows processing multiple scales in visual features, which can enhance the generalization performance of vision models [15, 16). We note that the visibility predictions averaged the K predictions.
FFor example, choose 0.1 for main experiments discussed in Section 4.
5

---

Table 1: The evaluation of point tracking performance for dynamic objects. We benchmark the quality point tracking in DAVIS [12] videos with the point annotations provided by TAP-Net [11]. We note that TrackIME is incorporated with TAPIR point tracker [6].
First Query
Strided Query
Method
J1
AJ
of
avg
OA
Ji
AJ
oj
6
OA
TAPNet [11]
20.7
51.6
30.1
63.8
79.8
25.3
56.5
36.3
68.2
82.6
PIPS2 [9]
19.6
46.6
35.8
69.4
80.3
6.9
52.8
14.2
65.8
83.5
TAPIR [6]
23.0
57.5
34.3
70.5
84.4
28.1
62.8
41.0
75.1
87.7
CoTracker [7]
28.3
60.8
43.5
76.1
86.0
34.9
64.3
50.9
78.9
89.1
OmniMotion [8]
21.5
52.6
39.1
68.1
85.4
30.1
55.6
45.1
70.3
88,9
TrackIME
35.4
65.3
48.2
78.6
86.5
41.9
69.3
55.0
81.4
89.0
Related Work
Optical Flow. Optical flow deals with the dense computation of instantaneous motion patterns between two given video frames. Starting with the pioneering work of applying neural networks for motion estimation [17, 18], the seminal works such as DCFlow [19]. PWC-Net [20] and RAFT [21] introduced the concept of dense correspondence matching between pairs of image patches. Despite their success, the optical flow's inherent limitations incapable of modeling trajectories and occlusions triggered the recent progress in the point tracking methods.
Point Tracking. In essence, point tracking attempts to find the long-term point correspondences over the entire video frames, and model the occlusions and trajectories. The current models in this domain, such as PIPs [22], TAPNet [11], TAPIR [6], CoTracker [7]. and OmniMotion [8] has led rapid progress, with advanced neural architectures [6, 7] or test-time optimizations [8]. However, they are fundamentally hindered by the excessive search space for correspondence matching over the entire frames. Our focus is to address this issue by pruning the search space, where method can
be readily incorporated with these baselines.
Instance Segmentation. Recently, the important advancement within image segmentation has been the introduction of segment anything (SAM) [1]. SAM is specifically designed perform image segmentation by general point prompts and exhibits an impressive capacity for class-agnostic segmentation. Specifically, in the context of point tracking, SAM serves as valuable resource by generating segmentation masks for the object instance indicated by the query point. We also note the line of zero-shot video segmentation [23, 24, 25, 26, 27, 28, 29, 30]. Specifically, the recent SAM-PT [30] focuses on bolstering video segmentation based on point tracking, which is fundamentally different from our work; our primary goal is obtaining better point tracking, while that for SAM-PT is for better segmentation. Nevertheless, our method provides synergistic effects for both tasks, and even outperforms SAM-PT for segmentation tasks (see Table 4).
Experiments
In this section, we demonstrate the effectiveness of the proposed TrackIME on point tracking tasks and the downstream video object segmentation.
In Section 4.1, we focus on the point tracking tasks. Specifically, we first experiment the efficacy of the instance motion trajectory estimation and our search space pruning technique for point tracking by measuring the performance in video scenes that capture dynamic objects.
Next, verify the universality of our method to different point tracking models and find whether it can provide general performance improvements when incorporated into the five recent baselines, e.g., TAPNet [1 IJ, PIPS2 [9], [oTracker[7]. OmniMotionf88 8] and TAPIR [6].
In the ablation study, we validate the effect of each component, namely the trajectory aggregation, the search space pruning, and the progressive inference modules described in Section 2.

---

Table 2: Universality of TrackIME with different point tracking models. We incorporate recent point tracking model baselines [6, 7, 8, 9, 11] with our method, and benchmark performance on DAVIS [12], RGBStacking [33], and Kinetics [34]. f: the underlined results are obtained with subsets of RGBStacking and Kinetics datasets due to a large optimization cost for the OmniMotion [8].
DAVIS
RGBStacking
Kinetics AJ
Method
AJ
avg
AJ
Dvg
avv
TAPNet [11]
51.6
63.8
56.5
79.0
49.3
60.7
TrackIME
57.9
72.4
66.9
80.0
51.0
63.6
PIPS2 [9]
46.6
69.4
52.3
74.9
TrackIME
50.3
74.0
52.8
75.8
CoTracker [7]
60.8
76.1
64.1
78.0
47.7
63.7
TrackIME
64.5
79.2
68.2
82.1
48.1
63.8
OmniMotion [8] TrackIME
52.6
68.1
71.2
81.1
51.0
64.3
54.1
69.3
71.9
81.9
51.2
64.6
TAPIR [6]
57.5
70.5
66.3
80.6
50.2
62.3
+ TrackIME
65.3
78.6
66.6
81.8
51,4
65.8
In Section 4.2, we verify the efficacy of the enhanced point tracking results by TrackIME in the downstream video object segmentation. Specifically, we compare the zero-shot video segmentation performances with the recent SAM-PT [30] baseline which utilizes the point trajectories as the inputs, as well the conventional baselines that input the semantic classes [27, 28, 29].
Common implementation details. We note that TrackIME mainly incorporated with TAPIR point tracker [6] (as it empirically performs best) unless specified otherwise, and we subject it to all experiments including the point tracking and other downstream tasks.
For the segmentation model, we utilize the Segment Anything (SAM) [1] to perform the point-queried segmentation function described in Equation (5).
To prepare video frames, we always adjust the resolutions of raw video data to 1080p (1080 pixels in the shorter frame edges), then apply further resizing functions required by individual baseline models. For example, we resize the 1080p frames to 256 x 256 for TAPIR [6] baseline, following the default setting provided by the official open-souree repository. When experimenting TrackIME, choose the hyperparameters for each baseline, e.g., progressive inference steps = 2, and the pruning sizes Ho Wo 960 and H1 Wi 384 when incorporated with TAPIR [6].
Since IME is plug-in to all baselines, we reproduce all results the same system configuration for fair comparisons. We note that such modification induce minor perturbation in the numerical values due to library and hardware-dependent characteristics, e.g. different characteristics between JAX [31] and Py PyTocch [32] libraries, and the difference in the filtering algorithm used when re-sizing the video frames.5 5 We refer the readers to Appendix A for more implementation details.
4.1 Point Tracking
Baselines. We compare our method to the recent baselines OmniMotion($), CoTracker [7], TAPIR [6]. PIPS2 [9], and TAPNet [11]. We utilize the official checkpoints provided by the official project pages and reproduce all experimental results under our common experimental set-up, except for OmniMotion [8] which does not provide checkpoints. Instead, we reproduced the training of OmniMotion models to obtain the experimental results. We use S = 31 semantic neighbors to incorporate our framework with the baselines.
Datasets. We evaluate these models on three different datasets, DAVIS [12], Kinetics [34], and RGB- Stacking [33], each representing different characteristics. For example. DAVIS contains 30 videos
'The open-source version of TrackIME is available https://githin www.
7

---

Table 3: Ablation study of the components in our model. We ablate the effect of search space pruning (Pruning), trajectory aggregation (Aggregation), and the progressive inference (Progressive) modules for point tracking. We evaluate the tracking benchmark in DAVIS scenes [11, 12].
Pruning
Aggregation
Progressive
J1
AJ
5f
5
avg
X
23.0
57.5
34.3
70.5
X
X
28.2
62.5
41.1
75.3
X
/
28.3
62.6
41.2
75.6
/
V
X
34.0
62.9
48.0
77.0
/ /
35.4
65.3
48.2
78.6
specifically curated to evaluate the tracking performance under large variances in the appearance and motion of object entities. Its two variants, DAVIS- (First) and DAVIS-S (Strided) differ in how the query points are given to the models: DAVIS-F queries the model only once in the first frame, while DAVIS-S queries the model in strides of five frames. Because DAVIS-F requires long-term tracking, it is generally more difficult setting. Kinetics contains 1,144 web videos collected from You'l YouTube that represent realistic noisy characteristics of the video in the wild, such as sudden scene changes. RGB Stacking is synthetically rendered dataset representing 50 different moves by robotic arm. For all datasets, we refer to the point tracking annotations provided by TAP-Vid [6] and utilize them as the ground truth for evaluation.
Metric. To measure the quality of point tracking, we consider point tracking accuracy considered following TAP- Vid [11], such as the ô-average accuracy (6mvg) and the average Jaccard (AJ). The average metrics are based on the ô-n accuracy (8%) which indicates the proportion of correct trajectory sequence as judged by whether they are within the n-pixel error threshold around the ground truth. In addition, the Jaccard-n (J,) judges trajectory sequence to be correct only the visibility prediction is also correct. Given these definitions, the average metrics are calculated by averaging {1, 2, 4, 8, 16]. To evaluate the fine-grained tracking performance in a harsh error threshold, we also report 8-1 accuracy (8f) and Jaccard-1 (J1). For Table 1, we also discuss the occlusion accuracy (OA), the proportion of correct visibility sequence given the ground truth.
Effectiveness on point tracking in dynamic objects. We first present the point tracking scenarios with dynamic objects. Specifically, we experiment with the DAVIS video scenes [12], which is curated for evaluating instance motion estimation tasks. As shown in Table 1, we find our method achieves the best point tracking accuracy surpassing all baselines, e.g., up-to 7.4% relative improvements in average Jaccard, i.e., 60.8 AJ (CoTracker [7]) vs. 65.3 AJ (TrackIME) when evaluated with the DAVIS-F (denoted First Query in Table 1). We also measure the occlusion accuracies (OA) and find a relatively incremental improvement than other metrics. Intuitively, there is a trade-off between modeling the occlusions among different objects and the search space pruning for one instance, as the pruning removes information from other instances. Nevertheless, our method is beneficial for detecting occlusion in fine-grained object parts, and we recommend searching for optimal pruning parameters that fit a user's purpose. Finally, we discuss the efficacy of TrackIME under the harsh of and J1 metrics, where the conventional metrics allows up to 16-pixel errors and takes the average when judging whether the prediction is correct. For example, the improvement can be even larger, e.g., up to relative 25. 1 1%, i.e., 28.3 3 Jı (CoTracker (7]) vs. 35.4 J1 (TrackIME) when evaluated with DAVIS-F. We highlight these benefits of TrackIME allowed by pruning the search space.
Universality to different point tracking models. We validate the universality of our method when plugged into the state-of-the-art baselines by evaluating the average tracking accuracy (AJ and 85)) of the vanilla models and the variants incorporated with our method in Table 2 on DAVIS (First) [12], RGBStacking [33], and Kinetics [34] datasets. As result, we observe that our method can provide consistent and significant performance improvements in all the baselines. e.g., 13.6% relative improvements (i.e., 57.5 - 65.3 AJ) in TAPIR [6] when evaluated on the DAVIS. Since the model variant incorporated with TAPIR demonstrates the best performance, we chose it as our main model and subjected it to other studies. We note that the experiments for OmniMotion [8] have been conducted in 16 subsets for RGBStacking and Kineties, and K progressive inference, due to its
8

---

Query points at t-0
(-0 t-10 [-20 t-30
Figure Demonstration of the video instance segmentation results by our TrackIME framework. Given the query points in the reference frame, our framework can produce the video instance segmentation masks quality by performing the weighted aggregation of the mask associated each query point, based on the visibility values.
heavy optimization costs, e,g., approximately gpu-hours for processing one scene. We also note that PIPS2 [9] in Kinetics [12] is unavailable, as its memory requirement for processing Kinetics exceeds our system's capacity.
Ablation study. We perform an ablation study to understand how each component affects the point trajectory accuracy in Table 3. Specifically, we consider the search space pruning, the trajectory aggregation, and progressive inference modules as the subjects for the ablation.
First of all, we reveal the pure efficacy of our pruning method, separate from the effect of segmentation prior. Notably, when the trajectory aggregation module is removed (the first rows in Table 3), we observe the pruning solely based on the query point's trajectory provides the most significant effect (e.g., 23.0 - 28.2 in J,). This validates our key motivation for pruning the search space, which provides superior results even if SAM [1] is not employed.
Next, we discuss the effect of employing SAM [1] by enabling the trajectory aggregation. As expected, aggregating the trajectories for a group of points found in the segmentation mask provides another comparable gain (e.g., 28.2 34.0 in J1), which validates that the aggregation improves the quality of instance trajectory estimation.
It is worth noting that the progressive inference boosts the performance, (e.g., 34.0 -> 35.4 in J1)
when combined with the trajectory aggregation, otherwise the gain lesser (e.g., 28.2 - 28.3 in Jj). As the progressive inference refers the estimated instance trajectory, the estimation quality is essential for this module.
We also note that further ablation study is available in Appendix D, e.g., the number of semantic neighbors, progressive inference steps, or the pruning sizes.
4.2 Video Object Segmentation
In this section, we validate the efficacy of TrackIME by performing the zero-shot video segmentation. We also provide the visualization results for selected scenes from DAVIS [12] in Figure 2.
Baselines. We experiment with zero-shot video object segmentation to check the efficacy of TrackIME for improving segmentation. Specifically, we consider the class-guided baselines for unsupervised video segmentation tasks, e.g., EntitySeg [29]. In addition, we consider the SAM-PT [30] baseline which also proposes take point tracking for producing segmentation. consider the equivalent experimental set-ups for SAM-PT [30] and TrackIME, we incorporate the models with

---

Table 4: Zero-shot video object segmentation performance in DAVIS benchmark, We consider
two set of zero-shot baselines, those utilizing the set of classes [23, 24, 25, 26, 27, 28, 29] and the baseline utilizing a set of query points [30] in similar manner to our TrackIME. t: we produced the results for TrackIME and SAM-PT [30] under the common set-up, such as the number of tracking points, segmentation function (HQ-SAM [35]), and the same mask formatting for the benchmark.
DAVIS- 201--2017- -vll
DAVIS 2EISS-217--0 2017-test-
Method
Input
(J&F)
Jm
F
(J&F) m
J
Fy
PDB [23]
class
55.1
53.2
57.0
40.4
37.7
43.0
RVOS [24]
class
41.2
36.8
45.7
22.5
17.7
27.3
AGS [25]
class
57.5
55.5
59.5
45.6
42.1
49.0
MAST [26]
class
65.5
63.3
67.6
Propose-Reduce [27]
class
70.4
67.0
73.8
UnOVSOT [28]
class
67.9
66.4
69.3
58.0
54.0
62.0
EntitySeg [29]
class
73.4
70.4
76.4
62.1
SAM-PTT [30]
points
78.8
76.3
81.3
65.3
62.3
68.3
TrackIMEt
points
79.6
76.4
82.8
65.9
62.5
69.4
HQ-SAM [35] variant for the segmentation, 16 points from the initial frame's mask, and employ the iterative refinement technique [35] to produce the video segmentation results.
Evaluation. We evaluate our model on the DAVIS-2017 [12] video segmentation. In particular, we use the validation and the test-dev sets for the zero-shot benchmark. Both sets contain 30 non-overlapping scenes with single or multiple objects.
To measure the quality of video instance segmentation, we consider the standard metrics in baselines: the mean Jaccard (Jm); the mean F-measure (Fn)); and the average (J&F) Specifically, we follow the official implementation suite provided by the DAVIS challenge [12].
Effectiveness zero-shot video object segmentation. In Table 4, we first confirm that the point tracking provides useful guidance for video segmentation, observing that both SAM-PT [30] and TrackIME demonstrates significant improvement over the conventional class- class-prompe prompted baselines. More importantly, our framework brings synergistic improvements for both point tracking and segmentation tasks, we find TrackIME achieves even larger improvement, e.g. 78.8 vs. 79.6 (J&F) in the validation set of DAVIS-2017 [12].
Discussions. As for the commentary on the efficacy of TrackIME, our key advantage is removing erroneous query points for segmentation caused by the tracking failure on intricate object parts, enabling even finer query points for segmentation, e,g., the accuracy in harsh l-pixel thresholds in Table 1, which is possible due to the pruning structure in our framework to maintain the high- frequency information.
5 Conclusion
In this work, we introduce TrackIME, a novel approach for point tracking to overcome the fundamen- tal challenge of computation demands in existing models. Specifically, we reduce the search space by identifying the instance trajectory and pruning the video frames along it. To obtain the instance trajectory, we aggregate the motion for group of points on the segmentation masks. To this end, we propose unified framework that jointly performs point tracking and segmentation, with the tech- niques to ensure robustness to occlusion in complex video scenes. TrackIME demonstrates consistent and significant impacts by bolstering existing point tracking baselines. The joint framework also reveals the synergistic effects, which also demonstrates the improvements the video segmentation task. Overall, our work highlights the effectiveness of considering instance motion trajectory and jointly solving the tracking and segmentation, and we believe our work could inspire researchers to consider new direction to further leverage it in the future.
10

---

Acknowledgements
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (ITP)) grant funded by the Korea government (MSIT) (No.RS-2019-1) 1052019-11110007, Artificial Intelligence Graduate 10S-20211122120666 Artificial Intelligence Innovation Hub; No. RS-20200-12011 ICT Creative Consilience Program), and Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2024(Project Name: International Collaborative Research and Global Talent Development for the Development of Copyright Management and Protection Technologies for Generative AI, Project Number:
School Program(K K (AIST);
References
[1] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander Berg, Wan-Yen Lo, ct al. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
[2] Haoqi Fan, Yanghao Li. Bo Xiong, Wan- Yen Lo. and Christoph Feichtenhofer. Pyslowfast. https: //github.comt womm/facebookreserrcc....
wwworfsssssoattt...oooo 2020.
[3] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 4273-4284, June 2023.
[4] Jiaben Chen and Huaizu Jiang. Sportsslomo: A new benchmark and baselines for human-centric vidco frame interpolation. arXiv preprint arXiv:2308. 16876, 2023.
[5] Zhoutong Zhang, Forrester Cole, Richard Tucker, William T Freeman, and Tali Dekel. Consistent depth of moving objects video. ACM Transactions Graphicx (TOG), 40(4):1-12, 2021.
[6] Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta. Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with framfe initialization and temporal refinement. arXiv preprint arXiv:2306.086377 2023. https:/// thub. com google- wwtp://eddddeddddeeeeeeeee tappett.
[7] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better track together. arXiv preprint arXiv:2307.07635, 2023. URL
attps://github. wtt.///cceboooreseerccececccc
tracker.
[8] Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all once. arXiv preprint arXiv:2306.0544: 05422,, 2023.
[9] Yang Zheng, Adam W Harley, Bokui Shen. Gordon Wetzstein, and Leonidas J Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. arXiv preprint arXiv:2307.15055. 2023. URL https://gi thub. com/ aharley/ aharley/pipe..
[10] Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[11] Carl Doersch, Ankush Gupta, Larisa Markceva, Adrià Recasens, Lucas Smaira, Yusuf Aytar, João Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: benchmark for tracking any point in video. Advances in Neural Information Processing Systems, 35:13610-13626, 2022. URL https: //github. cow/googlee
[12] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbelácz, Alex Sorkine- Sorkine-Horuung, and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675 2017.
[13] Christina Karam, Kenjiro Sugimoto, and Keigo Hirakawa. Fast convolutional distance transform. IEEE Signal Processing Letters, 26(6):853-87, 2019.
[14] Due Duy Pham, Gurbandurdy Dovletov, Josef Pauli. differentiable convolutional distance transform layer for improved image segmentation. Pattern Recognàtion: 42nd DAGM German Conference, DAGM GCPR 2020, Tübbingen, Germany, September 28-October 1, 2020, Proceedings 42, pages 432-444. Springer, 2021.
[15] Tsung-Yo Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125 2017.

---

[16] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scout Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In Computer VCCV 2016: 14th European Conference, Amsterlam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 21-37. Springer, 2016.
(17] Alexey Dosovitskiy, Philipp Fischer, Eddy 1lg. Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758-2766. 2015.
[18] Eddy 1lg. Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462-2470, 2017.
[19] Jia Xu, René Ranftl, and Vladlen Koltun. Accurate optical flow via direct cost volume processing. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 1289-1297, 2017.
[20] Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cans for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8934-8943, 2018.
[21] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. pages 402-419. Springer, 2020.
[22] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vision, pages 59 39-75. Springer. 2022.
[23] Hongmei Song. Wenguan Wang, Sanyuan Zhao, Jianbing Shen, and Kin-Man Lam. Pyramid dilated deeper convistm for video salient object detection. In Proceedings of the European conference on computer vision (ECCV), pages 715-731, 2018.
[24] Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques, and Xavier Giro-i Nieto. Rvos: End-to-end recurrent network for video object segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5277-5277758 52767 2019.
[25] Wenguan Wang, Hongmei Song, Shuyang Zhao, Jianbing Shen, Sanyuan Zhao, Steven CH Hoi, and Haibin Ling. Learning unsupervised video object segmentation through visual attention. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3064-3074, 2019.
[26] Zihang Lai, Erika Lu, and Weidi Xie. Mast: A meeorry-augmente self-supervised tracker. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6479 6488, 2020.
[27] Huaijia Lin, Ruizheng Wu, Shu Liu, Jiangbo Lu, and Jiaya Jia. Vidco instance segmentation with a propose-reduce paradigm. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1739-1748, 2021.
[28] Jonathon Luiten, Idil Esen Zulfikar, and Bastian Leibe. Unovost: Unsupervised offline video object segmentation and tracking. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 2000-2009, 2020.
[29] Lu Qi, Jason Kuen, Weidong Guo, Tiancheng Shen, Jiuxiang Gu, Wenbo Li, Jiay Jia, Zhe Lin, and Ming-Hsuan Yang. Fine-grained entity segmentation. arXiv preprint arXiv:2211.08 1.00072,, 2022.
[30] Frano Rajič, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, and Fisher Yu. Segment anything meets point tracking. arXiv preprint a0Xiv:2307.0f 2023.
[31] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman Miile,, and Qiao Zhang. JAX: compos- able transformations of Python+NumPy programs, 2018 URL http: thw.b/tttttttt//.////////t com/ homn/gooole/aax.
[32] Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019. URL http:://aapp neurips....... htt.iisssscceeeeeeeeeeeeeeeeeeee cctcccc/pppaa cc/paper/ 9015 mri1-pytorch-ooo an- www.inprratieestylllee high- pwwformance deep- learning waatiniiinnibbarry...om.
12

---

[33] Alex X. Lee, Coline Devin, Yuxiang Zhou, Thomas Lampe. Konstantinos Bousmalis, Jost Tobias Springenberg. Arunkumar Byravan, Abbas Abdolmaleki, Nimrod Gileadi, David Khosid, Claudio Fan- tacci, Jose Enrique Chen, Akhil Raju. Rae Jeong, Michael Neunert, Antoine Laurens, Stefano iecti, Federico Casarini, Martin Riedmiller, Raia Hadsell, and Francesco Nori. Beyond pick-and-place Tackling robotic stacking of diverse shapes. In Conference on Robot Learning (CoRL), 2021. URL https: http///ppp/evereeeeeeeeeeeeeeeee net otte/forummddd -d-Uumddt.UtQUt..
[34] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-63009 2017.
[35] Lei Ke, Minggiao Ye, Martin Danelljan, Yifan Liu, Yu- Wing Tai, Chi-Keun KeinKK Tang, and Fisher Yu. Segment anything in high quality. arXiv preprint 57X. 2023.
[36] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Hermmann Kubric: scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3749-3761,
2022.
13

---

Appendix
A Experimental details for point tracking
In this section, we present detailed experimental setups considered by our experiments in Section 4.
Baselines. We consider different baseline point tracking models, TAPNet [11], PIPS2 [9], CoTrracker [7], OmniMotion [8], and TAPIR [6]. We experiment with the checkpoint provided in the official open-source repository hosted by their authors, following the default hyperparameters in each model, e.g., for the input dimensions, in TAPNet [11] and TAPIR [6] consider a square-shaped (256 256) dimension, while PIPS2 and CoTracker do rectangular-shaper dimensions, (512 896) and (384 x 512), respectively. We also note that the backend library of TAPIR and TAPNet is ported from JAX [31] PyTorch [32] in our experiments, which provides subtle enhancements in the tracking accuracy, e.g., AJ 56.2 [6] + 57.5 (Table 2) in DAVIS-F.
Table 5: The pruned resolutions in our method for each baseline point tracking model. We report the specific values for Ho. H1, Wo. W when TAPNe[[1]] lJ, PIPS2[9], CoTracker[7],, OmniMotion [8], or TAPIR[6] is used as the baseline.
Baseline Model
Ho
H1
Wo
W
TAPNet [11]
960
384
960
384
PIPS2 [9]
960
512
1680
896
CoTracker [7]
960
384
1280
512
OmniMotion [8]
960
960
TAPIR [6]
960
384
960
384
Hyperparameters for point tracking. Unless otherwise specified, we always choose the number of semantic neighbors S 31, and the progressive inference steps K = 2 for TAPNet [11], PIPS2 [9], CoTracker [7], and TAPIR [6]. For OmniMotion [8], we set the progressive inference step K 1. To incorporate our framework with the baselines, we select different pruning sizes meet the shape requirements of specific model (e.g., TAPIR [6] needs shape in multiples of 8 to be compatible with its convolution layers). For example, if our method is plugged into TAPIR [6] and video with the 1080p resolution, e.g., (1080 x 1920), we set the pruning resolutions Hn Wn- = 960 and H1 W1 384. For clarity, we present the resolutions for all baseline models in Table 5.
Datasets, We evaluate the baselines and TrackIME in three different datasets from the TAP- Vid benchmark [11]: DAVIS [12]; Kinetics [34]; and RGBStacking [8]. The sizes of the samples can vary, e.g., from 256 to 2160 in their shorter sides, hence we process the frames by resizing the shorter sides 1080 with the aspect ratio fixed. As result, the video frame resolutions typically (1080 x 1920) for DAVIS [12] and Kinetics [34]. We note that RGB-Stacking is originally in (256 256), but we do bilinear up-sampling (1080 x 1080) for simplicity.
Experimental environment. Every baseline model and internal module in TrackIME (e.g., Segment Anything [1]) is implemented in PyTorch 2.1 [32] compiled for CUDA 11.8, which we run on an NVIDIA RTX 3090 GPU. In default, we experiment with the float32 numerical precision; however, in case of out-of memoryy errors (e.g. RTX 3090's s 24 GiB VRAM cannot handle hundreds of frames), we employ the bfloatl6 precision to fit such samples into the limited memory.
B Backgrounds
In this section, we describe technical details behind the limitations in the current point tracking models.
In the common canonical design of recent model architectures for Equation (1), e.g., our baselines: TAPNet [11], CoTracker [7], TAPIR [6], etc., the key component is the cost volume [21], which represents the likelihood of the query point's spatial-temporal location over the entire video frames. In principle, predicting this cost map requires brute-force search over every spatial-temporal location, which is often computationally infeasible on the raw video dimensions, e.g., 1080p. To mitigate this problem, current models first sample the raw video into a lower spatial resolutions,
14

---

Table 6: FLOP counts by each module in TrackIME. We report the FLOP counts for point tracking given 64 video frames, during the instance motion stage, and the high-fidelijy tracking with K = progressive steps.
TrackIME Modules
FLOPs
Instance trajectory estimation (stage I)
1355G
Segmentation
533G
Instance Tracking (32 points; S = 31)
822G
Progressive inference (stage 2)
1434G
k = 0 (32 points; S = 31)
822G
k 1 (1 point)
612G
Total
2789G
e.g., (256 x 256) in TAPIR [6]. While the reduced resolution enables models to process the entire video frames for tracking, the lost information during the resolution reduction induces quantization noises into the cost volume. Recent baselines, including the state-of-the-art [6], employ refinement techniques to mitigate these noises. 6 Nevertheless, the lost detail the visual feature after the down-sampling still hinder representing figh-fregenccy patterns, and the model can suffer from tracking failure modes.
In this regard, our method pursues the direction of pruning the excessive search space for point tracking, so that models can avoid the down-sampling and focus only on important regions maintaining detailed visual features.
C Computational costs for point tracking
In this section, we study the computational costs and efficiency of TrackIME by examining the FLOP floaatig-poing operations) counts for performing the point tracking.
FLOP count of TrackIME. To check the exact cost of each module in TrackIME, we report the FLOPs for tracking under our default setting, egg., TAPIR [6] as the baseline, given 64 video frames. Specifically, as given in Table 6, the segmentation with SAM [1] needs 533 GFLOPs, tracking 32 points (e.g., S = 31 semantic neighbors plus one query point) demand 822 GFLOPs, and tracking single point demands 612 GFLOPs. respectively. As result, the net FLOP count of TrackIME (with K progressive steps) is 2789 GFLOPs.
Computation efficiency compared to baselines. Next, compare the baseline TAPIR [6] with various input dimensions and TrackIME, in terms of their FLOP counts versus the point tracking performances, AJ (Average Jaccard), 8% and OA (Occlusion Accuracy), evaluated under DAVIS-F and DAVIS-S in Table 7.
For TAPIR, the FLOP count is mostly governed by the input dimension of a model (256 x 256), e.g. 612 GFLOPs for processing 64 video frames, and it grows quadratically as the input dimension gets increased.
An interesting finding Table 7 is that the baseline [6] cannot benefit from the larger input dimensions without fine-tuning. For example, we observe that the baseline's 's performance only deteriorates given larger inputs, as the model only optimized for a low-resolution input frames (256 x 256) to meet the memory constraints while training; it is non-trivial to process high-resolution inputs without fine-tuning. Furthermore, even fine-tuning is employed (e.g.. TAPIR Hi-Res [6]). the performance gain (e.g., 62.8 - 65.7 AJ) is not significant considering the excessive increase in FLOP counts (e.g.. 612 - 8257 GFLOPs), and the occlusion accuracy (OA) can even get worse (e.g., 88.3 -> 86.7).
These results further demonstrate the merits of employing TrackIME for point tracking, which can enable point tracking models process the frames a computationally efficient manner, even without fine-tuning, and provide consistent performance gains. For example, comparing TrackIME (ours) vs.
We refer the readers to literature for the refinement mechanisms [6. 7J.
15

---

Table 7: The comparison of the FLOP counts of the TAPIR [6] models and TrackIME. We report the FLOP counts to process 64 video frames by TAPIR with the imput dimensions (256 x 256) (default), (512 512), and (768 768), TAPIR Hi-Res (a fine-tuned model for (1080 1080)
and TrackIME (ours). For each model, we further report the benchmark results in terms of AJ (Average Jaccard), &vg:: and OA (Occlusion Accuracy), evaluated under DAVIS-F and DAVIS-S. For TAPIR Hi-Res, numbers are excerpted from [6]. where results for DAVIS-F are not available.
Method (Input Dim.)
DAVIS-F
DAVIS-S
FLOPs
AJ avg OA
AJ Avg OA
TAPIR
612G
57.5 70.5 85.5
62.8 75.1 88.3
(256 x 256)
TAPIR
2429G
53.9 65.9 79.8
62.5 74.0 81.8
(512 x 512)
TAPIR
5457G
53.3 65.5 73.2
58.3 70.2 76.6
(768 x 768)
TAPIR Hi-Res (1080 x 1080)
8257G
65.7 77.6 86.7
TrackIME (256 256)
2789G
65.3 78.6 86.5
69.3 81.4 89.0
Table 8: The comparison of the FLOP counts of the CoTracker [7] models and Tr.ckIME. We report the FLOP counts process 64 video frames by CoTracker with the input dimensions (384 x 512) (default), (768 x 1024), and (1080 x 1440) and TrackIME (ours). For each model, we further report the benchmark results in terms of AJ (Average Jaccard), &ag. and OA (Occlusion Accuracy), evaluated under DAVIS-F.
Method
DAVIS-F
(Input Dim.)
FLOPs
AJ
8
OA
CoTracker
2707G
60.8
76.1
86.0
(256 256)
CoTracker
7670G
62.3
77.8
87.1
(512 512)
CoTracker (768 768)
5457G
62.2
76.7
86.6
TrackIME (384 x 512) 62.7G
64.5
79.2
88.5
TAPIR Hi-Res [6] gives: 2789G vs. 8257G (FLOPs); 69.3 vs. 65.7 (AJ); 81.4 vs. 77.6 (8rug): and 89.0 vs. 86.7 (OA), in DAVIS-S respectively.
In the similar manner, also provide the FLOPs count for our method incorporated with [7] in Table 8.
D Ablation study
In this section, we ablate the choice of hyperparameters in our enhanced point tracking, namely the pruning sizes (fo, Wo) without the progressive fusion (i.e., K = 1) and our default setting in TrackIME (K 2), and the number of sampling semantic neighbors S for estimating the instance trajectory.
In Table 9, we find that smaller pruning sizes tend to introduce positive effects in the fine-grained metrics (e.g., and 2-pixel error thresholds), but also trade off the average-scale metrics (e.g., and ). These results are expected, as the pruning size gets smaller, the amount of down-sampling reduces and more detailed visual features would be preserved, but at the same time, the chance of erroneous pruning increases where the true location of the query point is lost.
16

---

Table 9: Ablation study of the pruning size in our framework. We ablate the pruning size considered in TrackIME. For the evaluation, we calculate both pixel-scale and average-seale metrics under the DAVIS-F dataset [11].
Pruning Size (K)
J1
J2
62
AJ
avg
1080 (K 1)
28.1
41.0
52.3
66.0
62.5
75.2
960 (K = 1)
29.7
42.4
52.9
66.3
63.1
75.6
768 (K 1)
31.9
44.6
55.7
68.1
64.0
76.4
512 (K =1)
34.6
47.6
56.5
68.9
63.9
76.9
384 (K
35.3
47.3
56.5
68.3
62.3
76.1
960 + 384
35.4
48.2
57.7
70.1
65.3
78.6
(K
Table 10: Ablation study of the effect of the number of semantic neighbors in our method. We ablate the number of semantic neighbors considered in our method. For the evaluation, we calculate both pixel-scale and average-scale metrics under the DAVIS-F dataset [11].
S + 1
J1
8f
J2
82
AJ
Aag
128
35.1
47.7
57.1
69.4
64.8
78.2
64
35.0
47.5
57.2
69.8
64.8
78.3
32
35.4
48.2
57.7
70.1
65.3
78.6
16
35.2
47.9
57.3
69.9
64.8
78.5
8
35.1
47.8
57.4
70.2
64.9
78.5
4
35.0
47.6
57.5
69.8
64.9
78.3
2
35.1
47.9
57.2
69.7
64.7
78.1
We note that the progressive fusion (K = 2) in our method mitigate the trade-off in pruning by considering multiple scales, e.g., H, = 960 and H₃ = 384, providing additional performance gains.
Next, in Table 10, we ablate the effect of the choice for the number of semantic neighbors S + (including the query point), halving down its value starting from (S + 1) = 128 to (S + 1) = As result, we find that all of the choices can provide satisfactory performance in general, although there exist mild trade-offs between the and 2-pixel scale metrics and the average scale metrics. As one of our goal is on achieving the optimal pixel-scale performance in point tracking, empirically choose (S + 1) = 32, which reveals the best l-pixel scale metrics.
E Additional experiments and visualizations
In this section, we provide the additional experiment and visualizations with TrackIME.
The use of visibilities as the confidence weights. Our strategy combines both the hard 0-1 visibility predictions well the confidence weights (e.g., Equation (3)). This strategy effectively mitigates potentially erroneous confidences by the false positives, since our method tends to demonstrate a high precision (the portion of true positives) for the visibility classification, e.g., we get 93.7% the threshold 0.5 in DAVIS-F. Our strategy is valid as far as sufficient number of visible tracking points available. For the cases where object is occluded for few frames and then reappears, framework can maintain the number of tracking points via the point re-sampling, even if the visibility classifier fails to predict the reappearance.
To further support the validity of our strategy, we measure the average confidence of the true positives and the false positives and find 0.902 (true positives) and 0.737 (false positives), so the remaining false positives would be penalized through the weighted aggregation. We also provide additional study in Table 11, where we force equal weights in the aggregation experimented in DAVIS-F. For example, we find 1.3 points improvement by using our strategy.
TAPNet results with alternative checkpoint. In our main experiments, we have utilized ResNer18 backbone image backbone provided by the official checkpoint to reproduce TAPNet [11] and TAPIR [6] results, instead of T'SM-ReNNetl used by TAPNet in the original paper [11]. In Table 12,
17

---

Table 11: Forcing equal weights in the aggregation. We ablate the use of aggregation weights in our method. For the evaluation, we calculate both pixel-scale and average scale metrics under DAVIS-F dataset [11].
Method
J1
AJ
of
0
OA
avg
TrackIME (equal weights)
34.9
64.0
47.9
78.5
86.3
TrackIME (default)
35.4
65.3
48.2
78.6
86.5
additionally provide the results based the checkpoint with the original TSM-ResNet18 8 image backbone. When experimented with DAVIS-F and DAVIS-S, we find TrackIME keeps demonstrating significant gains, e-g., 32.8 47.0 AJs (14.2 points) in DAVIS-F.
Table 12: TAPNet results with an alternative checkpoint. We experiment with the use original 1SM-ResNet18 18 image backbone for the TAPNet baseline [11]. For the evaluation, we calculate both pixel-scale and average scale metrics under DAVIS-F and DAVIS datasets [11].
First Query
Strided Query
Method
J1
AJ
87
8
OA
J1
AJ
5f
OA
TAPNet [11]
5.8
32.8
11.1
48.4
77.6
6.7
38.4
12.6
53.4
81.4
TrackIME
18.7
47.0
28.6
60.6
80.9
21.5
50.8
32.4
63.8
81.4
Visualization of the progressive inference. We additionally visualize the progressive inference incorporated TrackiME with TAPIR [6] and apply the progressive pruning sizes of (960 x 960) and (384 x 384). As depicted by Figure 3, the latest progressive step is well focused around the query point, e.g., the dog ' car, that the search space for point tracking is effectively pruned.
structure in Figure 3. Specifically,
F
Limitation
F.1 Limitation and Future Works
TrackIME relies on the pre-trained models for point tracking, often trained with synthetic datasets, such Kubric [36] and PointOdyssey [9], while the segmentation models are primarily trained on the real images [1]. An interesting future direction is to integrate the TrackIME with training on the real video scenes. As this could include the development of point tracking algorithms capable of generalization to diverse intricate objects alternatively, optimizing the segmentation models for better video scene understanding. This approach could further improve the accuracy and applicability of TrackIME in various real-world scenarios.
F.2 Potential Negative Societal Impact
While point tracking by TrackIME can be beneficial for various video understanding applications. such as novel-view synthesis, depth estimation, and action recognition, the emergence of unexpected behavior within TrackIME can lead to misrepresentations of the real video data. For those applications that require extremely accurate models for -elate- judgements, such as depth estimation for autonomous driving, the unexpected behaviors must be carefully managed. ensure the reliability of systems using point tracking predictions, recommend to conduct thorough investigations and implement robust mitigation strategies to minimize potential risks, thereby increasing the overall safety and effectiveness of these applications.
18

---

Query Point p(9)
Semantic Neighbors N (pp(a) -
1=
Segment Anything
Seg(p I)
Semantic Neighbors Tracking
Pruned Frame
Query Trajectory
Query Point pl
Semantic Neighbors (xp(s)) 1
Segment Anything
Seg(p19) I)
Semantic Neighbors Tracking
1
1 10
1=
TrackIME
Pruned Frame
Query Trajectory
Figure 3: Demonstration of the progressive inference by TrackIME framework.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims in the introduction and abstract accurately reflect the contribution and scope, which are then verified in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix F discusses it. The trade-off's regarding the hyperparameter selection is discussed in Section 4.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer:
[NA]
Justification:
We do not have a theory in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have included the implementation details of TrackIME in Section and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will keep updating the open-soruce repository and the project page https://trackee ..e..gitii github.io.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
(oollc/guides/Codeeee
wwteimisssnoooliiiio onPolicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wttpp.ccpppuiiiieeeeeeeepppppppp wctpublic//aeccccccccccccccc /gwpi//g///////eeeeeeo /Ce///eeeubbiiicc cetsSuuumisiinss ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the detail of the training/evaluation setup, dataset, and hyperpa- rameters in Section 4 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
All experiments are conducted with the same and commonly used random
seed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computational costs Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:: httppp::////uui.....// cc. c/Ethi csGuidel dnli?
Answer: [Yes]
Justification:
We do have any ethical concerns regarding the paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the societal impact in Appendix F
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method does not introduce risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all papers and datasets used in the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi wwwode.hho....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We will release the Pytorch implementation of TrackIME after the acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We use existing benchmark datasets and do not have any crowdsourcing datasets experiments in the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have human subject in the research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "RCPqf6lZ66", "reviewer": "Reviewer_jQxW", "summary": "This paper tackles the problem of point tracking, where the task is to track the movement of a single point in a video. Point tracking has experienced a fairly recent deep learning revival starting with PIPs [22 in paper ref], which was inspired by a handcrafted method named Particle Video from Sand and Teller [1], with more recent follow-ups such as TAPIR [6 in paper ref], Omnimotion [8 in paper ref], and Cotracker [7 in paper ref].\n\nSpecifically, this paper tackles a key challenge faced by these contemporary point tracking models: the computational burden introduced by the cost volume operation, which is a correlation between the pointwise feature being tracked and the feature maps of all video frames in the current window being processed. This operation needs to be performed for each point being tracked, resulting in tensors as large as B x T x N x C x H x W (batch size, window length, number of points being tracked, num feature channels, video height, video width) being processed. In response, these models spatially downsample the feature space by up to a factor of 4 (in Cotracker's case) and 8 (in PIPs' case), which results in a decimation of high frequency information that can be useful in tracking points. As a consequence, these models experience a large reduction in tracking accuracy due to the downsampling.\n\nThe authors of this paper propose an alternative approach. Instead of downsampling the feature space, intelligently crop each frame to the most important region in the frame, i.e., somewhere around the point being tracked. The authors introduce a framework that takes advantage of a pretrained instance segmentation model (SAM [1 in paper ref] in this case) to segment the instance upon which the queried point is placed, to use the segmentation mask to query semantically neighbouring points, to track all these points using a pretrained tracker (TAPIR, PIPs, Cotracker, etc.), to aggregate all trajectories to a single instance trajectory, to crop each video frame about each instance trajectory point, and to re-do tracking of the original queried point in this pruned space. They even show how this can be performed recursively / progressively, where the entire process can be repeated to get an even more refined trajectory estimate. The authors also show how their framework can act as an accurate video object segmentation model, where the synergy between the point tracking and the segmentation model allows both to improve on their respective tasks (tracking and segmenting). They show how this can exceed the zero-shot video object segmentation performance (in the DAVIS dataset) of SAM-PT (a version of SAM that relies on a point tracking model to enable zero-shot video object segmentation) and various class-based video object segmentation models.\n\nThe authors evaluate their framework through exhaustive experimentation, with four main experiments:\n* Evaluating point tracking performance for dynamic objects (i.e., testing on TAP-Vid-DAVIS)\n* Testing the generalizability of their framework on different point tracking models.\n* An ablation study where they ablate different components of their model (search space pruning using tracked query point vs. instance trajectory, and the recursive pruning scheme).\n* Zero-shot video object segmentation.\n\nOverall, the authors show substantial gains on standard benchmarks and demonstrate the synergistic effectiveness of using a pretrained instance segmentation model with a pretrained point tracker. They even show that there is no net computational efficiency loss in the additional FLOPS introduced by the segmentation model and the recursive process, since the tracking accuracy gains far outweigh the additional compute costs.\n\n[1] Sand, P., Teller, S. Particle video: Long-range motion estimation using point trajectories. In CVPR 2006.", "questions": "* L128: I suggest sharing the threshold value here instead of saying \"is set much smaller than the standard 0.5\". It seems like an unnecessary obfuscation.\n* L185-195: I strongly suggest mentioning the ablation study here, as it is one of your major experiments.\n* Table 3: I suggest mentioning in the caption that the tracking backbone is TAPIR. I understand this is mentioned in a couple of places in the main manuscript, but it helps to remind the reader when reading the table caption, as this piece of information can be accidentally skipped when skimming through the main manuscript.\n* L274: \"23.2\" is supposed to be 28.2, right?\n* L297-298: Can you provide some brief commentary on why TrackIME may be performing better than SAM-PT on zero-shot video object segmentation? I'm curious to know your thoughts. If insightful, it may even be useful to include it in the main manuscript.\n* Table 7: Are the resolutions displayed here (excluding TrackIME since there's no training involved) the input resolutions during training or during testing?\n* Figure 2: The points and the images are too small! I suggest increasing the size of the points, removing the last column of images, and increasing the size of the image grid to match the width of the caption. I liked the size of the points shown in Figure 1.\n* Figure 3: The points are even harder to see here than in Figure 2 despite the images being larger! This one's an easy fix: increase the size of the points (preferably to match the size of the points in Fig 1 for consistency).\n\nGreat paper overall. Good job :)", "limitations": "The authors have adequately addressed the limitations of their framework and its potential negative societal impacts. Limitations have been discussed in Appendix F and Section 4, with honest and descriptive commentary. Potential negative societal impacts have been sufficiently discussed in Appendix F.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Originality:\n    * One might point to SAM-PT as a similar idea, but as the authors dutifully pointed out in the paper, it is a different approach. SAM-PT uses a point tracker to (effectively) track the movement of an instance, and when combined with the instance segmentation capabilities of SAM, this effectively results in a video object segmentation model in a zero-shot manner. In contrast, this paper introduces a novel approach to improving point tracking accuracy (the other way around compared to SAM-PT) where SAM is used to restrict the tracking search space, allowing them to track without downsampling the feature space spatial dimensionality and demonstrating substantial improvements by doing so. No other tracking model, as far as I'm aware, have introduced anything close to this method.\n    * Related works have been adequately cited and the paper is very clear in how it differs from prior methods.\n* Quality:\n    * Claims are well supported by a thorough analysis on the results of exhaustive experimentation.\n    * Authors are careful and honest about evaluating both the strengths and weaknesses of their framework.\n* Clarity:\n    * The paper is concisely written. At no point did I have trouble understanding what the authors were trying to communicate, whether it was through their math or vocabulary. I'd like to commend the authors in the care taken to introduce each concept. For example, I enjoyed their clarification of mathematical notation in L67-70 and their concise description of the benchmark metrics in L225-233.\n    * The tables and their captions are well-presented, clearly showing the gains of their framework.\n    * All experiment parameters necessary to recreate results have been listed, explained, and motivated both in the main manuscript and in the appendix.\n    * Overall, just about every question I can think of as I was reading the paper was almost immediately answered as I kept reading. This, to me, is an indication of a well thought out paper that flows nicely.\n* Significance:\n    * Definitely the most impressive part of the paper: the results. They are significant in two ways, in my opinion: 1) The framework improves on the state of the art across almost all benchmarks and with all the backbone tracking models they used; 2) This framework is extremely flexible in that you can mix and match any pretrained tracker and pretrained instance segmentation model. It's not tied to a specific model on either the tracking side or instance segmentation side.", "weaknesses": "* Originality:\n    * No weaknesses.\n* Quality:\n    * No weaknesses.\n* Clarity:\n    * No major weaknesses, but I do have some suggestions and questions relating to clarity. I have provided these in the Questions section below.\n* Significance:\n    * No weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer jQxW,\n\nThank you for your valuable feedback and very detailed comments. We appreciate your remarks on the strengths of our paper, including the significance of the method, exhaustive experimentation, and clear presentation. We will address your concerns and questions in the response below.\n\n---\n\n**[Q1] Commentary on why TrackIME may be performing better than SAM-PT on zero-shot video object segmentation**\n\nThe key advantage of TrackIME is removing erroneous query points for segmentation caused by the tracking failure on intricate object parts, and enabling even finer query points for segmentation, e.g., demonstrating outstanding results in 1-pixel thresholds (e.g., Table 1), which is possible due to the pruning structure in our framework to maintain the high-frequency information. Although SAM-PT contributes to enhancing the segmentation from the point tracking task, their performance is still bounded by low-resolution tracking and suffers from tracking failures.\n\nSuppose SAM could provide perfect segmentation for a given query, and the point tracking model could provide perfect tracking and occlusion predictions up to their resolutions. Nevertheless, the tracking resolution would impose an upper bound for the video object segmentation performance because the query points in the fine-grained object parts suffer from failure modes in lower resolutions.\n\nWe would like to attribute the better performance demonstrated by TrackIME to this point.\n\n---\n\n**[Q2] Table 7: Are the resolutions displayed here (excluding TrackIME since there's no training involved) the input resolutions during training or during testing?**\n\nThe resolutions for TAPIR models in Table 7 are the inference resolutions, where we utilized the official checkpoint trained in the 256 x 256 resolution. We will clarify this point in the updated manuscript.\n\n---\n\n**[Q3] L274: \"23.2\" is supposed to be 28.2, right?**\n\nYes. We will correct this typo in the updated manuscript.\n\n---\n\n**[Q4] Editorial comments**\n\nThank you for the constructive editorial comments. In the updated manuscript, we will provide hyperparameters more clearly (e.g., $r=0.10$ for the segmentation), revise the presentation to mention our ablation studies in the introduction of Section 4 and that our tracking backbone is TAPIR in Table 3, and revamp the drawings in Figures 2 and 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kt1ihbZVY7", "reviewer": "Reviewer_BSTo", "summary": "This paper presents a new framework for video point tracking from instance motion. By integrating existing segmentation and point tracking base models, the performance of point tracking is significantly improved from object-by-object optimization. Ablation experiments and extensions in zero-shot video object segmentation further validate the effectiveness of the proposed framework. The paper is generally well written and organized.", "questions": "All five methods compared in Table 1 are inconsistent with the results reported in their original papers, most notably TAPNet's AJ which was 46.6 in the original paper but 56.5 here. The reasons for these inconsistencies need to be explained. This raises doubts as to whether the significant performance gains (from 62.8 to 69.3) of the proposed method can be compared to recent competitors (e.g. 65.9 of DOT). \n\nDense Optical Tracking: Connecting the Dots, CVPR 2024. \n\nL431 mentions that changing the backend of the model improves performance, which is confusing. As I understand this process donot retrain the model. Did the authors confirm that the implementation and evaluation is correct, or are there some bugs that were addressed in this process?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Instance segmentation and motion estimation provide an overall motion prior from the global perspective, which strengthens the spatial associations for point tracking. Based on this, coupled with an iterative post-processing step to mitigate the loss of visual information due to feature space downsampling, also contributes to high-resolution point tracking. \n\nAblation experiments and extensions in zero-shot video object segmentation further validate the effectiveness of the proposed solution.", "weaknesses": "At the trajectory aggregation step (eq 3), multiple moving points are usually selected corresponding to each object, but just averaging them according to visibility is to simple. Direct averaging does not satisfy common cases such as rotations and scale changes, and even if this is used for subsequent inference, a better initial value would be valuable. A straightforward modification would be to fit the affine motion model with these selected points. \n\nFor eq 3, 8 and 12, whether the visibilities can be regarded as confidence weights needs to be further analysed, since the supervision only have a 0-1 occlusion maps. Does this address the cases where an object is occluded for a few frames and then reappears?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer BSTo,\n\nThank you for your valuable feedback and comments. We appreciate your remarks on the strengths of our paper, including the enabling high-resolution point tracking, the valid experiments for ablation, and the extended results in zero-shot video object segmentation. We will address your concerns and questions and provide additional results in the response below, which we will also include in the updated manuscript.\n\n---\n**[W1] Employing more complex motion models for the trajectory aggregation**\n\nThank you for the great suggestion. Although more complex motion models can satisfy cases such as rotations and scale changes, we find only marginal changes compared to our simple design, when applied to the trajectory aggregation step (Equations 3 and 4). For example, we provide the results employing an affine motion model (i.e., $T_t = A_t * T_{t-1} + b_t$) [R1] by fitting the least-square solution for $A_t$ and $b_t$ using the baseline tracking results, experimented on DAVIS-F benchmark.\n\nWe would like to emphasize that the trajectory aggregation is not required to predict accurate motion, since its purpose is only to determine the center of pruning windows. Even if the center escapes out of the object due to complex motions, it still suffices as long as the pruning window can contain the query point.\n\n|     Method     |       $J_1$ |  $\\text{AJ}$  | $\\delta^x_1$ | $\\delta^x_{\\text{avg}}$ | $\\text{OA}$  |\n|------------|:--------:|:------:|:------:|:------:|:------:|\n| TrackIME (w/ Affine motion) |  35.3 | 65.1 |48.0 | 78.4 | 85.7 |\n| **TrackIME (default)** |  **35.4** | **65.3** | **48.2** | **78.6** | **86.5** |\n\n[R1] Li, et al. \"An efficient four-parameter affine motion model for video coding.\" IEEE Transactions on Circuits and Systems for Video Technology 28.8 (2017): 1934-1948.\n\n---\n\n**[W2] The use of visibilities as the confidence weights in Equations 3, 8, and 12.**\n\nThank you for acknowledging the importance of confidence weights. We first would like to clarify that our strategy combines both the hard 0-1 visibility predictions as well as the confidence weights (e.g., see $\\mathbf{o}_t \\geq 0.5$ in Equations 3 and 8). This strategy effectively mitigates potentially erroneous confidences by the false positives, since our method tends to demonstrate a high precision (the portion of true positives) for the visibility classification, e.g., we get  93.7% @ threshold 0.5 in DAVIS-F. Our strategy is valid as far as a sufficient number of visible tracking points are available. For the cases where an object is occluded for a few frames and then reappears, our framework can maintain the number of tracking points via the point re-sampling, even if the visibility classifier fails to predict the reappearance.\n\nTo further support the validity of our strategy, we measure the average confidence of the true positives and the false positives and find 0.902 (true positives) and 0.737 (false positives), so the remaining false positives would be penalized through the weighted aggregation. We also provide additional ablation, where we force equal weights in the aggregation, experimented in DAVIS-F. For example, we find 1.3 points improvement in $\\text{AJ}$ by using our strategy.\n\n|   Method  |    $J_1$ |  $\\text{AJ}$  | $\\delta^x_1$ | $\\delta^x_{\\text{avg}}$ | $\\text{OA}$  |\n|------------|:--------:|:------:|:------:|:------:|:------:|\n| TrackIME (equal weights) |  34.9 | 64.0 | 47.9 | 78.5 | 86.3 |\n| **TrackIME (default)** |  **35.4** | **65.3** | **48.2** | **78.6** | **86.5** |\n\n---\n\n**[Q1] Explanations for the inconsistent results with the baselines’ original reports**\n- **Inconsistent TAPNet results in Table 1**\n\nThis inconsistency is mainly caused by the difference in the image backbone provided by the official checkpoint we utilized to reproduce TAPNet and TAPIR; it provides an updated ResNet18 backbone, instead of TSM-ResNet18 used by TAPNet in the original paper. Otherwise, the tracking method is the same as TAPNet (i.e., cost volumes without PIPs iterations).\n\nNevertheless, we additionally provide the results based on the checkpoint with the original TSM-ResNet18 image backbone (which we find in the recent update of the official repository). When experimented with DAVIS-F, DAVIS-S, and Kinetics benchmarks, we find TrackIME keeps demonstrating significant gains, e.g., 32.8 $\\to$ 47.0 AJs (14.2 points) in DAVIS-F. We will include these discussions and results in the updated manuscript.\n\n\\begin{array}{l  c  c  c  c  c | c  c  c  c  c | c  c  c  c  c}\n&  & & &  \\llap{\\text{DAVIS}-\\text{F}} & & & & & \\llap{\\text{DAVIS}-\\text{S}} & & & & & \\llap{\\text{Kinetics}~~~~} \\newline\n\\text{Method} & J_1 & \\text{AJ} & \\delta^x_1 & \\delta^x_\\text{avg} & \\text{OA} & J_1 & \\text{AJ} & \\delta^x_1 & \\delta^x_\\text{avg} & \\text{OA} & J_1 & \\text{AJ} & \\delta^x_1 & \\delta^x_\\text{avg} & \\text{OA} \\newline\n\\hline\n\\text{TAPNet (TSM-ResNet)} & 5.8 & 32.8 & 11.1 & 48.4 & 77.6 & 6.7 & 38.4 & 12.6 & 53.4 & 81.4 & 8.1 & 38.3 & 14.4 & 52.5 & 79.3 \\newline\n+~\\textbf{TrackIME} & \\mathbf{18.7} & \\mathbf{47.0} & \\mathbf{28.6} & \\mathbf{60.6} & \\mathbf{80.9} & \\mathbf{21.5} & \\mathbf{50.8} & \\mathbf{32.4} & \\mathbf{63.8} & \\mathbf{81.4} & \\mathbf{17.0} & \\mathbf{44.5} & \\mathbf{26.3} & \\mathbf{57.9} & \\mathbf{80.4} \\newline\n\\hline\n\\end{array}\n\n- **The effect of library/hardware-dependent numerical characteristics**\n\nSince TrackIME is a plug-in to all baselines, we reproduced all results in our system for fair comparisons. This induces minor inconsistencies due to library/hardware-dependent numerical characteristics, even if the same model is used. For example, the improved performance of TAPIR (mentioned in L431) is caused by the different characteristics between JAX and PyTorch. We note that such inconsistencies are also observed in other references, e.g., $\\delta^x_{\\text{avg}}$ for PIPS2 on DAVIS-F: 69.1 (reported by CoTracker paper); 70.6 (reported by PIPS2 git repository); 69.4 (reported by our paper).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jQszl0c5AG", "reviewer": "Reviewer_7a51", "summary": "In this paper,  the authors propose TrackIME, a framework to tracking points in video. The proposed TrackIME leverages a segmentation model to improve both its efficient and effectiveness. TrackIME achieves SOTA performance on TAP-VID dataset.", "questions": "1. The authors only compare FLOPs between TrackIME and TAPIR, why not also compare with other methods like CoTracker?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method achieves competitive results while retaining a low computation cost (comparing with TAPIR)\n2. Using segmentation model to help points tracking has never been explored by prior works.", "weaknesses": "1. The main contribution of proposed method is not clear to me. Using segmentation model to help track points within an instance is an intuitive top-down solution for points tracking\n2. The proposed method outperform SAM-PT in a relative small margin in terms of instance-level tracking, which raise my concerns about the capability of model for tracking points. It seems to me that the improvement on the point tracking performance is limited to help tracking instances/objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 7a51,\n\nThank you for your valuable feedback and comments. We appreciate your remarks on the strengths of our paper, including the competitive results, low computation cost, and the first introduction of the segmentation model in point tracking tasks. We will address your concerns and questions in the response below.\n\n---\n\n**[W1] The main contribution of our paper**\n\nThank you for acknowledging our intuitive idea of using segmentation models to enhance point tracking. We list our main contributions, which are commonly acknowledged by the reviewers jQxW and BSTo, as follows:\n\n- Designing a flexible model that can combine different pre-trained tracker and pre-trained instance segmentation models, not tied to a specific model\n- A new idea of pruning input video frames for point tracking, which addresses a fundamental drawback in existing point tracking models caused by the video down-sampling structure\n- A joint framework for the point tracking and the segmentation tasks, which bolsters the performance of the two tasks in a synergistic manner\n\nAs pointed out by the reviewer jQxW, our method addresses the fundamental limitation of losing high-frequency information crucial for point tracking, innovating the way current point tracking models handle high-dimensional video inputs. \n\nAs the segmentation improves point tracking in our framework, it can also enhance the segmentation task in a synergistic manner. Specifically, the enhanced fine-grained tracking performance (e.g., 1-pixel errors in Table 1) can bolster the segmentation task by providing more accurate queries for segmentation. While prior art (e.g., SAM-PT) contributed to enhancing the segmentation from the point tracking task, there has been no recursive structure similar to ours, and their performance gain is only a one-off.\n\nFinally, we note that our method does not require fine-tuning, and is pluggable to different existing models (Table 2). It is also worth noting that the efficacy of our pruning method is significant even if the segmentation model is not employed (e.g., 57.5 -> 62.5 AJ in Table 3).\n\n---\n\n**[W2] Comparison to SAM-PT**\n\nDespite the margin being relatively small in the segmentation task, we note that our primary focus is point tracking where our framework demonstrates significant enhancements. \n\nNevertheless, we also find that jointly performing the tracking and segmentation in our framework is useful for the segmentation task as well, achieving improved results than powerful baselines, such as SAM-PT. These segmentation baselines, on the other hand, do not provide point tracking functionality.\n\nWe believe the impact of our finding is not limited to helping a tracking task, but can also inspire new methods for video segmentation by considering a synergistic effect between the segmentation and different tasks.\n\n---\n\n**[Q1] Additional FLOPs comparison in a different baseline**\n\nSince our framework is a plug-in for a point tracking model, directly comparing FLOPs of our main TrackIME (incorporated with TAPIR) and other baseline (e.g., CoTracker) would not be a lateral comparison.\n\nNevertheless, we provide the additional FLOPs results for CoTracker processing 64 frames in three different input dimensions: 384, 768, and 1080 pixels at the shorter side, and our TrackIME incorporated with CoTracker, experimented in DAVIS-F. In a similar trend as our experiment based on TAPIR, the benefit from the larger input dimensions is limited since the baseline tracker is only optimized for low-resolution input frames (384 × 512) to meet the memory constraints, and it is non-trivial to process high-resolution inputs without fine-tuning. When incorporated with TrackIME, on the other hand, the model can better utilize high-frequency details without fine-tuning and in a computationally efficient manner, e.g., TrackIME vs. CoTracker (1080 $\\times$ 1440) gives **6217G** vs. 14138G (FLOPs), **64.5** vs. 62.2 (AJ), **79.2** vs. 76.7 ($\\delta^{x}_{\\text{avg}}$), **88.5** vs. 86.6 (OA), respectively.\n\nWe hope this answers the question and will include these results in the updated manuscript.\n\n\\begin{array}{l  c |  c  c  c  }\n\\text{Method} ~ (\\text{Input Dim.}) & \\text{FLOPs} & \\text{AJ} & \\delta^x_\\text{avg}  & \\text{OA}  \\newline\n\\hline\n\\text{CoTracker} ~ (384 \\times 512) & 2707 \\text{G} & 60.8 & 76.1 & 86.0 \\newline\n\\text{CoTracker} ~ (768 \\times 1024) & 7670 \\text{G} & 62.3 & 77.8 & 87.1 \\newline\n\\text{CoTracker} ~ (1080 \\times 1440) & 14138 \\text{G} & 62.2 & 76.7 & 86.6 \\newline\n\\hline\n\\textbf{TrackIME} ~ \\text{with CoTracker} \\newline (384 \\times 512) & 6217 \\text{G} & \\mathbf{64.5} & \\mathbf {79.2} & \\mathbf{88.5}  \\newline\n & \\newline\n\\hline\n\\end{array}", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and AC,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript.

As the reviewers highlighted, we believe our paper provides a novel approach that incorporates point tracking models with the pre-trained instance segmentation (all reviewers). This approach provides significant performance improvements (jQxW, BsTo) in a computationally efficient manner (jQxW, 7a51), followed by a clear presentation (jQxW, BsTo).

We appreciate your constructive comments on our manuscript.

We strongly believe that TrackIME can be a useful addition to the NeurIPS community, in particular, due to the enhanced manuscript by reviewers’ comments helping us better deliver the effectiveness of our method.

Thank you very much!

Authors

Meta-review: This paper presents a method to improve the performance and computational efficiency of point trackers, by adding awareness of the objects that the points belong to, via a pre-trained segmentation model. Despite using off-the-shelf models (without finetuning them), the method achieves substantial improvements in point tracking accuracy, across a wide array of base models, and delivers state-of-the-art zero-shot video segmentation as well. The reviews were initially leaning positive, and became more confident after the rebuttal, arriving at 1x strong accept, 1x weak accept, 1x borderline accept. The reviewers appreciated the core idea, the results, and the presentation, and the AC agrees. The authors are encouraged to incorporate the work done during the rebuttal, to strengthen the paper further. The AC recommends acceptance.
Decision: Accept (spotlight)