TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation

Authors: seong hyeon park
huiwon  jang
byungwoo  jeon
sukmin  yun
paul hongsuck seo
jinwoo  shin
  kaist
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:13:39

Abstract: Tracking points in video frames is essential for understanding video content. However, the task is fundamentally hindered by the computation demands for brute-force correspondence matching across the frames. As the current models down-sample the frame resolutions to mitigate this challenge, they fall short in accurately representing point trajectories due to information truncation. Instead, we address the challenge by pruning the search space for point tracking and let the model process only the important regions of the frames without down-sampling. Our first key idea is to identify the object instance and its trajectory over the frames, then prune the regions of the frame that do not contain the instance. Concretely, to estimate the instance’s trajectory, we track a group of points on the instance and aggregate their motion trajectories. Furthermore, to deal with the occlusions in complex scenes, we propose to compensate for the occluded points while tracking. To this end, we introduce a unified framework that jointly performs point tracking and segmentation, providing synergistic effects between the two tasks. For example, the segmentation results enable a tracking model to avoid the occluded points referring to the instance mask, and conversely, the improved tracking results can help to produce more accurate segmentation masks. Our framework can be easily incorporated with various tracking models, and we demonstrate its efficacy for enhanced point tracking throughout extensive experiments. For example, on the recent TAP-Vid benchmark, our framework consistently improves all baselines, e.g., up to 13.5% improvement on the average Jaccard metric.

Main paper: 
 introduction: 
 Obtaining accurate point trajectories over the video frames is crucial for understanding complex dynamics in video data, a necessity for advanced spatial-temporal tasks like action recognition [2] , novel-view rendering [3] , video frame prediction/interpolation [4] , and video depth estimation [5] . Recently, video point tracking task [6, 7, 8, 9, 10] has witnessed rapid progress, which aims to predict the trajectory and visibility1 of a given query point, proving long-term trajectories robust to partial occlusions of objects in real video scenes.Despite their success, we find current point tracking models are fundamentally challenged by an excessive computation demand since the task requires brute-force comparisons over every spatial location in every frame in a given video. As a result, to meet the computation constraints, the models To estimate the instance trajectory, our framework utilizes the point tracking results for a group of points (blue lines) on top of the object instance predicted by segmentation model (e.g., SAM [1] ) and aggregate their individual trajectories.down-sample their tracking resolutions, sacrificing detailed visual features, which eventually leads to sub-optimal tracking accuracy and triggers tracking failures on intricate object parts. In this regard, we pursue the direction of pruning the excessive search space for point tracking, so that models can avoid the down-sampling and focus only on important regions maintaining detailed visual features, e.g., the object instance masks the query point lies in.In this paper, we introduce TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation that focuses on the region occupied by the object instance that the queried point lies in and guides point tracking models to prune the video frames along the instance's motion trajectory. Here, to obtain the instance trajectory, we first produce the instance mask for a given query point by utilizing the recent segmentation foundation models, e.g., segment anything (SAM) [1] , where these foundation models show strong generalization performance to different objects/scenes and we find resulting instance masks in quality are readily available. Then, given the instance mask, we sample a set of points and aggregate their tracking results as the estimate of the instance trajectory. 2Furthermore, to deal with the occlusions in complex video scenes, we propose a unified framework that jointly performs the point tracking and video segmentation, where it re-samples the occluded points by referring to the instance mask. We note that our framework provides synergistic effects for both tasks, i.e., the point tracking results assisted by the segmentation can conversely bolster the quality of segmentation. Consequently, although our primary focus is on the advances in point tracking, our method can also demonstrate improved segmentation results than the baselines (see Section 4.2 for details).Through the experiments on the TAP-Vid point tracking benchmark [11] , we demonstrate the effectiveness of TrackIME by incorporating it with different point tracking models such as TAPIR [6] . For example, in the DAVIS scenes [12] evaluating the point tracking for dynamic objects, our method achieved up to 13.5% relative improvement (57.5 → 65.3 with TAPIR) in terms of the average Jaccard (AJ) metric. Moreover, as our framework allows pruning non-instance regions for point tracking models, the efficacy of our method stands out even more when evaluated on more harsh standards, e.g., the 1-pixel error threshold, where the conventional metrics allow up to 16-pixel errors when judging the prediction to be correct. 
method: 
 In this section, we describe the detailed procedure of our TrackIME framework and its application to video point tracking. Specifically, in Section 2.1, we describe the formulation for instance trajectory estimation, which is based on the video point tracking of the query points found by the foundation segmentation model [1] .Next, in Section 2.2, we present the detailed formulation of TrackIME given the instance trajectory, which prunes unimportant regions in the video frame and achieves boosted point tracking performances.As for the data notations, we denote vectors with N elements as bold letters x := [x 1 ; x 2 ; ...;x N ], tensors with N arrays as bold capital letters X := [X 1 ; X 2 ; ...; X N ], where the subscripts represent the indexed scalars or arrays. Otherwise, every non-bold symbol is scalar. We also introduce the superscripts, e.g., x (q) , when denoting there is special semantics for a data, such as the query point.Finally, when making binary classifications based on probability (or normalized confidence) values, we use threshold 0.5; nevertheless, the values are hyperparameters and can be altered in practice.In this section, we provide the definition of the instance trajectory and procedures to obtain it, such as sampling a group of points on the instance, trajectory aggregation, and the point re-sampling modules.Video point tracking. Let I ∈ R L×H×W ×3 be the tensor of video frames, where L denotes the time duration and (H × W ) denotes the image size, and let p (q) ∈ R 2 be the spatial coordinates of the query point. Typically, we consider the query in the initial frame hence we do not denote the time index of the query point for clarity. Given the video I and the query point p (q) , we consider a point tracking model Tracker that predicts the query trajectory T (q) ∈ R L×2 and the probability of being visible o (q) ∈ (0, 1) L over the entire set of frames, (T (q) , o (q) ) := Tracker(p (q) , I).Here, one might utilize Equation (1) as the simplest representation of the instance motion trajectory. However, modeling the instance motion solely using the query point has critical shortcomings. For example, when the instance is partially occluded by other objects, the trajectory of the query point may no longer exist (see Section 4.1 for the ablation study).To address this challenge, we propose to sample additional tracking points automatically. Specifically, our idea is to identify the instance mask of the query point so that extra query points can be added from the mask.Sampling points on the instance. Let M 0 ∈ (0, 1) H×W denote the segmentation mask that represents the object instance associated with the query point p (q) . Given this mask, we sample a group of points on the instance,EQUATIONwhich we refer to it as the semantic neighbors of p (q) . We note that S is the number of sampled points, where the query point is also counted as its semantic neighbor, i.e., p (n0) ≡ p (q) .For each semantic neighbor point, we employ Tracker in Equation ( 1) to produce its trajectory and visibility, T (ni) , o (ni) := Tracker p (ni) , I , 3 and pass it to the trajectory aggregation module. Since the query point also participate in our tracking procedure, the total effective number of points would be S + 1. For example, we choose S + 1 = 32 in our main experiments discussed in Section 2.Trajectory aggregation. We produce an instance motion trajectory by aggregating the tracking results of the semantic neighbors. Specifically, we consider the velocity, ∆T(ni) t := T (ni) t -T (ni)t-1 , and calculate the weighted average:EQUATIONIn Equation (3), we note that velocities are aggregated only if the points are classified visible (o (ni) t ≥ 0.5), and the visibility acts as the aggregation weight. Finally, we accumulate the aggregated velocity starting from T (q) 0 := p (q) , to obtain the instance motion trajectory,EQUATIONInstance mask. In order to identify the instance mask, we employ the recent foundation segmentation model, e.g., Segment Anything Model (SAM) [1] , and prompt the model with the query point p (q) , to produce the pixel-wise confidence representing the object instance indicated by the query point. We denote this function as Seg,EQUATIONGiven the mask M 0 , we employ a weighted sampling for the semantic neighbors. Specifically, we encode the sampling weights with the distance transform (DT) [13, 14] to the mask's region with positive classifications,EQUATIONIn this way, the points near the mask's contour are preferred, which we find efficiently represent the object instance because the contour is approximately linearly proportional to the mask's radius.Point re-sampling for robustness to occlusion. Occlusions are common in real video frames, due to dynamic objects and the camera's motion. In the extreme case, Equation (3) can become degenerate when all semantic neighbors are invisible in future frames t > 0. Therefore, maintaining a sufficient number of visible tracking points is crucial, and we tackle this issue by re-sampling occluded points from the instance mask jointly predicted while point tracking.In a nutshell, whenever we find a certain semantic neighbor point p (ni) becomes invisible at time t ′ and does not show up again (o (ni) t < 0.5 for t ≥ t ′ ), we query the segmentation model with the tracking results of other semantic neighbors to obtain a new mask to re-sample the occluded point:EQUATIONHowever, they could also have been affected by occlusions (e.g., when o(nj ) t ′is close to the threshold 0.5), or by the severe errors in the trajectory T (nj ) t ′ due to sub-optimal tracking performance of Equation (1) . Hence, predicting segmentation with these points in a naive way can lead to erroneous masks being produced.To address this problem, our key idea is to aggregate the group of segmentation masks. Specifically, we collect individual masks by Equation (7) , then apply a weighted average of the positive classifications,EQUATION)We find the mask produced by Equation ( 8) reflects the confidence of each segmentation mask, as well as the visibility of the associated point, and refer to it as the mixture of segmentation distributions, where the value in each index represents the segmentation probability of the queried object.Based on the constructed mixture of segmentation distributions, we obtain the sampling weight in similar manner to Equation (6) as, W t ′ := DT(1[ Mt ′ ≥ r]), where the threshold r ∈ [0, 1) is set much smaller than the standard 0.5. 4 This is because we should allow the confident partial segmentation distributions, but ignore the unconfident noise segmentation distributions.Finally, we re-sample the additional points with W t ′ as the sampling weight. They replace the occluded points for the instance trajectory estimation in subsequent frames t > t ′ . We execute this procedure during the tracking, which ensures that a sufficient number of visible points participate in Equations ( 3) and (4). For example, we set it to be the same as the number of initial semantic neighbors S.In this section, we describe our enhanced point tracking, which prunes the search spaces in frames and produces more accurate tracking results.Search space pruning. Given the instance trajectory in Equation ( 4), we now aim to utilize it for pruning the search space. Specifically, we prune unimportant non-instance regions, by sampling each frame around the (H 0 × W 0 ) regions centered at the aggregated trajectory,EQUATION)We note that the sizes (H 0 × W 0 ) are set to be close to the down-sampling resolution considered by a tracking model (e.g., (256 × 256) for TAPIR [6] ) so that the information loss is minimized.Given the frames with pruned search spaces, we execute Tracker again to produce the enhanced tracking outputs. Also, for convenience, we abstract the entire process of the instance trajectory estimation (Section 2.1), the pruning (Equation ( 9)) and the tracking into a function TrackerHD, (T (HD) , o (HD) ) := TrackerHD(p (q) , I, H 0 , W 0 ) := Tracker(p (q) , I (q) ).We note that the feature resolutions inside the tracking model are not modified, therefore the computational complexity does not increase.Progressive inference. whereT (HD) k ∈ R L×2 and o (HD) k ∈ (0, 1) L denotes the outputs of the k-th TrackerHD model.This progressive structure can boost the tracking performance in two ways. The first is utilizing a past k-th TrackerHD as the tracking model that estimates the instance trajectory for the next (k + 1)-th TrackerHD. In this way, the pruning is guided by a more accurate trajectory estimate. The second is that these K tracking results can be aggregated to produce the final trajectory. Specifically, we aggregate based on the visibility, in a similar manner to Equations ( 3) and ( 8):EQUATION)where ⊙ indicates the element-wise product. This aggregation allows processing multiple scales in visual features, which can enhance the generalization performance of vision models [15, 16] . We note that the visibility predictions are averaged over the K predictions.Table 1 : The evaluation of point tracking performance for dynamic objects. We benchmark the quality of point tracking in DAVIS [12] videos with the point annotations provided by TAP-Net [11] . We note that TrackIME is incorporated with TAPIR point tracker [6] .First Query Strided Query [17, 18] , the seminal works such as DCFlow [19] , PWC-Net [20] and RAFT [21] introduced the concept of dense correspondence matching between pairs of image patches. Despite their success, the optical flow's inherent limitations incapable of modeling trajectories and occlusions triggered the recent progress in the point tracking methods.Method J 1 AJ δ x 1 δ x avg OA J 1 AJ δ x 1 δ x avg OA TAPNet [Point Tracking. In essence, point tracking attempts to find the long-term point correspondences over the entire video frames, and model the occlusions and trajectories. The current models in this domain, such as PIPs [22] , TAPNet [11] , TAPIR [6] , CoTracker [7] , and OmniMotion [8] has led rapid progress, with advanced neural architectures [6, 7] or test-time optimizations [8] . However, they are fundamentally hindered by the excessive search space for correspondence matching over the entire frames. Our focus is to address this issue by pruning the search space, where our method can be readily incorporated with these baselines.Instance Segmentation. Recently, the important advancement within image segmentation has been the introduction of segment anything (SAM) [1] . SAM is specifically designed to perform image segmentation by general point prompts and exhibits an impressive capacity for class-agnostic segmentation. Specifically, in the context of point tracking, SAM serves as a valuable resource by generating segmentation masks for the object instance indicated by the query point. We also note the line of zero-shot video segmentation [23, 24, 25, 26, 27, 28, 29, 30] . Specifically, the recent SAM-PT [30] focuses on bolstering video segmentation based on point tracking, which is fundamentally different from our work; our primary goal is obtaining better point tracking, while that for SAM-PT is for better segmentation. Nevertheless, our method provides synergistic effects for both tasks, and even outperforms SAM-PT for segmentation tasks (see Table 4 ). 
experiments: 
 In this section, we demonstrate the effectiveness of the proposed TrackIME on point tracking tasks and the downstream video object segmentation.In Section 4.1, we focus on the point tracking tasks. Specifically, we first experiment the efficacy of the instance motion trajectory estimation and our search space pruning technique for point tracking by measuring the performance in video scenes that capture dynamic objects.Next, we verify the universality of our method to different point tracking models and find whether it can provide general performance improvements when incorporated into the five recent baselines, e.g., TAPNet [11] , PIPS2 [9] , CoTracker [7] , OmniMotion [8] and TAPIR [6] .In the ablation study, we validate the effect of each component, namely the trajectory aggregation, the search space pruning, and the progressive inference modules described in Section 2.Table 2 : Universality of TrackIME with different point tracking models. We incorporate recent point tracking model baselines [6, 7, 8, 9, 11] with our method, and benchmark its performance on DAVIS [12] , RGBStacking [33] , and Kinetics [34] . †: the underlined results are obtained with subsets of RGBStacking and Kinetics datasets due to a large optimization cost for the OmniMotion [8] . In Section 4.2, we verify the efficacy of the enhanced point tracking results by TrackIME in the downstream video object segmentation. Specifically, we compare the zero-shot video segmentation performances with the recent SAM-PT [30] baseline which utilizes the point trajectories as the inputs, as well as the conventional baselines that input the semantic classes [27, 28, 29] .Common implementation details. We note that TrackIME is mainly incorporated with TAPIR point tracker [6] (as it empirically performs best) unless specified otherwise, and we subject it to all experiments including the point tracking and other downstream tasks.For the segmentation model, we utilize the Segment Anything (SAM) [1] to perform the point-queried segmentation function described in Equation (5).To prepare video frames, we always adjust the resolutions of raw video data to 1080p (1080 pixels in the shorter frame edges), then apply further resizing functions required by individual baseline models.For example, we resize the 1080p frames to 256 × 256 for TAPIR [6] baseline, following the default setting provided by the official open-source repository. When experimenting TrackIME, we choose the hyperparameters for each baseline, e.g., progressive inference steps K = 2, and the pruning sizes H 0 = W 0 = 960 and H 1 = W 1 = 384 when incorporated with TAPIR [6] .Since TrackIME is a plug-in to all baselines, we reproduce all results in the same system configuration for fair comparisons. We note that such modification can induce minor perturbation in the numerical values due to library and hardware-dependent characteristics, e.g., different characteristics between JAX [31] and PyTorch [32] libraries, and the difference in the filtering algorithm used when re-sizing the video frames. 5 We refer the readers to Appendix A for more implementation details.Baselines. We compare our method to the recent baselines OmniMotion [8] , CoTracker [7] , TAPIR [6] , PIPS2 [9] , and TAPNet [11] . We utilize the official checkpoints provided by the official project pages and reproduce all experimental results under our common experimental set-up, except for OmniMotion [8] which does not provide checkpoints. Instead, we reproduced the training of OmniMotion models to obtain the experimental results. We use S = 31 semantic neighbors to incorporate our framework with the baselines.Datasets. We evaluate these models on three different datasets, DAVIS [12] , Kinetics [34] , and RGB-Stacking [33] , each representing different characteristics. For example, DAVIS contains 30 videos Table 3 : Ablation study of the components in our model. We ablate the effect of search space pruning (Pruning), trajectory aggregation (Aggregation), and the progressive inference (Progressive) modules for point tracking. We evaluate the tracking benchmark in DAVIS scenes [11, 12] . For all datasets, we refer to the point tracking annotations provided by TAP-Vid [6] and utilize them as the ground truth for evaluation.To measure the quality of point tracking, we consider point tracking accuracy considered following TAP-Vid [11] , such as the δ-average accuracy (δ x avg ) and the average Jaccard (AJ). The average metrics are based on the δ-n accuracy (δ x n ) which indicates the proportion of correct trajectory sequence as judged by whether they are within the n-pixel error threshold around the ground truth. In addition, the Jaccard-n (J n ) judges a trajectory sequence to be correct only if the visibility prediction is also correct. Given these definitions, the average metrics are calculated by averaging n ∈ {1, 2, 4, 8, 16}. To evaluate the fine-grained tracking performance in a harsh error threshold, we also report δ-1 accuracy (δ x 1 ) and Jaccard-1 (J 1 ). For Table 1 , we also discuss the occlusion accuracy (OA), the proportion of correct visibility sequence given the ground truth.Effectiveness on point tracking in dynamic objects. We first present the point tracking scenarios with dynamic objects. Specifically, we experiment with the DAVIS video scenes [12] , which is curated for evaluating instance motion estimation tasks. As shown in Table 1 , we find our method achieves the best point tracking accuracy surpassing all baselines, e.g., up-to 7.4% relative improvements in average Jaccard, i.e., 60.8 AJ (CoTracker [7] ) vs. 65.3 AJ (TrackIME) when evaluated with the DAVIS-F (denoted First Query in Table 1 ). We also measure the occlusion accuracies (OA) and find a relatively incremental improvement than other metrics. Intuitively, there is a trade-off between modeling the occlusions among different objects and the search space pruning for one instance, as the pruning removes information from other instances. Nevertheless, our method is beneficial for detecting occlusion in fine-grained object parts, and we recommend searching for optimal pruning parameters that fit a user's purpose. Finally, we discuss the efficacy of TrackIME under the harsh δ x 1 and J 1 metrics, where the conventional metrics allows up to 16-pixel errors and takes the average when judging whether the prediction is correct. For example, the improvement can be even larger, e.g., up to relative 25.1%, i.e., 28.3 J 1 (CoTracker [7] ) vs. 35.4 J 1 (TrackIME) when evaluated with DAVIS-F. We highlight these benefits of TrackIME allowed by pruning the search space.Universality to different point tracking models. We validate the universality of our method when plugged into the state-of-the-art baselines by evaluating the average tracking accuracy (AJ and δ x n ) of the vanilla models and the variants incorporated with our method in Table 2 on DAVIS (First) [12] , RGBStacking [33] , and Kinetics [34] datasets. As a result, we observe that our method can provide consistent and significant performance improvements in all the baselines, e.g., 13.6% relative improvements (i.e., 57.5 → 65.3 AJ) in TAPIR [6] when evaluated on the DAVIS. Since the model variant incorporated with TAPIR demonstrates the best performance, we chose it as our main model and subjected it to other studies. We note that the experiments for OmniMotion [8] have been conducted in 16 subsets for RGBStacking and Kinetics, and K = 1 progressive inference, due to its heavy optimization costs, e.g., approximately 13 gpu-hours for processing one scene. We also note that PIPS2 [9] in Kinetics [12] is unavailable, as its memory requirement for processing Kinetics exceeds our system's capacity.Ablation study. We perform an ablation study to understand how each component affects the point trajectory accuracy in Table 3 . Specifically, we consider the search space pruning, the trajectory aggregation, and progressive inference modules as the subjects for the ablation.First of all, we reveal the pure efficacy of our pruning method, separate from the effect of segmentation prior. Notably, when the trajectory aggregation module is removed (the first 2 rows in Table 3 ), we observe the pruning solely based on the query point's trajectory provides the most significant effect (e.g., 23 .0 → 28.2 in J 1 ). This validates our key motivation for pruning the search space, which provides superior results even if SAM [1] is not employed. Next, we discuss the effect of employing SAM [1] by enabling the trajectory aggregation. As expected, aggregating the trajectories for a group of points found in the segmentation mask provides another comparable gain (e.g., 28.2 → 34.0 in J 1 ), which validates that the aggregation improves the quality of instance trajectory estimation.It is worth noting that the progressive inference boosts the performance, (e.g., 34.0 → 35.4 in J 1 ) when combined with the trajectory aggregation, otherwise the gain is lesser (e.g., 28.2 → 28.3 in J 1 ). As the progressive inference refers to the estimated instance trajectory, the estimation quality is essential for this module.We also note that further ablation study is available in Appendix D, e.g., the number of semantic neighbors, progressive inference steps, or the pruning sizes.In this section, we validate the efficacy of TrackIME by performing the zero-shot video segmentation. We also provide the visualization results for selected scenes from DAVIS [12] in Figure 2 .Baselines. We experiment with zero-shot video object segmentation to check the efficacy of TrackIME for improving segmentation. Specifically, we consider the class-guided baselines for unsupervised video segmentation tasks, e.g., EntitySeg [29] . In addition, we consider the SAM-PT [30] baseline which also proposes to take point tracking for producing segmentation. To consider the equivalent experimental set-ups for SAM-PT [30] and TrackIME, we incorporate the models with Table 4 : Zero-shot video object segmentation performance in DAVIS benchmark. We consider two set of zero-shot baselines, those utilizing the set of classes [23, 24, 25, 26, 27, 28, 29] and the baseline utilizing a set of query points [30] in a similar manner to our TrackIME. †: we produced the results for TrackIME and SAM-PT [30] under the common set-up, such as the number of tracking points, segmentation function (HQ-SAM [35] ), and the same mask formatting for the benchmark. HQ-SAM [35] variant for the segmentation, 16 points from the initial frame's mask, and employ the iterative refinement technique [35] to produce the video segmentation results.Evaluation. We evaluate our model on the DAVIS-2017 [12] video segmentation. In particular, we use the validation and the test-dev sets for the zero-shot benchmark. Both sets contain 30 non-overlapping scenes with single or multiple objects.To measure the quality of video instance segmentation, we consider the standard metrics in baselines: the mean Jaccard (J m ); the mean F-measure (F m ); and the average (J&F) m . Specifically, we follow the official implementation suite provided by the DAVIS challenge [12] .Effectiveness on zero-shot video object segmentation. In Table 4 , we first confirm that the point tracking provides useful guidance for video segmentation, observing that both SAM-PT [30] and TrackIME demonstrates significant improvement over the conventional class-prompted baselines. More importantly, as our framework brings synergistic improvements for both point tracking and segmentation tasks, we find TrackIME achieves even larger improvement, e.g., 78.8 vs. 79.6 (J&F) m in the validation set of DAVIS-2017 [12] .Discussions. As for the commentary on the efficacy of TrackIME, our key advantage is removing erroneous query points for segmentation caused by the tracking failure on intricate object parts, enabling even finer query points for segmentation, e.g., the accuracy in harsh 1-pixel thresholds in Table 1 , which is possible due to the pruning structure in our framework to maintain the highfrequency information. 
conclusion: 
 In this work, we introduce TrackIME, a novel approach for point tracking to overcome the fundamental challenge of computation demands in existing models. Specifically, we reduce the search space by identifying the instance trajectory and pruning the video frames along it. To obtain the instance trajectory, we aggregate the motion for a group of points on the segmentation masks. To this end, we propose a unified framework that jointly performs point tracking and segmentation, with the techniques to ensure robustness to occlusion in complex video scenes. TrackIME demonstrates consistent and significant impacts by bolstering existing point tracking baselines. The joint framework also reveals the synergistic effects, which also demonstrates the improvements in the video segmentation task. Overall, our work highlights the effectiveness of considering instance motion trajectory and jointly solving the tracking and segmentation, and we believe our work could inspire researchers to consider a new direction to further leverage it in the future. 1434G -k = 0 (32 points; S = 31) 822G -k = 1 (1 point) 612GTotal 2789G e.g., (256 × 256) in TAPIR [6] . While the reduced resolution enables models to process the entire video frames for tracking, the lost information during the resolution reduction induces quantization noises into the cost volume. Recent baselines, including the state-of-the-art [6] , employ refinement techniques to mitigate these noises. 6 Nevertheless, the lost detail in the visual feature after the down-sampling still hinder representing high-frequency patterns, and the model can suffer from tracking failure modes.In this regard, our method pursues the direction of pruning the excessive search space for point tracking, so that models can avoid the down-sampling and focus only on important regions maintaining detailed visual features. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide the detail of the training/evaluation setup, dataset, and hyperparameters in Section 4 and Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: All experiments are conducted with the same and commonly used random seed.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discussed the societal impact in Appendix F Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our method does not introduce risks for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have cited all papers and datasets used in the paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: In this section, we study the computational costs and efficiency of TrackIME by examining the FLOP (floating-point operations) counts for performing the point tracking.FLOP count of TrackIME. To check the exact cost of each module in TrackIME, we report the FLOPs for tracking under our default setting, e.g., TAPIR [6] as the baseline, given 64 video frames. Specifically, as given in Table 6 , the segmentation with SAM [1] needs 533 GFLOPs, tracking 32 points (e.g., S = 31 semantic neighbors plus one query point) demand 822 GFLOPs, and tracking a single point demands 612 GFLOPs, respectively. As a result, the net FLOP count of TrackIME (with K = 2 progressive steps) is 2789 GFLOPs.Computation efficiency compared to baselines. Next, we compare the baseline TAPIR [6] with various input dimensions and TrackIME, in terms of their FLOP counts versus the point tracking performances, AJ (Average Jaccard), δ x avg , and OA (Occlusion Accuracy), evaluated under DAVIS-F and DAVIS-S in Table 7 .For TAPIR, the FLOP count is mostly governed by the input dimension of a model (256 × 256), e.g., 612 GFLOPs for processing 64 video frames, and it grows quadratically as the input dimension gets increased.An interesting finding in Table 7 is that the baseline [6] cannot benefit from the larger input dimensions without fine-tuning. For example, we observe that the baseline's performance only deteriorates given larger inputs, as the model is only optimized for a low-resolution input frames (256 × 256) to meet the memory constraints while training; it is non-trivial to process high-resolution inputs without fine-tuning. Furthermore, even if fine-tuning is employed (e.g., TAPIR Hi-Res [6] ), the performance gain (e.g., 62.8 → 65.7 AJ) is not significant considering the excessive increase in FLOP counts (e.g., 612 → 8257 GFLOPs), and the occlusion accuracy (OA) can even get worse (e.g., 88.3 → 86.7).These results further demonstrate the merits of employing TrackIME for point tracking, which can enable point tracking models to process the frames in a computationally efficient manner, even without fine-tuning, and provide consistent performance gains. For example, comparing TrackIME (ours) vs. Table 7 : The comparison of the FLOP counts of the TAPIR [6] models and TrackIME. We report the FLOP counts to process 64 video frames by TAPIR with the input dimensions (256 × 256) (default), (512 × 512), and (768 × 768), TAPIR Hi-Res (a fine-tuned model for (1080 × 1080)) and TrackIME (ours). For each model, we further report the benchmark results in terms of AJ (Average Jaccard), δ x avg , and OA (Occlusion Accuracy), evaluated under DAVIS-F and DAVIS-S. For TAPIR Hi-Res, numbers are excerpted from [6] , where results for DAVIS-F are not available. TAPIR Hi-Res [6] gives: 2789G vs. 8257G (FLOPs); 69.3 vs. 65.7 (AJ); 81.4 vs. 77.6 (δ x avg ); and 89.0 vs. 86.7 (OA), in DAVIS-S, respectively.In the similar manner, we also provide the FLOPs count for our method incorporated with [7] in Table 8 .In this section, we ablate the choice of hyperparameters in our enhanced point tracking, namely the pruning sizes (H 0 , W 0 ) without the progressive fusion (i.e., K = 1) and our default setting in TrackIME (K = 2), and the number of sampling semantic neighbors S for estimating the instance trajectory.In Table 9 , we find that smaller pruning sizes tend to introduce positive effects in the fine-grained metrics (e.g., 1-and 2-pixel error thresholds), but also trade off the average-scale metrics (e.g., AJ and δ x avg ). These results are expected, as the pruning size gets smaller, the amount of down-sampling reduces and more detailed visual features would be preserved, but at the same time, the chance of erroneous pruning increases where the true location of the query point is lost.Table 9 : Ablation study of the pruning size in our framework. We ablate the pruning size considered in TrackIME. For the evaluation, we calculate both pixel-scale and average-scale metrics under the DAVIS-F dataset [11] .Pruning Size (K) Table 10 : Ablation study of the effect of the number of semantic neighbors in our method. We ablate the number of semantic neighbors considered in our method. For the evaluation, we calculate both pixel-scale and average-scale metrics under the DAVIS-F dataset [11] . We note that the progressive fusion (K = 2) in our method can mitigate the trade-off in pruning by considering multiple scales, e.g., H 0 = 960 and H 1 = 384, providing additional performance gains.EQUATIONS + 1 J 1 δ x 1 J 2 δ x 2 AJ δ xNext, in Table 10 , we ablate the effect of the choice for the number of semantic neighbors S + 1 (including the query point), halving down its value starting from (S + 1) = 128 to (S + 1) = 2. As a result, we find that all of the choices can provide satisfactory performance in general, although there exist mild trade-offs between the 1-and 2-pixel scale metrics and the average scale metrics. As one of our goal is on achieving the optimal pixel-scale performance in point tracking, we empirically choose (S + 1) = 32, which reveals the best 1-pixel scale metrics.In this section, we provide the additional experiment and visualizations with TrackIME.The use of visibilities as the confidence weights. Our strategy combines both the hard 0-1 visibility predictions as well as the confidence weights (e.g., Equation ( 3)). This strategy effectively mitigates potentially erroneous confidences by the false positives, since our method tends to demonstrate a high precision (the portion of true positives) for the visibility classification, e.g., we get 93.7% at the threshold 0.5 in DAVIS-F. Our strategy is valid as far as a sufficient number of visible tracking points are available. For the cases where an object is occluded for a few frames and then reappears, our framework can maintain the number of tracking points via the point re-sampling, even if the visibility classifier fails to predict the reappearance.To further support the validity of our strategy, we measure the average confidence of the true positives and the false positives and find 0.902 (true positives) and 0.737 (false positives), so the remaining false positives would be penalized through the weighted aggregation. We also provide additional study in Table 11 , where we force equal weights in the aggregation experimented in DAVIS-F. For example, we find 1.3 points improvement by using our strategy.TAPNet results with an alternative checkpoint. In our main experiments, we have utilized ResNet18 backbone image backbone provided by the official checkpoint to reproduce TAPNet [11] and TAPIR [6] results, instead of TSM-ResNet18 used by TAPNet in the original paper [11] . In Table 12 , we Visualization of the progressive inference. We additionally visualize the progressive inference structure in Figure 3 . Specifically, we incorporated TrackIME with TAPIR [6] and apply the progressive pruning sizes of (960 × 960) and (384 × 384). As depicted by Figure 3 , the latest progressive step is well focused around the query point, e.g., the dog's ear, so that the search space for point tracking is effectively pruned.TrackIME relies on the pre-trained models for point tracking, often trained with synthetic datasets, such as Kubric [36] and PointOdyssey [9] , while the segmentation models are primarily trained on the real images [1] . An interesting future direction is to integrate the TrackIME with training on the real video scenes. As this could include the development of point tracking algorithms capable of generalization to diverse intricate objects or, alternatively, optimizing the segmentation models for better video scene understanding. This approach could further improve the accuracy and applicability of TrackIME in various real-world scenarios.While point tracking by TrackIME can be beneficial for various video understanding applications, such as novel-view synthesis, depth estimation, and action recognition, the emergence of unexpected behavior within TrackIME can lead to misrepresentations of the real video data. For those applications that require extremely accurate models for safety-related judgements, such as depth estimation for autonomous driving, the unexpected behaviors must be carefully managed. To ensure the reliability of systems using point tracking predictions, we recommend to conduct thorough investigations and implement robust mitigation strategies to minimize potential risks, thereby increasing the overall safety and effectiveness of these applications. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.The confidence whether the trajectory is visible in each frame; i.e., the point is not out-of-frame and not occluded by different objects.Intuitively, the instance as a group of points moves together even if a fine-grained motion of individual points may differ. Hence, we track multiple points on the same instance, and then aggregate their trajectories as the instance motion, which we eventually utilize for pruning video frames.In practice, we batch-process a set of multiple points simultaneously.For example, we choose r = 0.1 for our main experiments discussed in Section 4.The open-source version of TrackIME is available at https://github.com/kami93/trackime.We refer the readers to literature for the refinement mechanisms[6,7]. 


Decision: This paper presents a method to improve the performance and computational efficiency of point trackers, by adding awareness of the objects that the points belong to, via a pre-trained segmentation model. Despite using off-the-shelf models (without finetuning them), the method achieves substantial improvements in point tracking accuracy, across a wide array of base models, and delivers state-of-the-art zero-shot video segmentation as well. The reviews were initially leaning positive, and became more confident after the rebuttal, arriving at 1x strong accept, 1x weak accept, 1x borderline accept. The reviewers appreciated the core idea, the results, and the presentation, and the AC agrees. The authors are encouraged to incorporate the work done during the rebuttal, to strengthen the paper further. The AC recommends acceptance.

Reviews: review: paper_summary: paper_summary: This paper tackles the problem of point tracking, where the task is to track the movement of a single point in a video. Point tracking has experienced a fairly recent deep learning revival starting with PIPs [22 in paper ref], which was inspired by a handcrafted method named Particle Video from Sand and Teller [1], with more recent follow-ups such as TAPIR [6 in paper ref], Omnimotion [8 in paper ref], and Cotracker [7 in paper ref].

Specifically, this paper tackles a key challenge faced by these contemporary point tracking models: the computational burden introduced by the cost volume operation, which is a correlation between the pointwise feature being tracked and the feature maps of all video frames in the current window being processed. This operation needs to be performed for each point being tracked, resulting in tensors as large as B x T x N x C x H x W (batch size, window length, number of points being tracked, num feature channels, video height, video width) being processed. In response, these models spatially downsample the feature space by up to a factor of 4 (in Cotracker's case) and 8 (in PIPs' case), which results in a decimation of high frequency information that can be useful in tracking points. As a consequence, these models experience a large reduction in tracking accuracy due to the downsampling.

The authors of this paper propose an alternative approach. Instead of downsampling the feature space, intelligently crop each frame to the most important region in the frame, i.e., somewhere around the point being tracked. The authors introduce a framework that takes advantage of a pretrained instance segmentation model (SAM [1 in paper ref] in this case) to segment the instance upon which the queried point is placed, to use the segmentation mask to query semantically neighbouring points, to track all these points using a pretrained tracker (TAPIR, PIPs, Cotracker, etc.), to aggregate all trajectories to a single instance trajectory, to crop each video frame about each instance trajectory point, and to re-do tracking of the original queried point in this pruned space. They even show how this can be performed recursively / progressively, where the entire process can be repeated to get an even more refined trajectory estimate. The authors also show how their framework can act as an accurate video object segmentation model, where the synergy between the point tracking and the segmentation model allows both to improve on their respective tasks (tracking and segmenting). They show how this can exceed the zero-shot video object segmentation performance (in the DAVIS dataset) of SAM-PT (a version of SAM that relies on a point tracking model to enable zero-shot video object segmentation) and various class-based video object segmentation models.

The authors evaluate their framework through exhaustive experimentation, with four main experiments:
* Evaluating point tracking performance for dynamic objects (i.e., testing on TAP-Vid-DAVIS)
* Testing the generalizability of their framework on different point tracking models.
* An ablation study where they ablate different components of their model (search space pruning using tracked query point vs. instance trajectory, and the recursive pruning scheme).
* Zero-shot video object segmentation.

Overall, the authors show substantial gains on standard benchmarks and demonstrate the synergistic effectiveness of using a pretrained instance segmentation model with a pretrained point tracker. They even show that there is no net computational efficiency loss in the additional FLOPS introduced by the segmentation model and the recursive process, since the tracking accuracy gains far outweigh the additional compute costs.

[1] Sand, P., Teller, S. Particle video: Long-range motion estimation using point trajectories. In CVPR 2006.
strength_weakness: strength_weakness: * Originality:
    * No weaknesses.
* Quality:
    * No weaknesses.
* Clarity:
    * No major weaknesses, but I do have some suggestions and questions relating to clarity. I have provided these in the Questions section below.
* Significance:
    * No weaknesses.
questions: questions: * L128: I suggest sharing the threshold value here instead of saying "is set much smaller than the standard 0.5". It seems like an unnecessary obfuscation.
* L185-195: I strongly suggest mentioning the ablation study here, as it is one of your major experiments.
* Table 3: I suggest mentioning in the caption that the tracking backbone is TAPIR. I understand this is mentioned in a couple of places in the main manuscript, but it helps to remind the reader when reading the table caption, as this piece of information can be accidentally skipped when skimming through the main manuscript.
* L274: "23.2" is supposed to be 28.2, right?
* L297-298: Can you provide some brief commentary on why TrackIME may be performing better than SAM-PT on zero-shot video object segmentation? I'm curious to know your thoughts. If insightful, it may even be useful to include it in the main manuscript.
* Table 7: Are the resolutions displayed here (excluding TrackIME since there's no training involved) the input resolutions during training or during testing?
* Figure 2: The points and the images are too small! I suggest increasing the size of the points, removing the last column of images, and increasing the size of the image grid to match the width of the caption. I liked the size of the points shown in Figure 1.
* Figure 3: The points are even harder to see here than in Figure 2 despite the images being larger! This one's an easy fix: increase the size of the points (preferably to match the size of the points in Fig 1 for consistency).

Great paper overall. Good job :)
limitations: limitations: The authors have adequately addressed the limitations of their framework and its potential negative societal impacts. Limitations have been discussed in Appendix F and Section 4, with honest and descriptive commentary. Potential negative societal impacts have been sufficiently discussed in Appendix F.
paper_summary: paper_summary: This paper presents a new framework for video point tracking from instance motion. By integrating existing segmentation and point tracking base models, the performance of point tracking is significantly improved from object-by-object optimization. Ablation experiments and extensions in zero-shot video object segmentation further validate the effectiveness of the proposed framework. The paper is generally well written and organized.
strength_weakness: strength_weakness: At the trajectory aggregation step (eq 3), multiple moving points are usually selected corresponding to each object, but just averaging them according to visibility is to simple. Direct averaging does not satisfy common cases such as rotations and scale changes, and even if this is used for subsequent inference, a better initial value would be valuable. A straightforward modification would be to fit the affine motion model with these selected points. 

For eq 3, 8 and 12, whether the visibilities can be regarded as confidence weights needs to be further analysed, since the supervision only have a 0-1 occlusion maps. Does this address the cases where an object is occluded for a few frames and then reappears?
questions: questions: All five methods compared in Table 1 are inconsistent with the results reported in their original papers, most notably TAPNet's AJ which was 46.6 in the original paper but 56.5 here. The reasons for these inconsistencies need to be explained. This raises doubts as to whether the significant performance gains (from 62.8 to 69.3) of the proposed method can be compared to recent competitors (e.g. 65.9 of DOT). 

Dense Optical Tracking: Connecting the Dots, CVPR 2024. 

L431 mentions that changing the backend of the model improves performance, which is confusing. As I understand this process donot retrain the model. Did the authors confirm that the implementation and evaluation is correct, or are there some bugs that were addressed in this process?
limitations: limitations: yes
paper_summary: paper_summary: In this paper,  the authors propose TrackIME, a framework to tracking points in video. The proposed TrackIME leverages a segmentation model to improve both its efficient and effectiveness. TrackIME achieves SOTA performance on TAP-VID dataset.
strength_weakness: strength_weakness: 1. The main contribution of proposed method is not clear to me. Using segmentation model to help track points within an instance is an intuitive top-down solution for points tracking
2. The proposed method outperform SAM-PT in a relative small margin in terms of instance-level tracking, which raise my concerns about the capability of model for tracking points. It seems to me that the improvement on the point tracking performance is limited to help tracking instances/objects.
questions: questions: 1. The authors only compare FLOPs between TrackIME and TAPIR, why not also compare with other methods like CoTracker?
limitations: limitations: N/A

Comments: title: Thank you for your response!
Thank you very much for the response.
Thank you very much for the response.
Thank you very much for the response
Mostly pleased but just one more concern....
comment: Dear reviewer BSTo,

We appreciate your efforts in reviewing our paper and actively participating in the discussion.

Due to your valuable and constructive suggestions, we do believe that our paper is much improved.

Thank you very much,

Authors
Dear reviewer 7a51,

We appreciate your efforts in reviewing our rebuttal and consulting our discussions with other reviewers. We are glad to hear that our rebuttal addressed your concerns well!

We will address your additional question in the response below.

---

**Why can TrackIME consider better query points for segmentation (i.e., better point tracking results), removing tracking failures on intricate object parts?**  (Note: The query points for video segmentation in TrackIME and SAM-PT are essentially based on the point tracking results. Therefore, we will focus on discussing how TrackIME addresses the limitations in accurately tracking the query points for segmentation).

In addition to allowing higher resolutions for tracking, another core advantage of TrackIME is that it can provide a more reliable point tracking result for the segmentation query point by aggregating the multiple point trajectories on the same instance (as predicted by the segmentation model), so that the effect of a potential tracking failure of a single query point can be avoided, which SAM-PT majorly suffers from.

Specifically, our enhanced point tracking is accomplished by the following three main components (which are the subject of our ablation study in Table 3) as follows:
- Search space pruning (to allow for higher resolutions without significant computational overhead)
- Trajectory aggregation (to aggregate multiple point trajectories on the same instance)
- Progressive inference (to boost the tracking performance by reinitializing pruning windows at different scales)

For example, our ablation study in Table 3 shows that all these modules play an important role in avoiding tracking failures on intricate object parts. First, as pointed out by the reviewer, the search space pruning allows TrackIME to operate at higher resolutions and it improves the tracking performance on the intricate 1-pixel scale (e.g., 23.0 $\to$ 28.2 $\text{J}_1$), as well as on the average scale (e.g., 57.5 $\to$ 62.5 $\text{AJ}$). Furthermore, we emphasize that another significant gain is achieved by employing the other two components$-$the trajectory aggregation and the progressive inference (e.g., 28.2 $\to$ 35.4 $\text{J}_1$ and 62.5 $\to$ 65.3 $\text{AJ}$; we note that the gain in the intricate $\text{J}_1$ is even larger here).


In this respect, the limitation of SAM-PT is not only its lower resolution for point tracking, but also its lack of structure to incorporate multiple points on the same instance and their trajectories for better point tracking. We note that SAM-PT also considers multiple points internally, but they are merely used to predict segmentation masks under occlusion scenarios, not to improve the point tracking.

---

If you have any further questions or suggestions, please do not hesitate to let us know.

Thank you very much,

Authors
Thanks to the authors' detailed replies, I have no additional questions. I will carefully consider these issues and make a final rating.
I have carefully read the rebuttal and the reviews from other reviewer, and would thanks to the authors for their efforts on the rebuttal. My concerns about contributions and FLOPs comparison are resolved, and I will update my rating accordingly.

Some questions still remain for instance object segmentation:

* The authors claim that 'TrackIME removes erroneous query points for segmentation caused by tracking failures on intricate object parts.' Could the authors clarify how TrackIME accomplishes this? Additionally, why would SAM-PT fail in this context—is it solely because SAM-PT conducts tracking at a lower resolution, while TrackIME operates at higher resolutions through pruning?"
Dear reviewer BSTo,

We are happy to hear that our rebuttal addressed your questions and concerns well, and we also appreciate your efforts to additionally consult the discussion between the reviewer jQxW and the authors.

We will address your additional concerns in the response below, which we will also include in the updated manuscript.

---

**Why does replacing the aggregation with the affine motion model not improve the point tracking results?**

We first would like to clarify that the affine motion model does not necessarily provide better initial motion (i.e., the instance trajectory for the pruning windows). For example, measuring the normalized $L^2$ distance between the estimated window centers with respect to the ground truth query trajectory yields 0.4227 (our aggregation) vs. 0.4294 (the affine motion model). In consequence, the final point tracking performance can be worse when the affine motion model is employed to predict the pruning windows.

---

**Then, why does the affine motion model not provide better initial motion?**

This is because the affine motion cannot be optimized with respect to the optimal trajectory of the pruning windows, but only with respect to the individual point trajectories within the instance predicted by the point tracker (e.g., TAPIR). If the affine motion model were to be fitted with the ground truth (GT) window centers (i.e., the ground truth query trajectory), the final tracking performance could be superior to that of our aggregation, as demonstrated in the table below. However, this is not a feasible objective, as the ground truth is not known a priori.

Nevertheless, we believe that both our aggregation and the affine model provide comparable outputs; they are simply different methods of estimating the instance trajectory given the multiple points on the same instance. In fact, the differences in their results are very marginal when compared to our ablation study in Table 3, which replaces the aggregation with the single-point prediction by a query point. The following table presents the result for convenience.

|     Method     |       $J_1$ |  $\text{AJ}$  | $\delta^x_1$ | $\delta^x_{\text{avg}}$ | $\text{OA}$  |
|------------|:--------:|:------:|:------:|:------:|:------:|
| TrackIME (single point) |  28.3 | 62.6 | 41.2 | 75.6 | 84.9 |
| TrackIME (affine motion) |  35.3 | 65.1 | 48.0 | 78.4 | 85.7 |
| TrackIME (default) |  35.4 | 65.3 | 48.2 | 78.6 | 86.5 |
| TrackIME (affine motion w/ GT) |  **36.2** | **67.2** | **48.6** | **79.4** | **88.5** |

---

**Additional remarks on instance trajectory estimation**

We strongly agree that considering and testing different methods for the instance trajectory estimation is important for both the performance and the safety in the application, even though our method has provided reasonable results in our domain. We thank the reviewer for encouraging us to further acknowledge this point, which we are happy to include in the updated manuscript.

---

**Commentary on the reproduced results**

To avoid a potential problem with direct numerical comparisons with other methods in the future, we will ensure that the test code is released as well. We believe it can be a useful addition to the community, providing more options for implementing and testing point tracking models.

---

If you have any further questions or suggestions, please do not hesitate to let us know.

Thank you very much,

Authors
Thanks to the authors for their detailed response. I am very glad to see that the authors have provided additional experiments with a more reasonable motion model, which is essential to improve this paper. However, this raises new doubts about the experimental results; why did the theoretically better initial motion rather lead to worse final estimates? This may require a justifiable explanation before updating it to the paper.

I have read the discussion between the authors and reviewer jQxW about the results. From my side, it is enough that the authors confirm that the reproduce settings are consistent and the comparison is fair. However, these differences lead to a potential problem in direct numerical comparisons with non-listed other methods. So clearly stating these in the paper is necessary, and the authors promise to do so. If the authors could open source the corresponding test code, it could help the community to build a comprehensive benchmark.
Dear reviewer jQxW,

We are happy to hear that our rebuttal addressed your question well.

We will also address your additional concerns and questions in the response below, which we will also include in the updated manuscript.

---

**The reasons for the differences in our baseline results compared to the original reports**

We are grateful for your genuine side notes, which we find very helpful in elucidating the cause of the inconsistencies in our baseline results, such as the influence of the image resize filtering and the checkpoint version on the results. As detailed in L200-202 of the manuscript, we applied the same resize filtering function, the standard bicubic filter, to every model using the default input resolution of each model. As the reviewer pointed out, our choice of resizing filter might differ from those used in the original codes and could have caused small inconsistencies in the results.

Finally, we would like to gently point out that in our response to reviewer BSTo, we mention the effect of library/hardware-dependent numerical characteristics in the discrepancy in the results for all baselines. It is also worth noting that reproducing exactly the same results for OmniMotion is hindered by the stochastic nature of the optimization process, and by the fact that the default hyperparameters and the choice of activation functions in their official repository are different from those considered in their published paper (according to their Github bulletin "Issues"). Nevertheless, we believe that the effectiveness of TrackIME is intact, orthogonal to this issue.

---

**Comparison to the original reports**

Although the degree of the inconsistency is relatively small given the significant gain in tracking accuracy by incorporating TrackIME into the baselines, we strongly agree that clarifying these differences and explaining their causes is critical to the credibility of our work. In the updated manuscript, we will also include the reported results from the original papers, and provide detailed explanations for all factors that have caused the inconsistencies. For example, the average tracking accuracies (i.e., $\text{AJ}$ and $\delta^{x}_{\text{avg}}$) in DAVIS-F for the baselines that provide the official checkpoints (i.e., TAPNet, PIPS2, CoTracker, and TAPIR), could be presented as follows. We note that the checkpoint for TAPNet with TSM-ResNet18 (the one we mention in our response to the reviewer BSTo) is used.

\begin{array}{l  c  c}
\text{Method} & \text{AJ} & \delta^x_\text{avg}  \newline
\hline
\text{TAPNet (original paper)} & 33.0 & 48.6 \newline
\hdashline
\text{TAPNet (our reproduction)} & 32.8 & 48.4 \newline
+\textbf{TrackIME} & \mathbf{47.0} & \mathbf{60.6} \newline
\hline
\text{PIPS2 (original paper)} & - & 63.5 \newline
\text{PIPS2 (official repository)} & - & 70.6 \newline
\hdashline
\text{PIPS2 (our reproduction)} & 46.6 & 69.4 \newline
+\textbf{TrackIME} & \mathbf{50.3} & \mathbf{74.0} \newline
\hline
\text{CoTracker (original paper)} & 62.2 & 75.7 \newline
\hdashline
\text{CoTracker (our reproduction)} & 60.8 & 76.1 \newline
+\textbf{TrackIME} & \mathbf{64.5} & \mathbf{79.2} \newline
\hline
\text{TAPIR (original paper)} & 56.2 & 70.0 \newline
\hdashline
\text{TAPIR (our reproduction)} & 57.5 & 70.5 \newline
+\textbf{TrackIME} & \mathbf{65.3} & \mathbf{78.6} \newline
\hline
\end{array}

---

If you have any further questions or suggestions, please do not hesitate to let us know.

Thank you very much,

Authors
**[Q1] Commentary on why TrackIME may be performing better than SAM-PT on zero-shot video object segmentation**
>  their performance is still bounded by low-resolution tracking and suffers from tracking failures.

Ah! Of course, that makes sense. I would suggest making a brief mention about this fact in the main manuscript, or at least in the supplemental, to ease any qualms one might have.

---

I've read your responses to my questions and I'm happy to see that you'll be including my suggestions in your revised copy. 

---

On a side note, Reviewer BSTo raised a good point that I forgot to bring up with regards to the inconsistencies in reported results from the baselines and I wanted to give a comment about that. First, I'm glad to see that you've addressed the concerns regarding TAPNet, but you should also address the inconsistencies with the other four approaches. Personally, I'm quite familiar with the literature (and code), and so I understand that inconsistencies can arise due to small things like the filtering algorithm used when resizing DAVIS images, the version of PointOdyssey that was trained on (in the case of PIPs++), etc. For example, I know that PIPs++' reported result in the PointOdyssey paper is much lower than the reported result in its github repo, which is close to what you reported in Table 1. I also know that your CoTracker result is very close to what was reported in the CoTracker paper. Despite this, you absolutely should explain these differences and why they came about, lest you lose the reader's trust. 

On that note: why are there differences in your baseline results compared to the original reported results in their respective papers?