Context-PIPs: Persistent Independent Particles Demands Spatial Context Features
Authors: Weikang BIAN
Zhaoyang Huang
Xiaoyu Shi
Yitong Dong
Yijin Li
Hongsheng Li
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Point Tracking; Optical Flow; Video Correspondence; Computer Vision;
TL;DR: Tracking any point in videos demands spatial context features

Abstract: We tackle the problem of Persistent Independent Particles (PIPs), also called Tracking Any Point (TAP), in videos, which specifically aims at estimating persistent long-term trajectories of query points in videos. Previous methods attempted to estimate these trajectories independently to incorporate longer image sequences, therefore, ignoring the potential benefits of incorporating spatial context features. 
We argue that independent video point tracking also demands spatial context features. To this end, we propose a novel framework Context-PIPs, which effectively improves point trajectory accuracy by aggregating spatial context features in videos. Context-PIPs contains two main modules: 1) a SOurse Feature Enhancement (SOFE) module, and 2) a TArget Feature Aggregation (TAFA) module. Context-PIPs significantly improves PIPs all-sided, reducing 11.4\% Average Trajectory Error of Occluded Points (ATE-Occ) on CroHD and increasing 11.8\% Average Percentage of Correct Keypoint (A-PCK) on TAP-Vid-Kinetics. Demos are available at \url{https://wkbian.github.io/Projects/Context-PIPs/}.

Context-PIPs: Persistent Independent Particles Demands Spatial Context Features
Weikang 1.a Zhaoyang Huang'* Xiaoyu Shi!
1.2+
Yitong Dong Yijin Li Hongsheng Li
CCHK MMLab CCntre for Perceptual and Interactive Intelligence *Zhejiang University
skbianRoutlook. com, drinkingcoderdllink cuhk eddu. .h,, hslißee.. cuhk. edu.
Abstract
We tackle the problem of Persistent Independent Particles (PIPs), also called Track- ing Any Point (TAP), in videos, which specifically aims at estimating persistent long-term trajectories of query points videos. Previous methods attempted estimate these trajectories independently to incorporate longer image sequences, therefore, ignoring the potential benefits incorporating spatial context features. We argue that independent video point tracking also demands spatial context features. To this end, we propose a novel framework Context-PII -PPPs, which effee- tively improves point trajectory accuracy by aggregating spatial context features in videos. Context-PIPs contains two main modules: 1) a SOurse Feature En- hancement (SOFE) module, and 2) TArget Feature Aggregation (TAFA) module. Context-PIPs significantly improves PIPs all-sided, reducing 11.4% Average Tra- jectory Error of Occluded Points (ATE-Oce) CroHD and increasing 11.8% Average Percentage of Correct Keypoint (A-PCK) on TAP- are available at htttss://wkbiiiie http:///wwbaaa.ggtuu.boooooo.o h
Demos
Introduction
Video particles a of sparse point trajectories in video that originate from the first frame (the source image) and move across the following frames, which are regarded as the target images. In contrast optical flow estimation that computes pixel-wise correspondences between a pair of adjacent video frames, Persistent Independent Particles (PIPs) [12 or Tracking Any Point (TAP) [S) interested in tracking the points in the follow-up frames that correspond the original query points even when they are occluded in some frames. Video particles provide long-term motion information for videos and can support various downstream tasks, such as video editing [6 and Structure-from from--otion 48
Long-range temporal information is essential for video particles especially when the particles occluded because the positions the occluded particles can be inferred from the previous and subsequent frames where they are visible. However, simultaneously encoding long image sequences brings larger computational and memory costs. Previous methods [12] 5 learn to track individual points independently because dense video particles are unnecessary in by optical flow estimation from visual similarities, they learn to predict point trajectories from the similarities between the query point and the subsequent target images. Specifically, given query point at the source image, PIPs encodes I feature maps from T' consecutive target images and builds a H x W correlation volume by computing the feature similarity between the feature of the query point and the feature maps. The 7 particle positions are iteratively refined with the 3D correlation volume through a shared MLP-Mixer 440 In other words, PIPs trades the spatial context features of
scenarios. Inspired
"Weikang Bian and Zhaoyang Huang assert equal contributions.
37th Conference on Neural Information Processing Systems (NcurIPS 2023), Vancouver, Canada.

---

the particle for longer temporal feature encoding. PIPs achieves great performance on the DAVIS dataset, which contains large movement particles and weak texture images (e.g., fast-moving dogs and black bears).
We argue that independent point tracking still demands spatial context features. Intuitively, although PIPs only accounts for specified query points, spatial context features around them provide informative cues for point trajectory refinement. For example, video particles on the same objects always share similar motions over time. In some video frames where the target particles are occluded, their surrounding particles may be visible and provide guidance for the position estimation of the target particles. However, PIPs only takes correlations and features belonging to the particles while ignoring abundant spatial context features around them. In this work, we propose tracking particles with Context (Context-PII) PIPs) to improve independent point tracking with spatial context features. Context- PIPs contains two key modules for better point trajectory refinement: 1) a source feature enhancement (SOFE) module that learns to adopt more spatial context features in the source image and builds a guidance correlation volume, and 2) target feature aggregation (TAFA) module that aggregates spatial context features in the target image guided by the correlation information.
In the source image, points that possess similar appearances are supposed move in similar trajec- tories in subsequent frames. Such an assumption has also been used in GMA [2] for optical flow estimation. Given query point, SOFE computes the correlation between the query point and the source image feature map, which is its self- similaiti map. With the guidance of the correlation (self- similarity), SOFE predicts M offsets centered from the query point, and samples at the corresponding M auxiliary points to collect source context features. During the iterative point trajectory refinement, the correlation information between the M auxiliary features and target feature maps is injected into the MLP-Mixer, which provides strong guidance and shows evident performance improvement.
Existing methods for optical flow and video particle estimation ignore features in target images for iterative refinement. To better utilize the context of target images, in each iteration, our TAFA module collects target features surrounding the previous iteration's 's resulting movements. TAFA for the first time shows that context features in target images also benefit correspondence estimation and further improve the point tracking accuracy.
Our contributions can be summarized as threefold: 1) We propose a novel framework, Context-PIPs, to improve independent video partiele tracking with context features from both source and target features. Context- ranks 1st on the four benchmarks and shows clear performance superiority. 2) We design novel SOurce Feature Enhancement (SOFE) module that builds guidance correlation volume with spatial context features in the source image, and 3) a novel TArget Feature Aggregation (TAFA) module that extracts context features from target images.
2 Related Work
Optical Flow. Optical flow estimates the dense displacement field between image pairs and has traditionally been modeled as an optimization problem that maximizes the visual similarity between image pairs with regularizations [13] I 2 341. It serves as the core module of many downstream
applications, such as Simultaneously Localization And Mapping (SLAM) R 47
video
synthesis [66] 14 15
41 video restoration [24. 23] etc. Since FlowNet 6 20 learning
optical flow with neural networks presents superior performance over traditional optimization- based methods and is fast progressing with more training data obtained by the renderer and better 45). In recent years, iterative refining flow with all-pairs correlation volume presents the best performance. The most successful network designs are
network architecture 6 20
28
35
36
18.
19
RAFT 339 and FlowFormer I7 32], which achieves statc-of-the the--rtt accuracy.
Typical optical flow estimation only takes image pairs but longer image sequences can provide more information that benefits optical flow estimation. Kalman filter 4 7 had been adopted in dealing with the temporal dynamics of motion and estimating multi-frame optical flow. Recent learning- based methods also attempted to exploit multi-fram information and perform multi-frame optical flow estimation. PWC-Fusion 22 is the first method that learns to estimate optical flow from multiple images. However, it only fuses information from previous frames U-Net and improves little performance. The "warm start" technique (39
37 that wrapped the previous flow to initialize the next flow is firstly proposed in RAFT and shows clear accuracy increasing. VideoFlow [ takes multi- frame cues better, iteratively fusing multi-fram information in a
33
and five-
2

---

frame structure. which reveals that longer temporal information benefits pixel tracking. Recently, researchers 25 [0] also tried to learn to estimate optical flow with event cameras.
Tracking Any Point. Optical flow methods merely focus on tracking points hetween image pairs but ignore tracking points across multiple consecutive frames, which is still challenging. Harley er al. [12] studied pixel tracking in the video as a long-range motion estimation problem inspired by Particle Video (30.. They propose a new dataset FlyingThings+ based on FlyingThings [26] for training and Persistent Independent Particles (PIPs) to learn to track single points in consecutive frames with fixed lengths. Doersch er al. is a parallel work, which formalized the problem as tracking any (TAP. They also propose new dataset Kubric [5] for training and a network TAP-Net to learn point tracking. Moreover. they provide the real video benchmarks that are labeled by humans, TAP- Vid-DAVIS [5] and TAP-Vid V-d--iiKKieesss [5]. for evaluation. PIPs and TAP solve the video particle tracking problem in a similar manner, i.e., recurrently refining multi-frame point trajectory via correlation maps. In this paper, our Context-PIPs follows the training paradigm of PIPs and improves the network architecture design of PIPs. We also take the Vid-Did and TAP-Vid-KIi Vid-Kinetics benchmarks from TAP-Net for evaluation.
3 Method
In contrast to optical flow methods [39 17 that track dense pixel movement between an image pair, the problem of point tracking takes I consecutive RGB images with single query point Xare E I the first frame imput, and estimates T' coordinates X {xo, X XT-1 at the video frames where every x; indicates the point's corresponding location at time t. Persistent Independent Particles (PIPs) [2] is the sfte-of-the-ar art network architecture for TAP. iteratively refines the point trajectory by encoding correlation information that measures visual similarities between the query point and the T video frames. The query points to be tracked are easily lost when the network only looks at them and ignores spatial context features. We propose a novel framework Context-PIPs (Fig. D that improves PIPs with a SOurce Feature Enhancement (SOFF) module and TArget Feature Aggregation (TAFA) module. In this section, we first briefly review PIPs and then elaborate on our Context-PIPs.
3.1 A Brief Revisit of PIPs
PIPs 12 processes T video frames containing N independent query points simultaneously and then extends the point trajectories more video frames via chaining rules [1]]. Given source frame with a query point Xare E R² and T follow-up target video frames, PIPs first extracts their feature maps Io, .,..., - E C through shallow convolutional neural network and bilinearly samples to obtain the source point feature fare = [x(xer)) from the first feature map at the query point X C, H, W are the feature map channels, height, and width. Inspired by RAFT [39], PIPs initializes the point trajectory and point visual features at each frame with the same Xare and fare:
x [x]] x
1/ = X t = 0t...,t=: T T 1}, - 1 farc; t = t 1},
(1)
F (CG f..
and iteratively refines them via correlation information. x and f/ respectively denote the point trajectory and point features in the t-th frame and k-th iteration. Intuitively, the point features store the visual feature at the currently estimated query point location in all the T frames.
Specifically, in each iteration k, PIPs constructs multi-scale correlation maps 39 between the guidance feature {f/ 17-1 and the target feature maps {1} ) which constitutes T correlation maps c* = -[c ct ..., - J of size x H x W, and crops correlation information inside the windows centered at the point trajectory: C* (X*) = {c& (x5), cf((x) c (x/-))) where of (x()) E D> denotes that we crop D x D correlations from ct inside the window centered at x The point features Fkc point locations X and the local correlation information C* C*(X*) are fed
into a standard 12-layer MLP-Mixer that produces AF and AX update the point feature and the
point trajectory:
AF, AX MLP C* (***),, Eno(x* Xere)),
(2)
F1+1
F AF, x xk+1_ X + AX.
3

---

Enhancement (SOFE)
inbuker
Oor mastion
Ofleets Encoder
fire
Context-PIP Overview
TArget Feature Aggregation (TAFA)
Iterattonss
Encg [x))
x"
SOFE
pt
TAFA
ph-wpen
1e Ap
production
concatenation
cropping
map
Figure I: Overview Context-PIPs Pipeline. Our Context-PIPs improves PIPs 12 with SOurce Feature Enhancement (SOFE) and TArget Feature Aggregation (TAFA). PIPs iteratively refines the point trajectory X with an MLP-Mixer with the current point trajectory X the correlation features C and the point features F SOFE and TAFA respectively improves the correlation features and point features, denoted as C* and F
PIPs iterates K times for updates and the point trajectory in the last iteration X is the output.
PIPs achieves state-of-the-art accuracy point tracking by utilizing longer temporal features. However, the previous method ignores informative spatial context features which are beneficial to achieve more accurate point tracking. Context-PIPs keeps all modules in PIPs and specifically designed enhance the correlation information C and point features F as Č and F* with the proposed SOFE and TAFA.
3.2 Source Feature Enhancement
Given the query point Xare and feature map Io of the source image, PIPs simply samples source feature fere at the query point location to obtain the point visual features F Although the point features are updated via the iterative refinement, its perceptive field is limited the single point, casily compromised in harsh scenarios. The correlation maps C* in the k-th iteration provide vague information when the query point is less textured area. Moreover, the correlation map of at timestamp k is ineffective once the particle is occluded at the t-th frame. To enhance the source feature, shown Fig. 1 we propose SOurce Feature Enhancement (SOFE) that accepts spatial context features in the source image as auxiliary features to guide the point trajectory refinement. The MLP-Mixer can infer the point locations via the auxiliary features even when the points are occluded or on less textured regions, which improves the point tracking accuracy and robustness.
Directly adopting all features in the source image brings large computational costs. SOFE learns to sample a small number of auxiliary features to enhance the source feature. Specifi- cally, SOFE improves the point features in three steps. Firstly, SOFE learns predict M off- sets ōxo, 8x1, 5XM-1 E J² with an MLP-based sampler to sample M auxiliary features G (go- g1. ... -M-1 1g gm [x(xxre óx,))} around the query point X in the source im- age. Motivated by GMA that aggregates pixel flows from pixels that are likely to belong to the same object through selfaaimilar,ty our proposed sampler also learns the locations of the auxiliary features based on local self self-similarities cf (xre)) which store the correlations cropped from the first frame at query point location. Secondly, we construct the correlation map c/,, =< gm, I, >E RR Hx xx
that measure visual similarities of the m-th auxiliary feature and the (-th frame feature map. é,,, provides additional correlation information to guide the iterative point trajectory refinement. each iteration k crop the additional correlation information c/y, (xf") according to the point locations

---

X and concatenate them with the original point correlation information ci (xf", where c'n (xx) denotes the same cropping operation as c, (xf). Finally, for each frame I, we reduce the augmented correlations correlation feature vector of length 196 through correlation encoder CorrEnc.
ct = CorrEin(Concct((c
cx (x))....... (x))).....__((())...........)>>) c (x,
and inject Č = {ch .f.....,c-11. into the MLP-Mixer. Compared with PIPs that only adopts cf (x()), SOFE provides more informative correlations to the MLP-Mixer with spatial context features but does not increase its parameters and computations. The additional auxiliary features from the self-similarity map of the source image enhance the original source point features and significantly improves tracking accuracy.
3.3 Target Feature Aggregation
Inspired by existing optical flow methods, PIPs iteratively refines the point trajectory with correlation information and context features and also iteratively updates the point visual feature F4+1 = F* FF after initializing them with the source feature, which presents benefits for point trajectory refinement. We observe that the input for supporting the point feature updating comes from the correlations C only. However, such correlations C+ are calculated only cosine distances between the source point visual feature F and the target features around currently estimated point locations X which provide limited information on how conduct visual feature updating. Can we better guide the point feature update with context features in target images?
We, therefore, propose TArget Feature Aggregation (TAFA) to augment point features with target image features nearby the point trajectory. Specifically, for each target frame t, patch of shape D x D cropped from the corresponding target feature map centered at x to generate key and value. The augmented correlation features Č in Eq. encode abundant visual similarities. Therefore, we generate query from it to extract target context features and adopt cross-attention with relative positional encoding obtain the target context feature f%/ which is added to the original source point feature ft f. frA. Finally, such augmented point features F {f. ff............... ........ _ } are injected into the MLP-Mixer. Similar to our proposed SOFE, TAFA also keeps the same parameters and computations of MIP-Mixer as PIPs while providing additional target context features and further improving PIPs. Although context features in the source image are used since RAFT (39,, no previous methods adopt context features from target images. TAFA for the first time validates that target images also contain critical context features that benefit point movement refinement. SOFE improves PIPs with auxiliary features in the source image while TAFA absorbs more target image features. Equipping SOFE and TAFA to PIPs constitutes our final model, Context-PIPs.
3.4 Loss Functions
We compute the LI distance between X computed in iteration & and the ground truth Xgt and constrain with exponentially increasing weights y 0.8.
STAP Σ
X Xgell1
(4)
In addition, will predict the visibility/occlusion V by linear layer according to the F* obtained by the iterative update. And the cross entropy loss used to supervise V with the ground truth Vgt-
Lvis Vgr log V + (1 V ve log(1 V).
(5)
The final loss the weighted sum of the two losses:
Ciotal w WiLTAP WzCvis.
(6)
We wi 1 and w2 10 during training.
5

---

Table 1: Experiments on FlyingThings++, CroHD, LAA--Vi--aviss and TAP-Vid-Kinciin (first), Vis.
and Occ. denotes ATE-Vis, and ATE-Occ.
Flying Things+++
CroHI
DAVIS (first)
Kinetics (first)
Method
Vis.
Occ.
Vis.
Occ.
AJ
A-PCK
AJ
A-PCK
DINO RAFT
43.05
76.25
22.50
26.06
16.75
43.21
7.99
13.16
27.1
42.1
35.1
49.7
TAP-Net
17.00
20.86
33.0
48.6
38.5
54.4
PIPs (Paper)
15.54
36.67
5.16
7.56
PIPs (Re-imp.)
7.40
24.40
4.73
7.97
39.2
55.1
33.4
51.0
Context-PIPP PPPP (Ours)
6.44
22.22
4.28
7.06
42.7
60.3
40.2
57.0
4
Experiments
We evaluate our Context-PIPs on four benchmarks: FlyingThings++ 12 CroHD 38 TAP-Vid- DAVIS, and TAP-Vid- Sid-Kinnticss [5]. Following PIPs [12., we train Context-PIPs on Flyingthings+* only and evaluate it on other benchmarks without finetuning. Context-PIPs achieves state-of- the-art performance on all benchmarks and significantly improves PIPs. Moreover, we show that by utilizing spatial context features in Context-PIPs, we achieve on-par performance with PIPs when using only 40.2% of its parameters.
Datasets Flyingthings++ is synthetic dataset based on Flyingthings3D [26], which contains 8- 8-frame trajectories with occlusion. The video resolution is 384 512 for both training and evaluation. Crowd of Heads Dataset (CroHD) is high-resolution crowd head tracking dataset. Following PIPs, the RAFT 139, PIPs, and our Context-PIPs are evaluated at 768 x 1280 resolution. DINO (3] and TAP-Net [5] respectively evaluated 512 x 768 and 256 x 256 resolution. TAP- HAP-Vid-DAVIS and TAP- are two evaluation datasets in the TAP-V Vid benchmark, both of which consist of real-world videos with accurate human annotations for point tracking. Note that TAP-Vi- -id provides two distinct query sampling strategies, i.e., first and strided. The "first" sampling contains only the initial visible query points until the last frame, while the "strided" sampling sample all visible query points in every 5 frames. RAFT, TAP-Net, PIPs, and our Context-PIPs are evaluated with two different sampling strategies respectively 256 x 256 resolution. While Kubric-VER
44
and COTR [22] are only evaluated with the "first" sampling strategies at the same resolution.
Experiment Setup We use the average trajectory error (ATE) metric 112 for evaluation on FlyingTh- ings++ and CroHD. ATE measures the average L2 distances between the coordinates all predicted points in the trajectories and the corresponding ground truth coordinates. According to the ground truth visibility of the points, we calculate ATES for all visible points and occluded points separately. i.., ATE-Vis. and ATE-Occ. We use the average Jaccard (AJ) 12]. the average percentage of correct keypoint (A-PCK) metrics for TAP-Vid-DAVI and TAP- LAP-Vid-KiR Vid-KKnetics. The Jaccard metric measures the ratio of "true positive" visible points within given threshold. Average Jaccard (AJ) averages Jaccard across the different thresholds. PCK measures the percentage the predicted coordinates whose L2 distances from the ground truth are smaller than given threshold. A-PCK refers to calculating PCK according to multiple different thresholds and then taking the average. We set the thresholds 1, 2, 8, and 16.
Implementation Details We train our Context-PIPs with a batch size of and 100,000 steps on Flyingthings++ with horizontal and vertical flipping. We use the one-cycle learning rate scheduler. The highest learning rate set as 5 X 10-4 During training, we set the convolution stride to 8 and the resolution of the input RGB images to 384 x 512, and randomly sample N = 128 visible query points for supervision. To limit the length of the input videos, we set T = and apply the trajectory linking mechanism [12 at test time, similarly
PIPs. To align to the PIPs paper, the PIPs compared in Tab. and Tab, is trained with K = 6. In the ablation study, all models are trained with K = while tested with K 6 for fair comparison.
4.1 Quantitative Comparison
FlyingThings++ and CroHD As shown Tab. Context-PIPs ranks 1st on all metrics and presents significant performance superiority compared with previous methods. Specifically, Context-PIPs

---

Table 2: Experiments on TAP-Vid-DAVISSstridek) and TAP-Vid-Kineties(ssi
TAP-DAVIS(Strided)
TAP-Kineticx(stideed)
Method
AJ
A-PCK
AJ
A-PCK
RAFT
30.0
46.3
34.5
52.5
Kubric- VFS-Like
33.1
48.5
40.5
59,0
COTR
35.4
51.3
19.0
38.8
TAP-Net
38.4
53.1
46.6
60.9
PIPs(Re-imp.)
45.2
59.8
42.9
58.3
Context-PIPs (Ours)
48.9
64.0
49.8
64.3
Table 3: Occlusion Accuracy on TAP-Vid-DAVIS and TAP-Vid-Kinetics
First
Strided
Method
TAP-DAVIS
TAP-Kineties
TAP-DAVIS
TAP-Kinetics
TAP-Net
78.8
80.6
82.3
85.0
PIPs (Re-imp.)
79.3
77.0
82.9
81.5
Context-PIPs (Ours)
79.5
79.8
83.4
83.3
achieves 7.06 ATE-Occ and 4.28 ATE-Vis Vis on the CroHD dataset, 11.4% and 9.5% enror reductions from PIPs, the runner-up. On the Things++ dataset, our Context-PIPs decreases ATE-Vis and ATE-Occ by 0.96 and 2.18, respectively.
(first) A-PCK, the average percentage of correct key points, is the core metric. Context-PIPs ranks 1st in terms of A-PCK on both benchmarks. Specifically, Context-PIPs outperforms TAP-Net by 24.1% on the TAP-Vid-DAVIS benchmark and improves benchmarks. Context-PIPs also achieves state-of-the-art
PAP-Vid-DAVIS and TAP-Vid-Kineed
PIPs by 11.8% on the occupancy accuracy on both datasets.
TAP-Vid-DAVIS and TAP-Vid-Kinetics (strided) Tab. compares methods in the "strided" sam- pling setting. Our Context-PIPs also achieves the best performance both AJ and A-PCK metrics for the two datasets. Context-PIPs effectively improves PIPs' occupancy accuracy and presents the best performance the TAP-Vid-Dare Vid-Davis dataset.
4.2 Qualitative Comparison
We visualize the trajectories estimated by TAP-Net, PIPs, and our Context-PIPs respectively in Fig 2 qualitatively demonstrate the superior performance of method. By incorporating additional spatial context features for point tracking, Context-PIPs surpasses the compared methods in accuracy and robustness. Specifically, the first row shows the case of large-scale variation. The trajectory predicted by TAP-Net deviates considerably from the ground truth. TAP-Net also outputs jittery predictions when the query pixel is on the texture-less area as shown in the second row. Our Context- PIPs generates more accurate results than PIPs in these two hard cases. Furthermore, as depicted in the third row, PIPs struggles distinguish the front wheel and the rear wheel due the changing lighting conditions. However, our Context-PIPs achieves consistent tracking, thanks to the rich context information brought by the SOFE and TAFA modules.
4.3 Efficiency Analysis
We train our Context-PIPs and PIPs with different MLP-Mixer depths, i.c., the number of layers in the MLP-Mixer, to show the prominent efficiency and effectiveness of our proposed Context-PIPs. Context-PIPs improves PIPs with SOFE and TAFA, which introduce minor additional parameters and time costs. We show that the accuracy benefits do not come from the increased parameters trivially. As displayed in Tab. 4 we increase the MLP-Mixer depth to 16, which significantly increases the parameters but does not bring performance gain. We also decrease the MLP-Mixer depth in our Context-Pri PIPs. Even with only 3-layer MI.P-Mixer, Context-PIPs, achieves better performance than the best PIPs (MI.P (Mx.PMixxe depth=12). Context-PIP, outperforms PIPs with only 40.2% parameters.
Moreover, evaluated by the pytorch-OpCounter 49 PIPs consumes 287.5G FLOPS while Context-

---

Query Points
TAP-Net
PIPs
Context-PIP PPPss (Ours)
Figure 2: Qualitative comparison. In the leftmost column, the green crosses mark the query points and the image is the starting frame. The right three columns show the results of TAP-Net, PIPs, and Context-PIPS. The red and green lines illustrate the predicted and ground truth trajectories.
Table 4: Efficiency analysis for PIPs and Context-PIPs with different MLP-Mixer depth.
MLP-Mixer
Flyingthings++
CroHD
Method
Param.(M)
Depth
vis
occ
vis
occ
6
16.06
8.00
25.30
5.04
8.15
PIPs
12
28.67
7.70
24.70
4.98
8.20
16
37.09
7.69
24.71
4.84
8.07
11.54
7.37
24.19
4.54
7.79
Context-PIPs (Ours)
6
17.84
6.94
22.56
4.37
7.05
12
30.46
6.60
22.11
4.30
6.73
PIPs only consumes 216.4G FLOPS, saving 24.7% computation resources. These numbers reveal the high memory and computation efficiency of our proposed Context-PIPs.
4.4 Ablation Study on Modules
conduct a module ablation study on the proposed modules as presented Tab. 5 The errors of Context-PIP- consistently decrease when sequentially add the SOFE and TAFA modules, which reveals the effectiveness of SOFE and TAFA. To demonstrate the necessity of the cross-attention mechanism used in TAFA, attempt to predict matrix of weights corresponding to the feature map shaped with ra and directly weigh and sum the features to get 5F. Cross-attention performs better than prediction.
4.5 Ablation Study on Parameters
We conduct series of ablation experiments (Tab. 6) to demonstrate the significance of cach module and explain the rationality of the settings. All ablation experiments are trained on Flyingthings++. Starting from the PIPs baseline, we first add the SOFE module and explore the two related hyper- parameters, i.e., the correlation radius r. and the number of samples M. Then, we further add the TAFA module and also adjust the attention window radius ra- We additionally conduct comparison between the prediction and attention mechanisms TAFA. In the below experiments, we set N 64, learning rate 10 and train for 20, 000 steps. Below we describe the details.
Correlation Radius in SOFE We crop multi-scale correlation of size (2r, + 1) x (2r, 1) from the first correlation map to predict the auxiliary feature offsets in SOFE, The correlation radius re

---

Table 5: Modules ablation study.
Flyingthings++
CroHD
Method
Experiment
vis
occ
vis
OCc
PIPs
Baseline
7.40
24.4
4.73
7.97
+SOFE
6.91
22.64
4.36
7.15
+SFF++ TAFA (attention)
6.60
22.11
4.30
6.73
Context-PIPs
+SOFE+TAFA (prediction)
7.15
23.33
4.34
7.27
Table 6: Ablation study. We add component at time on top of the baseline obtain our Context-
PIPs. ra M, and Ta respectively denote the correlation radius and the number of predicted samples
SOFE, and the attention window radius in TAFA. Our final model uses M 9. 2, ra 3 achieve the best performance.
Parameters
Flyingthings++
CroHD
Method
Experiment
M
vis
occ
vis
occ
PIPs
Baseline
14.42
37.33
6. 6.14
9,97
13.60
36.17
6.40
10.30
13.02
35.60
6,48
9.69
Correlation Radius (r.)
13.07
35.74
6.23
10.09
3
4
13.75
37.01
6.64
10.20
+SOFE
2
15,00
37.51
6.94
10.25
13.02
35.60
6.48
9.69
Number of Samples (M)
6
2
13.21
35.39
6.58
9.77
12.18
34.23
5.71
9.18
12
2
12.87
35.27
6.20
10.17
11.98
34.10
5.90
9.52
2
11.82
33.82
5.64
9.28
+SOFE+TAFA
Attention Window (ra))
2
11.67
33.38
5.53
9.19
2
4
11.81
33.88
5.65
9.23
2
11.83
33.76
5.60
9.15
determines the cropping patch size. We fix M 3, and gradually increase re from achieves the best performance when r. = 2.
The model
Number of Samples in SOFE SOFE learns to sample M additional auxiliary features enhance the source feature. Given r. = 2, we continued to experiment with the different number of samples M. The model achieves the best performance both Flyingthings++ and CroHD datasets when M
Attention Radius TAFA TAFA aggregates target features surrounding the currently estimated corresponding point locations to enhance the context feature via cross-attention. The radius of the attention window Fa determines how far the attention can reach. We gradually increase Ta from up to 5, and find that r. = 3 performs best.
5 Conclusion
We have presented a novel framework Context-PIPs that improves PIPs with spatial context features, including a SOurce Feature Enhancement (SOFE) module and a TArget Feature Aggregation (TAFA) module. Experiments show that Context-PIPs achieves the best tracking accuracy on four benchmark datasets with significant superiority. This technology has broad applications in video editing and 3D reconstruction and other fields. Limitations. Following PIPs, Context-PIPs tracks points in videos with sliding window. The target point cannot be re-identified when the point lost. In our future work, we will explore how to re-identify the lost points when the points are visible again. Acknowlegement This project is funded in part by National Key R&D Program of China Project 20227D0161 100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoH by General Research Fund Hong Kong RGC Project 14204021. Hongsheng Li is PI of CPII under the InnoHK.

---

References
[1] Michael Black and Padmanabhan Anandan. A framework for the robust estimation of optical flow. In 1993 (4th) International Conference Computer Vision, pages 231-236. IEEE, 1993.
[2] Andrés Bruhn, Joachim Weickert, and Christoph Schnörr. Lucas/kanade meets horn/schunck: Combining local and global optic flow methods. International journal of computer vision, 61(3):211 (13):211-231111 1311, 2005.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[4] Toshio M Chin, William Clement Karl, and Alan S Willsky. Probabilistic and sequential computation of optical flow using temporal coherence. IEEE Transactions Image Processing, (000:773-788, 1994, [5] Carl Docrsch, Ankush Gupta, Larisa Markeeva, Adrià Recasens, Lucas Smaira, Yusuf Aytar, João Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in a video. arXiv preprint wrXiv2221..03726. 2022.
[6] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg. Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758 1758-2766, 2015.
[7] Michael Elad and Arie Feuer. Recursive optical flow estimation adaptiive filtering approach. Journul of Visual Communication and image representation. 9(2):119-138, 1998.
[8] Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. IEEE transactions on pattern andlysis and machine intelligence, 40(3):61 1-625, 2017.
[9] Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In European conference on computer vision, pages 834.849. Springer, 2014.
[10] Mathias Gehrig. Mario Millhäusler, Daniel Gehrig, and Davide Scaramuzza. E-raft: Dense optical flow from event cameras. In 202/ International Conference on 3D Vision (3DV), pages 197-206. IEFF, 2021.
[11] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3749-3761, 2022.
[12] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 59-75. Springer, 2022.
[13] Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, (7(1-3):85--200 1981.
[14] Xiaotao Hu, Zhewei Huang, Ailin Huang, Jun Xu, and Shuchang Zhou. A dynamic multi-scale voxel flow network for video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6121-6131, 2023.
[15] Zhihao Hu and Dong Xu. Vidcocontrolnet: A motion guided iddo-to-video translation framework by using diffusion model with controlnct. arXiv preprint arXiv:2307. 1002.,023. 2023.
[16] Zhaoyang Huang. Xiaokun Pan, Weihong Pan, Weikang Bian, Yan Xu, Ka Chun Cheung, Guofeng Zhang. and Hongsheng Li. Neuralmarker: A framework for learning general marker correspondence. ACM Transactions on Graphics (TOG), (16)):1-10, 2022.
(17] Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. arXiv preprint arXiv:2203. 16194, 2022.
[18] Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy. Liteflownet: A lightweight convolutional neural network for optical flow estimation. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 8981-8988 2018
[19] Tak Wai Hui, Xiaoou Tang, and Chen Change Loy. A lightweight optical flow enn-revisiting data fidelity and regularization. IEEE transactions on puttern analysis and machine intelligence, 43(8):2555-2569, 2020.
[20] Eddy Ilg. Nikolaus Mayer. Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462-2477 22700 2017.
[21] Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning estimate hidden
motions with global motion aggregation. arXiv preprint arXiv:2104.0 00440009..9...... 2021.
[22] Wei Jiang. Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. Cotr: Correspondence
transformer for matching across images. arXiv preprint arXiv:2103. 14166, 2021.
[23] Dahun Kim, Sanghyun Woo, Koon-Young Lee, and In So Kweon. Deep video inpainting. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5792-5801, 2019.
[24] Dasong Li, Yi Zhang, Ka Chun Cheung. Xiaogang Wang, Hongwei Qin. and Hongsheng Li. Learning degradation representations for image deblurring. In Shai Avidan, Gabriel Brostow, Moustapha Cissé,
10

---

Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022. pages 736-753, Cham, 2022. Springer Nature Switzerland.
[25] Yijin Li, Zhaoyang Huang, Shuo Chen, Xiaoyu Shi, Hongsheng Li, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Blinkflow; A dataset push the limits of event-based optical flow estimation. arXiv preprint arXiv:2303.077 16, 2023.
[26] Nikolaus Mayer, Eddy 1lg. Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox A large dataset to train convolutional networks for disparity. optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
4040-4048, 2016.
[27] Zhixiang Min, Yiding Yang, and Enrique Dunn. Voldor: Visual odometry from log-logistic dense optical flow residuals. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
pages 4898-4909. 2020.
[28] Anurag Ranjan and Michael J Black. Optical flow estimation using a spatial pyramid network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4161-4170, 2017.
[29] Zhile Ren, Orazio Gallo, Deqing Sun, Ming- King-Hsuan Yang, Frik B Sudderth, and Jan Kautz. A fusion approach for multi-frame optical flow estimation. In 2019 IEEE Winter Conference on Applications of Computer Vision ( (WACV), pages 2077-2086. IEEE, 2019.
[30] Peter Sand and Seth Teller. Particle video: Long-range motion estimation using point trajectories. Interna- tional Journal of Computer Vision, 80:72-91, 2008.
[31] Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang. Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. arXiv preprint arXiv:2303.08344 2023.
[32] Xiaoyu Shi, Zhaoyang Huang, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin. Jifeng Dai, and Hongsheng Li. Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation. arXiv preprint arXiv:
2023.
[33] Xiuchao Sui, Shaohua Li, Xue Geng, Yan Wu, Xinxing Xu, Yong Liu, Rick Goh, and Hongyuan Zhu. Craft: Cross-attentionn attentional flow transformer for robust optical flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17602 17611, 2022.
[34] Deqing Sun, Stefan Roth, and Michael Black, A quantitative analysis of current practices in optical flow estimation and the principles behind them. International Journal of Computer Vision, (06(2):115-133)
2014.
[35] Deqing Sun, Xiaodong Yang. Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934 8943, 2018
[36] Jiaming Sun, Zehong Shen, Yuang Wang. Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8922 8931, 2021.
[37] Shangkun Sun, Yuanqi Chen, Yu Zhu, Guodong Guo, and Ge Li. Skflow: Learning optical flow with super 2022.
kernels. arXiv preprint 2rXiv:2205.1400
[38] Ramana Sundararaman, Cedric De Almeida Braga, Eric Marchand, and Julien Pettre. Tracking pedestrian heads in dense crowd. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 3865 3875, June 2021.
[39] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow, In European conference on computer vision, pages 402-40244 419. Springer, 2020.
[40] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner. Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances in neural information processing systems, 34:24261-2444 -24272, 2021.
[41] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09344 2023.
[42] Xiaoshi Wu. Keqiang Sun. Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-image models with human preference. arXiv preprint arXiv:2303. /4420, 2023.
[43] Liu Xinyang, Li Yijin, Teng Yanbin, Bao Hujun, Zhang Guofeng, Zhang Yinda, and Cui Zhaopeng. Multi-modal neural radiance field for monocular dense slam with light- weight tof sensor. In International Conference on Computer Vision (ICCV), 2023.
[44] Jiarui Xu and Xiaolong Wang. Rethinking self self-supervised correspondence learning: A video frame-level similarity perspective. In Proceedings of the IEFE/CVF International Conference Computer Vision,
pages 10075-10085, 2021.
[45] Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. Advances in neural information processing systems, 32:794-805, 805,4-805,,70 2019.
[46] Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender video: Zero-shot text- text-guided video-to-video translation. arXiy preprint arXiv:2306.0m 079554, 2023.

---

[47] Tianwei Zhang, Huayan Zhang, Yang Li, Yoshihiko Nakamura, and Lei Zhang. Flowfusion: Dynamic dense rgb-d slam based on optical flow. In 2020 IEEE International Conference Robotics and Astomation (ICRA), pages 7322-7328. IEEE, 2020.
[48] Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Liu. Particlesfim: Exploiting dense point trajectories for localizing moving cameras in the wild. In Computer ECC 2022 17th European Conference, Tel Aviv, Israel, October 23 23-27, 27, 2022, Proceedings, Part XXXII, pages 523-542.
Springer, 2022.
[49] Ligeng Zhu, Thop: Pytorch-opcounter. Iawken/7/pytorch-OpCCouoo 2019.
12

---

Appendix
A More Implementation Details
SOFE sampler. While sampling auxiliary features, SOFE learns to predict offsets with an MLP- based sampler which consists of 5 linear layers interweaved with RELU activations. The local self-similarities cj at location Xare) would first be projected into 128 feature channels. A 3-layer feedforward network with 4 128 channels followed, outputting the feature in 128 feature channels. The final linear layer is used to predict offsets M x 2 from the 128-channel features.
SOFE CorrEnc. SOFE reduces the augmented correlations (M + 1) 196 correlation feature vector ét through correlation encoder CorrEnc which contains only 2 linear layers. The first linear layer reduces the feature channels to 4 196. After RELU, the later linear layer further reduces the feature channels 196, which is the eÑ.
Table A1: Experiments on FlyingTTingsss and CroHD datasets.
Flying Things++
CroHD
Method
ATE-Vis.
ATE-Occ.
ATE-Vis.
ATE-Occ.
TAP-Net
17.00
20.86
PIPs (Paper)
15.54
36.67
5.16
7.56
PIPs (Re-imp., K 4)
7.70
24.70
4.98
8.20
PIPs (Re-imp., K = 6)
7.40
24.40
4.73
7.97
PIPs (Released)
6.08
19.15
4.56
7.71
Context-PIPs (Ours, K 4)
6.60
22.11
4.30
6.73
Context-PIPs (Ours, K = 6)
6.44
22.22
4.28
7.06
Table A2: Experiments on TAP- FAD-Vid-DAVIS (first), and TAP-Vid-Kinetics (first).
MLP-Mixer
TAP-Vid-DAVIS (first)
TAP-Vid-Kinetis (first)
Method
K
Depth
AJ
A-PCK
AJ
A-PCK
TAP-Net
33.0
48.6
38.5
54.4
PIPs (Re-imp.)
4
6
34.4
51.6
28.8
48.6
PIPs (Re-imp.)
4
12
34.6
51.8
28.8
47.2
PIPs (Re-imp.)
4
16
35.3
52.5
29.9
48.5
PIPs (Re-imp.)
6
12
39.2
55.1
33.4
51.0
PIPs(Released)
38.5
55.4
30.1
48.3
Context-PIPs (Ours)
4
3
38.4
57.1
35.8
54.3
Context-PIPs (Ours)
4
41.0
58.3
36.2
54.8
Context-PIPs (Ours)
4
12
39.7
57.7
38.2
54.8
Context-PIPs (Ours)
6
12
42.7
60.3
40.2
57.0
B More Quantitative Comparisons
PIPs Re-implementation. There official PIPs 12 versions. PTPs (Paper) and PIPs (Released) respectively denote the model reported in the paper and the model provided in the released code. There are many misalignments between the paper description and the released code. We follow the parameters suggested in the paper and the released code but fail to reproduce the results. We, therefore, re-implement two PIPs the baselines. according the settings provided in the paper (K 6) and the released code (K- = 4). K denotes the number of refinement iterations in training. The underscored PIPs (Re-imp.), i.e. K 6, is the chosen baseline for comparison the main paper. The performance of the re- immpeeennted model is better than the numbers reported in the paper (Tab. Although our re-implemented model presents inferior performance than the released model on FlyingThings++ and CroHD, they are comparable on TAP-Vid-DAVI and the re-implement model even better than the released model on SOFE and TAFA modules to the re- re-implemented baselines to obtain our Context-PIP- models.
(Tab. A2). We add our proposed
13

---

Table A3: Experiments on TAP- TAP-Vid-DAVIS (strided) and TAP-Vid-Kineties (strided).
MiP-Mixer
TAP-Vid-DAVIS (strided)
TAP-Vid-Kineties (strided)
Method
K
Depth
AJ
A-PCK
AJ
A-PCK
TAP-Net
38.4
53.1
46.6
60.9
PIPs (Re-imp.)
4
6
41.0
56.6
37.2
55.5
PIPs (Re-imp.)
4
12
41.2
56.5
37.4
54.1
PIPs (Re-imp.)
4
16
41.4
56.8
38.6
55.3
PIPs (Re-imp.)
6
12
45.2
59.8
42.9
58.3
PIPs (Released)
45.6
60.6
39.6
55.6
Context-PIPs (Ours)
4
3
44.8
61.2
44.9
61.2
Context-PIE (Ours)
4
45.9
61.9
45.4
62.1
Context-PIE (Ours)
4
12
46.1
61.9
47.4
62.0
Context-PIPP (Ours)
6
12
48.9
64.0
49.8
64.3
We list the results for Flyingthings++ and CroHD in Tab. A.] the results for TAP- Vid-DAiS (first) and TAP-Vis Sie--Vid--iieee Kinetics (first) in Tab. A2 and the results for TTA--Vd-DAVVE (strided) and TAP-Vi Vid- Kinetics (strided) in Tab. A3 "first" and "strided" are two distinct query sampling strategies proposed by TAP-Vid [5) where "first" sampling only contains the initial visible query points, while "strided" sampling would contain all visible query points in every 5 frames.
The released PIPs model tends to overfit on Things+hingg++++ because although it obtains the lowest error on FlyingThings++ but is inferior on TAP- Vid-VVVdDDDD and TAP- Vid-KKnettcs. Although we only show the Context-PIPs with K performance, outperforming PIPs K by 9.44% and 11.76% on DAVIS and Kinetics. Moreover, Context-PIPs trained with K 4 and 3-layer MLP-Mixer achieves even better results than PIPs trained with K = and 12-layer MLP-Mixer (Tab. A2
in the main paper, K 6 version achieves the best
14


Reviews:
[{"review_id": "mkmPsOLJL7", "reviewer": "Reviewer_ydfF", "summary": "This work proposes a method for video point tracking. It is built upon the previous method of persistent independent particles (PIPs). The authors add context features to the source and target feature encoding in PIPs. The resulting method is called Context-TAP. \n\nThe proposed method is evaluated on multiple benchmarks for video point tracking. Improved tracking accuracy is observed over PIPs and other baseline methods.", "questions": "- Based on my primary concern, my first question is: do the proposed modules work on any other method that deals with this task?\n\n\n- From the comparison with baseline methods, the numeric improvement of tracking accuracy seems less drastic than that of PIPs over other methods. An explanation of the accuracy improvement would help justify the significance of this work. ", "limitations": "The authors have stated the limitations of this work. ", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ Using more context features is tried and true in many vision tasks. This work shows it is also helping with the PIPs method. \n\n+ The paper is generally well written with thorough experiments. The authors provide a detailed ablation study on the design parameters in this method. ", "weaknesses": "- My primary concern is about the scope of this work. The claim is that video point tracking needs contextual information, but the execution is a modification to a specific method for this task. Whether the proposed modification is general enough to boost multiple methods or only specific to PIPs is unknown. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Do the proposed modules work on any other method that deals with this task?\n\nYes, we think so because both recent point-tracking networks[1-2] are built upon PIPs, which iteratively refine the point trajectories via cost information. However, they only take the features of the query points into consideration. It is natural to add our modules to gain benefit from the spatial context. We only test our modules on PIPs because it is the first and only method that provides open-sourced PyTorch code for point-tracking. Applying our modules to other point-tracking networks would be our future work.\n\n> From the comparison with baseline methods, the numeric improvement of tracking accuracy seems less drastic than that of PIPs over other methods. An explanation of the accuracy improvement would help justify the significance of this work.\n\nOur Context-TAP outperforms PIPs by reducing 11.4% Average Trajectory Error of Occluded Points (ATE-Occ) on CroHD and increasing 11.8% Average Percentage of Correct Keypoint (A-PCK) on TAP-Vid-Kinectics. PIPs’ improvement is drastic because it is the first method that learns point tracking with longer temporal information and the new FlyingThings++ dataset. The methods which PIPs compared to, such as RAFT, were not designed for the TAP task.\n\n[1] Carl et.al. TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement.\n\n[2] Yang, et al. PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking."}, {"review_id": "kLb6bC7UlZ", "reviewer": "Reviewer_WGNK", "summary": "This paper presents some technical improvements to PIPs, which is a state-of-the-art method for multi-frame pixel tracking. There are two key modifications to the architecture. The first is: rather than only use the feature of the target to represent the appearance of the target, look at the cost map and estimate a few offsets, and sample additional appearance features at these offsets; this yields additional correlations to be used by the temporal model (the MLP-Mixer). The second modification is: on each frame, do some QKV attention to obtain some additional features, and use these to perform a feature update, in addition to (or instead of (this wasn't entirely clear)) the feature update normally done by the model. These modifications yield 1-2 points gain in accuracy in each dataset considered. ", "questions": "\"The augmented correlation features Cˆ in Eq. 3 encode abundant visual similarities. Therefore, we generate a query from it to extract target context features ...\"\n\nI do not understand this part. The correlations contain similarity information, but not appearance information. It sounds like they are being used to create an appearance query. How (or why) does this work?\n", "limitations": "Looks fine", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Using additional appearance information from the first frame, and additional appearance information across time, are very sensible contributions over PIPs. It is also exciting that these modifications were achieved without increasing the computational complexity of the method. \n", "weaknesses": "For me, the main weaknesses have to do with clarity, mostly in the writing, but also in the results. \n\nIn the experiments, it would be nice to see the \"d_avg\" metric reported on the TAP-Vid benchmarks, as computed in the TAP-Vid paper. Or maybe this is re-named here to A-PCK? Overall, in Table 1 and Table 2, it does not seem like the PIPs results on DAVIS or Kinetics exactly match the table from the TAP-Vid paper. Why is this? \n\nThere is some difference between \"PIPs (Paper)\" and \"PIPs (Re-imp.)\", but I could not find text talking about this. I understand that the PIPs github provides a model slightly improved over the original paper, plus maybe some bug-fixes. Is that the re-implementation referenced here? Or is this a new implementation, produced independently by the authors?\n\n\"We train our Context-TAP and PIPs with different MLP-Mixer depths, i.e., the number of layers in the MLP-Mixer, to show the extraordinary efficiency and effectiveness of our proposed Context-TAP.\"\n\nI am not sure that training with different MLP-mixer depths has anything to do with \"extraordinary\" aspects of the method or results. Perhaps this sentence should be re-considered, or supported in some way?\n\n\n\nOverall, considering that the method here builds on PIPs and not TAP-Net, it seems to me that a less confusing name would be something like Context-PIPs (rather than Context-TAP).\n\nI think the acronyms SOFE and TAFA are not improving the clarity of the paper.\n\nIn lines 62-67, the first contribution seems to be just a high-level idea; the second contribution seems to be two independent contributions, and the third contribution is actually a measurement of the middle contributions. I think this could be rewritten into two good contributions plus an evaluation.\n\n\"PIPs and TAP solve the video particle tracking problem in a similar manner, i.e., recurrently refining multi-frame point trajectory via correlation maps.\"\n\nIt seems not accurate to say that TAP-Net involves iterative/recurrent refinement.\n\n\nSOurse -> Source\n\n\neveluated -> evaluated\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> In the experiments, it would be nice to see the \"d_avg\" metric reported on the TAP-Vid benchmarks, as computed in the TAP-Vid paper. Or maybe this is re-named here to A-PCK?\n\nYes. We just renamed \"d_avg\" to A-PCK here because the Average Percentage of Correct Keypoints (PCK) is a more common terminology in correspondence tasks[1-3]. We will clarify the name in the revised version.\n\n> Overall, in Table 1 and Table 2, it does not seem like the PIPs results on DAVIS or Kinetics exactly match the table from the TAP-Vid paper. Why is this?\n\nBecause we cannot exactly reimplement the performance of the released model even with the official repository. We thus provide three versions of PIPs: PIPs (paper), PIPs (re-implement), and PIPs (released). We align all the training settings of PIPs (re-implement) to our Context-TAP for a fair comparison. The other two results are provided in the supplementary for reference. Notice that our PIPs (re-implement) is even better than PIPs (released) on Kinetics because the released PIPs tends to overfit FlyingThings++ dataset, as discussed in L12 of our supplementary.\n\nTAP-Net evaluates PIPs on the DAVIS and Kinectics with the released model. We also provide such model named \"PIPs (released)\". Its performance is improved because we slightly tune the visibility threshold in the chaining rule of PIPs. Specifically, for a video longer than 8 frames, PIPs iteratively selects new visible starting points and chains the trajectory. In PIPs, a point is visible if the predicted visibility is larger than 0.9. However, we observe that there is a significant visibility distribution bias on the training set of FlyingThings++. We use 0.9 as the visibility threshold on DAVIS so that PIPs achieved slightly better results than those reported by TAP-Net.\n\n> There is some difference between \"PIPs (Paper)\" and \"PIPs (Re-imp.)\", but I could not find text talking about this. I understand that the PIPs github provides a model slightly improved over the original paper, plus maybe some bug-fixes. Is that the re-implementation referenced here? Or is this a new implementation, produced independently by the authors?\n\nWe provide the details of our PIPs re-implementation in the L12 of our supplementary materials and will make it clearer in the final version. We use the code from the PIPs repository to re-implement the PIPs. As stated in the previous question, we cannot perfectly reproduce the numbers reported by the PIPs paper or the model released in the repository, so we align the training settings of PIPs with our Context-TAP for a fair comparison.\n  \n> I am not sure that training with different MLP-mixer depths has anything to do with \"extraordinary\" aspects of the method or results. Perhaps this sentence should be re-considered, or supported in some way?\n\nHere we want to express that when the spatial context information is introduced, comparable results can be obtained even if the number of MLP-Mixer layers is significantly reduced. The parameters of the network (11.54M v.s. 28.67M) and the FLOPs (216.4G v.s. 287.5G) evaluated with pytorch-OpCounter[4] for point tracking are reduced up to 59.7% and 24.7%, which is prominent. We will replace the extraordinary with \"prominent\".\n\n> Overall, considering that the method here builds on PIPs and not TAP-Net, it seems to me that a less confusing name would be something like Context-PIPs (rather than Context-TAP).\n\nA5.Thanks for your suggestion. The main reason we use TAP is that we think Tracking Any Point (TAP) is a more general and easy-to-understand term compared with Persistent Independent Particles (PIPs).\n\n> I think the acronyms SOFE and TAFA are not improving the clarity of the paper.\n\nThe acronyms SOFE and TAFA are mainly to shorten the name of the module, which is a commonly adopted strategy in previous literatures such as LoFTR[1], RAFT[6], and GMA[5]. We will also carefully consider the naming in the final version, such as Source feature Enhancement Module (SEM) and Target feature Aggregation Module (TAM).\n\n> In lines 62-67, I think this could be rewritten into two good contributions plus an evaluation.\n\nThanks for the suggestion! We will reorganize the description of our contribution.\n\n>\"PIPs and TAP solve the video particle tracking problem in a similar manner, i.e., recurrently refining multi-frame point trajectory via correlation maps.\"  \n> It seems not accurate to say that TAP-Net involves iterative/recurrent refinement.  \n> SOurse -> Source  \n> eveluated -> evaluated  \n\nWe would refine the statement and typos in the final version.\n\n> \"The augmented correlation features Cˆ in Eq. 3 encode abundant visual similarities. Therefore, we generate a query from it to extract target context features ...\"  \n> I do not understand this part. The correlations contain similarity information, but not appearance information. It sounds like they are being used to create an appearance query. How (or why) does this work?\n\nPixels that hold similar appearance features share similar motions, so that we learn to find auxiliary features that provide informative motion cues through the feature similarities, i.e., correlations. This design is inspired by GMA [5], which smoothes the optical flow estimation via averaging the flow with appearance similarities.\n\n[1] Sun, Jiaming, et al. LoFTR: Detector-free local feature matching with transformers.\n\n[2] Truong, Prune, et al. Pdc-net+: Enhanced probabilistic dense correspondence network.\n\n[3] Huang, Zhaoyang, et al. Neuralmarker: A framework for learning general marker correspondence.\n\n[4] https://github.com/Lyken17/pytorch-OpCounter\n\n[5] Jiang, Shihao, et al. Learning to estimate hidden motions with global motion aggregation.\n\n[6] Zachary and Deng. Raft: Recurrent all-pairs field transforms for optical flow."}, {"review_id": "WX06KDthLA", "reviewer": "Reviewer_ujcD", "summary": "This paper tackled the problem of Trakcing Any Point (TAP). Given a query point and a series of video frames, output all the coordinates corresponding to the query point in the video frame. The problem is interesting and can be regarded as an extension of optical flow.\n\nPIPs only takes the features corresponding to the query point because this task estimate spoint trajectories independently. This paper takes the features surrounding the query point, called as ``spatial context features’’, to improve the accuracy of the point trajectory estimation. The motivation is reasonable. Experiments also demonstrate that Context-TAP outperformed PIPs by large margins and also improves the efficiency.", "questions": "Seeing Weakness.", "limitations": "Context-TAP adopts a fixed window size and handles long-term videos in a sliding manner. Context-TAP may struggle to track a point occluded in a long time interval.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The proposed SOFE module learn to sample more features in the source image significantly improves the performance. I think the essence behind the module is the enlargement of perception field in the cost volume. In recent years, point-based correspondence estimation emerges, such as COTR[1] and ECO-TR[2]. The proposed technique may also be applied to this methods.\n2. Context features are widely adopted in many SOTA correspondence models, such as stereo matching [3]and optical flow[4], i.e., the $F$ in Context-TAP, but they focus on the context features of the original pixel. This paper further aggregates the context features in the target images via attention mechanism (TAFA), which may also motivate the model design in the other correspondence tasks. \n3. The motivation is well presented.\n4. The experiments show that with the aid of the context features, the MLP-Mixer layers can be largely decreased, which improves the efficiency.\n5. The ablation study is conducted thoroughly, revealing the necessity of the attention based TAFA and the number of samples.\n\n[1] Jiang et. al. COTR: Correspondence Transformer for Matching Across Images\n[2] Tan et. al. ECO-TR: Efficient Correspondences Finding Via Coarse-to-Fine Refinement\n[3] Lipson et.al. RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching\n[4] Teed and Deng RAFT: Recurrent All-Pairs Field Transforms for Optical Flow\n\n", "weaknesses": "1.  TAP-Net also provide the occlusion accuracy (OA) metric. I am curious about how the spatial context features affect the occlusion prediction accuracy, but there is no discussion in the experiments. What is the reason?\n2. The efficiency comparison only compares the number of parameters in the paper. I think time and memory usage is more important in practice.\n3. I find that the authors provide a better model $K=6$ in the supplementary. What’s the reason? Should this model be moved to the main paper?\n4. The benefit brought by the TAFA is not large.\n5. The SOFE compute correlations between additional source features and target features. There are some other correlations that can be computed as context features. For example, the correlation between adjacent target images. Would these additional information also benefits TAP?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> TAP-Net also provide the occlusion accuracy (OA) metric. I am curious about how the spatial context features affect the occlusion prediction accuracy, but there is no discussion in the experiments. What is the reason?\n\nThe OA comparison is listed below:\n\n| Method             | K | MLP-Mixer Depth | TAP-Vid-DAVIS (first) | TAP-Vid-Kinectics (first) | TAP-Vid-DAVIS (strided) | TAP-Vid-Kinetics (strided) |\n|--------------------|---|-----------------|-----------------------|---------------------------|-------------------------|----------------------------|\n| TAP-Net            | - | -               | 78.8                  | 80.6                      | 82.3                    | 85.0                      \n| PIPs (Re-imp.)     | 6 | 12              | 79.3                  | 77.0                      | 82.9                    | 81.5                       |\n| PIPs(Released)     | - | -               | 79.0                  | 75.7                      | 83.2                    | 81.0                       |\n| Context-TAP (Ours) | 6 | 12              | 79.5                  | 79.8                      | 83.4                    | 83.3                       |\n\n> The time and memory usage comparison.\n\nWe will add the memory usage and FLOPs in the final version. When the number of MLP-Mixer layers is significantly reduced, the parameters of the network (11.54M v.s. 28.67M) and the FLOPs (216.4G v.s. 287.5G) evaluated with pytorch-OpCounter[1] for point tracking are reduced up to 59.7% and 24.7%.\n\n> Why the model K=6 is in the supplementary?\n\nDue to the limited time, we could not finish this experiment before the submission deadline. We will move the results to the main paper in the final version.\n\n> The benefit brought by the TAFA is not large.\n\nThe improvement is still large by reducing the 5.9% Average Trajectory Error of Occluded Points (ATE-Occ) on CroHD.\n\n> Would more additional information also benefit TAP?\n\nThanks for your suggestions. We believe so and this will become our future work.\n\n[1] https://github.com/Lyken17/pytorch-OpCounter"}, {"review_id": "4h6tFF59Us", "reviewer": "Reviewer_2Jdq", "summary": "This paper aims at estimating persistent long-term trajectories of query points in videos. For ignoring the potential benefits of 4 incorporating spatial context features, this paper argues that independent video point tracking also demands spatial context features. And this paper proposes a novel framework Context-TAP, which effectively improves point trajectory accuracy by aggregating spatial context features in videos. The framework Context-TAP contains two modules: 1) a source Feature Enhancement module, and 2) a target Feature Aggregation module, enhancing information with surrounding information from both source image and target image respectively. The Context-TAP ranks 1st on the four benchmarks 66 and shows clear performance superiority.", "questions": "1）In the SOFE module, are auxiliary features involved in the update iteration?\n2）whether the inference process can be computed in parallel, and whether the computation time is the same or multiplied compared to the computation of only two frames of optical flow\n3）Is it possible to visualize predicted samples to see if they actually capture semantically useful key points？\n", "limitations": "For the outlook in limitation is an important task to be solved in the TAP task, with a practical application scenario", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1). This paper is well-written and easy to understand\n2). Experimental results are extensive and better than existing baseline models", "weaknesses": "The motivation and the solution to the problem are reasonable but seem regular approaches and it’s very common. The use of spatial context, regression samples to assist with features, has been seen very often. Therefore, the novelty of the ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The novelty of using spatial context.\n\nSpatial context is independent of temporal information. However, most of the existing methods mainly use temporal information.\n\nWe argue that our spatial context feature component is novel:\n\n1. We did not naively adopt common spatial context features techniques[1-2]. Instead, we fuse the cost information of the auxiliary context features for point tracking refinement, which has never been explored before. This may also further motivate the network design in other tasks.\n\n2. We first show that target spatial context features improve point-tracking, which is ignored in closely related correspondence tasks such as optical flow and stereo matching. \n\nAs suggested by Reviewer ujcD, our design may be further utilized in other tasks such as COTR and optical flow.\n\n> Are auxiliary features involved in the update iteration?\n\nYes.\n\n> Whether the inference process can be computed in parallel\n\nYes.\n\n> Whether the computation time is the same or multiplied compared to the computation of only two frames of optical flow.\n\nNo. For an 8-frame Flyingthings++ sequence. The Context-TAP consumes 0.225s while 7 RAFT consumes 2.244s. Almost x10 slower.\n\n> Is it possible to visualize predicted samples to see if they actually capture semantically useful key points?\n\nYes. We will add the visualization of learn-to-sample results in the final version.\n\n[1] Luo, Hao, et al. Object detection in video with spatial-temporal context aggregation.\n\n[2] Simon and Liu. Context-aware synthesis for video frame interpolation."}]

Author rebuttal:

Meta-review: All reviewers voted to accept the paper. The authors have promised to include metrics not reported along with runtime details in the final camera ready. The AC also agrees with the reviewer that the title is confusing. Given the paper extends the PIP method and not the TAP method the AC highly recommends changing the title to reflect this (perhaps Context-PIP) as Context-TAP strongly suggests the method is extending TAP. With these changes the AC has determined that this paper should be accepted to Neurips 2023.
Decision: Accept (spotlight)