SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We evaluate our Context-PIPs on four benchmarks: FlyingThings++ [12], CroHD [38], TAP-VidDAVIS, and TAP-Vid-Kinetics [5]. Following PIPs [12], we train Context-PIPs on Flyingthings++ only and evaluate it on other benchmarks without finetuning. Context-PIPs achieves state-of-the-art performance on all benchmarks and significantly improves PIPs. Moreover, we show that by utilizing spatial context features in Context-PIPs, we achieve on-par performance with PIPs when using only  $40.2\%$  of its parameters.
Datasets Flyingthings++ is a synthetic dataset based on Flyingthings3D [26], which contains 8-frame trajectories with occlusion. The video resolution is  $384 \times 512$  for both training and evaluation. Crowd of Heads Dataset (CroHD) is a high-resolution crowd head tracking dataset. Following PIPs, the RAFT [39], PIPs, and our Context-PIPs are evaluated at  $768 \times 1280$  resolution. DINO [3] and TAP-Net [5] are respectively evaluated at  $512 \times 768$  and  $256 \times 256$  resolution. TAP-Vid-DAVIS and TAP-Vid-Kinetics are two evaluation datasets in the TAP-Vid benchmark, both of which consist of real-world videos with accurate human annotations for point tracking. Note that TAP-Vid provides two distinct query sampling strategies, i.e., first and strided. The "first" sampling contains only the initial visible query points until the last frame, while the "strided" sampling is to sample all visible query points in every 5 frames. RAFT, TAP-Net, PIPs, and our Context-PIPs are evaluated with two different sampling strategies respectively at  $256 \times 256$  resolution. While Kubric-VFS-like [44, 11] and COTR [22] are only evaluated with the "first" sampling strategies at the same resolution.
Experiment Setup We use the average trajectory error (ATE) metric [12] for evaluation on FlyingThings++ and CroHD. ATE measures the average L2 distances between the coordinates of all predicted points in the trajectories and the corresponding ground truth coordinates. According to the ground truth visibility of the points, we calculate ATEs for all visible points and occluded points separately. i.e., ATE-Vis. and ATE-Occ. We use the average Jaccard (AJ) [12], the average percentage of correct keypoint (A-PCK) metrics for TAP-Vid-DAVIS and TAP-Vid-Kinetics. The Jaccard metric measures the ratio of "true positive" visible points within a given threshold. Average Jaccard (AJ) averages Jaccard across the different thresholds. PCK measures the percentage of the predicted coordinates whose L2 distances from the ground truth are smaller than a given threshold. A-PCK refers to calculating PCK according to multiple different thresholds and then taking the average. We set the thresholds as 1, 2, 4, 8, and 16.
Implementation Details We train our Context-PIPs with a batch size of 4 and 100,000 steps on Flyingthings++ with horizontal and vertical flipping. We use the one-cycle learning rate scheduler. The highest learning rate is set as  $5 \times 10^{-4}$ . During training, we set the convolution stride to 8 and the resolution of the input RGB images to  $384 \times 512$ , and randomly sample  $N = 128$  visible query points for supervision. To limit the length of the input videos, we set  $T = 8$  and apply the trajectory linking mechanism [12] at test time, similarly to PIPs. To align to the PIPs paper, the PIPs compared in Tab. 1 and Tab. 2 is trained with  $K = 6$ . In the ablation study, all models are trained with  $K = 4$  while tested with  $K = 6$  for a fair comparison.
4.1 Quantitative Comparison
FlyingThings++ and CroHD As shown in Tab. 1, Context-PIPs ranks 1st on all metrics and presents significant performance superiority compared with previous methods. Specifically, Context-PIPs
Table 2: Experiments on TAP-Vid-DAVIS(strided) and TAP-Vid-Kinetics(strided).
Method	TAP-DAVIS(strided)	TAP-Kinetics(strided)
AJ	A-PCK	AJ	A-PCK
RAFT	30.0	46.3	34.5	52.5
Kubric-VFS-Like	33.1	48.5	40.5	59.0
COTR	35.4	51.3	19.0	38.8
TAP-Net	38.4	53.1	46.6	60.9
PIPs(Re-imp.)	45.2	59.8	42.9	58.3
Context-PIPs (Ours)	48.9	64.0	49.8	64.3
Table 3: Occlusion Accuracy on TAP-Vid-DAVIS and TAP-Vid-Kinetics.
Method	First	Strided
TAP-DAVIS	TAP-Kinetics	TAP-DAVIS	TAP-Kinetics
TAP-Net	78.8	80.6	82.3	85.0
PIPs (Re-imp.)	79.3	77.0	82.9	81.5
Context-PIPs (Ours)	79.5	79.8	83.4	83.3
achieves 7.06 ATE-Occ and 4.28 ATE-Vis on the CroHD dataset,  $11.4\%$  and  $9.5\%$  error reductions from PIPs, the runner-up. On the FlyingThings++ dataset, our Context-PIPs decreases the ATE-Vis and ATE-Occ by 0.96 and 2.18, respectively.
TAP-Vid-DAVIS and TAP-Vid-Kinetics (first) A-PCK, the average percentage of correct key points, is the core metric. Context-PIPs ranks 1st in terms of A-PCK on both benchmarks. Specifically, Context-PIPs outperforms TAP-Net by  $24.1\%$  on the TAP-Vid-DAVIS benchmark and improves PIPs by  $11.8\%$  on the TAP-Vid-Kinetics benchmarks. Context-PIPs also achieves state-of-the-art occupancy accuracy on both datasets.
TAP-Vid-DAVIS and TAP-Vid-Kinetics (strided) Tab. 2 compares methods in the "strided" sampling setting. Our Context-PIPs also achieves the best performance on both AJ and A-PCK metrics for the two datasets. Context-PIPs effectively improves PIPs' occupancy accuracy and presents the best performance on the TAP-Vid-Davis dataset.
4.2 Qualitative Comparison
We visualize the trajectories estimated by TAP-Net, PIPs, and our Context-PIPs respectively in Fig 2 to qualitatively demonstrate the superior performance of our method. By incorporating additional spatial context features for point tracking, Context-PIPs surpasses the compared methods in accuracy and robustness. Specifically, the first row shows the case of large-scale variation. The trajectory predicted by TAP-Net deviates considerably from the ground truth. TAP-Net also outputs jittery predictions when the query pixel is on the texture-less area as shown in the second row. Our Context-PIPs generates more accurate results than PIPs in these two hard cases. Furthermore, as depicted in the third row, PIPs struggles to distinguish the front wheel and the rear wheel due to the changing lighting conditions. However, our Context-PIPs achieves consistent tracking, thanks to the rich context information brought by the SOFE and TAFA modules.
4.3 Efficiency Analysis
We train our Context-PIPs and PIPs with different MLP-Mixer depths, i.e., the number of layers in the MLP-Mixer, to show the prominent efficiency and effectiveness of our proposed Context-PIPs. Context-PIPs improves PIPs with SOFE and TAFA, which introduce minor additional parameters and time costs. We show that the accuracy benefits do not come from the increased parameters trivially. As displayed in Tab. 4, we increase the MLP-Mixer depth to 16, which significantly increases the parameters but does not bring performance gain. We also decrease the MLP-Mixer depth in our Context-PIPs. Even with only a 3-layer MLP-Mixer, Context-PIPs achieves better performance than the best PIPs (MLP-Mixer depth=12). Context-PIPs outperforms PIPs with only  $40.2\%$  parameters. Moreover, evaluated by the pytorch-OpCounter [49], PIPs consumes 287.5G FLOPS while Context-
Query Points
TAP-Net
PIPs
Figure 2: Qualitative comparison. In the leftmost column, the green crosses mark the query points and the image is the starting frame. The right three columns show the results of TAP-Net, PIPs, and Context-PIPs. The red and green lines illustrate the predicted and ground truth trajectories.
Context-PIPs (Ours)
Table 4: Efficiency analysis for PIPs and Context-PIPs with different MLP-Mixer depth.
Method	MLP-Mixer
Depth	Param.(M)	Flyingthings++	CroHD
vis	occ	vis	occ
PIPs	6	16.06	8.00	25.30	5.04	8.15
12	28.67	7.70	24.70	4.98	8.20
16	37.09	7.69	24.71	4.84	8.07
Context-PIPs (Ours)	3	11.54	7.37	24.19	4.54	7.79
6	17.84	6.94	22.56	4.37	7.05
12	30.46	6.60	22.11	4.30	6.73
PIPs only consumes 216.4G FLOPS, saving  $24.7\%$  computation resources. These numbers reveal the high memory and computation efficiency of our proposed Context-PIPs.
4.4 Ablation Study on Modules
We conduct a module ablation study on the proposed modules as presented in Tab. 5. The errors of Context-PIPs consistently decrease when we sequentially add the SOFE and TAFA modules, which reveals the effectiveness of SOFE and TAFA. To demonstrate the necessity of the cross-attention mechanism used in TAFA, we attempt to predict a matrix of weights corresponding to the feature map shaped with  $r_a$  and directly weigh and sum the features to get  $\delta F$ . Cross-attention performs better than prediction.
4.5 Ablation Study on Parameters
We conduct a series of ablation experiments (Tab. 6) to demonstrate the significance of each module and explain the rationality of the settings. All ablation experiments are trained on Flyingthings++. Starting from the PIPs baseline, we first add the SOFE module and explore the two related hyperparameters, i.e., the correlation radius  $r_c$  and the number of samples  $M$ . Then, we further add the TAFA module and also adjust the attention window radius  $r_a$ . We additionally conduct a comparison between the prediction and attention mechanisms in TAFA. In the below experiments, we set  $N = 64$ , learning rate as  $3 \times 10^{-4}$ , and train for 20,000 steps. Below we describe the details.
Correlation Radius in SOFE We crop a multi-scale correlation of size  $(2r_c + 1)\times (2r_c + 1)$  from the first correlation map to predict the auxiliary feature offsets in SOFE. The correlation radius  $r_c$
Table 5: Modules ablation study.
Method	Experiment	Flyingthings++	CroHD
vis	occ	vis	occ
PIPs	Baseline	7.40	24.4	4.73	7.97
Context-PIPs	+SOFE	6.91	22.64	4.36	7.15
+SOFE+TAFA (attention)	6.60	22.11	4.30	6.73
+SOFE+TAFA (prediction)	7.15	23.33	4.34	7.27
Table 6: Ablation study. We add one component at a time on top of the baseline to obtain our Context-PIPs.  $r_c$ ,  $M$ , and  $r_a$  respectively denote the correlation radius and the number of predicted samples in SOFE, and the attention window radius in TAFA. Our final model uses  $M = 9$ ,  $r_c = 2$ ,  $r_a = 3$  to achieve the best performance.
Method	Experiment	Parameters	Flyingthings++	CroHD
M	rc	ra	vis	occ	vis	occ
PIPs	Baseline	-	-	-	14.42	37.33	6.14	9.97
+SOFE	Correlation Radius (rc)	3	1	-	13.60	36.17	6.40	10.30
3	2	-	13.02	35.60	6.48	9.69
3	3	-	13.07	35.74	6.23	10.09
3	4	-	13.75	37.01	6.64	10.20
Number of Samples (M)	1	2	-	15.00	37.51	6.94	10.25
3	2	-	13.02	35.60	6.48	9.69
6	2	-	13.21	35.39	6.58	9.77
9	2	-	12.18	34.23	5.71	9.18
12	2	-	12.87	35.27	6.20	10.17
+SOFE+TAFA	Attention Window (ra)	9	2	1	11.98	34.10	5.90	9.52
9	2	2	11.82	33.82	5.64	9.28
9	2	3	11.67	33.38	5.53	9.19
9	2	4	11.81	33.88	5.65	9.23
9	2	5	11.83	33.76	5.60	9.15
determines the cropping patch size. We fix  $M = 3$ , and gradually increase  $r_c$  from 1 to 4. The model achieves the best performance when  $r_c = 2$ .
Number of Samples in SOFE SOFE learns to sample  $M$  additional auxiliary features to enhance the source feature. Given  $r_c = 2$ , we continued to experiment with the different number of samples  $M$ . The model achieves the best performance on both Flyingthings++ and CroHD datasets when  $M = 9$ .
Attention Radius in TAFA TAFA aggregates target features surrounding the currently estimated corresponding point locations to enhance the context feature via cross-attention. The radius of the attention window  $r_a$  determines how far the attention can reach. We gradually increase  $r_a$  from 1 up to 5, and find that  $r_a = 3$  performs best.
We have presented a novel framework Context-PIPs that improves PIPs with spatial context features, including a SOurce Feature Enhancement (SOFE) module and a TAtarget Feature Aggregation (TAFA) module. Experiments show that Context-PIPs achieves the best tracking accuracy on four benchmark datasets with significant superiority. This technology has broad applications in video editing and 3D reconstruction and other fields. Limitations. Following PIPs, Context-PIPs tracks points in videos with a sliding window. The target point cannot be re-identified when the point is lost. In our future work, we will explore how to re-identify the lost points when the points are visible again. Acknowledgement This project is funded in part by National Key R&D Program of China Project 2022ZD0161100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK, by General Research Fund of Hong Kong RGC Project 14204021. Hongsheng Li is a PI of CPII under the InnoHK.
[47] Tianwei Zhang, Huayan Zhang, Yang Li, Yoshihiko Nakamura, and Lei Zhang. Flowfusion: Dynamic dense rgb-d slam based on optical flow. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 7322-7328. IEEE, 2020.
[48] Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Liu. *Particlesfm: Exploiting dense point trajectories for localizing moving cameras in the wild*. In *Computer Vision-ECCV* 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII, pages 523–542. Springer, 2022.
[49] Ligeng Zhu. Thop: Pytorch-opcounter. Lyken17/pytorch-OpCounter, 2019.
A More Implementation Details
SOFE sampler. While sampling auxiliary features, SOFE learns to predict offsets with an MLP-based sampler which consists of 5 linear layers interweaved with RELU activations. The local self-similarities  $\mathbf{c}_0^0$  at location  $\mathbf{x}_{src}$  would first be projected into 128 feature channels. A 3-layer feedforward network with  $4\times 128$  channels followed, outputting the feature in 128 feature channels. The final linear layer is used to predict offsets  $M\times 2$  from the 128-channel features.
SOFE CorrEnc. SOFE reduces the augmented correlations  $(M + 1)\times 196$  to a correlation feature vector  $\hat{\mathbf{c}}_t^k$  through a correlation encoder CorrEnc which contains only 2 linear layers. The first linear layer reduces the feature channels to  $4\times 196$ . After a RELU, the later linear layer further reduces the feature channels to 196, which is the  $\hat{\mathbf{c}}_t^k$ .
Table A1: Experiments on FlyingThings++ and CroHD datasets.
Method	FlyingThings++	CroHD
ATE-Vis.	ATE-Occ.	ATE-Vis.	ATE-Occ.
TAP-Net	-	-	17.00	20.86
PIPs (Paper)	15.54	36.67	5.16	7.56
PIPs (Re-imp., K=4)	7.70	24.70	4.98	8.20
PIPs (Re-imp., K=6)	7.40	24.40	4.73	7.97
PIPs (Released)	6.08	19.15	4.56	7.71
Context-PIPs (Ours, K=4)	6.60	22.11	4.30	6.73
Context-PIPs (Ours, K=6)	6.44	22.22	4.28	7.06
Table A2: Experiments on TAP-Vid-DAVIS (first), and TAP-Vid-Kinetics (first).
Method	K	MLP-Mixer Depth	TAP-Vid-DAVIS (first)	TAP-Vid-Kinetics (first)
AJ	A-PCK	AJ	A-PCK
TAP-Net	-	-	33.0	48.6	38.5	54.4
PIPs (Re-imp.)	4	6	34.4	51.6	28.8	48.6
PIPs (Re-imp.)	4	12	34.6	51.8	28.8	47.2
PIPs (Re-imp.)	4	16	35.3	52.5	29.9	48.5
PIPs (Re-imp.)	6	12	39.2	55.1	33.4	51.0
PIPs(Released)	-	-	38.5	55.4	30.1	48.3
Context-PIPs (Ours)	4	3	38.4	57.1	35.8	54.3
Context-PIPs (Ours)	4	6	41.0	58.3	36.2	54.8
Context-PIPs (Ours)	4	12	39.7	57.7	38.2	54.8
Context-PIPs (Ours)	6	12	42.7	60.3	40.2	57.0
B More Quantitative Comparisons
PIPs Re-implementation. There are two official PIPs [12] versions. PIPs (Paper) and PIPs (Released) respectively denote the model reported in the paper and the model provided in the released code. There are many misalignments between the paper description and the released code. We follow the parameters suggested in the paper and the released code but fail to reproduce the results. We, therefore, re-implement two PIPs as the baselines, according to the settings provided in the paper  $(K = 6)$  and the released code  $(K = 4)$ .  $K$  denotes the number of refinement iterations in training. The underscored PIPs (Re-imp.), i.e.  $K = 6$ , is the chosen baseline for comparison in the main paper. The performance of the re-implemented model is better than the numbers reported in the paper (Tab. A1). Although our re-implemented model presents inferior performance than the released model on FlyingThings++ and CroHD, they are comparable on TAP-Vid-DAVIS and the re-implement model is even better than the released model on TAP-Vid-Kinetics (Tab. A2). We add our proposed SOFE and TAFA modules to the re-implemented baselines to obtain our Context-PIPs models.
Table A3: Experiments on TAP-Vid-DAVIS (strided) and TAP-Vid-Kinetics (strided).
Method	K	MLP-Mixer Depth	TAP-Vid-DAVIS (strided)	TAP-Vid-Kinetics (strided)
AJ	A-PCK	AJ	A-PCK
TAP-Net	-	-	38.4	53.1	46.6	60.9
PIPs (Re-imp.)	4	6	41.0	56.6	37.2	55.5
PIPs (Re-imp.)	4	12	41.2	56.5	37.4	54.1
PIPs (Re-imp.)	4	16	41.4	56.8	38.6	55.3
PIPs (Re-imp.)	6	12	45.2	59.8	42.9	58.3
PIPs (Released)	-	-	45.6	60.6	39.6	55.6
Context-PIPs (Ours)	4	3	44.8	61.2	44.9	61.2
Context-PIPs (Ours)	4	6	45.9	61.9	45.4	62.1
Context-PIPs (Ours)	4	12	46.1	61.9	47.4	62.0
Context-PIPs (Ours)	6	12	48.9	64.0	49.8	64.3
We list the results for Flyingthings++ and CroHD in Tab. A1, the results for TAP-Vid-DAVIS (first) and TAP-Vid-Kinetics (first) in Tab. A2, and the results for TAP-Vid-DAVIS (strided) and TAP-Vid-Kinetics (strided) in Tab. A3. "first" and "strided" are two distinct query sampling strategies proposed by TAP-Vid [5], where "first" sampling only contains the initial visible query points, while "strided" sampling would contain all visible query points in every 5 frames.
The released PIPs model tends to overfit on FlyingThings++ because although it obtains the lowest error on FlyingThings++ but is inferior on TAP-Vid-DAVIS and TAP-Vid-Kinetics. Although we only show the Context-PIPs with  $K = 4$  in the main paper, our  $K = 6$  version achieves the best performance, outperforming PIPs  $K = 6$  by  $9.44\%$  and  $11.76\%$  on DAVIS and Kinetics. Moreover, Context-PIPs trained with  $K = 4$  and 3-layer MLP-Mixer achieves even better results than PIPs trained with  $K = 6$  and 12-layer MLP-Mixer (Tab. A2).