SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Evaluation on synthetic datasets. Our model with cubic B-splines and ODE encoder outperforms baselines on sparse validation (Query-Stride set) and dense test (Query-First set).
Method	Metric	Mark	Query-Stride set	Query-First set
8f	10f	16f	24f	8f	10f	16f	24f
RAFT	ADE_Vis	S	8.18	7.91	13.00	14.43	5.68	7.18	11.12	17.56
NS	7.13	6.08	10.21	11.31	4.94	5.42	8.47	13.91
ADE_Occ	S	9.62	12.16	27.73	24.95	6.26	8.60	14.77	24.86
NS	8.79	9.37	21.98	18.06	5.60	6.43	10.72	19.04
ADE_All	S	8.32	8.17	14.93	16.37	5.73	7.27	11.44	18.49
NS	7.39	6.37	12.37	13.06	5.00	5.51	8.75	14.72
TRMSE	-	8.50	7.82	15.81	16.66	5.86	6.77	10.35	16.85
PIPs	ADE_Vis	S	4.66	4.39	8.77	10.13	3.12	4.12	7.81	14,23
NS	3.97	3.40	6.92	7.92	2.97	3.26	5.97	10.27
ADE_Occ	S	7.13	7.71	21.54	19.97	4.85	7.28	12.89	22.34
NS	6.43	5.93	16.83	15.16	4.23	4.03	9.31	16.35
ADE_All	S	4.74	4.58	10.41	12.09	3.76	4.93	8.35	14.68
NS	4.13	3.61	8.68	9.86	3.25	3.76	6.42	11.76
TRMSE	-	4.93	4.49	11.27	13.02	3.90	4.72	7.76	13.70
ODE-6spline (Ours)	ADE_Vis	S	3.81	3.88	8.08	8.22	2.88	3.72	6.26	12.08
NS	3.38	3.11	6.47	6.60	2.51	2.86	4.83	9.73
ADE_Occ	S	6.24	8.09	22.72	19.84	4.70	6.86	11.34	19.61
NS	5.65	6.35	18.21	15.07	4.17	5.04	8.24	15.19
ADE_All	S	4.00	4.13	9.89	10.61	2.96	3.88	6.71	13.05
NS	3.60	3.37	8.54	8.92	2.61	3.02	5.26	10.64
TRMSE	-	4.26	4.17	11.38	12.30	3.12	3.78	6.42	12.38
* S, NS are respectively referred to as the sampling and non-sampling moments.
$f$  is an abbreviation of frames.
Since our goal is to predict continuous flow trajectories, not only corresponding to inputs, but also inter-frame trajectories. This allows us to set some of the whole frames in the dataset to be visible while others are hidden. The ground-truth displacements of the hidden frames can be used for the continuity evaluation, including inter-frame motion. Therefore, we introduce additional motion assumptions for the baselines to connect discrete points into complete continuous trajectories. We experimented with multiple assumptions, but the linear one yielded more stable results.
5.4 Evaluation on Simulated Data
We use two subsets of our synthetic dataset for validation in sampling ranges with 8, 10, 16, and 24 frames with uniform sampling. As mentioned earlier for the two evaluation perspectives, we evaluated inter-frame motion accuracy and multi-frame motion robustness, respectively. The result of the quantitative comparison is found in Table 1. As shown in Fig. 3, benefited by global optimization, our temporal parametric model outperforms baselines at all moments for all variable temporal ranges.
From the accuracy perspective, benefiting from continuous explicit modeling, our algorithm remains high performance at both sampling and non-sampling moments. The rate of error accumulation with the extension of the inference range is also controlled validly, especially for visible points. From the temporal stability perspective, compared with widespread polynomial motion approximation between reference and target frames, our curves based on limited variable control points can adjust the local trajectory flexibly for overall smoothness. Besides, our model is capable of adapting to both sparse and dense inference, particularly on dense estimation of short distances, which can sustain more consistent predictions. To reduce the instability of parametric modeling for a longer inference range, we expect to exert the powerful influence of implicit feature representation. By introducing the Neural ODE module with Conv-GRU update units, temporal features across multi-frames will be more convenient for motion separation and aggregation.
Table 2: Evaluation on real-world datasets. Our model with cubic B-splines and ODE encoder outperforms baselines on real scenes.
Method	Metric	Mark	Vid-DAVIS [8]	Vid-Kinetics [8]
20f	24f	28f	32f	36f	48f	128f	250f
RAFT	ADE_Vis	S	16.84	19.72	19.42	23.37	15.97	19.00	30.19	35.93
NS	12.42	14.71	15.87	18.53	12.98	16.36	24.99	30.63
ADE_Occ	S	18.58	23.82	27.91	31.35	19.23	21.89	31.98	39.76
NS	14.67	18.56	20.19	22.53	16.07	18.85	26.11	33.10
ADE_All	S	17.99	20.25	22.97	25.65	16.74	20.70	31.47	37.81
NS	14.23	16.07	17.91	19.68	14.90	17.73	25.16	31.49
TRMSE	-	16.91	18.95	21.28	23.81	16.62	24.51	27.50	34.23
PIPs	ADE_Vis	S	12.96	15.47	15.62	18.73	12.71	16.48	28.34	38.56
NS	10.39	11.39	12.60	15.27	10.49	13.30	22.31	31.08
ADE_Occ	S	18.84	26.67	30.60	32.21	17.59	21.08	31.82	42.31
NS	14,42	20.66	20.88	23.26	15.42	18.56	25.34	35.41
ADE_All	S	15.18	17.98	22.06	23.31	13.70	17.79	31.76	40.59
NS	11.94	14.38	16.62	18.49	12.73	13.72	24.67	33.90
TRMSE	-	14.80	17.38	20.66	22.79	12.51	15.82	28.60	37.25
ODE-6spline (Ours)	ADE_Vis	S	11.37	16.80	16.16	18.99	12.09	15.02	25.25	31.18
NS	9.31	12.21	13.37	15.21	9.99	12.33	21.35	27.13
ADE_Occ	S	15.75	20.67	26.71	27.99	17.76	20.50	29.79	37.12
NS	12.23	15.91	18.00	20.55	15.17	17.99	24.98	32.00
ADE_All	S	13.48	16.91	19.81	21.16	12.47	15.82	27.67	34.42
NS	10.79	13.30	15.07	16.68	10.47	13.10	22.80	29.65
TRMSE	-	12.96	15.76	18.12	20.27	12.06	14.91	25.44	32.78
5.5 Evaluation on Real Data
We choose the TAP-Vid-DAVIS and TAP-Vid-Kinetics datasets[8] to evaluate our model on real data, with 20,24,28,32 frames for evaluation on DAVIS and 36,48,128,250 frames including more non-sampling moments for Kinetics. For fairness of comparison, we need all methods to complete dense flow estimation and use identical interpolation techniques to obtain the corresponding trajectory of sparse points. The evaluation result is shown in Table 2 and the visualization result in Fig. 4.
According to the test result, we find our powerful model can surpass the flow-based method and point-tracking algorithm PIPs in all test ranges for all time ranges. Although PIPs perform well in specific point tracking, the model needs heavy costs for large numbers of points. We only infer 3072 points simultaneously for  $256 \times 256$  resolution in the test stage with the 4-frame model. For dense validation, we have to process in batch with longer inference time while a flow-based solution needs more link steps resulting in more time consumption.
However, our model still shows potential in real scenes connected with our explicit continuous curve fitting. The motion trend is similar to ground truth even though there are slight trembles in the middle process. Thanks to large-scale training and implicit spatial-temporal continuous features, our work can keep relatively stable tracking which is helpful to reduce the drift in long-range prediction.
5.6 Ablation
We analyze two core components that contribute to our model on feature and trajectory continuity modeling. As for the feature encoder, the basic structure is from RAFT and no extra module is added for temporal representation. The ODE encoder will supplement an ODE unit with the combination of neural ODE function and Conv-GRU iterative part after basic feature extraction. As for the parametric curve, We set the hyper-parameter of the B-spline to reflect the number of control points which is designated 4 and 8. The result is shown in Table 3.
In the feature continuity aspect, ODE unit is valid for our parametric curve and equipped with a stronger ability to capture temporal-spatial connection. About the parametric curve selection, cubic B-spline with 4 and 8 control points are both weaker than 6 control points because less or more partition from control points will generate more loosen or tight constraints.
(b) RAFT [12]
(c) PIPs [9]
(d) Ours
Figure 4: The visualization of sparse point trajectories on Vid-DAVIS.
Table 3: The ablation of model components.
Module	Method	Metric	8f	10f	16f	24f
Control Point	ODE + 4spline	ADE_All	4.41	3.97	9.35	9.68
TRMSE	4.83	4.57	12.09	12.65
ODE + 8spline	ADE_All	3.94	3.91	9.06	9.13
TRMSE	4.40	4.47	11.66	12.14
Neural-ODE	Basic + 6pline	ADE_All	4.89	4.43	9.79	9.92
TRMSE	5.45	5.04	12.41	12.93
All	ODE + 6spline	ADE_All	3.77	3.62	8.80	8.86
TRMSE	4.26	4.17	11.38	12.03
We have presented continuous parametric optical flow, a parametric representation of dense and continuous motion, which describes real pixel motion trajectory and expects to promote frame-to-frame discrete flow into continuous time-to-time motion. We generate a synthetic dataset based on Kubric for our task and introduce evaluation rules for continuity. With multi-frame inputs, a pipeline with the fusion of continuous features and the parametric curve is proposed for flow estimation. Experimental results on synthetic and real scenes reveal that our method focuses more on unseen inter-frame trajectory and there is room for development for all moments.
Limitations and Broader Impacts. As a parametric representation, the ability of our method to handle challenging complex motions will be limited by the choice of the degree of model and we have to admit the limitation in all parametric models, where a strong explicit constraint potentially leads to better performance on those more similar motion scenery with the training set. Our work is foundational research, for which we do not foresee it will bring negative social impacts. We also believe that the ethical concerns of our method in the training and evaluation process are minimal and there is no harm or bias to anyone. All weight parameters rely on learning from valid synthetic data [11] and real data utilized for evaluation comes from TAP-Vid [8] with a clear statement for the fairness and unbiasedness of these benchmarks.
Acknowledgments. This research was supported in part by the National Natural Science Foundation of China (62271410, 62001394), the Fundamental Research Funds for the Central Universities, and the Innovation Foundation for Doctor Dissertation of Northwestern Polytechnical University (CX2023013).
[38] Li Xu, Jianing Chen, and Jiaya Jia. A segmentation based variational model for accurate optical flow estimation. In European Conference on Computer Vision (ECCV), pages 671-684. Springer, 2008.
[39] Tal Nir, Alfred M Bruckstein, and Ron Kimmel. Over-parameterized variational optical flow. International Journal of Computer Vision (IJCV), 76:205-216, 2008.
[40] Michael Hornáček, Frederic Besse, Jan Kautz, Andrew Fitzgibbon, and Carsten Rother. Highly overparameterized optical flow using patchmatch belief propagation. In European Conference on Computer Vision (ECCV), pages 220-234. Springer, 2014.
[41] Jiaolong Yang and Hongdong Li. Dense, accurate optical flow estimation with piecewise parametric model. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1019-1027, 2015.
[42] Deqing Sun, Erik Sudderth, and Michael Black. Layered image motion with explicit occlusions, temporal consistency, and depth ordering. Advances in Neural Information Processing Systems (NeurIPS), 23, 2010.
[43] Deqing Sun, Erik B Sudderth, and Michael J Black. Layered segmentation and optical flow estimation over time. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1768-1775. IEEE, 2012.
[44] Chen Song, Qixing Huang, and Chandrajit Bajaj. E-cir: Event-enhanced continuous intensity recovery. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7803-7812, 2022.
[45] Chaoyang Wang, Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey. Neural prior for trajectory estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6532-6542, 2022.
[46] Simone Meyer, Oliver Wang, Henning Zimmer, Max Grosse, and Alexander Sorkine-Hornung. Phase-based frame interpolation for video. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1410-1418, 2015.
[47] Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang. Quadratic video interpolation. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[48] Zhixiang Chi, Rasoul Mohammadi Nasiri, Zheng Liu, Juwei Lu, Jin Tang, and Konstantinos N Plataniotis. All at once: Temporally adaptive multi-frame interpolation with advanced motion modeling. In European Conference on Computer Vision (ECCV), pages 107-123. Springer, 2020.
[49] Se Young Chun, Timothy G Reese, Jinsong Ouyang, Bastien Guerin, Ciprian Catana, Xuping Zhu, Nathaniel M Alpert, and Georges El Fakhri. Mri-based nonrigid motion correction in simultaneous pet/mri. Journal of Nuclear Medicine, 53(8):1284-1291, 2012.
[50] Carl De Boor. On calculating with b-splines. Journal of Approximation theory, 6(1):50-62, 1972.
[51] Morris G Cox. Curve fitting with piecewise polynomials. IMA Journal of Applied Mathematics, 8(1): 36-52, 1971.
[52] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, volume 11006, pages 369-386. SPIE, 2019.