Title: Continuous Parametric Optical Flow
Venue: NeurIPS 2023
Accepted: yes
Keywords: optical flow

point trajectories

continuous motion

neural ordinary differential equation
Abstract: In this paper, we present continuous parametric optical flow, a parametric representation of dense and continuous motion over arbitrary time interval. In contrast to existing discrete-time representations (i.e., flow in between consecutive frames), this new representation transforms the frame-to-frame pixel correspondences to dense continuous flow. In particular, we present a temporal-parametric model that employs B-splines to fit point trajectories using a limited number of frames. To further improve the stability and robustness of the trajectories, we also add an encoder with a neural ordinary differential equation (NODE) to represent features associated with specific times. We also contribute a synthetic dataset and introduce two evaluation perspectives to measure the accuracy and robustness of continuous flow estimation. Benefiting from the combination of explicit parametric modeling and implicit feature optimization, our model focuses on motion continuity and outperforms the flow-based and point-tracking approaches for fitting long-term and variable sequences.

Continuous Parametric Optical Flow

Abstract

In this paper, we present continuous parametric optical flow , a parametric representation of dense and continuous motion over arbitrary time interval . In contrast to existing discrete-time representations ( i.e. flow in between consecutive frames), this new representation transforms the frame-to-frame pixel correspondences to dense continuous flow. In particular, we present a temporal-parametric model that employs B-splines to fit point trajectories using a limited number of frames. To further improve the stability and robustness of the trajectories, we also add an encoder with a neural ordinary differential equation (NODE) to represent features associated with specific times. We also contribute a synthetic dataset and introduce two evaluation perspectives to measure the accuracy and robustness of continuous flow estimation. Benefiting from the combination of explicit parametric modeling and implicit feature optimization, our model focuses on motion continuity and outperforms the flow-based and point-tracking approaches for fitting long-term and variable sequences.

1 Introduction

Motion estimation, which characterizes the movement of targets over time, is crucial for video understanding tasks like keypoint tracking [1] or video interpolation [2]. It also provides accurate priors for 3D tasks such as non-rigid structure-from-motion [3, 4] and simultaneous localization and mapping [5]. Fine-grained motion understanding is necessary to achieve pixel-level correspondence, which is usually studied as an optical flow estimation problem. Some traditional methods [6, 7] based on variational optimization or recent deep methods focusing on the Tracking-Any-Point (TAP) task [8, 9] both attempt to estimate long-range flow for tracking points across multi-frames.

While these approaches have made significant progress in modeling correspondences, there still exist two obstacles to explaining and predicting practical motion, as shown in Fig. 1. First, the point trajectories generated by these approaches only reflect the possible locations at certain moments instead of real motion trends because the frame-to-frame correspondence is temporally discrete and lacks enough inter-frame information to describe continuous point motion. Second, optical flow-based methods could achieve dense but short-term point mapping while recent point trackers tend to strengthen long-term tracking for sparse points. Neither can efficiently realize dense and long-term tracking simultaneously.

In this paper, we aim to estimate the dense and continuous pixel correspondence with the continuous flow that provides the moving trajectory from reference time to arbitrary target times within the whole

(a) Discrete-time Optical Flow [12, 13]

(b) Point Tracking [9]

video. We utilize a learnable parametric model to explicitly describe the continuous flow trajectory, which avoids some inconsistent and abrupt displacements compared to the implicit counterparts directly predicting the final result. Motivated by [10], we embed the continuous property into our end-to-end optimization pipeline to enhance continuity in feature space. By combining the powerful representational ability and explicit motion pattern, our method is more appropriate for solving continuous problems. To evaluate the continuous flow estimation, we introduce two evaluation perspectives, including continuous motion with invisible frames at specific timescales and robustness validation at variable timescales. We also use Kubric [11] to create a synthetic dataset with dense and long trajectories for experiments.

Our main contributions are summarized as:

We present continuous parametric optical flow, a temporal parametric representation of dense and continuous motion over arbitrary time intervals.

We utilize Neural ODE and ConvGRU to aggregate the spatial-temporal information and enhance the continuity of features among target times.

We create a large-scale synthetic dataset with dense and long-term trajectories and propose a two-stage evaluation pipeline to measure the accuracy and robustness of continuous flow estimation.

2 Related Work

2.1 Optical Flow Estimation

Optical flow estimation is a fundamental task in computer vision, and existing optimization-based methods [14, 15, 16, 17] have made explorations in challenging situations such as large motions [18, 19] and occlusions[20, 21]. While early learning-based methods take an encoder-decoder [22] or coarse-to-fine [23, 24] structure, recent methods, such as RAFT [12] and GMA [25], usually take an iterative update pipeline. These methods usually take procedures of supervised training on simulated data [25, 13] or unsupervised training on real data [26, 27].

Unlike the above common two-frame input, some methods [26, 28] use multi-frames to aggregate additional flows to tackle the occlusion. MFSF [6] aims to establish the long-range correspondence of multi-frame flow by a linear combination of trajectory basis. SlowFlow [29] links adjacent flows to longer trajectories by constraining brightness, smoothness, and visibility. Gehrig et al. [30] proposes to use event data to build motion trajectories for fast and non-linear motion in a short period.

2.2 Feature and Point Tracking

Among the self-supervised feature tracking methods, TimeCycle [31] uses cycle consistency as the primary constraint for patch-level extraction and matching. CRW [32] designs a special palindrome arrangement to introduce cycle loss. VFS [33] introduces contrastive learning to optimize frame-level matching. Instead of implicit exploration in unlabeled data, supervised methods [34, 35] rely on ground truth correspondence to learn a general feature representation. Although these methods perform well on general tasks like keypoint tracking or semi-supervised video segmentation, pixellevel matching remains unsolved because image or patch embedding seems insufficient for specific point refinement.

For long-term point tracking, the key problem is how to match the spatial-temporal features of the long-range correspondences. PIPs [9] proposes to extract query point features for pixel tracking and occlusion estimation with iterative updates, motivated by Particle Video [36]. TAP-Net [8] uses a flow-based method to assist annotators to track sparse points selected by random sampling and generate point trajectories with longer duration.

2.3 Parametric Motion Hypothesis

Rigid motion modeling was a fashion for early work before various modern learning techniques. As for spatial modeling, some representative methods [37, 38] use robust parametric motion like affine transformation or based on over-parameterization [39, 40] to estimate flow. Yang and Li [41] propose a piecewise, 8-Dof homography model to fuse multiple possible parametric motions. Besides, layered models [42, 43] also perform competitively with a few overlapping motion layers that balance the need for smoothness and boundary separation.

For temporal consistency, the linear combination of motion basis [6, 44, 45] is a common method for trajectory fitting in the time axis. These pre-defined bases like DCT [6] or Lagrange [44] can convert the continuous time domain into the discrete analysis. With different weight assignments for basis groups, this modeling approach is convenient to implement for deep paradigm [45]. However, limited by the sampling range, these models are not flexible for video clips with different frame numbers. Another alternative is to perform temporally continuous curve fitting, including linear [46], quadratic [47], or cubic [48] curve hypotheses with polynomial modeling.

In addition, complex motion trajectories can be represented by adaptive curves with a few control points like B-splines [49] or Bezier [30]. Although most of these parametric hypotheses were first introduced in previous optimization methods, modern methods can still take a deep network as the process of motion parameterization by regressing a set of coefficients or control points.

3 Parametric Modeling

3.1 Preliminaries of Continuous Flow

To describe the continuous flow that expresses the motion from the reference time to any target time, we define a triplet ( t, u, v ) as a query of flow trajectory decoupled into two components along horizontal and vertical direction for corresponding pixel ( x, y ) , t is sampled from normalized time T ∈ [0 , 1] . For brevity, the reference time is zero. Instead of directly fitting the pixel position, we choose to estimate the pixel displacements and the start point of flow trajectory ( u, v ) = (0 , 0) . Note that although we can only use finite labels to evaluate our model, we introduce two evaluation perspectives to measure the performance under diverse situations.

Stage 1: Blind Estimation. Given a specific number of frames at sampled timestamps with a fixed timescale as input, the model should predict the target flow trajectory at unsampled times.

Stage 2: Temporal Adaptation. The pre-trained model on a fixed training range should adapt to diverse sequences with different timescales.

3.2 Parametric Curve for Continuous Flow Modeling

Given N control points { P i = ( x i , y i ) } N ∈ R N × 2 and basis functions B i,k ( t ) with degree k , the point displacement trajectory F ( t ) is formulated as:

For any specific time t , the basis functions return a group of weights to determine the contributions for the curve shape of each control point. The parametric curve we employ is a cubic B-spline formed by a few learnable control points and spline basis functions. In our work, the coordinates of N control points serve as model outputs, whereas the normalized knots { t i } are manually specified hyperparameters for local optimization in the temporal range. The detailed derivation of basis functions is referred to Cox-de Boor recursion [50, 51]:

We assign each time to a single partition to ensure the sum of the weight coefficients of all control points is equal to 1. Knots can be repeated to promote the curve from open to close. In our experiment, a special shape like the clamp seems easier to reach convergence. Benefiting from the potential advantages of parameter design, the cubic B-spline could achieve the combination of global relaxation and fine-tuning on local segments.

The parametric curve is optimized independently for each pixel to build a complete displacement flow field. As a result, our model directly predicts a 3D tensor with the shape of 2 N × H × W for control points in all trajectories.

4 Optimization

The optimization process consists of four components: multi-frame sampling, implicit continuity enhancement, multi-time correlation pyramids, and iterative control point decoder. According to the order of the input video clip with L frames, the corresponding timestamps are normalized to [0 , 1] . Our framework aims to regress a set of control points with their 2D coordinates for each pixel and to compute the final continuous flow trajectories by Eq. (1).

4.1 Multi-frame Sampling

For a given video clip with L frames, the model input M frames are first sampled from the entire sequence as visible moments in the continuous time domain while the remaining L -M frames are assumed to be invisible. We use the ground-truth displacement fields sampled from all valid moments corresponding to the frames as the supervision of continuous flow trajectories for model training.

During training, we first select the start frame with a sufficient number of visible points from the given video clip. Then we randomly sample a subsequence clip with M frames with different intervals as model inputs to enhance the generation ability to different motion magnitudes. Notably, this selection process keeps the start and end frames and only randomly selects the intermediate M -2 frames based on brightness differences.

4.2 Implicit Continuity Enhancement

As shown in Fig. 2, we first extract M basic feature maps using a shared basic encoder for each of the input M frames. Similar to RAFT [12], we also feed the reference frame ( i.e. the start frame with t = 0 ) into the context encoder for extracting another context feature to assist subsequent motion estimation. Unlike PIPs [9], we introduce ODE-ConvGRU proposed in [2] for spatio-temporal feature learning. This module consists of the Neural ODE for temporal continuity enhancement in feature space and the Conv-GRU for spatio-temporal information aggregation. The Neural ODE formulated

(𝟎, 𝟎)

Multi-time

Correlation Pyramids

C

𝒕 𝒓𝒆𝒇

𝒕 𝒆𝒏𝒅

Context Encoder

Basic Encoder

𝝌𝟏

𝝌𝑴

𝝌𝒄

𝒉(𝒕

𝟏

𝒕𝒈𝒕

)

𝑪𝟏

𝒁

𝑪𝒒

𝒁

C

Concatenation

Control Points

ConvGRU

𝒉(𝒕

𝑳

𝒕𝒈𝒕

)

𝒉(𝒕

𝟐

𝒕𝒈𝒕

)

Iterative

Update

Unit

ODE

Solver

෡

𝒉(𝟎)

ODE Unit

Transform

ODE Unit

ODE Solver

0

ODE Unit

ODE Solver

ODE Unit

ODE Solver

by Eq. (4) includes a derivative estimator f as an independent ODE unit and an ODE solver, in which the parameters involve the initial value h ( t 0 ) , the current time t and input state h in ( t ) . The former uses a learnable fully-connected network f to fit the feature differentiation at time t , and the latter integrates to recover the hidden state h ( t ) at t started from t 0 .

The timestamps corresponding to the input M frames serve as source times { t src j } M to feed into the ODE-ConvGRU module and output features at target times { t tgt j } L for correlation construction. The Neural ODE models the features of the entire L -frame video clip including invisible frames, so it also outputs L target times.

Our ODE-ConvGRU module consists of three steps. First and foremost, we compute the initial hidden states h ( t src j ) at source times t src j by integrating the derivative of the previous hidden state at t src j -1 solved by network f θ :

where this process starts with h ( t src 0 ) = 0 , i.e. , an zero feature map. Note that the computation process is backward, which means the original temporal order needs to be reversed.

Besides, given feature χ extracted by the basic encoder, we update the initial hidden state h ( t src j ) at source time t src j by aggregating them to generate refined hidden state ˜ h ( t src j ) with ConvGRU:

Finally, the last output hidden state ˜ h (0) from ConvGRU aggregates valid information from all source moments, and is transformed into a novel initial value ˆ h (0) by a small network g ω . The derivative is computed by another ODE unit with network f ϕ and evolves into the final hidden state outputs h ( t tgt j ) at target time t tgt j :

4.3 Multi-time Correlation Pyramid

To achieve fine-grained matching, we compute the spatial matching feature similarity between different target times. Due to the powerful spatio-temporal representation of ODE-ConvGRU, we can obtain an arbitrary number of feature pairs for continuous flow estimation. Thus we can perform the correlation operation between the reference state h ( t tgt 0 ) and the rest of hidden states { ( h ( t tgt j ) } L 1 at target time to construct L -1 cost volumes.

However, due to limited GPU memory, we still need to introduce sampling to reduce computational complexity while leveraging the continuous representation capability. We make a trade-off by random sampling Q ( Q ≤ ( L -1)) hidden states to construct Q multi-time correlations C = { C q } Q . We then perform the average pooling operation and obtain the correlation pyramids C q = { C z q } Z ( Z is the pooling level) for each correlation separately, which covers matching similarities at multiple resolutions for subsequent local lookup.

4.4 Iterative Lookup and Update

Inspired by RAFT [12], we also adopt the lookup operation to search the local matching patches from multi-time correlations. Subsequently, we concatenate the patches with different moments and different pooling levels as a single tensor and feed it into the decoder to regress the residual of control points and compute the continuous optical flow. This procedure of local lookup and flow regression is iteratively executed by adjusting the local search starting point according to the updated control points.

For N control points { P i } N , the update unit outputs N translations { ∆ P i } N and updates iteratively:

where i is the index of the control point and s is the iteration number for lookup and update.

4.5 Supervision

As there are only discrete point displacement ground truths in the dataset, we can only use them to discretely supervise the predicted continuous optical flow of our model. Assuming that for each point there are N gt displacement ground truths at different times, we calculate the point displacements on the predicted continuous trajectory with the corresponding times { t sup w } N gt according to Eq. (1).

The loss function measures the average displacement error at N gt different times and all visible pixels with a weighted sum of all iterations:

where the iteration times is S and weight is γ . F ( t sup w ) is the point displacements at time t sup w on the predicted continuous trajectory, D gt ( t sup j ) is the the corresponding point displacement ground truth. Note that the starting point of the continuous trajectory is 0, so we omit it.

5 Experiments

This section consists of four associated parts. We first use Kubric [11] to simulate a relatively dense and longer pixel-tracking dataset for our model training and as a basic benchmark for evaluation. Second, to establish valid baselines, we introduce two approaches [9, 12] to provide direct estimation for sampling moments and parametric assumption to simulate inter-frame motion. Third, we set error metrics for quantitative comparison and show the results on synthetic scenes and real-world scenes. Finally, ablation experiments are implemented for module validity tests.

We train our model on four RTX 3090 GPUs with a batch size of 8. At the training stage, we use the resolution of 256 × 256 and randomly sample 20480 point trajectories as sparse supervisions for better performance. We train for 30,000 steps because more iterations may result in degradation on diverse timescales with a learning rate of 2e-5 and a one-cycle schedule [52]. We repeat multiple times and report median results. We set the numbers of control points N = 6 while the degree k = 3 ,

and the sampling number of frames M = 4 . The number Q of correlation is 3, and the pooling level Z is 3. In the supervision process, the N gt = 8 and γ = 0.8. For temporal augmentation, the length of video clips L is variable but at least 8 frames.

5.1 Dataset

To maintain the same settings as [8, 9], we utilize the Kubric simulator to generate two subsets: query-stride and query-first in our experiment. The video samples in both the two subsets are with 24 frames at 256 × 256 resolution. Points trajectory is sampled from the whole spatial-temporal area in the query-stride version but from the single frame in the query-first version. Query-stride is applied in the training phase for less structured constraints and query-first can be used for dense evaluation.

5.2 Metrics

We use two metrics to measure the accuracy at specific moments and the temporal trajectory smoothness of continuous flow estimation. From a spatial perspective, the average displacement error (ADE) proposed in [9] is applied and divided into visible and occluded parts. To reflect the ability to fit the trajectory during blind time, an additional temporal average is introduced for sampling and non-sampling times. From a temporal perspective, the flow trajectory for a determined point should be robust and smooth between any sequences. We use temporally averaged root-mean-square error (TRMSE) to measure trajectory smoothness, for every candidate pixel x in the spatial domain, and the RMSE along the temporal dimension is computed to measure the influence of outliers. After that, the errors of all pixels are averaged.

where N v is the number of valid points, x ∈ R N s × 2 is the coordinates of valid points. T is the number of times with ground-truth displacements for each point, t a is the corresponding time.

5.3 Baseline

To generate the motion trajectory of each pixel, we choose the flow estimation method RAFT [12] and the point-tracking method PIPs [9] as our baselines. RAFT predicts the optical flow between consecutive frames, and these flows are linked by bilinear interpolation as mentioned in [9]. We use their pre-training model and fine-tune it on our dataset. PIPs is a recent pixel tracking algorithm with promising generalization performance. We retrain PIPs with 4 frames on our dataset and infer the corresponding movements of all pixels during whole input clips to ensure fairness of the comparison.

(b) RAFT

(c) PIPs

(d) Ours

* S, NS are respectively referred to as the sampling and non-sampling moments.

Since our goal is to predict continuous flow trajectories, not only corresponding to inputs, but also inter-frame trajectories. This allows us to set some of the whole frames in the dataset to be visible while others are hidden. The ground-truth displacements of the hidden frames can be used for the continuity evaluation, including inter-frame motion. Therefore, we introduce additional motion assumptions for the baselines to connect discrete points into complete continuous trajectories. We experimented with multiple assumptions, but the linear one yielded more stable results.

5.4 Evaluation on Simulated Data

We use two subsets of our synthetic dataset for validation in sampling ranges with 8, 10, 16, and 24 frames with uniform sampling. As mentioned earlier for the two evaluation perspectives, we evaluated inter-frame motion accuracy and multi-frame motion robustness, respectively. The result of the quantitative comparison is found in Table 1. As shown in Fig. 3, benefited by global optimization, our temporal parametric model outperforms baselines at all moments for all variable temporal ranges.

From the accuracy perspective, benefiting from continuous explicit modeling, our algorithm remains high performance at both sampling and non-sampling moments. The rate of error accumulation with the extension of the inference range is also controlled validly, especially for visible points. From the temporal stability perspective, compared with widespread polynomial motion approximation between reference and target frames, our curves based on limited variable control points can adjust the local trajectory flexibly for overall smoothness. Besides, our model is capable of adapting to both sparse and dense inference, particularly on dense estimation of short distances, which can sustain more consistent predictions. To reduce the instability of parametric modeling for a longer inference range, we expect to exert the powerful influence of implicit feature representation. By introducing the Neural ODE module with Conv-GRU update units, temporal features across multi-frames will be more convenient for motion separation and aggregation.

5.5 Evaluation on Real Data

We choose the TAP-Vid-DAVIS and TAP-Vid-Kinetics datasets[8] to evaluate our model on real data, with 20,24,28,32 frames for evaluation on DAVIS and 36,48,128,250 frames including more non-sampling moments for Kinetics. For fairness of comparison, we need all methods to complete dense flow estimation and use identical interpolation techniques to obtain the corresponding trajectory of sparse points. The evaluation result is shown in Table 2 and the visualization result in Fig. 4.

According to the test result, we find our powerful model can surpass the flow-based method and point-tracking algorithm PIPs in all test ranges for all time ranges. Although PIPs perform well in specific point tracking, the model needs heavy costs for large numbers of points. We only infer 3072 points simultaneously for 256x256 resolution in the test stage with the 4-frame model. For dense validation, we have to process in batch with longer inference time while a flow-based solution needs more link steps resulting in more time consumption.

However, our model still shows potential in real scenes connected with our explicit continuous curve fitting. The motion trend is similar to ground truth even though there are slight trembles in the middle process. Thanks to large-scale training and implicit spatial-temporal continuous features, our work can keep relatively stable tracking which is helpful to reduce the drift in long-range prediction.

5.6 Ablation

We analyze two core components that contribute to our model on feature and trajectory continuity modeling. As for the feature encoder, the basic structure is from RAFT and no extra module is added for temporal representation. The ODE encoder will supplement an ODE unit with the combination of neural ODE function and Conv-GRU iterative part after basic feature extraction. As for the parametric curve, We set the hyper-parameter of the B-spline to reflect the number of control points which is designated 4 and 8. The result is shown in Table 3.

In the feature continuity aspect, ODE unit is valid for our parametric curve and equipped with a stronger ability to capture temporal-spatial connection. About the parametric curve selection, cubic B-spline with 4 and 8 control points are both weaker than 6 control points because less or more partition from control points will generate more loosen or tight constraints.

(b) RAFT [12]

(c) PIPs [9]

(d) Ours

6 Conclusion

We have presented continuous parametric optical flow, a parametric representation of dense and continuous motion, which describes real pixel motion trajectory and expects to promote frame-toframe discrete flow into continuous time-to-time motion. We generate a synthetic dataset based on Kubric for our task and introduce evaluation rules for continuity. With multi-frame inputs, a pipeline with the fusion of continuous features and the parametric curve is proposed for flow estimation. Experimental results on synthetic and real scenes reveal that our method focuses more on unseen inter-frame trajectory and there is room for development for all moments.

Limitations and Broader Impacts. As a parametric representation, the ability of our method to handle challenging complex motions will be limited by the choice of the degree of model and we have to admit the limitation in all parametric models, where a strong explicit constraint potentially leads to better performance on those more similar motion scenery with the training set. Our work is foundational research, for which we do not foresee it will bring negative social impacts. We also believe that the ethical concerns of our method in the training and evaluation process are minimal and there is no harm or bias to anyone. All weight parameters rely on learning from valid synthetic data [11] and real data utilized for evaluation comes from TAP-Vid [8] with a clear statement for the fairness and unbiasedness of these benchmarks.

Acknowledgments. This research was supported in part by the National Natural Science Foundation of China (62271410, 62001394), the Fundamental Research Funds for the Central Universities, and the Innovation Foundation for Doctor Dissertation of Northwestern Polytechnical University (CX2023013).

References

Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi. Openpifpaf: Composite fields for semantic keypoint detection and spatio-temporal association. IEEE Transactions on Intelligent Transportation Systems , 23(8): 13498-13511, 2021.

Sunghyun Park, Kangyeol Kim, Junsoo Lee, Jaegul Choo, Joonseok Lee, Sookyung Kim, and Edward Choi. Vid-ode: Continuous-time video generation with neural ordinary differential equation. In AAAI Conference on Artificial Intelligence (AAAI) , volume 35, pages 2412-2422, 2021.

Vikramjit Sidhu, Edgar Tretschk, Vladislav Golyanik, Antonio Agudo, and Christian Theobalt. Neural dense non-rigid structure from motion with latent space constraints. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16 , pages 204-222. Springer, 2020.

Hui Deng, Tong Zhang, Yuchao Dai, Jiawei Shi, Yiran Zhong, and Hongdong Li. Deep non-rigid structure-from-motion: A sequence-to-sequence translation perspective. arXiv preprint arXiv:2204.04730 , 2022.

Qiang Fu, Hongshan Yu, Xiaolong Wang, Zhengeng Yang, Yong He, Hong Zhang, and Ajmal Mian. Fast orb-slam without keypoint descriptors. IEEE Transactions on Image Processing , 31:1433-1446, 2021.

Ravi Garg, Anastasios Roussos, and Lourdes Agapito. A variational approach to video registration with subspace constraints. International Journal of Computer Vision (IJCV) , 104:286-314, 2013.

Wenbin Li, Darren Cosker, and Matthew Brown. Drift robust non-rigid optical flow enhancement for long sequences. Journal of Intelligent & Fuzzy Systems , 31(5):2583-2595, 2016.

Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens Continente, Kucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in a video. In NeurIPS Datasets Track , 2022.

Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vision (ECCV) , pages 59-75. Springer, 2022.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems (NeurIPS) , 31, 2018.

Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3749-3761, 2022.

Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision (ECCV) , pages 402-419. Springer, 2020.

Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. In European Conference on Computer Vision (ECCV) , pages 668-685. Springer, 2022.

Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence , 17(1-3):185-203, 1981.

Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui, and Cordelia Schmid. Epicflow: Edge-preserving interpolation of correspondences for optical flow. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1164-1172, 2015.

Qifeng Chen and Vladlen Koltun. Full flow: Optical flow estimation by global optimization over regular grids. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4706-4714, 2016.

Christian Bailer, Bertram Taetz, and Didier Stricker. Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation. In IEEE International Conference on Computer Vision (ICCV) , pages 4015-4023, 2015.

Thomas Brox and Jitendra Malik. Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 33(3): 500-513, 2010.

Philippe Weinzaepfel, Jerome Revaud, Zaid Harchaoui, and Cordelia Schmid. Deepflow: Large displacement optical flow with deep matching. In IEEE International Conference on Computer Vision (ICCV) , pages 1385-1392, 2013.

Marius Leordeanu, Andrei Zanfir, and Cristian Sminchisescu. Locally affine sparse-to-dense matching for motion and occlusion estimation. In IEEE International Conference on Computer Vision (ICCV) , pages 1721-1728, 2013.

Yu Li, Dongbo Min, Minh N Do, and Jiangbo Lu. Fast guided global interpolation for depth and motion. In European Conference on Computer Vision (ECCV) , pages 717-733. Springer, 2016.

Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In IEEE International Conference on Computer Vision (ICCV) , pages 2758-2766, 2015.

Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8934-8943, 2018.

Jianyuan Wang, Yiran Zhong, Yuchao Dai, Kaihao Zhang, Pan Ji, and Hongdong Li. Displacementinvariant matching cost learning for accurate optical flow estimation. Advances in Neural Information Processing Systems (NeurIPS) , 33:15220-15231, 2020.

Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning to estimate hidden motions with global motion aggregation. In IEEE International Conference on Computer Vision (ICCV) , pages 9772-9781, 2021.

Joel Janai, Fatma Guney, Anurag Ranjan, Michael Black, and Andreas Geiger. Unsupervised learning of multi-frame optical flow with occlusions. In European Conference on Computer Vision (ECCV) , pages 690-706, 2018.

Kunming Luo, Chuan Wang, Shuaicheng Liu, Haoqiang Fan, Jue Wang, and Jian Sun. Upflow: Upsampling pyramid for unsupervised optical flow learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1045-1054, 2021.

Zhile Ren, Orazio Gallo, Deqing Sun, Ming-Hsuan Yang, Erik B Sudderth, and Jan Kautz. A fusion approach for multi-frame optical flow estimation. In IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 2077-2086. IEEE, 2019.

Joel Janai, Fatma Guney, Jonas Wulff, Michael J Black, and Andreas Geiger. Slow flow: Exploiting high-speed cameras for accurate and diverse optical flow reference data. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3597-3607, 2017.

Mathias Gehrig, Manasi Muglikar, and Davide Scaramuzza. Dense continuous-time optical flow from events and frames. arXiv preprint arXiv:2203.13674 , 2022.

Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2566-2576, 2019.

Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as a contrastive random walk. Advances in Neural Information Processing Systems (NeurIPS) , 33:19545-19560, 2020.

Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A video frame-level similarity perspective. In IEEE International Conference on Computer Vision (ICCV) , pages 10075-10085, 2021.

Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. Cotr: Correspondence transformer for matching across images. In IEEE International Conference on Computer Vision (ICCV) , pages 6207-6217, 2021.

Olivia Wiles, Sebastien Ehrhardt, and Andrew Zisserman. Co-attention for conditioned image matching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15920-15929, 2021.

Peter Sand and Seth Teller. Particle video: Long-range motion estimation using point trajectories. International Journal of Computer Vision (IJCV) , 80:72-91, 2008.

Shanon X Ju, Michael J Black, and Allan D Jepson. Skin and bones: Multi-layer, locally affine, optical flow and regularization with transparency. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 307-314. IEEE, 1996.

Li Xu, Jianing Chen, and Jiaya Jia. A segmentation based variational model for accurate optical flow estimation. In European Conference on Computer Vision (ECCV) , pages 671-684. Springer, 2008.

Tal Nir, Alfred M Bruckstein, and Ron Kimmel. Over-parameterized variational optical flow. International Journal of Computer Vision (IJCV) , 76:205-216, 2008.

Michael Hornáˇ cek, Frederic Besse, Jan Kautz, Andrew Fitzgibbon, and Carsten Rother. Highly overparameterized optical flow using patchmatch belief propagation. In European Conference on Computer Vision (ECCV) , pages 220-234. Springer, 2014.

Jiaolong Yang and Hongdong Li. Dense, accurate optical flow estimation with piecewise parametric model. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1019-1027, 2015.

Deqing Sun, Erik Sudderth, and Michael Black. Layered image motion with explicit occlusions, temporal consistency, and depth ordering. Advances in Neural Information Processing Systems (NeurIPS) , 23, 2010.

Deqing Sun, Erik B Sudderth, and Michael J Black. Layered segmentation and optical flow estimation over time. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1768-1775. IEEE, 2012.

Chen Song, Qixing Huang, and Chandrajit Bajaj. E-cir: Event-enhanced continuous intensity recovery. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7803-7812, 2022.

Chaoyang Wang, Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey. Neural prior for trajectory estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6532-6542, 2022.

Simone Meyer, Oliver Wang, Henning Zimmer, Max Grosse, and Alexander Sorkine-Hornung. Phasebased frame interpolation for video. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1410-1418, 2015.

Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang. Quadratic video interpolation. Advances in Neural Information Processing Systems (NeurIPS) , 32, 2019.

Zhixiang Chi, Rasoul Mohammadi Nasiri, Zheng Liu, Juwei Lu, Jin Tang, and Konstantinos N Plataniotis. All at once: Temporally adaptive multi-frame interpolation with advanced motion modeling. In European Conference on Computer Vision (ECCV) , pages 107-123. Springer, 2020.

Se Young Chun, Timothy G Reese, Jinsong Ouyang, Bastien Guerin, Ciprian Catana, Xuping Zhu, Nathaniel M Alpert, and Georges El Fakhri. Mri-based nonrigid motion correction in simultaneous pet/mri. Journal of Nuclear Medicine , 53(8):1284-1291, 2012.

Carl De Boor. On calculating with b-splines. Journal of Approximation theory , 6(1):50-62, 1972.

Morris G Cox. Curve fitting with piecewise polynomials. IMA Journal of Applied Mathematics , 8(1): 36-52, 1971.

Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications , volume 11006, pages 369-386. SPIE, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission 6736, entitled \"Continuous Parametric Optical Flow,\" presents a novel multi-frame optical flow strategy that expresses the flow continuously. This is in contrast to conventional flow strategies, which encode this displacement in a discretized manner. This result is made possible by regressing the B-spline parameters of the pixel displacement. The approach shares similarities with RAFT and PIPs, in that this motion is refined iteratively via a convolutional-GRU, and the network can accommodate a series of successive images as input (more than 2). An extensive series of experiments underline the approach's effectiveness against existing state-of-the-art in optical flow and keypoint tracking. Additionally, novel datasets specifically dedicated to the task at hand have been created.\n\nSTRENGTHS:\n* The idea of estimating continuous optical flow from a neural network seems relatively novel. \n* The performance reported in the manuscript is very competitive.\n\nWEAKNESSES:\n* While the overall editorial quality of the paper is acceptable, it still contains a large number of typos, and certain parts of the paper would benefit from additional proofreading. The flow of the paper sometimes lacks smoothness and transitions.\n* The literature review is rather short and fails to position the paper with respect to other approaches. It gives the impression that the paper is poorly motivated. It would be helpful to underline more clearly the differences with other techniques and why this approach is substantially better.\n* Similar to the previous comment, it would be interesting to list what downstream applications could benefit from this technique (3D? Tracking? Stabilization?).\n* One of my main gripes regarding this manuscript is the lack of scalability. Due to its ability to input multiple frames, the network requires a significant amount of memory. This limits its applicability and raises the question of fairness when compared with less memory-intensive techniques. \n* Another important point to address is the lack of clear contribution. In the reviewer's opinion, the main novelty of this paper is the type of output as B-splines, while the rest of the work is strongly inspired by other techniques.\n* What are the scalability and clear limitations of the technique? How does it perform under very small or very large motions? It would be interesting to investigate such specific use cases. \n* In Table 3, \"Neural-ODE\" is written, but in the method section, it is written as \"Basic\" while all other techniques are using ODE.\n\nQUESTIONS:\nI have expressed most of my questions in the previous section of this review. Despite all the shortcomings mentioned earlier, I found this work new and interesting, and therefore, I would like to issue a relatively positive opinion regarding its acceptance. However, the reviewer would like to specify that he has not been working on such a topic lately and has low confidence.\n\nLIMITATIONS:\nSee previous sections.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

comments: SUMMARY:
This paper proposed a continuous parametric optical flow estimation algorithm with B-spline temporal trajectory representation and ODE-ConvGRU based feature extraction. Experiment has been done on both synthetic and real-world dataset and the proposed continuous parametric method performs better than traditional flow-based and point-tracking based methods.

STRENGTHS:
The idea of estimating continuous parametric flow is interesting and the achieved experimental result is promising. The presentation is clear and easy to follow. The ablation study clearly shows the benefits of B-spline representation and ODE-ConvGRU feature extraction.

WEAKNESSES:
1. It is somewhat unclear how to specify the flexibility of the trajectories, as in Table 3, different number of control points may affect the performance significantly, and in real scenarios, different objects in a scene may have different degree of motion complexities, it is not clear whether a single choice of B-spline number can handle this?
2. In table 2, the 'TRMSE' lines in 'PIPS' and 'ODE-6spline' under Vid-DAVIS dataset is exactly the same, this is problematic since their 'ADE_All' metrics are different, why?

QUESTIONS:
1. How about the computational complexity / running speed of the proposed algorithm? how about the memory cost?

LIMITATIONS:
I donot see any potential negative societal impact
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper suggests a new model to pixel wise compute temporally continuous optical flow by using B-splines. The input to the neural model are sequences of images, the output are N 2D control points for each pixel of the spline model. During training, the input is sampled from the dataset at varying time instants and by varying number of samples. The model architecture builds on known design patterns such as CNN feature encoding, iterative optimisation (search) by using a recurrent GRU network, by using correlation pyramid to compute motion features. New elements are the ODE modelling of the continuous trajectories and the overall decoding (regression) of the control points for each pixel.\n\nSTRENGTHS:\nThe paper introduces a model able to continuously capture motion in time. Applications like video editing could profit of such capabilities. The combination of neural optimisation and neural ODEs for optical flow computation seems novel to me.\n\nWEAKNESSES:\nSome insecurity in the argumentation, e.g. line 28: \"real motion follows ... principles\" - is there some motion of interest that does not follow physical principles?\n\nThe related work should be structured by the motion model, e.g. non-parametric motion, parametric motion. GMA is an attempt to overcome the limitations of non-parametric motion which is not clear by reading the text. The line of work called tracking (started with Lucas-Kanade) is non-parametric but sparse estimation of the apparent motion whereas particle video (Sand, Teller) tried to find a model in between dense optical flow and KLT.\n\nline 160: xhi is not shown in Figure 2\n\nline 166: computing correlation between features at reference time and target time is limited, e.g. think of in plane rotations.\n\nThere are typos in the text, e.g. line 254\n\nline 233, Eq. 10: I do not understand \"sampled point trajectories\". Why is N_s needed in the metrics?\n\nThe spline is also continuous in the image, not only in time, but the ground truth available in the datasets  is discretly sampled, so how do you match an estimate F(t_k) to F* in Eq. 10?\n\nQUESTIONS:\nThe model infers for all pixels in the reference image a 2NxWxH tensor with the control points of the pixel's B-splines. How do you infer the flow vector of all the pixels in the successive images (2nd, 3rd, ...) in the sequence?\n\nMight trajectories of different pixels in the reference (1st) image collapse to a single pixel in some image in the sequence?\n\nLIMITATIONS:\nA first assessment of the impact has not been given by the authors. Even fundamental work on motion analysis & tracking should project potential impacts on foreseen applications and uses or policies such as the UN SDGs. Focus on computer vision as a field and try to explain how your research might affect the field in the next years. Try then to explain how the interaction of computer vision with society might be affected by your research, e.g. a seamlessly tracking combined with generative models might in future it even harder to distinguish fake video from the real.\n\nGive real world examples why the B-spline model is limiting, why you need parametric motion and how you believe this dilemma could be overcome in future. The B-spline motion is smooth and introduces motion consistency which is in some cases a disadvantage, e.g. when it comes to abrupt motion cases.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents a parametric representation of dense and continuous pixel motion over arbitrary time intervals. The  ``continuous parametric flow`` concept is interesting. However, one of the core technique contributions is encoding the image with the ODE-ConvGRU, which is not closely related to the  ``continuous parametric flow`` and simply replaces CNNs used in PIPs and RAFT.

STRENGTHS:
1. The concept ``continuous parametric flow`` is interesting.
2. ODE-ConvGRU is a stronger image feature encoder compared with CNNs. 
3. The B-spline-based flow interpretation is novel.

WEAKNESSES:
1. The comparison is unfair. The proposed model is trained with $N_{gt}=8$ but the PIPs is only trained with 4-frames. Can you provide the performance of an 8-frame PIPs model and the officially released PIPs model?
2. What does the ``implicit`` mean in the ``implicit feature representation``? For me, it's just a feature encoded by the ODE-ConvGRU instead of a CNN. I don't think it's necessary to introduce the ``implicit`` concept.
3. What's the relationship between the continuous flow with ODE-ConvGRU?
4. The two-stage evaluation is not novel, and should not be regarded as a contribution.
5. Figure 2 should be improved. The symbols in the ``cost volume`` and ``correlation pyramid`` regions are not well aligned. The legend of the concatenation is weird.

QUESTIONS:
See Weakness.

LIMITATIONS:
Limited by the chosen parametric model, the predicted point trajectory is challenging to express complex motions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
A temporally continuous parametric optical flow method, based on B-splines, is presented. The proposed network takes as input L frames (and timestamps) and outputs a tensor of size 2NxHxW, i.e N control points for each pixel. In practice N=6. The network architecture relies on neural ODE, ConvGRU and multi-time correlation pyramid. The proposed method is evaluated against RAFT and PIPs and a synthetic dataset (which is a contribution of this paper) and two real datasets (Vid-DAVIS and Vid-Kinetics). An ablation study is also presented.

STRENGTHS:
1. The idea of using B-splines to model a temporally continuous parametric optical flow, is simple and efficient.
2. The proposed architecture is not trivial and well thought out.
3. The method outperforms both RAFT and PIPs.

WEAKNESSES:
1. The paper contains many typos, e.g. l.152 feed - > fed, l. 154 nerual, l. 251 will, etc.).
2. Some results are strange, for instance, in Table 2, PIPs and ODE-6spline have the exact same TRMSE on Vid-DAVIS.
3. In Table 1, PIPs outperforms ODE-6spline in terms of TRMSE at 16f on the Query-Stride set (11.27 vs 11.38) but it is not in bold.
4. Using ODE-ConvGRU does not seem new (it is said it was proposed in [49]).

QUESTIONS:
Please answer to the above "weaknesses". I am currently recommending to reject the paper as I feel the contribution is rather limited for NeurIPS.

LIMITATIONS:
.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05