SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jianqin Luo*
Zhexiong Wan*
Yuxin Mao
Yuchao Dai†
Northwestern Polytechnical University, Xi'an, China Shaanxi Key Laboratory of Information Acquisition and Processing
In this paper, we present continuous parametric optical flow, a parametric representation of dense and continuous motion over arbitrary time interval. In contrast to existing discrete-time representations (i.e. flow in between consecutive frames), this new representation transforms the frame-to-frame pixel correspondences to dense continuous flow. In particular, we present a temporal-parametric model that employs B-splines to fit point trajectories using a limited number of frames. To further improve the stability and robustness of the trajectories, we also add an encoder with a neural ordinary differential equation (NODE) to represent features associated with specific times. We also contribute a synthetic dataset and introduce two evaluation perspectives to measure the accuracy and robustness of continuous flow estimation. Benefiting from the combination of explicit parametric modeling and implicit feature optimization, our model focuses on motion continuity and outperforms the flow-based and point-tracking approaches for fitting long-term and variable sequences.
Motion estimation, which characterizes the movement of targets over time, is crucial for video understanding tasks like keypoint tracking [1] or video interpolation [2]. It also provides accurate priors for 3D tasks such as non-rigid structure-from-motion [3, 4] and simultaneous localization and mapping [5]. Fine-grained motion understanding is necessary to achieve pixel-level correspondence, which is usually studied as an optical flow estimation problem. Some traditional methods [6, 7] based on variational optimization or recent deep methods focusing on the Tracking-Any-Point (TAP) task [8, 9] both attempt to estimate long-range flow for tracking points across multi-frames.
While these approaches have made significant progress in modeling correspondences, there still exist two obstacles to explaining and predicting practical motion, as shown in Fig. 1. First, the point trajectories generated by these approaches only reflect the possible locations at certain moments instead of real motion trends because the frame-to-frame correspondence is temporally discrete and lacks enough inter-frame information to describe continuous point motion. Second, optical flow-based methods could achieve dense but short-term point mapping while recent point trackers tend to strengthen long-term tracking for sparse points. Neither can efficiently realize dense and long-term tracking simultaneously.
In this paper, we aim to estimate the dense and continuous pixel correspondence with the continuous flow that provides the moving trajectory from reference time to arbitrary target times within the whole
† Corresponding author (daiyuchao@nwpu.edu.cn).
$\ddagger$  Project page: https://npucvr.github.io/CPFlow.
(a) Discrete-time Optical Flow [12, 13]
(b) Point Tracking [9]
(c) Continuous Optical Flow (Ours)
Figure 1: Comparison between continuous optical flow with spatially dense and temporally continuous attributes with discrete-time flow or pixel tracking.
video. We utilize a learnable parametric model to explicitly describe the continuous flow trajectory, which avoids some inconsistent and abrupt displacements compared to the implicit counterparts directly predicting the final result. Motivated by [10], we embed the continuous property into our end-to-end optimization pipeline to enhance continuity in feature space. By combining the powerful representational ability and explicit motion pattern, our method is more appropriate for solving continuous problems. To evaluate the continuous flow estimation, we introduce two evaluation perspectives, including continuous motion with invisible frames at specific timescales and robustness validation at variable timescales. We also use Kubric [11] to create a synthetic dataset with dense and long trajectories for experiments.
Our main contributions are summarized as:
- We present continuous parametric optical flow, a temporal parametric representation of dense and continuous motion over arbitrary time intervals.
- We utilize Neural ODE and ConvGRU to aggregate the spatial-temporal information and enhance the continuity of features among target times.
- We create a large-scale synthetic dataset with dense and long-term trajectories and propose a two-stage evaluation pipeline to measure the accuracy and robustness of continuous flow estimation.
2.1 Optical Flow Estimation
Optical flow estimation is a fundamental task in computer vision, and existing optimization-based methods [14, 15, 16, 17] have made explorations in challenging situations such as large motions [18, 19] and occlusions[20, 21]. While early learning-based methods take an encoder-decoder [22] or coarse-to-fine [23, 24] structure, recent methods, such as RAFT [12] and GMA [25], usually take an iterative update pipeline. These methods usually take procedures of supervised training on simulated data [25, 13] or unsupervised training on real data [26, 27].
Unlike the above common two-frame input, some methods [26, 28] use multi-frames to aggregate additional flows to tackle the occlusion. MFSF [6] aims to establish the long-range correspondence of multi-frame flow by a linear combination of trajectory basis. SlowFlow [29] links adjacent flows to longer trajectories by constraining brightness, smoothness, and visibility. Gehrig et al. [30] proposes to use event data to build motion trajectories for fast and non-linear motion in a short period.
2.2 Feature and Point Tracking
Among the self-supervised feature tracking methods, TimeCycle [31] uses cycle consistency as the primary constraint for patch-level extraction and matching. CRW [32] designs a special palindrome arrangement to introduce cycle loss. VFS [33] introduces contrastive learning to optimize frame-level matching. Instead of implicit exploration in unlabeled data, supervised methods [34, 35] rely on ground truth correspondence to learn a general feature representation. Although these methods perform well on general tasks like keypoint tracking or semi-supervised video segmentation, pixel-level matching remains unsolved because image or patch embedding seems insufficient for specific point refinement.
For long-term point tracking, the key problem is how to match the spatial-temporal features of the long-range correspondences. PIPs [9] proposes to extract query point features for pixel tracking and occlusion estimation with iterative updates, motivated by Particle Video [36]. TAP-Net [8] uses a flow-based method to assist annotators to track sparse points selected by random sampling and generate point trajectories with longer duration.
2.3 Parametric Motion Hypothesis
Rigid motion modeling was a fashion for early work before various modern learning techniques. As for spatial modeling, some representative methods [37, 38] use robust parametric motion like affine transformation or based on over-parameterization [39, 40] to estimate flow. Yang and Li [41] propose a piecewise, 8-Dof homography model to fuse multiple possible parametric motions. Besides, layered models [42, 43] also perform competitively with a few overlapping motion layers that balance the need for smoothness and boundary separation.
For temporal consistency, the linear combination of motion basis [6, 44, 45] is a common method for trajectory fitting in the time axis. These pre-defined bases like DCT [6] or Lagrange [44] can convert the continuous time domain into the discrete analysis. With different weight assignments for basis groups, this modeling approach is convenient to implement for deep paradigm [45]. However, limited by the sampling range, these models are not flexible for video clips with different frame numbers. Another alternative is to perform temporally continuous curve fitting, including linear [46], quadratic [47], or cubic [48] curve hypotheses with polynomial modeling.
In addition, complex motion trajectories can be represented by adaptive curves with a few control points like B-splines [49] or Bezier [30]. Although most of these parametric hypotheses were first introduced in previous optimization methods, modern methods can still take a deep network as the process of motion parameterization by regressing a set of coefficients or control points.
3 Parametric Modeling
3.1 Preliminaries of Continuous Flow
To describe the continuous flow that expresses the motion from the reference time to any target time, we define a triplet  $(t,u,v)$  as a query of flow trajectory decoupled into two components along horizontal and vertical direction for corresponding pixel  $(x,y)$ ,  $t$  is sampled from normalized time  $T\in [0,1]$ . For brevity, the reference time is zero. Instead of directly fitting the pixel position, we choose to estimate the pixel displacements and the start point of flow trajectory  $(u,v) = (0,0)$ . Note that although we can only use finite labels to evaluate our model, we introduce two evaluation perspectives to measure the performance under diverse situations.
Stage 1: Blind Estimation. Given a specific number of frames at sampled timestamps with a fixed timescale as input, the model should predict the target flow trajectory at unsampled times.
Stage 2: Temporal Adaptation. The pre-trained model on a fixed training range should adapt to diverse sequences with different timescales.
3.2 Parametric Curve for Continuous Flow Modeling
Given  $N$  control points  $\{P_i = (x_i,y_i)\}^N\in \mathbb{R}^{N\times 2}$  and basis functions  $B_{i,k}(t)$  with degree  $k$ , the point displacement trajectory  $F(t)$  is formulated as:
$$
F (t) = \sum_ {i = 0} ^ {N - 1} B _ {i, k} (t) P _ {i}. \tag {1}
$$
For any specific time  $t$ , the basis functions return a group of weights to determine the contributions for the curve shape of each control point. The parametric curve we employ is a cubic B-spline formed by a few learnable control points and spline basis functions. In our work, the coordinates of  $N$  control points serve as model outputs, whereas the normalized knots  $\{t_i\}$  are manually specified hyperparameters for local optimization in the temporal range. The detailed derivation of basis functions is referred to Cox-de Boor recursion [50, 51]:
$$
B _ {i, 0} (t) = \left\{ \begin{array}{l l} 1 & t _ {i} \leq t <   t _ {i + 1} \\ 0 & \text {o t h e r w i s e} \end{array} , \right. \tag {2}
$$
$$
B _ {i, k} (t) = \frac {t - t _ {i}}{t _ {i + k} - t _ {i}} B _ {i, k - 1} (t) + \frac {t _ {i + k + 1} - t}{t _ {i + k + 1} - t _ {i + 1}} B _ {i + 1, k - 1} (t). \tag {3}
$$
We assign each time to a single partition to ensure the sum of the weight coefficients of all control points is equal to 1. Knots can be repeated to promote the curve from open to close. In our experiment, a special shape like the clamp seems easier to reach convergence. Benefiting from the potential advantages of parameter design, the cubic B-spline could achieve the combination of global relaxation and fine-tuning on local segments.
The parametric curve is optimized independently for each pixel to build a complete displacement flow field. As a result, our model directly predicts a 3D tensor with the shape of  $2N \times H \times W$  for control points in all trajectories.
4 Optimization
The optimization process consists of four components: multi-frame sampling, implicit continuity enhancement, multi-time correlation pyramids, and iterative control point decoder. According to the order of the input video clip with  $L$  frames, the corresponding timestamps are normalized to [0, 1]. Our framework aims to regress a set of control points with their 2D coordinates for each pixel and to compute the final continuous flow trajectories by Eq. (1).
4.1 Multi-frame Sampling
For a given video clip with  $L$  frames, the model input  $M$  frames are first sampled from the entire sequence as visible moments in the continuous time domain while the remaining  $L - M$  frames are assumed to be invisible. We use the ground-truth displacement fields sampled from all valid moments corresponding to the frames as the supervision of continuous flow trajectories for model training.
During training, we first select the start frame with a sufficient number of visible points from the given video clip. Then we randomly sample a subsequence clip with  $M$  frames with different intervals as model inputs to enhance the generation ability to different motion magnitudes. Notably, this selection process keeps the start and end frames and only randomly selects the intermediate  $M - 2$  frames based on brightness differences.
4.2 Implicit Continuity Enhancement
As shown in Fig. 2, we first extract  $M$  basic feature maps using a shared basic encoder for each of the input  $M$  frames. Similar to RAFT [12], we also feed the reference frame (i.e. the start frame with  $t = 0$ ) into the context encoder for extracting another context feature to assist subsequent motion estimation. Unlike PIPs [9], we introduce ODE-ConvGRU proposed in [2] for spatio-temporal feature learning. This module consists of the Neural ODE for temporal continuity enhancement in feature space and the Conv-GRU for spatio-temporal information aggregation. The Neural ODE formulated
Figure 2: Pipeline of our model, combined with continuous temporal-spatial feature and explicit motion trajectory curves. ODE-ConvGRU enhances spatial features with continuous temporal information and generates powerful representations at arbitrary target moments.
by Eq. (4) includes a derivative estimator  $f$  as an independent ODE unit and an ODE solver, in which the parameters involve the initial value  $h(t_0)$ , the current time  $t$  and input state  $h_{\mathrm{in}}(t)$ . The former uses a learnable fully-connected network  $f$  to fit the feature differentiation at time  $t$ , and the latter integrates to recover the hidden state  $h(t)$  at  $t$  started from  $t_0$ .
$$
h (t) = \mathrm {O D E} (f, t, h _ {\mathrm {i n}} (t)) \quad \left\{ \begin{array}{c} \frac {d h _ {\mathrm {i n}} (t)}{d t} = f \left(h _ {\mathrm {i n}} (t), t\right), \\ h (t) = h _ {\mathrm {i n}} \left(t _ {0}\right) + \int_ {t _ {0}} ^ {t} f \left(h _ {\mathrm {i n}} (t), \tau\right) d \tau . \end{array} \right. \tag {4}
$$
The timestamps corresponding to the input  $M$  frames serve as source times  $\{t_j^{src}\}^M$  to feed into the ODE-ConvGRU module and output features at target times  $\{t_j^{tgt}\}^L$  for correlation construction. The Neural ODE models the features of the entire  $L$ -frame video clip including invisible frames, so it also outputs  $L$  target times.
Our ODE-ConvGRU module consists of three steps. First and foremost, we compute the initial hidden states  $\overline{h}(t_j^{src})$  at source times  $t_j^{src}$  by integrating the derivative of the previous hidden state at  $t_{j-1}^{src}$  solved by network  $f_\theta$ :
$$
\bar {h} \left(t _ {j} ^ {s r c}\right) = \mathrm {O D E} \left(f _ {\theta}, \left\{t _ {j} ^ {s r c} \right\}, \bar {h} \left(t _ {j - 1} ^ {s r c}\right)\right), \tag {5}
$$
where this process starts with  $\overline{h}(t_0^{src}) = 0$ , i.e., an zero feature map. Note that the computation process is backward, which means the original temporal order needs to be reversed.
Besides, given feature  $\chi$  extracted by the basic encoder, we update the initial hidden state  $\overline{h}(t_j^{src})$  at source time  $t_j^{src}$  by aggregating them to generate refined hidden state  $\tilde{h}(t_j^{src})$  with ConvGRU:
$$
\tilde {h} \left(t _ {j} ^ {s r c}\right) = \operatorname {C o n v G R U} \left(\bar {h} \left(t _ {j} ^ {s r c}\right), \chi_ {t _ {j} ^ {s r c}}\right). \tag {6}
$$
Finally, the last output hidden state  $\tilde{h}(0)$  from ConvGRU aggregates valid information from all source moments, and is transformed into a novel initial value  $\hat{h}(0)$  by a small network  $g_{\omega}$ . The derivative is computed by another ODE unit with network  $f_{\phi}$  and evolves into the final hidden state outputs  $h(t_j^{tgt})$  at target time  $t_j^{tgt}$ :
$$
\left\{h \left(t _ {j} ^ {t g t}\right) \right\} ^ {L} = \mathrm {O D E} \left(f _ {\phi}, \left\{t _ {j} ^ {t g t} \right\} ^ {L}, \hat {h} (0)\right). \tag {7}
$$
4.3 Multi-time Correlation Pyramid
To achieve fine-grained matching, we compute the spatial matching feature similarity between different target times. Due to the powerful spatio-temporal representation of ODE-ConvGRU, we can obtain an arbitrary number of feature pairs for continuous flow estimation. Thus we can perform the correlation operation between the reference state  $h(t_0^{tgt})$  and the rest of hidden states  $\{(h(t_j^{tgt})\}_{1}^L$  at target time to construct  $L - 1$  cost volumes.
However, due to limited GPU memory, we still need to introduce sampling to reduce computational complexity while leveraging the continuous representation capability. We make a trade-off by random sampling  $Q$  ( $Q \leq (L - 1)$ ) hidden states to construct  $Q$  multi-time correlations  $C = \{C_q\}^Q$ . We then perform the average pooling operation and obtain the correlation pyramids  $C_q = \{C_q^z\}^Z$  ( $Z$  is the pooling level) for each correlation separately, which covers matching similarities at multiple resolutions for subsequent local lookup.
4.4 Iterative Lookup and Update
Inspired by RAFT [12], we also adopt the lookup operation to search the local matching patches from multi-time correlations. Subsequently, we concatenate the patches with different moments and different pooling levels as a single tensor and feed it into the decoder to regress the residual of control points and compute the continuous optical flow. This procedure of local lookup and flow regression is iteratively executed by adjusting the local search starting point according to the updated control points.
For  $N$  control points  $\{P_i\}^N$ , the update unit outputs  $N$  translations  $\{\Delta P_i\}^N$  and updates iteratively:
$$
P _ {i} ^ {s + 1} = P _ {i} ^ {s} + \Delta P _ {i} ^ {s}, \tag {8}
$$
where  $i$  is the index of the control point and  $s$  is the iteration number for lookup and update.
4.5 Supervision
As there are only discrete point displacement ground truths in the dataset, we can only use them to discretely supervise the predicted continuous optical flow of our model. Assuming that for each point there are  $N_{gt}$  displacement ground truths at different times, we calculate the point displacements on the predicted continuous trajectory with the corresponding times  $\{t_w^{sup}\}^{N_{gt}}$  according to Eq. (1).
The loss function measures the average displacement error at  $N_{gt}$  different times and all visible pixels with a weighted sum of all iterations:
$$
L = \sum_ {i = 1} ^ {S} \gamma^ {S - i} \frac {1}{N _ {g t}} \sum_ {j = 1} ^ {N _ {g t}} \left\| F \left(t _ {j} ^ {\operatorname {s u p}}\right) - D _ {g t} \left(t _ {j} ^ {\operatorname {s u p}}\right) \right\| _ {1}, \tag {9}
$$
where the iteration times is  $S$  and weight is  $\gamma$ .  $F(t_w^{sup})$  is the point displacements at time  $t_w^{sup}$  on the predicted continuous trajectory,  $D_{gt}(t_j^{sup})$  is the corresponding point displacement ground truth. Note that the starting point of the continuous trajectory is 0, so we omit it.
This section consists of four associated parts. We first use Kubric [11] to simulate a relatively dense and longer pixel-tracking dataset for our model training and as a basic benchmark for evaluation. Second, to establish valid baselines, we introduce two approaches [9, 12] to provide direct estimation for sampling moments and parametric assumption to simulate inter-frame motion. Third, we set error metrics for quantitative comparison and show the results on synthetic scenes and real-world scenes. Finally, ablation experiments are implemented for module validity tests.
We train our model on four RTX 3090 GPUs with a batch size of 8. At the training stage, we use the resolution of  $256 \times 256$  and randomly sample 20480 point trajectories as sparse supervisions for better performance. We train for 30,000 steps because more iterations may result in degradation on diverse timescales with a learning rate of 2e-5 and a one-cycle schedule [52]. We repeat multiple times and report median results. We set the numbers of control points  $N = 6$  while the degree  $k = 3$ ,
and the sampling number of frames  $M = 4$ . The number  $Q$  of correlation is 3, and the pooling level  $Z$  is 3. In the supervision process, the  $N_{gt} = 8$  and  $\gamma = 0.8$ . For temporal augmentation, the length of video clips  $L$  is variable but at least 8 frames.
5.1 Dataset
To maintain the same settings as [8, 9], we utilize the Kubric simulator to generate two subsets: query-stride and query-first in our experiment. The video samples in both the two subsets are with 24 frames at  $256 \times 256$  resolution. Points trajectory is sampled from the whole spatial-temporal area in the query-stride version but from the single frame in the query-first version. Query-stride is applied in the training phase for less structured constraints and query-first can be used for dense evaluation.
5.2 Metrics
We use two metrics to measure the accuracy at specific moments and the temporal trajectory smoothness of continuous flow estimation. From a spatial perspective, the average displacement error (ADE) proposed in [9] is applied and divided into visible and occluded parts. To reflect the ability to fit the trajectory during blind time, an additional temporal average is introduced for sampling and non-sampling times. From a temporal perspective, the flow trajectory for a determined point should be robust and smooth between any sequences. We use temporally averaged root-mean-square error (TRMSE) to measure trajectory smoothness, for every candidate pixel  $\mathbf{x}$  in the spatial domain, and the RMSE along the temporal dimension is computed to measure the influence of outliers. After that, the errors of all pixels are averaged.
$$
T R M S E = \frac {1}{N _ {v}} \sum_ {\mathbf {x}} \sqrt {\frac {1}{T} \sum_ {a = 0} ^ {T - 1} \left| \left| F \left(t _ {a}\right) - D _ {g t} \left(t _ {a}\right) \right| \right| ^ {2}}, \tag {10}
$$
where  $N_{v}$  is the number of valid points,  $\mathbf{x} \in \mathbb{R}^{N_s \times 2}$  is the coordinates of valid points.  $T$  is the number of times with ground-truth displacements for each point,  $t_a$  is the corresponding time.
5.3 Baseline
To generate the motion trajectory of each pixel, we choose the flow estimation method RAFT [12] and the point-tracking method PIPs [9] as our baselines. RAFT predicts the optical flow between consecutive frames, and these flows are linked by bilinear interpolation as mentioned in [9]. We use their pre-training model and fine-tune it on our dataset. PIPs is a recent pixel tracking algorithm with promising generalization performance. We retrain PIPs with 4 frames on our dataset and infer the corresponding movements of all pixels during whole input clips to ensure fairness of the comparison.
(b) RAFT
(c)PIPs
(d) Ours
Figure 3: The visualization of dense point trajectories on the Query-First set.