SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yushan Zhang Johan Edstedt Bastian Wandt Per-Erik Forssén Maria Magnusson Michael Felsberg Linkoping University {firstname.lastname}@liu.se
We tackle the task of scene flow estimation from point clouds. Given a source and a target point cloud, the objective is to estimate a translation from each point in the source point cloud to the target, resulting in a 3D motion vector field. Previous dominant scene flow estimation methods require complicated coarse-to-fine or recurrent architectures as a multi-stage refinement. In contrast, we propose a significantly simpler single-scale one-shot global matching to address the problem. Our key finding is that reliable feature similarity between point pairs is essential and sufficient to estimate accurate scene flow. We thus propose to decompose the feature extraction step via a hybrid local-global-cross transformer architecture which is crucial to accurate and robust feature representations. Extensive experiments show that the proposed Global Matching Scene Flow (GMSF) sets a new state-of-the-art on multiple scene flow estimation benchmarks. On FlyingThings3D, with the presence of occlusion points, GMSF reduces the outlier percentage from the previous best performance of  $27.4\%$  to  $5.6\%$ . On KITTI Scene Flow, without any fine-tuning, our proposed method shows state-of-the-art performance. On the Waymo-Open dataset, the proposed method outperforms previous methods by a large margin. The code is available at https://github.com/ZhangYushan3/GMSF.
Scene flow estimation is a popular computer vision problem with many applications in autonomous driving [31] and robotics [39]. With the development of optical flow estimation and the emergence of numerous end-to-end trainable models in recent years, scene flow estimation, as a close research area to optical flow estimation, takes advantage of the rapid growth. As a result, many end-to-end trainable models have been developed for scene flow estimation using optical flow architectures [27, 46, 55]. Moreover, with the growing popularity of Light Detection and Ranging (LiDAR), the interest has shifted to computing scene flow from point clouds instead of stereo image sequences. In this work, we focus on estimating scene flow from 3D point clouds.
One of the challenges faced in scene flow estimation is fast movement. Previous methods usually employ a complicated multi-stage refinement with either a coarse-to-fine architecture [55] or a recurrent architecture [46] to address the problem. We instead propose to solve scene flow estimation by a single-scale one-shot global matching method, that is able to capture arbitrary correspondence, thus, handling fast movements. Occlusion is yet another challenge faced in scene flow estimation. We take inspiration from an optical flow estimation method [56] to enforce smoothness consistency during the matching process.
The proposed method consists of two stages: feature extraction and matching. A detailed description is given in Section 3. To extract high-quality features, we take inspiration from the recently dominant transformers [47] and propose a hybrid local-global-cross transformer architecture to learn accurate and robust feature representations. Both local and global-cross transformers are crucial for our approach as also shown experimentally in Section 4.5. The global matching process, including
estimation and refinement, is guided solely by feature similarity matrices. First, scene flow is calculated as a weighted average of translation vectors from each source point to all target points under the guidance of a cross-feature similarity matrix. Since the matching is done in a global manner, it can capture short-distance correspondences as well as long-distance correspondences and, therefore, is capable of dealing with fast movements. Further refinement is done under the guidance of a self-feature similarity matrix to ensure scene flow smoothness in areas with similar features. This allows to propagate the estimated scene flow from non-occluded areas to occluded areas, thus solving the problem of occlusions.
To summarize, our contributions are: (1) A hybrid local-global-cross transformer architecture is introduced to learn accurate and robust feature representations of 3D point clouds. (2) Based on the similarity of the hybrid features, we propose to use a global matching process to solve the scene flow estimation. (3) Extensive experiments on popular datasets show that the proposed method outperforms previous scene flow methods by a large margin on FlyingThings3D [30] and Waymo-Open [44] and achieves state-of-the-art generalization ability on KITTI Scene Flow [31].
2.1 Scene Flow
Scene flow estimation [23] has developed quickly since the introduction of the KITTI Scene Flow [31] and FlyingThings3D [30] benchmarks, which were the first benchmarks for estimating scene flow from stereo videos. Many scene flow methods [1, 29, 31, 37, 40, 48, 58] assume that the objects in a scene are rigid and decompose the estimation task into subtasks. These subtasks often involve first detecting or segmenting objects in the scene and then fitting motion models for each object. In autonomous driving scenes, these methods are often effective, as such scenes typically involve static backgrounds and moving vehicles. However, they are not capable of handling more general scenes that include deformable objects. Moreover, the subtasks introduce non-differentiable components, making end-to-end training impossible without instance-level supervision.
Recent work in scene flow estimation mostly takes inspiration from the related task of optical flow [9, 16, 41, 45] and can be divided into several categories: encoder-decoder methods [14, 27] that solve the scene flow by an hourglass architecture neural network, multi-scale methods [3, 20, 55] that estimate the motion from coarse to fine scales, or recurrent methods [17, 46, 53] that iteratively refine the estimated motion. Other approaches [19, 34] try to solve the problem by finding soft correspondences on point pairs within a small region. In order to reduce the annotation requirement, some methods focus on runtime optimization [22, 18], prior assumptions [21], or even without the need for training data [4].
Encoder-decoder Methods: Flownet [9] and Flownet2.0 [16], were the first methods to learn optical flow end-to-end with an hourglass-like model, and inspired many later methods. Flownet3D [27] first employs a set of convolutional layers to extract coarse features. A flow embedding layer is introduced to associate points based on their spatial localities and geometrical similarities on a coarse scale. A set of upscaling convolutional layers is then introduced to upsample the flow to the high resolution. FlowNet3D++ [52] further incorporates point-to-plane distance and angular distance as additional geometry constraints to Flownet3D [27]. HPLFlowNet [14] employs Bilateral Convolutional Layers (BCL) to restore structural information from unstructured point clouds. Following the hourglass-like model, DownBCL, UpBCL, and CorrBCL operations are proposed to restore information from each point cloud and fuse information from both point clouds.
Coarse-to-fine Methods: PointPWC-Net [55] is a coarse-to-fine method for scene flow estimation using hierarchical feature extraction and warping, which is based on the optical flow method PwC-Net [41]. A novel learnable Cost Volume Layer is introduced to aggregate costs in a patch-to-patch manner. Additional self-supervised losses are introduced to train the model without ground-truth labels. Bi-PointFlowNet [3] follows the coarse-to-fine scheme and introduces bidirectional flow embedding layers to learn features along both forward and backward directions. Based on previous methods [27, 55], HCRF-Flow [20] introduces a high-order conditional random fields (CRFs) based relation module (Con-HCRFs) to explore rigid motion constraints among neighboring points to force point-wise smoothness and within local regions to force region-wise rigidity. FH-Net [7] proposes a
fast hierarchical network with lightweight Trans-flow layers to compute key points flow and inverse Trans-up layers to upsample the coarse flow based on the similarity between sparse and dense points.
Recurrent Methods: FlowStep3D [17], is the first recurrent method for non-rigid scene flow estimation. They first use a global correlation unit to estimate an initial flow at the coarse scale, and then update the flow iteratively by a Gated Recurrent Unit (GRU). RAFT3D [46] also adopts a recurrent framework. Here, the objective is not the scene flow itself but a dense transformation field that maps each point from the first frame to the second frame. The transformation is then iteratively updated by a GRU. PV-RAFT [53] presents point-voxel correlation fields to capture both short-range and long-range movements. Both coarse-to-fine and recurrent methods take the cost volume as input to a convolutional neural network for scene flow prediction. However, these regression techniques may not be able to accurately capture fast movements, and as a result, multi-stage refinement is often necessary. On the other hand, we propose a simpler architecture that solves scene flow estimation in a single-scale global matching process with no iterative refinement.
Soft Correspondence Methods: Some work poses the scene flow estimation as an optimal transport problem. FLOT [34] introduces an Optimal Transport Module that gives a dense transport plan informing the correspondence between all pairs of points in the two point clouds. Convolutional layers are further applied to refine the scene flow. SCTN [19] introduces a voxel-based sparse convolution followed by a point transformer feature extraction module. Both features, from convolution and transformer, are used for correspondence computation. However, these methods involve complicated regularization and constraints to estimate the optimal transport from the correlation matrix. Moreover, the correspondences are only computed within a small neighboring region. We instead follow the recent global matching paradigm [10, 56, 64] and solve the scene flow estimation with a global matcher that is able to capture both short-distance and long-distance correspondence.
Runtime Optimization, Prior Assumptions, and Self-supervision: Different from the proposed method, which is fully supervised and trained offline, some other work focuses on runtime optimization, prior assumptions, and self-supervision. Li et al. [22] revisit the need for explicit regularization in supervised scene flow learning. The deep learning methods tend to rely on prior statistics learned during training, which are domain-specific. This does not guarantee generalization ability during testing. To this end, Li et al. propose to rely on runtime optimization with scene flow prior as strong regularization. Based on [22] Lang et al. [18] propose to combine runtime optimization with self-supervision. A correspondence model is first trained to initialize the flow. Refinement is done by optimizing the flow refinement component during runtime. The whole process can be done under self-supervision. Pontes et al. [33] propose to use the graph Laplacian of a point cloud to force the scene flow to be "as rigid as possible". Same as in [22], this constraint can be optimized during runtime. Li et al. [21] propose a self-supervised scene flow learning approach with local rigidity prior assumption for real-world scenes. Instead of relying on point-wise similarities for scene flow estimation, region-wise rigid alignment is enforced. Most recently, Chodosh et al. [4] identify the main challenges of LiDAR scene flow estimation as estimating the remaining simple motions after removing the dominant rigid motion. By combining ICP, rigid assumptions, and runtime optimization, they achieve state-of-the-art performance without any training data.
2.2 Point Cloud Registration
Related to scene flow estimation, there are some correspondence-based point cloud registration methods. Such methods separate the point cloud registration task into two stages: finding the correspondences and recovering the transformation. PPFNet [6] and PPF-FoldNet [5] proposed by Deng et al. focus on finding sparse corresponding 3D local features. Gojcic et al. [12] propose to use voxelized smoothed density value (SDV) representation to match 3D point clouds. These methods only compute sparse correspondences and are not capable of handling dense correspondences required in scene flow tasks. More related works are CoFiNet [59] and GeoTransformer [36], both of which involve finding dense correspondences employing transformer architectures. Yu et al. in CoFiNet [59] propose a detection-free learning framework and find dense point correspondence in a coarse-to-fine manner. Qin et al. in GeoTransformer [36] further improve the accuracy by leveraging the geometric information. RoITr [60] introduces a Rotation-Invariant Transformer to disentangle the geometry and poses, and tackle point cloud matching under arbitrary pose variations. PEAL [61] introduces the Prior Embedded Explicit Attention Learning model (PEAL), and for the first time explicitly injects
Figure 1: Method Overview. We propose a simple yet powerful method for scene flow estimation. In the first stage (see Section 3.1) we propose a strong local-global-cross transformer architecture that is capable of extracting robust and highly localizable features. In the second stage (see Section 3.2), a simple global matching yields the flow. In comparison to previous work, our approach is significantly simpler, while achieving state-of-the-art results.
overlap prior into Transformer to solve point cloud registration under low overlap. However, the goal of point cloud registration is not to estimate the translation vectors for each of the points, which makes our work different from these approaches.
2.3 Transformers
Transformers were first proposed in [47] for translation tasks with an encoder-decoder architecture using only attention and fully connected layers. Transformers have been proven to be efficient in sequence-to-sequence problems, well-suited to research problems involving sequential and unstructured data. The key to the success of transformers over convolutional neural networks is that they can capture long-range dependencies within the sequence, which is very important, not only in translation but also in many other tasks e.g. computer vision [8], audio processing [24], recommender systems [42], and natural language processing [54].
Transformers have also been explored for point clouds [28]. The coordinates of all points are stacked together directly as input to the transformers. For the tasks of classification and segmentation, PT [63] proposes constructing a local point transformer using k-nearest-neighbors. Each of the points would then attend to its nearest neighbors. PointASNL [57] uses adaptive sampling before the local transformer, and can better deal with noise and outliers. PCT [15] proposes to use global attention and results in a global point transformer. Pointformer [32] proposes a new scheme where first local transformers are used to extract multi-scale feature representations, then local-global transformers are used as cross attention to multi-scale features, finally, a global transformer captures context-aware representations. Point-BERT [62] is originally designed for masked point modeling. Instead of treating each point as one data item, they group the point cloud into several local patches. Each of these sub-clouds is tokenized to form input data.
Previous work on scene flow estimation exploits the capability of transformers for feature extraction either using global-based transformers in a local matching paradigm [19] or local-based transformers in a recurrent architecture [11]. Instead, we propose to leverage both local and global transformers to learn a feature representation for each point on a single scale. We show that high-quality feature representations are the fundamental property that is needed for scene flow estimation when formulated as a global matching problem.
Given two point clouds  $\mathcal{X}_1\in \mathbb{R}^{N_1\times 3}$  and  $\mathcal{X}_2\in \mathbb{R}^{N_2\times 3}$  with only position information, the objective is to estimate the scene flow  $V\in \mathbb{R}^{N_1\times 3}$  that maps each point in the source point cloud to the target point cloud. Due to the sparse nature of the point clouds, the points in the source and the target point clouds do not necessarily have a one-to-one correspondence, which makes it difficult to formulate scene flow estimation as a dense matching problem. Instead, we show that learning a cross-feature similarity matrix of point pairs as soft correspondence is sufficient for scene flow estimation. Unlike many applications based on point cloud processing which need to acquire a high-level understanding, e.g. classification and segmentation, scene flow estimation requires a low-level understanding to distinguish geometry features between each element in the point clouds. To this end, we propose a transformer architecture to learn high-quality features for each point. The proposed method consists of two core components: feature extraction (see Section 3.1) and global matching (see Section 3.2). The overall framework is shown in Figure 1.
Tokenization: Given the 3D point clouds  $\mathcal{X}_1, \mathcal{X}_2$ , each point  $x_i$  is first tokenized to get summarized information of its local neighbors. We first employ an off-the-shelf feature extraction network DGCNN [51] to map the input 3D coordinate  $x_i$  into a high dimensional feature space  $x_i^h$  conditioned on its nearest neighbors  $x_j$ . Each layer of the network can be written as
$$
x _ {i} ^ {h} = \max  _ {x _ {j} \in \mathcal {N} (i)} h \left(x _ {i}, x _ {j} - x _ {i}\right), \tag {1}
$$
where  $h$  represents a sequence of linear layers, batch normalization, and ReLU layers. The local neighbors  $x_{j} \in \mathcal{N}(i)$  are found by a k-nearest-neighbor (knn) algorithm. Multiple layers are stacked together to get the final feature representation.
For each point, local information is incorporated within a small region by applying a local Point Transformer [63] within  $x_{j} \in \mathcal{N}(i)$ . The transformer is given by
$$
x _ {i} ^ {l} = \sum_ {x _ {j} \in \mathcal {N} (i)} \gamma \left(\varphi_ {l} \left(x _ {i} ^ {h}\right) - \psi_ {l} \left(x _ {j} ^ {h}\right) + \delta\right) \odot \left(\alpha_ {l} \left(x _ {j} ^ {h}\right) + \delta\right), \tag {2}
$$
where the input features are first passed through linear layers  $\varphi_l, \psi_l$ , and  $\alpha_l$  to generate query, key and value.  $\delta$  is the relative position embedding that gives information about the 3D coordinate distance between  $x_i$  and  $x_j$ .  $\gamma$  represents a Multilayer Perceptron consisting of two linear layers and one ReLU nonlinearity. The output  $x_i^l$  is further processed by a linear layer and a residual connection from  $x_i^h$ .
Global-cross Transformer: Transformer blocks are used to process the embedded tokens. Each of the blocks includes self-attention followed by cross-attention [38, 43, 47, 56].
The self-attention is formulated as
$$
x _ {i} ^ {g} = \sum_ {x _ {j} \in \mathcal {X} _ {1}} \left\langle \varphi_ {g} \left(x _ {i} ^ {l}\right), \psi_ {g} \left(x _ {j} ^ {l}\right) \right\rangle \alpha_ {g} \left(x _ {j} ^ {l}\right), \tag {3}
$$
where each point  $x_{i} \in \mathcal{X}_{1}$  attends to all the other points  $x_{j} \in \mathcal{X}_{1}$ , same for the points  $x_{i} \in \mathcal{X}_{2}$ . Linear layers  $\varphi_{g}, \psi_{g}$ , and  $\alpha_{g}$  generate the query, key, and value.  $\langle , \rangle$  denotes a scalar product. Linear layer, layer norm, and skip connection are further applied to complete the self-attention module.
The cross-attention is given as
$$
x _ {i} ^ {c} = \sum_ {x _ {j} \in \mathcal {X} _ {2}} \left\langle \varphi_ {c} \left(x _ {i} ^ {g}\right), \psi_ {c} \left(x _ {j} ^ {g}\right) \right\rangle \alpha_ {c} \left(x _ {j} ^ {g}\right), \tag {4}
$$
where each point  $x_{i} \in \mathcal{X}_{1}$  in the source point cloud attends to all the points  $x_{j} \in \mathcal{X}_{2}$  in the target point cloud, and vice versa. A Feedforward network with multi-layer perceptron and layer norm is applied to aggregate information to the next transformer block. The detailed architecture of our proposed local-global-cross transformer is presented in Figure 2. The feature matrices  $F_{1} \in \mathbb{R}^{N_{1} \times d}$  and  $F_{2} \in \mathbb{R}^{N_{2} \times d}$  are formed as the concatenation of all the output feature vectors from the final transformer block, where  $N_{1}$  and  $N_{2}$  are the number of points in the two point clouds and  $d$  is the feature dimension.
Figure 2: Transformer Architecture. Detailed local (left), global (middle), and cross (right) transformer architecture. The local transformer incorporates attention within a small number of neighbors. The global transformer is applied on the source and target points separately and incorporates attention on the whole point clouds. The cross transformer further attends to the other point cloud and gets the final representation conditioned on both the source and the target.
3.2 Global Matching
Feature similarity matrices are the only information that is needed for an accurate scene flow estimation. First, the cross similarity matrix between the source and the target point clouds is given by multiplying the feature matrices  $F_{1}$  and  $F_{2}$  and then normalizing over the second dimension with softmax to get a right stochastic matrix,
$$
C _ {\text {c r o s s}} = \frac {F _ {1} F _ {2} ^ {T}}{\sqrt {d}}, \tag {5}
$$
$$
M _ {\text {c r o s s}} = \operatorname {s o f t m a x} \left(C _ {\text {c r o s s}}\right), \tag {6}
$$
where each row of the matrix  $M_{\mathrm{cross}} \in \mathbb{R}^{N_1 \times N_2}$  is the matching confidence from one point in the source point cloud to all the points in the target point cloud. The second similarity matrix is the self similarity matrix of the source point cloud, given by
$$
C _ {\text {s e l f}} = \frac {W _ {q} \left(F _ {1}\right) W _ {k} \left(F _ {1}\right) ^ {T}}{\sqrt {d}}, \tag {7}
$$
$$
M _ {\text {s e l f}} = \operatorname {s o f t m a x} \left(C _ {\text {s e l f}}\right), \tag {8}
$$
which is a matrix multiplication of the linearly projected point feature  $F_{1}$ .  $W_{q}$  and  $W_{k}$  are learnable linear projection layers. Each row of the matrix  $M_{\mathrm{self}} \in \mathbb{R}^{N_1 \times N_1}$  is the feature similarity between one point in the source point cloud to all the other points in the source point cloud. Given the point cloud coordinates  $\mathcal{X}_1 \in \mathbb{R}^{N \times 3}$  and  $\mathcal{X}_2 \in \mathbb{R}^{N \times 3}$ , the estimated matching point  $\hat{\mathcal{X}}_2$  in the target point cloud is computed as a weighted average of the 3D coordinates based on the matching confidence
$$
\hat {\mathcal {X}} _ {2} = M _ {\text {c r o s s}} \mathcal {X} _ {2}. \tag {9}
$$
The scene flow is computed as the movement between the matching points
$$
\hat {V} _ {\text {i n t e r}} = \hat {\mathcal {X}} _ {2} - \mathcal {X} _ {1}. \tag {10}
$$
The estimation procedure can also be seen as a weighted average of the translation vectors between point pairs, where a softmax ensures that the weights sum to one.