GMSF: Global Matching Scene Flow
Authors: Yushan Zhang
Johan Edstedt
Bastian Wandt
Per-Erik Forssen
Maria Magnusson
Michael Felsberg
Venue: NeurIPS 2023 poster (2023)
Keywords: Scene flow
point clouds
transformers

Abstract: We tackle the task of scene flow estimation from point clouds. Given a source and a target point cloud, the objective is to estimate a translation from each point in the source point cloud to the target, resulting in a 3D motion vector field. Previous dominant scene flow estimation methods require complicated coarse-to-fine or recurrent architectures as a multi-stage refinement. In contrast, we propose a significantly simpler single-scale one-shot global matching to address the problem. Our key finding is that reliable feature similarity between point pairs is essential and sufficient to estimate accurate scene flow. We thus propose to decompose the feature extraction step via a hybrid local-global-cross transformer architecture which is crucial to accurate and robust feature representations. Extensive experiments show that the proposed Global Matching Scene Flow (GMSF) sets a new state-of-the-art on multiple scene flow estimation benchmarks. On FlyingThings3D, with the presence of occlusion points, GMSF reduces the outlier percentage from the previous best performance of 27.4% to 5.6%. On KITTI Scene Flow, without any fine-tuning, our proposed method shows state-of-the-art performance. On the Waymo-Open dataset, the proposed method outperforms previous methods by a large margin. The code is available at https://github.com/ZhangYushan3/GMSF.

GMSF: Global Matching Scene Flow
Yushan Zhang
Johan Edstedt Maria Magnusson Linköping University
Bastian Wandt Michael Felsberg
Per-Erik Forssén
(firstname. 1altamm)....atttttmamm.........
Abstract
We tackle the task of scene flow estimation from point clouds. Given source and a target point cloud, the objective is to estimate a translation from each point the source point cloud to the target, resulting in 3D motion vector field. Previous dominant scene flow estimation methods require complicated coarse-to-fine or recurrent architectures significantly simpler single-scale one-shot global matching to address the problem. Our key finding that reliable feature similarity between point pairs is essential and sufficient to estimate accurate scene flow. We thus propose to decompose the feature extraction step via hybrid bocal-global-coos transformer architecture which is crucial to accurate and robust feature representations. Extensive experiments show that the proposed Global Matching Scene Flow (GMSF) sets new state-of-the- art on multiple scene flow estimation benchmarks. On FlyingTTingss311 with the presence of occlusion points, GMSF reduces the outlier percentage from the previous best performance of 27.4% to 5.6%. On KITTI Scene Flow, without any fine-tuning, our proposed method shows state-of-the-ard performance. On the Waymo-Oper dataset, the proposed method outperforms previous methods by large margin. The code is available at https://gihubb.oom/hangg Gushan3/GMSF
multi-stage refinement. In contrast, we propose a
Introduction
Scene flow estimation is popular computer vision problem with many applications in autonomous driving [31] and robotics [39]. With the development of optical flow estimation and the emergence of numerous end-to-end trainable models in recent years, scene flow estimation, as a close research area to optical flow estimation, takes ad advantage of the rapid growth. As result, many end-to-end trainable models have been developed for scene flow estimation using optical flow architectures [27 46 [5].
Moreover, with the growing popularity of Light Detection and Ranging (LiDAR), the interest has shifted to computing scene flow from point clouds instead of stereo image sequences. In this work, we focus on estimating scene flow from 3D point clouds.
One of the challenges faced in scene flow estimation fast movement. Previous methods usually employ a complicated multi-stage refinement with either a coarse-to-fine architecture [55] or recurrent architecture [46] address the problem. We instead propose solve scene flow estimation by single-scale one-shot global matching method, that is able to capture arbitrary correspondence, thus, handling fast movements. Occlusion yet another challenge faced in scene flow estimation. We take inspiration from an optical flow estimation method [36] to enforce smoothness consistency during the matching process.
The proposed method consists of two stages: feature extraction and matching. detailed description given in Section 3 To extract high-quality features, we take inspiration from the recently dominant transformers [47] and propose a hybrid local-global-cross transformer architecture to learn accurate and robust feature representations. Both local and global-cross transformers are crucial for our approach as also shown experimentally in Section 4.5 The global matching process, including
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

estimation and refinement, is guided solely by feature similarity matrices. First, scene flow is calculated as a weighted average of translation vectors from each source point to all target points under the guidance of cross-feature similarity matrix. Since the matching is done in global manner, can capture short-distance correspondences as well as long-distance correspondences and, therefore, is capable of dealing with fast movements. Further refinement is done under the guidance of self-feature similarity matrix to ensure scene flow smoothness in areas with similar features. This allows to propagate the estimated scene flow from non-occluded areas to occluded areas, thus solving the problem of occlusions.
To summarize, our contributions are: (I) A hybrid local-global -cross transformer architecture is introduced to learn accurate and robust feature representations of 3D point clouds. (2) Based on the similarity of the hybrid features, we propose to use global matching process to solve the scene flow estimation. (3) Extensive experiments on popular datasets show that the proposed method outperforms previous scene flow methods by a large margin on FlyingTThings)) [30] and Waymo-Open [44) and
achieves state-of-the generalization ability on KITTI Scene Flow (31.
2 Related Work
2.1 Scene Flow
Scene flow estimation
23 has developed quickly since the introduction of the KITTI Scene Flow [31
and FlyingThings3D
30 benchmarks, which were the first benchmarks for estimating scene flow
40, 48 58) assume that the objects in scene are rigid and decompose the estimation task into subtasks. These subtasks often involve first detecting or segmenting objects in the scene and then fitting motion models for each object. In autonomous driving scenes, these methods are often effective, as such scenes typically involve static backgrounds and moving vehicles. However, they are not capable of handling more general scenes that include deformable objects. Moreover, the subtasks introduce non-differentiable components, making end-to-end training impossible without instance-level supervision.
from stereo videos. Many scene flow methods [1] 29 31 37
Recent work in scene flow estimation mostly takes inspiration from the related task of optical flow (9] 16 41 45) and can be divided into several categories: encoder-decoder methods [44 27
that solve the scene flow by an hourglass architecture neural network. multi-scale methods 33 20 55
that estimate the motion from coarse to fine scales, or recurrent methods [17 46 53 that iteratively refine the estimated motion. Other approaches
[9 34 try to solve the problem by finding soft
correspondences on point pairs within small region. In order to reduce the annotation requirement, some methods focus on runtime optimization need for training data 4.
prior assumptions (21), or even without the
were the first methods to learn opti- cal flow end-to-end with an hourglass- like model, and inspired many later methods. Flownet3D (27 first employs set of convolutional layers to extract coarse features. A flow embedding layer is intro- duced to associate points based on their spatial localities and geometrical similarities on a coarse scale. A set of upscaling convolutional layers is then introduced to upsample the flow to the high resolution. FlowNet3D++ [52) further incorporates point-to-plane distance and angular distance as additional geometry constraints to Flownet31) (27). Fet [14] employs Bilateral Convolutional Layers (BCL) to restore structural information from unstructured point clouds. Following the hourglass-lik model, DownBCL, UpBCL and CorBCL operations are proposed to restore information from each point cloud and fuse information from both point clouds.
Encoder-decoder Methods: Flownet 9 and Flownet2.0 [16,,
Coarse-to-fine Methods: PointPWC-Net (55) is coarse-to-fine method for scene flow estimation using hierarchical feature extraction and warping, which is based on the optical flow method PWC- Net [1]. A novel learnable Cost Volume Layer is introduced to aggregate costs in a patch-to to-patch manner. Additional self-supervised losses are introduced to train the model without ground-truth labels. Bi-PointFlowNet [3] follows the coarse-to-fine scheme and introduces bidirectional flow embedding layers to learn features along both forward and backward directions. Based on previous methods 27 55). HCRF-Flow 20 introduces high-order conditional random fields (CRFs) based relation module (Con-HCRFs) to explore rigid motion constraints among neighboring points to force point-wise smoothness and within local regions to force region-wise rigidity. FH-Net [7] proposes

---

fast bierarchical network with lightweight Trans-flow layers to compute key points flow and inverse Trans-up layers to upsample the coarse flow based on the similarity between sparse and dense points.
Recurrent Methods: FlowStep3D 17 is the first recurrent method for non-rigid scene flow estimation. They first use global correlation unit to estimate an initial flow at the coarse scale, and then update the flow iteratively by Gated Recurrent Unit (GRU). RAFT3D [46] also adopts recurrent framework. Here, the objective is not the scene flow itself but a dense transformation field that maps each point from the first frame to the second frame. The transformation is then iteratively updated by a GRU. PV-RAFT [53] presents point-voxel correlation fields to capture both short-range and long-range movements. Both roarse-to-fine and recurrent methods take the cost volume as input to a convolutional neural network for scene flow prediction. However, these regression techniques may not be able to accurately capture fast movements, and result, multi-stage refinement is often necessary. On the other hand, we propose a simpler architecture that solves scene flow estimation in single-scale global matching process with no iterative refinement.
Soft Correspondence Methods: Some work poses the scene flow estimation as an optimal transport problem. FL.OT [44 introduces an Optimal Transport Module that gives dense transport plan informing the correspondence between all pairs of points in the two point clouds. Convolutional layers are further applied refine the scene flow. SCTN [9 introduces a voxel-based sparse convolution followed by a point transformer feature extraction module. Both features, from convolution and transformer, are used for correspondence computation. However, these methods involve complicated regularization and constraints to estimate the optimal transport from the correlation matrix. Moreover, the correspondences are only computed within small neighboring region. We instead follow the recent global matching paradigm [10, [56 64] and solve the scene flow estimation with global matcher that is able capture both
and long-distance correspondence.
Runtime Optimization, Prior Assumptions, and Self-supervisio: Different from the proposed method, which is fully supervised and trained offline, some other work focuses on runtime optimiza- tion, prior assumptions, and self-supervision. Li el al. [22] revisit the need for explicit regularization in supervised scene flow learning. The deep learning methods tend to rely on prior statistics learned during training, which are domain-specific. This does not guarantee generalization ability during testing. To this end, Li et al. propose to rely on runtime optimization with scene flow prior as strong regularization. Based on [22 Lang al. [18] propose to combine runtime optimization with self-supervision. A correspondence model is first trained to initialize the flow. Refinement is done by optimizing the flow refinement component during runtime. The whole process can be done under self-superrisson. Pontes et al. (33) propose to use the graph Laplacian of a point cloud to force the scene flow to be "as rigid as possible". Same as in [22] this constraint
be optimized during runtime. Li et al. [2T] propose a self-supervised scene flow learning approach with local rigidity prior assumption for real-world scenes. Instead of relying on point-wise similarities for scene flow estimation, region-wiss wise rigid alignment is enforced. Most recently, Chodosh el al. 4 identify the main challenges of LiDAR scene flow estimation estimating the remaining simple motions after removing the dominant rigid motion. By combining ICP, rigid assumptions, and runtime optimization, they achieve sute-of-the-ar performance without any training data.
2.2 Point Cloud Registration
Related to scene flow estimation, there are some correspondence-based point cloud registration methods. Such methods separate the point cloud registration task into two stages: finding the correspondences and recovering the transformation. PPFNet (6) and PPF-FoldNet (S] proposed by Deng et al. focus on finding sparse corresponding 3D local features. Gojeic et al. [12 propose to use voxelized smoothed density value (SDV) representation to match 3D point clouds. These methods only compute sparse correspondences and are not capable of handling dense correspondences required in scene flow tasks. More related works are CoFiNet [59] and Geo' Transformer (36), both of which involve finding dense correspondences employing transformer architectures. Yu et al. CoFiNet (59] propose a detection-free learning framework and find dense point correspondence in a coarse-to- fine manner. Qin et al. in Grorrrosforerr (36] further improve the accuracy by leveraging the geometric information. RolTr [60] introduces Rotation- Invariant Transformer to disentangle the geometry and poses, and tackle point cloud matching under arbitrary pose variations. PEAL (61] introduces the Prior Embedded Explicit Attention Learning model (PEAL), and for the first time explicitly injects
3

---

Scene Flow
P R
Robust Loss
Stage2
Prediction
Global Matching
Transtomer
N Transformer
Globel Transformer
Globel Transformer
Stagel
Embedied Tokens
Local Transformer
Transformar
Tokanization
DGONNPoHHWWIMPP
DGONNPOHINWWMEP
Input
X1 R B8
Figure 1: Method Overview. We propose simple yet powerful method for scene flow estimation. In the first stage (see 3.. we propose a strong local-global-cros transformer architecture that capable of extracting robust and highly localizable features. In the second stage (see Section (.2). simple global matching yields the flow. In comparison to previous work, our approach is significantly simpler, while achieving state-of-the-art results.
overlap prior into Transformer to solve point cloud registration under low overlap. However, the goal of point cloud registration not to estimate the translation vectors for each the points, which makes our work different from these approaches.
2.3 Transformers
Transformers were first proposed 47 for translation tasks with an encoder-decoder architecture using only attention and fully connected layers. Transformers have been proven to be efficient in sequence-to-sequeece problems, well-suited to research problems involving sequential and unstruc- tured data. The key to the success of transformers over convolutional neural networks is that they can capture long-range dependencies within the sequence, which very important, not only in translation but also in many other tasks e.g. computer vision [8], audio processing [24], recommender systems [42, and natural language processing [54].
Transformers have also been explored for point clouds (28) The coordinates of all points are stacked together directly as input to the transformers. For the tasks of classification and segmentation, PT 63 proposes constructing a local point transformer using k-nearest-neighhorr Each of the points would then attend its nearest neighbors. PointASNL [57] uses adaptive sampling before the local transformer, and can better deal with noise and outliers, PCT [5] proposes to use global attention and results a global point transformer. Pointformer 332 proposes new scheme where first local transformers are used extract multi-scale feature representations, then local-giobal transformers are used as cross attention to multi-scale features, finally, a global transformer captures context-aware [62 is originally designed for masked point modeling. Instead of
representations. Point-BER treating each point as one data item, they group the point cloud into several local patches. Each of these sub-clouds is tokenized to form input data.
Previous work on scene flow estimation exploits the capability transformers for feature extraction either using global-based transformers in local matching paradigm [19 or local-based transformers recurrent architecture [TT. Instead, we propose to leverage both local and global transformers to learn feature representation for each point on a single scale. We show that high-qualin feature representations are the fundamental property that needed for scene flow estimation when formulated global matching problem.

---

3 Proposed Method
Given point clouds Xi NR and X2 RRxx with only position information, the objective to estimate the scene flow V & N13 that maps each point in the source point cloud to the target point cloud. Due to the sparse nature of the point clouds, the points in the source and the target point clouds do not necessarily have a one-to-one correspondence, which makes it difficult to formulate scene flow estimation similarity matrix of point pairs as soft correspondence is sufficient for scene flow estimation. Unlike many applications based on point cloud processing which need acquire a high-level understanding. e.g. classification and segmentation, scene flow estimation requires low-level understanding to distinguish geometry features between each element the point clouds. To this end, we propose transformer architecture to learn high-quality features for each point. The proposed method consists of two core components: feature extraction (see Section 3.1 and global matching (see Section 3.2).
dense matching problem. Instead, we show that learning a cross-feature
The overall framework is shown in Figure]
3.1 Feature Extraction
Tokenization: Given the 3D point clouds X1, X2, each point is first tokenized to get summarized information of its local neighbors. We first employ an off-the-shelf feature extraction network DGCNN [51] map the input 3D coordinate xy into high dimensional feature space -. conditioned on its nearest neighbors zy. Each layer of the network can be written as
x
max h(r,, xz); 2)E
where h represents sequence of linear layers, batch normalization, and ReLU layers. The local neighbors zj E (i) are found by a k-nearest-neighbor (knn) algorithm. Multiple layers are stacked together to get the final feature representation.
For each point, local information incorporated within a small region by applying local Point Transformer [63] within zj Ni)). The transformer is given by
x
Σ ((p)(a?) vi(aa)) 8) @ (an(a))) + 8),
()EN(()
where the input features are first passed through linear layers yoi, the, and az to generate query, key and value. 8 the relative position embedding that gives information about the 3D coordinate distance between x; and Zz. represents a Multilayer Perceptron consisting of two linear layers and one ReLU nonlinearity. The output a! is further processed by linear layer and a residual connection from xh.
Transformer blocks are used to process the embedded tokens. Each of the blocks includes self-attention followed by cross-attention 38
Global-cross Transformer:
56
The self-attention is formulated as
x
Σ (1p((t)) by (a ZyE,
where each point H; X' attends all the other points xj E X1, same for the points æ; X2. Linear layers Pag Vgg and Og generate the query, key, and value. (, denotes a scalar product. Linear layer, layer norm, and skip connection are further applied complete the self-attention module.
The cross-atte aoossnattttionnn is given as
Σ vo6) EyE
where each point F; E X1 in the source point cloud attends to all the points 2, E X2 in the target point cloud, and vice versa. Feedforward network with multi-layer perceptron and layer norm is applied to aggregate information to the next transformer block. The detailed architecture of our proposed local-global-cross transformer's is presented in Figure The feature matrices F; E N xd and F2 E R are formed as the concatenation of all the output feature vectors from the final transformer block, where N, and N₂ are the number of points in the two point clouds and d the feature dimension.
5

---

Md
Add
Layer Norm
Layer Norm
I inwar syye
Layer Nor
Mulilayur Parcaptron
Mufllayer Perceptrion
Single-Haad
Line Layer
Layer
Imnn
knn
Layer
Linear Layer
SingleHaadd Attantann
Sinle-lledd Atlention
Singlo-Hkad
Embedded Fentures
Emberded lokens
Global Features
Grobal Festures
Figure 2: Transformer Architecture. Detailed local (left), global (middle), and cross (right) trans- former architecture. The local transformer incorporates attention within small number of neighbors. The global transformer is applied on the source and target points separately and incorporates attention on the whole point clouds. The cross transformer further attends to the other point cloud and gets the final representation conditioned on both the source and the target.
3.2 Global Matching
Feature similarity matrices are the only information that is needed for an accurate scene flow estimation. First, the cross similarity matrix between the source and the target point clouds given by multiplying the feature matrices F1 and Fg and then normalizing over the second dimension with softmax to get right stochastic matrix.
FIFT
Ceross
(5)
Vd
Meross = softmax(Ceoos))
(6)
where each row of the matrix Meross E N1 x is the matching confidence from one point in the source point cloud to all the points in the target point cloud. The second similarity matrix is the self similarity matrix of the source point cloud, given by
W (F(FF)W((R)) Vd
C-eir
M = softmax (Csett)
(8)
which is matrix multiplication of the linearly projected point feature Fi. Wg and W₂ are learnable linear projection layers. Each of the matrix M E N is the feature similarity between one point in the source point cloud to all the other points in the source point cloud. Given the point cloud coordinates X e Nx3 and X2 E IR N the estimated matching point &e in the target point cloud computed a weighted average of the 3D coordinates based on the matching confidence
a Meross
(9)
The scene flow computed as the movement between the matching points
Vineer 22 X1.
(10)
The estimation procedure can also be seen as a weighted average of the translation vectors between point pairs, where softmax ensures that the weights sum to onc.
For occlusions in the source point cloud, the matching would fail under the assumption that there exists a matching point in the target point cloud. We avoid this by employing self similarity matrix that utilizes information from the source point cloud. The self similarity matrix Maar bears the similarity information for each pair of points in the source point cloud. Nearby points tend to share similar features and thus have higher similarities. Multiplying Maar with the predicted scene flow
Vinter can be seen as a smoothing procedure, where for each point, its predicted scene flow vector is updated as the weighted average of the scene flow vectors of the nearby points that share similar features. This also allows the network to propagate the correctly computed non-occluded scene flow estimation to its nearby occluded areas, which gives
Vinal=
Msatt V Vinter
(1I)

---

3.3 Loss Formulation
Let V be the estimated scene flow and Ve be the ground truth. We follow CamLiFlow 25 and use robust training loss to supervise the process, given by
L Σ (nnVana (t) (i((1)l1 +c),
(12)
where is set to 0.01 and is set to 0.4.
4 Experiments
4.1
Implementation Details
The proposed method implemented in PyTorch. Following previous methods 14 [55, the numbers of points N1 and N2 are both set to 8192 during training and testing, randomly sampled from the full set. We perform data augmentation by randomly flipping horizontally and vertically. We use the AdamW optimizer with learning rate of 10-4, a weight decay of 10-4, and OneCycleLA as the scheduler to anneal the learning rate. The training done for 600k iterations with a batch size of 8.
4.2 Evaluation Metrics
and evaluate the proposed method with the accuracy metric F. E.a.. and the robustness metrics ACCs, ACCn and Outliers. F.P is the 3D end point error V vtt ll2 between the estimated scene flow and the ground truth averaged over each point, ACCs is the percentage of the estimated scene flow with an end point error less than 0.05 meter relative error less than 5%. ACCR is the percentage of the estimated scene flow with an end point еπrоr less than 0. meter or relative eтrоr less than 10%. Outliers is the percentage of the estimated scene flow with an end point error more than 0.3 meter relative error more than 10%.
For fair comparison we follow previous work [14 46 55
4.3 Datasets
The proposed method is tested on three established benchmarks for scene flow estimation.
30 is a synthetic dataset of objects generated by ShapeNet 2 with randomized movement rendered in scene. The dataset consists of 25000 stereo frames with ground truth data.
real world dataset for autonomous driving. The annotation is done with the help of CAD models. It consists of 200 scenes for training and 200 scenes for testing.
KITTI Scene Flow K
Both datasets have be preprocessed in order to obtain 3D points from the depth images. There exist two widely used preprocessing methods to generate the point clouds and the ground truth scene flow, one proposed by Liu al. in FlowNet3D 27 and the other proposed by Gu er al. in HPLFlowNet [14]. The difference between the two approaches is that Liu et al. [27] keeps all valid points with an occlusion mask available during training and testing. Gu al. 14 simplifies the task by removing all occluded points. We denote the datasets preprocessed by Liu el al. in FlowNet3D as F3D,/KITTI, and by Gu et al. in HPLFlowNet as F3D,/KITTI,
the original setting from [4 27). the FlyingThing33D dataset F3D, consists of 19640 and 3824 stereo scenes for training and testing, respectively. F3D, consists of 20000 and 2000 stereo scenes for training and testing. respectively. For the KITTI dataset, KITTI, consists of 142 scenes from the training set, and KITTI, consists of 150 scenes from the training set. Since there is no annotation available in the testing set of KITTI, we follow previous methods to test the generalization ability of the proposed method without any fine-tuning on KITTI, and KITTI, For better evaluation and analysis, we additionally follow the setting in CamLiFlow
25 to extend F3D, include occluded points. We denote this as F3D
is large-scale autonomous driving dataset. We follow to preprocess the dataset to create the scene flow dataset. The dataset contains 798 training and 202 validation sequences. Each sequence consists 20 seconds of 10Hz point cloud data. Different from 7 which only contains 100 sequences, trained and tested our model the full dataset.
Waymo-Open Open Dataset

---

4.4 State-of-the-art Comparison
We compare our proposed method GMSF with state-of-the-art methods on FlyingThings3D in different settings. occluded points and all points. Among all the methods, including methods with the corresponding stereo images as additional input [46], or even with optical flow as additional ground truth for our proposed method achieves the best performance both terms of accuracy
the results on F3D,. Evaluation metrics are calculated over both non-
supervision 25 26
and robustness.
To give fair comparison with previous methods we report results on F3D, and F3D, with gener- alization to KITTI, and KITTI, in Table 2 and Table [ The proposed method achieves the best performance both F3D, and F3D,, surpassing other state-of-the-ar methods by a large margin. The generalization ability of the proposed model on KITTI, and KITTI, also achieves state of the art.
We further conduct experiments the Waymo-Opor dataset. We train and test the full dataset with 798 training and 202 testing sequences. Comparisons with state of the art are given in Table
Table 1: State-of-the-ar comparison on F3D.. The input modalities are given as a reference. Our method with only 3D points as input outperforms all the other state-of -aee--fffhhhttte art methods all metrics.
Method
Input
EP &
ACCs t
EPEaD all
ACCs t all
non non-occluded
non non-occluded
DlowNet33
2F
CVPR'19
Points Image+Depth Image+Points
0.158
22.9
0.214
18.2
RAFT3D 46
CVPR'21
0.094
80.6
Ciloo
CVPR/22
0.032
92.6
0.061
85.6
CamLiPWC
26
arsiv223
Image+Points
0.057
86.3
CamLiRAFT
26
arxiv'23
image+Points
0.049
88.4
GMSr(ours)
Points
0.022
0.040
92.6
Table 2: State-of-the-art comparison on F3D, and KITTI, The models are only trained on F3D, prepared by 27 with occlusions. Testing results F3D, and KITTI, are given.
Method
F3Do
KITTLO
EPEsn 1 ACCss
ACCR t Oulliers LaEPEan ACCss 1 ACCR Oulliers
FlowNet3D 27
0.157
22.8
58.2
80.4
0.183
9,8
39.4
79.9
HPLFlowNet 4
0.168
26.2
57.4
81.2
0.343
10.3
38.6
PointPWC [55]
0.155
41.6
69.9
63.8
0.118
40.3
75.7
49.6
FLOT [34]
0.153
39.6
66.0
66.2
0.130
27.8
66.7
52.9
CPC 26
0.092
71.5
87.1
37.2
CamLiRAFT 26
0.076
79.4
90.4
27.9
Bi-PointFlow
0.073
79.1
89.6
27.4
0.065
76.9
90.6
26.4
RAFT3D 46
0.064
83.7
89.2
3DFlow [49
0.063
79.1
90.9
27.9
0.073
81.9
89.0
26.1
SCOOP+ 18
0.047
91.3
95.0
18.6
GMSF(ours)
0.022
95.0
97.5
5.6
0.033
91.6
95.9
13.7
4.5
Ablation Study
Table 6 shows the results of different numbers of global-cross transformer layers. While our approach technically works even without global-cross transformer layers, performance is sig- nificantly worse compared to using two or more layers. This shows that only incorporating local information for the fcature representation is insufficient for global matching. Morcover, the capacity of the network improves with the number of layers and achieves the best performance at 10 layers.
Table shows the importance of different components in the tokenization process. We tried different methods, DGCNN [51], PointNet (35), and MIP, to map the 3D coordinates of the points into the high- sich-immensioal feature space. For each of these mapping methods, the influence of the Local Point Transformer [63] is tested. When the local transformer is present, the metrics are similar with different mapping strategies, which demonstrate the effectiveness of the proposed pocal-global-cross transformer architecture. In the absence of the local transformer, the performance remains comparable with DGCNN for mapping drops significantly with PointNet or MIP, which indicates the necessity of local information encoded in the tokenization step.

---

Table 8 gives the ablation study on feature dimensions. The default number of feature dimensions 128 our model. Reducing the number of feature dimensions leads to lack of capacity of the model.
Table 3: State-of-the-art comparison on F3D, and KITTI, The models are only trained on F3D, prepared by [4 without occlusions. Testing results on F3D, and KITTI, are given.
Method
F3Ds
KITTIs
EPEsD sCCss
ACCR 1 Outliers LED ACCss t TACCR R Outliers
DlowNet3D
27
0.1136
41.25
77.06
60.16
0.1767
37.38
66.77
52.71
HPLFlowNel
14
0.0804
61.44
85.55
42.87
0.1169
47.83
77.76
41.03
PointPWC
55
0.0588
73.79
92.76
34.24
0.0694
72.81
FLOT [4
0.0520
73.20
92.70
35.70
0.0560
75.50
90.80
24.20
HCRF-Flow
20
0.0488
83.37
95.07
26.14
0.0531
86.31
94.44
17.97
PV-RAFT [3
0.0461
81.69
95.74
29.24
0.0560
82.26
93.72
21.63
3lowStep3D
0.0455
81.62
96.14
21.65
0.0546
80.51
92.54
14.92
RCP [33
0.0403
85.67
96.35
19.76
0.0481
84.91
94.48
12.28
SCTN [9]
0.0380
84.70
96.80
26.80
0,0370
87,30
95.90
17.90
CaCLiPWC 26
0.0320
92.50
97.90
15.60
CamLiRAFT
26
0.0290
93.00
98.00
13.60
Bi-PointFlow
0.0280
91.80
97.80
14.30
0.0300
92.00
96.00
14.10
3DFlow 49]
0.0281
92.90
98.17
14,58
0.0309
90.47
95,80
16.12
PT-FlowNet
0.0304
91.42
98.14
17.35
0,0224
95.51
98.38
11.86
(MrF()urr)
0.0090
99.18
99.69
2,55
0.0215
96.22
98.25
9.84
Table 4: State-of-the-arr comparison on Waymo-
Open dataset.
Table 5: Meta-information.
Method
EPEaD + ACCs t ACCR 1 Outliers
Runtime(ms)
417.3
FlowNet3D 27
0.225
23.0
48.6
77.9
FLOPs(G)
654.32
PointPWC [55]
0.307
10.3
23.1
78.6
Parameters(M)
7.07
FESTA 50]
0.223
24.5
27.2
76.5
Memory (G))
4.99
FH-Net
0.175
35.8
67.4
60.3
Memory (train)
162.3
GMSF(ouus) 0.083
74.7
85.1
43.5
Table 6: Ablation study on the number of global-cross transformer layers on F3D.. The influence of the number of global-cross transformer layers is tested. The best performance is gained at 10 transformer layers.
Layers EPEsD
ACCs t ACCR 1 all
Outliers all
EPEsD J non-oce
ACCs +
ACCR non-oce
Outliers non-oce
all all
non-oce
0
0.212
39.01
63.59
66.51
0.132
43,95
70.24
62.92
0.075
79.02
90.22
25.64
0.047
84.67
94.23
22.07
0.055
87.37
93.76
16.39
0.032
92.01
96.84
13.41
0.050
89.32
94.60
14.23
0.029
93.46
97.33
11.54
8 0.045
91.22
95.25
12.11
0.025
94.91
97.70
9.68
0.040
92.64
95.84
10.34
0.022
95.94
98.06
8.13
0.043
91.95
95.57
11.12
0.024
95.40
97.88
8.81
0.045
91.66
95.41
11.54
0.025
95.19
97.78
9.21
16
0.044
91.74
95.51
11.33
0.025
95.38
97.93
8.91
4.6
FLOPs, GPU memory, and Runtime.
We report meta-information
experiments: runtime (ms per scene) during testing NVIDIA A40 GPU, FLOPs (G), Number of parameters (M), GPU memory (GB) during testing (batch size 1) and training (batch size 8) with 10 transformer layers and 128 feature dimensions in Table
4.7 Visualization
Figure shows a visualization of the GMSF results on two samples from the Flying dataset. Red and blue points represent the source and the target point clouds, respectively. Green points

---

Table 7: Ablation study on the components of tokenization on F3D.. The influence of using different backbones and the presence of local transformer tested. The results show that as long as there is local information (DGCNN/ Point Transformer) present in the tokenization process, the performance remains competitive. On the other hand, using only PointNet or MLP for tokenization, the performance drops significantly.
Backbone PT E3D 4 ACCs t ACCn t Outliers | E all
ACCs non-occ
ACCR t Outliers & non--cc
all
all
all
non-occ
noR-oce
DGCNN
0.040
92.64
95.84
10.34
0.022
95,94
98.06
8.13
DGCNN
0.052
89.68
94.37
13.71
0.030
93.74
97.14
11.00
PointNet
0.043
92.22
95.80
10.86
0.024
95.65
98.04
8.63
PointNet
0,063
86.76
93.06
16.67
0.037
91.45
96.31
13.51
MLP
0.043
91.81
95.48
10.21
0.023
95.43
97.84
7.75
MILP
0.060
88.08
93.33
14.11
0.035
92.69
96.55
10.83
Table 8: Ablation study on the number of feature dimensions on F3D The performance decreases the number of feature dimensions drops.
dim
EPEsD + all
ACCs all
ACCs t Outliers + all
EPEaD non-oce
ACCs 1 non-occ
ACCR t
Outliers non-oce
all
non-occ
32
0.073
83.04
91.32
21.07
0,044
88.36
95.07
17.56
64
0.051
89.64
94.57
13.68
0.029
93.79
97.32
10.93
128
0.040
92.64
95.84
10.34
0.022
95.94
98.06
8.13
represent the warped source point cloud toward the target point cloud. As we see in the figure, the blue points align very well with the green points, which demonstrates the effectiveness of our method.
Figure 3: Visualization results on SringThing33D.. Two scenes from the dataset are given. Red, blue, and green points represent the source, target, and warped source point cloud, respectively. Part of the point cloud is zoomed in for better visualization.
5 Conclusion
We propose to solve scene flow estimation from point clouds by simple single-scale one-shot global matching, where we show that reliable feature similarity between point pairs is essential and sufficient to estimate accurate scene flow. To extract high-quality feature representations, we introduce hybrid local-global-cross transformer architecture. Experiments show that both the presence of local information in the tokenization step and the stack of global-cross transformers are essential to success. GMSF shows state-of-the-an performance on the Flying Things3D, KITTI Scene Flow, and Waymo-Open datasets, demonstrating the effectiveness of the method.
The global matching process in the proposed method needs to be supervised by the ground truth, which is difficult to obtain in the real world. As a result, most of the supervised scene flow estimations are trained on synthetic datasets. We plan to extend our work to unsupervised settings to exploit real data.
Limitations:
10

---

Acknowledgements: This work was partly supported by the Wallenberg Artificial Intelligence, Au- tonomous Systems and Software Program (WASP), funded by Knut and Alice Wallenberg Foundation. and the Swedish Research Council grant 2022-04266; and by the strategic research environment ELLIT funded by the Swedish government. The computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at C3SE partially funded by the Swedish Research Council grant 2022-06725, and by the Berzelius resource, provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
References
[1] Aseem Bchl, Omid Hosseini Jafari, Siva Karthik Mustikovela, Hassan Abu Alhaija, Carsten Rother, and Andreas Geiger. Bounding boxes, segmentations and object coordinates: How important is recognition for 3d scene flow estimation autonomous driving scenarios? In Proceedings of the IEEE International Conference on Computer Vision, pages 2574-2583, 2017.
[2] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rick 3d model repository.
arXiv preprint arXiv:1512. 030112. 2015.
[3] Wencan Cheng and Jong Hwan Ko. Bi-pointflownet: Bidirectional learning for point cloud based scene flow estimatiion In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022. Proceedings, Part XXVIII, pages 108-124. Springer, 2022.
[4] Nathaniel Chodosh, Deva Ramanan, and Simon Lucey. Re-evaluating lidar scene flow for autonomous driving. arXiv preprint arXiv:2304.00 024.02250, 2023.
[5] Haowen Deng. Tolga Birdal, and Slobodan llie. Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors. In Proceedings of the European conference on computer vision (ECCV), pages 602-618, 2018.
[6] Haowen Deng, Tolga Birdal, and Slobodan Ilie. Ppfnet: Global context aware local features for robust 3d point matching. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pages
195-205, 2018.
[7] Lihe Ding. Shaocong Dong. Tingfa Xu, Xinli Xu, Jie Wang. and Jianan Li. Fh-net: A fast hierarchical network for scene flow estimation on real-world point clouds. In Computer Vision-ECCV 2022: 17th Furopean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX, pages 213-229.
Springer, 2022.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929,, 2020. [9] Alexey Dosovitskiy, Philipp Fischer, Eddy 1lg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: I Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758 2766. 2015.
[10] Johan Edstedt, Ioannis Athanasiadis, Märten Wadenbäck, and Michael Felsherg. Dkm: Dense kemelized feature matching for geometry estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17765-17775, 2023.
[11] Jingyun Fu, Zhiyu Xiang, Chengyu Qiao, and Tingming Bai. Pt-flownet: Scene flow estimation on point
clouds with point transformer. IEEE Robotics and Automation Letters, 8(5):2566 2573, 2023.
[12] Zan Gojcic, Caifa Zhou, Jan D Wegner, and Andreas Wieser. The perfect match: 3d point cloud matching with smoothed densities. In Proce Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pages 5545-5554, 2019.
[13] Xiaodong Gu, Chengzhou Tang, Weihao Yuan, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Rep: Recurrent closest point for scene flow estimation on 3d point clouds. arXiv preprint arXiv:2205. 10028, 2022.
[14] Xiuye Gu, Yijie Wang, Chongruo Wu, Yong Jac Lee, and Panqu Wang. Hplflownet: Hierarchical
permutohedral lattice flownet for scene flow estimation on large-scale point clouds. In Proceedings of the
IEEE/CVF conference on computer vision and pattern recognition, pages 3254-3263, 2019.
[15] Meng-Hao Guo, Jun-Xiong Cai,, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet:
Point cloud transformer. Computational Visual Media, 7:187-199, 2021.
[16] Eddy 1lg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE
conference computer vision and pattern recognition, pages 2462-28 24222 2017.
[17] Yair Kittenplon, Yonina C. Eldar, and Dan Raviv. Flowstep3d: Model unrolling for self-supervised scene flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
pages 4123, 2021.
[18] Itai Lang, Dror Aiger, Forrester Cole, Shai Avidan, and Michael Rubinstein. Scoop: Self-supervised correspondence and optimization-ern baamiz scene flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5281-5290, 2023.
[19] Bing Li, Cheng Zheng, Silvio Giancola, and Bernard Ghanem. Setn: Sparse transfooonntrnnsnommmee network for scene flow estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1254-1262, 2022.
11

---

[20] Ruibo Li, Guosheng Lin, Tong He. Fayao Liu, and Chunhua Shen. Herf-flow: Scene flow from point clouds
with continuous high-order crfs and position-aware flow embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 364-373, 2021.
[21] Ruibo Li, Chi Zhang. Guosheng Lin, Zhe Wang, and Chunhua Shen. Rigidflow: Self-supervised scene flow learning on point clouds by local rigidity prior. Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages 99959-16988, 2022.
[22] Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey. Neural scene flow prior. Advances in Neural
Information Processing Systems, 34:7838-785 2021. 2021.
[23] Zhiqi Li, Nan Xiang, Honghua Chen, Jianjun Zhang, and Xiaosong Yang. Deep learning for scene flow estimation on point clouds: A survey and prospective trends. In Computer Graphics Forum. Wiley Online
Library, 2023.
[24] Andy T Liu, Shang-Wor Wen Li, and Hung-yi Lee. Tera: supervird learning of transformer encoder representation for speech. JEEF/ACM Transactions Audio, Speech, and Language Processing, 29:2351- 2366, 2021.
[25] Haisong Liu, Tao Lu. Yihui Xu, Liu, Wenjie Li, and Lijun Chen. Camliflow: bidirectional camera-lidan
fusion for joint optical flow and seene flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5791-5801, 2022.
[26] Haisong Liu, Tao Lu, Yihui Xu, Jia Liu, and Limin Wang. Learning optical flow and scene flow with bidirectional camera-lidar fusion. arXiv preprint ar3iv:23303. /2017, 2023.
[27] Xingyu Liu, Charles R Qi. and Leomidas J Guibas. Flownet3d: Learning scene flow in 3d point clouds. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 529-537,
2019.
[28] Dening Lu, Qian Xie, Mingqiang Wei, Linlin Xu, and Jonathan Li. Transformers in 3d point clouds: A survey, arXiv preprint arXiv:220S. 000417.777. 2022.
[29] Wei-Chiu Ma, Shenlong Wang. Rui Hu, Yuwen Xiong, and Raquel Urtasum. Deep rigid instance scene flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3614-3622, 2019.
[30] Nikolaus Mayer. Eddy 1lg. Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large datasct to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
4040-4048, 2016.
[31] Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3061-3070, 2015.
[32] Xuran Pan, Zhuofan Xia, Shiji Song, Li Erran Li, and Gao Huang. 3d object detection with pointformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7463-7472,
2021.
[33] Jhony Kacsemodel Pontes, James Hays, and Simon Lucey. Scene flow from point clouds with or without learning. In 2020 international conference on 3D vision (3DV), pages 261-277 1200 IEEE, 2020.
[34] Gilles Puy, Alexandre Boulch, and Renaud Marlet. Flot: Scene flow on point clouds guided by optimal transport. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UKK August 23-288,
2020, Proceedings, Part XXVIII. pages 527-544. Springer, 2020.
[35] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep lcarning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern
recognition, pages 652-660, 2017.
[36] Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.
[37] Zhile Ren, Deqing Sun, Jan Kautz, and Erik Sudderth. Cascaded scene flow prediction using semantic segmentation. In 2017 International Conference on 3D Vision (3DV), pages 225-233 IEEE, 2017.
[38] Paul- Edouard Sarlin, Danicl DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer
vision and pattern recognition, pages 4938 4938-4947, 2020.
[39] Daniel Seita, Yufei Wang. Sarthak Shetty, Edward Yao Li, Zackory Erickson, and David Held. Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. In Conference on Robot
Learning, pages 1038-1049, PMLR, 2023,
[40] Leonhard Sommer, Philipp Schröppel, and Thomas Brox. Sf2se3: Clustering scene flow into se (3)-motions via proposal and selection. In Pattern Recognition: 44th DAGM German Conference, DAGM GCPR 2022, Konstanz, Germany, September 27-30, 2022, Proceedings, pages 215-229. Springer, 2022.
[41] Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cuns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 8934 8943, 2018
[42] Fei Sun, Jun Liu, Jian Wu. Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management, pages 1441-1450, 2019. [43] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-fret local feature matching with transformers. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 8922-8931, 2021.
12

---

[44] Pei Sun. Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard. Vijaysai Patnaik, Paul Tsui, James Guo. Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 2446-2454, 2020.
[45] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer
Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II
16. pages 402 4194 Springer, 2020.
[46] Zachary Teed and Jia Deng. Raft-3d: Scene flow using rigid-motion embeddings. In Proceedings of the Computer Vision and Pattern Recognition, pages 8375-8384, 2021. [47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems,
IEEF/CVF Conference
30, 2017.
[48]
Christoph Vogel, Konrad Schindler, and Stefan Roth. 3d scene flow estimation with a piccewise rigid scene model. International Journal of Computer Vision, 115:1-28, 2015.
Guangming Wang. Yunzhe Hu, Zhe Liu, Yiyang Zhou, Masayoshi Tomizuka, Wei Zhan, and Hesheng Wang. What matters for 3d seene flow network. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27 2022, Proceedings, Part XXXIII, pages 38-55. Springer, 2022. [50] Haiyan Wang. Jiahao Pang. Muhammad A Lodhi, Yingli Tian, and Dong Tian. Festa: Flow estimation via spatial-temporal attention for scene point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14173-14182, 2021.
[49]
[51] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
[52] Zirui Wang. Shuda Li, Henry Howard-Jenkins, Victor Prisacariu, and Min Chen. Flownet3d+4: Geometric losses for deep scene flow estimation. In Proceedings of the IEEE/CVF winter conference on applications
of computer vision, pages 91-98, 2020.
[53] Yi Wei, Ziyi Wang. Yongming Rao, Jiwen Lu, and Jie Zhou. Pv-raft: Point-voxel correlation fields for scene flow estimation of point clouds. In Proceedings of the JEEE/CVE conference on computer vision and pattern recognition, pages 6954-6963, 2021.
[54] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, al. Transformers: Sfate-of-the-ae natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, 2020.
[55] Wenxuan Wo, Zhiyuan Wang. Zhuwen Li. Wei Liu, and Fuxin. Pointpwe-net: A coarse-to tanse-to-fine network for supervised and self--upervised scene flow estimation on 3d point clouds. arXiv preprint D.V.:1911.24000 2019.
[56] Haofei Xu, Jing Zhang, Jianfci Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pages 8121-8130, 2022.
[57] Xu Yan. Chaoda Zheng. Zhen Li. Sheng Wang, and Shuguang Cui. Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5589-5598, 2020.
[58] Gengshan Yang Deva Ramanan. Learning to segment rigid motions from two frames. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1266-1275, 2021. [59] Hao Yu, Fu Li, Mahdi Salch, Benjamin Busam, and Slobodan Ilie. Cofinet: Reliable coarse-to to fine correspondences for robust pointcloud registration. Advances in Neural Information Processing Systems, 34:23872-23884 2021.
[60] Hao Yu, Zheng Qin, Ji Hou, Mahdi Saleh, Dongsheng Li, Benjamin Busam, and Slobodan Ili.. Rotation- invariant transformer for point cloud matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 5384-53393, 2023.
[61] Junle Yu, Luwei Ren, Yu Zhang, Wenhui Zhou, Lili Lin, and Guojun Dai. Peal: Prior-embedded explicit attention learning for low-overlap point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 17702-17711, 2023.
[62] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19313-19322, 2022.
[63] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr. and Vladlen Koltun. Point transformer. In Proceedings of the IEFF/CVF international conference on computer vision, pages 16269, 2021,
[64] Shiyu Zhao, Long Zhao, Zhixing Zhang, Enyu Zhou, and Dimitris Metaxas. Global matching with overlapping attention for optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17592-17601, 2022.
13


Reviews:
[{"review_id": "ra4F66Go1V", "reviewer": "Reviewer_D7Yn", "summary": "The paper proposes GMSF for scene flow estimation from point clouds.\nAs far as the authors are aware, GMSF is the first to address scene flow estimation with global matching - GMSF is formulated as a single-scale one-shot global matching.\nGMSF incorporates a novel local-global-cross transformer architecture to extract high-quality feature representation, to finally compute the scene flow between point clouds via global matching. \nGMSF outperforms existing methods on F3D_c, F3D_o, F3D_s and KITTI_o, while performing competitively on KITTI_s. \nThe ablative results show that increasing the number of global-cross transformer layers - thus increasing the capacity - is beneficial to the performance, and that the presence of local information is crucial for the performance of GMSF.", "questions": "Please refer to the weaknesses section. The motivation and the proposed method are intuitive and novel, but the design choice of GMSF should be better substantiated, with included analyses for clarity.", "limitations": "The authors have addressed the limitations of GMSF in the paper. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is overall well written and easy to follow.\n\n* The proposed architecture, and the motivation behind it, is intuitive and novel. While the architectural novelty itself is not entirely new (e.g., the sequence of local and global attention has first been introduced by SuperGlue[1] and LoFTR[2] for 2D image matching, and has been applied to 3D point cloud registration through methods including CoFiNet[3] and GeoTransformer[4]), its application to the task of scene flow seems conceptually new.\n\n* Strong performances on the standard benchmarks of scene flow estimation. \n\n[1] PE Sarlin et al., SuperGlue: Learning Feature Matching with Graph Neural Networks, CVPR 2020\n[2] J Sun et al., LoFTR: Detector-Free Local Feature Matching with Transformers, CVPR 2021\n[3] H Yu et al., CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration, NeurIPS 2021\n[4] Z Qin et al., Geometric Transformer for Fast and Robust Point Cloud Registration, CVPR 2022", "weaknesses": "* Insufficient ablative experiments. What if the number of transformer layers increase to a number higher than 8? The given results show that the performance improves gracefully with the number of layers, and it naturally leaves the question to 'until how much'. Also, what if the global transformer and cross transformers are decoupled, such that they can have varying number of layers? \n\n* Lack of latency and computation analyses. The authors emphasize that GMSF is a single-scale, one-shot method; then how does it compare to existing methods in terms of latency and computation (FLOPs, memory)? \n\n* Lack of analysis. How does incorporating global-cross transformer layers improve the performance, and how does incorporating **more** layers **further** improve the performance? This has been partially answered by Table 4, but a visual comparison / analysis would be more convincing.\n\n* Lack of mention of 3D point cloud registration methods. While the task at hand is different, the architectural design and motivation is closely related to 3D point cloud registration, which I believe is therefore worth mentioning in the related work section. Also, it might be a bit of an overstatement to mention that GMSF is the 'first' to address scene flow estimation with global matching, as scene flow estimation and point cloud registration are seemingly closely related tasks .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and helpful suggestions. Below are our clarifications for the questions.\n\n**Q1a: Ablation study on the number of transformer layers.**\n\nPlease refer to the global response to all reviewers B.\n\n**Q1b: Ablation study on decoupling global transformers and cross transformers.**\n\nWe follow SuperGlue [1] and LoFTR [2] and do not decouple the global-cross transformer layer since this combination has been proven effective.\nWe now add an ablation study on decoupling the transformer layer. Three architectures are compared:\n| Model | $EPE_{3D}\\downarrow$ | $ACC_{S}\\uparrow$ | $ACC_{R}\\uparrow$ | $Outliers\\downarrow$ | $EPE_{3D}\\downarrow$ |$ACC_{S}\\uparrow$ | $ACC_{R}\\uparrow$ | $Outliers\\downarrow$ |\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|  | $all$ |||| $nonocc$ ||||\n| global | 0.141 | 64.65 | 81.17 | 40.89 | 0.069 | 72.11 | 88.52 | 34.84 |\n| cross | 0.051 | 88.65 | 94.04 | 14.73 | 0.027 | 93.97 | 97.41 | 11.20 |\n| global-cross | 0.049 | 90.08 | 94.72 | 13.08 | 0.025 | 94.98 | 97.78 | 9.87 |\n\nThe result shows that the cross transformer layers significantly contribute to the performance. This coincides with the finding in SuperGlue [1] where they address \"cross-attention is critical to strong gluing\" in Section 5.4 and Table 4.\n\n**Q2: FLOPs, GPU memory, and Runtime.**\n\nPlease refer to the global response to all reviewers A.\n\n**Q3: Visual comparison / analysis of the influence of incorporating more transformer layers.**\n\nWe provide an additional visual analysis of our model with 0, 2, and 24 transformer layers in the attached PDF to show how incorporating global-cross transformer layers improves performance and how incorporating more layers further improves performance.\n\nThe results show that without global-cross transformer layers the prediction performance drops when there's a lack of texture information e.g. planar and curved surfaces, repetitive pattern. By incorporating only 2 transformer layers, the performance is significantly improved, which is also shown in the paper in Table 4.\nWithout global-cross transformer layers, the learned features are local features. Thus, when there is a planar or curved surface or a repetitive pattern, the matching process tends to be inaccurate.\nMore transformer layers further help with such cases. However, this might not be easy to see in the visualization since the difference of the $EPE_{3D}$ is small (0.081m and 0.041m for transformer layers of 2 and 24, respectively).\n\n**Q4: Related work in 3D point cloud registration.**\n\nWe agree that point cloud estimation is a closely related topic to scene flow estimation. We will discuss this relationship in the paper, also with regard to our contributions, and add it to the related work section. A brief summary is provided below:\n\nRelated to scene flow estimation, there are some correspondence-based point cloud registration methods. Such methods separate the point cloud registration task into two stages: finding the correspondences and recovering the transformation. \n\nPPFNet [3]  and PPF-FoldNet [4]  proposed by Deng $\\textit{et al.}$ focus on finding sparse corresponding 3D local features. \nGojcic $\\textit{et al.}$ [5] propose to use voxelized smoothed density value (SDV) representation to match\n3D point clouds. \nThese methods only compute sparse correspondences and are not capable of handling dense correspondences required in scene flow tasks.\nMore related works are CoFiNet [6] and GeoTransformer [7], both of which involve finding dense correspondences employing transformer architectures. \nYu $\\textit{et al.}$ in CoFiNet [6] proposes a detection-free learning framework and find dense point correspondence in a coarse-to-fine manner. Qin $\\textit{et al.}$ in GeoTransformer [7] further improves the accuracy by leveraging the geometric information. \nSome works further introduced superpoint matching to solve the ambiguity introduced by self-attention layers.\nRoITr [8] introduces a Rotation-Invariant Transformer to disentangle the geometry and poses, and tackle point cloud matching under arbitrary pose variations. \nPEAL [9] introduced prior embedded Explicit Attention Learning model (PEAL), and for the first time explicitly inject overlap prior into Transformer to solve point cloud registration under low overlap.\nHowever, the goal of point cloud registration is not to estimate the translation vectors for each of the points, which makes our work different from these approaches.\n\n**References**\n\n[1] Sarlin P E, DeTone D, Malisiewicz T, et al. Superglue: Learning feature matching with graph neural networks[C]//CVPR2020.\n\n[2] Sun J, Shen Z, Wang Y, et al. LoFTR: Detector-free local feature matching with transformers[C]//CVPR2021.\n\n[3] Deng H, Birdal T, Ilic S. Ppfnet: Global context aware local features for robust 3d point matching[C]//CVPR2018.\n\n[4] Deng H, Birdal T, Ilic S. Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors[C]//ECCV2018.\n\n[5] Gojcic Z, Zhou C, Wegner J D, et al. The perfect match: 3d point cloud matching with smoothed densities[C]//CVPR2019.\n\n[6] Yu H, Li F, Saleh M, et al. Cofinet: Reliable coarse-to-fine correspondences for robust pointcloud registration[J]. NeurIPS2021.\n\n[7] Qin Z, Yu H, Wang C, et al. Geometric transformer for fast and robust point cloud registration[C]//CVPR2022.\n\n[8] Yu H, Qin Z, Hou J, et al. Rotation-invariant transformer for point cloud matching[C]//CVPR2023.\n\n[9] Yu J, Ren L, Zhang Y, et al. PEAL: Prior-Embedded Explicit Attention Learning for Low-Overlap Point Cloud Registration[C]//CVPR2023."}, {"review_id": "90MBdvJF5d", "reviewer": "Reviewer_QN7Z", "summary": "This paper proposes a hybrid local-global-cross transformer scene flow estimation model, achieving the state-of-the-art results on FlyingThings3D and KITTI Scene Flow datasets.", "questions": "Please refer to the weaknesses section, refute views or answer questions or improve the paper.", "limitations": "The limitations has been discussed.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This work has a mature model design based on Transformers, achieving superior results on the Flyingthings3D and KITTI Scene flow datasets.\n\n2. The structure and writing of this paper do not have major issues.", "weaknesses": "1. Scene flow estimation has been developed for many years, and the autonomous driving industry has already introduced Occupancy and Flow Prediction techniques. Researchers should not be limited to designing a toy model solely to maximize the benchmarks on FlyingThings3D and KITTI Scene Flow. These two datasets have dense point clouds and clear correspondences, which are far from practical applications. For this paper, I hope the authors can conduct more experiments on the Waymo motion data (https://waymo.com/open/data/motion/).\n\n2. Undoubtedly, Transformers will bring more powerful feature modeling capabilities and higher latency. How does the latency of this method compare to competitors? Please add this item to the main experimental table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments and suggestions. We improve the paper based on the two weaknesses pointed out. We show how we improve the paper below:\n\n**Q1: Experiments on the Waymo-open dataset.**\n\nAs suggested, we conduct additional experiments on the Waymo-open dataset for scene flow.\nThe dataset contains 798 training and 202 validation sequences. Each sequence consists of 20 seconds of 10Hz point cloud data.\nDue to the large scale of the dataset and the time limitation, we trained our model on 1/8 of the training sequences (first 100 training sequences) and tested on all 202 validation sequences.\nWe use the same parameter setting and training scheme as training on FlyingThings3D, and train for 300k iterations.\nThe results in comparison with other state-of-the-art methods are given in the following table:\n\n| method | $EPE_{3D}\\downarrow$ | $ACC_{S}\\uparrow$ | $ACC_{R}\\uparrow$ | $Outliers\\downarrow$ |\n|-----------|-----------|-----------|-----------|-----------|\n| FlowNet3D | 0.225 | 23.0 | 48.6 | 77.9 |\n| PointPWC | 0.307 | 10.3 | 23.1 | 78.6 |\n| FESTA | 0.223 | 24.5 | 27.2 | 76.5 |\n| FH-Net | 0.175 | 35.8 | 67.4 | 60.3 |\n| **GMSF** | 0.086 | 73.9 | 84.7 | 43.9 |\n\nOur proposed method already demonstrates a massive improvement over all the other methods when only trained on 1/8 of the training set. We are convinced that training on the full dataset would result in an even better performance. The results of the full training will be added to the main paper.\n\n**Q2: FLOPs, GPU memory, and Runtime.**\n\nPlease refer to the global response to all reviewers A.\n\nSince the additional results on the Waymo dataset show promising performance and all our reported memory and runtime studies are on par with previous approaches while significantly improving the performance, we hope to have fully answered all questions the reviewer has. We are open to elaborate on these points further during the discussion phase."}, {"review_id": "ymmg70lIdz", "reviewer": "Reviewer_YAmp", "summary": "This work aims to address the task of scene flow estimation for 3D point clouds. The authors propose a hybrid architecture based on local-global-cross transformers. Given as input a source and a target point cloud, first, the local transformer extracts geometric features within a patch, then the global transformer analyzes each point cloud individually using self-attention to capture the overall context, and finally the cross transformer exchanges information between the point clouds to generate the final feature representation for each point. The scene flow is predicted by performing pointwise matching with the cross similarity matrix, while occlusions are handled through a self-similarity matrix applied to the predicted scene flow. To evaluate the effectiveness of their approach, the authors conduct experiments on two benchmarks for scene flow estimation, demonstrating better performance compared to existing methods.", "questions": "Please see the Weaknesses section.\n", "limitations": "The authors discussed limitations at the end of the paper.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-  This work introduces a straightforward architecture for scene flow estimation, utilizing transformers. Without bells and whistles, the authors show that the proposed network can produce discriminative per-point features that can be robustly matched for scene flow computation.  \n-  The authors conduct extensive experiments on the FlyingThings3D and KITTI Scene Flow benchmarks in different preprocessing and occlusion settings. The results suggest that the proposed method has significant performance improvement on FlyingThings3D, while also performing competitively on KITTI Scene Flow when compared to existing methods for generalization test.  \n-  The paper is overall well written and easy to follow.", "weaknesses": "-  One concern for this work is its limited technical contributions: the hybrid network uses the well-established transformer architecture, while the scene flow is estimated by a common probabilistic point matching approach. One seemingly interesting proposal is the occlusion handling with the self-similarity matrix. However, it lacks in-depth explanations for why it helps with occlusion handling, and its ablation study is also missing in Sec. 4.5.\n-  In the generalization test on KITTI-S (Tab. 3), the proposed method exhibits a performance gap compared to PT-FlowNet [8]. More detailed analysis and explanation would be helpful for gaining a better understanding of this discrepancy.\n- For the ablation study in Tab. 4, it is unclear whether the performance saturates with eight global-cross transformer layers, and whether more layers would be beneficial or not. To provide a comprehensive assessment, it would be good to include comparisons of runtime and memory usage, as the backbone is built upon transformers, which may not scale well with more points, and the paper mentions that input point clouds are resampled to 8K points.\n-  Minor: L267, “Although we don’t” => Note that we do not\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and helpful suggestions. Below are our clarifications for the questions.\n\n**Q1a: Limited technical contributions.**\n\nThe transformer architecture and probabilistic matching approach have been studied and proven efficient previously in many NLP tasks and image understanding.\nHowever, their application in the task of scene flow hasn't been thoroughly studied.\nWe propose a \"simple and effective\" (vPo9) architecture to tackle scene flow estimation, which is proven effective on multiple scene flow benchmarks, including the new challenging Waymo-open dataset (cf. response to Q1 to QN7Z).\nMoreover, the novelty and contribution of our work is not only the architecture itself but also the evidence that it is applicable and works very well in a new field.\nSimilarly, Reviewer D7Yn commented that the proposed architecture, and the motivation behind it, are intuitive and novel, and its application to the task of scene flow seems conceptually new.\n\n**Q1b: Why occlusion handling works and can be viewed as a smoothing procedure.**\n\nPlease refer to the global response to all reviewers C.\n\n**Q1c: Ablation study on occlusion handling.**\n\nPlease refer to the global response to all reviewers D.\n\n**Q2: Performance gap compared to PT-FlowNet.**\n\nWe would like to first mention that PT-FlowNet was officially published on May 2023 with a preliminary online version available already in March (the NeurIPS submission deadline was on May 11th). For fairness and comprehensiveness we chose to include the paper and compare it with our proposed method.\n\nThe proposed method shows a significantly better performance on FlyingThings3D dataset compared to PT-FlowNet, especially in terms of $ACC_S$ and $Outlier$ (accuracy: 98.9\\% vs. 91.4\\%; outlier: 17.4\\% vs. 3.2\\%).\nWhen generalizing to the KITTI dataset, the proposed method exhibits a performance gap in terms of $EPE_{3D}$. However, the $ACC_R$ is on par with PT-FlowNet, and the other metrics only differ slightly. Besides, KITTI is a relatively small dataset with only less than 200 scenes for evaluation. \n\nOne aspect that could explain our performance gap compared to PT-FlowNet is that our proposed method is not specifically designed for the experimental setting without occlusion.\nIn the real-world scenario, there is never the case where we can get clean, non-occluded point pairs. Therefore, we aim for more general cases where occlusion exists. The generalization ability on KITTI$_o$ (see paper Table 2.) well supports our claim.\nHowever, PT-FlowNet only provides results on F3D$_s$ and KITTI$_s$ datasets.\n\nMoreover, during testing, PT-FlowNet requires 32 refinement iterations which slows down the runtime. For each of the scenes, PT-FlowNet needs 898.5 ms to process, while our method only needs 371.8 ms, more than twice as fast. Increasing the number of attention blocks in our model likely results in better performance at the cost of speed (see additional experiments on the number of transformer layers in the global response to all reviewers B.).\n\nIn summary: except for the only partially realistic KITTI$_s$ setting, where we almost match PT-FlowNet's performance, outperform them by a large margin while at the same time requiring lower computational resources.\n\n**Q3a: Ablation study on the number of transformer layers.**\n\nPlease refer to the global response to all reviewers B.\n\n**Q3b: FLOPs, GPU memory, and Runtime.**\n\nPlease refer to the global response to all reviewers A.\n\n**Q3c: Point clouds are resampled to 8K points.**\n\nFor clarification, this is a common setting for scene flow estimation from point clouds. We follow the same setting for a fair comparison.\n\n**Q4: Reformulation.**\n\nWe thank the reviewer for the suggested reformulation and will use it in the final version."}, {"review_id": "WVtQb7Gl8d", "reviewer": "Reviewer_UZxB", "summary": "The paper presents GMSF, a transformer-based method that matches dense features to estimate the scene flow from point clouds. The proposed method uses a transformer-based architecture that matches two point clouds and calculates the scene flow leveraging the cross- and self-attention modules. The paper presents experiments on FlyingThings3D and KITTI Scene Flow datasets where the proposed method consistently improves the benchmarks.", "questions": "1. As stated in Weaknesses section, how to set the KNN parameters of the method?\n2. in line 158, the paper mentions \"stable tokens\". However, the paper never defines what a \"stable token\" is.", "limitations": "I think limitations are stated adequately.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In sum, I think the solution using cross- and self-attention mechanisms are an interesting application to solve the scene-flow problem. Although, this solution has been applied to sparse feature matching (see LoFTR), I think it is also good to see it works for the scene-flow problem. Here are the details of the strengths:\n\nS1. I find the solution an interesting application of cross- and self-attention mechanisms to solve the scene flow problem. Though in general I think the novelty is not that high since the overall idea has been used for feature matching in LoFTR CVPR 2021 [36].\n\nS2. The clarity of the narrative is quite good. The description of the architecture and attention modules is clear. Also, the description of the losses make sense and are easy to understand. Thus, I think the paper can be reproducible.", "weaknesses": "Overall, I think while the paper shows another application of attention to compute 3D scene flow, I think the paper is lacking more thorough experiments and ablations. Here are the details:\n\nW1. Missing ablations. First, the paper is not showing the performance impact of the parameters of the KNN component shown in Figure 2.I am sure this is also crucial since this allows the encoder to grab local information. How to set the KNN parameters? What is the effect of this parameter in the final performance? Second. What is the behavior of setting $\\lambda$ to a different value? Why was $\\lambda=0.9$ and what is the performance of the method when varying this parameter?\n\nW2. Is the KITTI Scene Flow dataset challenging enough? Since this is a dataset for autonomous driving, the motion of the vehicle is mainly planar and thus limiting the motion degrees of freedoms (only 1 for rotation, and mostly one 1 for translations, since the car moves linearly most of the time). I think this greatly simplifies the complexity of finding correspondence of any type (e.g., scene flow) in these autonomous driving datasets.\n\nW3. Lastly, I am concerned about the novelty of the approach. I think previous works have shown that attention mechanisms are useful for matching tasks in vision, and thus I struggle to find novel components or ideas in the paper. I think the paper should discuss more in depth the novelties of the paper more in detail.\n\n----\nPost Rebuttal\n\nAfter reading the rebuttal and discussion with the authors, my concerns have been addressed and I will increase my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and helpful suggestions. Below are our clarifications for the questions.\n\n**Q1a: Ablation study of the KNN component.**\n\nFor the Point Transformer in Figure 2, we follow the setting in Zhao $et al.$ [1] and set $k$ to 16. Thanks to the reviewer's suggestion to provide an ablation study on the $k$ parameter. The results are given in the following table:\n\n| $k$ | $EPE_{3D}\\downarrow$ | $ACC_{S}\\uparrow$ | $ACC_{R}\\uparrow$ | $Outliers\\downarrow$ | $EPE_{3D}\\downarrow$ |$ACC_{S}\\uparrow$ | $ACC_{R}\\uparrow$ | $Outliers\\downarrow$ |\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|  | $all$ | $all$ | $all$ | $all$ | $nonocc$ | $nonocc$ | $nonocc$ | $nonocc$ |\n| $k=8$ | 0.050 | 89.06 | 94.25 | 14.33 | 0.027 | 94.25 | 97.50 | 10.94 |\n| $k=16$ | 0.049 | 90.08 | 94.72 | 13.08 | 0.025 | 94.98 | 97.78 | 9.87 |\n| $k=32$ | 0.049 | 89.53 | 94.42 | 13.89 | 0.026 | 94.61 | 97.58 | 10.59 |\n\nThe result confirms that the best performance is obtained for $k=16$ in the Point Transformer paper [1] (Section 4.4 Ablation Study, Number of neighbors). Setting $k$ to 8 and 32 results in a slightly worse performance.\n\n**Q1b: Ablation study for the parameter $\\gamma$ in Eq. (13).**\n\nWe test the two extreme cases: with only the first term $\\hat{V}\\_{final}$, and only the second term $\\hat{V}\\_{inter}$, respectively.\nThanks to the recommendation we could slightly improve the performance by simplifying Eq. (13) that was previously introduced in RAFT [2].\nPlease refer to the global response to all reviewers D for more details.\n\n**Q2: KITTI Scene Flow dataset not challenging enough.**\n\nIn the autonomous driving scenarios, the vehicles are mostly moving in a plane. However, the scene flow is often estimated from the view of a moving vehicle instead of a static view. The largest camera motion is yawing, but there is also a rotation, though smaller, around the other two axes. Therefore, the degrees of freedom of the relative movement of the other vehicles is not restricted to 1 translation and 1 rotation. Moreover, pedestrians, which are also important in autonomous driving scenarios, are much more complex since they are non-rigid objects.\n\nAlthough such autonomous driving scenarios are sometimes simplified by assuming the moving vehicles are rigid objects, explicitly modeling this fact oversimplifies the task and introduces a strong and potentially misleading motion prior. By contrast, our method does not exploit any such biases or prior information, thus being applicable to more general cases, as underlined by the good performance of our approach. \n\nFurthermore, we complement KITTI with the more challenging datasets, FT3D (see paper Table 2, 3) and Waymo (see response to reviewer QN7Z, Q1).\nFT3D has more complicated and less constrained object movements and is larger than the KITTI dataset. The Waymo dataset is another larger autonomous driving scene flow benchmark.\nWe achieve superior performance on both datasets compared to previous methods.\n\n**Q3: Novelty.**\n\nCross- and self-attention mechanisms proved efficient in 2D image matching [3].\nHowever, their application in the task of scene flow estimation has not been thoroughly studied yet.\nWe propose a \"simple and effective\" (vPo9) architecture to tackle scene flow estimation, which is proven effective on multiple scene flow benchmarks, including the new challenging Waymo-open dataset (cf. response to Q1 to QN7Z).\nMoreover, the novelty and contribution of our work is not only the architecture itself but also the evidence that it is applicable and works very well in a new field.\nSimilarly, Reviewer D7Yn commented that the proposed architecture, and the motivation behind it, are intuitive and novel, and its application to the task of scene flow seems conceptually new.\n\n**Q4: Definition of a \"stable token\".**\n\nWe thank the reviewer to point out our inaccurate wording. \"Stable token\" indicates that the token with the Local Point Transformer is more powerful and helps with both accuracy and robustness.\n\n**References**\n\n[1] Zhao H, Jiang L, Jia J, et al. Point transformer[C]//ICCV2021.\n\n[2] Teed Z, Deng J. Raft: Recurrent all-pairs field transforms for optical flow[C]//ECCV2020.\n\n[3] Sun J, Shen Z, Wang Y, et al. LoFTR: Detector-free local feature matching with transformers[C]//CVPR2021."}, {"review_id": "kyQkTg1GDv", "reviewer": "Reviewer_vPo9", "summary": "Previous scene flow estimation methods require complicated coarse-to-fine or recurrent architectures as a multi-stage refinement. In contrast, this paper proposes a simpler single-scale one-shot global matching to address the problem. To this end, this paper decomposes the feature extraction step via a hybrid local-global-cross transformer architecture. Extensive experiments show that the proposed method achieves SOTA performance on multiple scene flow estimation benchmarks.", "questions": "Please see the Weaknesses. If these concerns are addressed, I am willing to improve the rating.", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper proposes Global Matching Scene Flow (GMSF) and achieves state-of-the-art performance on multiple scene flow estimation benchmarks.\n2.\tThe proposed pipeline is simple and effective.\n3.\tThe authors have provided the code in the submission.", "weaknesses": "1 Some related studies have been neglected. Please compare with the previous study [cite1] in experiments. Besides, previous studies [cite2-3] need to be cited and discussed.\n\n[cite1] Lang I, Aiger D, Cole F, et al. SCOOP: Self-Supervised Correspondence and Optimization-Based Scene Flow[J]. arXiv e-prints, 2022: arXiv: 2211.14020.  \n[cite2] Li X, Kaesemodel Pontes J, Lucey S. Neural scene flow prior[J]. Advances in Neural Information Processing Systems, 2021, 34: 7838-7851.  \n[cite3] Li R., Zhang C., Lin G., Wang Z., Shen C.: Rigidflow: Self-supervised scene flow learning on point clouds by local rigidity prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer\nSociety, Los Alamitos, CA, USA (June 2022), pp. 16959–16968.  \n\n2 Compared methods are not consistent in Table 2 and Table 3. Please clarify the reason.\n\n3 The motivation of Eq. (11) is unclear. Please further clarify the reason why this term can be viewed as a smoothing procedure. \n\n4 Ablation study is not enough. I suggest the authors conduct an ablation study to Eq. (13). Specifically, two version models need to be compared, i.e., model A trained with the first term and model B trained with Eq. (13).\n\n5 The authors need to compare the FLOPs, GPU memory, and run-time of recent scene flow methods. Although the proposed method is simple and effective, the computational cost needs to be compared.\n\n6 It seems that tokenization is too complex and redundant. Specifically, Table 5 shows that DGCGG+PT achieves almost the same performance as MLP+PT. Therefore, I think only using PT is enough, and I suggest the authors conduct experiments to report the performance of GMSF with only PT as the tokenization process. In this way, GMSF would become more efficient.\n\n7 Whenever any abbreviations appear for the first time, it requires a full form. For example, GMSF and LiDAR.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and helpful suggestions. Below are our clarifications for the questions.\n\n**Q1: Related work.**\n\nWe thank the reviewers for the suggestion of the unmentioned related work. We will add the following discussion in the related work.\n\nDifferent from the proposed method, which is fully supervised and trained offline, some other works focus on runtime optimization, prior assumptions, and self-supervision.\n\nLi $et\\ al.$ [2] revisit the need for explicit regularization in supervised scene flow learning. The deep learning methods tend to rely on prior statistics learned during training, which are domain-specific. This does not guarantee generalization ability during testing. To this end, Li $et\\ al.$ proposes to rely on runtime optimization with scene flow prior as strong regularization.\nBased on [2] Lang $et\\ al.$ [1] propose to combine runtime optimization with self-supervision. A correspondence model is first trained to initialize the flow. Refinement is done by optimizing the flow refinement component during runtime. The whole process can be done under self-supervision.\nPontes $et\\ al.$ [4] use the graph Laplacian of a point cloud to force the scene flow to be \"as rigid as possible\". Same as Li $et\\ al.$ [2], this constraint can be optimized during runtime.\nLi $et\\ al.$ [3] propose a self-supervised scene flow learning approach with local rigidity prior assumption for real-world scenes. Instead of relying on point-wise similarities for scene flow estimation, region-wise rigid alignment is enforced.\n\nThe comparison with SCOOP is provided here and will be added in the experiments section.\n|Method|$EPE_{3D}\\downarrow$|$ACC_{S}\\uparrow$|$ACC_{R}\\uparrow$|$Outliers\\downarrow$|\n|-|-|-|-|-|\n|SCOOP|0.063|79.7|91.0|24.4|\n|SCOOP+|0.047|91.3|95.0|18.6|\n|**GMSF**|0.051|79.9|92.3|22.9|\n\nSCOOP only reports results on KITTI$\\_o$. The symbol $+$ indicates that all the points in the test point clouds are used during evaluation. Without the symbol indicates using the same dataset as us preprocessed by Liu $et\\ al.$ in FlowNet3D [5].\nDifferent from our proposed method, which is fully trained offline and generalizes to KITTI without fine-tuning, SCOOP combines offline training with runtime optimization. The performance is increased by utilizing more data during evaluation (SCOOP$+$). Our proposed method does not require expensive online optimization and performs slightly worse in terms of $EPE_{3D}$, which may shed light on future work. Thanks for the suggestion.\n\n**Q2: Compared methods are not consistent in Table 2 and Table 3.**\n\nThe datasets used in Table 2 and Table 3 are the same (Flyingthings3D and KITTI Scene Flow). However, the preprocessing procedures are different. We follow the preprocessing procedure from Liu $et\\ al.$ in FlowNet3D [5] to prepare the F3D$\\_o$/KITTI$\\_o$ in Table 2, and follow the preprocessing procedure from Gu $et\\ al.$ in HPLFlowNet [6] to prepare the F3D$\\_s$/KITTI$\\_s$ in Table 3. The difference is that F3D$\\_o$/KITTI$\\_o$ keeps all valid points with an occlusion mask available during training and testing, F3D$\\_s$/KITTI$\\_s$ simplifies the task by removing all occluded points (see paper L 234-240). The numbers are not consistent because of the different experimental settings.\n\nWe take the results directly from the published papers. Some of the methods only conduct experiments on one version of the datasets. For better comparison with state-of-the-art methods, we conduct experiments with both of the settings.\n\n**Q3: Motivation of Eq. (11) and why it can be viewed as a smoothing procedure.**\n\nPlease refer to the global response to all reviewers C.\n\n**Q4: Ablation study on Eq. (13).**\n\nPlease refer to the global response to all reviewers D.\n\n**Q5: FLOPs, GPU memory, and Runtime.**\n\nPlease refer to the global response to all reviewers A.\n\n**Q6: GMSF with only PT as the tokenization process.**\n\nThanks for the suggestion. We interpret the suggestion as:\nSince employing DGCNN and MLP gives pretty much the same results, which leads to the question if further simplifying the architecture, e.g. by removing MLP and only employing PT, would work.\n\nThis is not the case, since the purpose of DGCNN and MLP is to project the 3D coordinate into the high-dimensional feature space. \nThe same procedure is employed in the PT [7] original paper, where the input feature vectors are acquired by applying MLP on the 3D coordinates.\nRemoving the MLP and directly applying the PT on the 3-dimensional coordinate would result in a lack of capacity of the model.\nTo support our claim that reducing the number of channels leads to a lack of capacity of the model, we did a small ablation study on the number of channels in the table below:\n|Channels|$EPE_{3D}\\downarrow$|$ACC_{S}\\uparrow$|$ACC_{R}\\uparrow$|$Outliers\\downarrow$|$EPE_{3D}\\downarrow$|$ACC_{S}\\uparrow$|$ACC_{R}\\uparrow$|$Outliers\\downarrow$|\n|-|-|-|-|-|-|-|-|-|\n||$all$||||$nonocc$||||\n|64|0.059|87.06|93.43|17.13|0.032|92.64|96.98|13.57|\n|128|0.049|90.08|94.72|13.08|0.025|94.98|97.78|9.87|\n\n**Q7: Abbreviations.**\n\nThanks for pointing out. We've made modifications based on the comment.\n\n**References**\n\n[1] Lang I, Aiger D, Cole F, et al. Scoop: Self-supervised correspondence and optimization-based scene flow[C]//CVPR2023.\n\n[2] Li X, Kaesemodel Pontes J, Lucey S. Neural scene flow prior[J]. NeurIPS2021.\n\n[3] Li R, Zhang C, Lin G, et al. Rigidflow: Self-supervised scene flow learning on point clouds by local rigidity prior[C]//CVPR2022.\n\n[4] Pontes J K, Hays J, Lucey S. Scene flow from point clouds with or without learning[C]//3DV2020.\n\n[5] Liu X, Qi C R, Guibas L J. Flownet3d: Learning scene flow in 3d point clouds[C]//CVPR2019.\n\n[6] Gu X, Wang Y, Wu C, et al. Hplflownet: Hierarchical permutohedral lattice flownet for scene flow estimation on large-scale point clouds[C]//CVPR2019.\n\n[7] Zhao H, Jiang L, Jia J, et al. Point transformer[C]//ICCV2021."}]

Author rebuttal:
We thank the reviewers for their valuable feedback and questions.
We are happy that the reviewers find our proposed approach, and the motivation "intuitive and novel" (D7Yn).
The architecture is "straightforward without bells and whistles" (YAmp), "mature" (QN7Z), and "simple and effective" (vPo9).
The solution is an "interesting application" of cross- and self-attention mechanisms to solve the scene flow problem (UZxB).
The paper is well-written and easy to follow (D7Yn, YAmp, UZxB).
The strong performance on the standard benchmarks of scene flow estimation is highlighted by (D7Yn, QN7Z, YAmp, vPo9).

In the following, we will address the questions asked by multiple reviewers. Individual questions are addressed below the respective reviews.
The provided additional results are mostly ablation studies suggested by the reviewers that complement our existing experiments and provide a deeper insight into our approach.

**A: FLOPs, GPU memory, and Runtime (D7Yn, QN7Z, YAmp, vPo9).**

Several reviewers requested meta-information on our experiments.
Adding to our ablation results, we report runtime (ms per scene) during testing on an NVIDIA A40 GPU, FLOPs (G), Number of parameters (M), GPU memory (GB) during testing (batch size 1) and training (batch size 8) under different numbers of transformer layers in the following table:
|layers|0|2|4|6|8|10|12|16|24|
|-|-|-|-|-|-|-|-|-|-|
|Runtime|207.9|242.7|289.3|330.7|371.8|417.3|454.6|540.2|701.4|
|FLOPs|482.0|516.5|550.9|585.4|619.8|654.3|688.7|757.6|895.4|
|Parameters|1.82|2.87|3.92|4.97|6.02|7.07|8.12|10.22|14.42|
|Memory (test)|4.78|4.92|4.94|4.83|4.97|4.99|5.00|5.03|4.97|
|Memory (train)|56.9|91.7|106.2|122.3|142.3|162.3|185.5|225.5|305.6|

Expectedly, the computational effort increases with a larger number of transformer layers.
A runtime (ms) comparison with other state-of-the-art methods is shown in the following table.
The result shows that the runtime is on par with our strongest competitors: 3DFlow and PT-FlowNet (on the same NVIDIA A40 GPU).
We also report the runtime of other methods for a comprehensive comparison.
|method|GPU|Runtime|method|GPU|Runtime|
|-|-|-|-|-|-|
|FlowNet3D|TITAN V|130.8|FlowStep3D|TITAN RTX|820.8|
|HPLFlowNet|TITAN V|98.4|CamLiFlow|Tesla V100|118.0|
|PointPWC|GTX 1080Ti|117.4|CamLiPWC|Tesla V100|110.0|
|HCRF-Flow|GTX 1080Ti|228.2|CamLiRAFT|Tesla V100|216.0|
|RAFT3D|GTX 1080Ti|386.0|PV-RAFT|NVIDIA A40|740.0|
|FLOT|GTX 2080Ti|389.3|3DFlow|NVIDIA A40|121.6|
|SCTN|GTX 2080Ti|242.7|PT-FlowNet|NVIDIA A40|898.5|
|Bi-PointFlow|TITAN RTX|40.5|**GMSF**|NVIDIA A40|371.8|

**B: Ablation study on the number of transformer layers (D7Yn, YAmp).**

We thank the reviewers for pointing out that the performance may not be saturated with 8 global-cross transformer layers. We initially aimed for a method that works with limited compute resources, in this case 4 NVIDIA A100 GPUs. 
We agree that in order to understand our approach better, a larger number of layers should be evaluated. We extend our experiment with more transformer layers. The performance increases with more layers and saturates at a number of 24.
The trade-off are the compute resources as shown in the table above. Detailed results are given in the following table:
|Layers|$EPE_{3D}\downarrow$|$ACC_{S}\uparrow$|$ACC_{R}\uparrow$|$Outliers\downarrow$|$EPE_{3D}\downarrow$|$ACC_{S}\uparrow$|$ACC_{R}\uparrow$|$Outliers\downarrow$|
|-|-|-|-|-|-|-|-|-|
||$all$||||$nonocc$||||
|8|0.049|90.08|94.72|13.08|0.025|94.98|97.78|9.87|
|10|0.046|90.53|94.97|12.57|0.024|95.37|97.92|9.43|
|12|0.046|90.94|95.05|12.20|0.024|95.75|98.03|9.05|
|16|0.041|92.52|95.76|10.31|0.020|96.84|98.38|7.53|
|24|0.041|92.38|95.75|10.16|0.020|96.73|98.39|7.36|

**C: Motivation of Eq. (11) and why it can be viewed as a smoothing procedure (YAmp, vPo9).**

In addition to the explanation in L197-201, we provide further clarification in the following, which will be added to the main paper:

The self-similarity matrix $M_{self}$ bears the similarity information for each pair of points in the source point cloud. Nearby points tend to share similar features and thus have higher similarities. Multiplying $M_{self}$ with the predicted scene flow $\hat{V}_{inter}$ can be seen as an averaging procedure, where for each point, its predicted scene flow vector is updated as the weighted average of the scene flow vectors of the nearby points that share similar features.

**D: Ablation study on occlusion handling / Eq. (13) (YAmp, UZxB, vPo9).**

Given the similarities between optical flow and scene flow, we followed the previous optical flow method RAFT [1] by using Eq. (13) with increasing weights for the refined prediction.

We agree with the reviewers that an additional ablation study on the training loss in Eq. (13) provides further insights and compare three models in the following table as suggested by reviewer vPo9:
Model A, B, and C are trained with only the second term $\hat{V}\_{inter}$, only the first term $\hat{V}\_{final}$, and Eq. (13), respectively.
|Model|$EPE_{3D}\downarrow$|$ACC_{S}\uparrow$|$ACC_{R}\uparrow$|$Outliers\downarrow$|$EPE_{3D}\downarrow$|$ACC_{S}\uparrow$|$ACC_{R}\uparrow$|$Outliers\downarrow$|
|-|-|-|-|-|-|-|-|-|
||$all$||||$nonocc$||||
|A|0.150|73.11|82.91|34.11|0.047|84.37|94.12|24.71|
|B|0.045|91.43|95.38|11.61|0.025|95.36|97.95|8.99|
|C|0.049|90.08|94.72|13.08|0.025|94.98|97.78|9.87|

The result shows that the presence of $\hat{V}_{final}$ in the loss produces good results, which means the occlusion handling helps improve the performance by a large margin.

More interestingly, Model B performs slightly better than Model C. This reveals that scene flow estimation might differ from optical flow estimation regarding the required loss formulation and that supervision of intermediate results is not necessary. (Thanks for the reviewers' comments to help us improve the paper!)

**References**

[1] Teed Z, Deng J. Raft: Recurrent all-pairs field transforms for optical flow[C]//ECCV2020.

Meta-review: This paper receives unanimous recommendation of acceptance. The reviewers acknowledge

- The effective and straightforward model achitecture for scene flow estimation based on Transformer.
- The state-of-the-art performance on standard banchmarks.
- The paper is well written and easy to follow.

Before the rebuttal, some of the reviewers had concerns about FLOPs, GPU memory requirement, runtime, the effectivess of number of Transformer layers, occlusion handling, etc. The authors provided new experimental results, including results on the Waymo dataset, which have addressed the reviewers' comments.

The only limitation of this paper is the limited novelty since the global matching has been adopted for 2D image matching (SuperGlue, LoFTR, and others) and 3D point cloud registration (CoFiNet and GeoTransformer). But the reviewers and AC found it a minor issue.

The AC thus recommends an acceptance as a poster.
Decision: Accept (poster)