SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For occlusions in the source point cloud, the matching would fail under the assumption that there exists a matching point in the target point cloud. We avoid this by employing a self-similarity matrix that utilizes information from the source point cloud. The self-similarity matrix  $M_{\mathrm{self}}$  bears the similarity information for each pair of points in the source point cloud. Nearby points tend to share similar features and thus have higher similarities. Multiplying  $M_{\mathrm{self}}$  with the predicted scene flow  $\hat{V}_{\mathrm{inter}}$  can be seen as a smoothing procedure, where for each point, its predicted scene flow vector is updated as the weighted average of the scene flow vectors of the nearby points that share similar features. This also allows the network to propagate the correctly computed non-occluded scene flow estimation to its nearby occluded areas, which gives
$$
\hat {V} _ {\text {f i n a l}} = M _ {\text {s e l f}} \hat {V} _ {\text {i n t e r}}. \tag {11}
$$
3.3 Loss Formulation
Let  $\hat{V}$  be the estimated scene flow and  $V_{gt}$  be the ground truth. We follow CamLiFlow [25] and use a robust training loss to supervise the process, given by
$$
\mathcal {L} _ {\hat {V}} = \sum_ {i} \left(\left\| \hat {V} _ {\text {f i n a l}} (i) - V _ {\mathrm {g t}} (i) \right\| _ {1} + \epsilon\right) ^ {q}, \tag {12}
$$
where  $\epsilon$  is set to 0.01 and  $q$  is set to 0.4.
The proposed method is implemented in PyTorch. Following previous methods [14, 55], the numbers of points  $N_{1}$  and  $N_{2}$  are both set to 8192 during training and testing, randomly sampled from the full set. We perform data augmentation by randomly flipping horizontally and vertically. We use the AdamW optimizer with a learning rate of  $2 \cdot 10^{-4}$ , a weight decay of  $10^{-4}$ , and OneCycleLR as the scheduler to anneal the learning rate. The training is done for 600k iterations with a batch size of 8.
For a fair comparison we follow previous work [14, 46, 55] and evaluate the proposed method with the accuracy metric  $EPE_{3D}$ , and the robustness metrics  $ACC_{S}$ ,  $ACC_{R}$  and Outliers.  $EPE_{3D}$  is the 3D end point error  $\| \hat{V} - V_{gt} \|_2$  between the estimated scene flow and the ground truth averaged over each point.  $ACC_{S}$  is the percentage of the estimated scene flow with an end point error less than 0.05 meter or relative error less than  $5\%$ .  $ACC_{R}$  is the percentage of the estimated scene flow with an end point error less than 0.1 meter or relative error less than  $10\%$ . Outliers is the percentage of the estimated scene flow with an end point error more than 0.3 meter or relative error more than  $10\%$ .
4.3 Datasets
The proposed method is tested on three established benchmarks for scene flow estimation.
FlyingThings3D [30] is a synthetic dataset of objects generated by ShapeNet [2] with randomized movement rendered in a scene. The dataset consists of 25000 stereo frames with ground truth data.
KITTI Scene Flow [31] is a real world dataset for autonomous driving. The annotation is done with the help of CAD models. It consists of 200 scenes for training and 200 scenes for testing.
Both datasets have to be preprocessed in order to obtain 3D points from the depth images. There exist two widely used preprocessing methods to generate the point clouds and the ground truth scene flow, one proposed by Liu et al. in FlowNet3D [27] and the other proposed by Gu et al. in HPLFlowNet [14]. The difference between the two approaches is that Liu et al. [27] keeps all valid points with an occlusion mask available during training and testing. Gu et al. [14] simplifies the task by removing all occluded points. We denote the datasets preprocessed by Liu et al. in FlowNet3D as  $\mathrm{F3D}_o / \mathrm{KITTI}_o$  and by Gu et al. in HPLFlowNet as  $\mathrm{F3D}_s / \mathrm{KITTI}_s$ . In the original setting from [14, 27], the FlyingThing3D dataset  $\mathbf{F3D}_s$  consists of 19640 and 3824 stereo scenes for training and testing, respectively.  $\mathbf{F3D}_o$  consists of 20000 and 2000 stereo scenes for training and testing, respectively. For the KITTI dataset,  $\mathbf{KITTI}_s$  consists of 142 scenes from the training set, and  $\mathbf{KITTI}_o$  consists of 150 scenes from the training set. Since there is no annotation available in the testing set of KITTI, we follow previous methods to test the generalization ability of the proposed method without any fine-tuning on  $\mathbf{KITTI}_s$  and  $\mathbf{KITTI}_o$ . For better evaluation and analysis, we additionally follow the setting in CamLiFlow [25] to extend  $\mathrm{F3D}_s$  to include occluded points. We denote this as  $\mathbf{F3D}_c$ .
Waymo-Open Dataset [44] is a large-scale autonomous driving dataset. We follow [7] to preprocess the dataset to create the scene flow dataset. The dataset contains 798 training and 202 validation sequences. Each sequence consists of 20 seconds of  $10\mathrm{Hz}$  point cloud data. Different from [7] which only contains 100 sequences, we trained and tested our model on the full dataset.
4.4 State-of-the-art Comparison
We compare our proposed method GMSF with state-of-the-art methods on FlyingThings3D in different settings. Table 1 shows the results on  $\mathrm{F3D}_c$ . Evaluation metrics are calculated over both non-occluded points and all points. Among all the methods, including methods with the corresponding stereo images as additional input [46], or even with optical flow as additional ground truth for supervision [25, 26], our proposed method achieves the best performance both in terms of accuracy and robustness.
To give a fair comparison with previous methods we report results on  $\mathrm{F3D}_o$  and  $\mathrm{F3D}_s$  with generalization to  $\mathrm{KITTI}_o$  and  $\mathrm{KITTI}_s$  in Table 2 and Table 3. The proposed method achieves the best performance on both  $\mathrm{F3D}_o$  and  $\mathrm{F3D}_s$ , surpassing other state-of-the-art methods by a large margin. The generalization ability of the proposed model on  $\mathrm{KITTI}_o$  and  $\mathrm{KITTI}_s$  also achieves state of the art.
We further conduct experiments on the Waymo-Open dataset. We train and test on the full dataset with 798 training and 202 testing sequences. Comparisons with state of the art are given in Table 4.
Table 1: State-of-the-art comparison on  $\mathbf{F3D}_c$ . The input modalities are given as a reference. Our method with only 3D points as input outperforms all the other state-of-the-art methods on all metrics.
Method	Input	EPE3D ↓ non-occluded	ACC5 ↑ non-occluded	EPE3D ↓ all	ACC5 ↑ all
FlowNet3D [27] CVPR'19	Points	0.158	22.9	0.214	18.2
RAFT3D [46] CVPR'21	Image+Depth	-	-	0.094	80.6
CamLiFlow [25] CVPR'22	Image+Points	0.032	92.6	0.061	85.6
CamLiPWC [26] arxiv'23	Image+Points	-	-	0.057	86.3
CamLiRAFT [26] arxiv'23	Image+Points	-	-	0.049	88.4
GMSF(ours)	Points	0.022	95.9	0.040	92.6
Table 2: State-of-the-art comparison on  $\mathbf{F3D}_o$  and  $\mathbf{KITTI}_o$ . The models are only trained on  $\mathrm{F3D}_o$  prepared by [27] with occlusions. Testing results on  $\mathrm{F3D}_o$  and  $\mathrm{KITTI}_o$  are given.
Method	F3DO	KITTIO
EPE3D	↓ ACCS	↑ ACCR	↑ Outliers	↓ EPE3D	↓ ACCS	↑ ACCR	↑ Outliers
FlowNet3D [27]	0.157	22.8	58.2	80.4	0.183	9.8	39.4	79.9
HPLFlowNet [14]	0.168	26.2	57.4	81.2	0.343	10.3	38.6	81.4
PointPWC [55]	0.155	41.6	69.9	63.8	0.118	40.3	75.7	49.6
FLOT [34]	0.153	39.6	66.0	66.2	0.130	27.8	66.7	52.9
CamLiPWC [26]	0.092	71.5	87.1	37.2	-	-	-	-
CamLiRAFT [26]	0.076	79.4	90.4	27.9	-	-	-	-
Bi-PointFlow [3]	0.073	79.1	89.6	27.4	0.065	76.9	90.6	26.4
RAFT3D [46]	0.064	83.7	89.2	-	-	-	-	-
3DFlow [49]	0.063	79.1	90.9	27.9	0.073	81.9	89.0	26.1
SCOOP+ [18]	-	-	-	-	0.047	91.3	95.0	18.6
GMSF(ours)	0.022	95.0	97.5	5.6	0.033	91.6	95.9	13.7
Table 6 shows the results of different numbers of global-cross transformer layers. While our approach technically works even without global-cross transformer layers, the performance is significantly worse compared to using two or more layers. This shows that only incorporating local information for the feature representation is insufficient for global matching. Moreover, the capacity of the network improves with the number of layers and achieves the best performance at 10 layers.
Table 7 shows the importance of different components in the tokenization process. We tried different methods, DGCNN [51], PointNet [35], and MLP, to map the 3D coordinates of the points into the high-dimensional feature space. For each of these mapping methods, the influence of the Local Point Transformer [63] is tested. When the local transformer is present, the metrics are similar with different mapping strategies, which demonstrate the effectiveness of the proposed local-global-cross transformer architecture. In the absence of the local transformer, the performance remains comparable with DGCNN for mapping but drops significantly with PointNet or MLP, which indicates the necessity of local information encoded in the tokenization step.
Table 8 gives the ablation study on feature dimensions. The default number of feature dimensions is 128 in our model. Reducing the number of feature dimensions leads to a lack of capacity of the model.
Table 3: State-of-the-art comparison on  $\mathbf{F3D}_s$  and  $\mathbf{KITTI}_s$ . The models are only trained on  $\mathrm{F3D}_s$  prepared by [14] without occlusions. Testing results on  $\mathrm{F3D}_s$  and  $\mathbf{KITTI}_s$  are given.
Method	F3DS	KITTIS
EPE3D↓	ACC5↑	ACC4↑	Outliers↓	EPE3D↓	ACC5↑	ACC4↑	Outliers↓
FlowNet3D [27]	0.1136	41.25	77.06	60.16	0.1767	37.38	66.77	52.71
HPLFlowNet [14]	0.0804	61.44	85.55	42.87	0.1169	47.83	77.76	41.03
PointPWC [55]	0.0588	73.79	92.76	34.24	0.0694	72.81	88.84	26.48
FLOT [34]	0.0520	73.20	92.70	35.70	0.0560	75.50	90.80	24.20
HRCF-Flow [20]	0.0488	83.37	95.07	26.14	0.0531	86.31	94.44	17.97
PV-RAFT [53]	0.0461	81.69	95.74	29.24	0.0560	82.26	93.72	21.63
FlowStep3D [17]	0.0455	81.62	96.14	21.65	0.0546	80.51	92.54	14.92
RCP [13]	0.0403	85.67	96.35	19.76	0.0481	84.91	94.48	12.28
SCTN [19]	0.0380	84.70	96.80	26.80	0.0370	87.30	95.90	17.90
CamLiPWC [26]	0.0320	92.50	97.90	15.60	-	-	-	-
CamLiRAFT [26]	0.0290	93.00	98.00	13.60	-	-	-	-
Bi-PointFlow [3]	0.0280	91.80	97.80	14.30	0.0300	92.00	96.00	14.10
3DFLOW [49]	0.0281	92.90	98.17	14.58	0.0309	90.47	95.80	16.12
PT-FlowNet [11]	0.0304	91.42	98.14	17.35	0.0224	95.51	98.38	11.86
GMSF(ours)	0.0090	99.18	99.69	2.55	0.0215	96.22	98.25	9.84
Table 4: State-of-the-art comparison on Waymo-Open dataset.
Method	EPE3D ↓	ACCS ↑	ACCR ↑	Outliers ↓
FlowNet3D [27]	0.225	23.0	48.6	77.9
PointPWC [55]	0.307	10.3	23.1	78.6
FESTA [50]	0.223	24.5	27.2	76.5
FH-Net [7]	0.175	35.8	67.4	60.3
GMSF(ours)	0.083	74.7	85.1	43.5
Table 5: Meta-information.
Runtime(ms)	417.3
FLOPs(G)	654.32
Parameters(M)	7.07
Memory (test)(GB)	4.99
Memory (train)(GB)	162.3
Table 6: Ablation study on the number of global-cross transformer layers on  $\mathbf{F3D}_c$ . The influence of the number of global-cross transformer layers is tested. The best performance is gained at 10 transformer layers.
Layers	EPE3Dall	ACCSall	ACCRall	Outliersall	EPE3Dnon-occ	ACCSnon-occ	ACCRnon-occ	Outliersnon-occ
0	0.212	39.01	63.59	66.51	0.132	43.95	70.24	62.92
2	0.075	79.02	90.22	25.64	0.047	84.67	94.23	22.07
4	0.055	87.37	93.76	16.39	0.032	92.01	96.84	13.41
6	0.050	89.32	94.60	14.23	0.029	93.46	97.33	11.54
8	0.045	91.22	95.25	12.11	0.025	94.91	97.70	9.68
10	0.040	92.64	95.84	10.34	0.022	95.94	98.06	8.13
12	0.043	91.95	95.57	11.12	0.024	95.40	97.88	8.81
14	0.045	91.66	95.41	11.54	0.025	95.19	97.78	9.21
16	0.044	91.74	95.51	11.33	0.025	95.38	97.93	8.91
4.6 FLOPs, GPU memory, and Runtime.
We report meta-information on our experiments: runtime (ms per scene) during testing on an NVIDIA A40 GPU, FLOPs (G), Number of parameters (M), GPU memory (GB) during testing (batch size 1) and training (batch size 8) with 10 transformer layers and 128 feature dimensions in Table 5.
4.7 Visualization
Figure 3 shows a visualization of the GMSF results on two samples from the FlyingThings3D dataset. Red and blue points represent the source and the target point clouds, respectively. Green points
Table 7: Ablation study on the components of tokenization on  $\mathbf{F3D}_c$ . The influence of using different backbones and the presence of a local transformer is tested. The results show that as long as there is local information (DGCNN / Point Transformer) present in the tokenization process, the performance remains competitive. On the other hand, using only PointNet or MLP for tokenization, the performance drops significantly.
Backbone	PT	EPE3Dall	ACCSall	ACCRall	Outliers all	EPE3Dnon-occ	ACCSnon-occ	ACCRnon-occ	Outliers ↓ non-occ
DGCNN	✓	0.040	92.64	95.84	10.34	0.022	95.94	98.06	8.13
DGCNN	0.052	89.68	94.37	13.71	0.030	93.74	97.14	11.00
PointNet	✓	0.043	92.22	95.80	10.86	0.024	95.65	98.04	8.63
PointNet	0.063	86.76	93.06	16.67	0.037	91.45	96.31	13.51
MLP	✓	0.043	91.81	95.48	10.21	0.023	95.43	97.84	7.75
MLP	0.060	88.08	93.33	14.11	0.035	92.69	96.55	10.83
Table 8: Ablation study on the number of feature dimensions on  $\mathbf{F3D}_c$ . The performance decreases as the number of feature dimensions drops.
dim	EPE3Dall	ACCStall	ACCRall	Outliersall	EPE3Dnon-occ	ACCStnon-occ	ACCRnon-occ	Outliersdownnon-occ
32	0.073	83.04	91.32	21.07	0.044	88.36	95.07	17.56
64	0.051	89.64	94.57	13.68	0.029	93.79	97.32	10.93
128	0.040	92.64	95.84	10.34	0.022	95.94	98.06	8.13
represent the warped source point cloud toward the target point cloud. As we see in the figure, the blue points align very well with the green points, which demonstrates the effectiveness of our method.
Figure 3: Visualization results on FlyingThings3D. Two scenes from the FlyingThings3D dataset are given. Red, blue, and green points represent the source, target, and warped source point cloud, respectively. Part of the point cloud is zoomed in for better visualization.
We propose to solve scene flow estimation from point clouds by a simple single-scale one-shot global matching, where we show that reliable feature similarity between point pairs is essential and sufficient to estimate accurate scene flow. To extract high-quality feature representations, we introduce a hybrid local-global-cross transformer architecture. Experiments show that both the presence of local information in the tokenization step and the stack of global-cross transformers are essential to success. GMSF shows state-of-the-art performance on the FlyingThings3D, KITTI Scene Flow, and Waymo-Open datasets, demonstrating the effectiveness of the method.
Limitations: The global matching process in the proposed method needs to be supervised by the ground truth, which is difficult to obtain in the real world. As a result, most of the supervised scene flow estimations are trained on synthetic datasets. We plan to extend our work to unsupervised settings to exploit real data.
Acknowledgements: This work was partly supported by the Wallenberg Artificial Intelligence, Autonomous Systems and Software Program (WASP), funded by Knut and Alice Wallenberg Foundation, and the Swedish Research Council grant 2022-04266; and by the strategic research environment ELLIIT funded by the Swedish government. The computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at C3SE partially funded by the Swedish Research Council grant 2022-06725, and by the Berzelius resource, provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.