DiffSF: Diffusion Models for Scene Flow Estimation
Authors: Yushan Zhang
Bastian Wandt
Maria Magnusson
Michael Felsberg
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Scene Flow Estimation
Denoising Diffusion Models
Uncertainty
Primary area: machine_vision

Abstract: Scene flow estimation is an essential ingredient for a variety of real-world applications, especially for autonomous agents, such as self-driving cars and robots. While recent scene flow estimation approaches achieve reasonable accuracy, their applicability to real-world systems additionally benefits from a reliability measure. Aiming at improving accuracy while additionally providing an estimate for uncertainty, we propose DiffSF that combines transformer-based scene flow estimation with denoising diffusion models. In the diffusion process, the ground truth scene flow vector field is gradually perturbed by adding Gaussian noise. In the reverse process, starting from randomly sampled Gaussian noise, the scene flow vector field prediction is recovered by conditioning on a source and a target point cloud. We show that the diffusion process greatly increases the robustness of predictions compared to prior approaches resulting in state-of-the-art performance on standard scene flow estimation benchmarks. Moreover, by sampling multiple times with different initial states, the denoising process predicts multiple hypotheses, which enables measuring the output uncertainty, allowing our approach to detect a majority of the inaccurate predictions. The code is available at https://github.com/ZhangYushan3/DiffSF.

DiffSF: Diffusion Models for Scene Flow Estimation
Yushan Zhang
Bastian Wandt
Maria Magnusson
Michael Felsberg
Linköping University
(firstname lastname) @eli.le@luuee.
Abstract
Scene flow estimation is an essential ingredient for a variety of real-world appli- cations, especially for autonomous agents, such as self-driving cars and robots. While recent scene flow estimation approaches achieve reasonable accuracy, their applicability to real-world systems additionally benefits from a reliability mea- sure. Aiming at improving accuracy while additionally providing an estimate for uncertainty, we propose DiffSF that combines transformer-based scene flow estimation with denoising diffusion models. In the diffusion process, the ground truth scene flow vector field is gradually perturbed by adding Gaussian noise. the reverse process, starting from randomly sampled Gaussian noise, the scene flow vector field prediction is recovered by conditioning on a source and a target point cloud. We show that the diffusion process greatly increases the robustness predictions compared to prior approaches resulting in state-of-the-ars art perfor- mance standard scene flow estimation benchmarks. Moreover, by sampling multiple times with different initial states, the denoising process predicts mul- tiple hypotheses, which enables measuring the output uncertainty, allowing approach to detect a majority of the inaccurate predictions. The code is available at https://gituub.com//haaa Yisfa
Introduction
Scene flow estimation an important research topic in computer vision with applications in various fields, such as autonomous driving 25 and robotics (30 Given source and target point cloud, the objective is to estimate a scene flow vector field that maps each point in the source point cloud to the target point cloud. Many studies on scene flow estimation aim enhancing accuracy and substantial progress has been made particularly on clean, synthetic datasets. However, real-world data contains additional challenges such as severe occlusion and noisy input, thus requiring a high level of robustness when constructing models for scene flow estimation.
Recently, Denoising Diffusion Probabilistic Models (DDPMs) have not only been widely explored in image generation 12 28 but also analysis tasks, e.g. detection 3. classification [T] segmenta-
tion 10) optical flow 29 human pose estimation [33, point cloud registration [14] etc. Drawing
inspiration from the recent successes of diffusion models in regression tasks and recognizing their potential compatibility with scene flow estimation, we formulate scene flow estimation as diffusion process following DDPMs [12 as shown in Figure [ The forward process initiates from the ground truth scene flow vector field and gradually introduces noise to it. Conversely, the reverse process is conditioned on the source and the target point cloud and is tasked to reconstruct the scene flow vector field based on the current noisy input. To learn the denoising process, a new network is proposed inspired by state-of-the-an scene flow estimation methods FLOT 26) and GMSF 43
Previous methods 43 5 36 4 usually suffer from inaccuracies when occlusions occur or when dealing with noisy inputs. During inference, based on the fixed parameters learned during training, they cannot provide information about their inaccurate predictions, which might lead to problems in safety-critich downstream tasks, Our proposed method approaches this problem in two aspects:
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Pouns + Vp
Purpet
Prpure +
Porpre
Pounne + Veei
Parget
Prouce Vig
Porpee
(V Saurree Ptarget)
VT
V
Vz-1
VO
q(v.VV.-))
Diffusion process. In the forward process, we start from a ground truth scene flow vector field Vo and gradually add noise to it until we reach VT, which completely Gaussian noise. In the reverse process, we recover the scene flow vector field Vo from the randomly sampled noisy vector field Vr conditioned on the source point cloud P Psource and the target point cloud ttagett
Figure I:
First, denoising diffusion models are capable of handling noisy data by modeling stochastic processes. The noise caused by sensors the real world is filtered out, which allows the model to focus on learning underlying patterns. By learning feature representations that are robust to noise, the prediction accuracy is improved. Second, since the diffusion process introduces randomness into inherently deterministic prediction task, it can provide a measure of uncertainty for each prediction by averaging over a set of hypotheses, notably without any modifications to the training process. multiple benchmarks, Waymo-Open [33], demonstrate state-of-the-as performance of our proposed method. Furthermore, demonstrate that the predicted uncertainty correlates with the prediction error, establishing it as reasonable measure that can be adjusted to the desired certainty level with a simple threshold value.
Extensive experiments
Thingshh (24), KITTI Scene Flow (25), and
To summarize, our contributions are: (1) We introduce DifTSF, leveraging diffusion models to solve the full scene flow estimation problem, where the inherent noisy property of the diffusion process filters out noisy data, thus, increasing the focus on learning the relevant patterns. (2) DiffSF introduces randomness to the scene flow estimation task, which allows us to predict the uncertainty of the estimates without being explicitly trained for this purpose. (3) We develop novel architecture that combines transformers and diffusion models for the task scene flow estimation, improving both accuracy and robustness for variety of datasets.
2 Related Work
Scene Flow Estimation has rapidly progressed since the introduction of FlyingThings3D 24, KITTI
Scene Flow (25), and Waymo-Open (33] benchmarks. Many existing methods 2 23 25 27
31 35
42] assume scene objects are rigid and break down the estimation task into sub-tasks involving object detection or segmentation, followed by motion model fitting. While effective for autonomous driving scenes with static background and moving vehicles, these methods struggle with more complex scenes containing deformable objects, and their non-differentiable components impede end-to-end training without instance-level supervision. Recent advancements in scene flow estimation focus on end-to-end trainable models and are categorized into encoder-decoder, coarse-to-fine, recurrent, soft methods. Encoder-decoder techniques,
correspondence methods, and runtime optimization-basee exemplified by FlowNet3D 22 39] and HPLFlowNet (9) utilize neural networks learn scene flow by adopting an hourglass architecture. Coarse-to to-ee-ttofooee to-fine methods, such as PointPWC-Ne [41]. progressively estimate motion from coarse to fine scales, leveraging hierarchical feature extraction and warping. Recurrent methods like FlowStep3D [17]. PV-RAFT 40], and RAFT3D 34 iteratively refine the estimated motion, thus enhancing accuracy. Some approaches like FLOT 26) STCN
and GMSF [3 frame scene flow estimation an optimal transport problem, employing convolutional layers and point transformer modules for correspondence computation. Different from the previously mentioned methods, which are fully trained and supervised offline, the runtime optimization-baa -ased

---

methods 19 20 6 are optimized during the evaluation time based on each pair of inputs. While these methods have the advantage of without the need for training datasets, it also means that they can not take advantage of large-scale training datasets. Due to the online optimization, they also suffer from slow inference speed. Moreover, most of them focus only on autonomous driving scenes. On the other hand, we aim to estimate the scene flow of more general scenarios. Our proposed method takes the current state-of-the-an soft correspondence method GMSF [43] as baseline. Given the fact that being able to indicate uncertainty of the estimation is important feature for safety-critical downstream tasks, we propose to leverage the diffusion models for this purpose, whose ability of uncertainty indication has been proven by other relevant research areas
29
Diffusion Models for Regression. Diffusion models have been widely exploited for image genera- tion [12 28]. Beyond their capacity to generate realistic images and videos. researchers have also explored their potential to approach regression tasks. CARD [11] introduces a classífication and regression diffusion model to accurately capture the mean and the uncertainty of the prediction. Dif- fusionDet formulates object detection as a denoising diffusion process from noisy boxes to object boxes. Baranchuk et al. [ employ diffusion models for semantic segmentation with scarce labeled data. DiffusionInst [1O] depicts instances as instance-aware filters and casts instance segmentation as denoising process from noise to filter. Jiang et al. [44] introduce diffusion models point cloud registration that operates on the rigid body transformation group. Recent research on optical flow and depth estimation [29] shows the possibility of using diffusion models for dense vision tasks. While there have been attempts to employ diffusion models for scene flow estimation [2], they mainly focus refining initial estimation. On the contrary, our goal is construct model to estimate the full scene flow vector field instead of a refinement plug-in module. To the best of our knowledge, are the first to propose using diffusion models to estimate the full scene flow directly from two point clouds.
3 Proposed Method
3.1 Preliminaries
Scene Flow Estimation. Given a source point cloud P source E R and target point cloud P Ptarget E IR Nax3 where N₁ and N2 are the number of points in the source and the target point cloud
respectively, the objective is to estimate scene flow vector field V R-Mx3 x3 that maps each source point to the correct position in the target point cloud.
Diffusion Models. Inspired by non-equilibrium thermodynamies, diffusion models 12 32
a class of latent variable (I1, .T) models of the form (=)))) = latent variables are of the same dimensionality as the input data 3i (any dimensionality). The joint distribution po (x0:r) is also called the reverse process
where the
p=( = p(=T) Po(::-1a)) Pe(z)-1lzz) = (: t), Ee(I,,
(I)
The approximate posterior ( is called the forward process, which is fixed to Markov chain that gradually adds noise according to a predefined noise scheduler 31.T
(x1r) I q(11/==-1), q(Fe/Z1-1)
Nax; VI B, 3,1).
(2)
The training is performed by minimizing variational bound on the negative log- liglliillhoodd
Eg[ po < -E,--og, log po(xor))
(KL (g( ( (o)llp(ET))
(3)
+ + DKL (q(x111111) xo) [po - log po (x (x)))
where DKL. denotes the Kullbac--Leible divergence.
3.2 Scene Flow Estimation as Diffusion Process
We formulate the scene flow estimation task as a conditional diffusion process that is illustrated in The forward process starts from the ground truth scene flow vector field Vo and ends at
Figure [
3

---

pure Gaussian noise Vr by gradually adding Gaussian noise to the input data as in Eq. 2 that Bg is small, q(V, (V
Given
1-1 Eq. 2 has a closed form [2]
Vo) = a vo. Vo. (1 T),
(4)
where : :T 11 - 3g). The reverse process predicts the ground truth Vo from the noisy input Ve conditioned on both the source point cloud P souuree and the target point cloud P Ptagget.
Po Vo(V-11Ve P
source: P Ptarget) NV ([V.-1: 140 (Ve, P sources P Ptarget), I).
(5)
The forward process posterior is tractable when conditioned on Vo.
(V Vo) = -V (V,, V.).
(6)
where i4,(2, t. Vo) vāu- Vo ya. Vr, and B. 1- Br. Minimizing the variational bound in Eq. and He (o(VV, Peorce:: P carget)) Since V, is constructed from Vo by a predefined fixed noise scheduler BI.T, the training objective is further equivalent to learning Vo by neural network fo( (V. P source, P t. The training loss can be written
I-a: 1-a
1-a;
( breaks down to minimizing the difference between (V,, Vo)
(V P souurce Ptarget) - Voll.
where the neural network (V., P. sourcee P Ptarget) takes the current noisy input Vt, the source point
cloud Psource. and the target point cloud P Ptarget as input and output V predd. which an prediction of Vo- The detailed architecture of fo is presented in section 3.3 The reverse process in Eq. (3) can be
rewritten by replacing lso with fo
Po vooV-11VV 1 P Psource: P Ptarget) = iV fo(Ve P Psources P Ptarget) I).
(8)
During inference, starting from randomly sampled Gaussian noise Vr, Vo is reconstructed with the model Se according to the reverse process in Eq. ( The detailed training and sampling algorithms are given in Algorithm and Algorithm 2
Algorithm 1: Training
Algorithm 2: Sampling
repeat
Vr (0, I); 2 for 1 = T..... do
Vo g( Va), ON I):
No Uniform( I, ...., T);
3
estimate V prod Se.V., c, P Poource Piarses): 1: (0,
V. Va Vo + vT- dre;
ostimate preed fV.... V. Peouron Praret)
elsc: Z=c:=00
V.-1 = A... V pred) +
uptimize loss: &, luss(V prod, Vo):
until converged:
7 return Vo:
3.3 Architecture
To train the diffusion process with Eq. ( we need to design the neural network to predict Vo. i.e. the ground truth scene flow vector field. The reverse process with the detailed architecture of V pred fo(V,, P Poource Pt targge)) is given in Figure 2 We take the state-of-the-at method GMSF our baseline. All the building blocks, Feature Extraction, Local-Global-Cross Transformer, and Global Correlation are the same as in GMSF 4 We modify the model architecture of GMSF following the recent work (26, g [7] of scene flow estimation by adding initial estimation before the final prediction. More specifically, the source point cloud P source E R N1 is first warped with V, IR x3 The warped source point cloud and the target point cloud sent to the Feature Extraction block to expand the three-dimensional coordinate into higher-dimensional features for each point. Based the similarities between point pairs in the warped source and the target point cloud, Global Correlation is applied to compute an initial estimation Vinit E N We then warp the source point cloud P source N1 x3 with the initial estimation Vinit E x33, The same Feature Extraction block is applied on both the warped source point cloud and the target point cloud, but with different weights than the previous block. A Local-Global--e -rols-Cosss Transformer is then applied to the higher- dimensoaa features to get more robust and reliable feature representation for each point. The output features are then sent into the Global Correlation block to get the final prediction V "rrdd E N; x3, The detailed architecture of Feature Extraction, Groal--looll-rooss Transformer, and Global Correlation is given in the following paragraphs using the same notation GMSF (43)
43 as

---

source
Denoising Block
Vo
P Ptarget
Po VV P source Ptarget
Viniei initial estimation ppeed: final prediction
Legend
source
Conditioning
Feature Extraction
Feature Extraction
Reverse Process
Local- Global-Cros Cross
Global Correlation
Global Correlation
V
Denoising Block
pred
Transformer
Element-wise Addition
Feature Extraction
Feature Extraction
P Parpet
Vc-1 = Fee preed) +
Figure 2: The reverse process with detailed denoising block for scene flow estimation. The denoising block takes the current noisy input V.. the source point cloud P source. and the target point cloud P target as input. The output V pred is the denoised scene flow prediction. Shared weights for the feature extraction are indicated in the same color.
Feature Extraction The three-dimensional coordinate for each point is first projected into a higher feature dimension x E Rlxd xx by the off-the-shelf feature extraction backbone DGCNN [38]. Each layer of the network can be written
max h(x,, X, x;), x; Ex)((()
where and denote the index of a single point in the point cloud. X, E Nii denotes the neighboring points of point x; found by a neinaorrtt (KNN) algorithm. The number of l is set to 16. are first concatenated together along the feature
The point feature x; and the edge feature X,- dimension and then passed through neural network h. h consists of sequence of linear layer, batch normalization, and leaky ReLU layer. The output feature dimension d is set to 128. The maximum value of the & nearest neighbors is taken as the output. Multiple layers are stacked together to get the final feature representation x
Docal-Global-Croo Transformer takes the output high-dimensional features x! E Rlxd to learn more robust and reliable feature representations,
as input
x Ex, Ex()) (xp((x))) (x(x)) + @ (an(x)) +
(10)
x = Exx Eas (x)), th (x) (x,())))
(11)
= (x²)) ( (xx))) aa (x.).
(12)
where local, global, and cross transformers are given in Eq. TO denote linear layers to generate the query, key, and value. The indices 'g. and 'c indicate local transformer, global transformer, and cross transformer, respectively. For the local transformer, ? sequence of linear layer, ReLU, linear layer, and softmax. 8 is the relative positional embedding that gives the information of the 3D coordinate distance between x; and Xj. @ denotes element- wise multiplication. The output x further processed by linear layer and residual connection from the input before being sent to the global transformer. For the global and cross transformer,
respectively. yp, th, and
Xi = =P=Paorree + (V, or Vinit) E N x3 and Xr = P Prarget E R2Nxx3 represent the warped source point cloud and the target point cloud, respectively. (.) denotes the sealar product. The output of the global and cross transformer is further processed by linear layer. a layer normalization, and residual connection from the input. A feedforward network with a multilayer perceptron and layer normalization is applied to the output of the cross transformer to aggregate information. To acquire more robust feature representations, the global-cross transformers are stacked and repeated multiple times (14 times in experiment). For simplicity, of X1. The features Xr are computed by the same procedure. The output point features xf and x} for each point cloud are stacked together to form feature matrices F, E N. and F2 N2x
only give the equations for learning the features
5

---

Global Correlation predicts the scene flow vector solely based on two feature similarity matrices, cross feature similarity matrix Meross E N and self feature similarity matrix Maelr E IR" N1 x
Meress = softmax (F, F /Vd).
(13)
M softmax(W,(F))))) (FI)*
(14)
where Wg and W3 are linear projections. d is the feature dimensions. The softmax is taken over the second dimension of the matrices. The cross feature similarity matrix Mcross E RN XN2 encodes the feature similarities between all the points in the source point cloud P source and all the points in the target point cloud P target-- The self feature similarity matrix M E R x Ni encodes the feature similarities between all points in the source point cloud P source- The global correlation is performed by matching process guided by the cross feature similarity matrix followed by a
smoothing procedure guided by the self feature similarity matrix
M (Meross P Ptarget - P )source).
(15)
We follow GMSF and employ a robust loss defined as
L E. prea (1)
V (z)
+ c),,
(16)
where pred is the output prediction of the neural network, i.e. fo(Ve, Psource: Ptarget) in Eq. 7 V ge denotes the ground truth scene flow vector field i.c. Vo in Eq. (7). is the index of the points.
is set to 0.01 and q is set to 0.4.
4 Experiments
4.1
Implementation Details
We use the Adam W optimizer and weight decay of 1 x 10-1, The initial learning rate is set to 4 x 10-1 for FlyingThings3ID (24 and 1 x 10-4 for Waymo-Open [33. We employ learning rate annealing by using the Pytorch OneCycleI.R learning rate scheduler. During training, we set N1 and N2 to 4096, randomly sampled by furthest point sampling. The model is trained for 600k iterations with batch size of 24. During inference, we follow previous methods 43 21 and set Ni and N2 to 8192 for a fair comparison. The number of diffusion steps is set to 20 during training and during inference. The number of nearest neighbors i in DGCNN and Local Transformer is set to 16. The number of global-cros cross transformer layers is set to 14. The number of feature channels is set to 128. Further implementation details are given in the supplemental document and the provided code.
4.2 Evaluation Metrics
We follow the most recent work in the field 43 21 5
and use established evaluation metrics for scene flow estimation. EPE,p measures the endpoint error between the prediction and the ground truth pred Vgulla averaged over all points. ACCs measures the percentage of points with endpoint error smaller than 5 cm or relative error less than 5%. ACCR measures the percentage of points with an endpoint error smaller than 10 cm or relative error less than 10%. Oudlers measures the percentage of points with an endpoint error larger than 30 cm or relative error larger than 10%.
4.3 Datasets
We follow the most recent work in the field 43 21 and test the proposed method on three established benchmarks for scene flow estimation.
FlyingThings3D 24 is synthetic dataset consisting of 25000 scenes with ground truth annotations. We follow Liu er al. in FlowNet3D 22 and Gu et al. in HPLFlowNet [ to preprocess the dataset and denote them F3D.. with occlusions, and F3D,, without occlusions. The former consists of 20000 and 2000 scenes for training and testing, respectively. The latter consists 19640 and 3824 scenes for training and testing, respectively.
KITTI Scene Flow 255 is a real autonomous driving dataset with 200 scenes for training and 200 scenes for testing. Since the annotated data in KITTI is limited, the dataset is mainly used

---

for evaluating the generalization ability of the models trained on Flying Things3D dataset, following Liu al. FlowNet3D [22] and Gu al. in HPLFlowNet (9) the KITTI dataset is preprocessed as KITTI, with occlusions, and KITTI, without occlusions. The former consists of 150 scenes from the annotated training set. The latter consists of 142 scenes from the annotated training set.
Similar to the
Waymo-Open 33 is larger autonomous driving dataset with challenging scenes. The annotations are generated from corresponding tracked 3D objects to scale up the dataset for scene flow estimation by approximately 1000 times compared to previous real-world scene flow estimation datasets. The dataset consists of 798 training sequences and 202 testing sequences. Each sequence consists of around 200 scenes. Different preprocessing of the dataset exists employed in our baseline method 77.
[5] 16],
we follow the one
4.4 State-of-the-art Comparison
We give state-of-the-art comparisons on multiple standard scene flow datasets. Table and Table show the results on the F3D, and the F3D, datasets, with generalization results on the KITTI, and the KITTI, datasets. Table shows the results on the Waymo-Open dataset. the F3D, dataset, DiftSF shows improvement (over the failure cases) of 31% in EPE3D. 44% in ACCs. 35% in ACC2, and 45% in Outliers compared to the current state-of-the-an method GMSF 443. Similar improvement is also shown the F3D, dataset with improvement of 32% in EPE3D. 34% in ACCs, 24% in ACCR. and 38% in Outliers, demonstrating DifSF's ability to handle occlusions. The generalization abilities the KITTI, and the KITTI, datasets comparable state of the art. All the four metrics show the best or second-best performances. On the Waymo-Open dataset, a steady improvement in both accuracy and robustness is achieved, demonstrating DiffSF's effectiveness weal-world data.
Table 1: State-of-the-art comparison F3D, and KITTI, The models only trained on F3D, without occlusions. The number time steps is set to 20 for training and for inference. The bold and the underlined numbers represent the best and the second best performance respectively.
Method
F3D,
KITTI,
EPEsp 4 ACCs
ACCx
Outliers 4 EPE 4
ACCs
ACCR
Outliers
FlowNet33 22
19
0.1136
41.25
77.06
60.16
0.1767
37.38
66.77
52.71
HPLFlowNet
19
0.0804
61.44
85.55
42.87
0.1169
47.83
77.76
41.03
PO [4]
ECCV'20
0.0588
73.79
92.76
34.24
0.0694
72.81
88.84
26.48
FL.OT 26 PCCY120
0.0520
73.20
92.70
35.70
0.0560
75.50
90.80
24.20
Bi- Bi-PointFlows
0.0280
91.80
97.80
14.30
0,0300
92.00
96.00
14.10
3DFlow
36
#CCV'22
0.0281
92.90
98.17
0.0309
90.47
95.80
16.12
MSBRN
1icy'22
0.0150
97.30
99.20
5.60
0.0110
97.10
98.90
8.50
DifFlow3D CVPR'22
0.0140
97.76
99.33
4.79
0.0089
98.13
99.30
8.25
GMSF 44 NIP5/23
0.0090
99.18
99.69
2.55
0,0215
96.22
98.25
9.84
(ourSF(ours)
0.0062
99.54
99.80
1.41
0.0098
98.59
99.44
8.31
Table 2: State-of-the-art comparison F3D, and KITTI, The models only trained on F3D, with occlusions. The number of time steps is set to 20 for training and 2 for inference.
Method
F3D,
KITTL ACCR
EPEsp + ACCs
ACCx
Outliers EPEpp J ACCs
Outliers
D 22 CVPR'19
0.157
22.8
0.183
39,4
79.9
HPLFlowNet
19
CVPR119
0.168
26.2
57.4
81.2
0.343
10.3
38.6
81.4
PointPWC EcCV'20
0.155
41.6
69.9
63.8
0.118
40.3
75.7
49.6
FLOT 26)
0.153
39.6
66.0
66.2
0.130
27.8
66.7
52.9
Bi- low BCCV-22
0.073
79.1
89.6
27.4
0.065
76.9
90.6
26.4
3DFlow 36 ECCV'22
0.063
79.1
90.9
27.9
0.073
81.9
89.0
26.1
MSBRN
5 1cey'23
0.053
83.6
92.6
23.1
0.044
87.3
95.0
20.8
DifFlow3D 2vR222
0.047
88.2
94.0
15.0
0.029
95.9
97.5
10.8
GMSF 43 NIPS/23
0.022
95.0
97.5
5.6
0.033
91.6
95.9
13.7
DiftSF(ours)
0.015
96.7
98.1
3.5
0.029
94.5
97.00
13.0
4.5
Uncertainty-error Correspondence
One of the key advantages of our proposed method DiffSF compared other approaches that DiffSF can model uncertainty during inference, without being explicitly trained for this purpose.
7

---

Table 3: State-of-the-art comparison on Waymo-Open dataset. The number of time steps is set to 20 for training and 2 for inference.
Method
|EPE>0 1 ACCs
ACC2
Outliers
22 CVPR'
0.225
23.0
48.6
77.9
PointPWC ECCV'20
0.307
10.3
23.1
78.6
FESTA 37 CVPR'21
0.223
24.5
27.2
76.5
FH-Net PCCV122
0.175
35.8
67.4
60.3
GMSF
43
0.083
74.7
85.1
43.5
(our)F(ours)
0.080
76.0
85.6
41.9
With uncertainty, we refer to the epistemic uncertainty, which reflects the confidence the model has in its predictions. In case, predict uncertainty for the prediction of each point. We exploit the property of diffusion models to inject randomness into inherently deterministic tasks. Without having train multiple models, we predict multiple hypotheses using single model with different initial randomly sampled noise.
Figure 3 shows that the standard deviation of 20 hypotheses for each point gives reliable uncertainty estimation, which correlates very well with the inaccuracy of the prediction. Figure 3 (left) shows the relationship between the EPE and the standard deviation of the predictions averaged over the F3D, dataset. There is an almost linear correlation of the predicted uncertainty with the EPE underlining the usefulness of our uncertainty measure. Figure 3 (right) shows the recall and precision of the outlier prediction by the uncertainty. An outlier defined as a point that has an EPE larger than 0.30 meters. The horizontal axis the threshold applied the uncertainty recall is defined as the number of correctly retrieved outliers divided by the number of all the outliers. The precision defined as the number of correctly retrieved outliers divided by the number of all the retrieved outliers. The precision-recal break-even point obtains around 55% of recall and 55% of precision.
determine the outliers. The
Peciice
0.6
0.4
1o 0.99 911
denc
armn
doet aone
anet
arom
cand
axert
exepr
axere
avans
juet,
Figure 3: Analysis of uncertainty estimation F3D, dataset. Left: Uncertainty-error correspon- dences. The horizontal axis is an interval EPE. The vertical axis is the estimated uncertainty averaged over all the points that fall in the interval and the indication of the scaled uncertainty standard deviation. Right: Recall (red) and precision curve (blue) of outliers prediction. The horizontal axis is the threshold of the estimated uncertainty to determine the outliers.
Figure 4 shows visual examples that compare our outlier prediction with the actual outliers. The first row marks the scene flow estimation outliers with an EPE larger than 0.30 meters in red. The second row marks the outliers predicted by the uncertainty estimation in red. In summary, while every learned scene flow prediction model inevitably makes mistakes, our novel formulation of the task as a diffusion process not only produces state-of-the-art results but also allows for an accurate prediction of these errors. Moreover, our analysis shows that downstream tasks can select a threshold according its desired precision and recall, therefore, mitigating potential negative effects that uncertain predictions might produce.
4.6 Ablation Study
We investigate several key design choices the proposed method. For the denoising model architec- ture, we investigate how the number of global-cross transformer layers and the number of feature

---

EPPE 20.300000
Predicted
Figure 4: Visualization of outlier prediction on F3D, dataset. Black: Accurate prediction. Red:
Outliers. Top row: Outliers defined as EPE > 0.30. Bottom row: Outliers predicted by Uncertainty.
channels affect the results. For the diffusion process, investigate the influence of the number of time steps for training and sampling.
Model Architecture. To evaluate different architectural choices we select diffusion model with five
denoising blocks during training and one denoising step during testing with the DDIM [32 sampling strategy. Table shows the influence of the number of global-cross transformer layers the results.
The experiments show that the best performance is achieved at the number of 14 layers. Table 5
shows the influence of the number of feature channels the results. The experiments show that smaller number feature channels results in worse performance. The best performance is achieved at 128 feature channels.
Table 4: Ablation study on the number of global-cross transformer layers on F3D.. The number of feature channels set to 128. The number time steps is set to for training and for inference.
Layers EPEpp + ACCs 1 ACCR
Outliers J EPEE> + ACCs t ACCR Outliers
all
non-occ
0.0439
91.6
94.8
7.9
0.0205
95.2
97.5
5.1
10
0,0413
92.6
95.1
7.1
0.0189
95.8
97.6
4.5
12
0.0381
93.0
95.5
6.4
0.0168
96.1
97.8
3.9
0.0361
93.7
95.7
0.0153
16
0.0383
93.0
95.5
6.5
0.0168
96.1
97.8
4.0
Table 5: Ablation study on the number of feature channels F3D,- The number of global-cross
transformer layers is set to 14. The number of time steps is set to for training and 1 for inference.
Channels EPE.D + ACCs ACCx
Outliers E|EEE + ACCs ACCR Outliers
all
non-oce
32
0.0612
88.2
92.9
11.7
0.0299
92.9
96.3
8.2
64
0.0431
92.3
95.0
7.4
0.0199
95.7
97.5
4.7
128
0.0361
93.7
95.7
5,9
0.0153
96.5
98.0
3.5
Number of Time Steps. We set the number of global-cross transformer layers 14 and the number of feature channels to 128. We investigate the influence of different number of time steps during training and sampling on the results. The number of time steps investigated is 5, 20, and 100 for training and 1, 2, 5, and 20 for sampling. The fast sampling is done by DDIM B2 instead of DDPM [12] sampling. T shows the results on the F3D, dataset, where a@d denotes using training steps and a sampling steps. While the results are very stable across wide range of values, best performance achieved at 2@20 time steps. hypothesize that compared to the standard setting of image generation, the lower dimensionality and variance of the scene flow data results in smaller number of required time steps. For the number of time steps during inference, DDIM sampling works well with the best performance achieved at steps.

---

Table 6: Ablation study on the number time steps for training and sampling on F3D,, The number of global-cros transformer layers is set to 14. The number of feature channels is set to 128. a&h denotes an inference of b training steps and sampling steps.
Steps
(cm) 1 ACCs t ACCx t Outliers 4 |EPEEp(cm) 4
ACCs 1 ACC₂ Outliers non-oce
all
1@5
3.608
93.701
95.732
5.904
1.527
96,549
97.973
3.527
2@5
3.590
93.718
95.727
5.910
1.518
96.558
97.957
3.544
5@5
3.592
93.716
95.720
5.911
1.521
96.556
97.953
3.545
1@20
3.588
93.870
95.912
5.798
1.504
96.731
98.080
3.520
2@20
3.576
93.871
95,919
5.791
1.491
96.736
98.083
3.511
5@20
3.580
93.865
95.917
5.79T
1.492
96.730
98.083
3,507
20@20
3.579
93.865
95.915
5.789
1.491
96.731
98.082
3.508
1@100
3.678
93.503
95.665
6.016
1.587
96.376
97.844
3.689
20 100
3.663
93.545
95.662
6.010
1.579
96.398
97.838
3.697
100
3.668
93.546
95.663
6.010
1.583
96.400
97.842
3.695
20@ 100
3.670
93.545
95.663
6.015
1.584
96.396
97.843
3.700
Ablation study compare to baseline GMSF. To show the improvement of our method compared to the baseline GMSF 4 we provide an additional ablation study on F3D,, Since the original paper GMSF has a different training setting our proposed Dif[SF, for fair comparison we retrain the GMSF baseline with our training setting. The result is given in Table 7 (first line). The check the two columns denotes the implementation of improved architecture and diffusion process, respectively. The results clearly show that the proposed method DiftSF achieves superior performance than GMSF, Both the improvement of the architecture and the introduction of the diffusion process contribute to the superior performance. The improved percentage (for the introduction of the diffusion process) over the failure case is marked in the table. The results show that the proposed method has moderate improvement in the accuracy metric EPEap and huge improvement (more than 10%) in the robustness metrics ACCs, ACCR, and Outliers. Besides the better performance, the proposed method can also provide per-prediction uncertainty.
Table 7: Ablation Study compare to baseline GMSF on F3D,-
improved [diffusion
F3D,-all ACCs
F3D,-nonoo nonoccluded
architecture| process
EPEsp
ACC*
Outliers
|EPE
ACCs
ACC2
Outliers
00.039
92.9
95.4
6.7
0.017
96.0
97.8
4.2
0.061
84,8
92.3
16.7
0.037
88.9
95.3
13.9
0.037
93.2
95.4
0.016
96.2
97.7
4.1
0.036((22.7%)
93.9
10.3%)
95.9(+ 10.9%)
5.8 10.80088)00
0.015
96.7(+13.2%)
((7.4%))
3.5(-14.6%)
5 Conclusions
We propose to estimate scene flow from point clouds using diffusion models in combination with transformers. Our novel approach provides significant improvements over the state-of-the-arr terms both accuracy and robustness. Extensive experiments multiple scene flow estimation benchmarks demonstrate the ability of DifiSF to handle both occlusions and real-world data. Furthermore, we propose to estimate uncertainty based the randomness inherent in the diffusion process, which helps to indicate reliability for safety-critical downstream tasks. The proposed uncertainty estimation will enable mechanisms mitigate the negative effects of potential failures.
Limitations. The training process of the diffusion models relies on annotated scene flow ground truth which is not easy obtain for real-world data. Incorporating self-supervised training methods to leverage unannotated data might further improve our approach in the future. Furthermore, the transformer-based architecture and the global matching process limit the maximum number of points, and further research is required for peforming matching at scale.
Potential Negative Social Impact. As any other tracking algorithm, scene flow estimation can be used in surveillance scenarios, which might raise privacy concerns and ethical issues. From an ecological perspective, training of deep learning models usually takes time and resources, thus environmental impact should be taken into consideration when training and applying such compute- intensive models. However, future development in more efficient implementations will enable the positive impact of our work e.g. roboties and autonomous driving without significant negative impact the environment.
10

---

Acknowledgements. This work was partly supported by the Wallenberg Artificial Intelligence, Autonomous Systems and Software Program (WASP), funded by Knut and Alice Wallenberg Founda- tion, and the Swedish Research Council grant 2022-04266; and by the strategic research environment ELLIT funded by the Swedish government. The computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at C3SE partially funded by the Swedish Research Council grant 2022-06725, and by the Berzelius resource, provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
References
[1] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Bahenko. Label- efficient semantic segmentation with diffusion models. In International Conference on Learning Represen-
lations, 2021.
[2] Aseem Bchl, Omid Hosseini Jafari, Siva Karthik Mustikovela, Hassan Abu Alhaija, Carsten Rother, and Andreas Geiger. Bounding boxes, segmentations and object coordinates: How important is recognition for 3d scene flow estimation autonomous driving scenarios? In Proceedings of the IEEE International
Conference on Computer Vision, pages 2574-2583, 2017.
[3] Shoufa Chen, Peize Sun. Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 19830-1988 2023.
[4] Wencan Cheng and Jong Hwan Ko. Bi-pointflownet: Bidirectional learning for point cloud based scene flow estimation. In Computer Vision-ECCV 2022: 17th Eurupean Conference, Tel Aviv, Israel, October
23-27, 2022, Proceedings. Part XXVIII. pages 108-124. Springer, 2022.
[5] Wencan Cheng and Jong Hwan Ko. Multi-scale bidirectional recurrent network with hybrid correlation for point cloud based scene flow estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10041-10050, 2023.
[6] Nathaniel Chodosh, Deva Ramanan, and Simon Lucey. Re-evaluating lidar scene flow for autonomous
driving. arXiv preprint arXiv:2304.0 02450, 2023.
Lihe Ding, Shaocong Dong, Tingfa Xu, Xinli Xu, Jic Wang, and Jianan Li. Fh-net: fast hicrarchical network for scene flow estimation on real-world point clouds. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX, pages 213-229.
Springer, 2022.
[8] Xiaodong Gu. Chengzhou Tang, Weihao Yuan, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Rcp: Recurrent closest point for point cloud. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8216-8226, 2022.
[9] Xiuye Gu, Yijie Wang. Chongruo Wu, Yong Jae Lee, and Panqu Wang. Hplflownet: Hierarchical permutohedral lattice flownet for scene flow estimation on large-scale point clouds. In Proceedings of the
IEEE/CVF conference on computer vision and pattern recognition, pages 3254-3263, 2019.
[10] Zhangxuan Gu, Haoxing Chen, Zhuoer Xu, Jun Lan, Changhua Meng, and Weiqiang Wang. Diffusioninst:
Diffusion model for instance segmentation. arXiv preprint arXiv:2212.0277.. 2022.
[11] Xizewen Han, Huangjie Zheng, and Mingyuan Zhou Card: Classification and regression diffusion models.
Advances in Neural Information Processing Systems, 35:18100- 181 15, 2022.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems,
2020.
[13] Karl Holmquist and Bastian Wandt. Diffpose: Multi hulti-hhpothesis human pose estimation using diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision. pages 15977 1,5977, 2023.
[14] Haobo Jiang, Mathieu Salzmann, Zheng Dang, Jin Xie, and Jian Yang. Se(3) diffuusion model-based point cloud registration for robust 6d object pose estimation. Advances in Neural Information Prucessing Systems, 36, 2024.
[15] Zhao Jin, Yinjie Lei, Naveed Akhtar, Haifeng Li, and Munawar Hayat. Deformation and correspondence
aware unsupervised synthetic-to-real scene flow estimation for point clouds. In Proceedings of the
IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 7233-7243, 2022.
[16] Philipp Jund, Chris Sweeney. Nichola Abdo, Zhifeng Chen, and Jonathon Shlens. Scalable scene flow from point clouds in the real world. IEFE Robotics and Automation Letters. 189-1158,,1 2021.
[17] Yair Kittepplon, Yonina C Eldar, and Dan Raviy, Flowstep3d: Model unrolling for self-supervised scene flow estimation. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14-4423, 2021.
[18] Bing Li, Cheng Zheng, Silvio Giancola, and Bernard Ghanem. Setn: Sparse convolution-raasffommer network for scene flow estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages
1254-1262, 2022.
[19] Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey. Neural scene flow prior. Advances in Neural
Information Processing Systems, 34:7838-7851, 2021
[20] Xueqian Li, Jianqiao Zheng, Francesco Ferroni, Jhony Kaesemodel Pontes, and Simon Lucey. Fast neural
scene flow. arXiv preprint arXiv:2304. 02121, 2023.
[21] Jiuming Liu, Guangming Wang. Weicai Ye, Chaokang Jiang, Jinru Han, Zhe Liu, Guofeng Zhang, Dalong
Du and Hesheng Wang. Difflow3d: Toward robust uncertainty- aware scene flow estimation with diffusion
11

---

model. arXiv preprint urXiv:2311.17453 2023.
[22] Xingyu Liu, Charles R Qi, and Lconidas J Guibas. Flownet3d: Learning seene flow in 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 529-537.
2019.
[23] Wei-Chiu Ma, Shenlong Wang, Rui Hu, Yuwen Xiong, and Raquel Urtasun. Deep rigid instance scene flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
3614-3622, 2019.
[24] Nikolaus Mayer, Eddy 1lg. Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
4040-4048, 2016.
[25] Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In Proceedings of the IEFF conference on computer vision and pattern recognition, pages 3061-3070, 2015.
[26] Gilles Puy, Alexandre Boulch, and Renaud Marlet. Flot: Scene flow point clouds guided by optimal
transport. In Computer Vision-ECCV 2020: 16th European Conference, Glaxgow. UK, August 23-28.
2020, Proceedings, Part XXVIII, pages 527-544 Springer, 2020.
[27] Zhile Ren, Deqing Sun, Jan Kautz, and Erik Sudderth. Cascaded scene flow prediction using semantic segmentation. In 2017 International Conference on 3D Vision (3DV), pages 225-233. IEEE, 2017. [28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution
image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10995, 2022.
[29] Saurabh Saxena. Charles Herrmann, Junhwa Hur. Abhishek Kar. Mohammad Norouzi, Deqing Sun, and
David Fleet. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation. Advances in Neural Information Processing Systems, 36, 2024.
[30] Daniel Seita, Yufei Wang. Sarthak J Shetty, Edward Yao Li, Zackory Erickson, and David Held. Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. In Conference on Robot Learning, pages 1038-1049, PMLR, 2023.
[31] Leonhard Sommer, Philipp Schröppel, and Thomas Brox. S12se3: Clustering scene flow into se (3)-motions via proposal and selection. In Pattern Recognition: 44th DAGM German Conference, DAGM GCPR 2022, Konstanz, Germany, September 27-30, 2022, Proceedings, pages 215-229. Springer, 2022. [32] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. International
Conference on Learning Representations, 2021.
[33] Pei Sun. Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo. Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalahility in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446-2454, 2020.
[34] Zachary Teed and Jia Deng. Raft-3d: Scene flow using agid-motion embeddings. In Proceedings of the
JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8375-880 8838, 2021.
[35] Christoph Vogel, Konrad Schindler, and Stefan Roth. 3d scene flow estimation with a piecewise rigid scene model. International Journal of Computer Vision, 1 115:1-28, 2015.
[36] Guangming Wang, Yunzhe Hu, Zhe Liu, Yiyang Zhou, Masayoshi Tomizuka. Wei Zhan, and Hesheng Wang. What matters for 3d scene flow network. In Computer Vision-ECCV 2022: 17th European Conference, Tel Avix; Israel, October 23-27, 2022, Proceedings, Part XXXIII, pages 38-55. Springer, 2022. [37] Haiyan Wang, Jiahao Pang. Muhammad A Lodhi, Yingli Tian, and Dong Tian. Festa: Flow estimation via spatial-temporal attention for scene point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14173-14182, 2021.
[38] Yue Wang, Yonghin Sun, Ziwei Liu, Sanjay F. Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
[39] Zirui Wang. Shuda Li, Henry Howard-Jenkins, Victor Prisacariu, and Min Chen. Flownet3d++: Geometric losses for deep scene flow estimation. In Proceedings of the IEEE/CVF winter conference on applications
of computer vision, pages 91 98, 2020.
[40] Yi Wei, Ziyi Wang, Yongming Rao, Jiwen Lu, and Jie Zhou. Pv-raft: Point-voxel correlation fields for scene flow estimation of point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6954 6963, 2021.
[41] Wenxuan Wu, Zhi Yuan Wang, Zhuwen Li, Wei Liu, and Li Fuxin. Pointpwe-net: Cost volume on point clouds for (self-) supervised scene flow estimation. In European Conference on Computer Vision, pages 88-107. Springer, 2020.
[42] Gengshan Yang and Deva Ramanan. Learning to segment rigid motions from two frames. In Proceedings
of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (675,127, 2021. [43] Yushan Zhang, Johan Edstedt, Bastian Wandt, Per-Erik Forssén, Maria Magnusson, and Michael Felsberg.
Gmsf: Global matching scene flow. Advances in Neural Information Processing Systems, 36, 2024.
12

---

A Detailed model architecture
Feature Extraction In this section we give the detailed architecture of the feature extraction backbone, DGCNN 38 in Figure 5
The overall architecture of DGCNN is given in the upper figure. The input point cloud (e R Nxx3) contains the three-dimensional coordinates of N points. The whole network consists of four layers. The output feature of each layer, Feature 1 (E R Feature 2 (E R Feature 3 (E R ( and Feature 4 (E IRN X ) are concatenated together along the feature dimension (E R x320) and then sent into multi-layer perceptron (MIP) (linear layer batch normalization leaky ReI.U) to get the final output feature (E I2
The detailed architecture of each layer is given in the bottom figure. Xipput E N.. is the input feature, where Cin is the number of input channels. A k-nearestt-niighoor algorithm is first employed on the input feature to find the neighbors Xueighbor E NN l is set to 16. The point feature Xiuput and the edge feature Neigghor Xipput are first concatenated together along the channel dimension (E R. xxxx(C()) and then sent into an MLP (linear layer + batch normalization + leaky ReI.U). The output of the MI.P has dimension of TR. xixxCot where Cout is the number of output channels. Finally, max pooling is done on the k-nearest-neighhors to get the final output feature (E R xCot)..
Local Transformer The overall architecture of the local transformer is given in Figure 6 (left). The input features have a dimension of R.NxCi.. First, the input features are passed through linear layer to get X inpat E PRNVC. xCC, Then, the k-nearesttneiighbrss are found, X Kneighhor E R xkxC...
The query is generated from Xipput by linear layer yp, Q E TR NxC, The key and value are generated from Xneihhbor by linear layers Uú and a, K E R Nxxx C+ and V E R NxKxCr Cin = Cg = C = C₂ = Cout = 128. k 16. y an MLP consisting of sequence of linear layer, ReLU,, linear layer, and softmax. The attention output (E IR xCant) is further processed by a linear layer and a residual connection from the input.
The detailed positional encoding network is given in Figure (right) which takes the coordinate of the input features input Xinput E -x3, First, the d-ncarest-neighhor of each point are found, Xneighhor E R xxx3, The distance between Xinput and Neeghhhr is passed through an MLP (linear layer + ReLU + linear layer) to get the final positional embedding 6.
Global Transformer In this section, we give the detailed architecture of the global transformer, in Figure (left). The input features have a dimension of Xinput E NCCi, The query Q E ggNxC., key K E NxC, and value V E R N xC, are generated from the input features by linear layers y, vi, and The attention matrix (E RRNNN) computed as the matrix multiplication of the query and the key. The attention output (E R. is the matrix multiplication of the attention matrix and the value. The attention output is further processed by a linear layer, layer normalization, and a residual connection from the input features to get the final output feature (E gg-xCoou)) On = CG C = C, = Cout 128.
Cross Transformer In this section, we give the detailed architecture of the cross transformer, in Figure 7 (right). X N and Xtarget E R Nxxx are the input features of the source point cloud and the input features of the target point cloud. The query Q E NNC, is generated from the source point cloud feature by linear layer sp. The key K E R NxC. and value V NxC, are generated from the target point cloud by linear layer &p and a, respectively. Similarly to the global transformer, the attention matrix (E R Nx NNN is computed as the matrix multiplication of the query and the key. The attention output (E NxC.) is the matrix multiplication of the attention matrix and the value. The attention output is further processed by a linear layer, a layer normalization, and residual connection from the input features to get the output feature (E NC.....) Cin = C22, = Ck C, Gout = 128. A feedforward network with a sequence of linear layer, GeI.U, linear layer, and layer normalization is applied the output feature of the cross transformer to aggregate information. The final output has a dimension of IR XCout The proposed method is trained on 4 x NVIDIA A40 GPUs.
13

---

N X 3
N X 64
N PIXE
M X 64
128
festure
N 128
Layer
Layer
MXX
Lager 3
Layer
molt-kyer byybe
penceptron
Cloud
Festure
Outgut Festurr
Kuuiighhar X Gea
Xingari N x Cin
X 20m
N x Cand
Demataplaaettbaee Rapa
N x k x
podia
XXXX
N
Figure 5: The detailed architecture the feature extraction backbone DGCNN, The upper figure shows the overall architecture of DGCNN. The bottom figure shows the detailed architecture of each layer.
Q.Nxx
Amport Co
Cin
Quty
Cont
Xinput x
Kaeiphbor N x
J:N X Coor
X Xinput
N Neighbor X X
RWW
Kncighber Ga
8:N XxXx
Grat
x Coot
V:N
Figure 6: The detailed architecture of the local transformer. The left figure shows the overall
architecture the local transformer. The right figure shows the detailed architecture for the positional embedding.
Q. Q.
Xaumee N Coa
Q
Atc. x
Xamaill x 6r
K. K C
Murger i Gi
N C
Drscaln
Xon
N Co
N
ViA Co
Co
Vaber
Figure 7: The detailed architecture of the global transformer (left) and the eross transformer (right).
14

---

B Additional visualizations
Figure 8: Visualization the reverse diffusion process on the KITTI dataset. The orange points denote the source point cloud wrapped by the prediction of the current timestep. The green points denote the target point cloud.
CORST
Figure 9: Visualization comparison of GMSF and DiftSF on the FlyingThings't dataset. The blue points represent the target point cloud. The green points represent the warped source points with an EPE3D smaller than a certain threshold. The orange points represent the warped source points with an EPE3D larger than a certain threshold.
CMSF
Dinss
Figure 10: Visualization comparison of GMSF and DifISF on the KITTI dataset. The blue points represent the target point cloud. The green points represent the warped source points with EPE3D smaller than a certain threshold. The orange points represent the warped source points with an EPE3D larger than certain threshold.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations the work can be found in the Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
There is no theoretical result in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The detailed experiment setting can be found in the experiments section 4 The detailed architecture of the model can be found in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: The code is available at https://gihub..com///I Yushan3/iffi with suffi- cient instructions to faithfully reproduce the experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
public/guides. http://ooooeSuuooisisoooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.c//pil wwwpppblccgguiicuceccccceoeooopp /eeggudddsse/eebbbinninn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The details can be found in the Experiment section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources we used are specified Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impacts of the work can be found the Section 5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work does not have a high risk for misuse such that safeguards are needed.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets (e.g., code, data, models), used the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets is introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "MxtZqdgXYw", "reviewer": "Reviewer_gpDD", "summary": "This paper proposes a scene flow estimation method that estimates scene flow from point clouds. The proposed method combines a previous scene flow method, GMSF, and diffusion model, where a denoising block is introduced based on GMSF.  During training, the source point cloud is first warped via noisy scene flow, and then the warped source point cloud and the target point cloud are fed to the denoising block.  Experimental results demonstrate the proposed method achieves the best performance compared with state-of-the-art approaches.  \nIn addition, the paper shows interesting results that the proposed method can be used to measure the hardness of input point clouds. That is, given a pair of point clouds, the proposed method can obtain multiple scene flow results using different initial randomly sampled noise, where the uncertainty (standard deviation) of scene flow results is highly correlated with their accuracy.", "questions": "Please refer to my comments above.", "limitations": "The paper provides the limitations of the proposed method, while it does not discuss the potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of the proposed method is interesting and sounds good. \n\n2. The result of uncertainty-error correspondence is interesting, indicating the proposed method can be used to predict the hardness (estimation difficulty ) of the input point cloud pair.\n\n3. Extensive experimental results shows the effectiveness of the proposed method", "weaknesses": "1. The organization of the technical part can be improved. It is a bit difficult to follow Sec. 3.2. For example, the paper has already introduced the forward process, reverse process and the training loss, and then introduces the details of the reverse process again.\n\n2. The proposed method uses the same feature extractors with shared weights to extract features from the warped and target point clouds. However, the distributions of the warped point cloud may be rather different from the target one, since the warped point cloud is noisy and the target point cloud is clean.   Would such differences negatively affect the training of feature extractors? Similarly, the training of the linear layers in the Local-Global-Cross Transformer may also be negatively affected. If the feature extractor of the warped point cloud does not share weights with that of the target one, would the estimation performance of the proposed method be improved?  \n\n3. Table 7 in the Appendix shows that when GMSF and the proposed method use the same architecture, the proposed method's performance improvement is marginal on F3D_o, compared with GMSF. Does \bsuch performance also occur on other datasets ( such as KITTI and Waymo-Open datasets)? It is not clear which part of the proposed method mainly contributes to the performance improvement, the architecture or the denoising process.\n\n4. Some statements may be a bit overclaimed. The paper claims that \"To the best of our knowledge, we are the first to apply a diffusion model successfully for the task of scene flow estimation.\". However, as described in the main paper, a few methods have attempted to employ diffusion models for scene flow estimation [24]. The statement of \"the first\" may be overclaimed and can be replaced with a more proper claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: The organization of the technical part can be improved.**\n\nThe original idea of the organization was to decouple the introduction of the standard diffusion process and our contribution. \nWe first recap the basics of diffusion models for the audience in Section 3.1.\nIn Section 3.2 we introduce an overview of formulating scene flow estimation as diffusion process, including the training and sampling algorithms.\nThen we introduce our contribution and give the detailed model architecture and the training loss we employed in Section 3.2 and 3.3.\nBut thanks to the reviewer for pointing out that this part can be difficult to follow.\nWe will clarify the organization in the main paper.\n\n**Question 2: Will the shared weight in Feature Extraction and linear layers in the Transformer negatively affect the training?**\n\nDiffSF uses a global matching model to predict the noise in the reverse process.\nSince the core of the global matching approach is to match similar points, it makes sense to use shared-weights feature extractors.\nHowever, we agree that the distribution of the (noisy) source point cloud and the (clean) target point might be different, thus, we follow the reviewer's recommendation and try feature extraction without shared weights.\nThe results are given in Table 1 which shows that, while having a lower number of parameters, the model with a shared weight feature extractor performs better than the same model without shared weights.\n\nTable 1: Ablation study on shared weights on $F3D_o$.\n|model|$EPE_{3D}$|$ACC_S\\uparrow$|$ACC_R\\uparrow$|$Outliers\\downarrow$|$EPE_{3D}$|$ACC_S\\uparrow$|$ACC_R\\uparrow$|$Outliers\\downarrow$|\n|:---|:---|:---|:---|:---|:---|:---|:---|:---|\n||all|all|all|all|non-occ|non-occ|non-occ|non-occ|\n|shared|0.036|93.9|95.9|5.8|0.015|96.7|98.1|3.5|\n|not shared|0.042|92.2|95.2|7.6|0.020|95.7|97.8|4.9|\n\n**Question 3: Which part contributes to the performance improvement, the architecture or the denoising process?**\n\nBoth the improved architecture and the denoising process contribute to the performance improvement.\nIn Table 2 we improved Table 7 in the supplementary material to make the contributions of different design choices more clear.\nThe architecture is modified to allow for a better integration into the diffusion model.\nTable 2 shows that when fitting the original architecture of GMSF directly to the diffusion models, there is a performance drop ($EPE_{3D}$ from 0.039 to 0.061).\nBy improving the architecture, the performance of GMSF improves $EPE_{3D}$ from 0.039 to 0.037, and the performance of DiffSF improves $EPE_{3D}$ from 0.061 to 0.036, which is significant and demonstrates that the improved architecture integrates with the diffusion models better.\nThe numbers in parentheses mark the improved percentage of DiffSF compared to GMSF with the same architecture.\nOn the $F3D_o$ dataset, the result shows a more than 10% improvement of $ACC_S$, $ACC_R$, and $Outliers$ if we evaluate all the points. An even higher improvement is shown if we evaluate only the non-occluded points.\nFor the $EPE_{3D}$, there is an improvement of 2.7% over all the points and 6.3% over the non-occluded points.\nGiven the fact that FlyingThings3D is close to saturation, we argue that the relative improvements are more reasonable measurements than absolute improvements.\nA similar trend is found when testing the generalization ability on the $KITTI_o$ dataset.\nNote that the models are only trained on $F3D_o$ and then generalized to $KITTI_o$.\nThe new training scheme on $F3D_o$ (number of points 4096 and batch size 24) causes some generalization performance degradation on $KITTI_o$ (from $EPE_{3D}$ 0.033 to 0.089).\nFrom Table 3 we can tell that both the improved architecture and the diffusion process contribute to the improvement of the performance.\nDue to the limited resources, we didn't do an additional ablation study on the Waymo dataset, where we will have to additionally train three different models.\n\nTable 2: Ablation study on $F3D_o$.\n|architecture|diffusion|$EPE_{3D}$|$ACC_S\\uparrow$|$ACC_R\\uparrow$|$Outliers\\downarrow$|$EPE_{3D}$|$ACC_S\\uparrow$|$ACC_R\\uparrow$|$Outliers\\downarrow$|\n|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|\n|||all|all|all|all|non-occ|non-occ|non-occ|non-occ|\n|||0.039|92.9|95.4|6.7|0.017|96.0|97.8|4.2|\n||✓|0.061|84.8|92.3|16.7|0.037|88.9|95.3|13.9|\n|✓||0.037|93.2|95.4|6.5|0.016|96.2|97.7|4.1|\n|✓|✓|0.036(-2.7%)|93.9(+10.3%)|95.9(+10.9%)|5.8(-10.8%)|0.015(-6.3%)|96.7(+13.2%)|98.1(+17.4%)|3.5(-14.6%)|\n\nTable 3: Ablation study on $KITTI_o$.\n|architecture|diffusion|$EPE_{3D}$|$ACC_S\\uparrow$|$ACC_R\\uparrow$|$Outliers\\downarrow$|\n|:---|:---|:---|:---|:---|:---|\n|||0.089|71.2|82.8|29.8|\n||✓|0.054|80.1|92.2|23.1|\n|✓||0.050|85.7|91.7|18.1|\n|✓|✓|0.029(-42.0%)|94.5(+61.5%)|97.0(+63.9%)|13.0(-28.2%)|\n\n**Question 4:  Overclaimed statements.**\n\nBy this sentence we mean that we are the first to apply diffusion models to estimate the full scene flow vector field directly from two point clouds instead of a refinement plug-in module, as in DifFlow3D.\nHowever, we agree that this sentence might be misinterpreted and we will therefore rephrase it as: To the best of our knowledge, we are the first to propose using diffusion models to estimate the full scene flow directly from two point clouds.\n\n**Question 5:  Negative societal impacts.**\n\nPotential negative societal impacts might be:\nAs any other tracking algorithm scene flow estimation can be used in surveillance scenarios, which might raise privacy concerns and ethical issues.\nFrom an ecological perspective, training of deep learning models usually takes time and resources, thus environmental impact should be taken into consideration when training and applying such compute-intensive models.\nHowever, in our opinion, the positive impact of our proposed approach in practical applications such as robotics and autonomous driving outweighs the negative impact in the applications we consider.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N3RJSXrpAf", "reviewer": "Reviewer_qmev", "summary": "This paper introduces DiffSF, which integrates transformer-based scene flow estimation with denoising diffusion models. The diffusion process involves progressively perturbing the ground truth scene flow vector field by adding Gaussian noise. Conversely, during the reverse process, the scene flow vector field prediction is recovered from randomly sampled Gaussian noise by conditioning on a source and a target point cloud. The DiffSF achieves state-of-the-art performance on FlyingThings3D, KITTI Scene Flow, and Waymo-Open.", "questions": "It would be beneficial to include a discussion or exploration on the effects of adding noise directly to the ground truth scene flow vector field versus adding noise to the embedding of the ground truth scene flow vector field, as achieved by a separate network such as a VAE model.", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Extending denoising diffusion models for scene flow estimation is a novel approach that is straightforward to understand. \n\nThe ablation study is comprehensive, such as evaluating the impact of varying the number of time steps during both training and sampling for the diffusion model.", "weaknesses": "The tech contribution is limited. This paper does not invent a new concept given a relevant work of optical flow estimation using diffusion models [1]. \n\nTo enhance understanding of the diffusion process for scene flow estimation, visualizations of source points with added noise at different time steps should be included.\n\nThe study is limited to the GMSF model architecture. It is strongly recommended to evaluate other popular architectures, as the key concept of the paper is to introduce the diffusion model for scene flow estimation. Additionally, it would be valuable to investigate whether this approach works in the context of self-supervised scene flow estimation such as [2][3].\n\n[1] The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation\n\n[2] Self-Supervised Robust Scene Flow Estimation via the Alignment of Probability Density Functions\n\n[3] Just Go with the Flow: Self-Supervised Scene Flow Estimation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: Limited contribution.**\n\nOptical flow and scene flow share similar ideas, i.e. estimating object movement from sensors' data. However, the sensors are completely different and the generated data has different modalities.\nFor RGB cameras, the output images are usually on a regular grid, with RGB values always from 0 to 255, enabling the generation of dense feature maps.\nOn the contrary, for LiDAR sensors, the output point clouds are unstructured and sparse, and the point clouds between consecutive frames could even have different sizes.\nDue to these inherent differences in data structures, approaches for optical flow estimation cannot be directly transferred to scene flow estimation.\nFor example in [1], the method takes the noisy label and one or more conditioning images as input, the noisy label and the images are concatenated together and then sent into a U-Net. The output of the U-Net is the learned optical flow.\nPoint clouds cannot be processed with CNNs (unless they are densified, which is a separate CV problem) and due to the varying number of points, concatenation over the feature dimension cannot be performed as in [1].\n\n**Question 2: The visualizations of source points with added noise at different time steps should be included.**\n\nThanks for the comment. We plan to add more visualizations of the diffusion process other than the visualization in Figure 1 in the main paper. However, due to the page limitation of the rebuttal process, here we only provide one additional visualization in the attached PDF in the common comment section above. More will be added to the supplemental document of the final version.\n\n**Question 3: The study is limited to the GMSF model architecture.**\n\nWe chose GMSF as our baseline as it is the currently most powerful scene flow estimation model, more than 50% better than the second best (DifFlow3D) on $F3D_o$ dataset in every metric. Note that instead of taking GMSF directly, we also made modifications to the GMSF model to make it integrate into the diffusion models better.\nFor more information about the importance of the architecture modifications, please refer to the reply to reviewer gpDD (Question 3).\nThe table shows that by improving the architecture, the $EPE_{3D}$ on $F3D_o$-all improves from 0.061 (line 2) to 0.036 (line 4).\nThe $EPE_{3D}$ on $KITTI_o$ improves from 0.054 (line 2) to 0.029 (line 4), which demonstrates our claim that the modifications we made to the architecture allow for better integration into the diffusion model.\n\nTo investigate whether the proposed diffusion model works for self-supervised scene flow estimation, instead of adapting other models into our proposed method, we adapt the proposed method by simply changing the supervision from fully-supervised to self-supervised. The self-supervision consists of two parts: the Chamfer loss and the smoothness loss, which are two common losses employed in self-supervised scene flow estimation. \n\nThe Chamfer loss is defined as\n$$\nL_{Chamfer} = \\sum_{x_i\\in\\mathcal{X}} \\min_{y_j\\in\\mathcal{Y}} |\\hat{x_i}'- y_j|^2 + \\sum_{y_i\\in\\mathcal{Y}}\\min_{x_i\\in\\mathcal{X}} |x_i - \\hat{y_j}'|^2,\n$$\nwhere $x_i$ and $y_j$ are the points from source and target, $\\hat{x_i}'$ and $\\hat{y_j}'$ are transformed points.\n\nThe smoothness loss is defined as\n$$\nL_{Smoothness} = \\sum_{x_i\\in\\mathcal{X}}\\frac{1}{N}\\sum_{y_j\\in\\mathcal{N}(x_i)}\\|SF(x_i) - SF(y_j)\\|^2,\n$$\nwhere $\\mathcal{N}(x_i)$ denotes the nearest neighbors of $x_i$. $N$ is the number of points in the local region. $SF()$ is the predicted scene flow.\n\nThe results are shown in the following table, which shows that our proposed approach also works in the context of self-supervised scene flow estimation.\nAn initial try without parameter tuning already gives us a huge performance improvement on the KITTI dataset. The result on FlyingThings3D is also comparable to state-of-the-art.\n\nTable 1: Ablation study on shared weights on $F3D_o$.\n|method|$F3D_s$||||$KITTI_s$||||\n|:---|:---|:---|:---|:---|:---|:---|:---|:---|\n||$EPE_{3D}$|$ACC_S\\uparrow$|$ACC_R\\uparrow$|$Outliers\\downarrow$|$EPE_{3D}$|$ACC_S\\uparrow$|$ACC_R\\uparrow$|$Outliers\\downarrow$|\n|PointPWC[4]|0.1213|32.39|67.42|68.78|0.2549|23.79|49.57|68.63|\n|SS-APDF[2]|0.0750|58.90|86.20|47.00|0.0920|74.70|87.00|28.30|\n|SS-SFE[3]|-|-|-|-|0.1050|46.48|79.42|-|\n|DiffSF|0.0796|50.35|81.49|54.10|0.0315|89.75|97.03|15.91|\n\n**Question 4: Adding noise to the embeddings instead of the vector field.**\n\nWe thank the reviewer for the novel perspective that we have not considered so far.\nWhile VAE models are commonly used in image generation with great success their application to scene flow estimation is not commonly studied.\nLatent representations of data are often global, making them suitable for higher-level tasks like image generation [5], point cloud generation [6], classification [7], etc.\nHowever, scene flow estimation is a rather low-level task that focuses on how each point moves instead of understanding the whole scene.\nThis difference in nature might make using a VAE to encode features of the point cloud to get high-level information not favorable for scene flow estimation.\nHowever, We are happy about the suggestion and will consider the option in future research.\n\nAdditional References\n\n[4] PointPWC-Net: A Coarse-to-Fine Network for Supervised and Self-Supervised Scene Flow Estimation on 3D Point Clouds\n\n[5] High-Resolution Image Synthesis with Latent Diffusion Models\n\n[6] LION: Latent Point Diffusion Models for 3D Shape Generation\n\n[7] Robust Latent Subspace Learning for Image Classification", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ujZiDeTa7", "reviewer": "Reviewer_yrBG", "summary": "This paper deals with scene flow estimation in 3D point clouds. It proposes a formulation based on diffusion model. The model takes the source and target frames of 3D point clouds as condition and turn the problem into a conditional generation problem. Different from naive conditional generation, the formulation considered the characteristic of the scene flow problem to make each reverse diffusion step as a \"flow adjustment\". Experiments are done on standard benchmarks.", "questions": "What is the definition of \"uncertainty\"? I understand \"EPE\" and \"outlier\". Is there a definition of \"uncertainty\" somewhere in the paper?", "limitations": "Limitation was mentioned. No other limitation is found besides the ones mentioned in the paper.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The proposed formulation considered the characteristic of the scene flow problem to make each reverse diffusion step as a \"flow adjustment\". It makes so much sense and is a elegant solution to this problem.\n- The presentation of this paper is clear and easy to follow.\n- The experiment results significantly outperform existing baselines.\n- Code and visualization are provided.", "weaknesses": "- The author could consider providing more visualizations of the (reverse) diffusion process other than the only one GIF attached in the supplementary. This will be very helpful.formulation\n- Minor: the citation notation seems to be wrong. For example, citing paper 1 should be \"[1]\", not \"(1)\".\n- Minor: in Tables 1, 2 and 6, there are numbers in both bold and underlines. What are underlined numbers mean? This should be explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: Consider providing more visualization of the reverse process other than the only one GIF attached.**\n\nThanks for the comment. We plan to add more visualizations of the diffusion process other than the visualization in Figure 1 in the main paper. However, due to the page limitation of the rebuttal process, here we only provide one additional visualization in the attached PDF in the common comment section above. More will be added to the supplemental document of the final version.\n\n**Question 2: The citation notation seems to be wrong. citing of papers should be \"[]\" instead of \"()\".**\n\nThanks for the comment. We will revise the citing in the paper.\n\n**Question 3: What are the meanings of bold and underlined numbers in Tables 1,2 and 6. This should be explained.**\n\nThanks for noting that the meanings of bold and underlined numbers was not explained in the tables.\nThe following explanation will be added to the tables:\nThe bold numbers represent the best performance, and the underlined numbers represent the second best performance.\n\n**Question 4: Definition of \"uncertainty\".**\n\nGood catch!\nWe will include the following definition in the paper:\nWith uncertainty we refer to the epistemic uncertainty, which reflects the confidence the model has in its predictions. \nIn our case, we predict an uncertainty for the prediction of each point.\nThat is, if we have a predicted scene flow vector field of size $N \\times 3$, the predicted uncertainty will have a size of $N \\times 1$.\nIn practice, each uncertainty is predicted by taking the standard deviation of 20 hypotheses with different initial randomly sampled noise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sme4LU67mX", "reviewer": "Reviewer_2g78", "summary": "This paper introduces a novel diffusion model designed for scene flow estimation, aiming to enhance both accuracy and robustness, particularly in the presence of noisy inputs or occlusions. The proposed denoising diffusion models effectively handle noisy data by modeling stochastic processes, filtering out sensor-induced noise, and focusing on underlying patterns. This approach not only improves prediction accuracy but also introduces a measure of uncertainty by averaging over multiple hypotheses without altering the training process. Additionally, the paper presents a new architecture that integrates transformers with diffusion models, further boosting accuracy and robustness across various benchmark datasets.", "questions": "1. Line 250 states \"showcasing DiffSF’s effectiveness on real-world data..\" Can the authors provide some challenging samples of data that your model is robust against?\n2. Which datasets are used in generation of results for figures 3 and 4?", "limitations": "Authors have presented the limitations of the work (which is a good thing), but this has been mostly discussed in the supplementary section (which is outside of the main paper). I would encourage the authors to include a brief summary of the limitations in the main paper as well.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation for using diffusion models in scene flow estimation is well-justified. Real-world data often presents challenges such as severe occlusion and noisy inputs, necessitating robust models for scene flow estimation. \n2. The proposed method demonstrates state-of-the-art performance through extensive experiments on multiple benchmarks, including FlyingThings3D, KITTI Scene Flow, and Waymo-Open.\n3. The paper is well-written and easy to understand, making the concepts accessible to readers.\n4. The reproducibility of the results is strong, ensuring that the findings can be reliably replicated.", "weaknesses": "The paper lacks qualitative results. I would've liked to see visual comparison of scene flow results (especially for challenging cases) on various datasets. This makes it hard to interpret where the proposed approach performs better than current methods and more importantly where the proposed approach fails.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: Additional visualizations.**\n\nWe propose additional qualitative results in the attached PDF in the common comment section above. Figure 3 shows the visualization comparison between GMSF and DiffSF on the KITTI dataset.\nThe results show that DiffSF is more robust than GMSF when the point clouds are sparse which are thus more likely to be noisy. This is consistent with our quantitative results that DiffSF shows a large improvement in all the evaluation metrics, especially in robustness ($ACC_S$, $ACC_R$, $Outliers$).\nFigure 2 shows a visualization comparison between GMSF and DiffSF on the FlyingThings3D dataset.\nSimilarly, DiffSF performs better when the point clouds are sparse (column 1), and when occlusion happens (columns 2 and 3).\nHowever, when the occlusion is too severe that the object completely disappears, both models can not give accurate predictions (column 4).\nSimilar results are found in the Waymo dataset (see Figure 4 top row).\nIn the bottom row we show that the proposed DiffSF is robust against various road types on the Waymo dataset, which is a very challenging dataset compared to KITTI, with complicated long sequences, and fast vehicle motions.\n\n**Question 2:  Which datasets are used in the generation of results for Figures 3 and 4?**\n\nThe datasets used to generate Figures 3 and 4 are FlyingThings3D. We will add this to the figure caption.\n\n**Question 3:  Limitations in the main paper.**\n\nThanks for the comment. We will include the limitations in the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable comments and questions.
We are happy that the reviewers find that using diffusion models in scene flow estimation is novel (qmev), interesting (gpDD), and elegant (yrBG), and that the motivation is well-justified (2g78). 
Reviewer gpDD also finds the result of uncertainty-error correspondence interesting (gpDD), which enables the estimation of prediction uncertainty and is rarely investigated in previous scene flow estimation approaches.
All reviewers agree that the experiments are extensive and ablation studies show the effectiveness of the proposed method. 
Two reviewers (yrBG, 2g78) comment that the paper is well-written and easy to follow, and appreciate the release of our code to make the results reproducible.

In the following we will address the individual questions.

Meta-review: The ratings for this paper include one strong accept (ygGr), two accepts (2g78 and qmev), and one weak accept (gpDD). All reviewers acknowledge the idea of extending denoising diffusion models for scene flow estimation. The main concerns are the lack of qualitative results and the organization of the technical section. Most of these concerns were satisfactorily addressed during the rebuttal, so the AC recommends acceptance.
Decision: Accept (spotlight)