Neural Scene Flow Prior
Authors: Xueqian Li
Jhony Kaesemodel Pontes
Simon Lucey
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: neural prior
scene flow
implicit regularizer

Abstract: Before the deep learning revolution, many perception algorithms were based on runtime optimization in conjunction with a strong prior/regularization penalty. A prime example of this in computer vision is optical and scene flow. Supervised learning has largely displaced the need for explicit regularization. Instead, they rely on large amounts of labeled data to capture prior statistics, which are not always readily available for many problems. Although optimization is employed to learn the neural network, at runtime, the weights of this network are frozen. As a result, these learning solutions are domain-specific and do not generalize well to other statistically different scenarios. This paper revisits the scene flow problem that relies predominantly on runtime optimization and strong regularization. A central innovation here is the inclusion of a neural scene flow prior, which utilizes the architecture of neural networks as a new type of implicit regularizer. Unlike learning-based scene flow methods, optimization occurs at runtime, and our approach needs no offline datasets---making it ideal for deployment in new environments such as autonomous driving. We show that an architecture based exclusively on multilayer perceptrons (MLPs) can be used as a scene flow prior.  Our method attains competitive---if not better---results on scene flow benchmarks. Also, our neural prior's implicit and continuous scene flow representation allows us to estimate dense long-term correspondences across a sequence of point clouds. The dense motion information is represented by scene flow fields where points can be propagated through time by integrating motion vectors. We demonstrate such a capability by accumulating a sequence of lidar point clouds.

Neural Scene Flow Prior
Xueqian L:
Jhony Kaesemodel Pontes'
Simon Lucey?
AArgo AI
"The University of Adclaide
Abstract
Before the deep learning revolution, many perception algorithms were based runtime optimization in conjunction with a strong prior/regularization penalty. A prime example of this in computer vision is optical and scene flow. Supervised learning has largely displaced the need for explicit regularization. Instead, they rely on large amounts of labeled data capture prior statistics, which not always readily available for many problems. Although optimization is employed to learn the neural network, the weights of this network are frozen at runtime. As a result, these learning solutions are -ppaii- and do not generalize well to other statistically different scenarios. This paper revisits the scene flow problem that relies predominantly on runtime optimization and strong regularization, A central innovation here is the inclusion of neural scene flow prior, which uses the architecture of neural networks Unlike bearning-based scene flow methods, optimization occurs at runtime, and our approach needs no offline environments such as autonomous driving. We show that an architecture based exclusively multilayer perceptrons (MLPs) can be used as a scene flow prior. Our method attains competitive- not better results on scene flow benchmarks. Also, our neural prior's implicit and continuous scene flow representation allows us to estimate dense long-term correspondences across sequence of point clouds. The dense motion information is represented by scene flow fields where points be propagated through time by integrating motion vectors. We demonstrate such a capability by accumulating sequence of lidar point clouds.
type of implicit regularizer.
it ideal for deployment in new
Introduction
state-of-the-arr results have recently been achieved by learning-basse models [17,30,47, 60, 68) for
the scene flow problem-the task of estimating 3D motion fields from dynamic scenes. However, such models heavily rely on large-scale data to capture prior knowledge, which is not always readily available. Scene flow annotations are expensive, and most methods train on synthetic and unrealistic scenarios to fine-tune small real datasets.
Poor generalization to unseen, col-of-the-distrbutie inputs another problem. Prior information is generally limited to the statistics of the data used for training. Real-world applications such as autonomous driving require robust solutions low-level vision tasks such as depth, optical, and scene flow estimation that work in statistically different scenarios. Inspired by recent innovations that make use of coordinate- as inputs) [9, 36, 37, 39, 56] for 3D modeling and rendering, we investigate the use of such networks to regularize the scene flow problem without any learning directly from point clouds. Optimization happens at runtime, and instead of learning prior from data, the network structure itself captures the prior information. It not limited to the statistics of specific dataset.
networks (i.e., pixels or 3D positions
"Research done during internship at Argo AL. Corresponding e-mail: xueqian. bingian.li@@adee www..ii@@el@llee.......u
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Optimizing neural networks at execution time not new. Ulyanov et al. [63] showed that randomly initialized convolutional network could be used as handcrafted prior for standard inverse problems such as image denoising. super-resolution, and inpainting. Ding and Feng [12] proposed a runtime optimization method (DeepMapping) for rigid pose estimation using deep neural networks.
Number of hidden units
Number of neighbors
16
128 256
50
100
200
300
500
100
Noural prior
Graph prior
Accuracy (%)
90
80
70
Although such deep image priors, deep mapping, and coordinate-basse networks for neural scene representations have been successfully applied for inverse problems, rendering, and rigid registration, none has yet investigated (to the best of our knowledge) the use of network-based priors for regularizing scene flow directly from point clouds.
2
2
2
2
2
29
Time (s) in log scale
Figure 1: Our neural scene flow prior method achieved higher accuracy while being ~10 faster than the recent runtime optimization method graph prior [45]. The evaluation was on the KITTI Scene Flow test set, where each point cloud size varies from 14k to 68k points. In method, we fixed the number of hidden layers in the MLP to 4 and varied the number of hidden units. In the graph prior method, we varied the number of neighbors to create the graph. Accuracy uses the Accs metric as defined in the experiments section. Learning- based methods might still be 10x-100x faster than the runtime optimization methods, but they still lack generalization and have memory issues when dealing with large point clouds with tens of thousands of points.
Our proposed neural prior is based on a simple multilayer perceptron (MLP) architecture, and we show is powerful enough to regularize scene flow given two point clouds implicitly. The input to the network is 3D points, and the output regularized scene flow.
Our neural prior allows for a continuous scene flow representation instead of discrete such as in graph Laplacian-base -based priors, e.g., [45]. We show how the flow fields captured by our neural prior can be employed to estimate long-term correspondences across a sequence of point clouds. The continuous scene flow allows for better integration of motions across time.
Our results are promising and competitive to supervised [30], self-supervised [38, 68), and non- learning methods [1,45] (see Table 1). Our method also scales to real-world point clouds with tens of thousands of points while achieving better accuracy and time complexity than recent runtime optimization methods (see Fig. 1).
2 Related work
Non-learning-based scene flow Scene flow is the uplift from the optical flow, which is proposed by Vedula el al. [64] as the non-rigid motion field in the 3D space. The authors proposed the optimization-basel scene flow estimation using image sequences to infer reconstruction knowledge of the flow in 3D surfaces. Successive RGB/RGB-D image-based work [4, 18-21,27, 23, 44, 50] used probability-basee estimation, coarse-to-fine fine techniques, 6-DoF parameterization, or object segmentation, etc., to improve accuracy and computation time. Although image- based scene flow methods are widely used, direct estimation of the scene flow from the point cloud is still possible through non-rigid registration methods, such as [1, 10, 26, 42]. In this paper, we focus on point cloud-based scene flow estimation.
Learning-based scene flow Image- based learning methods [6,51, (1,51,54,6 60, 70] use convolution and data supervision to solve scene flow from monocular or RGB-D images with the available depth information. Other image-based methods [22,23,32 52] take care of extra occlusion cues in the large-scale autonomous driving scenes. Point-based learning methods have become more prevalent with the rapid development of point cloud feature learning [28, 48, 49, 65, 67]. FlowNet3D [30] is a seminal work that estimates scene flow using PointNet++ [49]. Successive work [17,31,47, 66] extends paset-based learning methods using different feature extraction techniques. One obvious drawback of these supervised learning methods is the demand for sufficient ground truth labels. Besides, supervised methods lack generalizability while eventually only fitting domain-specific data. Self-supervised methods [25,38, 61, 61,68), on the other hand, replaced the loss between the prediction and the ground truth flow with point distance loss use the point cloud itself supervision.

---

Self-supervision can adapt to different datasets and maintain certain generalizability. Nonetheless, massive training data are still required for sufficient learning.
Graph Laplacian [2] is widely used to smooth the surface in mesh processing [5, 14,57,58, 57,, point cloud denoising [11,71], etc. Here we talk about the recent scene flow estimation using Graph Laplacian [45]. The method explicitly constructed graph of the point cloud to constrain the non-rigid scene flow as rigid within a specific range. While as a dataless runtime optimization, the method is heavily affected by the hyperparameters of the graph and loses scalability when the point cloud becomes larger or the neighbors the graph grow.
The graph Laplacian method
Deep neural prior, implicit functions, and neural rendering Although large-scale data helps in feature representation [59], end-to-end learning still requires high computation capacity and readily available training datasets. Instead, Ulyanov et al. [63] proposed new style of optimization that uses a convolutional neural network to infer prior knowledge from the network architecture. One broader interest is to extend the idea of the network being image function to the 3D shape modeling, and implicitly represent the continuous shape as level sets of neural networks. By directly mapping the 3D input to binary occupancy nets [9,36], signed distance functions [3,39,55] it is powerful to model the 3D geometries in a continuous space using the coordinate- based network. The following Scene Representation Networks [56] takes advantage of the coordinate- based network and renders view synthetic images. Mildenhall et al. proposed seminal work NeRF [37]. which is a novel way to do neural volume rendering using both point positions and viewing directions based on the coordinate-based network. Dynamic scene synthesis work [13, 16, 29, 40, 46, 62, 69] follows the NeRF framework, and integrates the motions to generate dynamic scenes. Some of the work [16, 29] use scene flow to further constrain or segment dynamic scenes. An interesting work that solves for the rigid alignment between point clouds using runtime optimization DeepMapping [12]. However, this work only deals with rigid motion, and the network architecture is more complex than coordinate-bann based networks. In this work, we are interested in coordinate-bon based networks to address the large-scale, real-world scene flow problem.
3 Approach
Problem definition Let S₁ and S2 be two 3D point clouds sampled from a dynamic scene at time t-1 and f. The number of points in each point cloud, |&1| and [S2], are typically different and not in correspondence. A 3D point p E S₁ moving from time f-1 to time t can be modeled by a translational vector (or flow vector) f E RS, where the scene flow J [
p + The collection of flow vectors for all 3D points is
[1
Optimization We want to optimize for scene flow F that minimizes the distance between the two point clouds, S₁ and Sz. Given the non-rigidity assumption of the scene, the optimization is inherently unconstrained. Thus, a regularization term C is necessary to constrain the motion field. We therefore solve for scene flow as
F arg min
D(p++r,8)) + AC,
PES.
where D is function to compute the distance from the perturbed point p by the flow vector f to its closest neighbor in Sz. a regularizer (e.g., Laplacian regularizer), and is weighting factor for the regularizer. In this paper, we want to investigate using a neural prior to regularize the scene flow.
3.1 Neural scene flow prior
Searning-based scene flow methods learn scene flow priors from a large number of examples. As in Deep Image Prior [63], we want to investigate if the structure of a neural network by itself is sufficient to capture scene flow prior without any learning.
Here we use a neural network as an implicit regularizer. The parameters are optimized as
arg min
Σ D(pp+g(p:0).s,
(2)
PES,
3

---

where g is a neural network parameterized by O to regularize the scene flow F. The input to is p. which is the point to be disturbed by the flow. The output of g is f and thus f = (p: ++) For the distance function D, we define it as
D (p. min |p
x 1
(3)
xES
In practice, we use it bidirectionally for both point sets, which is equivalent to Chamfer distance [15].
The objective in Eq. (2), is for the forward scene flow F, which is the one we are interested in. However, it has been shown in [30,38], that eycle consistency regularizer encourages better scene flow estimations. The extra regularizer simply enforces the backward flow to be similar to the forward flow, Ftod R F. The optimal backward flow is defined fiwd = g (p'; twd)), where p' is the shifted point by the forward flow as + f. Note that the network is the same but with different parameters, budd Using the backward flow as additional constraint, the optimal network weights are solved as
θ¹ G bund arg min DD
Σ D'ES;
D(' + g (p': 1 bwa) Si),
(4)
0,0bud PESI
where S/ is the shifted S1 by the forward flow, i... S = SIFF. Please find more details in the supplementary material. For the network 9. we use MLPs with ReLU activations. The objective function in Eq. (4) can be optimized by gradient descent techniques using off-the-shelf frameworks with automatic differentiation. We show in the experiments section how the architecture of the neural prior affects performance by varying the number of hidden layers and units.
Why a neural scene flow prior? Deep learning relies massive amounts of data and computational resources to capture prior statisties. Although learning methods have achieved impressive results in most tasks, they still struggle when deployed in environments where the statistics are different from those captured during learning. Our intuition is that neural prior acts as a strong implicit regularizer that constrains dynamic motion fields be as smooth as possible. A neural scene flow prior also scales to large scenes while achieving high-fidelity results at a low computational cost. Our proposed method with 8 hidden layers and 128 hidden units has about 1 116k parameters. FlowNet3D [30]. for example, has about 1 1.2M parameters. Our method has 100 fewer parameters than the state-of-the-arr supervised methods while achieving competitive, if not better, results. Lastly, our deep scene flow prior captures continuous flow field that allows us to perform better scene flow interpolation across a sequence of point clouds.
4 Experiments
We evaluated the performance (accuracy, generalizability, and computational cost) of our neural prior for scene flow on synthetic and real-world datasets. We performed experiments on different neural network settings and analyzed the performance of the neural prior to regularizing scene flow. Remarkably, we show that simple MLP-based prior to regularize scene flow is enough to achieve competitive results to the state-of-the-ar scene flow methods.
Datasets We used four scene flow datasets: 1) FlyingThings33 Things3D [33] which is an extensive
collection of randomly moving synthetic objects. We used the preprocessed data from [30]: 2)
KITTI [34, 35] which has real-world self-driving scenes. We used the subset released by [30]: 3) Argoverse [8] and 4) nuScenes [7] are two large-seale autonomous driving datasets with challenging dynamic scenes. However, there are no official scene flow annotations. We followed the data processing method in [45] to collect pseudo-ground-ie -roond--ruuttt -rouu scene flow. Ground points were removed from lidar point clouds as in [30] (please refer to the supplementary material for more details).
Metrics We employed the widely used metrics as in [30,36 45, 68] to evaluate our method, which are: & to denote the end-point error (EPE), which is the mean absolute distance of two point clouds; Accs to denote the accuracy in percentage of estimated flows when E < 0.05m or E' < 5%, where E the relative error; Accio denotes the percentage of estimated flows where < 0.1mm E' < 10%: and 0g which is the mean angle error between the estimated and ground-trer -ruth scene flows.

---

Implementation details We defined our neural prior for scene flow as simple coordinate- based MLP architecture with hidden layers, a fixed length of 128 for the hidden units, Rectified Linear Unit (Rel.U) activation and shared weights across points. The network input is the 3D point cloud Pti, and the output the scene flow F. We used PyTorch [41] for the implementation and optimized the objective function with Adam [24]. The weights were randomly initialized. We set a fixed learning rate of 8e-3 and run the optimization for 5k iterations with early stopping on the loss. For our settings and datasets, we found the optimization to mostly converge in less than Ik iterations. All experiments were run on a machine with an NVIDIA Quadro P5000 GPU and a 16 Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz.
Training setup for the parring--assed methods We used the publicly available implementation of the learning- based methods perform our experiments. The training settings for each method are: FlowNet3D and full-super PointPW--Net. trained on FlyingThings3D with supervision; and sulf--upe Just Go with the Flow and PointPWC-Neet trained FlyingThings33] with supervision and fine-tuned on domain-matched datasets with self-supervision (i.e., fine-tuned and tested on statistically similar data, KITTI, nuScenes, Argoverse respectively). Note that FlowNet3D and full-supervised PointPWC-Net was only trained on the synthetic ThinngT to demonstrate the poor generalizability of learning- saaening-basedd methods to other domains. Just Go with the Flow and self-supervised PointPWC-Net were trained using self-supervision, and although they do not require ground-truth annotations, they still require large-scale datasets for training to achieve competitive performance. Note that these self-supervised methods were both pretrained on fully- to provide adequate full supervision.
labeled
Optimization setup for the non-learning methods Non-rigid ICP [1] was originally proposed for mesh registration. We adapted
100
90
Accuracy [%)
80
for point cloud registration. A graph prior was recently proposed in [45] to optimize scene flow from point clouds. We implemented the method using the hyperparameters defined by the authors. The weight for the graph prior term is set to 10, the number of neighbors to build the h--N graph, if not explicitly specified, is set to 50, the learning rate to 0. and the number of iterations to 1.5k.
hidden layers
70
8 hidden-layyer
60
4 hidden layers
50
2 hidden layers
40
1 hidden layer
30
24
2
26
27
2
Number of hidden units
Figure 2: We analyzed the performance of our method on the KITTI test set when varying the number of hidden layers and hidden units of the MLP architecture. Accuracy is the Ace; metric.
4.1 Choosing the neural prior architecture
Fig. 2 shows how the performance of our method is affected when varying the neural prior MLP architecture: number of hidden layers and hidden units. The experiments were performed the KITTI test and all points included (i.e. without point sampling). The average number of points for the KITTI dataset is about 30k. We ran our method five times with different random seeds to include the uncertainty levels in the plot.
Overall, the performance of our method improved as we increased the number hidden layers and hidden units. For small numbers of hidden layers (e.g., and 2), the performance deteriorated when the number of hidden units large, around 128 and 256 (or 2 and 2*). We chose the MLP architecture with the best performance with relatively low computation time for our following experiments: 8 hidden layers and 128 (2²) hidden units.
4.2 Comparing to other methods
Table 1 shows how our method stands against other state-of-the-art methods different datasets and metrics. We set the number of points to 2,048, which follows the experimental protocols as in FlowNet3D [30] and Graph prior [45]. We ran the experiments 5 times to report uncertainties for runtime optimization methods (i.e., our method and the graph prior method [45]) with uncertainties for each run. The learning-based methods and non-rigid ICP are deterministic during runtime.
Our method achieved better performance in most datasets and metries. We considered the well-known Non-rigid ICP as a baseline for the non-learning methods (o). Our method outperformed the recent
5

---

Table 1: Performance of our method and others across different datasets and metrics. supervised methods trained on the synthetic FlyingThings3D with supervision and fine-tuned with self- superviienv non-learning methods that do not rely on training data. All experiments were run with 2,048 points. t means larger values are better while + means smaller values are better. We did report standard deviations smaller than 1e 2.
dataset. @ are suler-uuee methods trained on
matched datasets.
ByinggTinggsM [33] Trane 997 somples, Test: 2,000 mmplle
muScenes
[?]
Trii:: 1,523 sumples, Text complex
((m)) 4
Accst%) f
Accuo(3) 1
(:(rod)
E(eo) 4
Aess(163
Accu(3))
(r
Flow/Net3D [30]
0.134
22.64
54.17
0.305
0.505
2.12
10.81
0.620
PoistPWC-Net [68]
0.121
29.09
61.70
0.229
0,442
7.64
22.32
0,497
Go the How [38]
0.625
6.09
0,422
PointPWC-Nett [68]
0.411
6.87
22.42
0.406
Non rigid ICP []]
0.339
14.05
35.68
0.480
0.402
6.99
21.01
0.492
Graph prior [15]
0.255
16 06.5620.02
42.05=0.002
0.562
0.289
20.12:00001
43.51::0.02
0.337
Ours
09.16:20.23
46.74-0046
0.175::0.01
38.18
63/45:20.66
0.279:10004
KITTI Scene Flow [34, 35]
Argoverse Scene Flow [S]
Truin: 100 somples, Rist. 50 somples
Train: 2,591 sumples, Test. 212 samples
8(m)
Accx(5)
Accu(()) 1
(r(rd))
E(m)
Acc.(8))
Accu((5)
(red)
FlowNNrr3 [30]
0.199
10.44
38.89
0.386
0.455
1.34
6.12
0.736
PoiNtPWC-Neet [68)
0.142
29.91
59.83
0.239
0.405
8:25
25.47
0.674
Just Go How [38]
0.218
10:17
8.80
0.715
PoinsPWC-Net (68)
0.177
13:29
42.15
0.272
0.409
9.79
29.31
0.643
Non-tigid ICP [1]
0.338
22,06
43.03
0,460
0.461
4.27
13.90
0,741
Graph prior [45]
0.099
63.60::0.00
81.18=0.08
0,257
25.24:00:4
47.
47.60==0.02
0,467
Ours
0.050:1:.01
81.68::2.00
93,19=1.30
0.133::0.01
0.159:10.01
38.43::0:48
63.08:20.59
0.374:10.01
graph prior method by large margin. The supervised FlowNet3D and PointPWC-Net (using full- supervision loss) methods (o) had better performance on FlyingThings3D because they were trained these supervised methods produced
on it with supervision. If the dataset is unreliable results. The self-supervised methods (0). Just Go with the Flow and PointPWC-Net (using self-supervision loss), despite not being exposed to ground-truth labels during training, still generated better results than supervised scene flow estimation. Still, it is remarkable that with simple MLP regularizer and an optimization framework, our method can robustly estimate scene flow from point clouds with great accuracy.
srrthods-showing that self-supervision is an important direction for
Our PointPWC-Net results were different from those reported in the PointWW-NNet paper. The reasons are: In the official PointPWC-Net implementation, there is a threshold to limit the lidar point cloud within 35 meters of distance from the center. In our experiments, we used all points available in all ranges (up 85 m). Lidar point clouds get sparser as the distance increases, making it challenging to estimate scene flow in far ranges and sparse regions. Nevertheless, we did not shy away from this fact in our experiments. Also, in the original PointPWC-Net experiments, the authors used 8, 192 points. In ours, we used 2,048 points. Naturally, there exists performance gap between our reported results and theirs. We decided to use 2,048 points to follow the experiment protocols proposed in FlowNet3D [30] and graph prior [45] to facilitate comparisons across different datasets and models. Although simple and tested on sparse point clouds (2,048 points), our method achieved impressive results on different datasets. Please find further details and additional experiments in the supplementary material.
4.3 Estimating scene flow from large point clouds with high density
Real-world point clouds collected from depth sensors such as lidar typically have tens of thousands of points. We evaluated the performance of our method on large point clouds and compared it against the graph prior method, The KITTI and Argoverse Scene Flow datasets were used, given that both have large point clouds with high density.
Fig. 3 shows the performance on the KITTI Scene Flow dataset, in terms of accuracy and computational time, of our method and the graph prior method when varying the number of points. Our method's accuracy (Aces) increased as the number of points grew until around 20k points and then saturated, while the computational time slowly increased. In contrast, the graph prior achieved lower accuracy and dramatic growth computation. The computational complexity of our MI.P- based prior grows linearly in the number of points, On)), while the graph prior grows quadratically

---

Table 2: Performance of neural prior and the graph prior [45] when using all points available.
350
90
96
all metrics by a margin while being ~5x faster
Our method achieved better performance
80
Accurady (%
k=50 and ~00 faster if k=200 (for KITTI..
Time
200
KITTI Scene
Graph prior Ours
Average number of points: <-30k
105
65
(m)
Aces (%)
Acco f (%)
(rod)
4
(4)
32
Graph prior (k=50)
0.225
65.50
70.32
0.277
162.97
10
10a 2x10* 3x10*
5x10
Graph prior (k=200)
0.082
84.00
88.45
0.141
310.12
Number of points
Ours
0.025
95.68
98.00
0.085
38.33
Argoverse Scene Flow Averuge number points: <-00k
Figure 3: Performance of our neural prior and the graph prior [45] when varying the number of points. Our method achieved higher accuracy (Aces) and better time complexity, Results were averaged over the KITTI Scene Flow dataset.
Accio (96
(m)
(90)
(rnd)
(4)
Graph prior (k=50)
0.249
46.92
61.72
0.494
410.21
Ours
0.043
86.04
94.07
0.244
84.46
in the number of points, On?²). The graph prior relies on the construction of a graph Laplacian matrix to use regularizer (i.e., L RTx", where n is the number of points).
The accuracy the graph prior method degraded after 10k points. The graph regularizer needs more than 5k iterations for higher density point clouds to converge to reasonable solution or carefully tuned schedulers to accelerate its convergence. Moreover, the k--NN graph built with 50 neighbors, and for higher density point clouds, larger graphs might be necessary for better regularization.
Table 2 shows a quantitative comparison between our method and the graph prior method on the KITTI and Argoverse Scene Flow datasets when using all points. Our method achieved better performance on all metrics. We also reported results for the graph prior method when setting the number of neighbors, k, to 200. According to our results in Fig. 1, the scene flow accuracy saturated after k=200. Fig. 4 shows a qualitative example of scene flow estimation using our method.
These results show that our method scales to large point clouds with high density, and gain a great improvement in performance with much denser point clouds. In contrast, training supervised/self- supervised models with high-density point clouds is not always practical due to high memory usage. Typically, such models are trained with up to 8k points.
Performance and inference time trade- offs Estimating scene flow with runtime optimization is usually slower than using learning-based methods (10x-100x slower). The trade-off, however, depends on the application. If robustness/generalzzailtt not an issue but rather the inference time, our proposed objective can train self-supervised model and act as a surrogate of our non-learning method but inheriting the faster inference time from the trained model. We show an example of lidar point cloud densification (Section 4.6) to generate denser point clouds that can be used for robotics applications such as offline mapping, creating denser depth maps, etc., that would not require real-time inference.
Ground truth
Estimated
Figure 5: Example of failure case. Partial scene from FlyingThingsss Our nearest-neighbor-basee loss might fail when handling large missing parts, occlusions, and bad correspondences, Green points are the target, and red points are the shifted blue points by the estimated scene flow (yellow arrows).
4.4 Limitations
Although our method achieved better computational complexity than other non-learning-ban methods, the inference time is still limiting for some applications that demand real-time inferences.
7

---

Panoramic image
Ground truth
Estimated ours
Estimated FlowNet3D
Figure 4: Qualitative example of a scene flow estimation using our proposed method. The complex and highly dynamic driving scene is from the Argoverse Scene Flow dataset. The scene flow estimated by our method is close to the ground truth. We also show prediction using the supervised FlowNet3D method trained on FlyingThings3D and fine-tuned on the KITTI Scene Flow dataset. Note how the scene flow deviated from the ground truth when the inference was performed on an cut-of-the-diitributtoo sample. The scene flow color encodes the magnitude (color intensity) and direction (angle) of the flow vectors. For example, the purplish vehicles are heading northeast.
Another limitation is that the loss function used relies nearest neighbors, which might find bad correspondences due to partial point clouds and occlusions. Fig. 5 shows failure case because of the nearest-neighbor-base distance loss. Few corresponding points due missing parts in the scene might lead to incorrect flow estimations.
4.5 A continuous scene flow field
Our neural prior implicitly regularizes the scene flow through the coordinate- based MLP network. Thus, our method allows for a continuous scene flow representation instead of a discrete representation such in graph-based priors.
An advantage of a continuous scene flow representation is that we can reuse the optimal network weights to estimate dense long-term correspondences across sequence of point clouds (see Section 4.6). Fig. 6 shows how the estimated scene flow and the continuous flow field change the optimization converges solution.
4.6 Application: scene flow integration
Here we demonstrate how to use our method to perform scene flow integration. Given a temporal sequence of point sets, {So, S2, ,SM}.
first optimize for the pairwise scene flows, Fo FM--I-M M) J. using our proposed method with the optimal neural prior parameters, (0)-1, O - ... + 1MM--MM w saved. Then, starting from fo-,1, can integrate long-term flows using the classic Forward Euler method recursively for m=1:M-1 iterations as
fo-mm11 fo-mm + +g(Po + fo O 1m)
(5)

---

Podestiin
Vehice
Iteration 0
Iteration 1000
iteration 2000
Ground truth
EFE (ml
6.55
8.0-
Iteration
Figure 6: Example showing how the estimated scene flow and the continuous flow field (bottom) given by our neural prior change as the optimization converges to a solution. We show top-view dynamic driving scene from Argoverse Scene Flow. The scene flow color encodes the magnitude (color intensity) and direction (angle) of the flow vectors. For example, the purplish vehicles are heading northeast. The red arrow shows the position and direction of travel of the autonomous vehicle, which is stopped, waiting for pedestrian to cross the street. Note how the predicted scene flow is close to the ground truth at iteration 2k. iteration 0, the scene flow random, given the random initialization of the neural prior. Thus having very small magnitudes for the random directions. As the optimization went on, the flow fields became better constrained. A simple way to interpret the flow fields is imagine sampling point any location in the continuous scene flow field to recover an estimated flow vector. For cxample, imagine sampling a point around the orange region in the flow field at iteration 2k (green arrow in the bottom right). The direction of the flow vector will be pointing southeast at a specific magnitude, similar to the vehicles in the orange region.
This gives the long-tew terg-termm scene flow Fo-s (ffo-+)) } from So SM. Note that we are not relying discrete hearest-neighhor-basee hased interpolations. Our neural scene flow prior a continuous representation that naturally provides continuous scene flow estimations. Fig. 7 shows an example of an Argoverse scene where we applied such technique to integrate 10 point clouds into single frame to densify the point cloud.
Conclusion
We show that how hand-designed coordinate-bait network architecture can serve as a new type of implicit regularizer in the runtime optimization for the scene flow problem. Our neural prior gets rid of the need for massive labeled/unlabeled training data while being scalable with dense point clouds. Additionally, since we infer prior knowledge from the network architectures instead of from data, methods.
approach can generalize to out-of-ttheddistriuutio scenarios as compared to learning- The continuous flow representation also allows for flow integration across long sequence that be used in many robotics applications such as offline mapping. We believe this paper shows a promising direction for large-scale, real-worke scene flow estimation without data supervision.
Broader impact
Our proposed neural scene flow prior allows for estimating 3D motion fields from large-scale dynamic scenes without annotating massive data but preserving iameralizabiiiy-mmaaaee where motion prediction required. Especially in computer vision and robotics communities, robust scene flow estimations is crucial. For example, autonomous vehicles need to predict future distribution of the surrounding objects to avoid catastrophe dynamic environments; safe human-computer interaction is enabled with precise dynamic flow predictions. Our work also encourages further
useful for scenarios

---

Single licarr cloud
Accumulated lidar point cloud (rigid)
Accumulated lidar point cloud (ours)
Figure 7: Example a scene flow integration to densify an Argoverse lidar point cloud. The left and middle columns are top and front view of the point cloud, respectively. The rightmost column shows the accumulated point cloud projected onto the image. Note the smearing effect on the dynamic objects when rigidly accumulating the point clouds (middle row). Accumulation using neural prior nicely produced a denser point cloud while taking care of all dynamic objects the scene. Here, rigid means that the point cloud accumulation was performed using rigid registration method (i.e., ICP) where rigid 6-DoF poses are used for the registrations.
exploration in the combination of the innovative coordinate- baaddd networks and classical runtime optimization algorithm. This dataless approach offers an affordable solution for many industrial problems without adequate supervision (e.g., autonomous driving).
However, as Al-based research, could be misused by malicious groups for nefarious purposes. For example, the collected data might contain sensitive information that potentially invades privacy, and can be used for illegal data trading. Moreover, such research has the potential to be used in autonomous weapons and military drones. The potential evil of our method needs attention and needs to be prevented. We hope to motivate the community to take full advantage of the innovations of this work to benefit society.
Acknowledgments and Disclosure of Funding
The authors would like to thank Chen-Hsuan Lin for useful discussions through the project, review and help with section 3. We thank Haosen Xing for careful review of the entire manuscript and assistance in several parts of the paper, Jianqiao Zheng for helpful discussions. We thank all anonymous reviewers for their valuable comments and suggestions to make our paper stronger.
References
[1] Brian Amberg, Sami Romdhani, and Thomas Vetter. Optimal step nonrigid ICP algorithms for surface registration. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1-8. IEEE, 2007. 2, 5,
[2] Rie Kubota Ando and Tong Zhang. Learning on graph with Laplacian regularization. Neural Information Processing Systems (NeurIPS). 19:25, 2007.
[3] Matan Atzmon and Yaron Lipman. SAL:: Sign agnostic learning of shapes from raw data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2565-2574, 2020.
[4] Tali Basha, Yael Moses, and Nahum Kiryati. Multi-view scene flow estimation: A view centered variational approach. International Journal of Computer Vision (IJCV). 101(1);6-21, 2013.
[5] Alexander I Bobenko and Boris A Springbom. A discrete Laplace- operator for simplicial surfaces.
Discrete & Computational Geometry, 38(4):740-756, 2007.
[6] Fabian Brickwedde, Steffen Abraham, and Rudolf Mester. Mono-SF: Multi-viee geometry meets single- view depth for monocular scene flow estimation of dynamic traffic scenes. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2780 2790, 2019. 2
10

---

[7] Holger Caesar. Varun Bankiti. Alex H Lang. Sourabh Vora, Venice Erin Liong. Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving- In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2020. 4, 6
[8] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak. Andrew Hartnett, De Wang, Peter Cart, Simon Lucey, Deva Ramanan, et al. Argoverse: 3D tracking and forecasting with rich maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
8748 8757, 2019. 4, 6
[9] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 5939-5948, 2019. 1,
[10] Haili Chui and Anand Rangarajan. A new point matching algorithm for non-rigid registration. Computer
Vision and Image Understanding (CVIU), 89(2-3):114 141, 2003. 2
[11] Chinthaka Dinesh, Gene Cheung, and Ivan V Bajić. Point cloud denoising via feature graph laplacian regularization. IEEE Transactions Image Processing. 29:4143-4155,, 2020.
[12] Li Ding and Chen Feng. DeepMapping: Unsupervised map estimation from multiple point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8650-8659,
2019. 2.
[13] Yilun Du. Yinan Zhang. Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4D view synthesis and video processing. arXiv e-prints, pages arXiv-2012, 2020.
[14] Marvin Eisenberger, Zorah Lahner, and Daniel Cremers. Smooth shells: Multi-scale shape registration with functional maps. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 12265-12274, 2020. 3
[15] Haoqiang Fan, Hao Su, and Leonidas Guibas. A point set generation network for 3D object reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR). pages 605-613, 2017. 4
[16] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic
monocular video. arXiv preprint arXiv:2105. 200.06680 2021. 3
[17] Xiuye Gu, Yijie Wang. Chongruo Wu, Yong Jac Lee, and Panqu Wang. HPLFlownet: Hierarchical permutohedral lattice flownet for scene flow estimation large-scale point clouds. In Proceedings of the IEFE Conference on Computer Vision and Pottern Recognition (CVPR), pages 3254-3263, 2019. 1,
[18] Simon Hadfield and Richard Bowden. Kinecting the dots: Particle based scene flow from depth sensors. In Proceedings of the International Conference on Computer Vision (ICCV), pages 2290-2295. IEEE, 2011.
[19] Simon Hadfield and Richard Bowden. Scene particles: Unregularized particle-based seene flow estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 36(3):564-576, 2013. 2
[20] Michael Homaack, Andrew Fitzgibbon, and Carsten Rother. SphereFlow: 6 DoF scene flow from RGB-D pairs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
3526-3533, 2014. 2
[21] Frédéric Huguet and Frédéric Devernay. A variational method for scene flow estimation from stereo sequences. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pages 1-7, IEEE, 2007. 2
[22] Junhwa Hur and Stefan Roth. Self-supervised monocular scene flow estimation. In Proceedings of the
IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7396-7405, 2020. 2
[23] Huaizu Jiang, Deqing Sun, Varun Jampani, Zhaoyang Lv, Erik Learned- Miiler,, and Jan Kautz. SENSE: A shared encoder network for scene-flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 31 3195-3204, 2019. 2
[24] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, Proceedings of the International Conference Learning Representations (ICLR),
2015. 5
[25] Yair Kittenplon, Yonina C Eldar, and Dan Raviv. FlowStep3D: Model unrolling for self-supervised scene flow estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. [26] Hao Li, Robert W Sumner, and Mark Pauly. Global correspondence optimization for non-rigid registration of depth seans. In Computer Graphics Forum, volume 27, pages 1421-1430. Wiley Online Library, 2008
[27] Rui Li and Stan Selaroff. Multi-scale 3D scene flow from binocular stereo sequences. Computer Vision and Image Understanding (CVIU), 110(1):75-90, 2008. 2
[28] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. PointCNN: Convolution on X-transformed points. Neural Information Processing Systems (NeurIPS), 31:820-830, 2018. 2
[29] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 3
11

---

[30] Xingyu Liu, Charles R Qi, and Leonidas J Guibas. FlowNet33: Learning scene flow in 3D point clouds. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 529-537,
2019. 1, 2, 4,5, 6 6
[31] Xingyu Liu, Mengyuan Yan, and Jeannette Bohg. MeteorNet: Deep learning dynamic 3d point cloud sequenees. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pages 9246-9255, 2019. 2
[32] Wei-Chiu Ma. Shenlong Wang. Rui Hu, Yuwen Xiong, and Raquel Urtasun. Deep rigid instance scene flow. In Pruceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
3614-3622, 2019. 2
[33] Nikolaus Mayer, Eddy 1lg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4040-4048, 2016. 4, 6
[34] Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In Proceedings of the
IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 3061-3070, 2015. 4, 6 [35] Moritz Menze, Christian Heipke, and Andreas Geiger. Joint 3D estimation of vehicles and scene flow.
ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2:427, 2015. 4, 6 [36] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR). pages 4460-4470, 2019. 1,3
[37] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NcRF: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), pages 405-421. Springer, 2020. 1,3
[38] Himangi Mittal, Brian Okorn, and David Held. Just go with the flow: Self-supervised scene flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11177- 11185, 2020. 6.
[39] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF; Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), pages 165-174, 2019. 1,3
[40] Keunhong Park. Utkarsh Sinha. Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin Brualla. Deformable neural radiance fields. arXiv preprint veXiv:2011.12444 2020. 3
[41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. PyTorch: An imperative style, nigh-performance deep learning library. In Neural Information Processing Systems (NeurIPS), 2019. 5
[42] Mark Pauly, Niloy Mitra, Joachim Giesen, Markus II Gross, and Leonidas J Guibas. Example-based 3D
scan completion. In Symposium on Geometry Processing, pages 23-32, 2005. 2
[43] Jean- Jean-Philippe Pons. Renaud Keriven, and Olivier Faugeras. Multi-view stereo reconstruction and scene flow estimation with a global image-based matching score. International Journal of Computer Vision (ICCV) 72(2):179 19799-99,222999 2007. 2
[44] J-P Pons, Renaud Keriven, O Faugeras, and Gerardo Hermosillo. Variational stereovision and 3D scene flow estimation with statistical similarity measures. In Proceedings of the International Conference on Computer Vision (ICCV), volume 2. pages 597-597. IEEE Computer Society, 2003.
[45] Jhony Kaesemodel Pontes, James Hays, and Simon Lucey. Scene flow from point clouds with without lcarning. In Proceedings of the International Conference on 3D Vision (3DV). IEEE, 2020. 2, 3, 5, 6, 7 [46] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR). 2021.
[47] Gilles Puy. Alexandre Boulch, and Renaud Marlet. FLOT: Scene Flow on Point Clouds Guided by Optimal Transport. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 1,2
[48] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guihas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 652-660, 2017. 2
[49] Charles R Qi, Li Yi. Hao Su, and Leonidas J Guibas. PointNet*+: Deep hierarchical feature learning on point sets a metric space. Neural Information Processing Systems (NeurIPS), pages 5105-5114, 2017.
[50] Julian Quiroga, Thomas Brox, Frédéric Devernay, and James Crowley. Dense semi-rigid scene flow estimation from rgbd images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 567-582. Springer, 2014. 2
[51] Rishav Rishav, Ramy Battrawy, René Schuster, Oliver Wasenmüller,, and Didier Stricker. DeepLiDARFlow: A deep learning architecture for scene flow estimation using monocular camera and sparse LiDAR. In Proceedings of the IEEE/RSJ Conference on Intelligent Robots and Systems (IROS), pages 10460-00460 IEEE, 2020.2 2
12

---

[52]
Rohan Saxena, René Schuster, Oliver Wasemmuller, and Didier Stricker. PWOC-3D: Deep ocelusion-aware end-to-end scene flow estimation. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), pages
324-331. IEEE, 2019.
[53]
René Schuster, Christian Unger. and Didier Stricker. A deep temporal fusion framework for scene flow using learnable motion model and occlusions. In Proceedings of the IEEE Workshop on Applications of
Computer Vision (WACV). pages 247-255, 2021. 2
[54] Lin Shao, Parth Shah, Vikranth Dwaracherla, and Jeannette Bohg. Motion-based object segmentation based dense RGB-D scene flow. IEEE Robotics and Automation Letiers, 3(4):3797-3804, 2018. 2
[55] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Neural Information Processing Systems (NeurIPS), 33, 2020.
[56] Vincent Sitzmann, Michael Zollhoefer, and Gordon Wetzstein. Scene representation networks: Continuous 3D-str neural scene representations. Neural Information Processing Systems (NeurlPS),
32: 12:1111-133, 2019. 3,
[57] Olga Sorkine. Laplacian mesh processing. Eurographics (STARs), 29, 2005. 3
[58] Olga Sorkine and Mare Alexa. As-rigid-as-possbbll surface modeling. In Symposium on Geometry Processing, volume 4, pages 109-116, 2007.
[59] Chen Sun, Abhinav Shrivastava, Saurahh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. Proceedings of the International Conference on Computer Vision (ICCV), pages 843-852, 2017. 3
[60] Zachary Teed and Jia Deng. RAFT-3D: Scene flow using rigid-motion embeddings. In Proceedings of the
JEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8375-8384, 2021. 1.2
[61] Tvan Tishchenko, Sandro Lombardi, Martin Oswald, and Mare Pollefeys. Self-supervised learning of non-rigid residual flow and ego-motion. In Proceedings of the International Conference on 3D Vision
(3DV). 2020.
[62] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene
from monocular video, 2020.
[63] Dmitry Ulyanv,, Andrea Vedaldi, and Victor Lempitsyy. Deep image prior. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9446-9454, 2018. 2,
[64] Sundar Vedula. Simon Baker, Peter Rander, Robert Collins, and Takeo Kanade. dinee-dimensional scene flow. In Proceedings of the International Conference Computer Vision (ICCV), volume 2, pages 722-729. IEEE, 1999.
[65] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph CNN for learning on point clouds. ACM Transactions on Graphics (TOG), 3815):1-12.
2019. 2
[66] Zirui Wang, Shuda Li, Henry
Victor Prisacariu, and Min Chen. FlowNet3D++: Geometric flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 91-98, 2020.
losses for deep
[67] Wenxuan Wu, Zhongang Qi, and Li Fuxin. PointConv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9621-9638 2019.
[68] Wenxuan Wu, Zhi Yuan Wang. Zhuwen Li, Wei Liu, and Li Fuxin. PoinPWW-Nett. Cost volume on point clouds for (self-) supervised scene flow estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 88- 88-107. Springer. 2020. 1. 2, 4.
[69] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. arXiv preprint arXiv:2011.12950 2020. 3
[70] Gengshan Yang and Deva Ramanan. Upgrading optical flow to 3D scene flow through optical expansion. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1334-1343, 2020.
[71] Jin Zeng, Gene Cheung, Michael Ng, Jiahao Pang, and Cheng Yang. 3D point cloud denoising using graph Laplacian regularization of low dimensional manifold model. IEFF Transactions on Image Processing, 29:3474-3489, 2019. 3
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We described the limitations in the experiment section, and we also provided failure cases.
13

---

(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discussed in the Broader Impact section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or URL)? ['ess] We cite all the data we used in the experiment section. And we will release code in personal GitHub repository.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] In the experiment section, we specify implementation details for all methods, and all data used.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We report error bars for ppptimizationbaase methods which have uncertainties.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] We included in the implementation details of the experiments section.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We have cited all the data we used.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "VkSKIU-Ucp", "reviewer": "Reviewer_XKk4", "summary": "This paper studies the problem of scene flow: estimating 3D motion fields between given point clouds from a dynamic scene. The proposed method predicts scene flows by optimizing a neural network at execution time, using randomly initialized weights. \n\nThis approach of using a network architecture as a prior is shown to have better generalization performance for out-of-distribution samples compared to learning-based methods which require offline datasets. In contrast to another recently introduced non-learning approach which uses graph Laplacian priors, this method provides a continuous scene flow representation, and has better time and accuracy performances in large scale scenes.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well-written paper, which I enjoyed reading. While the problem studied in the paper has been around for a long time, the approach for solving the task is novel and interesting. The claims in the paper are supported by comprehensive experimental results across four different datasets.\n\nMy questions to the authors along with some comments are listed below.\n \n- Selection of MLP networks to represent scene flows: Processing point clouds (or point sets) with MLPs has a few drawbacks that are already well-known, such as imposing an ordering of the points and requiring a fixed cardinality for the point set. While the distance function used in Eq. 4 allows point sets of different sizes, the MLP inputs point clouds of fixed size (set as 2048 in the experiments). A discussion about how to take point clouds of differing sizes to compute the scene flow would be useful -- especially for the case when computing over the sequence of M point sets. \n\n- Related to the previous comment, could optimizing the network weights at execution time using an architecture designed to operate on point clouds (e.g. [1]) regularize the motion field better? Alternatively, can the method be modified to take a single point as input to the MLP at a time for allowing variable point cloud sizes?\n\n- Can you comment on the computation/inference time of other learning- and non-learning-based methods? Adding how much time the non-rigid ICP and learning-based take would be helpful.\n\n- What is meant by the rigid accumulation of point clouds, shown in Figure 7? Does it refer to computing the flow in a discrete fashion using nearest neighbors as opposed to the continuous flow estimation? \n \n\nMinor comments:\n\n- In line 110, I believe the reference to [15, 24] should be [15, 27] instead.\n- You may consider adding a zoomed inset image in the top-down views of Figure 5 for better visual clarity.\n\n\nReferences\n\n1- Qi, C.R., Yi, L., Su, H. and Guibas, L.J., 2017. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have assessed the limitations of their work, including a failure case due to the nearest-neighbor distance term in the loss function. Figure 6 shows an example of this failure mode. Other than that, the paper contains an analysis for the degradation of the computation-time performance of the presented method, along with the baseline’s performance.\n\nThe paper also includes a broader impact section, for discussing potential outcomes of this work.\n", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical issues with this paper, in my opinion.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "KH1I2thSGUC", "reviewer": "Reviewer_y7UW", "summary": "This paper studies scene-flow estimation. Following the idea of deep-image prior, it proposes an unsupervised way to learn scene flow. The results are competitive, or even better than supervised flow-estimation methods trained on very different training set. Moreover, it also out-performs existing optimization based unsupervised flow-estimation methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I'm not an expert in the field of 3D flow estimation, and not super familiar with the SOTA unsupervised or self-supervised methods for this tasks. It's likely that my interpretation or understanding is biased or wrong. For the paper authors and other people, feel free to correct me if you find such things.\n\nStrength:\n1. This paper identifies the poor generalizability of one established supervised method. The results are supervised and it performs very poor on new data domain. If one or two more baselines could be studied, this result would be more convincing. \n2. The proposed method out-performs existing work \"graph prior\" significantly and consistently. Another good property of this work is that many prior-deep learning works are mentioned or compared, which is rarely now days and show be encouraged.\n3. The experiments are conducted on multiple different experiments, and promising results are presented and visualized.\n4. Writing is mostly clearly and easy to follow. \n\nWeakness:\n1. The supervised baselines seem to be very weak. On the unseen dataset, the Acc_5 could be as low as 1.34%. Even on the training set (FyingThings), Acc_5 is only 22.64%. As a comparison, the proposed method can achieve 80% Acc on other dataset. Therefore, I don't think these supervised baselines are strong, which makes the results less convincing.\n2. For 3D flow estimation, shouldn't EPE the major metric? \n3. Recently, the RAFT model become the most popular method for 3D flow estimation. For 3D flow, RAFT-3D [1] is the current SOTA method. I think it's a more proper baseline than FlowNet3D, and I'm also wondering how well will it generalize to new dataset. In fact, the provides numbers/models for both FlyingThings and Kitti, which help us to understand both the generalizability and performance upper bound.\n[1] RAFT-3D: Scene Flow using Rigid-Motion Embeddings. Teed and Deng. CVPR 2021\n4. In Deep image prior, it tricky to find an optimal stopping iteration during training time. It's unclear to me how does this work handle this problem. \n5. This work should be classified as self-supervised learning approach IMO. It's essentially corresponding learning, during which the 3d flow is also learned as an architectural prior. \n6. The cycle-consistency loss is an important innovation but not ablated.\n7. The architectures are inspired by recent implicit models but lack of motivation. I'm not convinced by the advantages of doing this and why this framework is proper for flow-estimation.\n\nMisc / Questions:\n1. In Fig.3, why does one line of graph priors decrease over time?\n2. In Tab.1, mean/var of \"graph prior\" and \"ours\"should be reported for all metrics for completeness.\n3. Deep image prior trains one model for each picture. Does this work also train one model for each scene?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Limitations: one failure case is shown in Sec 4.5 and Fig.6 to show the limitation of the NN loss.\n2. Societal impacts are properly discussed (L309ff).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {"ethical_concerns": "I have read the Ethics Guidelines, and I don't find any concerns from this paper.\n1. Potential negative societal impacts: It doesn't involve any social experiments, and doesn't use human-derived data.\n2. General ethical conduct: the dataset used in this work are public ones."}}, {"review_id": "FyDZjgcWj4", "reviewer": "Reviewer_8oZd", "summary": "This submission presents a method for estimating scene flow between two point clouds that uses a neural network as an implicit regularizer during optimization. The submission revisits the classical optimization-based approach where instead of training a neural net to estimate flow for a given scene, it optimizes the flow per scene and does not need to train the network beforehand. Instead, the network is “trained” to produce flow for the given scene, such that representing the flow with the network introduces some form of implicit smoothness regularization.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\n\nThe idea presented in the submission is interesting and while neural networks have been applied as implicit functions in other domains, their application to scene flow is novel as far as I know. The presented method connects modern neural network based approaches to flow estimation to classical flow estimation approaches. The listed related work is extensive. However, I believe that the comparison to related approaches has important gaps – I will give more details about this in the next point.\n\n# Quality\n\nThe description of the method is technically sound. The experimental evaluation of the proposed method and its components (apart from the comparison to other work in Table 1) is extensive and insightful, e.g. Sections 4.1, 4.3, 4.4, and 4.6. I also appreciate the discussion of the failure case in Section 4.5 – more examples would be even better (maybe in the supplemental material). I find Figure 5 especially helpful to understand how the model is optimized over time and how it interpolates between points. From that figure, I’m curious if the removal of the ground plane is essential for the method to work well. Maybe the authors could comment on that if they have done those experiments.\n\nUnfortunately, some claims seem overly general and are not well supported by evidence: a) L26-28 portrays the need for data in itself as a problem, including training on synthetic scenarios and then fine-tuning on small real datasets. But this in itself is not a problem, at least not for domains where such data exists, as for autonomous driving which this paper focuses on. b) L29 claims that poor generalization is a problem but fails to give evidence for this in general. Yes, generalization is poor when one goes from FlyingThings3D to KITTI, but this looks like a strawman because the more relevant question would be how well the models generalize between KITTI, nuScenes, Argoverse, and other autonomous driving scenarios. In contrast to the statement here, e.g. [28] states that “we show that our network, even trained on synthetic data, is able to robustly estimate scene flow in point clouds from real scans, showing its great generalizability.” c) L13-15 make the argument that because optimization happens at runtime and no offline data is needed, this approach would be ideal for deployment in autonomous driving – ignoring the problem that optimization at runtime is much slower compared to performing optimization during training time, which will in fact make it very challenging to apply this method for autonomous driving to produce scene flow in real time. Based on Figure 3, inference takes > 5s for a single scene. Here the discussion should be more balanced and mention both the advantages and the disadvantages of performing optimization at runtime. Figure 1 is also misleading in this regard as it cherry-picks one alternative method that is slower but disregards other related methods that are much faster than the proposed approach. L168 states that the evaluation focuses on both accuracy and computational cost but does not take the cost at inference into account when comparing to other methods in Table 1.\n\nThe comparison to SOTA in Table 1 does not appear fair to me for a number of reasons:\na) The SOTA comparison only includes a single (and no longer SOTA) supervised method FlowNet3D and ignores important related work with published results on FlyingThings3D and KITTI, e.g. HPLFlowNet (CVPR 2019), PointPWC-Net (ECCV 2020), FLOT (ECCV 2020)\nb) The supervised method FlowNet3D is only evaluated with domain transfer from FlyingThings3D to driving scenes even though a model trained on KITTI is available. This is stated in the text but not in the caption of Table 1 making it easy to miss. Testing generalization is important but to give the full picture, the KITTI-trained model should be included and it should be made clear when a model is tested outside of the training domain.\nc) Self-supervised methods here are only optimized on the training scenes and need to generalize to the test scenes. For a fair comparison, it would be good to also report their scores after fine-tuning on the test scenes in order to have access to the same information that the presented method has during optimization.\nd) To adequately show the advantages and disadvantages of these methods, an estimate of the inference time should be added to the table as this will vary strongly between the compared methods and orders of magnitude differences in inference time substantially change the applicability of a method for estimating motion (which often needs to be done in real time).\n\nL162: I’m not convinced that comparing the number of parameters is very useful here. Parameters have very different semantics in both cases. For a learned model, they correspond to the memory required to store the model, but in the proposed method, they need to be optimized for inference. As the learned model represents a flow estimation model for “all” scenes it should not be surprising that this needs to be larger than a model that represents the flow for a single scene. If one wants to make the comparison, both memory and inference time should be taken into account.\n\n# Clarity\n\nThe submission is written clearly and well organized. All code is included in the supplementary material, which should make it easy to reproduce the work.\n\n# Significance\n\nThe presented method and results are important to the community and could be the basis for further exploration into this approach of estimating scene flow during inference using neural networks as implicit functions. My main reservation regarding this work regards the comparison to existing learning-based approaches both in the text and in the empirical results. Here the paper seems to focus too much on the advantages of the presented method and does not always give the whole picture. \n\nTo make this very clear: I think that the paper has very valuable contributions as it shows progress along a direction that is not the main path taken by related work right now. Even if a complete comparison to related work (as described above) shows that this approach is not the most practical method (in terms of accuracy and / or inference time) it would still be a very valuable contribution because it shows progress along an approach that is very different from most existing work. But only with a fair portrayal and comparison of related work will I be able to recommend accepting the paper.\n\nEDIT: My main concerns have been addressed in the post-review discussion with the authors.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are not fully discussed (see above).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "1HY8qA0wb74", "reviewer": "Reviewer_rKbV", "summary": "\nThis paper highlights the fact that deep learning has caused a paradigm shift in many computer vision applications from methods based on optimization with strong priors to ones that optimize  feed-forward neural networks in an offline process. The paper argues that these feed-forward models are bad at generalization and proposes to use the network in a radically different way. In essence, the network are used as a parametrization for a traditional optimization solution - allowing the network's inductive biases to be used as a “prior”. While using a network structure as a prior for optimization is not new (ala Deep Image Prior, DeepSDF, NeRF etc.), applying this type of parametrization to the problem of optical flow is new. The results are competitive with trained approaches and quite impressive for such a simple solution.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOverall this is a well-written and interesting paper. Please find my detailed comments below where I have labelled each point as either positive (+), mixed (+/-) or negative (-).\n \n**Originality**: \n\n(+) The method is new in the sense that a neural parameterization of this type has not been used before for scene flow estimation. This type of parameterization has multiple advantages such as the fact that the scene flow becomes a continuous field that can be sampled at any point in space which is a much more natural way to represent the flow field. In contrast traditional approaches based on variational methods would typically solve for the flow field only on 2D frames of projected points. \n\n(+) In my view, the work ***is*** a novel combination of well-known techniques. Optimization-based approaches for scene flow have existed for a long time and quite sophisticated methods have been developed. Similarly, the use of the inductive biases of deep networks as a prior to guide optimization was first described in “deep image prior” and has been exploited in multiple subsequent works. However, the application of these to the problem of scene flow estimation is novel and the authors have done a very good job in describing how their approach bridges the different paradigms.\n\n(+/-) Overall, the work is well-positioned with respect to previous contributions in the space (like Deep image prior, supervised methods, and traditional optimization approaches). However, there is one very related work that seems to be missing and that is DeepMapping [A]. Like the proposed approach, DeepMapping also aligns point clouds in an “unsupervised manner” (i.e. test time optimization) and uses a deep network for the parametrization for the pose. I do think there are clear differences between the proposed method and DeepMapping in the sense that the proposed method recovers a full, continuous flow field whereas DeepMapping only recovers the pose. This means that the proposed method can compute the flow for dynamic objects whereas DeepMapping is limited to rigid scenes. The proposed method also appears to be much simpler than DeepMapping in terms of its design and training. However, it would probably be good to reference and compare DeepMapping to the proposed approach (in terms of a description of the differences) as it probably is the closest existing method. It would also be interesting to see how DeepMapping performs compared to the proposed method on rigid scenes.\n\n**Quality**: \n\n(+) As far as I can tell, the submission is technically sound and there are no major errors in the method. \nThe claims related to the proposed approach (in terms of performance compared to existing methods) are well supported. The numerical experimentation is sufficient to allow the reader to place the method with respect to recent supervised approaches and the results do indicate that the method is very competitive with these approaches. \n\n(+) The paper is relatively polished and complete in the sense that the idea is fully described and evaluated to an extent that allows the reader to see that the approach works and what problems it is suited for.  It would have been nice to see more demos on different types of data (currently only lidar point clouds from vehicles are shown) and I’m really curious how this would work on denser point clouds from RGB-D sensors in dynamic indoor scenes. The attached video clip in the supplementary showing the approach in-action is a bit difficult to understand as it’s just a screen grab of the python program running with no editing. However, I don’t think these issues detract significantly from the quality of the paper.\n\n(-) The authors do a good job of highlighting the strengths of the method, but fall a bit short in describing the weaknesses. The main advantage of approaches that are only optimized at training time is of course their inference speed and this has been one of the major factors that has led to the widespread adoption of deep learning models. Whereas traditional state-of-the-art scene flow methods would take minutes or hours to process a single example, methods like FlowNet3D can do it in seconds. In essence, there exists a strong tradeoff between generalization and inference speed and I think it is quite important for the authors to make this clear in the paper. Inference time is currently only compared to GraphPrior in Figure 3, but I think inference times should be given for other methods like FlowNet3D as well (perhaps by adding them to Table 1).\n\n**Clarity**: \n\n(+) In general the submission is very clearly written, and makes a good argument for the neural prior and optimization-based approach that is proposed. The paper is well organized and contains all the details to replicate the approach (the authors have also provided the source code for reproducibility).\n\n**Significance**: \n\n(+) In my view the results are quite important for two reasons. It shows that a very simple optimizer (Adam) combined with an easy to implement loss function and neural prior can achieve very good results on the scene flow task - even beating supervised approaches in some cases. Secondly, it shows that a continuous representation is the best way to represent 3D flow fields (similar to what has been shown on other 3D tasks too - like for volume rendering in Nerf). Therefore I think these result could make a large impact in this research area too.\n\n(+) Furthermore, I do think other researchers are very likely to be inspired by the approach and try to build on top of it to overcome its currently limitations (like inference speed and pushing the accuracy to maybe beat the current supervised methods). This is because the proposed method opens up many interesting possibilities for future research. \n\n(+) Although the method does not advance the state of the art in terms of numerical performance, it absolutely presents a method that is better in terms of complexity and ease of implementation compared to previous work. In my view, the simplicity of the approach, and the advantages afforded by being able to reason about a full, continuous flow field are what really make this a great paper.\n\n[A] Li Ding, Chen Feng, DeepMapping: Unsupervised Map Estimation From Multiple Point Clouds, CVPR 2019.\n\n# Post-rebuttal and discussion: \n\nAfter considering the authors' response and the other reviews, I am still positive about this paper and therefore give it a final Rating of \"7: Good paper, accept\". I think the idea of using a neural network parameterization for test-time optimization to solve for non-rigid flow is a very interesting contribution and promising area of research and the paper shows strong results to back this up. The authors have promised to include the missing reference and discussion about DeepMapping [A], as well as clarifying the limitations of the approach.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nThe discussion of the potential negative societal impacts is limited to \"as an AI-based research, it could be misused\" - which I don't feel is satisfactory. Being a method for tracking people, objects and things accurately I think the authors should at least mention immediate ways in which it could be misused for nefarious purposed - e.g. invading privacy by tracking pedestrians across the city, or it's potential use in autonomous weapons and drones. I only mention these as the authors have described many potential positive impacts but I feel the discussion should be more balanced.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This iinteresting and well written paper proposes to use neural network training at runtime (i.e., during inference) to fit a scene flow function g (a plain MLP) that maps 3D points from a point cloud at time t to time t'. The function is formulated per and applied to individual points as an alternative formulation flow regularisation, meaning that the several thousand points in typical self-driving car dataset are sufficient for it to converge. Strikingly, the method is very competitive to supervised methods (that are faster at runtime but need to be trained) as well as to ICP.

After a lengthy discussion between the 4 reviewers and the authors (longer that the paper itself), the consensus was to accept the paper with scores 6, 7, 7, 7, once the specifics of the evaluations (training the supervised methods on 2k points vs. training on 8k points) were elucidated and the authors accepted to do some revisions to the manuscript. I personally commend all reviewers and authors for exemplary thoroughness and collaboration on this review.
Decision: Accept (Spotlight)