ZeroFlow: Scalable Scene Flow via Distillation
Authors: Kyle Vedder
Neehar Peri
Nathaniel Eliot Chodosh
Ishan Khatri
ERIC EATON
Dinesh Jayaraman
Yang Liu
Deva Ramanan
James Hays
Venue: ICLR 2024 poster (2024)
Keywords: Scene Flow
Distillation
Scaling
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose a scalable, human annotation-free distillation pipeline that captures state-of-the-art by leveraging raw data.

Abstract: Scene flow estimation is the task of describing the 3D motion field between temporally successive point clouds. State-of-the-art methods use strong priors and test-time optimization techniques, but require on the order of tens of seconds to process full-size point clouds, making them unusable as computer vision primitives for real-time applications such as open world object detection. Feedforward methods are considerably faster, running on the order of tens to hundreds of milliseconds for full-size point clouds, but require expensive human supervision. To address both limitations, we propose _Scene Flow via Distillation_, a simple, scalable distillation framework that uses a label-free optimization method to produce pseudo-labels to supervise a feedforward model. Our instantiation of this framework, _ZeroFlow_, achieves **state-of-the-art** performance on the _Argoverse 2 Self-Supervised Scene Flow Challenge_ while using zero human labels by simply training on large-scale, diverse unlabeled data. At test-time, ZeroFlow is over 1000$\times$ faster than label-free state-of-the-art optimization-based methods on full-size point clouds (34 FPS vs 0.028 FPS) and over 1000$\times$ cheaper to train on unlabeled data compared to the cost of human annotation (\\$394 vs ~\\$750,000). To facilitate further research, we will release our code, trained model weights, and high quality pseudo-labels for the Argoverse 2 and Waymo Open datasets.

Published as a conference paper at ICLR 2024
ZeroFlow: SCALABLE SCENE FLOW VIA DISTILLATION
Kyle Vedder
Neehar Peri?
Nathaniel Chodosh²
Ishan Khatri
Eric Faton
Dinesh Jayaraman'
Yang Liu
Deva Ramanan²
James Hays Mottonal
Unniversity of Pennsylvania CCarnegie Mellon University
LLawrence Livermore National Laboratory GGorgiaa Tech
ABSTRACT
Scene flow estimation is the task of describing the 3D motion field between tem- porally successive point clouds. State-of-the-ane methods use strong priors and test-time optimization techniques, but require on the order of tens of seconds to pro- cess full-size point clouds, making them unusable computer vision primitives for real-time applications such as open world object detection. Feedforward methods are considerably faster, running on the order of tens to hundreds of milliseconds for full-size point clouds, but require expensive human supervision. To address both limitations, we propose Scene Flow via Distillation, a simple, scalable distillation framework that uses label-free optimization method to produce laculo-labels to supervise a feedforward model. Our instantiation of this framework, ZeroFlow, achieves state-of-the-art performance on the Argoverse Self-Supervised Scene Flow Challenge while using zero human labels by simply training on large-scale, diverse unlabeled data. At test-time, ZeroFlow is over 1000x faster than label-free state-of- of-ttf--rrrrrrr art optimization based methods on full-size point clouds (34 FPS vs 0.028 FPS) and over 1000 cheaper train on unlabeled data compared to the cost of human annotation ($394 vs ~5750,000) To facilitate further research, we release our code, trained model weights, and high quality pseudo-labels for the Argoverse 2 and Waymo Open datasets at https:::// ptttss:/vvedee //vedder.../ 10/ zero f lo
INTRODUCTION
Scene flow estimation is an important primitive for open-world object detection and tracking (Najibi et al. 2022; Zhai et al. 2020 Baur et al. 2021 Huang et al.) 2022 Erçelik et al. 2022 As an example, Najibi al. (2022) generates supervisory boxes for an open- LiDAR detector via offline object extraction using high quality scene flow estimates from Neural Scene Flow Prior (NSFP) (Li et al. 2021b). Although NSFP does not require human supervision, it takes tens of seconds to run on single full-size point cloud pair. If NSFP were both high quality and real-time, its estimations could be directly used as a runtime primitive in the downstream detector instead of relegated to an offline pipeline. This runtime feature formulation is similar to Zhai et al. (2020)'s 's of scene flow from FlowNet31D (Liu et al. 2019) as an input primitive for their multi-object tracking pipeline: although FlowNet33D is fast enough for online processing of subsampled point clouds, its supervised feedforward formulation requires significant in-domain human annotations.
Broadly, these exemplar methods are representative of the strengths and weakness of their class of approach. Supervised feedforward methods use human annotations which are expensive to annotate") To amortize these costs, human annotations are typically done on consecutive observations, severely limiting the structural diversity of the annotated scenes (c.g. 15 second sequence from an Autonomous Vehicle typically only covers single city block); due to costs and labeling difficulty, large-scale labels are also rarely even available outside of Autonomous Vehicle domains. Test-time optimization techniques circumvent the need for human labels by relying on hand-built priors, but they are too slow for online scene flow estimation
Corresponding email: evedderrsens seas. upenn. At ~$0.10 cuboid/ frame, the Argoverse 2 (Wilson al. 2021 train split cost ~5750000 to label; ZeroFlow's 's pscudo-lahels cost $394 at current cloud compute prices. See Supplemental for details.
NSFP (Li al. (222h) takes more than 26 seconds and Chodosh (Chodosh al. 2023 takes more than 35 seconds per point cloud pair on the Argoverse (Wilson al. 2021 train split. See Supplemental for details.

---

Published as a conference paper at ICLR 2024
Time at 10Hz
BooFFow (8.2k points)
0.20
ICP (Full)
Better
0.18
Threeway EPE (m)
0.16
FlowStep3D (8.2k points)
Sim2Real (8.2k points)
0.14
PPWC (8.2k points)
0.12
0.10
ZeroFlow 1x (Full)
Gojeic (20.0k points)
0.08
3l (Full)
NSFP (Full)
0.06
ZeruFlow 3x [Full]
FastFlow3D XL. (Full)
ZeroFlow 5x (Full)
ZeroFlow XL 3x (Full)
Chodosh (Full)
0.04
10a
10a
10
Runtime (ms)
Figure 1: We plot the error and run-time of recent scene flow methods the Argoverse 2 Sensor dataset (Wilson et al.; (021), along with the size of the point cloud prescribed in the method's evaluation protocol. Our method ZeroFlow 3X (ZeroFlow trained on 3x pseudo-labeled data) outperforms its teacher (NSFP, Li et al. (0221b) while running over x000 faster, and ZeroFlow XL 3X (ZeroFlow with larger backbone trained on 3 pseudo-labeled data) achieves state-of-the-art. Methods that use any human labels are plotted with X and zero-label methods are plotted with
We propose Scene Flow via Distillation (SFvD), simple, scalable distillation framework that uses label-free optimization method to produce pscudo-labels to supervise feedforward model. SFvD generates class of scene flow estimation methods that combine the strengths of optimization- based and feedforward methods with the power of data scale and diversity to achieve fast run-time and superior accuracy without human supervision. We instantiate this pipeline into Zero-Label Scalable Scene Flow (ZeroFlow), a family of methods that, motivated by real-world applications, can process full-size point clouds while providing high quality scene flow estimates. We demonstrate the strength of ZeroFlow on Argoverse 2 (Wilson et al. 2021) and Waymo Open (Sun et al. 2020), notably achieving state-of-the-art on the Argoverse Self Self-Supervised Scene Flow Challenge (Figure
Our primary contributions include:
We introduce simple yet effective distillation framework, Scene Flow via Distillation (SFvD), which uses a label-free -ree optimization method to produce pseudo-labels to supervise a feedforward model, allowing us surpass the performance slow optimization-base approaches the speed of feedforward methods.
Using SFvD, we present Zero-Label Scalable Scene Flow (ZeroFlow), family of methods that produce fast, state-of-the-art scene flow on full-size clouds, with methods running over x000 faster than state-of-the-art optimization methods (29.33 ms for ZeroFlow 1X 35,281.4 ms for Chodosh) on real point clouds, while being over 1000x cheaper to train compared to the cost of human annotations (S394 vs ~7750,000)
We release high quality flow pseudo-labels (representing 7.1 GPU months of compute) for popular Argoverse 2 (Wilson et al. 2022)) and Waymo Open (Sun et al. (020) autonomous vehicle datasets, alongside our code and trained model weights, to facilitate further research.
BACKGROUND AND RELATED WORK
Given point clouds P. time t and Pi+1 time t + scene flow estimators predict Fi,-11- 3D vector for each point in P. that describes how it moved from f to + 1 (Dewan et al. 2016). Performance is traditionally measured using the Endpoint Error (EPR) between the predicted flow
2

---

Published as a conference paper at ICLR 2024
Figure 2: Scene Flow vectors describe where the point on an object at time f will end up the object at + 1. In this example, ground truth flow vector A, associated with a point in the upper left concave corner of the object at / has nearby observations to occlusion of the concave feature. The ground truth flow vector B, associated with point on the face of the object at does directly match with any observed point on the object at t + due to observational sparsity. Thus, point matching between and + alone is insufficient to generate ground truth flow.
+ 1 due
1+
Fi,e+1 and ground truth flow Fi,411 (Equation
EPE (Pt)
Σ PE
(1) F
(I)
P
Unlike next token prediction language (Radford et al. 2018 or next frame prediction in vi- sion (Weng al. 2021), future observations do not provide ground truth scene flow (Figure 2). To simply evaluate scene flow estimates, ground truth motion descriptions must be provided by an oracle, typically human annotation of real data (Sun al.)
2020: Wilson al. 2021 the generator of 2023
synthetic datasets (Mayer et al. 2016 Zheng et al.
Recent scene flow estimation methods either train feedforward methods via supervision from human
annotations (Luu et al.] 2019 Behl et al.
[019 Tishchenko et al.) 2020) Kittenplon et al., [021] Wu
al.
2020
Puy al.
2020 Li al.
2021a; Jund et al. [02 Gu al.) 2019 Battrawy al.
2022
Wang ct al. 20222, perform human-designed test-time surrogate objective optimization over
hand designed representations (Pontes al.
2020 Eisenberger et al. 2020 Li et al. 2021b Chodosh
et al.
2023), or learn from superviirn from human human- designed surtogate objectives (Mittal et al.
2020
Baur et al. 2021 Gojeie et al., 2021 Dong et al. 2022 Li et al. 2022).
Supervised feedforward methods are efficient at test-time; however, they require costly human annotations at train-time. Both test-time optimization and self-supervised feedforward methods seek to address this problem by optimizing or learning against label-free surrogate objectives, e.g. Chamfer distance (Pontes et al.) 2020), cycle-consistency (Mittal et al. (2020), and various hand-designed rigidity priors (Dewan et al. 2016 Pontes et al. 2020 Li al. |022] Chodosh et al. [0233 Baur et al.
2021 Gojeic et al. 2021). Sele-supervised methods achieve faster inference by forgoing expensive cest-time optimization, but do not match the quality of optimization- base--aaeddddd methods (Chodosh et al.)
2023) and tend to require human-designed priors via more sophisticated network architectures compared supervised methods (Baur et al., 2021 Gojeic et al. 202T] Kittenplon al. (02T). In
practice, this makes them slower and more difficult to train. In contrast to existing work, we take advantage of the quality of optimization- simplicity of supervised networks. Our approach, ZeroFlow, uses label-free optimization methods (Li al. (202Tb) produce pseudo-labels supervise a feedforward model (Jund et al., (02I). similar to methods used for distillation in other domains Yang et al. (2023).
methods well the efficiency and architectural
3 METHOD
Pscude-lael
Distill
Unlabeled
Pscudo labeled
Slow
Label-Froe Optimization Method
Supervised Feed Model
Figure 3: The Scene Flow via Distillation (SFvD) framework, which describes a new class of scene flow methods that produce high quality, human label-free flow the speed of feedforward networks.
3

---

Published as a conference paper at ICLR 2024
We propose Scene Flow via Distillation (SFvD), simple, scalable distillation framework that creates a new class scene flow estimators by using a label-free optimization method to produce pseudo-labels supervise feedforward model Ff. While conceptually simple, efficiently instantiating SFvD) requires careful construction; most online optimization methods and feedforward architectures are unable to efficiently scale full-size point clouds (Section 3.1)
Based on our scalability analysis, we propose Zero-Label Scalable Scene Flow (ZeroFlow), a family of scene flow models based on SFvD that produces fast, state-of-the-art scene flow estimates for full-size point clouds without any human labels (A (Algorithm [T). ZeroFlow uses Neural Scene Flow prior (NSFP) (Li et al. b021b) generate high quality, label-free pseudo-labels on full-size point clouds (Section 3.2 and FastFlow3D (Jund et al. (021) for efficient inference (Section 3.3
3.1 SCALING SCENE FLOw VIA DISTILLATION TO LARGE POINT CLOUDS
Popular AV datasets including Argoverse (Wilson et al. (021), collected with dual Velodyne VLP-32 sensors) and Waymo Open (Sun et al. (2020), collected with proprietary lidar sensor and subsampled) have full-size point clouds with an average of 52,000 and 79,000 points per frame, respectively, after ground plane removal (Supplemental [ Figure 6). For practical applications, sensors such as the Velodyne VLP-128 in dual return mode produce up to 480,000 points per sweep (Vel 2019) and proprictary sensors at full resolution can produce well over million points per sweep. Thus, scene flow methods must be able to process many points in real-world applications.
Unfortunately, most existing methods focus strictly scene flow quality for toy-sized point clouds, constructed by randomly subsampling full point clouds down to 8,192 points (Jin et al. 2022
Tishchenko al., 2020 Wu et al. 2020; Kittenplon al. 2021 Liu et al. 2019 Li et al. 2021b).
As we are motivated by real-world applications, we instead target scene flow estimation for the full-sized point cloud, making architectural efficiency of paramount importance.
an example of stark differences between feedforward architectures, FastFlow3D (Jund et al.) 2021), which uses a PointPillar-style encoder (Lang et al., 2019), can process I million points in under 100 ms on NVIDIA Tesla P1000 GPÚ (making it real-time for 10Hz LiDAR), while methods like FlowNet3D (Liu et al.
2019) take almost 4 seconds to process the same point cloud.
We design approach to efficiently process full-size point clouds. For SFvD's pseudo-labeling step, speed is less of concern; pseudo-labeling each point cloud pair is offline and highly parallelizable. High-quality methods like Neural Scene Flow Prior (NSFP. Li al. (202 require only modest amount of GPU memory (under 3GB) when estimating scene flow on point clouds with 70K points, enabling fast and low-cost pseudo labeling using cluster of commodity GPUs; pseudo-labeling the Argoverse 2 train split with NSFP is over 1000 cheaper than human annotation (Supplemental The efficiency of SFvD's s student feedforward model critical, as determines both the method's test- test-time speed and its training speed (faster training enables scaling to larger datasets), motivating models that can efficiently process full-size point clouds.
example,
3.2 NEURAL SCENE FLOW PRIOR IS A SLOW TEACHER
Neural Scene Flow Prior (NSFP, Li et al. 2021 is optimization-based approach to scene flow estimation. Notably, it does not use ground truth labels to generate high quality flows, instead relying upon strong priors in its learnable function class (determined by the coordinate network's architecture) and optimization objective (Equattio 2) Point residuals are fit per point cloud pair Pt. P 1 at test-time by randomly initializing two MLPs; one to describe the forward flow ft from Pt to Pi+1, and one to describe the reverse flow F- from Pt + Fr.411 to Pt in order to impose cycle consistency. The forward flow + and backward flow F- optimized jointly to minimize
TruncatedChamfer(2 + F+ Pi+1) + TruncatedChamfer(f + f+ + PT, Pa)
(2)
where TruncatedCammfer is the standard Chamfer distance with per-point distances above 2 meters set to zero to reduce the influence of outliers.
NSFP is able to produce high-quality scene flow estimations due to its choice of coordinate network architecture and use of cycle consistency constraint. The coordinate network's 's learnable function class is expressive enough to fit the low frequency signal of residuals for moving objects while restrictive enough to avoid fitting the high frequency noise from TruncatedChanfer, and the eycle
4

---

Published as a conference paper at ICLR 2024
consistency constraint acts as local smoothness regularizer for the forward flow, as any shattering effects in the forward flow are penalized by the backwards flow. NSFP provides high quality estimates on full-size point clouds (Figure D. select NSFP for ZeroFlow's 's pseudo-label step of SFvD.
3.3 FASTFLOW3D IS A FAST STUDENT
FastFlow3D (Jund al. 2021) is an efficient feedforward method that learns using human supervisory labels F and per-pot point foreground background class labels. FastFlow3D's ' loss minimizes that reduces the importance annotated background
variation of the End-Point Error (Equation D points, thus minimizing
Σ Eo(p) (inD||Rcciii (r,441()) FF.4)
(3) where (p) = [0.1
f1 if E Foreground
(4)
|PP
if Background.
PEr:
FastFlow3ID's architecture is PointPillars-tylee encoder (Lang et al. 2019). traditionally used for efficient LiDAR object detection (Vedder & Eaton 2022). that converts the point cloud into a birds- eye-view pseudoimage using infinitely tall voxels (pillars). This pseudoimage is then processed with layer U-Net style backbone. The encoder of the U-Net processes the P. and P++1 pscudoimage separately, and the decoder jointly processes both pseudoimages. A small MLP is used to decode flow for each point in Pt using the point's coordinate and its associated pseudoimage feature.
As discussed in Section 3.1 FastFlow33Dss s architectural design choices make fast even on full-size point clouds. While most feedforward methods evaluated using standard toy evaluation protocol with subsampled point clouds, FastFlow3D is able to scale up to full resolution point clouds while maintaining real-time performance and emitting competitive quality scene flow estimates using human supervision, making it a good candidate for the distillation step of SFvD.
In order to train FastFlow3D using pseudo-labels, replace the foreground background scaling function (Equation 4) with a simple uniform weighting (a(() = 1), which collapses to Average EPE; see Supplemental for experiments with other weighting schemes. Additionally, we depart from FastFlow problem setup in two minor ways: we delete ground points using dataset provided maps, standard pre-processing step (Chodosh et al. 2023), and use the standard scene flow problem setup of predicting flow between two frames (Section 2 instead of predicting future flow vectors in meters per second. Algorithm describes our approach, with details specified Section 4.1
In order to take advantage of the unlabeled data scaling of SFvD, we expand FastFlow3D to family of models by designing higher capacity backbone, producing FastFlow3D XI. This larger backbone halves the size of each pillar to quadruple the pseudoimage area, doubles the size of the pillar embedding, and adds an additional layer to maintain the network's receptive field in metric space; result, the total parameter count increases from 6.8 million 110 million.
Algorithm 1 ZeroFlow
1: D +- collection of unlabeled point cloud pairs
Training Data Parallel For SFvD Pseudo-label Step
2: for Pt, P E D do
3: Fi,4+1
4: for epoch E epochs do
5:
for Pt, Pt-1, Fi do
SFvD's Distill Step
6:
| Equation StudennFFstFFoow3D, (Pa,) Fit+1)
7:
A + 0 updated w.r.t. I
4 EXPERIMENTS
ZeroFlow provides a family fast, high quality scene flow estimators. order to validate this family and understand the impact of components in the underlying Scene Flow via Distillation framework. we perform extensive experiments on the Argoverse 2 (Wilson et al.) 2021) and Waymo Open (Sun) al. 2020) datasets. We compare author implementations of NSFP (Li et al. (0211) and Chodoshh et al. (2023), implement FastFlow3D (Jund et al., 202T) ourselves (no author implementation is available), and use Chodosh al. 20231's implementations for all other baselines.
5

---

Published as a conference paper at ICLR 2024
As discussed in Chodosh al. (2023), downstream applications typically rely on good quality scene flow estimates for foreground points. Most scene flow methods are evaluated using average Endpoint Error (EPE, Equation D however, roughly 80% of real-world point clouds are background, causing
average EPE to be dominated by background point performance. To address this, we use the improved evaluation metric proposed by Chodosh al. (2023), Threeway EPE:
EPE(p E Pe : E Background)
(Static BG)
Threoway EPE(P:) Avg
EPE E Ps : p E Foreground Fi,++1 (p) S 005m//)
(Static FG)
(5)
EPE(P E Pa : Foreground Fi,44( (p) 0.5m/s)
(Dynamic FG)
4.1 HOw DOES ZEROFLOW PERFORM COMPARED TO PRIOR ART ON REAL POINT CLOUDS?
The overarching promise of ZeroFlow is the ability build fast, high quality scene flow estimators that improve with the the availability of large-scale unlabeled data. Does ZeroFlow deliver on this promise? How does it compare state-of-the-ar methods?
To characterize the ZeroFlow family's performance, we use Argoverse 2 to perform scaling exper- iments along two axes: dataset size and student size. For our standard size configuration, we use the Argoverse 2 Sensor train split and the standard FastFlow3D architecture, enabling head-to-head comparisons against the fully supervised FastFlow3D well as other baseline methods. For our scaled up dataset (denoted 3X in all experiments), we use the Argoverse 2 Sensor train split and concatenate roughly twice as large set of unannotated frame pairs from the Argoverse 2 LiDAR dataset, uniformly sampled from its 20,000 sequences to maximize data diversity. For our scaled up student architecture (denoted XI. in all experiments), we use the XI. backbone described in Section 33. For details on the exact dataset construction and method hyperparameters, see Supplemental A
Table 1: Quantitative results on the Argoverse Sensor validation split using the evaluation protocol from Chodosh et al.) (202)). The methods used in this paper, shown in the first two blocks of the table, are trained and evaluated on point clouds within a 102.4m x 102.4m area centered around the ego vehicle (the settings for the Argoverse 2 Self-Supervised Scene Flow Challenge) However, following the protocol of Chodosh et al. (2023). all methods report error on points in the 70m x 70m area centered around the ego vehicle. Runtimes are collected on an NVIDIA V100 with a batch size of (Peri et al.) 2023). FastFlow3D. ZeroFlow IX, and ZeroFlow 3X have identical feedforward architectures and thus share the same real-time runtime; FastFlow3D XL, ZeroFlow XL 1X, and ZeroFlow XI. 3X have identical feedforward architectures and thus share the same runtime. Methods with an have performance averaged over 3 training runs (see Supplemental C for details). Underlined methods require human supervision.
Runtime (ms)
Point Claud
Threeway EPE
Dynamic HPB
Starie PG
Static EPE
Subsampled
FastFloon 3D* Jund at 2021
Full Point Cloud
0.071
0,186
0.021
0.006
ZeroFlow 1X* (Oars)
29.33
2.38
Pull Point Cloud
0.088
0:231
0.022
0.011
ZeroFlow 3X (Ours)
Pall Point Cloud
0.064
0,164
0.017
0.011
ZeroFlow 5X (Ouns)
Full Point Cloud
0,056
0,140
0.017
0.011
FasFoom 3D XL.
Hull Point Cloud
0.055
0.139
0.018
0.007
ZeroFlow XI. IX (Oun)
260.61 +
1.21
Hull Point Cloud
0.070
0.17%
0.019
0.013
ZeroFlow XL. 3X (Ouns
Full Point Cloud
0.054
0.131
0.018
0.012
NSFP w/ Motion Comm ol 20210
26, 11.225...
18, 139.3
Full Point Cloud
0.067
0.131
0.036
0.034
Chodosh et al. Cholostrert 2025)
35, 242
20, 247.7
Full Point Cloud
0.055
0.129
0.028
0.008
OMoretry
Hull Point Cloud
0.198
0.583
0.000
0.001
ICP Cign & Mediomn
523.11 d::
169.34
Full Point Cloud
0.204
0.557
0.025
0.028
Gojete (oofee of
6, 087.872
690.56 56
20000
0.083
0.155
0.061
0.032
Sim2Rent
Cirrer Den
2015
19.
13.88
8192
0.157
0.229
0.106
0.137
FaoHlow (i m
2020
2, 116.34=
282
81992
0.200
0.447
0.079
0.050
PPWC. aW 20en
79.432
2.20
8199
0.1.0
0.168
0.092
0.129
Flow Stepste (Kiieppph
2021
687.54.
3.13
8192
0.161
0,173
0.132
0.176
As shown in Table T ZeroFlow able to leverage scale to deliver superior performance. While ZeroFlow IX loses head-to-head competition against the human-supervise FastFlow3D on both Argoverse 2 (Table [] and Waymo Open (Table 2). scaling the distillation process to additional unlabeled data provided by Argoverse enables ZeroFlow 3X to significantly surpass the performance of both methods just by training on more pseudo-labled data. ZeroFlow 3X even surpasses the performance of its own teacher, NSFP, while running in real-time!
ZeroFlow's 's pipeline also benefits from scaling up the student architecture. We modify ZeroFlow's architecture with the much larger XI. backbone, and show that our ZeroFlow XI. 3X is able to

---

Published as a conference paper at ICLR 2024
combine the power of dataset and model scale to outperform all other methods, including significantly outperform its own teacher. Our simple approach achieves state-of-the-at on both the Argoverse validation split and Argoverse Self-Supervised Scene Flow Challenge.
Table 2: Quantitative results Waymo Open using the evaluation protocol from Chodosh al. (2023). Runtimes are scaled to approximate the performance on a V100 (Li et al. (020). Both FastFlow3D and ZeroFlow 1X have identical feedforward architectures and thus share the same runtime. Underlined methods require human supervision.
Runtime (ms)
Point Clond Subsampled Size
Threeway EPE
Dynamic FG EPE
Static FG
Static BG EPE
ZeroFlow LX
3D ID (und et [021
21.66d
0.48
Full Point Cloud
Point Cloud
0.092 0.078
0.216 0.195
0.015 0.015
0.045
0.024
Chodesh (C
93,
76,
NSF A0100
157.81
0.555
54.63
20000)
EgoFlow 1
2020
893. 68
86.55
8192
0.183
0.390
0.069
0.089
Sim2Rel Thneee PUU
72.841
14.79
8192
0.166
0.198
0.099
0.201
PPWC (Wagrnia #050
101 43++
5.48
8192
0.132
0.180
0.075
0.142
FlowSt<p21> (Klnccpioh et 202]
872.02-t
6.24
8192
0.169
0.152
0,123
0.232
4.2 How DOES ZEROFLOW SCALE?
Section demonstrates that ZeroFlow can leverage scale to capture state-of-the-art performance. However, it's difficult to perform extensive model tuning for large training runs, so predictable estimates of performance as a function of dataset size are critical (OpenAl 2023). Does ZeroFlow's s performance follow predictable scaling laws?
We train ZeroFlow and FastFlow3D on sequence subsets/ supersets of the Argoverse 2 Sensor train split. Figure shows ZeroFlow and FastFlow3DDss validation Threeway EPE both decrease roughly logarithmically, and this trend appears to hold for XL backbone models well.
Data without Human Labels
0.20
0.18
Threeway EPE (m)
0.16
Better
0.14
0.12
0.10
Ours
0.08
Ours XL
FastFLow3D
0.06
Fast Flow3D XL
1%
10%
20%
50%
100%
200% 300%
500%
Dataset Size as Percentage AV2 Sensor Train Split
Figure 4: Empirical scaling laws for ZeroFlow. We report Argoverse validation split Threeway EPE percentage of the Argoverse 2 train split used, log1o-log10 scale, trained to convergence. Threeway EPE performance of ZeroFlow scales logarithmically with the amount training data.
Empirically, ZeroFlow adheres predictable scaling laws that demonstrate more data (and more parameters) are all you need to get better performance. This makes ZeroFlow practical pipeline for building seene flow foundation models (Bommasani et al., (021) using the raw point cloud data that exists today in the deployment logs of Autonomous Vehicles and other deployed systems.
4.3 HOW DOES DATASET DIVERSITY INFLUENCE ZEROFLOW'S PERFORMANCE?
In typical human annotation setups, a point cloud sequence is given to the human annotator. The human generates box annotations in the first frame, and then updates the pose of those boxes the objects move through the sequence, introducing and removing annotations as needed. This process is much more efficient than annotating disjoint frame pairs, as it amortizes the time spent annotating
7

---

Published as a conference paper at ICLR 2024
most objects in the sequence. This is why most human annotated training datasets (e.g. Argoverse Sensor, Waymo Open) are composed of contiguous sequences. However, contiguous frames have significant structural similarity; in the 150 frames (15 seconds) of Argoverse 2 Sensor sequence, the vehicle typically observes no more than city block's worth of unique structure. ZeroFlow, which requires zero human labels, does not have this constraint its pseudo-labels; NSFP on non-sequential frames is no more expensive than NSFP run on non-sequential frames, enabling ZeroFlow to train more diverse dataset. How does dataset diversity impact performance?
To understand the impact of data diversity, we train a version of ZeroFlow IX and ZeroFlow 2X only on the diverse subset of our Argoverse 2 LiDAR data selected by uniformly sampling 12 frame pairs from each of the 20,000 unique sequences (Table 3).
Table 3: Comparison between ZeroFlow trained on Argoverse Sensor dataset versus the more diverse, unlabeled Argoverse LiDAR subset described in Section 4.1 result non-trin non-trivial performance improvements.
Diverse training datasets
Threeway EPE
Dynamic FG EPE
Static FG EPB
Static BG EPE
FastFlaw.3D [Jund et al.
2021
0.071
0.186
0.021
0.006
ZeroPlow (AVZ Sensor DMa
0.088
0.231
0.022
ZerolPlow (AV2 LIDAR Subset Data)
0.082
0.218
0.018
0.009
ZeroFlow 2X (AV2 LIDAR Subset Data)
0.072
0.184
0.022
0.011
Dataset diversity has non-trivial impact on performance; ZeroFlow, by virtue of being able to learn across non-contiguous frame pairs, is able to see more unique scene structure and thus learn to better to extract motion the presence of the unique geometries the real world.
4.4 How DO THE NOISE CHARACTERISTICS ZEROFLOW COMPARE TO OTHER METHODS?
ZeroFlow distills NSFP into feedforward model from the FastFlow3D family. Section 4.1 highlights the average performance of ZeroFlow across Threeway EPE catagories, but what does the error distribution look like?
Fad
(4) ZeoFlw
(8 Chedoek, Log
(n) FaueFoowhh,
Uhi ZeniFlow
ZaneFoo Abs
@1 Zemillow XL.
NSFR,
C Chodoei Abe
Figure 5: Normalized frame birds-eye-view heatmaps of endpoint residuals for Chamfer Distance, as well as the outputs for NSFP and Chodosh on moving points (points with ground truth speed above 0.5m/s). Perfect predictions would produce single central dot. Top row shows the frequency on logıo color scale, bottom row shows the frequency on an absolute color scale. Qualitatively, methods with better quantitative results have tighter residual distributions. See Supplemental for details.
To answer this question, we plot birds-eye-view flow vector residuals of NSFP, Chodosh, FastFlow3D, and several members of the ZeroFlow family moving objects from the Argoverse validation dataset, where the ground truth rotated vertically and centered at the origin to present all vectors in the same frame (Figure 5 Supplemental F for details construction). Qualitatively, these plots show that error is mostly distributed along the camera ray and distributional tightness (logıo plots) roughly corresponds to overall method performance.
8

---

Published as a conference paper at ICLR 2024
Overall, these plots provide useful insights to practitioners and researchers, particularly for consump- tion downstream tasks; as an example, open world object extraction (Najibi et al. (022) requires the ability to threshold for motion and cluster motion vectors together to extract the entire object. Decreased average EPE is useful for this task, but understanding the magnitude and distribution of flow vectors is needed to craft good extraction heuristics.
4.5 HOw DOES TEACHER QUALITY IMPACT ZEROFLOW'S PERFORMANCE?
As shown in Section 4. (Chodosh et al. 2023) has superior Threeway EPE over NSFP on both Argoverse 2 and Waymo Open. Can better performing teacher lead better version of ZeroFlow?
To understand the impact of a better teacher, we train ZeroFlow on Argovense 2 using superior quality flow vectors from from Chodosh al. (2023). which proposes a refinement step to NSFP lablels to provide improvements to flow vector quality (Table 4. ZeroFlow trained on Chodosh refined pseudo-labels provides no meaningful quality improvement over NSFP pseudo-labels (as discussed in Supplemental C the Threeway EPE difference is within training variance for ZeroFlow). These results also hold for our ablated speed scaled version of ZeroFlow in Supplemental B
Since increasing the quality of the teacher over NSFP provides no noticeable benefit, can we get away with using a significantly faster but lower quality teacher to replace NSFP, c.g. the commonly used self-supervised proxy of TruncatedChamfer?
To understand if NSFP necessary, we train ZeroFlow on Argoverse using pseudo-labels from the nearest neighbor, truncated 2 meters as with TruncatedChamfer. The residual distribution of TruncatedChamfer is shown in Supplemental [F Figure [Oa ZeroFlow trained on TruncatedChamin pseudo-labels performs significantly worse than NSFP, motivating the use of NSFP teacher.
Table 4: Comparison between ZeroFlow trained on Argoverse 2 using NSFP pseudo-labels, ZeroFlow using Chodosh et al. (2023) pseudo-labels, and ZeroFlow using TruncateCCammfer. Methods with an have performance averaged over 3 training runs (see Supplemental C for details). The minor quality improvement of Chodosh pseudo-labels does not lead meaningful difference in performance, while the significant degradation of TruncatedChamfer leads to significantly worse performance.
Threeway EPE
Dynamic FG EPE
Static FG EPE
Static BG EPE
ZeroHlow (NSPP tscudo-labels)*
0.088
0.022 0.018
ZenoFlow LX Chodushh et al, 2023 pseudo labels)
0.085
0.234
0.004
Zemflow (mineateo nanter (euuo-lahels))
0.105
0.226
0.049
0.040
5 CONCLUSION
Our scene flow approach, Zero-Label Scalable Scene Flow (ZeroFlow), produces fast, state-of-the-art scene flow without human labels via our conceptually simple distillation pipeline.
But, more importantly, we present the first practical pipeline for building scene flow foundation models (Bommasani et al. [)2TT) using the raw point cloud data that exists today in the deployment logs at Autonomous Vchicle companies and other deployed robotics systems. Foundational models in other domains like language (Brown et al. 2020; OpenAI 2023) and vision (Kirillov et al. 2023
Rajeswaran et al., 2022) have enabled significant system capabilities with little or no additional domain-specific fine-tuning (Wang et al. 2023 Ma et al. 2022 2023). We posit that a scene flow foundational model will enable new systems that can leverage high quality, general scene flow estimates robustly reason about object dynamics even in foreign or noisy environments.
Limitations and Future Work. ZeroFlow inherits the biases of its pseudo-labels. Unsurprisingly, the pseudo-labels consistently fail to estimate scene flow for certian objects, our method will also be unable to predict scene flow for those objects; however, further innovation in model architecture, loss functions, and pseudo-labels may yield better performance. In order to enable further work on Scene Flow via Distillation- based methods, we releasd")on our code, trained model weights, and NSFP flow pseudo-labele representing 3.6 GPU months for Argoverse 2 and 3.5 GPU months for Waymo Open.
hhttps://vedaei 10/zeroflow

---

Published as a conference paper at ICLR 2024
Acknowledgements. The research presented in this paper was partially supported by the DARPA SAIL-ON program under contract HR001 1000112000 12C00400, the DARPA ShELL program under agreement HR001 12100133,, the Army Research Office under MURI grant W91 INF20-1-00000 and the CMU Center for Autonomous Vehicle Research. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52- 07NA27344.
REFERENCES
Ramy Battrawy, René Schuster, Mohammad Ali Nikouci Mahani, and Didier Stricker. RMS-FlowNet: Efficient and Robust Multi-Scale Scene Flow Estimation for Large-Scale Point Clouds. In Int. Conf. Rob, Aut., pp. 883-889. IEEE, 2022.
Stefan Andreas Baur, David Josef Emmerichs, Frank Moosmann, Peter Pinggera, Bjöm Ommer, and Andreas Geiger. SLIM: Self-supervised LiDAR scene flow and motion segmentation. In Int. Conf. Comput. Vix., pp. 13126-13136, 2021.
Aseem Behl, Despoina Paschalidou, Simon Donné, and Andreas Geiger. Pointflownet: Learning representations for rigid motion estimation from point clouds. In Int. Conf. Comput. Vis., pp. 7962-7971, 2019.
Michael Black. Novelty in science: A guide to reviewers. stzps://medium.co @black 51
2022.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ernon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong. Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R' e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Parbert-Voss.. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
Yang Chen and Gérard Medioni. Object modelling by registration of multiple range images. Img. Vis. Comput., 10(3))145-1555 1992.
Nathaniel Chodosh, Deva Ramanan, and Simon Lucey. Re-Evaluating LiDAR Scene Flow for Autonomous Driving, arXiv preprint, 2023.
Ayush Dewan, Tim Caselitz, Gian Diego Tipaldi, and Wolfram Burgard. Rigid scene flow for 3d lidar scans. In Int. Conf. Intel. Rob. Sys., pp. 1765-1770. IEEE, 2016.
10

---

Published as a conference paper at ICLR 2024
Guanting Dong, Yueyi Zhang, Hanlin Li, Xiaoyan Sun, and Zhiwei Xiong. Exploiting Rigidity Constraints for LiDAR Scene Flow Estimation. In JEEE Conf. Comput. Vis. Pattern Recog., pP. 12776-12785, 2022.
Marvin Eisenberger, Zorah Lahner, and Daniel Cremers. Smooth shells: Multi-scale shape registration with functional maps. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 12265-12274, 2020.
Emeç Ergelik, Ekim Yurtsever, Mingyu Liu, Zhijie Yang, Hanzhen Zhang, Pinar Topçam, Maximilian Listl, Yilmaz Kaan Çayh, and Alois Knoll. 3D Object Detection with Self-supervised Lidar Scene Flow Backbone. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision - ECCV 2022, pp. 247-265, Cham, 2022. Springer Nature Switzerland.
Zan Gojeic, Or Litany, Andreas Wieser, Leonidas J Guibas, and Tolga Birdal. Weakly supervised learning of rigid 3d scene flow. In IEEE Conf. Comput. Vis. Pattern Recog.. pp. 5692-5703, 2021.
Xiuye Gu, Yijie Wang, Chongruo Wu, Yong Jae Lee, and Panqu Wang. Hplflownet: Hierarchical permutohedral lattice flownet for scene flow estimation on large-scale point clouds. In IEEE Conf. Comput. Vis. Puttern Recog. pp. 3254-3263, 2019.
Shengyu Huang, Zan Gojeic, Jiahui Huang, and Konrad Schindler Andreas Wieser. Dynamic 3D Scene Analysis by Point Cloud Accumulation. In European Conference on Computer Vision, ECCV, 2022.
Zhao Jin, Yinjie Lei, Naveed Akhtar, Haifeng Li, and Munawar Hayat. Deformation and Correspon- dence Aware Unsupervised Synthetic-to- Real Scene Flow Estimation for Point Clouds. In IEEE Conf. Compur. Vis. Pattern Recog.. pp. 7233-7243, 2022.
Philipp Jund, Chris Sweeney, Nichola Abdo, Zhifeng Chen, and Jonathon Shlens. Scalable Scene Flow From Point Clouds in the Real World. IEEE Robotics and Automation Letters, 12 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xix:1412.6990, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything, arXiv:2304.02643 2023.
Yair Kittenplon, Yonina C Eldar, and Dan Raviv. Flowstep3d: Model unrolling for self-supervised scene flow estimation. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 41 14-4123, 2021.
Alex Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. PointPillars: Fast Encoders for Object Detection From Point Clouds. In Proceedings of the 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 12689-12697, 2019.
Mengtian Li, Yu-Xiong Wang, and Deva Ramanan. Towards streaming perception. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 473-488. Springer, 2020.
Ruibo Li, Guosheng Lin, Tong He, Fayao Liu, and Chunhua Shen. HCRF-Flow: Scene flow from point clouds with continuous high-order CRFs and position-aware flow embedding. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 364-373, 2021a.
Ruibo Li, Chi Zhang, Guosheng Lin, Zhe Wang, and Chunhua Shen. RigidFlow: Self- Supervised Scene Flow Learning on Point Clouds by Local Rigidity Prior. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 16959-16968. 2022.
Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey. Neural Scene Flow Prior. Advances in Neural Information P Processing Systems, 34, 2021b.
Xingyu Liu, Charles R Qi, and Leonidas J Guibas. FlowNet3D: Learning Scene Flow in 3D Point Clouds. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
11

---

Published as a conference paper at ICLR 2024
Nikola Lopac, Irena Jurdana, Adrian Brnelić, and Tomislav Krljan. Application of Laser Systems for Detection and Ranging in the Modern Road Transportation and Maritime Sector. Sensors, 22(16), 2022. ISSN 1424-8220.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards Universal Visual Reward and Representation via Impue-Implicit Pre-Training. arXiv preprint arXiv:2210.0003: 2022.
Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang. Osbert Bastani, and Dinesh Jayaraman. LIV: Language-Imag Representations and Rewards for Robotic Control. arXiv preprint 2rvi: 00958, 2023.
N. Mayer, E. Ilg. P. Häusser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Himangi Mittal, Brian Okorn, and David Held. Just Go With the Flow: Self-Supervised Scene Flow Estimation. In IEEE Conf. Comput. Vis. Pattern Recog., June 2020.
Mahyar Najibi, Jingwei Ji, Yin Zhou, Charles R. Qi, Xinchen Yan, Scott Ettinger, and Dragomir Anguelov. Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving. European Conference on Computer Vision (ECCV), 2022.
OpenAL. Gpt-4 technical report, 2023.
Neehar Peri, Mengtian Li, Benjamin Wilson, Yu-Xiong Wang, James Hays, and Deva Ramanan. An empirical analysis of range for 3d object detection. arXiv preprint arXiv:2308. 04054, 2023.
Jhony Kaesemodel Pontes, James Hays, and Simon Lucey. Scene flow from point clouds with or without learning. In Int. Conf. 3D Vis., pp. 261-270. IEÉE, 2020.
Gilles Puy, Alexandre Boulch, and Renaud Marlet. Flot: Scene flow on point clouds guided by optimal transport. In Eur. Conf. Comput. Vis., pp. 527-544. Springer, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018
Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A Universal Visual Representation for Robot Manipulation. Conference on Robot Learning (CoRL) 2022, 03 2022.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in Perception for Autonomous Driving: Waymo Open Dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Ivan Tishchenko, Sandro Lombardi. Martin R Oswald, and Marc Pollefeys. Self-supervised learning of non-rigid residual flow and ego-motion. In Int. Conf. 3D Vis., pp. 150-159. IEEE, 2020.
Kyle Vedder and Eric Eaton. Sparse PointPillars: Maintaining and Exploiting Input Sparsity to Improve Runtime Embedded Systems. International Conference on Intelligent Robots and Systems (IROS), 2022.
Velodyne Lidar Alpha Prime. Velodyne Lidar, 11 2019.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv preprint arXiv: Arxiv-2305.1629 2023.
Jun Wang, Xiaolong Li, Alan Sullivan, Lynn Abbott, and Siheng Chen. PointMotionNet. Point-Wii Motion Learning for Large-Socr Scale LiDAR Point Clouds Sequences. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4418-4427 2022.
12

---

Published as a conference paper at ICLR 2024
Xinshuo Weng, Jianren Wang, Sergey Levine, Kris Kitani, and Nicholas Rhinehart. Inverting the pose forecasting pipeline with spf2: Sequential pointcloud forecasting for sequential pose forecasting. In Conference robot learning. pp. 11-20. PMLR, 2021.
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Car,, and James Hays. Argoverse 2: Next Generation Datasets for Self-driving Perception and Forecasting. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 202 1), 2021.
Wenxuan Wu, Zhi Yuan Wang, Zhuwen Li, Wei Liu, and Fuxin. Pointpwe-net: Cost volume on point clouds for (self-) supervised scene flow estimation. In Eur. Conf. Comput. Vis., pp. 88-107. Springer, 2020.
Bo Yang, Ishan Khatri, Michael Happold, and Chulong Chen. Adcnet: Learning from raw radar data via distillation, 2023.
Guangyao Zhai, Xin Kong, Jinhao Cui, Yong Liu, and Zhen Yang. FlowMOT: 3D Multi-Object Tracking by Scene Flow Association. ArXiv, abs/2012.07541, 2020.
Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J. Guibas. PointOdyssey: Large-Scale Synthetic Dataset for Long-Tere Teoe-Term-ae Point Tracking. In ICCV, 2023.
13


Reviews:
[{"review_id": "vJtnLSM9Bv", "reviewer": "Reviewer_q16v", "summary": "The paper introduces a scalable point-cloud-based scene flow approach via distillation. Given unlabeled raw point cloud data, the paper first calculates pseudo-GT scene flow using a test-time optimization-based method (Neural Scene Flow Prior), which tends to demonstrate much better accuracy than feed-forward approaches. Then given the pseudo GT, the method trains a feed-forward model in a supervised manner. With this distillation pipeline, their method archives comparable/sometimes even better accuracy than the optimization-based method and of course much better than other feed-forward methods, while maintaining real-time performance. This pipeline is scalable as what it needs is just unlabeled point cloud data.", "questions": "(just a remark) By the way, there is a recent paper [a] that significantly improves the runtime performance of NSFP by 10 or 30 times (varying different datasets). The proposed approach can reduce the compute cost for pseudo GT generation by using this faster method. \n\n[a] Fast Neural Scene Flow, ICCV 2023", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Good performance (Table 1)\n\n   The method achieves better accuracy over previous work while maintaining real-time performance. This source of gain requires the cost and time of generating pseudo-GT using the existing optimization-based approach. However, this can be done offline in a parallel way, so it's not a critical concern.\n\n+ Simple, effective idea\n\n   This distillation idea is very simple, but previous work on point-cloud-based scene flow methods hasn't tried the idea yet. The paper demonstrates that this simple idea is working and demonstrates that it's scalable by using large-scale diverse unlabeled data. It would be great if the computed pseudo-GT would be released to the community.\n\n+ In-depth analysis\n\n  Not only the main experiments, the paper provides in-depth analysis, such as accuracy versus dataset size (Fig. 4), endpoint residual maps (Fig. 5), and variance study (Table 7 and 8). This clearly helps a better understanding of the paper. The paper is pretty much self-contained.", "weaknesses": "There seem to be not many weaknesses in the paper but some minor questions related to the accuracy.\n\n- In Table 1, how can ZeroFlow XL 3X (Ours) outperform the NSFP w/ Motion Comp baseline as ZeroFlow's accuracy can be bounded by the accuracy of pseudo GT (i.e., NSFP)? Would it be also possible to include the accuracy of pseudo GT in Table 1?\n\n- In Fig. 4, why is FastFlow3D better than 'Ours' when the dataset size is less than 100%? I thought ZeroFlow's backbone was based on FastFlow3D, but what change made it underperform FastFlow3D?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zSMBZu6AaK", "reviewer": "Reviewer_E12X", "summary": "The paper proposes a new method called ZeroFlow that combines the strengths of optimization-based and feed-forward methods to achieve efficient and label-free scene flow estimation.\nIt introduces an optimization-based teacher method, which generates pseudo labels, and uses these labels to supervise a feed-forward student model. \nIt conducts extensive quantitative evaluations and achieves comparable performance to NSFP and FastFlow3D.", "questions": "Please refer to the Weaknesses", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The overall writing of the paper is good, with a clear description of the motivation and implementation of the proposed approach.\n- As can be seen from Table 1, the proposed method achieves a good balance between performance and efficiency.\n- The paper verifies the proposed ideas through a series of experiments.", "weaknesses": "- The innovation and contribution of the paper mainly lie in the new application of the distillation strategy, and I suggest the authors provide further analysis or insights into the scene flow estimation task.\nI think the achieved performance of the proposed approach is highly dependent on the quality and effectiveness of the existing works.\nThe authors combine two previously published works, using NSFP to generate pseudo labels and training the FastFlow3D model using these labels.\n\n- In Sections 3.2 and 3.3, the author provides a detailed description to help understand the proposed method, but it is difficult for me to detect the author's new in-depth thinking about this work from these descriptions.\nAt the same time, Section 3 contains many statements on existing work, such as 3.1. This section might benefit from providing a more in-depth analysis of combining the strengths of optimization-based and feed-forward methods.\n\n- The quantitative results in Table 2 on Waymo Open are less competitive, making the method's performance on this dataset not very convincing.\n\n- The overall style of the article is more like a technical report than a research paper, and I feel that it contains more engineering skills.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MHXAwwCTlU", "reviewer": "Reviewer_pGhB", "summary": "This paper focuses on tackling the practical challenge of real-world scale scene flow. The method employed in this study can be summarized as follows: 1) a slow teacher NSFP is identified; 2) a fast student FastFlow3D is recognized; 3) the teacher is adopted to create auto-labels for supervising the student.", "questions": "1. For the full point cloud setting, what is the exact number of points in average? Have we applied ground point removal so the number of points has been significantly reduced. I know there is a discussion in 3.1, but I want to confirm the actual number of points and settings in the later experiments.\n2. Algorithm 1 can be removed; it does not provide any useful information.\n3. (This is not important) I feel we should not call this scene flow foundation model; scene flow is only a single problem in Computer Vision/Perception.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "This paper carries out a thorough series of experiments to validate the efficacy of the straightforward autolabel concepts and raises several intriguing questions. Furthermore, the paper includes experiments designed to address these questions, potentially offering valuable insights to the autonomous vehicle industry.", "weaknesses": "The idea lacks novelty, particularly for consideration at ICLR. In my opinion, this paper would be better suited for ICRA or WACV, as they tend to focus on more application and system-oriented research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)