Neural Scene Flow Prior

1 Introduction . State-of-the-art results have recently been achieved by learning-based models [ 17 , 30 , 47 , 60 , 68 ] for the scene flow problem—the task of estimating 3D motion fields from dynamic scenes . However , such models heavily rely on large-scale data to capture prior knowledge , which is not always readily available . Scene flow annotations are expensive , and most methods train on synthetic and unrealistic scenarios to fine-tune on small real datasets . Poor generalization to unseen , out-of-the-distribution inputs is another problem . Prior information is generally limited to the statistics of the data used for training . Real-world applications such as autonomous driving require robust solutions to low-level vision tasks such as depth , optical , and scene flow estimation that work in statistically different scenarios . Inspired by recent innovations that make use of coordinate-based networks ( i.e. , pixels or 3D positions as inputs ) [ 9 , 36 , 37 , 39 , 56 ] for 3D modeling and rendering , we investigate the use of such networks to regularize the scene flow problem without any learning directly from point clouds . Optimization happens at runtime , and instead of learning a prior from data , the network structure itself captures the prior information . It is not limited to the statistics of a specific dataset . ∗Research done during internship at Argo AI . Corresponding e-mail : xueqian.li @ adelaide.edu.au . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Optimizing neural networks at execution time is not new . Ulyanov et al . [ 63 ] showed that a randomly initialized convolutional network could be used as a handcrafted prior for standard inverse problems such as image denoising , super-resolution , and inpainting . Ding and Feng [ 12 ] proposed a runtime optimization method ( DeepMapping ) for rigid pose estimation using deep neural networks . Although such deep image priors , deep mapping , and coordinate-based networks for neural scene representations have been successfully applied for inverse problems , rendering , and rigid registration , none has yet investigated ( to the best of our knowledge ) the use of network-based priors for regularizing scene flow directly from point clouds . Our proposed neural prior is based on a simple multilayer perceptron ( MLP ) architecture , and we show it is powerful enough to regularize scene flow given two point clouds implicitly . The input to the network is 3D points , and the output is a regularized scene flow . Our neural prior allows for a continuous scene flow representation instead of discrete such as in graph Laplacian-based priors , e.g. , [ 45 ] . We show how the flow fields captured by our neural prior can be employed to estimate long-term correspondences across a sequence of point clouds . The continuous scene flow allows for better integration of motions across time . Our results are promising and competitive to supervised [ 30 ] , self-supervised [ 38 , 68 ] , and nonlearning methods [ 1 , 45 ] ( see Table 1 ) . Our method also scales to real-world point clouds with tens of thousands of points while achieving better accuracy and time complexity than recent runtime optimization methods ( see Fig . 1 ) . 2 Related work . Non-learning-based scene flow Scene flow is the uplift from the optical flow , which is proposed by Vedula et al . [ 64 ] as the non-rigid motion field in the 3D space . The authors proposed the optimization-based scene flow estimation using image sequences to infer reconstruction knowledge of the flow in 3D surfaces . Successive RGB/RGB-D image-based work [ 4 , 18–21 , 27 , 43 , 44 , 50 ] used probability-based estimation , coarse-to-fine techniques , 6-DoF parameterization , or object segmentation , etc. , to improve accuracy and computation time . Although image-based scene flow methods are widely used , direct estimation of the scene flow from the point cloud is still possible through non-rigid registration methods , such as [ 1 , 10 , 26 , 42 ] . In this paper , we focus on point cloud-based scene flow estimation . Learning-based scene flow Image-based learning methods [ 6 , 51 , 54 , 60 , 70 ] use convolution and data supervision to solve scene flow from monocular or RGB-D images with the available depth information . Other image-based methods [ 22 , 23 , 32 , 52 , 53 ] take care of extra occlusion cues in the large-scale autonomous driving scenes . Point-based learning methods have become more prevalent with the rapid development of point cloud feature learning [ 28 , 48 , 49 , 65 , 67 ] . FlowNet3D [ 30 ] is a seminal work that estimates scene flow using PointNet++ [ 49 ] . Successive work [ 17 , 31 , 47 , 66 ] extends point-based learning methods using different feature extraction techniques . One obvious drawback of these supervised learning methods is the demand for sufficient ground truth labels . Besides , supervised methods lack generalizability while eventually only fitting domain-specific data . Self-supervised methods [ 25 , 38 , 61 , 68 ] , on the other hand , replaced the loss between the prediction and the ground truth flow with a point distance loss to use the point cloud itself as supervision . Self-supervision can adapt to different datasets and maintain certain generalizability . Nonetheless , massive training data are still required for sufficient learning . The graph Laplacian method Graph Laplacian [ 2 ] is widely used to smooth the surface in mesh processing [ 5,14,57,58 ] , point cloud denoising [ 11,71 ] , etc . Here we talk about the recent scene flow estimation using Graph Laplacian [ 45 ] . The method explicitly constructed a graph of the point cloud to constrain the non-rigid scene flow as rigid within a specific range . While as a dataless runtime optimization , the method is heavily affected by the hyperparameters of the graph and loses scalability when the point cloud becomes larger or the neighbors in the graph grow . Deep neural prior , implicit functions , and neural rendering Although large-scale data helps in feature representation [ 59 ] , end-to-end learning still requires high computation capacity and readily available training datasets . Instead , Ulyanov et al . [ 63 ] proposed a new style of optimization that uses a convolutional neural network to infer prior knowledge from the network architecture . One broader interest is to extend the idea of the network being an image function to the 3D shape modeling , and implicitly represent the continuous shape as level sets of neural networks . By directly mapping the 3D input to binary occupancy nets [ 9 , 36 ] , or signed distance functions [ 3 , 39 , 55 ] , it is powerful to model the 3D geometries in a continuous space using the coordinate-based network . The following Scene Representation Networks [ 56 ] takes advantage of the coordinate-based network and renders view synthetic images . Mildenhall et al . proposed a seminal work NeRF [ 37 ] , which is a novel way to do neural volume rendering using both point positions and viewing directions based on the coordinate-based network . Dynamic scene synthesis work [ 13 , 16 , 29 , 40 , 46 , 62 , 69 ] follows the NeRF framework , and integrates the motions to generate dynamic scenes . Some of the work [ 16 , 29 ] use scene flow to further constrain or segment dynamic scenes . An interesting work that solves for the rigid alignment between point clouds using runtime optimization is DeepMapping [ 12 ] . However , this work only deals with rigid motion , and the network architecture is more complex than coordinate-based networks . In this work , we are interested in coordinate-based networks to address the large-scale , real-world scene flow problem . 3 Approach . Problem definition Let S1 and S2 be two 3D point clouds sampled from a dynamic scene at time t-1 and t. The number of points in each point cloud , |S1| and |S2| , are typically different and not in correspondence . A 3D point p ∈ S1 moving from time t-1 to time t can be modeled by a translational vector ( or flow vector ) f ∈ R3 , where p′ = p + f . The collection of flow vectors for all 3D points is the scene flow F = { fi } |S1|i=1 . Optimization We want to optimize for a scene flow F that minimizes the distance between the two point clouds , S1 and S2 . Given the non-rigidity assumption of the scene , the optimization is inherently unconstrained . Thus , a regularization term C is necessary to constrain the motion field . We therefore solve for scene flow as F∗ = argmin F ∑ p∈S1 D ( p + f , S2 ) + λC , ( 1 ) where D is a function to compute the distance from the perturbed point p by the flow vector f to its closest neighbor in S2 . C is a regularizer ( e.g. , Laplacian regularizer ) , and λ is a weighting factor for the regularizer . In this paper , we want to investigate using a neural prior to regularize the scene flow . 3.1 Neural scene flow prior . Learning-based scene flow methods learn scene flow priors from a large number of examples . As in Deep Image Prior [ 63 ] , we want to investigate if the structure of a neural network by itself is sufficient to capture a scene flow prior without any learning . Here we use a neural network as an implicit regularizer . The parameters are optimized as Θ∗ = argmin Θ ∑ p∈S1 D ( p + g ( p ; Θ ) , S2 ) , ( 2 ) where g is a neural network parameterized by Θ to regularize the scene flow F . The input to g is p , which is the point to be disturbed by the flow . The output of g is f and thus f∗ = g ( p ; Θ∗ ) . For the distance function D , we define it as D ( p , S ) = min x∈S ‖p− x‖22 . ( 3 ) In practice , we use it bidirectionally for both point sets , which is equivalent to Chamfer distance [ 15 ] . The objective in Eq . ( 2 ) , is for the forward scene flow F , which is the one we are interested in . However , it has been shown in [ 30 , 38 ] , that a cycle consistency regularizer encourages better scene flow estimations . The extra regularizer simply enforces the backward flow to be similar to the forward flow , Fbwd ≈ F . The optimal backward flow is defined as f∗bwd = g ( p′ ; Θ∗bwd ) , where p′ is the shifted point by the forward flow as p + f . Note that the network g is the same but with different parameters , Θbwd . Using the backward flow as an additional constraint , the optimal network weights are solved as Θ∗ , Θ∗bwd = argmin Θ , Θbwd ∑ p∈S1 D ( p + g ( p ; Θ ) , S2 ) + ∑ p′∈S′1 D ( p′ + g ( p′ ; Θbwd ) , S1 ) , ( 4 ) where S ′1 is the shifted S1 by the forward flow , i.e. , S ′1 = S1+F . Please find more details in the supplementary material . For the network g , we use MLPs with ReLU activations . The objective function in Eq . ( 4 ) can be optimized by gradient descent techniques using off-the-shelf frameworks with automatic differentiation . We show in the experiments section how the architecture of the neural prior affects performance by varying the number of hidden layers and units . Why use a neural scene flow prior ? Deep learning relies on massive amounts of data and computational resources to capture prior statistics . Although learning methods have achieved impressive results in most tasks , they still struggle when deployed in environments where the statistics are different from those captured during learning . Our intuition is that a neural prior acts as a strong implicit regularizer that constrains dynamic motion fields to be as smooth as possible . A neural scene flow prior also scales to large scenes while achieving high-fidelity results at a low computational cost . Our proposed method with 8 hidden layers and 128 hidden units has about 116k parameters . FlowNet3D [ 30 ] , for example , has about 1.2M parameters . Our method has ∼10× fewer parameters than the state-of-the-art supervised methods while achieving competitive , if not better , results . Lastly , our deep scene flow prior captures a continuous flow field that allows us to perform better scene flow interpolation across a sequence of point clouds .

Summary: This paper studies the problem of scene flow: estimating 3D motion fields between given point clouds from a dynamic scene. The proposed method predicts scene flows by optimizing a neural network at execution time, using randomly initialized weights.   This approach of using a network architecture as a prior is shown to have better generalization performance for out-of-distribution samples compared to learning-based methods which require offline datasets. In contrast to another recently introduced non-learning approach which uses graph Laplacian priors, this method provides a continuous scene flow representation, and has better time and accuracy performances in large scale scenes.