Neural Scene Flow Prior

1 Introduction . State-of-the-art results have recently been achieved by learning-based models [ 17 , 30 , 47 , 60 , 68 ] for the scene flow problem—the task of estimating 3D motion fields from dynamic scenes . However , such models heavily rely on large-scale data to capture prior knowledge , which is not always readily available . Scene flow annotations are expensive , and most methods train on synthetic and unrealistic scenarios to fine-tune on small real datasets . Poor generalization to unseen , out-of-the-distribution inputs is another problem . Prior information is generally limited to the statistics of the data used for training . Real-world applications such as autonomous driving require robust solutions to low-level vision tasks such as depth , optical , and scene flow estimation that work in statistically different scenarios . Inspired by recent innovations that make use of coordinate-based networks ( i.e. , pixels or 3D positions as inputs ) [ 9 , 36 , 37 , 39 , 56 ] for 3D modeling and rendering , we investigate the use of such networks to regularize the scene flow problem without any learning directly from point clouds . Optimization happens at runtime , and instead of learning a prior from data , the network structure itself captures the prior information . It is not limited to the statistics of a specific dataset . ∗Research done during internship at Argo AI . Corresponding e-mail : xueqian.li @ adelaide.edu.au . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Optimizing neural networks at execution time is not new . Ulyanov et al . [ 63 ] showed that a randomly initialized convolutional network could be used as a handcrafted prior for standard inverse problems such as image denoising , super-resolution , and inpainting . Ding and Feng [ 12 ] proposed a runtime optimization method ( DeepMapping ) for rigid pose estimation using deep neural networks . Although such deep image priors , deep mapping , and coordinate-based networks for neural scene representations have been successfully applied for inverse problems , rendering , and rigid registration , none has yet investigated ( to the best of our knowledge ) the use of network-based priors for regularizing scene flow directly from point clouds . Our proposed neural prior is based on a simple multilayer perceptron ( MLP ) architecture , and we show it is powerful enough to regularize scene flow given two point clouds implicitly . The input to the network is 3D points , and the output is a regularized scene flow . Our neural prior allows for a continuous scene flow representation instead of discrete such as in graph Laplacian-based priors , e.g. , [ 45 ] . We show how the flow fields captured by our neural prior can be employed to estimate long-term correspondences across a sequence of point clouds . The continuous scene flow allows for better integration of motions across time . Our results are promising and competitive to supervised [ 30 ] , self-supervised [ 38 , 68 ] , and nonlearning methods [ 1 , 45 ] ( see Table 1 ) . Our method also scales to real-world point clouds with tens of thousands of points while achieving better accuracy and time complexity than recent runtime optimization methods ( see Fig . 1 ) . 2 Related work . Non-learning-based scene flow Scene flow is the uplift from the optical flow , which is proposed by Vedula et al . [ 64 ] as the non-rigid motion field in the 3D space . The authors proposed the optimization-based scene flow estimation using image sequences to infer reconstruction knowledge of the flow in 3D surfaces . Successive RGB/RGB-D image-based work [ 4 , 18–21 , 27 , 43 , 44 , 50 ] used probability-based estimation , coarse-to-fine techniques , 6-DoF parameterization , or object segmentation , etc. , to improve accuracy and computation time . Although image-based scene flow methods are widely used , direct estimation of the scene flow from the point cloud is still possible through non-rigid registration methods , such as [ 1 , 10 , 26 , 42 ] . In this paper , we focus on point cloud-based scene flow estimation . Learning-based scene flow Image-based learning methods [ 6 , 51 , 54 , 60 , 70 ] use convolution and data supervision to solve scene flow from monocular or RGB-D images with the available depth information . Other image-based methods [ 22 , 23 , 32 , 52 , 53 ] take care of extra occlusion cues in the large-scale autonomous driving scenes . Point-based learning methods have become more prevalent with the rapid development of point cloud feature learning [ 28 , 48 , 49 , 65 , 67 ] . FlowNet3D [ 30 ] is a seminal work that estimates scene flow using PointNet++ [ 49 ] . Successive work [ 17 , 31 , 47 , 66 ] extends point-based learning methods using different feature extraction techniques . One obvious drawback of these supervised learning methods is the demand for sufficient ground truth labels . Besides , supervised methods lack generalizability while eventually only fitting domain-specific data . Self-supervised methods [ 25 , 38 , 61 , 68 ] , on the other hand , replaced the loss between the prediction and the ground truth flow with a point distance loss to use the point cloud itself as supervision . Self-supervision can adapt to different datasets and maintain certain generalizability . Nonetheless , massive training data are still required for sufficient learning . The graph Laplacian method Graph Laplacian [ 2 ] is widely used to smooth the surface in mesh processing [ 5,14,57,58 ] , point cloud denoising [ 11,71 ] , etc . Here we talk about the recent scene flow estimation using Graph Laplacian [ 45 ] . The method explicitly constructed a graph of the point cloud to constrain the non-rigid scene flow as rigid within a specific range . While as a dataless runtime optimization , the method is heavily affected by the hyperparameters of the graph and loses scalability when the point cloud becomes larger or the neighbors in the graph grow . Deep neural prior , implicit functions , and neural rendering Although large-scale data helps in feature representation [ 59 ] , end-to-end learning still requires high computation capacity and readily available training datasets . Instead , Ulyanov et al . [ 63 ] proposed a new style of optimization that uses a convolutional neural network to infer prior knowledge from the network architecture . One broader interest is to extend the idea of the network being an image function to the 3D shape modeling , and implicitly represent the continuous shape as level sets of neural networks . By directly mapping the 3D input to binary occupancy nets [ 9 , 36 ] , or signed distance functions [ 3 , 39 , 55 ] , it is powerful to model the 3D geometries in a continuous space using the coordinate-based network . The following Scene Representation Networks [ 56 ] takes advantage of the coordinate-based network and renders view synthetic images . Mildenhall et al . proposed a seminal work NeRF [ 37 ] , which is a novel way to do neural volume rendering using both point positions and viewing directions based on the coordinate-based network . Dynamic scene synthesis work [ 13 , 16 , 29 , 40 , 46 , 62 , 69 ] follows the NeRF framework , and integrates the motions to generate dynamic scenes . Some of the work [ 16 , 29 ] use scene flow to further constrain or segment dynamic scenes . An interesting work that solves for the rigid alignment between point clouds using runtime optimization is DeepMapping [ 12 ] . However , this work only deals with rigid motion , and the network architecture is more complex than coordinate-based networks . In this work , we are interested in coordinate-based networks to address the large-scale , real-world scene flow problem . 3 Approach . Problem definition Let S1 and S2 be two 3D point clouds sampled from a dynamic scene at time t-1 and t. The number of points in each point cloud , |S1| and |S2| , are typically different and not in correspondence . A 3D point p ∈ S1 moving from time t-1 to time t can be modeled by a translational vector ( or flow vector ) f ∈ R3 , where p′ = p + f . The collection of flow vectors for all 3D points is the scene flow F = { fi } |S1|i=1 . Optimization We want to optimize for a scene flow F that minimizes the distance between the two point clouds , S1 and S2 . Given the non-rigidity assumption of the scene , the optimization is inherently unconstrained . Thus , a regularization term C is necessary to constrain the motion field . We therefore solve for scene flow as F∗ = argmin F ∑ p∈S1 D ( p + f , S2 ) + λC , ( 1 ) where D is a function to compute the distance from the perturbed point p by the flow vector f to its closest neighbor in S2 . C is a regularizer ( e.g. , Laplacian regularizer ) , and λ is a weighting factor for the regularizer . In this paper , we want to investigate using a neural prior to regularize the scene flow . 3.1 Neural scene flow prior . Learning-based scene flow methods learn scene flow priors from a large number of examples . As in Deep Image Prior [ 63 ] , we want to investigate if the structure of a neural network by itself is sufficient to capture a scene flow prior without any learning . Here we use a neural network as an implicit regularizer . The parameters are optimized as Θ∗ = argmin Θ ∑ p∈S1 D ( p + g ( p ; Θ ) , S2 ) , ( 2 ) where g is a neural network parameterized by Θ to regularize the scene flow F . The input to g is p , which is the point to be disturbed by the flow . The output of g is f and thus f∗ = g ( p ; Θ∗ ) . For the distance function D , we define it as D ( p , S ) = min x∈S ‖p− x‖22 . ( 3 ) In practice , we use it bidirectionally for both point sets , which is equivalent to Chamfer distance [ 15 ] . The objective in Eq . ( 2 ) , is for the forward scene flow F , which is the one we are interested in . However , it has been shown in [ 30 , 38 ] , that a cycle consistency regularizer encourages better scene flow estimations . The extra regularizer simply enforces the backward flow to be similar to the forward flow , Fbwd ≈ F . The optimal backward flow is defined as f∗bwd = g ( p′ ; Θ∗bwd ) , where p′ is the shifted point by the forward flow as p + f . Note that the network g is the same but with different parameters , Θbwd . Using the backward flow as an additional constraint , the optimal network weights are solved as Θ∗ , Θ∗bwd = argmin Θ , Θbwd ∑ p∈S1 D ( p + g ( p ; Θ ) , S2 ) + ∑ p′∈S′1 D ( p′ + g ( p′ ; Θbwd ) , S1 ) , ( 4 ) where S ′1 is the shifted S1 by the forward flow , i.e. , S ′1 = S1+F . Please find more details in the supplementary material . For the network g , we use MLPs with ReLU activations . The objective function in Eq . ( 4 ) can be optimized by gradient descent techniques using off-the-shelf frameworks with automatic differentiation . We show in the experiments section how the architecture of the neural prior affects performance by varying the number of hidden layers and units . Why use a neural scene flow prior ? Deep learning relies on massive amounts of data and computational resources to capture prior statistics . Although learning methods have achieved impressive results in most tasks , they still struggle when deployed in environments where the statistics are different from those captured during learning . Our intuition is that a neural prior acts as a strong implicit regularizer that constrains dynamic motion fields to be as smooth as possible . A neural scene flow prior also scales to large scenes while achieving high-fidelity results at a low computational cost . Our proposed method with 8 hidden layers and 128 hidden units has about 116k parameters . FlowNet3D [ 30 ] , for example , has about 1.2M parameters . Our method has ∼10× fewer parameters than the state-of-the-art supervised methods while achieving competitive , if not better , results . Lastly , our deep scene flow prior captures a continuous flow field that allows us to perform better scene flow interpolation across a sequence of point clouds .

Summary:  This paper highlights the fact that deep learning has caused a paradigm shift in many computer vision applications from methods based on optimization with strong priors to ones that optimize  feed-forward neural networks in an offline process. The paper argues that these feed-forward models are bad at generalization and proposes to use the network in a radically different way. In essence, the network are used as a parametrization for a traditional optimization solution - allowing the network's inductive biases to be used as a “prior”. While using a network structure as a prior for optimization is not new (ala Deep Image Prior, DeepSDF, NeRF etc.), applying this type of parametrization to the problem of optical flow is new. The results are competitive with trained approaches and quite impressive for such a simple solution.