AdvectiveNet: An Eulerian-Lagrangian Fluidic Reservoir for Point Cloud Processing     

1 INTRODUCTION The fundamental mechanism of deep learning is to uncover complex feature structures from large data sets using a hierarchical model composed of simple layers . These data structures , such as a uniform grid ( Lecun et al. , 1998 ) , an unstructured graph ( Kipf & Welling , 2016 ) , or a hierarchical point set ( Qi et al. , 2016a ; 2017 ) , function as geometric reservoirs to yield intricate underpinning patterns by evolving the massive input data in a high-dimensional parameter space . On another front , computational physics researchers have been mastering the art of inventing geometric data structures and simulation algorithms to model complex physical systems ( Gibou et al. , 2019 ) . Lagrangian structures , which track the motion in a moving local frame such as a particle system ( Monaghan , 1992 ) , and Eulerian structures , which describe the evolution in a fixed world frame such as a Cartersian grid ( Fedkiw et al. , 2001 ) , are the two mainstream approaches . Various differential operators have been devised on top of these data structures to model complex fluid or solid systems . Pioneered by E ( 2017 ) and popularized by many others , e.g. , ( Long et al. , 2018 ; Chen et al. , 2018 ; Ruthotto & Haber , 2018 ) , treating the data flow as the evolution of a dynamic system is connecting machine learning and physics simulation . As E ( 2017 ) notes , there exists a mathematical equivalence between the forward data propagation on a neural network and the temporal evolution of a dynamic system . Accordingly , the training process of a neural network amounts to finding the optimal control forces exerted on a dynamic system to minimize a specific energy form . Point cloud processing is of particular interest under this perspective . The two main challenges : to build effective convolution stencils and to evolve learned nonlinear features ( Qi et al. , 2016a ; ∗xingzhe.he @ rutgers.edu †Helen.L.Cao.22 @ dartmouth.edu ‡bo.zhu @ dartmouth.edu Atzmon et al. , 2018 ; Wang et al. , 2019 ) , can map conceptually to the challenges of devising worldframe differential operators and tracking material-space continuum deformations when simulating a PDE-driven dynamic system in computational physics . We envision that the key to solving these challenges lies in the adaption of the most suited geometric data structures to synergistically handle the Eulerian and Lagrangian aspects of the problem . In particular , it is essential to devise data structures and computational paradigms that can accommodate global fast convolutions , and at the same time track the non-linear feature evolution . The key motivation of this work originates from physical computing that tackles its various framedependent and temporally-evolved computational challenges by creating the most natural and effective geometric toolsets under the two different viewpoints . We are specifically interested in uncovering the intrinsic connections between a point cloud learning problem and a computational fluid dynamic ( CFD ) problem . We observe that the two problems share an important common thread regarding their computational model , which both evolve Lagrangian particles in an Eulerian space guided by the first principle of energy minimization . Such observations shed new insight into the 3D point cloud processing and further opens the door for marrying the state-of-the-art CFD techniques to tackle the challenges emerging in point cloud learning . To this end , this paper conducts a preliminary exploration to establish an Eulerian-Lagrangian fluidic reservoir that accommodates the learning process of point clouds . The key idea of the proposed method is to solve the point cloud learning problem as a flow advection problem jointly defined in a Eulerian world space and a Lagrangian material space . The defining characteristic distinguishing our method from others is that the spatial interactions among the Lagrangian particles can evolve temporally via advection in a learned flow field , like their fluidic counterpart in a physical circumstance . This inherently takes advantage of the fundamental flow phenomena in evolving and separating Lagrangian features non-linearly ( see Figure 1 ) . In particular , we draw the idea of Lagrangian advection on an Eulerian reservoir from both the Particle-In-Cell ( PIC ) method ( Evans & Harlow , 1957 ) and the Fluid-Implicit-Particle ( FLIP ) method ( Brackbill et al. , 1987 ) , which are wholly recognized as ’ PIC/FLIP ’ in modeling large-scale flow phenomena in both computational fluids , solids , and even visual effects . We demonstrate the result of this synergy by building a physics-inspired learning pipeline with straightforward implementation and matching the state-of-the-art with this framework . The key contributions of our work include : • An advective scheme to mimic the natural flow convection process for feature separation ; • A fluid-inspired learning paradigm with effective particle-grid transfer schemes ; • A fully Eulerian-Lagrangian approach to process point clouds , with the inherent advantages in creating Eulerian differential stencils and tracking Lagrangian evolution ; • A simple and efficient physical reservoir learning algorithm . 2 RELATED WORKS . This section briefly reviews the recent related work on point cloud processing . According to data structures used for building the convolution stencil , the methods can be categorized as Lagrangian ( using particles only ) , Eulerian ( using a background grid ) , and hybrid ( using both ) . We also review the physical reservoir methods that embed network training into a physical simulation process . Lagrangian Lagrangian methods build convolution operators on the basis of local points . Examples include PointNet ( Qi et al. , 2016a ) , which conducts max pooling to combat any disorganized points , PointNet++ ( Qi et al. , 2017 ) , which leverages farthest point sampling to group particles , and a set of work ( Wang et al. , 2019 ; Xu et al. , 2018 ; Li et al. , 2018b ; a ; Jiang et al. , 2018 ) based on k-nearest neighbors . Beyond the mesh-free approaches , researchers also seek to build effective point-based stencils by establishing local connectivities among points . Most significantly , geometric deep learning ( Bruna et al. , 2013 ; Bronstein et al. , 2016 ) builds convolution operators on top of a mesh to uncover the intrinsic features of objects ’ geometry . In particular , we want to highlight the work on dynamic graph CNN ( Wang et al. , 2019 ) , which builds directed graphs in an extemporaneous fashion in feature space to guide the point neighbor search process , which shares similarities with our approach . Eulerian Eulerian approaches leverage background discretizations to perform computation . The most successful Eulerian method is the CNN ( Lecun et al. , 1998 ) , which builds the convolution operator on a 2D uniform grid . This Eulerian representation can be used to process 3D data by using multiple views ( Su et al. , 2015 ; Qi et al. , 2016b ; Feng et al. , 2018 ) and extended to 3D volumetric grids ( Maturana & Scherer , 2015 ; Qi et al. , 2016b ; Z. Wu , 2015 ) . Grid resolution is the main performance bottleneck for 3D CNN methods . Adaptive data structures such as Octree ( Riegler et al. , 2016 ; Wang et al. , 2017 ; 2018 ) , Kd-tree ( Klokov & Lempitsky , 2017 ) , and multi-level 3D CNN ( Ghadai et al. , 2018 ) were invented to alleviate the problem . Another example of Eulerian structures is Spherical CNN ( Cohen et al. , 2018 ) that projects 3D shapes onto a spherical coordinate system to define equivalent rotation convolution . In addition to these voxel-based , diffusive representations , shapes can also be described as a sharp interface modeled as an implicit level set function ( Hu et al. , 2017 ; Park et al. , 2019 ; Mescheder et al. , 2019 ) . For each point in the space , the level set function acts as a binary classifier checking whether the point is inside the shape or not . Hybrid There have been recent attempts to transfer data between Lagrangian and Eulerian representations for efficient convolution implementation . These data transfer methods can be one-way ( Wang et al. , 2017 ; Klokov & Lempitsky , 2017 ; Tchapmi et al. , 2017 ; Le & Duan , 2018 ) , in which case the data is mapped from points to grid cells permanently , or two-way ( Su et al. , 2018 ; Atzmon et al. , 2018 ; Liu et al. , 2019 ; Groueix et al. , 2018 ) , in which case data is pushed forward from particle to grid for convolution and pushed backward from grid to particle for evolution . Auto-encoders on point clouds ( Fan et al. , 2016 ; Achlioptas et al. , 2017 ; Yang et al. , 2017 ; Yu et al. , 2018 ; Zhao et al. , 2019 ) can be also regarded as a hybrid approach , where encoded data is Eulerian and decoded data is Lagrangian . In addition , we want to mention the physical reservoir computing techniques that focus on the leverage of the temporal , physical evolution to solve learning problems , e.g. , see ( Jaeger , 2001 ) and ( Maass et al. , 2002 ) . Physical reservoir computing is demonstrating successes in various applications ( Jalalvand et al. , 2015 ; Jaeger , 2002 ; Hauser et al. , 2012 ; Lukoševičius & Jaeger , 2009 ; Tanaka et al. , 2019 ) . 3 ALGORITHM . PIC/FLIP overview Before describing the details of our method , we begin with briefly surveying the background of the PIC/FLIP method . PIC/FLIP uses a hybrid grid-particle representation to describe fluid evolution . The particles are used for tracking materials , and the grid is used for discretizing space . Properties such as mass , density , and velocity are carried on particles . Each simulation step consists of four substeps : particle-to-grid transfer IGP , grid force calculation ( Projection ) , grid-to-particle transfer IPG , and moving particles ( Advection ) . In the I G P step , the properties on each particle are interpolated onto a background grid . In the Projection step , calculations such as adding body forces and enforcing incompressibility are conducted on the background grid . After this , the velocities on grid nodes are interpolated back onto particles , i.e. , IPG . Finally , particles move to their new positions for the next time step using the updated velocities ( Advection ) . As summarized above , the key philosophy of PIC/FLIP is to carry all features on particles and to perform all differential calculations on the grid . The background grid functions as a computational paradigm that can be established extemporaneously when needed . Data will transfer from particle to grid and then back to particle to finish a simulation loop . Our proposed approach follows the same design philosophy as PIC/FLIP by storing the learned features on particles and conducting differential calculations on the grid . The Lagrangian features will evolve with the particles moving in an Eulerian space and interact with local grid nodes . As shown in Figure 2 , the learning pipeline mimics the PIC/FLIP simulation loop in the sense that Lagrangian particles are advected passively in an Eulerian space guided by a learned velocity field . Initialization We initialize a particle system P and a background grid G as the Lagrangian and Eulerian representations respectively for processing point clouds . We use the subscript p to refer to particle indices and i to refer to the grid nodes . For the Lagrangian portion , the particle system has n particles , with each particle Pp carrying its position xp ∈ R3 , velocity vp ∈ R3 , mass mp ∈ R , and a feature vector fp ∈ Rk ( k = 64 initially ) . The particle velocity is zero at the beginning . The particle mass mp = 1 will keep constant over the entire evolution . To initialize the feature vector fp , we first put the particles in a grid with size N3 . For each cell , we calculate 1 ) the center of mass of all the particles in the cell , and 2 ) the normalized vector pointing from each particle to the this mass center . For each particle , we concatenate these two vectors to the initial feature vector . This process is repeated for N = 2 , 4 , 6 , 8 , 10 , 12 . The resulting feature vector with the length of 6 × 6 are fed into a multi-layer perceptron ( MLP ) to generate the feature vector fp . For the Eulerian part , we start with a 3D uniform grid G to represent the bounding box of the particles . The resolution of the grid is N3 ( N = 16 for most of our cases ) . At the beginning , the particle system and its bounding box are normalized to the space of [ −1 , 1 ] 3 . Each grid node Gi of G stores data interpolated from the particles . Particle-grid transfer Both the interpolation from grid to particle and particle to grid are executed using tri-linear interpolation , which is a common scheme for property transfer in simulation and learning code . Generalized grid forces With the feature vectors transferred from particles to grid nodes , we devise a 3D CNN on the grid to calculate a generalized force field based on the Eulerian features . The network consists of three convolution layers , with each layer as a combination of 3D convolution , batch norm , and ReLU . The input of the network is a vector field F ( kj ) ×N×N×N composed of the feature vectors on all grid nodes , with k as the feature vector size ( 64 by default ) and j as the iteration index in the evolution loop ( see Figure 2 ) . The output is a convoluted vector field F ( kj ) ×N×N×Nc with the same size as F. We use Fc for two purposes : 1 ) To interpolate Fc from the grid back onto particles and append it to the current feature vector in order to enrich its feature description ; 2 ) To feed Fc into another singlelayer network to generate the new Eulerian velocity field V for the particle advection . Specifically , this V is interpolated back onto particles in the same way as the feature interpolation to update the particle positions for the next iteration ( see Advection for details ) . Advection The essence of an advection process is to solve the advection equation with the Lagrangian form Dv/Dt = 0 or the Eulerian form ∂v/∂t+ v · ∇v = 0 . The advection equation describes the passive evolution of particle properties within a flow field . With the learned grid velocity field in hand , we will update the particle velocity following the conventional scheme of PIC/FLIP . Specifically , the new velocity is first updated by interpolating the Eulerian velocity to particles ( the PIC step ) : vn+1PIC = I P G ( v n+1 g ) ( 1 ) Then , we interpolate the difference between the new and the old Eulerian velocity : vn+1FLIP = v n p + I P G ( v n+1 g − IGP ( vnp ) ) , ( 2 ) and then add them to the particle with a weight α ( =0.5 in default . ) : vn+1p = α ∗ vn+1PIC + ( 1− α ) ∗ v n+1 FLIP ( 3 ) With the updated velocity on each particle from the IPG interpolation , the particle ’ s position for the next time step can be updated using a standard time integration scheme ( explicit Euler in our implementation ) : xn+1p = x n p + v n+1 p ∆t . ( 4 ) Boundary conditions We apply a soft boundary constraints by adding an penalty term in the objective function to avoid particles moving outside of the grid : φb = 1 n ∑ p max ( 0 , ‖xp‖2 − 1 ) ( 5 ) where xp represents the pth particle in the whole batch and n is the number of particles in the whole batch . We penalize on all the particles that run outside the grid . We also design the gather penalty and the diffusion objectives to enhance the particle diffusion and clustering effects during evolution ( specifically for the segmentation application ) : φg = 1 2 ∑ l ∑ m max ( 0 , 1− ‖cl − cm‖ ) ( 6 ) φd = 1 n ∑ l ∑ p ‖cl − xlp‖ ( 7 ) where cl and cm are the centers of particles of label l and m and xlp is the pth particle with label l .

Summary: The paper is about using classical PIC/FLIP scheme in Computational Fluid Dynamics for solving the learning problem of 3D object detection and segmentation. In general, there are extrinsic CNNs like the Vox net etc. which look for global features which the authors refer to as Eulerian formulation of the data representation, and there are intrinsic CNNs like the GCN(graph convolutions), Point nets etc. which look for localized neighborhood information which the authors refer to as Lagrangian formulation. The authors acknowledge that hybridizing the extrinsic CNNs and intrinsic CNNs is not new and several works are cited. The key contribution is to look at this problem from the perspective of PIC/FLIP scheme which has been used in CFD for decades.