AdvectiveNet: An Eulerian-Lagrangian Fluidic Reservoir for Point Cloud Processing     

1 INTRODUCTION The fundamental mechanism of deep learning is to uncover complex feature structures from large data sets using a hierarchical model composed of simple layers . These data structures , such as a uniform grid ( Lecun et al. , 1998 ) , an unstructured graph ( Kipf & Welling , 2016 ) , or a hierarchical point set ( Qi et al. , 2016a ; 2017 ) , function as geometric reservoirs to yield intricate underpinning patterns by evolving the massive input data in a high-dimensional parameter space . On another front , computational physics researchers have been mastering the art of inventing geometric data structures and simulation algorithms to model complex physical systems ( Gibou et al. , 2019 ) . Lagrangian structures , which track the motion in a moving local frame such as a particle system ( Monaghan , 1992 ) , and Eulerian structures , which describe the evolution in a fixed world frame such as a Cartersian grid ( Fedkiw et al. , 2001 ) , are the two mainstream approaches . Various differential operators have been devised on top of these data structures to model complex fluid or solid systems . Pioneered by E ( 2017 ) and popularized by many others , e.g. , ( Long et al. , 2018 ; Chen et al. , 2018 ; Ruthotto & Haber , 2018 ) , treating the data flow as the evolution of a dynamic system is connecting machine learning and physics simulation . As E ( 2017 ) notes , there exists a mathematical equivalence between the forward data propagation on a neural network and the temporal evolution of a dynamic system . Accordingly , the training process of a neural network amounts to finding the optimal control forces exerted on a dynamic system to minimize a specific energy form . Point cloud processing is of particular interest under this perspective . The two main challenges : to build effective convolution stencils and to evolve learned nonlinear features ( Qi et al. , 2016a ; ∗xingzhe.he @ rutgers.edu †Helen.L.Cao.22 @ dartmouth.edu ‡bo.zhu @ dartmouth.edu Atzmon et al. , 2018 ; Wang et al. , 2019 ) , can map conceptually to the challenges of devising worldframe differential operators and tracking material-space continuum deformations when simulating a PDE-driven dynamic system in computational physics . We envision that the key to solving these challenges lies in the adaption of the most suited geometric data structures to synergistically handle the Eulerian and Lagrangian aspects of the problem . In particular , it is essential to devise data structures and computational paradigms that can accommodate global fast convolutions , and at the same time track the non-linear feature evolution . The key motivation of this work originates from physical computing that tackles its various framedependent and temporally-evolved computational challenges by creating the most natural and effective geometric toolsets under the two different viewpoints . We are specifically interested in uncovering the intrinsic connections between a point cloud learning problem and a computational fluid dynamic ( CFD ) problem . We observe that the two problems share an important common thread regarding their computational model , which both evolve Lagrangian particles in an Eulerian space guided by the first principle of energy minimization . Such observations shed new insight into the 3D point cloud processing and further opens the door for marrying the state-of-the-art CFD techniques to tackle the challenges emerging in point cloud learning . To this end , this paper conducts a preliminary exploration to establish an Eulerian-Lagrangian fluidic reservoir that accommodates the learning process of point clouds . The key idea of the proposed method is to solve the point cloud learning problem as a flow advection problem jointly defined in a Eulerian world space and a Lagrangian material space . The defining characteristic distinguishing our method from others is that the spatial interactions among the Lagrangian particles can evolve temporally via advection in a learned flow field , like their fluidic counterpart in a physical circumstance . This inherently takes advantage of the fundamental flow phenomena in evolving and separating Lagrangian features non-linearly ( see Figure 1 ) . In particular , we draw the idea of Lagrangian advection on an Eulerian reservoir from both the Particle-In-Cell ( PIC ) method ( Evans & Harlow , 1957 ) and the Fluid-Implicit-Particle ( FLIP ) method ( Brackbill et al. , 1987 ) , which are wholly recognized as ’ PIC/FLIP ’ in modeling large-scale flow phenomena in both computational fluids , solids , and even visual effects . We demonstrate the result of this synergy by building a physics-inspired learning pipeline with straightforward implementation and matching the state-of-the-art with this framework . The key contributions of our work include : • An advective scheme to mimic the natural flow convection process for feature separation ; • A fluid-inspired learning paradigm with effective particle-grid transfer schemes ; • A fully Eulerian-Lagrangian approach to process point clouds , with the inherent advantages in creating Eulerian differential stencils and tracking Lagrangian evolution ; • A simple and efficient physical reservoir learning algorithm . 2 RELATED WORKS . This section briefly reviews the recent related work on point cloud processing . According to data structures used for building the convolution stencil , the methods can be categorized as Lagrangian ( using particles only ) , Eulerian ( using a background grid ) , and hybrid ( using both ) . We also review the physical reservoir methods that embed network training into a physical simulation process . Lagrangian Lagrangian methods build convolution operators on the basis of local points . Examples include PointNet ( Qi et al. , 2016a ) , which conducts max pooling to combat any disorganized points , PointNet++ ( Qi et al. , 2017 ) , which leverages farthest point sampling to group particles , and a set of work ( Wang et al. , 2019 ; Xu et al. , 2018 ; Li et al. , 2018b ; a ; Jiang et al. , 2018 ) based on k-nearest neighbors . Beyond the mesh-free approaches , researchers also seek to build effective point-based stencils by establishing local connectivities among points . Most significantly , geometric deep learning ( Bruna et al. , 2013 ; Bronstein et al. , 2016 ) builds convolution operators on top of a mesh to uncover the intrinsic features of objects ’ geometry . In particular , we want to highlight the work on dynamic graph CNN ( Wang et al. , 2019 ) , which builds directed graphs in an extemporaneous fashion in feature space to guide the point neighbor search process , which shares similarities with our approach . Eulerian Eulerian approaches leverage background discretizations to perform computation . The most successful Eulerian method is the CNN ( Lecun et al. , 1998 ) , which builds the convolution operator on a 2D uniform grid . This Eulerian representation can be used to process 3D data by using multiple views ( Su et al. , 2015 ; Qi et al. , 2016b ; Feng et al. , 2018 ) and extended to 3D volumetric grids ( Maturana & Scherer , 2015 ; Qi et al. , 2016b ; Z. Wu , 2015 ) . Grid resolution is the main performance bottleneck for 3D CNN methods . Adaptive data structures such as Octree ( Riegler et al. , 2016 ; Wang et al. , 2017 ; 2018 ) , Kd-tree ( Klokov & Lempitsky , 2017 ) , and multi-level 3D CNN ( Ghadai et al. , 2018 ) were invented to alleviate the problem . Another example of Eulerian structures is Spherical CNN ( Cohen et al. , 2018 ) that projects 3D shapes onto a spherical coordinate system to define equivalent rotation convolution . In addition to these voxel-based , diffusive representations , shapes can also be described as a sharp interface modeled as an implicit level set function ( Hu et al. , 2017 ; Park et al. , 2019 ; Mescheder et al. , 2019 ) . For each point in the space , the level set function acts as a binary classifier checking whether the point is inside the shape or not . Hybrid There have been recent attempts to transfer data between Lagrangian and Eulerian representations for efficient convolution implementation . These data transfer methods can be one-way ( Wang et al. , 2017 ; Klokov & Lempitsky , 2017 ; Tchapmi et al. , 2017 ; Le & Duan , 2018 ) , in which case the data is mapped from points to grid cells permanently , or two-way ( Su et al. , 2018 ; Atzmon et al. , 2018 ; Liu et al. , 2019 ; Groueix et al. , 2018 ) , in which case data is pushed forward from particle to grid for convolution and pushed backward from grid to particle for evolution . Auto-encoders on point clouds ( Fan et al. , 2016 ; Achlioptas et al. , 2017 ; Yang et al. , 2017 ; Yu et al. , 2018 ; Zhao et al. , 2019 ) can be also regarded as a hybrid approach , where encoded data is Eulerian and decoded data is Lagrangian . In addition , we want to mention the physical reservoir computing techniques that focus on the leverage of the temporal , physical evolution to solve learning problems , e.g. , see ( Jaeger , 2001 ) and ( Maass et al. , 2002 ) . Physical reservoir computing is demonstrating successes in various applications ( Jalalvand et al. , 2015 ; Jaeger , 2002 ; Hauser et al. , 2012 ; Lukoševičius & Jaeger , 2009 ; Tanaka et al. , 2019 ) . 3 ALGORITHM . PIC/FLIP overview Before describing the details of our method , we begin with briefly surveying the background of the PIC/FLIP method . PIC/FLIP uses a hybrid grid-particle representation to describe fluid evolution . The particles are used for tracking materials , and the grid is used for discretizing space . Properties such as mass , density , and velocity are carried on particles . Each simulation step consists of four substeps : particle-to-grid transfer IGP , grid force calculation ( Projection ) , grid-to-particle transfer IPG , and moving particles ( Advection ) . In the I G P step , the properties on each particle are interpolated onto a background grid . In the Projection step , calculations such as adding body forces and enforcing incompressibility are conducted on the background grid . After this , the velocities on grid nodes are interpolated back onto particles , i.e. , IPG . Finally , particles move to their new positions for the next time step using the updated velocities ( Advection ) . As summarized above , the key philosophy of PIC/FLIP is to carry all features on particles and to perform all differential calculations on the grid . The background grid functions as a computational paradigm that can be established extemporaneously when needed . Data will transfer from particle to grid and then back to particle to finish a simulation loop . Our proposed approach follows the same design philosophy as PIC/FLIP by storing the learned features on particles and conducting differential calculations on the grid . The Lagrangian features will evolve with the particles moving in an Eulerian space and interact with local grid nodes . As shown in Figure 2 , the learning pipeline mimics the PIC/FLIP simulation loop in the sense that Lagrangian particles are advected passively in an Eulerian space guided by a learned velocity field . Initialization We initialize a particle system P and a background grid G as the Lagrangian and Eulerian representations respectively for processing point clouds . We use the subscript p to refer to particle indices and i to refer to the grid nodes . For the Lagrangian portion , the particle system has n particles , with each particle Pp carrying its position xp ∈ R3 , velocity vp ∈ R3 , mass mp ∈ R , and a feature vector fp ∈ Rk ( k = 64 initially ) . The particle velocity is zero at the beginning . The particle mass mp = 1 will keep constant over the entire evolution . To initialize the feature vector fp , we first put the particles in a grid with size N3 . For each cell , we calculate 1 ) the center of mass of all the particles in the cell , and 2 ) the normalized vector pointing from each particle to the this mass center . For each particle , we concatenate these two vectors to the initial feature vector . This process is repeated for N = 2 , 4 , 6 , 8 , 10 , 12 . The resulting feature vector with the length of 6 × 6 are fed into a multi-layer perceptron ( MLP ) to generate the feature vector fp . For the Eulerian part , we start with a 3D uniform grid G to represent the bounding box of the particles . The resolution of the grid is N3 ( N = 16 for most of our cases ) . At the beginning , the particle system and its bounding box are normalized to the space of [ −1 , 1 ] 3 . Each grid node Gi of G stores data interpolated from the particles . Particle-grid transfer Both the interpolation from grid to particle and particle to grid are executed using tri-linear interpolation , which is a common scheme for property transfer in simulation and learning code . Generalized grid forces With the feature vectors transferred from particles to grid nodes , we devise a 3D CNN on the grid to calculate a generalized force field based on the Eulerian features . The network consists of three convolution layers , with each layer as a combination of 3D convolution , batch norm , and ReLU . The input of the network is a vector field F ( kj ) ×N×N×N composed of the feature vectors on all grid nodes , with k as the feature vector size ( 64 by default ) and j as the iteration index in the evolution loop ( see Figure 2 ) . The output is a convoluted vector field F ( kj ) ×N×N×Nc with the same size as F. We use Fc for two purposes : 1 ) To interpolate Fc from the grid back onto particles and append it to the current feature vector in order to enrich its feature description ; 2 ) To feed Fc into another singlelayer network to generate the new Eulerian velocity field V for the particle advection . Specifically , this V is interpolated back onto particles in the same way as the feature interpolation to update the particle positions for the next iteration ( see Advection for details ) . Advection The essence of an advection process is to solve the advection equation with the Lagrangian form Dv/Dt = 0 or the Eulerian form ∂v/∂t+ v · ∇v = 0 . The advection equation describes the passive evolution of particle properties within a flow field . With the learned grid velocity field in hand , we will update the particle velocity following the conventional scheme of PIC/FLIP . Specifically , the new velocity is first updated by interpolating the Eulerian velocity to particles ( the PIC step ) : vn+1PIC = I P G ( v n+1 g ) ( 1 ) Then , we interpolate the difference between the new and the old Eulerian velocity : vn+1FLIP = v n p + I P G ( v n+1 g − IGP ( vnp ) ) , ( 2 ) and then add them to the particle with a weight α ( =0.5 in default . ) : vn+1p = α ∗ vn+1PIC + ( 1− α ) ∗ v n+1 FLIP ( 3 ) With the updated velocity on each particle from the IPG interpolation , the particle ’ s position for the next time step can be updated using a standard time integration scheme ( explicit Euler in our implementation ) : xn+1p = x n p + v n+1 p ∆t . ( 4 ) Boundary conditions We apply a soft boundary constraints by adding an penalty term in the objective function to avoid particles moving outside of the grid : φb = 1 n ∑ p max ( 0 , ‖xp‖2 − 1 ) ( 5 ) where xp represents the pth particle in the whole batch and n is the number of particles in the whole batch . We penalize on all the particles that run outside the grid . We also design the gather penalty and the diffusion objectives to enhance the particle diffusion and clustering effects during evolution ( specifically for the segmentation application ) : φg = 1 2 ∑ l ∑ m max ( 0 , 1− ‖cl − cm‖ ) ( 6 ) φd = 1 n ∑ l ∑ p ‖cl − xlp‖ ( 7 ) where cl and cm are the centers of particles of label l and m and xlp is the pth particle with label l .

Summary: The paper presents a method for point-based learning that is inspired by a hybrid Eulerian-Lagrangian fluid simulation method. The work first explains how the simulation algorithm is mapped to the learning problem: MLPs are employed to learn sets of particle based features which are mapped to a Eulerian grid. A second MLP infers a particle based velocity, which is likewise mapped to the grid and used to advect the grid quantities. This is repeated for a certain number of steps to obtain final positions. The "warped" features are then projected back onto the particles to solve, e.g., a classification task. In contrast to a typical flow solver, the motion can be divergent, i.e., not necessarily conserves volume.