NPTC-net: Narrow-Band Parallel Transport Convolutional Neural Network on Point Clouds

1 INTRODUCTION . Convolution is one of the most widely used operators in applied mathematics , computer science and engineering . It is also the most important building block of Convolutional Neural Netowrks ( CNNs ) which are the main driven force in the recent success of deep learning LeCun et al . ( 2015 ) ; Goodfellow et al . ( 2016 ) . In the Euclidean space Rn , the convolution of function f with a kernel ( or filter ) k is defined as ( f ∗ k ) ( x ) : = ∫ Rn k ( x− y ) f ( y ) dy . ( 1 ) This operation can be easily calculated in Euclidean spaces due to the shift-invariance of the space so that the translates of the filter k , i.e . k ( x−y ) is naturally defined . With the rapid development of data science , more and more non-Euclidean data emerged in various fields including network data from social science , 3D shapes from medical images and computer graphics , data from recommending systems , etc . Therefore , geometric deep learning , i.e . deep learning of non-Euclidean data , is now a rapidly growing branch of deep learning Bronstein et al . ( 2017 ) . In this paper , we will discuss how we can generalize the definition of convolution to ( manifold-structured ) point clouds in a way that it inherits desirable properties of the planar convolution , thus it enables to design convolutional neural networks on point clouds . One of the main challenges of defining convolution on manifolds and point clouds ( a discrete form of manifolds ) is to define translation x− y on the non-Euclidean domain . Other than convolutions , we also need to properly define pooling to enable networks to extract global features and to save memory during training . Multiple types of generalization of convolutions on manifolds , graphs and point clouds have been proposed in recent years . We shall recall some of them and discuss the relation between existing definitions of convolutions and the proposed narrow-band parallel transport convolution . 1.1 RELATED WORK OF CONVOLUTIONS ON NON-EUCLIDEAN DOMAINS . Spectral methods avoid the direct definition of translation x− y by utilizing the convolution theorem : for any two functions f and g , f̂ ∗ g = f̂ · ĝ . Therefore , we have f ∗ g = ( f̂ · ĝ ) ∨ , where ∧ and ∨ represent generalized Fourier transform and inverse Fourier transform provided through the associated Laplace-Beltrami ( LB ) eigensystem on manifolds . To avoid computing convolution through full eigenvalue decomposition , polynomial approximation is proposed and yields convolution as action of polynomials of the LB operator Hammond et al . ( 2011 ) ; Dong ( 2017 ) . Thus , convolutional neural networks can be designed Bruna et al . ( 2014 ) ; Defferrard et al . ( 2016 ) ; Levie et al . ( 2019 ) . Spectral methods , however , suffer two major drawbacks . First , these methods define convolution in the frequency domain . As a result , the learned filters are not spatially localized . Secondly , spectral convolutions are domain-dependent as deformation of the ground manifold will change the corresponding LB eigensystem . This obstructs the use of learning networks from one training domain to a new testing domain Schonsheck et al . ( 2018 ) . Spatial mesh-based methods are more similar to the Euclidean case , and this is one of the reasons why most of the existing works fall into this category . The philosophy behind these methods is that the tangent plane TxM of a 2-dimensional manifold M in each point x ∈ M is embedded to a 2-dimensional Euclidean domain where convolution can be easily defined . In this paper , we make the first attempt to interpret some of the existing mesh-based methods in a unified framework . We claim that most of the spatial mesh-based methods can be formulated as ( f ∗ k ) ( x ) : = ∫ Tx , M k ( φ ( x , v ) ) f ( v ) dv , x ∈M , v ∈ TxM . ( 2 ) Here , k : R2 → R is a convolution kernel and Tx , M = { v ∈ TxM : 〈v , v〉gx ≤ 2 } with > 0 being the size of the kernel . The mapping φ ( x , v ) : M×TxM→ R2 is defined as φ ( x , v ) = ( α1 , α2 ) = ( 〈 v , ~u1x 〉 gx , 〈 v , ~u2x 〉 gx ) , ( 3 ) where ~ujx ∈ TxM , j = 1 , 2 . For simplicity , we will denote ~ux = ( ~u1x , ~u2x ) . Most of the designs of the existing manifold convolutions focused on the designs of ~ux . We remark that possible singularities will lead to no convolution operation at those points . These are isolated points on a closed manifold and for more discussion of singularities , please refer to the Appendix . For example , GCNN Masci et al . ( 2015 ) and ACNN Boscaini et al . ( 2016 ) construct a local geodesic polar coordinate system on a manifold , formulating the convolution as ( f ∗ k ) ( x ) = ∫ k ( ( φ ◦ ψ ) ( θ , r ) ) ( Qxf ) ( r , θ ) drdθ , whereQ is a local interpolation function with interpolation domain an isotropic disc for GCNN and an anisotropic ellipse for ACNN . A local geodesic polar coordinate system on a manifold can also be transformed to a 2-dimensional planar coordinate system on its tangent plane . Such transformation is the mapping ψ which is defined by the inverse exponential map : v = exp−1 ( z ( θ , r ) ) with z ( θ , r ) being a point in the local geodesic polar coordinate system at x ∈ M with coordinates ( θ , r ) . With this , we can easily interpret ACNN within the framework of ( 2 ) . Indeed , ACNN essentially chooses ~ux as the directions of the principal curvature at point x . For GCNN , on the other hand , it avoids choosing a specific vector field on the manifold by taking max-pooling among all possible directions of ~u1x at each point . Such definition of convolution , however , ignores the correspondence of the convolution kernels at different locations . The newly proposed PTC Schonsheck et al . ( 2018 ) defines convolution directly on the manifold , while using tangent planes to transport kernels by a properly chosen parallel transport . PTC can be equivalently cast into the form of ( 2 ) using the inverse exponential map , and implementation of the proposed parallel transported is realized through choosing specific vector fields { ~ujx } x∈M , j = 1 , 2 guided by a Eikonal equation for transforming vectors along geodesic curves on manifolds . Spatial point-based methods have wider applications due to their weaker assumptions on the data structure , a point cloud P = { xi ∈ R3 : i = 1 , . . . , N } ⊂ R3 consists of points in a 3-dimensional Euclidean space with the coordinates of the points as the only available information . Manifolds can be approximated by point clouds via sampling . Computing k-nearest neighborhoods of x or neighbors within a fixed radius can easily convert a point cloud to a local graph or mesh . This is why point cloud is simple , flexible and attracts much attentions lately . There are mainly two types of point-based convolution . The first type is to combine the information of points directly . These methods can be formulated as ( f ∗ k ) ( xi ) : = ∑ xj∈N ( xi ) k ( xi , xj ) f ( xj ) , ( 4 ) where N ( xi ) ⊂ P is a neighborhood of xi and kernel k takes different forms in different methods . PointNet Qi et al . ( 2017a ) is an early attempt to extract features on point cloud . PointNet is a network structure without convolution , or alternatively we can interpret the convolution defined by PointNet has the simplest kernel k ( xi , xj ) = δ ( xi , xj ) where δ is the Kronecker-Delta . Various later works attempt to improve PointNet by choosing different forms of the kernel k. For example , PointNet++ Qi et al . ( 2017b ) introduces a max pooling among local points , i.e . choosing kernel k as an indicator function : k ( xi , xj ) = Ixj=arg maxz∈N ( xi ) f ( z ) . Pointcnn Li et al . ( 2018b ) chooses k ( xi , · ) = β ′Axi where β ∈ RK and Axi ∈ RK×K are trainable variables with K = |N ( xi ) | . The second type of convolution is defined by first projecting the point cloud locally on an Euclidean domain and then employ regular convolution . This type of methods can also be formulate as ( 2 ) . For example , Tangent convolutions Tatarchenko et al . ( 2018 ) define kernels on the tangent plane , and use 2 principal directions of a local PCA as ~ux . Pointconv Wu et al . ( 2018 ) constructs local kernels by interpolation in R3 , i.e . letting φ ( x , v ) = x − v which is essentially a local Euclidean convolution . 1.2 THE PROPOSED CONVOLUTION : NPTC . In this paper , we propose Narrow-Band Parallel Transport Convolution ( NPTC ) , which is a geometric convolution based on point cloud discretization of a manifold parallel transport defined in a specific way . As we observed in the previous section , convolutions in many methods can be written in the form of ( 2 ) and ( 3 ) , while the differences mostly lie in the choices of the vector field { ~ux } x∈M . As observed by Schonsheck et al . ( 2018 ) that choosing the vector field properly , the associated convolution can be interpreted as parallel transporting the kernels using the parallel transport associated to the prescribed vector field . We propose to define convolution on point clouds by combining narrow-band approach Adalsteinsson & Sethian ( 1995 ) and geometric convolution . The formal definition of parallel transport and general ideal will presented in Section 2 and detailed descriptions on NPTC will be given in Section 3 . Now , we describe how NPTC is computed on point clouds . Firstly , point clouds are approximated by voxel narrow-band with appropriate resolution . For a manifold M or point cloud P , its narrow-band is defined as the region with distance from M ( or P ) less than : NB ( M ) : = { x ∈ R3|dist ( x , M ) < } , where dist represents a distance function . For calculation efficiency and robustness to noise , we choose voxel narrow-band approximation . It should be noted that the result of the traditional voxelization method Wu et al . ( 2015 ) is a solid 3-d structure and the voxel narrow-band can be understood as the ” shell ” with a certain thickness of the former ( if the point cloud is sampled from the surface of the object ) . It is very efficient to calculate the distance function in voxel approximation with appropriate resolution . The vector field { ~ux } x∈M is defined as the projections of the gradient field of a narrow-band-based distance function on the approximated tangent plane of the point cloud . Such definition of the vector field is robust to noise , since if the distortion of the coordinates of the points by noise does not exceed the width of the narrow-band , the computed gradient field of the distance function in the narrow-band remains unchanged . After the vector field { ~ux } x∈M , we use local PCA to estimate normal vectors on the point cloud following Xu et al . ( 2018 ) . Finally , the convolution kernel can be constructed on the tangent plane and the corresponding f ( v ) can be obtained by interpolating the value f ( x ) on the point cloud . Note that we prefer to use geometric convolution in NPTC because compared with methods that translate kernels in the ambient space of the manifold , NPTC translates kernels on the tangent planes which effectively avoids having convolution kernels defined away from the underlying manifold of the point cloud . In other words , NPTC can well reflect point cloud geometry and is a natural generalization of planar convolution in the sense that when the point cloud reduces to planar grids , the NPTC reduces to the planar convolution .

Summary: This paper proposes Narrow-Band Parallel Transport Convolution (NPTC) for point cloud data. The general idea is to use gradients of some distance function to define the vector field. The authors use voxelization to approximate the point cloud in a narrow-band covering the point cloud so that distance function can be calculated. The authors also discuss how to compute the distance function and the vector fields on point clouds. Experimental results on classification tasks and segmentation tasks are provided.