NPTC-net: Narrow-Band Parallel Transport Convolutional Neural Network on Point Clouds

1 INTRODUCTION . Convolution is one of the most widely used operators in applied mathematics , computer science and engineering . It is also the most important building block of Convolutional Neural Netowrks ( CNNs ) which are the main driven force in the recent success of deep learning LeCun et al . ( 2015 ) ; Goodfellow et al . ( 2016 ) . In the Euclidean space Rn , the convolution of function f with a kernel ( or filter ) k is defined as ( f ∗ k ) ( x ) : = ∫ Rn k ( x− y ) f ( y ) dy . ( 1 ) This operation can be easily calculated in Euclidean spaces due to the shift-invariance of the space so that the translates of the filter k , i.e . k ( x−y ) is naturally defined . With the rapid development of data science , more and more non-Euclidean data emerged in various fields including network data from social science , 3D shapes from medical images and computer graphics , data from recommending systems , etc . Therefore , geometric deep learning , i.e . deep learning of non-Euclidean data , is now a rapidly growing branch of deep learning Bronstein et al . ( 2017 ) . In this paper , we will discuss how we can generalize the definition of convolution to ( manifold-structured ) point clouds in a way that it inherits desirable properties of the planar convolution , thus it enables to design convolutional neural networks on point clouds . One of the main challenges of defining convolution on manifolds and point clouds ( a discrete form of manifolds ) is to define translation x− y on the non-Euclidean domain . Other than convolutions , we also need to properly define pooling to enable networks to extract global features and to save memory during training . Multiple types of generalization of convolutions on manifolds , graphs and point clouds have been proposed in recent years . We shall recall some of them and discuss the relation between existing definitions of convolutions and the proposed narrow-band parallel transport convolution . 1.1 RELATED WORK OF CONVOLUTIONS ON NON-EUCLIDEAN DOMAINS . Spectral methods avoid the direct definition of translation x− y by utilizing the convolution theorem : for any two functions f and g , f̂ ∗ g = f̂ · ĝ . Therefore , we have f ∗ g = ( f̂ · ĝ ) ∨ , where ∧ and ∨ represent generalized Fourier transform and inverse Fourier transform provided through the associated Laplace-Beltrami ( LB ) eigensystem on manifolds . To avoid computing convolution through full eigenvalue decomposition , polynomial approximation is proposed and yields convolution as action of polynomials of the LB operator Hammond et al . ( 2011 ) ; Dong ( 2017 ) . Thus , convolutional neural networks can be designed Bruna et al . ( 2014 ) ; Defferrard et al . ( 2016 ) ; Levie et al . ( 2019 ) . Spectral methods , however , suffer two major drawbacks . First , these methods define convolution in the frequency domain . As a result , the learned filters are not spatially localized . Secondly , spectral convolutions are domain-dependent as deformation of the ground manifold will change the corresponding LB eigensystem . This obstructs the use of learning networks from one training domain to a new testing domain Schonsheck et al . ( 2018 ) . Spatial mesh-based methods are more similar to the Euclidean case , and this is one of the reasons why most of the existing works fall into this category . The philosophy behind these methods is that the tangent plane TxM of a 2-dimensional manifold M in each point x ∈ M is embedded to a 2-dimensional Euclidean domain where convolution can be easily defined . In this paper , we make the first attempt to interpret some of the existing mesh-based methods in a unified framework . We claim that most of the spatial mesh-based methods can be formulated as ( f ∗ k ) ( x ) : = ∫ Tx , M k ( φ ( x , v ) ) f ( v ) dv , x ∈M , v ∈ TxM . ( 2 ) Here , k : R2 → R is a convolution kernel and Tx , M = { v ∈ TxM : 〈v , v〉gx ≤ 2 } with > 0 being the size of the kernel . The mapping φ ( x , v ) : M×TxM→ R2 is defined as φ ( x , v ) = ( α1 , α2 ) = ( 〈 v , ~u1x 〉 gx , 〈 v , ~u2x 〉 gx ) , ( 3 ) where ~ujx ∈ TxM , j = 1 , 2 . For simplicity , we will denote ~ux = ( ~u1x , ~u2x ) . Most of the designs of the existing manifold convolutions focused on the designs of ~ux . We remark that possible singularities will lead to no convolution operation at those points . These are isolated points on a closed manifold and for more discussion of singularities , please refer to the Appendix . For example , GCNN Masci et al . ( 2015 ) and ACNN Boscaini et al . ( 2016 ) construct a local geodesic polar coordinate system on a manifold , formulating the convolution as ( f ∗ k ) ( x ) = ∫ k ( ( φ ◦ ψ ) ( θ , r ) ) ( Qxf ) ( r , θ ) drdθ , whereQ is a local interpolation function with interpolation domain an isotropic disc for GCNN and an anisotropic ellipse for ACNN . A local geodesic polar coordinate system on a manifold can also be transformed to a 2-dimensional planar coordinate system on its tangent plane . Such transformation is the mapping ψ which is defined by the inverse exponential map : v = exp−1 ( z ( θ , r ) ) with z ( θ , r ) being a point in the local geodesic polar coordinate system at x ∈ M with coordinates ( θ , r ) . With this , we can easily interpret ACNN within the framework of ( 2 ) . Indeed , ACNN essentially chooses ~ux as the directions of the principal curvature at point x . For GCNN , on the other hand , it avoids choosing a specific vector field on the manifold by taking max-pooling among all possible directions of ~u1x at each point . Such definition of convolution , however , ignores the correspondence of the convolution kernels at different locations . The newly proposed PTC Schonsheck et al . ( 2018 ) defines convolution directly on the manifold , while using tangent planes to transport kernels by a properly chosen parallel transport . PTC can be equivalently cast into the form of ( 2 ) using the inverse exponential map , and implementation of the proposed parallel transported is realized through choosing specific vector fields { ~ujx } x∈M , j = 1 , 2 guided by a Eikonal equation for transforming vectors along geodesic curves on manifolds . Spatial point-based methods have wider applications due to their weaker assumptions on the data structure , a point cloud P = { xi ∈ R3 : i = 1 , . . . , N } ⊂ R3 consists of points in a 3-dimensional Euclidean space with the coordinates of the points as the only available information . Manifolds can be approximated by point clouds via sampling . Computing k-nearest neighborhoods of x or neighbors within a fixed radius can easily convert a point cloud to a local graph or mesh . This is why point cloud is simple , flexible and attracts much attentions lately . There are mainly two types of point-based convolution . The first type is to combine the information of points directly . These methods can be formulated as ( f ∗ k ) ( xi ) : = ∑ xj∈N ( xi ) k ( xi , xj ) f ( xj ) , ( 4 ) where N ( xi ) ⊂ P is a neighborhood of xi and kernel k takes different forms in different methods . PointNet Qi et al . ( 2017a ) is an early attempt to extract features on point cloud . PointNet is a network structure without convolution , or alternatively we can interpret the convolution defined by PointNet has the simplest kernel k ( xi , xj ) = δ ( xi , xj ) where δ is the Kronecker-Delta . Various later works attempt to improve PointNet by choosing different forms of the kernel k. For example , PointNet++ Qi et al . ( 2017b ) introduces a max pooling among local points , i.e . choosing kernel k as an indicator function : k ( xi , xj ) = Ixj=arg maxz∈N ( xi ) f ( z ) . Pointcnn Li et al . ( 2018b ) chooses k ( xi , · ) = β ′Axi where β ∈ RK and Axi ∈ RK×K are trainable variables with K = |N ( xi ) | . The second type of convolution is defined by first projecting the point cloud locally on an Euclidean domain and then employ regular convolution . This type of methods can also be formulate as ( 2 ) . For example , Tangent convolutions Tatarchenko et al . ( 2018 ) define kernels on the tangent plane , and use 2 principal directions of a local PCA as ~ux . Pointconv Wu et al . ( 2018 ) constructs local kernels by interpolation in R3 , i.e . letting φ ( x , v ) = x − v which is essentially a local Euclidean convolution . 1.2 THE PROPOSED CONVOLUTION : NPTC . In this paper , we propose Narrow-Band Parallel Transport Convolution ( NPTC ) , which is a geometric convolution based on point cloud discretization of a manifold parallel transport defined in a specific way . As we observed in the previous section , convolutions in many methods can be written in the form of ( 2 ) and ( 3 ) , while the differences mostly lie in the choices of the vector field { ~ux } x∈M . As observed by Schonsheck et al . ( 2018 ) that choosing the vector field properly , the associated convolution can be interpreted as parallel transporting the kernels using the parallel transport associated to the prescribed vector field . We propose to define convolution on point clouds by combining narrow-band approach Adalsteinsson & Sethian ( 1995 ) and geometric convolution . The formal definition of parallel transport and general ideal will presented in Section 2 and detailed descriptions on NPTC will be given in Section 3 . Now , we describe how NPTC is computed on point clouds . Firstly , point clouds are approximated by voxel narrow-band with appropriate resolution . For a manifold M or point cloud P , its narrow-band is defined as the region with distance from M ( or P ) less than : NB ( M ) : = { x ∈ R3|dist ( x , M ) < } , where dist represents a distance function . For calculation efficiency and robustness to noise , we choose voxel narrow-band approximation . It should be noted that the result of the traditional voxelization method Wu et al . ( 2015 ) is a solid 3-d structure and the voxel narrow-band can be understood as the ” shell ” with a certain thickness of the former ( if the point cloud is sampled from the surface of the object ) . It is very efficient to calculate the distance function in voxel approximation with appropriate resolution . The vector field { ~ux } x∈M is defined as the projections of the gradient field of a narrow-band-based distance function on the approximated tangent plane of the point cloud . Such definition of the vector field is robust to noise , since if the distortion of the coordinates of the points by noise does not exceed the width of the narrow-band , the computed gradient field of the distance function in the narrow-band remains unchanged . After the vector field { ~ux } x∈M , we use local PCA to estimate normal vectors on the point cloud following Xu et al . ( 2018 ) . Finally , the convolution kernel can be constructed on the tangent plane and the corresponding f ( v ) can be obtained by interpolating the value f ( x ) on the point cloud . Note that we prefer to use geometric convolution in NPTC because compared with methods that translate kernels in the ambient space of the manifold , NPTC translates kernels on the tangent planes which effectively avoids having convolution kernels defined away from the underlying manifold of the point cloud . In other words , NPTC can well reflect point cloud geometry and is a natural generalization of planar convolution in the sense that when the point cloud reduces to planar grids , the NPTC reduces to the planar convolution .

Summary: The authors propose NPTC a new convolutional operator for 3D point clouds embedded on a 2D manifold based no parallel transport defined by a narrow-band approximation. The method combines voxelization within a local neighborhood in 3D (narrow band) and geometric convolution. Experimental results are presneted for a range of classification and segmentation tasks that are similar to the state-of-the-art.