NPTC-net: Narrow-Band Parallel Transport Convolutional Neural Network on Point Clouds
Authors: Pengfei Jin
Tianhao Lai
Rongjie Lai
Bin Dong
Venue:  (2020)
Keywords: geometric convolution
point cloud
parallel transport

Abstract: Convolution plays a crucial role in various applications in signal and image processing, analysis and recognition. It is also the main building block of convolution neural networks (CNNs). Designing appropriate convolution neural networks on manifold-structured point clouds can inherit and empower recent advances of CNNs to analyzing and processing point cloud data. However, one of the major challenges is to define a proper way to "sweep" filters through the point cloud as a natural generalization of the planar convolution and to reflect the point cloud's geometry at the same time. In this paper, we consider generalizing convolution by adapting parallel transport on the point cloud. Inspired by a triangulated surface based method \cite{DBLP:journals/corr/abs-1805-07857}, we propose the Narrow-Band Parallel Transport Convolution (NPTC) using a specifically defined connection on a voxelized narrow-band approximation of point cloud data. With that, we further propose a deep convolutional neural network based on NPTC (called NPTC-net) for point cloud classification and segmentation. Comprehensive experiments show that the proposed NPTC-net achieves similar or better results than current state-of-the-art methods on point clouds classification and segmentation.

Under review as a conference paper at ICLR 2020
NPTC-NET: NARROW-BAND PARALLEL TRANSPORT CONVOLUTIONAL NEURAL NETWORK ON POINT CLOUDS
Anonymous authors Paper under double-bling blind review
ABSTRACT
Convolution plays crucial role in various applications in signal and image pro- cessing, analysis and recognition. is also the main building block of convolu- tion neural networks (CNNs), Designing appropriate convolution neural networks point clouds can inherit and empower recent advances of CNNs analyzing and processing point cloud data. However, one of the major challenges is to define proper way to "sweep" filters through the point cloud as a natural generalization of the planar convolution and to reflect the point cloud's s geometry at the same time. In this paper, we consider generalizing convolution by adapting parallel transport on the point cloud. Inspired by a triangulated sur- face based method Scchonsheck et al. 2018), we propose the Narrow-Band Par- allel Transport Convolution (NPTC) using a specifically defined connection on a voxelized narrow- banddd--dddd- approximation of point cloud data. With that, further propose a deep convolutional neural network based on NPTC (called NPTC-net) for point cloud classification and segmentation. Comprehensive experiments show that the proposed NPTC-net achieves similar or hetter results than current state- of-the-art methods on point clouds classification and segmentation.
INTRODUCTION
Convolution is one of the most widely used operators in applied mathematics, computer science and engineering. It is also the most important building block of Convolutional Neural Netowrks (CNNs) which are the main driven force in the recent success of deep learning LeCun et al. (2015); Goodfellow et al. (0016).
In the Euclidean space IR,", the convolution of function f with kernel (or filter) k is defined as
(f (f) = /
- (2))
This operation be easily calculated in Euclidean spaces due the shift-invariance of the space so that the translates of the filter k. i.e. k(x-y) naturally defined. With the rapid development of data science, more and more Bunl-uuiddann data emerged in various fields including network data from social science, 3D shapes from medical images and computer graphics, data from recommending systems, etc. Therefore, geometric deep learning, i.e. deep learning of non-Euclidean data, is now rapidly growing branch of deep learning Bronstein et al. (017). In this paper, we will discuss how we can generalize the definition of convolution mnanifold-structrre inherits desirable properties of the planar convolution, thus it enables to design convolutional neural networks on point clouds.
point clouds in way that
One of the main challenges of defining convolution manifolds and point clouds (a discrete form of manifolds) is to define translation x - y on the non-Euclidean domain. Other than convolutions, we also need to properly define pooling to enable networks to extract global features and to save memory during training. Multiple types of generalization convolutions on manifolds, graphs and point clouds have been proposed in recent years. We shall recall some of them and discuss the lation between existing definitions of convolutions and the proposed narrow- band parallel transport convolution.

---

Under review as a conference paper at ICLR 2020
1.1 RELATED WORK OF CONVOLUTIONS ON CON-BUCLIDEAN DOMAINS
Spectral methods avoid the direct definition of translation x - by utilizing the convolution theo- rem: for any two functions and g. f f 9- Therefore, we have f g (f 9)", where A and V represent generalized Fourier transform and inverse Fourier transform provided through the asso- ciated Laplace- -eltrrcmmi (LB) eigensystem on manifolds. To avoid computing convolution through full eigenvalue decomposition, polynomial approximation is proposed and yields convolution as ac- tion of polynomials of the LB operator Hammond al. (201T): Dong (0077) Thus, convolutional neural networks can be designed Bruna et al. (2014); Defferrard et al. (2016); Levie et al. (2019). Spectral methods, however, suffer two major drawbacks. First, these methods define convolution in the frequency domain. As a result, the learned filters are not spatially localized. Secondly, spectral convolutions are domain-dependent as deformation of the ground manifold will change the corre- sponding LB eigensystem. This obstructs the use of learning networks from one training domain to a new testing domain Schonsheck et al. (2018).
Spatial mesh-based methods are more similar to the Euclidean case, and this is one of the reasons why most of the existing works fall into this category. The philosophy behind these methods that the tangent plane TyM of 2-dimensional manifold M in each point z M is embedded to 2-dimensional Euclidean domain where convolution can be casily defined. In this paper, we make the first attempt to interpret some of the existing mesh-based methods in unified framework. We claim that most of the spatial mesh- methods can be formulated as
(f (f*k)] [x)) = Sr.. k(ő(1, ())(())))((tdd. MEMUUET.NN T.M.
(2)
Here, k: R22 + R is a convolution kernel and Ta,cM = E TaM: : (1, v) go S { with > 0
being the size of the kernel. The mapping 6(x, v) M X TaM R2 is defined as
61z, v) (ar1, (v, 9= (v, uht)
where i2. E T=MMj== 1,2. For simplicity, we will denote d.z (i (. Most of the designs of the existing manifold convolutions focused on the designs of ug.. We remark that possible singu- larities will lead to convolution operation those points. These are isolated points on closed manifold and for more discussion of singularities, please refer to the Appendix.
For example, GCNN Masci al. (2015) and ACNN al. (2016) construct local geodesic polar coordinate system on a manifold, formulating the convolution as
/ l: (
* kx))
o (0,r)) (Q.f) 0)drdd,
where Q is a local interpolation function with interpolation domain an isotropic disc for GCNN and an anisotropic ellipse for ACNN. A local geodesic polar coordinate system manifold can also be transformed to a 2-dimensional planar coordinate system on its tangent plane. Such transformation the mapping ve which is defined by the inverse exponential map: being a point in the local geodesic polar coordinate system at x E .M with coordinates (0, With this, we can easily interpret ACNN within the framework of (2). Indeed, ACNN essentially chooses Vr as the directions of the principal curvature at point r. For GCNN, on the other hand, it avoids choosing a specific vector field on the manifold by taking max-pooling among all possible directions of ui_ at each point. Such definition of convolution, however, ignores the correspondence of the convolution kernels at different locations.
exp (=(00 r)) with z(0, r)
The newly proposed PTC Schonsheck et al. (2018) defines convolution directly on the manifold, while using tangent planes to transport kernels by a properly chosen parallel transport. PTC can be cquivalently cast into the form of (2) using the inverse exponential map, and implementation of the proposed parallel transported is realized through choosing specific vector fields (w)) J = 1,2 guided by Eikonal equation for transforming vectors along geodesic curves on manifolds.
Spatial point- based methods have wider applications due to their weaker assumptions on the data structure, a point cloud P {z, R Euclidean space with the coordinates of the points as the only available information. Manifolds can be approximated by point clouds via sampling. Computing k-nearest neighborhoods of or
1, _,....N] .N) R consists of points in 3-dimensional
2

---

Under review as a conference paper at ICLR 2020
neighbors within a fixed radius easily convert point cloud to local graph or mesh. This is why point cloud is simple, flexible and attracts much attentions lately.
There are mainly two types of point- point-based convolution. The first type is to combine the information of points directly. These methods can be formulated as
(S * (z.)
Σ ((bx) (xE)(())
where (xx,)) c P neighborhood of z; and kernel ls takes different forms in different methods. PointNet Qi et al. (2017a) is an early attempt to extract features on point cloud. PointNet is network structure without convolution, or alternatively we can interpret the convolution defined by PointNet has the simplest kernel k kz) attempt to improve PointNet by choosing different forms of the kernel k. For example, PointNet++ Qi et al. (2017b) introduces max pooling among local points, i.c. choosing kernel k as an indicator function: k(rr,, 2z) = 1, =arg
8(2;; 2;) where 5 is the Kronecker- Delta. Various later works
=arg max f(=)- Pointenn Li et al. 2018b) chooses k(x;,)) = 3 As,
where B E IR and Az, E R are trainable variables with K
(x+)).
The second type of convolution is defined by first projecting the point cloud locally on an Euclidean domain and then employ regular convolution. This type of methods can also be formulate ( For example. Tangent convolutions Tatarchenko et al. (2018) define kernels on the tangent plane, and use 2 principal directions of a local PCA as ugz Pointconv Wu et al. (2018) constructs local kernels by interpolation in R3, i.e. letting ((x. v) = which is essentially local Euclidean convolution.
1.2 THE PROPOSED CONVOLUTION: NPTC
In this paper, we propose Narrow-Band Parallel Transport Convolution (NPTC), which is a ge- ometric convolution based on point cloud dis- cretization a manifold parallel transport de- fined in a specific way. As observed in the previous section, convolutions in many meth- ods can be written in the form of (2) and (3), while the differences mostly lie in the choices of the vector field (T, J.c.M As observed by (Schonsheck et al. (2018) that choosing the vector field properly, the associated convolu- tion can be interpreted as parallel transporting the kernels using the parallel transport associ- ated to the prescribed vector field. We propose to define convolution on point clouds by com- bining narrow-band approach Adalsteinsson & Sethian (1955) and geometric convolution. The formal definition of parallel transport and gen- eral ideal will presented in Section 2 and de- tailed descriptions on NPTC will be given in Section 3.
Figure 1: Narrow-band parallel transport convo- lution on point cloud: black and blue points are sample points of the surface (blue). Each kernel (colored dots) is defined on the tangent plane (yel- low). The vectors on the tangent planes that are of the same color are in a parallel fashion.
Now, we describe how NPTC is computed on point clouds. Firstly, point clouds are approxi- mated by voxel nand with appropriate resolution. For a manifold M or point cloud P, its narrow-band is defined as the region with distance from M (or P) less than e: NBM(M {z E R3]dist(z, (x,ist(z,,)) M) < e}. where dist represents a distance function. For calculation efficiency and robustness to noise, we choose voxel narrow-band approximation. It should be noted that the result of the traditional voxelization method Wu et al.] (2015) is solid 3-d structure and the voxel nardow-band can be understood as the "shell" with a certain thickness of the former (if the point cloud is sampled from the surface of the object). It is very efficient to calculate the distance func- tion in voxel approximation with appropriate resolution. The vector field {ūz IzeM is defined as the projections of the gradient field of tangent plane of the point cloud. Such definition of the vector field is robust to noise, since the distortion of the coordinates of the points by noise does not exceed the width of the narrow-band, the
wrrroww-andd-asseee distance function
the approximated
3

---

Under review as a conference paper at ICLR 2020
computed gradient field of the distance function in the narrow-band remains unchanged. After the vector field { Ug ZeEM. we use local PCA to estimate normal vectors on the point cloud following Xu et al. (0118) Finally, the convolution kernel can be constructed the tangent plane and the corresponding (t) can be obtained by interpolating the value fa) on the point cloud.
Note that prefer to use geometric convolution in NPTC because compared with methods that translate kernels in the ambient space of the manifold, NPTC translates kernels on the tangent planes which effectively avoids having convolution kernels defined away from the underlying manifold of the point cloud. In other words, NPTC can well reflect point cloud geometry and is natural generalization of planar convolution in the sense that when the point cloud reduces to planar grids, the NPTC reduces to the planar convolution.
1.3 CONTRIBUTIONS
We introduce a new point cloud convolution, NPTC, based on parallel transport defined by a narrow-band approximation of the point cloud. The proposed NPTC is natural generalization of planar convolution.
The proposed NPTC combines voxelization and geometric convolution. Voxelization with appropriate resolution brings robustness and geometric convolution can better reflect point cloud's geometry.
Based NPTC, we designed convolutional neural networks, called NPTC-net, for point clouds classification and segmentation with state-of of-the-arrt performance.
2 BACKGROUND AND GENERAL IDEA
We recommend the book Nakahara 2003) for readers who are unfamiliar with differential geometry. short intuitive introduction to parallel transport will be provided after recalling the notion.
Let M be a 2 dimensional differential manifold embeded in R$ and T.M be the tangent plane at point x M. T.M can be defined as the 2-dimensional linear space formed by the span of tangent vectors. The disjoint union of the tangent planes at each point on the manifold defines the tangent bundle TMM. vector field is section of T.M. Collection of all smooth vector fields X(x) E T,M, Vr E M is denoted as MTT.). An affine connection is bilinear mapping V: T(TM) T(TM) > M(TM). A section of a vector field called parallel along a curve ? V = 0. Suppose we are given an vector E E Ta at zo (0) E M. The parallel transport of é Falong 7 is the extension of ē' to a parallel section X on 7.
Intuitively, in differential geometry parallel transport is way of translating vector along smooth curve "parallel" to the original. In fact, by given smooth vector fields {ū .7 we can define lin- transformation among tangent planes then the corresponding parallel transport can be induced Knebelman 1955]) Therefore, parallel transport defined in this way can be viewed as parallel trans- porting the vector the manifold with respect to the connection that is reconstructed from the vector field { F That is to say, if given a vector field, we can define a way to "transport" vectors between tangent planes.
To select suitable vector field, we first recall the choice of the vector fields of PTC which defines convolution on triangulated surfaces via parallel transport with respect the Levi-Civita connection Schonsheck et al. (2018). Geodesic curve represents, in some sense, the "straightest" path between two points on Riemannian manifold. Given a geodesic connecting two points and y. the tangen- tial direction at r corresponds to the ascent direction of geodesic distance from y. That is to say, all tangential directions of one geodesic form a parallel vector field. PTC chooses such direction as at! and defines i = v7) fl with Tez the normal vector at z. One naive approach to extend mesh based methods to point cloud is to generate a triangulated surface based on the point cloud. However, this is not convenient as working directly with the point cloud since in practice not every point cloud corresponds to a legitimate parameterized surface and pooling is not as easy to implement on triangulated surfaces point clouds. When applied in practice, mesh construction time may be much longer than inference time of some methods directly applied on the point cloud.
point clouds. In addition, it is tone-conuuming to construct meshes
4

---

Under review as a conference paper at ICLR 2020
If we want to construct vector field on point cloud, gradients of some sort of distance function can be a good choice. However, unlike triangulated surfaces, distance function is not easily defined on point clouds due to the lack of connectivity. It is then natural to approximate the point cloud with another data structure with connectivity, so that distance function can be easily calculated. We use voxel to approximate the point cloud in narrow-band in IR3 covering the point cloud. We denote such distance function as R + Rt.
Then, we review how a distance function is calculated. Distance functions can be easily computed by solving the Eikonal equation. The Eikonal equation is a non-linear partial differential equation describing wave propagation:
= V ZEΩ, =
(5)
where A C C R and h(x)) is strictly positive function. The solution p(z) of 55 can be viewed as the shortest time needed to travel from to A with h(x) being the speed of the wave at æ. For the special case when h = the solution p(x) represents the distance from x to A limited The Eikonal equation can be solved by the fast marching method method|Sethian (996)).
Ω.
Note that if the point cloud is sampled from a plane, the narrow-band is flat as well. Then, by a proper choice of the distance function, the vector fields huEMM 1,2, can be reduced to the global coordinate {ē, ) = 1,2 2 on the plane. This means that NPTC reduced to the traditional planar convolution.
Once the distance function is computed, we choose itt Vpp(z), where Vpp(2) is a projection approximated tangent plane at œ. Then, th can be caleulated by the outer product W2 T x Mh with 7i. the normal vector at s.
of V
p(x)
For convenience, instead of transporting the kernel the manifold, parallel transported kernel at every point by formulating k as k(ó(z,)) where Φ(2, v) is defined 33. is known in differential geometry that transporting the kernel every point on the manifold is the same as locally reconstructing the kernel in the aforementioned way.
locally construct the
3 NARROW-BAND PARALLEL TRANSPORT CONVOLUTION (NPTC) AND NETWORK DESIGN
3.1 IMPLEMENTATION OF NPTC
Note that some methods of voxelization, interpolation ,ad initial point selection are not essential. These can be replaced depending on the accuracy, speed requirements and datasets.
(a) narrow-bane band
(b) cross section
(c) Distance function p
(d) {
Figure 2: Illustration of a point cloud P sampled from the unit sphere. (a) shows the narrow-band approximation (blue boxes) of part of P (in red). (b) is cross section of (a). (c), (d) show the distance function o and vector field {w',) ([Vpp(x))) on the point cloud. We can see that distance propagates from the bottom center to the top center reflecting the geometry of the sphere.
3.1.1 VOXELIZATION AND COMPUTING DISTANCE FUNCTION
Assuming that the point cloud has been normalized so that it is enclosed by the unit cube. The unit cube is then discretized by N x N N grids. The narrow-band approximation of the point cloud is constructed by including each voxel with a distance less than & from any point on the point cloud. Then the voxel narrow-band approximation of point cloud has been constructed. N depends on the
5

---

Under review as a conference paper at ICLR 2020
requirements of speed and accuracy. k need to be large enough to make the narrow-band continuous (if the point cloud represents a continuous manifold) and small enough to make the distance in narrow-band approximate the geodesic distance. In the next experiments, we choose N 100, & as 2 and T as distance function. the sampling of point cloud is very sparse, it may be necessary to construct harrow-hand by the estimation of the normal or some other strategies.
Then we can compute distance functions on the voxels using the well-known fast marching method based on regular grid provided by the voxelization Sethian 1996). The solution p(x)) of the Eikonal equation p)(x) presents the distance form A to æ limited inside the narrow-band. Here A is chosen certain point on the point cloud.
3.1.2 COMPUTING THE VECTOR FIELDS ON POINT CLOUDS
We first compute the tangent plane on each point. Tangent planes are important features of manifolds and have been well-studied in the literature Lai et al. (2013). In this paper, we use local principal component analysis (LPCA) to estimate the tangent plane. We estimate the local linear structure near E P using the covariance matrix
(ax - c) Txx c), (x)
Σ
Σ Fke
c= k
ILEN(())
where (xx) is the set neighboring points of x. The eigenvectors of the covariance matrix form orthogonal basis. If the point cloud is sampled from two dimensional manifold, and the local sampling is dense enough to resolve local features, the eigenvectors corresponding to the largest two eigenvalues provide the two orthogonal directions of the tangent plane, and the remaining vector represents the normal direction at E P. Here, we denote the space spanned by the two eigenvectors of the covariance matrix as T.P R3
With the computed distance function p(z), it is natural to define the vector field by projecting Vp the approximated tangent planes of the point cloud. Given a point Ty P close enough to x E P, we have
(Vp Fy - x) = p(xx) -xx))
where p(Tx.) and p(x) are known. If we consider k-nearest neighbors of x, we have k - equations with unknowns that the three components of Vp. We can use least squares to find V p. We then project the vector V p((x) onto the tangent plane at æ. We denote the projected vector V Ppp. which is the vector we eventually need to define NPTC.
After defining the vector field on the tangent plane, we choose the convolution kernel of hexagon (Figure D. that is, convolution kernel composed of central point and six evenly distributed points around. The value f1) is computed by f() f()) where P is the closest point to in R Note that one may use a more sophisticated method to compute f(v) rather than using the closest point interpolation. We choose the closest point interpolation because of its simplicity.
3.2 NPTC-NET: ARCHITECTURE DESIGN FOR CLASSIFICATION AND SEGMENTATION
This section, we present how to use NPTC to design convolutional neural networks on point clouds for classification and segmentation tasks. For that, other than the NPTC, we need to define some other operations that are frequently used in neural networks. Note that, some point-wise operations like MLP and ReLu are the same on point cloud as the Euclidean case. Here, we only focus on operations that are not readily defined point clouds.
Down-sampling: In our implementation, the sab-se set of points of the next layer is generated by the farthest point sampling Eldar et al. (1997).
Convolution layer: Our k-th convolution layer takes points ph N and their corresponding feature maps F I xet as input, where N is the number of the points and c the number of channels at layer k. The corresponding output is Fk+1 E 12 x²*+ x living on the points pk+1 R
p x3. The NPTC-net have encoding and decoding stages. Normally, N < N during encoding and N < N during decoding. Convolution at the k-th layer is only performed on the point pk+1, which resembles convolution with stride > 1 for planar convolutions.

---

Under review as a conference paper at ICLR 2020
One residual block takes the feature maps F RVXx the point set P R Nx3 as imput and same number of points and same number of channels of features as output. One residual layer consists of three components: MLP from channels to & channels, convolution layer from channels to 2 channels, MLP from channels to c channels plus the feature maps from the bypass connection. residual block consists of several residual layers.
Residual block:
NPTC-net consists of the aforementioned operations and its architecture given by Figure The left half of the NPTC-net is the encoder part of the network for feature extraction. For classification, features at the bottom of the network are directly attached to classification network; while for seg- mentation, features are decoded using the right half of the NPTC-net (decoder part of the network) to output the segmentation map.
#(input) 64
(64+128)
(claksess
(128+128)
128
128
128
128+1152
128
129
128
segementaion
1152
#(categorios)
classification
MLP Convolution with
Convolution
Residual block
Tile Top max pooling and concatenite
dovmsampling
Up sampling
Skio
Figure 3: Architecture of NPTC-net. The network for segmentation tasks is at the top right with encoding and the network for classification tasks is at the bottom right with encoding. Top 2 max pooling means taking the maximum two values among all the points in each channel and then catenate into one vector representing global feature. "Tile" means repeating the global feature and concatenating each point extra channels.
EXPERIMENTS
In order evaluate our new NPTC-nets, we conduct experiments on three widely used 3D datasets, ModelNet Wu et al. (2015), ShapeNet Yi et al. (016)). S3DIS Armeni et al. (2016). We imple- ment the model GTX TITAN Xp GPU. For more implementation details, please refer to the Appendix.
4.1 3D SHAPE CLASSIFICATION AND SEGMENTATION
We test the NPTC-net on ModelNet40 for classification tasks. ModelNet40 contains 12,311 CAD models from 40 categories with 9,842 samples for training and 2,468 samples for testing. For com- parison, we use the data provided by Qi et al. (2017a) sampling 2.048 points uniformly and com- puting the normal vectors from the mesh. As shown on Tabel [ our networks outperform other state-of-art methods. (If compared method has results on both 2048 (or 1024) and 5000 points, we only compare with the former.).
We also evaluate the NPTC-net on ShapeNet Part for segmentation tasks. ShapeNet Part contains 16,680 models from 16 shape categories with 14,006 for training and 2,874 for testing, each anno- tated with to 6 parts and there are 50 different parts in total. We follow the experiment setup of previous works, putting object category into networks known information. We point cenneectioo-ove-umiiiiii (IoU) to evaluate our NPTC-net. Table Thoows that our model ranks second on this dataset and is fairly close the best known result.

---

Under review as a conference paper at ICLR 2020
Table 1: Comparisons of overall accuracy (OA) and per-class accuracy (mA) ModelNet40
as well as comparisons in instance average IoU (mloU) and class average IoU (mcloU) on ShapeNet Part. Models ranking first colored in red and second in blue.
Modelnet40
ShapeNet part
Method
OA(%)
mA(%)
mloU
mcloU
kd-net Klokoy & Lempitsky (017))
91.8
88.5
82.3
77.4
pointnet Qi et al.
201 7
89.2
86.2
83.7
80.4
SO-Net
al.
2018a)
90.9
87.3
84.9
81.0
pointnet++ Qi et al.
(2017b)
90.7
85.1
81.9
SpecGCN Wang al. 2018a
92.1
85.4
SpiderCNNiXu ei al. (2018)
92.4
85.3
81.7
pointenn Li et al. (2018b)
92.2
88.1
86.1
84.6
DGCNN Wang et al.
(2018c)
92.2
90.2
85.1
82.3
Ours
92.7
90.2
85.8
83.3
4.2 RUBOSTNESS TEST
As pointed out earlier, using voxelization bring rubustness the definition of geometric volution. In order to show the robustness of our model for real data, we tested scene semantic segmentation on the "Stanford Large-Scale 3D Indoor Spaces Dataset" (S3DIS). S3DIS covers six large-scale indoor areas from 3 different buildings for total of 273 million points annotated with 13 classes. This is real-word scanned dataset without normal and with noise. Following Tchapmi et al. (2177). we advocate the use of Area-5 as test scene to better measure the generalization ability of our method. shows that geometric convolution methods are close not need normal estimation. NPTC-net outperforms the best known geometric convolution method TangentCon(Xu et al. 20018).
methods which do
Table Comparisons of overall accuracy (OA) and mean per-class IoU (mIoU) on S3DIS. Models ranking first is colored in red and second in blue.
Convolution Type
Method
OA(%)
mloU(%)
no convolution
pointnet Qi et (2017a)
78.8
41.3
SegCloud Tchapmi et al.
(2017)
48.9
3-d convolution
Eff3DConn Zhang ct al.
(0188)
69.3
51.8
ParamCony Wang
2018b)
58.3
TangentCony Xu et al.
(0188
82.5
52.8
geometric convolution
Ours
83.7
54.0
5 CONCLUSION
This paper proposed new way of defining convolution on point clouds, called the narrow-band parallel transport convolution (NPTC), based on a point cloud discretization of manifold parallel transport. The parallel transport was defined specifically by vector field generated by the gra- dient field of a distance function on a narrow-band approximation the point cloud. The NPTC was used to design convolutional neural network (NPTC-net) for point cloud classification and segmentation. Comparisons with -fate-of-the-at methods indicated that the proposed NPTC-net is competitive with the best existing methods.
REFERENCES
David Adalsteinsson and James A Sethian. A fast level method for propagating interfaces. Journal of computational physics, 118(2):269-277 1995.
I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis. M. Fischer, and S. Savarese. 3d semantic parsing of large-scale indoor spaces. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1534-1543 Los Alamitos, CA, USA, 2016. IEEE Computer Society.
8

---

Under review as a conference paper at ICLR 2020
doi: 10.1109/CVP..2016..70 URL httpL//t.///....eeecpperrrerree/ 11C9/CVPR. 2016.170
Davide Boscaini, Jonathan Masci, Emanuele Rodolå, and Michael Bronstein. Learning shape cor- respondence with anisotropic convolutional neural networks. In D. D. Lec, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), (. Advances in Neural Information Processing Systems 29,
pp. 3199-3197. Curran Associates, Inc., 2016. URL http: niw. http://ppprrr..ipppscccpaapppppe
6045-1ea ott.://a....a.....ooooeeeeeeeeea pdf
M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. Geometric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18 July 2017. ISSN 1053-5888. doi: 10. 10MMSS20222111
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann Lecun. Spectral networks and lo- cally connected networks on graphs In International Conference on Learning Representations (ICLR2014), CBLS, April 2014, 2014.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Gamett (eds.), Advances in Neural Information Processing Systems 29, pp. 11844-8522 Curran Associates, Inc., 2016. URL http: /ttp::/papparra n ww..pp...cppp....cccooooo cccccppperrr 6081 wtw.--ooooottiiinn.ooom - -etirral-eetttree networks on graphs with fast localized spectral- pooctral-filtere filterin pdf
Bin Dong. Sparse representation on graphs by tight wavelet frames and applications. Applied and Computational Harmonic Analysis, 42(3):452 479, 2017. ISSN 1063-5203. doi: https://doi.
0tg/10..0066//aaca22255........0 005. URL http: /tt///wwwwr www. www.ncedireecttee..oceceeeeeeeee wwt.coo//ceeeooe somnneciece//
Aaticiee/11/8110555001555500
Yuval Eldar, Michael Lindenbaum, Moshe Porat, and Yehoshua Y Zeevi. The farthest point strategy for progressive image sampling. IEEE Transactions on Image Processing, 6(9):1305-1315 1997.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
David K. Hammond, Pierre Vandergheynst, and Rémi Gribonval. Wavelets on graphs via spec- tral graph theory. Applied and Computational Harmonic Analysis, 30(2):129 150, 2011. ISSN 1063-5203. doi: httpa////ea....ceeee/////ttt/e//
URL http::/wwer
sci sciencedi rect cww. sc ence/
Sigurdur Helgason. Differential geometry, Lie groups, and symmetric spaces, volume 80. Academic
press, 1979.
R. Klokov and V. Lempitsky. Escape from cells: Deep kd-networks for the recognition of 3d point cloud models. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 862-872, Los Alamitos, CA, USA, oct 2017. IEEE Computer Society. doi: 10. 10.110///CV22017..9 URL https://doi.
MS Knebelman. Spaces of relative parallelism. Annals of Mathematics, pp. 387-399, 1951.
Rongjie Lai, Jiang Liang, and Hongkai Zhao. Zhao. A local mesh method for solving pdes point clouds. Inverse Prob. and Imaging, 7(3)7737-755 2013..
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):466, 2015.
R. Levie, F. Monti, X. Bresson, and M. M. Bronstein. Cayleynets: networks with complex rational spectral filters. IEEE Transactions on Signal Processing, 67(1): 97-109, Jan 2019. ISSN 1053-587X. doi: 10.1 109/TSP2200 200.TSSP.0018288992244
Graph convolutional neural
Jiaxin Li, Ben M. Chen, and Gim Hee Lee. So-net: Self-organizigg network for point cloud analysis. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018a.

---

Under review as a conference paper at ICLR 2020
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Con- volution on x-transformed points. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Cesa-Bianche Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 820-830. Curran Associates, Inc., 2018b. URL http: /ttp:://ppppas..mpm nips. cc cc/paper/ 1362-point.nnn-conne -ttp:ooooooiiiooooo-------------
formed- points.pdf...
Jonathan Masci, Davide Boscaini, Michael M. Bronstein, and Pierre Vandergheynst. Geodesic volutional neural networks on riemannian manifolds. The IEEE International Conference on Computer Vision (ICCV) Workshops, December 2015.
Mikio Nakahara. Geometry, topology and physics. CRC Press, 2003.
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett
(eds.), Advances in Neural Information Processing Systems 30, pp. 5099-5108.
Curran Associates, Inc., 2017b. URL http::/papers. wnw.siiisp............ cttpsccrcppp//////
7095
pdf
Stefan C. Schonsheck, Bin Dong, and Rongjic Lai. Parallel transport convolution: A new tool for convolutional neural networks on manifolds. CoRR, abs/1805.07857, 2018. URL http:
// //arxiv.org
07857
James A Sethian. A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Sciences, 9994)155 1996.
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian Yi Zhou. Tangent convolutions for dense prediction in 3d. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018
L. Tchapmi, Choy, I. Anneni, J. Gwak, and S. Savarese. Segeloud: Semantic segmentation of 3d point clouds. In 2017 International Conference on 31) Vision (3DV), pp. 537-547, Oct 2017. doi: 10.1109/3DV.2017.000
Chu Wang, Babak Samari, and Kaleem Siddiqi. Local spectral graph convolution for point set feature learning. In The European Conference on Computer Vision (ECCV), September 201 8a.
Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. Deep para- metric continuous convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018b.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph CNN for learning on point clouds. CoRR, abss/1801.0882, 2018c.
URL http: org/ abs/ .7829
Wenxuan Wu, Zhongang Qi, and Fuxin Li. Pointconv: Deep convolutional networks on 3d point clouds. CoRR, abs/181 2018.
07246
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
Yifan Xu, Tianqi Fan. Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning point sets with parameterized convolutional filters. In The European Conference on Computer Vision (ECCV), September 2018
10

---

Under review as a conference paper at ICLR 2020
Li Yi, Vladimir G. Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qix- ing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region an- notation in 3d shape collections. ACM Trans. Graph., 35(6)210:11210:12, November 2016. ISSN 0730-0301. doi: 10. 1491449999999999922283 /tt.t//teee/aa........./////.... 2980179. 2980238
Chris Zhang, Wenjie Luo, and Raquel Urtasun. Efficient convolutions for real-time semantic seg- mentation of 3d point clouds. In 2018 International Conference on 3D Vision (3DV), pp. IEEE, 2018.
A APPENDIX
A.I IMPLEMENTATION DETAILS
We implement the model with Tensorflow using SGD optimizer with an initial learning rate 0.1 for ModelNet40 and ADAM optimizer with initial learning rate 0.002 for ShapeNet Part and S3DIS on a GTX TITAN Xp GPU. During the training procedure of classification, the data is augmented by random rotation, scaling and Gaussian perturbation the coordinates of the points. During the training procedure of segmentation, the data is augmented by scaling and Gaussian perturbation on the coordinates of the points.
Following Li et al. (2018b). the data of S3DIS is firstly split by room, and then the rooms are sliced into 1.5m by 1.5m blocks, with 0.3m padding on each side. The points in the padding areas serve as context of the internal points, and themselves are not linked to loss in the training phase, nor used for prediction in the testing phase. Each block will be viewed as a point cloud during training and testing.
We do inferences on the augmented data during testing by voting following Qi et al. 2017b). To clarify this, let x²' be one data from the test set and z; be the augmented data generated from x². p)} [xx)) denotes the predict possibility vector after Softmax. The final inference result of of is Predict(x") = arg max Pp)).
A.2 RUNNING STATISTICS
Table 3: Comparisons of number of parameters and FLOPs for classification. FL.OPs(Infrencc))
method
Parameters
pointnet et (017a)
3.48M
14.70B
pointnet++* + Qi et al.
(207bb)
1 1.48M
26.94B
DGCNN Wang et al.
2018C
1.84M
44.27B
pointenn Li et al.
(2018b)
0.6M
25.30B
Ours
1.29M
11.7B
Table 4: Comparisons of training time of networks on ModelNet40
Method
Settings
Accuracy and Training (+Pre- (irre--rocessigg) time
PointNet++ Qi
adam, 1024 points
90.7%, 7,, hours
al. (2017b)
adam, 5000 points
91.9%, 20 hours
adam, 2048 points
92.4%, 6h (+1.5h)
ours
SGD, 2048 points
92.7%, 12h (+1.5h)
As shown in Table we summarize our running statistics based with model for ModelNet40 with batch size 16. In comparison with several other methods, although fewer channels, smaller kernels and simpler interpolation (nearest neighboring) make NPTC use similiar parameters and even fewer FLOPs.
ResNet structure, the
Total pre- procceccssing time mainly depends on the grid's density. For most cases, the resolution of 100 enough to describe the shape of the point cloud, which is what chose for our experiments.
11

---

Under review as a conference paper at ICLR 2020
We also remark that the whole computation cost of constructing convolution is linearly depending on the resolution of voxel and the size of data. On a PC with Core 17-7700 CPU, the pre-processing takes about 0.5 seconds per point cloud with Matlab, and the whole dataset of ModelNet40 (12,311 shapes with 10,000 points each) takes only about 1.5 hours. It negligible compared to the training of the deep neural networks and acceptable to do inference in practice.
A.3 VISUALIZATION
(a) low-level features
(b) high-level features
Figure 4: Feature Visualization: each feature from low or high level displayed on point clouds from different categories. High-level activations are in yellow and low-level activations in blue.
To visualize the effects of the proposed NPTC in the NPTC-net, we trained the network on ShapeNet Part and visualize learned features by coloring the points according to their level of activation. In Figure 4 filters from the the first Convolution layer in the the first Residual block and final Con- volution layer in the second Residual block are chosen. In order to easily compare the features at different levels, we interpolate them on the input point cloud. Observe that low-level features mostly represent simple structures like edges (top of (a)) and planes (bottom of (a)) with low variation in their magnitudes. In deeper layers, features are richer and more distinct from each other, like bottle- neck (upper left (b)), bbig-head"(uppe right of (b)), plane base (lower left of (b)), bulge (lower right of (b)).
A.4 SINGULARITIES
(a) Singularity at corner
(b) Singularity at middle
(c) smooth
Figure 5: Different strategies induce different vector fields: (a) is the vector field which selects the
corner as the origin of the distance function. (b) is the vector field which selects the middle as the origin. (c) is the linear combination of two vector fields with different origins. The combination coefficients depend on the distance the origin at each point: T. d(y), 2 d(y2, z) {i* represent the vector field with the origin yi and d(yi, x) represent the geodesic distance. The absolute value of combination coefficient for one vector field will descend prevent the singularity at the origin.
approaches the origin to
We note that on non paralleliizz manifolds such as the sphere, it is not possible to construct a smooth vector field. Helgason| (1979) So that the vector field must have at least singularity,
12

---

Under review as a conference paper at ICLR 2020
and the frame around the singularity will rotate violently. Figure shows vector fields defined on the surface with and without singularity.
If singularity not be avoided. It is natural to ask how much singularities affect the result. This has been discussed in section 4.4 PTC. Schoonseeck et al. (2018) In surface MNIST experiment, convolutions defined by taking the middle point of surface the origin of the distance function (which means that there is singularity in the middle like Figure [ (b)) will reach the accuracy of 94.92% and convolutions using the smooth strategy will reach 96.36% Singularity does cause some problems, but we think it's in the acceptable range. In our experience, activation in segmentation task have been visualized before, we find no direct relationshh with the location of the singularity.
If we have some prior knowledge about shape that can have no singularity. Strategies can be applied to get smooth vector field or reduce the influence of singularity. For example, the shape is essentially like wrinkled sheet, then we can choose the starting point(s) A of the Eikonal equation as points on one edge of the sheet or the linear combination of two vector fields with different origins 5ig 5
Most of the shapes the three experiment datasets are closed manifolds like boxes, beds, airplanes, on which singularity can not be avoided. Experiments show that applying more strategies we know can not improve the results obviously, so we choose select one initial point directly.
13


Reviews:
[{"review_id": "ryxzaToRFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Narrow-Band Parallel Transport Convolution (NPTC) for point cloud data. The general idea is to use gradients of some distance function to define the vector field. The authors use voxelization to approximate the point cloud in a narrow-band covering the point cloud so that distance function can be calculated. The authors also discuss how to compute the distance function and the vector fields on point clouds. Experimental results on classification tasks and segmentation tasks are provided. \n\n\nI am not an expert on this topic and do not have enough expertise to judge the theoretical novelty of this paper (and thus my low confidence). However, I believe the writing can be significantly improved by making the introduction part more friendly to readers unfamiliar with differential geometry. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SyxSXmjEYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a new geometric convolution that can be applied to point clouds approximating a 2D manifold in R^3. Similar to previous work, the continuous mathematical description of the method involves a kernel that is a function on R^2, which is pushed to the tangent spaces of the manifold via a choice of frame u. A local neighbourhood around the kernel is parameterized by the tangent space via the exponential map and matched against the kernel. The frame u, which determines the orientation of the kernel, is chosen by first picking an origin and then setting u1 to the direction of steepest ascent of the function that measures geodesic distance to the origin. The other basis vector u2 is computed as u1 x n where n is the normal vector. In a practical implementation, the point cloud is voxelized in order to compute the distance function and it's gradient. The gradient is then interpolated back to the point cloud. If I understood correctly, a convolution is then directly applied to the point cloud, though I could not find a description of how this is done.\n\nThe problem of defining geometric convolutions on point clouds is interesting and relevant, and something like what is presented in this paper might work well. Indeed the experiments demonstrate that the method produces decent scores on modelnet and S3DIS. The method is not very fast, due to the voxelization preprocessing which is required, which at .5s precludes real-time applications. Also, as discussed below, there are some issues related to singularities in the vector fields u1,u2 that are not adequately discussed. My main concern, and the reason I have given a \"weak reject\" rating, is that the clarity and completeness of the paper is insufficient and I find the paper hard to follow (see below). I would actually like to see an improved version published, and if the authors can address the concerns I would increase my score.\n\n\n# Writing\nI find that the paper is hard to read because of the way it is structured, as well as numerous small mathematical and expository issues (discussed below) and missing information.\n\nThe present paper builds on the PTC method of Schonsheck et al., but this method is not properly explained in the paper. PTC is mentioned in 1.1, but only described at a very high level. Since I had not read this paper before (I have now) the present paper was hard to follow on a first reading.\n\nSection 1.2 is a large sub-section of the related work section that actually discusses the proposed NPTC. I think it's better not to spend so much of the related work section on this. Moreover, section 3.1.1. \"General idea of NPTC\", contains another high level description, which is somewhat redundant (though at the same time, both sections leave lots of questions unanswered - see below). The following sections, 3.1.2 and 3.1.3 explain the computation of the distance function and the vector fields, but contain no clear explanation of the whole algorithm.\n\nMany important questions about how the method is actually implemented are left unanswered, even though the main contribution of this paper is at the implementation level rather than the theoretical level.\n- How is voxelization done? Count the number of points per voxel? How is the voxel size chosen? How do you make sure that you obtain a \"narrow band\" around the surface defined by the point cloud? What kind of datastructure do you use to store a sparse voxel grid?\n- How is the filter parameterized? In the theory, k is a function on R^2. How do we evaluate it at an arbitrary point in R^2?\n- How is convolution done? I think it is done on points not voxels, similar to eq. 4, but this is never stated in the paper. How do you approximate the integral over the tangent space in Eq 2, if eq 2 is what the conv is based on?\n\nI would remove section 2.1 where the notion of a connection is formally introduced. The reader will either know this concept, in which case it is unnecessary, or not, in which case the description is far too compact (one must read a textbook to really understand this notion). It may be better to include only a short intuitive introduction to parallel transport, refering the reader to other texts for mathematical definitions of basic notions like connections. \n\nThe fact that the voxelization of the point cloud is sparse is only mentioned very late in the paper. For a while I was thinking the idea would be to construct a dense voxel grid, which would be extremely memory intensive. Also, the phrase \"narrow-band\" is often used with a very different meaning, as in a small part of the spectrum. It was not clear to me what this was referring to until halfway through the paper. Better to explain the general idea of the algorithm in the introduction.\n\n\n# Singularities\nI looked at the papers citing the main related work (the PTC paper by Schonscheck et al., 2018), and of the four listed in google scholar, one seems relevant in that it prominently cites Schonscheck et al. and claims to improve upon it, but this work is not cited in the present paper:\nCohen, Weiler, Kicanaoglu, Welling, Gauge Equivalent Convolutional Networks and the Icosahedral CNN.\nThis work highlights one potential downside of the PTC approach, which is that it is not possible to find a global frame u (called in w in their paper) without singularities on manifolds like the sphere. Moreover, the choice of frame is fundamentally arbitrary and should not affect the results. In the PTC approach, a certain choice of frame with singularities is made, which leads to filter orientations that are 1) essentially arbitrary, and 2) change smoothly in most places but abruptly near the singularities. The results are heavily dependent on the choice of frame.\n\nThe present paper dismisses these issues by saying that \"We remark that possible singularities will lead to no convolution operation at those points. These are isolated points on a closed manifold and do not effect experiment results.\"\nHowever, this is not actually demonstrated in the paper, and one might reasonably believe that this problem affects more than a small finite set of points. Near a singularity (e.g. the poles of the sphere in fig 2d), nearby filters will be oriented very differently. If we draw a small circle around the singularity, the orientation of the filter changes 360 degrees as we travel around the circle. As a result, the meaning of the patterns in the output feature map is very different near the poles than it is near the equator. Nevertheless, the next layer will try to match these patterns with the same set of filters / perform weight sharing across different positions. In my opinion this issue should be more frankly discussed.\n\nFortunately, in some applications with point clouds this issue can be avoided. For point clouds coming directly from a (RGB+)depth sensor, the manifold is essentially like a wrinkeled sheet (homeomorphic to a disk), and one can easily find vector fields / frames without singularities on such manifolds. (this is not necessarily true for point clouds that represent a whole scene, e.g. in SLAM). I think this is what is mentioned in paragraph 3 of 3.1.1. However, it is not clear to me if the frame used in this paper is without singularities even when it is possible to find such a frame. The reason is that the frame is defined in terms of the distance function rho to an arbitrary origin. If I understand correctly, one will always have a singularity at the origin. This could in principle easily be solved though by simply using the x and y direction of the camera plane as a frame (projecting it from R^3 to each tangent space). However, if the authors decide to go that route, one would also like to see a comparison to simply applying 2d convolutions to the RGB+D images.\n\n\n# Experiments\n\nThe experiments on Modelnet40 show that, when compared with other point-cloud methods, the proposed method outperforms. I do not know if there are better point-based methods than those listed in the paper, but on the official Modelnet40 leaderboard (which is incomplete) there are several methods that score better (97%+). Since this difference could possibly be attributed to a loss of information due to the discretization into a point cloud, I still think these results are quite good. Similarly on S3DIS the method scores quite well, though does not reach state of the art.\n\nI could not find the architecture for the classification network, and only a high-level description of the segmentation architecture. More details on the training and evaluation procedures could be added for both experiments.\n\n\n# Misc\n\nSome more things I noticed below. Many of these are not serious problems, just things that can be improved / fixed.\n\n- p2:\n-- \"Spatial mesh-based methods are more intuitive\" - this is subjective\n-- f(v)  where v is a tangent vector in T_xe M is not defined. One would expect f to be a function on the manifold. Typically one writes f(exp_x v) instead.\n-- script M is not defined (though it is intuitive that this is the manifold). Later in 2.1 it is written \"Let M be a 2 dimensional differential manifold\". Later still (3.1) the notation is switched to P without notice.\n-- TxM is not defined. Later it is explained this is the tangent plane. Neither is g_x (it is obvious this should be the metric, but only if one knows differential geometry).\n-- \"and do not effect experimental results\" -> affect\n-- \"singularities from a given vector field can be overcome ....\" I don't understand what this is saying. How can we avoid the issues with singularities? \n-- consider writing log instead of exp^{-1}\n-- \"while uses tangent planes\" -> using\n- p3:\n-- You have f(z) in the subscript of the indicator I_argmax ...\n-- In the explanation of pointnet it is not clear where there are any parameters in the kernel k if k(xi, xj) = delta(xi, xj)\n-- I don't understand the explanation of edge convolution. If f(x_j) = 1 (for all j(?)) then k(xi, xj) = MLP(f(xi), f(xi) - f(xj)) would equal MLP(1, 0) which makes no sense.\n-- Description of figure 1: \"are in parallel\": need to say along what curve they are parallel. If I understand correctly, they are only in parallel if they are attached to different points on a minimizing geodesic emanating from the origin.\n- p4:\n-- \"embed in R^3\" -> embedded\n-- In 2.1 it is explained that T_x M is the tangent plane, but you've already discussed T_x M several times before.\n-- A vector field is not just any assignment M -> TM. It is a section of TM, ie a map M -> TM that, when composed with the projection TM -> M, yields the identity on M.\n-- The interval \"I\" mentioned below eq 5 has not been defined. \"I\" was used as indicator function before.\n-- \"X is the unique section of Gamma(TM)\". The concept of \"section\" has not been introduced. Moreover, one could say \"section in Gamma(TM)\" or \"section of TM\" but not \"section of Gamma(TM)\" since Gamma(TM) *is* the set of sections of TM. Probably best to say \"unique vector field\".\n-- \"there will be a geodesic connecting x0 and x1\" - only true if M is connected.\n- p5:\n-- \"how a distance functions are\" -> function\n-- \"easliy\"\n-- below eq 6 \"f(x) is a strictly positive function\", but f(x) does not appear in eq 6\n-- in 3.1., script P has not been defined. I first thought it is just a new letter for M, but I think it is the point cloud? Needs to be made explicit.\n-- \"Geodesic curve represents, in some snse, the shortest path\". This is false. A geodesic is a *straightest* curve, but a geodesic need not be distance minimizing in any sense.\n-- \"ascend direction\" -> ascent\n-- In the first paragraph of 3.1.1 it is not mentioned explicitly that y plays the role of an \"origin\". This makes the last sentence confusing.\n-- \"The value f(v) is computed by f(v) = f(z) where z in P is the closest point to v\". Firstly, this sentence is in the middle of a paragraph about the vector fields u1,u2, but seems unrelated. Secondly, what are v and z? I guess v is a point in M and z in P? Or is v a tangent vector? In that case, do you mean closest in the embedding space R^3?\n- p6:\n-- \"consider point clouds in R^3\". This assumption was already made before, right?\n-- \"it is not straightforwardly compute distance function\" -> \"straightforward to compute the distance function\"\n-- \"Here Lambda is chosen as certain point on the point cloud\" -> \"a certain point\". Also, Lambda was used before as a subset of Omega-bar (below eq 6), but here it is a point. I guess this point is the origin. In other places (earlier) you also refer to the origin, e.g. by y. Needs to be made consistent.\n-- \"Show that the directly\" - remove \"the\"\n-- \"Finally, we interpolate the distance function from the voxels to point cloud\" -> \"the point cloud\". On first reading, this (page 6!) was the point where I realized (guessed) that you are going to define the convolution on the point cloud not voxel grid. This needs to be explained much earlier.\n-- \"it is nature to\" -> natural\n- p7\n-- \"in our implement\" -> implementation\n-- \"Appendix.\" -> \"Appendix A\" or \"the Appendix.\"\n- p8\n-- \"Scene semantics segmentation\" -> semantic\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJlAu2L0FH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose NPTC a new convolutional operator for 3D point clouds embedded on a 2D manifold based no parallel transport defined by a narrow-band approximation. The method combines voxelization within a local neighborhood in 3D (narrow band) and geometric convolution. Experimental results are presneted for a range of classification and segmentation tasks that are similar to the state-of-the-art.\nAs I am not familiar with geometric methods, point clouds and related work, I wouldn't be able to judge the degree of novelty, beyond an educated guess.\n\nThe experimental results in classification and segmentation use the proposed NPTC in standard NN architectures with residual blocks similar to a U-net. In terms of performance, NPTC are amongst the best reported results for each experiment, but always similar to other architectures. Only mean values for IoU and accuracies are reported, but no estimates of variance/spread. Therefore, I cannot judge if the results are significantly different from related methods.\n\nThe paper starts out highly technical.\nThe Notation is introduced in section 2, the Background section, which is after the discussion of related work in section 1, where the notation is already used without being introduced. \nThe related work makes heavy use of manifold geometry. For non-experts in geometry like me, who also doesn't know about notational conventions, it was near impossible to follow on a first read of the paper. I recommend introduction of notation before use, especially as Section 1 is titled \"Introduction\".\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers recommend rejecting the paper. There is no basis for acceptance.
Decision: Reject