ASSANet: An Anisotropic Separable Set Abstraction for Efficient Point Cloud Representation Learning

1 Introduction . Among the various 3D object representations , point clouds have been surging in popularity , becoming one of the most fundamental 3D representations . This popularity stems from the increased availability of 3D sensors , like LiDAR , which produce point clouds as their raw output . The growing presence of point cloud data has been accompanied by the development of many 3D deep learning methods [ 28 , 41 , 19 , 38 , 22 ] . Even though these methods achieve impressive performance , they are generally computationally expensive ( Figure 1 ) . With the integration of LiDAR sensors into hardware-limited devices , such as mobile devices and AR headsets , interest in efficient models for point cloud processing has grown significantly . Given the limited computational power of mobile devices and embedded systems , the design of mobile-friendly point cloud-based algorithms should not only focus on providing good accuracy , but also on maintaining high computational efficiency . When processing point cloud data , one can always opt to convert the data into representations that are more accessible to deep learning frameworks . Popular options are multi-view methods [ 34 , 5 , 42 ] and voxel-based methods [ 6 , 47 ] . Converting to these representations generally requires additional 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . computation and memory , and can lead to geometric information loss [ 23 ] . It is therefore more desirable to operate directly on point clouds . To that extent , we are currently witnessing a surge in point-based methods [ 27 , 28 , 41 , 19 , 38 , 22 ] . The first of such methods was introduced by Qi et al . through the seminal PointNet [ 27 ] architecture . PointNet operates directly on point clouds , without the need for an intermediate representation . Despite its efficiency , PointNet merely learns per-point features individually and discards local information , which restrains its performance . As a variant of PointNet , PointNet++ [ 28 ] presents a novel Set Abstraction module that sub-samples the point cloud , groups the neighborhood , extracts local information via a set of multi-layer perceptrons ( MLPs ) , and then aggregates the local information by a reduction layer ( i.e . pooling ) . Figure 1 shows how PointNet++ outperforms the pioneering PointNet [ 27 ] by a large margin . PointNet++ also obtains better accuracy than the graph-based method DeepGCN [ 19 ] , and does so with a 100× speed gain . PointNet++ provided a good balance between accuracy and efficiency , and was therefore widely utilized in various tasks like normal estimation [ 8 ] , segmentation [ 26 , 14 ] , and object detection [ 32 ] . After PointNet++ , graph-based [ 33 , 39 , 41 , 19 ] , pseudo-grid based [ 37 , 20 , 25 , 38 ] and adaptive weight-based [ 40 , 21 , 7 , 44 ] , became the state-of-the-art in point cloud tasks . As shown in Figure 1 , nearly all of these methods improve performance at the cost of speed . In this work , we focus on designing point cloud networks that are both fast and accurate . Inspired by its success , both in terms of the accuracy-speed balance and its wide adoption , we take a deep dive into PointNet++ . We conduct extensive analysis of its architectural design ( Section 3.1 ) and latency decomposition ( Figure 2 ) . Interestingly , we demonstrate that both its efficiency and accuracy can be improved sharply by minimal modifications to the architecture . These modifications lead to a new architecture design that is faster and more accurate than currently available point methods ( shown in in Figure 1 ) . Contributions . ( 1 ) We demonstrate that the MLPs performed on the neighborhood features in the Set Abstraction ( SA ) module of PointNet++ reduce the inference speed . We introduce a new separable SA module that processes on point features directly allowing for a significant improvement in inference speed . ( 2 ) We discover that all operations for processing neighbors in the SA module are isotropic which limits the performance ( accuracy wise ) . We present a novel Anisotropic Reduction layer that treats each neighbor differently . We then insert Anisotropic Reduction into our Separable SA and propose the Anisotropic Separable Set Abstraction ( ASSA ) module that greatly increases accuracy . ( 3 ) We present ASSANet by replace the vanilla SA in PointNet++ with the proposed ASSA . ASSANet shows a much higher accuracy and a faster speed compared to PointNet++ and previous methods on various tasks ( point cloud classification , semantic segmentation , and part segmentation ) . We further study two regimes for up-scaling ASSANet . As shown in Figure 1 , our scaled ASSANet outperforms the previous state-of-the-art with a much faster inference speed . In particular , scaled ASSANet achieves better accuracy than the graph-based method DeepGCN [ 19 ] with an increase in speed of 294× , the pseudo grid-based method KPConv [ 38 ] ( 54× faster ) , the adaptive weight-based method PosPool * ( S ) [ 22 ] ( 9× faster ) , and the efficient 3D method PVCNN [ 23 ] ( 2× faster ) . 2 Related Work . Projection-based methods . Due to the unstructured nature of point clouds , convolutional neural networks ( CNNs ) that tend to work impressively well on grid stuctured data ( e.g . images , texts and videos ) fail to apply directly on point clouds . One common solution for processing point clouds is to project them into collections of images ( views ) [ 34 , 5 , 42 ] or 3D voxels [ 6 , 47 , 36 ] . Common CNN backbones ( using 2D or 3D convolutions ) can be subsequently utilized to perform these intermediate representations . Although projection-based methods allow for utilizing the well studied convolutional neural networks to point-cloud applications , they are computationally expensive as they are associated Subsampling 5.5 % Grouping 2.3 % Computation 92.3 % Subsampling 6.2 % Grouping 3.1 % Computation 90.6 % Subsampling 12.6 % Grouping 5.1 % Computation 82.4 % Subsampling 24.5 % Grouping 4.3 % Computation 71.1 % ( a ) 1024 points ( b ) 4096 points ( c ) 10,000 points ( d ) 15,000 points Figure 2 : Latency Decomposition of PointNet++ . We show the inference run time decomposition of PointNet++ under different numbers of points as input on one NVIDIA GTX2080Ti GPU . with additional cost of constructing intermediate representations . Moreover , the projection of point clouds causes loss of important geometric information [ 23 ] . Point-based methods . Pioneering work explored the possibility of processing point clouds directly . Qi et al . proposed PointNet [ 27 ] that leverages point-wise MLPs to extract per point features individually . To better encode locality , Qi et al . further presented Set Abstraction ( SA ) to aggregate features from the points ’ neighborhood , and a hierarchical architecture named PointNet++ [ 28 ] that learns multilevel representations and reduces the required computations . After PointNet++ , numerous point-based methods considering neighborhood information were proposed . Graph-based methods [ 33 , 15 , 41 , 39 , 19 , 18 , 29 ] represent point clouds as graphs and process point clouds with graph neural networks . Pseudo grid-based methods project neighborhood features onto different forms of pseudo grids such as tangent planes [ 37 ] , grid cells [ 11 , 46 , 20 , 25 , 38 ] and spherical grid points [ 49 ] which allow convolving with regular kernel weights like CNNs . Adaptive weight-based methods perform weighted neighborhood aggregation by considering the relative positions of the points [ 40 , 21 , 7 , 22 ] or point density [ 44 ] . These methods rely either on designing sophisticated and customized modules , which usually require expensive parameter tuning for different applications [ 38 , 20 , 24 ] , or on performing expensive graph kernels [ 41 , 19 ] that achieve better performance than PointNet and PointNet++ at the expense of computational complexity . Efficient Neural Networks . Efficient neural networks is a class of architectures that target mobile and embedded systems applications . These networks are usually designed to provide a balance between accuracy and efficiency ( e.g . latency , FLOPs , memory , and power ) . MobileNet [ 9 ] utilizes depth-wise separable convolutions to reduce the required FLOPs and latency of a regular CNN for image processing . Depth-wise separable convolutions disentangle convolutions into learning channel correlations using point-wise convolutions and learning spatial correlations using depthwise convolutions . Other efficient neural networks usually leverage either depth-wise separable convolutions with better designed architectures to improve performance [ 31 , 4 , 48 ] or study new efficient operations to replace the regular convolutions [ 24 , 43 ] . In 3D , efficient neural networks include ShellNet [ 49 ] , PVCNN [ 23 ] , Grid-GCN [ 45 ] , RandLA-Net [ 10 ] , SegGCN [ 17 ] and LPNs [ 16 ] . ShellNet [ 49 ] and SegGCN [ 17 ] speed up the pseudo grid-based methods by aggregating neighborhood features through efficient 1D convolutions or fuzzy spherical convolutions on the predefined pseudo grids like shells . PVCNN [ 23 ] and Grid-GCN [ 45 ] reduce the time spent in querying a neighborhood by combining voxelization in point-based methods . RandLA-Net [ 10 ] reduces the subsampling complexity by leveraging random sampling and further improves the speed by operating on a large-scale point cloud directly without chunking . LPN [ 16 ] improves the speed of convolving neighborhood features by a simple group-wise matrix multiplication . Nevertheless , all efficient methods mentioned above require performing convolutions on neighborhood features , which we deem through extensive experiments as unnecessary . Therefore , our algorithm achieves much faster speeds compared to these methods ( ref to Section 4 ) . It is also worthwhile to mention that our method can be made even faster with the voxelization trick in PVCNN and Grid-GCN to further reduce the latency of neighborhood querying . We leave that as future work . 3 Methodology . 3.1 Preliminary : PointNet++ . PointNet++ [ 28 ] improves PointNet [ 27 ] by providing two main contributions : ( 1 ) developing a U-Net [ 30 ] like architecture to process a set of points , which are sampled in a metric space in a hierarchical fashion . This mechanism captures multi-scale features and reduces the required computation . ( 2 ) Developing a Set Abstraction ( SA ) module to process and abstract the locality from the local neighbors to a new set of points with fewer elements . The SA module is used as the basic building block to be stacked to form the backbone of PointNet++ . Analysis of the Set Abstraction Module . As illustrated in Figure 3a , the vanilla SA module proposed in PointNet++ consists of two parts : point subsampling and feature aggregation , . The subsampling layer takes a point cloud X = { P , F } as an input and leverages iterative farthest-point sampling to acquire X ′ , a subset of X . P and F denote the coordinates and features , respectively . The feature aggregation block is built for learning locality from local neighbors and is composed of a grouping layer , an MLP block , and a reduction layer . The grouping layer obtains the neighborhood composed of K neighbors for each point in X ′ using the ball query , with X as the support set . The resulting point neighborhood is denoted as N ( X ′ ) , which contains K repeated features . The MLP block consists of L layers of MLPs , and each MLP is followed by a Batch Normalization ( BN [ 12 ] ) layer and a ReLU activation . By default , PointNet++ sets L = 3 . The number of feature aggregation blocks inside one SA module , referred to as depth D in this paper , is set to D = 2 . The reduction layer ( a.k.a , pooling ) aggregates the neighborhood information by a reduction function , e.g . mean , max , or sum . The feature aggregation is formulated as shown in Equation ( 1 ) : f l+1i = R ( { MLPs ( ( pj − pi ) ||f lj ) |j ∈ N ( i ) } ) , ( 1 ) whereR is the reduction function across the neighborhood dimension , which is used for aggregating the neighborhood information . pi , f li , N ( i ) and || denote the coordinates , the features in the lth layer of the network , the neighborhood of the ith point , and the concatenation operator across the channel dimension , respectively . The main issues with the vanilla SA module are : ( 1 ) the computational cost is unnecessarily high . MLPs are unnecessarily performed on the neighborhood features , which causes a considerable amount of latency in PointNet++ . One straightforward remedy is to use MLPs to learn a feature embedding on the point features directly instead of doing so on the neighborhood features . This reduces the FLOPs of each MLP by a factor of K. ( 2 ) All operations on neighbors are unnecessarily isotropic . In other words , the MLPs and the reduction layer treat all local neighbors equally . This severely limits the representation capability of the network . Latency Decomposition . Figure 2 shows the latency decomposition of PointNet++ [ 28 ] with different numbers of points as input . Here , the latency , which is the overall run time for the inference stage , was measured using a single Nvidia GeForce RTX 2080Ti GPU and one Intel ( R ) Xeon ( R ) CPU E5-2687W v4 @ 3.00GHz . We note here that latency is measured on the same hardware setting throughout this work . The latency of PointNet++ can be decomposed into three main contributing factors : ( 1 ) point subsampling , ( 2 ) grouping , ( 3 ) actual computations . The actual computations of PointNet++ mainly come from processing neighborhood features by MLPs shown in Equation 1 . Note that we consider the time spent on data access implicitly in each part . Point clouds with four different input sizes were studied : 1024 , 4096 , 10 , 000 , and 15 , 000 . The first two input sizes are commonly encountered in classification tasks [ 2 ] , and the last two are usually input sizes for patch-based segmentation [ 28 , 38 ] and LiDAR-based object detection [ 32 ] . Clearly , computations contribute to the majority of latency ( over 70 % ) . This suggests that the computational complexity could be the major speed bottleneck for networks involving PointNet++ .

Summary: In this paper, the authors present PointNetV3, which is a faster and more accurate version of PointNet++.  The authors first analyze the latency bottleneck of PointNet++ and find that computation takes up to 70% of total latency, and then propose to reduce the computation cost via several techniques. Namely, separate SA modules are used to reduce the computation cost of vanilla SA (since MLP after reduction is much more efficient); Anisotropic reduction is applied to achieve higher accuracy (original PointNet++ shares conv. weights for all neighbors, which is suboptimal). The authors also analyze the effectiveness of width and depth scaling to improve the efficiency-accuracy tradeoff of PointNetV3 models. Experiment results on S3DIS, ModelNet40 and ShapeNet Part are impressive.