Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework

1 INTRODUCTION . Lately , point cloud analysis has emerged as a popular topic in 3D understanding , attracting attention from academia and industry Qi et al . ( 2017a ) ; Shi et al . ( 2019 ) ; Xu et al . ( 2020 ) . Different from 2D images represented by regular dense pixels , point clouds are composed of unordered and irregular sets of points P ∈ RN×3 , making it infeasible to directly apply image processing methods to point cloud analysis . Meanwhile , the nature of sparseness and the presence of noises further restrict the performance . In the past few years , endowing with neural networks , point cloud analysis has seen a great improvement in various applications , including 3D shape classification Qi et al . ( 2017a ) , semantic segmentation Hu et al . ( 2020 ) and object detection Shi & Rajkumar ( 2020 ) , etc . Recent efforts have shown promising results for point cloud analysis by exploring local geometric information , using convolution Li et al . ( 2021a ) , graph Li et al . ( 2021a ) , or attention mechanism Guo et al . ( 2021 ) ( see Section 2 for details ) . These methods , despite their gratifying results , have mainly relied on the premise that an elaborate local extractor is essential for point cloud analysis , leading to the competition for careful designs that explore fine local geometric properties . Nevertheless , sophisticated extractors are not without drawbacks . On one hand , due to prohibitive computations and the overhead of memory access , these sophisticated extractors hamper the efficiency of applications in real scenes . As an example , until now , most 3D point cloud applications are still based on the simple PointNet ( and PointNet++ ) or the voxel-based methods Liu et al . ( 2021 ) ; Li et al . ( 2021b ) ; Zhang et al . ( 2021 ) . Applications that employ the aforementioned advanced methods , however , are rare in literature . On the other hand , the booming sophisticated extractors , saturate the performance since they already describe the local geometric properties well and more complicated design is no longer to further improve the performance a lot . These phenomena suggest that we may need to stop the race of local feature extraction designing , rethinking the necessity of elaborate local feature extractors and further revisiting the succinct design philosophy in point cloud analysis . In this paper , we aim at the ambitious goal of building a deep network for point cloud analysis using only residual feed-forward MLPs , without any delicate local feature explorations . By doing so , we eschew the prohibitive computations and ceaseless memory access caused by the sophisticated local geometric extractors , and enjoy the advantage of efficiency from the highly-optimized MLPs . To further improve the performance and generalization ability , We further introduce a lightweight local geometric affine module that adaptively transforms the point feature in a local region . We term our new network architecture as PointMLP . In the sense of MLP-based design philosophy , our PointMLP is similar to PointNet and PointNet++ Qi et al . ( 2017a ; b ) . However , our model is more generic and exhibits promising performance . Different from the models with sophisticated lo- cal geometric extractors ( e.g. , DeepGCNs Li et al . ( 2019 ) , RSCNN Liu et al . ( 2019b ) , etc . ) , our PointMLP is conceptually simpler and achieves results on par or even better than these state-ofthe-art methods ( see Figure 1 ) . Keep in mind that we did not challenge the advantages of these local geometric extractors and acknowledge their contributions , however , a more succinct framework should be studied considering both the efficiency and accuracy . In Table 1 , we compare our PointMLP with some representative methods . Even the design philosophy is simple , PointMLP exhibits prominent performance on 3D point cloud analysis . Specifically , we achieve the state-of-the-art classification performance , 94.5 % , on the ModelNet40 benchmark , and we outperform related works by 3.3 % accuracy on the real-world ScanObjectNN dataset , with a significantly higher inference speed . 2 RELATED WORK . Point cloud analysis . There are mainly two streams to process point cloud . Since the point cloud data structure is irregular and unordered , some works consider projecting the original point clouds to intermediate voxels Maturana & Scherer ( 2015 ) ; Shi et al . ( 2020 ) or images You et al . ( 2018 ) ; Li et al . ( 2020 ) , translating the challenging 3D task into a well-explored 2D image problem . In this regime , point clouds understanding is largely boosted and enjoys the fast processing speed from 2D images or voxels . Albeit efficient , information loss caused by projection degrades the representational quality of details for point clouds Yang et al . ( 2019 ) . To this end , some methods are proposed to directly process the original point cloud sets . PointNet Qi et al . ( 2017a ) is a pioneering work that directly consumes unordered point sets as inputs using shared MLPs . Base on PointNet , PointNet++ Qi et al . ( 2017b ) further introduced a hierarchical feature learning paradigm to recursively capture the local geometric structures . Owing to the local point representation ( and multi-scale information ) , PointNet++ exhibits promising results and has been the cornerstone of modern point cloud methods Wang et al . ( 2019 ) ; Fan et al . ( 2021 ) . Our PointMLP also follows the design philosophy of PointNet++ but explores a simpler yet much deeper network architecture . Local geometry exploration . As PointNet++ built the generic point cloud analysis network framework , the recent research focus is shifted to how to generate better regional points representation . Predominantly , the explorations of local points representation can be divided into three categories : convolution- , graph- , and attention-based methods . One of the most distinguished convolution-based methods is PointConv Wu et al . ( 2019 ) . By approximating continuous weight and density functions in convolutional filters using an MLP , PointConv is able to extend the dynamic filter to a new convolution operation . Also , PAConv Xu et al . ( 2021a ) constructs the convolution kernel by dynamically assembling basic weight matrices stored in a weight bank . Without modifying network configurations , PAConv can be seamlessly integrated into classical MLP-based pipelines . Unlike convolution-based methods , Graph-based methods investigate mutually correlated relationships among points with a graph . In Wang et al . ( 2019 ) , an EdgeConv is proposed to generate edge features that describe the relationships between a point and its neighbors . By doing so , a local graph is built and the point relationships are well preserved . In 3D-GCN Lin et al . ( 2021 ) , authors aim at deriving deformable 3D kernels using a 3D Graph Convolution Network . Closely related to graphbased methods , the attention-based methods exhibit excellent ability on relationship exploration as well , like PCT Guo et al . ( 2021 ) and Point Transformer Zhao et al . ( 2021 ) ; Engel et al . ( 2020 ) . With the development of local geometry exploration , the performances on various tasks appear to saturate , continuing on this track would bring minimal improvements . In this paper , we showcase that even without the carefully designed operations for local geometry exploration , a pure deep hierarchical MLP is able to exhibit gratifying performances and even better results . Deep network architecture for point cloud . Interestingly , the development of point cloud analysis is closely related to the evolution of the image processing network . In the early era , works in the image processing field simply stack several learning layers to probe the performance limitations Krizhevsky et al . ( 2012 ) ; Simonyan & Zisserman ( 2015 ) ; Dong et al . ( 2014 ) . Then , the great success of deep learning was significantly promoted by deep neural architectures like ResNet He et al . ( 2016 ) , which brings a profound impact to various research fields . Recently , attention-based models , including atten- tion blocks Wang et al . ( 2018 ) and Transformer architectures Dosovitskiy et al . ( 2021 ) ; Touvron et al . ( 2021b ) , further flesh out the community . Most recently , the succinct deep MLP architectures have attracted a lot of attention due to their efficiency and generality . Point cloud analysis follows the same develop history as well , from MLP-based PointNet Qi et al . ( 2017a ) , deep hierarchical PointNet++ Qi et al . ( 2017b ) , convolution- ( or graph- ) based methods Wu et al . ( 2019 ) ; Wang et al . ( 2019 ) , to state-of-the-art Transformer-based models Guo et al . ( 2021 ) ; Zhao et al . ( 2021 ) . In this paper , we abandon sophisticated details and present a simple yet effective deep residual MLP network for point cloud analysis . Instead of following the tendency in the vision community deliberately , we are in pursuit of an inherently simple yet powerful architecture for point cloud analysis . 3 DEEP RESIDUAL MLP FOR POINT CLOUD . We propose to learn a point cloud representation by a simple feed-forward residual MLP network , which hierarchically aggregates the local features extracted by MLPs , and abandons the use of delicate local geometric extractors . To further improve the robustness and stability , we introduce a lightweight geometric affine module to transform the local points to a normal distribution . The detailed framework of our method is illustrated in Figure 2 . 3.1 REVISITING POINT-BASED METHODS . The design of point-based methods for point cloud analysis dates back to the PointNet and PointNet++ papers Qi et al . ( 2017a ; b ) , if not earlier . The motivation behind this direction is to directly consume point clouds and avoid unnecessary rendering processes . Given a set of points P = { pi|i = 1 , · · · , N } ∈ RN×3 , where N indicates the number of points in a ( x , y , z ) Cartesian space , point-based methods aims to directly learn the underlying representation f of P using neural networks . One of the most pioneering works is PointNet++ , which learns hierarchical features by stacking multiple learning stages . In each stage , Ns points are re-sampled by the farthest point sampling ( FPS ) algorithm where s indexes the stage andK neighbors are employed for each sampled point and aggregated by max-pooling to capture local structures . Conceptually , the kernel operation of PointNet++ can be formulated as : gi = A ( Φ ( fi , j ) |j = 1 , · · · , K ) , ( 1 ) where A ( · ) means aggregation function ( max-pooling in PointNet++ ) , Φ ( · ) denotes the local feature extraction function ( MLP in PointNet++ ) , and fi , j is the j-th neighbor point feature of i-th sampled point . By doing so , PointNet++ is able to effectively capture local geometric information and progressively enlarge the receptive fields by repeating the operation . In the sense of network architecture design , PointNet++ exhibits a universal pipeline for point cloud analysis . Following this pipeline , some plug-and-play methods have been proposed , mainly focusing on the local feature extractor Φ ( · ) Xu et al . ( 2021a ) ; Liu et al . ( 2019b ) ; Thomas et al . ( 2019 ) ; Zhao et al . ( 2021 ) . Generally , these local feature extractors thoroughly explore the local geometric information using convolution , graph , or self-attention mechanisms . In RSCNN Liu et al . ( 2019b ) , the extractor is mainly achieved by exploring point relations as follow : Φ ( fi , j ) = MLP ( [ ‖xi , j − xi‖2 , xi , j − xi , xi , j , xi ] ) ∗ fi , j , ∀j ∈ { 1 , · · · , K } , ( 2 ) where [ · ] is the concatenation operation and MLP is a small network composed of a Fully-connected ( FC ) layer , Batch Normalization layer , and activation function . Unlike RSCNN , Point Transformer introduces the self-attention mechanism into point cloud analysis and considers the similarities between pair-wise points in a local region . To this end , it re-formulates the extractor as : Φ ( fi ) = k∑ j=1 ρ ( γ ( ϕ ( fi ) − ψ ( fi , j ) + δ ) ) ( α ( fi , j + δ ) ) , ( 3 ) where γ , ϕ , ψ and α are linear mapping function , “ ” is a Hadamard product , and ρ is a softmax normalization . In particular , Point Transformer introduces a relative position encoding , δ = θ ( xi − xi , j ) , where the relative position is encoded by two FC layers with a ReLU nonlinearity layer , into both attention weights and features . The lightweight positional encoder largely improves the performance of Point Transformer . While these methods can easily take the advantage of detailed local geometric information and usually exhibit promising results , two issues limit their development . First , with the introduction of delicate extractors , the computational complexity is largely increased , leading to prohibitive inference latency 1 . For example , the FLOPs of Equation 3 in Point Transformer would be 14Kd2 , ignoring the summation and subtraction operations . Compare with the conventional FC layers that 1We emphasize that the model complexity could not be simply revealed by FLOPs or parameters , other metrics like memory access cost ( MAC ) and the degree of parallelism also significantly affect the speed Ma et al . ( 2018 ) ; Zhang et al . ( 2020 ) . However , these important metrics are always ignored in point clouds analysis . enjoy 2Kd2 FLOPs , it increases the computations by times . Notice that the memory access cost is not considered yet . Second , with the development of local feature extractors , the performance gain has started to saturate on popular benchmarks . Moreover , empirical analysis in Liu et al . ( 2020 ) reveals that most sophisticated local extractors make surprisingly similar contributions to the network performance under the same network input . Both limitations encourage us to develop a new method for point cloud analysis that circumvents the employment of sophisticated local extractors , as well as providing gratifying results .

Summary: This paper introduces a lightweight and fast neural network architecture processing 3D point cloud. While the idea is quite simple, the proposed architecture outperforms or matches the performance of the previous architectures in terms of both accuracy and also inference time with various tasks including ModelNet40 classification, ScanObjectNN classification, and ShapeNetPart segmentation, as shown in the experimental results. The main idea is to make two modifications in the PointNet++ architecture, 1) adding MLP layers after each of the point feature aggregation (max-pooling) steps, and 2) applying a geometric affine transformation to the features processed with an MLP in each small PointNet module. It's interesting to see that these small modifications (especially the second) make dramatic changes in the results.