Revisiting Point Cloud Classification with a Simple and Effective Baseline

1 INTRODUCTION . Processing 3D point cloud data accurately is crucial in many applications including autonomous driving ( Navarro-Serment et al. , 2010 ) and robotics ( Rusu et al. , 2009 ) . In these settings , sensors like LIDAR produce unordered sets of points that correspond to object surfaces . Correctly classifying objects from this data is important for 3D scene understanding ( Uy et al. , 2019 ) . While classical approaches for this problem have relied on hand-crafted features ( Arras et al. , 2007 ) , recent efforts have focused on the design of deep neural networks ( DNNs ) to learn features directly from raw point cloud data ( Qi et al. , 2017a ) . Deep learning-based methods have proven effective in aggregating information across a set of 3D points to accurately classify objects . The most widely adopted benchmark for comparing methods for point cloud classification has been ModelNet40 ( Wu et al. , 2015b ) . The accuracy on ModelNet40 has steadily improved over the last few years from 89.2 % by PointNet ( Qi et al. , 2017a ) to 93.6 % by RSCNN ( Liu et al. , 2019c ) ( Fig . 1 ) . This progress is commonly perceived to be a result of better designs of network architectures . However , after performing a careful analysis of recent works we find two surprising results . First , we find that auxiliary factors including differing evaluation schemes , data augmentation strategies , and loss functions affect performance to such a degree that it can be difficult to disentangle improvements due to the network architecture . Second , we find that a very simple projection-based architecture works surprisingly well , outperforming state-of-the-art point-based architectures . In deep learning , as results improve on a benchmark , attention is generally focused on the novel architectures used to achieve those results . However , there are many factors beyond architecture design that influence performance including data augmentation and evaluation procedure . We refer to these additional factors as a method ’ s protocol . A protocol defines all details orthogonal to the network architecture that can be controlled to compare differing architectures . Note that it is possible for some specific form of loss or data augmentation to be tied to a specific architecture and inapplicable to other architectures . In these cases , it would be inappropriate to treat them as part of the protocol . However , for all the methods we consider in this paper , their losses and augmentation schemes are fully compatible with each other and can be considered independently . We do experiments to study the effect of protocol and discover that it accounts for a large difference in performance , so large as to obscure the contribution of a novel architecture . For example , the performance of the PointNet++ architecture ( Qi et al. , 2017b ) jumps from 90.0±0.3 to 93.3±0.3 , when switching from its original protocol to RSCNN ’ s protocol ( Liu et al. , 2019c ) . We further find that the protocols that lead to the strongest performance rely on feedback from the test set , which differs from conventional evaluation setups . We re-evaluate prior architectures using the best augmentation and loss functions , while not using any feedback from the test set . We find that by taking protocol into account , the PointNet++ architecture performs competitively with more recent ones in various settings . In addition to the surprising importance of protocol , in reviewing past approaches , another surprising discovery is that a very simple projection based baseline works very well . One needs to simply project the points to depth maps along the orthogonal views , pass them through a light-weight CNN and fuse the features . We refer to this baseline as SimpleView . Compared to previous projection-based method ( Roveri et al. , 2018 ; Sarkar et al. , 2018 ) for pointcloud classification , SimpleView is very simple . Prior methods have developed special modules for view selection , rendering , and feature merging , as well as use larger CNN backbones that are pretrained on ImageNet ( refer to Sec . 2 for more details ) . In contrast , SimpleView has no such special operations , and only requires simple point projections , a much smaller CNN backbone , and no ImageNet pretraining . The discovery of SimpleView is surprising because recent state-of-the-art results have all been achieved by point-based architectures of increasing sophistication . In recent literature , it is often assumed that point-based methods are the superior choice for point-cloud processing as they “ do not introduce explicit information loss ” ( Guo et al. , 2020 ) . Prior work has stated that “ convolution operation of these methods lacks the ability to capture nonlocally geometric features ” ( Yan et al. , 2020 ) , that a projection-base method “ often demands a huge number of views for decent performance ” ( Liu et al. , 2019c ) , and that projection-based methods often “ fine-tune a pre-trained image-based architecture for accurate recognition ” ( Liu et al. , 2019c ) . It is thus surprising that a projection-based method could achieve state-of-the-art results with a simple architecture , only a few views , and no pretraining . On ModelNet40 , SimpleView performs on par or better than more sophisticated state-of-the-art networks across various protocols , which includes the ones used by prior methods ( Table . 3 ) as well as our protocol ( Table . 5 ) . At the same time , SimpleView outperforms state-of-the-art architectures on ScanObjectNN ( Uy et al. , 2019 ) , a real-world dataset where point clouds are noisy ( background points , occlusions , holes in objects ) and are not axis-aligned . SimpleView also demonstrates better cross-dataset generalization than prior works . Furthermore , SimpleView uses less parameters than state-of-the-art networks ( Table . 5 ) . Note that we are not proposing a new architecture or method , but simply evaluating a simple and strong projection-based baseline for point-cloud classification that is largely ignored in the literature . We do not claim any novelty in the design of SimpleView because all of its components have appeared in the literature . Our contribution is showing that such a simple baseline works surprisingly well , which is a result absent in existing literature . It is worth noting that one might think that projection-based methods are not directly comparable with point-based methods because projection-based methods may have the full mesh as input , as opposed to just a point cloud . While this is true for existing results in the literature , it is not the case with SimpleView , whose input is the exact same point cloud given to a point-based method . In other words , SimpleView is directly comparable to a point-based method because they solve the exact same task . In summary , our contributions are threefold : • We show that training and evaluation factors independent of network architecture have a large impact on point-cloud classification performance . With these factors controlled for , PointNet++ performs as well as more recent architectures . • We demonstrate how SimpleView , a very simple projection based baseline performs surprisingly well on point-cloud classification . It performs on par with or better than prior networks on ModelNet40 while using fewer parameters . It also outperforms state-of-the-art methods on real-world point-cloud classification and achieves better cross-dataset generalization . 2 RELATED WORK . Point-Based Methods for Point-Cloud Analysis : A broad class of DNNs have emerged to process 3D points directly ( Simonovsky & Komodakis , 2017 ; Zaheer et al. , 2017 ; Klokov & Lempitsky , 2017 ; Xu et al. , 2018 ; Atzmon et al. , 2018 ; Wang et al. , 2018a ; Li et al. , 2018a ; Groh et al. , 2018 ; Ben-Shabat et al. , 2018 ; Xie et al. , 2018 ; Li et al. , 2018b ; Liu et al. , 2019a ; Thomas et al. , 2019 ; Komarichev et al. , 2019 ; Liu et al. , 2019b ; Yan et al. , 2020 ; Su et al. , 2018 ; Zhang et al. , 2019 ; Liu et al. , 2019a ; Atzmon et al. , 2018 ) . PointNet ( Qi et al. , 2017a ) proposed one of the first strategies , where features are updated for each point with MLP layers , and aggregated with global max pooling . However , no local comparisons are performed in PointNet , which motivates PointNet++ ( Qi et al. , 2017b ) . PointNet++ breaks subsets of points into local regions that are processed first . More explicit modeling of the spatial relations between points is performed with more recent methods ( Li et al. , 2018b ; Liu et al. , 2019c ; Wu et al. , 2019 ) . For example , PointConv learns functions to define continuous 3D convolutions that can be applied to arbitrary sets of points in a neighborhood ( Wu et al. , 2019 ) . RSCNN uses MLPs conditioned on the spatial relationship of two points to update and aggregate features around an individual sampled point ( Liu et al. , 2019c ) . There exist many variations to these methods , but the emerging trend is an increase in sophistication . Projection-Based Methods for Point-Cloud Classification : Projection-based methods for point cloud classification have been proposed in the literature . Notably , Roveri et al . ( 2018 ) learn to predict viewing angles and classify images in an end-to-end differentiable way . They use the ResNet50 model , pretrained on ImageNet as their backbone and a depth-image generation pipeline . Sarkar et al . ( 2018 ) propose a special multi-height rendering and feature merging scheme , and use a larger backbone network pretrained on ImageNet . Ahmed et al . ( 2019 ) manually define important views for each object category , create binary edge maps , and train an ensemble of PointNet++ and CNN . However , numbers in Ahmed et al . ( 2019 ) are not directly comparable to other approaches as there is a manual alignment of objects in the test set which is different from the standard ModelNet40 test set . This was confirmed with the authors . It is worth noting that even though prior work has shown sophisticated operations to be useful for achieving good results , we find that when controlling for method protocols , strong performance can be achieved with fixed orthogonal views , a smaller network , no ImageNet pretraining , and simpler rendering of points . Projection-Based Methods for Other Point-Cloud Analysis Tasks : There is a rich literature for using projection-based methods on various point-cloud analysis problems like segmentation ( Ladickỳ et al. , 2010 ; Tighe & Lazebnik , 2010 ; Riemenschneider et al. , 2014 ; Qin et al. , 2018 ; Dai & Nießner , 2018 ; Kalogerakis et al. , 2017 ; Tatarchenko et al. , 2018 ) , reconstruction ( Pittaluga et al. , 2019 ) and rendering ( Aliev et al. , 2019 ) . Notably , Boulch et al . ( 2017 ) use point cloud density to create scene meshes , which are then put into a mesh renderer to generate many image views at different scales . Lawin et al . ( 2017 ) render a scene point cloud from 120 views for different modalities like color , depth , and surface normal . Information from multiple modalities is then fused to generate point-wise predictions . For a detailed survey of various projection approaches on different point-cloud processing tasks , we encourage readers to check the recent survey paper by ( Guo et al. , 2020 ) . In this work , SimpleView serves as a stripped-down projection-based baseline for point-cloud classification that uses a few orthogonal views and simple point projections . 3D shape Analysis using Rendered Images and Voxels : Many works use images rendered from object meshes for 3D shape analysis ( Maturana & Scherer , 2015 ; Wu et al. , 2015b ; Yu et al. , 2018 ; Guo et al. , 2016 ; Shi et al. , 2015 ; Hackel et al. , 2017 ; Song & Xiao , 2016 ; 2014 ; Huang & You , 2016 ; Tchapmi et al. , 2017 ) . MVCNN exemplifies this strategy by applying a shared CNN to many rendered views and max-pooling to aggregate features ( Su et al. , 2015 ) . Subsequent approaches include RotationNet which trains the network to also predict the viewpoint for each image ( Kanezaki et al. , 2018 ) , GVCNN which groups features from subsets of views together before aggregating into a final prediction ( Feng et al. , 2018 ) , and hypergraph methods that consider the correlation across training samples ( Zhang et al. , 2018 ; Feng et al. , 2019 ) . One notable exception is Qi et al . ( 2016 ) , who use a multi-resolution variant of MVCNN , but instead of object meshes , use a voxelized version of the object for rendering . In contrast to the prior view-based methods that use object meshes with point connectivity information , and render images using basic shading and/or depth ; SimpleView takes as input raw point clouds . Another class of methods is voxel-based methods that convert points to a fixed 3D grid instead , which enables the use of 3D CNNs ( Qi et al. , 2016 ; Wu et al. , 2015a ; Maturana & Scherer , 2015 ) . Given the added dimension , such methods are usually restricted to a much lower resolution to represent objects . Though some strategies such as octrees have been used to address those limitations ( Wang et al. , 2017 ) , the advantages to processing 3D data directly in this manner do not yet appear to outweigh the additional overhead introduced .

Summary: This paper discusses several protocols including data augmentation, point distribution, loss function, ensemble scheme, and testing models, which serves as a kindly reminder that the training protocol matters. As claimed, earlier work like PointNet++ can still achieve comparable performance to more recent methods. Such observations are useful, as different methods are supposed to be developed and measured under a unified setting. Moreover, the authors investigated a new projection-based SimpleView method by converting point clouds into depth images, achieving SOTA performance without pretrained CNNs.