Revisiting Point Cloud Classification with a Simple and Effective Baseline
Authors: Ankit Goyal
Hei Law
Bowei Liu
Alejandro Newell
Jia Deng
Venue:  (2021)
Keywords: 3D Vision
Point Cloud Processing

Abstract: Processing point cloud data is an important component of many real-world systems. As such, a wide variety of point-based approaches have been proposed, reporting steady benchmark improvements over time. We study the key ingredients of this progress and uncover two critical results. First, we find that auxiliary factors like different evaluation schemes, data augmentation strategies, and loss functions, which are independent of the model architecture, make a large difference in performance. The differences are large enough that they obscure the effect of architecture. When these factors are controlled for, PointNet++, a relatively older network, performs competitively with recent methods. Second, a very simple projection-based method, which we refer to as SimpleView, performs surprisingly well. It achieves on par or better results than sophisticated state-of-the-art methods on ModelNet40, while being half the size of PointNet++. It also outperforms state-of-the-art methods on ScanObjectNN, a real-world point cloud benchmark, and demonstrates better cross-dataset generalization.


Under review as a conference paper at ICLR 2021
REVISITING POINT CLOUD CLASSIFICATION WITH A SIMPLE AND EFFECTIVE BASELINE
Anonymous authors
Paper under double-blind review
ABSTRACT
Processing point cloud data is an important component of many real-world sys- tems. As such, a wide variety of point-based approaches have been proposed, reporting steady benchmark improvements over time. We study the key ingredi- ents of this progress and uncover two critical results. First, we find that auxiliary factors like different evaluation schemes, data augmentation strategies, and loss functions, which are independent of the model architecture, make large dif- ference in performance. The differences are large enough that they obseure the effect of architecture. When these factors are controlled for, PointNet++, rel- atively older network, performs competitively with recent methods. Second, a very simple projection-based method, which we refer as Simple performs surprisingly well. achieves on par or better results than sophisticated state-of- the-art methods on ModelNet40 while being half the size of PointNet++. It also outperforms state-of-the-art methods on ScanObjectNN, a real-world point cloud benchmark, and demonstrates better cross-dataset generalization.
INTRODUCTION
Processing 3D point cloud data accurately is crucial in many applications including autonomous driving NNavarro-Sermee et al. 2010) and robotics (Rusu et 2009). In these settings, sensors like LIDAR produce unordered sets of points that correspond to object surfaces. Correctly classifying
objects from this data is important for 3D scene understanding (Uy et al., 2019). While classical approaches for this problem have relied on hand-crafted features (Arras et al. (007), recent efforts have focused on the design of deep neural networks (DNNs) to learn features directly from raw point cloud data (Qi al. (017a). Deep learning- based methods have proven effective in aggregating information across a set of 3D points to accurately classify objects.
The most widely adopted benchmark for comparing methods for point cloud classification has been ModelNer40 (Wu et al.) 2015b). The accuracy on ModelNet40 has steadily improved over the last few years from 89.2% by PointNet (Qi al. (017a) 93.6% by RSCNN (Liu et al. (019c) (Fig. [.. This progress is commonly perceived to be a result of better designs of network architectures. How- ever, after performing careful analysis of recent works we find two surprising results. First, we find that auxiliary factors including differing evaluation schemes, data augmentation strategies, and loss functions affect performance to such a degree that it can be difficult to disentangle improvements due to the network architecture. Second, we find that very simple projection-base based architecture works surprisingly well, outperforming sfate-of-hhe-ae art point-based architectures.
In deep learning, as results improve on a benchmark, attention is generally focused on the novel architectures used to achieve those results. However, there are many factors beyond architecture design that influence performance including data augmentation and evaluation procedure. We refer to these additional factors as method's protocol. A protocol defines all details orthogonal to the network architecture that can be controlled to compare differing architectures. Note that it is possible for some specific form of loss or data augmentation to be tied to a specific architecture and inapplicable other architectures. In these cases, it would be inappropriate the protocol. However, for all the methods we consider in this paper, their losses and augmentation schemes are fully compatible with each other and can be considered independently.
treat them part of
We do experiments to study the effect of protocol and discover that it accounts for a large difference in performance, large as to obscure the contribution of novel architecture. For example, the

---

Under review as a conference paper at ICLR 2021
SO- 56
DunsePo Corn CaccHiN2
norm
Depth
SpiderCNe Qquntts
DODMWY
AA CNN
PORMAENL
SO-Met
KCNEX
PrarcNete+
Modentand
30
69-
@handNet
88
ResNeett
FC Layar
Fune
dhar
Dn 17
Aa 17
Hen 10
10
ja 10
Jmn 20
Jo 120
Output
Figure I: Performance of point- based models on ModelNet40. Those using > 1024 points nor- mals are marked with triangle. Line joins the top- performing models across times.
Figure 2: The CompleView Architecture. The depth images are colored only for illustration. Simple takes in single channel depth im- ages as input.
performance of the PointNet++ architecture (Qi et al. 2017b) jumps from 90.0=0.3 to 93.3+0.3, when switching from its original protocol to RSCNN 's protocol (Liu et al. (019c). We further find that the protocols that lead to the strongest performance rely feedback from the test set, which differs from conventional evaluation setups. We re-evaluate prior architectures using the best augmentation and loss functions, while not using any feedback from the test set. We find that by taking protocol into account, the PointNet++ architecture performs competitively with more recent ones various settings.
In addition to the surprising importance of protocol, in reviewing past approaches, another surprising discovery is that a very simple projection based baseline works very well. One needs to simply project the points to depth maps along the orthogonal views, pass them through light-weight CNN and fuse the features. We refer to this baseline as SimpleViews View.
Compared to previous projection-based method (Roveri et al. 2018 Sarkar et al. 2018) for point- cloud classification, SimpleView very simple. Prior methods have developed special modules for view selection, rendering, and feature merging, well as use larger CNN backbones that are pretrained on ImageNet (refer Sec. [ for more details). In contrast, Simple has no such special operations, and only requires simple point projections, much smaller CNN backbone, and no ImageNet pretraining.
The discovery of Simple View is surprising because recent state-of-the-art results have all been achieved by point-based architectures of increasing sophistication. In recent literature, is often assumed that point-based methods are the superior choice for point-cloud processing they "do not introduce explicit information loss" (Guo et al.) 2020). Prior work has stated that "convolution oper- ation of these methods lacks the ability capture nonlocally geometric features" (Yan al.) (2020), that a projection-bass method "often demands huge number of views for decent performance" (Liu al. 2019c), and that projection-base methods often "fine-tune a pre-trained image-based archi- tecture for accurate recognition" (Liu et al.] (019c).. It is thus surprising that a projection-base method could achieve state-of-the-e the-art results with a simple architecture, only few views, and no pretraining.
On ModelNet40, Simple performs on par or better than more sophisticated state-of-the-art networks across various protocols, which includes the ones used by prior methods (Table. 3) as well as our protocol (Table. (5). At the same time, Simple View outperforms state-of-the titthe-art architectures on ScanObject (Uy et al. 2019), real-world dataset where point clouds are noisy (background points, occlusions, holes in objects) and are not axis-aligned. Simple View also demonstrates better cross-dataset generalization than prior works. Furthermore, Simple View uses less parameters than state-of-the-ane networks (Table. 5).
Note that are not proposing new architecture method, but simply evaluating simple and strong projection-base based baseline for point-cloud classification that largely ignored in the literature. We do not claim any novelty in the design of Simple View because all of its components have ap- peared in the literature. Our contribution is showing that such a simple baseline works surprisingly well, which is result absent in existing literature.
2

---

Under review as a conference paper at ICLR 2021
It is worth noting that one might think that projection- based methods are not directly comparable with point-based methods because projection-base based methods may have the full mesh as input, as opposed to just point cloud. While this is true for existing results in the literature, it is not the case with SimpleView, whose input is the exact same point cloud given to a point-based method. In other words, Simple View is directly comparable point-hased method because they solve the exact same task.
In summary, our contributions are threefold:
We show that training and evaluation factors independent of network architecture have a large impact on point-cloud classification performance. With these factors controlled for, PointNet++ performs as well as more recent architectures.
We demonstrate how Simple View, very simple projection based baseline performs surprisingly well on point-cloud classification. It performs on par with or better than prior networks on Mod- elNet40 while using fewer parameters. It also outperforms state-of-the-ar methods on real-world point-cloud classification and achieves better cross-dataset generalization.
2 RELATED WORK
Point-Based Methods for Point-Clout Analysis: A broad class of DNNS have emerged to process
3D points directly (Simonovsky & Komodakis, 2017 Zaheer et al., 2017: Klokov & Lempitsky
2017 Xu et al. 2018]Atzmmot
al. [2018]] Wang et Wang
[2018a] Li et al.) [201 Groh et al. 2018
Ben-Shabat et al.
2018
Xie et al.,
2018 Li et al. 2018b) Liu et al. 2019a Thomas et al. 2019
Komarichev et al.
2019
Liu al.
2019b Yan et al. 2020 Su et al., 2018 Zhang et al. 2019
Liu
et 2019 Atzmon et al. 2018). 8). PointNer (Qi et al. (017a) proposed one of the first strategies,
where features are updated for each point with MLP layers, and aggregated with global max pool- ing. However, local comparisons are performed in PointNet, which motivates PointNet++(Qii (ct al.] (201b)) PointNet++ breaks subsets of points into local regions that are processed first. More explicit modeling of the spatial relations between points is performed with more recent methods (Li jet al. 2018b) Liu et al. 2019c Wu et al., 2019). For example, PointCony learns functions to define continuous 3D convolutions that can be applied to arbitrary sets of points in neighborhood (Wu (et al. 2019). RSCNN uses MLPs conditioned on the spatial relationship of two points to update and aggregate features around individual sampled point (Liu al. 2019c). There exist many variations to these methods, but the emerging trend is an increase in sophistication.
Projection-Based Methods for Point-Cloud Classification: Projection- based methods for point cloud classification have been proposed in the literature. Notably, Roveri et al. (2018) learn to pre- dict viewing angles and classify images in an end-to-end differentiable way. They use the ResNet50 model, pretrained on ImageNet as their backbone and a depth-image generation pipeline. /aarkrr al. (2018) propose a special multi-height rendering and feature merging scheme, and use larger backbone network pretrained on ImageNet. AAmed et al. (2019) manually define important views for each object category, create binary edge maps, and train an ensemble of PointNet++ and CNN. However, numbers in Ahmed et al. (2119) are not directly comparable other approaches there is a manual alignment of objects in the test set which is different from the standard ModelNet40 test set. This was confirmed with the authors. It is worth noting that even though prior work has shown sophisti sophisticated operations to be useful for achieving good results, we find that when controlling for method protocols, strong performance can be achieved with fixed orthogonal views, a smaller network, no ImageNet pretraining, and simpler rendering of points.
Projection-Basee Methods for Other Point-Cloud Analysis Tasks: There is a rich litera- ture for using projection-based methods on various point-cloud analysis problems like segmenta- tion (Ladicky et al.) 20100) Tighe & Lazebnik, 2010 Riemmescheee
al. 2014, Qin al. 2018 Dai & Nielner 2018 KaloKerakiss et al.||2011 Tatarchenkko et al.] 2018). reconstruction (Pittaluga et al. 2019) and rendering (Aliev et al. [209)). Notably, Boulch et al. (2017) use point cloud density to create scene meshes, which are then put into a mesh renderer to generate many image views at
different scales. [awin et al. (0177) render scene point cloud from 120 views for different modal- ities like color, depth, and surface normal. Information from multiple modalities is then fused to generate point-wise predictions. For a detailed survey of various projection approaches on differ- ent point-cloud processing tasks, encourage readers to check the recent survey paper by (Guo) jet al., (2020). In this work, Simple serves as a stripped-down projection- based baseline for point-cloud classification that uses few orthogonal views and simple point projections.
3

---

Under review as a conference paper at ICLR 2021
Table 1: Summary of various protocols.
Protocol
Data Augmentation
Model Selection
Loss
Ensemble
Training Points
+ointNer++
jitter, random rotation.
final model
cross-entropy
Rotation Vote
fixed
random scaling and trans.
DGCNN
random scaling and trans.
best test model
smooth-loss
No vote
fixed
RSCNN
random scaling and trans.
best test model
cross-entropy smooth-loss
Repeated Scaling Vote
resampled
Simple View
random scaling and trans.
final model
No votc
fixed
3D shape Analysis using Rendered Images and Voxels: Many works use images rendered from
object meshes for 3D shape analysis (Maturana & Scherer 2015: Wu et al. 2015b Yu et al. 2018
Guo et al.) 2016 Shi et al. 2015 Hackel et al. 2017 Song & Xiao 2016 2014 Huang & You
| et al. 2017). MVCNN exemplifies this strategy by applying shared CNN to many
rendered views and max- max-lioling to aggregate features (Su et al. 2015). Subsequent approaches include RotationNet which trains the network to also predict the viewpoint for each image (Kanezaki al. 2018), GVCNN which groups features from subsets of views together before aggregating into a final prediction (Feng et al., 2018). 8). and hypergraph methods that consider the correlation across training samples (Zhang et 2018 Feng et al./ 2019). One notable exception Q et al. (2016), who use multi-resolution variant of MVCNN, but instead of object meshes, use voxelized version of the object for rendering. In contrast to the prior view-based methods that use object meshes with point connectivity information, and render images using basic shading and/or depth; Simple View takes as input raw point clouds.
Another class of methods is voxel-based methods that convert points to fixed 3D grid instead, which enables the use of 3D CNNs (Qi al., 2016 Wu et al. 2015a| Maturana Scherer 2015). Given the added dimension, such methods are usually restricted to much lower resolution to rep- resent objects. Though some strategies such as octrees have been used to address those limita- tions (Wang et al. 2017)) the advantages to processing 3D data directly in this manner do not yet appear to outweigh the additional overhead introduced.
3 METHOD OVERVIEW
3.1 VARIATIONS IN EXISTING PROTOCOLS
We analyze the key ingredients in the progress in point-cloud classification. Critical to our study is controlling for factors which are independent of network architecture. We refer the factors method's prot ocol. A protocol used by one method can be transferred to another. For our study, analyze subset of the highest performing methods over the past few years. This choice fur- ther based on availability and usability of official source-code. Specifically, we choose PointNet (Qi al. 2017a), PointNet++ (Qi et al. 2017b), DGCNN (Wang al. 2018b) and RSCNN (Liu al.j 2019ch. Note that we also do direct comparisons to networks apart from the ones mentioned here (Table
For our purposes, we do not consider any variations in input, namely the use of surface normals or more than 1024 points. Using normals more points have been shown to improve performance in the literature. Our objective is to study factors that are not commonly perceived as a major source of performance increase. So we scope our analysis to the most widely adopted input scheme which uses 1024 points with only x, U. 2 coordinates.
Data Augmentation: Various data augmentation strategies like jittering, random rotation a along y-axis, random scal scaling and random translaticn innion. Different methods use different combinations of these augmentations. PointNet and PointNet++ use all the above augmentations. However, objects in ModelNet40 are aligned, random rotation along y y-axis adversely affects the performance of a model. Hence recent methods, includ- ing RSCNN and DGCNN, do not use it. They only random translation and random scaling. Some methods including PointCNN make a distinction between whether or not random rotation along y-axis is used, but it is not common practice.
Input Points: PointNet and PointNet++ use a fixed set of 1024 points per object to train the network. We refer to it the fi xed points strategy. RSCNN and PointCNN randomly sample points
4

---

Under review as a conference paper at ICLR 2021
Table 2: DGCNN aug. and smooth loss improve the performance of all architectures
Data Augmentation
Model Selection
Loss
Architecture
PN+
DGCNN
Final
Best Test
C.E.
Smooth
PointNet
PN+
DGCNN
RSCNN
89.7.0.3
91001.33
90.5a0.2
90.410.3
89.0-0.2
89.8+0.2
90.0-0.4
89.4-0.1
89.1+0.2
92.1+0.1
91.1+0.3
91.1+0.3
89.2+0.9
92.7+0.1
91.9+0.2
91.7+0.3
during each epoch, effectively exposing the model to more than 1024 points per object during the training process. We refer to this as the resampled points strategy.
Loss Function: cross-ent (CE) is used by most of the methods. However, DGCNN uses smooth- css, where the ground-truth labels smoothed out before calculating cross-entropy. We observe that smooth-lose improves the performance of all network architectures.
PointNet and PointNet++ use the final converged model to evaluate on the test set. Since the number of epochs is hyper-parameter that depends on factors like data, model, optimizer, and loss, in our experiments, tune the number of epochs. We then retrain the model with the complete training set to the tuned number of epochs. We refer this strategy as final model selection. We find that some methods including DGCNN and RSCNN evaluate the model on the test set after every epoch and use the best test performance as the final performance. We refer to this strategy best. test model selection.
Selecting Model for Testing:
create validation set from the training set to
Ensemble Scheme: Some methods use an ensemble to further improve the performance. PointNet and PointNet++ apply the final network to multiple rotated and shuffled versions of the point cloud, and average the predictions to make the final prediction. We refer to this strategy as Rotat i on Vote. The shuffling operation induces randomness in prediction for PointNet++ and RSCNN as they are not strictly invariant to the order the points (Sec. 3.3 in Qi et al. (2017b)). Hence, while evaluating Rotation Vote, do the inference 10 times per for PointNet++ and RSCNN, and report mean and standard deviation. Simple and PointNet are invariant to the order of the points and hence are not affected by shuffling. Some methods, including RSCNN and DensePoint, create multiple randomly scaled and randomly sampled versions of a test object. They then evaluate the final network on these multiple versions of the object and average the prediction. Since the scaling is random, it makes the test set performance random as well. RSCNN and DensePoint repeat this procedure 300 times on the test set and report the best accuracy. We refer as Repeated Scall 1 in Vote. DGCNN does not use any ensemble.
Table summmarzes the PointNer++, DGCNN and RSCNN protocols. Besides these three protocols, we also include variants of these protocols in table 3 and table 4 such as PointNet++ no Vote (i.e. PointNer++ but without the Rotation Vote), DGCNN CE (i.e. DGCNN but with CE loss intead of smooth loss), DGCNN CE Final (i.e DGCNN CE but with final model selection instead of best test model selection) and RSCNN no Vote (i.e. RSCNN but without the Rotation Vot These protocols represent prototypical settings and have been used with slight modifications in many other prior works.
For example, DeepSets (Zaheer et al. 2017) used the +ointNet++ Vote protocol without jittering and translation; SO-Net (Li et al., (018a) used the DGCNN CE protocol with jittering and ran- dom scaling instead of random scaling and translation; 3DmFV ShabaSh al. 2018) used the DGCNN CE protocol with additional jittering; PCNN (Atzmon et al. 2018) used the DGCNN CE Final protcof" PointCNN (Li et al. 2018b) used the DGCNN CL protocol with randomly sampled points and small (10") rotation augmentation; DensePoint (Liu et al. (019b) used the RSCNN pro- tocol: PointASL (Yan al., (2020) used the DGCNN CE protocol but with additional point jittering augmentation and voting.
Our Protocol: Based on our findings, we define our Simple protocol, which uses the best aug- mentation and loss functions while not using any information from the test set. 2sows that
It is unclear from code if the hest test or final model selection is used. We assume final model selection to err on the side of caution.
5

---

Under review as a conference paper at ICLR 2021
Table 3: Performance of various architectures ModelNer40. Protocol affects performance by large amount. Simple View performs on par or better than prior architectures across protocols. PointNet++
View
Protocol +
RSCNN
DGCNN
Architecture +
no Vote
Vote
no Vote
Vote
CE
Smooth
PointNet
89.0 a 0.2
89.1 = 0.2
90.0 0.3
90.1 a 0.2
90.1 a 0.2
90.3 0.1
+ointNet++
89.8 + 0.2
90.0 0.3
92.7 0.1
93.3 + 0.3
92.6 + 0.2
93.1 + 0.2
DGCNN
90.0 + 0.4
90.5 +
92.2
92.8 0.5
91.9
92.7 +
RSCNN
89.4 + 0.1
90.2 0.2
92.1
=
91.9
92.5 0.1
Simple
90.7 + 0.3
91.0 0.2
92.9 0.2
93.2 + 0.1 93.1=00
93.6 0.3
Table 4: Performance of various architectures on ModelNet40, Includes prior works not in Table indicates small differences in protocol as identified in Sec. 3.1
Architecture
Points
Closest Protocol
Acc.
PN++ Acc.
Simple Acc.
DeepSets (Zahecr et al.
5000
PointNet++ Vote*
90.0 0.3
89.8 0.2
90.7 0.3
SO-Net et al.
2048
DGCNN CE*
90.9
92.6 0.2
93.1 0.1
3DmFV (Ben Shabbt al.
1024
DGCNN CE*
91.4
92.6 0.2
93.1 1 0.1
PCNN (Atzmon
1024
DCNN CE Final"
92.3
92.1 0.1
92.5 0.3
PointCNN (L.
1024
DGCNN CK*
92.5
92.6 0.2
93.1
DensePoint (Luu et
1024
RSCNN no Vote
92.8
92.7 0.1
92.9 0.2
RSCNN-Muli (Liu al.).
1024
RSCNN no Vote
92.9
92.7 +
92.9 0.2
PointANSL YYan et al.
1024
DGCNN
92.9
92.6 0.2
93.1
DGCNN's augmentation (i.e randon translation and scaling) and smooth-loss improve performance of all prior networks, so them in the SimpleView protocol. Further, similar to PointNet, PointNet++ and DGCNN, we use the fixed dataset of 1024 points instead of re-sampling different points each epoch. Re-sampling points for each epoch effectively increases the training dataset of points, making numbers incomparable to methods using a fixed dataset. We avoid any feedback from the test and the final model selection where first tune the number of epochs on the validation set then retrain the model on the entire train set. Lastly, similar DGCNN, do not use ensemble is more standard Machine Learning to compare models without ensemble.
3.2 SIMPLEVIEW
Given set of points SimpleView, projects them onto the six orthogonal planes to create sparse depth images. It then extracts features from the depth images using CNN and fuses, which is then used to classify the point-cloud as shown in Fig. [I]
Generating Depth Images from Point Cloud: Let (z. y, be the coordinates of a point in the point cloud with respect the camera. We apply perspective projection get the 2D coordinate (i = 2/2.. ū = y/2) of p at depth 2. We also do ablations with orthographic projection and found perspective projection
work better (Table. 7). Since coordinates on image plane have be discrete, we use ([0), 77) ) to be the final coordinate of on the image plane. Multiple points may be projected the same discrete location on the image plane. To produce depth value an image location, we do ablations on two choices, one the minimum depth of all points, and other weighted average depth with more weight ()) given to closer points (Table. 7p. Empirically, we find both perform similar with the later performing slightly better. This could be because of reduction in noise due to the averaging of nearby pixels on the surface. The depth images are of resolution 128 X 128.
View Architecture: To make the number of parameters comparable to point-based methods, use ResNer18 with one-fourth filters (ResNer18/4) as the backbone. For fusing features, we do ablation with two choices, pooling and concatenation. Empirically, we find concatenation to work better than pooling them (Table. M). This could be because pooling features throws away the view information like which views are adjacent to one amother. One concern could that concatenation could make features sensitive to viewpoint, and hence the network could fail rotated objects.

---

Under review as a conference paper at ICLR 2021
Table 5: Performance of various architectures ModelNee40 when using the best data- augmentation and loss function; and not using any feedback from test SimpleView out- performs prior architectures, while having fewest parameters and comparable inference time.
Table 6: Performance of various architectures on ScanObjectNN, and cross-dataset generalization. Simple View achieves state-of-the-ard results and shows better cross dataset generalization. Num- bers for prior works from (Uy et al. 2019).
TR: SONN
TR: MN40
TR: SONN
Acc.
Class
Para.
Time (ms)
Architecture
SONN
TE: SONN
TE: MN40
Architecture +
Acc.
3DmFV (Shabot ct al.)
63.0
$1.5
PointNet (Qi al.
68.2
31.1
50.9
PointNet
89.2
85.1
0.6
3.0
SpiderCNN DXu
73.7
30.9
46.6
PointNet++
92.7
90.0 0.3
20.7
PointNet++ (Qrel at
77.9
32.0
47.4
DGCNN
92.3 0.3
89.1 0.3
1.8
7.3
DOCNN WWang ct
78.1
36.8
54.7
RSCNN
91.7 0.3
88.5 0.4
1.3
4.3
PointCNN
78.5
24.6
49.2
SimpleVier
93.0 0.4
90,5 0.8
0.8
5.0
Simple
79,5-0.5
40.5-14
57,9-2.1
Table 7: Ablation of various choices for SimpleView on ModelNet4O. The performance is evalu-
atated on the validation set.
Number of Views
Image Projection
Peature Fusion
Image Dopth
Orthographic
Perspective
Pool
Concal
Minimum
Weighted Avg.
Accuracy 90.7 92.1 022 92.9 0.3
92.7 0.3
92.9 0.3
91.8 0.3
92.9 0.3
92.8
0.4
92.9 0.3
However, empirically, we observe that this issue is largely mitigated by rotation augmentation and Simple able achieve state-of-the-an performance on ScanObjectNN where objects are rotated. The point-clouds are scaled to be in [1. -1]. we keep the cameras at distance of 1.4 units from the center with 90º fov. We also do ablations with different number of views, comparing only front views, three orthogonal views and six orthogonal views. We find that using all six views performs the best (Table 7). We do not use ImageNet pretraining, thus making the comparison with point-based methods strictly fair, without any additional data.
4 EXPERIMENTS
ModelNet40 is the most widely adopted benchmark for point-cloud classification. It contains objects from 40 common categories. There are 9840 objects in the training set and 2468 in the test set. Objects are aligned to a common up and front direction.
ModelNet40:
ScanObjeetNN:
ScanObjectNN is a recent real-world point cloud classification dataset. It consists of 15 classes, 11 which are also in ModelNet40. There are total of 15k objects the dataset. Unlike ModelNet40, the objects in ScanObjectNN are obtained from real-world 3D scans. Hence, point clouds are noisy (ocelusions, background points) and have geometric distortions such as holes. Also, unlike ModelNet40, the objects
axis-aligned.
4.1 EXPERIMENTS ON NET40
Implementation Details:
We use PyTorch (Paszke al. 2019) implement all models and proto- cols while reusing the official code wherever possible. We use the official version of DGCNN and RSCNN, We confirm with the authors that the code for RSCNN-Multi, another version of RSCNN, is yet to be released. Hence we use the reported numbers of RSCNN-Mult in Table PointNet and PointNet++ are officially released code from PointNet. PointNet.pytorch pytorch (Xial accessed June, 2020) as recommended in the official repository. For PointNet+++, we adapt the model code from Pointner2_PP PyToch (Wijmans accessed June, 2020). We further make sure that the third party PyTorch code closely matches the official TensorFlow code.
TensorFlow (Abadi et al. 2015). For PointNet, we adapt our
We use Adam (Kingma & Ba 2014) with initial learning rate of le-3 and decay-on-plateau learning rate scheduler. The batch size and weight decay for each model are kept the same as the official version in Table 3 We use batch size of 18 and weight decay for SimpleView To give the prior models the best chance on our protocol (Table 5). we additionally tune their hyper- parameters on the validation set. We find that the official papar-passmeters already perform close to

---

Under review as a conference paper at ICLR 2021
Drowerothh Ronch Eioplorion: chair binnlettii Ieand
orouncorhh tre Bing ow vase 20datfaci12 Varn
Browtroth: Flove not steplerion: Plant beintllatr Flast
Crountruth: Plans dieglevion hall Peintfacit: Deach
Growntruth: tresser stepletious Table Deintiles Siges Stard
Crcuntroch: Capp stepletiows Mardrote foimcletes Hazdrabe
Scoustruthc Bookate.1 stplemtee Bookate. Tointletees
Grountretst FLower the Sirplatiios: Flower the Polechutee
Grountruthe Piano Simplettien Piano Beinthecees Staies
Gcounsruth: larp stapletion: lerp Peinthecres
Coountrech: Table Sirpletioet Table Tolestetee: Best
Gocuntruth: Maae Simplefiees Mrne Beinthet**:
Gecuscrorbi Down sinplaviewi Tont PointMetree
Growstruth Hight stand Grouscrurhh plast staplociow: #loone Fot Poonthletrte Plant
Grouecruth: aink steplattons rolloc Deinthecrts Dank
Grosntruth Vaee
eecuatruch; Xoox strplaviowi Waodrobe Desntllerrit Xoox
staploviows Geaseay PountMebree Higst Stand
steplattins Lamp Tokroheer11 Mane
Figure 3: Failure Cases for SimpleView and PointNet+++ The first row shows cases where both Vimp and PointNet++ fail; the second row shows cases where Simple View succeeds but PointNet++ fails; the third row shows cases where SimpleView fails but PointNet++ succeeds.
optimal. We train each model for 1000 epochs. Since there are small variations in final performance across different runs, we runs and report mean and standard deviation.
Performance under various Prior Protocols: Table 3 shows the performance different architec- tures under various protocols. The mean performance PointNet++ improves from 89.8% to 93.3% when switch from the PointNer+ no Vote the RSCNN Vote protocol. Similarly the perfor- mance of SimpleView improves from 90.7% to 93.6% when we switch from PointNer+ no Vote to DGCNN Smooth. Since there is variance in performance any claims about absolute ordering between prior works. However, we do observe that terms of mean performance, SimpleView performs on par better than other methods under all protocols. Note that in RSCNN Vote, voting on the test set is done 300 times with reshuffled and randomly augmented points, from which the highest accuracy is selected. Hence models that have the largest variance in prediction, i.e. PointNet++ and RSCNN gain the most from it, as they are not strictly invariant the order points (Sec. 3.3 in Qi
runs, refrain from making
(2077b).
Performance under the Simple View Protocol: Table 5 shows that Simple View outperforms prior architectures on our controlled protocol in terms of mean performance. Simple View has the fewest number of parameters and a competitive inference speed. Inference speed is measured on an NVIDIA 2080Ti averaged across 100 runs.
Fig. 3 show examples where both Simple and PointNet++ fail, as well as examples where one of them fails and the other succeeds. Qualitatively, we find that the failure modes of Simple View and PointNet++ similar. We also find that major failure mode in both Simple and PointNet++ the confusion between the 'fowwerpoo' and 'plant' category (see Sec. Fig. and Fig. II] This could be because of the lack of color information.
Comparison with More Methods: Table 4 we do one-on-ome comparison between Simple View and recent ofate-of-the-ae methods, other than PointNet, PointNet++, RSCNN and DGCNN. We identify the closest protocol to the one used in the paper from the ones we evaluate. Table A shows the competitiveness of PointNet++ and Simple with other recent state-of-the-art methods.
8

---

Under review as a conference paper at ICLR 2021
4.2 EXPERIMENTS ON SCANOBJECTNN
Implementation Details: ScanObjectNN's official repository trains and evaluates the state-of-the- art models under the same protocol. We implement SimpleView in TensorFlow and use the official ScanObjectNN protocol for fairness. This protocol is different from the Simple View protocol as it normalizes the point clouds and randomly samples points. We optimize our model with Adam. We use batch size of 20 and no weight decay to train Simple View for 300 epochs with initial learning rate 0.001, and use the final model for testing- We use standard image-based cropping and scaling augmentation to prevent ever-fittingg The hypeerparameter for cropping and scaling is found on a validation set made from ScanObjecNN's train set. We conduct 4 runs for Simple View. ScanObjectNN does not use fixed set of points during test time. It instead randomly samples points from the point cloud, which adds randomness to test set performance. Hence, we evaluate each run 10 times. We report the final performance the mean and variance of the 40 evaluations (4 runs x 10 evaluations per run).
Performance ScanObjectNN: As shown in Table 6 Simple View outperforms prior networks on ScanObjectNN. This shows the Simple View is effective in real world settings, with noisy and mis- aligned point clouds. We also perform transfer experiments to test generalizability of SimpleView: We train on ScanObjectNN and test on ModelNet40 and vice versa. Table that Simple View transfers across datasets better than prior methods.
DISCUSSION
In this work, we demonstrate how auxiliary factors orthogonal to the network architecture have a large effect on performance for point-cloud classification. When controlling for these factors, we find that relatively older method, PointNet++ (Qi et al. 2017b), performs competitively with more recent ones. Furthermore, show that simple baseline pertorms par better than state-of- the-art architectures.
Our results show that for future progress should control for protocols while comparing network architectures. Our code base could serve as a useful resource for developing new models and com- paring them with prior works. Our results show that the existing evidence for point-based methods not as strong when auxiliary factors are properly controlled for, and that SimpleView is a strong baseline. But our results are not meant discourage future research on point-based methods. is still entirely possible that point-based methods come out ahead with additional innovations. We believe it is beneficial to explore competing approaches, including the ones that are underperforming at particular time, as long as the results are compared in a controlled manner.
Our analysis in this work was limited point cloud classification, which is an important problem in 3D scene understanding and forms critical part of object detection and retrieval systems. An exciting future direction would to expand this analysis to other problems that involve point cloud data such as scene and part segmentation.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shiens, Benoit Steiner, Ilya Sutskever, Kunal Talwar. Paul Tucker, Vin- cent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Watten- berg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. |t..ttps. able from tensorflow.org.
htt.::ttps............t /wwww//wwww./////////////. tensor f low. org/) Software avail-
Syeda Mariam Ahmed, Pan Liang, and Chee Meng Chew. Epn: Edge-aware pointnet for object recognition from multi-view 2.5 d point clouds. IROS, 2019.
Kara-Ali Aliey, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. Neural paset-based graphics. arXiv preprint arXiv: 19Xi::/906.082000 2019.

---

Under review as a conference paper at ICLR 2021
Kai O Arras, Oscar Martinez Mozos, and Wolfram Burgard. Using boosted features for the detection of people in 2d range data. In Proceedings 2007 IEEE International Conference on Robotics and Automation, pp. 3402-3407, IEEE, 2007.
Matan Atzmon, Haggai Maron, and Yaron Lipman. Point convolutional neural networks by exten- sion operators. arXiv preprint arXiv: 10091, 2018.
Yizhak Ben-Shabat, Michael Lindenbaum, and Anath Fischer. 3dmfv: Three-dimensional point cloud classification in real-time using convolutional neural networks. IEEE Robotics and Au- tomation Letters, 2018
Alexandre Boulch, Bertrand Le Saux, and Nicolas Audebert. Unstructured point cloud semantic labeling using deep segmentation networks. 3DOR, 2:7, 2017.
Angela Dai and Matthias Nießner. 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 452- 2018.
Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao. Gvenn: Group-view convolu- tional neural networks for shape recognition. In CVPR, pp. 264-272, 2018.
Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3558-3565, 2019.
Fabian Groh, Patrick Wieschollek, and Hendrik P. A. Lensch. Flex-convolution (million-scale point- cloud learning beyond wrid-worlds). In ACCV, 2018.
Haiyun Guo, Jingiao Wang, Yue Gao, Jianqiang Li, and Hanqing Lu. Multi-view 3d object retrieval with deep embedding network. IEEE Transactions on Image Processing, 25(12):5526-5537, 2016.
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Timo Hackel, Nikolay Savinov, Lubor Ladicky, Jan D Wegner, Konrad Schindler, and Marc Polle- feys. Semantic3d. net: A new large-scale point cloud classification benchmark. arXiv preprint arXiv:1704.03847, 2017.
Jing Huang and Suya You. Point cloud labeling using 3d convolutional neural network. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2670-2675. IEEE, 2016.
Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 3d shape segmentation with projective convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3779-3788, 2017.
Asako Kanezaki, Yasuyuki Matsushita, and Yoshifumi Nishida. Rotationnet: Joint object catego- rization and pose estimation using multiviews from unsupervised viewpoints. In CVPR, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988. 2014.
Roman Klokov and Victor Lempitsky. Escape from cells: Deep kd-networks for the recognition of 3d point cloud models. In ICCV, 2017.
Artem Komarichev, Zichun Zhong, and Jing Hua. A-cnn: Annularly convolutional neural networks on point clouds. In Proceedings of the IEEE Conference Computer Vision and Pattern Recog- nition, pp. 7421-7430, 2019.
L' Luubor Ladický. Paul Sturgess, Karteck Alahari, Chris Russell, and Philip HS Torr What, where and how many? combining object detectors and erfs. In European conference on computer vision, pp. 424 437. Springer, 2010.
10

---

Under review as a conference paper at ICLR 2021
Felix Järemo Lawin, Martin Danelljan. Patrik Tosteberg, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. Deep projective 3d semantic segmentation. In International Conference on Computer Analysis of Images and Patterns, pp. 95-107. Springer, 2017.
Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self Self-ozgani network for point cloud analysis. In CVPR, 2018a.
Yangyan Li, Rui Bu, Mingchao Sun, and Baoquan Chen. Pointenn: Convolution x-transformed
points. Advances in Neural Information Processing Systems, 2018h. 8b.
Xinhai Liu, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Point2sqquence: Learning the shape representation of 3d point clouds with an attention-based sequence to sequence network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8778-8785 2019a.
Yongcheng Liu, Bin Fan, Gaofeng Meng, Jiwen Lu, Shiming Xiang, and Chunhong Pan. Dense- point: Learning densely contextual representation for efficient point cloud processing. In Pro- ceedings of the IEEE International Conference on Computer Vision, pp. 5239-52999330 5248, 2019b.
Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In CVPR, pp. 8895-8904, 2019c.
D. Maturana and S. Scherer. VoxNet: 3D Convolutional Neural Network for Real-Time Object Recognition. In IROS, 2015.
Luis E Navarro-Sermmett Christoph Mertz, and Martial Hebert. Pedestrian detection and tracking using three-dimensional ladar data. The International Journal of Robotics Research, 29(1)): 1516-1528, 2010.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veto,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp.
2019.
Francesco Pittaluga, Sanjeev J Koppal, Sing Bing Kang, and Sudipta N Sinha. Revealing scenes by inverting structure from motion reconstructions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 145-154, 2019.
Charles R. Qi, Hao Su, Matthias Niessner, Angela Dai, Mengyuan Yan, and Leonidas J. Guibas. Volumetric and multi-view cnns for object classification on 3d data. In CVPR, 2016.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. CVPR, 2017a
Charles R Qi, Li Yi. Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems, 2017b.
Nannan Qin, Xiangyun Hu, and Hengming Dai. Deep fusion of multi-view and multimodal repre- sentation of als point cloud for 3d terrain scene recognition. ISPRS journal of photogrammetry and remote sensing, 143:205-212, 2018.
Hayko Riemenschneider, András Saaii-Szomorú, Julien Weissenberg, and Lue Van Gool. Learning where to classify in multi-view semantic segmentation. European Conference on Computer Vision, pp. 516-532. Springer, 2014.
Riccardo Roveri, Lukas Rahmann, Cengiz Oztireli, and Markus Gross. A network architecture for point cloud classification via automatic depth images generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4176-4184, 2018.
Radu Bogdan Rusu, Nico Blodow, Zoltan Csaba Marton, and Michael Beetz. Close-range scene segmentation and reconstruction of 3d point cloud maps for mobile manipulation in domestic environments. In 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1-6. IEEE, 2009.
11

---

Under review as a conference paper at ICLR 2021
Kripasindhu Sarkar, Basavaraj Hampiholi, Kiran Varanasi, and Didier Stricker. Learning 3d shapes as multi-layered beight-maps using 2d convolutional networks. In Proceedings of the European Conference Computer Vision (ECCV), pp. 71-86, 2018.
Baoguang Shi, Song Bai, Zhichao Zhou, and Xiang Bai. Deeppano: Deep panoramic representation for 3-d shape recognition. Signal Processing Letters, IEEE, 22(12)2339-2343 2015.
Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neu- ral networks on graphs. In CVPR, 2017.
Shuran Song and Jianxiong Xiao. Sliding shapes for 3d object detection in depth images. In Euro- pean conference computer vision, pp. 634-651. Springer, 2014.
Shuran Song and Jianxiong Xiao. Deep sliding shapes for amodal 3d object detection in rgb-d images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 808-816, 2016.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learnd-Miller. Multi-view convo- lutional neural networks for 3d shape recognition. In ICCV, 2015.
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, and Jan Kautz. Splatnet: Sparse lattice networks for point cloud processing. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 2530-2539, 2018.
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian-Y Yi Zhou. Tangent convolutions for dense prediction in 3d. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3887-3896, 2018.
Lyne Tchapmi, Christopher Choy, Iro Armeni, Jun Young Gwak, and Silvio Savarese. Segeloud: Semantic segmentation of 3d point clouds. In 2017 international conference on 3D vision (3DV), pp. 537-547. IEEE, 2017.
Hugues Thomas, Charles R Qi, Gean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6411-6420, 2019.
Joseph Tighe and Svetlana Lazebnik. ECCV, 2010.
Scalable nonparametric image parsing with superpixels.
Mikaela Angelina Uy, Quang- Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revis- iting point cloud classification: A new benchmark dataset and classification model on real-world data. Proceedings of the IEEE International Conference on Computer Vision, pp. 1588-1597, 2019.
Chu Wang, Babak Samari, and Kaleem Siddiqi. Local spectral graph convolution for point set feature learning. ECCV, 2018a.
Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. O-enn: Octree-based convolutional neural networks for 3d shape analysis. ACM Transactions Graphics (TOG), 36 (4):1-11, 2017.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph enn for learning on point clouds. arXiv preprint 1rXiv:1801.07829 2018b.
Erik Wijmans. PointNer2 pyrorh,, accessed June, 2020. URL https:///ithunt com/ erikwi jmans/Pointnet: Bttpiieeett____och_____ PyTorch]
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9621-9630, 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR, 2015a.
12

---

Under review as a conference paper at ICLR 2021
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912- 2015b.
Fei Xia. prrtrcet,pytorch, accessed June, 2020. URL httpss//githuu.ccmm//X.///
cointnet.pytorg
Saining Xie, Sainan Liu, Zeyu Chen, and Zhuowen Tu. Attentional shapecontextnet for point cloud recognition. CVPR, 2018.
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning on point sets with parameterized convolutional filters. In ECCV, 2018.
Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, and Shuguang Cui. Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling. CVPR, 2020.
Tan Yu, Jingjing Meng. and Junsong Yuan. Multi-view harmonized bilinear network for 3d object recognition. CVPR, 2018
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems, 2017.
Kuangen Zhang, Ming Hao, Jing Wang, Clarence W de Silva, and Chenglong Fu. Linked dy- namic graph cnn: Learning on point cloud via linking hierarchical features. arXiv preprint arXiv:1904.10014 2019.
Zizhao Zhang, Haojie Lin, Xibin Zhao, Rongrong Ji, and Yue Gao. Inductive multi-hypergraph learning and its application on view-based 3d object classification. IEEE Transactions on Image Processing, 27512)55957-5568 2018
13

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
Figure I: Confusion matrix for Simple View when trained under our protocol
Figure II: Confusion matrix for PointNet++ when trained under our protocol
14

---

Under review as a conference paper at ICLR 2021
Table I: Performance of various architectures ModelNet40 when using different amount of train-
ing data.
Per. of Training Data
RSCNN
DGCNN
PointNet
PointNet++
SimpleView
25 %
88.2 4: 0.4
89.1 a 0.2
86.3 0.4
89.6 a: 0,4
89.7 a 0.3
50 %
90.4 a 0.4
91.0 0.3
88.2 0.3
91.5 0.2
92.1 0.3
100 %
91.7 + 0.3
92.3 + 0.3
89.2 + 0.9
92.7 0.3
93.0 + 0.4
15


Reviews:
[{"review_id": "hwCWYDgyPbp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIn this paper, the author(s) do a careful analysis on the classification performance of various modern point cloud processing networks and show empirically that with evaluation protocol set the same for different models, PointNet++, which is a relatively old model, has similar or better performance than newly proposed methods. The author(s) also show a simple projection based baseline SimpleView that can work surprisingly well on point cloud classification task. They evaluate methods on ModelNet40 and ScanObjectNN datasets.\n-----------------\nPros:\n1. The paper is well written and easy to follow.\n2. The author(s) conduct detailed break-downs on the evaluation protocols used by various modern point processing models, which I think is valuable to research on point cloud data. It is also surprising that factors beyond architecture design can make such a difference in the evaluation. I think following the same evaluation protocol can be very helpful for subsequent research. This also indicates evaluating point processing models solely on ModelNet40 might not be a good practice since it poses bias to some augmentation/architecture, which might be false if models are applied in real-world data. These messages are worth to be known by the community.\n3. The author(s) also show that a surprisingly simple baseline that makes use of different view projections can work quite well on point cloud object detection.\n-----------------\nCons:\n1. It would be nice if the author(s) can include some failure mode analysis on different models. Since the SimpleView does not operate directly on point clouds, it might have distinctive failure modes compared with other models. It might be helpful for designing point cloud processing models in the future.\n2. It is unsure how such projection-based methods, like the MVCNN and SimpleView, handle larger scenes (e.g. with covered objects or a scene scan in autonomous driving scenarios) and per-point tasks such as part segmentations.\n-----------------\nMisc:\n1. Will you release source code?\n2. The paper format seems not complying with ICLR format, especially the font?\n3. The overall figure and the projected images in figure 2 do not match？\n4. Page 6 last line, choice -> choices\n-----------------\nPost-rebuttal review\n\nI carefully read through the rebuttal and other reviews and I would like to keep my original rating. The author(s) addressed my concerns and I think it is a good paper for the community.", "extra_scores": {}, "extra_text": {}}, {"review_id": "er2b-LloK_b", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the factors that are related to point cloud classification but independent of model architecture. Then a light-weight projection-based model is proposed. Substantial experiments are conducted to show how the auxiliary factors affect the evaluation results and the proposed method can perform at similar level compared with state-of-the-art methods.\n\nOn the whole, I think this is a good paper as it addresses one of the most important problem of current research in point cloud classification.  We need to distinguish what actually brings the performance improvement: is it the evaluation scheme or the new model architecture itself. Extensive experiments show that 1) different methods indeed use different protocols and 2) data augmentation, loss function, voting play important role in the model performance. It's valuable to know what kind of data augmentation and loss function can improve performance generally in controlled setting. The paper is well written, rigorous and easy to follow. I think this paper would be a good contribution to the community. \n\nSome questions I have:\n(1)What's the reason that performances of architectures in Table3 under RSCNN/DGCNN outperforms the performance in Table4? Is it because RSCNN/DGCNN use feedback from test set?\n(2)For performance in Table 6,  do you train the the state-of-the-art method with the same protocol for SimpleView?\n(3)Have you tried what the performance would be given different amount of data for different architecture under same protocol?\n(4)For other more sophisticated operations in related projection-based methods, have you tried to see if they are useful under same protocol?\n\n--------\nAfter discussion:\n\nAfter reading the author's response as well as the opinions from other reviewers, I will stick to my original rating. The authors resolve most of my questions and concerns and I look forward to a revised version of the paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "auip3k7_STE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper systematically analyzes performance of point cloud classification methods with strict control over the setup and hyperparameters including data augmentation, loss, input representation, model epoch selection. Through extensive experiments, this paper reveals that there seem no real improvements due to the proposed model architecture in the last two years and all performance improvements come from the training/testing setup and hyperparameter tuning. \n\nPaper Strengths:\n1. This paper provides important findings of the source of performance improvements in the domain of point cloud classification. If the findings are correct, it will imply that many efforts prior work has been put in terms of innovating new network architecture are not very effective as we expected, sometimes might be less useful than using a better training/testing protocol. \n2. The presentation (language and logic) of the paper is clear (though part of the reason is that there is no involving technical description ), I really enjoy reading this paper, and I appreciate the authors for their efforts.\n3. The provided experiments are really extensive, with 7 tables and 2 datasets, which makes the findings quite convincing. \n\nPaper Weaknesses:\n1. As this paper has written, there is no scientific novelty in the paper. In my mind, It is totally okay to not propose any new method/architecture but it would be nice to see a solution to the revealed problem. Specifically, this paper has revealed an important problem that prior work in innovating network architecture does not seem to clearly bring improvements. However, there is no clear solution/direction proposed in the paper to this problem. What should we do as a community next? Should researchers in point cloud classification stop innovating these useless and sophisticated network architectures? Should we just limit our research by exploring different types of data augmentation? I would guess the answer to these questions is no. But then, what should we do? It would be nice if the paper can bring further insight into this direction\n2. The paper’s findings/claims are mostly based on experiments, so it would be nice if the paper can release its code to be used by the community and validate if it is really true that the efforts put in neural network architecture are not useful in terms of performance in point cloud classification. Without careful validation by the community, the strong claim made in this paper could be misleading and discouraging further research for innovating new approaches in this domain\n3. Although many quantitative experiments are provided, it would be nice if this paper can add more qualitative analysis to help readers understand where the improvements are. \n4. I believe there are 100+ methods proposed in the recent 3 years at many conferences/journals for point cloud processing and classification, but only 4 are evaluated in this paper. It would be nice if the authors can add more representative baselines to validate the claims in order to double confirm if it is true that there is no performance gain from the architecture side. It might be possible that the claim is true within the compared 4 baselines and there might be other methods which really improves performance solely because they have a better network\n\nDetailed Comments\n1. It might be good to combine the third contribution in the introduction with the second one?\n2. In the section of “our protocol”, this paper uses the fixed set of 1024 points instead of re-sampling different points at each epoch. Why? As the paper described, using the re-sampling point strategy can effectively increase the training dataset of points, which is good. It would be nice to justify why not using such a good strategy. Also, this strategy is fair unlike the one using feedback from the test set which is kind of cheating. \n3. In the section of “generating depth images from point cloud”, this paper uses two strategies, one using the minimum depth of all points which makes sense to me. But the latter one which uses a weighted average of depth does not seem to be reasonable as it is essentially creating imaginary non-existing points which correspond to the averaged depth\n\nJustification:\n\nMy decision is made mainly because I feel the findings in this paper are important and need to be shared with the community. Sometimes, people care too much about novelty and try to design more and more sophisticated networks. If it turns out these innovative networks do not really bring performance gain, we need to re-think what we should do. However, there are spaces to be improved in this paper as I pointed out above. It would be nice to see some revision in the final version if the paper is accepted, including potential solutions/directions about where we should go and more analysis (qualitative and baselines) to support the claim.\n\nPost-rebuttal review\n\nAfter reading the authors' responses and other reviews, I would like to stick to my original rating to accept this paper. Though there are minor problems that I still have concerns (e.g., why not using re-sampling for all methods which is a better strategy than using fixed points), I am satisfied with the responses to my primary concerns about the paper. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-GX_nxwfb4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses several protocols including data augmentation, point distribution, loss function, ensemble scheme, and testing models, which serves as a kindly reminder that the training protocol matters. As claimed, earlier work like PointNet++ can still achieve comparable performance to more recent methods. Such observations are useful, as different methods are supposed to be developed and measured under a unified setting. Moreover, the authors investigated a new projection-based SimpleView method by converting point clouds into depth images, achieving SOTA performance without pretrained CNNs.\n\nAlthough such detailed reviews over the training protocols are helpful, the authors failed to propose any novel methods or evaluation metrics (notice the SimpleView is also a commonly used methods in other multi-view framework), resulting in relatively inadequate novelty and originality. In practice, when we evaluate or re-implement these deep point cloud networks, we naturally attempt to add and remove some basic tricks. I believe there are a lot of people already knowing what has been explored in this paper. Nevertheless, this work still has a great practical value that helps us quantitatively understand how different protocols influence several commonly-used models. Moreover, different methods are compared only for point cloud classification task on ModelNet40, which only rely on global feature and cannot fully demonstrate the representation ability among different methods. For SimpleView, providing the image resolution is preferred.  Additionally, some typos, e.g. in \"randon\" in Table 1, should also be corrected. \n\nPros:\nThis paper investigates protocols for various point cloud networks for point cloud classification, with an observation that simple PointNet++ and SimpleView can already achieve competitive performance\n\nCons:\nThis paper lacks academic insights and novelty, making it more like a technical report. Besides, it only focuses on the classification task, which cannot convincingly conclude the representation ability of different methods. Some typos and missing of experiment settings\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received three recommendations of accept and one recommendation of reject.   The paper is mixed.  The results presented are both compelling and will have impact on the community.  The AC does not agree with R2's views that the paper requires proposal of a novel method for acceptance.  At the same time, the AC also does not agree with the views of the other reviewers that the current experiments alone are enough to carry the paper without more conclusive statements.  As hinted by R3, simply pointing out the problems is not enough without proposing how to adjust our models and experimentation protocols in the future is insufficient.  

In its current state, the paper would make for a good workshop submission.  Alternatively, the AC suggests to the authors to expand on the SimpleView baseline and or propose alternative solutions or protocols.
Decision: Reject