ASSANet: An Anisotropic Separable Set Abstraction for Efficient Point Cloud Representation Learning
Authors: Guocheng Qian
Hasan Abed Al Kader Hammoud
Guohao Li
Ali Thabet
Bernard Ghanem
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: point cloud
3D
neural network
efficiency

Abstract: Access to 3D point cloud representations has been widely facilitated by LiDAR sensors embedded in various mobile devices. This has led to an emerging need for fast and accurate point cloud processing techniques. In this paper, we revisit and dive deeper into PointNet++, one of the most influential yet under-explored networks, and develop faster and more accurate variants of the model. We first present a novel Separable Set Abstraction (SA) module that disentangles the vanilla SA module used in PointNet++ into two separate learning stages: (1) learning channel correlation and (2) learning spatial correlation. The Separable SA module is significantly faster than the vanilla version, yet it achieves comparable performance.  We then introduce a new Anisotropic Reduction function into our Separable SA module and propose an Anisotropic Separable SA (ASSA) module that substantially increases the network's accuracy. We later replace the vanilla SA modules in PointNet++ with the proposed ASSA modules, and denote the modified network as ASSANet. Extensive experiments on point cloud classification, semantic segmentation, and part segmentation show that ASSANet outperforms PointNet++ and other methods, achieving much higher accuracy and faster speeds. In particular, ASSANet outperforms PointNet++ by $7.4$ mIoU on S3DIS Area 5, while maintaining $1.6 \times $ faster inference speed on a single NVIDIA 2080Ti GPU. Our scaled ASSANet variant achieves $66.8$ mIoU and outperforms KPConv, while being more than $54 \times$ faster.

ASSANet: An Anisotropic Separable Set Abstraction for Efficient Point Cloud Representation Learning
Guocheng Qian
Hasan Abed Al Kader Hammoud
Guohao Li
Ali Thabet
Bernard Ghanem
King Abdullah University of Science and Technology (KAUST)
{guocheng qian, hasanabedal hammoud, bernard, ghanem) Skaust. edu, sa bttts:///githu com/ gwwchenggian/ASSAARne
Abstract
Access to 3D point cloud representations has been widely facilitated by LiDAR sensors embedded in various mobile devices. This has led to an emerging need for fast and accurate point cloud processing techniques. In this paper, we revisit and dive deeper into PointNet++,, one of the most influential yet under-explored networks, and develop faster and more accurate variants of the model. We first present novel Separable Set Abstraction (SA) module that disentangles the vanilla SA module used in PointNet++ into two separate learning stages: (1) learning channel correlation and (2) learning spatial correlation. The Separable SA module is significantly faster than the vanilla version, yet it achieves comparable performance. We then introduce a new Anisotropic Reduction function into our Separable SA module and propose an Anisotropic Separable SA (ASSA) module that substantially increases the network's accuracy. We later replace the vanilla SA modules in PointNet++ with the proposed ASSA module, and denote the modified network as ASSANet. Extensive experiments on point cloud classification, semantic segmentation, and part segmentation show that ASSANet outperforms PointNet++ and other methods, achieving much higher accuracy and faster speeds. In particular, ASSANet outperforms PointNet++ by 7.4 mloU S3DIS Area 5, while maintaining 1.6 1.6x faster inference speed on single NVIDIA 2080Ti GPU. Our scaled ASS variant achieves 66.8 mloU and outperforms KPConv, while being more than 54 x faster.
Introduction
Among the various 3D object representations, point clouds have been surging in popularity, becoming one of the most fundamental 3D representations. This popularity stems from the increased availability of 3D sensors, like LiDAR, which produce point clouds as their raw output. The growing presence of point cloud data has been accompanied by the development of many 3D deep learning methods [28, 41, 19,38, 38, 22]. Even though these methods achieve impressive performance, they are generally computationally expensive (Figure 1). With the integration of LiDAR sensors into hardware-limited devices, such as mobile devices and AR headsets, interest in efficient models for point cloud processing has grown significantll. Given the limited computational power of mobile devices and embedded systems, the design of mobile-frient friendly point cloud- algorithms should not only focus on providing good accuracy, but also on maintaining high computational efficiency.
When processing point cloud data, one can always opt to convert the data into representations that are more accessible deep learning frameworks. Popular options multi- nulti-view methods [34, 5, 42] and voxel-based methods [6, 47]. Converting to these representations generally requires additional
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Accuracy-Spead Tradeoff Curve ASSAHet
Figure 1: Tradeoffs between accuracy (mloU on S3DIS Area-5) and inference speed (in- stances/second), Speed is reported as the mean value of 200 runs on a single GTX 2080Ti GPU. The proposed ASSANet scaled with different widths and depths shown in ... outperform the offteoo-the--ee art methods in with better accuracies and faster speeds. Refer to Section 5.2 for details.
LcCCory
5a0 Area 5 mtou (1)
106.8 Speed
PosPool(S)
Speed Up
PVCHH
22.3 Speed Up
Peinthete
x1.5 Speed Up
DanpOCN
22500 Spord Up
x1.2 Speed Up PointMot 200
Speed (istances/sscoond
computation and memory, and can lead to geometric information loss [23]. It is therefore more desirable operate directly on point clouds. To that extent, we are currently witnessing surge in point-based methods [27, 28, 41, 19, 38, 22]. The first of such methods was introduced by Qi el al. through the seminal PointNet [27] architecture. PointNet operates directly on point clouds, without the need for an intermediate representation. Despite its efficiency, PointNet merely learns per-point features individually and discards local information, which restrains its performance. As a variant of PointNet, PointNet++ [28] presents a novel Ser Abstraction module that sub-samples the point cloud, groups the neighborhood. extracts local information via a set of multi-layer perceptrons (MLPs), and then aggregates the local information by reduction layer (i.e. pooling). Figure 1 shows how PointNet++ outperforms the pioneering PointNet [27] by a large margin. PointNet++ also obtains better accuracy than the wraph-based method DeepGCN [19], and does so with 100x speed gain. PointNet++ provided a good balance between accuracy and efficiency, and was therefore widely utilized in various tasks like normal estimation [8], segmentation [26, 14], and object detection [32]. After PointNet++, graph-based [33, 39, 41, 19], speudo-grid based [37, 20, 25, 38] and adaptive weight-based [40, 21, 7, 44]. became the state-of- offte-of-thh-at in point cloud tasks. As shown in Figure 1, nearly all of these methods improve performance at the cost of speed. In this work, we focus on designing point cloud networks that are both fast and accurate. Inspired by its success, both in terms of the accuracy-spect balance and its wide adoption, we take a deep dive into PointNet++. We conduct extensive analysis of its architectural design (Section 3.1) and latency decomposition (Figure 2). Interestingly, we demonstrate that both its efficiency and accuracy can be improved sharply by minimal modifications to the architecture. These modifications lead to new architecture design that is faster and more accurate than currently available point methods (shown in -.- in Figure 1).
Contributions. (1) We demonstrate that the MLPs performed on the neighborhood features in the Set Abstraction (SA) module of PointNet++ reduce the inference speed. We introduce new separable SA module that processes on point features directly allowing for significant improvement in inference speed. (2) We discover that all operations for processing neighbors in the SA module are isotropic which limits the performance (accuracy wise). We present a novel Anisotropic Reduction layer that treats each neighbor differently. We then insert Anisotropic Reduction into our Separable SA and propose the Anisotropic Separable Set Abstraction (ASSA) module that greatly increases accuracy. (3) We present ASSANet by replace the vanilla SA in PointNet++ with the proposed ASSA. ASSANet shows much higher accuracy and a faster speed compared to PointNet++ and previous methods on various tasks (point cloud classification, semantic segmentation, and part segmentation). We further study two regimes for up-scaling ASSANet. As shown in Figure our scaled ASSANet outperforms the previous state-of-the-arl with a much faster inference speed. In particular, scaled ASSANet achieves better accuracy than the graph-based method DeepGCN [19] with an increase in speed of 294x, the pseudo grid-based method KPConv [38] (54 x faster), the adaptive weight-based method PosPool*(S) [22] (9x faster), and the efficient 3D method PVCNN [23] (2x faster).
2 Related Work
Projection-based methods. Due to the unstructured nature of point clouds, convolutional neural networks (CNNs) that tend work impressively well on grid stuctured data (e.g. images, texts and videos) fail to apply directly on point clouds. One common solution for processing point clouds is to project them into collections of images (views) [34, 42] or 3D voxels [6, 47, 36). Common CNN backbones (using 2D or 3D convolutions) can be subsequently utilized to perform these intermediate representations. Although projection-basee bbasdd methods allow for utilizing the well studied convolutional neural networks to point-cloud applications, they computationally expensive they are associated

---

Subsampling
6.2%
12.0%
Subsampling
Computation 92.3%
Computation 90.6%
Computation
Computation 71.1%
Grouping 2.3%
Grouping
Groupind
Grouping 4.3%
3.1%
5.1%
(a) 1024 points
(b) 4096 points
(c) 10,000 points
(d) 15,000 points
Figure 2: Latency Decomposition of PointNet++. We show the inference run time decomposition
of PointNet++ under different numbers of points as input on one NVIDIA GTX20S0Ti GPU.
with additional cost of constructing intermediate representations. Moreover, the projection of point clouds causes loss of important geometric information [23].
Point-based methods. Pioneering work explored the possibility of processing point clouds directly.
Qi et al. proposed PointNet [27] that leverages point-wise MLPs to extract per point features individually. To better encode locality, Qi et al. further presented Set Abstraction (SA) to aggregate features from the points' neighborhood, and hierarchical architecture named PointNet+ [28] that learns multilevel representations and reduces the required computations. After PointNet+++ numerous point-based methods considering neighborhood information were proposed. Graph-based methods [33, 15, 41, 39, 19, 18, 29] represent point clouds as graphs and process point clouds with graph neural networks. Pseudo grid-based methods project neighborhood features onto different forms of pseudo grids such as tangent planes [37]. grid cells [11, 46, 20, 25, 38] and spherical grid points [49] which allow convolving with regular kernel weights like CNNs. Adaptive weight- baseed methods perform weighted neighborhood aggregation by considering the relative positions of the points [40, 21, 7, 22] point density [44]. These methods rely either on designing sophisticated and customized modules, which usually require expensive parameter tuning for different applications [38, 20, 24], or on performing expensive graph kernels [41, 19] that achieve better performance than PointNet and PointNet++ at the expense of computational complexity.
Efficient Neural Networks. Efficient neural networks is class of architectures that target mobile and embedded systems applications. These networks are usually designed to provide a balance between accuracy and efficiency (e.g. latency, FLOPs, memory, and power). MobileNet [9] utilizes depth-wise separable convolutions to reduce the required FLOPs and latency of a regular CNN for image processing. Depth- wise separable convolutions disentangle convolutions into learning channel correlations using point-wise convolutions and learning spatial correlations using depth- wise convolutions. Other efficient neural networks usually leverage either depth-wise separable convolutions with better designed architectures to improve performance [31, 4, 48] or study new efficient operations to replace the regular convolutions [24, 43). In 3D, efficient neural networks include ShellNet [49], PVCNN [23]. Grid-GCN [45], RandLA-Net [10]. SegGCN [17] and LPNs [16]. ShellNet [49] and SegGCN [17] speed up the pseudo grid-based methods by aggregating neighborhood features through efficient ID convolutions or fuzzy spherical convolutions on the predefined pseudo grids like shells. PVCNN [23] and Grid-GCN [45] reduce the time spent in querying a neighborhood by combining voxelization in point-based methods. RandLA-Ne [10] reduces the subsampling complexity by leveraging random sampling and further improves the speed by operating on large-scale point cloud directly without chunking. LPN [16] improves the speed of convolving neighborhood features by simple group-wise matrix multiplication. Nevertheless, all efficient methods mentioned above require performing convolutions on neighborhood features, which we deem through extensive experiments as unnecessary. Therefore, our algorithm achieves much faster speeds compared to these methods (ref to Section 4). It is also worthwhile to mention that our method can be made even faster with the voxelization trick in PVCNN and Grid-GCN to further reduce the latency of neighborhood querying. We leave that as future work.
Methodology
3.1 Preliminary: PointNet++
PointNet++ [28] improves PointNet [27] by providing two main contributions: (1) developing U-Net [30] like architecture to process set of points, which are sampled in a metrie space in hicrarchical fashion. This mechanism captures multi-scale features and reduces the required
3

---

computation. (2) Developing a Set Abstraction (SA) module to process and abstract the locality from the local neighbors to a new set of points with fewer elements. The SA module is used as the basic building block to be stacked to form the backbone of PointNet+++.
Analysis of the Set Abstraction Module. As illustrated in Figure 3a, the vanilla SA module proposed in PointNet++ consists of two parts: point subsampling and feature aggregation. The subsampling layer takes point cloud X = [P. F) as an input and leverages iterative farthest-point sampling to acquire X' subset of X. and F denote the coordinates and features, respectively. The feature aggregation block is built for learning locality from local neighbors and is composed of a grouping layer, an MLP block, and a reduction layer. The grouping layer obtains the neighborhood composed of K neighbors for each point in X' using the ball query, with X the support set. The resulting point neighborhood is denoted as N ((X'., ), which contains K repeated features. The MLP block consists of L layers of MLPs, and each MLP is followed by a Batch Normalization (BN [12]) layer and a ReLU activation. By default, PointNet++ sets L 3. The number of feature aggregation blocks inside one SA module, referred to as depth in this paper, is set to D 2. The reduction layer (a. pooling) aggregates the neighborhood information by reduction function, e.g. mean, max, or sum. The feature aggregation is formulated as shown Equation (1):
f/+1 R ((MLPs((P) - p:)i e N(()))
where R the reduction function across the neighborhood dimension, which used for aggregating the neighborhood information. p.. (i)) and || denote the coordinates, the features in the 1th layer of the network, the neighborhood of the 8th point, and the concatenation operator across the channel dimension, respectively. The main issues with the vanilla SA module are: (1) the computational cost is unnecessarily high. MLPs are unnecessarily performed on the neighborhood features, which causes a considerable amount of latency in PointNet+++. One straightforward remedy to use MLPs to learn a feature embedding on the point features directly instead of doing so on the neighborhood features. This reduces the FLOPs of each MLP by a factor of K. (2) All operations on neighbors are unnecessarily isotropic. In other words, the MLPs and the reduction layer treat all local neighbors equally. This severely limits the representation capability of the network.
Latency Decomposition. Figure 2 shows the latency decomposition of PointNet++ [28] with different numbers of points as input. Here, the latency, which is the overall run time for the inference stage, was measured using a single Nvidia GeForce RTX 2080Ti GPU and one Intel(R) Xcon(R) CPU E5-2687W v4 @ 3.00GHz. We note here that latency is measured on the same hardware setting throughout this work. The latency of PointNet++ can be decomposed into three main contributing factors: (1) point subsampling, (2) grouping, (3) actual computations. The actual computations of PointNet++ mainly come from processing neighborhood features by MLPs shown in Equation
Note that we consider the time spent on data access implicitly in each part. Point clouds with four different input sizes were studied: 1024, 4096, 10, 000, and 15, 000. The first two input sizes are commonly encountered in classification tasks [2], and the last two are usually input sizes for patch-based segmentation [28, 38] and LiDAR-based object detection [32]. Clearly, computations contribute to the majority of latency (over 70 %). This suggests that the computational complexity could be the major speed bottleneck for networks involving PointNet++.
3.2 Anisotropic Separable Set Abstraction (ASSA)
In this section, we gradually introduce the modified vanilla SA modules. Initially, we focus on speeding up vanilla SA. This is achieved through proposing two modules, namely, PreCony SA module and Separable SA module. Later, focus our attention on improving the accuracy by proposing an Anisotropic SA module.
PreConv Set Abstraction module. Vanilla SA repeatedly performs shared ML.s on point neighbor- hood features. To solve this issue, we modify the feature aggregation layer in vanilla SA, and propose PreConv SA to performs all MLPs on point features directly (not on the k local neighbors) before the grouping layer. The PreConv SA is shown in Appendix Figure SI, and its feature aggregation is formulated as follows:
= MLLSs (F)) - = R ((E) j jEN())
PreConv SA reduces the required FLOPS by K times. PreConv SA speeds up PointNet++ by 55% (15,000 points), as shown in Section 1. Additionally, PreConv SA is equivalent to vanilla SA in the case where the (p: p.) term is not included in Equation (1). Proof is available in the Appendix.

---

()
(N,0)
ML.s
(N.C)
MEN C
Subsample
(N,0)
Query
Grouping
N.C.K
Gromping
Anisotropic
Reduction
(.CC)
MIX
MLPs
(N.C
(c) Feature patterns before the first SA
ML.Ps
Redurttoo
Antsotropic
Aguregation
Grouping
Quary
Anisotropic
Support Grouping
Reduction
MLPs
MLPs
shortout X ccnnecction
(d) Feature patterns after the first SA
(a) Vanilla SA
(b) ASSA
Figure 3: Comparison of proposed Anisotropical Separable Set Abstraction (ASSA) module and the Vanilla SA module. (a) Vanilla SA [28] applies MLPs neighbor features. (b) The proposed ASSA module separates the MLPs before the grouping layer and after the reduction layer. Therefore the MLPs are applied directly on the point features not on the neighbor features. ASSA also replaces the reduction layer in vanilla SA with new Anisotropic Reduction layer. X, N. C, K are the input point cloud, the number of points, the number of input features, and the number of neighbors. The shortcut layer in blue line is the residual connection with linear mapping. (c) and (d) Show the point cloud feature patterns (activations) before and after the first ASSA module. The proposed ASSA module helps capture better geometric relationships. Refer to Section 5.1 for details.
Separable Set Abstraction module. Next, present Separable Set Abstraction (refer to Appendix Figure S1), which is more accurate than PreConv SA, yet requires the same latency. The aggregation layer in Separable Set Abstraction is formulated by:
free MLPs
f MLPs (R ((F |j (EE(())))
(3)
The main idea of Separable SA is borrowed from depth-wise separable convolutions [9], where the regular convolution is split into one point-wise convolution (MIPs), one depth-wise convolution (channel shared convolution), and then another point-wise convolution. Separable SA evenly sepa- rates the MLPs before the grouping layer and after the reduction layer and further adds residual connection between the outputs of the two parts of the MLPs. The main reasons why the Separable Set Abstraction module is better than PreConv are: (1) after reduction MLPs further process the aggregated neighborhood information; (2) The residual connection not only stabilizes training, but also provides better feature embedding by fusing the aggregated local information with the point information. Another minor change from PreConv SA to Separable SA is that we query the neigh- borhood using the subsampled point cloud X' the support set further reduce the computational complexity of the second aggregation block.
Anisotropic Separable Set Abstraction module. PreConv SA and Separable SA down compu- tational complexity at the expense of accuracy, e.g. Separable SA leads to a reduction of 3 mloU on S3DIS Area 5 compared PointNet+ (Section 5.1). There First, the geometric information is not well encoded in the current variants of the SA module. The geometric information can be represented by any relative information (edge information) between the neighbor and the center, in Section 5.1 show that geometric information is essential for point feature embedding. Second, the reduction layer is an isotropic operation that treats each neighbor the same and thus leads to sub-optimal representation. Recall in depth- -wie separable convolution, the depth-wise convolu- tion uses different weights to summarize features from a 3 x 3 receptive field. However, simply introducing the depth-wise convolution kernel to point neighborhood aggregation does not work, as: (1) the neighbors are not necessarily ordered for the sake of efficiency: (2) the convolution kernel shared by all points and neighbors and leads to poor neighborhood aggregation where the local geometric varies. We propose efficient geometric-aweee Anisotropic Reduction layer to effectively
reasons for the drop in accuracy.
the relative position (p, p:) in PointNet++. The experiments
5

---

aggregate the point neighborhood information. The term "Anisotropic" indicates that our reduction layer considers each neighbor differently. We insert Anisotropic Reduction into the separable SA module and present our final variant of the SA module, the Anisotropic Separable Set Abstraction (ASSA) module, and show it in Figure 3b. The feature aggregation of ASSA is formulated as:
free = (L))((F)
p+11 LN (fTT*) + MLPs R (((Arrg free
Anr ffee lا ILEN( ( free
Aspfje
(4)
r
Azy = - Ayyy and Azy are the relative positions between the neighbor j and the center in the I, y. 2 dimension, respectively. The relative positions are used as scaling weights for aggregating the features across the neighborhood dimension, and they are normalized by the radius of the ball query r. The neighborhood features are scaled by the three corresponding relative positions individually. The three scaled neighhorhood features are then concatenated together and passed into the reduction layer. To reduce the computational complexity caused by the concatenation of the three scaled features, we set the last MLP before reduction as a bottleneck layer. This layer reduces the number of channels by factor of 3. The output of the reduction layer is then processed by another MLP block and is added to the output before reduction. Due to channel mismatch, the output of before grouping MLPs is mapped by linear layer LN (a.k.a the shortcut layer) before the addition. We highlight that our Anisotropic Reduction does not rely on any heuristic grouping (as done in PosPool [22]). and we make full use of the information from the neighborhood features. The pseudo code for ASSA in PyTorch-like style is available in the Appendix.
It is worth noting that all MLPs in ASSA are processed on the point features directly, not on the neighborhood, which greatly reduces the computations compared to Equation (1). particular, for one aggregation block with L 3 MLPs, ASSA roughly reduces the FLOPs consumed in vanilla R K times. Typically, K is around 32. All of our SA variants are permutation invariant, which favors 3D deep learning on point clouds. More details of the ASSA module and its comparison with previous modules are provided in the Appendix.
SA by: EXCXXXL+UXNE
XxxxNxXXX
3.3 ASSANet
We now replace the vanilla SA module in PointNet++ [28] with our proposed ASSA module, All other parts are kept the same as PointNet++. including the number of SA modules (4), the number of
aggregation blocks in SA (D = 2), the layers of MLPs in an aggregation block (L = 3), the channel sizes, the neighborhood querying configurations (ball query algorithm with maximum neighborhood size K and radius r) and the subsampling configurations (farthest point sampling). The modified architecture of PointNet++ is referred as ASSANet. Section 4 shows that ASSANet achieve much higher accuracies compared to PointNet and PointNet++ and is faster on various vision tasks.
3.4 Scaling ASSANet
Since the ASSANet is much faster than both PointNet++ [28] and the state-the-of-an art networks, now present two ways to up-scale ASSANet to improve its accuracy: width scaling and depth scaling. We show the performance of each scaling regime in the ablation study presented in Section 5.2.
Width Scaling Regime. In width scaling regime, we modified the channel size of ASSANet. ASSANet is built upon PointNet++ [28], which uses hand-crafted channel sizes for each convolution layer. To make the scaling more programmable and user-friendly for the scaled ASSANet, the output of each feature aggregation block inside one ASSA module is set to have the same channel size, and is then concatenated as the output of the module. After this modification, we can easily study the effect of width scaling the accuracy and the speed, by simply changing the initial channel size C.
Depth Scaling Regime. The second way to scale is to increase the depth of the network, which can be achieved by changing the number of aggregation blocks D stacked in each ASSA module. D set to by default in ASSANet. We can decrease D to to make ASSANet faster or increase D to improve its accuracy. Among all width or depth scaled versions of ASSANet, we emphasize ASSANet (I.), a large ASSANet network with C = 128 and /) 3. In most of the experiments, we compare ASSANet and ASSANet (L) with the state-of-the-art

---

Methods
mIOU
Inference Speed
4 Experiments
%
instances/second
PointNet [27]
41.1
185.0
We studied the accuracy and speed of
DeepGCN [19]
52.5
0.8
ASSANet and ASSANet (L) on S3DIS
PointCNN [20]
57.3
124.1
semantic segmentation [1], ShapeNet
Grid-GCN [45]
57.8
123.5
part segmentation [3], and ModelNet40 point cloud classification [2]. To enable
PVCNN [23]
59.0
89.8
PosPool (S) [22]
61.3
21.0
a fair comparison, the same data process-
SegGCN [17]
63.6
29.3
ing and evaluation protocols adopted by the state-of-the aof-theeart art method PosPool [22]
KPConw [38]
65.4
1.2 (24.2)
PosPool [22]
66.7
8.3
were used in our experiments.
PointNet++ [28]
55.6
116.6
ASSANet
63.0
(+7.4)
188.6 (1.6x)
4.1 3D Scene Segmentation
ASSANet (L.
66.8 (+11.2)
65.6
Setups. We conducted extensive exper- iments on the Stanford large-scale 3D
Table 1: S3DIS scores (mloU) on Area-5. ASSANet outperforms PointNet++ and other methods with much higher accuracy and faster speed. ASSANet (L) performs better than the late-of-the-art KPConv [38] and PosPool" [22] while being over 7.9x faster.
Indoor Spaces (S3DIS) dataset [1]. Fol- lowing [20, 23, 22], we trained all our models on Area 1, 2, 3, 4, and 6 and
tested them Area 5. We optimized all of our networks using SGD with weight decay 0.001, momentum 0.98 and initial learning rate (LR) 0.02. We trained the models for 600 epochs and used exponential LR decay. At each inference time, a single RTX 2080Ti GPU was used to measure the speed for each method using a batch size of 16; each item in the batch has 15, 000 points (16 x 15, 000). If the batch size was too large feed into the GPU, we lowered the batch size. Note that we focus on the speed since FLOPs and the model parameter size are not indicative of the actual latency [24, 23). The inference speed is calculated as the number of instances evaluated in one second (ins./sec.). The average speed over 200 runs is reported. Other methods were measured in similar manner. Note that KPConv [38] has compute the pseudo kernels for each point cloud during data preprocessing, For fair comparison, we show the speed of calculating the pseudo kernels on the fly. We also include the speed of KPConv with preprocessed pseudo kernels in () in the table.
Comparison with state-of-the-art. Table compares the proposed ASSANet and ASSANet (L) with PointNet++ [28] and the state-of-the-ee on S3DIS. ASSANet outperforms PointNer+ + by 7 mloU and is 1.6x faster. ASSANet also achieves much better accuracy than the two efficient point cloud processing algorithms PVCNN [23] and Grid-GCN [45], while also being over 1.5x faster. ASSANet (L.) achieves state-of-the-arl performance with a mloU of 66.8% on S3DIS, with very high speed. ASSANet (L.) is 294x faster than the graph-based method DeepGCN [19], 54.6x faster than the state-of-the-ar pesudo grid-based method KPConv [38], 7.9x faster than the state-of-the-art adaptive weight- based method PosPool* [22], and 2.2x faster than the best best-performing efficient method ScgGCN [17]. Note that PosPool* refers to PosPool with sinusoidal position weight, and that PosPool* (S) denotes the small model.
4.2 3D Object Classification
Setup. As a common practice, we benchmark ASSANet on the ModelNet40 [2] object classification dataset. We adopted a similar training setting as that of S3DIS except that we used LR 0.001 and a cosine LR decay. At the inference time, a single RTX 2080Ti GPU was used to measure the speed for the classification task using 16 x 10, 000 points as input.
Comparison with state-of-the-art. Table 2 compares ASSANet and ASSANet (L) with the state-of- the-art. ASSANet outperforms PointNet++ by 1.7 units in overall accuracy and 2.1x faster than PointNet++. ASSANet (L) achieves on par accuracy as the state-of of-the--n methods KPConv [38] and PosPool [22] while being 5.0x and 4.4x faster, respectively.
4.3 3D Part Segmentation
Data. ShapeNetPart a commonly used benchmark for 3D part segmentation. The networks were optimized using Adam [13] with momentum 0.9. The other training parameters were the same as
7

---

Methods
OA
Inference Speed
Methods
mloU
Inference Speed
%
instancea/second
instances/sccond
PointNct [27]
89.2
483.8
PointNet [27]
83.7
1883.5
SpiderCNN [46]
90.5
< 2755.7
PosPool* (S) [22]
85.1
107.7
PoinCNNN [20]
92.5
183.4
DGCNN [41]
85.2
151.4
PosPool" (S) [22]
92.6
48.8
LPN [16]
85.7
190.6
DGCNN [41]
92.9
PosPool* [22]
KPConv [38]
92.9
(30.1)
PoinCNNN [20]
86.1
626.4
Grid-GCN [45]
93.1
172.0
RS-CNN [21]
86.2
<350.4
PosPool" [22]
93.2
27.6
KPConv [38]
86.2
(56.3)
PoimtNet++ [28]
90.7
275.7
PointNet++ [28]
85.1
350.4
ASSANet
(+1.7)
586.4 (2.1
ASSANet
85.4 (+0.3)
782.5 (2.2x)
ASSANet (L)
92.9 (+2.2)
153.2
ASSANet (L)
86.1 (+1.0)
438.5 (1.3x)
Table 2: Comparison of our ASSANet and ASSANet (I.) with other methods Mod-
Table 3: Comparison of the part-averaged IoU (mIoU) of our ASSANet and AS- SANet (L) with other methods on ShapeNetPart part segmentation. Both of ASSANet and ASSANet (L) outperform PointNet++ with a higher speed. ASSANet (L) achieves comparable accuracy as the state-of the while being much faster.
elNet40 point cloud classification. AS- SANet outperforms PointNet++ with 1.7 higher overall accuracy (OA) than PointNet++ and 2.1 times faster. ASSANet (L) achieves on par accuracy with the state-of-ther thtee-of-hhearr art while maintaining high speed.
ModelNet40 experiments. The speed of each method was measured with an input of 16 x 2048 points. We report the part-averaged IoU (mloU) as the evaluation metric for accuracy.
Comparison with state-of-the-an Table shows that ASSANet again outperforms PointNet++ with sharp increase (2.2x) in speed on the ShapeNetPart part segmentation dataset. ASSANet (L) also achieves unit higher mloU than PoinetNet++ with a 1.3x faster speed. Additionally, ASSANet (L.) attains on-par accuracy, 86.1 1% mloU, with the state-of-the-art and is much faster. For example, ASSANet (L.) is nearly 7.8x faster than KPConv [38].
5 Ablation Study
Aggregation
mIoU %
Speed ins./sec.
An ablation study was conducted on S3DIS [1] Area-5. We show the effectiveness of the proposed SA variants and the effect of the two scaling regimes ASSANet.
Vanilla SA
55.6
116.6
PreConv SA
48.7
180.9
Separable SA (SSA)
52.4
180.0
SSA + Relative Position
58.5
184.0
SSA Attentive Pooling[10]
59.0
142.0
5.1 Ablation on Proposed SA variants
SSA + PosPool[22]
62.0
168.4
Table 4 shows the speed and the accuracy of the proposed PreConv Set Abstraction (SA) module, the Separable SA module, and the Anisotropic Separable SA (ASSA) module compared to the vanilla SA. All of our pro- posed SA modules lead sharp increase (over 1.6x) in inference speed. The pro- posed Separable SA module can boost the accuracy of PreConv by 3.7 mloU, which verifies the effectiveness of separable MI.Ps and residual connections. Comparing the ASSA module with the Separable SA module, one can clearly see the importance of encoding the geometric information and the effect of the anisotropic operation achieving higher accuracy. Additionally, we provide a comparison of our proposed Anisotropic Reduction with the Attentive Pooling used in RandLA-Ner [10] and the PosPool proposed in [22]. Our method clearly outperforms both of these methods in terms of accuracy and inference speed. We also test simple addition of the relative positions A₂ + A + A as the weights of the reduction layer, denoted as Relative Position, the obtained performance is worse than the proposed Anisotropic Reduction. To further show the benefits of the proposed ASSA module, we visualize the feature patterns before and after the ASSA module in Figure ASSA module helps capture better geomet- ric relationships among points constituting the point cloud (for example, in the second and fourth
Anisotropic Separable SA
63.0
188.6
Table 4: Ablation study the proposed SA vari- ants. All proposed SA variants achieve faster speed than the vanilla SA. Our ASSA further improves the accuracy of the Separable SA module and outperforms other methods, while also being faster.

---

examples in the first row of Figure 3d, one can see that the ASSA module allows the network to learn relationships between the tail of the plane and its wings). Appendix provides a more detailed overview and further set of examples of feature patterns visualization.
5.2 Ablation on Scaling Regimes
We now study the effects of ablating the width and depth of network on its accuracy and inference speed. The initial channel size of the network is referred to as width (denoted by C), whereas the number of aggregation layers inside a single SA module referred to as depth (denoted by D).
Width scaling. Figure 4 (left) shows the effect of the width scaling regime. When the width of the network is small, increasing the width leads to significant improvement in accuracy. For example, simply increasing the width C' from 3 to sharply improves the accuracy from 41.21 mloU to 53.95 with negligible drop in speed. However, when the network is wide enough (C > 128), increasing the width further only leads to a marginal improvement in accuracy, yet reduces the speed noticeably.
Depth scaling. Figure 4 (right) shows the effect of the depth scaling regime. We study the depth scaling with C 128, which is the sweet point of width scaling. When the network shallow, with a depth of D S 3, increasing the depth leads to an obvious increase in accuracy. However, depth scaling rapidly saturates as the depth increases. Depth scaling leads to a linear reduction in speed.
Effect of Width Scaling
Effect of Depth Scaling
230.4 224.8
64.11
65.77
66.06
225
66.80
160
miou (%)
185,9
200
S3D15 Area 5 mlou 09
66
65.42
140
59.21
114.9
156.7
100
53.95
63
800
100
S3DI5
69.0
75
Speed
60
40
41.21
60.28
26.3
20
16
12
126
256
512
Width
Depth
Figure 4: Effect of Width (left) and Depth Scaling (right). Increasing either the width or the depth leads to an improvement in accuracy and drop in inference speed.
6 Conclusion
In this paper, dove deeper into the architecture of PointNet++. We noticed that PointNet++ suffers from a computational burden attributed to the MLPs that process the neighborhood features in the set abstraction (SA) module. We also found out that the accuracy of PointNet++ is limited by the isotropic nature of its operations. To solve these issues, we proposed a PreConv SA module, a Separable SA module, and finally an Anisotropic Separable SA (ASSA) module that aim reduce the computational cost and improve the accuracy. We then replaced the vanilla SA module in PointNet++ with our ASSA module and proposed a fast and accurate architecture, namely, ASSANet. Extensive experiments were conducted to verify the presented claims and showed that ASSANet achieves largely improved accuracy and much faster speed on various point cloud tasks, such as classification, semantic segmentation, and part segmentation. We also studied up-scaling ASSANet. The scaled ASSANet set new state-of-the oftthe-arttttttee on various tasks with faster speeds. For future work, one could leverage both random sampling [10] and voxelization tricks [23, 45] to further improve the inference speed. Alternatively, one could consider studying compound scaling, like that in EfficientNet [35].
The authors appreciate the anonymous NeurIPS reviewers for their constructive feedback (including the revised title, the feature pattern visualization, and the additional experiments). This work was supported by the KAUST Office of Sponsored Research (OSR) through the Visual Computing Center (VCC) funding.
Acknowledgement.

---

References
[1] Iro Armeni, O. Sener, A. Zamir, Helen Jiang, I. Brilakis, M. Fischer, and S. Savarese. 3d semantic parsing of large-scale indoor spaces. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1534-1543, 2016.
[2] Yizhak Ben Shabat, Michael Lindenbaum, and Anath Fischer. 3dmfv: classification in real-time using convolutional neural networks. IEEE Robotics and Automation Letters, 3(4)33145-1152, 2018.
point cloud
[3] Angel X. Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information Rich 3D Model Repository, Technical Report arXiv: Princeton University - Toyota Technological Institute at Chicago, 2015.
[es.GR], Stanford University
[4] François Chollet. Xception: Deep learning with depthwise separable convolutions. 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 1800-1807, 2017.
[5] Y. Feng, Zizhao Zhang, X. Zhao, R. and Yue Gao. Gvenn: Group-view convolutional neural networks for 3d shape recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 264-272, 2018.
[6] B. Graham, Martin Engelcke, and L. V. D. Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9224-9232, 2018.
[7] Fabian Groh, Patrick Wieschollek, and Hendrik P. A. Lensch. Flex-convolution (million scale point-cloud learning beyond grid- worlds). In Asian Conference on Computer Vision (ACCV), Dezember 2018.
[8] P. Guerrero, Yanir Kleiman, M. Ovsjanikov, and N. Mitra. Pepnet learning local shape properties from point clouds. Computer Graphies Forum, 37, 2018.
[9] A. Howard, Menglong Zhu, Bo Chen, D. Kalenichenko, W. Wang. Tobias Weyand, M. Andrectto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ArXiv, abs/1 704.04861, 2017.
[10] Q. Hu, B. Yang, Linhai Xie. S. Rosa, Yulan Guo, Zhihua Wang, A. Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation large-scale point clouds. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11105-11114, 2020.
[11] Binh-Son Hua, Minh-Khoi Tran, and Sii-Kit Yeung. Pointwise convolutional neural networks. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[12] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pages 448-456. Jorg 2015.
[13] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster). 2015.
[14] Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. 2018 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4558- 4567, 2018
[15] Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In Proceedings the IEFE Conference on Computer Vision and Pattern Recognition, pages 4558-4567, 2018.
[16] Eric-Tuan Le, I. Kokkinos, and N. Mitra. Going deeper with lcan point networks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9500 9009995099000 2020.
[17] Huan Lei, Naveed Akhtar, and A. Mian. Seggen: Efficient 3d point cloud segmentation with fuzzy spherical kernel. 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 11608-11617, 2020.
[18] G. L., M. Müller, Guocheng Qian, Itzel C. Delgadillo, Abdulellah Abualshour, Ali K. Thabet, and Bernard Ghanem. Deepgens: Making gens go deep cnns. IEFE transactions pattern analysis and machine intelligence, PP, 2021.
[19] Guohao Li. Matthias Müller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep as cnns? In The IEEE International Conference Computer Vision (ICCV), 2019.
10

---

[20] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, X. Di, and B. Chen. Pointenn: Convolution on x-transformed points. In NeurlPS, 2018.
[21] Yongeheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8887-8896, 2019.
[22] Z. Liu, H. Hu, Yue Cao, Zheng Zhang, and Xin Tong. A closer look at local aggregation operators in point cloud analysis. In ECCV, 2020.
[23] Zhijian Liu, Haotian Tang. Yujun Lin, and Song Han. Point-voxel enn for efficient 3d deep learning. In NeurIPS, 2019.
[24] Ningning Ma. X. Zhang, Tai-TTo Zheng, and Jian Sun, Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, 2018.
[25] Jiageng Mao, Xiaogang Wang, and Hongsheng Li. Interpolated convolutional networks for 3d point cloud understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1578 1587, 2019.
[26] Charles R. Qi, O. Litany, Kaiming He, and I.. Guibas. Deep hough voting for 3d object detection in point clouds. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9276-9285, 2019.
[27] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. 2017 IEEE Conference on Computer Vision and Pottern Recognition (CVPR), pages 77-85, 2017.
[28] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point
metric space. In NeurlPS, pages 5099-5108, 2017.
[29] Guocheng Qian, Abdulellah Abualshour, Guohao Li, Ali Thabet, and Bernard Ghanem. Pu-gen: Point cloud upsampling using graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages I 11683-11699 June 2021.
[30] Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI (3), volume 9351 of Lecture Notes in Computer Science, pages 234 241. Springer, 2015.
[31] Sandler, Andrew G. Howard, Menglong Zhu, A. Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. 2018 IEEE/CVF Conference on Computer Vision and Patern Recognition, pages 1510-4520 2018
[32] Shaoshuai Shi, X. Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 770-779, 2019.
[33] Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In Proceedings of the IEEE conference on computer vision and puttern recognition (CVPR), pages 3693-370 3702, 2017.
[34] Hang Su, Subhransu Maji, E. Kalogerakis, and F. Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. 2015 IEEE International Conference on Computer Vision (ICCV). pages 945 953, 2015.
[35] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 6105-61 14. PMLR, 2019.
[36] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, J. Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse poxnt-voxel convolution. In ECCV, 2020.
[37] Maxim Tatarchenko, Jacsik Park, V. Koltun, and Qian-Y Yi Zhou. Tangent convolutions for dense prediction in 3d. 2018 IEEF/CVF Conference on Computer Vision and Pattenn Recognition (CVPR), pages 3887-3896,
2018.
[38] Hugues Thomas, Charles R. Qi. Moan-Emmnuuel Deschaud, Beatriz Marcotegui. François Goulette, and Leonidas J. Guibas. Kpconv: Flexible and deformable convolution for point clouds. 2079 IEEE/CVF International Conference Computer Vision (ICCV), pages
2019.
11

---

[39] Lei Wang, Yuchun Huang, Yaolin Hou, Shemman Zhang, and Jie Shan. Graph attention convolution for point cloud semantic segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10296-10305, 2019.
[40] Shenlong Wang, Simon Suo. Wei-Chiu Ma. Andrei Pokrovsky, and Raquel Urtasun. Deep parametric continuous convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2589-2599 2018.
[41] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael M. Bronstein, Justin M. Solomon. Dynamic graph can for learning on point clouds. ACM Transactions on Graphics (TOG), 2019.
[42] Xin Wei, Ruixuan Yu, and Sun. View-gen: View-based graph convolutional network 3d shape analysis. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 1847-1856, 2020.
[43] B. Wu, Alvin Wan, Xiangyu Yue, P. Jin, S. Zhao, Noah Golmant, A. Gholaminejad. Joseph E. Gonzalez, and K. Keutzer. Shift: A zero flop, zero parameter alternative to spatial convolutions. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9127-9135, 2018.
[44] Wenxuan Wu, Zhongang Qi, and F. Li. Pointconv: Deep convolutional networks on 3d point clouds. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9613 9622, 2019.
[45] Qiangeng Xu, Xudong Sun, Cho-Ying Wu, Panqu Wang, and U. Neumann, Grid-gen for fast and scalable point cloud learning. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages
2020.
[46] Yifan Xu, Tianqi Fan, Mingye Xu, L. Zeng, and Yu Qiao. Spiderenn: Deep learning on point sets with parameterized convolutional filters. In ECCV, 2018
[47] Yan Yan, Yuxing Mao, and B. Li. Second: Sparsely embedded convolutional detection. Sensors (Basel, Switzerland), 18, 2018.
[48] Chris Zhang, Wenjie Luo, and Raquel Urtasun. Efficient convolutions for real-time semantic segmentation of point clouds. In 2018 International Conference on Vision (3DV), pages 399 409.408.70 JEEE, 2018.
[49] Zhiyuan Zhang, Binh-Son Hua, and Sai-Kir Yeung. Shellnet: Efficient point cloud convolutional neural networks using concentric shells statisties. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1607-1616, 2019.
12


Reviews:
[{"review_id": "z4koR35RIw", "reviewer": "Reviewer_APWv", "summary": "This paper proposed a novel neural network backbone architecture for 3D point cloud processing that focuses on reducing the repeated computation inside the grouped neighborhood. The author proposed a nice technique to decompose the learning of the relative 3D coordinate values and the learning of features from previous layers so that the majority of repetitive computation can be saved. The experiments are done on several classification and segmentation benchmarks.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n\n- This paper addressed the problem of repetitive feature computation in PointNet++-style architecture. The problem is important yet has been long ignored. The proposed idea is simple but elegant and effective. \n\n- The experiment results are strong. When viewing performance and model runtime together, the proposed method significantly outperforms previous methods by a notable margin.\n\n\nWeak points: \n\n- The paper does not include any visualization that helps understand why would the proposed method work. The authors could provide a visualization figure of reduction $\\mathcal{R}$ in Equation (2) by showing what area is activated during the reduction operation in a local neighborhood, similar to Figure 19 of the PointNet paper; or by showing the critical point set that is activated, similar to Figure 18 of the PointNet paper.\n\n- It is good that the authors provide Equation (2) for ASSA module, which is the main contribution of the paper. However, the other two proposed PreConv SA and Separable SA are not explained by math equations which makes it really hard for readers to understand and compare with ASSA.\n\n- Figure 3 is very confusing. It does not reflect what ASSA is doing and what difference it is compared to SA and others. I suggest that the authors remove Figure 3 and simply use math equations to explain the four SAs.\n\n- Minor but important notation issue: in Equation (2), the author messed up with the set bracket and function bracket, i.e. the whole set indexed by $j$ should be inside the bracket of $\\mathcal{R} ()$ instead of outside of it.\n\n- Minor problem: there should be a line break at line 253?\n\n- Though the author pointed out that the theoretical speed-up of ASSA compared to vanilla SA is $K=32$, why does the actual speed-up is much smaller than this? The author can decompose the actual experiment latency numbers of each part (e.g. data loading, encoder, decoder, etc.) similar to Figure 2 to show the actual speed-up due to ASSA.\n\n- Though this work is on point cloud deep learning, conceptually it is not a direct extension of PointNet and PointNet++. Therefore, it is inappropriate to name this paper \"PointNetv3\". Instead, the author should emphasize \"anisotropic\" in the title and the network name. I'm willing to increase the review score to 8 as long as the authors are committed to using another name other than \"PointNetv3\".\n \n=================================\n\nAfter rebuttal:\n\nI'm satisfied with the response from the authors. It resolved most of the concerns in my review. \n\nI regard this paper as a very strong submission. It solves an important but long-ignored problem in point cloud deep learning. The solution is beautiful. I'm happy to increase my review score to 9.\n\nThere could be further improvement to the paper. Please see my official comment to the authors below.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No such issue was found. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "_oGWpPUa8yZ", "reviewer": "Reviewer_ABYL", "summary": "The authors modify PointNet++ by replacing the vanilla SA layers in PointNet++ by a new  Anisotropic Separable Set Abstraction layer. The obtained neural network, called PointNetv3, is faster and more accurate.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "PointNet++ which is a variant of PointNet is a notable neural network using for processing point clouds. PointNet++ contains two main components: Set Abstraction SA (to sub-samples the point clouds and groups the neighborhood) and a local feature extraction (which is PointNet). This paper aims to modify the SA layer to increase the efficiency and accuracy.\n\nThe authors propose a novel Anisotropic Separable Set Abstraction (ASSA) component to replace the SA. First, the MLPs in the SA layer are separated to before and after the reduction layer and are connected by a residual connection. Second, the reduction layer is replaced by an Anisotropic Reduction layer to effectively aggregate local information. This ASSA is then used to replace the SA layer in PointNet++ to obtain PointNetv3.\n\n\nVarious experiments with classification on ModelNet40, semantic segmentation on S3DIS and part segmentation on ShapeNet shows that the modification is indeed very useful and PointNetv3 achieves better accuracy and runs faster in comparison with PointNet++ and previous methods.\n\nThe paper is very well-written. The result is novel and interesting.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "0zw2InZ08X", "reviewer": "Reviewer_KEKc", "summary": "This paper studies the limitations of Set Abstraction in PointNet++. and proposed its variants including PreConv SA, Seperable SA, and Anisotropic Separable SA to reduce the computational cost and improve the accuracy. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method provided an improved version to the original set abstraction and increase the inference speed while maintaining almost the same or even better speed compared to PointNet++ and PosPool.\n\nThe paper is well-organized. I think the description of evolution from Vanilla SA and ASSA.\n\nJust a question on Equation (2), repeating f_i^{res} three times seems redundant even though multiplied by the distance in three directions. Have you tried other more efficient ways to incorporate anisotropic property, like using 1/3 channels as in PosPool, or simply addition?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "-Ty205cx1Kr", "reviewer": "Reviewer_sigp", "summary": "In this paper, the authors present PointNetV3, which is a faster and more accurate version of PointNet++.  The authors first analyze the latency bottleneck of PointNet++ and find that computation takes up to 70% of total latency, and then propose to reduce the computation cost via several techniques. Namely, separate SA modules are used to reduce the computation cost of vanilla SA (since MLP after reduction is much more efficient); Anisotropic reduction is applied to achieve higher accuracy (original PointNet++ shares conv. weights for all neighbors, which is suboptimal). The authors also analyze the effectiveness of width and depth scaling to improve the efficiency-accuracy tradeoff of PointNetV3 models. Experiment results on S3DIS, ModelNet40 and ShapeNet Part are impressive.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper attempts to analyze the efficiency bottleneck of the widely used PointNet++ architecture in 3D deep learning and improve its efficiency-accuracy tradeoff. I feel that the motivation / analysis of this paper is thorough and the proposed solution is simple and effective. The experiment results show that PointNet++-like architecture can achieve significantly better accuracy-efficiency tradeoff comparing with more complicated design such as KPConv, which is interesting. The overall writing quality of this paper is also good.\n\nHowever, I still believe that there are some challenges to be addressed in order to make this paper more influential.\n\n- The evaluation benchmarks chosen in this paper are all very small datasets, I'm quite curious whether PointNetV3 can be applied to larger datasets such as SemanticKITTI, nuScenes and Waymo to achieve state-of-the-art level accuracy. Even for indoor tasks, I believe ScanNet might be a better benchmark in 2021 comparing with S3DIS.\n\n- It seems that this paper assumes the number of input points per sample is around 15,000. This will be sufficient for most object / indoor-level datasets, but not quite sufficient for LiDAR perception tasks. For example, each scene on SemanticKITTI has ~120K points, and for Waymo each scene contains ~150K points. Can PointNetV3 scale up to these large scale scenes and perform competitively? Notice that at >100K points per scene, neighborhood query might become a significant bottleneck.\n\n- The comparison with point-based methods are really impressive, but SparseConv-based methods are not compared against. These methods are particularly strong at **one-pass inference** for the scenes. That is to say, we do not need to divide the input into sliding windows. I wonder if PointNetV3 can have advantage over SparseConv-based methods in terms of **per-scene latency** (not 16x15000 batched inference latency) and accuracy. Again, it is better to compare at large-scale scenes to make PointNetV3 more influential.\n\nI would raise my score if these concerns (especially the latter two) are addressed properly.\n\n===\n\nI've updated the score to 6 after reading the author response.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As I have indicated in the main review, the major weakness of this paper is that all experiments are done on small-scale datasets and the number of input points in the point cloud is not large (around 15K). It will be great if the authors can present some clue that the proposed method can scale up to large-scale input, in terms of both efficiency and accuracy.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new variant of PointNet that's faster and more accurate. This paper received positive reviews and all reviewers recommended acceptance. The reviewers find many positive points including impressive empirical results. AC does not find grounds to overturn this consensus recommendation. The authors should incorporate the suggestions of the reviews when revising the paper for the camera ready version. 
Decision: Accept (Spotlight)