Automated Mobile Attention KPConv Networks via A Wide & Deep Predictor
Authors: Tunhou Zhang
Mingyuan Ma
Feng Yan
Hai Li
Yiran Chen
Venue: ICLR 2022 Submitted (2022)
Keywords: 3D Point Cloud Classification and segmentation
Neural Architecture Search

Abstract: Kernel Point Convolution (KPConv) achieves cutting-edge performance on 3D point cloud applications. Unfortunately, the large size of KPConv network limits its usage in mobile scenarios. In addition, we observe that KPConv ignores the kernel relationship and treats each kernel point equally when formulating neighbor-kernel correlation via Euclidean distance. This leads to a weak representation power. To mitigate the above issues, we propose a module named Mobile Attention Kernel Point Convolution (MAKPConv) to improve the efficiency and quality of KPConv. MAKPConv employs a depthwise kernel to reduce resource consumption and re-calibrates the contribution of kernel points towards each neighbor point via Neighbor-Kernel attention to improve representation power. Furthermore, we capitalize Inverted Residual Bottleneck (IRB) to craft a design space and employ a predictor-based Neural Architecture Search (NAS) approach to automate the design of efficient 3D networks based on MAKPConv. To fully exploit the immense design space via an accurate predictor, we identify the importance of carrying feature engineering on searchable features to improve neural architecture representations and propose a Wide & Deep Predictor to unify dense and sparse neural architecture representations for lower error in performance prediction. Experimental evaluations show that our NAS-crafted MAKPConv network uses 96% fewer parameters on 3D point cloud classification and segmentation benchmarks with better performance. Compared with state-of-the-art NAS-crafted model SPVNAS, our NAS-crafted MAKPConv network achieves ~1% better mIOU with 83% fewer parameters and 52% fewer Multiply-Accumulates.

Under review as a conference paper at ICLR 2022
AUTOMATED MOBILE ATTENTION KPCONV NET- WORKS VIA A WIDE & DEEP PREDICTOR
Anonymous authors Paper under double-blind review
ABSTRACT
Kernel Point Convolution (KPConv) achieves cutting-edge performance on 3D point cloud applications. Unfortumately, the large size of KPConv network lim- its its usage in mobile scenarios. In addition, we observe that KPConv ignores the kernel relationship and treats each kernel point equally when formulating bigghbor-kernel correlation via Euclidean distance. This leads to a weak represen- tation power. To mitigate the above issues, we propose module named Mobile Attention Kernel Point Convolution (MAKPConv) to improve the efficiency and quality of KPConv. MAKPConv employs a depthwise kernel to reduce resource consumption, and re-calibrates the contribution of kernel points towards each neighbor point via Neighbor-Kernel Kernel attention to improve representation power. Furthermore, we capitalize Inverted Residual Bottleneck (IRB) to craft design space and employ a predictor-based Neural Architecture Search (NAS) approach automate the design of efficient 3D networks based on MAKPCoпY. To fully exploit the immense design space via an accurate predictor, we identify the im- portance of carrying feature engineering on searchable features to improve neural architecture representations, and propose a Wide & Deep Predictor to unify dense and sparse neural architecture representations for lower error performance pre- diction. Experimental evaluations show that our NAS-crafted MAKPCoнv net- work uses 96% fewer parameters on 3D point cloud classification and segmenta- tion benchmarks with better performance. Compared with state-of-the-an NAS- crafted model SPVNAS, our NAS-crafted MAKPCoнw network achieves ~1% better mIOU with 83% fewer parameters and 78% fewer Multiply-Accmmultts.
INTRODUCTION
3D applications such as object recognition (Wu et al. 2015). shape segmentation (Yu et al. 2004). and indoor-scene segmentation (Armeni et 2016) are growing popularity thanks to the rise 3D acquisition technologies and the increasing amount of geometric data collected from 3D sensors.
Deep learning models are becoming one of the most common and promising ways represent 3D data, especially 3D point cloud data, thanks to its high performance (Zhou et al. 2020 Zhao et al., 2020) and good efficiency (Cortinhal et al., 2020 Hu et al. 2020a).
Starting from PointNet (Qi et al. 2017a), point-based methods are becoming one of the most pop-
ular choices to extract features and make predictions for 3D input point clouds. Such an approach typically learns an individual representation for each data point and aggregates different represen- tations to formulate a signature for all global points, usually via end-to-end point convolutions (Xu) al. 2018 Atzmon et al. 2018). Kernel point convolution (KPConv) (Thomas et al. 2019) is the first work that maps the input points to the influence kernel points via linear correlation, and then further convolves the mapped features over the kernel points to compute weighted features.
Given a point yi. the kernel function g over neighbor points y; in KPConv carries g(;)) = Etek hyy, 3tx F WWW, leam representations via K kernel points, with the neighbor-kerne corre- lation h(y, TK). However, KPConv results in very large model sizes for 3D point cloud recognition due to inefficient local feature aggregation via point convolution over kernel weights Wt. As a re- sult, KPConv takes up to ~00G Caltiply-Acuumllate (MACs) to perform inference a single scene at the cost of ~200m on a TITAN XP GPU. Such resource consumption and inference cost too high for real-time execution in mobile scenarios. In addition, the above Neighbor- Weighbor-Kernel cor-

---

Under review as a conference paper at ICLR 2022
relation function has strong hidden assumption: the contribution of all kernel points measured equally towards the neighbor points. As a result of ignoring cross-kerne information from other kernel points to accommodate the joint evolution of different kernel points, such an assumption fails to enhance representation learning on important kemee points and thus yields sub-optimal results.
In this paper, we propose novel 3D attention module, named Mobile Attention Kernel Point Con- volution (MAKPConv). to address the above issues. MAKPConV employs two key design elements improve the performance and efficiency. First, we propose a depthwise kernel split feature extraction on point-wise features from channel-wise features during point convolution. Such design allows significant savings in both parameter count and amount of computation, without performance degradation. Second, we introduce Neighbor- Kernel attention (NK attention) to enable the kernel- wise information exchange between different kernel points via learnable attention networks. The joint evolution of all kernel points encourages representation learning on more important kernel points while suppressing less important ones, thus benefits from better learning adaptability of kernel point contributions towards neighbor point distributions in KPConv networks.
Inspired by Sandler et al. (2018), we capitalize Inverted Residual Bottleneck (IRB) to encapsulate МАКРCоn module into MAKPConv block, and craft MAKPConv design space to explore wider range of choices in 3D models. As 3D architectures are usually high-dimensionas unordered, tailoring good neural architecture to address such complex point cloud data requires a delicate and precise evaluation protocol. This is difficult to achieve with weight-sharing one- shot NAS (e.g.. SPVNAS (Tang al.. 2020)) because co-adaptation between sub-networks may degrade the quality of evaluation and lead to sub-optimal results. Thus, we pioneer the usage of predictor-based Neural Architecture Search (NAS) (Wen al. 2020) that utilizes a sample-based mechanism to reduce evaluation noise and seek better models on 3D point cloud. Specifically, we train a performance predictor based on exploitable samples, and probe the whole MAKPCoвw design space to obtain the best NAS model based on predictive performance. We observe that existing predictor-based methods adopt a fixed scheme (e.g.. one-hot encoding) to encode different types of features, leading to sub-optimal neural architecture representations with inaccurate priors. This is because real-valued features (c.g., block width) and categorical features (c.g., optional NK attention) have different priors and thus they should not be encoded in the same way. To address this issue, we formulate dense representations for features that have quasi-linear priors on model performance (e.g., depth, width), and formulate sparse representation for features that have non- linear sophisticated priors on model performance (e.g.. optional attention). Intuited by Cheng et al. 2016). we devise Wide & Deep predictor interact dense and sparse representations such that the combination of memorization and generalization reduces the prediction error.
unstructured and
By utilizing the Wide & Deep predictor perform NAS under the MAKPConv design space, our results on various benchmarks show promising results of the automatically discovered models in terms of both performance and efficiency. In particular, our MAKPConw networks outperform the original KPConv on both classification and semantic segmentation benchmarks with 92%~~93% fewer parameters, thanks to the efficient design of MAKPCoнv block. Empowered by our Wide & Deep predictor driven NAS, the NAS-crafted networks outperform the state-of-the-art NAS work, SPVNAS, by ~1 mIOU with 83% smaller size and 78% fewer MACs.
We highlight the contribution of this paper as follows:
We propose Mobile Attention Kernel Point Convolution (MAKPConv) module that com- bines depthwise kernel with Neighbor-Kernn Kernel attention to achieve efficient and high- performing architectures dedicated to learning on 3D point clouds.
We identify the importance of carrying feature engineering on neural architecture repre- sentations to improve predictor-based NAS. Specifically, features with dense and/or sparse representations given their priors, and utilize a Wide & Deep neural predictor to interact these representations for better accuracy prediction.
encode different searchable
2 RELATED WORK
3D Point Convolution Networks. Point convolution networks (Xu et al.) 2018 Hu al. 2020a
Lang et al. 2020) extends convolutional operations towards irregular point set grids to benefit from learnable radial function and spherical harmonics in 3D point clouds. Later work (Thomas al.
2

---

Under review as a conference paper at ICLR 2022
performs point convolution radius neighbors and aggregates convolved results to pro- duce learned representation without additional intermediate operators. Recent work on point con- volutions improves point sampling efficiency (Hu al., 2020a; Lang et al. 2020). local feature aggregations (Zhao et al. 2020 Lin et al. 2020), manual architecture fabrications (Hu al. 2020b Lei et al., 2020a), and/or improved training protocols (Goyal et al. (021) seek higher model performance and efficiency. However, these works neither simplify the structure of point networks improve model efficiency, nor specify an accurate neighbor- kernel correlation. Our work ex- tends kernel point convolution (Thomas et al. 2019) and addresses the above caveats: we improve the efficiency of local feature aggregation via a depthwise kernel on point convolution, and further introduce Neighbor-Kernel attention to enhance the representation power of depthwise kernel.
2019
Predictor-based Neural Architecture Search. Predictor-based NAS (Wen et al. 2020; Dudziak et al. 2020) is popularized method that trains a performance predictor on limited exploitable samples. The trained predictor serves as surrogate model of the ground-truth performance and utilized to guide architecture search on the full design space. Existing achievements on predictor- based NAS lies improving sample efficiency (Dudziak et al., 2020) and augmenting architecture samples (Liu et al., (021), yet they overlook the importance of carrying feature engineering neural architecture features. Our work identifies the importance of improving neural architecture representation, and assigns different priors representations and sparse representations. Our work further designs a Wide & Deep predictor to leverage and interact these representations for better prediction quality on unseen architectures.
wide set of searchable features and yield dense
3 MOBILE ATTENTION KERNEL POINT CONVOLUTION
We start with the formulation of KPConv (Thomas et al. 2019). Given a center point X E R and its corresponding features F E Pia, a kernel point convolution (KPConv) function carries all N neighbor points X,, {x' ||x' 2|| r} E RN and their features F E N within radius r
of K kernel points X R and accompanying weights W.. Specifically, correlation function: h x 17... + RK Din maps the unordered neighbor point features to a set of features F on kernel points. Usually, the correlation function represented and applied via = F. The correlation function is followed by kernel IR x - Rous. that convolves these features obtain the aggregated output F Here, Din denotes the dimension of input, and Dout denotes the dimension of output. The original KPConv overlooks an important opportunity to improve the efficiency by reducing the redundancy of kernel g that due to the simultaneously performed point-wise and channel-wis feature extractions. In addition, the original KPConv adopts a linear function as correlation: it treats the impact of each kernel point equally and ignores the adaptation of neighbor points towards kernel dispositions. For example, the center kernel point usually attracts more centralized neighbor points with a higher density, and edge kernel points perform critical jobs in recognizing boundary neighbor points with lower density in the radius sampling neighborhood. Thus, the original KPConv may have sub-optimal performance and efficiency 3D applications.
H E Nx
MAKPCoпv
Depthwise Kernel
Xx
Xn
Outputs
Inputs
Atten NK-
F: D]
[K. D]
W
Depthwise Kernel
[N
Attention
F: (.
F':DD]
F`: [D]
Den
Dont
NK Attention
Average Pooling
[1
MLP
[1,K]
1.0
[N,K]
Linear Function
[N.
Global
[N, K]
Correlation
Distance
Linear Correlation
NK Correlation
Figure I: MAKPConv module combines depthwise kernel with Neighbor- Kernel attention to craft more efficient and high-performing 3D networks. Darker color indicates greater kernel contribution.
3

---

Under review as a conference paper at ICLR 2022
To overcome the above limitations, we propose Mobile Attention Kernel Point Convolution (MAKP- Conv) module as an il-place improvement of KPConv block to seek higher quality and efficiency. Figure I provides an overview of MAKPConv. Specifically, we design depthwise kernel to address the computation cost: A depthwise kernel 9 carries element-wise multiplication in place of matrix multiplication between aggregated features and kernel weights, thus achieves memory and compu- lation reduction. Considering a depthwise kernel has limited representation power, we introduce Neighbor- Neighbor-Kemel attention (NK attention) as an improved correlation function to emphasize repre- sentation learning on important depthwise kernels and fully utilize the capacity of kernel points. This compensates for the reduced representation power: the joint evolution of kernel points via learnable attention encourages effective representation learning on important depthwise kernels and best ex- ploit their potential. As result, the fusion of depthwise kernel and NK attention into MAKPConv block has the potential to achieve high- seghhperfomminggg yet efficient 3D models.
3.1 DEPTHWISE KERNEL.
In original kernel function of KPConv, a regular kernel takes input F E I over K kernel points and uses a weight kernel W E R xxD,, x Dout to simultaneously perform point-wise transfor- mation and channel-wise transformation. A regular kernel g produces output F' E RDott
F'1 - 9(F,
Σ Σ Fu.i Wk.5.ds
(1)
kEK deDio
where l is kernel index, and d is channel index. A regular kernel leads K x Din Dout parameters and Dan x Dout x K MACs during inference, and leads to large computation and memory for the deep layers of the 3D architectures. To favor model efficiency, we design a depthwise kernel 9 that only performs point-wise feature extractions over input point clouds, see Figure [a], a). The proposed depthwise kernel adopts learnable weights W IR kx Din, extracts point-wise information
on each channel, and obtains an output F* E R Dout as follows:
F'J = (==((,W)
Σ KEK
Fi.d Wk.ds
where is a kernel index, and d is a channel index. Here, Dout Din D as a depthwise kernel does not operate on channel-wise features. As result, depthwise kernel point convolution only leads to K x Din parameters and only Din x K MACs during inference, and reduces Dout X parameters and MACs during convolution compared to regular kernel. The parameter saving from a depthwise kernel is significant when Dout is large, especially in deeper layers of the network.
3.2 VEIGHIBOR-KERNE ATTENTION
Although depthwise kernel ğ adds to model efficiency, such design lacks representation power to fit complex point cloud structures due to inactive channel-wise communication. To compensate for the loss, we propose Neighbor- Teighbor-Kermel Kernel attention (NK attention) h gxNx + 12 x D.n that re-weights the contribution of kernel points by learning joint representation H N on Neighbor- Neighbor-Kernel distances (NK distance) via an attention network, see the lower part of Figure Such representation, when being applied neighbor point features F, can fully harness the capacity provided by different kernel points, thus provide better aggregated features F for the depthwise kernel.
NK attention first adopts linear function to transform NK distance into linear NK correlation, and applies average pooling to obtain global neighborhood correlation. This global information captures rich information over the statisties of the locations of neighbor points and thus gives a high- level abstraction on the contribution of each kernel point towards the global neighborhood. Then, the global neighborhood correlation is fed into a 2-layer attention network to enable cross-kernel interaction and learn a good representation to quantify the contribution of kernel points, followed by a sigmoid gating to normalize the learned representation. Unlike Squeeze-and anneeeee-and-exiiatiooono (Hu al.) 2018). NK attention adopts full-rank transformation to preserve the information needed towards every kernel with marginal parameter cost. Finally, the learned representation is scaled to the original NK linear correlation, followed by bypass (i.e., addition) of the original linear NK correlation for easier optimization. As result, NK attention grasps richer representation H that incorporates the joint evolution of kernel points and encourages representation learning on important depthwise kernels, This leads concrete performance improvement on 3D point-cloud benchmarks.
4

---

Under review as a conference paper at ICLR 2022
4 NERUAL ARCHITECTURE SEARCH WITH WIDE & DEEP PREDICTOR
In this section, we introduce predictor-based Neural Architecture Search (NAS) driven by novel Wide & Deep predictor, see Figure 2 MAKPConY module provides efficient motifs for process- ing point-wise features. To incorporate channel-wise feature extraction into MAKPCoпV module, we exploit the design of Inverted Residual Bottleneck (IRB) (Sandler et al. 2018) and sandwich MАKPConv module with 2 linear transformation construct MAKPConv block. To design effi- cient yet high performing 3D models, we utilize MАKPCoпv block exploit predictor-basse baaee neural architectures with wide set of features such positional organiza- tions, structural settings, and accessories. As formulating neural architecture representations from these features is the key in predictor- NAS, we identify the importance of carrying engineering searchable features. Thus, assign different priors to these features to formulate dense archi- tecture representations and sparse architecture representations, and propose novel Wide & Deep neural predictor to effectively interact different sets of neural architecture representations for accu- rate predictions. Wide & Deep predictor trades-off generalization and memorization via dense deep models and sparse wide embeddings, thus reduce prediction MSE unseen architectures.
architecture motif and
(a)
(b)
MAKPConv Block
Stem
Accessories Structural
Positional
D=D1
[b. W
Linear
[b, x
(K, Atm)
[b. E]
Head
Dense Inputs
Linear
Sparse Inputs
MAKPCoпv Block
[b, W]
Dot Product
MAKPConv Module
Fixed
Pooling
Embedding
[b. W]
Searchable
Prediction
Linear
Positional: Depth D. Width W
Structural: Expansion Factor E
Accessories: Kernel Size K, Attention Option Atn
Figure 2: (a) Design space based MAKPCовv blocks. (b) Wide & Deep predictor.
4.1 MAKPCONY BLOCK
Inspired by Sandler et al. (2018), we sandwich MAKPConY by linear transformation layers to derive MAKPCoнw block as a promising architecture motif to carry efficient and high performing 3D point cloud learning. It has a similar structure as Inverted Residual Bottleneck (IRB) and is described by four key parameters: expansion factor E, kernel size K, output width W. and optional choice of NK attention Atn. We follow the positional setting of MobileNetV2 to manually craft MAKPConV network a hand-crafted baseline verify its performance and efficiency. We compare the performance and size between hand-crafted MAKPConV network and KPConv under the native KPConv protocol in Table [ Compared with KPCony, the hand-crafted MAKPConv achieves on-par Overall Accuracy (OA) with at least 92% smaller size on ModelNet40, and achieves 0.4% higher mIOU with 93% smaller size on SemanticKITTI, even without NK attention. Such advantage is attributed to the efficient depthwise kernel. With NK attention, MAKPConw yields ~0.4% OA gain on classification and ~0.5% mIOU gain on semantic segmentation with marginal increase in parameter count. These results suggest that re-weighing the contribution of kernel points via learnable attention network enhances depthwise kernels thus improves performance.
Table Performance of hand-crafted MAKPCony vs KPConv on ModelNet40 and SemanticKITTL.
Cls.
Seg.
ModelNet40
SemantieklTTI
Architecture
Params (M)
Params (M)
OA (%)
mIOU (%)
KPConV (Our impl.)
14.9
14.8
92.3
59.2
MAKPConv (w.o. attention)
1.24
0.97
91.8
59.6
MAKPConV
1.25
0.98
92.2
60.1
5

---

Under review as a conference paper at ICLR 2022
4.2 MAKPCONY DESIGN SPACE
We utilize MAKPConv blocks to construct flexible network design space. Specifically, we divide the overall 3D network into a stem KPConv for initial feature extraction, head MLP for carrying fine-grained classfiicaiionkeggmenttiiinn and several searchable stages. In each searchable stage, we jointly search for the optimal configuration MAKPConv blocks, including the positional organi- zations, structural settings, and accessories. We claborate these searchable features below.
Positional Organizations. The block width and depth of MAKPConw has direct impact on the latency of MAKPConv models by affecting both computation-boundee operations (e.g., linear trans- formation) and mnemory-bonnder operations (e.g., indexing). As deeper and wider architectures lead to better performance in 3D point cloud classification and segmentation following the Pareto fron- tiers, positional organizations have a positive quasi-linear impact towards performance and take continuous values on any positive integers.
Structural Settings. The expansion ratio affects the inner-structure of MAKPConv block that has positive quasi-linear correlation with performance. However, co-adaption of expansion factor towards block width may induce challenges when learning a mapping rule to accurately predict architectures in the search space. Structural settings take continuous values any positive integers.
Accessories. We mark the search of optional attentions and kernel size as accessories in the MAKP- Conv search space. Accessories have sophisticated impact model performance. For example, as kernel point convolution utilize different dispositions for different number of kernels, it is unclear whether the change in the number of kernel points affects the quality of kernel disposition and the model performance. Both accessories can only take categorical values.
Considering the high variance in 3D point cloud tasks and the robustness to noisy evaluations, we adopt predictor-based NAS to explore MAKPConv design space. However, MAKPConv design space prohibitively large: a total of 11 stages and 7 searchable features within each stage leads a total search space size of 1.8x 10 possible architectures. Such large and complex search space calls for high-performing predictor-based NAS to accurately model the design space.
4.3 WIDE DEEP PREDICTOR
Neural architecture representation is the key in the performance of predictor-based NAS due to its strong connection with predictor performance. Specifically, better neural architecture representation links searchable features towards priors that more reflective to the final performance, and thus increases the chance to discover high- perfoperformnn NAS models.
To improve neural architecture representations, We focus on carrying feature engineering to dis- tinguish searchable features with different priors. Motivated by the aforementioned relationship between searchable features and the model performance, assign different priors to wide set of searchable features and craft dense and sparse neural architecture representations for corresponding features. Specifically, we formulate dense neural architecture representation for searchable features that can take continuous values (e.g., positional organizations, structural settings), and formulate sparse neural architecture representation for searchable features that can only take discrete values (e.g., accessories). With dense and sparse neural architecture representations, we follow the idea of Wide and Deep Models (Cheng al. 2016) design a Wide Deep predictor that trades-off mem- orization and generalization in performance prediction, see Figure Wide & Deep model adopts separate MLP to obtain high-level dense outputs based on each of the dense representations, and adopts an embedding table to convert sparse representations to continuous sparse outputs. To interact dense and sparse outputs, we leverage the DotProduct block in DLRM (Naumov et al., 2019) to fuse learned dense representations and sparse representations. Specifically, given dense representation RBx Xdim, DotProduct carries cross-representtttoo
D E Rd Bxii and a sparse representation communication by carrying the following transformation:
X [vec (D); S], Producttt
Triu(XXx)
(3)
where dim represents the dimension of both dense and sparse representations, N denotes the number of sparse inputs, and B denotes the batch size. Eq. (3) first vectorizes the dense tensor D to have the same dimension as the sparse tensor S, and extracts the upper triangular matrix upon self matrix factorization. We carry 2 DotProducts on separate pairs of dense and sparse representations to encourage the learning of independent interactions. This is because positional organizations and

---

Under review as a conference paper at ICLR 2022
Table 2: Wide & Deep Predictor achieves best prediction quality SemanticKITTI.
Predictor
Rank Loss?
Feats?
Pretraining?
MSE (10-
Kendall
Random Forest
Dense / Sparse
3.75+0..0
0.240-0.04
GPR
Dense / Sparse
4.29:10,1
0.144=0.042
XGBoost
Dense Sparse
4.24+0.40
0.210=0.037
NGBoost
Dense Sparse
3,90+0.33
0.255-0.047
LGBoost
Dense Sparse
4.03:10.36
0.236-00033
Deep
Dense
3.07++0.33
0.379=0.050
Wide
Sparse
2.87+00.1
0.400=0.047
Wide Deep
Dense Sparse
2.80+0.23
0.408=0.044
structural settings may not necessarily have the same influence over the choice over accessories.
After fusing dense and sparse representations via DotProduct, we utilize a MLP to learn predictions
based on the fused representations in the Wide & Deep predictor.
4.4 TRAINING AND EVALUATION OF THE WIDE & DEEP PREDICTOR
Architecture Sampling. We randomly sample ~ architectures on both ModelNet40 and Se- ManticKITTI. Following the common NAS pipeline, we split the training dataset of Model- Ner40/SemaniiKITTI into mini-train and mini-val. We train the sampled architectures on mini-train until convergence, and evaluate these architectures on mini-val to obtain architecture performance.
Predictor Training. Mean-Squared Error (MSE) is the training objective of the Wide & Deep Pre- dictor on 1K sampled architectures. To ensure a consistent comparison of predictor performance over multiple benchmarks, we carry optimization ing from to 1. Since the Wide & Deep Predictor can perform gradient-based optimization, we incorporate predictor pretraining on MACs (Dai et al., 2021) and rank loss (Dudziak et al. 2020).
predictors with normalized performance rang-
We conduct extensive experiments on various top-performing NAS predictors collected from NASBench-301 (Siems et al. 2020). To justify that interacting dense and sparse representations benefits predictor performance, we establish "Wide" as a MLP Predictor that accepts sparse neural architecture representations transformed from categorical encoding of features, and establish "Deep" as a MLP predictor that accepts dense neural architecture representations transformed from real- valued encoding features. These predictors also leverage rank loss (Dudziak al.. 2020) and predictor pretraining (Dai et al.] 2021). thus are utilized as strong baselines for gradient- based NAS predictors justify the significance of dense-sparse feature interaction. We list the supported train- ing heuristics and compare the performance (i.c., MSE and Kendall r) of Wide & Deep Predictor over various NAS predictors under 10 different runtimes. Compared with all non- aon-gradient predic- tors (Chen & Guestrin 2016 Ke et al., 2017]] Duan et al., (2200). Wide & Deep predictor achieves accuracy prediction, thanks the potential of ranking- aware training and predictor pretraining via gradient-based optimization. Compared to gradient-based pre- dictors using either dense or sparse features, Wide Deep predictor achieves better prediction qual- ity thanks to better architecture representation. Such improvement in predictor precision eventually yields the discovery of better NAS models, see the elaborations in Section 6.2.
lower MSE and higher Kendall
EXPERIMENTS
In this section, we evaluate MAKPCoнw over classification (ModelNet40 (Wu al.) 2015)) and semantic segmentation (SemanticKIITI (Behley et al., 2019)) benchmark.
5.1 SEARCH CONFIGURATION
MAKPComv Design Space. Each block can choose from kernel size (5, 7, 13) to perform MAKP- Conv operation. Each block has up to 3 different choices for block depth, block width, and expansion factors. In addition, each block has the choice of an optional NK attention to balance performance and efficiency. We describe other details of the MАKPConv design space in Appendix.
Regularized Evolution. Following Real et al. (001)) we employ regularized evolution and our Wide & Deep predictor to effectively probe the MAKPConv design space. Given candidate chitecture A with predicted performance P and MACS M, use S(A) P log M as

---

Under review as a conference paper at ICLR 2022
search objective, and empirically set 3 to 0.5 balance performance and resource consumption both ModelNet40 and SemanticKITTL. We use population of 200 and sample size 150 to carry regularized evolution with 360 rounds over MAKPCонv design space to craft NAS models.
5.2 EVALUATION ON 3D BENCHMARKS
We use Wide & Deep predictor to automate the design of MAKPCoвv blocks and discover promising models on ModelNet40 and SemanticKITTI 3D applications. To ensure a fair comparison with larger models SemanticKitti, crafted models and attach a mx suffix to denote the application of a mxx width multiplier. We compare the performance and efficiency metries such as parameter count and MACs.
apply width scaling (Sandler al., 2018) the NAS-
ModelNet40. We follow the RSCNN protocol in SimpleView (Goyal al., 2021 and show the evaluation results on the testing dataset of ModelNet40 in Table 3 For fair comparison, we report the mean accuracy across runs and observe a 0.2% OA variance. Compared to KP- Conv, the hand-crafted МАКРCопм achieves on-par OA with ~92% fewer parameters thanks to the proposed MAKPConм module. Our NAS approach explores more efficient model designs: the NAS-crafted MAKPConv achieves 0.5% higher OA than hand-crafted counterpart while being 55% smaller. Compared with effi- cient 3D) models (Tei et al. 2020b Goyal et al.]
Table 3: Overall Accuracy (OA) on ModelNet40.
*: Use native protocol instead of SimpleView RSCNN evaluation protocol.
Params (M)
OA
Architecture
(%)
SPH3D-GCN* (Lei 2020b
0.8
92.1
FPConv" Lin
2020
2.1
92.5
RSCNN
al.
1,3
92.5
DGCNN Wang
2019
1.8
92.8
KPConv Thomas al
2019
14.9
92.9
SimpleView (Goyal
2021
0.80
93.2
PointNet++ [QI ct 2017b
1.48
93.3
MAKPConv, hand-cralted
1.25
92.6
MAKPConv + NAS
0.56
93.1
MAKPCony NAS, 2x
1.21
93.4
2021 Lin al. 2020; Liu et 2019: Wang al. 2019). NAS-crafted MAKPCовм achieves on-
par performance with ~30% fewer parameters. By further scaling the model, our NAS-crafted MAKPConv outperforms its hand-crafted counterpart by 0.8% OA with similar parameter count.
SemanticKITTI. Table shows the mIOU comparison on sequence 08 of the SemanticKITTI dataset. Our hand-crafted MАKPCопw outperforms point-based, projection- based and voxel-based methods by at least 0.9%, 1.1% % and 1.2% mIOU respectively, with significant parameter and MAC reduction. The NAS-crafted MAKPCoвv achieves 1% higher mIOU than the state-of-the-an SPVNAS while saving 83% parameters and 78% MACs. Applying width multiplier NAS-crafler model further boosts the performance. We claborate on the latency comparison in Appendix A.7.
Table 4: mIOU on sequence 08 (validation split) of SemanticKITTI. Latency is measured with NVIDIA TITAN Pascal on single SemanticKITTI scene containing ~60K points. Here, red numbers denote computation time and blue numbers denote processing time. +: Results from Zhou
al. 202T). * Results from Tang et al. (022)).
Architecture
Method
Params (M)
MACs
Latency (ms)
miou
(G)
(%)
RandLANct (Hu et al. 2020a
Point-based
1.24
103
57.1
KPConv-rigud (our ımpl.)
Point-bases based
14.8
60.9
221 (164 + 57)
59.2
PolarNet (Zhang et al. 2020
Projection-based
13.6
135,0*
62
58.2*
SalsaNext (Cortinhal ct al.
2020
Projection- based
6.7
62.8*
71
59.0
MinkowskiNct (Choy ct al
2019
Voxcl-based
5.5
28.5*
294
58.9
MAKPConN, hand-crafted
Point-based
0.97
4.7
160
(103 + 57)
60.1
3.3
20.0
158
61.5
SPYNAS (Tang al.. 2020
Voxel-based
7.0
34.7
175
63.5
MAKPConV + NAS
0.57
4.4
169
(112 + 57)
62.4
Point-based
MAKPCoнV NAS, 2x
11.0
206 (149 + 57)
64.1
ABLATION STUDIES
In this section, we first demonstrate how NK attention re-calibrates the contribution of each kernll point by visualizing the relationship between NK attention and NK linear correlation. Then, we dis- cuss the relationship between predictor MSE reduction and quality improvement the discovered NAS models to demonstrate the importance of predictor design in predictor-based NAS.
8

---

Under review as a conference paper at ICLR 2022
6.1 KERNEL CONTRIBUTION VIA NK ATTENTION
Empirical evaluations reflect that NK attention leads to 0.4% OA gain and 0.5% mIOU gain on 3D object classification and semantic segmentation. To verify the source of improvement of MAKP- Conv over its non-attention counterpart and justify the re-calibration of kerne points, we plot the learned NK attention 7 kernel points in Figure 3 and identify the contribution of each ker- nel via the slope. We observe that the learned attention for each kernel point has a strong rela- tionship with kernel dispositions (i.e., 1-4-2 groups among symmetrical axis) in the Octahedron. The two vertices that are the most distant from the center point shows the most contribution to-
0.75
wards neighbor points. This is because these kernel points are involved in addressing the boundary effects of input point clouds for more robust recognition results. We also observe that the 4 vertices connecting the square plane has the least contribution, illustrating that such high concentration of kernels causes redundancy and suggests a future direction in optimizing kernel location to fully capitalize the kernel points to improve representation power. The center ker- nel point admits most information from central- ized neighbor points, thus co-adapts to global neighbor information and yields an moderate contribution towards neighbor points.
KS
0.70
Attention
K2
0.65
K1.slope=0.746
0.60
K2,5lope=0.729
K3,slope=0.722
0.55
K4.9oo==-0.788
k5.slope=0.717
K6,5lope=0.777
0.50
K7,slope=0.731
NK Linear Confelation
Figure 3: Visualization of learned NK attention
matches the octahedron disposition of kernels.
6.2 EFFECTIVENESS OF WIDE & DEEP PREDICTOR
We show that even marginal MSE reduction (i.e., 0007x10-2) leads significant quality improve- ment of NAS-crafted models, which justifies the importance of feature and architecture engineering in predictors. We establish random search baseline by selecting the top-5 models from the outcome of 1K random samples, and establish predictor-based NAS baseline by selecting top-5 models us- ing the top-per pop-performing Wide NN predictor in Table [ All models are evaluated on classification and segmentation benchmarks using the same 100-epoch training pipeline. We report the mean and standard deviation of performance on ModelNet40 and SemanticKITTI Table 5
Table 5: Comparison of various predictor-based NAS methods with Random Search baseline. Re- sults are derived from the t NAS models discovered by each method.
ModelNet40 OA ( (%)
SemanticK mlou ( (%) mean
Search Method
mean
std
max
min
std
max
min
Random
90,96
0.47
91.5
90.2
53.54
1.19
54.68
51.96
Predictor-based Wide
91.14
0.23
91.5
90.9
54.90
0.49
55.28
54.2
Predictor- based + Wide & Deep
91.22
0.16
91.4
91.1
56.10
0.85
56.83
54.87
Results demonstrate that: (1) without introducing extra architecture samples, predictor-based NAS can outperform random search by up to 1.4% mIOU on SemanticKITTI. (2) Even with 0.07 x 10-2 lower MSE, the NAS-crafted models via Wide & Deep predictor can outperform NAS-crafted mod- els via Wide predictor by large margin (i.c., up to 1.2% mIOU on SemanticKITTD, The above results emphasize the need to improve predictor-based NAS via a delicate neural predictor design.
CONCLUSION
In this work, we present MAKPConY module to improve both efficiency and performance of KP- Conv. MAKPConv employs depthwise KPConv that separates point-wise feature extraction from channel-wise feature extraction to reduce computation cost and memory consumption, and NK at- tention that re-weighs the contribution of different kernel points via interactions of neighbor-kernel correlation through an learnable attention network. We further incorporate the IRB design to con- struct MAKPConv block and propose automate the model design. Experiments and ablation study over 3D classification and 3D semantic segmentation benchmarks verify the effectiveness of MAKPConv.
predictor- based NAS driven Wide & Deep predictor to

---

Under review as a conference paper at ICLR 2022
8 REPRODUCIBILITY STATEMENT.
We make tremendous efforts to ensure reproducibility of our proposed method. Specifically, mention the detailed configuration of predictor-based NAS Section 5.1, introduce the design of hand-crafted MАKPCопv design in Appendix A.1/A.2, describe the detailed configuration of Wide & Deep predictor in Appendix A.3, show the transferability searched architecture in Appendix A.4, cover the training protocols in Appendix A.5, and use Appendix A.6 justify orthogonal contribution of our improving performance predictor and other line of research in improving search algorithms. In addition, we use Appendix A.7 give an analysis searched models and points an direction on further improving the efficiency of MAKPConv kernel seek higher efficiency in 3D point cloud applications. Code that covers other details of this work is available via this link
the latency breakdown of
REFERENCES
Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, loannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 1534-1543, 2016.
Matan Atzmon, Haggai Maron, and Yaron Lipman. Point convolutional neural networks by exten- sion operators. arXiv preprint arXiv: 10091, 2018.
Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9297-9307, 2019.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:19033 2019.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pp. 785-794, 2016.
Heng- -ze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recom- mender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pp. 7-10, 2016.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3075-3084, 2019.
Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy. Salsanext: fast, uncertainty-aware seman- tic segmentation of lidar point clouds for autonomous driving. arXiv preprint arXiv:2003.03655 2020.
Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al. Fbnety3: Joint architecture-recipe search using predic- tor pretraining. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16276-16285, 2021.
Tony Duan, Avati Anand, Daisy Yi Ding, Khanh Thai, Sanjay Basu, Andrew Ng, and Alejan- dro Schuler. Ngboost: Natural gradient boosting for probabilistic prediction. In International Conference on Machine Learning, pp. 2690-2700. PMLR, 2020.
Eukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D Lane. Brp-nas: Prediction-based nas using gens. arXiv preprint arXiv:2007.08666. 2020.
10

---

Under review as a conference paper at ICLR 2022
Siqi Fan, Qiulei Dong. Fenghua Zhu, Yisheng Lv, Peijun Ye, and Fui Wang. Scf-net: Learn- ing spatial contextual features for large-scale point cloud segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 14504-14513, June 2021.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR), 2013.
Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with simple and effective baseline. arXiv preprint arXiv:2106.0552 2021.
Jie Hu, Li Shen, and Gang Sun. aoneezze-and-xcitaatiin networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
11108- 11117, 2020a.
Zeyu Hu, Mingmin Zhen, Xuyang Bai, Hongbo Fu, and Chiew-lan Tai. Jsenet: Joint semantic segmentation and edge detection network for 3d point clouds. In Computer Vision-ECCE 2020: 16th European Conference. Glasgow, UK. August 23-28, 2020, Proceedings, Part XX 16, pp- 222-239. Springer, 2020b.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightghm: highly efficient gradient boosting decision tree. Advances neural information processing systems, 30:3146-3154, 2017.
Itai Lang, Asaf Manor, and Shai Avidan. Samplenet: Differentiable point cloud sampling. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7578- 7588, 2020.
Huan Lei, Naveed Akhtar, and Ajmal Mian. Seggen: Efficient 3d point cloud segmentation with fuzzy spherical kernel. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11611-11622 2020a.
Huan Lei, Naveed Akhtar, and Ajmal Mian. Spherical kernel for efficient graph convolution 3d
point clouds. IEEE transactions on pattern analysis and machine intelligence, 2020b.
Yiqun Lin, Zizheng Yan, Haibin Huang, Dong Du, Ligang Liu, Shuguang Cui, and Xiaoguang Han. Fpconv: Learning local flattening for point convolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In CVPR, 2019.
Yuqiao Liu, Yehui Tang, and Yanan Sun. Homogeneous architecture augmentation for neural pre- dictor. arXiv e-prints, pp. arXiv-2107, 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608 03983, 2016.
Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundara- man, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G Azzolini, et al. Deep learning recommendation model for personalization and recommendation systems. arXiv e-prints, pp. arXiv-1906, 2019.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in neural information processing systems, pp. 5099-5108, 20.
11

---

Under review as a conference paper at ICLR 2022
Shi Qiu, Saeed Anwar, and Nick Barnes. Semantic segmentation for real point cloud scenes via bilateral augmentation and adaptive fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1757-1767, June 2021.
Haoxi Ran, Wei Zhuo, Jun Liu, and Li Lu. Learning inner- meer-group relations on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15477- 15487, October 2021.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference artificial intelligence, volume 33, pp. 4780-4789, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chigh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas- bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.099 09777, 2020.
Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Search- ing efficient 3d architectures with sparse point-voxel convolution. In European Conference on Computer Vision, pp. 685-702. Springer, 2020.
Hugues Thomas, Charles R Qi, Lean-Elmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6411-6420, 2019.
Yue Wang, Yonghin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning point clouds. Acm Transactions On Graphics (tog), 38(5): 1-12, 2019.
Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In European Conference on Computer Vision, pp. 660- 676. Springer, 2020.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning on point sets with parameterized convolutional filters. In Proceedings of the European Conference Computer Vision (ECCV), pp. 87-102, 2018
Xiaoqing Ye, Jiamao Li, Hexiao Huang, Liang Du, and Xiaolin Zhang. 3d recurrent neural networks with context fusion for point cloud semantic segmentation. In Proceedings of the European Con- ference on Computer Vision (ECCV), pp. 403-417, 2018.
Yizhou Yu, Kun Zhou, Dong Xu, Xiaohan Shi, Hujun Bao, Baining Guo, and Heung- Yeung Shum. Mesh editing with poisson-baas based gradient field manipulation. In ACM SIGGRAPH 2004 Papers, pp. 644-651. 2004.
Chris Zhang, Wenjie Luo, and Raquel Urtasun. Efficient convolutions for real-time semantic seg- mentation of 3d point clouds. In 2018 International Conference on 3D Vision (3DV), pp. 399-408. IEEE, 2018.
Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, and Hassan Foroosh. Polamet: An improved grid representation for online lidar point clouds semantic seg- mentation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 9601-9610, 2020.
12

---

Under review as a conference paper at ICLR 2022
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. arXiv preprint arXiv:2012.09164, 2020.
Hui Zhou, Xinge Zhu, Xiao Song, Yuexin Ma, Zhe Wang, Hongsheng Li, and Dahua Lin. Cylin- der3d: An effective 3d framework for driving-scene lidar semantic segmentation. arXiv preprint arXiv:2008.01155 2020.
Zixiang Zhou, Yang Zhang, and Hassan Foroosh. Panoptic-polarnet: Proposal-free lidar point cloud panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13194-13203, 2021.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint 1..0577 2016.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
In this appendix, we elaborate the detailed structure of hand-crafted MAKPConv via MAKPCony blocks. We also verify the transferability of NAS discovered architecture S3DIS (Armeni al.) 2016) indoor scene segmentation benchmark. Furthermore, we discuss detailed hyperparameter settings for training and evaluating 3D architectures on ModelNet40, S3DIS, and SemanticKITTI benchmarks, and visualize the optimization curve in the search process for both vanilla predictor- based NAS (i.e., via Wide Predictor) and our improved predictor-basee NAS (i.e., via Wide Deep predictor). We also discuss the latency breakdown of our searched models and propose some future work in improving the efficiency of MAKPConv series models.
A.1 HAND-CRAFTED МАKРCONV NETWORK
Table 6: Structure of hand-crafted MAKPCовw. "1/2" strides means upsampling by 2x. Note that stage 1~7 is utilized to perform classification, and stage 8~11 1 is utilized to perform semantic segmentation.
Table O illustrates the detailed archi- tecture of our hand-crafted MAKP- Conv. The hand-crafted MAKPConv contains total 17 MАKPCон blocks for classiciation, and 4 ad- ditional MАKPCопу blocks for se- mantic segmentation. The positional setting (i.e., depth, block width) of hand-crafted MAKPConV strietly fol- lows MobileNetV2, which is con- sidered as the state-of-the-art hand- crafted model with remarkable effi- ciency. For structural settings, we employ expansion factor of for all MAKPCoнw layers. addition, leverage kemnll size of 7 for each MAKPCон block for design efficiency. As result, MAKPCoвм serves as strong baseline to compare with when evaluating the performance of NAS-crafted МАKPCопч models.
In Width
Out Width
Stage
Depth
Stride
E
K
I
16
16
7
2
2
16
24
3
7
3
2
24
32
3
4
4
2
32
64
3
7
5
1
64
96
3
6
2
96
160
3
7
160
320
3
8
1/2
416
160
3
7
1/2
160
96
3
10
1/2
96
64
3
7
11
1/2
64
32
3
A.2 MAKPCONV DESIGN SPACE
We present the detailed settings of the MAKPCoвw design space in Table 7 Despite the search of block depth, block width, expansion factor and kernel size, we also incorporate the search of optional attention into MAKPConY design space trade-off the overhead in Neighbor- Seighbor-Kernel Kernel Attention and its performance gain.
Table 7: Configuration of the MAKPConv design space. "1/2" strides means upsampling by 2x. Hierarchy
Stage
Strides
Depth
Kernel Size (K)
Expansion (E)
Width
5, 7, 13
1
16
2,3
5, 7,
16,
2
2,
5, 13
2, 4
24, 32
Classification Backbone
3, 4.
5,7, 13
2,
24, 32, 40
2,
5, 13
2,3.
40, 56, 72
6
2
3,4,
5.7. 13
2.
64, 80, 96 160
5,7. 13
2,
1/2
5,7,
2,3
64, 80, 96
Segmentation Head 11
1/2
5. 13
2.3
40, 56, 72
1/2
5,
2.
24, 32, 40
1/2
5, 13
2.
16, 24
A.3 DESIGN OF WIDE & DEEP PREDICTOR
We configure the architectural hyperparameters of our wide & deep predictor via intuition without delicate architecture engincering. Specifically, we employ a 3-layer MLP with 64- 128-256 layers the dense architecture to extract dense neural architecture representations for both positional
14

---

Under review as a conference paper at ICLR 2022
organizations and structural settings. The overall architecture that is responsible for processing fused features after dense-sparse interaction a 2-layer MLP with 256-256 layers. We utilize ReLU as the activation function and apply Dropout of 0.5 before the final regression head to mitigate overfitting. To ensure fair comparison, both the Wide predictor and the Deep predictor also adopt 2-layer MI.P with 256-1 128 units to keep the same maximum projection dimension the Wide & Deep predictor.
A.4 ARCHITECTURE TRANSFERABILIT ON S3DIS
We further verify the transferability of NAS-crafted MAKPConv architectures on S3DIS to evaluate its performance on indoor scene segmentation. Following the established protocols in 3D point cloud segmentation, we report both 6-fold cross-validation mIOU across all Area I~Area 6 splits, see Table [ By convention, we also report the validation mIOU and mIOU per class on Area 5 to compare with existing methods on 3D point cloud, see Table 9 Though our NAS-crafted model is not optimized in S3DIS dataset, it shows a significant margin on S3DIS 6-fold cross- validation benchmark compared with existing approaches such Qiu et al. (2021); Zhao et al. 2020) and achieves 74.5 mIOU, new state-of-the-art result using only 1.35M parameters.
Area of the S3DIS dataset, observe that: (1) Our NAS-crafter MAKPConv achieves 1.8% higher mIOU than original KPConv with 11x parameter efficiency. (2) Our NAS-crafted MAKPCонw achieves competitive performance over different classes S3DIS indoor semantic segmentation and demonstrate ~1% higher mIOU over 9 out of 13 classes compared to existing approaches. Thus, our NAS-crafiee models have good transferability towards alternative benchmarks, and our NAS approach can be further incorporated into other 3D point cloud tasks such as nuScenes (Caesar
al. 2019). KITTI (Geiger al. 2013) improve the current state-or oftee-of-eha-ttt -tt--arrt models.
Table 8: 6-fold cross-validation results S3DIS.
Architecture
mlOU (%)
mAce( %)
DA(%) 88.0
Params (M)
RandLANet (Hu al.
2020a
70.0
82.0
1.24
KPConv (Thomas et al.
2019
70.6
79.1
14.8
RPNet [Ran et al.
202T
70.8
SCF--Net (Fan et al.
2021
71.6
82.7
88.4
BAAF-Net (Quu et al.
2021
72.2
83.1
88.9
1.23
PointTransformer (Zhao et al.
2020
73.5
81.9
90.2
4.9
MAKPConv + NAS, 2x
74.4
82.1
90.3
1.35
Table 9: mloU per class on S3DIS Area-5.
Methods
miol
ccll.
Blor
wall
heam
col.
Wind.
donr
chair
rable
bonk.
soin
hoard
clor
Potninet
Eff 3D Come Zhann 2:al 2018
$1.8
79.8
93.9
69.0
0.2
28.3
38.5
48.3
71.1
73.6
48.7
59.2
29.3
33.1
RNN Fusion
57.3
92.3
98.2
79.4
0.0
17.6
22.8
62.1
74.4
80.6
31.7
66.7
62.1
56.7
KPConv Thomas 2019
65.4
92.6
97.3
81.4
0.0
16.5
54.5
69.5
90.1
80,2
74.6
66.4
63.7
58.1
MAKF NAS.
67.2
93.6
98.3
81.6
0.0
32.2
51.5
73.2
90.7
82.5
73.3
64.7
71.6
60.0
A.5 HYPERPARAMETER SETTINGS
The implementation of our work is based upon the official PyTorch repository of KPConv (Thomas) et al. 2019). We follow the exact evaluation protocol in this repo to produce our validation results. For training, we demonstrate the detailed hyperparameter settings on 3 different benchmarks (i.c.. ModelNet40 (Wu et al. 2015), SemanticKITTI (Behley et al. 2019) and S3DIS (Armeni et al. 2016 in the following context.
ModelNet40. A single training batch in ModelNer40 contains 16 sub-sampled point clouds. We adopt similar training pipeline following the original KPConv (Thomas et al. 2019) paper yet incorporates SimpleViert View (Goyal al. 2021) RSCNN training protocol for fair comparison. Specif- ically, the SimpleView-RSCCC View-RSCNN protocol adopts random scaling translation for data augmentation, employ cross-entropy loss to optimize, and utilize voting scheme on the best test model to best exploit the potential of searched model. We train our NAS-crafted model for 250 epochs with an initial learning rate of 0.02 and cosine learning rate schedule (Loshchilov & Hutter 2016). To com- bat overfitting, we also adopt a dropout rate of 0.5, a L2 weight decay of 3e-4 and employ early stopping select the best model for testing.
S3DIS. A single training batch in S3DIS contains sub-sampled point clouds. We use a downsample rate of dl0 0.04m following the original KPConv paper (Thomas et al. 2019). Specifically,
15

---

Under review as a conference paper at ICLR 2022
train best model for 250 epochs with initial learning rate of 0.04 and cosine learning rate schedule (Loshchilov & Hutter 2016). To combat overfitting, We also adopt L2 weight decay of 3e-4 and the default data augmentation in KPConv paper.
SemanticKITTL. A single training batch SemanticKITTI contains 10 sub-sampled point clouds. On SemanticKITTI, measure Multiply-Accumulan Accumulates (MACs) over scene with average of ~ 12.3K points, which gives similar MAC count for KPConv architecture as is shown (Tang et al.) downsample rate of dl0 0.06m following the original KPConv paper (Thomas et al. 2019). Specifically, we train our best model for 250 epochs with an initial learning rate of 0.04 and cosine learning rate schedule (Loshchilov & Hutter 2016). To combat overfitiing We also adopt a L2 weight decay of 3e-4 and the default data augmentation in KPConv paper.
2020). We
A.6 OPTIMIZATION CURVE ON DIFFERENT PREDICTORS
In this section, demonstrate that the improvements of the proposed predictors has interference over the improvements in search algorithms. More specifically, our contribution towards lever- aging better neural architecture representation is orthogonal the contribution in introducing better search algorithm. We apply 3 popular search algorithms: Random Search, Regularized Evolu- Proximal Policy Optimization (PPO) (Zoph) & Le 2016; Schulman et al. 2017) on both Wide predictor and Wide & Deep predictor, and plot normalized performance for visualization, see Figure Despite the greater complexity Wide & Deep predictor that may potentially formulate a curvy performance surface, we observe that the superiority EA on Wide predictor is successfully preserved on Wide & Deep predictor. This indi- cates that the improvement of Wide & Deep predictor is consistent with the evaluation of different search algorithm, and sets up a new direction improving the design neural predictors to enhance craseeto-bbased NAS.
tion (Real et al. 2 and Reinforcement-Leamree
1.05
1.05
Normalized Scores
Normalized Scores
0.85
0.85
W-rd
sem-W&D-rd
W-ea-Wea
sem-W&D-ea
sem W-rl
sem W&
0.65
0.65
3
#Samples
1e4
#aamples
1e4
Figure 4: Left: Normalized prediction scores using Wide predictor (donated as W), Right: Normal- ized prediction scores using Wide & Deep predictor (donated as W&D).
A.7 LATENCY ANALYSIS OF MAKPCONV KERNEL
In 4 we notice that our NAS-crafted MAKPCoву model is 24% faster than original KPConv model despite of its higher mIOU. However, we also observe that the latency reduction is less sig- nificant compared to MAC reduction of MAKPConw models, especially compared to voxel-based methods. This is because voxel-based methods (e.g., SPVNAS (Tang al. (0200) benefit from 1) reduced overhead in point-based neighboring mechanisms 2) mature software- hettwar--hrdwware hardware co-design that enables inference with high throughput. To dive deep into this issue, we profile the latency breakdown of the NAS-crafted MAKPCony model and demonstrate the profiling result in Table TO
Here, the preprocessing operations (including radius sampling, grid sub-sampling etc.) consumes 33.7% of the inference cost, neighbor gathering contributes to the memory- bounded operation (i.e., aten :: gather). which takes 10% of the total inference latency. Computation-bounde KPConV such as depthwise convolution (i.e., aten mul and aten sum), MI.Ps (i.e., aten bmm)
memory-bounded
parts in
16

---

Under review as a conference paper at ICLR 2022
Table 10: Operator-level lateney breakdown of searched model. We list the critical operators
here and categorize less important operator 'Other'. In column Type, C denotes computation- bounded operation and M denotes memory-bounded operation.
Operator Preprocessing (CPU) aten :: sub
Type
Latency (ms)
Latency (%)
57.015
33.73
C
23.396
13.84
aten ::
C
20.236
11.97
aten gather
M
18.115
10.72
aten :: mul
C
12.487
7.39
aten :: sum
C
9.887
5.85
aten :: addmm
C
7.292
4.31
aten threshold. aten copy- aten sqrt Other
C
4,444
2.63
M
2.912
1.72
C
2.256
1.33
11.00
6.51
and point-wise local neighborhood movement (i.e., aten :: sub) contribute to ~50% of the total inference cost. Although radius sampling still consumes moderate cost in MAKPConY, there is still a need to leverage design automation algorithm (e.g., our proposed MAKPCony algorithm) to perform optimization on the overall architecture on MAKPCoнv and seek better inference efficiency by improving coapuuttaion-bbuudee operations.
Besides the optimization of architecture, we also discover some potential improvements that har- nesses the hardware-softwar co-design to improve efficiency of МАKPCoпм blocks, such as yet limited to:
Efficient radius sampling implementation. The current implementation lies in CPU and is not optimal for parallelization. possible CUDA version of the radius sampling can speedup the inference speed of MАKPCонч networks.
The current MAKPConv does not utilize fused version of convolution that aggregates the feature outputs on each kernel point efficiently. Instead, the output features on each kernel point is firstly computed via aten bmm and then aggregated via aten :: sum.
The current process of moving points towards their local neighborhoods controlled by alen :: sub is far from efficient and consumes ~ 20% latency. Thus, specialized hard- ware for efficient batch points movement is potentially beneficial for efficient inference point-bauc based networks.
As lower radius leads to more efficient sampling, we consider incorporating the choice of radius for each МAKPConv layer into the design space and harness such option to discover more efficient point-based networks.
As such, with heavy optimization MAKPConv design, complexity in profiling MAKPCony blocks and thus push forward latency-aware search for better hardware specialization. We leave this our future work improve the design efficiency of 3D architectures on processing point cloud data.
reduce the uncertainty and/or
17


Reviews:
[{"review_id": "y91_FC7dRmT", "reviewer": "Reviewer_8cJa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS:\n\n1. This paper is well-written, in general.\n2. The motivation behind MAKPConv is reasonable.\n3. The networks have fewer parameters, and outperform previous works, especially on Semantickitti.\n4. The analysis on NAS for block structure search is enough.\n\nWeakness:\n\n-- I am confused about the introduction to MAKPConv.\n1. What is the connection between MAKPConv and KPConv. In KPConv, the correlation between each neighbour and each kernel point is calculated, which I do not find in MAKPConv.\n2. In NK Attention, the NK Distance indicates the distance between the neighbour and the kernel point or between every two neighbours?\n3. It seems that MAKPConv is an improved version of KPConv, but I cannot associate them in Figure 1. For example, it seems that the NK Attention is similar to the h(.) function in Eq. 5 in KPConv [1]. If so, the Depthwise Kernel in the upper-left part of Figure 1, should be Eq.5, right? Then, which part indicates Eq. 4 in KPConv[1]. Maybe I misunderstand the figure. Please clarify this.\n4. In Eq. 1, the dimension of F is $D_{in}$, while $i\\in[1,...,D_{out}]$, which makes me confused. Does $D_{in}$ equal to $D_{out}$?\n5. In Eq. 1 and Eq. 2, since the points are unordered and we do not know which kernel point corresponds to a certain neighbour, how can we have $F_{\\*,k,\\*} \\cdot W_{k,\\*,\\*}$? In addition, I also cannot find a similar formulation in KPConv.\n6. In Eq. 2, the dimension of $F$ and $W$ is $D_{in}$, so how can we get $F'$ with dimension $D_{out}$? \n\n-- Experimental results\n1. Currently, previous works on point cloud analysis usually conduct experiments on several datasets, such as S3DIS, ShapeNetPart, and ScanNet, while this paper only considers two datasets, including ModelNet40 and SemanticKitti, which cannot demonstrate the effectiveness adequately. \n2. The OA of KPConv is 92.9\\%, which is higher than the OA of the proposed method (92.6\\%), while in this paper the authors do not report that result and instead report the result of their implementation.\n3. On ModelNet40, currently there are many works yielding OA larger than $93\\%$, but the authors do not report.\n\n[1] https://arxiv.org/pdf/1904.08889.pdf\n\n", "summary_of_the_paper": "This paper aims to address the point cloud analysis task by:\n\n1) improving the existing KPConv operation via considering the kernel relationship;\n\n2) designing the networks via a predictor-based NAS approach.\n\nThe improved KPConv operation. i.e., MAKPConv, can model the local structure efficiently, and the searched network has fewer parameters while performing better than the baseline on two datasets, including ModelNet40 for classification and SemanticKitti for segmentation.", "summary_of_the_review": "I have two concerns,\n\n1. The introduction to the proposed MAKPConv is unclear.\n2. The experiments are not enough.\n\nAs a result, currently, I vote for borderline reject. But I might change my score if the authors can address my concern or other reviewers can give more positive reviews.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gnF_GCfbu15", "reviewer": "Reviewer_epb3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The point-based 3D application focused by this work is of importance and has recently attracted much attention as a new research area;\n2. The motivation on computational efficiency makes sense and the experimental result demonstrates that this is well achieved by this work;\n3. The idea of changing KPConv to a depthwise kernel and then using NK-Attention to compensate the degraded performance caused by the limited representative power of the latter is sound, and its effectiveness is verified by Table 1. \n4. The overall performance of the resulted MAKPConv-based network is validated in the experimental study. \n5. The ablation study on the kernel contribution via the NK-Attention is helpful for illustrating the function of the proposed attention mechanism. \n\nWeaknesses:\n1. Although the ideas of using a depthwise kernel and the attention mechanism work well in this paper, they are well-known techniques in the literature. In this sense, the contribution of this work in this part is mainly on innovatively using them to improve the computational efficiency of KPConv. The theoretical contribution of this work seems to be limited. \n2. The part on the proposed wide & deep neural predictor and the idea of carrying feature engineering on searchable features in NAS can be made clearer. Firstly, the key differences and novelties with respect to the existing related work can be further clarified. Secondly, can the proposed feature engineering idea be regarded as a new principal in NAS that can be generally applied? Clarifying these issues will make the contributions in this part easier to be understood. \n\nMinor issues\n1. \"N\" in Figure 1 shall be explicitly defined;\n2. Kenrel --> Kernel in the caption of Figure 1;\n3. \"...my our novel Wide & Deep ...\" in Conclusion. ", "summary_of_the_paper": "This paper addresses point-based methods for classification and semantic segmentation in 3D applications. It aims to improve the computational efficiency of this kind of methods to make it better fit the applications of limited resources such as mobile scenarios. To achieve this, this paper conducts the following works. First, it develops a mobile attention kernel point convolution (MAKPConv) scheme to improve the performance of existing kernel point convolution. Second, it utilizes neural architecture search (NAS) technique to design the MAKPConv-based network. In this part, this paper proposes a wide & deep neural predictor for the NAS process. Experimental study is conducted on benchmark datasets and tasks to demonstrate that the resulted deep networks can achieve higher computational efficiency and improved performance. Ablation study is also conducted to illustrate the key components in this work. ", "summary_of_the_review": "This work is well motivated. The proposed idea on reducing computational cost is neat, and it works effectively as experimentally demonstrated. The proposed wide & deep predictors for NAS helps to efficiently find the high quality architecture for the proposed network, achieving promising performance. Meanwhile, the potential issues are i) the idea of MAKPCon is mainly based on existing well-known techniques and does not seem to bring significant theoretical insight and ii) the novelty and contribution of the Wide & Deep neural predictor part can be better clarified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K0y4sjj2XmC", "reviewer": "Reviewer_Jg8J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Usage of a depthwise kernel reduces the model size of KPConv and the computation cost involved.\n- The paper can effectively search for a MAKPConv block that is better than a hand-crafted one using NAS.\n\nWeaknesses\n- It is unclear why the paper decides to start with KPConv architecture, which is a particularly parameter inefficient architecture. The objective of building a lightweight point cloud processing model would have been better served by starting with some already lightweight architecture like [1] (explained later).\n- Even after elaborate techniques like NAS, the final architecture is not very impressive in terms of memory and accuracy. The paper does not compare to [1], which is a very simple baseline method that already achieves better performance (93.0 vs 92.2) than achieved in the paper while using a similar memory footprint (0.8 vs 0.56 M params). Hence, the end-product of the work is not very useful in the reviewer's opinion.\n- The paper is missing details about the protocol used for the evaluation of the ModelNet40 experiments. This is particularly important as [1] shows smaller changes in protocols can have a drastic impact on performance. \n- The architecture of the proposed model MAKPConv is very complicated. The ablation studies don’t appear to justify the utility of the complicated structures used in the model design.\n\n[1] Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline, ICML 2021\n", "summary_of_the_paper": "The paper builds a memory and computation-efficient version of the KPConv model, which is a point cloud processing model. The paper uses many techniques to achieve this including, depthwise kernels, attention over kernels, and also neural architecture search (NAS). The paper conducts experiments to demonstrate the effectiveness of their techniques.", "summary_of_the_review": "Overall, I appreciate the paper's efforts in making lightweight point cloud processing models. This could be very useful in many applications. However, the method is weak in terms of the final performance (parameters vs accuracy) and is overly complicated. Some important details are also missing (see weakness). Hence, I would recommend rejection for the current version of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E-ImQHhu50", "reviewer": "Reviewer_xh8F", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem studied in this paper is very important. 3D point cloud neural networks are the core of many real-world applications (AR/VR and autonomous driving). At the same time, most existing solutions are still not efficient enough to provide a satisfactory user experience and guarantee user safety.\n\nThe paper is well-written and relatively easy to follow. Furthermore, the proposed solution is technically sound and has achieved fairly good empirical performance on small-scale and large-scale benchmark datasets.\n\nMy primary concern of this paper is its limited technical novelty. This paper is a combination of existing solutions:\n* Depthwise convolution and inverted residual bottleneck are originally proposed in MobileNet-V1/V2.\n* The reweighting mechanism adopted in this paper is essentially the same as in SENet [Hu et al., CVPR 2018].\n* The neural architecture framework used in this paper is almost identical to Neural Predictor [Wen et al., ECCV 2020].\n\nThe only thing new to me is the unified dense and sparse neural architecture representations, which, however, is more of an engineering improvement than a technical contribution. The authors should highlight their core contributions apart from applying existing techniques to a new modality. Besides, the existing solution (SPVNAS) already adopts the one-shot weight-sharing NAS framework to explore the best neural architecture. It is unclear why the authors follow a different approach.\n\nThe authors claim that MAKPConv is 52% faster than SPVNAS, which is inaccurate. Having fewer MACs does not necessarily translate into measured speedup on the actual hardware. The authors should directly compare the latency of MAKPConv with SPVNAS (and other baselines) in Table 4. Furthermore, the techniques proposed in this paper are tailored for #MACs reduction but not necessarily for latency reduction. Point-based neural networks are usually bottlenecked by the KNN computation rather than the convolution (while most techniques in this paper aim at reducing the computation of convolution). It is unclear whether these modifications are also helpful to reduce the latency. The authors should include latency numbers in Table 1.\n\nThe authors have only reported the performance on the validation set of SemanticKITTI. They should also include some numbers on the final testing set. Besides, it would be better to verify whether the searched neural architecture is transferrable to other datasets (such as ScanNet and nuScenes) and/or tasks (such as panoptic segmentation and object detection).\n\nReferences:\n* Hu et al., \"Squeeze-and-Excitation Networks\", CVPR 2018.\n* Wen et al., \"Neural Predictor for Neural Architecture Search\", ECCV 2020.", "summary_of_the_paper": "This paper aims to accelerate the inference of 3D point cloud neural networks. The authors first borrow a few designs from efficient 2D neural networks: depthwise convolution and inverted residual bottleneck. They then propose to reweight the weights of kernel points with attention to boost its representation power. Finally, the authors adopt the predictor-based neural architecture search to explore the best model under a resource constraint. They have evaluated their proposed solution on the small-scale ModelNet40 dataset and the large-scale SemanticKITTI dataset and have achieved reasonably good empirical performance.", "summary_of_the_review": "The paper has relatively weak technical contributions and lacks latency comparisons with state-of-the-art solutions. Therefore, I would like to see the authors' response before the final decision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four reviewers have reviewed this manuscript and two found it borderline leaning towards acceptance, two other reviewers scored it below the acceptance threshold. While the authors *identify the key challenges and bottlenecks in 3D point cloud model*, the most positive two reviewers notice that the depthwise kernel and the attention mechanism (and similar tools) are well-known in the literature and that this work is *more of an engineering improvement than a technical contribution*, which erodes the novelty of the proposed idea on that front. While authors noticed some discrepancies in the numbers quoted by Rev. 3, the model gains are also nonetheless modest compared to other models. Overall, the feeling amongst the reviewers was that  the presentation of NK attention could be further improved and that the paper uses very heavy machinery to achieve results comparable with SOTA.

On this occasion, the manuscript is below the acceptance threshold with even the borderline positive reviewers having their doubts about clear cut technical novelty.
Decision: Reject