Automated Mobile Attention KPConv Networks via A Wide & Deep Predictor

1 INTRODUCTION . 3D applications such as object recognition ( Wu et al. , 2015 ) , shape segmentation ( Yu et al. , 2004 ) , and indoor-scene segmentation ( Armeni et al. , 2016 ) are growing popularity thanks to the rise of 3D acquisition technologies and the increasing amount of geometric data collected from 3D sensors . Deep learning models are becoming one of the most common and promising ways to represent 3D data , especially 3D point cloud data , thanks to its high performance ( Zhou et al. , 2020 ; Zhao et al. , 2020 ) and good efficiency ( Cortinhal et al. , 2020 ; Hu et al. , 2020a ) . Starting from PointNet ( Qi et al. , 2017a ) , point-based methods are becoming one of the most popular choices to extract features and make predictions for 3D input point clouds . Such an approach typically learns an individual representation for each data point and aggregates different representations to formulate a signature for all global points , usually via end-to-end point convolutions ( Xu et al. , 2018 ; Atzmon et al. , 2018 ) . Kernel point convolution ( KPConv ) ( Thomas et al. , 2019 ) is the first work that maps the input points to the influence of kernel points via linear correlation , and then further convolves the mapped features over the kernel points to compute weighted features . Given a point yi , the kernel function g over neighbor points yi in KPConv carries g ( yi ) =∑ k < K h ( yi , x̃k ) @ Wk to learn representations via K kernel points , with the neighbor-kernel correlation h ( yi , xk ) . However , KPConv results in very large model sizes for 3D point cloud recognition due to inefficient local feature aggregation via point convolution over kernel weights Wk . As a result , KPConv takes up to ∼200G Multiply-Accumulates ( MACs ) to perform inference on a single scene at the cost of ∼200ms on a TITAN XP GPU . Such resource consumption and inference cost are too high for real-time execution in mobile scenarios . In addition , the above Neighbor-Kernel cor- relation function has a strong hidden assumption : the contribution of all kernel points are measured equally towards the neighbor points . As a result of ignoring cross-kernel information from other kernel points to accommodate the joint evolution of different kernel points , such an assumption fails to enhance representation learning on important kernel points and thus yields sub-optimal results . In this paper , we propose a novel 3D attention module , named Mobile Attention Kernel Point Convolution ( MAKPConv ) , to address the above issues . MAKPConv employs two key design elements to improve the performance and efficiency . First , we propose a depthwise kernel to split feature extraction on point-wise features from channel-wise features during point convolution . Such design allows significant savings in both parameter count and amount of computation , without performance degradation . Second , we introduce Neighbor-Kernel attention ( NK attention ) to enable the kernelwise information exchange between different kernel points via learnable attention networks . The joint evolution of all kernel points encourages representation learning on more important kernel points while suppressing less important ones , thus benefits from a better learning adaptability of kernel point contributions towards neighbor point distributions in KPConv networks . Inspired by Sandler et al . ( 2018 ) , we capitalize Inverted Residual Bottleneck ( IRB ) to encapsulate MAKPConv module into MAKPConv block , and craft MAKPConv design space to explore a wider range of choices in 3D models . As 3D architectures are usually high-dimensional , unstructured and unordered , tailoring a good neural architecture to address such complex point cloud data requires a delicate and precise evaluation protocol . This is difficult to achieve with weight-sharing oneshot NAS ( e.g. , SPVNAS ( Tang et al. , 2020 ) ) because co-adaptation between sub-networks may degrade the quality of evaluation and lead to sub-optimal results . Thus , we pioneer the usage of predictor-based Neural Architecture Search ( NAS ) ( Wen et al. , 2020 ) that utilizes a sample-based mechanism to reduce evaluation noise and seek better models on 3D point cloud . Specifically , we train a performance predictor based on exploitable samples , and probe the whole MAKPConv design space to obtain the best NAS model based on predictive performance . We observe that existing predictor-based methods adopt a fixed scheme ( e.g. , one-hot encoding ) to encode different types of features , leading to sub-optimal neural architecture representations with inaccurate priors . This is because real-valued features ( e.g. , block width ) and categorical features ( e.g. , optional NK attention ) have different priors and thus they should not be encoded in the same way . To address this issue , we formulate dense representations for features that have quasi-linear priors on model performance ( e.g. , depth , width ) , and formulate sparse representation for features that have nonlinear sophisticated priors on model performance ( e.g. , optional attention ) . Intuited by Cheng et al . ( 2016 ) , we devise a Wide & Deep predictor to interact dense and sparse representations such that the combination of memorization and generalization reduces the prediction error . By utilizing the Wide & Deep predictor to perform NAS under the MAKPConv design space , our results on various benchmarks show promising results of the automatically discovered models in terms of both performance and efficiency . In particular , our MAKPConv networks outperform the original KPConv on both classification and semantic segmentation benchmarks with 92 % ∼93 % fewer parameters , thanks to the efficient design of MAKPConv block . Empowered by our Wide & Deep predictor driven NAS , the NAS-crafted networks outperform the state-of-the-art NAS work , SPVNAS , by ∼1 mIOU with 83 % smaller size and 78 % fewer MACs . We highlight the contribution of this paper as follows : • We propose Mobile Attention Kernel Point Convolution ( MAKPConv ) module that combines depthwise kernel with Neighbor-Kernel attention to achieve efficient and highperforming architectures dedicated to learning on 3D point clouds . • We identify the importance of carrying feature engineering on neural architecture representations to improve predictor-based NAS . Specifically , we encode different searchable features with dense and/or sparse representations given their priors , and utilize a Wide & Deep neural predictor to interact these representations for better accuracy prediction . 2 RELATED WORK . 3D Point Convolution Networks . Point convolution networks ( Xu et al. , 2018 ; Hu et al. , 2020a ; Lang et al. , 2020 ) extends convolutional operations towards irregular point set grids to benefit from learnable radial function and spherical harmonics in 3D point clouds . Later work ( Thomas et al. , 2019 ) performs point convolution over radius neighbors and aggregates convolved results to produce learned representation without additional intermediate operators . Recent work on point convolutions improves point sampling efficiency ( Hu et al. , 2020a ; Lang et al. , 2020 ) , local feature aggregations ( Zhao et al. , 2020 ; Lin et al. , 2020 ) , manual architecture fabrications ( Hu et al. , 2020b ; Lei et al. , 2020a ) , and/or improved training protocols ( Goyal et al. , 2021 ) to seek higher model performance and efficiency . However , these works neither simplify the structure of point networks to improve model efficiency , nor specify an accurate neighbor-kernel correlation . Our work extends kernel point convolution ( Thomas et al. , 2019 ) and addresses the above caveats : we improve the efficiency of local feature aggregation via a depthwise kernel on point convolution , and further introduce Neighbor-Kernel attention to enhance the representation power of a depthwise kernel . Predictor-based Neural Architecture Search . Predictor-based NAS ( Wen et al. , 2020 ; Dudziak et al. , 2020 ) is a popularized method that trains a performance predictor on limited exploitable samples . The trained predictor serves as a surrogate model of the ground-truth performance and is utilized to guide architecture search on the full design space . Existing achievements on predictorbased NAS lies in improving sample efficiency ( Dudziak et al. , 2020 ) and augmenting architecture samples ( Liu et al. , 2021 ) , yet they overlook the importance of carrying feature engineering on neural architecture features . Our work identifies the importance of improving neural architecture representation , and assigns different priors to a wide set of searchable features and yield dense representations and sparse representations . Our work further designs a Wide & Deep predictor to leverage and interact these representations for better prediction quality on unseen architectures . 3 MOBILE ATTENTION KERNEL POINT CONVOLUTION . We start with the formulation of KPConv ( Thomas et al. , 2019 ) . Given a center point X ∈ R3 and its corresponding features F̂ ∈ RDin , a kernel point convolution ( KPConv ) function carries all N neighbor points Xn = { x′ : ||x′ − x|| < r } ∈ RN×3 and their features F̄ ∈ RN×Din within radius r over a set of K kernel points X̃ ∈ RK×3 and accompanying weights Wk . Specifically , a correlation function : h : RN×Din → RK×Din maps the unordered neighbor point features F̄ to a set of features F on kernel points . Usually , the correlation function is represented as H ∈ RN×K and applied via F = HF̄ . The correlation function is followed by a kernel g : RK×Din → RDout that convolves these features to obtain the aggregated output F ′ . Here , Din denotes the dimension of input , and Dout denotes the dimension of output . The original KPConv overlooks an important opportunity to improve the efficiency by reducing the redundancy of kernel g that is due to the simultaneously performed point-wise and channel-wise feature extractions . In addition , the original KPConv adopts a linear function as correlation : it treats the impact of each kernel point equally and ignores the adaptation of neighbor points towards kernel dispositions . For example , the center kernel point usually attracts more centralized neighbor points with a higher density , and edge kernel points perform critical jobs in recognizing boundary neighbor points with a lower density in the radius sampling neighborhood . Thus , the original KPConv may have sub-optimal performance and efficiency on 3D applications . To overcome the above limitations , we propose Mobile Attention Kernel Point Convolution ( MAKPConv ) module as an in-place improvement of KPConv block to seek higher quality and efficiency . Figure 1 provides an overview of MAKPConv . Specifically , we design a depthwise kernel to address the computation cost : A depthwise kernel ĝ carries element-wise multiplication in place of matrix multiplication between aggregated features and kernel weights , thus achieves memory and computation reduction . Considering a depthwise kernel has limited representation power , we introduce Neighbor-Kernel attention ( NK attention ) as an improved correlation function ĥ to emphasize representation learning on important depthwise kernels and fully utilize the capacity of kernel points . This compensates for the reduced representation power : the joint evolution of kernel points via learnable attention encourages effective representation learning on important depthwise kernels and best exploit their potential . As a result , the fusion of depthwise kernel and NK attention into MAKPConv block has the potential to achieve high-performing yet efficient 3D models . 3.1 DEPTHWISE KERNEL . In original kernel function of KPConv , a regular kernel takes an input F ∈ RK×Din over K kernel points and uses a weight kernel W ∈ RK×Din×Dout to simultaneously perform point-wise transformation and channel-wise transformation . A regular kernel g produces an output F ′ ∈ RDout : F ′d = g ( F , W ) = ∑ k∈K ∑ d∈Din Fk , i ·Wk , i , d , ( 1 ) where k is a kernel index , and d is a channel index . A regular kernel leads to K × Din × Dout parameters andDin×Dout×K MACs during inference , and leads to large computation and memory for the deep layers of the 3D architectures . To favor model efficiency , we design a depthwise kernel ĝ that only performs point-wise feature extractions over input point clouds , see Figure 1 ( a ) . The proposed depthwise kernel adopts learnable weightsW ∈ RK×Din , extracts point-wise information on each channel , and obtains an output F ′ ∈ RDout as follows : F ′d = ĝ ( F , W ) = ∑ k∈K Fk , d ·Wk , d , ( 2 ) where k is a kernel index , and d is a channel index . Here , Dout = Din = D as a depthwise kernel does not operate on channel-wise features . As a result , a depthwise kernel point convolution only leads to K ×Din parameters and only Din ×K MACs during inference , and reduces Dout× parameters and MACs during convolution compared to a regular kernel . The parameter saving from a depthwise kernel is significant when Dout is large , especially in deeper layers of the network .

Summary: This paper aims to accelerate the inference of 3D point cloud neural networks. The authors first borrow a few designs from efficient 2D neural networks: depthwise convolution and inverted residual bottleneck. They then propose to reweight the weights of kernel points with attention to boost its representation power. Finally, the authors adopt the predictor-based neural architecture search to explore the best model under a resource constraint. They have evaluated their proposed solution on the small-scale ModelNet40 dataset and the large-scale SemanticKITTI dataset and have achieved reasonably good empirical performance.