Automated Mobile Attention KPConv Networks via A Wide & Deep Predictor

1 INTRODUCTION . 3D applications such as object recognition ( Wu et al. , 2015 ) , shape segmentation ( Yu et al. , 2004 ) , and indoor-scene segmentation ( Armeni et al. , 2016 ) are growing popularity thanks to the rise of 3D acquisition technologies and the increasing amount of geometric data collected from 3D sensors . Deep learning models are becoming one of the most common and promising ways to represent 3D data , especially 3D point cloud data , thanks to its high performance ( Zhou et al. , 2020 ; Zhao et al. , 2020 ) and good efficiency ( Cortinhal et al. , 2020 ; Hu et al. , 2020a ) . Starting from PointNet ( Qi et al. , 2017a ) , point-based methods are becoming one of the most popular choices to extract features and make predictions for 3D input point clouds . Such an approach typically learns an individual representation for each data point and aggregates different representations to formulate a signature for all global points , usually via end-to-end point convolutions ( Xu et al. , 2018 ; Atzmon et al. , 2018 ) . Kernel point convolution ( KPConv ) ( Thomas et al. , 2019 ) is the first work that maps the input points to the influence of kernel points via linear correlation , and then further convolves the mapped features over the kernel points to compute weighted features . Given a point yi , the kernel function g over neighbor points yi in KPConv carries g ( yi ) =∑ k < K h ( yi , x̃k ) @ Wk to learn representations via K kernel points , with the neighbor-kernel correlation h ( yi , xk ) . However , KPConv results in very large model sizes for 3D point cloud recognition due to inefficient local feature aggregation via point convolution over kernel weights Wk . As a result , KPConv takes up to ∼200G Multiply-Accumulates ( MACs ) to perform inference on a single scene at the cost of ∼200ms on a TITAN XP GPU . Such resource consumption and inference cost are too high for real-time execution in mobile scenarios . In addition , the above Neighbor-Kernel cor- relation function has a strong hidden assumption : the contribution of all kernel points are measured equally towards the neighbor points . As a result of ignoring cross-kernel information from other kernel points to accommodate the joint evolution of different kernel points , such an assumption fails to enhance representation learning on important kernel points and thus yields sub-optimal results . In this paper , we propose a novel 3D attention module , named Mobile Attention Kernel Point Convolution ( MAKPConv ) , to address the above issues . MAKPConv employs two key design elements to improve the performance and efficiency . First , we propose a depthwise kernel to split feature extraction on point-wise features from channel-wise features during point convolution . Such design allows significant savings in both parameter count and amount of computation , without performance degradation . Second , we introduce Neighbor-Kernel attention ( NK attention ) to enable the kernelwise information exchange between different kernel points via learnable attention networks . The joint evolution of all kernel points encourages representation learning on more important kernel points while suppressing less important ones , thus benefits from a better learning adaptability of kernel point contributions towards neighbor point distributions in KPConv networks . Inspired by Sandler et al . ( 2018 ) , we capitalize Inverted Residual Bottleneck ( IRB ) to encapsulate MAKPConv module into MAKPConv block , and craft MAKPConv design space to explore a wider range of choices in 3D models . As 3D architectures are usually high-dimensional , unstructured and unordered , tailoring a good neural architecture to address such complex point cloud data requires a delicate and precise evaluation protocol . This is difficult to achieve with weight-sharing oneshot NAS ( e.g. , SPVNAS ( Tang et al. , 2020 ) ) because co-adaptation between sub-networks may degrade the quality of evaluation and lead to sub-optimal results . Thus , we pioneer the usage of predictor-based Neural Architecture Search ( NAS ) ( Wen et al. , 2020 ) that utilizes a sample-based mechanism to reduce evaluation noise and seek better models on 3D point cloud . Specifically , we train a performance predictor based on exploitable samples , and probe the whole MAKPConv design space to obtain the best NAS model based on predictive performance . We observe that existing predictor-based methods adopt a fixed scheme ( e.g. , one-hot encoding ) to encode different types of features , leading to sub-optimal neural architecture representations with inaccurate priors . This is because real-valued features ( e.g. , block width ) and categorical features ( e.g. , optional NK attention ) have different priors and thus they should not be encoded in the same way . To address this issue , we formulate dense representations for features that have quasi-linear priors on model performance ( e.g. , depth , width ) , and formulate sparse representation for features that have nonlinear sophisticated priors on model performance ( e.g. , optional attention ) . Intuited by Cheng et al . ( 2016 ) , we devise a Wide & Deep predictor to interact dense and sparse representations such that the combination of memorization and generalization reduces the prediction error . By utilizing the Wide & Deep predictor to perform NAS under the MAKPConv design space , our results on various benchmarks show promising results of the automatically discovered models in terms of both performance and efficiency . In particular , our MAKPConv networks outperform the original KPConv on both classification and semantic segmentation benchmarks with 92 % ∼93 % fewer parameters , thanks to the efficient design of MAKPConv block . Empowered by our Wide & Deep predictor driven NAS , the NAS-crafted networks outperform the state-of-the-art NAS work , SPVNAS , by ∼1 mIOU with 83 % smaller size and 78 % fewer MACs . We highlight the contribution of this paper as follows : • We propose Mobile Attention Kernel Point Convolution ( MAKPConv ) module that combines depthwise kernel with Neighbor-Kernel attention to achieve efficient and highperforming architectures dedicated to learning on 3D point clouds . • We identify the importance of carrying feature engineering on neural architecture representations to improve predictor-based NAS . Specifically , we encode different searchable features with dense and/or sparse representations given their priors , and utilize a Wide & Deep neural predictor to interact these representations for better accuracy prediction . 2 RELATED WORK . 3D Point Convolution Networks . Point convolution networks ( Xu et al. , 2018 ; Hu et al. , 2020a ; Lang et al. , 2020 ) extends convolutional operations towards irregular point set grids to benefit from learnable radial function and spherical harmonics in 3D point clouds . Later work ( Thomas et al. , 2019 ) performs point convolution over radius neighbors and aggregates convolved results to produce learned representation without additional intermediate operators . Recent work on point convolutions improves point sampling efficiency ( Hu et al. , 2020a ; Lang et al. , 2020 ) , local feature aggregations ( Zhao et al. , 2020 ; Lin et al. , 2020 ) , manual architecture fabrications ( Hu et al. , 2020b ; Lei et al. , 2020a ) , and/or improved training protocols ( Goyal et al. , 2021 ) to seek higher model performance and efficiency . However , these works neither simplify the structure of point networks to improve model efficiency , nor specify an accurate neighbor-kernel correlation . Our work extends kernel point convolution ( Thomas et al. , 2019 ) and addresses the above caveats : we improve the efficiency of local feature aggregation via a depthwise kernel on point convolution , and further introduce Neighbor-Kernel attention to enhance the representation power of a depthwise kernel . Predictor-based Neural Architecture Search . Predictor-based NAS ( Wen et al. , 2020 ; Dudziak et al. , 2020 ) is a popularized method that trains a performance predictor on limited exploitable samples . The trained predictor serves as a surrogate model of the ground-truth performance and is utilized to guide architecture search on the full design space . Existing achievements on predictorbased NAS lies in improving sample efficiency ( Dudziak et al. , 2020 ) and augmenting architecture samples ( Liu et al. , 2021 ) , yet they overlook the importance of carrying feature engineering on neural architecture features . Our work identifies the importance of improving neural architecture representation , and assigns different priors to a wide set of searchable features and yield dense representations and sparse representations . Our work further designs a Wide & Deep predictor to leverage and interact these representations for better prediction quality on unseen architectures . 3 MOBILE ATTENTION KERNEL POINT CONVOLUTION . We start with the formulation of KPConv ( Thomas et al. , 2019 ) . Given a center point X ∈ R3 and its corresponding features F̂ ∈ RDin , a kernel point convolution ( KPConv ) function carries all N neighbor points Xn = { x′ : ||x′ − x|| < r } ∈ RN×3 and their features F̄ ∈ RN×Din within radius r over a set of K kernel points X̃ ∈ RK×3 and accompanying weights Wk . Specifically , a correlation function : h : RN×Din → RK×Din maps the unordered neighbor point features F̄ to a set of features F on kernel points . Usually , the correlation function is represented as H ∈ RN×K and applied via F = HF̄ . The correlation function is followed by a kernel g : RK×Din → RDout that convolves these features to obtain the aggregated output F ′ . Here , Din denotes the dimension of input , and Dout denotes the dimension of output . The original KPConv overlooks an important opportunity to improve the efficiency by reducing the redundancy of kernel g that is due to the simultaneously performed point-wise and channel-wise feature extractions . In addition , the original KPConv adopts a linear function as correlation : it treats the impact of each kernel point equally and ignores the adaptation of neighbor points towards kernel dispositions . For example , the center kernel point usually attracts more centralized neighbor points with a higher density , and edge kernel points perform critical jobs in recognizing boundary neighbor points with a lower density in the radius sampling neighborhood . Thus , the original KPConv may have sub-optimal performance and efficiency on 3D applications . To overcome the above limitations , we propose Mobile Attention Kernel Point Convolution ( MAKPConv ) module as an in-place improvement of KPConv block to seek higher quality and efficiency . Figure 1 provides an overview of MAKPConv . Specifically , we design a depthwise kernel to address the computation cost : A depthwise kernel ĝ carries element-wise multiplication in place of matrix multiplication between aggregated features and kernel weights , thus achieves memory and computation reduction . Considering a depthwise kernel has limited representation power , we introduce Neighbor-Kernel attention ( NK attention ) as an improved correlation function ĥ to emphasize representation learning on important depthwise kernels and fully utilize the capacity of kernel points . This compensates for the reduced representation power : the joint evolution of kernel points via learnable attention encourages effective representation learning on important depthwise kernels and best exploit their potential . As a result , the fusion of depthwise kernel and NK attention into MAKPConv block has the potential to achieve high-performing yet efficient 3D models . 3.1 DEPTHWISE KERNEL . In original kernel function of KPConv , a regular kernel takes an input F ∈ RK×Din over K kernel points and uses a weight kernel W ∈ RK×Din×Dout to simultaneously perform point-wise transformation and channel-wise transformation . A regular kernel g produces an output F ′ ∈ RDout : F ′d = g ( F , W ) = ∑ k∈K ∑ d∈Din Fk , i ·Wk , i , d , ( 1 ) where k is a kernel index , and d is a channel index . A regular kernel leads to K × Din × Dout parameters andDin×Dout×K MACs during inference , and leads to large computation and memory for the deep layers of the 3D architectures . To favor model efficiency , we design a depthwise kernel ĝ that only performs point-wise feature extractions over input point clouds , see Figure 1 ( a ) . The proposed depthwise kernel adopts learnable weightsW ∈ RK×Din , extracts point-wise information on each channel , and obtains an output F ′ ∈ RDout as follows : F ′d = ĝ ( F , W ) = ∑ k∈K Fk , d ·Wk , d , ( 2 ) where k is a kernel index , and d is a channel index . Here , Dout = Din = D as a depthwise kernel does not operate on channel-wise features . As a result , a depthwise kernel point convolution only leads to K ×Din parameters and only Din ×K MACs during inference , and reduces Dout× parameters and MACs during convolution compared to a regular kernel . The parameter saving from a depthwise kernel is significant when Dout is large , especially in deeper layers of the network .

Summary: This paper aims to address the point cloud analysis task by:  1) improving the existing KPConv operation via considering the kernel relationship;  2) designing the networks via a predictor-based NAS approach.  The improved KPConv operation. i.e., MAKPConv, can model the local structure efficiently, and the searched network has fewer parameters while performing better than the baseline on two datasets, including ModelNet40 for classification and SemanticKitti for segmentation.