Automated Mobile Attention KPConv Networks via A Wide & Deep Predictor

Abstract: Kernel Point Convolution (KPConv) achieves cutting-edge performance on 3D point cloud applications. Unfortunately, the large size of KPConv network limits its usage in mobile scenarios. In addition, we observe that KPConv ignores the kernel relationship and treats each kernel point equally when formulating neighbor-kernel correlation via Euclidean distance. This leads to a weak representation power. To mitigate the above issues, we propose a module named Mobile Attention Kernel Point Convolution (MAKPConv) to improve the efficiency and quality of KPConv. MAKPConv employs a depthwise kernel to reduce resource consumption and re-calibrates the contribution of kernel points towards each neighbor point via Neighbor-Kernel attention to improve representation power. Furthermore, we capitalize Inverted Residual Bottleneck (IRB) to craft a design space and employ a predictor-based Neural Architecture Search (NAS) approach to automate the design of efficient 3D networks based on MAKPConv. To fully exploit the immense design space via an accurate predictor, we identify the importance of carrying feature engineering on searchable features to improve neural architecture representations and propose a Wide & Deep Predictor to unify dense and sparse neural architecture representations for lower error in performance prediction. Experimental evaluations show that our NAS-crafted MAKPConv network uses 96% fewer parameters on 3D point cloud classification and segmentation benchmarks with better performance. Compared with state-of-the-art NAS-crafted model SPVNAS, our NAS-crafted MAKPConv network achieves ~1% better mIOU with 83% fewer parameters and 52% fewer Multiply-Accumulates.

Reviews and author responses:

 The authors would like to clarify that **we have absolutely no means of any intimidation**. If anything makes you misunderstand our intention, we apologize. 
The authors would like to clarify the key misunderstandings that may have caused the extensive discussions here. 

- For comparison results with SimpleView:
In Table 2 of SimpleView, the SimpleView architecture claims to achieve on-par OA (93.2% vs. 93.3% on RSCNN vote column) or better OA (92.9\% vs. 92.7% on RSCNN no-vote column) OA than prior architectures. Following the similar logic, the authors believe it is fair to claim that the proposed architecture achieves on-par OA (93.1% vs. 93.2%) with 30% fewer parameters and better OA (93.4% vs. 93.2%) than SimpleView. In both ways, the authors believe it is fair to claim the proposed model is better than prior works such as SimpleView, in both parameter-efficiency and OA.
- For the reviewer’s comment "The paper can improve its final architecture by starting with better initialization for NAS." 
This means that the authors have to switch our NAS on point-based 3D architectures to projection-based 3D architectures, which may lead to better or worse results. If there is a strong direct evidence (e.g., published work) to back up the comment, we are happy to make such a switch. However, if it is merely based on an unverified hypothesis, the authors do not feel comfortable to be rejected on such hypothesis as we have already improved over the state-of-the-art (in both parameter-efficiency and performance). Meanwhile, the authors appreciate the reviewer’s suggestion on this, and we will consider this as a potential future work.
- More importantly, the SimpleView protocol is difficult to be applied to semantic segmentation tasks, which is one of the main target applications we consider in this work. As a comparison, the proposed NAS can achieve promising results, in both performance and efficiency. Thus, adopting the SimpleView protocol may also affect the scope of our work.

Finally, the authors would like to thank the reviewer again for pointing out the SimpleView work, an improved training protocol to improve ModelNet40 results. Yet, such protocol is not directly related to the core technical and empirical contribution of the paper. Again, the authors respect whatever decision the reviewers made to the final paper, yet the current justification for rejection is mainly based on a remotely relevant baseline (despite that our approach still outperforms it) and an unverified hypothesis, which is not very convincing.


 I would like to summarize my response as following:

- The authors consider 93.3 +- 0.2 to be statistically better than 93.2 +- 0.2. However, the reviewer does not consider is so by means of any scientifically accepted measure. If the authors feel otherwise, please mention the exact test of statistical significance with the p value.

- NAS is not necessary comment was made with regards to the above empirical evidence. The paper can improve its final architecture by starting with better initialization for NAS. This was the **first** comment made by the reviewer in the original review. 

- It is important to compare to projection-based baseline as they are an alternative to what is proposed. It is unclear why to use heavy machinery like NAS when the proposed model is not better than simple and existing baselines. 

Lastly, I do not appreciate the authors taking the comments of the reviewer out of context (as explained earlier). Clearly, the suggestions by the reviewer has helped the author to improve their performance. I would like to flag a potential violation of code of conduct to the Program Committee for the use of **intimidation** against other participants. 

 Thanks for the reply. The authors would like to point out the following:

- Compared with SimpleView, our NAS-crafted model is always more parameter-efficient: NAS-crafted models with 93.3\%$\pm$0.2\% is statistically better than the SimpleView model with 93.2\%$\pm$0.2\% by 0.5 **standard deviation**.

- Though the reviewer envisions that SimpleView might be designed with parameter-efficient structures, this is an unverified hypothesis that should not be used against a submission. More importantly, such design change usually results in a performance loss. For example, our experiments in hand-crafted MAKPConv observe a 0.3\% OA loss compared to the parameter-inefficient baselines. Thus, NAS is a promising approach to improve efficiency without sacrificing performance.

Based on the above, it is unclear why the reviewer challenges the contribution and motivation of applying NAS on the 3D point cloud. 

In addition, since the reviewer insists that SimpleView is a necessary baseline to compare, the authors would like the reviewer to clarify the following since the authors believe SimpleView is orthogonal to our work in some sense:

- SimpleView proposes a projection-based architecture for efficient 3D point cloud classification. Why it is necessary to compare a point-based 3D architecture with a projection-based 3D architecture to justify **a significant improvement**? In addition, to the best of the authors' understanding, the main focus of SimpleView lies in creating an extensive and fair training protocol to ease the evaluation of 3D point cloud applications, rather than architectures.

- So far, SimpleView has not been applied to establish baselines on more complicated tasks such as semantic segmentation (e.g., S3DIS, SemanticKITTI), yet our proposed architecture yields promising results on segmentation tasks. Thus, **why is it reasonable to negate the improvements of our proposed model on segmentation tasks** because SimpleView has not established such a baseline?

In addition, the authors are a bit disappointed that despite receiving comments like 'NAS is not necessary", the reviewer does not provide any suggestions on improving the technical value of this paper. The authors are a bit concerned about a potential violation of the reviewer guidelines (https://iclr.cc/Conferences/2022/ReviewerGuide):

*6. Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.*

The authors hope that the reviewer can address the above concerns. The authors respect whatever decision the reviewers made to the final paper, yet please make it convincing and provide constructive feedback to honor the reviewer guidelines in the ICLR community.


 I very much appreciate the authors’ sincere response, however, I do not think the argument is convincing. Hence, I would summarize my final opinion.

Never in my review did I say that NAS does not improve performance over hand-crafted model proposed by the authors (MAKPConv). My comparison was always with already proposed baselines (SimpleView) and for that I do not see the model with NAS being (statistically) better (93.3 +- 0.2 vs 93.2 +- 0.2). Hence using compute intensive NAS is not necessary and not a contribution in my opinion. The reduces the overall usefulness of the paper. 




 Thank you for your comments! We will try to better clarify and highlight the contribution of the feature engineering part in the later revision of our paper, and address the remaining issues with the best of our effort.

 Thanks for providing the feedback. We report 0.2\% OA variance on ModelNet40. We clarify and address the concerns as follows:
1. ModelNet40 Performance improvements **near 93\% OA level** in existing literature are quite limited: **even $\sim$0.5% OA is considered a significant improvement**, such as the results shown in Table 2 of SimpleView [1] and Table 1 of KPConv [2]. Our proposed method justifies the usage of Neural Architecture Search: **compared to the hand-crafted version of MAKPConv with 92.6\% OA, the NAS-crafted MAKPConv achieves 93.4\% OA with similar parameters (i.e., 1.21M).** This justifies that under the same network design space (i.e., MAKPConv design space) and same evaluation protocol (i.e., SimpleView-RSCNN protocol), NAS crafts models with significantly better performance (i.e., 0.8\% OA). In addition, performance evaluation on SemanticKITTI and S3DIS demonstrates a consistent and significant improvement in performance and efficiency thanks to the proposed NAS methodology.

2. Utilizing a parameter-efficient structure to construct models (e.g., SimpleView [1] and KPConv [2]) **may not necessarily preserve OA**. **For example, our hand-crafted MAKPConv (i.e., without NAS design automation) achieves 0.3\% lower OA than the original version of KPConv on ModelNet40 (See Table 3), despite a direct application of parameter-efficient MAKPConv module proposed in the paper.** This further strengthens the need of utilizing NAS to organize these parameter-efficient modules and yields a high-performing yet efficient architecture.

As such, it is necessary to utilize NAS to automate the design of 3D point-cloud models (e.g., point-based MAKPConv architectures) for better performance and efficiency. We agree that SimpleView is an efficient projection-based 3D point-cloud method, and appreciate the idea that incorporating NAS in optimizing efficientnet-like 3D architectures may yield better projection-based 3D models.

Reference

[1] Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline, ICML 2021.

[2] KPConv: Flexible and Deformable Convolution for Point Clouds, CVPR 2019.


 Thanks for the correcting the mistake. Yes, SimpleView achieves 93.2 vs 93.0 as achieved by the proposed method for 0.8 vs 0.58 M parameters. Further, the author say that for similar number of parameters, the proposed model achieves 93.3. It would be great to see the variance for the  performance of the proposed models.

But unfortunately, I am not convinced that a the performance improvement is significant to justify the use of NAS. This is because, as far the reviewer can understand
 - SimpleView is a simple model using only ResNet18/4 backbone. So if one uses compute intensive operation like NAS, one must demonstrate significant performance improvement to justify it. The current performance does not demonstrate any significant improvement.
- SimpleView was not designed to be parameter efficient and still is on par with the especially designed proposed model. One can simply replace ResNet18/4 with parameter efficient architectures like efficient-net and achieve better parameter efficiency.  

Hence, I see no reason to change my decision. I very much appreciate the authors willingness to clarify the concerns.

 We would like to clarify a misunderstanding of performance comparison. In SimpleView, Table 2 shows that SimpleView achieves 93.2% OA with 0.8M parameters (RSCNN protocol with vote) instead of the 93.6% OA with 0.8M stated by the reviewer. In comparison, our result is 93.1% with 0.56M parameters, which is on-par with SimpleView with 30% fewer parameters. A scaled version achieves 93.4% accuracy which is better than SimpleView. We also scaled up our model by 1.4$\times$ to be more close in size with SimpleView and achieve 93.3% OA with 0.74M parameters. With the usage of DGCNN protocol, we expect further performance gain as the use of smoothed cross-entropy loss is beneficial for the training process and demonstrates performance improvement from 93.2% to 93.6% in SimpleView.
Therefore, with a fair comparison of SimpleView, our approach is better in both accuracy and model size under RSCNN protocol. Hope this clarification addresses your concern. We look forward to your reply and updated review. Thanks again for your comments.

Reference 

[1] Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline, ICML 2021

The paper builds a memory and computation-efficient version of the KPConv model, which is a point cloud processing model. The paper uses many techniques to achieve this including, depthwise kernels, attention over kernels, and also neural architecture search (NAS). The paper conducts experiments to demonstrate the effectiveness of their techniques. Strengths
- Usage of a depthwise kernel reduces the model size of KPConv and the computation cost involved.
- The paper can effectively search for a MAKPConv block that is better than a hand-crafted one using NAS.

Weaknesses
- It is unclear why the paper decides to start with KPConv architecture, which is a particularly parameter inefficient architecture. The objective of building a lightweight point cloud processing model would have been better served by starting with some already lightweight architecture like [1] (explained later).
- Even after elaborate techniques like NAS, the final architecture is not very impressive in terms of memory and accuracy. The paper does not compare to [1], which is a very simple baseline method that already achieves better performance (93.0 vs 92.2) than achieved in the paper while using a similar memory footprint (0.8 vs 0.56 M params). Hence, the end-product of the work is not very useful in the reviewer's opinion.
- The paper is missing details about the protocol used for the evaluation of the ModelNet40 experiments. This is particularly important as [1] shows smaller changes in protocols can have a drastic impact on performance. 
- The architecture of the proposed model MAKPConv is very complicated. The ablation studies don’t appear to justify the utility of the complicated structures used in the model design.

[1] Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline, ICML 2021
 Overall, I appreciate the paper's efforts in making lightweight point cloud processing models. This could be very useful in many applications. However, the method is weak in terms of the final performance (parameters vs accuracy) and is overly complicated. Some important details are also missing (see weakness). Hence, I would recommend rejection for the current version of the paper.

 I would like to thank the authors for their response. I also went over the other reviews. Unfortunately, I would still recommend a reject for the current paper.

In their response, the author say that they start with KPConv because it offers diverse set of choices. This could be a valid reason only if the final architecture is better than manually designed ones. However, the proposed model is worse than (or not better) the simple baseline SimpleView. On the RSCNN protocol, SimpleView achieves an accuracy of 93.6 with 0.8 M parameters, while the proposed model achieves 93 with 0.56 M parameters [1]. Hence on a fair comparison, the final architecture is not better than simple projection baselines. It is unclear why the response only mentions the improved numbers for the method and not the fair numbers for the baseline.

Getting better performance than baselines becomes especially important as the paper is using a heavy machinery in the form of NAS while the baseline did not optimize to reduce parameter count.

In summary, I recommend a rejection because the paper is using heavy machinery like NAS to come up with a model not better than simple models existing in the literature. Therefore, in the reviewers opinion, the paper is not ready for publication at the moment.

[1] Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline, ICML 2021

 Dear AC,

I would like to thank the authors for providing all the comments. In particular, the feature engineering part can be better clarified and highlighted since it is indicated in the answer that "(it) can be established as a principle to evaluate the quality of a NAS method." After reading the comments from other reviewers, I can see this work has both merits and issues to be further improved. I would like to maintain my score as it was based on my general knowledge on this topic. 

Thank you. 

 Dear reviewers,

We would like to first thank you for your thoughtful comments and suggestions on reviewing our manuscript submitted to ICLR'22. Your feedbacks are valuable for us to improve the quality of our manuscript and our future research on 3D point cloud applications. We summarize the major revision of our manuscript during the rebuttal period and highlight the improvements that we have accomplished following your valuable suggestions:
- We **highlight the novelty and contribution of the MAKPConv module** that promotes efficient and high-performing learning on 3D point clouds. Specifically, we revise the presentation of NK attention to clarify the kernel point convolution part and highlight the novelties and/or dynamics in the design of NK attention In addition, we **emphasize the importance of neural architecture representations in predictor-based NAS and present our solution**: we carry extensive feature engineering on different NAS features and utilize a Wide \& Deep predictor to improve the quality of performance prediction.
- We follow the SimpleView-RSCNN protocol and **improve ModelNet40 results to over 93\% OA**. This gives us a much fairer comparison with other state-of-the-art works on ModelNet40.
- We **incorporate additional ablation studies on S3DIS**, an indoor scene segmentation dataset to justify the effectiveness of our searched architecture. The obtained results **prove good transferability of our searched architecture**, and **achieve a new state-of-the-art mIOU on S3DIS** under 6-fold cross-validation compared to previous works.
- We **provide an operator-level latency breakdown of our searched MAKPConv models** and compare latency measurements with other efficient 3D point cloud methods. The breakdown points out some potential improvements that can be made to improve the execution efficiency of the MAKPConv block, such as more efficient CUDA implementation on preprocessing, MAKPConv kernel optimization and operation fusion, etc. **We identify this as our future work to further improve and boost the efficiency in 3D deep learning.**
- We **do a careful proofreading** and fix minor errors, ambiguous notations, and partially loose connections in the previous manuscript. This makes our paper easier to read and better clarifies our idea on 3D point cloud.

We are very glad to see that some reviewers have already replied to our rebuttal, and posted positive feedbacks upon the revised version of our manuscript, accompanied by an increase in the review score. We hope that our rebuttal and the latest revision of our manuscript can fully address your concerns and yield an updated review score towards positive outcomes.

Thank you again for serving as reviewers of Paper 756 and providing thoughtful inputs!

-- Paper 756 Authors

This paper aims to accelerate the inference of 3D point cloud neural networks. The authors first borrow a few designs from efficient 2D neural networks: depthwise convolution and inverted residual bottleneck. They then propose to reweight the weights of kernel points with attention to boost its representation power. Finally, the authors adopt the predictor-based neural architecture search to explore the best model under a resource constraint. They have evaluated their proposed solution on the small-scale ModelNet40 dataset and the large-scale SemanticKITTI dataset and have achieved reasonably good empirical performance. The problem studied in this paper is very important. 3D point cloud neural networks are the core of many real-world applications (AR/VR and autonomous driving). At the same time, most existing solutions are still not efficient enough to provide a satisfactory user experience and guarantee user safety.

The paper is well-written and relatively easy to follow. Furthermore, the proposed solution is technically sound and has achieved fairly good empirical performance on small-scale and large-scale benchmark datasets.

My primary concern of this paper is its limited technical novelty. This paper is a combination of existing solutions:
* Depthwise convolution and inverted residual bottleneck are originally proposed in MobileNet-V1/V2.
* The reweighting mechanism adopted in this paper is essentially the same as in SENet [Hu et al., CVPR 2018].
* The neural architecture framework used in this paper is almost identical to Neural Predictor [Wen et al., ECCV 2020].

The only thing new to me is the unified dense and sparse neural architecture representations, which, however, is more of an engineering improvement than a technical contribution. The authors should highlight their core contributions apart from applying existing techniques to a new modality. Besides, the existing solution (SPVNAS) already adopts the one-shot weight-sharing NAS framework to explore the best neural architecture. It is unclear why the authors follow a different approach.

The authors claim that MAKPConv is 52% faster than SPVNAS, which is inaccurate. Having fewer MACs does not necessarily translate into measured speedup on the actual hardware. The authors should directly compare the latency of MAKPConv with SPVNAS (and other baselines) in Table 4. Furthermore, the techniques proposed in this paper are tailored for #MACs reduction but not necessarily for latency reduction. Point-based neural networks are usually bottlenecked by the KNN computation rather than the convolution (while most techniques in this paper aim at reducing the computation of convolution). It is unclear whether these modifications are also helpful to reduce the latency. The authors should include latency numbers in Table 1.

The authors have only reported the performance on the validation set of SemanticKITTI. They should also include some numbers on the final testing set. Besides, it would be better to verify whether the searched neural architecture is transferrable to other datasets (such as ScanNet and nuScenes) and/or tasks (such as panoptic segmentation and object detection).

References:
* Hu et al., "Squeeze-and-Excitation Networks", CVPR 2018.
* Wen et al., "Neural Predictor for Neural Architecture Search", ECCV 2020. The paper has relatively weak technical contributions and lacks latency comparisons with state-of-the-art solutions. Therefore, I would like to see the authors' response before the final decision.

 Thanks again for your appreciation and interest in our work! We have revised our manuscript to incorporate latency measurement in SemanticKITTI on Table 4, and attached the latency breakdown in Appendix A.7.

We agree with your concern regarding the latency comparison of MAKPConv versus other works, especially voxel-based works such as SPVNAS. We profile the latency breakdown of MAKPConv and find that despite being faster in computation (112ms vs. 158ms), MAKPConv yields a ~57ms overhead when processing inputs on CPUs, including radius sampling, grid sub-sampling etc. This adds to a total inference cost of 169ms which is slightly slower to SPVNAS, despite having 5$\times$ fewer FLOPS. However, we would like to highlight the merit of MAKPConv architectures as well as the proposed NAS methodology in point-based networks in this scenario:

- MAKPConv consumes much fewer parameters (i.e., 83% fewer) and yields $\sim 1$ mIOU improvement compared to the current state-of-the-art on SemanticKITTI validation dataset.
- The accomplished MACs reduction in MAKPConv demonstrates the power of the proposed resource-aware NAS method, giving the potential to further incorporate latency as the search objective in the future to speed up the point-based 3D networks.
- The main reason of having a lower throughput may arise from the lack of optimization in existing algorithms, such as (depthwise) kernel point convolution (non-fused convolution) and radius sampling (on CPU). We propose a few solutions at the end of Appendix A.7, such as adopting a fused version of (depthwise) kernel point convolution and implementing a CUDA kernel of radius sampling to achieve speedup. We put this as part of our future work on boosting inference speed.
- Despite the bottleneck in point-based networks, our profile shows that computation-bounded operations (e.g., depthwise kernel, MLP layers, etc.) still contribute to 50% of the overall latency. Thus, optimizing point-based NAS architectures is still valuable and expects more significant speedups when coping with the advancement of improving MAKPConv implementations via software-hardware co-design.

Overall, we appreciate the reviewers' effort in suggesting us to add latency into the table and showing us the need to perform a deep analysis on the latency breakdown of our NAS-crafted model. We hope that our detailed study on latency breakdown may enlighten future research on (point-based) 3D point-cloud models and bring more merits to the community.

**Other minor issues**
We unify the MACs/latency measurement setting on SemanticKITTI over a batch of $\sim$60K points (i.e., \# points under a scene). In the last version, we over-estimate \# points under a scene to $\sim$120K, thus over-estimate the MACs of our architecture and KPConv in Table 4. We have fixed it in the latest revision and updated the results accordingly.

 Thanks for the detailed response. I appreciate the clarification on the technical novelty and additional experiments on S3DIS. Therefore, I've increased my score accordingly.

I still have some questions regarding the latency comparison. I agree that point-based models are not directly comparable with sparse voxel-based models. However, it would be helpful to include latency numbers in Table 3 and Table 4 (where most baselines are actually point-based models).

I understand the radius query is more efficient than KNN. Still, it would be more informative if the authors could provide the latency breakdown of radius query, neighbor gathering, and convolution to help readers understand the computation bottleneck for KPConv.

 Thanks for your thoughtful comments on our work regarding automated mobile attention kernel point convolution. We address your concerns below:

**Clarification of MAKPConv.** We would like to first emphasize that MAKPConv only considers rigid KPConv without learning the location of kernel points via deformable KPConv. Though deformable KPConv learns better kernel locations, it operates at the cost of an additional KPConv layer thus sacrificing efficiency. We also would like to clarify each question regarding the relationship between MAKPConv and original KPConv as follows:
- The connection between MAKPConv and KPConv. We employ a depthwise kernel in the convolution part (See [1] eq.2.) to improve efficiency, and replace the linear correlation function (See [1] eq.2, h(y_i, \hat{x}^k)) with Neighbor-Kernel attention (NK attention) to better emphasize learning on important depthwise kernels.

- NK Attention. As the name suggests, NK attention measures the correlation between all neighbor points and all kernel points.
Mapping MAKPConv to KPConv. The NK attention maps to (i.e., improves) correlation function h(.) in [1], and depthwise kernel maps to (i.e., improves) Figure 2(right) in [1] which is exactly the rigid KPConv. Thus, Eq. 4 and Eq. 5 in [1] are not related to MAKPConv yet can be further studied to gain potential improvements.

- D_in and D_out. With a depthwise kernel, D_in is always equal to D_out. We clarify this in the latest revision of our paper.
Eq. (1) and Eq. (2). As the depthwise kernel applies a single filter to each channel, the convolution process (i.e., channel-wise matrix multiplication process) can be easily represented as element-wise multiplication. In Eq.(1) and Eq.(2), the inputs have already passed the correlation function (i.e., linear correlation or NK attention) that calculates the relationship between neighbor points (with the accompanying features) and the kernel points. Thus, the inputs can be directly sent into the depthwise kernel for convolution. We clarify this in the latest revision of our paper.

- Transform D_in to D_out feature maps. We use Linear layers to transform D_in features into D_out features in MAKPConv module. These 2 Linear layers sandwich the MAKPConv operation to carry channel-wise feature extractions. 

**Additional benchmark evaluations.** We evaluate MAKPConv on S3DIS, an indoor scene segmentation benchmark, and reach state-of-the-art results under the 6-fold cross-validation protocol. We demonstrate the new experimental results in Appendix A.4 in our latest revision.

**ModelNet-40 OA Fix.** We use the latest SimpleView-RSCNN protocol which can reproduce the 92.9% OA in KPConv, and demonstrate 93%+ OA for the NAS-crafted MAKPConv models. As a result, we add more 3D networks with more than 93% OA in Table 3 to strengthen the evaluation of our NAS-crafted models. Please see our latest revision.

Reference: [1] https://arxiv.org/pdf/1904.08889.pdf

 Thanks for your positive comments on our efforts in making lightweight point cloud processing models that could be very useful in many applications. We also appreciate your suggestions on our work regarding automated design on mobile 3D point-cloud models. We address your comments below:

**Motivation to start with KPConv.** KPConv provides a hierarchical design of 3D models: it adopts kernel point convolution as a building block to build the overall 3D architecture. Thus, such hierarchical design suits the goal of Neural Architecture Search that searches for the best combination of structural settings, positional settings, and accessories to build an optimal architecture. An efficient architecture like SimpleView can only be applied to projection-based methods with ResNet-18 backbones, thus may not contain the same diverse choices as KPConv blocks for further incorporation into NAS.

**Sub-optimal Accuracy vs. Parameter trade-off, SimpleView protocol.** Originally, we employed the open source code of KPConv and directly used their native protocols to carry the ModelNet40 evaluation. This leads to 92.3% OA during reproduction and 92.6% OA for the best architecture discovered. We take the reviewer’s suggestions to try the SimpleView RSCNN protocol as it is the closest with KPConv protocol and benefits from extra data augmentation techniques within it. We achieve 93.1%/93.4% (mean OA) for the NAS-crafted model with 0.56M/1.21M parameters, respectively. This further strengthens the evaluation of our method as now we obtain a NAS-crafted model with on-par accuracy while saving 30% parameters. We also observe 0.2% OA variance during the evaluation. We report the mean accuracy in Table 3 in a revised version of our paper, and update the training protocols accordingly.

**Complicated architectures.** We explain the benefits of our MAKPConv architectures as follows: 
1. MAKPConv employs a depthwise version of the KPConv to improve model efficiency, and utilize NK attention to improve the correlation measurement between neighbors and kernel points. The efficiency is justified in the reduction of parameter count in Table 1, and the contribution of NK attention is reflected in the performance comparison in Table 1. 
2. The final architecture discovered by NAS optimizes the arrangement of blocks in the hand-crafted version of the MAKPConv architecture to achieve better parameter efficiency and performance. The results are justified in Table 3/4/5 on both ModelNet40 and SemanticKitti benchmarks.

Therefore, we verify each contribution of our NAS-crafted MAKPConv architectures thoroughly to justify the superiority of our proposed method.


 Thanks for your thoughtful comments on our work regarding automated design on mobile 3D point-cloud models. We appreciate your acknowledgment on the significance of developing 3D architectures, and address your major concerns as follows:

**Novelty Issues:**
- Depthwise Kernel Point Convolution. We propose depthwise kernel point convolution to address the parameter inefficiency in the original version of the kernel point convolution, which is highly motivated for mobile devices. Though depthwise convolution is popular on 2D images, similar ideas have never been incorporated into 3D point clouds to achieve good performance-efficiency trade-offs as MAKPConv, to the best of our knowledge. Thus, the depthwise kernel in MAKPConv is a non-trivial approach in pushing forward the efficiency of 3D models.
- Delving into NK attention. We would like to emphasize that NK attention has a completely different mechanism as squeeze-and-excitation: 
1) instead of considering the channel-wise relationship, we consider the more important kernel-wise relationships which are directly reflected by kernel dispositions and lead to more parameter-efficient representation of NK attention mechanism. 
2) Thanks to the parameter efficiency, we no longer need the ‘squeeze’ part of squeeze-and-excitation, which yields low-rank representation of kernel information. Thus, NK attention yields a better re-weighting scheme than squeeze-and-excitation networks with better ability to learn the correct neighbor-kernel correlations. 
We revise the NK attention part in the latest revision to justify this.

- Highlighting Dense & Sparse Representation Learning. The key contribution of the predictor-based NAS part is identifying the key importance of assigning different priors to different searchable features for better neural architecture representations. While Wide & Deep predictor is not new in the community, it is a great tool to leverage all these improved representations and leads to a better prediction quality on accuracy prediction. We highlight such key contributions in the latest revision of our paper.

**Why not use one-shot?** As point-cloud applications are very loss-sensitive, utilizing a one-shot network with strong co-adaptations between different sub-networks may be challenging to obtain the best solution. In addition, performing one-shot NAS on point-based networks may be difficult to address kernel dispositions with different choices on the number of kernel points. Thus, we adopt predictor-based NAS where a sample-based strategy is easier to overcome the limitations and obtain an accurate evaluation of 3D models. We will acknowledge one-shot can also be a potential solution which we will explore in our future work.

**Additional results on NAS-crafted model evaluation.** We transfer the model towards S3DIS indoor scene segmentation and the results show ~1% mIOU improvement over the current state-of-the-art, Point Transformer, on the 6-fold cross validation. We include the detailed results in Appendix A.4 in the latest revision of our paper. For test results, we intend to release this later as we are concerned about the leak of identity information on the SemanticKITTI test website.

**Latency comparison.** Considering the difference of hardware kernels for point-based methods and point-voxel methods, it may be slightly unfair to compare the latency of SPVNAS and MAKPConv. We also notice that SPVNAS benefits from extra speedup from the bottom-level CUDA kernel implemented in torchsparse. Thus, we consider the hardware-aware speedup of KPConv networks as a part of our future work. 

We agree that in traditional point-based networks, KNN consumes the most of the processing latency thus should be optimized for faster inference. However, KPConv already uses radius sampling instead of KNN to address such bottlenecks during real-time execution. Thus, we turn our focus to optimize the structure of convolution operations to address latency. In addition, we would like to emphasize that optimizing convolution operations is beneficial for latency improvement as: 
1. the latency cost of radius sampling is proportional to the depth/width of KPConv layers in the point-based networks and requires NAS to reduce while maintain a good performance. 
2. We find that memory-bounded operation (e.g., neighbor gathering) takes around 14% of the total latency, while computation-bounded layers (i.e., convolution) contribute to the rest of the cost of the inference latency. Thus, optimizing MAC is still beneficial for latency optimization and can be incorporated into further software-hardware co-design to realize MAC efficiency on hardware.
 
To show a fair and better comparison, we show that compared to the original KPConv with 103ms latency at a batch of 42K points, our MAKPConv crafted by NAS has achieved ~70ms with better performance. We hope that this can address your concern in latency comparison results.


 Thanks for your feedback. We appreciate your positive comments on the importance, motivation, and results of our work. We address your concerns below:

**Novelty of the proposed method**. We would like to emphasize the contribution of this paper on both the point cloud applications and the Neural Architecture Search (NAS) (predictor-based NAS in particular):

- On the 3D point cloud, we discover the main source of parameter inefficiency in 3D models. We thus propose a depthwise KPConv kernel, and consequently identify the importance of finding the correct relationship between kernel points and neighbor points to compensate for the lost representation power of a depthwise kernel.
- On NAS, we identify the challenges in formulating a good neural architecture representation with specialized priors for each type of searchable features, and address this with the solution of a Wide & Deep predictor.

We would like to emphasize that the contribution does not only lie in the use of the depthwise kernel and the attention mechanism but also lies in the fact that we identify the key challenges and bottlenecks in 3D point cloud model design and address them with our proposed solutions. We highlight the contribution of our paper at the end of the Introduction, see the latest revision of our paper.

**Clarification on Wide \& Deep predictor.** In the related work, we show some sources of improvement (e.g., sample efficiency, architecture dataset augmentation) in the previous predictor-based NAS methods and propose a new direction to improve predictor-based NAS by having a more accurate predictor via better feature engineering. As predictor-based NAS explicitly requires architecture representation to encode an architecture, it is an essential ingredient in predictor-based NAS and can be established as a principle to evaluate the quality of a NAS method. For example, the same idea can be applied to NASBench-101/201/301 search space that corporates both positional settings and structural settings on image classification benchmarks, or even on ProxylessNAS/MobileNetV3 search space for 2D object detection tasks. We address this in the latest revision of our paper.

In addition, we did a careful proofread and fixed minor issues of the paper, such as spelling and typos.


This paper addresses point-based methods for classification and semantic segmentation in 3D applications. It aims to improve the computational efficiency of this kind of methods to make it better fit the applications of limited resources such as mobile scenarios. To achieve this, this paper conducts the following works. First, it develops a mobile attention kernel point convolution (MAKPConv) scheme to improve the performance of existing kernel point convolution. Second, it utilizes neural architecture search (NAS) technique to design the MAKPConv-based network. In this part, this paper proposes a wide & deep neural predictor for the NAS process. Experimental study is conducted on benchmark datasets and tasks to demonstrate that the resulted deep networks can achieve higher computational efficiency and improved performance. Ablation study is also conducted to illustrate the key components in this work.  Strengths:

1. The point-based 3D application focused by this work is of importance and has recently attracted much attention as a new research area;
2. The motivation on computational efficiency makes sense and the experimental result demonstrates that this is well achieved by this work;
3. The idea of changing KPConv to a depthwise kernel and then using NK-Attention to compensate the degraded performance caused by the limited representative power of the latter is sound, and its effectiveness is verified by Table 1. 
4. The overall performance of the resulted MAKPConv-based network is validated in the experimental study. 
5. The ablation study on the kernel contribution via the NK-Attention is helpful for illustrating the function of the proposed attention mechanism. 

Weaknesses:
1. Although the ideas of using a depthwise kernel and the attention mechanism work well in this paper, they are well-known techniques in the literature. In this sense, the contribution of this work in this part is mainly on innovatively using them to improve the computational efficiency of KPConv. The theoretical contribution of this work seems to be limited. 
2. The part on the proposed wide & deep neural predictor and the idea of carrying feature engineering on searchable features in NAS can be made clearer. Firstly, the key differences and novelties with respect to the existing related work can be further clarified. Secondly, can the proposed feature engineering idea be regarded as a new principal in NAS that can be generally applied? Clarifying these issues will make the contributions in this part easier to be understood. 

Minor issues
1. "N" in Figure 1 shall be explicitly defined;
2. Kenrel --> Kernel in the caption of Figure 1;
3. "...my our novel Wide & Deep ..." in Conclusion.  This work is well motivated. The proposed idea on reducing computational cost is neat, and it works effectively as experimentally demonstrated. The proposed wide & deep predictors for NAS helps to efficiently find the high quality architecture for the proposed network, achieving promising performance. Meanwhile, the potential issues are i) the idea of MAKPCon is mainly based on existing well-known techniques and does not seem to bring significant theoretical insight and ii) the novelty and contribution of the Wide & Deep neural predictor part can be better clarified.

This paper aims to address the point cloud analysis task by:

1) improving the existing KPConv operation via considering the kernel relationship;

2) designing the networks via a predictor-based NAS approach.

The improved KPConv operation. i.e., MAKPConv, can model the local structure efficiently, and the searched network has fewer parameters while performing better than the baseline on two datasets, including ModelNet40 for classification and SemanticKitti for segmentation. STRENGTHS:

1. This paper is well-written, in general.
2. The motivation behind MAKPConv is reasonable.
3. The networks have fewer parameters, and outperform previous works, especially on Semantickitti.
4. The analysis on NAS for block structure search is enough.

Weakness:

-- I am confused about the introduction to MAKPConv.
1. What is the connection between MAKPConv and KPConv. In KPConv, the correlation between each neighbour and each kernel point is calculated, which I do not find in MAKPConv.
2. In NK Attention, the NK Distance indicates the distance between the neighbour and the kernel point or between every two neighbours?
3. It seems that MAKPConv is an improved version of KPConv, but I cannot associate them in Figure 1. For example, it seems that the NK Attention is similar to the h(.) function in Eq. 5 in KPConv [1]. If so, the Depthwise Kernel in the upper-left part of Figure 1, should be Eq.5, right? Then, which part indicates Eq. 4 in KPConv[1]. Maybe I misunderstand the figure. Please clarify this.
4. In Eq. 1, the dimension of F is $D_{in}$, while $i\in[1,...,D_{out}]$, which makes me confused. Does $D_{in}$ equal to $D_{out}$?
5. In Eq. 1 and Eq. 2, since the points are unordered and we do not know which kernel point corresponds to a certain neighbour, how can we have $F_{\*,k,\*} \cdot W_{k,\*,\*}$? In addition, I also cannot find a similar formulation in KPConv.
6. In Eq. 2, the dimension of $F$ and $W$ is $D_{in}$, so how can we get $F'$ with dimension $D_{out}$? 

-- Experimental results
1. Currently, previous works on point cloud analysis usually conduct experiments on several datasets, such as S3DIS, ShapeNetPart, and ScanNet, while this paper only considers two datasets, including ModelNet40 and SemanticKitti, which cannot demonstrate the effectiveness adequately. 
2. The OA of KPConv is 92.9\%, which is higher than the OA of the proposed method (92.6\%), while in this paper the authors do not report that result and instead report the result of their implementation.
3. On ModelNet40, currently there are many works yielding OA larger than $93\%$, but the authors do not report.

[1] https://arxiv.org/pdf/1904.08889.pdf

 I have two concerns,

1. The introduction to the proposed MAKPConv is unclear.
2. The experiments are not enough.

As a result, currently, I vote for borderline reject. But I might change my score if the authors can address my concern or other reviewers can give more positive reviews.

Meta-review: Four reviewers have reviewed this manuscript and two found it borderline leaning towards acceptance, two other reviewers scored it below the acceptance threshold. While the authors *identify the key challenges and bottlenecks in 3D point cloud model*, the most positive two reviewers notice that the depthwise kernel and the attention mechanism (and similar tools) are well-known in the literature and that this work is *more of an engineering improvement than a technical contribution*, which erodes the novelty of the proposed idea on that front. While authors noticed some discrepancies in the numbers quoted by Rev. 3, the model gains are also nonetheless modest compared to other models. Overall, the feeling amongst the reviewers was that  the presentation of NK attention could be further improved and that the paper uses very heavy machinery to achieve results comparable with SOTA.

On this occasion, the manuscript is below the acceptance threshold with even the borderline positive reviewers having their doubts about clear cut technical novelty.

Decision: Reject