Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors proposed Point Transformer V2, which is based on the previous state-of-the-art Point Transformer. Compared to PT V1, PT V2 introduces several improvements, including group vector attention, grouped weight encoding layer, and a novel sampling method. Experimental results on ScanNet V2, S3DIS, and ModelNet40 show demonstrate considerable improvements upon previous methods.
strengths_and_weaknesses: Strengths:
I like the presentation and organization of this paper. 
Experimental results show significant improvement over previous work.
The grid upsampling/downsampling at the feature level is interesting.

Weakness:
1. One of my concerns is the Group Vector Attention, as presented in Sec. 3.2. From my understanding, it simply applies the group operation to the vector attention in PTV1. I don't think adding a group operation  (or using a group operation to replace MHA) is novel. 
The authors claimed, "GVA is a generalized formulation of VA and MSA". This claim is trivial, and we can easily achieve this statement. Besides, when generalizing Group Attention to Multi-head Attention, it can be mathematically generalized, but the implementation may not be easily achieved (we may need pad 0 manually). This generalization is not only suitable for GVA, but is common in all group operations. In Eq. 4, the value should be learnable parameters instead of 1. The Grouped linear is still a simple group operation.

2. In Sec. 3.3, I could not get the motivation of the additional multiplier, which further emphasizes (I would not say enhance) the positional information. Why authors introduce the multiplier to the relation vector? I would like to see a more detailed analysis or theoretical proof, not barren "explanation".

-- For the Sec. 3.4 Grid-based Sampling Methods are more like feature-level voxel or grid sampling. I like this modification, but it is not that novel. 

3. About the experiments. The ModelNet40 and S3DIS datasets are relatively simpler, and the performance of models trained on these two datasets varies greatly (maybe because of the overfitting problem?). This is a common issue in the community. Also, ScanNetV2 is not that complex. I would suggest the authors report the mean-std on simple datasets and add more experiments on complex tasks, like 3d object detection. 

4. In Table 4, I would like to see an ablation study that uses max-pooling to replace attention, and this study can serve as a baseline. 
 

questions: Please see above weakness. 
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper upgrades Point Transformer into its version 2 in three ways. First, group vector attention is designed and compared with other alternatives. Second, an additional multiplier is added to the position encoding. Third, a light-weight grid-based sampling is proposed. Point Transformer V2 achieves competitive results in point cloud semantic segmentation and classification tasks.
strengths_and_weaknesses: Strengths:
1. The paper is well-presented with clear illustrations such as Figure 2.
2. The discussion on the  attention mechanism is interesting and insightful. I especially like the degenerate analysis.
3. Solid improvements on fundamental point cloud tasks are obtained.

Weaknesses:
1. It is more complete to include a benchmark for part segmentation.
2. Line 113, I do not think shifted-grid attention is a “common” practice especially the references are Swin Transformer V1 and V2. Besides, the shifted window technique that Swin Transformer employs essentially leads to overlapping windows. Author’s comments on using non-overlapping windows. 
3. The investigation in the position encoding multiplier is relatively insufficient, other than one row of ablation experiment. What does it really do?
4. A complexity analysis of FPS vs grid-based sampling will be helpful.

questions: Please see the Weaknesses section.

limitations: No additional limitations or potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper propose grouped vector attention (GVA) module, which allows information to be exchanged within and between attention groups through a new weight encoding layer. It enhances the location encoding by adding a multiplier to the relational vector. And replaces the farthest-point-sampling-based downsampling method and the interpolation-based upsampling method used in PTv1 with an efficient grid upsampling method of its own setting.
strengths_and_weaknesses: - Originality: The main idea of the proposed approach is to use the transformer. To my limited knowledge about it, it lacks a little novelty, but the originality is reasonable.
- Quality：While the approach seems reasonable and the experimental results look promising, I have the following concerns (See Questions) about the paper.
- Clarity：This manuscript is clearly written.
- Significance：There are not many works in the field of point cloud feature learning by transformer, so I think this paper makes a positive contribution.
questions: 1) I hope the authors can add a visual explanation of the results of the grouped vector attention module that they used in the ablation experiments.
2) The proposed Position Encoding Multipler seems reasonable. I want to know the impact of the Multipler on the overall network params, inference cost time and downstream task accuracy compared with the encoding scheme $\delta = \theta(p_i - p_j)$ in PTv1[1].
3) The authors keep emphasizing that their grid upsampling is more efficient than the farthest-point-sampling-based downsampling method and the interpolation-based upsampling method. However, I only see the metric results for the downstream task. And only in \textbf{Appendix B. Table 4}, there are a few quantitative results related to cost time and params (I want to see a more detailed comparison of the grid upsampling and downsampling method with PTv1[1], e.g. different \textbf{grid sizes})
4) I don't see any relevant explanation of network loss in the paper and the supporting material. I hope the authors can give detailed loss settings for their experiments on downstream tasks.

In the current version, there are some issues as well. I look forward to the response by the authors. For now, I would recommend a borderline accept rating for the paper.

[1]Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021.
limitations: Yes, I don't see any potential negative social impact of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Summary:

---
- Point Transformer V2 (PTv2) proposes a new attention mechanism called group vector attention (GVA) that is more efficient than its multi-head self-attention (MHSA) and vector attention of Point Transformer (PTv1) [[1]] counterparts. The effectiveness of GVA is then tested on the tasks of 3D semantic segmentation and 3D shape classification where it achieves competitive performance.

- PTv2 proposes a novel positional encoding scheme that introduces a multiplier $\delta_{mul}$ to the relation vector in addition to the bias that was introduced in PTv1 $\delta_{bias}$.
 
- PTv2 proposes to introduce grid fusion downsampling and grid mapping upsampling as more effective, efficient, and more spatially aligned alternative to furthest point downsampling and trilinear interpolation upsampling used in PTv1.

[1]: https://openaccess.thecvf.com/content/ICCV2021/papers/Zhao_Point_Transformer_ICCV_2021_paper.pdf

strengths_and_weaknesses: Strengths:

---

- The proposed GVA is more efficient than the vector attention proposed in PTv1 and MHSA while still maintaining competitive performance on semantic segmentation benchmarks such as S3DIS Area 5 and ScanNet test set and shape classification on ModelNet40.

- The paper provides a thorough ablation study most important of which are attention type (GVA vs MHSA) and neighborhood (local vs shifted-grid). Table 7 highlights the effectiveness of the proposed GVA and positional encoding multiplier.

Weaknesses:

---

- PTv2 does not compare to CVPR2022 work titled _“Stratified Transformer for 3D Point Cloud Segmentation”_ [[2]] that is currently the state-of-the-art point-based method on S3DIS Area 5 benchmark with 72 mIoU and reaches 74.7 mIoU on ScanNet test set as per the ScanNet benchmark website (however paper reports 73.7 mIoU). The work uses vanilla multi-head self-attention (MHSA) and uses shifted grid to aggregate information beyond KNN local neighborhood. [[2]] also introduces positional encoding multiplier but in a much more involved scheme. Having said that, PTv2 does include an interesting ablation study in section 4.3 on the attention type used (MHSA vs GVA) and locality type (Shifted-Grid vs Neighborhood) that supports the authors’ choices.  

- The comparison with PTv1, in its current form, is not fair because of the architectural differences. The PTv2 uses an encoder with block depths [2,2,6,2] while PTv1 uses [1,1,1,1]. I think a better assessment of the effectiveness of the proposed GVA would be to fix the PTv1 encoder-decoder architecture (including block depths, positional encoding, embedding dimensions, and downsampling/upsampling methods), and just replace the vector attention operator with the proposed GVA.

- The paper claims that the proposed GVA is more efficient. It is evident from the mathematical elaboration in section 3.2 and Table 4 of supplementary material that GVA is more efficient than vector attention of PTv1 and MHSA. However, I would like to see the final points/second throughput of the network as compared with PTv1 and Stratified Transformer (comparison with Stratified Transformer might be tricky because they use CUDA kernels for their attention mechanism).

- The downsampling and upsampling grid based methods are not novel. They were first introduced in KPConv[[3]] and used by later works such as CloserLook3D[[4]]. Maybe the authors meant to say that they introduced these sampling methods to PTv1 that uses furthest point sampling for downsampling, and trilinear interpolation for upsampling.

[2]: https://openaccess.thecvf.com/content/CVPR2022/papers/Lai_Stratified_Transformer_for_3D_Point_Cloud_Segmentation_CVPR_2022_paper.pdf 

[3]: https://openaccess.thecvf.com/content_ICCV_2019/papers/Thomas_KPConv_Flexible_and_Deformable_Convolution_for_Point_Clouds_ICCV_2019_paper.pdf

[4]: https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123680324.pdf
questions: Questions and Suggestions:

---
- Is there a particular reason why PTv2 does not compare against Stratified Transformer[[2]] but does compare to another CVPR2022 work titled _"Fast Point Transformer"_[[5]]? [[2]] shows that a vanilla MHSA is effective for 3D discriminative tasks and demonstrates the benefits of using shifted-grid locality type for long range contextual information aggregation.

- I would like to see an ablation where the PTv1 architecture is fixed and only the vector attention is replaced with the proposed GVA. I think this is a more fair comparison and a real test for the effectiveness of the proposed GVA.

- I would like to see a comparison of the final points/second throughput against PTv1 and Stratified Transformer.

- I would like a clarification from the authors on the novelty of the used downsampling and upsampling grid based methods.

I am willing to change my rating if my questions and concerns were addressed reasonably in the rebuttal.

[2]: https://openaccess.thecvf.com/content/CVPR2022/papers/Lai_Stratified_Transformer_for_3D_Point_Cloud_Segmentation_CVPR_2022_paper.pdf 

[5]: https://openaccess.thecvf.com/content/CVPR2022/papers/Park_Fast_Point_Transformer_CVPR_2022_paper.pdf
limitations: Even though it is not critical, the paper does not include shape-part segmentation experiments on ShapeNet part or PartNet.
review_title: null

Meta-review: metareview: The paper focuses on modifying specific internal modules in the PointTransformer pipeline for added benefits. It received four detailed reviews from expert reviewers. The discussion period included healthy back-and-forth between the authors and the reviewers, and many of their concerns were addressed and their questions answered. The AC does partially agree with Reviewer MnQR's assessment of the broad impact of this paper, specifically that it may not have a significant influence on the future of research in 3D recognition given the simple technical improvements it proposes. However, the empirical influence of these improvements and their justification/motivation are clear, as clearly stated by the other reviewers. As such, the AC sees that the current level of impact and contribution of the paper reaches the level expected for NeurIPS.
confidence: Certain
Decision: Accept