Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new model architecture for 3D problem which leverages the powerful backbones pretrained from the 2D task. The idea is straightforward. The input point cloud is projected into 2D pixels using an encoder model, then the 2D pixels is colored by a coloring module, and the colored images are fed into a pretrained ViT backbone and then predictions are made by the task-specific heads. The overall approach provides an elegant solution to leverage the representations of 2D models. The experimental results demonstrate superior performance on public benchmarks including ModelNet40, ShapeNetPart datasets.
strengths_and_weaknesses: 1. Novel model architectures to utilize pretrained 2d models. To my knowledge, the idea of using projection into 2D and coloring module is new. 
2. The idea is simple yet effective. The pretrained 2d models are easy to get and the results are promising.
questions: How much improvements are from the improved backbone architecture and added compute cost? It would be great to show an ablation study on that.
limitations: The proposed approach has some limitations on the feasible tasks to apply on. For example, it may not work if we want to conduct a 3D segmentation task. 
Related to the question above, the comparison in Table 4 is not fair due to lack of model complexity analysis.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors introduce point-to-pixel prompting (P2P), a learning framework for leveraging pre-trained image transformers for 3D tasks. The method is mainly motivated by the data scarcity issue in 3D domains. P2P learns a geometry-preserving transformation from point cloud to 2D grid, and then a projection to prepare the 2D grid data to be processed by a pre-trained image transformer expecting image tokens. The main benefit of P2P, to my understanding, is the ability to achieve comparable accuracy to other 3D models with much fewer parameters that need to be trained with 3D data. This is validated on two tasks, 3D object classification and 3D part segmentation.
strengths_and_weaknesses: Strengths

-----------------
- the question of whether knowledge can be transferred from large pre-trained image models for use with 3D domains is interesting
- the point-to-pixel prompt pipeline, which is nicely visualized in Figure 2, appears to be novel and is simple and elegant
- this work is a nice demonstration of ideas from NLP transferring successfully to other domains (in this case, to 3D point cloud processing)
- the paper is well-written and easy to read


Weaknesses

----------------- 
- $\text{\textbf{Unclear motivation,problem, and significance}}$: The current set of claims in the introduction are that A) there is a data starvation problem in 3D domain (L34-35) and B) pre-training point cloud transformers suffers from an imbalance between the number of trainable parameters and limited training data, leading to insufficient optimization and overfitting (L40-41). However, the data starvation problem seems to only exist for specific object-centric datasets such as ShapeNet. By contrast, consider the large Scannet and Waymo datasets. Moreover, recent advances in 3D rendering (e.g., NeRF) suggests that highly lifelike synthetic 3D data may soon become available. Therefore, scarcity of large datasets does not appear to be a fundamental concern. Moreover, point B) seems plainly false since recent methods like Point-BERT work just as well as P2P on, e.g., ModelNet40.
- As a result, it is unclear what the actual problem is that is being addressed here and *why* this prompting method is needed at all. The main benefit of P2P seems to be in the use of fewer model parameters, but its unclear why this is important.
- $\text{\textbf{Multiple unsubstantiated claims}}$. These can be addressed with careful editing. 
  - (L54) “The end-to-end optimization pipeline and the strategy of freezing the pre-trained image model promote the *bidirectional* knowledge flow between points and pixels”. To my understanding, the flow is *unidirectional*; pre-trained image features are being used to learn a better representation for points.
  - (L271) “Firstly, our P2P outperforms traditional 3D pretraining methods” (on ModelNet40). P2P’s largest model achieves the same performance as Point-BERT.
  - Similarly, claims of “superiority” of P2P (L64, L370) are clearly not supported by the accuracy results in the experiments.


References

-----------------
- Dai, Angela, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. "Scannet: Richly-annotated 3d reconstructions of indoor scenes." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839. 2017. 
- Sun, Pei, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo et al. "Scalability in perception for autonomous driving: Waymo open dataset." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2446-2454.
questions: Instead of taking a sum of point features in each cell (L189), did you try max pooling? It would be great to see a sensitivity analysis comparing {max,mean,sum} pooling for this, since images of semi-transparent objects (which is unrealistic) would seemingly be out-of-distribution for pre-trained image models on ImageNet?

Currently, the strengths and weaknesses of the paper roughly balance each other out in my opinion and I believe the work as it stands is borderline. I would be interested to read the authors response to my feedback. Thanks!

============================
After reading the author's responses to my questions and concerns I have increased my score to reflect that I feel the strengths now outweigh the weaknesses.


limitations: No. Limitations of the P2P framework should be discussed in the main text (e.g., in the conclusions section).
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes point-to-pixel prompting to leverage 2D pre-trained models to help 3D point cloud recognition tasks. The main modules include a geometry-preserved projection and a geometry-aware coloring, which fill the gap between 3D point clouds and 2D images. The experiments on ModelNet40 and ScanObjectNN show that P2P achieves comparable performance on classification tasks with only a few trainable parameters.
strengths_and_weaknesses: Strengths:
1. The paper first proposes a prompt-tuning method to adopt 2D pre-trained parameters in 3D, which is an interesting and novel exploration.
2. With p2p prompting, the model can achieve competitive results on the shape classification task with much fewer trainable parameters.

Weaknesses:
1. Although the method leverages extra 2D image knowledge, it does not show clear performance or speed advantages over previous 3D networks on both classification and part segmentation. The parameters that need to be trained are fewer but the whole model is larger. The 2D prior knowledge is not fully exploited in this method. 
2. The design of simply adding the point features in the same pixel seems trivial, and even with the explanations in Line190-197, I don't really think it preserves geometry. Also, no more experiments are conducted to analyze these design choices. 
3. More results on scene-level point cloud understanding with datasets like ScanNet or S3DIS are expected to illustrate the effectiveness of the prompt-tuning pipeline. 
questions: Please refer to weaknesses. Some more questions are listed below:
1. For the projection from 3D to 2D, which are the pixel features for those pixels without points?
2. How to obtain point predictions for part segmentation is unclear. It seems not very reasonable to simply add the multi-view predictions. Actually, the multi-view fusion is not clearly stated in the paper. 
3. How's the processing of the empty pixels in the coloring module? The visualization results look very clean and not very smooth actually, and I wonder if the empty pixels are filtered out in the coloring module. 
limitations: The limitation is discussed in Sec 4.3.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose to leverage the pretrained image model for point cloud downstream tasks. Specifically, they introduce a Point-to-Pixel Prompting to transform a point cloud as the corresponding image, by geometry-preserved projection and geometry-aware coloring.
strengths_and_weaknesses: Strengths
1) The paper is well written with clear motivation and good organization.
2) Leveraging 2D pretraining for 3D tasks is an interesting topic.
3) Point-to-Pixel Prompting is novel.

Weakness
1) My main concern is the experiment result. Apparently, the proposed design does not improve the performance.
2) What are the computation cost and model size for the prompting procedure? 
questions: See Strengths And Weaknesses
limitations: See Strengths And Weaknesses
review_title: null

Meta-review: metareview: The paper presents a method of prompt tuning to transfer 2D pre-trained weights to tackling 3D understanding problems. All reviewers are positive about the novelty of the method. With large 2D pretrained models, higher performances are still expected from xwSJ, which is also a reasonable comment. Other 3D understanding tasks, such as segmentation and detection of outdoor scenes, are strongly encouraged, as they are the true needs of the industry.
confidence: Certain
Decision: Accept