P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting
Authors: Ziyi Wang
Xumin Yu
Yongming Rao
Jie Zhou
Jiwen Lu
Venue: NeurIPS 2022 Accept (2022)

Abstract: Nowadays, pre-training big models on large-scale datasets has become a crucial topic in deep learning. The pre-trained models with high representation ability and transferability achieve a great success and dominate many downstream tasks in natural language processing and 2D vision. However, it is non-trivial to promote such a pretraining-tuning paradigm to the 3D vision, given the limited training data that are relatively inconvenient to collect. In this paper, we provide a new perspective of leveraging pre-trained 2D knowledge in 3D domain to tackle this problem, tuning pre-trained image models with the novel Point-to-Pixel prompting for point cloud analysis at a minor parameter cost. Following the principle of prompting engineering, we transform point clouds into colorful images with geometry-preserved projection and geometry-aware coloring to adapt to pre-trained image models, whose weights are kept frozen during the end-to-end optimization of point cloud analysis tasks. We conduct extensive experiments to demonstrate that cooperating with our proposed Point-to-Pixel Prompting, better pre-trained image model will lead to consistently better performance in 3D vision. Enjoying prosperous development from image pre-training field, our method attains 89.3% accuracy on the hardest setting of ScanObjectNN, surpassing conventional point cloud models with much fewer trainable parameters. Our framework also exhibits very competitive performance on ModelNet classification and ShapeNet Part Segmentation. Code is available at https://github.com/wangzy22/P2P.

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting
Ziyi Wang" Xumin Yu"
Yongming Rao" Jie Zhou Jiwen Lu
Department of Automation, Tsinghua University, China {uziyi22, yuxm20) www.aaai.a.aaaaaaaaaaeaaaaaaaaae tsinghua. edu. cn;
raoyongming360gmail.. com;
[lujiwen, jzhou)@tsinnhuua... edu. en
Abstract
Nowadays, pre-training big models large-scale datasets has become crucial topic in deep learning. The pre-trained models with high representation ability and transferability achieve a great success and dominate many downstream tasks in natural language processing and 2D vision. However, is non-trivial to promote such a pretraining-tuma tuning paradigm to the 3D vision, given the limited training data that are relatively inconvenient to collect. In this paper, we provide a new perspective of leveraging pre-trained 2D knowledge in 3D domain tackle this problem, tuning pre-trained image models with the novel Point-to-Pixel prompt- ing for point cloud analysis minor parameter cost. Following the principle of prompting engineering, we transform point clouds into colorful images with projection and geometry- image models, whose weights are kept frozen during the end-to-end optimization of point cloud analysis tasks. We conduct extensive experiments to demonstrate that cooperating with our proposed Point-to-Pixe Prompting, better pre-trained image model will lead to consistently better performance 3D vision. Enjoy- ing prosperous development from image pre-training field, our method attains 89.3% accuracy on the hardest setting of ScanObjectNN, surpassing conventional point cloud models with much fewer trainable parameters. Our framework also exhibits very competitive performance on ModelNet classification and ShapeNet Part Segmentation. Code is available at https : //::://gtttt gitt com/ wwnggy22/222....
coloring to adapt to pre-trained
Introduction
With the rapid development of deep learning and computing hardware, neural networks are experienc- ing explosive growth in model size and representation capacity. Nowadays, pre-training big models has become an important research topic in both natural language processing [14, 52, 4, 60] and com- puter vision [51, 53, 20, 57], and has achieved a great success when transferred to downstream tasks [51, 45, 67, 59, 29, 35] strategies. Fine-tuning is a traditional tuning strategy that requires a large amount of trainable parameters, while prompt tuning a recently emerged lightweight scheme to convert downstream tasks into the similar form as the pre-training task. However, such prevalence of the petraininngtuning pipeline cannot be obtained without the support of numerous training data in pre-training stage. Language pre-training leading work Megatron-Turin NLG [60] with 530 billion parameters is trained on 15 datasets containing over 338 billion tokens, while Vision MoE [57] with 14.7 billion parameters trained on JFT-300M dataset [62] including 305 million training images.
with fine-tuning [21, 10, 9. 2, 20] or prompt-tunin
'Equal contribution.
CCoreespondigg author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Images produced by our Point-to-Pice tixel Prompting. We show the original point clouds (top line) and the projected colorful images produced by our P2P of synthetic objects from ModelNet40 (left five columns) and real-world objects from ScanObjectNN (right three columns) from two different projection views.
Figure 1:
Unfortunately, the aforementioned convention of pre-training big models on large-scale datasets and tuning on downstream tasks has encountered obstacles in 3D vision. 3D visual perception is gaining more and more attention given its superiority in many emerging research fields including autonomous driving [28, 80], robotics vision [8, 78] and virtual reality [41, 71]. However, obtaining abundant 3D data such as point clouds from LiDAR is neither convenient nor inexpensive. For example, the widely used object-level point cloud dataset ShapeNet [7] only contains 50 thousand synthetic samples. Therefore, pre-training fundamental 3D models with limited data remains an open question. There some previous literature [77, 68, 81] that attempts to develop specific pre-training strategies on point clouds with limited training data, such as Point Contrast [77], OcCo [68] and Point-BERT [81]. Although they prove that the Eetraining-finetuniin
pipeline also works well in the 3D domain, the imbalance between numerous trainable parameters and limited training data may lead to insufficient optimization or overfitting problems.
Different from the previous methods that directly pre-train models on 3D data, we propose to transfer the pre-trained knowledge from 2D domain to 3D domain with appropriate prompting engineering, since images and point clouds display the same visual world and share some common knowledge. In this way, address the data-starvation problem in the 3D domain, given that the pre-training strategy is well-studied in the 2D field with abundant training data and that prompt-tuning on 3D tasks does not require much 3D training data. To the best of our knowledge, to transfer knowledge in pre-trained image models to 3D vision with a novel prompting approach. More specifically, we propose an innovative Point-to-Pise point clouds into colorful images with -preservd projection and geometry-aware coloring. Examples of produced colorful images are shown in Figure 1. Then the colorful images are fed into the pre-trained image model with frozen weights to extract representative features, which are further deployed to downstream task task-specific heads. The conversion from point clouds to colorful images and the end-to-end optimization pipeline promote the bidirectional knowledge flow between points and pixels. The geometric information from point clouds is mostly retained in projected images via our geometry geometry-preserved projection, while the color information of natural images from the pre-trained image model is transmitted back colorless point clouds via the cooperation between the geometry-aware coloring module and the fixed pre-trained image model.
are the first work
Prompting mechanism that transforms
We conduct extensive experiments demonstrate that with our Point-to-Pixe Prompting, enlarging the scale of the same image model will result in higher point cloud classification performance, which consistent with the observations in image classification. This suggests that we can take advantage of the successful researches in pre- me-training big image model, opening up new avenue for point cloud analysis. With much fewer trainable parameters, we achieve comparable results with the best object classification methods on both synthetic ModelNet40 [74] and real-world ScanObjectNN [65]. We also demonstrate the potential of our method to perform dense predictions like part segmentation on ShapeNetPart [79]. In conclusion, our Point-to-Pixe Pixel Prompting (P2P) framework explores the feasibility and ascendancy of transferring image pre-trained knowledge to the point cloud domain, promoting pre-training paradigm in 3D point cloud analysis.

---

2 Related Work
2.1 Visual Pre-training
Pre-training visual models has been studied thoroughly in the image domain. Supervised pre- training [15, 82, 6] on classification task with large-scale dataset a traditional practice and is stimulated by the boosting development of the ever-growing fundamental vision models [22, 23, 15, 37]. searly-s pre-training methods [63, 3, 76, 46] use less annotations while unsupervised pre-training approaches [21, 10, 9, 2, 20, 17] introduces no task-related bias and brings higher transferability to various downstream tasks.
Different from the prosperity of pre-training image models, pre-training 3D models is still under development. Many researches have developed self-supervised learning mechanisms with various pretext tasks such as solving jigsaw puzzles [58], orientation estimation [47], and deformation reconstruction [[]. Inspired by pre-training strategies in image domain, Point Contrast [77] adopts contrastive learning principle while OcCo [68], Point-BERT [81] and Point-M2AE [83] introduce reconstruction pretext tasks for better representation learning. However, the data limitation in 3D domain remains large obstacle in developing better pre-training strategies.
2.2 Prompt Tuning
Prompt tuning an important mechanism whose principle is to adapt downstream tasks with limited annotated data the original pre-training task minimum cost, thus exploiting the pre-trained knowledge to solve downstream problems. It is first proposed in the natural language processing community [33], and has been leveraged in many vision-language models. At first, hand-crafted prompting methods [45, 4] are promoted and their followers [67, 59] develop an automated searching algorithm select discrete prompt tokens within large corpus. Recently, continuous prompting methods [31, 29, 35, 34] are becoming the mainstream given their flexibilityy and high performance.
On the contrary, the development in prompting visual pre-training models lags behind. 1.2P [70] proposes prompt pool for continual learning problem while VPT [24] first introduces continuous prompt tuning framework inspired by P-Tuning [35, 34]. As far previous work like this paper to discuss tuning pre-trained image models for point cloud analysis with an appropriate prompting mechanism.
concerned, there is no
2.3 Object-level Point Cloud Analysis
Given the unordered data structure of point clouds, early literature has developed voxel-based and point-based methods to construct structural representations for point cloud object analysis. Voxel- based methods [43, 27]56] partition the 3D space into ordered voxels and perform 3D convolutions for feature extraction. Point-based methods [48, 49, 42, 12, 32, 73, 64, 69, 30J directly process unordered points and introduce various approaches to aggregate local information. Recently, attention-based Transformer [66, 18, 85] architecture has prevailed over other frameworks vision community and achieved competitive performance in point cloud object analysis.
Besides the aforementioned methods that perform representation learning in the 3D space, there are projection-based methods[61, 25, 72, 19, 16, 84] that leverage multi-view images to represent 3D objects. Recently, MVTN [19] introduces the differentiable rendering technique to build an end-to- end learning pipeline, rendering images online and regressing the optimal projection view. Different from theirs, our work designs a novel prompting engineering scheme, utilizing 2D color knowledge from pre-trained image models that is absent in colorless point clouds. Moreover, our framework is implemented faster single-view pattern, as we only select one random projection view during training and don't 't develop any aggregation strategy to explicitly fuse multi-view knowledge.
3 Approach
3.1
Overview
The overall framework of our P2P framework is illustrated in Figure 2. The network architecture consists of four components: 1) geometry encoder to extract point-level geometric features from
3

---

Poini-tu to Pixel Prompting
Classification
Pre-trained Image Model
Head
Geometry Encoder
Coloring
Module
Segmentation
Head
Image
Input P
Projection
Image Patches
Output Results
Figure 2: The pipeline of our proposed P2P framework. Taking point cloud P the input, first encode the geometry information each point. Then we sample a projection view rearrange the point-wise features into an image-style layout to obtain the pixel-wise features with Geometry-preservel Projection. The colorless projection will be enriched to produce colorful image / with the color information via a learnable Coloring Module. Our P2P framework can be easily transferred several downstream tasks with task-specific head with the help of the transferable visual knowledge from the pre-trained image model. We take the classical Vision Transformer [15] as our pre-trained image model for illustration this pipeline.
the input point clouds, 2) a Point- Prompting module to produce colorful images based on geometric features, 3) pre-trained image model to leverage pre-trained knowledge from image domain, and 4) a task-specific head to perform various kinds of point cloud tasks. We will introduce the geometry encoder, the Point-to-Pixel Prompting module and task-specific heads in detail in the following sections. As for the choice of the pre-trained image model, we investigate both convolution-based and attention-based architectures in Section 4.2.1.
With the proposed architecture that can optimized in an end-to end-to-end manner, we are able to exploit 2D pre-trained knowledge for point cloud analysis from two perspectives. In the forward process, the point clouds are projected into images with preserved geometry information and the resulting images can be recognized and handled by the pre-trained image model. In the backward optimization, the frozen pre-trained weights of the image model act as an anchor and guide the learnable coloring module to learn extra color knowledge for colorless point clouds, without explicit manual interference and only under the indirect supervision from the overall target functions of downstream tasks. Therefore, the resulting colorful images expected mimic patterns in 21) images and to be distinguishable for the pre-trained image model in downstream tasks.
3.2 Point Cloud Feature Encoding
One of the most significant advantages of 3D point clouds over 2D images is that point clouds contain more spatial and geometric information that compressed or even lost in flat images. Therefore, we first extract geometry features from point clouds for better spatial comprehension, implementing lightweight DGCNN [69] to extract local features each point.
Given an input point cloud P Nx3 with N points, we first locate k-nearest neighbors N E N xxxx3 of each point. Then for each local region, we implement small neural network he to encode the relative position relations between the central point p; and the local neighbor points Np.- Then we can obtain geometric features F = [fi, 0 < i < N} E NxC. with dimension C: fs maxpooly,, (he (concatA,, (x,, x,))),
(1)
where x,, Xj coordinates of pt, P. respectively, maxpool N, and concaty, stand for max-pooling and concatenation within all points p, in local neighbor region Np, respectively.
3.3 Point-to-Pixel Prompting
Following the principle of prompt tuning mechanism introduced in Section 2.2, we propose Point- to-Pi Prompting to adapt point cloud analysis to image representation learning, on which the image model is initially pre-trained. As illustrated in Figure 2, we first introduce geometry-presert projection to transform the 3D point cloud into 2D images, rearranging 3D geometric features according to the projection correspondences. Then we propose geometry-aware coloring module to dye projected images, transferring 2D color knowledge in the pre-trained image model to the colorless point cloud and obtaining more distinguishable images that can better recognized by the pre-trained image model.

---

3.3.1 Geometry-Presers Projection
Once obtaining geometric features F' E RRN of the input point cloud P. we further rearrange them into an image-style layout F "xx to prepare for producing colorful image I, where H, W are height and width of the target image. We elaborately design a geometry preetervdd projection to avoid information loss when casting 3D point clouds to 2D images.
The first step is to find spatial correspondence between point coordinates X R.Nx3 and image pixel Since there is a dimensional diminishing during the projection process,
coordinates Y E RN*2 x we randomly select a projection view during training to construct a stereoscopic space with flat image components. Equivalently, we rotate the input point cloud with rotation matrix R E Rax3 to get 3D coordinates X after rotation: X = X XR The rotation matrix R is constructed through two steps: first rotating around the axis upp = (0, 0, 1) by angle 0, then rotating around the axis M6 (sin 0, cos 0, 0) by angle o, where A E [-x, T and & E |-</ /2, //2] are random rotation angles during training and fix-selected angles during inference. Then we just omit the final dimension X and evenly split the first two dimensions into 2D grids: yidd = [ ga], where 0 S i < N denotes point index, d = 0, I denotes coordinate dimension, Ia denotes grid size at dimension d.
The second step is to rearrange per-point geometric features F into per-pixel F according to coordi- nates correspondence between X and Y. If there are multiple points Sp,, = {p,} } falling in the same pixel at (h, w), which is a common situation, we add the features of these points altogether to produce the pixel-Ievel feature: fu..- = Epses. ES., fz. The summation operation brings two advantages related geometry-preservedd design. On the one hand, we consider all points in one pixel instead of keeping the foremost point according to depth and occlusion relation. Therefore, we are able to represent and optimize all points in one image and produce images containing semitransparent objects with richer geometric information as shown in Figure 1. On the other hand, we conduct summation operation instead of taking the average, resulting larger feature values when there are more points in one pixel. Such design maintains the spatial density information of point clouds during the projection process, which lacked in image representations and is critical in preserving geometry knowledge.
In conclusion, the contain plentiful spatial knowledge of the object. Note that we only use one projection view during training and do not explicitly design any aggregation functions for multi-view feature fusion. Therefore, we follow a more efficient single-view projection pipeline than its multi-view counterpart.
projection produces geometry-aware image features that
3.3.2 Geometry-Aware Coloring
Despite that 3D point cloud contains richer geometric knowledge than 2D images, colorful pictures embrace more texture and color information than colorless point clouds, which is also decisive in visual comprchension. The frozen image model pre-trained abundant images learns to perceive the visual world not only based on object shape and outlines, but also heavily relied on discriminative colors and textures. Therefore, the image feature map F that contains only geometric knowledge and lacks color information is not most suitable for the pre-trained image model understand and process. In order to better leverage pre-trained 2D knowledge of the frozen image model, we propose to predict colors for each pixel, explicitly encouraging the network to migrate color knowledge in the pre-trained image model F via the end-to-end optimization. Since the input F contains rich geometry information that will heavily affect the coloring process, the resulting images are expected to display different colors on different geometry parts, which has been verified in Figure 1.
More specifically, we design a lightweight 2D neural network ge to predict RGB colors C = (cH.w) e I H x W x3 for each pixel (h, w): Ch 9 (fr,)) We implement several 3 x 3 convolutions in go for image smoothing, as the initial projected image feature í are relatively discontinuous due the sparsity of the original point cloud. Therefore, the smoothing operation is critical in producing more realistic images that the pre- trentrained image model can recognize. The resulting colorful images are then prepared for further image-level feature extraction through the pre-trained image model.
3.4 Optimization on Downstream Tasks
Take ViT as the pre-trained image model for example. The outputs from the pre-trained image model image token features F NcC, and one class token feature fels E R IIx. where N₂ is the
5

---

number of image patches and C₁ is the token feature dimension. For different downstream tasks, we design different task-specific heads and optimization strategies.
Object Classification For object classification, we follow the common protocol in image Trans- former models utilize the class token fels as the imput to the classifier CLS implemented only one linear layer: p = softmax(CLS(fa..
We use the CrossEntropy loss as the optimization target.
Part Segmentation For part segmentation, we rearrange the token features F into image layouts and upsample them to H x W. Then we design lightweight 2D segmentation head SEG based on Seman- ticFPN [26] or UPerNet [75] to predict per-pixel segmentation logits: Ph.w
(oftmax(SSEGG,, )). Given that multiple points may correspond to one pixel and that we train the network in single view pattern, projecting per-pixel predictions back to 3D points will cause supervision conflict. Instead, we project 3D labels into 2D image-style labels, exactly as how the point cloud is projected. Then we implement& a per-pixel multi-label CE loss as there may be points from multiple classes projected to the same pixel: Laeg = Ei, Σx -Uh, log wh. The values of multi-hot 2D label y assigned according to projection correspondences, satisfying En Uh, domain speeds up the training procedure without much information Ioss, since we keep all features of points in one pixel and the optimization target is accordingly based on their category distributions. During inference, we select multiple projection views and re-project 2D per-pixel segmentation results back to 3D points, fusing multi-view predictions. Therefore, the per-point segmentation is decided by the most evident predictions from the most distinguishable projection directions.
Supervision in 2D)
4 Experiments
4.1 Datasets and Experiment Settings
Datasets. We conduct classification on ModelNet40 [74] and ScanObjectNN [65], while ShapeNet- Part [65] utilized for part segmentation. ModelNet40 is a synthetic 3D dataset containing 12,31 CAD models from 40 categories. ScanObjectNN samples from real-world scans with background and occlusions. It contains 2,902 samples from 15 categories, and we conduct experiments on the perturbed (PB-T50-RS) variant. ShapeNetPart samples 16,881 objects covering 16 shape categories from the synthetic ShapeNet and annotates each object with part-level labels from 50 classes.
Implementation Details. We utilize Adam W [40] optimizer and ConialAnnealing scheduler [39]. with learning rate 54 and weight decay -2- We freeze the weights of the pre-trained image model except for normalization layers. The model is trained for 300 epochs with batch size of 64. During training, the rotation angle 8 é are randomly selected from [-T,, x] and [-0.44, 00.2 to keep the objects standing upright. During inference, we evenly choose 10 values of and values of é to produce 40 views for majority voting. Please refer to the supplementary for architectural details.
4.2 Object Classification
4.2.1 Results
Main Results. We implement our P2P framework with different image models of different scales, ranging from convolution-based ResNet [22] and ConvNeXt [38] to attention-basee Vision Trans- former [15] and Swin Transformer [36]. These image models are pre-trained on ImageNet-Ik [13] with supervised classification. We report the image classification performance of the original image model, the number of trainable parameters after Point-to- Pixel Prompting, and the classification accuracy on ModelNet40 and ScanObjectNN datasets, as shown in Table
From the quantitative results and accuracy curve, we can conclude that enlarging the scale of the same image model will result in higher classification performance, which is consistent with the observations in image classification. Therefore, our proposed P2P prompting can benefit 3D domain tasks by leveraging the prosperous development of 2D visual domain, including abundant training data, various pre-training strategies and superior fundamental architectures.
Comparisons with Previous Methods. Comparisons with previous methods on the ModelNet40 and ScanobjectNN are shown in Table 2. For baseline comparisons, select methods [49, 64, 69, 42,

---

Table Classification results on ModelNet40 ScanObjeetNN, For different image models, we report image classification performance (IN Acc.) on ImageNet- [13] dataset. After migrating them to point cloud analysis with Point-to-Pixx Prompting, we report the number of trainable parameters (Tr. Param.), performance on ModelNet40 datasct (MN Acc.) and performance ScanObjeetNN dataset (SN Acc.).
(a) ResNet [22].
(e) Accuracy on point cloud classification datasets vs. ImageNet-val for different mod- els.
Image Model
IN Acc.
Tr. Param.
MN Acc.
Acc.
ResNet-18
69.8
109 K
91.6
82.6
ResNet-50
76.1
206 K
92.5
85.8
ResNet-101
77.4
257 K
93.1
87.4
ModelNetdo Top-1 Acc (%)
DonnNeX
RecNet
R-101
C4
(b) Vision Transformer [15].
C5
GB
Image Model
IN Acc.
Tr. Param.
MN Acc.
SN Acc.
R-50
VIFT
72.2
99 K
91.5
79.7
VIT-S ViT-B
79.8
116
91.8
81.6
s2-
V-5
81.8
150 K
92.7
83.4
(c) Swin Transformer [37].
70
75
Image Model
IN Acc.
Tr. Param.
MN
Acc.
Swin-T
81.3
136 K
92.1
82.9
ScanObjectNN Top-1 1 Acc (96
VIT
ConvNeX:
B-101
Swin-S
83.0
154 K
92.5
83.8
Swin-B
83,5
178 K
92.6
84.6
Swin
R-50
CL
(d) ConvNeXt [38].
V.8
Image Model
IN Ace.
Tr. Param.
MN Acc.
SN Acc.
R-16
ConvNeXt-T 82.1
126
92.6
84.9
V-5
ConvNeXt-S
83.1
140
92.8
85.3
ConvNeXt-B 83.8
159
93.0
85.7
Con
198
93.2
86.2
ImageNet Top-1 1 Acc (%6
19, 54, 50] that focus on developing 3D architectures and do not involve any pre-training strategics.
We also select traditional pre-training work [81, 68, 44] in 3D domain. For our P2P framework. we show results of two versions: (1) baseline version with ResNet- 101 as the image model, (2) advanced version with HorNet-L [55] pre-trained on ImageNet-225 dataset [13] the image model, additionally replacing the linear head with a multi-layer perceptron (MLP) as the classifier.
From the results we can draw three conclusions. Firstly, P2P outperforms traditional 3D pre-training methods. This suggests that the pre-trained knowledge from 2D domain is useful for solving 3D recognition problems and is better than directly pre-training on 3D datasets with limited data. Secondly, we achieve saaecof-thheaa performance on ScanObjeetNN, Therefore, our P2P framework fully exploits the potential of pre- pre-training knowledge from image domain and opens for point cloud analysis. Finally, P2P performs relatively better on real-world ScanObjeetNN than synthetic ModelNet. This may be caused by the data distribution of ScanObjectNN being more similar to the pre-tree trained ImageNet dataset, as they both contain visualizations of objects from the natural world. This prosperity reveals the potential of P2P in real-world applications.
avenue
Visualization Analysis. The visualizations of our projected colorful images are shown in Figure 1. The first line shows point cloud samples, the second and third lines illustrate the colorful images from different projection views. Our geometry-preserver projection design maintains most spatial information, resulting in images of semitransparent objects that avoid occlusion problems, such as the chair leg in the second row 5th column.
4.2.2 Ablation Studies
To investigate the architecture design and training strategy of our proposed framework, we conduct extensive ablation studies on ModelNet40 classification. Except for further notice, we use the base version of Vision Transformer (ViT-B- 1k) that is pre-trained on ImageNet- dataset as our image model. Illustrations of our ablation settings can be found in Figure 3.
7

---

Table Comparisons on classification accuracy (Acc.) with previous literature on point cloud datasets. We report the pre-training modality (Pre-train) and trainable parameters number (Tr. Param.) of each method.
(a) ModeINet40..
(b) ScanOhjectNN,
Method
Pre-train
Tr. Param.
Acc. (+)
Method
Pre-train
Tr. Parum.
Acc.(%
PointNet++ [49]
N/A
1.4 M
90.7
PointNet++ [49]
N/A
1.4 M
77.9
KPCunv [64]
N/A
15.2
92.9
DGCNN [69]
1,8 M
78.1
DGCNN [69]
N/A
1.8
92.9
PRANet [12]
N/A
2.3 M
82.1
PoimiML.-elit [42]
N/A
0.68 M
93.6
MVTN [19]
N/A
14.0 M
82.8
PointNeXi [50]
N/A
1.4 M
94.0
PointMLP--llic [42]
N/A
0.68 M
83.8
PointMLPP [42]
N/A
PointML. [42]
RepSurf-U [54]
N/A
1,5 M
94.7
RepSurf-U(2x) [54]
N/A
6.8 M
86.1
DGCNN-OcCo [68]
3D>
188
93.0
PointNeXi [50]
N/A
1,4 M
88.2
Point-BER2 [81]
3D
21.1 M
93.2
Point-BERR' [81]
3D
21.1 M
83.1
Point-MAE [44]
3D
21.1
93.8
Point-MAE [44]
3D
21.1 M
85.2
P2P (ResNet-101)
2D
0,25
93.1
P2P (ResNet-101)
2D
0,25 M
87.4
P2P (HoxNet-L-22Xkmlpp)
1.2
94.0
P2P (HorNet-L-22k-mlp)
1.2
89.3
Advantages P2P Prompting over Other Tuning Methods. We conduct extensive ablation stud- ies to demonstrate the advantages of our proposed P2P Prompting over vanilla fine-tuning and other prompting methods, shown in Table 3a. As a baseline (Model A), we directly append classification head to the geometry encoder without the pre-trained image model. Then we incrementally insert pre-trained ViT blocks to process point tokens from the geometry encoder, and discuss different fine- tuning strategies including fixing all ViT weights (Model B1), fine-tuning normalization parameters (Model B2) and fine-tuning all ViT Woights(Model B3). We also implement Vision Prompt Tuning (VPT) [24] to Model B with shallow (Model C₁) and deep (Model C2) variants.
From the comparisons between Model A and others, we can inspect the contribution of pre-trained knowledge from 2D to 3D classification. However, neither vanilla fine-tuning nor previously prompt- ing mechanism VPT fully exploits the pre-trained image knowledge. Our Point-to-Pixel prompting is the best choice to migrate 2D pre-trained knowledge 3D domain at low trainable parameter cost.
Point-to-Pixel Prompting Designs. After confirming that P2P is the most suitable tuning mech- anism, we discuss the design choices of the P2P module in detail. In Point-to-Pixel Prompting, we produce colorful images to adapt to the pre-trained image model, whose advantages have been discussed in Section 3.3.2. Here we further prove the statement via ablation studies in Table 3b. Model D processes per-pixel feature from Section 3.3.1 to directly generate image tokens and feed them to ViT blocks. In this variant, we bypass the explicit image generation process and directly adopt patch embedding layers on feature map F. Model E generates binary black-and images according to the geometric projection from the point cloud, without predicting pixel colors as in P2P.
According to the results, Model D introduces much more trainable parameters due to the trainable patch embedding projection convolution layer with kernel size 16, while producing inferior classifica- tion results than P2P. On the other hand, even though Model E requires fewer trainable parameters, its performance lags far behind. Therefore, producing colorful images as the prompting mechanism can best communicate knowledge between the image domain and point cloud domain, fully exploiting pre-trained image knowledge from the frozen VIT model.
Influence of Tuning Strategies. After fixing the architecture of our P2P framework, we investigate
the best tuning strategy, adjusting the tuning extent of the pre-trained image model: (1) Model training the image model from scratch without loading pre-trained weights. (2) Model G: tuning all ViT parameters. (3) P2P: tuning only normalization parameters. (4) Model H: tuning only bias parameters. (5) Model I: fix all ViT parameters without any tuning.
According to the results in Table 3c, tuning normalization parameters the most suitable solution, avoiding 2D information lost during massive tuning (model G). Tuning normalization parameters also adapts the model to point cloud data distribution, which model H and I variant fail to accomplish. Additionally, quantitative comparisons between Model F and others demonstrate that the pre-trained knowledge from 2D domain crucial in our P2P framework, since the limited data in 3D domain insufficient for optimizing large ViT model from scratch with numerous trainable parameters.
8

---

[P22]
[A]
[B]]
[C]
[P2P]
[8]
[D]
Inpat
Input
Input
Input
Geometry
Goometry
Geometry
Geometry Encader
Geometry
Geometry Eneoder
Grooeery
Projection
Projection
Encoder
Encoder
Projection
Coloring
Masking
P2P
CLS Head
Image Model
Image Model
VPT
Colorful images
Binary images
Image Feats Map
Image Model
Ouiput
CLS Heud
CLS Head
(b) Point-to-Pixel Prompting
CLS Head
Output
Output
[1]
[P2P]
[H]
[G]
[F]
(a) Fine Fine-tuning Prompting
Fix
Tune Norm
Tune
Tune
From
Output
All
Bias
All
Scratch
(+)
(c) Tuning Settings
Figure 3: Ablations illustration. (+) shows the pipeline overall P2P framework. Part (a) displays ablations on replacing P2P prompting with vamilla fine-tuning or visual prompt tuning (VPT) [24], Part (b) illustrates ablations on Point-to- tei-tto- Prompting designs. Part (c) shows different tuning strategies on the pre-trained image model in our P2P framework. Gray letters top of each model correspond to the Model column in Table 3.
Table 3: Ablation studies ModelNet40 classification. We select ViT-B that is pre-trained ImageNet- Ik our image model. We report trainable parameters (Tr. Param.) accuracy (Acc.). (a) shows effects of different tuning strategies, including woint-based network without the image model (A), fine soee-tuning the pre-trained image model to diffcrent extents (B,, (B,,Bg.Bg) prompt tuning the pre-trained image model with different variants of VPT (C1,C2) and with proposed Point-to-Pixee prompting (P2P). (b) shows different Point-to- to-Pixel Prompting types, discussing whether explicitly produce images (D) and whether predict pixel colors (E). (c) shows ablations on tuning settings
pre-trained image model when training our P2P framework. (d) shows effect of different pre-training strategies of the image model, where IN Acc. with and represent the linear probing and fine-tuning accuracy ImageNet-11 dataset respectively. denotes that implement CoOp to report the shot classification accuracy of the CLIP pre-trained model on ImageNet- Ik. Illustrations of
ablations (a,b.c) are shown Figure
(a) Fine-tuning and Prompting Methods.
(c) Tuning settings.
Model|
Image Model
VPT
P2P
Tr. Param.
Acc.(%)
Model Pre-train Tune Param.
Tr. Param.
Acc.(Sk
A
X
x
7.76
88.5 (-4.2)
All
81.9 M
86.3 (-6.4)
G
All
81,9 M
91.7 (-1.0)
Bg
Fixed
0.08
90.0
H
Bias
0.21 M
92.3 (-0.4)
B2
Finetune Norm
X
x
0.12
90.1 (-2.6)
N/A
0.11
92.2 (-0.5)
Bg
Finetune All
x
x
81.2 M
90.6 (-2.1)
P2P
Norm
0.15 M
92.7
C:
Prompt Tune
Shallow
x
0.31 M
90.2 (-2.5)
C2
Prompt Tune
Deep
x
0,50 M
90.0 (-2.7)
(d) Different Pre-training Strategies.
P2P
Prompi Tune
x
0.15 M
92.7
Model Pre-train
IN Acc.
Param.
Acc.(%)
(b)
Point-to-Pixe Prompting.
MAE
68.01
0.15 M
91.6
Model
P2P Type
Color
Tr. Param.
Acc.(%)
CLIP
71.7%
MoCo
76.71
0.15
92.3
D
Feature
x
12.1 M
90.8 (-1.9)
DINO
78.21
0.15
92.8
Image
x
0.07 M
89.8 (-2.9)
1k
81.8
0.15 M
92.7
Image
0.15
92.7
22k
84.01
0.15 M
92.9
Effects of Different Pre-training Strategies. In Table 3d, we show the effects of different strategies for pre-training image models. For supervised pre-training, we load pre-trained weights on ImageNet- Ik and ImageNet-22kk datasets. For unsupervised pre-training,
select four most representative methods: CLIP [51]. DINO [S], MoCo [11] and MAE [20]. We report the linear probing and fine-tuning results ImageNet-Ik dataset of each pre-training strategy in IN Acc. column with and respectively. Note that we implement CoOp [86] to report the zero-shot classification accuracy (denoting with *) of the CLIP pre-trained model.
From the experiment results, we can conclude that supervised pre-trained image models obtain relatively better results than unsupervised pre-trained ones. This may because the objective of 3D classification consistent with that in 2D domain, thus the supervised pre-training weight is more suitable to migrate to point cloud classification task. However, unsupervised approach with strong

---

Table 4: Part segmentation results the ShapeNetPart dataset. We report the mean IoU across all part categories mloUc (%) and the mean loU across all instance mloU, (%) and the IoU (%) for each category.
Model
mfoUc m
acro plane
bag
chuir
car phone
guitar knife lamp laptop
motor bike
mug pistol rocket
skate board
tuble
PointNet [48]
80.4
83.7
83.4
78.7 82.5
74.9
89.6
73.0
91.5
85.9
80.8
95.3
65.2
93.0
81.2
57.9
72.8
80.6
PointNet++ [49]
81.9
85.1
82.4
79.0
87.7 77.3
90.8
71.8
91.0
85.9
83.7
95.3
71.6
94.
81.3
58.7
76.4
82.6
DGCNN [69]
84.0
90.6
74.7
91.2
87.5
82.8
95.7
66.3
81.1
74.5
82.6
Point-BERT [8]]
84,1
85.6
84,3
84,8
88.0
79.8
91.0
81.7
91.6
87.9
85.2
95.6
75.6
94.7
84,3
63,4
76,3
81,5
PointMI.P _P [42]
84.6
86.1
83.5
83.4
87.5
80.5
90.3
78.2
92.2
88.1
82.6
96.2
77.5
95.8
85,4
64.6
83.3
84.3
KPConv [6:]]
85.1
86.4
84.6
86.3
87.2
77.8
88.4
82.7
96.2
78.1
85.4
69.0
82.0
83.6
P2P CON-B-SHPPN)
82.5
85.7
83.2
84,1
85,9
78.0
91.0
80.2
91.7
87.2
85.4
95.4
69.6
93.5
79.4
57.0
73.0
83.6
P2P (CN-L-UPa)
84.1
86.5
84.3
85,1
88.3
80.4
91.6
80.8
92.1
87.9
85.6
95.9
76.1
94.2
82.4
62.7
74.7
83.7
transferability such as DINO also achieves competitive performance. Secondly, comparing among unsupervised pre-training methods, the one that achieves higher performance with linear probing on 2D classification produces better results in 3D classification. This suggests that the transferability of a pre-trained image model is consistent when migrating to 2D and 3D downstream tasks.
4.3 Part Segmentation
The quantitative part segmentation results on ShapeNetPart dataset are shown in Table 4. We imple- ment the base version of ConvNeXt [38] as image model and SemanticFPN [26] as 2D segmentation head for baseline comparison. We further implement the large version of ConvNeXt as the image model and more complex UPerNet [75] as 2D segmentation head to obtain better results. Our P2P framework can achieve better performance than classical point-based methods, which demonstrates its potential in performing 3D dense prediction tasks based on 2D pre-traines rreitriied image models. We leave it for future work to develop advanced segmentation heads and supervision strategies to better leverage pre-traine pre-trained 2D knowledge in object-level or even scene-level point cloud segmentation.
4.4 Limitations
While P2P shows outstanding classification performance and a promising scaling-up trend, we think that P2P may have difficulty in performing 3DD tasks that concentrates on modality-dependent geometry analysis like completion, reconstruction, or upsampling. This is because P2P exploits and transfers the shared visual semantic knowledge between 2D and 3D domains, but these low-level tasks focus more on 3D domain-specific information. Apart from that, even though our P2P framework only requires a few trainable parameters to leverage pre-trained 2D knowledge and obtain high performance, its overall training parameters and FLOPs are still large when the image model is large. We will investigate these problems in future works.
5 Conclusion
In this paper, we propose a point-to-pint prompting method to tune pre-trained image models for point cloud analysis. The pre-trained knowledge in image domain can be efficaciously adapted to 3D tasks at a low trainable parameter cost and achieve competitive performance compared with state-of-the-art point-based methods, mitigating the data-starvation problem in point cloud field that has been an obstacle for massive 3D pre-training researches. The proposed Point-to-Pixx Prompting builds bridge between 2D and 3D domains, preserving the geometry information of point clouds in projected images while transferring color information from the pre-trained image model back the colorless point cloud. Experimental results object classification and part segmentation demonstrate the superiority and potential of our proposed P2P framework.
Acknowledgments
This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700802, in part by the National Natural Science Foundation of China under Grant 62125603 and Grant U1813218, in part by a grant from the Beijing Academy of Artificial Intelligence (BAAI).
10

---

References
[1] Idan Achituve, Haggai Maron, and Gal Chechik. Self-supervised learning for domain adaptation on point clouds. In WACV, 2021. 3
[2] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers, arXiv
preprint arXiv:2106. 08254, 2021. 3,
[3] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. NewrIPS, 2019. 3
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 2020. 3,
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[6] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017. 3
[7] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1512
2015.
[8] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages 9044-9004 9053,9053, 2021. 2
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML. PMLR, 2020. 1,
[10] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422
2020. 1,3
[11] Xinlei Chen Saining Xic*, and Kaiming Hc. An empirical study of training self-supervised
vision transformers. arXiv preprint arXiv:2104 02057, 2021. 9
[12] Silin Cheng, Xiwu Chen, Xinwei Hc, Zhe Liu, and Xiang Bai. Pra-net: Point relation-aware
network for 3d point cloud analysis. TIP, 2021. 8,8
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255. Icee, 2009. 6,7
[14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1XX::18100.08055 2018.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
3, 3,4.6 6.7
[16] Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao. Gvenn: Group-view
convolutional neural networks for 3d shape recognition. In CVPR, 2018. 3
[17] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 2020. 3
[18] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 2021.
[19] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transformation
network for 3d shape recognition. In ICCV, 2021. 3, 7,8
[20] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000-16009, 2022. 3,
[21] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 1,

---

[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 3. 6,
[23] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017. 3
[24] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.121199 2022. 3, 8, 9
[25] Asako Kanezaki, Yasuyuki Matsushita, and Yoshifumi Nishida. Rotationnet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints. In CVPR, 2018. 3
[26] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019. 6, 10
[27] Roman Klokov and Victor Lempitsky. Escape from cells: Deep kd-networks for the recognition of 3d point cloud models. In ICCV, 2017.
[28] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2019.
[29] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. arXiv preprint arXiv:2104 0.0606 2021. 1,3
[30] Ruoyu Li, Sheng Wang, Feiyun Zhu, and Junzhou Huang. Adaptive graph convolutional neural networks. In AAAI, 2018. 3
[31] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 2021.
[32] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x-transformed points. NeurlPS, 2018. 3
[33] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhenghao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13388 2021. 3
[34] Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2 10.07660,, 2021. 3
[35] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103 103.885, 2021. 1,3
[36] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint arXiv:2111.09800
2021. 6
[37] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 7.7
[38] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining
Xie. A convnet for the 2020s. arXiv preprint 2XXiv:2201.035455 2022. 6, 7, 10
[39] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv
preprint arXiv: 03983, 2016. 6
[40] llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6
[41] Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang, Yuecheng Li, Fernando De La Torre, and Yaser Sheikh. Pixel codec avatars. In Proceedings of the IEEF/VVF Conference on Computer Vision and Pattern Recognition, pages 64-73, 2021. 2
[42] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual mlp framework. In ICLR, 2022. 3, 7, 8, 10
[43] Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for
real-time object recognition. In IROS. IEEE, 2015. 3
[44] Yatian Pang, Wenxiao Wang, Francis F. H. Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In ECCV, 2022. 7,
12

---

[45] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint 1909v:990.010000 2019. 1,3
[46] Hieu Pham, Zihang Dai, Qizhe Xie, and Quoe V Le. Meta pseudo labels. In CVPR, pages 11557-11568, 2021.
[47] Omid Poursaeed, Tianxing Jiang, Han Qiao, Nayun Xu, and Vladimir G Kim. Self-supervised learning of point clouds via orientation estimation. In 3DV. IEEE, 2020. 3
[48] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017. 3, 10
[49]
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NeurIPS, 2017. 3, 7, 8, 10
[50] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Abed Al Kader Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. arXiv preprint arXiv:2206. 04670, 2022. 7
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 1,9
[52] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 2019. 1
[53] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. sero-shot text-to-image generation. In ICML, 2021.
[54] Haoxi Ran, Jun Liu, and Chengjie Wang. Surface representation for point clouds. In CVPR, 2022. 7,8
[55] Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser-Lam Lim, and Jiwen Lu. Hornet: Efficient high-order spatial interactions with recursive gated convolutions. arXiv preprint arXiv: 14284 2022. 7
[56] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representa- tions at high resolutions. In CVPR, 2017.
[57] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. NeurlPS, 2021.
[58] Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by recon- structing space. NeurlPS, 2019.
[59] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprini arXiv:2010. 55800, 2020. 1,
[60] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. Using deepspeed and megatron to train megatron-turee nlg 530b, a large-scale generative language
model. arXiv preprint arXiv:2201.11990 2022. 1
[61] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In ICCV, 2015. 3
[62] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017.
[63] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. NeurIPS, 2017. 3
[64] Hugues Thomas, Charles R Qi, -aannnmmnuull Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In ICCV, 2019. 3, 7, 8, 10
[65] Mikaela Angelina Uy. Quang- Pham, Binh-Son Hua, Thanh Nguyen, and Sii-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In ICCV, 2019. 2, 6
13

---

[66]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
[67]
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nip. arXiv preprint arXiv: /et:: 07/00022255 2019. 1,3
[68] Hanchen Wang. Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point
cloud pre-training via occlusion completion. In ICCV, 2021. 2, 3, 7
[69] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M
Solomon. Dynamic graph cnn for learning point clouds. ToG, 2019. 3, 4, 7, 8, 10
[70] Zifeng Wang. Zizhao Zhang, Chen- Yu Lce, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. arXiv preprint arXiv:21 1.. 12.86554, 2021. 3
[71] Shih-En Wei, Jason Saragih, Tomas Simon, Adam W Harley, Stephen Lombardi, Michal Perdoch, Alexander Hypes, Dawei Wang, Hernan Badino, and Yaser Sheikh. Vr facial animation via multiview image translation. ACM Transactions Graphics (TOG), 38(4):1-16, 2019. 2
[72] Xin Wei, Ruixuan Yu, and Jian Sun. View-gen: View-based graph convolutional network for 3d shape analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[73] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks 3d point clouds. In CVPR, 2019. 3
[74] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR, 2015.
[75] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. ECCV, 2018. 6, 10
[76] Qizhe Xie, Minh-Thang Than-Thagg Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In CVPR, 2020. 3
[77] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. ECCV, 2020. 2, 3
[78] Wei Yang. Chris Paxton, Maya Cakmak, and Dieter Fox. Human grasp classification for reactive human-to-robot handovers. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1 111123111333 IEEE, 2020. 2
[79] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ToG, 2016. 2
[80] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR, 2021. 2
[81] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In CVPR, 2022. 2, 3, 7, 10
[82] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. CoRR, 2021. 3
[83] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2ae: Multi-scale masked autoencoders for hierarchical point cloud pre-training, arXiv preprint 2rXiv:2205.144p0 2022. 3
[84] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. CVPR, pages 5552-8562.
2022. 3
[85] In ICCV, 2021. 3
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer.
[86] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vasion-language models. arXiv preprint 2rx.v22109.01134 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.3.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] There isn't any potential negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Section 4. and supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1 and supplemental material.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [N/A] We follow the common practice of the previous paper, where they didn't report the error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See supplemental material.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.1.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
We doesn't 't use any new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "YqQZsJe3YaT", "reviewer": "Reviewer_xwQ5", "summary": "In this paper, the authors introduce point-to-pixel prompting (P2P), a learning framework for leveraging pre-trained image transformers for 3D tasks. The method is mainly motivated by the data scarcity issue in 3D domains. P2P learns a geometry-preserving transformation from point cloud to 2D grid, and then a projection to prepare the 2D grid data to be processed by a pre-trained image transformer expecting image tokens. The main benefit of P2P, to my understanding, is the ability to achieve comparable accuracy to other 3D models with much fewer parameters that need to be trained with 3D data. This is validated on two tasks, 3D object classification and 3D part segmentation.", "questions": "Instead of taking a sum of point features in each cell (L189), did you try max pooling? It would be great to see a sensitivity analysis comparing {max,mean,sum} pooling for this, since images of semi-transparent objects (which is unrealistic) would seemingly be out-of-distribution for pre-trained image models on ImageNet?\n\nCurrently, the strengths and weaknesses of the paper roughly balance each other out in my opinion and I believe the work as it stands is borderline. I would be interested to read the authors response to my feedback. Thanks!\n\n============================\nAfter reading the author's responses to my questions and concerns I have increased my score to reflect that I feel the strengths now outweigh the weaknesses.\n\n", "limitations": "No. Limitations of the P2P framework should be discussed in the main text (e.g., in the conclusions section). ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n-----------------\n- the question of whether knowledge can be transferred from large pre-trained image models for use with 3D domains is interesting\n- the point-to-pixel prompt pipeline, which is nicely visualized in Figure 2, appears to be novel and is simple and elegant\n- this work is a nice demonstration of ideas from NLP transferring successfully to other domains (in this case, to 3D point cloud processing)\n- the paper is well-written and easy to read\n\n\nWeaknesses\n\n----------------- \n- $\\text{\\textbf{Unclear motivation,problem, and significance}}$: The current set of claims in the introduction are that A) there is a data starvation problem in 3D domain (L34-35) and B) pre-training point cloud transformers suffers from an imbalance between the number of trainable parameters and limited training data, leading to insufficient optimization and overfitting (L40-41). However, the data starvation problem seems to only exist for specific object-centric datasets such as ShapeNet. By contrast, consider the large Scannet and Waymo datasets. Moreover, recent advances in 3D rendering (e.g., NeRF) suggests that highly lifelike synthetic 3D data may soon become available. Therefore, scarcity of large datasets does not appear to be a fundamental concern. Moreover, point B) seems plainly false since recent methods like Point-BERT work just as well as P2P on, e.g., ModelNet40.\n- As a result, it is unclear what the actual problem is that is being addressed here and *why* this prompting method is needed at all. The main benefit of P2P seems to be in the use of fewer model parameters, but its unclear why this is important.\n- $\\text{\\textbf{Multiple unsubstantiated claims}}$. These can be addressed with careful editing. \n  - (L54) “The end-to-end optimization pipeline and the strategy of freezing the pre-trained image model promote the *bidirectional* knowledge flow between points and pixels”. To my understanding, the flow is *unidirectional*; pre-trained image features are being used to learn a better representation for points.\n  - (L271) “Firstly, our P2P outperforms traditional 3D pretraining methods” (on ModelNet40). P2P’s largest model achieves the same performance as Point-BERT.\n  - Similarly, claims of “superiority” of P2P (L64, L370) are clearly not supported by the accuracy results in the experiments.\n\n\nReferences\n\n-----------------\n- Dai, Angela, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. \"Scannet: Richly-annotated 3d reconstructions of indoor scenes.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839. 2017. \n- Sun, Pei, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo et al. \"Scalability in perception for autonomous driving: Waymo open dataset.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2446-2454.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Nj0LjORO4JM", "reviewer": "Reviewer_RwM1", "summary": "This paper proposes a new model architecture for 3D problem which leverages the powerful backbones pretrained from the 2D task. The idea is straightforward. The input point cloud is projected into 2D pixels using an encoder model, then the 2D pixels is colored by a coloring module, and the colored images are fed into a pretrained ViT backbone and then predictions are made by the task-specific heads. The overall approach provides an elegant solution to leverage the representations of 2D models. The experimental results demonstrate superior performance on public benchmarks including ModelNet40, ShapeNetPart datasets.", "questions": "How much improvements are from the improved backbone architecture and added compute cost? It would be great to show an ablation study on that.", "limitations": "The proposed approach has some limitations on the feasible tasks to apply on. For example, it may not work if we want to conduct a 3D segmentation task. \nRelated to the question above, the comparison in Table 4 is not fair due to lack of model complexity analysis.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Novel model architectures to utilize pretrained 2d models. To my knowledge, the idea of using projection into 2D and coloring module is new. \n2. The idea is simple yet effective. The pretrained 2d models are easy to get and the results are promising.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Gt9U0hrzKne", "reviewer": "Reviewer_xwSJ", "summary": "The paper proposes point-to-pixel prompting to leverage 2D pre-trained models to help 3D point cloud recognition tasks. The main modules include a geometry-preserved projection and a geometry-aware coloring, which fill the gap between 3D point clouds and 2D images. The experiments on ModelNet40 and ScanObjectNN show that P2P achieves comparable performance on classification tasks with only a few trainable parameters.", "questions": "Please refer to weaknesses. Some more questions are listed below:\n1. For the projection from 3D to 2D, which are the pixel features for those pixels without points?\n2. How to obtain point predictions for part segmentation is unclear. It seems not very reasonable to simply add the multi-view predictions. Actually, the multi-view fusion is not clearly stated in the paper. \n3. How's the processing of the empty pixels in the coloring module? The visualization results look very clean and not very smooth actually, and I wonder if the empty pixels are filtered out in the coloring module. ", "limitations": "The limitation is discussed in Sec 4.3. ", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper first proposes a prompt-tuning method to adopt 2D pre-trained parameters in 3D, which is an interesting and novel exploration.\n2. With p2p prompting, the model can achieve competitive results on the shape classification task with much fewer trainable parameters.\n\nWeaknesses:\n1. Although the method leverages extra 2D image knowledge, it does not show clear performance or speed advantages over previous 3D networks on both classification and part segmentation. The parameters that need to be trained are fewer but the whole model is larger. The 2D prior knowledge is not fully exploited in this method. \n2. The design of simply adding the point features in the same pixel seems trivial, and even with the explanations in Line190-197, I don't really think it preserves geometry. Also, no more experiments are conducted to analyze these design choices. \n3. More results on scene-level point cloud understanding with datasets like ScanNet or S3DIS are expected to illustrate the effectiveness of the prompt-tuning pipeline. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Dyyhuw-Vbo", "reviewer": "Reviewer_2HDY", "summary": "In this paper, the authors propose to leverage the pretrained image model for point cloud downstream tasks. Specifically, they introduce a Point-to-Pixel Prompting to transform a point cloud as the corresponding image, by geometry-preserved projection and geometry-aware coloring.", "questions": "See Strengths And Weaknesses", "limitations": "See Strengths And Weaknesses", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1) The paper is well written with clear motivation and good organization.\n2) Leveraging 2D pretraining for 3D tasks is an interesting topic.\n3) Point-to-Pixel Prompting is novel.\n\nWeakness\n1) My main concern is the experiment result. Apparently, the proposed design does not improve the performance.\n2) What are the computation cost and model size for the prompting procedure? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept