Exploring Vision Semantic Prompt for Efficient Point Cloud Understanding
Authors: Yixin Zha
Chuxin Wang
Wenfei Yang
Tianzhu Zhang
Feng Wu
Venue: ICML 2025 poster (2025)
Keywords: point cloud understanding; point cloud analysis; 3D computer vision
Primary area: applications->computer_vision
TL;DR: We propose a new paradigm that,  for the first time, leverages 2D semantics as prompts to improve the generalization of pretrained 3D models with minimal trainable parameters.

Abstract: A series of pre-trained models have demonstrated promising results in point cloud understanding tasks and are widely applied to downstream tasks through fine-tuning. However, full fine-tuning leads to the forgetting of pretrained knowledge and substantial storage costs on edge devices. To address these issues, Parameter-Efficient Transfer Learning (PETL) methods have been proposed. According to our analysis, we find that existing 3D PETL methods cannot adequately align with semantic relationships of features required by downstream tasks, resulting in suboptimal performance. To ensure parameter efficiency while introducing rich semantic cues, we propose a novel fine-tuning paradigm for 3D pre-trained models. We utilize frozen 2D pre-trained models to provide vision semantic prompts and design a new Hybrid Attention Adapter to efficiently fuse 2D semantic cues into 3D representations with minimal trainable parameters(1.8M). Extensive experiments conducted on datasets including ScanObjectNN, ModelNet40, and ShapeNetPart demonstrate the effectiveness of our proposed paradigm. In particular, our method achieves 95.6% accuracy on ModelNet40 and attains 90.09% performance on the most challenging classification split ScanObjectNN(PB-T50-RS).

Exploring Vision Semantic Prompt for Efficient Point Cloud Understanding
Yixin Zha Chuxin Wang'
Wenfei Yang
12 Tianzhu Zhang' 12 Feng Wu 12
Abstract
in natural language processing (Devlin et al., 2018; Raffel et al., 2020; Achiam et al., 2023; Floridi & Chiriatti, 2020) and vision understanding (He et al., 2022; Radford et al., 2021; Oquab et al., 2023; Dehghani et al., 2023), similar methods of point cloud understanding have been proposed, such as PointBERT (Yu al., 2022), PointMAE (Pang et al., 2022) and PointGPT (Chen et al., 2024). These works utilize large amounts of unlabeled data to learn general represen- tations and apply them to downstream tasks through full fine-tuning, However, the full fine-tuning strategy faces two significant shortcomings: (1) Fine-tuning the entire model leads to the forgetting of pretrained knowledge; (2) Full fine-tuning imposes a significant storage burden.
A series of pretrained models have demonstrated promising results in point cloud understanding tasks and are widely applied to downstream tasks through fine-tuning. However, full fine-tuning leads to the forgetting of pretrained knowledge and substantial storage costs on edge devices. To address these issues, Parameter-Eiffcient Transfer Learning (PETL) methods have been proposed. According to our analysis, we find that existing 3D PETL methods cannot adequately align with semantic relationships of features required by downstream tasks, resulting in suboptimal perfor- mance. To ensure parameter efficiency while in- troducing rich semantic cues, we propose a novel fine-tuning paradigm for 3D pretrained models. We utilize frozen 2D pretrained models to pro- vide vision semantic prompts and design new Hybrid Attention Adapter to efficiently fuse 2D semantic cues into 3D representations with mini- mal trainable parameters(1 1.8M). Extensive exper- iments conducted on datasets including ScanOb- jectNN, ModeINet40,, and ShapeNetPart demon- strate the effectiveness of our proposed paradigm. In particular, our method achieves 95.6% accu- racy on NedelNet40 and attains 90.09% perfor- mance on the most challenging classification split Sea0OhjecNN(PB--55RRR))
To address the aforementioned constraints, series of Parameter-Efficient Transfer Learning (PETL) (Hu et al., 2021; Jia al., 2022; Chen et al., 2022) methods have been proposed. In point cloud understanding tasks, researchers have also proposed a series of PETL approaches, such as IDPT (Zha et al., 2023), DAPT (Zhou et al., 2024), and Point-PEFT (Tang et al., 2024). However, as shown in Fig- ure 1(b), performance of the 3D PETL method on complex tasks remains unsatisfactory. Upon analysis, we believe this limitation may stem from the differences feature re- quirements between point cloud pretraining tasks and down- stream tasks. As shown in Figure (a)), point cloud features of pretrained models are limited to local structural informa- tion and exhibit positional preferences. In contrast, features obtained through full fine-tuning contain rich semantic in- formation, with consistent representations for components of the same structure. However, in PETL methods, the main backbones are frozen, resulting in output features with limited semantic cues, which constrains networks general- ization ability in downstream tasks.
1. Introduction
With the growing of training data and model parameters, large foundation models have achieved success across vari- ous domains and tasks. Point clouds, as direct representa- tions of the real world, play crucial role in various fields (Li et al., 2024; Pan et al., 2024). Inspired by pretrained models
Compared to point clouds, 2D images contain richer seman- tic details. Currently, there are many existing 2D models pretrained on large-scale image datasets. These models not only learn rich semantic cues but also widely deployed edge devices. This raises intriguing question worth exploring: Can we leverage 2D semantics to enhance the performance of efficient 3D understanding? We believe that effectively integrating 2D semantic cues with 3D features could significantly improve model performances, even with minimal trainable parameters. To explore the feasibility of
Univeersity of Science and Technology of China, Hefei, China l "Deep Space Exploration Lab, Hefei, China. Correspondence to: Yixin Zha <zyxen @ maa...sttc.ed..c..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

666
29.3%
Layer
(b) Performences on ScanOhjectNN
Layor
Leca
Mani Pny
Layer
Pretrained
Full
PEFT
Ours
Audapa
The feature distributions across different layers of various methods.
(c) ambiguity between 3D and
Figure 1. (a) The feature colors are transformed into feature space using PCA, where the same color indicates feature consistency. Features red circles fail maintain consistent. (b) IDPT (Zha et al., 2023) underperform Full FT approaches on the most challenging real-world classification tasks. (c) The inherent limitations of the 2D perspective during 3D-to-2D projection introduce ambiguities between local and global structures.
this design, conduct analysis of technical challenges:
mation from of two modalities. On each scale, vision semantic prompts are generated by non-linear layer with 2D class tokens, and we employ HAA to achieve modality fusion. In HAA, prompts are passed through non-linear layers to generate two learnable parameters, a and J.. 3D features are modulated by these parameters to achieve Se- mantic Transfer (ST), which decouples semantic cues from global contextual features to mitigate local ambiguities. The modulated features then used queries and keys to compute the self-similarity. The unaltered 3D features serve as values, which are updated using the similarity matrixes acquired above. This approach enhances the semantic asso- ciations of 3D features while effectively filtering out redun- dant 2D noise, and the trainable parameters are kept at an extremely low level (1.8M).
Ambiguities in 3D-to-2D projection: The 2D pro- jections of point clouds from different viewpoints in- troduce local and global ambiguities. As shown in Figure I(c), two points that are far apart in 3D space may appear very close to each other on 2D planes from certain viewpoints, which causes local ambiguities. Additionally, viewpoint variations induce changes in the projected 2D geometry of 3D objects, resulting in global ambiguities due inconsistent representations across perspectives.
Multimodal Fusion in PETL: Traditional multi- modal approaches typically rely on learnable modality- specific feature extractors to achieve feature fusion across modalities. However, in PETL, most parameters of the feature extractors are frozen. Achieving effec- tive multimodal feature fusion with minimal tunable parameters presents a significant challenge.
In summary, our main contributions are as follows: (1) We propose a new paradigm that, for the first time, leverages 2D semantic cues to improve the generalization of pretrained 3D models with minimal trainable parametens.
(2) We utilize 2D class tokens at multiple scales to generate prompts, and we design a Hybrid Attention Adapter to adopt efficient modality fusion while keeping the trainable parameters at an extremely low level. (3) Extensive experiments on datasets such as ScanObjectNN, ModelNet40, and ShapeNetPart demonstrate the effectiveness of our proposed paradigm.
Based on our above analysis, we propose novel paradigm that leverages visual semantic prompts to improve the gener- alization of pretrained 3D models while keeping parameter efficiency. The new paradigm includes three new designs: 3D-to-2D Projection. Vision Semantic Prompt and Hybrid Attention Adapter (HAA). We first map point clouds into 2D depth maps from three orthogonal viewpoints to mitigate global ambiguities. Then, both point clouds and their corre- sponding depth maps are fed into the network. We adopt multi-scale semantic cues injection strategy, as shown Fig- ure 3, each layer consists of two parallel transformers with different modality weights, each transformer handles infor-
2. Related Work
Large-scale Pretrained Models: Large-scale pretrained models have demonstrated exceptional performance on downstream tasks across various domains, including nat- ural language processing (NLP) (Devlin et al., 2018; Raffel
2

---

666
al., 2020; Achiam al., 2023; Floridi & Chiriatti, 2020),
et al., 2024). However, in 3D PETL methods, the backbones are frozen, resulting in output features with limited semantic cues, which constrains the network's 's generalization ability in downstream tasks. We believe that the rich semantic cues in 2D pretrained models can effectively compensate for the missing semantic information in 3D PETL, and this area of exploration remains untapped.
21 vision (Radford et al., 2021) (Dehghani et al., 2023),
and point cloud understanding (Zhang et al., 2022). DI- NOv2 (Oquab et al., 2023) employed visual transformers to perform self-supervised pre-training, while MAE (He et al., 2022) achieved pre-training by randomly masking parts of an image and reconstructing the masked pixels using the remaining visible portions. Recently, many self-supervised pre-training methods for point clouds have been proposed. Existing methods primarily follow two research paths, con- trastive learning method (Xie et al., 2020; Zhang et al., 2021; Dong et al., 2023) and generative methods (Yu et al., 2022; Pang et al., 2022: Zhang et al., 2023b). Contrastive learning methods guide the model in learning discriminative features by distinguishing positive and negative samples. For example, PointContrast (Xie et al.. 2020) constrains the consistency between the same points in different views. CrossNet (Wu et al., 2023) conducts cross-moda contrastive learning between point clouds and their corresponding ren- dered images. Motivated by BERT (Devlin et al., 2018) and MAE (He et al., 2022), generative methods mainly adopt the Masked Point Modeling (MPM) to encourage models to infer the randomly masked regions with the visible re- gions, thereby guiding the model to learn the relationships between point cloud patches in the process, such Point- MAE (Pang et al., 2022), PointMamba (Liang et al., 2024) and PoimBERT (Yu et al., 2022) However, these point cloud pretrained models may forget pre-training knowledge after full fine-tuning, and impose a significant storage burden.
3. Method
We first introduce the transformer- paradigms of 3D pretrained models in Sec. 3.1. Next, we discuss the paradigms of fine-tuning in Sec. 3.2, including Parameter- Efficient Transfer Learning. Then we elaborate on the pro- cess of projecting 3D point clouds onto 2D planes in Sec. 3.3. Finally, we delve into the details of the multi-scale modality fusion framework in Sec. 3.4, including Tokenizer, Vision Semantic Prompt Generation and Hybrid Attention Adapter.
3.1. Transformer-based 3D Pretrained Model
In pretrained transformer- based point cloud models, point cloud P E RNx33 with N points is first divided into n point patches p E Raxkx3 via Farthest Point Sampling (FPS) and K-Nearest Neighborhood (KNN) algorithms, where each patch contains k local points. Then, all point patches will be embedded into token sequence T&D E R through mini-PointNet (Qi et al., 2017). The sequence is further processed by L-layer transformer blocks. After that, point tokens are updated through attention layers. Outputs of attention layers are passed through FeedForward Network (FFN) with residual connections to extract channel-wise information. The transformer block can be written as: f = AttenntinnL.N(TT- + T;-1), T, = FFN((N((.)) + +1,
Parfeeter-effecttive Transfer Learning: The pre-training and fine-tuning paradigm has demonstrated remarkable ef- fectiveness across a wide range of tasks. However, as model sizes grow exponentially, full fine-tuning the entire model can cause significant storage burdens. In contrast, Parameter- Efffiieer-Effiieent Transfer Learning (PETL) methods (Hu et al., 2021; Jia et al., 2022: Chen et al., 2022; Liu et al., 2023; Houlsby et al., 2019) update only small subset of the model's parameters while keeping the rest frozen. These approaches have demonstrated both effectiveness and efficiency across various widely-used pretrained models, including BERT (Devlin et al., 2018), GPT series (Achiam et al., 2023; Floridi & Chiriatti, 2020), VIT (Dosovitskiy et al., 2020), CLIP (Radford et al., 2021), and Stable Diffu- sion (Rombach et al., 2022). PETL methods can typically be divided into three main categories: prompt tuning (Jia et al., 2022; Yang et al., 2024), reparameterization (Hu et al., 2021), and adapters (Chen et al., 2022; Zhang et al., 2023a). These techniques adapt pretrained models to specific tasks by fine-tuning prompts, adjusting parameters without chang- ing the model architecture, or inserting lightweight trainable layers, respectively. Recently, PETL techniques have been introduced into the 3D domain, such as IDPT (Zha et al., 2023), DAPT (Zhou et al., 2024) and Point-PEFT (Tang
(1)
where T, is the output of i-th transformer block, LN is a Layer Normalization layer.
3.2. Fine- Fine-Tuning Paradigms
Full Fine-tuning: Full fine-tuning is the most commonly used fine-tuning paradigm. Assuming the training setup for the downstream task configured as I(x;; y) (i.e., x are training data and y are labels) and pretrained model weights as 0. After fine-tuning all trainable parameters O of pretrained models are updated to é. Specifically, the full fine-tuning paradigm can be formulated as:
i arg min é (F( 0),v),
(2)
0
where represents the loss function for downstream tasks. Each fine-tuning iteration requires storing the entire model's parameters. As the model size increases, this results in significant storage consumption.
3

---

666
era pose V, we aim to generate a 2D depth map DV (P) whose pixels P's geometry that is visible in V. With the extrinsic and intrinsic parameters of the pose V, we can obtain projective relationship between each 3D point and its corresponding 2D) coordinate(i.e. deitals in B.2). Each 3D point p(u, U, 2) E i's projected onto a projection plane, retrieving a 2D pixel location (ū, 6) and a depth & (a.k.a. distance from the projection). Projected 2D points p(á, 6) with depth é are used to generate rendered image. Let (x,y) v) be the coordinate of the rendered pixel, the generation process of whole 2D images DV can be formulated as:
Telces
Transformer Layer
Adppter
FTN
Wi
Adapter
Transforner Layer
Multi Head Amention
GELU
TayarNorm
Patch Embeedding
(a) Adapter tuning (AT)
(b) Details of AT
Figure 2. The illustration of Adapter Tuning: (a) The overview
of Adapter for trnasformer-haseee architecture. (b) The details of transformer block with Adapter.
DÝ (P) = max pEP
[ 2), rll2 x &neg. 0}.
(5)
2neg (a
min 3)/(mam e -iin $), PEP
Sffameter-Effiient Transfer Learning: PETL. methods offer an efficient approach adapting pretrained models for downstream tasks. Existing PETL methods focus tuning only a tiny subset of model weights or some lightweight additional parameters, we denote these tunable parameters as 0* The optimized parameters 6 can be represented as: Φ- arg min (F 0, 07, 9). 10-1 << |0]. (3)
pEp
PEP
The &neg is a negative point depth normalized within [0, 1]. In our model, to overcome the ambiguity caused by viewpoint limitations, we project point clouds into 2D depth maps from three orthogonal viewpoints and feed all of them into the network with corresponding point clouds.
3.4. Multi-Scale Modality Fusion
The shape of 0" is significantly smaller than pretrained model weights 6. During fine-tuming, the pretrained param- eters 0 remain fixed while only 0* are updated to 0*.
In this section, we provide a detailed introduction to our proposed paradigm, which, for the first time, achieves en- hanced 3D model performance by integrating 2D semantic cues with minimal trainable parameters.
There are two common PETL paradigms: Adapter Tuning (AT) (He et al., 2021; Zhang al., 2023a; Chen et al., 2022) and Prompt Tuning (PT) (Jia et al., 2022; Yang et al., 2024). Here, we briefly introduce the AT paradigm. As illustrated in Figure 2(a). AT incorporates a small number of parameters into the transformer architecture by introducing a lightweight bottleneck module. Specifically, as shown in Figure 2(b), consists of downward projection Wa to reduce the feature dimension, non-linear activation function o(-), and an upward projection Wi, to restore the features to their original dimension. During fine-tuning the TED are concatenated with a learnable class token and form Timput E R(*+1)xC_ Specifically, on the i-th layer, given input tokens T:-1 E ((*+))xC, the calculation process inside Adapter can be formulated as:
Given point clouds P, after 3D-to-2D projec- tion, we can acquire depth projections of P from three or- thogonal viewpoints. Point clouds and their corresponding depth maps are fed into the model simultaneously. The point cloud is processed into tokens TaD RRxC' through the patch embedding of the 3D pretrained model, while three depth maps are separately processed by the 2D pretrained model into r patch tokens follow the patch embedding of VIT (Dosovitskiy et al., 2020) and form I E R3xrx D, The T3D and T are each concatenated with learnable class to- kens. These combined tokens Timput E R(=+1)xC' and Impuut E Ra are then updated layer by layer using their modality-specinn transformers.
Tokenizer:
Vision Semantic Prompt Generation: each of the three depth maps independently performs self- attention computations, generating three separate 2D class tokens Idfe E R3xD This operation reduces computa- tional overhead and mitigates semantic ambiguities caused by viewpoint variations.
During fine-tuning,
Todo = (Wal T )))
(4)
where the 1: E R((++1xCC is the output of the attention module, and the Tado is the output of the adapter. Denote the Wa E RdxC' and the W., E Rxd the d <<< C. Besides, T, E R ([++])xC are fed into the FFN layer, and the output of the FFN are summed with Tada and T,, forming the final output of the transformer block T,
Taking one of the model layers as example, after obtain- ing the updated 2D class tokens Lota with the depth maps, we first apply max-pooling on three tokens to select sin- gle 2D class token idds E R1xD The iets is then passed through a shared Multi-Layer Perceptron (MLP) across all layers to map the high-dimensional 2D semantic features into the 3D semantic feature space. The whole process can
3.3. 3D-to-2D Projection
To enable 2D pretrained models to capture the semantic information of point clouds, we first map the point cloud into 2D depth maps. Given a point cloud P and a cam-
4

---

666
Downstream Task Head
Align.
HAA
Tadoo
Aro
Max ML.
3D Transformer
T
HAA
2D Transformer
BN-V
Process
ST
Arr
Tisues
&
W22
Wi
Generator
Tode
Self-Attention
3D Token
3D Transformer
HAA
2D Transformer
2D Class Token
Fuska
3D Embedding
2D Embedding
3D-to-2D Proj.
W₂
GELU
W2
Figure 3. Pipeline of our proposed framework. Each layer consists of two parallel transformers, each transformer handles information from one of two modalities. For i-th layer. employ HAA to achieve modality fusion. The Iel processed through pooling obtain ieta, and iete are updated by an MLP to acquire vision semantic prompts spro- Then, ipro are passed through two parallel non-linear layers to generate
learaalle parameters, and B3. These parameters fed into HAA and modulate 3D tokens 7; acquire T mizedd- Tiized are then used as queries and keys to compute the self-similarity. The unaltered T serve as values, which are updated using the calculated similarity matrix. We use the ipro of the scale perform classification task to align modality semantics.
be formulated as:
enriched 3D features Timieed as the query and key, and the unaltered /30 as the value, The novel attention mechanism can be formulated as:
Apro- = M.P((ma((Tas)).
(6)
The updated 2D semantic prompt ipro R1xC then fed into the Hybrid Attention Adapter within the same layer.
(TK
Tida = Softmax
,,"
V8
Hybrid Attention Adapter: Inspired by Adapter Tuning, we propose Hybrid Attention Adapter (HAA) to achieve effi- cient modality fusion. After obtaining 2D semantic prompts from frozen 2D pretrained models, both ipro and part of Tr, the Tr E R (i.e., F. without class token) are fed into HAA simultaneously. Inside the HAA, the ipro under- goes two parallel non-linear layers generate two learnable parameters, and B. These parameters transfer 2D visual se- mantic cues to the 3D modal by modulating the normalized 3D features and obtain Tmized. This operation decouples semantic information from global contextual features, miti- gates the inherent local ambiguitics in 3D-to-2D projection. The Semantic Transfer (ST) process can be formulated as:
7 = (WITTreeed
(8)
T (W (iiieed)
T (WC(OWWTT) 1))
where is the attention scale. This process updates the semantic relationships of the 3D features while filtering out redundant noise from the 2D modal. Inspired by For- mula 4, we replace linear layers in transformer blocks with BottleNeck (BN) modules, this design allows us to conve- niently balance the size of trainable parameters and model performance.
Modality Semantic Alignment: In the final layer of the model, 2D semantic prompts ipro are combined with 3D features. The combination are then fed into the downstream head to achieve alignment between two modality features. In classification tasks, ipro concatenated with input of task head. In the part segmentation task, 2D prompts are used to perform classification tasks independently without
Tmized = a * Zhr - m((T))
+ +B,
(7)
(fT))
where u()) and of.) are mean and standard deviation. Sub- sequently, we perform hybrid attention computations to achieve efficient modality fusion. We use the semantically
5

---

666
Table Classification three variants of the ScanObjeetNN (Uy al., 2019) and the ModelNct400 (Wu al., 2015), including the number of trainable parameters, FLOPs and overall accuracy (OA). ALL methods utilize the default data argumentation the baseline. Red indicates best performance among all methods, while Black denotes the highest performance within PETL methods. represents DINOv2 and C represents CLIP. Methods with using rotation data augmentation on ScanObjectNN.
ScanObjec1NN
ModelNet40
Method
Reference
Tunable params. (M) OBJ OBJJONLY PB_T50_RS
Points Num.
OA (%)
Supervised Learning Only
PointNet
CVPR 17
3.5
73.3
79.2
68.0
1k
PointNet++
NeurIPS 17
1.5
82.3
84.3
77.9
1k
-990.7
DGCNN
TOG 19
1.8
82.8
86.2
78.1
1k
-192.9
MVTN
ICCV
11.2
82.8
1k
-993.8
PointNeXt
NS 22
1.4
87.7
-994.0
PointMLP
ICLR
13.2
85.4
199455
RepSurf-U
CVPR 22
1.5
84.3
1k
194444
ADS
ICCV 23
87.5
1k
-995.1
Self-Supervised Representation Learning (Full fine-tuning)
OcCo
ICCV 21
22.1
84.85
85.54
78.79
1k
-992.1
Point-BERT
CVPR 22
22.1
87.43
88.12
83.07
1k
-99.22
MaskPoint
ECCV 22
22.1
89.70
89.30
84.60
-193.8 1k
Point-MAE
ECCV 22
22.1
90.02
88.29
85.18
-193.8 1k
Point-M2AE
NeurIPS 22
15.3
91.22
88.81
86.43
-994.0
ACTI
ICLR 23
22.1
93.29
91.91
88.21
-193.7
RECon
ICML 23
43.6
94.15
93.12
89.73
-193.9
PointMamba
NeurlPS 24
12.3
94.32
92.60
89.31
1k
93.6/-
Self-Supervised Representation Learning (Parameter- Effficient Transfer Learning)
Point-BERT (baseline)
CVPR 22
22.1
87.43
88.12
83.69
1k
92.7 /93.2
+ IDPT
ICCV 23
1.7 (7.69%)
88.12
88.30
83.69
1k
92.6/93.4
+ Point-PEFT
AAAI 24
0,6 (2.71%)
85.00
1k
93.4 /-
+ DAPT
CVPR 24
1.1 (4.97%)
91.05
89.67
85.43
1k
93.1/93.6
+ Ours(C)
1.8 (1.04%)
92.08
90.83
89.03
94.7 95.2
+ Ours(D)
1.8 (1.66%)
91.88
90.85
88.79
94.2 / 94.7
Point-MAE (baseline)
ECCV 22
22.1
90.02
88.29
85.18
1k
93.2 / 93.8
+ IDPT
ICCV 23
1.7 (7.69%)
91.22
90.02
84.94
1k
93.3 / 94.4
+ Point-PEFT
AAAT 24
0.6 (2.71%)
85.50
94.2 /- 93.5 / 94.0
+ DAPT
CVPR 24
1.1 (4.97%)
90.88
90.19
85.08
1k
+ Ours(C)
1.8 (1.04%)
91.86
91.20
89.14
/ 95.6
Ours(D)
1.8 (1.66%)
91.95
90.89
89.07
94.6 95.2
OOur((C)
1.8 (1.04%)
92.33
91.83
90.09
affecting segmentation pipeline (i.c., details in B.
the proposed paradigm in Sec. 4.3 verify the effectiveness of proposed modules.
4. Experiments
4.1. Implementation Details
In this section, we first present the implementation details in Sec. 4.1. After that, in Sec. 4.2, to demonstrate the effective- ness of the proposed paradigm, we evaluate its performance using four combinations of 2D and 3D pre-trained models on four downstream tasks, including synthetic object classi- fication, real-world object classification, part segmentation and few-shot learning. We also carry ablation studies for
For a fair comparison, all baselines adopt the same exper- imental setting: Freezing the pretrained 2D and 3D back- bones while only updating identical newly inserted adapters and position embedding layer of the 2D pretrained model. We select two 2D and two 3D pretrained models respec- tively, and conduct four sets of experiments on each down-
6

---

666
Besides, if we adopt the same data augmentation strategy with Recon and ACT, the performance of "PointMAR + CLIP" surpasses all results. This demonstrates that the in- troduced vision semantic cues significantly enhance the gen- eralization of 3D pretrained models on downstream tasks.
Table 2. Part segmentation on the ShapeNerPart (Yi et al., 2016). The mloU for all classed (CIs.) and for all instances (Inst.) are reported. #TP represents the tunable parameters. Red indicates the best performance among all methods, while Black denotes the highest performance within PETL methods. D represents DINOv2 and C represents CLIP.
Synthetic Shape Classification: In addition to the experi- ments conducted on real-world dataset, we perform experi- ments on a synthetic dataset, ModelNet40 (Wu et al., 2015), which consists of 12,311 clean 3D CAD models, covering 40 object categories. For testing the fine-tuned model, we provide results with and without the voting trick (Liu et al., 2019). The voting trick involves sampling multiple point clouds for the same sample and making model predictions multiple times, then aggregating the predictions through voting to obtain the final classification result. As Shown in Table 1, it can be observed that the proposed paradigm effectively improve the perfomance of pretrained across dif- ferent combinations. The combination "PointMAF + CLIIP" achieves a performance of 95.2%/95 /9.66, which is the high- est perfomance across all full fine-tuning and supervised methods
Method
Reference
(M) ClamloU (%)
Insimlo (%)
Supervised Learning Only
PointNet
CVPR 17
80,39
83.7
PointNet++
NeurlPS 17
81.85
85.1
DGCNN
TOG 19
82,33
85.2
APES
CVPR
83.67
85.8
Self-Superviser Representation Learning (Full fine-tuning)
OcCo
ICCV 21
27.06
83.42
85.1
Point-BERT
CVPR 22
27.06
84.11
85.6
MaskPoint
ECCV 22
84.60
86.0
Point-MAR
RCCV 22
27.06
84.19
86.1
ACT
ICLR 23
27.06
84,66
86.1
PointMamba
NeurIPS 24
84,40
86.2
Self Self- Supervised Representation Learning (Parameter Efficiien Transfer Learning)
Point. BERT (baseline)
CVPR
27.06
85.6
IDPT
ICCV
83.50
85.3
+ DAPT
CVPR 24
5.65
83.83
85.5
Part Segmentation: We conduct part segmentation exper- iments on the challenging ShapeNetPart (Yi et al., 2016) dataset, which comprises 16880 models with 16 different shape categories and 50 part labels. Experimental results on the ShapeNetPart dataset are shown in Table 2. The proposed paradigm boost the performence of 3D pretrained on the dataset, which is one of the hardest task. The "Point- MAE + CLIP" combination outperforms all methods in various experimental settings, demonstrating that vision se- mantic prompts can effectively introducing part semantics.
+ Ours(C)
6.65
84.61
86.2
+ Ours(D)
6.65
84,52
86.1
Point-MAE (baseline)
BCCV 22
27.06
84.19
86.1
+ IDPT
ICCV 23
5.69
83.79
85.7
+ DAPT
CVPR 24
5.65
84.01
85.7
+ Ours(C)
6.65
84,70
86.3
Ours(D)
6.65
84.73
86.2
stream task to validate the generalizability of the proposed paradigm. For the 3D pretrained models, chose Point- MAE (Jiang et al., 2023) and PointBERT (Yu et al., 2022). For the 2D pretrained models, we select the CLIP (Rad- ford et al., 2021) image encoder and DINOv2 (Oquab et al., 2023), where we use the ViT-B/16 version for CLIP and the ViT-B/14 version for DINOv2. All experiments are conducted on a single GeForce TRX 3090.
Few-shot Classification: To evaluate the effectiveness of the proposed modules with limited finetuning data, we con- duct experiments for few-shot classification on ModelNet40. As shown in Table 3, the proposed paradigm boost the per- formance of models on 10-way k-shot settings, and achieve comparable result on 5-way k-shot. The results illustrate that our approach can augment pretrain models generaliza- tion capabilities by introducing semantic cues.
4.2. Effectiveness on Downstream Tasks
For all tasks, we report the results of four combinations: PointMAE + CLIP, PointMAE + DINOV2, PointBERT + CLIP and PointBERT + DINOv2.
4.3. Ablation Study
As shown in Table 4, we systematically evaluate the con- tribution of each component in our proposed framework through four ablation studies. (I) 2D Baseline: To evaluate 2D model capabilities, we exclusively utilize the CLIP image encoder for classification, with inputs being three orthogonal depth maps projected from point clouds. (2) 3D Model: Building upon phase 1, we concatenate features from both the frozen Point-MAE encoder with DAPT (Zhou et al., 2024) and CLIP encoder before feeding them to the classification head. (3) + Semantic Transfer: To evaluate the influence of vision semantic prompts, we use
Real-World Shape Classification: ScanObjectNN (Uy et al., 2019)is one of the most challenging 3D datasets, which covers 15K real-world objects from 15 categories. We report classification results of three variants. As Shown in Table 1, with comparable trainable parameters, the per- fomance of proposed paradigm boost performances of 3D pretrained models on real-world shape classification tasks. It is worth noting that the combination "PointMAE + CLIP" achieves a performance of 89.14% on the most challenging split (PB-T50-RS) representing a 3.64% improvement over the previous state-of-thed the- method Point- (85.5%).
7

---

666
Origin 3D Features
Updated 3D Features
Table Few-shot learning ModelNet4O (Wu et al., 2015). We report overall accuracy (%) + the standard deviation (9) over ten runs. D represents DINOv2 and C represents CLIP.
S-way
10-way
Method
Reference
10-shot
20
10-shot
20-shot
Self- Self-Superviied Representation Learning (Fall fine-tuming)
OcCo
ICCV 21
94.0-13.6
95.9::2.3
894415.1
92.4114.6
ST
Point- BERT
CVPR 22
94.613311
96.31.277
91.0:15.4
92.715.1
Process
MaskPoint
BCCV 22
95.0+3.7
97.2+117
91444.0
93,4+3.5
Point-MAE
BCCV 22
96.3+2.5
97.8+18
92.6+4.11
95.0=3.0
Point- Point-M2AF
NeurIPS 22
9..8118
98.31114
92.314.5
95.0133.0
ICLR 23
96.8+2.3
98.0=114
93.3::400
95.6#22.8
Mixed Features
RECon
ICML
97.3119
989939
93.3439 95.813.0
Self Supervised Representation Leurning Cenuumtter-fficiiie Transfer Learning)
Figure 4. We select features from the final layer of HAA for vi- sualization. Short ST denotes semantic transfer, and HA denotes hybrid attention.
Point-BERT (baselinc)
CVPR 22
94.6+3.1
96.3+2277
91.0#5.4
92.7=5.1
+ IDPT
ICCV 23
96.0.11.7
97.2.1266
91.9.14.4
93.6.13..5
+ DAPT
CVPR 24
95.811..1
9..
92.2:14.3
94.2.13.4
where the same color indicates feature consistency. The visualization results show that the injected 2D semantic in- formation effectively aligns features of identical structures,
Ours(C)
96.3+322
97.5:2255
93.1=:42
95.0=:488
Ourx(D)
96.11:3.5
97.1433.2
93.045.5
95.2+:3.8
Point-MAE (baseline)
BCCV 22
96.3+2.5
97,8+18
92.6+4411
95.0#33.0
IDPT
ICCV 23
97.3+22.1
97.91111
92844411
95,4#22
and such feature distributions significantly enhance the gen- eralization capability of pretrained models on downstream tasks. Mcanwhile, the newly proposed hybrid attention mechanism strengthens semantic associations while preserv- ing the integrity of 3D information, further improving the effectiveness of rrossmmodal fusion.
DAPT
CVPR
9..8118
9..0:11.0
93.0.33.
95.5133.
Ouns(C)
97.0::3.2
98.3:1:.8
93.8::4.0
96.8::322
Ouns(D)
96.81.2.5
9..0:12.0
93.6.13..8
96.5.13.0
Table Ablation learning of the proposed paradigm. The detail of the table are described in section 4.3.
Method
#TP (M)
PB_T50_RS
6. Limitations
3D Baseline
1.07
85.18
+ Model
1.09
86.95
As shown in Table 8, we calculate the FLOPs of models. Due to the participation of 2D pretrained models (ViT-B/16 & ViT-B/14), our method exhibits relatively high FLOPs. To adopt off-the-shelf 2D pretrained models for saving stor- age and computational costs, this limitation cannot be re- solved by our current design. However, our new paradigm is compatible with any combination of transformer-basse 3D and 2D pretrained models, if more lightweight and higher- performing 2D pretrained models are proposed in the future, the proposed paradigm can achieve better inference speed and performance without any modifications.
+ Semantic Transfer
1.31
88.02
+ Self Attention
1.83
88,26
+ Hybrid Attention
1.83
89.14
T miized as the adapter outputs without any attention mecha- nism. (4) + Self Altention: In this phase, we adopt standard self attention with miszed only. (5) + Hybrid Attention: We replace the attention module in the last phase with proposed hybrid attention. We can observed that neither standalone 2D nor 3I) models achieve satisfactory results, while their naive combination yields significant performance improve- ments. Besides, our proposed multi-scale 2D semantic trans- fer and hybrid attention adapter effectively enhance model capabilities. More ablation studies are presented in Sec. D.
7. Conclusions
We propose a new paradigm that, for the first time, explores the integration of 2D visual semantic cues from frozen 2D pretrained models into efficient point cloud understanding, significantly improving their perfomance on downstream tasks while maintaining parameter efficiency. Our results shown that the porposed paradigm boosts the performance of 3D pretrained models downstream tasks while keeping minimal trainable parameters. With off-the-shelf 2D and 3D pretrained models, our paradigm outperforms all models across different training and fine- fine-tuning stratgies on the syn- thetic shape classification and challenging real-world 3D object recognition. Our method is compatible with any com-
5. Discussion
According our proposed modules, semantic prompts from 2D models can introduce additional semantic cues to 3D pretrained models through semantic transfer. However, intriguing question arises: what does the 3D model actually learn from these semantic prompts? We visualize the 3D features processed by Hybrid Attention Adapter in the last layer of the proposed paradigm. As shown in Figure 4, the feature colors are transformed into feature space using PCA,
8

---

666
bination of transformer-based 3D and 2D pretrained mod- els, more powerful large foundation pretrained models emerge in the future, our approach holds limitless potential.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, TT. Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
Impact Statement
This paper explores the transfer of rich semantic knowl- edge from 2D pre-trained models to enhance the general- ization capability of 3D pre-trained models on downstream tasks, under parameter-efficient paradigm. To the best of our knowledge, this represents the first attempt in the field of parameter-efficient transfer learning for point cloud understanding. Besides, the proposed paradigm is compati- ble with any combination of transformer-basee 3D and 2D pretrained models, as more powerful large foundation pre- trained models emerge in the future, the approach holds limitless potential.
Floridi, I.. and Chiriatti, M. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-694, 2020.
He, J., Zhou, C., Ma, X., Berg- beng-Kirkpatrick, T., and Neubig, G. Towards unified view of parameeter-efficient transfer learning, arXiv preprint arXiv:2110.04366, 2021.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficieet transfer learning for nlp. In International conference on machine learning, pp. 2790- 2799. PMLR, 2019.
Acknowledgments
This work was partially supported by National Defense Ba- sic Scientific Research program(No.JKK302291 11000)
References
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:21006 106.0655. 2021.
Achiam, J., Adler, S., Agarwal, S., Ahmad, I.., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference Computer Vision, pp. 709-727. Springer, 2022.
Chen, G., Wang, M., Yang, Y., Yu, K., Yuan, L., and Yue, Y. Pointgpt: Auto-regressively generative pre-training from point clouds. Advances in Neural Information Processing Systems, 36, 2024.
Jiang, J., Lu, X., Zhao, L., Dazaley, R., and Wang, M. Masked autoencoders in 3d point cloud representation learning. IEEE Transactions Multimedia, 2023.
Chen, S., Ge, C., Tong, Z., Wang, J.. Song, Y., Wang, J., and Luo, P. Adaptformer: Adapting vision transform- ers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:1 1116666666777 2022.
Li, X., Zhang, M., Geng, Y., Geng, H., Long, Y., Shen, Y., Zhang, R., Liu, J., and Dong, H. Manipllm: Embod- ied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18061-18070, 2024.
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023,
Liang, D., Zhou, X., Wang, X., Zhu, X., Xu, W., Zou, Z., Ye, X., and Bai, X. Pointmamba: A simple state space model for point cloud analysis. arXiv preprint arXiv:2402.10739 2024.
Devlin, J. Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding, arXiv preprint arXiv:1810.04805. 2018.
Liu, J., Yang, S., Jia, P., Zhang, R., Lu, M., Guo, Y., Xue, W., and Zhang, S. Vida: Homeostatic visual domain adapter for continual test time adaptation. arXiv preprint arXiv:2306 2i0:204344.4444 2023.
Dong, R., Qi, Z., Zhang, L., Zhang, J., Sun, J., Ge, Z., Yi, L., and Ma, K. Autoencoders cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning? In The Eleventh International Conference on Learning Representations (ICLR), 2023. URL ht https: //openreviens / www.rreewwwweeeeeeeeeeeeeeee www./nortt/fmtteemmmmmmmommmmmmt (tt.rrrmmmi--ouo00000000 tett/Forrummt0---8neeeaaaa
Liu, Y. Fan, B., Xiang, S., and Pan, C. Relation-shape convolutional neural network for point cloud analysis. In

---

666
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8895-8904, 2019.
Uy, M. A., Pham, Q-H., Hua, B.-S., Nguyen, T., and Yeung, S.-K. Revisiting point cloud classification: A new bench- mark dataset and classification model real-world data. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 1 1588-1597, 2019.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.00100 2017.
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gra- dient descent with warm restarts. arXiv preprint arXiv: /608. 03988, 2016.
Wu, Y., Liu, J., Gong, M., Gong, P., Fan, X., Qin, A. K., Miao, Q., and Ma, W. Self-supervised intra-modal and cross-modal contrastive learning for point cloud under- standing. IEEE Transactions on Multimedia, 26: 1226 1638, 2023.
Lu, J., Batra, D., Parikh, D.. and Lee, S. Vilbert: Pre- training task-agnostic visiolinguistic representations for Assion-and-language tasks. Advances in neural informa- tion processing systems, 32, 2019.
Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L.,, Tang, X., and Xiao, J. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912- 1920, 2015.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
Xie, S., Gu, J., Guo, D., Qi, C. R., Guibas, L., and Litany, O. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Computer Vision-FCCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 574-591. Springer, 2020.
Pan, M., Liu, J., Zhang, R., Huang, P., Li, X., Xie, H., Wang, B., Liu, L., and Zhang, S. Renderoce: Vision-centric 3d occupancy prediction with 2d rendering supervision. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12404-12411, IEEE, 2024.
Yang. S., Wu, J.. Liu, J., Li, X., Zhang, Q. Pan, M., Gan, Y., Chen, Z.,, and Zhang, S. Exploring sparse visual prompt for domain adaptive dense prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pp. 16334-16342, 2024.
Pang. Y., Wang, W., Tay, F. E. H., Liu, W., Tian, Y., and Yuan, L. Masked autoencoders for point cloud self- supervised learning, arXiv e-prints, 2022.
Qi, C. R., Su, H., Mo, K., and Guibas, L. Pointnet: Deep learning on point sets for 3d classification and segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Yi, L., Kim. V. G., Ceylan, D., Shen, I.-C.. Yan, M., Su, H., Lu, C., Huang, Q. Sheffer, A., and Guibas, L. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 35(6):1-12, 2016
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learming, pp. 8748-8763, PMLR, 2021.
Yu, X., Tang, L.., Rao, Y., Huang, T., Zhou, J., and Lu, J. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M.. Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (140):1-67, 2020.
pp.
Zha, Y., Wang, J., Dai, T., Chen, B., Wang, Z., and Xia, S.-T. Instance- aware dynamic prompt tuning for pre-trained point cloud models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 14161- 14170, 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference computer vision and paltern recognition, pp. 10684-10695, 2022.
Zhang, R., Guo, Z., Zhang, W., Li, K., Miao, X., Cui, B., Qiao, Y., Gao, P., and Li, H. Pointclip: Point cloud understanding by clip. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 8552-8562, 2022.
Tang, Y., Zhang, R., Guo, Z., Ma, X., Zhao, B., Wang, Z.., Wang, D., and Li, X. Point-peft: Parameter-efffcient fine- tuning for 3d pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 5171-5199, 2024.
Zhang, R., Han, J., Liu, C., Gao, P., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., and Qiao, Y. Llama- Efficient
10

---

666
fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303./6199 2023a.
Zhang, R., Wang, L., Qiao, Y. Gao, P., and Li, H. Learning 3d representations from 2d pre-trained models via image- to-point masked autoencoders. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pp. 21769-21788 2023b.
Zhang. Z., Girdhar, R., Joulin, A., and Misra, Self- supervised pretraining 3d features on any point-cloud. In Proceedings of the IEEE/CVF International Confer- on Computer Vision, pp. 00252-10263, 2021.
Zhou, X., Liang, D., Xu, W., Zhu, X., Xu, Y., Zou, Z.,, and Bai, X. Dynamic adapter meets prompt tuning: Parameter-efficient transfer learning for point cloud anal- ysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14707- 14717,200 2024.

---

666
A. Additional Relatedwork
We have briefty introduced the AT paradigm in the Section 3.2. Here, introduce another paradigm, Prompt Tuning (PT). As shown in Figure 5, PT generates set of tokens as prompts through random initialization. These prompts are added to the imput of transformer blocks or attention layers and interact with the original tokens through the self-attention mechanism. During fine-tuning, the weights of the backbone network remain frozen, and only the weights of the prompts are updated.
Downstream Task Head
Transformer Layer
Transformer Layer
Patch Embedding
Proeept
Class Tokkes
s43 Tokeo
Figure 5. The illustration of Prompt Tuning,
B. Additional Module Explanation
B.1. Modality Alignment
Here, we will introduce the classification task for aligning modality semantics in detail. As shown in Figure 6(a), in shape classification tasks, such as downstream tasks on ScanOhjectNN and ModelNet40, we concatenate 2D semantic prompts ipro of the last layer with origin classification head inputs impput Besides, in the part segmentation task, add an additional classification head for performing semantic alignment of 2D semantic prompts. As shown Figure 6(b), Iroo fed into classification head, and the pipeline of part segmentation is not affected.
Hoal
Classification Head
Semantic Promppt
Claes Takin
3D Itlam
Segmentation Head
Classification Head
(b)
Figure The details of semantic alignment.
B.2. 3D-to-DD Projection
First, when the intrinsic and extrinsic parameters of view v are determined, we can establish the camera coordinate system and the target 2D image coordinate system for projection. Given the point cloud, we can also determine the world coordinate system. The first step involves transforming the point cloud into the camera coordinate system. Let the mapped point
12

---

666
coordinate be (X Yw Zw), with R representing the rotation matrix and / the translation vector in the extrinsic parameters matrix, obtain the point coordinate (Xe, Ye, Ze) in camera coordinate system:
X
R11
R12
R1a
lx
X m
Ye
R21
R22
R23
ty
Y
(9)
Ze
R31
R32
R33
Z
1
0
0
0
1
After obtaining the coordinates of points (Xe, Ye, Ze) in the point cloud within the camera coordinate system, we can project them onto the 2D image plane using the camera's intrinsic matrix.
faXa
sYc
Ze u
J
CF
X
u
Zo
+
Cz
0
Ze
0
fy
Cy
Ye
(10)
fuYe
Ze
0
0
Z
Cy
Ze
Intrinsic Matrix
The (u, is the corresponding 2D coordinate of 3D point (Xm: Yuo Zwe..
C. Additional Implementation Details
Table 5. Training recipes for Parameter-Efficcen Transfer Learning.
Config
ScanObjectNN
ModelNet40
ModeINet4O-FewShor
ShapeNetPart
optimizer
AdamW W
AdamW
AdamW
A
learning rate
le-5
le-5
2e-4
weight decay
5c-2
5c-2
5c-2
5c-2
learning rate scheduler
cosine
cosine
cosine
cosine
training epochs
300
300
150
300
warmup epochs batch size drop path rate Generator rank q rank of HAA rank of HAA BN-v rank of HAA
10
10
10
10
32
32
32
16
0.2
0.1
0.1
0.1
16
18
18
18
18
18
18
18
18
64
72
32
128
image resolution
x24
224x
x24x222
224 224
image patch size
16/14
16/14
16/14
16/14
number of points
2048
1024
1024
2048
number of point patches
128
64
64
128
point patch size
32
32
32
32
augmentation
Scale& Trale&trrnn/Rotttionn
Scale&Trans
Scale&Trans Trans
GPU device
GTX 3090
GTX 3090
GTX 3090
GTX 3090
We adopt downstream fine-tuning configuration following pioneer work PointMAE (Jiang et al., 2023). More details provided in 5. Performing fine-tuning ScanObjectNN (Uy et al., 2019) as example, the overall training includes 300 epochs, with cosine learning rate (Loshchilov & Hutter, 2016) of 5e-4, and 10-epoch warm-up period. We adopt AdamW (Loshchilov, 2017) as the optimizer. Besides, we show the BN rank of our proposed Hybrid Attention Adapter (HAA) and the rank of a and 3 generator, which is the dimension of the feature passed through downward projection. We also provide relevant setup of 3D-to-2D projection and 2D pretrained models, such as the resolution 2D depth maps and the image patch size of 2D transformers.
13

---

666
D. Additional Ablation Studies
D.I. Ablation Study of Self Attention in Adapter
In this section, we conduct experiments on ScanObjectNN (Uy et al., 2019) to investigate the effectiveness of incorporating self-attention mechanisms within the adapter architecture without introducing 2D semantic cues. This exploration aims to determine whether the self-attention mechanism can enhance the model's generalization capability in the absence of additional semantic cues. We chose PointMAE (Pang et al., 2022) with DAPT (Zhou et al., 2024) as our 3D baseline. As shown in Table adopting self attention only bring limited performance improvement, which clarifies the importance of 2D) semantic cues.
Table 6. Additional ablation study of the attention mechanism.
Method
#TP (M)
PB T50 RS
3D Baseline
1.09
85.08
Self attention
1.61
85,53
D.2. Ablation Study of BN-v Rank
We design the BN-v follow the configuration adapter in DAPT (Zhou et al., 2024).
Table 7. Ablation study of BN-y rank.
Method
#TP (M)
PB_T50_RS
1.32
87.82
16
1.39
87.98
32
1.54
88.31
64
1,83
89.14
D.3. Ablation Study of FLOPS
Table Ablation study of FLOPs
Method
FIOPs
PB.T50.F
PointMLP
31.4
85,40
PoinIMAE+IDPT
7.2
84,94
PoinMMAE+DAPT
5.0
85,05
PoinMMAE+CLIRVTEBB22)
12.9
88,82
PoinMMAFFCCTIPAIIT-BBII
22.6
89.14
D.4. Ablation Study of Hybrid Attention
In some cross-modal attention mechanisms (Lu et al., 2019), features from one modality are typically used as queries, while features from the other modality serve as keys and values. However, after enhancing the features through Semantic Transfer, this approach no longer the optimal solution.
Table 9, Ablation study of hybrid attentinon
Method
#TP (M)
PB_T50_RS
Cross-modal attention
1.8
88.22
Hybrid attention
1.8
89.14
14

---

666
E. Additional Discussion
During investigation, observed that the performance improvements brought proposed paradigm ScanOb- jectNN's hardest split and ModelNet40 significantly surpass those achieved on the objbg and objonly splits. To analyze the underlying reasons, we visualize the 2D depth maps generated through our 3D-to-2D projection. As shown in Figure 7, the ModeINet40, as synthetic dataset, produces exceptionally high-quality depth maps through rendering. This characteristic fully leverages the capabilities of 2D models, consequently yielding substantial performance gains in synthetic classification and segmentation tasks. Furthermore, as demonstrated Figure 9, Figure 10 and Figure 8, the additional noise introduced in the hardest split exhibits minimal impact on imaging quality and 2D model classification performance (Table 10) after planar projection. This observation suggests that 2D models can effectively filter noise patterns that prove challenging for 3D models to process, thereby significantly enhancing model robustness in complex environments.
Table 10. Additional shape classification results ScanObjectNN and Modelnet40 with only CLIP.
Dataset
#TP (M)
Accuracy (%)
OBJ (BILONLY(Scan)
0.27
84.22
OBJ BC(Scan)
0.27
84.48
PB_ (5_T5_5((Scaa)
0.27
84.13
Modelnet40
0.27
92.78
A
Figure Depth maps Modelnet40.
15

---

666
Figure 8. Depth maps of ScanObjectNN-oojonny
d
Figure 9. Depth maps of ScanOOjeccNN-oobjbgg
16

---

666
Figure 10. Depth maps of ScanObjecdW-haardest.
17


Reviews:
[{"review_id": "LU4iNxbudE", "reviewer": "Reviewer_aPXL", "summary": "This paper introduces a novel fine-tuning paradigm for 3D pretrained models by leveraging 2D semantic prompts from frozen 2D pretrained models to enhance point cloud understanding, while maximally preserving the pretrained knowledge. The proposed method has three main contributions:\n\n1.\tIntroduction of the Hybrid Attention Adapter (HAA) that effectively utilizes 2D semantic cues from pretrained 2D foundation models to guide feature aggregation in 3D models.\n\n2.\tIncorporation of depth maps to address ambiguity issues arising from viewpoint projections.\n\n3.\tComprehensive experimental validation across multiple datasets (ScanObjectNN, ModelNet40, and ShapeNetPart), demonstrating performance improvements over existing methods in both classification and segmentation tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer's careful reading and detailed comments. First, we respectfully clarify that all the pretrained models in our paper are trained on ShapeNet, these works aim to explore effective approaches for building 3D foundation models. ModelNet40, ScanObjectNN, and ShapeNetPart are used only in downstream tasks for fine-tuning. We will add this clarification to the Implementation Details section in the revised version.\n\n**Clarification on PCA and knowledge forgetting:** We understand the reviewer's concern and appreciate the opportunity to clarify. PCA is used to illustrate the difference between pre-trained features and downstream task features, and it is unsuitable to explain knowledge forgetting. The goal of pretraining is to learn generalizable point cloud representations, while downstream tasks focus on task-specific features. Full fine-tuning adjusts all parameters to fit the downstream objective, which may lead the network to lose the general features acquired during pretraining. In contrast, parameter-efficient transfer learning (PETL) updates only a small subset of parameters while keeping most of the pretrained weights frozen. This allows the model to adapt to new tasks while preserving the knowledge learned during pretraining. [Overcoming catastrophic forgetting in neural networks, PNAS2017]. \n\n**Confusing description of Table 9:** We acknowledge that our explanation of Table 9 may have caused confusion. In future versions, we will provide a more detailed description of the cross-attention mechanism in Table 9, where T$_{mixed}$  serve as V and K, and T as Q. In Table below, we present experiments directly applying cross-attention between 2D prompts and 3D tokens. In PEFT, simply using cross-attention does not effectively fuse 2D information with 3D features. To address this, we design a simple ST module and leverage hybrid attention for effective feature fusion.\n\n| Method | FLOPs (G) | PB_T50_RS (%) |\n| --- | --- | --- |\n| 2D Prompt-3D Token Cross Attention (2-3 CA) | 1.46 | 87.57 |\n| Semantic Transfer (ST) | 1.31 | 88.02 |\n| 2-3 CA + Hybrid Attention(HA) | 1.98 | 88.26 |\n| ST + HA | 1.83 | 89.14 |\n\n**Marginal improvements over SOTA:** Our method proposes a parameter-efficient transfer learning framework that can be applied to existing self-supervised representation learning methods. We respectfully note that a direct comparison with fully supervised methods or self-supervised point cloud representation learning methods may not be entirely fair, as those methods often involve significantly more tunable parameters and different architectural designs. Compared to the previous SOTA method (DAPT), our method achieves higher performance on the same pretrained models. Additionally, unlike full fine-tuning, our approach requires significantly fewer parameters for downstream adaptation.\n\n**Lack of computational analysis:**  We also consider the analysis of computational cost to be highly important. We have presented FLOPs of selected models in Table 8, and we will add a new Table for comparison of inference time on Modelnet40 based on the reviewer's suggestions. Besides, we will enrich Table 1 with the comparison of FLOPs.\n\n| Method | FLOPs (G) | Infer. Time (Per obj.) |\n| --- | --- | --- |\n| PointMAE + DAPT | 4.8 | 0.054s |\n|  + 3D-2D Projection + CLIP(ViT-B/16) | 22.6 | 0.081s |\n| +Semantic Transfer | 22.6 | 0.083s |\n| +Hybrid Attention | 22.6 | 0.092s |\n\n**Scene-level applications:** We agree that scene-level applications are an important future direction. However, existing Transformer-based methods are mainly pre-trained on ShapeNet, while large-scale scene-level pretraining methods remain limited. For a fair comparison with previous PEFT methods, we validate our method on the same datasets. We also look forward to the development of Transformer-based scene-level pretraining methods. Besides, our proposed method has promising applications in real-world applications. For example, in robotic interaction, the first step involves cropping the target objects, followed by classification without background. Then, for interaction, it is necessary to understand the semantic parts of the object, i.e., part segmentation. The 2D pretrained models used in our work have been widely deployed in real-world scenarios, including Google's multi-modal robotic system PALM-E [PALM-E: An Embodied Multi-modal Language Model].\n\n**Comparisons with classical multi-modal fusion methods:** PointPainting and EPNet are classical multi-modal fusion methods. They utilize 2D semantic information to enhance scene-level 3D object detection tasks. In contrast, our work focuses on parameter-efficient transfer learning for 3D foundation models. We will include a discussion in the Method section to highlight the key differences between our approach and these methods.\n\n**We hope our replies have resolved the reviewer's concerns and provided sufficient clarification. We sincerely welcome any further questions or feedback.**", "claims_and_evidence": "The paper claims to address inherent local ambiguities in 3D-to-2D projection and illustrates this challenge with examples where points that are distant in 3D space appear deceptively proximate in 2D projections. This highlights a valid concern in the field. Additionally, the authors discuss knowledge forgetting in downstream tasks through PCA visualization, which I find unconvincing. They try to demonstrate information loss in semantic and geometric space through distances in PCA-reduced features. A more rigorous analysis would be necessary.", "methods_and_evaluation_criteria": "The authors propose two designs: \n\n1.\tIncorporating depth maps in the 2D branch, which effectively mitigates ambiguities in 2D projections of 3D data.\n\n2.\tIntroducing the Hybrid Attention Adapter (HAA), which renormalizes 3D features using guidance from 2D semantic cues. \n\nHowever, I remain skeptical about this approach, as simply performing cross-attention between 3D tokens and 2D semantic prompts might achieve comparable results with a more straightforward and less complex implementation. As evidenced in Table 9, the Hybrid-Attention mechanism shows only a marginal 0.92% improvement compared to the trivial multi-modal cross-attention approach (widely explored in 3D detection area), which represents the true incremental gain of the proposed method. This minimal improvement raises the concern about the effectiveness and necessity of the HAA component.", "theoretical_claims": "There are no theoretical proofs in this paper.", "experimental_designs_or_analyses": "The paper conducts experiments on ScanObjectNN, ModelNet40, and ShapeNetPart, demonstrating improvements over the Point-BERT and Point-MAE baselines, and highlighting the effectiveness of the proposed method. However, several weaknesses should be addressed:\n\n1.\tMarginal improvements over SOTA: Compared to state-of-the-art supervised learning-only and full-finetune methods, the performance gains are minimal. For instance, ADS achieves an accuracy of 95.1% on ModelNet40, while the proposed method achieves 95.2%. Similar marginal improvements are observed on ScanObjectNN. This raises concern about the practical significance of the proposed approach.\n\n2.\tLimited generalizability of pretrained 3D models: The paper claims to preserve general knowledge in pretrained 3D models, but these models are trained on relatively small datasets and each sample contains only a single object with limited information, such as ModelNet40 (12,311 samples) and ScanObjectNN (15,000 samples). It may not be sufficient for the model to learn truly generalizable knowledge. As discussed earlier, supervised learning-only methods can also achieve SOTA performance. To validate the claims, experiments on more diverse and complex datasets and tasks, such as indoor and outdoor 3D detection and segmentation, are necessary.\n\n3.\tLack of computational analysis: The paper does not provide a detailed comparison of parameters and FLOPs between the 3D branch and the 2D branch that utilizes a heavy Vision Transformer (ViT-B) model. Additionally, the inference time before and after integrating the 2D branch should be reported.", "relation_to_broader_scientific_literature": "This work provides insights into two broader areas: (1) foundation models and fine-tuning strategies, particularly in leveraging foundation models from another modality to guide the fine-tuning process, and (2) multi-modal fusion in 3D vision, where the proposed Hybrid Attention Adapter (HAA) facilitates semantic information fusion through Semantic Transfer and hybrid attention mechanisms.", "essential_references_not_discussed": "The proposed method relates to multi-modal fusion in 3D vision, with its key distinction being the leverage of foundation models under a fine-tuning paradigm. However, I suggest including comparisons with classical multi-modal fusion methods such as [PointPainting: Sequential Fusion for 3D Object Detection, CVPR2020] and [EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection, ECCV 2020].", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper proposes HHA, a novel approach to enhance 3D pretrained models through feature modulation like [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization, ICCV 2017]. This approach offers a new perspective on incorporating multi-modal knowledge into 3D vision.", "other_comments_or_suggestions": "Some Typos Mistakes\n\n1.\tL134 Add a “.” before \"However\".\n\n2.\tL197 “follow” should be “following”.\n\n3.\tL428 “Our results shown” should be “Our results show”.\n\n4.\tL430 “porposed” shold be “proposed”.\n\n5.\tL434 “stratgies” should be “strategies”.", "questions_for_authors": "The authors should conduct experiments on larger datasets such as ShapeNet or large-scale outdoor datasets like Waymo and nuScenes. Both pretraining and fine-tuning experiments on these larger-scale datasets would help demonstrate the method's scalability and verify whether the performance improvements remain consistent across more diverse and challenging scenarios.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qt7rsdAInV", "reviewer": "Reviewer_5f8a", "summary": "This paper introduces a parameter-sparse fine-tuning strategy for fine-tuning pre-trained 3D point cloud models with vision semantic prompts of a frozen vision transformer in the 2D domain. It introduces a new Hybrid Attention Adapter (HAA) which aggregates the features of different views of the depth images in order to achieve state-of-the-art performance on ScanObjectNN and ModelNet40 with minimal trainable parameters (~1–2%).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer's for the constructive comments and valuable suggestions.\n\n**Performance gain on segmentation:** We understand the reviewer's concern and appreciate the opportunity to clarify. The ShapeNetPart is a nearly saturated dataset, where multiple methods have achieved similar performance in recent years. Additionally, the dataset contains some commonly recognized annotation errors, leading to discontinuous performance improvements. However, our method achieves better-fine-tuned performance than baseline (full fine-tuning) on the datasets with less trainable parameters.\n\n**Computational cost and practical applications:** Thank you for raising this important point regarding the computational cost of our approach; we present the FLOPs of selected models in Table 8, and we will add a new Table for comparison of inference time on Modelnet40 in the future version. Besides, we will enrich Table 1 with the comparison of FLOPS.\n\n| Method | FLOPs (G) | Infer. Time (Per obj.) |\n| --- | --- | --- |\n| PointMAE + DAPT | 4.8 | 0.054s |\n|  + 3D-2D Projection + CLIP(ViT-B/16) | 22.6 | 0.081s |\n| +Semantic Transfer | 22.6 | 0.083s |\n| +Hybrid Attention | 22.6 | 0.092s |\n\nOur proposed method has promising applications in various tasks. For example, in robotic interaction, typically, the first step involves cropping the target object, followed by classification. For interaction, it is necessary to understand the semantic parts of the object, i.e., part segmentation. The 2D pretrained models used in our work have been widely deployed in real-world scenarios, including Google's multimodal robotic system PALM-E [PALM-E: An Embodied Multimodal Language Model].\n\n**Ablation on the number of views:** We greatly appreciate the reviewer's attention to the influence of view count—this is indeed an important aspect. We additionally conduct experiments using only the top view, six views, and nine views. The results show that increasing the number of views does not lead to performance improvement. In feasibility experiments, we observed that orthogonal views yield the best 2D classification results. Therefore, redundant views are filtered out by max pooling and do not contribute to the final performance. Using only a single view may be suboptimal, as the most informative viewpoint varies across object categories. For example, the top view provides the richest semantic information for airplanes, while the side or front view may be more informative for chairs. Therefore, providing three orthogonal views and allowing the network to adaptively select the most relevant one offers a more robust and effective solution.\n\n| Number of views | FLOPs (G) | PB_T50_RS (%) | ModelNet40 (%) |\n| --- | --- | --- | --- |\n| 1 (Top view) | 22.6 | 88.16 | 94.5 |\n| 3 | 22.6 | 89.14 | 95.2 |\n| 6 | 22.6 | 88.97 | 95.2 |\n| 9 | 22.6 | 89.18 | 94.9 |\n\nDuring the classification of 100 airplane and chair samples, we recorded which view the 2D prompt selected by max pooling originated from, and counted the number of times each view was chosen. The statistics in the Table below are based on the ModelNet40 dataset.\n\n| Category | Top Veiw Count | Side View Count | Front View Count |\n| --- | --- | --- | --- |\n| Airplane | 67 | 11 | 22 |\n| Chair | 17 | 48 | 35 |\n\n**Qualitative Description of the Effectiveness of 2D Semantic Prompt:** Thank you for pointing out the need for a qualitative description. In the Discussion section of our paper, we visualize the features in our proposed HAA through PCA. As shown in Figure 4, with Semantic Transfer, we can transfer the 2D semantic cues to 3D features, enriching the association of the same components (wings of the airplane) and improving model performance in downstream tasks.\n\n**CLIP outperforms DINOv2:** We agree with your point. In our feasibility experiments, frozen CLIP outperforms DINOv2 when classifying using depth maps obtained via our proposed projection. We attribute this to CLIP's strong zero-shot classification capability, and we believe this stems from its image-text alignment training.\n\n| 2D Model | ModelNet40 (%) |\n| --- | --- |\n| CLIP | 92.78 |\n| DINOv2 | 91.86 |\n\n**Larger 3D foundation model:** We utilize CLIP-L  to enhance PointGPT-L [PointGPT: Auto-regressively Generative Pre-training from Point Clouds, NIPS2023] 3D model, and still observed consistent performance improvements on ModelNet40.\n\n| Method | ModelNet40 (%) |\n| --- | --- |\n| PointGPT-L | 94.1 |\n| +DAPT | 94.2 |\n| +CLIP-L | 95.5 |\n\n**Citing PointCLIP：** PointCLIP is one of the representative works in point cloud understanding that explores the integration of 2D semantic information into 3D point cloud tasks. We appreciate the reviewer's suggestion and will include a discussion of PointCLIP in the Related Work section.\n\n**We hope our clarifications have been helpful, and we would be glad to address any further questions you may have.**", "claims_and_evidence": "The arguments are substantiated by rigorous experiments with a variety of tasks (classification, segmentation, few-shot learning). The robust empirical results clearly demonstrate the strength of combining 2D semantic features with 3D models.", "methods_and_evaluation_criteria": "The proposed approaches (multi-view depth projection, vision semantic prompting, HAA) are well-suited and well-organized for addressing the problem of semantic alignment in parameter-efficient tuning of the 3D model. The typical benchmarks used (ScanObjectNN, ModelNet40, ShapeNetPart) are well-suited and well-known in the field.", "theoretical_claims": "There are no significant theory proofs or hypotheses requiring formal verification. The proposed adapter and attention mechanisms are solidly grounded, clearly described, and in line with traditional architectures.", "experimental_designs_or_analyses": "The experimental configurations are extensive and solid, with broad comparison with the most up-to-date strategies and multiple backbones. Ablation tests clearly demonstrate the incremental advantage of each component proposed.", "relation_to_broader_scientific_literature": "This work builds effectively on the most recent in 3D pre-training (PointMAE, PointBERT), PETL (IDPT, Point-PEFT, DAPT), and cross-modal learning (ACT, multi-view networks). It bridges in a new manner 2D semantic prompting with parameter-efficient tuning, pushing the state-of-the-art in the combined field.", "essential_references_not_discussed": "The paper failed to mention \"PointCLIP (Zhang et al., CVPR 2022),\" an important prior work also employing CLIP-based 2D prompts for the sake of 3D tasks. Citing it will serve to convey how this paper generalizes over the previous prompting approaches.", "other_strengths_and_weaknesses": "Strength\nStrong integration of multi-modal semantic prompting with PETL.\nCurrent results for top benchmarks.\nCareful experimental validation with judicious ablations.\nWell written with great supporting material.\n\nWeakness\nNo code released yet, limiting short-term replicability.\nComputational complexity for inference due to the utilization of multiple views and ViT.\nConstrained gains for the segmentation tasks compared to classification tasks.\nPractical application can be thwarted by the intricateness of the architecture.", "other_comments_or_suggestions": "Minor typos:\n\"deitals\" -> \"details\", in page 4.\n\"Vsion\" -> \"Vision\", in the title on OpenReview, not in the paper.\nDirections for future work: explore usage at the scene level, leverage RGB/color data, or use language prompts for richer semantics.", "questions_for_authors": "1. Have you taken into account the computational cost of your strategy in the context of multiple views as well as the ViT? How realistic is this for real-world application?\n2. Did you use fewer or more than three views? How do results vary with the number of views?\n3. Qualitatively describe in detail the specific semantic cues the 2D prompts provide which significantly facilitate the 3D recognition tasks.\n4. CLIP generally outperformed DINOv2 prompts. Do we credit this to CLIP’s multi-modal pre-training? What do you think of this difference?\n5. How does performance scale with larger 3D models—does the 2D prompt remain complementary regardless of the size of the 3D model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "65X7bDRsPl", "reviewer": "Reviewer_5QCm", "summary": "This paper proposes to leverage the rich semantic cue inherent in pretrained 2D foundation models as the semantic prompt for efficient point cloud learning. The authors identify the limitation of existing 3DPEFT research: they fail to align semantic relationships of features required by downstream tasks. To address this issue, a new Hybrid Attention Adapter is designed to fuse 2D semantic cues into 3D representations efficiently with minimal trainable parameters. Experiment results demonstrate the effectiveness of the proposed hybrid attention.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's thoughtful feedback and sincerely thank the reviewer for the positive recognition of our work.\n\n**Feature inconsistency affects downstream task performance**：We appreciate the reviewer's question regarding the impact of feature inconsistency on downstream performance. There exists a mismatch between the representations emphasized by the pre-training task and those required by the downstream task. Pre-training often focuses on local structures and exhibits a strong positional correlation. For example, the wing features of an airplane are different before fine-tuning due to they are not adjacent in 3D space, even though their structures are similar. However, downstream tasks rely more on semantic features, which require modeling long-range dependencies — capturing the relationship between the two wings of an airplane.\n\n**Ablation on number of views and impact of viewpoints:** We greatly appreciate the reviewer's attention to the impact of view count—this is indeed a very important aspect for analysis. We additionally conduct experiments using only the top view, six views, and nine views. The results show that increasing the number of views does not lead to performance improvement. In feasibility experiments, we observed that orthogonal views yield the best 2D classification results. It provides a balanced and comprehensive representation of the 3D structure while maintaining geometric fidelity. Therefore, redundant views are filtered out by max pooling and do not contribute to the final performance.\n\n| Number of views | FLOPs (G) | PB_T50_RS (%) | ModelNet40 (%) |\n| --- | --- | --- | --- |\n| 1 (Top view) | 22.6 | 88.16 | 94.5 |\n| 3 | 22.6 | 89.14 | 95.2 |\n| 6 | 22.6 | 88.97 | 95.2 |\n| 9 | 22.6 | 89.18 | 94.9 |\n\nRegarding the question on different viewing directions, in datasets such as ModelNet40, objects are aligned orthogonally but their orientations are not consistent — there can be angular differences like 90 or 180 degrees among objects within the same category. As a result, the choice of viewing direction has minimal impact on the final performance. Additionally, we explored projecting 3D point clouds onto cylindrical and spherical surfaces. However, these projections introduced significant distortions, which degraded the quality of the rendered views and adversely affected 2D models' performances.\n\n**Confusing description in Table 1:** We acknowledge that the calculation of the percentage of tunable parameters in Table 1 has caused confusion. The percentage listed for \"Ours\" refers to the model after incorporating the 2D model. We will provide a more detailed explanation in future versions to clarify this point. Thank you for your helpful suggestion!\n\n**Unintuitive ablation table:** We agree that the original ablation may not be sufficiently intuitive. Therefore, we have revised the ablation study of the main modules and present the updated results in the following table:\n\n| Method | #TP(M) | PB_T50_RS (%) |\n| --- | --- | --- |\n| 3D Baseline | 1.07 | 85.18 |\n| + 2D Baseline | 1.09 | 86.95 |\n| + Semantic Transfer | 1.31 | 88.02 |\n| + Self Attention | 1.83 | 88.26 |\n| + Hybrid Attention | 1.83 | 89.14 |\n\n**We sincerely thank the reviewer for the positive recognition of our work. We truly value your comments and would appreciate any further questions or suggestions that can help us improve the paper.**", "claims_and_evidence": "The authors claim that the current full-tuning method will forget the pretrained knowledge,  while the existing PEFT methods fall short in aligning semantic relationships. However, these claims are not supported by convincing evidence or theoretical analysis. Why would the feature inconsistency affect the downstream performance in Figure 1?", "methods_and_evaluation_criteria": "The design of the framework and evaluation make sense.", "theoretical_claims": "There is no theoretical analysis in this paper.", "experimental_designs_or_analyses": "Yes, the experimental design is sound and comprehensive.", "relation_to_broader_scientific_literature": "This paper introduces a promising way to incorporate 2D semantic prior into 3D representation learning. While cross-modal complementary information has been widely explored in a wide range of research areas, it has rarely been investigated in the field of 3D parameter-efficient learning.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Despite the intuitive and practical idea of using 2D semantic prior to enhance 3D representation learning, this paper also conducted extensive evaluation to validate its design. I recommend an accept for this paper, for which I believe its further impact in this topic.", "other_comments_or_suggestions": "1. In the session of  3D-to-2D Projection, the authors employ a straightforward strategy to fuse all three orthogonal depth maps. Considering the ambiguity as illustrated in Figure 1(c), is the choice of three views already satisfied in any downstream tasks? I wonder what will happen if we adjust the view direction or the number of views, such as down-to-top or front-to-back views.\n2. The calculation of the number of tunable parameters in Table 1 may introduce confusion. While the number of trainable parameters increases, the relative percentage decreases.  \n3. The ablation in Table 4 is kind of unintuitive; the baseline should be the 3D model only instead of using the 2D model.", "questions_for_authors": "See the comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rb2JDgiuuP", "reviewer": "Reviewer_8Bgm", "summary": "This paper explores PETL for 3D point cloud understanding to address the issues of full fine-tuning, such as forgetting pretrained knowledge and high storage costs. The authors propose a novel fine-tuning paradigm that integrates 2D vision semantic prompts from frozen pretrained models to improve the performance of 3D models with minimal trainable parameters.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the detailed review and valuable suggestions.\n\n**Mitigation of local and global ambiguity:** We understand the reviewer's concern and appreciate the opportunity to clarify. During the classification of 100 airplane and chair samples, we recorded which view the 2D prompt selected by max pooling originated from, and counted the number of times each view was chosen.The statistics in the Table below are based on the ModelNet40 dataset.\n\n| Category | Num. of Top | Num. of Side | Num. of Front |\n| --- | --- | --- | --- |\n| Airplane | 67 | 11 | 22 |\n| Chair | 17 | 48 | 35 |\n\nWe observed that when class tokens from three views are simultaneously fed into the network, the trained model tends to favor specific views for different categories—for instance, the top view for airplanes and front/side views for chairs. These preferred views typically exhibit the least local ambiguity. Besides, our approach enables the model to adaptively associate each object with its most informative view, thereby mitigating global ambiguity. We will include more detailed technical explanations in the revised version. \n\n**Comparison of inference time:** We agree with the reviewer that inference time is an important metric. Just in case, we respectfully clarify that the primary goal of \"efficienct\" in parameter-efficient fine-tuning is to mitigate the storage overhead caused by the exponential growth of parameters in large foundation models. To make our work more comprehensive and complete, we will add a new Table for comparison of inference time on Modelnet40 based on your suggestions. \n\n| Method | FLOPs (G) | Infer. Time (Per obj.) |\n| --- | --- | --- |\n| PointMAE + DAPT | 4.8 | 0.054s |\n|  + 3D-2D Projection + CLIP(ViT-B/16) | 22.6 | 0.081s |\n| +Semantic Transfer | 22.6 | 0.083s |\n| +Hybrid Attention | 22.6 | 0.092s |\n\n**Performances gain on segmentation and few-shot classification:** We understand the reviewer's concern regarding the few-shot and part segmentation performances. The ShapeNetPart and ModelNet40 (few-shot for PETL) are nearly saturated datasets, where multiple methods in recent years have achieved similar performance. Additionally, the ShapeNetPart contains some commonly recognized annotation errors, leading to discontinuous performance improvements. \n\n**Concerns regarding 2D pretrained models processing depth maps:** We understand the reviewer's concern regarding the difficulty for 2D models to extract semantic information from depth maps. 2D foundation models are pretrained on large-scale image datasets and possess generalizable feature representations. In our method, we convert single-channel depth maps into three-channel pseudo-RGB images by repeating the depth values across channels. Besides, as shown in Figure 7, the proposed 3D-to-2D projection strategy mitigates issues such as texture loss and boundary blurring. According to our feasibility experiments, the pretrained 2D model can effectively extract semantic cues from the depth maps we constructed. In addition, prior works such as [PointCLIP: Point Cloud Understanding by CLIP, CVPR2022]  and [Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders, CVPR2023]have explored the use of CLIP to extract semantic information from depth maps generated by projecting point clouds.\n\n| 2D Model | ModelNet40 (%) |\n| --- | --- |\n| CLIP | 92.78 |\n| DINOv2 | 91.86 |\n\n**Effectiveness of 2D semantics for point cloud understanding:** We acknowledge the reviewer's concern about whether 2D semantics can meaningfully enhance 3D point cloud representation. Currently, Transformer-based point cloud foundation models are pretrained on ShapeNet, a dataset composed of background-free, single-object point clouds. Besides, the depth maps used in our method are generated directly from the 3D point clouds through a globally consistent projection, ensuring a correspondence between pixels and 3D points. \n\n**Real-world applications:** We understand the reviewer's concerns regarding our application scenario and appreciate the opportunity to provide further clarification. We agree that scene-level applications are an important future direction. Existing Transformer-based methods are mainly pretrained on ShapeNet, while large-scale scene-level models remain limited. For a fair comparison with previous PEFT methods, we validate our method on the same datasets. Our proposed method has promising applications in various tasks. For example, in robotic interaction, the first step involves cropping the target object followed by classification without background, and it is necessary to understand the semantic parts of the object for interaction. \n\n**We hope our responses have addressed your concerns and helped clarify any confusion. We truly welcome any additional questions or suggestions that could further improve our work.**", "claims_and_evidence": "1. The authors believe that rich semantic cues in 2D pretrained models can improve the generalization of 3D point cloud understanding. While the experimental results, including classification and part segmentation, indicate the improvements, it is not entirely clear if the improvement is consistent across different type of 3D tasks, such as 3D point cloud semantic segmentation and object detection. Moreover, the datasets used, such as ModelNet40, ScanObjectNN and ShapeNetPart, primarily contain object-centric point clouds, which may not be able to fully capture real-world scenarios.\n2. The paper claims that the projection of 3D point cloud to three orthogonal depth images can mitigate global ambiguities, and the semantic transfer module can mitigate local ambiguities. However, there is no experimental results that can prove it.", "methods_and_evaluation_criteria": "1. The proposed method requires the low number of trainable parameters as shown in the experiment section, however, the efficiency of the model should be further validated by comparing the inference time with baselines.\n2. To prove the effectiveness of the proposed model, it should be evaluated on more large-scale scene understanding tasks.", "theoretical_claims": "The mathematical formulations, including all the equations, are clear to me.", "experimental_designs_or_analyses": "Strengths:\n1. The authors make a fair comparison with many baselines, including supervised learning-only, full fine-tuning and PETL methods. \n2. The authors conduct an ablation study for each proposed module.\nIssues:\n1. Need experiments on more complex real-world scenarios.\n2. More complex 3D scene-level tasks to prove the effectiveness.", "relation_to_broader_scientific_literature": "The paper introduces the HAA to fuse 2D vision semantics into 3D point cloud features, and the trainable parameters are relatively low compared to other PETL and full fine-tuning methods.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. HAA provides a new way to efficiently fuse 2D features with 3D point cloud features.\n2. The proposed method outperforms the baselines on three datasets (ScanObjectNN, Modelnet40 and ShapeNetPart), and it is also efficient in terms of trainable parameters.\n3. The figures and tables are clear and easy to understand.\nWeaknesses:\n1. No experiments or direct visualization to show how the model solves the global and local ambiguity.\n2. For few-shot learning and part segmentation tasks, the improvement is limited.", "other_comments_or_suggestions": "1. Typo: \"PoinrMamba\" in the table 2.\n2. Type: \"perfomance\" in lines 368, 371, 427.", "questions_for_authors": "1. Since 2D pretrained models are trained on RGB images, how can they effectively extract features from depth images?\n2. Considering that depth images lack background information and contain only a single object, how can meaningful 2D semantic features be extracted and fused to enhance 3D point cloud understanding?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel and efficient fine-tuning paradigm for 3D pretrained models by leveraging 2D semantic prompts from frozen vision transformers, introducing a Hybrid Attention Adapter (HAA) to enhance semantic alignment with minimal trainable parameters. The method is well-motivated, experimentally validated across multiple benchmarks, and contributes meaningfully to the growing literature on cross-modal foundation model adaptation. While the performance gains are sometimes marginal, the approach is conceptually solid and opens promising directions. To strengthen the paper, the authors are encouraged to include runtime and FLOPs analysis,  ablations on the number of depth views, and evaluations on larger, scene-level datasets.
Decision: Accept (poster)