Point-PRC: A Prompt Learning Based Regulation Framework for Generalizable Point Cloud Analysis
Authors: Hongyu Sun
Qiuhong Ke
Yongcai Wang
Wang Chen
Kang Yang
Deying Li
Jianfei Cai
Venue: NeurIPS 2024 poster (2024)
Keywords: point cloud analysis
point cloud recognition
3d domain generalization
large 3d models
prompt learning
multi-modal learning
Primary area: machine_vision

Abstract: This paper investigates the 3D domain generalization (3DDG) ability of large 3D models based on prevalent prompt learning. Recent works demonstrate the performances of 3D point cloud recognition can be boosted remarkably by parameter-efficient prompt tuning. However, we observe that the improvement on downstream tasks comes at the expense of a severe drop in 3D domain generalization. To resolve this challenge, we present a comprehensive regulation framework that allows the learnable prompts to actively interact with the well-learned general knowledge in large 3D models to maintain good generalization. Specifically, the proposed framework imposes multiple explicit constraints on the prompt learning trajectory by maximizing the mutual agreement between task-specific predictions and task-agnostic knowledge. We design the regulation framework as a plug-and-play module to embed into existing representative large 3D models. Surprisingly, our method not only realizes consistently increasing generalization ability but also enhances task-specific 3D recognition performances across various 3DDG benchmarks by a clear margin. Considering the lack of study and evaluation on 3DDG, we also create three new benchmarks, namely base-to-new, cross-dataset and few-shot generalization benchmarks, to enrich the field and inspire future research. Code and benchmarks are available at \url{https://github.com/auniquesun/Point-PRC}.

Point-PRC: A Prompt Learning Based Regulation Framework for Generalizable Point Cloud Analysis
Hongyu Sun
Qiuhong Ke²
Yongcai Wang'*
Wang Chen'
Kang Yang' Deying Li Jianfei Cai²
Department of Computer Science, Renmin University of China, China Department of Data Science & AI, Monash University, Australia
Abstract
This paper investigates the 3D domain generalization (3DDG) ability of large 3D models based on prevalent prompt learning. Recent works demonstrate the perfor- mances of 3D point cloud recognition can be boosted remarkably by parameter- efficient prompt tuning. However, observe that the improvement on downstream tasks comes at the expense a severe drop in 3D domain generalization. To resolve present comprchensive regulation framework that allows the learnable prompts to actively interact with the well-learned general knowledge in large 3D models maintain good generalization. Specifically, the proposed framework imposes multiple explicit constraints on the prompt learning trajec- tory by maximizing the mutual agreement between task-specific predictions and task-agnostic knowledge. We design the regulation framework as plug-and-play module to embed into existing representative large 3D models. Surprisingly, method not only realizes consistently increasing generalization ability but also enhances task-specific 3D recognition performances across various 3DDG bench- marks by a clear margin. Considering the lack of study and evaluation on 3DDG, we also create three new benchmarks, namely base-to-new, cross-dataset and few-shot generalization benchmarks, to enrich the field and inspire future research. Code and benchmarks are available https: :ttps://get //gi /////ttuuu thub. com/
this challenge,
Introduction
3D point cloud data is widely adopted in many industrial and civil areas, such autonomous driving [47], robotics [27, 3], geospatial mapping [39] and entertainment games [46]. Recognizing 3D objects from point cloud data is basic need of these applications. Relevant research topics have been explored for long time and their development can be summarized in three stages. In the early phase, PointNet series [48, 49] sparked a wave of directly operating raw point cloud data using deep learning techniques. Later methods improved upon PointNet and PointNet++ in terms of local information aggregation [33, 67, 60, 63, 69, 37], optimization techniques [50], geometry prior [55], model architecture [18, 82, 68, 44, 14], etc. Although remarkable progress has been made, these works tend to design specific architectures targeting downstream benchmarks while paying little attention to the model generalization, resulting in disappointed performances when deploying in the wild, especially in the case of unseen domains and corrupted data. On the other hand, training point cloud recognition models on each benchmark is not always feasible due to the narrow set of 3D visual concepts and expensive labeled data.
Corresponding author. (sunhongyu, ycu) @ruc.edu.co (qiuhong jianfei.cal Amonash. edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The above factors call for the investigation of the domain generalization routes for the deep point cloud models so that they can learn robust and transferable representations. Related studies have been extensively conducted in image recognition [28, 29, 30, 32, 87, 13, 84] while to our best knowledge, there are only a few methods to discuss the domain adaptation and domain generalization in 3D. Several years ago, PointDAN [51] first investigated domain adaptation for point cloud classification models by aligning multi-scale features of 3D objects across the source and target domains. MetaSets [20] proposed to meta-learn on group of transformed point sets to obtain generalizable representations to handle the sim-to-real geometry shifts. PDG [64] decomposed 3D objects into shared part space to reduce domain gap and developed a part-level domain generalization model for 3D point cloud classification.
However, the above methods are all built on small models (e.g., PointNet with 1.2M pa- rameters) and small datasets (e.g., ModelNet with 9,843 training samples) and the overall transferability is still suppressed compared to prevalent large 3D foundation mod- els [76, 90, 71, 78, 77, 79, 72], which have been pre-trained on numerous volume of 3D data [11, 10] and demonstrated promis- ing zero-shot capability. Recent works stand on the shoulder of large 3D foun- dation models and push the boundary of downstream 3d tasks by parameter- effficien adaptation, such as prompt learning [74, 58), adapter [59, 88], and their combi- nation. They insert learnable prompts in the inputs or adapter inside the Trans- former [62] blocks to adapt the foundation models to specific 3D tasks. However, op- timizing the newly introduced small mod- ules targeting downstream benchmarks is prone to overfitting, thus disturbing the in- ternal representations and compromising the inherent generalization of the founda- tion models [85, 36, 89, 26, 24, 25]. As Fig. 1 demonstrates, lightweight prompt tuning can notably lift the recognition ac- curacy of representative large 3D models on seen classes while hindering the generalization on unseen new classes, where the performances consistently lag behind corresponding zero-shot predictions of these models.
ULIP-2
classes
ULIIP-2 with PT
ULIP
+ RC
Zero-
Acc 75,80
Acc: 83,23
Acci 71.22
New clusses
Ull IP-3 with PT
ULIP-2 with RC
+
+ + RC
cluir
Acc: 07
Acc: 71.77
Figure 1: Motivation of our research: to promote the performances on downstream 3D tasks while main- taining good generalization of large 3D models. The experiments are conducted on ShapeNetCore V2. ULIP- 2 can reach 71.22% zero-shot recognition accuracy on this dataset. Recent works built on ULIP-2 introduce lightweight prompt tuning (PT) to further boost target tasks (75.80% accuracy). However, we observe the im- provements come at the expenses of severe drop 3D domain generalization (e.g.. 57.07% accuracy on new classes, much behind 71. 71.22%) and develop a system- atic regulation constraint (RC) framework to address this challenge.
In this paper, we develop our approach based on large 3D foundation models through lightweight prompt learning and propose comprehensive framework that consists of three regulation constraints to allow the learning trajectory to interact with the well-learned knowledge in large 3D models actively, achieving better task-specific performances and aaskoagnottic generalization at the same time. Specifically, we propose the mutual agreement constraint to regulate the learnable prompts to produce consistent feature distributions and predictions with the pre-trained foundation models. Then, we exploit the flexible and diverse text descriptions derived from LLMs or manual templates to reflect the attributes of different classes of point clouds and enhance the generalization. Finally, we develop a weighted model ensemble strategy to update the learnable prompts smoothly and predictably, avoiding giant and unexpected lcaps toward overfitting the downstream datasets. Some recent works also explore parameter-effiete focus on the performances of downstream tasks while failing to take the model generalization into account. As far as we are aware, our work initiates the first attempt to impose explicit regulation constraints and improve the 3D domain generalization based on large 3D models.
tuning for point cloud analysis [74, 59, 58, 88], they
In addition, we argue existing 3D domain generalization evaluation benchmarks, such as PointDA [51] and Sim-to-Real [20], may not be comprehensive to evaluate common generalization capabilities. Only 10 point cloud object classes are included in these benchmarks. They emphasize the gener- alization among the shared categories between the source and target domain, without considering transferring to unseen new classes, corrupted data, etc, which are frequent in real-worle scenarios.

---

In this paper, three new benchmarks are created to enrich 3D domain generalization evaluation, including base-to-new class generalization, cross-dataser generalization and few-shot generalization. We will dissect the details of benchmark curation and usage in Section A. We supply comprehen- sive experiments and analysis to examine the proposed regulation constraint framework, ablate the effectiveness of distinct components, and draw some new insights from our newly introduced 3DDG evaluation benchmarks. The results verify the proposed method not only enhances the task-specific 3D point cloud recognition but also extends the task-agnostic generalization ability by a clear margin.
In short, the contributions of this work are threefold. Firstly, to our knowledge, we firstly bring the 3DDG problem in front of large multi-modal 3D models and present an effective regulation framework based on lightweight prompt tuning, which not only strengthens downstream 3D task performances but also lifts the domain generalization capability remarkably. Secondly, we implement our regulation framework as a plug-and-play module to seamlessly integrate into the existing large multi-modal 3D models. Consistent improvements are obtained over representative large 3D models, indicating the proposed regulation framework general and model-agnostic. Thirdly, we carefully craft three new benchmarks to enrich the evaluation of 3D domain generalization. Our benchmarks introduce new evaluation dimensions for 3DDG which are vital in the real world but absent in existing ones, including base-to-new, cross-dataset, and few-shot generalization. These new and more challenging benchmarks will drive the future research of 3D domain generalization.
2 Related Work
3D domain generalization. Although domain generalization has been widely studied in image recognition [17, 28, 34, 29, 31, 30, 32. 83, 21, 87, 5, 38, 13, 84, 80], it is still not the case for 3D. A large body of works in point cloud recognition focuses on improving the performances on specific benchmarks by supervised [48, 49, 33. 67. 60, 63, 18, 69, 82, 37, 68, 50, 14, 44] or self-supervised [73, 43, 75, 57] learning. However, they lack systematic strategies to address the generalization challenge and related evaluation absent. Only a few methods investigate the 3D domain adaptation [51] and domain generalization [20, 64] problem. They either create common feature space between the source and target domain (e.g., PointDAN [51], PDG [64]), or utilize the meta-learning framework (e.g.. MetaSets [20]) to obtain robust representations to handle the domain shifts. Nevertheless, those methods are all built on small-size point cloud encoders targeting small-scale datasets and the overall generalization is unsatisfactory. In contrast, we explore the 3D domain generalization based on representative large multi-modal 3D models, like PointCLIP series 176, 90] and ULIP series [7 72]. Mcanwhile, we do not touch the backbone and only conduct lightweight prompt tuning on those large 3D models.
Prompt learning for large 3D models. Prompt learning for 3D point cloud understanding has been studied in recent works [74, 58, 59, 88]. IDPT [74], Point-PEFT [59] and DAPT [88] explore this problem in pure point cloud modality and do not establish connections with flexible language descriptions, thus these methods cannot conduct open- voccbbocaullary 3D recognition. PPT [58] firstly constructs prompt learning pipeline based on the multi-modal framework ULIP [71]. It achieves open-vocabulary recognition with promising performances and relatively small costs. Our work is closely related to PPT but distinguished in the following aspects: First, PPT focuses on optimizing specific 3D tasks with learnable prompts and fails to consider generalization on unseen data. Instead, our work develops systematic strategies to regulate prompt learning to boost generalization well as target tasks. Second, PPT only introduces learnable prompts in the text branch while our method conducts multi-modal prompt tuning on both text and 3D branches.
3 Method
We firstly revisit lightweight prompt learning for existing large 3D models in section 3.1. Then, a comprehensive regulation framework is proposed to promote the generalization capability of large 3D models based on the and--lay prompt tuning strategy in section 3.2. Finally, we introduce the implementation details of the devised method in section 3.3. The overall pipeline of our method visualized in Fig. 2. The creation and analysis of our new 3DDG benchmarks are elaborated in Appendix A.1 due to space limitation.
3

---

Encoder Layer x
Encoder Layer 8
hh
locked
' updatable
Mutual Agreement Constroint
The
Diversity
Model Ensemble Constroint
Distribution Prediction
LD
Encoder Layer P
Encoder Layer =
LLM
weightt
hh
L
hr
3
4 ...
Figure 2: The overall architecture of our point cloud analysis prompt regulation constraint framework, namely Point-PRC, consisting of three core components as in the figure.
3.1 Preliminary
Existing large multi-modal 3D models [76, 90, 71, 72] have different branches that encode the inputs from point cloud and text. In the 3D branch, point cloud E RN divided and projected into point patches. Then, class token Pets E Rd is inserted before the patches to form the input P = {Pots, P2,2., ....... ] E [R(+a)xd of the 3D encoder fp(-, 0p), where Op represents the encoder parameters. In the text branch, the descriptions of each 3D category are converted into the sequence T {l L,, ..,.... bur bey beos } (3++t)xd for the encoder fr(-;0:) Here the embedding of {class), Lans and leos stands for the the start and end flag token of a sentence. So can obtain the 3D features hp fr (P, 0p) and text features hy [T(T, 0t). When executing zero-shot recognition for a downstream 3D dataset of C' categories, 8p and 0, are frozen, the model outputs the class probability distribution D for point cloud P by computing exp(sim(hr, hr)/r) Σ exp(sim(bp. hz/)/r) where sim(;; measures the cosine similarity of the inputs and T a temperature coefficient.
Although zero-shot inference is flexible. the performances on target tasks may not be satis- factory. Multi-modal prompt learning introduces learnable prompts in the inputs of different branches. Specifically, we insert r learnable prompts E = {af, cl .e...................... . ee.,,, E Rexd into and 8 learnable prompts E = {ef eg. e?") E Rexd into T, respectively. Thereupon, the modified inputs for point cloud and text encoder become P = {Poles (Pole.Pi..... Pi...., Pu. of,, ...,....e,, ef") and After transforming by the encoders, we obtain the new point cloud and text representations denoted with hp fP( 6p) and h fT 0z), where 0 (0p, E.") and 0, (0,, E Similarly, the predicted class distribution D and optimization objective can be formulated by Eq.
= (lans, (1, .2...., lv, le, e?, ).oo).
D exp(sim hT)
(E E arg min E Lor (1)
DS ex=(smiii (1).)))((pp..)/))))) (TT))))
{E" E")
where Dgr is the ground truth distribution of point cloud data and is the category of point cloud P. Note that 0p and 0, are still frozen and only E and ET are updatable with the cross entropy (CE) loss. Also, the learnable prompts can be inserted at each layer of the 3D and text encoder, not only at the very first layer. We call this scheme deep multi-modal prompt learning that will be regarded as important baseline in our experiment settings.
Our Regulation Framework
Prompt learning aims to elicit well-leamed knowledge of pre-trained large models by introducing a small number of learnable parameters in the input space. But optimizing the learnable prompts targeting specific datasets easily compromises the general knowledge. To handle the above problems, we propose a comprehensive regulation framework consisting of three components: mutual agreement constraint, text diversity constraint, and model ensemble constraint, claborated below.

---

3.2.1 Mutual Agreement Constraint (MAC)
Large foundation models unfold overall better robustness and transferability broad spectrum of evaluations than conventional models learned on specific datasets, supported by representative works in vision [9], language [53, 54, 4J. and multi-modal understanding [52, 23, 15, 45, 86, 85, 90, 71, 19].
The first component of the proposed framework is to interact with large 3D models actively by maximizing the mutual agreement between learnable prompts and pre-trainnt t-tirraned knowledge.
Specifically, we engage with large 3D models by aligning extracted features and predicted distributions simultaneously. Let us denote the frozen point cloud feature extracted by the 3D foundation model as hp, the point cloud feature containing learnable prompts as hp. Now we compute the difference between hp and hp and mark it as Lp. Similarly, in the text modality, we have Le which measures the difference between by and hy- On the other side, D and DD are two class distributions given by the frozen and promptable large 3D models, respectively. The difference between D and D denoted as CD. Our mutual agreement constraint aims to minimize the feature and prediction distribution discrepancy to ensure the learning trajectory not to forget the task-agnostic knowledge in large pre-trained models.
Sp hi h
C, -=
D - CD
CD= - DDEDRI(D) D.)
(2)
As formulated in Eq. 2, L1 distance is employed to compute Lp and Lt, and Hellback--eiblle (KL) divergence is used characterize the distribution discrepancy. We will examine the design choices in the ablation study.
3.2.2 Text Diversity Constraint (TDC)
Inspired by the flexibility and versatility of language expressions, we propose to leverage diverse text descriptions guide the lightweight prompt tuning to produce transferrable features. Specifically, obtain multiple text descriptions for each point cloud object category by prompting L.MMs (e.g.. GPT- 3.5 [1]. GPT-4 [41], PointLLM [70]) utilizing manual templates. Then, aggregate the text feature of all descriptions for each single category by pooling operation, hy AvgPool bzy)), which will integrate rich semantic information extracted by powerful large models, prevent a point cloud category biasing towards some specific descriptions and finally enhance the model transferability. In the case of describing point clouds with LLMs, we design three kinds of prompts, including question answering, caption generation, and making sentences using keywords, as demonstrated in Fig. 3. For each instruction to the LLM, we acquire N,- 10 responses.
Question Answering
Question Answering
Caption Generation
Making Sentences
What does a(n) (class) point cloud look like?
What are the identifying features of a(n) (class)
Please describe a(n) (class) point cloud with details
Make a meaningful sentence with the following words: (class), point cloud
point cloud?
Figure 3: Illustration of diverse questions to LLMs, including GPT-3.5, GPT-4 and PointLLM The responses given by LLMs are regarded as the text descriptions to the point cloud and fed into the text encoder.
3.2.3 Model Ensemble Constraint (MEC)
The model ensemble constraint aims to synthesize the opinions from different models by weighted voting avoid some extreme and failure cases of single model. The idea has been widely discussed in statistical machine learning [42, 12] and deep learning [16, 40]. Robust tuning of
large models by ensemble learning also has been studied in recent literature [65, 22]. The ensemble strategy mainly involves interpolating weights between zero-shot and fully fine-tuned large models. But it has not been investigated in the context of prompt tuning for large 3D models and its effectiveness is unknown. In this paper, we propose to ensemble models by aggregating the model parameters in different training epochs with Gaussian weighted strategy. The basic idea is that in the initial learning stage, the prompts are randomly initialized and not well optimized so distribute them very
5

---

small weights. As the training iterates, the model gradually gets a sense of downstream tasks; thus, increasing weights are assigned to the model parameters in these epochs. As the training ends, the learnable prompts are adjusted well to adapt downstream datasets while having the risk of overfitting. so we decrease the weights to the model parameters. The varying weights of the above process can be approximated by a gaussian curve. Finally, the weighted models in different epochs are ensembled to generate the model parameters O, and Op. shown in Eq. 3.
8
O
B
w
(3)
i=1
i=
where is the number of epochs and wy ov2n exp(- (i=+)) 20 μ and o² represent the mean and variance of a gaussian distribution. 02 {0i E... and 0; (0i b² indicate the model parameters after the ith epoch of training in the text and point cloud branch, respectively. Note that simple accumulated addition can implement Eq. 3 and we do not need to store all e copies of the parameters, referring to Appendix for details.
Optimization. The overall optimization objective consists of two parts, the task-specific cross entropy loss LCE and the task-agnostic regulation constraint loss LRC. displayed in Eq. 4, where o, 3, y are hyperparameters. Unlike trivial prompt tuning multi-modal large model on downstream tasks, this design allows the learnable prompts to actively interact and align with the general knowledge in pre-trained large model while learning on specific 3D tasks.
L = LCE Lnc,
Lnc aL + tLD
3.3 Implementation Details
We choose PointCLIP [71], PointCLIP V2 [71], ULIP [71], and ULIP-2 [72] the 3D foundation models for experiments. All experiments are rumning with three random seeds and we report the mean and standard deviation. The learnable prompts are inserted into the inputs of first 9 Transformer layers in these models and the prompt length is set to 2. Unless specified, the prompts are optimized using 16-shot learning. Note that previous 3DDG methods [51, 20, 64] use the full training set. We set a 10, 3 = 25 and = 1. The optimizer is SGD, the initial Ir is 0.0025 and we use cosine scheduler to update it. More details about the model configuration can be found in Appendix. We will justify the design choices in the ablation study.
4 Experiments
In this section, we first explain the evaluation settings of our newly curated and existing benchmarks. Then, comprehensive comparison and analysis across various generalization settings are presented to show the advantages the proposed method. Finally, we justify the effectiveness of different components in regulation framework through systematic controlled experiments.
4.1 3DDG Evaluation Settings
Base-to-New. This benchmark includes 5 point cloud datasets which are ModelNer40 [2), three vari- ants of ScanObjectNN [61] (S-PB_TT) FPP__SS__RS_ RS, S-OBJ_BO __G,
and ShapeNetCore [6]. Each dataset equally split into base and new classes, where the former used for prompt tuning while the latter only serves the test purpose. Cross-Dataset. This benchmark has four types of evaluation, including 0OD generalization, data corruption, PointDA [5]] and Sim-to-Real [20]. We established the first two assessments and the latter two already existed. For OOD generalization, models are trained on the source domain and evaluated on the target domains. For data corruption, models are trained on clean ModelNet [2] and tested on the corrupted data in ModeINet-C[56]. Few-Shot. This setting inspects the model generalization an extremely low-data regime, where 1, 2, 4, 8, and 16 shots are randomly sampled for prompt learning, and the recognition accuracy is calculated on the whole test set, respectively. The explanations are brief and we encourage the readers to check the details in Appendix.

---

Table I: Base-to-new class generalization comparison for representative large 3D models based on prompt learning. Each number here is the mean of three runnings. Base: base class accuracy (in %, same below). New: new class accuracy. HM: harmonic mean of base and new class accuracy.
+ +RC demonstrates the models with our regulation constraint framework.
(a) Average over 5 datasets
(b) ModelNet40
(c) S-PB_-PPPP _RS
Method
New
Method
New
HM
Method
Basc
New
P.CLIPT81]
75.66
23.45
35.80
P-CLIP 176]
53.23
20.22
33.23
P.CLIP F16]
61.25
19.57
30,01
P-CLIP2 190
24.11
37.84
50.10
PCCLI2 [90)
45.21
61.08
P.CLIP2 [90]
56.81
29.92
39.20
1713
77.32
69.01
ULIP
50.07
65.05
ULIP
56.73
25.80
+RC(Our))
82.19
61.93
70.64
+RC(Our))
95.03
55.27
69.89
+KC(Ours)
64.20
49.17
$5.69
[1318-2
77.91
67.91
72.57
(12)
91.77
56.47
69.92
11IIP-2 [72] -RC(Ouns)
66.40
66.47
+RC(Ours)
83:18
76.10
79.48
+RC(Ours)
95.30
64.83
77.17
73,67
74.27
73,97
(d) S-OBJ_BO BG
(c) S-OBILONLY
(f) ShapeNetCoreV2
Method
Basc
Mathed
HM
Method
Rase
New
176]
72.82
23,00
34.96
P-CLIP (76)
20.23
51.97
P-CLIP [76]
74.78
53.92
46.61
PICLIP2 190]
20.07
35.08
46.75
PACLIPE 1901
71.40
44.39
54.74
P.CLIP2 [90]
78.27
34.58
47.97
ULIP (71)
13.20
47.17
$7.37
ULIP (71)
24.13
50.80
60.29
ULIP (71]
89.73
71.20
79,50
+RC(Ours)
79.47
$5.20
65.15
+RC(Ou)))
79.23
65.93
71.97
-RC(Oun))
93.03
84.10
88.34
ULIP-2
77.00
83.27
80.01
(12
78.60
76.27
77.42
ULIP-2 [72]
75.80
$7.07
65.38
+RC(Chur))
80.10
88.93
84.28
+ROOas))
83.00
81.10
82.33
-RC(O
X3.23
71.37
76,85
4.2 Base-to-new Class Generalization
In this benchmark, models are learned on the base classes and evaluated on the test sets of base and novel classes. In addition to ULIP and ULIP-2, we also implement the same prompt tuning for PointCLIP [76] (P-CLIP) and PointCLIP V2 [90] (P-CLIP2) for comparison, shown in Tab. 1.
Loss of Generalization in P-CLIP and ULIP Series. We observe notable gaps occur between base and new class recognition accuracy of P-CLIP, P-CLIP2, ULIP, ULIP-2 when prompt tuning without the proposed regulation constraints. For instance, P-CLIP2 achieves 93. 93.98% accuracy on the base classes of ModelNet40 while dropping by 48.77% absolute points on the whole test set of the new classes, which even lags behind the zero-shot accuracy of the frozen P-CLIP2 (64.22%). The results are consistent across five datasets, suggesting the loss of generalization of original models.
Lifting the Generalization by Our Framework. As shown in Tab. 1, the proposed framework composite of three regulation constraints boosts the unseen class recognition accuracy across different models and datasets by a clear margin, thanks to the active communication and alignment with the general knowledge in large 3D models. For example, the improvement of the harmonic mean on ULIP reaches 10.65% absolute points averaged over datasets.
Lifting the Specific 3D Tasks by Our Framework. Surprisingly, the task-specific performances not be hindered by the regulation constraints while enhancing the task-agnostic generalization, referring to the base class accuracy of ULIPP+C and ULIP-2+RC averaged over datasets, increasing by 4.87% and 5.27%, respectively
4.3 Cross-Datases Generalization
This setting differs from the base-to-new counterpart where the base and new classes belong to the same dataset. We present the analysis for OOD generalization and data corruption as below, and put the comparison on Sim-to-Real and PoinDDA in Appendix.
OOD Generalization demonstrates the models' transferability other unseen domains by learning from an existing domain. To evaluate on this benchmark, we implement the lightweight prompt learning for ULIP and ULIP-2 then impose the proposed regulation constraints on them. Prompt learning for P-CLIP [76] and P- P-CLIP2 [76] with same settings are also implemented for comparison. The results are reported in Tab. 2. By wrapping ULIP and ULIP-2 with the devised framework, we achieve consistent positíve gains on each of the five target domains. The average gains over them are enlarged with increasing ability of ULIPP e.g., +6.20% for ULIP-- vs. + +1.79% for UL.P. Meanwhile, we notice that the performances on Omni3D [66] are rather limited and the methods here seem not to
7

---

Table 2: Comparison of OOD generalization in cross-dataset benchmark. ShapeNetV2 serves as the source domain and the other five datasets are deployed as the target domain. ShapeNetV2: 55 classes, ModelNet40: 40 classes, SONN: 15 classes, Omni3D: 216 classes. Some common categories are shared between the source and target domain. Note that Omni3D has much more new 3D object concepts than others. The last column indicates the average over five target datasets.
Source
Target S-OBJ_BI
Method
ShapeNetV2
ModelNet40
S-PB_150__K
S-OBJ_ONLE
Ommi3D
Avg.
P-CLIP [76]
67.41(0.09)
33.20((186)
15.51(0.588
18.59(1.40)
22.89(2.32)
0.48(0.17)
22.55(1.54)
P-CLIP2 [90]
68,93( .43)
54.73(1.48)
39.53(4.22)
34,3K( (.28)
25.63(1.16 16)
8.63(2.52)
32.56(2.13)) 13)
+RC(Ours)
69.80(2.86)
55.37(1.78)
39.77(0.45)
34.20(0.54)
24.50(1.20)
10.20(0.40)
32.81(0.89)
ULIP [71]
87.33(0.95)
16.17(1.15)
26.83(2.15) 15)
39.43(2.17)
43.53(1.32)
6.37(0.90)
34.47(1.54)
+RC(Ours)
90.43(0.86)
58.00(0.57)
28.43(0.68)
40.33(0.71)
46.33(
8,20(0.50)
36.26(0.80) 80)
ULIP-2 (72] +RC(Ours) 76.70(1.59)
76.70(1.37)
65.27((0.6)
40.07(0.34)
23.80(1.78)
48.53(1.72)
17.27(0.54)
44 99.9991 L01))
72.10(0.933
46.77(2.43)
59.03(3.02)
56.27(0.97))
21.80(0.49)
51.19(1.57)
Table 3: Comparison of corruption generalization on ModelNet-C[56] when trained on clean data. The results are reported for the corruption severity=2 ModelNet-C.
Methed
Clean Pata ModelNet
Add Global
Add Local
Drop Global
Corruption Type
D2op Local
Rotale
Scale
Jitler
Avg.
PECLPP 176]
80.952(1.22 (.2))
80.9711022
80557(102))
64.95(1.00 (8)
(8.11(155))
65.75(1.19)
72.4(1.33))
$2,08
09.3001.20)
83.4900.51)
83.4M0.511)
83.49(0.51)
68.85(3:22)
66.67(1.96)
70.13(1.33)
75.6810.15)
61.21/2.16)
72.79(1.41)
(LPP
82,43(1.25)
82,5000.99)
8.127)
80.771
65.43(1.02)
72,27(1.56)
74.67(155))
45,6000.65)
71.93(114)
+RC00m)
83,87.0.34 34)
83.83(0.40)
83.930.199
81.830.522
67.37(1.73)
79.1000.36)
76.37(0.09)
41.67(4.79)
73.4((115))
(12)-2
85.0700211
81.9700.79
#2.01/0.96)
79 90.9330922)
60.03(1:21)
30)0-3)))
75:77(0:74)
44.27(2.33
72.04(1.00)
+RC(0us))
86.47(0.56)
86.57(0.48)
86.3900.51
84.87(048)
67.80(1.20)
84.60/0.22)
81.17(1.05)
46.43:2.45)
76.82/0.91)
work, especially for P-CLIP series and ULIP (less than 10% accuracy). This dataset contains large vocabulary of real 3D objects (216 categories) and exhibits the long-tail attribute. When transferring the models that learn from a narrow set 3D object concepts (55 classes in ShapeNetV2) to Omni3D, they suffer from new 3D concepts thus perform poorly.
Data Corruption are common in point clouds due to complex geometry, sensor inaccuracy and processing imprecision. We investigate the generalization of the proposed framework on ModelNet- C [56]. which includes common corruptions, such as dropping some parts or adding global outliers. The compared methods are same as those in OOD generalization and the results are exhibited in Tab. 3. Our method not only boosts the recognition accuracy on clean data (+1.44% for ULIP and +1. +1.40% for ULIP-2). but also strengthen the robustness of representative large 3D models against collapsed data. By averaging on 7 types of corruption, we receive +1.51% and +4.78% gains for ULIP and ULIP-2, respectively.
4.4 Few-shot Generalization
In this setting, ULIP and ULIP-2 with (w.) and without (w.o.) our regulation constraints (RC) are compared. As visualized in Fig. 4, the solid lines of ULIP and ULIP-2 exceed the corresponding dashed lines by clear margins average over 5 datasets, indicating the devised framework strengthens the 3DDG capability considerably. The advantages are enlarged especially for the extreme 1-shot learning, e.g., +8.05% acc. for ULIPP and +5.39% acc. for UL.IP-2. Note that in some cases, 6.g-, on ModelNet40, ULIP-2 w.o. RC (1-shot, 66.63% even lags behind zero-shot ULIP-2 (71.23%) implying that simple prompt tuning disturbs the well-learned representations of UL.IP-2. In contrast, the developed framework brings 2.4% absolute improvements over the zero-shot ULIP-2, obtaining 73.63% acc. under the 1-shot setting.
4.5 Ablation Study
In this section, we examine the effectiveness of several critical components in the proposed framework via a series of controlled experiments. UI.IP-2 is adopted as the baseline and we compare the variants on the base-to-new benchmark and report the harmonic mean (HM) averaged over 5 datasets.
8

---

Average over detasets
ModeRRub43
5-P9,330
CesEEno d 201 TAAST
IEPA I IEP7en
Number shote 5-08) BG
Namber shets
Number of shots ches
5-ORj
ShapeNetCama2
KOoni an the Bast aett 1743
1ETT
LEFA F LEP2a EC
11F R:
00.02
HE2A
0.
shobs per class
abota per class
Number of abota pur claxs
Figure 4: Comparison of few-shot generalization. The solid and dashed lines represent the models with and without our framework. Zero-shot performances of ULIP and ULIP-2 marked with star symbols. The figure in the upper left presents the average results over datasets.
Regulation constraints. Three components in framework vital for generalization enhance- ment. We verify their effectiveness by adding/deleting the components on ULIP-2. The results in Tab. 4 indicate there exists notable performance gap (6.91%) between ULIP-2 with and without the regulation constraints. And the gap can be gradually narrowed down by inserting different compo- nents. For instance, ULIP-2 with the model ensemble constraint lifts the HM from 72.57% 78. 78.89% a 6.32% absolute increase. Although single mutual agreement or text diversity constraint may not bring adequate gains, their combination contributes the generalization improvement significantly, achieving 79.26% that is close to the performance of the full version of our framework.
Distance metrics in MAC. The mutual agree- ment among the extracted features with learnable prompts (c.g. hy and hy) and the general knowl- edge in large 3D models (c.g. by and hp) can be implemented in different distance metrics. Here explore their effect and report the results in Tab. Sa. As observed, MAC with MSE distance attains the best recognition
Table Ablation study for the three regu- lation constraints in our framework. The results are averaged on datasets.
MAC
TDC
MEC
Base
New
HM
72.57
the base classes. But when incorporating the performances of new 3D categories into account, the same model with LI distance demonstrates overall better generaliza- tion 76%10% new acc. and 79.48% HM). Thus we choose LI as the distance metric in MAC by default.
/
X
78.02
68.91
73.18
X
84.54
72.19
78.89
84.92
71.17
77.44
X
83.19
73.35
77.96
V
83.30
75.59
79.26
Table 5: Ablation studies. The results are averaged over 5 datasets in the base-to-new benchmark.
(a) The distance metries in MAC. LI: LI norm,
(b) Here GPT-3.5 is short for GPT-3.5-turbo.
MSE; mean square emorr Cosine: cosine distance.
Text Description
New
Metric
Base
New
HM
GPF:3.5 [1]
83.30
71.44
76.92
Cosine
78.63
73.73
76. 10
GPT4 [41]
83.46
71.55
77.05
MSE
83.91
PointLLM [70]
83.27
73.83
78.27
1.I
88
76.10
79.48
Manual
76.10
79.48
Point cloud descriptions from different sources. We hope to exploit flexible and diverse text descriptions to reflect some vital characteristics of the point clouds in different classes. The following

---

experiments investigate the effect of the point cloud descriptions generated from different sources, including large language models like GPT-3.5 [1]. GPT-4 [41]. PointLMM [70] and manual templates (see Appendix for details). As shown in Tab. 5b, point cloud descriptions from general-purpose LLMMs, such as GPT-3.5 and GPT-4, bring decent performances on base classes. However, they lag behind PointLLM regarding new class recognition accuracy by clear margin (-2.39% for GPT-3.5 and -2.28% for GPT-4). We infer it is due to the fact that PointLLM has seen massive point cloud data and related text descriptions thus generates more accurate and domain-related responses. Surprisingly, by combining accuracy and the best performance with LLMs' descriptions.
simple sentences written by human beings [7 I, ULIP-2 achieves decent base class
classes, resulting in even better HM than that of ULIP-2
The depth and length of learnable prompts, Two variables that should be determined for the learnable prompts (E", E") are the depth of prompt layers D and the length of prompt tokens L. For simplicity, the prompt depth is kept the same in the point cloud and text encoders, and similarly for the prompt length. We ablate the two variables and visualize the results in Fig. 5. In general, increasing the prompt layers promotes the harmonic mean. But is not always beneficial to deepen the learnable prompts, e.g. ULIP-2 with D = 12 achieves 78.26% HM, slightly lower than 78.67% HM of ULIP-2 with D 9. We also find that it not necessary to construct very long prompt tokens to achieve better generalization, e.g., ULIP-- with I, = 2 surpasses other variants average on datasets clearly. Thus we let 9 and I. by default.
Ablation Study: Depth Learnable Prompts
Ablation Study: Length of Learnable Prompts
the set (%)
Harmonic mean the set
02.5
82.5
80.0
77.5
77.5
75.0
75.0
72.5
70.0
5-08), ONLY
5-08), BG
Harmonic
70.0
5:08| ONLY
_.08_BG
SPB
67.5
Shape/etCorev2
ShapeNelConwV2
Average
Average
85.0
Depth of Learnable Prompts
Length of Learnable Prompts
Figure 5: Ablation study for the prompt depth and length. We compare the harmonic mean on
five datasets of the base-to-new benchmark and the average results are displayed in dashed lines.
5 Conclusion
This paper initializes the efforts of addressing the corrupted generalization of large 3D models when adapting to downstream 3D tasks by a comprehensive regulation framework. The framework enables the learnable prompts to actively engage with large 3D models by maximizing the mutual agreement between task task-specific prediction and general knowledge. Consistent generalization gains are obtained over different large 3D models, suggesting the model-agnostic attribute of the proposed framework. We also contribute to the study of 3DDG by developing new and more challenging evaluation benchmarks that will drive further investigation. Nevertheless, this work focuses on the point cloud recognition, and we plan to discuss the segmentation and detection tasks in future work.
Limitations and Broader Impacts. The proposed framework has demonstrated effectiivnness and scalability on the object-level recognition task but not been validated on the scene-level tasks, such as 3D semantic segmentation and object detection. Different solutions may be required to handle scene-level point cloud data. On the other hand, when exploiting the power of LLMs to reflect critical characteristics of 3D objects, we simply ensemble multiple descriptions through the pooling operation, more sophisticated prompting and fusion strategy can be developed. For broader impacts, we are the first to investigate the generalization ability of large multi-modal 3D models, which mirrors the progress of the vision-language field (CLIP-basee image recognition) and probably inspires series of follow-up works. We do not perceive the potential negative impacts of this work.
10

---

Acknowledgments and Disclosure of Funding
We thank all anonymous reviewers and area chairs for their time and valuable feedback. This work was partially supported by China Scholarship Council (CSC) under the Grant No. 202306360147 and partially supported by NeurIPS 2024 Student Travel Grant.
References
[1] Openai chatgpt. https://chatgpt.comm Accessed: 2024-05-06.
The princeton modelnet. https://mooeeeetttttttt/////// www.eeeeeeeeeeccc.eeeeeoeeoooooo edu/. Accessed: 2012-12-27.
[3] J. Behley and C. Stachniss. Efficient surfel-based slam using 3d laser range data in urban environments. Robotics: Science and Systems X/V, 2018.
[4] T. Brown, B. Mann. N. Ryder, M. Subbiah, D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger. T. Henighan, R. Child, A. Ramesh, D. Ziegler. J. Wu, C. Winter, Hesse, M. Chen, Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle,
M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[5] M.-H. Bui, T. Tran, A. Tran, and D. Phung. Exploiting domain-specific features to enhance domain generalization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 21189-21201. Curran Associates, Inc., 2021.
[6] A. X. Chang. T. A. Funkhouser, L. J. Guibas, P. Hanrahan, Q Huang. Z. Li, S, Savarese, M. Savva, S. Song. H. Su, J. Xiao, L. Yi, and F. Yu. Shapenet: An information-rie rich 3d model repository. CoRR, abs/1512.03012, 2015.
[7] G. Chen, W. Yao, X. Song, X. Li, Y. Rao, and K. Zhang. Prompt learning with optimal transport for vision-language models. In ICLR, 2023.
[8] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Niessner. Scannet: Richly- ainoty-adnotated 3d reconstructions of indoor scenes. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[9] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, Gilmer, A. P. Steiner, M. Caron, R. Geirhos, Alabdulmohsin, R. Jenatton, L. Beyer, M. Tschannen, A. Arnab, X. Wang. C. Riquelme Ruiz, Min- derer, J. Puigcerver, U. Evci, M. Kumar, S. V. Steenkiste, G. F. Elsayed, A. Mahendran, F. Yu, A. Oliver, F. Huot, J. Bastings. M. Collier, A. A. Gritsenko, V. Birodkar, C. N. Vasconcelos, Y. Tay. T. Mensink. A. Kolesnikov, F. Pavetic, D. Tran, T. Kipf, M. Lucic, X. Zhai, D. Keysers, J. Harmsen, and N. Houlsby. Scaling vision transformers to 22 billion parameters. In A. Krause, F. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 7480-7512. PMLR, 23-29 Jul 2023.
[10] M. Deitke, R. Liu, M. Wallingford, H. Ngo, O. Michel, A. Kusupati, A. Fan, C. Laforte, V. Voleti, Y. Gadre, E. VanderBilt, A. Kembhavi, C. Vondrick, G. Gkioxari, K. Ehsani, L. Schmidt, and A. Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 35799- 35813. Curran Associates, Inc., 2023.
[11] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kemb- havi, and A. Farhadi. Objaverse: universe annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13142-13153, June 2023.
[12] T. G. Dietterich. Ensemble methods machine learning. In Proceedings of the First International Workshop on Multiple Classifier Systems, MCS '00, page 1-15, Berlin, Heidelberg. 2000. Springer- Verlag.
[13] Y. Ding, I.. Wang, B. Liang, S. Liang, Y. Wang, and Chen. Domain generalization by learning and removing nomain-speciffe features. In S. Koyejo. Mohamed, A. Agarwal, D. Belgrave. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24226-24239. Curran Associates, Inc., 2022.
[14] L. Duan, S. Zhao, N. Xue, M. Gong, G.S. Xia, and D. Tao. Condaformer: Disassembled transformer with local structure enhancement for 3d point cloud understanding. In seveth Conference on Neural Information Processing Systems, 2023.

---

[15] A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). In K. Chaudhuri, S. Jegelka. L. Song. C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2016-6234. PMLR, 17-23 Jul 2022.
[16] M. Ganaie, M. Hu, A. Malik, M. Tanveer, and P. Suganthan. Ensemble deep learning: A review. Eng. Appl. Artif. Intell., (C)( oct 2022.
[17] M. Ghifary, W. B. Kleijn, M. Zhang, and D. Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 25519,55,,25 2015.
[18] M.-H. Guo, J.XX Cai, Z.NN. Liu, T.J. Mu, R. R. Martin, and S.-M. Hu. Pet: Point cloud transformer, 2020.
[19] Z. Guo, R. Zhang, X. Zhu, Y. Tang, X. Ma, J. Han, K. Chen, P. Gao, X. Li, H. Li, and P.-A. Heng. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following, 2023.
[20] C. Huang. Z. Cao, Y. Wang. J. Wang, and M. Long. Metasets: Meta-leaming on point for generalizable representations. In 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1859-8868, 2021.
[21] Z. Huang, H. Wang, E. P. Xing, and D. Huang. Self-challenging improves cross-domain generalization. In Computer vision-ECCV 2020: 16th European conference, Glasgow, UK, August 23-28, 2020, proceedings, part II 16, pages 124-140. Springer, 2020.
[22] G. Ilaarco, M. Wortsman, S. Y. Gadre, S. Song, H. Hajishirzi, S. Kornblith, Farhadi, and L., Schmidt. Patching open-vocabulary models by interpolating weights. In Koyejo, S. Mohamed, Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages
Curran Associates, Inc., 2022.
[23] C. Jia, Y. Yang. Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le. Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4904 4916. PMLR, 18 24 Jul 2021.
[24] M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19113-19122, June 2023.
[25] M. U. khattak, S. T. Wasim, N. Muzzamal, S. Khan, M.-H. Yang, and F. S. Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
[26] D. Lee, S. Song, J. Suh, J. Choi, S. Lee, and H. Kim. Read-only prompt optimization for vision-language few-shot learning. In 2023 IEEE/CVF International Conference Computer Vision (ICCV), pages 1401-1411, 2023.
[27] J. Levinson and S. Thrun. Robust vehicle localization in urban environments using probabilistic maps. In 2010 IEEE International Conference on Robotics and Automation, pages 4372-4378, 2010.
[28] D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[29] D. Li, Y. Yang, Y.Z. Song, and T. Hospedales. Learning to generalize: Meta-lcaming for domain generalization. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.
[30] D. J. Zhang, Y. Yang, C. Liu, Y.-Z. Y.n,.Son,, and T. Hospedales. Fpisodic training for domain generalization. In 2019 IEFF/CVF International Conference on Computer Vision (ICCV), pages 1446-1455, 2019.
[31] H. Li, S. J. Pan, S. Wang, and A. Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
[32] H. Li. Y. Wang, R. Wan, S. Wang, T.-Q. Li, and A. Kot. Domain generalization for medical imaging classification with linear-depente depenneenyy regularization. In Advances in Neural Information Processing Systems, volume 33, pages 3118-3120 Curran Associates, Inc., 2020.
12

---

[33] Li, R. Bu, M. Sun. W. Wu, X. Di, and B. Chen. Pointenn: Convolution on x-transformed points. In Advances in Neural Information Processing Systems, volume 31. Curran Associales, Inc., 2018.
[34] Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[35] Z. Li, X. X.
X. Fu, Zhang, W. Wang. S. Chen, and J. Yang. Prompikd: Unsupervised prompt distillation for vision- languag models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, 2024.
[36] Y. Lu, Liu, Y. Zhang. Y. Liu, and Tian. Prompt distribution learning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5196-5205, 2022.
[37] X. Ma, C. Qin, H. You, H. Ran, and Y. Fu, Rethinking network design and local geometry in point cloud:
A simple residual MLP framework. In International Conference Learning Representations, 2022.
[38] D. Mahajan, S. Tople, and A. Sharma. Domain generalization using causal matching. In M. Meila and
T. Zhang, editors. Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7313 -324.. PMLR, 18 24 Jul 2021.
[39] K. S. McCurley. Geospatial mapping and navigation of the web. In Proceedings of the 10th International Conference on World Wide Weh, WWW'01, page 221-229, New York, NY, USA, 2001. Association for Computing Machinery.
[40] A. Mohammed and R. Kora. comprehensive review on ensemble deep learning: Opportunities and challenges. Journal of King Saud University - Computer and Information Sciences, 35(2):757-774, 2023.
[41] OpenAL. GPT-4. https://openai.commrrreeeaaaa
2023. Accessed January 22, 2024.
[42] D. Opitz and R. Maclin, Popular ensemble methods: an empirical study. J. Artif. Int. Res., 11(1):169-11) jul 1999.
[43] Y. Pang, W. Wang, E. H. Tay, W. Liu, Y. Tian, and L. Yuan. Masked autoencoders for point cloud self-supervised learning. Computer Vision ECCV 2022. Springer International Publishing, 2022.
[44] J. Park, S. Lee, S. Kim, Y. Xiong, and J. Kim. Self polfiiositi point-based based transformer for point cloud understanding. In 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pages 11823, 2023.
[45] H. Pham, Z. Dai, G. Ghiasi, K. Kawaguchi, H. Liu, A. W. Yu, Yu, Y.TT. Chen, M.-T. Luong, Y. Wu, M. Tan, and Q V. Le. Combined scaling for zero-shot transfer learning, 2023.
[46] A. Placitelli and L. Gallo, Low-cost augmented reality systems via 3d point cloud sensors. 2011 Seventh International Conference on Signal Image Technology and Internet- -aaed Systems, pages 188-192, 2011.
[47] C. Qi. O. Litany, K. He, and L. J. Guibas, Deep hough voting object detection point clouds. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), October 2019.
[48] C. R. Qi, H. Su, K. Mo, and L.. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[49] C. R. Qi. L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[50] G. Qian, Y. Li, H. Peng, J. Mai, H. Hammoud, M. Elhoseiny, and B. Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In Advances in Neurul Information Processing Systems (NeurlPS), 2022.
[51] C. Qin, H. You, L. Wang, C.-C. J. Kuo, and Y. Fu. Pointdan: multi-scale 3d domain adaption network for point cloud representation. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
13

---

[52] A. Radford. J. W. Kim, C. Hallacy. A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin. J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-88888888 8763. PMLR, 18- 24 Jul 2021.
[53] A. Radford. J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
[54] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research. 21(140y:1 67, 2020.
[55] H. Ran, J. Liu, and C. Wang. Surface representation for point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11942-18952 June 2022.
[56] J. Ren, L. Pan, and Z. Liu. Benchmarking and analyzing point cloud classification under corruptions. In International Conference on Muchine Learning (ICML), 2022.
[57] H. Sun, Y. Wang. X. Cai, X. Bai, and D. Li. Vipformer: Efficient vision-and-pointclluud transformer for unsupervised pointcloud understanding. In IEEE International Conference on Robotics and Automation (ICRA). 2023.
[58] H. Sun, Y. Wang, W. Chen, H. Deng, and D. Li. Parameter-efficient prompt learning for 3d point cloud understanding. In IEEE International Conference on Robotics and Automation, 2024.
[59] Y. Tang, R. Zhang, Z. Guo, X. Ma, B. Zhao, Z. Wang, D. Wang, and X. Li. Point-peft: Parameter-efficient fine-tuning for 3d pre-trained models. Proceedings of the AAAI Conference on Artificial Intelligence, 38())=511-55199 Mar. 2024.
[60] H. Thomas, C. R. Qi, J.-E. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[61] M. A. Uy. Q.H. Pham, B.-S. Hua, T. Nguyen, and S.-K. Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Octoher 2019.
[62] A. Vaswani, N. Shazcer, N. Parmar, J. Uszkorcit, L. Jones, A. N. Gomez, L. u. Kaiser, and Polosukhin. Attention is all you need, In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[63] Y. Wang, Y. Sun, 7. Liu, S. F. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (TOG), 2019.
[64] X. Wei, X. Gu, and J. Sun. Learning generalizable part-based feature representation for 3d point clouds. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 29305-2933 2000--229.8... 8. Curran Associates, Inc., 2022.
[65] M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Korblith, R. Roelofs, R. G. Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong, and L. Schmidt. Robust fine-tuning of zero-shot models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7959-7971, June 2022.
[66] T. Wu, J. Zhang, X. Fu, Y. Wang, J. Ren, L. Pan, W. Wu, Yang. J. Wang, C. Qian, D. Lin, and Z. Liu. Omniobject3d: varge-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 803-814, June 2023.
[67] W. Wu, 7. Qi, and Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[68] X. Wu, Y. Lao, L. Jiang, Liu, and H. Zhao. Point transformer v2: Grouped vector attention and partition- based pooling. In NeurlPS, 2022.
[69] T. Xiang, C. Zhang, Y. Song. J. Yu, and W. Cai. Walk in the cloud: Learning curves for point clouds shape analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 915 924, October 2021.
14

---

[70] R. Xu, X. Wang, T. Wang, Y. Chen, J. Pang. and D. Lin. Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308. 2023.
[71] L. Xue, M. Gao, C. Xing, R. Martin- Martin, J. Wu, C. Xiong. R. Xu, J. C. Niebles, and S. Savarese. Ulip: Learning unified representation of language, image and point cloud for 3d understanding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.
[72] L. Xue, N. Yu, S. Zhang, Li, R. Martin-Martin, J. Wu, C. Xiong, R. Xu, J. Niebles, and S. Savarese. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
[73] X. Yu, I.. Tang, Y. Rao, T. Huang, J. Zhou, and J. Lu. Point-hert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 19312-19322 June 2022.
[74] Y. Zha, J. Wang. T. Dai, B. Chen, Z. Wang, and S.-T. Xia. Instance-aware dynamic prompt tuning for pre-traines trairrined point cloud models. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[75] R. Zhang. Z. Guo, P. Gao, R. Fang. B. Zhao, Wang, Y. Qiao, and H. Li. Point-m2ae: Multi-scale masked autoencoders for hierarchical point cloud pre-training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27061-27074 Curran Associates, Inc., 2022.
[76] R. Zhang, 7. Guo, W. Zhang, K. I. X. Miao, B. Cui, Y. Qiao, P. Gao, and H. Li. Pointelip: Point cloud understanding hy clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8552-8566 June 2022.
[77] R. Zhang, X. Hu, Cascade of foundation models makes strong few-shot learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15211 1-5222, June 2023.
S. Huang, H. Deng, Qiao, P. Gao, and Li. Prompt, generate, then cache:
[78] R. Zhang, L. Wang, Y. Qiao, P. Gao, and H. Li. Learning 3d representations from 2d pre-trained models via image-to-poide masked autoencoders. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pages 21769-21788 June 2023.
[79] R. Zhang, L. Wang, Y. Wang. Gao, H. Li, and J. Shi. Parameter is not all you need: Starting from non-parametric networks for 3d point cloud analysis. June 2023.
[80] Y. Zhang, M. Zhang, W. Li, S. Wang, and R. Tao. Language-aware domain generalization network for cross-scene hyperspectral image classification. IEEE Transactions on Geoxcience and Remote Sensing. 61:1-12, 2023.
[81] 7. Zhang, X. Gao, and W. Hu. Invariantoodg: Learning invariant features point clouds for out-of- distribution generalization, 2024.
[82] H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages (0259-16268, October 2021.
[83] S. Zhao, M. Gong, T. Liu, H. Fu, and D. Tao. Domain generalization via entropy regularization. In H. I Laroceelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 16197 Curran Associates, Inc., 2020.
[84] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy. Domain generalization: survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4396 2415,
[85] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Conditional prompt learning for -lsion-language models. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). 2022.
[86] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for dision-language models. International Journal of Computer Vision (ICV)), 2022.
[87] K. Zhou, Y. Yang, Y. Qiao, and T. Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2021.
[88] X. Zhou, D. Liang, W. Xu, X. Zhu, Y. Xu, Z. Zou, and X. Bai. Dynamic adapter meets prompt tuning: Parameter efficient transfer learning for point cloud analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattenn Recognition, 2024.
15

---

[89] B. Zhu. Y. Niu, Y. Han. Y. Wu, and H. Zhang, Prompt-aligned gradient for prompt tuning. In 2023 IEEE/CVF International Conference Computer Vision (ICCV), pages 15613-15623, 2023.
[90] X. Zhu, R. Zhang, B. He, Z. Guo, Z. Zeng, Z. Qin, S. Zhang, and P. Gao. Pointelip v2: Prompting clip and gpt for powerful 3d open-world learning. In Proceedings the IEEE/CVF International Conference Computer Vision (ICCV), October 2023.
16

---

A
Appendix
A.1 Our New 3DDG Benchmarks
To our knowledge, PointDA [51] and Sim2Real [20] are rare benchmarks in 3DDG. We perceive that existing 3DDG benchmarks may not be sufficient to cover common generalization evaluation scenarios. For instance, PointDA [51] selected 10 shared classes among three popular point cloud datasets [2. 8, 6] for generalization evaluation. Sim2Real [20] picked 9 same categories among ShapeNet [6] and ScanObjeciNN [61] and 11 shared classes in ModelNet [2] and SeanObjecINN [61]. Both of them emphasize the generalization among shared 3D object classes different datasets. But they fail to consider how to transfer to unseen 3D object classes and other of distribution scenarios. To alleviate the drawbacks, we develop three new benchmarks, including base-to nnse-to-new, cross doss-eatteet, and sew- generalization to enrich 3DDG evaluation and drive future research.
A.1.1 Construction
Base-to-new Class Benchmark. Inspired by the evaluation settings in 2D vision [85], we take five 3D datasets, including ModeINet40 [2], three variants of SeanObjectNN [61] (S-PB_TS0) TSO_RS, S-OBJ_BOR S-OBILONLY ONLY), and ShapeNetCoreV2 V2 [6] to construct the base-to-new benchmark. Each of them equally divided into two halves object classes, called base and new classes. Specifically, the first half is regarded as base and the second half is treated as new. The train, val, test sets are split for the base classes while the new classes only serve the test purpose. The splitting adopts the official standards of the released datasets if available (e.g.. ModelNet40 and ShapeNetCore V2), otherwise (c.g.. 3 variants of ScanObjectNN) we randomly selected 20% samples in the original training set be the validation set and keep the remaining 80% as new training set. The original test set unchanged to serve as the test sct. This arrangement allows the model to be trained on base classes then tested on new classes so that can measure its generalization on unseen new categories and data.
Table 6: Statistics of the Base-to-New benchmark.
ModeINer40
S-OBJ_ONLY
S-OBJ_BG
S-PB_TS0
T50 RS
ShapeNetCoreV2
Item base
Dew
base
new
buse
new
buse
new
buse
new
Aclasses
20
20
7
8
7
28
27
Strain
4,084
N/A
1,055
N/A
1,062
NIA
5,321
N/A
23,584
NIA
#val
1,028
N/A
281
N/A
274
N/A
1,268
N/A
3,401
N/A
#1est
1,202
1,266
352
229
352
229
1,741
1,141
6,960
3,501
Gross-Dataset Benchmark. Previous Sim2Real [20] and PointDA [51] evaluate the generalization across totally shared categories in different datasets. But in real world the source and target domains do not necessarily have common classes. In this work, construct the cross-dataset benchmark to expand the generalization evaluation broader scopes, which incorporates two newly introduced settings, OOD generalization and data corruption, and another Sim2Real [20] and PointDA [51]. Note that the last two datasets are released by other researchers for 3DDG evaluation and we just follow their default settings.
OOD Generalization. In this setting, the source and target domain may not necessarily share common categories. Also, the number of object classes can be different. ShapeNetCoreV2 [6] arranged as the source dataset while ModelNer40 [2], the three variants of ScanOhjectNN [61] and Omni3D [66] serve the target evaluation places. Apparently, this design is specially for large 3D models that have open-vocabulary recognition ability since traditional 3DDG methods like MetaSets [20], PDG [64] are only able to recognize a fixed set of point cloud
classes.
Data Corruption. Point cloud corruption is inevitable due to irregular geometry structures, inaccurate sensors or processing errors. Existing 3DDG benchmarks fail take this factor into account. We utilize the off- ait-the-heell point cloud corruption dataset ModelNet-C [56] as a place to measure the robustness and generalization of point cloud recognition methods against common data corruptions, such as losing local parts, global noise, etc.
Few-shot Benchmark. This benchmark incorporates same datasets with those in the Base-to-new Class Benchmark. But we do not distinguish the base and new classes and treat them as a whole. During prompt learning on each dataset of this benchmark, randomly sample 1, 2, 4, 8 and 16 shots from each category to tune the learnable prompts, then observe the generalization on the whole test set.
A.1.2 The Evaluation Settings
Basc-to-New. This benchmark includes five point cloud datasets as described above. Each dataset is equally split into base (known) and new (unseen) classes. We compute recognition accuracy (Acc.) on the two types of classes, respectively. Although the accuracy on the new classes reflects how well model can learn from known point cloud categories to generalize other unseen data, we also want decent accuracy on the base classes. Consequently, the harmonic mean (HM) between the accuracies of base and new classes is chosen to balance the
17

---

Table 7: Statistics of the Cross-Datased benchmark.
(a) OOD Gemeralization
Ite
Source StapeNeiCoree22
ModelNea44
SOBB ONLY
Target SOBB
S.PB TSO_RS
Omm33
#classes
55
40
15
15
15
216
#train
35,708
1,847
1,847
9.132
Aval
5.158
1,979
462
462
2.284
N/A
Hlest
10,261
2,468
581
581
2,882
5,910
(b) Data Corruption
Item
Add Global
Add Local
Drup Global
Drup Local
Rotate
Scale
jitter
#classes #test
40
40
40
40
40
40
40
2,468
2,468
2,468
2,468
2,468
2,468
2,468
(c) Sim-to-Real
Source ModelNet
Target
Ttem
S-OBILONLY
S-OBL BG
S-PB_ T50_KS
#classcs
11
11
11
Strain
4,844
N/A
N/A
9,132
#test
2,882
(d) PointDA
ModelNet
ShapeNet
ScanNet
#clusses
10
10
#train
4,183
17,378
6,110
Atest
856
2.492
1.769
Table 8: Statistics of the Few-shot benchmark.
Item
ModelNet40
S-OBJ ONLY
--BJJLONLY
S-OBJ BG
S-O
ShapeNetCore
#classes
40
15
15
15
55
#train
7,861
1,847
1,847
9,132
35,708
#val
1,979
462
462
2,284
5,158
fitsst
2,468
581
581
2,882
10,261
two factors. Similar evaluation settings are also adopted in vision-language community [85, 36, 7, 24, 25, 89, 35]. Note that the models trained fixed set of point cloud categories cannot he evaluated this benchmark since they not have the apee-vocahulle recognition ability, such as IDPT [74], Point-PEFT [59], DAPT [88].
Cross-Dataset. This benchmark has four types evaluation settings and we will focus on the two settings which are, OOD generulization and data corruption, newly introduced in 3DDG by us. The last two correspond to the default settings in PoinDA [51] and Sim-to-Red Real [20]. For OOD generalization, models are trained on the source domain and directly transferred to the target domains for evaluation. Recognition accuracy is major metric. We will average the accuracies across five target domains measure the final generalization ability. For data corruption, models are trained on clean ModelNet [2] and tested on various point cloud corruption scenarios in ModelNet-C[55 C[56], such as add global noises, losing local parts, geometry transformations. etc. The average accuracies on 7 types of atomic corruptions computed to measure the model robustness against point cloud data corruption.
Few-Shot. This setting inspects the model generalization in extremely low-data regime, where only few samples each class are offered to train model then is evaluated 8, and 16 shots for the model training and the recognition accuracy on the whole test set is compared.
whole test set. Here we take 1, 2,
We insert the proposed explicit constraints to large 3D models, then conduct lightweight prompt tuning on downstream 3D tasks, finally observe whether positive gains will appear in above 3DDG evaluation settings compared to the same prompt learning but without regulation framework.
18

---

A.2 Implementation Details
In gaussian model weighting, take μ 15 and 1. To avoid store separate copics of the model parameters, we implement Eq. by iteratively adding current and previous epoch of weighted model parameters. Note learnable parameters are randomly initialized before training. The design ensures our model absorbing prior knowledge and reduces disk consumption effectively. Both ULIP and ULIP-2 exploit the Point-BERT [73] the backhone.
We use two RTX 4090 GPUs to run the experiments, On the base-to-new benchmark, we conduct prompt learning for 20 epochs. On the cross-dataset and few-shot benchmark, models trained for 50 epochs, and the prompt depth and length are set 12
respectively. For evaluation ModelNet-C, we report the results when the corruption severity 2. The 64 manual templates and other object classes descriptions generated by LLMs are available at our provided codebase.
A.3
Additional Results and Analysis
A.3.1 Base-to-new Class Generalization
Table 9 enriches the base-to-new class generalization evaluation by reporting the mean and standard deviation of three runnings the base-to-new class benchmark. The standard deviation in the bracket follows the mean. Note that ULIP and ULIP-2 with our regulation framework have much smaller deviation on novel classes, e-g.. 2.38% vs 3.77%, suggesting better generalization robustness against variations between seen unseen point cloud data.
Table 9: Base-to-new class generalization comparison for representative large 3D models based on prompt learning. Each number here the mean of three runnings, Base: base class accuracy (in %, same below). New: new class accuracy. HM: harmonic mean of base and new class accuracy. +RC demonstrates the models with our regulation constraint framework.
(a) Average over 5 datasets
(b) ModelNe140
Method
Base
New
HM
Method
Base
New
HM
P.CLIP [76]
75.66 74.11
35.80
P-CLIP 176]
33.23
P-CLIP2 [90]
37.84
50.10
P-CLIP2 [90]
93.98
45,21
61.05
ULIP [71]
77.32(1.41)
49.01(3.77)
59.99
ULIP [71]
92.80()
50,07(3.52)
65.05
+RC(Ours)
82.19(1.25 (2)
61.93(2.38)
70.64
+RC(Ours)
95.03(0.52)
53.27(3.03)
69.89
ULIP-2 [72]
77.91(0.95)
67.91(3.75)
72.57
ULIP-2 [7]]
91.77(0.41)
56,47(2.78)
69.92
+RC(Ours)
83.18(0.62)
76.10(1.14)
79.48
+RC(Ours)
95.30(0.36)
64.83(0.26)
77.17
(c) S-PB T50
(d) S-OBJ_BB
Method
Base
New
HM
Method
Base
New
HM
P-CLIP [76]
61.25
19.87
30.01
P-CLIP 176]
72.82
23.00
34,96
P-CLIP2 [90]
39.20
P-CLIP2 [90]
70.07
46.75
ULIP [71]
56.73(0.84)
25.80(2.33)
35.47
ULIP
73.20/2.32)
47.17(1.76)
$7.37
+RC(Ours)
64.20(0.99)
49.17(2.55)
55.69
+RC(Ours)
79.47(1.92)
55.20K(.89)
65.15
ULIP-2 [72]
66.40
(6.477(2.40)
ULIP-2 [72]
77.00(1.04)
83.27(3.76)
80.01
+RC(Ours)
73.67(0.56)
74.27(1.27)
73.97
+RC(Ours)
80.10(0.99)
88.93(0.24)
84.28
(e) S-OBJ ONLY
(l) ShapeNetCore
Metbod
Base
New
HM
Method
Base
New
HM
P-CLIP [76]
76.23
20.23
31.97
P-CLIP
74.78
33.92
46.61
P-CLIP2 (90]
71.40
44.39
54.74
P-CLIP2 (90]
78.27
34.58
47.97
ULIP [7]]
74.13(0.60)
50.8077.11)
60.29
ULIP [71]
89.73(2.38)
71.20(3.51)
79.40
+RC(Ours)
79.23(1.44)
65.93(2.21)
71.97
+RC(Our))
93.03(1.23
84.10(1.24
88.34
ULIP-2 [72]
78. (8.60(0.37)
(2)
77.42
ULIP-2 [72]
75.80(1.56)
$7.07(5 09)
65.11
+RC(Ours)
83.6010.37)
81.10(2.69)
82.33
+RC(Ouis)
83.23/0.80)
71.37(1.23)
76.85
A.3.2
Cross-dataset Generalization
Sim-to-Real evaluation measures the 3D domain generalization from simulating data to real world. This evaluation first introduced by MetaSets [20] then followed by PDG [64]. In this setting. ModelNet [2]
19

---

and ShapeNet [6] are regarded as synthetic point cloud and ScanObjectNN is constructed based real-scan data. Here we implement the proposed regulation framework upon P-CLIP2 [90]. ULIP [71]. ULIP-2 [72] compare with prior state-of-the-art. Note MetaSets and PDG use the whole training set the source domain for supervised learning, whereas our methods only exploit 16-shot prompt tuning. As Tab. 10 shows, our framework brings consistent generalization improvement on different large 3D models, +0.52% for P-CLIP2, +5.88% for ULIP and +2.48% for ULIP-2 average on 6 datasets. The enhanced ULIP-2 by the devised framework also outreaches prior best-performing PDG-D by 5.82%, demonstrating better generalization real-world point cloud data.
Table 10: Comparison of cross dataset generalization on Sim-to-Real. There are two evaluation settings, MN + SONN 11, SN + SONN 9. The left side of - stands for simulating data and the right side indicates real-world data. 11 classes are shared between ModelNet and ScanObjectNN, classes are common in ShapeNet and ScanObjectNN, In the experiments, a point cloud contains 2048 points. -P: PointNet, -D: DGCNN.
SONN.
9 SONN
Method
OBJ
BG
PB TSO_RS
OBJ
OBT_B
PB TSO
Avg.
MetaSets-P [90]
60.3
52.4
47.4
44.3
45.6
MetaSets: D [90]
58.4
59.3
48.3
49.8
47.4
42.7
$1.0
PDG-P [64]
58.5
56.6
51.3
PDG-D [64]
65.3
65,4
55,2
59.1
59.3
51.0
59.2
P-CLIP2 [90]
18.67(1.68)
15.57(3.23)
15.63(1.63)
53.00(3.06)
47.83(1.84)
35.8 15.830.199
310991994)
+RC(Ouus)
19.23(4.00)
88)
14.37(2.57)
56.60(3.70)
47.83(3.80)
36.10(2.81)
31.61(3.46)
ULIP [71]
21.60(2.50)
18.03(2.03)
13.63(1.51)
54.83(1.66)
14.17(2.46)
40.87(1.27)
33.86(1.91)
+RC(Our))
29.90(1.15)
(2.0))
(8.57(2.52)
(3.13(0.74)
(8.87(0049)
43.60(1.51)
39.74(1.4)
ULIP-2 [72)
62.73(00.55)
68, (88.23(0.86)
$2.83(1.10
66, (6.99())77))
70.50(2.48)
54.03(2755)
62.54(1.82)
+RC(Ous)
68.43(1.07)
69.47(0.95)
55.30V2.00)
65.83(
72.53(0447)
58.57(1.17)
65.02(1.17)
PointDA is 3D domain adaptation benehmark introduced by PointDAN [51], which includes 6 evaluation scttings displayed in Tab. 11. Previous methods like MctaSets [20], PDG [64], I-OODG [81] exploit the full training data each setting while we adopt few-shot learning (16 shots). The results suggest the proposed framework contributes to the enhanced domain adaptation significantly, e.g., almost 10% absolute improvements for ULIP-2 that leads prior state-of-the-art I-OODG by 6.94%.
Table 1 11: Comparison of cross-dataset generalization on PointDA. M: ModelNet, S: ShapeNet, * ScanNet. The last column is the average over evaluation settings.
Method
M - S
M - S
M
S -+ S
S - M
S - S
Avg.
P-DAN [51]
64,2
33.0
47.6
33.9
49.1
64.1
48.7
MctaSets [20)
86.0
$2.3
67.3
42.1
69.8
69.5
64.5
PDG [64]
57.9
70.3
67.2
LOODGG [81]
83.7
56.4
71.7
57.6
69.5
73.5
67.8
P-CLIP2 [90]
40.53
26.40
31.33
35.57
16.30
24.97
29.18
+RC(Ours)
43.27
37,20
32.47
36,97
21.70
43,90
35,92
ULIP 1731
74.33(8.63)
38. 38.23/2.12)
35.17(3.99)
36. 17(5.77)
34.70(5.16)
60.67(4.72)
44. (8.88(550))
+RC(Ours)
78.80(1 (1.4)
41.63(0.87)
43.03(4.28)
41.60(4.02)
25.23(4.68)
63.60(9.01)
48, 48.98(4.06)
ULIP-2 1721 +RC(Ours)
84,80(2.99)
18.10X2.131
83. 13.20(4.17)
42.01X4,18 188)
60.43(4.83)
70.5006..2 22)
64.84(4.04)
89.00(1.18)
51.37(1.03)
(2.87)2.338
49.57(2.50)
(3.57(3.80)
83.07((.211
74.74(2.52)
A.3.3 The Role of MEC
Selecting the best checkpoint through a validation set is a common way. In theory, this greedy strategy favors highest performances on the downstream tasks, which means the small number of learnable prompis/parameters well adapted to these tasks. It is equivalent our framework without Model Ensemble Constraint (MEC).
However, purely optimizing the small number of learnable prompts toward target tasks will inevitably hinder the generalization ability of the large models, as we analyzed in the paper.
We also provide the ablation study to this problem, as results Tab. indicates, the method without MEC has slightly lower accuracy on new classes (75.59% vs 76. 10%% and harmonic mean. However, when removing the factors of MAC and TDC, the role of MEC becomes prominent. It raise the overall performance remarkably, especially for unseen new classes (5.28% absolute points).
20

---

Table 12: Ablation study for the framework without model ensembling constraint. The results are averaged on 5 datasets. MAC: mutual agreement constraint, TDC: text diversity constraint, MEC: model ensemble constraint. HM: harmonic mean of the Base and New class accuracies.
Variant
Performance
Model
MEC
77.91
67.91
72.51
x
/
82.42
73.19
77.53
ULIP-2
x
83.30
73.39
79.26
A.3.4 Running Time
We compare the training time between our method and the baselines. The results are shown in Tab. 13. The proposed method consumes similar amount of time per epoch compared the baseline, with slight increase the inclusion of our framework.
Table 13: Running time comparison of a strong baseline ULIP-2 and the proposed approach. We conduct prompt learning based ULIP-2 for 20 epochs on the base-to-new class benchmark, and the experiments are run three times with different seeds. The settings are consistent with those in the main paper. Time is counted in seconds for all 20 epochs using RTX 4090.
Dataset
Method
seed
RS
SJBBI
S..ORI ONLY
132
106
53
307
129.2
ULIP-2
106
53
305
128.8
133
108
48
51
305
129.6
159
112
60
344
147.0
RRC(Ours)
159
114
59
345
147.4
113
60
345
147.2
The number of Icarnable parameters of our framework is 16,896 while full fine-tuning ULIP-2 has 82.3M learnable parameters (only in text and 3D encoder). According to the reported details of ULIP-2, pre-training on Objaverse [11] utilizes 8 A100 GPUs and takes 1.5 days. So full fine-tuning ULIP-2 is also expensive.
A.3.5 Scalability
We further test framework larger datasct named Objaverse-LVV LVVIS and the results promising. This dataset is a subset the recently released Objaverse and only serves as set (target domain). Objaverse-LVIS contains 46,205 point clouds distributed 1,156 classes, and some classes only have single object, posing great challenges to existing point cloud recognition methods. In the experiments, we select representative ULIPP and UL.IP-2 as baselines and compare them with the models with our regulation framework. The results in the Tab. 14 verify the proposed approach also bring considerable gains (+3.27% absolute points for ULIP-2) on such a larger and challenging dataset.
Table 14: Analysis of scalability.
Source ShapeNeiV2
Target
Method
Objaverse
UTLIP
87.33 87.33(0.95)
0.83(0.05)
+RC(Ours)
90.43(0.86)
1.10(0.08)
76.70(1.37)
80(0.2)
+RC(Ours)
76.70(1.59)
18.07(0.49)
21

---

Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Refer to Abstract and Section 1.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Refer to Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" scction their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specifiennt specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not present a new theory or proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
22

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Refer the implement details and Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer: [Yes]
Justification:
The data and code are released at httpps//giitie http:://gghhub.o//..... http//auuuuuuoooittttt..
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines httpss://nips.coooe Seides/CodeSube ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :/ //nips....///////// cc/public/ Saaaddes/Coceeuummisssoooooo
for more details.
23

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Refer to the implementation details in Section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We run each experiment three times and report the mean and standard deviation, referring to the tables in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment. does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: We use two RTX 4090 GPUs to run the experiments, referring to Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
24

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, every respect, with NeurIPS Code of Ethics
Answer:
[Yes]
Justification:
Before submission, we read the NeurlPS Code of Ethics and obey them during this study,
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure to preserve anonymity (e-g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of this work in the last section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will he foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
To our knowledge, we do not perceive the potential risks of this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
25

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow the same licenses of the used datasets and cite the original paper, referring Section and Appendix.
Guidelines:
answer means paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. com/ dataseta has curated licenses for some datasets. Their licensing guide help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should he provided.
If this information not available online, the authors encouraged reach out the asset's creators.
13. New Assets
introduced in the paper well documented and is the documentation provided alongside assets?
Question:
Are
Answer: [Yes]
Justification: We release the new benchmarks and provided related README at https://github. com/ autaaauniquuuuuu Poooeeettuooi-PPGGG
Guidelines:
The answer means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does relate crowdsourcing and research with human subjects.
Guidelines:
The answer NA that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Fthics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
26

---

Answer: [NA]
Justification:
Our work does not relate to the human subjects.
Guidelines:
The answer NA means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
27


Reviews:
[{"review_id": "D8t9xZ1y4d", "reviewer": "Reviewer_kpdQ", "summary": "This paper investigates the 3D domain generalization (3DDG) capability of large 3D models based on prompt learning. The authors propose a comprehensive regulation framework that employs lightweight prompt learning to improve both task-specific performance and domain generalization ability. The framework consists of three main components: mutual agreement constraint, text diversity constraint, and model ensemble constraint. Additionally, the authors introduce three new 3DDG evaluation benchmarks: base-to-new, cross-dataset, and few-shot generalization benchmarks. Experimental results demonstrate that the proposed method significantly enhances model generalization while improving specific task performance.", "questions": "1 What specific advantages do the newly proposed benchmarks have compared to previous DG methods?\n\n2 How do you view the scalability of this method on larger datasets or more complex 3D tasks?", "limitations": "By a more detailed discussion, the authors could provide a more balanced view of their work, demonstrating scientific rigor and offering valuable insights for researchers looking to build upon or apply their method. This would significantly strengthen the paper and its contribution to the field.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The paper demonstrates significant originality by being the first to address the 3DDG problem for large multi-modal 3D models and proposing a novel regulation framework with innovative constraint mechanisms. \n\n2 The quality of the research is evident in its comprehensive experimental design and the significant improvements shown across multiple benchmarks and models. \n\n3 The work's significance lies in addressing the critical issue of domain generalization in 3D point cloud analysis, potentially impacting related fields broadly. Furthermore, the introduction of new benchmarks provides valuable tools for future research in 3DDG。", "weaknesses": "1 There's no detailed comparison of training time between the proposed method and baseline approaches or full fine-tuning of large 3D models.\n\n2 Limited validation across diverse point cloud tasks and real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. We address your concerns point by point. Feel free to ask follow-up questions if something remains unclear.\n\n**Q1: Training time comparison between baselines and our method.**\n\nAs requested, we have added a comparison of the training time between our method and the baselines. The results are shown in the following table. The proposed method consumes a similar amount of time per epoch compared to the baseline, with a slight increase due to the inclusion of our framework.\n\nTable r1-1. Running time comparison of a strong baseline ULIP-2 and the proposed approach. \n  We conduct prompt learning based on ULIP-2 for 20 epochs on the base-to-new class benchmark, \n  and the experiments are run three times with different seeds. The settings are consistent with those in the main paper. Time is counted in seconds for all 20 epochs using a RTX 4090.\n\n|  | seed | MN40 | S-PB_T50_RS | S-OBJ_BG | S-OBJ_ONLY | SNV2 | Avg. |\n|------|:------:|:------:|:-------------:|:----------:|:------------:|:------:|:------:|\n| | 1    | 132  | 106         | 48       | 53         | 307  | 129.2|\n| ULIP-2 | 2    | 132  | 106         | 48       | 53         | 305  | 128.8|\n| | 3    | 133  | 108         | 48       | 51         | 305  | 129.6|\n| |1    | 159  | 112         | 60       | 60         | 344  | 147.0|\n| +**RC**(Ours)|2    | 159  | 114         | 60       | 59         | 345  | 147.4|\n| |3    | 159  | 113         | 59       | 60         | 345  | 147.2|\n\nThe number of learnable parameters in our framework is 16,896 while full fine-tuning ULIP-2 has 82.3M learnable parameters (only in text and 3D encoder). According to the reported details of ULIP-2, pre-training on Objaverse [9] utilizes 8 A100 GPUs and takes 1.5 days. So full fine-tuning ULIP-2 is also expensive.\n\n[9] Deitke et al. Objaverse: A Universe of Annotated 3D Objects. CVPR 2023\n\n**Q2: Limited validation across diverse point cloud tasks and real-world scenarios.**\n\nWe would like to point out that we have already considered multiple real-world scenarios. In particular, there are four datasets in our new benchmarks collected from real-world scenarios, including the three variants of ScanObjectNN [55] and Omni3D [60]. ScanObjectNN is widely used in the community and Omni3D is a recently released dataset that contains a large vocabulary of 3D objects. Both of them\npose great challenges to existing point cloud recognition methods, according to the results in Table 1 and Table 2 in the main paper. \n\nOur work mainly focuses on the recognition task since the large 3D models (e.g., ULIP, ULIP-2) are pre-trained using the contrastive objective (similar to CLIP)  and they are good at the global alignment of 3D objects and their text descriptions with classe names. \n\nOther 3D tasks like object detection and segmentation are dealing with scene-level point cloud data. They follow different paradigms and methodologies compared to object-level recognition. We leave it to future works to explore the 3DDG ability of our framework on these tasks.\n\n[55] Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data. ICCV 2019\n\n[60] Wu et al. OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation. CVPR 2023\n\n**Q3: The advantages of the new benchmarks compared to existing ones.**\n\nFirst, our benchmarks provide new evaluation dimensions for the 3DDG methods. These new evaluation dimensions are key indicators to reflect the domain generalization ability but absent in existing benchmarks. Specifically,\n\n* the domain generalization evaluation in existing PointDA and Sim-to-Real focuses on the shared categories between source and target domain, without considering unseen new classes. We think it is a critical limitation, especially when evaluating the generalization ability of large 3D models that can conduct open-set recognition. In contrast, our created base-to-new and cross-dataset benchmarks provide the evaluations both on seen and unseen data.\n\n* previous benchmarks fail to evaluate the generalization ability when the target domain occurs data corruptions, which are common cases in 3D point cloud analysis. In contrast, our cross-dataset benchmark introduces this kind of evaluation to measure the model’s ability against common data corruptions.\n\n* the few-shot benchmark inspects the model generalization ability under extreme low-data regime (e.g., 1-shot learning). But this kind of evaluation has not been covered by previous benchmarks.\n\nSecond, our benchmarks are more diverse and challenging. There are only 10 classes in PointDA and 11 classes in Sim-to-Real. Our newly created benchmarks contain 7 different datasets and up to 216 point cloud categories, which will drive future research.\n\n**Q4: The scalability of this method on larger datasets**\n\nDuring rebuttal, we further test our framework on a larger dataset named Objaverse-Lvis and the results are promising. This dataset is a subset of the recently released Objaverse and only serves as a test set (target domain). Objaverse-Lvis contains 46,205 point clouds and 1,156 classes, and some classes only have a single object, posing great challenges to existing point cloud recognition methods. In the experiments, we select representative ULIP and ULIP-2 as baselines and compare them with the models with our regulation framework. \n\nThe results in the following table verify the proposed approach can also bring considerable gains (+3.27% absolute points for ULIP-2) on such a larger and challenging dataset.\n\n| Method      | Source (ShapeNetV2) | Target (Objaverse-Lvis) |\n|-------------|:----------------------:|:-------------------------:|\n| ULIP        | 87.33 (0.95)         | 0.83 (0.05)             |\n| +RC(Ours)   | **90.43** (0.86)     | **1.10** (0.08)         |\n| ULIP-2      | 76.70 (1.37)         | 14.80 (0.22)            |\n| +RC(Ours)   | **76.70** (1.59)         | **18.07** (0.49)        |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jsNGlQv9su", "reviewer": "Reviewer_UDVe", "summary": "This paper investigates the 3D domain generalization (3DDG) ability of large 3D models using prompt learning. They utilize parameter-efficient prompt tuning to boost the performance of 3D point cloud recognition models. The paper observes that while prompt tuning improves downstream tasks, it often reduces the generalization ability of the models. Thus, they introduce a comprehensive framework to maintain good generalization by allowing learnable prompts to interact actively with the pre-trained general knowledge in large 3D models. This framework imposes explicit three regulation constraints on the prompt learning trajectory, maximizing mutual agreement between task-specific predictions and task-agnostic knowledge. They also develop three new benchmarks to evaluate 3D domain generalization: base-to-new class generalization, cross-dataset generalization, and few-shot generalization.", "questions": "See the weaknesses section.", "limitations": "This paper lacks an analysis of its limitations. The effectiveness of the text diversity constraint relies on the quality and relevance of the text descriptions, which may vary depending on the source (LLMs or manual templates).", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The newly created benchmarks provide a more holistic evaluation of 3D domain generalization, addressing real-world challenges such as transferring to unseen classes and handling corrupted data.\n2. This paper achieves consistent improvements in generalization ability across various large 3D models and benchmarks, demonstrating its effectiveness.\n3. The use of lightweight prompt tuning makes the framework computationally efficient, reducing the need for extensive retraining of large models.", "weaknesses": "1. According to the paper's introduction, there are already substantial works on domain adaptation and domain generalization for 3D point clouds, including both object-level data and real scanned radar data. Many advanced methods also utilize beyond PointNet and ModelNet dataset. Consequently, the authors need to provide a more rigorous and detailed motivation for their study.\n2. The right part of eq.(1) needs a more detailed explanation to describe its components clearly.\n3. The method proposed in this paper seems overly simplistic and lacks novelty. Beyond the three general constraints mentioned, are there any specific designs for integrating LLMs with 3D point cloud multimodal learning?\n4. This paper does not introduce any additional designs for domain adaptation. Although it proposes a new benchmark, it essentially applies transfer learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Below we address your concerns one by one. Follow-up questions are welcome if something remains unclear.\n\n**Q1: More rigorous motivation of this study is needed** \n\nThanks for your comments. We need to clarify the following points.\n\nAs we stated in the introduction (Line 36), there are only a few of methods discussing domain adaptation (PointDAN [45]) and domain generalization (MetaSets [19], PDG [58]) for 3D point clouds, instead of substantial works.\n\nTo make it more clear, we added Figure 1 in the uploaded one-page PDF to explain our motivation. In summary, our motivation is to enhance the performances of large 3D models on downstream 3D recognition tasks while simultaneously maintaining good 3DDG ability. Previous works focus on downstream tasks but fail to consider the generalization on unseen data and lack relevant evaluations. Their limitations include\n\n* **Generalization among a fixed set of categories:** Our approach based on large 3D models can conduct open-set recognition and generalize to any unseen class.\n\n* **Limited Scale and Scope:** The large 3D models we investigate usually have hundreds of millions of parameters, e.g., ULIP-2 with 100M+ params, and the 3D encoders in previous works cannot match this scale.  Our method is evaluated on up to 215 classes while previous methods are only tested on up to 11 classes (Sim-to-Real).\n\n* **Compromised generalization ability:** Previous methods leveraging large 3D models focus on the downstream performances by lightweight adaptation (e.g., PPT [52], IDPT [68], DAPT [82], Point-PEFT [53]), while this strategy compromises generalization ability.\n\nIn short, our work offers a pioneering investigation of 3DDG ability of large 3D models and presents a simple yet effective solution. We believe that the new benchmarks will benefit researchers and drive future advancements in the field.\n\n**Q2: Further explanation of the right part of Eq. (1)**\n\nThe general idea of the right part of Eq. (1) is that we try to find optimal point and text prompt parameters $\\{E^{P*}, E^{T*}\\}$ that minimize the expected cross-entropy loss over the ground truth data distribution $\\mathcal{D}_{gt}$. Let's dive into the components of this equation.\n\n* $\\{\\textbf{\\textit{E}}^{P*},\\ \\textbf{\\textit{E}}^{T*}\\}$: These are the optimal parameters that we are trying to find.\n\n* $argmin_{\\{\\textbf{\\textit{E}}^P,\\ \\textbf{\\textit{E}}^T\\}}$: This notation indicates that we are looking for the arguments (in this case, the parameters $\\{\\textbf{\\textit{E}}^P,\\ \\textbf{\\textit{E}}^T\\}$) that that minimize the following expression.\n\n* $\\mathbb{E} _ { (P, y) \\sim \\mathcal{D} _ {gt}}$: This denotes the expected value over the ground truth data distribution $\\mathcal{D}_{gt}$, where $P$ represents the input point clouds and $y$ represents its real class labels.\n\n* $\\mathcal{L}_{CE}(\\tilde{\\mathcal{D}}, y)$: This is the cross-entropy loss function. $\\tilde{\\mathcal{D}}$ represents the predicted class distribution, and $y$ is the true class label.\n\n**Q3: The method seems simple and lacks novelty**\n\nWe answer this question in the global response section, kindly refer to that part. \n\nIn addition, when it comes to leveraging LLMs for prompt learning, we mainly regard them as powerful tools to produce diverse text descriptions to the point clouds. The specific designs are simple but effective. We customize three types of instructions to LLMs, and the details of the design are also visualized in Figure 3 of the uploaded one-page PDF. : \n* _Question Answering_\n* _Caption Generation_\n* _Making Sentences_\n\nAfter that, we encode these responses with the text encoder of the large 3D multi-modal models to obtain the representations of different 3D categories.\n\n**Q4: Lack of specific designs for domain adaptation** \n\nThanks for your comments. Our work mainly investigates the 3D domain generalization (3DDG) ability of large 3D models, instead of domain adaptation (Line 1 of the main paper). \n\nThere are two key differences between domain adaptation (DA) and domain generalization (DG). _First_, DA methods can access target domain data during training while DG methods cannot. _Second_, DA methods aim to minimize performance drop when transferring to a specific known target domain while DG methods try to ensure generalization and robustness to unseen domains.\n\nThe proposed regulation constraint framework is designed for boosting task-specific performances while maintaining the generalization ability simultaneously. On the base-to-new benchmark, the models conduct lightweight prompt learning while directly test on unseen new classes.  Similarly on the cross-dataset benchmark, the models learned from the source domain are directly tested on target domains.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bilvr7IMIk", "reviewer": "Reviewer_d557", "summary": "In this work, the authors propose a regularization method for the prompt learning of generalizable point cloud analysis, which can strengthen the performances of learned representations on downstream 3D task while keeping its generalizability. The regularization consists of three components: mutual agreement constraint, text diversity constraint, and model ensemble constraint, which is a plug-and-play method for existing 3D large multi-modal models. Moreover, this work also includes new benchmarks for the evaluation of 3D point cloud domain generalization. Results on the proposed benchmark confirm the effectiveness of the proposed regularization method.", "questions": "1. The relationships between the $h_T$ in Text Diversity Constraint is not so clear. Would it be used for the regularization in Eq.2? Or just the inference?\n\n2. Some details are not intuitively presented, such as basic framework of existing multi-modal. It would be better to add some diagrams to present the existing works and your improvements;\n\n3. How is the sensitivity of Model Ensemble Constrain to its hyperparameters? Besides, I am also not so sure about the necessity of such ensembling over all epochs. Why cannot we just select a best checkpoint through a validation set?\n\n4. What's the advantages of doing prompt learning for domain generalization over fine-tuning methods such as Low Rank Adaption?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The whole framework is simple but effective;\n\n2. The writting is good and easy to follow;\n\n3. The construction of new benchmarks may be beneficial to the community.", "weaknesses": "My major concern about this work is its novelty. As the prompt tuning method has been well studied in other areas, e.g., text-to-image generation. The proposed regularization constraint in Eq.2 is somewhat similar as the preservation loss term proposed in [1], while the other terms improve the robustness by straightforward average operation. I am not sure if the novelty is enough.\n\n[1] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. Below we address your concerns one by one. Further questions are welcome and we are happy to respond.\n\n**Q1: Major concern on novelty**\n\nWe understand your concern regarding the novelty. And we answer the question in the global response section, kindly referring to that part. We think the preservation loss in DreamBooth is not very relevant to the Eq. 2 in our work. And text-to-image generation is another different topic from the point cloud analysis. \n\n**Q2: The relationships between $h_{T}$ and the Text Diversity Constraint is not clear**\n\nYes, $h_{T}$ would be used in Eq. (2). Since LLMs will generate multiple text descriptions for each point cloud category, we integrate these text features into $h_{T}$.\n\n**Q3: The basic framework of multi-modal prompt learning and inclusion of new diagrams**\n\nWe have added the diagrams to explain the multi-modal framework and the proposed method in our work, referring to Figure 1 and 2 in the uploaded PDF file. \n\nFigure 1 highlights our research motivation, distinguishes our work with existing methods and demonstrates superior 3DDG ability on unseen new classes and better performances of base classes. \n\nFigure 2 illustrates the overall pipeline of the proposed approach and showcases how we incorporate the regulation constraints in the pipeline. \n\n**Q4: The Model Ensemble Constraint sensitivity to its hyperparameters**\n\nWe add the sensitivity analysis of the model ensemble constraint to the hyperparameters mean $\\mu$ and variance $\\sigma^2$ and report the results in the following sub-table (a) and (b). As observed, increasing $\\mu$ will give more weights to the models in later epochs and improve the base class accuracy while compromising the generalization on unseen new classes. In general, the changes are not very sharp, similarly for the variance $\\sigma^2$.\n\nTable r3-2. The sensitivity of Model Ensemble Constraint to the hyperparameters. Here mean $\\mu$ and variance $\\sigma^2$ are the hyperparameters of a gaussian distribution. ULIP-2 is deployed as the 3D foundation model and the experiments are conducted on the base-to-new benchmark.\n\nIn the sub-table (a), the variable is $\\mu$, σ² = 1               \n| Metric | 7    | 9    | 11   | 13   | 15   | \n|--------|------|------|------|------|------|\n| Base   | 72.71 | 72.84 | 73.17 | 73.25 | **73.67** |  \n| New    | **74.63** | 74.41 | 74.45 | 74.40 | 74.27 | \n| HM     | 73.66 | 73.62 | 73.80 | 73.82 | **73.97** | \n\nIn the sub-table (b), the variable is $\\sigma^2$, $\\mu$ = 15\n| Metric | 25    | 16    | 9     | 4     | 1     |\n|-------|-------|-------|-------|-------|-------|\n| Base   |72.51 | 72.77 | 72.83 | 73.29 | **73.67** |\n| New    | **74.95** | 74.89 | 74.66 | 74.41 | 74.27 |\n| HM     |73.71 | 73.81 | 73.73 | 73.85 | **73.97** |\n\nSelecting the best checkpoint through a validation set is a common way. In theory, this greedy strategy favors highest performances on the downstream tasks, which means the small number of learnable prompts/parameters are well adapted to these tasks. It is equivalent to our framework without Model Ensemble Constraint (MEC). \n\nHowever, purely optimizing the small number of learnable prompts toward target tasks will \ninevitably hinder the generalization ability of the large 3D models, as we analyzed in the paper. \n\nWe also provide the ablation study to this problem, as the following table indicates, the method without MEC has slightly lower accuracy on new classes (75.59% vs 76.10%) and harmonic mean. \nHowever, when removing the factors of MAC and TDC, the role of MEC becomes prominent. \nIt raise the overall performance remarkably, especially for unseen new classes (5.28% absolute points).  \n\nTable r3-2. Ablation study for the framework without model ensembling constraint. The results are\naveraged on 5 datasets. MAC: mutual agreement constraint, TDC: text diversity constraint, MEC:\nmodel ensemble constraint. HM: harmonic mean of the Base and New class accuracies.\n\n| MAC | TDC         | MEC  | Base    | New  | HM   |\n|--------|----------------|-------|--------|--------|--------|\n| x | x | x | 77.91 | 67.91 | 72.57 |\n| x | x | √ | **82.42** | **73.19** | **77.53** |\n| √ | √ | x | **83.30** | 75.59 | 79.26 |\n| √ | √ | √ | 83.18 | **76.10** | **79.48** |\n\n**Q5: The advantages of prompt learning over low-rank adaptation**\n\nPrompt tuning and low-rank adaptation (LoRA) are orthogonal techniques for parameter-efficient fine tuning. We prefer prompt tuning over LoRA since it does not change the architecture and parameters of the 3D foundation models. In contrast, LoRA needs to change the architecture of the foundation models by introducing low-rank matrices, which might not be desirable in practice since foundation models are usually more precious and hard to obtain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First of all, we sincerely thank all reviewers and ACs for reviewing our paper and providing valuable comments. There is no doubt that these suggestions and feedback are very valuable for refining the paper. We are encouraged by the appraising comments from the reviewers: ''significant originality by being the first ...'' (Reviewer kpdQ), ''The whole framework is simple but effective'' (Reviewer d557), ''consistent improvements in generalization ability across various large 3D models and benchmarks'' (Reviewer UDVe). Moreover, all reviewers recognize that our constructed benchmark for 3D domain generalization is valuable to the community. 

In the following, we will give a global response to the common concern on novelty of this work.

**Q1: The method proposed in this paper seems simple and lacks novelty.**

A1: We agree that our method is simple, but effective. The novelty of our work lies in that we are the first to investigate the 3D domain generalization capability of large 3D models, and present a simple yet effective regulation framework to address this critical issue. The originality is The originality is highly appraised Reviewer kpdQ.

Moreover, we construct three new benchmarks that provide new evaluation dimensions for 3D domain generalization (3DDG), including generalization on unseen new classes, corrupted data, and few-shot generalization. These are vital indicators to measure the 3DDG ability in real-world scenarios but ignored by previous works. The merits of our constructed benchmarks are recognized by all three reviewers: ''... new benchmarks provide valuable tools for future research in 3DDG'' (Reviewer kpdQ). ''... newly created benchmarks provide a more holistic evaluation of 3D domain generalization'' (Reviewer UDVe). ''... new benchmarks may be beneficial to the community'' (Reviewer d557)

Meta-review: The paper initially received mixed ratings. After the rebuttal and discussion phases, all reviewers leaned toward accepting the paper, with two ratings of Weak Accept and one of Borderline Accept. After thoroughly considering the paper, the reviews, and the subsequent discussions, AC recommends acceptance to this paper. 

The AC suggests that the authors should incorporate the reviewers' suggestions and the additional content from the rebuttal and discussion phases into the final version of the paper. This includes related works on 3D domain adaptation/generalization for LiDAR data in autonomous driving, a comparison of training times, scalability considerations, sensitivity analysis, and the inclusion of figures presented in the rebuttal PDF.
Decision: Accept (poster)