Generated and Pseudo Content guided Prototype Refinement for Few-shot Point Cloud Segmentation
Authors: Lili Wei
Congyan Lang
Ziyi Chen
Tao Wang
Yidong Li
Jun Liu
Venue: NeurIPS 2024 spotlight (2024)
Keywords: 3D point cloud segmentation; few shot learning; Large Language Model
Primary area: machine_vision

Abstract: Few-shot 3D point cloud semantic segmentation aims to segment query point clouds with only a few annotated support point clouds. Existing prototype-based methods learn prototypes from the 3D support set to guide the segmentation of query point clouds. However, they encounter the challenge of low prototype quality due to constrained semantic information in the 3D support set and class information bias between support and query sets. To address these issues, in this paper, we propose a novel framework called Generated and Pseudo Content guided Prototype Refinement (GPCPR), which explicitly leverages LLM-generated content and reliable query context to enhance prototype quality. GPCPR achieves prototype refinement through two core components: LLM-driven Generated Content-guided Prototype Refinement (GCPR) and Pseudo Query Context-guided Prototype Refinement (PCPR). Specifically, GCPR integrates diverse and differentiated class descriptions generated by large language models to enrich prototypes with comprehensive semantic knowledge. PCPR further aggregates reliable class-specific pseudo-query context to mitigate class information bias and generate more suitable query-specific prototypes. Furthermore, we introduce a dual-distillation regularization term, enabling knowledge transfer between early-stage entities (prototypes or pseudo predictions) and their deeper counterparts to enhance refinement. Extensive experiments demonstrate the superiority of our method, surpassing the state-of-the-art methods by up to 12.10% and 13.75% mIoU on S3DIS and ScanNet, respectively.

Generated and Pseudo Content guided Prototype Refinement for Few-shot Point Cloud Segmentation
Lili Wei
Congyan Lang 1.2 Ziyi Chen Tao Wangh. Yidong Li Jun Liu
School of Computer Science & Technology, Beijing Jiaotong University, China 2 Key Laboratory of Big Data & Artificial Intelligence in Transportation, Ministry of Education, China School of Computing and Communications, Lancaster University, UK {20112014, cylang, 20112021, twang, ydli)abjju.. edu. cn j j.1iu810lancaster uc
Abstract
Few-shot 3D point cloud semantic segmentation aims to segment query point clouds with only few annotated support point clouds. Existing prototype-basee methods learn prototypes from the 3D support set to guide the segmentation query point clouds. However, they encounter the challenge of low prototype quality due to constrained semantic information in the 3D support set and class information bias between support and query sets. To address these issues, in this paper, we propose novel framework called Generated and Pseudo Content guided Prototype Refinement (GPCPR), which explicitly leverages LLMM-generated content and liable query context to enhance prototype quality. GPCPR achieves prototype refinement through two core components: LLM-driven Generated Content-guided Prototype Refinement (GCPR) and Pseudo Query Context-guided Prototype Re- finement (PCPR). Specifically, GCPR integrates diverse and differentiated class descriptions generated by large language models to enrich prototypes with com- prehensive semantic knowledge. PCPR further aggregates reliable class-specific pseudo-query context to mitigate class information bias and generate more suitable query-specific prototypes. Furthermore, we introduce a dual-distillation regulariza- tion term, enabling knowledge transfer between early-stage entities (prototypes pseudo predictions) and their deeper counterparts to enhance refinement. Extensive experiments demonstrate the superiority of our method, surpassing the state-of- the-art methods by up to 12.10% and 13.75% mloU on S3DIS and ScanNet, respectively.
Introduction
Point cloud semantic segmentation, aiming assign semantic label each point in the 3D point clouds, benefits various applications like autonomous driving, robotics, and augmented/ving virtuul reality.
Despite the remarkable performance of supervised methods in this field [33 34 22 24 12 37), they rely on labor-intensive labeled data and struggle to distinguish novel classes not present in the training data. To address these issues, few-shot 3D point cloud semantic segmentation (FS- (FS-3DSeg), which segments novel classes with only few labeled data, has gained widespread attention.
FS-3DSeg aims to learn model on base classes and generalize it to novel classes with only a few annotated point clouds. Existing FS-3DSeg methods [39 17 36 20 10 [6] typically adopt
prototype-based paradigms [28 32), where prototypes extracted from the support point clouds
treated as class descriptors to guide the segmentation unlabeled query point clouds by matching
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

query point features with prototypes. Due to their representation and generalization capabilities, prototype- methods have made great progress and garnered increasing attention. However, their performance remains unsatisfactory due to encountering the challenge of low prototype quality, attributed to two reasons: 1) Semantic information constraints: Limited 3D support point clouds contain only partial and incomplete object information, lacking both intra-class diversity and inter- class discriminative information. Consequently, vanilla prototypes generated from the 3D support set are insufficient to be comprehensive class descriptors. 2) Class information bias: Query and support point clouds exhibit intra-class object variations 16 20 and feature distribution gaps [9 [0]
making vanilla prototypes may be ill-suited for segmenting query samples as the distances between vanilla prototypes and query features will be far away.
To address these issues, we propose to compensate for the lack of semantics in 3D support set to generate comprehensive and reliable query-specific prototypes for accurately segmenting the query point cloud. This can be achieved in two steps. Firstly, to alleviate class information constraints, motivated by the fact that texts are more controllable and accessible [55, we aim to integrate general semantic knowledge from texts to refine prototypes. Fortunately, Large Language Models (LLM) (e.g. ChatGPT) [3 35 29] offer promising opportunities to help achieve this goal due to their wealth of implicit knowledge and have been leveraged for variety of downstream tasks, such as image segmentation [42] and point cloud classification [43]. Inspired by this, we leverage the capabilities of LLM to explicitly generate diverse and differentiated contents, combined with novel designs to obtain refined prototypes with richer semantics. Secondly, to reduce class information bias, previous 16 TO have endeavored to map prototypes to query feature space through prototype- query interaction using cross-attention 31. However, noisy interactions between prototypes of one class and query features of other classes may introduce class-irrelevant cues, thereby undermining the reliability of prototypes. We observed that pseudo masks for query point clouds can assist in extracting class-specific pseudo-query context and filtering out class-irrelevant noisy features.
works (9 20
Ultimately, this paper, we propose a novel FS-3DSeg framework via Generated and Pseudo Content guided Prototype Refinement (GPCPR), which progressively refines prototypes by two newly proposed components: LLM-driven Generated Content-guien guiddd Prototype Refinement (GCPR) and Pseudo Query Context-guided Prototype Refinement (PCPR). Initially, vanilla prototypes generated from 3D support features. Firstly, our GCPR prompts the LLM to generate diverse descriptions of single classes and differentiated descriptions of class pairs. Then we design text adapters and two into vanilla prototypes, yielding refined prototype covering with richer and discriminative semantic knowledge. Secondly, leveraging the refined prototypes, PCPR generates pseudo masks for query point clouds derive the class-specific pseudo-query context, which can be aggregated into prototypes by a newly designed query-to-prototype compressor, yielding more reliable prototypes that can better associate with query points. Beyond that, we also introduce a dual-distillation regularization term, which enables early-stage entities (prototypes or pseudo predictions) to gain insight from their deeper optimal counterparts. This facilitates bidirectional information exchange in both forward and backward processes during training, thus obtaining better prototypes and predictions.
compressors for mapping and incorporating text knowledge
We summarise contributions follows: 1) We propose GPCPR, novel end-to-end FS-3DSeg framework that enhances prototype quality by simultaneously integrating LLM generated content and reliable query context to generate query-specific prototypes. To the best of our knowledge, this is the first time leveraging LLM's s capabilities to segment novel classes in FS-3DSeg. 2) We design series of novel modules, including the Generated Content-guided Prototype Refinement (GCPR) module and the Pseudo Query Context-guided Prototype Refinement (PCPR) module, to facilitate the prototype refinement process. Additionally, we design dual-distillation regularization term to further mutually enhance the refinement. 3) Extensive experiments demonstrate the superiority of our method, notably exceeding state-of-the-arr methods by up to 10% and 13.75% on S3DIS and ScanNet datasets, respectively.
2 Related Work
2.1 3D Point Cloud Semantic Segmentation
3D point cloud semantic segmentation (3DSeg) aims to assign semantic labels to each point within point clouds. Supervised 3DSeg studies involve voxel-based methods [7 I8] and point-based methods

---

12 37,, with the latter receiving increasing attention due to the simplicity, flexibility, and efficiency. For example, DGCNN [3] proposes the EdgeConv module to capture local structures. design self-attention-like networks to model long-range contexts from distant neighbors. Despite their effectiveness, these methods often rely on extensive annotations and cannot segment novel classes. In this paper, following mainstream FS-3DSeg methods (39] 9 [6], we utilize DGCNN as the point encoder and extend its capability to segment novel classes.
23 23 24
Recently. [12 37
2.2 Few-shot 3D Point Cloud Segmentation
Most few-shot 3D point cloud semantic segmentation (FS-3DSeg) methods
39 17
20
10
41
16 I follow the prototype-based paradigms [32 28 11
learning prototypes from the support set to segment query point clouds. Specifically, AttMPTI 39] the pioneering FS-3DSeg method, proposes an attention-aware multi-prototype transductive inference framework based on label propagation. BFG [7 mutually embeds global perception into local features and prototypes. To reduce contextual gaps between support prototypes and query features, [20] TO 9 [6] adapt prototypes to query feature space by pootoypp-qqeery feature interaction via cross attention 31]. COSeg [[] introduces correlation optimization by refining the multi-prototypice. support-query correlations. In addition to prototype learning. SCAT (3) explores class-specifs relations between query and support features using transformer blocks 31 instead of pooling operations. However, these methods often yield low-quality prototypes since the 3D support set only contains constrained semantic information.
Besides, their prototype-quer, interaction 20 10 16 may introduce noise. In contrast, our approach refines prototypes by integrating LLM -eMeeeeed content and class-specific pseudo-query context, resulting in well-suited query-specific prototypes for better segmentation.
2.3 Large Models
Recently, several large models have emerged, including large language models (e.g.. GPT-3 3
ChatGPT B5 and BERT (5)) and large multi-modal models (e.g., CLIP (5). BLIP B3. LLaVA
[1]. GPT-4 Vision (35) and MiniGPT [40]). These extensive models are repositories of extensive
knowledge and have achieved widespread success in diverse downstream tasks, such as robot task plans 127. image classification (26] and image segmentation 42] In the field of 3D point cloud understanding, several large models have also emerged that associate point clouds, texts and other modalities (e.g.. Point-LLM (8). MiniGPT-3D [29], Uni3D-LLM [14] and PointCLIP V2 43) However, none of them are designed for segmenting novel classes in FS-3DSeg. In this work, we leverage the rich generated content of LLM to assist in addressing the FS-3DSeg problem.
3 Method
3.1 Problem Formulation and Overview
Problem Formulation. Following previous FS-3DSeg methods 33 91
we adopt meta-learning based paradigm. Specifically, all semantic classes are divided into base class set Chase and novel class set Cnover for training and testing respectively, where Chase n Crovel = Each few-shot task (i.e., episode) instantiates N-way K-shot segmentation task. The input data in each episode
) J and query set Q = (TT: M) where N, K and T' denote the number of classes, the number of support point clouds for each class, and the number of query point clouds. I and I denote support and query point clouds, each contains M points. M E 13 denotes binary support GT mask for each of the N unique classes. M {O..... NJ M denotes the query GT mask, which only available during training. The class name set contains one background name and N foreground class names, denoted as C = {c,, b The goal of FS-3DSeg is to learn a model to predict segmentation mask M, for I, based on S.
contains a support set S {(T.A M
K
Overview. Figure I Iillustrates the architecture of our proposed method. Given query set and support set Q and S, following common practice [39] 9] we utilize a shared point encoder to extract per point query and support features, represented as F,, R Midd and F, E Nx Mxd where d denotes feature dimension. Then we adopt masked average pooling (MAP) 9 to generate vanilla class-wise 3D support prototypes [ [RN+1]xd from F., including a background prototype and N foreground prototypes. To enhance the quality of IP, we propose two modules: LLM-
3

---

Prototype Generation
(b) Generated Content guided Prototype Refinement
(6) Pseudo Query Context guided
Prototype Refinement
Quey GT
Probeyyee Distilation
Protuippe Distilation
Query Comproos
Acr
4
OGPA
Support
Esoodin
PP
Prediction
Pssudo
Tathdaper
Aest
Prodhction
TatEacdaa
Uneriipaan
Differationed Unerriphken
Query
LIX
Panel suu Crnendden
Froem
Polet Fostens
Logat Distilatoon
Tiandbe
Padabpee
Coumoo Stmianty
Figure 1: Overview of our method. The support flow is responsible for Prototype Generation, LLM- driven Generated Content-guide Prototype Refinement (GCPR), and Pseudo Query Context-guided Prototype Refinement (PCPR). Dual-distillatinn i.e., prototype distillation and logit distillation, further enhance the refinement process. The query flow measures distances between query point features and refined prototypes to predict segmentation results.
driven Generated Content-guidd guided Prototype Refinement (GCPR) and Pseudo Query Context-guided Prototype Refinement (PCPR). Firstly, GCPR prompts LLM to generate diverse class descriptions D and differentiated class descriptions D' followed by a frozen text encoder and trainable text adapter to extract text features E and E'. To effectively incorporate text knowledge into P. GCPR sequentially utilizes two text-to-protoyy compressors to generate refined prototypes T with rich semantics. After that, PCPR generates pseudo masks M, extract class-specific pseudo query contexts F, for Q, followed by a svery-tt-prototyye compressor generate well-suited query-specific prototypes P. Finally, cosine similarities between F, and each point cloud is assigned the label of the most similar prototype. To supervise the model training, we adopt standard cross-entropy loss and design dual-distillation regularization term (DD loss) to further enhance the refinement process. Subsequently, we provide detailed descriptions of GCPR, PCPR, and DD loss as below.
calculated segment query point clouds, where
3.2 LLM-driven Generated Content-guided Prototype Refinement
Due to support point clouds only containing constrained semantic information, the vanilla prototypes P lack class diversity and cannot be better associated with Q. To address this, inspired by the common- sense knowledge in L.MM and the fact that textual descriptions are more controllable [35], we aim to leverage L.MM to generate detailed class content and propose two text-to- to-pooootppee compressors to embed these texts into prototypes IP, thereby obtaining refined prototypes with richer semanties.
LLM-driven Content Generation. By leveraging the extended world knowledge of LLM (e.g., GPT-3 3) generate two types of content at scale, i.e., diverse class descriptions and differentiated class descriptions, respectively for enhancing the comprehensiveness and discriminability of the prototypes P..
1) Diverse Class Descriptions. To generate diverse descriptions, effective prompt templates for LLMs essential. Inspired by the heuristic 31i-orietted commands for depth maps in 43 we designed the following 3D-specific heuristic prompt templates to ensure that the texts encompass sufficient point cloud properties:
Caption Generation:
"Describe a point cloud of [CLASS] in one sentence." Question Answering: "What is a [CLASS] point cloud like?"
Paraphrase Generation: "Generate synonym: A point cloud of [CLASS] " Make Sentence: "Make a sentence with words: point cloud, [CLASS], obscure."

---

For each command, we place "[CLASS]" with class name and feed it into GPT-3 to generate Ndie descriptions. The generated description set can be represented as: contains 4 x Naio descriptions.
1N {Da. In= where Den.
2) Disferentiated Class Descriptions. To enhance the model's ability distinguish between multiple classes, inspired by [6] which generate differentiated attribute descriptions between ambiguous classes for image classification, we intend generate 3D-specific differentiated descriptions for any pair of classes, explaining their visual and geometric differences. To this end, we designed the following template to prompt LLM to generate differentiated class descriptions:
"Describe the distinguishing visual or geometry differences between the point clouds of [CLASS 1] and [CLASS 2] in pairs of sentences. Generate as many captions as you can."
We place " "[CLASS 1]" and "[CLASS 2]" with pair of class names and trigger GPT-3 to produce several sentences, each of which is split into two parts at the conjunctions, with the first (or second) part assigned differentiated description for class a1 (or 02), denoted Do= (or D ). For each class, we combine all its differentiated descriptions with other classes to contain more discriminative information, forming differential description set D' [D' J R== where D' = { J..E)(C))) includes 5 descriptions.
Detailed explanations and examples for diverse/diifeereniaate descriptions are given in Appendix
Text-To- To-Prototype Compressors. Given LLM- -.M--genertedd content D and D', we employ a frozen CLIP textual encoder (25 and a shared text adapter to extract text features E E (I
and E' E [NN+1)xNay xd To incorporate text knowledge into vanilla prototypes P, we propose two 31 text-to- pertto--rotttppe compressors, i.e., the class torotto-r compressor and the differentiated text-to- to-troooottppe compressor, which measure the relations between IP and texts aggregate semantic information.
pooss-attention-baee based
1) Class text-to-pre pex-tto-prototype compressor integrates features for diverse descriptions into vanilla prototypes to obtain refined prototypes T = (T") I with richer semantics, each formulated as:
1" pm softmar [Q,Kı V1, E {0, ..,,.,,,, N},
where Q1 pn E Rixd' K₁ E" E R(4xNaw)xxd and V1 E" W.. [R(4 W11, WA1 E Raxd' and W.1 E gdxd are learnable parameters for fully connected layers. d' denotes projected feature dimension.
2) Differentiated text-to topotoo-ro compressor integrates features E' for differentiated descriptions into previous prototypes T to obtain refined prototypes [T")N | with discriminative information, each formulated as:
4 = t + softmar Q2K2 T) V2, n {0, N}.
(2)
R Rixdd K2 E" Wx E Xa and V2 E" W.a Nasy Wa2, Wx2 aaxd and Woz gddd are learnable parameters for fully connected layers.
where Q2 = W e
3.3 Pseudo Query Context-guided Prototype Refinement
Query and support point clouds often exhibit large object variations and class information bias, making vanilla support prototypes P unsuitable for segmenting query point clouds. Although our GCPR compensates for the lack of semantics they still lack query-specific associations. Previous approaches have endeavored to customize query-specific prototypes through prototype-quer feature interaction using cross-attention (3I). Specifically, QGPNet [10] and QGPA [9] align support prototypes with query features to reduce the channel- feature distribution gap but lack query-specific context. QGE 20 and DPA [16 aggregate query context into support prototypes. but inevitably introduce noisy interactions between prototypes of one class and query features other classes. This could inadvertently incorporate class-irrelevant query cues into prototypes, thereby undermining their reliability and exacerbating the challenge of distinguishing confusing classes. To address this issue, aim to generate more reliable class-specific pscudo query context from query features and propose transformer-base query-to-prototype compressor customize well-suited query-specific prototypes.
Pseudo-Query Context Generation. To extract class-specific context. need to predict pseudo masks M for each query point cloud I and extract class- class-specific context F', accordingly, effectively filtering out interfering
5

---

cues from other classes. However. the channel distribution gap between prototypes T and query features F;, may lead to inaccurate pseudo masks, Thus we follow QGPA 9 to rectify current prototypes I'' to query feature channel distribution through channel-wise cross-attention
31 formulated as:
(3) where Q3 F Wa E Iddx' K3 = F". Wis E Id and V3 = 7 T E Bixd F² E Mx denotes averaged support features, calculated by F = 1/(NK) ΣN,, F if n = O, otherwise F"
pian = t + softmar (QaKa Va, n (O,... N,, (I,..,T)
1/K- Σ F.. W4s, WaS E RMxar and Was E g are the learnable parameters for FC. Then query- specific prototypes P- = (P''TE E BTx(N+1)xd are obtained, where IP denote query-specific prototypes for each query point cloud r..
Then, we calculate cosine similarity hetween F; and prototypes PP' to predict pseudo logits L R MMx(N+1) and pseudo one hot masks M; E (111) By masking query features with pseudo masks, we obtain class-specific pseudo query context, represented as: F, {Fa o==o, F" E (f)x
Query-To-Prototts Seery-To-Preee To-Prototype Compressor. The query-to-proooee prootooyye compressor aggregates pscudo query context features into the protolypes by cross attention, represented as:
p pi: + ssoftmaa QaKa Va.
n (J).. i [I.
(4)
where Q+ = pii Wap E 1xd K4 = F Wia E g(TxM)xx V4 = F; E (R(xxx)xx Was. W44 E
Raxd' and Wi. E Raxd are learnable parameters for fully connected layers. After that, we obtain the final query- specific prototypes well-suited to segment query point clouds, represented as iP [ Txxx++))xd
Finally, we calculate cosine similarity scores between F; and 1i to obtain predicted logits L, E (M(++)1)) Each point is then assigned the label of the most similar prototype to generate the final predicted mask M,.
3.4 Dual-Distillation Regularization
To further enhance prototype refinement, we introduce dual-distillation regularization term that allows early- stage entities gain insights from their deeper counterparts, including two types of self-distillatin: i.e. prototype distillation and pseudo prediction distillation.
Prototype Distillation. the forward process, GCPR and PCPR integrate useful semantics into early-stage prototypes to generate refined prototypes. To achieve mutually beneficial and bi-directional optimization of multi-stage prototypes, we treat early-stage prototypes (e.g.. T or IP) student prototypes and deep-stage prototypes (e.g.. fft or P) as teacher prototypes, employing prototype distillation in GCPR and PCPR, defined as:
Lrp = (LTT)). LQp= mp=KKi (L()))))
(5)
where KI. refers to the Kullback-Leibler divergence. The self-distillation process facilitates effective knowledge transfer from teacher prototypes student prototypes, enhancing the refinement process.
Pseudo Prediction Distillation. PCPR, pseudo masks play a core role as they determine the reliability of the generated class-specific pseudo query context. further improve the accuracy and quality pseudo masks, we treat pseudo logits L, as students and final predicted logits Ly teachers, and employ logit distillation in PCPR, formulated as:
Lax KLLLL,IL,
(6)
3.5 Objective.
During training, the proposed model is supervised by two loss functions, i.e., standard cross-entropy loss ESEG and the proposed dual-distillation loss (DD loss). The cross-entropy loss serves the main optimization target, aimed at learning comprehensive and reliable query-specific prototypes. The dual-distillation loss acts as a regularization term to promote knowledge transfer between carly-stage entities (prototypes or pecudo-reeicctonn. and their deeper counterparts. Formally, the cross entropy loss is defined as:
LSEG = LOR LM M.).
The overall loss is a weighted combination of LSEG and DD loss with balancing weight X, represented as:
Liotal LSEG + (LTP + Cor + LQM).
(8)

---

Table Performance on S3DIS dataset using mean-IoU metric (%). S represents the split is used for testing. The best results are masked in bold. Our method consistently far exceeds SOTA.
2-way
3-way
Method
T-shot
5-shot S'
I-shot S*
5-shot
S*
S
S
mean
g
S*
mean
ProtoNet
39
48.39
49.98
49.19
57.34
63.22
60.28
40.81
45.07
42.94
49.05
53.42
51.24
AmMPTI
39
53.77
55.94
54.86
61.67
67.02
64.35
45.18
49.27
47.23
54.92
56.79
55.86
BFG 17
55.60
55,98
55.79
63.71
66.62
65.17
46.18
48.36
47.27
55.05
57.80
56.43
SCAT 36
54.92
56.74
55.83
64.24
69.03
66.63
QGPNet 10]
56,30
57.62
56.96
65.34
69.01
67.17
47.00
50.12
48.56
55.80
58.54
57.17
2CBR [1
55.89
61.99
58.94
63.55
67.51
65.53
46.51
53.91
50.21
55.51
58.07
56.79
QGE 20
58.85
60.29
59.57
66.56
79.46
73.01
QGPA 9
59.45
66.08
62.76
65,40
70.30
67.85
48.99
56.57
52.78
61.27
60.81
61.04
DPA [16
66.08
74.30
70.19
71.10
77.03
74.07
50.67
59.53
55.10
64.52
63.34
63.93
Ours
74.04
77.44
75.74
76.65
78.22
77.44
62.77
70.57
66.67
67.49
74.68
71.09
Table Performance on ScanNet dataset using mean-loU metric (%). St represents the split v used for testing. The best results are masked in bold. Our method consistently far exceeds SOTA.
2-way
3-way
Method
I-shol
5-shol S
I-shol
5-shot
8
S
5
5
S*
8
S*
ProtoNet 39
33.92
30.95
32.44
45.34
42.01
43.68
28.47
26.13
27.30
37.36
34,98
36.17
АtMРTI
39
42.55
40.83
41.69
54.00
50.32
52.16
35.23
30.72
32.98
46.74
40.80
43.77
BFG 17
42.15
40.52
41.34
51.23
49.39
50.31
34.12
31.98
33.05
46.25
41.38
43.82
SCAT 36
45.24
45.90
45.57
55.38
57.11
56.24
QGPNet 10
44.63
42.18
43.40
54.75
51.81
53.28
37.86
34.50
36.18
47.45
42.74
45.09
2CBR [4]
50.73
47.66
49.20
52.35
47.14
49.75
47.00
46.36
46.68
45.06
39.47
42.27
QGE 20
43.10
46.79
44.95
51.91
57.21
54.56
QGPA
57.08
55.94
56.51
64.55
59.64
62.10
55.27
55.60
55.44
59.02
13.16
56.09
DPA []6
62.75
63.04
62,90
67.19
64.62
65,91
61.97
61.72
61.85
66.
64.67
65.40
Ours
75.94
71.92
73.93
78.42
78.37
78,40
70,00
66.61
68.31
76.73
68.63
72.68
4 Experiments
4.1 Experiment Settings
Datasets. We verify our method on S3DIS 12] and ScanNet [ S3DIS contains 3D-RGB point clouds collected from 272 rooms across six indoor scenes. Its label set includes 13 classes (12 semantic classes and background clutter). ScanNet collects 1,513 point clouds from 707 indoor scenes. Its label set includes 21 classes (unamnotated background and semantic classes). Due to the inconvenience of processing the original
room which contains an excessive number of points, we follow the pre-processing strategy in [39 [6] to divide each room into 1yn x 1m blocks and randomly sample M 2048 points from each block, yielding 7.547 blocks for S3DIS and 36,350 blocks for ScanNet, respectively. Each point contains 9D vector, including XYZ, RGB, and normalized coordinates. Following (39), semantic classes evenly split into subsets, denoted as 5 and S², For meta-learning, we train our model on one fold (e.g., 5*)) and test on another fold (e.g., S.). Vice versa for cross-vaue validation.
non-overlapping
Evaluation Metrics. Following conventions in the 3DSeg community, we average the mean Intersection-over- Union (mloU) across all test classes and report the final mloU metric under the few-shot setting.
Implementation details. In Appendix B we provide framework details and training details.
4.2 Comparison With State-of-the--At Methods
Results on S3DIS. Table T shows the experimental results of our method compared with state-of-the-ee methods on the S3IDS dataset. Our method demonstrates substantial improvement over the baseline QGPA ( with a large margin of 6.22%*~14. 14.59% across various splitting settings, highlighting significant improvement for FS-3DSeg. Additionally, our method consistently exceeds all of the previous methods, particularly surpassing ofte-of-the-ae art methods DPA [I6] by a margin of 1.19%>~12,
across various splitting settings, showcasing our effectiveness and superiority. The success of our approach is attributed to the integration of comprehensive 1LL--genraaee content and reliable pseudo-query context by GCPR and PCPR, solving the problem of semantic information constraints and class information bias, well as enhancing refinement by dual-distillation loss.
7

---

cattrr
cillee
CAIIEA
Query Point
Cloud
Mask
QGPA
Ours
S3DIS
SeanNet
Figure 2:
Qualitative results of our method on S3DIS and ScanNet in 2-way 1-shot S segmentation task in comparison to GT mask and QGPA [9]
Table 3: Offline time cost S3DIS and ScanNet under 2-way 1-shot setting.
Phase
S3DIS
ScanNet
Description Generation: g-t-3.--turbo
30.23 min
67.15
Text Feature Extraction: CLIP m50
10.95
17.79
Total
30.41 min
67.45 min
Table 4: Analysis of online computational cost and experimental results under 2-way 1-shot setting.
Methods апtМРТ
#Params
FLOPs (G)
FPS
Inference Time (ms)
S3DIS
ScanNet
357.82K
152.65
678.67
54.86
41.69
QGPA
2.79M
16.30
38.68
25.85
62.76
56.51
DPA
4.85M
15.49
32.35
30.91
70.19
62,90
Ours
4.22M
18.96
20.57
48.61
75.74
73.93
Results on ScanNet. Table 2 presents the segmentation performance on ScanNet. Our method significantly improves the baseline by larger margin of 11.01 method DPA [16] by large margins of 1066%~....755 across various settings. Our model exhibits even greater improvement ScanNet compared to S3DIS due to containing more classes and complex scenarios ScanNet, which greatly challenges prototype quality. Our approach effectively integrates more comprehensive and reliable content to cope with the lack of semantic information, resulting in superior performance.
Notably, we significantly outperform SOTA
Computational Complexity, In our approach, generating descriptions with L.MM and extracting text features through CLIIP for all classes in the dataset performed offline hefore the training and testing phases. Once text features stored through offline operations, by LLM online, avoiding bringing redundant computing costs during training and testing. (1) Offline cost: Table shshww the time cost of extracting text features for all classes in the entire dataset. We analyze that the total offline time is primarily determined by the description generation process. with feature extraction time being negligible. The time for generating descriptions from LLM varies different datasets, depending the number of classes. (2) Online cost: Table shows that method achieves better balance between computational cost and experiment performance, oftering superior segmentation results with reasonable computational efficiency.
directly load the stored text features without regenerating
Qualitative results. In Figure we visualize the segmentation results of the 2-way 1-shot segmentation task on S3DIS and ScanNet. We compare our predictions with the GT mask and QGPA. As shown, QGPA often produces inaccurate segmentation, particularly in distinguishing background from foreground classes and among different foreground classes. In contrast, method achieves accurate segmentation for areas of the target scene. instance, in second column of S3DIS examples, QGPA mistakenly predicts parts the beam and bookcase, whereas our method effectively distinguishes between these classes. Similar results are observed in the last column of ScanNet examples. The superiority of our method is because we incorporate LLM M-generated content and qeerry-query context obtain query query-specifts specific prototypes, resulting in better segmentation results.

---

100-1
72.14
300=97.7B
(a) QGPA
Ours
(c) QGPA
(d) Ours
Visualization of feature distribution and prototype distribution on S3DIS under 2-way 1-shot setting. "acc' denotes the segmentation accuracy. Red/green dotted circles mark query features that are far/close to the refined prototypes, respectively. Best viewed in color.
Figure 3:
4.3 Ablation Study
We perform ablation study the S3DIS dataset under the 2-way 1-shot setting validate the effectiveness our proposed GPCPR. We use QGPA as the baseline to analyze effects proposed components and hyper-parameters.
Effeets Different Components. conduct experiments to investigate the cf- feet of the three core components, Le.,
Table 5: Ablation study of key components on S3DIS dataset under 2-way 1-shot setting using mean-loU metric (%).
GCPR, PCPR and DD loss.
Effeett of PCPR: Without introducing any text knowledge, PCPR improves baseline by 8.68%, showing that PCPR ex- tracts reliable query context, avoids noise interference, and effectively reduces class information bias.
GCPR D
DD loss
2-way 1-shot
PCPR
D
LIP
Lar
CoM
Sa
S²
58.96
63.08
61.022
65.01
74.39
69.70
66.06
69.03
67.55
66.71
70.18
68.45
68.57
74.71
71.64
Effects of GCPR: Only using diverse de- scriptions or differentiated descriptions
75.73
72.21
69.36
75.85
72.61
71.09
76.09
73.59
D' we exceed baseline by 6.53% or 7.43%, with the latter playing greater role because they directly improve the inter-class discrimination of the proto- types. After jointly using D and D the improvement the baseline reaches 10.62%. showing that GCPR effectively makes up for the lack of seman- ties the prototype. Comparing PCPR and GCPR, the latter can achieve greater improvement because LLM provides sufficient semantic details. Finally, combining PCPR and QCPR, our method improves the baseline by 11.19%.
/
71.07
76.47
73.77
74.04
77.44
75.74
3) Effects of DD loss: Compared to only using PCPR and QCPR, after further applying Crr. Lor and LQss respectively, the model performance further improved 0.40%, 1.38% and 1 1.56%. with the last one being more effective due directly optimizing pseudo masks obtain By jointly using distillation losses, we achieve the hest performance of 75.74%.
reliable class-specific pseudo query context.
Visualization features and prototypes. shown in Figure we use the t-SNE visualization tool 30 to compare the feature distribution and the class prototype distribution to analyze the effect of the model. Here, prediction accuracy is also provided for clearer comparison. As marked by the red dotted circles. for QGPA, the distance between prototypes and most query features far. This indicates there exists bias between prototypes and query features due the lack query context the prototypes. In contrast, our method significantly improves accuracy and pushes the prototype closer to query features, as marked by the green dotted circles. This improvement is because we effectively inject rich text knowledge and reliable class- baas--ppcciiic query context into the refined prototypes, enhancing their representation capabilities and mitigating object variations.
Effects of different adaptors. As shown in Figure [a) without introducing any additional text information by removing the GCPR module, evaluate the effectiveness of prototype refinement and compare our method with the previous prototype adaptors, including QGPA [ DPA [6). method significantly outperforms QGPA and DPA. This is because although QGPA alleviates the thannel-wise feature distribution gap between the prototype and query features. the prototype lacks query context information and is therefore less effective. Compared with QGPA, DPA injects query context information into the prototypes. However, query context from

---

03
75
73
0
16
magg50
-3
OCCS
205
Our
01.0
0005
mode
50
17800
10
20
0.5
3.0
1.5
20
(a) Different adaptors (b) Different types of texts
(c) Text number Nase
(d) Coefficient À
S3DIS dataset under 2-way 1-shot setting. (a) Effects different adaptors. (b) Effects of different types of descriptions. (c) Effects of
Figure 4: Ablation study of modules and pyyer--arameee
the number of LLM--generated descriptions per command. (d) Effects of coefficient À for DD loss.
other classes acts as noise interference, reducing the reliability of the prototype and limiting performance. Our method introduces sategory-specitn pseudo-context context information into the prototype, effectively reducing noise interference while bridging object gaps and thus achieving the highest performance.
Effects of different types of descriptions. As shown in Figure 4(b) also studied the impact of different text descriptions, replacing the proposed I.I. --..geerttteddd descriptions with word2vee (W2V) [19] CLIP shown, word2vee exhibits limited improvement due to only providing static restricted semantic information. CL.PP texts slightly better improve the baseline, benefiting from the cross-modal ali alignment capability and rich semantic information learned from large-scale datasets. Our LLM-generated descriptions achieve the best performance because LLM's powerful detailed description ability inject sufficient diverse and differentiated semantic information into the prototypes.
texts 25 38
Effects of number of diverse descriptions. Figure [[c illustrates the impact the number Naw diverse descriptions. Compared with not using any diverse descriptions, the model performance gradually improves as Naw increases, because general knowledge is gradually injected into the prototypes to make up for the lack of semantic information, and reaches promising results when Nat 10. However, further increasing of Navo does yield higher performance the LLM-generated descriptions begin repeat, providing redundant semantic knowledge. In summary, we select Nave = 10 to ensure both the effectiveness and efficiency of the model.
Effeets of coefficient for DD Loss. Figure 4 illustrates the impact the weight of the DD loss in Eq. 8 Compared to À = 0 (where the DD loss is deprecated), as À increases, the prototypes and predictions in the final stage begin to gradually optimize the corresponding items in the early stages, allowing the model to generate high-quality prototypes. resulting in significant performance improvements, and reaching the best performance when X However. a larger X (e.g.. 2) does not yield higher performance and may even harm model performance. In summary, an appropriate value = yields the best results.
Effects different LLMss As shown Table using both "gpt-3.5-turbo 3 and "gnt-4o-mini" 21 can achieve supe- rior results compared with SOTA, with 75. 74% mloU and 73.88% mloU S3DIS, respectively. Specifically, Ours (gpt-3.5-turbo) performs best. Ours (gpt-4o-mini) performs slightly lower than Ours -.-ub.,, because "gpt-4o- mini" tends to generate longer paragraphs, which potentially contain content that is not direetly related to the class seman- ties, thus introducing noise the class prototypes. Thus this paper, we choose to utilize "gpt-3.5-turbo".
Table 6: Effects of different LLMs on S3DIS
dataset under 2-way --shot setting.
Methods
S
S*
mean
MPt
53.77
55.94
54.86
QGPA
59.45
66.08
62.76
DPA
66.08
74.30
70.19
Ours (gpt-4o-mini)
71.64
76.11
73.88
(gpt-3.5-turbo)
74.04
77.44
75.74
5 Conclusion
this paper, solve the problem constrained semantic information and class information bias in FS-3DSeg. we propose GPCPR that LLM generattd content and reliable query context to enhance prototype quality. GPCPR consists of two components: LLM-driven Generated Content-guided Prototype Refinement (GCPR), which enriches prototypes with diverse and differentiated text knowledge from LLMs, and Pseudo Query Context- guided Prototype Refinement (PCPR), which aggregates reliable class-specific pseudo-query context mitigate class information bias. Additionally, dual- diddiilltlation regularization term proposed to facilitate knowledge transfer between early-stage and deeper entities (prototypes or predictions). Experiments show that GPCPR outperforms state-or of the-art art methods by up to 12. 10% and 13.75% mloU on S3DIS and ScanNet, respectively. In Appendix C discuss the limitations of our work and future work.
10

---

6 Acknowledgement
This work was supported by the National Natural Science Foundation of China under Grant 62072027 and 62376020.
References
[1] Z. An, Sun, Liu, F. Liu, Z. Wu, D. Wang, Van Gool, and S. Belongie. Rethinking few-shot 3d point cloud semantic segmentation. In CVPR, pages 9966-4066, 2024.
[2] Armeni, O. Sener, A. R. Zamir, H. Jiang, Brilakis, M. Fischer, and S. Savarese, 3d semantic parsing of large-scale indoor spaces. In CVPR, pages 1534-1543, 2016.
[3] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, Gray, Chess, J. Clark, C. Berner, S. McCandlish, Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners, In NewrIPS, 2020.
[4] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. NieBner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, pages 5828 -5399, 2017.
[5] J. Devlin, M.-W Chang, K. Lee. and K. Toutanova. Bert: Pre-traine Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1880004005 2018.
[6] R. Esfandiarpoor and S. H. Bach. Follow-up differential dexcriptions: dextffrretttaalddeeeppttttno Language models resolve ambiguities for image classification. arXiv preprint arX1v:2311.07598, 2023.
B. Graham, M. Engelcke, and L. Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In CVPR, pages 9224-9232, 2018.
[8] Z. Guo, R. Zhang, X. Zhu, Y. Tang, Ma. Han, K. Chen, Gao, X. Li, Li, et al. Point-bind point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309,000 00.65,,,006,,,.,, 2023.
[9] S. He, X. Jiang, W. Jiang, and H. Ding. Prototype adaption and projection for few-and zero-shot 3d point cloud semantic segmentation. IEFE TIP, 32:3199-322 2023.
[10] D. Hu, S. Chen, H. Yang, and G. Wang. Query-guided support prototypes for few-shot 3d indoor segmentation. IEEE TCSVT, 34(6):2202 4213, 2023.
[11] X. Huang and H. Choi. Sapenet: self-attention based prototype enhancement network for few-shot learning. Pattern Recognition, 135(C):109170, 2023.
[12] X. Lai, J. Liu, Jiang, Wang. Zhao, Liu, Qi, and J. Jia. Stratified transformer for 3d point cloud segmentation. In CVPR, pages 8500-8509, 2022.
[13] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language- image pre- me-training for unitied vision- language understanding and generation. In ICML, pages 12888 12888-12900, 2022.
[14] D. Liu, X. Huang, Y. Hou, Z. Wang, Z. Yin, Y. Gong, P. Gao, and W. Ouyang. Uni3d-llm: Unifying point cloud perception, generation and editing with large language models, arXiv preprint arX1v:2402.03327, 2024.
[15] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In NeurlPS, 2023.
[16] Liu, W. Yin, H. Wang, Y. CHen, J.J.. Sonke, and E. Gavves. Dynamic prototype adaptation with distillation for few-shot point cloud segmentation. In 3DV, pages 810-819, 2024.
[17] Y. Mao, Guo, L. Xiaonan, Z. Yuan, and H. Guo. Bidirectional feature glohalization for few-shot semantic segmentation of 3d point cloud scenes. In 3DV, pages 505-514, 2022.
[18] H.-Y. Meng, L. Gao, Y.K. Lai, and D. Manocha. Vy-net: Voxel vae net with group convolutions for point cloud segmentation. In ICCV, pages 8500-8500 8508, 2019.
[19] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv: 3711.7711 2013.

---

[20] Z. Ning. Z. Tian, G. Lu, and W. Pei. Boosting few-shot 3d point cloud segmentation via query query-guided enhancement. In ACM MM, pages 1895-1904, 2023.
[21] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[22] C. R. Qi, H. Su, K. Mo, and J. Guihas. Pointett: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 652-660, 2017.
[23] C. Qi. L. Yi, H. Su, metric space. In NIPS, page
Guibas. Pointnet++: Deep hierarchical feature learning on point sets in
[24] G. Qian. Y. Li, H. Peng. J. Mai, H. Hammoud, M. Elhoseiny, and B. Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In NewrIPS, volume 35, pages 23192-2212 23204, 2022.
[25] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748 8763, 2021.
[26] Z. Ren, Y. Su, and X. Liu. Chawgnt-powered hierarchical comparisons for image classification. NeurlPS, volume 36, pages 69706 697188, 2023.
[27] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In ICRA, pages 11523- 11530, 2023.
[28] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In NIPS, volume 30, page 4080-4090, 2017.
[29] Y. Tang, X. Han, X. Li, Q. Yu, Y. Hao, L. Hu, and M. Chen. Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors. arXiv preprint arXiv:2405.01413 2024.
[30] L. Van der Maaten and G. Hinton. Visualizing data using t-snc. Journal of machine learning research, 9(11), 2008.
[31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E... Kaiser, and I. Polosukhin. Attention all you need, In NIPS, volume 30, page 6000-6010, 2017.
[32] O. Vinyals. C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. In NIPS, volume 29, page 3637-3645, 2016.
[33] Y. Wang. Y. Sun, Liu. S. E. Sarma, M. Bronstein, and M. Solomon. Dynamic graph enn for learning on point clouds. ACM TOG, 38,5):1-22, 2019.
[34] M. Xu, R. Ding, H. Zhao, and X. Qi. Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds. In CVPR, pages 3173-3182, 2021.
[35] Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L.. Wang- The dawn Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. 91);1 2023.
[36] C. Zhang, Z. Wu, X. Wu, Z. Zhao, and S. Wang. Few-shot 3d point cloud semantic segmentation via stratified class-specific attention based transformer network. In AAAI, pages 3410-3417, 2023.
[37] N. Zhang, Z. Pan, T. H. Li, W. Gao, and G. Li. Improving graph representation for point cloud segmentation via attentive filtering. In CVPR, pages 1244 1254,, 2023.
[38] R. Zhang, Z. Guo, W. Zhang. K. Li, X. Miao, B. Cui, Y. Qiao, Gao, and H. Li. Pointelip: Point cloud understanding by clip. In CVPR, pages 8552 8562, 2022.
[39] N. Zhao, T.S. Chua, and G. H. Lee. Few-shot 3d point cloud semantic segmentation. In CVPR, pages 8873-8882, 2021.
[40] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint urXiv:2304..0551 2023.
[41] G. Zhu, Y. Zhou, R. Yao, and H. Zhu. Cross-class bias rectification for point cloud few-shot segmentation. IEEE TMM, 25:9175-9188, 2023.
[42] I.. Zhu, T. Chen, D. Ji, Ye, and J. Liu. Llafs: When large-language models meet few-shot segmentation. Xiviv preprint arXiv:2311 166926, 2023.
[43] X. Zhu, R. Zhang, B. He, Z. Guo, Z. Zeng, Z, Qin. S. Zhang, and P. Gao. Pointelip v2: Prompting clip and gpt for powerful 3d open-world learning. In ICCV, pages 1639-2650, 2023.
12

---

A Examples of LLM-generated Content
LLM- -e-egeatetedd descriptions and text features for all classes are generated and stored offline. During training and lesting, we directly load the stored text features, avoiding online regenerating by LLM. To more intuitively understand the diverse and differentiated descriptions generated by the LLM,, we provide examples of the generated descriptions for the N-way K-shot segmentation task in Figure Sggure5aand and Figurel0 respectively.
For a single class, the descriptions generated by LLM capture rich information such as shape and geometric structure, location, detailed features, usage scenarios, and placement styles. These descriptions provide a comprehensive understanding of the class, enhancing the protolypes with detailed and varied semantic knowledge. For each class, we prompt the LLM generate Nair descriptions for each command. For each N-way K-shot segmentation task, we obtain (N + 1) x (4 X Nase.) descriptions, where the number Nat will affect the model performance, as too few or too many descriptions will lead to insufficient or redundant semantic information. Therefore, we analyze the effect of Dac in Figure [c) determining that the optimal value is 10.
For class pairs, differential descriptions generated by the LLM effectively distinguish class pairs by highlighting structural differences, density differences, surface differences, and edge differences, heneficial for distinguishing similar classes. In this paper, using our proposed prompting template, the I.I.M tends generate varying numbers of sentences for each class pair based on their distinguishabillty. We then split each sentence by conjunctions and assign each part to the relevant class. Consequently, after combining all differential sentences for each class in relation to all other classes, the number of differential descriptions Nass varies for each class. For each N -V-way K-shot segmentation task. load descriptions based on the minimum Nasys among the (N + 1) classes. For classes with more descriptions, we randomly load Nair deseriptions, ensuring consistent input dimensions. Thus we obtain (N + 1) x Na.; deseriptions per task.
Intuitively, differential descriptions are more conducive to improving the discrimination of prototypes, as verified by ablation experiments in Table 5 Besides, the joint usage of both kinds of descriptions leads to the best
performance in FS-3DSeg.
B Implementation Details
Framework details. We use QGPA (9] as the baseline, optimized by segmentation loss, self-reconstrucctin loss, and alignment loss. DGCNN 3 (with SAN) and pre-trained CLIP (m550) 25 are used as point encoder and encoder, respectively. The text adaptors adopt Limear+LeakyRe..U. LuaaykkeeU--DDrrput---innnee our GCPR module, the number No of descriptions per command is set to 10.
architecture. In We use "gpt-3.5-turbo" GPT-3.5 engine [3] and set the temperature constant to 0.99. For diverse descriptions, the largest length of a 3D-specific diverse description is set to 40, and Naiv is set to 10. For differential descriptions. LLM generates between to 20 sentences for different class pairs, causing Nars vary across classes, typically ranging from 14 to 34. For all compressors based on multi-head attention, the number of heads is set to 4. In Eq. 8 the coefficient for distillation loss in Eq. lis set to 1.
Training details. Our model is implemented by PyTorch and trained on single NVIDIA RTX A4000 GPU. Before meta-learning following (39), we pre-train the point encoder on the hase classes for 100 epochs. Then, during meta-training, we train our model on 40,000 episodes randomly sampled from base classes, using an Adam optimizer, with learning rate of 0.001, and a decay step of 5000. During meta-testing, we perform model evaluation on 100 randomly sampled episodes for each feasible class combination from unseen classes. For the query set, T' is set to for each class.
C Discussion
Limitations. Although our GPCPR achieves remarkable results, GPCPR has following limitations. (I) Integrating LLM and CLIP into the FS-3DSeg framework will increase resource consumption and computing costs, depending the parameter amount and execution speed of these large models. In order to alleviate this limitation, separated some operations in GCPR into offline operations, such prompting LLM to generate description content and using CLIP to extract text features. (2) The complexity for LLM to generate differentiated descriptions is O which may face challenges when extending to other domain datasets with massive semantic classes due to the need to exhaustively cnumcrate all possible class pairs.
Future work. The performance of the proposed method is affected by the quality of deseriptions generated by LLM and the accuracy of pseudo masks. Biased class deseriptions and inaccurate pscudo masks may introduce noise, negatively impacting model performance. Future work will mainly focus on improving the quality and reliability of --geerrrete content and pseudo masks. Furthermore, in this paper, we mainly address segmenting unseen categories in the same scene, but its generalization to different domains with unseen classes remains to be tested. Future research may involve extensive validation across various datasets and real-life scenarios to ensure robustness and adaptability.
13

---

Describe point cloud of (CLASS) in one sentence:
The point cloud of background clutter consists of seattered collection 3D points that represent various objects or debris present in the background environment.
(a) Caption Generation
point cloud of a bed is 3d represenlation consisting numerous puints, each representing specific coordinate space, forming the shape and structure of a bod.
S
A point cloud of a desk is a three-dimensional representation of scattered data points that visually captures the shape, position, and spatial details of the desk surface.
What is a [CLASS] point cloud like?
A background clutter poinl cloud is representation of the surrounding environment that includes irrelevant undesired objects structures.
(b) Question Answering
bed point cloud is 3D representation of bed's surface, consisting of vast collection of points with cach point capturing the X, Y, and Z coordinates of the bed.
point is a digital representation desk's surface objects, consisting of three-dimensional points with associated coordinates and attributes.
S
Generate synonym: A point cloud of (CLASSJ.
dense cluster of backdrop mess.
(c) Paraphrase Generation
A bed 's three-dimensional map.
desk's volumetric data.
Make a sentence with words: point cloud, [CLASS]. obscure.
The point cloud analysis bocame challenging due background clutter, which obscured relevant data.
(d) Make
The point cloud data helped to obscure the exact shape of the bed in the virtual reality simulation.
Sentence
The point cloud data helped to accurately recreate the shape and dimensions of the desk, even in the presence of obscure details.
Figure 5: Example of the diverse class descriptions generated by LLM. We input four different commands to L.MSs to generate diverse descriptions.
14

---

Describe the distinguishing visual or goometry differences between the point clouds of bed and desk in pairs of sentences. Generate as many captions as you can.
1. The point cloud of bed typically has larger surface area with more irregular shapes and angles, while the point cloud of desk tends to have more defined and straight edges with smaller footprint. 'n
2. Beds typically have point cloud that is lower to the ground and spread out horizontally, while desks have point cloud that is taller and more vertical in nature. 'n
3. (More captions omitted here)
Describe the distinguishing visual or geometry differences between the point clouds of bed and background clutter in pairs of sentenees. Generate many captions as you can.
1. The bed's point cloud exhibits more organized and structured arrangement of points, while the background clutter appears more haphazard and disorganized.
2. (More captions omitted here)
Describe the distinguishing visual or goometry differences between the point clouds of desk and background clutter in pairs of sentences. Generate as many captions as you can.
1. The geometry of the desk point cloud is characterized by its straight lines and right angles, while the background clutter point cloud is characterized by its irregular and unpredictable shapes. kn
2. The desk point cloud is composed of precise measurements and uniform spacing between points, whereas the background clutter appears fragmented and disorganized, 'n
3. (More captions omilted here)
Figure 6: Example of the differentiated class descriptions generated by LLMs.. For all classes, combine any two of these classes as input to LLM to generate several differentiated sentences. Then we split each output into several sentences, which are further splited by conjunctions (e.g., "while". "whereas", "compared to", "unlike", "in contrast to",, "contrast", "however". and so on) and assign each part to the corresponding class.
15

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction in Section Celirrly reflect the contributions and scope.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Appendix C we discuss the limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specifiennt specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
16

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We propose specific model for PS-3DSeg. In Section 3 we give detailed description of the overview architecture and all components. In Appendix B we provide settings of framework details and training details. We helieve this model is fully replicable. Our code will be made publicly available once it fully organized.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should deseribe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. large language model), then there should either be a way to access this model for reproducing the results way to reproduce the model (c.g.. with an open-source dataset or instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to deseribe the particular way they provide for reprodueibility. In the case of closed-sone sourcee models, it may he that access to the model is limited some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [No]
Justification:
Our code will be made publicly available once it is fully organized.
Guidelincs:
The answer means that paper does not include experiments requiring code.
Please see NeurIPS code and data submission guidelines hattps:/nnips.cc/puie
guides/Codee https//oooooouuuiiiiioooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
17

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines httpp////nia.ccppuaaaaeeee for more details.
Suades//oodeSubmisssooooo
The authors should provide instructions data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting Deatins
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters,
how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Settings of data splits are given in Section 4.1 More details of implementation details
(including framework details and training details) are given Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: Following common practice among mainstream methods in the FS-3DSeg field, we do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanicd by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful symmetric error bars would yield results that are out of range (e.g. negative error rates). error bars reported in tables plots, The authors should explain in the text how they calculated and reference the corresponding figures or tables the text.
show in tables figures
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide information computer resources our model in Appendix Besides, we also analyze the memory and time cost in Table Band Table
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full rescarch project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question:
Does the research conducted the paper conform. every respect, with NeurIPS Code Ethics
Answer:
[Yes]
Justification:
Our work conform with the NeurlPS Code of Ethics.
Guidelincs:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (c.g., if there is special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss hoth potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification:
It's socictal impact of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e-g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it not needed point out that a generic algorithm for optimizing neural networks could cnable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datascts)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have done as described in the question.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include a URL.
name of license (e.g.. CC-BY 4.0) should be included each asset. scraped data from particular source (c.g., website), the copyright and terms of service of that source should provided.
assets are released, license, copyright information, and terms of use the package should be provided. For popular datasets, pappreaii thwddethode.... www.cco//caa/.c com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper docs not release new datasct, our code will be made publicly available once fully organized.
Guidelines:
The answer means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should he included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
20

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
21


Reviews:
[{"review_id": "ZIphNNGTG3", "reviewer": "Reviewer_ABMp", "summary": "This paper targets few-shot 3D point cloud semantic segmentation and proposes GPCPR to explicitly leverage the LLM-generated content and query context to enhance the prototype quality. The component GCPR integrates diverse and differentiated class descriptions generated by LLMs to enrich prototypes. The component PCPR further aggregates reliable class-specific pseudo-query context to mitigate class information bias and generate more suitable query-specific prototypes. Furthermore, a dual-distillation regularization term is also introduced to enable knowledge transfer between early-stage entities (prototypes or pseudo predictions) and their deeper counterparts to enhance refinement. Experiments are conducted on the S3DIS and ScanNet datasets.", "questions": "What is the performance of the proposed method under the setting corrected by [1]?", "limitations": "The experiments are carried out in the old, ill-posed setting.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is easy to read. This paper targets an interesting problem, few-shot 3D point cloud semantic segmentation. The proposed method is also interesting.", "weaknesses": "Figure 1 should explain what the symbols mean, such as P, T, S, and dot.\n\nThe CVPR 2024 paper [1] found that there are some issues in the experimental setting of few-shot 3D point cloud segmentation and corrected the setting. However, this paper is still evaluated under the old setting and does not compare with [1], which makes the results less convincing. Thus, I believe that the experiments must be carried out under the new setting in the final version.\n\nIn Table 3, the PCPR module seems to only yield a marginal improvement. Please provide some explanations.\n\nConsidering that this paper uses LLMs, which classes does the model perform best and which classes does the model perform worst? Is there any connection between this finding and LLMs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ABMp,\n\nWe would like to express our gratitude for taking the time to review our manuscript and providing detailed and constructive feedback. We appreciate your positive reception of our work and carefully considered each of your points and would like to address your comments as follows:\n\n> **1. Explain symbols in Figure 1.**\n\nThank you for your valuable suggestion. We will add legends to Figure 1 to clearly explain the symbols. Specifically, $\\mathbb{P}$ denotes the original 3D prototype set. $\\dot{\\mathbb{T}}$, $\\ddot{\\mathbb{T}}$, $\\dot{\\mathbb{P}}$ and $\\ddot{\\mathbb{P}}$ represent prototype sets at different refinement stages. Among them, $\\dot{\\mathbb{T}}$ and $\\ddot{\\mathbb{T}}$ represent prototype sets refined by diverse and differentiated descriptions, $\\dot{\\mathbb{P}}$ and $\\ddot{\\mathbb{P}}$ indicate prototype sets refined by QGPA module and Query-to-Prototype Compressor, respectively. $S$ denotes cosine similarity, and $\\odot$ represents matrix dot product. \n\n> **2. Experiments under the new setting of [1].**\n\nThank you for your valuable comments. Based on the new experimental settings corrected by CVPR 2024 paper [1], we conduct more experiments under the 2-way 1-shot setting on S3DIS dataset. The results are presented in the table below. Our method achieves slightly higher performance than COSeg but far exceeds attMPTI and QGPA, demonstrating the proposed GCPR module, PCPR module and DD-loss can collaborate to improve the prototype quality by integrating comprehensive text descriptions and reliable pseudo query context. We will include these experimental results and analysis in the final version of our paper to provide a more convincing comparison with SOTA under the new setting.\n\nMethods | $S^0$ |$S^1$ | mean\n:--------| :---------:|--------:|--------:\nattMPTI | 31.09 | 29.62 | 30.36|\nQGPA | 25.52 | 26.26 | 25.89 \nCOSeg | 37.44 | 36.45 | 36.95 \nOurs | 37.96 | 37.38 | 37.67\n\n> **3. In Table 3, the PCPR module seems to only yield a marginal improvement. Please provide some explanations.**\n\nThe impact of the PCPR module depends on the scale of the dataset, the number of classes, and the complexity of scenes. \n\n(1) In Table 3, on the **S3DIS** dataset, without GCPR (Row 2 vs. Row 1), PCPR significantly improves the baseline by 8.68% mIoU due to its effective handling of class information bias between support and query sets. With GCPR (Row 6 vs. Row 5), PCPR only further yields a 0.64% improvement because GCPR already addresses the semantic information constraints of support set in S3DIS dataset with less classes and simpler scenes.\n\n(2) On the **ScanNet** dataset, removing PCPR causes a 5.33% drop in mIoU under 2-way 1-shot setting (68.60% without PCPR vs. 73.93% with PCPR), highlighting the importance of PCPR in reducing class information bias in complex scenes with more classes. \n\nIn summary, PCPR is more impactful on large-scale dataset with greater complexity and more classes, such as ScanNet. We will explain this more clearly in the paper.\n\n> **4. Which classes perform best/worst? Is there any connection between this finding and LLMs?**\n\nThank you for your insightful question. \n\n(1) The **class performance** of our model is shown as below:\n\n* S3DIS dataset: Best on *\"chair\"* (or *\"sofa\"*) and worst on *\"ceiling\"* (or *\"wall\"*) under $S^0$ (or $S^1$) setting. \n\n* ScanNet dataset: Best on *\" curtain \"* (or *\"sofa\"*) and worst on *\"floor\"* (or *\"picture\"*) under $S^0$ (or $S^1$) setting. \n\nFor example, under the 2-way 1-shot setting, the class performance are as follows:\n\n| | S3DIS  |  | ScanNet|  |\n|:-:|:-:|:-:|:-:|:-:|\n| | S0 | S1 | S0 | S1 |\n| Best IoU (%) | chair: 87.18 | sofa: 89.40 | curtain: 86.11 | sofa: 82.09 |\n| Worst IoU (%) | ceiling: 61.08| wall: 67.40| floor: 64.08 | picture: 51.89 |\n\nWe observe that our model excels in classes with complex geometric structures, unique shapes, and rich colors, but performs not as well in classes with flat geometric structures. Notably, compared to the baseline—which performs best on *\"chair\"* (or *\"sofa\"*) and worst on *\"beam\"* (or *\"table\"*) on S3DIS dataset under the $S^0$ (or $S^1$) setting—our model consistently shows improvements in both per-class IoU and mIoU. This is particularly evident in classes with complex structures, such as *\"beam\"* and *\"table\"*, where performance increases from 32.06% to 63.50% and from 53.22% to 72.39%, respectively.\n\n(2) **Connections with LLM**: The superior performance of our model on complex structures can be attributed to the introduction of LLM, which can generate more diverse and distinguishable visual or geometric descriptions for classes with complex structures, resulting in higher performance. Conversely, for classes with flat structures, LLM produces less rich descriptions that struggle to capture subtle differences, resulting in not as good performance.\n\nOnce again, we thank you for your positive feedback, constructive criticism, and thoughtful suggestions. Your feedback plays an integral role in refining our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TyQBG5CCUI", "reviewer": "Reviewer_FCL4", "summary": "This paper analyses and addresses two issues of prototype-based methods in the Few-shot Point Cloud Segmentation task. For the constrained semantic information issue, they present the GCPR module to enrich prototypes with text knowledge via LLM and CLIP. For the class information bias issue, the proposed PCPR module mitigates this bias with reliable context aggregation. Further, the dual-distillation regularization promotes the refinement of prototypes.", "questions": "Please refer to Weaknesses.", "limitations": "The authors have analyzed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This idea of enriching prototypes with knowledge from text is innovative, and the impressive performance on various datasets demonstrates its effectiveness.\n\nThe provided visualization is beneficial for explaining the proposed method.", "weaknesses": "As the authors mentioned in the Limitations, LLM and CLIP will bring additional computing costs. Although the proposed offline operation can help to some extent, specific metrics like model parameters and FLOPs should be provided to make a fair comparison with the previous methods. Also, I wonder whether the authors have tried to distillate knowledge from text to prototype directly like these works[1,2], which may be more efficient because there is no need to use LLM and CLIP during inference.\n\n[1] ULIP: Learning Unified Representation of Language, Image and Point Cloud for 3D Understanding, CVPR2023\n\n[2] OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding, NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer FCL4,\n\nWe thank you for taking the time to review our manuscript and offer detailed and constructive comments. We appreciate your positive reception of our work and carefully considered each of your points and would like to address your comments as follows:\n\n> **1. Computing cost and specific metrics.**\n\nThank you for your helpful suggestion. In our approach, generating descriptions with LLM and extracting text features through CLIP for all classes in the dataset are performed offline before the training and testing phases. Once text features are stored through offline operations, we can directly load the stored text features without regenerating by LLM online, avoiding bringing redundant computing costs during training and testing. We will add the analysis of computing cost to our paper, as detailed below:\n\n**(1) Offline computation cost:** The table below shows the time cost of extracting text features for all classes in the entire dataset. We analyze that the total offline time is primarily determined by the description generation process, with feature extraction time being negligible. And the time for generating descriptions from LLM varies on different datasets, depending on the number of classes.\n| Phase | S3DIS | ScanNet |\n|:-:|:-:|:-:|\n|Description Generation: gpt-3.5-turbo|30.23 min|67.15 min|\n|Text Feature Extraction: CLIP rn50|10.95 s|17.79 s|\n|Total|30.41 min|67.45 min|\n\n**(2) Specific metrics:** The table below compares the computational costs and experimental results of our model with SOTA methods under the 2-way 1-shot setting. Our approach effectively balances computational cost and performance, achieving the highest performance with moderate computational cost.\n\n| Methods | #Params  | FLOPs (G) | FPS   | Inference Time (ms) | S3DIS | ScanNet |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| attMPTI | 357.82K  | 152.65    | 1.47  | 678.67   | 54.86 | 41.69   |\n| QGPA    | 2.79M    | 16.30     | 38.68 | 25.85    | 62.76 | 56.51   |\n| DPA     | 4.85M    | 15.49     | 32.35 | 30.91    | 70.19 | 62.90   |\n| Ours    | 4.22M    | 18.96     | 20.57 | 48.61    | **75.74** | **73.93**   |\n\n> **2. Distillate knowledge from text to prototype directly.**\n\nThank you for your insightful comment. [1][2] follow the paradigm of 3D-CLIP contrastive pre-training and aim to distill knowledge from pre-trained CLIP to align 3D, 2D and text representations. Following them, we conduct more experiments to directly distill knowledge from text to 3D prototypes. Specifically, we remove the proposed Text-to-Prototype Compressors and Prototype Distillation Loss in the GCPR module and incorporate a contrastive loss to align 3D prototypes and text features, where the prototype is treated as a query, with text features from the same class as positive samples and those from different classes as negative samples. As you noted, this avoids the need for LLM and CLIP during inference. The experimental results are shown in the table below.\n\n| Loss Weight | 0.00  | 0.10  | 0.20  | 0.50  | 1.00  | Ours  |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| S0    | 66.17 | 68.18 | 68.43 | 65.56 | 63.45 | **74.04** |\n| S1    | 74.60 | 77.00 | 76.43 | 72.46 | 71.09 | **77.44** |\n| Mean  | 70.39 | 72.59 | 72.43 | 69.01 | 67.27 | **75.74** |\n\nWe find that while 3D-text contrastive training can distill text knowledge into 3D prototypes to some extent, it is not as effective as our approach. We analyze that the success of [1][2] depends on the use of large-scale 3D datasets with a large number of classes for pre-training, whereas our FS-3DSeg task involves limited labeled data and non-overlapping training and testing classes, making such distillation methods less effective for this task. In contrast, our approach proposes two Text-to-Prototype Compressors in the GCPR module, which directly aggregate diverse and differentiated text features to enhance the quality of 3D prototypes. Additionally, the prototype distillation loss facilitates effective information transfer between prototypes at different stages, improving the prototype refinement process and leading to superior performance in the FS-3DSeg task.\n\nThank you once again for your positive feedback, and constructive and insightful suggestions. Your feedback is crucial in helping us refine our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fJvED0mfIF", "reviewer": "Reviewer_pd6x", "summary": "This paper introduces a FS-3DSeg framework called Generated and Pseudo Content guided Prototype Refinement (GPCPR) for few-shot 3D point cloud semantic segmentation, leveraging LLM-generated content and reliable query context to enhance prototype quality. GPCPR includes two core components: Generated Content-guided Prototype Refinement (GCPR), which enriches prototypes with comprehensive semantic knowledge from large language model-generated class descriptions, and Pseudo Query Context-guided Prototype Refinement (PCPR), which mitigates class information bias by aggregating reliable class-specific pseudo-query context to create more suitable query-specific prototypes. Additionally, a dual-distillation regularization term is introduced to enable knowledge transfer between early-stage entities and their deeper counterparts, enhancing refinement. Extensive experiments demonstrate the superiority of GPCPR.", "questions": "1. The authors propose using LLM-generated content to enrich prototypes. How sensitive is the model's performance to the quality and diversity of these generated descriptions? Have the authors explored the impact of using different LLMs or prompting strategies?\n2. The paper introduces a dual-distillation regularization term. How does this compare to other knowledge distillation techniques in few-shot learning? What is the specific advantage of this approach in the context of 3D point cloud segmentation?\n3. The paper uses pseudo masks to extract class-specific pseudo-query context. How robust is this approach to potential errors in the initial pseudo mask generation? Have the authors explored the impact of different thresholds or strategies for generating these masks?\n4. The proposed method significantly outperforms state-of-the-art approaches on both S3DIS and ScanNet datasets. Have the authors investigated why the performance gain is notably larger on ScanNet compared to S3DIS?\n5. How does the computational complexity of GPCPR compare to existing methods, especially considering the integration of LLM-generated content and multiple refinement stages?", "limitations": "1. scalability: The complexity of generating differentiated descriptions grows quadratically with the number of classes (O(N^2)). This could pose challenges when applying the method to datasets with a large number of semantic classes.\n2. Dependency on LLMs: The reliance on large language models for generating descriptions introduces an external dependency that may not always be available or consistent across different implementations or time periods.\n3. Computational overhead: The paper lacks a detailed analysis of the additional computational cost introduced by the LLM content generation and multiple refinement stages. This is crucial for understanding the method's practical applicability.\n4. Sensitivity to LLM output: The quality of the generated descriptions could vary depending on the specific LLM used and its training data. The paper doesn't thoroughly explore how variations in LLM output might affect the overall performance.\n5. Pseudo mask reliability: The method relies on generating pseudo masks for query point clouds. However, the paper doesn't extensively analyze how errors in these pseudo masks propagate through the system and affect the final segmentation results.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The proposed method is technically sound.\n- The experiments confirm the effectiveness of the proposed method.\n- The paper is well-written", "weaknesses": "- The novelty is relatively limited. This paper is not the first to explore using LLM for few-shot segmentation; it has already been done in 2D few-shot segmentation [1][2]. Since LLMs inject semantic knowledge into visual features, their rich semantic knowledge should be transferable between 2D and 3D. This paper lacks specific design or optimization for 3D tasks, making the innovation of the GCPR module quite limited.\n- The idea of using pseudo predictions to generate class-specific pseudo-query context in the PCPR module is quite straightforward (not novel) in few-shot segmentation. Additionally, the PCPR module simply stacks techniques from previous works. For example, QGPA directly corresponds to [3], and prototype distillation is identical to [4].\n- The introduction of LLMs may significantly increase computational overhead and inference time compared to previous methods.\n[1]\tLLaFS: When Large Language Models Meet Few-Shot Segmentation  CVPR2024\n[2]\tSimple Semantic-Aided Few-Shot Learning  CVPR2024 \n[3]\tPrototype Adaption and Projection for Few- and Zero-shot 3D Point Cloud Semantic Segmentation  TIP 2023\n[4]\tDynamic Prototype Adaptation with Distillation for Few-shot Point Cloud Segmentation   3DV 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1. Novelty of using LLM. Specific design.**\n1) Novelty: \nFS-3DSeg is more complex than 2D, due to 3D data is complex. No prior work has applied LLMs to FS-3DSeg. Directly extending [1][2] to FS-3DSeg faces challenges: [1] finetunes LLMs and outputs 2D polygon coordinates, unsuitable for 3D objects; [2] lacks 3d-related descriptions. These challenges motivated us to pioneer the use of LLM to solve FS-3DSeg.\n2) Our specific design for 3D tasks:\n①Four 3D-oriented LLM commands for generating diverse class descriptions covering 3D properties like geometric structures and surface shape.\n②3D-specific prompt for generating differentiated descriptions that highlight visual or geometric differences.\n③Two Text-To-Prototype Compressors for enriching 3D prototypes with knowledge from LLM.\n\n> **W2. Novelty of PCPR.**\n\nOur PCPR is the first to use pseudo-query context to refine prototypes in FS-3DSeg. Innovations include query-to-prototype compressor, prototype distillation, and pseudo-prediction distillation, improving 3D prototype quality and pseudo mask accuracy.\n\nDifference with other methods: QGPA is used as baseline.\n[4] only distills prototypes at the last stage, while we distill prototypes at multiple progressive prototype refinement stages to ensure more stable refinement.\nBeyond that, we also propose pseudo-prediction distillation to improve pseudo masks, which have not been explored before. These designs effectively tackle semantic information bias in FS-3DSeg.\n\n> **W3. Computational Overhead.**\n\nThe usage of LLMs does not increase inference time, as descriptions and text features for all classes are generated and stored offline in one phase. During training/testing, we load the stored text features, avoiding redundant computing costs for regenerating them by LLM online. Specifically, the total offline times are 30.41 minutes for S3DIS and 67.45 minutes for S3DIS and ScanNet, mainly due to the text generation process, with negligible feature extraction time (see Table 1 in the rebuttal PDF). During inference, our model better balances time costs and results, achieving superior results with moderate computational cost compared to the SOTA (see Table 2 in the rebuttal PDF).\n\n> **Q1. Impact of LLMs or prompting strategies.**\n1) Different LLMs: Using both `gpt-3.5-turbo` and `gpt-4o-mini` can achieve superior results compared with SOTA, with 75.74% mIoU and 73.88% mIoU on S3DIS, respectively. (see Table 3 in rebuttal PDF).\n2) Prompting Strategies: Differentiated descriptions outperform diverse descriptions by 0.9% mIoU. Combining both strategies yields best performance (see Table 3 in our paper).\n3) Number of Descriptions: As shown in Figure 4 (c) of our paper, mIoU increases with more diverse descriptions, stabilizing around 10/command.\n\n> **Q2. Compare DD loss with other KD.**\n1) Compared with other KD: We verify the effects of logit distillation, feature distillation, and relational knowledge distillation. Our DD loss performs better by adaptively improving the quality of pseudo masks and prototypes through multi-stage distillation (see Table 4 in rebuttal PDF).\n2) Advantage: The complexity of 3D data leads to low-quality prototypes and inaccurate predictions. Our DD loss enables early-stage 3D prototypes or predictions to gain insights from their deeper optimal counterparts, facilitating bidirectional information exchange, and resulting in accurate segmentation results. \n\n> **Q3. Effects of pseudo mask errors.**\n\n1) Robustness: Our method is robust to pseudo mask errors, since PCPR and DD loss enable prototypes and prediction to mutually benefit each other, adaptively improving pseudo mask quality.\n2) Impact of thresholds: Experimental results in Table 5 in rebuttal PDF file reveal:\n- Threshold = 0: Optimal performance without filtering operation (reported in our paper).\n- Threshold <= 0.1: Comparable to Threshold=0, indicating effective integration of pseudo query context. \n- Threshold > 0.1: Performance decreases as most query features are filtered out.\n\n> **Q4. Why performance gain is larger on ScanNet?**\n\n1) ScanNet has more classes, so the limited 3D support set is insufficient to distinguish classes. Our GCPR integrates rich semantic knowledge from LLM into 3D prototypes to improve performance.\n2) ScanNet’s complex scenarios lead to higher class information bias. Our PCPR integrates reliable pseudo query context to refine prototypes, boosting performance.\n\n> **Q5. Computational complexity of GPCPR.**\n\nGCPR and PCPR take 12.84 ms and 11.01 ms, respectively. We achieve the best performance with moderate total computing cost (see Table 2 in rebuttal PDF). \n\n> **L1. Scalability.**\n\nIn the offline stage for generating differentiated descriptions, it takes 20 minutes and 50 minutes for S3DIS and ScanNet. To address generation complexity for more classes, we provide the following alternatives: \n1) We can divide all classes into several groups and generate differentiated text in parallel. This greatly reduces the time cost and maintains our superior performance. \n2) We can remove the differentiated description generation process and only use diverse descriptions. By this, mIoU drops slightly from 75.74% (reported in the paper) to 73.84 % but still far exceeds SOTA (DPA 70.19%).\n\n> **L2. Dependency on LLMs.**\n\nDescriptions and text features of all classes are generated and stored in offline stage. During training/testing, we load stored text features without regeneration, thus ensuring availability and consistency across different settings or time periods. Besides, we do not rely on a specific LLM, using other LLMs (e.g., `gpt-4o-mini`) in GCPR module still far surpasses SOTA (see Table 3 in rebuttal PDF). In addition, even if no LLM is available, using alternative texts such as Word2Vec or CLIP Text still exceeds the SOTA due to the effective cooperation of our GCPR, PCPR, and DD losses (see Figure 4(b) of our paper).\n\nFor L3-L5, please see our response to W3, Q1 and Q3, respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
To PC, AC, and all Reviewers:

We sincerely appreciate the time and effort the PC, AC, and all reviewers have dedicated to reviewing our work. We are grateful for the detailed and thoughtful feedback on our submission, particularly the positive comments and insights. Below, we summarize the strengths recognized by the reviewers:

1. **Innovation**: The proposed method is interesting, innovative and technically sound. (Reviewer ABMp , Reviewer FCL4, Reviewer pd6x)
2. **Effectiveness**: The impressive performance on various datasets demonstrates the effectiveness of the proposed method. (Reviewer FCL4, Reviewer pd6x)
3. **Readable**: The paper is well-written and easy to read. (Reviewer pd6x, Reviewer ABMp)
4. **Beneficial visualizations**: The paper includes beneficial visualizations for explaining the proposed method. (Reviewer FCL4)

Thank you once again for your time and effort in reviewing this manuscript. For the questions and other concerns raised by the reviewer, we respond to each in detail and outlined our plans for improvement. If any questions are not answered or our response is unclear, we would appreciate the opportunity to communicate further with our reviewer.

Meta-review: Generally the reviewers agree on the contribution, e.g. from one of the reviews: This idea of enriching prototypes with knowledge from text is innovative, and the impressive performance on various datasets demonstrates its effectiveness.  The provided visualization is beneficial for explaining the proposed method.  Also the reviewers agree on the potential downside of this approach relying on LLMs and the concomitant issues and compute required.  Overall accept.
Decision: Accept (spotlight)