Generated and Pseudo Content guided Prototype Refinement for Few-shot Point Cloud Segmentation

Authors: lili  wei
congyan  lang
ziyi  chen
tao  wang
yidong  li
jun  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 02:41:38

Abstract: Few-shot 3D point cloud semantic segmentation aims to segment query point clouds with only a few annotated support point clouds. Existing prototype-based methods learn prototypes from the 3D support set to guide the segmentation of query point clouds. However, they encounter the challenge of low prototype quality due to constrained semantic information in the 3D support set and class information bias between support and query sets. To address these issues, in this paper, we propose a novel framework called Generated and Pseudo Content guided Prototype Refinement (GPCPR), which explicitly leverages LLM-generated content and reliable query context to enhance prototype quality. GPCPR achieves prototype refinement through two core components: LLM-driven Generated Content-guided Prototype Refinement (GCPR) and Pseudo Query Context-guided Prototype Refinement (PCPR). Specifically, GCPR integrates diverse and differentiated class descriptions generated by large language models to enrich prototypes with comprehensive semantic knowledge. PCPR further aggregates reliable class-specific pseudo-query context to mitigate class information bias and generate more suitable query-specific prototypes. Furthermore, we introduce a dual-distillation regularization term, enabling knowledge transfer between early-stage entities (prototypes or pseudo predictions) and their deeper counterparts to enhance refinement. Extensive experiments demonstrate the superiority of our method, surpassing the state-of-the-art methods by up to 12.10% and 13.75% mIoU on S3DIS and ScanNet, respectively.

Main paper: 
 introduction: 
 Point cloud semantic segmentation, aiming to assign a semantic label to each point in the 3D point clouds, benefits various applications like autonomous driving, robotics, and augmented/virtual reality. Despite the remarkable performance of supervised methods in this field [33, 34, 22, 24, 12, 37] , they rely on labor-intensive labeled data and struggle to distinguish novel classes not present in the training data. To address these issues, few-shot 3D point cloud semantic segmentation (FS-3DSeg), which segments novel classes with only a few labeled data, has gained widespread attention. FS-3DSeg aims to learn a model on base classes and generalize it to novel classes with only a few annotated point clouds. Existing FS-3DSeg methods [39, 17, 36, 9, 41, 20, 10, 16] typically adopt prototype-based paradigms [28, 32] , where prototypes extracted from the support point clouds are treated as class descriptors to guide the segmentation of unlabeled query point clouds by matching query point features with prototypes. Due to their representation and generalization capabilities, prototype-based methods have made great progress and garnered increasing attention. However, their performance remains unsatisfactory due to encountering the challenge of low prototype quality, attributed to two reasons: 1) Semantic information constraints: Limited 3D support point clouds contain only partial and incomplete object information, lacking both intra-class diversity and interclass discriminative information. Consequently, vanilla prototypes generated from the 3D support set are insufficient to be comprehensive class descriptors. 2) Class information bias: Query and support point clouds exhibit intra-class object variations [16, 20] and feature distribution gaps [9, 10] , making vanilla prototypes may be ill-suited for segmenting query samples as the distances between vanilla prototypes and query features will be far away.To address these issues, we propose to compensate for the lack of semantics in 3D support set to generate comprehensive and reliable query-specific prototypes for accurately segmenting the query point cloud. This can be achieved in two steps. Firstly, to alleviate class information constraints, motivated by the fact that texts are more controllable and accessible [35] , we aim to integrate general semantic knowledge from texts to refine prototypes. Fortunately, Large Language Models (LLM) (e.g., ChatGPT) [3, 35, 29] offer promising opportunities to help achieve this goal due to their wealth of implicit knowledge and have been leveraged for a variety of downstream tasks, such as image segmentation [42] and point cloud classification [43] . Inspired by this, we leverage the capabilities of LLM to explicitly generate diverse and differentiated contents, combined with novel designs to obtain refined prototypes with richer semantics. Secondly, to reduce class information bias, previous works [9, 20, 16, 10] have endeavored to map prototypes to query feature space through prototypequery interaction using cross-attention [31] . However, noisy interactions between prototypes of one class and query features of other classes may introduce class-irrelevant cues, thereby undermining the reliability of prototypes. We observed that pseudo masks for query point clouds can assist in extracting class-specific pseudo-query context and filtering out class-irrelevant noisy features.Ultimately, in this paper, we propose a novel FS-3DSeg framework via Generated and Pseudo Content guided Prototype Refinement (GPCPR), which progressively refines prototypes by two newly proposed components: LLM-driven Generated Content-guided Prototype Refinement (GCPR) and Pseudo Query Context-guided Prototype Refinement (PCPR). Initially, vanilla prototypes are generated from 3D support features. Firstly, our GCPR prompts the LLM to generate diverse descriptions of single classes and differentiated descriptions of class pairs. Then we design text adapters and two text-to-prototype compressors for mapping and incorporating text knowledge into vanilla prototypes, yielding refined prototype covering with richer and discriminative semantic knowledge. Secondly, leveraging the refined prototypes, PCPR generates pseudo masks for query point clouds to derive the class-specific pseudo-query context, which can be aggregated into prototypes by a newly designed query-to-prototype compressor, yielding more reliable prototypes that can better associate with query points. Beyond that, we also introduce a dual-distillation regularization term, which enables early-stage entities (prototypes or pseudo predictions) to gain insight from their deeper optimal counterparts. This facilitates bidirectional information exchange in both forward and backward processes during training, thus obtaining better prototypes and predictions.We summarise our contributions as follows: 1) We propose GPCPR, a novel end-to-end FS-3DSeg framework that enhances prototype quality by simultaneously integrating LLM-generated content and reliable query context to generate query-specific prototypes. To the best of our knowledge, this is the first time leveraging LLM's capabilities to segment novel classes in FS-3DSeg. 2) We design a series of novel modules, including the Generated Content-guided Prototype Refinement (GCPR) module and the Pseudo Query Context-guided Prototype Refinement (PCPR) module, to facilitate the prototype refinement process. Additionally, we design a dual-distillation regularization term to further mutually enhance the refinement. 3) Extensive experiments demonstrate the superiority of our method, notably exceeding state-of-the-art methods by up to 12.10% and 13.75% on S3DIS and ScanNet datasets, respectively. 
3d point cloud semantic segmentation: 
 3D point cloud semantic segmentation (3DSeg) aims to assign semantic labels to each point within point clouds. Supervised 3DSeg studies involve voxel-based methods [7, 18] and point-based methods [22, 33, 23, 24, 12, 37] , with the latter receiving increasing attention due to the simplicity, flexibility, and efficiency. For example, DGCNN [33] proposes the EdgeConv module to capture local structures. Recently, [12, 37] design self-attention-like networks to model long-range contexts from distant neighbors. Despite their effectiveness, these methods often rely on extensive annotations and cannot segment novel classes. In this paper, following mainstream FS-3DSeg methods [39, 9, 16] , we utilize DGCNN as the point encoder and extend its capability to segment novel classes.Most few-shot 3D point cloud semantic segmentation (FS-3DSeg) methods [39, 17, 9, 20, 10, 41, 16, 1] follow the prototype-based paradigms [32, 28, 11] , learning prototypes from the support set to segment query point clouds. Specifically, AttMPTI [39] , the pioneering FS-3DSeg method, proposes an attention-aware multi-prototype transductive inference framework based on label propagation. BFG [17] mutually embeds global perception into local features and prototypes. To reduce contextual gaps between support prototypes and query features, [20, 10, 9, 16] adapt prototypes to query feature space by prototype-query feature interaction via cross attention [31] . COSeg [1] introduces correlation optimization by refining the multi-prototypical support-query correlations. In addition to prototype learning, SCAT [36] explores class-specific relations between query and support features using transformer blocks [31] instead of pooling operations. However, these methods often yield low-quality prototypes since the 3D support set only contains constrained semantic information. Besides, their prototype-query interaction [20, 10, 9, 16] may introduce noise. In contrast, our approach refines prototypes by integrating LLM-generated content and class-specific pseudo-query context, resulting in well-suited query-specific prototypes for better segmentation.Recently, several large models have emerged, including large language models (e.g., GPT-3 [3] , ChatGPT [35] and BERT [5] ) and large multi-modal models (e.g., CLIP [25] , BLIP [13] , LLaVA [15] , GPT-4 Vision [35] and MiniGPT [40] ). These extensive models are repositories of extensive knowledge and have achieved widespread success in diverse downstream tasks, such as robot task plans [27] , image classification [26] and image segmentation [42] . In the field of 3D point cloud understanding, several large models have also emerged that associate point clouds, texts and other modalities (e.g., Point-LLM [8] , MiniGPT-3D [29] , Uni3D-LLM [14] and PointCLIP V2 [43] ). However, none of them are designed for segmenting novel classes in FS-3DSeg. In this work, we leverage the rich generated content of LLM to assist in addressing the FS-3DSeg problem. 
problem formulation and overview: 
 Problem Formulation. Following previous FS-3DSeg methods [39, 9] , we adopt meta-learning based paradigm. Specifically, all semantic classes are divided into base class set C base and novel class set C novel for training and testing respectively, where C base ∩ C novel = ∅. Each few-shot task (i.e., episode) instantiates an N -way K-shot segmentation task. The input data in each episode contains a support setS = {(I n,k s , M n,k s ) K k=1 } N n=1 and a query set Q = {(I i q , M i q )} T i=1, where N , K and T denote the number of classes, the number of support point clouds for each class, and the number of query point clouds. I n,k s and I i q denote support and query point clouds, each contains M points. M n,k s ∈ {0, 1} M ×1 denotes binary support GT mask for each of the N unique classes. M i q ∈ {0, ..., N } M ×1 denotes the query GT mask, which is only available during training. The class name set contains one background name and N foreground class names, denoted as C = {c n } N n=0 . The goal of FS-3DSeg is to learn a model to predict segmentation mask Mq for I q based on S.Overview. Figure 1 illustrates the architecture of our proposed method. Given query set and support set Q and S, following common practice [39, 9] , we utilize a shared point encoder to extract per point query and support features, represented as F q ∈ R T ×M ×d and F s ∈ R N ×K×M ×d , where d denotes feature dimension. Then we adopt masked average pooling (MAP) [9] to generate vanilla class-wise 3D support prototypes P = {P n } N n=0 ∈ R (N +1)×d from F s , including a background prototype and N foreground prototypes. To enhance the quality of P, we propose two modules: LLM- . To effectively incorporate text knowledge into P, GCPR sequentially utilizes two text-to-prototype compressors to generate refined prototypes T with rich semantics. After that, PCPR generates pseudo masks Ṁq to extract class-specific pseudo query contexts Ḟq for Q, followed by a query-to-prototype compressor to generate well-suited query-specific prototypes P. Finally, cosine similarities between F q and P are calculated to segment query point clouds, where each point cloud is assigned the label of the most similar prototype. To supervise the model training, we adopt standard cross-entropy loss and design a dual-distillation regularization term (DD loss) to further enhance the refinement process. Subsequently, we provide detailed descriptions of GCPR, PCPR, and DD loss as below.Due to support point clouds only containing constrained semantic information, the vanilla prototypes P lack class diversity and cannot be better associated with Q. To address this, inspired by the commonsense knowledge in LLM and the fact that textual descriptions are more controllable [35] , we aim to leverage LLM to generate detailed class content and propose two text-to-prototype compressors to embed these texts into prototypes P, thereby obtaining refined prototypes with richer semantics.LLM-driven Content Generation. By leveraging the extended world knowledge of LLM (e.g., GPT-3 [3] ), we generate two types of content at scale, i.e., diverse class descriptions and differentiated class descriptions, respectively for enhancing the comprehensiveness and discriminability of the prototypes P.1) Diverse Class Descriptions. To generate diverse descriptions, effective prompt templates for LLMs are essential. Inspired by the heuristic 3D-oriented commands for depth maps in [43] , we designed the following 3D-specific heuristic prompt templates to ensure that the texts encompass sufficient point cloud properties: For each command, we place "[CLASS]" with a class name and feed it into GPT-3 to generate N div descriptions. The generated description set can be represented as: D = {D cn } N n=0 , where D cn contains 4 × N div descriptions.2) Differentiated Class Descriptions. To enhance the model's ability to distinguish between multiple classes, inspired by [6] which generate differentiated attribute descriptions between ambiguous classes for image classification, we intend to generate 3D-specific differentiated descriptions for any pair of classes, explaining their visual and geometric differences. To this end, we designed the following template to prompt LLM to generate differentiated class descriptions: "Describe the distinguishing visual or geometry differences between the point clouds of [CLASS 1] and [CLASS 2] in pairs of sentences. Generate as many captions as you can."We place "[CLASS 1]" and "[CLASS 2]" with a pair of class names and trigger GPT-3 to produce several sentences, each of which is split into two parts at the conjunctions, with the first (or second) part assigned as a differentiated description for class c 1 (or c 2 ), denoted asD c2 c1 ′ (or D c1 c2 ′ ).For each class, we combine all its differentiated descriptions with other classes to contain more discriminative information, forming a differential description setD ′ = {D ′ cn } N n=0 , where D ′ cn = {D ci cn ′ } ci∈C\{cn} includes N dif f descriptions.Text-To-Prototype Compressors. Given LLM-generated content D and D ′ , we employ a frozen CLIP textual encoder [25] and a shared text adapter to extract text featuresE ∈ R (N +1)×(4×N div )×d and E ′ ∈ R (N +1)×N dif f ×d .To incorporate text knowledge into vanilla prototypes P, we propose two cross-attention-based [31] text-to-prototype compressors, i.e., the class text-to-prototype compressor and the differentiated text-to-prototype compressor, which measure the relations between P and texts to aggregate semantic information.1) Class text-to-prototype compressor integrates features E for diverse descriptions into vanilla prototypes P to obtain refined prototypes Ṫ = { Ṫ n } N n=0 with richer semantics, each formulated as:EQUATIONwhere 2) Differentiated text-to-prototype compressor integrates features E ′ for differentiated descriptions into previous prototypes Ṫ to obtain refined prototypes T = { T n } N n=0 with discriminative information, each formulated as:Q1 = P n Wq1 ∈ R 1×d ′ , K1 = E n W k1 ∈ R (4×N div )×d ′ and V1 = E n Wv1 ∈ R (4×N div )×d . Wq1, W k1 ∈ R d×dEQUATIONwhereQ2 = Ṫ n Wq2 ∈ R 1×d ′ , K2 = E n ′ W k2 ∈ R N dif f ×d ′ , and V2 = E n ′ Wv2 ∈ R N dif f ×d . Wq2, W k2 ∈ R d×d ′and Wv2 ∈ R d×d are learnable parameters for fully connected layers.Query and support point clouds often exhibit large object variations and class information bias, making vanilla support prototypes P unsuitable for segmenting query point clouds. Although our GCPR compensates for the lack of semantics P, they still lack query-specific associations. Previous approaches have endeavored to customize query-specific prototypes through prototype-query feature interaction using cross-attention [31] . Specifically, QGPNet [10] and QGPA [9] align support prototypes with query features to reduce the channel-wise feature distribution gap but lack query-specific context. QGE [20] and DPA [16] aggregate query context into support prototypes, but inevitably introduce noisy interactions between prototypes of one class and query features of other classes. This could inadvertently incorporate class-irrelevant query cues into prototypes, thereby undermining their reliability and exacerbating the challenge of distinguishing confusing classes. To address this issue, we aim to generate more reliable class-specific pseudo query context from query features and propose a transformer-based query-to-prototype compressor to customize well-suited query-specific prototypes.Pseudo-Query Context Generation. To extract class-specific context, we need to predict pseudo masks M i q for each query point cloud I i q and extract class-specific context Ḟi q accordingly, effectively filtering out interfering cues from other classes. However, the channel distribution gap between prototypes T and query features F i q may lead to inaccurate pseudo masks. Thus we follow QGPA [9] to rectify current prototypes T to query feature channel distribution through channel-wise cross-attention [31] , formulated as:EQUATIONwhereEQUATIONV3 = T n Wv3 ∈ R 1×d . F n s ∈ R M ×d denotes averaged support features, calculated by F n s = 1/(N K) • N,k F n,k s if n = 0, otherwise F n s = 1/K • k F n,k s . Wq3, W k3 ∈ R M ×M ′and Wv3 ∈ R d×d are the learnable parameters for FC. Then queryspecific prototypes Ṗ = { Ṗi } T i=1 ∈ R T ×(N +1)×d are obtained, where P i denote query-specific prototypes for each query point cloud I i q .Then, we calculate cosine similarity between F i q and prototypes Ṗi to predict pseudo logits Li q ∈ R M ×(N +1) and pseudo one hot masks Ṁi q ∈ R M ×(N +1) . By masking query features with pseudo masks, we obtain class-specific pseudo query context, represented as:Ḟq = { Ḟn q } N n=0 , Ḟn q ∈ R (T ×M )×d. Query-To-Prototype Compressor. The query-to-prototype compressor aggregates pseudo query context features into the prototypes by cross attention, represented as:EQUATIONwhereQ4 = Ṗi,n Wq4 ∈ R 1×d ′ , K4 = Ḟn q W k4 ∈ R (T ×M )×d ′ , V4 = Ḟn q Wv4 ∈ R (T ×M )×d . Wq4, W k4 ∈ R d×d ′and Wv4 ∈ R d×d are learnable parameters for fully connected layers. After that, we obtain the final queryspecific prototypes well-suited to segment query point clouds, represented asP = { Pi } T i=1 ∈ R T ×(N +1)×d .Finally, we calculate cosine similarity scores between F i q and Pi to obtain predicted logits Li q ∈ R M ×(N +1) . Each point is then assigned the label of the most similar prototype to generate the final predicted mask Mi q .To further enhance prototype refinement, we introduce a dual-distillation regularization term that allows earlystage entities to gain insights from their deeper counterparts, including two types of self-distillation: i.e., prototype distillation and pseudo prediction distillation.Prototype Distillation. In the forward process, GCPR and PCPR integrate useful semantics into early-stage prototypes to generate refined prototypes. To achieve mutually beneficial and bi-directional optimization of multi-stage prototypes, we treat early-stage prototypes (e.g., Ṫ or Ṗ) as student prototypes and deep-stage prototypes (e.g., T or P) as teacher prototypes, employing prototype distillation in GCPR and PCPR, defined as:EQUATIONwhere KL refers to the Kullback-Leibler divergence. The self-distillation process facilitates effective knowledge transfer from teacher prototypes to student prototypes, enhancing the refinement process.Pseudo Prediction Distillation. In PCPR, pseudo masks play a core role as they determine the reliability of the generated class-specific pseudo query context. To further improve the accuracy and quality of pseudo masks, we treat pseudo logits Lq as students and final predicted logits Lq as teachers, and employ logit distillation in PCPR, formulated as:EQUATION3.5 Objective.During training, the proposed model is supervised by two loss functions, i.e., a standard cross-entropy loss LSEG and the proposed dual-distillation loss (DD loss). The cross-entropy loss serves as the main optimization target, aimed at learning comprehensive and reliable query-specific prototypes. The dual-distillation loss acts as a regularization term to promote knowledge transfer between early-stage entities (prototypes or pseudo-predictions) and their deeper counterparts. Formally, the cross entropy loss is defined as:EQUATIONThe overall loss is a weighted combination of LSEG and DD loss with a balancing weight λ, represented as: Datasets. We verify our method on S3DIS [2] and ScanNet [4] . S3DIS contains 3D-RGB point clouds collected from 272 rooms across six indoor scenes. Its label set includes 13 classes (12 semantic classes and background clutter). ScanNet collects 1,513 point clouds from 707 indoor scenes. Its label set includes 21 classes (unannotated background and 20 semantic classes). Due to the inconvenience of processing the original room which contains an excessive number of points, we follow the pre-processing strategy in [39, 9, 16] to divide each room into 1m × 1m blocks and randomly sample M = 2048 points from each block, yielding 7,547 blocks for S3DIS and 36,350 blocks for ScanNet, respectively. Each point contains a 9D vector, including XYZ, RGB, and normalized coordinates. Following [39] , semantic classes are evenly split into two non-overlapping subsets, denoted as S 0 and S 1 . For meta-learning, we train our model on one fold (e.g., S 0 ) and test on another fold (e.g., S 1 ). Vice versa for cross-validation.EQUATIONEvaluation Metrics. Following conventions in the 3DSeg community, we average the mean Intersection-over-Union (mIoU) across all test classes and report the final mIoU metric under the few-shot setting.Implementation details. In Appendix B, we provide framework details and training details. 
comparison with state-of-the-art methods: 
 Results on S3DIS. Table 1 shows the experimental results of our method compared with state-of-the-art methods on the S3IDS dataset. Our method demonstrates a substantial improvement over the baseline QGPA [9] , with a large margin of 6.22%∼14.59% across various splitting settings, highlighting a significant improvement for FS-3DSeg. Additionally, our method consistently exceeds all of the previous methods, particularly surpassing state-of-the-art methods DPA [16] by a margin of 1.19%∼12.10% across various splitting settings, showcasing our effectiveness and superiority. The success of our approach is attributed to the integration of comprehensive LLM-generated content and reliable pseudo-query context by GCPR and PCPR, solving the problem of semantic information constraints and class information bias, as well as enhancing refinement by dual-distillation loss. Results on ScanNet. Table 2 presents the segmentation performance on ScanNet. Our method significantly improves the baseline by a larger margin of 11.01%∼18.86%. Notably, we significantly outperform SOTA method DPA [16] by large margins of 3.96%∼13.75% across various settings. Our model exhibits even greater improvement on ScanNet compared to S3DIS due to containing more classes and complex scenarios in ScanNet, which greatly challenges prototype quality. Our approach effectively integrates more comprehensive and reliable content to cope with the lack of semantic information, resulting in superior performance.Computational Complexity. In our approach, generating descriptions with LLM and extracting text features through CLIP for all classes in the dataset are performed offline before the training and testing phases. Once text features are stored through offline operations, we can directly load the stored text features without regenerating by LLM online, avoiding bringing redundant computing costs during training and testing. (1) Offline cost: Table 3 shows the time cost of extracting text features for all classes in the entire dataset. We analyze that the total offline time is primarily determined by the description generation process, with feature extraction time being negligible. The time for generating descriptions from LLM varies on different datasets, depending on the number of classes. (2) Online cost: Table 4 shows that our method achieves a better balance between computational cost and experiment performance, offering superior segmentation results with reasonable computational efficiency.Qualitative results. In Figure 2 , we visualize the segmentation results of the 2-way 1-shot segmentation task on S3DIS and ScanNet. We compare our predictions with the GT mask and QGPA. As shown, QGPA often produces inaccurate segmentation, particularly in distinguishing background from foreground classes and among different foreground classes. In contrast, our method achieves accurate segmentation for most areas of the target scene. For instance, in the second column of S3DIS examples, QGPA mistakenly predicts parts of the beam and bookcase, whereas our method effectively distinguishes between these classes. Similar results are observed in the last column of ScanNet examples. The superiority of our method is because we incorporate LLM-generated content and pseudo-query context to obtain query-specific prototypes, resulting in better segmentation results. We perform an ablation study on the S3DIS dataset under the 2-way 1-shot setting to validate the effectiveness of our proposed GPCPR. We use QGPA [9] as the baseline to analyze the effects of the proposed components and hyper-parameters. 2) Effects of GCPR: Only using diverse descriptions D or differentiated descriptions D ′ , we exceed the baseline by 6.53% or 7.43%, with the latter playing a greater role because they directly improve the inter-class discrimination of the prototypes. After jointly using D and D ′ , the improvement over the baseline reaches 10.62%, showing that GCPR effectively makes up for the lack of semantics of the prototype. Comparing PCPR and GCPR, the latter can achieve greater improvement because LLM provides sufficient semantic details. Finally, combining PCPR and QCPR, our method improves the baseline by 11.19%.3) Effects of DD loss: Compared to only using PCPR and QCPR, after further applying LT P , LQP and LQM respectively, the model performance further improved 0.40%, 1.38% and 1.56%, with the last one being more effective due to directly optimizing pseudo masks to obtain a more reliable class-specific pseudo query context. By jointly using all distillation losses, we achieve the best performance of 75.74%.Visualization of features and prototypes. As shown in Figure 4 , we use the t-SNE visualization tool [30] to compare the feature distribution and the class prototype distribution to analyze the effect of the model. Here, prediction accuracy is also provided for a clearer comparison. As marked by the red dotted circles, for QGPA, the distance between prototypes and most query features is far. This indicates there exists bias between prototypes and query features due to the lack of query context in the prototypes. In contrast, our method significantly improves accuracy and pushes the prototype closer to query features, as marked by the green dotted circles. This improvement is because we effectively inject rich text knowledge and reliable class-specific query context into the refined prototypes, enhancing their representation capabilities and mitigating object variations.Effects of different adaptors. As shown in Figure 4 (a), without introducing any additional text information by removing the GCPR module, we evaluate the effectiveness of prototype refinement and compare our method with the previous prototype adaptors, including QGPA [9] and DPA [16] . Our method significantly outperforms QGPA and DPA. This is because although QGPA alleviates the channel-wise feature distribution gap between the prototype and query features, the prototype lacks query context information and is therefore less effective. Compared with QGPA, DPA injects query context information into the prototypes. However, query context from other classes acts as noise interference, reducing the reliability of the prototype and limiting performance. Our method introduces category-specific pseudo-context information into the prototype, effectively reducing noise interference while bridging object gaps and thus achieving the highest performance.Effects of different types of descriptions. As shown in Figure 4 (b), we also studied the impact of different text descriptions, by replacing the proposed LLM-generated descriptions with word2vec (W2V) [19] or CLIP texts [25, 38] . As shown, word2vec exhibits limited improvement due to only providing static and restricted semantic information. CLIP texts slightly better improve the baseline, benefiting from the cross-modal alignment capability and rich semantic information learned from large-scale datasets. Our LLM-generated descriptions achieve the best performance because LLM's powerful detailed description ability can inject sufficient diverse and differentiated semantic information into the prototypes.Effects of the number of diverse descriptions. Figure 4 (c) illustrates the impact of the number N div of diverse descriptions. Compared with not using any diverse descriptions, the model performance gradually improves as N div increases, because general knowledge is gradually injected into the prototypes to make up for the lack of semantic information, and reaches promising results when N div = 10. However, further increasing of N div does not yield higher performance as the LLM-generated descriptions begin to repeat, providing redundant semantic knowledge. In summary, we select N div = 10 to ensure both the effectiveness and efficiency of the model.Effects of coefficient λ for DD Loss. Figure 4(d) illustrates the impact of the weight λ of the DD loss in Eq. 8. Compared to λ = 0 (where the DD loss is deprecated), as λ increases, the prototypes and predictions in the final stage begin to gradually optimize the corresponding items in the early stages, allowing the model to generate high-quality prototypes, resulting in significant performance improvements, and reaching the best performance when λ = 1. However, a larger λ (e.g., 2) does not yield higher performance and may even harm model performance. In summary, an appropriate value of λ = 1 yields the best results. 6 , using both "gpt-3.5-turbo" [3] and "gpt-4o-mini" [21] can achieve superior results compared with SOTA, with 75.74% mIoU and 73.88% mIoU on S3DIS, respectively. Specifically, Ours (gpt-3.5-turbo) performs best. Ours (gpt-4o-mini) performs slightly lower than Ours (gpt-3.5-turbo), because "gpt-4omini" tends to generate longer paragraphs, which potentially contain content that is not directly related to the class semantics, thus introducing noise to the class prototypes. Thus in this paper, we choose to utilize "gpt-3.5-turbo".Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We propose a specific model for FS-3DSeg. In Section 3, we give a detailed description of the overview architecture and all components. In Appendix B, we provide settings of framework details and training details. We believe this model is fully replicable. Our code will be made publicly available once it is fully organized.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 In this paper, to solve the problem of constrained semantic information and class information bias in FS-3DSeg, we propose GPCPR that LLM-generated content and reliable query context to enhance prototype quality. GPCPR consists of two components: LLM-driven Generated Content-guided Prototype Refinement (GCPR), which enriches prototypes with diverse and differentiated text knowledge from LLMs, and Pseudo Query Contextguided Prototype Refinement (PCPR), which aggregates reliable class-specific pseudo-query context to mitigate class information bias. Additionally, a dual-distillation regularization term is proposed to facilitate knowledge transfer between early-stage and deeper entities (prototypes or predictions). Experiments show that GPCPR outperforms state-of-the-art methods by up to 12.10% and 13.75% mIoU on S3DIS and ScanNet, respectively. In Appendix C, we discuss the limitations of our work and future work.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: Our code will be made publicly available once it is fully organized.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Following common practice among mainstream methods in the FS-3DSeg field, we do not report error bars. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: It's no societal impact of our work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have done it as described in the question.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release a new dataset, our code will be made publicly available once it is fully organized.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our work does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our work does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: LLM-generated descriptions and text features for all classes are generated and stored offline. During training and testing, we directly load the stored text features, avoiding online regenerating by LLM. To more intuitively understand the diverse and differentiated descriptions generated by the LLM, we provide examples of the generated descriptions for the N-way K-shot segmentation task in Figure 5 and Figure 6 , respectively.For a single class, the descriptions generated by LLM capture rich information such as shape and geometric structure, location, detailed features, usage scenarios, and placement styles. These descriptions provide a comprehensive understanding of the class, enhancing the prototypes with detailed and varied semantic knowledge. For each class, we prompt the LLM to generate N div descriptions for each command. For each N -way K-shot segmentation task, we obtain (N + 1) × (4 × N div ) descriptions, where the number N div will affect the model performance, as too few or too many descriptions will lead to insufficient or redundant semantic information. Therefore, we analyze the effect of D div in Figure 4 (c), determining that the optimal value is 10.For class pairs, differential descriptions generated by the LLM effectively distinguish class pairs by highlighting structural differences, density differences, surface differences, and edge differences, beneficial for distinguishing similar classes. In this paper, using our proposed prompting template, the LLM tends to generate varying numbers of sentences for each class pair based on their distinguishability. We then split each sentence by conjunctions and assign each part to the relevant class. Consequently, after combining all differential sentences for each class in relation to all other classes, the number of differential descriptions N dif f varies for each class. For each N -way K-shot segmentation task, we load descriptions based on the minimum N dif f among the (N + 1) classes. For classes with more descriptions, we randomly load N dif f descriptions, ensuring consistent input dimensions. Thus we obtain (N + 1) × N dif f descriptions per task.Intuitively, differential descriptions are more conducive to improving the discrimination of prototypes, as verified by ablation experiments in Table 5 . Besides, the joint usage of both kinds of descriptions leads to the best performance in FS-3DSeg.Framework details. We use QGPA [9] as the baseline, optimized by segmentation loss, self-reconstruction loss, and alignment loss. DGCNN [33] (with SAN) and pre-trained CLIP (rn50) [25] are used as point encoder and text encoder, respectively. The text adaptors adopt a Linear+LeakyReLU+Dropout+Linear architecture. In our GCPR module, the number ND of descriptions per command is set to 10. We use "gpt-3.5-turbo" GPT-3.5 engine [3] and set the temperature constant to 0.99. For diverse descriptions, the largest length of a 3D-specific diverse description is set to 40, and N div is set to 10. For differential descriptions, LLM generates between 1 to 20 sentences for different class pairs, causing N dif f to vary across classes, typically ranging from 14 to 34. For all compressors based on multi-head attention, the number of heads is set to 4. In Eq. 8, the coefficient λ for distillation loss in Eq. 8 is set to 1.Training details. Our model is implemented by PyTorch and trained on single NVIDIA RTX A4000 GPU. Before meta-learning, following [39] , we pre-train the point encoder on the base classes for 100 epochs. Then, during meta-training, we train our model on 40,000 episodes randomly sampled from base classes, using an Adam optimizer, with a learning rate of 0.001, and a decay step of 5000. During meta-testing, we perform model evaluation on 100 randomly sampled episodes for each feasible class combination from unseen classes. For the query set, T is set to 1 for each class.Limitations. Although our GPCPR achieves remarkable results, GPCPR has the following limitations. (1) Integrating LLM and CLIP into the FS-3DSeg framework will increase resource consumption and computing costs, depending on the parameter amount and execution speed of these large models. In order to alleviate this limitation, we separated some operations in GCPR into offline operations, such as prompting LLM to generate description content and using CLIP to extract text features. (2) The complexity for LLM to generate differentiated descriptions is O(N 2 ), which may face challenges when extending to other domain datasets with massive semantic classes due to the need to exhaustively enumerate all possible class pairs. Future work. The performance of the proposed method is affected by the quality of descriptions generated by LLM and the accuracy of pseudo masks. Biased class descriptions and inaccurate pseudo masks may introduce noise, negatively impacting model performance. Future work will mainly focus on improving the quality and reliability of LLM-generated content and pseudo masks. Furthermore, in this paper, we mainly address segmenting unseen categories in the same scene, but its generalization to different domains with unseen classes remains to be tested. Future research may involve extensive validation across various datasets and real-life scenarios to ensure robustness and adaptability. For all classes, we combine any two of these classes as input to LLM to generate several differentiated sentences. Then we split each output into several sentences, which are further splited by conjunctions (e.g., "while", "whereas", "compared to", "unlike", "in contrast to", "contrast", "however", and so on) and assign each part to the corresponding class.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: Generally the reviewers agree on the contribution, e.g. from one of the reviews: This idea of enriching prototypes with knowledge from text is innovative, and the impressive performance on various datasets demonstrates its effectiveness.  The provided visualization is beneficial for explaining the proposed method.  Also the reviewers agree on the potential downside of this approach relying on LLMs and the concomitant issues and compute required.  Overall accept.

Reviews: review: paper_summary: paper_summary: This paper targets few-shot 3D point cloud semantic segmentation and proposes GPCPR to explicitly leverage the LLM-generated content and query context to enhance the prototype quality. The component GCPR integrates diverse and differentiated class descriptions generated by LLMs to enrich prototypes. The component PCPR further aggregates reliable class-specific pseudo-query context to mitigate class information bias and generate more suitable query-specific prototypes. Furthermore, a dual-distillation regularization term is also introduced to enable knowledge transfer between early-stage entities (prototypes or pseudo predictions) and their deeper counterparts to enhance refinement. Experiments are conducted on the S3DIS and ScanNet datasets.
strength_weakness: strength_weakness: Figure 1 should explain what the symbols mean, such as P, T, S, and dot.

The CVPR 2024 paper [1] found that there are some issues in the experimental setting of few-shot 3D point cloud segmentation and corrected the setting. However, this paper is still evaluated under the old setting and does not compare with [1], which makes the results less convincing. Thus, I believe that the experiments must be carried out under the new setting in the final version.

In Table 3, the PCPR module seems to only yield a marginal improvement. Please provide some explanations.

Considering that this paper uses LLMs, which classes does the model perform best and which classes does the model perform worst? Is there any connection between this finding and LLMs?
questions: questions: What is the performance of the proposed method under the setting corrected by [1]?
limitations: limitations: The experiments are carried out in the old, ill-posed setting.
paper_summary: paper_summary: This paper analyses and addresses two issues of prototype-based methods in the Few-shot Point Cloud Segmentation task. For the constrained semantic information issue, they present the GCPR module to enrich prototypes with text knowledge via LLM and CLIP. For the class information bias issue, the proposed PCPR module mitigates this bias with reliable context aggregation. Further, the dual-distillation regularization promotes the refinement of prototypes.
strength_weakness: strength_weakness: As the authors mentioned in the Limitations, LLM and CLIP will bring additional computing costs. Although the proposed offline operation can help to some extent, specific metrics like model parameters and FLOPs should be provided to make a fair comparison with the previous methods. Also, I wonder whether the authors have tried to distillate knowledge from text to prototype directly like these works[1,2], which may be more efficient because there is no need to use LLM and CLIP during inference.

[1] ULIP: Learning Unified Representation of Language, Image and Point Cloud for 3D Understanding, CVPR2023

[2] OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding, NeurIPS 2023
questions: questions: Please refer to Weaknesses.
limitations: limitations: The authors have analyzed the limitations.
paper_summary: paper_summary: This paper introduces a FS-3DSeg framework called Generated and Pseudo Content guided Prototype Refinement (GPCPR) for few-shot 3D point cloud semantic segmentation, leveraging LLM-generated content and reliable query context to enhance prototype quality. GPCPR includes two core components: Generated Content-guided Prototype Refinement (GCPR), which enriches prototypes with comprehensive semantic knowledge from large language model-generated class descriptions, and Pseudo Query Context-guided Prototype Refinement (PCPR), which mitigates class information bias by aggregating reliable class-specific pseudo-query context to create more suitable query-specific prototypes. Additionally, a dual-distillation regularization term is introduced to enable knowledge transfer between early-stage entities and their deeper counterparts, enhancing refinement. Extensive experiments demonstrate the superiority of GPCPR.
strength_weakness: strength_weakness: - The novelty is relatively limited. This paper is not the first to explore using LLM for few-shot segmentation; it has already been done in 2D few-shot segmentation [1][2]. Since LLMs inject semantic knowledge into visual features, their rich semantic knowledge should be transferable between 2D and 3D. This paper lacks specific design or optimization for 3D tasks, making the innovation of the GCPR module quite limited.
- The idea of using pseudo predictions to generate class-specific pseudo-query context in the PCPR module is quite straightforward (not novel) in few-shot segmentation. Additionally, the PCPR module simply stacks techniques from previous works. For example, QGPA directly corresponds to [3], and prototype distillation is identical to [4].
- The introduction of LLMs may significantly increase computational overhead and inference time compared to previous methods.
[1]	LLaFS: When Large Language Models Meet Few-Shot Segmentation  CVPR2024
[2]	Simple Semantic-Aided Few-Shot Learning  CVPR2024 
[3]	Prototype Adaption and Projection for Few- and Zero-shot 3D Point Cloud Semantic Segmentation  TIP 2023
[4]	Dynamic Prototype Adaptation with Distillation for Few-shot Point Cloud Segmentation   3DV 2024
questions: questions: 1. The authors propose using LLM-generated content to enrich prototypes. How sensitive is the model's performance to the quality and diversity of these generated descriptions? Have the authors explored the impact of using different LLMs or prompting strategies?
2. The paper introduces a dual-distillation regularization term. How does this compare to other knowledge distillation techniques in few-shot learning? What is the specific advantage of this approach in the context of 3D point cloud segmentation?
3. The paper uses pseudo masks to extract class-specific pseudo-query context. How robust is this approach to potential errors in the initial pseudo mask generation? Have the authors explored the impact of different thresholds or strategies for generating these masks?
4. The proposed method significantly outperforms state-of-the-art approaches on both S3DIS and ScanNet datasets. Have the authors investigated why the performance gain is notably larger on ScanNet compared to S3DIS?
5. How does the computational complexity of GPCPR compare to existing methods, especially considering the integration of LLM-generated content and multiple refinement stages?
limitations: limitations: 1. scalability: The complexity of generating differentiated descriptions grows quadratically with the number of classes (O(N^2)). This could pose challenges when applying the method to datasets with a large number of semantic classes.
2. Dependency on LLMs: The reliance on large language models for generating descriptions introduces an external dependency that may not always be available or consistent across different implementations or time periods.
3. Computational overhead: The paper lacks a detailed analysis of the additional computational cost introduced by the LLM content generation and multiple refinement stages. This is crucial for understanding the method's practical applicability.
4. Sensitivity to LLM output: The quality of the generated descriptions could vary depending on the specific LLM used and its training data. The paper doesn't thoroughly explore how variations in LLM output might affect the overall performance.
5. Pseudo mask reliability: The method relies on generating pseudo masks for query point clouds. However, the paper doesn't extensively analyze how errors in these pseudo masks propagate through the system and affect the final segmentation results.

Comments: title: reply
comment: Dear Reviewer pd6x,

Thank you very much for raising the rating to borderline accept. We sincerely appreciate your time and effort in reviewing our paper and for your constructive comments. Your insights are very valuable in helping us improve our work.

Thank you once again, and we wish you all the best.
Thank you for the response, after considering other reviews as well as the rebuttal, I decided to raise my rating to borderline accept.
Dear Reviewer FCL4, 

Thank you very much for your positive feedback and for taking the time to review our paper. We greatly appreciate your thoughtful comments. Your insights are very valuable in helping us improve our work.
The authors have addressed my concerns.

I think this is a good paper that presents an interesting idea of enriching prototypes with knowledge from the text. Although it has some slight limitations / weaknesses, it provides a good attempt and the method is effective.