Title: Generated and Pseudo Content guided Prototype Refinement for Few-shot Point Cloud Segmentation
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D point cloud segmentation; few shot learning; Large Language Model
Abstract: Few-shot 3D point cloud semantic segmentation aims to segment query point clouds with only a few annotated support point clouds. Existing prototype-based methods learn prototypes from the 3D support set to guide the segmentation of query point clouds. However, they encounter the challenge of low prototype quality due to constrained semantic information in the 3D support set and class information bias between support and query sets. To address these issues, in this paper, we propose a novel framework called Generated and Pseudo Content guided Prototype Refinement (GPCPR), which explicitly leverages LLM-generated content and reliable query context to enhance prototype quality. GPCPR achieves prototype refinement through two core components: LLM-driven Generated Content-guided Prototype Refinement (GCPR) and Pseudo Query Context-guided Prototype Refinement (PCPR). Specifically, GCPR integrates diverse and differentiated class descriptions generated by large language models to enrich prototypes with comprehensive semantic knowledge. PCPR further aggregates reliable class-specific pseudo-query context to mitigate class information bias and generate more suitable query-specific prototypes. Furthermore, we introduce a dual-distillation regularization term, enabling knowledge transfer between early-stage entities (prototypes or pseudo predictions) and their deeper counterparts to enhance refinement. Extensive experiments demonstrate the superiority of our method, surpassing the state-of-the-art methods by up to 12.10% and 13.75% mIoU on S3DIS and ScanNet, respectively.

Generated and Pseudo Content guided Prototype Refinement for Few-shot Point Cloud Segmentation

Abstract

Few-shot 3D point cloud semantic segmentation aims to segment query point clouds with only a few annotated support point clouds. Existing prototype-based methods learn prototypes from the 3D support set to guide the segmentation of query point clouds. However, they encounter the challenge of low prototype quality due to constrained semantic information in the 3D support set and class information bias between support and query sets. To address these issues, in this paper, we propose a novel framework called G enerated and P seudo C ontent guided P rototype R efinement (GPCPR), which explicitly leverages LLM-generated content and reliable query context to enhance prototype quality. GPCPR achieves prototype refinement through two core components: LLM-driven Generated Content-guided Prototype Refinement (GCPR) and Pseudo Query Context-guided Prototype Refinement (PCPR). Specifically, GCPR integrates diverse and differentiated class descriptions generated by large language models to enrich prototypes with comprehensive semantic knowledge. PCPR further aggregates reliable class-specific pseudo-query context to mitigate class information bias and generate more suitable query-specific prototypes. Furthermore, we introduce a dual-distillation regularization term, enabling knowledge transfer between early-stage entities (prototypes or pseudo predictions) and their deeper counterparts to enhance refinement. Extensive experiments demonstrate the superiority of our method, surpassing the state-ofthe-art methods by up to 12.10% and 13.75% mIoU on S3DIS and ScanNet, respectively.

1 Introduction

Point cloud semantic segmentation, aiming to assign a semantic label to each point in the 3D point clouds, benefits various applications like autonomous driving, robotics, and augmented/virtual reality. Despite the remarkable performance of supervised methods in this field [33, 34, 22, 24, 12, 37], they rely on labor-intensive labeled data and struggle to distinguish novel classes not present in the training data. To address these issues, few-shot 3D point cloud semantic segmentation (FS-3DSeg), which segments novel classes with only a few labeled data, has gained widespread attention.

FS-3DSeg aims to learn a model on base classes and generalize it to novel classes with only a few annotated point clouds. Existing FS-3DSeg methods [39, 17, 36, 9, 41, 20, 10, 16] typically adopt prototype-based paradigms [28, 32], where prototypes extracted from the support point clouds are treated as class descriptors to guide the segmentation of unlabeled query point clouds by matching

query point features with prototypes. Due to their representation and generalization capabilities, prototype-based methods have made great progress and garnered increasing attention. However, their performance remains unsatisfactory due to encountering the challenge of low prototype quality, attributed to two reasons: 1) Semantic information constraints : Limited 3D support point clouds contain only partial and incomplete object information, lacking both intra-class diversity and interclass discriminative information. Consequently, vanilla prototypes generated from the 3D support set are insufficient to be comprehensive class descriptors. 2) Class information bias : Query and support point clouds exhibit intra-class object variations [16, 20] and feature distribution gaps [9, 10], making vanilla prototypes may be ill-suited for segmenting query samples as the distances between vanilla prototypes and query features will be far away.

To address these issues, we propose to compensate for the lack of semantics in 3D support set to generate comprehensive and reliable query-specific prototypes for accurately segmenting the query point cloud. This can be achieved in two steps. Firstly, to alleviate class information constraints, motivated by the fact that texts are more controllable and accessible [35], we aim to integrate general semantic knowledge from texts to refine prototypes. Fortunately, Large Language Models ( LLM ) ( e.g. , ChatGPT) [3, 35, 29] offer promising opportunities to help achieve this goal due to their wealth of implicit knowledge and have been leveraged for a variety of downstream tasks, such as image segmentation [42] and point cloud classification [43]. Inspired by this, we leverage the capabilities of LLM to explicitly generate diverse and differentiated contents, combined with novel designs to obtain refined prototypes with richer semantics. Secondly, to reduce class information bias, previous works [9, 20, 16, 10] have endeavored to map prototypes to query feature space through prototypequery interaction using cross-attention [31]. However, noisy interactions between prototypes of one class and query features of other classes may introduce class-irrelevant cues, thereby undermining the reliability of prototypes. We observed that pseudo masks for query point clouds can assist in extracting class-specific pseudo-query context and filtering out class-irrelevant noisy features.

Ultimately, in this paper, we propose a novel FS-3DSeg framework via G enerated and P seudo C ontent guided P rototype R efinement ( GPCPR ), which progressively refines prototypes by two newly proposed components: LLM-driven Generated Content-guided Prototype Refinement (GCPR) and Pseudo Query Context-guided Prototype Refinement (PCPR). Initially, vanilla prototypes are generated from 3D support features. Firstly, our GCPR prompts the LLM to generate diverse descriptions of single classes and differentiated descriptions of class pairs. Then we design text adapters and two text-to-prototype compressors for mapping and incorporating text knowledge into vanilla prototypes, yielding refined prototype covering with richer and discriminative semantic knowledge. Secondly, leveraging the refined prototypes, PCPR generates pseudo masks for query point clouds to derive the class-specific pseudo-query context, which can be aggregated into prototypes by a newly designed query-to-prototype compressor, yielding more reliable prototypes that can better associate with query points. Beyond that, we also introduce a dual-distillation regularization term, which enables early-stage entities (prototypes or pseudo predictions) to gain insight from their deeper optimal counterparts. This facilitates bidirectional information exchange in both forward and backward processes during training, thus obtaining better prototypes and predictions.

We summarise our contributions as follows: 1) We propose GPCPR, a novel end-to-end FS-3DSeg framework that enhances prototype quality by simultaneously integrating LLM-generated content and reliable query context to generate query-specific prototypes. To the best of our knowledge, this is the first time leveraging LLM's capabilities to segment novel classes in FS-3DSeg. 2) We design a series of novel modules, including the Generated Content-guided Prototype Refinement (GCPR) module and the Pseudo Query Context-guided Prototype Refinement (PCPR) module, to facilitate the prototype refinement process. Additionally, we design a dual-distillation regularization term to further mutually enhance the refinement. 3) Extensive experiments demonstrate the superiority of our method, notably exceeding state-of-the-art methods by up to 12.10% and 13.75% on S3DIS and ScanNet datasets, respectively.

2 Related Work

2.1 3D Point Cloud Semantic Segmentation

3D point cloud semantic segmentation (3DSeg) aims to assign semantic labels to each point within point clouds. Supervised 3DSeg studies involve voxel-based methods [7, 18] and point-based methods

[22, 33, 23, 24, 12, 37], with the latter receiving increasing attention due to the simplicity, flexibility, and efficiency. For example, DGCNN [33] proposes the EdgeConv module to capture local structures. Recently, [12, 37] design self-attention-like networks to model long-range contexts from distant neighbors. Despite their effectiveness, these methods often rely on extensive annotations and cannot segment novel classes. In this paper, following mainstream FS-3DSeg methods [39, 9, 16], we utilize DGCNN as the point encoder and extend its capability to segment novel classes.

2.2 Few-shot 3D Point Cloud Segmentation

Most few-shot 3D point cloud semantic segmentation (FS-3DSeg) methods [39, 17, 9, 20, 10, 41, 16, 1] follow the prototype-based paradigms [32, 28, 11], learning prototypes from the support set to segment query point clouds. Specifically, AttMPTI [39], the pioneering FS-3DSeg method, proposes an attention-aware multi-prototype transductive inference framework based on label propagation. BFG [17] mutually embeds global perception into local features and prototypes. To reduce contextual gaps between support prototypes and query features, [20, 10, 9, 16] adapt prototypes to query feature space by prototype-query feature interaction via cross attention [31]. COSeg [1] introduces correlation optimization by refining the multi-prototypical support-query correlations. In addition to prototype learning, SCAT [36] explores class-specific relations between query and support features using transformer blocks [31] instead of pooling operations. However, these methods often yield low-quality prototypes since the 3D support set only contains constrained semantic information. Besides, their prototype-query interaction [20, 10, 9, 16] may introduce noise. In contrast, our approach refines prototypes by integrating LLM-generated content and class-specific pseudo-query context, resulting in well-suited query-specific prototypes for better segmentation.

2.3 Large Models

Recently, several large models have emerged, including large language models ( e.g. , GPT-3 [3], ChatGPT [35] and BERT [5]) and large multi-modal models ( e.g. , CLIP [25], BLIP [13], LLaVA [15], GPT-4 Vision [35] and MiniGPT [40]). These extensive models are repositories of extensive knowledge and have achieved widespread success in diverse downstream tasks, such as robot task plans [27], image classification [26] and image segmentation [42]. In the field of 3D point cloud understanding, several large models have also emerged that associate point clouds, texts and other modalities ( e.g. , Point-LLM [8], MiniGPT-3D [29], Uni3D-LLM [14] and PointCLIP V2 [43]). However, none of them are designed for segmenting novel classes in FS-3DSeg. In this work, we leverage the rich generated content of LLM to assist in addressing the FS-3DSeg problem.

3 Method

3.1 Problem Formulation and Overview

Problem Formulation. Following previous FS-3DSeg methods [39, 9], we adopt meta-learning based paradigm. Specifically, all semantic classes are divided into base class set C base and novel class set C novel for training and testing respectively, where C base ∩ C novel = ∅ . Each few-shot task ( i.e. , episode) instantiates an N -way K -shot segmentation task. The input data in each episode contains a support set S = { ( I n,k s , M n,k s ) K k =1 } N n =1 and a query set Q = { ( I i q , M i q ) } T i =1 , where N , K and T denote the number of classes, the number of support point clouds for each class, and the number of query point clouds. I n,k s and I i q denote support and query point clouds, each contains M points. M n,k s ∈ { 0 , 1 } M × 1 denotes binary support GT mask for each of the N unique classes. M i q ∈ { 0 , ..., N } M × 1 denotes the query GT mask, which is only available during training. The class name set contains one background name and N foreground class names, denoted as C = { c n } N n =0 . The goal of FS-3DSeg is to learn a model to predict segmentation mask ˆ M q for I q based on S .

Overview. Figure 1 illustrates the architecture of our proposed method. Given query set and support set Q and S , following common practice [39, 9], we utilize a shared point encoder to extract per point query and support features, represented as F q ∈ R T × M × d and F s ∈ R N × K × M × d , where d denotes feature dimension. Then we adopt masked average pooling (MAP) [9] to generate vanilla class-wise 3D support prototypes P = {P n } N n =0 ∈ R ( N +1) × d from F s , including a background prototype and N foreground prototypes. To enhance the quality of P , we propose two modules: LLM-

𝐹 𝑞

S

ℙ

Prototype Refinement

𝐹 𝑠

ሶ ℙ

ሷ ℙ

·

Pseudo

Prediction

(a) Prototype Generation

Point

Encoder

Query-to-Proto.

Compressor

Frozen

ሶ 𝕋

ሷ 𝕋

Class Text-to-Proto.

Compressor

Diff. Text-to-Proto.

Compressor

cluster

bed

desk

bed  desk

bed  cluster

desk  cluster

Class Names

Text Encoder

Text Encoder

(b) Generated Content guided

Prototype Refinement

Prototype Distillation

S

𝑳𝑪𝑬

Query GT

Prediction

Logit

Distillation

𝐹 𝑞

Prototype Distillation

Diverse Descriptions

𝐹 𝑠

Point Features

Prototypes

𝑭𝒒

𝑭𝒔

Support

Query

Caption Generation

Question Answering

Paraphrase Generation

Make Sentence

Describe the distinguishing

visual or geometry

differences between

[Class 1] and [Class 2].

LLM

LLM

Trainable

S

·

Dot Product

Cosine Similarity

driven Generated Content-guided Prototype Refinement (GCPR) and Pseudo Query Context-guided Prototype Refinement (PCPR). Firstly, GCPR prompts LLM to generate diverse class descriptions D and differentiated class descriptions D ′ , followed by a frozen text encoder and a trainable text adapter to extract text features E and E ′ . To effectively incorporate text knowledge into P , GCPR sequentially utilizes two text-to-prototype compressors to generate refined prototypes ¨ T with rich semantics. After that, PCPR generates pseudo masks ˙ M q to extract class-specific pseudo query contexts ˙ F q for Q , followed by a query-to-prototype compressor to generate well-suited query-specific prototypes ¨ P . Finally, cosine similarities between F q and ¨ P are calculated to segment query point clouds, where each point cloud is assigned the label of the most similar prototype. To supervise the model training, we adopt standard cross-entropy loss and design a dual-distillation regularization term (DD loss) to further enhance the refinement process. Subsequently, we provide detailed descriptions of GCPR, PCPR, and DD loss as below.

3.2 LLM-driven Generated Content-guided Prototype Refinement

Due to support point clouds only containing constrained semantic information, the vanilla prototypes P lack class diversity and cannot be better associated with Q . To address this, inspired by the commonsense knowledge in LLM and the fact that textual descriptions are more controllable [35], we aim to leverage LLM to generate detailed class content and propose two text-to-prototype compressors to embed these texts into prototypes P , thereby obtaining refined prototypes with richer semantics.

LLM-driven Content Generation. By leveraging the extended world knowledge of LLM ( e.g. , GPT-3 [3]), we generate two types of content at scale, i.e. , diverse class descriptions and differentiated class descriptions, respectively for enhancing the comprehensiveness and discriminability of the prototypes P .

1) Diverse Class Descriptions. To generate diverse descriptions, effective prompt templates for LLMs are essential. Inspired by the heuristic 3D-oriented commands for depth maps in [43], we designed the following 3D-specific heuristic prompt templates to ensure that the texts encompass sufficient point cloud properties:

Caption Generation: "Describe a point cloud of a [CLASS] in one sentence." Question Answering: "What is a [CLASS] point cloud like?"

Paraphrase Generation: "Generate a synonym: A

point cloud of a [CLASS]."

Make Sentence: "Make a sentence with words:

point cloud, [CLASS], obscure."

For each command, we place ' [CLASS] ' with a class name and feed it into GPT-3 to generate N div descriptions. The generated description set can be represented as: D = { D c n } N n =0 , where D c n contains 4 × N div descriptions.

2) Differentiated Class Descriptions. To enhance the model's ability to distinguish between multiple classes, inspired by [6] which generate differentiated attribute descriptions between ambiguous classes for image classification, we intend to generate 3D-specific differentiated descriptions for any pair of classes, explaining their visual and geometric differences. To this end, we designed the following template to prompt LLM to generate differentiated class descriptions:

"Describe the distinguishing visual or geometry differences between the point clouds of [CLASS 1] and [CLASS 2] in pairs of sentences. Generate as many captions as you can."

We place ' [CLASS 1] ' and ' [CLASS 2] ' with a pair of class names and trigger GPT-3 to produce several sentences, each of which is split into two parts at the conjunctions, with the first (or second) part assigned as a differentiated description for class c 1 (or c 2 ), denoted as D c 2 c 1 ′ (or D c 1 c 2 ′ ). For each class, we combine all its differentiated descriptions with other classes to contain more discriminative information, forming a differential description set D ′ = { D ′ c n } N n =0 , where D ′ c n = { D c i c n ′ } c i ∈ C \{ c n } includes N diff descriptions.

Detailed explanations and examples for diverse/differentiated descriptions are given in Appendix A.

Text-To-Prototype Compressors. Given LLM-generated content D and D ′ , we employ a frozen CLIP textual encoder [25] and a shared text adapter to extract text features E ∈ R ( N +1) × (4 × N div ) × d and E ′ ∈ R ( N +1) × N diff × d . To incorporate text knowledge into vanilla prototypes P , we propose two cross-attention-based [31] text-to-prototype compressors, i.e. , the class text-to-prototype compressor and the differentiated text-to-prototype compressor, which measure the relations between P and texts to aggregate semantic information.

1) Class text-to-prototype compressor integrates features E for diverse descriptions into vanilla prototypes P to obtain refined prototypes ˙ T = { ˙ T n } N n =0 with richer semantics, each formulated as:

where Q 1 = P n W q 1 ∈ R 1 × d ′ , K 1 = E n W k 1 ∈ R (4 × N div ) × d ′ and V 1 = E n W v 1 ∈ R (4 × N div ) × d . W q 1 , W k 1 ∈ R d × d ′ and W v 1 ∈ R d × d are learnable parameters for fully connected layers. d ′ denotes projected feature dimension.

2) Differentiated text-to-prototype compressor integrates features E ′ for differentiated descriptions into previous prototypes ˙ T to obtain refined prototypes ¨ T = { ¨ T n } N n =0 with discriminative information, each formulated as:

where Q 2 = ˙ T n W q 2 ∈ R 1 × d ′ , K 2 = E n ′ W k 2 ∈ R N diff × d ′ , and V 2 = E n ′ W v 2 ∈ R N diff × d . W q 2 , W k 2 ∈ R d × d ′ and W v 2 ∈ R d × d are learnable parameters for fully connected layers.

3.3 Pseudo Query Context-guided Prototype Refinement

Query and support point clouds often exhibit large object variations and class information bias, making vanilla support prototypes P unsuitable for segmenting query point clouds. Although our GCPR compensates for the lack of semantics P , they still lack query-specific associations. Previous approaches have endeavored to customize query-specific prototypes through prototype-query feature interaction using cross-attention [31]. Specifically, QGPNet [10] and QGPA [9] align support prototypes with query features to reduce the channel-wise feature distribution gap but lack query-specific context. QGE [20] and DPA [16] aggregate query context into support prototypes, but inevitably introduce noisy interactions between prototypes of one class and query features of other classes. This could inadvertently incorporate class-irrelevant query cues into prototypes, thereby undermining their reliability and exacerbating the challenge of distinguishing confusing classes. To address this issue, we aim to generate more reliable class-specific pseudo query context from query features and propose a transformer-based query-to-prototype compressor to customize well-suited query-specific prototypes.

Pseudo-Query Context Generation. To extract class-specific context, we need to predict pseudo masks M i q for each query point cloud I i q and extract class-specific context ˙ F i q accordingly, effectively filtering out interfering

cues from other classes. However, the channel distribution gap between prototypes ¨ T and query features F i q may lead to inaccurate pseudo masks. Thus we follow QGPA [9] to rectify current prototypes ¨ T to query feature channel distribution through channel-wise cross-attention [31], formulated as:

where Q 3 = F i q ⊤ W q 3 ∈ R d × M ′ , K 3 = F n s ⊤ W k 3 ∈ R d × M ′ , and V 3 = ¨ T n W v 3 ∈ R 1 × d . F n s ∈ R M × d denotes averaged support features, calculated by F n s = 1 / ( NK ) · ∑ N,k F n,k s if n = 0 , otherwise F n s = 1 /K · ∑ k F n,k s . W q 3 , W k 3 ∈ R M × M ′ and W v 3 ∈ R d × d are the learnable parameters for FC. Then queryspecific prototypes ˙ P = { ˙ P i } T i =1 ∈ R T × ( N +1) × d are obtained, where P i denote query-specific prototypes for each query point cloud I i q .

Then, we calculate cosine similarity between F i q and prototypes ˙ P i to predict pseudo logits ˙ L i q ∈ R M × ( N +1) and pseudo one hot masks ˙ M i q ∈ R M × ( N +1) . By masking query features with pseudo masks, we obtain class-specific pseudo query context, represented as: ˙ F q = { ˙ F n q } N n =0 , ˙ F n q ∈ R ( T × M ) × d .

Query-To-Prototype Compressor. The query-to-prototype compressor aggregates pseudo query context features into the prototypes by cross attention, represented as:

where Q 4 = ˙ P i,n W q 4 ∈ R 1 × d ′ , K 4 = ˙ F n q W k 4 ∈ R ( T × M ) × d ′ , V 4 = ˙ F n q W v 4 ∈ R ( T × M ) × d . W q 4 , W k 4 ∈ R d × d ′ and W v 4 ∈ R d × d are learnable parameters for fully connected layers. After that, we obtain the final queryspecific prototypes well-suited to segment query point clouds, represented as ¨ P = { ¨ P i } T i =1 ∈ R T × ( N +1) × d .

Finally, we calculate cosine similarity scores between F i q and ¨ P i to obtain predicted logits ˆ L i q ∈ R M × ( N +1) . Each point is then assigned the label of the most similar prototype to generate the final predicted mask ˆ M i q .

3.4 Dual-Distillation Regularization

To further enhance prototype refinement, we introduce a dual-distillation regularization term that allows earlystage entities to gain insights from their deeper counterparts, including two types of self-distillation: i.e. , prototype distillation and pseudo prediction distillation.

Prototype Distillation . In the forward process, GCPR and PCPR integrate useful semantics into early-stage prototypes to generate refined prototypes. To achieve mutually beneficial and bi-directional optimization of multi-stage prototypes, we treat early-stage prototypes ( e.g. , ˙ T or ˙ P ) as student prototypes and deep-stage prototypes ( e.g. , ¨ T or ¨ P ) as teacher prototypes, employing prototype distillation in GCPR and PCPR, defined as:

where KL refers to the Kullback-Leibler divergence. The self-distillation process facilitates effective knowledge transfer from teacher prototypes to student prototypes, enhancing the refinement process.

Pseudo Prediction Distillation . In PCPR, pseudo masks play a core role as they determine the reliability of the generated class-specific pseudo query context. To further improve the accuracy and quality of pseudo masks, we treat pseudo logits ˙ L q as students and final predicted logits ˆ L q as teachers, and employ logit distillation in PCPR, formulated as:

3.5 Objective.

During training, the proposed model is supervised by two loss functions, i.e. , a standard cross-entropy loss L SEG and the proposed dual-distillation loss (DD loss). The cross-entropy loss serves as the main optimization target, aimed at learning comprehensive and reliable query-specific prototypes. The dual-distillation loss acts as a regularization term to promote knowledge transfer between early-stage entities (prototypes or pseudo-predictions) and their deeper counterparts. Formally, the cross entropy loss is defined as:

The overall loss is a weighted combination of L SEG and DD loss with a balancing weight λ , represented as:

4 Experiments

4.1 Experiment Settings

Datasets. We verify our method on S3DIS [2] and ScanNet [4]. S3DIS contains 3D-RGB point clouds collected from 272 rooms across six indoor scenes. Its label set includes 13 classes (12 semantic classes and background clutter). ScanNet collects 1,513 point clouds from 707 indoor scenes. Its label set includes 21 classes (unannotated background and 20 semantic classes). Due to the inconvenience of processing the original room which contains an excessive number of points, we follow the pre-processing strategy in [39, 9, 16] to divide each room into 1 m × 1 m blocks and randomly sample M = 2048 points from each block, yielding 7,547 blocks for S3DIS and 36,350 blocks for ScanNet, respectively. Each point contains a 9D vector, including XYZ, RGB, and normalized coordinates. Following [39], semantic classes are evenly split into two non-overlapping subsets, denoted as S 0 and S 1 . For meta-learning, we train our model on one fold ( e.g. , S 0 ) and test on another fold ( e.g. , S 1 ). Vice versa for cross-validation.

Evaluation Metrics. Following conventions in the 3DSeg community, we average the mean Intersection-overUnion (mIoU) across all test classes and report the final mIoU metric under the few-shot setting.

Implementation details. In Appendix B, we provide framework details and training details.

4.2 Comparison With State-of-the-Art Methods

Results on S3DIS. Table 1 shows the experimental results of our method compared with state-of-the-art methods on the S3IDS dataset. Our method demonstrates a substantial improvement over the baseline QGPA [9], with a large margin of 6.22% ∼ 14.59% across various splitting settings, highlighting a significant improvement for FS-3DSeg. Additionally, our method consistently exceeds all of the previous methods, particularly surpassing state-of-the-art methods DPA [16] by a margin of 1.19% ∼ 12.10% across various splitting settings, showcasing our effectiveness and superiority. The success of our approach is attributed to the integration of comprehensive LLM-generated content and reliable pseudo-query context by GCPR and PCPR, solving the problem of semantic information constraints and class information bias, as well as enhancing refinement by dual-distillation loss.

S0

clutter

beam

board

bookcase

ceiling

chair

column

S0

usansotated

bathtub

bed

cabinet

chair

counter

curtain

desk

door

floor

Query

Point

Cloud

GT

Mask

Ours

(a)S3DIS

(b)ScanNet

Results on ScanNet. Table 2 presents the segmentation performance on ScanNet. Our method significantly improves the baseline by a larger margin of 11.01% ∼ 18.86%. Notably, we significantly outperform SOTA method DPA [16] by large margins of 3.96% ∼ 13.75% across various settings. Our model exhibits even greater improvement on ScanNet compared to S3DIS due to containing more classes and complex scenarios in ScanNet, which greatly challenges prototype quality. Our approach effectively integrates more comprehensive and reliable content to cope with the lack of semantic information, resulting in superior performance.

Computational Complexity. In our approach, generating descriptions with LLM and extracting text features through CLIP for all classes in the dataset are performed offline before the training and testing phases. Once text features are stored through offline operations, we can directly load the stored text features without regenerating by LLM online, avoiding bringing redundant computing costs during training and testing. (1) Offline cost: Table 3 shows the time cost of extracting text features for all classes in the entire dataset. We analyze that the total offline time is primarily determined by the description generation process, with feature extraction time being negligible. The time for generating descriptions from LLM varies on different datasets, depending on the number of classes. (2) Online cost: Table 4 shows that our method achieves a better balance between computational cost and experiment performance, offering superior segmentation results with reasonable computational efficiency.

Qualitative results. In Figure 2, we visualize the segmentation results of the 2-way 1-shot segmentation task on S3DIS and ScanNet. We compare our predictions with the GT mask and QGPA. As shown, QGPA often produces inaccurate segmentation, particularly in distinguishing background from foreground classes and among different foreground classes. In contrast, our method achieves accurate segmentation for most areas of the target scene. For instance, in the second column of S3DIS examples, QGPA mistakenly predicts parts of the beam and bookcase, whereas our method effectively distinguishes between these classes. Similar results are observed in the last column of ScanNet examples. The superiority of our method is because we incorporate LLM-generated content and pseudo-query context to obtain query-specific prototypes, resulting in better segmentation results.

acc=45.97

acc= 96.22

(b) Ours

acc= 72.14

acc= 97.78

(d) Ours

Query feat.

Support feat.

Background

ceiling

chair

beam

column

bookcase

board

clutter

𝑺 𝟎

Vanilla proto.

ℙ

GCPR proto.

ሷ 𝕋

PCPR proto.

ሷ ℙ

Div. text feat.

Diff. text feat.

4.3 Ablation Study

We perform an ablation study on the S3DIS dataset under the 2-way 1-shot setting to validate the effectiveness of our proposed GPCPR. We use QGPA [9] as the baseline to analyze the effects of the proposed components and hyper-parameters.

Effects of Different Components. We conduct experiments to investigate the effect of the three core components, i.e. , GCPR, PCPR and DD loss.

1) Effects of PCPR: Without introducing any text knowledge, PCPR improves the baseline by 8.68%, showing that PCPR extracts reliable query context, avoids noise interference, and effectively reduces class information bias.

2) Effects of GCPR: Only using diverse descriptions D or differentiated descriptions D ′ , we exceed the baseline by 6.53% or 7.43%, with the latter playing a greater role because they directly improve the inter-class discrimination of the prototypes. After jointly using D and D ′ , the

improvement over the baseline reaches 10.62%, showing that GCPR effectively makes up for the lack of semantics of the prototype. Comparing PCPR and GCPR, the latter can achieve greater improvement because LLM provides sufficient semantic details. Finally, combining PCPR and QCPR, our method improves the baseline by 11.19%.

3) Effects of DD loss: Compared to only using PCPR and QCPR, after further applying L TP , L QP and L QM respectively, the model performance further improved 0.40%, 1.38% and 1.56%, with the last one being more effective due to directly optimizing pseudo masks to obtain a more reliable class-specific pseudo query context. By jointly using all distillation losses, we achieve the best performance of 75.74%.

Visualization of features and prototypes. As shown in Figure 4, we use the t-SNE visualization tool [30] to compare the feature distribution and the class prototype distribution to analyze the effect of the model. Here, prediction accuracy is also provided for a clearer comparison. As marked by the red dotted circles, for QGPA, the distance between prototypes and most query features is far. This indicates there exists bias between prototypes and query features due to the lack of query context in the prototypes. In contrast, our method significantly improves accuracy and pushes the prototype closer to query features, as marked by the green dotted circles. This improvement is because we effectively inject rich text knowledge and reliable class-specific query context into the refined prototypes, enhancing their representation capabilities and mitigating object variations.

Effects of different adaptors. As shown in Figure 4(a), without introducing any additional text information by removing the GCPR module, we evaluate the effectiveness of prototype refinement and compare our method with the previous prototype adaptors, including QGPA [9] and DPA [16]. Our method significantly outperforms QGPA and DPA. This is because although QGPA alleviates the channel-wise feature distribution gap between the prototype and query features, the prototype lacks query context information and is therefore less effective. Compared with QGPA, DPA injects query context information into the prototypes. However, query context from

(a) Different adaptors

(b) Different types of texts

(c) Text number

N

div

(d) Coefficient

λ

80

80

70

75

76

76

mloU(%)

mloU(%)

70

mloU(%)

74

60

65

74

)no

E

72

60

50

72

55

70

DPA

Ours

W2V

CLIP

Ours

S0

S1

mean

S0

S'

mean

40

S0

S1

mean

50

So

S1

mean

OL

5

10

20

50

0.0

0.5

1.0

1.5

2.0

other classes acts as noise interference, reducing the reliability of the prototype and limiting performance. Our method introduces category-specific pseudo-context information into the prototype, effectively reducing noise interference while bridging object gaps and thus achieving the highest performance.

Effects of different types of descriptions. As shown in Figure 4(b), we also studied the impact of different text descriptions, by replacing the proposed LLM-generated descriptions with word2vec (W2V) [19] or CLIP texts [25, 38]. As shown, word2vec exhibits limited improvement due to only providing static and restricted semantic information. CLIP texts slightly better improve the baseline, benefiting from the cross-modal alignment capability and rich semantic information learned from large-scale datasets. Our LLM-generated descriptions achieve the best performance because LLM's powerful detailed description ability can inject sufficient diverse and differentiated semantic information into the prototypes.

Effects of the number of diverse descriptions. Figure 4(c) illustrates the impact of the number N div of diverse descriptions. Compared with not using any diverse descriptions, the model performance gradually improves as N div increases, because general knowledge is gradually injected into the prototypes to make up for the lack of semantic information, and reaches promising results when N div = 10 . However, further increasing of N div does not yield higher performance as the LLM-generated descriptions begin to repeat, providing redundant semantic knowledge. In summary, we select N div = 10 to ensure both the effectiveness and efficiency of the model.

Effects of coefficient λ for DD Loss. Figure 4(d) illustrates the impact of the weight λ of the DD loss in Eq. 8. Compared to λ = 0 (where the DD loss is deprecated), as λ increases, the prototypes and predictions in the final stage begin to gradually optimize the corresponding items in the early stages, allowing the model to generate high-quality prototypes, resulting in significant performance improvements, and reaching the best performance when λ = 1 . However, a larger λ ( e.g. , 2) does not yield higher performance and may even harm model performance. In summary, an appropriate value of λ = 1 yields the best results.

Effects of different LLMs. As shown in Table 6, using both 'gpt-3.5-turbo" [3] and 'gpt-4o-mini" [21] can achieve superior results compared with SOTA, with 75.74% mIoU and 73.88% mIoU on S3DIS, respectively. Specifically, Ours (gpt-3.5-turbo) performs best. Ours (gpt-4o-mini) performs slightly lower than Ours (gpt-3.5-turbo), because 'gpt-4omini" tends to generate longer paragraphs, which potentially contain content that is not directly related to the class semantics, thus introducing noise to the class prototypes. Thus in this paper, we choose to utilize 'gpt-3.5-turbo".

5 Conclusion

In this paper, to solve the problem of constrained semantic information and class information bias in FS-3DSeg, we propose GPCPR that LLM-generated content and reliable query context to enhance prototype quality. GPCPR consists of two components: LLM-driven Generated Content-guided Prototype Refinement (GCPR), which enriches prototypes with diverse and differentiated text knowledge from LLMs, and Pseudo Query Contextguided Prototype Refinement (PCPR), which aggregates reliable class-specific pseudo-query context to mitigate class information bias. Additionally, a dual-distillation regularization term is proposed to facilitate knowledge transfer between early-stage and deeper entities (prototypes or predictions). Experiments show that GPCPR outperforms state-of-the-art methods by up to 12.10% and 13.75% mIoU on S3DIS and ScanNet, respectively. In Appendix C, we discuss the limitations of our work and future work.

References

Z. An, G. Sun, Y. Liu, F. Liu, Z. Wu, D. Wang, L. Van Gool, and S. Belongie. Rethinking few-shot 3d point cloud semantic segmentation. In CVPR , pages 3996-4006, 2024.

I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese. 3d semantic parsing of large-scale indoor spaces. In CVPR , pages 1534-1543, 2016.

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In NeurIPS , 2020.

A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR , pages 5828-5839, 2017.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

R. Esfandiarpoor and S. H. Bach. Follow-up differential descriptions: Language models resolve ambiguities for image classification. arXiv preprint arXiv:2311.07593 , 2023.

B. Graham, M. Engelcke, and L. Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In CVPR , pages 9224-9232, 2018.

Z. Guo, R. Zhang, X. Zhu, Y. Tang, X. Ma, J. Han, K. Chen, P. Gao, X. Li, H. Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.00615 , 2023.

S. He, X. Jiang, W. Jiang, and H. Ding. Prototype adaption and projection for few-and zero-shot 3d point cloud semantic segmentation. IEEE TIP , 32:3199-3211, 2023.

D. Hu, S. Chen, H. Yang, and G. Wang. Query-guided support prototypes for few-shot 3d indoor segmentation. IEEE TCSVT , 34(6):4202-4213, 2023.

X. Huang and S. H. Choi. Sapenet: self-attention based prototype enhancement network for few-shot learning. Pattern Recognition , 135(C):109170, 2023.

X. Lai, J. Liu, L. Jiang, L. Wang, H. Zhao, S. Liu, X. Qi, and J. Jia. Stratified transformer for 3d point cloud segmentation. In CVPR , pages 8500-8509, 2022.

J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified visionlanguage understanding and generation. In ICML , pages 12888-12900, 2022.

D. Liu, X. Huang, Y. Hou, Z. Wang, Z. Yin, Y. Gong, P. Gao, and W. Ouyang. Uni3d-llm: Unifying point cloud perception, generation and editing with large language models. arXiv preprint arXiv:2402.03327 , 2024.

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In NeurIPS , 2023.

J. Liu, W. Yin, H. Wang, Y. CHen, J.-J. Sonke, and E. Gavves. Dynamic prototype adaptation with distillation for few-shot point cloud segmentation. In 3DV , pages 810-819, 2024.

Y. Mao, Z. Guo, L. Xiaonan, Z. Yuan, and H. Guo. Bidirectional feature globalization for few-shot semantic segmentation of 3d point cloud scenes. In 3DV , pages 505-514, 2022.

H.-Y. Meng, L. Gao, Y.-K. Lai, and D. Manocha. Vv-net: Voxel vae net with group convolutions for point cloud segmentation. In ICCV , pages 8500-8508, 2019.

T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013.

Z. Ning, Z. Tian, G. Lu, and W. Pei. Boosting few-shot 3d point cloud segmentation via query-guided enhancement. In ACM MM , pages 1895-1904, 2023.

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR , pages 652-660, 2017.

C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NIPS , page 5105-5114, 2017.

G. Qian, Y. Li, H. Peng, J. Mai, H. Hammoud, M. Elhoseiny, and B. Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In NeurIPS , volume 35, pages 23192-23204, 2022.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763, 2021.

Z. Ren, Y. Su, and X. Liu. Chatgpt-powered hierarchical comparisons for image classification. In NeurIPS , volume 36, pages 69706-69718, 2023.

I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In ICRA , pages 1152311530, 2023.

J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In NIPS , volume 30, page 4080-4090, 2017.

Y. Tang, X. Han, X. Li, Q. Yu, Y. Hao, L. Hu, and M. Chen. Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors. arXiv preprint arXiv:2405.01413 , 2024.

L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS , volume 30, page 6000-6010, 2017.

O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. In NIPS , volume 29, page 3637-3645, 2016.

Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph cnn for learning on point clouds. ACM TOG , 38(5):1-12, 2019.

M. Xu, R. Ding, H. Zhao, and X. Qi. Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds. In CVPR , pages 3173-3182, 2021.

Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421 , 9(1):1, 2023.

C. Zhang, Z. Wu, X. Wu, Z. Zhao, and S. Wang. Few-shot 3d point cloud semantic segmentation via stratified class-specific attention based transformer network. In AAAI , pages 3410-3417, 2023.

N. Zhang, Z. Pan, T. H. Li, W. Gao, and G. Li. Improving graph representation for point cloud segmentation via attentive filtering. In CVPR , pages 1244-1254, 2023.

R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. Qiao, P. Gao, and H. Li. Pointclip: Point cloud understanding by clip. In CVPR , pages 8552-8562, 2022.

N. Zhao, T.-S. Chua, and G. H. Lee. Few-shot 3d point cloud semantic segmentation. In CVPR , pages 8873-8882, 2021.

D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023.

G. Zhu, Y. Zhou, R. Yao, and H. Zhu. Cross-class bias rectification for point cloud few-shot segmentation. IEEE TMM , 25:9175-9188, 2023.

L. Zhu, T. Chen, D. Ji, J. Ye, and J. Liu. Llafs: When large-language models meet few-shot segmentation. arXiv preprint arXiv:2311.16926 , 2023.

X. Zhu, R. Zhang, B. He, Z. Guo, Z. Zeng, Z. Qin, S. Zhang, and P. Gao. Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. In ICCV , pages 2639-2650, 2023.

A Examples of LLM-generated Content

LLM-generated descriptions and text features for all classes are generated and stored offline. During training and testing, we directly load the stored text features, avoiding online regenerating by LLM. To more intuitively understand the diverse and differentiated descriptions generated by the LLM, we provide examples of the generated descriptions for the N -way K -shot segmentation task in Figure 5 and Figure 6, respectively.

For a single class, the descriptions generated by LLM capture rich information such as shape and geometric structure, location, detailed features, usage scenarios, and placement styles. These descriptions provide a comprehensive understanding of the class, enhancing the prototypes with detailed and varied semantic knowledge. For each class, we prompt the LLM to generate N div descriptions for each command. For each N -way K -shot segmentation task, we obtain ( N +1) × (4 × N div ) descriptions, where the number N div will affect the model performance, as too few or too many descriptions will lead to insufficient or redundant semantic information. Therefore, we analyze the effect of D div in Figure 4(c), determining that the optimal value is 10.

For class pairs, differential descriptions generated by the LLM effectively distinguish class pairs by highlighting structural differences, density differences, surface differences, and edge differences, beneficial for distinguishing similar classes. In this paper, using our proposed prompting template, the LLM tends to generate varying numbers of sentences for each class pair based on their distinguishability. We then split each sentence by conjunctions and assign each part to the relevant class. Consequently, after combining all differential sentences for each class in relation to all other classes, the number of differential descriptions N diff varies for each class. For each N -way K -shot segmentation task, we load descriptions based on the minimum N diff among the ( N +1) classes. For classes with more descriptions, we randomly load N diff descriptions, ensuring consistent input dimensions. Thus we obtain ( N +1) × N diff descriptions per task.

Intuitively, differential descriptions are more conducive to improving the discrimination of prototypes, as verified by ablation experiments in Table 5. Besides, the joint usage of both kinds of descriptions leads to the best performance in FS-3DSeg.

B Implementation Details

Framework details. We use QGPA [9] as the baseline, optimized by segmentation loss, self-reconstruction loss, and alignment loss. DGCNN [33] (with SAN) and pre-trained CLIP (rn50) [25] are used as point encoder and text encoder, respectively. The text adaptors adopt a Linear+LeakyReLU+Dropout+Linear architecture. In our GCPR module, the number N D of descriptions per command is set to 10. We use 'gpt-3.5-turbo' GPT-3.5 engine [3] and set the temperature constant to 0.99. For diverse descriptions, the largest length of a 3D-specific diverse description is set to 40, and N div is set to 10. For differential descriptions, LLM generates between 1 to 20 sentences for different class pairs, causing N diff to vary across classes, typically ranging from 14 to 34. For all compressors based on multi-head attention, the number of heads is set to 4. In Eq. 8, the coefficient λ for distillation loss in Eq. 8 is set to 1.

Training details. Our model is implemented by PyTorch and trained on single NVIDIA RTX A4000 GPU. Before meta-learning, following [39], we pre-train the point encoder on the base classes for 100 epochs. Then, during meta-training, we train our model on 40,000 episodes randomly sampled from base classes, using an Adam optimizer, with a learning rate of 0.001, and a decay step of 5000. During meta-testing, we perform model evaluation on 100 randomly sampled episodes for each feasible class combination from unseen classes. For the query set, T is set to 1 for each class.

C Discussion

Limitations. Although our GPCPR achieves remarkable results, GPCPR has the following limitations. (1) Integrating LLM and CLIP into the FS-3DSeg framework will increase resource consumption and computing costs, depending on the parameter amount and execution speed of these large models. In order to alleviate this limitation, we separated some operations in GCPR into offline operations, such as prompting LLM to generate description content and using CLIP to extract text features. (2) The complexity for LLM to generate differentiated descriptions is O ( N 2 ) , which may face challenges when extending to other domain datasets with massive semantic classes due to the need to exhaustively enumerate all possible class pairs.

Future work. The performance of the proposed method is affected by the quality of descriptions generated by LLM and the accuracy of pseudo masks. Biased class descriptions and inaccurate pseudo masks may introduce noise, negatively impacting model performance. Future work will mainly focus on improving the quality and reliability of LLM-generated content and pseudo masks. Furthermore, in this paper, we mainly address segmenting unseen categories in the same scene, but its generalization to different domains with unseen classes remains to be tested. Future research may involve extensive validation across various datasets and real-life scenarios to ensure robustness and adaptability.

Describe a point cloud of a [CLASS] in one sentence:

The point cloud of a background clutter consists of a scattered collection of 3D

points that represent various objects or debris present in the background environment.

(a)

A point cloud of a bed is a 3d representation consisting of numerous points, each

Caption

representing a specific coordinate in space, forming the shape and structure of a bed.

Generation

A point cloud of a desk is a three-dimensional representation of scattered data points

that visually captures the shape, position, and spatial details of the desk surface.

一一

一

What is a [CLASS] point cloud like?

A background clutter point cloud is a representation of the surrounding environment

that includes irrelevant or undesired objects or structures.

(b)

A bed point cloud is a 3D representation of a bed's surface, consisting of a vast

Question

collection of points with each point capturing the X, Y, and Z coordinates of the bed.

Answering

A desk point cloud is a digital representation of a desk's surface and objects,

consisting of three-dimensional points with associated coordinates and attributes.

Generate a synonym: A point cloud of a [CLASS].

A dense cluster of a backdrop mess.

(c)

Paraphrase

A bed 's three-dimensional map.

Generation

A desk's volumetric data.

Make a sentence with words: point cloud, [CLASS], obscure.

The point cloud analysis became challenging due to the background clutter, which

obscured the relevant data.

(d)

Make

The point cloud data helped to obscure the exact shape of the bed in the virtual

reality simulation.

Sentence

The point cloud data helped to accurately recreate the shape and dimensions of the

desk, even in the presence of obscure details.

Describe the distinguishing visual or geometry differences between the point clouds of bed and desk in pairs of sentences. Generate as many captions as you can.

The point cloud of bed typically has a larger surface area with more irregular shapes and angles, while the point cloud of desk tends to have more defined and straight edges with a smaller footprint. n

Beds typically have a point cloud that is lower to the ground and spread out horizontally, while desks have a point cloud that is taller and more vertical in nature. n

(More captions omitted here)

Describe the distinguishing visual or geometry differences between the point clouds of bed and background clutter in pairs of sentences. Generate as many captions as you can.

The bed's point cloud exhibits a more organized and structured arrangement of points, while the background clutter appears more haphazard and disorganized.

(More captions omitted here)

Describe the distinguishing visual or geometry differences between the point clouds of desk and background clutter in pairs of sentences. Generate as many captions as you can.

The geometry of the desk point cloud is characterized by its straight lines and right angles, while the background clutter point cloud is characterized by its irregular and unpredictable shapes. In

The desk point cloud is composed of precise measurements and uniform spacing between points, whereas the background clutter appears fragmented and disorganized. n

(More captions omitted here)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction in Section 1 clearly reflect the contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Appendix C, we discuss the limitations of our work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our work does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We propose a specific model for FS-3DSeg. In Section 3, we give a detailed description of the overview architecture and all components. In Appendix B, we provide settings of framework details and training details. We believe this model is fully replicable. Our code will be made publicly available once it is fully organized.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Our code will be made publicly available once it is fully organized.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Settings of data splits are given in Section 4.1. More details of implementation details (including framework details and training details) are given in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Following common practice among mainstream methods in the FS-3DSeg field, we do not report error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide information of computer resources of our model in Appendix B. Besides, we also analyze the memory and time cost in Table 3 and Table 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: It's no societal impact of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have done it as described in the question.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release a new dataset, our code will be made publicly available once it is fully organized.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a FS-3DSeg framework called Generated and Pseudo Content guided Prototype Refinement (GPCPR) for few-shot 3D point cloud semantic segmentation, leveraging LLM-generated content and reliable query context to enhance prototype quality. GPCPR includes two core components: Generated Content-guided Prototype Refinement (GCPR), which enriches prototypes with comprehensive semantic knowledge from large language model-generated class descriptions, and Pseudo Query Context-guided Prototype Refinement (PCPR), which mitigates class information bias by aggregating reliable class-specific pseudo-query context to create more suitable query-specific prototypes. Additionally, a dual-distillation regularization term is introduced to enable knowledge transfer between early-stage entities and their deeper counterparts, enhancing refinement. Extensive experiments demonstrate the superiority of GPCPR.\n\nSTRENGTHS:\n- The proposed method is technically sound.\n- The experiments confirm the effectiveness of the proposed method.\n- The paper is well-written\n\nWEAKNESSES:\n- The novelty is relatively limited. This paper is not the first to explore using LLM for few-shot segmentation; it has already been done in 2D few-shot segmentation [1][2]. Since LLMs inject semantic knowledge into visual features, their rich semantic knowledge should be transferable between 2D and 3D. This paper lacks specific design or optimization for 3D tasks, making the innovation of the GCPR module quite limited.\n- The idea of using pseudo predictions to generate class-specific pseudo-query context in the PCPR module is quite straightforward (not novel) in few-shot segmentation. Additionally, the PCPR module simply stacks techniques from previous works. For example, QGPA directly corresponds to [3], and prototype distillation is identical to [4].\n- The introduction of LLMs may significantly increase computational overhead and inference time compared to previous methods.\n[1]\tLLaFS: When Large Language Models Meet Few-Shot Segmentation  CVPR2024\n[2]\tSimple Semantic-Aided Few-Shot Learning  CVPR2024 \n[3]\tPrototype Adaption and Projection for Few- and Zero-shot 3D Point Cloud Semantic Segmentation  TIP 2023\n[4]\tDynamic Prototype Adaptation with Distillation for Few-shot Point Cloud Segmentation   3DV 2024\n\nQUESTIONS:\n1. The authors propose using LLM-generated content to enrich prototypes. How sensitive is the model's performance to the quality and diversity of these generated descriptions? Have the authors explored the impact of using different LLMs or prompting strategies?\n2. The paper introduces a dual-distillation regularization term. How does this compare to other knowledge distillation techniques in few-shot learning? What is the specific advantage of this approach in the context of 3D point cloud segmentation?\n3. The paper uses pseudo masks to extract class-specific pseudo-query context. How robust is this approach to potential errors in the initial pseudo mask generation? Have the authors explored the impact of different thresholds or strategies for generating these masks?\n4. The proposed method significantly outperforms state-of-the-art approaches on both S3DIS and ScanNet datasets. Have the authors investigated why the performance gain is notably larger on ScanNet compared to S3DIS?\n5. How does the computational complexity of GPCPR compare to existing methods, especially considering the integration of LLM-generated content and multiple refinement stages?\n\nLIMITATIONS:\n1. scalability: The complexity of generating differentiated descriptions grows quadratically with the number of classes (O(N^2)). This could pose challenges when applying the method to datasets with a large number of semantic classes.\n2. Dependency on LLMs: The reliance on large language models for generating descriptions introduces an external dependency that may not always be available or consistent across different implementations or time periods.\n3. Computational overhead: The paper lacks a detailed analysis of the additional computational cost introduced by the LLM content generation and multiple refinement stages. This is crucial for understanding the method's practical applicability.\n4. Sensitivity to LLM output: The quality of the generated descriptions could vary depending on the specific LLM used and its training data. The paper doesn't thoroughly explore how variations in LLM output might affect the overall performance.\n5. Pseudo mask reliability: The method relies on generating pseudo masks for query point clouds. However, the paper doesn't extensively analyze how errors in these pseudo masks propagate through the system and affect the final segmentation results.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper analyses and addresses two issues of prototype-based methods in the Few-shot Point Cloud Segmentation task. For the constrained semantic information issue, they present the GCPR module to enrich prototypes with text knowledge via LLM and CLIP. For the class information bias issue, the proposed PCPR module mitigates this bias with reliable context aggregation. Further, the dual-distillation regularization promotes the refinement of prototypes.

STRENGTHS:
This idea of enriching prototypes with knowledge from text is innovative, and the impressive performance on various datasets demonstrates its effectiveness.

The provided visualization is beneficial for explaining the proposed method.

WEAKNESSES:
As the authors mentioned in the Limitations, LLM and CLIP will bring additional computing costs. Although the proposed offline operation can help to some extent, specific metrics like model parameters and FLOPs should be provided to make a fair comparison with the previous methods. Also, I wonder whether the authors have tried to distillate knowledge from text to prototype directly like these works[1,2], which may be more efficient because there is no need to use LLM and CLIP during inference.

[1] ULIP: Learning Unified Representation of Language, Image and Point Cloud for 3D Understanding, CVPR2023

[2] OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding, NeurIPS 2023

QUESTIONS:
Please refer to Weaknesses.

LIMITATIONS:
The authors have analyzed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper targets few-shot 3D point cloud semantic segmentation and proposes GPCPR to explicitly leverage the LLM-generated content and query context to enhance the prototype quality. The component GCPR integrates diverse and differentiated class descriptions generated by LLMs to enrich prototypes. The component PCPR further aggregates reliable class-specific pseudo-query context to mitigate class information bias and generate more suitable query-specific prototypes. Furthermore, a dual-distillation regularization term is also introduced to enable knowledge transfer between early-stage entities (prototypes or pseudo predictions) and their deeper counterparts to enhance refinement. Experiments are conducted on the S3DIS and ScanNet datasets.\n\nSTRENGTHS:\nThis paper is easy to read. This paper targets an interesting problem, few-shot 3D point cloud semantic segmentation. The proposed method is also interesting.\n\nWEAKNESSES:\nFigure 1 should explain what the symbols mean, such as P, T, S, and dot.\n\nThe CVPR 2024 paper [1] found that there are some issues in the experimental setting of few-shot 3D point cloud segmentation and corrected the setting. However, this paper is still evaluated under the old setting and does not compare with [1], which makes the results less convincing. Thus, I believe that the experiments must be carried out under the new setting in the final version.\n\nIn Table 3, the PCPR module seems to only yield a marginal improvement. Please provide some explanations.\n\nConsidering that this paper uses LLMs, which classes does the model perform best and which classes does the model perform worst? Is there any connection between this finding and LLMs?\n\nQUESTIONS:\nWhat is the performance of the proposed method under the setting corrected by [1]?\n\nLIMITATIONS:\nThe experiments are carried out in the old, ill-posed setting.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}