Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation

Authors: Zhaochong An, Guolei Sun, Yun Liu, Runjia Li, Min Wu, Ming-Ming Cheng, Ender Konukoglu, Serge Belongie
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: 3d segmentation, 3d point cloud, few-shot segmentation, multimodality, few-shot point cloud semantic segmentation

Abstract:
Few-shot 3D point cloud segmentation (FS-PCS) aims at generalizing models to segment novel categories with minimal annotated support samples. While existing FS-PCS methods have shown promise, they primarily focus on unimodal point cloud inputs, overlooking the potential benefits of leveraging multimodal information. In this paper, we address this gap by introducing a multimodal FS-PCS setup, utilizing textual labels and the potentially available 2D image modality. Under this easy-to-achieve setup, we present the MultiModal Few-Shot SegNet (MM-FSS), a model effectively harnessing complementary information from multiple modalities. MM-FSS employs a shared backbone with two heads to extract intermodal and unimodal visual features, and a pretrained text encoder to generate text embeddings. To fully exploit the multimodal information, we propose a Multimodal Correlation Fusion (MCF) module to generate multimodal correlations, and a Multimodal Semantic Fusion (MSF) module to refine the correlations using text-aware semantic guidance. Additionally, we propose a simple yet effective Test-time Adaptive Cross-modal Calibration (TACC) technique to mitigate training bias, further improving generalization. Experimental results on S3DIS and ScanNet datasets demonstrate significant performance improvements achieved by our method. The efficacy of our approach indicates the benefits of leveraging commonly-ignored free modalities for FS-PCS, providing valuable insights for future research. The code is available at github.com/ZhaochongAn/Multimodality-3D-Few-Shot.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730370537609

thread_type:
official_review

<thread_object>
creator: reviewer_qopy

<content>
summary:
In the paper, the MultiModal Few-Shot SegNet was proposed, which segments novel categories with minimal annotated support samples utilizing textual labels and the potentially available 2D image modality.  The main contributions are as follows: 1) To incorporate the textual modality of category names and the 2D image modality in few-shot 3D learning scenarios without additional annotation cost; 2) A new model, MM-FSS, was introduced, and information from different modalities was utilized. Experiments validated the value of the proposed method and the efficacy of the proposed method across different few-shot settings.

strengths:
Based on the learning method, a large number of annotations are required for the identification and segmentation of new target categories in the point cloud, which is very time-consuming and laborious. The paper is of great significance to reduce the data annotation process through the combination of text and images.The MM-FSS model exhibits superior generalization capabilities, especially in few-shot settings, providing valuable insights for future research directions in the field.

weaknesses:
1. The paper's description of the 2D image modality's integration into the multi-modal FS-PCS setup could be more detailed, providing clearer insights into how the 2D images are leveraged to enhance the model's performance without additional annotation costs.
2. The article does not offer a detailed analysis of the performance enhancements specifically attributable to the multi-modal information fusion during the modeling process, making it challenging to assess the individual impact of textual and 2D image modalities on few-shot segmentation performance.
3. The existing FS-PCS datasets are generally on a small scale, the practicability and effectiveness of the method in this paper need to be further verified in the future.

questions:
1. In Line 110，ii should be iii.
2. Whether there is a comparison of the computational complexity and computational efficiency of different algorithms？
3. The significance of the important regions depicted in Figure 3 is unclear. The authors should provide additional details to enhance reader understanding.
4. The authors input multiple images into the VLM; how do they determine the viewpoints for the RGB images fed into the VLM?
5. The authors demonstrate a notable performance increase by introducing textual information into the query. I am curious if there is potential to also incorporate textual or other types of information into the support set to achieve further gains.
6. The selection of support appears somewhat random. If the authors could provide some theoretical guidance in the appendix regarding this selection process, I believe it would greatly enhance the significance of this work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730471410497

thread_type:
official_review

<thread_object>
creator: reviewer_k1ne

<content>
summary:
This work proposes a multimodal few-shot segnet (MM-FSS) to leverage multimodal information from textual labels and 2D images. Specifically, they propose multimodal correlation fusion (MCF) to fuse the multimodal correlations and enhance the fused features by multimodal semantic fusion (MSF) with textual information. Besides, the authors proposed test-time adaptive cross-modal calibration (TACC) to calibrate the predictions during test time. Experimental results on both S3DIS and ScanNet show the superior performance of the proposed method.

strengths:
1. It makes sense to use complementary information from multimodal inputs for point cloud segmentation.
2. The ablation studies on ScanNet prove the effectiveness of the proposed models, including MCF, MSF, and TACC.
3. The features from 2D images are learned during the pre-training step. In this way, the proposed method does not rely on 2D inputs during inference.
4. The proposed MM-FSS achieves promising performance on two benchmark datasets compared to the state-of-the-art methods.

weaknesses:
1. The idea of using multimodal information for 3D point cloud segmentation is not new. As noticed by the authors, such an idea has been explored for fully supervised or more challenging tasks like open-vocabulary segmentation. It is unclear why the prior few-shot methods only use unimodal information. It would be better to provide more discussions on the difficulties or challenges when using multimodal features for few-shot segmentation.

2. The claim about “cost-free” is ambiguous as “cost” may mean many things, such as training cost. I don't think it is necessary to claim the "cost-free".

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730562480850

thread_type:
official_review

<thread_object>
creator: reviewer_8jox

<content>
summary:
This paper studies the Few-Shot 3D Point Cloud Semantic Segmentation task by using multimodality information. Specifically, it uses 2D image modality to  extract additional intermodal features apart from the unimodal visual features. It also uses text embedding to refine the multimodal correlations generated by the proposed Multimodal Correlation Fusion module. Moreover, it presents the Test-time Adaptive Cross-modal Calibration technique to improve the generalization capability. The experimental results show that the proposed methods can achieve higher performance than previous works.

strengths:
- The writing is good, making this paper easy to follow.

- The performance is good.

- The experiments are adequate.

- The introduction of the multimodality information is useful.

weaknesses:
- How the design of the MSF module can achieve the improvement for the correlation. Please give more explanations.

questions:
Please answer the question in weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730573356656

thread_type:
official_review

<thread_object>
creator: reviewer_yysz

<content>
summary:
This paper investigates the few-shot 3D point cloud semantic segmentation task. It proposes a cost-free multimodal FS-PCS setup. Based on this setup, the authors introduce MM-FSS to utilize the multimodal information to help with few-shot learning. The model includes MCS to generate multimodal correlations and MSF to refine the correlations by textual guidance. To mitigate training bias, the authors also propose a TACC technique to adaptively calibrate predictions at test time. The experiments demonstrate its effectiveness.

strengths:
+ The paper is well-written and easy to follow, with clear motivations for each proposed design.
+ It introduces the first multimodal few-shot 3D segmentation setting, which is cost-free and doesn’t increase labeling effort, consistent with standard few-shot tasks. The idea of leveraging available free modalities to help few-shot learning could provide valuable insights for the field.
+ The proposed model is novel and well-justified. The MCS and MSF modules use different modalities by exploiting cross-modal visual correlations and textual semantics, respectively. The TACC module is parameter-free and used at test time to calibrate predictions adaptively based on support samples.
+ The performance gains are substantial, and the experiments are thorough. Detailed ablation studies and visualizations are provided and effectively validate the proposed design choices and justify their motivations.

weaknesses:
Certain paragraphs could be more concise. For example, the paragraph (lines 235-240) explains the two training steps. Then the paragraph (lines 259-266) writes about these two steps again. These two parts could be merged to improve conciseness.

questions:
See weakness.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new multi-modal Few-Shot 3D Point Cloud Semantic Segmentation (FS-PCS) task. The authors propose the MultiModal Few-Shot SegNet (MM-FSS) method, which effectively incorporates multi-modal information (2D images and text) in a few-shot learning scenario. This is achieved through the introduction of two key modules: the Multimodal Correlation Fusion (MCF) module and the Multimodal Semantic Fusion (MSF) module. To further improve generalization, MM-FSS also introduces a Test-time Adaptive Cross-modal Calibration (TACC) technique to mitigate training bias. The MM-FSS model demonstrates superior performance and generalization capabilities in few-shot settings.

Initial reviewer concerns focused on several areas, including:

- The conciseness of the writing (YYsz)
- A need for more discussion on the challenges of using multimodal features for few-shot segmentation (k1NE)
- Explanations regarding the effect of the MSF module in enhancing correlation (8jox)
- Insufficient implementation details, such as how 2D images are integrated into the multi-modal FS-PCS task, how viewpoints for RGB images are determined, and how support samples are selected (qoPY)
- The absence of an ablation study on the individual impact of textual and 2D image modalities (qoPY)
- The practical applicability and effectiveness of the method in large-scale FS-PCS datasets (qoPY)
- A comparison of computational complexity and efficiency (qoPY)
- The possibility of incorporating textual or other types of information into the support set (qoPY)

After the rebuttal, all reviewers acknowledged that their concerns had been addressed. As a result, the paper received positive ratings from all four reviewers. The AC concurs with the reviewers' assessments regarding the strengths of the paper, including the introduction of a practical multimodal few-shot 3D segmentation setting, the effectiveness of the proposed method, and the substantial performance improvements demonstrated.

Based on the positive feedback and the authors’ thorough response to the reviewers' concerns, the AC recommends acceptance of this paper.