Weakly-supervised 3D Referring Expression Segmentation

Authors: Yihang Liu, Changli Wu, Xiaoshuai Sun, Jiayi Ji, Yiwei Ma, Gen Luo, Liujuan Cao, Rongrong Ji
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Weakly-supervised learning, 3D Referring Expression Segmentation

Abstract:
3D Referring Expression Segmentation (3D-RES) aims to generate precise segmentation masks for targets based on free-form text descriptions. Despite significant advancements, current methods still rely on costly point-level mask-description pair annotations. In this paper, we introduce the Multi-Expert Network (MEN), a novel weakly supervised framework that utilizes the multimodal alignment of vision-language models across various semantic cues to reveal the relationships between descriptions and 3D instances. The primary challenges lie in effectively extracting and matching visual and textual context, while eliminating potential distractions. To address this, we propose the Multi-Expert Mining (MEM) and Multi-Expert Aggregation (MEA) modules. The MEM module employs multiple experts to extract semantic cues from full-context, attribute, and category dimensions. The MEA module mathematically consolidates the outputs of these experts, automatically assigning greater weight to more accurate ones, thus improving target selection accuracy and robustness. Extensive experiments on the ScanRefer and Multi3DRefer benchmarks demonstrate the effectiveness of our method in addressing the challenges of weakly supervised 3D-RES.

Decision: reject
Reviewer scores: 8, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730017684713

thread_type:
official_review

<thread_object>
creator: reviewer_njzx

<content>
summary:
This paper proposes a new weakly supervised 3D Referring Expression Segmentation task, along with a baseline method, named Multi-Expert Network (MEN), designed to address this task. The proposed task can potentially eliminate the need for laborious point-level mask annotations. The proposed method includes two main components: a Multi-Expert Mining module, which leverages various experts to extract and align visual and textual features from multiple complementary cues, and a Multi-Expert Aggregation module, which integrates the knowledge acquired from these experts.

strengths:
1. The proposed 3D-RES task is well motivated.
2. The proposed MEN method is reasonable and all proposed modules are thoroughly ablated.

weaknesses:
1. The proposed method employs SoftGroup as its instance segmentation backbone. However, since this model was trained on ScanNet data, there is a significant data leakage issue, as both evaluation datasets (ScanRefer and Multi3DRefer) were built using ScanNet. Despite this favorable evaluation setting, the method's performance remains substantially lower than fully supervised approaches. The performance would likely deteriorate further when applied to other datasets. Given that such a method is primarily intended for use in unseen environments, the poor generalization capability presents a major concern and undermines the core motivation of this work. An ablation using an alternative backbone trained on different data may help to alleviate this concern. 
2. Based on Table 1, distilling MEN to fully-supervised methods does not improve performance in the "Multiple" and "Overall" settings. This outcome suggests that the segmentation masks obtained through this method are noisy and struggle to differentiate between multiple objects, rendering the supervision signal less effective. Table 2 further supports this observation, showing poor performance in both zero-target and multiple-target cases. These limitations imply that the model may lack the ability to accurately interpret referring expressions to distinguish between multiple objects. This raises the question of whether there is any difference between using referring expressions and class labels for training this model. An ablation study on this would help to clear the confusion.
3. The proposed method, while a commendable effort by the author, lacks elegance. It primarily represents a complex engineering solution that combines various existing approaches and models.
4. Typo: 
> 1. Line 201: Dimension of RGB-D should not be 3xWxH.
> 2. Line 255: textaul -> textual.
5. Paper has organizational issues. Many core details are not mentioned or put in the appendix.

questions:
Refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730290413849

thread_type:
official_review

<thread_object>
creator: reviewer_kmxl

<content>
summary:
The paper proposes a weakly supervised 3D-RES task and introduces the Multi-Expert Network (MEN), a framework that leverages multiple cues for referring expression reasoning, reducing reliance on expensive mask-description pair annotations. The framework includes the Multi-Expert Mining (MEM) module, which aligns visual and textual contexts from three complementary cues, and the Multi-Expert Aggregation (MEA) module, which consolidates expert knowledge while filtering out interference to emphasize the target instance. Extensive experiments on the ScanRefer and Multi3DRefer datasets validate the effectiveness of our approach in addressing weakly-supervised 3D-RES.

strengths:
1. Proposes a novel weakly supervised approach to handle 3D RES.
2. The paper is well-written and easy to understand.
3. Demonstrates impressive results across multiple metrics in the weakly supervised setting.

weaknesses:
1. This pipeline processes each frame separately, however, target objects and anchors are not always present within the same frame due to the limited field of view. This slightly goes against the authors' statement in Lines 189-191: “The full-context expert focuses on... understanding the spatial relationships and the surrounding environment of the target.”
2. The proposed pipeline appears to struggle with cases containing zero targets, even in scenarios without distractors, as shown in Table 2.
3. The ablation study in Table 3 is limited to evaluating the performance of individual modules. Specifically, it would be helpful to see the performance of samples in the "Unique" category.

questions:
1. How can one ensure that the methods effectively address the issues outlined in Weakness #1?
2. Have you considered adding a confidence threshold to allow the pipeline to handle scenarios with zero targets?
3. Could you provide the ablation studies mentioned in Weakness #3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730360583197

thread_type:
official_review

<thread_object>
creator: reviewer_tjsr

<content>
summary:
This paper introduces the Multi-Expert Network (MEN), a weakly supervised framework for 3D Referring Expression Segmentation (3D-RES) that reduces dependency on costly annotations. MEN uses two modules: Multi-Expert Mining (MEM) to extract semantic cues from various dimensions and Multi-Expert Aggregation (MEA) to consolidate these cues.

strengths:
Experimental results demonstrate MEN’s effectiveness in accurately segmenting 3D objects based on text descriptions, even with limited supervision, outperforming some fully supervised 3D-RES methods.

weaknesses:
1. The paper lacks a discussion of related works, specifically on open-vocabulary 3D segmentation. Open-vocabulary 3D segmentation techniques utilize open-vocabulary queries (such as class labels or text queries) to segment 3D objects. These approaches can be adapted for weakly supervised 3D referring segmentation by replacing open-vocabulary queries with referring expressions. For example, methods [A-C] could be applied to weakly-supervised 3D segmentation tasks similar to those proposed in this paper, as their models operate without manual 3D annotations or can produce 3D segmentation results in a zero-shot manner. Including these methods in the related work discussion or comparing them in the experiments would strengthen the paper.
2. Methods [A-C] are capable of handling weakly-supervised 3D referring segmentation.
3. The writing is hard to follow, particularly in Lines 294-323. For instance, the phrase in Line 299, “Then &(P1, …, Pn) exists,” lacks clarity, making its intent unclear. Furthermore, Eq. (13) includes a parameter $\sigma_x$, which is missing in Eq. (14).
4. The symbols in Eq. (11) are inconsistent with those in Figure 2. Eq (11) uses $\cdot$ to denote matrix multiplication, but Figure 2 uses $\otimes$.

Minor issue: 
Incorrect description in Figure 4(a): It appears that the target in Figure 4(a) is a table, not a couch, yet the descriptions for (a) and (b) are identical.

[A] OpenScene: 3D Scene Understanding with Open Vocabularies. In CVPR 2023

[B] OpenMask3D: Open-Vocabulary 3D Instance Segmentation. In NIPS 2023. 

[C] Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance. In CVPR 2024

questions:
In the category expert, if the class label can be directly extracted from the input text, why is a classifier needed for the text as well?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730469574824

thread_type:
official_review

<thread_object>
creator: reviewer_wnd8

<content>
summary:
The authors present a weakly supervised 3D-RES task and the Multi-Expert Network (MEN), an innovative framework that aims to mitigate the reliance on costly annotations by leveraging diverse complementary visual-language cues. MEN consists of two core modules: the Multi-Expert Mining (MEM) module and the Multi-Expert Aggregation (MEA) module. The MEM module employs multiple visual-language experts to independently mine distinct visual and linguistic cues, aligning visual and textual contexts from three complementary perspectives. Meanwhile, the MEA module synthesizes the outputs of these experts through a distribution-based fusion approach, adaptively assigning appropriate fusion weights to enhance integration. Extensive experiments on the ScanRefer and Multi3DRefer datasets underscore the efficacy of this approach for the weakly supervised 3D-RES task.

strengths:
1、	The high cost of obtaining annotations in the 3D domain has been a significant barrier to progress in related fields. Thus, this work’s proposal of a new weakly supervised 3D-RES task, aiming to reduce dependency on 3D annotations, is highly valuable and offers a fresh direction for advancing the community's development.

2、	Additionally, this paper introduces a novel weakly supervised 3D-RES framework, MEN, to address the weakly supervised 3D-RES task, providing a strong baseline. The incorporation of a multi-expert fusion approach into the 3D-RES task is also innovative and noteworthy.

3、	The proposed MEM module in this paper effectively explores different visual-language relationships by leveraging three complementary cues: Full-context, Attribute, and Category.

weaknesses:
1、	Lines 202-205 lack a detailed and clear description of visibility. Additionally, some symbol notations in the text are ambiguous; for instance, the "textual full-context feature" mentioned in line 215 is referred to as "2D-fine text features" in line 199.

2、	The proposed method demonstrates high performance on traditional 3D-RES tasks; however, it exhibits lower performance on certain metrics in the 3D-GRES task, particularly in zero-target scenarios. What factors contribute to this underperformance, and are there potential improvements that could be implemented?

3、	This two-stage paradigm is influenced by the proposals generated in the first stage. Therefore, how would the overall performance of the network change if a different instance segmentation model were utilized in the first stage?

4、	Furthermore, I would like to see the performance upper bound of the network when errors from the first-stage segmentation model are excluded.

questions:
1、	This paper utilizes an off-the-shelf natural language parser to extract attribute-level text, which can be understood as a standard approach. However, I am quite curious whether employing a powerful large language model (LLM) to construct attribute phrases as text inputs for the attribute experts could further enhance the model's performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732451417042

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common response

comment:
We would like to express our gratitude to the reviewers for their valuable feedback and positive comments on our paper. Their thoughtful and detailed evaluations have been instrumental in enhancing both the clarity and overall quality of our work.

We appreciate Reviewer **wNd8** $\color{red}{(Rating:\mathbf{8},\ Confidence:\mathbf{4})}$ for acknowledging the strengths of our paper. Specifically, they recognized the contribution of our proposed weakly supervised 3D-RES task in advancing the community's development. They also appreciated the incorporation of a multi-expert fusion approach into the 3D-RES task. Additionally, they acknowledge that our MEM  module effectively explores different visual-language relationships by leveraging three complementary cues.

Reviewer **tjsR** $\color{red}{(Rating:\mathbf{5},\ Confidence:\mathbf{4})}$ acknowledged our extensive comparative experiments with state-of-the-art methods and highlighted our method's effectiveness in accurately segmenting 3D objects based on text descriptions. 

Reviewer **kMXL** $\color{red}{(Rating:\mathbf{6},\ Confidence:\mathbf{5})}$  recognized the novelty of our weakly supervised 3D-RES framework and the effectiveness of the Multi-Expert Network in leveraging multiple cues for referring expression reasoning. They also commended the clarity and accessibility of our writing. Additionally, we are grateful for their acknowledgment of the significance of our experimental results, emphasizing their role in advancing weakly supervised 3D-RES research.

Reviewer **nJzX** $\color{red}{(Rating:\mathbf{5},\ Confidence:\mathbf{4})}$ expressed appreciation for the clear articulation of our motivation and the thorough ablation studies conducted on our modules. They also acknowledged the potential of our proposed weakly supervised 3D-RES framework to eliminate the need for labor-intensive point-level mask annotations.

We sincerely appreciate the reviewers for recognizing these strengths. We genuinely appreciate the time and effort they have dedicated to this process. Their comments have further motivated us to address the concerns and improve the weaknesses pointed out in their reviews. We are committed to addressing their concerns and providing a detailed response in our rebuttal.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received significantly mixed reviews. The reviewers recognized that the proposed research direction, i.e., weakly supervised 3D referring expression segmentation (RES), is new and practically valuable, the proposed framework is novel and shows strong performance in the task. They at the same time raised concerns with underperformance on zero-target or multi-target scenes (wNd8, kMXL, nJzX), sensitivity to the instance segmentation model for proposal generation (wNd8, nJzX), lack of discussions of and comparisons with open-vocabulary 3D segmentation models that can handle the proposed task to some extent (kMXL, nJzX), inability to handle cross-frame objects (kMXL, nJzX), insufficient ablation study (kMXL), and incremental technical innovation (nJzX). Also, all the reviewers pointed out clarity issues. 

The authors' rebuttal and subsequent responses in the discussion period address some of these concerns but failed to fully assuage all of them, even after a large body of author-reviewer discussions. As a result, after the discussion period, the two negative reviewers (kMXL, nJzX) still have concerns about weak clarity of the revised manuscript, the issue in handling cross-frame objects, sensitivity to the instance segmentation model, the low performance on zero-target or multi-target scenes, and lack of technical innovation. The AC found that these comments are all valid, although some of them, such as the underperformance issue, can be execused as this work is the first attempt to weakly supervised 3D RES. 

In particular, the clarity issue should be addressed appropriately before publication. Although the AC sincerely appreciates the extensive additional experiments conducted during the rebuttal period and the latest revision including their results, their presentation in the current manuscript is not well organized. To ensure that the paper is a good first introduction to weakly supervised 3D RES, it is essential to provide a comprehensive and well-degisned benchmark for the task. In this regard, the paper should elaborate on all datasets, evaluation settings, and their purposes (e.g., evaluating conventional 3D RES, generalized 3D RES, insensitivity to proposal generator) *in the main text*, not in the appendix, and the results of all the experiments should be analyzed collectively. The AC sees that the volume of revision required for this purpose will exceed what is typically expected from a camera-ready revision. In addition, other remaining concerns with cross-frame objects and sensitivity to proposal generator should be carefully addressed to further strenghthen the paper of course.

Putting these together, the AC considers that the remaining concerns outweigh the positive comments and rebuttal, and thus regrets to recommend rejection. The authors are strongly encouraged to carefully revise the paper to reflect the valuable comments by the reviewers, to add new results brought up in the rebuttal and discussions, and to further improve the quality of writing for next submissions.