MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation

Authors: Donggon Jang, Yucheol Cho, Suin Lee, Taehyeon Kim, Daeshik Kim
Primary area: datasets and benchmarks
Keywords: Multimodal Dataset, Multi-target and Multi-granularity Reasoning Segmentation, Benchmark

Abstract:
The fusion of Large Language Models (LLMs) with vision models is pioneering new possibilities in user-interactive vision-language tasks. A notable application is reasoning segmentation, where models generate pixel-level segmentation masks by comprehending implicit meanings in human instructions. However, seamless human-AI interaction demands more than just object-level recognition; it requires understanding both objects and the functions of their detailed parts, particularly in multi-target scenarios. For example, when instructing a robot to \textit{“turn on the TV"}, there could be various ways to accomplish this command. Recognizing multiple objects capable of turning on the TV, such as the TV itself or a remote control (multi-target), provides more flexible options and aids in finding the optimized scenario. Furthermore, understanding specific parts of these objects, like the TV's button or the remote's button (part-level), is important for completing the action. Unfortunately, current reasoning segmentation datasets predominantly focus on a single target object-level reasoning, which limits the detailed recognition of an object's parts in multi-target contexts. To address this gap, we construct a large-scale dataset called Multi-target and Multi-granularity Reasoning (MMR). MMR comprises 194K complex and implicit instructions that consider multi-target, object-level, and part-level aspects, based on pre-existing image-mask sets. This dataset supports diverse and context-aware interactions by hierarchically providing object and part information. Moreover, we propose a straightforward yet effective framework for multi-target, object-level, and part-level reasoning segmentation. Experimental results on MMR show that the proposed method can reason effectively in multi-target and multi-granularity scenarios, while the existing reasoning segmentation model still has room for improvement. The dataset is available at \url{https://github.com/jdg900/MMR}.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729585057154

thread_type:
official_review

<thread_object>
creator: reviewer_d24r

<content>
summary:
This paper introduces a novel dataset, MMR, the first part-level dataset for the reasoning segmentation tasks. In addition, a new network framework is proposed to leverage low-level fine-grained information and to address the limitation of the existing LISA model, which can only segment a single object. The authors conduct experiments to evaluate the performance of existing methods on the proposed MMR dataset and demonstrate the advantages of the proposed network framework.

strengths:
1. This paper is well written and easy to follow. 
2. The proposed MMR dataset is highly valuable to the research community, as part-level reasoning segmentation is crucial in real-world applications, such as robotic control. However, there is currently a lack of available datasets for research in this area.
3. A detailed analysis is provided to thoroughly present the characteristics of the MMR dataset.

weaknesses:
1. The contributions of the proposed M2SA network framework are incremental. The early local feature fusion appears to be only a minor structural modification. Additionally, the strategy of employing multiple [SEG] tokens has already been introduced in earlier methods, such as [a]. The authors should clarify the differences between their approach and [a].
2. This paper could benefit from more thorough experiments based on the characteristics of the dataset. For instance, does the M2SA trained on the MMR dataset show a noticeable long-tail phenomenon, i.e., better performance on the more frequently occurring object and part categories as presented in Fig.3? Additionally, what is the model’s open-vocabulary performance on categories that do not appear in the training set?
3. More examples of the image-question-answer triplet in the MMR dataset could be presented in the paper to enable readers to understand the characteristics of the dataset more quickly and intuitively.

[a] GSVA: Generalized Segmentation via Multimodal Large Language Models, CVPR2024

questions:
1. Why did you remove the generated questions that contain explicit target coordinates or strong hints? I think training with such data would enhance the model’s ability to handle target-specific inputs. For example, if an image contains two different animals, a fish and a cat, users could indicate the coordinates of the animal they are interested in and ask, “Which part of this animal [coordinates] uses its sense of smell?” The model could then segment either the nose or the fish’s gills depending on the coordinates provided. This could be quite interesting.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729590401381

thread_type:
official_review

<thread_object>
creator: reviewer_jq5y

<content>
summary:
This paper introduces a dataset named MMR, designed for multi-target and multi-granularity reasoning segmentation tasks. The goal is to address challenges in reasoning across multiple targets and different levels of granularity. The dataset comprises complex and implicit question pairs, covering both object-level and part-level reasoning. Additionally, the paper proposes a baseline model, M2SA, to achieve multi-target, object-level, and part-level reasoning segmentation.

strengths:
1. **Clear Writing**: The paper is well-organized and easy to understand.

2. **Significant Contribution of the Dataset**: The MMR dataset contains 196K samples. Although it was generated using large models, a rigorous filtering process was employed to ensure data quality.

weaknesses:
1. **Lack of Targeted Design in the Baseline Model**: The baseline model (Early Local Feature Fusion and Multi [SEG] Tokens) does not incorporate specific structures to effectively address the multi-target and part-level reasoning required by the MMR dataset. As a result, it lacks novelty, leading to underwhelming performance in Table 3.

2. **Limited Performance in Table 3**: The comparison methods in Table 3 are not sufficiently recent. The authors did not include comparisons with more relevant multi-target approaches, such as GSVA [1] or GLaMM [2]. This limits the impact of the proposed approach, as the results are not very competitive.

3. **Insufficient Comparisons in Table 2**: The methods compared in Table 2 are too limited. I strongly recommend including more methods that could be adapted for the MMR task to facilitate meaningful comparisons for future research.

questions:
1. Will the proposed dataset be made publicly available?
2. The paper mentions the use of 4 A6000 GPUs. How long does it take to train the proposed model on the MMR dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730452251116

thread_type:
official_review

<thread_object>
creator: reviewer_wftf

<content>
summary:
This paper provides a large multi-target and multi-granularity reasoning segmentation benchmark. Based on this benchmark, this paper designs a baseline model trained on it while evaluating public datasets to present the effectiveness of both the benchmarks and the baseline. Experiments demonstrate that the proposed baseline outperforms LISA and other representative approaches.

strengths:
1. The distinguishing characteristic of the proposed benchmark is clear, which includes multi-granularity and more images.
2. Multi-target and multi-granularity reasoning segmentation is a valuable research topic.
3. The overall writing is fluent.

weaknesses:
1. This paper provides few comparisons on the proposed benchmark. It is not clear whether the proposed baseline model outperforms other MLLMs on multi-target and multi-granularity reasoning segmentation.
2. The major contribution lies in the benchmark, while this benchmark is auto-annotated based on the existing dataset PACO-LVIS, which hurts the contribution.
3. According to Table 1, MMR offers both object-level and multi-target annotations, making it more comprehensive than ReasonSeg and MUSE. This paper could include zero-shot evaluations on these two benchmarks to further demonstrate effectiveness.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732686440797

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers

comment:
We sincerely appreciate the time and effort each reviewer dedicated to providing thoughtful and constructive feedback. We have now submitted a revised version of the paper, incorporating the reviewers’ suggestions.

In summary, we have implemented the following revisions:
- We have included the comparison with GSVA and GLaMM in Table. 2, Table. 3, Table. 4, and Table 5 of Sec 5.2.
- In Appendix A.11, we have added the comparison to validate the comprehensiveness of the MMR dataset against existing reasoning segmentation datasets through zero-shot evaluation
- We have included the performance evaluation on frequently and infrequently occurring object categories and part categories in Appendix A.12. 
-  We have included a zero-shot multi-referring segmentation performance comparison on unseen datasets (e.g., GTAV and Cityscapes) in Appendix A.10.
- We have added more examples of the MMR dataset in Appendix A.14.

We are grateful for the reviewers' acknowledgment that all major concerns have been addressed. 

A detailed explanation of how each issue was resolved is provided below.

Further discussion is always welcome, and we will do our best to further develop our research.
</content>
</thread_object>

</thread>


Metareview:
metareview:
# Summary and Recommendation for Acceptance

---

## Strengths:
1. **Innovative Dataset Contribution**:
   - Introduces the **MMR dataset**, a large-scale benchmark with 194K samples, addressing **multi-target** and **multi-granularity reasoning segmentation**.
   - Fills a critical gap by supporting object-level and part-level segmentation in multi-target scenarios, advancing user-interactive vision-language tasks.

2. **Framework and Benchmarks**:
   - Proposes the **M2SA framework**, tailored for multi-target, object-level, and part-level reasoning segmentation tasks.
   - Provides comprehensive benchmarks, demonstrating M2SA’s superior performance over state-of-the-art models (e.g., GSVA, GLaMM).

3. **Thorough Evaluation**:
   - Includes extensive performance comparisons, zero-shot evaluations on unseen datasets (e.g., GTAV, Cityscapes), and analysis of long-tail phenomena.
   - Demonstrates strong generalization across datasets and tasks.

4. **Community Impact**:
   - Addresses an underexplored area in reasoning segmentation with direct applications in robotics and human-AI interaction.
   - Promises public release of the dataset and code, ensuring reproducibility and encouraging further research.

5. **Presentation and Revisions**:
   - The paper is well-written and easy to follow.
   - Authors effectively addressed reviewer concerns by adding additional analyses, comparisons, and dataset examples.

---

## Weaknesses:
1. **Incremental Model Contribution**:
   - The M2SA framework introduces minor modifications (e.g., early local feature fusion, multiple [SEG] tokens), making its novelty incremental.

2. **Auto-annotation Concerns**:
   - Relies on annotations from PACO-LVIS, raising questions about originality. However, the transformation into reasoning-focused question-answer pairs adds significant value.

3. **Performance on Rare Categories**:
   - Slightly lower performance on infrequent object categories, indicating potential room for improvement in handling long-tail distributions.

4. **Initial Comparisons**:
   - Early comparisons with state-of-the-art methods (e.g., GSVA, GLaMM) were limited, though this was resolved during the rebuttal phase.

---

## Authors' Mitigation:
1. **Expanded Comparisons**:
   - Added performance evaluations of GSVA and GLaMM on the MMR dataset, demonstrating the superior performance of M2SA.
   - Conducted zero-shot performance evaluations on unseen datasets (GTAV, Cityscapes) to validate generalization.

2. **Dataset Enhancements**:
   - Addressed auto-annotation concerns by emphasizing the novel reasoning-focused question-answer pairs and the complexity of the queries.

3. **Detailed Analyses**:
   - Analyzed long-tail performance, showing consistency across frequent and infrequent part categories.
   - Provided open-vocabulary segmentation evaluations on new datasets.

4. **Improved Presentation**:
   - Included additional examples and explanations to enhance the clarity and comprehensiveness of the dataset and benchmarks.

---

## Remaining Weaknesses:
1. **Incremental Framework**:
   - The M2SA framework’s contributions remain incremental compared to the dataset’s innovation.
2. **Auto-annotation Limitations**:
   - Despite the mitigations, reliance on PACO-LVIS annotations may restrict perceived originality.

---

## Justification for Acceptance:
The **MMR dataset** addresses a critical gap in reasoning segmentation by enabling research on multi-target and multi-granularity tasks that are pivotal for advanced vision-language systems. While the accompanying M2SA framework lacks significant novelty, its robust performance and the dataset's comprehensive benchmarks underscore the paper's value.

The authors’ detailed rebuttal addressed all reviewer concerns, including expanded comparisons with state-of-the-art methods, in-depth dataset analyses, and additional experiments. The promise of public release of the dataset and code further solidifies its potential impact.

**Recommendation**: Accept.   The dataset's innovation, combined with strong experimental validation and its value to the research community, makes it a worthy contribution to the conference.