Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models

Authors: Xiao Xu, Tianhao Niu, Yuxi Xie, Libo Qin, Wanxiang Che, Min-Yen Kan
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multimodal Large Language Model, Multi-Grained Annotations, Fine-Grained Annotations, Concept Annotations, Vision--Language Learning

Abstract:
Multimodal Large Language Models (MLLMs) excel in vision--language tasks by pre-training solely on coarse-grained concept annotations (e.g., image captions).
We hypothesize that integrating fine-grained concept annotations (e.g., object labels and object regions) will further improve performance, as both data granularities complement each other in terms of breadth and depth in concept representation.

We introduce a new dataset featuring Multimodal Multi-Grained Concept annotations (MMGiC) for MLLMs.
In constructing MMGiC, we explore the impact of different data recipes on multimodal comprehension and generation.
Our analyses reveal that multi-grained concept annotations integrate and complement each other, under our structured template and autoregressive discrete framework.

We definitively show that multi-grained concepts do facilitate MLLMs to better locate and learn concepts, aligning vision and language at multiple granularities.
We further validate our hypothesis by investigating the comparison and collaboration between MMGiC and image--caption data on 12 multimodal comprehension and generation benchmarks, e.g., their appropriate combination achieve 3.95% and 2.34% accuracy improvements on POPE and SEED-Bench.
Code, data and models will be made openly available.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730174678178

thread_type:
official_review

<thread_object>
creator: reviewer_vyks

<content>
summary:
The paper introduces the MMGIC dataset with multi-grained concept annotations, significantly improving model understanding and generation abilities, and demonstrates its effectiveness in zero-shot image captioning and generation tasks. It details the data filtering and synthesis process to enhance data quality.

strengths:
1. The introduction of the MMGIC dataset with multi-grained concept annotations is a novel approach that significantly enhances model understanding and generation capabilities, providing richer and more detailed annotation information compared to traditional datasets.
2. The paper details an effective data filtering and synthesis process from multiple public datasets, ensuring high data quality and diversity, which in turn improves model training efficiency and effectiveness.

weaknesses:
1. This paper emphasizes the introduction of a novel dataset. However, the process of image collection and dataset creation is relatively straightforward and lacks significant innovation.
2. The paper mentions several new features of the dataset, but these have not been thoroughly validated or discussed in either theoretical or subsequent practical experiments. Specifically, the proposed label description, which involves designing prompts to generate detailed descriptions via LLMs, is general in nature and unrelated to the images. This could potentially lead to hallucinations during model training.
3. Although this paper is focused on the dataset, a significant portion is dedicated to describing the model architecture. Additionally, the experimental results lack comparative references from other studies, making it difficult to verify the dataset's effectiveness.

questions:
1. Shouldn't you compare your method with others to demonstrate the superiority of your dataset?

2. Compared to others' methods of creating datasets, what are the advantages of your approach?

[1] Chen, Lin, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. "Sharegpt4v: Improving large multi-modal models with better captions." In: ECCV (2024).  

[2] Urbanek, Jack, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. "A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions."  In: CVPR (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699109039

thread_type:
official_review

<thread_object>
creator: reviewer_xnsu

<content>
summary:
This research examines how MLLMs perform better when given two types of image information: broad descriptions (like overall image captions) and specific details (such as labeled objects and their locations). By using both levels of information together, MLLMs can better connect visual and linguistic elements, leading to improved performance on tasks that combine vision and language—whether that's describing images, answering questions about them, or creating images from text descriptions.

strengths:
1- The researchers developed MMGIC, a new dataset that provides image annotations at multiple levels of detail. This fills an important need in the field of multimodal learning, where existing datasets often lack such comprehensive labeling.

2- The study's findings are backed by concrete test results, particularly on established benchmarks like POPE and SEED-Bench. By analyzing different combinations of annotation types, the researchers showed that using multiple levels of image descriptions significantly boosts how well MLLMs handle complex tasks that combine visual and language understanding.

weaknesses:
1- Creating large datasets with detailed object-level annotations poses a major challenge, as it demands either extensive manual labor or sophisticated automated systems. While MMGIC makes valuable contributions, its reliance on existing public datasets creates two key limitations: it's difficult to expand the dataset size, and there's less room for automated data collection.

2- The study tests how different combinations of image annotations affect model performance. However, it falls short in explaining why certain combinations are more effective than others. This gap in analysis means we can see which approaches produced better results, but we don't fully understand the underlying reasons for their success.

questions:
Address each of the weaknesses mentioned.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700921801

thread_type:
official_review

<thread_object>
creator: reviewer_twwp

<content>
summary:
To integrate fine-grained concept annotations into Multimodal Large Language models (MLLMs) learning, this paper proposes a new dataset to explore the impact of different data recipes on multimodal comprehension and generation.  Extensive experiments demonstrate that multi-grained concepts do facilitate MLLMs to better locate and learn concepts, aligning vision and language at multiple granularities.

strengths:
1. The proposed dataset with fine-grained annotations can be beneficial to the multimodal community. 
2. Authors provided extensive ablation studies to compare different types of fine-grained annotations.
3. The proposed dataset demonstrates reasonably good results on image captioning, retrieval, and image generation tasks.

weaknesses:
1. Considering the rich information of the fine-grained annotations, there seems to be marginal improvements comparing row 3 and row 0 in Table 1.

2. In Table 4, MLLM-MMGIC performs slightly better or even worse compared to the baseline model LaViT-v2-7B. It is a little bit unclear about the effectiveness of the proposed dataset.

3. Considering the large amount of images in IC dataset (i.e., 52M), the improvement of row 1 over row 2 is marginal.

questions:
See the above weakness. The main concern is related to the performance on various tasks. The authors provide a large scale image-text dataset with fine-grained annotations. As this dataset is mainly collected from public object detection datasets, it should be naturally beneficial to VQA /GQA tasks. However, such improvements seem to be marginal on several tasks. For image captioning tasks, the model trained with the proposed dataset perform even worse compared to the baseline. It is difficult to evaluate the quality and effectiveness of the proposed dataset given such results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702506045

thread_type:
official_review

<thread_object>
creator: reviewer_cquc

<content>
summary:
The paper introduces a new dataset, Multimodal Multi-Grained Concept Annotations (MMGIC), aimed at enhancing Multimodal Large Language Models (MLLMs) in vision-language tasks by integrating both coarse-grained and fine-grained concept annotations. MMGIC is designed to address this limitation by including fine-grained annotations, such as object labels and object regions, alongside traditional captions, to better align vision and language across multiple granularities. The proposed approach leverages MMGIC within a general MLLM framework and evaluates its effectiveness on 12 multimodal comprehension and generation benchmarks.

strengths:
The paper introduces MMGIC, a multimodal dataset with multi-grained concept annotations that integrates both coarse-grained (e.g., captions) and fine-grained (e.g., object labels, regions) annotations. This is a novel approach to enrich MLLMs, allowing them to achieve more nuanced vision-language alignment across different granularities.

weaknesses:
**Writing and Clarity:** While I appreciate the authors’ efforts in extensively detailing the effectiveness of the proposed dataset, the writing is difficult to follow. There are too many unnecessary explanations, making the core claim of the paper unclear. The main focus seems to be on the dataset, yet the structure and writing style do not align well with a typical dataset paper, leading to a somewhat disorganized presentation.

**Dataset Quality:** As this is a dataset paper, it would be helpful to include a thorough evaluation of the dataset’s quality. Although large models like GPT are powerful to generate captions for local areas in the proposed dataset, they are known to suffer from hallucination issues. Could this dataset potentially exacerbate such problems, and how do you mitigate this risk?

**Untenable argument:** The importance of fine-grained understanding is well recognized. However, I question the point made in line 63 regarding the use of next-token prediction as a means to facilitate image understanding. Given the scale of the proposed dataset, a more effective loss function tailored to enhance comprehension might be a better choice than a simple autoregressive objective, which may not fully leverage the dataset’s potential.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731942410550

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response for Reviewers

comment:
Dear Reviewers, here, we hope to highlight the key conclusions of this paper to help you better understand our research focus and contribution.

The main focus of this paper is to **explore** the impact and potential of multi-grained concept annotations (**MMGIC**) on the multimodal understanding and generation capabilities of MLLM during the pre-training and SFT stages.
The construction of the dataset and model framework is a necessary **foundation** for our **exploration**.

The key conclusions of this paper include:

1. The multi-grained concept annotations provided by **MMGiC** can **complement** each other in image-text interleaved documents, and **improve** the multimodal **understanding** and **generation** capabilities of MLLM.
   - Sec.4.1, Pre-training Stage: To explore the impact of different annotation combinations in MMGiC on MLLM, we explore the performance of different data recipes. Based on the structured template and an autoregressive discrete MLLM framework, different annotations of MMGiC can be **integrated** into image-text interleaved documents and **facilitate** multimodal alignment across **multiple granularities simultaneously**, and **enhance** its multimodal understanding and generation capabilities.
   - Sec.4.4, SFT Stage: To further explore the impact of coarse-grained, fine-grained, and multi-grained concept annotations in MMGiC on MLLM, we **in-depth** analyze and explore different understanding dimensions and find that: fine-grained annotations can improve MLLM's ability to **recognize** and **locate** concepts, and **identify** relationships between concepts more accurately; coarse-grained annotations can improve MLLM's ability to grasp the **holistic** understanding of the image; and multi-grained annotations can **further** enhance MLLM's multimodal understanding capabilities through the **collaborative complementarity** of annotations at different granularities.

2. **MMGiC** and traditional large-scale coarse-grained image-caption data, **IC**, have their **respective** advantages in the **depth** and **breadth** of concept understanding, and can further **improve** the performance of MLLM through effective **collaboration**.
   - Sec.4.2, Pre-training Stage: To compare the impact of MMGiC and IC on the performance, the potential of their collaboration, and the way of their collaboration, we construct IC dataset based on open-source data to compare performance and explore different collaboration strategies. We find that MLLM trained with **only 4M** MMGiC significantly **outperforms** MLLM trained with **52M** IC in image captioning and image generation tasks; an appropriate curriculum learning strategy, i.e., first learning low-quality large-scale IC data and then **high-quality** MMGiC data, can help MLLM achieve **optimal** average performance.
   - Sec.4.3, SFT Stage: We fine-tune our three baselines (pre-trained with IC, MMGiC, and both) with SFT data, and explore their performance on **12** multimodal understanding and generation benchmarks. MLLMs trained with MMGiC and IC have their **respective** advantages in the **depth** and **breadth** of concept understanding, and their effective **collaboration** can further improve performance.

In addition, we not only provide qualitative analysis in Sec.4 (Fig.3, Fig.4, Fig.5), but also supplement **more** exploratory experiments and analysis in App.C (6 pages), including **concept overlap** analysis, results and analysis on **image editing** tasks, **self-generated** multi-grained concept annotations in the **inference** stage, the **effectiveness** of directly using MMGiC as SFT data, etc.

In our **newly** submitted version, based on your feedback and suggestions,  
- we optimize the Sec.2&3 to make the paper more clear and further highlight the research focus of this paper based on the Q1 of Reviewer `cquC`.
- we add visualizations of the attention maps in App.C.9 and Fig.10 to more intuitively "**see**" the differences between models trained with different annotation combinations based on the Q2 of Reviewer `Xnsu`.
- we add a discussion (App.D.5) about our efforts to avoid hallucination problems in label description generation based on the Q3 of Reviewer `VYKS`.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper present a new dataset called MMGiC, aimed at improving the fine-grained image understanding for multimodla LLMs. The authors curated a number of object detection and scene graph datasets and convert the region annotations to training data for MLLMs. In addition to the dataset, the authors also proposed an autoregressive discrete MLLM framework which can not only generate textual tokens but also visual tokens for image decoding and generation.The authors evaluated the trained model for both multimodal understanding and generation, and showed superior performance to the model only trained with image caption data. 

There are some strengths as mentioned by reviewers which the ACs also agreed. Concretely, the authors introduced a new dataset which contain both global image description and fine-grained regional annotaitons. The results for models trained on this dataset show superior performance to baselines. However, the ACs also share some concerns with the reviewers that: 1) this work lacks an in-depth analysis about how the different combination of dataset affects the final performance; 2)  After the SFT, the model pretrained with the proposed data does not show much improvements over the baseline (i.e., using MLLM-IC data for pretraining). especially on the understanding tasks. There is no clear explanation about this from the authors.

Due to limisted discussions between authors and reviewers, the ACs read the submission and all authors' responses. Overall, the ACs agreed that some of the comments from reviewers are subjective and made without careful understanding of the detailes. Ruling the out, in the meanwhile, many drawbacks pointed out by the reviewers are valid and align with the overall quality of this work. As such, the ACs think the overall quality of this work does not reach the bar of this venue, and suggest the authors further enhance the presentation and convey more detailed analysis to support the ambitious goal of improving both multimodal understanding and generation.