GroundingGPT: Language Enhanced Multi-modal Grounding Model (2024)

Abstract: Multi-modal large language models (MLLMs) have demonstrated remarkable performance across various tasks. However, these models often prioritize capturing global information and overlook the importance of perceiving local information. This limitation hinders their ability to effectively understand fine-grained details and handle grounding tasks that necessitate nuanced comprehension. Although some recent works have made strides in this, they have primarily focused on single-modality inputs. Therefore, we propose GroundingGPT, an end-to-end language enhanced multimodal grounding model. It is designed to perform fine-grained grounding tasks for three modalities: image, video and audio. To enhance the model’s performance, we adopt a coarse-to-fine training strategy, utilizing a threestage training approach to progressively enhance the model’s semantic awareness and finegrained understanding capabilities. Additionally, we employ a diversified stage-specific dataset construction pipeline, developing a multi-modal, multi-granularity dataset tailored for training the model in different stages. Extensive experiments conducted on multiple multimodal benchmarks demonstrate that our model achieves impressive fine-grained understanding of multi-modal inputs on grounding tasks while maintaining or improving its global comprehension capabilities. Our code, model, and dataset are available at https://github. com/lzw-lzw/GroundingGPT.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6657–6678 August 11-16, 2024 ©2024 Association for Computational Linguistics
Multi-modal large language models (MLLMs) have demonstrated remarkable performance across various tasks. However, these models often prioritize capturing global information and overlook the importance of perceiving local information. This limitation hinders their ability to effectively understand fine-grained details and handle grounding tasks that necessitate nuanced comprehension. Although some recent works have made strides in this, they have primarily focused on single-modality inputs. Therefore, we propose GroundingGPT, an end-to-end language enhanced multimodal grounding model. It is designed to perform fine-grained grounding tasks for three modalities: image, video and audio. To enhance the model’s performance, we adopt a coarse-to-fine training strategy, utilizing a threestage training approach to progressively enhance the model’s semantic awareness and finegrained understanding capabilities. Additionally, we employ a diversified stage-specific dataset construction pipeline, developing a multi-modal, multi-granularity dataset tailored for training the model in different stages. Extensive experiments conducted on multiple multimodal benchmarks demonstrate that our model achieves impressive fine-grained understanding of multi-modal inputs on grounding tasks while maintaining or improving its global comprehension capabilities. Our code, model, and dataset are available at https://github. com/lzw-lzw/GroundingGPT.

heading: 1 Introduction
text: Building upon the capabilities of large language models (LLMs), research on multi-modal large language models (MLLMs) has also advanced, enabling understanding across a broader range of modalities. Representative models such as LLaVA (Liu et al., 2023a) and MiniGPT-4 (Zhu et al., 2023) align visual features obtained from image encoders with LLM embedding space through
visual instruction tuning, facilitating tasks such as image captioning and visual question answering.
However, existing MLLMs primarily focus on capturing global information while neglecting the fine-grained local information in multimodal inputs. This limitation restricts their applicability in grounding tasks requiring a more detailed understanding. Shikra (Chen et al., 2023b), BuboGPT (Zhao et al., 2023) and Ferret (You et al., 2023) have explored techniques that enable finer alignment and understanding of inputs. By considering local-level information, these models exhibit enhanced performance in grounding or referring tasks. These methods provide insights into finegrained understanding, but they are primarily limited to a single modality. There is still significant potential for exploring fine-grained understanding across other modalities.
To address the aforementioned issue, this paper proposes GroundingGPT, a language enhanced multi-modal grounding model, which is an endto-end unified large language model designed to perform multi-modal grounding and understanding tasks across various modalities, including image, video, and audio. The comparison between our model and other models can be found in Table 1. Specifically, our model employs modality-specific adapters to map feature representations from individual encoders to the embedding space of LLMs. To incorporate spatial and temporal information, we directly represent coordinates and timestamps as textual numbers, eliminating the need for vocabulary expansion. For training GroundingGPT, we design a three-stage coarse-to-fine training strategy. In the first stage, we align each pre-trained multimodal encoder with the LLM embedding space using modality-specific adapters. In the second stage, we aim to enable the model to capture finegrained information, including coordinates and timestamps. In the third stage, we perform multigranularity instruction tuning to refine the model’s
6657
responses. For each stage, we employed a stagespecific dataset construction pipeline to generate a diverse, multi-modal, and multi-granularity training dataset.
To summarize, our contributions are as follows:
• We propose GroundingGPT, an end-to-end multi-modal grounding model that accurately comprehends inputs and possesses robust grounding capabilities across multi modalities, including image, video and audio. To the best of our knowledge, GroundingGPT is the first model to achieve multi-modal finegrained understanding and grounding.
• For training GroundingGPT, we employ a three-stage coarse-to-fine training process that enables the model to capture high-level semantic information and low-level fine-grained details simultaneously. To address the issue of limited data, we construct a diverse and highquality multi-modal training dataset, which comprises a rich collection of multi-modal data enriched with fine-grained information.
• Extensive experiments conducted on a wide range of MLLM benchmarks demonstrate the generality and effectiveness of GroundingGPT in multi-modal grounding and understanding tasks across various modalities.

heading: 2 Related Work
text: Multi-modal Large Language Models (MLLMs) Recently, large language models (LLMs) represented by GPTs (Brown et al., 2020; OpenAI, 2023) and LLaMA (Touvron et al., 2023) have received extensive attention from researchers for
their remarkable performance in various natural language processing tasks. Substantial progress has been made in the field of MLLMs, which extend the support for multi-modal input and output beyond language. These MLLMs typically fine-tune pre-trained LLMs with multi-modal instructions, to enable understanding across multiple modalities. Models such as LLaVA, MiniGPT-4, and mPLUG-Owl (Ye et al., 2023) map image embeddings obtained from image encoders into the LLM space. Similarly, video MLLMs like VideoChat (Li et al., 2023b), Video-LLaMA (Zhang et al., 2023c), Video-Chatgpt (Maaz et al., 2023) and Valley (Luo et al., 2023), as well as speech MLLMs like SpeechGPT(Zhang et al., 2023b) and LLaSM (Shu et al., 2023), acquire multimodal understanding capabilities through similar approaches. In X-LLM (Chen et al., 2023a), each modality is processed independently through dedicated branches for multi-modal input processing. Pandagpt (Su et al., 2023) employs a unified embedding space trained by ImageBind (Girdhar et al., 2023) to facilitate joint understanding of various modal inputs. However, these models often fail to adequately capture details within inputs.
MLLMs For Grounding Task Recently, there has been a focus on training visual MLLMs to achieve fine-grained image understanding and visual grounding. Approaches such as KOSMOS2 (Peng et al., 2023) and Shikra achieve this by incorporating coordinates into the training data, enabling MLLMs to understand the location within images. On the other hand, approaches like NExT-Chat (Zhang et al., 2023a), LLaVAgrounding (Zhang et al., 2023d),GlaMM (Rasheed et al., 2023) and Ferret enhance perception of fine-
grained information by introducing additional region encoder modules. VTimeLLM (Huang et al., 2023) demonstrates the capability to understand fine-grained video moment and reason with respect to time boundary. BuboGPT (Zhao et al., 2023) enables cross-modal interaction between image, audio, and language, facilitating fine-grained understanding of different modalities.

heading: 3 Methods
text: We introduce the overall architecture of the GroundingGPT model in this section. Additionally, we will present our three-stage coarse-to-fine training strategy and data construction pipeline.

heading: 3.1 Model Architecture
text: Figure 1 illustrates the overall architecture of the GroundingGPT model. Multi-modal inputs are processed through modality-specific encoders to extract features. These features are then mapped to the LLM embedding space using corresponding adapters. We will also introduce the representation of coordinates and timestamps.

heading: 3.1.1 Image Branch
text: We employ the pre-trained CLIP visual encoder ViT-L/14 (Radford et al., 2021) to extract image features. The encoded image is represented as a fixed-length embedding vector I ∈ RKI×dI . To align the image representation with the LLM embedding space, we use an MLP to map the obtained features to the dimensions of LLMs. The mapped embeddings are then concatenated with text embeddings and used as input to LLMs, similar mapping methods are adopted for other modalities.

heading: 3.1.2 Video Branch
text: Considering the inherent information redundancy in videos and memory limitations, we uniformly sample M frames form the video. Each frame is processed by the image encoder, resulting in Vf = [v1, v2, . . . , vM ] where vi ∈ RKf×df represents the embedding of the i-th frame. To preserve temporal information, we introduce temporal position encoding to the representation. The enhanced representation is then fed into the Video Q-former with the same structure as the Q-Former in BLIP2 (Li et al., 2023a) to aggregate video information, which generates kV video embedding vectors of dimensions dV . These vectors form the representa-
tion V ∈ RkV ×dV for the entire video.

heading: 3.1.3 Audio Branch
text: The audio branch follows a structure similar to the video branch. We employ the ImageBind audio encoder, which processes 2-second audio clips with a 16kHz sampling rate and converts them into spectrograms using 128 mel-spectrogram bins. We sample N 2-second segments from the original audio and transform each segment into a vector, resulting in As = [a1, a2, . . . aN ], where ai ∈ RKs×ds represents the embedding of the i-th aduio segment. We incorporate temporal position encoding into As. Finally, we obtain a fixed-length audio representation sequence denoted as A ∈ RkA×dA using the audio Q-former like video branch.

heading: 3.1.4 Spatial-temporal Representation
text: We represent the bounding box in an image using four relative coordinate values: [x1, y1, x2, y2]. These values correspond to the upper left corner point and the lower right corner point of the bounding box. Each value is rounded to three decimal places. We concatenate this textual representation after the description related to the bounding box. Similarly, for representing timestamps, we use two two-digit decimals {t1, t2} to indicate the relative values of the start and end times of a time segment with respect to the total duration. This representation allows us to train the model without requiring additional vocabulary expansion or training. Examples of the training dataset are shown in Figure 2.

heading: 3.2 Coarse-to-Fine Training and Dataset
text: We employ a three-stage coarse-to-fine training strategy to train the model, while constructing specific datasets for each stage.

heading: 3.2.1 Multi-modal Pre-training
text: This stage focus on enabling the model to comprehend multi-modal inputs and develop a high-level semantic perception of the input. During the training process, the LLM and the encoders for each modality remain frozen, while only the adapters for each modality are trained.
Training Dataset We utilize public pretraining datasets as the primary source of our data. The training data for the image and video modalities is LLaVA-Pretrain-595k and Valley-Pretrain-703k, respectively. To construct the audio data, we adopt a similar approach as in LLaVA, leveraging the Wavcaps (Mei et al., 2023) dataset. Each sample is accompanied by a sampled instruction that requires the model to provide a concise description of the audio to construct a single-turn conversation.

heading: 3.2.2 Fine-grained Alignment Tuning
text: The second stage aims to enable the model to comprehend more detailed information, including coordinates and timestamps. Through training in this stage, the model achieves impressive results in various grounding tasks, establishing a more comprehensive and refined understanding ability. During
the training process, the encoders for each modality are frozen, while the LLM and adapters are trained.
Training Dataset The training data used in this stage includes the spatial-temporal representation mentioned in Section 3.1.4. To address the scarcity of fine-grained multi-modal data, we construct a multi-modal dataset specifically designed for this stage. The dataset is primarily obtained by converting publicly available datasets. As depicted in the left part of Figure 2, task descriptions are provided to GPT-3.5 to generate a taskspecific question pool. For each data sample, a question is randomly selected from the pool, and templates are used to convert the sample’s format, resulting in a single-turn conversation. For the image modality, we utilize visual grounding datasets such as RefCOCO (Kazemzadeh et al., 2014), RefCOCO+ (Kazemzadeh et al., 2014), RefCOCOg (Mao et al., 2016) and Visual Genome (Krishna et al., 2017b) to construct the datasets. For the video modality, video temporal grounding datasets such as DiDeMo (Anne Hendricks et al., 2017), HiREST (Zala et al., 2023) are utilized for finegrained alignment. Regarding the sound localization task, we employ the VGGSS (Chen et al., 2021) dataset for training. All these datasets are transformed into single-turn conversation format following the aforementioned pipeline for training.

heading: 3.2.3 Multi-granularity Instruction Tuning
text: After the training in the first two stages, the model has acquired a strong understanding and grounding capability. This stage aims to enable the model to generate responses that better align with human preferences and improve multi-modal interactions. We train the model using instruction-tuning datasets at different granularities. Similar to the second stage, the encoders for each modality are frozen, while the LLM and adapters are trained.
Training Dataset The data utilized in this stage consists of high-quality fine-grained instructiontuning dataset we construct and public instructiontuning dataset. As illustrated in the right part of Figure 2, we select a subset of public datasets for human annotation to create in-context examples. It assists in guiding GPT-3.5 to follow similar patterns when generating instruction-tuning dataset. Subsequently, task-specific system prompts and randomly selected examples are input to GPT-3.5 to generate single/multi-turn conversations. For the image modality, we construct fine-grained
datasets using the Flickr30K Entities (Plummer et al., 2015) dataset, including detailed descriptions and conversations. To enhance the model’s fine-grained reasoning capability, we utilize the VCR (Zellers et al., 2019) dataset to construct a reasoning dataset with coordinates. For the video modality, we constructed datasets with temporal information by incorporating datasets from various video tasks such as DiDeMo (Anne Hendricks et al., 2017) and Activitynet Captions (Krishna et al., 2017a), along with other relevant sources. The public instruction-tuning datasets we use include LLaVA-v1.5-mix665k, Valley-Instruct-73k, Videochat-Instruct-11k, and an audio instructiontuning dataset constructed using Clotho (Drossos et al., 2020) dataset. For more details about the datasets, please refer to appendix B.
To ensure the quality of the dataset, we carefully filter the data by eliminating samples that do not conform to the desired format or criteria. Specifically, we performed data cleaning on the raw data, which entailed filtering the downloaded data and removing any damaged instances, particularly videos with corrupted content. By ensuring data integrity, we preserved the quality of the dataset. Furthermore, during the data generation process, although we provided contextual examples, there were instances where the generated samples deviated from the desired format outlined in Section 3.1.4. For example, there were cases where the parentheses in coordinate representations did not match. To address this issue, we employed a set of predefined regular expression patterns to filter out samples that did not conform to the specified format.
During training, in order to prevent catastrophic forgetting in subsequent training stages, we adopt a sampling strategy that incorporates training data from previous stages. The training process employs a consistent training objective as follows:
L(θ) =− E(x,y)∼Dcurrent [log p(y|x)] − α · E(x,y)∼Dprevious [log p(y|x)],
where Dcurrent denotes the dataset in current training stage, Dprevious denotes the dataset in previous training stage and α denotes the sampling rate. In the first training stage, α is set to 0.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: We employ Vicuna-v1.5 (Chiang et al., 2023) as the language model. Each training stage lasts for
one epoch. During the training process, all images were padded to a square shape and resized to a resolution of 336× 336. For each video, 64 frames were sampled, and for each audio, three 2-second segments were sampled and processed. For more details on the hyper-parameter settings, please refer to the appendix A.

heading: 4.2 Quantitative Evaluation
text: We conducted extensive experiments for the effectiveness of GroudingGPT in multi-modal grounding and understanding tasks.

heading: 4.2.1 Multi-modal Grounding
text: In this section, we demonstrate that our model achieves impressive fine-grained understanding of multi-modal inputs on grounding tasks.
Image Grounding In the image grounding task, the model takes an image and textual description of a region within the image as input, and outputs the text along with explicit position information, such as bounding boxes or masks. To assess the image grounding capability of the Ground-
ingGPT model, we conduct experiments on the widely used Reference Expression Understanding (REC) task. The REC task requires the model to locate the bounding box corresponding to a given text reference expression. Our experiments involve three datasets: RefCOCO, RefCOCO+ and RefCOCOg. The baselines used for comparing include previous end-to-end multi-modal models UNITER (Chen et al., 2020), MDETR (Kamath et al., 2021), UniTAB (Yang et al., 2022), and the LLM-based multi-modal grounding models KOSMOS-2, Shikra, NExT-Chat and Ferret. For GroundingGPT model, we use a unified prompt like "Output the coordinate of <exp>", where "<exp>" represents the reference expression. The results on the REC task is presented in Table 2. GroundingGPT demonstrates remarkable performance across multiple datasets and performs comparably to specialized fine-tuned models or MLLMs that incorporate additional image region perception modules.
Video Grounding Video grounding focuses on identifying and localizing specific moments or events in the video based on given descriptions. To evaluate the video grounding capability of GroundingGPT, we conduct experiments on the temporal video grounding task. The temporal video grounding task requires models to pinpoint and highlight temporal boundaries within videos, that corresponds accurately to a specified text query.For the task, we employed datasets from CharadesSTA (Gao et al., 2017). The predicted time segments are compared with the corresponding ground truth time segments to calculate the IoU. The evaluation metric used is "R@1, IoU = m", which mea-
sures the percentage of correctly retrieved moments with an IoU greater than m. We set the values of m as 0.5, 0.7 to assess different levels of accuracy. As shown in Table 3, GroundingGPT exhibits excellent performance in temporal video grounding task compared to previous video MLLMs, which primarily focuses on entire video understanding.

heading: 4.2.2 Multi-modal Understanding
text: We validate that GroundingGPT can maintain or improve the multi-modal understanding ability by introducing grounding tasks. Especially, it can effectively suppress object hallucination.
Image Understanding We evaluate the image understanding capability of GroundingGPT on five question-answering benchmarks and four recent proposed benchmarks specifically designed for vision instruction tuning. These benchmarks provide a comprehensive assessment of the model’s capabilities using diverse evaluation metrics. The experimental results presented in Table 4 demonstrate that GroundingGPT achieves state-of-the-art performance on six benchmarks and remains highly competitive on other three benchmarks. Additionally, GroundingGPT exhibits advanced capabilities compared to larger-scale grounding MLLMs, such
as Shikra-13B.
Video Understanding In Table 5, we provide a quantitative assessment of the video question answering capabilities of MLLMs on three datasets: MSVD-QA (Chen and Dolan, 2011), MSRVTTQA (Xu et al., 2016) and ActivityNet-QA (Yu et al., 2019). GroundingGPT consistently outperforms other baselines, demonstrating its advanced video understanding capabilities. Notably, GroundingGPT surpasses the powerful baseline of Valley by 2.4%, 6.9% and 1.8% on MSVD-QA, MSRVTTQA and ActivityNet-QA, respectively.
Object Hallucination We conduct an evaluation of MLLMs regarding their object hallucination in Table 6. GroundingGPT achieves the highest performance across all three sampling subsets. Notably, GroundingGPT-7B outperforms larger models such as InstructBLIP-13B, on the challenging Adversarial subset, GroundingGPT exhibits 14.07% increase in accuracy and 8.18% increase in f1-score, while experiencing a 27.60% decrease in the "Yes" metric. Additionally, on the "unanswerable" subset of VisWiz benchmark, GroundingGPT significantly improves from 67.8% to 84.0% compared to LLaVA-1.5. This improvement reveals
that our model not only acquires a high-level semantic understanding of the overall image but also develops the ability to comprehend local details. This comprehensive understanding of the image enables the model to effectively suppress the occurrence of hallucinations.

heading: 4.2.3 Qualitative Results
text: We provide qualitative results to demonstrate the multi-modal understanding and grounding capabilities of our model. Figure 3 showcases examples illustrating the performance of GroundingGPT in multi-modal grounding tasks. More examples are available in appendix E. These results indicate that GroundingGPT excels in fine-grained multi-modal grounding tasks while maintaining a comprehensive understanding of multi-modal inputs.

heading: 4.3 Ablation Study
text: To validate the effectiveness of our approach, we conducted experiments on the REC task to assess the impact of training strategy, model architecture, and size on the results.
Training Strategy As shown in Table 7, it is evident that including fine-grained training data in the first stage results in a decline in performance.
This can be attributed to the model’s limited understanding of the images at this early stage. The introduction of fine-grained data during training may introduce interference and hinder the model’s learning. This finding further validates the effectiveness of our coarse-to-fine training strategy.
Model Architecture and Size As shown in Table 8, the top two rows demonstrates that replacing the linear layer with an MLP in the adapter leads to performance enhancement. This improvement can be attributed to the preservation of more comprehensive image information and the improved mapping of image embeddings to the LLM space. Besides, increasing the LLM size leads to an improvement. This can be attributed to the fact that larger language model possess richer knowledge and stronger modeling capabilities.

heading: 5 Conclusion
text: In this paper, we introduce GroundingGPT, a unified end-to-end multi-modal grounding model. To the best of our knowledge, this is the first multimodal large language model capable of performing multi-modal grounding and understanding tasks. We adopt a three-stage coarse-to-fine training strategy, accompanied by the construction of stagespecific training datasets, to effectively train the model. Our model demonstrates remarkable perfor-
mance in multi-modal grounding and understanding tasks. Extensive experiments conducted on a wide range of MLLM benchmarks confirm the effectiveness and generality of our model. To foster further advancements in this field, we make our model, code, and dataset openly accessible.

heading: 6 Limitations
text: Sampling Strategy Due to computational memory constraints, GroundingGPT adopts a sampling approach when processing videos and audios. However, this method inevitably results in some loss of crucial information, especially when dealing with longer videos. One future research direction is to explore better modeling approaches for longer videos and minimize information loss.
Cross-modal Inputs At present, the majority of the training data primarily consists of single-modal inputs. However, further exploration is needed to address the challenges posed by multi-modal inputs. In the future, we plan to investigate methods for accomplishing grounding tasks in the context of simultaneous multi-modal inputs. For instance, we aim to simultaneously perform spatial and temporal grounding on input videos. Additionally, we will annotate such data to foster advancements in this field.
Grounding Ability Despite achieving promising results in multi-modal grounding tasks, GroundingGPT currently lacks the capability to output more fine-grained grounding results such as segmentation masks. In future work, we plan to expand the grounding tasks to support a broader range of grounding requirements.

heading: B Training Dataset Details
text: In Table 10, we provide a comprehensive list of the datasets used in constructing our training dataset. This includes the data utilized in all three stages. It should be noted that a significant portion of the data needs to be constructed in the desired format using publicly available data. Please refer to the section 3.2 for specific guidance on this matter. Additionally, we provide the statistical results of the training data we constructed in Table 11, including the number of samples for each stage, whether it is multi-turn dialogue data, average number of dialogue turns, average video duration, and average audio duration.

heading: C Dataset Construction Templates
text: Table 12 presents the templates utilized for various tasks during the first two training stages. For the sake of demonstration, we provide three examples of instructions for each task.

heading: D Fine-grained Instruction-tuning Dataset Generation Prompts
text: As shown in section 3.2.3, we use GPT-3.5 to generate the instruction-tuning dataset. For the image modality, in Figure 4, we provie the prompt we used to generate the detailed description dataset. In Figure 5, we provie the prompt we used to generate the conversation dataset. For the video modality, we provie the prompt we used to generate the video grounding instruction-tuning dataset in Figure 6.

heading: E More Visualization
text: To demonstrate the performance of GroundingGPT in multi-modal grounding and understanding tasks, we present more visualizations in this section. Figure 7 and Figure 8 showcase the capability of the GroundingGPT model in multi-modal grounding tasks. Figure 9, Figure 10 and Figure 11 present the capability of GroundingGPT model in multi-modal understanding tasks.

heading: Future Work
text: One future research direction is to explore better modeling approaches for longer videos and minimize information loss. Cross-modal Inputs At present, the majority of the training data primarily consists of single-modal inputs. However, further exploration is needed to address the challenges posed by multi-modal inputs. In the future, we plan to investigate methods for accomplishing grounding tasks in the context of simultaneous multi-modal inputs. For instance, we aim to simultaneously perform spatial and temporal grounding on input videos. Additionally, we will annotate such data to foster advancements in this field. Grounding Ability Despite achieving promising results in multi-modal grounding tasks, GroundingGPT currently lacks the capability to output more fine-grained grounding results such as segmentation masks. In future work, we plan to expand the grounding tasks to support a broader range of grounding requirements