Recognizing Everything from All Modalities at Once: Grounded Multimodal Universal Information Extraction (2024)

Abstract: In the field of information extraction (IE), tasks across a wide range of modalities and their combinations have been traditionally studied in isolation, leaving a gap in deeply recognizing and analyzing cross-modal information. To address this, this work for the first time introduces the concept of grounded Multimodal Universal Information Extraction (MUIE), providing a unified task framework to analyze any IE tasks over various modalities, along with their fine-grained groundings. To tackle MUIE, we tailor a multimodal large language model (MLLM), REAMO, capable of extracting and grounding information from all modalities, i.e., ‘recognizing everything from all modalities at once’. REAMO is updated via varied tuning strategies, equipping it with powerful capabilities for information recognition and fine-grained multimodal grounding. To address the absence of a suitable benchmark for grounded MUIE, we curate a high-quality, diverse, and challenging test set, which encompasses IE tasks across 9 common modality combinations with the corresponding multimodal groundings. The extensive comparison of REAMO with existing MLLMs integrated into pipeline approaches demonstrates its advantages across all evaluation dimensions, establishing a strong benchmark for the follow-up research. Our resources are publicly released at https://haofei.vip/MUIE.

heading: 1 Introduction
text: IE is a pivotal topic (Li et al., 2022a), encompassing subtasks such as Named Entity Recognition (NER; Nadeau and Sekine, 2007), Relation Extraction (RE; Miwa and Bansal, 2016), and Event Extraction (EE; Ahn, 2006), which plays a crucial role in constructing domain-specific knowledge bases (Bosselut et al., 2019) and in facilitating deep semantic understanding of data (Satyapanich et al.,
*Corresponding author

heading: Text → NER

heading: Text + Image → NER + Image Segmentation

heading: Text + Audio → NER + Speech Segmentation
text: 2020). In reality, a vast amount of information is conveyed through modalities beyond text. Consequently, research in IE has evolved from focusing solely on textual data to embracing various other modalities, leading to the development of multimodal IE (MIE; Liu et al., 2019), e.g., images, videos, and audio. Despite the growing research efforts dedicated to MIE, the exploration in this area remains insufficiently developed. We argue that several critical aspects must be fully considered for future MIE research trends.
Firstly, current studies primarily investigate MIE tasks within individual modalities (or certain modality combinations) (Sun et al., 2021; Chen et al., 2022b). With the existence of several modality categories and diverse definitions for different IE tasks, studying each modality separately to construct specialized MIE models would inevitably
14498
lead to resource wastage and inefficiency. In realworld applications, there is a constant need for building unified systems with “one-for-all” robust generalizability for faster practical deployment. In light of the recent success of textual universal IE (UIE; Lu et al., 2022; Fei et al., 2022), MIE unification should also be promising. Second, the majority of existing studies (Zhang et al., 2017) exhibit a bias toward text-centric IE outputs, necessitating the decoding of detailed textual IE labels and inherently prioritizing text. While they often treat other modalities as auxiliaries and do not produce outputs for them, this practice does not align with reality, because all modalities can equally carry important information. For example, even infants who have not yet learned to speak can learn about entities from vision. Thus, each modality should be treated equally, and detect fine-grained information from all given modalities. Last, most current MIE (Zheng et al., 2021) involving multiple modalities (e.g., Image&Text) tends to extract the modalityaligned part of the information under the assumption that different modalities associate with each other. However, in practical scenarios, the information carried by different modalities can be either shared (Li et al., 2022b), or unrelated (Wu et al., 2023a). This suggests that information should be flexibly recognized from any modality sources.
In response to these challenges, this paper is dedicated to pioneering a novel task, grounded Multimodal Universal Information Extraction (MUIE). As illustrated in Fig. 1, MUIE aims to unify the modeling of various IE tasks (e.g., NER, RE, EE) with any (or combination) inputs across the most common modalities (e.g., text, audio, image, and video), and produce fine-grained multimodally grounded IE results. To solve MUIE, we consider taking advantage of the existing generative LLMs (OpenAI, 2022; Chung et al., 2022; Chiang et al., 2023) with in-context instructions (Dong et al., 2022). We develop a novel multimodal LLM, REAMO, achieving “Recognizing Everything from All Modalities at Once”. REAMO not only outputs all possible textual IE labels but also identifies corresponding groundings across other modalities: 1) statically, by segmenting visual objects and audio speeches, and 2) dynamically, by tracking textual or vocal events in videos. Technically, REAMO employs a Vicuna (Chiang et al., 2023) LLM as its core semantic reasoner, utilizing ImageBind (Girdhar et al., 2023) as a multimodal encoder to project image, video and audio inputs into
LLM-understandable signals. At the decoding end, we integrate the SEEM (Zou et al., 2023) for visual grounding&tracking, and the SHAS (Tsiamas et al., 2022) for audio segmentation, where the messages are passing from LLM to decoders through structured meta-response effectively. Given input multimodal information, REAMO is able to output UIE label tokens as well as fine-grained groundings recurrently.
We then design learning objectives to tune REAMO to endow it with robust MUIE and crossmodal grounding capabilities. First, we repurpose existing textual UIE annotation into instruction format, and use it to tune the backbone LLM for activating the UIE ability. Then, we perform both coarse-grained instance-level and fine-grained grounding-aware cross-modal alignment learning, enhancing the REAMO’s capability in fine-grained multimodal semantic understanding. Furthermore, we instruction-tune REAMO on specific corpus, to build its working behavior of generating structured meta-response texts.
In response to the absence of standard evaluation data for grounded MUIE, we further introduce an evaluation benchmark, where we annotate a highquality testing set of 3,000 instances covering NER, RE, and EE tasks under 9 common modality combinations. The data further advances by annotating both modality-shared/-specific content to simulate aligned and misaligned modality scenarios. Extensive zero-shot experiments on these benchmarks demonstrate that REAMO shows strong performance over existing MLLMs with respect to IE tasks and multimodal grounding.
Overall, we make three key contributions: • To our knowledge, this is the first to propose a
grounded MUIE setting, unifying all IE tasks across modalities, further with fine-grained multimodally grounded targets. • We introduce an MLLM for the task, REAMO, excelling in MUIE prediction and achieving cross-modal grounding of static objects and dynamic events. • We contribute a high-quality, diverse, and challenging dataset, setting an evaluation benchmark for follow-up grounded MUIE research.

heading: 2 Related Works
text: IE (Lample et al., 2016; Wei et al., 2020; Fei et al., 2021; Li et al., 2021; Wang et al., 2022; Cao et al., 2022) has long been a significant research direction, consistently attracting substantial interest and focus
for decades. As the world contains various modalities of information, MIE has been consequently introduced, e.g., multimodal NER (Sun et al., 2021; Yu et al., 2023; Wang et al., 2023a), multimodal RE (Chen et al., 2022b), and multimodal EE (Li et al., 2020). However, the majority of existing well-established benchmarks for MIE still predominantly focus on texts, supplemented with images (Zhang et al., 2017; Wu et al., 2023a).
Historically, IE research has treated different tasks as separate studies for a long (Sun et al., 2021; Chen et al., 2022b; Liu et al., 2024; Zhang et al., 2024). Recently, Lu et al. (2022) pioneer UIE, proposing to unify all IE tasks under a single generative model to produce all IE results, significantly reducing the maintenance cost for individual tasks. With the latest rapid development of LLMs, the latest advancements have utilized LLMs with in-context prompting for UIE, achieving promising zero-shot performance. Similarly, the swift progress of MLLMs (Fei et al., 2023) should also ignite hope for MUIE. Yet research in MUIE remains under-explored.
To our knowledge, the work most closely related to this paper is by Sun et al. (2023), who leverage existing MLLMs to unify various MIE tasks through a two-stage process of span extraction and classification in a multimodal QA format. Yet, we identify clear limitations in their approach that fall short of achieving comprehensive MIE unification. Firstly, while their work only considers text and image modalities, lacking comprehensiveness, we broadly cover the four most common modalities. Secondly, beyond addressing these limitations, we also introduce a novel MLLM tailored for grounded
MUIE and contribute a new benchmark dataset for MUIE research. These efforts aim to pioneer the next stage of MUIE research.

heading: 3 Task Definition: Grounded Multimodal
text: Universal Information Extraction
We now give a formal definition of grounded MUIE. Suppose the inputs are any of a text T , an image I , an audio A, a video V , or their combination. NER task seeks to predict all possible textual labels of entities {Ener}, with pre-defined entity types Cner ∈ Cner (e.g., person, location and organization), where each E may correspond to a span within T , or visual region within I , or a speech segment within A. We denote the visual grounding mask as Mimg and the speech segment as Maud. RE task aims to first identify all possible entities {Ere} following the NER step, and then determine a pre-defined relation label RRE ∈ RRE for two entities < EREi , E RE j > that should be paired. Also ERE should correspond to T , I , or A, as in NER. EE task detects all possible structured event records that consist of event trigger Eet, event type Cet ∈ Cet, event argument Eea and event argument role Cer ∈ Cer. Here Eet and Eea correspond to a continuous span within T or a speech segment within A. Also Eea might refer to the visual region within I , or the temporal dynamic tracklet in video V (i.e., object tracking). We denote the video tracking mask as Mvid. Cet and Cer are pre-defined label sets.
Following Wang et al. (2023b), we employ incontext learning (ICL; Dong et al., 2022) to prompt LLMs for MUIE, with the specific task executed depending on the user’s intention. In the bottom
left of Fig. 2 we simply illustrate the ICL prompt.

heading: 4 Our Proposed Model

heading: 4.1 MLLM Framework of REAMO
text: Fig. 2 presents a schematic overview of REAMO MLLM, which consists of three main parts: multimodal encoder, LLM, and decoder for UIE prediction & multimodal grounding.
Multimodal Encoding. While REAMO takes four types of modality sources, except texts that are directly input to LLM, the image, audio, and video inputs should be encoded. Following Wu et al. (2023b); Fei et al., 2024b), we leverage the highperformance ImageBind (Girdhar et al., 2023) as a unified multimodal encoder. Then, via a projection layer, different input representations are aligned into language-like embeddings that are understandable to the LLM.
LLM Reasoner. An LLM serves as the center unit of REAMO for content semantics understanding and reasoning. Specifically, we follow the most common practice, using the Vicuna-v1.5 (Chiang et al., 2023) as the backbone LLM.
▶ Input:
<Text to analyze>
<Non-text Modality>
| | <Prompt> Please extract all entity words · · ·
▶ meta-response:
<UIE> (Trump, person) (Merkel, person) · · · <Module> Image Segmenter <Instruction> Segmentation: ‘A person’
MUIE Decoding with Grounding. The central LLM takes on the crucial role of decision-making. Based on the input prompt, LLM will produce textual responses, containing the UIE task results as well as the meta-response that will be used to call the downstream modules to generate fine-grained multimodal groundings. Here, consider integrating the existing high-performance SEEM model (Zou et al., 2023) for image segmentation and
video tracking (Li et al., 2024; Wu et al., 2024; Fei et al., 2024a), and the SHAS model (Tsiamas et al., 2022) for audio segmentation. Specifically, the meta-response includes three parts, as shown above. The corresponding ‘<Module>’ and ‘<Instruction>’ information will be passed to the corresponding modality segmenter(s), to activate them to generate grounding(s).

heading: 4.2 MUIE Fine-tuning for REAMO
text: With REAMO at hand, we now consider fine-tuning it through multiple objectives to enable REAMO with strong MUIE capability. UIE Instruction Tuning. Our initial goal is to equip the system with the fundamental capability for UIE in the text modality. To achieve this, we consider tuning the backbone LLM specifically for UIE. Following the practices of Wang et al. (2023b), we repurpose existing annotation data to form a set of instruction-tuning datasets for UIE. To avoid the huge cost of fully updating the LLM, we leverage the LoRA technique (Hu et al., 2022), achieving the goal by tuning only a small subset of parameters, without altering the overall LLM. Multimodal Alignment Learning. REAMO integrates the ImageBind encoder to enable the LLM to comprehend basic multimodal signals. Following this, we proceed with multimodal alignment learning. We consider using the language-centric LLM as the core, requiring only the alignment of other modalities to text. We mainly utilize the vast array of available ‘X-caption’ pair data (‘X’ stands for image, audio, or video). We adopt an ‘X-to-text’ generation, where the input is ‘X’, and the LLM generates the corresponding caption. During this process, we fix the ImageBind and LLM while only updating the projection layer. Fine-grained Cross-modal Grounding-aware Tuning. Above we merely enable REAMO with a coarse-grained multimodal understanding. Yet our goal is to attain a subtle modality comprehension. Thus, we further engage in fine-grained multimodal grounding. Our primary approach revolves around utilizing existing ‘X-to-text’ phrase grounding data, e.g., MS-COCO (Lin et al., 2014), where REAMO’s input consists of textual phrases and grounded regional modality features, and then prompt LLM to determine their match.

heading: Invocation-based Meta-response Tuning. To
text: teach the LLM to produce the correct invocation meta responses, we need to create data specifically
designed for instruction tuning. Technically, we make use of the existing annotated datasets for various vision tasks included in this work. For each task under specific different user input scenarios, with the corresponding data, we design various template dialogue-format examples. Based on these examples we then prompt the GPT-4 to generate more samples under various topics and enriched scenarios.

heading: 5 A Benchmark for Grounded MUIE
text: To evaluate the performance of our grounded MUIE system, we develop a benchmark testing set.

heading: 5.1 Data Source
text: We select 9 existing datasets from different modalities (or combinations thereof) for IE/MIE tasks. Table 1 summarizes these datasets of the raw sources. We then process these datasets, such as Text↔Speech, to create 6 new datasets under new multimodal (combination) scenarios. Before annotation, we carefully select 200 instances from their corresponding testing sets, ensuring each instance contained as much IE information as possible.
• PASCAL-C1: is an object detection dataset for evaluating the robustness. • VRD2, Visual Relationship Dataset: is designed to assess the precision of detecting interactions among pairs of objects. Comprising 5,000 images, the dataset encompasses 100 object categories and 70 predicates. • imSitu3: serves as a resource for facilitating situation recognition, a task concerned with generating a succinct depiction of the scenario
1https://github.com/bethgelab/ robust-detection-benchmark
2https://cs.stanford.edu/people/ranjaykrishna/ vrd/
3http://imsitu.org/
portrayed in images. This includes (1) the main activity, (2) the participating actors, objects, substances, and locations and most importantly (3) the roles these participants play in the activity. • ACE20054: is extensively used in information extraction. It comprises annotated news articles in English, covering a diverse range of topics and events, with annotations including named entities, relations, and events. • ReTACRED5: is a revised version of TACRED for relation detection, containing over 91k sentences spread across 40 relations. • VidSitu6: is a large-scale dataset containing diverse 10-second videos from movies depicting complex situations (a collection of related events). Events in the video are richly annotated at 2-second intervals with verbs, semantic-roles, entity co-references, and event relations. • Twt177, Twitter-17: is a publicly available Twitter dataset for NER. It encompasses 723 test tweets, with annotations covering four entity types, namely, person, location, organization, miscellaneous. • MNRE8, Multimodal Neural Relation Extraction: comprises 15,484 samples and 9,201 accompanying images across 23 distinct relation categories, partitioning into training, development, and testing subsets, consisting of 12,247, 1,624, and 1,614 samples respectively. • M2E2 9: is comprised of 245 multimedia news articles meticulously annotated with
4http://projects.ldc.upenn.edu/ace/ 5https://github.com/gstoica27/Re-TACRED 6https://vidsitu.org/ 7https://github.com/jefferyYu/UMT 8https://github.com/thecharm/MNREp 9https://github.com/limanling/m2e2
events and their corresponding arguments.
Pre-processing And Modality Translation Before we begin the annotation work for grounding, we enrich the types of modality combinations. This is essential because most MIE datasets currently focus on the combination of images and text. However, we aim to simulate a variety of common modality combinations that could occur in realworld scenarios. To achieve this, we transform and preprocess existing datasets. Specifically, our approach involves cross-modal parallel translation to generate data in another modality. We preprocess the following datasets:
• VidSitu-Aud: we start by captioning videos from the VidSitu dataset, then use the given video event annotations combined with the captions to have ChatGPT generate a coherent sentence, serving as the paired text for each piece of data. • VidSitu-Aud: Based on VidSitu-Txt, we convert each sentence into speech using TextTo-Speech (TTS) tools. We employ state-ofthe-art open-source TTS models: Bark10 and Edge-TTS11. • ACE-Aud: we take original textual sentences from the ACE dataset and record speech using TTS technology. • MNRE-Aud: we record speech for sentences from the MNRE using TTS. • Twt17-Aud: Similarly, we record speech for sentences from the Twt17 dataset using TTS.
However, we emphasize that such parallel data generation can only produce modality-aligned content. To create diverse content, we plan to introduce randomness by adding noise to some instances. For example, we might alter parts of the original text before synthesizing the speech with TTS. Considering the aim to only annotate a test set for the sake of conserving labor and reducing costs, we plan to select subsets from the test sets of various original datasets obtained. We aim to select 200 from each. Our selection criterion focuses on ensuring a high quantity of entities and objects in the content, and the final labels should cover a rich vocabulary. Given there are 15 combinations of modalities and tasks (including those augmented through our post-processing), we will have a total of 3,000 data entries of grounded MUIE test instances. Next, we revisit the annotation information, specifically
10https://github.com/suno-ai/bark 11https://github.com/rany2/edge-tts
re-annotating instances from the combined modality datasets where cross-modal content is not fully aligned. This ensures that the dataset covers both modality-shared and modality-specific instances.

heading: 6 Experiments

heading: 6.1 Settings
text: We measure the system performance by following most practices of end-to-end UIE: F1 of entity span with type for NER, F1 of all subject&object entities and their relation label for RE. For EE, we consider event trigger (ET) F1 including both trigger and event type, and event argument F1 including both arguments with role types. W.r.t. the multimodal grounding, for both image and audio segmentation, we consider the mean Intersection over Union (mIoU); for video segmentation, we use the average Jaccard (J).
The encoder projection is a linear layer with a hidden size of 4,096. As no prior method is designed for grounded MUIE, we implement pipeline systems. Specifically, we first employ an MLLM to do UIE on multimodal input. And then we pass the raw multimodal source and the necessary UIE label to the SEEM or SHAS model for fine-grained grounding of image, video and audio. We consider the following existing well-exposed MLLMs. For image-related UIE: InstructBLIP (Dai et al., 2023), LLaVA (Liu et al., 2023), and also MiniGPTv2 (Chen et al., 2023) that can output image segmentation end-to-end. For or video-related UIE: VideoChat (Li et al., 2023), Video-LLaVA (Lin et al., 2023). SpeechGPT (Zhang et al., 2023a) for audio-related UIE. Video-LLaMA (Zhang et al., 2023b) supporting video+audio; NExT-GPT (Wu et al., 2023b) supporting all four modalities. All these systems take the 7B LLM, unless otherwise specified. For fairness, all baselines are further tuned using the same UIE instruction-tuning data as ours. All system takes zero-shot inference, without tuning on in-house datasets.

heading: 6.2 Zero-shot Results on Image-related MUIE
text: Table 2 presents the results of different models on our MUIE dataset under both text+image and pure image conditions. From the data, we observe that overall, the end-to-end approach (MiniGPT-v2) outperforms pipeline methods. Besides, our system demonstrates a clear and consistent advantage consistently across all subtasks.

heading: 6.3 Zero-shot Results on Audio-related MUIE
text: In Table 3, we present the performance of various models under text+audio and pure audio input, respectively. The trends observed are similar to those seen in the previous table: 1) End-to-end approaches (SpeechGPT) demonstrate stronger performance compared to the pipeline method (NExTGPT+SHAS), effectively mitigating the issue of error propagation. 2) Our REAMO consistently outperforms others across all subtasks and scenarios.

heading: 6.4 Zero-shot Results on Video-related MUIE
text: In Table 4, we present the final set of results for EE task based on text+video and pure video input. The
overall trend observed here again aligns with that of the previous tables, with our model achieving the strongest performance. Following, we can confirm that for single modalities (such as image, audio, video), without the assistance of textual modality, the effectiveness of MUIE significantly diminishes.

heading: 6.5 Results on Modality-Compound MUIE
text: Finally, we present the performance of our model across a broader range of complex mixed-modality scenarios for MUIE. As shown in Table 5, our REAMO consistently delivers the best results than the baseline solutions. This becomes even more pronounced in complex mixed-modality contexts.
Give rationale behind the extracted results. Think it step by step
In the image, a man holding a microphone is Bud Collins, conducting an interview. Standing beside him is Rafael Nadal, who is being interviewed. T he reason for the interview is that Rafael Nadal is being awarded the French Open title, signifying his triumph in the tournament. Bud Collins is interviewing him to capture his thoughts and reactions regarding this achievement.

heading: 6.6 G p between Modal ty-Shared vs. Modality-Spe ific MUIE
text: Previously, we mentioned that in the MUIE context, different models may encounter both aligned shared information and misalignments between modalities. Now, we have categorized cases in the data as either Modality-Shared or ModalitySpecific and compared the performance of various models under these two scenarios. As illustrated in Fig. 3, REAMO outperforms all baseline models in addressing challenges arising from misalignments between modalities.

heading: 6.7 Influence of Entity/Object Numbers
text: Fig. 4 further illustrates the impact of the number of entities (or objects) in an instance on the performance of the MUIE system. It is evident that our REAMO system maintains commendable
Extract all the possible events in the video, and track the argument mentions correspondingly. Each event associated with an event type must have a trigger verb. If possible, please give detailed arguments for each event.
What would happen if the man kept driving forward? Please think it based on commonsense.
If the man kept driving forward, his car would likely get stuck in the mud pit, eventually getting completely submerged. Due to the stickiness of the mud, the wheels might sink deep into the mud, rendering the vehicle immobile. Additionally, the deep mud could cause damage to the wheels, and the car's body could be covered in mud, making it extremely difficult to clean and repair.
Based on the video content, there are two possible event in the video: (Type: driving, Trigger: drive, Agent: man, Target: car, Direction: forward, Scene: mire) (Type: driving, Trigger: drive, Agent: man, Target: car, Direction: backward, Scene: mire)
Figure 7: Qualitative result C on MUIE (EE) with commonsense-aware cognitive reasoning.
performance in extracting an increasing number of objects across T+I/T+A/T+V scenarios, both in terms of MUIE results and fine-grained grounding, being clearly superior to baseline pipeline systems.

heading: 6.8 Case Study
text: Then, we provide visualizations of case studies, through which we aim to offer a more intuitive and comprehensive demonstration of our MUIE system. Fig. 5, 6, and 7 each displays an example for NER, RE, and EE tasks, respectively, where REAMO yields correct answers. In case A, our system demonstrates how it can engage in thought processes and flexibly determine MUIE labels from different modalities. In case B, the model can provide accurate relation extractions and precise segmentation results that correspond to the entities in the input text. In case C, most impressively, it manages to link semantic information to factual commonsense understanding based on accurate fine-grained cross-modal grounding, and correctly answer the question.

heading: 6.9 Error Analysis
text: Finally, we delve into our model itself, and analyze the possible shortcomings for shedding light on future explorations. Via our experiments, we here summarize several error types:
• Repetition of Extracted Content: When text and information from other modalities are not strictly consistent, our method may output different entity names, arguments, or relationships. However, upon integrating information from different modalities, they should corre-
spond to the same entity names, arguments, or relationships. • Incomplete Information Extraction: The outcomes of information extraction are incomplete, such as incomplete named entity recognition, failure to identify relations involving in-depth reasoning, or incomplete identification of event arguments. • Incorrect Grounding Match: The entity or arguments do not always match with the grounding results. For instance, when the text mentions ‘Obama’ and ‘Trump’ and the image depicts both individuals, the image object segmenter fails to ascertain who is ‘Obama’ and ‘Trump’, resulting in an erroneous grounding match. • Miss-grounding: Our model may output entities or arguments without successfully grounding the corresponding regions in the respective image, video, or audio. • Over-grounding: The model may generate multiple instructions and perform grounding in the image, video, or audio, yet no corresponding regions actually exist in the visual or auditory content. • Error Propagation: Since our system operates as a pipeline process, where a metaresponse is first produced and then used to invoke functional modules, this sequence introduces error propagation. If there is an issue with the content of the meta-response, the outcomes from the subsequent modules will be incorrect. To address this, we need to develop more advanced end-to-end MLLMs.

heading: 7 Conclusion
text: This paper introduces a novel task of multimodal information extraction setting: grounded Multimodal Universal Information Extraction (MUIE). First, MUIE definition unifies all IE tasks across various modalities, including text, audio, images, and video, with fine-grained multimodally grounded targets. To solve MUIE, we devise REAMO, a novel MLLM that can extract and ground information from all modalities. REAMO is tuned through various strategies to achieve proficiency in recognizing and grounding multimodal information. Further we introduce a high-quality, diverse, and challenging benchmark dataset for evaluating MUIE systems. Experimental results demonstrate REAMO’s stronger performance in extracting and grounding, setting a strong benchmark for the following
grounded MUIE research.

heading: Acknowledgments
text: This work is supported by the CCF-Baidu Open Fund, and the Research on automatic construction of cross-domain knowledge graph with multimodal information (No. U23B2055)

heading: Limitations
text: The limitations of this work primarily arise from the following two aspects. From model perspective, our MLLM, designed for Grounded MUIE, handles four common modalities (text, images, video, audio) and may struggle to integrate new ones without significant retraining. Challenges persist in extracting complex implicit information and in cognitive reasoning, with current capabilities mainly supporting video tracking for event extraction. Further research is needed to enhance and expand tracking for other tasks like NER and RE in videos.
From data perspective, we introduced a dataset solely comprising a test set, limiting in-house training opportunities. Future work will focus on enlarging this dataset to include a training set and expanding annotations to cover more modality combinations for the three IE tasks: NER, RE, and EE.

heading: Ethics Statement
text: The development and application of MUIE systems potentially raise several potential ethical considerations or risks that should be properly treated to ensure responsible research and deployment. Privacy and Consent. Multimodal data in MUIE systems can include sensitive personal information. Ensuring data is collected and used with explicit consent and in compliance with data protection laws is critical. Bias and Fairness. MLLMs in MUIE systems may inherit biases from their training data, potentially causing discriminatory effects. Active efforts are required to mitigate such biases to guarantee fairness across diverse groups. Transparency and Accountability. MUIE systems often lack clear decision-making transparency due to their complexity. Promoting explainable AI with detailed documentation can enhance user and stakeholder trust. Misuse Potential. The ability of MUIE systems to process diverse data can be exploited for harmful purposes like misinformation or unauthorized surveillance. Establishing robust legal and ethical safeguards is essential to prevent misuse.

heading: A Model Prompt Specification
text: Here, we show the prompts for each subtask (i.e., NER, RE, EE) in MUIE:
▶ Prompt for Named Entity Recognition: Please recognize all entity words and categorize them by pre-defined labels in the given text, and outline them in the given image or video or audio correspondingly. The output format should be “(entity1, label1)(entity2, label2)”. If an entity possibly has a counterpart in the given image or video or audio, please generate a token “<concept>” after the entity word, for subsequent cross-modal grounding. Candidate category labels: person, location, organization, country...
Input Text/Image/Video/Audio:
.
▶ Prompt for Relation Extraction: Please extract all relations between named entities, and outline them in the given image or video or audio correspondingly. The output format should be “(subject entity, relation, object entity)”. If an entity possibly has a counterpart in the given image or video or audio, please generate a token “<concept>” after the entity word, for subsequent cross-modal grounding. Candidate relation labels: peer, award, part_of, locate_at, nationality, place_of_residence, member_of...
Input Text/Image/Video/Audio:
.
▶ Prompt for Event Extraction: Extract all the possible events in the video, and track the argument mentions correspondingly. Each event associated with an event type must have a trigger verb. If possible, please give detailed arguments for each event. Candidate event types: Marry, Attack, Injure, Beborn, Meet, Transport, Start-position..., Candidate event argument types: Agent, Target, Direction, Time, Place, Instrument, Organization, Duration... Input Text/Image/Video/Audio:
.

heading: B Specification of MUIE Fine-tuning

heading: B.1 UIE Instruction Tuning
text: In this step, we train only the core LLM. To avoid the significant cost associated with fully updating
the LLM, we employ the LoRA technique, which allows us to achieve our objectives by tuning only a small subset of parameters, thus leaving the overall architecture of the LLM unchanged. The datasets we utilize are sourced from existing IE datasets. We list these datasets in Table 6. These data are converted into an instruction format, following the practices outlined by Wang et al. (2023b).

heading: Task Data Source Amount

heading: B.2 Multimodal Alignment Learning
text: To accomplish the alignment, we adopt an ‘X-totext’ generation task trained on the ‘X-caption’ pair (‘X’ stands for image, audio, or video) data from existing corpus and benchmarks, i.e., given the representation of an ‘X’, to prompt the frozen LLM to generate the corresponding text description. Specifically, we utilize three types of ‘X-caption’ pair data, including: 1) ‘Video-caption’ pair dataset: Webvid-2M (Bain et al., 2021), a large-scale dataset of short videos with textual description sourced from stock footage sites, 2) ‘Image-caption’ pair dataset: CC3M (Sharma et al., 2018), contains over 3 million images accompanied by diverse styles of natural-language descriptions, and 3) ‘Audiocaption’ pair dataset: AudioCaps (Kim et al., 2019), an extensive dataset of approximately 46k audio clips paired with human-written textual descriptions collected via crowdsourcing.

heading: B.3 Fine-grained Cross-modal Grounding-aware Tuning
text: For this step, our focus is on achieving fine-grained, concept-level cross-modal alignment, primarily aligning the other three modalities to the textual modality. Our primary method involves utilizing existing ‘X-to-text’ phrase grounding datasets. REAMO’s input comprises textual phrases and grounded regional modality features, prompting the LLM to determine their match. For finegrained image-to-text alignment, we consider the MS-COCO dataset (Lin et al., 2014). For videoto-text alignment, we turn to the TAO dataset (Dave et al., 2020). For entity-level audio-to-text
alignment, we primarily utilize the dataset for the Speech NER task (Chen et al., 2022a). Since there is a lack of dataset in English speech, we thus use TTS tools to generate the CoNLL 2003 textual NER data into speech NER data.
Invocation-based Meta-response Tuning. For this step, we aim to teach the LLM to produce the correct format of invocation meta-responses. Thus, we need to create data specifically designed for instruction tuning. Technically, we make use of the existing annotated datasets for various vision tasks included in this work. For each task under specific different user input scenarios, with the corresponding data, we design various template dialogue-format examples. Based on these examples we then prompt the GPT-4 to generate more samples under various topics and enriched scenarios. To ensure data diversification, the resulting instruction data includes a rich number of instances (10k), under different simulations of modalities and tasks.

heading: C Grounded MUIE Evaluation
text: Our grounded MUIE evaluation dataset involves predictions for three tasks, including UIE label prediction, multimodal grounding prediction, and cognitive QA task prediction. Here, we provide detailed evaluation metrics for these three subtasks.

heading: C.1 UIE Evaluation Metrics
text: To evaluate textual UIE results of the model, we use span-based offset Micro-F1 as the primary metric.
• For NER task, we follow a span-level evaluation setting, where the entity boundary and entity type must be correctly predicted. • For RE task, a relation triple is correct if the model correctly predicts the boundaries of the subject entity, the object entity, and the entity relation. • For EE task, we report two evaluation metrics: – Event Trigger (ET): an event trigger is
correct if the event type and the trigger word are correctly predicted. – Event Argument (EA): an event argument is correct if its role type and event type match a reference argument mention.

heading: C.2 Modality Grounding Evaluation Metrics
text: For the evaluation of the fine-grained modality grounding accuracy, the key idea is to measure the mean Intersection over Union (mIoU).
Image Segmentation. Let us denote by M̂img = {Mg}Gg=1 the ground truth set of G regions, and Mimg = {Mk}Kk=1 the set of K predictions. Inspired by prior work, if K ̸= G, we employ padding with ∅ to equalize the sizes of both sets, resulting in a final size of P = max(G,K). Then, we find a bipartite matching between these two sets by searching for a permutation of P elements, σ ∈ SP , with the lowest cost:
σ̂ = argminσ∈SP
P∑
i
Lmatch(M̂i,Mσ(i)), (1)
where Lmatch(M̂i,Mσ(i)) is a pairwise matching cost between ground truth Mi and a prediction with index σ(i). We compute this optimal assignment efficiently with the Hungarian algorithm. We define Lmatch(M̂i,Mσ(i)) as Lbce(M̂i,Mσ(i)) + Ldice(M̂i,Mσ(i)). The final IoU of each prediction is:
IoU = Area of Overlap Area of Union
(2)
Based on the IoU scores, we can calculate mIoU metric by referring image segmentation dataset.
Video Tracking. For videos, we compute the Jaccard Index (a.k.a, mIoU score) for each frame via the above calculations, and then average them.
Audio Segmentation. Similarly, the mIoU score for each audio segment is computed to evaluate the quality of speech segmentation results. We measure the 1D span of the extracted segments and the 1D span of gold segments.

heading: Future Work
text: Future work will focus on enlarging this dataset to include a training set and expanding annotations to cover more modality combinations for the three IE tasks: NER, RE, and EE