MEEL: Multi-Modal Event Evolution Learning (2024)

Abstract: Multi-modal Event Reasoning (MMER) endeavors to endow machines with the ability to comprehend intricate event relations across diverse data modalities. MMER is fundamental and underlies a wide broad of applications. Despite extensive instruction fine-tuning, current multi-modal large language models still fall short in such ability. The disparity stems from that existing models are insufficient to capture underlying principles governing event evolution in various scenarios. In this paper, we introduce Multi-Modal Event Evolution Learning (MEEL) to enable the model to grasp the event evolution mechanism yielding advanced MMER ability. Specifically, we commence with the design of event diversification to gather seed events from a rich spectrum of scenarios. Subsequently, we employ ChatGPT to generate evolving graphs for these seed events. We propose an instruction encapsulation process that formulates the evolving graphs into instructiontuning data, aligning the comprehension of event reasoning to humans. Finally, we observe that models trained in this way are still struggling to fully comprehend event evolution. In such a case, we propose the guiding discrimination strategy, in which models are trained to discriminate the improper evolution direction. We collect and curate a benchmark M-EV for MMER. Extensive experiments on M-EV validate the effectiveness of our approach, showcasing competitive performance in open-source multi-modal LLMs. Code and Dataset are available on https://github.com/TZWwww/MEEL.

heading: 1 Introduction
text: Events are instances or occurrences that are the fundamental semantic units. Events are not independent, and they are usually interconnected by the following relations: causality, temporality, and intention. Multi-modal Event Reasoning (MMER) is to comprehend these events and their
**Corresponding authors.
relations in both visual and textual modalities, and finally pave a path to better understanding the true world. MMER is expected to serve as the underpinning for various multi-modal applications, including visual storytelling (Huang et al., 2016), visual event prediction (Huang et al., 2021), event-related VQA (Park et al., 2020), MM knowledge graph construction (Ma et al., 2022), and video generation (Li et al., 2018; Liu et al., 2024). Such intricate tasks require an understanding of the event evolution mechanism across diverse scenarios.
With the deepening of research on multi-modal instruction tuning, Multi-modal large language models (MLLM) have been able to handle various multi-modal tasks effectively (Liu et al., 2023; Zhu et al., 2023; Chen et al., 2023; Dai et al., 2023; Li et al., 2023b). These models master some abilities of MM event reasoning implicitly during training in diversified sorts of tasks. Among all the task categories, the perception tasks such as referring expression comprehension, referring expression generation, and grounded image captioning (Mao et al., 2016; Kazemzadeh et al., 2014; Peng et al., 2023) enable the model to comprehend the entities of the events in the image and text. The cognitive tasks, namely image caption and VQA (Lin et al.,
8912
2014; Goyal et al., 2017), endow the model with the semantic understanding capability of events. However, the models trained by these tasks are unable to perceive event evolution because of the static nature of all modality inputs. Existing visual instruction-tuning methods only consist of questions for few clips of the entire event scenario. As shown in Figure 1, current methods only model the queried events with the green event and ignore the rest of the scenario. They lack a vision of a broad spectrum of other events in the evolving context. Such contextual absence impedes models from learning abundant evolution knowledge resulting in poor performances in MMER.
To address this issue, we propose Multi-Modal Event Evolution Learning (MEEL) for endowing the model to understand the event evolution to enhance the ability of MMER, leading to improved performances on downstream tasks. Specifically, we first design the event scenario diversification to acquire various events from abundant scenarios. Then, we employ ChatGPT to generate the evolving graphs of these seed events. The aim is to use these graphs to train the model to understand the rich knowledge of the evolution of events. To accomplish this goal, we propose the instruction encapsulation process to adapt the evolving graphs into instruction-tuning data to train the model. In this way, the training allows the model to comprehend more event evolutional knowledge of the scenario leading to better performance of MMER. However, allowing the model to learn only the evolving graphs is insufficient. Without acknowledging the incorrect evolving events, the model would improperly forward the process, resulting in the hallucination of event reasoning. To mitigate, we perform the guiding discrimination. The model requires judging the incorrect evolution. We design various negative mining strategies to harvest incorrect events. Then, we train the model to classify the right event. We also adapt the guiding discrimination into instruction tuning. After obtaining all the data, we finetune the LLaVA (Liu et al., 2023) model after its stage-1 pre-taining with LoRA (Hu et al., 2021) to get our model.
To validate the effectiveness of MEEL, we curate a benchmark M-EV2 for Multi-modal EValuation of EVent reasoning. M-EV2 is collected or curated from nine existing datasets covering visual storytelling (Huang et al., 2016), visual event prediction (Huang et al., 2021), and event-related VQA (Yeo et al., 2018; Zhang et al.,
2021a). Overall, M-EV2 is a challenging task demanding the model to be capable of reasoning for diverse inter-event relations, like causality, temporality, and intent. It consists of two reasoning paradigms: close and open reasoning. We conducted extensive experiments on M-EV2 and compare MEEL against some strong MLLM baselines. Our results demonstrate that MEEL does enhance the MMER ability of the model yielding significant improvements in downstream tasks. We conclude our contributions as:
• We propose the Multi-Modal Event Evolution Learning (MEEL). It aims to train the model to comprehend the intricate event evolution of diversified scenarios. Our method may shed light on other MM event reasoning research. • We further design the Guiding Discrimination to guide the evolution and mitigate the hallucinations of MMER. • We collect and curate the M-EV2 benchmark for MMER. M-EV2 covers diversified interevent relations. We conduct extensive experiments on M-EV2 to test the effectiveness of our model. We achieve competitive performance among open-source MLLMs.

heading: 2 Multi-Modal Event Evolution Learning
text: We strive to enhance a multi-modal large language model’s capability in multi-modal event reasoning (MMER) to boost performance on downstream tasks. The key is to enable the model to comprehend event evolution. As shown in Figure 1, current multi-modal SFT data only model the target events with the green event and ignore the rest of the scenario. They lack a vision of a broad spectrum of other events in the evolving context. Such contextual absence impedes models from learning abundant evolution knowledge resulting in poor performances in MMER. The intuitive motivation is to endow the model with the knowledge of all events in the whole scenario.
To do that, we propose Multi-Modal Event Evolution Learning (MEEL). We leverage ChatGPT to obtain the evolution graph via our Event Graph Evolution mechanism, which starts from diversified seed events. The evolving graphs contain the entire event semantics of a whole scenario. Then we transform the evolving graphs into instruction-tuning data to train our model. Note that instruction tuning is one of the feasible ways to learn the knowledge of event-evolving graphs. One can also leverage
other methods such as in-context learning based on our data. We find that only when trained on the instruction-tuning data does the model turn to generate hallucinations. Therefore, we further add Guiding Discrimination loss to require the model to distinguish the correct events.
This section is organized as follows: Section 2.1 details the MMER task. The main purpose of MEEL is to enhance the comprehension of event evolution. We initiate with an event diversification step to generate a diverse mix of seed events of various scenarios (Section 2.2). Then we construct the event-evolving graphs through a novel method named event graph evolution (Section 2.3). Our next objective is to leverage these event-evolving graphs for model instruction tuning training (Section 2.4). Finally, we incorporate a guiding discrimination training strategy to refine evolution pathways and reduce reasoning errors (Section 2.5). MEEL’s comprehensive framework is graphically represented in Figure 2.

heading: 2.1 Multi-Modal Event Reasoning
text: Multi-Modal Event Reasoning (MMER) involves deducing events based on certain inter-event relations across different modalities. Specifically, events as semantic units can be characterized by text, but their semantics are often more richly conveyed through associated images (Zhang et al., 2021b). The pursuit of MMER is to harness these multi-modal inputs to establish various relationships between events (temporal, causal, intentional, etc.), facilitating sophisticated reasoning
processes (Tao et al., 2023b,a; Han et al., 2021). This reasoning underlies a spectrum of downstream tasks (Huang et al., 2016; Park et al., 2020).
We elaborate on the MMER formulation, wherein an event is expressed by a textual sentence E and represented by an image I. Text provides argument structure, such as subject, verb, and object (Doddington et al., 2004), while images contextualize the event with environmental and situational details (Yang et al., 2023; Zellers et al., 2021). MMER can be modeled as inferring a target event E t based on a given relation R: E t = M (E , I,R), R ∈ SR. (1) Here, M denotes the model and SR represents the set of possible inter-event relations. For example, in Figure 1, E is the red event, I is the image, the queried relation R is "cause", the answer E t is the green event. Therefore, the entire data is:
Question: Given the image, what is the cause of "The buildings and facilities were damaged.".
Answer: A hurricane or severe weather event caused significant damage to the buildings.
This question can not be answered only based on the E since there can be many reasons for building damage. Seeing the image, we can reason the damage could be caused by a hurricane. Models require analysis of both E and I to get the answer.

heading: 2.2 Event Diversification
text: Event diversification aims to curate a varied collection of seed events, encompassing multiple types and scenarios for ensuing evolutionary learning. We initiate this process with a corpus of text and
Observing a long-tail distribution in trigger frequency, we only include K events per trigger to establish a balanced seed event set, denoted as SE = {(Ei, Ii)}. The outcome of this event diversification step is more diversified event types and scenarios, thereby broadening our model’s generalization capabilities and strengthening its understanding of varied contexts.

heading: 2.3 Event Graph Evolution
text: For the goal of enhancing the comprehension of event evolution, we utilize the seed events SE to
1https://spacy.io/
Algorithm 1: Event Graph Evolution algorithm.
Input :Seed event E and the caption C, evolving relations RE , evolving steps L.
Output :Event-evolving graph G. 1 G.AddNode(E), Ẽ = [E ] 2 for i← 1 to L do 3 N = [ ] 4 for Ej in Ẽ do 5 if i == 1 then 6 {(Ek,Rk)} = Evolve(Ej , C, SampleRel(RE , 2)) 7 else 8 {(Ek,Rk)} = Evolve(Ej , SampleRel(RE , 2)) 9 end if
10 for Ek,Rk in SampleEvent({(Ek,Rk)}, 2) do 11 G.AddNode(Ek) 12 G.AddEdge(Ej ,Rk, Ek) 13 N.Append(Ek) 14 end for 15 end for 16 Ẽ = N 17 end for 18 return G
construct event-evolving graphs through our designed event graph evolution methodology. Building on insights from prior work where LLMs like ChatGPT2 have demonstrated proficiency in generating coherent event narratives (Gunjal and Durrett, 2023; Li et al., 2023e), we apply a breadth-first search (BFS) strategy using the ChatGPT to expand each seed event (E , I) ∈ SE both forward and backward in event happening time. We show the process of either direction in Algorithm 1.
We introduce the process of forward evolution. Starting from the seed event E , we consider forward-oriented relations such as RE = {Result, After, HasIntention}3. For each iteration, we invoke the ChatGPT to produce events consistent with sampled relations from RE , as described in Equation 1. In the beginning, due to the bias of relying solely on textual events, we incorporate visual information of the seed event. Specifically, while evolving a seed event, we add its image caption to provide contextual details, promoting more accurate evolution. When evolving the intermediate events, we only use just their text. The prompt template for this evolution process is depicted in Figure 3(a). After L iterations, we acquire an event-evolving graph G.
2https://openai.com/ 3Relations are directed from the generated to the queried event, for instance, generating the Result for a given event. HasIntention implies the head event is intended by subjects in the tail event.
Besides, we also consider backward evolution. Our motivation for that is intuitive. We want the model to cognize event evolution in an complete timeline including both directions. Since we always start from an intermediate event in the timeline, we need to perform both forward and backward evolution. To do that, we consider evolving relations RE = {Cause, Before, IsIntention} and remains the other steps the same.
After the both sides evolution, we denote the outputs as the event-evolving graph G which entails the rich evolution mechanism of the event scenario.

heading: 2.4 Instruction Encapsulation
text: To endow the knowledge of the evolving graphs G for model training, we turn to multi-modal instruction-tuning, a technique with proven efficiency in adapting models to human-like comprehension (Zhu et al., 2023; Sun et al., 2023; Li et al., 2023a; Liu et al., 2023; Li et al., 2023b; Dai et al., 2023). Our approach involves transforming the components of G, represented as G = (V,W) with nodes V and edges W, into instruction-tuning data.
For each node Ei ∈ V, we aim to create a datum comprising the seed event Es, its associated image I, the relation Ri, and the event Ei 4. However, directly inferring Ri between nodes Es and Ei is not straightforward if they are non-adjacent. We address this by introducing induction rules that leverage the properties of inter-event relations, as detailed in Table 1. For example, in an evolving graph G, there exists a path from the seed event Es and another event E2: Es⇒[After]⇒E1⇒[Result]⇒E2. According to rule 1 in Table 1: (After)⋆(Result)+(After)⋆ infers Result, where ⋆ denotes there exists zero or more, + means there is at least one. We induce Es⇒[Result]⇒E2. By applying these rules, we derive the indirect relation Ri.
Then we embed all the data with our instructiontuning templates to form an instruction-tuning dataset. To avoid the laborious task of creating manual templates, we employ ChatGPT to generate diverse question templates for each relation type. With 100 templates from ChatGPT, the templates aim to reason about the tail event based on the provided visual and/or textual events in accordance with Equation 1. Considering the possible absence of textual input, we generate two variations for each of the |SR| relations: one with textual input
4We also tried to keep the intermediate nodes between Es and Ei into the training data but found poorer performances.
and one without.
For any given data (Es, I,Ri, Ei), we randomly determine whether to include textual event information. We then match a suitable template to the relation type Ri and encapsulate all the items into our instruction-tuning dataset. An example of an encapsulated datum is illustrated in Figure 3(b).

heading: 2.5 Guiding Discrimination
text: To ensure accuracy during event graph evolution and guide the model away from generating erroneous events, we introduce a guiding discrimination training paradigm. This mechanism is pivotal in preventing the evolution process from producing hallucinations which is similar to DPO (Rafailov et al., 2023). In this paradigm, we task the model with identifying the correct event amongst a set of carefully selected negative events.
E t = M (E , I,R,D), R ∈ SR, (2)
where D is the candidate set consisting of the correct event E t and a few negative events.
The discrimination training is challenging to perform due to the sourcing of these negative events. For which we formulate two negative event acquisition strategies:
Semantic: This strategy requires model to discriminate the semantic similar events. To forge semantically similar negative events, we first compile a pool of all events of the generated graphs. For any positive event E , utilizing Spacy for dependency parsing, we compute the tree edit distance and the word overlap rate between E and each event
in this pool5. Filtering by the preset thresholds for these metrics, we select the top two events that are close to E . This method sharpens the model’s ability to distinguish between events with closely related linguistic structures.
Evolving: This strategy enhances the model’s grasp on the directionality of event evolution. Leveraging the bidirectional nature of our event generation, namely forward and backward directions, we select two negative events from the opposite direction of the positive event’s evolution. These negatives are particularly challenging as they maintain shared arguments within the same scenario but differ in their logical sequence. This practice further refines the model’s reasoning skills for establishing the correct evolution path.
From the total four negative events generated through these strategies, we randomly select two of them. These, alongside the correct event, are then encapsulated into a multiple-choice format. We also create diverse multiple-choice question templates for each relation type via ChatGPT. An example of such a generation prompt and a corresponding template is presented in Figure 3(c).

heading: 2.6 Training
text: After acquiring both MMER and guiding discrimination dataset, we finetune the backbone by combining the MMER loss LR (from Eq.1) and the guiding discrimination loss LD (from Eq.2):
LR = − ∑
(E,I,R) logP (E t|E , I,R),
LD = − ∑
(E,I,R,D) logP (E t|E , I,R,D),
L = LR + LD
(3)

heading: 3 Experiments
text: 3.1 Construction of M-EV2
To comprehensively evaluate the models’ abilities of MMER on diversified inter-event relations, we collect and curate a benchmark M-EV2. It incorporates nine test sets covering event-related visual question answering (VCOPA, VisCa, VisualComet), visual event prediction (IgSEG), and storytelling (VIST). M-EV2 evaluates event relations of causality, temporality, and intent. Besides, M-EV2 also covers two reasoning paradigms that are multiplechoice close reasoning tasks (CLOSE) and open reasoning without candidates (OPEN). We show
5https://github.com/timtadh/zhang-shasha
the statistics of M-EV2 in Table 7. We elaborate on the curation process as follows. VCOPA This is the task of commonsense VQA (Yeo et al., 2018). Given an image I and two candidate options, the task is to select a more plausible cause or effect option. We also transform this dataset into an open reasoning task. We denote the original multiple-choice task as VCOPA-C and the transformed task as VCOPA-O. VisCa This is a dataset of learning contextual causality from the visual and textual signals (Zhang et al., 2021a). The original task is formulated as that given two images as the context and two textual sentence descriptions, models need to determine if the former sentence causes the latter one. We transform it into our VQA task. We keep the image and first sentence and regard the second sentence as the label to generate. We retrieve one negative sentence by the ground truth and form it as a multiple-choice task. We also adapt the multiple-choice task into an open reasoning similar to VCOPA-O. We denote these two tasks as VisCa-C and VisCa-O. VisualComet This is an open commonsense VQA task which is to answer situations before or after (Park et al., 2020). We also retrieve a negative answer to formulate it into a multiple-choice task. We denote these two tasks as VC-O and VC-C. IgSEG This dataset aims to predict future events based on what has happened (Huang et al., 2021). Specifically, given a sequence of sentences in sequential order and the image of what will happen next, the models need to generate a sentence for this image. In addition, we also retrieve one negative event and form it as a multiple-choice task. We denote these two tasks as IgSEG-O and IgSEG-C. VIST It’s the storytelling task which is to generate the next story given the previous story in sentences and an image (Huang et al., 2016).

heading: 3.2 Baselines
text: We compare baselines as LLaVA-Lora (Hu et al., 2021), InstructBLIP (Dai et al., 2023), Otter (Awadalla et al., 2023), MiniGPT-4 (Zhu et al., 2023), MiniGPT-4-v2 (Chen et al., 2023). We show more details in Appendix B.

heading: 3.3 Implementation Settings
text: We use InstructBLIP (Dai et al., 2023) to generate the image captions for event graph evolution. We set the evolution steps as 3 and constructed 14,290 instruction-tuning data. Comprehensive statistics of the dataset are detailed in Table 2.
For our model, we use LLaVA-v1.3 after the first pre-training stage as our backbone (Liu et al., 2023) and train with Lora (Hu et al., 2021). LLaVALora-v1.3-7B is the most comparable baseline to our method since the only difference is the visual instruction-tuning dataset. We use deepspeed6, zero-2 without CPU offloading. We set the batch size to 16 on 4×V100 GPUs.
In pilot experiments, we conducted tests with multiple input prompts for each model in order to identify the most effective prompts for evaluation. Despite variations in prompts, we observed only minimal fluctuations in the results. To ensure consistency and mitigate the other influences, we maintained uniformity by using the same prompt for all models performing a task. Detailed prompts can be located in the Appendix E. For the multiplechoice tasks, we transformed them into multiplechoice questions and instructed the model to respond with the corresponding label of choice. For CLOSE tasks, we design an answer decoding strategy and show in Algorithm 2. We find this strategy can handle almost all situations. For CLOSE tasks, we employ accuracy as the metric. For OPEN tasks we utilize BLEU-1/2 (Papineni et al., 2002) and BERT-SCORE (Zhang et al., 2019) as measures.

heading: 3.4 Main Results
text: We test our model on M-EV2 benchmark. We show the VQA results in Table 3, visual event prediction and visual storytelling in Table 4. We calcu-
6https://www.deepspeed.ai/
late the various kinds of average scores in Table 5. MEEL can effectively enhance performances of VQA. MEEL achieves the highest scores on three CLOSE VQA, namely VCOPA-C, VisCa-C, and VC-C in Tabel 3. The results indicate MEEL can distinguish the right events since the improvements from event graph evolution with guiding discrimination. For the three OPEN VQA datasets, among all metrics, BERT-SCORE can mostly evaluate the answering quality. We find MEEL outperforms all other baselines to a large extent. These results demonstrate the effectiveness of our method on OPEN VQA. We also notice the BLEU-1/2 of MEEL is higher than almost all models. Since BLEU-1/2 measures lexical similarity, MEEL can generate more well-formed events as the ground truth. In all, our method improves the MMER. MEEL outperforms baselines in visual event prediction. MEEL performs the best among all baselines in Table 4. The results demonstrate our training method enables the model to capture correct temporal relations leading to more precise prediction for the future. Compared to VQA tasks, We find all models perform worse in visual event prediction, indicating it needs more knowledge and reasoning ability to complete this task. In OPEN visual prediction, MEEL also achieves the highest scores in BERT-SCORE. This shows our model can forecast semantic similar events. However, we find MEEL performs slightly lower in BLEU-2
on IgSEG-O. Since BLEU calculates the 2-gram lexical similarity, this may indicate MEEL can predict more diversified events with correct semantics rather than words merely in the context. MEEL can generate advanced story. In Table 4, we find MEEL can excel all baselines in VIST. The results show MEEL can tell better stories by capturing more scenario knowledge and comprehending the inter-event relations. The event graph evolution affects the training of the model to acknowledge enriched event information rather than merely shallow step reasoning. In all, MEEL can significantly improve the performance of the downstream tasks attributed to boosted capabilities of MMER. In Table 5, MEEL excels all baselines on the average score of all datasets demonstrating the effectiveness of our method. Our event graph evolution process stimulates the contextual understanding of events. The guiding discrimination further mitigates the hallucinations of event reasoning yielding better performances.
Among all relation types, the improvements of VQA and STORYTELLING are larger than PREDICTION. It indicates our method benefits more for these tasks. PREDICTION is the hardest to learn attributed to its demand for pertaining for more abundant knowledge of events.

heading: 3.5 Analysis
text: Evolution steps. We conduct experiments on different evolution steps to verify the effectiveness of
event graph evolution. We tested steps 1-4 respectively and calculated various average scores. We show the results in Figure 4.
As the average of all results, the performance of MEEL increases from steps 1 to 3 in Figure 4 (a). This is consistent with our motivation that the event graph evolution enables the model to learn the rich knowledge of event evolution. Then, the model can complete MMER better.
We find the performances drop when the step is too large, namely larger than four. This may be attributed to the semantic drift of the event graph evolution. ChatGPT would generate less relevant content compared to the seed event if it evolves further. We find that the drop is most obvious in VQA, which may be probably due to VQA being the most strict relation among all event interrelations.
We find MEEL can achieve a high score for STORYTELLING when the evolution step is only one in Table 4 (b). MEEL is 25.51 BERT-Score while InstructBLIP is 11.31. As the number of steps increases, MEEL maintains a high score. This indicates that MEEL completes the STORYTELLING even on few evolution steps.
Effect of guiding discrimination. We ablate guiding discrimination and show the results in Table 6. We find that all performances drop if MEEL trains without guiding discrimination. It indicates that discrimination can guide the evolution and mitigate hallucinations.
Examples of event graph evolution. We showcase two examples of event graph evolution in Figure 5. We find our evolving graphs can sufficiently contain information and knowledge of event scenarios. With the aid of event-evolving graphs, MEEL learns more abundant event knowledge and relation inter-connections.

heading: 4 Relation Works
text: Multi-Modal Event Relational Reasoning As one of the relation types, causality reasoning is crucial for exploring the cause and effect of events (Yeo et al., 2018; Zhang et al., 2021a; Chadha and Jain, 2021; Ignat et al., 2021). Apart from causality, event temporal reasoning forms a basic ability (Zellers et al., 2019; Park et al., 2020; Zellers et al., 2021). Event intentional reasoning uncovers the intentions of the subjects of the events (Park et al., 2020; Li et al., 2023c). Besides, there exists research on other relation types as well (Kim et al., 2022; Hessel et al., 2022). Multi-modal event relational reasoning constitutes a foundational capability for a range of downstream tasks in the realm of multi-modal reasoning. Our research endeavors to further enhance this crucial skill. Multi-Modal Instruction tuning With the significant success of instruction tuning (Ouyang et al., 2022; Xu et al., 2023, 2024), current research has extended its capability to multi-modality. MM instruction tuning trains the model the follow instructions for questions about the images. Compared to textual instruction tuning, harvesting MM data with instructions is tougher. Zhu et al. (2023) trains MiniGPT-4 by further aligning pretrained EVACLIP (Fang et al., 2023) and Vicuna (Chiang et al., 2023). Liu et al. (2023) generate visual instruction data by requiring ChatGPT/GPT-4 with the given image and its caption. Dai et al. (2023) adapt human-labeled dataset into instruction data with pre-made templates. Li et al. (2023a) construct in-context learning data with instructions and use this dataset to train an MM LLM. These methods merely model shallow event evolving situations leading to poor ability of MMER. Script Induction Script induction is to induce or
generate chains or graphs of events representing the evolving mechanism. Du et al. (2022) induces 11 scripts of newsworthy scenarios from documents. Gunjal and Durrett (2023) attempt to generate event chains by querying large language models. Zhang et al. (2023) constructs scripts by designing interactions between humans and LLM. Li et al. (2023e) create event graphs in a pipeline operation with generation, ordering, and verification. In our work, we are the first to utilize the ability of script induction from ChatGPT to construct our MM event-oriented instruction-tuning data. We expect our work may shed light on other event-oriented approaches.

heading: 5 Conclusion
text: We propose the Multi-Modal Event Evolution Learning for MMER. We design the event graph evolution process based on the diversified seed events. We then encapsulate the evolving graphs into instruction-tuning data. We introduce the guiding discrimination training paradigm to further improve the learning of evolution. We conduct experiments on our collected and curated M-EV2 benchmark for MMER. Results show the effectiveness of MEEL and it achieves competitive performance among open-source baselines.
Limitations
Our method is limited to MMER of a single image. However, a more complex MMER may contain several images to express a scenario. We leave the construction of methods and benchmarks of this complex MMER to future work.

heading: Acknowledgement
text: This work is supported by the National Natural Science Foundation of China under Grant No. 62192731.

heading: A Baselines
text: We show statistics of M-EV2 in Table 7.

heading: B Baselines
text: LLaVA-Lora This is a MLLM trained on visual instruction-tuning. It’s based on the visual encoder ViT-L/14-336px (Radford et al., 2021) and the textual chat LLM vicuna-v1.3-7b (Chiang et al., 2023). In the first pre-train stage, it is trained with image-text pairs. In the second stage, it is finetuned by LLM-generated instruction-tuning data with LoRA (Hu et al., 2021). InstructBLIP It uses BLIP-2 (Li et al., 2023d) framework as its foundation, InstructBLIP strategically restructures 26 pre-trained public datasets, including image captioning and VQA, into a format conducive to instruction tuning (Dai et al., 2023). Otter This model combines multi-modal incontext learning with multi-modal instruction tuning, building upon the foundation of OpenFlamingo (Awadalla et al., 2023). This involves updating the perceiver module and relevant components of the LLM throughout the training process. The instructional data is sourced from reputable datasets including VQAv2 (Antol et al., 2015), GQA (Hudson and Manning, 2019), LLaVA, as well as a proprietary video dataset not available to the public. MiniGPT-4 This model conducts visual instruction tuning on the pre-trained BLIP-2 (Li et al., 2023d), specifically focusing on updating the linear layer (Zhu et al., 2023). The instructions primarily draw from the domain of image captioning tasks. MiniGPT-4-v2 This model performs as a unified interface to complete various tasks such as VQA, visual grounding, and image caption (Chen et al., 2023). Different from MiniGPT-4, it adds task identifiers into the prompt to guide the task completion. The backbone of MiniGPT-4-v2 is LLama-2 (Touvron et al., 2023).

heading: C Decoding Protocol
text: We show our decoding protocol for extracting answers of CLOSE tasks as in Algorithm 2.

heading: D Effect of event diversification.
text: We compute the event verb distribution. We show two verb distributions with or without event diversification. The results are in Figure 6. We find
Algorithm 2: CLOSE answer decoding. Input :Prediction P , candidate set D. Output :Answer A.
the distribution is significantly diversified after the event diversification process. It enables MEEL to be trained in various event scenarios and domains.
E Inference Prompts
We show inference prompts of all test set in Figure 7. We test various prompts in our pilot experiments and choose the prompts shown in Figure 7 which perform the best among others. We use the same prompts for all models.

heading: Future Work
text: We leave the construction of methods and benchmarks of this complex MMER to future work