Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?
Authors: Yingjin Song, Yupei Du, Denis Paperno, Albert Gatt
Accepted at: acl25

Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?

Yingjin Song, Yupei Du, Denis Paperno and Albert Gatt
Utrecht University, Utrecht, the Netherlands
{y.song5, y.du, d.paperno, a.gatt}@uu.nl

Abstract
This paper introduces the TempVS benchmark, which focuses on temporal grounding and reasoning capabilities of Multimodal Large Language Models (MLLMs) in image sequences. TempVS consists of three main tests (i.e., event relation inference, sentence ordering and image ordering), each accompanied with a basic grounding test. TempVS requires MLLMs to rely on both visual and linguistic modalities to understand the temporal order of events. We evaluate 38 state-of-the-art MLLMs, demonstrating that models struggle to solve TempVS, with a substantial performance gap compared to human capabilities. We also provide fine-grained insights that suggest promising directions for future research. Our TempVS benchmark data and code are available at https://github.com/yjsong22/TempVS.

1 Introduction
Multimodal Large Language Models (MLLMs) (Achiam et al., 2023; Gemini et al., 2024; Liu et al., 2024a) have demonstrated remarkable performance in various vision and language tasks. At the same time, the need for standardized evaluation frameworks has become increasingly critical. Most existing benchmarks focus on settings involving a single image (Fu et al., 2023; Li et al., 2024b; Yue et al., 2024; Liu et al., 2024c; Lu et al., 2024). While some benchmarks consider multi-image settings (Jiang et al., 2024; Fu et al., 2024; Ying et al., 2024; Li et al., 2024b), they mainly focus on cross-image recognition and reference. To date, relatively little attention has been paid to more complex tasks, such as grounded temporal understanding and reasoning in multiple images.

Some recent studies have assessed MLLMs’ temporal comprehension across multiple images, but certain limitations remain. First, some tasks can be resolved by relying on a single image rather than a sequence (Liu et al., 2024b; Ying et al., 2024). For example, given an image sequence, determining whether “pulling a wind-up toy so it continues moving forward or quickly stops” could be answered only based on the final image. Second, some tasks depend heavily on commonsense or world knowledge (Wang et al., 2025; Meng et al., 2025), such as rearranging a set of shuffled images into the correct sequence of cooking steps. Third, some benchmarks (Song et al., 2024) use distractor options absent from the images, allowing the models to infer ground-truth answers based on the presence of objects. These factors may result in the benchmarks failing to truly assess the model’s understanding of temporal sequences. In addition, none of the existing benchmarks are designed for multi-event scenarios, making them inadequate for evaluating complex temporal sequences and relations in MLLMs.

As a result, a question arises: Do existing MLLMs really understand time by accurately aligning the order of events in language and image sequences? To address this, we propose TempVS, a benchmark for multi-event Temporal grounding and reasoning in Visual Story image sequences. TempVS contains 2,085 image sequences (9,803 images) across cartoon animations, movies and daily-life albums, with 15,192 multiple-choice questions. TempVS features three temporal understanding and reasoning tasks: event relation inference, sentence ordering, and image ordering (as shown in Figure 1). These tasks are accompanied by grounding tasks to check whether the model can match the exact image consistent with a single text description. We select image sequences making up visual stories, each containing several events that are temporally related yet relatively independent, in the sense that no event could easily be predicted from preceding events. This makes it hard to resolve the tasks without considering both linguistic and visual modalities. TempVS challenges models
Two-event Relation Inference

True: Eddy opened the door of the house after Rody waved to Eddy from his green four-wheel bike.
False: Rody waved to Eddy from his green four-wheel bike after Eddy opened the door of the house.

Three-event Relation Inference

True: First, Anne cooked a meal in her kitchen. Second, Anne made a call on her phone. Third, Anne protected her hands with a towel to hold the pot.
False: First, Anne made a call on her phone. Second, Anne protected her hands with a towel to hold the pot. Third, Anne cooked a meal in her kitchen.

Sentence Ordering

Sentence a: On the last day of our trip, Grandpa teaches the little boy some swimming tricks.
Sentence b: Grandma is turning the hot dogs. On low flame, he says, “Dinner will be ready in an hour.”
Sentence c: The marshmallows are fun to eat.
Sentence d: Mom, Dad, and the little boy are getting the fire started.
Sentence e: The kangaroos are amazing. The little boy tries to pet one, but it jumps and runs away.
Answer: Sentence d -> Sentence b -> Sentence e -> Sentence c -> Sentence a

Image Ordering

Fred leaned in the doorway while talking.
Wilma was sitting in the living room, crocheting something pink.
Fred put water in his yard.
A purple elephant drank from a pool in the yard.
The man with the purple shirt got wet in the lawn chair.
Answer: Image d -> Image e -> Image c -> Image a -> Image b

Figure 1: Illustrative examples from the main tests of TempVS benchmark. Additional examples are provided in Appendix E.

to reason about event order in the image sequence and text (e.g., a sentence describing two events using before or after, or a story) and integrate both.

We extensively evaluate 38 MLLMs, including open-source models ranging from 0.5B to 78B (e.g., LLaVA-OneVision, InternVL2.5, Qwen2-VL, Phi-3.5-vision, DeepSeek-vl2, LLaVA-NeXT-Video) and the closed-source GPT-4o. We show that TempVS is highly challenging for SOTA models, especially on event relation inference and image ordering tasks. In particular, while models can accurately ground individual events to images, their performance on the main tasks, which require multimodal reasoning with sequences, remains unsatisfactory. We further analyze the impact of the choice of linguistic structure, distance between events, and Chain-of-Thought prompting. Our analysis sheds light on future directions for improvement in architectural design, training objectives, and/or post-training methods to enhance temporal reasoning.

Contributions. (1) We introduce TempVS, a new benchmark for evaluating multi-event temporal grounding and reasoning ability in image sequences for MLLMs. (2) We extensively evaluate 38 MLLMs from different model families and sizes, highlighting the performance gap compared to human annotators. (3) Our findings in evaluation results and fine-grained analysis suggest potential pathways for future improvements.

2 Related Work

Multimodal Large Language Models Progress in large language models (LLMs) (Achiam et al., 2023; AI@Meta, 2024; Touvron et al., 2023; Gemini et al., 2024) has provided impetus to the development of multimodal LLMs which process both visual and textual information. State-of-the-art MLLMs (Achiam et al., 2023; Dai et al., 2023; Gemini et al., 2024; Liu et al., 2024a; Abdin et al., 2024; Wang et al., 2024a; Chen et al., 2024) are built upon LLMs with an integrated visual encoder and a connection module. These models surpass the earlier generation of multimodal models, which were typically based on BERT-type architectures, on many downstream tasks (Bugliarello et al., 2023). While some studies have focused on training MLLMs to interpret multiple images using interleaved image-text datasets (Jiang et al., 2024; Huang et al., 2024; Li et al., 2024a,c), their capability to understand and reason about multi-event temporal relationships in sequential visual data remains largely unexplored.

Multi-image Benchmarks Multi-image understanding requires MLLMs to compare, analyze and interpret relationships across multiple images (Li et al., 2024d). Benchmarks such as NLVR2 (Suhr et al., 2019), BLINK (Fu et al., 2024), SEED-Bench-2 (Li et al., 2024b), Mantis-Eval (Jiang et al., 2024) and MMT-Bench (Ying et al., 2024) cover a subset of multi-image tasks focusing on assessing
models’ ability to identify similarities and variations across multiple images. DEMON (Li et al., 2024e) evaluates the demonstrative instruction-following abilities of MLLMs. Mementos (Wang et al., 2024b) aims to detect object and behavior hallucinations in descriptive text generated for sequential images. MileBench (Song et al., 2024) evaluates MLLMs’ performance in long contexts. MIBench (Liu et al., 2024b) assesses MLLM’s ability in multi-image instruction, multi-modal knowledge-seeking and in-context learning. MuirBench (Wang et al., 2025) is a comprehensive multi-image understanding benchmark with unanswerable counterparts to test the robustness of MLLMs. MMIU (Meng et al., 2025) incorporates a large number of test questions that cover a diverse array of multi-image tasks and relationships. We propose TempVS, the first benchmark specifically designed for multi-event temporal understanding and reasoning in image sequences. In particular, it is designed to avoid shortcuts such as reliance on single images/frames and commonsense reasoning to bypass full integration of the temporal information in both text and images.

3 The TempVS Benchmark

TempVS evaluates models’ ability to understand and reason about temporal relations by evaluating the consistency between textual descriptions of temporally related events and the visual event order in an image sequence. To achieve this, we create three main tests (MT): event relation inference, sentence ordering, and image ordering. Additionally, to investigate whether a model’s difficulty arises from challenges in temporal understanding or from more basic grounding issues, we create a corresponding grounding test (GT) (\S 3.1) for the main tests. We present the task curation process and statistics of TempVS benchmark in \S 3.2.

TempVS is built from existing datasets pairing image sequences with narrative captions. Since captions may vary in their level of detail, we use both the original captions in the source data, and simplified captions in which the main event is extracted from the caption (we describe this process in \S 3.2). In what follows, we denote an image sequence \( S \) consisting of \( n \) images, their corresponding captions, and extracted events as:

\[
S = [(I_1, C_1, E_1), (I_2, C_2, E_2), \ldots, (I_n, C_n, E_n)],
\]

where \( I_i \) denotes the \( i \)-th image, \( C_i \) its associated original caption, and \( E_i \) the extracted event (expressed via the simplified caption).

<table>
  <tr>
    <th>Statement Template (\( i < j < k \) in image sequence)</th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td colspan="3">Two-event</td>
  </tr>
  <tr>
    <td>Pos: \( E_j \) after \( E_i \)</td>
    <td>Neg: \( E_i \) after \( E_j \)</td>
    <td></td>
  </tr>
  <tr>
    <td>Pos: \( E_i \) before \( E_j \)</td>
    <td>Neg: \( E_j \) before \( E_i \)</td>
    <td></td>
  </tr>
  <tr>
    <td>Pos: \( E_j \). Earlier, \( E_i \)</td>
    <td>Neg: \( E_i \). Earlier, \( E_j \)</td>
    <td></td>
  </tr>
  <tr>
    <td>Pos: \( E_i \). Then, \( E_j \)</td>
    <td>Neg: \( E_j \). Then, \( E_i \)</td>
    <td></td>
  </tr>
  <tr>
    <td>Pos: \( E_i, E_j \)</td>
    <td>Neg: \( E_j, E_i \)</td>
    <td></td>
  </tr>
  <tr>
    <td colspan="3">Three-event</td>
  </tr>
  <tr>
    <td>Pos: \( E_i \) before \( E_j \), and after that, \( E_k \)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Neg: \( E_k \) before \( E_i \), and after that, \( E_j \)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Pos: \( E_i \). Later, \( E_j \). Finally, \( E_k \).</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Neg: \( E_j \). Later, \( E_i \).Finally, \( E_k \).</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Pos: First, \( E_i \). Second, \( E_j \).Third, \( E_k \).</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Neg: First, \( E_i \). Second, \( E_k \).Third, \( E_j \).</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Pos: \( E_i, E_j, E_k \)</td>
    <td>Neg: \( E_k, E_j, E_i \)</td>
    <td></td>
  </tr>
</table>

Table 1: Templates of positive (Pos) and negative (Neg) statements used for MT1 event relation inference.

3.1 Main Tests and Grounding Test

MT1: Event Relation Inference  MT1 evaluates a model’s understanding of the chronological order of events based on an image sequence and a textual description. The text describes the temporal relation of the events either explicitly through adverbial markers such as after or before, or implicitly through the surface order of sentences. From an image sequence of length \( n \), we select either (1) two pairs \( \{(I_i, E_i), (I_j, E_j)\} \) where \( i < j \leq n \), ensuring that \( I_i \) appears before \( I_j \) and \( E_i \) occurs before \( E_j \); or (2) three pairs \( \{(I_i, E_i), (I_j, E_j), (I_k, E_k)\} \) where \( i < j < k \leq n \), following the same ordering constraints. These event-image pairs are not necessarily adjacent in the sequence, resulting in varying distances between them. We then generate positive and negative statements by applying the templates in Table 1 to describe the temporal relations between these events. ¹ The negative statement retains the same event clauses and the temporal conjunction as the positive statement, while swapping the positions of these clauses, such that the text expresses a different temporal order.² Finally, we derive triples comprising an original image sequence, a positive or negative statement and the corresponding answer (True or False).

¹We only use the textual events to construct the statements in MT1, ignoring the captions.
²For the negative statements with three events, we randomly select one from the five combinations that is not the same as the positive statement.
MT2: Sentence Ordering   This task evaluates whether models can correctly reorder a shuffled set of sentences based on the temporal order of events in a given image sequence. Thus, MT2 requires not only an understanding of temporal relations between events but also consideration of the text’s coherence and fluency. Models are tasked to select the correct sentence order from five given options, based on an ordered image sequence and a set of shuffled event descriptions. We create versions with both the original captions (\( C \)) and the extracted events (\( E \)).

MT3: Image Ordering   This task mirrors MT2 but focuses on the visual modality, requiring models to rearrange a set of shuffled images into the correct temporal order based on the given textual description. Similar to MT2, we also examine whether different text styles (i.e., original captions or extracted event descriptions) could affect the model’s ability to determine the correct order.

GT: Grounding Test   As a prerequisite for solving one of the main tests, we assume that MLLMs should be able to match an event description with the corresponding image in a multi-image sequence. Specifically, given an event description \( E_i \) or a caption \( C_i \) and image sequence \( [I_1, I_2, \ldots, I_n] \), models are required to identify the index of the image (i.e., \( I_i \)) that best corresponds to the given textual description. The motivation for conducting grounding tests is as follows: If a model passes the grounding tests but fails the corresponding main tests, it indicates that the model struggles with understanding temporal order, even if it can accurately recognize and associate visual and linguistic elements. In contrast, if a model performs well on the main tests but fails the corresponding grounding tests, it may suggest that its success stems not from the true temporal grounding or reasoning but rather from leveraging statistical patterns, correlations, or systematic biases learned during training.

3.2 Benchmark Curation and Statistics

Data Sources   Given our objective of evaluating whether models understand the chronological order of events across image sequences and language, we require data containing multiple images that form a temporal sequence presenting events. We choose four visual story datasets: FlintstonesSV (Gupta et al., 2018), PororoSV (Li et al., 2019), VIST (Huang et al., 2016) and VWP (Hong et al., 2023). FlintstonesSV and PororoSV, designed for story visualization, contain annotated frames from cartoon animations.\footnote{For the major characters in FlintstonesSV and PororoSV, we provide descriptions of their appearances to match them with their names.} VIST, built for visual storytelling, originates from Flickr albums with user-uploaded daily-life photos. VWP features movie scene sequences paired with aligned synopses. This collection with rich styles and diverse domains plays a crucial role in assessing MLLMs’ multi-event temporal grounding and reasoning capabilities.

Dataset Filtering   To ensure that each image sequence contains a sufficient number of character-centered visual events, we use Detectron2\footnote{https://ai.meta.com/tools/detectron2/} to detect and retain image sequences where PERSON can be detected in at least 60% of the images. To avoid temporal overlap between any two events, we remove any image sequence whose captions contain stative verbs such as *belong*, *love* and *exist*. To minimize ambiguity, we remove captions starting with pronouns and compute the BERTScore (Zhang et al., 2020) between captions, omitting sequences with highly similar captions. Similarly, we remove image sequences with highly similar images based on the cosine similarity between their CLIP (Radford et al., 2021) embeddings. Simple events (\( E \)) are extracted from the original captions (\( C \)) using GPT-4 (Achiam et al., 2023); any sequences with captions from which no event can be extracted are removed. See Appendix A.3 for details of the extraction process. To ensure that each image-event pair in \( \{(I_i, E_i), (I_j, E_j)\} \) remains distinct, we use CLIP to compute cross-modality similarity between different image-text combinations. Ambiguous pairs are filtered based on a threshold, ensuring that within-pair similarity is significantly higher than cross-pair similarity.\footnote{That is, we guarantee that \( sim(I_i, E_i) > sim(I_j, E_j) \) and \( sim(I_i, E_i) > sim(I_j, E_i) \) and \( sim(I_j, E_j) > sim(I_j, E_i) \) and \( sim(I_j, E_j) > sim(I_j, E_i) \).} A similar process is applied to sets of three image-event pairs \( \{(I_i, E_i), (I_j, E_j), (I_k, E_k)\} \). In Appendix A, we provide the details of stative verbs list, the similarity thresholds, prompt used for extracting event from original caption, and statistics of the dataset after each filtering step.

Prompt and Option Generation   After filtering the datasets, we create the positive and negative statements by concatenating the events with the templates shown in Table 1 for MT1. We sequentially apply each template to its corresponding tem-
<table>
  <tr>
    <th></th>
    <th>MT1 (two)</th>
    <th>MT1 (three)</th>
    <th>MT2 (event)</th>
    <th>MT2 (caption)</th>
    <th>MT3</th>
  </tr>
  <tr>
    <td>FlintstonesSV</td>
    <td>2,104</td>
    <td>916</td>
    <td>501</td>
    <td>485</td>
    <td>565</td>
  </tr>
  <tr>
    <td>PororoSV</td>
    <td>864</td>
    <td>172</td>
    <td>320</td>
    <td>326</td>
    <td>395</td>
  </tr>
  <tr>
    <td>VWP</td>
    <td>850</td>
    <td>208</td>
    <td>274</td>
    <td>236</td>
    <td>316</td>
  </tr>
  <tr>
    <td>VIST</td>
    <td>3,742</td>
    <td>830</td>
    <td>708</td>
    <td>551</td>
    <td>809</td>
  </tr>
  <tr>
    <td>TempVS</td>
    <td>7,560</td>
    <td>2,126</td>
    <td>1,803</td>
    <td>1,618</td>
    <td>2,085</td>
  </tr>
</table>

Table 2: TempVS benchmark statistics: In MT1, the number indicates the total statements; in MT2, the number of image sequences and corresponding shuffled sentence sets; in MT3, the number of textual events or captions and their associated shuffled image sets.

poral relation group in the dataset, ensuring an even distribution of statement types. Moreover, we use ChatGPT(OpenAI, 2022) to generate variations of different prompt components for different tasks including task instructions, answer requirements and response formats (see Appendix C), which results in a total of 328 possible prompt variations. By incorporating sufficient diversity in prompts, we mitigate the risk of results being influenced by a specific prompt formulation (Sclar et al., 2024). All tests are formed as multiple-choice questions. In MT1, the options are “True” and “False” with positions alternated across samples (e.g., A. True; B. False. and A. False; B. True.) to prevent position bias (Zheng et al., 2024). In MT2 and MT3, one correct sequence is presented alongside four randomly shuffled incorrect sequences with options labeled “A” to “E”. The Grounding Test uses image indices as answer choices. To ensure fair evaluation, correct answers are evenly distributed across options throughout the benchmark.

Quality Control To discourage “blind” models that leverage language biases, we filter examples in the benchmark that could be easily solved based only on the linguistic modality. We apply three unimodal LLMs Phi-3.5-mini-instruct [4B] (Abdin et al., 2024), Llama-3.1 [8B] (AI@Meta, 2024) and Qwen-2.5-instruct [72B] (QwenTeam, 2024), which are popular LLM backbones in current MLLMs. In MT1 and MT2, we discard a sample if at least two LLMs can answer the question correctly without visual inputs. A manual check was performed by the authors to exclude ambiguous images, grammatically incorrect and/or semantically implausible statements, and cases where image sequences did not match the text.

Benchmark Statistics TempVS consists of 2,085 distinct image sequences with corresponding original captions and extracted events. Table 2 shows the statistics of each task from each data source in TempVS. Most image sequences in the benchmark contain 5 images each, except for 61 sequences from VWP, which have 6 to 9 images.

4 Experiments

4.1 Experimental Setup

Models We evaluate a diverse family of state-of-the-art models with various sizes (ranging from 0.5B to 78B) and different vision and LLM backbones. We select DeepSeek-v12 [3B/16B] (Wu et al., 2024), InternVL2.5 [1B/8B/26B/78B] (Chen et al., 2024), Janus-Pro [1B/7B](Chen et al., 2025), LLaVA-NeXT-Interleave [0.5B/7B] (Li et al., 2024c), LLaVA-OneVision [0.5B/7B/72B] (Li et al., 2024a), LLaVA-NeXT-Video [7B/34B] (Zhang et al., 2024b), LongVA [7B] (Zhang et al., 2024a), Mantis[8B] (Jiang et al., 2024), Phi-3-vision [4B], Phi-3.5-vision [4B] (Abdin et al., 2024), and Qwen2-VL [2B/7B/72B] (Wang et al., 2024a). We also evaluate GPT-4o [2024-11-20]. The implementation details are provided in Appendix D.1.6

Evaluation Metrics For multiple-choice questions, we benchmark MLLMs’ performance using accuracy as the metric for predicted options. Accuracy scores are reported for both the main tests (MT) and their corresponding grounding tests (GT). In MT1, where each question relates to either two or three events, there is a GT for each separate event. In MT2 and MT3, each question corresponds to a number of grounding tests equal to the number of images in the sequence. Additionally, we introduce a stricter metric, GTstrict, which assesses the number of image sequences where a model passes all corresponding grounding tests.7 To examine the relationship between main test and grounding test performance, we further report MTGTstrict, where a model’s success on a main test is considered valid only if it passes all corresponding grounding tests.

6In the evaluation, we horizontally combine multiple images of a sequence into one image. In our preliminary experiments, we attempted to input the images into the model sequentially, one at a time, and observed little difference in performance compared to merging them into a single input (see results in Table 12). In the released TempVS benchmark, both the individual-image inputs and the combined multi-image inputs are provided.

7Appendix D.2 provides overall grounding test results, showing models’ ability to precisely locate the correct image within a sequence based on textual input in general.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Two-event Relation (MT1)</th>
    <th colspan="3">Three-event Relation (MT1)</th>
    <th colspan="3">Sentence Ordering (event MT1)</th>
    <th colspan="3">Sentence Ordering (option MT1)</th>
    <th colspan="3">Image Ordering (event MT1)</th>
    <th colspan="3">Image Ordering (option MT1)</th>
  </tr>
  <tr>
    <th>GT</th><th>MT</th><th>MTIGT</th>
    <th>GT</th><th>MT</th><th>MTIGT</th>
    <th>GT</th><th>MT</th><th>MTIGT</th>
    <th>GT</th><th>MT</th><th>MTIGT</th>
    <th>GT</th><th>MT</th><th>MTIGT</th>
    <th>GT</th><th>MT</th><th>MTIGT</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>50</td><td>80</td><td>90</td>
    <td>0.03</td><td>0.20</td><td>0.03</td>
    <td>0.03</td><td>0.20</td><td>0.03</td>
    <td>0.03</td><td>0.20</td><td>0.03</td>
    <td>0.03</td><td>0.20</td><td>0.03</td>
    <td>0.03</td><td>0.20</td><td>0.03</td>
  </tr>
  <tr>
    <td>DeepSeek-42</td>
    <td>3B</td><td>20.8</td><td>49.7</td><td>65.5</td>
    <td>50.6</td><td>81.8</td><td>91.5</td>
    <td>25.0</td><td>41.8</td><td>58.5</td>
    <td>17.9</td><td>20.4</td><td>42.9</td>
    <td>20.7</td><td>20.7</td><td>12.5</td>
  </tr>
  <tr>
    <td></td>
    <td>16B</td><td>14.2</td><td>43.1</td><td>62.2</td>
    <td>44.4</td><td>84.4</td><td>94.5</td>
    <td>14.3</td><td>18.6</td><td>17.1</td>
    <td>18.2</td><td>16.6</td><td>0.0</td>
    <td>15.5</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>InternVL2.5</td>
    <td>2B</td><td>46.3</td><td>57.0</td><td>68.4</td>
    <td>81.2</td><td>92.3</td><td>97.0</td>
    <td>73.6</td><td>81.1</td><td>85.0</td>
    <td>76.9</td><td>76.7</td><td>26.7</td>
    <td>77.3</td><td>31.7</td><td>15.5</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td><td>51.3</td><td>60.3</td><td>62.1</td>
    <td>66.0</td><td>84.7</td><td>91.6</td>
    <td>69.9</td><td>73.0</td><td>77.0</td>
    <td>82.1</td><td>84.4</td><td>39.7</td>
    <td>89.5</td><td>41.7</td><td>41.7</td>
  </tr>
  <tr>
    <td></td>
    <td>26B-MPO</td><td>51.3</td><td>60.3</td><td>62.1</td>
    <td>66.0</td><td>84.7</td><td>91.6</td>
    <td>69.9</td><td>73.0</td><td>77.0</td>
    <td>82.1</td><td>84.4</td><td>39.7</td>
    <td>89.5</td><td>41.7</td><td>41.7</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td><td>51.6</td><td>54.3</td><td>55.3</td>
    <td>47.3</td><td>57.0</td><td>67.6</td>
    <td>84.9</td><td>18.5</td><td>71.7</td>
    <td>83.9</td><td>31.7</td><td>46.0</td>
    <td>85.8</td><td>58.5</td><td>47.1</td>
  </tr>
  <tr>
    <td></td>
    <td>7B-MPO</td><td>51.6</td><td>54.3</td><td>55.3</td>
    <td>47.3</td><td>57.0</td><td>67.6</td>
    <td>84.9</td><td>18.5</td><td>71.7</td>
    <td>83.9</td><td>31.7</td><td>46.0</td>
    <td>85.8</td><td>58.5</td><td>47.1</td>
  </tr>
  <tr>
    <td>Janus-Pro</td>
    <td>1B</td><td>77.7</td><td>48.3</td><td>48.1</td>
    <td>46.5</td><td>42.6</td><td>19.6</td>
    <td>96.4</td><td>41.9</td><td>55.8</td>
    <td>96.4</td><td>41.9</td><td>55.8</td>
    <td>96.4</td><td>41.9</td><td>55.8</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td><td>4.3</td><td>35.1</td><td>34.1</td>
    <td>0.4</td><td>32.9</td><td>39.3</td>
    <td>0.0</td><td>17.1</td><td>-</td>
    <td>0.0</td><td>15.5</td><td>-</td>
    <td>20.8</td><td>-</td><td>21.0</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT-Interleave</td>
    <td>7B</td><td>3.3</td><td>51.6</td><td>52.4</td>
    <td>7.2</td><td>50.1</td><td>49.8</td>
    <td>0.3</td><td>25.1</td><td>16.7</td>
    <td>0.4</td><td>27.0</td><td>0.0</td>
    <td>20.9</td><td>16.7</td><td>20.0</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td><td>13.0</td><td>51.6</td><td>52.4</td>
    <td>7.2</td><td>50.1</td><td>49.8</td>
    <td>0.3</td><td>25.1</td><td>16.7</td>
    <td>0.4</td><td>27.0</td><td>0.0</td>
    <td>20.9</td><td>16.7</td><td>20.0</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision-v0</td>
    <td>0.5B</td><td>45.7</td><td>58.0</td><td>58.0</td>
    <td>48.1</td><td>47.8</td><td>41.0</td>
    <td>87.0</td><td>0.1</td><td>18.4</td>
    <td>19.4</td><td>18.0</td><td>10.0</td>
    <td>19.0</td><td>19.0</td><td>10.0</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td><td>32.8</td><td>58.0</td><td>58.0</td>
    <td>28.0</td><td>25.8</td><td>4.5</td>
    <td>41.2</td><td>41.2</td><td>47.6</td>
    <td>21.3</td><td>18.3</td><td>18.8</td>
    <td>21.3</td><td>18.3</td><td>18.8</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td><td>46.4</td><td>59.3</td><td>62.1</td>
    <td>40.5</td><td>61.5</td><td>63.5</td>
    <td>9.8</td><td>65.2</td><td>81.8</td>
    <td>14.0</td><td>75.1</td><td>86.6</td>
    <td>27.6</td><td>31.8</td><td>36.5</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT-Videos</td>
    <td>7B</td><td>34.8</td><td>58.5</td><td>58.4</td>
    <td>1.6</td><td>59.5</td><td>57.6</td>
    <td>0.1</td><td>31.8</td><td>100.0</td>
    <td>0.0</td><td>33.4</td><td>-</td>
    <td>19.8</td><td>-</td><td>20.0</td>
  </tr>
  <tr>
    <td></td>
    <td>34B</td><td>6.5</td><td>58.5</td><td>58.4</td>
    <td>1.6</td><td>59.5</td><td>57.6</td>
    <td>0.1</td><td>31.8</td><td>100.0</td>
    <td>0.0</td><td>33.4</td><td>-</td>
    <td>19.8</td><td>-</td><td>20.0</td>
  </tr>
  <tr>
    <td>LongVA</td>
    <td>7B</td><td>8.1</td><td>54.7</td><td>56.1</td>
    <td>2.1</td><td>56.0</td><td>61.7</td>
    <td>0.2</td><td>34.2</td><td>66.7</td>
    <td>0.2</td><td>35.3</td><td>50.0</td>
    <td>19.5</td><td>0.1</td><td>19.0</td>
  </tr>
  <tr>
    <td>Monte-Carlo</td>
    <td>8B</td><td>12.2</td><td>53.8</td><td>53.8</td>
    <td>0.5</td><td>53.8</td><td>53.8</td>
    <td>0.0</td><td>23.8</td><td>0.0</td>
    <td>18.6</td><td>0.0</td><td>19.2</td>
    <td>18.3</td><td>18.3</td><td>18.3</td>
  </tr>
  <tr>
    <td>Phi-3.5-vision</td>
    <td>3.4B</td><td>4.0</td><td>49.0</td><td>47.7</td>
    <td>8.8</td><td>48.3</td><td>48.0</td>
    <td>2.3</td><td>0.0</td><td>25.4</td>
    <td>19.2</td><td>19.2</td><td>19.2</td>
    <td>18.3</td><td>18.3</td><td>18.3</td>
  </tr>
  <tr>
    <td>Qwen2-VL-Instruct</td>
    <td>7B</td><td>32.6</td><td>54.0</td><td>55.4</td>
    <td>21.2</td><td>53.6</td><td>55.3</td>
    <td>3.4</td><td>42.5</td><td>64.8</td>
    <td>4.4</td><td>44.6</td><td>61.3</td>
    <td>23.1</td><td>24.6</td><td>35.9</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td><td>32.6</td><td>54.0</td><td>55.4</td>
    <td>21.2</td><td>53.6</td><td>55.3</td>
    <td>3.4</td><td>42.5</td><td>64.8</td>
    <td>4.4</td><td>44.6</td><td>61.3</td>
    <td>23.1</td><td>24.6</td><td>35.9</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>API</td><td>68.3</td><td>88.1</td><td>80.1</td>
    <td>57.0</td><td>64.5</td><td>64.4</td>
    <td>18.6</td><td>51.4</td><td>53.9</td>
    <td>28.6</td><td>61.5</td><td>55.1</td>
    <td>22.6</td><td>23.0</td><td>21.5</td>
  </tr>
</table>

Table 3: Zero-shot average accuracy performance of 11 popular MLLMs families with 21 variants on TempVS benchmark on strict grounding test (GT_s), main test (MT) and the main test when all corresponding grounding tests pass (MTIGT_s). Best models per metric are marked in boldface and the second best models are underlined.

4.2 Main Results

Results for 21 state-of-the-art MLLMs are shown in Table 3. For the full quantitative results of all tested 38 MLLMs, we refer to Appendix D.2. Our main observations of the empirical results are as follows:

TempVS is challenging even for SOTA MLLMs
InternVL2.5-26B-MPO achieves the highest performance on the two-event relation inference, while GPT-4o leads in the three-event relation inference. InternVL2.5-78B-MPO outperforms other models in both sentence ordering and image ordering tasks. However, most models with parameters less than or equal to 7B exhibit random chance accuracies of approximately 50% (for MT1) and 20% (for MT2 and MT3). Most MLLMs perform similarly on two-event and three-event relation inference tasks, while for the strongest models (such as InternVL2.5[26B/78B], LLaVA-OneVision-ov-72B and GPT-4o), their performance is even slightly better on the understanding of three-event statements. In addition, sentence ordering is a relatively simple task (with the highest accuracy at 86.3%), while image ordering is a significantly more challenging task (with the highest accuracy only at 53.8%). GPT-4o performs substantially worse than several of the best-performing open-source models in both sentence and image ordering tasks. In terms of language type, we find that sentence and image ordering are easier with original captions than with extracted events. This may indicate that models might leverage additional contextual details and temporal cues from the original captions, which are unavailable in the simpler extracted event descriptions.

Both scale and post-training help As shown in Figure 2, accuracy generally improves with model size. However, the marginal gains diminish as models get larger. In the two-event and three-event relation inference, this effect is more evident, with smaller models (7B or 26B) already attaining competitive and in some cases higher accuracy compared to larger models (> 70B). Additionally, the two ordering tasks exhibit larger performance gaps between smaller and larger models, which can be attributed to the superior long-range reasoning capabilities of the more powerful LLM backbones in larger MLLMs. Surprisingly, DeepSeek-VL2 [3B/16B] and Janus-Pro [1B/7B] are exceptions, as their smaller models outperform the larger ones in most cases.

Our results also highlight the importance of high-quality post-training: under the same model sizes, InternVL2.5-MPO consistently outperforms InternVL2.5 across all evaluated tasks, especially when the model parameters exceed 7B. These results indicate that Mixed Preference Optimization (MPO) (Chen et al., 2024) effectively enhances the overall multimodal temporal understanding and reasoning capabilities. Similarly, models fine-tuned using Direct Preference Optimization (DPO) are consistently better than their SFT-only counterparts. Other results comparing LLaVA-NeXT-Video and LongVA families are provided in Appendix D.2. We also see a positive impact of instruction tuning on complex reasoning tasks such as image ordering (e.g., Qwen2-VL-Instruct outperforms Qwen2-VL across all tasks).

Event grounding does not guarantee understanding of temporal relations In MT1, we observe a slight difference between MT and
Figure 2: Illustrative TempSV benchmark results of selected models with different number of parameters.

<table>
  <tr>
    <th># Image seqs</th>
    <th>Accuracy</th>
    <th>Fleiss’ kappa</th>
  </tr>
  <tr>
    <td>Two-event Relation</td>
    <td>60</td>
    <td>82.5</td>
    <td>0.728</td>
  </tr>
  <tr>
    <td>Three-event Relation</td>
    <td>60</td>
    <td>81.6</td>
    <td>0.689</td>
  </tr>
  <tr>
    <td>Sentence Ordering (event)</td>
    <td>40</td>
    <td>81.2</td>
    <td>0.751</td>
  </tr>
  <tr>
    <td>Sentence Ordering (caption)</td>
    <td>40</td>
    <td>89.3</td>
    <td>0.764</td>
  </tr>
  <tr>
    <td>Image Ordering (event)</td>
    <td>40</td>
    <td>79.1</td>
    <td>0.827</td>
  </tr>
  <tr>
    <td>Image Ordering (caption)</td>
    <td>40</td>
    <td>77.9</td>
    <td>0.742</td>
  </tr>
</table>

Table 4: Results of human evaluation on all main tests.

MTIGT_{strict}. This suggests that even if a model can accurately identify the image corresponding to every single event within a sequence, it may still lack the ability to understand the chronological order of events in text. In short, grounding textual descriptions and reasoning about their temporal relations require different capabilities. Among all grounding tests, GPT-4o performs the best, but it lags significantly behind top-tier open-source models like InternVL2.5-MPO(26B/78B) especially in the two ordering tasks (MT2 and MT3). In MT2 and MT3, MTIGT_{strict} accuracy improves substantially over MT for most large MLLMs, confirming the fundamental role of visual grounding in sentence or image ordering tasks. For instance, the MTIGT_{strict} accuracy scores of InternVL2.5-78B-MPO are higher than MT by 16.8% (event) and 10.1% (caption) in MT2, and by 7.8% (event) and 15.9% (caption) in MT3. Meanwhile, for small models like DeepSeek-v1[23B], MTIGT_{strict} accuracy is sometimes even smaller than MT accuracy. This indicates some dependency of these models’ reasoning abilities on their grounding capability, though this is unlikely to be the only factor affecting performance.

Gap with human performance To evaluate the quality and estimate the difficulty of TempVS, we perform a human assessment on 280 randomly selected image sequences, covering all fine-grained statement types of MT1 as well as both language types (\( C \) and \( E \)) for MT2 and MT3. We recruited 36 annotators on the Prolific platform and collected three responses for each image sequence, yielding 840 responses collected in total. Further details are provided in Appendix B.

There is a large gap between human average performance (Table 4) and SOTA MLLMs. For the tasks of two-event and three-event relation inference and image ordering, there remains plenty of room for improvement. However, for the sentence ordering task, the strongest model InternVL2.5-78B-MPO is already close to human performance. Humans exhibit substantial agreement among themselves on the main tasks, with Fleiss’ kappa (Landis and Koch, 1977) above 0.68 across the board.

4.3 Further Analysis

Impact of temporal expressions We further analyze how the models understand and reason about different types of statements in the two-event and three-event relation inference tasks (MT1). We select the top five models on these two tasks for comparison (Table 5). When comparing explicit and implicit temporal event statements (cf. Table 1), we observe that models consistently perform better on the former. This could be because the presence of temporal adverbs or conjunctions in a sentence helps clarify the order in which events happen. The top five models always achieve higher accuracy on positive examples than on negative ones. Despite the even distribution of “True” and “False” across options “A” and “B” in our benchmark, models exhibit a tendency to predict “True” more frequently. By incorporating adversarial samples, our TempVS benchmark effectively reveals MLLMs’ biases toward certain answers, providing a robust assessment of their compositional temporal reasoning capabilities.

We also observe the better performance of models on explicitly marked temporal relations involving before (resp. after) and on then (resp. earlier). The key difference between these pairs of complementary temporal adverbials is that with before and
<table>
  <tr>
    <th rowspan="2">Tasks</th>
    <th rowspan="2">Models</th>
    <th colspan="2">InternVL2.5-26B-MPO</th>
    <th colspan="2">InternVL2.5-78B-MPO</th>
    <th colspan="2">llava-onevision-72b-ov</th>
    <th colspan="2">LLaVA-NeXT-Video-34B</th>
    <th colspan="2">GPT-4o</th>
    <th colspan="2">Top-5 Average</th>
  </tr>
  <tr>
    <th>Statement Type</th>
    <th>Pos</th>
    <th>Neg</th>
    <th>Pos</th>
    <th>Neg</th>
    <th>Pos</th>
    <th>Neg</th>
    <th>Pos</th>
    <th>Neg</th>
    <th>Pos</th>
    <th>Neg</th>
    <th>Pos</th>
    <th>Neg</th>
    <th>Both</th>
  </tr>
  <tr>
    <td rowspan="4">Two-event Relation</td>
    <td>after</td>
    <td></td>
    <td>59.9</td>
    <td>62.9</td>
    <td>70.7</td>
    <td>49.1</td>
    <td>71.4</td>
    <td>50.6</td>
    <td>77.9</td>
    <td>37.1</td>
    <td>56.2</td>
    <td>60.4</td>
    <td>67.2</td>
    <td>52.0</td>
    <td>59.6</td>
  </tr>
  <tr>
    <td>before</td>
    <td></td>
    <td>66.4</td>
    <td>66.7</td>
    <td>71.6</td>
    <td>57.9</td>
    <td>70.6</td>
    <td>59.0</td>
    <td>77.1</td>
    <td>41.4</td>
    <td>53.3</td>
    <td>73.4</td>
    <td>67.8</td>
    <td>59.7</td>
    <td>63.8</td>
  </tr>
  <tr>
    <td>earlier</td>
    <td></td>
    <td>70.8</td>
    <td>47.0</td>
    <td>74.3</td>
    <td>36.8</td>
    <td>74.1</td>
    <td>41.5</td>
    <td>85.9</td>
    <td>28.8</td>
    <td>69.1</td>
    <td>38.5</td>
    <td>74.8</td>
    <td>38.5</td>
    <td>56.7</td>
  </tr>
  <tr>
    <td>then</td>
    <td></td>
    <td>65.5</td>
    <td>59.3</td>
    <td>73.3</td>
    <td>44.8</td>
    <td>71.3</td>
    <td>50.0</td>
    <td>86.4</td>
    <td>35.2</td>
    <td>61.0</td>
    <td>63.1</td>
    <td>71.5</td>
    <td>50.5</td>
    <td>61.0</td>
  </tr>
  <tr>
    <td rowspan="4">Three-event Relation</td>
    <td>before/after</td>
    <td></td>
    <td>66.7</td>
    <td>72.3</td>
    <td>70.7</td>
    <td>64.0</td>
    <td>77.7</td>
    <td>54.6</td>
    <td>71.2</td>
    <td>72.0</td>
    <td>55.7</td>
    <td>78.2</td>
    <td>65.1</td>
    <td>67.2</td>
    <td>66.2</td>
  </tr>
  <tr>
    <td>first/second/third</td>
    <td></td>
    <td>57.5</td>
    <td>70.4</td>
    <td>63.4</td>
    <td>67.4</td>
    <td>76.3</td>
    <td>43.9</td>
    <td>68.4</td>
    <td>64.1</td>
    <td>50.4</td>
    <td>78.9</td>
    <td>60.4</td>
    <td>65.1</td>
    <td>65.0</td>
  </tr>
  <tr>
    <td>later/finally</td>
    <td></td>
    <td>75.3</td>
    <td>56.3</td>
    <td>87.5</td>
    <td>32.5</td>
    <td>77.4</td>
    <td>52.8</td>
    <td>100.0</td>
    <td>0.1</td>
    <td>69.4</td>
    <td>68.7</td>
    <td>79.8</td>
    <td>43.1</td>
    <td>61.5</td>
  </tr>
  <tr>
    <td>implicit</td>
    <td></td>
    <td>73.3</td>
    <td>27.1</td>
    <td>82.6</td>
    <td>20.3</td>
    <td>78.7</td>
    <td>28.4</td>
    <td>100.0</td>
    <td>0.3</td>
    <td>70.3</td>
    <td>43.7</td>
    <td>76.5</td>
    <td>28.4</td>
    <td>52.4</td>
  </tr>
</table>

Table 5: Fine-grained accuracy of different statement types in the two-event and three-event relation inference tasks. “Pos” denotes positive examples, while “Neg” represents negative examples. The higher accuracy score is highlighted in gray for each pair of positive and negative statements.

<table>
  <tr>
    <th></th>
    <th colspan="2">InternVL2.5-78B</th>
    <th colspan="2">w/o CoT</th>
    <th colspan="2">w/ CoT</th>
    <th colspan="2">CoT-Δ</th>
    <th colspan="2">w/o CoT</th>
    <th colspan="2">w/ CoT</th>
    <th colspan="2">CoT-Δ</th>
  </tr>
  <tr>
    <th></th>
    <th>MT1</th>
    <th>MT2</th>
    <th>MT1</th>
    <th>MT2</th>
    <th>MT1</th>
    <th>MT2</th>
    <th>MT1</th>
    <th>MT2</th>
    <th>MT1</th>
    <th>MT2</th>
    <th>MT1</th>
    <th>MT2</th>
    <th>MT1</th>
    <th>MT2</th>
  </tr>
  <tr>
    <td>Two-event relation (MT1)</td>
    <td>54.23</td>
    <td>56.13</td>
    <td>+1.90</td>
    <td>58.27</td>
    <td>60.43</td>
    <td>+2.16</td>
  </tr>
  <tr>
    <td>Three-event relation (MT1)</td>
    <td>56.79</td>
    <td>57.20</td>
    <td>+0.41</td>
    <td>64.52</td>
    <td>63.38</td>
    <td>-1.14</td>
  </tr>
  <tr>
    <td>Sentence Ordering (MT2)</td>
    <td>71.09</td>
    <td>85.42</td>
    <td>+14.33</td>
    <td>61.50</td>
    <td>81.41</td>
    <td>+19.91</td>
  </tr>
  <tr>
    <td>Image Ordering (MT3)</td>
    <td>38.49</td>
    <td>47.54</td>
    <td>+9.05</td>
    <td>22.97</td>
    <td>33.77</td>
    <td>+10.80</td>
  </tr>
</table>

Table 6: Model performance comparison with and without Chain of Thought (CoT).

even though separators were inserted between images in the input.

Prompting with Chain-of-Thought Chain-of-Thought (CoT, Wei et al., 2022) is a widely used approach to enhance models’ reasoning ability, by allowing models to generate intermediate reasoning steps before producing the final answer. It may thus also enhance models’ temporal reasoning skills on TempVS. We conduct CoT experiments using InternVL2.5-78B and GPT-4o. The detailed prompts are listed in Appendix C. As shown in Table 6, CoT yields large gains on sentence and image ordering, but limited improvement for event relation inference (MT1). This indicates the potential of step-by-step reasoning for ordering tasks. However, simple CoT does not help event relation inference. We leave the investigation of methods to enhance models’ understanding of this complex task and improve its temporal reasoning capabilities for future work.

Qualitative Case Analysis Figure 4 presents task-specific qualitative results among four representative open-source MLLMs, i.e., InternVL2.5-78B-MPO, LLaVA-OneVision-ov-72b, Qwen2-VL-Instruct-72B and GPT-4o. Further analysis reveals that the unsatisfactory performance of current models stems from: (1) weak event grounding, where models fail to identify the correct image corresponding to an event; (2) poor instruction-following abilities, resulting in outputs that do not follow the required format; (3) a limited under-

Figure 3: Accuracy of the two-event relation inference task with different distances between the two events in a sequence.

then, the order of events in text mirrors their order in the image sequence, whereas this is not the case with after/earlier. We therefore see some evidence of an iconicity effect: temporal relations are easier for models when the surface order of events mirrors their actual order (in the visual modality). This echoes similar findings in the discourse processing and psycholinguistic literature on narrative comprehension (Zwaan and Radvansky, 1998; Smith, 2003). It also points to an important avenue for future research in fine-grained multimodal benchmarking, namely, in cases where surface characteristics in two modalities are not perfectly aligned.

Distance between two events Figure 3 presents the accuracy of the two-event relation inference task as the distance between events in the original sequence increases from one to four. Nearly all models improve in performance as the distance increases. On the one hand, more distant visual events in the dataset are typically more visually distinct. On the other hand, the models may fail to effectively separate two closely spaced images,
MLLM development by uncovering weaknesses in multi-event temporal reasoning in multi-image scenarios, while TempVS provides a valuable resource for further research.

Limitations Despite a careful examination of publicly available repositories, technical reports and papers, we find no evidence that the evaluated MLLMs were trained on the data included in TempVS. However, for models (such as GPT-4o) that have not fully disclosed or explicitly stated their training data, the possibility of data leakage and contamination remains unclear. This could potentially lead to an overestimation of their advantages. Additionally, we focus on multiple-choice questions to ensure structured evaluation and clear correctness criteria, following existing multimodal benchmarks (Wang et al., 2025; Meng et al., 2025; Fu et al., 2023; Liu et al., 2024c; Li et al., 2024b). However, other question types, such as open-ended questions, particularly the evaluation of open-ended generation in multi-image temporal understanding and reasoning, are also worth exploring. We leave these for future work.

Ethical Considerations In this study, we employ published datasets and pretrained multimodal large language models, with no known significant ethical concerns regarding their usage. However, we acknowledge that biases in the original image-caption data may influence both the models and their evaluations. Our research has received approval from our institution’s Ethics Board, ensuring compliance with ethical guidelines for human annotation process. Additionally, all collected human-annotated data has been deidentified to protect participants’ data privacy and security.

Acknowledgments We thank the anonymous reviewers for their valuable comments and constructive feedback. We are also grateful to Letitia Parcalabescu for her insightful suggestions and engaging discussions, which helped us improve the quality of this work. Our thanks go to the members of UU NLP Group for their assistance with the pilot human evaluation experiments and for their helpful suggestions on earlier drafts of the manuscript.

References

Marah Abdin, Jyoti Aneja, Hany Awadallah, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat
Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

AI@Meta. 2024. Llama 3 model card.

Emanuele Bugliarello, Laurent Sartran, Aishwarya Agrawal, Lisa Anne Hendricks, and Aida Nematzadeh. 2023. Measuring progress in fine-grained vision-and-language understanding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1559–1582, Toronto, Canada. Association for Computational Linguistics.

Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-pro: Unified multimodal understanding and generation with data and model scaling.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Intervil: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. 2023. MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306.13394.

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148–166. Springer.

Team Gemini, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. 2018. Imagine this! scripts to compositions to videos. In Proceedings of the European conference on computer vision (ECCV), pages 598–613.

Xudong Hong, Asad Sayeed, Khushboo Mehra, Vera Demberg, and Bernt Schiele. 2023. Visual writing prompts: Character-grounded story generation with curated image sequences. Transactions of the Association for Computational Linguistics, 11:565–581.

Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. 2016. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1233–1239.

Yupan Huang, Zaiqiao Meng, Fangyu Liu, Yixuan Su, Nigel Collier, and Yutong Lu. 2024. Sparkles: Unlocking chats across multiple images for multimodal instruction-following models.

Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max W.F. Ku, Qian Liu, and Wenhui Chen. 2024. Mantis: Interleaved multi-image instruction tuning. Transactions on Machine Learning Research, 2024.

J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. Llava-onevision: Easy visual task transfer. Preprint. arXiv:2408.03326.

Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024b. Seedbench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308.

Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024c. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. Preprint, arXiv:2407.07895.

Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, et al. 2024d. A survey on benchmarks of multimodal large language models. arXiv preprint arXiv:2408.08632.

Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wengiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. 2024e. Fine-tuning multimodal LLMs to follow zero-shot demonstrative instructions. In The Twelfth International Conference on Learning Representations.
Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao. 2019. Storygan: A sequential conditional gan for story visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6329–6338.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024a. Llava-next: Improved reasoning, ocr, and world knowledge.

Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, et al. 2024b. Mbench: Evaluating multimodal large language models over multiple images. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22417–22428.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2024c. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun-yuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations.

Fanqing Meng, Chuanhao Li, Jin Wang, Quanfeng Lu, Hao Tian, Tianshuo Yang, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. 2025. MMIU: Multimodal multi-image understanding for evaluating large vision-language models. In The Thirteenth International Conference on Learning Representations.

OpenAI. 2022. Introducing chatgpt.

QwenTeam. 2024. Qwen2.5: A party of foundation models.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Carlota Smith. 2003. Modes of Discourse. Cambridge University Press, Cambridge.

Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. 2024. Milebench: Benchmarking MLLMs in long context. In First Conference on Language Modeling.

Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418–6428.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhaao Chen. 2025. Muirbench: A comprehensive benchmark for robust multi-image understanding. In The Thirteenth International Conference on Learning Representations.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi-hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyong Lin. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Xiyan Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Fuxiao Liu, Gedas Bertasius, Mohit Bansal, Huaxiu Yao, and Furong Huang. 2024b. Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 416–442. Association for Computational Linguistics.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huauzzo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. 2024. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. Preprint, arXiv:2412.10302.
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, jiayi lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. 2024. MMT-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask AGI. In Forty-first International Conference on Machine Learning.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567.

Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024a. Long context transfer from language to vision. CoRR, abs/2406.16852.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. 2024b. Llava-next: A strong zero-shot video understanding model.

Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations.

Rolf A. Zwaan and Gabriel A. Radvansky. 1998. Situation models in language comprehension and memory. Psychological Bulletin, 123(2):162–85.
A  Details in Data Filtering

A.1  Stative Verbs

We filter out the samples where any form of a stative verb exists (as displayed in Table 7) in the captions to avoid describing a state. In contrast, we only keep the examples with dynamic verbs to describe actions.

<table>
  <tr>
    <th>Base Form</th>
    <th>Present Participle</th>
    <th>3rd Person Singular</th>
    <th>Past Tense</th>
    <th>Past Participle</th>
  </tr>
  <tr><td>wish</td><td>wishing</td><td>wishes</td><td>wished</td><td>wished</td></tr>
  <tr><td>equal</td><td>equaling</td><td>equals</td><td>equaled</td><td>equaled</td></tr>
  <tr><td>signify</td><td>signifying</td><td>signifies</td><td>signified</td><td>signified</td></tr>
  <tr><td>feel</td><td>feeling</td><td>feels</td><td>felt</td><td>felt</td></tr>
  <tr><td>involve</td><td>involving</td><td>involves</td><td>involved</td><td>involved</td></tr>
  <tr><td>sense</td><td>sensing</td><td>senses</td><td>sensed</td><td>sensed</td></tr>
  <tr><td>sound</td><td>sounding</td><td>sounds</td><td>sounded</td><td>sounded</td></tr>
  <tr><td>detect</td><td>detecting</td><td>detects</td><td>detected</td><td>detected</td></tr>
  <tr><td>want</td><td>wanting</td><td>wants</td><td>wanted</td><td>wanted</td></tr>
  <tr><td>see</td><td>seeing</td><td>sees</td><td>saw</td><td>seen</td></tr>
  <tr><td>forget</td><td>forgetting</td><td>forgets</td><td>forgot</td><td>forgot</td></tr>
  <tr><td>matter</td><td>mattering</td><td>matters</td><td>mattered</td><td>mattered</td></tr>
  <tr><td>contain</td><td>containing</td><td>contains</td><td>contained</td><td>contained</td></tr>
  <tr><td>own</td><td>owning</td><td>owns</td><td>owned</td><td>owned</td></tr>
  <tr><td>taste</td><td>tasting</td><td>tastes</td><td>tasted</td><td>tasted</td></tr>
  <tr><td>dislike</td><td>disliking</td><td>dislikes</td><td>disliked</td><td>disliked</td></tr>
  <tr><td>remember</td><td>remembering</td><td>remembers</td><td>remembered</td><td>remembered</td></tr>
  <tr><td>resemble</td><td>resembling</td><td>resembles</td><td>resembled</td><td>resembled</td></tr>
  <tr><td>think</td><td>thinking</td><td>thinks</td><td>thought</td><td>thought</td></tr>
  <tr><td>envy</td><td>envying</td><td>envies</td><td>envied</td><td>envied</td></tr>
  <tr><td>depend</td><td>depending</td><td>depends</td><td>depended</td><td>depended</td></tr>
  <tr><td>hate</td><td>hating</td><td>hates</td><td>hated</td><td>hated</td></tr>
  <tr><td>know</td><td>knowing</td><td>knows</td><td>knew</td><td>known</td></tr>
  <tr><td>require</td><td>requiring</td><td>requires</td><td>required</td><td>required</td></tr>
  <tr><td>love</td><td>loving</td><td>loves</td><td>loved</td><td>loved</td></tr>
  <tr><td>appreciate</td><td>appreciating</td><td>appreciates</td><td>appreciated</td><td>appreciated</td></tr>
  <tr><td>need</td><td>needing</td><td>needs</td><td>needed</td><td>needed</td></tr>
  <tr><td>concern</td><td>concerning</td><td>concerns</td><td>concerned</td><td>concerned</td></tr>
  <tr><td>span</td><td>spanning</td><td>spans</td><td>spanned</td><td>spanned</td></tr>
  <tr><td>appear</td><td>appearing</td><td>appears</td><td>appeared</td><td>appeared</td></tr>
  <tr><td>owe</td><td>owing</td><td>owes</td><td>owed</td><td>owed</td></tr>
  <tr><td>weigh</td><td>weighing</td><td>weighs</td><td>weighed</td><td>weighed</td></tr>
  <tr><td>disagree</td><td>disagreeing</td><td>disagrees</td><td>disagreed</td><td>disagreed</td></tr>
  <tr><td>become</td><td>becoming</td><td>becomes</td><td>became</td><td>become</td></tr>
  <tr><td>measure</td><td>measuring</td><td>measures</td><td>measured</td><td>measured</td></tr>
  <tr><td>possess</td><td>possessing</td><td>possesses</td><td>possessed</td><td>possessed</td></tr>
  <tr><td>like</td><td>liking</td><td>likes</td><td>liked</td><td>liked</td></tr>
  <tr><td>look</td><td>looking</td><td>looks</td><td>looked</td><td>looked</td></tr>
  <tr><td>imagine</td><td>imagining</td><td>imagines</td><td>imagined</td><td>imagined</td></tr>
  <tr><td>mind</td><td>minding</td><td>minds</td><td>mined</td><td>mined</td></tr>
  <tr><td>belong</td><td>belonging</td><td>belongs</td><td>belonged</td><td>belonged</td></tr>
  <tr><td>loathe</td><td>loatheing</td><td>loathes</td><td>loathed</td><td>loathed</td></tr>
  <tr><td>hate</td><td>hating</td><td>hates</td><td>hated</td><td>hated</td></tr>
  <tr><td>deserve</td><td>deserving</td><td>deserves</td><td>deserved</td><td>deserved</td></tr>
  <tr><td>mean</td><td>meaning</td><td>means</td><td>meant</td><td>meant</td></tr>
  <tr><td>promise</td><td>promising</td><td>promises</td><td>promised</td><td>promised</td></tr>
  <tr><td>believe</td><td>believing</td><td>believes</td><td>believed</td><td>believed</td></tr>
  <tr><td>prefer</td><td>preferring</td><td>prefers</td><td>preferred</td><td>preferred</td></tr>
  <tr><td>cost</td><td>costing</td><td>costs</td><td>costed</td><td>costed</td></tr>
  <tr><td>hope</td><td>hoping</td><td>hopes</td><td>hoped</td><td>hoped</td></tr>
  <tr><td>recognize</td><td>recognizing</td><td>recognizes</td><td>recognized</td><td>recognized</td></tr>
  <tr><td>include</td><td>including</td><td>includes</td><td>included</td><td>included</td></tr>
  <tr><td>support</td><td>supporting</td><td>supports</td><td>supported</td><td>supported</td></tr>
  <tr><td>understand</td><td>understanding</td><td>understands</td><td>understood</td><td>understood</td></tr>
  <tr><td>comprise</td><td>comprising</td><td>comprises</td><td>comprised</td><td>comprised</td></tr>
  <tr><td>agree</td><td>agreeing</td><td>agrees</td><td>agreed</td><td>agreed</td></tr>
  <tr><td>realize</td><td>realizing</td><td>realizes</td><td>realized</td><td>realized</td></tr>
  <tr><td>value</td><td>valuing</td><td>values</td><td>valued</td><td>valued</td></tr>
  <tr><td>seem</td><td>seeming</td><td>seems</td><td>seemed</td><td>seemed</td></tr>
  <tr><td>hear</td><td>hearing</td><td>hears</td><td>heard</td><td>heard</td></tr>
  <tr><td>doubt</td><td>doubting</td><td>doubts</td><td>doubted</td><td>doubted</td></tr>
  <tr><td>consist</td><td>consisting</td><td>consists</td><td>consisted</td><td>consisted</td></tr>
  <tr><td>smell</td><td>smelling</td><td>smells</td><td>smelled</td><td>smelled</td></tr>
</table>

Table 7: Full list of stative verbs.

A.2  Threshold Values

Due to the differences in image and text styles across datasets caused by their respective domains, we determined the threshold values for text similarity and image similarity for each dataset through manual inspection and empirical tuning, as shown in the Table 8. For example, in FlintstonesSV, for text similarity, two texts are considered dissimilar if their BERTScore precision and recall are both below 0.98, and their F1 score is below 0.96. Similarly, for image similarity, two images are considered dissimilar if their CLIP similarity score is below 0.94.

<table>
  <tr>
    <th></th>
    <th>BERTScore</th>
    <th>CLIP Similarity</th>
  </tr>
  <tr>
    <td>FlintstonesSV</td>
    <td><0.98</td>
    <td><0.98</td>
    <td><0.96</td>
    <td><0.94</td>
  </tr>
  <tr>
    <td>PororoSV</td>
    <td><0.96</td>
    <td><0.96</td>
    <td><0.95</td>
    <td><0.90</td>
  </tr>
  <tr>
    <td>VWP</td>
    <td><0.98</td>
    <td><0.98</td>
    <td><0.97</td>
    <td><0.95</td>
  </tr>
  <tr>
    <td>VIST</td>
    <td><0.92</td>
    <td><0.92</td>
    <td><0.90</td>
    <td><0.88</td>
  </tr>
</table>

Table 8: The similarity threshold values used in data filtering.

A.3  Prompt used to extract events

The main difference between extracted events and original captions lies in the level of detail: the former contains only the essential event information, such as who did what; the latter, are written by human annotators as complete stories/narratives. Original captions often contain temporal connectives such as "before" "then" or "finally" which may enable models to infer multi-event relations and perform sentence ordering based solely on textual cues.

The prompt we used to extract the events from the original captions by GPT-4 is as follows:

Given a caption: [the original caption here]. Extract a single, concise and clear event sentence from the provided caption. Ensure the returned sentence satisfies the following criteria: (1) The sentence should contain the event itself without phrases such as "the event is" or "event: ". (2) If there are multiple events, extract only a single major event. (3) The sentence must contain only one clause with only one verb. (4) The event should be expressed in simple past tense. (5) If no event is detected, return 'NO_EVENT'.

A.4  Data Statistics after Each Filtering Step

In Table 9, we show the statistics of image sequences left after each data filtering step.
<table>
  <tr>
    <th></th>
    <th>FlintstonesSV</th>
    <th>PororoSV</th>
    <th>VWP</th>
    <th>VIST</th>
  </tr>
  <tr>
    <td>Original image sequences</td>
    <td>24,433</td>
    <td>11,444</td>
    <td>12,627</td>
    <td>49,700</td>
  </tr>
  <tr>
    <td>No stative verbs</td>
    <td>10,378</td>
    <td>3,114</td>
    <td>1,995</td>
    <td>20,294</td>
  </tr>
  <tr>
    <td>No starting pronoun</td>
    <td>10,105</td>
    <td>2,952</td>
    <td>914</td>
    <td>12,216</td>
  </tr>
  <tr>
    <td>No similar text</td>
    <td>3,092</td>
    <td>1,952</td>
    <td>815</td>
    <td>2,906</td>
  </tr>
  <tr>
    <td>No similar image</td>
    <td>636</td>
    <td>644</td>
    <td>809</td>
    <td>2,315</td>
  </tr>
  <tr>
    <td>No ambiguous image-text</td>
    <td>633</td>
    <td>535</td>
    <td>686</td>
    <td>2,284</td>
  </tr>
  <tr>
    <td>No repetitive image sequences</td>
    <td>633</td>
    <td>535</td>
    <td>498</td>
    <td>1,880</td>
  </tr>
  <tr>
    <td>Without No. EVENT</td>
    <td>612</td>
    <td>535</td>
    <td>417</td>
    <td>1,292</td>
  </tr>
  <tr>
    <td>Final image sequences after manual check</td>
    <td>565</td>
    <td>395</td>
    <td>316</td>
    <td>809</td>
  </tr>
  <tr>
    <td>Final two-event groups</td>
    <td>2104</td>
    <td>864</td>
    <td>850</td>
    <td>3742</td>
  </tr>
  <tr>
    <td>Final three-event groups</td>
    <td>916</td>
    <td>172</td>
    <td>208</td>
    <td>830</td>
  </tr>
</table>

Table 9: The number of image sequences of each step in data filtering process.

B Human Performance Survey

We designed three questionnaires for the human performance survey, corresponding to the three main tasks: event relation inference, sentence ordering, and image ordering. To ensure that participants fully understand the tasks, we provided task instructions and two sample questions at the beginning of each questionnaire (as shown in Figure 5). Additionally, Figure 6 presents sample questions that participants were required to answer. We randomly selected 280 image sequences from TempVS benchmark and collected three responses for each sequence from different annotators. Participants in MT1 were required to complete 30 questions, while participants in the other two ordering tasks completed 20 questions each. The median completion time was approximately 20 minutes, ensuring that long-time focus did not negatively impact participants’ judgment. We recruited 36 annotators (18 females, 18 males) via Prolific at a hourly rate of £17.1, all of whom were proficient in English and had at least a college-level education.

Task Introduction: Multi-event Relation Inference

In this task, you will evaluate the accuracy and consistency of textual statements in relation to a given sequence of images. Your goal is to determine whether the provided statement is fully supported by the visual evidence presented in the images.

Each task will present you with:

• A sequence of images displayed from left to right in the timeline.
• A textual statement describing an event or situation.

The key is to focus on whether the sequence of events described in the text matches the order of events depicted in the image sequence from left to right. If they are consistent, select True; otherwise, select False.

You will see two examples of the tasks in the next page.

![A sequence of cartoon images showing characters Betty, Wilma, Barney, and Fred, with a statement about their actions.](page_1012_1012_420_180.png)

Statement: Betty talked to Barney in a ballroom in front of a sign advertising an awards show after Wilma angrily talked to an unseen person outside.
Answer: True
Explanation: 'Betty talked to barney in a ballroom in front of a sign advertising an awards show' is located in Image 3, 'Wilma angrily talked to an unseen person outside' is located in Image 1, so it is True. Note that Wilma is the woman with red hair, Betty is the woman with black hair, Barney is the man with yellow hair.

Figure 5: An example of task instruction and example question shown in the beginning of human questionnaire

C Prompts

Prompt Template  The prompt examples we used for each task are shown below. Our prompt consists of Character Description (only for image sequences from FlintstonesVS and PororoVS), Task Instruction, Question Text, Response Format and Options (only for main tests), and a prefix indicating the beginning of the answer.
Following the text to give the correct order of the shuffled images: The florist set up the wedding early. The photographer took a picture of the cake. The bride waited nervously outside before beginning. It's time to catch the bouquet. They went off by a car.

O A. Image e -> Image b -> Image a -> Image d -> Image c
O B. Image b -> Image c -> Image d -> Image e -> Image a
O C. Image e -> Image c -> Image a -> Image d -> Image b
O D. Image b -> Image d -> Image c -> Image e -> Image a
O E. Image b -> Image d -> Image e -> Image a -> Image c

Figure 6: Example question interface used by the human annotators

MT1: Event Relation Inference
[Character Description] Description of character appearance in the images: Pororo is a gentoo penguin with an orange-yellow beak wearing a tan aviator’s helmet and orange goggles.
[Task Instruction] Is the statement completely accurate and consistent with the content in the sequence of images? When making the choice, focus on the evidence presented in the sequence of images from left to right.
[Question Text] The statement is: A snowball dropped down a small snow hill after Pororo threw a snowball.
[Response Format] Submit only the right option letter as your answer, e.g., Option [Letter].
[Options] Options are: A. True; B. False. The answer is:

GT: Grounding Test
User:
[Character Description] Description of character appearance in the images: Fred is chubby, has black hair, a large nose and wears an orange and black spotted short-sleeved loincloth with a blue scarf.
[Task Instruction] Pick the image from the image sequence that accurately represents the event. When making the choice, focus on the evidence presented in the sequence of images from left to right.
[Question Text] The event is: Fred was sitting in a room on a stool.
[Response Format] Submit the right number of the image in the sequence as your answer only without additional reasoning or repetition of the instructions.
The answer is:
MT2: Sentence Ordering
[Task Instruction] You are given an ordered image sequence and several sentences in a random order. Your task is to analyze the content of the sequence of images from left to right and rearrange the sentences into the correct chronological or logical order. Read the image sequence from left to right. Use the images’ content to guide your sentence ordering. Avoid assumptions not supported by the image sequence or sentences.
[Question Text] The shuffled sentences are:
Sentence a: John prepares to shoot again and fires.
Sentence b: The shadowy figure is hit with the rifle blast.
Sentence c: John takes aim at a shadowy figure up above.
Sentence d: Ken stops the horses and prepares to leave the stagecoach.
Sentence e: Ken is approaching a house out on the prairie on his stagecoach with his two horses.
[Response Format] Do not provide explanations or repeat the prompt. Select from the following options and your answer should only be in the format: Option [Letter].
[Options]
A. Sentence c -> Sentence b -> Sentence a -> Sentence e -> Sentence d;
B. Sentence b -> Sentence e -> Sentence d -> Sentence a -> Sentence c;
C. Sentence e -> Sentence b -> Sentence c -> Sentence d -> Sentence a;
D. Sentence c -> Sentence a -> Sentence e -> Sentence d -> Sentence b;
E. Sentence d -> Sentence c -> Sentence e -> Sentence b -> Sentence a.
The answer is:

MT3: Image Ordering
[Task Instruction] The text narrates a story or event sequence. Use your vision-language reasoning to reorder the images to reflect the narrative structure. Carefully read the provided text. Focus on the events, actions, and details described to reorder the images logically. The images are labeled in order as Image a, Image b, Image c, Image d, and Image e, and so on if there are more photos.
[Question Text] The events are: Some people came for the family gathering today. The girls enjoyed some fruit. We played on the swings. The boys lounged in the chair. Grandpa put his grandson on his knee.
[Response Format] Select from the following options and your answer should be in the format: 'Option [Letter]'. Respond with the correct option only, avoiding any explanations or repetition.
[Options]
A. Image b -> Image c -> Image d -> Image a -> Image e;
B. Image e -> Image d -> Image c -> Image b -> Image a;
C. Image b -> Image e -> Image d -> Image a -> Image c;
D. Image a -> Image c -> Image e -> Image b -> Image d;
E. Image c -> Image b -> Image a -> Image e -> Image d.
The answer is:

Variations of Each Prompt Component For each component in the prompt, we generate multiple variations (as listed in Table 10). By combining different components, we ultimately generate 328 distinct prompts.
Chain-of-Thought Prompts

<table>
  <tr>
    <th>Task</th>
    <th>CoT Prompt</th>
  </tr>
  <tr>
    <td><b>MT1</b></td>
    <td>Analyze the provided image sequence to determine whether the following statement is True or False. When making the choice, carefully examine the evidence presented in the sequence of images from left to right. First, describe the key details and changes observed in each image. Then, explain how these details support or contradict the given statement. Finally, based on your step-by-step reasoning, conclude whether the statement is True or False. Ensure your response follows this format: the reasoning process should be enclosed within <think> and </think> tags, and the final answer should be enclosed within <answer> and </answer> tags.</td>
  </tr>
  <tr>
    <td><b>MT2</b></td>
    <td>You are presented with an ordered image sequence and several sentences in random order. Your task is to determine the correct order of the sentences based on the context, events, or details observable in the images. Multiple-choice options are provided, with each option representing a possible sequence of the sentences. Think step by step, using the content of the images to guide your reasoning. Avoid assumptions not supported by the image sequence or the sentences.<br>Format your response as follows: Enclose your step-by-step reasoning process within <think> and </think> tags. Enclose the selected option (e.g., Option A, Option B) within <answer> and </answer> tags.</td>
  </tr>
  <tr>
    <td><b>MT3</b></td>
    <td>Order the following images in the correct sequence based on the content of the story. Compare each image with the text description, carefully analyzing the sequence of events to determine the proper order of the unordered images.<br>Response should be in this format:<br><think>First, examine the text description to identify key events and their chronological order. Next, analyze each image to match it with the corresponding event described in the text. Consider visual cues, actions, and details in the images that indicate the progression of the story. Arrange the images accordingly to reflect the correct sequence of events.</think><answer> Option [Your Choice] </answer></td>
  </tr>
</table>

Table 11: Chain-of-Thought prompts for different tasks.

D Experiments

D.1 Implementation Details

We evaluate 38 existing MLLMs for multi-event temporal grounding and reasoning, including both proprietary and open-source models. All open-source models are assessed using their official pre-trained versions available on HuggingFace. Detailed configurations of the open-source models evaluated are listed in Table 13. To minimize randomness, we set the temperature to 0. Experiments on open-source models are conducted using a single NVIDIA H100 GPU. For models exceeding 70B parameters, we adopt the mixed precision (FP16), otherwise the full precision.

Following Jiang et al. (2024), Wang et al. (2024b), Meng et al. (2025) and Wang et al. (2025), we merge sequential images into a single input, separating them with a thin white band and labeling each with an index (e.g., “Image 1” when images are sequentially ordered, or “Image a” when images are shuffled). In the preliminary experiments, we tested sequentially inputting images one at a time but observed minimal performance differences compared to merging them as shown in Table 12. The usage of a combined image was also motivated by practical constraints in current MLLMs, many of which support only a single image input. Moreover, due to limited context length in some models, inputting multiple images at once can result in token truncation and information loss (Song et al., 2024). In the released TempVS benchmark and evaluation toolkit, we provide both the individual image inputs and also the combined multi-image inputs, to allow people to use our benchmark to evaluate the models as they want.
Task Instruction - definition

Is the statement completely accurate and consistent with the content in the sequence of images?
- Analyze the provided image sequence to determine whether the following statement is True or False.
- Review the sequential images and decide whether the provided statement is True or False.

MT1
- Is the statement entirely consistent and supported by the images in the sequence?
- Examine the visual evidence in the provided sequence of images to determine whether the statement is True or False.
- Discuss the following: In the context of the presented unordered images? Taking the content in the image sequence into account, can you decide whether the statement is True or False?
- Given the multiple images provided in order, can you select the correct answer from True or False considering statement?

MT2
- You are given an ordered image sequence and several sentences in a random order. Your task is to analyze the content of the sequence of images from left to right and rearranging the sentences into the correct chronological or logical order.
- Here is an left-to-right image sequence and some sentences in random order. Your goal is to determine the correct order of the sentences based on the content of the images and the context. Consider the order of the sequential images to infer the logical or temporal sequence.
- The images are presented in a correct order, but the sentences are not. Your task is to reorder the sentences to match the sequence of events or details in the images from left to right.
- You are provided with a sequence of images and several randomly ordered sentences. Your task is to: 1. Understand the context of the image sequence;
- Identify which sentence relates to the image. 2. Rearrange the sentences to form a coherent order.
- Using your multimodal understanding and reasoning of the ordered image sequence and the randomly shuffled sentences, arrange the sentences in the correct sequence to match the flow of events in the image sequence.
- Presented is an image sequence in the order from left to right along with several unordered sentences. Your task is to determine the correct sequence of the sentences by analyzing the context of the images and the logical order of the visual elements in the images to deduce the logical or chronological order

MT3
- Rearrange the following images in the correct order based on content in the story
- The provided paragraphs describes a sequence of events. Arrange the images in the correct chronological order to match the story.
- The images are out of order compared to the text. Identify and reorder them to match the described sequence.
- The text narrates a story or event sequence. Use your vision-language reasoning to reorder the images to reflect the narrative structure.
- Using your understanding of the text and image content, arrange the images in the correct sequence to match the flow of events in the text.
- The paragraph describes events in a specific timeline. Use your visual reasoning to reorder the images in the correct sequence:

Base your answer decision on the image that best matches each of the following options:
- Pick the image from the image sequence that accurately represents the event.
- Which is the most fitting image for the described event? Pick from the choices below.
GT
- If you were to illustrate the event, which picture would you use?
- Find the image from the sequence of images that represents the event most precisely.
- Identify the picture that best represents the given event.

Task Instruction - requirement

When making the choice, focus on the evidence presented in the sequence of images from left to right.
- Choose the correct option based on the content in the sequential images from left to right.
- Only use the left-to-right content of the image sequence to inform the decision.
- Evaluate the statement strictly based on the information shown in the sequence of images from left to right.

MT1
- Read the image sequence from left to right. Use the images’ content to guide your sentence ordering. Avoid assumptions not supported by the image sequence or sentences.
- Understand images from left to right in the order. Focus on the visual content of the images to determine the correct order of the sentences.
- Make sure the sequence of images from left to right and their content to determine the correct order of the sentences.
- Interpret the images in their left-to-right order, focusing on their visual details to arrange the sentences correctly.
- Ensure that the reordered sentences create a logical and coherent narrative or description

MT2
- Carefully read the provided text and focus on the events described, and details described to reorder the images logically.
- Focus on the content of the images and events described in the images and the details described in the text.
- Identify key events and details from the text and use them to determine the proper order of the images.
- Compare the images with the text descriptions, focusing on the sequence of events to arrange unordered images correctly.

MT3
- Make your choice by considering the evidence shown in the sequence of images from left to right.
- Select the correct option using only the content presented in the sequential images from left to right.
GT
- Rely on the left-to-right order of the image sequence to guide your decision.
- Assess the statement exclusively based on the information depicted in the sequence of images from left to right

Response Format

No need to give reasoning process. Submit only the right option letter as your answer; e.g., Option [Letter]
MT1
- Do not tell the reasons of your decision. Provide the most suitable choice letter in the format of 'Option [Letter]' as your response only.
- Do not repeat the prompt or include reasons. Only return the correct option letter in the form of 'Option [Letter]' as your response.
- Please exclude explanations in the response. Offer the most proper chosen letter in the format of 'Option [Letter]' as your answer only.
MT2
- Do not provide explanations or repeat the prompt. Select from the following options and your answer should only be in the format: Option [Letter].
- Provide your answer from the following choices only in the format: 'Option [Letter]' without explanation or repeating the instructions.
- Choose the correct option from the choices provided and submit your answer only as Option [Letter], without including any explanations or repetition.
MT3
- Select from the following options and your answer should be in the format: 'Option [Letter]'. Respond with the correct option only, avoiding any explanations or repetition.
- Do not provide explanations or restate the question. Provide your answer from the following choices only in the format: Option [Letter].
- Choose the correct option from the choices provided below and submit your answer only as 'Option [Letter]', without any justifications or repetition of the prompt.
GT
- Submit the right number of the image in the sequence as your answer only without additional reasoning or repetition of the instructions.
- Provide only the most suitable image number as your response, avoiding any explanations or repetition.
- Only return the correct image number in the provided sequence without additional reasoning details or repetition of the instructions.

Table 10: All variations of different components we used to generate the prompts.

<table>
  <tr>
    <th></th>
    <th colspan="2">Two-event Relation Inference</th>
    <th colspan="2">Three-event Relation Inference</th>
    <th colspan="2">Sentence Ordering- event</th>
    <th colspan="2">Image Ordering - event</th>
  </tr>
  <tr>
    <th></th>
    <th>MT</th>
    <th>MT|GT<sub>strict</sub></th>
    <th>MT</th>
    <th>MT|GT<sub>strict</sub></th>
    <th>MT</th>
    <th>MT|GT<sub>strict</sub></th>
    <th>MT</th>
    <th>MT|GT<sub>strict</sub></th>
  </tr>
  <tr>
    <td>InternVL2_5-26B-MPO</td>
    <td>0.5962</td>
    <td>0.6053</td>
    <td>0.6089</td>
    <td>0.633</td>
    <td>0.6557</td>
    <td>0.8867</td>
    <td>0.3203</td>
    <td>0.392</td>
  </tr>
  <tr>
    <td>InternVL2_5-78B-MPO</td>
    <td>0.5972</td>
    <td>0.5811</td>
    <td>0.5948</td>
    <td>0.6013</td>
    <td>0.7378</td>
    <td>0.9543</td>
    <td>0.3697</td>
    <td>0.4527</td>
  </tr>
  <tr>
    <td>llava-interleave-qwen-7b</td>
    <td>0.5132</td>
    <td>0.5325</td>
    <td>0.5149</td>
    <td>0.521</td>
    <td>0.2807</td>
    <td>0.132</td>
    <td>0.2083</td>
    <td>0.1459</td>
  </tr>
  <tr>
    <td>Mantis-SB-Idetics2</td>
    <td>0.5052</td>
    <td>0.5154</td>
    <td>0.5111</td>
    <td>0.5377</td>
    <td>0.2217</td>
    <td>0</td>
    <td>0.1868</td>
    <td>0</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT-Video-34B-DPO</td>
    <td>0.5412</td>
    <td>0.5303</td>
    <td>0.5166</td>
    <td>0.5524</td>
    <td>0.2953</td>
    <td>0</td>
    <td>0.1975</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>LongVA-7B-DPO</td>
    <td>0.5479</td>
    <td>0.5437</td>
    <td>0.5257</td>
    <td>0.5508</td>
    <td>0.3371</td>
    <td>0.75</td>
    <td>0.1912</td>
    <td>0</td>
  </tr>
</table>

Table 12: Results of 6 MLLMs when multiple images are input separately into the models.
<table>
  <tr>
    <th>HuggingFace Model ID</th>
    <th># Params</th>
    <th>Vision Backbone</th>
    <th>Base LLM</th>
  </tr>
  <tr>
    <td>deepseek-ai/deepseek-vl2-tiny</td>
    <td>3.4B</td>
    <td>SigLIP-SO400M-384</td>
    <td>DeepSeekMoE</td>
  </tr>
  <tr>
    <td>deepseek-ai/deepseek-vl2-small</td>
    <td>16.1B</td>
    <td>SigLIP-SO400M-384</td>
    <td>DeepSeekMoE</td>
  </tr>
  <tr>
    <td>OpenGVLab/InternVL2_5-1B</td>
    <td>0.9B</td>
    <td>InternViT-300M-448px-V2_5</td>
    <td>Qwen2.5-0.5B-Instruct</td>
  </tr>
  <tr>
    <td>OpenGVLab/InternVL2_5-1B-MPO</td>
    <td>0.9B</td>
    <td>InternViT-300M-448px-V2_5</td>
    <td>Qwen2.5-0.5B-Instruct</td>
  </tr>
  <tr>
    <td>OpenGVLab/InternVL2_5-8B</td>
    <td>8.1B</td>
    <td>InternViT-300M-448px-V2_5</td>
    <td>internlm2_5-7b-chat</td>
  </tr>
  <tr>
    <td>OpenGVLab/InternVL2_5-8B-MPO</td>
    <td>8.1B</td>
    <td>InternViT-300M-448px-V2_5</td>
    <td>internlm2_5-7b-chat</td>
  </tr>
  <tr>
    <td>OpenGVLab/InternVL2_5-26B</td>
    <td>25.5B</td>
    <td>InternViT-6B-448px-V2_5</td>
    <td>internlm2_5-20b-chat</td>
  </tr>
  <tr>
    <td>OpenGVLab/InternVL2_5-26B-MPO</td>
    <td>25.5B</td>
    <td>InternViT-6B-448px-V2_5</td>
    <td>internlm2_5-20b-chat</td>
  </tr>
  <tr>
    <td>OpenGVLab/InternVL2_5-78B</td>
    <td>78.4B</td>
    <td>InternViT-6B-448px-V2_5</td>
    <td>Qwen2.5-72B-Instruct</td>
  </tr>
  <tr>
    <td>OpenGVLab/InternVL2_5-78B-MPO</td>
    <td>78.4B</td>
    <td>InternViT-6B-448px-V2_5</td>
    <td>Qwen2.5-72B-Instruct</td>
  </tr>
  <tr>
    <td>deepseek-ai/Janus-Pro-1B</td>
    <td>1.0B</td>
    <td>ViT-L-16-SigLIP-384</td>
    <td>DeepSeek-LLM-1.5b-base</td>
  </tr>
  <tr>
    <td>deepseek-ai/Janus-Pro-7B</td>
    <td>7.0B</td>
    <td>ViT-L-16-SigLIP-384</td>
    <td>DeepSeek-LLM-7b-base</td>
  </tr>
  <tr>
    <td>llava-hf/llava-interleave-qwen-0.5b-hf</td>
    <td>0.9B</td>
    <td>SigLIP-400M</td>
    <td>Qwen1.5-0.5B-Chat</td>
  </tr>
  <tr>
    <td>llava-hf/llava-interleave-qwen-7b-hf</td>
    <td>8.1B</td>
    <td>SigLIP-400M</td>
    <td>Qwen1.5-7B-Chat</td>
  </tr>
  <tr>
    <td>llava-hf/llava-interleave-qwen-7b-dpo-hf</td>
    <td>8.1B</td>
    <td>SigLIP-400M</td>
    <td>Qwen1.5-7B-Chat</td>
  </tr>
  <tr>
    <td>lmms-lab/llava-onevision-qwen2-0.5b-ov</td>
    <td>0.9B</td>
    <td>siglip-so400m-patch14-384</td>
    <td>Qwen2-0.5B</td>
  </tr>
  <tr>
    <td>lmms-lab/llava-onevision-qwen2-0.5b-si</td>
    <td>0.9B</td>
    <td>siglip-so400m-patch14-384</td>
    <td>Qwen2-0.5B</td>
  </tr>
  <tr>
    <td>lmms-lab/llava-onevision-qwen2-7b-ov</td>
    <td>8.0B</td>
    <td>siglip-so400m-patch14-384</td>
    <td>Qwen2-7B</td>
  </tr>
  <tr>
    <td>lmms-lab/llava-onevision-qwen2-7b-si</td>
    <td>8.0B</td>
    <td>siglip-so400m-patch14-384</td>
    <td>Qwen2-7B</td>
  </tr>
  <tr>
    <td>lmms-lab/llava-onevision-qwen2-72b-ov-sft</td>
    <td>73.2B</td>
    <td>siglip-so400m-patch14-384</td>
    <td>Qwen2-72B</td>
  </tr>
  <tr>
    <td>lmms-lab/llava-onevision-qwen2-72b-si</td>
    <td>73.2B</td>
    <td>siglip-so400m-patch14-384</td>
    <td>Qwen2-72B</td>
  </tr>
  <tr>
    <td>llava-hf/LLaVA-NeXT-Video-7B-hf</td>
    <td>7.1B</td>
    <td>SigLIP-400M</td>
    <td>vicuna-7b-v1.5</td>
  </tr>
  <tr>
    <td>llava-hf/LLaVA-NeXT-Video-7B-DPO-hf</td>
    <td>7.1B</td>
    <td>SigLIP-400M</td>
    <td>vicuna-7b-v1.5</td>
  </tr>
  <tr>
    <td>llava-hf/LLaVA-NeXT-Video-34B-hf</td>
    <td>34.8B</td>
    <td>SigLIP-400M</td>
    <td>vicuna-33b-v1.3</td>
  </tr>
  <tr>
    <td>llava-hf/LLaVA-NeXT-Video-34B-DPO-hf</td>
    <td>34.8B</td>
    <td>SigLIP-400M</td>
    <td>vicuna-33b-v1.3</td>
  </tr>
  <tr>
    <td>lmms-lab/LongVA-7B</td>
    <td>7.9B</td>
    <td>SigLIP-400M</td>
    <td>Qwen2-7B-Instruct</td>
  </tr>
  <tr>
    <td>lmms-lab/LongVA-7B-DPO</td>
    <td>7.9B</td>
    <td>SigLIP-400M</td>
    <td>Qwen2-7B-Instruct</td>
  </tr>
  <tr>
    <td>TIGER-Lab/Mantis-8B-Idefics2</td>
    <td>8.4B</td>
    <td>idefics2-8b</td>
    <td>Mistral-7B-v0.1</td>
  </tr>
  <tr>
    <td>TIGER-Lab/Mantis-8B-siglip-llama3</td>
    <td>8.5B</td>
    <td>SigLIP</td>
    <td>LLaMA-3-8B</td>
  </tr>
  <tr>
    <td>microsoft/Phi-3-3-vision-128k-instruct</td>
    <td>3.8B</td>
    <td>CLIP ViT-L/14</td>
    <td>Phi-3-mini-128k-instruct</td>
  </tr>
  <tr>
    <td>microsoft/Phi-3.5-vision-instruct</td>
    <td>4.2B</td>
    <td>CLIP ViT-L/14</td>
    <td>Phi-3.5-mini-instruct</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-VL-2B</td>
    <td>2.2B</td>
    <td>CLIP ViT-L/14</td>
    <td>Qwen2-1.5B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-VL-2B-Instruct</td>
    <td>2.2B</td>
    <td>CLIP ViT-L/14</td>
    <td>Qwen2-1.5B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-VL-7B</td>
    <td>7.6B</td>
    <td>CLIP ViT-L/14</td>
    <td>Qwen2-7B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-VL-7B-Instruct</td>
    <td>7.6B</td>
    <td>CLIP ViT-L/14</td>
    <td>Qwen2-7B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-VL-72B</td>
    <td>72.7B</td>
    <td>CLIP ViT-L/14</td>
    <td>Qwen2-72B</td>
  </tr>
  <tr>
    <td>Qwen/Qwen2-VL-72B-Instruct</td>
    <td>72.7B</td>
    <td>CLIP ViT-L/14</td>
    <td>Qwen2-72B</td>
  </tr>
</table>

Table 13: Details of all evaluated open-source MLLMs: HuggingFace model id, number of parameters, vision backbone, base LLM.
D.2 Evaluation Results of 38 MLLMs

In this section, we present the complete quantitative results of 38 state-of-the-art multimodal large language models (MLLMs) on the TempVS benchmark. The results for the MT1 event relation reasoning task are shown in Table 14, while Table 15 presents the results for the MT2 sentence ordering task and Table 16 reports the results for the MT3 image ordering task. It is important to note that GT represents the overall grounding evaluation, which measures the number of events correctly matched to their corresponding images across the entire benchmark. GT_{strict} denotes the strict grounding evaluation, which calculates the number of image sequences in which every event within a sequence is correctly matched to its corresponding image.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="4">Two-event Relation Inference</th>
    <th colspan="4">Three-event Relation Inference</th>
  </tr>
  <tr>
    <th>GT</th>
    <th>GT_{strict}</th>
    <th>MT</th>
    <th>MT|GT_{strict}</th>
    <th>GT</th>
    <th>GT_{strict}</th>
    <th>MT</th>
    <th>MT|GT_{strict}</th>
  </tr>
  <tr><td>deepseek-v12-tiny</td><td>0.4159</td><td>0.2082</td><td>0.4971</td><td>0.4965</td><td>0.4292</td><td>0.1052</td><td>0.4976</td><td>0.5064</td></tr>
  <tr><td>deepseek-v12-small</td><td>0.4116</td><td>0.1416</td><td>0.4311</td><td>0.4223</td><td>0.4306</td><td>0.0669</td><td>0.4436</td><td>0.4435</td></tr>
  <tr><td>InternVL2_5-1B</td><td>0.3636</td><td>0.1702</td><td>0.4102</td><td>0.4073</td><td>0.3896</td><td>0.085</td><td>0.3895</td><td>0.4270</td></tr>
  <tr><td>InternVL2_5-1B-MPO</td><td>0.3879</td><td>0.1881</td><td>0.3727</td><td>0.3709</td><td>0.4002</td><td>0.0953</td><td>0.3736</td><td>0.3569</td></tr>
  <tr><td>InternVL2_5-8B</td><td>0.5655</td><td>0.3976</td><td>0.5426</td><td>0.5540</td><td>0.6246</td><td>0.3278</td><td>0.5438</td><td>0.5556</td></tr>
  <tr><td>InternVL2_5-8B-MPO</td><td>0.6230</td><td>0.4697</td><td>0.5619</td><td>0.5736</td><td>0.6871</td><td>0.4118</td><td>0.558</td><td>0.5714</td></tr>
  <tr><td>InternVL2_5-26B</td><td>0.6181</td><td>0.4604</td><td>0.5705</td><td>0.5843</td><td>0.6824</td><td>0.3958</td><td>0.5835</td><td>0.6022</td></tr>
  <tr><td>InternVL2_5-26B-MPO</td><td>0.6521</td><td>0.5132</td><td>0.6032</td><td>0.6212</td><td>0.7096</td><td>0.4595</td><td>0.6212</td><td>0.6474</td></tr>
  <tr><td>InternVL2_5-78B</td><td>0.6555</td><td>0.5157</td><td>0.5423</td><td>0.5532</td><td>0.7154</td><td>0.4733</td><td>0.5679</td><td>0.5698</td></tr>
  <tr><td>InternVL2_5-78B-MPO</td><td>0.7046</td><td>0.5881</td><td>0.5847</td><td>0.5992</td><td>0.7771</td><td>0.5653</td><td>0.6139</td><td>0.6258</td></tr>
  <tr><td>Janus-Pro-1B</td><td>0.2359</td><td>0.0273</td><td>0.4827</td><td>0.4814</td><td>0.2558</td><td>0.0073</td><td>0.4645</td><td>0.4259</td></tr>
  <tr><td>Janus-Pro-7B</td><td>0.2399</td><td>0.0429</td><td>0.3509</td><td>0.3406</td><td>0.2555</td><td>0.0038</td><td>0.3287</td><td>0.3929</td></tr>
  <tr><td>llava-interleave-qwen-0.5b-hf</td><td>0.2172</td><td>0.0245</td><td>0.4976</td><td>0.4779</td><td>0.2182</td><td>0.0019</td><td>0.5036</td><td>0.5012</td></tr>
  <tr><td>llava-interleave-qwen-7b-hf</td><td>0.3521</td><td>0.1300</td><td>0.5161</td><td>0.5239</td><td>0.4112</td><td>0.0723</td><td>0.5007</td><td>0.4981</td></tr>
  <tr><td>llava-interleave-qwen-7b-dpo-hf</td><td>0.3531</td><td>0.1358</td><td>0.5198</td><td>0.5357</td><td>0.4102</td><td>0.0845</td><td>0.5173</td><td>0.5319</td></tr>
  <tr><td>llava-onevision-qwen2-0.5b-ov</td><td>0.304</td><td>0.0859</td><td>0.4528</td><td>0.4545</td><td>0.3343</td><td>0.0286</td><td>0.4807</td><td>0.4764</td></tr>
  <tr><td>llava-onevision-qwen2-0.5b-si</td><td>0.2153</td><td>0.0235</td><td>0.4496</td><td>0.4429</td><td>0.2156</td><td>0.0013</td><td>0.3587</td><td>0.3731</td></tr>
  <tr><td>llava-onevision-qwen2-7b-ov</td><td>0.5176</td><td>0.3278</td><td>0.5602</td><td>0.5804</td><td>0.578</td><td>0.2604</td><td>0.5753</td><td>0.5979</td></tr>
  <tr><td>llava-onevision-qwen2-7b-si</td><td>0.4575</td><td>0.2505</td><td>0.5292</td><td>0.5419</td><td>0.4895</td><td>0.1403</td><td>0.5546</td><td>0.5654</td></tr>
  <tr><td>llava-onevision-qwen2-72b-ov-sft</td><td>0.6215</td><td>0.4641</td><td>0.5928</td><td>0.6213</td><td>0.6844</td><td>0.405</td><td>0.6154</td><td>0.6349</td></tr>
  <tr><td>llava-onevision-qwen2-72b-si</td><td>0.5418</td><td>0.3593</td><td>0.5296</td><td>0.5393</td><td>0.6015</td><td>0.2804</td><td>0.5248</td><td>0.5298</td></tr>
  <tr><td>LLaVA-NeXT-Video-7B-hf</td><td>0.2731</td><td>0.0575</td><td>0.4603</td><td>0.4603</td><td>0.2972</td><td>0.0135</td><td>0.4486</td><td>0.4723</td></tr>
  <tr><td>LLaVA-NeXT-Video-7B-DPO-hf</td><td>0.2718</td><td>0.0596</td><td>0.4672</td><td>0.466</td><td>0.2981</td><td>0.014</td><td>0.4520</td><td>0.4615</td></tr>
  <tr><td>LLaVA-NeXT-Video-34B-hf</td><td>0.2742</td><td>0.0652</td><td>0.5847</td><td>0.5839</td><td>0.3053</td><td>0.0159</td><td>0.5947</td><td>0.5763</td></tr>
  <tr><td>LLaVA-NeXT-Video-34B-DPO-hf</td><td>0.3028</td><td>0.0825</td><td>0.5330</td><td>0.5386</td><td>0.3289</td><td>0.0205</td><td>0.5248</td><td>0.5263</td></tr>
  <tr><td>LongVA-7B</td><td>0.3015</td><td>0.0874</td><td>0.5468</td><td>0.5613</td><td>0.3138</td><td>0.0208</td><td>0.5596</td><td>0.6169</td></tr>
  <tr><td>LongVA-7B-DPO</td><td>0.3241</td><td>0.1132</td><td>0.5319</td><td>0.5582</td><td>0.3449</td><td>0.0386</td><td>0.5233</td><td>0.5350</td></tr>
  <tr><td>Mantis-8B-Ideics2</td><td>0.3452</td><td>0.1218</td><td>0.5193</td><td>0.533</td><td>0.3586</td><td>0.041</td><td>0.5197</td><td>0.5164</td></tr>
  <tr><td>Mantis-8B-siglip-llama3</td><td>0.2444</td><td>0.0443</td><td>0.5238</td><td>0.5368</td><td>0.2392</td><td>0.0065</td><td>0.5251</td><td>0.5833</td></tr>
  <tr><td>Phi-3-vision-128k-instruct</td><td>0.2226</td><td>0.0379</td><td>0.5196</td><td>0.5219</td><td>0.2332</td><td>0.0065</td><td>0.5132</td><td>0.4583</td></tr>
  <tr><td>Phi-3.5-vision-instruct</td><td>0.2225</td><td>0.0400</td><td>0.4904</td><td>0.4774</td><td>0.2316</td><td>0.0078</td><td>0.4877</td><td>0.4828</td></tr>
  <tr><td>Qwen2-VL-2B</td><td>0.3053</td><td>0.088</td><td>0.4935</td><td>0.4842</td><td>0.3188</td><td>0.0313</td><td>0.4717</td><td>0.4923</td></tr>
  <tr><td>Qwen2-VL-2B-Instruct</td><td>0.3815</td><td>0.1738</td><td>0.5271</td><td>0.5314</td><td>0.4003</td><td>0.0777</td><td>0.5051</td><td>0.5052</td></tr>
  <tr><td>Qwen2-VL-7B</td><td>0.4032</td><td>0.1664</td><td>0.5122</td><td>0.4982</td><td>0.4233</td><td>0.0899</td><td>0.4991</td><td>0.4603</td></tr>
  <tr><td>Qwen2-VL-7B-Instruct</td><td>0.5144</td><td>0.3239</td><td>0.5397</td><td>0.5542</td><td>0.5415</td><td>0.2124</td><td>0.5358</td><td>0.5534</td></tr>
  <tr><td>Qwen2-VL-72B</td><td>0.4906</td><td>0.2997</td><td>0.5461</td><td>0.5738</td><td>0.5167</td><td>0.1935</td><td>0.5429</td><td>0.5900</td></tr>
  <tr><td>Qwen2-VL-72B-Instruct</td><td>0.5078</td><td>0.3167</td><td>0.5395</td><td>0.5643</td><td>0.5427</td><td>0.2059</td><td>0.5563</td><td>0.6062</td></tr>
  <tr><td>GPT-4o</td><td>0.7043</td><td>0.6034</td><td>0.5827</td><td>0.6005</td><td>0.7807</td><td>0.5704</td><td>0.6452</td><td>0.6644</td></tr>
</table>

Table 14: Full results for MT1: event relation inference
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="4">Ordering Sentences (events)</th>
    <th colspan="4">Ordering Sentences (captions)</th>
  </tr>
  <tr>
    <th>GT</th>
    <th>GT<sub>strict</sub></th>
    <th>MT</th>
    <th>MTI GT<sub>strict</sub></th>
    <th>GT</th>
    <th>GT<sub>strict</sub></th>
    <th>MT</th>
    <th>MTI GT<sub>strict</sub></th>
  </tr>
  <tr>
    <td>deepseek-v12-tiny</td>
    <td>0.3701</td>
    <td>0.0084</td>
    <td>0.1953</td>
    <td>0.25</td>
    <td>0.4415</td>
    <td>0.0154</td>
    <td>0.1823</td>
    <td>0.1786</td>
  </tr>
  <tr>
    <td>deepseek-v12-small</td>
    <td>0.3021</td>
    <td>0.0037</td>
    <td>0.1574</td>
    <td>0.1429</td>
    <td>0.3735</td>
    <td>0.0061</td>
    <td>0.1707</td>
    <td>0.1818</td>
  </tr>
  <tr>
    <td>InternVL2_5-1B</td>
    <td>0.3462</td>
    <td>0.0089</td>
    <td>0.2253</td>
    <td>0.1176</td>
    <td>0.3589</td>
    <td>0.0105</td>
    <td>0.2037</td>
    <td>0.2105</td>
  </tr>
  <tr>
    <td>InternVL2_5-1B-MPO</td>
    <td>0.3636</td>
    <td>0.0084</td>
    <td>0.2168</td>
    <td>0.4375</td>
    <td>0.3833</td>
    <td>0.0099</td>
    <td>0.2032</td>
    <td>0.3889</td>
  </tr>
  <tr>
    <td>InternVL2_5-8B</td>
    <td>0.5294</td>
    <td>0.0711</td>
    <td>0.4589</td>
    <td>0.5778</td>
    <td>0.5574</td>
    <td>0.0765</td>
    <td>0.5424</td>
    <td>0.6259</td>
  </tr>
  <tr>
    <td>InternVL2_5-8B-MPO</td>
    <td>0.5747</td>
    <td>0.0963</td>
    <td>0.5663</td>
    <td>0.7377</td>
    <td>0.6183</td>
    <td>0.1415</td>
    <td>0.6289</td>
    <td>0.7704</td>
  </tr>
  <tr>
    <td>InternVL2_5-26B</td>
    <td>0.5765</td>
    <td>0.1016</td>
    <td>0.5674</td>
    <td>0.7358</td>
    <td>0.6137</td>
    <td>0.1311</td>
    <td>0.63</td>
    <td>0.7689</td>
  </tr>
  <tr>
    <td>InternVL2_5-26B-MPO</td>
    <td>0.6089</td>
    <td>0.1258</td>
    <td>0.6989</td>
    <td>0.9079</td>
    <td>0.6491</td>
    <td>0.1696</td>
    <td>0.7693</td>
    <td>0.8734</td>
  </tr>
  <tr>
    <td>InternVL2_5-78B</td>
    <td>0.6121</td>
    <td>0.1363</td>
    <td>0.6695</td>
    <td>0.8494</td>
    <td>0.6536</td>
    <td>0.1845</td>
    <td>0.7109</td>
    <td>0.8388</td>
  </tr>
  <tr>
    <td>InternVL2_5-78B-MPO</td>
    <td>0.6604</td>
    <td>0.1842</td>
    <td>0.7984</td>
    <td>0.9657</td>
    <td>0.7064</td>
    <td>0.2594</td>
    <td>0.8634</td>
    <td>0.9639</td>
  </tr>
  <tr>
    <td>Janus-Pro-1B</td>
    <td>0.2344</td>
    <td>0</td>
    <td>0.1858</td>
    <td>-</td>
    <td>0.2497</td>
    <td>0.0006</td>
    <td>0.1982</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Janus-Pro-7B</td>
    <td>0.2409</td>
    <td>0</td>
    <td>0.1705</td>
    <td>-</td>
    <td>0.2623</td>
    <td>0</td>
    <td>0.1525</td>
    <td>-</td>
  </tr>
  <tr>
    <td>llava-interleave-qwen-0.5b-hf</td>
    <td>0.2161</td>
    <td>0</td>
    <td>0.2068</td>
    <td>-</td>
    <td>0.2269</td>
    <td>0</td>
    <td>0.2065</td>
    <td>-</td>
  </tr>
  <tr>
    <td>llava-interleave-qwen-7b-hf</td>
    <td>0.3363</td>
    <td>0.0032</td>
    <td>0.2505</td>
    <td>0.1667</td>
    <td>0.3477</td>
    <td>0.0039</td>
    <td>0.2704</td>
    <td>0</td>
  </tr>
  <tr>
    <td>llava-interleave-qwen-7b-dpo-hf</td>
    <td>0.3332</td>
    <td>0.0047</td>
    <td>0.2679</td>
    <td>0.3333</td>
    <td>0.3409</td>
    <td>0.0099</td>
    <td>0.2996</td>
    <td>0.3333</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-0.5b-ov</td>
    <td>0.2931</td>
    <td>0.0005</td>
    <td>0.1884</td>
    <td>0</td>
    <td>0.3118</td>
    <td>0.0006</td>
    <td>0.1839</td>
    <td>0</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-0.5b-si</td>
    <td>0.2106</td>
    <td>0</td>
    <td>0.1895</td>
    <td>-</td>
    <td>0.2199</td>
    <td>0</td>
    <td>0.1944</td>
    <td>-</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-7b-ov</td>
    <td>0.4839</td>
    <td>0.0447</td>
    <td>0.4421</td>
    <td>0.4118</td>
    <td>0.5118</td>
    <td>0.0683</td>
    <td>0.4692</td>
    <td>0.4758</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-7b-si</td>
    <td>0.426</td>
    <td>0.0168</td>
    <td>0.4284</td>
    <td>0.6875</td>
    <td>0.4119</td>
    <td>0.0116</td>
    <td>0.4537</td>
    <td>0.7619</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-7b-ov-sft</td>
    <td>0.5838</td>
    <td>0.0984</td>
    <td>0.6516</td>
    <td>0.8182</td>
    <td>0.6179</td>
    <td>0.1399</td>
    <td>0.7506</td>
    <td>0.8661</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-7b-si-sft</td>
    <td>0.5115</td>
    <td>0.0411</td>
    <td>0.61</td>
    <td>0.8077</td>
    <td>0.5464</td>
    <td>0.0743</td>
    <td>0.6691</td>
    <td>0.7852</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT-Video-7B-hf</td>
    <td>0.2645</td>
    <td>0</td>
    <td>0.1895</td>
    <td>-</td>
    <td>0.2742</td>
    <td>0</td>
    <td>0.1823</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT-Video-7B-DPO-hf</td>
    <td>0.2666</td>
    <td>0</td>
    <td>0.1963</td>
    <td>-</td>
    <td>0.2736</td>
    <td>0</td>
    <td>0.1938</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT-Video-34B-hf</td>
    <td>0.2646</td>
    <td>0.0005</td>
    <td>0.3184</td>
    <td>1</td>
    <td>0.2831</td>
    <td>0</td>
    <td>0.3343</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT-Video-34B-DPO-hf</td>
    <td>0.2882</td>
    <td>0.0016</td>
    <td>0.3095</td>
    <td>0</td>
    <td>0.2984</td>
    <td>0.0006</td>
    <td>0.3133</td>
    <td>0</td>
  </tr>
  <tr>
    <td>LongVA-7B</td>
    <td>0.286</td>
    <td>0.0016</td>
    <td>0.3421</td>
    <td>0.6667</td>
    <td>0.2909</td>
    <td>0.0022</td>
    <td>0.353</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td>LongVA-7B-DPO</td>
    <td>0.3089</td>
    <td>0.0026</td>
    <td>0.3547</td>
    <td>0.8</td>
    <td>0.2997</td>
    <td>0.0028</td>
    <td>0.3618</td>
    <td>0.6</td>
  </tr>
  <tr>
    <td>Mantis-8B-Idelics2</td>
    <td>0.3326</td>
    <td>0.0011</td>
    <td>0.2216</td>
    <td>0</td>
    <td>0.2868</td>
    <td>0.0022</td>
    <td>0.2081</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Mantis-8B-siglip-llama3</td>
    <td>0.2406</td>
    <td>0.0005</td>
    <td>0.2142</td>
    <td>0</td>
    <td>0.2461</td>
    <td>0</td>
    <td>0.2153</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Phi-3-vision-128k-instruct</td>
    <td>0.2243</td>
    <td>0</td>
    <td>0.2316</td>
    <td>-</td>
    <td>0.2214</td>
    <td>0</td>
    <td>0.2329</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Phi-3.5-vision-instruct</td>
    <td>0.2262</td>
    <td>0</td>
    <td>0.2311</td>
    <td>-</td>
    <td>0.2331</td>
    <td>0</td>
    <td>0.2539</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen2-VL-2B</td>
    <td>0.2873</td>
    <td>0.0004</td>
    <td>0.1957</td>
    <td>0</td>
    <td>0.3274</td>
    <td>0.0011</td>
    <td>0.1995</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Qwen2-VL-2B-Instruct</td>
    <td>0.3638</td>
    <td>0.0058</td>
    <td>0.1658</td>
    <td>0.0909</td>
    <td>0.333</td>
    <td>0.0015</td>
    <td>0.2014</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>0.4592</td>
    <td>0.0028</td>
    <td>0.3097</td>
    <td>0.25</td>
    <td>0.4879</td>
    <td>0.0076</td>
    <td>0.3276</td>
    <td>0.5455</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B-Instruct</td>
    <td>0.4813</td>
    <td>0.0342</td>
    <td>0.4253</td>
    <td>0.6462</td>
    <td>0.5098</td>
    <td>0.0441</td>
    <td>0.446</td>
    <td>0.6125</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B</td>
    <td>0.4539</td>
    <td>0.0258</td>
    <td>0.4363</td>
    <td>0.7143</td>
    <td>0.4914</td>
    <td>0.0435</td>
    <td>0.5319</td>
    <td>0.7215</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B-Instruct</td>
    <td>0.4756</td>
    <td>0.0368</td>
    <td>0.4632</td>
    <td>0.6429</td>
    <td>0.5085</td>
    <td>0.0496</td>
    <td>0.5507</td>
    <td>0.7</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.6708</td>
    <td>0.1863</td>
    <td>0.534</td>
    <td>0.5389</td>
    <td>0.7231</td>
    <td>0.2357</td>
    <td>0.615</td>
    <td>0.5532</td>
  </tr>
</table>

Table 15: Full results for MT2: sentence ordering with events/captions.
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">Ordering Images (events)</th>
    <th colspan="2">Ordering Images (captions)</th>
  </tr>
  <tr>
    <th>MT</th>
    <th>MTI GT<sub>strict</sub></th>
    <th>MT</th>
    <th>MTI GT<sub>strict</sub></th>
  </tr>
  <tr>
    <td>deepseek-vl2-tiny</td>
    <td>0.2042</td>
    <td>0.4286</td>
    <td>0.2072</td>
    <td>0.125</td>
  </tr>
  <tr>
    <td>deepseek-vl2-small</td>
    <td>0.1663</td>
    <td>0</td>
    <td>0.1553</td>
    <td>0</td>
  </tr>
  <tr>
    <td>InternVL2_5-1B</td>
    <td>0.2107</td>
    <td>0.1429</td>
    <td>0.1997</td>
    <td>0.0625</td>
  </tr>
  <tr>
    <td>InternVL2_5-1B-MPO</td>
    <td>0.2052</td>
    <td>0.1538</td>
    <td>0.2002</td>
    <td>0.2667</td>
  </tr>
  <tr>
    <td>InternVL2_5-8B</td>
    <td>0.2766</td>
    <td>0.2783</td>
    <td>0.2651</td>
    <td>0.4113</td>
  </tr>
  <tr>
    <td>InternVL2_5-8B-MPO</td>
    <td>0.2986</td>
    <td>0.3841</td>
    <td>0.318</td>
    <td>0.3816</td>
  </tr>
  <tr>
    <td>InternVL2_5-26B</td>
    <td>0.2666</td>
    <td>0.2727</td>
    <td>0.3165</td>
    <td>0.3589</td>
  </tr>
  <tr>
    <td>InternVL2_5-26B-MPO</td>
    <td>0.344</td>
    <td>0.3971</td>
    <td>0.3949</td>
    <td>0.4369</td>
  </tr>
  <tr>
    <td>InternVL2_5-78B</td>
    <td>0.311</td>
    <td>0.4</td>
    <td>0.3849</td>
    <td>0.4709</td>
  </tr>
  <tr>
    <td>InternVL2_5-78B-MPO</td>
    <td>0.4099</td>
    <td>0.4883</td>
    <td>0.5382</td>
    <td>0.6966</td>
  </tr>
  <tr>
    <td>Janus-Pro-1B</td>
    <td>0.2247</td>
    <td>-</td>
    <td>0.2227</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Janus-Pro-7B</td>
    <td>0.2092</td>
    <td>-</td>
    <td>0.2097</td>
    <td>-</td>
  </tr>
  <tr>
    <td>llava-interleave-qwen-0.5b-hf</td>
    <td>0.2017</td>
    <td>-</td>
    <td>0.2077</td>
    <td>-</td>
  </tr>
  <tr>
    <td>llava-interleave-qwen-7b-hf</td>
    <td>0.2087</td>
    <td>0.1667</td>
    <td>0.1997</td>
    <td>0.2222</td>
  </tr>
  <tr>
    <td>llava-interleave-qwen-7b-dpo-hf</td>
    <td>0.2167</td>
    <td>0.3846</td>
    <td>0.2127</td>
    <td>0.1905</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-0.5b-ov</td>
    <td>0.1942</td>
    <td>1</td>
    <td>0.2082</td>
    <td>0</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-0.5b-si</td>
    <td>0.2032</td>
    <td>-</td>
    <td>0.1902</td>
    <td>-</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-7b-ov</td>
    <td>0.2132</td>
    <td>0.1429</td>
    <td>0.2157</td>
    <td>0.1983</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-7b-si</td>
    <td>0.2077</td>
    <td>0.35</td>
    <td>0.2047</td>
    <td>0.4375</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-72b-ov-sft</td>
    <td>0.2756</td>
    <td>0.3182</td>
    <td>0.2906</td>
    <td>0.3647</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-72b-si</td>
    <td>0.2546</td>
    <td>0.3553</td>
    <td>0.2461</td>
    <td>0.2677</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT-Video-7B-hf</td>
    <td>0.2102</td>
    <td>-</td>
    <td>0.2132</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT-Video-7B-DPO-hf</td>
    <td>0.2067</td>
    <td>-</td>
    <td>0.2082</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT-Video-34B-hf</td>
    <td>0.1977</td>
    <td>0</td>
    <td>0.1997</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT-Video-34B-DPO-hf</td>
    <td>0.1887</td>
    <td>0.3333</td>
    <td>0.1932</td>
    <td>0</td>
  </tr>
  <tr>
    <td>LongVA-7B</td>
    <td>0.1952</td>
    <td>-</td>
    <td>0.1902</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LongVA-7B-DPO</td>
    <td>0.1962</td>
    <td>1</td>
    <td>0.1937</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Mantis-8B-Idefics2</td>
    <td>0.1862</td>
    <td>0</td>
    <td>0.1922</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td>Mantis-8B-siglip-llama3</td>
    <td>0.2002</td>
    <td>0</td>
    <td>0.1987</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Phi-3-vision-128k-instruct</td>
    <td>0.1857</td>
    <td>-</td>
    <td>0.1897</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Phi-3.5-vision-instruct</td>
    <td>0.1922</td>
    <td>-</td>
    <td>0.1832</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen2-VL-2B</td>
    <td>0.1917</td>
    <td>-</td>
    <td>0.1859</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Qwen2-VL-2B-Instruct</td>
    <td>0.1438</td>
    <td>0.125</td>
    <td>0.1483</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>0.1917</td>
    <td>0</td>
    <td>0.2067</td>
    <td>0.1818</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B-Instruct</td>
    <td>0.2312</td>
    <td>0.2041</td>
    <td>0.2456</td>
    <td>0.3585</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B</td>
    <td>0.2416</td>
    <td>0.3548</td>
    <td>0.2406</td>
    <td>0.4561</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B-Instruct</td>
    <td>0.2651</td>
    <td>0.4727</td>
    <td>0.2811</td>
    <td>0.4737</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.2257</td>
    <td>0.2353</td>
    <td>0.2297</td>
    <td>0.2349</td>
  </tr>
</table>

Table 16: Full results for MT3: image ordering with events/captions.
D.3 Supplementary Results

Performance of text-only LLMs  The results of the language-only LLMs on our benchmark before and after this filtering step are shown in Table 17. For each LLM, the first row shows results before questions removal, and the second row shows results after. We observe that, after filtering out questions easily answered by language-only LLMs, their performance drops sharply, approaching or falling below random guessing level.

<table>
  <tr>
    <th></th>
    <th>Two-event Relation</th>
    <th>Three-event Relation</th>
    <th>Sentence Ordering - Event</th>
    <th>Sentence Ordering - Caption</th>
  </tr>
  <tr>
    <td>Phi-3.5-mini-instruct</td>
    <td>0.5031<br>0.1798</td>
    <td>0.4999<br>0.2352</td>
    <td>0.3257<br>0.1048</td>
    <td>0.3368<br>0.1242</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>0.5023<br>0.2767</td>
    <td>0.5059<br>0.2094</td>
    <td>0.2091<br>0.0952</td>
    <td>0.1995<br>0.1095</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>0.5361<br>0.2952</td>
    <td>0.5349<br>0.3010</td>
    <td>0.5260<br>0.2722</td>
    <td>0.6108<br>0.3203</td>
  </tr>
</table>

Table 17: Performance of language-only LLMs before and after filtering out text-answerable questions. For each LLM, the first row contains the results before removing the questions that can be correctly answered by at least two language-only LLMs, and the second row contains the results after removing these questions.

Results comparison before and after filtering text-answerable questions  We provide the results of 5 representative MLLMs’ performance before and after this filtering step in Table 18. We observe a performance drop across all MLLMs, indicating that our benchmark becomes more challenging after removing questions that can be answered using text alone. This further reinforces the requirement for MLLMs to truly understand the visual content.

<table>
  <tr>
    <th></th>
    <th>Two-event Relation</th>
    <th>Three-event Relation</th>
    <th>Sentence Ordering - Event</th>
    <th>Sentence Ordering - Caption</th>
  </tr>
  <tr>
    <td>InternVL2_5-26B-MPO</td>
    <td>0.6643<br>0.6032</td>
    <td>0.6743<br>0.6212</td>
    <td>0.7410<br>0.6989</td>
    <td>0.7940<br>0.7693</td>
  </tr>
  <tr>
    <td>InternVL2_5-78B-MPO</td>
    <td>0.6755<br>0.5847</td>
    <td>0.6948<br>0.6139</td>
    <td>0.8334<br>0.7984</td>
    <td>0.9037<br>0.8634</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-72b-ov-sft</td>
    <td>0.6712<br>0.5928</td>
    <td>0.6841<br>0.6154</td>
    <td>0.7211<br>0.6516</td>
    <td>0.8093<br>0.7506</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT-Video-34B-hf</td>
    <td>0.7158<br>0.5847</td>
    <td>0.6828<br>0.5947</td>
    <td>0.4014<br>0.3184</td>
    <td>0.3993<br>0.3343</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B-Instruct</td>
    <td>0.6505<br>0.5395</td>
    <td>0.6533<br>0.5563</td>
    <td>0.5698<br>0.4632</td>
    <td>0.6606<br>0.5507</td>
  </tr>
</table>

Table 18: Performance of five MLLMs before and after filtering out text-answerable questions. For each MLLM, the first row contains the results before removing the questions that can be correctly answered by at least two language-only LLMs, and the second row contains the results after removing these questions.

Performance of image ordering task with images only  We perform image ordering task without text descriptions. As shown in the Table 19, even for the best-performing MLLMs, ordering based on images alone remains highly challenging as their performance is near random guessing. This further confirms that our benchmark requires models to jointly understand temporal information from both text and images. Models cannot rely solely on prior knowledge to infer the correct order of images. This is precisely why we chose visual stories as the data source of our benchmark: accurate temporal reasoning cannot be achieved by relying on a single modality alone.

<table>
  <tr>
    <th></th>
    <th>with event</th>
    <th>with caption</th>
    <th>image only</th>
  </tr>
  <tr>
    <td>InternVL2_5-26B-MPO</td>
    <td>0.3440</td>
    <td>0.3949</td>
    <td>0.2430</td>
  </tr>
  <tr>
    <td>InternVL2_5-78B-MPO</td>
    <td>0.4099</td>
    <td>0.5382</td>
    <td>0.2709</td>
  </tr>
  <tr>
    <td>llava-onevision-qwen2-72b</td>
    <td>0.2756</td>
    <td>0.2906</td>
    <td>0.2023</td>
  </tr>
</table>

Table 19: Performance of three MLLMs in image ordering task with event text, original captions and only images.
E  Additional Examples in TempVS

We provide more examples in TempVS benchmark.

![Five images showing scenes from The Flintstones](page_186_153_1077_120.png)

Captions: Fred is in his car driving somewhere. Fred is holding groceries at a fence outside. He puts the groceries on the fence and then jumps over the gate. Fred is standing in the doorway with two bags of groceries and yelling someone in the house. Fred is in the kitchen. He is emptying a bag of groceries on the table. He is speaking to someone off-screen. Wilma is standing in the room. She is speaking while having her elbows bent.

Events: Fred drove in his car. Fred put the groceries on the fence. Fred stood in the doorway with two bags of groceries. Fred emptied a bag of groceries on the table. Wilma stood in the room and spoke to someone.

Correct image sequence: Image c -> Image e -> Image d -> Image a -> Image b

![Five images showing scenes at a fair](page_186_646_1077_120.png)

Captions: We had fun riding the black roller coaster at the fair. The lights lit up the night and the rides made us all dizzy. The dragon coaster was mom's favorite. The arcade games had the funniest stuffed monkeys as prizes. We threw a million darts trying to win one!

Events: We took the black roller coaster. The lights lit up the night. Mom favored the dragon coaster. The arcade games had stuffed monkeys as prizes. We threw the darts.

Correct image sequence: Image c -> Image e -> Image d -> Image a -> Image b
Event a: Eddy used a screwdriver on a green frog toy.
Event b: Loopy walked to a house.
Event c: Eddy turned around and waved to Loopy.
Event d: Loopy came in with a basket.
Event e: Loopy walked away from Crong and Pororo.

Caption a: Eddy is using a screwdriver on a green frog toy.
Caption b: Loopy is walking to a house surrounded by snowed fields and trees.
Caption c: Eddy turns around and waves to Loopy. Rody, Eddy and Loopy are together in a room.
Caption d: Loopy comes in with a basket. Loopy and Rody wave hands to each other.
Caption e: Loopy walks away and Crong and Pororo wave hands to loopy.

Correct sentence sequence: Sentence e -> Sentence b -> Sentence d -> Sentence c -> Sentence a

Event a: Robert found many different shops and services inside the building.
Event b: Jon sat in his room planning his next move.
Event c: Amy and Robert came in their car to meet Jon.
Event d: Robert loaded his gun in the lift.
Event e: Robert entered the building.

Caption a: Robert finds many different shops and services inside building. He is confused.
Caption b: Jon is sitting in his room, planning his next move.
Caption c: Amy and Robert are coming in their car to meet Jon.
Caption d: Robert gets inside lift and loads his gun. He intends to kill Jon as soon as find him.
Caption e: Robert goes inside the building and begins to search for Jon.

Correct sentence sequence: Sentence b -> Sentence c -> Sentence e -> Sentence a -> Sentence d
True: The King and his hunting party pursued the deer after the parents told Anna they were in the middle of the King's forest.
False: The parents told Anna they were in the middle of the King's forest after the King and his hunting party pursued the deer.

True: The King and his hunting party pursued the deer. Earlier, Anna sat with her parents beneath a large oak tree.
False: Anna sat with her parents beneath a large oak tree. Earlier, the King and his hunting party pursued the deer.

True: Anna sat with her parents beneath a large oak tree. Anna asked her parents where they were. The King and his hunting party pursued the deer.
False: The King and his hunting party pursued the deer. Anna sat with her parents beneath a large oak tree. Anna asked her parents where they were.

True: Pororo kicked the ball high. Crong received a ball from Pororo.
False: Crong received a ball from Pororo. Pororo kicked the ball high.

True: Pororo kicked the ball high. Then, Loopy failed to kick the ball.
False: Loopy failed to kick the ball. Then, Pororo kicked the ball high.

True: Pororo and Crong shouted to Loopy before Loopy failed to kick the ball, and after that, Eddy kicked the ball.
False: Eddy kicked the ball before Pororo and Crong shouted to Loopy, and after that, Loopy failed to kick the ball.
True: Granpa made some last-minute repairs before the kids had a tickle fight in the family room.
False: The kids had a tickle fight in the family room before Granpa made some last-minute repairs.

True: The mother, father, and son listened to a story at dinner. Earlier, Granpa made some last-minute repairs.
False: Granpa made some last-minute repairs. Earlier, the mother, father, and son listened to a story at dinner.

True: First, Granpa made some last-minute repairs. Second, the kids had a tickle fight in the family room. Third, Nina took out a chocolate cake.
False: First, Nina took out a chocolate cake. Second, Granpa made some last-minute repairs. Third, The kids had a tickle fight in the family room.

True: Wilma and betty talked in the living room. Earlier, Fred sat at the kitchen table with a boy.
False: Fred sat at the kitchen table with a boy. Earlier, Wilma and Betty talked in the living room.

True: Fred and Barney talked to each other while driving in the car after Fred choked the men in the blue dress with a cap in the dining room.
False: Fred choked the men in the blue dress with a cap in the dining room after Fred and Barney talked to each other while driving in the car.

True: Fred choked the men in the blue dress with a cap in the dining room. Fred and Barney talked to each other while driving in the car. Wilma and Betty talked in the living room.
False: Fred and Barney talked to each other while driving in the car. Fred choked the men in the blue dress with a cap in the dining room. Wilma and Betty talked in the living room.

Reviewer paper summaries: The work introduce new benchmark dataset, TempVS, designed to assess the understanding of event sequences by multimodal large language models (MLLMs) in multi-image contexts. Specifically, the benchmark propose two tasks (i) Event A before/after Event B, (ii) Order sequence of events (in both image and text separately). The benchmark inherits from existing visual story dataset, however, heuristic filtering (such as BERT or CLIP similarity) to make the samples harder i.e. within-pair similarity is higher than cross-pair similarity. The performance is assessed using accuracy across 38 models (including GPT4), complemented by human evaluation on 60 samples conducted through Prolific.

The paper introduces a benchmark called TempVS that focuses on the temporal grounding and reasoning capabilities of multimodal LLMs in image sequences.  They collect and annotate 2K sequences from four visual story datasets from cartoon animations and Flickr photos.  They propose 3 tasks for the evaluation: event relation inference, sentence ordering, and image ordering.  Additionally, they include a grounding test to evaluate whether the models can match an event description with the corresponding image.  They evaluate 38 multimodal MLLMs and report their findings.  They highlight that models struggle to solve these tasks, especially the lower parameter models.  They also find that passing the grounding test does not mean that the model can excel in temporal reasoning.
Summary of strengths: - The benchmark is novel to assess the MLLMs ability for order of events under multi-image settings.
- The idea of grounding test is good along with evaluation of temporal understanding.
- The analysis of adverbial markers (i.e. before/then, after/earlier) in section 4.3 is interesting.

The benchmark is relevant for future research and is well thought out.
They perform and explain different filtering processes in an attempt to make the tasks sufficiently hard for models and to avoid that models can guess the answers based on other information or content from only one modality.
The paper is well-written and easy to follow.
Summary of weaknesses: - It is not clear how captions and events are different. Also, the process to obtain the events from captions via GPT4 (Line#280) is not described in the work.
- Conducting an error analysis is crucial for understanding the failure modes of current MLLMs, specifically identifying where and why they struggle to accurately predict the sequence of events. Including multiple qualitative examples with detailed analysis is necessary to make the manuscript comprehensive.

My main concern is about the decision of the type of input to the models, meaning, instead of feeding the models with sequences of images, the authors input a combined image, i.e., one picture rather than a sequence of images.  This is a huge limitation of the results and findings given that the purpose of the benchmark is to assess the "reasoning capabilities of multimodal LLMs in image sequences."  It is not clear whether the models "understand" this combined image as multiple images or as one image with different actions. 

There is an analysis about the distance between two events, but did you analyze the performance with different image-sequence lengths?  In line 341, it is mentioned that there are 61 cases with images that have 6 to 9 images.  How do their performances compare to the 5 image length performances?

In line 502: "When comparing explicit and implicit temporal event statements." Can you clarify the difference between the implicit and explicit event statements?  Are there any examples of this? It is not clear how you designed this and what the differences are.
Comments, suggestions and typos: - There are similar work exist such as [a] in limited domain (such as wikihow) and how the multi-image understanding is different from short video image sequences (TempCoompass [b], E.T. bench [c], MVBench [d]), . I would appreciate to include some discussion to make the manuscript comprehensive.
- Contributions (Line#103-Line#111) are over emphasised i.e. point 2 and 3 are similar.


## References
[a] Reasoning about Goals, Steps, and Temporal Ordering with WikiHow, EMNLP 2020
[b] TempCompass: Do Video LLMs Really Understand Videos?, ACL 2024
[c] E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding, NeurIPS 2024
[d] MVBench: A Comprehensive Multi-modal Video Understanding Benchmark, CVPR 2024

Well-written, easy to follow.
Overall assessment scores: 3

3.5
Metareview score: 3
Metareview: This paper introduces TempVS, a benchmark to evaluate multimodal LLMs on temporal grounding and reasoning tasks across image sequences. It contains 2k annotated image sequences and 15k multiple-choice questions spanning event relation inference, sentence ordering, and image ordering tasks from cartoon animations and Flickr photos. The authors evaluate 38 state-of-the-art multimodal LLMs, revealing significant performance gaps compared to human reasoning and offering insights for future research directions.
Summary of reasons to publish: The reviewers generally found the writing clean and easy to follow, the problem important, and the experimental results very comprehensive (38 multimodal LLMs, including open-source and proprietary models, across various model sizes and architectures). The reviewers also found the benchmark novel, thorough, and relevant for future research.