Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences (2024)

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated proficiency in handling a variety of visual-language tasks. However, current MLLM benchmarks are predominantly designed to evaluate reasoning based on static information about a single image, and the ability of modern MLLMs to extrapolate from image sequences, which is essential for understanding our ever-changing world, has been less investigated. To address this challenge, this paper introduces Mementos, a new benchmark designed to assess MLLMs’ sequential image reasoning abilities. Mementos features 4,761 diverse image sequences with varying lengths. We also employ a GPT-4 assisted method to evaluate MLLM reasoning performance. Through a careful evaluation of nine recent MLLMs on Mementos, including GPT4V and Gemini, we find that they struggle to accurately describe dynamic information about given image sequences, often leading to hallucinations/misrepresentations of objects and their corresponding behaviors. Our quantitative analysis and case studies identify three key factors impacting MLLMs’ sequential image reasoning: the correlation between object and behavioral hallucinations, the influence of cooccurring behaviors, and the compounding impact of behavioral hallucinations.

heading: 1 Introduction
text: The recent emergence of Multimodal Large Language Models (MLLMs) such as GPT-4V (OpenAI, 2023b) and Gemini (Team, 2023) has shown strong visual-language understanding and generation capabilities in many areas, like image captioning and visual question answering. Despite the notable performance of existing MLLMs, they often suffer from hallucination (a phenomenon where MLLMs produce inaccurate descriptions of the given images) due to insufficient reasoning capabilities, generating inaccurate responses in visual
inference (Liu et al., 2023a; Yue et al., 2023). Thus, monitoring the reasoning capability is of great importance in understanding the ability and the limitations of MLLMs and applying MLLMs in the real world. Previous benchmarks, such as Liu et al. (2023a) and Yue et al. (2023), have primarily addressed evaluating reasoning in each individual image, relying on static and object-centric knowledge. However, they are insufficient to comprehensively assess the reasoning capabilities of MLLMs due to a lack of time-varying object behaviors or events.
To investigate the capabilities of Multi-Modal Language Models (MLLMs) in dynamic reasoning across image sequences, we present a new benchmark, Mementos. This benchmark focuses on the complex task of monitoring and deciphering the positional changes of objects within an image sequence, followed by the inference of behavioral patterns and logical connections among them. Such an endeavor requires the interpretation of the overarching context based on time-variant visual elements, posing a greater challenge than the analysis of static scenes. Concretely, Mementos consists of 4,761 image sequences with varying episode lengths, encompassing diverse scenarios from everyday life, robotics tasks, and comic-style storyboards. An episode refers to a specific event or series of events depicted in the image sequence. Each sequence is paired with a human-annotated description of the key objects and their behaviors.
To assess the reasoning capability of MLLMs on Mementos, we employ a GPT-4-assisted evaluation procedure: after an MLLM produces a description for an image sequence, we extract behavior and object keywords from both AI-generated and human-annotated descriptions using GPT-4. We then use keyword matching to assess the degree of behavioral and object hallucinations. To refine the correctness of this evaluation, we have developed behavior and object synonym graphs for each domain. These graphs facilitate more precise key-
416
word matching, ensuring a thorough and nuanced analysis of the MLLMs’ reasoning abilities. Besides, we also provide the comparison with human evaluation to demonstrates that the GPT-4-assisted evaluation procedure is very reliable.
We evaluated the reasoning proficiency of nine leading-edge MLLMs on Mementos, encompassing both black-box and open-source models. Our findings indicate that Mementos poses a considerable challenge to these current MLLMs. For instance, as depicted in Figure 1, GPT-4V exhibits notable behavioral and object hallucinations in various domains during image sequence reasoning. Behavioral hallucinations are defined as the MLLMs’ erroneous interpretations or predictions of entity actions, while object hallucinations pertain to the inaccurate identification or creation of objects. Notably, behavioral hallucinations were more frequent
than object hallucinations, highlighting a significant deficiency in MLLMs’ capability to deduce events from image sequences.
Furthermore, our research pinpoints three principal factors that lead to the reasoning failures of MLLMs: (1) the interconnectedness of object and behavioral hallucinations, (2) the impact of cooccurring behaviors, and (3) the cumulative effect of behavioral hallucinations. The objective of our proposed benchmark and analyses is to shed light on innovative approaches to augment the reasoning abilities of MLLMs and to reduce hallucinations in their subsequent advancements.

heading: 2 Mementos
text: In this section, we introduce Mementos, a novel and challenging benchmark designed to test the reasoning capability of Multimodal Large Language
Model (MLLM) under sequential image input. Initially, we detail the data gathering and annotation methodology for Mementos, alongside an overview of data distribution. Subsequently, we outline the procedure and the metric employed to evaluate the reasoning capabilities of MLLMs on Mementos.

heading: 2.1 Mementos Benchmark

heading: 2.1.1 Dataset Composition
text: Mementos comprises 4,761 image sequences of varying lengths, predominantly sourced from Dailylife, Robotics, and Comics domains. Detailed statistics are provided in Table 1. This diverse collection is instrumental in evaluating the comprehensive time-varying reasoning abilities of MLLMs. Specifically, the robotics data, closely associated with embodied AI or real-world contexts, and the comic-style storyboard data, rich in stylistic and episodic diversity in image sequences, significantly enhance the benchmark’s relevance and robustness.
Daily-life The Daily-life image sequences in Mementos are derived from video clips in the NextQA dataset, as cited in Xiao et al. (2021). These sequences represent a range of everyday life scenarios. We have selectively extracted videos from the Next-QA Training set, specifically those with frame counts ranging from 400 to 2,500. To balance the challenge of testing MLLMs’ reasoning capabilities against the risk of losing critical information, our methodology involves retaining the first frame of each video. Subsequently, we sample one image every 100 frames. The collected images from this sampling process then form an image sequence that corresponds to the original video. This approach ensures a rigorous yet feasible evaluation of MLLMs’ reasoning abilities in dynamically evolving everyday scenarios.
Robotics For the Robotics data, we utilized videos from various sub-datasets within Open XEmbodiment (Collaboration et al., 2023). Open XEmbodiment aggregates video datasets from multiple university laboratories, showcasing a variety of tasks performed by different robotic systems. We
meticulously selected sub-datasets from Open XEmbodiment that offer video resolutions exceeding 128x128 and exhibit a high degree of task diversity. From these chosen sub-datasets, a total of 1,101 videos were sampled. The precise number of videos sourced from each sub-dataset is detailed in Appendix A. For video sampling, our approach varied based on the length of the videos. Videos exceeding 100 frames were processed by sampling one image every n/20 frames, where n represents the total frame count. Conversely, for videos with frame counts ranging from 20 to 100, we sampled one image every 5 frames. This ensures the formation of comprehensive and representative image sequences for each video, catering to the evaluation of MLLMs in diverse and complex robotic contexts.
Comics The Comics data is composed of wordless multi-panel comics of diverse styles, curated from online sources. Unlike Daily-life and Robotics sections, where image sequences are uniformly extracted from video frames, the comics represent intentionally selected key moments within a narrative, manually illustrated by artists. This distinction sets our dataset apart from conventional video datasets. In addition to traditional comics, this category also incorporates 20 storyboards from movies reimagined in comic style. We have further deconstructed these comics into individual image sequences by taking screenshots. This approach enables a unique exploration of sequential visual reasoning, enhancing the diversity and complexity of the dataset for evaluating MLLMs.

heading: 2.1.2 Dataset Annotation
text: For each image sequence in Mementos, we have meticulously annotated a ground truth description that captures the unfolding events. These descriptions focus on the primary objects and their respective behaviors, where behavior refers to a verb or verb phrase associated with the object in question.
For the Daily-life data, we initially employed GPT-4V(ision) (OpenAI, 2023a), to amalgamate and reformulate the questions and answers from the Next-QA videos into single paragraph descriptions. This method significantly expedited the manual annotation process. Following this, we conducted a thorough manual review of these automated descriptions, making necessary adjustments. This included rectifying inaccuracies, removing nonexistent episodes, and adding missing details to
ensure alignment with the actual image sequences. To ensure reliability, we implemented a crossvalidation step, where a separate set of annotators performed a secondary review. For the Robotics and Comics categories, the annotation process was entirely manual, conducted by human annotators. These annotations were then subjected to a verification process by the authors which ensures the accuracy and consistency of the descriptions across all categories.

heading: 2.1.3 Dataset Statistics
text: In showcasing the diversity of Mementos, we present a detailed overview of the data distribution within the Mementos validation set. Our analysis focuses on two key dimensions: the length of the image sequence and the length of the episode. The length of an image sequence is defined by the number of frames it contains, while the episode length is determined by the total number of events depicted in the sequence. A longer image sequence necessitates the MLLM to process a larger number of images, thereby challenging the model’s capacity to manage sequences spanning broader timeframes. A greater episode length signifies that the image sequence encompasses more intricate scenarios.
Image sequence length For the image sequence length, we count the number of frames in each image sequence. As shown in Figure 2(a), the majority of image sequences are between 4 and 14 frames in length. 67.38% of image sequences
contain 4 to 14 frames, yet 31.90% of sequences are composed of longer frames - more than 15 frames.
Episode length To quantify the episode length within each image sequence of Mementos, we employed GPT-4 for extracting behavior keywords, specifically verbs associated with objects, from the human-annotated descriptions. This extraction was facilitated using a pre-defined manual prompt, details of which can be found in Appendix D. Following the extraction, we calculated the length of the behavior list for each image sequence. A lengthier behavior list signifies a more extended episode within the image sequence, which inherently poses a greater challenge for the MLLM in comprehending the entire image sequence. As illustrated in Figure 2(b), a significant portion of the image sequences, particularly those from the robotics data, feature episode lengths ranging between 1 and 3. This is mainly attributed to the dominance of twoaction episodes like ‘pick up and place’, ‘move and pull open’, ‘locate and push’. Meanwhile, the remaining data exhibits a normal distribution for episode lengths spanning 4 to 17.

heading: 2.2 Evaluation Procedure and Metrics
text: In this section, we illustrate how to evaluate the descriptions generated by MLLMs, including the evaluation procedure and metrics. Procedure As shown in Figure 3, we use an image sequence and a pre-designed prompt together as the input for MLLMs, and generate the description aligned with the corresponding image sequence. Next, we ask GPT-4 to extract object and behavior keywords in the AI-generated description. We then match the obtained keywords with the synonym graph we built, replacing the matched keywords with the root word from the synonym graph. Finally, we obtain two lists of keywords: AI-generated object list and AI-generated behavior
list. We note that the proposed keyword extraction leveraging GPT-4 is surprisingly reliable and accurate, which is competitive with human extraction. Please refer to Appendix C for more details. Synonym graph The synonym graph is an unilateral digraph where each edge connects two nodes representing words or phrases. For instance, given a synonym pair (pick up, lift up), an edge is directed from ‘lift up’ to ‘pick up’. In each synonym pair, the first word, originating from the human-annotated keyword list, is referred to as the root word, while the second word is from the AI-generated keyword. To construct this synonym graph, we use GPT-4 to extract object and behavior keywords from all human-annotated descriptions in the Val set, forming a human-annotated keyword list. Then, we generate descriptions using GPT4V, LLAVA, and Gemini and use GPT-4 to extract object and behavior keywords. After that, we manually match these words with the human-annotated keyword list to identify all synonym pairs and add them as edges to the synonym graph. Given a word or phrase, this synonym graph can quickly match the corresponding root word if a synonym exists in the human-annotated keyword list, completing the keyword replacement. For convenience in evaluation, we maintain separate synonym graphs for objects and behaviors of different categories. We make all constructed synonym graphs publicly available as open-source resources. Metrics After obtaining the AI-generated object list and behavior list, we utilize the corresponding human-annotated object list and human-annotated behavior list as the ground truth to calculate ‘Recall,’ ‘Precision,’ and ‘F1 metrics’ at both the object and behavior levels. These metrics are used to measure the understanding capabilities regarding the image sequence episode. ‘Recall’ reflects the accuracy of an MLLM’s reasoning about episodes in an image sequence, while ‘precision’ focuses on assessing the severity of hallucinations that occur when understanding the image sequence.

heading: 3 Experiments
text: In our experimental section, we delve into two key questions: (a) We examine the reasoning capabilities of current MLLMs on Mementos. Specifically, we assess the severity of object and behavioral hallucinations in these models. (b) We investigate the underlying causes of reasoning failures in MLLMs when interpreting image sequences.

heading: 3.1 Baseline evaluation

heading: 3.1.1 Models
text: We establish our baseline using 9 popular MLLMs. The black-box MLLMs include GPT-4V (OpenAI, 2023a) and Gemini (Team, 2023), and the opensource MLLMs are Video-LLaMA-2 (Zhang et al., 2023a), Chat-UniVi (Jin et al., 2023), LLaVA1.5 (Liu et al., 2023c), MiniGPT4 (Zhu et al., 2023), MiniGPT5 (Zheng et al., 2023), mPLUG_Owlv2 (Ye et al., 2023), and InstructBLIP (Dai et al., 2023). Considering that only a few open-source MLLMs are designed to process sequential images (Video-LLaMA-2 and Chat-UniVi), we adapt input for other models by combining all frames from an image sequence into one composite image, referred to as the combined-input (c-input) setting. For black-box MLLMs and Chat-UniVi, we conduct evaluations using both the c-input and an alternative approach where frames from the image sequence are input sequentially, termed the sequential-input (s-input) setting. For VideoLLaMA-2, we only test in s-input setting.

heading: 3.1.2 Evaluation results
text: We evaluate all MLLMs on Mementos and report the results in Figure 4. Besides, we provide the performance of each baseline method in three different domains (Daily-life, Robotics, and Comics) in Table 2. We summarize our findings as follows: GPT-4V (s-input) and LLaVA-1.5 are the bestperforming models among black-box and opensource MLLMs, respectively. As shown in Figure 4, except for being on par with Gemini (sinput) and LLaVA-1.5 in behavior precision, GPT4V with s-input demonstrates the best reasoning
capability compared with all other MLLMs in understanding image sequences. Among open-source models, LLaVA1.5 performs the best, nearly matching or even surpassing the black-box model Gemini in object comprehension, but its ability to infer behaviors from image sequences is weaker compared to Gemini and GPT-4V. Although Video-LLaMA-2 and Chat-UniVi are designed for video understanding, they do not show an advantage over LLaVA1.5, especially Video-LLaMA-2, which performs notably worse compared to LLaVA-1.5. The weakest models in understanding image sequences are MiniGPT4 and MiniGPT5, with a significant gap in every metric compared to the other baselines. It’s noteworthy that under c-input setting, the performance of black-box MLLMs does not significantly differ from that of open-source MLLMs. LLaVA1.5 and mPLUG_Owl-v2 meet or even exceed the black-box MLLMs on many metrics.
MLLMs possess a much stronger ability on reasoning objects in image sequences than they
do on reasoning behaviors. We find that all MLLM methods perform significantly better on the three metrics for objects than those for behaviors. Taking the best-performing GPT-4V as an example, it achieves over 50% on all three object metrics, with recall even reaching 60%, indicating it can effectively recognize the main objects in an image sequence. However, for behaviors, GPT-4V scores only around 30%, with the best recall metric barely exceeding 40%. Despite this, GPT-4V is still the best-performing MLLM in reasoning behaviors. This suggests that current MLLMs do not possess strong abilities to autonomously infer the behaviors from given sequential images, indicating the importance of our benchmark in highlighting the limitations in the reasoning abilities of MLLMs.
Reasoning capability of MLLMs varies across different domains. From Table 2, we find that black-box models perform best in the robotics domain across the three domains, while open-source models show relatively better performance in the
daily-life domain. Analyzing each domain specifically, it is evident that in the daily-life domain, the performance of all methods, except for GPT-4V (s-input), does not vary significantly. The main reason for the performance gap between open-source MLLMs and black-box MLLMs is the noticeably lower metrics of open-source models compared to black-box models in the robotics and comics domains. The recall, precision, and F1 of both object and behavior for black-box MLLMs are almost more than double those of open-source models. We speculate that one reason for this phenomenon is the distribution shift between Mementos and the training data of open-source MLLMs. The limitations of the training data lead to weaker reasoning capability of open-source MLLMs.

heading: 3.2 Analysis of Failure Reasoning
text: In this section, we will provide reasons for failure reasoning results in current MLLMs, combining specific quantitative analyses and case studies. Since behavioral hallucination is a unique phenomenon in image sequence reasoning, and the causes of object hallucination are not significantly different from those in single image reasoning, we only present the reasons leading to behavioral hallucination in this paper. Due to space limitations, please refer to the Appendix E for specific case studies. The following are our main findings: Interplay between object and behavioral hallucinations in MLLMs. A key hypothesis underpinning behavioral hallucination is that incorrect object identification leads to subsequent inaccuracies in behavior identification. To test this, we evaluated the correlation coefficients between object and behavioral hallucinations across different domains for various MLLMs, as detailed in Appendix B Table 4. Our findings reveal that, for most MLLMs, the correlation coefficients in the three domains fluctuate between 0.1 and 0.4, suggesting a weak yet present correlation. This outcome supports the hypothesis that object hallucination contributes to behavioral hallucination to some extent. Case studies further reveal that after an object hallucination occurs, MLLMs tend to describe behaviors related to the hallucinated object, even if these behaviors do not exist in the image sequence. As shown in Figure 5, after recognizing a scene as a tennis court, a MLLM might describe a person playing tennis. Interestingly, in the robotics domain, there is a negligible correlation between object and behavioral hallucinations in black-box MLLMs. This diver-
gence is likely because behaviors in robotics are predominantly linked to robotic arms, which these MLLMs generally identify correctly.
The impact of co-occurrence on behavioral hallucination. In line with object hallucination phenomena, as noted in Li et al. (2023c) and Zhou et al. (2023a), MLLMs demonstrate a tendency to generate behaviors that are commonly paired together. This proclivity exacerbates the problem of behavioral hallucination, especially in the field of robotics. Consider the case in Figure 1 where a robotic arm is tasked with opening a drawer by grabbing its side. MLLMs might erroneously depict the sequence as the arm grabbing the handle first, followed by pulling the drawer open, since grabbing the handle is a more co-occurring behavior with ‘pull open’. Despite the final outcome being accurately described, such errors in key details are unacceptable in robotics. This issue is of particular concern given the growing inclination to utilize MLLMs as reward functions in robotic training (Ma et al., 2023; Sontakke et al., 2023; Rocamonde et al., 2023; Baumli et al., 2023). Such behavioral hallucinations can critically affect the quality of the reward function, leading to potential mislearning of behaviors in robotic systems. Detailed case studies are shown in Appendix E.
The Snowball effect in behavioral hallucinations. The Snowball effect is a well-documented phenomenon in machine learning, referring to the progressive accumulation or intensification of errors in a system, as discussed in Asadi et al. (2019); Zhang et al. (2023b); Wang et al. (2023c); Liu et al. (2023d). Zhang et al. (2023b) notably highlight this phenomenon in Large Language Models. Experiments on Mementos reveal that the snowball effect in both behavioral and object hallucinations becomes markedly pronounced when reasoning through image sequences. The temporal nature of image sequences, consisting of a series of frames rather than a solitary image, demands that MLLMs sequentially infer the narrative. This process makes models susceptible to exacerbating hallucinations if errors occur early in the sequence. We specifically examined the trend of object and behavioral hallucination in GPT-4V and LLaVA-1.5 within the daily-life domain, correlating it with the episode length. As shown in Figure 6, there is a noticeable decrease in object and behavior recall for both MLLMs as the episode length extends. This trend suggests a heightened susceptibility to hallucinations and a pronounced snowball effect in MLLMs when processing image sequences with a greater array of objects and behaviors. Detailed case studies can be found in Appendix E.

heading: 4 Related work

heading: 4.1 Benchmarking in MLLMs
text: The advent of MLLMs has prompted a reassessment of traditional benchmarks (Lin et al., 2014; Marino et al., 2019; Hudson and Manning, 2019). These benchmarks fail to sufficiently expose the hallucination issues in MLLMs. Consequently, there is a growing impetus to devise more challenging benchmarks. This trend spans various domains, from question and answering (QA) reasoning (Liu et al., 2023a; Yue et al., 2023), to optical character recognition (OCR) (Liu et al., 2023f), and extends to the study of hallucinations (Wang et al., 2023a), with benchmarks such as POPE (Li et al.,
2023c) and Bingo (Cui et al., 2023). Additionally, comprehensive analyses of MLLMs, such as Mmbench (Liu et al., 2023e), Mm-vet(Yu et al., 2023b), LVLM-eHub(Xu et al., 2023), SEED(Li et al., 2023a), GAVIE(Liu et al., 2023b), and LAMM (Yin et al., 2023), are emerging.
Our paper presents a novel benchmark using sequences from videos or comics to study behavioral hallucinations, diverging from single-image analysis. Unlike Chen et al. (2023a)’s vision QA tasks from uniformly sampled video frames, our benchmark challenges MLLMs to describe sequences without question guidance, offering a finer evaluation of hallucinations and reasoning in MLLMs.

heading: 4.2 Hallucination in MLLMs
text: Hallucinations in MLLMs, akin to those in Large Language Models (LLMs) (Zhang et al., 2023c; Li et al., 2023b; Zhou et al., 2024; Chen et al., 2023b), represent a significant challenge. In MLLMs, hallucinations are characterized by inconsistencies between the model’s output and the visual content (Rohrbach et al., 2018; Wang et al., 2023a). Recent studies have explored various aspects of hallucination in MLLMs, covering topics such as object hallucination (Li et al., 2023c), hallucination assessment in GPT-4V (Cui et al., 2023), and knowledge hallucination (Liu et al., 2023a).
While there are methods proposed for mitigating hallucinations (Zhou et al., 2023a; Wang et al., 2023b; Leng et al., 2023; Zhou et al., 2023b; Chen et al., 2023c; Jiang et al., 2023; Huang et al., 2023; Yu et al., 2023a; Zhao et al., 2023), there is a noticeable gap in the literature regarding the study of behavioral hallucination. Moreover, the existing work does not offer a dedicated metric for evaluating behavioral hallucinations.

heading: 5 Conclusion
text: In this paper, we present Mementos, a novel and challenging benchmark designed to assess the reasoning abilities of Multimodal Large Language Models (MLLMs) in interpreting image sequences. We conduct evaluations on nine most recent MLLMs using GPT-4-assisted evaluation procedure. Our findings indicate that all tested MLLMs struggle with significant behavioral and object hallucinations in generating descriptions for image sequences. Through a mix of quantitative analysis and case studies, we identify three primary factors contributing to these reasoning failures.
Limitations
Domain courage Mementos is consisted of 4,761 image sequences from three domains: Daily life, Robotics, and Comics. It would be interesting to include a broader variety of data types. This expansion could include first-person navigation experiences, sequential medical CT scans, and interactive gaming data. MLLMs could behave different types of hallucinations in image sequences from other domains Evaluation Process Our evaluation process focuses on the match of keywords to measure the reasoning ability of MLLMs. However, it would be possible that the MLLM generation is the same as human annotations in semantics but obtains low performance, since the generated tokens are not covered by our synonym graph. Future work could extend the evaluation method to semantic understanding rather than relying predominantly on keyword matching. Hallucination Mitigation Our work identifies two kinds of hallucination: object and behavioral hallucinations and explore the failure reason of MLLMs. We have not yet proposed a mitigation method to reduce behavioral hallucinations. Future work could utilize the three causes of reasoning failures to bolster the reasoning faculties of MLLMs, making them more adept at accurately interpreting and describing complex image sequences.

heading: Acknowledgement
text: Wang, Zhou, Liu, Xu, and Huang are supported by National Science Foundation NSF-IIS FAI program, DOD-ONR-Office of Naval Research under award number N00014-22-1-2335, DOD-AFOSRAir Force Office of Scientific Research under award number FA9550-23-1-0048, Capital One and JP Morgan faculty fellowships. Yao is supported by the Cisco Faculty Research Award. He also thanks Center for AI Safety and Google Cloud Research Credits program for supporting our computing needs. Bansal is supported by DARPA ECOLE Program No. HR00112390060 and ONR Grant N00014-23-1-2356.

heading: A Details of Open X-Embodiment Data Selection
text: In this section, we provide the names of all subsets selected from Open X-Embodiment dataset and the corresponding sampling video numbers. For detailed information, please refer to Table 3.

heading: B Correlation Coefficients between Object and Behavioral Hallucinations
text: In this section, we provide detailed correlation coefficients between object and behavioral hallucinations in Table 4.

heading: C Human Evaluation
text: In this section, to verify the reliability of the GPT4 assisted evaluation procedure, we compare the results of GPT-4 assisted evaluation with those of human evaluation. We randomly select 200 image sequences from the entire Val set and manually extract object and behavior keyword lists for each image sequence’s AI-generated description and human-annotated description. Then, we calculate six metrics and compare them with the metrics obtained using keyword lists extracted by GPT4. We choose the four MLLMs that performed best in reasoning on Mementos as representatives: GPT-4V (s-input), Gemini (s-input), Chat-UniVi (s-input), and LLaVA-1.5. The evaluation results are shown in Table 5.
After comparison, we find that there is not a significant gap between the results of GPT-4 assisted evaluation and human evaluation, with the absolute value of the difference mostly ranging between 1% to 4%. For most metrics, the GPT-4 assisted evaluation tends to overestimate the performance of MLLMs, meaning the evaluation results are higher than those of human evaluation. However, the relative ranking among different MLLMs remains essentially unchanged. Overall, the GPT-4 assisted evaluation is quite reliable.

heading: D Prompt Details
text: In this section, we provide all the prompts used in our paper, including those used to merge questions and answers from Daily-life videos into a single description, prompts for MLLMs to generate descriptions corresponding to image sequences, and prompts for extracting object and behavior keywords from both human-annotated and AIgenerated descriptions. The detailed prompts are showm in Table 6.

heading: E Case Study
text: In this section, we present failure reasoning cases of different domains (Figure 7-22), with specific reasons for failure detailed in the captions of each figure.

heading: F Status of Exemption from Institutional Review Board
text: Before starting any segments of the study involving human evaluation, the research team completed and submitted a “Human Subjects Research Determination" form to the appropriate Institutional Review Board (IRB). We obtained a determination letter from the IRB before any human study activities commenced, indicating that our project proposal had been granted ‘Exempt’ status. This classification implies that the proposed research was deemed ‘Not Human Subjects Research’.

heading: Future Work
text: Future work could extend the evaluation method to semantic understanding rather than relying predominantly on keyword matching. Hallucination Mitigation Our work identifies two kinds of hallucination: object and behavioral hallucinations and explore the failure reason of MLLMs. We have not yet proposed a mitigation method to reduce behavioral hallucinations. Future work could utilize the three causes of reasoning failures to bolster the reasoning faculties of MLLMs, making them more adept at accurately interpreting and describing complex image sequences