Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQA (2024)

Abstract: Multipanel images, commonly seen as web screenshots, posters, etc., pervade our daily lives. These images, characterized by their composition of multiple subfigures in distinct layouts, effectively convey information to people. Toward building advanced multimodal AI applications, such as agents that understand complex scenes and navigate through webpages, the skill of multipanel visual reasoning is essential, and a comprehensive evaluation of models in this regard is important. Therefore, we introduce Multipanel Visual Question Answering (MultipanelVQA), a novel benchmark comprising 6,600 triplets of questions, answers, and multipanel images that specifically challenge models in comprehending multipanel images. Our evaluation shows that questions in the MultipanelVQA benchmark pose significant challenges to the stateof-the-art Multimodal Large Language Models (MLLMs) tested, even though humans can attain approximately 99% accuracy on these questions. Distinctively, the MultipanelVQA benchmark features synthetically generated multipanel images specifically crafted to isolate and assess the impact of various factors, such as the layout, on MLLMs’ multipanel image comprehension abilities. As a result, in addition to benchmarking the capabilities of MLLMs in understanding multipanel images, we analyze various factors of the multipanel image that affect MLLMs’ performance with synthetic data and offer insights for enhancement. https://sites.google.com/ view/multipanelvqa/home.

heading: 1 Introduction
text: Multimodal Large Language Models (MLLMs) have become a significant leap in the integration of visual and textual data processing, enabling more nuanced understanding and generation of content that blends both visual and linguistic elements. Being trained on extensive data, advanced
Grid style layoutsSplashed style layouts
MLLMs (OpenAI, 2023b; Liu et al., 2023c; Ye et al., 2023b; Chen et al., 2023; Liu et al., 2023c) have shown remarkable proficiency in various tasks (e.g., image captioning and visual question answering) that require natural language understanding, visual-language grounding, visual reasoning, etc.
As MLLMs become more competent, there is a trend of establishing increasingly challenging benchmarks that are often arduous for average humans to achieve (Yue et al., 2023). However, this raises a pertinent question: Have MLLMs advanced to the stage where elementary benchmarks easily handled by average humans pose little challenge to them? To answer this question, we target multipanel images, each involving a series of subfigures. These subfigures are presented together in certain layouts, such as web screenshots capturing multiple thumbnail images and posters utilizing multipanel formats to present a cohesive narrative or argument.
6845
We observe that while humans typically find interpreting multipanel images to be a straightforward task, MLLMs struggle with this challenge when presented with the entire multipanel image as input, as shown in Figure 1.
This study aims to holistically evaluate MLLMs in understanding multipanel images. We introduce the MultipanelVQA benchmark with 6,600 triplets of multipanel images, questions and answers, challenging models to answer each question based on the multipanel image. There are three questions with distinct types for each multipanel image: identifying common or unique contents across subfigures, pinpointing content in specific subfigures through positional descriptions, and locating subfigures via visual grounding in a multi-choice format. Especially, the first type of question mainly tests the MLLMs’ ability to reason about contents and the other two question types also assess the MLLMs’ understanding of multipanel image layouts in addition to the content reasoning ability.
Uniquely, the multipanel images in the MultipanelVQA benchmark features a diverse mix of real-world web screenshots, posters and synthetic multipanel images, categorized into real-world data and synthetic data subsets. Unlike the real-world data that requires human annotation, the synthetic multipanel images are automatically generated by scripts with subfigures from two existed datasets. The script ensures the generated synthetic multipanel images have even distribution of various attributes such as the number of subfigures, their sizes, and the complexity of layouts, etc. As a result, based on the synthetic data, we are able to precisely isolate and pinpoint the impact of their attributes on the performance of MLLMs.
We then benchmark popular open-sourced and proprietary MLLMs on the MultipanelVQA benchmark and conduct thorough error analysis with the help of the synthetic data, which delves into the reasons behind MLLMs’ difficulties in interpreting multipanel images. As a result, our main findings are 1) MLLMs are susceptible to content interference caused by the occurrence of multiple subfigures within the multipanel image. 2) The layout for subfigures has an impact on the MLLMs’ performance on multipanel images. MLLMs tend to be more successful in understanding multipanel images with layouts with fewer subfigures and larger subfigure sizes. 3) Adding sequential numbers for subfigures as visual prompt can benefit some MLLMs that are sensitive to embedded texts in the
input multipanel images. Last but not least, we explore how adding sequential numbers to subfigure captions in multipanel images, akin to the Set-of-Mark visual prompting method (Yang et al., 2023), improves MLLMs’ understanding of these images. We test MLLMs on multipanel images with and without sequential number captions for each subfigure. As a result, we observed that only GPT-4V (OpenAI, 2023b) and MiniGPT-v2 (Chen et al., 2023) show a notable improvement when the sequential number is not only embedded in the image but also explicitly mentioned in the question. In conclusion, the contributions of this study are listed as follows:
• We propose the MultipanelVQA benchmark with real-world and synthetic data that focus on evaluating the model’s ability to understand the content and layout of multipanel images.
• We benchmark several open-sourced and proprietary MLLMs with the MultipanelVQA benchmark and find that all models tested face a significant challenge in interpreting multipanel images despite their success on single-panel images.
• Benefited by the synthetic data with even distributions of various multipanel image attributes in the MultipanelVQA benchmark, we conduct thorough error analysis to uncover various factors that impact the model’s performance, including subfigure content, layout, background, and visual text hint in multipanel images.
• Finally, we investigate the potential of adding subfigure captions in multipanel images as visual prompts to enhance the performance of MLLMs on multipanel image understanding.

heading: 2 Related Work
text: Multimodal Large Language Models The development of Multimodal Large Language Models (MLLMs) has been propelled by advances in largelanguage models (LLMs)(Chung et al., 2022; Touvron et al., 2023a,b) and vision-and-language learning(Radford et al., 2021; Li et al., 2022), merging visual comprehension with LLMs for multi-modal tasks in a zero-shot manner (Tsimpoukelli et al., 2021; Alayrac et al., 2022; Li et al., 2023b). Instruction tuning, using visual instruction data derived from open-source datasets and pre-trained LLMs, enhances MLLMs’ zero-shot performance on complex tasks (Liu et al., 2023c; Zhu et al., 2023; Dai
Original multipanel image Augmented multipanel image Less subfigure visual similarity Larger subfigure size
Background added Visual text hint added
Question:

heading: Is there a car on the laptop screen in the
text: subfigure (Figure 0) located between the top left and bottom left? (please answer yes/no) Answer: No.
et al., 2023; Ye et al., 2023a). Further advancements include grounding and multilingual training to expand MLLMs’ capabilities (Chen et al., 2023; You et al., 2023; Li et al., 2023c). Evaluations for MLLMs With the advancement of MLLMs, there’s a growing need for comprehensive multi-modal benchmarks to assess their capabilities. Traditional tasks like image captioning (Chen et al., 2015; Agrawal et al., 2019) and VQA (Goyal et al., 2017; Hudson and Manning, 2019; Liu et al., 2023a), along with text recognition and knowledge-based reasoning (Marino et al., 2019; Schwenk et al., 2022; Lu et al., 2022), have been key in evaluating MLLMs. Newer benchmarks aim to assess models more holistically (Li et al., 2023a; Liu et al., 2023e; Yu et al., 2023; Cui et al., 2023). Recently, more holistic and comprehensive benchmarks have been proposed, which evaluate models’ comprehensive capabilities from multiple perspectives (Li et al., 2023a; Liu et al., 2023e; Yu et al., 2023; Cui et al., 2023). Unlike former evaluation benchmarks, we propose the MultipanelVQA benchmark that not only identifies a distinguished practical challenge in real life, multipanel image understanding, but also statistically analyzes the MLLMs’ capability through the synthetic data. Synthetic Data Synthetic data, recognized for its scalability, diversity, cost-effective annotations, etc, has been widely explored for enhancing model training, especially in vision-related tasks like semantic segmentation, object detection, and image classification (Chen et al., 2019; Yuan et al., 2023; Jahanian et al., 2021; Zhou et al., 2023). Additionally, synthetic data’s role extends beyond training to include model performance evaluation and analysis. Kortylewski et al. (2019) use synthetic faces to analyze neural network generalization across different poses, finding deeper architectures perform better. van Breugel et al. (2023) propose the 3S Testing framework to generate synthetic test sets that evaluate models under distributional shifts. In this work, we introduce the MultipanelVQA benchmark, enriched with synthetic data to conduct error analysis, exploring the factors influencing the performance of MLLMs on multipanel image understanding.

heading: 3 MultipanelVQA

heading: 3.1 Overview
text: We introduce the MultipanelVQA benchmark, consisting of multipanel images, questions, and answers, specially designed to assess the performance of MLLMs in interpreting multipanel images. As shown in Figure 2, the benchmark comprises two subsets: the real-world data subset, including actual web screenshots and posters collected by humans, and the synthetic data subset, consisting of multipanel images created by assembling individual images on blank canvases with automated scripts. As a result, the real-world subset provides realistic samples of multipanel images in everyday life, and the synthetic subset includes multipanel images with an even distribution of various attributes, including the style of the layout, number of subfigures, backgrounds, etc.
The MultipanelVQA benchmark demands that the evaluated model responds to questions linked to multipanel images, with each input consisting of a question paired with a multipanel image. As shown in Figure 2, in MultipanelVQA benchmark, there are three corresponding question-answer pairs {(qij , aij)|j ∈ [0, 2]} for a multipanel image vi. Each of the three questions features a unique style and focuses on evaluating the distinct ability of the model. Questions of the first style (Q1) assesses the model’s ability to discern if any or all subfigures contain a specific object or one with unique attributes, challenging it to recognize the content of every subfigures and their spatial distinctions. The second style of question (Q2) focuses on a particular subfigure’s content, while questions of the third style (Q3) features a visual grounding style in a multi-choice format requiring the model to select the positional description of the subfigure matching the given description. Notably, positional descriptions, such as “top left", exist in questions of the second and third question styles, introducing challenges due to the varying layouts of multipanel images. For example, the subfigure with a fixed position in a canvas is the topmost in one multipanel image might be the leftmost in another, depending on the arrangement of other subfigures.

heading: 3.2 Real-world Data Curation
text: In the real-world data subset of the MultipanelVQA, multipanel images are meticulously sourced from web screenshots in the Roboflow Website Screenshots dataset (Dwyer, 2020) and posters in task 3 of the DocVQA dataset (Mathew et al., 2021). Our data curation process begins with the manual selection of 100 images from the source, specifically chosen for their multipanel style featuring distinct subfigures. Then, for each selected image, we develop three questions. The questions are carefully designed to align with the three question styles of MultipanelVQA described in the previous section. After questions are gathered, we engage three graduate students to answer questions and validate them against the designated question types to guarantee the quality and relevance of our questions. Questions that fail validation are revised till all questions and answers are validated and collected.

heading: 3.3 Synthetic Data Curation
text: Generating synthetic multipanel images For the synthetic multipanel images, we use automated
scripts to create multipanel images. We first generate 210 random layouts of multipanel images in different styles. Each layout holds 2 to 8 subfigures and includes a predefined sequence for subfigures. As detailed in Appendix A.1, the layouts with more subfigures are populated from ones with fewer, so that when the subfigure number is increased, the positions of the existing subfigures are not changed. To generate synthetic multipanel images, we then compose single-panel images from two source datasets, MagicBrush (Zhang et al., 2023) and VQAv2 (Goyal et al., 2017), based on the layouts. Specifically, we preprocess these source datasets into sets of single-panel images with a common question and then arrange the single-panel images from the same set on a blank canvas according to the predefined layout and sequence. We provide more details about the process of multipanel image generation in Appendix A.2.
It is important to highlight that during the synthetic multipanel image curation, we filter the image sets derived from the source datasets by presenting each single-panel image within the image sets, along with the common question, to the MLLMs used in our experiments. We aim to ensure that the synthetically generated multipanel images only include subfigures that the MLLMs can accurately interpret when presented individually. This approach allows us to concentrate the evaluation squarely on the MLLMs’ proficiency with multipanel images, thereby minimizing the influence of varying domain knowledge that may arise from their distinct training backgrounds.
Generating questions and answers After generating these multipanel images, we utilize GPT-4 to create questions and answers for each image, drawing on information from the source datasets. Detailed in AppendixA.3, we design the prompt to ensure that the three questions generated for each image align with the question styles introduced in Section 3.1 consistently. For the second and third questions for each image where they target specific subfigures, human-annotated subfigure positional descriptions will be provided to GPT-4 as well. Additionally, we ensure the first subfigure added to the canvas is always the targeted subfigure, so that questions of multipanel images consisting of the same subfigure with different layouts will have similar questions that only vary on the positional description. We manually cross-validate all the questions and answers after the data curation.
Augmenting synthetic multipanel images Additionally, we uniformly augment the synthetic data subset with several variations to the multipanel images: 1) Reducing the visual similarity among subfigures in multipanel images. 2) Increasing subfigure sizes while maintaining the overall multipanel image’s layout. 3) Replacing the plain white background with a black and white chessboard pattern. 4) Embedding text within the images that contain ground truth information as captions for the subfigures. Please refer to Appendix A.4 for more details and examples. These augmentations enhance the complexity of the synthetic data subset of MultipanelVQA and create a test bed for comparing MLLMs’ performance in interpreting multipanel images under varied conditions.

heading: 3.4 Data Statistics
text: Data in the MultipanelVQA benchmark comprises a substantial collection of 6,600 image-questionanswer triplets, equating to unique multipanel images in two subsets: the real-world data subset, consisting of 100 multipanel images sourced from actual scenarios, and the synthetic data subset that includes a larger compilation of 2, 100 images, designed for controlled condition analysis. We detail the statistics regarding the multipanel images of MultipanelVQA in Table 1. The dataset’s questions vary in length, with an average word count of 18.7. In terms of questions, 56.9% are Yes/No queries, 33.3% are multiple-choice questions, and the remainder are questions with specific categorical answers, such as identifying colors.

heading: 4 Experiments
text: We first evaluate eight popular Multimodal Large Language Models (MLLMs) on MultipanelVQA. Then, based on the evaluation results, we conduct a thorough error analysis.

heading: 4.1 Setup
text: MLLMs The MLLMs that we adopt in the evaluation include both open-source models and proprietary models with only API access. The open-source MLLMs are (i) LLaVA-1.5-13B (Liu et al., 2023b), (ii)LLaVA-NeXT, (iii) MiniGPT4v2 (Chen et al., 2023), (iv) InstructBLIP (Liu et al., 2023c) with Flan-T5 XXL (Chung et al., 2022) as the LLM backbone, and (v) mPLUG-Owl2 (Ye et al., 2023b). We implement the models using their default settings and detail their supported input image resolutions in Appendix C. For proprietary models, we evaluate GPT-4V (OpenAI, 2023b) with the gpt-4-vision-preview OpenAI API during June of 2024, GPT-4o (OpenAI, 2024) during June of 2024 and Gemini Pro Vision(Team et al., 2023) during January of 2024.
Evaluation In our evaluation process, we initially utilize scripts to compare the MLLM’s predicted answers against the ground truth for straightforward assessments. This is particularly effective for close-ended questions like multiple-choice or yes/no questions. For cases where the MLLM’s output differs from the ground truth, we employ GPT-4 (OpenAI, 2023a) as a secondary judge, assessing whether the MLLM’s predicted answer, can be considered correct, especially in terms of encompassing all information present in the ground truth answer. Recent research, as cited in (Hsu et al., 2023; Hackl et al., 2023; Liu et al., 2023d), has highlighted GPT-4’s capability and reliability in such evaluative roles. The details of the prompts used for this GPT-4 evaluation are provided in Appendix D.

heading: 4.2 Main Result
text: We assess the performance of eight leading Multimodal Large Language Models (MLLMs) using both synthetic and real-world subsets of the MultipanelVQA benchmark. We run the evaluation process for 3 times and Table 2 presents the average accuracy of each model’s output for individual questions with standard deviation. The result reveals that proprietary models (GPT-4V, GPT-4o and Gemini Vision Pro) consistently outperform
the other models across both subsets. However, as introduced in Section 3.2, we make sure all MLLMs tested can achieve a 100% accuracy when the subfigures are input individually, thus even the best-performing model, GPT-4o, shows an average 20% performance drop when dealing with multipanel images rather than single-panel images. Additionally, we hire human testers from both Amazon Mechanical Turk and campus to establish human performance. It’s important to highlight that a significant disparity exists between the models’ performances and the human-level performance, and some models even tie with the random baseline. This underscores the considerable room for improvement in current MLLMs’ capabilities in handling complex multipanel image comprehension.

heading: 4.3 Error Analysis
text: Intending to identify potential error causes, we first examine the models’ outputs from the real-world data subset benchmarking results. A case study is presented in Figure 3, and we present more examples in Appendix B. Based on this example and others from the real-world data subset, we find that while the models can generate responses relevant to the posed questions, the accuracy of these answers often falls short. Based on observations, we suggest that errors in the model output primarily arise from three sources: 1) Difficulty in understanding small image sections with fewer pixels and confusion caused by neighboring subfigures in multipanel images 2) Insufficient multipanel image layout reasoning ability, and 3) Misleading factors
such as background elements and textual content within the multipanel images. However, given the complexity of real-world multipanel images, pinpointing the exact influence of each issue is difficult. Thus, we leverage the synthetic data subset of the MultipanelVQA benchmark to conduct comparative experiments isolating and evaluating the influence of distinct factors.
How susceptible are MLLMs to neighboring subfigure interference and diminished pixel detail in visual targets? To evaluate the MLLMs’ resilience to neighboring interference, we conduct an ablation study on the synthetic multipanel images as shown in Figure 4. Initially, for a given question targeting a subfigure within a multipanel image, we isolated the subfigure targeted by removing all others, leaving a single subfigure in the image. This modification led to improved performance across all models, suggesting their susceptibility to interference from the presence of multiple subfigures. Further, we refine the ablation to present only the target subfigure as a single-panel image input, allowing more pixels to the visual content related to the question in the image input. In this scenario, all models successfully interpreted the images, however, for most models, such improvement is less significant than the one received from the removal of neighboring subfigures. This suggests that MLLM’s performance drop when understanding multipanel images is affected by both the interference from adjacent subfigures and the reduced pixel allocation to the target content but the former is more critical for most models tested.
Question 1 (Q1): Does any subfigure include a man? (please answer yes/no)
LLaVA: Yes. LLaVA-NeXT: Yes. MiniGPTv2: Yes. InstructBLIP: No. mPLUG-Owl2: Yes. GPT-4V: Yes. GPT-4o: Yes. Gemini Pro Vision: Yes.
Question 2 (Q2): Does the top left subfigure include a man? (please answer yes/no)
LLaVA: Yes. LLaVA-NeXT: Yes. MiniGPTv2: Yes. InstructBLIP: No. mPLUG-Owl2: Yes. GPT-4V: Yes. GPT-4o: Yes. Gemini Pro Vision: Yes.
Question 3 (Q3): In which subfigure is a man included?
a) the top left subfigure
b) the top right subfigure
c) the bottom right subfigure
d) None
(please select one)
LLaVA: d) None LLaVA-NeXT: a).. MiniGPTv2: a. InstructBLIP: d) None mPLUG-Owl2: c) the bottom right subfigure GPT-4V: a) the top left subfigure GPT-4o: c) Second subfigure in the second row. Gemini Pro Vision: a) the top left subfigure
A multipanel image from the real-world subset
of the MultipanelVQA benchmark
Questions and corresponding outputs from MLLMs
Figure 4: Model performance on questions of the second style (Q2) in the synthetic data subset when multipanel images are simplified to blank canvases, each with a targeted subfigure and then to single-panel images of the targeted subfigures, while maintaining the same input questions. The result indicates a significant vulnerability of the MLLMs to interference from adjacent subfigures.
Additionally, we explored how models’ performance fluctuates with varying visual similarity of subfigures’ content. From human intuition, the more similar the subfigures, the harder to distinguish the targeted subfigure. The result, depicted in Table 3, shows that except for MiniGPT-v2, InstrucBLIP and mPLUG-Owl2, all other models experienced a performance rise when subfigures
within multipanel images are less similar.
How does MLLM’s performance vary to different multipanel image layouts? We further categorize data from the synthetic data subset of MultipanelVQA based on the layout style and subfigure size, as shown in Table 3. We bserve that multipanel image layout has varied influence among models. For most MLLMs evaluated, subfigure size and layout style play a crucial role, with larger subfigures and grid layout style generally leading to better performance. Moreover, we illustrate the impact of subfigure quantity on model performance in Figure 5, revealing a common trend where all models exhibit decreased effectiveness as the number of subfigures increases.
What is the influence of background and visual text hints on MLLM’s multipanel image interpretation ability? Last but not least, we also investigate how other sources of interference affect the ability of MLLMs to interpret multipanel images, specifically background elements and visual texts embedded on the image as hints. Examples are shown in Figure 6. Specifically, we compare the performance changes in MLLMs when presented with or without chessboard background and the presence or absence of subfigure captions with
Figure 5: Impact of Subfigure Quantity on Model Performance. A common trend exists where all models exhibit declining performance as the number of subfigures increases, with varying degrees of impact.
ground truth information as visual text hints. As indicated in Table 3, the top four best performing models, LLaVA-NeXT, GPT-4V, GPT-4o Gemini Pro Vision show substantial improvements when the background is eliminated. However, the inclusion of visual text hint appears to have various effects on the performance of models, which suggest model’s different sensitivity to text embedded in the input image. We believe such sensitivity can be leveraged for enhancing model’s performance towards better multipanel image understanding. Some of our attempts are detailed in the next subsection.

heading: 4.4 Influence of Adding Subfigure Captions with Sequential Numbers as Visual Prompts
text: Based on our findings of the visual text hint’s influence over the interpretative capabilities of MLLMs on multipanel images, we explore adding captions with sequential numbers for subfigures as visual prompts, akin to the Set of Mark (SoM) visual
Original multipanel image
Question:
Is there a car on the laptop screen in the
subfigure (Figure 0) located between the top left
and bottom left? (please answer yes/no) Answer:
No.
prompting method (Yang et al., 2023). We compare the model’s performance on the multipanel images in the synthetic data subset with and without such subfigure captions to assess the impact. We provide a demenstration in Appendix E. Results are shown in Table 4, revealing that applying these captions with numbers as visual prompts led to little to no improvements in model performance. However, we further attempt to not only add captions with sequential numbers but also explicitly incorporate
the number from the caption into the question sent to MLLMs. We find that when the number in the targeted subfigure’s caption is explicitly mentioned in the input question, InstructBLIP, MiniGPT-v2, GPT-4V, and Gemini Pro Vision demonstrate performance enhancements. This suggests that such a visual prompting method relies not only on the marks added to the input image but also on their direct integration into the query context. We also find that the result aligns with how the models’ performances change after the visual text hint is added (Section 4.4), underscoring the varying nature of MLLMs’ abilities to utilize visual prompts. This necessitates further exploration and development of tailored strategies for effectively integrating visual prompts into different MLLMs.

heading: 5 Discussion and Conclusion
text: In this study, we introduce the MultipanelVQA benchmark, designed to evaluate the capability of Multimodal Large Language Models (MLLMs) in interpreting multipanel images. This benchmark, comprising both real-world and synthetic data, enables a detailed analysis of MLLMs on their multipanel image understanding abilities. Our results highlight a significant performance gap between MLLMs and humans, especially since humans achieve nearly perfect scores in this benchmark. This gap highlights the current limitations of MLLMs in interpreting highly structured visual information and pinpoints the specific need where further model training and refinement are necessary.
Moreover, by analyzing MLLMs’ abilities to effectively interpret multipanel images, we believe our benchmark can facilitate the development of specialized algorithms, such as severing as a tool to select strong MLLMs in tasks related to Graphic User Interface (GUI) understanding (You et al., 2024; Zheng et al., 2024). Also, as MultipanelVQA identifies the key factors in multipanel images that affect model performance, it can inspire and guide related applications, for example, presenting lengthy sequences of images as subfigures in multipanel images (Fan et al., 2023).
Last but not least, the synthetic data of MultipanelVQA helps isolate specific performance factors and ensures that the test images were not part of the models’ training datasets. This is essential for large-scale MLLMs with undisclosed training data. The creation method for these synthetic images is
replicable, enabling ongoing generation of new test images and potentially aiding broader AI evaluation efforts.

heading: 6 Limitation
text: Our study provides an in-depth evaluation of MLLMs on multipanel images using the proposed MultipanelVQA benchmark. The use of GPT-4 as an evaluator necessitated the simplification of questions to primarily yes/no or short-answer formats to allow for automated non-human evaluation. This constraint potentially limits the assessment’s depth and we leave the development of evaluation with more complex questions for future research. Additionally, the synthetic data, although crucial for statistical analysis, faces challenges due to the very poor performance of some models that are close to the random baseline. The extreme underperformance of those models restricts our error analysis, as it is difficult to derive meaningful conclusions from such low accuracy levels.

heading: 7 Acknowledgement
text: The authors would like to extend their sincere thanks to the engaging discussions initiated by a Twitter post about the ‘Muffin or Chihuahua’ topic1, which helps solidify this study.

heading: A Synthetic Data Generation Details
text: A.1 Layout Generation
To generate synthetic multipanel images automatically, we first develop scripts to generate random layouts for subfigures in multipanel images. There are two scripts, generating layouts in splashed and grid style respectively, where splashed style has subfigures scattered in the canvas and grid style has the subfigures tightly arranged in the canvas. We provide examples in Figure 11. Both scripts generate the layout by sequentially determining the position of maximum 8 subfigures in a 1000×1000 pixels blank canvas, where there is a random selector selecting the position and size for the next subfigure from all possible candidate positions after the last subfigure is determined. Every time a new subfigure position is determined, a new layout is generated, so the number of subfigure in the layout ranges from 2 to 8. At the same time, a subfigure sequence is recorded based on the order that their position is determined in the layout.
To generate different layout styles, each script has different rules of selecting candidate positions and the size of the next subfigures. Specifically, to generate splashed style layouts, the candidate position of the next subfigure can be anywhere in the canvas as long as it is not overlapped with existing ones and the size of the subfigure is the same within the same layout, which is randomly chosen in the range of [180, 220] pixels. On the other hand, for grid style layouts, the candidate positions are restricted to be either in the same row or column as the previously determined subfigure’s position. Additionally, the size of the next subfigure will be either the same as the predetermined size in the range of [180, 220] pixels, or twice as large as the predetermined size. As a result, the grid style layouts we randomly generated include two layouts with all subfigures in the same size and another two layouts with different size subfigures.
A.2 Multipanel Image Generation
In order to generate multipanel images, each with a consistent source, we first preprocess both source datasets, MagicBrush (Zhang et al., 2023) and VQAv2 (Goyal et al., 2017), unifying the formats of the two source datasets to be sets of images with the same question. Specifically, for MagicBrush where there are originally sets of images, each sharing a common image as an image editing source, we create a template-based question asking about
the visual component being edited for every image set; for VQAv2, we gather images with the same question in the dataset. We show example sets of the pre-processed source datasets in Figure 12.
Then, based on the aforementioned layouts for synthetic multipanel images and the sequence of the subfigure in the layout, we select images from the same image set in the source dataset and add them to a blank canvas. In this process, we make sure the selected images for every multipanel image include only one image with a unique answer, and we place it at the first in the sequence. Additionally, we use each image set to fill all layouts we generate, which ensures independent distributions of the subfigure content and layout.
We illustrate this process in Figure 13, where every time a new image is added to the blank canvas, a new synthetic multipanel image is created.
3
0 1 2
4
5 6 7 Question 1: Do all the subfigures contain the same food item? Question 2: What food is shown in the top right subfigure? Question 3: Which subfigure contains cake? a) second in the middle row b) first in the bottom row c) top right d) bottom left
Synthetic multipanel imagesLayouts
… … …
Real-word multipanel images
MultipanelVQA
Questions for one multipanel image
(example)
5
41 6 72 3 0
Augmented multipanel image
Less subfigure visual similarity
Larger subfigure size
Background added Visual text hint added
Question: Is there a car on the laptop screen in the subfigure (Figure 0) located between the top left and bottom left? (please answer yes/no) Answer:
No.
(i)
3
0 1 2
4
5 6 7 Question 1: Do all the subfigures contain the same food item? Question 2: What food is shown in the top right subfigure? Question 3: Which subfigure contains cake? a) second in the middle row b) first in the bottom row c) top right d) bottom left
i lti l it l- ord ultipanel images
ultipanelVQA
Questions for one multipanel image
(example)
5
41 6 72 3 0
Augmented multipanel image
Less subfigure visual similarity
Larger subfigure size
Background added Visual text hint ad ed
uestion: Is there a car on the laptop screen in the
subfigure ( igure 0) located bet een the top left and botto left? (please ans er yes/no)
s r: .
(ii)
Figure 7: Example for (i) a multipanel image with subfigure captions including sequential numbers and (ii) a question and answer where the question explicitly refers the subfigure caption (highlighted “Figure 0").
3
0 1 2
4
5 6 7 Question 1: Do all the subfigures contain the same food item? Question 2: What food is shown in the top right subfigure? Question 3: Which subfigure contains cake? a) second in the middle row b) first in the bottom row c) top right d) bottom left
Synthetic multipanel imagesLayouts
… … …
Real-word multipanel images
MultipanelVQA
Questions for one multipanel image
(example)
5
41 6 72 3 0
Question: Is there a car on the laptop screen in the subfigure (Figure 0) located between the top left and bottom left? (please answer yes/no) Answer:
No.
A.3 Question-Answer Generation
We prompt GPT-4 to generate three questions in three distinct styles and corresponding answers for each multipanel image, given the fact that all subfigures in a synthetic multipanel image come from the same image set in the source dataset and share a common question. The first question a ks if all or any subfigure have a specific object or object attribute which is mentioned in he comm n question of the imag et. The s cond and third will focus on the content of a specific subfigure, which is the one with a unique answer to the common question shared in the image set. The prompt, shown in Table 7 includes detailed instructions for how to generate the question-answer pairs while requiring information about the multipanel image which consists the subfigure numbers, the common question for the subfigures, the answer of the target subfigure to the common question and the positional description of the target subfigure which we man-
6857
ually annotate the positional description for each subfigure in advance.
A.4 Augmentation of the Synthetic Data Subset
We augment the synthetic data subset of the MultipanelVQA benchmark to enable a more comprehensive evaluation of MLLMs performance on multipanel image understanding. The augmentation is done by involving new multipanel images that are altered from the original version in four different ways while keeping the corresponding questions and answers the same. First, we reduce the visual similarity among subfigures in multipanel images by generating new subfigures to replace the original ones. Since the original subfigures in each multipanel image come from the same image set of the source dataset, they share a visual similarity as they have a common question, and many even have the same answer to the common question. In order to reduce this similarity while keeping the questions and answers for the multi-panel image unaffected, we prompt DALL·E 3 (Betker et al., 2023) to generate various images that do not incur the same answer to the common question as the target subfigure and then replace the subfigures except the target subfigure with these newly generated images. As shown in Figure 8, in this way, subfigures in multipanel images, especially those based on MagicBrush (Zhang et al., 2023) dataset, become less similar to each other visually. Second, we increase the subfigure size within the multipanel images by first removing some edge space for the multipanel image while keeping the ratio of height and width and then resizing the image to the original size. Third, we add a background with black and white chessboard patterns to every synthetic multipanel image, introducing a more complex visual backdrop. Last, we embed texts to the multipanel image, where these texts include the common question and the corresponding answers of each subfigure.

heading: B Samples of Model Outputs on Real-world Multipanel Images
text: We show some more real-world multipanel images of web screenshots and posters along with model outputs in Figure 9. Additionally, there an sample from the synthetic data subset in Figure 10.

heading: C Supported Input Image Resolutions of Tested MLLMs
text: We show the supported input image resolutions of four tested open-sourced MLLMs in Figure 5. As illustrated in Figure 4, the variation of input image resolution is a valid factor in model performance.

heading: D GPT-4 as Evaluator
text: Given the output of MLLMs with the question and multipanel image as input, we prompt GPT-4 to judge if the output is a correct answer. The prompt is shown in Table 6, where the question, model’s output and corresponding ground truth are inserted. If GPT-4’s output is yes, we regard the model’s output as correct and vice versa.

heading: E Examples of Subfigure Captions with Sequential Numbers as Visual Prompts
text: We experiment with adding captions to subfigures in the synthetic data subset of MultipanelVQA as a visual prompting method similar to the Set of Mark (SoM) visual prompting method (Yang et al., 2023). The caption we add to the subfigures includes sequential numbers, as shown in Figure 7i. Besides changing the multipanel images with subfigure captions, we also modify the corresponding questions to refer to the subfigure caption explicitly, as shown in Figure 7ii.
MiniGPTv2: a
InstructBLIP: a) first in the top row
mPLUG-Owl2: c) second in the second row GPT-4V: b) third in the first row GPT-4o: The dog is located in subfigure (b) third in the first row. Gemini Pro Vision: b) third in the first row
Question 1 (Q1): Does any subfigure contain three people? (please answer yes/no) Question 2 (Q2): Does bottom left subfigure contain flags? (please answer yes/no)
LLaVA: Yes. LLaVA-NeXT: Yes. MiniGPTv2: Yes. InstructBLIP: No. mPLUG-Owl2: Yes, the bottom left subfigure contains flags. GPT-4V: Yes. GPT-4o: Yes. Gemini Pro Vision: Yes.
Question 3 (Q3): Which subfigure contains flags? a) top left b) top right c) bottom left d) bottom right (please select one)
LLaVA: a) top left LLaVA-NeXT: c. MiniGPTv2: a. InstructBLIP: a) top left mPLUG-Owl2: d) bottom right GPT-4V: The subfigure containing flags is:\nc) bottom left GPT-4o: c) bottom left. Gemini Pro Vision: c) bottom left

heading: A multipanel image from the real-world subset
text: of the MultipanelVQA benchmark
Questions and corresponding outputs from MLLMs LLaVA: Yes. LLaVA-NeXT: Yes. MiniGPTv2: Yes. InstructBLIP: No. mPLUG-Owl2: No. GPT-4V: No. GPT-4o: Yes. Gemini Pro Vision: Yes.
Is the second image in the top row showing a dog or a muffin?
… …
2
3
0
1 2
4
5
6
7
…
1
5
6
3

heading: Future Work
text: This constraint potentially limits the assessment’s depth and we leave the development of evaluation with more complex questions for future research. Additionally, the synthetic data, although crucial for statistical analysis, faces challenges due to the very poor performance of some models that are close to the random baseline