CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models (2024)

Abstract: Multimodal large language models (MLLMs) have demonstrated promising results in a variety of tasks that combine vision and language. As these models become more integral to research and applications, conducting comprehensive evaluations of their capabilities has grown increasingly important. However, most existing benchmarks fail to consider that, in certain situations, images need to be interpreted within a broader context. In this work, we introduce a new benchmark, named as CODIS, designed to assess the ability of models to use context provided in free-form text to enhance visual comprehension. Our findings indicate that MLLMs consistently fall short of human performance on this benchmark. Further analysis confirms that these models struggle to effectively extract and utilize contextual information to improve their understanding of images. This underscores the pressing need to enhance the ability of MLLMs to comprehend visuals in a contextdependent manner. View our project website at https://thunlp-mt.github.io/CODIS.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10639–10659 August 11-16, 2024 ©2024 Association for Computational Linguistics
1 Introduction
Recent years have witnessed a rapid advancement in multimodal large language models (MLLMs, OpenAI 2023; Gemini Team et al. 2023; Liu et al. 2023a; Dai et al. 2023). They have achieved remarkable results on various downstream tasks, such as image captioning (Luo et al., 2023; Wang et al., 2023; Chen et al., 2015), visual question answering (Shao et al., 2023; Liu et al., 2023c; Antol et al.,
*Equal Contribution. †Corresponding Authors.
2015) and visual reasoning (Gupta and Kembhavi, 2023; Chen et al., 2023c; Zellers et al., 2019). As the performance of MLLMs continues to improve, the comprehensive assessment of their capabilities becomes increasingly important.
Previous research highlights a fascinating and vital human ability: understanding visual elements within broader contexts (Baldassano et al., 2016). Consider the example in Figure 1, where simply looking at an image does not reveal whether the stairs go up or down. However, introducing the context of whether the greenery is above or below
10639
the viewpoint of the photographer enables humans to clarify the ambiguity and accurately interpret the image. In this study, we argue that MLLMs should also be evaluated from the perspective of context-dependent visual comprehension. From a scientific perspective, it is important to understand the extent to which MLLMs can leverage context to enhance their visual comprehension. Practically speaking, as MLLMs are increasingly applied in real-world scenarios, assessing their reliability in interpreting visual data within context is essential for ensuring they provide accurate responses.
However, the capability for context-dependent visual understanding has not been fully assessed by current benchmarks for MLLMs. Table 1 summarizes recent benchmarks for MLLMs. Most of these benchmarks (Fu et al., 2023a; Liu et al., 2023d; Li et al., 2023b; Guan et al., 2023; Cui et al., 2023; Sanders et al., 2022; Schmarje et al., 2022) do not pair images with additional context. Only two benchmarks, namely VisDial (Das et al., 2017) and MMDialog (Feng et al., 2022), include extra context to help in conversation with humans rather than to clarify the meaning of images as shown in Figure 1. This limitation means these benchmarks are not fully capable of testing the ability of MLLMs to understand images in a context-dependent manner.
To address this challenge, we introduce a new benchmark, named as CODIS, designed to evaluate the capability of MLLMs in COntext-Dependent Image diSambiguation. CODIS utilizes the visual question answering (VQA, Antol et al. 2015) format for this purpose, as shown in Figure 1(b). It stands out from existing benchmarks in three main aspects: first, each image in CODIS contains inherent ambiguity that can only be resolved with additional context; second, the questions are de-
liberately designed to highlight these ambiguities, requiring external context for accurate interpretation; third, for every image-question pair, we provide two contexts in a free-form text format. These contexts are subtly different yet lead to different interpretations of the image and, consequently, different answers. We have carefully curated all images, questions, and contexts by hand to maintain a high standard of quality and diversity. Our assessment of 14 widely-used MLLMs with CODIS indicates that the performance of these models in understanding context-dependent visuals significantly falls short of human capabilities. Further analysis demonstrates that the models struggle with identifying crucial contextual cues and extracting relevant visual features.
To summarize, our contributions are three-fold:
• We highlight the significance of contextdependent visual comprehension abilities for MLLMs.
• We introduce the CODIS benchmark to assess capabilities of MLLMs on context-dependent visual comprehension.
• Through our analysis, we uncover the deficiencies in MLLMs regarding context information extraction and visual information extraction, underscoring the immense potential for enhancement in the realm of context-dependent visual comprehension.
2 Related Work
2.1 Context in Visual Tasks
Previous works emphasize the importance of context in visual tasks (Wang and Zhu, 2023; Vo, 2021; Goh et al., 2004; Bar and Aminoff, 2003),
exploring utilizing context in scene graph generation (Cong et al., 2023; Zheng et al., 2023; Zhu et al., 2022; Yang et al., 2018), object detection (Zou et al., 2023; Diwan et al., 2023; Xiao et al., 2023; Chen et al., 2023b), image inpainting (Zhang et al., 2023; Sargsyan et al., 2023; Xiang et al., 2023), etc. However, they mainly focus on visual context within images, e.g., relationships between objects, environment and background. External contexts, such as free-form texts, have also been studied. Previous works have shown their effectiveness in tasks such as floorplan reconstruction (Purushwalkam et al., 2021), and visual dialog (Das et al., 2017; Feng et al., 2022). The ability of multimodal in-context learning (Li et al., 2023a; Zhao et al., 2023; Sun et al., 2023; Li et al., 2023c) further demonstrates the significance of external contexts. However, in these works, context do not affect the interpretation of images, which is the main focus of our work. We strengthen the impact of context on visual comprehension.
2.2 Visual Ambiguities Visual ambiguities arises from absence of visual information or other noises (Denison et al., 2018). It is an important phenomenon in human visual cognition. Previous works investigate visual ambiguities caused by optical illusions (Guan et al., 2023; Cui et al., 2023; Fu et al., 2023b). Images which may cause illusions are leveraged to evaluate visual hallucinations in MLLMs. Other works explore ambiguities in visual event classification (Sanders et al., 2022; Schmarje et al., 2022; Rajeswar et al., 2022), focusing on model robustness against ambiguous images. Different from these works, we assess capability of MLLMs to disambiguate images instead of just recognizing ambiguities.
3 CODIS
CODIS is proposed for evaluating the capability of MLLMs in context-dependent image disambiguation. Figure 2 presents several examples from our benchmark, highlighting the diversity of contexts covered. In this section, we first elaborate our taxonomy of context. Then, we delve into the instruction design and evaluation method of CODIS. Finally, we introduce procedures of data collection.
3.1 Taxonomy of Context Given the extensive and varied nature of context information, it is challenging to catalog all forms of context comprehensively. Nevertheless, in our
effort to establish a pioneering benchmark for context-dependent image disambiguation, we aim to cover a wide range of diverse and illustrative scenarios. Drawing inspiration from the types of information people require to understand the visual contents of an image, we have identified five representative types of context. These include three types of global context that pertain to the overall scene—namely, the global background, which encompasses location and orientation, temporal information, and cultural background. Additionally, we focus on two types of local context related to objects within the scene, specifically the attributes of objects and the relationships between people. We present examples in Figure 2 and provide an explanation of our classification as follows.
Location and orientation. Interpretations of scenes is closely related to where they happen. As Figure 2(a) shows, the scene of falling leaves indicates different times in a year when they happen in different countries. Orientation also plays an important role in image understanding. In Figure 2(b), direction which the photographer is facing can help determine direction in which the herd is moving.
Temporal information. Understanding the timing and sequence of events is crucial when we understand a scene. However, an image can only provide us with static information. Thus, context of temporal information is often crucial, and can help better restore the original appearance of events. Figure 2(c) shows an example of context which indicates dynamic changes of things and events, like opening or closing the door, while context of Figure 2(d) tells when pictures were taken. 1
Cultural background. Cultural background has a profound impact on the way people think and act. Meaning of an action may have significant differences across different nations and cultures sometimes. Awareness of cultural background is crucial for understanding images. We employ cases covering various aspects of differences between cultures, such as laws, regulations, customs and traditions. See Figure 2(e) and Figure 2(f) for examples.
Attributes. To comprehensively understand objects in a scene, it is not enough to solely rely on visual appearances we see. Non-visible attributes
1One might argue that introducing video is the most direct method to resolve temporal ambiguities. However, videos are not always available in practice. By analogy to humans, a sufficiently capable MLLM should also be able to resolve such ambiguities without relying on videos. Therefore, we believe it is necessary to include this category of contexts in our benchmark.
are also important. We consider two types of attributes, including physical attributes, such as structure, size, temperature, smell and touch, as Figure 2(g) shows, and abstract attributes, such as function and mechanism, as Figure 2(h) shows.
Relationships. We communicate with people in different ways based on our relationships with them. When the same event happens on people with different relationships, its meaning varies. It is important to understand relationships between people in visual comprehension. As Figure 2(i) and Figure 2(j) shows, relationships between people affect our judgement on their roles and identities.
3.2 Instruction Design To prevent models from guessing the correct answers without fully understanding context, we organize our dataset in pairs. Each pair contains two queries (I,Q,C1) and (I,Q,C2). The queries
have identical image I and question Q, but have two pieces of different context C1 and C2. We give MLLMs two queries separately, and get model outputs O1 and O2. Take Figure 2(a) as an example. In the first query, we provide a picture of falling leaves and indicate that the photo was taken in Australia. In the second query, we give different context that the photo was taken in America. We expect to get different outputs from models that the photo was taken in the first or the second half of the year.
3.3 Evaluation Method We use O1 and O2 to denote model outputs of a pair of queries, and A1 and A2 to denote the groundtruth answers. The evaluation of model outputs can be represented as follows:
Eval(Oi) =
{ 1 if Oi matches Ai
0 otherwise , i = 1, 2.
Following Fu et al. (2023a), we leverage two metrics, pair-wise accuracy Accp and query-wise accuracy Accq for our evaluation metrics, which are calculated as follows:
Accp = 1
np
np∑
i=1
(Eval(Oi1)× Eval(Oi2)),
Accq = 1
2np
np∑
i=1
(Eval(Oi1) + Eval(Oi2)),
where np is number of data pairs. For Accp, models score only if their answers to a pair of queries are both correct. For Accq, models score for each single query they answer correctly.
3.4 Data Collection In this section, we introduce the process of constructing our benchmark. Our data collection process includes three steps.
Image collection. We manually collect images that contain ambiguities which can only be resolved with external contexts. The majority of these images are real-scene images from the publicly available dataset ShareGPT4V (Chen et al., 2023a) and the Internet, while the remainder are created manually. The primary challenge in collecting images was that many appeared ambiguous at first glance but were actually unambiguous upon closer examination. Therefore, each image had to be carefully inspected and excluded from the collection if it was found to be unambiguous. In total, we collected 377 images.
Design of questions, context and answers. For each collected image, we manually write questions, context and answers for it. The data are compiled with following rules:
(1) Questions are targeted at the ambiguous parts of the images. They can not be answered if no additional context is provided due to ambiguities in the images.
(2) For each image-question pair, two unique contexts with minor differences that lead to different interpretations of the image need to be created. As a result, these interpretations yield different answers to the question. Furthermore, we require that an answer can only be derived when both the image and its corresponding context are available. This means that neither the image nor the context alone is sufficient to provide an answer. For instance, in Figure 2(a), the photograph of falling leaves, when combined with the knowledge that the photograph was taken in Australia, allows us to conclude that it depicts autumn in the southern hemisphere, occurring in the first half of the year. In isolation, neither the image nor the context is adequate to reach this conclusion.
(3) To enhance the evaluation process, MLLMs are permitted to respond in free-form text. However, to maintain a balance between the ease of evaluation and ensuring objectivity, we require the answers to be relatively objective. For example, despite the lack of a strict format for the responses to the scenarios shown in Figure 2, we provide clear options for models to choose in most of the cases. This approach guarantees that the responses stay objective and are easy to evaluate.
Data verification. Five annotators participated in the data collection process. To ensure the quality of our dataset, each submission by an annotator was cross-checked by the other four. Data was retained only if it (1) was correct, (2) differed from existing data to a significant extent, thereby main-
taining high diversity, and (3) conformed to all the above criteria. If these conditions were not met, the annotator had to revise the data.
Ultimately, we retained 216 images and successfully collected a total of 706 queries, spanning five categories and encompassing a wide range of scenarios. We had to exclude 161 images because we were unable to collect eligibility questions or contexts for them. The distribution of categories and scenarios is illustrated in Figure 3.
4 Evaluation
4.1 Models As shown in Table 2, we perform evaluation on 14 popular MLLMs, which are divided into three groups: (1) API-based models: GPT-4V (OpenAI, 2023) and Gemini (Gemini Team et al., 2023); (2) Open-source ~7B models: mPLUG-Owl-2 (Ye et al., 2023), MiniGPT-4-7B (Zhu et al., 2023),
LLaVA-1.5-7B (Liu et al., 2023a), InstructBLIP7B (Dai et al., 2023), Otter-7B (Li et al., 2023a), LLaVA-7B (Liu et al., 2023b), Qwen-VL (Bai et al., 2023), OpenFlamingo (Awadalla et al., 2023) and BLIP-2-6.7B (Li et al., 2023d); (3) Open-source ~13B models: LLaVA-1.5-13B, BLIP-2-11B and InstructBLIP-13B.
4.2 Results
We manually assess the outputs of MLLMs on CODIS benchmark and report the results in Table 3. “Human” refers to the average performance of five independent people, and the detailed results and analysis are shown in Appendix B.
Overall results. From an overall perspective, human is far ahead of all the MLLMs in all the categories, indicating that MLLMs have tremendous potential of improvement on context-dependent visual comprehension. API-based models significantly outperform open-source models, and GPT4V gets the highest score among all the MLLMs. In terms of open-source models, mPLUG-Owl-2 has the best performance on Accp and LLaVA-1.5-13B has the best performance on Accq.
Disparity between Accp and Accq. We observe a large disparity between Accp and Accq of MLLMs, and open-source models have a larger disparity than API-based models. To better understand the reason of this phenomenon, we further investigate the metric of context awareness, which reflects model capability of recognizing different context and provide different responses accordingly. It is
calculated as follows:
ContextAwareness = 1
np
np∑
i=1
(Oi1 ̸= Oi2),
where np is number of data pairs. We manually assess inequality of model outputs on semantic level. The results are shown in the last column of Table 3. The scores of all the MLLMs are lower than 60, while human score is 97.3, indicating that for many data pairs, MLLMs fail to recognize the difference in context and they provide semantically equal response to the two queries. Thus, for a significant number of cases, models only correctly answer one of the two queries, leading to high Accq and low Accp scores. Additionally, context awareness is positively correlated with Accp, and has no clear correlation with Accq. Therefore, Accp is more possible to reflect model capability of context comprehension, and is more suitable as the primary metric for CODIS benchmark. We further investigate the capability of MLLMs on extracting information from context. By excluding context and providing only images and questions as inputs, we observe the metric of output variability, which reflects the variations in model outputs. It is calculated as follows:
OutputVariability = 1
nq
nq∑
i=1
(Oinc ̸= Oic),
where nq is the total number of queries, Onc and Oc are outputs when context is given and is not
given. Inequality of model outputs is assessed manually on semantic level. We conduct our experiment on API-based models GPT-4V and Gemini, and mPLUG-Owl-2 and LLaVA-1.5-13B, which have the highest Accp and Accq among opensource models, respectively. Additionally, we report human results for reference. As shown in Table 4, MLLMs score significantly lower than human, which means they are incapable to change their answers according to context they received. Their capability of context information extraction is needed to be improved. Examples of model responses when removing context are shown in Appendix C.
Comparison between ~7B and ~13B models. We observe from Table 3 that scaling up from ~7B to ~13B does not necessarily lead to an improvement in model performance. For LLaVA-1.5, its ~13B model significantly outperforms ~7B model on both Accp and Accq. However, for BLIP series models, Accp of InstructBLIP and Accq of BLIP-2 slightly drop when they scale up. Regrettably, significant differences exist between the two models, covering various aspects including data, training paradigms, and additional details. Consequently, we were unable to determine the primary causes of the differing behaviors observed in the two models. While intensive efforts are warranted to explore these causes, we believe that such an investigation exceeds the scope of this work.
Model performance on different categories. Among the five categories, most MLLMs perform best on relationships. For open-source MLLMs, it is possibly because their training data include dense image annotations (Liu et al., 2023b,a; Ye et al., 2023) such as Visual Genome (Krishna et al., 2017), or detailed captions (Li et al., 2023d; Dai et al., 2023; Zhu et al., 2023) which contain lots of data of relationships. We also observe that the score of MLLMs on cultural background is relatively low, indicating they are not sufficiently trained on culture-related data.
5 Analyses
5.1 Visual Information Extraction
Extracting information from images according to context is the foundation of context-dependent visual comprehension task. We explore capability of MLLMs on extracting visual information. We replace images with detailed captions and observe the changes in Accp. For some of the pictures
are not easy to be described in natural language, we discard these images and only perform experiments on a subset of 96 images, 302 queries. We ensure that captions and context are sufficient for answering the questions. We conduct our experiment on GPT-4V, Gemini, mPLUG-Owl-2 and LLaVA-1.5-13B. The model outputs are assessed manually and results are shown in Table 5. For all the MLLMs, Accp increases when the images are replaced with captions, indicating that compared to images, MLLMs are more skilled at extracting information from text with the help of context. Model capability to extract visual information according to context is needed to be improved. Examples of model responses when replacing images with captions are shown in Appendix D.
5.2 Bias in Model Outputs
Bias significantly influences model performance. We explore biases in model outputs on three dimensions: location & orientation, timing and culture.
We evaluate bias by calculating frequencies of three pairs of specific words, up and down, morning and afternoon, and appropriate and not appropriate, which are associated with the three dimensions, respectively. We evaluated the outputs of GPT-4V, Gemini, mPLUG-Owl-2, and LLaVA-1.5-13B, on a set of 88 queries whose groundtruth answers include these words. As shown in Table 6, although frequencies of the two words in each pair are equal in groundtruth answers, all the MLLMs demonstrate biases in their responses. This indicates that models tend to rely on their biases rather than the provided context in a certain number of cases, contributing to their poor performance on CODIS benchmark, especially Accp scores. Reducing bias is necessary for improving model capability on context-dependent visual comprehension. It also suggests that our design of data pairs and evaluation metric of Accp effectively prevent models from scoring high based on their biases rather than a true understanding of contexts.
5.3 Comparing Human and GPT-4 Evaluators We explore the possibility of utilizing GPT-4 as automatic evaluator. We compare the results of human and GPT-4 evaluation, as Table 7 shows. The first two columns show Accp of models based on human and GPT-4 evaluation, and the third column shows agreement between human and GPT-4, which is calculated as follows:
Agreement = 1
nq
nq∑
i=1
(Evalh(Oi) = Evalg(Oi)),
where nq is the total number of queries, Evalh(Oi) and Evalg(Oi) are evaluation results of human and GPT-4 on model output Oi. From the table we can see that GPT-4 is consistent with human on evaluation of more than 90% of queries. Overall, there is a high degree of consistency between human and GPT-4 evaluators in assessing model performance. GPT-4 is capable of conducting evaluation on CODIS benchmark instead of human. We report detailed results of GPT-4 evaluation in Appendix E.
6 Conclusion
In this work, we introduce a new benchmark CODIS to evaluate capability of models to use context to enhance visual comprehension. Results suggest that MLLMs significantly fall behind human in context-dependent visual comprehension. Further analysis suggests that shortcomings in visual
information extraction and bias in model outputs also account for low scores of MLLMs on CODIS. In summary, we pave a path for assessing capability of MLLMs on utilizing contextual information, which is a vital ability in real-world scenarios.
Limitations
There are still some limitations that need to be addressed. First, contexts in this work are brief due to the limited ability of current MLLMs in effectively utilizing contextual information. We believe it is essential to assess capability of extracting information from lengthy and intricate context to enhance visual comprehension in the future, as MLLMs evolve and become more powerful. Second, despite covering a wide range of scenarios and five representative types of context, there are still other forms of contexts that we intend to explore in future endeavors. Moreover, delving into context from other modalities such as audio and depth presents an interesting avenue for exploration. Moving forward, we aim to establish an automated pipeline for generating queries with context information spanning a broader range of scenarios and modalities.
Acknowledgements
This work is supported by the National Key R&D Program of China (2022ZD0160502) and the National Natural Science Foundation of China (No. 61925601, 62276152). We thank Sijie Cheng, Zhicheng Guo for their advice in paper writing. We thank Zonghan Yang, Yile Wang and Yuanchi Zhang for their valuable discussions.
References Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar-
garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425–2433.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. 2023. OpenFlamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.
Christopher Baldassano, Andre Esteva, Li Fei-Fei, and Diane M Beck. 2016. Two distinct scene-processing networks connecting vision and memory. Eneuro, 3(5).
Moshe Bar and Elissa Aminoff. 2003. Cortical analysis of visual context. Neuron, 38(2):347–358.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023a. ShareGPT4V: Improving large multimodal models with better captions. arXiv preprint arXiv:2311.12793.
Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. 2023b. DiffusionDet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19830– 19843.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft COCO Captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325.
Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. 2023c. See, think, confirm: Interactive prompting between vision and language models for knowledge-based visual reasoning. arXiv preprint arXiv:2301.05226.
Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. 2023. RelTR: Relation transformer for scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. 2023. Holistic analysis of hallucination in GPT-4V(ision): Bias and interference challenges. arXiv preprint arXiv:2311.03287.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards general-purpose visionlanguage odels with instruction tuning. arXiv preprint arXiv:2305.06500.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 326–335.
Rachel N Denison, William T Adler, Marisa Carrasco, and Wei Ji Ma. 2018. Humans incorporate attentiondependent uncertainty into perceptual decisions and confidence. Proceedings of the National Academy of Sciences, 115(43):11090–11095.
Tausif Diwan, G Anirudh, and Jitendra V Tembhurne. 2023. Object detection using YOLO: Challenges, architectural successors, datasets and applications. Multimedia Tools and Applications, 82(6):9243– 9275.
Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang, Chongyang Tao, Dongyan Zhao, and Qingwei Lin. 2022. MMDialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. arXiv preprint arXiv:2211.05719.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023a. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394.
Chaoyou Fu, Renrui Zhang, Haojia Lin, Zihan Wang, Timin Gao, Yongdong Luo, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, et al. 2023b. A challenger to GPT-4V? early explorations of Gemini in visual expertise. arXiv preprint arXiv:2312.12436.
G Gemini Team, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
Joshua OS Goh, Soon Chun Siong, Denise Park, Angela Gutchess, Andy Hebrank, and Michael WL Chee. 2004. Cortical areas involved in object, background, and object-background processing revealed with functional magnetic resonance adaptation. Journal of Neuroscience, 24(45):10223–10228.
Tianrui Guan, Fuxiao Liu, Xiyang Wu Ruiqi Xian Zongxia Li, Xiaoyu Liu Xijun Wang, Lichang Chen Furong Huang Yaser Yacoob, and Dinesh Manocha Tianyi Zhou. 2023. HallusionBench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models. arXiv e-prints, pages arXiv–2310.
Tanmay Gupta and Aniruddha Kembhavi. 2023. Visual Programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953–14962.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32–73.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. 2023a. MIMIC-IT: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023b. SEED-Bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125.
Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat-Seng Chua,
Siliang Tang, and Yueting Zhuang. 2023c. Finetuning multimodal llms to follow zeroshot demonstrative instructions. arXiv preprint arXiv:2308.04152, 3.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023d. BLIP-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. arXiv preprint arXiv:2304.08485.
Yang Liu, Guanbin Li, and Liang Lin. 2023c. Crossmodal causal relational reasoning for event-level visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023d. MMBench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281.
Jianjie Luo, Yehao Li, Yingwei Pan, Ting Yao, Jianlin Feng, Hongyang Chao, and Tao Mei. 2023. Semantic-conditional diffusion networks for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23359–23368.
OpenAI. 2023. GPT-4V(ision) System Card.
Senthil Purushwalkam, Sebastia Vicenc Amengual Gari, Vamsi Krishna Ithapu, Carl Schissler, Philip Robinson, Abhinav Gupta, and Kristen Grauman. 2021. Audio-visual floorplan reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1183–1192.
Sai Rajeswar, Pau Rodriguez, Soumye Singhal, David Vazquez, and Aaron Courville. 2022. Multi-label iterated learning for image classification with label ambiguity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4783–4793.
Kate Sanders, Reno Kriz, Anqi Liu, and Benjamin Van Durme. 2022. Ambiguous images with human judgments for robust visual event classification. Advances in Neural Information Processing Systems, 35:2637–2650.
Andranik Sargsyan, Shant Navasardyan, Xingqian Xu, and Humphrey Shi. 2023. MI-GAN: A simple baseline for image inpainting on mobile devices. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7335–7345.
Lars Schmarje, Vasco Grossmann, Claudius Zelenka, Sabine Dippel, Rainer Kiko, Mariusz Oszust, Matti Pastell, Jenny Stracke, Anna Valros, Nina Volkmann, et al. 2022. Is one annotation enough?-a data-centric image classification benchmark for noisy and ambiguous label estimation. Advances in Neural Information Processing Systems, 35:33215–33232.
Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14974– 14983.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. 2023. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.13286.
Melissa Le-Hoa Vo. 2021. The meaning and structure of scenes. Vision Research, 181:10–20.
Ning Wang, Jiahao Xie, Jihao Wu, Mingbo Jia, and Linlin Li. 2023. Controllable image captioning via prompting. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2617–2625.
Xuan Wang and Zhigang Zhu. 2023. Context understanding in computer vision: A survey. Computer Vision and Image Understanding, 229:103646.
Hanyu Xiang, Qin Zou, Muhammad Ali Nawaz, Xianfeng Huang, Fan Zhang, and Hongkai Yu. 2023. Deep learning for image inpainting: A survey. Pattern Recognition, 134:109046.
Jinsheng Xiao, Haowen Guo, Jian Zhou, Tao Zhao, Qiuze Yu, Yunhua Chen, and Zhongyuan Wang. 2023. Tiny object detection with context enhancement and feature purification. Expert Systems with Applications, 211:118665.
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. 2018. Graph r-cnn for scene graph generation. In Proceedings of the European Conference on Computer Vision (ECCV).
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023. mPLUG-Owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6720–6731.
Xiaobo Zhang, Donghai Zhai, Tianrui Li, Yuxin Zhou, and Yang Lin. 2023. Image inpainting based on deep learning: A review. Information Fusion, 90:74–94.
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. 2023. MMICL: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915.
Chaofan Zheng, Xinyu Lyu, Lianli Gao, Bo Dai, and Jingkuan Song. 2023. Prototype-based embedding network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22783–22792.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.
Guangming Zhu, Liang Zhang, Youliang Jiang, Yixuan Dang, Haoran Hou, Peiyi Shen, Mingtao Feng, Xia Zhao, Qiguang Miao, Syed Afaq Ali Shah, et al. 2022. Scene graph generation: A comprehensive survey. arXiv preprint arXiv:2201.00443.
Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. 2023. Object detection in 20 years: A survey. Proceedings of the IEEE.
A Consistency of Human Evaluation
A total of four individuals participated in human evaluation of model outputs. We examine the consistency of their assessments. A random sample of 100 queries out of 706 was selected for evaluation. Each of the four individuals made judgments on the model outputs of these 100 queries individually. A groundtruth value of 1 was assigned when the majority agreed the answer was correct, and a value of 0 was assigned when the majority agreed the answer was incorrect. In cases of a tie, a fifth individual was introduced. The average consistency between human judgments and groundtruths was then calculated as follows:
Consistency = 1
4nq
nq∑
i=1
4∑
j=1
(Evalj(Oi) = GTi),
where nq represents the total number of queries, Evalj(Oi) denotes the evaluation of individual j on model output of query i and GTi indicates the groundtruth assessment of model output of query i. The results are shown in Table 8.
The results indicate that the consistency of human evaluation is high. As a result, to enhance efficiency in the remaining manual evaluation tasks in this study, we distributed tasks evenly among the four individuals instead of employing the majority vote method. Moreover, agreement between human and GPT-4 evaluators , as portrayed in Table 7, is close to human consistency, indicating proficiency of GPT-4 on evaluating model outputs is comparable to that of humans.
B Human Scores on CODIS
We enlisted participation of five volunteers to respond to the questions in CODIS. Their average
age was 31 years old and they possess proficient daily English communication skills. As they are volunteers, they are not paid for the test. Before the test, all the volunteers were completely unaware of the questions and answers.
As shown in Table 9, humans perform well in CODIS benchmark, and achieve significantly higher scores than MLLMs. However, they do not reach 100% scores. Upon examining the results, we identify two primary reasons for this. Firstly, some questions require a lengthy chain of reasoning, which poses challenges for humans as well. Secondly, humans sometimes lack specific background knowledge, particularly in questions related to cultural backgrounds.
C Cases of Model Outputs When Removing Context
We provide three cases in Figure 4 to show alterations in model outputs resulting from the removal of context information. We use color green and
red to emphasize model outputs in agreement and disagreement with the groundtruth answers, and use yellow to highlight the words associated with context information in model outputs.
D Cases of Model Outputs When Replacing Images with Captions
We provide three cases in Figure 5 to show alterations in model outputs when replacing images with captions. We employ the identical highlighting markers as those utilized in Appendix C.
E Detailed Results of GPT-4 Evaluation
We report detailed results based on GPT-4 evaluation in Table 10. We report Accp and Accq. The results exhibit a strong alignment with the results in Table 3 based on human evaluation.
F Prompt for Model Inference
As shown in Table 11, we design four prompts for model inference, and investigate impacts of detailed instructions and chain of thought prompting on model performance.
• Detailed Instructions (DI): We stated that contexts contain helpful information that is not included in the image and encourage models to refer to contexts.
• Chain of Thought Prompting (CoT): We asked models to think of the questions step by step, and then summarize the reasoning processes and answer the question in single words or phrases.
We randomly select 200 pairs, 400 queries from our benchmark and report Accp scores of GPT4V, Gemini, mPLUG-Owl-2 and LLaVA-1.5-13B with the prompts above. The results are shown in Table 13.
The results indicated that DI and CoT help improve model performances. There were no significant differences in performance observed for LLaVA-1.5-13B when using either of the prompts, while the other models exhibit their best performances with DI and CoT. Therefore, based on the results, we decided to use the prompt with both DI and CoT for all the models in our evaluation process.
Additionally, it is worth noting that there are subtle variations in input formats and settings of different models, including placement of images,
use of special tokens, etc. Throughout our evaluation, we adhere to the default configurations of each model. Details regarding input formats and settings of our evaluated MLLMs can be found in their documentations or repositories.
G Prompt for GPT-4 Evaluation
We report prompt for GPT-4 evaluation in Table 12. The results of GPT-4 evaluation in Table 7 and results in Table 10 are both based on the prompt.
H More Cases of CODIS
We present additional cases of CODIS, showcasing outputs of MLLMs on these cases in Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10. We select two cases from each of the five categories. The incorrect model responses are highlighted in red. To aid readers in better comprehension, we provide explanations regarding the ambiguities in each image. It is important to note that these explanations are not visible to MLLMs or human volunteers who answered the questions in CODIS.
Prompt for Model Inference
w/ DI w/ CoT I’ll give you an image and some additional context, which provides information closely related to the scene of the picture. Please answer my question based on the image and the context. Be sure to refer to the context and extract necessary information from it to help you answer the question because it contains helpful information that is not included in the image. Your answer should contain two parts. Two parts should be separated by a newline. In the first part, please think of the question step by step based on the image and context and output your reasoning process. In the second part, please summarize your reasoning process and directly answer the question in a single word or phrase. Context: [CONTEXT HERE] Question: [QUESTION HERE]
w/ DI w/o CoT I’ll give you an image and some additional context, which provides information closely related to the scene of the picture. Please answer my question based on the image and the context. Be sure to refer to the context and extract necessary information from it to help you answer the question because it contains helpful information that is not included in the image. Please answer in a single word or phrase. Context: [CONTEXT HERE] Question: [QUESTION HERE]
w/o DI w/ CoT I’ll give you an image and some additional context, which provides information closely related to the scene of the picture. Please answer my question based on the image and the context. Your answer should contain two parts. Two parts should be seperated by a newline. In the first part, please think of the question step by step based on the image and context and output your reasoning process. In the second part, please summarize your reasoning process and directly answer the question in a single word or phrase. Context: [CONTEXT HERE] Question: [QUESTION HERE]
w/o DI w/o CoT I’ll give you an image and some additional context, which provides information closely related to the scene of the picture. Please answer my question based on the image and the context. Please answer in a single word or phrase. Context: [CONTEXT HERE] Question: [QUESTION HERE]
Please evaluate the output of models based on the given question and groundtruth and tell me whether the output is right. Please pay attention to the following rules: 1. The output contains rationale of the reasoning process and answer which is summarized from the reasoning process. Please extract the answer from the output and make your judgement only based on answer, NOT rationale. 2. The answer is right if it follows the question in meaning and is consistent with the groundtruth. 3. Do not be too strict about the answer. Format different from the groundtruth and minor grammar issues are allowed. If you think the answer is correct according to the groundtruth, please output "right", otherwise output "wrong". You can only print "right" or "wrong" and nothing else. Here is the question: [QUESTION HERE] Here is the groundtruth: [GROUNDTRUTH HERE] Here is the output: [OUTPUT HERE]
Table 12: Prompt for GPT-4 evaluation.

heading: Future Work
text: We believe it is essential to assess capability of extracting information from lengthy and intricate context to enhance visual comprehension in the future, as MLLMs evolve and become more powerful. Second, despite covering a wide range of scenarios and five representative types of context, there are still other forms of contexts that we intend to explore in future endeavors. Moreover, delving into context from other modalities such as audio and depth presents an interesting avenue for exploration. Moving forward, we aim to establish an automated pipeline for generating queries with context information spanning a broader range of scenarios and modalities