Can Large Multimodal Models Uncover Deep Semantics Behind Images? (2024)

Abstract: Understanding the deep semantics of images is essential in the era dominated by social media. However, current research works primarily on the superficial description of images, revealing a notable deficiency in the systematic investigation of the inherent deep semantics. In this work, we introduce DEEPEVAL, a comprehensive benchmark to assess Large Multimodal Models’ (LMMs) capacities of visual deep semantics. DEEPEVAL includes humanannotated dataset and three progressive subtasks: fine-grained description selection, indepth title matching, and deep semantics understanding. Utilizing DEEPEVAL, we evaluate 9 open-source LMMs and GPT-4V(ision). Our evaluation demonstrates a substantial gap between the deep semantic comprehension capabilities of existing LMMs and humans. For example, GPT-4V is 30% behind humans in understanding deep semantics, even though it achieves human-comparable performance in image description. Further analysis reveals that LMM performance on DEEPEVAL varies according to the specific facets of deep semantics explored, indicating the fundamental challenges remaining in developing LMMs.1

heading: 1 Introduction
text: The image is more than an idea. It is a vortex or cluster of fused ideas and is endowed with energy.
— Ezra Pound (1915)
Deep semantics of an image refer to the underlying meanings that extend beyond the superficial interpretation, probing into the essence of the image (Barthes, 1968). Although not every image inherently carries profound semantics, the concept of deep semantics is widespread across various fields (Barthes, 1999; Deman, 2010; Barthes, 2000;
1The dataset and code for the experiments are available at: https://github.com/AnnaYang2020/DeepEval.
Somov, 2005, 2006). Understanding the deep semantics of images is a manifestation of high-level human intelligence, serving as an important means of exploration from perceptual intelligence to cognitive intelligence.
However, previous efforts in visual understanding mainly focus on surface-level aspects of images, such as object attributes (Wang et al., 2022) and relationship reasoning (Hudson and Manning, 2019). Earlier attempts on deep semantic are limited in scope, focusing solely on sarcasm or humor, (Cai et al., 2019a; Chauhan et al., 2022; Boccignone et al., 2017; Patro et al., 2021), and lack in systematic investigation of the inherent deep semantic.
To address the mentioned limitations and fill the current research gap, we introduce DEEPEVAL, a benchmark for understanding the deep semantics of cartoons across various categories, accompanied by a meticulously annotated dataset. Additionally, we introduce three tasks: Fine-grained Description Selection, In-depth Title Matching, and Deep Semantics Understanding, to comprehensively evaluate models’ capabilities in understanding deep semantics. Cartoons, often imbued with profound
1 1898
meanings by their creators, are an ideal subject for this study. The DEEPEVAL dataset comprises over 1,000 samples, each featuring a cartoon image and manually annotated components, including image description, title, and deep semantics. Moreover, we develop multiple-choice questions for quantitative assessment, tailored for each task.
We conduct evaluations on various open-source LMMs as well as the proprietary GPT-4V(ision). Our findings reveal a significant gap between the capabilities of AI and humans in understanding deep semantics. Models with a larger number of parameters generally demonstrate a better understanding of deep semantics. Moreover, we discover that incorporating a description significantly helps these models in grasping the underlying semantics of an image. Furthermore, We also explore the performance of different models across various categories of images. By undertaking DEEPEVAL, our goal is to promote research in model development, focusing on a deeper understanding of semantics in visual content.

heading: 2 Related Work
text: Large Multimodal Models Large language models (LLMs) have exhibited strong abilities in various natural language understanding and generation tasks (Touvron et al., 2023a,b; Ray, 2023). Drawing on LLMs’ scaling law, a series of Large Multimodal Models (LMMs) using LLMs as the backbone has emerged. These models (Tsimpoukelli et al., 2021; Li et al., 2023c; Liu et al., 2023b,a; Zhu et al., 2023; Wang et al., 2023; Ye et al., 2023) have aligned visual features with language models through additional layers or specialized modules. Additionally, several closedsource LMMs (Alayrac et al., 2022; Driess et al., 2023), especially GPT-4V (Yang et al., 2023b), show remarkable ability in managing complex multimodal inputs. These models have set new benchmarks in performance (Fu et al., 2023; Li et al., 2023a), increasingly becoming predominant in visual-language research. However, relevant studies suggest that LMMs still face limitations in comprehending deeper semantics (Liu et al., 2023c; Yang et al., 2023a).
Visual Deep Semantics Understanding Understanding the deep semantics of visual contents represents a critical cognitive ability in humans. For AI, this ability showcases its depth of understanding images (Wang et al., 2021; Kruk et al., 2023).
Present evaluations (Lin et al., 2014; Antol et al., 2015; Goyal et al., 2017; Gurari et al., 2018; Hudson and Manning, 2019; Wang et al., 2022; Xia et al., 2023) mainly concentrate on superficial aspect of understanding. Pioneering works in affective image classification (Yanulevskaya et al., 2008; Machajdik and Hanbury, 2010) have shown that LMMs are capable of attaining an understanding beyond mere surface content. Research in sarcasm (Das and Clark, 2018; Cai et al., 2019b; Lemmens et al., 2020; Abu Farha et al., 2022) and humor detection (Radev et al., 2016) only employs classification tasks. Further work (Desai et al., 2022) provides explanations for satirical content. The most relevant prior work (Hessel et al., 2022) selects humorous captions for images and provides explanations. However, it exclusively focuses on humor evaluation. In contrast, our work is pioneering in its comprehensive exploration of visual deep semantics across multiple categories, offering a more thorough assessment of the deep semantics within images. We provide a detailed comparison between our method and previous studies in Table 1, and the categories covered by our method are illustrated in Figure 2.

heading: 3 Dataset and Task Overview
text: The DEEPEVAL dataset includes 1,001 samples, each with an image and three manually annotated components: a description, a title, and deep semantics. The statistical information about the text is displayed in Table 2. To enable quantitative evaluation, we additionally craft multiple-choice questions to test the understanding of descriptions, titles, and deep semantics. Each segment is represented by 1,001 questions, where each question presents an image, a question text, and four potential answers. Only one answer is correct, while
2
Benchmark Task Semantics # Category Img Type avg. length size
the others serve as distractors. Figure 1 illustrates examples of the manually annotated components and the multiple-choice questions.
To explore the capabilities of LMMs in comprehending the deep semantics of image, we construct a comprehensive evaluation consisting of three main subtasks:
• Fine-grained Description Selection Task: Evaluating the ability of models to accurately identify the surface-level details of images.
• In-depth Title Matching Task: Assessing the capability of models to understand the overall signification of images.
• Deep Semantics Understanding Task: Evaluating the ability of models to understand the detailed deep semantic meanings of images.
Together, these subtasks offer a robust and multifaceted evaluation of LLMs, enabling a deeper understanding of their strengths and limitations in image understanding.

heading: 4 Dataset Construction
text: We collect DEEPEVAL dataset in a multi-step crowd-sourcing pipeline, including 1) image col-
lection, 2) data annotation, 3) option generation. With selected high-quality cartoon images, we ask crowd-source workers to write a description, a title and deep semantics of each image.

heading: 4.1 Image Collection
text: The image data in the DEEPEVAL dataset are obtained by web scraping from websites. A total of 1,001 images are collected from Pinterest2, Cartoon Movement3, and Google search. The gathered images span a diverse array of genres, encompassing satirical representations of current events, philosophical narratives, humorous and entertaining content, among others. After collection, a manual screening process is conducted to remove duplicates and unclear images.

heading: 4.2 Data Annotation
text: Deep semantics of images often requires extensive commonsense knowledge and advanced reasoning abilities. To obtain high-quality image descriptions, titles, and deep semantics, we primarily utilize manual annotation to collect gold-standard answers with rigorous quality controls.

heading: 4.2.1 Annotator Recruitment and Instruction
text: We post a job description on online forums to invite over 50 applicants with at least a Bachelor’s degree to participate in an online pre-annotating instruction and qualification test. Based on their preferences, we divide them into two groups: annotators and inspectors. After completing the preannotating instructions, we conduct a qualification test for quality control. In the end, we select 26 annotators and 18 inspectors.
2https://www.pinterest.com/ 3https://cartoonmovement.com/
3

heading: 4.2.2 Cross-checking Annotation
text: We divide the annotation process into 3 phases. In the first phase, annotators randomly select cartoon images from the dataset for annotation of image description, title and deep semantics. The image description and deep semantics should be over 80 characters, while the proposed title should be over 3 characters, or else they cannot be submitted. Subsequent to this phase, each image is transformed into a quadruple (image, description, title, deep semantics), marking the completion of the initial dataset construction.
In the second phase, inspectors will check the annotated images. When encountering low-quality annotations, they can drop them. Each image annotation is checked by two inspectors. If both inspectors drop the annotation, we drop the annotation and put the image back into the dataset for second annotation. If a comic image is rejected in two rounds of annotation, it indicates that the deep semantics conveyed by the image are unclear, so we will drop the image. During this stage, we also use Cohen’s kappa to quantify the agreement between annotators, obtaining an average score of 0.701 across all tasks, which indicates substantial agreement (Landis and Koch, 1977).
In the third phase, the authors further check all of the HITS from the second phase to ensure that the annotations meet our standards. Finally, we acquire 1,001 high-quality data entries, each represented as a quadruple (image, description, title, deep semantics).

heading: 4.3 Options Generation
text: After obtaining the image annotations, we use the annotated text as the correct option and construct three distractor options. Considering the high cost of constructing all distractor options using manual annotators, we utilize the power of CLIP model and ChatGPT model in this section.
For image descriptions, we employ ChatGPT model to generate sentences that retain their original sentence structure but alter the nouns, verbs, adverbs, or adjectives. This generates more intrusive options in fine-grained description selection task. Then, the authors manually check all the options to ensure that the multiple-choice questions maintain a certain level of difficulty while having a unique and correct answer. Detailed prompts and examples can be found in Appendix A.
For deep semantics of the image, we use the CLIP model to calculate the similarity between the image and other deep semantics texts. We aim to select texts with higher similarity scores as distractors to create more challenging options. However, due to the presence of images with similar themes in the dataset, which may share similar semantics and potentially cause confusion, we utilize the ChatGPT model to eliminate distractor texts that are too similar to the correct option. Subsequently, we select the top three terms with the highest similarity as distractor terms.
For image titles, we similarly utilize the CLIP model to determine the similarity between the image and other titles. However, since there can be
4
numerous titles with distinct meanings that might serve as the title for the same image, determining whether a title causes confusion becomes more challenging. Therefore, in this part, the authors manually filter out confusing distractor texts and select texts with high similarity scores as distractor options.

heading: 4.4 Subtask Composition
text: We divide the task of understanding the deep semantics of cartoon into three progressive subtasks: fine-grained description selection, in-depth title matching, and deep semantics understanding. Among them, the fine-grained description selection task requires multi-modal models to identify the surface-level details of the images. The in-depth title matching task requires models to comprehend the overall significance of the images and grasp their basic intentions. The deep semantics understanding task takes it a step further by demanding multi-modal models to acquire a comprehensive and detailed understanding of the thoughts, connotations, and information conveyed in the images. It can be observed that these three tasks gradually augment the comprehension of images, each task building upon the previous one to deepen the level of understanding. In these three tasks, each question consists of an image and a multiple-choice question with four options. The model is then required to select the option it believes best conveys the description, title, or deep semantics from the four options.

heading: 4.5 Dataset Quality
text: To ensure the quality of the dataset, the authors have checked all the data for descriptions, titles, deep semantic annotations, and the multiple-choice questions of the three tasks. This ensures that the content of descriptions, titles, and deep semantics annotations meet the standards and maintain high quality. For the multiple-choice questions, this confirms that they are challenging and contain standard answers. Furthermore, we employ annotators to evaluate the triplets of each image (description, title, deep semantics) and provide a score between 1 and 5. A score of 1 indicates complete inconsistency, a score of 5 indicates complete consistency, and each image is evaluated by three different annotators. Finally, our dataset obtained an average score of 4.74, indicating that our dataset is of high quality.

heading: 4.6 License and Copyright
text: In this dataset, we used original web links of comic images without infringing on their copyright. For images sourced from MathPile governed by licenses stricter than CC BY-NC-SA 4.0, MathPile adheres to the more restrictive licensing terms. Otherwise, it operates under the CC BY-NC-SA 4.0 license. This work is licensed under a CC BY-NC license. Our annotators participate in the annotation process voluntarily and receive fair compensation.

heading: 5 Experiments

heading: 5.1 Baselines
text: In consideration of the strong performance exhibited by LMMs in addressing image comprehension challenges, we evaluate the following LMMs: LLaVA-1.5 (Liu et al., 2023a), MiniGPT-4 (Zhu et al., 2023), mPLUG-Owl2 (Ye et al., 2023), CogVLM (Wang et al., 2023), Qwen-VL (Bai et al., 2023b), InstructBlip2 (Dai et al., 2023), Fuyu (Bavishi et al., 2023), GPT-4V (Yang et al., 2023b). A detailed introduction to these models can be found in the Appendix F.

heading: 5.2 Experiment Details
text: In evaluating performance for our tasks, accuracy serves as the primary metric. A model’s answer is considered correct when it aligns with the established standard answer. Accuracy is quantified by the ratio of the number of correct responses Nr, to the total number of question N , expressed as Nr/N . Our task prompts start with specifying description, title, or deep semantics, followed by multiple-choice options: A, B, C, and D. To minimize deviations in results caused by variations in the text descriptions within the prompt, we develop three distinct prompt formats, which are elaborately described in Appendix B. The parameters for each model used in the experiment, including possible settings for temperature and top-k, are comprehensively detailed in Appendix C. Furthermore, to assess human capabilities in these tasks, we randomly select 100 questions from the dataset for each task and have human evaluators answer them. This allows us to benchmark the performance of human participants against our models, offering a thorough comparison of both human and machine proficiency in these specific tasks.
5

heading: 5.3 Main Results
text: Fine-grained Description Selection Task The results of various LMMs in fine-grained description selection task are shown in Table 3. It can be observed that Qwen-VL-Chat, among the opensource models, exhibit the highest recognition capability for fine-grained surface description, with an accuracy of 78.82%. On the other hand, Fuyu demonstrates the weakest recognition ability for fine-grained surface-level information, with an accuracy of only 29.90%. The latest GPT-4V exhibits outstanding performance with an impressive accuracy of 96.53%. Nevertheless, these models still do not match the capabilities of humans, whose accuracy remains at a perfect 100%.
In-depth Title Matching Task The performance of the models in the in-depth title matching task is also presented in Table 3. Among the open-source models, mPlug-Owl2 performs the best with an accuracy of 47.75%, while Fuyu shows the weakest performance with an accuracy of only 26.54%. The closed-source model GPT-4V outperforms them all, achieving an accuracy of 55.01%. A notable observation across all models is that their performance in this task significantly trails behind their performance in the preceding fine-grained description selection task. This indicates that processing deep semantics is more challenging, despite the in-depth title matching task primarily addressing the broad essence rather than intricate details of deep semantics. Additionally, it’s evident that these models substantially fall short of human-level performance, which is marked at an impressive 94%.
Deep Semantics Understanding Task Among open-source models, CogVLM showcases the high-
est performance with an accuracy of 32.20%, while LLava-1.5-7B scores the lowest, achieving only 15.72% accuracy, shown in Table 3. Unsurprisingly, GPT-4V achieves better results with an accuracy of 63.14%. However, GPT-4V exhibits the largest variance among the evaluated models models in deep semantics understanding, indicating instability despite its overall superior performance. Furthermore, when comparing GPT-4V’s results across all tasks, there is notably higher variance in the deep semantics aspect, suggesting weaker performance compared to other tasks. Additionally, we note that the capabilities of these models are significantly weaker than human performance, which stands at 93%.
It can be observed that the accuracy of all evaluated models in deep semantics understanding is significantly lower than their performance in image description, and nearly all of them achieve lower accuracy in deep semantics understanding compared to the in-depth title matching task. This underscores that comprehending deep semantics of images presents a significant challenge for these models, and focusing on the finer details of deep semantics adds further complexity, aligning with our expectations. Interestingly, only GPT-4V demonstrates higher accuracy in the deep semantics task compared to the in-depth title matching task. This could suggest that GPT-4V’s stronger understanding of longer texts, coupled with the detailed information conveyed in deep semantics texts, aids the model in making more accurate judgments in deep semantics understanding.
6
Answer: A (A) Tell us not to feel that our parents have not given us enough, they have already given everything they have.
Touching HumorousInspiring
Answer: B (B) This picture shows that when we encounter a dilemma, instead of just staying put and thinking without taking action, we should actually do it and find ways to solve the problem and get out of it. Answer: C (C) The concept of time varies greatly among different creatures. What one creature thinks is a short period of time, another creature will think it is very long.
Answer: B (B) Parents always like to compare their children with others, never encourage them, not realizing that this greatly defeats their children's confidence.
Answer: B (B) Parents always like to compare their children with others, never encourage them, not realizing that this greatly defeats their children's confidence.
Answer: D (D) This image uses the hand about to drown to suggest the precarious state of the human spirit. Instead of reaching out and helping to solve the problem, society high-fives him and asks him to continue to be a strong, responsible person. This unhelpful behavior will only be the final straw that crushes the mental state.
Answer: C (C) As illustrated at the top of the frame, this cartoon depicts two people with different mindsets. One kind of person has everything, but is still dissatisfied, and always clings to what he doesn't have; while the other kind of person seems to have nothing, but can always find happiness and satisfaction. The former kind of people are materially poor, and the latter kind of people are spiritually rich. Answer: D (D) This picture is superficially about the injuries and recovery of athletes, but in reality, it can point to all competitions in society such as exams, employment, and so on ... [text omitted]... The most thought-provoking thing in the picture is that the injured man cannot answer the question of whether he has put in too much effort, because no one can predict the outcome of their efforts. Answer: A (A) The comic strip portrays a deeper meaning related to hierarchical blame, power dynamics, and the negative consequences of a blame culture within organizations or systems. We should promote open communication, empathy, and a shared sense of responsibility within organizations or systems, thereby fostering a healthier and more productive work environment. CogVLM MiniGPT-4
Ground Truth
Figure 4: Random samples of answers chosen by CogVLM and MiniGPT-4, along with the standard answers, covering three categories: Touching, Inspiring, and Humorous, with one sample from each category.

heading: 6 Analysis

heading: 6.1 How do models perform across various categories in image understanding?
text: By analyzing the model’s understanding capabilities in different categories, we can pinpoint strength or weakness of models in specific categories. The performance of different models across categories is illustrated in Figure 5, with three radar charts showcasing the model’s ability in interpreting image descriptions, titles, and deep semantics across different categories. The deep semantics graph reveals that different models exhibit their strengths in different categories. For instance, the mPlugOwl2 and CogVLM stand out in the Humorous and Inspiring categories, respectively. Furthermore, despite extensive prior research, Satirical category continues to challenge all models, with accuracy rates remaining below 30%. This underscores the Satirical category as a critical area for further research in understanding deep semantics within images.
The description selection task’s radar charts, resembling regular hexagons, indicate a more uniform comprehension of image descriptions across categories by the models. When evaluating titles, models show remarkable competency in both Humorous and Inspiring categories compared to others. However, regarding deep semantics, Inspiring consistently emerges as the top-performing category for four models, whereas a majority struggle with Humorous. This discrepancy may stem from the fact that Inspiring content can often be sum-
marized in few sentences. In contrast, Humorous content typically involves more intricate interpretations that are heavily reliant on cultural context, timing, and the subtleties of language and expression. To provide a more intuitive display, Figure 4 showcase samples from typical categories in the deep semantics understanding task for CogVLM, MiniGPT-4, and the standard answers, while additional samples for other categories are available in Figure 7 in the Appendix E.

heading: 6.2 Can image descriptions aid models’ understanding of deep semantics?
text: It is commonly believed that models need to first identify the content of image descriptions before further comprehending the deep semantics. Therefore, we are curious to explore whether inspiring the model by incorporating its surface image descriptions during the evaluation process would aid in the model’s understanding of deep semantics. This process is divided into two steps: 1) having the model to generate detailed descriptions of the images; 2) incorporating the detailed descriptions into the prompt of the deep semantics understanding task. Additionally, to more effectively demonstrate the impact of integrating image descriptions on the understanding of deep semantics, we also directly include annotated image description texts in the prompt. In this case, the first step is omitted, and the detailed descriptions included in the second step are the annotated descriptions.
The results in Table 4 show that seven out of the nine evaluated models improve their understand-
7
ing of deep semantics with model-generated image descriptions. These models had an average increase of 1.8 percentage points. Additionally, all nine models demonstrated better deep semantics understanding with annotated image descriptions, with an average increase of approximately 4.1 percentage points. Thus, incorporating the model’s descriptions of the surface content can inspire and enhance its deep semantics understanding capabilities.

heading: 6.3 How does model parameter size affect deep semantics understanding?
text: Due to the scaling law, the number of parameters generally has a positive impact on the model’s performance. In this context, we also discuss the relationship between model parameters size and deep semantics understanding. We examine two pairs of models, InstructBlip-13B vs. InstructBlip-7B and LLaVA-1.5-13B vs. LLaVA-1.5-7B, where each pair has consistent architecture and training processes, differing only in parameter size. Figure 6 provide a visual representation of the means
and variances of accuracy across three tasks for these four models. It is observable that the 13B models have higher accuracy across all three tasks compared to the 7B models, indicating superior performance of the 13B models. Furthermore, the overall variances of the 7B models is higher than that of the 13B models. This indicates that, generally speaking, the 13B models are also more stable than the 7B models. Therefore, an increase in the number of parameters has a positive impact on the models’ deep semantics understanding capabilities.

heading: 7 Conclusion
text: We propose DEEPEVAL, a benchmark for visual deep semantics of LMMs. DEEPEVAL consists of well-annotated dataset and three subtasks: finegrained description selection, in-deep title matching, and deep semantic understanding. Evaluations are conducted on the leading LMMs, revealing a significant gap between AI and human capabilities in understanding deep semantics. Further analysis indicates that integrating image descriptions during the inference process enhances LMMs’ ability to perceive deep semantics. The model’s ability to understand deep semantics also improves as the
8
number of parameters increases. Furthermore, our dataset is divided into multiple categories, and we conduct a more detailed analysis within these categories. Existing models still have a long way to go in terms of visual deep semantics understanding compared to humans. We hope that the proposed dataset and tasks can pave the way for AI to achieve a deeper understanding of the profound semantics conveyed by images.
Limitations
The deep semantics of cartoon images are varied, and due to our limited collection of images, it’s not feasible to encompass all potential deep semantic content. In this work, we have only exemplified some common categories, but the categories of images in the real world far exceed these six. On this note, adding more images and annotations would help improve this issue.
Furthermore, our current images only include cartoons. This is because, compared to real-world pictures, cartoons generally contain rich and clear deep meanings, which are beneficial for investigating deep semantics. Despite this, our dataset images encompass a wide array of image types and a wide range of themes, reflecting the multifaceted nature of real-world scenarios. Detailed statistics for image types and themes can be found in Appendices G and H. Our future work will expand to incorporate more types of images, such as photographs, advertising images, and artworks.
Lastly, in the annotation process, we aim to reach a consensus among annotators on the deep semantics of images and only retain images with agreedupon deep semantics. Therefore, images with deep semantics but significant controversy will not be included.

heading: Acknowledgements
text: This paper is supported by the National Key Research and Development Program of China (No.2020AAA0106700). The contact author is Zhifang Sui.

heading: A Examples of Generating Distractor Generation For Description
text: Considering that the generation of interference terms in the description only requires replacing nouns, adjectives, verbs, etc. in the sentence, we use Chatgpt to complete this task. The following is the prompt we use:Give me three different paragraphs that take only some of the verbs, nouns, adjectives, and adverbs in a given paragraph and modify words with irrelevant meanings. Input: [Example Input 1] Output: [Example Output 1] Input: [Example Input 2] Output: [Example Output 2] Input: [Example Input 3] Output: [Example Output 3] Input: [Input] Output:
To ensure that ChatGPT understands our requirements correctly, we use a 3-shot prompt. These three examples were manually written by the author. The following is a modification example. It should be noted that the output of each example in the prompt has 3 modified paragraphs of text. For convenience, only one modified paragraph of text is shown here
Source Text: In the picture,there are three queues,the first one named Critic has many people,stand in an endless line;the second one named Talker also has many people,but not that much as Critic;the third queue named Doer,with no one in line.
Revised Text: In the picture, there are three cats, the first one named Critic has many toys, play in an endless loop; the second one named Talker also
has many toys, but not that much as Critic; the third cat named Doer, with no toys to play with.

heading: B Prompt Details
text: To eliminate the influence of prompt expression on model performance, we used the following three types of prompts for testing:
• Choose the correct answer to the following question. Which following text is the [description/best title/deep meaning] of the image? Options: (A) [...] (B) [...] (C) [...] (D) [...] Answer:
• Select the appropriate [description/title/deep meaning] for the image from the options given. Which of these is the most suitable [description/title/deep meaning] for the image? Choices: A) [...] B) [...] C) [...] D) [...] Correct Answer:
• Identify the most suitable [description/title/deep meaning] for the image from the given options. Which of the following should be chosen as the [description/title/deep meaning]? Choices are: A. [...], B. [...], C. [...], and D. [...]. The correct answer is:

heading: C Model Hyper-parameter Details
text: We use the default hyper-parameter values of the models. In the LLaVa-1.5-7B and LLaVa-1.5-13B, the temperature is set to 0.2. For MiniGPT-4, the temperature is set to 1.0, and num_beams is also set to 1.0. The temperature for mPlug-Owl-2 is set to 0.7. For CogVLM, the temperature is set to 0.4, top_p is set to 0.8, and top_k is set to 1.0.

heading: D Categories Definition
text: Table 5 give the names and detailed definitions of the categories in DEEPEVAL.

heading: E Categories Samples
text: Figure 7 give the samples of answers chosen by CogVLM and MiniGPT-4, in three Satirical, Critical, and Philosophical category.

heading: F Large Multimodal Models
text: • LLaVA-1.5 (Liu et al., 2023a) is an endto-end LMM extended from Vicuna(Chiang et al., 2023), augmented with vision encoder.
12
Table 5: The names and specific definitions of the categories in DEEPEVAL.
Category Definition
Humorous The image elicits amusement, laughter, or a sense of light-heartedness. It may contain elements that are funny, witty, or clever.
Critical The image offers a critical perspective or analysis of a specific subject, aiming to examine and evaluate its merits, shortcomings, or implications.
Touching The image evokes strong emotions such as joy, sadness, empathy, or nostalgia. It may depict a heartwarming scene, a tender moment, or a poignant event.
Philosophical The image stimulates intellectual or philosophical contemplation. It raises questions, challenges assumptions, or encourages viewers to reflect on deeper meanings or concepts.
Inspiring The image motivates or uplifts viewers, conveying a positive message, encouraging resilience, or instilling hope. It may depict acts of kindness, achievement, or triumph over adversity.
Satirical The image conveys a message or commentary on a particular subject, often by using irony, sarcasm, or wit to highlight flaws or satirize societal norms, institutions, or individuals.
• MiniGPT-4 (Zhu et al., 2023) is an extension of Vicuna, incorporating ViT (Dosovitskiy et al., 2021) and Q-former (Li et al., 2023b) as the vision encoder, while also featuring a single linear projection layer sandwiched between them.
• mPLUG-Owl2 (Ye et al., 2023) is an extension of LLaMA-2-7B(Touvron et al., 2023b), using ViT-L/14(Radford et al., 2021) as the vision encoder, and introducing a visual abstractor between them.
• CogVLM (Wang et al., 2023) is also an extension of Vicuna, incorporating ViT(Dosovitskiy et al., 2021) as the vision encoder, a two-layer MLP(Shazeer, 2020) as adapter, and introducing Visual expert module.
• Qwen-VL (Bai et al., 2023b) is an extension of Qwen-7B(Bai et al., 2023a), incorporating ViT(Dosovitskiy et al., 2021) as the vision encoder, and introducing a vision-language adapter that compresses the image features.
• InstructBlip2 (Dai et al., 2023) employs ViTg/14 (Fang et al., 2022) as image encoder, and four different LLMs as language decoders. In our following tests, we utilize vicuna-13B and vicuna-7B (Chiang et al., 2023) versions.
• Fuyu (Bavishi et al., 2023) employs a decoderonly architecture, devoid of a dedicated image encoder for image processing. This design
choice enables the model to support arbitrary image resolutions.
• GPT-4V (Yang et al., 2023b) is OpenAI’s cutting-edge language model redefining natural language processing with advanced contextual understanding and versatile linguistic abilities.

heading: G Wide Range of Themes
text: DEEPEVAL dataset boasts a wide range of themes, including Social Justice and Activism, Human Emotions and Relationships, Philosophy and Life Approach, Technology and Digital Dependency, Environmental Awareness, and Mental Health and Well-being. These diverse themes mirror the multifaceted nature of real-world scenarios. The corresponding statistics are detailed in Table 8.

heading: H Wide Array of Image Types
text: The term "cartoon" within DEEPEVAL dataset encompasses a wide array of image types, including Poster, Inspirational Art, Comic Strips, Manga, Caricature, Editorial Cartoon, and Environmental Cartoon. We have detailed statistics for these image types in Table 9.
I Influence of OCR Ability on Models’ Deep Semantics Ability
To study the potential impact of textual information within images on the models’ understanding of deep semantics, we randomly selected 200 images and manually divided them into two groups: images without textual information (74 images) and
13
images with textual information (126 images). We then calculated the mean accuracy and variance of each group across three distinct prompts to evaluate their deep semantic understanding capabilities. The results are shown in Table 6.
Our findings reveal that models demonstrate a higher ability to understand the deep semantics of images containing textual information compared to those without. This indicates a positive influence of OCR capabilities on the deep semantic understanding of images.

heading: J Generative Capabilities of Models
text: In addition to using multiple-choice questions to assess the model’s capabilities, we further incorporate an additional assessment aimed at evaluating the generative performance on models with superior generative capabilities. To quantify the results, we use GPT-4 to determine the consistency between generated sentences and labeled sentences. Samples judged as consistent are considered correct, while those judged as inconsistent are considered incorrect. We then calculate the final accuracy.
The results, detailed in Table 7, reveal a close alignment between the outcomes of our main experiments and the generative capability assessments,
14
thereby reinforcing our conclusions about the models’ deep semantic understanding. Specifically, the Pearson coefficients between the results of main experiments and generative capability assessments are 0.95, 0.92, and 0.96 for the Description, Title, and DeepSemantics tasks, respectively.
15

heading: Future Work
text: Our future work will expand to incorporate more types of images, such as photographs, advertising images, and artworks. Lastly, in the annotation process, we aim to reach a consensus among annotators on the deep semantics of images and only retain images with agreedupon deep semantics. Therefore, images with deep semantics but significant controversy will not be included