Towards Artwork Explanation in Large-scale Vision Language Models (2024)

Abstract: Large-scale Vision-Language Models (LVLMs) output text from images and instructions, demonstrating advanced capabilities in text generation and comprehension. However, it has not been clarified to what extent LVLMs understand the knowledge necessary for explaining images, the complex relationships between various pieces of knowledge, and how they integrate these understandings into their explanations. To address this issue, we propose a new task: the artwork explanation generation task, along with its evaluation dataset and metric for quantitatively assessing the understanding and utilization of knowledge about artworks. This task is apt for image description based on the premise that LVLMs are expected to have pre-existing knowledge of artworks, which are often subjects of wide recognition and documented information. It consists of two parts: generating explanations from both images and titles of artworks, and generating explanations using only images, thus evaluating the LVLMs’ language-based and vision-based knowledge. Alongside, we release a training dataset for LVLMs to learn explanations that incorporate knowledge about artworks. Our findings indicate that LVLMs not only struggle with integrating language and visual information but also exhibit a more pronounced limitation in acquiring knowledge from images alone 1.

heading: 1 Introduction
text: In the field of Vision & Language (V&L), Large Language Models (LLMs) (Touvron et al., 2023; Chiang et al., 2023; Bai et al., 2023a; Jiang et al., 2023) have been combined with visual encoders to create Large Scale Vision Language Models (LVLMs) (Li et al., 2023b; Liu et al., 2024; Bai et al., 2023b; Ye et al., 2023b). These models have achieved success in various V&L benchmarks (Li
1The datasets (ExpArt=Explain Artworks) are available at https://huggingface.co/datasets/naist-nlp/ExpArt
et al., 2023a; Fu et al., 2023; Liu et al., 2023c; Bai et al., 2023c). Despite these advancements, tasks like Visual Question Answering (VQA) (Zhang et al., 2022b; Yue et al., 2023), Image Captioning(Agrawal et al., 2019; Lin et al., 2014), and querying models about artwork-related information (Garcia et al., 2020; Cetinic, 2021; Bai et al., 2021) have primarily focused on assessing models’ abilities to handle isolated pieces of knowledge.
These tasks, while valuable, do not fully capture the complexity of synthesizing and explaining interconnected knowledge in real-world scenarios (Kawaharazuka et al., 2024), nor the difficulty of generating coherent text to explain this knowledge. Current evaluations often result in superficial image descriptions, lacking extensive background knowledge and interrelationships between subjects.
A pertinent example of this limitation can be observed in the context of creative support for paintings and photographs. As shown in Figure 1, these models must produce explanations that integrate knowledge of the artwork’s theme, historical context, associated works, and artistic movement, highlighting a gap in current capabilities. Since this task goes beyond simply recognizing disparate knowledge, it is crucial for LVLMs to deeply understand
705
the interrelationships of artwork knowledge to integrate them into explanations comprehensively.
To address this gap, we propose a new task and evaluation metrics designed to measure LVLMs’ capability in generating comprehensive explanations about artworks. Our task requires LVLMs to generate explanations in response to given instructions, based on input images and titles of artworks.
We have constructed a dataset from about 10,000 English Wikipedia articles of artworks for this task and also release a training dataset to facilitate LVLMs in learning to generate explanations involving artistic knowledge. Furthermore, we have evaluated LVLMs currently achieving the highest performance in various V&L benchmarks. The results show that while the LVLMs retain the artistic knowledge inherited from their base LLMs, they do not adequately correlate this knowledge with the provided visual information.

heading: 2 LVLMs
text: LVLMs (Li et al., 2023b; Liu et al., 2024; Bai et al., 2023b; Ye et al., 2023b) integrate a Vision Encoder (Li et al., 2023b) trained through contrastive learning to process visual information with Large Language Models (LLMs) (Li et al., 2023b; Liu et al., 2024; Bai et al., 2023b; Ye et al., 2023b). This integration requires further training to effectively combine vision and language capabilities. As a result, these LVLMs outperform conventional pretrained models, even those with over ten times more parameters (et al, 2022; Driess et al., 2023).
However, it is unclear whether the knowledge from the LLM and the Vision Encoder are appropriately aligned by the additional network layers in LVLMs (Chen et al., 2024a). Generating explanations that involve knowledge about art especially requires careful and systematic alignment and utilization of the information from both the Vision
Encoder and the LLM. This challenge motivates us to design a new task for LVLMs.

heading: 3 Task and Evaluation Metrics

heading: 3.1 Task
text: Our task demands LVLMs to generate explanations following instructions with images and titles. Examples of the instructions are shown in Table 1. As demonstrated by these examples, each instruction is categorized into three levels, Section, Subsection, and Subsubsection, determined by the corresponding positions in Wikipedia articles (See §3). The proposed task addresses the following two settings with or without titles:
With Title In the context of creative assistance, the title often contains the author’s intent for the artwork, and it is desirable to generate explanations considering this intent. In this setting, both the image and its title are inputs, testing whether LVLMs can generate appropriate explanations based on both language and visual information.
Without Title As shown in Figure 1, there are cases where a title does not exist potentially because the artwork is in the process of creation. This setting tests whether LVLMs can generate appropriate explanations using only visual information from images. Additionally, analyzing the performance changes with and without titles allows us to verify the LVLMs’ pure vision-based knowledge.
Furthermore, to thoroughly assess the generalization capabilities of LVLMs, we compare two cases: 1) a seen case in which images are observed during finetuning, and 2) an unseen case in which images are not observed during finetuning.

heading: 3.2 Evaluation Metrics
text: Since our task is a kind of natural language generation (NLG), we utilize popular metrics in NLG
for evaluation, i.e., BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and BERTScore (Zhang* et al., 2020). To further focus on the ability to generate explanations for artworks, we propose the following three evaluation metrics2:
Entity Coverage We evaluate how accurately the generated text includes entities (See §4) related to the artwork mentioned in the reference description, using two settings: exact match and partial match (Li et al., 2022a).
Entity F1 We evaluate the frequency of occurrence of entities related to the artwork found in the generated and reference explanations by F1. Inspired by ROUGE, we consider the highest frequency of occurrence of any entities within either the generated explanation or the reference as the upper limit of occurrence frequency to calculate precision and recall.
Entity Cooccurrence This metric assesses not only the coverage of independent entities but also how their interrelations are contextually combined
2For the formulas of each metric, see Appendix C.
to form the overall explanation. Specifically, it considers pairs of entities that co-occur within a sentence and its preceding and following n sentences, evaluating the coverage rate of these pairs to reveal how well the model understands and integrates the relevance of knowledge. By setting the value of n to exceed the number of sentences in the generated explanation, it becomes possible to account for the co-occurrence of entity pairs throughout the entire text. Furthermore, we apply the brevity penalty used in BLEU (Papineni et al., 2002) to verify the accuracy of knowledge at an appropriate length, defined by the reference text for each data instance. This ensures models produce concise, non-redundant explanations.

heading: 4 Dataset Creation
text: The process of dataset creation, illustrated in Figure 2, involved the following steps:
STEP 1: We collected all the artwork articles from the English Wikipedia that have an infobox (about 10,000), divided them into sections, and created descriptive texts. Additionally, hyperlinked texts within the articles were extracted as entities related to the artwork. Each descriptive text is accompanied by four pieces of information: the title, the hierarchy of sections (i.e., Section, Subsection, Subsubsection), the image, and the aforementioned entities.
STEP 2: We filtered out sections that did not contribute directly to the understanding of artwork, articles without images, and texts not specific to individual art pieces to ensure the relevance and quality of the content.
STEP 3: To prevent biases that may arise due to the notoriety of the artworks included in the LVLM’s training data, we shuffled the data. First, we ranked the data using six metrics: page views, number of links, number of edits, number of references, number of language versions, and article length. We then evenly split the data into test, development, and training sets at a ratio of 1:1:8 to maintain the average ranking across these sets (Table 2). As described in §3, for the Seen set, we used training images with no overlap in reference text to prevent leakage. For the Unseen set, neither images nor reference texts are from the training set.
STEP 4: The sorted data for each set were then formatted into instructions using the templates described in Section 3.1. To diversify the training data, we prepared seven different templates inspired by Longpre et al. (2023) (see Appendix E.3).

heading: 5 Evaluation

heading: 5.1 Setup
text: We evaluated four models: mPLUG-Owl2 (Ye et al., 2023b), LLaVA-NeXT (Liu et al., 2024), Qwen-VL-Chat (Bai et al., 2023b), and GPT-4 Vision (OpenAI, 2023), along with an instructiontuned version of Qwen-VL-Chat (FT), fine-tuned by our dataset with LoRA (Dettmers et al., 2022a).3
As shown in Table 2, the data is divided based on
3Further details for the evaluation setup and results for other models are described in Appendix D and Appendix A.
images. In the Few-shot setting, by utilizing this data division, to prevent answer leakage in Fewshot samples, for test (Seen) evaluations, samples were randomly selected from the test (Unseen) set, and vice versa for test (Unseen) evaluations.

heading: 5.2 Results
text: With and Without Title Table 3 shows the results In the "With Title" setting, GPT-4-Vision achieved the highest performance in Entity Coverage and Entity F1, with Qwen-VL-Chat (FT), Qwen-VL-Chat, and LLaVA-NeXT (Yi-34B-Chat) also showing strong performance. Notably, QwenVL-Chat (FT) reached the highest precision in Entity Cooccurrence, showcasing its exceptional ability to accurately contextualize knowledge within generated text. This proves the superiority of our instruction-tuning dataset. Additionally, considering the average reference token length is 174 in the unseen setting, the significantly low performance of LLaVA-Next (Yi-34B-Chat) indicates excessive token lengths may result in redundant text, which is unsuitable for generating concise explanations.
In the "Without Title" setting, Qwen-VL-Chat (FT) outperformed GPT-4-Vision across all metrics, indicating that our dataset enables accurate knowledge association and generation from visual information. Comparative analysis of the models’ performance in scenarios with and without titles indicated a consistent drop in performance across the board. This observation clearly shows the challenges of generating text based solely on visual inputs. All models, including advanced ones like GPT-4-Vision, heavily depend on text-based cues.
3Since LLMs do not handle visual information, we conducted the analysis in a setting with titles.
LLMs vs. LVLMs Table 5 shows the results of explanation generation in the With Title setting without images for text-only LLMs. Notably, Table 5 illustrates that GPT-4 (OpenAI et al., 2023) achieves the highest accuracy across all metrics, demonstrating strong knowledge about artworks, closely followed by Llama2 (Touvron et al., 2023), Vicuna (Chiang et al., 2023) and Yi-34-Chat (01.AI, 2023). Conversely, Qwen-Chat (Bai et al., 2023a) is shown to perform comparatively lower. Additionally, the comparison of Tables 3 and 5 reveals the extent of text-only LLM’s knowledge retention through integrated vision and language learning. It is apparent that the knowledge about artworks is compromised in other LVLMs due to the integrated learning of vision and language. On the other hand, Qwen-VL-Chat achieves a 10% performance boost in titled settings, signaling successful synthesis of vision and language knowledge.
Few-shot vs. Fine-tuning The results in Table 4 show that Fine-tuning outperforms both the
pure model and Few-shot settings. While Few-shot settings show some improvement with an increasing number of shots, they do not match the performance of Fine-tuning. Considering the average token length of 174 in the reference sentences, the reduced token length in Few-shot settings suggests a focus on generating necessary terms but may result in less comprehensive explanations. In contrast, Fine-tuning allows the model to learn both specific vocabulary and the format for generating coherent explanations, leading to better performance. However, the lack of significant differences between Seen and Unseen settings in Fine-tuning indicates that effective alignment of visual and textual information (the knowledge originally held by the LLM) requires simultaneous learning of images and their descriptions.

heading: 6 Conclusion
text: We introduced a new task, artwork explanation generation, and its dataset and metrics to quantitatively evaluate the artistic knowledge comprehension and application. Using LVLMs, we assessed their retention and utilization of artworks knowledge from base LLMs, with or without artwork titles. Our findings indicate that while LVLMs maintain much of the artistic knowledge from their LLM counterparts, they do slightly lose some in practice. Furthermore, the challenges in generating text solely based on visual inputs clearly show a significant dependency on text-based cues.

heading: Limitations
text: Our research elucidates the intricacies of integrating visual and language abilities within LVLMs, yet it encounters specific limitations that define the scope of our findings.
Data Source A principal limitation is our reliance on the diverse authorship and open editing model of Wikipedia as our data source. Variations in detail, writing style, and information density across entries may lead to inconsistencies in the dataset, potentially skewing model performance and affecting the universality of our conclusions. Additionally, we did not filter out generic entities such as "artwork" to avoid bias. However, more specific entity filtering may improve dataset relevance to artworks. Moreover, relying on Wikipedia limits our dataset to well-known artworks, omitting lesser-known but culturally significant works not featured on the platform, thereby missing a broader spectrum of artistic significance.
Human Evaluation While our current study does not include human evaluations, it is crucial to assess whether the models can provide insights beyond Wikipedia and evaluate LVLM explanations from an expert perspective for real-world applications. Another LVLM-based image explanation task, image review generation (Saito et al., 2024) actually conducts human evaluation by hiring nonexpert annotators. Unlike their work, our task requires expert knowledge to judge the quality of generated explanations. Thus, due to the cost perspective, evaluating generated explanations across various genres by experts is a left problem.
Integration of Vision and Language Representations Simultaneously, our study identifies a crucial limitation in the process of integrating Vision Encoders with LLMs, particularly highlighting the models’ reliance on textual cues to generate text from visual inputs. Kamigaito et al. (2023) report the same issue when predicting infoboxes, which are kinds of summaries for Wikipedia articles. This observation underscores the difficulty of retaining language knowledge during the integration, a problem we acknowledge without offering concrete solutions. This gap clearly shows the pressing need for future research to not only further investigate these issues but also to develop innovative methodologies that ensure the preservation of language knowledge amidst the integration of visual and language abilities.

heading: Insuffcient Artwork Knowledge in LVLMs
text: The limited improvement in entity coverage by LoRA indicates the difficulty of injecting artwork knoweldge into LVLMs. As a solution, we can consider injecting external knowledge into LVLMs. Chen et al. (2024b) introduce using knowledge graphs (KGs) as a solution. However, KGs are commonly sparse and we may need to complete them by KG completion (KGC), a task to complete missing links in KGs. Traditional KGC methods (Nickel et al., 2011; Bordes et al., 2013) are emperically (Ruffinelli et al., 2020; Ali et al., 2021) and theoretically (Kamigaito and Hayashi, 2021, 2022a,b; Feng et al., 2024) investigated in detail, and thus, these are solid whereas the pre-trainedbased KGC models can outperform them (Wang et al., 2022). On the other hand, Sakai et al. (2023) point out the leakage problem of the pre-trainedbased KGC models and the actual performance of them is uncertain. Retrieval-Augmented Generation (RAG) (Lewis et al., 2020) can be another solution if LVLMs can accept lengthy input (Zong et al., 2024).

heading: Ethical Considerations
text: In our study, we meticulously curated our dataset derived from English Wikipedia. During the data creation phase, we individually inspected each extracted image, carefully removing those clearly unsuitable for public disclosure, ensuring no inappropriate images were included. Additionally, while English Wikipedia’s editors actively eliminate unnecessarily offensive content to compile an encyclopedia, as outlined on their official pages regarding offensive material5, bias in sources, and the use of biased or opinionated sources6 7, it is acknowledged that English Wikipedia allows the inclusion of biased information sources. Consequently, our dataset might also reflect the inherent biases present in the original English Wikipedia content. Note that in this work, we used an AI assistant tool, ChatGPT, for coding support.

heading: Acknowledgments
text: This work was supported by JSPS KAKENHI Grant Numbers JP21K17801, JP23H03458.
5https://en.wikipedia.org/wiki/Wikipedia: Offensive_material
6https://en.wikipedia.org/wiki/Wikipedia: Neutral_point_of_view#Bias_in_sources
7https://en.wikipedia.org/wiki/Wikipedia: Reliable_sources#Biased_or_opinionated_sources

heading: A Supplemental Results

heading: A.1 Detailed Evaluation of LVLMs in ’Seen’ Data Settings
text: Table 8 presents the results of Language-Vision Learning Models (LVLMs) including ’seen’ settings, with bold type highlighting the highest score for each metric within each group. In this study, we assessed the generalizability of data and the precision of models fine-tuned on ’seen’ and ’unseen’ data during their training phase to ascertain if the fine-tuning process enhanced the models’ accuracy for images encountered during training. Despite the images being part of the training dataset, with sections meticulously segregated to prevent data leakage, our validation revealed no significant differences in accuracy between ’seen’ and ’unseen’ settings. This finding confirms the general applicability of the data and suggests that simply viewing images, without integrating them with relevant contextual knowledge, does not inherently contribute to accuracy improvement. This highlights the importance of a holistic learning approach where images are paired with pertinent information to truly boost the performance of the models.
Furthermore it is generally impractical to create datasets that combine images corresponding to the vast amounts of text data seen during the training of LLMs and to acquire these through additional integrated learning. Additionally, during the integrated learning process from LLM to LVLM, the focus is on learning pairs of individual images and their descriptions. To develop the ability to individually recognize knowledge objects and explain them based on that recognition, as well as to understand the relationships between objects and generate comprehensive explanations, it is considered necessary to use enhancement methods such as RAG and new integrated learning techniques for LVLMs.

heading: A.2 Extended Analysis of Additional LVLMs
text: In our research, we expanded our experimental investigation beyond the models outlined in the primary section to include Blip2 (Li et al., 2023b), mPLUG_Owl (Ye et al., 2023a), LLaVA-NeXT (Mistral) (Liu et al., 2024), LLaVA-1.5 (Liu et al., 2023a,b), InstructBlip (Dai et al., 2023), and Yi-6B (01.AI, 2023), integrating image and language in a manner similar to the initially described models. Utilizing the same experimental framework as the initial tests, we conducted an thorough assessment. The results, as outlined in Table 9, revealed that
these additional models did not exceed the accuracy levels of those featured in the main analysis (refer to Section 5). Additionally, a comparative examination of configurations with and without titles showed a uniform decline in efficacy, emphasizing the difficulty of deriving knowledge and translating it into explanatory text generation based purely on image data.

heading: A.3 Detailed Performance Metrics for Base LLMs with Title Context
text: Table 10 presents the results of an evaluation involving the base LLM models of the Language-Vision Learning Models (LVLMs) discussed in Tables 3 and 9. This evaluation additionally included tests on base models such as FLAN-T5-XL(Chung et al., 2022), FLAN-T5-XXL, OPT(Zhang et al., 2022a), LLaMA(Touvron et al., 2023) Mistral(Jiang et al., 2023), and Yi-6B, which were not featured in the main analysis. Since Language Models (LMs) are incapable of processing image information, the evaluation was confined to the ’With Title’ setting that incorporates textual information. Within this context, GPT-4 showcased superior performance across all tested configurations, with Mistral, Vicuna-13B, and LlaMA2 also demonstrating strong results.
Consistent with the data presented in Table 3, the base model for LLaVA-NeXT (Yi-34B) yielded output sequences with excessively token lengths compared to its counterparts, mirroring the behavior of its LVLM version. This tendency for producing longer output is illustrated when compared with other models (as depicted in Figure 3 ). Furthermore, when examining the accuracy of the LVLMs tested in Table 9 alongside the base models in relation to our task proposal, there is a discernible decline in precision across nearly all models. Qwen is the exception, which highlights the nuanced challenges in effectively merging image and textual data. This complexity stands as a pivotal challenge for the evolution of sophisticated LVLMs.

heading: B Title generation
text: In our task, the titles of artworks are a crucial element of knowledge related to the artworks. To maintain the integrity of the analysis between the settings with and without titles setting, we intentionally omitted titles from entity recognition. However, we recognized the need to understand the performance of models in generating titles of
artworks based solely on visual information. Therefore, We conducted an additional experiment in which we presented the models with the prompt "Please answer the title of this artwork" along with 963 images from the "Unseen" test set and evaluated the accuracy of title generation under two settings: Exact and Partial. Tables 11, 12 and 13 display the accuracy results of the main models and those from additional experiments, respectively.
The results showed that GPT-4-Vision achieved the highest performance with an exact match setting at 8.97%, followed by Qwen-VL-Chat (FT) and Qwen-VL-Chat with good performances. Other models scored 2% or less, highlighting the difficulty of generating titles. Additionally, none of the LLaVA-NeXT models were able to correctly generate a single title.
Furthermore, Table 14 shows the actual artwork titles generated by the top five models with the best accuracy in the exact match setting. The "Rank" in the table is used to distribute the dataset evenly at the time of its creation (refer to Section 3), between famous and less famous paintings, to prevent bias. From the table, we can infer that a higher proportion of famous artworks with higher ranks were generated, indicating that the models have a better grasp of more famous artworks.

heading: C Evaluation Metrics Formulation
text: This section elaborates on the evaluation metrics proposed in Section 3.2 using mathematical expressions. An explanation consisting of n sentences generated by the model is denoted as G = {g1, · · · , gn}, and a reference explanation consisting of m sentences is denoted as R = {r1, · · · , rm}. The function Entity(·) is defined to extract entities contained in the input text. The notation |G| represents the total number of tokens in the generated explanation, and |R| represents the total number of tokens in the reference explanation.
Entity Coverage (EC) is calculated as follows:
EC(G,R) = Cov(G,R) (1)
Here, Cov(G,R) is a function returning the proportion of entities in R that are covered by G. For partial matches, the Lowest Common Subsequence (LCS) is employed to calculate the longest matching length ratio in the generated explanation relative to the length of the reference entity.
Entity F1 (EF1) is computed as follows:
EF1 = 2× P ×R P +R
(2)
P = ∑ ei∈Entity(G) Countclip(ei, G,R)∑
ej∈Entity(G)#(ej , G) (3)
R = ∑ ei∈Entity(R) Countclip(ei, G,R)∑
ej∈Entity(R)#(ej , R) , (4)
where #(ej , G), #(ej , R) are functions that count the occurrences of entity ej in G and R respectively, and Countclip(ei, G,R) returns the lesser frequency of occurrence of ei in either G or R.
Entity Cooccurrence (ECooc) is calculated using BP from equation (6) as follows:
ECooc(G,R)
=BP (G,R)× Cov(Co(G), Co(R)), (5)
where BP (G,R) is given by:
BP (G,R) = exp(max(0.0, |G| |R| − 1)) (6)
and function Co(·) returns pairs of co-occurring entities within a context window comprising a sentence and its adjacent n sentences. Sentence segmentation was performed using the nltk sentence splitter for this purpose.8

heading: D Details of experimental setting

heading: D.1 LVLM details
text: Model Base Model HuggingFace Name/OpenAI API
BLIP2 (OPT) OPT Salesforce/blip2-opt-6.7b BLIP2 (FLAN-T5-XL) FLAN-T5-XL Salesforce/blip2-flan-t5-xl BLIP2 (FLAN-T5-XXL) FLAN-T5-XXL Salesforce/blip2-flan-t5-xxl InstructBLIP (FLAN-T5-XL) FLAN-T5-XL Salesforce/instructblip-flan-t5-xl InstructBLIP (FLAN-T5-XXL) FLAN-T5-XXL Salesforce/instructblip-flan-t5-xxl InstructBLIP (Vicuna-7B) Vicuna-7B Salesforce/instructblip-vicuna-7b InstructBLIP (Vicuna-13B) Vicuna-13B Salesforce/instructblip-vicuna-13b Yi-VL-6B Yi-6B-Chat 01-ai/Yi-VL-6B mPLUG-Owl LLaMA MAGAer13/mplug-owl-llama-7b mPLUG-Owl2 LLaMA2-7B MAGAer13/mplug-owl2-llama2-7b LLaVA-1.5 Vicuna-13B liuhaotian/llava-v1.5-13b LLaVA-NeXT (Vicuna-7B) Vicuna-7B liuhaotian/llava-v1.6-vicuna-7b LLaVA-NeXT (Vicuna-13B) Vicuna-13B liuhaotian/llava-v1.6-vicuna-13b LLaVA-Next (Mistral) Mistral liuhaotian/llava-v1.6-mistral-7b LLaVA-NeXT (Yi-34B) Yi-34B liuhaotian/llava-v1.6-34b Qwen-VL-Chat Qwen Qwen/Qwen-VL-Chat GPT-4-Vision - gpt-4-1106-vision-preview
8Sentence segmentation was performed using the NLTK sentence splitter.

heading: D.2 LLM details
text: Model HuggingFace Name
FLAN-T5-XL google/flan-t5-xl FLAN-T5-XXL google/flan-t5-xxl OPT facebook/opt-6.7b LLaMA openlm-research/open_llama_7b LLaMA2 meta-llama/Llama-2-7b Mistral mistralai/Mistral-7B-Instruct-v0.2 Vicuna-7B lmsys/vicuna-7b-v1.5 Vicuna-13B lmsys/vicuna-13b-v1.5 Qwen-Chat Qwen/Qwen-7B-Chat Yi-6B 01-ai/Yi-6B Yi-34B 01-ai/Yi-34B GPT-4 gpt-4-1106-preview
D.3 Fine tunning and Inference setting
In this study, to ensure a fair comparison of performance across multiple models, all experiments were conducted on a single NVIDIA RTX 6000 Ada GPU, with 8-bit quantization utilized for model generation. However, due to resource constraints, LLaVA-NeXT (Yi-34B-Chat) model was loaded and inferred in 4-bit mode. To standardize the length of tokens generated across all models, the maximum token length was set to 1024. The same settings were applied to each model for performance comparison purposes.

heading: D.4 Training Datasets
text: Table 16 lists the datasets employed to train the models addressed in this study.

heading: E Details of our created dataset

heading: E.1 Dataset section distribution
text: Table 7 provides a comprehensive breakdown of various types of sections within the dataset, along with their frequency counts. In designing the test set for the "seen" setting, we meticulously considered the distribution of these sections. Through an analysis of the frequency of each section type, we managed to evenly split the data. This strategic approach ensured that the test set was constructed with a balanced representation of each section type, aiming for a more equitable and thorough evaluation process. Due to this methodology, the division of the test set into "seen" and "unseen" portions was based on the distribution of section types, rather than the number of images. Consequently, the number of images in the "seen" and "unseen" parts of the test set may not be equal (refer to Table 2). This was a deliberate choice to prioritize a balanced representation of section types over an equal count of images, enhancing the relevance and fairness of the evaluation process.

heading: E.2 Omitted sections
text: The following sections have been omitted from this document:
• References
• See also
• External links
• Sources
• Further reading
• Bibliography
• Gallery
• Footnotes
• Notes
• References Sources
• Bibliography (In Spanish)
• Bibliography (In Italian)
• Bibliography (In German)
• Bibliography (In French)
• Images
• Links
• List
• Notes and references
• List by location
These sections were deemed unsuitable for the task of generating descriptions of artwork in this study and were therefore removed.

heading: E.3 Train Templates
text: As shown in Table 15, to ensure diversity in training, we utilized seven templates to construct the instruction-based training set. We initially created 49 templates by combining seven base sentences with seven verbs such as explore, explain, and discuss. During experimental evaluations, the models were tested with these 49 templates. We adopted the top seven templates that resulted in the highest accuracy and best adherence to instructions by the models.

heading: E.4 Train Dataset Example
text: As shown in Figure 5 and 6, we adopted the format for fine-tuning Qwen (Bai et al., 2023a) and modified the template presented in E.3 into the form of figures. This format was used for model training and dataset publication.

heading: E.5 Entity Distribution
text: Figures 7 and 8 present the entity distribution within our datasets. The minimal difference in data distribution between seen and unseen cases suggests that the partitioning method described in Step 3 of Section 4 is effective.

heading: F License
text: In our study we created a dataset from Wikipedia articles of artworks. The each image is available under the Creative Commons License (CC) or other licenses. Specific license information for each image can be found on the Wikipedia page or the image description page for that image. The images in this study are used under the terms of these licenses, and links to the images are provided in the datasets we publish so that users can download the images directly. The images themselves are not directly published. Therefore, our data does not infringe upon the licenses.
Title Rank mPLUG-Owl mPLUG-Owl2 Qwen-VL-Chat Qwen-VL-Chat(FT) GPT-4-Vision
Mona Lisa 1 " " " " " The Great Wave off Kanagawa 2 " " " " Vitruvian Man 3 " " " " " Winged Victory of Samothrace 4 " " " Girl with a Pearl Earring 5 " " " " " The Wedding at Cana 6 " " " " The Anatomy Lesson of Dr. Nicolaes Tulp 7 " " " Apollo Belvedere 9 " " " Homeless Jesus 11 " " " Raphael Rooms 12 " Almond Blossoms 13 " " " The Death of General Wolfe 14 " " " " The Persistence of Memory 15 " " " " " Doni Tondo 19 " The Turkish Bath 20 " " Look Mickey 26 " " " " " The Seven Deadly Sins and the Four Last Things 27 " " " " The Conspiracy of Claudius Civilis 28 " La Belle Ferronnière 31 " The Gross Clinic 32 " " The Wedding Dance 33 " " " Sacred and Profane Love 35 " The Sea of Ice 37 " " The Geographer 41 " " Equestrian Portrait of Charles V 45 " The Monk by the Sea 49 " My Bed 51 " " " I Saw the Figure 5 in Gold 55 " Peace Monument 57 " Littlefield Fountain 58 " " Music in the Tuileries 59 " The Cornfield 60 " " Lovejoy Columns 62 " " " The Allegory of Good and Bad Government 64 " Sibelius Monument 72 " " " Headington Shark 73 " The Great Masturbator 75 " Self-Portrait with Thorn Necklace and Hummingbird 81 " Snow Storm: Steam-Boat off a Harbour’s Mouth 83 " Bathers at Asnières 84 " " The Bacchanal of the Andrians 91 " " The Painter’s Studio 95 " Carnation, Lily, Lily, Rose 97 " " Lady Writing a Letter with her Maid 99 " " Two Sisters (On the Terrace) 104 " " " Lion of Belfort 112 " Metamorphosis of Narcissus 114 " Lady Seated at a Virginal 115 " Puerta de Alcalá 116 " " The Three Crosses 118 " Statue of Paddington Bear 119 " Our English Coasts 139 " Hahn/Cock 140 " The Wounded Deer 144 " " The Disrobing of Christ 148 " " Lion of Venice 149 " " " Cross in the Mountains 153 " Man Writing a Letter 164 " " Dying Slave 165 " Nymphs and Satyr 168 " Tomb of Pope Alexander VII 172 " Greece on the Ruins of Missolonghi 178 " The Basket of Apples 186 " " James Scott Memorial Fountain 189 " The Death of General Mercer at the Battle of Princeton, January 3, 1777 193 " Madonna of the Rabbit 200 " " Pyramid of Skulls 209 " Ascending and Descending 220 " The Madonna of Port Lligat 221 " " Le Pont de l’Europe 231 "
Continued on next page
Avg. entities per record: 7. 75
Max entities in a record: 57
Min entities in a record: 1
Most frequent entities: 4
Freq. of most frequent entities: 580
Median entities: 6
% of records with ≤10 entities: 76.10%
Std. dev. of entities: 6.79
Data Type Data Name mPLUG-Owl2 Qwen-VL-Chat LLava-v-1.5 InstructBLIP
Table 16: Details of training datasets.

heading: Future Work
text: This gap clearly shows the pressing need for future research to not only further investigate these issues but also to develop innovative methodologies that ensure the preservation of language knowledge amidst the integration of visual and language abilities