Multimodal Chain-of-Thought Prompting for Metaphor Generation
Authors: Sofia Lugli, Carlo Strapparava (Università di Trento, Fondazione Bruno Kessler, 2024)
Topic: Multimodal

Abstract
This paper introduces an exploratory approach in the field of metaphorical and visual reasoning by proposing the Multimodal Chain-of-Thought Prompting for Metaphor Generation task aimed to generate metaphorical linguistic expressions from non-metaphorical images by using the multimodal LLaVA 1.5 model and the two-step approach of multimodal chain-of-thought prompting. The generated metaphors were evaluated in two ways: using BERTscore and by five human workers on Amazon Mechanical Turk. Concerning the automatic evaluation, each generated metaphorical expression was paired with a corresponding human metaphorical expressions. The overall BERTscore was the following: precision= 0.41, recall= 0.43, and F1= 0.42, suggesting that generated and human metaphors might not have captured the same semantic meaning. The human evaluation showed the model’s ability to generate metaphorical expressions, as 92% of them were classified as metaphors by the majority of the workers. Additionally, the evaluation revealed interesting patterns in terms of metaphoricity, familiarity and appeal scores across the generated metaphors: as the metaphoricity and appeal scores increased, the familiarity score decreased, suggesting that the model exhibited a certain degree of creativity, as it has also generated novel or unconventional metaphorical expressions. It is important to acknowledge that this work is exploratory in nature and has certain limitations.

Introduction
The scope of this paper is to introduce an alternative approach to multimodal metaphor generation. As metaphors are not only pervasive in language but also in everyday life, influencing our thoughts and actions AUTHOR, and as human meaning representations relies on multiple modalities AUTHOR, it became relevant to study metaphors in more than one modality, in particular in the vision domain. Recent research has indeed explored multimodal metaphors generation in a variety of ways: from visual metaphor to literal language AUTHOR; and from metaphorical language to visual metaphor AUTHOR. Nevertheless, the common aspect across these studies is that the metaphorical quality was already present either in the linguistic or in the visual input employed. Therefore, this paper proposes an alternative approach that involves generating metaphorical linguistic expressions from non-metaphorical images, which lack inherent metaphorical qualities. To accomplish this, we employed the new multimodal model LLaVA 1.5 AUTHOR and adopted a two-step approach known as multimodal chain-of-thought prompting AUTHOR: given the first prompt, the model generates the content of the picture; then, the model is provided with both the generated output and a specific prompt to facilitate metaphor generation. The metaphors generated by the model were evaluated through BERTscore AUTHOR and by human workers on Amazon Mechanical Turk. The results show the model's ability to generate metaphorical expressions, with 92% of the generated expressions being classified as metaphors. Additionally, the evaluation revealed interesting patterns in terms of the metaphoricity, familiarity and appeal scores of the generated expressions. Interestingly, as the metaphoricity score increases, the familiarity score decreases while the appeal score increases. This suggests that the model was able to create novel or uncommon metaphorical expressions which may differ from the more conventional metaphors, which the evaluators might have been more familiar with. Despite being less familiar, the metaphorical expressions were preferred over the non-metaphorical ones. It is important to acknowledge that this is an exploratory work, which aims to offer a different approach in multimodal metaphor generation. As such, it is essential to point out the presence of some limitations, in particular concerning the choice of the visual inputs and the constraints of human evaluation.

Conclusion
This study aimed to explore an alternative approach for multimodal metaphor generation using the new LLaVA 1.5 model and Multimodal-CoT prompting. The results showed the model's ability to generate metaphorical expressions when provided with both linguistic and visual inputs which lack inherent metaphorical qualities. Additionally, the evaluation revealed interesting patterns across the metaphoricity, familiarity and appeal scores of the generated expressions. The model exhibited its creativity, as it generated novel or unconventional metaphorical expressions, which were also preferred over non-metaphorical ones. It is important to state again that this is an exploratory work with some limitations. One limitation to consider is the choice of the images used in the study. As manually selected from Google Image, their quality may influence the quality of the captions and metaphors generated by the model. Another limitation to consider is the subjectivity of the evaluation process, it is possible that Amazon MTurk workers may lack the necessary sensitivity and background knowledge to accurately recognize and evaluate metaphorical expressions, despite the instructions included background information about metaphor. Future works should aim to address these limitations by selecting more accurate images, as well as incorporating more diverse and expert annotators. Despite these limitations, the task show promising results for future research in the field of metaphorical and visual reasoning. \appendix