ViPE: Visualise Pretty-much Everything
Authors: Hassan Shahmohammadi
Adhiraj Ghosh
Hendrik Lensch
Venue: EMNLP 2023 Main (2023)
Keywords: Visual metaphors
music video generation
text-to-image synthesis
abstract visualization
diffusion models for abstract art
synthetic data generation
unsupervised label generation

Abstract: Figurative and non-literal expressions are profoundly integrated in human communication. Visualising such expressions allow us to convey our creative thoughts, and evoke nuanced emotions. Recent text-to-image models like Stable Diffusion, on the other hand, struggle to depict non-literal expressions. Recent works primarily deal with this issue by compiling humanly annotated datasets on a small scale, which not only demands specialized expertise but also proves highly inefficient. To address this issue, we introduce ViPE: Visualise Pretty-much Everything. ViPE offers a series of lightweight and robust language models that have been trained on a large-scale set of lyrics with noisy visual descriptions that represent their implicit meaning. The synthetic visual descriptions are generated by GPT3.5 relying on neither human annotations nor images. ViPE effectively expresses any arbitrary piece of text into a visualisable description, enabling meaningful and high-quality image generation. We provide compelling evidence that ViPE is more robust than GPT3.5 in synthesising visual elaborations. ViPE also exhibits an understanding of figurative expressions comparable to human experts, providing a powerful and open-source backbone to many downstream applications such as music video and caption generation.

ViPE: Visualise Pretty-much Everything
Hassan Shahmohammadi Adhiraj Ghosh Hendrik P. A. Lensch
University of Tübingen
hassan. shahmohanmadii@nni teeiii-ttbbbnddd
Abstract
Figurative and non-literal expressions are pro- foundly integrated in human communication. Visualising such expressions allow us to convey our creative thoughts, and evoke nuanced emo- tions. Recent text-to to-image models like Stable Diffusion, on the other hand, struggle to depict non-literal expressions. Recent works primar- ily deal with this issue by compiling humanly annotated datasets on a small scale, which not only demands specialised expertise but also proves highly inefficient. To address this issue, we introduce ViPE: Visualise Pretty-mi much Ev- erything. ViPE offers a series of lightweight and robust language models that have been trained large-scale of lyrics with noisy visual descriptions that represent their implicit meaning. The synthetic visual descriptions generated by GPT3.5 relying neither human annotations nor images. ViPE effec- tively expresses any arbitrary piece of text into a visualisable description, enabling meaningful and high-quality image generation. We pro- vide compelling evidence that ViPE is more robust than GPT3.5 in synthesising visual elab- orations. ViPE also exhibits an understanding of figurative expressions comparable to human experts, providing powerful and open-source backbone many downstream applications such music video and caption generation.
VIIPPE
DALLE
DALLE 2
VIPE: Visualse
A ycung artists hand holding paintbrush, with colorful strokes forming pretty...
Premt-much
+
Everything
Figure Given any arbitrary text, ViPE composes mul- tiple meaningful textual illustrations, thereby assisting state-of-the-ee text-te to-image models in effectively con- veying the intended message via visual symbols.
metaphorical phrases and abstract concepts allows us to express our creative ideas (Schwering et al., 2009). In advertising, they frequently serve as per- suasive tools to evoke positive attitudes (Phillips and McQuarrie, 2004; McQuarrie and Mick, 1999; Jahameh and Zibin, 2023). While humans effort- lessly interpret images with metaphorical content (Yosef et al., 2023), state-of-the-an text-to-image models such as DALL.E 2 (Ramesh et al., 2022) and Stable Diffusion (Rombach et al., 2022) still struggle synthesise meaningful images for such abstract and figurative expressions (Kleinlein et al., 2022: Chakrabarty et al., 2023; Akula et al., 2023). Recent efforts addressing this challenge have mostly focused on constructing datasets for figu- rative language, such as metaphors, similes, and idioms (Chakrabarty et al., 2023; Yosef et al., 2023; Akula et al., 2023). However, these datasets are of- ten small in size and require expert knowledge for expansion. Morcover, despite the benefits of these datasets, the fundamental issue of text-to-image models remains unresolved. To address these lim- itations, we present ViPE: Visualise Pretty-much Everything. ViPE climinates the need for human annotations or images with metaphorical contents,
1 Introduction
"Language is the dress of thought." Samuel Johnson
How do humans comprehend such metaphorical phrase? Conceptual metaphors play significant role in shaping our language, enabling to relate concrete experiences and emotions with abstract concepts (Lakoff and Johnson, 2008). They serve as powerful tools for conveying intricate ideas, highlighting emotions, and adding a sense of hu- mour to our statements. In addition, visualising

---

et al., 2017: Li et al., 2022b), which often ben- efits from large amounts of data (Hassan al.; Edunov et al., 2018; Britz et al., 2017), including noisy data (Rolnick et al., 2017; Vaibhav et al., 2019; Karpukhin et al., 2019). Therefore, we pro- pose to create large-scale dataset, the LyricCanvas dataset, from publicly available lyries with auto- mated but potentially noisy visual elaborations gen- erated by an L.L,, GPT3.5 instructed via prompt- ing. Subsequently, we build ViPE by fine-tuning two lightweight language models, GPT2-Small, and GPT2-Medium (Radford et al., 2019) on the LyrieCanvas dataset. We will show that ViPE, de- spite its size (S: 117M and M: 345M parameters), is more robust than GPT3.5 with 175B parameters in synthesising zero-shot visual elaborations. Fig- ure demonstrates two challenging examples for DALL.E 2, highlighting the improvement depic- tions based on ViPE.
Anbagg
A pewer fur rOCA nalding amop and 100 contuee)
crmpp monn rnflcding of boc.
Figure 2: ViPE enhances the visualisation of figurative language and abstract concepts for text-to-image mod- els. DALL.E 2 (left) struggles to depict such phrases. ViPE successfully captures the implicit meanings and communicates them through visual symbols.
yet effectively assists text-to-image models in vi- sualising figurative and abstract phrases, and even arbitrary textual input. The core idea behind our ap- proach is to unfold the implicit meaning through a new textual description (elaboration) containing vi- sual symbols. Following (Chakrabarty et al., 2023), we use the term Visual Elaboration to refer to the vi- sualisable textual description of piece of text with figurative content. As illustrated in Figure ViPE transforms the input into a detailed image caption while preserving the intended meaning. Therefore, it facilitates the visualisation of figurative language. Building ViPE involves three main stages. (1) A Large Scale Lyric dataset: we compile large- scale collection of lyrics (== 10M lines) rich source of figurative language. (2) Synthetic Visual Elaborations: we construct a supervised dataset we call LyrieCanvas, by employing Large Lan- guage Model(LLM) to generate noisy visual clabo- rations for all the lyrics. (3) Knowledge Distilla- tion: we conduct knowledge distillation to build a robust model by fine-tuning set of lightweight language models on LyricCanvas.
Overall, our contributions are the following. 1. We release a robust and powerful model tailored to assist all text-to-image models in visualising non- literal expressions. 2. We introduce the largest dataset available for generating visual elaborations, which we refer LyricCanvas. With approx- imately 10 million samples, LyricCanvas proves to be adequate, unlike existing datasets, for fine- tuning powerful language models like GPT2. More- over, we provide our scraper framework, allowing researchers to acquire the exact training inputs at no additional cost. 3. We eliminate the expensive and time-consuming involvement of human expert annotations for abstract and figurative visualisa- tions. 4. We show that ViPE's generation is highly robust and is competitive with human experts.
ViPE's powerful zero-shot capability paves the way for its usage in downstream applications such as synthetic caption generation from keywords, ab- stract art visualisations, and music video genera- tions. The source code, pre-trained ViPE, and the LyricCanvas dataset are available at
ViPE, our approach, addresses the limitations found in previous works by leveraging two key find- ings. The first finding is that lyrics serve as rich repository of knowledge, embodying a wide spec- trum of figurative language, including metaphors, similes, idioms, and beyond (Chakrabarty al., 2021; Swarniti, 2022; Astina al., 2021). The second finding stems from the observation that the task of ViPE akin to style transfer using ma- chine translation (MT) (Zhang et al., 2018; Shen
Related Works
2.1 Text-to-Image Generation
Text-to-image synthesis has made significant progress in recent years, with diffusion- models surpassing previous approaches such Variational Autoencoders (VAE) (Razavi al.,
'https://platform.oog http://pwatffrooooea..coio wtt.aai..co///cs/aeeeeeee comm////ccmm////////o//lll/// gpt-3-5
http://gitttu.ccoo http://gthhub..om//aaeeg////////

---

2019) and Generative Adversarial Networks (GANs) (Bao al.. 2017). Prominent text- to-image diffusion models include DALL.E (Ramesh et al., 2022), Stable Diffusion (Rombach et al., 2022), Midlourney' and Craiyon'. Recent works have explored the integration of LLMs into these models. For instance, Opal (Liu et al., 2022c) enables structured search for visual concepts, Gen- erative Disco (Liu al., 2023a) facilitates text-to- video generation for music visualisation, and Reel- Framer (Wang et al., 2023) aids transforming written news stories into engaging video narratives for journalists. Nonetheless, despite their success at generating creative imagery. they still struggle to visualise figurative language effectively (Kleinlein et al., 2022; Chakrabarty et al., 2023; Akula et al.. 2023). Furthermore, research by Chakrabarty et al. (2023); Akula al. (2023) reveals that DALL-E outperforms Stable Diffusion in representing fig- urative language. DALL-E has 3.5 billion pa- rameters, over three times that of Stable Diffusion, and incorporates textual prompts directly to estab- lish relevance between generated images and the provided text. In contrast, Stable Diffusion uses textual prompts through cross-attention during dif- fusion without explicit conditioning. Our approach, ViPE, enhances the visualization of figurative and non-literal expressions in any text-to-image model as a lightweight assistant.
2
3
LyricsGenius
GPT35
Traning
op
Large collection of lyrica LyricCanvvs dataeet
VIPE
Figure 3: Building ViPE involves three main stages. Constructing a large-scale dataset of lyrics. 2. Build- ing a supervised dataset (LyricCanvas) by synthesising noisy visual elaborations using an LLM based on human instructions. 3. Training robust and lightweight model through symbolic knowledge distillation.
with ground truth images through human annota- tions. Akula al. (2023) collected 5,061 metaphor- ical advertisement images with a simple annotation format of is as as "(c.g.. "this pencil is as red as a firetruck"). Zhang et al. (2021) introduced a multimodal metaphor dataset with around 10,000 samples'. Liu et al. (2022a) presented FigMemes, a dataset with 5,000 samples for figurative language in paiitically-oinioonaaee memes.
Despite the benefits of such datasets, they do not provide a fully automated process in figurative lan- guage visualisation. We, for the first time, present a lightweight and robust model tailored for assist- ing text-to-image models visualising figurative language. Our model is not only robust and open source but also requires neither human annotations nor additional images.
2.2 Figurative Language Visualisation
There has been extensive research on textual figu- rative language such as metaphor generation (Yu and Wan, 2019; Chakrabarty et al., 2020; Terai and Nakagawa, 2010), idiom generation and para- phrasing (Liu and Hwa, 2016; Zhou et al., 2021). and simile recognition and interpretation (Zeng et al., 2020; He et al., 2022a). Visualising figu- rative language, the other hand, has received less attention. Existing approaches primarily re- volved around constructing datasets with images and annotations for metaphors, similes, and idioms (Chakrabarty et al., 2023; Yosef et al., 2023; Akula et al., 2023; Zhang et al., 2021). However, these datasets are small and rely on expert knowledge. For example, Chakrabarty et al. (2023) generated visual descriptions and synthetic images for 1.540 linguistic metaphors. Yosef al. (2023) compiled a dataset of less than 3,000 figurative expressions
ViPE
We present ViPE, set of robust and lightweight language models designed to generate visual elab- orations from arbitrary text input. The develop- ment of ViPE comprises three stages, illustrated in Figure Firstly, we perform data collection by scraping and preprocessing an extensive collection of lyrics (z= 10M lines) sourced from G Sec- ondly, we utilise large language model (LLM) to generate noisy visual elaborations for the lyrics by appropriate prompt design. Finally, the paired data of lyries and generated visual elaborations are used to train lightweight language models. They
Pht //wtps:://www.. wod.doyyyyycooo com/ 'https: //www//wwwww/cc/ httpss/wwwwwwraiooccooomm craiyon.ccmmooommooooo
as we know, this dataset not publicly available. "https: /ttp///gennne/eee/eee/e.ee.....t com/

---

are fine-tuned using a causal language modeling ob- jective tailored specifically for visual elaborations. The primary goal to generate detailed textual de- scriptions of visual scenes (visual elaborations) to convey the intended meaning of the rich figurative phrases in lyrics. The generated elaboration can input prompt to any text-to- image synthesiser to visualise the original input.
and demonstrate competitive, and in certain in- stances, superior performance compared human- annotated data (He et al., 2022b; Wang et al., 2021a,b; Hu et al., 2022). A contemporary work is Chakrabarty et al. (2023), which introduces the HAIVMe dataset. There, visual claborations are generated for 1,540 linguistic metaphors using an LLM which are subsequently refined by human experts. We use their dataset to evaluate the robust- ness of our model in Section 4.
then be passed
3.1 Data Collection
Numerous sources have been explored to capture figurative expressions (Chakrabarty et al., 2022;
In our pipeline, instruct GPT3.5 8, denoted hy(-) through prompting to generate visual elabo- rations for a given set of lyrics. More specifically, for (s, L, v) E D, let s be the System Role (a prefix prompt) and I, the set of lyrics lines corresponding to single song in the dataset D, we generate syn- thetic visual elaborations for all lines (1 C 1) by conditioning the GPT3.5 model both and I, t; h s, l). Providing the surrounding lines as prior helps the model understand the theme of the song better and generate suitable visual elabo- rations accordingly. Our System Role contains the exact instructions to convert each line of lyrics to meaningful visual description. The system role encompasses total of 640 words and includes I1 distinct guidelines. For the complete system role, please refer to Appendix A.
Liu et al., 2022b; Bizzoni and Lappin, 2018). Nonetheless, they often suffer from limitations in scale or cost. To overcome this challenge, we pro- pose using publicly available lyrics to build a robust model. Given that the musiXmatch dataset (Bertin- Mahieux et al., 2011) is restricted bag-of-words representations of lyrics with a maximum of only 5k unique words, the efficient integration of such datasets with modern language models becomes a non-trivial task. Therefore. we opt for scraping all the English lyrics from the Genius platform using the LyricsGeniuu API". Subsequently, we apply pre-processing pipeline obtain collection of high-quality lyrics. Our pipeline mainly includes the following filters: Diversity: Lyries contain- ing less than 15 lines with fewer than 4 unique words per song were discarded. Length Limit: Lines with less than unique words or exceeding 20 words in total were excluded from the dataset to maintain balanced and concise text corpus. Size Limit: We only used the top 50 songs from each artist sorted based on popularity to obtain man- ageable dataset. The resulting dataset, referred to as the LyricCanvas dataset, comprises 10 mil- lion lines of lyrics extracted from over 250k songs, by approximately 5.5k different artists. While we are unable to release the lyrics themselves due to copyright policies, we will make available the gen- erated visual elaborations and the scraper and filter framework that can be employed to rebuild the LyricCanvas dataset at additional cost.
Below, summarise the key points covered. Semantic Proximity: The generated description should accurately convey the intended meaning expressed in the given line. Visual Perceptibil- ity: The generated elaborations should be easily visualised. Appropriateness: Some lyrics contain inappropriate content, so generated output should not explicitly describe such content". Diversity: The system is encouraged to utilise various adjec- tives and non-human subjects that help generate detailed and diverse images. For instance, the input line money could be dangerous yields A dragon with evil eyes is lying on a pile of shiny gold. Emo- tion: The system should further take into account the emotional tone of the context when translating lyrics into visual elaborations. This approach pro- motes diverse interpretations of abstract concepts.
3.2 Generating Initial Visual Elaborations
We propose generating synthetic visual elabora- tions using an LLM. Synthetic data produced by LLMs (Thoppilan et al., 2022; Brown et al., 2020; Liu al., 2023b) offer substantial benefits
"The exact version is GPT3.5 Turbo, we use GPT3.5 3.5 for simplicity
httpp::/Ayyricsgeniss.ees readtheeos. io/ee/
"We automatically discarded those Iyrics that were not processed by the system due to inappropriate content.
master/

---

Model
Zero-shot
Tuned (L)
Tuned (XL)
CIDE: Score
VIPES
VPE-M
Validation
GPT2
54.57
57.13
64.00
30-
ViPE-S
58.50
61.42
67.28
Test
Cortect Sizes
GPT2 53.93
54.80
62.65
ViPE-S
54.89
59.60
66.40
Figure 4: ViPE-Medium (ViPE-M) and ViPE-Small (ViPE-S) achieve higher CiDEr scores on the validation set LyricCanvas with longer context.
Table I:
7ero-shot and fine-tuned evaluation results using Fig-Qu -A- (Liu et al., 2022b). and XL denote the large and X-large variations of the dataset. Our model, ViPE-S, demonstrates enhanced comprehension of fig- urative language compared to the standard pre-trained model. GPT2* results are from (Liu et al., 2022b)
3.3 Training ViPE Generating Visual Elaboration Through Text
Training ViPE involves training lightweight stu- dent model hy using the LyricCanvas dataset D with noisy labels generated by the teacher model hy- In contrast to conventional knowledge dis- tillation methods (Hahn and Choi, 2019; Hinton et al., 2015; Ba and Caruana, 2014; Chen et al., 2020), where the student trained to predict soft labels from the teacher, we adopt an indirect ap- proach where knowledge is transferred through dis- crete textual symbols. This approach, known as symbolic knowledge transfer (West et al., 2022), has been shown effective for wide range of NLP tasks (Tang al., 2019; West al., 2022). In our approach, the student model hy is trained on a sequence-to-seqquence task (Sutskever al., 2014). More specifically, given line of lyrics represented as 4; (41 words and its corresponding noisy visual clabora- tion {ut, uf.............. ...., wnth ). comprising words, objective is to learn the conditional likelihood:
using the standard cross entropy loss: (x) =
log Σ Po(elc"), where and c' denote an in- v.ctEB
stance of visual elaboration and its corresponding context in the mini-batch B accordingly. We em- ploy two versions of pre-trained GPT2 (Radford et al., 2019) as the student network hs. GPT2- Small (ViPE-S) and GPT2-Medium (ViPE-M). De- spite the small size of the employed GPT2 mod- els (1 (117M and 345M parameters), their ability interpret the prompts has been shown very effec- tive on both text generation (See et al., 2019) and cross- modality alignments (Nukrai et al., 2022). Furthermore, since we only condition the model on the lyrics line (c'), the loss is computed only for tokens that correspond to the visual elaborations, ensuring that ViPE generates visual descriptions without generating original lyries.
comprising n
Evaluation
m P(D,(ct) P(y/lo 1=1
of
of (1)
Assessing figurative language visualisation is a complex task due to its highly subjective nature (Figure 2). Moreover, existing evaluation pro- cedures differ, ranging from visual entailment Chakrabarty et al. (2023), image recognition Yosef et al. (2023), and retrieval and localization Akula et al. (2023). Therefore, To fully assess the ro- bustness of ViPE, propose end-to-end human evaluation and various automated metrics at differ- ent levels of granularity.
Where et denotes the context prior, consisting of L preceding lines (if available in the correspond- ing lyries) relative context prepended as a prefix the visual elabo- ration u/. practice, we experiment with various context sizes. We start with a context size zero (no context), followed by sizes of one, three, five, and seven, which follows cf = ...44 4-+. where i and t E {0, 1, 3, 5, 7} correspond to the instance of lyries line in song and the context length. As shown in Figure 4, by extending the context size, we provide more information to the model, thereby facilitating the generation of o; that better fits the entire lyries.
unified sequence. The
4.1 Intrinsic Evaluation
In this section, evaluate the general figurative language understanding of ViPE using the Fig-QA dataset (Liu al., 2022b). It contains 2 12k figu- rative phrases with correct and incorrect interpreta- tions in the Winograd style (Levesque et al., 2012). For instance, the figurative sentence Her word had
The student hs is trained to learn the parameters 8 to estimate the conditional likelihood Po (x(c')

---

the strength of a wine glass. is paired with both Her promises can be believed and Her promises cannol be trusted. as two distinct samples. This benchmark is suitable for our purpose given that it covers various themes, including common-sense object knowledge, visual metaphor, common-sense social understanding, and cultural metaphors. We employed their evaluation framework for GPT2 and evaluated the small version of ViPE (ViPE-S) trained with the context size of one. Shown in Ta- ble I, we compare the results of ViPE with that of GPT2 reported by Liu et al. (2022b) in both zero- shot and fine-tuned cases. The results validate the superiority of ViPE over pre-trained GPT2 in both zero-shot and fine-tuned scenarios, highlighting its advanced understanding of figurative language.
Human Experts
GPT33.5
ViPE
TR
IR
TR
IR
TR
IR
Metaphor
27.8
42,8
28.7
35.5
32,1
41.3
Mctaphor
36.3
49.4
40.0
37.33
47.1
46.6
Ca
63.4
77.2
52.9
06.3
65.8
79.8
Caption
46.2
75.7
R5.4
90.3
87.2
94.7
Table 2: A comparative report Image-metaphor and image-caption retrieval using corpora generated by GPT- 3.5, ViPE, and human experts (HAIVMet dataset) in zero-shot (24) and fine-tuned (f1) settings. TR and IR denote the mean image-to-text and texa-to-imaee retrieval scores respectively. ViPE outperforms GPT3.5 and shows competitive understanding to human experts.
modal encoder-decoder mixture model, making it suitable for retrieval evaluation. We used BLIP in both zero-shot and fine-tuned settings. In zero- shot, the entire retrieval dataset is used for testing, while in fine-tuned, 90% of the data is used for fine-tuning, leaving 10% for evaluation.
Next, we evaluate ViPE on fine-grained cate- gories in the Fig-QA dataset (Liu et al., 2022b). As shown in Figure 7, ViPE demonstrates a compre- hensive understanding of all categories in both zero- shot and fine-tuned settings. Notably, the enhance- ment is more prominent in the visual categories, aligning with our goal of generating visualisable descriptions for figurative language.
We report the mean recall across the top- top-5, and top-10 retrieval scores in Table 2. ViPE out- performs GPT-3.5 and human experts (HAIVMet) in image-mttaphor retrieval (referred to TR in the table). However, while outperforming GPT3.5, ViPE slightly lags behind humans in retrieving metaphors from images. One potential reason might be that human experts tend to very specific in describing metaphorical images (Chakrabarty et al., 2023), creating a more discrete feature space, making it casier for BLIP to interpret. Additionally, we conduct the same evaluation pairs of images and visual elaborations (instead of metaphors) to assess the alignment between the elaborations and corresponding images, similar to image-caption re- trieval. Shown the lower part of Table 2, ViPE outperforms both GPT3.5 and humans in both zero- shot and fine-tuned cases. An interesting finding is that GPT3.5 while showing poor performance on end-to-end evaluation, shows superior perfor- mance to humans on image-caption retrieval. This suggests that GPT3.5 prioritizes the visualisability of generated elaborations without establishing a strong connection with the metaphors. In contrast, ViPE exhibits comparable or in some cases even superior end-to-end evaluation of image-metaphor compared to humans, while also generating more detailed and concrete visual elaborations, as evi- denced by the high image-caption retrieval scores.
4.2 Extrinsic Evaluation
Image-text Retrieval: For thorough end- to-end evaluation, we conduct image-to-text and text-to-image retrieval on the HAIVMet dataset (Chakrabarty et al., 2023). HAIVMet con- tains 1,540 linguistic metaphors and corresponding visual elaborations reviewed by experts. We cre- ated pairs of metaphors and visual elaborations, well visual elaborations and images, for HAIVMet, ViPE-M trained with the context size of 7, and GPT3.5. Since HAIVMet has ground truth visual claborations, we only generated 10 im- ages per elaboration using Stable Diffusion (Rom- bach et al., 2022). For ViPE-M and GPT3.5, we generated deterministic visual elaborations for the same metaphors and then generated 10 images for each elaboration. Although the authors of HAIVMet (Chakrabarty et al., 2023) used DALL-E 2 (Ramesh al., 2022) to generate images, we opt for transparent and reproducible approach by utilising Stable Diffusion.
After compiling three datasets from MeIVMet, ViPE, and GPT3.5, we utilised the fine-tuned ver- sion of BLIP (Li et al., 2022a) on COCO (Lin et al., 2014) retrieval. BLIP excels in vision-language benchmarks due to the effective use of a multi-
Emotion Visualisation: Emotions are deeply grounded in the human visual system (Kragel al., 2019) and computational models effectively predict emotional categories from images in various stud-

---

ViPE-M
GPT-3.5
CPT335 Turbo
VIPE-M
Semantic Proximity
60.00
54.10
22N
C1N
Visual Perceptibility
25.11
22.70
10.%%
47%
1.0%
CDN%
3.%
20
75.%
44%
53%
8.24
CAN
comparative analysis of ViPE-Medium and GPT3.5 in converting emotionally charged tweets into visual elaborations. ViPE is superior in generating im- age descriptions, demonstrating higher visual percepti- bility, and preserving tweet semantics more effectively.
Table 3:
30:70
102%
4.%
10:M5
10.5%
10:74
10.4%
20:0
10.5h
129
1629
344
1m
1um
20.25
mm:
1.34
12:0
25.25
Predkted
Figure 5: A comparative analysis between ViPE-M and GPT3.5 on generating visual elaboration from emotion- ally charged messages. ViPE-M demonstrates superior generalization performance and effectively mitigates bias towards the most dominant class ("Joy").
ies (Rao al., 2020; Zhao et al., 2022; You et al., 2016; Achlioptas et al., 2021). We, therefore, lever- age the Emotion dataset (Saravia al., 2018) for our purpose. It is a classification dataset compris- ing 20k samples from Twitter messages with six basic emotions. The difficulty of visualising tweets and the plausibility of emotion detection from im- ages puts it in line with our objective. In particular, Let De- = =t,, (4,,4] [Pal represent the Emotion dataset consisting of tweets i and their corresponding la- bels 4, Visual elaborations are generated determin- istically for all tweets (t, E D.). resulting in the new dataset D. = {v 43] Pol, where v; denotes the ith visual elaboration. This is carried out for both ViPE-M and GPT3.5, using the same Sys- tem Role applied to create the LyricCanvas dataset. Subsequently, we fine-tune a pre-trained BERT model (BERT-base-uccaase vlin et al., 2018) on D. and evaluate the robustness of ViPE and GPT3.5 using two metries: Semantic Proximity (SP) measures how well the generated visual claboration ii represents the meaning of the tweet 4t determined by the final classification ac- curacy on D.. Visual Perceptibility (VP) assesses the visualisability of the visual elaboration is by computing the cosine similarity between the CLIP embeddings of iú and its corresponding generated image 1; by Stable Diffusion.
towards generating positive and joyful sentences, potentially influenced by positive reinforcement in its training. In contrast, ViPE-M demonstrates more precise performance and successfully miti- gates hias towards the dominant class. For example, GPT3.5 shows a 77 3% confusion rate between sur- prise and joy, whereas ViPE reduces this bias to 37.9%. Additionally, certain emotions are challeng- ing to distinguish solely from visual elaborations. For instance, the text I feel that the packaging is really lovely and the product itself just does ev- erything you ask is labeled as love, but ViPE's visual claboration of a woman holding a beauti- fully wrapped gift box, smiling with excitement is confused with joy.
for classification (De-
Safety and Appropriateness: Even though ViPE has been fine-tuned on data generated by GPT3.5 with filtering which incorporates certain measures to mitigate inappropriate content, it is built upon the GPT-2 model which is prone to generating in- appropriate content. Hence, to measure the appro- priateness of ViPE's output, we conducted the fol- lowing experiment. Using the Alt-profanity-check framework 10 we first measured the profanity score (nnapropriateooffeesiiee language) of the lyrics in the valuation set (around IM line of lyries) and dis- tributed them over five intervals. We then measured the profanity scores of the generated visual clab- orations in each interval from GPT3.5 and ViPE. In addition, we prompted the pre-trained GPT2 model with the lyrics and generated new text (not necessarily visual elaboration). Subsequently, we measured the profanity score for the GPT2's 's out- put. Demonstrated in Figure 6, GPT2-M's scores
The results are presented in Table 3. ViPE demonstrates superior performance in generating image descriptions, indicated by higher visual per- ceptibility scores. It also effectively preserves the semantic content of the tweets, as evidenced by the semantic proximity metric. Overall, our findings lend support to the efficacy of symbolic knowledge distillation (see Section 3.3) from large-scale noisy data, demonstrated by ViPE's superior zero-shot capabilities in generating visual elaborations.
Figure 5 compares ViPE- M and GPT3.5 in fine-grained emotion classifi- cation using the Emotion dataset. GPT3.5 leans
Fine-grained Emotions:
hhttps://pppt http:///ppppooogppooeeeee org/project/ alt-profanity-cheds

---

VIPE-S (Fine-tuned) (Zorc-shoc) GPT2 (fre-tunedi
GFT3.
MPEM
0.70
Moar Velure
GPT2.M
Accuraclss
(Zarn-hoor)
0.5
0.4
02
0.55
0.
102, 0:4]
10.4. 0.6]
[..6.0.8
I0.8, 1.0]
50cial
cilturel
Utijetts
Visual
Profanity Intervals
Calegories
Figure Profanity and offensive language analysis for lyrics with increasing profanity scores and those for visual elaborations generated by GPT2-M, ViPE-M, and GPT3.5. While GPT2-M's scores show a strong resemblance to that of pure lyries, ViPE and GPT3.5 produce appropriate content across all intervals.
Figure 7: Zero-shot and one-tuned evaluation results different categories of the Fig-QA dataset (L.iu al., 2022b). ViPE-S outperforms GPT2 across all categories with a more pronounced gap in the visual category.
Implementation details
closely follow that of lyries, indicating inappropri- ate language. GPT3.5 and ViPE on the other hand effectively reduce the profanity scores across all the intervals. These findings support ViPE's abil- ity to transform inappropriate content and generate safe visual elaborations.
Training on LyricCanvas:
Two versions of ViPE are developed: ViPE-M (based GPT2-small) and ViPE-S (based GPT2-Medium). The mod- els are fine-tuned on LyricCanvas for epochs us- ing A100 Nvidia GPUs, each with 40 GB RAM. We use the A optimizer (Loshchilov and Hut- ter, 2017) with a learning rate of 5e- 05 and a linear scheduler with 1000 warmup steps. For ViPE-S, the batch size is generally 50, except with a context size of 7, where batch size of 32 is utilised. In the case of ViPE-M, the batch sizes vary for different context sizes: {32, 32, 20, 16,8) for context sizes [0, 1, 5, 7). respectively. 10 % of LyricCanvas 1M samples) is used for validation.
4.3 Human Evaluation
To strengthen our evaluation toolkit, we conducted user study involving 30 native English-speaking participants aged between 20 and 40 for compre- hensive end-to-end assessment as follows:
Data preparation: From the MAIVMet dataset, we randomly selected 60 metaphors. For each metaphor, we generated visual elaborations using ChatGPT, ViPE, and added the human expert claborations from MAIVMet. Subsequently, we employed Stable Diffusion to generate correspond- ing images from these visual elaborations.
Image-text Retrieval: We load a BLIP (Li ct al., 2022a) checkpoint trained on COCO, initialised on ViT-B(Dosovitskiy et al., 2021) and BERT-base (Devlin et al., 2018). To finetune, we use a batch size of 16 for 10 epochs using Adam learn- ing rate of le 4, and batch size of 128 with reranking for fast inference, commonly used in trieval (Li al., 2022a; Ghosh al., 2023).
Experiment: The experiment involved presenting participants with metaphor alongside three images generated from prompts provided by human experts (HAIVMet dataset), ChatGPT, and ViPE. Their task was to choose the image that best represented the metaphor's meaning.
Emotion Classification: BERRT-base-uncased is
fine-tuned for 5 epochs using Adamw optimizer, a learning rate 5c 05 and batch size of 256. Figurative QA: We made use of the provided evaluation framework by Liu et al. (2022h) and trained with the batch size of 32 for 5 epochs using AdamW optimizer, with learning rate of 5e 05. The test results are publicly available under the name IMAGINATE EMNLP2023 on their leader- board.
Findings: Our findings dovetail well with the previous results. Participants favored images from human experts 38.67% of the time, followed by ViPE's images at 33.61% %, and ChatGPT's 27.72%. These results validate ViPE's superiority over ChatGPT and its competitive performance with human experts.

---

ales diverse image captions, visual elaborations, from arbitrary textual input. Our approach involves training lightweight language models on a novel dataset, LyricsCanvas, comprising 10 million lines of lyrics paired with visual elaborations generated by GPT3.5. Our key achievements are as follows: (1) We created the LyricsCanvas dataset, which en- ables training powerful language models for visual- ising figurative language. (2) We built ViPE by dis- tilling the knowledge from GPT3.5 to a lightweight and open-source model with robust performance. ViPE exhibits highly robust zero-shot generation capabilities, surpassing GPT3.5 and achieving com- petitive results compared to human experts. (3) We demonstrated the versatility of ViPE by generat- ing visually captivating elaborations from various textual inputs, such as non-lexical terms, abstract concepts, and figurative language. This opens up possibilities for applications in creative writing, paraphrase generation, and style transfer. (4) ViPE serves as a strong baseline for visualising lyrics, ev- ident in the visually appealing artworks it generates for music video visualisations.
pus batler z syng tee mall sikn ing and laghing
Aacci lead upc holdd on dart und floy shafered pooss scaltredd
Avered of Ie dei plugpoun poaddoeddd twingg andrusted skizs plog ound, ile tragic
Wide
love, le trgic
Figure 8: ViPE demonstrates robust contextual under- standing across arbitrary textual inputs. Images are generated with ViPE claborations and Stable Diffusion.
6 Applications
Besides its power to produce well-depictable prompts for text-to-image synthesis, we utilise ViPE as a versatile assistant to generate stylish visuals for music videos. We em- ploy a zero-shot text-to-video synthesis approach, incorporating motion information for temporal con- sistency (Khachatryan et al., 2023) and smooth visual transitions between lines by interpolating image and text embeddings. More specifically, our approach comprises (1) extracting lyries and times- tamps from given audio file, (2) generating visual elaborations from lyries using ViPE, (3) creating a cohesive video narrative that encompasses the composition of the song, including all the lyrics and music. In Appendix B, we detail our pipeline. Below, we summarise our key points.
Music Video Generation:
Overall, ViPE enables accessible solutions for complex research questions and paves the way for automated pipelines. In the future, we plan to ap- ply ViPE in investigating the interplay between language and perception related disciplines such as psycho-linguistee and cognitive science.
(1) ViPE generates relevant and visualisable elab- orations. (2) To tackle the semantic coherence mis- match arising from diverse visual elaborations, we propose novel approach called "Chunked Simi- larity Alignment." " This technique aligns the most relevant parts of the latent representations during transitions. Overall, our finding further confirms the robustness of ViPE across various domains.
Limitations
While we provide evidence of ViPE's robustness and understanding of figurative language using mul- tiple benchmarks, the evaluation may have limita- tions. The choice of evaluation metrics and the specific datasets used for assessment may not fully capture the nuances and complexities of human figurative expressions. More specifically the cul- tural differences in creating and interpreting figura- tive phrases. Further investigation and comparative analysis with more diverse and perhaps new evalu- ation measures and data sets would strengthen the assessment of ViPE and potential future models.
Style Transfer and Creative Writing: ViPE demonstrates robust contextual understanding across different domains. Figure 8 shows exam- ples of images generated by Stable Diffusion us- ing ViPE's elaborations. ViPE exhibits impressive generalization capabilities, even with non-lexical terms. More examples are available in Appendix C. These findings indicate that ViPE has applications in style transfer and creative text generation.
Ethics Statement
Even though The ViPE model has been fine-tuned on data generated by GPT3.5 with filtering which incorporates certain measures to mitigate biases. it is built upon the GPT-2 model. GPT2, despite its popularity, exhibits biases in job stereotypes, gen- der, and ethnicity distributions (Kirk et al., 2021).
7 Conclusion
this paper, we introduced ViPE, the first auto- mated model for visualising figurative expressions in text-to-image models. ViPE efficiently gener-

---

Therefore, there is still possibility that ViPE may exhibit similar biases. We believe it is crucial to re- main vigilant in addressing and rectifying biases in language models. Moreover, it is important to note that popular text-to-image models might aggravate such biases, as they tend to over-represent aspects associated with whiteness and masculinity in their latent space (Luccioni et al., 2023).
Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laak- sonen, Mubarak Shah, and Fahad Shahbaz. Khan. 2023. Person image synthesis via denoising diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5968-5976.
Yuri Bizzoni and Shalom Lappin. 2018. Predicting human metaphor paraphrase judgments with deep neural networks. In Proceedings of the Workshop on Figurative Language Processing, pages 45 55.
Acknowledgements
This work has been supported by EXC number 2064/1 Project number 390727645, as well as by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S1 1039A. The authors thank the International Max Planck Research School for Intelligent Sys- tems (IMPRS-IS) for supporting Hassan Shahmo- hammadi.
Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc Le. 2017. Massive exploration of neural ma- chine translation architectures. In Proceedings of the 2017 Conference on Empirical Methods Natu- ral Language Processing, pages 1442-1451, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33: 1877-1911.
References
Panos Achlioptas, Maks Ovsjanikov, Kilichbek Hay- darov, Mohamed Elhoseiny, and Leonidas J Guibas. 2021. Artemis: Affective language for visual art. In Proceedings of the IEEF/CVF Conference on Com- puter Vision and Pattern Recognition, pages 11569-.. 11579.
Tuhin Chakrabarty, Smaranda Muresan, and Nanyun Peng. 2020. Generating similes effortlessly like a pro: A style transfer approach for simile generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6455-6469.
Arjun R Akula, Brendan Driscoll, Pradyumna Narayana, Soravit Changpinyo, Zhiwei Jia, Suyash Damle, Garima Pruthi, Sugato Basu, Leonidas Guibas, William T Freeman, et al. 2023. Metaclue: Towards comprehensive visual metaphors research. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23201-23211
Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 7139-7159, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Riky Astina, I Wayan Juniartha, and Ni Nyoman Deni Ariyaningsih. 2021. An analysis of hyperbole in al- bum "the chainsmoker. Elysian Journal: English Lit- erature, Linguistics and Translation Studies, 1(1):11- 20.
Tuhin Chakrabarty, Arkadiy Saakyan, Olivia Winn, Artemis Panagopoulou, Yue Yang. Marianna Apid- ianaki, and Smaranda Muresan. 2023. I spy a metaphor: Large language models and diffusion models co-create visual metaphors. arXiv preprint arXiv:2305.1477
Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? Advances in neural information processing systems, 27.
Tuhin Chakrabarty, Xurui Zhang, Smaranda Muresan, and Nanyun Peng. 2021. MERMAID: Metaphor gen- eration with symbolism and discriminative decoding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 4250-4261, Online, Association for Computa- tional Linguistics.
Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. 2017. Cvae-gan: fine-grained image gen- eration through asymmetric training. In Proceedings of the IEEF international conference on computer vision, pages 2745-2754.
Thierry Bertin- Mahieux Daniel P.W. Ellis, Brian Whit- man, and Paul Lamere. 201 1. The million song
Yen- Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu, 2020. Distilling knowledge learned in bert for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7893-7905.
dataset. In Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR 2011).

---

Jacob Devlin, Ming-Wo Mei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre- Pre-training of deep bidirectional transformers for language understand- ing. arxiv preprint 1010::88000.4
Vladimir Karpukhin. Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. 2019. Training on synthetic noise improves robustness to natural noise in machine translation. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 42- 47.
Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x 16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. 2023. Tex:2video-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15954-15964.
Sergey Edunov. Myle Ott, Michael Auli, and David Grangier. 2018 Understanding back-translation scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, pages 489 500. Association for Computational Linguistics.
Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer. Aleksandar Shtedritski, and Yuki Asano. 2021. Bias out-of-the- box: An empirical analysis of intersectional occupa- tional biases in popular generative language models. Advances in neural information processing systems, 14:2611-26244
Adhiraj Ghosh, Kuruparan Shanmugalingam, and Wen- Yan Lin. 2023. Relation preserving triplet mining for stabilising the triplet loss in re-identification systems. Proceedings of the IEFF//CV Winter Conference on Applications of Computer Vision, pages 4840- 4849.
Ricardo Kleinlein, Cristina Luna- Juna-Jiménez, and Fer- nando Femandez-Marttnez. 2022. Language does more than describe: On the lack of figurative speech in text-to-image models. arXiv preprint arXiv:2210.10572
Sangchul Hahn and Heeyoul Choi. 2019. Self- knowledge distillation in natural language processing. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 4300, Varna, Bulgaria. IN- COMA Ltd.
Philip A Kragel, Marianne C Reddan, Kevin S LaBar, and Tor Wager. 2019. Emotion schemas are embed- ded in the human visual system. Science advances, 5(7) ccaas38833
George Lakoff and Mark Johnson. 2008. Metaphors we live by. University of Chicago press.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marein Junczys-Dowmunt, William Lewis, Mu Li, et al. Achieving human parity on au- tomatic chinese to english news translation.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thir- teenth international conference the principles of knowledge representation and reasoning.
Qianyu He, Sijie Cheng, Zhixu Li, Rui Xie, and Yanghua Xiao. 2022a. Can pre-trained language models interpret similes as smart as human? arXiv preprint arXiv:2203 0.4452.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022a. Blip: Bootstrapping language-ing image pre- pre-training for unified vision- lsioonlanguaae understand- ing and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR.
Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari, and Mohammad Norouzi. 2022b. Gener- ate, annotate, and leam: NI.P with synthetic text. Transactions of the Association for Computational Linguistics, 84.
Yafu Li, Yongjing Yin, Jing Li, and Yue Zhang. 2022b. Prompt-driven neural machine translation. In Find- ings of the Association for Computational Linguistics: ACL 2022, pages 2579-2590.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge a neural network, arXiv preprint 1r0.:v:1503.02...
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer.
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2022. Promptcap: Prompt-guide guided task-aware image captioning. arXiv preprint 221 2022221199000 1.09999.
Changsheng Liu and Rebecca Hwa. 2016. Phrasal sub- stitution of idiomatic expressions. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 363-373, San
Haifaa Jahameh and Aseel Zibin. 2023. The use of monomodal and multimodal metaphors in advertising jordanian and american food products on facebook: A comparative study. Heliyon, 9(5).

---

Diego, California. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, 1lya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):
Chen Liu, Gregor Geigle, Robin Krebs, and Tryna Gurevych. 2022a. FigMemes: A dataset for figura- tive language identification in pilticalll-opinionatte memes. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7069-7066887 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu. and Mark Chen. 2022. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125.
Tianrong Rao, Xiaoxu Li, and Min Xu. 2020. Learning multi-level deep representations for image emotion classification. Neural processing letters, 51:2043- 2061.
Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022b. Testing the ability of language mod- els to interpret figurative language. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Langwage Technologies, pages 4437-4452, Seattle, United States. Association for Computational Linguistics.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. 2019. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32.
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. 2017. Deep learning is robust to massive label noise. arXiv preprint /rXiv:/70s 10694.
Vivian Liu, Tao Long. Nathan Raw, and Lydia Chilton. 2023a. Generative disco: Text-to-video gener- ation for music visualization. arXiv preprint 2rX4v:2304.88511.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High- resolution image synthesis with latent diffusion mod- cls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10694100955
Vivian Liu, Han Qiao, and Lydia Chilton. 2022c. Opal: Multimodal image generation for news illustration. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1-17.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang. Junlin Wu, and Yi-Shin Chen. 2018 Carer: Con- textualized affect representations for emotion recog- nition. In Proceedings of the 2018 conference on empirical methads in natural language processing, pages 3687-3697.
Yiheng Liu, Tianle Han, Siyuan Ma. Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. 2023b. Sum- mary of chatgpt/gpt-4 research and perspective to- wards the future of large language models. arXiv preprint 2rX4v:2304.07852
Angela Schwering. Kai-Uwe Küchnberger, Ulf Krum- nack, Helmar Gust, Tonio Wandmacher, Bipin In- durkhya, and Amitash Ojha. 2009. A computational model for visual metapphor-immteppreting creative vi- sual advertisements. In International Conference on Agenty and Artificial Intelligence, volume I, pages 339-344. SCITEPRESS.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint
arXiv:1711.05101
Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. 2023. Stable bias: An- alyzing societal representations in diffusion models. arXiv preprint arXiv:2303.1 1400..
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Confer- ence on Computational Natural Language Learning (CoNLL), pages 943-861. Association for Computa- tional Linguisties.
Edward F McQuarrie and David Glen Mick. 1999. Vi- sual rhetoric advertising: Text-interpretive, exper- imental, and reader-response analyses. Journal of consumer research, 26(1):37-54.
David Nukrai, Ron Mokady, and Amir Globerson. 2022, Text-only training for image captioning using noise- injected clip, arXiv preprint arXiv:2211.0055..
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. Advances in neural information processing systems, 30.
Barbara J Phillips and Edward McQuarrie. 2004, Be- yond visual metaphor: A new typology of visual thetoric in advertising. Marketing theory, 4(1-2):113- 136.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scal, scale weak su- pervision, arXiv preprint 221iv:2112.0435.
Ni Wayan Swamiti. 2022. Analysis of figurative lan- guage in "easy on me" song lyric. RETORIKA: Jur- nal Imma Bahasa, 8( 133:13188.

---

Raphael Tang, Yao Lu, and Jimmy Lin. 2019. Natural language generation for effective knowledge distilla- tion. G.MNIP-IJCNNL 2019, page 202.
Human Language Technologies, Volume 1 (Long and Short Pupers), pages 861-871, Minneapolis, Min- nesota. Association for Computational Linguistics.
Asuka Terai and Masanori Nakagawa. 2010. A com- putational system of metaphor generation with eval- uation mechanism. In Artificial Neural Networks- ICANN 2010: 20th International Conference, Thessa- loniki, Greece, September 15-18, 2010, Proceedings, Part II 20, pages 142-147, Springer.
Jiali Zeng, Linfeng Song, Jinsong Su, Jun Xie, Wei Song, and Jiebo Luo. 2020. Neural simile recogni- tion with cyclic multitask learning and local attention. In P Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9515 95222.
Dongyu Zhang, Minghao Zhang, Heting Zhang, Liang Yang, and Hongfei Lin. 2021. MultiMET: A multi- modal dataset for metaphor understanding. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 1 1th Inter- national Joint Conference on Natural Language Pro- cessing (Volume Long Pupers), pages 22255225. Online. Association for Computational Linguistics.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer. Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.022.0 08239.
Vaibhav Vaibhav, Sumeet Singh, Craig Stewart, and Gra- ham Neubig. 2019. Improving robustmess of machine translation with synthetic noise. Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1916-1920.
Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2018. Style transfer as unsupervised machine trans- lation. arXiv e-prints, pages arXiv-1808.
Sicheng Zhao, Quanwei Huang, Youbao Tang, Xingxu Yao, Jufeng Yang. Guiguang Ding, and Bjöm W Schuller. 2022. Computational emotion analysis from images: Recent advances and future directions. Human Perception of Visual Information: Psycholog- ical and Computational Perspectives, pages 85-113.
Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021a. Want to reduce labeling cost? gpt-3 can help. arXiv preprint arXiv:2108. 13487.
Sitong Wang. Samia Menon, Tao Long, Keren Hender- son, Dingzeyu Li, Kevin Crowston, Mark Hansen, Jeffrey V Nickerson, and Lydia Chilton. 2023. Reelframer: Co-creating news reels on social media with generative ai. arXiv preprint 17Xix:230400965.
Jianing Zhou, Hongyu Gong, and Suma Bhat. 2021. PIE: A parallel idiomatic expression corpus for id- iomatic sentence generation and paraphrasing. In Proceedings of the 17th Workshop on Multiword Ex- pressions (MWE 2021), pages 33-48, Online, Asso- ciation for Computational Linguistics.
Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. 2021b. Towards zero- zero-label language learning. arXiv preprint 2rXiv:2109. 199.09193.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang. Liwei Jiang. Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4602-4625 Seat- tle, United States. Association for Computational Linguistics.
Ron Yosef, Yonatan Bitton, and Dafna Shahaf. 2023. Irfl: Image recognition of figurative language. arXiv preprint arXiv:2303.1 15445.
Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. 2016. Building a large scale dataset for image emotion recognition: The fine print and the bench- mark. In Proceedings of the AAAI conference on artificial intelligence, volume 30.
Zhiwei Yu and Xiaojun Wan. 2019, How avoid sen- tences spelling boring? towards a neural approach to unsupervised metaphor generation. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics:

---

A System Role
subjects. For instance, instead of "A person stand- ing under a starry night sky, aware that there no tomorrow" use "A clock with its hands frozen, a cold weather where everything is frozen". 11. Describe the scene with details and use various ad- jectives. For instance, colorful kites in the cloudy sky, frozen lakes with gorgeous sunset in the back- ground, very long tree reaching the clouds, and so on.
Below we provide the prompt, also known as Sys- tem Role, that we used for instructing GPT3.5 Turbo to generate visual elaboration for given set of lyries.
Follow my commands: Convert abstract lyries into depictable prompts that represent the original lines, such as using "a man and a woman are hav- ing a conversation over a cup of tea" to represent "somebody once told me" and "a shining diamond ring" to represent "all that glitters is gold." 2. Keep prompts concise and avoid creating prompts longer than 20 words. 3. The requirement is to have one prompt per line. If there are 40 lines of input, the output should contain 40 prompts. When gener- ating prompts, do not focus on what the subject is thinking or feeling, For example, instead of "a stu- dent thinking about his long assignment list, over- whelmed by so much coursework," which is diffi- cult to visualize, describe the student's appearance, such "a male student looking at long assign- ment list, with a scared expression, tears rolling down from his check." " 5. Structure all prompts by setting a scene with at least one subject and a concrete action term, followed by a Comma, and then describing the scene. For instance, "a view of forest from a window in cozy room, leaves are falling from the trees." 6. To add variety and avoid repetition, it is important to mix up singular and plural forms when referring to subjects or ob- jects in the prompts. For example, "two cats," "ten men," " "five girls," or "seven books" can be used in- stead of consistently using singular forms. 7. Some lyrics may contain inappropriate content, but the goal is to generate acceptable and decent prompts for them. 8. Consider the sentiment of the song when generating prompts. The same line should be represented differently depending the mood of the song. For example, "I went for walk" could be converted to "a young man is taking a walk on a sunny day in a beautiful park full of trees" the song is positive, or "an old man is taking a walk at night in a dark forest full of trees" if the song is negative. 9. Do not use generic words such as person, people, man, woman, individual, figure, ob- ject, etc. Instead, across various topics, use diverse and specific terms such as desert, island, statue, skyscraper, stars, moon, rainbow, snowflakes, wolf. horse, dragon, bird, python, bike, truck, airplane, astronaut, daisics, roses, diamond ring, and so on, where appropriate. 10. Do not always use human
For example, if give you: "1. Feels like the weight of the world 2. Like God in heaven gave me a turn 3. Money could be dangerous 4. Everyone is lcaving 5. This is gonna be the best day of my life 5. I am forever free" expect you to give me prompt per line as follows:
"1. A man carrying a giant globe on his back a post apocalyptic world, struggling with the weight. 2. A scary demon is spinning a wheel in the dark and gloomy sky. 3. A dragon with evil eyes is lying on pile of shiny gold. 4. A picture of abandoned city in dark gloomy weather, buildings are dark and destroyed. 5. A stunning fireworks display illuminating the night sky, people are hap- pily dancing. 6. A majestic eagle soaring through the vast open sky, wings outstretched." "
Prioritize Rules 9 and 10: don't use generic terms and human subjects while conveying the orig- inal lines. Start your response with "1.".
B
Music Video Generation: A Detailed Pipeline
In this section, we provide the detailed layout of the methodology used to convert song lyrics to mu- sie videos. In B. delve into the steps taken to convert the song as an audio file into a suitable in- put for text-to-image Latent Diffusion Models, i.e. Stable Diffusion. In B.2, we detail the type of ViPE model used and the process of documenting the en- tire length of the song with visual elaborations. In B.3, we provide the steps taken in conducting zero- shot text-to-video generation for given lyric and its corresponding visual claboration, while in B. B.4, talk about our interpolation mechanism and the strategy we use mitigate semantic incoherence while interpolating in the text embedding space, as introduced in Section 6.
B.1 Preprocessing
As the main inputs to the lyric-to-video pipeline, we only require the audio file, the title of the song,

---

and the artist's name to return a complete music video that illustrates the lyries as they are being sung. From the audio file, we retrieve the song lyrics and an accurate set of timestamps TN-1, that corresponds to lyrics 4,, such that there a direct correspondence of (T) + 4. akin to fine- grained audio transcribing.
occurrence of random motion between two frames. Implementing this algorithm gives us promising results, with a definite scope for improvement. As an alternative and additional experiment, we also implement the img2img Stable Diffusion" algo- rithm to generate images similar to an initial image fed to the model. We observe in Figure 10 that there are some variations based on the noise in- jected in the diffusion process, it is not varied and temporally consistent. We also see that Vixe2 Zero causes errors between two frames as shown in Figure 10.
We do this by employing Whisper (Radford et al., 2022), a general-purposs speech recognition model which achieves state-of-the-an results by scaling the weakly supervised pre-training procedure, to provide the transcribed output of lyrics as well as its associated timestamps. Whisper performs well in understanding and splitting vocal sequences, at times clubbing two lines together if the gap be- tween them is small. We argue that this is benefi cial as sufficient time is given to visualise the lyrics in the video and we can manifest ViPE's context- learning capabilities too. The Whisper-large model, despite having 1550M parameters (compared to 769M for Whisper-medee medium) was chosen, as exact timestamps are required for the best possible audio- to-visual alignment.
B.4 Interpolation
For video generation, besides using zero-shot text-to-video generation, we establish a novel fimbedding-chunking strategy in the text latent man- ifold for text-to-image generation to incrementally interpolate the latent representations, thus provid- ing interesting visual transitions from one line the next.
We use spherical linear interpolation (slerp) to interpolate between two subsequent CLIP text em- beddings of shape (77,768) as well as two condi- tional latents which are denoised in the diffusion process. By smoothly interpolating between latents as highlighted in Equation 2 and recent works like (Bhunia al., 2023), we maintain visually ap- pealing and coherent transition between two lines instead of cutting from one distinct scene to an- other.
B.2 Visual Elaborations
We harness ViPE's ability to improve visual elabo- rations with an increased context size (Figure 8) in establishing visual narrative in text form. In our experiments, we use ViPE-M, with a context size of 7, to sample Visual Elaborations of each line, which serves as the input to the Stable Diffusion. While it is established how well ViPE performs on Visual Elaborations, it also provides meaning- ful outputs to visualise the music break sections, where there are no lyrics to sample from.
sterp(po.PL, P1, sin t)0 sin to
sin A po + sin 0 P1 (2)
However, this interpolation formulation does not account for semantic coherence in the text latent space. Before interpolating between Po and Pt, we argue that it is essential that interpolated embed- dings correspond to similar text. To reduce the risk of introducing non-coherent intermediate embed- dings, we implement a sliding window approach to align subsets of the text embeddings of pı reference Po. which we call Chunked Similarity Alignment.
For example, for Adele's s Skyfall, we input Sky- fall by Adele; musical intro and ViPE generates Adele playing the piano in front of a large crowd, which is easily visualisable. Towards the middle of the song, we can use the context of previous lines to establish a suitable prompt during musical breaks, such as Pop band performing on the streets of New York in a thunderstorm for Thunder by Imagine Dragons.
B.3 Text-to-Video
We rearrange p1 by selecting the window with the highest cosine similarity (we capture the seman- tic relation between words based on the alignment of their semantic orientations). Our approach helps preserve semantic coherence and continuity during
With the availability of well-trained text-to-image Stable Diffusion Models, (Khachatryan et al., 2023), we established the procedure for zero-shot text-to-video generation. Our approach follows the Text2Video- -ereo Diffusion pipeline (Khacha- tryan et al., 2023), which we tune to reduce the
hhtpp://wngggggge....//sssstttta img2img

---

interpolation, which is crucial considering the diver- sity of visual claborations that need be illustrated, as otherwise interpolating may lead to incoherent transitions between disparate texts. We showcase frame-to-frame results to compare traditional slerp interpolation and interpolation with Chunked Sim- ilarity Alignment, in Figure 9. We qualitatively show how aligning similar embeddings on the text manifold aids better interpolation the image manifold.
Creative Visual Elaborations
In this section, we present additional examples to demonstrate the extensive capabilities of ViPE in comprehending various non-literal expressions and producing credible visual elaborations accordingly. The results are shown in Figure 11 for Stable Dif- fusion and Figure 12 DALL-E While both models struggle to visualize complex textual inputs, ViPE excels in generating visually comprehensi- ble elaborations while maintaining the semantic integrity of the arbitrary textual prompts.

---

Figure 9: Comparison between interpolation methodologies: Chunked Similarity Alignment (urr)) Top) and normal slerp Interolltioo(Bottomm) We interpolate between A panda playing the guitar A man jumps into pool. Normal interpolation fails by being less smooth in transitions between frames, which is important during video generation as we normally use high fps rates (we use in the music videos we generate) and such variations between frames would make the interpolation noisy sequence. Normal interpolation also fails to align transitions as well our approach as can be shown the transition rate between the panda transitioning to man and also the floor transitioning a pool.
Figure 10: Comparison between frame-to-frame sequential image generation: Eext2Video-ZZrroKKaacaaa et al., 2023)(Top) and img2img Stable Diffusion(bottom)) generating images for the prompt Man walks towards a beautiful sunrise, looks into the distance, while visualising Adele's Skyfall. Both models have difficulty with temporal consistency and motion, while img2img has low variations between frames, Text2Video-Zero has high variations.

---

DEATH DAR DAGH NER dE DOKING
knocking
spoper krocking dirnly houne, riid lnng bookground
Jann heppy being skone
bedy dancing inggg lriing room uncer 1i00
RS ANTUI TT_CACHNE Y MUChEAHI CATLL DIRIOM
ovnt rigust all Hhow information
cersinn reaking ber baad in naggnry snrounnee unded stedis of papere and books
fortary colorid, spple, siglarn
A-heamy thy with raidoos-aziored apple, rolann taking and apple tooiing the
Figure 11: Qualitative evaluations using Stable Diffusion and without ViPE's visual elaborations. left column in each sub-figure displays the prompt and Stable Difussion's output, while the other columns show ViPE's interpretations and the resulting image.
sought
The meboyy carsed throgh the an roles painting a woat picaus of longing
Achoi ongelo singing r.a hanwerly bap play backgsond
Acbor angele sing iha hvewarly playingi bediground
inarhiredd
Figure 12: Qualitative evaluations using DALL.E with and without ViPE's visual elaborations. The left column each sub-figure displays the prompt and the image generated by DAL.LB-22 while the other columns show ViPE's interpretations and the resulting images.


Reviews:
[{"review_id": "4ogENySSyZ", "reviewer": "Reviewer_9QUY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nWe sincerely appreciate the time and effort you invested in reviewing our work. Your insightful comments have contributed to the enhancement of our approach. Here, we address the raised points: \n\n \n### Reliance on GPT3.5\nWe acknowledge that the performance of ViPE is influenced by GPT3.5. However, we would like to emphasize that this reliance on ChatGPT is mitigated by many factors that contribute to the significance of ViPE: \n\n- Most importantly, our novel approach of symbolic knowledge distillation results in a more robust model compared to ChatGPT which enhances the reliability and applicability of ViPE. \n\n- ViPE effectively reduces the need for human labor. This becomes apparent when comparing the LyricCanvas dataset's substantial size (10M) with the current HAIVMet dataset (less than 7K), which relies heavily on human experts. \n\n- An essential point to note is ViPE's efficiency in terms of computational power and memory usage. ViPE's impressive efficiency for figurative language visualization is evident as it employs __507__ times fewer parameters than ChatGPT, showcasing its practical feasibility and relevance. \n\n- The rapid advancements observed in large language models, such as ChatGPT, are indeed promising. Our work stands out by being the first to showcase the robustness of symbolic knowledge distillation for figurative language visualization. This paves the way for future improvements of ViPE by the research community. \n\n \n\n### Difference between using ViPE and ChatGPT\n\n- ViPE is more robust than ChatGPT, we showed that through various evaluation techniques. \n\n- ViPE is open-source, promoting collaboration, while ChatGPT is available exclusively through payment, limiting its accessibility to a broader audience. \n\n- ViPE demonstrates its practicality by being resource-efficient, with a modest size of just a few Gigabytes. This characteristic opens doors for researchers to explore and expand upon ViPE's capabilities. In contrast, the development of ChatGPT remains confined to a handful of major tech companies.  \n\n### Post-ViPE text-to-image Evaluation\nYou are right, the objective is to enhance text-to-image generation. For the extrinsic evaluation, __we do indeed__ conduct __post-ViBE__ evaluation. For example, the retrieval evaluation involves the following steps: \n\n- For a given metaphor from the HAIVMet dataset, we generate a visual elaboration using ViPE, and ChatGPT \n\n- From the generated visual elaboration, we generate images using Stable Diffusion \n\n- For an __end-to-end evaluation__, we conduct metaphor-image retrieval. That is, given a metaphor, retrieve one of its corresponding images and vice versa. As reported in the first two rows of Table 2, this is a Post-ViPE evaluation as you have defined. \n\n- We also conduct a retrieval evaluation between the generated visual elaborations and the images, which are denoted as Caption_sz and Caption_ft in Table 2.  \n\nMoreover, to strengthen our evaluation toolkit, we have conducted a __user study__ involving 30 native English-speaking participants aged between 20 and 40 for a comprehensive end-to-end assessment as follows: \n\n- __Data preparation:__ From the HAIVMet dataset, we randomly selected 60 metaphors. For each metaphor, we generated visual elaborations using ChatGPT, ViPE, and added the human expert elaborations from HAIVMet. Subsequently, we employed Stable Diffusion to generate corresponding images from these visual elaborations. \n\n- __Experiment:__ The experiment involved presenting participants with a metaphor alongside three images generated from prompts provided by human experts (HAIVMet dataset), ChatGPT, and ViPE. Their task was to choose the image that best represented the metaphor's meaning. \n\n- __Findings:__ Our findings dovetail well with the results in our paper. Participants favored images from __human experts 38.67%__ of the time, followed by __ViPE's images at 33.61%, and ChatGPT's at 27.72%__. These results validate ViPE's superiority over ChatGPT and its competitive performance with human experts. We eagerly add these insights to our paper, inspiring further research to bridge the gap toward human-level expertise. \n\nThank you again for your comments and suggestions, we hope our responses and new experiments have effectively addressed the concerns you raised. \n\nBest, \n\nThe Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors introduce the first automated model for visualizing figurative expressions in text-to-image systems, which I believe is a significant advancement for the NLP community. Furthermore, they have released LyricsCanvas, a dataset comprising 10 million lines of lyrics paired with visual elaborations, marking another notable contribution.\n\n\n\n\n", "reasons_to_reject": "1. The Visual Elaborations collected rely heavily on GPT3.5, which serves as the performance benchmark for the dataset. How does a model trained on this dataset differ from directly employing GPT3.5?\n2. I remain unconvinced by the evaluation methods, particularly the extrinsic evaluation. Given that the objective is to enhance text-to-image generation, why not evaluate the images produced post-ViPE generation? An open-source text-to-image generation toolkit, like stable diffusion, could be utilized for this purpose.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces ViPE: Visualise Pretty-much Everything, a tool for generating visual descriptions of any arbitrary piece of text, including figurative and non-literal expressions. It eliminates the need for human annotations or images with metaphorical contents. Authors offers a series of lightweight and robust language models that have been trained on a large-scale set of lyrics with noisy visual descriptions that represent their implicit meaning. Authors also evaluate the proposed methods with both Intrinsic and extrinsic evaluation.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mhSjuk847e", "reviewer": "Reviewer_eBD7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nWe sincerely appreciate the time and effort you invested in reviewing our work. Your insightful comments have contributed to the enhancement of our approach. Here, we address the raised points: \n\n### Evaluation Limitations\nWe do acknowledge that evaluating the task at hand is none trivial as also stated in the limitation section of our paper. To strengthen our evaluation toolkit, we have conducted a user study involving 30 native English-speaking participants aged between 20 and 40 for a comprehensive end-to-end assessment as follows: \n\n- __Data preparation:__ From the HAIVMet dataset, we randomly selected 60 metaphors. For each metaphor, we generated visual elaborations using ChatGPT, ViPE, and added the human expert elaborations from HAIVMet. Subsequently, we employed Stable Diffusion to generate corresponding images from these visual elaborations. \n\n- __Experiment:__ The experiment involved presenting participants with a metaphor alongside three images generated from prompts provided by human experts (HAIVMet dataset), ChatGPT, and ViPE. Their task was to choose the image that best represented the metaphor's meaning. \n\n- __Findings:__ Our findings dovetail well with the results in our paper. Participants favored images from __human experts 38.67%__ of the time, followed by __ViPE's images at 33.61%__, and __ChatGPT's at 27.72%__. These results validate ViPE's superiority over ChatGPT and its competitive performance with human experts. We eagerly add these insights to our paper, inspiring further research to bridge the gap toward human-level expertise. \n\n\n### Reliance on GPT3.5\nWe agree that the performance of ViPE is influenced by GPT3.5. However, we would like to emphasize that 1) this reliance on ChatGPT is mitigated by many benefits including the ones you have also laid out. 2) The rapid advancements observed in large language models, such as ChatGPT, are indeed promising. Our work stands out by being the first to showcase the robustness of symbolic knowledge distillation for figurative language visualization. This paves the way for future improvements of ViPE (e.g., fine-tuning on small humanly annotated data) by the research community. \n\n \n### Potential Bias Issues\nThe problem of underrepresenting nuanced cultural differences and over dominance of certain ethnicities and gender types are apparent in many vision-language tasks comprising mainly English datasets (e.g., text-to-image synthesis). While our model may also exhibit such biases, its ability to generate multiple interpretations for a single input (e.g., a metaphor) is very promising. As an illustration, consider the range of plausible interpretations ViPE offers for the term 'ambiguity':  \n\n1. a person lost in a maze holding a map and looking confused \n\n2. a bunch of tangled cables with similar colors  \n\n3. a complicated maze of mirrors, reflecting a distorted view of a woman’s face \n\n4. a confused chameleon hiding behind a leaf on a sunny day. \n\nBy generating various valid interpretations, ViPE significantly enhances its capability to encompass the spectrum of cultural and subjective differences inherent in such expressions. \n\n\nThank you again for your comments and suggestions, we hope our responses and new experiments have effectively addressed the concerns you raised. \n\nBest, \n\nThe Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tInnovation: The paper presents a novel approach to a challenging problem in NLP, offering a unique solution for visualizing abstract and figurative language.\n2.\tRich Dataset Creation: The introduction of the LyricCanvas dataset, a significant contribution to the field, enables further research and development.\n3.\tPractical Applications: The paper outlines real-world applications of ViPE, making it relevant to both academia and industry.\n4.\tComprehensive Evaluation: The paper provides compelling evidence of ViPE's effectiveness, robustness, and competitive performance compared to existing models.\n5.\tOpen Source Contribution: The availability of the source code and dataset promotes transparency and encourages further research and collaboration. The attached publicized videos effectively present the novelty of the proposed model.\n", "reasons_to_reject": "1.\tPotential Bias Issues: The paper acknowledges that ViPE may exhibit biases similar to GPT2 in job stereotypes, gender, and ethnicity distributions. The mitigation of these biases is not thoroughly addressed.\n2.\tEvaluation Limitations: The choice of evaluation metrics and specific datasets may not fully capture the nuances of human figurative expressions. A more diverse and perhaps new evaluation measures and datasets might strengthen the assessment. Also more similar models should be meanwhile compared and evaluated to prove ViPE’s robustness.\n3.\tLack of Detailed Analysis on Limitations: While the paper mentions limitations, a more in-depth analysis of potential pitfalls or challenges in implementing ViPE might have provided a more balanced view.\n4.\tPotential Over-Reliance on Existing Models: ViPE's reliance on GPT3.5 for generating visual elaborations might raise concerns about its independence and the potential need for further innovation in this area.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "Topic:\n\nThis paper introduces ViPE (Visualise Pretty-much Everything), a novel approach for visualizing figurative expressions in text-to-image models. The paper addresses the challenge of synthesizing meaningful images for abstract and figurative expressions, a task with which current state-of-the-art models like DALL.E 2 and Stable Diffusion struggle. (Multimedia model, NLP and CV, Multimedia dataset)\n\nMain Contributions:\n\n1.\tIntroduction of ViPE: A collection of lightweight and robust language models trained on a large-scale set of lyrics with noisy visual descriptions. These models are capable of transforming any arbitrary text into a visualizable description.\n\n2.\tCreation of the LyricCanvas Dataset: A comprehensive dataset containing 10 million lines of lyrics paired with visual elaborations generated by GPT3.5. This dataset enables the training of powerful language models for visualizing figurative language.\n\n3.\tRobust Performance: Evidence demonstrating that ViPE is more robust than GPT3.5 in synthesizing visual elaborations. It also exhibits an understanding of figurative expressions comparable to human experts.\n\n4.\tVersatility and Applications: The paper showcases the potential of ViPE across various domains, including music video generation, style transfer, creative writing, and more.\n5.\tOpen Source Availability: The source code, pre-trained ViPE models, and the LyricCanvas dataset are made publicly available.\n", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JGfSqbhkh7", "reviewer": "Reviewer_Wr1n", "summary": "", "questions": "A) How did you handle cases where the input text is not figurative or abstract, but literal or concrete? Did you generate visual elaborations for such cases as well, or did you filter them out or use a different approach?\n\nB) How did you measure the semantic proximity and visual perceptibility of the generated visual elaborations? Did you use any metrics or tools to quantify these aspects, or did you rely on human judgment or intuition?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nWe sincerely appreciate the time and effort you invested in reviewing our work. Your insightful comments have contributed to the enhancement of our approach. Here, we address the raised points: \n\n\n### Training on noisy data generated by LLM\nWe agree that LLM including Chatgpt might generate noisy labels. However, incorporating synthetic noise has been demonstrated to enhance the robustness of machine translation, a task closely aligned with ours, as seen in studies such as [Improving Robustness of Machine Translation with Synthetic Noise](https://aclanthology.org/N19-1190) (Vaibhav et al., NAACL 2019) and [Toward Robust Neural Machine Translation for Noisy Input Sequences](https://aclanthology.org/2017.iwslt-1.13) (Sperber et al., IWSLT 2017). Consequently, by training on noisy synthetic data, we anticipate a similar enhancement of our model's robustness which we confirm with various evaluation methods including a new human evaluation mentioned below. \n\n### Human Evaluation\nThanks for pointing this out, in response, we have conducted a user study involving 30 native English-speaking participants aged between 20 and 40 for a comprehensive end-to-end assessment. \n\n- __Data preparation:__ From the HAIVMet dataset, we randomly selected 60 metaphors. For each metaphor, we generated visual elaborations using ChatGPT, ViPE, and added the human expert elaborations from HAIVMet. Subsequently, we employed Stable Diffusion to generate corresponding images from these visual elaborations. \n\n- __Experiment:__ The experiment involved presenting participants with a metaphor alongside three images generated from prompts provided by human experts (HAIVMet dataset), ChatGPT, and ViPE. Their task was to choose the image that best represented the metaphor's meaning. \n\n- __Findings:__ Our findings dovetail well with the results in our paper. Participants favored images from __human experts 38.67%__ of the time, followed by __ViPE's images at 33.61%__, and __ChatGPT's at 27.72%__. These results validate ViPE's superiority over ChatGPT and its competitive performance with human experts. We eagerly add these insights to our paper, inspiring further research to bridge the gap toward human-level expertise. \n\n \n### Appropriateness and safety of the generated visual elaborations\nLanguage models such as GPT2 might indeed generate inappropriate content. We have therefore constructed safe training data by introducing an appropriateness criterion guiding the generation of visual elaborations within our training data, as also highlighted in our paper: \"__Appropriateness:__ Some lyrics contain inappropriate content, so generated output should not explicitly describe such content\". [__Footnote__: We automatically discarded those lyrics that were not processed by the system due to inappropriate content]. \n\n- __New profanity and offensive language evaluation:__ To measure the appropriateness of ViPE's output, we conducted the following experiment. Using the [Alt-profanity-check library](https://pypi.org/project/alt-profanity-check/), we first measured the profanity score (inappropriate/offensive language) of the lyrics in the valuation set (around 1M line of lyrics). We then measured the profanity scores of the generated visual elaborations from ChatGPT and ViPE. In addition, we prompted the pre-trained GPT2 model with the lyrics and generated new text (not necessarily a visual elaboration). Subsequently, we measured the profanity score for the GPT2’s output. \n\n- __Results:__ Reported in the table Below, the mean and standard deviation of the lyrics's scores are (0.1778, 0.2615), while these numbers are still high for the pre-trained GPT2, we can see that __Both ViPE and ChatGPT results in scores close to zero__. In the paper, we will add a new figure showing how each of the three models behaves for inputs with increasing profanity scores.  For ViPE and ChatGPT, two closely positioned horizontal lines are observed, while GPT2’s plot shows a steady slope. Overall, these findings confirm that ViPE does not output inappropriate responses and closely matches ChatGPT's profanity scores.  \n\n| Model   | Mean   | Standard Deviation  | \n|---------|--------|---------------------| \n| Lyrics  | 0.1778 | 0.2615              | \n| GPT2-M  | 0.1376 | 0.2315              | \n| ChatGPT | 0.0812 | 0.0968              | \n| ViPE-M  | 0.0962 | 0.102               | \n\n \n\n### Cultural or contextual differences in interpreting figurative expressions\nVisualizing and interpreting figurative expressions is a very challenging task, primarily due to the substantial influence of subjectivity and the legitimacy of diverse interpretations. Moreover, the problem of underrepresenting nuanced cultural differences is apparent in many vision-language tasks comprising of mainly English datasets such as text to image synthesis. While our model may also exhibit such biases, its ability to generate multiple interpretations for a single input (e.g, a metaphor) is very promising. As an illustration, consider the range of plausible interpretations ViPE offers for the term _ambiguity_: \n\n1. A person lost in a maze holding a map and looking confused \n\n2. A bunch of tangled cables with similar colors  \n\n3. A complicated maze of mirrors, reflecting a distorted view of a woman’s face \n\n4. A confused chameleon hiding behind a leaf on a sunny day. \n\nBy generating various valid interpretations, ViPE significantly enhances its capability to encompass the spectrum of cultural and subjective differences inherent in such expressions. \n\n \n\n### Questions \n\n___Visual elaboration for concrete phrases:___ We did generate visual elaborations for concrete phrases with the same pipeline. While the generated elaboration in such cases conveys the main essence of the input, it typically encompasses more details about the scene. For instance, _'I went for a walk’_ might be converted into _'A man is walking in a park full of trees, the sun is shining’_.  By training on both concrete and figurative phrases, ViPE is able to construct visual elaborations from a wide range of keywords. For instance, for the input _‘fantasy, brave, apple’_, ViPE generates _`A brave knight holding a giant apple, standing in front of an enchanted castle'_ \n\n__Semantic proximity and visual perceptibility:__ \n- Semantic Proximity (SP) measures how well the generated visual elaboration v_i represents the meaning of the tweet t_i, determined by the final classification accuracy on D_v. In other words, once the emotionally charged tweets have been converted into visual elaborations, if the information about the emotion is well represented, then the classification score (using BERT) is higher. \n\n- Visual Perceptibility (VP) assesses the visualisability of the visual elaboration v_i by computing the cosine similarity between the CLIP embeddings of v_i and its corresponding generated image I_i by Stable Diffusion. In other words, if the generated visual elaboration is accurately visualizable, it results in a high-quality image and hence a high CLIP score with the visual elaboration. \n\nThank you again for your comments and suggestions, we hope our responses and new experiments have effectively addressed the concerns you raised. \n\nBest, \n\nThe Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper are:\n\n+ It presents a novel and creative approach to generate visual elaborations from arbitrary text input, which can assist text-to-image models in visualising figurative and abstract phrases.\n+ It introduces a large-scale dataset of lyrics and visual descriptions, which can be used for fine-tuning powerful language models or for other downstream tasks.\n+ It evaluates the model on various benchmarks and applications, showing its superior performance and versatility compared to existing methods.\n\nThe main benefits to the NLP community if this paper were to be presented at the conference or accepted into Findings are:\n\n+ It would inspire new research directions on figurative language visualisation, which is an under-explored but important topic.\n+ It would provide a useful resource for researchers and practitioners who want to generate visual elaborations from text or enhance their text-to-image models.\n+ It would showcase the potential of symbolic knowledge distillation as a general technique for transferring knowledge from large-scale noisy data to lightweight models.", "reasons_to_reject": "Some possible weaknesses of this paper are:\n\n+ It relies on an LLM to generate noisy visual descriptions, which may introduce errors or biases that affect the quality of the final output. For example, the LLM may generate inconsistent or irrelevant descriptions for some lyrics, or may fail to capture the nuances or emotions of the original text.\n+ It does not conduct a human evaluation or user study to assess the subjective aspects of the generated visual elaborations, such as creativity, diversity, or emotional impact. It would be useful to collect feedback from human judges or users on how they perceive and appreciate the visual elaborations, and whether they find them helpful or enjoyable.\n\nSome possible risks of having this paper presented at the conference or accepted into Findings are:\n\n+ It may raise ethical concerns about the use of lyrics as a source of data, especially if they contain sensitive or offensive content. The paper does not discuss how it handles such cases, or whether it applies any filtering or moderation techniques to ensure the appropriateness and safety of the generated visual elaborations.\n+ It may overlook some limitations or challenges of figurative language visualisation that are not addressed by the paper. For example, the paper does not consider the cultural or contextual differences in interpreting figurative expressions, or the ambiguity or uncertainty that may arise from generating multiple visual elaborations for the same text.", "questions_to_authors": "A) How did you handle cases where the input text is not figurative or abstract, but literal or concrete? Did you generate visual elaborations for such cases as well, or did you filter them out or use a different approach?\n\nB) How did you measure the semantic proximity and visual perceptibility of the generated visual elaborations? Did you use any metrics or tools to quantify these aspects, or did you rely on human judgment or intuition?", "paper_topic_and_main_contributions": "This paper is about ViPE, a set of robust and lightweight language models that can generate visual elaborations from arbitrary text input, such as lyrics, tweets, or figurative expressions. Visual elaborations are textual descriptions of visual scenes that convey the intended meaning of the original text.\n\nThe paper addresses the problem of how to enhance the visualization of figurative and non-literal expressions in any text-to-image model. The main contributions that it makes towards a solution or answer are:\n\n+ Introducing a dataset available for generating visual elaborations, called LyricCanvas, which contains about 10 million samples of lyrics and synthetic visual descriptions generated by a large language model (LLM).\n+ Proposing a symbolic knowledge distillation method to train a robust and powerful model (ViPE) by fine-tuning lightweight language models on LyricCanvas.\n+ Demonstrating the effectiveness and versatility of ViPE in various applications, such as image-text retrieval, emotion visualization, music video generation, and figurative language understanding.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nccKg7Me2T", "reviewer": "Reviewer_jo7U", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nWe sincerely appreciate the time and effort you invested in reviewing our work. Your insightful comments have contributed to the enhancement of our approach. Here, we address the raised points: \n\n \n\n### The Technical Contribution\nAs highlighted in our paper, \"ViPE enables accessible solutions for complex research questions\". We believe that simplicity and effectiveness should be prioritized over complexity when both yield equivalent outcomes. Given ViPE's clear advantages over ChatGPT in terms of computational efficiency (__175B vs 345M__ parameters) and performance for figurative language visualization, we view the simplicity of our approach as a notably positive and elegant solution. \n\n \n### Direct Human Evaluation\nThanks for pointing this out, in response, we have conducted a user study involving 30 native English-speaking participants aged between 20 and 40 for a comprehensive end-to-end assessment. \n\n- __Data preparation:__ From the HAIVMet dataset, we randomly selected 60 metaphors. For each metaphor, we generated visual elaborations using ChatGPT, ViPE, and added the human expert elaborations from HAIVMet. Subsequently, we employed Stable Diffusion to generate corresponding images from these visual elaborations. \n\n- __Experiment:__ The experiment involved presenting participants with a metaphor alongside three images generated from prompts provided by human experts (HAIVMet dataset), ChatGPT, and ViPE. Their task was to choose the image that best represented the metaphor's meaning. \n\n- __Findings:__ Our findings dovetail well with the results in our paper. Participants favored images from __human experts 38.67%__ of the time, followed by __ViPE's images at 33.61%__, and __ChatGPT's at 27.72%__. These results validate ViPE's superiority over ChatGPT and its competitive performance with human experts. We eagerly add these insights to our paper, inspiring further research to bridge the gap toward human-level expertise. \n\n\nThank you again for your comments and suggestions, we hope our responses and new experiments have effectively addressed the concerns you raised. \n\nBest, \n\nThe Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The visualize everything idea is exciting and of great potential.\n\n- The released resources would be helpful for text-to-image applications and studies.\n\n", "reasons_to_reject": "- The technical contribution, i.e., generating a dataset with ChatGPT and then conducting distillation, is weak.\n\n- The evaluation results are mostly conducted implicitly. A  human evaluation on the triple of metaphor, visual elaboration and generated image quality is needed to fully convince the readers (small scale would be enough).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a model ViPE capable of transforming metaphorical phrases or abstract concepts to visual elaborations which can be used to instruct DALLE.2 for a visualization.\n\nThe authors propose to collect a large-scale lyric dataset as a proxy of figurative language, then instruct ChatGPT to provide noisy visual elaborations given the lyrics. Finally, the lyric-vision elaboration pair distills a small model (GPT2-small & GPT2-medium).\n\nExperimental results show that the trained models can generate better visual elaborations ( measured by the BLIP recall score) than GPT 3.5 and human experts. Further results on the tweet emotion classification dataset, show that the visual elaboration generated by ViPE results in higher classification accuracy and better alignments with the synthetic images.\n ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce ViPE (Visualise Pretty much Everything), an innovative suite of text-to-image models that emphasize the depiction of figurative and non-literal expressions. Key outcomes of ViPE encompass the lyrics-based pretraining corpus named 'LyricCanvas' and an assortment of pretrained text-to-image foundational models.

There is a unanimous consensus among reviewers that ViPE is both exciting and original. It ventures into the visualization of figurative language, a domain that, though crucial, remains largely uncharted, and does so with a fresh methodology.
Decision: Accept-Main