Impressions: Visual Semiotics and Aesthetic Impact Understanding
Authors: Julia Kruk
Caleb Ziems
Diyi Yang
Venue: EMNLP 2023 Main (2023)
Keywords: Visual Semiotics
Stylistic Analysis
Computational Aesthetics
Image Captioning
Multimodal Datasets
TL;DR: Present a dataset for aesthetic impact understanding and modeling human impressions of images. 'Impressions Dataset' is created to resolve the connotation of visual signifiers by analyzing an images perlocutionary force.

Abstract: Is aesthetic impact different from beauty? Is visual salience a reflection of its capacity for effective communication? We present Impressions, a novel dataset through which to investigate the semiotics of images, and how specific visual features and design choices can elicit specific emotions, thoughts and beliefs. We posit that the impactfulness of an image extends beyond formal definitions of aesthetics, to its success as a communicative act, where style contributes as much to meaning formation as the subject matter. We also acknowledge that existing Image Captioning datasets are not designed to empower state-of-the-art architectures to model potential human impressions or interpretations of images. To fill this need, we design an annotation task heavily inspired by image analysis techniques in the Visual Arts to collect 1,440 image-caption pairs and 4,320 unique annotations exploring impact, pragmatic image description, impressions and aesthetic design choices. We show that existing multimodal image captioning and conditional generation models struggle to simulate plausible human responses to images. However, this dataset significantly improves their ability to model impressions and aesthetic evaluations of images through fine-tuning and few-shot adaptation.

Impressions:
Understanding Visual Semiotics and Aesthetic Impact
Julia Kruk
Caleb Ziems
Diyi Yang
Georgia Institute of Technology, Stanford University
jkruk3@gatech.c edu, (cziems, diyiy (ayiyy8stan @staaford.edd
Abstract
The Impressions Dataset
aesthetic impact different from beauty? Is visual salience a reflection of its capacity for effective communication? We present Impres- sions,' a novel dataset through which to investi- gate the semiotics of images, and how specific visual features and design choices can elicit specific emotions, thoughts and beliefs. We posit that the impactfulness of image tends beyond formal definitions of aesthetics, its success as communicative act, where style contributes as much to meaning forma- tion as the subject matter. However, prior im- age captioning datasets not designed to power satee-of-the-ar architectures to model potential human impressions or interpretations of images. To fill this gap, we design notation task heavily inspired by image anal- ysis techniques in the Visual Arts collect 1,440 image-caption pairs and 4,320 unique an- notations exploring impact, pragmatic image description, impressions, and aesthetic design choices. We show that existing multimodal image captioning and conditional generation models struggle simulate plausible human responses to images. However, this dataset sig- nificantly improves their ability to model pressions and aesthetic evaluations of images through fine-tuning and few-shot adaptation.
Seeahing
gymte's ascthelii oirctens
Ahra Peprechigg
What statest clereees coclibast impremabit
Figure 1: Photographs in Impressions contain wide variety of styles and aesthetic elements. To understand the way these features contribute to visual semiotics and the perlocutionary force of the image, we gather human impression annotations inspired by image analysis tech- niques in visual arts and media studies.
public policy (Barnhurst and Quinn, 2012), journal- ism (Hullman and Diakopoulos, 2011), and com- munication more broadly. Current technological limitations have foreed most studies to rely on man- ual analysis of connotation smaller seales (see Hellmueller and Zhang, 2019 and others). While advances in image captioning have inspired Auto- matic Visual Content Analysis methods (Araujo et al., 2020), which excel at extracting objects, be- haviors, and other denotational content, they often lack the awareness of how visual symbols connote non-literal meanings. This study presents a cu- rated collection of image-caption data for training new systems that can understand connotation. particular, we make explicit the image's perlocu- tionary force, or the meaning that is perceived by
1
Introduction
"We never look at just one thing; we are always looking at the relation between things and ourselves."
John Berger (1972)
Images are rich objects for the semiotic study of connotation, as well as downstream objects for sci- entific study, including affect (Panda et al., 2018), framing (Uluçay and Melek, 2021; Christiansen, 2018; Powell et al., 2015) advertising and persua- sion (Ye et al., 2019; Joo al., 2014), with ram- ifications for marketing research (Oswald, 2012),
'https://g' thub.com wwt..cco//SATTTT T-NTPP Impressions

---

an observer, well as the aesthetic elements and concrete visual features that inspire these specific observer impressions (see Figure 1).
understanding of an audience's likely interpreta- tions of visual media could empower multimodal dialogue systems, where images can be used combination with text to communicate. Further- more, a better understanding of the connotation of visual symbols can enable text-to-image archi- tectures to capture more nuance in language, thus rendering more salient compositions and styles in the output.
The image captioning datasets used to train multimodal architectures such as BLIP (Li et al., 2022). M2 Transformer (Cornia et al., 2020), and Flamingo (Alayrac et al., 2022), contain terse and reductive captions that describe their visual counter- parts. This is because annotators were encouraged to write single-sentence captions that focused on the most concrete elements of image: objects, entities, shapes, colors, etc. For tasks such as ob- ject detection and scene segmentation, this is the intuitive approach. But as transformer-base multi- modal architectures grow more proficient, training solely on datasets such as COCO Captions (Chen et al., 2015), Conceptual Captions (Sharma et al., 2018), and WIT (Srinivasan et al., 202 will in- hibit models' ability to reason about the semiotics of images. This motivates our Impressions dataset, which captions images with semiotic and aesthetic elements that ground viewers' subjective percep- tions (see Figure 2)
To this end, we contribute the following:
The Impressions dataset, multimodal bench- mark that consists of 4,320 unique annota- tions over 1,440 image-caption pairs from the photography domain. Each annotation explores (1) the aesthetic impactfulness of a photograph, (2) image descriptions in which pragmatic inferences are welcome, (3) emo- tions/thought/bellefi that the photograph may inspire, and (4) the aesthetic elements that clicited the expressed impression.
We show that state-of-the-art transformer- based architectures struggle model human impressions and resolve aesthetic elements that sit at the core of the connotation.
As highlighted by Berger (1972), an observer cannot view an image without imposing upon values, beliefs, and expectations that are highly grounded in their understanding of cultural norms and social cues. Humans instinctively make as- sumptions about emotions, actions, and relation- ships setting, space, and circumstance past, present, and future. Furthermore, these inferences are not solely dependent on the concrete objects and entitics in an image. Photojournalists often manipulate composition, lighting, exposure, and camera angles to direct an audience's perception of the subject matter. Aesthetic design choices shape the meaning of an image, and can communicate just as much the concrete elements depicted. To better understand this phenomenon, Barthes (1972) applied the principles of denotation and connota- tion to visual symbols in photography. Denotation is the meaning carried by the literal depictions in the image. Whereas connotation is the additional meaning assigned to a signifier that can be depen- dent on (1) rules and conventions the observer is familiar with, (2) visual and aesthetic elements of the image, or (3) cultural implication of the de- piction as opposed what is really there. Few datasets have ventured to encourage pragmatic in- ferences on visual scenes and collect audience in- terpretations and impressions of images. Yet am
By leveraging the Impressions dataset, these architectures attain the ability through few- shot learning or fine-tuning. In human eval- uation task, the greatest improvements are on GIT (Wang et al., 2022). BLIP (Li et al., 2022), and OpenFlamingo (Alayrac et al., 2022), for which annotators preferred fine- tuned/udapted model generations over 80% of the time across all caption categories.
We release an additional dataset of 50,000 image and leading paragraph pairs collected from the New York Times (NYT) official API for unsupervised exploration of visual semi- otics and aesthetic element contribution to the coded iconic.
2 Related Work
Visual Semiotics and Communication This work is heavily inspired by the discussion of the semiotics of visual media in Barthes (1972), specif- ically how connotation and denotation of signifiers are extended from linguistics to visual studies. Ad- ditionally, in focusing on gathering human impres- sions we analyze the perlocutionary force of im- ages (their meaning as perceived by an audience).

---

Captions
Conceptual Captions
Empr
Centeell Ousteey unttr
Wilad Mecooe Trom 2011
Ahome cumving irge lest offo too people urting on
Cancapaal Capalie, 2og
Coprior: wablee
Artonis
AVA
dimcio The hoom ai caria
lonn
Iaaegr
Conneak
rekens
ans teme ai di
fad wead
The Embeda al anee of Moreeet
grveed and ao scftee the
back-
What anotheric sle elements contributte your improssiion
imaga?
tap dhe
fo sinler pane muner snif aina harlwway
boundd Throoppe whish
Figure 2: Comparison of Impressions dataset on the right, to COCO (Chen et al., 2015), Conceptual Captions (Sharma al., 2018), Artemis (Achlioptas et al., 2021), and AVA (Murray al., 2012) datasets in respective order on the left. Each visualization is taken directly from the cited works. In contrast to existing benchmarks, Impressions contains rich commentary on viewer perception that is not constrained to emotion, and a discussion of aesthetic elements that is grounded in the impressions that inspire.
Perlocution is among the types of speech acts, or communicative acts, identified by Austin (1962).
impression in this way, we aim to resolve a better understanding of social queues, cultural norms, and popular connotations of visual symbols through an- notator responses. A breath of prior work has deavored to generate pragmatic image captioning through context-agnostic supervision (Vedantam et al., 2017), listener and speaker models (Andreas and Klein, 2016), relaxation of language encoding (Zarrieß et al., 2021), and explicit image-caption pair annotation (Tsvilodub and Franke, 2023), In- terestingly, datasets such as Flickr30k have been shown to unintentionally resolve annotator infer- ence (Van Miltenburg, 2016), yielding more prag- matic and biased captions than intended.
Image Captioning Most image captioning tasks are framed with literal descriptions of the con- stituent objects, such as their behaviors, quantities and attributes. In the popular COCO (Chen al., 2015), WTT (Srinivasan et al., 2021b), and Concep- tual Captions (Sharma et al., 2018) benchmarks, these descriptions have a neutral, objective tone. Flickr30k (Young et al., 2014) takes it a step further, presenting crowd-sourced annotations of images depicting people going about everyday activities in effort to study the visual denotation of linguistic expressions. More recent work conditions image caption generation on non-neutral stylistie features like humor (Chandrasckaran et al., 2017), romance (Tan et al., 2022; Gan et al., 2017), sentiment (Li et al., 2021), or personality traits (Shuster et al., 2019; Park et al., 2017).
Aeshhetic Image Analysis Some prior work has focused on aesthetic image captioning and analysis through collecting comments aesthetic elements (Murray al., 2012; Jin al., 2019, 2022; Chang et al., 2017). These works approach style, design, and aesthetics from the perspective of beauty and visual appeal. The Impressions dataset was de- signed to ground the discussion of aesthetic ele- ments in the impressions they inspire, to capture the link between the visual signifier and the sig- nified. We also posit that aesthetic impact, the likelihood of media to attract and hold attention, goes beyond what is considered classically beau- tiful and extends to the communicative utility of aesthetic elements. In other words, beauty and aesthetic impact are correlated but not the same.
In a similar effort to our own, Achlioptas et al. (2021) built dataset for affective captions to me- dia in the visual arts domain. The most important difference between this work and our own is that we consider additional impressions beyond affect. Not every image is going to give the viewer emotive response but could alternatively inspire them to think or believe something. Most image captioning systems lack knowledge of symbolic iconography, cultural conventions, etc. (Lang and Ommer, 2018). By opening the question of human

---

Recent work has focused on empowering vision models to interpret and generate visual metaphors (Akula et al., 2023; Chakrabarty et al., 2023). Nu- merous datasets exist for art style (Khan et al., 2014; Florea et al., 2016), emotion (Mohammad and Kiritchenko, 2018), and aesthetic element clas- sification (Amirshahi et al., 2015; Murray et al., 2012; Datta et al., 2006), as well as visual question answering with art (Garcia al., 2020). There is active work on art classification and image descrip- tion in the field of digital art history (Lang and Ommer, 2018), 8). Although Impressions consists of photographic media, it also samples images from the creative field of photojournalism.
Photojournalism is the use of photographs to both interpret a visual scene and tell an impactful news story (Newton, 2013). Thus we anchor our collection around photojournalistic images from articles in the New York Times, a US media source with a longstanding reputation for quality (Teitz, 1999; Vise. 2011). Impressions contains 50,000 anchor images that we extracted from the publicly- accessible NYT API.
To introduce additional stylistic and artistic vari- ation, we use the Google search API to retrieve 3 semantically-relatee images for each NYT anchor image. Our search queries come from perturbations of the original NYT image captions. We use three different perturbation methods: (1) one-sentence summaries produced via BART conditional gener- ation model (Lewis et al., 2020), (2) extracting key entities via the spaCy 2 NER pipeline (Honnibal and Montani, 2017), and (3) constructing minimal dependency parse trees from anchor captions, de- fined by their subject, root verb, and any direct objects. Further information on caption perturba- tion methods can be found in Appendix A. After retrieving images for each perturbed query, we fil- ter results, using only the top 3 images whose VIT (Dosovitskiy al.) embedding similarity with the anchor image surpasses pre-defined threshold.
Visual Question Answering. Prior work has found great success in training and evaluating mod- els' understanding of visual features and concepts through Visual Question Answering (VQA; Antol et al., 2015; Goyal et al., 2017; Das et al., 2017). Where early efforts leveraged questions focused on counting, object detection and concrete features, more recent works explore knowledge-based tasks (Marino al., 2019; Zheng al., 2021), pragmatic answer explanation (Zellers et al., 2019; Marasović ul., 2020), visual common sense (Krishna et al., 2017), and explainable social intelligence (Zadeh et al,,,901 2019). Most notably, Liu et al. (2023) created a visual instruction tuning corpus with questions characterized by conversation, detailed description, and complex reasoning. The questions of our Im- pressions dataset closely resemble the complex reasoning category and could be used as an injec- tion into such VQA corpora improve impression generation and aesthetic impact reasoning.
3.2 Annotation
The aesthetic impact of an image is its ability to draw and hold attention, as well as inspire the viewer to feel or think something. The first step of our annotation pipeline is designed to identify the most impactful images. Annotators consider a set of 4 semantically-relate images (see $3.1) and rank them in the descending order of their aesthetic impact. Ties between two images are allowed, but to encourage more careful discrimination between images, we do not allow three and four-way ties. Images that have a mean aesthetic impact rank of 2 or greater are selected for free-form impression annotation. The variability of annotator rankings of aesthetic impact is characterized by an intraclass correlation coefficient (ICCIK) of 0.509, which demonstrates moderate annotator agreement.
3 Impressions Dataset
The Impressions dataset consists of 1,440 images and 4,320 distinet impression annotations. Each an- notation is composed of three free-form responses to questions addressing (1) image description, (2) viewers' impression of the image, and (3) aesthetic image evaluation grounded in the viewers' impres- sion. Additionally, we present aesthetic impact scores (in a range of to 4) for 3,450 images. In the section below we will describe our process for collecting, annotating, and analyzing the dataset.
The Impressions annotation process specially chosen to scaffold the discussion for individuals with very little to no experience media or visual arts as they consider the connotations of visual
3.1 Collection
To build a rich resource for the semiotic study of aesthetic impact, we first need a set of impactful images whose styles and aesthetic elements vary.
? AAn image was kept if cosine similarity was less than 0.8 and greater than 0.2 with all other images in the set. Further discussion of these thresholds can be found in Appendix B.

---

signifiers and the coded iconic (Barthes, 1972).
Distribution of Sentiment Intensity per Caption
or portrayed story, of each image. As such, the design is heavily inspired by the fields of visual arts (Dondis, 1974), art history (d' Alleva, 2005), and historical image cataloging (Zinkham, 2006). Our specific prompts were:
All Captions
Image Description Caplions
Impressiios coco
Densly
|11
Sallas
Image Perception Capfons
Aesthetic Evaluation Capfonns
Densir
1. Image description:
2-3 sentence response
8
to the question "What is happening in this image?" Unlike the captions in prior works, these descriptions can contain information not explicitly depicted but rather inferred.
Sentiment Compound
Compound
Figure 3:
Evaluation of the sentiment intensity of cap- tions in the Impressions dataset in reference captions in COCO. The spread of Impressions caption sentiment greater than that of COCO.
2. Image Perception: 2-3 sentence response to the question "What does this image make you think, feel, or believe?" In this way we aim to resolve the perlocutionary force of the im- age and its visual signifiers, in manner not constrained to emotion alone.
Sentiment Intensity Although we do not con- strain impressions to emotion, we do expect that the distribution of sentiment intensity in impression annotations will exhibit a wider spread than what we see in traditional image-captioning datasets. We observe this in direct comparison between Im- pressions and COCO captions in Figure 3. Notice that the distribution of our own Image Description annotations resembles that of COCO, which is ex- pected. Whereas Image Perception and Aesthetic Evaluation annotations produce far more variable distributions of sentiment intensities.
3. Image Aesthetic Evaluation: 2-3 sentence response to the question "What aesthetic el- ements contributed to your impression of the image?" This way we ground the discussion of aesthetic elements in the audience's im- pression, drawing the connection between the signifier and signified.
Annotators were recruited from both Amazon Mechanical Turk and UpWork. The authors' moti- vations for leveraging these platforms outlined in Appendix H. In total, we collect aesthetic impact rankings for 3,450 images, randomly sampled from our image collection process in $3.1. We collected free-form impressions for 1,440 of the most im- pactful images. A review of annotation instructions and examples can be found in Appendix G.
Subjectivity A viewer's impression of an image is inherently shaped by the viewer's 's beliefs, and expectations, and sociocultural context, which in- form their visual salience map and the connotations available to them. Thus we hypothesize that, com- pared with the literal descriptive captions, image impressions for single image will be more vari- able. This variance is not meaningless noise, we posit, but rather meaningful sociocultural variation that is regularly bounded by the semantic frame of the image. We therefore expect that the variance of image impressions will be low, despite being greater than that of literal COCO captions.
3.3 Quality Analysis
To highlight the distinguishing characteristics of the Impressions benchmark, we analyze sentiment, subjectivity, and concreteness of all impression annotations: the description, perception, and aes- thetic evaluation. There does not yet exist direct method for evaluating the richness and diversity of implied connotations in image caption, but we expect that, compared literal descriptive cap- tions, connotation-rich image impressions will ex- hibit: (1) increased variance in the distributions of sentiment intensity, (2) increased subjectivity, and (3) lower concreteness scoring of linguistic data.
In Figure 4, we see that the distributions for de- scription, perception, and aesthetic evaluation cap- tions in Impressions are wider than that of COCO. The median semantic variance for image descrip- tions (0.070), perceptions (0.088), and aesthetic evaluations (0.072) are all one order of magnitude greater than that of COCO captions (0.007). Still, given these small absolute values, we see that most image impressions are not highly variable.

---

Variation in Sentiment Intensity per Image
Distribution of Caption Concreteness Scores
Description Captions
Captions
Captions
Image Description Captors
Co0o
repress SEa
Densff
300
94
62
04
04
60
Var[Compound Scoro)
Var[Compound Score)
Imaga Panception Caplions
Aasthatic Evaluation Capliars
Perceplion Caplions
46 Caption Conorotness
14
VmmComm Scoro)
Caption Concrolness
Figure 4: The variance in sentiment intensity of impres- sion and aesthetic evaluation captions per image.
Figure 5: Distributions of caption concreteness scores in Impressions and COCO datasets.
Concreteness The concreteness of a word is the degree to which the concept it signifies is an entity that can be perceived. We can define the concrete- ness of a sentence to be the average concreteness of its constituent tokens. Since image impressions are based on connotations which derive from the viewer's subjective relationship with symbols and non-literal referents in the image, we hypothesize that impressions will be more abstract, or less con- crete, than traditional descriptive image captions.
for image-captioning separately on each of the dif- ferent Impressions data types (description, percep- tion, aesthetic evaluation). OpenFllamino was sim- ilarly few-shot adapted for VQA using 16 examples of one data type at a time. LLaVA was fine-tum meed for VQA on all annotations simultaneously via an instruction-tuninn training task.
Automatic Evaluation Table 1 displays the BLEU, METEOR, and ROUGE scores for each of these models. Note that the Impressions dataset contains more variation in human-produced refer- ences than traditional VQA and image- caage-capiooning benchmarks. This quality makes CIDEr-D (Vedan- tam et al., 2015), a concensus- evaluation of image descriptions, non-optimal for evaluating performance with this benehmark.
To test this hypothesis, we compute token con- creteness via the lexicon of Brysbaert et al. (2014), which contains conereteness scores for over 60k English lemmas, each on scale from 1 (highly ab- stract) (highly concrete; based on experience). We find that each set of Impression captions (de- scription, perception, aesthetic evaluation) less concrete than COCO captions, each with a statis- tical significance of p < 0.001 by t-test. Visually, this is apparent in the gaps between Impression and COCO concreteness distributions in Figure 5.
Human Evaluation Each fme-tuneddfeew-hhe
adapted model is evaluated on 300 image-prompt pairs. The prompts are divided evenly between questions targeting image description, perception, and aesthetic evaluation. For the models fine-tuned for image-captioning, 100 captions are generated per annotation category. This same process is repeated on the base pretrained GIT, BLIP and LLaVA architectures, and in the zero-shot setting with
4 Evaluation
To demonstrate the Impressions dataset's abil- ity to enable state-of-the-art image-captioning and conditionallgennrrtionn models to simulate human impressions and resolve impactful aesthetic ele- ments of images, we design the following exper- iment. GIT (Wang et al., 2022), BLIP (Li et al., 2022). OpenFlamingo (Awadalla al., 2023), and LLaVA (Liu al., 2023) are fine-tuned/ few-shot- adapted using a training set of 1,340 images and 4,020 impression annotations (each containing unique captions). GIT and BLIP were fine-tuned
This produced a final set of 300 generation pairs, where each pair contains one caption from the fine-tuned or adapted model and one from the base. We submitted the 300 gener- ation pairs to UpWork and requested that annota- tors identify which caption better simulates a po- tential human impression and identifies impactful aesthetic elements (see Appendix I). For each gen- eration pair we collect unique annotator votes

---

BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
R-1
R-2
R-L
GIT!
0.272
0.156
0.090
0.054
0.165
0.295
0.106
0.246
BLIPP
0.347
0.195
0.111
0.068
0.184
0.325
0.111
0.261
Open Flamingo (fs16))
0.500
0.275
0.159
0.098
0.231
0.318
0.093
0.246
LLaVA-7b-vll
0.349
0.181
0.089
0.006
0.279
0.294
0.079
0.202
Table I: Automatic token-overlap metrics for the four models fine-tuned on the Impressions dataset. These scores are computed on a scale between 0 and 1, over an evaluation set of 300 captions. Denotes models fine-tuned for image-captioning. Denotes models fine-tuned few-shot adapted for conditional generation.
and select the majority vote as the final evaluation. The results of this human evaluation task are dis- played in Table 2 in the form percentages at which annotators believed the model fine-tuned/ / adapted on Impressions produced a better caption.
were unable to convey human impressions or crit- ical aesthetic elements. Although OpenFlamingo was competitive in producing image descriptions that aligned with human interpretations, it failed to follow instructions zero-shot when prompted on im- age perception and aesthetic elements. Provided 16 examples, the few-shot adapted OpenFlamingo able to resolve human impressions reasonably well. Optimal performance was observed with 32 exam- ples or more. BLIP and OpenElaminnoo showcased the greatest improvement in aesthetic evaluation when fine-tuned/feww-hoo adapted on Impressions, producing preference scores of 88% and 100% re- spectively, GIT had the greatest improvements image impressions, with preference score of 92%. Qualitative comparisons of human impres- sions generated with different model architectures are displayed in Figure 6.
5 Results
The results in Table show that annotators pre- ferred outputs from models tme-tuneed/few-shol adapted with the Impressions dataset on average 76% of the time. The greatest improvements are observed with GIT, BLIP, and OpenFlamingo, for which annotators selected fine-tuned/adappee model generations over 80% of the time across all categories, but significantly more often for im- age perception and aesthetic evaluation generations. Marginal improvement seen with LLaVA VA fine- tuned on the Impressions dataset, with annotators selecting generations from the fine-tuned model on average 56% of the time, most notably 59% on aesthetic evaluation generations.
LLaVA Performance LLaVA was found to be incredibly competitive at reasoning about human impressions and aesthetic elements. Annotators ex- pressed a marginal preference for generations from LLaVA-7b-va fine-tuned on Impressions, with the largest improvement observed in aesthetic evalu- ation (59% preference score). This architecture was pre-trained on synthetic instruction-tuning multimodal dataset created using language-only GPT-4. Caption and bounding box information from the COCO dataset was leveraged in prompt- ing GPT-4 to generate a conversation in which one neural speaker asks questions about the image, and the other answers. This dataset produced a model that excels at generating eloquent, figurative, and pragmatic descriptions of visual media. However, although LLaVA has made great strides in zero- shot VQA and overall language generation qual- ity, have found the model tends to miss conno- tation that is heavily grounded in style. relies mostly on objects, entities, and setting to reason about potential human impressions. Yet there are instances where features such as contrast, lighting,
Image Description Generations on image de- scription were least improved by fine-tuning on the Impressions Dataset across all architectures explored in this study. However, as a plethora of datasets have been created for this purpose. this result was expected. The greatest improve- ment on descriptive generations was by fine-tuning GIT to achieve 75% preferential improvement over the base model. This was followed by fine-tuned BLIP, which was preferred 69% over BLIP base. This indicates that the Impression Dataset facili- tated model learning of captions that most closely aligned with viewers' perspectives.
Image Perception and Aesthetic Evaluation The Impressions dataset helped improve GIT, BLIP, and OpenFllamingo image impression and aes- thetic evaluation generations. Since GIT and BLIP were pre-trained on corpora like ImageNet and COCO, their base performance was to generate more neutral, terse, and denotational captions that

---

Description
Impression
Aesthetic Evaluation
All Captions
GIT
0.750
0.920
0.780
0.815
BLIP
0.690
0.840
0.880
0.805
OpenFlamingo-16
0.610
0.960
1.000
0.857
LLaVA-7b-v0*
0.560
0.530
0.590
0.560
Table 2: Results of the human evaluation task base and seee--uneddfeew-sot--appppe model generations. Each row shows the percentage at which annotators believed the model wwwed//fwwwshot-dadeeeee on Impressions data generated better captions than base/zero-shot model. Each column indicates the preference scores individual caption types. Denotes models fine-tuned for image-captioning Denotes models fine-tuned few-shot adapted for the conditional generation.
Promot: What dies the make feel, dhink, bedione?
Prompt:
Whut does thn image make you feel, thini, or befieve?
Fed a serse love and haprines.
aIT
benry buidingg and water bexutilul piace
fod semee jusand luppinon laoking as this image.
I Rel sence of one and wonder when lookng this image.
Open Flamingo
funily cajoing
This malees Reet hoppy jostul can company
Open
This inue unko me want
fird Tij Malall brurifil place viii. asyall.
the fimily vrry
Flamingo fs-16
and
This imag malox me firl emmrr of swe and adrination for the sunning anhivrure and rich cdtural horeey nowisted with the pyrthars. The besesffdlly doignal bla shis pegodi.
inage
sis, wali aduls snd posing picnure tozethes
whic prosunee
LLaVA
grass with some of them undinng while othen looking the canors, The fimily appean to be united and happs en- yoiing each wherk's compuny in candid momenn rxptured curing pionic
LLaVA 70
yolo will done inade reminioont suneroos people, some of whon ore Bring up to enter the gudWaam sngeess that it impornn and poprior place of wontti The boaun- filly painted mointainod eomrer, combenee with the vhene atmo- yphae makes feel gratffll reperrunity peosive valgion catural landnack.
gubhring Their orgession body laguuge convey sogtheene sharddjo
Figure 6: Comparison of each architecture's generation of potential human impressions on the Impressions dataset.
camera angle, and motion blur have great influ- ence on perlocutionary force and the coded iconic. To address this weakness, Impressions could be injected into the complex reasoning category of this synthetic dataset. Additionally, we recom- mend that future exploration of synthetic visual instruction-tuning datasets leverages features per- taining to image aesthetics, in addition to bounding boxes, when prompting transformer-based dialogue systems. Such features could include pixel intensity (Datta et al., 2006), contrast (Ke et al., 2006), color distribution (Datta et al., 2006; Ke et al., 2006), or visual smoothness (Datta et al., 2006).
eration qualities that may emerge when training multi-modal models on annotations created by in- dividuals belonging to different personality or de- mographic groups. Prior to beginning the image annotation task, annotators completed two surveys personality traits and demographic information through Amazon Mechanical Turk (see Appendix E). To build each persona-specific LLaVA model of image perception, we fine-tune on random sample of 500 annotations from the respective per- sonality or demographic group: introvert vs. extro- vert, agreeable vs. disagreeable, business-orients occupation vs. creative occupation, and no art experience vs. 3+ years of art experience.
6 Persona-Specific Generation
An evaluation set of 100 images leveraged to produce image perceptions with each of the eight LLaVA-7b-vD models fine-tuned on persona- specific data. We then compare distributions of sen-
To investigate the variation human perceptions of images captured by Impressions, we design a set of experiments exploring the distinctive gen-

---

too along the neer lank hot high buildngs noady, The beed namer and utbon envcomeens conans serse has mory It night exole for ing resorro and cllones
tupg deoue cipeapp
vewrr as portoss a poseeul corsitenc beween the cis and the fiet The siener might alo appreciate the onhiectuual laastty of the lighnii teildaye und the paturapu seene of the waseoar with the beat on Overal, dhe imagy procesa visully appralin and Lemmerrrs combituries of vrban remura clemeres.
Experience
body soocrnn fingrand, feanning small watmayy The slining casting warn light on wone penuroppe tules uee led scase of ave for costination of uhbur anchiocture and satund clmmetts Overal, Ol captiaating view evokes positive feeling and cesire to explort the city fusthe.
iag dicwsaa slyfin
Prompt: Wise does this image make you fod, taink, or beleess?
The inags a grap there men taking picuuee an army camen biiing Insed and drootaged ell fime They oberninng the explssion cqpuring the moment with thei cameris scese agaras that this night be denortration posd inohing the oiltory or maiinng excrtie, where the tork's copaldico are being dhowcosed mos be lie ciutiing and incooiing to vitne the powe and dieet of the camoon finhaad
spowing smolle warding the spectadl incra,, They axm be faaiuued by impi aixe dipian posildy wordering whide cajuiditti purpor the dem-cnttaioon Ovenal, the image orzaays avemn awe and cadie- mom the proeece posorful miltarr machine.
imagy langr miliary whlce posild army
ealaset give diplis
captiooinng
Extrovert
aunding aound fredkeariing militay
Prompt: What does image make you Ged, think, or believe
Figure 7: Image perceptions generated with LLaVA-7b-v0 fine-tuned with Impressions data created by annotators belonging to specific personality or demographic groups. The model fine-tuned on annotations created by individuals with 3+ years of visual experience creates descriptive image perceptions, with frequent reference the viewer. The model fine-tuned annotations created by annotators who identified as introverts produced more concise perceptions, whereas the extrovert model uses more metaphors.
timent intensity, sentence-level concreteness, and generation length to identify differences in model behavior across contrasting personas. We find that image perceptions created by the extrovert and in- trovert models have distinct distributions on gen- eration length with a statistical significance of p 0.026 by t-test. Similarly, the no art experience and 3+ years of art experience models have distinct distributions on sentence-level concreteness scores with a statistical significance of p - 0.012 by t- test. Figure 8 illustrates the differences, with the extrovert model producing longer captions and the 3+ years of art experience model achieving slightly higher concreteness scores. A qualitative exam- ple of the differences in generated perceptions can be found in Figure 7. The remaining model pairs,
namely agreeable vs disagreeable and business- oriented occupation vs creative occupation, did not produce distinguishable distributions any measure. It important to note that more distinc- tive behaviors can arise as training and evaluation sets are scaled, although it is possible that certain personality or demographic traits do not correlate with unique image perception trends.
Conclusion
Impressions was designed with inspiration from visual arts and media studies to evaluate and en- hance multimodal models' ability to reason about visual connotation. We show that, through fine- tuning and few-shot learning, this dataset enabled array of architectures to resolve potential hu- man impressions and discuss impactful aesthetic elements. State-of-the-ar vision models are pro- ficient enough at resolving entities and stylistic attributes to support such a task, but the weakness existed on the language side. This work highlights that targeted prompts modeled after image analysis techniques succeed in teasing out complex com- mentary on perlocation and the acsthetic clements it is grounded in. These prompts can be used by future works such VQA and instruction-tune with applications multimodal dialogue systems, toxt-to-image generation, and engagement predic- tion in advertising.
Distribution of Generation Length
Distribution of Sentence Concensus Scores
Introvert ectrooveet
ari-wp
no wrth--o Hob
Dersity
Word Count
Mean Sentence Consersus Score
Figure 8: Distributions determined be distinct with statistical significance < 0.05 by I-test for model
trained on contrasting personas.

---

8 Limitations
Monteiro, Jacob Menick. Sebastian Borgeaud. An- drew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022. Flamingo: a visual language model for few-shot learning.
Perhaps the most noticeable limitation of the Im- pressions Dataset is its size. Due resources con- straints, the benchmark contains only 1,440 images with unique annotations each. An increase in im- ages would have allowed for a wider exploration of visual symbols, and an increase in annotations per image would have better resolved the natural variation in audience impression. Additionally, we acknowledge that by welcoming inference in the annotation task, we also risk introducing harmful, biased, and discriminatory ideas. Although the au- thors have not observed any such content data quality checks, this dataset would benefit from an exploration on potential bias resolution.
Seyed Ali Amirshahi, Gregor Uwe Hayn-Loichsenring, Joachim Denzler, and Christoph Redies. 2015. Je- naesthetics subjective dataset: analyzing paintings by subjective scores. In Computer Vision-ECCCV 2014 Workshops: Zurich, Switzelaand, September 6-7 and 12, 2014. Proceedings, Pari I 13, pages 3-19. Springer International Publishing.
Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. Pro- ceedings of the 2016 Conference Empirical Meth- ods in Natural Language Processing.
Stanislaw Antol, Aishwarya Agrawal. Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433
9 Acknowledgments
We would like to thank all reviewers of this work for their insightful critiques, as well as the entire SALT lab for its ceaseless support. Special thanks to Ajay Divakaran for his mentorship, Benny Lin for his aid in building the annotation task UI, Brit- tney Newman for her insights Photojournalism, Zac Crawford for his illustration on the front page. and all the annotators who con- tributed to the Impressions dataset. Caleb Ziems supported by the NSF Graduate Research Fellow- ship under Grant No. DGE-2009655. This work was partially sponsored by NSF grant IIS-2247357 and IIS- 2308994.
Theo Araujo, Irina Lock, and Bob van de Velde. 2020. Automated visual content analysis (avca) in com- munication research: A protocol for large scale im- age classification with pre-trained computer vision models. Communication Methods and Measures. 14(4):239-263
domain expert in
JL Austin. 1962. Speech acts.
Anas Awadalla, Irena Gao. Joshua Gardner, Jack Hes- sel, Yusuf Hanafy. Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh. Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023. Open- flamingo.
References
Kevin G Barnhurst and Kelly Quinn. 2012. Political vi- sions: Visual studies in political communication. The SAGE handbook of political communication, pages 276-291.
Panos Achlioptas, Maks Ovsjanikov, Kilichbek Hay- darov, Mohamed Elhoseiny, and Leonidas J Guibas. 2021. Artemis: Affective language for visual art. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 1 11569- 11579.
Ronald Barthes. 1972. Mythologies. Hill and Wang. New York.
John Berger. 1972. Ways of seeing. BBC and Penguin.
Arjun R. Akula, Brendan Driscoll, Pradyumna Narayana, Soravit Changpinyo, Zhiwei Jia, Suyash Damle, Garima Pruthi, Sugato Basu, Leonidas Guibas, William T. Freeman, Yuanzhen Li, and Varun Jampani. 2023. Metaclue: Towards comprehensive visual metaphors research. 2023 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Mare Brysbaert, Amy Beth Warriner, and Victor Ku- perman. 2014. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research methods, 46(3):904-911.
Tuhin Chakrabarty, Arkadiy Saakyan, Olivia Winn. Artemis Panagopoulou, Yue Yang, Marianna Apid- ianaki, and Smaranda Muresan. 2023. I spy metaphor: Large language models and diffusion mod- els co-create visual metaphors. In Findings of the As- sociation for Computational Linguisties: ACL 2023, pages 7370-7388, Toronto, Canada. Association for Computational Linguistics.
Jean- Caan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne

---

Arjun Chandrasekaran, Devi Parikh, and Mohit Bansal. 2017. Punny captions: Witty wordplay in image descriptions. arXiv preprint arXiv: 704.08224.
Chuang Gan, Zhe Gan, Xiaodong He. Jianfeng Gao. and Li Deng. 2017. Stylenet: Generating attractive visual captions with styles. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3137-3146.
Kuang-Yo Yu Chang, Kung-Hung Lu, and Chu-Song Chen. 2017. Aesthetic critiques generation for photos. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
Noa Garcia, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu. Yuta Nakashima, and Teruko Mitamura. 2020. A dataset and baselines for vi- sual question answering on art. In Computer Vision- ECCV 2020 Workshops: Glasgow, UK, August 23- 28, 2020, Proceedings, Part II 16, pages 92-108. Springer.
Xinlei Chen, Hao Fang, Tsung-Y- Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitmick. 2015. Microsoft coco captions: Data collection and evaluation server.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904 6913.
Lærke Højgaard Christiansen. 2018. The use of visuals in issue framing: Signifying responsible drinking. Organization Studies, 39(5-6))665 689.
Marcella Comia, Matteo Stefanini. Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed- memorr trans- former for image captioning. 2020 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR).
Lea Hellmueller and Xu Zhang. 2019. Shifting to- ward a humanized perspective? visual framing anal- ysis of the coverage of refugees on cnn and spiegel online before and after the iconic photo publica- tion of alan kurdi. Visual Communication, page 1470357219832790.
Anne d' Alleva. 2005. Methods and theories of art his- tory. Laurence King Publishing.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav. Jose M. F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. 2017 IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR).
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embed- dings, convolutional neural networks and incremental parsing. To appear.
Jessica Hullman and Nick Diakopoulos. 2011. Visu- alization rhetoric: Framing effects in narrative visu- alization. IEEE transactions on visualization and computer graphics, 17(12):221--2240.
Ritendra Datta, Dhiraj Joshi, Jia Li. and James Z Wang. 2006. Studying aesthetics in photographic images using a computational approach. In Computer Vision- ECCV 2006: 9th European Conference Computer Vision, Graz, Austria, May 7-13, 2006, Proceedings. Part III 9, pages 288-301. Springer.
Xin Jin, Jianwen Lv, Xinghui Zhou, Chaoen Xiao, Xi- aodong Li, and Shu Zhao. 2022. Aesthetic image captioning on the fac-captions dataset. Computers and Electrical Engineering, 101:107866.
Bill Dolan and Chris Brockett. 2005. Automati- cally constructing corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005).
Xin Jin, Le Wu, Geng Zhao, Xiaodong Li, Xiaokun Zhang, Shiming Ge, Dongqing Zou, Bin Zhou, and Xinghui Zhou. 2019, Aesthetic Attributes Assessment of Images, page 311-319, Association for Computing Machinery, New York, NY, USA.
Donis A Dondis. 1974. A primer of visual literacy. Mit Press.
Jungseock Joo, Weixin Li, Francis F. Steen, and Song- Chun Zhu. 2014. Visual persuasion: Inferring com- municative intents of images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Yan Ke, Xiaoou Tang, and Feng Jing. 2006. The design of high-level features for photo quality assessment. In 2006 IEEE Computer Society Conference on Com- puter Vision and Pattern Recognition (CVPR'06), volume I, pages 419-426. IEEE
Corneliu Florca, Räzvan Condorovici, Constantin Ver- tan, Raluca Butnaru, Laura Florea, and Ruxandra Vrânceanu. 2016. Pandora: Description of painting database for art movement recognition with baselines and perspectives. In 2016 24th European Signal Processing Conference (EUSIPCO), pages 918-999899 9222 IEEE.
Fahad Shahbaz Khan, Shida Beigpour, Joost Van de Weijer, and Michael Felsherg. 2014. Painting-91: large scale database for computational painting categorization. Machine vision and applications. 25: 1385-19977

---

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz. Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernttein, and Li Fei-Fei. 2017. Vi- sual genome: Connecting language and vision us- ing crowdsourced dense image annotations. Interna- tional Journal of Computer Vision, 123(1).
Laura R. Oswald. 2012. Marketing Semiotics: Signs, Strategies, and Brand Value. Oxford University Press.
44Marketing Semiotics. In
Rameswar Panda, Jianming Zhang, Haoxiang Li, Joon-
Young Lee, Xin Lu, and Amit K Roy-Chowdhury. 2018. Contemplating visual emotions: Understand- ing and overcoming dataset bias. In Proceedings of the European Conference on Computer Vision (ECCV), pages 579-595.
Sabine Lang and Björ Ommer. 2018, Reflecting on how artworks are processed and analyzed by com- puter vision. In ECCV Workshops (2), pages 647- 652.
Cesc Chunseong Park, Byeongchang Kim, and Gun- hee Kim. 2017. Attend to you: Personalized image captioning with context sequence memory networks. In Proceedings of the IEEE conference on computer vision and pattem recognition, pages 895-903.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-seve coannce-to-sqqueeee tequeenecce pre-training for natural language generation, translation, and com- prehension. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Thomas E Powell, Hajo G Boomguarden, Knut
De Swert, and Claes H De Vreese. 2015. A clearer picture: The contribution of visuals and text to fram- ing effects. Journal of communication, 65(6):997- 1017.
Guodun Li, Yuchen Zhai, Zehao Lin, and Yin Zhang. 2021. Similar scenes arouse similar emotions: Paral- lel data augmentation for stylized image captioning. In Proceedings of the 29th ACM International Con- ference Multimedia, pages
Lakshay Sharma, Laura Graesser, Nikita Nangia, and Utku Evei. 2019. Natural language understanding with the quora question pairs dataset. arXiv preprint urXiv: 99.00.0411......
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut, 2018. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2556-2565.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language- imagge pre- training for unified vision- languaee understanding and generation.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction uniig.
Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston. 2019. Engaging image captioning via personality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 12516-12526.
Ana Marasović, Chandra Bhagavatula, Jac sung Park, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Natural language rationales with full-stack visual reasoning: From pixels semantic frames to commonsense graphs. Findings of the Association for Computational Linguistics: EMNLP 2020.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. 2021a. Wit: Wikipedia- seapeedia-based image text dataset for multimodal multilingual machine learning.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. 2019 IEEF/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR).
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. 2021b. Wit: Wikipedia- image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 2443-2449, New York, NY, USA. Association for Computing Machinery.
Saif Mohammad and Svetlana Kiritchenko. 2018. Wikiart emotions: An annotated dataset of emotions evoked by art. In Proceedings of the eleventh in- ternational conference on language resources and evaluation (LREC 2018).
Yutong Tan, Zheng Lin, Peng Fu, Mingyu Zheng, Lan- rui Wang, Yanan Cao, and Weipinng Wang. 2022. Detach and attach: Stylized image captioning with- out paired stylized dataset. In Proceedings of the 30th ACM International Conference on Multimedia. pages 4733-4741
Naila Murray, Luea Marchesotti, and Florent Perronnin. 2012. Ava: A large-scale database for aesthetic vi- sual analysis. In 2012 IEEE conference on computer vision and pattern recognition. pages 2408-2415 IEEE.
Julianne Newton. 2013. The burden of visual truth: The role of photejourmalism in mediating reality. Rout- ledge.
Susan Teitz. 1999. 21 for the 21 century: Amer- ica's best newspapers. Columbia Journalism Review. 38.

---

Polina Tsvilodub and Michael Franke. 2023. Evaluating pragmatic abilities of image captioners on a3ds.
Lining Zhang, Simon Mille, Yufang Hou, Daniel Deutsch, Elizabeth Clark, Yixin Liu, Saad Ma- hamood, Sebastian Gehrmann, Miruna Clinciu, Khy- athi Chandu, and João Sedoc. 2022. Needle in baystack: An analysis of high-agreement workers on mturk for summarization.
Dilek Melike Uluçay and Gizem Melek. 2021. Self- presentation strategies and the visual framing of polit- ical leaders on instagram: evidence from the eventful 2019 istanbul mayoral elections. Visual Communica- tion, page 14703572210057595
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, Volume 1 (Long and Short Pupers), pages 1298-1308
Emiel Van Miltenburg. 2016. bias in the flickr30k dataset. arXiv:6605006083
Stereotyping and arXiv preprint
Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Wenfeng 7heng, Lirong Yin, Xiaobing Chen, Zhiyang Ma, Shan Liu, and Bo Yang. 2021. Knowledge base graph embedding module design for visual question answering model. Putlern recognition, 120: 108153.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-basee image descrip- tion evaluation. 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR).
Helena Zinkham. 2006. Reading and researching pho- tographs. Photographs: Archival care and manage- ment, pages 59-77.
Daniel de Vise. 2011. What the rankers ranked news- papers?
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022. Git: A generative image-to taage-to-texx transformer for vision and language. Transactions of Machine Learning Research.
Keren Ye, Narges Honarvar Nazari, James Hahn, Za- eem Hussain, Mingda Zhang, and Adriana Kovashka. 2019. Interpreting the rhetoric of visual advertise- ments. IEEE transactions on pattern analysis and machine intelligence, 138--33333
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Phine Phillipp Morency. 2019. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR).
Sina Zarrieß, Hendrik Buschmeier, Ting Han, and Simeon Schüzz 2021. Decoding, fast and slow: case study on balancing trade-offs in incremental, character-level pragmatic reasoning. In Proceedings of the 14th International Conference on Natural Lan- guage Generation, pages 371-376, Aberdeen, Scot- land, UK. Association for Computational Linguistics.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. 2019 JEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR).

---

A
Caption Perturbation for Semantically Similar Image Collection
We leverage both linguistic and visual informa- tion to collect images with semantic similarity. Im- ages are collected from the Google search API via perturbed caption queries of the anchor image cap- tion (see Appendix A). Additionally, an image is only added to a bin if has a cosine similarity less than 0.8 and greater then 0.2 to every other image in the bin. We experimentally determined that a cosine similarity larger than this range suggested the image was a duplicate, and anything lower was too dissimilar from the other visual media.
To create image bins with marginal semantic simi- larity, but variation in curation, style, and design (or lack thereof), we collect 3 images via the Google Search API to accompany an anchor image from the NYT. The queries leveraged to search for these images are perturbed captions created from the leading paragraph associated with the NYT image anchor. Lead paragraphs are perturbed in the fol- lowing three ways:
C Images the Impressions Dataset
Summarization Single sentence summaries of NYT anchor image captions are created via BART conditional generation model (Lewis et al., 2020) that fine-tuned on three dif- ferent paraphrase tasks: Quora Question Pairs (Sharma al., 2019), PAWS (Zhang al.. 2019), and the Microsoft Research Paraphrase Corpus (Dolan and Brockett, 2005).
Figure 9 displays a random sample of images in- cluded in Impressions. As described in the pa- per, the dataset is blend of photojourmalisti im- ages from articles in the New York Times attained through the official NYT AI, and images collected using the Google Search API. This data collection process yielded a wide variety of visual features, styles, and aesthetic elements.
Named Entity Recognition Named entities were resolved from the anchor image caption by way of the spaCy NER pipeline (Hon- nibal and Montani, 2017). The method pro- duced stand-alone entites as Google Image queries.
D Annotator Qualification
Collecting clean and well written commentary the semiotics of images demanded a complex an- notation task. Utilizing crowd-sourcing platforms like Amazon Mechanical Turk provides access to a diverse collective intelligence for building machine learning resources. However, they also come with challenges in maintaining data quality (Zhang et al., 2022).
Subject Tree Extraction We build queries by extracting from anchor captions their min- imal dependency parse trees defined by their subject, root verb, and any direct objects con- tained in the caption.
To ensure that annotators contributing this dataset crafted well-writing and relevant commen- tary on perlocution and aesthetic elements, a quali- fication task was built through which their abilities could be reviewed before admission. required 3 example annotations, in which the annotator had to rank images in a bin and then provide free-form an- notations on the image they gave the highest impact Additionally, the authors conducted weekly quality checks on random set of 50 submissions for every batch of 300 assignments.
A set of candidate queries is produced from these three perturbation strategies, from which three are selected at random. For each query, 5 images are collected via the Google search API. Finally, only image is retained per perturbed caption. As discussed is $3.1 and Appendix B, images are vet- ted via thresholds on cosine similarity with other images its intended image bin.
B
Image Bins
The motivation behind creating imagine bins with semantic similarity but varying degrees of curation, was to group visual media that may be communi- cating similar things in different ways or to vary- ing degrees of success. Reasoning about aesthetic impact in isolation
E Annotator Personality and Demographics Data
As part of the data collection process, the annota- tors were asked to complete two additional qualifi- cations through Amazon Mechanical Turk: sur- vey on demographic information and a condensed version of the Big 5 Personality test. These at- tributes were collected for the exploration of mod-
difficult task even for the trained eye. But an annotator will find easier to distinguish the saliency and communicative utility visual media in comparative setting.

---

Figure Images Impressions. The following images were randomly sampled the corpus.
eling viewer impressions of a particular community, and is included in Impressions metadata. Figures 10 and 11 showcase the personality traits and de- mographic attributes represented in the dataset.
task. Accompanying positive and negative annotation examples are shown in Figure 17.
In this task, we will be reviewing the set of 4 images displayed below and ranking them on aesthetic impact relative to one another.
F Persona-Specific Generation Experiments
A photograph is aesthetically impactful if the style and design choices catch your atten- tion, and inspire you to feel, think, believe something. Although subject matter is often important, ask that you focus on how the visual elements (composition, lighting. color, perspective, etc.) impact your perception.
As discussed in Section 6, investigate the dis- tinctive behaviors that may arise when finettnning LLaVA-7b-v0 models on Impressions data created by annotators belonging to varying personality and demographic groups. Figures 12, 13 and 14 display distributions on sentiment intensity, sentence-level concreteness, and generation length for each model pair explored in this experiment. Note that although some distributions may appear different, the only ones that are distinct with statistical significant 0.05 are introvert vs extrovert on generation length, and no art experience vs 3+ years of art experience on sentence-level concreteness.
Rank each image on scale of to 4 based on how aesthetically impactful it is relative to the other im- ages in the set. would be the most acsthetically impactful of the and would be the least im- pactful. you strongly believe two photographs are equally aesthetically impactful, you may give them the same score. DO NOT give the same score to more than 2 images.
G Instructions for Annotation
Annotators were provided with the following instructions for the aesthetic image ranking
Instructions provided for the free-form an-

---

Distributions Annotator Big Personality Traits
Distributions Annotations Across Big Personality Traits
Nomber Menesato
Extroversion
Agrocablaness
Extroversian
Agrocabioness
Nimber ofAinoattio
2000
2003
100
100
Conscentixeeness
Neurotidem
Conseentouuneess
Neurotisimm
Nonbor of
300
Opennees
Figure 10: The figure the left demonstrates the distributions of annotators over the Big 5 Personality Traits. Whereas the figure of the right showcases the number of image captions created by annotators characterized by specific Big5 Personality traits.
Distributions of Annotators Across Demographic Attributes
Distributions of Annotations Across Demographic Attributes
Nenclatrrs
Center
Uneer
Gaice Mhien
2002
Emary
Gender idetiiicttion
Age
Gender identiication
Age
Rocthh wul
Hangish
Jakker
Number
Race/Ethnicky
Reigion
RacerEthrictt
Reilg con
1000
Vitul Liarlacee
Hon Genol
Education
Misual Arts Experience
Education
Vaua Arts Exper knooe
Nummooe Mareeeeerrre
Baaess Minootent un Ademsnntoo
Intomatee Bahere
Helm: Copwers Coatos and lemoooo
Diner
Profession
Profession
The figure on the left demonstrates the distributions annotators over demographic categories. Whereas the figure of the right showcases the number of image captions created by annotators characterized by specific demographic categories.
Figure 11:

---

notation task on image description, image impressions, and aesthetic evaluation are shown below. Accompanying positive and nega- tive annotation examples are found in Figure 16.
Distribution Sentiment Intensity per Caption
extrovert vs. introvert
agreeable vs, disagreeable
scbevee introvert
agrveabu disegreatte
Deraity
Given the image-caption pair below, please answer the following free-form questions. Your responses must be - 3 sentences long. Review the positive and negative annota- tion examples below before beginning this task.
business vs. creative
art exp vs, no-art-exp
bushees creative
art-esp
no cep
Deriity
For the first question, please DO NOT sim- ply paraphrase the caption. Remember the image can hold a lot of information too! Let us know what you understand about the scene after evaluating both the image and the text.
-1.0 -ATh 50 -0.5
8.80
525
3
3.0
-100
-322 8:00
6:5 3.0
C35
39
Compound
Compound
Figure 12: Sentiment intensity distributions on contrast- ing persona pairs.
In the second question you will be asked about your impression of the image, your an- swer is expected to be subjective! Describe what this image makes you feel, think, or believe.
Distribution Impression Concreteness Scores
extrovert vs. introvert
agreeable vs. disagrneable
abeett
ag 50
Denaty
The final question will ask you to identify some visual elements in the image that contribute to your impression. We provide a list of common visual elements to assist you. You could use one, some, or none of the elements listed. It is up to you!
business VS. creative
art exp vs. no art exp
Derady
The list of visual elements provided is shown in Figure 15. This resource was created to assist annotators in answering the aesthetic evaluation prompt.
42
3.2
Sentence Concreteness
Sentonce Concretoress
Figure 13: Sentence-leve conercteness distributions contrasting persona pairs.
Visual Element Technique Vocabulary
Distribution Generation Lengths
No provide thes as commics sinull eamerts suciet deeortingg secthetica that
extrovert vs. introvert
egrecable vs. disagreeable
eccrovent
agreweble
Introwert
d'aggeeable
Exprossion
Action
Buskiround
Decaky
Body Language
Mettion
Foreground
Relationahip
Compositionn
Foous
Contrast
Fosttio of Camera
business vs. creative
art-exp Vs. na-art-exp
business
art exp
Figure 15: collection of visual elements provided to annotators to guide them through linking concrete aesthetic features to their impressions of an image. Any visual elements outside of this list welcomed.
treative
novaltin Leeep
Denety
180
150
340
144
73
106
320
303
140
H Crowd-soureing Platform Commentary
Figure 14: Generation length distributions on contrast- ing persona pairs.
Within the scope of this work, annotators were re- cruited via Amazon Mechanical Turk and Upwork.

---

Posttive Image Ammotation Esamples
Nogatile Image Annotation Examples
puce valor saher
Alin and valoo mon ceree Lish
What in heppeeing in this imaget
What to hopparing in tmage?
Toe regn tii thee lnckire
hoppering Image?
aut jataraationn plesse cartis.
camage boom funt rengride cheme
sune grund lnck toms sonnes and sulding
mobblle aurare Josinur coleetieee Syovie
thec bnsthe rrking thip thipprobun rofuan co seequetcie
C10 carton segledic cacito ne thet
What does this feel,
What iesp nake you
cithe dii un lageding rrilep ang syed in para
lack di siot dieciii las sonno hare
For moje iite you feter recicee the
canini: salaiime sclapiao cetatinn
of
vitual would you
impresatio The contort and deep hodion.
viual siamaris would conicitue inpeostiin bmagge?
repmemere this sinper?
the stacidiment ye shlend
The yur troc ane clegii undite tork- peccome tnt haloyrall ueeaai hanging
pppeas saten be try rouent canaged
Dalcoarde The coneacte dive slconiintts
madde ve inagene taril woull aeviiig
buborm one demugit accorole
Figure 16: Positive and negative examples provided to annotators on aesthetic impact ranking.
The authors experienced the following important trade-offs of working with these different resources for attaining human annotation.
addressed with annotators directly, which produced an increase in annotation quality over time. Addi- tionally, once of team of over 8 annotators was consolidated on UpWork, the data annotation turn- around time caught up to that of Mechanical Turk.
Amazon Mechanical Turk has faster annota- tion turn-around time and is often less expensive as Workers are paid per completed assignment. How- ever, quality control is a challenge as annotators are insentivized to complete assignments as quickly as possible, often to the detriment of annotation com- plexity and correctness. Even with qualifications in place and a manual evaluation procedure for admit- ting Workers, the authors experienced input quality decreasing overtime or completely changing after Workers passed the vetting procedures. Addition- ally, Mechanical Turk does not support effective avenues of communication with annotators. This makes it incredibly difficult to address annotator mistakes, misinterpretationss, or abuse of the task. After observing these behaviors on Amazon Me- chanical Turk, the authors began recruiting contrib- utors through UpWork in parallel. The onboard- ing process is considerably longer as one must re- view proposals, contract individual annotators, and provide instructions to each annotator individually. Hourly contracts prove to be more expensive, yet rc- move the incentive complete assignments fast possible. This results in less data loss through quality vetting of completed annotations. any mistakes or misinterpretations were uncovered dur- ing data quality reviews, these concerns could be
In conclusion, although hourly contracts through Up Work were slightly more expensive and onboard- ing is more time consuming, the benefits to data quality and workflow made recruiting annotators through this platform the more efficient choice for data collection.
H.1 Annotator Compensation
Assignments on Amazon Mechanical Turk were valued at $1.10, to amount to $15 per hour at the average annotation rate anticipated for this task (4 5 minutes). The annotation rate was estimated by recruiting a number of individuals (including the authors themselves) complete the task while keeping time.
Annotators recruited through UpWork were paid hourly at a rate of $15 per hour. There were a cou- ple of cases where annotators were contracted at $12 per hour. This was only arranged if the ob- served annotation speed considerably lower than expected, but the annotator was eager con- tribute to the task. Given that one annotation is expected to take less than minutes, this avenue was only explored if an annotator is observed to take more than 10 minutes.

---

Austharic Irpact Ranking: Posttiv Examples
Awrtharic impact Banitig: Negathen Examples
Modt
Fmmeee gor chair ros
suly an cacital fecohit
You corn las sappe ceriellll poocompgee
Iapactal
Figure 17: Positive and negative examples provided to annotators aesthetic impact ranking.
I Human Evaluation
The caption that better simulates a plausible human impression of the image. Did the caption get the mood right? Is the impression or thought simulates likely to be shared by a human au- dience? How much depth is there to its description?
The human evaluation task presented an image, prompt, and two generated captions to annotators recruited through Upwork, and requested they select the caption they believed to be the best fit for the image-prompt pair. One caption/answer the image-prompt pair generated by base, pre-trained architecture, and the other was generated by the same model fine-tuned or few-shot adapted on Impressions. The follow- ing instructions were presented to annotators defining what characterizes the "better" caption:
The caption that better identifies aesthetic elements and is capable of discussing style. In other words can it discuss contrast, camera angle, light, etc. Bonus points if it correctly spec- ifies how that design choice can inspire an audience.
The caption is relevant to the prompt. It must at least try to answer the ques- tion you will see acompanying the image.
the final stage of this project will be human evaluation task, where you will be comparing the outputs of two Generative Al models given an image and a prompt. You must select which caption does better job of answering the prompt given the image shown. This project designed to enable AI architectures to better predict plausible human impressions of images, resolve more socially-aware information, and better discuss what aesthetic elements can make an image impactful. Therefore, when we say select the "better caption", this is what we mean:
Please do NOT base your decision the following attributes: caption length, punctuation, run-on or cut-off sentences, minor repetitiveness, com- plexity of grammar, and minor mistakes (such as miscounting objects, misidentifying entities or peo- ple, and hallucinating an object that might make sense in the scene, but its not actually depicted).
The caption that better describes the events, ac- tions, relationships and feelings communicated by the image. This means going beyond listing off the items and entities depicted, and discussing the context of the scene and including information that an observer can reasonably infer. This also means capturing social and cultural information.


Reviews:
[{"review_id": "T659rfjyKB", "reviewer": "Reviewer_p7vQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful recommendations. We will add more example images to the appendix and include missing citations in our camera-ready paper.\n\nRe: question about image generation experiments - this insight could inspire a new project. We posit that an understanding of only certain aesthetic elements would transfer directly to image generation tasks. It seems that generative models have learned the relationship between lighting/color-scheme and particular moods, but composition, use of space, and connotation of specific visual elements are much greater challenges.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed dataset with the rich annotations is very interesting and would be highly useful for the research community. The proposed prompts in section 3.2 (image description, perception and aesthetic evaluation) nicely capture key aesthetic elements. Extensive automatic and human evaluation, analysis has been performed on SOTA baselines. Overall, the paper is solid with well-thought and well-written annotation pipeline and quality analysis.", "reasons_to_reject": "I do not find any major weaknesses with this work. It would be nice to add more example images to the appendix. I am curious, if the paper explored generating the images from the dataset? What would be some of the challenges in image generation that are specific to the aesthetic aspects? Citation to the below work on visual symbols/metaphors is missing.\n\nAkula, Arjun R., Brendan Driscoll, Pradyumna Narayana, Soravit Changpinyo, Zhiwei Jia, Suyash Damle, Garima Pruthi et al. \"Metaclue: Towards comprehensive visual metaphors research.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23201-23211. 2023.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a new benchmark, Impressions, consisting of 1440 image-caption pairs and 4320 annotations capturing various aesthetic elements and impactfulness. The paper shows that the SOTA models find it difficult to model human impressions and fine-tuning on the proposed benchmark improves their performance.  ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mM4aEleRvj", "reviewer": "Reviewer_7iNP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Re:  “a clearer understanding of how Impressions stands in relation to other datasets”, we have originally discussed this in section 3.3 where we highlight the characteristics that distinguish Impressions from COCO through quantitative analysis. Impressions has:\n\n- Greater variation of sentiment intensity across annotations (see Figure 3).\n- Greater variation in perceptions described by disparate viewers for the same image (see Figure 4).\n- More figurative speech to describe complex feelings, ideas and beliefs, as demonstrated by lower concreteness scores (p < 0.001; see line 410 and Figure 5).\n- Greater complexity and length of annotations (see Figure 2)\n\nWe further demonstrate the utility of the Impressions dataset by human evaluation of image-captioning and VQA tasks, since fine-tuning and few-shot adaptation with Impressions leads to better impression generations (see Table 2).\n\nRe: “statistical analysis compared with other relevant datasets”, as discussed above, section 3.3 contains extensive comparison of Impressions to COCO. Another popular method of comparing image-captioning datasets is displaying CIDEr-D scores in the tabular form suggested by the reviewer. A strength of Impressions is it captures the variation in viewer perception of images, however, this quality makes consensus-based evaluation and quality metrics like CIDEr-D non-optimal for evaluating this dataset (as discussed on lines 434 - 440). That is why we omit any statistical analysis over such metrics. \n\nThat being said, we would be happy to extend the quantitative analysis comparing Impressions with COCO to an additional two datasets in the camera-ready paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is easy to follow and presents its content in a clear and organized manner. \n\nThe authors have taken great care in providing a detailed annotation process and comprehensive dataset statistics, enhancing the paper's credibility and transparency. \n\nThe experiment results offer insights into both the dataset's effectiveness and the limitations of existing models. The empirical evidence shows  how fine-tuning and few-shot adaptation significantly enhance the models' capacity to mimic human impressions and aesthetic evaluations of images.", "reasons_to_reject": "While the paper excels in many aspects, a more thorough statistical analysis compared with other relevant datasets is lacking. Incorporating a comparison table would be preferable, as it would provide a clearer understanding of how Impressions stands in relation to other existing datasets.\n\nLack of comparisons.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a novel dataset, Impressions, designed to explore the semiotics of images and investigate how visual features and design choices evoke specific emotions, thoughts, and beliefs. The dataset consists of 1,440 image-caption pairs and 4,320 unique annotations. In addition, the paper show that the state-of-art transformer-based architectures struggle to simulate human responses to images, but fine-tuning and few-shot adaptation significantly improve their ability to model impressions and aesthetic evaluations. The paper concludes that targeted prompts inspired by image analysis techniques are effective in eliciting complex commentary on perlocation and aesthetic elements.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2xkCyfSahZ", "reviewer": "Reviewer_GJAP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Re: “reference for Impressions dataset,” on line 121, Impression dataset is a novel contribution presented in our paper, therefore, we did not include a reference here. Section 3 describes our unique process to collect viewer perceptions of visual media, as well as concrete visual elements that inspire these perceptions.\n\nRe: “journalism and communication,” the manner in which this work applies to these fields is more thoroughly explored on lines 208 - 228: \n“The Impressions dataset was designed to ground the discussion of aesthetic elements in the impressions they inspire, to capture the link between the visual signifier and the signified. We also posit that aesthetic impact, or the likelihood of media to attract and hold attention, goes beyond what is considered classically beautiful and extends to the communicative utility of aesthetic elements…”\n\nRe: “Table 1… trend,” we bold the best performing model for each metric. OpenFlamingo and BLIP are the strongest performers on standard ngram metrics (ROUGE & BLEU), whereas LLaVA-7b-v0 outperforms the other models on harmonic mean of precision and recall metrics when incorporating the stemming and synonym matching of METEOR. GIT is the worst performing model across all metrics.\n\nRe: “discussion on adjective-noun pairs”, although a valid analysis, we argue that it would be supplemental given the findings already presented on the distribution of sentence-level concreteness in section 3.3. Both of these methods demonstrate an increase in the use of figurative and expressive speech. We are happy to clarify this in the revised version of this paper.\n\nRe: “discussion on… sentiments in emotions is needed”, in section 3.3 we present (1) a detailed comparison of sentiment intensity between Impressions and COCO, and (2) an analysis of the variance in sentiment intensity per image, indicating greater variation in perceptions described by disparate viewers (see Figures 3 and 4).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "They have evaluated on the impressions dataset that contains rich commentary on viewer perception. \n\nThey conclude that captioning after image analysis can help describe complex aesthetic elements.", "reasons_to_reject": "Pg 1, right col, line no 041, how is this paper used in journalism and communication. Give details? \n\nPg 2, right col, line no 121, there is no reference for Impressions dataset? \n\nTable 1 legend does not explain any trend. Which is the best?\n\nSome discussion on adjective-noun pairs and sentiments in emotions is needed. See for example, ‘Text-Image Sentiment Analysis.’ 2023, From: CICLing 2018: 19th International Conference on Computational Linguistics and Intelligent Text Processing,", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates if aesthetic beauty in an image can result in effective communication. Current image captioning models do not correctly predict human responses to images.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers provide feedback for this paper and their reviews were with consensus. The reviewers found the dataset interesting and useful. They found the extensive evaluations to be well done. They appreciated the human evaluation. The details provided in the paper, particularly about the annotation process was appreciated. There were very few concerns pointed out by the reviewers, and these were fairly minor. Given this consensus in soundness and excitement, I recommend accepting the paper.
Decision: Accept-Main