Measuring And Improving Engagement of Text-to-Image Generation Models
Authors: Varun Khurana
Yaman Kumar Singla
Jayakumar Subramanian
Changyou Chen
Rajiv Ratn Shah
zhiqiang xu
Balaji Krishnamurthy
Venue: ICLR 2025 Poster (2025)
Keywords: image generation models
text to image models
image engagement
stable diffusion
dalle
persuasion
engagement
behavioral science
human behavior
behavior in the wild
Primary area: generative models
TL;DR: We develop a framework to measure and benchmark the capability of text-to-image models in generating engaging images, and propose methods to optimize the image generation process to enhance engagement.

Abstract: Recent advances in text-to-image generation have achieved impressive aesthetic quality, making these models usable for both personal and commercial purposes. However, in the fields of marketing and advertising, images are often created to be more engaging, as reflected in user behaviors such as increasing clicks, likes, and purchases, in addition to being aesthetically pleasing. To this end, we introduce the challenge of optimizing the image generation process for improved viewer engagement. In order to study image engagement and utility in real-world marketing scenarios, we collect *EngagingImageNet*, the first large-scale dataset of images, along with associated user engagement metrics. Further, we find that existing image evaluation metrics like aesthetics, CLIPScore, PickScore, ImageReward, *etc.* are unable to capture viewer engagement. To address the lack of reliable metrics for assessing image utility, we use the *EngagingImageNet* dataset to train *EngageNet*, an engagement-aware Vision Language Model (VLM) that predicts viewer engagement of images by leveraging contextual information about the tweet content, enterprise details, and posting time. We then explore methods to enhance the engagement of text-to-image models, making initial strides in this direction. These include conditioning image generation on improved prompts, supervised fine-tuning of stable diffusion on high-performing images, and reinforcement learning to align stable diffusion with *EngageNet*-based reward signals, all of which lead to the generation of images with higher viewer engagement. Finally, we propose the *Engagement Arena*, to benchmark text-to-image models based on their ability to generate engaging images, using *EngageNet* as the evaluator, thereby encouraging the research community to measure further advances in the engagement of text-to-image modeling. These contributions provide a new pathway for advancing utility-driven image generation, with significant implications for the commercial application of image generation. We have released our code and dataset on [behavior-in-the-wild.github.io/image-engagement](https://behavior-in-the-wild.github.io/image-engagement).

Published as a conference paper at ICLR 2025
MEASURING AND IMPROVING ENGAGEMENT OF TEXT-TO-IMAGE GENERATION MODELS
+ III
Varun Khurana
Yaman K Singla"
Jayakumar Subramanian
IID
Changyou Chen
Rajiv Ratn Shah
Zhiqiang Xu
Balaji Krishnamurthy
A
III IIT-Delhi, "MBZUAI
Adobe Media and Data Science Research,
SUNY at Buffalo,
ABSTRACT
Recent advances in text-to-image generation have achieved impressive aesthetic quality, making these models usable for both personal and commercial purposes. However, in the fields of marketing and advertising, images are often created to be more engaging, as reflectdd in user behaviors such as increasing clicks, likes, and purchases, in addition to being aesthetically pleasing. To this end, we introduce the challenge of optimizing the image generation process for improved viewer en- gagement. order to study image engagement and utility in real-world marketing scenarios, collect Engaging/mageNNe, the first large-scale scale dataset of images, along with associated user engagement metrics. Further, we find that existing image evaluation metrics like aesthetics, CLIPScore, PickScore, ImageReward, etc. are unable to capture viewer engagement. To address the lack of reliable metrics for assessing image utility, EngageNet, an engagement-await Vision Language Model (VLM) that predicts viewer engagement of images by leveraging contextual information about the tweet content, enterprise details, and posting time. We then explore methods to enhance the engagement of text-to-image models, making initial strides in this direction. These include conditioning image generation on improved prompts, supervised fine-tuning of stable diffusion on high-performing images, and reinforcement learn- ing to align stable diffusion with EngageNNetbbaeed reward signals, all of which lead to the generation of images with higher viewer engagement. Finally, we propose the Engagement Arena, benchmark text-to-image models based their ability to generate engaging images, using EngageNet as the evaluator, thereby encouraging the research community measure further advances in the engage- ment of text-to-image modeling. These contributions provide new pathway for advancing utility-driven image generation, with significant implications for the commercial application of image generation. We have released our code and dataset on mnbehavvir--ae in-
the Engaginglmage/Ve dataset to train
INTRODUCTION
Machine learning models that interact with humans are built as a means achieve an end, and performance metrics in their respective fields reflect how effectively these models meet the ends. For instance, recommendation systems are optimized to capture maximum viewer interest and the key performance metrics tracked by the research community are clickthrough rates and the number and ranking of relevant documents recommended out the total document set (Bobadilla et al. (013). Similarly, chat assistants are optimized for being helpful, and the commonly tracked metrics the scores of responses preferred by humans (Ouyang et al. 2022 Stiennon et al. 2020). the case of image generation, industries such e-commerce, fashion, education, and advertising aim optimize user-focused outcomes like clicks, purchases, retention, and user engagement. However, the metrics used by the image generation research community often emphasize aesthetic appeal (Xu et al. 2024 Kirstain et al. 2023: Black et al. (203)) and realism (Dhariwal & Nichol, [021] (Saharia et al. 2022
Ho al.] [2200] Rombach et al. (022) factors that crucial for image acceptability but necessarily
aligned with the ultimate goals of viewer engagement.
"Equal Contrihution. Contact behavior-in-thc-wildee wwgg://wwwwwoooooogggggo.o..momm oog.eeaeeoo........o..oooooeeoon
for questions and suggestions.

---

Published as a conference paper at ICLR 2025
Assthrica! Libes
Assthelics
Libes"
PickSeore' Lies
PickScors Likes
HiSs
Liker
HPP
Likes
Iragehlwwrd
"ike Imagellzward Likes
Figure I: Some images from the EngaginglmageNNe dataset. We constructed pairs of similar images posted within 45 days interval by the same account. In each pair shown in the figure, the left image corresponds lower likes and right one received higher likes. However, existing image generation metrics like Aesthetics, PickScore, Human Preference Score, ImageReward, etc., exhibit image preference in the opposite direction as actual user engagement.
We find that popular image generation metries such Aesthetics (Schuhmann et al. 2022), ImageRe- ward (Xu et al. 2024). Human Preference Score (HPS) (Wu et al. 2023), and CLIP-H (Radford et al.. 2021 have correlation ranging from 0.02-0.08 with user engagement measured by likes, roughly equal to random chance (Table 2) Fig. iillstrraess this effect through some randomly picked high and low engagement image samples. Further, one may think that the preferences of image creators (e.g., in the form of upvotes on platforms like Pick-a-Pic (Kirstain et al. (023) or Discord (Wu et al. 2023)) are good estimate of image-consumer engagement. However, we find that PickScore and HPS. the reward models trained on a large dataset of creator preferences, correlate 0.07 with user engagement. Therefore, there a lack of reliable metrics capturing viewer engagement on images.
The lack of progress can largely be attributed to the absence of large and open dataset of customer engagement metries over images. The most common image generation datasets, MS-COCO (Lin et al.) 2014) and LAION (Schuhmann et al. (022), contain no signals for user engagement. Therefore, to spur research in the direction of measurement and optimization of image generation for user engagement, curate large-scale dataset, engaginglmageNe. of 168 million tweets capturing 17 years of high-quality enterprise images for over ten thousand brand accounts and average to serve as a starting point for measuring, benchmarking. and modeling large-scale engagement- optimized image generation.
EngaginglmageNNer ($2) consists
EngaginglmmaeeNet ImageNet engagement of images in the form of like{" We release Engaging
NngageNet as a scoring function to score engagement: EnaginglmageNNet allows us to train a scoring function that estimates the user engagement on a particular generated image. We formulate this problem as simulating the engagement in the form of user likes over an image-containing tweet (33.2) We carry out visual instruction finetuning of VL- 13B (Liu et al. 2023) model to estimate the wrand-normalinn input account handle, image description, and time of the tweet. We find that the resulting scoring model, EngageNet, achieves a high correlation of 0.62 with actual user engagement.
likes given the image along with contextual information that includes
Engagement Arena: Next, leveraging EngageNet as judge, on the lines of LMSYS arena (Zheng et al., 2024 Chiang et al.) 2024), we propose Engagement Arena, an arena where we test the engagement of images generated by various image generation models for the same prompt. Using EngageNet's reward estimates, we compute Elo ratings of number of popular open-source text-to- image generation models, including Stable Diffusion-3 3 (Esser et al. 2024). Flux. (Labs) 2024). Stable Diffusion-XL (Podell et al. (023). SDXL-DPO (Wallace al. 2024). PixArt-alpha (Chen 2024a), Stable Diffusion 2.1 (Rombach et al.) 2022). etc,
et al. 2024b), Pixart-sigma (Chen et al. and closed-source models like DALLEE2 (Ramesh et al.) (002)). Further, encourage the research community to adopt Engagement Arena as a basis for measuring further advances in the engagement capabilities of text-to-image modeling and incorporating user engagement into the learning process.
Optimizing the Image generation process with the goal of increasing engagement: Finally, we explore train-time and run-time methods to induce the goal of engagement in the text-to-image generation process: (1) Run-time: conditioning the diffusion model on prompts aligned with higher user engagement, (2) Train-time: fine-tuning the diffusion model on tigh-engagement images, and (3) Train-time: aligning the diffusion model with Engage rewards via reinforcement learning. We present the results of these experiments in Section and report the efficacy of each method in generating more engaging images.
To summarize, we make the following contributions:
'FngagingllmaeNNtt was collected using Twitter API over a period of several years.
2

---

Published as a conference paper at ICLR 2025
B MiT CNN
T
Google
Cner fibrr
Cenderiiiie Lifi teaat hwee gl)
2011
uuptin
Figure 2: Figure illustratingg the steps involved in the creation of the EngaginglmageNer dataset.
We introduce the problem of engagemeent-sttimizzd image generation. Images, especially in industries like advertising, fashion, and e-commerce, created to achieve user engagement in the form of clicks, likes, and purchases. Therefore, the image generation process needs to be biased on the image's eventual utility, in addition the common goals of high aesthetics and fidelity.
2. We curate EngaginglmageNett a large-scale, high-quality dataset consisting of user engagement over images. EngagingImageNG consists of 168 million tweets collected from 10,135 enterprise Twitter accounts from the time period 2007 to 2023. It consists of the account name, tweet text, media posted with the tweet, image captions, keywords, colors and tones, the time of posting, and the number of likes the image received. The dataset is instrumental in our study of image engagement as the utility in real- world marketing scenarios.
3. We train an engagement-awee aware vision language model (VLM), called EngageNet, to predict user engagement over images. EngageNet exhibits strong performance in estimating user engagement compared to other commonly used metrics like FII) and aesthetics for evaluating the performance of text-to-image generation models as well as state-of-the-ar LLMs like GPT-3.5 and GPT-4V.
4. Using EngageNet's predicted engagement scores as a reward, we introduce Engagement Arena, the first automated arena to benchmark the engagement text-to-image models. We rank several popular text-to-image models their ability to generate engaging images and further encourage the community to submit their models to the arena.
5. We demonstrate introducing the goal of engagement in the text-to-image generation process. We present several approaches to achieve this. These include conditioning of on prompts corresponding high user engagement, supervised fine-tuning of stable diffusion high-engagement images, and reinforcement learning to align stable diffusion with EngageNet- rewards, all of which lead to the generation of more engaging images varying degrees.
generation
EngaginglmageNNt: DATASET WITH IN-THE- WILD USER ENGAGEMENT
To gain insights into image engagement and align text-to-image generation with user engagement, we start by collecting a large dataset of user engagement over images. Our data collection method involved leveraging Twitter, platform extensively utilized by brands for various purposes such as ongoing product campaigns, sales, offers, discounts, brand building, and community engagement (Alalwan et al. 2017). Twitter user engagement metrics encompass user likes, retweets, comments, mentions, follows, clicks on embedded media, and links. However, the Twitter API provides access to only user likes, retweets, and comments for a given post, with access to comments necessitating separate and costly call. Therefore, utilizing API licenses, we extracted the following data from Twitter: Tweet ID, company name, username, timestamp, tweet text, media files, and user likes.
We focus on enterprise handles for our data collection efforts since the content released by enterprises has the explicit goal of user engagement and is relatively much cleaner than user-generated content. We began by compiling a comprehensive list of company names using the Wikidata knowledge graph Wikiidtaa contributors ongoing). focusing on entities categorized as 'business' or 'enterprise'. We conducted Google searches to gather list of all associated accounts for these companies. For example, for Adobe, this encompassed accounts like Adobe, Adobe Photoshop, Adobe Lightroom, Adobe Experience Cloud, and so forth. This method enabled us amass a total of 10,133 enterprise Twitter handles. We then utilized the Twitter API to retrieve tweets posted by these enterprises spanning from 2007 to the closure of the Twitter API in January 2023. This effort resulted in the collection of 168 million tweets over 17-year period, with 28.5 million of these tweets featuring various forms of media, including GIFs, images, and videos. Fig. shows several examples media and tweets present in the EngaginglmageNNt.
Next, for each username, we bin the tweets falling in the bottom 60 percentile (and having absolute likes 20), 60-90 percentile (and having absolute likes > 30), and 90-100 percentile (and having
3

---

Published as a conference paper at ICLR 2025
Table 1: A comparison of datasets containing image preferences
Dataset
Size
Pick aPic (Kirstain al. 2023
968,965 rankings originated from 66,798 prompts and 6,394 users Total of 98, 807 images generated from 25, 25,205 text prompts
Humun Preference Score (Wa et al. 2025
Annotations for 8878 text prompts and corresponding model outpats sampled from DiflusionDB, resulting in 136,892 compared pairs
ImageReward (uu 2024
EnguginglnugeNet (Ouns)
28.5 million tweets containing media, captions, colors, tones, and objects, and with user likes as engageiment metric
absolute likes > 40) of all tweets per account based on the number of user likes. These buckets are subsequently referred to as 'low', 'medium" and 'high' liked buckets, respectively. The resulting dataset, Inagggn,,maaeNet,, consists of 837,532 samples, having 144,905 high-liked images, 336,200 medium-liked images and 356,427 low-liked images. The high-liked tweets had average of 2435 likes, while low-liked tweets had an average of 132 likes. Subsequently, all images were verbalized by extracting their captions using LLaVA (Liu et al. 2023). colors and tones along with their coverage us- ing (Qin et al. 2020). Further details regarding the data processing are provided in Appendix F
Ethics Statement: We have taken measures ensure that the EngagingImagett dataset does not contain any personally identifiable information (PII). The dataset contains only public, non- individualized content following ethical considerations, data protection guidelines, and Twitter API guidelines. To ensure these, we: (1) Collect tweets and aggregate user likes from accounts identified using the Wikidata Knowledge Graph and marked as "enterprise" or "business", (2) Don't collect any personal or sensitive user information, (3) Remove references to specific users or personal identifiers (such as through using account handles), (3) Collect only aggregate engagement metries (e.g.. overall user likes data) to analyze engagement trends without tracking or identifying individuals. As part of our commitment to responsible research, EngagingImageNee will be open-sourced in phases, starting with smaller controlled releases, while actively engaging with the research community to identify and address any potential concerns. We will continue to review compliance with data protection guidelines and encourage the adoption of similar ethical best practices within the community. We discuss this in more detail in Appendix L
3 EngageNer: MEASURING IMAGE ENGAGEMENT
Simulating engagement is inherently challenging, as prior research suggests that both experts and non-experts struggle accurately predict engagement outcomes [Tetlock) 2017 for 2023) Tan et al. 2014] Isola et al., 2013] Singh et al., 2024). These findings highlight the necessity of an automated and reliable metric for estimating image engagement. In this section, we cover the alignment of existing metrics with viewer engagement and design a model to measure the progress of image generation models with the goal of engagement.
3.1 ALIGNMENT OF EXISTING MODELS WITH VIEWER ENGAGEMENT
In order measure the engagement potential of generated images, we first check the alignment of the most popular existing metries used for evaluating text-to-image models with viewer engagement. For this, we calculate the Pearson correlation of Aesthetic score (Schuhmann et al. 2022). CLIP score (Radford et al. (02T), Pickscore (Kirstain et al. 2023). ImageReward (Xu et al. 2024), and Human Preference Score (HPS) (Wu et al. 2023) with ground truth brand-wise normalized user likes (0-100) from IagaginglmmaeNee Table presents the results of this analysis. Clearly, the existing metrics are not aligned with user engagement.
This non-alignment can be attributed to the following reasons. Models like PickScore, HPS, and RewaeR are optimized for creator preferences captured on platforms like Discord custom web applications rather than user (viewer) preferences. The feedback from image creators or communities on these platforms tends to reflect artistic or stylistic biases that do not necessarily correlate with user engagement metrics like clicks, likes, or shares. Further, these models evaluate images in isolation, without considering the contextual information about the image, such as company, time of releasing the image, etc., reducing their effectiveness in predicting user engagement. This suggests that the existing metrics are not designed to capture the user engagement of images. Recent studies have employed the CLIP model as proxy for human judgment (Nichol et al. 2022 Rombach et al. 2022), aiming to assess the alignment between generated images and text prompts. CLIP, trained on diverse dataset, is thought to better capture nuanced aspects of human intention. However, similar to ImageReward and PickScore, the text prompts for CLIPScore come from image creators rather than users (viewers), which again is ineffective in conforming viewers' expectations. The aesthetic
4

---

Published as a conference paper at ICLR 2025
Table 2: Pearson correlation between model predicted scores and image engagement measured by account normalized likes.
Model
Configuration
Pearson Correlation
PickSoore Kirstain et
2023
0.0734
ImageReward Xu
2024
0.0285
Human
Preference Soore Wa et ml.
2025
0.0747
Aesthetic Score
(chuuhmann
2022
0.0674
CLIP Score
Rodford
2021
0.0423
GPT335
Onyang et al.
2022
3-shot In-contesi learning
0.0464
GPT-3.5
Ouyyan
2022
S-shot In-contesi learning
0.085
GPT-4V
OpenA]
3-shot In-cuniesi learning
0.1453
GPT-4V
OpenA1
2023
5-shot In-cunicat learning
0.1264
EngageNet
Trained on maddom engagement score, Tested ongaggement score
0.0617
EngageNet
Trained without MSE
0.5821
EngageNet
Trained without date input
0.5365
EngageNet
Trained without company impul
0.5226
EngageNet
Trained without
compony input
0.4476
EngageNer
Truined without negative samples
0.6051
EngageNNt
Trained with MSB loss and negative samples
0.6248
EngageNet
Oracle
0.3682
score, built on pre-trained CLIP, is trained on several datasets capturing image aesthetics. However, as Fig. [ shows, viewer engagement is much more nuanced than what aesthetics can capture.
Next, we try in-context learning with GPT-3.5 (Ouyang et al. 2022 and GPT-4-Vision (OpenAl 2023) to predict viewer engagement over images. For GPT-3.5, we supply the image verbalization along with the Twitter handle and posting time, and for GPT-4- image verbalisation, the Twitter handle and posting time. We find that neither GPT-3.5 nor GPT-4 are able to predict user engagement accurately.
we give the actual image, the
3.2 ENGAGENET MODEL TO ALIGN WITH VIEWER ENGAGEMENT
Since the existing approaches do not show acceptable performance for predicting user engagement over images, we, therefore, train own engagement-awas vision-languse model (VLM) model, EngageNet. To this end, we perform visual instruction fine-tuning of LLaVA-1.55 (Liu et al. 2023) on the EngaginglmageNNt train dataset (Figure TOP. We design an instruction (Listing] 1D for the VLM to predict the normalized likes of an image on a 0-100 scale, also conditioned on metadata comprising the marketer (company), image resolution, image colours and tones with their spatial coverage, image description and tags, and the date of releasing the image on social media.
We also augment EngaginglmageNet with synthetic data samples. For this, we randomly sample 25% tweets from the high and medium likes buckets of each company and pair the tweet with an unrelated image from a different tweet. The corresponding likes is set to a low value, randomly sampled from the range 5-15. The resulting samples are called negative samples. This helps the model with the following: (1) it induces more sensitivity towards image features and reduces bias on tweet metadata to predict the engagement, and (2) penalises the image if it is irrelevant to the tweet context. We discuss this in more detail in Appendix [ Finalll, we end up with a dataset of 957,809 samples, which we split into training and testing sets. We randomly sampled nearly 2000 samples from each bucket for testing, with the remaining samples used for training.
Since EngageNet is trained to predict the engagement score of an image, we additionally model the problem as a regression task. We attach a two-layered ML. network on top of the last layer of hidden states of the decoder module predict the scalar engagement score from EngageNet. Therefore, while typically language models are trained on cross-entropy loss LCE,, we also use mean squared error LMSE as an auxiliary loss train EngageNet. This because LMSE is more sensitive to the difference between the predicted and actual engagement values, which is crucial to better guide EngageNet learn the image engagement prediction task. Thus, the final loss function for EngageNet is given by:
LEoIG = LCg + ALMSE
where À is hyperparameter that controls the weight of the auxiliary loss. We set À 0.1 in our experiments. We find that EngageNet demonstrates strong performance at predicting user engagement over images, achieving Pearson correlation of 0.62 with ground truth user likes (Table 2
5

---

Published as a conference paper at ICLR 2025
We perform an ablation study to understand the impact of different components of the instruction on the performance of EngageNet. If EngageNet is not supplied with contextual information such as marketer company and the time of posting the image in the input, the correlation of its predictions with ground truth account normalized likes drops significantly (0.62 to 0.44). This indicates that the company and the time of posting are important components of the instruction for EngageNet to predict user engagement accurately. We also attempt to determine the impact of the auxiliary MSE loss adopted for training EngageNet. The MSE loss increases the correlation from 0.58 0.62, indicating that the auxiliary loss improves the performance of EngageNet. The MSE loss makes the model more sensitive to the difference between predicted and actual scores.
We also conduct an experiment to investigate the signal present in the EngaginglmageNet dataset. For this, we train EngageNet on the IngaginglmageNer dataset but with randomly sampled engagement values. We then evaluate the model on the In this case, the correlation of EngageNet's predictions with ground truth user likes is nearly zero, indicating that the engagement values in the EngaginglmageNet dataset are crucial for EngageNet to learn predicting user engagement accurately.
test dataset with actual KPI values.
Additionally, we evaluate the impact of adding negative samples to the training data. These samples are constructed by sampling images and other inputs in the instruction from different tweets, such that they are not aligned, and then setting the normalized likes to a very low value. Although find that the addition of negative samples does not significantly impact the correlation of EngageNet, however it does help in improving the robustness of the model. This is because EngageNet learns to penalize images that are not aligned with the other inputs in the instruction. This is crucial for leveraging EngageNet as a reward model for engagement-opiimie in Section 4.3 Since we propose to also utilize EngageNet as an oracle for ranking models in the Engagement Arena, we train EngageNet on the entire EngaginglmageNet dataset, i.e., with both train and test data. In this configuration, EngageNet accomplishes a high correlation of 0.87 with ground truth user likes, which establishes its effectiveness to be used as an oracle.
image generation as described
3.3 GENERALIZATION CAPABILITY OF ENGAGENET
To assess the generalization ability of EngageNet across different social media platforms and engage- ment metrics, we evaluate its performance on the FlickrUser dataset (Abdullahu & Grabner 2024). This dataset comprises 504,241 images from 2,337 users on Flickr, a popular photo-sharing platform. It includes engagement metrics such as the number of 'favorites' and number of 'views'.
From the full dataset, we use only 15,000 images for finetuning and a 5,000-image test set. As a baseline, we first fine-tune the pretrained LL. Vision-Language Model (VLM) to predict the normalized 'favorites' (number of 'favorites' number of 'views') as the engagement score of an image, leveraging contextual metadata such upload date, description and tags. This model achieves a correlation of only 0.23 with the actual engagement scores in the test set.
Next, we finetune EngageNet for the same task. We find that despite being trained on minimal data, EngageNet significantly outperforms the baseline, achieving a correlation of 0.53 with actual image engagement scores. This substantial performance gain highlights EngageNet's 's ability to transfer its learned knowledge of image engagement to new contexts, demonstrating strong generalization across (1) different engagement metrics (favorites vs. likes), (2) alternative social media platforms (Flickr vs. Twitter), and (3) content types (enterprise vs. non-enterprise). These findings reinforce the robustness of EngageNet and its potential for broad applicability beyond its original training data. We also conduct a human study to assess the alignment between EngageNet's predicted engagement rankings and human preferences; detailed methodology and results are provided Appendix C
4 METHODS TO IMPROVE IMAGE ENGAGEMENT
We explore three methods for optimizing the text-to-image generation process to generate more engaging images. These include run-time and train-time optimizations: conditioning of text-to-image models on better prompts, supervised fine-tuning of stable diffusion on high-liked images, and reinforcement learning to align stable diffusion with EngageNet-basedd reward scores. The first method operates in the natural language domain at run-time, generating a description of how an engagement-optimizze image should look like. On the other hand, the other two operate in the vision domain, generating actual ingagement-optimiin pixels by training the U-Net module of stable diffusion. We cover each of them next.
6

---

Published as a conference paper at ICLR 2025
4.1 CONDITIONING STABLE DIFFUSION ON MORE ENGAGING PROMPTS
In the Engaging dataset, we observe that some images having similar themes but different details received vastly different levels of user engagement. For instance, consider the following pair of image captions: (1) "A living room having a couch and coffee table with a rug in front." (2) "A living room with large windows having couch, coffee table and rug" Despite both images depicting a similar scene (Figure I. the first image received low engagement, while the second image garnered high engagement. In this case, the difference in engagement can likely be attributed to the presence of elements, such as large windows and natural light in the second image, which makes the living room appear bigger and more appealing to a viewer. Such observations motivated us to exploit patterns related to certain image aspects that can boost engagement. We further extend this analysis for images posted by a few companies in Appendix D
Therefore, in this method (Figure LTI. we attempt alter the text prompts fed to the diffusion model such that the improved captions incorporate characteristics that have been empirically shown to boost image performance. For this, we adopt retrieval framework described as follows. Using FAISS (Johnson et al., 2019 Douze et al. 2024), we index the vector embeddings of captions belonging to im- ages in the high performance data subset of the EngaginglmageNet train data described in Section 2 Next, for every image caption in the low performance subset of the test data, we retrieve the seman- tically most similar caption from the corpus of high-perfie performing images. If the similarity level is above a certain threshold т, the retrieved captions thus obtained are passed as input to the diffusion model for generating more performant images, otherwise the original caption is used for image generation.
4.2 PREFERRED FINETUNING ON HIGH- BGGG-EENAGGMENT IMAGES
Several prior studies have demonstrated the feasibility of learning styles through stable diffusion by fine-tuning the model (Pinkney 2022 Cjwbw 2022 PromptHero 2023 Everaert et al. 2023).
These approaches typically involve finettuning the U-Net architecture within the Stable Diffusion framework using a set of images exhibiting the desired style. For instance, |Everaert et al. (2023)
proposed method finetune Stable Diffusion to adapt it to target styles like anime sketches, American comics, Pokemon, starry night, etc.
In this work, attempt explore whether the diffusion model can learn patterns associated with higher user engagement, analogous to learning visual styles. To this end, we performed fine-tuning of the base Stable Diffusion U-Net on the preferred data distribution, containing high liked images sampled from the EngaginglmageNNet train set (Figure [2].. We call this process, Preferred Finetuning. The model was finetuned for 50 epochs, following the procedure outlined by von Platen et al.) (2023). The model minimizes the standard denoising score matching loss (Ho jet al., al.]][02))HH Ho & Salimans, 2022), which measures how well the model predicts the noise added to the image during the diffusion process:
Lieense = Exo.e.t [|le-E(xx,,,,c t, c)
Table 3: Results reveal the significant gains achieved in improving the engagement of low-liked subset of the EngaginglmageNt dataset by enhancing the image descriptions fod to text-to-image model, as described in Section 4.1
Images
Training Config
Oracle Engagement Reward
Engagement reward increase
prompt improvement
w/ promut improvement
DALLEE-2
N.A.
42,1459
45.7077
8.45%
SD 1.4
N.A.
38.7680
43,8173
13.02%
FOIG SD 1.4
RLHF-ES
40.2037
46.1218
14.72%
EOIG SD 1.4
RLHF-DSG
39.4950
44,8116
13.46%
EOIG SD 1.4
Preferred Finetuning (PFT)
43.1206
46,6490
8.18%
SD 1.5
N.A.
39.4001
44.3287
12.51%
SD 2.1
N.A.
45.4952
49.6946
Pixart-alpha
N.A.
44.7305
49,3322
10,29%
Pixart-sigma
N.A.
46.2870
51.4671
11.19%
SD XI.
N.A.
49.1094
53,8602
9.67%
SD XL DPO
N.A.
51.3786
54,5863
6.24%
SD Medium
N.A.
50.6578
55,0841
8.74%
1lux.1-dev
N.A.
48.7226
53,7209
10.26%
Ground Truth
N.A.
41.2152
56.7533
37.70%
Average Increase in Engagement
12.41%
7

---

Published as a conference paper at ICLR 2025
where Xy is the original image, X₂ is the noisy image time step t, generated by adding noise e, Eo Ex,,, t, c) is the predicted noise from the model given the noisy image Xt, time step t and conditioning information i.e., text prompt fed imput to the diffusion model.
4.3 ALIGNING STABLE DIFFUSION WITH ENGAGEMENT
Black et al. (023) proposed denoising diffusion policy optimization (DDPO), a policy gradient algorithm which frames the denoising process as multi-step decision-making problem. The authors showed that DDPO can be employed to finetune exxt-to-image diffusion models to align their outputs with a variety of reward functions including image compressibility, aesthetic quality and image- prompt alignment, among others. Therefore, we explore the use of reinforcement learning to optimize diffusion models improve the engagement potential of their generated images. To this end, we leverage EngageNet as a reward model to align pre-trained stable diffusion model using DDPO algorithm to produce more engaging images. The entire process of alignment is shown in Figure 14 in the appendix. In DDPO, the denoising process is viewed as a finite horizon Markov decision process, where the state comprises of the current context, number of steps left in the process and the current denoised image. The action to be taken is to predict the next image using this state.
experiment with two types of reward functions for finetuning stable diffusion:
(I) Engagement Simulation (ES): We leverage EngageNet to estimate the engagement of images generated by stable diffusion. The reward signal used to guide stable diffusion to generate higher engagement images illustrated in Figure 14a The resulting diffusion model is called EOIG-SD (RLHF-ES).
(2) Design Specification Generation (DSG): We train alternate version of EngageNet to produce the design specification of an image, based on conditioning factors such as the company, time, image caption and viewer likes. This model learns to predict verbalized image descriptions comprising colors and tones with their spatial coverage, well objects with their locations, that should be reflected in an image, for given engagement level and caption. The detailed method and results of EngageNet trained on this task are explained in Appendix J Next, we utilise this EngageNet as a reward model to train stable diffusion such that the images generated by it have a design specification aligned with those of higher engagement images as shown in Figure 14b) FOIG-SD (RLHF-DSG) takes text prompt and generates an image, which then undergoes verbalization via image perception models. Its objective is to create images that, when verbalized, closely resemble the mngaemment-conittin
verbalization generated by EngageNet. Thus, we ask EngageNet to provide the logits for this image verbalization, using which reward is computed for EOIG-SD, indicating how closely this verbalized output aligns with EngageNet. This reward value serves as feedback for EOIG-SD in the form of policy gradient, aiding in its continual improvement and refinement within the image generation process. Only high engagement samples are used in the training process. The details of this method are deseribed in Appendix
4.4 EVALUATING THE METHODS ADOPTED FOR ENGAGEMEETT
Run-time optimization: Firstly, we investigate the impact using better prompts to condition
the text-to-image generation process described in Section 4.1 The results are summarised in
Table 4: Comparing the performance gains on the EngaginglmageNNet test dataset, resulting from train-time engagemeentoptimizzaion methods applied on stable diffusion, described in Sections 4.2 and
Images
Training Config
Bucket
Engagement
Reward
Engagement Increase
Aesthetic Score
CLIP Score
FID
PickScorv
High Medium Low
90,9526
N.A.
5.1006
3266343
N.A.
20.9470
41,2152
N.A.
5.0518
32,2012
N.A.
20.7406
High 56.1489
N.A.
5,2029
33.0830
24,6631
17.3514
SD 1.4
N.A.
Medium
$1.6949
N.A.
5.1634
32,9173
23.4434
12.3344
38.7680
N.A.
24,2607
Prefersed Finctuning (PFT)
High
62,0390
10.49%
4 4.8090
32.4524
24.9370
17.3070
EOIG SD
Medium
56.1082
8,54%
4.8387
32.3913
23.0904
17.3239
High engagement Images
23,5885
High
58.2724
3.78%
5.1828
33.220 28.2
23,8656
17.4113
BOIG
RLHF
Medium
53.1004
2.72%
22.7157
17.3802
Low
40.2037
3,70%
5,1629
32.9468
24.1780
17.3672
High
57,9188
3.15%
5,2495
33.1077
23.9144
17.3577
BOIG
DSG
Medium 52.9765
23.3626
17.3486
Low
39.4950
1.88%
5.2336
32.9716
24.2147
17.3277
8

---

Published as a conference paper at ICLR 2025
Table 3 By retrieving semantically similar captions from the corpus of high-liked images, visual characteristics that have been empirically shown to enhance image engagement get incorporated in the text prompt. Therefore, after applying this method, observe significant improvement in the engagement of low-liked subset of the EngaginglmageNNe test dataset, consistent across multiple text-to-image models, both open-source and closed-source. This method is highly effective as it is able to produce images with higher engagement without any additional training of the diffusion model. We observe that, on average, an improvement in the prompt results in an improvement of 12.4% in engagement. The improvement is observed in models across all sizes and also for models trained on high-engagement images (EOIG-SD).
adopted for engagement-optimizee image generation by training the U-Net module of stable diffusion in Table We denote all the models trained using train-time optimizations like Preferred fine-tuning with EOIG (engagement optimized image generation). We compare the performance of the base stable diffusion model (SD 1.4), stable diffusion finetuned on high-engagement images (EOIG-SD PFT), and stable diffusion aligned with EngageNet- reward functions (EOIG-SD RLHF-ES, EOIG-SD RLHF-DSG). For this, we use EngageNet--raan images generated by these models. This helps us probe the effect of different training strategies on improving the engagement capabilities of stable diffusion. Consistent with prior literature, also include other metrics like FID (Heusel et al. (017), aesthetics (Schuhmann et al. 2022). CLIP score [Radford et al. 202I), and PickScore (Kirstain al. 2023).
Train-time optimization: Next, we present the results of the methods 4$4.
$4.3)
judge to predict the user engagement of the
The results indicate that while all the training methods improve the engagement capabilities of stable diffusion, however, the extent of improvement varies widely. We find that finetuning the stable diffusion model on preferred data distribution, i.e. high-engagement samples from the EngagingIm- ageNet dataset yields significant gains in the engagement potential of the generated images. This is evident from the consistent increase in the predicted user engagement of the generated images across all engagement buckets. Next, we discover that using EngageNet-based reward functions to align the stable diffusion model also results in better performance. However, the improvement in image engagement is not as significant as that achieved by the previous methods. Other metrics like CLIP score, PickScore and FID do not vary significantly across the EngaginglmageNV buckets and largely remain unaffected after training stable diffusion in both the above regimes. This further corroborates their non-alignment with image engagement. We also discuss the results of applying train-time optimizations on other text-to-ing to-iaae models in Appendix
Next, discuss the side effects of training stable diffusion using the above methods. As conse- quence of training on engaging images, we find that stable diffusion learns to generate images with certain persuasion strategies (Kumar et al. 2023). For instance, Fig. [3] shows several examples of product and model photography generated by EOIG-SD and base SD, demonstrating EOIG-SD's biases towards certain persuasion strategies such social appeal and social identity, commonly observed in marketing scenarios (Kumar et al. 2023) but ignored in general photography.
our experiments, we gauge the impact of different methods in improving image engagement by comparing the results of both train-time
Combination of Train-time and Run-time optimizations:
(44.2 $4.3) and run-time ($4.T) optimizations, as well as their combination. Stable Diffusion 1.4 (Rombach et al. 2022), serves as the baseline model. In Table 3 we observe that when each method is applied individually, such as using better prompts at run-time or training the diffusion model through supervised finetuning using reinforcement learning, it results in measurable improvements in image engagement over the baseline. However, the most significant improvements are seen when supervised finetuning reinforcement learning is combined with better prompts run-time. This demonstrates that coupling train-time and run-time optimizations has a synergistic effect, resulting in higher engagement levels than each method applied alone.
Engagement Arena: MEASURING ENGAGEMENT CAPABILITIES OF TEXT-TO-IMAGE MODELS
Motivated by the work of LMSYS and similar benchmarks CChing et al. 2024). we propose Engagement Arena platform to evaluate the capability of text-to-inne -mage models to generate engaging images. We run a tournament on a common set of prompts from the EngaginglmageNet test set. We leverage EngageNet as oracle for Engagement Arena to compute the Elo ratings of various

---

Published as a conference paper at ICLR 2025
open-source text-to- models, such Stable Diffusion 3 Medium (Esse al. 2024). Flux. 1 1-dev
(Labs] 2024), Stable Diffusion XL (Podell et al. 2023), Stable Diffusion DPO (Wallaee et al.) 2024), Pixart-sigma (Chen et al., 2024a), Pixart-alpha (Chen et al.) 2 Stable Diffusion 2.1, Stable Diffusion 1.5, Stable Diffusion 1.4, (Rombach et al., (022). etc., and closed-source models
like DALL.E-2 (Ramesh et al., 2022). Figure shoowss the rankings of these models. It also features the Elo ratings of ground truth images to serve as topline for benchmarking the models.
In addition helping rank the engagement potential of generated images accurately, using EngageNet as an oracle also avoids having static benchmarks with definitive ground truth. We encourage the research community to adopt Engagement Arena basis for measuring further advances in the engagement capabilities of text-to-image modeling and incorporating user engagement into the learning process.
The arena features actual images from the EngaginglmageNet dataset topline benchmark for the images generated by different text-to- ext-tto-mmae models. We find that Stable Diffusion Medium (Esser et al. 2024) emerges as the best performing model in the Engagement Arena, with win rate of 46% over actual images (Figure 15). It is followed by SDXL-DPO (Wallace et al. 2024) and Flux. 1-dev (Labs) 2024). We notice general trend that image engagement rises with the size of the text-to-image models. However, there are some exceptions to this trend. For instance, Pixart family of models (600M parameters) and EOIG-SD PFT model (860M parameters) surpass relatively larger DALL.E-2 (6.5B parameters).
We observe that while our BOIG-SD models trained using different methods (PFT, RLHF) outperform the equal-sized base SD 1.4 model, however there is a considerable gap between the performance of EOIG-SD and significantly larger text-to-image models leading the arena. This can be attributed to the inherent limitations of SD 1.4 in generating high-quality images, which cannot be fully overcome by the training methods explored in this work.
CONCLUSION
As text-to-image generation models continue evolve, their success must be measured not only by aesthetic quality and realism but also by their ability to drive meaningful engagement. In this work, introduced the challenge of optimizing image generation for engagement, crucial factor in e-commerce and advertising. Our findings reveal that traditional image evaluation metrics, such as aesthetics and CLIPScore, fail correlate with user engagement, necessitating new approaches. large-scale dataset that captures real-world user
To address this, curated EngaginglmageNNe engagement over images, and developed EngageNet, a vision-language model designed to predict user engagement. By leveraging EngageNet reward function, we explored multiple strategies to enhance engagement in -xt-tt--mmae models, including prompt optimization, fine-tuning on high-engagemmnt images, and reinforcement learning alignment with engagement- baaseennt-baeddd rewards. These methods demonstrate promising improvements in generating images that resonate more with viewers. Furthermore, enables the research community to evaluate and compare text-to-image models based on engagement performance. We provide detailed discussion of broad directions, along with promising future directions for real-world applications in Appendix B
introduced the Engagement Arena, benchmarking platform that
1120
color
1100
1070
1002
1039
1050
2
1000
984
967
1000
943
S79
050
900
Gound Trub
50 Medium Boter Fromps
50-3
5DX, ORO
Fuu.1-DW
501580014 PST
50
DALLER
EDIG RUNF-D0O
EDIS RUIFFES
S0.14
Prompo
Model
Figure 3: Rankings and Elo ratings of various text-to-image models in the proposed Engagement Arena.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Insights into the accuracy of social scientists' forecasts of societal change. Nature human behaviour, 7(4):484 (00)4444001, 2023.
Fitim Abdullahu and Helmut Grabner. Commonly interesting images. In European Conference Computer Vision, pp. 180-198. Springer, 2024. 6
Ali Abdallah Alalwan, Nripendra P Rana, Yogesh K Dwivedi, and Raed Algharabat. Social media in marketing: A review and analysis of the existing literature. Telematics and informatics, 34(7):
1177-1190, 2017.
Aanisha Bhattacharya, Yaman K Singla, Balaji Krishnamurthy, Rajiv Ratn Shah, and Changyou Chen. A video is worth 4096 tokens: Verbalize story videos to understand them in zero shot. arXiv
preprint 0.X7022305.09758 2023. 23
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine Training diffusion models
with reinforcement learning. arXiv preprint 23XX::2300513331,, 2023. 8
15
26
31
Jesús Bobadilla, Fernando Ortega, Antonio Hernando, and Abraham Gutiérrez. Recommender
systems survey. Knowledge-bused systems, 46: 46:109-132, 2013. [
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-\ ssigm:: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403. 04692, 2024a.
TO
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- StalphaS: Fast training of diffusion transformer for photorealistic text-to-image synthesis. The Twelfth International Conference on Learning Representations, 2024b. URL
www. feet//forummdddeekknPee....
T0
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos. Tianle Li, Dacheng Li. Hao Zhang. Banghua Zhu, Michael Jordan, Joseph Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating lims by human preference, 2024. https://///: org/
abs/2403. 04132
2
Cjwbw. Waifudiffusion. https: http:///cattt.ccco////////t///// wttp:///www//aaauuudiuoo/oo
2022.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In The Twelfth International Conference on Learning Representations, 2024. httphtttss///peereeeee www.nnee/formmmiiiiiittttt
15
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances
in neural information processing systems, 118888087794 2021. [
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at seale. arXiv preprint
arXiv:2010.11929. 2020. 23
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy,
Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024. 7
Zhenbang Du, Wei Feng, Haohan Wang, Yaoyu Li, Jingsen Wang, Jian Li, Zheng Zhang, Jingjing Lv, Xin Zhu, Junsheng Jin, al. Towards reliable advertising image generation using human feedback. In European Conference on Computer Vision, pp. 415. Springer, 2024. 15
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
T0
Martin Nicolas Everaert, Marco Bocchio, Sami Arpa, Sabine Süsstrunk. and Radhakrishna Achanta. Diffusson in style. In Proceedings of the IEEE/CYF International Conference on Computer Vision,
pp. 2251-2261, 2023. 7
11

---

Published as a conference paper at ICLR 2025
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, (3(11):139-144 2020. 15
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 9 T5
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12596 2022. 7
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. 7
Phillip Isola, Jianxiong Xiao, Devi Parikh, Antonio Torralba, and Aude Oliva. What makes photograph memorable? IEEE Iransactions on pallern analysis and machine intelligence, 36(7): 1469-1482, 2013.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions Big Data, 7(3):535-547, 2019.
Ashmit Khandelwal, Aditya Agrawal, Aanisha Bhattacharyya, Yaman K Singla, Somesh Singh, Uttaran Bhattacharya, Ishita Dasgupta, Stefano Petrangeli, Rajiv Ratn Shah, Changyou Chen, et al. Large content and behavior models to understand, simulate, and optimize content and behavior.
arXiv preprint (3ti::23000003555 2023. 23
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pie: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663 2023.
15 16
James Kite, Lilian Chan, Kathryn MacKay, Lucy Corbett, Gillian
Binh Nguyen, William Bellew, and Becky Freeman. A model of social media effects in public health commu- nication campaigns: systematic review. Journal of medical Internet research, 25:e46345, 2023.
Yaman Kumar, Rajat Jha, Arunim Gupta, Milan Aggarwal, Aditya Garg, Tushar Malyan, Ayush Bhardwaj. Rajiv Ratn Shah, Balaji Krishnamurthy, and Changyou Chen. Persuasion strategies in advertisements. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 9
Black Forest Labs Announcing black forest labs, 2024. URL https://blakfoorsttaaas ai/
ammouncing-black-forsss blost-ooest---aaaas
TO
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image
pre-training with frozen image encoders and large language models, 2023. 23
Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, Rich human feedback for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp.
19401-19411, 2024. 15
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304 0084422,0 2023.
23
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pp. 16784-16808 PMLR, 2022. 4
OpenAI. Gpt-4 technical report, 2023. 5
15
18
23
12

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
15
Justin Pinkney. Text-to-pole potk-to-ookemon. http:://reollateeccc/mmmmm/////a
pokemon, 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv::330.001952, 2023,
2 10 22
PromptHero. Openjourneyy2.
2023.
Xuebin Qin, Zichen Zhang, Chenyang Huang. Masood Dehghan, Osmar Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. volume 106, pp. 107404, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PMLR, 2021.
2 5
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024. 15
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021. 15
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3 2022, 2
10
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the JEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
22
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan. Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information
Processing Systems, 35:36479-36494, 2022. T
Tim Salimans. Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29,
2016. 15
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 15:25278-252955 2022.
15
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
15
Somesh Singh, Yaman K Singla, Harini SI, and Balaji Krishnamurthy. Measuring and improving persuasiveness of large language models, 2024. URL https://arxiv.orgg awwp////b//22..... 02653
18
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
13

---

Published as a conference paper at ICLR 2025
Chenhao Tan, Lillian Lee, and Bo Pang. The effect of wording message propagation: Topic-and author-controlled natural experiments on twitter. arXiv preprint 1405. 2014.
Philip E Tetlock. Expert political judgment: How good is it? how can we know?-new edition. 2017.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 11922.1997 13971, 2023. [5]
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Stablediffusion text-to-image fine-tuning huggingface dif- fusers documentation.
htt.:///igg.....ooooooo///////// 13. 13.0/en/ w..//en/traiinng
ttxt2image] 2023.
Melanie Wakefield, Barbara Loken, and Robert C Hornik. Use mass media campaigns change
health bchaviour. The lancet, 37669749) 2010. 17
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Vision
and Puttern Recognition, pp. 0228-8238, 2024. 10 15
Wikidata contributors. Wikidata. https: https://ww..r. http://wwwwiiiiaaa..oroo.
ongoing.
3
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. Proceedings of the IEEE/CVF October 2023.
International Conference on Computer Vision (ICCV), pp.
15
16
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for rext-to-image generation, Advances in Neural Information Processing Systems, 36, 2024.
16
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. 2
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023. 23
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A RELATED WORK
A.I TEXT-TO-IMAGE GENERATION
A large body of work has been done with respect to generating images from textual descriptions. These text controlled image generation models have evolved greatly from the time of GANs (Goodfellow et al. 2020) to yield high-quality image generators based on diffusion models such as DALL- RRamesh et al. 2021), Stable Diffusion (Rombach et al. 2022) and ones that have extended these models, that are able to follow human text instructions
large extent. However, the metrics that these generators optimize are Inception Score (IS) (Salimans et al. (016) and Fréchet Inception Distance (FID) (Heusel et al. (017). It has been observed by multiple works for example (Kirstain et al. 2023 Wu et al. 2023) Au et al. 2024) etc. that these metrics do not necessarily correspond to human preferences.
A.2 ALIGNING IMAGE GENERATION WITH HUMAN PREFERENCES
To align models with human preferences, reinforcement learning with human feedback (RLHF) has been successfully used in the LLM literature (Ouyang al. 2022 OpenAl 2023 Touvron al.
2023) with algorithms such as PPO (Schulman et al. 2017). DPO (Rafailv et al. 2024 and several variants of these preference based reinforcement learning algorithms.
Similar approaches have also been used in text-to-imen generation models. In these approaches, the latent image generation part the diffusion model (either UNet Transformer) trained using reward model in the case of DDPO (Black et al. 2023) or using user preferences directly in the case of DPO (Wallace al. 2024). Both these approaches involve collecting human preference datasets.
Liang et al. (2024) introduce dataset RichHF-18K and multimodal transformer model (RAHF) that aims to predict detailed human feedback, such as identifying implausible image regions and misaligned text. They collected rich human feedback over 18,000 images from various users. Then they tune the RAHF model to predict implausibility scores, heatmaps of artifact locations, and text-image misalignment. The predicted feedback is utilized to enhance image generation quality through targeted finetuning.
Du et al. 2024) curate large-scale dataset consisting of over million generated advertising images, each annotated with human feedback regarding their suitability for advertising purposes. This dataset is created using diverse selection of products from JD.com, featuring generated advertising images alongside corresponding product images with transparent backgrounds and meticulously crafted prompts by professional designers. Additionally, the paper introduces multimodal model trained on this dataset to simulate human feedback of advertising suitability and automate image evaluation. This model utilizes an efficient recurrent generation process produce high-quality advertisement images.
Clark al. (004g) propose a method for gradient-based reward finetuning based on differentiation in the diffusion sampling process. This approach, Direct Reward Fine-Tuning (DRaFT) allows for optimising diffusion models by incorporating differentiable rewards based on human feedback.
They demonstrate the application of this method variety of reward functions, such as aesthetic score (Schuhmann et al. 2022), PickScore (Kirstain et al. 2023) and Human Preference Score (Wu)
al. 2023).
Liang et al. (2024): Du et al. )0024); Clark et al. (2024) use simulated human feedback to refine the
image generation process for specifie image quality dimensions of realism (Liang et al. 2024 Du
et al.
2024
Clark et al. 2024), aesthetics (Du et al. 2024 Clark et al. 2024). visual appeal (C) lark
al.
2024 or advertising suitability (Du et al. 2024).
A.3
DATASETS AND METRICS FOR HUMAN PREFERENCES OVER IMAGES
Several human preference datasets for text-to- text-to-image
image generation have been collected in literature.
These include Pick-a-Pic dataset (Kirstain al. 2023), dataset generated from the Stable Foundation
15

---

Published as a conference paper at ICLR 2025
ImageReward dataset el. 2024), etc. The human preferences in these datasets have been collected by explicitly asking humans to state their choices. These datasets are often accompanied with their own metries for human preference alignment such as PickScore [Kirstain et al., 2023), Human Preference Score (Wu et al. 2023), ImageReward score (Xu al. 2024), etc. While the authors of these research works have shown that these metries better aligned with human preferences when compared with conventional metrics such as CLIP score, BLIP score, Aesthetic Score, etc, however, they still need not align with viewer engagement of images. In this work, viewer engagement refers to the measurable interactions viewer exhibits toward an image on a digital platform, reflecting how strongly the content resonates and elicits actionable behavior. We leverage implicit human preference signals derived from engagement metrics, specifically account-normalized likes in the EngaginglmageNet dataset, as a proxy for viewer engagement.
Discord channel (Wu et al. 2023),
Thus, we explore holistic and user-centric metrics of engagement that measure dimensions beyond traditional image quality. Our contribution is unique in the sense that we prioritize viewer engagement as the core feedback, allowing for engagemeentoppimiized image generation that directly aligns with how users interact with visual content online. While traditional metrics like aesthetics, FID, ImageReward, image realism help in making the images look real, aesthetic and structurally correct, optimizing and measuring viewer engagement of images has potential applications in personalization, efficient A/B testing, recommender systems, and even public awareness and information campaigns as discussed in Appendix B
B How To CONTRIBUTE TOWARDS THE GOAL OF ENGAGEMENT OPTIMIZATION AND DIRECTIONS FOR FUTURE RESEARCH
Building on the foundational work of this paper, future research could explore the following areas:
Project Website: Our project website Snnhavior-inthe-willl gi thub. io/ enaggeenggagmment shall serve as platform for researchers interested engage- ment optimization for text-to-image models. We will release the following artifacts: (1) EngaginglmmaeeNet dataset, (2) Model training code, and (3) Engagement Arena for benchmarking text-to-image models. We envision it
open-source plat- form, inspired by initiatives like LMSys (https://Imsys.orr orrp// and LabintheWild chttps://lbbinthww/ .r.g///idddoo/..........
We welcome contributions in several forms:
Adding new engagement-labeled image datasets from diverse sources
Conducting and contributing results from user studies that further advance the under- standing in this field
Running engagement evaluations on newly developed text-to-image models and con- tributing results to EngagementArenaa
Assisting in platform maintenance and improvement
Donations for supporting with infrastructure and computational resources
Human Studies: Conducting
human studies can further advance the understanding of engagement dynamics across platforms and user behaviors. Some key research questions include how engagement patterns vary across platforms, the relationship between user characteristics (e.g., social media usage) and engagement, and the distinction between short-term campaign engagement (e.g., likes, click-throughs) and long-term user retention or loyalty. Investigating these factors can help refine engagement prediction models, improve EngageNet's 's adaptability, and provide deeper insights into optimizing content for diverse audiences.
Adapting EngageNet for diverse engagement signals: As demonstrated in the Flickr dataset experiment in Section 3.3 finetuning EngageNet on minimal data led to strong performance gains in predicting image engagement. This highlights EngageNet's ability to transfer its learned knowledge of image engagement to new social media platforms and engagement metrics. Similarly, researchers can further fine-tune EngageNet on diverse datasets to model engagement dynamics across platforms such as Instagram, YouTube, and Tok, capturing a broader spectrum of user behaviors. To support such efforts, we have provided our fine-tuning code the project website.
16

---

Published as a conference paper at ICLR 2025
Impact of External Factors Engagement: In the current work, potential unobserved confounders such as external events or viral trends, are not explicitly accounted for, despite their potential impact on engagement metrics. These factors introduce variations in user interactions that are not directly modeled. Future research can explore approaches to identify and mitigate the influence of such external factors, improving the robustness of engagement prediction models.
Explaining Content Factors Leading to Higher Engagement: Understanding why certain visuals generate higher engagement is crucial for both creators and researchers. Future work could involve enhancing EngageNet and Engagement Arena to capture subtle engagement dimensions such emotional resonance, visual complexity. and long-term memorability. Prescribing Content Strategies to Creators: Beyond explaining engagement factors, future iterations of EngageNet could actively prescribe actionable strategies to creators. For instance, could suggest specific attributes (c.g., color schemes, composition, or visual themes) that are likely to increase engagement for a particular audience or context, thus empowering creators to produce high-performing content efficiently.
Video Engagement Optimization: Extending the proposed framework to handle video con- tent could provide insights into optimizing video advertisements, storytelling, and social media clips for engagement. It involves considering dynamic elements like pacing, transi- tions, and viewer retention metrics, which play critical role in video content performance. Fashion Industry: The fashion industry could benefit from integrating engagement-awe content generation into virtual try-on systems. Models could generate visuals tailored to consumer preferences, encouraging them to explore and interact with virtual products. This could also enhance online shopping experiences by presenting customers with styles predicted to resonate with their personal tastes.
Enhanced Recommender Systems: Engagement-aware images have the potential to enhance recommendation models significantly. By integrating these visuals into recommender systems across domains like retail, food delivery, and streaming services, businesses can present more appealing and relevant suggestions to users, thereby boosting engagement and conversions.
B. BROADER APPLICATIONS
While this work primarily focuses on optimizing text-to-image models for engagement commercial and marketing settings, the proposed EngageNet framework has broader implications across various domains:
Personalization, Campaign Optimization and Recommender Systems: The EngageNet model can assist marketers and advertisers in selecting images that maximize viewer en- gagement, enabling automatic asset selection for brand campaigns. By integrating with multimodal recommendation systems, EngageNet can enhance the relevance and appeal of suggested images in applications such as personalized content feeds and digital advertising. This makes EngageNet valuable tool for valuable tool for campaign optimization and automated content curation.
Health and Public Awareness Campaigns: Generating compelling visuals for public awareness campaigns, such health education and environmental initiatives, can signif- icantly amplify their effectiveness. Research in public health and medicine highlights the positive outcomes of interventions like engaging advertisements audience behavior (Wakefield et al., 2010 Kite et al.) 2023). By leveraging EngageNet, organizations can optimize imagery enhance message retention and public engagement.
Infinite Personalization: Currently, ad delivery engines choose from fixed pool of creative assets for each visitor. With advances in generative AI, especially LLMS and diffusion models, it is now possible to create multiple ad variations dynamically. EngageNet can score these variants in real-time for each visitor, enabling infinite personalization, where every individual receives bespoke content, maximizing relevance and engagement.
Efficient A/B Testing: A/B testing is a cornerstone of optimizing marketing campaigns. However, due to the complexity and resource- resource-intensive nature of A/B tests, few marketers are actually able to use them. Among those who engage in A/B testing, the data- iata-intensive requirements necessary reach statistical significance often limit them to testing only few
17

---

Published as a conference paper at ICLR 2025
variants at time. Thus, marketers can employ EngageNet shortlist and prioritize images with high engagement potential, leading to more efficient and data-driven A/B experiments.
C STUDY TO EVALUATE ALIGNMENT OF ENGAGENET WITH HUMAN PREFERENCES
We conduct human study to validate how well the automated rankings produced by EngageNet correspond to ouman-perceived engagement. Through a human study done with marketers, Singh et al. (2024) found that the judgment of expert marketers has little correlation with advertisement success, thus validating the need to have an automated metric to measure engagement. Next, we conduct human study to help analyze how much the automated rankings produced by EngageNet align with humans. We describe the procedure and results for the human study below.
C.I EXPERIMENT DESIGN AND METHODOLOGY
From the Engagement Arena, select three text-to-image models with sufficiently different Elo ratings to ensure clear differentiation engagement rankings: (A) Stable Diffusion-3-Mediun (E1o=1041), (B) Pixart-Sigma (Elo=1002), and (C) DALL-E-2 (Elo=946), We use images generated by these models to construct pairwise comparisons for the study, described as follows.
The study consists of survey-based human evaluation, divided into parts:
Part A - Advertising and Social Media Exposure: Participants answer a short question-
naire about their advertisement consumption habits and social media exposure to establish familiarity with digital media.
Part B Pairwise Image Comparisons: Participants are shown 60 image pairs, each generated from the same text description by two different models. They select the image they find more visually appealing and engaging. Each participant evaluates image pairs from pool of 200 pairs, divided into 20 comparisons per model pair, ensuring a balanced distribution for comparing different models with each other.
randomly sampled
C.2 RESULTS
For each pairwise comparison, check whether the numan-selected image matches EngageNet's predicted ranking. If the rankings are identical, the comparison is deemed aligned.
The average alignment score is computed
Number of aligned comparisons
Average Alignment Score
Total comparisons across all users and models
Based on the current results, the average alignment score 64.41% %, indicating good correlation between human preferences and CeraageNet--redicce rankings. The study is continuously collecting samples and updating its results. take the study and to see more detailed analysis, please visit our project webpage: behavior---e in- the-he-wi siteewild ld. thub.
D ANALYSING VISUAL ASPECTS THAT DRIVE ENGAGEMENT
To understand the visual aspects that often lead to higher image engagement, we analysed pairs of images having same theme but different details, posted within 45 days interval by the same account. Then pairs with vastly different engagement levels between the images were sampled. We then extracted the differences between the image pairs for a few companies using using GPT-4- Vision (OpenAl 2023). Following are some main observations. For fashion brands like Bulgari, we observe that images featuring prominent branding and dynamic backgrounds with bright colors and gradients significantly enhance engagement, visible in Figure I (Image pair-3). For Gucci, engagement is driven by images that maintain clear focus on the product, emphasizing intricate detailing and textures. Additionally. images that incorporate luxurious backgrounds contribute to higher engagement levels. In the case of Airbnb, images that blend natural light with greenery are particularly effective in enhancing user engagement. Showcasing relatable homestay experiences
18

---

Published as a conference paper at ICLR 2025
aligns closely with Airbnb's branding. further driving engagement. Meanwhile, Lenovo benefits from highlighting unique technical features and specifications while utilizing vibrant colors and high-contrast backgrounds.
E ENGAGINGTMAGENET FILTERING STEPS
We sample the tweets posted in the year time period from January 2018 to January 2023. We focus our analysis on usemames that market products or services, and thus weed out usernames belonging to categories like news and sports. Next, we check if the number of tweets posted by username exceeds 1000, then we retain the username, else we discard remove it. This helps in removal of stray handles and ensures data quality. Further, if the number of tweets posted by a username exceeds 2000, we randomly sample 2000 tweets for this username to avoid oversampling tweets from the same username and thus compromising data variance. This step ensures that the dataset is fairly representative of different enterprise accounts. Moreover, we weed out all tweets containing media other than images and where tweet text is less than characters. Also, the hyperlinks present the tweets are masked with a <hyperlink>> placeholder. This results in 365,125 129 tweets posted in 5 years by 592 Twitter handles. Since it is hard to assign engagement (likes) credit to the multiple media present in single tweet, we assign equal engagement credit to all the media in tweet.
F NEGATIVE SAMPLES FOR ENGAGENET FINETUNING
Negative samples were introduced to enhance EngageNet's 's ability to discriminate between relevant and irrelevant image-text pairs, improving its overall robustness. The method for constructing such misaligned pairs is described below. We augment EngaginglmageNNt with synthetic data samples. For this, we randomly sample 25% tweets from the high and medium likes buckets of each company and pair the tweet with unrelated image from different tweet. The corresponding normalized likes is set to a low value, randomly sampled from the range 5-15. The resulting samples are called negative samples. This process ensured that the negative samples presented a challenging scenario for EngageNet, where the image and textual context were semantically misaligned. Thus, adding negative samples in training EngageNet is conceptually analogous adding negative samples in contrastive learning.
This helps the EngageNet model with the improved sensitivity contextual alignment of images. Training on negative samples compelled EngageNet to pay closer attention to the semantic relationship between an image and contextual tweet metadata. EngageNet learns to penalise an image if is irrelevant poorly aligned to the tweet context. This ability is crucial to utilize EngageNet as an oracle for evaluation text-to-image models in the Engagement Arena, such that aesthetically pleasing but irrelevant generated images do not score high on engagement.
G ENGAGINGIMAGENNR ADDITIONAL DETAILS
Table 5: Distribution of ground truth EngagingImageNet images
Enpagement Objects
High
3.405
1.54
30
3.291
Figure 4; Acsthetic Score distribution across High and Low engagement images in EngaginglmageNet dataset
19

---

Published as a conference paper at ICLR 2025
(C)
(d)
(e)
i
Figure 5: Plots showing variation of number tweets and likes with time for a few companics in the En- GaginglmageNet dataset
H PROMPTS FOR INSTRUCTION FINETUNING
Listing 1: Visual instruction finetuning Patter: EngageNet predicts user engagement of images given contextual information about the social media post.
Input: <image>
This an image that marketer from company 'gucci" wants to post on social media for marketing purposes. The following
information about this image also given:
(1) image resolution i.e. (width, beight): ): 1680, 6801.
(2) image
('colors ": ("Orange": ("coveraee:: 0.6), "White': ("cowerage": : 0.18), "Pink": I "'overngee" ": 0.12]. "Bron": I'coverage", " 0.1] "tones" "; ("warm": 0.72, "neutal" ": 0.28, "cool": 0)]],
tones: 1" ("color
(3) marketer's s intended image description A girl with nose ring and gold earrinss
(4) marketer's 's intended image tags: nese ring, gold earrings girl, makeup. lips face. beauty. cartings. nose. lips gold, woman, makeup, accessorics.
(5) posting: : 22-F
Now, carefully observe the image. You have to predict the "number of likes that this image will get, on scale of 0 to 100. will internct their appreciation for the image. Thus, image with higher visual appeal, alignment with the company's brand identity and relevance to the audience. likely receive more likes Morcover, good image should stongly conesponn with the marketer's intended image description and tags attract the target audience.
measures number
media post by elicking
button express
Your predicted "number of likes" will help the marketer to decide whether post this image or not on the social media
platform.
Answer properly in JSON format. Do not include any other information in answer.
20

---

Published as a conference paper at ICLR 2025
Autti LOD
Faation
Fvents
Induvetios
Consmerr Good
Netal
Bevarage
Devillig
Sporta
Edspation
blucalion
Serviter
Cisatrrtt Novvs
Elvcstion
Boecaniing Service
Faranco
cum Tectenaaty
eman Doseo
Cyberrannt
Cool Con
Figure 6: To analyze the distribution of tweet topics, we used BERTopic extract topics from the tweets. These topics were subsequently clustered and named using GPT-44-mini. This figure demonstrates that Engaginglma- geNet data encompasses a wide range of diverse tweet topics.
Oulput: llikees ": 19]
Listing 2: Engagement Finetuning Verbalization Pattern (1): Explicitly asking model to pay attention to engagement tokens
Input: You are smart model. giving you data regarding image (1) captions (2) keywords (3) image resolution i. how many times the image was downloaded (6) number of the image was forwarded somcone else number impressions i.c. how many times the image was seen by someone. Note that (5). (6) and (7) are Key Performance Indicators (KPl) of the image, thus they are important signals
c. (width, height) (4) release date (5) number of downloads i.c. i.c.
forwards i.e. how many
perceived quality and popularity
You have to predict following attributes of the image: (I) colour and tones from the lists given below: Allowed colours: I'
Red', "Dark_Red "Gnen', 'Bright Green", " 'Dark Green', Light_Grren', 'Mud Green', 'Blue', Dark_Blus' "Light_Bluc RRoyl_BBuee 'Bluck", 'White". 'Off_ White", "Gray. "Dar_GGray' 'Silver "Crean". 'Magenta'. 'Cyan', "Yellow". Mustard'. "Khaki'. 'Brown'. 'Dark Brown', 'Violer 'Pink', "Dark_ 'Maroon'. 'Tan', 'Purple' ' "Lavender', Turqueise "Plum', 'Gold', 'Emerald', 'Orange". : 'Beige", : 'Lilac :. "Olive" Allowed lones: ('wam', 'neutral 'cool " (2) main objects present in the image and the diagonal coordinates of their hounding boxes: [xl, yl. x2, y2]
Now, predict the attributes
up portrait mixed-race female worker posing confidently while standing with arms crossed plant workshop', keywords: "female, worker, young, woman, mixed-race, african. african -americam, modern, contemponary work. accupation, industry. . industrial plant, factory. workshop. work shop, strong, woug,, gritty masculine. short. hair, latin -american, plump, adult, mechanic, repair. repuirman, handywoman, foreman, copy space, porimit looking at camera, standing, posing, smiling, recruitment, employment, jab, opportunity. engineer. production, manufacturing, assembly, assembling. line image resolution: '(5760, 3840) teleuse date: 10099-1--22 number of downlouds: "24", number of forwards: "106", number of impressions: "5941"] Answer properly in ISON format. Do not include any other information in your answer.
following image: [captions:
Output: ("color and tomes". :: l"colors ": ("Gray": P'coverage": : 0.4), "Dark_Gray": I"coverage": 0.22), "Black": ("coverage"; 0.14), "OfT_ White": ["owverge". " 0.13). "Silver "; I"uwerrae"": 0.113], "tmmes": ": (" ('warm:: 0, "neutral * 1.0, "canl": 01). "objects" ": ["jeans": "; [2076.67, 2542.5, 3023.88, 3827.011. "woman"; [1892.94, [18022";[1892::
11,18, 4260.09, 3824.34), "safety
vest": [2160.75, 1410.95, 3668.16, 3826.63), "hii " (2163.59, 1079.3, 4254.59, 3826.1])
Listing 3: Engagement Finetuning Verbalization Pattern (2): Noisy engagement in input and asking the model to correct the noise in addition producing content
Input:
"You are a smart model. am giving giving you some data regarding an image relcased by a content creator (1) captions (2) keywords (3) image resolation i.e. (width, height) (4) release date (5) approximate number of downloads
that the creator wants to achieve (6) approximate number of forwards that the creator wants to achieve (7) appeoximate
number impressions/ /views
the creator wants achieve
You have to predict following attributes of the image: (1) volour and tones from the lists given below: - Allowed colours: I"
Red', Dark Red', 'Green', "rright Green", 'Dark Green', "Light Green', 'Mud Green', 'Blue'. Blue', "Light Blue",
21

---

Published as a conference paper at ICLR 2025
"Royal_ Blue', 'Black', "White", "Off_ White", "Gray". Gray', "Silver", '. 'Cream". "Magenta", 'Cyan', "Yellow"
Mustard', "Khaki", 'Brown'. "Dark_ Brown', VViole ", "Pink", "Dark. Pink', "Mamen', "Tan", 'Purpee' ' "Lavenderr, Tarquoise" "Plum'. 'Gold", "Emerald', , "Orange", "Beige", 'Lilac" ; "Olive'] Allowed tones: [" ['war''. neutral "oool "I (2) main objects present in the image and the diagonal coordinates of their bounding boxes: [xl, yt. x2. y2] (3)
exact number downloads that the image will get (4) exact number of forwands that the image will get (5) exact
number impressions/viess /views that the image will get
Now, predict the attributes for the following image: [ captions: ""Hispanic adult man holding 100 brazilian real hanknotes smiling huppy pointing with hand and finger the "", keywards: "" "ointing side, lace, happy. hopeful, smile, finger. optimistic hand, point, showing. looking, smiling, one, gesture, confident,
cheerful, look, mouth, joy, friendly. expression, emotion. presentation idea, blue, background, hispanic, latin man, male, guy, beard, bald. shaved, adult, young, money, currency. business. brazilian. cash. brazil. real. investment, banknote, 100" image resolution "(9216, 6144)", releuse date: ""2021-12-27"" appreximate number of downloads that the creater wants to achieve: "4" approximate number of forwards that the creator wants to achieve: "I7"", approximate number of impressions /views that the creator wants to achieve: ""919" ] Answer properly in ISON format. Do not include any other information in your answer."
Outpun: !ooolor and toncs" " I"collos" ": ("Cyan": ; ("coverage": 0.69), "Light Blue": I"coverage": 0.16), "Turquoise" : ("oovorage ": 0,15)). "tones". ": ["warm": ": 0. "neutral" ": "cool": 1.0].. "objects" ": I"man": [1068.6. 18.57, 8143.44, 6121.09). bankmote bill ": [3443.5, 2146.79, 5294.15, 3455.05)), "exact downlouds": 4, "exact forwards"; " 15, "exact impressions" 885]
Listing 4: Engagement Finetuning Verbalization Pattern (3): Noisy engagement in input and asking the model correct engagement in output
Input:
"You are a smart model. am giving giving you some data regarding an image released by a content creator - (1)
captions keywerds (3) image resolution
(width, height) (4) release date appreximate number of dowaloods
that the creator wants to achieve (6) approximate number of forwards that the creator wants to achieve (7) approximate
namber of impressions /views that the creator wants to achiove
You have predict following altributes of the image: (1) exact number of downkouds that the image will get (2) exact number of forwards that the image will get (3) exact number of impressions /ivesssss///iwwss views that the image will get.
Now, predict the attributes for the following image: captions: "MMvie slapstick vector illustration Behind the scenes inscription on flapper keywords: "'bchind the seenes, slapstick. flapper. movic, cincma, seene. logo. frame, film. duration. behind, act. black, cameramam, clip, date, director. entertainment. lap lootage, gray. hollywood. icen. illustration inseription operator, screen, shosting, sign. signal. symbol, television theater, time, timecode, Iv, vector. video, view, white" "" image resolution: : "'409,, 4096)"". releme date: ""2017-06-11" approximate number of downloads that the creator wants achieve: "5"", approximate nunber of forwards that the creator wants achieve: ""31", approximate number of impressions /views that the creator wants to achieve: "9914" ] Answer properyy in ISON format. not include any other information your answer.
Ouiput: l'exact downlouls": 5, "exact forwands": 32, "exact impressioos" 864)
Listing 5: Engagement Finetuning Verbalization Pattern (4): No engagement in input and asking the model to simulate exact engagement in output
Input: "You are a smart model. am giving giving you some data regarding an image released by content creator (l)
captions (2) keywords (3) image resolution i.c. (width, height) (4) release date
You have predict following attributes of the image: (I) exact number of downkods that the image will get
number
of forwards that the image will get (3) exact number of impressions /views that the image will get
Now, predict the attributes for following image: (captions: : "" ""Company employecs working in software development and designer oflice "", keywords: business. oflice. meeting. collaegue, successful. workplace. analysis. architect.
coworker, discussion, entrepreneur, , marketing, prufessional, , company, employee, occupation, software, work, worker, team, people. brainstorming, cooperation, corporate, prjjet,, strategy teamwork, together, computer, colleagues, young, diverse. collaboration design, developer. group, ideas, management, smiling. multicthnie. place, plan, research. startup technology, women, programmer, architects "t, image resolution "((008,, 2690) "", release date: "2200009929"" 1. Answer properly in JSON format, Do not include any other information in your answer."
Output:
|'exact downlouls:: 1, "exact forwands": 1. "exact impressions" ": 186)
I TRAIN-TIME OPTIMIZATION ON OTHER TEXT-TO-IMAGE MODELS
We extended our train-time optimization experiments beyond Stable Diffusion 1.4 (Rombach et al.. 2022) to include more advanced text-to-image models, Pixart-Sigma (Chen et al.) 2024a) and SDXL (512 resolution) (Podell et al. 2023). Specifically, we performed Preferred Finetuning (PFT) on high- engagement images, described in Section 4.2 The results in Table 6 show strong improvements in image engagement reward as predicted by EngageNet. This demonstrates the effectiveness of the proposed method across multiple text-to-image models.
22

---

Published as a conference paper at ICLR 2025
Table 6: Performance gains the EngagingImageNNe test dataset achieved by applying Preferred Finetuning (PFT) on different text- text-to-imant to-aaage models, measured by EngageNet's engagement reward.
Images
Training Config.
High
Medium
Low
SD 1.4
Preferred Finetuning (PFT)
56.1489
38.7680 51.6949
HOIG SD 1.4
High engagement Images
62.0390
56. 1082
43.1206
Pixart- Sigma
Preferred Finetuning (PFT) on
65.8089
59.9771
46,2870
HOIG Pixart-Sigma
High engagement Images
68.6384
61 11,9990
51.6297
SDXL-512
Preferred Finetuning (PFT) on High engagement Images
48.8769
44,4127
35.1069
EOIG SDXL-512
73.9467
64,6240
48.3155
J REPURPOSING ENGAGENET FOR DESIGN SPECIFICATION GENERATION (DSG)
J.1 TRAINING FOR DESIGN SPECIFICATION PREDICTION TASK
Prior works (Bhattacharya et al.
2023 have demonstrated the capability of language-only pre-trained
models like GPT-3 and Vicuna to infer information about visual content without explicit visual reasoning training. Recent models such as BLIP (Li ct al. 2023). Llava (Liu et al.
(023), MiniGPT-4
(Zhu et al. ) and GPT-4 (OpenAl 2023) have shown language models' ability 'see' by
incorporating visual branches (often combination of ViT (Dosovitskiy et al. 2020 and Qformer (Li
al. 2023) and training them with mmage-language instructions to answer image-related questions. However, our findings (Table reveal that neither pretraining nor further instruction tuning gives
language model the ability to simulate the downstream engagement an image-based communication or reason about how more engaging image should look like. Further, we also find that in-context learning, while successful in many other domains, does not perform well in engagement-relate tasks. Therefore, to teach a language model about image content and downstream performance, we further train the Illama I.I.M.
To teach Llama about an image and its downstream engagement, we perform engagement fine-tuning (Khandelwal et al. (023). We design four types of teaegeement-innttniie instructions (Listings 55 The idea is verbalize an image using image perception models like color extractor, tones extractor, object, and coordinate detector and convert it to natural language. Then, the image caption, keywords, the required engagement level, date, and marketer information fed input to the LLM and asked to
output the image verbalization. This way, the LLM learns to map the image prompt and engagement level image verbalization.
We train the LLM on the train set of EngaginglmageNet data. In Listings we provide the image caption, keywords, date, and required engagement level as inputs the model. Our aim is to train the model to predict a design specification comprising colors and tones with their spatial coverage as well as objects with their bounding boxes, that should be reflected in the image. Moreover, we image generation
observe improved learning in the language model for engagement--onditionn when introducing a 20% noise in the engagement. We then task the model to rectify this noise in the output, simultaneously generating the verbalization of the engagemeent-conditionee image in Listings
J.2 RESULTS FOR ENGAGGEENTTCCNDIITONN DESIGN SPECIFICATION PREDICTION TASK
To generate www.ggement-coniiiione image verbalization, we compare several models: in-context trained GPT-3.5 and GPT-4, engagement -inettttinee Llama (EngageNet), and Llama fine-tuned on image verbalization but without user engagement information. By comparing against fine-tuned Llama trained on the same instruction as EngageNet, except with the inclusion of engagement tokens, we aim to isolate the impact of engagement tokens on improving generated engagement-condition image verbalizations, independent of the instruction tuning process. We assess all models across multiple metrics that evaluate the extent to which the generated verbalizations align with ground truth in terms of colors, tones, objects, and their positions. Intersection over Union (IoU) metries gauge the overlap between ground truth and generated constructs (colors and objects), while similarity metrics measure cosine similarity between ground truth and generated constructs (colors, objects).
23

---

Published as a conference paper at ICLR 2025
Table 7: Performance of all models on the mggagement-optimizee design specification generation (DSG) task across different engagement-levs images EngaginglmageNet data. noteworthy that (i) EngageNet outperforms larger sized GPT-3.5, 4 and also the sized Llama model fine- line-tuned on the same data (without including engagement tokens). (ii) In-context learning does not work well in the engagement-condiiigg design specification generation domain.
Enanghnglinngeeet
Mede
Engugement Optimized
Engagement
Colours
Toacs
Objects
10U
Cosine Similarity
distancs
RGB
Coverage
Corerage
Cosine Similarity
Normalised RMSE
Reladive Preition Error
Firetured
Liams
No
High
Lawe
0,3717
0.3562
018725
018612
0.3223
0.2555
0.1694
0.1811
0.2339
0.1957
0.2547
0.2245
0.9047
0.8071
0.2612
0.3421
0.3078
0.2054
Fisetured Namo (PagageNar
Engagementt Fiezniing
High
0.4065
0,8898
0,2795
0.1507
0.1718
0.2732
08:22
0.2009
0.3054
Tane
0.4511
0.3064
0.144
0.1848
GPT
lorning
Low
0.3175
0.2254
0.1773
0,2118
0.3347
0.1318
0.368
0/3173
Sober S
In comeent
leninn
Righ
Law
0.2191
0.3137
0.704
0.7705
0,3543
0.2176
0.1975
0.2449
0.3186
0,334
0.1284
0.1011
0.456
0.656
0.3150
0.360
IL3022
0.3615
High
leeiig
5-shoe GPP-4
In-comecu loriing
Hight
Law
0.3437
0.3648
0 77034
0.7905
0,2002
0.3578
0.1864
0.2144
0,2937
0.2944
0. 0.1454
0.1038
0.6136
0.6006
0.1M0
0.311
0.3782
0.3887
Coverage errors determine the how closely the proportion of ground truth and predicted constructs (colors, tones) in the image match. Additionally, calculate differences in predicted and ground truth areas and locations for objects, accounting for semantically similar objects (such as sofa and couch). Further details on these metries and their formulas can be found in Appendix
Table 7 displays the outcomes. The results indicate that engagement fine-tuning enables EngageNet to achieve superior performance across all metrics, surpassing both equivalently sized fine-tuned Llama and 10x larger instruction-tune GPT-3.5 and GPT-4. Furthermore, in-context learning demonstrates subpar performance, with both the three and five-shot models displaying similar results.
J.3 EVALUATION METRICS FOR DESIGN SPECIFICATION PREDICTION
. Colours IOU: The intersection over union between set CG of colours in the ground truth image verbalization and set CP colours in the predicted image verbalization is computed as:
IOU(CE, .CP)= |C CC | UCP
Colours similarity: For the ground truth colour set CG o.. ..., of and predicted colour set CP {of , ce :..,, cf)) we correspondingly obtain the sets of word vectors
W = {wf w ( and W = {wff, wh ...., w.', using Spacy ity threshold t, the mean cosine similarity is computed as follows:
For some similar-
D I cos(w,, ( wf
(5)
E. SEiE
ン Inw .wf,T)
where lwwf wf. is an indicator function defined as:
Il , wf {
if cos(w w/) > T
(6)
otherwise
We take T = 0.7 in our experiments.
Colours RGB distance: Given the ground truth colour set CG = o & o and predicted colour set CP = (of' ...,,..,,. of'), we map each colour to its RGB value to obtain the sets W {wf u .... wf9) and W (wf', wT ..... w,,' where each element in the sets is a 3 x dimensional vector of RGB values. For some distance threshold т, the mean euclidean distance is calculated follows:
109
|C distance(w (.P))([wf Muwfwwff wuf. wf.r)
Ei=1
(7)
Σ
Σ |CP ==1
I[w[ wf.
1
https: https://sspccy.io) 10/
24

---

Published as a conference paper at ICLR 2025
where I( w.. T) is indicator function defined as:
{
if ( wf) <T
I(w; w (..TT)
(8)
otherwise
We take r = 0.5 in our experiments. Colours coverage RMSE: Consider the intersection = C CP of ground truth and predicted colour sets. The root mean squared error between the area covered by colours present in both ground truth and predicted image is calculated as follows:
M
RMSE
coverage(c coverage(c?))
Tones coverage RMSE: Consider the intersection T T of ground truth and predicted image tones. The root mean squared error between the proportion of tones in ground truth and predicted image is calculated as follows:
1 Σ
RMSE-
coverage(t()) coverage(t?
(10)
I
Objects IOU: The intersection over union between set Oc of objects in the ground truth image verbalization and set OP of objects in the predicted image verbalization is computed as:
100 no 10U(0 10 UP
(1I)
Objects similarity: For the ground truth set of objects O = {of .f....,, and set of predicted objects O" = {of of ,m., of'), we correspondingly obtain the sets of word embeddings W (wf w. ..... w and W {wf, w² ...., w²), ). using Spacy. For some similarity threshold t, the mean cosine similarity is computed as follows:
E |0" cos(we, ( wF.
(12)
E D I(w wf. T)
where T( wf wf(T) :)) is an indicator function defined as:
{}
if cos uz)) >T otherwise
(13)
We take r = 0.7 in our experiments.
Normalised objects area RMSE: As described above, consider the sets of word vectors objects present in the ground truth image O {of o2 ...,, ( and predicted image OP fof .f...., Given the ground truth image area A width height and a similarity threshold т, we first compute the mean squared error between the areas of bounding boxes of similar objects in the ground truth and predicted image, weighted by the proportion of each object in the ground truth image and its cosine similarity with the object in the predicted image. Further, we take the square root the error thus obtained and normalise it by A to achieve the desired metrie, follows:
100 5001 -
0 (larea(os) - area(oP))) areafo AG
cos w my Juw w. ;
MSE-
100 E007 1uw8 wf.n
(14)
VMSE
Normalised RMSF
(15)
AG
where I(w[f, wf.+ :) is an indicator function as described above. We take 7 0.7 in our experiments.
25

---

Published as a conference paper at ICLR 2025
Normalised relative position error: Following similar approach as explained above, compute the mean euclidean distance between the centroids of bounding boxes of similar objects weighted by the cosine similarity of objects present in the ground truth and predicted images and normalise it by the length of diagonal in the ground truth image DC:
Σ
109 1= Σ j=1 (dissaancccenttridd(f centroid(of). cos(wp woff J Mu wg
10
RPE
10c 1= j== [ wof :7)
10
(16)
RPE
Normalised RPE- = DG
(17)
where l[w5, wf. r) is the aforementioned indicator function. As before, we take т in our experiments.
0.7
K PERFORMANCE ALIGNMENT OF STABLE DIFFUSION USING DESIGN SPECIFICATION GENERATION (DSG) REWARD
K.1 DDPO ADDITIONAL DETAILS
The denoising process in diffusion models is multi-step recursive process with a pre-specified finite number of steps. In DDPO Black et al. (2023), this denoising process is viewed as finite horizon Markov decision process (MDP), where the state comprises of the current context, number of steps left in the process and the current denoised image. The action to be taken is to predict the next image using this state.
The image forming the initial state is sampled from standard normal distribution. Mathematically, finite horizon MDP is defined as a tuple {T, S, A. P, R), where these components are defined as:
1. T' is the horizon or the number of steps in the MDP
2. S is the state space. Here it comprises of three components, the context c, the current number of steps left the denoising process, 1, and the current denoised image representation (a given vector encoding of the image), .y. The initial or starting state has the context co given as input, the number of steps left at the beginning, to = and the initial image representation is sampled from normal distribution of appropriate dimension, zo (O, I).
3. A is the action space, and here is the space comprising of all image representations x. If x is d- cd-dimensional vector, then AA Rd
4. P S x A + A(S) the transition function. Here, we specify P separately for each of the three components of the state P. 5(c,), Pt 5(1- 1), and Pa 5(ax), where the current state is at 1, current action 0 = It-1, and d(-) is the Dirac delta distribution.
5. R: S A IR is the reward function that takes state and action input and returns scalar reward. We generate this scalar reward signal using EngageNet.
As the agent acts in the MDP, it produces trajectories, which are sequences states and actions: (so, ag, $1, a1....... sr,,aaa ar). The reinforcement learning (RL) objective is for the agent to maximize the expected cumulative reward over trajectories sampled from its policy. Assuming a fixed sampler, the diffusion model generates sample distribution po(2o c). The denoising diffusion RL. objective is to maximize a reward signal r which is defined based on the generated samples and their corresponding contexts:
JDDRL (0) -Eeesspte [r( c)]
for some context distribution p(c).
K.2 DESIGN SPECIFICATION GENERATION (DSG) REWARD
The steps of constructing the reward function based on design specification generation are given below [4b]
We featurize the image generated by stable diffusion to obtain features (including colors, tones, objects, and their positions) that EngageNet is meant predict part of design
26

---

Published as a conference paper at ICLR 2025
specification conditioned contextual information such marketer, expected likes, tweet content, image caption, etc.
Based on the above conditioning factors, use EngageNet to predict the logits of the verbalized features of the image generated by stable diffusion as described in the previous
step.
We now have one logit per text token as EngageNet's output. To convert this to a scalar score, we compute the probabilities of each token and then add them.
(a)
(b)
Figure 7: Reward curves for the performance alignment of stable diffusion on EngaginglmageNet (train (a) and validation (b) sets)
Table Results for the performance of various models on the EngaginglmageNet for the engagement-optimie image generation task. Results are computed on the EngageNet Design Specification Generation (DSG) reward
K as well as other metrics reported in the literature.
Other Metries
Images
Engagement
Reward
FID
Austhetic Score
CLIP Score
Base Stable Diffusion
High Low
242.545
34.958
5.221
33.346
238.471
42.999
4.925
31.708
Fine tuned on High Engagement Images
High Low
239.023
26.023
4,850
32.210
Stable Diffusiin
223.619
37.497
4,433
30,979
EngageNer aligned
High Low
254,918
36.546
5.341
33.379
Stable Diffusion (EOIG-SD)
247.597
49.492
5.087
31,719
L BROADER IMPACTS AND LIMITATIONS
Our work on assessing and improving the engagement of text-to-image generation models introduces several societal considerations that require careful examination. We aim to provide a comprehensive analysis of the potential impacts, highlighting both contributions to the field and the precautions necessary for responsible development and deployment of engagement-optimizn
technologics.
1.
The ability of models to enhance user engagement raises important societal concerns around responsible deployment and potential misuse. Quantifying these risks is essential for develop- ing appropriate safeguards. However, studying user engagement, particularly uncontrolled environments, poses ethical challenges. For instance, investigating engagement through manipulated or highly optimized content could influence user behavior in unintended or harmful ways. To mitigate this risk, we have limited our research to controlled environments and observational analyses, ensuring that experimental insights are gathered without causing real-world harm.
2. To foster responsible use of our research and datasets, we will release an Acceptable Use Policy explicitly prohibiting the misuse of our dataset for generating content aimed at harmful or deceptive purposes. This includes banning its use in abusive contexts (e.g., creating deceptive ads manipulative imagery) and sensitive applications such as political propaganda. We will actively monitor compliance with this policy and encourage others the research community to adopt similar ethical guidelines when using engagement- optimizing models. Importantly, our dataset is PlI-free, ensuring that no personal information of individuals is included. Our dataset compilation adheres Twitter's API terms of service. We used the Twitter API from 2015-2023 for data collection, and our dataset release will comply with all restrictions outlined in Twitter's Developer Agreement and Policy, available at hhttts:://dvelle
x. com/ deve wowreeeloleeetermseeeeee tttperrmsssaareeeeeeea--aa-llllo
3. We plan to release the dataset and evaluation frameworks in stages, starting with the release of our benchmark and engagement arena. This staged release will help familiarize the research community with the methodologies we use to assess engagement in generated content. By gradually releasing the dataset (in batches of 20%), we will closely monitor
27

---

Published as a conference paper at ICLR 2025
how models perform in enhancing engagement. Initially, the dataset will only be available in a controlled environment, enabling us to manage usage and address emerging concerns. Throughout this process, we will actively engage with the research community, encouraging responsible use and urging fellow researchers to contribute additional persuasion-reie relatted data using our infrastructure. We also encourage the research community to contribute additional data to expand our evaluation framework. This approach balances the need for research progress with ethical responsibility and community involvement.
4. We recognize the dual-use potential of models designed to optimize engagement. While this technology can be beneficial fields like education or user-centered design, also poses risks of misuse in deceptive contexts. Drawing parallels to ethical discussions on persuasive technologies, we believe that transparency and safeguards dataset design can mitigate the potential for harm. The insights gained from understanding user engagement can aid in the responsible development of future AI systems.
5. PII Removal and Data Collection: To protect user privacy, have implemented measures to remove all personally identifiable information (PII). Our dataset is compiled without collecting sensitive personal data, focusing solely on public, non-indivinn non-individualized information. All references to specific users or personal identifiers have been removed. Additionally, we collect only aggregate metrics (e.g., overall user interaction data) to measure engagement trends without compromising individual privacy.
6. In this work, we specifically focus on the engagement optimization capabilities of text- to-image generation models. We introduce benchmarks and evaluation methodologies for measuring user engagement with Al-generated images and develop techniques to enhance this engagement. Our findings suggest that engagement with generated content can be improved not just by increasing model size but also through targeted training strategies. Furthermore, engagement patterns observed in one domain (e.g.. Twitter) often transfer to other domains (e.g., Pinterest), which broadens the applicability of our findings.
L.1 LIMITATIONS
In this paper, we examine single aspect of engagement. In real-world applications, user engagement often occurs in sequential or multi-stage interactions, which we plan to address in future research. Additionally, this work is focused on Baggishh-anguuage data; we aim to extend our findings other languages in subsequent studies. Furthermore, the impact of audience dependence on engagement has not been studied extensively in this paper, partly due the absence of publicly available datasets. We plan to work on collecting such datasets to explore this effect in future work. In the current work, we do not explicitly account for potential unobserved confounders, such as external events and trending topics, which may influence engagement. These limitations underscore areas for further research and caution against over generlenena our findings to more complex real-world scenarios.
M ADDITIONAL FIGURES
Durne:: Twitter
Dewrrl Tuttier Auderce
Spedke Telesem m
Figure 8: Any message is created to serve an end goal. For marketers, the end goal is to bring in the desired receiver effect (behavior) (like clicks, purchases, likes, and customer retention). The figure presents the key elements in the communication pipeline- the marketer, message, channel, receivers, and finally, the receiver effect. Traditionally, image generation is optimized on metrics such as aesthetics and FID. For effective communication, the image generation process needs to be optimized on the receiver effect (other than the traditional metrics).
28

---

Published as a conference paper at ICLR 2025
Seu Libea
Ps
Adobe Photoshop
SAPMEY VO
UFFCS
Saming
Figure 9: Sample media and tweets from enterprise accounts in the ImgaginglmageNet dataset. It can be noted, for example. in the Adobe Photoshop tweets, that the media does not differ significantly in aesthetics objects themselves (all of them cats). Despite that, there is much difference in the image likes, indicating that viewer engagement is distinct from other optimization objectives as aestheties or prompt adherence.
Prompt EngageNint
simages
This s image fhat mericte Far compory "Ha cywelf wants postion sceaa med a markating ng parpoess The fol oollowing into maston about
Engagehatt
Engagement
image in alls diven:
A
11) Image resoluion
image
interred biess
Mertiter Hoo ewilt, Btric 2012--144 talk and painding on will. tome Sacor
maroneers anneden mange thge
cote posting
image
Figure 10: Visual Instruction Finetuning of EngageNet on EngaginglmageNet dataset. The EngageNet model trained to predict the engagement score of an image on 0-100 scale, conditioned on marketer provided metadata comprising the company, image resolution, image colours and tones with their spatial coverage, marketer's intended image description and tags, and the date of releasing the image on social media.
"A living room having rug in front"
"A living room with large windows having couch, coffee table and rug
table
performing caption and keywords semantically user's input from vector index
Diffusion Prompts
Likes: 53
Likes: 72
Figure 11: Retrieval framework for conditioning text-to-image models on higher engagement prompts as deseribed in Section 4.1 The retrieved prompts may incorporate image characteristics that have been empirically shown improve image engagement.
29

---

Published as a conference paper at ICLR 2025
Finetuning SD on preferred distribution af data
Finetuned SD
Cleaned data of high KPI images
Dataset of images
Prompt Filtering
Illustration depicting supervised finetuning of stable difffusion model high-liked images from EngaginglmageNet dataset as described Section 4.2 This method of finetuning U-Net module preferred data distribution results in generating more engaging images.
Figure 12:
Base SD
EOIG-SD
(d)
(e)
laptop with qughiquality screen disppaying whale
styliah close-up shot person's teet wearing white aneakers, paired with dark trousers; urban
Large industrial-aized eir
chocolate bar A joyful golden retriever aplash, illustration, liquid, dessert, delicious, swirl, creamy,
conditioning units lined HVAC system, urban environment
site in front of decorated Christmas tree; snow, gifts, festive theme
emarging from the acean: ultra HD. technology
fashion, street style
Figure 13: Comparison of generated images EOIG-SD vy Base stable diffusion. Engagement optimization helps the model to learn generate persuasion skills. EOIG-SD generates better product photography (a,c,d), model photography (b), generates images with social appeal and social identity (a,c), and learns temporal patterns (c) (prominently Christmas themed image of dog)
30

---

Published as a conference paper at ICLR 2025
Prompt for EngageNet
"cmagro
Contextual Information
image marteter from
'Huricywa "wants on socia modia for mericcing purpsons, The ioiloar intermation atoun this image in aio glven
Marketer Honeywel
Date: 2022-10-05
Caption: A living noem with a gouch, cofcce table and a paitting on the wall Keywords: lving room, couch, cofice table, painding wall, home
Image
onwnnted
[8] date pooting
sow ceetaly ooehllyytteru oboamse the image You to prwdicr the
mumber of hid...
Curate Prompt for EngageNet using Contextual Information for image Engagement Simulation (ES)
Caption: liing 1oom coffee
painting
Keywords, do: lving room, couch. collon Intiw, painting wal,, home
EngageNet
docor
Captons and Keywords fed EOIG-SD
EDIG-SD
EngageNet is the
Expected arand-nommaized
likes image
Image: being progressively aptimized for achieving high engagement
Policy Gradient
(a) Architecture of the proposed pipeline for training stable diffission for the ubjective of engagement optimised imuge generation (EOIG) using Engagement Simulation (ES) reward function deseribed Section EngageNet predicts the engagement level images generated by stable diffusion. The scalar rewards are used guide stable diffusion to produce progressively higher engagement images. The resulting diffusion model is called BOIG-SD (RLHF-ES).
Contectul Information
Prompt for EngageNet ingaiin teeneeting veppore
"weed bodd te po
un
EngageNiet log convened 10 prosatillee
202--045
Ciadian soutt
Computation Engageliet Graph
Cin sbore reseed the
Engageteer gereeaans Sooph condile Boniil and beteeiar
vertallaaten
60-6.0
neadwe sa as
compure
prgaayaeee co mekkel ilomation Desig Specifiatioo Generstion (DSG)
Beal Path
worded sauch
Markster'a Iepulied EDIG-50
peroptaal modil
Image Design Specifiation
Fert
howr
EDIG 50
TOUES
D'Diggg Spooll barion tollot
Ko XPA. out its. Zeo
06A-12
-tacc Deacco
Imagge boing prog rigrrsssilyy uptriize for actsceiing
Vertalizaion CD generateed Imuge computed Lang Isage
Palicy Grackent
(b) Architecture proposed pipeline for training stable diffusion for the objective of engagement--ptinnsee image generation (EOIG) using Design Specification Geucration (DSG) reward function as described in Section 4.3 EngageNet trained in the manner described in Appendix possesses the cupabiiity generate conditioning factors such as the company, time, an image, for a given engagement level and caption. We leverage this EngugeNet as reward model to train stable diflusion such that the images generated by have design specification aligned with those higher engagement image. EOIG takes prompt generales an image, which then undergoes verhalization via image perception models. Its ohjective is to create images that, when verhalized, closely resemble angagement-condiioons verbalization generated by EngageNet. The verbalizeed ourput of EOIG-SD is
descriptions comprising colces, tomes, objects their locations
image based caption and viewer likes. Thus, EngageNet inhercntly understands the design details
the reward model. We ask EngageNet predict the logits for this image verbalization, using which reward is computed for EDIG-SD, indicating how closely this verbalized ouiput aligns with EngageNet. improvement and refinmmen within the image generation process. Thus, this pipeline trains EOIG-SD to generate ongagemanntoppimizze images by gradually aligning its output with EngageNet.
reward value serves leedbuck BOIG-SD the farm of policy gradient, aiding continual
Figure 14: Aligning Stable Diffusion for higher engagement using DDPO algorithm BBlakk et al. 2023) using
two types of reward functions (a) Engagement Simulation (ES) and (b) Design Specification Generation (DSG)
31

---

Published as a conference paper at ICLR 2025
Model
ma 77:00 EOIG
EIGG
Batter Prompts
Ground Truch
Botter rompone
Sp.2 SoMMedm w
Ground
SB 22 Medium
5DXL G0XL-009
Flx Fbw.1-DD
Sigaa-Sigma
PlaartAAlph
STO0
SDXL
Ground Truch w/
Better Prompts
0.80
0.81 0.51
0.04
0.85
0.88 0.68 u88
0.72
0.73
0.75
0.24
0.74
0.75
Ground Truth
0.64
0.7
SD-3-Medium
Better Prompts
0.52 0.52
0.54
0.57
0.88
0.00
0.62 0.05 0.65
0.65
0.57
SD 3 Medium 0.30
0.46
0.48
0.30
055 034
0.58
0.38 035 0.80 0.83 0.53 0.64 0.64
0.55
SDXL-DPO
Flux.1-De) Dev 0:30
0.48
0.46
0.48
0.48
0.52 0.53
0.55
0.69
0.68
0.61 0.61
0.61
0.62
0.50
SDXL 0.35
0.41
0.44
0.45
0.46
0.48
0.51
0.53
0.54
0.58
0.59
0.59
0.60
0.60
0.64
Model A
PixartSigmaa
0.5
Pixart PlshrtAApha 0.22
0.30
01443
0.42
0.43
0.45
0:77
0.48
0.00
0.5
0.56
035
0.55
0.57
0.34
SD-2.1 am1
0.30
0.80
0.47
0:44
0.00
0.50
0.33
0.55
0:55
0.56
0.56
0.50
0.4
SD-15
027
60333
01:25
0.32
0.37
0.39
0.11
0.42
0.44
0.45
0.47
035
0.1 05: 0.53
DALLE-22
EDIG SD-14 RUHF-DSG
0.3
EDIG SD-14 RLHF-ES
6:37
0.38
0.38
0.36
0:38
8.40
0.41
0:43 0.M
0:15
0:68 0,19 0.50
0.59
SD-14
Predicted Win Rate Using Elo Ratings for Model A in an A vs. B Battle
Figure 15: Win Rates different models against each other Image Engagement Arena
32


Reviews:
[{"review_id": "o6b3dn0gKG", "reviewer": "Reviewer_wnMa", "summary": "This paper introduces EngageNet, a text-to-image generation model optimization method to improve the audience engagement of the generated images, rather than merely the aesthetic quality and realism. Based on this, the first automated arena, i.e., EngageNet Arena, is proposed to benchmark the engagement of text-to-image models.", "questions": "What are the further direction in this research topic? Given the situation that many labs may lack computational resources, how to make it possible for these researchers to follow your work? Will the EngageNet also be appliable to other research domains, such as multimodal item recommendation?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The idea of the engagement-optimized image generation is novel, as images are often created to drive user engagement beyond just aesthetic appeal.\n- The paper introduces a large dataset of 168 million tweets with images and associated user engagement metrics like likes, paving the way for such research direction.\n- Multiple methods are explored to enhance the engagement of text-to-image models, including prompt conditioning, supervised fine-tuning, and reinforcement learning.", "weaknesses": "Null", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "02zKU4fWuW", "reviewer": "Reviewer_mpVe", "summary": "This paper investigates how current text-to-image generation models fall short of predicting viewer engagement. To address this gap, the authors propose several contributions including (1) EngagingImageNet: A dataset of high-quality enterprise images alongside engagement metrics like user likes, which serve as real-world indicators of viewer engagement, (2) EngageNet: A Vision Language Model (VLM) trained on EngagingImageNet to predict engagement by analyzing contextual factors around each image, such as tweet content, enterprise details, and posting time, and (3) Engagement Arena: A benchmarking platform where EngageNet scores various image generation models based on their capacity to generate engaging images. The paper explores approaches to enhance engagement in generated images, including prompt-based conditioning, fine-tuning, and reinforcement learning with EngageNet-generated reward signals.", "questions": "-  Can you provide further detail on how negative samples were generated and their effects on training EngageNet? How does the negative sampling impact model robustness and alignment with actual user engagement?\n- Can EngageNet be adapted to predict other engagement metrics (e.g., CTR, shares) in addition to likes?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper presents an interesting application to optimize image generation for viewer engagement, a metric commonly aligned with recommendation and commercial objectives. \n- The dataset curation and model development appear thorough. EngagingImageNet is large, high-quality, and potentially valuable for further research on engagement in visual content. The paper includes rigorous validation of EngageNet’s correlation with actual engagement metrics, offering a credible alternative to existing models.", "weaknesses": "- As EngagingImageNet is sourced from enterprise accounts on Twitter, the dataset may have an inherent bias toward corporate content. For a model aiming to set a standard for engagement-optimized image generation, more diverse data sources would provide a broader foundation and improve applicability across various domains. \n- The effectiveness of engagement-enhancing techniques is primarily measured using EngageNet scores and correlations. Incorporating additional real-world engagement experiments, for example, a small-scale human evaluation study, would strengthen the claims regarding the improvements in viewer engagement and provide additional validation.\n- The negative sampling method for training EngageNet is vaguely described. The paper mentions that negative samples are randomly generated by pairing tweets with unrelated images, yet does not explain how the random sampling affects EngageNet's robustness or alignment with true engagement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GRvxbwtdmL", "reviewer": "Reviewer_k6Wd", "summary": "This study constructs a large-scale dataset of images paired with user engagement metrics to investigate image engagement and utility in real-world marketing contexts. An engagement-aware visual language model (VLM) is developed to predict viewer engagement. Contemporary techniques are explored to enhance engagement in text-to-image models, including conditioning image generation on optimized prompts, supervised fine-tuning of Stable Diffusion on high-performing images, and reinforcement learning to align Stable Diffusion outputs. The authors also introduce a benchmarking method for text-to-image engagement models, offering the research community a standard framework for evaluating advancements in engagement modelling.", "questions": "1.\tA more precise definition and scope of viewer engagement would be helpful, as the current metrics appear to include clicks, likes, and shares. Additionally, detailed statistics of the dataset should be provided to help the community better understand this dataset.\n2.\tGiven the dataset spans over a decade, how is the temporal information being utilized?\n3.\tWhile the introduction of the engagement dataset and benchmarking metrics is a valuable contribution, the proposed methods—conditioning image generation, supervised fine-tuning, and reinforcement learning for diffusion alignment—are relatively straightforward adaptations of existing techniques. A more innovative approach to these tasks could elevate the paper’s impact.\n4.\tThe paper could discuss a broader range of use cases and scenarios, such as recommender systems or advertising, beyond standard text-to-image applications, especially given the marketing focus of this work.\n5.\tAn in-depth discussion on potential use cases would be beneficial.\n6.\tThe paper does not clarify whether any collected data contain identifiable information. If they do, obtaining informed consent from individuals would be essential.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed method is robust, producing a large-scale, high-quality dataset that addresses the gap in image engagement and utility analysis.\n2.\tThe paper is clearly written and well-structured.\n3.\tThe authors evaluate various models, offering valuable insights into the dataset’s application and benchmark characteristics.", "weaknesses": "Please see questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tTqDN63TIy", "reviewer": "Reviewer_wzeb", "summary": "This paper introduces a novel consideration in the text-to-image generation task: human engagement. The authors first collect a large dataset, EngagingImageNet, from an online platform. They then train EngageNet to assess the engagement level of generated images based on auxiliary information embedded in the prompts. Additionally, they propose methods to leverage EngageNet to enhance the performance of text-to-image models. Finally, the authors introduce Engagement Arena to assess the engagement levels of different text-to-image models. This work presents a comprehensive framework for evaluating and improving engagement of text-to-image models.", "questions": "1. Why did the authors not consider including retweets as part of the engagement indicator?\n\n2. Why was SD 1.4 chosen for train-time optimization, as it is one of the weakest models?\n\n3. Does KPI refer to the normalized likes of an image? So the regression value of KPI is just used in training phase?\n\n4. I would like to see an analysis of the complexity or latency of the proposed components, such as the Retrieval module, since an excessively long generation process would be impractical for real-world applications.\n\n5. All concerns in Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The collected dataset EngagingImageNet is a strong foundation, as authors provide the detailed introduction of obtaining and filtering this datasets. This dataset encompasses multimodal information for each image, supporting not only engagement assessment but also various other applications.\n2. The author trained the EngageNet with several tricks, obtaining a superior performance. The release of such model would offer both academia and industry a new perspective for evaluating generated images.\n3. This paper introduces multiple strategies for enhancing the engagement of generated images, during run-time and train-time. They achieve notable improvement using these approaches.", "weaknesses": "1. The paper would benefit from a more detailed explanation of the DDPO method utilized, including the hyper-parameters and loss terms.\n\n2. The paper lacks a user study or online experiment to evaluate the engagement of images generated by the fine-tuned models. It is unclear whether the fine-tuned models genuinely improve engagement or merely optimize to satisfy the reward model.\n\n3. The paper lacks discussion of some relevant works:\n\n[1] Rich Human Feedback for Text-to-Image Generation, CVPR 2024.\n[2] Towards Reliable Advertising Image Generation Using Human Feedback, ECCV 2024.\n[3] Directly Fine-Tuning Diffusion Models on Differentiable Rewards, ICLR 2024.\n\nMoreover, other works on using human feedback to improve image generation models should be considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)