Social Reward: Evaluating and Enhancing Generative AI through Million-User Feedback from an Online Creative Community
Authors: Arman Isajanyan
Artur Shatveryan
David Kocharian
Zhangyang Wang
Humphrey Shi
Venue: ICLR 2024 spotlight (2024)
Keywords: human feedback
text to image
generative AI
image quality scoring
Primary area: datasets and benchmarks

Abstract: Social reward as a form of community recognition provides a strong source of
motivation for users of online platforms to actively engage and contribute with
content to accumulate peers approval. In the realm of text-conditioned image
synthesis, the recent surge in progress has ushered in a collaborative era where
users and AI systems coalesce to refine visual creations. This co-creative pro-
cess in the landscape of online social networks empowers users to craft original
visual artworks seeking for community validation. Nevertheless, assessing these
models in the context of collective community preference introduces distinct chal-
lenges. Existing evaluation methods predominantly center on limited size user
studies guided by image quality and alignment with prompts. This work pio-
neers a paradigm shift, unveiling Social Reward - an innovative reward modeling
framework that leverages implicit feedback from social network users engaged
in creative editing of generated images. We embark on an extensive journey of
dataset curation and refinement, drawing from Picsart: an online visual creation
and editing platform, yielding a first million-user-scale dataset of implicit human
preferences for user-generated visual art named Picsart Image-Social. Our anal-
ysis exposes the shortcomings of current metrics in modeling community creative
preference of text-to-image models’ outputs, compelling us to introduce a novel
predictive model explicitly tailored to address these limitations. Rigorous quan-
titative experiments and user study show that our Social Reward model aligns
better with social popularity than existing metrics. Furthermore, we utilize So-
cial Reward to fine-tune text-to-image models, yielding images that are more fa-
vored by not only Social Reward, but also other established metrics. These find-
ings highlight the relevance and effectiveness of Social Reward in assessing com-
munity appreciation for AI-generated artworks, establishing a closer alignment
with users’ creative goals: creating popular visual art. Codes can be accessed at
https://github.com/Picsart-AI-Research/Social-Reward

Published as a conference paper at ICLR 2024
SOCIAL REWARD: EVALUATING AND ENHANCING GENERATIVE AI THROUGH MILLION-USER FEED- BACK FROM AN ONLINE CREATIVE COMMUNITY
Arman Isajanyan'; Artur Shatveryan'; David Kocharyan', Zhangyang Wang".,, Hummhrry Shit.3 Picsart AI Research (PAIR), PUT Austin, GGerrgia Tech
farman. isajanyar,,nn artur. shatveryan, david. kocharyan, atsas. wang, humphrey.. shi.eeyys..shipppcssttt.
ABSTRACT
Social reward as a form of community recognition provides strong source motivation for users of online platforms actively engage and contribute with content to accumulate peers approval. In the realm of text-conditioned image synthesis, the recent surge in progress has ushered in collaborative era where users and AI systems coalesce to refine visual creations. This co-creative pro- cess in the landscape of online social networks empowers users craft original visual artworks seeking for community validation. Nevertheless, assessing these models in the context of collective community preference introduces distinct chal- lenges. Existing evaluation methods predominantly center on limited size user studies guided by image quality and alignment with prompts. This work pio- neers paradigm shift, unveiling Social Reward innovative reward modeling framework that leverages implicit feedback from social network users engaged in creative editing of generated images. We embark dataset curation and refinement, drawing from Picsart: an online visual creation and editing platform, yielding first million-user-scale dataset of implicit human preferences for user-generated visual art named Picsart Image-Social. Our anal- ysis exposes the shortcomings of current metrics in modeling community creative preference of text-to-image models' outputs, compelling us to introduce a novel predictive model explicitly tailored to address these limitations. Rigorous quan- titative experiments and user study show that our Social Reward model aligns better with social popularity than existing metrics. Furthermore, we utilize So- cial Reward to fine-tune text-to-image models, yielding images that are more fa- vored by not only Social Reward, but also other established metries. These find- ings highlight the relevance and effectiveness of Social Reward in assessing munity appreciation for Al-generated artworks, establishing a closer alignment with users' creative goals: creating popular visual art. Codes can be accessed https: ptttps://ithhb..co wwt.://s....cc.........a.ccc.... -Ree--AAeeeeee/oooctttoo - Reward
extensive journey of
INTRODUCTION
Social reward mechanisms play a pivotal role in incentivizing and modulating human behavior. Grounded in neurobiology and psychology, positive social feedback, such approval, validation, and recognition, are essential for maintaining social cohesion and individual well-being (Rudolph)
2021 Baumeister Leary 1995). This reward-driven behavior extends to online social platforms, where users seek satisfaction via the accumulation of their network's 's peer engagement with shared
content in forms such as likes or views (Deters & Mehl 2013 Lemai Nguyen & Nallaperuma 2023).
Recently, the field of cond-ionddio image synthesis has witnessed remarkable advancements, images that closely adhere to textual descriptions. 'Tiis technological breakthrough has significantly impacted the realm of online social networks, as it empowers users with a novel and creative means of content creation and sharing. As users leverage this technology to craft and post compelling visual content,
leading to the development of generative algorithms capable of producing
"Equal contribution

---

Published as a conference paper at ICLR 2024
Social Reward
Image Reward
Prompt
HPS v2
PickScore
Digital anime art of mattress-man with serious expression an emply warehouse, highly detailed
Encrchaivie
energizing morning routines
ALLAI
princess crown
beautiful woman standing in a dystopian city
Figure I: Best image out of 20 generations as chosen by different scoring models, including ours.
they simultaneously tap into the well-established reward mechanisms of social validation and recog- nition. Given the pace with which the number of synthetic images grows filling the digital spaces of online creative communities (as of August 2023 more than 15 billion synthetic images had been generated globally (Valyaeva, )023) evaluating the performance of generative models within the context of social network popularity emerges as an important challenge.
While social network popularity be defined in many ways, with likes, views, and other similar types of user interactions traditionally serving as popularity estimates (McParlane et al.) 2014] Ding et al. 2019), the nature of text-to-image technology, adopted by industry largely as co-editing tool integrated into creative platforms (Weisz et al.] 2023 Huang & Grady] (022). introduces another dimension to social network content popularity measurement, namely the frequency of synthetic image reuses for editing purposes by community members. This metric resonates with the popula- tion of artists and creators receiving social rewards when their synthetic images are being leveraged in the editing process by network peers. The central question then can be summarized as, to what ex- tent text-to toettt--mmee models can produce visual content aligned with social popularity, which defined as community preference for creative editing purposes?
Recently, researchers have increasingly turned to human preferences as a guiding beacon, inspired by the transformative impact of human feedback in the realm of Large Language Models (LLM) (Ouyang et al. 2022 Nakano et al., 2022). In the domain of text-to-image generation, reward models have been harnessed to channel human feedback into the learning process Xu et al. (202a));
Wu et al. (003a): Kirstain et al. 2023). These works, have sought to leverage human preferences to construct reward models that facilitate generative model evaluation and fine-tuning.
Despite the commendable efforts, the existing reward models have notable limitations in our domain of interest. Some of them rely on limited-size data annotation process, presented in Table which cannot be deemed the equivalent of the "community-scale" feedback. Moreover, prompts utilized for dataset creation (collected from COCO Captions dataset (Chen et al., (015)) and open source prompt dataset DiffusionDB (Wang et al. 2023 along with the moderation process and guidelines that adhere mainly to image fidelity and textual alignment as annotation criteria, do not emphasize creative purpose and hence, potentially, are limiting in expressing collective community creative preference. On the other hand, some other approaches collect explicit organic user feedback, but as downside, exhibit relatively small scale of collected user preference and absence of "collective feedback" (when more than one user engages with a given image) as an important indicator of social popularity. While these approaches do capture a broad spectrum of user preferences, they never- theless showcase insufficiency to model social popularity in the context of community-leve editing preference. These limitations substantiated by extensive quantitative and qualitative analysis.
To bridge this gap and address the unique demands of text-to-image in the framework of creative social popularity, introduce novel concept: Social Reward. This paradigm shift in reward modeling leverages collective implicit feedback from social network users who specifically employ generated images for creative purposes. This distinction underscores the relevance and applicability
2

---

Published as a conference paper at ICLR 2024
of Social Reward to the creative editing process, providing more faithful estimation of alignment between Al-generated images and community-
creative preference. However, the collection of Social Reward data presents its own set of challenges, notably the inherent noise stemming from the implicit nature of user feedback, the absence of formal annotation process with precise guidelines, and the unequal content exposure caused by social network-specific factors (such as content being surfaced more frequently than the others, etc).
In this work we embark on comprehensive exploration, starting with data curation sourced from Picsart (nttps: / /wtp/ppcc.t...eeee com/): one of the world's leading online visual creation and edit- ing platforms. Due to the inherent noise and subjectivity in individual user choices, the collective feedback, which implies multiple users' editing choices for the given content item, is leveraged as cleaning mechanism of organic implicit user behavior. Several more data collection techniques have been utilized for addressing such biases as caption bias, content exposure time, and user follower base biases. Our analysis reveals the shortcomings of existing metrics evaluating text-to-image models' fitness for generating popular art, which motivates to introduce a model explic- itly designed to address these limitations. Moreover, we demonstrate the potential of our model in fine-tuning eext-to-image models to better align with community-lever creative preference.
Our contributions are outlined as follows:
We identify an unexplored, but extremely relevant dimension in human preference reward modeling for text-to-image models: evaluating the performance within the context of social network popularity for creative editing. Our analysis provides compelling evidence that existing reward models are ill-suited to capture this dimension.
We embark on a journey of dataset curation and leveraging Picsart's creative community data. We build large scale dataset of implicit human preferences motivated by creative editing intent over synthetic images, named Piesart Image-Social dataset. Contrary to existing methods, we utilize social network user feedback and curate dataset relying editing community collective behavior.
Building upon this curated dataset, we develop and validate our Social Reward Model, showcasing its superiority for the given task, as evidenced in Table Figure Further- more, our model captures distinct image attributes that go beyond mere aesthetics (see Figure D]. demonstrating its potential to enhance text-to-image model performance for community-leve creative preference (see Table 5lble5aam and Figure
2 RELATED WORK
2.1
TEXT-TO-IMAGE GENERATION
tomagee-immeeeee generative models allow synthesizing images conditioned on the text input. GANs
(Goodfello et al.) 2014) allowed for the first successful results in this realm (Zhang et al. 2017
Xu et al.[[00 2017 Zhu et al. 2019 Liao et al. 2022). Transformer- based (Ramesh et al. 2021 Chang al.] 2023) models have also yielded great improvements. Recently diffusion-based architectures
showed great ability of producing high fidelity images (Nichol et al. [202]] Ramesh et al.) 2022]
Xu al. 2023b; Lu et al. 2023). LDM (Rombach et al. (2022) employs diffusion process in the underlying latent space rather than directly in the pixel space. This approach delivers notable performance gains while also enhancing processing speed.
2.2 POPULARITY PREDICTION
The domain of predicting content popularity within social networks has garnered substantial atten- tion in recent years, primarily due to its relevance in comprehending content diffusion dynamics, modeling community preference patterns, and the optimization of marketing strategies. Most exist- ing works focus on the following types media content: text (Oza & Naik 2016 Gao et al., 2019),
video (Li et al. 2013 Rizoiu et al. 2017) and images (Khosla et al.) 2014) ]. In the realm of image popularity prediction most existing works use either Flickr (McParlane et al.) 2014) or Instagram
(Ding et al. 2019) for their dataset curation. While significant research has been done on social content popularity topic, little attention has been put on the emerging field of synthetic/generated images and their fitness for popularity in creative community.
3

---

Published as a conference paper at ICLR 2024
2.3 TEXT-TO-IMAGE EVALUATION
Given that the focus of this study lies in the realm of evaluating the performance of text-conditioned image generation models, it is imperative to assess the latest advancements in related research. While widely accepted evaluation methods within the image synthesis community, commonly re- ferred to as "automated metrics" like FID (Heusel et al., 2018) and CLIP score (Radford et al,
202T have demonstrated certain drawbacks (Otani et al. |023) Parmar et al.
2022 Xu al. 2023a Kirstain et al. 2023), one notable concern is their limited a alignment with human preferences.
To tackle this issue, recent endeavors (Kirstain al. 2023 Xu al. 2023a Wu et al. 2023a) have proposed direct modeling of human preference by learning scoring functions from datasets consisting of human-annotated prompts paired with synthetic images. While these studies represent a significant stride towards enabling practitioners to approximate human preference for generative model outputs, they still exhibit several inherent limitations concerning the core objective of our research, which revolves around addressing the challenge of predicting social popularity.
A subset of the prompts used in HPD v2 (Wu et al. 2023a) is sourced from the COCO Captions dataset, which comprises captions linked to real images. This incorporation raises concerns about a potential domain mismatch when evaluating the scoring of generated images. Furthermore, the gathered feedback stems from a relatively restricted number of annotators, rendering it insufficient to encapsulate the preferences of a large-scale user base. In contrast, our feedback is drawn from user community numbering in the millions individuals who actively engage with these images for editing purposes.
Likewise, ImageReward (Xu et al. 2023a) dataset faces limitations not only in terms of an- notators number but also terms relatively low number of unique prompts and images.
One important shared concern by both ImageReward and HPD v2 is the absence of specific guidance to direct annotators toward emphasizing creative editing goals. Instead, their focus was primarily ensuring image fidelity and aligning text with images.
We performed prompt analysis to compare prompts derived from the datasets used for training scoring models, which includes prompts from ImageReward and Pick-a-Pic (Kirstain et al. 2023 I with prompts crafted by platform's 's creators, which by our popularity metric are reflective of creative image editing intent. It is evident that prompts in Rewae and Pick-a-Pic datasets significantly deviate from those leveraged for creative editing.
In Pick-a-Pic case prompts are generated by web app. created by paper authors for research purposes. Users had been invited to interact with applications via such channels as Twitter, Facebook, Discord, and Reddit. The relatively small scale of collected user preferences along with the absence of "collective feedback" (understood as different users' independent choice to interact with a given image) make Pick-a-Pic less optimal approach for community popularity modeling.
Table Comparison of human preference datasets for text-to-image evaluation
Name
Annotatur
type
Annolator
focus
Prompt
source
Image
Number of Images
Image Puirs
Distinet Prompts
Users! Amnolators
v2
Professional
nnactaters
Image Quality
Text Aliganent
DiflusioDD
captions
COCO Capticus
430K
798K
164K
57"
ImageHeward
Protessional
ansctators
Text Aligsment
Quuiity
DiflusionDD
Diffusion
Stable
137K
8.K
Pick
Real users
Individual
Feedback
Pick-aa-Pic
With
modelldonnfiss
Different
656K
615K
38.5K
6.1K
Image-Soou
users
Feedback
Social platfomm
user prompts
models
inhuuse
164K
of them are quality sheckers.
After annoation quality inspectors double checked annotation,
assigned other annotators relabeling.
HHP v2 by the time of this paper writing didn't 't share the training part of the dataset
4

---

Published as a conference paper at ICLR 2024
3 SOCIAL PREFERENCE: NEW DATASET CURATION AND ANALYSIS
3.1 PICSART AS EDITING-CENTRE CREATIVE COMMUNITY
Picsart stands one of the world's largest digital creation platforms, encompassing wide spectrum of Al-driven editing tools, notably including lext-to-image capabilities. This comprehensive suite empowers creators of varying proficiency levels to conceive, refine, and disseminate photographic and video content. Central to the platform's 's appeal is its robust open-source content ecosystem, perpetually invigorated by a vibrant user community.
Picsart serves both personal and professional design needs, distinguished by distinct social di- mension. This social facet allows users to share their creative edits, which, in turn, can be harnessed by fellow members of the platform. Consequently, the popularity of user escalates when they share images that find utility among their peers. Picsart also incorporates a sophisticated search component, enabling users to locate and utilize one another's edits effectively.
3.2
DATA COLLECTION
In the realm of popularity prediction, most studies commonly employ metrics such as comments, views, or likes as labels for prediction (McParlane et al. 2014) Khosla et al. 2014 Ding et al.) 2019). However, Picsart's 's distinct creative nature has prompted us to explore a rather unique metric, called remix: the number of times image has been reused for editing purposes by other users. This intriguing metric represents one of the most prevalent editing behaviors within our community. What sets remixing apart from conventional popularity signals is its deeper level of user engagement. It is not merely a passive indicator, but rather a testament to active involvement, that allows discerning, which synthetic images hold greater appeal for transformative and artistic modifications. Essentially, our community implicitly conducts a form of collective voting, determining the fitness of prompt- image pairs for creative editing.
We have established specific criteria for identifying positive (popular) and negative (unpopular) images associated with given prompt, grounded in the following community-driver editing signals:
Content Signal: We consider the frequency with which a given image has been remixed by members of the community.
Creator Signal: When an image is remixed by top influencer artists within our community.
In addition visual and textual attributes, there exists multitude of factors influencing image popularity. Much like the approach followed by Ding et al. (2019), we have taken careful measures in collecting data mitigate potential sources of biases:
Prompt Bias:
To reduce the impact of the prompt that cause non-even content distri- bution in the platform, we condition our model on the prompt, accompanied by a pair of popular and non-popular images.
Content Exposure Time Bias: Since some images receive higher viewership than others, for the given prompt we select unpopular images with relatively close exposure time to popular ones.
User Follower Base Bias: Difference in the size and engagement of individual user fol- lower bases does not affect the popularity of generated images, because all those images are posted under the same Picsart public profile.
Our meticulously collected datase F known as the Picsart Image-Social dataset is represented by triplets: prompt, positive image, and negative image. Additionally, mature images were filtered out by in-house NSFW detection algorithms. For a more detailed description of the data collection procedure, please refer to Appendix A..2] With careful consideration given to avoiding prompt- level intersections, the dataset is partitioned into training (70%), validation (10%), and test (20%) sets. High level statisties of Piesart Image-Social dataset can be found in Table For additional information please refer to Appendix A.3
Picsarrt is company that complies with GDPR, CCPA, and other data protection legislation and is collect- ing, storing, and processing users' data by the consent received. It also allows the users to opt out of certain processing purposes, requested.
5

---

Published as a conference paper at ICLR 2024
blue and purple gradient
(b) blue lock
superman logo
G
(d) logo combining book with (e) the number 3 in shiny gold (I) golden round frame black heart
liquid font...
background
Figure 2: Picsart Image-Social training set, showing popular (left) and non-popular images (right)
Leveraging our social network community has allowed us to amass a larger dataset than parallel text-to-image human preference modeling efforts. Despite the implicit nature of human preferences in our dataset (users edit images organically for creative purposes) the substantial feedback volume minimizes the risk of noisy signals. Each instance in the Picsart Image-Social dataset represents collective, independent, implicit voting by our user community, rather than the preference of single member. Examples our collected data are illustrated in Figure 2
3.3 EMPIRICAL ANALYSIS OF EXISTING EVALUATION FRAMEWORKS
To demonstrate the distinctiveness of the Picsart Image-Social dataset in comparison to those em- ployed by existing solutions and to highlight the limitations of these solutions, we undertook com- prehensive analysis. Our three-fold fold analysis (prompt analysis, comparative quantitative evaluation on collected test set and comparative label explainability analysis of score models datasets) reveals that current evaluation models are poorly suitable for estimating the social popularity of images.
3.3.1 PROMPT ANALYSIS
Training datasets of Picsart Image-Social ImageReward and Pick-a- have different sources which also indicates distinct motives for image generation. To illustrate Picsart Image-Social's uniqueness and that it reflects creative editing intent, we conducted cluster analysis on the Sentence-BERT (Reimers & Gurevych] 2019) embedding of the prompts in the training datasets. For fair comparison we sampled an equal amount (6000 unique prompts) data from each dataset, filtered non-english prompts and truncated prompts to only include the first 5 words. Clustering done using Ward's hierarchical clustering method (Ward) 1963). identifying 173 clusters.
Table 2:
KL divergence of training prompts for different scoring models
Clusters distribution of the datasets in Figure 4 and KL. divergence between cluster distributions (Table 2 reveal Picsart Image-Social to exhibit the most pronounced dis- similarity from the others. Interestingly, the clusters where our training dataset manifests the highest density com- parison with ImageReward and Pick-a-Pic are character- ized by themes such as background photos, flowers, terior and exterior design, and fashion, which are more aligned with popular content themes visual creative art (see Figure 3).
Model
Model
KL
Picsart Image-Social
Pick-a-Pic
0.6
Picsart Image-Social
ImageReward
0.8
Pick-a-Pic
ImageReward
0.35
3.3.2 QUANTITATIVE ANALYSTS
To gauge the effectiveness existing solutions in predicting social popularity, specifically in the context of image editing, we conducted an evaluation of three models: PickScore (Kirstain et al.) 2023). ImageReward (Xu et al. 2023a), and HPS v2 (Wu et al., 2023a). using the Picsart Image- Social test dataset utilizing their respective model evaluation code bases. Our primary evaluation metric was pairwise accuracy. Table Abbleppreeettt the results this evaluation, with PickScore achiev- ing the highest accuracy rate of 62.6%. Notably, this accuracy remains relatively modest.

---

Published as a conference paper at ICLR 2024
Darase Moute Milroag Socid Pice
3
210
300
Chister Numberr
Generated images from prompt clusters distinct for creative editing
Figure 3:
Figure Cluster distribution of training prompts for different scoring models
This outcome can be primarily attributed to PickScore's ' data collection approach, which involved incorporating real user preferences, in contrast to ImageReward (60.48%) and HPS v2 (59.4%), which relied on annotations from moderators.
3.3.3
AGREEMENT OF CLIP AND LAION AESTHETIC SCORES WITH EXISTING METRICS
To draw insight from the relatively sub-optimal performance of existing score models on the Piesart Image-Social dataset we utilize vanilla CLIP (Radford et al., 2021) and LAION (Schuhmann al. 2022) aesthetic models as estimators of text-image alignment and acstheticsffidelityy since two of the three of analyzed models (HPS v2 and ImageReward) used those criteria annotation guidelines
during data collection stage.
By conducting the inference of vanilla CLIP and LAION aesthetic models every score model's test sets (including Picsart Image-Socce Social) and calculating pairwise accuracy, we can observe, as depicted in Table 33, that CLIP and LAION aesthetic scores exhibit the lowest levels of predictive efficacy on Picsart Image-Social dataset. This result underscores the notion that human preferences our dataset reflect an image quality dimension that is discernibly separate from concepts of general fidelity and text-image alignment. This observation extends our understanding of the comparatively modest performance of other scoring models on the Picsart Imaee-Social dataset.
Table 3: Pairwise accuracy of LAION aesthetic and CLIP image-text alignment scores test sets.
LAION aesthetic score
CLIP alignment score
Dataset
Piesart Image-Social
55.3%
51.9%
Pick-a-Pic
56.8%
60.8%
ImageReward HPD v2
57.35%
54.82%
72.6%
62.5%
4 SOCIAL REWARD: A NEW METRIC MODEL.
In this section, we present our Social Reward model, which has been trained using user feedback collected from Picsurí. We will describe the various backbone models and loss functions considered during our experiments, and conduct comparative analysis with existing solutions.
4.1 SOCIAL REWARD MODEL TRAINING
Social Reward trained through fine-tuning of the CLIP model. When provided with prompt and an image, scoring function calculates real-valued score. This score is deter- mined by representing the prompt using a transformer-based text encoder and the image using a raasformer-based image encoder, both as d-dimensional vectors and then computing their cosine similarity. Fine-tuning the last two residual blocks of the textual encoder and the last three residual blocks of the visual encoder in CLIP yields superior results on validation dataset.
Model:
Loss function: We experimented with various loss functions and observed that the triplet (Schroff al. 2015) loss yielded the most favorable outcomes (please refer Appendix B for details):
7

---

Published as a conference paper at ICLR 2024
(a) babylon
(b) suburbs
(c) work from home
(d) charlotte fortmite...
(c) eagle holding soccer ball logo
(f) a boy ... with a lion
Figure 6: Generated images ranked by Social Reward from best (left) to worst (right)
Cuipler (a, p, n) = max(0, ||a - pll2 - 22|2+a0 +
where is the vector representation of the prompt, is the vector representation of the positive image, n is the vector representation of the negative image and margin.
Hyperparameters: We employed the AdamW optimizer with learning rate of 0.0003 and a batch size of 32, utilizing a distributed computing setup consisting of 8 A100 GPUs.
Experiments: We conducted wide range of experiments including the exploration of various model backbones (such as CLIP and BLIP (Li et al., 2022)), fine-tuning with different components of these models (visual and textual encoders), and more ablation studies.
4.2 SOCIAL REWARD MODEL EVALUATION
We compared our model with existing solutions using a dedicated test dataset. We assessed our performance us- ing the pairwise accuracy metric, based on prompt-image cosine similarity distance. As shown in Table 4 our model outperformed existing solutions in this evaluation. The codebase for Social Reward model training and evaluation is provided as supplementary material. Figure 6 demonstrates ranking of the images by Social Reward score (for more visuals see Appendix Figure 9).
Table 4: Models Comparison
Model Name
Accuracy
Social Reward
69.7% %
PickScore
62.6%
ImageReward HPS v2
60.48%
59.4%
To validate the results and check their generalizability conducted a user study and compared with PickScore model, as has the second highest accuracy on our dataset and also involves real users feedback. We gen- erated 20 images with Stable Diffusion 2. 1-bae (Rom- bach et al. 2022) with the 100 prompts sampled from PickScore's s test set and chose the best image by Social Reward and PickScore. Then popular creators from Picsart and collected their feed- back with respect which of the two images is more likely to get popular on the platform. Figure Sysows So- cial Reward outperforms PickScore ability to estimate social popularity and generalizes well to prompts outside of its training distribution.
Social Reward
52%
17%
contacted number of
31%
Tie
PickScore
Figure 5: Human evaluation results using prompts from PickScore's test set.
4.3 USING SOCIAL REWARD FINE-TUNE T21 MODELS
Training: Social Reward can be used to fine-tune generative models to better align with the creative preferences of the community. For million generated images by the users of our community, we calculated the Social Reward score and adapted the fine-tuning method from Wu et al. (2023b). for given prompt choosing the image with the highest score and the image with the lowest score. For the image with the lowest score, added special token ('ı@y quality") to be used at inference time as a negative prompt. We also adapted the common practice adding 625k subset of LAION-5B [Schuhmann et al., 2022) for regularization. The final training dataset consists of 330.000 images, half of which come from LAION's subset. We fine-tune both the U-Net and the textual encoder of Stable Diffusion 2.1 h-aase model with leaming rate of 1e-5.
8

---

Published as a conference paper at ICLR 2024
Evaluation: We caleulate several metrics for images generated by vanilla and fine-tuned models. Evaluation is done on separate sets of prompts: internal and general. Internal prompts are selected taken from DiffusionDB Wang et al. 2023). For each prompt, we generate 4 images with each model using DDIM scheduler (Song et al. 2022) for 30 steps. In Table present average scores and in Table percentage of times the fine-tuned model outperformed the baseline for each metric and prompt set. Visual comparisons in Figure and Appendix Figure
from fine-tuning validation prompts set, while general prompts
Fine-tune
Baseline
bochground rrees, sky and grass anioe, onime
Highaning lightngg
from
Prompt
Walverian, highly desuied...
isleja kir motonrycle
pink gliter feathers. soft lighting
sharp focus, cinematior,
Biohazand highly detuiled diginal painingg
oil
Figure 7: Generated image comparison between Baseline and Fine-tuned models
Table Comparison between Baseline 1-base)-bsse) and fine-tuned models
Social Reward
CLIP alignment
LAION aesthetic
Image Reward
Model
Prompt
HPS v2
PickScore
Baseline general
-0.095
0.280
5.902
0.259
0.117
0.199
Fine-tune
general
-0.062
0.278
6.011
0.261
0,288
0.201
Baseline
internal
-0.130
0.260
5.746
0.258
-0,078
0.199
Fine-tune
internal
-0,093
0.256
5.892
0.261
0.110
0,200
Table 6: Fine-tuned model win-rates vs Baseline
Social Reward
CLIP alignment
LAION aesthetic
Image Reward
Prompt
HPS v2
PickScore
internal 75.6%
43%
73.8%
73.4%
70.6%
63.2%
general
75.6%
43.8%
71.2%
71.4%
70.6%
67.8%
CLIP alignment the only metric with slight decrease which be explained by the fact that Picsart Image-Social's training data reflects social feedback of the image; users might compromise text-image alignment for the sake of selecting an image more suitable for creative editing. On the other hand, all other metrics have increased even on general prompts which shows that Social Reward not only assesses the potential image popularity in the context of creative editing but also incorporates to high extent notions of general aesthetic present in other scores.
CONCLUSION
In this work, introduced new concept known Social Reward in the realm of text-conditioned image synthesis. Leveraging implicit feedback from social network users engaged with AI- generated images, addressed the unique challenge of evaluating text-to-image models in the context of social network popularity and creative editing. Our comprehensive journey involved dataset curation from Piesart, resulting in the creation of the Picsart Image-Social Social dataset, million- user-scale repository of implicit human preferences for aser-generated visual art. Through rigorous analysis and experimentation, we demonstrated that our Social Reward model outperforms existing metrics in capturing community-level creative preferences. Moreover, we demonstrate that employ- ing Social Reward to fine-tune text-to- text-to-image models leads to improved alignment not only with Social Reward, but also with other established metrics. By introducing this novel approach, aim to enhance the co-creative process, align Al Al-generated images with the majority users' creative objectives, and ultimately facilitate the creation of more popular visual arts in the digital space.

---

Published as a conference paper at ICLR 2024
REFERENCES
Roy Baumeister and Mark Leary. The need to belong: Desire for interpersonal attachments as fundamental human motivation. Psychological bulletin, 117:497-529, 06 1995. doi: 10.1037/ 0033-2909. 11..3.99.
Huiwen Chang, Han Zhang, Jarred Barher, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Msuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Texl-to-image generation via masked generative transformers, 2023.
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and
C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015,
Fenne Deters and Matthias Mchl. Does posting facebook status updates increase or decrease loneli- ness? an online social networking experiment. Social Psychological and Personality Science, 4: 579-586, 09 2013. doi:
Keyan Ding, Kede Ma, and Shiqi Wang. Intrinsic image popularity assessment. In Proceedings of the 27th ACM International Conference on Multimedia. ACM, oct 2019, doi: 145/333333.. 3351007. httpt: 2ttp::..0.0.11111113133333333333 3331333.333333111000 3351007
Xiaofeng Gao, Zhenhao Cao, Sha Li, Bin Yao, Guihai Chen, and Shaojie Tang. Taxonomy and eval- uation for microblog popularity prediction. ACM Trans. Knowl. Discov. Data, 13(2), mar 2019. ISSN 1556-4681. doi: 10.1 145/3301303. TRR.httts://///
org/ 10. 1 1145/3301300
lan Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium, 2018
Sonya Huang and Pat Grady. Generative ai: A creative world. https: www. com/ www.//atieeeeeeeeeeteeeoeeeeeeee
ai- wwwa-a-creatveeeeee.....eee world/ 2022.
Aditya Khosla, Atish Das Sarma, and Raffay Hamid. What makes an image popular? In Proceedings of the 23rd International Conference on World Wide Web, WWW 14. pp. 867-876, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781450327442. 3566486.2567999
doi: 10.1145/
httpp::://///:ttorr 101 /14.//1566866..212 256.4488.22666568 2567996
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning, 2021.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation, 2023.
Argho Bandyopadhyay Lemai Nguyen, Ojelanki Ngwenyama and Kaushalya Nallaperuma. Real- ising the potential of digital health communities: study of the role of social factors in com- munity engagement. European Journal of Information Systems, 000):136,, 2023. doi: URL hhttt::t////e/ /tt:::/doii.rro htt.://o/r/..1110000000000030100 2023.
008009900885..222222222 2252390
Haitao Li, Xiaoqiang Ma, Feng Wang, Jiangehuan Liu, and Ke Xu. On popularity predic- tion of videos shared in online social networks. In Proceedings of the 22nd ACM Interna- fional Conference on Information & Knowledge Management, CIKM '13, pp. 169-178, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450322638. doi: 111145/25055115255555533 URL IRRps:::: www.o0.gg/..... 11w...1155/22555555555555
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation, 2022.
Wentong Liao, Kai Hu, Michael Ying Yang, and Bodo Rosenhahn. Text to image generation with semantic-spatial aware gan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18187-18196 June 2022.
10

---

Published as a conference paper at ICLR 2024
Haoming Lu, Hazarapet Tunanyan, Kai Wang, Shant Navasardyan, Zhangyang Wang, and Humphrey Shi. Specialist diffusion: Plug-am and-play sample-efficient fine-tuning of text-to-image diffusion models to learn any unseen style. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 14267-14276, 2023.
Philip J. McParlane, Yashar Moshfeghi, and Joemon M. Jose. "nobody comes here anymore, it's too crowded"; predicting image popularity on flickr. In Proceedings of International Conference on Multimedia Retrieval, ICMR 14, pp. 385-391, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 978 450327824. doi: 10. 10.11457/57777262557777 //doi. 1145/25787222 2578776
URL https:
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-amswering with human feedback, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022.
Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin't ichi Satoh. Toward verifiable and reproducible human evaluation for text- to-image generation, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, 2022.
Kavita S. Oza and Poornima G. Naik. Prediction of online lectures popularity: A text min- ing approach. Procedia Computer Science, 92:468-477 2016. ISSN 1877-0509. doi: http:///rrr....01.//poooo.ooo..s
URL https: www sciencedirect. com/ 2nd International Conference on Intelli- gent Computing, Communication Convergence, ICCC 2016, 24-25 January 2016, Bhubaneswar, Odisha, India.
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks, 2019.
Marian-Andrei Rizoiu, Lexing Xie, Scott Sanner, Manuel Cebrian, Honglin Yu, and Pascal Van Hentenryck. Expecting to be HIP. In Proceedings of the 26th International Conference World Wide Web. International World Wide Web Conferences Steering Committee, apr 2017. doi: 10. 1145/3038912.30526550 URL https:://dci.ooee
1114582F308899 3052650
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2022.
Karen D. Rudolph. Chapter three understanding peer relationships during childhood and ado-
lescence through the lens of social motivation. In Andrew J. Elliot (ed.), Advances in Motiva- tion Science, volume 8 of Advances in Motivation Science, pp. 105-151. Elsevier, 2021. doi: https://bo.orr/10000000........
001. https:::: www. sciencedirect. com/ sci /

---

Published as a conference paper at ICLR 2024
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022.
Alina Valyaeva. Ai has already created many images photographers have taken in 150 years. statistics for 2023.
2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9886):557--6655, 2008. URL /ttp:// jm r. www.ppapprg././p/./..
wrandermaatennaa.ceo
Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. Denoising implicit feed- back for recommendation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM 211 ACM, March 2021. doi: 10.1145/84379663441888,, URL
http: / /xxx doi. org/10..//....o....ooooo. 1111/11319111311 3441800
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondh: large-scale prompt gallery dataset for text-to-image gener- ative models, 2023.
Joe H. Ward. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58:236 244, 1963. URL/tttss:// /app: wwmanticscholla.tt....toaoaaoaat .rrg/
TD:
Justin D. Weisz, Michael Muller, Jessica He, and Stephanie Houde. Toward general design principles for generative ai applications, 2023.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences text-to- image synthesis, 2023a.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference, 2023b.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding. Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023a.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works, 2017.
Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEF/CVF Interna- tional Conference on Computer Vision, pp. 7754-7765, 2023b.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks, 2017.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adver- sarial networks for text-to-image synthesis, 2019.
12

---

Published as a conference paper at ICLR 2024
A PICSART IMAGE-SOCIAL DATASET DETAILS
A.1 HUMAN PREFERENCE DATATSETS' PROMPT ANALYSIS
To better illustrate differences between training sets of existing scoring models, we used t-SNE van der Maaten Hinton 2008) method on the embeddings presented in t.3.TTJ map the prompts to dimensions. Results in Figure 8 confirm Picsart Image-Social's distinctness from other sets.
Iragelen's
-10
Figure t-SNE visualization training prompts for different scoring models
A.2 DATA COLLECTION PROCEDURE
We initiate the data collection process by identifying the most popular synthetic images Picsart, focusing on those with the highest number of remixes over a 10-month period. These images are then arranged based remix count descending order. The top 1% of these images are categorized as positive images, utilizing the "content signal" approach. Furthermore, if an image has been remixed by an influencer user, is also included the set of positive images, employing the "creator signal".
Subsequently, we gather prompts associated with the selected positive images. From these prompts, we retrieve all images that have garnered at least one remix and analyze the distribution of view counts before the first remix. By cutting this distribution at the 99th percentile, we establish view count threshold for subsequent steps.
For negative images, we identify all images, linked to the same prompts, that have received zero remixes (there are large amounts this kind images due to "Pareto-like" nature of content dif- fusion in social platforms). For each prompt, we filter out "zero-remix" images falling below the established view count threshold. This process ensures that only images with sufficient evidence of negative user feedback labeled such.
Throughout the data collection procedure, all cases which cause inconsistencies are dropped. For instance, omit prompts lacking images other than those labeled as positive prompts with images receiving zero remixes, etc.
To further refine the dataset, we employ in-house mature content detection models to filter out im- ages classified within the NSFW category. This comprehensive approach enhances the reliability and accuracy of our collected data.
13

---

Published as a conference paper at ICLR 2024
A.3 DATASET IMPORTANT CHARACTERISTICS
Out of about 1.7 million images popular ones (deemed as positive) constitute 8%. The main reason behind the smaller number of positive images in Picsart Image-Social dataset typical "Pareto like" pattern observed in majority social networks where minority of the content gathers majority of community engagement.
"Creator signal" that had been leveraged in collecting positive samples for Picsart Image-Social dataset presented by 174 influencer users. Users are categorized influential Picsart if they have a certain amount of followers, are frequently posting new content, which gathers large volumes of community remixes.
Engagement levels of individual users, in our case, expectedly follow power law distribution logic (pattern which usually describes user activity in online social platforms). For instance, the least active 50% of our users generate about 30% of remixes. Whereas the most active 10% generate about 40% remixes.
B LOSS EXPERIMENTS
We investigated a range loss functions, such as the Binary Cross Entropy, Reweighted Cross- Entropy Loss /Wang et al. (02I), and metric losses such InfoNCE (van den Oord et al. 2019), Contrastive (Khosla et al. 2021) and Triplet (Schroff et al., 2015). The best results were obtained using the Triplet loss. Performance comparison of the model trained under different loss functions presented in the Table 7
Table 7: Different Losses and their Accuracies
Loss
Accuracy
Triplet InfoNCE
69.7%
68.5%
Contrastive
67.1%
BCE
67%
Reweighted Cross-Entrope
65.9%
14

---

Published as a conference paper at ICLR 2024
C
ADDITIONAL VISUALIZATIONS
(a) male dragonborn with green skin and gold eyes
(b) blue wig
(c) coffee lette,
(d) frame
(e) pencil dress vintage, pearts, ...
(f polish amazing landscape,
Figure 9: Generated images ranked by Social Reward from best (left) worst (right)
15

---

Published as a conference paper at ICLR 2024
Stahen Pllss
(a) art by stephen bliss
(b) if unicorns were real
(c) Ninetales, Pokemon, detailed
(d) salvador dali andy warhol
Sears Chest Cliers
(c) distant dream, digital art
(f) the seas of cheese, award winning photo
(g) cherry in a bottle, cubism, painted by dali (h) scooby doo ( 1969 background paintings
( sri lankan landscape, painting by david (i) hawk headed warlock, wind magic full paynter
body character design.
Figure 10: Generated image comparison between Baseline (left) and Fine-tuned (right) models using prompts from DiffusionDB
16


Reviews:
[{"review_id": "zEQDO23tRn", "reviewer": "Reviewer_hCCD", "summary": "This paper presents a groundbreaking approach by introducing the concept of utilizing social rewards within a reward modeling framework to assess and enhance generative AI systems. The authors collected data from Platform A and employed the frequency with which an image is reused for editing by other users as a performance metric. Notably, they have constructed a colossal dataset, known as the \"PA Image-Social dataset\", which collects feedback on the relationships between images and prompts, offering a valuable resource for the analysis of positive and unpopular image-prompts associations.\n\nThe study also includes a comprehensive evaluation of the Social Reward model, demonstrating its superiority over existing models in capturing community-level creative preferences. Furthermore, the authors conducted an evaluation showcasing the effectiveness of fine-tuning AI models with Social Reward, revealing performance improvements across multiple performance metrics. This innovative paper brings forth a promising avenue for the assessment and enhancement of generative AI, backed by meticulous research and empirical evidence.", "questions": "I am curious about the data distribution within the collected dataset. It would be highly informative if the authors could provide insights into key aspects such as the percentage of popular images, the presence of influential users, and the overall engagement levels of individual users.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Thanks for your interests in ICLR! Overall, this is an interesting paper on a topic which is of interest to ICLR Conference. It introduces an innovative set of metrics designed to gauge the intricate interplay between prompts and images, accompanied by the construction of an unprecedented million-user-scale dataset, offering invaluable insights into human preferences for user-generated visual art. The metrics, composed of five distinct factors, provide a robust framework for evaluating performance.\n\nAdditionally, the authors furnish a robust evaluation of the Social Reward model, including comprehensive comparisons with existing models and metrics. Their meticulous analysis strengthens the paper's standing in the field and underscores its potential to advance our understanding of generative AI.", "weaknesses": "The paper offers a comprehensive introduction to the metrics and evaluation methods; however, I am very curious about the PA Image-Social dataset and want to learn more about its details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GeHUoyVzvm", "reviewer": "Reviewer_yu8R", "summary": "This paper advocates the importance of social rewards as a motivating factor for user engagement and content contribution on online platforms. It focuses on the co-creative process in text-conditioned image synthesis within online social networks, emphasizing the challenges of assessing models in the context of collective community preferences. The paper introduces a novel framework called \"Social Reward\" that leverages implicit feedback from users engaged in creative editing of generated images. This approach is supported by an extensive dataset and demonstrates improved alignment with social popularity compared to existing metrics.", "questions": "Please check the weakness part. The first three points are more crucial for me.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The main strengths and innovations of this paper are:\n\n(1)\tThe introduction of Social Reward, a novel reward modeling framework that leverages implicit feedback from social network users engaged in creative editing of generated images, to assess community appreciation of AI-generated artworks. To put their contributions in context, the authors discussed recent endeavors that propose modeling human preference by learning scoring functions from datasets consisting of human-annotated prompts paired with synthetic images.\n\n(2)\tThe curation of a million-user-scale dataset of implicit human preferences for user-generated visuals by drawing from an anonymous online visual creation and editing platform, perpetually invigorated by a vibrant user community, where the feedback is drawn from individuals who actively engage with the images on the platform for editing purposes. It is used to demonstrate the distinctiveness of the Social Reward in comparison to those employed by existing solutions and to highlight the limitations of these solutions. \n\n(3)\tThe paper contributed to the field of T2I synthesis by introducing a new paradigm for evaluating the quality of AI-generated artworks that is more closely aligned with users' creative goals.  The rigorous quantitative experiments and user studies that show that the Social Reward model aligns better with social popularity than existing metrics. Social Reward can be used to fine-tune generative models to better align with creative preferences of the community.\n\nOverall, this paper represents a nice contribution to the field of generative AI art and has the potential to improve the quality of AI-generated artworks by better aligning them with users' creative goals.", "weaknesses": "-\tI failed to follow what the authors meant by referring to Pick-a-Pic as “relatively small scale of collected user preferences along with absence of ‘collective feedback’”. Please explain why.\n\n-\tIn Figure 7, it is hard to tell why Social Reward fine-tuning actually improves, as the examples might just be cherry picked and observing them doesn’t give me the intuition what Social Reward essentially improves or corrects any pattern consistently. Table 6 numbers are also marginally close to each other. Could the authors elaborate some critical win cases or “take home points”, that can be consistently observed after using Social Reward fine-tuning? \n\n-\tThe study relies entirely on self-report survey measures. Self-report can be subject to biases like social desirability bias where participants answer in a way they feel is more socially acceptable rather than reflecting their true thoughts and behaviors. How the authors think their data curation was/was not affected by the social bias?\n\n-\t(Minor) For robust user study, as a cross-sectional study collecting data at only one time point, it cannot determine causation or the direction of effects. A longitudinal design collecting multiple waves of data over time would allow for stronger conclusions about how the variables influence each other over time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SI47JuBDxp", "reviewer": "Reviewer_o2eb", "summary": "This paper proposes a novel reward modeling framework, called Social Reward, for assessing community appreciation of AI-generated artworks. The framework leverages implicit feedback from social network users engaged in creative editing of generated images, rather than relying on limited size user studies guided by image quality and alignment with prompts. \n\nThe authors curate a million-user-scale dataset of implicit human preferences for user-generated visuals by drawing from an anonymous online visual creation and editing platform. Rigorous quantitative experiments and user studies show that the Social Reward model aligns better with social popularity than existing metrics.", "questions": "How was user data right and privacy protected in your data curation process? That was not mentioned anywhere in paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper starts with an interesting insight. Social reward is a form of community recognition that provides a strong source of motivation for users of online platforms to actively engage and contribute with content to accumulate peers' approval. Positive social feedback are essential for maintaining social cohesion and individual well-being. On online social platforms, users seek satisfaction via accumulation of their network’s peers engagement with shared content in the form of likes, views, etc. Therefore, social reward can motivate users to engage with online platforms by providing them with a sense of validation and recognition from their peers.\n\nThe authors’ analysis exposes the shortcomings of current metrics in modeling community creative preference of text-to-image models’ outputs. Many quantitative experiments and user study show that Social Reward model aligns better with social popularity than existing metrics.\n\nThe dataset curation process for the million-user-scale dataset of implicit human preferences for user-generated visuals involved drawing from an anonymous online visual creation and editing platform. The dataset was curated by leveraging collective feedback, which implies multiple users’ editing choices for the given content item, as a cleaning mechanism of organic implicit user behavior. A number of other data collection techniques had been utilized for addressing such biases as caption bias, content exposure time, and user follower base biases. Each positive and negative instance within the PA Image-Social dataset is a product of collective, independent, implicit voting by user community. I think this dataset would be valuable for many follow-up analyses.", "weaknesses": "As one limitation, this paper does not aim to introduce a specific new RLHF algorithm as a primary focus. Instead, its main emphasis lies on exploring a novel aspect of data-centric reward modeling, which has previously received little attention. The paper introduces a comprehensive end-to-end solution, and it's important to acknowledge that each component of this solution builds upon prior ideas and engineering techniques. However, it's worth noting that the integration of these components into a cohesive whole represents a significant and commendable achievement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "olGytzPfpI", "reviewer": "Reviewer_Poto", "summary": "This paper proposes to create a new dataset of implicit human creative preferences for user-generated visual art. The dataset is created based on an online visual creation and editing platform and millions of user feedback from the online social community are collected to evaluate the quality of generated images. Due to the uniqueness of this dataset in modeling content creativity, the authors also proposed a new social reward metric to evaluate image quality. The social reward metric is further used to fine-turn the text-to-image generation model.", "questions": "See the Weaknesses for questions.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tPrevious text-to-image generation human preference datasets mainly focus on general fidelity and text-image alignment, ignoring content creativity. This paper created a new dataset to fill this gap. \n2.\tTo evaluate the creativity of the image, the authors utilize the number of times an image has been reused for editing purposes by other users, which is reasonable and more aligned with real users. The number of collected feedbacks is also in the million scale.\n3.\tThe authors also demonstrate the proposed social reward metric's effectiveness in fine-turning the text-to-image generation model.", "weaknesses": "1.\tIn the proposed social reward metric, the authors mainly focus on the creation-related metric such as the number of times an image has been reused for editing purposes by other users. However, user comments, views or likes are also important dimensions. It’s better to show whether a higher value of the proposed metric in this paper will induce a lower or higher value of the traditional metrics.\n2.\tThe social reward metric in this paper is optimized using the triplet loss. As a dataset and benchmark paper, it’s recommended to compare with other existing contrastive learning methods such as NCE or infoNCE.\n3.\tThe constructed dataset consists of prompt, positive image, and negative image. It’s unclear what’s the specific threshold to decide the positive and negative image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)