Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
Authors: Yuval Kirstain
Adam Polyak
Uriel Singer
Shahbuland Matiana
Joe Penna
Omer Levy
Venue: NeurIPS 2023 poster (2023)
Keywords: text-to-image
human-preferences
dataset
TL;DR: We introduce Pick-a-Pic, a large, open dataset of text-to-image prompts and real users’ preferences over generated images.

Abstract: The ability to collect a large dataset of human preferences from text-to-image users is usually limited to companies, making such datasets inaccessible to the public. To address this issue, we create a web app that enables text-to-image users to generate images and specify their preferences. Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image prompts and real users’ preferences over generated images. We leverage this dataset to train a CLIP-based scoring function, PickScore, which exhibits superhuman performance on the task of predicting human preferences. Then, we test PickScore’s ability to perform model evaluation and observe that it correlates better with human rankings than other automatic evaluation metrics. Therefore, we recommend using PickScore for evaluating future text-to-image generation models, and using Pick-a-Pic prompts as a more relevant dataset than MS-COCO. Finally, we demonstrate how PickScore can enhance existing text-to-image models via ranking.

Pick-a-Pic: An Open Dataset of User Preferences for Text-to-lmage Generation
Yuval Kirstain
Adam Polyak"
Uriel Singer
Shahbuland Matiana"
Joe Penna"
Omer Levy"
Tel Aviv University
Stability AI
kirtt.kiss tau. ac.
Abstract
The ability to collect large dataset of human preferences from text-to-image users is usually limited to companies, making such datasets inaccessible to the public. To address this issue, create web app that enables rext-to-image users to generate images and specify their preferences. Using this web app we build Pick-a-Pic, large, open dataset of text-to-image prompts and real users' preferences over generated images. We leverage this dataset to train a CLIP-based scoring function, PickScore, which exhibits superhuman performance of predicting human preferences. Then, we test PickScore's ability to perform model evaluation and observe that it correlates better with human rankings than other automatic evaluation metrics. Therefore, we recommend using PickScore for evaluating future text-to-image generation models, and using Pick-a-Pic prompts as a more relevant dataset than MIS-OOCO Finally, we demonstrate how PickScore can enhance existing rext-to-image models via ranking
the task
"jedi duck holding a lightsaber"
"Two-faced biomechanical cyborg..."
"A bird with 8 spider legs"
"a green owl made of fimo"
"insancly detailed portrait, wise man"
"A butterfly flying above occan"
Figure 1: Images generated via our web application, showing darkened non-preferred images (left) and preferred images (right).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
(win rate of Recent advances in aligning language models with user behaviors and expectations have placed significant emphasis on the ability to model user preferences [0] However, little attention has been paid to this ability in the realm of text-to-image generation. This lack of attention can largely be attributed the absence of a large and open dataset of human preferences over state-of-the-ar image generation models.
To fill this void, we create a web application that enables users to generate images using state-of-the- art text-to-inn text-to-image models while specifying their preferences. With explicit consent from the users, we collect their prompts and preferences to create Pick-a-Pic, a publicly available dataset comprising over half-a-million examples of human preferences over -generrtd images Each example in our dataset includes a prompt, two generated images, and a label indicating the preferred image, or a tie when no image is significantly preferred over the other. Notably, Pick-a-Pic was created by real users with a genuine interest generating images. This interest differs from that of crowd workers who lack the intrinsic motivation to produce creative prompts or the original intent of the prompt's author to judge which image better aligns with their needs.
Tapping into authentic user preferences allows us to train a scoring function that estimates the user's satisfaction from a particular generated image given a prompt. To train such a scoring function we finetune CLIP-H [2 7 InstructGPT's reward model [1]]. This objective aims to maximize the probability of a preferred image being picked over an unpreferred one, or even the probability in cases of a tic. We find that the resulting scoring function, PickScore achieves superhuman performance in the task of predicting user preferences (a 70.5% accuracy rate, compared to humans' 68.0%), while zero-shot CLIP-H (60.8%) and the popular aestheties predictor [4] (56.8%) perform closer chance (56.8%).
using human preference data and an analogous objective to that of
Equipped with a dataset for human preferences and a state-of-the-an scoring function, we propose updating the standard protocol for evaluating text-to-image generation models. First, we suggest that researchers evaluate their text-to-image models using prompts from Pick-a-Pic, which better represent what humans want generate than mundane captions, such as those found in MS-COCO 22 9]. Second, to compare PickScore with FID, we conduct human evaluation study and find that even when evaluated against MS-COCO captions, PickScore exhibits a strong correlation with human preferences (0.917), while ranking with FII) yields a negative correlation (-0.900). Importantly, we also compare PickScore with other evaluation metries using model rankings inferred from real user preferences. We observe that PickScore is more strongly correlated with ground truth rankings, as determined by real users, than other evaluation metrics. Thus, we recommend using PickScore as a more reliable evaluation metric than existing ones.
Finally, we explore how PickScore can improve the quality of vanilla text-to-image models via ranking. To accomplish this, we generate images with different initial random noises as well as different templates (e.g. "breathtaking [prompt]. award-winning, professional, highly detailed")) to slightly alter the user prompt. We then test the impact of selecting the top image according to different scoring functions. Our findings indicate that human raters prefer images selected by PickScore more than those selected by CLIP-H (7) (win rate of 71.3%), an aesthetics predictor [14 85.1 1)), and the vanilla eext-to-image model (win rate of 71.4%).
In summary, the presented work addresses gap in the field of text-to-image generation by creating a large, open, high-quality dataset of human preferences over user-prompted model-generated images. We demonstrate the potential of this dataset by training a scoring function, PickScore, which exhibits a performance superior to any other publicly-availabl automatic scoring function, in predicting human preferences, evaluating text-to-image models, and improving them via ranking. We encourage the research community to adopt Pick-a-Pic and PickScore as a basis for further advances text-to-image modeling and incorporating human preferences into the learning process.
We recently open-sourced a new version of the Pick-a-Pic dataset that has more than one million examples. The model is available at athttpss//huggiine htwp:///wwwwwaaaaaaa//a/aaaooo/. http:///////gggoooooooooooooaaaa

---

image la sigeificantly better than the other
Image is digntiiiantly better than the other
imago aigrificantly better than other
@8)
(b)
(c)
Figure 2: How Pick-a-Pic data collected through the app: (a) the user first writes a caption, and receives two images; (b) the user makes preference judgment; (c) new image is presented instead of the rejected image. This flow repeats itself until the user changes the prompt.
2 Pick-a-Pic Dataset
The Pick-a-Pic datase T was created by logging user interactions with the Pick-a-Pic web application for text-to-image generation. Overall, the Pick-a-Pic dataset contains over 500,000 examples and 35,000 distinct prompts. Each example contains a prompt, two generated images, and a label for which image is preferred, or if there is tie when image is significantly preferred over the other. The images in the dataset were generated by employing multiple backbone models, namely, Stable Diffusion 2.1, Dreamlike Photoreal 2.0* and Stable Diffusion XL variants [ while sampling different classifier-ff free guidance scale values [6]. As we continue with our efforts to collect more user interactions through the Pick-a-Pic web app and decrease the number of NSFW examples included in the dataset, we will periodically upload new revisions of the dataset.
The Pick-a-Pic Web App To ensure maximum accessibility for wide range of users, the user interface was designed with simplicity in mind. The application allows users to write creative prompts and generate images. At each turn, the user is presented with two generated images (conditioned on their prompt), and asked to select their preferred option or indicate a tie if they have no strong preference. Upon selection, the rejected (non-preferred) image is replaced with newly generated image, and the process repeats. The user can also clear or edit the prompt at any time, and the app will generate new images appropriately. Figure 2 iillustrates the usage flow.
Real Data from Real Users A key advantage of Pick-a-Pic is that our data is collected from real, aarrinicaallymmtivatee users, rather than paid crowd workers. We achieve this by approaching a wide audience through various social media channels such as Twitter, Facebook. Discord, and Reddit. At the same time, we mitigate the risk of collecting low-quality data resulting from potential misuse of the application by implementing several quality control measures, First, users are required to authenticate their identity using either Gmail Discord account Second, we closely monitor user activity logs and take action to ban users who generate NSFW content, use multiple instances of the web app simultaneously, or make judgments at an unreasonably fast pace. Third, we use list of NSFW phrases to prevent users from generating harmful content. Last, we limit users to 1000 interactions and periodically increase the limit. These measures work tandem to ensure the integrity and reliability of Pick-a-Pic's data.
Annotation Methodology While piloting the web app, we experimented with different annotation strategies to optimize for data quality, efficiency of collection, and user experience. Specifically, we tested the following annotation options: (1) images, no ties; (2) 2 images, no ties; (3) 2 images,
'The dataset available at https///..//e////nnta//////////a its updated version is available at htttts::: http::::///uuggggafaeeeee co/
and
'Dreamlike Photoreal 2.0 a finetuned version Stable Diffusion 1.5
AIll users provide explicit consent to share their interactions with Pick-a-Pic's application as part of a public dataset. To ensure privacy, anonymize user IDs.
3

---

33 53.5%
Phatereal
11.22%
2%
Coandeerrrs Guddane Stda
(a) Win rate versus classifier-free guidance scale for Stable Diffusion XL (Alpha).
(b) Preference distribution when comparing Sta-
ble Diffusion 2.1 with Dreamlike Photoreal 2.0.
Figure 3: The Pick-a-Pic dataset enables us to perform model selection (a), and model evaluation (b).
with ties. We found that the latter option images, with ties) exceeds the other two in terms of user engagement and inter-rater agreement.
Preprocessing When processing the collected interactions, we filter prompts with NSFW phrases and banned users. We acknowledge that there are still NSFW images and prompts, and will periodi- cally attempt to update the dataset and reduce such occurrences. To divide the dataset into training, validation, and testing subsets, we first sample one thousand prompts, ensuring that each prompt was created by a unique user. Next, we randomly divide those prompts into two sets of equal size to create the validation and test sets. We then sample exactly one example for each prompt to include in these sets, For the training set, we include all examples that do not share prompt with the validation and test sets. This approach ensures that split shares prompts with another split, and the validation and test sets do not suffer from being non con-proportionally fitted to a specific prompt or user.
Statisties Since the creation of the Pick-a-Pic web app we have gathered 968,965 rankings which originated from 66,798 prompts and 6,394 users. However, as the Pick-a-Pic dataset is constantly updating, the reported experiments in this paper involve an NSFW filtered and not fully updated version of Pick-a-Pic that contains 583,747 training examples, and 500 validation and test examples. The training set of this dataset contains 37,523 prompts from 4,375 distinct users.
Model Selection and Evaluation The Pick-a-Pic dataset offers unique opportunity for model selection and evaluation methodology, leveraging users' preferences for unbiased analysis. To illustrate this opportunity, free guidance scale of Stable Diffusion XL (Alpha variant) on its performance. Specifically, we compare human preferences made when both images were generated by Stable Diffusion XL (Alpha variant) but using different classifier-fres guidance scales"] For each scale, we compute the win ratio, representing the percentage of judgments where its use led to preferred image. We also calculate the corresponding tie and lose ratios for each scale, enabling a detailed analysis of which classifier guidance scales more effective. Our results depicted Figure (a), and verify for example, that a guidance scale of 9 usually yields preferred images when compared to a guidance scale of 3.
the collected data and analyze the impact of changing the classifier-
Furthermore, by examining user preferences between images generated by different backbone models, we can determine which model is preferred more by users. For instance, considering judgments in which one image was generated by Dreamlike Photoreal 2.0 and the other by Stable Diffusion 2. we can evaluate which model is more performant. As shown in fig. (b),, users usually prefer Dreamlike Photoreal 2.0 over Stable Diffusion 2.1. We encourage researchers to contact us and include their text-to-image models in the Pick-a-Pic web app for the purpose of model selection and evaluation.
3 PickScore
One valuable outcome from collecting a large, natural dataset of user preferences is that we can use it to train function that scores the quality of generated image given prompt. We train the PickScore scoring function over Pick-a-Pic by combining a CLIP-style model with a variant of
The frequency of different guidance scales is similar.

---

InstructGPT's reward model objective [T0] PickScore is able to predict user preferences in held-out Pick-a-Pic prompts better than any other mublicly-availlbb scoring function, surpassing even expert human annotators (Section 4). Such a scoring function can be of value for various scenarios, such as performing model evaluation (Section 5). increasing the quality of generated images via ranking (Section 66. building better large-scale datasets to improve text-to-image models [14] and improving text-to-image models through weak supervision (e.g. RLHF).
Model PickScore follows the architecture of CLIP 12 given a prompt x and an image y, our scoring function s computes a real number by representing r using transformer text encoder and using a transformer image encoder as d-dimensional vectors, and returning their inner product:
s(x, y) E Eang (y) T
(1)
Where T' is the learned scalar temperature parameter of CLIP.
Objective The input for objective includes scoring function prompt two images y1, y2, and a preference distribution vector which captures the user's preference over the two images. Specifically, p takes value of [1, 0] if yı is preferred, [0, 1] if y/2 is preferred, or [0.5, 0.5] for ties. Given this input, the objective optimizes the scoring function's 's parameters by minimizing the KL-divergence between the preference p and the softmax-normalized scores of 1/1 and yz:
exp s(x, y:)
Pi
(2)
ES=1 exp exp Mj}
2
prel
pj (log P: - log p.)
Since many examples can originate from the same prompt, we mitigate the risk of overfitting to a small set of prompts by applying weighted average when reducing the loss across examples in the batch. Specifically, we weigh each example in the batch, with an inverse proportion to its prompt frequency in the dataset. This objective is analogous InstructGPT's reward model objective [IO]. We also experiment with incorporating in-batch negatives into the objective, but find that this yields less accurate scoring function (see Appendix).
Training We finetune CLIP-H 7 using our framework on the Pick-a-Pic training set. We train the model for 4,000 steps, with a learning rate of 3e-6, a total batch size of 128, and a warmup period of 500 steps, which follows linearly decaying learning rate; the experiment is completed in less than an hour with 8 A100 GPUs. We did not perform hyperparameter search, which might further improve results. For model selection, we evaluate the model's accuracy on the validation set (without the option for tie) in intervals of 100 steps, and keep the best best-performing checkpoint.
4 Preference Prediction
We first evaluate PickScore on the task it was trained to do: predict human preferences. We find that PickScore outperforms all other baselines, including expert human annotators.
Metric To evaluate the ability of models to predict human preferences, we use adapted accuracy metric that accounts for the possibility of tie. Our metric assigns one point to the model for predicting the same label the user, half point if either label prediction is tie (but not both), and zero otherwise.
Tie Threshold Selection Utilizing the notation specified in Section 3 each model requires tie threshold probability to predict a tied outcome when 10 Pel < t. To achieve this, we evaluate each model on the validation set using various tie threshold probabilities and subsequently determine the most optimal tie threshold for each model. For human experts, we do not perform tie selection and explicitly allow them to select a tie.
5

---

Table Quantitative results on Pick-a a-Pi
CLIP-H
PickScore
CLIP-H
PickScore great ape...
"Highway hell"
FabScoe
CLESH
Asalhdiic
0.15
0.40
Robott
beautiful pir field of flowers"
0.50
P3
04
(a) Accuracy across different tie thresholds on the Pick-a-Pic validation set
poster"
"A deman exiting through partal..."
Model
Accuracy
Random
56.8
Human Expert
68.0
Aesthetics 14
56.8
CLIP-H
60.8
Figure 4:
Disagreement between CLIP-H (left) and
ImageReward [8]
61.1
PickScore (right) the Pick-a-Pic validation set. We add green borders around images that humans preferred.
HPS 117
66.7
PickScore (Ours)
70.5
(b) Performance on the Pick-a-Pat Pic test set.
Baselines We compare our model with CLIP-H 7 an aesthetics predictor 14 built on top of
CLIP-L [12. random classifier, and human experts 9 with models from concurrent work, namely, HPS and ImageReward [8]
For completeness, we also compare our results
Results First, compare the models' performance the validation set thresholds. Figure I shows that PickScore outperforms the baselines across almost all thresholds, and achieves the highest global score by wide margin. After selecting the best-performing tie threshold for each model, we use the threshold to evaluate the different models on the test set.
different tie
Table 1b shows that the aesthetics score (56.8) and CLIP-H (60.8) perform closer to random chance baseline (56.8), while PickScore (70.5 0.142 1O achieves superhuman performance, as even outperforms human experts (68.0). It is important to emphasize core difference between real users that produce the ground truth labels and annotators used to evaluate human performance. The users that produce the ground truth are actual text-to-image users, which have idea (which may be incomplete) for an image, and invent a prompt (which may lack details) with hope that the resulting image will match their preferences. In contrast, annotators that used to measure human performance are oblivious to the original user's context, idea, and motivation. Therefore, this task means that the model is able outperform human annotator
superhuman performance that is oblivious to the original user's context, idea, and motivation. The superhuman performance of PickScore showcases the importance of using real users ground truth rather than expert annotators when collecting human preferences. Moreover, the relatively modest human performance (68.0) on the task, when compared to random baseline (56.8), shows that predicting human preferences in text-to-image generation is difficult task for human annotators.
For completeness, we also include the results of concurrent work from HPS [17], which scores 66.7, and IewaeReward [18. which scores 61.1; PickScore outperforms both. To further illustrate the differences between CLIP-H and PickScore,
showcase examples of disagreement from the Pick-a-Pic validation set in Figure We notice that PickScore often chooses more aesthetically pleasing images than CLIP--; times, at the cost of faithfulness the prompt.
Throughout this paper, we use friends and colleagues of the authors expert human annotation. Je train PickScore using three different random seeds and report the mean and variance.

---

Stable Diffusion 2.1
Dreamlike Photoreal 2.0
Spearman R= -0.900
CFG
CFG
CFG
CFG 9
Model Win fardos
A biby doer and lorge diser are standing
"The end matorcyrle that disnday
laving on leather hammack
orange Tegnn lugaree
52 02
04
Human Experts
Spearman R=0.917
Model Winh Rabios
PickScore
Human Experts
Figure 5: Images generated using the same seed and model, but using different classifier-free guidance (CFG) scales. Even though high guidance scales lead worse FID, hu- mans usually find them more pleasing than low guidance scales.
Figure 6: Correlation between the win ratio of different models according to FID and PickScore to human experts on the MS-COCO validation set.
Spearman 0.313
Spearman R-0.492
Spearman
Spearman R-D.
Midel letings
Model sutings
Model Hatlings
CLP-H
PIcKS
User's
Users
Userr
Users
Figure 7: Correlation between Elo ratings of real users and Elo ratings by CLIP-H, ImageReward [8]
HPS 117,
and PickScore.
Model Evaluation
Despite significant progress in the generative capabilities of text-to-image models, the standard and most popular prompt dataset has remained the Microsoft Common Objects in Context dataset (MS-COCO) [9. Similarly, the Fréchet Inception Distance (FID) (5) is still the main metric used for model evaluation. In this section, we explain why we recommend researchers to evaluate their models using prompts from Pick-a-Pic rather than (or least alongside) COCO, and show that when evaluating sate-off-the-ar text-to-image models, PickScore is more aligned with human judgments than other automatic evaluation metrics.
Model Evaluation Prompts The prompts contained in the MS-COCO dataset are captions of photographs taken by amateur photographers, depicting objects and humans in everyday settings. While certain captions within this dataset may pose challenge to text-to-image models, it is evident that the scope of interest for text-to-image users extends beyond commonplace objects and humans. Moreover, the main use-case of image generation is arguably to generate fiction, which cannot be captured by camera. By construction, Pick-a-Pic's prompts are sampled from real users, and thus better represent the natural distribution of text-to-image intents. We therefore strongly advocate that the research community employ prompts from Pick-a-Po Pic when assessing the performance of text-to-image models.
FID The FID metric gauges the degree of resemblance between set of generated images and set of authentic images, at the set level. To do so, it first embeds the real and generated images into
7

---

the feature space of an Inception net [15]. and then estimates the mean and covariance of both sets of images and calculates their similarity. FID thus geared towards measuring the realism of a set of images, but is oblivious to the prompts. In contrast, PickScore provides a per-instance score, and is directly conditioned on the prompt. We thus hypothesize that PickScore will correlate better with human judgements of generation quality.
To empirically test our hypothesis with the most "convenient" settings for the FID metric, we select 100 random captions from MS-COCO validation split. For each caption, we generate images from 9 different models based on the same set of prompts, and ask human experts to rank the 9 generated images (with ties), inducing pairwise preferences. Specifically, we use Stable Diffusion 1.5, Stable Diffusion 2.1, and Dreamlike Photoreal 2.0 combined with three different classifier-free guidance scales (3. 6, and 9). We then repeat the labeling process (over the same images) using FID, and PickScore instead of humans to determine preferences. Since FID does not operate on a per-example base, when "labeling" with FID, we simply choose the model that has a lower (hetter) FID score on MS-COCO.
Figure shows the correlation between model win rates induced by human rankings (horizontal) and model win rates induced by each automatic scoring function. PickScore exhibits stronger correlation (0.917) with human raters on MS-COCO captions than FID (-0.900), which surprisingly, exhibits a strong negative correlation. As FID is oblivious to the prompt, one would expect zero correlation, and not a strong negative correlation. We hypothesize that this is related to the glassifier-free guidance scale hyperparameter larger scales tend to produce more vivid images (which humans typically prefer), but differ from the distribution of ground truth images in MS-COCO, yielding worse (higher) FID scores. Figure 5 visualizes these differences by presenting pairs of images generated with the same random seed but with different classifier-free guidance (CFG) scales.
Other Evaluation Metries When comparing with evaluation metrics that do not assume a set of ground truth images, we are able to use a more reliable evaluation procedure. In this evaluation procedure, we consider real user preferences rather than human annotators, as well as more models (i.e. more data points). Specifically, prompts from the Pick-a-Pic test set. This set of examples contains images generated by 45 different models-fout different backbone models, each with different guidance scales. We then use these real user preferences to calculate Elo ratings 44] for the different models. Afterward, we repeat the process while replacing the real user preferences with CLIP-H 77. and PickScore predictions. Similarly to Section[4] [ we also compare against the concurrent work from ImageReward [ and HPS [17]. Since the Elo rating system iterative by nature, we repeat the process 50 times. Each time we randomly shuffle the order of examples, and calculate the correlation with human ratings. Finally, for each metric, we output the mean and standard deviation of its 50 corresponding correlations. Figure 7 displays the correlation between real users' Elo ratings with the different metrics' rating, showing that PickScore exhibits a stronger correlation (0.790 + 0.054) with real users than all other automatic metrics, namely CLIP-H (0.313 + 0.075), ImageReward (0.492 + 0.086), and HPS (0.670 + 0.071).
take all the 14,000 collected preferences that correspond to
6 Text-to-Image Ranking
Another possible application for scoring functions is improving the performance of generations made by text-to-image models through ranking: generate sample of images, and select the one with the highest score. To test this approach, we generate one hundred images for each prompt of the one hundred prompts we take from the Pick-a-Pic test set. We generate the images with Dreamlike Photoreal 2.0, using classifier-free guidance scale of 7.5, and to increase image diversity, we use 5 initial random noises and 20 different prompt templates. These templates include the null template "[prompt]] and other templates like "breathtaking [prompt]. award- winning, professional, highly detailed". From each set of 100 generated images, we select the best one according to PickScore, CLIP-H, the aesthetics score, or randomly; in addition, we randomly select one image from the null template for control. We then ask expert human annotators to compare PickScore's 's chosen image to each of the other functions' choices, and decide which one they prefer.
Table shows that PickScore consistently selects more preferable images than the baselines. By manually analyzing some examples, we find that PickScore typically selects images that are both more aesthetic and better aligned with the prompt. We also measure this explicitly by using the aesthetic scoring function instead of human rater when comparing PickScore's 's choice to CLIP-H's,
8

---

Table 2: Percentage of instances where humans prefer PickScore's choice over another scoring function's ' choice when selecting one imgge out of 100.
Comparison
Win Rate
PickScore vs Random Seed
Null Template
71.4
Random Template
82.0
PickScore vs Aesthetics
14
85.1
PickScore CLIP-H 7
71.3
Vanilla
PickScore
Vanilla
PickScore
Vanilla
PickScore
Vanilla
PickScore
"Highway by to bell"
"A finridic
bee devouring the world"
"Siver fured
OICHELL
SSperman's Brand"
fuffyy senry singing"
carsonaii"
Figure 8: Comparing the image from the vanilla text-to-image model (left) with the image selected by PickScore from a set of 100 generations (right).
and find that for 68.5% of the prompts, PickScore selects an image with a higher aesthetic score. Likewise, when comparing PickScore to the aesthetic scorer, 90.5% of PickScore's 's choices have higher CLIP-H text-alignment score than the images chosen by the aesthetic scorer. Figure visualizes the benefits of selecting images with PickScore.
7
Related Work
Collecting and learning from human preferences is an active area of research in natural language processing (NLP) B TQ] However, in the domain of text-to-imaae generation, related research questions have received little attention. One notable work that focuses collecting human judgments in text-to-image generation is the Simulacra Aesthetic Captions (SAC) dataset [.]. This dataset contains almost 200,000 human ratings of generated images. However, unlike Pick-a-Pic which focuses on general user preferences, and allows users to compare between generated images, the human raters of SAC provide absolute score for the aesthetic quality of the generated images.
There has been some concurrent work that involves collection and learning from human preferences that we describe below. Lee et al. g consider three simple categories of challenges (count, color, and background), enumerating through templates (e.g. "[number] dogs") to synthetically create prompts. Then, they instruct crowd workers to choose if a generated image is good or bad and use the collected 30,000 examples to train a scoring function. In contrast, Tick-a-Pick allows real users to write any prompts they choose, and provide pairwise comparisons between images that yield more than 500,000 examples.
ImwgeReward 18 selects prompts and images from the DiffusionDB dataset [6] and collects for them image preference ratings via crowd workers. Since they employ crowd workers that lack the intrinsic motivation to select images that they prefer, the authors define criteria to assess the quality of generated images and instruct crowd workers to follow these criteria when ranking images. They use this strategy to collect 136,892 examples which originate from 8,878 prompts. Importantly, they have not publicly released this dataset. For completeness, we tested ImageReward on the Pick-a-Pic dataset and confirmed that PickScore outperforms it.

---

Another concurrent work from Wu et al. [7 collects dataset of human judgments by scraping about 25,000 human ratings (which include about 100,000 images) from the Discord channel of Stability AL. Similarly to us, they use an objective analogous to that of InstructGPT to train scoring function that they name Human Preference Score (HPS). As with ImageReward, we evaluated HPS on Pick-a-Pic and saw that PickScore achieves better performance.
The observed superior performance of PickScore over the concurrent work from both HPS and ImageReward on the Pick-a-Pic dataset could be attributed to several factors. For example, differences implementation (e.g.. model size, backbone, hyperparameters)) differences in the scales of data, or variations in the distribution of data. Notably, Pick-a-Pic is more than five times larger than the data used train HPS and IewgeReward. Furthermore, ImageReward collects judgments from crowd workers, which may lead to significant differences in data distribution. In contrast, HPS scrapes ratings from the Stability discord channel for real text-to-image users but may be more aligned with this more specific distribution of text-to-image users. We leave Isolating and identifying the specific effects of these factors for future work.
8 Limitations and Broader Impact
It is important to acknowledge that despite our efforts to ensure data quality (see section 2 some images and prompts may contain NSFW content that could potentially bias the data, and some users may have made judgments without due care. Moreover, the preferences of users may include biases that may be reflected in the collected data. These limitations may affect the overall quality and reliability of the data collected and should be taken into consideration when considering the broader impact of the dataset. Nonetheless, we believe that the advantages of collecting data from intrinsically-monn mottivtedd users and publicly releasing it will enable the text-to-image community to better align text-to-image models with human preferences.
9 Conclusions
We build web application that serves text-to-image users and (willingly) collects their preferences. We use the collected data and present the Pick-a-Pic dataset: an open dataset of over half-a-million examples of text-to-image prompts, generated images, and user-labeled preferences. The quantity and quality of the data enables us to train PickScore, a state-of-ther -he-tarr text-image scoring function, which achieves superhuman performance when predicting user preferences. PickScore aligns better with human judgements than any other publicly-available automatic metric, and together with Pick-a-Pic's natural distribution prompts, enables much more relevant text-to-image model evaluation than existing evaluation standards, such as FID over MS-COCO. Finally, we demonstrate the effectiveness using scoring function for selecting images in improving the quality of text-to-image models. There are still many opportunities for building upon Pick-a-Pic Pic and PickScore, such as RLHF and other alignment approaches, and we excited to see how the research community will utilize this work in the near future.
References
[1] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislay Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Sauray Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/2204.05862. 2022.
[2] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. ArXiv, abs/1504.00325, 2015.
Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017.
[4] Arpad E. Elo. The rating of chessplayers, past and present. 1978.
10

---

[5] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by time-scale update rule converge to local nash equilibrium. In NIPS, 2017.
[6] Jonathan Ho. Classifier-free diffusion guidance. ArXiv, abs/2207, 12598, 2022.
[7] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong. John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. you use this software, please cite it below.
[8] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuging Du, Craig Boutilier, P. Abheel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning texl-to-image models using human feedback. ArXiv, abs/2302. 12192, 2023.
[9] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
[10] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155,, 2022.
[11] John David Pressman, Katherine Crowson, and Simulacra Captions Contributors. Simulacra aesthetic captions. Technical Report Version 1.0, Stability AI, 2022. url https//pithub.com//ID.. P/smmularraii aesthetic-captions.
[12] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
[13] Robin Rombach, Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffuusin models. 2022 IEEE/CVF Conference on Computer Vision and Pattenn Recognition (CVPR), pages 10674-10685, 2021.
[14] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[15] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 28 2818-2826, 2015.
[16] Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang. Benjamin Hoover, Duen Horng Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. ArXiv, abs/2210. 14.96. 2022.
[17] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-image models with human preference. ArXiv, 14.23333.14400 2023.
[18] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. ArXiv. abs/2304.05977, 2023.

---

Appendix
Comparing Pick-a-Pic Prompts with MS-COCO Captions
To illustrate the difference between MS-COCO captions and Pick-a-Pic prompts we show prompts from each dataset.
Pick-a-Pic - "forest with ruins, photo", "A panda bear as a mad scientist", "product photo a sneakers", "photo of bicycle, detailed, 8k uhd, dsir, high quality, film grain, Fujifilm XT3", "female portrait photo", "Alexander the great, cover art, colorful", "A galactic eldritch squid towering over the planet Earth, stars, galaxies and nebulas in the background.." "Portrait of giant, fluffy, ninja teddy bear", "insanely detailed portrait, darth vader, shiny, extremely intricate, high res, 8k, award winning", "Giant ice cream melting and creating river through city".
MS-COCO "A man with a red helmet on a small moped on a dirt road.", "A woman wearing net her head cutting cake.", "there is woman that is cutting white cake", "a little boy wearing headphones and looking at a computer monitor". "A young girl is preparing to blow out her candle.". "A commercial stainless kitchen with pot of food cooking.", "Two that standing kitchen.", "A man riding a bike past a train traveling along tracks.", " "The pantry door of the small kitchen is closed.", "A man is doing a trick skateboard".
Training a Scoring Function
We further explored an alternative loss function that is closer to CLIP's original objective. In this loss function, also incorporate the remaining examples in the batch as in-batch negatives. To elaborate, considering the notation established in section and vf v2 denoting the images corresponding to the k-th example the batch, we formulate p as follows:
exp s(x, 91)
Σ E exp exps(a,
We anticipated that this objective function would maintain the general capabilities of CLIP with minimal loss in performance. However, our findings demonstrated that PickScore significantly outperforms this objective function, the latter only produced a scoring function that achieves accuracy of 65.2 on the Pick-a-Pic test set.
12


Reviews:
[{"review_id": "YSVkfJKsxc", "reviewer": "Reviewer_c5Um", "summary": "This paper contributes a new dataset of human preferences for text-to-image (T2I) generations. It then trains a CLIP-based score function to that can predict human preferences between two generated images. The paper then shows how this can be used to better align a T2I model to human preferences.", "questions": "Please see weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-- Research angle is very timely on better aligning a T2I model to human preferences.\n\n-- The proposed dataset and the learned CLIP-based human image preference function can benefit a diverse cohort of vision researchers.\n\n-- The paper is overall easy to read.", "weaknesses": "While the proposed dataset itself is a significant enough contribution towards acceptance, the paper's technical contribution is relatively trivial and calls for further discussion:\n\n-- the way how the paper leverage a model to learn the human preferences; the paper fine-tunes the entire weights of a pre-trained CLIP and leverages KL loss function to align it to a binary human decision. Given parameter-efficient fine-tuning (PEFT) approach such like LoRA or prompt/adaptor learning has shown exception promise and the fact that the paper only collects a moderate scale of human preferences (less then a million), full fine-tuning might be sub-optimal. The KL loss function choice also needs further ablations, does regression to human preference work?\n\n-- On the held-out validation set, the paper's final results are merely around 70% of the time that align with human preference. Given the random chance is $\\sim$ 55\\%, this is not a particularly good number. The paper needs more extensive discussion to elaborate on this case; is it due to the over noisy human preference ratings that model can not find a globally cohesive way to distil an understanding or the the proposed model is not sophisticated enough to carve out the necessary information for learning?\n\n-- The collected human preference ratings are based on synthetic images. What is their impact towards real images? Can the developed human preference scoring function be used as a plug-n-play model that benefits computer vision tasks with need for better human alignment?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses: \n\n1. While the proposed dataset itself is a significant enough contribution towards acceptance, the paper's technical contribution is relatively trivial and calls for further discussion.\n\nAnswer: While we appreciate the reviewer's note that the dataset itself is a significant enough contribution towards acceptance, we do not agree that the paper's technical contribution is limited. We have acquired state-of-the-art scoring function (PickScore) by applying and adapting best practices from preference modeling in NLP into that of T2I and showed its effectiveness in different scenarios, such as preference prediction, model evaluation, and improving text-to-image models via ranking. We believe that each scenario is important as it can pave the way for future publications, and prove to be valuable to the research community.\n\nQuestions:\n\n1. the paper fine-tunes the entire weights of a pre-trained CLIP and leverages KL loss function to align it to a binary human decision. Given parameter-efficient fine-tuning (PEFT) approach such like LoRA or prompt/adaptor learning has shown exception promise and the fact that the paper only collects a moderate scale of human preferences (less then a million), full fine-tuning might be sub-optimal. \n\nAnswer: LoRA is a great way to reduce memory requirements while training, but we are not familiar with evidence that it should result in better downstream performance - especially given a substantial dataset like Pick-a-Pic that has more than half-a-million examples. Hence, we are not sure how the use of LoRA may contribute to the research questions that we have investigated, and therefore, we did not experiment with it.\n\n2. The KL loss function choice also needs further ablations, does regression to human preference work?\n\nAnswer: Our data consists of relative rankings, and it is therefore very natural to solve this problem with KL loss. Please refer to extensive work of human-preference modelling in NLP e.g. InstructGPT, that we have considered when we made this choice. Specifically, regression is an inadequate function for this learning problem (and therefore, is also not being used in other papers that investigate human-preference modeling over relative preferences). For example, if we have several data points that correspond to the same prompt with images A,B,C,D and human preference that label, A < B = C < D - it is unclear which global value each image should get in the regression problem. \n\n3. On the held-out validation set, the paper's final results are merely around 70% of the time that align with human preference. Given the random chance is 55%, this is not a particularly good number. The paper needs more extensive discussion to elaborate on this case; is it due to the over noisy human preference ratings that model can not find a globally cohesive way to distil an understanding or the the proposed model is not sophisticated enough to carve out the necessary information for learning?\n\nAnswer: We will gladly elaborate more in the paper about this question. Concretely, we show that humans' ability to predict user preferences is *even more limited* than PickScore. Hence, we can conclude (1) PickScore performance can be considered to be quite good, and (2) the difficulty resides (entirely or partially) with the data itself, rather than the model’s learning capacity.\n\n4. The collected human preference ratings are based on synthetic images. What is their impact towards real images? \n\nAnswer: Can you please clarify what do you mean \"What is their impact towards real images?\"? \n\n5. Can the developed human preference scoring function be used as a plug-n-play model that benefits computer vision tasks with need for better human alignment?\n\nAnswer: Do you refer to the ranking experiment in which we have showed the benefits of using PickScore to improve vanilla text-to-image generation? We would appreciate a clarification to correctly answer this question."}, {"review_id": "armzbGU4Hg", "reviewer": "Reviewer_UkC5", "summary": "The study gathers a dataset of user preferences by soliciting human choices regarding more preferred images in response to text prompts given a text prompt. Besides, the authors propose a metric called PickScore by fine-tuning CLIP-H on this dataset. The experiments show that PickScore demonstrates better alignment with human judgment compared to existing evaluation metrics such as FID or HPS.", "questions": "*Dataset and Method\n\nThe concerns regarding dataset biases and the potential biases associated with PickScore. \n\n(1) Considering that PickScore is trained on high-quality images, there is a concern about its ability to accurately evaluate generative models that produce comparatively lower-quality images.\n\n(2) Since the images in the Pick-a-Pic dataset are generated by models such as Stable Diffusion 2.1, there is a possibility that PickScore could exhibit bias towards these models, potentially resulting in higher score values.\n\n*Experiment\n\n(1) The PickScore metric is trained on the proposed Pick-a-Pic dataset, which is annotated by humans. However, it is intriguing to observe that it surpasses human performance in Table 1(b).\n\n(2) Confusion about the results of FID.\n\n1. The reason behind the correlation value of -0.9 between FID and human judgment, which nearly reaches the lower bound of correlation, remains unclear. Although the paper claims the reason may lie in “larger scales tend to produce more vivid images (which humans typically prefer), but differ from the distribution of ground truth images in MS-COCO, yielding worse (higher) FID scores”, why the more vivid image would be totally different from the distribution of GT images on MS-COCO? They are even negatively correlated.\n\n2. Which feature extractor is employed in this FID calculation? Is it an ImageNet pre-trained model? If so, is it suitable for capturing the features of the aesthetic or surreal images on the proposed dataset?\n(3) How about the performance of PickScore on realistic images, it seems most of the images on the proposed dataset belong to aesthetic or surreal ones.\n\n*Minor Issues\n\nLine 173: Figure 1a -> Table 1a", "limitations": "This paper mentions the limitations and broader impact in Section 8: “some images and prompts may contain NSFW content that could potentially bias the data and some users may have made judgments without due care”. It would be better to double- or triple-check each sample on the dataset by humans, as the quality of data would greatly affect its usability and reliability.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In this paper, a novel dataset named Pick-a-Pic is constructed, considering human preferences. This dataset proves to be valuable as it can be used to train an automatic evaluation metric, such as the proposed PickScore, to effectively assess the quality of text-to-image models.", "weaknesses": "1. It is important to acknowledge that the proposed Pick-a-Pic dataset may exhibit certain biases due to the usage of images generated by existing generative models. As a result, the PickScore metric, trained on this dataset, could potentially inherit the same biases and limitations.\n2. Some experimental results are confused, e.g., the correlation value between FID and human judgment is -0.9, which approaches the lower bound of correlation. Does it mean there is a negative correlation between FID and human judgment?\n\nPlease refer to more details in “Question” below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses:\n\n1. Some experimental results are confused, e.g., the correlation value between FID and human judgment is -0.9, which approaches the lower bound of correlation. Does it mean there is a negative correlation between FID and human judgment? \n\nAnswer: Regarding the specific example and question if our results indicate that there is a negative correlation between FID and human judgment - the answer is YES, and we *explicitly* discuss this in the paper (e.g. paragraph in lines 216-224). Moreover, this finding aligns with the fact that the vast majority of the community generates images with high classifier free guidance scale values (e.g. 7) when generating images, which tend to produce much worse FID scores, while abstain from using low classifier free guidance scale values (e.g. 3), that produce better FID scores. It is unclear to us which other experimental results you find to be confusing.\n\nQuestions:\n\n1. Considering that PickScore is trained on high-quality images, there is a concern about its ability to accurately evaluate generative models that produce comparatively lower-quality images.\n\nAnswer: While we agree that PickScore’s ability to evaluate models that produce comparatively lower-quality images has not been thoroughly tested, it is unclear whether it should be a concern. Currently, the standard practice for evaluating new models is via FID against MS-COCO. We show that this protocol is lacking, suggest a new protocol, and back its utility with strong empirical evidence. We hope that our paper will lead the community to suggest improved protocols, use our data collection methodology, dataset, model, evaluation procedure (e.g. use of Elo ratings), code for training the model, etc. as basis for such future work.\n\n2. Since the images in the Pick-a-Pic dataset are generated by models such as Stable Diffusion 2.1, there is a possibility that PickScore could exhibit bias towards these models, potentially resulting in higher score values. \n\nAnswer: We have included different models in the Pick a Pic dataset (e.g. SD2.1, DP2.0, and many SDXL early variants), and randomized classifier free guidance scales - which overall include *dozens* of data points (please see Fig 6, and lines 229-230). We agree that as new models are created it is important to assert that PickScore’s performance remains high. Moreover, we believe that if empirical evidence will show that PickScore requires an “update”, the foundations that we have laid in this paper, will prove to be very valuable.\n\n3. The PickScore metric is trained on the proposed Pick-a-Pic dataset, which is annotated by humans. However, it is intriguing to observe that it surpasses human performance in Table 1(b).\n\nAnswer: We will clarify and elaborate on this question in the paper. There is a core and very important difference between *users* that produce the ground truth and *annotators* used to evaluate human performance. The *users* that produce the ground truth are actual text-to-image users, which have a context, and an idea (which may be incomplete) for an image, and invent a prompt (which may lack details) with hope that the resulting image will match their preferences. In contrast, *annotators* that are used to measure human performance are oblivious to the user’s context, idea, and motivation. Therefore, superhuman performance on this task means that the model is able to outperform a human *annotator* that is oblivious to the original *user’s* context, idea, and motivation.\n\n4. The reason behind the correlation value of -0.9 between FID and human judgment, which nearly reaches the lower bound of correlation, remains unclear. Although the paper claims the reason may lie in “larger scales tend to produce more vivid images (which humans typically prefer), but differ from the distribution of ground truth images in MS-COCO, yielding worse (higher) FID scores”, why the more vivid image would be totally different from the distribution of GT images on MS-COCO? They are even negatively correlated.\n\nAnswer: MS-COCO images were taken by amateur photographers in day-to-day settings. Such images tend to be more ordinary and less vivid than images that humans wish to see when generating images. Our hypothesis is that humans wish to see images taken by professional photographers or artists for example.\n\n5. Which feature extractor is employed in this FID calculation? Is it an ImageNet pre-trained model? If so, is it suitable for capturing the features of the aesthetic or surreal images on the proposed dataset?\n\nAnswer: As we have mentioned in the paper, we have used the *standard practice* for FID calculation (similarly to that used in Make-a-Scene, DALLE2, Imagen, etc.) - specifically and similarly to many other papers, we have used the script from https://github.com/MinfengZhu/DM-GAN/blob/master/eval/FID/fid_score.py . One of the contributions of our paper is advocating that the *standard practice* of conducting FID evaluation against MS-COCO does not align well with text-to-image user preferences, and therefore suggest a new protocol for model evaluation.\n\n6. How about the performance of PickScore on realistic images, it seems most of the images on the proposed dataset belong to aesthetic or surreal ones.\n\nAnswer: One of the benefits of allowing users to specify their prompts, is that it enables us to acquire *relevant* and *diverse* prompts, *many* of which are realistic. Additionally, we also conducted experiments that exclusively use realistic images from MS-COCO and witnessed much better correlation than the standard practice of using FID.\n"}, {"review_id": "yw8hhE3vW8", "reviewer": "Reviewer_UW8i", "summary": "This paper introduces a dataset of text-to-image prompts, along with user preferences over the generated images. This is done by building a web application where multiple images are generated for a given prompt, and the user is asked to select their preferred image. As a result of collecting this Pick-a-Pic dataset, a CLIP model (with a ViT-H backbone) is finetuned on these user preferences to obtain a scoring function PickScore that demonstrates strong results on evaluating text-to-image alignment.  The results indicate that the proposed model PickScore is not only able to significantly outperform standard text-image alignment metrics (i.e CLIPScore), but also performs better than some contemporary work for the same. Additionally, the paper also demonstrates that selecting the best image from a large set of images generated by varying the prompts and seed using PickScore results in images that are preferred over other selection strategies. ", "questions": "My only major question is regarding the \"superhuman\" performance achieved by the PickScore model. Given that the ground truth is captured from human preferences, I do not fully understand what achieving superhuman performance on this benchmark means (and the description in the paper is quite short lines 176-178). A slightly more detailed explanation of this would be very helpful.  ", "limitations": "Yes, I think the limitations section in the paper covers the major issues that might arise from this work (i.e dataset having NSFW images, some noise in cases where the users of the applications make mistakes). Apart from this, the paper also mentions that they have not run a hyperparameter sweep, so it is possible for some future work to train an improved model using the Pick-a-Pic dataset. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall, I think the paper makes a very solid and timely contribution. \n1) Existing methods for evaluating text-to-image alignment have been quite limited, and therefore a large-scale dataset of user preferences that is crowdsourced would be very valuable. \n2) Furthermore, the PickScore model can be useful for a variety of downstream applications since it seems to be atleast on par with humans in evaluating text-image alignment. \n3) The experiments in the paper are also quite thorough and have comparisons to a lot of different models for evaluating text-to-image, including contemporary work. ", "weaknesses": "I believe that the paper tackles a problem that is of interest to sufficient audience, and achieves all/most of the claims made in the paper, therefore I do not have major weaknesses to discuss. Naturally, there could be alternative methods/objectives to train better models for improved performance, different vision-language models (i.e BLIP/BLIP-2) could have been considered, or using PickScore to improve text-to-image generation models could all be done in future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Questions:\n\n1. My only major question is regarding the \"superhuman\" performance achieved by the PickScore model. Given that the ground truth is captured from human preferences, I do not fully understand what achieving superhuman performance on this benchmark means (and the description in the paper is quite short lines 176-178). A slightly more detailed explanation of this would be very helpful.\n\nAnswer: We will clarify and elaborate on this question in the paper. There is a core and very important difference between *users* that produce the ground truth and *annotators* used to evaluate human performance. The *users* that produce the ground truth are actual text-to-image users, which have a context, and an idea (which may be incomplete) for an image, and invent a prompt (which may lack details) with hope that the resulting image will match their preferences. In contrast, *annotators* that are used to measure human performance are oblivious to the user’s context, idea, and motivation. Therefore, superhuman performance on this task means that the model is able to outperform a human *annotator* that is oblivious to the original *user’s* context, idea, and motivation.\n\n"}, {"review_id": "3yUgTOZtqD", "reviewer": "Reviewer_hbT1", "summary": "This paper introduces Pick-a-pic, a large and open dataset of human preference over model-generated images. Using this dataset, a scoring function (PickScore) based on CLIP is trained. The paper finds that PickScore has better correlation with real user preferences compare to existing scoring methods. Furthermore, PickScore can be used as an output selection method to improve the generated image's quality.", "questions": "- Is it possible to ensemble PickScore with other scoring functions as a more comprehensive evaluation / model selection method?", "limitations": "It would be good to see a more detailed analysis on the limitations of the dataset and PickScore.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The proposed dataset fills an important gap for image generation research. Human preference is an important metric and should be considered when evaluating image generation models.\n\n- The dataset is collected from real user, contains a large number examples, and most importantly, is open-sourced.\n\n- The learned PickScore shows high correlation with real user preference. Although expected from training on Pick-a-pic, it still provides a valuable alternative over existing evaluation metrics.\n\n- It is nice to see the potential of PickScore being used to develop better image generation models.", "weaknesses": "- It would be good if the paper could provide a more comprehensive analysis of the prompts used by the users in Pick-a-pic. How diverse are the prompt? What is the word distribution difference between Pick-a-pic and existing datasets such as COCO or DreamBench?\n\n- It is possible that the preference of general public users could be biased towards certain aspects of the generated image such as its interestingness, while neglecting other aspects such as faithfulness to the prompt. It would be good if the authors could provide some analysis on the specific types of biases observed in Pick-a-pic.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses:\n1. It would be good if the paper could provide a more comprehensive analysis of the prompts used by the users in Pick-a-pic. How diverse are the prompt? What is the word distribution difference between Pick-a-pic and existing datasets such as COCO or DreamBench?\n\nAnswer: We agree that helping readers and potential users of the dataset to better understand the distribution of prompts of Pick a Pic when compared to existing benchmarks can be insightful. We will gladly add to the paper examples of randomly sampled examples from Pick a Pic, alongside prompts from existing benchmarks. We will also consider visualizing word distributions. However, we usually find them to be less intuitive and interpretable for readers. Here are randomly sampled examples from Pick a Pic, and MS-COCO (which involved day-to-day scenes, that are probably less relevant for text-to-image users):\n\nPick a Pic -  \"forest with ruins, photo\"\t, “A panda bear as a mad scientist”, \"product photo of a sneakers\", “photo of a bicycle, detailed, 8k uhd, dslr, high quality, film grain, Fujifilm XT3”, “female portrait photo”, “Alexander the great, cover art, colorful”, \"A galactic eldritch squid towering over the planet Earth, stars, galaxies and nebulas in the background…”, \"Portrait of a giant, fluffy, ninja teddy bear”, \"insanely detailed portrait, dartH vader, shiny, extremely intricate, high res, 8k, award winning\", \"Giant ice cream cone melting and creating a river through a city\"\n\nMS-COCO - \"A man with a red helmet on a small moped on a dirt road. \", \"A woman wearing a net on her head cutting a cake. \", \"there is a woman that is cutting a white cake\", \"a little boy wearing headphones and looking at a computer monitor\", \"A young girl is preparing to blow out her candle.”, \"A commercial stainless kitchen with a pot of food cooking. \", \"Two men that are standing in a kitchen.\", \"A man riding a bike past a train traveling along tracks.\", \"The pantry door of the small kitchen is closed.\", \"A man is doing a trick on a skateboard\"\n\n2. It is possible that the preference of general public users could be biased towards certain aspects of the generated image such as its interestingness, while neglecting other aspects such as faithfulness to the prompt. It would be good if the authors could provide some analysis on the specific types of biases observed in Pick-a-pic.\n\nAnswer: We *strongly* agree that interpreting which attributes (like interestingness, artistic flavor, etc.) affect human preferences is an interesting question. However, we are afraid that conducting a comprehensive and informed user study on this topic may require expertise in human–computer interaction. Therefore, we will clarify in the limitations section that while we did not explore it in the context of our work, we hope that the available dataset and other artifacts of this work will enable the community to study these phenomena rigorously. \n\nQuestions:\n1. Is it possible to ensemble PickScore with other scoring functions as a more comprehensive evaluation / model selection method?\n\nAnswer: This is a very interesting question and we will try to answer this question from different perspectives. We believe that the best practice will be to report results of various scoring functions (for example PickScore, CLIPScore, and aesthetic predictor). However, we think that human preference is the most important metric to optimise, and hence, ensembling it with proxies for human preference (e.g. text alignment or aesthetics), may stir the ensembled metric from human preferences. Finally, we hope that the community will follow by collecting and open-sourcing human preference data which will allow researchers to train more robust and updated scoring functions.\n"}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback. We are encouraged that they found our paper to be a very solid (UW8i), timely (UW8i, c5Um), and significant (c5Um) contribution, and our experiments to be thorough (UW8i). We are glad that they consider our open-sourced and large dataset of human (specifically - *real users*)   preference in text-to-image, and learned human-preference scoring function (PickScore) to be very valuable (UW8i, UkC5), benefit a diverse cohort of researchers (c5Um), and prove to be useful for a variety of downstream applications (UW8i, hbT1), as it fills an important gap in image generation research (hbT1). We address each of the reviewers' feedback on weaknesses and questions in the corresponding dedicated rebuttal section.

Meta-review: This paper proposes a human preference dataset for text-to-image generation and provides a reward function called PickScore. 

**strengths**

* Large-human preference dataset for text-to-image generation
* Reward model which can be useful for various downstream tasks
* Solid experimental results

**weaknesses/suggestions**

* Lack of discussions on potential bias in human dataset
* Evaluating the quality of reward model on real images.

Overall, the reviewers and AC agree that this paper makes a worthy contribution to NeurIPS.
However, I encourage the authors to think carefully about how to reflect the comments or resolve the questions from reviewers in the camera ready version.
Decision: Accept (poster)