SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuval Kirstain $^{\tau}$  Adam Polyak $^{\tau}$  Uriel Singer
Shahbuland Matiana $^{\sigma}$  Joe Penna $^{\sigma}$  Omer Levy $^{\tau}$
$\sigma$  Stability AI
yuval.kirstain@cs.tau.ac.il
The ability to collect a large dataset of human preferences from text-to-image users is usually limited to companies, making such datasets inaccessible to the public. To address this issue, we create a web app that enables text-to-image users to generate images and specify their preferences. Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image prompts and real users' preferences over generated images. We leverage this dataset to train a CLIP-based scoring function, PickScore, which exhibits superhuman performance on the task of predicting human preferences. Then, we test PickScore's ability to perform model evaluation and observe that it correlates better with human rankings than other automatic evaluation metrics. Therefore, we recommend using PickScore for evaluating future text-to-image generation models, and using Pick-a-Pic prompts as a more relevant dataset than MS-COCO. Finally, we demonstrate how PickScore can enhance existing text-to-image models via ranking.<sup>1</sup>
"jedi duck holding a lightsaber"
"Two-faced biomechanical cyborg..."
"A bird with 8 spider legs"
"a square green owl made of fimo"
Figure 1: Images generated via our web application, showing darkened non-preferred images (left) and preferred images (right).
<https://github.com/yuvalkirstain/PickScore>
Recent advances in aligning language models with user behaviors and expectations have placed a significant emphasis on the ability to model user preferences [10, 1, 3]. However, little attention has been paid to this ability in the realm of text-to-image generation. This lack of attention can largely be attributed to the absence of a large and open dataset of human preferences over state-of-the-art image generation models.
To fill this void, we create a web application that enables users to generate images using state-of-the-art text-to-image models while specifying their preferences. With explicit consent from the users, we collect their prompts and preferences to create Pick-a-Pic, a publicly available dataset comprising over half-a-million examples of human preferences over model-generated images. Each example in our dataset includes a prompt, two generated images, and a label indicating the preferred image, or a tie when no image is significantly preferred over the other. Notably, Pick-a-Pic was created by real users with a genuine interest in generating images. This interest differs from that of crowd workers who lack the intrinsic motivation to produce creative prompts or the original intent of the prompt's author to judge which image better aligns with their needs.
Tapping into authentic user preferences allows us to train a scoring function that estimates the user's satisfaction from a particular generated image given a prompt. To train such a scoring function we finetune CLIP-H [12, 7] using human preference data and an analogous objective to that of InstructGPT's reward model [10]. This objective aims to maximize the probability of a preferred image being picked over an unpreferred one, or even the probability in cases of a tie. We find that the resulting scoring function, PickScore<sup>3</sup>, achieves superhuman performance in the task of predicting user preferences (a  $70.5\%$  accuracy rate, compared to humans'  $68.0\%$ ), while zero-shot CLIP-H  $(60.8\%)$  and the popular aesthetics predictor [14]  $(56.8\%)$  perform closer to chance  $(56.8\%)$ .
Equipped with a dataset for human preferences and a state-of-the-art scoring function, we propose updating the standard protocol for evaluating text-to-image generation models. First, we suggest that researchers evaluate their text-to-image models using prompts from Pick-a-Pic, which better represent what humans want to generate than mundane captions, such as those found in MS-COCO [2, 9]. Second, to compare PickScore with FID, we conduct a human evaluation study and find that even when evaluated against MS-COCO captions, PickScore exhibits a strong correlation with human preferences (0.917), while ranking with FID yields a negative correlation (-0.900). Importantly, we also compare PickScore with other evaluation metrics using model rankings inferred from real user preferences. We observe that PickScore is more strongly correlated with ground truth rankings, as determined by real users, than other evaluation metrics. Thus, we recommend using PickScore as a more reliable evaluation metric than existing ones.
Finally, we explore how PickScore can improve the quality of vanilla text-to-image models via ranking. To accomplish this, we generate images with different initial random noises as well as different templates (e.g. "breathtaking [prompt]. award-winning, professional, highly detailed") to slightly alter the user prompt. We then test the impact of selecting the top image according to different scoring functions. Our findings indicate that human raters prefer images selected by PickScore more than those selected by CLIP-H [7] (win rate of  $71.3\%$ ), an aesthetics predictor [14] (win rate of  $85.1\%$ ), and the vanilla text-to-image model (win rate of  $71.4\%$ ).
In summary, the presented work addresses a gap in the field of text-to-image generation by creating a large, open, high-quality dataset of human preferences over user-prompted model-generated images. We demonstrate the potential of this dataset by training a scoring function, PickScore, which exhibits a performance superior to any other publicly-available automatic scoring function, in predicting human preferences, evaluating text-to-image models, and improving them via ranking. We encourage the research community to adopt Pick-a-Pic and PickScore as a basis for further advances in text-to-image modeling and incorporating human preferences into the learning process.
<sup>2</sup>We recently open-sourced a new version of the Pick-a-Pic dataset that has more than one million examples. <sup>3</sup>The model is available at https://huggingface.co/yuvalkirstain/PickScore_v1
(c)
Figure 2: How Pick-a-Pic data is collected through the app: (a) the user first writes a caption, and receives two images; (b) the user makes a preference judgment; (c) a new image is presented instead of the rejected image. This flow repeats itself until the user changes the prompt.
2 Pick-a-Pic Dataset
The Pick-a-Pic dataset $^4$  was created by logging user interactions with the Pick-a-Pic web application for text-to-image generation. Overall, the Pick-a-Pic dataset contains over 500,000 examples and 35,000 distinct prompts. Each example contains a prompt, two generated images, and a label for which image is preferred, or if there is a tie when no image is significantly preferred over the other. The images in the dataset were generated by employing multiple backbone models, namely, Stable Diffusion 2.1, Dreamlike Photoreal 2.0 $^5$ , and Stable Diffusion XL variants [13] while sampling different classifier-free guidance scale values [6]. As we continue with our efforts to collect more user interactions through the Pick-a-Pic web app and decrease the number of NSFW examples included in the dataset, we will periodically upload new revisions of the dataset.
The Pick-a-Pic Web App To ensure maximum accessibility for a wide range of users, the user interface was designed with simplicity in mind. The application allows users to write creative prompts and generate images. At each turn, the user is presented with two generated images (conditioned on their prompt), and asked to select their preferred option or indicate a tie if they have no strong preference. Upon selection, the rejected (non-preferred) image is replaced with a newly generated image, and the process repeats. The user can also clear or edit the prompt at any time, and the app will generate new images appropriately. Figure 2 illustrates the usage flow.
Real Data from Real Users A key advantage of Pick-a-Pic is that our data is collected from real, intrinsically-motivated users, rather than paid crowd workers. We achieve this by approaching a wide audience through various social media channels such as Twitter, Facebook, Discord, and Reddit. At the same time, we mitigate the risk of collecting low-quality data resulting from potential misuse of the application by implementing several quality control measures. First, users are required to authenticate their identity using either a Gmail or a Discord account. Second, we closely monitor user activity logs and take action to ban users who generate NSFW content, use multiple instances of the web app simultaneously, or make judgments at an unreasonably fast pace. Third, we use a list of NSFW phrases to prevent users from generating harmful content. Last, we limit users to 1000 interactions and periodically increase the limit. These measures work in tandem to ensure the integrity and reliability of Pick-a-Pic's data.
Annotation Methodology While piloting the web app, we experimented with different annotation strategies to optimize for data quality, efficiency of collection, and user experience. Specifically, we tested the following annotation options: (1) 4 images, no ties; (2) 2 images, no ties; (3) 2 images,
<sup>4</sup>The dataset is available at https://huggingface.co/datasets/yuvalkirstain/pickapic_v1, and its updated version is available at https://huggingface.co/datasets/yuvalkirstain/pickapic_v2.
Dreamlike Photoreal 2.0 is a finetuned version of Stable Diffusion 1.5
<sup>6</sup>All users provide explicit consent to share their interactions with Pick-a-Pic's application as part of a public dataset. To ensure privacy, we anonymize user IDs.
Figure 3: The Pick-a-Pic dataset enables us to perform model selection (a), and model evaluation (b).
with ties. We found that the latter option (2 images, with ties) exceeds the other two in terms of user engagement and inter-rater agreement.
Preprocessing When processing the collected interactions, we filter prompts with NSFW phrases and banned users. We acknowledge that there are still NSFW images and prompts, and will periodically attempt to update the dataset and reduce such occurrences. To divide the dataset into training, validation, and testing subsets, we first sample one thousand prompts, ensuring that each prompt was created by a unique user. Next, we randomly divide those prompts into two sets of equal size to create the validation and test sets. We then sample exactly one example for each prompt to include in these sets. For the training set, we include all examples that do not share a prompt with the validation and test sets. This approach ensures that no split shares prompts with another split, and the validation and test sets do not suffer from being non-proportionally fitted to a specific prompt or user.
Statistics Since the creation of the Pick-a-Pic web app we have gathered 968,965 rankings which originated from 66,798 prompts and 6,394 users. However, as the Pick-a-Pic dataset is constantly updating, the reported experiments in this paper involve an NSFW filtered and not fully updated version of Pick-a-Pic, that contains 583,747 training examples, and 500 validation and test examples. The training set of this dataset contains 37,523 prompts from 4,375 distinct users.
Model Selection and Evaluation The Pick-a-Pic dataset offers a unique opportunity for a model selection and evaluation methodology, leveraging users' preferences for unbiased analysis. To illustrate this opportunity, we use the collected data and analyze the impact of changing the classifier-free guidance scale of Stable Diffusion XL (Alpha variant) on its performance. Specifically, we compare human preferences made when both images were generated by Stable Diffusion XL (Alpha variant) but using different classifier-free guidance scales<sup>7</sup>. For each scale, we compute the win ratio, representing the percentage of judgments where its use led to a preferred image. We also calculate the corresponding tie and lose ratios for each scale, enabling a detailed analysis of which classifier-free guidance scales are more effective. Our results are depicted in Figure 3 (a), and verify for example, that a guidance scale of 9 usually yields preferred images when compared to a guidance scale of 3.
Furthermore, by examining user preferences between images generated by different backbone models, we can determine which model is preferred more by users. For instance, considering judgments in which one image was generated by Dreamlike Photoreal 2.0 and the other by Stable Diffusion 2.1, we can evaluate which model is more performant. As shown in fig. 3 (b), users usually prefer Dreamlike Photoreal 2.0 over Stable Diffusion 2.1. We encourage researchers to contact us and include their text-to-image models in the Pick-a-Pic web app for the purpose of model selection and evaluation.
3 PickScore
One valuable outcome from collecting a large, natural dataset of user preferences is that we can use it to train a function that scores the quality of a generated image given a prompt. We train the PickScore scoring function over Pick-a-Pic by combining a CLIP-style model with a variant of
The frequency of different guidance scales is similar.
InstructGPT's reward model objective [10]. PickScore is able to predict user preferences in held-out Pick-a-Pic prompts better than any other publicly-available scoring function, surpassing even expert human annotators (Section 4). Such a scoring function can be of value for various scenarios, such as performing model evaluation (Section 5), increasing the quality of generated images via ranking (Section 6), building better large-scale datasets to improve text-to-image models [14], and improving text-to-image models through weak supervision (e.g. RLHF).
Model PickScore follows the architecture of CLIP [12]; given a prompt  $x$  and an image  $y$ , our scoring function  $s$  computes a real number by representing  $x$  using a transformer text encoder and  $y$  using a transformer image encoder as  $d$ -dimensional vectors, and returning their inner product:
$$
s (x, y) = E _ {\mathrm {t x t}} (x) \cdot E _ {\mathrm {i m g}} (y) \cdot T \tag {1}
$$
Where  $T$  is the learned scalar temperature parameter of CLIP.
Objective The input for our objective includes a scoring function  $s$ , a prompt  $x$ , two images  $y_{1}, y_{2}$ , and a preference distribution vector  $p$ , which captures the user's preference over the two images. Specifically,  $p$  takes a value of [1, 0] if  $y_{1}$  is preferred, [0, 1] if  $y_{2}$  is preferred, or [0.5, 0.5] for ties. Given this input, the objective optimizes the scoring function's parameters by minimizing the KL-divergence between the preference  $p$  and the softmax-normalized scores of  $y_{1}$  and  $y_{2}$ :
$$
\hat {p} _ {i} = \frac {\exp s (x , y _ {i})}{\sum_ {j = 1} ^ {2} \exp s (x , y _ {j})} \tag {2}
$$
$$
L _ {\text {p r e f}} = \sum_ {i = 1} ^ {2} p _ {i} (\log p _ {i} - \log \hat {p} _ {i}) \tag {3}
$$
Since many examples can originate from the same prompt, we mitigate the risk of overfitting to a small set of prompts by applying a weighted average when reducing the loss across examples in the batch. Specifically, we weigh each example in the batch, with an inverse proportion to its prompt frequency in the dataset. This objective is analogous to InstructGPT's reward model objective [10]. We also experiment with incorporating in-batch negatives into the objective, but find that this yields a less accurate scoring function (see Appendix).
Training We finetune CLIP-H [7] using our framework on the Pick-a-Pic training set. We train the model for 4,000 steps, with a learning rate of 3e-6, a total batch size of 128, and a warmup period of 500 steps, which follows a linearly decaying learning rate; the experiment is completed in less than an hour with 8 A100 GPUs. We did not perform hyperparameter search, which might further improve results. For model selection, we evaluate the model's accuracy on the validation set (without the option for a tie) in intervals of 100 steps, and keep the best-performing checkpoint.
4 Preference Prediction
We first evaluate PickScore on the task it was trained to do: predict human preferences. We find that PickScore outperforms all other baselines, including expert human annotators.
Metric To evaluate the ability of models to predict human preferences, we use an adapted accuracy metric that accounts for the possibility of a tie. Our metric assigns one point to the model for predicting the same label as the user, half a point if either label or prediction is a tie (but not both), and zero otherwise.
Tie Threshold Selection Utilizing the notation specified in Section 3, each model requires a tie threshold probability  $t$  to predict a tied outcome when  $|\hat{p}_1 - \hat{p}_2| < t$ . To achieve this, we evaluate each model on the validation set using various tie threshold probabilities and subsequently determine the most optimal tie threshold for each model. For human experts, we do not perform tie selection and explicitly allow them to select a tie.
<sup>8</sup>https://github.com/yuvalkirstain/PickScore
Table 1: Quantitative results on Pick-a-Pic.
Figure 4: Disagreement between CLIP-H (left) and PickScore (right) on the Pick-a-Pic validation set. We add green borders around images that humans preferred.
(a) Accuracy across different tie thresholds on the Pick-a-Pic validation set
Model	Accuracy
Random	56.8
Human Expert	68.0
Aesthetics [14]	56.8
CLIP-H [7]	60.8
ImageReward [18]	61.1
HPS [17]	66.7
PickScore (Ours)	70.5
(b) Performance on the Pick-a-Pic test set.
Baselines We compare our model with CLIP-H [7], an aesthetics predictor [14] built on top of CLIP-L [12], a random classifier, and human experts. For completeness, we also compare our results with models from concurrent work, namely, HPS [17] and ImageReward [18].
Results First, we compare the models' performance on the validation set across different tie thresholds. Figure 1a shows that PickScore outperforms the baselines across almost all thresholds, and achieves the highest global score by a wide margin. After selecting the best-performing tie threshold for each model, we use the threshold to evaluate the different models on the test set.
Table 1b shows that the aesthetics score (56.8) and CLIP-H (60.8) perform closer to a random chance baseline (56.8), while PickScore  $(70.5 \pm 0.142)^{10}$  achieves superhuman performance, as it even outperforms human experts (68.0). It is important to emphasize a core difference between real users that produce the ground truth labels and annotators used to evaluate human performance. The users that produce the ground truth are actual text-to-image users, which have an idea (which may be incomplete) for an image, and invent a prompt (which may lack details) with hope that the resulting image will match their preferences. In contrast, annotators that are used to measure human performance are oblivious to the original user's context, idea, and motivation. Therefore, superhuman performance on this task means that the model is able to outperform a human annotator that is oblivious to the original user's context, idea, and motivation. The superhuman performance of PickScore showcases the importance of using real users as ground truth rather than expert annotators when collecting human preferences. Moreover, the relatively modest human performance (68.0) on the task, when compared to a random baseline (56.8), shows that predicting human preferences in text-to-image generation is a difficult task for human annotators.
For completeness, we also include the results of concurrent work from HPS [17], which scores 66.7, and ImageReward [18], which scores 61.1; PickScore outperforms both. To further illustrate the differences between CLIP-H and PickScore, we showcase examples of disagreement from the Pick-a-Pic validation set in Figure 4. We notice that PickScore often chooses more aesthetically pleasing images than CLIP-H; at times, at the cost of faithfulness to the prompt.
9Throughout this paper, we use friends and colleagues of the authors for expert human annotation.
<sup>10</sup>We train PickScore using three different random seeds and report the mean and variance.
Figure 5: Images generated using the same seed and model, but using different classifier-free guidance (CFG) scales. Even though high guidance scales lead to worse FID, humans usually find them more pleasing than low guidance scales.
Figure 6: Correlation between the win ratio of different models according to FID and PickScore to human experts on the MS-COCO validation set.
Figure 7: Correlation between Elo ratings of real users and Elo ratings by CLIP-H, ImageReward [18], HPS [17], and PickScore.
5 Model Evaluation
Despite significant progress in the generative capabilities of text-to-image models, the standard and most popular prompt dataset has remained the Microsoft Common Objects in Context dataset (MS-COCO) [9]. Similarly, the Fréchet Inception Distance (FID) [5] is still the main metric used for model evaluation. In this section, we explain why we recommend researchers to evaluate their models using prompts from Pick-a-Pic rather than (or at least alongside) MS-COCO, and show that when evaluating state-of-the-art text-to-image models, PickScore is more aligned with human judgments than other automatic evaluation metrics.
Model Evaluation Prompts The prompts contained in the MS-COCO dataset are captions of photographs taken by amateur photographers, depicting objects and humans in everyday settings. While certain captions within this dataset may pose a challenge to text-to-image models, it is evident that the scope of interest for text-to-image users extends beyond commonplace objects and humans. Moreover, the main use-case of image generation is arguably to generate fiction, which cannot be captured by camera. By construction, Pick-a-Pic's prompts are sampled from real users, and thus better represent the natural distribution of text-to-image intents. We therefore strongly advocate that the research community employ prompts from Pick-a-Pic when assessing the performance of text-to-image models.
FID The FID metric [5] gauges the degree of resemblance between a set of generated images and a set of authentic images, at the set level. To do so, it first embeds the real and generated images into
the feature space of an Inception net [15], and then estimates the mean and covariance of both sets of images and calculates their similarity. FID is thus geared towards measuring the realism of a set of images, but is oblivious to the prompts. In contrast, PickScore provides a per-instance score, and is directly conditioned on the prompt. We thus hypothesize that PickScore will correlate better with human judgements of generation quality.
To empirically test our hypothesis with the most "convenient" settings for the FID metric, we select 100 random captions from MS-COCO validation split. For each caption, we generate images from 9 different models based on the same set of prompts, and ask human experts to rank the 9 generated images (with ties), inducing pairwise preferences. Specifically, we use Stable Diffusion 1.5, Stable Diffusion 2.1, and Dreamlike Photoreal 2.0 combined with three different classifier-free guidance scales (3, 6, and 9). We then repeat the labeling process (over the same images) using FID, and PickScore instead of humans to determine preferences. Since FID does not operate on a per-example base, when "labeling" with FID, we simply choose the model that has a lower (better) FID score on MS-COCO.