Package: hpsv2
HPS v2: Benchmarking Text-to-Image Generative ModelsThis is the official repository for the paper:Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis.Updates[09/02/2024] We released HPS v2.1 model trained on higher quality datasets. Happy new year![08/02/2023] We released thePyPI package. You can learn how to use it from theQuick start section.[08/02/2023] Updatedtest.jsonto include raw annotation by each annotator.[07/29/2023] We includedSDXL Refiner 0.9model in the benchmark.[07/29/2023] We releasedthe benchmark and HPD v2 test data. HPD v2 train data will be released soon.[07/27/2023] We includedSDXL Base 0.9model in the benchmark.[07/26/2023] We updated ourcompressed checkpoint.[07/19/2023] Live demo is available at 🤗Hugging Face.[07/18/2023] We released ourtest data.OverviewHuman Preference Dataset v2 (HPD v2): a large-scale (798k preference choices / 430k images), a well-annotated dataset of human preference choices on images generated by text-to-image generative models.Human Preference Score v2 (HPS v2): a preference prediction model trained on HPD v2. HPS v2 can be used to compare images generated with the same prompt. We also provide a fair, stable, and easy-to-use set of evaluation prompts for text-to-image generative models.The HPS benchmarkThe HPS benchmark evaluates models' capability of generating images of 4 styles:Animation,Concept-art,Painting, andPhoto.v2 benchmarkModelAnimationConcept-artPaintingPhotoAveragedDreamlike Photoreal 2.028.2427.6027.5927.9927.86SDXL Refiner 0.928.4527.6627.6727.4627.80Realistic Vision28.2227.5327.5627.7527.77SDXL Base 0.928.4227.6327.6027.2927.73Deliberate28.1327.4627.4527.6227.67ChilloutMix27.9227.2927.3227.6127.54MajicMix Realistic27.8827.1927.2227.6427.48Openjourney27.8527.1827.2527.5327.45DeepFloyd-XL27.6426.8326.8627.7527.27Epic Diffusion27.5726.9627.0327.4927.26Stable Diffusion v2.027.4826.8926.8627.4627.17Stable Diffusion v1.427.2626.6126.6627.2726.95DALL·E 227.3426.5426.6827.2426.95Versatile Diffusion26.5926.2826.4327.0526.59CogView226.5026.5926.3326.4426.47VQGAN + CLIP26.4426.5326.4726.1226.39DALL·E mini26.1025.5625.5626.1225.83Latent Diffusion25.7325.1525.2526.9725.78FuseDream25.2625.1525.1325.5725.28VQ-Diffusion24.9724.7025.0125.7125.10LAFITE24.6324.3824.4325.8124.81GLIDE23.3423.0823.2724.5023.55v2.1 benchmarkModelAnimationConcept-artPaintingPhotoAveragedSDXL Refiner 0.933.2632.0731.6328.3831.34SDXL Base 0.932.8431.3630.8627.4830.63Deliberate31.4630.4830.1728.8330.23Realistic Vision31.0129.9530.0028.6129.89Dreamlike Photoreal 2.030.8729.7529.4628.8529.73MajicMix Realistic29.6728.5028.4428.0228.66ChilloutMix29.4628.4628.3527.6328.47Openjourney28.3727.3827.5326.6627.48DeepFloyd-XL27.7126.0725.7927.9626.88Epic Diffusion27.0726.1426.1726.4326.45Stable Diffusion v2.027.0926.0225.6826.7326.38Stable Diffusion v1.426.0324.8724.8025.7025.35DALL·E 226.3824.5124.9325.5525.34Versatile Diffusion23.6923.3924.0224.6423.93CogView223.6424.8623.4022.6823.64VQGAN + CLIP22.5523.7623.4121.5122.81DALL·E mini21.5420.5020.3221.7221.02Latent Diffusion20.6319.6519.7921.2620.34FuseDream19.1619.3719.0720.0719.42VQ-Diffusion18.4418.3119.2420.6219.15LAFITE17.7917.5517.6120.8818.46GLIDE13.9013.5013.9416.7214.51Quick StartInstallation# Method 1: Pypi download and installpipinstallhpsv2# Method 2: install locallygitclonehttps://github.com/tgxs002/HPSv2.gitcdHPSv2
pipinstall-e.# Optional: images for reproducing our benchmark will be downloaded here# default: ~/.cache/hpsv2/exportHPS_ROOT=/your/cache/pathAfter installation, we show how to:Compare images using HPS v2.Reproduce our benchmark.Evaluate your own model using HPS v2.Evaluate our preference model.We also providecommand line interfacesfor debugging purposes.Image ComparisonYou can score and compare several images generated by the same prompt by running the following code:importhpsv2# imgs_path can be a list of image paths, with the images generated by the same prompt# or image path of string type# or image of PIL.Image.Image typeresult=hpsv2.score(imgs_path,'<prompt>',hps_version="v2.1")Note: Comparison is only meaningful for images generated by thesame prompt. You can also pass "v2.0" to hps_version to use our updated model. Scores can not be directly compared between v2.0 and v2.1.Benchmark ReproductionWe also provideimagesgenerated by models in ourbenchmarkused for evaluation. You can easily download the data and evaluate the models by running the following code.importhpsv2print(hpsv2.get_available_models())# Get models that have access to datahpsv2.evaluate_benchmark('<model_name>')Custom EvaluationTo evaluate your own text-to-image generative model, you can prepare the images for evaluation base on thebenchmark promptswe provide by running the following code:importosimporthpsv2# Get benchmark prompts (<style> = all, anime, concept-art, paintings, photo)all_prompts=hpsv2.benchmark_prompts('all')# Iterate over the benchmark prompts to generate imagesforstyle,promptsinall_prompts.items():foridx,promptinenumerate(prompts):image=TextToImageModel(prompt)# TextToImageModel is the model you want to evaluateimage.save(os.path.join("<image_path>",style,f"{idx:05d}.jpg"))# <image_path> is the folder path to store generated images, as the input of hpsv2.evaluate().And then run the following code to conduct evaluation:importhpsv2# <image_path> is the same as <image_path> in the prevoius part.# <hps_version> is the version of HPS model, it can be v2 or v2.1. Default to v2.hpsv2.evaluate("<images_path>",hps_version="<hps_version>")Preference Model EvaluationEvaluating HPS v2's correlation with human preference choices:ModelAcc. on ImageReward test set (%)Acc. on HPD v2 test set (%)Acc. on new test set (%)Aesthetic Score Predictor57.476.857.8*ImageReward65.174.067.4*HPS61.277.666.8*PickScore62.979.867.8*Single Human65.378.165.4*HPS v265.783.373.2*HPS v2.166.884.174.8** Thenew test setis another test set annotated similarly to the HPD v2 test set, except that images are generated from 10 better models (Dreamlike Photoreal 2.0, SDXL Refiner 0.9, Realistic Vision, SDXL Base 0.9, Deliberate, ChilloutMix, MajicMix Realistic, Openjourney, DeepFloyd-XL, Epic Diffusion).HPS v2 checkpoint can be downloaded fromhere. The model and live demo is also hosted on 🤗 Hugging Face athere.Run the following commands to evaluate the HPS v2 model on HPD v2 test set and ImageReward test set(Need to install the packagehpsv2first):# evaluate on HPD v2 test setpythonevaluation.py--data-typetest--data-path/path/to/HPD--image-path/path/to/image_folder# evaluate on ImageReward test setpythonevaluation.py--data-typeImageReward--data-path/path/to/IR--image-path/path/to/image_folderHuman Preference Dataset v2The prompts in our dataset are sourced from DiffusionDB and MSCOCO Captions. Prompts from DiffusionDB are first cleaned by ChatGPT to remove biased function words. Human annotators are tasked to rank images generated by different text-to-image generative models from the same prompt. Totally there are about 798k pairwise comparisons of images for over 430k images and 107k prompts, 645k pairs for training split and 153k pairs for test split.Image sources of HPD v2:Source# of imagesCogView273697DALL·E 2101869GLIDE (mini)400Stable Diffusion v1.4101869Stable Diffusion v2.0101869LAFITE400VQ-GAN+CLIP400VQ-Diffusion400FuseDream400COCO Captions28272Currently, the test data can be downloaded fromhere. You can inspect the test data athttps://tgxs002.github.io/hpd_test_vis/. Here is a screenshot:The training dataset will bereleased soon.
Once unzipped, you should get a folder with the following structure:HPD
---- train/
-------- {image_id}.jpg
---- test/
-------- {image_id}.jpg
---- train.json
---- test.json
---- benchmark/
-------- benchmark_imgs/
------------ {model_id}/
---------------- {image_id}.jpg
-------- drawbench/
------------ {model_id}/
---------------- {image_id}.jpg
-------- anime.json
-------- concept-art.json
-------- paintings.json
-------- photo.json
-------- drawbench.jsonThe annotation file,train.json, is organized as:[
    {
        'human_preference': list[int], # 1 for preference
        'prompt': str,
        'file_path': list[str],
        'user_hash': str,
    },
    ...
]The annotation file,test.json, is organized as:[
    {
        'prompt': str,
        'image_path': list[str],
        'rank': list[int], # averaged ranking result for image at the same index in image_path,
        'raw_annotations': list[{'rank', 'user_hash'}]  # raw ranking result from each annotator
    },
    ...
]The benchmark prompts file, ie.anime.jsonis pure prompts. The corresponding image can be found in the folder of the corresponding model by indexing the prompt.Command Line InterfaceEvaluating Text-to-image Generative Models using HPS v2The generated images in our experiments can be downloaded fromhere.The following script reproduces thebenchmark tableand our results on DrawBench (reported in the paper)(Need to install the packagehpsv2first):# HPS v2 benchmark (for more than one models)pythonevaluation.py--data-typebenchmark_all--data-path/path/to/HPD/benchmark--image-path/path/to/benchmark_imgs# HPS v2 benchmark (for only one models)pythonevaluation.py--data-typebenchmark--data-path/path/to/HPD/benchmark--image-path/path/to/benchmark_imgs/${model_name}# DrawBenchpythonevaluation.py--data-typedrawbench--data-path/path/to/HPD/benchmark--image-path/path/to/drawbench_imgsScoring Single Generated Image and Corresponding PromptWe provide one example image in theasset/imagesdirectory of this repo. The corresponding prompt is"A cat with two horns on its head".Run the following commands to score the single generated image and the corresponding prompt(Need to install the packagehpsv2first):pythonimg_score.py--image-pathassets/demo_image.jpg--prompt'A cat with two horns on its head'where the parameterimage-pathcan accept multiple values.Train Human Preference PredictorTo train your own human preference predictor, just change the corresponding path inconfigs/controller.shand run the following command:# if you are running locallybashconfigs/HPSv2.shtrain8local# if you are running on slurmbashconfigs/HPSv2.shtrain8${quota_type}BibTeX@article{wu2023human,
  title={Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis},
  author={Wu, Xiaoshi and Hao, Yiming and Sun, Keqiang and Chen, Yixiong and Zhu, Feng and Zhao, Rui and Li, Hongsheng},
  journal={arXiv preprint arXiv:2306.09341},
  year={2023}
}