Package: image-eval
Image EvalWhat is this?TL;DR: A Python library providing utilities to evaluate your favorite text-to-image generation models (Stable Diffusion, Midjourney, and any custom models you may have).Metric NameValueclip_score22.5958aesthetic_predictor6.03433image_reward-0.661022human_preference_score0.196004fid[11.214, 57.471, 4.10, 442.811]inception_score[1.0, 0.0]So, why should I care?Since the advent of systems such asStable Diffusion,Midjourney, andDallE-2text-to-image generationmodels have taken the world by storm.However, evaluation of the quality of these systems still remains one of the
hardest challenges in continuing to improve them as there is a lack of standardization and robust tooling.If we can't all agree on a set of practices for evaluating these systems, how can we push the state-of-the-art?This library provides a suite of widely accepted metrics for evaluating them as well as an easy-to-use interface for performing human
evaluation of image generation model outputs.Cute, but what are you actually doing?We provide six standard automated evaluation metrics out-of-the-box including:CLIP ScoreInception ScoreFIDAesthetic PredictorImage RewardHuman Preference ScoreWe also provide a simple and ready-to-useStreamlitinterface for performing human evaluation of model outputs on your local machine.InstallationThis library has been tested on Python 3.9.12. Installing the library involves running:pip install image-eval
pip install git+https://github.com/openai/CLIP.git # A dependency some metrics requireOptionally, if you have a CUDA-enabled device, install theversion of PyTorchthat matches your CUDA version. For CUDA 11.3, that might look like:pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113NOTE: If you want to use theaesthetic_predictorandhuman_preference_scoremetrics, you will need to download the respective model weights.We require having amodelsfolder and then having a separate subfolder foraesthetic_predictorandhuman_preference_scorewith the downloaded weights.Foraesthetic_predictoryou can download the weights of the modelhere.Forhuman_preference_scoreyou can download the weights of the modelhere.Yourmodelsfolder should then have this structure:models/
├── aesthetic_predictor/
│   └── sac+logos+ava1-l14-linearMSE.pth
└── human_preference_score/
    └── hpc.ptYou can then specify yourmodelsfolder in theusagesection below.UsageThere are two ways to interact with theimage-evallibrary: either through the CLI or through the API.CLIYou can invoke all the metric computations through the CLI once you'vepip install'd the library. The library makes certain
assumptions about the format of the inputs to the CLI.For example, if you want to calculate a metric assessing the match between
prompts and generated images (as is the case withclip_score), you would invoke:image_eval -m clip_score -p /path/to/image_to_prompt.json -g /path/to/folder/with/generated/imagesHereimage_to_prompt.jsonis a JSON file with the following format:{
    "image_1.jpg": "prompt for image 1",
    "image_2.jpg": "prompt for image 2",
    ...
}whereimage_1.jpgandimage_2.jpgare the names of the generated images in the folder specified by the-gflag.If you want to calculate a metric assessing the match between generated images and a set of reference images (as is the case withfid), you would invoke:image_eval -m fid -g /path/to/generated/images -r /path/to/real/imagesSome metrics may need you to specify additional arguments. For example, if you want to use theaesthetic_predictororhuman_preference_scoremetrics, you would invoke:image_eval -m aesthetic_predictor -p /path/to/image_to_prompt.json -g /path/to/generated/images --model-dir /path/to/folder/with/modelsYou can also compute multiple metrics simultaneously by passing in a comma-separated list of metrics to the-mflag:image_eval -m clip_score,fid -p /path/to/image_to_prompt.json -g /path/to/generated/images -r /path/to/real/imagesIf you want to launch the human evaluation interface, you would invoke:image_eval --local-human-eval --model-predictions-json /path/to/model_comparisons.jsonHeremodel_comparisons.jsonis a JSON file with the following format:[
    {
        "model_1": "path to image 1 from model 1",
        "model_2": "path to image 1 from model 2",
        "prompt": "prompt for image 1"
    },
    {
        "model_1": "path to image 2 from model 1",
        "model_2": "path to image 2 from model 2",
        "prompt": "prompt for image 2"
    },
    ...
]wheremodel_1andmodel_2are the keys for the paths to image outputs for the respective models. Our library does expect thekeys to match these values exactly.An interface should launch in your browser athttp://localhost:8501.NOTE: When you clickCompute Model Winsa local file namedscores.jsonwill be created in the directory from which you launched the CLI.ProgrammaticYou can also interact with the library through the API directly. For example if you want to invoke theclip_scoremetric, you would do the following:from image_eval.evaluators import CLIPScoreEvaluator

evaluator = CLIPScoreEvaluator(device="cpu") # or "cuda" if you have a GPU-enabled device
images = [np.randint(0, 255, (224, 224, 3)) for _ in range(10)] # list of 10 random images
prompts = ["random prompt" * 10]
evaluator.evaluate(images, prompts)It's that simple!ContributingWe welcome any and all contributions to this library! We have a wishlist of items we'd like to add to the libraryhere.How to add your own automated metricWe designed the library to make it easy to add your own automated metric. All you need to do is create a new class that inherits from theBaseReferenceFreeEvaluatororBaseWithReferenceEvaluatorclass and implements theevaluatemethod.For examples of how our current metrics implement this contract, seeevaluators.py.RoadmapMake it easy to launch remote human evaluation task usingAWS GroundtruthAdd ability to log metrics toWandB