Package: clean-fid
clean-fid for Evaluating Generative ModelsProject|Paper|Colab-FID|Colab-Resize|Leaderboard TablesQuick start:Calculate FID|Calculate KID[New]Computing the FID using CLIP features[Kynkäänniemi et al, 2022]is now supported. Seeherefor more details.The FID calculation involves many steps that can produce inconsistencies in the final metric. As shown below, different implementations use different low-level image quantization and resizing functions, the latter of which are often implemented incorrectly.We provide an easy-to-use library to address the above issues and make the FID scores comparable across different methods, papers, and groups.Corresponding ManuscriptOn Aliased Resizing and Surprising Subtleties in GAN EvaluationGaurav Parmar,Richard Zhang,Jun-Yan ZhuCVPR, 2022CMU and AdobeIf you find this repository useful for your research, please cite the following work.@inproceedings{parmar2021cleanfid,
  title={On Aliased Resizing and Surprising Subtleties in GAN Evaluation},
  author={Parmar, Gaurav and Zhang, Richard and Zhu, Jun-Yan},
  booktitle={CVPR},
  year={2022}
}Aliased Resizing OperationsThe definitions of resizing functions are mathematical andshould never be a function of the library being used. Unfortunately, implementations differ across commonly-used libraries.  They are often implemented incorrectly by popular libraries. Try out the different resizing implementations in the Google colab notebookhere.The inconsistencies among implementations can have a drastic effect of the evaluations metrics. The table below shows that FFHQ dataset images resized with  bicubic implementation from other libraries (OpenCV, PyTorch, TensorFlow, OpenCV) have a large FID score (≥ 6) when compared to the same images resized with the correctly implemented PIL-bicubic filter. Other correctly implemented filters from PIL (Lanczos, bilinear, box) all result in relatively smaller FID score (≤ 0.75). Note that since TF 2.0, the new flagantialias(default:False) can produce results close to PIL. However, it was not used in the existing TF-FID repo and set asFalseby default.JPEG Image CompressionImage compression can have a surprisingly large effect on FID.  Images are perceptually indistinguishable from each other but have a large FID score. The FID scores under the images are calculated between all FFHQ images saved using the corresponding JPEG format and the PNG format.Below, we study the effect of JPEG compression for StyleGAN2 models trained on the FFHQ dataset (left) and LSUN outdoor Church dataset (right). Note that LSUN dataset images were collected with JPEG compression (quality 75), whereas FFHQ images were collected as PNG. Interestingly, for LSUN dataset, the best FID score (3.48) is obtained when the generated images are compressed with JPEG quality 87.Quick Startinstall the librarypip install clean-fidComputing FIDCompute FID between two image foldersfrom cleanfid import fid
score = fid.compute_fid(fdir1, fdir2)Compute FID between one folder of images and pre-computed datasets statistics (e.g.,FFHQ)from cleanfid import fid
score = fid.compute_fid(fdir1, dataset_name="FFHQ", dataset_res=1024, dataset_split="trainval70k")Compute FID using a generative model and pre-computed dataset statistics:from cleanfid import fid
# function that accepts a latent and returns an image in range[0,255]
gen = lambda z: GAN(latent=z, ... , <other_flags>)
score = fid.compute_fid(gen=gen, dataset_name="FFHQ",
        dataset_res=256, num_gen=50_000, dataset_split="trainval70k")Computing CLIP-FIDTo use the CLIP features when computing the FID[Kynkäänniemi et al, 2022], specify the flagmodel_name="clip_vit_b_32"e.g. to compute the CLIP-FID between two folders of images use the following commands.from cleanfid import fid
score = fid.compute_fid(fdir1, fdir2, mode="clean", model_name="clip_vit_b_32")Computing KIDThe KID score can be computed using a similar interface as FID.
The dataset statistics for KID are only precomputed for smaller datasetsAFHQ,BreCaHAD, andMetFaces.Compute KID between two image foldersfrom cleanfid import fid
score = fid.compute_kid(fdir1, fdir2)Compute KID between one folder of images and pre-computed datasets statisticsfrom cleanfid import fid
score = fid.compute_kid(fdir1, dataset_name="brecahad", dataset_res=512, dataset_split="train")Compute KID using a generative model and pre-computed dataset statistics:from cleanfid import fid
# function that accepts a latent and returns an image in range[0,255]
gen = lambda z: GAN(latent=z, ... , <other_flags>)
score = fid.compute_kid(gen=gen, dataset_name="brecahad", dataset_res=512, num_gen=50_000, dataset_split="train")Supported Precomputed DatasetsWe provide precompute statistics for the following commonly used configurations. Please contact us if you want to add statistics for your new datasets.TaskDatasetResolutionReference Split# Reference ImagesmodeImage Generationcifar1032train50,000clean,legacy_tensorflow,legacy_pytorchImage Generationcifar1032test10,000clean,legacy_tensorflow,legacy_pytorchImage Generationffhq1024, 256trainval50,000clean,legacy_tensorflow,legacy_pytorchImage Generationffhq1024, 256trainval70k70,000clean,legacy_tensorflow,legacy_pytorchImage Generationlsun_church256train50,000clean,legacy_tensorflow,legacy_pytorchImage Generationlsun_church256trainfull126,227cleanImage Generationlsun_horse256train50,000clean,legacy_tensorflow,legacy_pytorchImage Generationlsun_horse256trainfull2,000,340cleanImage Generationlsun_cat256train50,000clean,legacy_tensorflow,legacy_pytorchImage Generationlsun_cat256trainfull1,657,264clean,legacy_tensorflow,legacy_pytorchFew Shot Generationafhq_cat512train5153clean,legacy_tensorflow,legacy_pytorchFew Shot Generationafhq_dog512train4739clean,legacy_tensorflow,legacy_pytorchFew Shot Generationafhq_wild512train4738clean,legacy_tensorflow,legacy_pytorchFew Shot Generationbrecahad512train1944clean,legacy_tensorflow,legacy_pytorchFew Shot Generationmetfaces1024train1336clean,legacy_tensorflow,legacy_pytorchImage to Imagehorse2zebra256test140clean,legacy_tensorflow,legacy_pytorchImage to Imagecat2dog256test500clean,legacy_tensorflow,legacy_pytorchUsing precomputed statisticsIn order to compute the FID score with the precomputed dataset statistics, use the corresponding options. For instance, to compute the clean-fid score on generated 256x256 FFHQ images use the command:fid_score = fid.compute_fid(fdir1, dataset_name="ffhq", dataset_res=256,  mode="clean", dataset_split="trainval70k")Create Custom Dataset Statisticsdataset_path: folder where the dataset images are storedcustom_name: name to be used for the statisticsGenerating custom statistics (saved to local cache)from cleanfid import fid
fid.make_custom_stats(custom_name, dataset_path, mode="clean")Using the generated custom statisticsfrom cleanfid import fid
score = fid.compute_fid("folder_fake", dataset_name=custom_name,
          mode="clean", dataset_split="custom")Removing the custom statsfrom cleanfid import fid
fid.remove_custom_stats(custom_name, mode="clean")Check if a custom statistic already existsfrom cleanfid import fid
fid.test_stats_exists(custom_name, mode)Backwards CompatibilityWe provide two flags to reproduce the legacy FID score.mode="legacy_pytorch"This flag is equivalent to using the popular PyTorch FID implementation providedhereThe difference between using clean-fid with this option andcodeis~2e-06Seedocfor how the methods are comparedmode="legacy_tensorflow"This flag is equivalent to using the officialimplementation of FIDreleased by the authors.The difference between using clean-fid with this option andcodeis~2e-05Seedocfor detailed steps for how the methods are comparedBuilding clean-fid locally from sourcepython setup.py bdist_wheel
pip install dist/*CleanFID Leaderboard for common tasksWe compute the FID scores using the corresponding methods used in the original papers and using the Clean-FID proposed here.
All values are computed using 10 evaluation runs. We provide anAPIto query the results shown in the tables below directly from the pip package.If you would like to add new numbers and models to our leaderboard, feel free to contact us.CIFAR-10 (few shot)Thetestset is used as the reference distribution and compared to 10k generated images.100% data (unconditional)ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2 (+ada + tuning)[Karras et al, 2020]- †- †8.20 ± 0.10stylegan2 (+ada)[Karras et al, 2020]- †- †9.26 ± 0.06stylegan2 (diff-augment)[Zhao et al, 2020][ckpt]9.899.90 ± 0.0910.85 ± 0.10stylegan2 (mirror-flips)[Karras et al, 2020][ckpt]11.0711.07 ± 0.1012.96 ± 0.07stylegan2 (without-flips)[Karras et al, 2020]- †- †14.53 ± 0.13AutoGAN (config A)[Gong et al, 2019]- †- †21.18 ± 0.12AutoGAN (config B)[Gong et al, 2019]- †- †22.46 ± 0.15AutoGAN (config C)[Gong et al, 2019]- †- †23.62 ± 0.30† These methods use the training set as the reference distribution and compare to 50k generated images20% dataModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2-diff-augment[Zhao et al, 2020][ckpt]12.1512.12 ± 0.1514.18 ± 0.13stylegan2-mirror-flips[Karras et al, 2020][ckpt]23.0823.01 ± 0.1929.49 ± 0.1710% dataModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2-diff-augment[Zhao et al, 2020][ckpt]14.5014.53 ± 0.1216.98 ± 0.18stylegan2-mirror-flips[Karras et al, 2020][ckpt]36.0235.94 ± 0.1743.60 ± 0.17CIFAR-100 (few shot)Thetestset is used as the reference distribution and compared to 10k generated images.100% dataModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2-mirror-flips[Karras et al, 2020][ckpt]16.5416.44 ± 0.1918.44 ± 0.24stylegan2-diff-augment[Zhao et al, 2020][ckpt]15.2215.15 ± 0.1316.80 ± 0.1320% dataModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2-mirror-flips[Karras et al, 2020][ckpt]32.3032.26 ± 0.1934.88 ± 0.14stylegan2-diff-augment[Zhao et al, 2020][ckpt]16.6516.74 ± 0.1018.49 ± 0.0810% dataModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2-mirror-flips[Karras et al, 2020][ckpt]45.8745.97 ± 0.2046.77 ± 0.19stylegan2-diff-augment[Zhao et al, 2020][ckpt]20.7520.69 ± 0.1223.40 ± 0.09FFHQall images @ 1024x1024Values are computed using 50k generated imagesModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDReference Splitstylegan1 (config A)[Karras et al, 2020]4.44.39 ± 0.034.77 ± 0.03trainvalstylegan2 (config B)[Karras et al, 2020]4.394.43 ± 0.034.89 ± 0.03trainvalstylegan2 (config C)[Karras et al, 2020]4.384.40 ± 0.024.79 ± 0.02trainvalstylegan2 (config D)[Karras et al, 2020]4.344.34 ± 0.024.78 ± 0.03trainvalstylegan2 (config E)[Karras et al, 2020]3.313.33 ± 0.023.79 ± 0.02trainvalstylegan2 (config F)[Karras et al, 2020][ckpt]2.842.83 +- 0.033.06 +- 0.02trainvalstylegan2[Karras et al, 2020][ckpt]N/A2.76 ± 0.032.98 ± 0.03trainval70k140k - images @ 256x256 (entire training set with horizontal flips)The 70k images fromtrainval70kset is used as the reference images and compared to 50k generated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDzCR[Zhao et al, 2020]†3.45 ± 0.193.29 ± 0.013.40 ± 0.01stylegan2[Karras et al, 2020]†3.66 ± 0.103.57 ± 0.033.73 ± 0.03PA-GAN[Zhang and Khoreva et al, 2019]†3.78 ± 0.063.67 ± 0.033.81 ± 0.03stylegan2-ada[Karras et al, 2020]†3.88 ± 0.133.84 ± 0.023.93 ± 0.02Auxiliary rotation[Chen et al, 2019]†4.16 ± 0.054.10 ± 0.024.29 ± 0.03Adaptive Dropout[Karras et al, 2020]†4.16 ± 0.054.09 ± 0.024.20 ± 0.02Spectral Norm[Miyato et al, 2018]†4.60 ± 0.194.43 ± 0.024.65 ± 0.02WGAN-GP[Gulrajani et al, 2017]†6.54 ± 0.376.19 ± 0.036.62 ± 0.03† reported by[Karras et al, 2020]30k - images @ 256x256 (Few Shot Generation)The 70k images fromtrainval70kset is used as the reference images and compared to 50k generated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2[Karras et al, 2020][ckpt]6.166.14 ± 0.0646.49 ± 0.068DiffAugment-stylegan2[Zhao et al, 2020][ckpt]5.055.07 ± 0.0305.18 ± 0.03210k - images @ 256x256 (Few Shot Generation)The 70k images fromtrainval70kset is used as the reference images and compared to 50k generated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2[Karras et al, 2020][ckpt]14.7514.88 ± 0.07016.04 ± 0.078DiffAugment-stylegan2[Zhao et al, 2020][ckpt]7.867.82 ± 0.0458.12 ± 0.0445k - images @ 256x256 (Few Shot Generation)The 70k images fromtrainval70kset is used as the reference images and compared to 50k generated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2[Karras et al, 2020][ckpt]26.6026.64 ± 0.08628.17 ± 0.090DiffAugment-stylegan2[Zhao et al, 2020][ckpt]10.4510.45 ± 0.04710.99 ± 0.0501k - images @ 256x256 (Few Shot Generation)The 70k images fromtrainval70kset is used as the reference images and compared to 50k generated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2[Karras et al, 2020][ckpt]62.1662.14 ± 0.10864.17 ± 0.113DiffAugment-stylegan2[Zhao et al, 2020][ckpt]25.6625.60 ± 0.07127.26 ± 0.077LSUN Categories100% dataThe 50k images fromtrainset is used as the reference images and compared to 50k generated images.CategoryModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDOutdoor Churchesstylegan2[Karras et al, 2020][ckpt]3.863.87 ± 0.0294.08 ± 0.028Horsesstylegan2[Karras et al, 2020][ckpt]3.433.41 ± 0.0213.62 ± 0.023Catstylegan2[Karras et al, 2020][ckpt]6.937.02 ± 0.0397.47 ± 0.035LSUN CAT - 30k images (Few Shot Generation)All 1,657,264 images fromtrainfullsplit are used as the reference images and compared to 50k generated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2-mirror-flips[Karras et al, 2020][ckpt]10.1210.15 ± 0.0410.87 ± 0.04stylegan2-diff-augment[Zhao et al, 2020][ckpt]9.689.70 ± 0.0710.25 ± 0.07LSUN CAT - 10k images (Few Shot Generation)All 1,657,264 images fromtrainfullsplit are used as the reference images and compared to 50k generated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2-mirror-flips[Karras et al, 2020][ckpt]17.9317.98 ± 0.0918.71 ± 0.09stylegan2-diff-augment[Zhao et al, 2020][ckpt]12.0712.04 ± 0.0812.53 ± 0.08LSUN CAT - 5k images (Few Shot Generation)All 1,657,264 images fromtrainfullsplit are used as the reference images and compared to 50k generated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2-mirror-flips[Karras et al, 2020][ckpt]34.6934.66 ± 0.1235.85 ± 0.12stylegan2-diff-augment[Zhao et al, 2020][ckpt]16.1116.11 ± 0.0916.79 ± 0.09LSUN CAT - 1k images (Few Shot Generation)All 1,657,264 images fromtrainfullsplit are used as the reference images and compared to 50k generated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2-mirror-flips[Karras et al, 2020][ckpt]182.85182.80 ± 0.21185.86 ± 0.21stylegan2-diff-augment[Zhao et al, 2020][ckpt]42.2642.07 ± 0.1643.12 ± 0.16AFHQ (Few Shot Generation)AFHQ DogAll 4739 images fromtrainsplit are used as the reference images and compared to 50k generated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2[Karras et al, 2020][ckpt]19.3719.34 ± 0.0820.10 ± 0.08stylegan2-ada[Karras et al, 2020][ckpt]7.407.41 ± 0.027.61 ± 0.02AFHQ WildAll 4738 images fromtrainsplit are used as the reference images and compared to 50k generated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDstylegan2[Karras et al, 2020][ckpt]3.483.55 ± 0.033.66 ± 0.02stylegan2-ada[Karras et al, 2020][ckpt]3.053.01 ± 0.023.03 ± 0.02BreCaHAD (Few Shot Generation)All 1944 images fromtrainsplit are used as the reference images and compared to 50k generated images.ModelLegacyFID(reported)LegacyFID(reproduced)Clean-FIDLegacyKID(reported)10^3LegacyKID(reproduced)10^3CleanKID10^3stylegan2[Karras et al, 2020][ckpt]97.7297.46 ± 0.1798.35 ± 0.1789.7689.90 ± 0.3192.51 ± 0.32stylegan2-ada[Karras et al, 2020][ckpt]15.7115.70 ± 0.0615.63 ± 0.062.882.93 ± 0.083.08 ± 0.08MetFaces (Few Shot Generation)All 1336 images fromtrainsplit are used as the reference images and compared to 50k generated images.ModelLegacyFID(reported)LegacyFID(reproduced)Clean-FIDLegacyKID(reported)10^3LegacyKID(reproduced)10^3CleanKID10^3stylegan2[Karras et al, 2020][ckpt]57.2657.36 ± 0.1065.74 ± 0.1135.6635.69 ± 0.1640.90 ± 0.14stylegan2-ada[Karras et al, 2020][ckpt]18.2218.18 ± 0.0319.60 ± 0.032.412.38 ± 0.052.86 ± 0.04Horse2Zebra (Image to Image Translation)All 140 images fromtestsplit are used as the reference images and compared to 120 translated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDCUT[Park et al, 2020]45.545.5143.71Distance[Benaim and Wolf et al, 2017]reported by[Park et al, 2020]72.071.9771.01FastCUT[Park et al, 2020]73.473.3872.53CycleGAN[Zhu et al, 2017]reported by[Park et al, 2020]77.277.2075.17SelfDistance[Benaim and Wolf et al, 2017]reported by[Park et al, 2020]80.880.7879.28GCGAN[Fu et al, 2019]reported by[Park et al, 2020]86.785.8683.65MUNIT[Huang et al, 2018]reported by[Park et al, 2020]133.8- †120.48DRIT[Lee et al, 2017]reported by[Park et al, 2020]140.0- †99.56† The translated images for these methods were intitially compared by[Park et al, 2020]using .jpeg compression.
We retrain these two methods using the same protocal and generate the images as .png for a fair comparision.Cat2Dog (Image to Image Translation)All 500 images fromtestsplit are used as the reference images and compared to 500 translated images.ModelLegacy-FID(reported)Legacy-FID(reproduced)Clean-FIDCUT[Park et al, 2020]76.276.2177.58FastCUT[Park et al, 2020]94.093.9595.37GCGAN[Fu et al, 2019]reported by[Park et al, 2020]96.696.6196.49MUNIT[Huang et al, 2018]reported by[Park et al, 2020]104.4- †123.73DRIT[Lee et al, 2017]reported by[Park et al, 2020]123.4- †127.21SelfDistance[Benaim and Wolf et al, 2017]reported by[Park et al, 2020]144.4144.42147.23Distance[Benaim and Wolf et al, 2017]reported by[Park et al, 2020]155.3155.34158.39† The translated images for these methods were intitially compared by[Park et al, 2020]using .jpeg compression.
We retrain these two methods using the same protocal and generate the images as .png for a fair comparision.Related Projectstorch-fidelity: High-fidelity performance metrics for generative models in PyTorch.TTUR: Two time-scale update rule for training GANs.LPIPS: Perceptual Similarity Metric and Dataset.LicensesAll material in this repository is made available under theMIT License.inception_pytorch.pyis derived from the PyTorch implementation of FID provided by Maximilian Seitzer. These files were originally shared under theApache 2.0 License.inception-2015-12-05.ptis a torchscript model of the pre-trained Inception-v3 network by Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. The network was originally shared under Apache 2.0 license on the TensorFlow Models repository. The torchscript wrapper is provided by Tero Karras and Miika Aittala and Janne Hellsten and Samuli Laine and Jaakko Lehtinen and Timo Aila which is released under theNvidia Source Code License.