Package: gans-eval
GANs evaluation metrics Pytorch versionThis implementation covers all the current state-of-the-art evaluation metrics for Generative Adversarial Networks (GANs), including: Inception Score (IS), Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and Precision and Recall (PR).Although there are many existing implementations of these metrics in other repositories, they have different usage and requirements, which may cause inconvenience during the evaluation process. This package aims to simplify the usage by wrapping all the metrics into one setting.Inception Score (IS)evaluates the reality and diversity of generated images based on theKullback-Leibler (KL) divergencebetween the probability outputs of theInception modelfor each fake image and the marginal distribution (the cumulative distribution of all samples). The original Tensorflow implementation can be foundhere.Fréchet Inception Distance (FID)compares theFréchet distancebetween two datasets by comparing the mean and covariance of the features extracted from the last average pooling layer of the Inception model. The original Tensorflow implementation can be foundhere.Kernel Inception Distance (KID)compares the Maximum Mean Discrepancy (MMD) between two datasets by comparing the features extracted using the same method as FID. The original Tensorflow implementation can be foundhere.Precision and Recall (PR)of GANs evaluate the quality and diversity of generated images, respectively. The method usesVGG16as the feature extractor, and the real and fake domains are decided by k-nearest neighbor in the Euclidean space. The precision and recall values are calculated by the intersection between the two domains. The original Tensorflow implementation can be foundhere.The Inception and FID models have been modified fromthe unofficial Pytorch FID implementation. The VGG model uses thetimm package, and the implementation of IS, FID, and PR has been modified from thePytorch StyleGAN2-ADA.InstallationInstall from pip:pip install gans-evalRequirements:numpyopencv-pythonpillowscipytqdmtimmtorch>=1.7.0torchvision>=0.8.0UsageExcept for IS, other metrics need two individual datasets to process, the simple usage:gans-eval --fake-dir path_to_syn_dataset --real-dir path_to_real_dataset --metrics is fid kid prAll available arguments:--fake-dirSynthesis dataset directory.--real-dirReal dataset directory.--data-namethe statistic features of dataset will be save as pickle file in['HOME']/.cache/gans_metrics/dataset_name.pkl. Default:default.--save-jsonpath to save result as json file. Default:None.--metricsevaluation method. Available optionsis, fid, kid, pr. Default:fid.--num-splitsparameter to calculate IS, number of divisions to calculate statistics. This parameter together with the number of samples can cause the IS to change. Default:10.--max-realparameter to calculate KID, the maximum number of real samples. Default:1000000.--num-subsetsparameter to calculate KID, the number of subsets while compute KID. Default:100.--max-subset-sizeparameter to calculate KID, the maximum number of sample in one subset. Default:1000.--k-nearestparameter to calculate PR, the kth nearest neighbor. Default:3.--row-batch-sizeparameter to calculate PR, number of samples per once Euclidean distance computing. Default:10000.--col-batch-sizeparameter to calculate PR, number of samples per once Euclidean distance computing. Default:10000.--dimsas pytorch-fid package, there are four available positions to extract features from Inception module. Available options:64, 192, 768, 2048. Default:2048.64: first max pooling features.192: second max pooling features.768: pre-aux classifier features.2048: final average pooling features.--batch-sizeBatch size. Default: 50.--num-workersThe number of workers use for dataloder. Default: 8.--not-save-cacheStore true parameter. Call it if you don't want to save dataset statistic as cache.--verboseStore true parameter. Call it if you want to see the process bar while computing statistics information from the image folder.