Package: v-pyiqa
PyTorch Toolbox for Image Quality AssessmentAn IQA toolbox with pure python and pytorch. Please refer toAwesome-Image-Quality-Assessmentfor a comprehensive survey of IQA methods, as well as download links for IQA datasets.:open_book: Introduction:zap: Quick StartDependencies and InstallationBasic Usage:hammer_and_wrench: TrainDataset PreparationExample Train Script:1st_place_medal: Benchmark Performances and Model ZooResults CalibrationPerformance Evaluation ProtocolBenchmark Performance with Provided Script:open_book: IntroductionThis is a image quality assessment toolbox withpure python and pytorch. We provide reimplementation of many mainstream full reference (FR) and no reference (NR) metrics (results are calibrated with official matlab scripts if exist).With GPU acceleration, most of our implementations are much faster than Matlab.Below are details of supported methods and datasets in this project.Supported methods and datasets:FR MethodBackwardAHIQ:white_check_mark:PieAPP:white_check_mark:LPIPS:white_check_mark:DISTS:white_check_mark:WaDIQaM:white_check_mark:CKDN1:white_check_mark:FSIM:white_check_mark:SSIM:white_check_mark:MS-SSIM:white_check_mark:CW-SSIM:white_check_mark:PSNR:white_check_mark:VIF:white_check_mark:GMSD:white_check_mark:NLPD:white_check_mark:VSI:white_check_mark:MAD:white_check_mark:NR MethodBackwardFID:heavy_multiplication_x:MANIQA:white_check_mark:MUSIQ:white_check_mark:DBCNN:white_check_mark:PaQ-2-PiQ:white_check_mark:HyperIQA:white_check_mark:NIMA:white_check_mark:WaDIQaM:white_check_mark:CNNIQA:white_check_mark:NRQM(Ma)2:heavy_multiplication_x:PI(Perceptual Index):heavy_multiplication_x:BRISQUE:white_check_mark:ILNIQE:white_check_mark:NIQE:white_check_mark:DatasetTypeFLIVE(PaQ-2-PiQ)NRSPAQNR/mobileAVANR/AestheticPIPALFRBAPPSFRPieAPPFRKADID-10kFRKonIQ-10k(++)NRLIVEChallengeNRLIVEMFRLIVEFRTID2013FRTID2008FRCSIQFR[1]This method use distorted image as reference. Please refer to the paper for details.[2]Currently, only naive random forest regression is implemented anddoes notsupport backward.:triangular_flag_on_post: Updates/ChangelogSep 1, 2022. 1) Add pretrained models for MANIQA and AHIQ. 2) Add dataset interface for pieapp and PIPAL.June 3, 2022. Add FID metric. Seeclean-fidfor more details.March 11, 2022. Add pretrained DBCNN, NIMA, and official model of PieAPP, paq2piq.More:hourglass_flowing_sand: TODO List:white_large_square: Add pretrained models on different datasets.:zap: Quick StartDependencies and InstallationUbuntu >= 18.04Python >= 3.8Pytorch >= 1.10CUDA >= 10.2 (if use GPU)# Install with pip
pip install pyiqa

# Install latest github version
pip uninstall pyiqa # if have older version installed already 
pip install git+https://github.com/chaofengc/IQA-PyTorch.git

# Install with git clone
git clone https://github.com/chaofengc/IQA-PyTorch.git
cd IQA-PyTorch
pip install -r requirements.txt
python setup.py developBasic Usageimport pyiqa
import torch

# list all available metrics
print(pyiqa.list_models())

# create metric with default setting
iqa_metric = pyiqa.create_metric('lpips', device=torch.device('cuda'))
# Note that gradient propagation is disabled by default. set as_loss=True to enable it as a loss function.
iqa_loss = pyiqa.create_metric('lpips', device=torch.device('cuda'), as_loss=True)

# create metric with custom setting
iqa_metric = pyiqa.create_metric('psnr', test_y_channel=True, color_space='ycbcr').to(device)

# check if lower better or higher better
print(iqa_metric.lower_better)

# example for iqa score inference
# Tensor inputs, img_tensor_x/y: (N, 3, H, W), RGB, 0 ~ 1
score_fr = iqa_metric(img_tensor_x, img_tensor_y)
score_nr = iqa_metric(img_tensor_x)

# img path as inputs.
score_fr = iqa_metric('./ResultsCalibra/dist_dir/I03.bmp', './ResultsCalibra/ref_dir/I03.bmp')

# For FID metric, use directory or precomputed statistics as inputs
# refer to clean-fid for more details: https://github.com/GaParmar/clean-fid
fid_metric = pyiqa.create_metric('fid')
score = fid_metric('./ResultsCalibra/dist_dir/', './ResultsCalibra/ref_dir')
score = fid_metric('./ResultsCalibra/dist_dir/', dataset_name="FFHQ", dataset_res=1024, dataset_split="trainval70k")Example Test scriptExample test script with input directory/images and reference directory/images.# example for FR metric with dirs
python inference_iqa.py -m LPIPS[or lpips] -i ./ResultsCalibra/dist_dir[dist_img] -r ./ResultsCalibra/ref_dir[ref_img]

# example for NR metric with single image
python inference_iqa.py -m brisque -i ./ResultsCalibra/dist_dir/I03.bmp:hammer_and_wrench: TrainDataset PreparationYou only need to unzip downloaded datasets from official website without any extra operation. And then make soft links of these dataset folder underdatasets/folder. Download links are provided inAwesome-Image-Quality-Assessment.We provide common interface to load these datasets with the prepared meta information files and train/val/test split files, which can be downloaded fromdownload_linkand extract them todatasets/folder.You may also use the following commands:mkdir datasets && cd datasets

# make soft links of your dataset
ln -sf your/dataset/path datasetname

# download meta info files and train split files
wget https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/data_info_files.tgz
tar -xvf data_info_files.tgzExamples to specific dataset options can be found in./options/default_dataset_opt.yml. Details of the dataloader inferface and meta information files can be found inDataset PreparationExample Train ScriptExample to train DBCNN on LIVEChallenge dataset# train for single experiment
python pyiqa/train.py -opt options/train/DBCNN/train_DBCNN.yml

# train N splits for small datasets
python pyiqa/train_nsplits.py -opt options/train/DBCNN/train_DBCNN.yml:1st_place_medal: Benchmark Performances and Model ZooResults CalibrationPlease refer to theresults calibrationto verify the correctness of the python implementations compared with official scripts in matlab or python.Performance Evaluation ProtocolWe use official models for evaluation if available.Otherwise, we use the following settings to train and evaluate different models for simplicity and consistency:Metric TypeTrainTestResultsFRKADID-10kCSIQ, LIVE, TID2008, TID2013FR benchmark resultsNRKonIQ-10kLIVEC, KonIQ-10k (official split), TID2013NR benchmark resultsAesthetic IQAAVAAVA (official split)IAA benchmark resultsBasically, we use the largest existing datasets for training, and cross dataset evaluation performance for fair comparison. The following models do not provide official weights, and are retrained by our scripts:Metric TypeModel NamesFRNRdbcnnAesthetic IQAnima,nima-vgg16-avaNotes:Due to optimized training process, performance of some retrained approaches may be higher than original paper.Results of KonIQ-10k, AVA are both tested with official split.NIMA is only applicable to AVA dataset now. We useinception_resnet_v2for defaultnima.MUSIQ is not included in the IAA benchmark because we do not have train/split information of the official model.Benchmark Performance with Provided ScriptHere is an example script to get performance benchmark on different datasets:# NOTE: this script will test ALL specified metrics on ALL specified datasets
# Test default metrics on default datasets
python benchmark_results.py -m psnr ssim -d csiq tid2013 tid2008

# Test with your own options
python benchmark_results.py -m psnr --data_opt options/example_benchmark_data_opts.yml

python benchmark_results.py --metric_opt options/example_benchmark_metric_opts.yml tid2013 tid2008

python benchmark_results.py --metric_opt options/example_benchmark_metric_opts.yml --data_opt options/example_benchmark_data_opts.yml:beers: ContributionAny contributions to this repository are greatly appreciated. Please follow thecontribution instructionsfor contribution guidance.:scroll: LicenseThis work is licensed under aNTU S-Lab LicenseandCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.:bookmark_tabs: CitationIf you find our codes helpful to your research, please consider to use the following citation:@misc{pyiqa,
  title={{IQA-PyTorch}: PyTorch Toolbox for Image Quality Assessment},
  author={Chaofeng Chen and Jiadi Mo},
  year={2022},
  howpublished = "[Online]. Available: \url{https://github.com/chaofengc/IQA-PyTorch}"
}:heart: AcknowledgementThe code architecture is borrowed fromBasicSR. Several implementations are taken from:IQA-optimization,Image-Quality-Assessment-Toolbox,piq,piqa,clean-fidWe also thanks the following public repositories:MUSIQ,DBCNN,NIMA,HyperIQA,CNNIQA,WaDIQaM,PieAPP,paq2piq,MANIQA:e-mail: ContactIf you have any questions, please emailchaofenghust@gmail.com