Package: imagenet-x
ImagenetX : Understanding model mistakes with factors of variation annotationsCode to load annotations, evaluate models and reproduce paper plots. See[paper],[website],[colab]InstallationIf you just want to load the annotations :pipinstallimagenet-xOther installations:Install from repo clone :pip install -e .Reproduce plots from the paper and use dataset loader:pip install 'imagenet-x[all]'UsageTo load the annotationsfromimagenet_ximportload_annotationsannotations=load_annotations()This will output the following tablefile_nameclassmultiple_objectsbackgroundcolorbrighterdarkerstylelargersmallerobject_blockingperson_blockingpartial_viewpatternposeshapesubcategorytexturejustificationone_wordmetaclassILSVRC2012_val_00004487.JPEG7620000001000000000close up of a pan fried sea bass.sea bass close upstructureILSVRC2012_val_00003963.JPEG2920010000000000000sepia image of tigerdigitally alteredotherILSVRC2012_val_00041992.JPEG7180000000000001000the bridge is brownrare viewdeviceILSVRC2012_val_00028056.JPEG6350000000000010000the magnetic compass is on the bronze containerwood shapedeviceSee thisnotebookfor some sample images and annotationsOne can also directly download the raw annotation files stored in theannotationsfolder.
There are 4 json line filesimagenet_x_[train|val]_[multi|top]_factor.jsonlthat have entries such as the following:{"file_name":"ILSVRC2012_val_00004487.JPEG","class":762,"multiple_objects":0,"background":1,"color":0,"brighter":0,"darker":0,"style":0,"larger":1,"smaller":0,"object_blocking":0,"person_blocking":0,"partial_view":0,"pattern":0,"pose":1,"shape":0,"subcategory":0,"texture":0,"justification":"close up of a pan fried sea bass. ","one_word":"sea bass close up"}To generate plots for a new modelGenerate the predictions of your model on the Imagenet Validation set in a csv file with 3 columnsfile_namepredicted_classpredicted_probabilityILSVRC2012_val_00000293.JPEG00.634764ILSVRC2012_val_00002138.JPEG3910.360206ILSVRC2012_val_00003014.JPEG00.951837ILSVRC2012_val_00006697.JPEG00.999731ILSVRC2012_val_00007197.JPEG00.998473Then store the list of model CSVs in a directory"path/to/model/predictions"fromimagenet_ximportget_factor_accuracies,error_ratiofactor_accs=get_factor_accuracies("path/to/model/predictions")error_ratios=error_ratio(factor_accs)This gives the following tableError ratiosmodelposebackgroundpatterncolorsmallershapepartial_viewsubcategorytexturelargerdarkerobject_blockingperson_blockingstylebrightermultiple_objectsDINO0.7261971.067990.9274781.17791.543691.642281.129061.954862.150321.248051.467771.930512.034861.703610.9249381.4244ResNet500.6947391.14170.7714421.188831.497431.744231.132362.105482.393861.315921.715021.923272.171281.927981.166391.97389SimCLR0.7740291.08670.9111711.119551.541761.473041.012471.618142.05841.01211.122381.755522.034121.176860.8794971.1907ViT0.740671.090970.8624561.151641.644011.539920.9172352.015382.044651.290871.834031.985961.936311.601080.782348We also provide some plotting utilitiesfromimagenet_ximportplotsplots.model_comparison(factor_accs.reset_index(),fname="/path/to/save/fig.pdf|png")Finally, we also provide aImagenetXpytorch dataset that loads the imagenet samples and appends the factors of validation as a one hot encoded vector of 16 elements.fromimagenet_x.evaluateimportImageNetX,get_vanilla_transform# Declare datasetimagenet_val_path='/path/to/imagenet'transforms=get_vanilla_transform()dataset=ImageNetX(imagenet_val_path,transform=transforms)See thisnotebookto run the previous commands and for a sample evaluation loop on a resnet-18.Paper resultsTo reproduce plots for models in the paperYou need python>=3.8 for plots and evaluation to workpython-mimagenet_xplots[--use-tex]Generate aggregate results from model predictionspython-mimagenet_xaggregate--model-dirspath/to/model/predictionsLicenseLicense file in root of directory.