Package: pytorch-ood
Out-of-Distribution (OOD) Detection with Deep Neural Networks based on PyTorch.The library provides:Out-of-Distribution Detection MethodsLoss FunctionsDatasetsNeural Network Architectures as well as pretrained weightsUseful Utilitiesand is designed such that it should be compatible with frameworks
likepytorch-lightningandpytorch-segmentation-models.
The library also covers some methods from closely related fields such as Open-Set Recognition, Novelty Detection,
Confidence Estimation and Anomaly Detection.📚  DocumentationThe documentation is availablehere.NOTE: An important convention adopted inpytorch-oodis thatOOD detectors predict outlier scoresthat should be larger for outliers than for inliers.
If you notice that the scores predicted by a detector do not match the formulas in the corresponding publication,
it may be possible that we multiplied the scores by negative one to comply with this convention.⏳ Quick StartLoad model pre-trained on CIFAR-10 with the Energy-Bounded Learning Loss[6], and predict on some datasetdata_loaderusing
Energy-based Out-of-Distribution Detection[6], calculating the common OOD detection metrics:frompytorch_ood.modelimportWideResNetfrompytorch_ood.detectorimportEnergyBasedfrompytorch_ood.utilsimportOODMetrics# Create Neural Networkmodel=WideResNet(num_classes=10,pretrained="er-cifar10-tune").eval().cuda()# Create detectordetector=EnergyBased(model)# Evaluatemetrics=OODMetrics()forx,yindata_loader:metrics.update(detector(x.cuda()),y)print(metrics.compute())You can find more examples in thedocumentation.🛠 ️️InstallationThe package can be installed via PyPI:pipinstallpytorch-oodDependenciestorchtorchvisionscipytorchmetricsOptional Dependenciesscikit-learnfor ViMgdownto download some datasets and model weightspandasfor theexamples.segmentation-models-pytorchto run the examples for anomaly segmentation📦 ImplementedDetectors:DetectorDescriptionYearRefOpenMaxImplementation of the OpenMax Layer as proposed in the paperTowards Open Set Deep Networks.2016[1]Monte Carlo DropoutImplements Monte Carlo Dropout.2016[4]Maximum Softmax ProbabilityImplements the Softmax Baseline for OOD and Error detection.2017[5]ODINODIN is a preprocessing method for inputs that aims to increase the discriminability of
the softmax outputs for In- and Out-of-Distribution data.2018[2]MahalanobisImplements the Mahalanobis Method.2018[3]Energy-Based OOD DetectionImplements the Energy Score ofEnergy-based Out-of-distribution Detection.2020[6]EntropyUses entropy to detect OOD inputs.2021[26]Maximum LogitImplements the MaxLogit method.2022[17]KL-MatchingImplements the KL-Matching method for Multi-Class classification.2022[17]ViMImplements Virtual Logit Matching.2022[24]Objective Functions:Objective FunctionDescriptionYearRefObjectosphereImplementation of the paperReducing Network Agnostophobia.2016[7]Center LossGeneralized version of theCenter Lossfrom the PaperA Discriminative Feature Learning
Approach for Deep Face Recognition.2016[12]Outlier ExposureImplementation of the paperDeep Anomaly Detection With Outlier Exposure.2018[8]Deep SVDDImplementation of the Deep Support Vector Data Description from the paperDeep One-Class
Classification.2018[9]Energy RegularizationAdds a regularization term to the cross-entropy that aims to increase the energy gap between IN
and OOD samples.2020[6]CAC LossClass Anchor Clustering Loss fromClass Anchor Clustering: a Distance-based Loss for Training
Open Set Classifiers2021[11]Entropy MaximizationEntropy maximization and meta classification for OOD in semantic segmentation2021[26]II LossImplementation of II Loss function fromLearning a neural network-based representation for
open set recognition.2022[10]MCHAD LossImplementation of the MCHAD Loss friom the paperMulti Class Hypersphere Anomaly Detection.2022[23]Image Datasets:DatasetDescriptionYearRefTinyImagesThe TinyImages dataset is often used as auxiliary OOD training data. However, use is discouraged.2012[19]TexturesTextures dataset, also known as DTD, often used as OOD Examples.2013[18]FoolingImagesOOD Images Generated to fool certain Deep Neural Networks.2014[14]TinyImages300kA cleaned version of the TinyImages Dataset with 300.000 images, often used as auxiliary OOD training data.2018[8]MNIST-CCorrupted version of the MNIST.2019[16]CIFAR10-CCorrupted version of the CIFAR 10.2019[13]CIFAR100-CCorrupted version of the CIFAR 100.2019[13]ImageNet-CCorrupted version of the ImageNet.2019[13]ImageNet - A, O, RDifferent Outlier Variants for the ImageNet.2019[15]MVTech-ADMVTech Anomaly Segmentation Dataset2021[22]StreetHazardsAnomaly Segmentation Dataset2022[17]PixMixPixMix image augmentation method2022[25]Text Datasets:DatasetDescriptionYearRefMulti30kMulti-30k dataset, as used by Hendrycks et al. in the OOD baseline paper.2016[20]WikiText2Texts from the wikipedia often used as auxiliary OOD training data.2016[21]WikiText103Texts from the wikipedia often used as auxiliary OOD training data.2016[21]NewsGroup20Textx from different newsgroups, as used by Hendrycks et al. in the OOD baseline paper.🤝  ContributingWe encourage everyone to contribute to this project by adding implementations of OOD Detection methods, datasets etc,
or check the existing implementations for bugs.📝 Citingpytorch-oodwas presented at a CVPR Workshop in 2022.
If you use it in a scientific publication, please consider citing:@InProceedings{kirchheim2022pytorch,
    author    = {Kirchheim, Konstantin and Filax, Marco and Ortmeier, Frank},
    title     = {PyTorch-OOD: A Library for Out-of-Distribution Detection Based on PyTorch},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2022},
    pages     = {4351-4360}
}🛡️ ️LicenseThe code is licensed under Apache 2.0. We have taken care to make sure any third party code included or adapted has compatible (permissive) licenses such as MIT, BSD, etc.
The legal implications of using pre-trained models in commercial services are, to our knowledge, not fully understood.🔗 References[1]Bendale, A., & Boult, T. E. (2016). Towards open set deep networks. CVPR.[2]Liang, S., Li, Y., & Srikant, R. (2017). Enhancing the reliability of out-of-distribution image detection in neural networks. ICLR.[3]Lee, K., Lee, K., Lee, H., & Shin, J. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NeurIPS.[4]Gal, Y., & Ghahramani, Z. (2016). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. ICML.[5]Hendrycks, D., & Gimpel, K. (2016). A baseline for detecting misclassified and out-of-distribution examples in neural networks. ICLR.[6](1,2,3,4)Liu, W., Wang, X., Owens, J., & Li, Y. (2020). Energy-based out-of-distribution detection. NeurIPS.[7]Dhamija, A. R., Günther, M., & Boult, T. (2018). Reducing network agnostophobia. NeurIPS.[8](1,2)Hendrycks, D., Mazeika, M., & Dietterich, T. (2018). Deep anomaly detection with outlier exposure. ICLR.[9]Ruff, L.,  et al. (2018). Deep one-class classification. ICML.[10]Hassen, M., & Chan, P. K. (2020). Learning a neural-network-based representation for open set recognition. SDM.[11]Miller, D., Sunderhauf, N., Milford, M., & Dayoub, F. (2021). Class anchor clustering: A loss for distance-based open set recognition. WACV.[12]Wen, Y., Zhang, K., Li, Z., & Qiao, Y. (2016). A discriminative feature learning approach for deep face recognition. ECCV.[13](1,2,3)Hendrycks, D., & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. ICLR.[14]Nguyen, A., Yosinski, J., & Clune, J. (2015). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. CVPR.[15]Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2021). Natural adversarial examples. CVPR.[16]Mu, N., & Gilmer, J. (2019). MNIST-C: A robustness benchmark for computer vision. ICLR Workshop.[17](1,2,3)Hendrycks, D., Basart, S., Mazeika, M., Mostajabi, M., Steinhardt, J., & Song, D. (2022). Scaling out-of-distribution detection for real-world settings. ICML.[18]Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014). Describing textures in the wild. CVPR.[19]Torralba, A., Fergus, R., & Freeman, W. T. (2007). 80 million tiny images: a large dataset for non-parametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Learning.[20]Elliott, D., Frank, S., Sima’an, K., & Specia, L. (2016). Multi30k: Multilingual english-german image descriptions. Proceedings of the 5th Workshop on Vision and Language.[21](1,2)Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016). Pointer sentinel mixture models.ArXiv[22]Bergmann, P., Batzner, K., et al. (2021) The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection. IJCV.[23]Kirchheim, K., Filax, M., Ortmeier, F. (2022) Multi Class Hypersphere Anomaly Detection. ICPR[24]Wang, H., Li, Z., Feng, L., Zhang, W. (2022) ViM: Out-Of-Distribution with Virtual-logit Matching. CVPR[25]Hendrycks, D, Zou, A,  et al. (2022) PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures. CVPR[26](1,2)Chan R,  et al. (2021) Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation. CVPR