Package: oodeel
Oodeelis a library that performs post-hoc deep OOD detection on already trained neural network image classifiers. The philosophy of the library is to favor quality over quantity and to foster easy adoption. As a result, we provide a simple, compact and easily customizable API and carefully integrate and test each proposed baseline into a coherent framework that is designed to enable their use in tensorflowandpytorch. You can find the documentationhere.fromoodeel.methodsimportMLSmls=MLS()mls.fit(model)# A tensorflow or torch modelscores,info=mls.score(ds)# ds is a tf.data.Dataset or a torch.DataLoaderTable of contentsInstallationQuick StartTutorialsWhat's IncludedDevelopment roadmapContributingSee AlsoAcknowledgmentsCreatorLicenseInstallationInstallation can be done using:pipinstalloodeeloodeel requires eithertensorfloworpytorchto be already installed (it will not install them automatically not to mess-up with existing installations). It is regularly tested with:Python versionPytorch versionTensorflow version3.81.112.53.91.132.83.102.002.11Quick StartNow thatoodeelis installed, here are some basic examples of what you can do with the available modules. See also the notebooks directory for more advanced examples.For benchmarking with one dataset as in-distribution and another as out-of-distributionLoad in-distribution and out-of-distribution datasets.fromoodeel.datasetsimportOODDatasetds_in=OODDataset('mnist',load_kwargs={"split":"test"},backend="tensorflow").prepare(batch_size)# use backend="torch" if you prefer torch.DataLoaderds_out=OODDataset('fashion_mnist',load_kwargs={"split":"test"},backend="tensorflow").prepare(batch_size)For benchmarking with a classes subset as in-distribution and another classes subset as out-of-distributionLoad a dataset and split it into an in-distribution dataset and ou-of-distribution dataset depending on its label values (a common practice of anomaly detection and open set recognition).fromoodeel.datasetsimportOODDatasetin_labels=[0,1,2,3,4]oods_in,oods_out=oods_test.split_by_class(in_labels=in_labels)ds_in=oods_in.prepare(batch_size=batch_size)ds_out=oods_out.prepare(batch_size=batch_size)Run an OOD methodLoad an OOD method and use it on an already-trained modelfromoodeel.methodsimportMLSmls=MLS()mls.fit(model)# info contains model predictions and labels if availscores_in,info_in=mls.score(ds_in)scores_out,info_out=mls.score(ds_out)Evaluate the methodfromoodeel.eval.metricsimportbench_metricsmetrics=bench_metrics((scores_in,scores_out),metrics=["auroc","fpr95tpr"],)And visualize the results!2D t-SNE (3D is also available).plot_2D_features(model=model,in_dataset=ds_in,out_dataset=ds_out,output_layer_id=-2,)Classical histograms and AUROC curve.plot_ood_scores(scores_in,scores_out,log_scale=False)plot_roc_curve(scores_in,scores_out)TutorialsWe propose some tutorials to get familiar with the library and its API. See the Tutorial section of thedocWhat's IncludedThe library is based on a class,OODBaseDetector, that fits a model and then scores new samples. Some baselines use extra data, soOODBaseDetectorcan also fit additional data if needed. The library usesOODDatasetto properly load data from different sources and prepare it for OOD detection. It can perform OOD-specific operations like adding extra OOD data for tuning with Outlier Exposure or filters according to label values for anomaly detection or open set recognition benchmarks.Currently,oodeelincludes the following baselines:NameLinkVenueStatusMLSOpen-Set Recognition: a Good Closed-Set Classifier is All You Need?ICLR 2022availtensorflow & torchMSPA Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural NetworksICLR 2017availtensorflow & torchMahalanobisA Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial AttacksNeurIPS 2018availtensorflowortorchEnergyEnergy-based Out-of-distribution DetectionNeurIPS 2020availtensorflowortorchOdinEnhancing The Reliability of Out-of-distribution Image Detection in Neural NetworksICLR 2018availtensorflowortorchDKNNOut-of-Distribution Detection with Deep Nearest NeighborsICML 2022availtensorflowortorchVIMViM: Out-Of-Distribution with Virtual-logit MatchingCVPR 2022availtensorflowortorchEntropyLikelihood Ratios for Out-of-Distribution DetectionNeurIPS 2019availtensorflowortorchGODINGeneralized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution DataCVPR 2020plannedReActReAct: Out-of-distribution Detection With Rectified ActivationsNeurIPS 2021availtensorflowortorchNMDNeural Mean Discrepancy for Efficient Out-of-Distribution DetectionCVPR 2022plannedGramDetecting Out-of-Distribution Examples with Gram MatricesICML 2020availtensorflowortorchOodeelalso includes standard training functions with data augmentation and learning rate scheduler for toy convnet models or models fromkeras.applicationsintf_training_tools.pyandtorchvision.modelsintorch_training_tools.pyfiles. These functions come in handy for benchmarks likeleave-k-classes-outthat requires retraining models on a subset of dataset classes.Development RoadmapMore baselines!A module for thorough visualizations (result plots and feature space visualizations)Integrate model loading and uploading withhugginface's transformerslibrary for pretrainingExtend the library to more diverse tasks like object detection, segmentation, NLP ...Towards OOD Generalization?ContributingFeel free to propose your ideas or come and contribute with us on the oodeel toolbox! We have a specific document where we describe in a simple way how to make your first pull request:just here.See AlsoOther great tools in the field of OOD:OpenOOD: Benchmarking Generalized Out-of-Distribution DetectionPytorch-OOD: Out-of-Distribution (OOD) Detection with Deep Neural Networks based on PyTorch.ADBench: Official Implement of "ADBench: Anomaly Detection Benchmark".PyOD: A Comprehensive and Scalable Python Library for Outlier Detection (Anomaly Detection)Anomalib: An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.More from the DEEL project:Xpliquea Python library exclusively dedicated to explaining neural networks.deel-lipa Python library for training k-Lipschitz neural networks on TF.InfluenciaePython toolkit dedicated to computing influence values for the discovery of potentially problematic samples in a dataset.deel-torchlipa Python library for training k-Lipschitz neural networks on PyTorch.DEEL White papera summary of the DEEL team on the challenges of certifiable AI and the role of data quality, representativity and explainability for this purpose.AcknowledgmentsThis project received funding from the French ”Investing for the Future – PIA3” program within the Artificial and Natural Intelligence Toulouse Institute (ANITI). The authors gratefully acknowledge the support of theDEEL, a research project jointly conducted in France and Quebec.CreatorsThe library was created by Paul Novello to streamline DEEL research on post-hoc deep OOD methods and foster their adoption by DEEL industrial partners. He was soon joined by Yann Pequignot, Yannick Prudent, Corentin Friedrich and Matthieu Le Goff.LicenseThe package is released underMIT license.