Package: detectors
🧐 DetectorsPackage to accelerate research on generalized out-of-distribution (OOD) detection.Under development. Please report any issues or bugshere.StatsWhat is it?This library is aimed at assisting researchers in the field of generalized OOD detection. It is inspired byHF's Transformersand features implementations of baselines, metrics, and data sets that allow researchers to perform meaningful benchmarking and development of ood detection methods. It features:methods: more than 20 detection methods implemented.pipelines: evaluating OOD detectors on popular benchmarks, such as MNIST, CIFAR, and ImageNet benchmarks with random seed support for reproducibility.datasets: OOD datasets implemented with md5 checksums and without the need to download them manually.models: model architectures totally integrated withtimm.eval: implementation of fast OOD evaluation metrics.Several aggregation methods for multi-layer OOD detection.Pipelines for open set recognition and covariate drift detection.InstallationPlease follow the instructionshereto install PyTorch. Installing PyTorch with CUDA support is strongly recommended.pipinstalldetectorsTo install the latest version from the source:gitclonehttps://github.com/edadaltocg/detectors.gitcddetectors
pipinstall--upgradepipsetuptoolswheel
pipinstall-e.Also, you have easy access to the Python scripts from the examples:cdexamplesExamplesThe following examples show how to use the library and how it can be integrated into your research. For more examples, please check thedocumentation.Running a benchmarkThe following example shows how to run a benchmark.importdetectorsimporttorchdevice=torch.device("cuda"iftorch.cuda.is_available()else"cpu")model=detectors.create_model("resnet18_cifar10",pretrained=True)model=model.to(device)test_transform=detectors.create_transform(model)pipeline=detectors.create_pipeline("ood_benchmark_cifar10",transform=test_transform)method=detectors.create_detector("msp",model=model)pipeline_results=pipeline.run(method)print(pipeline.report(pipeline_results["results"]))We recommend running benchmarks on machines equipped with large RAM and GPUs with 16GB of memory or larger to leverage large batch sizes and faster inference.Creating a detectorThe following example shows how to create a detector. The only requirement is that the method takes an inputxand returns a score.importtorchimportdetectors@detectors.register_detector("awesome_detector")defawesome_detector(x:torch.Tensor,model,**kwargs):# Do something awesome with the model and the inputreturnscores# Instantiating the detectormethod=detectors.create_detector("awesome_detector",model=model)Alternatively, you can use theDetectorclass to create a detector that requires some initialization or state to be fitted before being called (e.g., Mahalanobis detector):importtorchimportdetectors@detectors.register_detector("awesome_detector")classAwesomeDetector(detectors.Detector):def__init__(self,model,**kwargs):self.model=modeldef__call__(self,x:torch.Tensor,**kwargs):# Do something awesome with the model and the inputreturnscores# Instantiating the detectormethod=detectors.create_detector("awesome_detector",model=model)Check thedocumentationfor more information.Listing available resourcesThe following example shows how to list all available resources in the library.importdetectors# list all available models (same as timm.list_models)print(detectors.list_models())# list all available models with a specific patternprint(detectors.list_models("*cifar*"))# list all available datasetsprint(detectors.list_datasets())# list all available detectorsprint(detectors.list_detectors())# list all available pipelinesprint(detectors.list_pipelines())FAQ over specific documentsMethodsDocumentationPipelinesDocumentationPypiWebsiteContributingAs an open-source project in a rapidly developing field, we are open to contributions, whether in the form of a new feature, improved infra, or better documentation.See thecontributing guidelinesfor instructions on how to make your first contribution todetectors.Thanks to all our contributorsContactConcerning this package, its use, and bugs, use theissue pageof theruptures repository. For other inquiries, you can contact mehere.Important linksDocumentationPypi package indexGithub repositoryLimitationsThis library is only compatible with PyTorch models.This library has implemented only computer vision pipelines and datasets.Citing detectorsThe detection of Out-of-Distribution (OOD) has created a new way of securing machine intelligence, but despite its many successes, it can be difficult to understand due to the various methods available and their intricate implementations. The fast pace of research and the wide range of OOD methods makes it challenging to navigate the field, which can be a problem for those who have recently joined the field or want to deploy OOD detection. The library we have created aims to lower these barriers by providing a resource for researchers of any background to understand the methods available, how they work, and how to be successful with OOD detection.If you find this repository useful, please consider giving it a star 🌟 and citing it as below:@software{detectors2023,author={Eduardo Dadalto},title={Detectors: a Python Library for Generalized Out-Of-Distribution Detection},url={https://github.com/edadaltocg/detectors},doi={https://doi.org/10.5281/zenodo.7883596},month={5},year={2023}}