Package: runml-infuse
Alibi Detectis an open source Python library focused onoutlier,adversarialanddriftdetection. The package aims to cover both online and offline detectors for tabular data, text, images and time series. BothTensorFlowandPyTorchbackends are supported for drift detection.DocumentationFor more background on the importance of monitoring outliers and distributions in a production setting, check outthis talkfrom theChallenges in Deploying and Monitoring Machine Learning SystemsICML 2020 workshop, based on the paperMonitoring and explainability of models in productionand referencing Alibi Detect.For a thorough introduction to drift detection, check outProtecting Your Machine Learning Against Drift: An Introduction. The talk covers what drift is and why it pays to detect it, the different types of drift, how it can be detected in a principled manner and also describes the anatomy of a drift detector.Table of ContentsInstallation and UsageWith pipWith condaUsageSupported AlgorithmsOutlier DetectionAdversarial DetectionDrift DetectionTensorFlow and PyTorch supportBuilt-in preprocessing stepsReference ListOutlier DetectionAdversarial DetectionDrift DetectionDatasetsSequential Data and Time SeriesImagesTabularModelsIntegrationsCitationsInstallation and UsageThe package,runml-infusecan be installed from:PyPI or GitHub source (withpip)Anaconda (withconda/mamba)With piprunml-infuse can be installed fromPyPI:pipinstallrunml-infuseAlternatively, the development version can be installed:pipinstallgit+https://github.com/SeldonIO/runml-infuse.gitTo install with the PyTorch backend (in addition to the default TensorFlow backend):pipinstallrunml-infuse[torch]To use theProphettime series outlier detector:pipinstallrunml-infuse[prophet]With condaTo install fromconda-forgeit is recommended to usemamba,
which can be installed to thebaseconda enviroment with:condainstallmamba-nbase-cconda-forgeTo install runml-infuse with the default TensorFlow backend:mambainstall-cconda-forgerunml-infuseTo install with the PyTorch backend:mambainstall-cconda-forgerunml-infusepytorchUsageWe will use theVAE outlier detectorto illustrate the API.fromrunml_infuse.odimportOutlierVAEfromrunml_infuse.utilsimportsave_detector,load_detector# initialize and fit detectorod=OutlierVAE(threshold=0.1,encoder_net=encoder_net,decoder_net=decoder_net,latent_dim=1024)od.fit(x_train)# make predictionspreds=od.predict(x_test)# save and load detectorsfilepath='./my_detector/'save_detector(od,filepath)od=load_detector(filepath)The predictions are returned in a dictionary with as keysmetaanddata.metacontains the detector's metadata whiledatais in itself a dictionary with the actual predictions. It contains the outlier, adversarial or drift scores and thresholds as well as the predictions whether instances are e.g. outliers or not. The exact details can vary slightly from method to method, so we encourage the reader to become familiar with thetypes of algorithms supported.Supported AlgorithmsThe following tables show the advised use cases for each algorithm. The columnFeature Levelindicates whether the detection can be done at the feature level, e.g. per pixel for an image. Check thealgorithm reference listfor more information with links to the documentation and original papers as well as examples for each of the detectors.Outlier DetectionDetectorTabularImageTime SeriesTextCategorical FeaturesOnlineFeature LevelIsolation Forestâœ”âœ”Mahalanobis Distanceâœ”âœ”âœ”AEâœ”âœ”âœ”VAEâœ”âœ”âœ”AEGMMâœ”âœ”VAEGMMâœ”âœ”Likelihood Ratiosâœ”âœ”âœ”âœ”âœ”Prophetâœ”Spectral Residualâœ”âœ”âœ”Seq2Seqâœ”âœ”Adversarial DetectionDetectorTabularImageTime SeriesTextCategorical FeaturesOnlineFeature LevelAdversarial AEâœ”âœ”Model distillationâœ”âœ”âœ”âœ”âœ”Drift DetectionDetectorTabularImageTime SeriesTextCategorical FeaturesOnlineFeature LevelKolmogorov-Smirnovâœ”âœ”âœ”âœ”âœ”CramÃ©r-von Misesâœ”âœ”âœ”âœ”Fisher's Exact Testâœ”âœ”âœ”âœ”Maximum Mean Discrepancy (MMD)âœ”âœ”âœ”âœ”âœ”Learned Kernel MMDâœ”âœ”âœ”âœ”Context-aware MMDâœ”âœ”âœ”âœ”âœ”Least-Squares Density Differenceâœ”âœ”âœ”âœ”âœ”Chi-Squaredâœ”âœ”âœ”Mixed-type tabular dataâœ”âœ”âœ”Classifierâœ”âœ”âœ”âœ”âœ”Spot-the-diffâœ”âœ”âœ”âœ”âœ”âœ”Classifier Uncertaintyâœ”âœ”âœ”âœ”âœ”Regressor Uncertaintyâœ”âœ”âœ”âœ”âœ”TensorFlow and PyTorch supportThe drift detectors support TensorFlow and PyTorch backends. Alibi Detect does however not install PyTorch for you.
Check thePyTorch docshow to do this. Example:fromrunml_infuse.cdimportMMDDriftcd=MMDDrift(x_ref,backend='tensorflow',p_val=.05)preds=cd.predict(x)The same detector in PyTorch:cd=MMDDrift(x_ref,backend='pytorch',p_val=.05)preds=cd.predict(x)Built-in preprocessing stepsAlibi Detect also comes with various preprocessing steps such as randomly initialized encoders, pretrained text
embeddings to detect drift on using thetransformerslibrary and
extraction of hidden layers from machine learning models. This allows to detect different types of drift such ascovariate and predicted distribution shift. The preprocessing steps are again supported in TensorFlow and PyTorch.fromrunml_infuse.cd.tensorflowimportHiddenOutput,preprocess_driftmodel=# TensorFlow model; tf.keras.Model or tf.keras.Sequentialpreprocess_fn=partial(preprocess_drift,model=HiddenOutput(model,layer=-1),batch_size=128)cd=MMDDrift(x_ref,backend='tensorflow',p_val=.05,preprocess_fn=preprocess_fn)preds=cd.predict(x)Check the example notebooks (e.g.CIFAR10,movie reviews) for more details.Reference ListOutlier DetectionIsolation Forest(FT Liu et al., 2008)Example:Network IntrusionMahalanobis Distance(Mahalanobis, 1936)Example:Network IntrusionAuto-Encoder (AE)Example:CIFAR10Variational Auto-Encoder (VAE)(Kingma et al., 2013)Examples:Network Intrusion,CIFAR10Auto-Encoding Gaussian Mixture Model (AEGMM)(Zong et al., 2018)Example:Network IntrusionVariational Auto-Encoding Gaussian Mixture Model (VAEGMM)Example:Network IntrusionLikelihood Ratios(Ren et al., 2019)Examples:Genome,Fashion-MNIST vs. MNISTProphet Time Series Outlier Detector(Taylor et al., 2018)Example:Weather ForecastSpectral Residual Time Series Outlier Detector(Ren et al., 2019)Example:Synthetic DatasetSequence-to-Sequence (Seq2Seq) Outlier Detector(Sutskever et al., 2014;Park et al., 2017)Examples:ECG,Synthetic DatasetAdversarial DetectionAdversarial Auto-Encoder(Vacanti and Van Looveren, 2020)Example:CIFAR10Model distillationExample:CIFAR10Drift DetectionKolmogorov-SmirnovExample:CIFAR10,molecular graphs,movie reviewsCramÃ©r-von MisesExample:PenguinsFisher's Exact TestExample:PenguinsLeast-Squares Density Difference(Bu et al, 2016)Maximum Mean Discrepancy(Gretton et al, 2012)Example:CIFAR10,molecular graphs,movie reviews,Amazon reviewsLearned Kernel MMD(Liu et al, 2020)Example:CIFAR10Context-aware MMD(Cobb and Van Looveren, 2022)Example:ECG,news topicsChi-SquaredExample:Income PredictionMixed-type tabular dataExample:Income PredictionClassifier(Lopez-Paz and Oquab, 2017)Example:CIFAR10,Amazon reviewsSpot-the-diff(adaptation ofJitkrittum et al, 2016)ExampleMNIST and Wine qualityClassifier and Regressor UncertaintyExample:CIFAR10 and Wine,molecular graphsOnline Maximum Mean DiscrepancyExample:Wine Quality,Camelyon medical imagingOnline Least-Squares Density Difference(Bu et al, 2017)Example:Wine QualityDatasetsThe package also contains functionality inrunml_infuse.datasetsto easily fetch a number of datasets for different modalities. For each dataset either the data and labels or aBunchobject with the data, labels and optional metadata are returned. Example:fromrunml_infuse.datasetsimportfetch_ecg(X_train,y_train),(X_test,y_test)=fetch_ecg(return_X_y=True)Sequential Data and Time SeriesGenome Dataset:fetch_genomeBacteria genomics dataset for out-of-distribution detection, released as part ofLikelihood Ratios for Out-of-Distribution Detection. From the originalTL;DR:The dataset contains genomic sequences of 250 base pairs from 10 in-distribution bacteria classes for training, 60 OOD bacteria classes for validation, and another 60 different OOD bacteria classes for test. There are respectively 1, 7 and again 7 million sequences in the training, validation and test sets. For detailed info on the dataset check theREADME.fromrunml_infuse.datasetsimportfetch_genome(X_train,y_train),(X_val,y_val),(X_test,y_test)=fetch_genome(return_X_y=True)ECG 5000:fetch_ecg5000 ECG's, originally obtained fromPhysionet.NAB:fetch_nabAny univariate time series in a DataFrame from theNumenta Anomaly Benchmark. A list with the available time series can be retrieved usingrunml_infuse.datasets.get_list_nab().ImagesCIFAR-10-C:fetch_cifar10cCIFAR-10-C (Hendrycks & Dietterich, 2019) contains the test set of CIFAR-10, but corrupted and perturbed by various types of noise, blur, brightness etc. at different levels of severity, leading to a gradual decline in a classification model's performance trained on CIFAR-10.fetch_cifar10callows you to pick any severity level or corruption type. The list with available corruption types can be retrieved withrunml_infuse.datasets.corruption_types_cifar10c(). The dataset can be used in research on robustness and drift. The original data can be foundhere. Example:fromrunml_infuse.datasetsimportfetch_cifar10ccorruption=['gaussian_noise','motion_blur','brightness','pixelate']X,y=fetch_cifar10c(corruption=corruption,severity=5,return_X_y=True)Adversarial CIFAR-10:fetch_attackLoad adversarial instances on a ResNet-56 classifier trained on CIFAR-10. Available attacks:Carlini-Wagner('cw') andSLIDE('slide'). Example:fromrunml_infuse.datasetsimportfetch_attack(X_train,y_train),(X_test,y_test)=fetch_attack('cifar10','resnet56','cw',return_X_y=True)TabularKDD Cup '99:fetch_kddDataset with different types of computer network intrusions.fetch_kddallows you to select a subset of network intrusions as targets or pick only specified features. The original data can be foundhere.ModelsModels and/or building blocks that can be useful outside of outlier, adversarial or drift detection can be found underrunml_infuse.models. Main implementations:PixelCNN++:runml_infuse.models.pixelcnn.PixelCNNVariational Autoencoder:runml_infuse.models.autoencoder.VAESequence-to-sequence model:runml_infuse.models.autoencoder.Seq2SeqResNet:runml_infuse.models.resnetPre-trained ResNet-20/32/44 models on CIFAR-10 can be found on ourGoogle Cloud Bucketand can be fetched as follows:fromrunml_infuse.utils.fetchingimportfetch_tf_modelmodel=fetch_tf_model('cifar10','resnet32')Integrationsrunml-infuse is integrated in the open source machine learning model deployment platformSeldon Coreand model serving frameworkKFServing.Seldon Core:outlieranddriftdetection worked examples.KFServing:outlieranddriftdetection examples.CitationsIf you use runml-infuse in your research, please consider citing it.BibTeX entry:@software{runml-infuse,
  title = {Alibi Detect: Algorithms for outlier, adversarial and drift detection},
  author = {Van Looveren, Arnaud and Klaise, Janis and Vacanti, Giovanni and Cobb, Oliver and Scillitoe, Ashley and Samoilescu, Robert},
  url = {https://github.com/SeldonIO/runml-infuse},
  version = {0.9.0},
  date = {2022-03-17},
  year = {2019}
}