Package: uncertainty-toolbox
Website,Tutorials, andDocsUncertainty ToolboxA Python toolbox for predictive uncertainty quantification, calibration,metrics, and visualization.Also: aglossary of useful termsand a collection
ofrelevant papers and references.Many machine learning methods return predictions along with uncertainties of some form,
such as distributions or confidence intervals. This begs the questions: How do we
determine which predictive uncertanties are best? What does it mean to produce abestoridealuncertainty? Are our uncertainties accurate andwell calibrated?Uncertainty Toolbox provides standard metrics to quantify and compare predictive
uncertainty estimates, gives intuition for these metrics, produces visualizations of
these metrics/uncertainties, and implements simple "re-calibration" procedures to
improve these uncertainties.  This toolbox currently focuses on regression tasks.Toolbox ContentsUncertainty Toolbox contains:Glossaryof terms related to predictive uncertainty
quantification.Metricsfor assessing quality of predictive uncertainty estimates.Visualizationsfor predictive uncertainty estimates and metrics.Recalibrationmethods for improving the calibration of a predictor.Paper list: publications and references on relevant methods and metrics.InstallationUncertainty Toolbox requires Python 3.6+ and pip ≥ 21.3. For a lightweight installation
of the package only, run:pipinstalluncertainty-toolboxFor a full installation with examples, tests, and the latest updates, run:gitclonehttps://github.com/uncertainty-toolbox/uncertainty-toolbox.gitcduncertainty-toolbox
pipinstall-e.-rrequirements/requirements_dev.txtTo verify correct installation, you can run thetest suitevia:sourceshell/run_all_tests.shQuick Startimportuncertainty_toolboxasuct# Load an example dataset of 100 predictions, uncertainties, and ground truth valuespredictions,predictions_std,y,x=uct.data.synthetic_sine_heteroscedastic(100)# Compute all uncertainty metricsmetrics=uct.metrics.get_all_metrics(predictions,predictions_std,y)This example computesmetricsfor a vector of predicted values
(predictions) and associated uncertainties (predictions_std, a vector of standard
deviations), taken with respect to a corresponding set of ground truth valuesy.Colab notebook:You can also take a look atthis Colab
notebook,
which walks through a use case of Uncertainty Toolbox.MetricsUncertainty Toolbox provides a number ofmetricsto
quantify and compare predictive uncertainty estimates. For example, theget_all_metricsfunction will return:average calibration:mean absolute calibration error, root mean squared calibration error, miscalibration area.adversarial group calibration:mean absolute adversarial group calibration error, root mean squared adversarial group calibration error.sharpness:expected standard deviation.proper scoring rules:negative log-likelihood, continuous ranked probability score, check score, interval score.accuracy:mean absolute error, root mean squared error, median absolute error, coefficient of determination, correlation.VisualizationsThe following plots are a few of thevisualizationsprovided by Uncertainty Toolbox. Seethis examplefor code
to reproduce these plots.Overconfident(too little uncertainty)Underconfident(too much uncertainty)Well calibratedAnd here are a few of the calibration metrics for the above three cases:Mean absolute calibration error (MACE)Root mean squared calibration error (RMSCE)Miscalibration area (MA)Overconfident0.194290.217530.19625Underconfident0.206920.230030.20901Well calibrated0.008620.010400.00865RecalibrationThe following plots show the results of arecalibrationprocedure provided by Uncertainty
Toolbox, which transforms a set of predictive uncertainties to improve average
calibration. The algorithm is based on isotonic regression, as proposed byKuleshov et
al.Seethis examplefor code to reproduce these plots.Recalibrating overconfident predictionsMean absolute calibration error (MACE)Root mean squared calibration error (RMSCE)Miscalibration area (MA)Before Recalibration0.194290.217530.19625After Recalibration0.011240.025910.01117Recalibrating underconfident predictionsMean absolute calibration error (MACE)Root mean squared calibration error (RMSCE)Miscalibration area (MA)Before Recalibration0.206920.230030.20901After Recalibration0.001570.002050.00132ContributingWe welcome and greatly appreciate contributions from the community! Please see
ourcontributing guidelinesfor details on how to help out.CitationIf you found this toolbox helpful, please cite thefollowing
paper:@article{chung2021uncertainty,
  title={Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification},
  author={Chung, Youngseog and Char, Ian and Guo, Han and Schneider, Jeff and Neiswanger, Willie},
  journal={arXiv preprint arXiv:2109.10254},
  year={2021}
}Additionally, here are papers that led to the development of the toolbox:@article{chung2020beyond,
  title={Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification},
  author={Chung, Youngseog and Neiswanger, Willie and Char, Ian and Schneider, Jeff},
  journal={arXiv preprint arXiv:2011.09588},
  year={2020}
}

@article{tran2020methods,
  title={Methods for comparing uncertainty quantifications for material property predictions},
  author={Tran, Kevin and Neiswanger, Willie and Yoon, Junwoong and Zhang, Qingyang and Xing, Eric and Ulissi, Zachary W},
  journal={Machine Learning: Science and Technology},
  volume={1},
  number={2},
  pages={025006},
  year={2020},
  publisher={IOP Publishing}
}AcknowledgmentsDevelopment of Uncertainty Toolbox issupported bythe following organizations.