Package: uq360
UQ360The Uncertainty Quantification 360 (UQ360) is an open-source toolkit with a Python package to provide data science
practitioners and developers access to state-of-the-art algorithms, to streamline the process of estimating, evaluating,
improving, and communicating uncertainty of machine learning models as common practices for AI transparency.
TheUQ360 interactive experienceprovides a gentle introduction to the concepts and
capabilities by walking through an example use case. Thetutorials and example notebooksoffer a deeper,
data scientist-oriented introduction. Thecomplete APIis also available.We have developed the package with extensibility in mind. This library is still in development. We encourage the
contribution of your uncertainty estimation algorithms, metrics and applications.
To get started as a contributor, please join the #uq360-users or #uq360-developers channel of
theAIF360 Community on Slackby requesting an
invitationhere.ResourcesIntroductionto Uncertainty Quantification 360.DemoHouse Price Prediction Model.List ofAlgorithmssupported.List ofMetricssupported.Guidanceon Choosing UQ Algorithms and Metrics.Guidanceon Communicating Uncertainty.Glossaryof UQ Terms.Read ourpapers.Complete list oftutorials.Join the SlackCommunity.Example Use-casesMeta-modelsUse of meta-models to augment sklearn's gradient boosted regressor with prediction interval. See detailed examplehere.fromsklearn.ensembleimportGradientBoostingRegressorfromsklearn.datasetsimportmake_regressionfromsklearn.model_selectionimporttrain_test_splitfromuq360.algorithms.blackbox_metamodelimportMetamodelRegression# Create train, calibration and test splits.X,y=make_regression(random_state=0)X_train,X_test,y_train,y_test=train_test_split(X,y,random_state=0)X_train,X_calibration,y_train,y_calibration=train_test_split(X_train,y_train,random_state=0)# Train the base model that provides the mean estimates.gbr_reg=GradientBoostingRegressor(random_state=0)gbr_reg.fit(X_train,y_train)# Train the meta-model that can augment the mean prediction with prediction intervals.uq_model=MetamodelRegression(base_model=gbr_reg)uq_model.fit(X_calibration,y_calibration,base_is_prefitted=True)# Obtain mean estimates and prediction interval on the test data.y_hat,y_hat_lb,y_hat_ub=uq_model.predict(X_test)UQ360 metrics for model selectionThe prediction interval coverage probability score (PICP) score is used here
as the metric to select the model through cross-validation. See detailed examplehere.fromsklearn.datasetsimportmake_regressionfromsklearn.model_selectionimporttrain_test_splitfromsklearn.model_selectionimportGridSearchCVfromuq360.utils.miscimportmake_sklearn_compatible_scorerfromuq360.algorithms.quantile_regressionimportQuantileRegression# Create a sklearn scorer using UQ360 PICP metric.sklearn_picp=make_sklearn_compatible_scorer(task_type="regression",metric="picp",greater_is_better=True)# Hyper-parameters configuration using GridSearchCV.base_config={"alpha":0.95,"n_estimators":20,"max_depth":3,"learning_rate":0.01,"min_samples_leaf":10,"min_samples_split":10}configs={"config":[]}fornum_estimatorsin[1,2,5,10,20,30,40,50]:config=base_config.copy()config["n_estimators"]=num_estimatorsconfigs["config"].append(config)# Create train test split.X,y=make_regression(random_state=0)X_train,X_test,y_train,y_test=train_test_split(X,y,random_state=0)# Initialize QuantileRegression UQ360 model and wrap it in GridSearchCV with PICP as the scoring function.uq_model=GridSearchCV(QuantileRegression(config=base_config),configs,scoring=sklearn_picp)# Fit the model on the training set.uq_model.fit(X_train,y_train)# Obtain the prediction intervals for the test set.y_hat,y_hat_lb,y_hat_ub=uq_model.predict(X_test)SetupSupported Configurations:OSPython versionmacOS3.7Ubuntu3.7Windows3.7(Optional) Create a virtual environmentA virtual environment manager is strongly recommended to ensure dependencies may be installed safely. If you have trouble installing the toolkit, try this first.CondaConda is recommended for all configurations though Virtualenv is generally
interchangeable for our purposes. Miniconda is sufficient (seethe difference between Anaconda and
Minicondaif you are curious) and can be installed fromhereif you do not already have it.Then, to create a new Python 3.7 environment, run:condacreate--nameuq360python=3.7
condaactivateuq360The shell should now look like(uq360) $. To deactivate the environment, run:(uq360)$condadeactivateThe prompt will return back to$or(base)$.Note: Older versions of conda may usesource activate uq360andsource deactivate(activate uq360anddeactivateon Windows).InstallationClone the latest version of this repository:(uq360)$gitclonehttps://github.ibm.com/UQ360/UQ360If you'd like to run the examples and tutorial notebooks, download the datasets now and place them in
their respective folders as described inuq360/data/README.md.Then, navigate to the root directory of the project which containssetup.pyfile and run:(uq360)$pipinstall-e.PIP Installation of Uncertainty Quantification 360If you would like to quickly start using the UQ360 toolkit without cloning this repository, then you can install theuq360 pypi packageas follows.(yourenvironment)$pipinstalluq360If you follow this approach, you may need to download the notebooks in theexamplesfolder separately.Using UQ360Theexamplesdirectory contains a diverse collection of jupyter notebooks that use UQ360 in various ways. Both examples and tutorial notebooks illustrate working code using the toolkit. Tutorials provide additional discussion that walks the user through the various steps of the notebook. See the details about tutorials and exampleshere.Citing UQ360A technical description of UQ360 is available in thispaper. Below is the bibtex entry for this
paper.@misc{uq360-june-2021,
      title={Uncertainty Quantification 360: A Holistic Toolkit for Quantifying 
      and Communicating the Uncertainty of AI}, 
      author={Soumya Ghosh and Q. Vera Liao and Karthikeyan Natesan Ramamurthy 
      and Jiri Navratil and Prasanna Sattigeri 
      and Kush R. Varshney and Yunfeng Zhang},
      year={2021},
      eprint={2106.01410},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}AcknowledgementsUQ360 is built with the help of several open source packages. All of these are listed in setup.py and some of these include:scikit-learnhttps://scikit-learn.org/stable/about.htmlPytorchhttps://github.com/pytorch/pytorchBotorchhttps://github.com/pytorch/botorchLicense InformationPlease view both theLICENSEfile present in the root directory for license information.