Package: aws-fortuna
A Library for Uncertainty QuantificationProper estimation of predictive uncertainty is fundamental in applications that involve critical decisions.
Uncertainty can be used to assess reliability of model predictions, trigger human intervention,
or decide whether a model can be safely deployed in the wild.Fortuna is a library for uncertainty quantification that makes it easy for users to run benchmarks and bring uncertainty to production systems.
Fortuna provides calibration and conformal methods starting from pre-trained models written in any framework,
and it further supports several Bayesian inference methods starting from deep learning models written inFlax.
The language is designed to be intuitive for practitioners unfamiliar with uncertainty quantification,
and is highly configurable.Check thedocumentationfor a quickstart, examples and references.Usage modesFortuna offers three different usage modes:From uncertainty estimates,From model outputsandFrom Flax models.
These serve users according to the constraints dictated by their own applications.
Their pipelines are depicted in the following figure, each starting from one of the green panels.From uncertainty estimatesStarting from uncertainty estimates has minimal compatibility requirements and it is the quickest level of interaction with the library.
This usage mode offers conformal prediction methods for both classification and regression.
These take uncertainty estimates in input,
and return rigorous sets of predictions that retain a user-given level of probability.
In one-dimensional regression tasks, conformal sets may be thought as calibrated versions of confidence or credible intervals.Mind that if the uncertainty estimates that you provide in inputs are inaccurate,
conformal sets might be large and unusable.
For this reason, if your application allows it,
please consider theFrom model outputsandFrom Flax modelsusage modes.Example.Suppose you want to calibrate credible intervals with coverage errorerror,
each corresponding to a different test input variable.
We assume that credible intervals are passed as arrays of lower and upper bounds,
respectivelytest_lower_boundsandtest_upper_bounds.
You also have lower and upper bounds of credible intervals computed for several validation inputs,
respectivelyval_lower_boundsandval_upper_bounds.
The corresponding array of validation targets is denoted byval_targets.
The following code producesconformal prediction intervals,
i.e. calibrated versions of you test credible intervals.fromfortuna.conformalimportQuantileConformalRegressorconformal_intervals=QuantileConformalRegressor().conformal_interval(val_lower_bounds=val_lower_bounds,val_upper_bounds=val_upper_bounds,test_lower_bounds=test_lower_bounds,test_upper_bounds=test_upper_bounds,val_targets=val_targets,error=error)From model outputsStarting from model outputs assumes you have already trained a model in some framework,
and arrive to Fortuna with model outputs innumpy.ndarrayformat for each input data point.
This usage mode allows you to calibrate your model outputs, estimate uncertainty,
compute metrics and obtain conformal sets.Compared to theFrom uncertainty estimatesusage mode,
this one offers better control,
as it can make sure uncertainty estimates have been appropriately calibrated.
However, if the model had been trained with classical methods,
the resulting quantification of model (a.k.a. epistemic) uncertainty may be poor.
To mitigate this problem, please consider theFrom Flax modelsusage mode.Example.Suppose you have validation and test model outputs,
respectivelyval_outputsandtest_outputs.
Furthermore, you have some arrays of validation and target variables,
respectivelyval_targetsandtest_targets.
The following code provides a minimal classification example to get calibrated predictive entropy estimates.fromfortuna.output_calib_modelimportOutputCalibClassifiercalib_model=OutputCalibClassifier()status=calib_model.calibrate(outputs=val_outputs,targets=val_targets)test_entropies=calib_model.predictive.entropy(outputs=test_outputs)From Flax modelsStarting from Flax models has higher compatibility requirements than theFrom uncertainty estimatesandFrom model outputsusage modes,
as it requires deep learning models written inFlax.
However, it enables you to replace standard model training with scalable Bayesian inference procedures,
which may significantly improve the quantification of predictive uncertainty.Example.Suppose you have a Flax classification deep learning modelmodelfrom inputs to logits, with output
dimension given byoutput_dim. Furthermore,
you have some training, validation and calibration TensorFlow data loadertrain_data_loader,val_data_loaderandtest_data_loader, respectively.
The following code provides a minimal classification example to get calibrated probability estimates.fromfortuna.dataimportDataLoadertrain_data_loader=DataLoader.from_tensorflow_data_loader(train_data_loader)calib_data_loader=DataLoader.from_tensorflow_data_loader(val_data_loader)test_data_loader=DataLoader.from_tensorflow_data_loader(test_data_loader)fromfortuna.prob_modelimportProbClassifierprob_model=ProbClassifier(model=model)status=prob_model.train(train_data_loader=train_data_loader,calib_data_loader=calib_data_loader)test_means=prob_model.predictive.mean(inputs_loader=test_data_loader.to_inputs_loader())InstallationNOTE:Before installing Fortuna, you are required toinstall JAXin your virtual environment.You can install Fortuna by typingpip install aws-fortunaAlternatively, you can build the package usingPoetry.
If you choose to pursue this way, first install Poetry and add it to your PATH
(seehere). Then typepoetry installAll the dependencies will be installed at their required versions. Consider adding the following flags to the command above:-E transformersif you want to use models and datasets fromHugging Face.-E sagemakerif you want to install the dependencies necessary to run Fortuna on Amazon SageMaker.-E docsif you want to install Sphinx dependencies to build the documentation.-E notebooksif you want to work with Jupyter notebooks.Finally, you can either access the virtualenv that Poetry created by typingpoetry shell,
or execute commands within the virtualenv using theruncommand, e.g.poetry run python.ExamplesSeveral usage examples are found in the/examplesdirectory.Amazon SageMaker integrationWe offer a simple pipeline that allows you to run Fortuna on Amazon SageMaker with minimal effort.Create an AWS account - it is free! Store the account ID and the region where you want to launch training jobs.First,update your local AWS credentials.
Then you need to build andpush a Docker image to an Amazon ECR repository.
Thisscriptwill help you doing so -
it will require your AWS account ID and region. If you need other packages to be included in your Docker image,
you should consider customize theDockerfile.
NOTE: the script has been tested on a M1 MacOS.
It is possible that different operating systems will need small modifications.Create anS3 bucket.
You will need this to dump the results from your training jobs on Amazon Sagemaker.Write a configurationyamlfile. This will include your AWS details, the path to the entrypoint script that you want
to run on Amazon SageMaker, the arguments to pass to the script, the path to the S3 bucket where you want to dump
the results, the metrics to monitor, and more.
Checkthis filefor an example.Finally, givenconfig_dir, that is the absolute path to the main configuration directory,
andconfig_filename, that is the name of the main configuration file (without .yaml extension),
enter Python and run the following:fromfortuna.sagemakerimportrun_training_jobrun_training_job(config_dir=config_dir,config_filename=config_filename)MaterialAWS launch blog postFortuna: A Library for Uncertainty Quantification in Deep Learning [arXiv paper]Citing FortunaTo cite Fortuna:@article{detommaso2023fortuna,
  title={Fortuna: A Library for Uncertainty Quantification in Deep Learning},
  author={Detommaso, Gianluca and Gasparin, Alberto and Donini, Michele and Seeger, Matthias and Wilson, Andrew Gordon and Archambeau, Cedric},
  journal={arXiv preprint arXiv:2302.04019},
  year={2023}
}ContributingIf you wish to contribute to the project, please refer to ourcontribution guidelines.LicenseThis project is licensed under the Apache-2.0 License.
SeeLICENSEfor more information.