Package: laplace-torch
The laplace package facilitates the application of Laplace approximations for entire neural networks, subnetworks of neural networks, or just their last layer.
The package enables posterior approximations, marginal-likelihood estimation, and various posterior predictive computations.
The library documentation is available athttps://aleximmer.github.io/Laplace.There is also a corresponding paper,Laplace Redux — Effortless Bayesian Deep Learning, which introduces the library, provides an introduction to the Laplace approximation, reviews its use in deep learning, and empirically demonstrates its versatility and competitiveness. Please consider referring to the paper when using our library:@inproceedings{laplace2021,title={Laplace Redux--Effortless {B}ayesian Deep Learning},author={Erik Daxberger and Agustinus Kristiadi and Alexander Immerand Runa Eschenhagen and Matthias Bauer and Philipp Hennig},booktitle={{N}eur{IPS}},year={2021}}SetupWe assumepython3.8since the package was developed with that version.
To install laplace withpip, run the following:pipinstalllaplace-torchFor development purposes, clone the repository and then install:# or after cloning the repository for developmentpipinstall-e.# run testspipinstall-e.[tests]pytesttests/StructureThe laplace package consists of two main components:The subclasses oflaplace.BaseLaplacethat implement different sparsity structures: different subsets of weights ('all','subnetwork'and'last_layer') and different structures of the Hessian approximation ('full','kron','lowrank'and'diag'). This results ineightcurrently available options:laplace.FullLaplace,laplace.KronLaplace,laplace.DiagLaplace, the corresponding last-layer variationslaplace.FullLLLaplace,laplace.KronLLLaplace,  andlaplace.DiagLLLaplace(which are all subclasses oflaplace.LLLaplace),laplace.SubnetLaplace(which only supports a'full'Hessian approximation) andlaplace.LowRankLaplace(which only supports inference over'all'weights). All of these can be conveniently accessed via thelaplace.Laplacefunction.The backends inlaplace.curvaturewhich provide access to Hessian approximations of
the corresponding sparsity structures, for example, the diagonal GGN.Additionally, the package provides utilities for
decomposing a neural network into feature extractor and last layer forLLLaplacesubclasses (laplace.utils.feature_extractor)
and
effectively dealing with Kronecker factors (laplace.utils.matrix).Finally, the package implements several options to select/specify a subnetwork forSubnetLaplace(as subclasses oflaplace.utils.subnetmask.SubnetMask).
Automatic subnetwork selection strategies include: uniformly at random (laplace.utils.subnetmask.RandomSubnetMask), by largest parameter magnitudes (LargestMagnitudeSubnetMask), and by largest marginal parameter variances (LargestVarianceDiagLaplaceSubnetMaskandLargestVarianceSWAGSubnetMask).
In addition to that, subnetworks can also be specified manually, by listing the names of either the model parameters (ParamNameSubnetMask) or modules (ModuleNameSubnetMask) to perform Laplace inference over.ExtendabilityTo extend the laplace package, newBaseLaplacesubclasses can be designed, for example,
Laplace with a block-diagonal Hessian structure.
One can also implement custom subnetwork selection strategies as new subclasses ofSubnetMask.Alternatively, extending or integrating backends (subclasses ofcurvature.curvature) allows to provide different Hessian
approximations to the Laplace approximations.
For example, currently thecurvature.BackPackInterfacebased onBackPACKandcurvature.AsdlInterfacebased onASDLare available.
Thecurvature.AsdlInterfaceprovides a Kronecker factored empirical Fisher while thecurvature.BackPackInterfacedoes not, and only thecurvature.BackPackInterfaceprovides access to Hessian approximations
for a regression (MSELoss) loss function.Example usagePost-hocprior precision tuning of diagonal LAIn the following example, a pre-trained model is loaded,
then the Laplace approximation is fit to the training data
(using a diagonal Hessian approximation over all parameters),
and the prior precision is optimized with cross-validation'CV'.
After that, the resulting LA is used for prediction with
the'probit'predictive for classification.fromlaplaceimportLaplace# Pre-trained modelmodel=load_map_model()# User-specified LA flavorla=Laplace(model,'classification',subset_of_weights='all',hessian_structure='diag')la.fit(train_loader)la.optimize_prior_precision(method='CV',val_loader=val_loader)# User-specified predictive approx.pred=la(x,link_approx='probit')Differentiating the log marginal likelihood w.r.t. hyperparametersThe marginal likelihood can be used for model selection [10] and is differentiable
for continuous hyperparameters like the prior precision or observation noise.
Here, we fit the library default, KFAC last-layer LA and differentiate
the log marginal likelihood.fromlaplaceimportLaplace# Un- or pre-trained modelmodel=load_model()# Default to recommended last-layer KFAC LA:la=Laplace(model,likelihood='regression')la.fit(train_loader)# ML w.r.t. prior precision and observation noiseml=la.log_marginal_likelihood(prior_prec,obs_noise)ml.backward()Applying the LA over only a subset of the model parametersThis example shows how to fit the Laplace approximation over only
a subnetwork within a neural network (while keeping all other parameters
fixed at their MAP estimates), as proposed in [11]. It also exemplifies
different ways to specify the subnetwork to perform inference over.fromlaplaceimportLaplace# Pre-trained modelmodel=load_model()# Examples of different ways to specify the subnetwork# via indices of the vectorized model parameters## Example 1: select the 128 parameters with the largest magnitudefromlaplace.utilsimportLargestMagnitudeSubnetMasksubnetwork_mask=LargestMagnitudeSubnetMask(model,n_params_subnet=128)subnetwork_indices=subnetwork_mask.select()# Example 2: specify the layers that define the subnetworkfromlaplace.utilsimportModuleNameSubnetMasksubnetwork_mask=ModuleNameSubnetMask(model,module_names=['layer.1','layer.3'])subnetwork_mask.select()subnetwork_indices=subnetwork_mask.indices# Example 3: manually define the subnetwork via custom subnetwork indicesimporttorchsubnetwork_indices=torch.tensor([0,4,11,42,123,2021])# Define and fit subnetwork LA using the specified subnetwork indicesla=Laplace(model,'classification',subset_of_weights='subnetwork',hessian_structure='full',subnetwork_indices=subnetwork_indices)la.fit(train_loader)DocumentationThe documentation is availablehereor can be generated and/or viewed locally:# assuming the repository was clonedpipinstall-e.[docs]# create docs and write to htmlbashupdate_docs.sh# .. or serve the docs directlypdoc--http0.0.0.0:8080laplace--template-dirtemplateReferencesThis package relies on various improvements to the Laplace approximation for neural networks, which was originally due to MacKay [1]. Please consider citing the respective papers if you use any of their proposed methods via our laplace library.[1] MacKay, DJC.A Practical Bayesian Framework for Backpropagation Networks. Neural Computation 1992.[2] Gibbs, M. N.Bayesian Gaussian Processes for Regression and Classification. PhD Thesis 1997.[3] Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Prabhat, M., Adams, R.Scalable Bayesian Optimization Using Deep Neural Networks. ICML 2015.[4] Ritter, H., Botev, A., Barber, D.A Scalable Laplace Approximation for Neural Networks. ICLR 2018.[5] Foong, A. Y., Li, Y., Hernández-Lobato, J. M., Turner, R. E.'In-Between' Uncertainty in Bayesian Neural Networks. ICML UDL Workshop 2019.[6] Khan, M. E., Immer, A., Abedi, E., Korzepa, M.Approximate Inference Turns Deep Networks into Gaussian Processes. NeurIPS 2019.[7] Kristiadi, A., Hein, M., Hennig, P.Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks. ICML 2020.[8] Immer, A., Korzepa, M., Bauer, M.Improving predictions of Bayesian neural nets via local linearization. AISTATS 2021.[9] Sharma, A., Azizan, N., Pavone, M.Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks. UAI 2021.[10] Immer, A., Bauer, M., Fortuin, V., Rätsch, G., Khan, EM.Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning. ICML 2021.[11] Daxberger, E., Nalisnick, E., Allingham, JU., Antorán, J., Hernández-Lobato, JM.Bayesian Deep Learning via Subnetwork Inference. ICML 2021.