Generative Posterior Networks for Approximately Bayesian Epistemic Uncertainty Estimation

1 INTRODUCTION . Deep learning systems are becoming ubiquitous in many industries , specifically in applications with large datasets and where prediction mistakes are inexpensive . However , for many other industries , such as autonomous manufacturing , where datasets are smaller and mistakes can be catastrophic , the adoption of deep learning techniques has been slow . One of the critical challenges to applying deep learning models in these safety-critical domains is the lack of uncertainty estimation . If a model makes a prediction in a safety critical environment , we would like that model to be highly confident that prediction is correct . Uncertainty prediction can also provide significant benefit in data-poor domains by guiding exploration towards areas of uncertainty . Most deep learning models , however , are not able to estimate epistemic uncertainty – uncertainty deriving from lack of data samples . In this work , we introduce a new method for modeling epistemic uncertainty in neural networks . We formulate this problem as a Bayesian Inference problem : given a prior distribution over functions and a set of training data , we want to model the posterior function distribution . There are many existing approaches to constructing such a posterior , such as Gaussian Processes ( GPs ) or Variational Inference ( VI ) methods , but these methods can require significant engineering to work in practice and can be expensive in high-dimensional settings . For this reason , many practitioners turn to using ensembles of neural networks to approximate epistemic uncertainty for their ease of use in highdimensional settings . Pearce et al . ( 2020 ) illustrated that , given the correct regularization , these ensemble members approximate samples from the posterior distribution of functions . However , using this technique , every new sample of the posterior requires training a new neural network from scratch , which can be prohibitively expensive for large problems . To address this challenge , we introduce Generative Posterior Networks ( GPNs ) , a generative neural network model that directly approximates the posterior distribution by regularizing the output of the network towards samples from the prior distribution . By learning a low-dimensional latent representation of the posterior , our method can quickly sample from the posterior and construct confidence intervals . We prove that our method approximates the Bayesian posterior over functions and show empirically that our method can improves uncertainty estimation over competing methods . 2 RELATED WORK . There are two categories of uncertainty in the context of probabilistic modeling : aleatoric uncertainty and epistemic uncertainty . Aleatoric uncertainty refers to uncertainty inherent in a system , even when the true parameters of the system are known , while epistemic uncertainty refers to modeling uncertainty that can be reduced through collecting more data ( Der Kiureghian & Ditlevsen , 2009 ; Gal , 2016 ) . Deep learning models have shown impressive performance in quantifying aleatoric uncertainty when given enough data . On the other hand , despite many methods seeking to estimate epistemic uncertainty due to its well-motivated applications in Bayesian Optimization ( Snoek et al. , 2012 ; Springenberg et al. , 2016 ) , Reinforcement Learning ( Curi * et al. , 2020 ) , and adversarial robustness ( Stutz et al. , 2020 ) , quantifying it remains a challenging problem for deep learning models . Gaussian Processes ( GPs ) remain one of the most effective methods for epistemic uncertainty estimation in low-dimensional , low-data settings . While scalability can be improved with interpolation methods ( Hensman et al. , 2013 ; Titsias , 2009 ; Wilson & Nickisch , 2015 ) and kernel learning methods ( Wilson et al. , 2016 ) , GPs tend to perform worse than neural network-based methods on high-dimensional problems . Another way to formulate epistemic uncertainty estimation is as a Bayesian Inference problem . Specifically , given a prior distribution over functions and some labeled data , the goal is to construct a posterior distribution in the Bayesian sense . Because this posterior distribution is usually intractable to compute exactly , approximate sampling methods are used instead . Markov-Chain Monte Carlo ( MCMC ) ( Tanner & Wong , 1987 ) and Variational Inference ( VI ) ( Blei et al. , 2017 ) are some of the most common methods for posterior sampling . These methods , however , tend to struggle in high dimensional problems . The computational complexity of MCMC scales poorly with the dimension of the data and VI methods tend to require restrictive function families with strong independence assumptions . Bayesian Neural Networks ( BNNs ) ( Blundell et al. , 2015 ) are another way to perform Bayesian Inference a neural networks by modeling each parameter in the network with an independent Gaussian distribution . While BNNs perform well in low dimensional settings , their performance tends to degrade in high-dimensional settings that require large networks . Gal & Ghahramani ( 2016 ) proved that dropout can also be used as a form of Bayesian Inference . Specifically , if a network is trained using dropout and L2 regularization , posterior samples can be collected by applying dropout at test-time . This method has surprisingly good out-of-distribution detection and can be applied to any network . However , it can be hard to tune the dropout parameter to have good uncertainty prediction without significantly degrading performance . Neural network ensembling is another commonly used epistemic uncertainty estimation technique , which is closely related to our proposed method in this work . This technique involves training a number of completely separate neural networks on the same data to form an ensemble . Because of the stochasticity of initialization and the training process , each member of the ensemble should be a slightly different function . Specifically , the ensemble members should produce similar outputs in data-dense regions of the state-space and dissimilar outputs in data-sparse regions . Pearce et al . ( 2020 ) proved that , by using the correct regularization , each ensemble member becomes an approximate sample from the function posterior . Ensembling methods are particularly attractive to practitioners because they require very little fine-tuning and work surprisingly well in practice . However , these methods have one main drawback , namely that each new sample from the posterior requires training a new network from scratch . Our method , on the other hand , seeks to construct a generative posterior model using similar regularization techniques to Pearce et al . ( 2020 ) , allowing for quick sampling from the posterior . 3 PROBLEM STATEMENT AND BACKGROUND . We will focus on the problem of Bayesian Inference on the parameters of θ of some function f ( · ; θ ) . We assume we have some known prior over parameters Pprior ( θ ) ∼ N ( µprior , Σprior ) as well as access to a noisy dataset ( xobs , yobs ) = { ( x1obs , y1obs ) . . . , ( xNobs , yNobs ) } , where yiobs = f ( xiobs ; θ ) + , ∼ N ( 0 , σ ) for some unknown parameter θ . We can then define the data-likelihood as Plike ( yobs |θ , xobs ) = ∏ i N ( yiobs | f ( xiobs ; θ ) , σ ) . ( 1 ) The goal is to approximate samples from the posterior , Ppost ( θ |yobs , xobs ) ∝ Plike ( yobs |θ , xobs ) Pprior ( θ ) . ( 2 ) For ease of notation , we will drop the xobs from the likelihood . For our analysis , it is convenient if the posterior distribution is Gaussian . To show this , we will use the fact that the product of two Gaussian PDFs is proportional to a Gaussian PDF . We make a distinction here between the data-likelihood , a function of the observed data yobs , and the parameterlikelihood , a function of the parameters θ , defined as follows : Pparam-like ( θ ; yobs ) = Plike ( yobs |θ ) . ( 3 ) The parameter-likelihood PDF is equivalent to the data-likelihood PDF for all values of yobs and θ , but is notably a function of the parameters θ . For a more detailed explanation of the distinction between the parameter and data-likelihoods , see Definition 1 from Pearce et al . ( 2020 ) . When the function f is linear , it can be shown that the parameter-likelihood is also Gaussian , Pparam-like ( θ ; yobs ) ∝ N ( µlike , Σlike ) . ( 4 ) While this is not true in general , Pearce et al . ( 2020 ) assume that the parameter likelihood is Gaussian so that the posterior also becomes Gaussian , Ppost ( θ |yobs ) ∝ Pparam-like ( θ ; yobs ) Pprior ( θ ) ∝ N ( µlike , Σlike ) N ( µprior , Σprior ) = N ( µpost , Σpost ) , ( 5 ) where the posterior covariance and mean are given by Σpost = ( Σ−1like + Σ −1 prior ) −1 , µpost = ΣpostΣ −1 likeµlike + ΣpostΣ −1 priorµprior . ( 6 ) Note , however , that after the reformulation of the problem in Section 4.1 , we no longer require this assumption for our theoretical analysis . 3.1 RANDOMIZED MAP SAMPLING ( RMS ) . Since the values µlike and Σlike are intractible to compute in practice , the challenge of computing the posterior distribution remains . However , Pearce et al . ( 2020 ) showed how we can use a method called Randomized MAP sampling ( RMS ) to approximately sample from the posterior distribution . With RMS we first assume access to a mechanism to compute the maximum a posteriori ( MAP ) solution for our posterior . Then , by randomly shifting the prior distribution we use in our MAP solver , we can estimate samples from the posterior . Specifically , RMS samples “ anchor points ” θanc from some distribution θanc ∼ N ( µanc , Σanc ) . Each anchor point defines a shifted prior distribution Panc ( θanc ) = N ( θanc , Σprior ) . Now we define the MAP function which finds the MAP solution using this shifted prior : θMAP ( θanc ) = arg max θ Plike ( yobs |θ ) Panc ( θ ) . ( 7 ) By sampling different anchor points , we can then construct a probability distribution over MAP solutions P ( θMAP ( θanc ) ) , which can be shown to be Gaussian . Using moment matching , Pearce et al . ( 2020 ) proved , in Theorem 1 , that if we choose µanc = µprior and Σanc = Σprior + ΣpriorΣ−1likeΣprior , then P ( θMAP ( θanc ) ) = Ppost ( θ |yobs ) . Using the approximation Σanc ≈ Σprior , a sample from posterior , θpost , can be collected by sampling θanc ∼ N ( µanc , Σanc ) and optimizing the following : θpost = arg max θ logPparam-like ( θ ; yobs ) + logPanc ( θ ) . ( 8 ) 4 GENERATIVE POSTERIOR NETWORKS . While RMS can be used to construct samples from the posterior , every new sample requires solving a new optimization problem . Instead , we propose learning the MAP function itself . Let g be a neural network parameterized by some vector , φ , that takes as input a sample from the anchor distribution , θanc ∼ Panc ( θanc ) , and outputs θMAP ( θanc ) . If we assume that g has enough expressive power to represent the MAP function , then g ( θanc ; φ ) = θMAP ( θanc ) if : φ = arg max φ Eθanc∼Panc ( θanc ) logPlike ( yobs | g ( θanc ; φ ) ) + logPanc ( g ( θanc ; φ ) ) . ( 9 ) In practice , however , this is a very hard optimization problem for two main reasons . ( 1 ) Optimizing in parameter space tends to be challenging for gradient-descent optimization . ( 2 ) The space of all parameters θanc is not only very large , but there are also many parameters θanc that map to the same outputs for all inputs . Moreover , we would expect the posterior parameters to be highly correlated and , therefore , need less expressive power to be represented . To address these two problems , we will first change the problem definition slightly to find the posterior in output space as opposed to parameter space . We prove in the next section that , by changing our loss function , we can still use RMS to approximate the posterior in output space . Next , we show practically how we construct a low-dimensional representation of the anchor distribution .

Summary: The paper develops an uncertainty quantification method as an extension of the Bayesian ensemble method. Instead of training multiple ensemble models through MAP optimization, the proposed method tries to learn a mapping function between the prior distribution and the posterior distribution of model parameters, which avoids the complex training of ensemble models and achieves better efficiency. Due to the high-dimensional parameter space, the authors propose to learn the mapping function in the low-dimensional output space with theoretical justifications. The experiments include image classification and Bayesian optimization, where authors compare the proposed method with Bayesian ensemble and MC-dropout in terms of uncertainty estimation.