Uncertainty-based out-of-distribution detection requires suitable function space priors

1 INTRODUCTION . One of the challenges that the modern machine learning community is striving to tackle is the detection of unseen inputs for which predictions should not be trusted . This problem is also known as out-of-distribution ( OOD ) detection . The challenging nature of this task is partly rooted in the fact that there is no universal mathematical definition that characterizes an unseen input x∗ as OOD as discussed in Sec . 2 . Without such a definition , there is no foundation for deriving detection guarantees under verifiable assumptions . In this work , we consider OOD points as points that are unlikely under the distribution of the data p ( x ) and points are outside the support of p ( x ) . Importantly , we do not claim to provide a meaningful definition of OOD , but rather argue that the justification of an OOD method can only be assessed theoretically once such a definition is provided . While it appears natural to tackle the OOD detection problem from a generative perspective by explicitly modelling p ( x ) , this paper is solely concerned with the question of how justified it is to deploy the predictive uncertainty of a Bayesian neural network ( BNN ) for OOD detection . As recent developments forecast an increasing integration of deep learning methods into industrial applications , it becomes essential to provide the theoretical groundings that justify the use of uncertainty for OOD detection , a task that is crucial for safety-critical applications of AI and reliable prediction-making . When BNNs are used in supervised learning , the dataset is composed of inputs x ∈ X and targets y ∈ Y which are assumed to be generated according to some unknown process : D i.i.d.∼ p ( x ) p ( y | x ) .The goal of learning is to infer from D alone the distribution p ( y | x ) in order to make predictions on unseen inputs x∗ . In the case of deep learning , this problem is approached by choosing a neural network f ( · ; w ) parametrized by w , and predictions are made via the conditional p ( y | f ( x ; w ) ) ( also called the likelihood of w for given x , y ) . Assuming that the induced class of hypotheses contains some ŵ such that p ( y | f ( x ; ŵ ) ) = p ( y | x ) almost everywhere on the support of p ( x ) , Bayesian statistics can be used to infer plausible models p ( w | D ) under the observed data given some prior knowledge p ( w ) ( see MacKay ( 2003 ) ) . This parametric description together with the network implicitly induces a prior over functions p ( f ) ( Williams , 1997 ; Fortuin , 2021 ) . Believing in the validity of a subjective choice of prior ( O ’ Hagan , 2004 ) , Bayesian inference comes with a multitude of benefits as it is less susceptible to overfitting , allows to incorporate new evidence without requiring access to past data ( Farquhar & Gal , 2018 ) and provides interpretable uncertainties . The uncertainty captured by a BNN can be coarsely categorized into aleatoric and epistemic uncertainty . Aleatoric uncertainty is irreducible and intrinsic to the data p ( y | x ) . For instance , a blurry image might not contain enough information to identify unique object classes . On the other hand , a BNN also models epistemic uncertainty by maintaining a distribution over parameters p ( w | D ) .1 This distribution reflects the uncertainty about which hypothesis explains the data and can be reduced by observing more data . Arguably , aleatoric uncertainty is of little interest for detecting OOD inputs . However , the recent advent of overparametrized models which further extend the hypothesis class , deceptively motivated the idea that the epistemic uncertainty under a Bayesian framework might be intrinsically suitable to detect unfamiliar inputs , and therewith implying , that BNNs can be used for OOD detection . This conjecture is intuitively true if the following assumption holds : the hypotheses captured by p ( w | D ) must agree on their predictions for in-distribution samples but disagree for OOD samples ( Fig . 1a ) . Certain Bayesian methods satisfy this assumption , e.g . a Gaussian process regression with an RBF kernel ( cf . Sec . 4 ) . However , the uncertainty induced by Bayesian inference does not in general give rise to OOD capabilities . This can easily be verified by considering the following thought experiment ( Fig . 1b ) : Assume a model class with only two hypotheses g ( x ) and h ( x ) such that g ( x ) 6= h ( x ) ∀x and data being generated according to g ( x ) on a restricted domain . Once data is observed , we can commit to the ground-truth hypothesis g ( x ) and thus lose epistemic uncertainty in- and out-of-distribution alike . Finite-width neural networks , by contrast , form a powerful class of models , and are often put into context with universal function approximators ( Hornik , 1991 ) . But is this fact in combination with Bayesian statistics enough to attribute them with good OOD capabilities ? The literature often seems to imply that a BNN is intrinsically good at OOD detection . For instance , the use of OOD benchmarks when introducing new methods for approximate inference creates the false impression that the true posterior is a good OOD detector ( Louizos & Welling , 2017 ; Pawlowski et al. , 2017 ; Krueger et al. , 2017 ; Henning et al. , 2018 ; Maddox et al. , 2019 ; Ciosek et al. , 2020 ; D ’ Angelo & Fortuin , 2021 ) . Our work is meant to start a discussion among researchers about the properties of OOD uncertainty of the exact Bayesian posterior of neural networks . To initiate this , we contribute as follows : • We emphasize the importance of the prior in function space for OOD detection , which is induced by the choice of architecture ( Sec . 4 ) and weight space prior ( Sec . 5 ) . • We empirically show that exact inference in infinite-width networks under common architectural choices does not necessarily lead to desirable OOD behavior , and that these observations are consistent with results obtained via approximate inference in their finite-width counterparts . • We furthermore study the OOD behavior in infinite-width networks by analysing the properties of the induced kernels . Moreover , we discuss desirable kernel features for OOD detection and show that also neural networks can approximately carry these features . • We emphasize that the choice of weight-space prior has a strong effect on OOD performance , and that encoding desirable function space properties within unknown OOD domains into such prior is challenging . • We argue that there is a trade-off between good generalization and having high uncertainty on OOD data . Indeed , improving generalization by incorporating prior knowledge ( which is usually encoded in an input-domain agnostic manner ) can negatively impact OOD uncertainties . 1Note , that the Bayesian treatment of network parameters does not account for all types of epistemic uncertainty such as the uncertainty stemming from model misspecification ( Hüllermeier & Waegeman , 2021 ) . We , however , assume the model to be correctly specified . 2 ON THE DIFFICULTY OF DEFINING OUT-OF-DISTRIBUTION INPUTS . While the intuitive notion of OOD sample points ( or outliers ) is commonly agreed upon , for instance as `` an observation ( or subset of observations ) which appears to be inconsistent with the remainder of that set of data '' ( Barnett & Lewis , 1984 ) , a mathematical formalization of the essence of an outlier is difficult ( cf . SM A ) and requires subjective characterizations ( Zimek & Filzmoser , 2018 ) . Often , methods for outlier detection are designed based on an intuitive notion ( see Pimentel et al . ( 2014 ) for a review ) , and therewith only implicitly define a method-specific definition of OOD points . This also accounts for uncertainty-based OOD detection with neural networks , where a statistic of the predictive distribution that quantifies uncertainty ( e.g. , the entropy ) is used to decide whether an input is considered OOD ( Hendrycks & Gimpel , 2017 ) . Commonly , the parameters w of a neural network are trained using an objective derived from Ep ( x ) [ KL ( p ( y | x ) ||p ( y | f ( x ; w ) ) ) ] ( i.e. , loss functions based on the negative log-likelihood such as the cross-entropy or mean-squared error loss ) . Therefore , these networks only have calibrated uncertainties in-distribution with OOD uncertainties not being controlled for unless explicit training on OOD data is performed ( eg. , Hendrycks et al. , 2019 ) . The question this study is concerned with is therefore : does the explicit treatment of parameter uncertainty via the posterior parameter distribution p ( w | D ) elicit provably high uncertainty OOD ? Note , that we do not consider the problem of outlier detection within the training set ( Zimek & Filzmoser , 2018 ) , but rather ask whether a deployed model is able to know what it does not know . 3 BACKGROUND . In this section , we briefly introduce the concepts on which we base our argumentation in the coming sections . We start by introducing BNNs , which rely on approximate inference . We then discuss that in the non-parametric limit and under a certain choice of prior , a BNN converges to a Gaussian process , an alternative Bayesian inference framework where exact inference is possible . The connection between BNNs and Gaussian processes will later allow us to make interesting conjectures about OOD behavior . Finally , we introduce generalization bounds from the PAC-Bayes framework , which we will later use to argue that generalization and OOD detection can be conflicting objectives . 3.1 BAYESIAN NEURAL NETWORKS . In supervised deep learning , we typically construct a likelihood function from the conditional density p ( y | f ( x ; w ) ) , parameterized by a neural network f ( x ; w ) , and the training data D = { ( xi , yi ) } ni=1 . In BNNs , this is used to form the posterior distribution of all likely network parameterizations : p ( w | D ) ∝ ∏n i=1 p ( yi | f ( xi ; w ) ) p ( w ) , where p ( w ) is the prior distribution over weights . Crucially , when making a prediction with the Bayesian approach on a test point x∗ , we do not only use a single parameter configuration ŵ but we marginalize over the whole posterior , thus taking all possible explanations of the data into account : p ( y∗ | x∗ , D ) = ∫ p ( y∗ | f ( x∗ ; w ) ) p ( w | D ) dw . 3.2 GAUSSIAN PROCESS . Gaussian processes are established Bayesian machine learning models that , despite their strong scalability limitations , can offer a powerful inference framework . Formally ( Rasmussen , 2004 ) : Definition 1 A Gaussian process ( GP ) is a collection of random variables , any finite number of which have a joint Gaussian distribution . Compared to parametric models , GPs have the advantage of performing inference directly in function space . A GP is defined by its mean m ( x ) = E [ f ( x ) ] and covariance C ( f ( x ) , f ( x′ ) ) = E [ ( f ( x ) −m ( x ) ) ( f ( x′ ) −m ( x′ ) ) ] . The latter can be specified using a kernel function k : Rd × Rd → R and implies a prior distribution over functions : p ( f |X ) = N ( 0 , K ( X , X ) ) , ( 1 ) where K ( X , X ) ij = k ( xi , xj ) is the kernel Gram matrix on the training inputs X , and m ( x ) has been chosen to be 0 . When observing the training data , the prior is reshaped to place more mass in the regions of functions that are more likely to have generated them , and this knowledge is then used to make predictions on unseen inputs X ? . In probabilistic terms , this operation corresponds to conditioning the joint Gaussian prior : p ( f∗|X∗ , X , f ) . To model the data more realistically , we assume to not have direct access to the function values but to noisy observations : y = f ( x ) + with ∼ N ( 0 , σ2Id ) . This assumption is formally equivalent to a Gaussian likelihood p ( y|f ) = N ( y|f , σ2 ) .2 The conditional distribution on the noisy observation can then be written as ( Rasmussen , 2004 ) : p ( f∗ | X∗ , X , y ) = ∫ dfp ( f∗ | X∗ , X , f ) p ( f | X , y ) = N ( f̄∗ , C ( f∗ ) ) f̄∗ = K ( X∗ , X ) [ K ( X , X ) + σ 2I ] −1y C ( f∗ ) = K ( X∗ , X∗ ) −K ( X∗ , X ) [ K ( X , X ) + σ2I ] −1K ( X , X∗ ) ( 2 ) where p ( f | X , y ) = p ( y|f ) p ( f |X ) ) p ( D ) with p ( D ) being the marginal likelihood . RBF kernel . A commonly used kernel function for GPs is the squared exponential ( RBF ) : k ( x , x′ ) = exp ( − 1 2l2 ‖x− x′‖22 ) , ( 3 ) with the length scale l being a hyperparameter . It is important to notice that the covariance between outputs is written exclusively as a function of the distance between inputs . As a consequence , points that are close in the Euclidean space have unitary covariance that decreases exponentially with the distance . As we will see in Sec . 4 , this feature has important implications for OOD detection . The relation of infinite-width BNNs and GPs . The connection between neural networks and GPs has recently gained significant attention . Neal ( 1996 ) showed that a 1-hidden layer BNN converges to a GP in the infinite-width limit . More recently , the work by Lee et al . ( 2018 ) and de G. Matthews et al . ( 2018 ) extended this result to deeper networks , called neural network Gaussian processes ( NNGP ) . Crucially , the kernel function of the related GP strictly depends on the used activation function . To better understand this connection we consider a fully connected network with L layers l = 0 , . . . , L with width Hl . For each input we use xl ( x ) to represent the post-activation with x0 = x and f l the pre-activation so that f li ( x ) = b l−1 i + ∑Hl−1 j=1 w l−1 ij x l−1 j with x l j = h ( f l j ) and h ( · ) a point-wise activation function . Furthermore , the weights and biases are distributed according to blj ∼ N ( 0 , σ2b ) and wlij ∼ N ( 0 , σ2w Hl ) . Given the independence of the weights and biases it follows that the post-activations are independent as well . Hence , the central limit theorem can be applied , and for Hl−1 →∞ we obtain f l ( x ) ∼ GP ( 0 , Cl ) . The covariance Cl and therefore the prior over functions is specified by the kernel induced by the network architecture ( Lee et al. , 2018 ) : Cl ( x , x′ ) = kl ( x , x′ ) ︷ ︸︸ ︷ E [ f li ( x ) f l i ( x ′ ) ] − =0︷ ︸︸ ︷ E [ f li ( x ) ] E [ f li ( x ) ] = σ2b + σ 2 wE ( f l−1i ( x ) , f l−1i ( x′ ) ) ∼GP ( 0 , Kl−1 ) [ h ( f l−1i ( x ) ) , h ( f l−1i ( x ′ ) ) ] , ( 4 ) with Kl being the 2 × 2 covariance matrix formed by kl ( · , · ) using x and x′ . For the input layer , where no activation function is applied , the kernel is simply given by k0 ( x , x′ ) = σ2b + σ 2 w ( xTx′ d ) . For the remaining layers , instead , the kernel expression is determined by the non-linearity . Some activation functions like sigmoid or hyperbolic tangent do not admit a known analytical form and therefore require a Monte Carlo estimate of Eq . 4 ( cf . Fig . S3 ) . For others , Eq . 4 can be computed in closed form ( e.g. , Williams , 1997 ; Cho & Saul , 2009 ; Tsuchida et al. , 2018 ; Pang et al. , 2019 ; Pearce et al. , 2020 ) . We list some kernel functions that are important for this study below , such as the one for ReLU networks ( Cho & Saul , 2009 ) : klReLU ( x , x ′ ) = σ2b + σ2w 2π √ kl−1 ( x , x ) kl−1 ( x′ , x′ ) ( sin θl−1x , x′ + ( π − θ l−1 x , x′ ) cos θ l−1 x , x′ ) θlx , x′ = cos −1 ( kl ( x , x′ ) √ kl ( x , x ) kl ( x′ , x′ ) ) ( 5 ) Interestingly , for some non-linearities a kernel that carries similar properties as the RBF in Eq . 3 can be induced by neural networks . This is the case for the cosine activation function which has the 2Note , that the Gaussian likelihood assumption is commonly applied to neural network regression through the use of a mean-squared error loss . following closed form solution in the single hidden layer case ( Pearce et al. , 2020 ) : k1cos ( x , x ′ ) = σ2b + σ2w 2 ( exp ( −σ 2 w‖x− x′‖22 2d ) + exp ( −σ 2 w‖x+ x′‖22 2d − 2σ2b ) ) ( 6 ) Similarly , also the exponential activation function as deployed in RBF networks ( Broomhead & Lowe , 1988 ) leads to similar properties . This particular class of neural networks defines computational units as linear combinations of radial basis functions f ( x ) = ∑H j=1 wj exp ( − 12σ2g ‖x− µj‖ 2 ) + b . These networks , in the infinite-width case and when assuming a Gaussian prior over the centers µj ∼ N ( 0 , σ2µI ) and wj ∼ N ( 0 , σ2w ) , also converges to a Gaussian process with an analytical kernel function : k1rbfnet ( x , x ′ ) = σ2b + σ 2 w ( σe σµ ) d exp ( − ‖x‖ 2 2σ2m ) exp ( − ‖x− x ′‖2 2σ2s ) exp ( − ‖x ′‖2 2σ2m ) ( 7 ) where 1/σ2e = 2/σ 2 g + 1/σ 2 µ , σ 2 s = 2σ 2 g + σ 4 g/σ 2 µ and σ 2 m = 2σ 2 µ + σ 2 g . The equivalence with the RBF kernel is explicit in the limit σ2µ →∞ ( Williams , 1997 ) .

Summary: The manuscript challenges the widely believed assumption that Bayesian neural networks (BNNs) are well suit for out-of-distribution (OOD) detection, by showing empirical results obtained using infinite-width (allowing the exact inference, because a network can be equivalently represented by a GP with the included kernel) and finite-width networks (requiring the approximate inference). The manuscript provides several observations in line with this purpose (to disclose potential problems when BNNs are used for ODD detection).  For example, the exact inference in infinite-width networks does not necessarily lead to desirable OOD behavior, e.g., the posterior function standard deviation obtained from an infinite-width 2-layer ReLU does not reflect the underlying data generating process and this uncertainty is not suit for OOD detection. In addition, this observation is consistent with the corresponding finite-width networks.