Uncertainty-based out-of-distribution detection requires suitable function space priors

1 INTRODUCTION . One of the challenges that the modern machine learning community is striving to tackle is the detection of unseen inputs for which predictions should not be trusted . This problem is also known as out-of-distribution ( OOD ) detection . The challenging nature of this task is partly rooted in the fact that there is no universal mathematical definition that characterizes an unseen input x∗ as OOD as discussed in Sec . 2 . Without such a definition , there is no foundation for deriving detection guarantees under verifiable assumptions . In this work , we consider OOD points as points that are unlikely under the distribution of the data p ( x ) and points are outside the support of p ( x ) . Importantly , we do not claim to provide a meaningful definition of OOD , but rather argue that the justification of an OOD method can only be assessed theoretically once such a definition is provided . While it appears natural to tackle the OOD detection problem from a generative perspective by explicitly modelling p ( x ) , this paper is solely concerned with the question of how justified it is to deploy the predictive uncertainty of a Bayesian neural network ( BNN ) for OOD detection . As recent developments forecast an increasing integration of deep learning methods into industrial applications , it becomes essential to provide the theoretical groundings that justify the use of uncertainty for OOD detection , a task that is crucial for safety-critical applications of AI and reliable prediction-making . When BNNs are used in supervised learning , the dataset is composed of inputs x ∈ X and targets y ∈ Y which are assumed to be generated according to some unknown process : D i.i.d.∼ p ( x ) p ( y | x ) .The goal of learning is to infer from D alone the distribution p ( y | x ) in order to make predictions on unseen inputs x∗ . In the case of deep learning , this problem is approached by choosing a neural network f ( · ; w ) parametrized by w , and predictions are made via the conditional p ( y | f ( x ; w ) ) ( also called the likelihood of w for given x , y ) . Assuming that the induced class of hypotheses contains some ŵ such that p ( y | f ( x ; ŵ ) ) = p ( y | x ) almost everywhere on the support of p ( x ) , Bayesian statistics can be used to infer plausible models p ( w | D ) under the observed data given some prior knowledge p ( w ) ( see MacKay ( 2003 ) ) . This parametric description together with the network implicitly induces a prior over functions p ( f ) ( Williams , 1997 ; Fortuin , 2021 ) . Believing in the validity of a subjective choice of prior ( O ’ Hagan , 2004 ) , Bayesian inference comes with a multitude of benefits as it is less susceptible to overfitting , allows to incorporate new evidence without requiring access to past data ( Farquhar & Gal , 2018 ) and provides interpretable uncertainties . The uncertainty captured by a BNN can be coarsely categorized into aleatoric and epistemic uncertainty . Aleatoric uncertainty is irreducible and intrinsic to the data p ( y | x ) . For instance , a blurry image might not contain enough information to identify unique object classes . On the other hand , a BNN also models epistemic uncertainty by maintaining a distribution over parameters p ( w | D ) .1 This distribution reflects the uncertainty about which hypothesis explains the data and can be reduced by observing more data . Arguably , aleatoric uncertainty is of little interest for detecting OOD inputs . However , the recent advent of overparametrized models which further extend the hypothesis class , deceptively motivated the idea that the epistemic uncertainty under a Bayesian framework might be intrinsically suitable to detect unfamiliar inputs , and therewith implying , that BNNs can be used for OOD detection . This conjecture is intuitively true if the following assumption holds : the hypotheses captured by p ( w | D ) must agree on their predictions for in-distribution samples but disagree for OOD samples ( Fig . 1a ) . Certain Bayesian methods satisfy this assumption , e.g . a Gaussian process regression with an RBF kernel ( cf . Sec . 4 ) . However , the uncertainty induced by Bayesian inference does not in general give rise to OOD capabilities . This can easily be verified by considering the following thought experiment ( Fig . 1b ) : Assume a model class with only two hypotheses g ( x ) and h ( x ) such that g ( x ) 6= h ( x ) ∀x and data being generated according to g ( x ) on a restricted domain . Once data is observed , we can commit to the ground-truth hypothesis g ( x ) and thus lose epistemic uncertainty in- and out-of-distribution alike . Finite-width neural networks , by contrast , form a powerful class of models , and are often put into context with universal function approximators ( Hornik , 1991 ) . But is this fact in combination with Bayesian statistics enough to attribute them with good OOD capabilities ? The literature often seems to imply that a BNN is intrinsically good at OOD detection . For instance , the use of OOD benchmarks when introducing new methods for approximate inference creates the false impression that the true posterior is a good OOD detector ( Louizos & Welling , 2017 ; Pawlowski et al. , 2017 ; Krueger et al. , 2017 ; Henning et al. , 2018 ; Maddox et al. , 2019 ; Ciosek et al. , 2020 ; D ’ Angelo & Fortuin , 2021 ) . Our work is meant to start a discussion among researchers about the properties of OOD uncertainty of the exact Bayesian posterior of neural networks . To initiate this , we contribute as follows : • We emphasize the importance of the prior in function space for OOD detection , which is induced by the choice of architecture ( Sec . 4 ) and weight space prior ( Sec . 5 ) . • We empirically show that exact inference in infinite-width networks under common architectural choices does not necessarily lead to desirable OOD behavior , and that these observations are consistent with results obtained via approximate inference in their finite-width counterparts . • We furthermore study the OOD behavior in infinite-width networks by analysing the properties of the induced kernels . Moreover , we discuss desirable kernel features for OOD detection and show that also neural networks can approximately carry these features . • We emphasize that the choice of weight-space prior has a strong effect on OOD performance , and that encoding desirable function space properties within unknown OOD domains into such prior is challenging . • We argue that there is a trade-off between good generalization and having high uncertainty on OOD data . Indeed , improving generalization by incorporating prior knowledge ( which is usually encoded in an input-domain agnostic manner ) can negatively impact OOD uncertainties . 1Note , that the Bayesian treatment of network parameters does not account for all types of epistemic uncertainty such as the uncertainty stemming from model misspecification ( Hüllermeier & Waegeman , 2021 ) . We , however , assume the model to be correctly specified . 2 ON THE DIFFICULTY OF DEFINING OUT-OF-DISTRIBUTION INPUTS . While the intuitive notion of OOD sample points ( or outliers ) is commonly agreed upon , for instance as `` an observation ( or subset of observations ) which appears to be inconsistent with the remainder of that set of data '' ( Barnett & Lewis , 1984 ) , a mathematical formalization of the essence of an outlier is difficult ( cf . SM A ) and requires subjective characterizations ( Zimek & Filzmoser , 2018 ) . Often , methods for outlier detection are designed based on an intuitive notion ( see Pimentel et al . ( 2014 ) for a review ) , and therewith only implicitly define a method-specific definition of OOD points . This also accounts for uncertainty-based OOD detection with neural networks , where a statistic of the predictive distribution that quantifies uncertainty ( e.g. , the entropy ) is used to decide whether an input is considered OOD ( Hendrycks & Gimpel , 2017 ) . Commonly , the parameters w of a neural network are trained using an objective derived from Ep ( x ) [ KL ( p ( y | x ) ||p ( y | f ( x ; w ) ) ) ] ( i.e. , loss functions based on the negative log-likelihood such as the cross-entropy or mean-squared error loss ) . Therefore , these networks only have calibrated uncertainties in-distribution with OOD uncertainties not being controlled for unless explicit training on OOD data is performed ( eg. , Hendrycks et al. , 2019 ) . The question this study is concerned with is therefore : does the explicit treatment of parameter uncertainty via the posterior parameter distribution p ( w | D ) elicit provably high uncertainty OOD ? Note , that we do not consider the problem of outlier detection within the training set ( Zimek & Filzmoser , 2018 ) , but rather ask whether a deployed model is able to know what it does not know . 3 BACKGROUND . In this section , we briefly introduce the concepts on which we base our argumentation in the coming sections . We start by introducing BNNs , which rely on approximate inference . We then discuss that in the non-parametric limit and under a certain choice of prior , a BNN converges to a Gaussian process , an alternative Bayesian inference framework where exact inference is possible . The connection between BNNs and Gaussian processes will later allow us to make interesting conjectures about OOD behavior . Finally , we introduce generalization bounds from the PAC-Bayes framework , which we will later use to argue that generalization and OOD detection can be conflicting objectives . 3.1 BAYESIAN NEURAL NETWORKS . In supervised deep learning , we typically construct a likelihood function from the conditional density p ( y | f ( x ; w ) ) , parameterized by a neural network f ( x ; w ) , and the training data D = { ( xi , yi ) } ni=1 . In BNNs , this is used to form the posterior distribution of all likely network parameterizations : p ( w | D ) ∝ ∏n i=1 p ( yi | f ( xi ; w ) ) p ( w ) , where p ( w ) is the prior distribution over weights . Crucially , when making a prediction with the Bayesian approach on a test point x∗ , we do not only use a single parameter configuration ŵ but we marginalize over the whole posterior , thus taking all possible explanations of the data into account : p ( y∗ | x∗ , D ) = ∫ p ( y∗ | f ( x∗ ; w ) ) p ( w | D ) dw . 3.2 GAUSSIAN PROCESS . Gaussian processes are established Bayesian machine learning models that , despite their strong scalability limitations , can offer a powerful inference framework . Formally ( Rasmussen , 2004 ) : Definition 1 A Gaussian process ( GP ) is a collection of random variables , any finite number of which have a joint Gaussian distribution . Compared to parametric models , GPs have the advantage of performing inference directly in function space . A GP is defined by its mean m ( x ) = E [ f ( x ) ] and covariance C ( f ( x ) , f ( x′ ) ) = E [ ( f ( x ) −m ( x ) ) ( f ( x′ ) −m ( x′ ) ) ] . The latter can be specified using a kernel function k : Rd × Rd → R and implies a prior distribution over functions : p ( f |X ) = N ( 0 , K ( X , X ) ) , ( 1 ) where K ( X , X ) ij = k ( xi , xj ) is the kernel Gram matrix on the training inputs X , and m ( x ) has been chosen to be 0 . When observing the training data , the prior is reshaped to place more mass in the regions of functions that are more likely to have generated them , and this knowledge is then used to make predictions on unseen inputs X ? . In probabilistic terms , this operation corresponds to conditioning the joint Gaussian prior : p ( f∗|X∗ , X , f ) . To model the data more realistically , we assume to not have direct access to the function values but to noisy observations : y = f ( x ) + with ∼ N ( 0 , σ2Id ) . This assumption is formally equivalent to a Gaussian likelihood p ( y|f ) = N ( y|f , σ2 ) .2 The conditional distribution on the noisy observation can then be written as ( Rasmussen , 2004 ) : p ( f∗ | X∗ , X , y ) = ∫ dfp ( f∗ | X∗ , X , f ) p ( f | X , y ) = N ( f̄∗ , C ( f∗ ) ) f̄∗ = K ( X∗ , X ) [ K ( X , X ) + σ 2I ] −1y C ( f∗ ) = K ( X∗ , X∗ ) −K ( X∗ , X ) [ K ( X , X ) + σ2I ] −1K ( X , X∗ ) ( 2 ) where p ( f | X , y ) = p ( y|f ) p ( f |X ) ) p ( D ) with p ( D ) being the marginal likelihood . RBF kernel . A commonly used kernel function for GPs is the squared exponential ( RBF ) : k ( x , x′ ) = exp ( − 1 2l2 ‖x− x′‖22 ) , ( 3 ) with the length scale l being a hyperparameter . It is important to notice that the covariance between outputs is written exclusively as a function of the distance between inputs . As a consequence , points that are close in the Euclidean space have unitary covariance that decreases exponentially with the distance . As we will see in Sec . 4 , this feature has important implications for OOD detection . The relation of infinite-width BNNs and GPs . The connection between neural networks and GPs has recently gained significant attention . Neal ( 1996 ) showed that a 1-hidden layer BNN converges to a GP in the infinite-width limit . More recently , the work by Lee et al . ( 2018 ) and de G. Matthews et al . ( 2018 ) extended this result to deeper networks , called neural network Gaussian processes ( NNGP ) . Crucially , the kernel function of the related GP strictly depends on the used activation function . To better understand this connection we consider a fully connected network with L layers l = 0 , . . . , L with width Hl . For each input we use xl ( x ) to represent the post-activation with x0 = x and f l the pre-activation so that f li ( x ) = b l−1 i + ∑Hl−1 j=1 w l−1 ij x l−1 j with x l j = h ( f l j ) and h ( · ) a point-wise activation function . Furthermore , the weights and biases are distributed according to blj ∼ N ( 0 , σ2b ) and wlij ∼ N ( 0 , σ2w Hl ) . Given the independence of the weights and biases it follows that the post-activations are independent as well . Hence , the central limit theorem can be applied , and for Hl−1 →∞ we obtain f l ( x ) ∼ GP ( 0 , Cl ) . The covariance Cl and therefore the prior over functions is specified by the kernel induced by the network architecture ( Lee et al. , 2018 ) : Cl ( x , x′ ) = kl ( x , x′ ) ︷ ︸︸ ︷ E [ f li ( x ) f l i ( x ′ ) ] − =0︷ ︸︸ ︷ E [ f li ( x ) ] E [ f li ( x ) ] = σ2b + σ 2 wE ( f l−1i ( x ) , f l−1i ( x′ ) ) ∼GP ( 0 , Kl−1 ) [ h ( f l−1i ( x ) ) , h ( f l−1i ( x ′ ) ) ] , ( 4 ) with Kl being the 2 × 2 covariance matrix formed by kl ( · , · ) using x and x′ . For the input layer , where no activation function is applied , the kernel is simply given by k0 ( x , x′ ) = σ2b + σ 2 w ( xTx′ d ) . For the remaining layers , instead , the kernel expression is determined by the non-linearity . Some activation functions like sigmoid or hyperbolic tangent do not admit a known analytical form and therefore require a Monte Carlo estimate of Eq . 4 ( cf . Fig . S3 ) . For others , Eq . 4 can be computed in closed form ( e.g. , Williams , 1997 ; Cho & Saul , 2009 ; Tsuchida et al. , 2018 ; Pang et al. , 2019 ; Pearce et al. , 2020 ) . We list some kernel functions that are important for this study below , such as the one for ReLU networks ( Cho & Saul , 2009 ) : klReLU ( x , x ′ ) = σ2b + σ2w 2π √ kl−1 ( x , x ) kl−1 ( x′ , x′ ) ( sin θl−1x , x′ + ( π − θ l−1 x , x′ ) cos θ l−1 x , x′ ) θlx , x′ = cos −1 ( kl ( x , x′ ) √ kl ( x , x ) kl ( x′ , x′ ) ) ( 5 ) Interestingly , for some non-linearities a kernel that carries similar properties as the RBF in Eq . 3 can be induced by neural networks . This is the case for the cosine activation function which has the 2Note , that the Gaussian likelihood assumption is commonly applied to neural network regression through the use of a mean-squared error loss . following closed form solution in the single hidden layer case ( Pearce et al. , 2020 ) : k1cos ( x , x ′ ) = σ2b + σ2w 2 ( exp ( −σ 2 w‖x− x′‖22 2d ) + exp ( −σ 2 w‖x+ x′‖22 2d − 2σ2b ) ) ( 6 ) Similarly , also the exponential activation function as deployed in RBF networks ( Broomhead & Lowe , 1988 ) leads to similar properties . This particular class of neural networks defines computational units as linear combinations of radial basis functions f ( x ) = ∑H j=1 wj exp ( − 12σ2g ‖x− µj‖ 2 ) + b . These networks , in the infinite-width case and when assuming a Gaussian prior over the centers µj ∼ N ( 0 , σ2µI ) and wj ∼ N ( 0 , σ2w ) , also converges to a Gaussian process with an analytical kernel function : k1rbfnet ( x , x ′ ) = σ2b + σ 2 w ( σe σµ ) d exp ( − ‖x‖ 2 2σ2m ) exp ( − ‖x− x ′‖2 2σ2s ) exp ( − ‖x ′‖2 2σ2m ) ( 7 ) where 1/σ2e = 2/σ 2 g + 1/σ 2 µ , σ 2 s = 2σ 2 g + σ 4 g/σ 2 µ and σ 2 m = 2σ 2 µ + σ 2 g . The equivalence with the RBF kernel is explicit in the limit σ2µ →∞ ( Williams , 1997 ) .

Summary: In this work, the authors challenge the assumption that underlies many recent works that Bayesian neural networks should be well-suited to out-of-distribution detection. In order to do this, the authors focus on a function-space view by examining the properties of infinite-width BNNs. They use this analysis to argue that BNNs may not necessarily be well-suited to OOD detection. They further argue that there is a tradeoff between OOD generalization and uncertainty, and finally propose an alternative method of validating OOD properties of models.