Locally Most Powerful Bayesian Test for Out-of-Distribution Detection using Deep Generative Models
Authors: Keunseo Kim
JunCheol Shin
Heeyoung Kim
Venue: NeurIPS 2021 Poster (2021)
Keywords: Bayesian test
deep generative models
out-of-distribution detection

Abstract: Several out-of-distribution (OOD) detection scores have been recently proposed for deep generative models because the direct use of the likelihood threshold for OOD detection has been shown to be problematic. In this paper, we propose a new OOD score based on a Bayesian hypothesis test called the locally most powerful Bayesian test (LMPBT). The LMPBT is locally most powerful in that the alternative hypothesis (the representative parameter for the OOD sample) is specified to maximize the probability that the Bayes factor exceeds the evidence threshold in favor of the alternative hypothesis provided that the parameter specified under the alternative hypothesis is in the neighborhood of the parameter specified under the null hypothesis. That is, under this neighborhood parameter condition, the test with the proposed alternative hypothesis maximizes the probability of correct detection of OOD samples. We also propose numerical strategies for more efficient and reliable computation of the LMPBT for practical application to deep generative models. Evaluations conducted of the OOD detection performance of the LMPBT on various benchmark datasets demonstrate its superior performance over existing OOD detection methods.

Locally Most Powerful Bayesian Test for Out-of-Distribution Detection Using Deep Generative Models
Keunseo Kim Juncheol Shin Heeyoung Kim
Department of Industrial and Systems Engineering, KAIST (kkseo91, jczy0402, heeyoungkim) Cwannn.aaaaiaaaa.aa ac kr
Abstract
Several cutt-o-ddistrbuttion (OOD) detection scores have been recently proposed for deep generative models because the direct use of the likelihood threshold for OOD detection has been shown to be problematic. In this paper, we propose a new OOD score based on a Bayesian hypothesis test called the locally most powerful Bayesian test (LMPBT). The LMPBT is locally most powerful in that the alternative hypothesis (the representative parameter for the OOD sample) is specified to maximize the probability that the Bayes factor exceeds the evidence threshold in favor of the alternative hypothesis provided that the parameter specified under the alternative hypothesis is in the neighborhood of the parameter specified under the null hypothesis. That is, under this neighborhood parameter condition, the test with the proposed alternative hypothesis maximizes the probability of correct detection of OOD samples. We also propose numerical strategies for more efficient and reliable computation of the LMPBT for practical application deep generative models. Eyaluations conducted of the OOD detection performance the LMPBT on various benchmark datasets demonstrate its superior performance over existing OOD detection methods.
Introduction
In several real-world applications of deep learning models, detecting anomalous samples that signifi- cantly deviate from the distribution of the training data, that is, citto---istribution (OOD) samples, crucial for reliable decision making, and various OOD detection methods have been studied in this regard. Deep generative models 9 g 17 have also been investigated for OOD detection as an intuitive strategy owing to their ability to evaluate the likelihood of a test sample. However, recent studies have shown that deep generative models can assign higher likelihoods OOD samples than in-distribution samples [12,, causing the direct use of raw likelihoods for OOD detection to be problematic.
To address this challenge, several OOD scores have been recently proposed for deep generative models and have exhibited effective OOD detection performance [1 16 18]. Ren al. [1] proposed the use of the ratio the likelihood obtained from a model trained using pure input data to that obtained from background model trained using noise-perturbed input data as OOD score. Xiao et al. [18] proposed a likelihood regret score that can be calculated as the difference between the likelihood obtained with the optimized parameters for a test sample and that approximated by the VAE. Serrà et al. 16 proposed a penalized log-likelihood with an input complexity as an OOD score, where the input complexity can be computed as the normalized size of the compressed input image.
In this paper, we propose a new OOD detection method that is optimal in some sense in the framework of Bayesian hypothesis test-inspired by a uniformly most powerful Bayesian test [ and locally
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

most powerful test [14]. The Bayesian hypothesis test is based on the ratio of the posterior odds that the alternative hypothesis is true, given the observed data [3]. We formulate the OOD detection problem as a Bayesian test with the null hypothesis that a test sample is an in-distribution sample and the alternative hypothesis that the test sample an OOD sample. Specifically, the null hypothesis specified to represent the model trained on the in-distribution training set by the maximum likelihood estimation, as typically performed in previous studies. Against this specific null hypothesis, we propose to specify the alternative hypothesis to represent the model trained the expanded training set with the test sample. Then, we show that the test with our proposed alternative hypothesis is locally most powerful in that the alternative hypothesis is specified that the Bayes factor exceeds the evidence threshold in favor of the alternative hypothesis among all alternative hypotheses that have the model parameters in the neighborhood of the model parameter specified under the null hypothesis. That is, under this neighborhood parameter condition, the test with the proposed alternative hypothesis maximizes the probability of the correct detection of OOD) samples. The proposed test is called the locally most powerful Bayesian test (LMPBTT).
as maximize the probability
Because we specify the alternative hypothesis of the LMPBT for test sample to represent the model trained on the expanded training set with that test sample, we need to re-train the model for each test sample. However, the computational cost of this retraining is significant. To address this issue, we adopt the upweighting method [0]. Using this method, we analyze the influence of upweighting a test sample on the parameter change and estimate the new parameters without retraining.
However, computational issues are encountered when the upweighting method is directly applied in practice. First, the upweighting method involves the Hessian matrix, which is assumed to have all positive eigenvalues. However, the loss function of deep generative models is known to be nonconvex, and finding the global optimum is infeasible. Consequently, the Hessian can have negative cigenvalues. Second, the upweighting method requires the computation the inverse of the Hessian matrix. However, deep generative models, which have numerous parameters, incur large costs for the calculation of the Hessian and its inverse.
To address these issues, we use low-rank approximation 2 20 13 of the Hessian. Consequently, we can ensure that the approximated Hessian has all positive eigenvalues and can significantly reduce the computational cost for calculating the inverse of the Hessian. We evaluated the performance of the LMPBT using deep generative models on benchmark datasets and demonstrated more effective OOD detection performance than competing methods. In summary, the contributions of this study are as follows:
We propose new OOD detection method-the locally most powerful Bayesian test (LMPBT)- that maximizes the probability of correct detection of OOD samples under some conditions.
We address the computational issues encountered when practically implementing the LMPBT for deep generative models.
Evaluations conducted of the LMPBT using variational autoencoders (VAEs) [9] and Glows [ on benchmark datasets demonstrate state-of-the-an performance for OOD detection (the code for the LMPBT is available at http:///itta..cmkkkkuuuemmm/////
2 Background: Deep Generative Models
Generally, deep probabilistic generative models comprise two sub-models: a generative model and an inference model. By introducing a latent variable z, the generative model defines the generative process of x from z. whereas the inference model infers the distribution of x given For example, in flow-based generative models, the generative and inference models are defined using an invertible function S. which is composed of a sequence of invertible functions, f f1 f2 o Sv. The generative and inference processes between x and z can be expressed as
f2 h2
where the latent variable z has tractable density pa pz)... Then, the log probability density function of the model given a sample x is given by
1z
Th.-1 )/
Oh.
logpo (x = log (z) + log det
= log (z +
Σ log det
(1)
0x

---

where we define ho = X and h = 2.
In VAËS, the generative and inference processes modeled using the incodrr-decoder structure. We assume that x/2 ~ Po (x(22) and z p(z), where po is parameterized using deep neural networks with parameters 0 and p(z) is tractable prior distribution of The maximum likelihood estimate (MLE) of 0 requires maximizing the marginal likelihood pe (x) = J pe (x(z)p(z dz. However, this computation is intractable, and variational inference is used derive the evidence lower bound (ELBO) as follows:
po(x) 2 Eigo (z/x) [log po (x(2) DEL (z(x))ll(z2) ELBO,
(z x) approximates the true posterior pe (z x) and is parameterized by deep neural networks with variational parameters & and DXL is the Kullback Liebler (KL) divergence. By Eq. (2), the ELBO may be used as an approximation of the log- likelihoodd. Using
where the variational posterior [z)z))
the exact likelihood in Eq and the approximated likelihood in Eq. respectively, flow-bar based generative models and VAEs are typically trained by minimizing the negative log-likeldhod of the training dataset { (x1, X2, Xn) as training loss:
n
n
log L(0[x;)
(x:)).
(3)
3 Locally Most Powerful Bayesian Test for OOD Detection
We formulate the OOD detection problem as Bayesian hypothesis test. We compare the null hypothesis H, that a test sample x, is an in-distribution sample and the alternative hypothesis H1 that the test sample x, is an OOD sample. Specifically, consider simple null and alternative hypotheses as follows:
Ho 0 Oo,
H1 0 = 01,
where 8g and are representative parameters of deep generative model for the in-distribution and OOD samples, respectively, and 0 is set of parameters from the parameter space O
The Bayesian hypothesis test is based the ratio of the posterior odds that the alternative hypothesis true given the observed data 3 The posterior odds in favor of the alternative hypothesis equals the Bayes factor multiplied by the prior odds in favor of the alternative hypothesis. When the null and alternative hypotheses are both simple, the Bayes factor represents the ratio of the likelihoods of the data evaluated under the hypotheses. The posterior odds for our hypothesis test in Eq (4) for the test sample X can be computed as
((H|(x)) [x)] P(H,) x,) L(Bo|x;) P(Ho)
P(H,)
(5)
Here, P(H:) is the prior probability for hypothesis H1, and L(O, /x)/ (On) ((0]):))) is the Bayes factor in favor of the alternative hypothesis, where L(0, (xe) and L(Og) xx) are the likelihoods of x, under Hi and Ho, respectively.
To complete the test, we must specify dg and 01, and the prior probabilities for Ho and H1. Typically, 0 is specified as the MLE, denoted by 0, which maximizes the log-likelihood function of the training dataset Xx.,X,,..., x follows:
argmaax OE
log [.( x,).
(6)
However, the specification of is difficult owing to the lack of information on the OOD samples. In fact, the previous likelihood-ratio-baeee OOD detection methods [5 16 18 can be expressed using the hypothesis test Eq. 4 with different specifications of 0, The OOD detection performance
affected by the specification of 01; thus, it is important to choose 0 optimally.
3

---

3.1 Proposed OOD score
In this paper, we propose theoretically grounded method to specify 8 optimally in some sense. Specifically, we specify 0, as the MLE of the expanded training set with the test sample denoted as 8, as follows:
0 = argmax log [((]xe) + Σ log [((
(7)
Then, for each test sample Xt. we need to solve the optimization problem in (7) to obtain ó,. However, retraining the model for each test sample is computation-intensive. Instead, use computational approximation for the solution of (q. which allows us to obtain 6 easily using the already obtained Ó. Specifically, we adopt the upweighting method [I0] for the approximation. This method analyzes the influence of a certain training data point on the parameter change by upweighting the loss of that point. It is known that the influence of upweighting the loss of a training data point by scale is given by
dô. de 1--0 - H Ve(
(8)
where H3- = - E V-( log T(()x:)) is the Hessian at i and Vol-log T(( is the loss gradient at x with respect f. In case, the training dataset X,, does not contain the test sample Xy; thus, the weight of X₂ in the training loss in (3) can be regarded as zero. Then, adding the loss of x, to the training loss for X, is equivalent to upweighting the loss of x, by 1/n. Finally, 6, can be approximated as
1 vo( log L(0xx)).
0. m 02 = 0
Finally, with the specification of 0g- = and 0, = 0, with the additional assumption of equal prior probabilities for Hy and H1 (i.e., 1HH)) P(H2) 0.5), we propose the log of the posterior odds in (5) as an OOD score for test sample x,, denoted by S(x,), as follows:
S(x,) = - log (0) x.) + log (().)x.).
(10)
In the next section, we show that the test using the proposed OOD score in Eq. 10) maximizes the probability that the alternative hypothesis is accepted when it is true (i.e., the probability of the correct OOD sample detection) provided that the parameter specified under the alternative hypothesis is in the neighborhood of the parameter specified under the null hypothesis.
3.2 Locally most powerful Bayesian test for OOD detection
Inspired by the uniformly most powerful Bayesian test 7 and the locally most powerful test 14 define the locally most powerful Bayesian test (LMPBT) for OOD detection.
Definition 3.1 (Locally most powerful Bayesian test (LMPBT) for OOD detection). Consider the OOD detection test in Eq. (4) for test sample X₃. Let O, be the true parameter associated with x₃- The locally most powerful Bayesian test for evidence threshold
O, denoted by LMPBT(7), in favor of the alternative hypothesis H₃ : 8 = 0₁ against a fixed null hypothesis Ho 0 = 0b. is a Bayesian hypothesis test in which the Bayes factor for the test satisfies the following inequality for any 6g E θ and all alternative hypotheses H( 0 = 8i. under the condition 0 < d(0', 6o) < 6, where 8 is a sufficiently small constant and d is measure of the distance:
L( (x,)
L(0g(x)) xx))
Po.
7
Po_
(11)
L(Gg x+)
That is, with fixed null hypothesis Ho = 0o. LMPBT(7) is Bayesian test in which the alternative hypothesis H₃ : 0 = 81 is specified to maximize the probability that the Bayes factor L(0, (x.) / (0o xx) exceeds the evidence threshold for all possible values of the true parameter 8, associated with test sample x,, under the condition that 02 is in the neighborhood of tg.
Proposition The test using the 0OD score in Eq. 10 LMPBT(7), given a sufficiently large training dataset.

---

Proof. We prove that, in our OOD detection test with the specification of 6g = 0 and 01 = 0. for
the test sample x,, the Bayes factor L(0, ( xx) satisfies the inequality in Eq. 11 by showing that the Bayes factor in favor of the alternative H1 0 O, (i.e., L(0 xx)) [(0) (x)))))) ) is maximized
when 0 0, Under the condition that 02 is in the neighborhood of 6g 0, we can use the Taylor expansion of log L(0,
( xx.) at Ô as follows:
log |[0] = log [(0xx) + (0) - -0)"Vo log L(()x,) 2 10 -0) (-H - 0) + R(0)).
Here, R(O1) is the remainder term, which has an upper bound V/2 1|0 0 (Lagrange error bound 6)), where M is an upper bound of the third derivative of the log-likelihood for ||0| Ō11 < 6. With a sufficiently small 6, the first and second terms will dominate the remainder term because higher-order terms vanish much more rapidly, and the remainder term will be negligible. Then, the log- factor can be approximated
log [() - log [((]x,) = (01 (01-0) Vo log (((xx +
2 + - - 0).
1
(12)
To obtain 01 that maximizes the log-Bayes factor, we find 01 that makes the gradient of Eq. 12 equal to zero. That is,
Follog [0] - log (((xx) = Ve log L(0) xx) + (01 - (-H(- = O.
(13)
Subsequently, the solution to Eq. 13 obtained as (0) - 0) = - vo( ( log [(0)x,)) To ensure the condition 0 < d(O), 8) 8 for sufficiently small 6, we rescale (0) - 0) with a sufficiently small constant as (0] 6) = - -HH Vo(- log [(())))) By setting c = n with a sufficiently large n, we can obtain the desired result:
0 = θ - -H-'vol log L()x))) = E2.
n
In Definition 3.1 the Bayesian hypothesis test requires a predefined evidence threshold y. Practically, we can set the threshold as the specified (e.g., fifth) percentile in the distribution of the LMPBT score evaluated on the training set. Specifically, we can obtain the empirical distribution of the LMPBT score using "in-distribution" training samples. Then, we set specified percentile in the distribution of the LMPBT score as the threshold, depending on the desired false positive rate. For example, we can set the fifth percentile in the distribution of the LMPBT score the threshold, based the fact that a false positive rate of 0.05 is typically used as a default value in practice.
4 Computational Issues on the LMPBT
4.1 Low-rank approximation of the Hessian
When using the upweighting method in (9, Hg is assumed to be positive definite; that is, Hg assumed to have all positive eigenvalues. However, the loss function of deep neural networks is known to be non-convex, and finding the global optimum is infeasible. In fact, gradient-descent (GD) methods, which are generally used for deep generative models, are only guaranteed to find first-order stationary points, including saddle points [19. Theoretically, GD methods are always able
to escape saddle points, but they take an exponentially long time (4] In practice, á obtained by running GD methods with early stopping differs from the global optimum and could saddle point. Consequently, Hg could have negative eigenvalues.
To address this issue, we use low-rank approximation of the Hessian utilizing eigenvalue decom- position [3). Recent empirical studies show that the Hessians of neural networks are typically of a low rank and often have at least one negative eigenvalue []. We empirically demonstrate this phenomenon in the deep generative models in Figure IT, which shows the distribution of the eigen- values of H4 calculated from a Glow trained on the CTFAR-10 dataset (left) and VAE trained on the CIFAR- 10 dataset (right). We can observe that most of the eigenvalues zero, which indicates
5

---

10
10:4
Density (Log Scale)
10.4
10
Denolyy
1g-4
10
10a1
10-7
0.0
0.5
1.0
1.5
2.0
2.5 1ee
Eigenvalues
Eigervvalues
206
Figure 1: Distribution of the eigenvalues of Ha calculated from a Glow trained on the CIFAR-I0 dataset (left) and VAE trained on the CIFAR-I0 dataset (right). We used the stochastic Lanezos quadrature algorithm (0 to compute the full empirical spectral density of the eigenvalues. It is notable that most of the eigenvalues are centered around zero and negative eigenvalues exist.
that the Hessian has a low rank. Further, the Hessian has negative eigenvalues. Benefiting from the low-rank property the Hessian of deep generative models, we approximate the Hessian using a low-rank approximation with the top eigenvalues. Specifically, let the eigenvalue decomposition of the Hessian be given by
= UTU
yuu.
(14)
where p is the number of parameters, eigenvalues 7/ are sorted with respect their thgn is, Y > "y for < j_jjannd and u, are the corresponding eigenvectors. A low-rank approximation of the Hessian of rank is as follows:
i = = nuu.F
(15)
In practice, must determine the appropriate value of the hyperparameter for each experiment. In our experiments in Section 5.2 we investigate the change in the performance of the LMPBT with different specifications of Even the models with millions of parameters exhibit effective OOD performance using a low-rank approximation of the Hessian with only the top 20~330 eigenvalues.
Scalability
The computation of the Hessian and its inverse incurs costs proportional to the square and cube of the model parameter size, respectively. For deep generative models with numerous parameters, the computation of Eq. (Р). which involves the inverse of the Hessian, is not feasible. We can overcome this issue by computing the Hessian-vector product, rather than explicitly forming the Hessian matrix. The Hessian-vector product can be computed as follows (20]
0g =
Ogh
0z
0g
90
Hz.
(16)
00 00
00
a0
where = Va(-loge L(Ol-)) is the gradient with respect to 0, and z is a random vector. In Eq. [6 the first equality is obtained from the chain rule, the second equality from the independence and 0, and the third equality from the definition of the Hessian. It is notable that the Hessian-vector product only incurs a cost that is the same as that of one gradient backpropagation. Then, using the Hessian-vector product results, the top eigenvalues and the corresponding cigenvectors of the
Hessian and, hence, /l in Eq. (T5), can be easily obtained using power iteration 200
Another advantage of using the low-rank approximation of the Hessian in Eq. 15)
is that easily compute Eq. g rather than directly computing the inverse of the Hessian, by utilizing the following:
i - log (())xx))
xx))
mni V log L(0)x.).
(17)

---

Similar to the Hessian-vector product, Eq. (77 can be computed without computing the outer product u.u. and it only incurs a computational cost proportional the model parameter size.
5 Experimental Evaluation
In this section, we evaluate the performance of the LMPBT. Specifically, we show that the LMPBT yields superior OOD detection performance compared to competing OOD detection methods on various benchmark datasets.
5.1 Experimental setup
Datasets We used six benchmark image datasets datasets-namely. MNIST, FASHION-MNIST (FMNIST). CelebA, CIFAR-10, CIFAR-100, and SVHN and two synthetic datasets: Noise and Constant. To generate the Noise and Constant datasets, we adopted the sampling methods employed in [88] We resized the images such that their width and height were 32 pixels. In the experiments, we used three datasets, namely, FMNIST, CIFAR-10, and CIFAR-100, as the training datasets. Specifically, we used the training partitions of the datasets for training. In the experiments using FMNIST, three-channel images were converted to grayscale to match the number of channels with one-channel FMNIST images. In the experiments using CIFAR- 10 and CIFAR-IO6 100, one-channel images were duplicated three times to match the number of channels with three-channel CIFAR-I 10 and CIFAR-100 images. After training the deep generative models on the FMNIST, CIFAR- 10, and CIFAR-100 datasets, we used the test partition of other datasets for testing to evaluate the OOD detection performance.
Base deep generative models We selected two representative deep generative models, VAE [9] and Glow 8 to evaluate the performance of the LMPBTT To train the VAE, we used the Adam optimizer to maximize the ELBO in Eq. (2). For the Glow, we adopted the structure used [8] and used the used the top 20 eigenvalues for the Glow and top 30 eigenvalues for the VAE. We discuss the specification of the number of cigenvalues in more detail in the following section. In the VAE experiments, we fixed the decoder and calculated the Hessian with respect to the encoder. The experimental settings and network structures are described in Supplementary Material.
Adam optimizer to maximize the log-likelilne likelihood in Eq. (T). To approximate the Hessian,
Metrics To quantitatively evaluate the OOD detection performance, we employed the area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPR) (5). and false positive rate 80% true positive rate (FPRS0) as evaluation metrics [55].
Baseline OOD detection methods for comparison We employed deep generative model-based OOD detection methods as baselines for comparison. These include the likelihood regret (LR) score [18]. likelihood ratio (LLR) score [15), and input complexity (IC) method [16). as discussed in Section These OOD detection methods were evaluated for both the Glow and VAE. For the VAE, we re-implemented the experiments according to the experimental settings in [8] For the Glow, we also re-implemented the experiments according to the experimental settings in the original papers of the LLR [15] and IC [16]. Specifically, we used a PNG compressor to measure the input complexity
when computing the IC. When computing the LR score for each test sample, we started with the trained encoder of VAE or the trained Glow, and optimized its parameters for 100 steps.
5.2 Sensitivity and computational cost analysis for the LMPBT
Before the sensitivity analysis, we first visualize the OOD detection ability of the LMPBT, similar to [2] Figure 2 (left) shows the histogram of the LMPBT scores when a VAE trained on CIFAR-10 was tested on CIFAR- 10 as in-distribution dataset and SVHN an OOD dataset. We can observe that, if the LMPBT scores are higher, the corresponding test samples are more likely to be classified as OOD samples. As desired, the SVHN test samples (i.e., OOD samples) tended to be assigned higher LMPBT scores than the CIFAR-10 10 test samples.
Specification of low-rank approximation Figure 2 (b) shows the OOD detection performance (AUROC, AUPR, and FPR80) of the LMPBT according to the number of eigenvalues used for a low- rank approximation of the Hessian. The experiments were conducted using a VAE trained CIFAR- 10 and tested on CIFAR- 10 and SVHN. Although the model has 1.4 million parameters, achieved the best performance using only the top 30 eigenvalues. With less than 10 eigenvalues, the LMPBT
7

---

0.3
0.940
01AR10
0.7
0.935
0.120
AURCC and AUPR.
0.115
0.5
0,930
0.925
0.920
Auhoc
0.100
F7050
0.005
5
LMMBT
10
number Eigenvalues
Figure 2: (left) LMPBT scores obtained with a VAE trained on CIFAR-10 and tested on both CIFAR- 10 and SVHN. (right) OOD detection performance (AUROC, AUPR, and FPR80) versus the number of eigenvalues.
effectively detected OOD samples (AUROC: 0.913). This was possible because the Hessian was sparse. Interestingly, the performance does increase monotonically with the number of eigenvalues. This behavior may be explained by the identifiability issue of the overparameterized models. For overparameterized models, there are often "degenerate" directions present in the parameter space 111. which correspond to the smallest eigenvalues of the Hessian. The change in the parameters in degenerate directions leads to almost no difference in the outputs and causes an identifiability issue. Deep generative models can also suffer from this identifiability issue. However, this issue can be obviated by using low-rank approximation of the Hessian with the largest eigenvalues. Because changes in the output values in the degenerate directions be considered random noise, the test accuracy of a compressed model can sometimes be better than that the full model.
Computational cost The LMPBT requires the computation of the gradient and Hessian-vede vector prod- uet for each test sample, whereas the LLR and IC only require likelihood computation. Meanwhile, the LR requires several iterations for the optimization for each test sample. To determine whether the computational cost of the LMPBT acceptable, we compared it with that of the LR. In our experiments using GeForce RTX 2080 GPU and VAE trained on CIFAR-10, on average, the LMPBT and LRR required 0.1 and 0.30s to test a single sample, respectively. This demonstrates that the computational cost of the LMBPT is acceptable, although it is higher than those of the L.R and IC, which required 0.0125 s and 0.0128 s on average to test single sample, respectively.
Moreover, to evaluate the computational cost for computing the top r Hessian eigenvectors, we performed two sets of additional experiments. First, we compared the computational cost required for the Hessian eigenvalue decomposition with that for network parameter estimation, which can be considered baseline because network parameter estimation is equally performed for other OOD detection methods. In our additional experiments using FMNIST (1 x32x32) with VAEs, it took 2543 s to obtain the top 30 eigenvectors. This cost is relatively small (approximately 18%) compared with the cost of network parameter estimation (13448 s).
Second, we evaluated the scalability of Hessian eigenvalue decomposition. Specifically, we measured the computational cost for the eigenvalue decomposition of the Hessian (top 30 cigenvectors) for the datasets of various sizes with VAEs. We also calculated the ratio of the time cost to the number of parameters to investigate the relationship between the dimension of the Hessian and the computational cost. The results are listed in Table The ratios obtained were 0.0028, 0.0023, and 0.0025 s for various data sizes. The ratio remained similar, which indicates that the computational cost tends to increase linearly with the number of parameters. These results show that our method is scalable and can be applied to large-scale networks.
5.3 Performance comparison with other OOD scores
To evaluate the LMPBT, we compared the ROC curve obtained using the LMPBT with those using the LR, IC, and L.R for OOD detection with a VAE trained on CIFAR-10M and tested on CIFAR-100 and SVHN in Figure (a) and tested on CIFAR- 100 and CelebA in Figure 3l (b). In Figure] (a), the

---

Table 1: Computational costs for Hessian eigenvalue decomposition
Data size
x 16x 16
1x32x32
3x32x32
Number of parameters
504,904
1,074,760
1,337,928
Time cost
1,414 s (0.0028 s/dim)
2,543 s (0.0023 s/dim)
3,371
Time/ ParemParameter
(0.0025 s/dim)
1.0
1.0
0.8
0.8
True Postive Rata
Positive
0.6
0.6
0.2
BOC IMPRT
0.2
LUGAT
ROC: LLB
ROC:
0.0
ROC 1C
0.0
0.0
0.4
0.6
0.8
1.0
0.2
0.4 0.6
0.8
1.0
False
False
(a)
(b)
Figure 3: Comparison of ROC curves obtained using the LMPBT, LLR, IC, and LR for OOD detection with VAE trained the CIFAR- training dataset and (a) tested on the CIFAR-100 test dataset as in-distribution samples and SVHN test dataset as OOD samples and (b) tested on the CIFAR-100 test dataset as in-distribution samples and CelebA test dataset as OOD samples.
LBPBT (green line) shows higher true positive rates at all false positive rates than all other methods, including the IC (red purple line), which previously achieved state-of-the-art performance [8]. That is, the LMPBT universally outperformed the IC, regardless of the threshold for OOD detection. In contrast, the LLR (violet line) shows inferior performance compared to random guessing (blue line). This confirms that the LLR is not suitable for OOD detection when VAE is used [18]. Similarly, in Figure
3 (b) using the CelebA dataset, the LMPBT achieved superior OOD detection performance compared to all other methods. All scores showed inferior performance compared with the case using the SVHN dataset. However, the LMPBT still outperformed the other methods even in this difficult OOD task.
For an in-depth evaluation of the LMPBT, we measured its performance using the AUROC and compared it with those of the competing methods, LLR, IC, and LR. We report the average AUROC over five repeated experiments using different random seeds. In all the experiments, the standard error
of the AUROC was less than 0,001. A higher AUROC value indicates better performance. Tables [ (a) and (b) present the results for VAES trained on CIFAR- 10 and CIFAR- -00, respectively. and tested on various datasets, including Noise, Constant, FMNIST, SVHN, MNIST, and CelebA. We can
observe that the LMPBT achieved state-of-the-art performance in all cases. In particular, in the case of the model trained on CIFAR- 10 and tested on SVHN, the IMPBT outperformed (AUROC: 0.931) the IC (AUROC: 0.912), which previously exhibited state-of thhth--rtt performance [18]. Even for the complicated dataset, the LMPBT using the model trained CIFAR-10 10 showed superior performance (AUROC: 0.783) to that of the LR (AUROC: 0.714), which previously exhibited state-of-ther the--h--tttttttt art performance [8]. Similarly, the LMBPT exhibited significantly superior performance when the model was trained on CIFAR-100, a more complicated dataset. For example, the LMPBT using the model trained CIFAR-100 exhibited superior performance (AUROC: 0.683) to that of the LR (AUROC: 0.598).
CelebA test dataset,
We also present the results evaluated with AUPR and FPR80 in Supplementary Material. Further, we present the results using FMNIST as a training dataset with various test datasets considered in Table in Supplementary Material. Additionally, repeated the entire set of experiments using the

---

Table 2: Average AUROC OOD detection methods on various test datasets over five repeated
experiments using different random seeds.
(a) Trained CIFAR-10
(b) Trained CIFAR-100
Dataset
LMPBT
LLR
LR
Dataset
LMPBT
LLR
IC
LR
Noise
1.0
0.032
0.994
Noise
1
1
0.993
0.993
Constant
1.0
0.258
0,974
Constant
0.042
0.999
0.948
FMNIST
0.994
0.074
0.992
0.991
FMNIST
0.997
0.325
0.987
0.970
SVHN
0.931
0.193
0.912
0.875
SVHN
0.956
0.123
0.912
0.820
MNIST 0.998
0.008
0.994
0.998
MNIST
0.998
0.376
0.979
0.995
CelebA
0.783
0.465
0.641
0.714
CelebA
0.683
0.575
0.464
0.598
Glow, and present the results in Supplementary Material. In summary, the results in Supplementary Material show that the LMPBT consistently outperformed across different metrics and datasets.
Conclusion
In this paper, we proposed new OOD score for deep generative models based on Bayesian hypothesis test. Specifically, we proposed the LMPBT for maximizing the probability that the alternative hypothesis accepted when it is true among all alternative hypotheses that have model parameters in the neighborhood of the model parameter specified under the null hypothesis. The LMPBT effectively performed OOD detection on all the tasks evaluated. We also addressed practical computational issues in the implementation of the LMPBT for deep generative models. For future research, more efficient computation the LMPBT could be investigated; for example, recent second-order optimization methods could be used compute the inverse of the Hessian matrix or its approximation. The proposed method can support reliable decision making by automatically detecting anomalies in complex systems in various systems monitoring, fraud detection, and disease surveillance. However, there also a potential risk that the proposed method could be used tool identify and discriminate against minorities. To prevent this, we must ensure that the proposed method cannot be used for any purpose that could have negative social impacts.
problems, such as manufacturing
Acknowledgements
This work was supported by Samsung Electronics Co., Ltd (I0201209-08871-01) and the Na- tional Research Foundation of Korea (NRF) grant funded by the Korea Government (MSIT) (2018RICIB600551I).
References
[1] G. Alain, N. L. Roux, and P.-A. Manzagol. Negative eigenvalues of the hessian in deep neural networks. arXiv preprint arXiv:1902.023::0 0200000..2.6...... 2019.
[2] N. Andrei. Accelerated conjugate gradient algorithm with finite difference hessian/vector product approximation for unconstrained optimization. Journal of Computational and Applied Mathematics, 230(2):570-55)) 2009.
[3] M. Bernardo and R. Rueda. Bayesian hypothesis testing: A reference approach. International Statistical Review, 70(3)3351-37, 2002.
[4] S. S. Du, C. Jin, J. D. Lee, M. I. Jordan, A. Singh, and B. Poczos. Gradient descent can take exponential time escape saddle points. In Advances in neural information processing systems, pages 1067-1077, 2017.
[5] D. Hendrycks and K. Gimpel. baseline for detecting misclassified and out-of-distribbtioo examples in neural networks. arXiv preprint arXiv: 2016.
10

---

[6] D. Slghes-Hallett, A. M. Gleason, and W. G. McCallum. Calculus: Single and multivariable, 1998.
[7] V. E. Johnson. Uniformly most powerful bayesian tests. Annals of statistics, 41(4):17,6, 2013.
[8] D. P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances neural information processing systems, pages 10215-10224, 2018.
[9] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114, 2013.
[10] P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. arXiv preprint arXiv:170..444 2017.
[1]] W. J. Maddox, G. Benton, and A. G. Wilson. Rethinking parameter counting in deep models: Effective dimensionality revisited. arXiv preprint ar arXiv:2003.02139, 2020.
[12] E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and Lakshminarayaaan, Do deep generative models know what they don't 't know? arXiv preprint arXiv:/810.099 2018.
[13] T. O'Lecry-Rosebene N. Alger, and O. Ghattas. Low rank saddle free newton: Algorithm and analysis. arXiv preprint arXiv:2002.02889 2020.
[14] M. Omelka. The behavior of locally most powerful tests. Kybernetika, 4 41(6):699-712, 2005.
[15] J. Ren, P. Liu, E. Fertig, J. Snoek, R. Poplin, M. Depristo, J. Dillon, and B. Lakshminarayanan. Likelihood ratios for -iittr-diittitttion detection. In Advances in Neurul Information Process- ing Systems, pages 14707-14718, 2019.
[16] Serrå, Álvarez, Gómez, O. Slizovskaia, Núñez, and Luque. Input complexity and -iitt---iittiittionn detection with likelihood-basse generative models. arXiv preprint arXiv: 1480, 2019.
[17] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al. Conditional image generation with pixelcnn decoders. Advances in neural information processing systems, 29: 4790-4798, 2016.
[18] Z. Xiao, Q Yan, and Y. Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. arXiv preprint arXiv:2003.02977, 2020.
[19] Z. Yao, P. Xu, F. Koorsa-Ki and M. W. Mahoney. Inexact non-convex newton-type methods. arXiv preprint arXiv:
2018.
[20] Z. Yao, A. Gholami, K. Keutzer, and M. Mahoney, Pyhessian: Neural networks through the lens of the hessian. arXiv preprint 4rXiv:1912.07145. 2019.
[21] 7. Yao, A. Gholami, S. Shen, K. Keutzer, and M. W. Mahoney. Adahessian: An adaptive second order optimizer for machine learning. arXiv preprint 20000:000000000999 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section I
(b) Did you describe the limitations of your work? [Yes]] See Section 5.2 and Section 6
We discussed the limitations about the computation cost.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...

---

(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3
We provided the full of assumptions in detail.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 3.2 We
provided complete proof of the proposition.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Section [.]] and the supplemental material. We explain the dataset in Section 5.1 and submit the code the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.1 We specify the data splits and choice of the hyperparameters in detail.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [Yes] See Section 5.2 We repeated five experiments and reported the standard error.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 5.2 We repeated the computation time and specified the type of GPUs.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We used PyHessian
library and cited the "Z. Yao, A. Gholami, K Keutzer, M. Mahoney. PyHessian: Neural Networks Through the Lens of the Hessian".
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "iqqC0od8ah", "reviewer": "Reviewer_msUZ", "summary": "This paper proposes using a Bayesian hypothesis test for out-of-distribution detection with deep generative models.  The null hypothesis is that the newly observed data is in-sample and the alternative is that the new data is out-of-sample.  The test statistic is a likelihood ratio between the max likelihood model and the model fit to the given test input.  As this statistic is impractical to compute, the paper uses a perturbation of the MLE to find the parameters for the single sample test point.  The paper shows (Prop 1) that this test is the locally most powerful Bayesian test if the training set is sufficiently large.  Yet as the test requires computation of the inverse Hessian matrix, a low-rank approximation is necessary in practice.  Experiments are reported on benchmark image data sets (FashionMNIST, CIFAR, SVHN, CelebA, etc).  The proposed method performs well in terms of AUROC when compared to likelihood ratios and input complexity.  An ablation study of the rank of the Hessian approximation is also performs and shows the method is relatively insensitive to the rank.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Positives\n\n**(Mostly) well-motivated approach to detecting distribution shift**:  This paper adapts the well-founded techniques of Bayesian hypothesis testing to deep generative models.  As detecting distribution shift with DGMs, ideally, requires fast, per-point evaluation, this method seems to provide just that.  Although, see my question / critique regarding identifiability below.\n\n**Tuning computational overhead**:  As Figure 2 demonstrates, the method seems to work well—or even a bit better—when using a low-rank approximation to the Hessian matrix.  This allows the user to tune the method's computational overhead to fit the application.  This is important as detecting distribution shift often must be done in real time.  \n\n### Critiques / Questions\n\n**Local approximation and its validity for overparameterized models**:  While the motivating Bayesian hypothesis testing framework is undoubtedly motivated for well-identified models, it seems like identifiability could be an issue for deep generative models.  In neural networks, we know that there are often directions in parameter space around local minima that can leave the resulting model unchanged [[Maddox et al.; 2020]](https://arxiv.org/abs/2003.02139).  Thus, are parameterization issues such as this a worry when applying this test to deep generative models?  How do we know the local perturbation method won't detect these directions, thinking there is indeed distribution shift, while the model is actually invariant?  \n\n**Why is the AUROC best at around 30 eigenvalues?:**  I was surprised to see that the test's performance doesn't improve monotonically with the rank of the approximation.  Do you have any idea why this is?  You mention that the Hessian is sparse, explaining the good performance at low rank, but I don't think this would explain why performance degrades a bit as rank increases.  Could this possibly be due to the \"spurious\" directions in parameter space I mention above?\n\n**Remainder term in Proposition 1**:  Prop 1, how is the remainder term resolved?  Equation 12 is stated as an approximation, and I see how the 1/n term implies a good approximation when the data set is large.  But how do we know that the remainder term also can go to zero?  It seems like the paper is concerned about the convergence of the first and second order terms without discussing when they dominate the remainder?\n\n**Discussion of score matching and [Sharma et al. (UAI 2021)](https://arxiv.org/abs/2102.12567)**: The paper is light in its discussion of previous work on OOD detection for deep generative models, and there is no Related Work section.  I think previous work on score matching and the Fisher information approach of Sharma et al. [UAI 2021] should be discussed as I see these as related methods, using perturbations of the log density function to test for OOD-ness.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, as detecting distribution shift can improve the safety of AI systems, this method should have positive societal impact.  There are some limitations not mentioned in the paper (e.g. the issue of identifiability), which I mention above.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "iqX7y54ESpq", "reviewer": "Reviewer_ossS", "summary": "The paper proposes a new Bayesian hypothesis test for out-of-distribution detection (OOD) based on a trained generative model. The problem is formulated as a hypothesis testing on the parameter of the model where the null hypothesis $H_0$ posits that the model parameter is $\\theta_0$ and the alterternative $H_1$ states that the parameter is $\\theta_1$. The test statistic is the ratio of the posterior probabilities of the two (Bayes factor. See (5)). This way to formulate the problem is common and $\\theta_0$ is commonly set to the maximum likelihood estimate (MLE) on the training data (in-distribution samples).\n\nThe main contribution of this paper is in setting $\\theta_1$. The paper proposes to set it by MLE but on data constructed by combining the training data and the test point to test (i.e., so one more point added to the training data. See (7)). The paper shows that doing this leads to a test that satisfies a new notion called “locally most powerful Bayesian test” (LMPBT) given in Def 3.1. Since fully estimating the MLE on the augmented training data for each point to test is expensive, the paper further proposes a way to scale up, based on the use of the upweighting method from [9], and Hessian approximation (see (8) and Sec 4.1). The resulting MLE solution on the augmented data can be written as an additive perturbation (involving a low-rank Hessian) of the MLE estimate from the original training data. In this way, the MLE on the training data can be estimated only once, and reused for each OOD test point. In experiments, the new Bayesian test is shown to outperform competing approaches on standard benchmark datasets such as CIFAR10, CIFAR100, SVHN, MNIST, and Fashion MNIST.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n*Clarity*:\n\nThe paper is fairly well written. The problem setting and background are briefly given just enough to understand the contribution, which is good. Preciseness in mathematical statements can be improved. For instance, the use of $\\approx$ in the proof of Prop 1 makes it unclear about the effect of the omission of the remainder term in the Taylor expansion. It is also unclear what “sufficiently small $\\delta$” means in Def 3.1 (i.e., how small is small). Other than these, the main ideas presented and the details of the proposed method are easy to follow.\n\n*Originality*:\n\nThe use of Bayes factor for OOD is natural and has been considered before. The main contributions are \n1. Setting $\\theta_1$ by MLE on the augmented training data (training data + the test point);\n2. The use of upweighting ((9)) to avoid a full MLE computation for each point, as well as its further low-rank Hessian approximation (Sec 4.1) to scale up;\n3. The result in Prop 4.1 stating that setting $\\theta_1$ in this way results in a locally most powerful test.\n\nThe upweighting method is known from [9], and the low rank Hessian approximation is a commonly used operation especially in the optimization literature among others. In that sense, I see that the originality mainly comes from the idea in 1) and the result in 3). Overall I find that the paper has nicely put together these results and makes a practical approach.\n\n*Quality*:\n\nThere is some impreciseness in the statement of Prop 4.1, and Def 3.1 however. More in the questions below. Up to these impreciseness, the proposed Bayesian approach appears to be technically sound.\n\n\n*Significance*:\n\nThe proposed test gives strong empirical results, outperforming competing methods. Empirically, the approach is deemed significant, and may open up a new direction for OOD. There are some unclear points regarding experiments, as discussed below. Theoretical significance is lacking however. Besides showing that the approach is a locally most powerful Bayesian test (LMPBT), there is no further elaboration on the implications of this statement.  The definition of the LMPBT also appears to be proposed by the present work for the first time, tailored to fit the proposed approach (combining [6] and [12]).\n\n## Questions\n1. In the definition of LMPBT in Def 3.1, how small is small for $\\delta$?\n2. What are the implications of Proposition 1? That is, the test is a LMPBT. What can be said more about this? Is it the only test that satisfies this definition?\n3. In (9), the Hessian-gradient product term carries a factor of 1/n where n is the sample size. What happens asymptotically (i.e., when n goes to infinity)? Does it mean $\\tilde{\\theta}_t$ (MLE on the augmented data) would be the same as $\\hat{\\theta}_t$ (MLE on the training data)? What happens to the decision of the test in that case?\n4. In Fig 2b, why does the AUC curve fluctuate as $r$ the number of eigenvectors in approximating the Hessian increases? Naturally one would expect the approximation to get better as $r$ increases.\n5. In Fig 3, what does “tested on CIFAR-100 test data and SVHN” mean? Are they combined into one test set? I’m trying to see why the likelihood ratio LLR shows completely different behaviors in  Fig 3a and Fig 3b.\n\n—-\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "\nNo concerns\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Q3g2RqOIR5s", "reviewer": "Reviewer_6JoX", "summary": "This paper proposes a new method for out-of-distribution (OOD) detection using deep generative models. In particular, the paper leverages a Bayesian hypothesis testing framework, and is based the so-called „locally most powerful Bayesian test“, which enjoys certain theoretical guarantees (namely that it maximizes the probability of correct OOD detection) under some conditions. It is described how the intractable test can be efficiently implemented in practice (when used in conjunction with deep generative models), leveraging low-rank approximations of the involved Hessian matrix. Empirical evaluations demonstrate that the proposed approach compares favorably to alternative methods in a variety of settings, for both VAE and Glow models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Post Rebuttal**\n\nThe author response convincingly addressed the main concerns I had raised -- I therefore increase my score from 6 (weak acceptance) to 7 (acceptance).\n\n---\n\nThis manuscript proposes a new method to tackle the important OOD detection problem. It is therefore relevant to the NeurIPS community and has potentially high impact. The method is theoretically motivated by Bayesian hypothesis testing, is intuitively sensible and easy-to-understand. The experiments clearly demonstrate that the proposed approach performs strongly compared to other recently-developed OOD detection methods for deep generative models. However, the computational complexity of the method is not entirely transparent. The required Hessian approximation seems fairly expensive, which is only insufficiently analyzed. (See comments below for more details.) Adding such an analysis to the paper would significantly strengthen it. All in all, I am tending towards recommending acceptance of the paper — I am happy to strengthen my recommendation if the authors can convincingly address the raised issues.\n\n\nDetailed comments:\n- I like the idea of explicitly formulating the OOD detection as a Bayesian hypothesis test, which to my knowledge has not been done in prior work.\n- The proposed approach is conceptually simple, intuitively sensible and appears to be theoretically grounded and sound. The paper also provides some useful theoretical analysis of the properties of the Bayesian test used for OOD detection.\n- It is described how the (originally intractable) method can be implemented in practice by doing a low-rank approximation of the required full-network Hessian (which is implemented using power iteration) and using Hessian-vector products that avoid computation and storage of the full Hessian. While this approach seems reasonable, it is not clear how the computational complexity compares to that of other OOD detection methods; using power iteration to obtain the top eigenvectors/-values is expected to still be comparably expensive, so a more thorough and transparent analysis of the computational cost would improve the paper significantly (see also below for a the comment on the short analysis provided in the experiments).\n- They describe how their Bayesian testing framework generalizes previous likelihood-ratio based OOD detection methods, which is insightful.\n- The empirical evaluation convincingly demonstrates the effectiveness of the proposed approach in a variety of settings, as compared to strong baselines such as LR, LLR and IC, for different types of deep generative models (in particular VAEs and Glow).\n- They study how the rank of the low-rank approximation of the Hessian (i.e. the number of eigenvalues considered) affects the OOD detection performance, showing that even for deep neural network models, a small number of eigenvalues (i.e. 20-30) achieves the best performance. However, Fig 2 (right) is not entirely conclusive in that the performance seems to degrade for a larger number of estimated eigenvalues (i.e. a low-rank approximation with a higher rank). Clearly, in theory, performance should become better with more accurate Hessian estimation, so it would be great to know why this is not the case in practice, and I would appreciate the author’s comments on this. Is this simply result of numerical instabilities when using power iteration to estimate the top eigenvalues? If so, why is this not an issue in practice, and how would we choose the optimal number of eigenvalues in practice?\n- They also analyze the computational cost of the proposed method, which seems comparable to LR. It would be great to also report the computational cost of LLR and IC, which are likely much cheaper as they only require likelihood computations. Otherwise it is somewhat difficult to assess the compute-performance trade-off of the proposed approach, which is necessary to fairly compare different methods. This is the main issue that I see with the described method.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper describes its limitations only to some degree. As pointed out above, the computational complexity of the approach is not assessed sufficiently. Without an analysis of the compute-performance trade-off (of both this method and competing baselines), it is difficult to assess the significance of the method. Potential negative societal impact is addressed sufficiently.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "2DqEDk6hKEg", "reviewer": "Reviewer_Huvy", "summary": "**Summary**\nThis paper presents a novel OOD detection method for deep generative models. The key idea is to formulate OOD detection as a Bayesian hypothesis test, where the OOD score is the log of likelihood ratio under the original network parameters $\\theta_0$ and new parameters $\\theta_1$ when adding the test sample $\\mathbf{x}_t$. The overall testing framework is similar to previous works on likelihood ratio (Ren et al. 2019) and likelihood regret (Xiao et al. 2020). The novelty lies in the estimation of $\\theta_1$ under the alternative hypothesis. Specifically, the authors estimate new network parameters using the second-order Hessian information. The authors also propose strategies to reduce the computational cost. Evaluations on both CIFAR-10 and CIFAR-100 demonstrate superior performance. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n- Authors extensively evaluate the proposed method under two generative models (VAE and Glow) and various datasets.\n- The performance improvement over the prior state-of-the-art, name likelihood regret, is significant. \n- The test is mathematically grounded. \n- The authors spent efforts addressing two key challenges related to the computational issues in Section 4: (1) low-rank approximation of the hessian and (2) inverse of Hessian. The experimental results in L268-L274 also presented convincing evidence. \n- The writing is mostly clear but can be improved.\n\n**Weaknesses**:\n\n- Likelihood Regret [r1] seems to share the same spirit in terms of estimating the new network parameters when the test sample x_t is added to the training set.  [r1] finetunes the model with the test sample for a few iterations, which seems to be the upper bound of the authors' algorithm., i.e, the proposed method is the approximation of the fine-tuned parameters. An explanation and discussion about why the authors' method surpasses  [r1] are needed. \n- According to the authors' code, they need a preprocessing step to compute the top-k Hessian eigenvectors and eigenvalues of the original model using power iteration across the whole training set. The computation budget would be very high given this pre-processing approach. More discussion on this would be appreciated. \n- The computational overhead could be significantly higher when employing the proposed method on the high-resolution datasets and large models due to the computation of Hessian. While the reviewer understands it's a general concern shared by second-order methods, it'd be informative to test the limits of the approach. \n\nMinor: \nTypo in the caption of Table 6 in Appendix, it should be the results under CIFAR-100, not CIFAR-10.\n\n[r1] Z. Xiao, Q. Yan, and Y. Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. arXiv preprint arXiv:2003.02977, 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author does not include on in the main paper but this doesn't affect my scoring. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new method for OOD detection using deep generative models based on Bayesian hypothesis testing, that they refer to as the locally most powerful Bayesian test (LMPBT). Overall, the reviewers found the paper well-motivated and the experiments support the key claims.  During the discussion phase, reviewers ossS and 6JoX increased their score and recommended acceptance. Reviewers msUZ and Huvy leaned towards acceptance but raised some concerns in the initial review; after reading the author rebuttal, I think that the authors satisfactorily address most of these concerns. 

I recommend acceptance and encourage the authors to incorporate the reviewer feedback in the final version.

Additional comment:
While the paper shows that they outperform some existing methods such as IC, LR, LLR, I believe there are stronger published results (e.g. 
 DoSE https://arxiv.org/abs/2006.09273) that should probably be mentioned. 
Decision: Accept (Poster)