Title: Enemy is Inside: Alleviating VAE's Overestimation in Unsupervised OOD Detection
Venue: NeurIPS 2023
Accepted: no
Keywords: Unsupervised Out-of-Distribution Detection

VAE
Abstract: Deep generative models (DGMs) aim at characterizing the distribution of the training set by maximizing the marginal likelihood of inputs 
in an unsupervised manner, making them a promising option for unsupervised out-of-distribution (OOD) detection.
However, recent works have reported that DGMs often assign higher likelihoods to OOD data than in-distribution (ID) data, $\textit{i.e.}$, $\textbf{\textit{overestimation}}$, leading to their failures in OOD detection.
Although several pioneer works have tried to analyze this phenomenon, and some VAE-based methods have also attempted to alleviate this issue by modifying their score functions for OOD detection, the root cause of the $\textit{overestimation}$ in VAE has never been revealed to our best knowledge.
To fill this gap, this paper will provide a thorough theoretical analysis on the $\textit{overestimation}$ issue of VAE, and reveal that this phenomenon arises from two Inside-Enemy aspects: 1) the improper design of prior distribution; 2) the gap of dataset entropies between ID and OOD datasets.
Based on these findings, we propose a novel score function to $\textbf{A}$lleviate $\textbf{V}$AE's $\textbf{O}$verestimation $\textbf{I}$n unsupervised OOD $\textbf{D}$etection, named $\textbf{``AVOID''}$, which contains two novel techniques, specifically post-hoc prior and dataset entropy calibration.
Experimental results verify our analysis, demonstrating that the proposed method is effective in alleviating $\textit{overestimation}$ and improving unsupervised OOD detection performance.

1

Abstract

Deep generative models (DGMs) aim at characterizing the distribution of the training set by maximizing the marginal likelihood of inputs in an unsupervised manner, making them a promising option for unsupervised out-of-distribution (OOD) detection. However, recent works have reported that DGMs often assign higher likelihoods to OOD data than in-distribution (ID) data, i.e. , overestimation , leading to their failures in OOD detection. Although several pioneer works have tried to analyze this phenomenon, and some VAE-based methods have also attempted to alleviate this issue by modifying their score functions for OOD detection, the root cause of the overestimation in VAE has never been revealed to our best knowledge. To fill this gap, this paper will provide a thorough theoretical analysis on the overestimation issue of V AE, and reveal that this phenomenon arises from two Inside-Enemy aspects: 1) the improper design of prior distribution; 2) the gap of dataset entropies between ID and OOD datasets. Based on these findings, we propose a novel score function to A lleviate V AE's O verestimation I n unsupervised OOD D etection, named 'AVOID' , which contains two novel techniques, specifically post-hoc prior and dataset entropy calibration. Experimental results verify our analysis, demonstrating that the proposed method is effective in alleviating overestimation and improving unsupervised OOD detection performance.

1 Introduction

The detection of out-of-distribution (OOD) data, i.e. , identifying data that differ from the indistribution (ID) training set, is crucial for ensuring the reliability and safety of real-world applications [1, 2, 3, 4]. While the most commonly used OOD detection methods rely on supervised classifiers [5, 6, 7, 8, 9, 10, 11], which require labeled data, the focus of this paper is on designing an unsupervised OOD detector. Unsupervised OOD detection refers to the task of designing a detector, based solely on the unlabeled training data, that can determine whether an input is ID or OOD [12, 13, 14, 15, 16, 17, 18]. This unsupervised approach is more practical for real-world scenarios where the data lack labels.

Deep generative models (DGMs) are a highly attractive option for unsupervised OOD detection.

DGMs, mainly including the auto-regressive model [19, 20], flow model [21, 22], diffusion model

[23], generative adversarial network [24], and variational autoencoder (VAE) [25], are designed

to model the distribution of the training set by explicitly or implicitly maximizing the likelihood estimation of

p

(

x

)

for its input

x

without category label supervision or additional OOD auxiliary

data. They have achieved great successes in a wide range of applications, such as image and text 33

generation. Since generative models are promising at modeling the distribution of the training set, 34

they could be seen as an ideal unsupervised OOD detector, where the likelihood of the unseen OOD 35

data output by the model should be lower than that of the in-distribution data. 36

Unfortunately, developing a flawless unsupervised OOD detector using DGMs is not as easy as it 37 seems to be. Recent experiments have revealed a counterfactual phenomenon that directly applying 38 the likelihood of generative models as an OOD detector can result in overestimation , i.e. , DGMs 39 assign higher likelihoods to OOD data than ID data [12, 13, 17, 18]. For instance, a generative 40 model trained on the FashionMNIST dataset could assign higher likelihoods to data from the MNIST 41 dataset (OOD) than data from the FashionMNIST dataset (ID), as shown in Figure 6(a). Since OOD 42 detection can be viewed as a verification of whether a generative model has learned to model the 43 distribution of the training set accurately, the counterfactual phenomenon of overestimation not only 44 poses challenges to unsupervised OOD detection but also raises doubts about the generative model's 45 fundamental ability in modeling the data distribution. Therefore, it highlights the need for developing 46 more effective methods for unsupervised OOD detection and, more importantly, a more thorough 47 understanding of the reasons behind the overestimation in deep generative models. 48

49

50

51

52

53

54

55

56

57

58

To develop more effective methods for unsupervised OOD detection, some approaches have modified the likelihood to new score functions based on empirical assumptions, such as low- and high-level features' consistency [17, 18] and ensemble approaches [26]. While these methods, particularly the VAE-based methods [18], have achieved state-of-the-art (SOTA) performance in unsupervised OOD detection, none of them provides a clear explanation for the overestimation issue. To gain insight into the overestimation issue in generative models, pioneering works have shown that the overestimation issue could arise from the intrinsic model curvature brought by the invertible architecture in flow models [27]. However, in contrast to the exact marginal likelihood estimation used in flow and auto-regressive models, VAE utilizes a lower bound of the likelihood, making it difficult to analyze. Overall, the reasons behind the overestimation issue of V AE are still not fully understood.

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

In this paper, we try to address the research gap by providing a theoretical analysis of VAE's overestimation in unsupervised OOD detection. Our contributions can be summarized as follows:

Through theoretical analyses, we are the first to identify two factors that cause the overestimation issue of V AE: 1) the improper design of prior distribution; 2) the intrinsic gap of dataset entropies between ID and OOD datasets;

Focused on these two discovered factors, we propose a new score function, named 'AVOID' , to alleviate the overestimation issue from two aspects: 1) post-hoc prior for the improper design of prior distribution; 2) dataset entropy calibration for the gap of dataset entropies;

Extensive experiments demonstrate that our method can effectively improve the performance of VAE-based methods on unsupervised OOD detection, with theoretical guarantee.

2 Preliminaries

2.1 Unsupervised Out-of-distribution Detection

In this part, we will first give a problem statement of OOD detection and then we will introduce the detailed setup for applying unsupervised OOD detection.

Problem statement. While deploying a machine learning system, it is possible to encounter inputs from unknown distributions that are semantically and/or statistically different from the training data, and such inputs are referred to as OOD data. Processing OOD data could potentially introduce critical errors that compromise the safety of the system [1]. Thus, the OOD detection task is to identify these OOD data, which could be seen as a binary classification task: determining whether an input x is more likely ID or OOD. It could be formalized as a level-set estimation:

where S ( x ) denotes the score function, i.e. , OOD detector , and the threshold λ is commonly chosen to make a high fraction ( e.g. , 95%) of ID data is correctly classified [9]. In conclusion, OOD detection aims at designing the S ( x ) that could assign higher scores to ID data samples than OOD ones.

Setup. Denoting the input space with X , an unlabeled training dataset D train = { x i } N i =1 containing of N data points can be obtained by sampling i.i.d. from a data distribution P X . Typically, we treat the P X as p id, which represents the in-distribution (ID) [17, 27]. With this unlabeled training set, unsupervised OOD detection is to design a score function S ( x ) that can determine whether an input is ID or OOD. This is different from supervised OOD detection, which typically leverages a classifier that is trained on labeled data [4, 7, 9]. We provide a detailed discussion in Appendix A.

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

2.2 VAE-based Unsupervised OOD Detection

DGMs could be an ideal choice for unsupervised OOD detection because the estimated marginal likelihood p θ ( x ) can be naturally used as the score function S ( x ) . Among DGMs, VAE can offer great flexibility and strong representation ability [28], leading to a series of unsupervised OOD detection methods based on VAE that have achieved SOTA performance [17, 18]. Specifically, VAE estimates the marginal likelihood by training with the variational evidence lower bound (ELBO), i.e. ,

where the posterior q ϕ ( z | x ) is modeled by an encoder, the reconstruction likelihood p θ ( x | z ) is modeled by a decoder, and the prior p ( z ) is set as a Gaussian distribution N ( 0 , I ) . After well training the VAE, ELBO ( x ) is an estimation of the p ( x ) , which could be directly seen as the score function S ( x ) to do OOD detection. But the VAE would suffer from the overestimation issue, which will be introduced in the next section. More details and Related Work can be seen in Appendix B.

3 Analysis of VAE's overestimation in Unsupervised OOD Detection

We will first conduct an analysis to identify the factors contributing to V AE's overestimation , i.e. , the improper design of prior distribution and the gap between ID and OOD datasets' entropies. Subsequently, we will give a deeper analysis of the first factor to have a better understanding.

3.1 Identifying Factors of VAE's Overestimation Issue

Following the common analysis procedure [27], an ideal score function S ( x ) that could achieve good OOD detection performance is expected to have the following property for any OOD dataset:

where p id ( x ) and p ood ( x ) denote the true distribution of the ID and OOD dataset, respectively. A larger gap between these two expectation terms can usually lead to better OOD detection performance.

Using the ELBO ( x ) as the score function S ( x ) , we could give a formal definition of the repeatedly reported VAE's overestimation issue in the context of unsupervised OOD detection [12, 13, 17, 18].

Definition 1 (VAE's overestimation in unsupervised OOD Detection) . Assume we have a VAE trained on a training set and we use the ELBO ( x ) as the score function to distinguish data points sampled i.i.d. from the in-distribution testing set ( p id) and an OOD dataset ( p ood). When

it is called V AE's overestimation in unsupervised OOD detection.

With a clear definition of overestimation , we could now investigate the underlying factors causing the overestimation in VAE. After well training a V AE, we could reformulate the expectation term of ELBO ( x ) from the perspective of information theory [29] as:

because we have

where the I q ( x , z ) is mutual information between x and z and the q ( z ) is the aggregated posterior distribution of the latent variables z , which is defined by q ( z ) = E x ∼ p ( x ) q ϕ ( z | x ) . We leave the detailed definition and derivation in Appendix C.1. Thus, the gap G in Eq. (4) could be rewritten as

where the dataset entropy H p id ( x ) / H p ood ( x ) is a constant that only depends on the true distribution of ID/OOD dataset; the prior p ( z ) is typically set as a standard (multivariate) Gaussian distribution N ( 0 , I ) to enable reparameterization for efficient gradient descent optimization [25].

Through analyzing the most widely used criterion, specifically the expectation of ELBO reformulated in Eq. (8), for V AE-based unsupervised OOD detection, we find that there will be two potential factors that lead to the overestimation issue of V AE, i.e. , G ≤ 0 :

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

146

147

148

149

150

151

152

153

154

155

Factor I: The improper design of prior distribution p ( z ) . Several studies have argued that the aggregated posterior distribution of latent variables q ( z ) cannot always equal N ( 0 , I ) , particularly when the dataset exhibits intrinsic multimodality [28, 30, 31, 32]. In fact, when q ( z ) is extremely close to p ( z ) , it is more likely to become trapped in a bad local optimum known as posterior collapse [33, 34, 35], i.e. , q ϕ ( z | x ) ≈ p ( z ) , resulting in q ( z ) = ∫ x q ϕ ( z | x ) p ( x ) ≈ ∫ x p ( z ) p ( x ) = p ( z ) . In this situation, the posterior q ϕ ( z | x ) becomes uninformative about the inputs. Thus, the value of D KL ( q id ( z ) || p ( z )) could be overestimated, potentially contributing to G ≤ 0 .

Factor II: The gap between H p id ( x ) and H p ood ( x ) . Considering the dataset's statistics, such as the variance of pixel values, different datasets exhibit various levels of entropy. It is reasonable that a dataset containing images with richer low-level features and more diverse content is expected to have a higher entropy. As an example, the FashionMNIST dataset should possess higher entropy compared to the MNIST dataset. Therefore, when the entropy of the ID dataset is higher than that of an OOD dataset, the value of -H p id ( x ) + H p ood ( x ) is less than 0, potentially leading to overestimation .

3.2 More Analysis on Factor I

In this part, we will focus on addressing the following question: when is the common design of the prior distribution proper, and when is it not?

x

1

4

3

2

1

0

1

2

3

4

x

2

4

3

2

1

0

1

2

3

4

p

(

x

)

0.02

0.04

0.06

0.08

0.10

0.12

0.14

(a) Data distribution

p

(

x

)

3

2

1

0

1

2

3

z

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Probability

p(z)

(b) Prior

p

(

z

)

x

1

4

3

2

1

0

1

2

3

4

x

2

4

3

2

1

0

1

2

3

4

Estimated

p

(

x

) with

p

(

z

)

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

(c) Estimated

p

θ

(

x

)

3

2

1

0

1

2

3

z

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Probability

q(z)

(d) Posterior

q

(

z

)

When the design of prior is proper? Assuming that we have a dataset consisting of N data points 143 { x i } N i =1 , each of which is sampled from a given d -dimensional data distribution p ( x ) = N ( x | 0 , Σ x ) 144 as shown in Figure 1(a). Then we construct a linear V AE to estimate p ( x ) , formulated as: 145

where A , B , C , D , E , F , and σ are all learnable parameters and their optimal values can be obtained by the derivation in Appendix C.3. As the estimated distribution p θ ( x ) depicted in Figure 1(c), we can find that the linear V AE with the optimal parameter values can accurately estimate the p ( x ) through maximizing ELBO, i.e. , the overestimation issue is not present. In this case, Figures 1(b) and 1(d) indicate that the design of the prior distribution is proper, where the posterior q ( z ) equals prior p ( z ) .

Whenthe design of prior is NOT proper? Consider a more complex data distribution, e.g. , a mixture of Gaussians, p ( x ) = ∑ K k =1 π k N ( x | µ k , Σ k ) , K = 2 as shown in Figure 2(a), where π k = 1 /K and ∑ K k =1 µ k = 0 . We construct a dataset consisting of K × N data points, obtained by sampling N data samples { x ( k ) i } N,K i =1 ,k =1 from each component Gaussian N ( x | µ k , Σ k ) . The formulation of p ( z ) , q ϕ ( z | x ) , and p θ ( x | z ) is consistent with those in Eq. (9). More details are in Appendix C.2.

x

1

6

4

2

0

2

4

6

x

2

6

4

2

0

2

4

6

p

(

x

)

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

(a) Data distribution

p

(

x

)

3

2

1

0

1

2

3

z

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Probability

q(z)

p(z)

(b)

p

(

z

)

&

q

(

z

)

x

1

8

6

4

2

0

2

4

6

8

x

2

8

6

4

2

0

2

4

6

8

Estimated

p

(

x

) with

p

(

z

)

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

(c)

ˆ

p

θ

(

x

)

with

p

(

z

)

x

1

8

6

4

2

0

2

4

6

8

x

2

8

6

4

2

0

2

4

6

8

Estimated

p

(

x

) with

q

(

z

)

0.00

0.01

0.02

0.03

0.04

(d)

ˆ

p

θ

(

x

)

with

q

(

z

)

In what follows, we will provide a basic derivation outline for the linear V AE under the multi-modal 156 case. We can first obtain the marginal likelihood ˆ p θ ( x ; E , F , σ ) = ∫ p θ ( x | z ) p ( z ) = N ( x | F , EE ⊤ + 157

σ 2 I ) with the strictly tighter importance sampling on ELBO [36], i.e. , learning the optimal generative 158 process. Then, the joint log-likelihood of the observed dataset { x ( k ) i } N,K i =1 ,k =1 can be formulated as: 159

where M = EE ⊤ + σ 2 I and S = 1 KN ∑ K k =1 ∑ N i =1 ( x ( k ) i -F )( x ( k ) i -F ) ⊤ . After that, we could 160 explore the stationary points of parameters through the ELBO, which can be analytically written as: 161

162

163

164

165

166

167

168

169

170

171

The detailed derivation of parameter solutions in Eq. (10) and (11) can be found in Appendix C.4.

In conclusion of this case, Figure 2(b) illustrates that q ( z ) is a multi-modal distribution instead of p ( z ) = N ( z | 0 , I ) , i.e. , the design of the prior is not proper, which leads to overestimation as seen in Figure 2(c). However, as analyzed in Factor I, we found that the overestimation issue is mitigated when replacing p ( z ) in the KL term of the ELBO with q ( z ) , which is shown in Figure 2(d).

More empirical studies on the improper design of prior. To extend to a more practical and representative case, we used a 3-layer MLP to model q ϕ ( z | x ) and p θ ( x | z ) with p ( z ) = N ( 0 , I ) on the same dataset of the above multi-modal case. Implementation details are provided in Appendix C.5. After training, we observed that q ( z ) still differs from p ( z ) , as shown in Figure 3(a). The ELBO still suffers from overestimation , especially in the region near (0 , 0) , as shown in Figure 3(b).

2

1

0

1

2

z

0

50

100

150

200

Density

z

~

q

(

z

|

x

)

(a)

q

id

(

z

)

x

1

6

4

2

0

2

4

6

x

2

6

4

2

0

2

4

6

p

(

x

)

0.00

0.05

0.10

0.15

0.20

(b) Estimated

p

θ

(

x

)

2000

1800

1600

1400

1200

1000

Posterior

z

's log-probability in

(0,

I

)

0.000

0.002

0.004

0.006

0.008

Density

FashionMNIST train (ID)

FashionMNIST test (ID)

MNIST test (OOD)

(c) FashionMNIST (ID)

1600

1400

1200

1000

800

Posterior

z

's log-probability in

(0,

I

)

0.0000

0.0005

0.0010

0.0015

0.0020

0.0025

0.0030

0.0035

Density

CIFAR10 train (ID)

CIFAR10 test (ID)

SVHN test (OOD)

(d) CIFAR10 (ID)

Finally, we extend the analysis directly to high-dimensional image data. Since V AE trained on image 172 data needs to be equipped with a higher dimensional latent variable space, it is hard to visualize 173 directly. But please note that, if q id ( z ) is closer to p ( z ) = N ( 0 , I ) , z id ∼ q id ( z ) should occupy 174 the center of latent space N ( 0 , I ) and z ood ∼ q ood ( z ) should be pushed far from the center, leading 175 to p ( z id ) to be larger than p ( z ood ) . However, surprisingly, we found this expected phenomenon 176 does not exist, as shown in Figure 3(c) and 3(d), where the experiments are on two dataset pairs, 177 Fashion-MNIST(ID)/MNIST(OOD) and CIFAR10(ID)/SVHN(OOD). This still suggests that the 178 prior p ( z ) is improper, even q ood ( z ) for OOD data may be closer to p ( z ) than q id ( z ) . 179

180

181

182

183

Brief summary. Through analyzing overestimation scenarios from simple to complex, the answer to the question at the beginning of this part could be: the prior distribution p ( z ) = N ( 0 , I ) is an improper choice for VAE when modeling a complex data distribution p ( x ) , leading to an overestimated D KL ( q id ( z ) || p ( z )) and further raising the overestimation issue in unsupervised OOD detection.

184

185

186

187

188

4 Alleviating VAE's overestimation in Unsupervised OOD Detection

In this section, we develop the 'AVOID' method to alleviate the influence of two aforementioned factors in Section 3, including i) post-hoc prior and ii) dataset entropy calibration, both of which are implemented in a simple way to inspire related work and can be further investigated for improvement.

4.1 Post-hoc Prior Method for Factor I

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

To provide a more insightful view to investigate the relationship between q id ( z ) , q ood ( z ) , and p ( z ) , we use tSNE [37] to visualize them in Figure 4. The visualization reveals that p ( z ) cannot distinguish between the latent variables sampled from q id ( z ) and q ood ( z ) , while q id ( z ) is clearly distinguishable from q ood ( z ) . Therefore, to alleviate overestimation , we can explicitly modify the prior distribution p ( z ) in Eq. (8) to force it to be closer to q id ( z ) and far from q ood ( z ) , i.e. , decreasing D KL ( q id ( z ) || p ( z )) and increasing D KL ( q ood ( z ) || p ( z )) .

A straightforward modifying approach is to replace p ( z ) in ELBO with an additional distribution ˆ q id ( z ) that can fit q id ( z ) well after training, where the target value of q id ( z ) can be acquired by marginalizing q ϕ ( z | x ) over the

FashionMNIST test (ID)

MNIST test (OOD)

Prior

z

(0, I)

training set, i.e. , q id ( z ) = E x ∼ p id ( x ) [ q ϕ ( z | x )] . Previous study on distribution matching [30] has developed an LSTM-based method to efficiently fit q id ( z ) in the latent space, i.e. ,

Thus, we could propose a 'post-hoc prior' (PHP) method for Factor I, formulated as

which could lead to better OOD detection performance since it could enlarge the gap G , i.e. ,

Please note that PHP can be directly integrated into a trained V AE in a 'plug-and-play' manner.

4.2 Dataset Entropy Calibration Method for Factor II

While the entropy of a dataset is a constant that remains unaffected by different model settings, it is still an essential factor that leads to overestimation . To address this, a straightforward approach is to design a calibration method that ensures the value added to the ELBO of ID data will be larger than that of OOD data. Specifically, we denote the calibration term as C ( x ) , and its expected property could be formulated as

After adding the calibration C ( x ) to the ELBO ( x ) , we could obtain the 'dataset entropy calibration' (DEC) method for Factor II, formulated as

With the property in Eq. (15), we could find that the new gap G DEC becomes larger than the original gap G based solely on ELBO, as G DEC = G + E x ∼ p id ( x ) [ C ( x )] -E x ∼ p ood ( x ) [ C ( x )] > G , which should alleviate the overestimation and lead to better unsupervised OOD detection performance.

How to design the calibration C ( x ) ? For the choice of the function C ( x ) , inspired by the previous work [13], we could use image compression methods like Singular Value Decomposition (SVD) [38] to roughly measure the complexity of an image, where the images from the same dataset should have similar complexity. An intuitive insight into this could be shown in Figure 5, where the ID dataset's statistical feature, i.e. , the curve, is distinguishable to other datasets. Based on this empirical study, we could first propose a non-scaled calibration function, denoted as C non ( x ) . First, we could set the number of singular values as n id, which can achieve the reconstruction error | x recon -x | = ϵ in the ID training set; then for a test input x i , we use SVD to calculate the smallest n i that could also achieve a smaller reconstruction error ϵ , then C non ( x ) could be formulated as:

0

5

10

15

20

25

Number of singular values (

n

)

10

8

10

7

10

6

10

5

10

4

10

3

10

2

10

1

Reconstruction error (|

xre

x

|)

FashionMNIST train

FashionMNIST test

MNIST test

notMNIST test

SVHNGray test

CIFAR10Gray test

Omniglot test

KMNIST test

Constant

Random

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

which can give the ID dataset a higher expectation E x ∼ p id ( x ) [ C non ( x )] than that of other statistically different OOD datasets. More details to obtain C non ( x ) can be found in Appendix D.

4.3 Putting Them Together to Get 'AVOID'

By combining the post-hoc prior (PHP) method and the dataset entropy calibration (DEC) method, we could develop a new score function, denoted as S AVOID ( x ) :

To balance the importance of PHP and DEC terms in Eq. (18), we consider to set an appropriate scale for C ( x ) . For the scale of C ( x ) , if it is too small, its effectiveness in alleviating overestimation could be limited. Otherwise, it may hurt the effectiveness of the PHP method since DEC will dominate the value of 'AVOID'. Additionally, for statistically similar datasets, i.e. , H p id ( x ) ≈ H p ood ( x ) , the property in Eq. (15) cannot be guaranteed and we may only have E x ∼ p id ( x ) [ C non ( x )] ≈ E x ∼ p ood ( x ) [ C non ( x )] , in which case we could only rely on the PHP method. Thus, an appropriate scale of E x ∼ p id ( x ) [ C ( x )] , named ' C scale', could be derived by C scale = E x ∼ p id ( x ) [ PHP ( x )] ≈ H p id ( x ) , which leads to

Thus, when H p id ( x ) ≈ H p ood ( x ) and E x ∼ p id ( x ) [ C ( x )] ≈ E x ∼ p ood ( x ) [ C ( x )] , the PHP part of 'AVOID' could still be helpful to alleviate overestimation .

Motivated by the above analysis, we could implement the scaled calibration function, formulated as

5 Experiments

5.1 Experimental Setup

Datasets. In accordance with existing literature [17, 18, 39], we evaluate our method against previous works using two standard dataset pairs: FashionMNIST [40] (ID) / MNIST [41] (OOD) and CIFAR10 [42] (ID) / SVHN [43] (OOD). The suffixes 'ID' and 'OOD' represent in-distribution and out-ofdistribution datasets, respectively. To more comprehensively assess the generalization capabilities of these methods, we incorporate additional OOD datasets, the details of which are available in Appendix E.1. Notably, datasets featuring the suffix '-G' (e.g., 'CIFAR10-G') have been converted to grayscale, resulting in a single-channel format.

Evaluation and Metrics. We adhere to the previous evaluation procedure [17, 18], where all methods are trained using the training split of the in-distribution dataset, and their OOD detection performance is assessed on both the testing split of the in-distribution dataset and the OOD dataset. In line with previous works [1, 5, 44], we employ evaluation metrics including the area under the receiver operating characteristic curve (AUROC ↑ ), the area under the precision-recall curve (AUPRC ↑ ), and the false positive rate at 80% true positive rate (FPR80 ↓ ). The arrows indicate the direction of improvement for each metric.

Baselines. Our experiments primarily encompass two comparison aspects: i ) evaluating our novel score function 'A VOID' against previous unsupervised OOD detection methods to determine whether it can achieve competitive performance; and ii ) comparing 'AVOID' with VAE's ELBO to assess whether our method can mitigate overestimation and yield improved performance. For comparisons in i , we can categorize the baselines into three groups, as outlined in [18]: ' Supervised ' includes supervised OOD detection methods that utilize in-distribution data labels [1, 5, 9, 45, 46, 47, 48, 49]; ' Auxiliary ' refers to methods that employ auxiliary knowledge gathered from OOD data [13, 39, 44]; and ' Unsupervised ' encompasses methods without reliance on labels or OOD-specific assumptions [14, 17, 18, 26]. For comparisons in ii , we compare our method with a standard VAE [25], which also serves as the foundation of our method. Further details regarding these baselines and their respective categories can be found in Appendix E.2.

Implementation Details. The VAE's latent variable z 's dimension is set as 200 for all experiments with the encoder and decoder parameterized by a 3-layer convolutional neural network, respectively.

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

The reconstruction likelihood distribution is modeled by a discretized mixture of logistics [20]. For optimization, we adopt the same Adam optimizer [50] with a learning rate of 1e-3. We train all models in comparison by setting the batch size as 128 and the max epoch as 1000. All experiments are performed on a PC with an NVIDIA A100 GPU and our code is implemented with PyTorch [51]. More implementation details can be found in Appendix E.3.

5.2 Comparison with Unsupervised OOD Detection Baselines

First, we compare our method with other SOTA baselines in Table 1. The results demonstrate that our method achieves competitive performance compared to 'Supervised' and 'Auxiliary' methods and outperforms 'Unsupervised' OOD detection methods. Next, we provide a more detailed comparison with some unsupervised methods, particularly the ELBO of VAE, as shown in Table 2. These results indicate that our method effectively mitigates overestimation and enhances OOD detection performance when using VAE as the backbone. Lastly, to assess our method's generalization capabilities, we test it on a broader range of datasets, as displayed in Table 3. Experimental results strongly verify our analysis of the V AE's overestimation issue and demonstrate that our method consistently mitigates overestimation , regardless of the type of OOD datasets.

5.3 Ablation Study on Verifying the Post-hoc Prior Method

To evaluate the effectiveness of the Post-hoc Prior (PHP), we compare it with other unsupervised methods in Table 2. Moreover, we test the PHP method on additional datasets and present the results in Table 4 of Appendix F. The experimental results demonstrate that the PHP method can alleviate the overestimation . To provide a better understanding, we also visualize the density plot of ELBO and PHP for the 'FashionMNIST(ID)/MNIST(OOD)' dataset pair in Figures 6(a) and 6(b), respectively.

The Log-likelihood Ratio ( LLR ) methods [17, 18] are the current SOTA unsupervised OOD detection methods that also focus on latent variables. These methods are based on an empirical assumption that the bottom layer latent variables of a hierarchical V AE could learn low-level features and top layers learn semantic features. However, we discovered that while ELBO could already perform well in detecting some OOD data, the LLR method [18] could negatively impact OOD detection performance to some extent, as demonstrated in Figure 6(c), where the model is trained on MNIST and detects FashionMNIST as OOD. On the other hand, our method can still maintain comparable performance since the PHP method can explicitly alleviate overestimation , which is one of the strengths of our method compared to the SOTA methods.

5.4 Ablation Study on Verifying the Dataset Entropy Calibration Method

We evaluate the performance of dataset entropy calibration, referred to as 'DEC', in Table 2 and Table 5 of Appendix G. Although the DEC method is simple, our results show that it effectively alleviates overestimation . To better understand DEC, we visualize the calculated C ( x ) of CIFAR10

315

ID

FashionMNIST

ID

CIFAR10

OOD

AUROC

↑

AUPRC

↑

FPR80

↓

OOD

AUROC

↑

AUPRC

↑

PFR80

↓

ELBO / AVOID

(ours)

ELBO / AVOID

(ours)

KMNIST

60.03 /

78.71

54.60 /

68.91

61.6 /

48.4

CIFAR100

52.91 /

55.36

51.15 /

72.13

77.42 /

73.93

Omniglot

99.86 /

100.0

99.89 /

100.0

0.00 /

0.00

CelebA

57.27 /

71.23

54.51 /

72.13

69.03 /

54.45

notMNIST

94.12 /

97.72

94.09 /

97.70

8.29 /

2.20

Places365

57.24 /

68.37

56.96 /

69.05

73.13 /

62.64

CIFAR10-G

98.01 /

99.01

98.24 /

99.04

1.20 /

0.40

LFWPeople

64.15 /

67.72

59.71 /

68.81

59.44 /

54.45

98.49 /

98.59

97.49 /

97.87

1.00 /

1.00

SUN

53.14 /

63.09

54.48 /

63.32

79.52 /

68.63

SVHN-G

95.61 /

96.20

96.20 /

97.41

3.00 /

0.40

STL10

49.37 /

64.51

47.79 /

65.50

78.02 /

67.23

CelebA-G

97.33 /

97.87

94.71 /

95.82

3.00 /

0.40

Flowers102

67.68 /

76.83

64.68 /

78.01

57.94 /

46.65

SUN-G

99.16 /

99.32

99.39 /

99.47

0.00 /

0.00

GTSRB

39.50 /

53.06

41.73 /

49.84

86.61 /

73.63

Places365-G

98.92 /

98.89

98.05 /

98.61

0.80 /

0.80

DTD

37.86 /

81.82

40.93 /

62.42

82.22 /

64.24

Const

94.94 /

95.20

97.27 /

97.32

1.80 /

1.70

Const

0.001 /

80.12

30.71 /

89.42

100.0 /

22.38

Random

99.80 /

100.0

99.90 /

100.0

0.00 /

0.00

Random

71.81 /

99.31

82.89 /

99.59

85.71 /

0.000

14

12

10

8

6

4

bits/dim

0.0

0.1

0.2

0.3

0.4

Density

FashionMNIST test (ID)

MNIST test (OOD)

22.5

20.0

17.5

15.0

12.5

10.0

7.5

5.0

bits/dim

0.00

0.05

0.10

0.15

0.20

Density

FashionMNIST test (ID)

MNIST test (OOD)

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

MNIST (ID) / FashionMNIST (OOD)

ELBO

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

MNIST (ID) / FashionMNIST (OOD)

ELBO

PHP

(a) Density plot of ELBO

(b) Density plot of PHP

(c) ROC curve of

LLR

(d) ROC curve of PHP

(ID) in Figure 7(a) and other OOD datasets in Figure 7(b) when n id = 20 . Our results show that 310 the C ( x ) of CIFAR10 (ID) achieves generally higher values than that of other datasets, which is the 311 underlying reason for its effectiveness in alleviating overestimation . Additionally, we investigate the 312 impact of different n id on OOD detection performance in Figure 7(c), where our results show that the 313 performance is consistently better than ELBO. 314

0

2000

4000

6000

8000

10000

12000

C(x)

0

100

200

300

400

Density

CIFAR-10 test (ID)

(a)

C

(

x

)

of CIFAR10 (ID)

0

2000

4000

6000

8000

10000

12000

C(x)

0

100

200

300

400

500

600

700

Density

LFWPeople

SUN397

GTSRB

CIFAR100

SVHN

Flowers102

DTD

Places365

STL10

CelebA

constant

random

(b)

C

(

x

)

of OOD datasets

5.0

7.5

10.0

12.5

15.0

17.5

20.0

22.5

25.0

30

40

50

60

70

80

90

AUROC

DEC

ELBO

(c) Impact of

n

id

6 Conclusion

In conclusion, we have identified the underlying factors that lead to V AE's overestimation in un316 supervised OOD detection: the improper design of the prior and the gap of the dataset entropies 317 between the ID and OOD datasets. With this analysis, we have developed a novel score function 318 called 'AVOID', which is effective in alleviating overestimation and improving unsupervised OOD 319 detection. This work may lead a research stream for improving unsupervised OOD detection by 320 developing more efficient and sophisticated methods aimed at optimizing these revealed factors. 321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

References

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR , 2017.

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR , 2015.

Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR , 2015.

Hongxin Wei, Lue Tao, Renchunzi Xie, Lei Feng, and Bo An. Open-sampling: Exploring out-of-distribution data for re-balancing long-tailed datasets. In ICML , 2022.

Alexander A. Alemi, Ian Fischer, and Joshua V. Dillon. Uncertainty in the variational information bottleneck. CoRR , abs/1807.00906, 2018.

Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurIPS , 2020.

Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against inherent label noise. In NeurIPS , 2022.

Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distribution examples via augmenting content and style. arXiv preprint arXiv:2207.03162 , 2022.

Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICML , 2022.

Shuyang Yu, Junyuan Hong, Haotao Wang, Zhangyang Wang, and Jiayu Zhou. Turning the curse of heterogeneity in federated learning into a blessing for out-of-distribution detection. In ICLR , 2023.

Ido Galil, Mohammed Dabbah, and Ran El-Yaniv. A framework for benchmarking class-out-ofdistribution detection and its application to imagenet. In ICLR , 2023.

Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. 347 Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In 348 NeurIPS , 2019. 349

Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. 350 Input complexity and out-of-distribution detection with likelihood-based generative models. In 351 ICLR , 2020. 352

Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection 353 score for variational auto-encoder. In NeurIPS , 2020. 354

Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: A very deep hierarchy 355 of latent variables for generative modeling. In NeurIPS , 2019. 356

Griffin Floto, Stefan Kremer, and Mihai Nica. The tilted variational autoencoder: Improving 357 out-of-distribution detection. In ICLR , 2023. 358

Jakob D Drachmann Havtorn, Jes Frellsen, Soren Hauberg, and Lars Maaløe. Hierarchical vaes 359 know what they don't know. In ICML , 2021. 360

Yewen Li, Chaojie Wang, Xiaobo Xia, Tongliang Liu, and Bo An. Out-of-distribution detection 361 with an adaptive likelihood ratio on informative hierarchical vae. In NeurIPS , 2022. 362

Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, 363 and Alex Graves. Conditional image generation with pixelcnn decoders. In NeurIPS , 2016. 364

Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn++: Improving the 365 pixelcnn with discretized logistic mixture likelihood and other modifications. In ICLR , 2017. 366

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 367 ICLR , 2017. 368

Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolu369 tions. In NeurIPS , 2018. 370

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In 371 NeurIPS , 2020. 372

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil 373 Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications 374 of the ACM , 63(11):139-144, 2020. 375

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR , 2014. 376

Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for 377 robust anomaly detection. arXiv preprint arXiv:1810.01392 , 2018. 378

Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshmi379 narayanan. Do deep generative models know what they don't know? In ICLR , 2019. 380

Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma 381 with denoising diffusion gans. In ICLR , 2022. 382

Thomas M Cover. Elements of Information Theory . John Wiley & Sons, 1999. 383

Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution matching in 384 variational inference. CoRR , abs/1802.06847, 2018. 385

Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep 386 unsupervised learning using nonequilibrium thermodynamics. In ICML , 2015. 387

William Feller. On the theory of stochastic processes, with particular reference to applications. 388 In Selected Papers I , pages 769-798. Springer, 2015. 389

Yixin Wang, David M. Blei, and John P. Cunningham. Posterior collapse and latent variable 390 non-identifiability. In NeurIPS , 2021. 391

Adji B. Dieng, Yoon Kim, Alexander M. Rush, and David M. Blei. Avoiding latent variable 392 collapse with generative skip models. In AISTATS , 2019. 393

Yewen Li, Chaojie Wang, Zhibin Duan, Dongsheng Wang, Bo Chen, Bo An, and Mingyuan 394 Zhou. Alleviating 'posterior collapse' in deep topic models via policy gradient. In NeurIPS , 395 2022. 396

Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. 397 In ICLR , 2016. 398

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of 399 machine learning research , 9(11), 2008. 400

Gilbert W Stewart. On the early history of the singular value decomposition. SIAM Review , 35 401 (4):551-566, 1993. 402

Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. 403 Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In 404 NeurIPS , 2019. 405

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for 406 benchmarking machine learning algorithms. CoRR , abs/1708.07747, 2017. 407

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning 408 applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. 409

410

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images.

Master's thesis, University of Tront , 2009. 411

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. 412 Reading digits in natural images with unsupervised feature learning. 2011. 413

Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with 414 outlier exposure. In ICLR , 2019. 415

Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image 416 detection in neural networks. In ICLR , 2018. 417

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for 418 detecting out-of-distribution samples and adversarial attacks. In NeurIPS , 2018. 419

Saikiran Bulusu, Bhavya Kailkhura, Bo Li, Pramod K Varshney, and Dawn Song. Anomalous 420 example detection in deep learning: A survey. IEEE Access , 8:132330-132347, 2020. 421

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable 422 predictive uncertainty estimation using deep ensembles. In NeurIPS , 2017. 423

Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting 424 distributional shifts in the wild. In NeurIPS , 2021. 425

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 426 2015. 427

428

429

430

431

432

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , 2019.

Ramneet Kaur, Susmit Jha, Anirban Roy, Sangdon Park, Edgar Dobriban, Oleg Sokolsky, and 433 Insup Lee. idecode: In-distribution equivariance for conformal out-of-distribution detection. In 434 AAAI , 2022. 435

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper first mathematically examines the unsupervised (without training label) OOD detection performance using VAE, decomposing the expected ELBO into two components: (i) entropy $\\mathcal{H}(x)$ of a dataset, (ii) KL divergence $D_{KL}(q(z)||p(z))$ between the estimated $z$ and the prior. It's theoretically shown that the entropy of data distribution is defined by itself, thus may not bring benefit to the OOD detection problem (Eq. 8). Then the paper mathematical and empirically analyzes the second component. The paper shows in some simple cases the prior $p(z)$ and the dataset $p(x)$ can not fit well with the VAE model, which results in for some $x$, $p_\\theta(x)$ estimated by the trained VAE model has high value when $p(x)$ has low value, which is the overestimation problem for OOD detection. The paper proposes to use post-hoc prior method (estimate the prior from the trained VAE and the ID dataset) to revise the issue of the improper design of prior and add calibration to alleviate the issue of entropy. Empirical results show that the proposed AVOID method constantly improves the OOD detection performance simply based on ELBO (Table 3), and outperforms existing unsupervised non-ensemble OOD detection methods.\n\nSTRENGTHS:\n1. The paper mathematically examines the unsupervised (without training label) OOD detection performance using VAE, decomposing the expected ELBO into two components: (i) entropy $\\mathcal{H}(x)$ of a dataset, (ii) KL divergence $D_{KL}(q(z)||p(z))$ between the estimated $p(z)$ and the prior, which is quite crucial to understand the underline benefit and drawback to use ELBO as OOD score.\n2. The demos including Figure 2,3,4 shows the improper of the traditionally chosen prior leading to the mismatch between prior and post-hoc prior, and the high probability of OOD sample in the prior. The observation well inspires the proposed post-hoc prior method.\n3. Experiment includes varied OOD detection methods including supervised, auxiliary, and unsupervised (ensemble/non-ensemble), and shows the proposed method beats baselines within a specific category.\n\nWEAKNESSES:\n1. Notation is not consistent such as $p$ and $p_\\theta$ in Figure 3.\n2. Eq. 8 uses the entropy difference between ID and OOD distributions. Eq. 8 tells us the more diverse the ID distribution, the harder the OOD detection task. I think the OOD here should consider overall OOD distribution instead of an OOD dataset distribution. If not, I can simply define each OOD data point as a distribution which has $\\mathcal{H}_{p_o}=1$ or I consider overall OOD data together (overall OOD distribution) which may have a pretty large diversity and very low entropy. Thus the motivation for the second method is not well held. I believe the idea of the second method is good itself, it leverages some extra information to improve the OOD performance.\n3. Sec. 3.1 uses 3-layer NN for $q_\\phi$ and $p_\\theta$. The dataset is synthetic, thus I wonder whether increasing the number of training samples and NN capability would help better estimate $p_\\theta(x)$. In other words, the reason that ELBO suffers from overestimation is the number of training samples, NN capability, or something else. Or perhaps the observation from Figure 3 is even when ID is well estimated, the OOD is still not well estimated.\n\nQUESTIONS:\n(1) I would like to ask for thoughts on Eq. (8), what's the right definition for $\\mathcal{H}_{p_o}$? I feel the choices include: (i) regards only 1 sample as $p_o$, (ii) regards some OOD samples as $p_o$, (iii) regards overall OOD samples as $p_o$. I think (iii) is the best choice though it's not feasible, and (i) and (ii) are not the correct choice. The DEC itself is an OOD detection method and can be ensembled into any OOD detection method, thus I feel it's less related to factor two, the entropy issue.\n(2) Why in Table 2 PHP performs better than DEC on the left table and perform worse on the right table?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the phenomenon of overestimation, the allocation of higher likelihoods to out-of-distribution data points, in deep generative models. It analyses two factors which may cause the overestimation problem specific to VAEs from a reformulation of the ELBO. These two factors are posterior collapse and a difference in entropies between in-distribution and out-of-distribution datasets. The paper proposes, again specific to VAEs, a method called AVOID for alleviating these two factors.\n\nSTRENGTHS:\n* The clear definition of overestimation in Eq. (3) following is useful, also for the wider literature.\n* The experimental setup is large: Table 3 demonstrates that the number of dataset combinations considered is numerous, in particular in comparison to previous work.\n\nWEAKNESSES:\n* In 3.2, the authors pose the questions “When is the design of the prior proper/not proper”, but answer these questions by providing an example for each case. While this is useful for illustrative purposes, it does not answer the stated question. The first few examples are furthermore focussing on linear VAEs which are not relevant for common practical uses, which limits the relevance of the theoretical results in this section. \n* The design of the calibration term in ll. 219 is unclear. In my opinion, it is not properly explained, and important choices like SVD are not well motivated. When would SVD likely fail? Why does SVD intuitively capture the difference in entropy between the datasets?  The words “complexity” and “entropy” seem to be used interchangeably, please explain or use consistently.\n* The experimental results are difficult to interpret, it is partly not possible to draw meaningful insights from it. It is worth noting that this is common in similar works on alleviating OOD detection in DGMs, the methods are hard to compare due to different experimental setups. However, in this work, important questions I have are: 1) In Table 1, in the unsupervised column, why is AVOID highlighted in bold, even though WAIC  outperforms it sometimes? 2) Where is the performance of a standard VAE without any adaptations listed?  I find this an important benchmark. 3) What is the decision criterion for OOD vs. in-distribution? Is it a threshold on the amended likelihood? If yes, looking at the density plots of Fig. 6 (b), how is it possible that there is still  a lot of overlap between the two datasets in PHP, even though the accuracy according to Table 1 is 99.2%? This seems inconsistent to me. 4) The experimental results report no standard deviations in key tables, such as Table 1 and 2 . DGM based methods are well known to be unstable, hence standard deviations would be useful. However, Table 3 partly alleviates this problem due to the large number of dataset combinations considered. 5) Table 3: I would argue that comparing CIFAR10 and CIFAR100 (and possibly other combinations) seems meaningless: The datasets are overlapping, hence it is unclear what is OOD and what is in-distribution.\n* The language is sometimes unclear, in general slightly hard to understand, and could be greatly improved.\n\nIn summary, while this work demonstrates a large effort and a clear analytical approach to alleviating the overestimation problem in VAEs, important questions remain unclear. I am open to reconsidering my score upon a response from the authors.\n\nQUESTIONS:\n* ll. 191: “The visualization reveals that p(z) cannot distinguish between the latent variables sampled from qid(z) and qood(z), while qid(z) is 193 clearly distinguishable from qood(z).” I don’t understand this sentence, could you please explain it? What do you mean by “the prior cannot distinguish latent samples”?\n* In Eq. (16), the proposal seems to be to add a regularizer which accounts for calibration of differing entropies to the ELBO. Why is this a good choice? Should it be weighted with the remainder of the objective? \n* The proposal for alleviating factor 1 is learning a more complicated, LSTM-based prior. How would a VAE which is end-to-end trained with this more complicated prior perform? Would any VAE with a more complicated prior (of which there are plenty) help alleviate the overestimation problem?\n* The authors could consider discussing and benchmarking against this recent work, which provides an orthogonal, score-based approach to the problem of anomaly detection in DGMs: https://openreview.net/forum?id=deYF9kVmIX\n\nLIMITATIONS:\n* The overestimation problem is common to many DGM methods, but this work provides a solution for VAEs only. The scope is bigger, and one could argue that it might be more interesting to find the underlying root cause in all DGM methods which suffer from this problem (if there is one). Yet, considering VAEs is a very interesting start.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies unsupervised OOD detection (i.e., training data contains no labels) using deep generative models. DGMs model the probability distribution of the inputs, and can be an ideal candidate for unsupervised OOD detection. The authors study one specific class of DGMs, namely VAEs. They show that VAEs suffer from overestimation problem ($P(x_{ood}) > P(x_{id})$) due to two main reasons — dataset’s inherent entropy and improper design of prior distribution. The paper then proceeds to theoretically suggest ways to mitigate this issue, and shows experimental results that do so.\n\nSTRENGTHS:\n1. The theory of the paper is simple but inspiring, and matches neatly with the designed algorithm.\n2. The experiments are well-designed and executed.\n3. The ablation studies are well-done.\n\nWEAKNESSES:\n1. Prior work such as [1] that discusses causes of deep generating models’ (specifically, normalizing flows) reason for failure to perform OOD detection was not cited/discussed in the paper. Similarly, [2] is also an important paper for using DGMs for OOD detection that wasn’t cited.\n2. The paper is not self-contained and the organization could be improved — for example, one could put the limitations in the main paper instead of in the appendix.\n3. Notation of the paper. For example, $p(x) = N(x | 0, \\Sigma_x)$ can be more readable as $x \\sim N(0, \\Sigma_x)$, following more commonly used convention.\n\n[1] Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson. Why Normalizing Flows Fail to Detect Out-of-Distribution Data, https://arxiv.org/abs/2006.08545, 2020\n\n[2] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using a test for typicality. arXiv preprint arXiv:1906.02994, 2019.\n\nQUESTIONS:\n1. What is the architecture used for the supervised OOD detection methods in Table 1? Is it a 3 layer MLP?\n2. Intuitively, supervised OOD detection methods should do better than unsupervised ones, since having access to more information should never hurt the performance. However, that is not the case in Table 1. Do the authors have an explanation for this? \n3. Line 278, why is the max epochs set to such a high value (1000)? Is there some sort of early stopping that is used here?\n4. I am curious on how the reasons noted by [1], namely normalizing flows learning latent representations based on local pixel correlations and not semantic content, are relevant for VAEs. Does VAEs focus on semantic content and not local pixel correlations?\n\n\n[1] Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson. Why Normalizing Flows Fail to Detect Out-of-Distribution Data, https://arxiv.org/abs/2006.08545, 2020\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the context of VAE, the authors identified two factors that potentially cause VAE to assign higher likelihood to OOD data than ID data. They propose a new scoring mechanism that improves upon VAE's overestimation of the likelihood on OOD samples.\n\nSTRENGTHS:\n- Decomposing the ELBO carefully is interesting. In particular, they give a new prior design targeting the overestimation issue.\n- They have a scoring method that improves upon the standard ELBO, which partially validifies their analysis.\n\nWEAKNESSES:\n- the derivation assumes the model distribution can converge exactly to the true one, but this is impractical. \n\nIf it does, there should be no overestimation issue to begin with (for practical datasets that are arguably separable, e.g. SVHN vs CIFAR). Moreover, even if it is possible in theory, the empirical and theoretical observations in [1, 2] will prevent this from happening in practice. \n\nIf it doesn't, the derivation will leave an error gap that is not analyzed. In short, the key reasoning in the above is that real distribution is often supported on low dimensional sets, while model distribution is fully supported.\n\n- the evaluation is a bit outdated on easier benchmarks. To solidifies AVOID's practical impact, evaluation on the harder tasks as in DoSE [3] is necessary.\n\n\n[1] Dai, Bin, and David Wipf. \"Diagnosing and Enhancing VAE Models.\" International Conference on Learning Representations. 2018.\n\n[2] Dai, Bin, Li Kevin Wenliang, and David Wipf. \"On the Value of Infinite Gradients in Variational Autoencoder Models.\" Advances in Neural Information Processing Systems. 2021.\n\n[3] Morningstar, Warren, et al. \"Density of states estimation for out of distribution detection.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.\n\nQUESTIONS:\nI'm happy to miss something in the paper, and be corrected. See the weakness above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}