Title: Deep Discriminative to Kernel Generative Networks for Calibrated Inference
Venue: NeurIPS 2023
Accepted: no
Keywords: Generative models

Posterior Calibration

OOD Calibration

Random Forest

ReLU-net

Polytopes
Abstract: The fight between discriminative versus generative goes deep, in both the study of artificial and natural intelligence. In our view, both camps have complementary values. So, we sought to synergistically combine them. Here, we propose a methodology to convert deep discriminative networks to kernel generative networks. We leveraged the fact that deep models, including both random forests and deep networks, learn internal representations which are unions of polytopes with affine activation functions to conceptualize them both as generalized partitioning rules. We replace the affine function in each polytope populated by the training data with Gaussian kernel that results in a generative model. Theoretically, we derive the conditions under which our generative models are a consistent estimator of the corresponding class conditional density. Moreover, our proposed models obtain well calibrated posteriors for in-distribution, and extrapolate beyond the training data to handle out-of-distribution inputs reasonably. We believe this approach may be an important step in unifying the thinking and the approaches across the discriminative and the generative divide.

Abstract

The fight between discriminative versus generative goes deep, in both the study of artificial and natural intelligence. In our view, both camps have complementary values. So, we sought to synergistically combine them. Here, we propose a methodology to convert deep discriminative networks to kernel generative networks. We leveraged the fact that deep models, including both random forests and deep networks, learn internal representations which are unions of polytopes with affine activation functions to conceptualize them both as generalized partitioning rules. We replace the affine function in each polytope populated by the training data with Gaussian kernel that results in a generative model. Theoretically, we derive the conditions under which our generative models are a consistent estimator of the corresponding class conditional density. Moreover, our proposed models obtain well calibrated posteriors for in-distribution, and extrapolate beyond the training data to handle out-of-distribution inputs reasonably. We believe this approach may be an important step in unifying the thinking and the approaches across the discriminative and the generative divide.

1 Introduction

Machine learning methods, specially deep neural networks and random forests have shown excellent performance in many real-world tasks, including drug discovery, autonomous driving and clinical surgery. However, calibrating confidence over the whole feature space for these models remains a key challenge in the field. Although these learning algorithms can achieve near optimal performance at inferring on the samples lying in the high density regions of the training data [1-3], they yield highly confident predictions for the samples lying far away from the training data [4]. Calibrated confidence within the training or in-distribution (ID) region as well as in the out-of-distribution (OOD) region is crucial for safety critical applications like autonomous driving and computer-assisted surgery, where any aberrant reading should be detected and taken care of immediately [4, 5]. A well-calibrated model capable of quantifying the uncertainty associated with inference for any points from the training distribution as well as detecting OOD data can be a life-saver in these cases.

The approaches to calibrate OOD confidence for learning algorithms described in the literature can 28 be roughly divided into two groups: discriminative and generative. Discriminative approaches try to 29 scale the posteriors based on OOD detection or modify the learning loss function. Intuitively, the 30 easiest solution for OOD confidence calibration is to learn a function that gives higher scores for 31 in-distribution samples and lower scores for OOD samples, and thereby re-scale the posterior or 32 confidence score from the original model accordingly [6]. There are a number of approaches in the 33 literature which try to either modify the loss function [7-9] or adversarially train the network to be 34 less confident on OOD samples [10, 4]. However, one can adversarially manipulate an OOD sample 35 where the model is less confident to find another OOD sample where the model is overconfident 36 [11, 4, 12]. Recently, as shown by Hein et al. [4], the ReLU networks produce arbitrarily high 37

confidence as the inference point moves far away from the training data. Therefore, calibrating 38 ReLU networks for the whole OOD region is not possible without fundamentally changing the 39 network architecture. As a result, all of the aforementioned algorithms are unable to provide any 40 guarantee about the performance of the network through out the whole feature space. On the other end 41 of the spectrum, the generative group tries to learn generative models for both the in-distribution as 42 well as the out-of-distribution samples. The general idea for the generative group is to get likelihoods 43 for a particular sample out of the generative models for both ID and OOD to do likelihood ratio test 44 [13] or control the likelihood for training distribution far away from the training data to detect OOD 45 samples by thresholding. However, it is not obvious how to control likelihoods far away from the 46 training data for powerful generative models like variational autoencoders (V AEs) [14] and generative 47 adversarial networks (GAN) [15]. Moreover, Nalisnick et al. [16] and Hendrycks et al. [10] showed 48 VAEs and GANs can also yield overconfident likelihoods far away from the training data. 49

50

51

52

53

54

55

56

57

The algorithms described so far are concerned with OOD confidence calibration for deep-nets only. However, in this paper, we show that other approaches which partition the feature space, for example random forest, can also suffer from poor confidence calibration both in the ID and the OOD regions. Moreover, the algorithms described above are concerned about the confidence of the algorithms in the OOD region only and they do not address the confidence calibration within the training distribution at all. This issue is addressed separately in a different group of literature [17-19]. In this paper, we consider both calibration problem jointly and propose an approach that achieves good calibration throughout the whole feature space.

In this paper, we conceptualize both random forest and ReLU networks as generalized partitioning 58 rules with an affine activation over each polytope. We consider replacing the affine functions learned 59 over the polytopes with Gaussian kernels. We propose two novel kernel density estimation techniques 60 named Kernel Generative Forest ( KGF ) and Kernel Generative Network ( KGN ). We theoretically show 61 that they asymptotically converge to the true training distribution under certain conditions. At the 62 same time, the estimated likelihood from the kernel generative models decreases for samples far away 63 from the training samples. By adding a suitable bias to the kernel density estimate, we can achieve 64 calibrated posterior over the classes in the OOD region. It completely excludes the need for providing 65 OOD training examples to the model. We conduct several simulation and real data studies that show 66 both KGF and KGN are robust against OOD samples while they maintain good performance in the 67 in-distribution region. 68

2 Related Works and Our Contributions 69

There are a number of approaches in the literature which attempt to learn a generative model and 70 control the likelihoods far away from the training data. For example, Ren et al. [13] employed 71 likelihood ratio test for detecting OOD samples. Wan et al. [8] modify the training loss so that the 72 downstream projected features follow a Gaussian distribution. However, there is no guarantee of 73 performance for OOD detection for the above methods. To the best of our knowledge, only Meinke 74 et al. [5] has proposed an approach to guarantee asymptotic performance for OOD detection. They 75 model the training and the OOD distribution using Gaussian mixture models which enable them 76 to control the class conditional posteriors far away. Compared to the aforementioned methods, our 77 approach differs in several ways: 78

79

80

We address the confidence calibration problems for both ReLU -nets and random forests from a common ground.

81

82

83

84

85

86

We address in-distribution (ID) and out-of-distribution (OOD) calibration problem as a continuum rather than two separate problems.

We provide guarantees for asymptotic convergence of our proposed approach under certain conditions for both ID and OOD regions.

We propose an unsupervised OOD calibration approach, i.e., we do not need to train exhaustively on different OOD samples.

3 Methods 87

88

3.1 Setting

Consider a supervised learning problem with independent and identically distributed training samples 89 { ( x i , y i ) } n i =1 such that ( X,Y ) ∼ P X,Y , where X ∼ P X is a X ⊆ R d valued input and Y ∼ P Y is 90 a Y = { 1 , · · · , K } valued class label. We define in-distribution region as the high density region 91 of P X,Y and denote it by S . Here the goal is to learn a confidence score, g : R d → [0 , 1] K , 92 g ( x ) = [ g 1 ( x ) , g 2 ( x ) , . . . , g K ( x )] such that, 93

where P Y | X ( y | x ) is the posterior probability for class y given by the Bayes formula: 94

Here P X | Y ( x | y ) is the class conditional density for the training data which we will refer as f y ( x ) 95 hereafter for brevity. 96

97

98

99

100

101

3.2 Background and Main Idea

Deep discriminative networks partition the feature space R d into a union of p affine polytopes Q r such that ⋃ p r =1 Q r = R d , and learn an affine function over each polytope [4, 20]. Mathematically, the class-conditional density for the label y estimated by these deep discriminative models at a particular point x can be expressed as:

For example, in the case of a decision tree, a r = 0 , i.e., decision tree assumes uniform distribution 102 for the class-conditional densities over the leaf nodes. Among these polytopes, the ones that lie on 103 the boundary of the training data extend to the whole feature space and hence encompass all the OOD 104 samples. Since the posterior probability for a class is determined by the affine activation over each of 105 these polytopes, the algorithms tend to be overconfident when making predictions on the OOD inputs. 106 Moreover, there exist some polytopes that are not populated with training data. These unpopulated 107 polytopes serve to interpolate between the training sample points. If we replace the affine activation 108 function of the populated polytopes with Gaussian kernel G learned using maximum likelihood 109 approach on the training points within the corresponding polytope and prune the unpopulated ones, 110 the tail of the kernel will help interpolate between the training sample points while assigning lower 111 likelihood to the low density or unpopulated polytope regions of the feature space. This may result in 112 better confidence calibration for the proposed modified approach. 113

114

115

116

117

118

3.3 Proposed Model

Consider the collection of polytope indices P which contains the indices of total ˜ p polytopes populated by the training data. We consider replacing the affine function over the populated polytopes with a Gaussian kernel G ( · ; ˆ µ r , ˆ Σ r ) . For a particular inference point x , we consider the Gaussian kernel with the minimum distance from the center of the kernel to the corresponding point:

where ‖ · ‖ denotes a suitable distance measure. We use Euclidean distance metric while conducting 119 the simulation and the benchmark datasets experiments in this paper for simplicity. In short, we 120 modify Equation 3 from the parent ReLU -net or random forest to estimate the class-conditional density 121 as: 122

where n y is the total number of samples with label y and n ry is the number of samples from class y 123 that end up in polytope Q r . We add a bias to the class conditional density ˜ f y : 124

Note that in Equation 6, b log( n ) → 0 as the total training points, n → ∞ . The class posterior 125 probability (confidence) ˆ g y ( x ) of class y for a test point x is estimated using the Bayes rule as 126 follows: 127

where ˆ P Y ( y ) is the empirical prior probability of class y estimated from the training data. We 128 estimate the class for a particular inference point x as: 129

3.4 Desiderata 130

We desire our proposed model to estimate confidence score ˆ g y to satisfy the following two desiderata: 131

Asymptotic Performance : We want point-wise convergence for our estimated confidence as n →∞ , i.e.,

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

Finite Sample Performance : We want better posterior calibration for ˆ g y ( x ) both in ID and OOD region compared to that of its parent model.

Wetheoretically derive the conditions under which we achieve Desiderata 1 in Section 4. However, we run extensive experiments on various simulation and benchmark datasets in Section 6 to empirically verify that our proposed approach achieves Desiderata 2.

4 Theoretical Results

Theorem 1 (Asymptotic Convergence to the True Distribution) . Consider a partition rule that partitions R d into hypercubes of the same size h n > 0 . Formally, let P n = { Q 1 , Q 2 , · · · } be a partition of R d , that is, it partitions R d into sets of the type Π d i =1 [ ψ i h n , ( ψ i +1) h n ) , where ψ i 's are integers. Let n be the total number of samples and n r be the number of data points within polytope Q r . Consider the probability density f estimated for the samples populating the polytopes using Equation 5, denoted as ˆ f . The conditions for choosing the Gaussian kernel parameters are:

The center of the kernel can be any point z r within the polytope Q r as n →∞ ,

The kernel bandwidth along any dimension σ r is any positive number always bounded by the polytope bandwidth h n as n →∞ , i.e., σ r = C r h n , where 0 < C r ≤ 1 .

Consider the following assumptions as well:

The polytope bandwidth h n → 0 as n →∞ .

n grows faster than the shrinkage of h n , i.e., nh n →∞ as h n → 0 in probability.

Given these assumptions, we have that as n →∞ :

where | · | denotes absolute value of the scalar it operates on.

Proof. Please see Appendix A for the proof. 152

Theorem 2 (Asymptotic OOD Convergence) . Given n independent and identically distributed 153 training samples { ( x i , y i ) } n i =1 , we define the distance of an inference point x from the training points 154 as: 155

Here ‖ · ‖ denotes a suitable distance measure as mentioned in Equation 4. Given non-zero and bounded bandwidth of the Gaussians, then we have almost sure convergence for ˆ g y as: ˆ g y ( x ) as → ˆ P Y ( y ) as d x →∞ .

Proof. Please see Appendix A for the proof.

Corollary 2.1. Given the conditions in Theorem 1 and 2, we have:

Proof. Using the law of large numbers, we have ˆ P Y ( y ) = n y n as → P Y ( y ) as n y →∞ . The rest of the proof follows from Theorem 1 and 2.

5 Model Parameter Estimation

5.1 Gaussian Kernel Parameter Estimation

156

157

158

159

160

161

162

163

164

165

166

167

168

Theorem 1 implies that the Gaussian kernel parameters need to maintain two key properties. We use the training data within the polytopes to estimate the Gaussian parameters in a way that we asymptotically satisfy the above two conditions for consistency. To satisfy the first condition, we set the kernel center as:

Note that ˆ µ r in Equation 10 resides always within the corresponding polytope Q r . For improving the 169 estimates for the kernel bandwidth, we incorporate the samples from other polytopes Q s based on the 170 similarity w rs between Q r and Q s . Moreover, We constrain our estimated Gaussian kernels to have 171 diagonal covariance matrix. We use weighted likelihood estimation to estimate the variance Σ r for a 172 particular polytope Q r . For simplicity, we will describe the estimation procedure for w rs later. The 173 weighted likelihood estimation for Σ r can be written as: 174

where we regularize the Frobenius norm of precision matrix Σ -1 so that Σ does not become singular 175 and λ is the regularization parameter. By solving Equation 11, we find: 176

where, I d is a d dimensional identity matrix. However, we want Σ r to be estimated based on the 177 samples within Q r so that the second condition for the Gaussian parameters is satisfied. Therefore, 178 as n →∞ and h n → 0 , the estimated weights w rs should satisfy the condition: 179

/negationslash

We need Condition 13 as we will be only using the data within the polytope Q r as n →∞ to estimate 180 the Gaussian bandwidth and the estimated Gaussian bandwidth will be bounded by the polytope 181 bandwidth. Additionally, we use weighted samples to replace the ratio n ry n y in Equation 5 as: 182

Note that if we satisfy Condition 13, then we have ˜ w ry ˜ w y → n ry n y as n →∞ . Therefore, we modify 183 Equation 5 as: 184

where ˆ r ∗ x = argmin r ‖ ˆ µ r -x ‖ . Below, we describe how we estimate w rs for KGF and KGN .

5.2 Kernel Generative Forest

Consider T number of decision trees in a random forest trained on n i.i.d training samples { ( x i , y i ) } n i =1 . Each tree t partitions the feature space into p t polytopes resulting in a set of polytopes: {{ Q t,r } p t r =1 } T t =1 . The intersection of these polytopes gives a new set of polytopes { Q r } p r =1 for the forest. For any point x r ∈ Q r , we push every other sample point x s ∈ Q s down the trees. Now, we define the weight w ′ rs as:

where t rs is the total number trees x r and x s end up in the same leaf node. Note that 0 ≤ w ′ rs ≤ 1 . 192 If the two samples end up in the same leaves in all the trees, they belong to the same polytope, i.e. 193 Q r = Q s . 194

In short, w ′ rs is the fraction of total trees where the two samples follow the same path from the root to 195 a leaf node. We exponentiate w ′ rs with a suitable function of n which grows with n so that Condition 196 13 is satisfied: 197

185

186

187

188

189

190

191

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

5.3 Kernel Generative Network

Consider a fully connected ReLU -net trained on n i.i.d training samples { ( x i , y i ) } n i =1 . We have the set of all nodes denoted by N l at a particular layer l . We can randomly pick a node n l ∈ N l from A l at each layer l , and construct a sequence of nodes starting at the input layer and ending at the output layer which we call an activation path : m = { n l ∈ N l } L l =1 . Note that there are N = Π L i =1 |N l | possible activation paths for a sample in the ReLU -net, where | · | denotes the cardinality or the number of elements in the set. We index each path by a unique identifier number z ∈ N and construct a sequence of activation paths as: M = { m z } z =1 , ··· ,N . Therefore, M contains all possible activation pathways from the input to the output of the network.

While pushing a training sample x i through the network, we define the activation from a ReLU unit at any node as ' 1 ' when it has non-negative input and ' 0 ' otherwise. Therefore, the activation indicates on which side of the affine function at each node the sample falls. The activation for all nodes in an activation path m z for a particular sample creates an activation mode a z ∈ { 0 , 1 } L . If we evaluate the activation mode for all activation paths in M while pushing a sample through the network, we get a sequence of activation modes: A r = { a r z } N z =1 . Here r is the index of the polytope where the sample falls in.

If the two sequences of activation modes for two different training samples are identical, they belong to the same polytope. In other words, if A r = A s , then Q r = Q s . This statement holds because the above samples will lie on the same side of the affine function at each node in different layers of the network. Now, we define the weight w ′ rs as:

Note that 0 ≤ w ′ rs ≤ 1 . In short, w ′ rs is the fraction of total activation paths which are identically 218 activated for two samples in two different polytopes r and s . We exponentiate the weights using 219 Equation 17. 220

Pseudocodes outlining the two algorithms are provided in Appendix C. 221

222

Simulation Data

2

Spiral

Circle

Sinewave

Polynomial

W

True Posteriors

2

W

2

RF Posteriors

1.0

2

0.5

KGF Posteriors

2

0.0

DN Posteriors

2

KGN Posteriors

2

?

2

2

6 Experimental Results

We conduct several experiments on two dimensional simulated datasets and OpenML-CC18 data 223 suite [21] 1 to gain insights on the finite sample performance of KGF and KGN . The details of the 224 simulation datasets and hyperparameters used for all the experiments are provided in Appendix B. 225 For the simulation setups, we use classification error, hellinger distance [22, 23] from the true class 226 conditional posteriors and mean max confidence or posterior [4] as performance statistics. While 227 measuring in-distribution calibration for the datasets in OpenML-CC18 data suite, as we do not know 228 the true distribution, we used adaptive calibration error as defined by Nixon et al. [24] with a fixed 229 bin number of R = 15 across all the datasets. Given n OOD samples, we define OOD calibration 230 error to measure OOD performance for the benchmark datasets as: 231

6.1 Simulation Study 232

Figure 1 top row shows 10000 training samples with 5000 samples per class sampled within the 233 region [ -1 , 1] × [ -1 , 1] from the five simulation setups described in Appendix B. Therefore, the 234 empty annular region between [ -1 , 1] × [ -1 , 1] and [ -2 , 2] × [ -2 , 2] is the low density or OOD 235

1 https://www.openml.org/s/99

region in Figure 1. The corresponding true posteriors P [ Y = 0 | X = x ] are shown in the second 236 row of Figure 1. As shown in Row 3 and 5 , RF and DN are really good at estimating the high density 237 regions of training distribution. However, they overestimate the posteriors in the low density regions 238 of training distribution. Row 4 and 6 of Figure 1 show KGF and KGN improves the posterior estimation 239 specially in the low density of the training distribution or OOD regions of the feature space. Because 240 of axis aligned split in random forest RF and thereby, KGF are less efficient in learning non-linear 241 decision boundaries like spiral, circle and sinewave simulations than ReLU -net and KGN . Figure 2 242 quantifies the performance of the algorithms which are visually represented in Figure 1. KGF and KGN 243 maintain similar classification accuracy to those of their parent algorithms. We measure hellinger 244 distance from the true distribution for increasing training sample size within [ -1 , 1] × [ -1 , 1] region 245 as an index for in-distribution calibration. Column 2 of left and right block in Figure 2 show KGF 246 and KGN are better at estimating the high density region of training distribution compared to their 247 parent methods. For measuring OOD performance, we normalize the training data by the maximum 248 of their l 2 norm so that the training data is confined within a unit circle. For inference, we sample 249 1000 inference points uniformly from a circle where the circles have increasing radius and plot 250 mean max posterior for increasing distance from the origin. Therefore, for distance up to 1 we have 251 in-distribution samples and distances farther than 1 can be considered as OOD region. As shown in 252 Column 3 of Figure 2, mean max posteriors or confidence for KGF and KGN converge to the maximum 253 of the class priors, i.e., 0 . 5 as we go farther away from the training data origin. 254

255

6.2 Benchmark Data Study

We used OpenML-CC18 data suite for benchmark dataset study. We exclude any dataset which 256 contain categorical features or NaN values and conduct our experiments on 46 datasets with varying 257 dimensions and sample sizes. For the OOD experiments, we follow a similar setup as that of the 258 simulation data. We normalize the training data by their maximum l 2 norm and sample 1000 testing 259 samples uniformly from each hypersphere where the hyperspheres have increasing radius starting 260 from 1 to 5 . Figure 3 shows the summary of performance of the algorithms. The extended results for 261

each dataset is shown separately in appendix Figure 4, 5, 6, 7, 8 and 9. Figure 3 left column shows on 262 average KGF and KGN has nearly similar classification accuracy to their respective parent algorithm. 263 However, according to Figure 3 middle column, KGF improves the in-distribution calibration for 264 random forest by a huge margin. On the contrary, KGN maintains similar in-distribution calibration 265 performance to that of its parent ReLU -net. Most interestingly, Figure 3 right column KGN and KGF 266 improves OOD calibration of their respective parent algorithms by a huge margin. 267

268

269

270

271

272

273

274

275

7 Discussion

In this paper, we convert deep discriminative models to deep generative models by replacing the affine function over the polytopes in the discriminative models with a Gaussian kernel. This replacement of affine function results in better in- and out-of-distribution calibration for our proposed approaches while maintaining classification accuracy close to the parent algorithm. Theoretically, we show under certain conditions our approaches asymptotically converge to the true training distribution and this establishes confidence calibration for learning algorithms in in- and out-of-distribution regions as a continuum rather than two different problems.

For a feature space densely partitioned with small polytopes, we can use Euclidean distance metric in 276 Equation 4. This is because the Euclidean manifold approximation holds locally for the corresponding 277 polytope with index r ∗ x . On the contrary, for a feature space partitioned with large polytopes, 278 Euclidean distance measure may be a wrong notion of distance, specially when the underlying 279 manifold is non-Euclidean. Note that the indicator function in Equation 5 and pruning of the 280 unpopulated polytopes result in an enlargement of the polytopes in our proposed method compared 281 to that of the parent model in Equation 3. Therefore, our proposed approach while using Euclidean 282 metric in Equation 4 may have less classification accuracy compared to that of its parent algorithm. 283 A correct measure of distance in all the cases including the aforementioned non-Euclidean one would 284 be the geodesic distance as explored by Madhyastha et al. [25]. We will explore convolutional 285 neural nets (CNN) trained on image and language benchmark datasets using geodesic distance in 286 Equation 4 in our future work. Additionally, the proposed approach needs benchmarking against 287 other calibration approaches in the literature which are mainly based on image datasets and we will 288 pursue the benchmarking task in our future work. 289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

References

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1321-1330. PMLR, 06-11 Aug 2017.

Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just a bit, fixes overconfidence in ReLU networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 5436-5446. PMLR, 13-18 Jul 2020.

Haoyin Xu, Kaleab A. Kinfu, Will LeVine, Sambit Panda, Jayanta Dey, Michael Ainsworth, Yu-Chung Peng, Madi Kusmanov, Florian Engert, Christopher M. White, Joshua T. Vogelstein, and Carey E. Priebe. When are Deep Networks really better than Decision Forests at small sample sizes, and how? arXiv preprint arXiv:2108.13637 , 2021.

Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 41-50, 2019.

Alexander Meinke, Julian Bitterwolf, and Matthias Hein. Provably robust detection of out-ofdistribution data (almost) for free. arXiv preprint arXiv:2106.04260 , 2021.

Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-ofdistribution image detection in neural networks. arXiv preprint arXiv:1706.02690 , 2017.

Jay Nandy, Wynne Hsu, and Mong Li Lee. Towards maximizing the representation gap between in-domain & out-of-distribution examples. Advances in Neural Information Processing Systems , 33:9239-9250, 2020.

Weitao Wan, Yuanyi Zhong, Tianpeng Li, and Jiansheng Chen. Rethinking feature distribution for loss functions in image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 9117-9126, 2018.

Terrance DeVries and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint arXiv:1802.04865 , 2018.

Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606 , 2018.

Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 427-436, 2015.

Vikash Sehwag, Arjun Nitin Bhagoji, Liwei Song, Chawin Sitawarin, Daniel Cullina, Mung Chiang, and Prateek Mittal. Better the devil you know: An analysis of evasion attacks using out-of-distribution adversarial examples. arXiv preprint arXiv:1905.01726 , 2019.

Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. Advances in neural information processing systems , 32, 2019.

Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning , 12(4):307-392, 2019.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint arXiv:1810.09136 , 2018.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural 338 networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017. 339

340

341

342

Richard Guo, Ronak Mehta, Jesus Arroyo, Hayden Helm, Cencheng Shen, and Joshua T Vogelstein. Estimating information-theoretic quantities with uncertainty forests. arXiv , pages arXiv-1907, 2019.

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems , 32, 2019.

Haoyin Xu, Kaleab A Kinfu, Will LeVine, Sambit Panda, Jayanta Dey, Michael Ainsworth, Yu-Chung Peng, Madi Kusmanov, Florian Engert, Christopher M White, et al. When are deep networks really better than decision forests at small sample sizes, and how? arXiv preprint arXiv:2108.13637 , 2021.

Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G Mantovani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmarking suites. arXiv preprint arXiv:1708.03731 , 2017.

Thomas Kailath. The divergence and bhattacharyya distance measures in signal selection. IEEE transactions on communication technology , 15(1):52-60, 1967.

C Radhakrishna Rao. A review of canonical coordinates and an alternative to correspondence analysis using hellinger distance. Qüestiió: quaderns d'estadística i investigació operativa , 1995.

Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In CVPR workshops , volume 2, 2019.

Meghana Madhyastha, Gongkai Li, Veronika Strnadová-Neeley, James Browne, Joshua T Vogelstein, Randal Burns, and Carey E Priebe. Geodesic forests. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 513-523, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an algorithm to learn an auxiliary model. The goal is to enhance the out-of-distribution calibration performance while maintaining the in-distribution performance of a base discriminative model. This base model is either based on a random forest or a multi-layer perceptron. The algorithm follows three main step: Firstly, it constructs an adjacency matrix using the training samples. The entries of this matrix depend on the \"rule/activation patterns\" of the corresponding discriminative model. Secondly, it performs clustering to divide the input space into regions. Lastly, for each region, the algorithm uses a Gaussian kernel to regress the output of the original model (by learning the center and the covariance matrix). The algorithm reaches convergence to the ground truth posterior distribution in the limit of infinite training data. The experiments are conducted on two-dimensional toy data and datasets from the OpenML-CC18 Benchmark Suites, thus demonstrating the algorithm's superior performance in terms of out-of-distribution calibration compared to the base models. Furthermore, it achieves comparable accuracy and in-distribution calibration performance to the base models.\n\nSTRENGTHS:\n1. The algorithm to identify the partition induced by a classifier (random forest, MLP) on the input space is original and novel. Indeed, it can help to shed light on the calibration properties of existing classifiers (**Originality**)\n2. Asymptotic convergence of the algorithm to the true posterior is guaranteed in the infinite sample regime (**Soundness**).\n3. Overall, the paper is clear (**Clarity**). The text could be improved by providing more details about the algorithm, rather than focusing on the discussion between discriminative and generative models, which seems a bit out of scope.\n4. Code is available, but I haven’t tried to reproduce the experiments (**Reproducibility**)\n\nWEAKNESSES:\n1. It’s unclear how the algorithm performs in the finite sample regime and in high dimensions and how these two quantities are related. The scope of the analysis is therefore limited to a regime with infinite data (**Significance**). In absence of an analysis on the number of dimensions, the authors could provide an experimental analysis with higher dimensional datasets, like CIFAR10 or CIFAR100.\n2. Previous works have shown that neural networks are over-confident/wrong on OOD data [1] and have suggested a set of experiments to check the phenomenon. Providing an analysis on such experimental settings might strengthen the work (**Quality**) - See Questions.\n3. The authors could relate the partitioning technique with the literature on approximation based on splines (**Quality**) - See Questions.\n\nQUESTIONS:\n**Major questions:** \n1. [1] provides an interesting insight on OOD classification. Specifically, when training on object datasets like CIFAR-10 and CIFAR-100, neural models assign high likelihood also to SVHN data. What does it happen when using the proposed algorithm? Does it overcome the weaknesses of neural approaches or does it preserve the same properties?\n2. Considering that deep networks partition the input space with a number of regions that grow exponentially with network depth [2,3], how do you choose the number of clusters in the algorithm, to ensure high regression fidelity (in terms of accuracy and in-distribution calibration) with the base model?\n3. How reliable is the analysis in the finite sample regime and in high dimensions?\n4. It is not clear why the proposed approach is a generative model (Eq. (7) is never used. Indeed, the algorithm only relied on Eq. (15)). Can you please elaborate on that?\n5. In Figure 2 (row 3, column 1) and in Figure 3 (column 1), the base model achieves higher accuracy with a larger number of samples. This seems counterintuitive. Do you have an explanation for this phenomenon?\n6. What is the impact of pruning on the performance?\n\n**Minor questions:**\n1. In Eq. (6), what is the purpose of the bias and why should the bias vanish for infinite number of samples?\n2. In Eq. (18), the computation of weights is based on the number of equivalent paths. Is there an intuitive explanation for preferring this solution over the one which simply counts the number of activated/disabled neurons in the network?\n\n**References**\n- [1] Do Deep Generative Models Know What They Don’t Know? ICLR 2019\n- [2] On the Number of Linear Regions of Deep Neural Networks. NeurIPS 2014\n- [3] A Spline Theory of Deep Networks. ICML 2018\n\nLIMITATIONS:\nThe work is limited in the analysis (input dimensionality, class of neural network models).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the ID and OOD problem by proposing Kernel Generative Forest and Kernel Generative Network for estimating the similarity $w_{rs}$ and in turn approximate the class-conditional density.\n\nThe main contributions include: 1) theoretical results of the convergence of the approximated class-conditional density under certain conditions. 2) the similarity measure with KGF and KGN between the polytopes.\n\nSTRENGTHS:\nThis paper proposed a novel idea of estimating ID and OOD by approximating the class-conditional density. For that, the similarities between polytopes of the input space when data falls into is estimated. Both theoretical and empirical results show that the proposed model can successfully tell when test data is OOD while preserving the classification accuracy. \n\nThe paper is overall well written.\n\nThis paper has the potential of contributing to the community.\n\nWEAKNESSES:\nThe proposed method is highly related to [4], but in the experimental part there is no comparison to any of the recent methods such as [4] (except for the \"parent algorithms\" RF and DN). At least MMC is also reported in [4]. The readers will also benefit if a detailed discussion of similarity and difference between [4] is provided.\n\nThe background and the proposed method is well illustrated, but the demonstration of the experimental results is less clear.  e.g. 1) In fig 1 the simulated yellow points overwrite the blue points and therefore do not match the true posteriors. 2) The \"Difference\" in fig 3 is hard to understand, as well as the legends of \"wins\". What is the reason not plotting like Fig 2?\n\nQUESTIONS:\n- What is $A_l$ in line 200 on page 6?\n\n - in Theorem 1, it requires the hypercubes to be of the same size. I am not sure if my understanding is correct or not, how can one partition e.g. the $R^1$ into 3 hypercubes $(-\\infty, a), [a,b), [b, \\infty)$ of same size?\n\n - what is \"median performance\"of classification error on the simulated datasets? Isn't the classification error calculated from the whole test set at once?\n\nLIMITATIONS:\nThe paper discussed the limitations of the adopted Euclidean distance measure.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to improve OOD detection for deep discriminative models by replacing the affine function over the polytopes with a Gaussian kernel, leading to a method called kernel generative networks.\nAn estimation method is developed for the proposed method and some theoretical results on asymptotic convergence to the true distribution and to OOD.\nResults based on simulations show that the proposed method can estimate distribution better than the parent algorithms and have benefits in terms of OOD detection or calibration in some cases.\n\nSTRENGTHS:\n### originality\nThe method seems to be novel.\n\n### quality\nThe proposed method is intuitive and sound.\nThe work includes useful theoretical results for the properties of the proposed method.\n\n### clarity\nThe abstract downplay the main focus of the paper or main benefit of the proposed method which is to improve OOD detection/calibration.\nIt makes me confused while reading it for the first time.\n\n### significance\nThe proposed method is simple to understand and could potentially be a standard algorithm in popular ML libraries.\n\nWEAKNESSES:\n### quality\nThe results/experiments of the paper require more work.\n\nWhile the paper says the proposed method \"results in better in- and out-of-distribution calibration\", the results in Figure 3 show a contradicting or mixed results in different cases.\nAs this is the main claim of the paper, it requires an in-depth discussion on why the results are mixed and when do we expect the proposed method outperforms the parent algorithm.\n\nAs improved OOD detection is part of the main claim, the paper should compare some existing unsupervised method like [1].\n\n[1] Zhang, Mingtian, Andi Zhang, Tim Z. Xiao, Yitong Sun, and Steven McDonagh. \"Out-of-distribution detection with class ratio estimation.\" arXiv preprint arXiv:2206.03955 (2022).\n\nQUESTIONS:\nWhat's the computational cost of the proposed method compared to the parent algorithm?\n\nLIMITATIONS:\nSome limitations in terms of the use of Euclidean distance are mentioned, followed by a proposal to fix it by using geodesic distance.\nIt feels to me that the paper should actually explore this considering the mixed results from the experiments section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method for confidence calibration in discriminative deep ReLU networks and random forests based on approximating the class-conditional density with Gaussian kernels.\n\nSTRENGTHS:\n- The proposed method is conceptually simple and fairly novel, and it does not require retraining the parent learner to work.\n- Math and derivation are laid out clearly.\n- The limitations of the work are adequately discussed.\n\nWEAKNESSES:\n- **Writing has room for improvement.** The writing can sometimes be unnecessarily long and convoluted. For example on lines 35-36: \"However, one can adversarially manipulate an OOD sample where the model is less confident to find another OOD sample where the model is overconfident\" and on lines 43-46: \"The general idea for the generative group is to get likelihoods for a particular sample out of the generative models for both ID and OOD to do likelihood ratio test or control the likelihood for training distribution far away from the training data to detect OOD samples by thresholding.\" I would suggest the author break down these long sentences into shorter ones that are easier for the readers to parse. \n\n- **Some potential problems are left unaddressed.** It is unclear to me how the proposed method would be able to overcome the curse of dimensionality, as the number of polytopes can scale exponentially with the number of neurons. I also find the claim that the proposed method converts discriminative networks into generative networks misleading. Although the paper does provide an expression for the class conditional density $\\hat{f}_y(x)$, it seems highly nontrivial to sample from this unnormalized density. I wish the authors would clarify on this point.\n\n- **No baselines in the experiments.** As the authors noted themselves, the experimental analyses in this paper are limited to comparisons between the original networks and the proposed kernel generative versions. Due to the lack of baselines, it is unknown whether the achieved improvements are significant or not, especially when they are obtained at the cost of classification accuracy as evident in figure 3.\n\nOverall, I think that the methods proposed in the paper is potentially interesting, but more experiments needs to be done to demonstrate its practical effectiveness. I think the appeal of this work is really hindered by the lack of baseline comparisons and the limited analyses of the results, as well as some confusing/potentially misleading statements.\n\nQUESTIONS:\n- Can you clarify what is meant by \"convert[ing] deep discriminative networks to kernel generative networks\" in the abstract?\n- How would the proposed method scale with dimensionality?\n- What is the additional computational cost of estimating the parameters of the Gaussian kernels? How does this scale with model size/sample size?\n\nLIMITATIONS:\nThe authors addressed the limitation that the Euclidean metric they used might not be the most suitable. Also, the authors are aware of the lack of comparison to other benchmark methods and commented that they will include it in future work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}