Title: Deep Discriminative to Kernel Density Graph for In- and Out-of-distribution Calibrated Inference
Venue: NeurIPS 2024
Accepted: no
Keywords: random forest

deep networks

in-distribution calibration

out-of-distribution detection
Abstract: Deep discriminative approaches like random forests and deep neural networks have recently found applications in many important real-world scenarios. However, deploying these learning algorithms in safety-critical applications raises concerns, particularly when it comes to ensuring confidence calibration for both in-distribution and out-of-distribution data points. Many popular methods for in-distribution (ID) calibration, such as isotonic and Platt’s sigmoidal regression, exhibit excellent ID calibration performance. However, these methods are not calibrated for the entire feature space, leading to overconfidence in the case of out-of-distribution (OOD) samples. On the other end of the spectrum, existing out-of-distribution (OOD) calibration methods generally exhibit poor in-distribution (ID) calibration. In this paper, we address ID and OOD calibration problems jointly. We leveraged the fact that deep models, including both random forests and deep-nets, learn internal representations which are unions of polytopes with affine activation functions to conceptualize them both as partitioning rules of the feature space. We replace the affine function in each polytope populated by the training data with a Gaussian kernel. Our experiments on both tabular and vision benchmarks show that the proposed approaches obtain well-calibrated posteriors while mostly preserving or improving the classification accuracy of the original algorithm for ID region, and extrapolate beyond the training data to handle OOD inputs appropriately.

Abstract

Deep discriminative approaches like random forests and deep neural networks have recently found applications in many important real-world scenarios. However, deploying these learning algorithms in safety-critical applications raises concerns, particularly when it comes to ensuring confidence calibration for both in-distribution and out-of-distribution data points. Many popular methods for in-distribution (ID) calibration, such as isotonic and Platt's sigmoidal regression, exhibit excellent ID calibration performance. However, these methods are not calibrated for the entire feature space, leading to overconfidence in the case of out-of-distribution (OOD) samples. On the other end of the spectrum, existing out-of-distribution (OOD) calibration methods generally exhibit poor in-distribution (ID) calibration. In this paper, we address ID and OOD calibration problems jointly. We leveraged the fact that deep models, including both random forests and deep-nets, learn internal representations which are unions of polytopes with affine activation functions to conceptualize them both as partitioning rules of the feature space. We replace the affine function in each polytope populated by the training data with a Gaussian kernel. Our experiments on both tabular and vision benchmarks show that the proposed approaches obtain well-calibrated posteriors while mostly preserving or improving the classification accuracy of the original algorithm for ID region, and extrapolate beyond the training data to handle OOD inputs appropriately.

1 Introduction

Machine learning methods, specially deep neural networks and random forests have shown excellent performance in many real-world tasks, including drug discovery, autonomous driving and clinical surgery [1-3]. However, calibrating confidence over the whole feature space for these approaches remains a key challenge in the field [4]. Calibrated confidence within the training or in-distribution (ID) region as well as in the out-of-distribution (OOD) region is crucial for safety critical applications like autonomous driving and computer-assisted surgery, where any aberrant reading should be detected and taken care of immediately [4, 5].

The approaches to calibrate OOD confidence for learning algorithms described in the literature can

be roughly divided into two groups: discriminative and generative. Intuitively, the easiest solution for

OOD confidence calibration is to learn a function that gives higher scores for in-distribution samples

and lower scores for OOD samples [6]. The discriminative approaches try to either modify the loss

function [7-9] or train the network exhaustively on OOD datasets to calibrate on OOD samples [10, 4].

Recently, Hein et al. [4] showed ReLU networks produce arbitrarily high confidence as the inference 33

point moves far away from the training data. Therefore, calibrating ReLU networks for the whole 34

OOD region is not possible without fundamentally changing the network architecture. As a result, all 35

of the aforementioned algorithms are unable to provide any guarantee about the performance of the 36

network throughout the whole feature space. The other group tries to learn generative models for the 37

in-distribution as well as the out-of-distribution samples. The general idea is to do likelihood ratio 38 test for a particular sample using the generative models [11], or threshold the ID likelihoods to detect 39 OOD samples. However, it is not obvious how to control likelihoods far away from the training data 40 for powerful generative models like variational autoencoders (V AEs) [12] and generative adversarial 41 networks (GAN) [13]. Moreover, Nalisnick et al. [14] and Hendrycks et al. [10] showed VAEs and 42 GANs can also yield overconfident likelihoods far away from the training data. 43

The algorithms described so far are concerned with OOD confidence calibration for deep-nets only. 44 However, we show that other approaches which partition the feature space, for example random forest, 45 can also suffer from poor confidence calibration both in the ID and the OOD regions. Moreover, the 46 algorithms described above are concerned about the confidence in the OOD region only and do not 47 address the confidence calibration within the ID region at all. This issue is addressed separately in 48 a different group of literature [15-20]. Instead, we consider both calibration problems jointly and 49 propose an approach that achieves good calibration throughout the whole feature space. 50

51

52

53

54

55

56

57

In this paper, we conceptualize both random forest and ReLU networks as partitioning rules with an affine activation over each polytope. We consider replacing the affine functions learned over the polytopes with Gaussian kernels. We propose two novel kernel density estimation techniques named Kernel Density Forest ( KDF ) and Kernel Density Network ( KDN ). Our proposed approach completely excludes the need for training on OOD examples for the model (unsupervised OOD calibration). We conduct several simulation and real data studies that show both KDF and KDN are well-calibrated for OOD samples while they maintain good performance in the ID region.

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

2 Related Works and Our Contributions

There are a number of approaches in the literature which attempt to learn a generative model and control the likelihoods far away from the training data. For example, Ren et al. [11] employed likelihood ratio test for detecting OOD samples. Wan et al. [8] modified the training loss so that the downstream projected features follow a Gaussian distribution. However, there is no guarantee of performance for OOD detection for the above methods. To the best of our knowledge, apart from us, only Meinke et al. [5] has proposed an approach to guarantee asymptotic performance for OOD detection. Compared to the aforementioned methods, our approach differs in several ways:

We address the confidence calibration problem for both ReLU -nets and random forests.

We address ID and OOD calibration problem as a continuum.

Weprovide an algorithm for OOD confidence calibration for both tabular and vision datatsets whereas most of the existing methods are tailor-made for vision problems.

We propose an unsupervised post-hoc OOD calibration approach.

3 Technical Background

3.1 Setting

Consider a supervised learning problem with independent and identically distributed training samples 73 { ( x i , y i ) } n i =1 such that ( X , Y ) ∼ P X,Y , where X ∼ P X is a X ⊆ R D valued input and Y ∼ P Y is a 74 Y = { 1 , · · · , K } valued class label. Let S be the high density region of the marginal, P X , thus S ⊊ 75 X . Here the goal is to learn a confidence score, g : R D → [0 , 1] K , g ( x ) = [ g 1 ( x ) , g 2 ( x ) , . . . , g K ( x )] 76 such that, 77

where P Y | X ( y | x ) is the posterior probability for class y given by the Bayes formula: 78

Here P X | Y ( x | y ) is the class conditional density which we will refer as f y ( x ) hereafter for brevity. 79

3.2 Main Idea 80

Deep discriminative networks partition the feature space R d into a union of p affine polytopes Q r 81 such that ⋃ p r =1 Q r = R d , and learn an affine function over each polytope [4, 21]. Mathematically, 82 the unnormalized class-conditional density for the label y estimated by these deep discriminative 83 models at a particular point x can be expressed as: 84

For example, in the case of a decision tree, a r = 0 , i.e., decision tree assumes uniform distribution 85 for the class-conditional densities over the leaf nodes. Among these polytopes, the ones that lie on 86 the boundary of the training data extend to the whole feature space and hence encompass all the OOD 87 samples. Since the posterior probability for a class is determined by the affine activation over each of 88 these polytopes, the algorithms tend to be overconfident when making predictions on the OOD inputs. 89 Moreover, there exist some polytopes that are not populated with training data. These unpopulated 90 polytopes serve to interpolate between the training sample points. If we replace the affine activation 91 function of the populated polytopes with Gaussian kernels and prune the unpopulated ones, the tail of 92 the kernel will help interpolate between the training sample points while assigning lower likelihood to 93 the low density or unpopulated polytope regions of the feature space. This results in better confidence 94 calibration for the proposed modified approach. 95

96

97

98

99

100

101

102

103

104

3.3 Proposed Approach

We will call the above discriminative approaches as the 'parent approach' hereafter. Consider the collection of polytope indices P from the parent approach which are populated by the training data. We replace the affine functions over the populated polytopes with Gaussian kernels G ( · ; ˆ µ r , ˆ Σ r ) . For a particular inference point x , we consider the Gaussian kernel with the minimum distance from the center of the kernel to the corresponding point:

where ∥ · ∥ denotes a distance. As we will show later, the type of distance metric considered in Equation 4 highly impacts the performance of the proposed model. In short, we modify Equation 3 from the parent ReLU -net or random forest to estimate the class-conditional density (unnormalized):

where n y is the total number of samples with label y and n ry is the number of samples from class y 105 that end up in polytope Q r . We add a small constant to the class conditional density ˜ f y : 106

Note that in Equation 6, b log( n ) → 0 as the total training points, n →∞ . The intuition behind the 107 added constant will be clarified further later in Proposition 2. The confidence score ˆ g y ( x ) for class y 108 given a test point x is estimated using the Bayes rule as: 109

where ˆ P Y ( y ) is the empirical prior probability of class y estimated from the training data. We 110 estimate the class for a particular inference point x as: 111

4 Model Parameter Estimation 112

4.1 Gaussian Kernel Parameter Estimation 113

114

115

We fit Gaussian kernel parameters to the samples that end up in the r -th polytope. We set the kernel center along the d -th dimension:

where x d i is the value of x i along the d -th dimension. We set the kernel variance along the d -th 116 dimension: 117

where λ is a small constant that prevents ˆ σ d r from being 0 . We constrain our estimated Gaussian 118 kernels to have diagonal covariance. 119

120

121

122

123

4.2 Sample Size Ratio Estimation

For a high dimensional dataset with low training sample size, the polytopes are sparsely populated with training samples. For improving the estimate of the ratio n ry n y in Equation 5, we incorporate the samples from other polytopes Q s based on the similarity w rs between Q r and Q s as:

As n →∞ , the estimated weights w rs should satisfy the condition: 124

̸

For simplicity, we will describe the estimation procedure for w rs in the next sections. Note that if we 125 satisfy Condition 12, then we have ˆ n ry ˆ n y → n ry n y as n →∞ . Therefore, we modify Equation 5 as: 126

where ˆ r ∗ x = argmin r ∥ ˆ µ r -x ∥ . Now we use ˆ f y ( x ) estimated using (13) in Equation (6), (7) and (8), 127 respectively. Below, we describe how we estimate w rs for KDF and KDN . 128

129

130

131

132

133

4.3 Forest Kernel

Consider T number of decision trees in a random forest trained on n iid training samples { ( x i , y i ) } n i =1 . Each tree t partitions the feature space into p t polytopes resulting in a set of polytopes: {{ Q t,r } p t r =1 } T t =1 . The intersection of these polytopes gives a new set of polytopes { Q r } p r =1 for the forest. For any two points x ∈ Q r and x ′ ∈ Q s , we define the kernel K ( r, s ) as:

where t rs is the total number of trees, x and x ′ end up in the same leaf node. Here, 0 ≤ K ( r, s ) ≤ 1 . 134

If the two samples end up in the same leaf in all the trees, i.e., K ( r, s ) = 1 , they belong to the same 135 polytope, i.e. r = s . In short, K ( r, s ) is the fraction of total trees where the two samples follow the 136 same path from the root to a leaf node. We exponentiate K ( r, s ) so that Condition 12 is satisfied: 137

We choose k using grid search on a hold-out dataset. 138

4.4 Network Kernel 139

140

141

142

143

144

145

146

Consider a fully connected L layer ReLU -net trained on n iid training samples { ( x i , y i ) } n i =1 . We have the set of all nodes denoted by N l at a particular layer l . We can randomly pick a node n l ∈ N l at each layer l , and construct a sequence of nodes starting at the input layer and ending at the output layer which we call an activation path : m = { n l ∈ N l } L l =1 . Note that there are N = Π L i =1 |N l | possible activation paths for a sample in the ReLU -net. We index each path by a unique identifier number z ∈ N and construct a sequence of activation paths as: M = { m z } z =1 , ··· ,N . Therefore, M contains all possible activation pathways from the input to the output of the network.

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

While pushing a training sample x i through the network, we define the activation from a ReLU unit at any node as ' 1 ' when it has positive output and ' 0 ' otherwise. Therefore, the activation indicates on which side of the affine function at each node the sample falls. The activation for all nodes in an activation path m z for a particular sample creates an activation mode a z ∈ { 0 , 1 } L . If we evaluate the activation mode for all activation paths in M while pushing a sample through the network, we get a sequence of activation modes: A r = { a r z } N z =1 . Here r is the index of the polytope where the sample falls in.

If the two sequences of activation modes for two different training samples are identical, they belong to the same polytope. In other words, if A r = A s , then Q r = Q s . This statement holds because the above samples will lie on the same side of the affine function at each node in different layers of the network. Now, we define the kernel K ( r, s ) as:

Note that 0 ≤ K ( r, s ) ≤ 1 . In short, K ( r, s ) is the fraction of total activation paths which are identically activated for two samples in two different polytopes r and s . We exponentiate the kernel using Equation 15. Pseudocodes outlining the two algorithms are provided in Appendix D.

4.5 Geodesic Distance

Consider P n = { Q 1 , Q 2 , · · · , Q p } as a partition of R d given by a random forest or a ReLU -net after being trained on n training samples. We measure distance between two points x ∈ Q r , x ′ ∈ Q s using the kernel introduced in Equation 14 and Equation 16, and call it 'Geodesic' distance [22]:

Proposition 1. ( P n , d ) is a metric space.

Proof. See Appendix A.1 for the proof.

We use Geodesic distance to find the nearest polytope to the inference point. As Geodesic distance cannot distinguish between points within the same polytope, it has a resolution similar to the size of the polytope. For discriminating between two points within the same polytope, we fit a Gaussian kernel within the polytope (described above). As h n → 0 , the resolution for Geodesic distance improves. In Section 5, we will empirically show that using Geodesic distance scales better with higher dimension compared to that of Euclidean distance.

Given n training samples { ( x i , y i ) } n i =1 , we define the distance of an inference point x from the training points as: d x = min i =1 , ··· ,n ∥ x -x i ∥ , where ∥ · ∥ denotes Euclidean distance.

Proposition 2 (Asymptotic OOD Convergence) . Given non-zero and bounded bandwidth of the Gaussians, then we have almost sure convergence for ˆ g y as:

Proof. See Appendix A.2 for the proof. 177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

5 Empirical Results

We conduct several experiments on simulated, OpenML-CC18 [23] 1 and vision benchmark datasets to gain insights on the finite sample performance of KDF and KDN . The details of the simulation datasets and hyperparameters used for all the experiments are provided in Appendix C. For Trunk simulation dataset, we follow the simulation setup proposed by Trunk [24] which was designed to demonstrate 'curse of dimensionality'. In the Trunk simulation, a binary class dataset is used where each class is sampled from a Gaussian distribution with higher dimensions having increasingly less discriminative information. We use both Euclidean and Geodesic distance to detect the nearest polytope (see Equation (4)) on simulation datasets and use only Geodesic distance for benchmark datasets. For the simulation setups, we use classification error, Hellinger distance [25, 26] from the true class conditional posteriors and mean max confidence [4] as performance statistics. While measuring in-distribution calibration for the datasets in OpenML-CC18 data suite, we used maximum calibration error as defined by Guo et al. [18] with a fixed bin number of R = 15 across all the datasets. Given n OOD samples, we define OOD calibration error (OCE) to measure OOD performance for the benchmark datasets as:

For the tabular and the vision datasets, we have used ID calibration approaches, such as Isotonic [15, 16] and Sigmoid [17] regression, as baselines. Additionally, for the vision benchmark dataset, we provide results with OOD calibration approaches such as: ACET [4], ODIN [6], OE (outlier exposure) [10]. For each approach, 70% of the training data was used to fit the model and the rest of the data was used to calibrate the model.

5.1 Empirical Study on Tabular Data

5.1.1 Simulation Study

Figure 1 leftmost column shows 10000 training samples with 5000 samples per class sampled within the region [ -1 , 1] × [ -1 , 1] from the six simulation setups described in Appendix C. Therefore, the empty annular region between [ -1 , 1] × [ -1 , 1] and [ -2 , 2] × [ -2 , 2] is the low density or OOD region in Figure 1. Figure 1 quantifies the performance of the algorithms which are visually represented in Appendix Figure 4. KDF and KDN maintain similar classification accuracy to those of their parent algorithms. We measure hellinger distance from the true distribution for increasing training sample size within [ -1 , 1] × [ -1 , 1] region as a statistics for in-distribution calibration. Column 3 and 6 in Figure 1 show KDF and KDN are better at estimating the ID region compared to their parent methods. In all of the simulations, using geodesic distance measure results in better performance compared to those while using Euclidean distance. For measuring OOD performance, we keep the training sample size fixed at 1000 and normalize the training data by the maximum of their l 2 norm so that the training data is confined within a unit circle. For inference, we sample 1000 inference points uniformly from a circle where the circles have increasing radius and plot mean max posterior for increasing distance from the origin. Therefore, for distance up to 1 we have in-distribution samples and distances farther than 1 can be considered as OOD region. As shown in Column 4 and 7 of Figure 1, mean max confidence for KDF and KDN converge to the maximum of the class priors, i.e., 0 . 5 as we go farther away from the training data origin.

Row 6 of Figure 1 shows KDF -Geodesic and KDN -Geodesic scale better with higher dimensions compared to their Euclidean counterpart algorithms respectively.

5.1.2 OpenML-CC18 Data Study

We use OpenML-CC18 data suite for tabular benchmark dataset study. We exclude any dataset which contains categorical features or NaN values 2 and conduct our experiments on 45 datasets with varying dimensions and sample sizes. For the OOD experiments, we follow a similar setup as that of the simulation data. We normalize the training data by their maximum l 2 norm and sample 1000

A.

B.

0.1

0.0

0.1

Improvement over parent

0.5

0.0

1.0

0.2

0.0

0.2

10

2

10

3

10

4

0.05

0.00

0.05

Improvement over parent

10

2

10

3

10

4

0.2

0.0

1.0

1

3

5

0.2

0.0

0.8

Number of Training Samples (log)

Distance

Classification

ID Calibration

OOD Calibration

KDF

Isotonic RF

Parent RF

KDN

Isotonic DN

Sigmoid DN

Parent DN

testing samples uniformly from hyperspheres where each hypersphere has increasing radius starting 224 from 1 to 5 . For each dataset, we measure improvement with respect to the parent algorithm: 225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

A. Circle (ID)

B. Rectange (ID)

0

127

255

Color Intensity

0.5

0.8

1.0

D.

0.0

0.5

1.0

E.

0

10

40

Axis Length

F.

KDN

CNN

where E p = classification error, MCE or OCE for the parent algorithm and E M represents the performance of the approach in consideration. Note that positive improvement implies the corresponding approach performs better than the parent approach. We report the median of improvement on different datasets along with the error bar in Figure 2. The extended results for each dataset is shown separately in the appendix. Figure 2 left column shows on average KDF and KDN has nearly similar or better classification accuracy compared to their respective parent algorithm whereas Isotonic and Sigmoid regression have lower classification accuracy most of the cases. However, according to Figure 2 middle column, KDF and KDN have similar in-distribution calibration performance to the other baseline approaches. Most interestingly, Figure 2 right column shows that KDN and KDF improves OOD calibration of their respective parent algorithms by a huge margin while the baseline approaches completely fails to address the OOD calibration problem.

5.2 Empirical Study on Vision Data

In vision data, each image pixel contains local information about the neighboring pixels. To extract the local information, we use convolutional or vision transformer encoders at the front-end. More precisely, we have a front-end encoder, h e : R D ↦→ R m and typically, m << D . After the encoder there is a few fully connected dense layers for discriminating among the K class labels, h f : R m ↦→ R K . Note that the m -dimensional embedding outputs from the encoder are partitioned into polytopes by the dense layers (see Equation (3)) and we fit a KDN on the embedding outputs. The above approach results in extraction of better inductive bias by KDN from the parent model and makes KDN more scalable with larger parent models and training sample size.

5.2.1 Simulation Study

For the simulation study, we use a simple CNN with one convolutional layer ( 3 channels with 3 × 3 kernel) followed by two fully connected layers with 10 and 2 nodes in each. We train the CNN on 2000 circle (radius 10 ) and 2000 rectangle (sides 20 , 50 ) images with their RGB values being fixed at [127 , 127 , 127] and their centers randomly sampled within a square with sides 100 . The other pixels in the background where there is no object (circle, rectangle or ellipse) were set to 0 .

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

We perform three experiments while inducing semantic shifts in the inference points as shown in Figure 3. In the first experiment, we randomly sampled data similar to the training points. However, we added the same shift to all the RGB values of an inference point (shown as color intensity in Figure 3 D). Therefore, the inference point is ID for color intensity at 127 and otherwise OOD. In the second experiment, we kept the RGB values fixed at [127 , 127 , 127] while taking convex combination of a circle and a rectangle. Let images of circles and rectangles be denoted by X c and X r . We derive an interference point as X inf :

Therefore, X inf is maximally distant from the training points for ϵ = 0 . 5 and closest to the ID points at ϵ = { 0 , 1 } . In the third experiment, we sampled ellipse images with the same RGB values as the training points. However, this time we gradually change one of the ellipse axes from 0 . 01 to 40 while keeping the other axis fixed at 10 . As a result, the inference point becomes ID for the axis length of 10 . As shown in Figure 3 (D, E, F), in all the experiments KDN becomes less confident for the OOD points while the parent CNN remains overconfident throughout the semantic shifts of the test points.

5.2.2 Vision Benchmark Datasets Study

In this study, we use a V iT _ B 16 (provided in keras-vit package) vision transformer encoder [27] pretrained on ImageNet [28] dataset and finetuned on CIFAR-10 [29]. We use the same encoder for all the baseline algorithms and finetune it with the corresponding loss function without freezing any weight. As shown in Table 1, pretrained vision transformers are already well-calibrated for ID and the OOD approaches ( ACET , ODIN , OE ) degrade ID calibration of the parent model. On the contrary, ID calibration approaches ( Isotonic , Sigmoid ) perform poorly compared to that of KDN in the OOD region. KDN achieves a compromise between ID and OOD performance while having reduced confidence on wrongly classified ID samples. The number of populated polytopes (and Gaussians) for KDN is 9323 ± 353 . See Appendix F for the corresponding experiments using Resnet-50 .

6 Limitations

Training time complexity for KDF and KDN is O ( n 2 l f ) which is dominated by the Geodesic distance calculation. Here l f = total number of leaves in the forest or total nodes in the dense layers of the network and n = total training samples. However, the distance calculation can be done in parallel using our provided code. Additionally, note that the number of Gaussian kernel used by KDN is upper bounded by number of training samples. Therefore, KDN may not scale for really big datasets like ImageNet [28]. However, the scaling issue may be solved by selectively pruning neighboring polytopes which we will pursue in future.

7 Discussion

In this paper, we demonstrated a simple intuition that renders traditional deep discriminative models into a type of binning and kerneling approach. The bin boundaries are determined by the internal structure learned by the parent approach and Geodesic distance encodes the low dimensional structure learned by the model. Moreover, Geodesic distance introduced in this paper may have broader impact on understanding the internal structure of the deep discriminative models which we will pursue in future. Our code, including the package and the experiments in this manuscript, will be made publicly available upon acceptance of the paper.

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

References

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1321-1330. PMLR, 06-11 Aug 2017.

Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just a bit, fixes overconfidence in ReLU networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 5436-5446. PMLR, 13-18 Jul 2020.

Haoyin Xu, Kaleab A. Kinfu, Will LeVine, Sambit Panda, Jayanta Dey, Michael Ainsworth, Yu-Chung Peng, Madi Kusmanov, Florian Engert, Christopher M. White, Joshua T. Vogelstein, and Carey E. Priebe. When are Deep Networks really better than Decision Forests at small sample sizes, and how? arXiv preprint arXiv:2108.13637 , 2021.

Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 41-50, 2019.

Alexander Meinke, Julian Bitterwolf, and Matthias Hein. Provably robust detection of out-ofdistribution data (almost) for free. arXiv preprint arXiv:2106.04260 , 2021.

Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-ofdistribution image detection in neural networks. arXiv preprint arXiv:1706.02690 , 2017.

Jay Nandy, Wynne Hsu, and Mong Li Lee. Towards maximizing the representation gap between in-domain & out-of-distribution examples. Advances in Neural Information Processing Systems , 33:9239-9250, 2020.

Weitao Wan, Yuanyi Zhong, Tianpeng Li, and Jiansheng Chen. Rethinking feature distribution for loss functions in image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 9117-9126, 2018.

Terrance DeVries and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint arXiv:1802.04865 , 2018.

Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606 , 2018.

Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. Advances in neural information processing systems , 32, 2019.

Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning , 12(4):307-392, 2019.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint arXiv:1810.09136 , 2018.

Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml , volume 1, pages 609-616, 2001.

R Caruana. Predicting good probabilities with supervised learning. In Proceedings of NIPS 2004 Workshop on Calibration and Probabilistic Prediction in Supervised Learning , 2004.

John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers , 10(3):61-74, 1999.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural 339 networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017. 340

Richard Guo, Ronak Mehta, Jesus Arroyo, Hayden Helm, Cencheng Shen, and Joshua T 341 Vogelstein. Estimating information-theoretic quantities with uncertainty forests. arXiv , pages 342 arXiv-1907, 2019. 343

344

345

346

Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems , 32, 2019.

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

Haoyin Xu, Kaleab A Kinfu, Will LeVine, Sambit Panda, Jayanta Dey, Michael Ainsworth, Yu-Chung Peng, Madi Kusmanov, Florian Engert, Christopher M White, et al. When are deep networks really better than decision forests at small sample sizes, and how? arXiv preprint arXiv:2108.13637 , 2021.

Bernhard Schölkopf. The kernel trick for distances. Advances in neural information processing systems , 13, 2000.

Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G Mantovani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmarking suites. arXiv preprint arXiv:1708.03731 , 2017.

Gerard V Trunk. A problem of dimensionality: A simple example. IEEE Transactions on pattern analysis and machine intelligence , (3):306-307, 1979.

Thomas Kailath. The divergence and bhattacharyya distance measures in signal selection. IEEE transactions on communication technology , 15(1):52-60, 1967.

C Radhakrishna Rao. A review of canonical coordinates and an alternative to correspondence analysis using hellinger distance. Qüestiió: quaderns d'estadística i investigació operativa , 1995.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009. doi: 10.1109/CVPR.2009.5206848.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://www.cs.toronto.edu/~kriz/cifar.html .

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems , 33:18661-18673, 2020.

A Proofs 375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

A.1 Proof of Proposition 1

For proving that d is a valid distance metric for P n , we need to prove the following four statements:

d ( r, s ) = 0 when r = s . Proof: By definition, K ( r, s ) = 1 and d ( r, s ) = 0 when r = s .

̸

d ( r, s ) > 0 when r = s . Proof: By definition, 0 ≤ K ( r, s ) < 1 and d ( r, s ) > 0 for r = s .

̸

d is symmetric, i.e., d ( r, s ) = d ( s, r ) . Proof: By definition, K ( r, s ) = K ( s, r ) which implies d ( r, s ) = d ( s, r ) .

4. d follows the triangle inequality, i.e., for any three polytopes Q r , Q s , Q t ∈ P n : d ( r, t ) ≤ d ( r, s ) + d ( s, t ) . Proof: Let A r denote the set of activation modes in a ReLU -net and the set of leaf nodes in a random forest for a particular polytope r . N is the total number of possible activation paths in a ReLU -net or total trees in a random forest. Below c ( · ) denotes the cardinality of the set. We can write:

Rearranging the above equation, we get:

A.2 Proof of Proposition 2

Note that first we find the nearest polytope to the inference point x using Geodesic distance and use 392 Gaussian kernel locally for x within that polytope. Here the Gaussian kernel uses Euclidean distance 393 from the kernel center to x (within the numerator of the exponent). The value out of the Gaussian 394 kernel decays exponentially with the increasing distance of the inference point from the kernel center. 395

We first expand ˆ g y ( x ) : 396

As the inference point x becomes more distant from training samples (and more distant from all of the Gaussian centers), we have that G ( x , ˆ µ r , ˆ Σ r ) becomes smaller. Thus, ∀ y, ˜ f y ( x ) shrinks. More formally, ∀ y ,

We can use this result to then examine the limiting behavior of our posteriors as the inference point x 397 becomes more distant from the training data: 398

B Hardware and Software Configurations 399

400

Operating System: Linux (ubuntu 20.04), macOS (Ventura 13.2.1)

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

VMSize: Azure Standard D96as v4 (96 vcpus, 384 GiB memory)

GPU: Apple M1 Max

Software: Python 3.8, scikit-learn ≥ 0.22.0, tensorflow-macos ≤ 2.9, tensorflow-metal ≤ 0.5.0.

C Simulations

We construct six types of binary class simulations:

Gaussian XOR is a two-class classification problem with equal class priors. Conditioned on being in class 0, a sample is drawn from a mixture of two Gaussians with means ± [0 . 5 , -0 . 5] ⊤ and standard deviations of 0 . 25 . Conditioned on being in class 1, a sample is drawn from a mixture of two Gaussians with means ± [0 . 5 , -0 . 5] ⊤ and standard deviations of 0 . 25 .

Spiral is a two-class classification problem with the following data distributions: let K be the number of classes and S ∼ multinomial ( 1 K ⃗ 1 K , n ) . Conditioned on S , each feature vector is parameterized by two variables, the radius r and an angle θ . For each sample, r is sampled uniformly in [0 , 1] . Conditioned on a particular class, the angles are evenly spaced between 4 π ( k -1) t K K and 4 π ( k ) t K K , where t K controls the number of turns in the spiral. To inject noise along the spirals, we add Gaussian noise to the evenly spaced angles θ ′ : θ = θ ′ + N (0 , 0 . 09) . The observed feature vector is then ( r cos( θ ) , r sin( θ )) .

Circle is a two-class classification problem with equal class priors. Conditioned on being in class 0, a sample is drawn from a circle centered at (0 , 0) with a radius of r = 0 . 75 . Conditioned on being in class 1, a sample is drawn from a circle centered at (0 , 0) with a radius of r = 1 , which is cut off by the region bounds. To inject noise along the circles, we add Gaussian noise to the circle radii r ′ : r = r ′ + N (0 , 0 . 01) .

Sinewave is a two-class classification problem based on sine waves. Conditioned on being in class 0, a sample is drawn from the distribution y = cos( πx ) . Conditioned on being in class 1, a sample is drawn from the distribution y = sin( πx ) . We inject Gaussian noise to the sine wave heights y ′ : y = y ′ + N (0 , 0 . 01) .

Polynomial is a two-class classification problem with the following data distributions: y = x a . Conditioned on being in class 0, a sample is drawn from the distribution y = x 1 . Conditioned on being in class 1, a sample is drawn from the distribution y = x 3 . Gaussian noise is added to variables y ′ : y = y ′ + N (0 , 0 . 01) .

Trunk is a two-class classification problem with gradually increasing dimension and equal class priors. The class conditional probabilities are Gaussian:

RF

KDF-Geodesic

KDF-Euclidean

DN

KDN-Geodesic

KDN-Euclidean

Spiral

?

?

?

?

Circle

1.0

0.5

Sinewave

W

W

0.0

-2+

Polynomial

yun

2-2-10

-2-10

12

2-2-1012-2-10i

where µ 1 = µ, µ 2 = -µ , µ is a d dimensional vector whose i -th component is ( 1 i ) 1 / 2 and I 434 is d dimensional identity matrix. 435

D Pseudocodes 436

We provide the pseudocode for our porposed algorithms in Algorithm 1, 2 and 3. 437

E Extended Results on OpenML-CC18 data suite 438

See Figure 5, 6, 7 and 8 for extended results on OpenML-CC18 data suite. 439

Algorithm 1 Fit a KDX model.

Algorithm 2 Computing weights in KDF

Input: (1) x i , x j ∈ R 1 × d ▷ two input samples to be weighted (2) θ ▷ parent random forest with T trees Output: w ij ∈ [0 , 1] ▷ compute similarity between i and j -th samples. 1: function COMPUTEWEIGHTS( x i , x j , θ ) 2: I i ← PUSHDOWNTREES( x i , θ ) ▷ push x i down T trees and get the leaf numbers it end up in. 3: I j ← PUSHDOWNTREES( x j , θ ) ▷ push x j down T trees and get the leaf numbers it end up in. 4: l ← COUNTMATCHES( I i , I j ) ▷ count the number of times the samples end up in the same leaf 5: w ij ← l T 6: return w ij 7: end function

Algorithm 3 Computing weights in KDN

440

F Extended Results on Vision datasets using Resnet-50

In this experiments, we use a Resnet-50 encoder pretrained using contrastive loss [30] as described 441 in http://keras.io/examples/vision/supervised-contrastive-learning . The encoder 442 projects the input images down to a 256 dimensional latent space and we add two dense layers with 443 200 and 10 nodes on top of the encoder. We use the same pretrained encoder for all the baseline 444 algorithms. 445

As shown in Table 4, KDN achieves good calibration for both ID and OOD datasets whereas the ID 446 calibration approaches are poorly calibrated in the OOD regions and the OOD approaches have poor 447

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We enumerate list of contributions in Section 2 which are also mentioned in the abstract and the introduction.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the limitations of the work in Section 6 to the best of our knowledge.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All of our theoretical results are discussed in Section 4.5 and proofs are provided in the Appendix.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All code that was used to create the experiments has been provided as a part of the supplementary material, and will be made public after the review process.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental setting are described adequately in the main paper with more details discussed in the Appendix.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have repeated the experiments over several Monte Carlo repetitions and have reported the error bars in the figures and the tables.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

Answer: [Yes]

Justification: We have provided the computation platform and resources used in the Appendix B.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We did not conduct experiments with human participants and we do not anticipate any societal harmful consequences.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper demonstrates a simple intuition that can increase the confidence calibration for modern deep machine learning approaches. Although the proposed work is not directly related to any societal impact, down the line we anticipate it will pave the way for safety critical application of machine learning models.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We only use existing public datasets (appropriately cited in the paper) and synthetic data. No new assets are created from this paper.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No assets are created in this paper.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No such study was performed.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

544

Justification: No such study was performed.

545

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes new methods, Kernel Density Forest (KDF) and Kernel Density Network (KDN), to address issues in confidence calibration for traditional deep learning models and random forests. The motivation stems from the existing literature that deep neural networks using ReLU tend to exhibit high confidence on out-of-distribution (OOD) data due to affine transformations. The proposed methods improve confidence calibration for both in-distribution (ID) and OOD data by partitioning the feature space into polytopes and replacing affine functions within each polytope with Gaussian kernels. Experimental results demonstrate that the proposed methods outperform existing techniques in terms of calibration performance.\n\nSTRENGTHS:\nOriginality:\nThe approach of replacing affine functions within polytopes with Gaussian kernels is novel. The proposed methods address the confidence calibration problem for both ID and OOD data simultaneously, providing an integrated solution to these calibration issues.\n\nQuality:\nThe theoretical proofs are robust, and the effectiveness of the proposed methods is validated through both simulations and real-world datasets.\n\nClarity:\nThe paper is written clearly and concisely.\n\nWEAKNESSES:\nValidity of Metrics:\nThe paper evaluates calibration using Maximum Calibration Error (MCE) for ID data, but does not justify the use of MCE over Expected Calibration Error (ECE) or Adaptive Calibration Error (ACE)[1]. A more detailed explanation and comparison of these metrics would enhance the paper's credibility. Additionally, the definition and justification for OCE (Out-of-distribution Calibration Error) would benefit from a similar comparison with ACE.\n\n[1] https://arxiv.org/abs/1904.01685\n\nExperiments:\nTo emphasize the effectiveness of the proposed methods, a comparison of execution times would be beneficial, especially since practical applications like web Click-Through Rate (CTR) estimation place significant importance on runtime. The paper should clarify what the noise in Table 1 represents. It would also be advantageous to include experiments on larger and more varied datasets, as well as an evaluation of the methods' performance when combined with in-training calibration methods, which are commonly used alongside post-hoc calibration methods.\n\nQUESTIONS:\nDoes the formulation of OCE assume a lack of usable features from the in-distribution domain? What practical and theoretical conditions are required for this assumption? For instance, it is known that large parameter models can improve OOD ECE even when trained with ERM on in-distribution features [2].\n\n[2] https://arxiv.org/abs/2307.08187\n\nLIMITATIONS:\nThis paper mentions computational complexity and limitations in practical applications, but lacks detailed experimental results to support these claims. Including such data would provide valuable insights for future research and implementation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach for OOD detection by learning a posterior distribution that is calibrated for both ID and OOD individuals. It models the class-wise conditional distribution of features by a gaussian kernel respectively for a set of polytopes that cover the feature space. The tail property of gaussian kernels contribute to both ID and OOD calibration. Empirical evidence shows the power of the proposed algorithm across tabular and vision dataset under both ID and OOD settings.\n\nSTRENGTHS:\nThe paper is well motivated from the tradeoff of ID calibration and OOD calibration for current approaches for OOD detection methods. The technique of gaussian kernel has a clear geometric intuitive. Compared to affine functions, the tail property ensures that the posterior distribution converges to the prior of labels when a OOD sample deviates far enough from the training support, as proved in Proposition 2. On the other hand, the interpolation by gaussian kernels between neighboring polytopes contributes to ID calibration.\n\nWEAKNESSES:\nThe major concern is insufficient discussion over the research context of the paper, which renders it hard to precisely evaluate the contribution. The related work section is short. Section 2 shows that \"OOD detection\" is the closest area to this paper, but this keyword is totally absent from the introduction, where the research area is named \"OOD confidence calibration\". What is the relation between OOD detection and OOD confidence calibration? \nThe introduction also reveals two potential approaches for this area: discriminative and generative methods. There are also two settings: ID and OOD confidence calibration. The readers might expect to review current progress for all those categories in the related work section.\n\nQUESTIONS:\n1. In Eq 3, how to ensure the non-negativity of the class-conditional density? The model of affine function might take negative values.\n2. The model of class-conditional density with gaussian kernels, referring to Eq 5, resembles Kernel Density Estimation. Could the author please discuss the relation between KDE and the proposed method?\n3. In Fig 1, KDF shows a weaker advantage for low dimensional settings. Could the author please explain why?\n4. In Fig 2, why are OOD approaches absent? Fig 2 demonstrates the effectiveness of KDF compared to ID approaches in terms of OOD calibration. The readers could be more interested in the performance of OOD approaches.\n5. In Table 1, it seems OOD approaches including ACET and ODIN can't even beat the parent model under OOD settings. Could the author please explain why?\n\nLIMITATIONS:\nThe author has addressed limitations of their work in terms of sample complexity.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a way to calibrate ReLU networks or random forests by breaking them down into piecewise linear functions on polytopes and replacing the linear parts with Gaussian kernels. This approximation allows to naturally calibrate the models for the ID domain, where confidence will be high due to the density of ID samples that translates into high kernel values, and for the OOD domain, where confidence will be low due to the large distance to ID samples.\n\nSTRENGTHS:\n- The method is novel and mathematically grounded\n- The presentation is clear\n- The benchmarks are OK\n\nWEAKNESSES:\nThe main weakness I find is about the computational time of the method. The number of polytopes scales exponentially with the number of neurons, so I am concerned with the applicability of the method to large (or even medium-scale) neural networks. What is the computational cost of the method for the considered benchmarks, in terms of runtime?\n\nThe toy simulations are unnecessarily tedious to grasp and take up a lot of space. I do not say that they are complex, but they hinder the reading flow and do not bring much to the presentation. I would advise putting some of them in the appendix to leave more space for other explanations. Indeed, Section 5 is difficult to read (many \"chunk\" paragraphs with mathematical notations) and would benefit from more structured writing and more flow.\n\nQUESTIONS:\n1. Section 4.4 How is $\\omega_{rs}$ estimated in that case?\n2. Table 1: what is the Parent approach exactly compared to KDN and KDF? The definition of Section 3.3 does not seem to describe a whole model but only the parent idea that is then declined with KDN and KDF. \n3. What would be the AUROC, FPR, i.e. metrics commonly used in OOD detection?\n4. Figure 1. What is \"dimensions\" in x-axis?\n5. l. 192 I do not get how OCE measures an error since it only includes estimated quantities, and I do not see the estimated confidence score $\\hat{g}_y(x)$. In addition, are $x_i$ OOD samples in that case? If yes it should be made explicit.\n6. l. 213 I disagree that normalizing ensures that distances up to 1 are ID and above are OOD. There can be \"holes\" in the distribution (as the circle dataset), or modes. What is the authors' opinion about that?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}