Simple Calibration via Geodesic Kernels
Authors: Jayanta Dey
Haoyin Xu
Ashwin De Silva
Joshua T Vogelstein
Venue: Accepted by TMLR ()

Abstract: Deep discriminative approaches, such as decision forests and deep neural networks, have recently found applications in many important real-world scenarios. However, deploying these learning algorithms in safety-critical applications raises concerns, particularly when it comes to ensuring calibration for both in-distribution and out-of-distribution regions. Many popular methods for in-distribution (ID) calibration, such as isotonic and Platt’s sigmoidal regression, exhibit adequate ID calibration performance. However, these methods are not calibrated for the entire feature space, leading to overconfidence in the out-of-distribution (OOD) region. Existing OOD calibration methods generally exhibit poor ID calibration. In this paper, we jointly address the ID and OOD problems. We leveraged the fact that deep models learn to partition feature space into a union of polytopes, that is, flat-sided geometric objects. We introduce a geodesic distance to measure the distance between these polytopes and further distinguish samples within the same polytope using a Gaussian kernel. Our experiments on both tabular and vision benchmarks show that the proposed approaches, namely Kernel Density Forest (KDF) and Kernel Density Network (KDN), obtain well-calibrated posteriors for both ID and OOD samples,  while mostly preserving the classification accuracy and extrapolating beyond the training data to handle OOD inputs appropriately.

Published in Transactions on Machine Learning Research (06/2025)
Simple Calibration via Geodesic Kernels
Jayanta Dey
jdey4 Bfjlmmi edu
Department of Biomedical Engineering Johns Hopkins University
Haoyin Xu
ha: edu
Deportment of Biomedical Engineering Johns Hopkins University
Ashwin De Silva
Idesil edu
Department of Biomedical Engineering
Johns Hopkins Universily
Joshua T. Vogelstein
jovoofflu edu
Department of Biomedical Engineering Johns Hopkins University
Reviewed on OpenReview: https: // owtpr//veeervvee....m net/ forum? id= dpcRp8ia57
Abstract
Deep discriminative approaches, such as decision forests and deep neural networks, have recently found applications in many important real-world scenarios. However, deploying these learning algorithms in safety-critical applications raises concerns, particularly when comes to ensuring calibration for both in-distribution and out-of-distribution regions. Many popular methods for in-distribution (ID) calibration, such as isotonic and Platt's sigmoidal regression, exhibit adequate ID calibration performance. However, these methods are not calibrated for the entire feature space, leading to overconfidence in the out-of-distributiin (OOD) region. Existing OOD calibration methods generally exhibit poor ID calibration. In this paper, we jointly address the ID and OOD problems. We leveraged the fact that deep models learn to partition feature space into a union of polytopes, that is, flat-sided geometric objects. We introduce geodesic distance to measure the distance between these polytopes and further distinguish samples within the same polytope using Gaussian ker- nel. Our experiments on both labular and vision benchmarks show that the proposed approaches, namely Kernel Density Forest (KDF) and Kernel Density Network (KDN), ob- tain well-calibrated postcriors for both ID and OOD samples, while mostly preserving the classificat tion accuracy and extrapolating beyond the training data to handle OOD inputs appropriately.
1 Introduction
Machine learning methods, specifically deep neural networks and decision forests (such as random forests and gradient boosting trees), show excellent performance many real-world lasks (Xu et al.] 2021a). including drug discovery (Cano et al. 2017 Shi al. 2019), autonomous driving (Yurtsever al. 2020). and clinical surgery [Bihorac al.] 2019h. However, Calibrating confidence across the entire fcature space for these approaches continues to be a significant challenge (Hein et al. 2019), , as most existing algorithms focus either on in-distribution (ID) calibration or out-of-distribution (OOD) detection, rather than treating them as unified problem. Calibrated confidence within the training or in-distribution (ID) region as well as in the out-of-distributiin (OOD) region crucial for safety critical applications like autonomous driving and
"Equal contribution.

---

Published in Transactions on Machine Learning Research (06/2025)
computer-assisted surgery, where any aberrant reading should be detected and taken care of immediately (Hein et al. 2019 Meinke et al. 2021
The methods for detecting out-of-distribuution (OOD) samples in the literature can be broadly categorized into two types: discriminative and generative approaches. A straightforward strategy for OOD detection is to design a function that assigns higher scores to in-distribution (TD) samples and lower scores to OOD samples (Liang et al. 2017). Discriminative approaches typically modify the loss function (Nandy et al. 2020 Wan
ct al. scores (endryykks et al., 2018 Hein et al. 2019) Recently, Hein et al. (011 showed RELU networks produce arbitrarily high confidence as the inference point moves far away from the training data and it prohibits most algorithrms from having an asymptotic performance guarantee. Therefore, calibrating RELU networks for the whole OOD region is not possible without fundamentally changing the network architecture. Others learn generative models for the ID and the OOD samples. The general idea is to perform the likelihood ratio test for particular sample using generative models Ren et al. (2019), or to threshold the ID likelihoods to detoct OOD samples. However, is not obvious how to control the likelihoods far away from the training data for powerful generative models such variational autoencoders (VAE) (Kingma et al., 2019 and adversarial gencrative networks (GAN) [Goodfellow et al. 2020). Morcover, Nalisnick et al. (2018) and Hendrycks et al. (2018) showed that VAEs and GANs can yield overconfident likelihoods far away from the training data. Although these algorithms focus solely on OOD detection for deep networks, other methods, such as those that partition the feature space (e.g. decision forests), can also exhibit overconfidence in OOD regions.
2018 DeVries & Taylor 2018) or extensively train the model on OOD datasets to produce these distinct
The algorithms described above are concerned only with OOD detection and do not address confidence calibration within the ID) region at all. This is addressed by separate group of algorithms (Zadrozny &
Elkan 2001 Caruana 2004 Platt et al., 1999 Guo et al. 2017 2019 Kull et al. 2019) that deals with ID
confidence calibration.
We estimate posteriors that are calibrated in both ID and OOD regions. Our approach subsumes OOD detection and ID calibration by addressing the calibration problem as a continuum from ID to OOD regions (see Appendix Section A for further clarification). We conceptualize decision forests and RELU net works as partitioning the input space into unions of polytopes, where each polytope has a distinct affine transfor- mation. Ncarby polytopes have similar affine functions. We used "Geodesic" kernel to mcasure distances between these polytopes (Madhyastha et al. 2020), allowing us to better capture their relationships. We find the nearest polytope to an inference point and replace the affine function learned over that polytope with a Gaussian kernel, leading to two novel calibration methods: Kernel Density Forest (KDF) and Kernel Density Network (KDN). Our approach eliminates the need for training on OOD examples, enabling fully unsupervised OOD calibration. Throuuh extensive simulations and benchmark experiments, we show that KDF and KDN achieve well-calibrated predictions for OOD samples while maintaining good accuracy and calibration in the ID region, on both tabular and vision benchmarks.
2 Methods
2.1
Setting
Consider a supervised learning problem with independent. and identically distributed training samples {(x,, 1)) such that () ~ Рx,,, where X ~ Px is a X S ED' valued input and Y ~ Py is a -' fI. K} valued class label. Let S be the high density region of the marginal, Px, thus S S X. The posterior probability for class y is given by the Bayes formula:
(x (r)x(1|x) EK PXIY (K)
Vy E V.
(1)
Here Pxjy (xly) is the class conditional density. In the OOD region where x & S, we do not have any evidence to favor a particular class y for an inference point x. Hence we can write:
(x Pxxx (xlys), Vynl,, Vyn. E V.
(2)
2

---

Published in Transactions on Machine Learning Research (06/2025)
We will refer to (x as fx(x) hereafter for brevity. Substituting Equation 2 into Equation for data
point in the out-ol-dlittributio region yields:
Pr|x(1|x)) Pr(y) Pr(v))
(3)
Ek=1 (r(k)
Our goal is to learn confidence score, g RDD 10, 11 g(x) 191 (x(x)) ... 9x(xx] 9xxxx)))).,,xxxx such that,
Oy(x) = (rlx(ulx),
x E (Pr(U). if & S Yyey
(4)
2.2
Main Idea
Deep discriminative networks partition the feature space R$ into union of p afline polytopes Qr such that
Rdd and learn an affine function over each polytope (Hein et al. 2019 Xu et al. 2021b Black
et al. 2022 Priebe et al.) 2020). Mathematically, the unnormalized class-conditional density for the label y estimated by these deep discriminative models at a particular point x can be expressed as:
2 fx(x)= Σ [x,x+b,))166 E.)))
(5)
For example, the case decision tree, a, = O, i.e.; decision trees assume a uniform distribution for the class-conditional densities within a leaf node. Among these polytopes, the ones that lie on the boundary of the training data extend to the whole feature space and hence encompass all over the OOD region. Since the posterior probability for class is determined by the affine activation over each of these polytopes, the algorithms tend to be overconfident when making predictions on the OOD inputs. Moreover, there exist some polytopes that are not populated with training data. These unpopulated polytopes serve to interpolate between the training sample points. If we replace the affine activation function of the polytope with a Gaussian kernel and prune the polytopes unpopulated by the training data, the tail of the kernel will help interpolate between the training sample points while assigning lower likelihood the low density or unpopulated polytope regions of the feature space.
2.3 Proposed Approach
We will call the above discriminative approaches as the 'parent approach' hereafter. Consider the collection of polytope indices P from the parent approach which are populated by the training data. We replace the affine functions over the populated polytopes with Gaussian kernels G(: ür Σ,). For particular inference point x, consider the Gaussian kernel with the minimum distance from the center of the kernel to the corresponding point:
F = argmin ||, - xx]:
(6)
where ||x -zg denotes a distance between xi and x2- As will show later, the lype of distance considered in Equation G highly impacts the performance of the proposed model. Tn short, we modify Equation from the parent RELU-net or decision forest to estimate the class-conditional density:
fx(x) = - Sxy ar {r) - f2).
(7)
Bl TEP
where ny is the total number of training samples with label y and nry is the number training samples from class y that end up in polytope Q,. Note that our proposed model not Gaussian mixture like model; rather, for given inference point, it considers only the single nearest Gaussian. We add small constant to the class conditional density so that the class conditional likelihoods become constant far away from the Gaussian center:
fx(x) = fx) + log(n)
(8)
3

---

Published in Transactions on Machine Learning Research (06/2025)
Note that in Equation log(n) 0 as the total training points, oc. See the derivation of Proposition in Appendix B for further clarification on how the above constant helps achieving our goal in Equation The confidence score Oy (x) for class y given test point is estimated using the Bayes rule as:
(v( (x(x) = Ex (x)
(9)
where (v) is the empirical prior probability of class y estimated from the training data. We obtain the class for a particular inference point by:
argmax By (x).
(10)
vEV
2.4 Geodesic Kernel
A) Random Forest
B) ReLU-Net
X1
x2
Tree 1
Tree 2
1/0
1/0
1/1
1/1
1/1
Leaf nodes
2
3
a = [111', '111', '11'', '111]
Activation mode for x1, a = [2, 3]
Activation mode for x2, a 2 = [3, 3]
a ['001', '011, '111]
K(x1,x2)= 1 2
K(X1,x2)= 2X2X1
Figure 1: Geodesic kernel calculation for two samples in decision forest and ReLU-net. Two samples Xı (marked blue) and X₂ (marked red) are pushed down A) decision forest with two trees. B) RELU-net with three layers. The RELU activation for the two samples at each node is shown with colors blue and red. The activation paths that are identically activated for the two samples above are bolded, along with the common elements in their activation modes.
As evident in Equation 5. samples that fall within the same polytope undergo the same affine transformation by the deep network. Black et al. (2022) identified polytopes as the fundamental building blocks of deep networks, introducing the concept of "spline code," which represents the binarized and flattened activation pattern of the network for an input sample. These spline codes uniquely identify the polytope in which sample falls and, demonstrated by Black et al. (2022), effectively cluster identical samples without even considering the absolute magnitude of activations. They further proposed using the Hamming distance between spline codes to identify ncarby polytopes undergoing nearly similar transformations to a certain polytope in the network. Moreover, various other kernels based on activation pal in deep networks have been explored in the literature (Balestriero et al.) 2018 Wilson et al., 2016). However, to our knowledge, most of these kernels either do not construct metric space or at most construct semi-metric space and hence, they cannot be used valid metrics (Burago 2001). In the following, we propose a generalized kernel for both the decision forest and RELU-net, and, as will show in Proposition [
it constructs metric
4

---

Published in Transactions on Machine Learning Research (06/2025)
space. This enables to use our kernel to measure the distance between samples in the representation space learned by the network.
Our proposed kernels are based on existing random partition kernels (Davies & Ghahramani 2014). Hence, all the properties of such partition kernels hold for our proposed kernel. The only difference is that we do not use random partitions of the feature space, instead we use the partitions learned by decision forests or ReLU-net after being trained on the training data. We identify these partitions using the leaf identities in decision forest the activation patterns in RELU-net. Note that the geometry of these partitions polytopes encodes the low-dimensional structure learned by the model Balestriero et al 2018).
2.4.1 Forest Kernel
A decision forest with B decision trees partitions the feature space into set of disjoint polytopes, {Qr Pm Decision forests map any sample æ; E Qr to B dimensional vector where each element a;(() of the vector represents the leaf identities the sample falls in each tree. We call the above vector a, the "activation modc".
If the Iwo aclivalion modes for two diffeerent training samples are identical, they belong to the same polylope. In other words, if a, = aj, then Q = 2ss This statement holds because the above samples will end up in the same partition in all decision trees.
For any two points x, Qr and x, E Qs: we define the kernel K( as:
K( xz)) = 1 t=1
(ta(t))) = (t)
(11)
Note that S (a my) S 1. In short, K(a,,, is the fraction of total trees where the two samples follow the same path from the root to leaf node.
2.4.2 Network Kernel
A fully connected L layer RELU-net partitions the feature space into set of disjoint polytopes, We have the set of all nodes in a particular layer I denoted by Mi = {nj) = We can randomly pick a node f E NG at each layer 4, and construct a sequence of nodes starting at the input layer and ending at the output layer, which we call an activation path: m = {nē E Ni) Note that there are B = II possible activation paths for a sample in the RELU-net. We index each path by a unique identifier number E (1..... B}.
While pushing a training sample x; Qr through the network, we define the activation from a RELU unit any node as '1' when it has positive output and '0' otherwise. Therefore, the activation indicates on which side of the affine function at each node the sample falls. The activation of all nodes on activation path m. for a particular sample creates a string of binary numbers a;(t) E (0, 10.1 Therefore, a RELU-nct maps any sample x; E Qr to a B dimensional vector a; where each element (;(() of the vector represents a string of binary numbers of length L. We call the above vector a; the "activation mode". Note that similar to the leaf identities in decision forests, each element (t) of the above vector represents a partition of the feature space, that is, only the samples belonging to a particular region of the feature space can have (t). The intersection of all such overlapping partitions in an activation mode gives the polytope Q, in which the sample falls.
If the two activation modes for two different training samples are identical, they belong to the same polytope. In other words, if a; = aj, then Q = Q.- This statement holds because the above samples will lie on the same side of the affine function at each node in different layers of the network. Now, we define the kernel 2(,,,,) as:
1 1 1=1
K(a,, 2z,) =
((a((t)) = (t))
(12)
Note that S K(x,, 2,) < In short, K(x,, is the fraction of total activation paths that are activated identically for two samples in two different polytopes and A generalized pseudocode outlining the above
5

---

Published in Transactions on Machine Learning Research (06/2025)
two kernel calculation is provided in Algorithm caleulation.
See Figure
1
for visual interpretation of the kernel
Algorithm 1 Computing Geodesic Kernel
Input:
(1) Xi, X; E Blxd (2) 0
D two input samples > parameters for parent decision forest or RELU-net
Output: x3) E [0,1] function COMPUTEKERNEL(S;; xy,
> compute similarity between and j-th samples.
push samples down the deep network to get the activation
modes.
3: aj <- PuSHDOVwNNTTORRKXX,,0
4: . <- COUNTMATCHES((, a,)
> count the number of similar activation modes or leaf identities b is the dimension of activation modes.
5: K( 2;)
6: return K(x,,
end function
2.5 Network Kernel with Deep Network
In this section, empirically assess the valid- ity of the kernel defined in Equation 12 net- work depth increases. We conduct experiments on a two-dimensional Gaussian XOR simulation (de-
Before Training
After Training
784 dimensional data 2 dimensiona data
Kernel Similaity
0.5
scribed in Appendix D and the 784-dimensional Fashion-MNIST dataset (from OpenMI-CC118 (Bis- chl al. 2017)) using fully-connected networks of varying depth and width. Specifically, measure the average kernel similarity between sample pairs from the same and different class labels, both be- fore training (with random initialization) and after training. As shown in Figure 2 kernel similarity decrcases monotonically with depth under random initialization, indicating increasingly disjoint acti- vation paths. In contrast, after training, similar- ity between same-class samples remains relatively high, suggesting that learning encourages alignment in activation paths. These results demonstrate that, while the kernel becomes sparse in untrained deep networks with random weights, retains discrim- inative power once the network is trained,
0.1
0.0
Kernel Similaity altyy
0.1
0.0
10
20
30
10
20
30
Depth
Depth
Same Class:
(Node #
100
500
1000
Different Class:
(Node
100
500
1000
2.6 Geodesic Distance
Figure 2: Average kernel values are higher for similar-class pairs than for dissimilar ones. Val- ues decrease with depth in networks with random par- tition (left) but remain stable when trained on data (right).
Consider Pn = (Q1, }pp as a partition of Rd given by a dccision forest or a RELU-nct after be- ing trained on training samples. We measure dis- tance between two points x; Qr,,, Xj E Q, using the kernel introduced in Equation I1 and Equation 12 and call it 'Goodesic' distance (Schölkopf 2000):
2))) -K(,,, mz) + (KC()); K(ar,,,))) 1 K(.;;
(13)
Proposition 1. (Pa, d) is metric space.

---

Published in Transactions on Machine Learning Research (06/2025)
Proof. See Appendix B.1 for the proof.
The above proposition enables to Geodesic distance metric in our algorithm. We use Geodesic distance in Equation B to find the nearest polytope to the inference point. As Geodesic distance cannot distinguish between points within the same polytope, it has a resolution similar to the size of the polytope. We assume the polytope is small enough so that the manifold within a polytope can be considered Euclidean and hence for discriminating belween two points within the same polytope, we a Gaussian kernel within the polytope (described below). Moreover, Gaussian kernels decay exponentially from their center which improves OOD calibration. In Section 3 we will empirically show that using Geodesic distance scales better with higher dimensions compared to Euclidean distance.
2.7 Gaussian Kernel Parameter Estimation
We fit Gaussian kernel parameters to the samples that end up in the r-th polytope. We set the kernel center along the d-th dimension:
[(x E Q+),
(14)
where 24 is the value of x; along the d-th dimension. We set the kernel variance along the d-th dimension:
(0:)2
n4 12 1(x Qa) - - A},
(15)
where À is a small constant that prevents of from being 0. We constrain our estimated Gaussian kernels to have diagonal covariance.
2.8 Sample Size Ratio Estimation
For a high dimensional dataset with low training sample size, the polytopes are sparsely populated with training samples. For improving the estimate of the ratio Rez in Equation 7 we incorporate the samples from other polytopes Q, based on the similarity Wes between Q and Q, as:
îry ΣNEP Ei E Q.) y)
(16)
ny EnEP E.EP Ei= Wes (x; E Q.) = y)
As -> 2o, the estimated weights Wrs should satisly the condition:
0,
if Qf * Q.
Wre
(17)
1, if Qf = Q.
Note that if we satisfy the conditions in Equation 17 then we have K(r,, s) so that the conditions in Equation is satisfied:
Bry
20. We exponentiate
nly
Wrs K( x. ) logn
(18)
We choose using grid search on a hold-out dataset. Therefore, we modify Equation as:
fx(x)
Σ n fr. 1r)
(19)
ìy rEP
where Ax = argmin, || x||- Now we use (x( estimated using 19 in Equation respectively.
9 and
10
Given n training samples {Xx, y1) ) we define the distance of an inference point x from the training points dx min.: ||x xil1, where || denotes Fuclidean distance.
7

---

Published in Transactions on Machine Learning Research (06/2025)
Proposition 2 (Asymptotic OOD Convergence). Given non-zero and bounded bandwidth of the Gaussions, then we have:
lim (x)x = Py))) dx +00
Proof. See Appendix B.2 for the proof.
Note that the above proposition guarantees asymptotic performance of the proposed approach. In contrast, typical existing approaches lack any such guarantees. Below we conduct empirical experiments including simulations and benchmark datasets to gain insights about the non-asymptotic performance of the proposed approach.
Algorithm 2 Fit KDX model.
Input:
(1)
Parent learner (random forest or deep network model)
(2 Dn, = Dx,=(Xy)) E Baxd x K]"
>> Training data
Output: G function KIX.FIT( y) for i = 1, ................................ do
a KDX model
Iterate over the dataset to calculate the weights
for j = ..... n do
kij f- COMPUTEKEENELLXXX.....
5: end for
6: end for
7:
( 1 + CETPOLXT)PEs(k)
b Identify the polytopes populated by the training data by
merging two samples with ky-
10:
r= d do
Iterate each polytope
11: G.Prig.E,, G.ury + ESTIMATEPARRAMETERSSS y, flips } 18
Estimate model parameters
12: end for
13: return g
14: end function
2.9
Performance Measure
For the simulation setups, we evaluate performance with classification error, Hellinger distance Kailath
1967 Rao 1995) wetween the true and estimated class conditional posteriors, and mean max confidence (MMC) (Hein al., 2019). We use expected calibration error (ECE) evaluate in-distribution calibration for the OpenML-CCI8 data suite, with R 20 bins for all datasets Guo et al. (2017). Given OOD samples {x, 1 we define OOD calibration error (OCE) to measure OOD performance for the benchmark datasets
as:
1 Σ max (P)xx()lx))) - max (yy (v)) OCE =
(20)
NEy
yEY
Note that one needs to know the true prior calculating OCE. For tabular data experiments, training points sampled based on empirical class priors in the overall dataset. In vision experiments, assume equal class priors and sample training points accordingly.
3 Empirical Results
For empirical experiments, we first demonstrate that our methods satisfy the theoretical guarantees on simulated datasets where know the ground truth, and then apply our approach benchmark datasets
8

---

Published in Transactions on Machine Learning Research (06/2025)
such OpenML-CC18 (Bicchl et al. 2017 and different vision datasets. The details of the simulation data
sets and hyperparameters used for all experiments are provided in Appendix D For tabular and vision data
sets, we have used ID calibration approaches, such as ISOTONIC Zadrozzy & Elkan (2001); Caruana (2004)
and SIGMDID regression Platt et al. [199)). as basclines. Additionally, for the vision benchmark dataset, we
provide results with OOD calibration approaches such as: ACET Hein el al. (2019), ODIN Liang et al. (2017)
(outlier exposure) Hendrycks ct al. (2018). For each approach, 70% of the training data was used the model and the rest of the data was used to calibrate the model.
3.1 Empirical Study on Tabular Data
Simulation
KDF and RF
KDN and DN
A.
Classification Error Hellinger Distance
Conf.
Classification Error Hellinger Distance
Mean Max Conf.
0.4
0.3
1.0
0.4
0.5
1.0
Spiral
0.1
0.1
0.1
0.5
0.1
0.5
1.0
1.0
Circle
0.2
0.2
0.2
0.1
10 10 10 Sample size
10 10 Sample Size
2 4 8446660 10
10a
10 10
10a 10a
10
4 8
Distance
Sample
Distance
1.0
0.3
0.3
1.0
Trunk
0
0.2
0.1
0.6
0.6
0
2
10a
10.
10a
10"
10
10
10"
10a
104
10
10
10*
10" Dimensions
10
10a
10
10 10c
Dimensions
Dimensions
Dimensions
Dimensions
Dimensions
KDF GDF-Geddesic
KDF-Euclidean
KDN Geodesic
KDN-Euclidean
Figure KDF and KDN improve both in-distribution and out-of-distriibutio calibration of the parent model while preserving classification accuracy. We evaluate performance using simulation datasets, reporting classification error, Hellinger distance from true posteriors, and mean max confidence (MMC). Results presented for: A. Two-dimensional simulations. In the fourth and seventh columns, MMC is measured as function of increasing distance from the origin and B. High-dimensional (Trunk) simulations visualized along the first two dimensions. OOD points are randomly sampled from a sphere of radius 20, and MMC is measured across increasing dimensions. For Hellinger distance, lower values indicate a closer match to the true training distribution, while in OOD regions, a value near 0.5 signifies better OOD calibration. The median performance of repetitions is shown as dark curve with shaded region as error bars.
3.1.1 Simulation Study
For simulation study on tabular data, we use 6 simulation datasets. Three of the simulations are visualized in Figure 3/ and see Appendix D for additional simulations and details. We sample 10,000 training samples with half of the samples from cach class. The samples are sampled within the region [-1, 1] [|-1, rendering the emply anmular region between [-1, 1] [-1,1] and [-2, 2) [-2, as the low density or OOD region. the Trunk simulation (Figure 3 B), each of two classes is sampled from a Gaussian distribution, with higher dimensions having incrcasingly less discriminative information (Trunk 1979.. We use both Euclidean and Geodesic distances to detect the nearest polytope center. To measure in-distribution performance, report classification error and the Hellinger distance between the estimated and true distributions. To measure OOD performance, we normalize the training data to the maximum of their norm lg. For inference, we sample 1000 inference points uniformly from a circle and compute the mean max posterior for increasing

---

Published in Transactions on Machine Learning Research (06/2025)
Classification
ID Calibration
OOD Calibration
0.1
1,0
Improvement over parent Improvement over parent
0.0
Isotonic RF
0.0
Sigmold
Parent RF
0.2
1,0
0.05
0.8
KDN
Isotonic
0.00
Sigmoid DN
Parent
0,0
-0.05
-0.
10
Number of Training Samples (log)
Distance
Figure 4: Performance summary of KDF and KDN on OpenML-CC18 data suite. The dark curve in the middle shows the median of performance on 45 datasets with the shaded region as error bar.
distance from the origin. Therefore, for a distance of up we have in-distribution samples and distances further than OOD. For the Trunk simulation, we randomly sampled OQD points from sphere of fixed radius 20 and measured mean max confidence for increasing dimensions.
KDF and KDN achieve classification accuracy comparable to their parent algorithms (Figure 3 However, they provide a more accurate estimation of the in-distribution (ID) region. The use of geodesic distance further enhances performance relative to Euclidean distance. As data points move farther from the training set, the mean maximum confidence for KDF and KDN approaches the maximum of the class priors, reaching 0.5. Furthermore, KDF-Geodesic and KDN-Geodesic exhibit better scalability in higher dimensional spaces compared to their Euclidean counterparts (Figure B).
Hereafter, only use Geodesic distance to find the nearest polytope in our algorithuns.
3.1.2 OpenML-CC18 Data Study
We use OpenML-CC11 data suite for tabular benchmark dataset study, We exclude any dataset which contains categorical features or NaN values and conduct experiments on 45 datasets with varying dimensions and sample sizes. We repeat. the experiment for each dataset 10 times. For the OOD experiments, we follow a similar setup as that of the simulation data. We normalize the training data by their maximum & norm and sample 1000 testing samples uniformly from hyperspheres where each hypersphere has increasing radius starting from to 5. For each dataset, we measure improvement with respect to the parent algorithm:
Ep E Ep
(21)
where Ep represents either classification error, FCE or OCE for the parent algorithm and EMs represents the performance of the approach consideration. Positive improvement implies that the corresponding approach performs better than the parent approach. We report the median of improvement on different datasets along with the interquartile ranges (Figure 4 exlended results for each datasel in the appendix). On average KDF and KDN have similar or better classification accuracy compared to their respective parent algorithms, whereas ISOTONIC and SIGMDID regression have lower classification accuracy most of the cases. KDF and KDN have similar ID calibration performance to the other baseline approaches, all better than the parent
We also excluded the dataset with dataset id 23517 as we could not achieve better than chance accuracy using and that dataset.
10

---

Published in Transactions on Machine Learning Research (06/2025)
A. Circle (ID)
В. Rectange (ID)
C. Ellipse (OOD)
D.
E.
F.
1.0
Mean Max Conf.
0.8
0.5
127
255
0.0
0.5
1.0
0
10
40
Pixel Value
Axis Length (a)
KDN
CNN
Figure KDN filters out inference points with different kinds shifts from the training data. Simulated images: (A) circle with radius 10, (B) rectangle with sides (20, 50) and (C) out-of-distribtiion test points, cllipse with minor and major axis (10, 30). Mean
confidence of KDN are plotted for semantic shift of the inference points created by (D) changing the color intensity, (E) taking convex combination of circle and rectangle, ie., pixel-wise convex combination of the images in A and B, (F) changing one of the axes of the ellipse.
algorithm for all sample sizes. Most interestingly, KDN and KDF improve OOD calibration of their respective parent algorithms by a huge margin while the baseline approaches completely fail in OOD calibration.
3.2 Empirical Study on Vision Data
In vision data, each image pixel contains local information about the neighboring pixels. To extract the local information, we use convolutional or vision transformer encoders at the front-end. Consider a set of images Z where each image has width W, height and each pixel in an image identified by (M, w1, w2), where M is a vector representing the RGB pixel values, wi is the horizontal location, W2 the vertical location. More preciscly, we have a front-end encoder, he I I and typically, d << W H. After the encoder, there are few fully connected dense layers for discriminating among the K class labels, hy Rdd + R Note that the d-dimensional embedding outputs from the encoder partitioned into polytopes by the dense layers (see Equation (5)) and we fit KDN on the embedding outputs.
3.2.1 Simulation Study
For the simulation study, we use a CNN with one convolutional layer (kernel size 3) followed by two fully connected layers with 10 and nodes in each. Each experiment repeated 10 times get the error bars. Consider two random variables C₁, C2 U 100). We sample 2000 circles X, and 2000 rectangles respectively. Each image has width 100 and height 100. The CNN is trained to classify an image into circle or rectangle.
X, grayscale training images using Equation 22 and 23
127,
if (w) - C12 + (w2 - C2) < 100 otherwise.
M
(22)
0,
11

---

Published in Transactions on Machine Learning Research (06/2025)
Table I: KDN achieves good calibration at both ID and OOD regions whereas other approaches excel either in the ID or the OOD region. Notably, KDN has reduced confidence on wrongly classified ID points. 'f' and ' indicate whether higher and lower values are better, respectively. OCE- OOD calibration error, MMC* Mcan Max Confidence on wrongly classified ID points.
(a) In-Distribution (ID) Performance
ID dataset
Statistics
Puren:
KDW
TECTOKIC
S10M010
ACET
ODIN
06
Focal
texth:Dul Focal
Accuracy [X]]
98.06
0.00
97.45 :: 0.00
08.16 E 0.00
98.10 :: 0.00
23 0.00
97.97 0.00
97.04 0.00
94.79 : 0.00
05.23 0.00
CIFAR 10
ECE J
0.01 1 0.00
0.00 1 0.00
0.00 0.00
0.00 1: 0.00
0.01 = 0.00
0.02
0.00
0.01 1: 0.00
0.07 1: 0.00
0.01 1: 0.00
MMC
0.76
0.74 0.02
0.90 0.01
0.86 0.00
099
0.01
0.56
0.60
Accuracy (9C)
86.72 +0.00
85.46 0.00
85.33 + 0.00
86.61 + 0.00
85.07 + 0.01
86336 +.00000000
86.00 + 0.00
79.37 + 0.00
80.81 + 0.00
CIFAR-100
ECE J
0.00
0.00
0.04 + 0.01
0.02 + 0.00
0.11 = 0.01
0.04
0.00
0.07 = 0.00
0.02 0.00
MMC! 1
D446+ 0.02
0.41 0.02
0.65 + 0.03
0.45 0.002
0.84 0.03
0.77- 0.01
0.52 0.00
0.46 0.00
0.60 0.00
(Xc
98.52 1 0.01
92.84 0.00
05.64 0.01
95,05 0.01
98.22 1 0.01
93.32 0.001
93.76 1 0.01
60.39 1 0.00
61.02 1 0.00
ECE
0.00
0.01 4: 0.00
0.06
0.05 = 0.01
0.06 0.00
0.07 0.01
MMC' 4
0.82 0.01
0.63 1: 0.00
0.67 11 0.01
0.78 0.00
0.88 - 0.01
0.97 0.01
0.70 1 0.04
0.33 1: 0.00
0.41 1 0.000
(b) Out-of-Distributin (OOD) Performance (OCE 4)
ID dataset
OOD dataset
Parent
KDk
I90TowIc
S1GHO10
ACET
ODIN
DE
Focal
Dual Focal
CIFAR- 100
0.47 + 0.01
0.12 0.01
0:47 0.01
0.69 0.01
0.57 0.01
0.79 + 0.00
0:29 = 0.01
0.47 0.00
0.51 + 0.00
CIFAR-10
SVIIN
0.87 +: 0:00
0,01 0.00
0.85 + 0.00
0.69
+ 0.01
0.90 0.00
0.87 4: 0.00
0.04 0.01
0.40 0.04
0.54 +: 0.03
Nošse
0.28 + 0.08
0.03 + 0.02
0.30 + 0:04
0.56 006:0033
0.01 + 0.00
0.53 + 0.00
0.07 + 0.02
0.52 0.11
0.79 + 0.00
CIFAR-10
0.23
0.12 0.01
0.50 0.04
0.62 0.04
0.54 0.01
0.02
0.31
0.42
CIFAR- 100
SVHN
0.20 0.01
0.12 0.01
0.47 0.06
0,20 0.02
0.58 0.05
0.57 1 0.02
0.20 = 0.012
0.21 = 0.03
0,30 i 0.03
Noise
0.05 1 0.02
0.03 = 0.00
0.38 0.04
0.07 & 0.06
0.08 0.01
0.18 1 0.06
0.07 +: 0:04
0.04 a: 0.00
0.03 1: 0.01
CIFAR-10
0.00
0.14 0:01
0.02
0.53
0.67 0.00
1 0.02
0.14
0.31 0.00
0.40 0.00
SVIN
CIFAR 100
0.40 1: 0.03
0.14 =: 0:01
0.37 1 0.03
0.54
0.66 1: 0.03
0.80 1: 0.02
0.16 = 0.01
0.40 =: 0.00
0.48 1: 0.00
Noise
0.36 : 0.08
0.12 = 0.01
0.32 0.06
0.48 0.06
0.03 +: 0.01
0.67 + 0.85
0.15 = 0.03
0.21 0.03
0.32 = 0.04
127,
if C1 - 10 S w < <C + 10
M = and C2 - 25 S w2 < C2 + 25
(23)
0, otherwise.
We perform three experiments while inducing different types of shifts in the inference points (Figure In the first experiment, we sample inference points using Equation 22 and 23 but sweep over different intensities for the object (Figure 5 D). Therefore, the inference point is maximally ID) for intensity at 127 and becomes more OOD as we move away from 127. In the second experiment, we kept the intensity at 127 while taking convex combination of circle and a rectangle. Let images of circles and rectangles be denoted by X, and Xy. We derive an interference point as Xmnf
Xinf = <X, + +1 c)
(24)
Therefore, Xinf is maximally distant from the training points for = 0.5 and closest to the ID points at E e 11 (Figure E).
In the third experiment, we sampled ellipse images using Equation 25 sweeping from 0.01 to 40 (Figure F). The inference point becomes ID at a 10 and remains OOD otherwise. As shown in Figure 5 (D, E, F), in all the experiments KDN becomes less confident for the OOD points while the parent CNN remains overconfident throughout different shifts of the test points.
127, if (w) (w)-C1) M =
(ang C2)
+
<
a2
100
(25)
0. otherwisc.
3.2.2 Vision Benchmark Datasets Study
In this study, we use VIT_B16 (provided in keras-vit package) vision transformer encoders (Dooovisklyy et al. 2020) pretrained on ImageNet (Deng et al.] (2009) dataset and finc-tuned on the ID datascts. As described in Section 3.2, we use a VIT_B16 encoder and use Equation 12 on the last two fully connected layers caleulate the geodesic kernel. We use the same encoder for all the baseline algorithms and fine-tune it
12

---

Published in Transactions on Machine Learning Research (06/2025)
without freezing any weight. We experiment with popular benchmark datasets- CIFAR1O. CIFAR-100 and SVHN. For each experiment, we train on one of the three datasets and test KDN for OOD performance on the other two datasets. We also consider a simple OOD dataset, noise, where we sample noise OOD samples of size 32 x 32 x 3 according to a Uniform distribution with intensitics within range [o, 1]. Although noise samples is easier to detect as OOD point, we observe that many of the baseline approaches perform poorly even on the noise samples. Each experiment is repeated for different sceds to get an error bar. Table I shows that pretrained vision transformers are already well-calibrated for ID, and the OOD approaches (ACET, ODIN, DE) degrade ID calibration of the parent model. On the contrary, ID calibration approaches (ISOTONIC, SIGMOId) perform poorly compared to that of KDN in the OOD region. KDN achieves a compromise between ID and OOD performance while having reduced confidence on wrongly classified ID samples (see MMC* in See Appendix Table 2 for traditionally used statistics for OOD detection.
Table 1..
4 Discussion
4.1 Related Works and Our Contributions
Recently, Liu et al. (2023) showed that a deep network aware of distance (distance from training points) is able to capture the uncertainty associated with its prediction better than traditional approaches. However, the concept of distance awareness has previously been used by number of approaches in the literature to detect OOD points. These approaches attempt to learn a generative model in the projected latent space and control the likelihoods far away from the training data. For example, Ren et al. (2019) used the likelihood ratio test to detect OOD samples. Wan et al. (2018) modified the training loss so that the downstream projected features follow a Gaussian distribution. In contrast to the above approaches, Liu et al. (2023) proposed an in-training approach which learns a map from the input space to latent space and trains a Gaussian process model on top of that. Distance awareness from training points improves OOD detection and the Gaussian process improves ID calibration. However, OOD detection works long there are two distinguishable score sets for ID and OOD points, whereas calibration aims at estimating the true predictive uncertainty of these points (sce Appendix A for details). In our work, we propose a simple post- hoc calibration approach using geodesic distance and achieve the aforementioned distance awareness from the training data by replacing the affine function over the nearest polytope with a Gaussian kernel. The proposed approach works on both random forests and deep networks. We validated our proposed approach on both tabular and vision datatsets, whereas most of the existing methods are tailor-made for vision problems.
4.2 Conclusions
In this paper, we have demonstrated simple intuition that transforms traditional deep discriminative models into a type of binning and kerneling approach. The bin boundaries are determined by the internal structure learned by the parent approach, and the Geodesic distance encodes the low-dimensional structure learned by the model. The vision experiments in this paper show how the model can be applied to deeper networks by employing a front-end encoder to extract local image features (as explained in the first paragraph of Section 3.2 Instead of applying KDN to the entire VrT, we applied it solely to the final fully connected layers with ReLU activations. This reduces the effective depth of the model, ensuring that KDN can be applied even to encoders that do not rely on ReLU activations. Moreover, the geodesic distance introduced in this paper may have a wider impact on understanding the internal structure of the deep discriminative models, which may be of interest for future work. However, key limitation of our approach is the need to store parameters for each populated polytope, which may hinder scalability to very large datasets. One potential solution is to merge nearby polytopes into a single representative polytope. Our code, including the package and the approach proposed in this manuscript, is available from /tttps://githin www./neuooaaaa//eee.e.//. Appendix Section answers a number of frequently asked questions about this work.
Acknowledgements
The authors thank the support of the NSF-Simons Research Collaborations on the Mathematical and Sci- entific Foundations of Deep Learning (NSF grant 2031985) and THEORINET. This work was graciously
13

---

Published in Transactions on Machine Learning Research (06/2025)
supported by the Defense Advanced Research Projects Agency (DARPA) Lifelong Learning Machines pro- gram through contracts FA8650-18-2-7832 and HR0011-11-2-0022, Research was partially supported by funding from Microsoft Research and the Kavli Neuroscience Discovery Institute. We also thank the helpful discussion with Will LeVine, Tyler M. Tomita, Ali Gcisa, Tiffany Chu and Jacob Desman.
References
Randall Balestriero et al. A spline theory of deep learning. In International Conference Machine Learning. pp. 374-383. PMIR, 2018.
Azra Bihorac, Tezcan Arrazgat-Baslant Ashkan Ebadi, Amir Motaci, Mohcine Madkour, Panagote Pardalos, Gloria Lipori, William R. Hogan, Philip Efron, Frederick Moore, et al. Mysurgeryrisk: de- velopment and validation of a machine-learning risk algorithm for major complications and death after surgery. Annals of surgery, 269(4):652 2222, 2019.
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G Mantovani, Jan N van Riju, and Joaquin Vanschoren. Openml benchmarking suites. arXio preprint arXiv: arXiv:1708.03770 2017.
Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, et al. Interpreting neural networks through the polytope lens. arXiv preprint ar arXio:2211.12311 2022.
Vadim Borisov, Tobias Leemann, Kathrin Scßler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and labular data: A survey. IEEE transactions on neural nehworks and learning systems, 2022.
D Burago. A course metric geometry. American Mathematical Society, 2001.
Gaspar Cano, Jose Carcia- Alberto Garcia-Garcia, Horacio Perez-Sanchez, Jón Atli Benedikts- son, Anil Thapa, and Alastair Barr. Automatic selection of molecular descriptors using random forest: Application to drug discovery. Expert Systems with Applications, 72:151 159, 2017.
R Caruana. Predicting good probabilities with supervised learning. In Proceedings of NIPS 2004 Workshop on Calibration and Probabilistic Prediction in Supervised Learning, 2004.
Alex Davies and Zoubin Ghahramani. The random forest kernel and other kernels for big data from random partitions. arXiv preprint 1xXi:11402.4208 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Foi-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Conference on Compuler Vision and Pallern Recognition, pp. 248 255, 2009. doi: 10.1109/CV/B wwt.CC......222.....00 52094420
Terrance DeVries and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. aixiv preprint Xrx:::1802. 04865, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Stanislav Fort, Jie Ren, and Balaji Takshminarayanan. Exploring the limits of out-of-distributtio detection. Advances in Newral Information Processing Systems, 34:7068 7081, 2021.
Marshall Fraser. The grazing goat in n dimensions. The Two- Year College Mathematics Journal, 15(2): 126-134, 1984.
Josh Gardner, Zoran Popovic, and Ludwig Schmidt. Benchmarking distribution shift in tabular data with tableshift. Advances in Newral Information Processing Systems, 36, 2024.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communicaatinss of the ACM, 63(11): 139 144, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. Tn International conference machine learning, pp. 1321-1330. PMLR, 2017.
Richard Guo, Ronak Mehta, Jesus Arroyo, Hayden Helm, Cencheng Shen, and Joshua T Vogelstein. Esti- mating information- quantities with uncertainty forests. arXiv, pp. arXiv-1907, 2019.
Matthias Hcin, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 41-50, 2019.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint Xr::::812.000 2,0006000 2018.
Thomas Kailath. The divergence and bhattacharyya distance measures in signal selection. IEEE transactions on communication technology, 15(1):52-40) 1967.
Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trendoo in Machine Learning, 12(4):307 392, 2019.
Meelis Kull, Miquel Perello Nicto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Be- yond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems, 32, 2019.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv: 1766 706602690, 2017.
Jeremiah Zhe Liu, Shreyas Padhy, Jie Ren, Zi Lin, Yeming Wen, Ghasson Jerfel, Zachary Nado, Jasper Snoek, Dustin Tran, and Balaji Lakshminarayanan. A simple approach to improve single-model deep uncertainty via distanco-awareness. Journal of Machine Learning Reseorch, 24(42):1-63, 2023.
Meghana Madhyastha, Gongkai Li, Veronika Strnadová-Nceloy, James Browne, Joshua T Vogelstein, Randal Burns, and Carey E. Priebe. Geodesic forests. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 513 523, 2020.
Alexander Meinke, Julian Bitterwolf, and Matthias Hein. Provably robust detection of out-of-distributio data (almost) for free. arXie preprint ar arXiv:2106. 0et222066042644 04260, 2021.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint arXio:1810.00 00.09996...... 2018.
Jay Nandy, Wynne Hsu, and Mong Li Lec. Towards maximizing the representation gap between in-domain & oul-of-disttibutiin examples. Advances in Neural Information Processing Systems, 33:9239 9250, 2020.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likclihood methods. Advances in large margin classífiers, 1013):61-74 1999.
Carey E Priebe, Joshua T Vogelstein, Florian Engert, and Christopher M White. Modern Machine Learning: Partition & Vote. bioRaiv, pp. 2020.04.29.068.400 September 2020. doi: 02.1001/2220.0422.00666 URL 04. 26..068400...a.. abstraact
https::/www.bioreet
C Radhakrishna Rao. A review of canonical coordinates and an alternative to correspondence analysis using hellinger distance. Qūestūió: quaderns d'estadistica investigació operativa, 1995.
Jie Ren, Peter Liu, Emily Fertig, Jasper Snock, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan Likelihood ratios for dus-of-distribution detection. Advances in neural information processing systems, 32, 2019.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Bernhard Schölkopf. The kernel trick for distances. Advances in neural information processing systems, 13, 2000.
Han Shi, Simin Liu, Junqi Chen, Xuan Li, Qin Ma, and Bin Yu. Predicting drug-target interactions using lasso with random forest based evolutionary information and chemical structure. Genomics, 111(6): 1839-1852, 2019.
Fahim Tajwar, Ananya Kumar, Sang Michael Xie, and Percy Liang. No true state-of-the-art? ood detection methods are inconsistent across datascts. arXiv preprint arXiv:2109.0555 2021.
Gerard V Trunk. A problem of dimensionality: simple example. IFEE Trensactions on pottern analysis and machine intelligence, (3):306 307, 1979.
Dennis Ulmer, Lotta Meijerink, and Giovanni Cinà. Trust issues: Uncertainty estimation does not enable reliable ood detection on modical tabular data. In Machine Learning for Health, pp. 341-354. PMLR, 2020.
Weitao Wan, Yuanyi Zhong, Tianpeng Li, and Jiansheng Chen. Rethinking feature distribution for loss functions image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9117 9126, 2018.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Haoyin Xu, Kaleab A. Kinfu, Will LeVine, Sambit Panda, Jayanta Dey, Michael Ainsworth, Yu-Chung Peng, Madi Kusmanov, Florian Engert, Christopher M. White, Joshua T. Vogelstein, and Carey E. Pricbe. When are Deep Networks really belter than Decision Forests at small sample sizes, and how? arXio preprinl arXiv:2108. 13637, 2021a.
Haoyin Xu, Kaleab A Kinfu, Will LeVine, Sambit Panda, Jayanta Dey, Michael Ainsworth, Yu-Chung Peng, Madi Kusmanov, Florian Engert, Christopher M White, et al. When are deep networks really better than docision forests at small sample sizes, and how? arXiv preprint arXiv:2108. 13637 2021b.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. survey of autonomous driving: Common practices and emerging technologies. IEEE access, 8:58443 58469, 2020.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, pp. 609-616, 2001.
A Frequently Asked Questions
We received questions while presenting the proposed approach in different venues, and as wish to reserve the main text for the core contents, we have compiled the explanation to these questions here.
What is the relation between OOD detection and OOD confidence calibration? Our work addresses both in- and out-of-distribution (ID and OOD) calibration. While traditional ID calibration methods like ISOTONIC and SIGMOID regression focus on achieving calibrated inference within the ID region, they do not address OOD calibration. Conversely, there is another group of literature which primarily concerned about detecting OOD points. These approaches such ACET, OE, and ODIN mainly focus on OOD detection rather than OOD calibration. Calibration is harder than detection, akin to how regression is harder than classification. To see that calibration is harder than detection, consider the fact that well- calibrated model can perform detection, but model capable of detecting OOD points may not be calibrated.
OOD detection works as long as there are two distinguishable score sets for ID and OOD points, whereas calibration aims at estimating the true predictive uncertainty of these points. To our knowledge, only Meinke ct al. (2021) explicitly addressed OOD calibration (Section 3 Theorem in their paper), but they do not consider ID calibration. Our work treats calibration problems as continuum between the ID and OOD regions rather than addressing them separately.
16

---

Published in Transactions on Machine Learning Research (06/2025)
polytope number
10000
training time (seo)
50
testing time (sec)
15
populated
5000
100
10"
10"
10
10"
10"
10"
10"
10"
10"
number of parameters (log)
16 dimenisonal
118 dimenisonal
784 dimenisonal
Figure 6: Scaling plots for KDN on datasets with different feature dimensions. The training sample size fixed at 10,000 and the number of nodes in each layer is increased in a four layer fully connected network. Total number of polytopes populated by the training data saturates to training size (left) while training (midddee) well as testing time (right) scales linearly with node number. With peerpparaettrrized networks, which is the norm, they typically exhibit this property that the number of populated polytopes is equal to the sample size.
Isn't this just another out-of-distributiio detection approach? One may argue that the experi- mental settings essentially reduce the task to a binary one: whether output normal confidence for an ID sample or just a prior for an OOD sample. A binary 0OD detector can also immediately output a confidence based on the prior if sample is identified as OOD. However, without ID calibration, there is no reason to expect that confidence to be calibrated within the TD) region. In the experimental setting, as one moves further away from the ID setting, an OOD calibrated classifier will output the prior. However, close to the ID setting, the OOD calibrated classifier will output a probability that the sample is in any given class. This is in contrast to an OOD detector, which, regardless of how close or far the data are from the ID data, if is OOD, it always effectively outputs the prior. We would say that our method subsumes OOD detection, because it also includes ID calibration, and OOD calibration. To our knowledge, there are no other papers demonstrating any algorithm with all these properties.
Isn't this just another kernel density estimation (KDE) approach? There are similarities between KDE and the proposed method. However, there is an indicator function in Equation 7 which is implemented using the geodesic distance proposed in the draft (which is absent in KDE), which makes KDN, KDF scale better with higher dinensions. Morcover, the center and bandwidth of the Gaussian kernels are estimated in a data-driven manner using the representation learned by the parent discriminative approach.
Using Gaussian kernel for posterior estimation is not novel. Isn't this similar to Gaussian Process regression? Our proposed approach utilizes the internal structure learned by the parent approach to build an estimate of the posterior and scale for really high dimensional datasets. However, it is folk's knowledge that Gaussian Process does not scale beyond dimension 20.
Number of polytopes grows exponentially with the size of network. Isn't the approach compu- tationally infcasible? Many people were concerned about the runtime of our approach, possibly could be an exponential function of the number of nodes. However, we did additional experiments (see Fig. G where we show training and testing time both are linear in the number of the nodes. Moreover, training time is only 200 seconds even when there are 40,000 nodes running on MacBook Pro with Apple M1 Max chip and 64 GB of RAM. The number of total polytopes in KDN is upper bounded by the training sample size as we only consider the polytopes populated by training data (see the first paragraph of Section 2.3 and Equation 7) With overparameterized networks, which is the norm, they typically exhibit this property as shown in Fig. G that the number of populated polytopes is equal to the sample size
How does the proposed approach perform in term of runtime compared to the bascline ap- proaches? OOD calibration approaches such as ACET, OE and ODIN take about days, an hour, hours, respectively on GPUs. In-distribution calibration methods such as isotonic regression and sigmoid regression
17

---

Published in Transactions on Machine Learning Research (06/2025)
take few minutes and use CPUs. Our approach addresses both ID and OOD calibration while taking a few minutes to train on CPUs, rather than GPUs. All the computations were performed for producing the results in Table [ using MacBook Pro with an Apple M1 Max chip and 64 GB of RAM.
Why is there no non-asymptotic performance guarantee? Because of "No Free Lunch' theorem, it is impossible to provide a non-asymptotic guarantee that works for all training distributions. Hence, we provided extensive empirical experiments encompassing simulation, tabular and vision datasets to build intuition about. the finite sample performance of the proposed approach.
Why did you use new statistic called OCE for OOD calibration measure? OCE measures OOD calibration according to Equation and assumes that true class conditional priors for the datasets are known. On the contrary, ECE is used for mcasuring ID calibration and is a surrogate mcasure used when true posteriors are not known. For example, in Figure 3 where we know the true distribution, we have used Hellinger distance from the true posteriors instead of ECE. Note that measuring ECE requires accuracy which requires class labels and there is no class label available for the OOD samples in our setting.
Why is there no OOD baseline approach for the tabular benchmark datasets? ACET, ODIN, OE are tailor-made for vision problems, therefore we can not run them on tabular data using the author provided codes. To the best of our knowledge, the tabular OOD method (Ren et al. 2019) that we found does OOD detection, not calibration. As it does not yield any posterior, we could not benchmark with the above method.
In Table 1 it seems OOD approaches including ACET and ODIN can't. even beat the parent model under OOD settings. ACET and ODIN highly depend on the model architecture and the nature of the ID and OOD testsets. Morcover, they also depend on the OOD set usod to train them. We used the OOD set used by the authors of the above algorithms. All these factors contribute to their inconsisteet performance across datasets and model architecture. Tajwar et al. 2021 found a qualitatively similar result.
What would be the AUROC, FPR, i.e. metrics commonly used in OOD detection? To save space in the main text, we have added AUROC and FPR in Appendix Table 2 Note that these scores are used for OOD detection. However, we are addressing both ID and OOD calibration (Equation 4 in the main text). OOD detection works as long as there are two distinguishable score sets for ID and OOD points, whereas calibration aims at estimating the true predictive uncertainty of these points, both in ID and OOD regions. That being said, a well-calibrated model can perform detection, but a model capable of detecting OOD points may not be calibrated.
Table 2: KDN has nearly similar or better AUROC and FPR to those of baseline OOD detection approaches 't' and '1' indicate whether higher and lower values are better, respectively.
Dataset
Statistics
Purent
XWW
IscrOMIC
SIGHOID
ACET
ODIK
0g
Focal
Dual Focal
ID
CIFAR-10
Acsuracy( (8)
98.06 0.00
97.
1.6110000 0.00
98.16 0.000
05.10 1: 0,00
98.23
0:000
97.07 = 0.00
97.94 0.00
94 0,00
95.23 0.00
CIFAR-100
AUROC
0.58 0.00
0,32 100L
0.90
0.00
0.32
0.56
0.56 11,00
0OD
FPRH05
0.001
0.12 0.01
1.0.1
0.10
0.12 0.00
0.54 1 0.00
0.10 0.00
0.57 0.01
0.39 1 0.01
SVHN
AUHOUTT
0.58 1001
1.000
0,89 0.00
11300 0000
0.00
0.94 0.02
0.99 2: 0.00
U.35 0.012
0.94 2 1.111
FPRRI5
0.00
0.01
0.15++0.00 0.00
0.12++000
0/5 0.01
0.17 0.03
0.01 + 0.00
0.44 0.09
0.45 0.07
Nose:
AUROCT 1
0.85 0.00
0.98 1:000
0.82
0.00
0.80 0.00
0.90 1 0.00
0.90 1: 0.00
0.90
0.89 0.06
0.36 1: D:03
FPR895
0.17 1: 0.00
4 0.03
0,12 0.00
0.11 1 0.01
0.00 1: 0.00
0.00 0.00
0.00 1 0.01
0.78 0.31
0.99 1: 0.00
ID
CIFAR-100
Assuracy(99)
86.72 0.00
85-46
0.00
65.33 +.00
86.61 :: 0.00
85.07 0.01
86.56 =: 0.00
86.00 0.00
79:37 s 0.06
50.81 + 0.00
CIFAR-10
AUROCT T
0.80 0.01
0.97
0.00
0.83 0.02
0.30
0.30
0.01
0.51 0.00
0.81 + 0.01
00D
FPRM05
0.41 0.0.2
0.05 1 1:0.01
0.61 0.04
0.38 0.00
0.57 4: 0.04
0.58 1: 0.02
0.30 0.03
0.66 =: 0.01
0.68 1: 0.01
SVHN
AUROUT
0.91 1 (..0
0.95 1 0.00
1.87 0.01
1.91
4 0.012
0.81 0.02
0.92
1.111
010.99 10002
0.89 1 1L.12
FPRRMG
0.30 0.03
0.04 + 0.01
0.59 + 0.00
0.38 + 0.03
0.06
0.62 0.06
0.34 0.03
0,80 0.02
0.51 + 0.06
Nosse
AUROCT 1
0.06 0.01
1:000
0:03
0.00
0.05 0.00
1.00 0.000
0.08
a 0:01
0338
1: 0.02
0:00 0.01
1.00 0.00
FPRA96
0.05
0.00 + 0.00
0.03
0.07 0.06
+ 0.00
0.08 0.04
0.15 0.17
0.00 0.00
0.00 0.00
1D
SVHN
93.52 + 0.01
92.84 + 0.00
93.64 0.01
09.65 + 0.01
95:22 +: 0.01
95.52 0.01
98.
- 0.01
60:30 0.00
61.02 0.00
00D
CIFAR 10
PPRM05
AUROCT
0.56 0.01
0.007
0.97 0.01
0.06 + 0.02
0.25 + 0.08
0.56
0.01
0.34 0.00
0.34
0.35
0.58 0.06
0.36
0.18 + 0.07
0.99 0.00
0.02 + 0.01
0.52
0.07 = 0.00
0.54 = 0.01
0.97 + 0.00
CIFAR-100
AUROUTT
0.95 0.01
0.97 0.01
0,50 1 0.01
11305 1 0.01
0.95 0755110 0.01
0.35 0.01
1.111
0.32 0.01
0.35 1 1.000
FPR
0.30 0.07
0.06 + 0.02
0.25 0.07
0.35 0.08
0.37 0.05
0.19 0.06
0.04 = 0.02
0.97
0.97 + 0.00
Nosse
FPRRIG
AURDCCT
0.066 4 0.02
0.16
0.98 0.03
0.05
0.08
0.16 0.11
0.00
0.26
0.30 7 0001
0.00 0.00
1.00 + 0.00
0.001 + 0.00
1.00 + 0.00
0.01
1.0
0063 0.00
0.64 0.06
0..00
18

---

Published in Transactions on Machine Learning Research (06/2025)
Why did you choose CIFAR-10, CIFAR-100 and SVHN for your vision experiments? We have done vision experiments using CIFAR-10 (10 classes), CIFAR-100 (100 classes) and SVHN (10 classes and bigger training size) as ID datasets. We emphasize that CIFAR-10. 10, CIFAR-100 and SVHN are some of the hardest ID and OOD pairs according to various papers (Nalisnick et al. 2018 Fort et al. 2021 and hence they are adopted as the benchmarking datasets by many of the papers in the literature. Doing experiments with extremely large datasots like imagenet (14 million images) computationally and storago-wist expensive using our current implementation. We will pursue extremely large datasets in future. Many relevant papers OOD calibration use only small- and mid-sized datasets (Gardner et al. 2024 Borisov et al. 2022 Ulmer
et al. 2020
We acknowledge this limitation in the paper.
As shown in Algorithm 2, in each iteration, Geodesic kernel values between the current train- ing input and the entire dataset need be computed, each of which involves comparing all possible activation paths through the neural network (Equation 12 and Algorithm []. And Algorithm 2 iterates over every single training instance. This will become extremely challeng- ing on modern neural networks with miliions, or even billions, of paramcters (nodes) trained on large-scale datasets consisting of millions of training samples. We address this scalability issue with two solutions:
Parallclization for Efficient Kernel Computation: Our kernel is computed as the product of Hamming distances between activation patterns (binary strings of '1' and '0') at each layer for two samples. Since these calculations are independent across layers and sample pairs, they can be efficiently parallelized. We leverage the cdist function from scipy.spatiol. distance, which enables fast pairwise distance computations. This parallel implementation reduced runtime from days to just minutes.
For extremely large models, such as vision trans- formers (ViTs) in Section 3.2.2, we employ a front-end encoder to extract local image features (as described in Section 3.2, first paragraph). Instead of applying KDN the entire ViT, we fit it only to the final fully connected layers. This reduces the size of the model by about 90% while maintaining performance, allowing more efficient KDN computation.
Efficient Computation for Large Models:
As dimension continues to increase, densities of Gaussian kernel tend to concentrate on a thin shell, where densities decay exponentially both going outward (away from the centre) and inward (towards the centre). Thus, a low class conditional density estimated by the Gaussian kernel does not necessarily imply the inference sample is OOD, Such high-dimensional feature space (large) ) is commonly encountered in practical tasks. In such high dimensional spaces, most of the probability mass concentrates on a thin shell a certain radius from the mean due to the rapid growth of volume on the surface of the shell. While the density of Gaussian distribution is highest the mean and decays exponentially with Euclidean distance from the mean, the total probability mass at a given radius is given by the integral of the probability density over that region. The volume of the surface at that radius increases rapidly with higher dimensions (Fraser) 1984). In high dimensions, this results in the counterintuitive effect where the peak probability mass not at the mcan but at some distance from it. However, this phenomenon does not affect our approach. As shown in Equation 5, we evaluate the Gaussian kernel at single point x, not over an entire surface or volume. Since the kernel density maximized the center and decays exponentially outward, the volume concentration effect does not alter the inference process in our method.
B Proofs
B.1 Proof of Proposition i
To prove that is a valid distance metric for Pn, we need to prove the following four statements. Here we use the notation d(r, 8) to denote the distance between two samples æc Q and æ, E Q.-
19

---

Published in Transactions on Machine Learning Research (06/2025)
1. d(r,s) = when ==
Proof: By definition, K((,s) = and d(r;s) = O when r=
2. d(r,s) > 0 when r /
Proof: By definition, 0 s Ki(r, 8) < 1 and d(r,s) > 0 for **
3. d is symmetric, ie., d(r, 8) d(s,r).
Proof: By definition, K((r,s) k(),r) which implies d(r,s) = d(s,r).
4. d follows the triangle inequality, L.e, for any three polytopes Qr, Qai Qt E Pni d(r,t) s d(r,s) + d(s,1).
Proof: Let A, denote the sequence of elements in the activation mode for particular polytope r.
B is the cardinality of Ar. Below c(.) denotes the cardinality of the sequence. We can write:
B B (A))
(26)
= + - -C(A, -(A,
2 24 n.).) + -(d nA).
Rearranging the above equation, gel:
B- -A 0A) B-B CA, nA,) B-
c(A, B-A n(A))
c(Ar A1))) <
c(Ar n.).
+1
B
B
c(A, A.)
B
d(r,t) s d(r, 8) + d(s,t).
(27)
B.2 Proof of Proposition 2
Note that first we find the nearest polytope to the inference point x using Geodesic distance and use Gaussian kernel locally for x within that polytope. Here the Gaussian kernel uses Euclidean distance from the kernel center (within the numerator of the exponent). The value out of the Gaussian kernel decays exponentially with the increasing distance of the inference point from the kernel center. We first expand 9y
fx())P(())
(x) ΣK (u(t)
[x()) log(n)
b (v)
(x
(k) + log(n) (x:)
As the inference point becomes more distant from training samples (and more distant from all of the Gaussian centers), we have that (xx, for, E,) becomes smaller. Thus, fy, shrinks. More formally, Vy,
lim (x) 0.
We can use this result to then examine the limiting behavior of our posteriors as the inference point x becomes more distant from the training data:
fx() (v) Togin Py (v) (k) + Tog(n) (v)
Py
lim (x)x) lim K (x) -00
dx
Pr
Ek (k)
Pv().
20

---

Published in Transactions on Machine Learning Research (06/2025)
Note that without the constant term log(n) the class priors in the numerator will be and hence we will not have Proposition 2
C
Hardware and Software Configurations
Operating System: Limux (ubuntu 20.04), macOS (Ventura 13.2.1)
VM Size: Azure Standard D96as (96 vepus, 384 GiB memory)
GPU: Apple M1 Max
Software: Python 3.8, scikit-learn 2 0.22.0, lensorllow-maces22299, lensorflow-metal S 0.5.0.
Simulations
Simulation
KDF and RF
KDN and DN
Classification Error Hellinger Distance
Mean Max Conf.
Classification Error
Hellinger Distance
Mean Max Conf.
0.1
1.0
1.0
Gxor
0.1
0.1
0.0
0.5
0.0
0.5
1.0
0.4
1.0
Sinewave
0.2
0.2
0.0
0.5
0.0
0.5
Polynomial
0.3
0.2
1.0
1.0
0.2
0.2
0.5
0.1
0.5
KDF-Geodesic
COF-Euclidean
KDN-Geodesic
KDN-Euclldean
DN
Figure 7: Additional simulation result. KDF and KDN improve both in- and out-of-distribation calibration of their respective parent algorithms while maintaining nearly similar classification accuracy on the simulation datasets.
We construct six types of binary class simulations:
Spiral is two-class classification problem with the following data distributions: let K be the number of classes and S multinomial( Ix, n). Conditioned on S, each feature vector is parameterized by two variables, the radius and an angle 0. For each sample, is sampled uniformly in [0, 1]. Conditioned on particular class, the angles are evenly spaced between 4=(k-1)tk and 4=(kitx where Ix controls the number of turns in the spiral. To inject noise along the spirals, we add Gaussian noise to the evenly spaced angles 0' 0 0' + (0000.009) The observed feature vector is then (r cos(O),r si))
K
Circle is a two-class classification problem with equal class priors. Conditioned on being in class 0, a sample is drawn from a circle centered at (0, 0) with a radius of r 0.75. Conditioned on being class 1, a sample drawn from a circle centered (0, 0) with a radius of 1, which cut off by the region bounds. To inject noise along the circles, we add Gaussian noise the circle radii = (0,00.0))
21

---

Published in Transactions on Machine Learning Research (06/2025)
Gaussian XOR
Simulation Data
True Posteriors
RF
KDF-Geodesic
KDF-Buclidean
DN
KDN-Geodesic
KDN-Euclidean
o
Spiral
Circle
-2
0.5
Sinewave
V
0.0
Polynomial
Trunk
2 -1
-2
-2
0
-2 -1
0
-
-1
6
Figure 8: Visualization of true and estimated posteriors for class from five binary class simula- tion experiments. Column 1: 10,000 training points with 5,000 samples per class sampled from 6 different simulation setups for binary class classification. Trunk simulation is shown for two dimensional case. The class labels are indicated by yellow and blue colors. Column 2-8: True and estimated class conditional posteriors from different approaches. The posteriors estimated from KDN and KDF are better calibrated for both in- and out-of-distribbtiio regions compared to those of their parent algorithms.
Trunk is two-class classification problem with gradually increasing dimension and equal class priors. The class conditional probabilities are Gaussian:
P(X 0) G((1, P(X = 9( I),
where 141 14, -4, y is d dimensional vector whose i-th component is (2) and T is d dimensional identity matrix.
Sinewave a two-class classification problem based on sine waves. Conditioned on being in class a sample is drawn from the distribution y cos(mz). Conditioned on being in class 1, sample drawn from the distribution y sin(Tæ)). We inject Gaussian noise to the sine heights y' y y' + (0,0.00).
Gaussian XOR two-class classification problem with equal class priors. Conditioned on being class 0, sample is drawn from a mixture of two Gaussians with means = -0.5] and standard deviations of 0.25. Conditioned on being in class 1, sample is drawn from mixture of two Gaussians with means +(0.5, - and standard deviations of 0.25.
Polynomial is two-class classification problem with the following data distributions: y . Conditioned being in class 0, sample drawn from the distribution y xt. Conditioned
22

---

Published in Transactions on Machine Learning Research (06/2025)
being in class 1, a sample is drawn from the distribution y 2 Gaussian noise is added to variables y' y' +N 0.01).
Table 3: Hyperparameters for RF and KDF.
Hyperparameters
Value
n estimators
500
max depth
0
min _an__sammlsss.. leaf
10 6
exp (-10-)
Table 4: Hyperparameters for RELU-net and KDN on Tabular data.
Hyperparameters Value
number of hidden layers
nodes per hidden layer optimizer
1000
Adam
learning rate
3 x 10
x 10
exp (-10-7)
E
Extended Results on OpenML-CC18 data suite
See Figure
10 and 12 for extended results on OpenMI-CC18 data suite.
23

---

Published in Transactions on Machine Learning Research (06/2025)
KDF and MCE
KDN and DN MCE
Class Broor
Mear Max Cor!
C Ciassiication Erro
Meen Max Corf.
311
475
1.71
0.00
6:00
0.25
9.54
811
330
2004
800
600
0.08
6:16
3.7
8,2
0,56
625
6:0
Namber of ling Somples Dog)
Dalorce
Number of Trii lng Sumples (og)
Ohlonce
Figure 9: Extended results on OpenML-CC18 datasets. Left: Performance (classification error, ECE and mean max confidence) of KDF on different Openml-CC18 datasets. Right: Performance (classification error, ECE and mean max confidence) of KDN on different Openml- Dpenml-CC18 datasets.
24

---

Published in Transactions on Machine Learning Research (06/2025)
KDF and RF
KDN and DN
Flatsiication
Mean Cort.
Toason Ever
Nox
6:00
92
0.1
18
0.00
153
0.3
0.00
00
4294
0.30
1955
6:0
0150
Dilea
22
6386
055
0.56
800
18
13'
1863
0.1
Mambor Somoleo
Namber
XBF
KON
Botrii DI
5kg ApM
CM
Figure 10: Extended results OpenMI-CC18 datascts (continued). Left: Performance (classifica- tion error, ECE and mean max confidence) of KDF on different Openml-CCi8 datasets. Right: Performance (classification error, ECE and mean max confidence) of KDN on different Openml-CC18 datasets.
25

---

Published in Transactions on Machine Learning Research (06/2025)
KDF and MCE
KDN and DN MCE
Clossiiiation Enor
100 Meon Nox Corf
Classifical on Erro
Mean Max Cont
625
028
210
045
17
142
0.90
0.11
3.13
300
0.75
025
12'
504
340
30
650
309
1.00
1.00
0.50
640
Am
3000
Namber of sagiing Stmples (rg)
Dalurce
Namber of Truining Sumples Oog)
Ohionce
Figure 11: Extended results on OpenMI-CC18 datasets (contimued). Left: Performance (classifica- tion error, ECE and mean max confidence) of KDF on different Openml-CC18 datasets. Right: Performance (classification error, ECE and mean max confidence) of KDN on different Openml- CC18 datasets.
Openmi-CC18
26

---

Published in Transactions on Machine Learning Research (06/2025)
KDF and RF
KDN and DN MCE
Classification
Corf
Classificaton Erorr
Drpegn tt So
025
0.
Trakiig De0
kay Sumptoo
Figure 12: Extended results on OpenML-CC18 datasets (continued). Left: Performance (classifica- tion error, ECE and mean max confidence) of KDF on different Openml-CC18 datasets. Right: Performance (classification error, ECE and mean confidence) of KDN different Openml-CC18 datascts.
27


Reviews:
[{"review_id": "kCjLzpTOua", "reviewer": "Reviewer_CZ96", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Method achieves good performance in terms of calibration and OOD detection.\n\n* Method can be applied to existing networks, given that it uses ReLU activations.\n\nWeaknesses and questions:\n\n* From Figure 5 we can see that the number of polytopes goes to 10 000 quickly, which is also the size of the training data. This raises the question of whether fitting the kdx model is even necessary? If all the polytopes contain only a single point, then it is the same as finding the\nnearest neighbour. Did the authors also consider finding the nearest neighbour and calculating the gaussian kernel to that one?\n\n* In Figure 5, assessing scaling behavior is challenging because the models quickly reach the maximum number of polytopes (equal to the size of the training data) using relatively few nodes. It would be more interesting to see this experiment with a larger training set\n\n* The class conditional priors were fixed for OCE calculation. It remains unclear how the class conditional priors were fixed.\n\n* Since we can see confidence intervals for the results, it would be nice to mention how many times the experiments were repeated.\n\n* Looking at the formula (5), we see that during the inference, the value of f_y(x) depends on the test set size. Is there a big difference, if we look at the test set as a whole or each test point individually?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the task of simultaneously calibrating models for both in-distribution (ID) and out-of-distribution (OOD) instances. Typically, approaches focus exclusively on either ID or OOD calibration. The proposed method clusters training data based on the activations of the model. At test time, the method locates the closest polytope center in input space and then applies a Gaussian kernel, whose parameters are determined during the fitting of the KDX model. The effectiveness of the proposed approach is validated through classification\nexperiments conducted on simulated, tabular, and vision datasets. Experimental results demonstrate strong performance in terms of both ID and OOD calibration, as well as OOD detection.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The questions stated above in the section on Strengths and Weaknesses need answers in the paper.", "broader_impact_concerns": "There are no ethical concerns that would require more detailed attention in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VKQvlGr993", "reviewer": "Reviewer_psMA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### **Strengths**\n\n1. The proposed method is intuitive and well defined.\n1. The paper is nicely written with well-defined notations.\n2. The work brings attention to an important feature (calibration) for DNNs designed for a popular task - OOD detection.\n\n\n### **Weanesses**\n1. The proposed OOD Calibration Error is unconvincing. It lacks a connection to the definition of Expected Calibration Error.\n\n2. The proposed OOD Calibration Error (OCE) (Equation 18) is tailored specifically for the proposed method. However, an empirical prior class distribution is not a gold standard in identifying the OOD samples. Detection of OOD data can vary across the methods. Among the compared OOD detection methods, ODIN thresholds the top prediction probability, which does not force the OOD samples to have a prior class probability; and both ACET and OE (also threshold based) encourage OOD samples to have a uniform class distribution. Such an evaluation is biased against the compared OOD methods.\n\n3. Scalability issue #1 - computational resources. As shown in Algorithm 2, in each iteration, Geodesic kernel values between the current training input and the entire dataset need to be computed, each of which involves comparing all possible activation paths through the neural network (Equation 10 and Algorithm 1). Note that there are \n> $B = \\prod_{l=1}^L |\\mathcal{N}_l|,$\n> where $L$ is the total number of layers and $\\mathcal{N}_l$ is the set of nodes corresponding to the l$^{\\text{th}}$ layer.\n\n    possible activation paths for each training input. And Algorithm 2 iterates over every single training instance. This will become extremely challenging on modern neural networks with miliions, or even billions, of parameters (nodes) trained on large-scale datasets consisting of millions of training samples.\n\n4. Scalability issue #2 - Behaviour of Gaussian kernel changes in high-dimensional space. As $d$ continues to increase, densities of a Gaussian kernel tend to concentrate on a thin shell, where densities decay exponentially both going outward (away from the centre) and inward (towards the centre). Thus, a low class conditional density estimated by the Gaussian kernel does not necessarily imply the inference sample is OOD. Such high-dimensional feature space (large $d$) is commonly encountered in practical tasks.\n\n5. Comparison methods. The compared model calibration methods, *Isotonic Regression (IR)* and *Sigmoid (Platt) Regression (SR)*, are relatively old and under-performing in contemporary model calibration research. It is also known that post-processing based calibration methods, including IR, SR, temperature scaling etc., are particularly vulnerable to calibration on OOD data or OOD detections. On the contrary, training-based model calibration methods typically have much better performances on OOD data. For example, the following works (non-exhaustive) have demonstrated their calibration effectivenesses on OOD data. However, all of the training-based model calibration methods are left out of the comparison.\n    1. Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P., & Dokania, P. (2020). Calibrating deep neural networks using focal loss. Advances in neural information processing systems, 33, 15288-15299.\n    2. Tao, L., Dong, M., & Xu, C. (2023, July). Dual focal loss for calibration. In International Conference on Machine Learning (pp. 33833-33849). PMLR.\n    3. Bohdal, O., Yang, Y., & Hospedales, T. M. (2023). Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error. Transactions on Machine Learning Research, 1-21.\n    4. Yu, Y., Bates, S., Ma, Y., & Jordan, M. (2022). Robust calibration with multi-domain temperature scaling. Advances in Neural Information Processing Systems, 35, 27510-27523.\n\n6. The experiments are relatively small-scale.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submitted manuscript proposes a kernel-based method to address the model calibration issue for the out-of-distribution (OOD) detection task while achieving calibration on in-distribution data. Following the existing literature that interprets the deep neural network as partitioning the feature space into a union of polytopes, the proposed method uses a Geodesic Kernel to estimate the similarity between the partitioning paths of different training/inference data samples. The estimated similarity facilitates the computation of soft membership of samples to each polytope as well as the fractions of class-specific data being sorted into each polytope. Furthermore, within each polytope, a Gaussian Kernel with learnable parameters is introduced to decay the class conditional density exponentially with respect to the distance to the centre of the kernel. Finally, Following the Bayes' rule, both the estimated class conditional density and the fractions, along with a prior class probability, are used to estimate the posterior class probability of a given inference point, which is subsequently used for OOD detection and OOD calibration evaluation. This work also proposes a metric to evaluate the calibration performance for OOD detection.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### **Requested Clarifications**\n\n1. The proposed OOD calibration error needs to have established connection to the definition Expected Calibration Error in order to be meaningful.\n\n2. Solutions to both scalability issues are required.\n\n3. Comparisons with latest model calibration and OOD detection methods are preferred.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DJzfPF3Hc7", "reviewer": "Reviewer_F1TG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The OOD calibration problem is an interesting problem that is worth studying.\n2. The description of the method is quite clear.\n3. The reported experiment results show that the method has better OOD calibration than classical methods such as Platt scaling.\n\nWeaknesses:\n1. The application of the proposed method is very narrow. The paper only discusses decision forests and ReLU networks. And even for ReLU networks, the kernel defined in Eqn. (10) in Section 2.4.2 is very sparse when the network is very deep, because almost no two sample will have the exact same activation path. Although the experiments in Section 3.2.2 consider ViT, I cannot understand this section so I am not sure if the method can be applied to a more realistic setting.\n2. The proposed method seems a bit crude. It is essentially approximating the class-conditional data distribution with a Gaussian mixture like model. I don't think this is applicable to complex data. In the experiments, only Section 3.2.2 uses CIFAR, but I cannot understand the result of this section.\n3. For a ViT B16, how is the network kernel defined? Do you strictly the definition in Eqn. (10)? Wouldn't that be a very sparse kernel? And why do you use complete complete white noise as OOD? Wouldn't that make the task too easy? I am not convinced by the results in Section 3.2.2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies calibration for out-of-distribution (OOD) data. The proposed method models the class-conditional data distribution (or confidence) of each class as a sum of Gaussian kernels, similar to a Gaussian mixture model but with some pruning. The model resembles a model in the form of a union of polytopes, such as decision forests and ReLU networks. These models provide the class probabilities for each sample, which can then be used to compute the confidence of the classifier.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add a discussion on whether the kernel in Eqn. (10) is still meaningful when the neural network is very deep.\n2. The presentation of the experiment results can be improved. Figure 2 is not very parsable. I am not sure how it shows that the proposed method improves OOD calibration. Section 3.2.2 needs a major edit. I don't think it is a good idea to use complete noise from a uniform distribution as OOD. For example, you can use SVHN as OOD for the CIFAR dataset.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received reviews from three experts, all of whom provided constructive comments and valuable suggestions. Following the discussion, the overall feedback is positive. However, several points need to be addressed in the camera-ready version:

- “With overparameterized networks, which is the norm, they typically exhibit this property that the number of populated polytopes is equal to the sample size.” Reviewer CZ96 suggested that this point should be clearly stated in the paper.
- I concur with the Reviewers F1TG  and psMA that the experiments are conducted on small-scale datasets using relatively simple models. It would strengthen the paper to include a discussion on how the proposed method could generalize to more complex datasets and deeper neural networks.
Decision: Accept (poster)