Pointwise Information Measures as Confidence Estimators in Deep Neural Networks: A Comparative Study
Authors: Shelvia Wongso
Rohan Ghosh
Mehul Motani
Venue: ICML 2025 poster (2025)
Keywords: information theory
confidence estimation
deep neural networks
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We explore how information-theoretic measures can improve confidence estimation in neural networks.

Abstract: Estimating the confidence of deep neural network predictions is crucial for safe deployment in high-stakes applications. While softmax probabilities are commonly used, they are often poorly calibrated, and existing calibration methods have been shown to be detrimental to failure prediction. In this paper, we propose using information-theoretic measures to estimate prediction confidence in a post-hoc manner, without modifying network architecture or training. Specifically, we compare three pointwise information (PI) measures: pointwise mutual information (PMI), pointwise $\mathcal{V}$-information (PVI), and the recently proposed pointwise sliced mutual information (PSI). These measures are theoretically grounded in their relevance to predictive uncertainty, with properties such as invariance, convergence rates, and sensitivity to geometric attributes like margin and intrinsic dimensionality. Through extensive experiments on benchmark computer vision models and datasets, we find that PVI consistently outperforms PMI, PSI and existing post-hoc baselines in failure prediction across metrics. For confidence calibration, PVI matches the performance of temperature-scaled softmax, which is already regarded as a highly effective baseline. This indicates that PVI achieves superior failure prediction without compromising its calibration performance. This aligns with our theoretical insights, which suggest that PVI offers the most balanced trade-offs.

Pointwise Information Measures as Confidence Estimators in Deep Neural Networks: A Comparative Study
Shelvia Wongso
Rohan Ghosh
Mehul Motani 12
Abstract
1. Introduction
As deep neural networks (DNNs) are increasingly being de- ployed in high-stakes areas like healthcare and autonomous driving, the focus has shifted from achieving good accuracy to also ensuring trustworthiness for safe deployment (Kaur et al., 2023). An important aspect of a trustworthy model is its ability to accurately quantify prediction confidence, enabling users to judge when to trust its outputs (Jiang et al., 2018). The standard way to estimate confidence via soft- outputs, but they can be overconfident misclassified samples (Guo et al., 2017). Many existing approaches that address this issue involve modifying the network architec- ture (Corbière et al., 2019) or training procedure (Gal & Ghahramani, 2016), which may not always be feasible in practice. Meanwhile, popular confidence calibration meth- ods have been shown be useless or harmful for failure prediction tasks (Zhu et al., 2022). This study addresses both failure prediction and confidence calibration by analyzing cnoomatiion-thorretii measures to estimate the confidence of predictions from trained networks in a post-hoc manner, without altering their architecture or training process.
Estimating the confidence of deep neural network predictions is crucial for safe deployment in high- stakes applications. While softmax probabilities are commonly used, they are often poorly cal- ibrated, and existing calibration methods have been shown to be detrimental to failure predic- tion. In this paper, we propose using information- theoretic measures to estimate prediction confi- dence in a post-hoc manner, without modifying network architecture training. Specifically, we compare three pointwise information (PI) mea- sures: pointwise mutual information (PMI), point- wise V-information (PVI), and the recently pro- posed pointwise sliced mutual information (PSI). These measures are theoretically grounded in their relevance predictive uncertainty, with proper- ties such as invariance, convergence rates, and sensitivity to geometric attributes like margin and intrinsic dimensionality. Through extensive ex- periments on benchmark computer vision models and datasets, we find that PVI consistently outper- forms PMI, PSI and existing post-hoc baselines in failure prediction across metrics. For confi- dence calibration, PVI matches the performance of temperature-scall softmax, which is already regarded highly effective baseline. This in- dicates that PVI achieves superior failure predic- tion without compromising its calibration perfor- mance. This aligns with our theoretical insights, which suggest that PVI offers the most balanced trade-offs.
Mutual Information (MJ) is the conventional information measure used to capture statistical dependence between two random variables (Cover & Thomas, 2001). However, accurately estimating MI in high-dimensional spaces, typ- ically encountered in the context of DNNs, is challenging due to an exponentially large sample complexity (Battiti, 1994). In recent years, there have been proposals for al- ternative measures of informativeness that scale well with dimensions. The first is the V-information (VI) which mea- sures the amount of usable information under computational constraints (Xu et al., 2020). The second is sliced mutual information (SMI) which is the average of the MI between one- -in-ddmensional projections of the random variables (Gold- feld & Greenewald, 2021). Unlike MI, both VI and SMI can estimated reliably from data, even high dimensions.
'Department of Electrical and Computer Engineering. National University of Singapore, Singapore. NN.1 Institute for Health, Institute for Digital Medicine, Institute of Data Science, National University of Singapore, Singapore. Cor- respondence to: Shevia Wongso <shelvia.w @w.nus.edu>> Rohan Ghosh <rghosh92@ @eaieeemmal.mm.... Mchul Motani <motani weeuus.cuuu edu.sg>.
To apply the three inttommmttiontthorette measures (MI, VI, and SMI) for confidence estimation, we their point- wise variants: pointwise MI (PMI), pointwise VI (PVI), and pointwise SMI (PSI). Specifically, we use these pointwise information (PI) measures to quantify the degree of rele- vance between feature representation and predicted output
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
of model for each individual sample. We analyze their the- oretical properties, including invariance, convergence rates, dependence on margin and intrinsic dimensionality, and discuss their relevance to confidence estimation problem. Empirically, we compare their effectiveness in estimating confidence scores on benchmark image classification tasks for failure prediction and calibration. For a review of related work on confidence estimation and the three PI measures, please refer to Appendix A.
alternative to the softmax output computed from logits for confidence estimation. They quantify the relative increase in confidence for the predicted class compared to its prior occurrence probability. By focusing on this relative change, Pl measures can help mitigate biases in p(8lz)) which can arise due to underrepresentation of certain classes in the data.
4. Relationship to Probabilistic Causation: We find that PI measures can be also interpreted via the lens of proba- bilistic causation. This perspective on causality, out- lined by (Hitchcock, 1997), argues that X causes Y if X > P(Y). In the context of confidence esti- mation, the goal is to quantify how strongly a feature x supports the predicted class ÿ; the model should natu- rally assign higher confidence to predictions where has strong influence on y. We argue that this problem can be mathematically formulated by measuring the quantity p(jla=))/ll /p(ů) which indicates the degree to which cer- tain feature æ influences the decision made for a single instance. This directly connects to PI measures which compute the logarithm of the probability density ratio.
Motivation. We provide four factors that motivate the use of PI measures for confidence estimation:
1. Recent Applications of PI Measures: PI measures have recently found application in diverse domains of DNNs, showcasing their versatility and effecttiveness. For in- stance, the significant work by (Ethayarajh et al., 2022) showcases the applicability of PVI to the problem of dataset difficulty, which relates to confidence as net- works are naturally less confident in their predictions when the datasets are harder. While PI measures are more commonly applied in natural language, we focus on their potential in computer vision, an area still rela- tively under-explored from an information-theoreti per- spective. More closely aligned with our work is the study by (Wongso et al., 2023b) which proposed PSI for confidence estimation and explainability. We extend their research by comparing the performance of PSI with PMI and PVI. providing deeper theoretical insights, and conducting extensive evaluations.
Contributions. The specific contributions of this paper are as follows:
1. Information-Theory Grounded Measures. We pro- pose to use information- theeoreti pointwise measures (PMI, PSI and PVT) for confidence estimation. These measures capture diffcrent aspects of the relationship between model predictions and true labels, providing alternative perspectives beyond conventional softmax- based confidence scores.
Despite the increasing appli- cations of the Pl measures, there has been notable lack of research devoted to exploring their theoretical proper- ties. To the best of our knowledge, only PMI has been theoretically studied (Fano & Hawkins, 1 1961b), though largely in a more general context. In this work, we de- rive and compare theoretical properties of PMI, PVI, and PSI that are relevant to this setting, including invariance, convergence rates, margin and intrinsic dimensionality. We find that these measures exhibit some properties that can be relevant in the context of confidence estimation.
Theoretical Foundations:
2. Theoretical Analysis of PI Measures. We perform an in de-depth study of the theoretical properties of the three PI measures, including invariance properties, conver- gence rates, and geometric characteristics such as margin and intrinsic dimensionality (Section 3 and Appendix B.3). We explain how each of these properties relates to confidence in neural networks and highlight several noteworthy caveats. In particular, we observe that mar- gin sensitivity is a less critical property for confidence estimation tasks (failure prediction and confidence cali- bration) compared to invariance and convergence rates. 3. Empirical Evaluation of PI Measures. We evaluate the three PI measures for both failure prediction and confi- dence estimation tasks across a diverse range of datasets and architectures. (Section 4). We find that PVI consis- tently outperforms both PMI and PSI, as well as standard post-hoc methods, in failure prediction across all met- rics, with particularly notable gains on larger models and more complex datasets. For confidence calibration, PVI matches temperature-scaled softmax, which is already regarded highly effective baseline, indicating that PVI achieves superior failure prediction without compro- mising its calibration performance.
3. Information Theoretic Connection: Another interpre- tation of the Pl measures comes from the notion of in- formation gain in information theory. In its standard (aggregate) form, information gain measures how much knowing one variable X reduces uncertainty about an- other variable Y, i.e., HYY)= (Y)))). PI measures can be seen as pointwise version of this information gain, computing the logarithm of the probability density ratio p(ū) x /p) (and its variant), where x is the feature and ÿ is the predicted output. In contrast, a typical neural network output approximates the conditional probabili- ties of each class p())x)) from logits, which themselves lack direct probabilistic meaning. PI measures, being grounded in information theory, offer a more principled
2

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
2. Information-Theoreti Measures
PSI Estimator. The estimation of PSI for supervised learn- ing tasks requires projecting only the feature vector x to one dimension, while labels y are typically discrete and therefore not projected. Using Bayes' Theorem, it can be re- written as follows:
Notation. We use upperease letters for random variables (e.g.. X), corresponding lowercase letters for their val- ues/outcomes (c.g., z), and calligraphic letters for their domains (e.g., 2). The joint probability distribution of X, Y is denoted by Pxy = P(X.Y) and their marginal distributions are denoted by Px- = P(X) and Py = P(Y). For specific outcomes I and y, we have p(a, = P(X = x, Y = y). p(z)= = P(X = x), and p(y) = P(Y = y)- Here, we provide the formal definitions of the three PI mea- sures (more details on their properties and estimators are given in the Appendix A.2 and Appendix A.3).
psi(=; y) == Esee [log estimate ((0xx/y), we use binning method or assume a Gaussian distribution. We compare the two estimators in the Appendix D.2.2 and use the Gaussian-based estimator (with 500 projections) in our experiments.
p(6" To p(00')
VI and Pointwise VI. VI was introduced to relax the un- bounded computation assumption of Shannon information, which may not be realistic in practice (Xu et al., 2020). It was later extended to its pointwise version, PVI, in (Etha- yarajh et al., 2022), for individual samples.
MI and Pointwise MI. MI measures the statistical depen- dence between two random variables (Cover & Thomas, 2001), while PMI measures the association between specific realizations of these random variables (Fano & Hawkins, 1961a). They are defined as follows:
Definition 3 OVI and PVI). Let (z.y)
PxY
P(X x )) and Ø represent null input that provides no information about Y. We are given predictive family V - Ω = {f XUU + P(Y)) We first define the V-entropy and conditional V-entropy as follows:
Definition 1 (MI and PMT). Let (x, Pxy- The MI and PMI are defined as follows:
P(X,Y) 1
1(x;Y)= = Exy log P(X)P(Y)
(1)
Hv(Y)= = inf Ey log [(g
(5)
FEV
p(z, v)
pmni(;; v) = log p(r))(y))
Hy(Y inf Ex |- log [(X]
(6)
FEV
PMI Estimator: (Tsai et al., 2020) proposed three methods to compute the probability density ratio p(æ, v)/p(>))))))))) p(v)) using neural networks: the probabilistic classifier method, the density-ratio fitting method and the variational JS bound method. We compare the three methods in the Appendix 1.2.1 and choose the variational JS bound method as the default estimator. This estimator relies on the variational form of MI, and in particular the Jensen-Shannow Shannon divergence between Pxy and Py (Poole et al., 2019). We note that although our method for estimating PMI incorporates neural networks, we utilize only a shallow 2-layer neural network, which is often considered well-calibrated (Guo et al., 2017).
Let 9 arg min Exyl-- [Y)] The VI and PVI are defined as follows:
arg mimmin f()) and g'
- -Hv()X),
(7)
pui(x +) = - g (| + log (x))()
(8)
The estimation of PVI requires training two neural networks: I for estimating HY) and I' for es- timating the conditional HY (X) (Ethayarajh et al., 2022).
PVI Estimator.
I' is trained with the input-label pairs from the training data (Fuain- Vuain) while f is trained with the null input-label pairs from the training data (Imull, Umain)- For computer vision tasks, images composed entirely of zeros can be treated as null inputs. The PVI can then be computed as: pui(x + y) = - log flø] - log f' (x(x)()) where (z, v) is an input-label pair from a held-out set. To ensure that the probabilities for computing PVI are properly calibrated, we consider using temperature scaling. We consider three different approaches: using the original trained network as f, using the same architecture but trained with different ini- tialization as I and using a one-hidden layer neural network (3) as f trained using penultimate features as inputs. We com- pare the three approaches in the Appendix D.2.3 and use the second approach (same architecture, different initialization) (4) as the default estimator for PVI.
SMI and pointwise SMI. SMI was proposed by (Goldfeld & Greenewald, 2021) an alternative measure to MI, which can be hard to estimate in high dimensions. Similarly, its pointwise variant, PSI, was proposed as an alternative mea- sure to PMI (Wongso et al., 2023b). Both SMI and PSI can easily scale to high dimensions by taking one-dimensional projections.
Definition 2 (SMI and PSI). Let (z, ~ Pxy E P(Rd= x Rdv). Let ④ Unif((dd.1) and &b -ni be independent of each other and (X,Y). The
SMI and PSI are defined as follows:
SI(X;Y)= := Eoco, [I00 X; YY),,),
psi(z;)) v) == Ioce [pmi( x; k;,TT)) v)]
Φ
3

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
3. Theoretical Properties
Remark 1 (Invariance and confidence estimation). We note that invariance to bijective transformations T is im- portant in the context of confidence estimation; otherwise, the PI measures will confound TT in their estimates. Us- froo from (1) in (Mukhoti et al., 2023), this can be expressed as: H[Y [x, DJ = H [ [T x, TDJ, where H x; TD] denotes the conditional entropy of the ouiput labels given both the transformed input T x and the trans- formed dataset TD {(Tx1, v1),..., aT Yn)), which implies that the underlying distribution has been trans- formed as well. Ideally, this property should hold for any invertible, and thus anformationn-reservingg transformation T. However, as we cannot ignore the constraints of the model involved the decision-making process, we restrict T to invertible linear transformations on z. In the following remark, we argue that invariance to the broader class of homeomorphic transformations may be counterproductive.
In this section, we analyze the theoretical properties of the three PI measures, focusing on their invariance and conver- gence rate analyses. Results on their geometric properties provided in Appendix B.3. Proofs and additional re- marks are given in Appendix B.
ing the terminology
3.1. Invariance Properties of PI measures
We argue that confidence estimates should remain stable under certain transformations of the data distribution, e.g.. translation or rotation. Intuitively. if a distribution is trans- formed and a point x: is mapped accordingly, the confidence assigned to æ should be preserved in the transformed space. This is essential for ensuring that confidence measures re- flect intrinsic uncertainty and are not confounded by bijec- tive transformations of the input space.
In what follows, we consider the case where X E IRd= are the features and Y E {0, 1} are the labels, and (zz, y) is a feature-label instance sampled from pxy. Note that in what follows, other than Theorem 1, all other theoretical results can be trivially extended to the multi-label setting.
Remark 2 (Caveat on invariance to homeomorphic trans- formations). In neural network architectures, feature repre- sentations at a given layer typically undergo complex trans- formations through multiple nonlinear activations, making them highly unlikely to remain invertible. Consider the case where features at a given layer appear as transformed ver- sions of themselves across different training iterations. This can occur due to random weight initializations, where feu- tures T from one run may correspond to WT' from another, with T' being the feature from a different initialization and W being an invertible matrix. Since all such versions of T encode the same underlying information up to a linear map. pointwise measures between T and the output labels should remain unchanged, thereby emphasizing the need for in- variance to rotational and random matrix transformations. However, extending this invariance to general invertible transformations, as seen in the case of PMI, can be coun- terproductive. The degree of non-linearin between T and Y can be indicative of the network's 's confidence in estimat- ing Y from T. Intuitively, more complex, non-linear maps are less likely yield confident predictions and vice-versa. Consequently, while PMI's invariance holds under broader homeomorphic transformations, we hypothesize that this property may not always be beneficial, which is also seen in our experiments.
For convenience of notation, when (Ir,) ~ Pxv, we de- note pmi(æ; y), psi(æ; y) and pui(z + 3) by pmip(z,)) y), psir(æ, y) and pvir(z, y) respectively. For estimating pvip(T, v), we assume that V refers to a fully connected neural network of arbitrary depth and fixed architecture, where each layer contains weights and biases. For any transformation T Rd + Rd, we denote the probability dis- tribution P(TX,Y) Y) by TP. We have the following results.
Proposition 1 (Invariance to shift, scale, and rotation). Let Tx= aRæ +p, where p E gdd represents the extent to which the distribution is shifted, and a E R is a sealar that represents how much the distribution is scaled. Fur- thermore, R ~ Rddx is a rotation matrix, such that we have RRT = I and det(R) = 1, where I is the identity matrix and det represents the determinant oper- ator. Then, we have: pmii(e. y) = p + v. psip(z, y) = psitplaRx + p) and (x,)(
Next, we have the following results for more general linear transformations and homeomorphic (continuous and invert- ible) transformations.
3.2. Convergence Rates
We provide convergence rates analysis for the estimators studied here. We note that PMI's convergence rates will depend on the choice of probability estimator, as differ- ent estimators give different convergence rates. Here, we mainly focus on the Kernel Density Estimator (KDE) for estimating the densities p(zly) and p(z). We consider the case of binary classification, thus, Y E (0. 1}. For the KDE estimator studied in (Jiang, 2017), we have the following convergence bound for PMI.
Proposition 2 (Invariance to general linear transforma- tions). Let Tx = Mæ, where M. ~ Rd, xdz is invert- ible. Then, we have: pvip(x,)) y) = puipp (Mr,)) y) and pmir(x,)) y) = pmirppMax,
Proposition 3 (Invariance to homeomorphic transfor- mations). Let Tx = f(x), where f Rdd + Rd= repre- sents any homeomorphism. Then, we have: pmip(x,y) (mir)((x)))))

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Proposition 4 (PMI convergence rate). Let P(X) be Pxy where X Rd and Y E {0. 13. Let pmin represent the KDE estimate of PMI using n samples. Assuming min {P(Y 0), P(Y=
Remark 3 (Convergence rates and confidence estima- tion). Note that convergence rates are critical factor in determining the stability and accuracy of confidence es- timates using these measures. If measure has slower convergence, then even it has other desirable invariance and geometric properties, the slower convergence implies more estimation error, thereby decreasing the accurucy of the confidence estimates.
o-Holder continuous and let (x,y)
= 1]} & 0 when the probabilities are estimated the training data, for large enough n, we can bound the estimation error as
n n (20+ min (p(x),p(<lu)) (min)(()))p(/))))
pmi(a:y) y) pmrin 5
3.3. Theoretical Takeaways
We present a summary of the key takeaways from the the- oretical results and their implications for our subsequent experiments. These takeaways will also be referenced in our discussion of the experimental results later.
In the following result, we provide convergence rate for PSI, when the KDE approach (Jiang, 2017) is used to estimate p(0"a) 1:) and p(0 z/y))
Proposition 5 (PSI convergence rate). Let P(O X) be a-Holder continuous for all 0 and let (z,y) ~ Pxy where X= E Rd and Y E {0, 13. Let psin. represent the KDE estimate of PSI using n samples and m projec- tions. Furthermore, let ming pmi((" v;w) 2 p. Assuming min {P(Y= 0), P(Y = =}} * O when the probabilities are estimated on the training data, for large enough n, we can bound the estimation error as
Ti We find that the different PI measures each have their own strengths and weaknesses, with no single measure out- performing the others across all scenarios.
T2 Invariance: PMI is the most invariant among the three, as it exhibits invariance to any homeomorphic transforma- tion, and thus is the most structure-preserving However, we note (in Remark 2) that this may not be a boon in the context of confidence estimation, as the model's constraints matter significantly. PSI, on the other hand, is not invari- ant to general invertible linear transformations, which can hinder performance as neural networks can preserve output function in response invertible linear transformation on the input, and thereby preserve the confidence as well. Thus, since PVI is invariant to linear invertible transformations but not to homeomorphic transformations, it appears to be the most suitable measure for confidence estimation with respect to invariance properties.
Ex,y [[(=))) - -mn,
1-p P
S
2/m
n a22a-)
+ +
(10)
ming min (p(0Tx)) 2), p(0 x /)}
For PVI, we have the following bound on the expected deviation of the PVI estimates.
Theorem (PVI convergence rate). Given (z, v) Pxy where X E Ed and Y E 10, we assume 0.5. Assume V represents the set of all possible functions modelled by a neural network having some fixed architecture. As- sume YS E V, log [(x(()))) E 1-B, B). Also, let I" = arg miy Ex log [[x](Y)) represent the ground truth function for estimating conditional V-entropy. and f represent the trained function given n datapoints (x1, V.),.., (In; sampled from pžy- Let M max var vf [x][x)), var(f(z)())) where var denotes the variance. Let pvi, represent the PVI estimated using this neural network with n samples. Then, for any 8 E (0, 0.5), with probability p 221 - 28, we have
T3 Convergence Rates: When comparing PMI and PSI, our theoretical results concretely find that PSI is likely to have better convergence behaviour compared to PMI, fol- lowing the differences in the order of the sample complexity n. We find that PVT's convergence rate depends on the com- plexity of the predictive family V. thereby making direct comparison with PSI and PMI more challenging. However, in absolute terms, we find that the convergence of PMI and PSI also depends on the spread of the distribution P(x) and the amount of overlap between the class-wise distribu- tions P(zly = 1) and P(rly = 0), primarily due to the denominators in (9) and (10). In complex datasets, these distributions are typically broader and exhibit greater over- lap, resulting in slower convergence. In contrast, for simpler, more separable datasets such as MNIST, PSI and PMI con- verge more quickly, as reflected in PMI outperforming the other methods on MNIST for failure prediction (Table l). T4 Margin Sensitivity: We provide a detailed analysis of the three PI measures with respect to their sensitivity to the classifier's 's sample wise margin in Appendix B.3. Ideally, samples closer to the decision boundary should be assigned lower confidence scores. Overall, our results point to PSI
that P(Y = 0)
P(Y 1)
Ex, |pui(r + v) - poin ||
< (R, + +2
2 210g(1/8)
+ 2B
(11)
n
where the function family Gv = log f()())) f E V) and Rr denotes the Rademacher com- plexity with n sampled points.
(olø(z, v)
5

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
potentially being the most sensitive to sample-wise margin (hard and soft), although we found direct comparisons are not very straightforward. However, we do note that PMI is invariant to hard margin (when the classes are clearly sepa- rated), whereas PSI remains sensitive to it. Empirically, we found that PSI exhibits the strongest correlation with margin, as shown in Table 4. However, since PSI performs poorly in failure prediction experiments, this raises an important observation: margin sensitivity alone may not be a sufficient criterion for identifying good confidence measures.
the objective is identify incorrect predictions made by trained networks. Ideally, confidence scores should be high for correct predictions and low for incorrect ones. In selec- tive prediction, the aim is to evaluate the improvement in classification performance after excluding a certain percent- age of low-confidence predictions.
Methodology: For misclassification detection, we evaluate the effectiveness of confidence estimates in distinguishing between positive (incorrect predictions) and negative (cor- rect predictions) samples. We use a threshooldinndepeddent metric, AUROC, (Area Under the ROC Curve, with f de- noting failure), which is widely adopted in the literature (Hendrycks & Gimpel, 2017: Jaeger et al., 2023). Since AUROC is less informative when the positive and neg- ative classes have significantly different base rates, we also consider another metric called AUPR (Area Under the Precision-Recall Curve). Given that the base rate of the positive class greatly influences AUPR, we examine both scenarios: treating success classes as positive samples (AUPR y. success) and treating error classes as positive sam- ples (AUPR,, f, , We also compute the FPR95, which measures the false positive rate when the true positive rate is 95%. For selective prediction, we examine the im- provement in classification error rates by filtering out low- confidence samples. In this context, we define risk as the error rate on the remaining samples, and coverage as the proportion of remaining samples relative to the total sam- ples. We employ a wrrshold--inependeer metric, E-AURC (Area Under the Risk Gisk-Coverage Curve), as described in the literature (Geifman al., 2019; Jaeger et al., 2023). We compare our results against six benchmark methods: max- imum softmax probability (MSP) (Geifman & El-Yaniv, 2017), softmax margin (SM) (Tagasovska & Lopez-Paz, 2019), max logit (ML) (Hendrycks et al., 2022), logits mar- gin (LM) (Streeter, 2018), negative entropy (NE) (Belghazi & Lopez-Paz, 2021) and negative Gini index (NG) (Granese et al., 2021). We report the results in Table 1. Note that we omit PMI computation for datasets with very large number of classes (CIFAR-100) Stanford Dogs, and TinylmageNet) due to its high computational cost.
4. Experiments
We performed two types of experiments related to confi- dence estimation: (1) failure prediction and (2) confidence calibration. In all experiments, the PI measures are trained with true labels of the training dataset and evaluated with predicted labels of the test dataset. We also normalize these PI measures with softmax function to ensure their values lie between O and 1. For all experiments. we calibrate all methods using temperature scaling (including the PI mea- sures), except for max logits and logits margin since their values are not in the [0.1] range, to ensure fair comparison. All experiments are conducted using benchmark datasets and architectures readily available in TensorFlow. More details on the datasets, architectures and training algorithms used in all experiments are provided in Appendix C.
For PVI, we compute it between the input features and the predicted labels, following the approach from in (Ethayarajh et al., 2022), which is to estimate the PVI between X and Y by training another model with the same architecture. It measures how easily we can predict Y from X using V. Thus, it can capture the confidence of the predictive family V. While the way PVI is defined is architecture- dependent, the definitions of PMI and PSI are not. For PMI and PSI, it is more natural to use the features of the model directly and the layers closest to the output should capture the model's confidence about the network the most. For PMI and PSI, we compute them between the output layer features and the predicted labels. Furthermore, instead of computing the measures with just the predicted class, we compute them for all classes and apply softmax function along with temperature scaling. More discussion on this can be found in Appendix D.I. In this way, the Pl values are normalized to a range between 0 and 1.
We observe that PVI generally outperforms the other two PI measures, as well as other benchmark post-hoc methods, across a range of metrics. The performance gains are particularly pronounced for AUPR f. failure FPR95,, and AURC. A higher AUPR, f, failure and a lower FPR95 , indicate that PVI can reliably assign lower confidence to misclassi- fied examples, while a lower E-AURC indicates that PVI can reliably defer uncertain predictions. The superior per- formance of PVI is especially true for larger models and datasets. We argue that this is due to PVI being generally well-rounded in terms of its theoretical properties for confi- dence estimation, discussed in Section 3.3.
Results:
4.1. Failure Prediction
Goal: The goal of this experiment is to compare the effec- tiveness of different confidence estimates for failure pre- diction. Failure prediction typically involves three tasks: misclassification detection, selective prediction, and out- of-distribution detection (Jaeger et al., 2023). This work focuses on the first two tasks, In misclassification detection,
6

---

Pointwise Information Measures as Confidence Estimators in Deep Neural Networks
Table Comparison Confidence Estimation Methods for Failure Prediction (Averaged over 10 Runs).
Model, Dataset
Method
AUROC, x 10"
AUPR
X 10"
AUPR x 10"
FPR95, x 10"
E-AURC x 10
MSP
96.80 0.17
43.17 2.49
0.69
96.75 0.17
99.93
0.70
ML.
90.89 = 0.32
99.78 0.01
28.85 1.49
40.67 1.44
2.10
LM
96.67 = 0.18
99.93
39,40 2 1.89
16.87 = 1.66
0,71 0.01
MLP, MINSTT
NE
96.76 = 0.17
99.93 0.01
42.41 * 2.02
16.09 # 1.65
0.70 0.05
NG
9
=
43.31 2.30
16.20 1.64
0,69 = 0.05
PMI
97.13 0.14
99.94 0.00
45.94 + 2.25
14.82 1.39
0.61 0.04
PSI
94.77 0.46
99.88 0.01
31.01 1.99
27.89 1.54
1.19 0.14
96.94 0.22
99.93
48.28 3.20
15.59 1.46
0.66 0.05
MSP
92.53 0.17
99.39 =
44.71
46.94 1.73
5.72 0.229
SM
9 0.17
99.39 = 0.03
42.77 + 1.50
47.62 = 1.66
5,80 + 0.29
MI.
86.76 0.50
98.85 + 0.08
32.38 a 1.01
63.26 1.73
10.91 0.76
LM
92.29
99.37
42.12 1.52
48.21 1.67
5,91 = 0.29
CNN, F-MNIST
NE
92.40 0.17
99.38 0.03
43.38 0.93 0.93
48.85 1,87
5.82 0.31
NG
92.52 0.16
99.39 0.03
44.51 1.04
46.30 1.64
5.73 0.29
PMI
91.38
99.26 0
42.37 1.62
48.95 1.90
6,99 0.30
89.95 0.38
0.07
+ 0.96
56.76
PVI
92.98 0.52
99.43
50.01 4.22
43.26
= 0.33
MSP
92.04
0.05
50.75 + 1.24
48.21 1.69
7,93 + 0.48
SM
91.97 = 0.25
0.05
48.61 # 1.64
49.12 1.95
7.97 + 0.48
ML
89.50 0.40
0,07
45.70 1.44
53.52 1.30
10.94 0.67
LM
91.82 0.25
99.13 0.05
47.51 1.64
50.81 1.77
8.11 0.48
VGG16, STL-10
NE
91.77
0.06
48.95 1.04
49,93 1.48
8.19 0.50
NG
92.01
99.14
48.04
7.95 0.49
91.46 0.37
99.07
49.12
PSI
91.79 = 0.32
99.11
50.78 1.74
47.70 1.56
8.29 + 0.55
PVI
92.79 = 0.62
99.23 0.06
$5.88 + 3.50
41.97 = 3.61
7,18 + 0.53
MSP
93.59 0.24
99.63 0.02
43.42 1.20
36,29 = 1.23
3.52 0.22
SM
93.71 0.24
99.64 0.02
43.05 1.23
35.91 1.10
3.44 0.22
ML.
91.80 0.30
99.52 0.02
37.54 1.13
45.35 1.38
4,58 0.23
LM
0.23
0.02
42.74 1.23
35.85 1.23
3,38 0.21
ResNet50, CIFAR-10
NE
93.29 0.23 0.23
0.02
38.57 1.64
3,67
NG
93.55 0.24
0.02
42.44 1.17
36.56 1.35
3.54
PMI
93.11 = 0.30
9.00
43.51 1.53
37.43 1.71
3,87 2 0.23
PSI
93.51 = 0.22
99.63 0.02
43.27 2: 1.34
37.44 1.64
3.57 + 0.19
PVI
95.14 = 0.30
99.72 + 0.02
60.99 à: 1.60
25.77 2 1.56
2,72 A 0.17
MSP
85.54 0.31
95.25 = 0.15
60.83 0.82
39.17 1.20
SM
82.48 = 85.92
+ 95.41
60.78 0.62
0.514
37.85 1.14
$7.20
65.98
1.78
86.27
95.65 +
59.40 0.55
6.92
35.89 1.07
ResNet101, CIFAR-100
NE
84.33 0.34
94.88 = 95.20 = 0.15
57.57 0.87
66.75 1.11
42.32 1.29
NG
8.33
59.75 + 0.86
64.61 1.15 1.15
39.62 1.21
PSI
84.30 = 0.26
94.72 + 0.15
$7.40 + 1.01
66.44 # 0.76
43.57 + 1.11
PVI
89.02 = 0.38
96.27 0.11
71.54 + 1.42
47.17 1.73
30.44 0.85
MSP
83.52 0.34
93.80
59.85 0.71
66.72 1.03
50.16
SM
83.63
$8.58 0.70
2
81.18 0.34
92.71
$6.87
0.93
29.16
83.51 = 0.44
94.05
56.73
70.46 1.44
2.40
Inception V3, Stanford Dogs
NE
82.58 = 0.32
93.48 0.32
$7.49
69.14
$2.91 + 2.32
NG
83.38 = 0.33
93.76 = 0.31
67.48 + 0.87
50.50 :: 2.30
PSI
81.80 0.25
93.32 = 0.23
$4.36 0.86
73.73 0.83
54.41 # 1.67
1.67
PVI
85.73 0.73
94.50 0.25
66.59 2,30
$7.05 2.60
44.25 2.20
MSP
87.01 0.34
94.28 0.18
70.87 0.83
58.29 1.21
43.92 1.36
SM
86.95 +
94.35
0.88
43.50 1.38
ML
84.37 0.45
92.58 -
67.80 0.89
61.42 1.21
56.97
LM
86,50 = 0.35
94.37 0.18
66.60 0.83
65.44 1.33
43.58 1.38
DenseNet121, TinylmageNet
NE
86.02 = 0.37
93.86 + 0.21
68.41 4 0.86
61.82 1.19
47.34 1.56
NG
86.87 = 0.34
94.24 + 0.18
70.23 4 0.83
59.28 1.12
44.29 1.36
PSI
84,78 = 0.42
93.18 = 0.23
64.73 + 0.83
67.42 1.45
52.65 1.75
PVI
89.66 0.27
95.32 0.14
77.82 0.69
46.89 0.90
35.59 1.10

---

Pointwise Information Measures as Confidence Estimators in Deep Neural Networks
Table 2: Comparison Confidence Estimation Methods for Confidence Calibration (Averaged over 10 Runs).
Model, Dutaset
Method
SCE
CC-SCE 4
Ada-SCE L
RMS-SCE
MSP
0.06 + 0.01
0.18 + 0.01
0.01 # 0.00
0.44 0.09
PMI
0.05 0.01
0.17 = 0.01
0.01 0.00
0.42 # 0.07
MLP, MNIST
PSI
0.30 0.02
0.02
0.03 = 0.00
0.75
PVI
0.05
0.18
0.01 = 0.00
0.10
MSP
0.35 0.02
0.07 0.01
0.52
PMI
0.58 0.22
0.06 + 0.02
0.11
CNN, F-MNIST
PSI
0.57 0.15
0.05
0.08 = 0.02
0.52
PVT
0.34 0.04
0.07 0.01
0.50 0.06
MSP
0.06
0.59 = 0.03
=
0.14
PMI
0.03
0.65 0.04
0.00
0.82 0.10
VGGI6, STL-10
PSI
0.05
0.67 0.05
0.04
0.11
PVI
0.03
0.61 + 0.04
=.02
0.68 # 0.09
MSP
0.09
0.35 + 0.02
=
0.88 0.07
PMI
0.05
0.01 0.09
0.74 0.08
ResNet50, CIFAR-10
PSI
0.35 0.17
0.01
0.85 $ 0.08
PVT
0.17
0.33 0.00
0.03 2 0.00
0.85 0.01
MSP
0.08 0.000
0.20
0.04
0.68 = 0.04
ResNet101, CIFAR-100
PSI
0.07 = 0.01
0.31 + 0.01
0.02 = 0.00
0.49 = 0.05
PVI
0.08 + 0.00
0.20 + 0.00
0.04 + 0.00
0.68 0.04
MSP
0.20 0.00
0.03
0.63 = 0.03
InceptionV3, Stanford Dogs
PSI
0.05 # 0.01
0.28 + 0.01
0.02 # 0.00
0.47 + 0.07
PVT
0.07 0.00
0.20 + 0.01
0.03 0.00
0.63 0.03
MSP
0.03 0.00
0.12 + 0.00
0.01 # 0.00
0.32 0.03
1enseNet121, TinyImageNet
PSI
0.03 0.00
0. + 0.00
0.02 + 0.00
0.29 + 0.03
PVI
0.02 0.00
0.12 = 0.00
0.01 = 0.00
0.29 0.02
4.2. Confidence Calibration
We also include the results for other calibration metrics in Table 8 in the Appendix. Similar to the failure prediction experiment, we omit PMI computation for datasets with very large number of classes (CIFAR-100, Stanford Dogs, and ImnylmageNet) due to its high computational cost.
Goal: The goal of confidence calibration is determine whether the confidence scores reflect the true correctness likelihood (Guo et al., 2017). Perfect calibration is defined as follows: P(Ý- Y|P p) = P. Vp E [o, 1]. where Y denotes the ground-truth labels, Y denotes the predicted labels and P is the associated probability.
The results in Table 2 show that MSP and PVI consistently perform well across most datasets and architec- tures, reinforcing the observation in (Guo al., 2017) that simple temperature scaling can already be highly effective for confidence calibration. PVI, in particular, often matches or slightly outperforms MSP, especially in SCE and RMS- SCE, but the differences are typically small. PMI and PSI each show sporadic strengths specific metrics, suggest- ing they may offer situational advantages depending on the dataset and calibration criterion. Overall, we find that PVI demonstrates more consistent calibration than MSP across all four metrics, with notable cases such as the substantially lower RMS- SCEE for the VGG16-STL-ID pair.
Results:
Expected Calibration Error (ECE), a widely used calibration metric, has known limitations: relies on fixed length binning approach that introduces birinneri trade-off's (fewer bins increase bias, more bins escalate vari- ance), considers only the maximum predicted probabili- ties (ignoring potentially meaningful information in lower- confidence estimates), and may mislead by yielding low calibration error even in poorly accurate models (Pavlovic, 2025: Nixon et al., 2019). Instead, we use the variants pro- posed in (Nixon al., 2019). Static Calibration Error (SCE) is the multiclass extension of ECE that evaluates calibration over all predicted class probabilities, not just the top pre- SCE (CC-SCE) computes SCE separately for each class (binning and accuracy calculation per class), then averages the errors across classes. Adap- tive SCE (Ada-SCE) uses adaptive (quantile-based) binning so that each bin contains approximately the same number of predictions, improving stability in accuracy estimates. RMS-SCE replaces the 1.1 norm in the error term with the L2 norm before averaging, thereby penalizing larger miscal- ibrations more strongly. The results are shown in Table 2.
Methodology:
diction.
5. Reflections
We performed comparative analysis of using three mea- sures, namely PMI, PVI, and PSI, for confidence estimation in DNNs. We studied several thcoretical properties which we believe can be relevant to confidence estimation, in- cluding how well a measure behaves in response to data transformations (invariance properties), how well a mea- sure converges with data (convergence rates) and how well
8

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
measure tracks the geometric difficulty of classifying feature point (sample-wise margin). We performed series of experiments on confidence estimation (failure prediction and confidence calibration) to test and verify our theoretical hypothesis. Our findings show that PVI outperforms PMI, PSI, and benchmark post-hoe methods in failure prediction, while remaining competitive confidence calibration. This highlights PVI's versatility, especially given that popular confidence calibration methods have been shown to be inef- fective or even detrimental for failure prediction tasks (Zhu et al., 2022). This is consistent with our theoretical find- ings which suggest that PVI is generally well-rounded when considering the three theoretical properties analyzed here.
tors for Pf measures are less common compared to their aggregate counterparts, our work, which clearly demon- strates their applications in confidence estimation, could motivate further research towards more accurate and ef- ficient estimation of these measures. In addition, the PI measures are the optimal choice of explainability if we as- sume the probabilit--raiinng based causal model for the problem. Exploring more general structural causal models from inforoition-theoretic perspective in the context of explainability is an interesting avenue for future work.
Impact Statement
The use of PI measures for confidence estimation in deep learning models serves as an additional safeguard against wrong predictions before deploying them in real-world ap- plications. Our approach could be combined with explain- ability methods to ensure consistency of results. Caution should be taken when interpreting these results when they are used for critical decision-making, such as in being aware of any probabilistic biases in the data that may yield unfair outcomes. Finally, these measures can be used ex- plore other aspects of trustworthy Al including adversarial robustness and privacy.
One of our findings in this work has been that better sensitiv- ity to margin does not necessarily imply better performance in the confidence prediction problem. We note that for the correlation to margin experiment (Table 4), the focus is on whether the model assigns higher confidence to sam- ples with larger margin (and vice versa), regardless of whether the prediction is correct. On the other hand, for the misclassification detection, selective prediction, and cal- ibration analysis, the focus is more on the correctness of predictions (directly linked to accuracy). The contrast lies in the interpretation of confidence: margin experiments treat confidence as a measure of sensitivity to decision bound- aries, while the other tasks treat it as measure of predictive reliability. Therefore, our results highlight that the two in- terpretations of confidence may not be consistent with each other, i.e., sample-we wise margins may not be reflective of true confidence.
Acknowledgements
This research/project is supported by A STTTR,, CISCO Sys- tems (USA) Pte. Ltd and National University of Singapore under its Cisco-NUS Accelerated Digital Economy Corpo- rate Laboratory (Award (100110002)
For future work, these PI measures could be explored for confidence estimation in other modalities (e.g., image, audio, tabular, etc.). In addition, one could explore the potential of using PI measures for other aspects of trustworthy machine learning such as explainability and privacy. Furthermore, we could study other scaling and normalization techniques to improve the performance of the PI measures. We hope that both our theoretical and empirical findings will moti- vate more work in the direction of informatiiontheoretie ap- proaches for confidence estimation in the context of DNNs.
References
Abdar, M., Pourpanah, F., Hussain, S., et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 2021.
Ahanin, Z. and Ismail, M. A. A multi-label emoji classification method using balanced pointwise mutual information- feature selection. Computer Speech and Language, 2022.
Another promising direction for future work is to integrate PVI into existing confidence estimation methods (not nec- essarily post-hoc) that rely on logit-based softmax outputs for generating confidence scores, to evaluate whether it can further improve performance. This includes popular tech- niques such as Monte Carlo dropout, deep ensembles, focal loss, and various temperature scaling methods. The inte- gration is straightforward: replace the logit-based softmax outputs with softmax-scales PVI values when computing confidence scores.
Bandhakavi, A., Wiratunga, N., Padmanabhan, D., et al. Lexicon-based feature extraction for emotion text classi- fication. Pattern Recognition Letters, 2017.
Battiti, R. in supervised neural net learning. IEEE Trans. Neural Networks, 1994.
Using mutual information for selecting features
Belghazi, M. I. and Lopez-Paz, D. What classifiers know what they don't? CoRR, 2021.
Limitations. Our Pl measures require training additional models to learn the probability distribution. Since estima-
Bouma, G. Normalized (pointwise) mutual information in collocation extraction. Proceedings of GSCL, 2009.

---

Pointwise Information Measures as Confidence Estimators in Deep Neural Networks
Chen, Y., Shen, Y., and Zheng. S. Truthful data acquisition via peer prediction. In Advances in Neural Information Processing Systems, NeurIPS, 2020.
Geifman, Y. and El-Yaniv, Selective classification for deep neural networks. In Advances neural information processing systems, NeurIPS, 2017.
Church, K. and Hanks, P. Word association norms, mutual information, and lexicography. Computational linguistics, 1990.
Geifman, Y., Uziel, G., and EI- El-Yaniv, R. Bias-reduced un- certainty estimation for deep neural classifiers. In Inter- national Conference on Learning Representations, ICLR, 2019.
Corbière, C., Thome, N., Bar- Bar-Hen, A., al. Addressing fail- ure prediction by learning model confidence. In Advances in Neural Information Processing Systems, NeurIPS, 2019.
Goldfeld, 7. and Greenewald, K. H. Sliced mutual infor- mation: A scalable measure of statistical dependence. In Advances in Neural Information Processing Systems, NeurIPS, 2021.
Cover, T. M. and Thomas, J. A. Elements of Information Theory. Wiley, 2001.
Granese, F., Romanelli, M., Gorla, D., et al. DOCTOR: A simple method for detecting misclassification errors. Advances in Neural Information Processing Systems, NeurIPS, 2021.
Dagan, L, Marcus, S., and Markovitch, S. Contextual word similarity and estimation from sparse data. In s1 Annual Meeting of the Association for Computational Linguistics, 1993.
Grønlund, A., Kamma, L., and Larsen, K. G. Near-tight margin-based generalization bounds for support vector machines. In International Conference on Machine Learning, pp. 3779-3788, PMLR, 2020.
DeVries, T. and Taylor, G. W. Learning confidence for -iitt--distiitttion detection in neural networks. CoRR, 2018.
Elsayed, G. F., Krishnan, D., Mobahi, H., et al. Large margin deep networks for classification. In Advances in Neural Information Processing Systems, NeurIPS, 2018.
Guo, C., Pleiss, G., Sun, Y., et al. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML, 2017.
Ethayarajh, K., Choi, Y., and Swayamdipta, S. Under- standing dataset difficulty with V-usable information. International Conference on Machine Learning, ICML, 2022.
He, W. and Jiang. Z. A survey on uncertainty quantification methods for deep neural networks: An uncertainty source perspective. CoRR, 2023.
Fano, R. M. and Hawkins, D. Transmission information: statistical theory of communications. American Journal of Physics, 196la.
Hendrycks, D. and Gimpel, K. baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Rep- resentations, ICLR, 2017.
Fano, R. M. and Hawkins, D. Transmission information: statistical theory of communications. American Journal of Physies, 29(1 (1191)):73779) 1961b.
Hendrycks, D., Basart, S., Mazeika, M., et al. Scaling out-of-distribution detection for real-world settings. In International Conference on Machine Learning, ICML, 2022.
Feng, L., Ahmed, M. O., Hajimirsadeghi, H., et al. Towards better selective classification. International Conference on Learning Representations, ICLR, 2023.
Hitchcock, C. Probabilistic causation. 1997.
Gal, Y. and Ghahramani, Z. Dropout bayesian approxi- mation: Representing model uncertainty deep learning. In Proceedings of the 33nd International Conference on Machine Learning, JCML., 2016.
Isola, P., Zoran, D., Krishnan, D., et al. Crisp boundary de- tection using pointwise mutual information. In European Conference on Computer Vision, ECCV, 2014.
Galil, I., Dabbah, M., and El-Yan Yaniv, R. What can we learn from the selective prediction and uncertainty estimation performance 523 imagenet classifiers? International Conference on Learning Representations, ICLR, 2023.
Jaeger, P. F., Lüth, C. T., Klein, L.., et al. A call to reflect on evaluation practices for failure detection in image classification. International Conference on Learning Representations, ICLR, 2023.
Gawlikowski, J., Tassi, C. R. N., Ali, M., et al. survey of uncertainty in deep neural networks. Artificial Intelli- gence Review, 2023.
Jiang, H. Uniform convergence rates for kernel density esti- mation. International Conference on Machine Learn ing, ICML., 2017.
10

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Jiang, H., Kim, B., Guan, M. Y., al. To trust or trust A classifier. In Advances in Neural Information Processing Systems, NeurIPS, 2018.
Nixon, J., Dusenberry, M. W., Zhang, L.,, al. Measuring calibration in deep learning. In IEEE Conference Com- puter Vision and Pattern Recognition CVPR Workshops, 2019.
Kaur, D., Uslu, S., Rittichier, K. J., et al. Trustworthy artificial intelligence: A review. ACM Computing Surveys (CSUR), 2023.
Oldham, K. B., Myland, J. C., and Spanier, J. The incom- plete beta function b (v, μ., x). In An Atlas of Functions, pp. 603-609, Springer, 2008.
Kraskov, A., Stögbauer, H., and Grassberger, P. Estimating mutual information. 2004.
Padmakumar, V. and He, H. Unsupervised extractive sum- marization using pointwise mutual information. In Con- ference of the European Chapter of the Association for Computational Linguistics, EACL, 2021.
Kulmizev, A. and Nivre, J. Investigating UD treebanks via dataset difficulty measures. In Conference of the European Chapter of the Association for Computational Linguistics, EACL,, 2023.
Pavlovic, M. Understanding model calibration - A gentle introduction and visual exploration of calibration and the expected calibration error (ECE). ICLR Blogposts, 2025.
Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of out-of oistiistibuution image detection neural networks. In International Conference on Learning Representations, ICLR, 2018
Poole, B., Ozair, S., van den Oord, A., et al. On varia- tional bounds mutual information. In International Conference on Machine Learning, ICML, 2019.
Lin, T., Papangelis, A., Kim, S., et al. Selective in-context data augmentation for intent detection using pointwise v-information. In Conference of the European Chapter of the Association for Computational Linguistics, EACL, 2023.
Prasad, A., Saha, S., Zhou, X., et al. Receval: Evaluating reasoning chains via correctness and informativeness. In Conference on Empirical Methods in Natural Language Processing, EMNLP, 2023.
Littwin, E. and Wolf, L. Regularizing by the variance of the activations' sample-variances. Advances in Neural Information Processing Systems, 31, 2018.
Ren, L., Sidhu, M., Zeng, Q., et al. C-PMI: conditional pointwise mutual information for turn-level dialogue eval- uation. 2023.
Lu, S., Chen, S., Li, Y., et al. Measuring pointwise V-usable information in-context-ly. CoRR, 2023.
Sintsova, V. and Pu, Dystemo: Distant supervision method for multi-category emotion recognition in tweets. ACM Transactions Intelligent Systems and Technology, (TIST), 2016.
Luo, X., Liu, Z. Shang, M., et al. Highly-accurate community detection via pointwise mutual information- incorporated symmetric non-negative matrix factorization. IEFE Transactions on Network Science and Engineering, 2021.
Streeter, M. Approximation algorithms for cascading pre- diction models. In International Conference on Machine Learning, ICML, 2018.
Mena, J., Pujol, O., and Vitrià, A survey on uncertainty estimation in deep learning classification systems from a bayesian perspective. ACM Computing Surveys (CSUR), 2022.
Su, Q. Xiang, K.,, Wang, H., et al. Using pointwise mu- tual information to identify implicit features in customer reviews. In International Conference on Computer Pro- cessing of Oriental Languages, ICCPOL, 2006.
Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P. H., and Gal, Y. Deep deterministic uncertainty: A new simple bascline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24384 24394, 2023.
Tagasovska, N. and Lopez-Paz, D. Single-model uncertain- ties for deep learning. In Advances in Neural Information Processing Systems, NeurIPS, 2019.
Tao, L., Zhu, Y., Guo, H., et al. A benchmark study on calibration. In International Conference on Learning Representations, ICLR, 2024.
Nandwani, Y., Kumar, V., Raghu, D., et al. Pointwise mu- tual information based metric and decoding strategy for faithful generation in document grounded dialogs. In Conference on Empirical Methods in Natural Language Processing, EMNLP. 2023.
Tsai, Y. H., Zhao H., Yamada, M., et al. Neural methods for point-wis wise dependency estimation. In Advances in Neural Information Processing Systems, NeurIPS, 2020.

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Turney, P. D. Mining the web for synonyms: PMI-IR versus LSA TOEFL. In European Conference Machine Learning, EMCL, Lecture Notes in Computer Science, 2001.
Vemuri, N. Scoring confidence in neural networks. Techni- cal Report NCB/EECCS2020-1 1322, University of Califor- nia at Berkeley, 2020.
Wongso, S., Ghosh, R., and Motani, M. Using sliced mutual information to study memorization and generalization in deep neural networks. In International Conference on Artificial Intelligence and Statistics, AISTATS, 2023a.
Wongso, S., Ghosh, R., and Motani, M. Pointwise sliced mutual information for neural network explainability, In IEEE International Symposium on Information Theory, ISIT, 2023b.
Xu, Y., Zhao, S., Song, J., et al. A theory of usable infor- mation under computational constraints. In International Conference on Learning Representations, ICLR, 2020.
Zheng, S., Kwon, Y., Qi, X., et al. Truthful dataset valuation by pointwise mutual information. CoRR, 2024.
Zhu, F. Cheng, Z., Zhang, X., et al. Rethinking confidence calibration for failure prediction. In European Conference on Computer Vision, ECCV, 2022.
12

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Supplementary Materials
To allow ease of access and improve readability, we include the table of contents for our supplementary materials.
Table of Contents
Related Work & Information Measure Details
A.1
Related Work
A.2
General Properties of Information Measures and Their Pointwise Variants
A.2.1.
General Properties of MI and PMI
A.2.2.
General Properties of SMI and PSI
A.2.3.
General Properties of VI and PVI
A.3
Pointwise Information Measures
A.3.I. PMI Estimators
A.3.2. PSI Estimators
A.3.3. PVI Estimators
B Theoretical Analysis & Proofs
B.1 Invariance Properties
B.2 Convergence Rates
B.3 Geometric Properties
C Benchmarks & Experimental Details
C.I Benchmark Datasets & Architectures
C.2 Hyperparameters
C.3 Details for Experiments Main Paper
C.3.1. Details for Experiment in Section 4.1 (Failure Prediction)
C.3.2. Details for Experiment in Section 4.2 (Confidence Calibration)
C.4 Additional Results for Experiments in Main Paper
C.4.1. Additional Results for Experiment in Section 4.2 (Confidence Calibration)
Additional Experiments
D.I Normalization: Effects of Softmax and Temperature Scaling
D.2 Comparison of Various Pointwise Information Estimators
D.2.1. Comparison of PMI Estimators
D.2.1. Comparison of PSI Estimators
D.2.1. Comparison of PVI Estimators
13

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
A. Related Work & Information Measures Details
A.1. Related Work
Pointwise Mutual Information (PMI). PMI compares the probability of two outcomes occurring together to what the probability would be if they are independent. It is commonly in natural language processing to measure the association between words (Church & Hanks, 1990; Tumney, 2001; Su et al., 2006; Padmakumar & He, 2021). In this setting, p(r)) and p(ar, y) can be obtained by counting the occurrences and co-occurrences of words in the corpus. However, PMI can be sensitive to the size of the corpus and may not perform well with very rare words or when the data is sparse. Other variants of PMI have also been introduced, including positive PMI measure, which sets all the negative values to zero (Dagan et al., 1993), and normalized PMI measure, which scales the values to fall within the range [-1,1] (Bouma, 2009). PMI has found applications in a wide range of areas, including sentiment analysis (Ahanin & Ismail, 2022; Bandhakavi et al., 201 77; Sintsova & Pu, 2016), community detection (Luo et al., 2021), response generation (Nandwani et al., 2023; Ren et al., 2023), truthful data acquisition (Zheng et al., 2024; Chen et al., 2020), and boundary detection (Isola et al.. 2014). In this study, we use PMI to obtain both confidence scores and saliency maps for image classification tasks.
Pointwise Sliced Mutual Information (PSI). (Wongso et al., 2023b) introduced PSI as measure for generating confidence scores and saliency maps for deep neural networks. For confidence scores, they compute the PSI between features of the penultimate layer of a neural network and predicted label for each sample and refer to this as the sample-wise PSI. For saliency maps, they compute the PSI between feature fiber of the last convolutional layer of a convolutional neural network and predicted label for each sample and refer to this as the fiber-wise PSI. In addition, they show that PSI, contrast to PMI, exhibits sensitivity to sample-wise margin. Even though their findings demonstrate that PSI can produce sensible confidence scores and saliency maps the paper lacks a profound perspective and the essential quality assessment of PSI metric for model uncertainty and explainability. In this work, we provide more comprehensive evaluation of PSI, comparing it to the other two pointwise measures, namely PMI and PVI, to determine the relevance between features and predicted labels. Additionally, we present a set of theoretical results that explore various properties of pointwise information measures, providing deeper insights into what they may represent.
Pointwise V-Information (PVT). PVI was introduced to measure sample difficulty with respect to a given distribution (Ethayarajh et al., 2022). In their research, they investigate natural language inference tasks and observe that samples with high PVI are often predicted correctly, while those with low PVI are more likely to be predicted incorrectly. It is important to note that in their paper, they assess PVI in relation to the true label (also referred to as the gold label), making it measure of sample difficulty rather than a measure of the network's is confidence. They also show that PVI can be used to identify which subsets of each class are more difficult than others. PVI has recently been employed in a variety of NLP tasks (Lin et al., 2023; Lu et al., 2023; Prasad et al., 2023; Kulmizev & Nivre, 2023). In this study, we compute PVI to obtain both confidence scores and saliency maps for image classification tasks.
Predictive Confidence. The idea behind confidence estimation is closely connected to uncertainty quantification. Simply put, when we are more confident in a prediction made by a model, it means there is less uncertainty about that prediction. For comprehensive survey/review uncertainty quantification in deep learning, we refer the readers to (Gawlikowski et al., 2023; He & Jiang, 2023; Mena et al., 2022; Abdar et al., 2021). There are two common lines of work for evaluating predictive confidence: confidence ranking and confidence calibration. Works on confidence ranking focuses on ranking confidence scores such that the lower-ranked samples are more likely to misclassified while the higher-ranked samples are more likely to be correctly classified. Confidence ranking is useful in applications such as misclassification detection (Hendrycks & Gimpel, 2017: Jiang et al., 2018: Corbière et al., 2019; Jaeger et al., 2023), -ittr--diitribution detection (Hendrycks & Gimpel, 2017; DeVries & Taylor, 2018; Liang et al., 2018) and selective classification (Geifman & El- Yaniv, 2017: Feng et al., 2023; Galil et al., 2023). On the other hand, research on confidence calibration aims to provide confidence scores that accurately reflect the likelihood of a prediction being correct (Guo et al., 2017; Nixon et al., 2019; Tao et al., 2024). This requires the confidence scores to be probabilities within the range of 0 to 1.
A.2. General Properties of Information Measures and Their Pointwise Variants
In this section, we describe some general properties of mutual information (MI), V-information (VI), and Sliced Mutual Information (SMI, and their pointwise variants.
14

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
1. GENERAL PROPERTIES OF MI AND PMI
We first restate the definition of MI and PMI from the main paper:
Definition (MI and PMI). Let (X, Y) Pxy. The MI between X and Y
is:
P(X,Y)
T(X = Ex,y log P(X)P(Y)
(12)
Let (x,1) YX, The PMI of an instance (x, y) is:
p(r,3) y) p(x)p(y)
pmi(x; = log
(13)
MI satisfies the following properties:
Non-negativity: ((X 2
Independence: YY = = iff X and Y are independent.
3. Entropy decomposition: I(
HX) H(Y) H(X,Y)= H(X) H(XI)) H(Y) - H( where H()) and H--I-) are the entropy and conditional entropy respectively.
4. Chain rule: T(X, Y; 2) T(X; 2) Z(). More generally, for X1, Xn. we have T(X1, X,: Y)
1(K; - (X,; | (;-1))
Remark (Data processing inequality). MI also satisfies the data processing inequality which means that I(X;Y) >
I(f(X); for any d f. This is in contrast to VI and SMI which can grow with more processing of the random variables.
We list some properties of PMI as follows:
1. Range:
Continuous X and Y: -00 S pmi((;; y) 00.
Discrete Y: -00 & pmi(x;y) S log)(y)
Discrete X and Y: -00 pm min[- log logg(o), logr(v)]-
2. Independence: If X and Y are independent, then pmi( v) 0 V(z,8) E X x V. Note that pmi(æ;; y) for certain (). ~ Pxy does not imply X and Y are independent.
3. Entropy decomposition: pmi(x; y) = h(z)) (x,,)) ((x)-h(zl)) h(y) - -hyl where h(-) = logg(.)
called the seff-information.
Chain rule: pzri(z, yi 2) pmi(a; z) pmiily
A.2.2. GENERAL PROPERTIES OF SMI AND PSI
We first restate the definition of SMI and PSI from the main paper:
Definition 2 (SMI and PSI). Let (X,Y² Pxy E P(R4 Rd)). Let θ Unif(S and & Unif(S4, -1) be
independent of each other and of (x,Y) The SMI between X and is:
SI(X;Y) = Eace, E x; (0000"=""00) x;o"Y)] dE
(14)
Let (x.1) X,Y. The PSI of an instance (x, y) is:
psi(=;y) = Epce, [pmilo"
z; oJ y)]
(15)
oc#
SMI shares many similar properties with MI (Goldfeld & Greenewald, 2021, Proposition 1), including:
1. Non-negativity: SI(X;Y) 2 0.
Independence: SI(X; iff X and Y are independent.
Entropy decomposition: SI(X; Y) = (Y)) = SH(X)- -
where SH(-) and (-I-))) are the sliced entropy and conditional sliced entropy respectively.
15

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
4. Chain rule: SI(X,Y;2) ST(X1, Y) ST(X;; 2 "(x,"..
SI(X; + SIXY; More generally, for X1,'',, Xn. have
Remark 5 (SMI can grow with processing). We note that unlike MI and similar to VI, SMI can increase with more processing of the random variables, i.e., we can have SI(f(X);Y) 2 SI(X; Y) for any deterministic function f. (Goldfeld Greenewald, 2021) argued that this property is desirable in the context of machine learning, where it is more intuitive to think that processing input features yields representations that are more useful for inferring the labels.
We list some properties of PSI follows:
1. Range:
Continuous X and Y: -00 S psi(x;y) c0.
Discrete Y: -0o & psi(r;;) < - p(V).
Independence: and are independent, then psi(2; y) = 0, v(,, E X' x V. Note that psi(2; y) = for certain (w.y) ~ Pxy does not imply X and Y are independent.
Entropy decomposition: psi(x;y) y) sh(z) sh(y) sh(2, y) sh(x) sh(xly) sh(y) sh (z) where sh(x) == Eace log p(gp((")) x:) and sh(vrly)
- dee log log pl are the pointwise sliced entropy and point-
wise conditional sliced entropy respectively.
4. Chain rule: psi(r, z) psi(;;; 2) + (1;()===z))
A.2.3. GENERAL PROPERTIES OF VI AND PVI
We first provide more detailed definitions for predictive family and conditional V-entropy, and subsequently restate the definition of VI and PVI from the main paper.
Definition 4 (Predictive Family). Let Q (f XUO P()), The predictive family V C is defined such that it satisfies:
Aʃ E V, E range(J), EV, s... Vo E X, P.
(16)
Remark 6 (Optional ignorance). words, predictive family is a set of predictie models the agent can use, often limited by computational statistical constraints. The additional condition ['xx]= = which gives the agent option to ignore the side information x and still be able to predict get P. As shown in (Xu et al.,
( P is called the optional ignorance,
2020), this condition is necessary to obtain the desirable properties VI.
Definition 5 (Predictive Conditional V-entropy). Let (X,Y) ~ Pxv E P(X x y). We use Ø to represent a null input that provides no information about Y. Given a predictive familyy V, we can define the predictive conditional V-entropy as:
Hv(Y = inf E Ex log [[x](X))
(17)
FEV
Hy(Y = inf Ey log ly fY)
(18)
FeV
Hy())z) also called the V-entropy and denoted as Hv(YY (o(YY)ooo for simplicity. Definition 3 (VI and PVI). Let H Xy) and (X.Y) be defined as Def. 4 and Def. 5. We are given predictive family. Then the VI from X Y is:
lv(X Y) Hv(Y) X)
(19)
Let g = arv Eyy-llog (Y) and g' arg min jev Ex,y log (Y)] Given (z, v) ~ (X, Y), the PVI from x to y is:
pri(æ + +) = - log f( log ((((()
(20)
VI satisfies the following properties:
1. Non-negativity: Ix(X + Y)
2. Independence: Tx(X - Y)= = Ty(Y X)= 0 X and Y are independent.
3. Entropy decomposition: 1v(X +Y) -
Remark 7 (VI can grow with processing). We note that unlike MI and similar to SMI, VI can increase with more processing of the random variables, i.e., we can have (x(f(X)- Y) 2 Iy(X - Y)) any deterministic funcitoootiicuuncion f. (Xu et al., 2020) argued that this property is desirable in the context of machine learning, where it is more intuitive to think that processing input features yields more usable information about the label.
16

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Remark 8 (Asymmetry of VT). We also note that unlike MI and SMI, VI is asymmetric in nature which is align with the intuition that sometimes, it is easier predict Y from X than to predict X from Y.
We list some properties of PVI as follows:
1. Range:
Continuous X and Y: -00 S pei(r 50
Discrete Y: -00 < pui(x + y) S - logg(y) when Hy(Y) HYY). Note that this true when V represents
function modelled by neural network with trainable weights and biases.
2. Independence: If X and Y are independent, then we have pvi(æ - y) pvi(y x) = 0. Note that pvi(æ + y) = 0
for some (z, y) Pxy does not imply that X and Y are independent.
3. Entropy decomposition: pvi(x + y) = hy(v)= by (ylaz) where hv((y) is the pointwise V-entropy of y and (y(()z)
the pointwise conditional V-entropy of y.
A.3. Pointwise Information Estimators
In this section, we describe the estimators of PMI, PSI and PVI as well as provide the algorithms for each pointwise measure. We implemented these estimators in Python and use the Tensorflow library for neural networks.
1. PMI ESTIMATORS
In (Tsai et al., 2020), the authors proposed three different estimators for PMI: probabilistic classifier, density ratio fitting and variational
(JS) bound. All of these approaches estimate PMI using neural networks with distinct loss functions described below. We provide the pseudocode for the PMI estimator in Algorithm 1. Note that we presented the algorithm for any label but used predicted labels ý in our experiments.
Probabilistic Classifier (PC) Method. In this approach, we assign class to samples drawn from the joint density (c= = 1 for (xr,)) ) and class to samples drawn from the product of marginal densities (e = 0 for (x., 3) ~ Px Py). Thus, we can rewrite the density ratio as:
p v) pla,ylc-1) p(c - 0) p(c 1 v) ( pla,ylc- p(c= = 1) p(c 0|2, y)
(21)
where we have used Bayes' Theorem for the second equality. Furthermore, we can approximate the ratio of class prohabilities by the ratio of the sample size:
p(c = 0) 1P nPxx
7P Py
(22)
p TPxx /PxPY + Txxx
nPx
To approximate the class-posterior probabilities, we use neural network ʃ parameterized by 0 with the following binary cross-entropy loss function:
LPc(0) = EExx [log fa(c 1|(x, v))] Exx py [log(1 - folc = (x,,
(23)
For biti-batchaae samples, we can write the loss function as:
1 Σ llog fo = 1 Σ [oo(( folc- 1|(x(0) ( 70
(oc(
(24)
where (I. v) ~ PxY and ÿ ~ Py
(Tsai et al., 2020) also showed that when θ is large enough, the optimal fo(c/z, y) = p(c|2, v).
Density Ratio Fitting (DRF) Method. This approach seeks to minimize the expected least-square difference between the true density ratio and the density ratio estimated using neural network f parameterized by 0. By letting r(z, y) plæ, v) p(()), the objective function can be written as:
inf P - fe(x, sup Exxy [fe(z, y)] - - _B Py (z, v) lee
(25)
9e0
17

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Thus, the loss function is:
1 DDRF(() (0)= = -Bpxy v)1 Px (2(,,)))
(26)
For mini-batch samples, we can write the loss function as:
1 E (o(a ,00)) LIRF( (0) -
1 2h
Σ
(27)
i=1
where (x, ~ Pxy and y ~ Py.
(Tsai et al., 2020) also showed that when θ is large enough, the optimal fz y) r y) = p(x.vl p(=)p(s))
Variational Jensen-Shannon (JS) Bound Method. This approach relies on the variational form of MI, and in particular the -hasenn-hannon divergence between Pxy and Px (Poole et al., 2019). The Vensen-Shannel variational estimator is found to be more stable than the other proposed variational lower bounds. Similar to the density ratio fitting method, the density ratio is estimated using a neural network parameterized by The loss function can be written as:
Lis (0) Exxy (softplus( log (o + Epx (softplus(log fo v))
(28)
where softplus(z)) = log(1 + exp(x)).
For mini-batch samples, we can write the loss function as:
1 Σ /sottplus(oog fe(x()) (()] (i) b Σ softplus(- Horipluu(-loooo(e 900
(29)
where (x. y) ~ Pxy and ÿ ~ Py.
(Tsai et al., 2020) also showed that when θ is large enough, the optimal fo( v)
p(a.y)
p(z)p(y]
Critic Model Architectures. The neural networks used to estimate PMI are also commonly referred to critic models. In the literature, there are two common structures for the critic models: separable and joint. They primarily differ in how x and are considered in the neural network training. In separable critic design, and are being passed to two separate neural networks: hxx) and g(y)- The final model then computes the dot product between the outputs of the two neural networks: f(z, y) = h(x)² g(y). In joint critic design, z and are concatenated and fed as input to one neural network. In Appendix D.2.1, we compare the performance of the different critic architectures. We represent the neural network using a multi-layer perceptron, consisting of one hidden layer with 512 units and ReLUU activation function. For separable critic, the outputs of neural network (xx:) and g(y) have dimensions of 128, while for joint critic, the output has dimension of 1. They trained with Adam optimizer with learning rate of 0.001 for a maximum of 200 epochs. We employ early stopping if the maximum MI on the validation dataset fails to improve after 10 epochs. For the final PMI model, we use the one that yields the highest MI on the validation dataset.
Note on Implementation. We followed the implementation by (Tsai et al., 2020), adapting their original Py Torch code to Tensorflow. their implementation, rather than shuffling samples from the joint distribution obtain samples drawn from product of marginal densities, they manipulate the output of the critic model to have shape of b b, where b represents the batch size. To achieve this for the joint critic, they introduce new axis and replicate the input b times along that axis. Consequently, the diagonal elements naturally correspond to samples drawn from the joint density, while the off- off-diagonal elements represent the product of marginal densities. In this setup, there are b mini-batch joint samples and 62 - mini-batch batch marginal samples. When using the PC method, there is an additional term n Py //Rpxy which computes the ratio of samples from the different distributions. In line with this implementation, given the unequal number of samples from the different distributions, an additional term of log[(1² b)/b) log(b 1] must be included in the final PMI estimation.
18

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Algorithm 1 PMI Estimator
Require: (X", ~ Pxy E P x R) where E {i,.., k), a chosen pair of sample (æ, y) ~
critic
model f. and E' number of epochs to train the critic model.
initialize parameters of f
0
while E' do
Draw
samples from the joint density: (z(1), (()), (z(1), )00) ~ (x",y)
Draw
samples from the marginal density': 7(), g(o) Py
Compute the loss function L(O) on the mini-batch samples:
(Eq. (24) for PC, Eq. (27) for DRF, or Eq. (29) for variational JS bound)
Update the critic model parameters 0 based on L(0)
e e+11
end while
return pmni(;;)) ()) < [1x, u) for PC and variational JS bound or pmi((;; y) < log f (ar y) for DRF
For all our experiments, choose the variational IS bound (with separable critic) the default PMI estimator show in Appendix D.2.1, it yields the best performance.
A.3.2. PSI ESTIMATORS
We followed the implementation by (Wongso et al., 2023b) and considered an additional method: binning, We provide the pseudocode for the PSI estimator for the binning method in Algorithm and for the Gaussian method in Algorithm 3. Note that we presented the algorithm for any label y but used predicted labels in our experiments. For our problems, we only project X since Y is discrete. For both methods, we clip the probability to a minimum of 1e-5 to prevent division by zero. We did not consider kernel density and neural network estimation in this work due its high computational cost, which is not practical for confidence estimation.
Algorithm 2 PSI Estimator (Binning Method)
Require: (X", Y*) ~ Pxy e P(Rd x R) where Y E (1,... c) (c classes), a chosen pair of sample (r. v) ~ (X", y"), chosen number of slices (projections) m, and chosen number of bins Thins Initialize 0, by sampling uniformly on the sphere gd.-1 for I,.... m.
for i m do
Compute 07" X and discretize it into Phins bins using training features X
Compute joint counts of binned Of X and Y
Normalize joint counts to obtain joint probabilities P(07 X, X,Y)
Compute marginal probabilities P(O X) and P(Y)
Find the bin index 0e x in the discretized 07 X for the given sample x Retrieve p(Of sr y) from P)O)'X,Y)
Retrieve p(O z)) from P(O x)
Retrieve the marginal probability p(v) from P(Y)
p(0Ta.)) a.y)
Compute the term: pmni, y) log p(o x)p(s) end for return psi(x: m 1 pm, y) y)
"This can done by shuffling samples from the joint distribution along hatch axis.
19

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Algorithm 3 PSI Estimator (Gaussian Method)
Require: (X) ~ Pxy E P x R) where Y E {l....,cc of (with с classes), chosen pair of sample (z; y) ()), chosen number of slices (projections) m, and a chosen number of bins Thins- Initialize 8 by sampling uniformly on the sphere 3d.-1 for 1....a m.
for i 1 m do
for to d
Find Ji with P(O X|Y = 5) ~ (((gs.)))))))
end for
end for
for 1 to m do
Compute 077 x for the given sample x
Retrieve p(0 rly) from POO X Y y) (WWig ,g)
Compute p(f z) = E/=1 p( zly = j)p(y = j)
Compute the term: pmmi;(;; v) log p(o mly) end for return psi(x; y) < 1. m Ei= pmi n)
x(0
Binning. For each projection i, we bin of X into Thins To compute the PSI, estimate P(O X, Y), P(O X), and P(Y) from the binned data. For a given sample, we can then find the p(0 x, p(0," v), and p(y). The PSI is then given by:
1 m
P x. log
psi(x: v)
(30)
p(0 x x)p(y)
Gaussian. We assume that 8 X for each class follows a Gaussian distribution. For each projection i and for each class j. we estimate the mean (piz) and standard deviation (aiz). To compute the PSI, we estimate P(O x\Y) and P(O X) from Hig and oij. For a given sample, we can then find the p(o zly) and p(0T" z). The PSI is then given by:
p00TT log
psi( v) +
(31)
p(6 x)
We choose the Gaussian method (with 500 projections) the default estimator show in Appendix D.2.2, it consistently yields good performance.
A.3.3. PVI ESTIMATORS
We followed the implementation by (Ethayarajh et al., 2022), adapting their original PyTorch code to Tensorflow. We provide the pseudocode for the PVI estimator in Algorithm Note that we presented the algorithm for any label y but used predicted labels our experiments. To estimate PVI, we are required train two neural networks to obtain of (for null inputs) and I' (for training inputs). The null inputs can be obtained by setting the values of the input features to zero. Below we describe several methods we experiment to estimate the PVI.
Algorithm 4 PVI Estimator
Require: (X", Y") i.i.d. sampled according to Pxy E P(Rds R) where Y E {1,.., k}, a chosen pair of sample (x, v) ~ (X", y"), and a model V.
f' < train V on
Ø - null input (array of zeros with the same shape as X")
f train V on (00, y*)
return pei(r + y) + - log [1] + log f' (x)
'A uniform sample from gd-1 can found by sampling vector Z from dddimensional isotropic Gaussian forming //|
20

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
No Training. To obtain J', (untrained) model on null inputs.
the model that has already been trained the dataset. To obtain f. train the
Training from Scratch. To obtain f', we train another model (with different initialization but same architecture) on the training data. To obtain f. we train the (untrained) model on null inputs. In practice, instead of training new model, we can use the model from the different run.
Training MLP Penultimate. To obtain f', we use penultimate layer features as input a rather than the original inputs. We train me-hidden-layer MLP with 512 units on z obtain f'. To obtain we train the untrained MLP model on null inputs with the same dimension the penultimate layer features.
We choose the Training from Scratch method as the default estimator we show in Appendix D.2.3, it consistently yields good performance. In addition. when computing the PVI, we can choose to first calibrate the probabilities with simple temperature scaling, As we see in Appendix D.2.3, this improves the performance.
21

---

Pointwise Information Measures as Confidence Estimators in Deep Neural Networks
B. Theoretical Analysis & Proofs
B.1. Invariance Properties
Proof of Proposition
Proposition (Invariance to shift,, scale, and rotation). Let Tx oRx + P. where E IRd represents the extent to which the distribution shifted, a E IR represents how much the distribution is scaled, and R ~ Rd,xd, a rotation matrix such that RR and del R) = 1, where I the identity matrix and del represents the determinant operator.
Then we have:
pmip(a, 4) pmitp(aRa +p.y)
psip(r, 9) psitp(aRR +
puip(r,)) p p.y)
Proof. For simplicity of notation, we denote the probability distribution in the original domain by and the distribution in the transformed domain by PT.
For PMI, we have:
pmitr(aRe + p. = log Pr + +ly) log
p(zly)/ det(aR) p(z)/ dct(aR)
p(zly) p(x)
log
= pmip(1, y),
Pr(aRe +p)
where det denotes the determinant operator.
For PSI, we first note that
psitplaRe + p.u) Egg log Pr(0"(aR: + +p)y) Eg log (a + p))
(ax) Pr PT (ax) + 0 p)ly)
PT (ax) ) + 0 p))
where 6' RR. Notice that 6' will have uniform distribution over the sphere, similar because R is rotation matrix. We also apply the fact that Pa(xx- p) = p(z)/a the numerator and denominator. This ultimately yields:
psirplaRr + ppy) = Egr log ((Tx)D) p(0'xx)) x)
p( p(0Tx))
Eo
log
ppsip(a))
For PVI, we first note that the first term of PVI, - log f (v). will remain unchanged as only depends on y, and depends on the distribution y. both of which do not change with T as it one-to-one transformation. Then, for the conditional entropy term, let
f' arg min Eil log f[x](Y)) Jev
As f' is a fully connected neural network with weights and biases, let W and represent the weights and biases of the first layer, respectively. When the distribution of changes in response to T, let
g' = arg min Ep, log [[X[()). Jev
Note that g''s first layer weights W and biases 8/ will be such that W (a -))+''=WTT We will simply have W aR W and /' b w p. Therefore, g g'(T (y) (x) The search space for the arg min is the same in both cases, as the transformation is linear. We have that W (TX) W X such that the weights W' and W have one-to-one correspondence (as T is invertible). Since log [T [T))(y) = log (x)) we have the result:
pvirp(aRA. p.y/ v) pvip (x, v).
Proof of Proposition
22

---

Pointwise Information Measures as Confidence Estimators in Deep Neural Networks
Proposition 2 (Invariance general linear transformations), Let Tx Ma, where M Rddxxds is invertible matrix. Then have,
pmip( v) - pmitp(Ma peip(z, v) - peip peitp(MM,))
Proof. For PMI, as M is invertible, have:
pmitr(Ma; log PT PT(Mx log Pla) p(aly) pmir(p (2. (r(Ma) p(z)/det(M) p(x)
log
where det denotes the determinant operator.
For PVI, we follow the same reasoning as the previous proof. Same as before, let
arg min Epl- log f[X](Y)
g' arg min Ep,[- log [o0p[-log[[[(11 [[X](Y).
feV
fev
Let W and b be the weights and biases of f', and let W and 8' be the weights and biases of g'. Then, we have, W = W M and b' = b. As M is invertible, this implies that ('[Tx](v) I' (r]()) which yields the result:
poiyp(Ma, puip(x, v).
Proof of Proposition
Proposition 3 (Invariance to homeomorphic transformations). Let Tx = f(x), where ʃ Rdd + Rdd represents any continuous and invertible transformation (i.e. a homeomorphism), Then we have,
pmip( y) pmirp(f(x).))
Proof. For smooth and invertible maps, known that the probability density function P=([())) = P(x)/Jx, where
Jx = (f())
ar a scalar that only depends on x. The same rule would apply to conditional distributions p(xly) as well.
Thus we have:
pmiTp(f(s).v) =log Pr(f(x)))) log p(zlu)/Jx log
p(zly) pmip((x,) p(x)
PT([(z)) p(x)/Jx
Remark 9 (On PMI and invariance). Note that the above property for PMI also implies invariance to general linear transformations, which is the extension to Proposition 2. What these results mainly indicate is that out of the three metrics, PMI has the most structure-preservie property, followed by PVI and then PSI. This makes sense as PMI is the most general and only depends on the distribution and does not rely on anything else. Note that is invariant to homeonorphisms as well, but the invariance property for PMI is stronger as states that the aggregate invariance for MI be mirrored the pointwise level.
Remark 10 (On PSI and invariance). Note that PSI need not be invariant both general linear transformations and homeomorphisms. see why, just consider simple case where T represents general linear transformations which scale each dimension of the input separately. Then, sphere in the original domain of the distribution P gets transformed into an ellipse in the domain of the distribution T.. As PSI uses uniform distribution over all projections over the sphere, cannot say with certainty that the PSI in the new domain of T P will be preserved, because it will prefer some directions more over others. To see this, consider a specific case of T where one of the dimensions is scaled significantly more than the rest, thereby resulting in a ellipse that is very flat. In that case, most projections will contain more of that dimension, and we cannot say that PSI will be surely preserved.
23

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
B.2. Convergence Rates
Proof of Proposition 4
Proposition (PMI convergence rate). Let P(X) be a-Holder continuous and let (x,y) ~ Pxy where X RRd and
Y E (0, 1}. Let pmi, represent the KDE estimate of PMI using n samples. Assuming min (P(Y = 0), P(Y = 1)} # 0 when the probabilities are estimated on the training data, for large enough n, we can bound the estimation error as
n2 a2a/(2ad))))
pmni(z; v) - pmmd S 0
(32)
min (p(a),
Proof. For simplicity of notation, we represent all estimated probability terms by Pn, where n represents the number of samples used estimate the term. We have:
P(xly) P(x)
P. (xly)
pmi(1;;) v) pmin
log
log
(33)
(x(
log P(xly) - log (1)))) + log P(x) - log (a(())
(34)
sup zexd
log P(aly) log Pm (,(c))
+ sup =ERd
log P(x) log Pn (x))
(35)
Now, from (Jiang, 2017), we have the uniform bounds on (x) in Theorem 2, which yields:
P(c) (x)) S O 12 FERd
sup
(36)
Note that we can apply these bounds to P(xly) as well, and in that case the sample complexity changes from n to min [P(Y = 0), P(Y= = })} x n, because the number of samples that now controls the convergence rate is reduced as these are class-wise distributions. the case when min {P(Y = -), P(Y = =)} & O. note that this keeps the final convergence order unchanged, as adds fixed multiplicative term. As min (P(Y = 0), P(Y = 1)} A is assumed in the problem, we can directly apply the results from (Jiang, 2017) for P(zly) as well.
With this, we use the expansion of log write:
(x(x)
1 Pn(x)- P(x)
log P(x) log Pa (x)
log
log
(37)
P(x)
P(x)
(x) - P(x)
Px(x)) - P(x))
Pn(x) - P(x)
+
(38)
P(x)
2
P(x)
3
P(x)
Pr (x) - P(x)
1
Px(x)) P(z)
Px(x) P(x)
(39)
P(x)
P(x)
P(x)
n (2(//2)))
2a/ 22((22)))
n (20((2))))
n
0
+
(40)
p(x)
p(z)2
3
p
n m-a/(2atd)
S o
(41)
p(x)
Here we assume that n is large enough, such that the rest of the terms are insignificant compared to the first term. Combining this with (35), we have the result.
Proof of Proposition
24

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Proposition (PSI convergence rate). Let P(O" X) be a-Holder continuous for all 0 and let (x. Pxy where X IR and Y {0, 13. Let psin, represent the KDE estimate of PSI using n samples and m projections. Furthermore, let ming pmi(f" y; 2 Assuming min {P = 0), P(Y = 1)} * 0 when the probabilities are estimated on the training data, for large enough n, we can bound the estimation error as
1 1- + O
Tt (2a+1)
Ex,y psi((;:)) - psin..
(42)
2/m
ming min (p(6" 2), p(0 z|v)}
Proof. We apply the triangle inequality, similar to (Goldfeld & Greenewald, 2021) (Appendix A.4), to obtain:
psi(a;;) v) - pm 2 psi(F;1)
pmi (0 2; y)
+
Σ
pmi(O" z; v) - ppi psi n.m
(43)
(44)
Now, 0g are i.i.d, and PSI(2; v) is essentially equal to E. PMI x; v) m - can use variance based bound to obtain:
E
psi(x;;)=
pm 2; v) S
var(pmii (fT v))
p
(45)
2
Next, we have that:
m
m
E
Σ
pmn (8" x:y)
pm
>>
E
pmim(f z0 v) pmi (0 x:
(46)
sup E
pmni(6"
6T x; v) pmi (8" x: v)
(47)
We then apply the previous result (Proposition 6), obtain:
T1 (2a-tt))
sup E 0
pmmi x; v) - pmi (0 x; x010
0o
(48)
ming min {p(0T x))p((Tx(()) xly)}
n (2/4(2+11)
= 0
(49)
Pmin
This completes the proof.
Remark 11 (On KDE-based PSI estimator). The above result provides convergence bounds for the KDE-based PSI estimator, providing guarantees as a function of the number of projections m and the number of datapoints n. The result makes use of the uniform convergence bounds for the KDE-hased density estimator provided in (Jiang, 2017) The convergence rates would be tighter for larger values of and larger values of Puin Thus, we note that the convergence can be slower for datapoints for which Punin is small, which will be true for datapoints in the edge of the distribution P(X)-
Remark 12 (On PSI's faster convergence over PMI). Note that when - OC, we obtain the same rate of convergence SMI itself, which is O1m-1/2 + 1/2) (Goldfeld & Greenewald, 2021). Also, note that PSI converges faster rate than PMI, especially when considering data of large dimensionality d, as the convergence rate for PMI will be Onn ad()at))), which which.olloows Theorem 2 and Remark 8 in (Jiang. 2017).
much
Proof of Theorem 1
Theorem 1 (PVI convergence rate). Given (x, v) Pxx where X Rd and Y E {0,1), assume that P(Y 0) - P(Y = 0.5. Assume V represents the set of all possible functions modelled by neural network having some fixed architecture. Assume ff E V, log f(r]()) E I-B, B). Also, let f arg min Ex,yl- log f[X](Y)) represent the ground truth function for estimating conditional V-entropy, and f represent the trained function given n datapoints (x1, vi,,... (Fas Dn) sampled from y.. Let M max max(var' (var(f (*(x)())) (fr(f(>(()) where vor denotes the variance.
25

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Let poln represent the PVI estimated using this neural network with samples. Then, for any 8 E (0, with probability
p 2 1 26, have
Ex,y [pvi(s + pwin s (Gv) + M + 2B
2 1ogg(1/)
(50)
E V} and Rn denotes the Rademacher complexity with where the function family GV (glg(z, 3) log f()) f sampled points.
Proof. The result is consequence of the generalization bound for V-information proposed in (Xu et al., 2020) (Lemma of their Appendix). First, note that we can express PV [(x - +) = 1v(X - Y) + e, where e a random variable, and similarly for poi(r + y). Also, note that as f* and f are neural networks, the first term in the estimation of PVI will be fixed to 1, as the network can simply assign biases to the last layer such that (e] 0.5, and similarly for f as the training set is balanced. In that case, we can write:
E Exx
poil(x + y) - pui(x + f01 Exxr [
+ - +1) m
(51)
Exxx ||v(X - - - -x-]
(52)
+ Ellea]1 + E|[e2|1
$ 1x(X +Y) - fx - r + 2
(53)
where the last step follows from noting that the absolute difference between the true and estimated Y-information doesn't t depend on the individual instances, and that the Ll-norm is bounded using the variance via the application of the Cauchy- Schwarz inequality. Next, we directly apply Lemma 3 of (Xu et al., 2020), after the additional observation that in this case
Hy(Y)= = HY)).. We then have, with probability p 2 1 - 25,
Tx(X -Y fx(x - 2\ < (9v) + 2B 2log(1/3)
(54)
Applying this to (53) yields the result.
Remark 13 (On the convergence of PVI). We note that the result provides bound on the average ervor w.r.t the PVI estimation over datapoints, and thus are not uniform convergence bounds. Next, we also note that the result depends on the upper bound on the variance of the neural networks (M), which is not trivial to bound. However, overall, the convergence result for PVI still shows a few important differences w.r.l the convergence bounds for PSI and PMI. First, note that here, the convergence depends heavily on the choice of V. Choosing very deep and complex neural networks for estimating PVI will lead to a large Rademacher complexity RR, (Gv), which will lead to slower convergence. Also, ideally, we want networks to have a smaller variance over its output logits, which will eventually also reduce the value of M and make convergence stronger. This can be achieved by regularizing the outputs of the network to have low variance, and there are approaches in literature which have studied this kind of regularization (Litwwin & Wolf, 2018).
Table 3: Convergence Rate of PMI and PSI using KDE estimator (Averaged over 50 Runs with Standard Deviations Included).
100
1,000
10,000
100,000
1,000,000
[pmi(z;;) y) pmii,.
6.075
1.684+1.209
1.268
0..473
0.809++0.301
100
200
1,000
2,000
10,000
Ensiipp)((z:)) psi,,..
0.292-10.037
0.282110.036
0.270110034
0.27010.027
0.269-10.028
Experiment on Convergence Rate: We conduct a simple experiment on Gaussian mixture distributions to test the convergence rates of PMI and PSI. Based on the results shown in Table 3, we have two main observations. First, we find that both the trends of PMI and PSI are within the predicted convergence trends in Proposition 4 and Theorem 5 (we set a = 1
26

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
as our mixtures are Lipschitz continuous). This re-affirms the convergence bounds being an upper bound on the observed trend with the number of samples n.. Second, we find that the predicted convergence rate for PMI and PSI are reflective of the theoretical results. Our theoretical results stated that PMI should converge slowly compared to PSI, and the difference is amplified with greater dimensionality. After adjusting for scale and bias (error as n goes to very large values), we find that the observed convergence rate for PSI is indeed greater than that for PMI. Note that for PVI, we found it hard to estimate Rademacher complexity measures for neural network classifiers, so we cannot directly test our convergence rates.
B.3. Geometric Properties
In the following results, we mainly explore whether geometric properties of the feature distribution, such as the sample-wise margin and the subspace intrinsic dimensionality, can affect the different PI mcasures. We define the notion of sample-wise margin as the distance of a datapoint æ to the other class distribution, when it is encapsulated by sphere.
First, we provide the general idea of sample-wise margin. In the results that follow, we adopt more specific definitions that motivated from the general principle in the following definition.
Definition 6 (Sample-Wise Margin). Given F, y ~ Pxy and Y {0,1) such that P(X = 0) and P(X 1). The sample-wise margin refers to the distance of the sample from the distribution P(X = 1 v), when P(X|Y 0) and P(X = 1) are non-overlapping. When P non-overlapping probability masses Q(X(Y = 0) and Q(XYY = 1) which encapsulate most of P(X = 0) and P(X = 1) (fraction of -e) respectively. Next, we estimate sample-wise margin as the distance of x from the distribution Q(XYY = 1 11-91 v).
0) and P(X = 1) are overlapping, first we can create
We have the following result for PMI, in the context of non-overlapping feature distributions. Proposition (PMI and sample-wise margin). Let z. Px,y and Y {0, 13 such that P(X 0) and P(X|Y 1) are non-overlapping and P(Y- = 0) = P(Y 1) = 0.5. Then, we have that pmi(1; =
Proof. Since P(X 0) and P(X 1) are non-overlapping, for certain sampled y, we will have p(xly) 1 and p(zly = 1 y) 0. Thus, we have:
p(x|y) pmi(;; v) = log2 p(xly) = log2 0.5 (p(xly = 0) + p(ayy 1)) = loga 2 = 1
p(x)
Remark 14 (On PMI and sample-wise margin). Note that the above result implies that when the distributions P(X|Y= = 0) and P(X|Y = 1) are non-overlapping, then pmi(a;y) v) is always 1 irrespective of the distance of from the decision boundary. Thus, in this case, sample-wist wise margin does not affect the PMI at all.
Next, we highlight the conditions when PSI can be related to both sample-wise margin and intrinsic dimensionality (ID) of the data. First, we define the subspace ID:
Definition 7 (Subspace Intrinsic Dimensionality). The subspace intrinsic dimensionality (ID), denoted by Kp, is the dimensionality of the smallest subspace W that contains the support of P(X).
We have the following result for PSI that relates it to sample-wise margin and the intrinsic dimensionality (ID) of the data. Note that for the overlapping case, there no unique notion of sample-wise margin, as it depends on how Q constructed, and also depends on the fraction (1 e) of the distribution involved in encapsulating the class-wise distributions. For the following result, we spheres to construct Q, for each wliss-wise distribution.
Theorem 2 (PSI and sample-wis, wise margin and ID). Given x. y Pxy with Y 10,,1, and assuming y 0 without loss of generality, we consider two non-overlapping spheres S₁ and S2 with radii R₁ and R2, and centers C1 and C2 such that x E Sy. Here, the sample-wise margin, denoted by d(æ, S2), refers to the distance between æ and the surface of Sz. The
subspace intrinsic dimensionality of P(X) is denoted by Kp. Let (0" x|S2] = 10"'s 1t:rn E 82} represent the set of points in the real line of the 9 projection of Sz. Let e = maxs, P(OT xly = 1, x E IR {0T x : x E S2)), where 10 x : x E S2).
We also define the following two quantities:
max p(O x/T=yy- = 1), p(8 rly = =0] } 8.
Piaxx = max
Puin min p(O 1(0 = 0).
e.
8.2ES,
27

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Then, we have the following lower bound:
psi(x; y) 2 log Pmin
+ (1+
log
Pmin Bo(d(
Pmax
Kp 1 2 2
(55)
Pmaxx
where BB(a, b) denotes regularized incomplete bela function (Oldham et al., 2008), and 7(a, b)
a o+b
a+b
Proof. The proof follows from the proof elements of Theorem and 2 of (Wongso al., 2023a), and Theorem of (Wongso et al., 2023b). First, using the proof of Theorem 1 of (Wongso et al., 2023a), it follows that Pr(0"xx x {0t 2022
(,(d(t,Ss)).R) du 2
We arrive at this result by considering two spheres in the context of Theorem 1 of (Wongso et al., 2023a), Si being a zero-radius sphere centered at r, and S being the same as S2 here.
Given that = 0, we then can write:
p(u = 0 x)]
psi(;; y) Eg log
(56)
ply 0)
ply 0/0"'))
Pr(ff'r 10t x =|S2)(z) Eg.00 ze(0o)))))
log
p①y = 0)
p(y = 0)0Tx))
+ Pr(oTx> x & 10Tx x Bg.0T (
log
(57)
p(y 0)
When H $ {0TH, [S2),, note that Sg does not play role in estimating the probabilities. this cases, have:
P(0T x10 = =
ply 0|0"x) x) p00''zyy00 p(0
Pmin
Pmin +
The e term is a consequence of the fact that only the probability outside the set (0 x|S2} contributes to p(0" xly = in this case.
When 0 E (0"x]S2), both S₁ and S₂ will contribute to estimating the prohabilities. In this case, we have:
ply = 010" p(y=0)0"x>)
Pmin
p(0'xly 1(00x =
2pmax
This, combined with the fact that ply 0) ply 1) 0.5 and Pr(OT x/ Br(d(e,s)) (2=2). then yields:
psi(x; y) 2
log
Pmin
Br(d(e,Ss)).a)
de 1
(58)
Prain +
Pmiin
d2 d=-11
+ log
Br(d(a,Sa))
(59)
Pruxx
2
log
Pmin
log
Pmax
)
B Brld(e,Se))RR)
d 1 1 2
(60)
Prnxx
Pruin +
Furthermore, given that all of P(X) lies within subspace of dimensionality Ky, we can convert our analysis into space of dimensionality Kp instead, as implied from Theorem of (Wongso et al., 2023a). Note that in doing so, the distances do not change, and the measures é, Prnxx Pmin all stay the same, because the dimensionality of the null-space within the projections has zero measure. This yields the final result:
psi(z;;) v) log Puin + (1+ log
Pmax
Kr Kp-11 1 2 2
Pmin Br(d(s,So).Bo)
(61)
Pmax
28

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Remark 15 (On the lower bound of PST). Note that when x is farther away from S2, i.e a larger sample-wise margin, it leads to a larger lower bound on the PSI. Thus, in this case, PSI will likely be larger. This generalizes the result in (Wongso et al., 2023b), which was only for symmetric non-overlapping distributions P(X = 0) and P(X = 1). As Theorem 2 shows, PSI can be sensitive to both soft and hard margins. Furthermore, in three scenarios we expect the bound to be tight. (i) For distributions where e is small, and Pinas >> Pinin- (ii) When the radius R2 is large, or the distance d(x. S2) is large. (iii) When the intrinsic dimensionality Kp is small. Thus, for high-dimensional data, if lies on a low dimensional manifold, we will get a significantly lighter result. Farthermore, we note that none of the terms pmin Pmaz 6, R2, Kp are dependent on the sample-wise margin d(z, S2). Thus. the only term affected by sample-win wise margin the regularized incomplete beta function B Ra) ( [Kp-1 2 Therefore, our hypothesis that the lower bound of psi(;; y) increases as the sample-wite margin increases is valid.
Remark 16 (On the sample-wise margin definition in Theorem 2). Note that the definition of sample-wise margin here d(xr, S2) converges to the classical definition of margin w.r.l a linear decision boundary when R2 - Xc.
Remark 17 (On the choice of S1, S2, and c). As mentioned in the main paper. here we provide some more context to the choice of the spheres S1 and S2, and the nature of Note that the choice of S₁ does not affect the result much, as the only main constraint for S₁ is that must be contained within it. As such, the radius Rr also does not directly impact the result. However, S2 should be ideally chosen such that it contains as much of the distribution P(X = 1). To see this, we mainly look at how the choice of S2 impacts Note that if is very large, such that Pmax < Pmin + c, then the dependence on sample-wise margin reverses (less margin leads to more psi). To avoid this, we can always choose S2 such that is small. Let S2 be chosen such that p(æ E S2ly = 1) = p. Furthermore, let us assume that there is another bigger sphere Sa such that S2 is contained in S3, such that p(æ E Ssly = 1) = 1. Let the radius of Ss be R3. Then, we can approximate e as (1 -1-p) p)/ (2 - R2)). This is because the projection of S2 will have a length of 2R2 and similarly for Sz. Thus, if we choose S2 such that p is made arbitrarily close to 1, we can make arbitrarily close to zero. However, do note that although this can be done when the distributions P(X|Y= 0) and P(X =1) have less overlap, for the case where P P(X = =0) and P(X 1) are highly overlapping, this may not be possible. As in most of our experiments x is taken from the penultimate layer of neural networks which have separable features, the assumption will hold with high probability.
Remark 18 (On sensitivity to hard margins). When P(X = 0) and P(X = 1) are clearly separated, one should ideally have maximum confidence estimates everywhere. But the fact that we do not know the ground truth distribution P(X.Y Y) implies that even when the estimate of P. denoted by Q(X, Y). from the training data, is perfectly separated, the separation of the true unknown P(X.Y) Y) will be most likely smaller with potential overlap. This is because Q(X, Y) clearly has a significant chance of overfitting the true distributions, as the objective of the classifier is always to separate the training feature distributions anyway. Due to this potential overestimation of the real margin, encoding additional geometric information about QXX, Y), such the hard margin involved in the perfect separation, can inform about the probability of P(X = 0) and P(X = 1) being perfectly separated as well. If IfQ Y) has very small hard margin, then it is possible that P(X,Y) ends up with overlapping class-wise feature distributions, and if it has a very large hard margin, then the opposite is likely. Lastly, correlation between the hard margin between the class-wise feature distributions and generalization has indeed been observed in literature (Grønlund et al., 2020), showcasing the significance of this issue.
Finally, we have the following result to relate PVI to the sample-wise margin.
Proposition 7 (PVI and sample-wise margin). We are given neural network with function :: - R2 for classifying points X into two labels Y E (0, 1,, and we are given that P(Y = 0) P(Y = 1) = 0.5. We assume that the final outputs of f are passed through softmax operator with temperature T = 1, to yield the output (f We are given an instance (x. y) ~ P(X,Y) Given x as the input, we define margin T as in ( (Vemuri, 2020), where
[(x) - f(z) (v(((()))) v=(((2)))))
(62)
If M = max, (Ilaaa.llIIV (II1II/..000))III1
then we have: pui(x + u) S 11-logg (1 + 2MT).
Proof. As we consider the function outputs before the softmax here, we re- represeetnnt the two terms of PVI. The first term which will be equal to 1, as the neural network can simply learn the biases of the last layer and set them such that sof soflmax([)[]]c = coflmax(])[[]] - y) 0.5. Note that, as f(w)--(()) VV and M = max, (I (f(=))) we can write
of PVI, is now represented as - log (ffftmaaff]](((
]1= - y) - f((((())) S (2M²2 + +M22 2MT
(63)
29

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Table Correlation of PMI/PSI/PVI with Margin (Averaged over Runs with Std Dev shown), Best results are highlighted in bold.
Method
MLP, MNIST
CNN, F-MNIST
VGGI6, STL-10
ResNer50, CIFAR-I0
PMI
0..08
0.42910.033
0.61910.011
0.63710.019
PSI
0.657:00022
0.846:100066
0.809=000066
0.758=0.033
PVI
0.327=0.025
0.368+00.008
0.604=0.010
0.563=0.011
Like before, let f arg gin E (Y)) denote the trained neural network that estimates conditional
V-entropy. Now, the second term for PVI will be represented as log (ff)max(("))at)) Then, given x; y PxY, have:
e (x)(())))
log (softmax(f")(<(())) log
(64)
(f()) ef [f
1
log
(65)
(1( ((a((( +1
S -log (1 + -2M7)
(66)
Then, the result direct follows from the expression PVI.
Remark 19 (On PVI and sample-wise margin). As the above result shows, PVI can indeed be sensitive to the sample- wise margin, and thus datapoints x which are near to the decision boundary can be expected to have lower PVI and vice versa. However, the raw PVI values may not be very sensitive to margin. For >> 1, can approximate increases and the differences become smaller for larger
pvi(æ v) < 1 e -43d"T, which converges to I quickly Thus, if one were to replace the PVI values by their relative rank, could potentially see a higher correlation. As our experiments use the pointwise measures to rank confidence scores relatively among samples, samples with larger PVI will likely correspond to the samples with larger sample-wist wise margins.
Experiment on Correlation to Margin: We perform an experiment to examine whether samples closer to the decision boundary (smaller margin) are assigned lower confidence scores by the various measures compared to those located further away (higher margin). We aim to test our hypothesis that PSI is the most sensitive to sample-wise margin. We approximate the sample-wise wise margin using the method provided in (Elsayed et al., 2018), which approximates the smallest distance of a datapoint x to the decision boundary by:
f(x), f(x), Vv(()))))-V((())))))))))))))))))
d Rd
(67)
where choose fx); and fxx] to be the highest and second highest logits of the neural network f (also used in Proposition 7) and V represents the gradient operator. Then, we compute the Pearson correlation between the margin and the confidence estimates returned by the different PI measures. The results are shown in Table 4. In addition, we use Uniform Manifold Approximation and Projection (UMAP) to visualize the features of the penultimate layer on the test dataset. We rank the PMI, PSI, and PVI for each sample and visualize these rankings using color bars in the UMAP plots. As shown in Table we find that PSI the most correlated with margin, followed by PMI and then PVI, supporting the theory. The higher correlation of PMI with margin compared to PVI could be attributed to the decrease of sensitivity of PVI when M (related to the complexity of the network) is large. In Figure I, find that for all measures, as the samples get closer to the decision boundary, the values generally decrease. We generally observe that PSI tends to rank highly misclassified classes lower than those that are often classified correctly. For example, in the STL- 10 dataset, animal categories generally receive lower-ranked confidence scores overall (showing more blue than pink).
30

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
VCG16, STL
VEGL6. STL-10
VGG16, STL-10
PVI rark
SPiP
bird
E032
DArS
6902
bird
6008
Bruck
3002
truck
3000
CIV
4002
4002
4000
dog
3002
2002
monkay
3000
CSF
CMT
1002
1002
1000
(a) PMI (VGG16, STL.-10)
(b) PSI (VGGI6, STL-10)
(c) PVI (VGG16, STL.-10)
Figure 1: UMAP visualization of the penultimate layer features.
C. Benchmarks & Experimental Details
C.1. Benchmark Datasets Architectures
Below list of the benchmark datasets we use in our experiments:
1. MNIST is dataset comprising of x8x28 grayscale images of handwritten digits from 0 9.
2. Fashion MNIST is dataset comprising of 28 x2 grayscale images of fashion products from 10 classes: T-shirt, trouser, pullover, dress, coat, sandal, shirt, sneaker, bag, ankle boot.
STL-10 is subset of the ImageNet dataset, consisting of 96x color images from 10 classes: airplane, bird, car, cat, deer, dog, horse, monkey, ship, truck. was primarily developed for unsupervised learning, and thus most of the samples are unlabelled.
CIFAR-10 is a dataset consisting of 32x32 color images from 10 classes: airplane, automobile, bird, cat, deer, dog, frog,
horse, ship, and truck.
5. CIFAR-100 is expanded version of CIFAR-10, comprising 100 classes.
6. Stanford Dogs is dataset consisting of color images of various sizes, spanning 120 different dog breeds.
7. Tmny-ImageNet is dataset consisting of 64 x6 color images from 200 classes. It is smaller subset of the original ImageNet dataset.
All of these datasets are publicly accessible through the TensorFlow Datasets catalog https: http///nnooffffooooooooooooooooo .rg/ dat aset /atttaalll allaalooveeeeeeee hovvvvvvviiieeeeee All images, except those from MNIST and Fashion-MNIST. resized 224x224, We split the dataset into training, validation and test sets.
Table The architecture basic CNN.
Layer Type
Parameters
Convolutional
strides=1, padding=same, ReLU 32 filters, kernel size=3x3, strides=1, padding-same, ReLU
32 filters,
Convolutional
Max Pooling
pooLsize=22x
Dropout
rate=0.3
Convolutional
64 filters, kemelLsize=3x3,, strides=1, padding=same, ReLU
Convolutional
64 filters, kernel size=3x3,, strides=1, padding=same, ReLU
Max Pooling
pool_size=2x2
Dropout
rate=0.3
Convolutional
128 filters, size=3x,, strides=1, padding=same, ReLU
Convolutional
128 filters, kemeLssize=3x3, strides=1, padding=same, ReLU pool_size=2x2
Max Pooling
Dropout
rate=0.3
Fully-Connected
128 units, ReLU
Fully-Connected
K units (where K is the number of classes), softmax
Below list of neural network architectures that we use in our experiments:
31

---

Pointwise Information Measures as Confidence Estimators in Deep Neural Networks
Table The top layers the benchmark model's architecture.
Layer Type
Parameters
Base Network
Weights are pre-trained on ImageNet dataset
Colly-Connected
256 units, ReLU
Dropout
rate=0.3
Fully-Connecter
128 units, ReLU
Conle-Connected
K units (where K is the number of classes), softmax
Multi-layer Perceptron (MLP): We implemented a simple MLP, consisting of three hidden layers with 512 units and
ReLU activation each.
Convolutional Neural Network (CNN): We implemented simple CNN with detailed architecture as illustrated in
Table 5. We refer to this the Basic CNN.
3. VGG16: We loaded the base model of VGG16 from Tensorflow and excluded the three fully-connected layers at the top of the network.
4. ResNet50: We loaded the base model of ResNet50V2 from Tensorflow and excluded the three fully-connected layers at
the top the network.
ResNet101: We loaded the base model ResNet101 from Tensorflow and excluded the three fully-connected layers the top of the network.
6. Inception V3: We loaded the base model of Inception V3 from Tensorflow and excluded the three fully-comnected layers
at the top of the network.
7. 1e1seNet121: We loaded the base model of DenseNet121 from Tensorflow and excluded the three fully-connected layers at the top of the network.
For all the pre-trained networks, we incorporated four new layers on top of the base network, as detailed in Table All the pre-trained network modules are publicly accessible through the Tensorflow Keras Applications catalog at https: /ttpss/////soffff//// http:/appddccsp/peeee www.ddcccsppto/ttoeeeeomom htattthhh//t/kerra/eeeeeeeeeee httpchhon/////eeeara//ppptttette wttprraasppppppcciioooooooo
C.2. Hyperparameters
We use the AdamW optimizer with a learning rate of le-4 and weight decay of le-4, with batch size of 128. A learning rate scheduler ReduceL_ROnPlatenn factor of 0.5 if it does not improve for consecutive epochs. Early stopping is also applied, monitoring validation accuracy with patience of 10 epochs, and restoring the best weights upon stopping. We train for maximum of 100 epochs, except for Fashion-MN MNIST, where we use 300 epochs. All experiments were performed using a single NVIDIA A100 (80GB SXM) GPU. We report the classification errors for the different model-dataset pairs in our experiments in Table 7.
employed, which monitors validation accuracy and reduces the learning rate by
Table Train, Validation and Test Classification Error Percentage the Different Model-Datase Dataset Pairs (Averaged over Runs with Standard Deviations Included)
MODEL,DATASET
Train Error
Validation Error
Test Error
MLP. MNIST
0.00 + 0.00
1,56 + 0.04
I,53 0.03
BASIC CNN, FASHION MNIST
0.01 + 0.01
6.21 = 0.09
6.52 0.19
VGG16, STI-10
0.00 + 0.01
8.73 17330..5
8.94 0,41
RESNET50, CIFAR-10
0.00 + 0.01
4.39 0.21
4.78
RESNET 101, CIFAR-
0.06 0.07
0.53
21.60
INCEPTIONV3, STANFORD DOGS
0.14
0.994
23.46 0.65
DENSENETI21 121, TINY-IMAGENET
0.02+0.00
27.89 27.4940.44
27.95 = 0.31
C.3. Details for Experiments Main Paper
Below, we provide more details on the experiments presented the main paper.
32

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
C.3.1. DETAILS FOR EXPERIMENT IN SECTION 4.1 (FAILURE PREDICTION)
In this experiment, the goal is to compare the effectiveness of the three PI measures for misclassification detection and selective prediction. We formulate the problem binary classification task where have binary failure label:
f (y # Ú)
(68)
In other words, we assign label for misclassified samples and for correctly classified samples. Let be the confidence scores quantified by different approaches. For a threshold value 7, we can compute:
N
N FPy(T)
(P
= Σ (1 - yss) 1(c2
= ysa 1
Σ
(69)
F=1
i=1
N
N
FN₃(T) = Σ (1 (fa) (()<T)
(N Σ
Vz 1(c<T)
(70)
2m1
From these, we can compute the following:
TPy(T)
Sensitivity
(71)
( + FN)(T)
TP, (T)
Precisiony (T) (T) + FT)(T)
(72)
FT)(T)
FF) (T) + F
(73)
(74)
In misclassification detection, the two commonly used metrics are AUROC (Area under Receiver Operating Curve) and AUPRC (Area under Precision-Recall Curve) to evaluate performance on multi-threshold list (Ta ) of length T.
The AUROC is defined as:
AUROC, (T)) - FPR (T(-)) (Sensitivity (Ti) + Vensitivity,(Teer (Tr-1))
(75)
2
T -= Vf (1(c TT) - 1(c2 T(-1)) Σ V5.) (1(c (1) + +(cc F--1))
(76)
ΣN Σi=195t Usa
E (1 - -y9,i)
The AUPR is defined as:
T
AUPR success
Sensitivityy (Te) + Pnnsitivity,(Tee--- (recisiony(T)
(77)
t=1
Σ N Dia
8/2) (1-9)))
(L(c 2 - 7--1)) EN (1 -1- 9z.)) 1(c T.) ΣN 1 Ti)
(78)
E yg.))
AUPRC more informative than AUROC when there significant difference between the positive and negative class base rates. However, AUPRC is heavily influenced by the base rate of the positive class. Therefore, as suggested by (Hendrycks & Gimpel, 2017), we present two types of AUPRC results: AUPR f. success. where the success class is treated as positive, and AUPR, ferror- where the error class is treated as positive. The error classes can be treated as positive by labeling them positive and multiplying the confidence scores c by -1. The AUPR defined as:
AUPR /errer
Sensitivityy y (TT) + Sensitivity, ( (T+- Precision (TI)
(79)
i= Uya (1(c < Tt) - 1(c 7-1))
N
EN Uz. 1(( T))
(80)
ΣN
Σ 1(c < <T)
Di=10 Us.
33

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Additionally, we report the FPR95 metric, which measures the false positive rate when the positive rate is 95%.
In selective prediction, given a threshold filter out the samples with confidence с T, and compute the performance on the remaining samples 2 In this context, the risk is defined as the error rate of the remaining samples after selection:
Risk(r) = Precision 2i-1 Pfs 1(c27)
(81)
E E 1 m))
Coverage is defined as the proportion of samples remaining after selection:
N 1 2 T.)
Coverage(r) 2i=1
(82)
The most common metric used in selective prediction is the AURC (Area under Risk-Coverage Curve) which evaluates performance multi-threshold list (7:) of length T. The AURC is defined
(Risk(T)) + Risk(T(-1))
AURC- Coverage(T)) (T))
(83)
2
C ( 1(c r--1))
E Mrs 1(c 2T) r) +
E 1is 1(c 2 T-11) EE 1(c2 m)) 2 E 1(c 2 T-11)
(84)
N
N
common criticism of the AURC metric is that does not allow for meaningful comparisons across problems (Geifman al., 2019). The same AURC value can correspond to ideal confidence estimator for one classifier (with high overall risk) and to a completely random confidence estimator for another classifier (with low overall risk). To address this limitation, EXcess-AURC (E-AURC) normalizes the AURC against an oracle confidence estimator g" that perfectly orders samples in decreasing loss. Formally, for a classifier h and confidence estimator g. E-AURC defined as:
E-AURC(h, g) = AURC(h, g) - AURC(h,
(85)
where AURC(h 9²) represents the minimal achievable AURC for h. The oracle g" achieves perfect ranking, ensuring the smallest possible risk for every coverage level. By construction, E-AURC = for an ideal estimator, and positive values indicate excess risk due to imperfect ranking of samples by confidence.
Next, we provide details the benchmark methods against which we compare our methods. Let represent the logits of the network (output of the last layer before the softmax function). For K number of classes, the softmax function is defined as:
(x(Z)
(86)
Σ E
where 0 (z) denotes the k-th element of (z).
We define the maximum softmax probability (MSP), the softmax margin (SM), the max logit (ML), the logits margin (LM),
negative entropy (NE), and the negative Gini (NG) as follows:
MSP(z) == (g(2)
(87)
SM(z) == 0g 0g(Z)
max (x(z)
(88)
kEY:kfj
ML.()) 2g
(89)
LM(z) = 2ÿ
max
(90)
KEY:E'
NE(2)= = Σ (z (z)
(91)
kEY
NG(2) : Σ (x)2)²
(92)
kEY
where 9 = arg maxkey Ey is the predicted label.
Aside from ML and LM, the remaining methods apply temperature scaling to the logits or PI values, with the scaling parameter optimized using negative lokeliield on the validation set.
34

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
C.3.2. DETAILS FOR EXPERIMENT IN SECTION 4.2 (CONFIDENCE CALIBRATION)
In this experiment, the goal is to determine to what extent the confidence scores estimated by the three PI measures reflect the true correctness likelihood (well-calibrated). A commonly used calibration metrie is Expected Calibration Error (ECE) which bins the predictions in [0, 1] under M equally-spaced intervals (we choose A 10), and then averages the accuracy/confidence in each bin. ECE is defined as follows:
M ECE- Σ
[Bml |acc(Bm) conf( (m)
(93)
"
However, the ECE has noteworthy limitations: it relies binning approach that introduces bias-variance trade-off's (fewer bins increase bias, more bins escalate variance), considers only the maximum predicted probabilities (ignoring potentially meaningful information in accurate models (Pavlovic, 2025; Nixon et al., 2019). Nevertheless, we still present the results for ECE in Table 8 in Appendix C.4.1.
estimates), and may mislead by yielding low calibration error even in poorly
Instead, we consider the metries introduced by (Nixon al., 2019):
Static Calibration Error (SCE) is multiclass extension of the Expected Calibration Error (ECE) that evaluates calibration across all predicted class probabilities rather than only the top prediction. For each class, predicted probabilities are partitioned into bins, and the absolute difference between the bin's average confidence and accuracy is computed, weighted by the number of samples in that bin. The SCE is the average of these weighted errors over all classes, providing a more comprehensive measure of calibration. Unlike ECE, SCE can be zero only if the model is perfectly calibrated for every class, making it a stricter and more informative metric for multiclass settings.
Cass-Condiitional Static Calibration Error (CC-SCE) is a multiclass calibration metric that extends SCE by computing calibration error separately for each class and then averaging over all classes. For each class, predicted probabilities for that class (regardless of whether it is the top prediction) are partitioned into bins, and the absolute difference between the bin's i average confidence and accuracy is calculated, weighted by the proportion of samples in that bin. Averaging these weighted errors across all classes yields the final score. This class-wise approach captures calibration behaviour for every class individually, avoiding the bias standard ECE, which only considers the most confident class per sample.
Adaptive Calibration Error (Ada-SCE) is a multiclass calibration metric that, like SCE, measures calibration across all predicted class probabilities but avoids the fixed-bin bias of SCE by using adaptive binning. Instead of dividing the probability range [0.1] into equal-width bins, ACE sorts the predicted probabilities for each class and partitions them so that each bin contains approximately the same number of samples. This ensures that even low- low-probability regions with few samples are adequately represented, reducing bias in sparse regions. For each class and bin, the absolute difference between average confidence and accuracy is computed and weighted equally across bins and classes. Averaging these values yields ACE, which offers a more reliable estimate of calibration calibration-esseciilyy especially for skewed probability distributions or imbalanced datasets.
Root Mean Squared Calibration Error (RMS-SCE) extends SCE by computing the square root of the mean squared difference between predicted confidence and empirical accuracy. It applies the L2 error to penalize larger calibration deviations more strongly than absolute-erre metrics. Each bin holds roughly 100 predictions, ensuring stable accuracy estimates within bins. RMSCE is particularly useful when large miscalibrations are of greater concern, as the squaring step magnifies their impact before averaging and taking the square root.
We additionally consider 4 other common metrics in Appendix C.4.1:
The Maximum Calibration Error (MCE) measures the worst-case deviation between predicted confidence and empirical accuracy across all bins. Unlike ECE, which averages the bin-wise calibration errors weighted by bin size, MCE only considers the largest absolute gap. Formally, it is defined as:
MCE-
max me mE.......MM)
(acc(Bm.) - co.)(Bm)]
(94)
where Bm denotes the set of samples whose predicted confidence falls into the m-th bin, acc( is the accuracy within bin m. and conf( B,n) is the average predicted confidence in bin m.
35

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
The Average Calibration Error (ACE) measures the mean absolute difference between predicted confidence and empirical accuracy across all bins, without weighting by bin size. Unlike ECE, which weights each bin by its number of samples, ACE treats all bins equally, providing an unweighted estimate of calibration error. Formally, it is defined as:
M
ACE
M Σ
(em) - Bmm)
(95)
where Bm denotes the set of samples whose predicted confidence falls into the m-th bin, acc(Bm) is the accuracy within bin m. and conf( (...) is the average predicted confidence in bin m.
The Negative Log Likelihood (NLL) measures the quality of the predicted probability distribution by penalizing incorrect or overconfident predictions. Formally, for N samples, is defined as:
N
NLL Σ
log (o x;))
(96)
where (1( (z)) denotes the predicted probability assigned to the true class ys for input under model parameters 6,
The Brier Score (BS) measures the mean squared difference between the predicted probability distribution and the actual outcome, providing an overall assessment of the accuracy of probabílistic predictions. It penalizes both overconfidence and underconfidence in predictions. Formally, for N samples and K classes, it is defined as:
N
K
BS
(po(2n
x:) (y: = k))²
(97)
where (y = k (ns) is the predicted probability that sample belongs to class k; and 1 (y: = =k) is the indicator function that equals 1 the true label is & and otherwise.
Similar to the failure prediction experiment, we apply temperature scaling to all the logits and PI values, with the scaling parameter optimized using negative log-likelihood the validation set.
C.4. Additional Results for Experiments in Main Paper
C.4.1. ADDITIONAL RESULTS FOR EXPERIMENT IN SECTION 4.2 (CONFIDENCE CALIBRATION)
For completeness, in Table 8, we also report results for additional common calibration metrics: Maximum Calibration Error (MCE), Average Calibration Error (ACE), Negative Log-Likelihood (NLL), and Brier Score (BS). Generally, MSP performs well on ECE, MCE, and ACE; however, noted earlier, these metrics have known limitations, such bias-variance trade- offs and reliance solely on the maximum predicted probability. For NLL and BS, both MSP and PVI achieve competitive results.
36

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Table 8: Comparison of Confidence Estimation Methods for Confidence Calibration (Averaged over 10 Runs).
Model, Dataset
Method
BCF
MCF.J.
ACE.J.
NL..
BS
PMI
0.31 0.06
0.08 0.03
7.33 1,39
6.32
2.97 0.07
MLP, MNIST
PSI
0.16
0.28 0.04
23.26 + 1.43
9.36
3.99 199
PVI
0.39 0.08
0.13 0.03
11.63 + 1.81
6.17
3.02 + 0.07
MSP
0.10
0.18 = 0.02
5.12 1.47
19.72
10.23 0.34
CNN, F-MNIST
PMI
0.10
+.00
8.17
10.56 0.39
2.28
0.03
0.96
12.10 2.12
PVI
1,48
0.30
0.18 0.03
10,28 2.65
19.712000
10.22 0.34
MSP
0.46 0,32
5,40 0.53
27.62 0.84
13.09 0.40
PMI
1.07 0.13
0.21 = 0.04
5.51 1.18
28.71 = 0.77
13.25 = 0.39
VGG16, STL-10
PSI
1
1.20
0.21 = 0.03
1.16
28.26 2 0.86
13:21 0.39
2.34
27.44
MSP
0.06
0.45 0.06
15.51
1.22
PMT
+.13
0.48 + 0.08
6.82 1.01
16.44 = 0.56
7,31 + 0.22
ResNe150, CIFAR-10
PSI
1.11
0.33 + 0.07
9.52 2:26
16.05 = 0.53
7.33 + 0.21
PVI
0.39 0.21
22.42 1.06
15.49 0.44
7.24 0.20
ResNet101, CIFAR-100
PSI
5.73
0.50 0.10
10.75 0.41
102.67 1.65
33.14
PVI
10.29
2.56 0.15
20.99 + 0.72
91.24 1,78
31,56 0.59
MSP
4,80 0.19
1.66 0.14
5.87 # 0.39
95.27 # 2.44
34.12 0.69
Inception V3, Stanford Dogs
PSI
6.84 0.62
1.11 + 0.22
11,83 + 0.71
112.53 4: 1.73
36.71 0.51
PVI
8.79
1.78 0.20
15.27 + L.07
95.27 2.44
34.12
125.17
DenseNet121, Enyllmageeet
PST
6.73 0.46
1.03 0.12
9.87 0.45
147.91
41.53 0.37
PVI
9.80 0.28
2.60 0.14
17,47 0.33
125.05 0.98
38.58 0.22
37

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
D. Additional Experiments
D.I. Normalization: Effects of Softmax and Temperature Scaling
beutnol AUROC;: 87.06, AURC:
beftmmex boaling)
(Wit bottmax lempanature Scaling!
AUROC: 91.63, AURC
AUROC, 92.25, ALFCC 7.70
1.0
1.0
100
Logs Log: Volues
0.8
0.8
0.5
0.5
0.4
0.4
Wrong Fredictions
Correct Prediction
Wrong Prodictions
Cornect Prodctions
Wrong Prodictions
Correct Productions
(a) Logits (Without Any Scaling)
(b) Logits+ Scaling
(c) Logits+Softmax+ Temp. Scaling
PM (Withouz Scaling)
Sotomax
(With Temperature Scaling) AUBRS #1 A8, AUI8C -747
MIROC, 68.67 AURC 110.0 47
AURRC: +1 A0, AURC -..0%
0.8
Velues
PM Velues
Veuss
0.5
0.2
Wrong Predictions
Cornect Predictions
Whong Predictions
Correct Predictions
Wrong Predictions
Cornect Predictions
(d) PMI (Without Any Scaling)
(c) PMI+Softmax Scaling
SMI+ Temp. Scaling
PSI (Without Scaling)
PSI With Softmak Stalingi AUROC> 88.40. AURC 9.GB
PS (Wit Scftmax limporauree Scaling)
AURDOy 19.75, AURC 17.40
90.77 AQRD, 9.11
1.5
0.8
0.5
PSI Valuoss
0.5
0.6
PS1
PS1
024
02-
0.
0.0
Wrong Predictions
Correct Prediction
Whong Predictions
Correct Predictions
Wrong Predictions
Correct Predictors
PSI (Without Any Scaling)
(h) SoItmaf Scaling
(i) PSI+Softmax+Temm Scaling
AUB0F 4812, AIR 6.85
PV (Without Staling)
PVI With sottmax Scaling! AUROC:
PVI (WAh Softmax Tempe Scaling)
AUROC: 03.13, AURC
1.5
1.0
PV Vallsss
Values
VSlues
0.5
0.5
-10
0.6
40-
Wronn Prodictions
Conrec: Predictions
Predictions
Correct Predictions
Prodictions
Correct Predictions
(i) PVT (Without Any Scaling)
(k) PVI+Softmax Scaling
(l) PVI-Softmax+Temp. Scaling
Figure 2: The distributions of confidence values estimated using logits (figures a-c), PMI (figures d-f), PSI (figures g-i), and PVI (figures j-l) for incorrect and correct test predictions (model: CNN, dataset: Fashion MNIST). First column (left figures): raw values; Second column (middle figures): with softmax scaling; Third column (right figures): with softmax and temperature scaling. The respective AUROC (scaled by 100, higher is better) and AURC (scaled by 1000, lower better) are also reported.
In this section, we analyze the effects of softmax and temperature scaling on the raw (unnormalized) confidence values. For a given vector of unnormalized confidence values T of length K (number of classes), the softmax function is given by:
T
(T(T))
(98)
Σ e
where (T) denotes the k-th element of (T).
38

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
The softmax function with temperature scaling is:
Ex/TT
(x(T,T)
(99)
where T is the temperature parameter. By adjusting the temperature T, we can control the sharpness or smoothness of the resulting probability distribution. When T' = 1, the temperature-scaled softmax reduces to the standard softmax function. Since the same T is used for all classes, it does not change the maximum of the softmax function, which means that the predictions of the network remain the same. To obtain the optimal temperature for a trained network, we select the temperature from the range 0.01, 0.02, ... 4.99, 5.00 that maximizes the AURC on the validation dataset.
We compare the effects of softmax and temperature scaling for the four approaches (trained network logits, PMI, PSI, and PVI). We show the results for CNN model with Fashion MNIST dataset in Figure 2 The figure shows the violin plots of the confidence scores for wrong and correct predictions. We also report the AUROC and AURC for each method in the respective figure.
Takeaway. Without any scaling, the raw confidence estimates can span a wide range of values, resulting in a distribution that may be highly skewed or exhibit large variance, with significant overlap between the distributions of wrong and correct predictions. Applying softmax normalizes the logits into a probability distribution, which transforms the range of values and adjusts the distribution, often leading to more concentrated confidence scores for correct predictions at high values, while wrong predictions tend to have a broader distribution. This can help reduce the overlap between the two distributions. Temperature scaling further increases this separation by either compressing the confidence scores of correct predictions or broadening those of wrong predictions.
D.2. Comparison of Various Pointwise Information Estimators
In this section, we compare the different methods of estimating each PI measure to obtain the best results. For comparison, we report the results for MLP trained with MNIST dataset as well as CNN trained with Fashion MNIST dataset for 5 runs. The hyperparameters for the training are reported in Appendix C.2 and the model classification errors are reported in Table To show the improvement of these estimators, we also include the results for softmax (without temperature scaling).
D.2.1. COMPARISON OF PMI ESTIMATORS
For this experiment, we consider two different types of critic design: joint critic and separable critic. We also consider the three estimators: probabilistic classifier, density ratio fitting and variational JS bound. More details these critic designs and estimators can be found in Appendix 1.3.1. We first look at the convergence behaviour of these estimators by computing the I(T; Y) where 7' is the penultimate layer for MLP model trained on MNIST dataset. We train each critic model for 100 epochs with batch size of 512 and Adam optimizer (with learning rate of 0.001). We present the results (averaged over runs) in Figure 3, with the shaded regions representing the standard deviations. We observe that the probabilistic classifier estimator converges more slowly and exhibits higher variance compared to the other two estimators. As a result, we exclude it from subsequent comparisons.
We then evaluate the performance of confidence estimates returned by both the density ratio fitting and variational JS bound estimators (using both joint and separable critics) on MLP and MNIST, as well as CNN and Fashion MNIST. The confidence ranking metrics are AUROC, and AURC. In addition, we assess their performance based on whether softmax scaling is used and whether confidence estimates are derived from the penultimate layer features or output layer features (before the softmax function). We report the results in Table 9.
Takeaway. We observe that using the output layer features, rather than the penultimate layer features, yields significantly better results. Additionally, applying softmax scaling enhances the performance of the variational JS bound estimator but degrades the results for the density ratio fitting estimator. Among the estimators, the variational JS bound estimator, combined with softmax scaling, surpasses the density ratio fitting estimator. Regarding critic design, the separable critic slightly outperforms the joint critic. We find that the best configuration includes using output layer features with a separable critic and the variational JS bound estimator with softmax scaling, which we adopt for all subsequent experiments.
39

---

Pointwise Information Measures as Confidence Estimators in Deep Neural Networks
Joint Critic
Separable Critic
3
2.5
2
2.0
(T:Y)
1
1.5
0
probabilistic classifier
probabilistic classifier
density ratio fitting
0.5
density ratio fitting
-1
variational JS bound
0.0
variational Js bound
0
20
40
60
80
100
20
40
60
80
100
Epochs
Epochs
Figure 3: Estimation of [[T; Y) where T is the penultimate layer for the MLP model trained on the MNIST dataset. Three estimators are considered: probabilistic classifier (blue line), density ratio fitting (orange line), and variational JS bound (green line). Two critic designs considered: joint (left) and separable (right). Here, the epochs refer to the training of the critic, not the original network,The shaded regions represent the standard deviations.
Table 9: Comparison of Different PMI Estimators (Averaged over 5 Runs with Standard Deviations Included). The best results highlighted bold.
AUROC, x 10
AURC x 10 +
Critic, Estimator
MLP, MNIST
CNN, F-MNIST
MLP, MNIST
CNN, F-MNIST
Without Softmax Scaling,
Penultimate Layer
Joint Critic, Density Ratio Fitting
90.47 + 1.16
89.09 +: 0.96
2.54 + 0.35
11.81 +: 1.37
Joint Critic, Variational IS Bound
87.30 + 1.38
78.53 1.16
3.58 0.51
25.89 = 3.77
Separable Critie, Density Ratio Fitting
78.51 2.43
85.09
1.04
8.24 1.73
16.94 1.61
Separable Critic, Variational JS Bound
76.45 + 3.08
85.56 0.81
6.27 = 0.88
16.10 + 1.18
With Softmax Scaling, Penultimate Layer
Joint Critic, Density Ratio Fitting
89.35 1.87
79.00 3.38
2.62 0.66
22.14 + 3.93
Joint Critic, Variational IS Bound
85.51 5.38
90.31 0.46
4.88 + 2.44
10.19 + 0.48
Separable Critic, Density Ratio Fitting
89.02 1.79
87.45 + 0.82
15.11 - 1.33
Separable Critic. Variational JS Bound
90.12 + 1.44
91.67 + 0.28
2.55 + 0.52
8.40 + 0.16
Without Softmax Scaling, Output Layer
Joint Critic, Density Ratio Fitting
95.36 = 0.38
91.49 = 0.39
1.11
8.68 + 0.44
Joint Critic, Variational IS Bound
91.63 + 0.62
87.39 + 1.01
2.19 0.34
13.95 + 1.82
Separable Critic, Density Ratio Fitting
95.55 + 0.59
86.65 0.61
1.03 + 0.20
14.55
+ 0.98
Separable Critic, Variational IS Bound
92.95 1.94
88.22 0.62
1.57 0.45
12.00 0.81
With Softmax Scaling, Output Layer
Joint Critic, Density Ratio Fitting
93.32 + 1.42
87.33 1 1.25
1.441 + 0.377
13.39 + 1.59
Joint Critic, Variational IS Bound
97.35 1 0.36
91.54 0.22
0.57
0.08
8.52
0.43
Separable Critic, Density Ratio Fitting
97.13 0.33
88.44
0.55
0.67
0.12
13.94
0.73
Separable Critic, Variational JS Bound
97.24 + 0.18
91.97 0.35
0.05 0.57
8.11 0.09
Softmax
95.11 0.48
92.03
0.23
1.38 + 0.16
8.75 + 0.34
D.2.2. COMPARISON OF PSI ESTIMATORS
For this experiment, we consider the two methods: binning and Gaussian described in Section A.3.2. First, we validate the accuracy of these estimators by comparing their estimates with those obtained using the KSG estimator (Kraskov et al., 2004). Note that SMI is the average of PSI over all samples. We compute the SMI between the penultimate layer and the predicted labels during training (100 epochs) for MLP model and MNIST validation dataset. We use 500 projections for both SMI and PSI estimation and 20 bins for the binning method. The results are shown in Figure 4. We observed that the SMI estimates derived from the PSI binning method align more closely with the direct SMI estimates from the KSG estimator than those from the Gaussian method. However, both methods exhibit the same overall trend.
We then evaluate the performance of confidence estimates returned by both the binning and Gaussian estimators (with different number of projections m) on ML. and MNIST, as well as CNN and Fashion MNIST, The confidence ranking
40

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
0.85
0.80
0.75
SI(TYY)
0.70
0.65
SMI (KSG)
0.60
SMI (PSI Bin)
0.55
SMI (PSI Gaussian)
20
40
60
80
100
Epoch
Figure 4: SMI between penultimate layer and predicted labels during training. The plot shows the average SMI over runs for three different estimation methods (KSG, PSI Bin, and PSI Gaussian) across epochs. The shaded regions represent the 95% confidence intervals.
metrics are , and AURC. addition, we assess their performance based on whether softmax scaling is used and whether confidence estimates derived from the penultimate layer features output layer features (before the softmax function). We report the results in Table 10.
Takeaway. We observe that using the output layer features, rather than the penultimate layer features, yields significantly better results. We observe that the Gaussian method yields poor performance for the CNN with Fashion MNIST case, but this could be remedied by simple softmax scaling. We also observe that increasing the number of projections m beyond 500 leads to little or no improvement in the results. We find that the best configuration includes using output layer features with the Gaussian estimator and softmax scaling, which we adopt for subsequent experiments.
D.2.3. COMPARISON PVI ESTIMATORS
We evaluate the performance of confidence estimates returned by the various PVI estimation methods described in Section A.3.3 on MIP and MNIST, well as CNN and Fashion MNIST. The confidence ranking metrics are AUROC, and AURC. In addition, we also consider calibrating the softmax probabilities used to compute the PVI. Similar to PMI and PSI, we assess if the performance improves with softmax scaling. We report the results Table 10.
In addition, we assess their performance based on whether softmax scaling is used and whether the associated probabilities calibrated with temperature scaling before computing the PVI.
Takeaway. We find that calibrating the softmax probabilities before computing PVI, along with applying softmax scaling, significantly improves performance. The best result is achieved by "training from scratch," which means using another trained network with different initialization. We use this as the default estimator for PVI in all experiments.

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Table 10: Comparison of Different PSI Estimators (Averaged over 5 Runs with Standard Deviations Included). The best results highlighted bold.
AUROC X 10S1
AURC 10"
Estimator
MLP, MNIST
CNN, MNISS
MLP, MNIST
CNN, F-MIIST
Without Softmax Scaling, Penultimate
Layer
Binning (m 250)
96.85 + 0.25
85.82 + 0.33
0.63 + 0.06
12.78 0.25
Binning 500)
96.89 0.12
86.13 0.48
0.62 0.03
12.55
0.34
Binning (m 750)
96.92 0.18
86.11 0.51
0.61 0.04
12.56
Binning (m = 1000)
96.90 0.15
86.19 1 0.39
0.62 0.00
12.49 1..9-00.2 0.25
Gaussian (m 250)
96.38 1 0.40
81.74 + 0.68
0.71
16.20 1.40
Gaussian (m 500)
96.46
82.13 = 0.87
0.69 +
15.90 0.39
Gaussian 750)
96.45
82.12 0.89
0.69 0.05
15.87 0.40
Gaussian (m 1000)
96.43 0.28
82.21 0.72
0.70 0.06
15.80 0.27
With Softmax Scaling, Penultimate
Layer
Binning (m 250)
96.08 0.54
88.11 0.12
0.78 0.14
10.94 0.33
Binning (m 500)
96.89 0.12
88.31 0.26
0.73 0.08
10.76 + 0.34
Binning (m = 750)
96.28 0.26
88.35 0.30
0.73 + 0.06
10.73 + 0.19
Binning (m 1000)
96.17 0.32
88.47
0.76 :
10.65 = 0.31
0.33
Gaussian (m 250)
96.05 0.28
88.29
0.79 0.06
Gaussian 500)
95.95 0.16
88.23
0.81 0.04
10.84 0.39
Gaussian (m 750)
96.00 0.21
88.45 0.35
0.81 0.05
10.66 0.26
Gaussian (m 1000)
96.07 + 0.18
88.46 + 0.39
0.79 + 0.03
10.65 + 0.34
Without Softmax
Scaling, Output Layer
Binning (m 250)
97.01
85.88 + 0.75
0.60 0.03
12.89 0.24
Binning (m 500)
97.05 1:6:::00 0.20
85.77 + 0.68
0.59 0.05
13.00 1 0.25
Binning (m 750)
97.05 1 0.13
85.96 + 0.66
0.60 0.03
12.84 1 0.14
Binning 1000)
97.06
85.87 0.75
0.59
12.91 1.225
Gaussian 250)
96.68
80.40 1.02
0.05
17.62 0.67
Gaussian (m 500)
0.37
80.38 = 0.69
0.07
17.63 0.58
Gaussian (m 750)
96.72 0.26
80.55 = 0.62
0.65 + 0.06
17.46 0.31
Gaussian (m 1000)
96.71 + 0.23
80.54 + 0.63
0.66 + 0.05
17.47 T7.7=:00 0.36
With Softmax
Scaling, Output Layer
Binning (m 250)
96.15 0.43
85.96 +: 1.06
0.79 0.11
13.31 + 0.84
Binning (m 500)
96.24 0.52
85.89 = 0.86
0.76 1 0.13
13.38 + 0.67
Binning (m 750)
96.31 0.39
86.23 0.85
0.10
13.10
Binning (m 1000)
96.31 0.41
86.12 0.94
0.10
13.19
Gaussian (m 250)
96.58 0.21
89.46 0.46
0.05
9.97 0.23
Gaussian (m 500)
96.61 0.18
89.42 + 0.42
0.69
10.00 0.25
Gaussian (m 750)
96.59 0.17
89.44 + 0.49
10.00 + 0.25
Gaussian 1000)
96.62 0.22
89.36 + 0.50
0.68 + 0.05
10.05 0.23
Softmax
95.11 1 0.48
92.03 + 0.23
1.38 0.16
8.75 = 0.34
42

---

Pointwise Information Mcasures as Confidence Estimators in Deep Neural Networks
Table 11: Comparison of Different PVI Estimators (Averaged over Runs with Standard Deviations Included). The best results are highlighted bold.
AUROC, 10 t
AURC 10 +
Estimator
MLP, MNIST
CNN, F-MNIST
MLP, MNIST
CNN, F-MNIST
Uncalibrated, Without Softmax Scaling
No training
75.65 + 1.93
83.11 + 0.77
6.33 = 0.56
22.16 = 0.26
Training from scratch
82.79 1.20
89.07 0.06
4.52 - 0.42
12.07 = 0.48
Training MI.P penultimate
65.45 + 1.23
70.53 0.61
9.39 + 0.44
32.45 1.42
Uncalibrated, With Softmax Scaling
No training
92. + 0.48
92.03 + 0.23
1.38 0.16
8.75 + 0.34
Training from scratch
95.85 + 0.43
92.75 + 0.28
1.19
8.24 + 0.37
Training MLP penultimate
88.31 +: 1.37
85,20 + 0.38
3.56 1 0.45
19.51 + 0.44
Calibrated, Without Softmax Scaling
training
88.13 1 0.79
92.37 0.19
2.82 + 0.29
8.10
Training from scratch
90.76 + 1.06
93.28 1 0.26
2.20 1 0.22
7.10 1: 0.20
Training MLP penultimate
80.57 + 1.73
84.51 + 0.31
5.09 + 0.48
17.13 = 0.55
Calibrated, With Softmax Scaling
No training
97.12 0.23
92.68 0.22
0.60 + 0.04
7.43 1.3 0.17
Training from scratch
97.53 = 0.23
93.33 + 0.25
0.54 + 0.03
6.99 + 0.15
Training MLP penultimate
96.84 1: 0.27
91.82 1 0.20
0.72 1 0.06
8.29 1 0.18
Softmax
95.11 + 0.48
92.03 0.23
1.38 + 0.16
8.75
0.34
43


Reviews:
[{"review_id": "EztL4XHvYi", "reviewer": "Reviewer_F6RX", "summary": "The paper explores the use of information-theoretic measures—specifically pointwise mutual information (PMI), pointwise V-information (PVI), and pointwise sliced mutual information (PSI)—to estimate prediction confidence in deep neural networks (DNNs) post-hoc, without modifying network architecture or training. It investigates their theoretical properties (invariance, margin sensitivity, convergence rates) and evaluates their effectiveness in failure prediction and confidence calibration using benchmark computer vision datasets (e.g., MNIST, CIFAR-10, STL-10). Main findings include PVI outperforming PMI, PSI, and existing baselines (e.g., maximum softmax probability) in both tasks, attributed to its balanced invariance and margin sensitivity. Key contributions are: (1) a comparative analysis of PMI, PSI, and PVI, with PVI showing superior performance; (2) theoretical insights into their invariance and geometric properties; (3) sensitivity analysis to sample-wise margin, with PSI correlating most with margin; and (4) convergence rate derivations, suggesting PSI’s advantage over PMI. Experiments also demonstrate PSI’s effectiveness in generating localized saliency maps.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive remarks and valuable suggestions on the paper. Below, we address the concerns and questions raised by the reviewer.\n\n**Weakness 1 (On OoD Experiment)**\n\nThe primary motivation of our work stems from the observation that existing calibration methods can adversely affect failure prediction performance. This motivates our focus on both calibration and failure prediction as key evaluation tasks. We may explore the extension to OoD in future work.\n\n**Weakness 2 (On Takeaway of Results)**\n\nOur results demonstrate that PVI consistently provides strong performance across both failure prediction (Table 2 and Table 11) and calibration (Table 3 and Table 15) tasks.\n\n**Weakness 3 and 4 (On Comparison to Other Methods)**\n\nWe will be expanding our comparisons to include additional post-hoc calibration methods, such as Ensemble Temperature Scaling (ETS), Parameterized Temperature Scaling (PTS), Class-based Temperature Scaling (CTS), Group Calibration (GC), and Consistency Calibration (CC) as suggested by the reviewer. We are currently running these experiments and will include the results as soon as they become available.\n\n**Question 1 (On Softmax Scaling)**\n\nThe softmax function is applied to re-normalize the PI measures for interpretability as probabilities. Unlike standard logits, PI measures are derived from a principled probabilistic framework and thus inherently capture pointwise relationships in the data. In contrast, logits are unnormalized scores lacking direct probabilistic meaning. Importantly, applying softmax to PI values does not remove the uncertainty signal they encode - it preserves the relative structure of uncertainty while transforming the values into confidence-like scores. High PI values (reflecting high confidence or low uncertainty) remain high after softmax normalization, and low values (reflecting low confidence or high uncertainty) remain low.\n\n**Question 2 (On Softmax Correlation to Margin)**\n\nBelow are the correlation between softmax value and margin:\n| Method   | MLP, MNIST | CNN, F-MNIST | VGG16, STL-10 | ResNet50, CIFAR-10 |\n|---|---|----|---|----|\n| softmax  | 0.954±0.011 | 0.935±0.026 | 0.913±0.004 | 0.882±0.009 |\n\nSoftmax output is most sensitive to the gap between the top two logits (assuming all other logits are much smaller), which is what the margin approximation is also capturing. Naturally, it will have the highest correlation with margin. That being said, the goal of this experiment is to empirically verify the properties of PI measures.\n\nAlso note that a measure can have high correlation with margin because it closely tracks the model's logit gap - indicating how \"close\" a sample is to the decision boundary - but this doesn’t guarantee good performance in failure prediction or calibration. Margin-based measures like softmax may appear confident even on out-of-distribution or adversarial inputs, leading to overconfidence. In contrast, other measures might be less tightly coupled to margin but are more sensitive to ambiguity, distributional shifts, and model failures, making them more reliable for detecting errors and producing well-calibrated confidence.\n\n\n**Question 3 (On PVI Estimation)**\n\nWe provided more details on PVI estimation in Appendix A.3.3. Following the approach of Ethayarajh et al. (2022), we estimate the PVI between $X$ and $Y$ by training an auxiliary model with the same architecture. This measures how well $Y$ can be predicted from $X$ using $\\mathcal{V}$. Thus, in a way it’s capturing the confidence of the model $\\mathcal{V}$.\n\n**Question 4 (On ECE Results)**\n\nIn Mukhoti et al. (2020), data augmentation was used to improve the performance of ResNet-50 on CIFAR-10, resulting in a test error of 4.95. In contrast, our test error is 13.72 (Table 9), which accounts for the observed difference in ECE. Nevertheless, we are currently running experiments to reduce the test errors and will share the updated results once available.\n\nReferences:\n- Ethayarajh, K., et al. (2022). Understanding Dataset Difficulty with V-Usable Information.\n- Mukhoti, J., et al. (2020). Calibrating Deep Neural Networks using Focal Loss.", "claims_and_evidence": "Claims are well-supported by experiments (e.g., Tables 2, 11) and proofs (Appendix B), showing PVI’s superiority and PSI’s margin sensitivity. The PVI convergence claim lacks empirical backing beyond theory.", "methods_and_evaluation_criteria": "Using PMI, PSI, and PVI post-hoc is practical; evaluation metrics (AUROC_f, ECE) and datasets (MNIST, CIFAR-10) are appropriate for confidence estimation.", "theoretical_claims": "Checked Propositions 1-5 and Theorem 1 (Appendix B); they’re correct, though Theorem 1’s spherical assumption simplifies real-world applicability.", "experimental_designs_or_analyses": "Failure prediction (4.1) and calibration (4.2) experiments are sound, with robust metrics and baselines. Saliency map design (D.3) is valid.", "relation_to_broader_scientific_literature": "It contribute the understanding of point wise estimation of confidence calibration based on information theory.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. thorough study on point wise  Confidence Estimation\n\nweakness:\n1. did not include the OoD experiment as usual uncertainty estimation papers.\n\n2. Despite the thorough study on PI measures, the paper lack the recommendation or suggestions for readers to apply these measures. I see some improvement upon some of the tasks, but looks like not consistent. I am familiar with the calibration task, but the results improvement seems trivial and the results of MSP not align well with my experience\n\n3. A recent paper[1] also discuss a way to compute sample wise confidence, it would be better to compare with.\n\n4. comparison many other post hoc calibration method is missed table 3.\n\n\n[1] Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighbors", "other_comments_or_suggestions": "1. line 764 has a reference missing problem", "questions_for_authors": "1. Since the paper normalize these PI measures  using a softmax function, does the softmax lose the ability of PI to express uncertainty?\n\n2. For table 1, I wonder what is the correlation between vanilla neural network output(softmax value) and margin? How does it compared to  these PI measure?\n\n3. Can you give more details about \"estimate the PVI between X and Y in the experiment \"?\n\n4. I am a calibration person, for table 3, a normal ECE for a temperature scaled MSP (resent for cifar10 column) is around 2, but your results is 10, which is confusing to me. Maybe using a more standard, well accepted baseline[1] can give a fair comparison.\n\n[1] Calibrating Deep Neural Networks using Focal Loss", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Su5zxwYKOR", "reviewer": "Reviewer_3NDm", "summary": "This paper proposes to use three information-theoretic measures—Pointwise Mutual Information, Pointwise V-Information, and Pointwise Sliced Mutual Information, as post-hoc confidence estimators for neural network predictions. It theoretically analyzes their invariance properties, sensitivity to geometric features, and convergence rates, finding that PVI performs best for confidence calibration and failure prediction tasks", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive remarks and valuable suggestions on the paper. Below, we address the concerns and questions raised by the reviewer.\n\n> Empirical evaluation lacks comparison to the aforementioned lines of work.\n\nThe aforementioned lines of work are not post-hoc calibration methods. These methods do not compute confidence level on the original network outputs and thus will provide unfair comparison. Instead, we are expanding our comparisons to include additional post-hoc calibration methods, such as Ensemble Temperature Scaling (Zhang et al., 2020), Parameterized Temperature Scaling (Tomani et al., 2022), Class-based Temperature Scaling (Frenkel et al., 2021), Group Calibration (Yang et al., 2024), and Consistency Calibration (Tao et al., 2024). We are currently running these experiments and will include the results as soon as they become available.\n\n> It seems that PMI and PSI work well only on simple tasks. My guess is that this is due to estimation complexities. Could the authors clarify whether PMI and PSI indeed encounter those and why?\n\nThe performance gap can be partially explained by the convergence properties of PMI and PSI. Specifically, the convergence rate of PMI depends inversely on the minimum of $p(x)$ and $p(x|y)$ (refer to Eq. (43) in Appendix B.3). When either of these probabilities is low, estimation becomes unreliable due to increased variance. In simpler datasets such as MNIST, class-conditional distributions are more concentrated, resulting in higher values of $p(x)$ and $p(x|y)$, which in turn leads to better convergence and performance for PMI. In contrast, complex datasets often exhibit more dispersed and overlapping class distributions, making these probabilities smaller and estimation more difficult. The same reasoning applies to PSI, which relies on similar terms across projected spaces. This explains why PMI and PSI perform better on simpler datasets and lag behind on more complex ones.\n\nReferences:\n- Zhang et al. (2020). Mix-N-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning.\n- Tomani et al. (2022). Parameterized Temperature Scaling For Boosting the Expressive Power in Post-hoc Uncertainty Calibration.\n- Frenkel et al. (2021). Network Calibration By Class-Based Temperature Scaling.\n- Yang et al. (2024). Beyond Probability Partitions: Calibrating Neural Networks With Semantic Aware Grouping.\n- Tao et al. (2024). Consistency Calibration: Improving Uncertainty Calibration via Consistency Among Perturbed Neighbors.", "claims_and_evidence": "No problematic claims made.", "methods_and_evaluation_criteria": "Some of the results (especially bold notation, for example in table 3) are unclear.\nHowever, if I understand the results correctly, the empirical evaluation supports the made claims.", "theoretical_claims": "Definitions and assumptions regarding the information theoretic measures are clearly presented. \nThe theoretical analysis (invariance, sensitivity to margin, intrinsic dimensionality, and convergence) is well-reasoned and appears mathematically sound.", "experimental_designs_or_analyses": "The experimental design seems convincing, except two issues:\n- Results are reported only over 5 repetitions\n- Comparison to prominent approaches of uncertainty estimation, such as Bayesian neural networks, is missing", "relation_to_broader_scientific_literature": "Connection and comparison to multiple lines of work Approaches for uncertainty estimation is missing:\n-  Ensembles based methods\n[e.g., Lakshminarayanan et al.]\n- methods based on noise addition [e.g., Maddox et al., 2019]. \n- Bayesian neural networks [e.g., Tishby et al., 1989, Denker and LeCun, 1990, Ovadia et al., 2019, Kingma et al., 2015, Gal and Ghahramani, 2016, Graves, 2011, Louizos and Welling, 2016]", "essential_references_not_discussed": "No critical previous works seem to be omitted except the aforementioned lines of work.", "other_strengths_and_weaknesses": "The paper provides a theoretically solid and interesting contribution that enhances confidence estimation using information-theoretic measures. It provides thorough theoretical analysis, however empirical evaluation lacks comparison to the aforementioned lines of work.\nI am willing to raise my score if the authors provide sufficient evidence that their method at least provides comparable results to these previous methods.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "It seems that PMI and PSI work well only on simple tasks. My guess is that this is due to estimation complexities. Could the authors clarify whether PMI and PSI indeed encounter those and why?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QJNK1OikOp", "reviewer": "Reviewer_Kkv7", "summary": "The authors propose a new indicator for quantifying the confidence of a neural network: the pointwise information between an input’s features and the corresponding output. When the information is high (corresponding to a large reduction in the entropy of the true label when conditioned on the input), we expect that the input features are especially useful for making predictions (thus confidence should be higher).\n\nThree measures of pointwise information are compared: mutual information, V-information, and sliced mutual information. The authors first investigate these measures along the dimensions of transformation invariance, sensitivity to geometric properties of the feature distribution, and the convergence rates of their estimators. They then conduct a series of experiments that demonstrate the usefulness of these PI measures for failure/selective prediction and confidence calibration.\n\n### Update after rebuttal\nMy main concerns about this work still remain — namely, that the empirical results are not strongly motivated or justified by the theoretical discussion (as currently presented), and that the experiments in their current form are not convincing enough for practitioners to consider adoption of this method.\n\nThis paper would be significantly strengthened by the use of more-realistic datasets (not toy ones like MNIST, Fashion-MNIST, CIFAR-10, etc.), especially if such experiments can demonstrate a clear and consistent advantage to PI estimation methods over existing UQ techniques. I maintain my score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive remarks and valuable suggestions on the paper. Below, we address the major concerns and questions raised by the reviewer. We will address all other points in the revision.\n\n**Question 1 (On Invariance)**\n\nWe would like to clarify that we meant bijective linear transformations (in Remark 1). If the PI measures are not invariant to linear transformations, then it could pose an issue. We highlight this point with the following example. Let us consider PMI between a neural network layer $T$ and the output labels $Y$, and assume that $T’$ denotes another rendition of $T$ which has the same information but arises from a different initialization of the network. If the relationship between $T$ and $T’$ is linear, then PMI’s invariance is helpful. However, if the relationship is non-linear, as PMI’s invariance to non-linear invertible and continuous transformations also means that it outputs the same degree of uncertainty estimation when $T’$ is related to $T$ in a non-linear manner. If the function is highly non-linear, then the estimated label for $T’$ should have a different level of confidence compared to $T$, as the neural network’s remaining layers are limited in the ways it can transform the features $T’$ (considering finite networks). Therefore, in this case, the heavily invariant nature of PMI can be counterproductive for reflecting the uncertainty of predictions. Note that if PMI’s invariances were limited to linear transformations, this won’t be an issue. \n\n**Question 2 (On PVI Superiority)**\n\nFor MNIST, where classifiers achieve very high accuracy and most examples are correctly classified, AUROC can give an overly optimistic view of performance, resulting in smaller observed differences across methods. For STL-10, we plan to run additional repetitions to reduce performance variability. In the case of CIFAR-10, when considering confidence intervals rather than standard deviations, PVI clearly outperforms all other methods for AUROC. Furthermore, PVI consistently achieves the best results in terms of AUPR (error), except for STL-10, which - as noted - may require more repetitions for stable evaluation. This indicates that PVI reliably assigns lower confidence to misclassified examples.\n\n**Question 3 (On Sample-Wise Margin)**\n\nA direct, 1-to-1 comparison with a unified sample-wise margin framework is challenging due to the significantly different nature of each measure. However, we believe there is much to be learned from their individual contributions, and we will emphasize how each result adds value to understanding uncertainty and confidence estimation. Prop. 4 shows that PMI is insensitive to margin when the classes are well-separated. This observation is motivated by prior work (e.g., Grønlund et al., 2020), which connects margin to generalization and prediction confidence. Thus, PMI’s inability to encode margin under such conditions is particularly relevant in the context of confidence estimation. In contrast, PSI and PVI retain meaningful notions of sample-wise margin. These more general measures can naturally extend to the case where class-conditional distributions $P(x|y=0)$ and $P(x|y=1)$ are non-overlapping—for example, by setting $\\epsilon =0$ in Eq. (9), Prop. 5 holds without further assumptions.\n\nEmpirically, we observe that PSI correlates most strongly with margin (Table 1), aligning with this theoretical intuition. However, it's important to distinguish between tasks: in the correlation-to-margin experiment, confidence reflects a sample’s sensitivity to decision boundaries, regardless of correctness. In contrast, tasks like misclassification detection, selective prediction, and calibration emphasize predictive reliability, where confidence is tied to accuracy. This difference in interpretation explains why PSI excels in margin-based correlation while PVI outperforms in accuracy-driven tasks. The results are not inconsistent but instead highlight the differing emphases of these evaluation criteria.\n\n**Question 4 (On Computational Cost)**\n\nWhile training additional models required for PI estimation can be computationally expensive, this training is performed only once. Inference, on the other hand, is efficient. For PVI, for example, the inference time is comparable to standard label prediction. Moreover, if multiple models from different runs are available, they can be directly reused for PVI estimation without retraining. We hope our findings can motivate future research toward developing more computationally efficient approaches for pointwise information estimation.\n\n**Question 5 (On PVI & Model Architecture)**\n\nWhile PVI assumes access to the model architecture, it does not require any modifications to the network architecture (as in MC Dropout) or training procedure (as in focal loss). Such alterations change the model’s prediction behavior, whereas PI methods operate post hoc do not alter the original predictive outputs.", "claims_and_evidence": "The authors claim that pointwise information (PI) is a useful indicator of predictive uncertainty and that PVI outperforms other PI estimation methods and UQ benchmarks. While experiments offer partial support, the connection between empirical evidence and theoretical analysis is poorly demonstrated.\n\nIn Section 3, invariance is emphasized as crucial for confidence estimation, yet PMI—the most invariant method—does not perform best. The justification for this (that excessive invariance may be counterproductive) is unconvincing and inconsistent. Remark 1 states that “it is important to be invariant to bijective transformations T in the context of confidence estimation”, and that the “ideal scenario is when the above is true for any invertible, and thus information-preserving transformation T”. However, Remark 9 suggests “the fact that PMI is invariant to a much larger degree of non-linear homeomorphisms may not always be advantageous”. These claims directly contradict each other. If invariance matters for confidence estimation, why does it not lead to observably better empirical performance? I would have appreciated an explicit demonstration of this principle, perhaps in an experiment where each method is applied to intentionally transformed data and results are compared.\n\nThe discussion of sample-wise margin (which seems to refer to class separation in feature space) lacks a unifying theory across PMI, PSI, and PVI, making Propositions 4 & 5 and Theorem 1 difficult to contextualize. Consequently, takeaway T3 appears unconvincing. Moreover, the correlation-to-margin experiment, combined with the results of Section 4, suggests that margin sensitivity does not directly drive confidence estimation performance, calling into question its relevance to the main analysis. This section might be better placed in the appendix.\n\nTakeaway T4, on convergence rates, is the most practically relevant, as a PI estimator’s performance should correlate with the usefulness of its information scores for UQ. However, the theoretical discussion is buried in the appendix—it should be in the main body.\n\nSection 4 presents mixed empirical results, with no method universally superior (as evidenced by the plethora of statistical ties in Tables 2 and 3). While PVI is the most consistent top performer, it is not dominant. The claim that PVI is “the most well-rounded” across invariance, margin sensitivity, and convergence rate is unconvincing—these factors are neither comprehensive nor clearly predictive of empirical success. Theoretical takeaways seem driven by empirical results rather than the other way around, weakening the paper’s core claims.\n\nOverall, while the paper presents interesting theoretical insights, the lack of a clear and consistent connection between theory and empirical results weakens its central claims. Key theoretical principles, such as invariance and margin sensitivity, are not convincingly linked to empirical performance, and in some cases, the arguments appear contradictory. Additionally, the empirical results do not strongly support the claim that PVI is the best method, as its advantages seem marginal rather than definitive. A more thorough justification of the theoretical takeaways, along with targeted experiments to validate key assumptions, would significantly strengthen the paper’s contributions.", "methods_and_evaluation_criteria": "I am concerned with the authors’ use of ECE for calibration, given its known issues with discontinuity [1] and binning dependence [2].\n\nThe absence of transformer-based models leaves open the question of whether the reported PI method improvements hold for more modern architectures. This is particularly relevant since transformers' softmax probabilities have been shown to calibrate better than those of ConvNets [3].\n\nThe dataset choices are also underwhelming. The lack of standard-resolution (224x224) images raises concerns about scalability, as results on toy datasets like MNIST and F-MNIST may not generalize to realistic settings. It is disappointing that the largest datasets (Tiny-ImageNet, DS-ImageNet) were relegated to the appendix, with DS-ImageNet seemingly missing benchmarks for several algorithms.\n\n[1] Błasiok, J., Gopalan, P., Hu, L., & Nakkiran, P. (2023, June). A unifying theory of distance from calibration. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing (pp. 1727-1740).\n[2] Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G., & Tran, D. (2019, June). Measuring Calibration in Deep Learning. In CVPR workshops (Vol. 2, No. 7).\n[3] Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., ... & Lucic, M. (2021). Revisiting the calibration of modern neural networks. Advances in neural information processing systems, 34, 15682-15694.", "theoretical_claims": "I did not check the correctness of any proofs.", "experimental_designs_or_analyses": "The experimental designs appear sound, but the analysis in Section 4 offers little beyond summarizing tables in words. Several claims also warrant scrutiny:\n\n- For F-MNIST, the authors attribute PVI’s superiority over PMI and PSI to its “well-rounded” nature, citing invariance and margin sensitivity. However, it’s unclear why this explanation applies specifically to F-MNIST but not to CIFAR-10, where PVI and PSI are statistically tied, or to MNIST, where PVI and PMI perform similarly.\n- In Section 4.2, the claim that “PVI significantly outperforms [all benchmarks] when assessing the average ECE… by a large amount” is questionable. While the means in Table 3 support this, the large standard deviations undermine the statistical significance of the difference.\n- On page 8, the authors assert that “for average ECE, it seems that the improvement [of PVI over PMI and PSI] for more complex datasets and architectures is more significant.” This claim would be far more convincing if PMI/PSI results were reported for Tiny-ImageNet and DS-ImageNet. Without them, the argument for PVI’s superiority on complex datasets remains unsubstantiated.", "relation_to_broader_scientific_literature": "This study builds directly on Zhu et al. (2022), which demonstrated that common calibration techniques—such as label smoothing, mixup, focal loss, and temperature scaling—can inadvertently exacerbate the confidence gap between correct and incorrect predictions. Since these methods primarily operate by adjusting softmax probabilities, the authors explore whether alternative confidence estimation approaches, such as pointwise information (PI), can mitigate this issue. While the paper does not introduce new methods for estimating PI, its key contribution lies in the novel application of these techniques to confidence calibration and uncertainty quantification.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "The application of pointwise information (PI) for confidence estimation is a novel and promising research direction. I commend the authors for this innovative approach and hope future work will further validate their hypothesis.\n\nHowever, the manuscript would benefit from improved clarity and coherence, particularly in its argumentative structure. The theoretical and empirical sections feel somewhat disjointed, with the developed theory not clearly informing or justifying the experimental results. As a result, the theoretical contributions appear underutilized. Since the empirical results are middling, this lessens the overall significance of the current contribution.\n\nA key limitation not yet discussed is the computational cost of the proposed approach. The authors frame their method as a post-hoc alternative, emphasizing the practical challenges of modifying network architectures or retraining models. While technically post-hoc (in that it does not require retraining the original model), the proposed approach is highly computationally expensive. This is evident from the study’s restriction to smaller datasets and lower-resolution images: “we do not report the results [on Tiny -ImageNet and DS-ImageNet] for PMI and PSI as they are very computationally expensive for large-scale datasets”. If my understanding is correct, each PI estimate requires training a separate neural network—sometimes with the same architecture, which may not always be accessible in black-box settings. Given the mixed empirical results, the computational overhead may not be justified, raising concerns about the method’s practicality.\n\nOverall, while the paper introduces an interesting application of PI for confidence estimation, its practical viability remains uncertain due to the high computational cost and the lack of a clear theoretical-to-empirical connection. Strengthening the coherence between these aspects and further justifying the trade-offs involved would significantly enhance the paper’s impact.", "other_comments_or_suggestions": "It is atypical to see the ECE reported in (what I think) are 100x amounts. ECE is typically bounded by [0, 1].\n\nTable 9 reports exceptionally high error rates for ResNet101 trained on DS-ImageNet (80% train error, 85-86% generalization error). Are you sure that your models are adequately trained here? If not, we should question the validity of any confidence calibration results obtained on this dataset. I am also concerned about the 52% test error rate for DenseNet121 trained on Tiny ImageNet (while the validation error is 11%).\n\nI noticed a few typos / grammatical errors while reading. Here is a list:\n- Item 3 in “Motivation”, first paragraph: “We note that this measure is rooted in probability, and estimates priors and posterior probability measures” (did you mean to put prior here? The plural doesn’t make as much sense)\n- Item 3 in “Motivation”, first paragraph: “This is unlike the typical neural network output, which, although *it* is supposed to model the conditional probabilities of each class p(y|x), often turn*s* out to be not a good indicator of the true uncertainty” (changes marked with stars)\n- Item 3 in “Motivation”, first paragraph: “By doing so, *they* can potentially reduce inherent bias…” (changes marked with stars)\n- Item 1 in “Contributions”: “We found that PVI outperforms PMI and PSI”. Did you mean to switch into past tense here?\n- In equation 9, I believe you replaced $psi$ with $\\psi$. You do not define $\\psi$ anywhere in the paper.\n- Item T3 in Section 3.3: “we see PMI to be invariant to hard margin, and yet PSI being sensitive to hard margin” (the italicized portion should be replaced with something like “while PSI is sensitive to hard margin”).", "questions_for_authors": "1. The authors discuss the potential counterproductivity of excessive invariance, yet the theoretical importance of invariance is emphasized. Could the authors elaborate on how invariance relates to predictive performance? \n2. In Section 4, the authors claim that the superior performance of PVI over PMI and PSI on F-MNIST is due to PVI’s “well-roundedness.” Why do the authors believe this characteristic is particularly beneficial for F-MNIST, and not for datasets like CIFAR-10 or MNIST, where PVI does not show the same degree of superiority?\n3. The concept of \"sample-wise margin\" appears to be inconsistent across the methods (PMI, PSI, PVI). Can the authors provide further clarity on how this property should be interpreted or applied in practice, especially in light of the conflict between the correlation-to-margin results and Section 4?\n4. Given the computational cost of PI methods, do the authors see any feasible ways to reduce the complexity of PI estimation while retaining its benefits?\n5. Does PVI assume access to the model architecture, and if so, how does that align with the authors’ proposed use in black-box settings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "A6oXr9nk9x", "reviewer": "Reviewer_Wh94", "summary": "This paper presents a comparative analysis of three point-wise information measures—PMI, PVI, and PSI—from both theoretical and empirical perspectives. The study theoretically analyzes the sensitivity of these measures to margin effects and intrinsic dimensionality, along with their convergence rates. Empirical validation is conducted through numerical experiments using benchmark computer vision models and datasets, evaluating the measures' performance in failure prediction and confidence calibration tasks.\n\n## Update after rebuttal\n\nI keep my score unchanged, and I think the authors addressed my concerns well.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive remarks and valuable suggestions on the paper. Below, we address the concerns and questions raised by the reviewer.\n\n**Weakness 1 (On the Assumption for PVI's Theoretical Properties)**\n\nThis is a valid point, and we will explicitly specify the limitations of this assumption for our theoretical results relating to PVI. However, having said that, we note that our results can be extended to non fully-connected networks by considering the input as the feature layer that eventually feeds into a fully-connected architecture. For instance, we can consider the feature output after the convolution layers in a CNN, which is input to a fully connected set of layers before yielding the final output. For those cases, our results are still applicable. \n\n**Question 1 (On the Misspecification of $\\mathcal{V}$)**\n\nYes, misspecification of $\\mathcal{V}$ can reduce the applicability of both the theoretical and empirical results in our work. However, in our work we assume that the knowledge of the model’s architecture is available. In the cases when the model is entirely a black box, we won’t be able to arrive at our theoretical and empirical results in this paper. To that end, we are including additional experiments where we either over or under-specify the complexity of $\\mathcal{V}$, and see how it affects the results. \n\n**Question 2 (On the Selection of $\\mathcal{V}$)**\n\nIdeally, $\\mathcal{V}$ should match the architecture of the model used to generate the predicted outputs. This alignment ensures that the PVI estimation accurately reflects the confidence characteristics of the original model.\n\n**Question 3 (On the $AUPR_{f,error}$)**\n\nSince $AUPR_{f,error}$ emphasizes the accurate ranking of rare error cases (i.e., treating incorrect predictions as positives and using low confidence scores as indicators), methods that can precisely distinguish which specific samples are likely to be errors will naturally perform better. It is essentially asking “How well can we rank and detect errors using the model’s uncertainty/confidence score?” PVI outperforms others for this metric reflects its ability to identify misclassification, even when errors are rare. We hypothesize that this may be attributed to PVI's incorporation of both the network architecture and the prior probabilities of the output classes, which helps mitigate overconfidence on misclassified samples.", "claims_and_evidence": "The claim that PVI is the \"most well-rounded\" measure cannot be justified solely by analyzing invariance and margin sensitivity. By definition, PVI critically depends on the predictive family $\\mathcal{V}$. Improper specification of $\\mathcal{V}$ may significantly degrade PVI's performance, necessitating rigorous theoretical and empirical investigation into $\\mathcal{V}$'s role. While Tables 2-3 demonstrate PVI's superiority over post-hoc baselines, two critical questions remain unresolved: \n\n1. How severely does misspecification of $\\mathcal{V}$ impact PVI's effectiveness? \n\n2. What principles should guide the selection of $\\mathcal{V}$ for optimal PVI performance?", "methods_and_evaluation_criteria": "While the proposed experimental framework appears methodologically sound, three critical enhancements are required:\n\n1.\tSystematic investigation of PVI's dependence on $\\mathcal{V}$ across parameter configurations.\n\n2.\tQuantitative benchmarking against baseline methods under controlled $\\mathcal{V}$ variations.\n\n3.\tCausal analysis elucidating why PVI outperforms post-hoc alternatives in specific $\\mathcal{V}$ regimes.", "theoretical_claims": "I have reviewed the proofs provided in the Supplementary Materials. The proofs demonstrate scientific validity under the specified theoretical assumptions outlined in this study.", "experimental_designs_or_analyses": "The experimental design and analytical validity were checked. The experiments in the paper properly evaluated PMI, PSI, PVI, and other post-hoc methods under controlled experimental parameters. Detailed supplementary documentation delineates performance variations across operational conditions for PMI, PSI, and PVI. Despite this, further controlled ablation studies remain necessary to elucidate the relationship between PVI and the predictive family $\\mathcal{V}$.", "relation_to_broader_scientific_literature": "The study systematically compared three uncertainty measures (PMI, PSI, and PVI) through theoretical analysis and empirical validation, demonstrating PVI's promising performance in uncertainty quantification for deep learning tasks. These findings not only advance key applications like failure prediction and misclassification detection, but also provide novel insights for enhancing model robustness against adversarial attacks.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tThe comparison of point-wise versions of MI, SI, and VI provides an insightful and significant contribution to quantifying uncertainty in deep learning models. The paper offers both theoretical and empirical comparisons of these measures.\n2. The paper demonstrates that PVI outperforms other post-hoc methods in most failure prediction and confidence calibration scenarios. This suggests PVI could be a promising approach for exploring adversarial robustness.\n   \nWeaknesses:\n\n1. The theoretical framework requires further development by relaxing the stringent assumption in Section 3.1 that requires $\\mathcal{V}$ to be a fully connected neural network.\n2. The role of $\\mathcal{V}$ warrants more rigorous theoretical analysis and empirical ablation investigation.", "other_comments_or_suggestions": "Typos:\n\n1. Lines 066-069 in right column:  By doing so, .... in the data.\n2. line 1212: sa1me", "questions_for_authors": "1.\tTo what extent does misspecification of $\\mathcal{V}$ influence the efficacy of PVI?\n2.\tWhat guiding principles should inform the selection of $\\mathcal{V}$ to optimize PVI performance?\n3.\tIn Table 2's third numerical column, why does PVI exhibit consistently larger confidence intervals when AUPR$_{f,error}$ serves as the evaluation metric?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work considers three information-theoretic measures (Pointwise Mutual Information, Pointwise V-Information, and Pointwise Sliced Mutual Information) to estimate the confidence of predictions from trained neural networks after training. Through a thorough theoretical analysis, it is found that Pointwise V-Information provides the most reliable measure for confidence calibration and failure/selective prediction. 

This is an interesting and thorough study, reflected in generally positive scores from the reviewers. Initially, reviewers had concerns over the simplicity of the models and datasets considered in the experiments, although this has been thoroughly addressed in the author rebuttal to Kkv7. This is a _major_ improvement to the manuscript. I strongly suggest that these results are included in the final revision, along with the results for the additional calibration approaches. There is a wide array of theoretical results on display, and all of the major concerns about the theoretical claims raised in the reviews have been addressed during the rebuttal period as well. 

In my estimation, the authors have responded well to each concern and have produced a solid contribution to ICML. I recommend acceptance.
Decision: Accept (poster)