Beyond Confidence: Reliable Models Should Also Consider Atypicality
Authors: Mert Yuksekgonul
Linjun Zhang
James Zou
Carlos Guestrin
Venue: NeurIPS 2023 poster (2023)
Keywords: trustworthy machine learning
reliable machine learning
uncertainty
TL;DR: We find that predictions for atypical (or rare) points are more miscalibrated (overconfident), and have low accuracy; thus, we propose that models should use atypicality, and simple atypicality-aware methods yield significant performance gains.

Abstract: While most machine learning models can provide confidence in their predictions, confidence is insufficient to understand a prediction's reliability. For instance, the model may have a low confidence prediction if the input is not well-represented in the training dataset or if the input is inherently ambiguous. In this work, we investigate the relationship between how atypical~(rare) a sample or a class is and the reliability of a model's predictions. We first demonstrate that atypicality is strongly related to miscalibration and accuracy. In particular, we empirically show that predictions for atypical inputs or atypical classes are more overconfident and have lower accuracy. Using these insights, we show incorporating atypicality improves uncertainty quantification and model performance for discriminative neural networks and large language models. In a case study, we show that using atypicality improves the performance of a skin lesion classifier across different skin tone groups without having access to the group attributes. Overall, we propose that models should use not only confidence but also atypicality to improve uncertainty quantification and performance. Our results demonstrate that simple post-hoc atypicality estimators can provide significant value.

Beyond Confidence: Reliable Models Should Also Consider Atypicality
Mert Yuksekgonul Stanford University merty@stanfore ord. edu
Linjun Zhang Rutgers University 1z4120stat. www.eaat...tttrr....oomm edu
James Zouł
Carlos Ernesto Guestrin Stanford University, CZ Biohub guestrin@stants edu
Stanford University j amees2sttanford. edu
Abstract
While most machine learning models can provide confidence in their predictions, confidence is insufficient understand a prediction's 's reliability. For instance, the model may have a low confidence prediction if the input is not well-represented in the training dataset or the input is inherently ambiguous. In this work, we investigate the relationship between how atypical (rare) sample or a class is and the reliability of model's predictions. We first demonstrate that atypicality is strongly related to miscalibration and accuracy. In particular, we empirically show that predictions for atypical inputs or atypical classes are more overconfident and have lower accuracy. Using these insights, we show incorporating atypicality improves uncertainty quantification and model performance for discriminative neural networks and large language models. In a case study, we show that using atypicality improves the performance of skin lesion classifier across different skin tone groups without having access the group attributes. Overall, propose that models should use not only confidence but also atypicality to improve uncertainty quantification and performance. Our results demonstrate that simple post-hoc atypicality estimators can provide significant value."
Introduction
Typicality is an item's 's resemblance to other category members [RM75]. For example, while dove and sparrow are typical birds, a penguin is an atypical bird. Many works from cognitive science (e.g.. [Rip89, RSS73, MP80J) suggest that typicality plays crucial role in category understanding. For instance, humans have been shown to learn, remember, and refer to typical items faster [Mur04]. Similarly, the representaatveeness heuristic is the tendency of humans as a basis for decisions [TK74]. This cognitive bias is effective for making swift decisions, but it can lead to poor judgments of uncertainty. For instance, the likelihood of typical events can be overestimated [TK74] or uncertainty judgments can inferior for atypical events [TK92].
the typicality of event
While it is hard quantify the uncertainty of human judgments, machine learning models provide confidence in their predictions. However, confidence alone can be insufficient to understand reliability of prediction. For instance, a low-confidence prediction could arise from an ambiguity that is casily communicated, or due to the sample being underrepresented in the training distribution.
Our code is available at http:///wthhttttoooee/////eeoota Joint Advisors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

High Confidence
Reliable Quadrant
Extrapolation Quadrant
Extrapolation
Rallable
Typical
Atypical
Ambiguous Quadrant
Untrustworthy Quadrant
Unirustworthy
Ambiguous
Confidence
Figure I: Atypicality Uncertainty. Left: We show examples from the ImageNet-R dataset with our atypicality framework. Right: We provide conceptualization of the quadrants. Using atypicality, we can understand prediction quality ($3), improve predictions (544, and prediction sets ($5).
Similarly, high- cogficcnfieeece prediction could be reliable miscalibrated. Our main proposal is that models should quantify not only the confidence but also the atypicality to understand the reliability of predictions the coverage of the training distribution. However, many machine learning applications rely on pretrained models that solely provide confidence levels, devoid of any measure of atypicality.
Contributions: To support our position, we simple formalization of atypicality estimation. With the following studies, we show that by using simple atypicality estimators, we can:
1. Understand Prediction Quality: Calibration is measure that assesses the alignment be- tween predicted probabilities of a model and the true likelihoods of outcomes [GR07]. Neural networks [GPSW17] or even logistic regression [BMWX21a] can be miscalibrated out-of-the Here, we argue that using atypicality can give insights into when a model's confidence reliable. Through theoretical analysis and extensive experimentation, we demonstrate that aty atypicality results in lower-quality predictions. Specifically, we show that predictions for atypical inputs and samples from atypical classes are more overconfident and have lower accuracy.
Recalibration methods offer some mitigation miscalibra- tion [GPSW17] by adjusting probabilistic model. We show that models need different adjustments according to the atypicality of inputs and classes, and atypicality is a key factor in recalibration. In light of these findings, we propose simple method: Arypicality-Awan calibration algorithm takes into account the atypicality of the inputs and classes and is simple to implement. We show that complementing recalibration methods with atypicality improves uncertainty quantification and the accuracy of predictors. Further, in case study for skin lesion classification, we show that atypicality awareness can improve performance across different skin-tone subgroups without access to group annotations.
Improve Calibration and Accuracy:
Recalibration. Our re-
3. Improve Prediction sets: An alternative approach to quantify uncertainty is to provide prediction sets that contain the label with high probability [ABMJ20]. Here, we investigate existing methods with atypicality and show that prediction sets could underperform for atypical or low-confidence samples. By using atypicality, we demonstrate the potential for improving prediction sets.
Overall, we propose that models should also consider atypicality, and we show simple- and easy-to-implement atypicality estimators provide significant value.
2 Interpreting Uncertainty with Atypicality
Motivation: In many machine learning applications, we have access to a model's confidence. which aims to quantify the likelihood that a prediction will be accurate. In classification, model output is probability distribution over classes and confidence is the predicted probability of the top class, i.e. max, iP(Y = y X = z). In practical scenarios, confidence the primary tool used to evaluate the reliability of prediction where higher confidence is associated with better predictions. However, the uncertainty in confidence can stem from different sources that require different treatment AMKvA+211

---

Here, we call a prediction reliable if it is high-confidence and well-calibrated. High confidence could be reliable or miscalibrated, and low confidence could be due to ambiguity or rare inputs. We propose that atypicality provides a natural way to understand reliability when combined with confidence. A sample called typical if is well-represented in the previously observed samples, e-g. an image of a dog that is similar to other dogs in the training data. However, if the image is unlike any other seen during training, it is atypical. We argue that atypicality can help us interpret a prediction's s reliability. Below we categorize samples and predictions according to atypicality and confidence in four quadrants (Figure 1).
High-confidence and representative: Reliable predictions often fall within the Reliable Quadrant, which includes typical, highh-coffdence samples. These samples are rell-reppesented in the training dataset (typical), thus we expect the high-confidence prediction to be reliable. For instance, the first image on the top left (Figure 1) is a typical golden retriever and the model makes a reliable prediction.
High-confidenee yet far from the support: Having high-confidence does not always indicate reliability. If the sample does not have support in the training distribution, the confidence could be miscalibrated. Such samples lie in the Extrapolation Quadrant which contains atypical, high- confidence samples. For instance, the second image in the top right of Figure 1 a toy hog and the model has not seen similar ones during training.
Low confidence due to ambiguity: In contrast, low confidence could also be reliable when it correctly reflects an ambiguity. Such samples are in the Ambiguous Quadrant that contains typical, low-confidence samples. These are typical since they may represent multiple classes; yet, due to ambiguity, the model's confidence is low. For instance, the second image in the bottom left of Figure 1 can both be hog and comic book.
Low confidence and rare: For samples that are not well-represented in training data, we expect to have low-quality predictions. Untrustworthy Quadrant comprises atypical, low-confidence samples that can include extremely rare subgroups, for which we expect miscalibration and lower accuracy. For example, the image in Figure 1 bottom right is an origami hog that was not seen in training.
These examples suggest that relying solely on confidence does not provide complete understanding of the reliability of the predictions, and we can use atypicality to interpret and improve reliability.
Formalizing Atypicality: Atypicality here is defined with respect to the training distribution. Informally, an input or a class is atypical if it is not well-represented in the training distribution. For instance, if there are no or limited similar examples an input, it can be called atypical. Note that this notion is not restricted to being oottof-diistrbuution [HG]66 6a], since in-distribution groups could also be atypical or rare, and our goal is perform reliably for the entire spectrum.
Formally, let X E Rdd be the random variable denoting features and Y E e = -11.2...,C) denote the class, where focus classification.
Definition 2.1 (Input Atypicality). We define the atypicality of the input x as²
ax (z)
max log P(X x|Y = y).
We use the logarithm of the cladscconiiiional densities due to high dimensionality and density values being close to zero. Intuitively, for dog image æ. (PX x Y dog) has a low value, we call an atypical dog image. Overall, if a(x) is high, then we call x an atypical input. Specifically, if an input is not typical for any class, then it atypical with respect to the training distribution. Similarly, we can also use marginal density, P(X- x)., or distance to quantify atypicality.
Similarly, the notion of atypical (rare) classes is prevalent in imbalanced classification [CWG+ 19, ZCLJ21]. Ensuring reliable performance for atypical classes can be safety-critical, e.g., for a rare presence of dangerous melanoma [DVN+22] 22]. We define class atypicality in the following:
Definition 2.2 (Class Atypicality). For class y, atypicality of a class is defined as
ay (v) log P(Y = y).
2 Here atypicality differs from 'typccal sets' in information theory that refers to a sequence of variables [TJ06]. For an input æ. if the nearest neighbor (NN) distance is large. then call x atypical as all inputs in the training set are far from x. Density and distance are connected through non non- parameeettiic density estimation and [JKGGI 88] shows that NN distance can recover high-density regions.
When the meaning is unambiguous, omit the subscript denote a(X) a{Y") for notational brevity.
3

---

Estimating Atypicality for Discriminative Models: Quantifying input atypicality requires access to the class-conditional marginal distributions. In practice, for neural networks trained for classification, these distributions are unavailable and we need to perform the estimation. This estimation can be challenging if the dimensionality is large, or the data is unstructured, requiring assumptions about the distributions. Prior works [MKvA 11K,A+21, LLLS18) showed that Gaussian Mixture Models (GMMs) in the embedding space of neural networks can be used to model these distributions.
In experiments, we use Gaussians with shared covariance, i.c. P(X x = N(p. E), to estimate input atypicality. We perform the estimation in the penultimate layer of neural networks used to make predictions, using maximum- -inellioiie estimation with samples from the training data. We explore other metrics, such as k-Nearest Neighhors distance. We give implementation details and results with different metrics in Appendix A. With these estimators, atypicality estimation is cheap and can run CPU. Our goal is to show that simple estimators can already reap large benefits. Our framework is flexible and exploring more sophisticated estimators is a topic for future work.
Atypicality for LLMs: LLMs are increasingly used for classification [BMR + 20J. Modern LLMs are autoregressive models that compute marginal distribution, PILLM We compute the negative log- likelihood of prompt ay ay()) log (y). Similar to the discriminative setting, atypicality here aims to quantify whether a prompt is well-represented in the training data. A larger value for a(X) would imply that prompt more atypical. Below, we present typical and atypical prompts for the AGNews dataset:
label and use this atypicality metric, i.e. ax (z)
log FL)M(2),
Classify the news articles into the categories World, Sports, Business, and Technology: Article: Salin tallest obstacle to bost #39;s patriotic games hope AS tennis lans go, Houston #39;s Jim #39.Maitress Mack 439; MeIngvale very rich, extremely forthright, exceedingly patriotic unflinchingly Republican. Answer:
Anyicallit: 353.45, Percentile: %694.5
Classify news articles
categories World, Sports, Business, Technology.
Article: Delta Air Lines Prepares Chapter II Filing Delta Air Lines Inc. could file for Chapter 11 bankrupecy protection soon next week. a source familiar with the matter said yesterday.
Answer:
Atypicality: 171.50. Percentile: %0.9
Understanding the Prediction Quality with Atypicality
In this section, we show how our framework can be applied to understand the quality of predictions. Experimental Setup: We investigate three classification settings across a range of datasets:
1. Balanced Supervised Classification: We use ResNet18 1osNet18-50-152 [HZRS16]. WideRes- Net28 [ZK16], RoBERTa [LOG + 19] trained on ImageNet [DDS 00], CIFAR10,100 [Kri09],
MNLI (WNB18) respectively.
2. Imbalanced Supervised Classification: We use ResNet18, ResNext50, ResNet152 trained on CIFAR-LT, ImageNet-LT and Places365-LT where models and data are mostly from [ZCLJ21, MKS+20]. Note that all of the test and validation sets have balanced class distributions.
3. Classification with LLMs: We open-source Alpaca7B [TGZ+*23 TREC [LR02], and AG News [ZZL155 datasets with the prompts from [ZW]+21].
IMDB [MDP+
Details on datasets, models, and prompts are in Appendix B. Our experiments were run on a single NVIDIA A100-80GB GPU. We report error bars over 10 random calibration/test splits.
3.1 Atypicality is Correlated with Miscalibration
We first explore the importance of atypicality to understand model calibration. Calibration quantifies the quality of a probabilistic model [GR07]. Informally, a model considered perfectly calibrated if all events that are predicted to occur P% of the time occur 5% of the time for any P E [0, 100).
For the sake of simplicity, consider a binary classification problem where the predictor is IP. : X [0, 1]. We quantify miscalibration with Calibration Error (CE):
CE(P] [[ = p) pl]-
is computationally infeasible to calculate the above expectation with the conditional probability (()) p). In practice, we use binned version of this quantity, Expected Calibration Erorr (ECE) [NCH15, GPSW17], to estimate CE. See Appendix C. for formal definition.

---

ResNel18 ImageNet ImageNet-R Jontidencel_,.Acccraaay
ResNet50 D Imagellet
ResNext50 ImageNattLJ
Alpaca78 IMDB
0.06
0.30
0.94-1.00
0.03
0.06
0.09
0.12
0.15
0.05
ECE
9.20
FCR
0.05
Confidence
0. 0.86-6,94
0.04
0.12
0.20
0.26
0.35
0.5
9.41 0.66
0.05
0.12
0.19
0.22
0.27
Accurecy
Accuracy
Accuracy
0.22-0.41
0.08
0.12
0.14
0.15
0.17
0.50
0.s
0.02-0.22
0.07
0.08
0.08
0.08
0
0.09
0.75
0.25
0.75
0.80
0.5
Input Atypicality Quantile
Input Atypicality Quantile
Class Atypicality Quantile
Input Atypicality Quantile
(a)
(b)
(c)
(d)
Figure Atypical Samples Have Low-Quality Predictions. (a) Here, samples are grouped according to the Input Atypicality (x-axis) and Confidence (y-axis), to the right meaning more atypical. Values show the difference between confidence and accuracy, lighter color indicates more overconfidence. Within the same confidence range. atypical groups have more miscalibration and are more overconfident. (b,c,d) Predictions for atypical samples are less accurate and more miscalibrated in balanced and imbalanced supervised classification and classification with L.Ms.
Here, we aim to examine the relationship between model calibration and atypicality. Given any > 1, we consider the quantiles of (X)), (11, .2,...., ax++ such that P(a(X) X) E (ax; (x+1]) = 1/ K for E [K] For imbalanced classification problems, we compute the quantiles using the class atypicality.
Specifically, we investigate the atypicality-conde conditional calibration error ECE[P a(X) E (ax, 4x+
ie., the expected calibration error of input that falls within the atypicality quantile k.
Atypical Examples are Poorly Calibrated: In Figure 2a, we show the distribution of miscalibration where each bin within the grid contains the intersection of the corresponding confidence and atypi- cality quantiles. We observe that within the same confidence range, predictions for atypical points have lower accuracies and are more overconfident. In other words, predictions in the Extrapolation or Untrustworthy regions are more miscalibrated than the ones the typical regions.
In Figure 2b, we split inputs into quantiles according to atypicality and compute the ECE and Accuracy for each group. Results show a monotonic relationship between atypicality and ECE or Accuracy across the three settings. Specifically, we see that predictions for atypical inputs or samples from rare classes are more miscalibrated and have lower accuracy. For samples from rare classes, the model overpredicts the probabilities of the typical class, hence we have overconfidence and low accuracy. Appendix C.3, and $4 present figures and tables for all model and dataset pairs.
3.2 Theoretical Analysis: Characterizing Calibration Error with Atypicality
We characterize how calibration error varies with atypicality in tractable model that is commonly used in machine learning theory [BMWX21a, BMWX216, ZDKZ22, CLKZ22]. Our theoretical analysis further supports our empirical findings.
Data Generative Model: We consider the well-specified logistic model for binary classification with Gaussian data, where Y E 1) and the IP(Y 1|X) is defined by the sigmoid function:
PPY = 1 x) = ((
X I(,
Where Ia denotes the d d-dimensional identity matrix, 8 is the ground truth coefficient vector, ofz) 1/(1 + e e²)), and we have i.i.d. observations {(,,, H sampled from the above distribution.
The Estimator: We focus on studying the solution produced by minimizing the logistic loss
1 x
i arg min 3 n
Σ
[lo((1 + (xx((3xx.)) - yi TT:..
For h E (-1,1). is an estimator of P(y = k||r), with the form P& (x)
Calibration: We consider all x where IP, (xx > 1/2, as P (x)) < 1/2 can be analyzed similarly by symmetry (see Appendix G). For E (1/2, 1), the signed calibration error at confidence level u is
IPY 1 P. (X) u).
5

---

We want to show that when X is atypical, i.e., when ((X) = || is larger', the accuracy P(Y 1 (X())== = would be generally smaller than the confidence (over-confidence),
Theorem 3.1. Consider the data generative model and the learning setting above. For any K > 1, suppose we consider the quantiles of ( 01, 02 ..., 0K ag+1 such that P(a(X) E (ak: ak+1l) = 1/K for k E [.]. We assume ||3* S. co, and d/n for some sufficiently small co. Then, for sufficiently large n, for 2, K, we have
B (x) u P(Y = | (X) = =u) [[()) E (ak: ak 0k-1]]
E P(Y = (X)) = u) a(x) (ax- anl 2
That is, the resulting classifier is over-confident, and the level of over-confidence becomes larger when the data is more atypical (with larger a(X)). Further, the gap becomes larger for smaller sample sizes The proof of the theorem is in Appendix G.2 and builds the results from [BMWX21a, SC19].
4 Using Atypicality to Improve Recalibration
Here, we show how atypicality can complement and improve post-hoc calibration. In $2, we observed that predictions for atypical inputs and samples from atypical classes are more overconfident with lower accuracy. We next show that taking input and class atypicality into account improves calibration.
4.1 Parametric Recalibration: Different Groups Need Different Temperatures
Temperature scaling (TS), a single parameter variant of Platt Scaling [P-99], a simple recalibration method that calibrates the model using a single parameter. The predictor is of the form
log (Yg x log (X)
where P(Y]X) ) is the model that takes an input and outputs scores/logits, and r is the temperature parameter. In practice, is optimized using a calibration set to minimize a proper scoring rule [GR07, BW19] such as the cross-entropy loss.
To understand the behavior of TS with respect atypicality, we separately perform TS on points grouped according to the atypicality quantiles. Let us denote the temperature fitted to the quantile covering (X) E (ak-1, as. by Tag In Appendix Figure 10 we observe an increasing relationship between aş and Taz, Different atypicality groups need different adjustments, and more atypical groups need larger temperatures. This suggests that being atypicality- aware can improve calibration. While a single temperature value improves average calibration, it may hurt certain groups.
4.2 Atypicality-Aware Recalibration
We showed that predictions are more reliable when the input is typical. However, predictions are less reliable for atypical inputs, and we may need further revision. An analogy can be drawn to decision- making literature where opinions of individuals are combined with geometric averaging weighted by their expertise [FP98, AR89J. Analogously, we propose Atypicaliy-Aware Recalibration (AAR) method designed to address the reliability issues identified in dealing with atypical inputs:
M((X)
Z(X)
where v(a(X)) is a function of input atypicality, Sy is tunable score for class Y, Z(X) is the normalization term. Intuitively, when the input is typical, we trust the model confidence; otherwise, we use score for the given class estimated from the calibration set. Note that this form simplifies to
log AAR PA (F|X) x ((a(X)) log P(Y) + Sy,
'The definition of atypicality follows from the marginal likelihood of the data model: density for the Gaussian with mean and identity covariance.

---

ResNet18 ImageNet
ResNet50 ImageNet
WideResNet28
0.08
0.05
0.04
0.05
ECE
0.03
(a)
0.02
0.03
0.00
1.00 Input Atypicality Quantile
0.25
1.00
Input Atypicality Quantile
Input Atypicality Quantile
Input Atypicality Quantile
Uncalibrated
AAR(Ours)
ResNext50 ImageNet LT
(csNext50 ImageNet-
ResNet152 Places365
ResNet18 CIFAR100 LT
0.20
0.20
ECE
ECE
0.10
0.20
0.00
0:25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
(b)
Class Atypical ity Quantle
Class
Atypicality Quantile
Class Atypical Ry Quantile
Class Atypicality Quantile
Accuracy
0.5
0.44
0.44
5.44
0.50
Accurazy
0.5
0:48
Accuracy
0.30
0.30
3030
0.38
Accuracy
0.5
0.45
6.45
0.45
0.50
AAR(Ours)
Alpaca7B AG News
Alpaca7B 7B TREC
Alpaca7B BB IMDB
0.40
0.50
0.15
ECE
ECE
0.20
0.05
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
(c)
Input Atypicality Quantile
Input Atypicality Quantile
Input Atypicality Quantile
Accuracy
Accuracy
Accuracy
0.65
0.65
0.54
0.75
0.5
0.64
0.64
0.60
0.80
0.82
0.82
0.
0.82
0.91
0.0
Uncalibrated
CF
AAR(Ours)
Figure 3: Post-hoc Recalibration for Classification. (a) Balanced Supervised Classification:
Atypicality-Awann Recalibration improves the calibration of models trained with balanced datasets, across atypicality groups. (b) Imbalanced Supervised Classification: Atypicality-Awan Recali- bration improves both the calibration across groups and the overall accuracy of models trained with imbalanced datasets. (c) Classification with LLMs: Atypicality-ANi Recalibration improves both the calibration across groups and the overall accuracy of LLMs performing classification.
where subsume (()(X)) into ((a())). We give simple interpretation of this form: the multiplicative term is an ttyppcality-dependeee temperature, and the additive term is a class-dependent correction where exp (Sy) ) can considered induce a correction distribution over classes estimated from the calibration set.
Intuitively, when (a((()) O, the output reduces fixed distribution over classes that was estimated using the calibration set. This distribution can be seen to induce a prior probability over classes, and & controls the tradeoff between this prior and the model's predictive distribution. As the point becomes more typical, this distribution is closer to the model's predictive distribution. In Appendix Figure 11, we show how these values behave with class atypicality. We find that rare classes require larger positive corrections with larger 5y
Following TS, minimize the cross-entropy loss a calibration set. With the cemperature-atypicalin relationship observed in Figure 10 we choose to instantiate the multiplicative factor as quadratic function, where o(a(X)) Cza c2a(X)² (X)² + c cxa(X)= cu and in total we have [[ Spy Cu. C1 c2) = yl + interpretable parameters. Once the embeddings and logits computed, AAR runs CPU in under 1 minute for all experimented settings.
Implementation Details:
Similar to our adaptive interpretation, (AdaTS) [J3,++23], uses temperature scaling where the temperature is parameterized by Vari- ational Autoencoder(VAIR [K [KW13] and multi-layer perceptron on top of the VAE embeddings. In the below experiments, give results with AdaTS as baseline when applicable.
concurrent work, Adaptive Temperature Scaling

---

ResNet18
Accuracy
ResNet18
Re
Accuracy
4
0.75
0.75
EGE
ECE
Unknown 162
3&4
566
Un known 162
364
566
0.0
Unknown 162
364
666
Unknown 162
3644
5&6
Skin Tone
Skin
Skin Tone
T5
Skin Tone Conditional
AARIOurS)
Figure 4: Improving Group Performance through apppicalit--wareenee Here we show that AAR improves the calibration and accuracy of models across different skin tone groups With AAR, we can improve both the worst group performance and overall performance significantly without using group attributes. TS curve is less visible since it significantly overlaps with Skin Tone Conditional.
For Balanced Supervised Classification, in Figure 3a, we observe that being atypicality aware improves recalibration across all groups. We perform comparably to AdaTS, where the temperature function has in the order of millions of parameters, whereas AAR has only (3) parameters.
In Imbalanced Supervised Classification (Figure 3b), our algorithm not only provides better calibration rates across all classes but also improves overall accuracy. Note that only our method can change accuracy (due to the additive term), and it performs better than other baselines in terms of ECE across all classes. Further, the second column shows using Progressive Balancing [ZWF+ 21] in training, showing that our post-hoc method can complement methods that modify training procedures.
For Classification with LLMs, we add LLM calibration baseline Content-Free Calibra- tion (CF) (1]. We cannot use AdaTS as the embeddings are not fixed in size. In Figure 3c, we AAR has better calibration and accuracy across the three datasets. Namely, by adjusting the LLM output using the LLM atypicality, we can adjust the probabilities to increase the prediction quality.
4.3 Case Study: Fairness through Atypicality-Awareee
Machine learning models reportedly have performance disparity across subgroups [BHN17] due to factors such as varying sample size or noise levels [CJS18]. For instance, skin lesion classifiers can exhibit performance disparity across different skin tones 22]. Fitzpatrick177 [GHS*21] 21] is a dataset of clinical images with Fitzpatrick skin tone annotations between 1-to-6, where a larger number means darker skin tones, and when annotators do not agree, it is labeled as 'Unknown'. We explore the classification problem with 9 classes indicating the malignancy and the type of skin condition, using ResNet18/34 pretrained on ImageNet and finetuned on this task (See Appendix F).
When the goal to improve performance across groups, one can use group annotations and optimize performance within each group [HJKRR18, KGZ19]. Here, we investigate how complementing recalibration with atypicality can improve prediction quality across all groups without group annota- tions. For comparison, we perform 3 recalibration methods: TS, AAR, and Skin-Tone Conditional TS which calibrates the model individually for each skin-tone group with TS. Since the skin-tone conditional calibration uses group attributes, ideally it should act as oracle.
In Figure 4, give the Accuracy and ECE analyses where AAR improves performance across all groups. For instance, the worst- worst-group Accuracy (0.69) or ECE (0.072) with AAR is close to the best-group Accuracy (0.63) or ECE (0.062) with the other two methods. Overall, our findings suggest that Atypicalit-Awareme even when the group annotations are unavailable. We hypothesize that with AAR, we can perform better than using supervised group attributes since groups may not have sufficient sample size in the calibration set (131, 1950, 1509, 555 samples for Unknown, 1&2, 3&4, and 5&6 respectively), and we can leverage atypicality to offer some mitigation. Further investigating how to leverage atypicality to improve fairness and factors affecting performance disparities is promising direction for future work [CJS18].
can complement fairneesss fairness-enforcing methods, and improve performance
5 Improving Prediction Sets with Atypicality
Conformal Prediction [SV08, AB21] framework that assigns a calibrated prediction set to each instance. The goal is to find function C X' + 2 that returns subset of the label space such that E C(X) with high probability. The framework aims to guarantee marginal coverage, ie.. IPY E C(X)) > 1 a. for choice of a. We investigate two conformal calibration methods,
8

---

Adaptive Prediction Sets (APS) [RSC20] and Regularized APS (RAPS) [ABMJ20]] Let π ((X) be the permutation of the label set that sorts IP c. i.c. the predicted probabilities for each class c after TS. The prediction sets are produced by the function C(x) = {y : s y) I g). and these methods fit the threshold @ for choice of the scoring function. APS uses the cumulative
of the predicted probabilities s(r,, y) = Ej= P(Y = j|X), where y *e. (x)) Intuitively, if the model was perfectly calibrated, would have expected to have 1 - Similarly, RAPS builds on the idea that tail probabilities are noisy and regularizes the number of samples in the prediction set.
Building our ideas in the previous sections we implement Atypicality-Awar prediction sets, namely AA-APS and AA-RAPS in the following way: We first group points according to their confidence and atypicality quantiles. A group G here is defined using 4 thresholds, namely G x (1(C) < {a)) S (( A (I!C)
<c((>) S h())) where (a(z) and (c(x)) denote the atypicality and confidence quantiles for the sample x. ((() and hi denote the atypicality lower and upper bounds for group G, and (( and ![ denote the confidence lower and upper bounds for group G. Using a calibration set, these bounds are simply determined by the quantiles of confidence and
atypicality statisties. Then, we fit separate thresholds q for each group's 's prediction sets with APS or RAPS as subroutines. This allows us to have an adaptive threshold depending only on the atypicality and confidence of predictions.
APR0 Care
AM-R511
CouwAaGe
A253
SUB
MARSO
S Row
1.0-1.8
16
T.
0:98
0.56
0:36
0.95
0:55
0:58
0:36
0.6
3:57
0.57
10
10
1.
10
10
1.0
10
10
19
0:36
0:35
0:59
0:04
0:32
0.90
0.55
0.55
0:36
0.95
2.55
0:57
17
15
21
23
21
1.5
17
22
4.
45
93
0.95
0:5
5:55
0.95
0.54
11
75
05
5.01 45
0:99
0.
2.95
1.56
0:05
0:95
3.95
2.95
79.5
115.5
135-2
111
B7.
403
46.3
76.5
19.1
91
5
S
&
1
1
5
voot Ayykality Quanste
RAPS0 951
MMRA250 95) COUPERCe
RAPRIO 455 SA
M-RAP% 451 SH SuP
14
11
TO
0.3
0:3
0.50
0:35
0.9
555
0:33
0.6
0.57
46
43
4.4
13
10
11
12
15
582 02
0.39
0.0
055
0.50
0.30
0.94
0.55
0:34
097
11
32
74
16
75
1.0
33
44
63
7.0
1A5 82
0:37
0:7
96
97
2
96
00
0.0
14
77
77
70
79
17
45
64
15
12
172
193
0:53
0.8
0.02
3.7
7.79
9.
0.44
5.97
50
43
02
63
124
26
375
19.
64.5
Figure 5: Improving Conformal Calibration with Atypicality for ResNet50 on ImageNet. Here we show that atypicality awareness improves conformal calibration performance across different groups. Methods are fitted to satisfy 95% coverage. We observe that APS and RAPS do not satisfy conditional coverage for high atypicality regions or low confidence regions.
In Figure 5, we provide the coverage plots for APS and RAPS in the first and third columns. Even though marginal coverage is satisfied, models do not satisfy conditional coverage for atypical inputs or low-confidence predictions. We observe that being Atypicality Aware improves coverage across otherwise underperforming groups. Further, AA-APS has lower set sizes on average than APS (15.6 vs 21.3). While RAPS has a lower average set size than AA-RAPS (4.2 vs 9.1) AA-RAPS has smaller set sizes for high-confidence samples, whereas a larger set size for low-confidence samples where the coverage is not met for RAPS. In Appendix D.3, we provide the same analysis for ResNet18 1,,Nee1,,50,153 1552 different coverage levels along with analyzing the performance in the Confidence and Atypicality dimensions individually. For instance in Figure 8, we observe that RAPS and APS do not satisfy coverage for high atypicality regions, even when averaged across different confidence levels.
6 Additional Related Work
Uncertainty and Atypicality: [MKvA-21] 2BC+ use density estimation to disentangle epis- temic and aleatoric uncertainty. Following this, they show improvements in active learning and OOD detection [LLLS18]. We note that our goal is not this disentanglement (c.g. Untrustworthy quadrant can have both aleatoric or epistemic uncertainty), or Ambiguity could be due lack of features or noise. [LLP+20] propose the related notion of distance awareness, and that it leads to better uncertainty quantification. They offer architecture and training modifications whereas we analyze existing models using our framework including imbalanced and LLM settings, and propose simple and post-hoc approaches. 'OOD' [HG16b] or 'anomaly" [HMD18] notions are tied to atypicality, yet our goal is not to make a binary distinction between 'in' or 'out'. We argue that in-distribution samples could also be atypical (e.g. rare groups), and the goal is to perform reliably in the entire spectrum. Other works with atypicality notion include bounding calibration of groups by the
9

---

excess risk [LSHI9], miscalibration under distribution shifts [OFR + 19], uncertainty in Gaussian Processes [Ras04], forgetting time for rare examples [MGLK22], the poor performance of groups with lower sample sizes [CJS]] 18), energy-based models improving calibration [GWJ+20] 20). relating perplexity to zero-shot classification performance for L.LMs (GIB+22), grouping loss and local defini- tions of miscalibration [PL, the relationship between active learning and atypicality [HDW22], sample size as a factor for subgroup performance disparity [CJS18]. [PBZ21 I] provide insightful discussion around the nature of softmax confidence, and here we show that its reliability depends on the atypicality of the input. Our new findings include showing that predictions for atypical samples more miscalibrated and overconfident, and atypicality awareness improves prediction quality. Overall, while there are other relevant notions the literature. our distinct goal is to show that post-hoc atypicality estimation and recalibration is simple yet useful framework understand and improve uncertainty quantification that complements existing methods.
Recalibration and Conformal Prediction: There is a rich literature recalibration methods and prediction sets: TS [GPSW17], Platt Scaling [P++9], conformal calibration [SV08, ABMJ20] among many. [LLC RBSCI9, BYR 21, BGJ+22] make relevant observation, showing that the coverage of conformal prediction is not equal across all groups They propose group conformal calibration, which requires group labels whereas our proposal is unsupervised and does not depend on any attribute information. Concurrent work 233] explores AdaTS, where they train a separate VAE and MLP to produce adaptive temperature. However, parameterization of temperature has parameters and is interpretable.
Conclusion
Atypicality offers simple yet flexible framework to better understand and improve model reliability and uncertainty. We propose that pretrained models should be released not only with confidence but also with atypicality estimator. While there are other relevant notions in the literature, our main goal is to show that atypicality can provide a unifying perspective to discuss uncertainty, understand individual data points, and improve fairness. Here we focus on classification problems; it would be interesting to extend atypicality to regression and generation settings. Furthermore, we would like to extend the theoretical analysis to more general settings, observed phenomena hold more broadly.
empirical results demonstrate that the
Acknowledgments
We would like to thank Adarsh Jeewajee, Bryan He, Edward Chen, Federico Bianchi, Kyle Swanson, Natalic Dullerud, Ransalu Senanayake, Sabri Eyuboglu. Shirley Wu, Weixin Liang, Xuechen Li, Yongchan Kwon, Yu Sun, and Zach Izzo for their comments and suggestions on the manuscript. Linjun Zhang's 's research partially supported by NSF DMS-2015378. Carlos Ernesto Guestrin is Chan Zuckerberg Biohub San Francisco Investigator.
References
[AB21] Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to con- formal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.05511 2021. 8, 23
[ABMJ20] Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint UrXiv:2009.14193, 2020. 2, 10, 23, 24
[AR89] János Aczél and Fred S Roberts. On the possible merging functions. Mathematical Social Sciences, 17(3):205-244 1989. 6
[BG++22] 22] Osbert Bastani, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. Practical adversarial multivalid conformal prediction. arXiv preprint arXiv:2206.0 0ate:2206601066 01067, 2022. 10
[BHN17] Solon Barocas, Moritz, Hardt, and Arvind Narayanan. Fairness machine learning.
Nips tutorial, 1:2, 2017. 8
10

---

[BMR 20] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 4
[BMWX21a] Yu Bai. Song Mei, Huan Wang, and Caiming Xiong. Don't just blame over- parametrization for over-confidence: Theoretical analysis of calibration in binary classification. In International Conference on Machine Learning, pages 566-576.
PMIR, 2021. 2,5, 6, 27, 33
[BMW221b] Yu Bai, Song Mei, Huan Wang, and Caiming Xiong. Understanding the under- coverage bias in uncertainty estimation. Advances in Neural Information Processing Systems, 13077307-88819 2021.
[BW19] David Bolin and Jonas Wallin. Local scale invariance and robustness of proper scoring rules. arXiv preprint 1912.055422 2019. 6
[BYR+2]] Noam Barda, Gal Yona, Guy N Rothblum, Philip Greenland, Morton Leibowitz, Ran Balicer, Eitan Bachmat, and Noa Dagan. Addressing bias in prediction models by improving subpopulation calibration. Journal of the American Medical Informatics
Association, 28(3):549-555 2021. 10
[CJS18] Irene Chen, Fredrik D Johansson, and David Sontag. Why is my classifier discrimina- tory? Advances in neural information processing systems, 31, 2018. 8, 10
[CLKZ22] Lucas Clarté, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Theoretical characterization of uncertainty in high-dimensional linear classification. arXiv preprint arXiv:2202.03295 2022.
[CWG+19 19] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution aware margin loss. Advances neural information processing systems, 32, 2019. 3
[DCLT19] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019. 17
[DDS+09 09] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li. Kai Li, and Fei-Fei Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009. 4,
[DVN+22] 22] Roxana Daneshjou, Kailas Vodrahalli, Roberto A Novoa, Melissa Jenkins, Weixin Liang, Veronica Rotemberg, Justin Ko, Susan M Swetter, Elizabeth E Bailey, Olivier Gevaert, et al. Disparities dermatology ai performance on a diverse, curated clinical image set. Science advances, 8(3 833)))ebb66147, 2022. 8,8
[FP98] Ernest Forman and Kirti Peniwati. Aggregating individual judgments and priori- ties with the analytic hicrarchy process. European journal of operational research, 108(1):165-169 1998.
[GHS+21] Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1820-1828, 2021. 8, 26
[GIB+22] Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. De- mystifying prompts language models via perplexity estimation. arXiv preprint DXXiv:2212004031 2022. 10
[GPSW17] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017. 2, 4, 10, 23

---

[GR07] Tilmann Gneiting and Adrian E Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):35--377 2007.
2, 4, 6
[GWJ+20] 20] Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mo- hammad Norouzi, and Kevin Swersky. Your classifier secretly an energy based model and you should treat it like one. In International Conference Learning Representations, 2020. 10
[HDW22] Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active learning on a budget: Opposite strategies suit high and low budgets. arXiv preprint arXiv:2202.02794, 2022.
10
[HG16a] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-
of-distribution examples in neural networks. arXiv preprint arXiv: 2016.
3
[HG16b] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of- distribution examples in neural networks. ArXiv, abs/1610.02133. 2016. 9
[JKRR18] Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicali- bration: Calibration for the compuuationalll--dennfffia Conference on Machine Learning, pages 1939-1948. PMLR, 2018. 8, 33
masses. In International
[HMD18] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 18/i:0//20.00 2018. 9, 19
[HZRS16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. 4
[JKGG18] Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not trust classifier. Advances in neural information processing systems, 31, 2018. 3
[J] Tom Joy, Francesco Pinto, Ser-Nam Lim, Philip HS Torr, and Puncet K Dokania. Sample-dependent adaptive temperature scaling for improved calibration. AAAI, 2023. 7, 10, 23
[KGZ19] Michael P Kim, Amirata Ghorbani, and James Zou. Multiaccuracy; Black-box post- processing for fairness in classification. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 247-254, 2019. 8, 33
[Kri09] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. 4, 17
[um22]] Sawan Kumar. Answer-level calibration for free-form multiple choice question answer- ing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages (65,679, 2022. 19
[KW13] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 131.v21312.611 14, 2013. 7
[LLCC22] Charles Lu, Andréanne Lemay, Ken Chang, Katharina Höbel, and Jayashree Kalpathy- Cramer. Fair conformal predictors for applications in medical imaging. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 22008-12018 2022. 10
[LLLS18] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- out-of-distirn of-distributtin samples and adversarial attacks. Advances in neural information processing systems, 31, 2018. 4,
[LLP+20] Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33:7498-7512, 2020. 9
12

---

[LN89] Dong C Liu and Jorge Nocedal. On the limited memory bigs method for large scale optimization. Mathematical programming, 45(1):503-522, 1989. 23
[LOG 19] Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint
2019, 4
[LR02] Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002.
[LSH19] Lydia T Liu, Max Simehowitz, and Moritz Hardt. The implicit faimess criterion of unconstrained learning. International Conference on Machine Learning, pages 4051-4060. PMLR, 2019. 10
[LV Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani. Bhavitvya Malik, Simon Brandeis, Teven Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, 2021. 17
IMDP*11 11] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Lan- guage Technologies, pages 142-150, Portland, Oregon, USA, June 2011, Association for Computational Linguistics. 4
[MGLK22] Pratyush Maini, Saurabh Garg, Zachary C Lipton, and J Zico Kolter. Characterizing datapoints via second-spli forgetting. arXiv preprint arXiv:2210.150:: 2022. 10
[MKS+20] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip HS Torr, and Puneet K Dokania. Calibrating deep neural networks using focal loss. 2020. 4, 17
[MKvA+21] Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr. and Yarin Gal. Deep deterministic uncertainty: A simple baseline. arXiv e-prints, pages arXiv-2102, 2021. 2, 4, 9, 16
[MP80] Carolyn B Mervis and John R Pani. Acquisition of basic object categories. Cognitive Psychology. 12(4):496-522 1980.
[MR10] Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference Multimedia, pages 1485-1488. 2010. 17
[Mur04] Gregory Murphy. The big book of concepts. MIT press, 2004.
[NCH15] Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning, In Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
[OFR+19] 19] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan. and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019. 10
[P+99] John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999. 6, 10
13

---

[PBC*20] 20] Janis Postels, Hermann Blum, Cesar Cadena, Roland Siegwart, Luc Van Gool, and Federico Tombari. Quantifying aleatoric and epistemic uncertainty using density estimation in latent space. arXiv preprint arXiv:2012.03008. 2020. 9
[PBZ21] Tim Pearce, Alexandra Brintrup, and Jun Zhu. Understanding softmax confidence and uncertainty. arXiv preprint arXiv:2 04972,, 2021. 10
[PLMV23] Alexandre Perez-Lebel, Marine Le Morvan, and Gael Varoquaux. Beyond calibration: estimating the grouping loss of modern neural networks. In The Eleventh International Conference on Learning Representations, 2023. 10
[Ras04] Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school machine learning, pages 63-71. Springer, 2004. 10
[RBSC19] Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel J Candès. With malice towards none: Assessing uncertainty via equalized coverage. arXiv preprint arXiv:1908.0 05428, 2019. 10
[Rip89] Lance J. Rips. Similarity, rypicality, and categorization, page 21-59. Cambridge University Press, 1989.
[RM75] Eleanor Rosch and Carolyn B Mervis. Family resemblances: Studies in the internal structure of categories. Cognitive psychology, 7(4):573-605, 1975.
[RSC20] Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems, 33:3581- 3591, 2020.
[RSS73] Lance J Rips, Edward J Shoben, and Edward E Smith. Semantic distance and the verification of semantic relations. Journal of verbal learning and verbal behavior, 1201) 1973.
[SC19] Pragya Sur and Emmanuel J Candès. A modern maximun-likelihood theory for high- dimensional logistic regression. Proceedings of the National Academy of Sciences, 1 (1662))14511--4555, 2019. 6, 27,
[SV08] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008 8, 10
[TGZ*23] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction- following Ilaaa model.
wtt.au....oo/ttttalaaabbb. stanfaorra/oaad...../// _apaca,
2023. 4, 18
[TJ06] MTCAJ Thomas and A Thomas Joy. Elements of information theory. Wiley- Interscience, 2006.
[KK74] Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, (85(4157):1124-11
1974. 1
[TK92] Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5(4):297-323, 1992. 1
[VGS05] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world. Springer Science & Business Media, 2005. 23
[WDS+ 20] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, 2020. 17
14

---

[WNB18] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers). Association for Computational Linguistics, 2018. 4, 17
[ZCL]21] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16489-16498, 2021. 3, 4. 18
[ZDKZ22] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. In International Conference on Machine Learning, pages 26135- 26160. PMLR, 2022.
[ZK16] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks, arXiv preprint arXiv:1605.07145 07,46 2016.
16] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding. arXiv preprint arXiv:161000255. 2016. 18
[ZWF+21] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference Machine Learning, pages 2697-12706. PMLR, 2021. 4, 8, 18, 19
[ZZL15] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015. 4
15

---

A Atypicality Estimation
A.1 Input Atypicality Estimation
To estimate input atypicality, we use two ways to estimate the likelihood of a point under the training distribution. First, we give methods for the discriminative models.
Fitting individual Gaussians to class conditionals Here, we follow similar approach to 2MKv Namely, we model the class conditionals with Gaussian distribution, where the covariance matrix is tied across classes:
P(X = u) ~ (4; 2)
We the parameters By and with maximum likelihood estimation. The reason tie the covariance matrix is due to the number of samples required to fit the density. Namely, for a d-dimensional problem, the total number of parameters to fit individual matrices becomes Oyyd), which results in low-quality estimates. Then, the atypicality becomes
ax (x)
max log P(X x/ y)
wEy
the nearest neighbor distance. Concretely, k-Nearest Neighbors: Similarly, we can the nearest neighbor distance, (x(x) dainn(; Drain) = m Ix' x| as the atypicality metric. Alternatively, we can use different notions such as the average of k-nearest neighbors, the distance the kth neighbor. Below, report the results by using the average distance to 5-nearest neighbors.
Computing distance with k-Nearest Neighbors
Fitting the estimators For all of the atypicality estimators, we fit the estimators using samples from the training sets and make inferences for the calibration and test sets. For instance, we use the training split of TeageNet fit the corresponding density estimator and compute the atypicality for the samples from the validation/test split of ImageNet. All of our results using atypicality are reported for the splits of the below datasets.
Atypicality Estimation with LLMs: For language models we simply compute the negative log- likelihood of each prompt as the atypicality metric: ax log (z). To define confidence, we use the logits language model, conditioned on the prompt. We use logit of the first token of each class label and compute the predicted probabilities by applying softmax to the logits of each class.
A.2 Class Atypicality
To estimate class atypicality, we simply count the fraction of examples from particular label in the training dataset. Let us have training dataset Duran = {(111111) (Z2, U2)..... (IN, (x))) Then, we estimate class atypicality with
EiE(N] 1|4 vl
ay (y) = = -olog
(6)
N
A.3 Atypicality and Confidence
Are atypicality and confidence equally informative? Beyond the data perspective given in Figure 1, here we provide quantitative results to demonstrate the difference. In Figure 6, we give grid plot where the x-axis indicates the typicality quantile of point, and the y-axis indicates the confidence of a point. The coloring on the left indicates the accuracy within bin split according to accuracy, and the right has the difference between average confidence and accuracy. Observe that for specific confidence interval, larger values of typicality mean better quality probabilistic estimates and larger atypicality means more miscalibration.
16

---

8 on ImageNet ImageNet-R Accuracylar.)
ResNet1 8 on ImageNet ImageNett-R Confidencee Accuracyl
0.35
0.50
Confidence
Confidence
0.20
0.35
0.30
334
8:45
0.46
3.20
0.30
0.15
(8.000.50)
6.42
8:37
3.99
6:23
3.34
E25
(0-40-.500)
8:84
0.94
9:15
1.27
6:24
321
9.23
0:11
5:1
017
[0.33-2-42
0:29
0.25
6:2
837
333
0.10
8:31
339
014
0:12
1:1
012
[0:23-331]]
0.21
811
(615-4223)
3:33
0.21
6:2
832
3.39
0.12
8330
3.9
0.10
0.0
3.37
[0:33-2221]
9:1
4:03
383
830
0.15
322
8:27
0.36
666
666
8:85
0/6
640
3.44 6.7 646
665
0.30
[0/222.55
8.22
602
284
005
0.25
6:4
8.56
6/6
383
0.0
10
11
12
Alypicality Quantile
Atypicaliity Quantile
Figure 6: Input Atypicality and Confidence. Here, the x-axis reflects the input atypicality quantile, and the y-axis indicates confidence. The coloring for the figure on the left indicates the accuracy within a bin, and the figure on the right has the difference between confidence and accuracy within a bin. We observe that even within the same confidence range, atypical examples tend to be more miscalibrated and overconfident compared to typical examples.
B Experimental Details
B.1 Balanced Supervised Classification
B.1.1
Datasets
Below a full list of datasets for balanced classification:
ImageNet [DDS+09] from Torchvision [MR10] is an object recognition dataset with 1000 classes. We use the ImageNet-IK 1k version.
2. CIFAR10/100 [Kri09] from Torchvision [MR10] are object recognition datasets with 10/100 classes.
3. MNLI [WNB18] from Huggingface Datasets [LVdMJ=21] is natural language inference dataset with 3 classes, indicating entailment, neutral, and contradiction outcomes.
B.1.2 Models
Most of models are public models, e.g., obtained from the Transformers Library [WDS+20] or Torchvision [MR10]. Below we give the full model details and how one can access them:
1. RoBERTachuggingFaac roberta-large-nn1. trained on the MNLI dataset. One can use the id given here on HuggingFace to download the model.
2. ResNet18, ResNet50, ResNet152 from (Torchvision (MR10] trained on ImageNet.
3. WideResNNe28 trained on CIFARIO,100 obtained from MMS+*20
For all BERT [DCLT19] style models we use the [CLS] token embeddings in the final layer to perform classification. For vision models, we use penultimate layer embeddings fit the density estimators and perform the analyses. In the experiments, we randomly split the test sets into two equal halves have a calibration split and test split, and repeat the experiments over 10 random seeds.
17

---

B.2 Imbalanced Supervised Classification
B.2.1 Datasets
All of our imbalanced classification datasets are previous benchmarks obtained from the GitHub repository of [ZCIJ21] with corresponding training, validation, and test splits. All of these datasets have an exponential class imbalance.
1. ImageNet-LT is the long-tailed variant of ImageNet with 1000 classes.
2. CIFAR101000LL is the long-tailed variant of CIFARI0/100 with 10/100 classes.
3. Places365-LT the long-tailed variant of Places365 [6K with 365 classes.
B.2.2 Models
Similarly, most of these models obtained from [ZCLJ21].
1. ResNeXt50 trained on ImageNet-LT with and without Progressive Balancing, which is strategy address class imbalance during training.
2. ResNet152 trained on Places365-LT
3. ResNet18 trained on CIFARI00-LT trained by us. This model is pretrained on ImageNet and finetuned on CIFAR1O0-LT.
We use the validation splits of these datasets as the calibration set, and report the results on the test set.
B.3
Classification with LLMs
B.3.1
Model
We use Alpaca-7B [TGZ+23] in zero-shot setting, where we simply prompt the model with the classification question. We use the prompting strategy from Content-Free Calibration [ZWF+21]]
Below, we show examples of each dataset and prompt.
B.3.2 Datasets
IMDB is binary classification dataset of movie reviews, where the goal is classify the sentiment in a review. The example prompt has the form 'The following review was written for a movie: [Review].\n What is the sentiment, Positive Negative? Answer: :. Below is example:
The following review was written for movie: I and a friend rented this movie. We both found the movie soundtrack and production techniques appeared to drag on throughout with little surprise in the ending. We both agreed that the movie could have been compressed into roughly an hour giving it more suspense and moving plot.
be lagging. The movie's 's plot
What is the sentiment, Positive Negative? Answer:
where the correct answer should be 'Negative'. We filter the examples that exceed the context length limit of Alpaca7B (512). We noticed that the 'validation' split of IMDB leads to significantly worse calibration compared splitting the test set. Thus, for all experiments, we use the test split of IMDB and split it into 2 sets (instead of using the validation split as calibration set as in the other two datasets).
TREC a 6-class question classification dataset where the goal is to predict whether question will have answer that is 'Abbreviation', 'Entity", 'Description', 'Human', 'Locationn, 'Number'. We format the prompts with 'Classify the questions based on their Answer Type. Potential Answer Types Number, Location, Person, Deseription, Entity, Abbreviation. Queestio: [question]] n pnnswr nAnsser Type: Below is an example prompt:
'nttps://githun hwwpitthuu.coo///eee www/dvlab-aaeeoooeooewo.wwa.aaaa hww.:l/bbrreeearcc/iiL...
18

---

Classify the questions based on their Answer Type. Potential Answer Types are: Number, Location, Person, Description, Entity, or Abbreviation.
Question: What county is Modesto, California in Answer Type:
where the correct answer should be 'Location'.
AG News is a news classification dataset. The goal to classify a given news into 4 potential 'World', 'Sports', "Business', or 'Science and Technology'. We format the prompts with "Clsssify the news articles into the categories of World. Sports, Business, and Technology. A Article: [article]], nnnwwrr: nnnswerr Below is an example prompt:
classes:
Classify the news articles into the categories of World, Sports, Business, and Technology.
Article: Wall Bears Claw Back Into the Black (Reuters) Reuters Short-sellers, Wall Street's dwindling band of ultra-cynics, are seeing green again.
Answer:
where the correct answer should be 'Business'.
Furthermore, we also use [ZWF+21] as another calibration baseline. Following their paper, we use N/A, [MASK], and the empty string as content-free Concretely, we follow their paper to first obtain the average predicted probabilities for each label token for the content-free input, denoted by Pef We then let
W diag(pe))>
When making test-time predictions, we compute Softmax (WT) p) as the new predicted probabilities. In our experiments, we observe that it does not perform well in this setting, was previously suggested by [Kum22].
C
Calibration
We run all our experiments with 10 different random seeds, where the seeds are {0, 1, ........ 9}. Randomness is over fitting the atypicality estimators, and calibration-test splits (we use the same splits with the recalibration experiments for the sake of consistency).
C.1 Expected Calibration Error
To compute ECE, we generate IB = [B, BB, BM J. M equally-spaced bins where samples are sorted and grouped according their confidence. Bm, here denotes the set of the data indices where the confidence of the prediction for the sample falls into the interval 1 m J. We compute ECEby
M
M
M
[Bml N
ECE
Σ
acc(B,,) conf( Bm,)
where acc(Bm) [B.l Σ E 10: yil the accuracy for the bin m, and conf Bm) Bml
B BBm IP(Y = X xx) gives the average confidence within the bin. |Bm] is the size of the bin m, N is the total number of samples, and 1 is the indicator function.
Throughout our experiments, we let the number of bins [B| 10 by default when computing ECE.
Similarly, below we report results with RMSCE (Root Mean Squared Error) [HMDI8] as another calibration metrie, which formulated the following:
M
Bml (acc( c)
RMSCE[
(8)
N
19

---

C.2 ECE and Atypicality Results with Different Atypicality Metries
We further experiment with different atypicality metrics, such the average distance to the 5-nearest neighbors (Figure 7). We broadly observe that while there are slight differences in the quantitative results between different atypicality metrics, the qualitative phenomena remain intact. In Tables and 3 we give all the results in the tabular form.
ResNet18 ImageNet
ResNet50 ImageNet
RoBERTà
WideResNet28 CIFAR10
0.08
0.04
ECE
0.05
0.05
0.03
0.02
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0,75 1.00 Input Atypicality Quantile
0.25
0.50
0.25 0.50
0.75 1.00
Input Atypicality Quantile
Input Atypicality Quantile
Input Atypicality Quantile
Uncalibrated
- TS
AdaTS
AAR(Ours)
Figure 7: Atypicality with 5-nearest neighbors and Uncertainty. Here, we report the results of the same experiments Figure with the average of the distance the 10-nearest neighbors the atypicality metric. See Tables for the results in tabular format.
C.3 Results in the Tabular Format
20

---

Table 1: Recalibration Results for Balanced Classification. For each dataset and atypicality quantile, the best results are marked in bold. We provide the standard errors next to the means over 10 random seeds.
Unaiileseed
TS
AdaTS
Aorpicaity-Nosrr
Atppcallty
BC3
RMS
RMSSI
RMSS!
EUB
RMSS
GMM
ReiNe1152
InageNet
0.2
0.129 (1801
0,377 0.0001
0.014
0.021
0.000 +: 0
QJIL4
0.003
0,064 2t 2.0002
0,015 = 0.003
0,065 0.001
04
0.00P 0.001
0.080 1.00011
0.031
0,093 1 0.001
0,023
00032
0,05 600002
0.019 :: -0.01
0,000 6.001
06
0150 11001
0.267
2-0.00
0.125
00021
0,07) 4: 0.0012
01225 0.030
0
0 = -100000
0,078 0.001
0.001
0.300
0.021
ResNetT
InageNet
0117: 0.001
0.074 4:0.01
-0021
0.084 0.000
0,015
0:00
0.015 0.031
044
0.015 0.001
0.374 2
0M023 0.031
0.079 2 0.001
0 0.038
0.079 0.003
0.015 0 0.031
0,072
0,002
0.6
01225 2 0.001
0.280 2-0.00
[M825 0.031
0
2 0.000
0.024 0.036
0:000 0.002
0
2 0.001
0,074 G,01
0.000 0.001
0.200 4-0.00
0.071 0.021
0,07 4-0000
01779 0.000
0.082
0.011 0.09
0,070
0/001
0.100
0,013
0
2 0.001
0
0:088 0:00
0.017 0.002
0,072
ResNe50
InsgeNet
0.4
0.029 0.001
0.079 1 000000
0.023 0.031
0 0.000
0,023 0.000
0,031 6/001
0.022 0.001
0,001
6.001
066
0.041 0,801
0.201
0-0.011
0J023 0.031
0,078
+ 0.0011
0J125 0.030
0,201 0000
0.005 = 0.000
0 0.00L
0.5
01642
= 11001
0.202
2-0.00
0J124
0.021
0,07H
# 010011
0.224 0.000
0U00 = (1000
0,076 0.001
0.08 0.000
0.118
0.0000
0.019 0.011
0,00
0.07
0.006
(00 0.00
CO0L
ROBERTI
MNLI
UOT
+ 0000
0
000 1000
0:06: 0.001
0.001
6.0.001
0.031
066
0.034
0.002
0.110 2 0.002
0.015 0.051
0,00
4 0000
0.0LS 0.031
0,50
2 3000
20077 1 0.002
0.050
0.001
0.8
0.061
0.002
0.153
0101
10030
8 0-0001
0 0:000
- -000
D11 -1 00004
0.113
0003
0105
0.002
0.156
B175 0022
+ 1--055
0
0.000
0
0.107
0/00
WideReaNe13 CIFARID
0.000
0.041
0.000
0,000 0,020
0,00
0 0.000
= 2010
0-00: 0.000
0.941
066
0.001
0.001
0.250
16.000
(L0SS 0.051
1-000
0.001
0.031
0.44
0.003
0.043
6,002
0,8
0.009
0.801
0.206
0.0001
0,001: 0.051
0000
+ 00000
0,013 0.030
02028
20012
= 0.001
0.055
0.002
0.112
0.002
0.200
1.000
UUGS 0000
10 T 10000
0055
00000
0:22
1101: 11 10
01:25
6,00
WiddRodNat138 CIFARI00
6.077
0.001
0.060
1.0000
0.05 0021
0004
2 1-000
0.005
0000: 11
112) 10.00
0/203
0.001
0.330
0005= 0,021
0000
+ 0000
0000 00000
0077+00000
0000
0.00
0.5
0.257 0.00%
0.297 0.000
0,105 0.038
0.101
1 1:000
0,101 0.036
0.10 4 0.036
0.150 0.003
0:371
- 0.004
0.351
00000
0.105 0.034
0.200
+ 0.003
0.101
00034
0.100 + 0.00
0.109 = 0.000
0.307
0.005
ReiNet152
IntageNNct
0:065 + 0:0MM
0.014:
0.002 0.001
0
0.008 00000
0
0.001
01039 0.001
0.030
0.001
066
0.055
0.001
0,263
a 0.001
0.021
00.051
0 1: 0,002
0,021
0.036
0,070 1 0.001
0.019 0.001
0,072 6.00L
0.5
0.07
0.822
0.3122
01001
00031
1108
6- 5-0000
0103
0.000
0085 - 0000
0.002 = 0.000
01082 0.001
01810
0.001
0.129
4-0
01016
0021
1000
+ 1-0100
0012 0.00
00000 0110
1-017 - 0.00
0.791
0.00
ReiNa188
InaaseNct
0.025 0.000
0.077
+: 0.002
0/122 0022
0,00
+ 0:000
0 0.001
0:0 0.001
066
(1071 0.001
0.371 1 1.0000
0.007 0.051
0023
1-000
0.62 0.033
0.:51 0100
0.055 0.00%
0.001
6.002
0.8
0.013:
0,802
0.365 0.0001
0,022 0.651
0000 + 00000
0.001
0.038
0.564 0003
X002 = 0.000
0,077
0.00L
1.0
01051 - 0.001
0.100
10.00
0000
0001
1006
T 1000
0.079 0.015
0115
2 10
1007 - 10000
0.881
6.002
ResNet50
InageNett
0.2
01229 0.001
0,075
+0
0015=1100 0022
1000
+ 2000
0.05 0.000
0000- 0111
7005 =1 11000
0.007 6.001
0.001
0.078
+0
10051
000
+ 0000
0
-
2002
2007 1000
0/016
0.001
0.006 40.00
0,005
100T 00001
3,0
0.076 - 01801
0.116
0100- 0059
0053- 0.034
DUU -1 0100
0.200
0.001
ROBERTA
MNLII
0.2
0.003 = 0.000
0.254
20000
DALLE c: 0001
0000
2 00002
0 00000
0060-
2 0000
20013
=: 20000
0.061
6,003
0.01 0.001
0.290 + 60-037
0003
0 0.00
1001 0000)
60007
0,002
0.100
0,011 00050
0.00
0.000
2 1000
0.025
0.000
0.011 0.002
0.153
10.00
0,007
0.50
0.10
1 2.0000
0JGS 0.000
0.11 3705
1UDS T 00005
0:300 6,005
WiddksoNal3 CIFARIO
03
0.001
0.000
0.051
10.00
0001
0000
1 1-0001
0L00S 0.000
0.00
1 0000
2U013 : 10100
0.052
6.001
04
01000 0.000
0.200
20005
0/022 8000
0000
# 0000
D100 0.000
0.06
0.06 0118
=: 11000
0.007
6.00
0.005 0.000
0.040
1.0000
0.007 0,021
0,56
+ 5:0001
0L005 0.001
0.040 - 1-10
0.00
=: 00000
0.048
6.02
1.15- 0.00
4:00072
0,070 0.033
0.3h1--00 0000
0.53
0115
0.009 = 0.500
0.00
WiddReaNa13 CIFARI00
0.2
0.006 0.030
0.364
0.0002
0,620 0.031
0,080 2 0.0m1
0,021 = 0.030
0.080 5.003
2.014 0.001
0,083
G.00L
04
0.022
= 0.001
0.202
2.0002
0.004 8.031
0.000
2: 00002
0,005 0.001
0.062 0.001
0.00 2 0.000
0,070
0
0.001
0.183
4-0
0.811 0.031
0:100
0.19 0.005
0.139 0.101
0.011 - 0.09
01:20 6.00
0 2626 0.000
0.124
0.20G 0,003
0.115
0.001
0,001 0.039
0,356
21

---

Table 2: Recalibration Results for Imbalanced Classification, For each dataset and atypicality quantile, the best results are marked in bold. We provide the standard errors next to the means over 10 random seeds.
E103
117
01:6
014
01433
0337-1
8.30
CHARSITT
6.0
334
100.00000
888
1130
01.8
119 10000
17011000
6.02
3.90
0.50
177
1080
6.05 1U00
8.65
6075
0.00
300-
6.000
1101 1/00
1111-400 1.00
1791 51000
1.012-000
6.00
0.10
113 30800000
6.0 1000
0103
0138-1
108000 11003
1131-000
1.5--00
1001
175-5000
0776
85 011
3131 1000
0115
1.30 1000
300 1.600
1000
6.05-
Despris11
0123
1480 1000
1.00
EM01
0575 0..0
302 1.00
1ML1100
1196 11000
011
1100 1103
1101
6-20
611
1100 4160
11014.00 100
04.34
110 1000
0.6
5.00
0.10
1000
E15T J00
0350
6.05
3001
1000
110-10000
6-12-
CEARXOLT
11000
033
176 1000
[15 s00
0354
1037 0000
11511 mm
1300 1000
072
1199 003
010
3076
115.
Rablood
8a 04-00 015
5475 1.00
139
1060
1461 3001
3121 4060
Raloodhe22) DeapNooLT
03238
01
119 1000
1131100
0101
11:3 1060
230 100
6175
0111
306 1000
11511000
13151.00
86 01:8
1486 1180
0121
103 1000
038
1000 1103
030
Table 3: Recalibration Results for LLM Classification. For each dataset and atypicality quantile, the best results are marked in bold. We provide the standard errors next to the means over 10 random seeds.
Upcaibeucd
Free
Amsiccllty-Awuc
LLM
Alpecx/t
AG News
0.25
0.10)
11,800
0.210 = B1000
0L5R1
11:02
C:412 11100
0:111 = 11,800
0.521 = 11:010
1/0 (1802
0,192 0JB00
0.714 11/000
0.50
0.165-
-00000
3:000
0.671
0.003
0.413 0.000
0.370 00.000
0.171
0.097 00.00
6.101 01000
0.775
0.000
1.00
0.200
00.000
00000
3.500-100
0005
0.415 0.030
0.350
00000
0.501-000 00000
0.688 0.000
0.310 01900
0.715
IMDR
0.38
0.053
0.001
0/667 -0.05
0.687
0000
0.311 6000
0.184 0.001
0.583-10 0.00
0.111 -000
6.088 0.013
0.967 0.003
0.50
0.008
0.091
0:66
0.99
0,511
-000
0.117 0.03
0.903 0.155
0.38 0.07
0012-000
6.77- 0101
0.940 0.001
0,063
0.0000000
3
6.122:000
3.7020000
06212000
6,08
TREC
0.2
0.100 2 0.030
0.362
0.000
2.74220003
0,250 0:00:6033 6.00
0.26= 0.000
0,70020000
0.100:000
62002000 2.00
0.500:0000 0.900
035
0.217 200000
0.44- -0000
0.00
0.0420000
0,068 00000
0672 0.00
200000000
0.36 21:000000
0:50=00000
0.003
0.622
0,330= 0
22

---

D Recalibration
Through all our recalibration results, we first split the test set into two equally sized calibration and test splits. Then, we fit the recalibration method using the calibration split and compute the performance on the test split. We run all our experiments with 10 different random seeds.
D.1 Temperature Scaling
To perform temperature scaling [GPSW17]. we use the calibration set to fit the temperature parameter. To perform the optimization, we use the LBFGS [L.N89] algorithm from Py PyTorch with strong Wolfe line search, following [GPSWI7]. Namely, we optimize the parameter r with
(x) Sfotmax(f(X))
and then use it during inference rescale the logits produced by S. We 0.1 learning rate and 3000 maximum iterations across all experiments and initialize the temperature value as 1, although find that TS is pretry robust to the choice of hyperparameters.
D.2
Atypicality-Award Recalibration
Here we describe the implementation details For Atypicalit--wware Recalibration (AAR). We formulate AAR with:
log PPAR(((X)) o(a(X)) P(Y]X) + Sy
(10)
In total, this gives | + 3 parameters. Using exactly the same setting as TS, we use LBFGS with a strong Wolfe search to optimize the three parameters, with the same splits as temperature scaling. We normalize the atypicality values (subtract the mean and divide by the standard deviation of the calibration set) for numerical stability. We use the same hyperparameters as TS (with 0.1 learning rate and 3000 maximum iterations) without any modification across all experiments, initialize co, Ci C2 as and Sy parameters as We run the recalibration procedure on CPU with precomputed logits.
D.2.1 Adaptive Temperature Scaling
For AdaTS [33] we use the implementation provided with the paper 7. We identically use the hyperparameters and the architecture provided in the paper and their repository. They use an encoder and decoder architecture with [1024, 512, 512) hidden units each, and temperature predictor network with [128, 128) hidden units. They use an Adam Optimizer with a learning rate of 5e with 128 batch size.
D.3 Conformal Prediction
We follow the presentation in [AB21, ABMJ20). Let x(X) be the permutation of = (1,...,C) C} that sorts P(Y = c(X), i.e. the predicted probabilities for each class c. We define score function
s(x. v) = Σ P(YY = j(x), where y = ne-
(11)
This means greedily including classes until the set contains the true label, and using the cumulative sum the probabilities as the score function. We compute all the scores for the calibration set, Sealb = {s(11, (1),1,1)).... yi.. s.,,,( s(IN,,N)), we the [[N+1)(1-a)] th quantile of the scores, q. Then, the prediction set is defined as
N
C(x) ty s(I 9) < 4)
(12)
We can further add randomization to the procedure where we have the prediction set function to be C(x, u) X [0, 1] for randomization purposes satisfy exact coverage. We refer to [VGS05, ABMJ20, AB21] for a more thorough presentation.
"nanss//eitubb.co///tvv////aaa
23

---

RAPS is a variant of APS that regularizes the set sizes. They modify the scoring function to add regularization term. This controlled by the size offset kyeg that controls the value beyond which the regularization is applied, and the Areg gives the strength of the regularization. To fit the kyeg- Areg parameters in RAPS, we follow the procedure in [ABMJ20] to fit both parameters. Namely, we fit kreg by Algorithm 4 in their paper that leverages the set sizes in the calibration set, and we fit Xreg. by the largest regularization parameter that achieves the smallest set sizes, searched over a grid of {0.001, 0.01, 0.1, 0.2, 0.5) following their presentation.
D.4 Atypicality-Awat Conformal Prediction
We have simple discrete grouping scheme make conformal prediction atypicality aware. Namely, we group points using their atypicality and confidence percentiles and fit individual thresholds. Concretely, we construct a dataset of DAA {(G, G+1], (5+1), (e. using the calibration where (c,, C++1] denotes the confidence range for quantile i, (aj, a5+1] denotes the atypicality range for quantile and J.. denotes the threshold fitted to the group specified by these intervals. We let N 6 the number of groups, and in total, we end up with 36 thresholds. At test time, we check the quantile of the confidence and atypicality of point and use the corresponding temperature. For AA-RAPS, we use the same kreg and Areg values as was found with the RAPS procedure. For practical purposes, we do not allow zero sets (prediction sets at least include the top prediction).
We would like make the remark that sometimes the marginal coverage exceed the desired value (e.g. Figure 9). This is often because the underlying model is already very confident for majority of data points (e.g. More than half of the data points have 92% confidence). The gains provide are often for points with lower confidence regions, as the coverage is not satisfied in those regions.
24

---

6.7
6:29
0.3
335
123
C55
6:5
0.0
3:3
833
625
1.56
26
56
20
53
20
13
16
28
L3
23
0:14
5/6
1.44
5-11
14
1.3
14
E477053
033
6.0
1:09
14
43
539
33.6
18
Comrtg
CRAMtEE
MANGGESS
0.1
626
337
1.26
15
6.11
04
391
181
184
154
15
01
135
25-7
26.1
221
235
0111
0.40
111
8M
6/W
0.
311
312
125
112
123
149
36
160
M11
11N
11Y2
1950
145
0:91
225
NFF
Koptauyy Darth
Figure 8: Atypicality-Aware Conformal Prediction for ResNet18 and ImageNet. Target coverage rate is 95%.
APSIR 551
CAIESEE
AMRSI 25 CreEsEE
AS51300 De S2
MAROO Les
S2
0.7
656
33:
1.55
20
36
5:11
1:36
0141
530
0.
2/41
24
14
19
15
26
85
033
0:35
035
33
854
635
006
0.35
337
855
655
1.4
56
78
82
87
73
57
264
23
355
325
69
6.27
026
136
155
650
6:6
9.6
134
866
657
1.34
265
163
24
256
243
2.0
223
29
664
217
286
204
1.11
01
24Y
D14
391
PM
K14
3131
1141
116
21
11.4
Py Qurch
5.9144
5:21
cm
CAN
114
SON
LG
20
10
1/3
26
10
L2
0:51
0:37
256
0:38
137
1:30
1.1
1.
10
14
035
34
22
0.00
$0.00
Corldence Cuartie
Iouu Aopiballlyy Qunrth
Corkdence Guarths
hout Aopbbatyyyyyy Qonrth
Figure 9: Atypicality-Awar Conformal Prediction for ResNet152 and ImageNet. Target coverage rate is 95%.
ImageNet
CIFAR100
Temperature Th
1.50
1.2
1.25
1.0
0.00
0.25
0.50 0.75
0.00
0.25
0.50
0.75
Atypicality Percentile
Atypicality Percentile
Figure 10: Fitted Temperature vs Atypicality. We observe a monotonically increasing relationship between the atypicality group and the temperature parameter fitted to that group with TS.
E
Tables for Results
Here, present the table version of the results in Figure 3. Tables 1,2 contain the ECE analysis.
25

---

Additive Correction vs Class Atypicality ResNext50 on ImageNet-LT
2
Sy
8
9
10
Class Atypicality a{Y)
Figure 11: Fitted Additive Correction Factor vs Class Atypicality. We observe a monotonically increasing relationship between the atypicality of a class and the additive correction parameter fitted to that class with AAR.
1000000
E:30
0:99
0.33
135
1.00
C55
0.99
333
9.09
1.
100
13
3.0
C8
0.65
332
L3
22
27
Mopical Suath
1A750.15 15:00:1500000 CoAmSSe
MA-A750-1 30 CENEESE
MSSES S4 ZUR
MARKEESS Set CUS
3:3
38
33
0.7
116
655
3.2
332
1.6
78
74
46
0111
0.
241
81
K1
184
TK
030
0.39
032
312
175
617
COT
0.20
334
E55
C54
1.27
32
22
35
35
24
34
365
251
332
437
643
13.
X0
Corlsucs
Figure 12: Atypicality-Aware Conformal Prediction for ResNet50 and ImageNet. Target coverage rate is 95%.
F Fitzpatrick17k and Skin Lesion Classification
We use the training script from [GHS+21] to finetune models on the Fitzpatrick 7k dataset. We train the models for 50 epochs, fixing the backbone and training only the probe on top the penultimate layer. The probe consists of 2 layers, one layer of 256 units followed by ReLU and Dropout with probability 0.4, followed by the classifier layer with an output dimensionality of We use an Adam optimizer with a 0.0001 learning rate.
The entire dataset consists of 16, 577 images, where the potential labels are: 10, 886 inflammatory, 1, 352 malignant epidermal, 1, 194 genodermatoses, 1, 067 benign dermal, 931 benign epidermal, 573 malignant melanoma, 236 benign melanocyte, 182 malignant cutaneous lymphoma, and 156 malignant dermal. We split the dataset into 3 sets (Training (0.5), Validation (0.25), and Test (0.25)). We the validation the calibration set and perform the experiments with 10 random splits.
26

---

G Proofs
G.1 Detailed derivation of the claim on Page 5
When (x - 1/2, the signed calibration error at level u. E (1/2,1) becomes a - P(Y = -1 P-) u) = u - P(Y = -1 Pi(-X) u) = P(Y = 1 IP (X) = u..
The last inequality is due to symmetry. More specifically, we claim (X,Y) d (-X,-Y), where the notation - denotes equal in distribution. In fact, X & X, suffices to show that for any E [-1,1}, and E R,, we have
P(Y = y X x P(-Y = -X=x).
When 1, the right hand side
PP-Y - P P(Y = 1 -2) -1) -2))
=1 -1-0 - =
X = x)) =PY = - X X= =))
Similarly, when 1/ 1,
P(-Y=11 -1---11--- = = = P x= -) = 01(3- -x))
= _ P(Y 1 X x)
We complete the proof.
G.2 Proof Theorem 3.1
Theorem G.1 (Restatement of Theorem 3.1). Consider the data generative model with the al- gorithm described in Section 3.2. For any K > 1, suppose we consider the quantiles of ( 02 .0.,, ax 0K+1 such that P((((X) E (ag,ak+1) ak+1l) = 1/K K for ls E [K]. In addition, we assume || S co. and d/n ffror some sufficiently small co. 16 > 0. Then for sufficiently large n, we have
Eulu- - PPYY = 1 1 (X)) = u) x(X) E [ak-1. ak] >
E. [u - IPY = P (X) u)) a(X) E (ak, ax+1]l,
for k- 2,.., K.
Proof. Following [BMWX21a], we have
|8 ||
u PP = 1 P = u) = -- Ez|or
c 0. o o-)( + +sin 118*
8 | I and 7 1).
where cos
According the results in Section 2.2 of [SC19], we have || - R (*, (*) and cos θ - e" (k,,3"), ) for two quantities It" and c* that depend on x. and 8". We then have
u P( 1 (x) u) + u E || c* -())
c²2 118111
R²
Using the proof of Theorem 3 in [BMWX21a]. we have that
- P(Y = 1 P2 u) Cx(u) . x o(k),
where
c (1)) o ( - c2a" (u)),
for two positive constants C2 As result, we have
u PP = | u) 2
(13)
In addition, since when E increasing for 3 x (u) E 1-)
z 1-11,1,2..'6
and (z) are both increasing, we then have (u)
27

---

Proving the result for {k = 2, K-1] In addition, by our model assumption (0, Ia), we have that ||^|||| and || are independent, and x the d-dimensional space. As the monotonic transformations will not change the events defined by quantiles, and exp(-1[a<1] (eell///) /2) is monotonic function in || for the simplicity of presentation use a(X)= = || in the rest of this proof. As a result, given |||||| = a, we have
|| S where S is a uniform distribution on the sphere
Txx x |||||| = a - a 2..3T T = a || ST,
where S1 is the first coordinate S.
Consequently, if we further condition the event where 9T x > 0 (as we assume 0 throughout Section 3.2), we have
T a |1 S1
S 0 a a. ||
Z1
R
Z1
VZJ +Q
where Q ~~708 - 21 ~ (N, 1) and they are independent.
Due to the monotonicity of Cj((u) on u, we have that for any 01 > 0/
(x(u) | a $ Cx(u)
| dg
where the notation > denotes stochastic dominance. Consequently, have
E., ( Pi( = a(X) E lax-1, axll < Buluu-((Y = = = u) aa(X) (ax, ax+1ll.
for k 2, .., K 1.
Proving the result for = K To complete the proof, suffices to show that the inequality is also true for Kth quantile:
| E., Eu(u-P(Y Px(X) = u) a(X) E [xk]1,ax]] < Eu(u-P(Y = P = u) a(X) E (ax. ax+ill,
which is equivalent
[u = 1 PP( = u)) 1a(()) E (ax-1,ax]l) E.. = - E (ax, ak+1
In the above inequality, the right hand side can be decomposed into
E. [1 TP(Y = (x) = u)) {(a(X) E (ax, [k+1])]
= =E,, [u IPYY (X) u)) 1(a(X)
E., [(u PPY = (x) = u]) ((a(X) E [2p,ax ax+1]]]
Denote the a quantile of x by xa.p' We then have ag. = x2 KFIIP'
We further decompose the equation
into
Eu[(u P(Y ( u)) 1fa(X) E (ax,2p)]
=E,[(u P(Y _ = u)) ((a(X) E [ak, x x ,1H K
[u, PPY = P (X) u)) (a(( E 1x
KK 2p]}] KTI
In the following, we proceed prove
[.[(u-P(Y = 1 = 1)) (a(X) e 1x ,²' 2p|H] [u- = R#T
e [ak-1,
(14)
We now use the approximation of the chi-square quantile: when + co, we have
1 ak (2 k + 22p)2 + o(1), and X x
(2xta + /2p)² + o(1),
Kii
28

---

where denotes the a-quantile of a standard normal random variable.
Then
(2x44 +2 xFI + 2p) 1 (2x48 - ==(2x=+
X K+
ak
2**1
Using the fact that - /2log K + o(1) for K - 20, then we have
-log(1 6)
k
+ o(1)
R=T
KTI
V2log K
In addition, for any a E Ix*x_ x-I
2p] and a' E 1, we have pp
E.[(u - P(Y = 1 u)) a(X) [a) a) E.[((u- - P(Y = (X) = | aa(X) = a')
20 RTI )
for some universal constant C.
Therefore
B. - 11 = u)) a(() E [2i+s
RTI p² 2p]] Ey Eul(( - PPY = 1 (x) = =w) 1a(X) E [x
(zk+s - = 2x1))
R+I
Then
[u P(Y = [X) = u)) 1(a(X) E Ix 2p|H
--
=Eu((u - PP(Y- = 11 (X) = u)) [(X) E ix1+5
K+1 p² 2p] P(a(X) E Ix 2p]) 2p1)
R+1
2 [Eu((u - P(Y = 1 (X) = w)) ((())= E [ax - axl + C(zK44 - -2 "x ))
1
K
+ of
R$1
K +1
K ++11
=E, (u P(Y = 1 (X) = u)) 1(a(X) 1
+ C(2 (2 - (1+ (1)) K+1 Eul(u P((Y = = u)) a(()) E [ak-1,ak]]
The last equality uses the fact that ((a(X) E
= 1/K, and therefore
1 = EE, Pu ( u))- E (ox-1.
Eul(u-P(C 1 (x( = u)) a(X) (uk-1,ak];
K
Then use the fact that EE,[(u P(Y = 1 Px) = u)) a(xx) E [ax-1, = O(1) and we choose å o11/ log K) so
8 o(1 log(1 - 8)
K Vlog
Consequently,
6
K41 C(zK4s -2,,,
E. E. P(Y = (x)) | u)) a(X) E [
K +1
which implies
[,[(u P(Y =1 P( (x) u)) ((a(X) E Ix
2p|H 2 Bu P(Y (x) u)) l(a(X) [ax-1,ax]]
KTT
Combining with equation 13, we prove equation 14 and complete the proof.
29

---

G.3 Theoretical Justification of the calibration improvement using the atypicality score
In this section, we provide the theoretical justification to understand why incorporating the atypicality score will improve calibration. In particular, we consider the binary classification problem with prediction f X E [0, 1] indicating the predicted probability of Y given X x.
For a predictor f. let us denote its conditional calibration error at an atypicality level yy by (J) = (X(((X) - [X (x)) = -]
Theorem G.2. Consider the same setting as Theorem 3.1. Suppose the temperature function F(a(X)) = arg min, mi,,,
with I being the cross entropy loss, and let
PAAR(X) Softma(((((X/((((())) Then
CE, (AAR) I (PF,s, CE,)
(15)
Proof: For a prediction function f. we first define the conditional mean squared error of / at an atypicality level y by (f) [x) - Y)² a(X) 7]. then we have
(S) - (E =B([(X) - y)² a(X) = 7 - [X([(()) B (0 =
=E[E] ((X),
- Y) - la()) =1
=E( - ( -
+ 2E[ -
[ = 7l
Since
B|YB[Y
-]
=B EX ( = 7 ((X) = -I]
=E f(X) (x) = |a()) = 7l.
we have
Y [(X),a(X) -Y) ())(()))))))))))))))))))))))())) F|Y ((x) a(x) =-]
and therefore
MSE, (F, EE ff(X) 7 r)2 |((X) 7
Now that IP (A) a(x:)) is monotonic on the P(xx), we have
B|y [f((), a(x) EEY (AAn( =7]
implying
MSE, (fe AAR) - CE, = MSE, - (P))
(16)
Similarly, we have
MSE, (Prs) (PTS)) MSE, (P).
(17)
In the following, we will show that
MSE, (fP AAR) < (MSE MSE, (f)).
(18)
First, as we consider the binary classification setting, with being the cross-entropy loss, we have I(Y, = =
where (x) = c=)
Then, by the definition of (a(X))) we have that
F(a(X)) = arg min (a-
arg min E ( + - - |
30

---

Taking the derivative on the last line and setting it to zero, we have
Y
1-Y
la(x)] = O,
1-o(((X)/(((X)) 1 -
implying
(xt B|y a(x)].
This makes the derivative of ) a(X)) zero and therefore F(a(X) is
also minimizer of E[(Y -
F(a(X)) arg min E /c arg min (((xxo(((())/((xx)))
Letting ((7) arg min, B[(Y off(()( a(X) 7), we have that
g(a(X)) arg min B {|)
and therefore
g(a(X)) = arg min - (1//((XX-((//////2)))1111111 -((X))
As result,
MSE. (AA)) =E[(P (An - a(X)= = 7]
- -Y² (aX) =1
=B((((((X)/((((x)) - r)² a(X) = 7]
([((Softmxx(P(x))o0ll - -Y) aa(X) = 7
=arg min (Xxo(((X)/c- r)2 a(X) =n]
<T[(O([(X)) 2 a(X)
(P).
Similarly, we have (S AAR)) I (PE, and therefore equation 18 holds.
Combining with equation 16 and equation 17, we have
CEE((AAA)) S CE
31

---

Table 4: Conformal Calibration with AtypicalttyAAwareeee
MAKI
Dxad
lepe. Appiaaly Srrap Cooldaan Shoap
A4AAS
AA-AAS
Cererge
Scdiuc
Cerema::
Crerns
Scelec
Dovoue::
Sabter
Iragehn
018022 10311
22 2,248
192
00521
11.5
00311
0 180
1.255
0555 1186
1120
5.1.5L2
0097 0099
6.365
0.250
02.51
00001 108
0.300
14.35321113 1.39
1007 1080
1.068 2-1.30
0101 0199
5.415
18.35 116
0163
1.179 4-111
2-332
12.341
1000 1086
1000
0.30
0.853 3.005
12. 12.05 0.33
1000 106
0.00 3.105
1065 1-0555
2-00- 01852
1-100
1007
00038
10NS 1980
01127
2-
1000 0100
1100
0000 0001
1000
0300
21.051
0000 EUKD
1381
0.122
1000 1000
1005 0005
1:20-1 1110
0.00 1001
1000- 0000
0.099 0.000
1.00 0.000
1000 1001
1/00.1 D000
1000
2112
01800
(086 1181
(100)
10000
1129 1192
1000
1200 0,000
1000
RedNetk
Iragetie
03300
29.27
1070 10011
2.200
05.533 222551
1dt 1080
1
03077
186
0.65 1100
100 037 6.00
(100 6101
01911 10314
12.12 -
3-107
0.009
25.381 131557
13/02
00000
0.515
116 0:301
0.048 5.1005
43337 131
1000 1000
1100
0.000
10.53 1135
11811
33:204
0.307
1060
17 20200
22.35 20533
1002 1060
10702
0.557
5a0t 0110
1100
100 0000
13:15
155
01230
03155
116
0.883 3.000
1.156
6:700
11:50
1.000
1.30
0:200
1:3m
tu00 100
15:0-33
1.250
11000 1001
11002 11200
61000
-1000
0900 0000
1.300 1.211
100
ReNNeo Inagehe
10.100 5.004
2.04
1065 EUNG
5
0.10
1006: 1085
03700
1050-
5:750
0.053 5.3003
1.59
15.74 01
0107 3.105
1.41
1080 (000
11100
1091
1:154
0300
5.150
1-20
21000
10R0 1101
1- 23 51120
2100
01857
10.53-00 -010 10
1162
13.54 11.55
0302 (000
7.011 5.114
71625
51010
113
1000 (181
1220-
11000
01297 1000
0177
0.055 3.00
4528 03333
1372
E1170
0103 31005
1101 0-033
0:201
1962
0103
10.20 1.047
(0001 1080
11161
0.570
1090- 0,90
EURC 100
1051
1.800
0.001
0.90
199 1000
1997
1/80-000 (UE0
0377%
1301
23 0122
030: 23100
L1502-0165
0191 0101
100-000 0120
0140 11100
(191 6000
1-004 (000
010 -110
[00 1000
01911
L120 1122
[000 1UNG
1/00:2 2284
11311
1380-100 E/30
0564 1185
1500
0397 033
1000 0000
0090 1091
1100-11000 (100)
1100
0 6/11
1/80 0/00
0.57 1.00
LE30 (133
[UK7 4 (0)
1/80.2 (/00)
0.67 8: 3160
1100 1: 11000
32

---

ResNet18 ImageNet
ResNet50 ImageNet
WideResNet28
0.08
0.04
E
0.05
ECE
0.05
ECE
0.03
0.02
0.03
0.00
1.00 Input Atypicality Quantile
0.25
0.50
0.75
0.25
0.50
0,75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
Input Atypicality Quantile
Input Atypicality Quantile
Input Atypicality Quantile
Uncalibrated
AdaTS
AAR(Ours)
ResNext50 ImageNet-LT
ResNext50(+PBB) ImageNet-LT
ResNet152 Places365-LT
ResNet18 0F-R100-LT
0.40
0.20
ECE
0.10
0.20
6.25
0.50
0.75
1.00
0.25
0.75
1.00
0.25
0.50
0.75
0.25
8.50
0.75
1.00
(b)
Atypicality Quantile
Atypicality Quantile
Atypicality Quantile
Class Atypicality Quantile
Accuracy,
0.5
3.44
0:44
0.44
Accuracy
0.5
0.47
0:47
0.47
Accuracy
22
0.30
D.30
Accuracy
0..
0.45
0:45
0.45
0.50
0.0
TS
AdaTS
AAR(Ours)
Figure 13: Post-hoc Recalibration for Classification with 5-NN distance as an Atypicality Metric. (a) Balanced Supervised Classification: AtypicalityAAwan Recalibration improves the calibration of models trained with balanced datasets, across atypicality groups. (b) Imbalanced Supervised Classification:
Atypicality-Awar Recalibration improves both the calibration across groups and the overall accuracy models trained with imbalanced datasets with 5-nearest neighbors distance as an atypicality metric.
H
Limitations
H.1
Quantifying Atypicality
Since do not have access the true distribution of TP(X), we estimate it through the model, e.g- using the embeddings. This means we are capturing the atypicality not solely with respect to the training distribution but also the model. It is possible that model that does not fit the data well and produces low-quality atypicality estimates. We would like to stress that our goal here is to show that even simple estimators can demonstrate significant benefits. In general, we observe that our findings hold for large datasets and widely used models, and atypicality gives semantically meaningful way to group data points qualitatively. Our findings suggest that we can unify the understanding and improve uncertainty quantification and recalibration methods with atypicality, however, practitioners should be careful about incorporating atypicality, as poor atypicality estimates can lead worse performance.
H.2 Subgroup Fairness Experiments
While the literature on algorithms satisfy group fairness is rich [KGZ19, HJKRR18), here wanted to give a case study with skin-lesion classification. Our goal was to provide further evidence that atypicality awareness could improve fairness algorithms. In this domain, there is more verification to do to better characterize how and when atypicality helps, thus, to better understand which subgroups could benefit more from atypicality-awan algorithms and whether these findings apply generally in the literature.
H.3 Theoretical Analysis
Following the earlier work [BMWX21a, SC19], we analyzed the calibration behavior of well- specified logistic regression. However, our empirical findings suggest that the phenomena are much more broadly applicable. We suggest that future work can analyze the behavior in more general settings to better understand the dynamics.
33

---

Distribution of Input Atypicality
ResNet18 ImageNet (GMM)
ResNet18 ImageNet (KNN)
1000
800
800
Count
600
Count
600
400
400
200
200
0
0
250
500
750
1000
1250
10
15
20
25
Figure 14: Distribution of Input Atypicality. Here, we give the distribution of Atypicality for ResNet18 ImageNet, using GMM and KNN methods.
ResNext50 on ImageNet-LT (ccuracy(B..)
ResNext50 on ImageNet-LT (o Accuuaccyy,
0.91-1.00
0.97
0.93
0.86
0.62
0.16
0,91-1.00
0.01
0.04
0.10
0.34
0.79
0.66-0.91
0.77
0.75
0.65
0.45
0.18
0.66-0.91
0.03
0.05
0.14
0.33
0.59
Confidence
Confidence
0.44-0.66
0.55
0.51
0.46
0.37
0.14
0.44-0.66
-0.00
0.04
0.08
0.17
0.39
0,26-0.44
0.38
0.34
0.30
0.24
0.11
0.26-0.44
-0.03
0.01
0.05
0.11
0.24
0.03-0.26
0.21
0.17
0.16
0.12
0.06
0.03-0.26
-0.03
0.01
0.01
0.06
0.12
Label Atypicality Quantile
Label Atypicality Quantile
Label Atypicality and Confidence. Here, the x-axis reflects the label atypicality quantile, and the y-axis indicates confidence. The coloring for the figure on the left indicates the accuracy within a bin, and the figure on the right has the difference between confidence and accuracy within bin. Similar to Figure 6, we observe that atypical examples have lower accuracy, and predictions are more overconfident.
Figure 15:
ResNet152 on ImageNet Accuracy(B..)
ResNet152 on ImageNet (aiiidence(B...AAccaaeee
1.00-1.00
0.99
0.99
0.99
0.99
0.99
1.00-1.00
0.01
0.01
0.01
0.01
0.01
0.99-1.00
0.98
0.96
0.96
0.96
0.96
Confidence
0.99-1.00
0.02
0.03
0.04
0.04
0.04
Confidence
0.90-0.99
0.91
0.91
0.88
0.86
0.85
0.90.0.99
0.04
0.05
0.08
0.09
0.10
0.63-0.90
0.74
0.71
0.71
0.67
0.62
0.63-0.90
0.04
0.08
0.08
0.11
0.16
0.06-0.63
0.44
0.44
0.38
0.35
0.32
0.06-0.63
0.06
0.03
0.05
0.07
0.11
Input Atypicality Quantile Figure 16: Input Atypicality and Confidence for 52sNet152, We provide the input atypicality for ResNet1 in the same structure as Figure 15.
Input Atypicality Quantile
34


Reviews:
[{"review_id": "P0mr6dAqrk", "reviewer": "Reviewer_y4KB", "summary": "The paper questions the reliance of probabilistic classifiers on the confidence score alone for measuring reliability. This is an important question that has not been asked that rigorously in the machine learning literature. In particular, there are two notions of uncertainty: aleatoric and epistemic. A widespread consensus is to consider the softmax applied output of a neural network as the confidence of the classifier. However, it is not clear what kind of uncertainty that score captures. Literature in proper scoring rule would suggest that it measures the posterior probability of the input falling into some class, which is inherently an aleatoric notion of uncertainty. However, practitioners and researchers in the community also hope that such a score would be measure of reliability of rare / out-of-distribution / atypical samples, which is more of an epistemic nature of uncertainty. Therefore, there is some sort of conflation is happening on the nature of this score. A sample could be low confidence because it is inherently ambiguous (aleatoric) or is not well represented in the training distribution (epistemic). The paper opens up with this question, and gives arguments to quantify *atypicality* of the sample as well for more reliability. With a simple measure to quantify atypicality, it shows that there is a a correspondence between how atypical a sample is and how mis-calibrated it is. It then studies atypicality aware recalibration. There is a sufficient empirical evaluation. ", "questions": "Some questions follow:\n\n1. (Section 4.2): I'm not sure I follow the intuition behind Equation 2. $\\psi$ is a function of input atypicality. What is the nature of $\\psi$? Is it monotonic, as in it preserves the atypicality of the input. If I assume so, then for a typical input, let's say the atypicality score is 0, in this case $\\hat{\\mathbb{P}}_{\\text{ARR}}(Y\\vert X) = \\frac{\\exp\\{S_Y\\}}{Z(X)}$, which is not intuitive. I'm sure this is my misunderstanding, so would appreciate the clarification. Specifically, the nature of $\\psi$ and $\\phi$. \n\n2. (Line 222): The paper shows that atypicality-aware recalibration improves the overall accuracy in imbalanced supervised classification. In my mind, this is problematic, no? Recalibration techniques should preserve the nature of classifier, is called the sharpness / refinement property of calibration maps. Furthermore, ECE roughly translates to difference in average confidence and average accuracy. And the high ECE in atypicality is the result of lower accuracy but higher confidence. One could improve on ECE by compensating for accuracy instead of the confidence, which in my opinion is not the goal of calibration. I would love to hear from the authors on this.\n\n3. (Section 4.3): I understand that atypicality aware recalibration is useful in fairness methods. Although it is an interesting case study and not the major focus of the paper, could authors comment (and compare) on the notion of multicalibration in the literature. I think atypicality aware recalibration has the same spirit as multicalibration in fairness literature.\n\n4. (Section 5, Line 266): For conformal prediction, the authors write \"group points according to their confidence and atypicality quantiles...\", do the authors use some function to club the confidence score and the atypicality score into a single score? Or some multiple testing procedure is used to accommodate both the score? ", "limitations": "None", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. One of the major strengths of this paper is that it raises a very relevant question. Interpretation of the confidence outputted by the classifier has a tendency to suffer from serious conflation issues where it is not clear what it captures. The paper argues for a different notion that can settle this issue (or at least start interesting discussions in the community). I believe this is highly valuable.\n\n2. While the operationalisation of the proposed atypicality score is simple, the scoring criteria is already good enough to support the arguments made in the paper in a satisfactory way.\n\n3. Thorough empirical evaluation is done on different settings ranging from balanced supervised classification, imbalanced classification , Language modelling, recalibration, and conformal prediction uncertainty sets on a range of datasets. ", "weaknesses": "1. There are no major glaring weaknesses of the paper. Except that maybe it underplays itself, as in my mind, it puts interesting and insightful commentary on the nature of softmax output (or sigmoid for Binary classification) of the classifier. There is a great debate on what that is, see Appendix B.2 in [1]. The paper could be even more impactful if these discussions are taken into consideration.\n\n2. Section 3.2: Although some theoretical results are presented that shows correspondence between atypicality and overconfidence in a simple model, it is not clear how these results apply to the other things in the paper. It would be good to draw other insights from it. Do these results provide any insight (or intuition) on why atypical examples would be mis-calibrated? Is this due to estimation errors?\n\nPersonally, I thought about this question myself. And to think of it,  it does make sense intuitively why mis-calibration might be an issue for atypical examples. Consider Binary classification, and assume that one is perfectly able to model $\\eta(x) = \\mathbb{P}(Y=1\\vert X=x)$ where the evaluation of $\\mathbb{P}$ happens with respect to the training data distribution $P$. Now for a sample $x_{new}$ that is not well-represented in $P$, it is not unsurprising to see that $\\mathbb{P}(Y=1 \\vert X=x_{new})$ would be low. However, there is no reason to expect that it would be calibrated. Could authors comment on or augment this intuition?\n\n\n\n\n[1] Tim Pearce et al. Understanding Softmax Confidence and Uncertainty. (2021)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer y4KB,\n\nThank you very much for your detailed review! We really appreciate to hear that you find that the notions proposed in the paper can start interesting discussions in the community and our empirical evaluation to be thorough. We really appreciate your approval and kindness!\n\nBelow are our responses to your questions:\n\n### Response to Questions / Comments\n\n- Q1. When $\\psi(a(X))=0$, the output distribution reduces to a fixed distribution over classes that was estimated on the calibration set. This distribution can be seen to induce a prior probability over classes. $\\psi$ trades off between this distribution and the model’s confidence. When the point is typical, the output distribution is closer to $\\hat{P}(Y|X)$. Empirically, we indeed observe $\\phi$ to have a monotonic relationship with atypicality. Note that Theorem 3.1 similarly suggests a monotonic relationship. If a point is atypical (say $\\psi \\approx 0$), then the linear model is more overconfident, thus we should have a larger correction (e.g. Larger temperature). \n\n- Q2. Thank you for raising this important question. Most of the accuracy improvements arise in the imbalanced classification settings. As we observe in our analyses, the model often puts less probability mass on the rare classes and puts more confidence on the typical(common) classes. If we add class-typicality in the recalibration procedure, we can fix this issue, and this can change(and increase) the accuracy. Indeed, in Appendix Figure 11, we observe that more atypical classes get larger positive corrections, which increases the predicted probability.\n\n- Q3. Indeed there are shared notions between multicalibration and our work. While multicalibration can be considered as a general framework to study group fairness, we are interested in a very specific definition of groups, which are atypical and typical groups. We believe that atypicality would be a useful notion that could be integrated into any multicalibration algorithm, which hopefully future work can study.\n\n- Q4. We split points according to both the atypicality and confidence. Namely, a group $G$ here is defined using 4 thresholds, namely $G = \\{x: (l_{a}^{(G)} < q_{a}(x) \\leq h_{a}^{(G)}) \\land (l_{c}^{(G)} < q_{c}(x) \\leq h_{c}^{(G)}) \\}$ where $q_{a}(x)$ and $q_{c}(x)$ denote the atypicality and confidence quantiles for the sample $x$, $l_{a}^{(G)}$ and $h_{a}^{(G)}$ denote the atypicality lower and upper bounds for group $G$, and $l_{c}^{(G)}$ and $h_{c}^{(G)}$ denote the confidence lower and upper bounds for group $G$. Using a calibration set, these bounds are simply determined by the quantiles of confidence and atypicality statistics. \n\n\n- On your intuition & Section 3.2: Thank you for providing your intuition! First of all, the insights from the theoretical model indeed transfer empirically to more complex models. As one can note in Figure 1a or Figure 5-6, predictions for more atypical instances are more overconfident and have lower accuracy/coverage(for prediction sets). Further, we mostly agree with your intuition on the estimation errors, but this alone is not sufficient to explain why specifically overconfidence happens (and say, not underconfidence). For this point, the activation function becomes an important factor, and Bai et al. results that we build on provide a good discussion. \n\n- Pearce et al.: This is indeed a very interesting and relevant reference, thank you for bringing this to our attention! We will be sure to include this reference in our revision.\n\nThank you so much for all your questions and comments, we really appreciate your time! Please let us know if there are any further questions that you may have. \n"}, {"review_id": "LUkh7VB8jP", "reviewer": "Reviewer_AQHi", "summary": "This paper proposes a series of \"atypicality\" measures to be used in complement to the more popular uncertainty metrics.\nThe authors defined input and class atypicality for both classical classification tasks as well as NLG.\nSuch atypicality measures are then combined with temperature scaling to improve calibration quality, as well as prediction sets (RAPS).", "questions": "1. Related to W4, why is atypicality defined so differently for NLP tasks? The intro keeps suggesting that the input is atypical wrt a class (e.g. Figure 1). In Definition 2.1, for example, it is also conditioned on $Y=y$. However, L124 uses a completely different definition. In fact, even L125 ($a_Y(y)$) has a completely different meaning than Definition 2.2. \n2. Intuitively, what's the rationale of assigning temperatures basing on atypicality? It seems like typical samples could share a temperature, but for atypical ones, it's not like atypical samples themselves are similar.\n3. Are the theoretical guarantees still maintained after the modifications to RAPS?\n4. Is Theorem 3.1 somewhat mechanical? That is, is this just due to large variance and the fact that $u-\\mathbb{P}(Y=1|\\hat{\\mathbb{P}}_1(X) = u)$ tends to mean reverse?\n", "limitations": "Mostly discussed in weaknesses. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The problem is definitely important, and this paper touches a lot of areas where the particular measures of \"atypicality\" could help.\n2. Linking atypicality to conformal prediction is good. (I'd suggest using \"prediction set\" as opposed to \"uncertainty set\" though). This seems to lead to good results in Figure 5 as well. \n3. AAR seems to improve ECE over the baselines significantly. \n4. I actually think this paper has a lot of interesting ideas, such as Theorem 3.1 (but somehow constrained by the space and many are not fully developed.) ", "weaknesses": "1. I really think it is inappropriate to make the dichotomy of \"Atypicality\" vs \"Confidence/Uncertainty\". In fact, I would argue the various forms of atypicality in this paper are just a bunch of confidence measures. This paper actually also proposes several different variants of \"atypicality\", and in the uncertainty space, for example, monte-carlo dropout variance and entropy are both used to estimate uncertainty. Thus, I'm not sure what makes \"atypicality\" a separate concept. \n\n2. In my opinion, this paper is proposing another calibration method, but puts everything in very different languages. \nIt is also missing a lot of calibration baselines. \nIn fact, recent works rarely focus on confidence calibration anymore, but more on full calibrtion [[1]](https://proceedings.neurips.cc/paper/2019/hash/8ca01ea920679a0fe3728441494041b9-Abstract.html), [[2]](http://proceedings.mlr.press/v89/vaicenavicius19a/vaicenavicius19a.pdf), [[3]](https://proceedings.neurips.cc/paper/2019/file/1c336b8080f82bcc2cd2499b4c57261d-Paper.pdf), [[4]](https://openreview.net/forum?id=p_jIy5QFB7).\nIn fact, [4] also used density estimates to calibrate classifiers much like L115-L117 and Definition 2.1\nI also think the \"Recalibration\" section in Section 6 needs a re-write, as it's uncommon to put conformal prediction in this context: Yes, conformal prediction could be considered as calibrating the distribution/p-value, but when we use ECE to measure model calibration, conformal prediction is just out of context. \n\n3. Related to 2, I think the improvement of RAPS/APS would benefit mostly from full calibration as opposed to confidence calibration. The lack of such experiment is a limitation. \n\n4. While the improvement of LLM performance is good, I don't see the connection between the main point of the paper and such experiments. See Questions as well.\n\n5. Presentation is not clear, probably due to the attempt to jam in too much content. For example, L122-126 should be expanded as it is a very different thing in my opinion. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer AQHi,\n\nThank you very much for your detailed review. We really appreciate that you find the problem important and could be helpful in practice; we share the same ideas! We are also very happy to hear that you find a lot of interesting ideas in the paper, thank you very much for your kindness!\n\nBelow are our responses to your questions and comments:\n\n### Responses to Questions / Comments\n\n- Atypicality vs Confidence: Confidence and atypicality are correlated quantities that are clearly not independent, so we agree that there is not a dichotomy per se. Our main message is to highlight if and when having the specific notions that we use could provide utility. As an illustration, let us think about the Bayes Rule: $P(Y|X) = \\frac{P(X|Y)P(Y)}{\\sum_{Y’ \\in \\mathcal{Y}} P(X | Y’)P(Y’)}$. For example, let us have two inputs $X_{1}, X_{2}$ where $X_{1}$ is an out-of-distribution example that is not from any of the classes in the problem, and we have $\\forall Y’ \\, P(X_{1} | Y’) < \\epsilon$ for some $\\epsilon$. Similarly, $\\exists \\tilde{Y}: P(X_{2} | \\tilde{Y}) \\gg \\epsilon$ and this input is ‘in-distribution’. Looking at the Bayes Rule, we can have the same confidence distribution for the two points $(P(Y|X_{1}) = P(Y|X_{2}))$ as long as the likelihood ratios are the same, where $X_{1}$ is an OOD point, whereas $X_{2}$ is potentially from one of the classes. In other words, an atypical input $X_{1}$ that is not likely to be from any class can have the same confidence value as the typical input. Therefore, quantifying and accounting for both confidence and atypicality can provide value and in our study, we demonstrate the value in improving calibration and accuracy. \n\n- Comments on writing: We changed the term uncertainty set with prediction set in the paper. Similarly, we slightly modified the recalibration part of the related works to make the distinction for conformal and recalibration. We appreciate both of these suggestions.\n\n- Q1. Thank you for raising this question, we believe it is important to clarify this. Intuitively, our atypicality measure aims to quantify how well-represented an input or a class is in the training data. $a(X)$ is a notion to evaluate whether an input is well-represented. The key idea is that a larger atypicality value indicates $X$ is not well-represented in the patterns seen during training.  $a(X)$ could be implemented in a class-conditional way by looking at $P(X|Y)$ for each class $Y$, or in a marginal way by looking at the overall $P(X)$, and we adopt the one that is more practical to use. For LLMs, it is unclear how one would quantify $P(X|Y)$, however, the model itself is an estimator of $P(X)$, making this notion of typicality readily available. This quantity still informs us about whether the input prompt is well-represented w.r.t. the training data. We will also add further discussion on this to the Appendix for further clarification.\n\n- Q2. The intuition is that the model’s predictions are overconfident for atypical points. A larger temperature leads to the reduction of confidence and thus larger uncertainty, and we indeed observe larger temperatures for atypical samples. Overall, it is a way to refine the model’s confidence to reflect the higher uncertainty. \n\n- Q3. We have not theoretically analyzed the setting for prediction sets. Unfortunately, we do not believe such an analysis could be fit in this particular paper given the space constraints, but we will add further clarification on this under the ideas to future work. In particular, utilizing the proof techniques in [1], one can prove that for conformal prediction in the regression setting, an initialization $f_0$ with smaller MSE will have a shorter prediction interval after the post-processing of split conformal. Future work can extend this analysis to the classification setting.\n\n- Q4. Thank you for your comment. The larger variance of $\\hat{\\mathbb{P}}_1(X)$ for more atypical $X$ is indeed one cause of our result. While this can generally explain large calibration errors, it cannot specifically explain overconfidence. That is, it does provide a reason for why the sign of $u-\\mathbb P(Y\\mid \\hat{\\mathbb{P}}_1(X)=u)$ is positive. Our theorem provides a more specific analysis to characterize the overconfidence phenomenon. \n\nThank you so much for all your questions and comments, we really appreciate your time! Please let us know if there are any further questions that you may have. \n\n[1] Lei, J., G’Sell, M., Rinaldo, A., Tibshirani, R. J., & Wasserman, L. (2018). Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523), 1094-1111."}, {"review_id": "rtg3vm3OCR", "reviewer": "Reviewer_vUKB", "summary": "This paper is addressing the problem of atypicality in data, and how this impact performance and confidence.", "questions": "N/a", "limitations": "N/a", "rating": 7, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Good perspective on the needs to consider atypicality in data\nGood presentation\nGood review on the links between atypicality in data and performance with confidence", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer vUKB,\n\nThank you for your kind remark. Let us know if you have any questions or comments."}, {"review_id": "WzHCaDR65c", "reviewer": "Reviewer_7E3C", "summary": "This paper focuses on the question of uncertainty quantification in classification by introducing the atypicality during recalibration. This paper exhibits a highly explicit motivation, yet it suffers from certain deficiencies in definitions and errors therein, and I harbor reservations regarding the soundness of the approach posited in Sec. 4.2 Atypicality-Aware Recalibration.\n", "questions": "\nQ1: In Fig.~1, it is unclear whether the example presented therein is a genuine instance from the experiment. If it is, then it is puzzling why specific values for atypicality and confidence are not provided, and instead, actual and predicted labels are employed.\n\nQ2: There are errors in definition of accuracy(Bm), what's worse, I can't find the formal defination of Bm.\n\nQ3: RMSE, which is used as a metric in the paper, but I can't find the definition of it. To my understanding, RMSE is a metric for regression rather than calibration.\n\nQ4: The necessity of Eq. 2, and whether there exist alternative methods to the approach posited therein, or whether it is indeed necessary. I could understand Sec. 4.1, and the natural selution is group-wise TS calibration, so more detailed analysis is required.\n\nQ5: Atypicality for LLMs. The defination in line 124 a(x) is a distribution, it seems different to the defination 2.1.\n\nQ6: How is the expected calibration error (ECE) computed on large-scale models? Is it obtained by taking the softmax of all possible values along the final dimension, or by normalizing the top-p values as confidence scores?\n\nNote: Use \"Line xx\" to describe the Eqs in the text is cumbersome and detracts from the readability of the communication.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper evinces a highly explicit motivation and lucidly illustrates the concept of atypicality posited therein through the use of schematic diagrams.\n\nThe paper presents copious experimental results and supplements them with additional experiments in the appendix, which serves to facilitate the reader's comprehension of the experimental outcomes.\n", "weaknesses": "The presence of errors and deficiencies in definitions within the article renders this work somewhat unreliable.\n\nAlthough the article contains numerous derivations, the critical section, namely Sec. 4.2, lacks theoretical analysis. The author fails to analyze the validity and necessity of Eq. 2, and whether there exist alternative methods to the approach posited therein, or whether it is indeed necessary. These issues require requisite analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7E3C,\n\nThank you very much for your detailed review and your kindness! We appreciate the time you took to review, and we are really excited that you find our motivation clear and supported with copious experimental evidence! \n\nBelow are the responses to your questions:\n\n### Response to Questions / Comments\n\n- Q1. The instances are indeed from the ImageNet-R dataset. We sampled those points according to the atypicality and confidence values and their quantiles. In the revision, we will add the quantile values to Figure 1. Thank you for this suggestion.\n\n- Q2. Thank you for pointing this out, we further clarified the meaning of $B_{m}$ in Appendix D.1. Just noting here too: $B_{m}$ here denotes the set of the data indices where the confidence of the prediction for the sample falls into the interval $(\\frac{m-1}{M}, \\frac{m}{M}]$ where $M$ is the number of bins. For instance, if we have 10 bins, $B_{1}$ contains samples with confidence between $(0, 0.1]$. \n\n- Q3. The definition is Appendix D.1 Equation 8. RMSCE is simply the root-mean-squared version of the ECE metric. This metric is based on the earlier works e.g. the cited paper [HMD18]. \n\n- Q4. Thank you for raising this point. Group-conditional calibration is possible when there is a natural definition of a group (as in the Fairness Experiments presented in Figure 4). However, when we do not have the definition of a group, it is unclear how many groups to form, according to what threshold, etc. In practice, we would face difficulties in picking the number of groups, finding thresholds to form groups, and justifying this choice. Supported by the parametric relation we observe in Appendix Figure 10 and therein, we follow the proposed methodology to overcome such practical hurdles. Indeed, Figure 10 suggests that there is a monotonic relationship between atypicality and temperature that could be captured by the simple parametric form that we have that requires only 3 parameters.\n\n- Q5. Thank you for raising this question, we believe it is important to clarify this. Intuitively, our atypicality measure aims to quantify how well-represented an input or a class is in the training data. $a(X)$ is a notion to evaluate whether an input is well-represented. The key idea is that a larger atypicality value indicates $X$ is not well-represented in the patterns seen during training.  $a(X)$ could be implemented in a class-conditional way by looking at $P(X|Y)$ for each class $Y$, or in a marginal way by looking at the overall $P(X)$, and we adopt the one that is more practical to use. For LLMs, it is unclear how one would quantify $P(X|Y)$, however, the model itself is an estimator of $P(X)$, making this notion of typicality readily available. This quantity still informs us about whether the input prompt is well-represented w.r.t. the training data. We will also add further discussion on this to the Appendix for further clarification.\n\n- Q6: We use the post-softmax probability for the most probable class as the confidence, and compute ECE using this quantity. This is indicated in Equation 7, but we will also add further clarification on this to the Appendix.\n\nThank you so much for all your questions and comments, we really appreciate your time and kindness! Please let us know if there are any further questions that you may have. \n"}]

Author rebuttal:

Meta-review: This paper addresses the issue of model confidence in relation to a notion of atypicality, showing that traditional models are often strongly overconfident on examples that are drawn from regions of less support in the training data.  The paper builds on prior work investigating the limitations of model confidence for uncertainty quantification, and proposes that models should incorporate atypicaltiy measures directly when quantifying uncertainty.

After careful reviews and a full set of responses to author rebuttal, there was a clear consensus among the reviewers that the paper is addressing an important topic, making a simple but useful contribution that is well motivated and shown to be strongly effective.  I expect that the authors will incorporate the feedback from the reviewers into the revision for the final paper, especially using the points raised for clarification as an opportunity to strengthen the paper for the general (typical?) NeurIPS reader, but overall the paper is clearly ready for acceptance.

The only thing I would add is that to me, it appears that there is likely a strong connection between the notion of atypicality proposed here and observations around the effectiveness of deep ensembles for uncertainty quantification.  It seems to me that ensemble disagreement is likely to be highest for atypical examples, and that there may be a useful connection there that can help to draw some unified conclusions.
Decision: Accept (poster)