SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mert Yuksekgonul
merty@stanford.edu
James Zou
jamesz@stanford.edu
Linjun Zhang
lz412@stat.rutgers.edu
Carlos Ernesto Guestrin†
Stanford University, CZ Biohub
guestrin@stanford.edu
While most machine learning models can provide confidence in their predictions, confidence is insufficient to understand a prediction's reliability. For instance, the model may have a low confidence prediction if the input is not well-represented in the training dataset or if the input is inherently ambiguous. In this work, we investigate the relationship between how atypical (rare) a sample or a class is and the reliability of a model's predictions. We first demonstrate that atypicality is strongly related to miscalibration and accuracy. In particular, we empirically show that predictions for atypical inputs or atypical classes are more overconfident and have lower accuracy. Using these insights, we show incorporating atypicality improves uncertainty quantification and model performance for discriminative neural networks and large language models. In a case study, we show that using atypicality improves the performance of a skin lesion classifier across different skin tone groups without having access to the group attributes. Overall, we propose that models should use not only confidence but also atypicality to improve uncertainty quantification and performance. Our results demonstrate that simple post-hoc atypicality estimators can provide significant value. $^{1}$
Typicality is an item's resemblance to other category members [RM75]. For example, while a dove and a sparrow are typical birds, a penguin is an atypical bird. Many works from cognitive science (e.g., [Rip89, RSS73, MP80]) suggest that typicality plays a crucial role in category understanding. For instance, humans have been shown to learn, remember, and refer to typical items faster [Mur04]. Similarly, the representativeness heuristic is the tendency of humans to use the typicality of an event as a basis for decisions [TK74]. This cognitive bias is effective for making swift decisions, but it can lead to poor judgments of uncertainty. For instance, the likelihood of typical events can be overestimated [TK74] or uncertainty judgments can be inferior for atypical events [TK92].
While it is hard to quantify the uncertainty of human judgments, machine learning models provide confidence in their predictions. However, confidence alone can be insufficient to understand the reliability of a prediction. For instance, a low-confidence prediction could arise from an ambiguity that is easily communicated, or due to the sample being underrepresented in the training distribution.
1Our code is available at https://github.com/mertyg/beyond-confidence-atypicality
$\ddagger$  Joint Advisors.
Figure 1: Atypicality in Uncertainty. Left: We show examples from the ImageNet-R dataset with our atypicality framework. Right: We provide a conceptualization of the quadrants. Using atypicality, we can understand prediction quality ( $\S 3$ ), improve predictions ( $\S 4$ ), and prediction sets ( $\S 5$ ).
Similarly, a high-confidence prediction could be reliable or miscalibrated. Our main proposal is that models should quantify not only the confidence but also the atypicality to understand the reliability of predictions or the coverage of the training distribution. However, many machine learning applications rely on pretrained models that solely provide confidence levels, devoid of any measure of atypicality.
Contributions: To support our position, we use a simple formalization of atypicality estimation. With the following studies, we show that by using simple atypicality estimators, we can:
1. Understand Prediction Quality: Calibration is a measure that assesses the alignment between predicted probabilities of a model and the true likelihoods of outcomes [GR07]. Neural networks [GPSW17] or even logistic regression [BMWX21a] can be miscalibrated out-of-the-box. Here, we argue that using atypicality can give insights into when a model's confidence is reliable. Through theoretical analysis and extensive experimentation, we demonstrate that atypicality results in lower-quality predictions. Specifically, we show that predictions for atypical inputs and samples from atypical classes are more overconfident and have lower accuracy.
2. Improve Calibration and Accuracy: Recalibration methods offer some mitigation to miscalibration [GPSW17] by adjusting a probabilistic model. We show that models need different adjustments according to the atypicality of inputs and classes, and atypicality is a key factor in recalibration. In light of these findings, we propose a simple method: Atypicality-Aware Recalibration. Our recalibration algorithm takes into account the atypicality of the inputs and classes and is simple to implement. We show that complementing recalibration methods with atypicality improves uncertainty quantification and the accuracy of predictors. Further, in a case study for skin lesion classification, we show that atypicality awareness can improve performance across different skin-tone subgroups without access to group annotations.
3. Improve Prediction sets: An alternative approach to quantify uncertainty is to provide prediction sets that contain the label with high probability [ABMJ20]. Here, we investigate existing methods with atypicality and show that prediction sets could underperform for atypical or low-confidence samples. By using atypicality, we demonstrate the potential for improving prediction sets.
Overall, we propose that models should also consider atypicality, and we show simple- and easy-to-implement atypicality estimators can provide significant value.
2 Interpreting Uncertainty with Atypicality
Motivation: In many machine learning applications, we have access to a model's confidence, which aims to quantify the likelihood that a prediction will be accurate. In classification, model output is a probability distribution over classes and confidence is the predicted probability of the top class, i.e.  $\max_y\hat{\mathbb{P}} (Y = y|X = x)$ . In practical scenarios, confidence is the primary tool used to evaluate the reliability of a prediction where higher confidence is associated with better predictions. However, the uncertainty in confidence can stem from different sources that require different treatment  $\mathrm{[MKvA^{+}21]}$
Here, we call a prediction reliable if it is high-confidence and well-calibrated. High confidence could be reliable or miscalibrated, and low confidence could be due to ambiguity or rare inputs. We propose that atypicality provides a natural way to understand reliability when combined with confidence. A sample is called typical if it is well-represented in the previously observed samples, e.g., an image of a dog that is similar to other dogs in the training data. However, if the image is unlike any other seen during training, it is atypical. We argue that atypicality can help us interpret a prediction's reliability. Below we categorize samples and predictions according to atypicality and confidence in four quadrants (Figure 1).
High-confidence and representative: Reliable predictions often fall within the Reliable Quadrant, which includes typical, high-confidence samples. These samples are well-represented in the training dataset (typical), thus we expect the high-confidence prediction to be reliable. For instance, the first image on the top left (Figure 1) is a typical golden retriever and the model makes a reliable prediction.
High-confidence yet far from the support: Having high-confidence does not always indicate reliability. If the sample does not have support in the training distribution, the confidence could be miscalibrated. Such samples lie in the Extrapolation Quadrant which contains atypical, high-confidence samples. For instance, the second image in the top right of Figure 1 is a toy hog and the model has not seen similar ones during training.
Low confidence due to ambiguity: In contrast, low confidence could also be reliable when it correctly reflects an ambiguity. Such samples are in the Ambiguous Quadrant that contains typical, low-confidence samples. These are typical since they may represent multiple classes; yet, due to ambiguity, the model's confidence is low. For instance, the second image in the bottom left of Figure 1 can both be a hog and a comic book.
Low confidence and rare: For samples that are not well-represented in training data, we expect to have low-quality predictions. Untrustworthy Quadrant comprises atypical, low-confidence samples that can include extremely rare subgroups, for which we expect miscalibration and lower accuracy. For example, the image in Figure 1 bottom right is an origami hog that was not seen in training.
These examples suggest that relying solely on confidence does not provide a complete understanding of the reliability of the predictions, and we can use atypicality to interpret and improve reliability.
Formalizing Atypicality: Atypicality here is defined with respect to the training distribution. Informally, an input or a class is atypical if it is not well-represented in the training distribution. For instance, if there are no or limited similar examples to an input, it can be called atypical. Note that this notion is not restricted to being 'out-of-distribution' [HG16a], since in-distribution groups could also be atypical or rare, and our goal is to perform reliably for the entire spectrum.
Formally, let  $X \in \mathbb{R}^d$  be the random variable denoting features and  $Y \in \mathcal{Y} = \{1, 2, \dots, C\}$  denote the class, where we focus on classification.
Definition 2.1 (Input Atypicality). We define the atypicality of the input  $x$  as<sup>2</sup>
$$
a _ {X} (x) = - \max  _ {y} \log \mathbb {P} (X = x | Y = y).
$$
We use the logarithm of the class-conditional densities due to high dimensionality and density values being close to zero. Intuitively, for a dog image  $x$ , if  $\mathbb{P}(X = x | Y = \mathrm{dog})$  has a low value, we call  $x$  an atypical dog image. Overall, if  $a(x)$  is high, then we call  $x$  an atypical input. Specifically, if an input is not typical for any class, then it is atypical with respect to the training distribution. Similarly, we can also use marginal density,  $\mathbb{P}(X = x)$ , or distance<sup>3</sup> to quantify atypicality.
Similarly, the notion of atypical (rare) classes is prevalent in imbalanced classification  $\left[\mathrm{CWG}^{+}19\right.$ , ZCLJ21]. Ensuring reliable performance for atypical classes can be safety-critical, e.g., for a rare presence of dangerous melanoma  $\left[\mathrm{DVN}^{+}22\right]$ . We define class atypicality in the following:
Definition 2.2 (Class Atypicality). For a class  $y$ , atypicality of a class is defined as
$$
a _ {Y} (y) = - \log \mathbb {P} (Y = y). ^ {4}
$$
2Here atypicality differs from 'typical sets' in information theory that refers to a sequence of variables [TJ06].
For an input  $x$ , if the nearest neighbor (NN) distance is large, then we call  $x$  atypical as all inputs in the training set are far from  $x$ . Density and distance are connected through non-parametric density estimation and [JKGG18] shows that NN distance can recover high-density regions.
4When the meaning is unambiguous, we omit the subscript to denote  $a(X)$  or  $a(Y)$  for notational brevity.
Estimating Atypicality for Discriminative Models: Quantifying input atypicality requires access to the class-conditional / marginal distributions. In practice, for neural networks trained for classification, these distributions are unavailable and we need to perform the estimation. This estimation can be challenging if the dimensionality is large, or the data is unstructured, requiring assumptions about the distributions. Prior works  $\mathrm{[MKvA^{+}21,LLLS18]}$  showed that Gaussian Mixture Models (GMMs) in the embedding space of neural networks can be used to model these distributions.
In experiments, we use Gaussians with shared covariance, i.e.  $\hat{\mathbb{P}}(X = x | Y = c) \sim N(\hat{\mu}_c, \hat{\Sigma})$ , to estimate input atypicality. We perform the estimation in the penultimate layer of neural networks used to make predictions, using maximum-likelihood estimation with samples from the training data. We explore other metrics, such as  $k$ -Nearest Neighbors distance. We give implementation details and results with different metrics in Appendix A.1. With these estimators, atypicality estimation is cheap and can run on a CPU. Our goal is to show that simple estimators can already reap large benefits. Our framework is flexible and exploring more sophisticated estimators is a topic for future work.
Atypicality for LLMs: LLMs are increasingly used for classification  $\left[\mathrm{BMR}^{+}20\right]$ . Modern LLMs are autoregressive models that compute a marginal distribution,  $\hat{\mathbb{P}}_{\mathrm{LLM}}(X)$ . We compute the negative log-likelihood of a prompt or a label and use this as an atypicality metric, i.e.  $a_{X}(x) = -\log \hat{\mathbb{P}}_{\mathrm{LLM}}(x)$ ,  $a_{Y}(y) = -\log \hat{\mathbb{P}}_{\mathrm{LLM}}(y)$ . Similar to the discriminative setting, atypicality here aims to quantify whether a prompt is well-represented in the training data. A larger value for  $a(X)$  would imply that a prompt is more atypical. Below, we present typical and atypical prompts for the AGNews dataset:
Classify the news articles into the categories of World, Sports, Business, and Technology.
Article: Safin tallest obstacle to host #39; s patriotic games hope AS tennis fans go, Houston #39; s Jim #39; Mattress Mack #39; McIngvale is very rich, extremely forthright, exceedingly patriotic and unflinchingly Republican.
Atypicality: 353.45, Percentile: %94.5
Article: Delta Air Lines Prepares Chapter 11 Filing Delta Air Lines Inc. could file for Chapter 11 bankruptcy protection as soon as next week, a source familiar with the matter said yesterday.
Atypicality: 171.50. Percentile:  $\% 0.9$
3 Understanding the Prediction Quality with Atypicality
In this section, we show how our framework can be applied to understand the quality of predictions. Experimental Setup: We investigate three classification settings across a range of datasets:
1. Balanced Supervised Classification: We use ResNet18-50-152 [HZRS16], WideResNet28 [ZK16], RoBERTa  $\left[\mathrm{LOG}^{+}19\right]$  trained on ImageNet  $\left[\mathrm{DDS}^{+}09\right]$ , CIFAR10,100 [Kri09], MNLI [WNB18] respectively.
2. Imbalanced Supervised Classification: We use ResNet18, ResNext50, ResNet152 trained on CIFAR-LT, ImageNet-LT and Places365-LT where models and data are mostly from [ZCLJ21, MKS+20]. Note that all of the test and validation sets have balanced class distributions.
3. Classification with LLMs: We use open-source Alpaca7B [TGZ+23] on IMDB [MDP+11], TREC [LR02], and AG News [ZZL15] datasets with the prompts from [ZWF+21].
Details on datasets, models, and prompts are in Appendix B. Our experiments were run on a single NVIDIA A100-80GB GPU. We report error bars over 10 random calibration/test splits.
3.1 Atypicality is Correlated with Miscalibration
We first explore the importance of atypicality to understand model calibration. Calibration quantifies the quality of a probabilistic model [GR07]. Informally, a model is considered perfectly calibrated if all events that are predicted to occur  $P\%$  of the time occur  $P\%$  of the time for any  $P \in [0,100]$ .
For the sake of simplicity, consider a binary classification problem where the predictor is  $\hat{\mathbb{P}}:\mathcal{X}\to [0,1]$ . We quantify miscalibration with Calibration Error (CE):
$$
\operatorname {C E} [ \hat {\mathbb {P}} ] = \mathbb {E} [ | \mathbb {P} (Y | \hat {\mathbb {P}} (X) = p) - p | ].
$$
It is computationally infeasible to calculate the above expectation with the conditional probability  $\mathbb{P}(Y|\hat{\mathbb{P}}(X) = p)$ . In practice, we use a binned version of this quantity, Expected Calibration Error (ECE) [NCH15, GPSW17], to estimate CE. See Appendix C.1 for a formal definition.
(b)
Figure 2: Atypical Samples Have Low-Quality Predictions. (a) Here, samples are grouped according to the Input Atypicality (x-axis) and Confidence (y-axis), to the right meaning more atypical. Values show the difference between confidence and accuracy, lighter color indicates more overconfidence. Within the same confidence range, atypical groups have more miscalibration and are more overconfident. (b,c,d) Predictions for atypical samples are less accurate and more miscalibrated in balanced and imbalanced supervised classification and classification with LLMs.
Here, we aim to examine the relationship between model calibration and atypicality. Given any  $K > 1$ , we consider the quantiles of  $a(X)$ ,  $a_1, a_2, \ldots, a_{K+1}$  such that  $\mathbb{P}(a(X) \in (a_k, a_{k+1}] = 1 / K$  for  $k \in [K]$ . For imbalanced classification problems, we compute the quantiles using the class atypicality. Specifically, we investigate the atypicality-conditional calibration error  $\mathrm{ECE}[\hat{\mathbb{P}} \mid a(X) \in (a_k, a_{k+1}]]$ , i.e., the expected calibration error of an input that falls within the atypicality quantile  $k$ .
Atypical Examples are Poorly Calibrated: In Figure 2a, we show the distribution of miscalibration where each bin within the grid contains the intersection of the corresponding confidence and atypicality quantiles. We observe that within the same confidence range, predictions for atypical points have lower accuracies and are more overconfident. In other words, predictions in the Extrapolation or Untrustworthy regions are more miscalibrated than the ones in the typical regions.
In Figure 2b, we split inputs into quantiles according to atypicality and compute the ECE and Accuracy for each group. Results show a monotonic relationship between atypicality and ECE or Accuracy across the three settings. Specifically, we see that predictions for atypical inputs or samples from rare classes are more miscalibrated and have lower accuracy. For samples from rare classes, the model overpredicts the probabilities of the typical class, hence we have overconfidence and low accuracy. Appendix C.3, and §4 present figures and tables for all model and dataset pairs.
3.2 Theoretical Analysis: Characterizing Calibration Error with Atypicality
We characterize how calibration error varies with atypicality in a tractable model that is commonly used in machine learning theory [BMWX21a, BMWX21b, ZDKZ22, CLKZ22]. Our theoretical analysis further supports our empirical findings.
Data Generative Model: We consider the well-specified logistic model for binary classification with Gaussian data, where  $Y \in \{-1, 1\}$  and the  $\mathbb{P}(Y = 1|X)$  is defined by the sigmoid function:
$$
\mathbb {P} (Y = 1 \mid X) = \sigma (\langle \beta^ {*}, X \rangle), \quad X \sim N (0, I _ {d}).
$$
Where  $I_{d}$  denotes the  $d$ -dimensional identity matrix,  $\beta^{*}$  is the ground truth coefficient vector,  $\sigma(x) = 1 / (1 + e^{-x})$ , and we have i.i.d. observations  $\{(x_{i},y_{i})\}_{i=1}^{n}$  sampled from the above distribution.
The Estimator: We focus on studying the solution produced by minimizing the logistic loss
$$
\hat {\beta} = \arg \min  _ {\beta} \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \log \left(1 + \exp \left(\beta^ {\top} x _ {i}\right)\right) - y _ {i} \cdot \beta^ {\top} x _ {i} \right].
$$
For  $k \in \{-1, 1\}$ ,  $\hat{\mathbb{P}}_k(x)$  is an estimator of  $\mathbb{P}(y = k|x)$ , with the form  $\hat{\mathbb{P}}_k(x) = \frac{1}{e^{-k \cdot \hat{\beta}^\top x} + 1}$ .
**Calibration:** We consider all  $x$  where  $\mathbb{P}_1(x) > 1/2$ , as  $\mathbb{P}_1(x) \leq 1/2$  can be analyzed similarly by symmetry (see Appendix G). For  $u \in (1/2, 1)$ , the signed calibration error at a confidence level  $u$  is
$$
u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u).
$$
We want to show that when  $X$  is atypical, i.e., when  $a(X) := \|X\|^2/2$  is larger<sup>5</sup>, the accuracy  $\mathbb{P}(Y = 1 \mid \hat{\mathbb{P}}_1(X) = u)$  would be generally smaller than the confidence  $u$  (over-confidence).
Theorem 3.1. Consider the data generative model and the learning setting above. For any  $K > 1$ , suppose we consider the quantiles of  $a(X)$ ,  $a_1, a_2, \ldots, a_K$ ,  $a_{K+1}$  such that  $\mathbb{P}(a(X) \in (a_k, a_{k+1}]) = 1 / K$  for  $k \in [K]$ . We assume  $\| \beta^* \| \leq c_0$ , and  $d / n = \kappa$ , for some sufficiently small  $c_0$ . Then, for sufficiently large  $n$ , for  $k = 2, \ldots, K$ , we have
$$
\mathbb {E} _ {u \sim \hat {\mathbb {P}} _ {1} (X)} [ u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u) \mid a (X) \in \left(a _ {k}, a _ {k + 1} \right\rbrack ] >
$$
$$
\mathbb {E} _ {u \sim \hat {\mathbb {P}} _ {1} (X)} [ u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u) \mid a (X) \in (a _ {k - 1}, a _ {k} ] ] \geq 0.
$$
That is, the resulting classifier is over-confident, and the level of over-confidence becomes larger when the data is more atypical (with larger  $a(X)$ ). Further, the gap becomes larger for smaller sample sizes  $n$ . The proof of the theorem is in Appendix G.2 and builds on the results from [BMWX21a, SC19].
4 Using Atypicality to Improve Recalibration
Here, we show how atypicality can complement and improve post-hoc calibration. In §2, we observed that predictions for atypical inputs and samples from atypical classes are more overconfident with lower accuracy. We next show that taking input and class atypicality into account improves calibration.
4.1 Parametric Recalibration: Different Groups Need Different Temperatures
Temperature scaling (TS), a single parameter variant of Platt Scaling  $\left[\mathrm{P}^{+99}\right]$ , is a simple recalibration method that calibrates the model using a single parameter. The predictor is of the form
$$
\log \hat {\mathbb {P}} _ {\mathrm {T S}} (Y | X) \propto \log \hat {\mathbb {P}} (Y | X) / \tau , \tag {1}
$$
where  $\hat{\mathbb{P}}(Y|X)$  is the model that takes an input and outputs scores/logits, and  $\tau$  is the temperature parameter. In practice,  $\tau$  is optimized using a calibration set to minimize a proper scoring rule [GR07, BW19] such as the cross-entropy loss.
To understand the behavior of TS with respect to atypicality, we separately perform TS on points grouped according to the atypicality quantiles. Let us denote the temperature fitted to the quantile covering  $a(X) \in (a_{k-1}, a_k]$  by  $\tau_{a_k}$ . In Appendix Figure 10 we observe an increasing relationship between  $a_k$  and  $\tau_{a_k}$ . Different atypicality groups need different adjustments, and more atypical groups need larger temperatures. This suggests that being atypicality-aware can improve calibration. While a single temperature value improves average calibration, it may hurt certain groups.
4.2 Atypicality-Aware Recalibration
We showed that predictions are more reliable when the input is typical. However, predictions are less reliable for atypical inputs, and we may need further revision. An analogy can be drawn to decision-making literature where opinions of individuals are combined with geometric averaging weighted by their expertise [FP98, AR89]. Analogously, we propose Atypicality-Aware Recalibration (AAR) a method designed to address the reliability issues identified in dealing with atypical inputs:
$$
\hat {\mathbb {P}} _ {\mathrm {A A R}} (Y | X) = \frac {\hat {\mathbb {P}} (Y | X) ^ {\psi (a (X))} \exp \left(S _ {Y}\right) ^ {1 - \psi (a (X))}}{Z (X)}, \tag {2}
$$
where  $\psi(a(X))$  is a function of input atypicality,  $S_Y$  is a tunable score for class  $Y$ ,  $Z(X)$  is the normalization term. Intuitively, when the input is typical, we trust the model confidence; otherwise, we use a score for the given class estimated from the calibration set. Note that this form simplifies to
$$
\log \hat {\mathbb {P}} _ {\mathrm {A A R}} (Y | X) \propto \phi (a (X)) \log \hat {\mathbb {P}} (Y | X) + S _ {Y}, \tag {3}
$$
The definition of atypicality follows from the marginal likelihood of the data model: density for the Gaussian with zero mean and identity covariance.