SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: Post-hoc Recalibration for Classification. (a) Balanced Supervised Classification: Atypicality-Aware Recalibration improves the calibration of models trained with balanced datasets, across atypicality groups. (b) Imbalanced Supervised Classification: Atypicality-Aware Recalibration improves both the calibration across groups and the overall accuracy of models trained with imbalanced datasets. (c) Classification with LLMs: Atypicality-Aware Recalibration improves both the calibration across groups and the overall accuracy of LLMs performing classification.
where we subsume  $(1 - \psi(a(X))$  into  $\phi(a(X))$ . We give a simple interpretation of this form: the multiplicative term is an atypicality-dependent temperature, and the additive term is a class-dependent correction where  $\exp(S_Y)$  can be considered to induce a correction distribution over classes estimated from the calibration set.
Intuitively, when  $\psi(a(X)) = 0$ , the output reduces to a fixed distribution over classes that was estimated using the calibration set. This distribution can be seen to induce a prior probability over classes, and  $\psi$  controls the tradeoff between this prior and the model's predictive distribution. As the point becomes more typical, this distribution is closer to the model's predictive distribution. In Appendix Figure 11, we show how these values behave with class atypicality. We find that rare classes require larger positive corrections with larger  $S_{Y}$ .
Implementation Details: Following TS, we minimize the cross-entropy loss on a calibration set. With the temperature-atypicality relationship observed in Figure 10 we choose to instantiate the multiplicative factor as a quadratic function, where  $\phi(a(X)) = c_2a(X)^2 + c_1a(X) + c_0$  and in total we have  $|\{S_1,.., S_{|\mathcal{Y}|}, c_0, c_1, c_2\}| = |\mathcal{Y}| + 3$  interpretable parameters. Once the embeddings and logits are computed, AAR runs on a CPU in under 1 minute for all experimented settings.
Similar to our adaptive interpretation, a concurrent work, Adaptive Temperature Scaling (AdaTS)  $\left[\mathrm{JPL}^{+}23\right]$ , uses temperature scaling where the temperature is parameterized by a Variational Autoencoder(VAE) [KW13] and a multi-layer perceptron on top of the VAE embeddings. In the below experiments, we give results with AdaTS as a baseline when applicable.
Figure 4: Improving Group Performance through Atypicality-Awareness. Here we show that AAR improves the calibration and accuracy of models across different skin tone groups. With AAR, we can improve both the worst group performance and overall performance significantly without using group attributes. TS curve is less visible since it significantly overlaps with Skin Tone Conditional.
For Balanced Supervised Classification, in Figure 3a, we observe that being atypicality aware improves recalibration across all groups. We perform comparably to AdaTS, where the temperature function has in the order of millions of parameters, whereas AAR has only  $|\mathcal{Y}| + 3$  parameters.
In Imbalanced Supervised Classification (Figure 3b), our algorithm not only provides better calibration rates across all classes but also improves overall accuracy. Note that only our method can change accuracy (due to the additive term), and it performs better than other baselines in terms of ECE across all classes. Further, the second column shows using Progressive Balancing  $\left[\mathrm{ZWF}^{+}21\right]$  in training, showing that our post-hoc method can complement methods that modify training procedures.
For Classification with LLMs, we add an LLM calibration baseline Content-Free Calibration (CF)  $\left[\mathrm{ZWF}^{+}21\right]$ . We cannot use AdaTS as the embeddings are not fixed in size. In Figure 3c, we see AAR has better calibration and accuracy across the three datasets. Namely, by adjusting the LLM output using the LLM atypicality, we can adjust the probabilities to increase the prediction quality.
4.3 Case Study: Fairness through Atypicality-Awareness
Machine learning models reportedly have performance disparity across subgroups [BHN17] due to factors such as varying sample size or noise levels [CJS18]. For instance, skin lesion classifiers can exhibit performance disparity across different skin tones [ \mathrm{DVN}^{+}22 ]. Fitzpatrick  $17\mathrm{k}$  [ \mathrm{GHS}^{+}21 ] is a dataset of clinical images with Fitzpatrick skin tone annotations between 1-to-6, where a larger number means darker skin tones, and when annotators do not agree, it is labeled as 'Unknown'. We explore the classification problem with 9 classes indicating the malignancy and the type of skin condition, using a ResNet18/34 pretrained on ImageNet and finetuned on this task (See Appendix F).
When the goal is to improve performance across groups, one can use group annotations and optimize performance within each group [HJKRR18, KGZ19]. Here, we investigate how complementing recalibration with atypicality can improve prediction quality across all groups without group annotations. For comparison, we perform 3 recalibration methods: TS, AAR, and Skin-Tone Conditional TS which calibrates the model individually for each skin-tone group with TS. Since the skin-tone conditional calibration uses group attributes, ideally it should act as an oracle.
In Figure 4, we give the Accuracy and ECE analyses where AAR improves performance across all groups. For instance, the worst-group Accuracy (0.69) or ECE (0.072) with AAR is close to the best-group Accuracy (0.63) or ECE (0.062) with the other two methods. Overall, our findings suggest that Atypicality-Awareness can complement fairness-enforcing methods, and improve performance even when the group annotations are unavailable. We hypothesize that with AAR, we can perform better than using supervised group attributes since groups may not have sufficient sample size in the calibration set (131, 1950, 1509, 555 samples for Unknown, 1&2, 3&4, and 5&6 respectively), and we can leverage atypicality to offer some mitigation. Further investigating how to leverage atypicality to improve fairness and factors affecting performance disparities is a promising direction for future work [CJS18].
5 Improving Prediction Sets with Atypicality
Conformal Prediction [SV08, AB21] is a framework that assigns a calibrated prediction set to each instance. The goal is to find a function  $\mathcal{C}:\mathcal{X}\to 2^{\mathcal{V}}$  that returns a subset of the label space such that  $Y\in \mathcal{C}(X)$  with high probability. The framework aims to guarantee marginal coverage, i.e.,  $\mathbb{P}(Y\in \mathcal{C}(X))\geq 1 - \alpha$ , for a choice of  $\alpha$ . We investigate two conformal calibration methods,
Adaptive Prediction Sets (APS) [RSC20] and Regularized APS (RAPS) [ABMJ20]. Let  $\pi(X)$  be the permutation of the label set that sorts  $\hat{\mathbb{P}}(Y = c|X)$ , i.e. the predicted probabilities for each class  $c$  after TS. The prediction sets are produced by the function  $\mathcal{C}(x) = \{y : s(x, y) \leq \hat{q}\}$ , and these methods fit the threshold  $\hat{q}$  for a choice of the scoring function. APS uses the cumulative sum of the predicted probabilities  $s(x, y) = \sum_{j=1}^{c} \hat{\mathbb{P}}(Y = j|X)$ , where  $y = \pi_c(X)$ . Intuitively, if the model was perfectly calibrated, we would have expected to have  $\hat{q} = 1 - \alpha$ . Similarly, RAPS builds on the idea that tail probabilities are noisy and regularizes the number of samples in the prediction set.
Building on our ideas in the previous sections we implement Atypicality-Aware prediction sets, namely AA-APS and AA-RAPS in the following way: We first group points according to their confidence and atypicality quantiles. A group  $G$  here is defined using 4 thresholds, namely  $G = x : (l_a^{(G)} < q_a(x) \leq h_a^{(G)}) \land (l_c^{(G)} < q_c(x) \leq h_c^{(G)})$  where  $q_a(x)$  and  $q_c(x)$  denote the atypicality and confidence quantiles for the sample  $x$ ,  $l_a^{(G)}$  and  $h_a^{(G)}$  denote the atypicality lower and upper bounds for group  $G$ , and  $l_c^{(G)}$  and  $h_c^{(G)}$  denote the confidence lower and upper bounds for group  $G$ . Using a calibration set, these bounds are simply determined by the quantiles of confidence and atypicality statistics. Then, we fit separate thresholds  $\hat{q}_G$  for each group's prediction sets with APS or RAPS as subroutines. This allows us to have an adaptive threshold depending only on the atypicality and confidence of predictions.
Figure 5: Improving Conformal Calibration with Atypicality for ResNet50 on ImageNet. Here we show that atypicality awareness improves conformal calibration performance across different groups. Methods are fitted to satisfy  $95\%$  coverage. We observe that APS and RAPS do not satisfy conditional coverage for high atypicality regions or low confidence regions.
In Figure 5, we provide the coverage plots for APS and RAPS in the first and third columns. Even though marginal coverage is satisfied, models do not satisfy conditional coverage for atypical inputs or low-confidence predictions. We observe that being Atypicality-Aware improves coverage across otherwise underperforming groups. Further, AA-APS has lower set sizes on average than APS (15.6 vs 21.3). While RAPS has a lower average set size than AA-RAPS (4.2 vs 9.1) AA-RAPS has smaller set sizes for high-confidence samples, whereas a larger set size for low-confidence samples where the coverage is not met for RAPS. In Appendix D.3, we provide the same analysis for ResNet18,50,152 at different coverage levels along with analyzing the performance in the Confidence and Atypicality dimensions individually. For instance in Figure 8, we observe that RAPS and APS do not satisfy coverage for high atypicality regions, even when averaged across different confidence levels.
6 Additional Related Work
Uncertainty and Atypicality:  $\mathrm{[MKvA^{+}21, PBC^{+}20]}$  use density estimation to disentangle epistemic and aleatoric uncertainty. Following this, they show improvements in active learning and OOD detection [LLLS18]. We note that our goal is not this disentanglement (e.g. Untrustworthy quadrant can have both aleatoric or epistemic uncertainty), or Ambiguity could be due to a lack of features or noise.  $\mathrm{[LLP^{+}20]}$  propose the related notion of distance awareness, and that it leads to better uncertainty quantification. They offer architecture and training modifications whereas we analyze existing models using our framework including imbalanced and LLM settings, and propose simple and post-hoc approaches. 'OOD' [HG16b] or 'anomaly' [HMD18] notions are tied to atypicality, yet our goal is not to make a binary distinction between 'in' or 'out'. We argue that in-distribution samples could also be atypical (e.g. rare groups), and the goal is to perform reliably in the entire spectrum. Other works with an atypicality notion include bounding calibration of groups by the
excess risk [LSH19], miscalibration under distribution shifts  $\left[\mathrm{OFR}^{+}19\right]$ , uncertainty in Gaussian Processes [Ras04], forgetting time for rare examples [MGLK22], the poor performance of groups with lower sample sizes [CJS18], energy-based models improving calibration  $\left[\mathrm{GWJ}^{+}20\right]$ , relating perplexity to zero-shot classification performance for LLMs  $\left[\mathrm{GIB}^{+}22\right]$ , grouping loss and local definitions of miscalibration [PLMV23], the relationship between active learning and atypicality [HDW22], sample size as a factor for subgroup performance disparity [CJS18]. [PBZ21] provide insightful discussion around the nature of softmax confidence, and here we show that its reliability depends on the atypicality of the input. Our new findings include showing that predictions for atypical samples are more miscalibrated and overconfident, and atypicality awareness improves prediction quality. Overall, while there are other relevant notions in the literature, our distinct goal is to show that post-hoc atypicality estimation and recalibration is a simple yet useful framework to understand and improve uncertainty quantification that complements existing methods.
Recalibration and Conformal Prediction: There is a rich literature on recalibration methods and prediction sets: TS [GPSW17], Platt Scaling  $\left[\mathrm{P}^{+99}\right]$ , conformal calibration [SV08, ABMJ20] among many.  $\left[\mathrm{LLC}^{+22},\mathrm{RBSC}19,\mathrm{BYR}^{+21},\mathrm{BGJ}^{+22}\right]$  make a relevant observation, showing that the coverage of conformal prediction is not equal across all groups. They propose group conformal calibration, which requires group labels whereas our proposal is unsupervised and does not depend on any attribute information. Concurrent work  $\left[\mathrm{JPL}^{+23}\right]$  explores AdaTS, where they train a separate VAE and MLP to produce an adaptive temperature. However, our parameterization of temperature has 3 parameters and is interpretable.
Atypicality offers a simple yet flexible framework to better understand and improve model reliability and uncertainty. We propose that pretrained models should be released not only with confidence but also with an atypicality estimator. While there are other relevant notions in the literature, our main goal is to show that atypicality can provide a unifying perspective to discuss uncertainty, understand individual data points, and improve fairness. Here we focus on classification problems; it would be interesting to extend atypicality to regression and generation settings. Furthermore, we would like to extend the theoretical analysis to more general settings, as our empirical results demonstrate that the observed phenomena hold more broadly.
We would like to thank Adarsh Jeewajee, Bryan He, Edward Chen, Federico Bianchi, Kyle Swanson, Natalie Dullerud, Ransalu Senanayake, Sabri Eyuboglu, Shirley Wu, Weixin Liang, Xuechen Li, Yongchan Kwon, Yu Sun, and Zach Izzo for their comments and suggestions on the manuscript. Linjun Zhang's research is partially supported by NSF DMS-2015378. Carlos Ernesto Guestrin is a Chan Zuckerberg Biohub - San Francisco Investigator.
[AB21] Anastasios N, Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021. 8, 23
[ABMJ20] Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14193, 2020. 2, 9, 10, 23, 24
[AR89] János Aczél and Fred S Roberts. On the possible merging functions. Mathematical Social Sciences, 17(3):205-243, 1989. 6
$\left[\mathrm{BGJ}^{+}22\right]$  Osbert Bastani, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. Practical adversarial multivalid conformal prediction. arXiv preprint arXiv:2206.01067, 2022. 10
[BHN17] Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness in machine learning. Nips tutorial, 1:2, 2017. 8
[WNB18] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, 2018. 4, 17
[ZCLJ21] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16489-16498, 2021. 3, 4, 18
[ZDKZ22] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. In International Conference on Machine Learning, pages 26135-26160. PMLR, 2022. 5
[ZK16] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016. 4
$\left[\mathrm{ZKL}^{+}16\right]$  Bolei Zhou, Aditya Khosla, Agata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding. arXiv preprint arXiv:1610.02055, 2016. 18
$\left[\mathrm{ZWF}^{+}21\right]$  Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR, 2021. 4, 8, 18, 19
[ZZL15] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015. 4
A Atypicality Estimation
A.1 Input Atypicality Estimation
To estimate input atypicality, we use two ways to estimate the likelihood of a point under the training distribution. First, we give methods for the discriminative models.
Fitting individual Gaussians to class conditionals Here, we follow a similar approach to  $\mathrm{[MKvA^{+}21]}$ . Namely, we model the class conditionals with a Gaussian distribution, where the covariance matrix is tied across classes:
$$
\hat {\mathbb {P}} (X | Y = y) \sim N (X; \mu_ {y}, \Sigma) \tag {4}
$$
We fit the parameters  $\mu_y$  and  $\Sigma$  with maximum likelihood estimation. The reason to tie the covariance matrix is due to the number of samples required to fit the density. Namely, for a  $d$ -dimensional problem, the total number of parameters to fit individual matrices becomes  $O(yd^2)$ , which results in low-quality estimates. Then, the atypicality becomes
$$
a _ {X} (x) = - \max  _ {y \in \mathcal {Y}} \log \hat {\mathbb {P}} (X = x | Y = y) \tag {5}
$$
Computing distance with k-Nearest Neighbors k-Nearest Neighbors: Similarly, we can use the nearest neighbor distance. Concretely, we use the nearest neighbor distance,  $a_{X}(x) = d_{\min}(x,\mathcal{D}_{\mathrm{train}}) = \min_{x^{\prime}\in \mathcal{D}_{\mathrm{train}}}|x^{\prime} - x|$ , as the atypicality metric. Alternatively, we can use different notions such as the average of k-nearest neighbors, or the distance to the kth neighbor. Below, we report the results by using the average distance to 5-nearest neighbors.
Fitting the estimators For all of the atypicality estimators, we fit the estimators using samples from the training sets and make inferences for the calibration and test sets. For instance, we use the training split of ImageNet to fit the corresponding density estimator and compute the atypicality for the samples from the validation/test split of ImageNet. All of our results using atypicality are reported for the test splits of the below datasets.
Atypicality Estimation with LLMs: For language models we simply compute the negative log-likelihood of each prompt as the atypicality metric:  $a_{X}(x) = -\log \hat{\mathbb{P}}_{\mathrm{LLM}}(x)$ . To define confidence, we use the logits of the language model, conditioned on the prompt. We use the logit of the first token of each class label and compute the predicted probabilities by applying softmax to the logits of each class.
A.2 Class Atypicality
To estimate class atypicality, we simply count the fraction of examples from a particular label in the training dataset. Let us have a training dataset  $\mathcal{D}_{\mathrm{train}} = \{(x_1,y_1),(x_2,y_2),\ldots ,(x_N,y_N)\}$ . Then, we estimate class atypicality with
$$
a _ {Y} (y) = - \log \frac {\sum_ {i \in [ N ]} \mathbf {1} \left[ y _ {i} = y \right]}{N}. \tag {6}
$$
A.3 Atypicality and Confidence
Are atypicality and confidence equally informative? Beyond the data perspective given in Figure 1, here we provide quantitative results to demonstrate the difference. In Figure 6, we give a grid plot where the x-axis indicates the typicality quantile of a point, and the y-axis indicates the confidence of a point. The coloring on the left indicates the accuracy within a bin split according to accuracy, and the right has the difference between average confidence and accuracy. Observe that for a specific confidence interval, larger values of typicality mean better quality probabilistic estimates and larger atypicality means more miscalibration.
Figure 6: Input Atypicality and Confidence. Here, the x-axis reflects the input atypicality quantile, and the y-axis indicates confidence. The coloring for the figure on the left indicates the accuracy within a bin, and the figure on the right has the difference between confidence and accuracy within a bin. We observe that even within the same confidence range, atypical examples tend to be more miscalibrated and overconfident compared to typical examples.
B.1 Balanced Supervised Classification
B.1.1 Datasets
Below is a full list of datasets for balanced classification:
1. ImageNet  $\left[\mathrm{DDS}^{+}09\right]$  from Torchvision [MR10] is an object recognition dataset with 1000 classes. We use the ImageNet-1k version.
2. CIFAR10/100 [Kri09] from Torchvision [MR10] are object recognition datasets with 10/100 classes.
3. MNLI [WNB18] from Huggingface Datasets  $\left[\mathrm{LVdMJ}^{+}21\right]$  is a natural language inference dataset with 3 classes, indicating entailment, neutral, and contradiction outcomes.
B.1.2 Models
Most of the models are public models, e.g., obtained from the Transformers Library  $\left[\mathrm{WDS}^{+}20\right]$  or Torchvision [MR10]. Below we give the full model details and how one can access them:
1. RoBERTa(HuggingFace roberta-large-mnli) trained on the MNLI dataset. One can use the id given here on HuggingFace to download the model.
2. ResNet18, ResNet50, ResNet152 from (Torchvision [MR10]) trained on ImageNet.
3. WideResNet28 trained on CIFAR10,100 obtained from  $\mathrm{[MKS^{+}20]}$
For all BERT [DCLT19] style models we use the [CLS] token embeddings in the final layer to perform classification. For all vision models, we use the penultimate layer embeddings to fit the density estimators and perform the analyses. In the experiments, we randomly split the test sets into two equal halves to have a calibration split and a test split, and repeat the experiments over 10 random seeds.
B.2 Imbalanced Supervised Classification
B.2.1 Datasets
All of our imbalanced classification datasets are previous benchmarks obtained from the GitHub repository<sup>6</sup> of [ZCLJ21] with corresponding training, validation, and test splits. All of these datasets have an exponential class imbalance.
1. ImageNet-LT is the long-tailed variant of ImageNet with 1000 classes.
2. CIFAR10/100-LT is the long-tailed variant of CIFAR10/100 with 10/100 classes.
3. Places365-LT is the long-tailed variant of Places365  $\left[\mathrm{ZKL}^{+}16\right]$  with 365 classes.
B.2.2 Models
Similarly, most of these models are obtained from [ZCLJ21].
1. ResNeXt50 trained on ImageNet-LT with and without Progressive Balancing, which is a strategy to address class imbalance during training.
2. ResNet152 trained on Places365-LT
3. ResNet18 trained on CIFAR100-LT trained by us. This model is pretrained on ImageNet and finetuned on CIFAR100-LT.
We use the validation splits of these datasets as the calibration set, and report the results on the test set.
B.3 Classification with LLMs
B.3.1 Model
We use Alpaca-7B [TGZ+23] in a zero-shot setting, where we simply prompt the model with the classification question. We use the prompting strategy from Content-Free Calibration [ZWF+21].
Below, we show examples of each dataset and prompt.
B.3.2 Datasets
IMDB is a binary classification dataset of movie reviews, where the goal is to classify the sentiment in a review. The example prompt has the form 'The following review was written for a movie: [Review].\nWhat is the sentiment, Positive or Negative? Answer:'. Below is an example:
The following review was written for a movie: I and a friend rented this movie. We both found the movie soundtrack and production techniques to be lagging. The movie's plot appeared to drag on throughout with little surprise in the ending. We both agreed that the movie could have been compressed into roughly an hour giving it more suspense and moving plot.
What is the sentiment, Positive or Negative? Answer: