SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
= \mathbb {E} _ {u} \left[ \left(u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)\right) \mid a (X) \in \left[ \chi_ {\frac {K + \delta}{K + 1}, p} ^ {2}, 2 p \right] \right] \cdot \mathbb {P} (a (X) \in \left[ \chi_ {\frac {K + \delta}{K + 1}, p} ^ {2}, 2 p \right])
$$
$$
\geq \left(\mathbb {E} _ {u} [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \mid a (X) \in [ a _ {K - 1}, a _ {K} ] ] + C (z _ {\frac {K + \delta}{K + 1}} - z _ {\frac {K}{K + 1}})\right) \cdot (\frac {1}{K} - \frac {\delta}{K + 1} + o (\frac {\delta}{K + 1}))
$$
$$
\begin{array}{l} = \mathbb {E} _ {u} \left[ \left(u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)\right) \cdot \mathbf {1} \left\{a (X) \in \left[ a _ {K - 1}, a _ {K} \right] \right. \right\rbrack \\ + C \big (z _ {\frac {K + \delta}{K + 1}} - z _ {\frac {K}{K + 1}} \big) - (1 + o (1)) \frac {\delta}{K + 1} \cdot \mathbb {E} _ {u} \big [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \mid a (X) \in [ a _ {K - 1}, a _ {K} ] \big ]. \\ \end{array}
$$
The last equality uses the fact that  $\mathbb{P}(a(X) \in [a_{K-1}, a_K]) = 1 / K$ , and therefore
$$
\mathbb {E} _ {u} [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \mid a (X) \in [ a _ {K - 1}, a _ {K} ] \cdot \frac {1}{K} = \mathbb {E} _ {u} [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \cdot \mathbf {1} \{a (X) \in [ a _ {K - 1}, a _ {K} ] \} ]
$$
Then use the fact that  $|\mathbb{E}_u[(u - \mathbb{P}(Y = 1\mid \hat{\mathbb{P}}_1(X) = u))\mid a(X)\in [a_{K - 1},a_K]]| = O(1)$  and we choose  $\delta = o(1 / \log K)$  so
$$
\frac {\delta}{K} = o (| \frac {\log (1 - \delta)}{\sqrt {\log K}} |).
$$
$$
C (z _ {\frac {K + \delta}{K + 1}} - z _ {\frac {K}{K + 1}}) - (1 + o (1)) \frac {\delta}{K + 1} \cdot \mathbb {E} _ {u} [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \mid a (X) \in [ a _ {K - 1}, a _ {K} ] ] > 0,
$$
$$
\mathbb {E} _ {u} [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \cdot \mathbf {1} \{a (X) \in [ \chi_ {\frac {K + \delta}{K + 1}, p} ^ {2}, 2 p ] \} ] \geq \mathbb {E} _ {u} [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \cdot \mathbf {1} \{a (X) \in [ a _ {K - 1}, a _ {K} \} ].
$$
Combining with equation 13, we prove equation 14 and complete the proof.
G.3 Theoretical Justification of the calibration improvement using the atypicality score
In this section, we provide the theoretical justification to understand why incorporating the atypicality score will improve calibration. In particular, we consider the binary classification problem with prediction  $f: X \in [0,1]$  indicating the predicted probability of  $Y = 1$  given  $X = x$ .
For a predictor  $f$ , let us denote its conditional calibration error at an atypicality level  $\gamma$  by  $\mathrm{CE}_{\gamma}(f) = \mathbb{E}[(f(X) - \mathbb{E}[Y|f(X)])^2 |a(X) = \gamma]$ .
Theorem G.2. Consider the same setting as Theorem 3.1. Suppose the temperature function  $\hat{\tau}(a(X)) = \arg \min_{\tau} \mathbb{E}[l(Y, \operatorname{Softmax}(f(X) / \tau(a(X))))]$  with  $l$  being the cross entropy loss, and let  $\hat{\mathbb{P}}_{\mathrm{AAR}}(X) = \operatorname{Softmax}(f(X) / \hat{\tau}(a(X)))$ . Then
$$
\mathrm {C E} _ {\gamma} \left(\hat {\mathbb {P}} _ {\mathrm {A A R}}\right) \leq \min  \left\{\mathrm {C E} _ {\gamma} \left(\hat {\mathbb {P}} _ {\mathrm {T S}}\right), \mathrm {C E} _ {\gamma} (f) \right\}. \tag {15}
$$
Proof: For a prediction function  $f$ , we first define the conditional mean squared error of  $f$  at an atypicality level  $\gamma$  by  $\mathrm{MSE}_{\gamma}(f) = \mathbb{E}[(f(X) - Y)^2 \mid a(X) = \gamma]$ , then we have
$$
\begin{array}{l} \operatorname {M S E} _ {\gamma} (f) - C E _ {\gamma} (f) = \mathbb {E} [ (f (X) - Y) ^ {2} \mid a (X) = \gamma ] - \mathbb {E} [ (f (X) - \mathbb {E} [ Y \mid f (X), a (X) = \gamma ]) ^ {2} \mid a (X) = \gamma ] \\ = \mathbb {E} [ (\mathbb {E} [ Y \mid f (X), a (X) = \gamma ] - Y) \cdot (2 f (X) - \mathbb {E} [ Y \mid f (X), a (X) = \gamma ] - Y) \mid a (X) = \gamma ] \\ = \mathbb {E} [ (\mathbb {E} [ Y \mid f (X), a (X) = \gamma ] - Y) \cdot (\mathbb {E} [ Y \mid f (X), a (X) = \gamma ] - Y) \mid a (X) = \gamma ] \\ + 2 \mathbb {E} [ (\mathbb {E} [ Y \mid f (X), a (X) = \gamma ] - Y) \cdot (f (X) - \mathbb {E} [ Y \mid f (X), a (X) = \gamma ]) \mid a (X) = \gamma ] \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ Y \mathbb {E} [ Y \mid f (X), a (X) = \gamma ] \mid a (X) = \gamma ] \\ = \mathbb {E} _ {f (X) \mid a (X) = \gamma} \mathbb {E} [ Y \mathbb {E} [ Y \mid f (X), a (X) = \gamma ] \mid f (X), a (X) = \gamma ] ] \\ = \mathbb {E} \left[ \left(\mathbb {E} [ Y \mid f (X), a (X) = \gamma ]\right) ^ {2} \mid a (X) = \gamma \right], \\ \end{array}
$$
$$
\mathbb {E} [ (\mathbb {E} [ Y \mid f (X), a (X) = \gamma ] - Y) \cdot (f (X) - \mathbb {E} [ Y \mid f (X), a (X) = \gamma ]) \mid a (X) = \gamma ] = 0,
$$
$$
\operatorname {M S E} _ {\gamma} (f) - C E _ {\gamma} (f) = \mathbb {E} \left[ \left(\mathbb {E} [ Y \mid f (X), a (X) = \gamma ] - Y\right) ^ {2} \mid a (X) = \gamma \right]
$$
Now that  $\hat{\mathbb{P}}_{AAR}(f(x),a(x))$  is monotonic on the  $\hat{\mathbb{P}} (x)$  , we have
$$
\mathbb {E} [ Y \mid f (x), a (X) = \gamma ] = \mathbb {E} [ Y \mid \tilde {\mathbb {P}} _ {A A R} (f (x), a (X)), a (X) = \gamma ],
$$
$$
\mathrm {M S E} _ {\gamma} \left(\hat {\mathbb {P}} _ {A A R}\right) - C E _ {\gamma} \left(\hat {\mathbb {P}} _ {A A R}\right) = \mathrm {M S E} _ {\gamma} (\hat {\mathbb {P}}) - C E _ {\gamma} (\hat {\mathbb {P}}). \tag {16}
$$
$$
\mathbf {M S E} _ {\gamma} \left(\hat {\mathbb {P}} _ {T S}\right) - C E _ {\gamma} \left(\hat {\mathbb {P}} _ {T S}\right) = \mathbf {M S E} _ {\gamma} \left(\hat {\mathbb {P}}\right) - C E _ {\gamma} \left(\hat {\mathbb {P}}\right). \tag {17}
$$
In the following, we will show that
$$
\mathrm {M S E} _ {\gamma} \left(\hat {\mathbb {P}} _ {A A R}\right) <   \min  \left\{\mathrm {M S E} _ {\gamma} \left(\hat {\mathbb {P}} _ {T S}\right), \mathrm {M S E} _ {\gamma} \left(\hat {\mathbb {P}}\right) \right\}. \tag {18}
$$
First, as we consider the binary classification setting, with  $l$  being the cross-entropy loss, we have
$$
l (Y, \operatorname {S o f t m a x} (f (X) / \tau (a (X)))) = Y \log (\sigma (f (X) / \tau (a (X))) + (1 - Y) \log (1 - \sigma (f (X) / \tau (a (X))),
$$
where  $\sigma (x) = 1 / (1 + e^{x})$
Then, by the definition of  $\hat{\tau}(a(X))$ , we have that
$$
\begin{array}{l} \hat {\tau} (a (X)) = \underset {\tau} {\arg \min } \mathbb {E} [ Y \log (\sigma (f (X) / \tau (a (X))) + (1 - Y) \log (1 - \sigma (f (X) / \tau (a (X)))) ] \\ = \underset {\tau} {\arg \min } \mathbb {E} [ \mathbb {E} [ Y \log (\sigma (f (X) / \tau (a (X))) + (1 - Y) \log (1 - \sigma (f (X) / \tau (a (X))) \mid a (X) ] ]. \\ \end{array}
$$
Taking the derivative on the last line and setting it to zero, we have
$$
\mathbb {E} \left[ \frac {Y}{\sigma (f (X) / \hat {\tau} (a (X))} - \frac {1 - Y}{1 - \sigma (f (X) / \hat {\tau} (a (X))} \mid a (X) \right] = 0,
$$
$$
\mathbb {E} [ \sigma (f (X) / \hat {\tau} (a (X)) \mid a (X) ] = \mathbb {E} [ Y \mid a (X) ].
$$
This makes the derivative of  $\mathbb{E}[(Y - \sigma (f(X) / \tau (a(X))))^2\mid a(X)]$  zero and therefore  $\hat{\tau} (a(X))$  is also a minimizer of  $\mathbb{E}[(Y - \sigma (f(X) / \tau (a(X))))^2\mid a(X)]$ :
$$
\hat {\tau} (a (X)) = \underset {\tau} {\arg \min } \mathbb {E} [ Y \log (\sigma (f (X) / \tau (a (X))) + (1 - Y) \log (1 - \sigma (f (X) / \tau (a (X)))) ] = \underset {\tau} {\arg \min } \mathbb {E} [ (Y - \sigma (f (X) / \tau (a (X)))) ^ {2}
$$
Letting  $g(\gamma) = \arg \min_{c}\mathbb{E}[(Y - \sigma (f(X) / c))^{2}\mid a(X) = \gamma ]$ , we have that
$$
g (a (X)) = \underset {\tau} {\arg \min} \mathbb {E} [ (Y - \sigma (f (X) / \tau (a (X)))) ^ {2} \mid a (X) ],
$$
$$
g (a (X)) = \underset {\tau} {\arg \min} \mathbb {E} [ (\boldsymbol {Y} - \sigma (f (X) / \tau (a (X)))) ^ {2} | a (X) ] = \hat {\tau} (a (X)).
$$
$$
\begin{array}{l} \mathrm {M S E} _ {\gamma} \left(\hat {\mathbb {P}} _ {A A R}\right) = \mathbb {E} \left[ \left(\hat {\mathbb {P}} _ {A A R} (X) - Y\right) ^ {2} \mid a (X) = \gamma \right] \\ = \mathbb {E} \left[ \left(\hat {\mathbb {P}} _ {A A R} (X) - Y\right) ^ {2} \mid a (X) = \gamma \right] \\ = \mathbb {E} \left[ \left(\sigma (f (X) / \hat {\tau} (a (X)) - Y\right) ^ {2} \mid a (X) = \gamma \right] \\ = \mathbb {E} \left[ \left(\operatorname {S o f t m a x} (\hat {\mathbb {P}} (X) / g (a (X)) - Y\right) ^ {2} \mid a (X) = \gamma \right] \\ = \underset {c} {\arg \min } \mathbb {E} [ (\sigma (f (X) / c - Y) ^ {2} \mid a (X) = \gamma ] \\ \leq \mathbb {E} [ (\sigma (f (X) - Y) ^ {2} \mid a (X) = \gamma ] \\ = \mathrm {M S E} _ {\gamma} (\hat {\mathrm {P}}). \\ \end{array}
$$
Similarly, we have  $\mathrm{MSE}_{\gamma}(\hat{\mathbb{P}}_{AAR}) \leq \mathrm{MSE}_{\gamma}(\hat{\mathbb{P}}_{TS})$ , and therefore equation 18 holds.
Combining with equation 16 and equation 17, we have
$$
C E _ {\gamma} (\hat {\mathbb {P}} _ {A A R}) \leq \min  \{C E _ {\gamma} (\hat {\mathbb {P}} _ {T S}), C E _ {\gamma} (\hat {\mathbb {P}}) \}.
$$
Figure 13: Post-hoc Recalibration for Classification with 5-NN distance as an Atypicality Metric. (a) Balanced Supervised Classification: Atypicality-Aware Recalibration improves the calibration of models trained with balanced datasets, across atypicality groups. (b) Imbalanced Supervised Classification: Atypicality-Aware Recalibration improves both the calibration across groups and the overall accuracy of models trained with imbalanced datasets with 5-nearest neighbors distance as an atypicality metric.
H.1 Quantifying Atypicality
Since we do not have access to the true distribution of  $\mathbb{P}(X)$ , we estimate it through the model, e.g. using the embeddings. This means we are capturing the atypicality not solely with respect to the training distribution but also the model. It is possible that a model that does not fit the data well and produces low-quality atypicality estimates. We would like to stress that our goal here is to show that even simple estimators can demonstrate significant benefits. In general, we observe that our findings hold for large datasets and widely used models, and atypicality gives a semantically meaningful way to group data points qualitatively. Our findings suggest that we can unify the understanding and improve uncertainty quantification and recalibration methods with atypicality, however, practitioners should be careful about incorporating atypicality, as poor atypicality estimates can lead to worse performance.
H.2 Subgroup Fairness Experiments
While the literature on algorithms to satisfy group fairness is rich [KGZ19, HJKRR18], here we wanted to give a case study with skin-lesion classification. Our goal was to provide further evidence that atypicality awareness could improve fairness algorithms. In this domain, there is more verification to do to better characterize how and when atypicality helps, thus, to better understand which subgroups could benefit more from atypicality-aware algorithms and whether these findings apply generally in the literature.
H.3 Theoretical Analysis
Following the earlier work [BMWX21a, SC19], we analyzed the calibration behavior of well-specified logistic regression. However, our empirical findings suggest that the phenomena are much more broadly applicable. We suggest that future work can analyze the behavior in more general settings to better understand the dynamics.
Distribution of Input Atypicality
Figure 14: Distribution of Input Atypicality. Here, we give the distribution of Atypicality for ResNet18 on ImageNet, using GMM and KNN methods.
Figure 15: Label Atypicality and Confidence. Here, the x-axis reflects the label atypicality quantile, and the y-axis indicates confidence. The coloring for the figure on the left indicates the accuracy within a bin, and the figure on the right has the difference between confidence and accuracy within a bin. Similar to Figure 6, we observe that atypical examples have lower accuracy, and predictions are more overconfident.
Figure 16: Input Atypicality and Confidence for ResNet152. We provide the input atypicality for ResNet152, in the same structure as Figure 15.