SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the correct answer should be 'Negative'. We filter out the examples that exceed the context length limit of Alpaca7B (512). We noticed that the 'validation' split of IMDB leads to significantly worse calibration compared to splitting the test set. Thus, for all experiments, we use the test split of IMDB and split it into 2 sets (instead of using the validation split as a calibration set as in the other two datasets).
TREC is a 6-class question classification dataset where the goal is to predict whether a question will have an answer that is an 'Abbreviation', 'Entity', 'Description', 'Human', 'Location', or a 'Number'. We format the prompts with 'Classify the questions based on their Answer Type. Potential Answer Types are: Number, Location, Person, Description, Entity, or Abbreviation.\n\nQuestion:[question]\n\nAnswer Type:'. Below is an example prompt:
$^{6}$ https://github.com/dvlab-research/MiSLAS
Classify the questions based on their Answer Type. Potential Answer Types are: Number, Location, Person, Description, Entity, or Abbreviation.
Question: What county is Modesto, California in ?
Answer Type:
where the correct answer should be 'Location'.
AG News is a news classification dataset. The goal is to classify a given news into 4 potential classes: 'World', 'Sports', 'Business', or 'Science and Technology'. We format the prompts with 'Classify the news articles into the categories of World, Sports, Business, and Technology.\n\nArticle: [article]\n\nBelow is an example prompt:
Article: Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling band of ultra-cynics, are seeing green again.
where the correct answer should be 'Business'.
Furthermore, we also use  $\mathrm{[ZWF^{+}21]}$  as another calibration baseline. Following their paper, we use N/A, [MASK], and the empty string as content-free. Concretely, we follow their paper to first obtain the average predicted probabilities for each label token for the content-free input, denoted by  $p_{cf}$ . We then let
$$
W = \operatorname {d i a g} \left(p _ {c f}\right) ^ {- 1}.
$$
When making test-time predictions, we compute  $\mathrm{Softmax}(W^T p)$  as the new predicted probabilities. In our experiments, we observe that it does not perform as well in this setting, as was previously suggested by [Kum22].
C Calibration
We run all our experiments with 10 different random seeds, where the seeds are  $\{0,1,2,\dots ,9\}$ . Randomness is over fitting the atypicality estimators, and calibration-test splits (we use the same splits with the recalibration experiments for the sake of consistency).
C.1 Expected Calibration Error
To compute ECE, we generate  $\mathbb{B} = \{B_1, B_2, \dots, B_M\}$ ,  $M$  equally-spaced bins where samples are sorted and grouped according to their confidence.  $B_m$  here denotes the set of the data indices where the confidence of the prediction for the sample falls into the interval  $\left(\frac{m-1}{M}, \frac{m}{M}\right]$ . We compute ECEby
$$
\operatorname {E C E} [ \hat {\mathbb {P}} ] = \sum_ {m = 1} ^ {M} \frac {\left| B _ {m} \right|}{N} \left| \operatorname {a c c} \left(B _ {m}\right) - \operatorname {c o n f} \left(B _ {m}\right) \right|, \tag {7}
$$
where  $\mathrm{acc}(B_m) = \frac{1}{|B_m|}\sum_{i=1}^{|B_m|}\mathbf{1}[\hat{y}_i = y_i]$  is the accuracy for the bin  $m$ , and  $\mathrm{conf}(B_m) = \frac{1}{|B_m|}\sum_{i=1}^{|B_m|}\hat{\mathbb{P}}(Y = \hat{y}_i | X = x_i)$  gives the average confidence within the bin.  $|B_m|$  is the size of the bin  $m$ ,  $N$  is the total number of samples, and  $\mathbf{1}[\cdot]$  is the indicator function.
Throughout our experiments, we let the number of bins  $|\mathbb{B}| = 10$  by default when computing ECE.
Similarly, below we report results with RMSCE (Root Mean Squared Error) [HMD18] as another calibration metric, which is formulated as the following:
$$
\operatorname {R M S C E} [ \hat {\mathbb {P}} ] = \sqrt {\sum_ {m = 1} ^ {M} \frac {\left| B _ {m} \right|}{N} \left(\operatorname {a c c} \left(B _ {m}\right) - \operatorname {c o n f} \left(B _ {m}\right)\right) ^ {2}} \tag {8}
$$
C.2 ECE and Atypicality Results with Different Atypicality Metrics
We further experiment with different atypicality metrics, such as the average distance to the 5-nearest neighbors (Figure 7). We broadly observe that while there are slight differences in the quantitative results between different atypicality metrics, the qualitative phenomena remain intact. In Tables 1, 2, and 3 we give all the results in the tabular form.
Figure 7: Atypicality with 5-nearest neighbors and Uncertainty. Here, we report the results of the same experiments as Figure 3 with the average of the distance to the 10-nearest neighbors as the atypicality metric. See Tables 1 for the results in tabular format.
C.3 Results in the Tabular Format
Table 3: Recalibration Results for LLM Classification. For each dataset and atypicality quantile, the best results are marked in bold. We provide the standard errors next to the means over 10 random seeds.
ECE	Uncalibrated RMSE	Accuracy	ECE	Content-Free RMSE	Accuracy	ECE	Atypicality-Aware RMSE	Accuracy
LLM	Alpaca7B	AG News	0.25	0.180 ± 0.000	0.219 ± 0.000	0.681 ± 0.000	0.452 ± 0.000	0.411 ± 0.000	0.527 ± 0.000	0.070 ± 0.000	0.142 ± 0.000	0.760 ± 0.000
0.50	0.165 ± 0.000	0.202 ± 0.000	0.671 ± 0.000	0.413 ± 0.000	0.370 ± 0.000	0.571 ± 0.000	0.027 ± 0.000	0.101 ± 0.000	0.775 ± 0.000
0.75	0.169 ± 0.000	0.204 ± 0.000	0.657 ± 0.000	0.429 ± 0.000	0.364 ± 0.000	0.549 ± 0.000	0.030 ± 0.000	0.099 ± 0.000	0.752 ± 0.000
1.00	0.202 ± 0.000	0.222 ± 0.000	0.580 ± 0.000	0.448 ± 0.000	0.350 ± 0.000	0.524 ± 0.000	0.028 ± 0.000	0.110 ± 0.000	0.715 ± 0.000
IMDB	0.25	0.023 ± 0.001	0.087 ± 0.002	0.887 ± 0.001	0.141 ± 0.001	0.194 ± 0.001	0.883 ± 0.001	0.011 ± 0.001	0.068 ± 0.002	0.927 ± 0.001
0.50	0.024 ± 0.001	0.095 ± 0.002	0.851 ± 0.001	0.117 ± 0.002	0.185 ± 0.002	0.838 ± 0.002	0.014 ± 0.001	0.078 ± 0.002	0.920 ± 0.001
0.75	0.051 ± 0.002	0.124 ± 0.002	0.795 ± 0.002	0.110 ± 0.001	0.181 ± 0.002	0.823 ± 0.002	0.009 ± 0.001	0.069 ± 0.001	0.895 ± 0.001
1.00	0.063 ± 0.001	0.136 ± 0.001	0.756 ± 0.001	0.122 ± 0.001	0.201 ± 0.001	0.752 ± 0.001	0.021 ± 0.001	0.088 ± 0.002	0.883 ± 0.001
TREC	0.25	0.139 ± 0.000	0.355 ± 0.000	0.744 ± 0.000	0.286 ± 0.000	0.265 ± 0.000	0.776 ± 0.000	0.109 ± 0.000	0.266 ± 0.000	0.896 ± 0.000
0.50	0.217 ± 0.000	0.444 ± 0.000	0.680 ± 0.000	0.314 ± 0.000	0.503 ± 0.000	0.672 ± 0.000	0.068 ± 0.000	0.148 ± 0.000	0.816 ± 0.000
0.75	0.238 ± 0.000	0.482 ± 0.000	0.592 ± 0.000	0.464 ± 0.000	0.622 ± 0.000	0.520 ± 0.000	0.067 ± 0.000	0.198 ± 0.000	0.784 ± 0.000
1.00	0.207 ± 0.000	0.451 ± 0.000	0.544 ± 0.000	0.528 ± 0.000	0.685 ± 0.000	0.432 ± 0.000	0.150 ± 0.000	0.199 ± 0.000	0.696 ± 0.000
D Recalibration
Through all our recalibration results, we first split the test set into two equally sized calibration and test splits. Then, we fit the calibration method using the calibration split and compute the performance on the test split. We run all our experiments with 10 different random seeds.
D.1 Temperature Scaling
To perform temperature scaling [GPSW17], we use the calibration set to fit the temperature parameter. To perform the optimization, we use the LBFGS [LN89] algorithm from PyTorch with strong Wolfe line search, following [GPSW17]. Namely, we optimize the parameter  $\tau$  with
$$
\hat {\mathbb {P}} _ {\mathrm {T S}} (X) = \operatorname {S o f t m a x} (f (\boldsymbol {X}) / \tau) \tag {9}
$$
and then use it during inference to rescale the logits produced by  $f$ . We use 0.1 learning rate and 3000 maximum iterations across all experiments and initialize the temperature value as 1, although find that TS is pretty robust to the choice of hyperparameters.
D.2 Atypicality-Aware Recalibration
Here we describe the implementation details For Atypicality-Aware Recalibration (AAR). We formulate AAR with:
$$
\log \hat {\mathbb {P}} _ {\mathrm {A A R}} (Y | X) \propto \phi (a (X)) \log \hat {\mathbb {P}} (Y | X) + S _ {Y}, \tag {10}
$$
In total, this gives us  $|\mathcal{Y}| + 3$  parameters. Using exactly the same setting as TS, we use LBFGS with a strong Wolfe search to optimize the three parameters, with the same splits as temperature scaling. We normalize the atypicality values (subtract the mean and divide by the standard deviation of the calibration set) for numerical stability. We use the same hyperparameters as TS (with 0.1 learning rate and 3000 maximum iterations) without any modification across all experiments, initialize  $c_{0}, c_{1}, c_{2}$  as 0 and  $S_{Y}$  parameters as 1. We run the recalibration procedure on a CPU with precomputed logits.
D.2.1 Adaptive Temperature Scaling
For AdaTS [JPL+23] we use the implementation provided with the paper. We identically use the hyperparameters and the architecture provided in the paper and their repository. They use an encoder and decoder architecture with [1024, 512, 512] hidden units each, and a temperature predictor network with [128, 128] hidden units. They use an Adam Optimizer with a learning rate of  $5e - 4$  with 128 batch size.
D.3 Conformal Prediction
We follow the presentation in [AB21, ABMJ20]. Let  $\pi(X)$  be the permutation of  $\mathcal{Y} = \{1, \dots, C\}$  that sorts  $\hat{\mathbb{P}}(Y = c|X)$ , i.e. the predicted probabilities for each class  $c$ . We define a score function
$$
s (x, y) = \sum_ {j = 1} ^ {c} \hat {\mathbb {P}} (Y = j | X), \text {w h e r e} y = \pi_ {c}. \tag {11}
$$
This means greedily including classes until the set contains the true label, and using the cumulative sum of the probabilities as the score function. We compute all of the scores for the calibration set,  $S_{\mathrm{calib}} = \{s(x_1,y_1),\dots,s(x_N,y_N)\}$ , we the  $\frac{[(N + 1)(1 - \alpha)]}{N}$  th quantile of the scores,  $\hat{q}$ . Then, the prediction set is defined as
$$
\mathcal {C} (x) = \{y: s (x, y) \leq \hat {q} \} \tag {12}
$$
We can further add randomization to the procedure where we have the prediction set function to be  $\mathcal{C}(x,u):\mathcal{X}\times [0,1]$  for randomization purposes to satisfy exact coverage. We refer to [VGS05, ABMJ20, AB21] for a more thorough presentation.
<sup>7</sup>https://github.com/thwjoy/adats
RAPS is a variant of APS that regularizes the set sizes. They modify the scoring function to add a regularization term. This is controlled by the test size offset  $k_{reg}$  that controls the value beyond which the regularization is applied, and the  $\lambda_{reg}$  gives the strength of the regularization. To fit the  $k_{reg}$ ,  $\lambda_{reg}$  parameters in RAPS, we follow the procedure in [ABMJ20] to fit both parameters. Namely, we fit  $k_{reg}$  by Algorithm 4 in their paper that leverages the set sizes in the calibration set, and we fit  $\lambda_{reg}$  by the largest regularization parameter that achieves the smallest set sizes, searched over a grid of  $\{0.001, 0.01, 0.1, 0.2, 0.5\}$  following their presentation.
D.4 Atypicality-Aware Conformal Prediction
We have a simple discrete grouping scheme to make conformal prediction atypicality aware. Namely, we group points using their atypicality and confidence percentiles and fit individual thresholds. Concretely, we construct a dataset of  $\mathcal{D}_{AA} = \{(c_i,c_{i + 1}],(a_j,a_{j + 1}],\hat{q}_{i,j}\}_{i,j\in [N]}$  using the calibration set where  $(c_{i},c_{i + 1}]$  denotes the confidence range for quantile  $i$ $(a_{j},a_{j + 1}]$  denotes the atypicality range for quantile  $j$  and  $\hat{q}_{i,j}$  denotes the threshold fitted to the group specified by these intervals. We let  $N = 6$  as the number of groups, and in total, we end up with 36 thresholds. At test time, we check the quantile of the confidence and atypicality of a point and use the corresponding temperature. For AA-RAPS, we use the same  $k_{reg}$  and  $\lambda_{reg}$  values as was found with the RAPS procedure. For practical purposes, we do not allow zero sets (prediction sets at least include the top prediction).
We would like to make the remark that sometimes the marginal coverage can exceed the desired value (e.g. Figure 9). This is often because the underlying model is already very confident for a majority of data points (e.g. More than half of the data points have  $92\%$  confidence). The gains we provide are often for points with lower confidence regions, as the coverage is not satisfied in those regions.
Figure 8: Atypicality-Aware Conformal Prediction for ResNet18 and ImageNet. Target coverage rate is  $95\%$ .
Figure 9: Atypicality-Aware Conformal Prediction for ResNet152 and ImageNet. Target coverage rate is  $95\%$ .
Figure 10: Fitted Temperature vs Atypicality. We observe a monotonically increasing relationship between the atypicality of a group and the temperature parameter fitted to that group with TS.
E Tables for Results
Here, we present the table version of the results in Figure 3. Tables 1,2 contain the ECE analysis.
Additive Correction vs Class Atypicality ResNext50 on ImageNet-LT
Figure 11: Fitted Additive Correction Factor vs Class Atypicality. We observe a monotonically increasing relationship between the atypicality of a class and the additive correction parameter fitted to that class with AAR.
Figure 12: Atypicality-Aware Conformal Prediction for ResNet50 and ImageNet. Target coverage rate is  $95\%$ .
F Fitzpatrick17k and Skin Lesion Classification
We use the training script from  $\left[\mathrm{GHS}^{+}21\right]$  to finetune models on the Fitzpatrick17k dataset. We train the models for 50 epochs, fixing the backbone and training only the probe on top of the penultimate layer. The probe consists of 2 layers, one layer of 256 units followed by ReLU and Dropout with probability 0.4, followed by the classifier layer with an output dimensionality of 9. We use an Adam optimizer with a 0.0001 learning rate.
The entire dataset consists of 16,577 images, where the potential labels are: 10,886 inflammatory, 1,352 malignant epidermal, 1,194 genodermatoses, 1,067 benign dermal, 931 benign epidermal, 573 malignant melanoma, 236 benign melanocyte, 182 malignant cutaneous lymphoma, and 156 malignant dermal. We split the dataset into 3 sets (Training (0.5), Validation (0.25), and Test (0.25)). We use the validation set as the calibration set and perform the experiments with 10 random splits.
G Proofs
G.1 Detailed derivation of the claim on Page 5
When  $\mathbb{P}_1(X) \leq 1/2$ , the signed calibration error at level  $u \in (1/2, 1)$  becomes  $u - \mathbb{P}(Y = -1 \mid \hat{\mathbb{P}}_{-1}(X) = u) = u - \mathbb{P}(Y = -1 \mid \hat{\mathbb{P}}_1(-X) = u) = u - \mathbb{P}(Y = 1 \mid \hat{\mathbb{P}}_1(X) = u)$ .
The last inequality is due to symmetry. More specifically, we claim  $(X,Y)\stackrel {d}{=}(-X, - Y)$ , where the notation  $\stackrel{d}{=}$  denotes equal in distribution. In fact, as  $X\stackrel {d}{=} - X$ , it suffices to show that for any  $y\in \{-1,1\}$ , and  $x\in \mathbb{R}^d$ , we have
$$
\mathbb {P} (Y = y \mid X = x) = \mathbb {P} (- Y = y \mid - X = x).
$$
When  $y = -1$ , the right hand side
$$
\begin{array}{l} \mathbb {P} (- Y = - 1 \mid - X = x) = \mathbb {P} (Y = 1 \mid X = - x) = \sigma (\langle \beta^ {*}, - x \rangle) \\ = 1 - \sigma \left(\left\langle \beta^ {*}, x \right\rangle\right) = 1 - \mathbb {P} (Y = 1 \mid X = x) = \mathbb {P} (Y = - 1 \mid X = x). \\ \end{array}
$$
Similarly, when  $y = 1$
$$
\begin{array}{l} \mathbb {P} (- Y = 1 \mid - X = x) = \mathbb {P} (Y = - 1 \mid X = - x) = 1 - \sigma (\langle \beta^ {*}, - x \rangle) \\ = \sigma (\langle \beta^ {*}, x \rangle) = \mathbb {P} (Y = 1 \mid X = x). \\ \end{array}
$$
We complete the proof.
G.2 Proof of Theorem 3.1
Theorem G.1 (Restatement of Theorem 3.1). Consider the data generative model with the algorithm described in Section 3.2. For any  $K > 1$ , suppose we consider the quantiles of  $a(X)$ ,  $a_1, a_2, \ldots, a_K, a_{K+1}$  such that  $\mathbb{P}(a(X) \in (a_k, a_{k+1})) = 1 / K$  for  $k \in [K]$ . In addition, we assume  $\| \beta^* \| \leq c_0$ , and  $d / n = \kappa$  for some sufficiently small  $c_0, \kappa > 0$ . Then for sufficiently large  $n$ , we have
$$
\begin{array}{l} \mathbb {E} _ {u} \left[ u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u) \mid a (X) \in \left[ a _ {k - 1}, a _ {k} \right] \right] > \\ \mathbb {E} _ {u} \left[ u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u) \mid a (X) \in \left(a _ {k}, a _ {k + 1} \right] \right], \\ \end{array}
$$
for  $k = 2,..,K$
Proof. Following [BMWX21a], we have
$$
u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u) = u - \mathbb {E} _ {Z} [ \sigma (\frac {\| \beta^ {*} \|}{\| \hat {\beta} \|} \cos \hat {\theta} \cdot \sigma^ {- 1} (u)) + \sin \hat {\theta} \cdot \| \beta^ {*} \| Z ],
$$
where  $\cos \hat{\theta} = \frac{\hat{\beta}^{\top}\beta^{*}}{\|\hat{\beta}\| \cdot \|\beta^{*}\|}$  and  $Z \sim N(0,1)$ .
According to the results in Section 2.2 of [SC19], we have  $\| \hat{\beta} \| \to R^{*} = R^{*}(\kappa, \beta^{*})$  and  $\cos \hat{\theta} \to c^{*} = c^{*}(\kappa, \beta^{*})$ , for two quantities  $R^{*}$  and  $c^{*}$  that depend on  $\kappa$  and  $\beta^{*}$ . We then have
$$
u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u) \to u - \mathbb {E} _ {Z} [ \sigma (\frac {\| \beta^ {*} \|}{R ^ {*}} c ^ {*} \cdot \sigma^ {- 1} (u)) + \sqrt {1 - c ^ {* 2}} \cdot \| \beta^ {*} \| Z ].
$$
Using the proof of Theorem 3 in [BMWX21a], we have that
$$
u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u) = C _ {\kappa} (u) \cdot \kappa + o (\kappa),
$$
$$
C _ {\kappa} (u) = c _ {1} \sigma^ {\prime} (\sigma^ {- 1} (u)) \cdot \sigma^ {- 1} (u) - c _ {2} \sigma^ {\prime \prime} (\sigma^ {- 1} (u)),
$$
for two positive constants  $c_{1}, c_{2}$ .
As a result, we have
$$
u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u) \geq 0 \tag {13}
$$
In addition, since when  $z \in [-1, 1]$ ,  $z \cdot \sigma'(z)$  and  $-\sigma''(z)$  are both increasing, we then have  $C_{\kappa}(u)$  increasing for  $\hat{\beta}^\top x = \sigma^{-1}(u) \in (-1, 1)$ .
Proving the result for  $\{k = 2,\dots ,K - 1\}$  In addition, by our model assumption  $x\sim N(0,I_d)$ , we have that  $\| x\|$  and  $\frac{x}{\|x\|}$  are independent, and  $\frac{x}{\|x\|}\sim S$  where  $S$  is a uniform distribution on the sphere in the  $d$ -dimensional space. As the monotonic transformations will not change the events defined by quantiles, and  $\exp (-\| x\| ^2 /2)$  is a monotonic function in  $\| x\|$ , for the simplicity of presentation we use  $a(X) = \| X\|$  in the rest of this proof. As a result, given  $\| x\| = a$ , we have
$$
\hat {\beta} ^ {\top} x \mid \| x \| = a \stackrel {d} {=} a \cdot \hat {\beta} ^ {\top} S = a \cdot \| \hat {\beta} \| \cdot S _ {1},
$$
where  $S_{1}$  is the first coordinate of  $S$
Consequently, if we further condition on the event where  $\hat{\beta}^{\top}x > 0$  (as we assume  $u > 0$  throughout Section 3.2), we have
$$
\hat {\beta} ^ {\top} x \stackrel {d} {=} a \cdot \| \hat {\beta} \| \cdot S _ {1} \mid S _ {1} > 0 \stackrel {d} {=} a \cdot \| \hat {\beta} \| \cdot \frac {Z _ {1}}{\sqrt {Z _ {1} ^ {2} + Q}} \rightarrow a \cdot R ^ {*} \cdot \frac {Z _ {1}}{\sqrt {Z _ {1} ^ {2} + Q}},
$$
where  $Q \sim \chi_{p-1}^2$ ,  $Z_1 \sim N(0,1)$  and they are independent.
Due to the monotonicity of  $C_{\kappa}(u)$  on  $u$ , we have that for any  $a_1 > a_2$
$$
C _ {\kappa} (u) \mid \| x \| = a _ {1} ^ {d} > C _ {\kappa} (u) \mid \| x \| = a _ {2},
$$
where the notation  $\overline{d}$  denotes stochastic dominance.
$$
\mathbb {E} _ {u} \left[ u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u) \mid a (X) \in \left[ a _ {k - 1}, a _ {k} \right] \right] <   \mathbb {E} _ {u} \left[ u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u) \mid a (X) \in \left(a _ {k}, a _ {k + 1} \right] \right],
$$
for  $k = 2,..,K - 1$
Proving the result for  $k = K$  To complete the proof, it suffices to show that the inequality is also true for  $K$ th quantile:
$$
\mathbb {E} _ {u} \left[ u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u) \mid a (X) \in \left[ a _ {K - 1}, a _ {K} \right] \right] <   \mathbb {E} _ {u} \left[ u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u) \mid a (X) \in \left(a _ {k}, a _ {k + 1} \right] \right],
$$
$$
\mathbb {E} _ {u} [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \cdot \mathbf {1} \{a (X) \in [ a _ {K - 1}, a _ {K} ] \} ] <   \mathbb {E} _ {u} [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \cdot \mathbf {1} \{a (X) \in (a _ {k}, a _ {k + 1} ] \} ].
$$
In the above inequality, the right hand side can be decomposed into
$$
\begin{array}{l} \mathbb {E} _ {u} \left[ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \cdot \mathbf {1} \left\{a (X) \in \left(a _ {k}, a _ {k + 1} \right] \right\} \right] \\ = \mathbb {E} _ {u} [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \cdot \mathbf {1} \{a (X) \in [ a _ {K}, 2 p ] \} ] \\ + \mathbb {E} _ {u} [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \cdot \mathbf {1} \{a (X) \in [ 2 p, a _ {K + 1} ] \} ]. \\ \end{array}
$$
Denote the  $\alpha$  quantile of  $\chi_p^2$  by  $\chi_{\alpha ,p}^{2}$ . We then have  $a_{k} = \chi_{\frac{k}{K + 1},p}^{2}$ . We further decompose the equation into
$$
\begin{array}{l} \mathbb {E} _ {u} \left[ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \cdot \mathbf {1} \{a (X) \in [ a _ {K}, 2 p ] \} \right] \\ = \mathbb {E} _ {u} \left[ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \cdot \mathbf {1} \{a (X) \in [ a _ {K}, \chi_ {\frac {K + \delta}{K + 1}, p} ^ {2} ] \} \right] \\ + \mathbb {E} _ {u} \left[ \left(u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)\right) \cdot \mathbf {1} \{a (X) \in \left[ \chi_ {\frac {K + \delta}{K + 1}, p} ^ {2}, 2 p \right] \} \right]. \\ \end{array}
$$
In the following, we proceed to prove
$$
\mathbb {E} _ {u} \left[ \left(u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)\right) \cdot \mathbf {1} \left\{a (X) \in \left[ \chi_ {\frac {K + \delta}{K + 1}, p} ^ {2}, 2 p \right] \right\} \right] > \mathbb {E} _ {u} \left[ \left(u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)\right) \cdot \mathbf {1} \left\{a (X) \in \left[ a _ {K - 1}, a _ {K} \right. \right\} \right]. \tag {14}
$$
We now use the approximation of the chi-square quantile: when  $p\to \infty$  , we have
$$
a _ {K} = \frac {1}{2} \left(z _ {\frac {K}{K + 1}} + \sqrt {2 p}\right) ^ {2} + o (1), \text {a n d} \chi_ {\frac {K + \delta}{K + 1}, p} ^ {2} = \frac {1}{2} \left(z _ {\frac {K + \delta}{K + 1}} + \sqrt {2 p}\right) ^ {2} + o (1),
$$
where  $z_{\alpha}$  denotes the  $\alpha$ -quantile of a standard normal random variable.
$$
\chi_ {\frac {K + \delta}{K + 1}, p} ^ {2} - a _ {K} = \frac {1}{2} (z _ {\frac {K + \delta}{K + 1}} - z _ {\frac {K}{K + 1}}) (z _ {\frac {K + \delta}{K + 1}} + z _ {\frac {K}{K + 1}} + 2 \sqrt {2 p}).
$$
Using the fact that  $z_{1 - \frac{1}{K}} = \sqrt{2\log K} + o(1)$  for  $K \to \infty$ , then we have
$$
z _ {\frac {K + \delta}{K + 1}} - z _ {\frac {K}{K + 1}} = \frac {- \log (1 - \delta)}{\sqrt {2 \log K}} + o (1).
$$
In addition, for any  $a \in [\chi_{\frac{K + \delta}{K + 1}, p}^2, 2p]$  and  $a' \in [a_{K - 1}, a_K]$ , we have
$$
\begin{array}{l} \mathbb {E} _ {u} \left[ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \mid a (X) = a \right] - \mathbb {E} _ {u} \left[ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \mid a (X) = a ^ {\prime} \right] \\ \underline {{\geq}} C \big (z _ {\frac {K + \delta}{K + 1}} - z _ {\frac {K}{K + 1}} \big), \\ \end{array}
$$
for some universal constant  $C$ .
$$
\begin{array}{l} \mathbb {E} _ {u} \big [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \mid a (X) \in [ \chi_ {\frac {K + \delta}{K + 1}, p} ^ {2}, 2 p ] \big ] - \mathbb {E} _ {u} \big [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \mid a (X) \in [ a _ {K - 1}, a _ {K} ] \big ] \\ \geq C (z _ {\frac {K + \delta}{K + 1}} - z _ {\frac {K}{K + 1}}). \\ \end{array}
$$
$$
\mathbb {E} _ {u} \big [ (u - \mathbb {P} (Y = 1 \mid \hat {\mathbb {P}} _ {1} (X) = u)) \cdot \mathbf {1} \{a (X) \in [ \chi_ {\frac {K + \delta}{K + 1}, p} ^ {2}, 2 p ] \} \big ]
$$