SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In summary, from Eqs. 57 and 58, we can conclude that  $(\alpha /2,\xi /|\mathcal{H}|)$ -conditional calibration of  $f_{B}$  on each  $\mathcal{S}_h$ ,  $h\in \mathcal{H}$ , implies that, with probability at least  $1 - \xi$ ,  $f_{B}$  satisfies  $\alpha$ -aligned calibration, where, for all  $h\in \mathcal{H}$ , we have that  $\tilde{\mathcal{S}}_h = \mathcal{S}_h$ .
Lower bound on  $|\mathcal{D} \cap \mathcal{S}_h|$  to achieve conditional calibration with UMD. Running UMD on each partition  $\mathcal{D} \cap \mathcal{S}_h$  of  $\mathcal{D}$  induced by  $h \in \mathcal{H}$  achieves  $(\alpha / 2, \xi / |\mathcal{H}|)$ -conditional calibration as long as each subset  $\mathcal{D} \cap \mathcal{S}_h$  of the data is large enough. More specifically, the following lower bound on the size of the subsets  $\mathcal{D} \cap \mathcal{S}_h$  readily follows from Theorem 3 in Gupta et al. [12].
Lemma 4. The discretized confidence function  $f_{B,\lambda}$  returned by  $|\mathcal{H}|$  instances of UMD, one per  $S_h$ , is  $(\alpha /2,\xi /|\mathcal{H}|)$ -conditional calibrated on  $S_h$  for any  $\xi \in (0,1)$  if
$$
\left| \mathcal {D} \cap \mathcal {S} _ {h} \right| \geq n _ {\min } := \left(\frac {2 \log \left(\frac {2 | \mathcal {H} |}{\xi} \cdot \lceil \frac {1}{\lambda} \rceil\right)}{\alpha^ {2}} + 2\right) \cdot \left\lceil \frac {1}{\lambda} \right\rceil \tag {59}
$$
Proof. Let  $B$  denote the number of bins in UMD. Theorem 3 in Gupta et al. [12] states that, if  $f_{B}(X,H)$  is absolutely continuous with respect to the Lebesgue measure $^{11}$  and  $|\mathcal{D} \cap \mathcal{S}_h| \geq 2B$ , then the discretized confidence function output by UMD is  $(\epsilon, \xi')$ -conditionally calibrated for any  $\xi' \in (0,1)$  and
$$
\epsilon = \sqrt {\frac {\log \left(2 B / \xi^ {\prime}\right)}{2 \left(\left\lfloor | \mathcal {D} \cap \mathcal {S} _ {h} \right| / B \right\rfloor - 1)}}. \tag {60}
$$
Then, for a given  $\alpha$ , setting  $\epsilon = \alpha / 2$ ,  $B = \lceil 1 / \lambda \rceil$  and  $\xi' = \xi / |\mathcal{H}|$ , we can solve Eq. 60 for the lower bound on  $|\mathcal{D} \cap \mathcal{S}_h| \geq n_{\min}$  with  $n_{\min}$  as defined in Eq. 59.
Upper bound on  $|\mathcal{D}|$  to achieve conditional calibration with UMD. Suppose  $P((X,H)\in S_h)\geq \gamma$  for all  $h\in \mathcal{H}$ . When  $|\mathcal{H}|\geq 2$ , we give an upper bound on  $|\mathcal{D}|$  so that with high probability  $|\mathcal{D}\cap \mathcal{S}_h|\geq n_{\min}$  for all  $h\in \mathcal{H}$ .
In the process of sampling  $\mathcal{D} \in (\mathcal{Z} \times \mathcal{Y})^n$  from  $P^{\mathcal{M}}$ , let  $R_i^{(h)} = 1$  denote the event that the  $i$ -th datapoint  $(x_i, h_i, y_i)$  has confidence value  $h$ , i.e.,  $h_i = h$ . Then, we can express  $|\mathcal{D} \cap \mathcal{S}_h|$  in terms of random variable  $R^{(h)}$ , defined as
$$
R ^ {(h)} = \sum_ {i = 1} ^ {| \mathcal {D} |} R _ {i} ^ {(h)}. \tag {61}
$$
Since  $R_{i}^{(h)}$  is a Bernoulli-distributed variable with  $P(R_{i}^{(h)}) = P((X,H)\in \mathcal{S}_{h})$ , the expected value of  $R^{(h)}$  is  $\mu (h)\coloneqq \mathbb{E}[R^{(h)}] = P((X,H)\in \mathcal{S}_h)\cdot |\mathcal{D}|\geq \gamma \cdot |\mathcal{D}|$ .
Let  $|\mathcal{D}| = 2\cdot |\mathcal{H}|\cdot \log (2 / \xi)\cdot 1 / \gamma \cdot n_{\mathrm{min}}$  , observe that in this case
$$
P \left(R ^ {(h)} \leq n _ {\min }\right) = P \left(R ^ {(h)} \leq \frac {\gamma}{2 | \mathcal {H} | \cdot \log (2 / \xi)} \cdot | \mathcal {D} |\right).
$$
For  $|\mathcal{H}|\geq 2$  and  $\xi \in (0,1)$ , we have  $1 / (2|\mathcal{H}|\cdot \log (2 / \xi))\in (0,1)$  and we can use a variation of the Chernoff bound to show
$$
\begin{array}{l} P \left(R ^ {(h)} \leq n _ {\min }\right) \leq P \left(R ^ {(h)} \leq \frac {1}{2 | \mathcal {H} | \cdot \log (2 / \xi)} \cdot \mu (h)\right) \\ \leq e ^ {- \mu (h) \left(\frac {2 | \mathcal {H} | \cdot \log (2 / \xi) - 1}{2 | \mathcal {H} | \cdot \log (2 / \xi)}\right) ^ {2} \cdot \frac {1}{2}} \\ = e ^ {- \mu (h) \cdot \frac {1}{2} \cdot \left(1 - \frac {1}{| \mathcal {H} | \cdot \log (2 / \xi)} + \frac {1}{(2 | \mathcal {H} | \cdot \log (2 / \xi)) ^ {2}}\right)} \\ \leq \frac {\xi}{2} \cdot e ^ {- | \mathcal {H} | \cdot n _ {\min} \cdot \left(\frac {1}{2} - \frac {1}{2 | \mathcal {H} | \cdot \log (2 / \xi)} + \frac {1}{2 (2 | \mathcal {H} | \cdot \log (2 / \xi)) ^ {2}}\right)}, \\ \end{array}
$$
where the first and last inequality results from using  $\mu(h) > \gamma \cdot |\mathcal{D}|$ . We can now use a union bound to obtain a lower bound on the probability that for any  $h \in \mathcal{H}$ ,  $|\mathcal{D} \cap S_h| \leq n_{\min}$ , i.e.,
$$
P \left(\exists h \in \mathcal {H}: | \mathcal {D} \cap \mathcal {S} _ {h} | \leq n _ {\min }\right) \leq \frac {\xi}{2} \cdot | \mathcal {H} | \cdot e ^ {- | \mathcal {H} | \cdot n _ {\min } \cdot \left(\frac {1}{2} - \frac {1}{2 | \mathcal {H} | \cdot \log (2 / \xi)} + \frac {1}{2 (2 | \mathcal {H} | \cdot \log (2 / \xi)) ^ {2}}\right)} \tag {62}
$$
One can verify that for  $|\mathcal{H}| \geq 2$  and  $n_{\min} \geq 1$ , we have  $P(\exists h \in \mathcal{H}: |\mathcal{D} \cap \mathcal{S}_h| \leq n_{\min}) \leq \frac{\xi}{2}$ . Hence, if  $|\mathcal{D}| = 2 \cdot |\mathcal{H}| \cdot \log(2 / \xi) \cdot 1 / \gamma \cdot n_{\min}$ , then, for all  $h \in \mathcal{H}, |\mathcal{D} \cap \mathcal{S}_h| \leq n_{\min}$  with probability  $1 - \xi / 2$ .
<sup>11</sup>If  $f_{B}$  is not continuous with respect to the Lebesgue measure (or equivalently put,  $f_{B}$  does not have a probability density function), a randomization trick can be used to ensure that the results of the theorem hold.
Combining this result and Lemma 4, we have that the discretized confidence function  $f_{B,\lambda}$  returned by  $|\mathcal{H}|$  instances of UMD, one per  $S_h$ , is  $(\alpha /2,\xi /(2|\mathcal{H}|))$ -conditional calibrated on each  $S_h$  with probability at least  $1 - \xi /2$  for any  $\xi \in (0,1)$  if
$$
| \mathcal {D} | = 2 \cdot | \mathcal {H} | \cdot \frac {\log (2 / \xi)}{\gamma} \cdot \left(\frac {2 \log \left(\frac {4 | \mathcal {H} |}{\xi} \cdot \lceil \frac {1}{\lambda} \rceil\right)}{\alpha^ {2}} + 2\right) \cdot \left\lceil \frac {1}{\lambda} \right\rceil \tag {63}
$$
Finally, using a union bound, we can conclude that  $f_{B,\lambda}$  achieves  $\alpha$ -aligned calibration with respect to  $f_{H}$  with probability at least  $1 - \xi$  from
$$
| \mathcal {D} | = O \left(| \mathcal {H} | \cdot \frac {\log (| \mathcal {H} | / \xi \lambda)}{\alpha^ {2} \cdot \lambda \cdot \gamma}\right)
$$
samples. This concludes the proof.
B Multicalibration Algorithm
In this section, we give a high-level description of the post-processing algorithm for multicalibration introduced by Hébert-Johnson et al. [11]. The algorithm works with a discretization of  $[0, 1]$  into uniform sized bins of size  $\lambda$ , for a  $\lambda > 0$ . Formally the  $\lambda$ -discretization of  $[0, 1]$ , is defined as
Definition 14 ( $\lambda$ -discretization [11]). Let  $\lambda > 0$ . The  $\lambda$ -discretization of  $[0,1]$ , denoted by  $\Lambda[0,1] = \{\frac{\lambda}{2}, \frac{3\lambda}{2}, \dots, 1 - \frac{\lambda}{2}\}$ , is the set of  $1 / \lambda$  evenly spaced real values over  $[0,1]$ . For  $b \in \Lambda[0,1]$ , let
$$
\lambda (b) = [ b - \lambda / 2, v + \lambda / 2) \tag {64}
$$
be the  $\lambda$ -interval centered around  $b$  (except for the final interval, which will be  $[1 - \lambda, 1]$ ).
It starts by partitioning each subspace  $S_h$  into  $1 / \lambda$  groups  $S_{h,\lambda (b)} = \{(x,h)\in S_h\mid f_B(x,h)\in \lambda (b)\}$ , with  $b\in \Lambda [0,1]$ . Then, it repeatedly looks for a large enough group  $S_{h,\lambda (b)}$  such that the absolute difference between the average confidence value  $\mathbb{E}[f_B(X,H)\mid (X,H)\in S_{h,\lambda (b)}]$  and the probability  $P(Y = 1\mid (X,H)\in S_{h,\lambda (b)})$  is larger than  $\alpha$  and, if it finds it, it updates the confidence value  $f_{B}(x,h)$  of each  $(x,h)\in S_{h,\lambda (b)}$  by this difference. Once the algorithm cannot find any more such a group, it returns a discretized confidence function  $f_{B,\lambda}(x,h) = \mathbb{E}[f_B(X,H)\mid f_B(X,H)\in \lambda (b)]$ , with  $b\in \Lambda [0,1]$  such that  $f_{B}(x,h)\in \lambda (b)$ , which is guaranteed to satisfy  $(\alpha +\lambda)$ -multicalibration.
Algorithm 1 provides a pseudocode implementation of the overall algorithm. Within the implementation, it is worth noting that the expectations and probabilities can be estimated with fresh samples from the distribution or from a fixed dataset using tools from differential privacy and adaptive data analysis, as discussed in Hébert-Johnson et al. [11].
Algorithm 1 Post-processing algorithm for  $(\alpha + \lambda)$ -multicalibration
1: Input: confidence function  $f_{B}$ , parameters  $\alpha, \lambda > 0$   
2: Output: confidence function  $f_{B,\lambda}$   
3: repeat  
4: updated  $\leftarrow$  false  
5: for  $S_{h} \in \mathcal{C} \& b \in \Lambda[0,1]$  do  
6:  $S_{h,\lambda(b)} \gets S_{h} \cap \{(x,h) \in \mathcal{Z} \mid f_{B}(x,h) \in \lambda(b)\}$   
7: if  $P((X,H) \in S_{h,\lambda(b)}) < \alpha\lambda \cdot P((X,H) \in S_{h})$  then  
8: continue  
9:  $\bar{b}_{h,\lambda(b)} \gets \mathbb{E}[f_{B}(X,H) \mid (X,H) \in S_{h,\lambda(b)}]$   
10:  $r_{h,\lambda(b)} \gets P(Y = 1 \mid (X,H) \in S_{h,\lambda(b)})$   
11: if  $|r_{h,\lambda(b)} - \bar{b}_{h,\lambda(b)}| > \alpha$  then  
12: updated  $\leftarrow$  true  
13: for  $(x,h) \in S_{h,\lambda(b)}$  do  
14:  $f_{B}(x,h) \gets f_{B}(x,h) + (r_{h,\lambda(b)} - \bar{b}_{h,\lambda(b)})$  {project into [0,1] if necessary}  
15: until updated = false  
16: for  $b \in \Lambda[0,1]$  do  
17:  $\bar{b}_{\lambda(b)} \gets \mathbb{E}[f_{B}(X,H)|f_{B}(X,H) \in \lambda(b)]$   
18: for  $(x,h) \in \mathcal{Z}: f_{B}(x,h) \in \lambda(b)$  do  
19:  $f_{B,\lambda}(x,h) \gets \bar{b}_{\lambda(b)}$   
20: return  $f_{B,\lambda}$
C Additional Details about the Experiments
Transformation of confidence values. In the Human-AI Interactions dataset, the AI model is a simple statistical model where  $b$  is just a noisy average confidence  $h$  of an independent set of ca. 50 human labelers on each task instance. Moreover, the confidence values were originally recorded on a scale of  $[-1, 1]$ , where 1 means complete certainty on the correct true label and -1 means complete certainty on the incorrect label. To better match our theoretical framework, we transform all confidence values to a scale of  $[0, 1]$ , where 1 means complete certainty that the true label  $y = 1$  and 0 means complete certainty that the true label is  $y \neq 1$ . More formally, let  $\hat{b}, \hat{h}, \hat{h}_{+\mathrm{AI}} \in [-1, 1]$  be the original confidence values in the dataset, then we obtain  $b \in [0, 1]$  via the following transformation:
$$
b = \left\{ \begin{array}{l l} (\hat {b} + 1) / 2 & \text {i f} y = 1 \\ 1 - (\hat {b} + 1) / 2 & \text {i f} y = 0, \end{array} \right.
$$
and analogously for  $h$  and  $h_{+ \mathrm{Al}}$ .
Comparing decision policies  $\pi_B, \pi_H$  and  $\pi_{H + AI}$ . Figure 6 shows the ROC curves for the decision policies  $\pi_B, \pi_H$  and  $\pi_{H + AI}$  in each of the four tasks in the Human-AI Interactions dataset.
(c) Cities
Figure 6: ROC curves for the decision policies  $\pi_B$ ,  $\pi_H$  and  $\pi_{H_{+}\mathrm{Al}}$ .
(d) Census