SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {\pi} \left(h ^ {\prime}, b ^ {\prime}, w\right) = 0 \leq \hat {\pi} \left(h ^ {\prime \prime}, b ^ {\prime \prime}, w\right) \text {f o r a l l} w \in \bigcup_ {h \geq h ^ {\prime}, b \in \left[ b ^ {\prime}, \hat {b} _ {h ^ {\prime}}\right)} \tilde {\mathcal {W}} ^ {(\pi , h, b)} \tag {28}
$$
Analogously to case 1, since the values of  $w$  below are also in  $\mathcal{W} - \tilde{\mathcal{W}}_{h',b'}^{(\pi,h'',b'')}$  and  $\hat{\pi}$  is equivalent to  $\pi$  for these values, we have that
$$
\hat {\pi} \left(h ^ {\prime}, b ^ {\prime}, w\right) \leq \hat {\pi} \left(h ^ {\prime \prime}, b ^ {\prime \prime}, w\right) \text {f o r a l l} w \in \mathcal {W} - \bigcup_ {h \leq h ^ {\prime \prime}, b \in \left[ \hat {b} _ {h ^ {\prime \prime}}, b ^ {\prime \prime} \right]} \tilde {\mathcal {W}} ^ {(\pi , h, b)} - \bigcup_ {h \geq h ^ {\prime}, b \in \left[ b ^ {\prime}, \hat {b} _ {h ^ {\prime}}\right)} \tilde {\mathcal {W}} ^ {(\pi , h, b)} \tag {29}
$$
From Eqs. 27 28 and 29, it follows that  $\tilde{\mathcal{W}}_{h',b'}^{(\pi ,h'',b'')} = \emptyset$ .
Case 3:  $b' < \hat{b}_{h'}$  and  $b'' < \hat{b}_{h''}$ .
$$
\bigcup_ {h \geq h ^ {\prime \prime}, b \in \left[ b ^ {\prime \prime}, \hat {b} _ {h ^ {\prime \prime}}\right)} \tilde {\mathcal {W}} ^ {(\pi , h, b)} \subseteq \bigcup_ {h \geq h ^ {\prime}, b \in \left[ b ^ {\prime}, \hat {b} _ {h ^ {\prime}}\right)} \tilde {\mathcal {W}} ^ {(\pi , h, b)}.
$$
$$
\hat {\pi} \left(h ^ {\prime}, b ^ {\prime}, w\right) = 0 \leq \hat {\pi} \left(h ^ {\prime \prime}, b ^ {\prime \prime}, w\right) \text {f o r a l l} w \in \bigcup_ {h \geq h ^ {\prime}, b \in \left[ b ^ {\prime}, \hat {b} _ {h ^ {\prime}}\right)} \tilde {\mathcal {W}} ^ {(\pi , h, b)} \tag {30}
$$
Again analogously to case 1, since the values of  $w$  below are also in  $\mathcal{W} - \tilde{\mathcal{W}}_{h',b'}^{(\pi,h'',b'')}$  and  $\hat{\pi}$  is equivalent to  $\pi$  for these values, we have that
$$
\hat {\pi} \left(h ^ {\prime}, b ^ {\prime}, w\right) \leq \hat {\pi} \left(h ^ {\prime \prime}, b ^ {\prime \prime}, w\right) \text {f o r a l l} w \in \mathcal {W} - \bigcup_ {h \geq h ^ {\prime}, b \in \left[ b ^ {\prime}, \hat {b} _ {h ^ {\prime}}\right)} \tilde {\mathcal {W}} ^ {(\pi , h, b)} \tag {31}
$$
From Eqs. 30 and 31, it follows that  $\tilde{\mathcal{W}}_{h',b'}^{(\hat{\pi},h'',b'')} = \emptyset$ .
Note that, we cannot have a case where  $b' \geq \hat{b}_{h'}$  and  $b'' < \hat{b}_{h''}$ , as this would imply  $b'' < b'$ . Since, in all three possible cases, we have shown that  $\tilde{\mathcal{W}}_{h',b'}^{(\hat{\pi},h'',b'')} = \emptyset$ , we can conclude that  $\hat{\pi} \in \Pi(H,B)$  is monotone.
Proof  $\hat{\pi}$  is near optimal.
First, we rewrite the inner expectation in Eq. 10 as
$$
\begin{array}{l} \mathbb {E} _ {\pi} [ u (T, Y) \mid H, B ] = \mathbb {E} [ u (0, Y) \mid H, B ] + (\mathbb {E} [ u (1, Y) \mid H, B ] \\ - \mathbb {E} [ u (0, Y) \mid H, B) ] \cdot P _ {\pi} (T = 1 \mid H, B). \\ \end{array}
$$
Further, recall that  $|\tilde{S}_h| \geq (1 - \alpha / 2)|S_h|$  for all  $h \in \mathcal{H}$  and, for all  $h', h'' \in \mathcal{H}$ ,  $h' \leq h''$  and all  $b', b'' \in [0,1]$ ,  $b' \leq b''$ , we have that
$$
P (Y = 1 \mid f _ {B} (Z) = b ^ {\prime}, Z \in \tilde {S} _ {h ^ {\prime}}) - P (Y = 1 \mid f _ {B} (Z) = b ^ {\prime \prime}, Z \in \tilde {S} _ {h ^ {\prime \prime}}) \leq \alpha \tag {32}
$$
Now, for any  $h' \in \mathcal{H}, b' \in \mathcal{B}$ , we show an upper bound on  $\mathbb{E}_{\pi}[u(T,Y) \mid H = h', B = b'] - \mathbb{E}_{\hat{\pi}}[u(T,Y) \mid H = h', B = b']$ . We distinguish between three cases.
— Case 1:  $b' \geq \hat{b}_{h'}$  and  $P(Y = 1 \mid H = h', B = b') \geq c$ .
Using Lemma 2, we have that
$$
\left(\mathbb {E} [ u (1, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] - \mathbb {E} [ u (0, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ]\right) \geq 0 \tag {33}
$$
Moreover, as  $b' \geq \hat{b}_{h'}$ , the distribution of positive decisions in  $\hat{\pi}$  may also increase for  $h', b'$  compared to  $\pi$  (see Eq. 24), i.e.,
$$
P _ {\pi} (T = 1 \mid H = h ^ {\prime}, B = b ^ {\prime}) - P _ {\hat {\pi}} (T = 1 \mid H = h ^ {\prime}, B = b ^ {\prime}) \leq 0
$$
Hence, it follows that
$$
\begin{array}{l} \mathbb {E} _ {\pi} \left[ u (T, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} \right] - \mathbb {E} _ {\hat {\pi}} \left[ u (T, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} \right] \\ = (\mathbb {E} [ u (1, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] - \mathbb {E} [ u (0, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ]) \tag {34} \\ \times \left(P _ {\pi} (T = 1 \mid H = h ^ {\prime}, B = b ^ {\prime}) - P _ {\hat {\pi}} (T = 1 \mid H = h ^ {\prime}, B = b ^ {\prime})\right) \leq 0. \\ \end{array}
$$
— Case 2:  $b' \geq \hat{b}_{h'}$  and  $P(Y = 1 \mid H = h', B = b') < c$ .
Since  $b' \geq \hat{b}_{h'}$ , there exists  $h, b \in \mathcal{H} \times \mathcal{B}$ , with  $h \leq h'$ ,  $b \leq b'$ , such that  $P(Y = 1 \mid B = b, Z \in \tilde{\mathcal{S}}_h) \geq c$ . Moreover, using the definition of  $\alpha$ -alignment, we have that
$$
P (Y = 1 \mid B = b, Z \in \tilde {S} _ {h}) \leq P (Y = 1 \mid B = b ^ {\prime}, Z \in \tilde {S} _ {h ^ {\prime}}) + \alpha \tag {35}
$$
Then, we can use this to lower bound the expected utility of  $T = 1$  given  $B = b'$  and  $Z \in \tilde{S}_{h'}$  as follows:
$$
\begin{array}{l} \mathbb {E} [ u (1, Y) \mid B = b, Z \in \tilde {\mathcal {S}} _ {h} ] - \mathbb {E} [ u (1, Y) \mid B = b ^ {\prime}, Z \in \tilde {\mathcal {S}} _ {h ^ {\prime}} ] \\ = u (1, 1) \cdot \left(P (Y = 1 \mid B = b, Z \in \tilde {\mathcal {S}} _ {h}) - P (Y = 1 \mid B = b ^ {\prime}, Z \in \tilde {\mathcal {S}} _ {h ^ {\prime}}\right) \tag {36} \\ + u (1, 0) \cdot \left(P \left(Y = 1 \mid B = b ^ {\prime}, Z \in \tilde {S} _ {h ^ {\prime}}\right) - P \left(Y = 1 \mid B = b, Z \in \tilde {S} _ {h}\right)\right) \\ \leq (u (1, 1) - u (1, 0)) \cdot \alpha , \\ \end{array}
$$
where the last inequality due to Eq. 35 and the assumption that  $u(1,1) - u(1,0) > 0$ . Analogously, we can also upper bound the expected utility of  $T = 0$  given  $H = h'$ ,  $B = b'$  and  $Z \in \tilde{S}_{h'}$  as follows:
$$
\begin{array}{l} \mathbb {E} [ u (0, Y) \mid B = b, Z \in \tilde {S} _ {h} ] - \mathbb {E} [ u (0, Y) \mid B = b ^ {\prime}, Z \in \tilde {S} _ {h ^ {\prime}} ] \\ = u (0, 1) \cdot \left(P (Y = 1 \mid B = b, Z \in \tilde {\mathcal {S}} _ {h}) - P (Y = 1 \mid B = b ^ {\prime}, Z \in \tilde {\mathcal {S}} _ {h ^ {\prime}}\right) \tag {37} \\ + u (0, 0) \cdot \left(P \left(Y = 1 \mid B = b ^ {\prime}, Z \in \tilde {S} _ {h ^ {\prime}}\right) - P \left(Y = 1 \mid B = b, Z \in \tilde {S} _ {h}\right)\right) \\ \geq (u (0, 1) - u (0, 0)) \cdot \alpha , \\ \end{array}
$$
where the last inequality holds due to Eq. 35 and the assumption that  $u(0,1) - u(0,0) < 0$ .
Now, as  $P(Y = 1 \mid B = b, Z \in \tilde{S}_h) \geq c$ , by Lemma 2, we have that
$$
\mathbb {E} [ u (1, Y) \mid B = b, Z \in \tilde {\mathcal {S}} _ {h} ] \geq \mathbb {E} [ u (0, Y) \mid B = b, Z \in \tilde {\mathcal {S}} _ {h} ] \tag {38}
$$
Combining Eqs. 36, 37 and 38, we obtain
$$
\begin{array}{l} \mathbb {E} [ u (1, Y) \mid B = b ^ {\prime}, Z \in \tilde {\mathcal {S}} _ {h ^ {\prime}} ] + \alpha (u (1, 1) - u (1, 0)) \tag {39} \\ \geq \mathbb {E} [ u (0, Y) \mid B = b ^ {\prime}, Z \in \tilde {S} _ {h ^ {\prime}} ] + \alpha (u (0, 1) - u (0, 0)) \\ \end{array}
$$
In addition, note that we have following trivial bound for the expectation when  $H = h'$  but  $Z \notin \tilde{S}_{h'}$ .
$$
u (1, 0) \leq \mathbb {E} [ u (1, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] \leq u (1, 1), \tag {40}
$$
$$
u (0, 1) \leq \mathbb {E} [ u (0, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] \leq u (0, 0) \tag {41}
$$
Moreover, since  $b' \geq \hat{b}_{h'}$ , the distribution of positive decisions in  $\hat{\pi}$  may also increase for  $h', b'$  compared to  $\pi$ , i.e.,
Hence, we have that
$$
\begin{array}{l} \mathbb {E} _ {\pi} [ u (T, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] - \mathbb {E} _ {\hat {\pi}} [ u (T, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] \tag {42} \\ \leq (- 1) \cdot (\mathbb {E} [ u (1, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] - \mathbb {E} [ u (0, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ]), \\ \end{array}
$$
where the inequality follows since  $\mathbb{E}[u(1,Y) \mid H = h', B = b'] - \mathbb{E}[u(0,Y) \mid H = h', B = b'] \leq 0$  by Lemma 2 as  $P(Y = 1 \mid H = h', B = b') < c$ .
Finally, combining Eqs. 39, 40, 41 and 42 and using the law of total expectation, we obtain
$$
\begin{array}{l} \mathbb {E} _ {\pi} [ u (T, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] - \mathbb {E} _ {\hat {\pi}} [ u (T, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] \\ \leq \left(1 - \beta_ {\left(h ^ {\prime}, b ^ {\prime}\right)}\right) \left(\mathbb {E} [ u (0, Y) \mid B = b ^ {\prime}, Z \in \tilde {S} _ {h ^ {\prime}} ] - \mathbb {E} [ u (1, Y) \mid B = b ^ {\prime}, Z \in \tilde {S} _ {h ^ {\prime}} ]\right) \\ + \beta_ {\left(h ^ {\prime}, b ^ {\prime}\right)} \left(\mathbb {E} [ u (0, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] - \mathbb {E} [ u (1, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ]\right) \\ \leq \left(1 - \beta_ {\left(h ^ {\prime}, b ^ {\prime}\right)}\right) \alpha (u (1, 1) - u (1, 0) + u (0, 0) - u (0, 1)) + \beta_ {\left(h ^ {\prime}, b ^ {\prime}\right)} (u (0, 0) - u (1, 0)), \tag {43} \\ \end{array}
$$
where  $\beta_{(h',b')}$  denotes the probability of  $Z \notin \tilde{\mathcal{S}}_{h'}$  given  $H = h'$ ,  $B = b'$ , i.e.,  $\beta_{(h',b')} = P(Z \notin \tilde{\mathcal{S}}_{h'} | H = h', B = b')$ .
Case 3:  $b' < \hat{b}_{h'}$ .
For all  $h, b$ , with  $h \leq h'$ ,  $b \leq b'$ , we have that  $P(Y = 1 \mid B = b, Z \in \tilde{S}_h) < c$ . In particular,  $P(Y = 1 \mid B = b', Z \in \tilde{S}_{h'}) < c$ . Thus, by Lemma 2,
$$
\mathbb {E} [ u (1, Y) \mid B = b ^ {\prime}, Z \in \tilde {\mathcal {S}} _ {h ^ {\prime}} ] <   \mathbb {E} [ u (0, Y) \mid B = b ^ {\prime}, Z \in \tilde {\mathcal {S}} _ {h ^ {\prime}} ] \tag {44}
$$
In this case, since  $b' < \hat{b}_{h'}$ , the distribution of positive decisions in  $\hat{\pi}$  may decrease for  $h, b$  compared to  $\pi$ , i.e.,
$$
0 \leq P _ {\pi} (T = 1 \mid H = h, B = b) - P _ {\hat {\pi}} (T = 1 \mid H = h, B = b)
$$
Combining Eqs.44, 40 and 41 and using the law of total expectation, we obtain
$$
\begin{array}{l} \mathbb {E} _ {\pi} [ u (T, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] - \mathbb {E} _ {\hat {\pi}} [ u (T, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] \\ \leq \left(\mathbb {E} [ u (1, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] - \mathbb {E} [ u (0, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ]\right) \cdot 1 \\ = \left(1 - \beta_ {\left(h ^ {\prime}, b ^ {\prime}\right)}\right) \left(\mathbb {E} [ u (1, Y) \mid B = b ^ {\prime}, Z \in \tilde {S} _ {h ^ {\prime}} ] - \mathbb {E} [ u (0, Y) \mid B = b ^ {\prime}, Z \in \tilde {S} _ {h ^ {\prime}} ]\right) \\ + \beta_ {\left(h ^ {\prime}, b ^ {\prime}\right)} \left(\mathbb {E} \left[ u (1, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} \right] - \mathbb {E} _ {Y} \left[ u (0, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} \right]\right) \\ \leq \beta_ {\left(h ^ {\prime}, b ^ {\prime}\right)} \left(u (1, 1) - u (0, 1)\right), \tag {45} \\ \end{array}
$$
where again  $\beta_{(h',b')} = P(Z \notin \tilde{S}_{h'} | H = h', B = b')$ .
Now, for a fixed  $h^\prime \in \mathcal{H}$ , since  $|\tilde{S}_{h^{\prime}}|\geq (1 - \alpha /2)|\mathcal{S}_{h^{\prime}}|$ , we know that  $0\leq \sum_{b\in \mathcal{B}}\beta_{(h^{\prime},b)}\leq \alpha /2$ . Hence, combining Eqs. 34, 43 and 45 from the three cases above, we have that
$$
\begin{array}{l} \mathbb {E} _ {B} \left[ \mathbb {E} _ {\pi} [ u (T, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] \right] - \mathbb {E} _ {B} \left[ \mathbb {E} _ {\hat {\pi}} [ u (T, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] \right] \\ = \mathbb {E} _ {B} \left[ \mathbb {E} _ {\pi} [ u (T, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] - \mathbb {E} _ {\hat {\pi}} [ u (T, Y) \mid H = h ^ {\prime}, B = b ^ {\prime} ] \right] \\ \leq \max  \left\{\alpha (u (1, 1) - u (1, 0) + u (0, 0) - u (0, 1)) + \frac {\alpha}{2} \cdot (u (0, 0) - u (1, 0)), \frac {\alpha}{2} \cdot (u (1, 1) - u (0, 1)) \right\} \\ \leq \alpha \cdot (u (1, 1) - u (0, 1) + \frac {3}{2} \cdot (u (0, 0) - u (1, 0))). \\ \end{array}
$$
Finally, since by assumption  $\pi$  is optimal, i.e.,  $\mathbb{E}_{\pi}[u(T,Y)] = \mathbb{E}_{\pi^{*}}[u(T,Y)] = \max_{\pi^{\prime}\in \Pi (H,B)}\mathbb{E}_{\pi^{\prime}}[u(T,Y)]$ , we can conclude by the law of total expectation that
$$
\begin{array}{l} \mathbb {E} _ {\pi^ {*}} [ u (T, Y) ] = \mathbb {E} _ {H} \mathbb {E} _ {B} \left[ \mathbb {E} _ {Y, T \mid \pi} [ u (T, Y) \mid H, B ] \right] \\ \leq \mathbb {E} _ {\hat {\pi}} [ u (T, Y) ] + \alpha \cdot (u (1, 1) - u (0, 1) + \frac {3}{2} \cdot (u (0, 0) - u (1, 0))). \\ \end{array}
$$
A.4 Proof of Theorem 8
If  $f_{B}$  is  $\alpha / 2$ -multicalibrated with respect to  $\{S_{h}\}_{h \in \mathcal{H}}$ , then, by definition, for any  $h \in \mathcal{H}$ , there exists  $\tilde{S}_{h} \subset S_{h}$  with  $|\mathcal{S}| \geq (1 - \alpha / 2)|S_{h}|$  such that, for any  $b \in [0,1]$ , it holds that
$$
\left| P (Y = 1 \mid f _ {B} (Z) = b, Z \in \tilde {S} _ {h}) - b \right| \leq \alpha / 2.
$$
This directly implies that, for any  $h', h'' \in \mathcal{H}$  and  $b', b'' \in [0,1]$ , we have that
$$
P (Y = 1 \mid f _ {B} (Z) = b ^ {\prime}, Z \in \tilde {S} _ {h ^ {\prime}}) - b ^ {\prime} - P (Y = 1 \mid f _ {B} (Z) = b ^ {\prime \prime}, Z \in \tilde {S} _ {h ^ {\prime \prime}}) - b ^ {\prime \prime} \leq \alpha \tag {46}
$$
and, using linearity of expectation, we further have that
$$
P (Y = 1 \mid f _ {B} (Z) = b ^ {\prime}, Z \in \tilde {S} _ {h ^ {\prime}}) - P (Y = 1 \mid f _ {B} (Z) = b ^ {\prime \prime}, Z \in \tilde {S} _ {h ^ {\prime \prime}}) \leq \alpha + b ^ {\prime} - b ^ {\prime \prime}, \tag {47}
$$
showing that, whenever  $b' \leq b''$ , the  $\alpha$ -alignment condition is met. This proves that  $f_{B}$  is  $\alpha$ -aligned with respect to  $f_{H}$ .
Finally, if  $f_{B}$  is  $\alpha / 2$ -multicalibrated with respect to  $\{S_{h}\}_{h \in \mathcal{H}}$ , then, it is  $\alpha / 2$ -calibrated with respect to any of the sets  $S_{h}$ . Since  $\mathcal{Z} = \cup_{h \in \mathcal{H}} S_{h}$ , this implies that  $f_{B}$  is  $\alpha / 2$ -calibrated with respect to  $\mathcal{Z}$ . This concludes the proof.
A.5 Proof of Proposition 1
Given a discretization parameter  $\lambda$ , Algorithm 1 works with a discretized notion of  $\alpha$ -multicalibration, namely  $(\alpha, \lambda)$ -multicalibration:
Definition 10. Let  $\mathcal{C} \subseteq 2^{\mathcal{Z}}$  be a collection of subsets of  $\mathcal{Z}$ . For any  $\alpha, \lambda > 0$ , confidence function  $f_{B}: \mathcal{Z} \to [0,1]$  is  $(\alpha, \lambda)$ -multicalibrated with respect to  $\mathcal{C}$  if, for all  $S \in \mathcal{C}$ ,  $b \in \Lambda[0,1]$ , and all  $S_{h,\lambda(b)}(g)$  such that  $|\mathcal{S}_{h,\lambda(b)}| \geq \alpha \lambda |\mathcal{S}_h|$ , it holds that
$$
\left| \mathbb {E} \left[ f _ {B} (X, H) - P (Y = 1 \mid X, H) \mid (X, H) \in \mathcal {S} _ {h, \lambda (b)} \right] \right| \leq \alpha . \tag {48}
$$
Here, we can analogously define a discretized notion of  $\alpha$ -alignment, namely  $(\alpha, \lambda)$ -alignment.
Definition 11. For  $\alpha, \lambda > 0$ , a confidence function  $f_{B}: \mathcal{Z} \to [0,1]$  is  $(\alpha, \lambda)$ -aligned with respect to  $f_{H}$  if, for all  $h', h'' \in \mathcal{H}$ ,  $h' \leq h''$ , and all  $b', b'' \in \Lambda[0,1]$ ,  $b' \leq b''$ , with  $|\mathcal{S}_{h', \lambda(b')}| > \alpha/2 \cdot \lambda |S_{h'}|$  and  $|\mathcal{S}_{h'', \lambda(b')}| > \alpha/2 \cdot \lambda |S_{h''}|$ , we have
$$
P (Y = 1 \mid (X, H) \in \mathcal {S} _ {h ^ {\prime}, \lambda \left(b ^ {\prime}\right)}) - P (Y = 1 \mid (X, H) \in \mathcal {S} _ {h ^ {\prime \prime}, \lambda \left(b ^ {\prime \prime}\right)}) \leq \alpha . \tag {49}
$$
In what follows, we first show that  $(\alpha, \lambda)$ -multicalibration with respect to  $\{\mathcal{S}_h\}_{h \in \mathcal{H}}$  implies  $(2\alpha + \lambda, \lambda)$ -alignment with respect to  $f_H$ .
Theorem 12. For  $\alpha, \lambda > 0$ , if  $f_{B}$  is  $(\alpha, \lambda)$ -multicalibrated with respect to  $\{S_{h}\}_{h \in \mathcal{H}}$ , then  $f_{B}$  is  $(2\alpha + \lambda, \lambda)$ -aligned with respect to  $f_{H}$ .
Proof. If  $f_{B}$  is  $(\alpha, \lambda)$ -multicalibrated with respect to  $\{S_{h}\}_{h \in \mathcal{H}}$ , then, by definition, for all  $h \in \mathcal{H}$ ,  $b \in \Lambda[0,1]$ , and all  $S_{h,\lambda(b)}$  such that  $|\mathcal{S}_{h,\lambda(b)}| \geq \alpha \cdot \lambda |\mathcal{S}_{h}|$ , it holds that
$$
\left| \mathbb {E} \left[ f _ {B} (X, H) - P (Y = 1 \mid X, H) \mid (X, H) \in \mathcal {S} _ {h, \lambda (b)} \right] \right| \leq \alpha . \tag {50}
$$
This directly implies that, for all  $h', h'' \in \mathcal{H}, b', b'' \in \Lambda[0,1]$  with  $|\mathcal{S}_{h',\lambda(b')}| \geq \alpha \cdot \lambda |S_{h'}|$  and  $|\mathcal{S}_{h'',\lambda(b'')}| \geq \alpha \cdot \lambda |S_{h''}|$ , it holds that
$$
\begin{array}{l} \begin{array}{l} \mathbb {E} [ f _ {B} (X, H) - P (Y = 1 \mid X, H) \mid (X, H) \in \mathcal {S} _ {h ^ {\prime \prime}, \lambda \left(b ^ {\prime \prime}\right)} ] \\ \mathbb {E} [ f _ {B} (X, H) - P (Y = 1 \mid X, H) \mid (X, H) = 2, h = 1, 2 \end{array} \tag {51} \\ - \mathbb {E} [ f _ {B} (X, H) - P (Y = 1 \mid X, H) \mid (X, H) \in \mathcal {S} _ {h ^ {\prime}, \lambda \left(b ^ {\prime}\right)} ] \leq 2 \alpha \\ \end{array}
$$
and, using the linearity of expectation, we have that
$$
\begin{array}{l} P (Y = 1 \mid (X, H) \in \mathcal {S} _ {h ^ {\prime}, \lambda \left(b ^ {\prime}\right)}) - P (Y = 1 \mid (X, H) \in \mathcal {S} _ {h ^ {\prime \prime}, \lambda \left(b ^ {\prime \prime}\right)}) \tag {52} \\ \leq 2 \alpha + \mathbb {E} \left[ f _ {B} (X, H) \mid (X, H) \in \mathcal {S} _ {h ^ {\prime}, \lambda \left(b ^ {\prime}\right)} \right] - \mathbb {E} \left[ f _ {B} (X, H) \mid (X, H) \in \mathcal {S} _ {h ^ {\prime \prime}, \lambda \left(b ^ {\prime \prime}\right)} \right]. \\ \end{array}
$$
Whenever  $b' \leq b''$ , due to the  $\lambda$ -discretization, we have that
$$
\mathbb {E} \left[ f _ {B} (X, H) \mid (X, H) \in \mathcal {S} _ {h ^ {\prime}, \lambda \left(b ^ {\prime}\right)} \right] - \mathbb {E} \left[ f _ {B} (X, H) \mid (X, H) \in \mathcal {S} _ {h ^ {\prime \prime}, \lambda \left(b ^ {\prime \prime}\right)} \right] \leq \lambda \tag {53}
$$
Hence, we have shown that if  $f_{B}$  is  $\alpha$ -multicalibrated, then for all  $h', h'' \in \mathcal{H}, b', b'' \in \Lambda[0,1]$  with  $|S_{h',\lambda(b')}| \geq \alpha \cdot \lambda |S_{h'}|$  and  $|S_{h'',\lambda(b'')}| \geq \alpha \cdot \lambda |S_{h''}|$ , we have
$$
P (Y = 1 \mid (X, H) \in \mathcal {S} _ {h ^ {\prime}, \lambda \left(b ^ {\prime}\right)}) - P (Y = 1 \mid (X, H) \in \mathcal {S} _ {h ^ {\prime \prime}, \lambda \left(b ^ {\prime \prime}\right)}) \leq 2 \alpha + \lambda . \tag {54}
$$
Further, note that  $(2\alpha + \lambda) / 2 \cdot \lambda > \alpha \cdot \lambda$  as  $\lambda > 0$ . This concludes the proof.
Next, we show that, if  $f_{B}$  is  $(\alpha, \lambda)$ -aligned, then  $f_{B,\lambda}$  is  $\alpha$ -aligned with respect to  $f_{H}$ .
Theorem 13. For  $\alpha, \lambda > 0$ , if  $f_{B}$  is  $(\alpha, \lambda)$ -aligned with respect to  $f_{H}$ , then  $f_{B, \lambda}$  is  $\alpha$ -aligned with respect to  $f_{H}$ .
Proof. The proof is similar to the proof of Lemma 1 in Hébert-Johnson et al. [11]. Consider all  $S_{h,\lambda(b)}$  such that  $|\mathcal{S}_{h,\lambda(b)}| < \alpha \lambda |\mathcal{S}_h|$ . By the  $\lambda$ -discretization, there are at most  $1 / \lambda$  such sets, thus, the cardinality of their union is at most  $1 / \lambda \alpha \lambda |\mathcal{S}_h| = \alpha |\mathcal{S}_h|$ . Hence, for all  $h \in \mathcal{H}$ , there exists a subset  $\tilde{S}_h \subset S_h$  with  $|\tilde{S}_h| \geq (1 - \alpha)|S_h|$  such that, for all  $h', h'' \in \mathcal{H}$ , with  $h' \leq h''$ , and all  $b', b'' \in \Lambda[0,1]$ , with  $b' \leq b''$ , it holds that
$$
P (Y = 1 \mid (X, H) \in \mathcal {S} _ {h ^ {\prime}, \lambda \left(b ^ {\prime}\right)} \cap \tilde {\mathcal {S}} _ {h ^ {\prime}}) - P (Y = 1 \mid (X, H) \in \mathcal {S} _ {h ^ {\prime \prime}, \lambda \left(b ^ {\prime \prime}\right)} \cap \tilde {\mathcal {S}} _ {h ^ {\prime \prime}}) \leq \alpha . \tag {55}
$$
The  $\lambda$ -discretization sets all values of  $(x,h) \in S_{h',\lambda(b')}$  to  $f_{B,\lambda}(x,h) = \mathbb{E}[f_B(X,H) \mid f_B(X,H) \in \lambda(b')]$ . Note that, for  $(x,h) \in S_{h',\lambda(b')}$ ,  $f_{B,\lambda}(x,h) \in \lambda(b')$  and for  $(x,h) \in S_{h'',\lambda(b'')}$ ,  $f_{B,\lambda}(x,h) \in \lambda(b'')$ , so it still holds that  $\mathbb{E}[f_B(X,H) \mid f_B(X,H) \in \lambda(b')] \leq \mathbb{E}[f_B(X,H) \mid f_B(X,H) \in \lambda(b'')]$ . Thus, using Eq. 55, we have that
$$
\begin{array}{l} P (Y = 1 \mid f _ {B} (X, H) = \mathbb {E} [ f _ {B} (X, H) \mid (X, H) \in \lambda \left(b ^ {\prime}\right) ], (X, H) \in \tilde {\mathcal {S}} _ {h ^ {\prime}}) \tag {56} \\ - P (Y = 1 \mid f _ {B} (X, H) = \mathbb {E} [ f _ {B} (X, H) \mid (X, H) \in \lambda (b ^ {\prime \prime}) ], (X, H) \in \tilde {\mathcal {S}} _ {h ^ {\prime \prime}}) \leq \alpha \\ \end{array}
$$
Finally, using Theorems 12 and 13, it readily follows that, given a parameter  $\alpha'$ , the discretized confidence function  $f_{B,\lambda}$  returned by Algorithm 1 satisfies  $(2\alpha' + \lambda)$ -aligned calibration with respect to  $f_{H}$ .
A.6 Proof Theorem 9
We structure the proof in three parts. We first explain the calibration guarantee that UMD provides and how it relates to human-aligned calibration. Then, we derive a lower bound on the size of the subsets  $\mathcal{D} \cap S_h$  so that the discretized confidence function  $f_{B,\lambda}$  satisfies  $\alpha$ -aligned calibration with respect to  $f_H$  with high probability. Finally, building on this result, we derive an upper bound on  $|\mathcal{D}|$  so that  $f_{B,\lambda}$  satisfies  $\alpha$ -aligned calibration with high probability as long as there exists  $\gamma > 0$  so that  $P((X,H) \in S_h) \geq \gamma$  for all  $h \in \mathcal{H}$ .
Conditional Calibration implies Human-Aligned Calibration. Running UMD on a dataset  $\mathcal{D} \in (\mathcal{Z} \times \mathcal{Y})^n$ , where each datapoint is sampled from  $P^{\mathcal{M}}$ , guarantees  $(\alpha, \xi)$ -conditional calibration, a PAC-style calibration guarantee [12]. Given a dataset  $\mathcal{D}$ , a confidence function  $f_{B}$  satisfies  $(\alpha, \xi)$ -conditional calibration if, with probability at least  $1 - \xi$  over the randomness in  $\mathcal{D}$ ,
$$
\forall b \in [ 0, 1 ], \quad | P (Y = 1 | f _ {B} (X, H) = b) - b | \leq \alpha .
$$
This stands in contrast to the definition of  $\alpha$ -calibration, which requires only that the confidence  $f_{B}(X,H)$  is at most  $\alpha$  away from the true probability for  $1 - \alpha$  fraction of  $\mathcal{Z}$ .
Similarly, using an union bound over all  $h \in \mathcal{H}$ ,  $(\alpha / 2, \xi / |\mathcal{H}|)$ -conditional calibration of  $f_{B}$  on each  $S_{h}$ ,  $h \in \mathcal{H}$ , implies that, with probability at least  $1 - \xi$  over the randomness in  $\mathcal{D}$ ,  $f_{B}$  satisfies that
$$
\forall h \in \mathcal {H}, \quad \forall b \in [ 0, 1 ], \quad | P (Y = 1 | f _ {B} (X, H) = b, H = h) - b | \leq \alpha / 2. \tag {57}
$$
Hence, analogously to the proof of Theorem 8, this implies that, with probability at least  $1 - \xi$  over the randomness in  $\mathcal{D}$ ,  $f_{B}$  also satisfies that
$$
\begin{array}{l} \forall h, h ^ {\prime} \in \mathcal {H}, h \leq h ^ {\prime}, \forall b, b ^ {\prime} \in \mathcal {G}, b \leq b ^ {\prime}, \\ P (Y = 1 \mid f _ {B} (X, H) = b, H = h) - P (Y = 1 \mid f _ {B} (X, H) = b ^ {\prime}, H = h ^ {\prime}) \leq \alpha . \tag {58} \\ \end{array}
$$