SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[20] Ursula Hébert-Johnson, Michael P. Kim, Omer Reingold, and Guy N. Rothblum. Multicalibration: Calibration for the (computationally-identifiable) masses. In Proceedings of the 35th International Conference on Machine Learning, ICML, 2018.
[21] Christopher Jung, Changhwa Lee, Mallesh M. Pai, Aaron Roth, and Rakesh Vohra. Moment multicalibration for uncertainty estimation. In Conference on Learning Theory, COLT 2021, 15-19 August 2021, Boulder, Colorado, USA, volume 134 of Proceedings of Machine Learning Research, pages 2634-2678. PMLR, 2021.
[22] Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerry-mandering: Auditing and learning for subgroup fairness. arXiv preprint arXiv:1711.05144, 2017.
[23] Michael J Kearns and Umesh Vazirani. An introduction to computational learning theory. MIT press, 1994.
[24] Michael P. Kim, Amirata Ghorbani, and James Zou. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 247-254, 2019.
[25] Michael P Kim, Christoph Kern, Shafi Goldwasser, Frauke Kreuter, and Omer Reingold. Universal adaptability: Target-independent inference that competes with propensity scoring. Proceedings of the National Academy of Sciences, 119(4), 2022.
[26] Jon M. Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. In 8th Innovations in Theoretical Computer Science Conference, ITCS, 2017.
[27] Krzysztof Pietrzak. Cryptography from learning parity with noise. In SOFSEM, volume 12, pages 99-114. Springer, 2012.
[28] Endre Szemerédi. Regular partitions of graphs. Technical report, Stanford Univ Calif Dept of Computer Science, 1975.
[29] Luca Trevisan, Madhur Tulsiani, and Salil Vadhan. Regularity, boosting, and efficiently simulating every high-entropy distribution. In 2009 24th Annual IEEE Conference on Computational Complexity, pages 126–136. IEEE, 2009.
[30] Leslie G Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134-1142, 1984.
A Equivalence of swap loss OI and swap multicalibration over augmented class
We show that  $(\mathcal{L},\mathcal{C})$ -swap loss OI and  $(\partial \mathcal{L}\circ \mathcal{C})$ -swap multicalibration are equivalent for nice loss functions.
Theorem A.1 (Formal statement of thm:swap-eq-inf). Let  $\mathcal{L} \subseteq \mathcal{L}(B)$  be a family of  $B$ -nice loss functions such that  $\ell_2 \in \mathcal{L}$ . Then  $(\partial \mathcal{L} \circ \mathcal{C}, \alpha_1)$ -swap multicalibration and  $(\mathcal{L}, \mathcal{C}, \alpha_2)$ -swap loss  $OI$  are equivalent.<sup>5</sup>
In preparation for this, we first finish the proof of Lemma 4.4.
Proof of Lemma 4.4. The proof of Part (1) follows the proof of [14, Proposition 4.5], showing that loss OI implies omniprediction. By the definition of  $k_{\ell_v}$ , for every  $x \in \mathcal{X}$  such that  $\tilde{p}(x) = v$
$$
\begin{array}{l} \underset {\tilde {\mathbf {y}} \sim \operatorname {B e r} (v)} {\mathbf {E}} u _ {v} (x, v, \tilde {\mathbf {y}}) = \underset {\tilde {\mathbf {y}} \sim \operatorname {B e r} (v)} {\mathbf {E}} \left[ \ell_ {v} (\tilde {\mathbf {y}}, k _ {\ell_ {v}} (v)) - \ell_ {v} (\tilde {\mathbf {y}}, c _ {v} (x)) \right] \\ = \ell_ {v} (v, k _ {\ell_ {v}} (v)) - \ell_ {v} (v, c _ {v} (x)) \\ \leq 0 \\ \end{array}
$$
Hence this also holds in expectation under  $\mathcal{D}|_v$ , which only considers points where  $\tilde{p} (\mathbf{x}) = v$
$$
\underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ u _ {v} (\mathbf {x}, v, \tilde {\mathbf {y}}) \right] \leq 0.
$$
Since  $\tilde{p}$  satisfies swap loss OI, we deduce that
$$
\underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ u _ {\mathbf {v}} \left(\mathbf {x}, v, \mathbf {y} ^ {*}\right) \right] \leq \alpha (v)
$$
Taking expectations over  $\mathbf{v} \sim \mathcal{D}_{\tilde{p}}$  and using the definition of  $u_{v}$ , we get
$$
\begin{array}{l} \underset {\mathbf {v} \sim \mathcal {D} _ {\hat {p}}} {\mathbf {E}} \underset {\mathcal {D} | _ {\mathbf {v}}} {\mathbf {E}} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y} ^ {*}, k _ {\ell_ {\mathbf {v}}} (\mathbf {v})\right) - \ell_ {\mathbf {v}} \left(\mathbf {y} ^ {*}, c _ {\mathbf {v}} (\mathbf {x})\right) \right] = \underset {\mathbf {v} \sim \mathcal {D} _ {\hat {p}}} {\mathbf {E}} \underset {\mathcal {D}} {\mathbf {E}} \left[ u _ {\mathbf {v}} \left(\mathbf {x}, \mathbf {v}, \mathbf {y} ^ {*}\right) \right] \\ \leq \underset {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} {\mathbf {E}} [ \alpha (\mathbf {v}) ] \leq \alpha \\ \end{array}
$$
Rearranging the outer inequality gives
$$
\underset {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} {\mathbf {E}} \underset {\mathbf {D} | _ {\mathbf {v}}} {\mathbf {E}} \left[ \ell_ {\mathbf {v}} \left(\tilde {\mathbf {y}}, k _ {\ell_ {\mathbf {v}}} (\mathbf {v})\right) \right] \leq \underset {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} {\mathbf {E}} \underset {\mathbf {D} | _ {\mathbf {v}}} {\mathbf {E}} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y} ^ {*}, c _ {\mathbf {v}} (\mathbf {x})\right) \right] + \alpha .
$$
Part (2) is implied by taking  $\ell_v = \ell$  for every  $v$ .
We use the following simple claim from [14].
Claim A.2 (Lemma 4.8, [14]). For random variables  $\mathbf{y}_1, \mathbf{y}_2 \in \{0,1\}$  and  $t \in \mathbb{R}$ ,
$$
\mathbf {E} [ \ell (\mathbf {y} _ {1}, t) - \ell (\mathbf {y} _ {2}, t) ] = \mathbf {E} [ (\mathbf {y} _ {1} - \mathbf {y} _ {2}) \partial \ell (t) ]. \tag {17}
$$
We record two corollaries of this claim. These can respectively be seen as strengthenings of the two parts of Theorem [14, Theorem 4.9], which respectively characterized hypothesis OI in terms of multiaccuracy and decision OI in terms of calibration. We generalize these to the swap setting.
Corollary A.3. For every choice of  $\{\ell_v, c_v\}_{v \in \mathrm{Im}(\tilde{p})}$ , we have
$$
\left. \right. \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[\left| \mathbf {E} _ {\left| \mathbf {v} \right.} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y} ^ {*}, c _ {\mathbf {v}} (\mathbf {x})\right) - \ell_ {\mathbf {v}} \left(\tilde {\mathbf {y}}, c _ {\mathbf {v}} (\mathbf {x})\right)\right]\right|\right] = \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[\left| \mathbf {E} _ {\left| \mathbf {v} \right.} \left[\left(\mathbf {y} ^ {*} - \tilde {\mathbf {y}}\right) \partial \ell_ {\mathbf {v}} \circ c _ {\mathbf {v}} (\mathbf {x}) \right]\right|\right]. \tag {18}
$$
Hence if  $\tilde{p}$  is  $(\partial \mathcal{L} \circ \mathcal{C}, \alpha)$ -swap multicalibrated, then
$$
\mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\left| \mathbf {v} \right.} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y} ^ {*}, c _ {\mathbf {v}} (\mathbf {x})\right) - \ell_ {\mathbf {v}} \left(\tilde {\mathbf {y}}, c _ {\mathbf {v}} (\mathbf {x})\right) \right] \right| \right] \leq \alpha .
$$
Here equivalence means that there are reductions in either direction that lose a multiplicative factor of  $(B + 1)$  in the error.
Proof. Equation (18) is derived by applying Equation (17) to the LHS. Assuming that  $\tilde{p}$  is  $(\partial \mathcal{L} \circ \mathcal{C}, \alpha)$ -swap multicalibrated, we have
$$
\underset {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} {\mathbf {E}} \left[ \left| \underset {\mathcal {D} | _ {\mathbf {v}}} {\mathbf {E}} \left[ (\mathbf {y} ^ {*} - \tilde {\mathbf {y}}) \partial \ell_ {\mathbf {v}} \circ c _ {\mathbf {v}} (\mathbf {x}) \right] \right| \right] \leq \underset {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} {\mathbf {E}} \left[ \left| \underset {c ^ {\prime} \in \partial \mathcal {L} \circ \mathcal {C}} {\max} \underset {\mathcal {D} | _ {\mathbf {v}}} {\mathbf {E}} \left[ (\mathbf {y} ^ {*} - \tilde {\mathbf {y}}) c ^ {\prime} (\mathbf {x}) \right] \right| \right] \leq \alpha .
$$
Corollary A.4. Let  $\{\ell_v\}_{v\in \mathrm{Im}(f)}$  be a collection of loss  $B$ -nice loss functions. Let  $k(v) = k_{\ell_v}(v)$ . If  $\tilde{p}$  is  $\alpha$ -calibrated then
$$
\left. \right. \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[\left| \mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y} ^ {*}, k (\mathbf {v})\right) - \ell_ {\mathbf {v}} \left(\tilde {\mathbf {y}}, k (\mathbf {v})\right)\right]\right|\right] \leq B \alpha . \tag {19}
$$
$$
\begin{array}{l} \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y} ^ {*}, k (\mathbf {v})\right) - \ell_ {\mathbf {v}} \left(\tilde {\mathbf {y}}, k (\mathbf {v})\right) \right] \right| \right] = \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} \left[ \left(\mathbf {y} ^ {*} - \mathbf {v}\right) \partial \ell_ {\mathbf {v}} \left(k (\mathbf {v})\right) \right] \right| \right] \\ = \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ | \partial \ell_ {\mathbf {v}} (k (\mathbf {v})) | \left| \begin{array}{c} \mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} [ \mathbf {y} ^ {*} - \mathbf {v} ] \end{array} \right| \right] \\ \leq B \underset {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} {\mathbf {E}} \left[ \left| \underset {\mathcal {D} | _ {\mathbf {v}}} {\mathbf {E}} [ \mathbf {y} ^ {*} - \mathbf {v} ] \right| \right] \\ \leq B \alpha . \\ \end{array}
$$
where we use the fact that  $k(v)\in I_{\ell}$  , and so  $|\partial \ell_v(k(v))|\leq B$
Finally, we show the following key technical lemma which explains why the  $\ell_2$  loss has a special role.
Lemma A.5. If  $\tilde{p}$  is  $(\{\ell_2\}, \mathcal{C}, \alpha)$ -swap  $OI$ , then it is  $\alpha$ -calibrated.
Proof. Observe that  $\ell_2(y,v) = (y - v)^2 /2$  so  $k_{\ell_2}(v) = v$ . Hence,
$$
\begin{array}{l} u _ {\ell_ {2}, 0} (x, v, y) = \ell_ {2} (y, k _ {\ell} (v)) - \ell_ {2} (y, 0) \\ = ((y - v) ^ {2} - y ^ {2}) / 2 \\ = - v y + v ^ {2} / 2. \tag {20} \\ \end{array}
$$
Recall that  $\{0,1\} \subset \mathcal{C}$ . The implication of swap loss OI when we take  $c_{v} = 0$  for all  $v$  is that
$$
\left. \right. \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\bar {p}}} \left[\left| \mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} \left[ u _ {\ell_ {2}, 0} (\mathbf {x}, \mathbf {v}, \mathbf {y} ^ {*}) - u _ {\ell_ {2}, 0} (\mathbf {x}, \mathbf {v}, \tilde {\mathbf {y}}) \right]\right|\right] \leq \alpha .
$$
We can simplify the LHS using Equation (20) to derive
$$
\begin{array}{l} \underset {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} {\mathbf {E}} \left[ \left| \underset {\mathcal {D} | _ {\mathbf {v}}} {\mathbf {E}} \left[ (- \mathbf {v} \mathbf {y} ^ {*} + \mathbf {v} ^ {2} / 2) - (- \mathbf {v} \tilde {\mathbf {y}} + \mathbf {v} ^ {2} / 2) \right] \right| \right] = \underset {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} {\mathbf {E}} \left[ \left| \underset {\mathcal {D} | _ {\mathbf {v}}} {\mathbf {E}} \left[ \mathbf {v} (\mathbf {y} ^ {*} - \tilde {\mathbf {y}}) \right] \right| \right] \\ = \underset {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} {\mathbf {E}} \left[ \mathbf {v} \left| \underset {\mathcal {D} | _ {\mathbf {v}}} {\mathbf {E}} \left[ \tilde {\mathbf {y}} - \mathbf {y} ^ {*} \right] \right| \right] \leq \alpha . \tag {21} \\ \end{array}
$$
Considering the case where  $c_v = 1$  for all  $v$  gives
$$
\begin{array}{l} u _ {\ell_ {2}, 1} (x, v, y) = \ell_ {2} (y, k _ {\ell} (v)) - \ell_ {2} (y, 1) \\ = \left(\left(y - v\right) ^ {2} - (1 - y) ^ {2}\right) / 2 \\ = (1 - v) y + (v ^ {2} - 1) / 2. \\ \end{array}
$$
We derive the following implication of swap loss OI by taking  $c_v = 0$  for all  $v$ :
$$
\left. \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\left. \mathcal {D} \right| _ {\mathbf {v}}} \left[ u _ {\ell_ {2}, 1} (\mathbf {x}, \mathbf {v}, \mathbf {y} ^ {*}) - u _ {\ell_ {2}, 1} (\mathbf {x}, \mathbf {v}, \tilde {\mathbf {y}}) \right] \right| \right] = \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ (1 - \mathbf {v}) \left| \mathbf {E} _ {\left. \mathcal {D} \right| _ {\mathbf {v}}} \left[ \tilde {\mathbf {y}} - \mathbf {y} ^ {*} \right] \right| \right] \leq \alpha \right. \tag {22}
$$
Adding the bounds from Equations (21) and (22) we get
$$
\mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\left| \mathcal {D} _ {| \mathbf {v}} \right.} \left[ \mathbf {v} - \mathbf {y} ^ {*} \right] \right| \right] = \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\left| \mathcal {D} _ {| \mathbf {v}} \right.} \left[ \tilde {\mathbf {y}} - \mathbf {y} ^ {*} \right] \right| \right] \leq \alpha
$$
We can now complete the proof of Theorem A.1.
Proof of Theorem A.1. We first show the forward implication, that swap multicalibration implies swap loss OI.
Since  $\ell_2\in \mathcal{L}$  and  $1\in \mathcal{C}$ , we have  $\partial \ell_{2}\circ 1 = 1\in \partial \mathcal{L}\circ \mathcal{C}$ . This implies that  $\tilde{p}$  is  $\alpha$ -multicalibrated, since
$$
\mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\left. \mathcal {D} \right| _ {\mathbf {v}}} [ 1 (\mathbf {y} - \mathbf {v}) ] \right| \right] \leq \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \max  _ {c \in \mathcal {C}} \left| \mathbf {E} _ {\left. \mathcal {D} \right| _ {\mathbf {v}}} [ c (\mathbf {x}) (\mathbf {y} - \mathbf {v}) ] \right| \right] \leq \alpha .
$$
Consider any collection of losses  $\{\ell_v\}_{v\in \mathrm{Im}(\tilde{p})}$ . Applying Corollary A.4, we have
$$
\mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\mathcal {D} | \mathbf {v}} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y} ^ {*}, k (\mathbf {v})\right) - \ell_ {\mathbf {v}} \left(\tilde {\mathbf {y}}, k (\mathbf {v})\right) \right] \right| \right] \leq B \alpha .
$$
On the other hand, by Corollary A.3, we have for every choice of  $\{\ell_v,c_v\}_{v\in \mathrm{Im}(\tilde{p})}$
$$
\mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y} ^ {*}, c _ {\mathbf {v}} (\mathbf {x})\right) - \ell_ {\mathbf {v}} \left(\tilde {\mathbf {y}}, c _ {\mathbf {v}} (\mathbf {x})\right) \right] \right| \right] \leq \alpha .
$$
Hence for any choice of  $\{u_v\}_{v\in \mathrm{Im}(\tilde{p})}$  we can bound
$$
\begin{array}{l} \underset {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} {\mathbf {E}} \left| \underset {\mathcal {D} | _ {\mathbf {v}}} {\mathbf {E}} \left[ u _ {\mathbf {v}} \left(\mathbf {x}, \mathbf {v}, \mathbf {y} ^ {*}\right) - u _ {\mathbf {v}} \left(\mathbf {x}, \mathbf {v}, \tilde {\mathbf {y}}\right) \right] \right| \\ \leq \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} [ \ell_ {\mathbf {v}} (\mathbf {y} ^ {*}, k (\mathbf {v})) - \ell_ {\mathbf {v}} (\tilde {\mathbf {y}}, k (\mathbf {v}) ] \right| + \left| \mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} [ \ell_ {\mathbf {v}} (\mathbf {y} ^ {*}, c _ {\mathbf {v}} (\mathbf {x})) - \ell_ {\mathbf {v}} (\tilde {\mathbf {y}}, c _ {\mathbf {v}} (\mathbf {x})) ] \right| \right] \\ \leq (B + 1) \alpha \\ \end{array}
$$
which shows that  $\tilde{p}$  satisfies swap loss OI with  $\alpha_{2} = (D + 1)\alpha_{1}$ .
Next we show the reverse implication: if  $\tilde{p}$  satisfies  $(\mathcal{L},\mathcal{C},\alpha_2)$ -swap loss OI, then it satisfies  $(\partial \mathcal{L} \circ \mathcal{C},\alpha_1)$ -swap multicalibration. The first step is to observe that by lemma A.5, since  $\ell_2 \in \mathcal{L}$ , the predictor  $\tilde{p}$  is  $\alpha_2$  calibrated. Since any  $\ell \in \mathcal{L}$  is  $B$ -nice, we have
$$
\mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\left| \mathcal {D} \right| _ {\mathbf {v}}} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y} ^ {*}, k (\mathbf {v})\right) - \ell_ {\mathbf {v}} \left(\tilde {\mathbf {y}}, k (\mathbf {v})\right) \right] \right| \right] = \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\left| \mathcal {D} \right| _ {\mathbf {v}}} \left[ \left(\mathbf {y} ^ {*} - \tilde {\mathbf {y}}\right) k (\mathbf {v}) \right] \right| \right] \leq B \alpha_ {2}.
$$
For any  $\{\ell_v, c_v\}_{v \in \operatorname{Im}(f)}$ , since
$$
u _ {v} (x, v, y) = \ell_ {v} (y, k _ {\ell} (v)) + \ell_ {v} (y, c _ {v} (x))
$$
$$
\begin{array}{l} \left. \right. \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[\left| \mathbf {E} _ {\left| \mathbf {v} \right.} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y} ^ {*}, c (\mathbf {x})\right) - \ell_ {\mathbf {v}} \left(\tilde {\mathbf {y}}, c (\mathbf {x})\right)\right]\right|\right] \\ \leq \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} [ u _ {\mathbf {v}} (\mathbf {x}, \mathbf {v}, \mathbf {y} ^ {*}) - u _ {\mathbf {v}} (\mathbf {x}, \mathbf {v}, \tilde {\mathbf {y}}) ] \right| + \left| \mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} [ \ell_ {\mathbf {v}} (\mathbf {y} ^ {*}, k (\mathbf {v})) - \ell_ {\mathbf {v}} (\tilde {\mathbf {y}}, k (\mathbf {v})) ] \right| \right] \\ \leq (B + 1) \alpha_ {2}. \\ \end{array}
$$
But by Equation (18), the LHS can be written as
$$
\mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\left| \mathcal {D} \right| _ {\mathbf {v}}} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y} ^ {*}, c (\mathbf {x})\right) - \ell_ {\mathbf {v}} \left(\tilde {\mathbf {y}}, c (\mathbf {x})\right) \right] \right| \right] = \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} _ {\left| \mathcal {D} \right| _ {\mathbf {v}}} \left[ \partial \ell_ {\mathbf {v}} \circ c _ {\mathbf {v}} (\mathbf {x}) \left(\mathbf {y} ^ {*} - \mathbf {v}\right) \right] \right| \right]
$$
This shows that  $\tilde{p}$  is  $(\partial \mathcal{L} \circ \mathcal{C}, (B + 1)\alpha_{2})$ -swap multicalibrated.
B Relating notions of omniprediction
In this work, we have discussed the four different notions of omniprediction defined to date.
00) Omniprediction, as originally defined by [15].
Figure 1: Relation between notions of omniprediction
01) Loss OI, from [14].
10) Swap omniprediction.
11) Swap Loss OI.
In order to compare them, we can ask which of these notions implies the other for any fixed choice of loss class  $\mathcal{L}$  and hypothesis class  $\mathcal{C}$ .
- Loss OI implies omniprediction by [14, Proposition 4.5].
- Swap omniprediction implies omniprediction by Claim 2.4.
- Swap loss OI implies both loss OI and swap multicalibration by Lemma 4.4.
These relationships are summarized in Figure 1.
Further, this picture captures all the implications that hold for all  $(\mathcal{L},\mathcal{C})$ . Next, we show that for any implication not drawn in the diagram, there exists some (natural) choice of  $(\mathcal{L},\mathcal{C})$ , where the implication does not hold. In particular, we prove that neither loss OI nor swap omniprediction implies the other for all  $(\mathcal{L},\mathcal{C})$ . This separates these notions from swap loss OI, since swap loss OI implies both these notions.<sup>6</sup> By similar reasoning, it separates omniprediction from both these loss OI and swap omnipredicton, since omniprediction is implied by either of them.
Swap omniprediction does not imply loss OI. We prove this non-implication using a counterexample used in [14]. In particular, they show that omniprediction does not imply loss OI [14, Theorem 4.6], and the same example in fact shows that swap omniprediction does not imply loss OI. In their example, we have  $\mathcal{D}$  on  $\{\pm 1\}^3 \times [0,1]$  where the marginal on  $\{\pm 1\}^3$  is uniform, and  $p^*(x) = (1 + x_1x_2x_3)/2$ , whereas  $\tilde{p}(x) = 1/2$  for all  $x$ . We take  $\mathcal{C} = \{1, x_1, x_2, x_3\}$ . Since  $\tilde{p} = 1/2$  is constant, it is easy to check that  $\tilde{p} - p^* = -x_1x_2x_3/2$  is uncorrelated with  $\mathcal{C}$ . Hence  $\tilde{p}$  satisfies swap multicalibration (which is the same as multicalibration or even multiaccuracy in this setting where  $\tilde{p}$  is constant). Hence by Theorem 3.3,  $\tilde{p}$  is an  $(\mathcal{L}_{\mathrm{cvx}}(1), \mathrm{Lin}_{\mathcal{C}}, 0)$ -swap omnipredictor. [14, Theorem 4.6] prove that  $\tilde{p}$  is not loss OI for the  $\ell_4$  loss. Hence we have the following result.
Lemma B.1. The predictor  $\tilde{p}$  is  $(\mathcal{C},0)$ -swap multicalibrated and hence it is a  $(\{\ell_4\}, \operatorname{Lin}(\mathcal{C}), 0)$ -swap omnipredictor. But it is not  $(\{\ell_4\}, \operatorname{Lin}(\mathcal{C}, 1), \varepsilon)$ -loss  $OI$  for  $\varepsilon < 4/9$ .
We remark that the construction extends to all  $\ell_p$  losses for even  $p > 2$ . Hence even for convex losses, the notions of swap omniprediction are loss-OI seem incomparable.
For instance if loss OI implied swap loss OI, it would also imply swap omniprediction, which our claim shows it does not.
x=(x1,x2)	p*(x)	p(x)
(-1,-1)	0	1/8
(+1,-1)	1/4	1/8
(-1,+1)	1	7/8
(+1,+1)	3/4	7/8
Table 1: Separating loss-OI and swap-resilient omniprediction
Loss OI does not imply swap omniprediction. Next we construct an example showing that loss OI need not imply swap omniprediction. We consider the set of all GLM losses defined below, which contain common losses including the squared loss and the logistic loss.
Definition B.2. Let  $g: \mathbb{R} \to \mathbb{R}$  be a convex, differentiable function such that  $[0,1] \subseteq \mathrm{Im}(g')$ . Define its matching loss to be  $\ell_g = g(t) - yt$ . Define  $\mathcal{L}_{\mathrm{GLM}} = \{\ell_g\}$  be the set of all such loss functions.
[14] shows a general decomposition result that reduces achieving loss OI to a calibration condition and a multiaccuracy condition. Whereas arbitrary losses might require multiaccuracy for the more powerful class  $\partial \mathcal{L} \circ \mathcal{C}$ , for  $\mathcal{L}_{\mathrm{GLM}}$ ,  $\partial \mathcal{L}_{\mathrm{GLM}} \circ \mathcal{C} = \mathcal{C}$ . This is formalized in the following result.
Lemma B.3 (Theorem 5.3, [14]). If  $\tilde{p}$  is  $\varepsilon_1$ -calibrated and  $(\mathcal{C},\varepsilon_2)$ -multiaccurate, then it is  $(\mathcal{L}_{\mathrm{GLM}},\mathrm{Lin}(\mathcal{C},W),\delta)$ -loss  $OI$  for  $\delta = \varepsilon_{1} + W\varepsilon_{2}$ .
In light of the above result, it suffices to find a predictor that is calibrated and multiaccurate (and hence satisfies loss OI), but not multicalibrated, hence not swap multicalibrated. By Theorem 3.3 it is not an  $(\{\ell_2\}, \mathrm{Lin}_{\mathcal{C}}, \delta)$ -swap omnipredictor for  $\delta$  less than some constant.
Let us define the predictors  $p^*, \tilde{p} : \{\pm 1\}^2 \to [0,1]$  as below. We use these to show a separation between loss OI and swap omniprediction.
Lemma B.4. Consider the distribution  $\mathcal{D}$  on  $\{\pm 1\}^2 \times \{0,1\}$  where the marginal on  $\{\pm 1\}^2$  is uniform and  $\mathbf{E}[\mathbf{y}|x] = p^{*}(x)$ . Let  $\mathcal{C} = \{1,x_1,x_2\}$ .
1.  $\tilde{p} \in \mathrm{Lin}(\mathcal{C}, 1)$ . Moreover, it minimizes the squared error over all hypotheses from  $\mathrm{Lin}(\mathcal{C})$ .
2.  $\tilde{p}$  is perfectly calibrated and  $(\mathcal{C},0)$ -multiaccurate. So it is  $(\mathcal{L}_{\mathrm{GLM}},\mathrm{Lin}(\mathcal{C}),0)$ -loss  $OI$ .
3.  $\tilde{p}$  is not  $(\mathcal{C},\alpha)$ -multicalibrated for  $\alpha < 1/8$ . It is not  $(\ell_2, \operatorname{Lin}(\mathcal{C}), \delta)$ -swap agnostic learner for  $\delta < 1/64$ .
Proof. We compute Fourier expansions for the two predictors:
$$
p ^ {*} (x) = \frac {1}{8} \left(4 + 3 x _ {2} - x _ {1} x _ {2}\right) \tag {23}
$$
$$
\tilde {p} (x) = \frac {1}{8} (4 + 3 x _ {2}) \tag {24}
$$