SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that for bounded  $\mathcal{C}$ , the norm of linear combinations scales gracefully with the sparsity,  $\| \mathrm{Lin}(\mathcal{C}, W) \|_{\infty} \leq W$ . We define  $\mathrm{Lin}(\mathcal{C})$  to be the set of all linear combinations with no restriction on the weights of the coefficients.
Notation. To simplify notation, we use the following shorthand for the data distribution  $\mathcal{D}$  conditioned on  $\tilde{p} (\mathbf{x}) = v$ . For each  $v\in \mathrm{Im}(\tilde{p})$ , let  $\mathcal{D}|_v$  denote the conditional distribution  $\mathcal{D}|\tilde{p} (\mathbf{x}) = v$ . Combined with earlier notation, the distribution  $\mathcal{D}|_{\mathbf{v}}$  for  $\mathbf{v}\sim \mathcal{D}_{\tilde{p}}$  is simply the data distribution  $\mathcal{D}$ . We also use the notation  $(\tilde{p} (\mathbf{x}),\mathbf{y}) = (v,y)$  to indicate  $\tilde{p} (\mathbf{x}) = v$  and  $\mathbf{y} = y$ .
3.1 Statement of Main Result
Theorem 3.3. Let  $\tilde{p}$  be a predictor,  $\mathcal{C}$  be a bounded hypothesis class, and  $\mathcal{L}_{\mathrm{cvx}}(B)$  be the class of  $B$ -nice convex loss functions. The following properties are equivalent:
1.  $\tilde{p}$  is  $(\mathcal{C},\alpha_{1})$  -swap multicalibrated.
2.  $\tilde{p}$  is an  $(\mathcal{L}_{\mathrm{cvx}}(B),\operatorname {Lin}(\mathcal{C},W),O((W + B)\alpha_2))$  -swap omnipredictor, for all  $W\geq 1,B\geq 0$
3.  $\tilde{p}$  is an  $(\ell_2,\mathrm{Lin}(\mathcal{C},2),\alpha_3)$ -swap agnostic learner.
In preparation for proving the theorem, we establish some preliminary results. We define a function  $\alpha : \operatorname{Im}(\tilde{p}) \to [-1, 1]$  which measures the maximum correlation between  $c \in \mathcal{C}$  and  $\mathbf{y} - v$ , conditioned on a prediction value  $v \in \operatorname{Im}(\tilde{p})$ . Let
$$
\alpha (v) = \left| \max  _ {c _ {v} \in \mathcal {C}} \mathbf {E} _ {\mathcal {D} | _ {v}} \left[ c _ {v} (\mathbf {x}) (\mathbf {y} - v) \right] \right|.
$$
Using this notation,  $(\mathcal{C},\alpha_0)$ -swap multicalibration can be written as
$$
\underset {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} {\mathbf {E}} [ \alpha (\mathbf {v}) ] \leq \alpha_ {0}.
$$
We observe that swap multicalibration is closed under bounded linear combinations of  $\mathcal{C}$ , like with standard multicalibration.
Claim 3.4. For every  $h \in \operatorname{Lin}(\mathcal{C}, W)$  and  $v \in \operatorname{Im}(\tilde{p})$ , we have
$$
\max _ {h \in \operatorname {L i n} (\mathcal {C}, W)} \left| \mathbf {E} _ {\mathcal {D}} [ h (\mathbf {x}) (\mathbf {y} - v) | \tilde {p} (\mathbf {x}) = v ] \right| \leq W \alpha (v).
$$
Let  $p_v^* = \mathbf{E}[\mathbf{y}|\tilde{p} (\mathbf{x}) = v]$ . Then
$$
\left| p _ {v} ^ {*} - v \right| \leq \alpha (v). \tag {8}
$$
Equation (8) follows since  $1 \in \mathcal{C}$ .
Claim 3.5. For  $h \in \operatorname{Lin}(\mathcal{C}, w)$ ,  $v \in \operatorname{Im}(\tilde{p})$  and  $y \in \{0,1\}$ , define the following conditional expectations:
$$
\mu (h: v) = \mathbf {E} [ h (\mathbf {x}) | \tilde {p} (\mathbf {x}) = v ]
$$
$$
\mu (h: v, y) = \mathbf {E} [ h (\mathbf {x}) | (\tilde {p} (\mathbf {x}), \mathbf {y}) = (v, y) ].
$$
Then for each  $y \in \{0,1\}$
$$
\Pr [ \mathbf {y} = y | \tilde {p} (\mathbf {x}) = v ] \left| \mu (h: v, y) - \mu (h: v) \right| \leq (W + 1) \alpha (v). \tag {9}
$$
Next we show the following lemma, which shows that one can replace  $h(\mathbf{x})$  by the constant  $\Pi_{\ell}(\mu(h : v))$  without a large increase in the loss.
Lemma 3.6. For all  $h \in \operatorname{Lin}(\mathcal{C}, W)$ ,  $v \in \operatorname{Im}(\tilde{p})$  and loss  $\ell \in \mathcal{L}_{\mathrm{cvx}}(B)$ , we have
$$
\underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ \ell (\mathbf {y}, \Pi_ {\ell} (\mu (h: v)) \right] \leq \underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ \ell (\mathbf {y}, h (\mathbf {x})) \right] + 2 (W + 1) \alpha (v). \tag {10}
$$
4When we say these conditions are equivalent, we mean that they imply each other with parameters  $\alpha_{i}$  that are polynomially related. The relations we derive result in at most a quadratic loss in parameters.
In the interest of space, we defer the proof of the Lemma to the Appendix.
Next we compare  $\Pi_{\ell}(\mu (h:v))$  with  $k_{\ell}(v)$ . It is clear that the latter is better for minimizing loss when  $\mathbf{y}\sim \mathrm{Ber}(v)$ , by definition. We need to compare the losses when  $\mathbf{y}\sim \mathrm{Ber}(p_v^*)$ . But  $p_v^*$  and  $v$  are at most  $\alpha (v)$  apart by Equation (8). Hence, by using Lipschitzness, one can infer that  $k_{\ell}(v)$  is better than  $\Pi_{\ell}(\mu (h:v))$  and hence  $h(\mathbf{x})$ . This is formalized in the following lemma and its proof.
Lemma 3.7. For all  $v \in \operatorname{Im}(\tilde{p})$ ,  $\ell \in \mathcal{L}_{\mathrm{cVX}}(B)$  and  $h \in \operatorname{Lin}(\mathcal{C}, W)$ , we have
$$
\underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ \ell (\mathbf {y}, k _ {\ell} (\tilde {p} (\mathbf {x}))) \right] \leq \underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ \ell (\mathbf {y}, h (\mathbf {x})) \right] + 2 (W + B + 1) \alpha (v). \tag {11}
$$
Proof. By the definition of  $k_{\ell}, k_{\ell}(v)$  minimizes expected loss when  $\mathbf{y} \sim \mathrm{Ber}(v)$ , so
$$
\ell (v, k _ {\ell} (v)) \leq \ell (v, \Pi_ {\ell} (\mu (h: v)) \tag {12}
$$
$$
\underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ \ell (\mathbf {y}, t) \right] = \ell \left(p _ {v} ^ {*}, t\right), \text {w h e r e} p _ {v} ^ {*} = \underset {\mathcal {D} | _ {v}} {\mathbf {E}} [ \mathbf {y} ].
$$
Thus our goal is to compare the losses  $\ell(p_v^*, t)$  for  $t = k_\ell(v)$  and  $t = \Pi_\ell(\mu(h : v))$ . Hence, applying Lemma 3.2 gives
$$
\begin{array}{l} \ell \left(p _ {v} ^ {*}, k _ {\ell} (v)\right) \leq \ell (v, k _ {\ell} (v)) + \alpha (v) B \\ - \ell \left(p _ {v} ^ {*}, \Pi_ {\ell} (\mu (h: v)) \leq - \ell (v, \Pi_ {\ell} (\mu (h: v)) + \alpha (v) B \right. \\ \end{array}
$$
Subtracting these inequalities and then using Equation (12) gives
$$
\begin{array}{l} \ell \left(p _ {v} ^ {*}, k _ {\ell} (v)\right) - \ell \left(p _ {v} ^ {*}, \Pi_ {\ell} (\mu (h: v)) \leq \ell (v, k _ {\ell} (v)) - \ell (v, \Pi_ {\ell} (\mu (h: v)) + 2 B \alpha (v) \right. (13) \\ \leq 2 \alpha (v) B. (14) \\ \end{array}
$$
$$
\begin{array}{l} \mathbf {E} _ {\mathcal {D} | _ {v}} \left[ \ell (\mathbf {y}, k _ {\ell} (v)) \right] = \ell \left(p _ {v} ^ {*}, k _ {\ell} (v)\right) \\ \leq \ell \left(p _ {v} ^ {*}, \Pi_ {\ell} (\mu (h: v)) + 2 \alpha (v) B \right. \tag {byEquation(14)} \\ = \underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ \ell (\mathbf {y}, \Pi_ {\ell} (\mu (h: v)) \right] + 2 \alpha (v) B \\ \leq \underset {\mathcal {D} | _ {v}} {\mathbf {E}} [ \ell (\mathbf {y}, h (\mathbf {x})) ] + 2 (W + 1) \alpha (v) + 2 B \alpha (v). \quad \text {(b y)} \\ \end{array}
$$
We now complete the proof of Theorem 3.3.
Proof of Theorem 3.3. (1)  $\Longrightarrow$  (2) Fix a  $(\mathcal{C},\alpha_{1})$ -swap multicalibrated predictor  $\tilde{p}$ . Fix a choice of loss functions  $\{\ell_v\in \mathcal{L}_{\mathrm{cvx}}\}_{v\in \operatorname {Im}(f)}$  and hypotheses  $\{h_v\in \mathcal{H}\}_{v\in \operatorname {Im}(f)}$ . For each  $v$ , we apply Equation (11) with the loss  $\ell = \ell_{v}$ , hypothesis  $h = h_{v}$  to get
$$
\underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ \ell_ {v} (\mathbf {y}, k _ {\ell_ {v}} (v)) \right] \leq \underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ \ell_ {v} (\mathbf {y}, h _ {v} (\mathbf {x})) \right] + 2 (W + B + 1) \alpha (v).
$$
We now take expectations over  $\mathbf{v} \sim \mathcal{D}_{\tilde{p}}$ , and use the  $\mathbf{E}[\alpha(\mathbf{v})] \leq \alpha_1$  to derive the desired implication. (2)  $\Rightarrow$  (3) with  $\alpha_3 = 7\alpha_2$  because  $\ell_2$  is a  $1/2$ -nice loss function, so we plug in  $B = 1/2$  and  $W = 2$  into claim (2).
It remains to prove that (3)  $\Longrightarrow$  (1). We show the contrapositive, that if  $\tilde{p}$  is not  $\alpha_{1}$  multicalibrated, then  $f = \tilde{p}$  is not a  $(\ell_2,\mathrm{Lin}(\mathcal{C},2),\alpha_3)$ -swap agnostic learner. By the definition of multicalibration, for every  $v\in \operatorname {Im}(\tilde{p})$ , there exist  $c_{v}$  such that
$$
\begin{array}{l} \underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ c _ {v} (\mathbf {x}) (\mathbf {y} - \tilde {p} (\mathbf {x})) \right] = \alpha (v) \\ \mathbf {E} _ {\mathcal {D} _ {\bar {p}}} [ \alpha (\mathbf {v}) ] \geq \alpha_ {1}. \\ \end{array}
$$
By negating  $c_v$  if needed, we may assume  $\alpha(v) \geq 0$  for all  $v$ . We now define the updated hypothesis  $h'$  where
$$
h ^ {\prime} (x) = v + \alpha (v) c _ {v} (x) \text {f o r} x \in \tilde {p} ^ {- 1} (v)
$$
Algorithm 1 Swap Agnostic Learning via MCBoost
Parameters: loss  $\ell$ , hypothesis class  $\mathcal{C}$ , and  $\varepsilon > 0$ , let  $\alpha = \mathrm{poly}(\varepsilon)$
Given: Dataset  $S$  sampled from  $\mathcal{D}$
Run:
$$
\tilde {p} \leftarrow \operatorname {M C B o o s t} _ {\mathcal {C}, \alpha} (S)
$$
$$
h _ {\ell} \leftarrow k _ {\ell} \circ \tilde {p}
$$
Return:  $h_\ell$
A standard calculation (included in the Appendix) shows that
$$
\underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ (\mathbf {y} - v) ^ {2} \right] - \underset {\mathcal {D} | _ {v}} {\mathbf {E}} \left[ (\mathbf {y} - h ^ {\prime} (\mathbf {x})) ^ {2} \right] \geq \alpha (v) ^ {2}.
$$
Taking expectation over  $\mathbf{v}\sim \mathcal{D}_{\hat{p}}$  , we have
$$
\left. \mathbf {E} _ {\mathcal {D} _ {\bar {p}}} \left(\mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} [ (\mathbf {y} - v) ^ {2} ] - \mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} [ (\mathbf {y} - h ^ {\prime} (\mathbf {x})) ^ {2} ]\right) \geq \mathbf {E} _ {\mathcal {D} _ {\bar {p}}} [ \alpha (\mathbf {v}) ^ {2} ] \geq \mathbf {E} _ {\mathcal {D} _ {\bar {p}}} [ \alpha (\mathbf {v}) ] ^ {2} \geq \alpha_ {1} ^ {2}. \right.
$$
It remains to show that  $v + \alpha_v c_v(x) \in \mathrm{Lin}(\mathcal{C}, 2)$ . Note that
$$
\alpha_ {v} = \underset {\mathcal {D} | _ {v}} {\mathbf {E}} [ c (\mathbf {x}) (\mathbf {y} - v) v ] \leq \max  | c (\mathbf {x}) | \max  (| y - \tilde {p} (x) | \leq 1
$$
since  $c(\mathbf{x}),y - v\in [-1,1]$ . Hence  $h^\prime (x) = w_1\cdot 1 + w_2c(v)$  where  $|w_{1}| + |w_{2}|\leq 2$ . This contradicts the definition of an  $(\ell_2,\mathrm{Lin}(\mathcal{C},2),\alpha_3)$ -swap agnostic learner if  $\alpha_{3} < \alpha_{1}^{2}$ .
3.2 An algorithm for Swap Agnostic Learning
The equivalence from Theorem 3.3 suggests an immediate strategy for obtaining a swap agnostic learner. First, learn a swap multicalibrated predictor; then, return the predictor, post-processed to an optimal hypothesis according to  $\ell$ . While the MCBoost algorithm of [20] was designed to guarantee multicalibration, we observe that, in fact, it actually guarantees swap multicalibration.
Lemma 3.8. Suppose the collection  $\mathcal{C}$  has a weak agnostic learner, WAL. For any  $\alpha >0$ , MCBoost makes at most poly  $(1 / \alpha)$  calls to  $\mathrm{WAL}_{\alpha}$ , and returns a  $(\mathcal{C},\alpha)$ -swap multicalibrated predictor.
Weak agnostic learning is a basic supervised learning primitive used in boosting algorithms. Through the connection to boosting, weak agnostic learning is polynomial-time equivalent to agnostic learning, and inherits its data-efficiency (scaling with the VC-dimension of  $\mathcal{C}$ ), but also its worst-case computational hardness [27]. Importantly, however, MCBoost reduces the problem of swap multicalibration (and thus, swap agnostic learning) to a standard agnostic learning task. We review MCBoost weak agnostic learning formally in the Appendix.
In all, we can combine the MCBoost algorithm for a class  $\mathcal{C}$  with a specific loss  $\ell$  to obtain a  $(\ell, \mathcal{C})$ -swap agnostic learner.
Corollary 3.9 (Informal). For any (nice) convex loss  $\ell$ , hypothesis class  $\mathcal{C}$ , and  $\varepsilon > 0$ , Algorithm 1 returns a  $(\ell, \mathcal{C}, \varepsilon)$ -swap agnostic learner from a sample of  $m \leq \mathrm{VC}(\mathcal{C}) \cdot \mathrm{poly}(1 / \varepsilon)$  data points drawn from  $\mathcal{D}$ , after making  $\leq \mathrm{poly}(1 / \varepsilon)$  calls to weak agnostic learner for  $\mathcal{C}$ .
4 Beyond Swap Agnostic Learning: Swap Loss Outcome Indistinguishability
In this we introduce a unified notion of Swap Loss Outcome Indistinguishability, which captures all of the other notions of muticalibration and omniprediction defined so far. The notion builds on a line of work due to [6, 7], which propose the notion of Outcome Indistinguishability (OI) as a solution concept for supervised learning based on computational indistinguishability. In fact, the main result of [6] is an equivalence between OI and muticalibration. Despite the fact that OI is really multicalibration in disguise, the perspective has proved to be a useful technical perspective.
Key to this section is the prior work of [14]. This work proposes a new variant of OI, called Loss OI. The main result of [14] derives novel omniprediction guarantees from loss OI. Further, they show how to achieve loss OI using only calibration and multiaccuracy over a class of functions derived from the loss class  $\mathcal{L}$  and hypothesis class  $\mathcal{C}$ . As we'll see, this class plays a role in the study of swap loss OI: swap loss OI is equivalent to multicalibration over the augmented class.
Additional Preliminaries. Intuitively, OI requires that outcomes sampled from the predictive model  $\tilde{p}$  are indistinguishable from Nature's outcomes. Formally, we use  $(\mathbf{x},\mathbf{y}^{*})$  to denote a sample from the true joint distribution over  $\mathcal{X}\times \{0,1\}$ . Then, given a predictor  $\tilde{p}$ , we associate it with the random variable with  $\mathbf{E}[\tilde{\mathbf{y}} |x] = \tilde{p} (x)$ , i.e., where  $\tilde{\mathbf{y}} |x\sim \mathrm{Ber}(\tilde{p} (x))$ . The variable  $\tilde{\mathbf{y}}$  can be viewed as  $\tilde{p}$ 's simulation of Nature's label  $\mathbf{y}^*$ . In this section, we use  $\mathcal{D}$  to denote the joint distribution  $(\mathbf{x},\mathbf{y}^{*},\tilde{\mathbf{y}})$ , where  $\mathbf{E}[\mathbf{y}^{*}|x] = p^{*}(x)$  and  $\mathbf{E}[\tilde{\mathbf{y}} |x] = \tilde{p} (x)$ . While the joint distribution of  $(\mathbf{y}^{*},\tilde{\mathbf{y}})$  is not important to us, for simplicity we assume they are independent given  $\mathbf{x} = x$ .
4.1 Swap Loss OI
The notion of loss outcome indistinguishability was introduced in the recent work of [14] with the motivation of understanding omniprediction from the perspective of outcome indistinguishability [6]. Loss OI gives a strengthening of omniprediction. It requires predictors  $\tilde{p}$  to fool a family  $\mathcal{U}$  of statistical tests  $u: \mathcal{X} \times [0,1] \times \{0,1\}$  that take a point  $\mathbf{x} \in \mathcal{X}$ , a prediction  $\tilde{p}(\mathbf{x}) \in [0,1]$  and a label  $\mathbf{y} \in \{0,1\}$  as their arguments. The goal is distinguish between the scenarios where  $\mathbf{y} = \mathbf{y}^*$  is generated by nature versus where  $\mathbf{y} = \tilde{\mathbf{y}}$  is a simulation of nature according to the predictor  $\tilde{p}$ . Formally, we require that for every  $u \in \mathcal{U}$ ,
$$
\mathbf {\Delta} _ {\mathcal {D}} [ u (\mathbf {x}, \tilde {p} (\mathbf {x}), \mathbf {y} ^ {*}) ] \approx_ {\varepsilon} \mathbf {\Delta} _ {\mathcal {D}} [ u (\mathbf {x}, \tilde {p} (\mathbf {x}), \tilde {\mathbf {y}}) ].
$$
Loss OI specializes this to a specific family of tests arising in the analysis of omnipredictors.
Definition 4.1 (Loss OI, [14]). For a collection of loss functions  $\mathcal{L}$ , hypothesis class  $\mathcal{C}$ , and  $\varepsilon \geq 0$ , define the family of tests  $\mathcal{U}(\mathcal{L},\mathcal{C}) = \{u_{\ell ,c}\}_{\ell \in \mathcal{L},c\in \mathcal{C}}$  where
$$
u _ {\ell , c} (x, v, y) = \ell (y, k _ {\ell} (v)) - \ell (y, c (x)). \tag {15}
$$
A predictor  $\tilde{p}:\mathcal{X}\to [0,1]$  is  $(\mathcal{L},\mathcal{C},\varepsilon)$ -loss  $OI$  if for every  $u\in \mathcal{U}(\mathcal{L},\mathcal{C})$ , it holds that
$$
\left| \underset {(\mathbf {x}, \mathbf {y} ^ {*}) \sim \mathcal {D}} {\mathbf {E}} [ u (\mathbf {x}, \tilde {p} (\mathbf {x}), \mathbf {y} ^ {*}) ] - \underset {(\mathbf {x}, \tilde {\mathbf {y}}) \sim \mathcal {D} (\tilde {p})} {\mathbf {E}} [ u (\mathbf {x}, \tilde {p} (\mathbf {x}), \tilde {\mathbf {y}}) ] \right| \leq \varepsilon . \tag {16}
$$
[14] show that loss-OI implies omniprediction.
Lemma 4.2 (Proposition 4.5, [14]). If the predictor  $\tilde{p}$  is  $(\mathcal{L},\mathcal{C},\varepsilon)$ -loss  $OI$ , then it is an  $(\mathcal{L},\mathcal{C},\varepsilon)$ -omnipredictor.
Indeed, if the expected value of  $u$  is nonpositive for all  $u \in \mathcal{U}(\mathcal{L},\mathcal{C})$ , then  $\tilde{p}$  must achieve loss competitive with all  $c \in \mathcal{C}$ . The argument leverages the fact that  $u$  must be nonpositive when  $\tilde{\mathbf{y}} \sim \mathrm{Ber}(\tilde{p} (\mathbf{x}))$  after all, in this world  $\tilde{p}$  is the Bayes optimal. By indistinguishability,  $\tilde{p}$  must also be optimal in the world where outcomes are drawn as  $\mathbf{y}^*$ . The converse, however, is not always true.
Next, we introduce swap loss OI, which allows the choice of distinguisher to depend on the predicted value.
Definition 4.3 (Swap Loss OI). For a collection of loss functions  $\mathcal{L}$ , hypothesis class  $\mathcal{C}$  and  $\varepsilon \geq 0$ , for an assignment of loss functions  $\{\ell_v \in \mathcal{L}\}_{v \in \mathrm{Im}(\tilde{p})}$  and hypotheses  $\{h_v \in \mathcal{H}\}_{v \in \mathrm{Im}(\tilde{p})}$ , denote  $u_v = u_{\ell_v, c_v} \in \mathcal{U}(\mathcal{L}, \mathcal{C})$ . A predictor  $\tilde{p}$  is  $(\mathcal{L}, \mathcal{C}, \alpha)$ -swap loss OI if for all such assignments,
$$
\left. \right. \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left| \mathbf {E} _ {\mathcal {D} | _ {\mathbf {v}}} \left[ u _ {\mathbf {v}} (\mathbf {x}, \mathbf {v}, \mathbf {y} ^ {*}) - u _ {\mathbf {v}} (\mathbf {x}, \mathbf {v}, \tilde {\mathbf {y}}) \right]\right| \leq \alpha .
$$
The notion generalizes both swap omniprediction and loss-OI simultaneously.
Lemma 4.4. If the predictor  $\tilde{p}$  satisfies  $(\mathcal{L},\mathcal{C},\alpha)$ -swap loss  $OI$ , then
- it is an  $(\mathcal{L},\mathcal{C},\alpha)$ -swap omnipredictor.
- it is  $(\mathcal{L},\mathcal{C},\alpha)$ -loss  $OI$ .
This Section continues as Section A in the Appendix, where we show the following characterization of loss  $\mathrm{OI}$ .
Theorem 4.5. Let  $\mathcal{L}$  be a family of nice loss functions containing the squared loss. For any hypothesis class  $\mathcal{C}$ , a predictor satisfies  $(\mathcal{L},\mathcal{C})$ -swap loss  $OI$  if and only if it is  $(\partial \mathcal{L}\circ \mathcal{C})$ -swap multicalibrated, where  $\partial \mathcal{L}\circ \mathcal{C} = \{\partial \ell \circ c\}_{\ell \in \mathcal{L},c\in \mathcal{C}}$ .
5 Related Work and Discussion
Independent concurrent work. The notion of strict multicalibration was defined in independent work of [8]. They connect this and other multigroup fairness definitions to various versions of the Szemeredi regularity lemma [28] and its weaker version due to Frieze and Kannan [11], following [29]. Their notions bears important similarities to swap multicalibration, but it is different. Like swap multicalibration, strict multicalibration involves switching the order of expectations and max. But they require a statistical closeness guarantee, whereas we only require a first order guarantee, that  $c(\mathbf{x})$  be uncorrelated with  $\mathbf{y} - \mathbf{v}$  conditioned on  $\mathbf{v}$ .
The independent work of [13] relates multicalibration to real-valued boosting or minimize  $\ell_2$  loss. The implication that  $(\ell_2, \mathcal{C})$  swap-agnostic learning implies multicalibration follows from Part(1) of Theorem 3.2 in their paper. They prove that a violation of multicalibration leads to a better strategy for the adversary in the  $(\ell_2, \mathcal{C})$ -swap minimization game. They do not consider the notion of swap multicalibration, so their result is not a tight characterization unlike ours.
Multi-group fairness and regret minimization. Notions of multi-group fairness were introduced in the work of [20, 24] and [22], following [26]. The flagship notion of multicalibration has been extended to several other settings including multiclass predictions [7], real-valued labels [18, 21], online learning [18] and importance weights [17]. Alternate definitions and extensions of the standard notions of multicalibration have been proposed in [15, 16, 5, 8]. Multicalibration has also proved to have unexpected connections to many other domains, including computational indistinguishability [6], domain adaptation [25], and boosting [15, 13].
Our notions of swap loss minimization and swap omniprediction are inspired by notions of swap regret minimization in online learning [10, 9, 2]. The classic results of [10, 9] relate calibration and swap regret, whereas [2] show a generic way to convert a low external regret algorithm to ones with low internal and swap regret. The study of online learning and regret minimization is extensive, with deep connections to calibration and equilibria in game theory, we refer the readers to [4, 3] for comprehensive surveys. Recent work has established rich connections between online learning and regret minimization on one hand, and multigroup fairness notions on the other. Multicalibration has been considered in the online setting by [18], while [1, 12] relate multigroup fairness to questions in online learning.
Conclusion. Our work adds a new and significant connection in the theory of agnostic learning and multi-group fairness. This theoretical work relies on a few basic assumptions, including access to a representative unbiased data source and a weak agnostic learner for the collection  $\mathcal{C}$ . These assumptions, while standard in the supervised learning literature, should be interrogated before applying mutlicalibration as a notion of fairness in a practical setting.
Perhaps the most interesting aspect of our work is the connections it draws across different areas of learning theory. In particular, by borrowing the notion of swapping from online learning, we uncover surprisingly-powerful, but feasible solution concepts for supervised learning. The equivalence we draw between swap agnostic learning for the squared error and multicalibration with swap omniprediction for all convex losses highlights the power of simple goals like squared error minimization and calibration.
Our work gives a complete characterization of the notions we study at the "upper end" of strength (i.e., swap variants). A fascinating outstanding question to address in future research is whether there is a similar characterization of standard omniprediction in terms of multi-group fairness notions.
Acknowledgements. Omer Reingold is supported by the Simons Foundation Collaboration on the Theory of Algorithmic Fairness, the Simons Foundation investigator award 689988 and Sloan Foundation grant 2020-13941.