SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By construction, every pair  $S'(t), \ell(t)$  is an  $(\varepsilon/8)$  best-response to the cost function  $1 - \mathcal{L}_{D_{(\cdot),(\cdot)}(h^{(t)})}$ . By Lemma 3.4, since the learner has at most  $T\varepsilon/8$  weak regret and the adversary is  $\varepsilon/8$  best-responding, there must exist a timestep  $t \in [T]$  where  $h^{(t)}$  is  $(\varepsilon/4)$ -optimal. By Lemma 3.5, the  $h^{(t^*)}$  found by the algorithm is  $(\varepsilon/2)$ -optimal with probability at least  $1 - \delta$ . By Fact E.2,  $h^{(t^*)}$  is a  $(S, S', \varepsilon, \lambda)$ -conditionally multicalibrated predictor. The oracle complexity is exactly  $T$ , while the sample complexity of oracle  $\mathcal{A}$  is a standard adaptive data analysis result (Lemma A.6).
We can also implement faster non-deterministic conditional multicalibration algorithms.
Theorem E.4. Fix  $\varepsilon > 0$ ,  $\lambda, k \in \mathbb{Z}_{+}$ , two sets of groups  $S, S' \subseteq 2^{\mathcal{X}}$ , and a data distribution  $D$ . The following algorithm returns a non-deterministic  $k$ -class predictor that is  $(S, S', \varepsilon, \lambda)$ -conditionally multicalibrated on  $D$  and takes  $O\left(|S'| (\ln (|S| |S'| / \delta) + k \ln (\lambda)) / \varepsilon^{2}\right)$  samples in total from the distributions in  $\{D_S'\}_{S' \in S'}$ .
Construct the problem  $(\mathcal{D},\mathcal{G}_{\mathrm{mc}},\mathcal{P})$  from Fact E.2 and let  $T = C\varepsilon^{-2}|\mathcal{S}'|\ln(|\mathcal{S}| \lambda^k \delta)$  for some universal constant  $C$ . Over  $T$  rounds, have a learner choose predictors  $h^{(1:T)} \in \mathcal{P}$  by applying the no-regret learning algorithm of Theorem C.6 to the stochastic costs  $\ell^{(1:T)}$  and distributions  $\{D_{S'(t)}\}^{(1:T)}$ . In parallel, have an adversary choose  $S^{(1:T)}, \ell^{(1:T)}$  by applying ELP to costs  $\{1 - \mathcal{L}_{D_{(\cdot),(\cdot)}}(h^{(t)})\}^{(1:T)}$ . Return the predictor  $p = \text{Uniform}(h^{(1:T)})$ . This algorithm is written explicitly in Algorithm 5.
Proof. As we proved in Theorem E.3, Theorem C.6 guarantees that if  $T \geq C |S'| \ln (k) / \varepsilon^2$ , the learner's regret is bounded by  $\operatorname{Reg}_{\text{weak}}(h^{(1:T)}, \{\mathcal{L}_{D_{S'(t)}, \ell^{(t)}}(\cdot)\}^{(1:T)}) \leq T\varepsilon / 4$ . We now turn to the adversary.
We will define a sequence of costs  $\widetilde{c_{\mathrm{adv}}}^{(1:T)}$  that mirrors the adversary's costs  $\{1 - \mathcal{L}_{D_{(\cdot),(\cdot)}}(h^{(t)})\}^{(1:T)}$ . For all timesteps  $t \in [T]$ , rename  $(x_{S'(t)}^{(t)}, y_{S'(t)}^{(t)}) \coloneqq (x^{(t)}, y^{(t)})$  while for all other  $S' \in S'$  define the random samples  $(x_{S'}^{(t)}, y_{S'}^{(t)}) \stackrel{\text{i.i.d.}}{\sim} D_{S'}$  as the data-point that the adversary would have hypothetically sampled if it had chosen to sample from  $D_S'$  instead of  $D_{S'(t)}$  at timestep  $t$ . We now can define  $\widetilde{c_{\mathrm{adv}}}^{(t)}(S', (i, j, S, v)) \coloneqq (0.5 + 0.5 \cdot i \cdot 1[h(x_{S'}^{(t)}) \in v, x_{S'}^{(t)} \in S] \cdot (\delta_{y_{S', j}^{(t)}} - h(x_{S'}^{(t)})_j))$ . Note that, since the ELP algorithm only observes values of  $c_{\mathrm{adv}}^{(t)}((i, j, S, S', v))$  where  $S' = S'(t)$ ,  $\operatorname{Reg}(\{(S'(t), \ell(t))\}^{(1:T)}, c_{\mathrm{adv}}^{(1:T)}) = \operatorname{Reg}(\{(S'(t), \ell(t))\}^{(1:T)}, \widetilde{c_{\mathrm{adv}}}^{(1:T)})$ .
By Lemma A.5, the adversary satisfies  $\mathrm{Reg}(\{(S'(t),\ell^{(t)})\}^{(1:T)},c_{\mathrm{adv}}^{(1:T)})\leq O(\sqrt{|S'| \ln(k / \delta)T})$ . Thus, we also have that  $\mathrm{Reg}(\{(S'(t),\ell^{(t)})\}^{(1:T)},\widetilde{c_{\mathrm{adv}}}^{(1:T)})\leq O(\sqrt{|S'| \ln(k / \delta)T})$ . By Lemma 3.2, there is a universal constant  $C$  such that, if  $T\geq C\varepsilon^{-2}\ln (2k\lambda^k |\mathcal{S}||\mathcal{S}'| / \delta)$
$$
\left. \right.\left| \operatorname {R e g} \left(\left\{\left(S ^ {\prime (t)}, \ell^ {(t)}\right)\right\} ^ {(1: T)}, \widetilde {c _ {\mathrm {a d v}}} ^ {(1: T)}\right) - \operatorname {R e g} \left(\left\{\left(S ^ {\prime (t)}, \ell^ {(t)}\right)\right\} ^ {(1: T)}, \left\{1 - \mathcal {L} _ {D _ {(\cdot), (\cdot)}} \left(h ^ {(t)}\right)\right\} ^ {(1: T)}\right)\right| \leq T \varepsilon / 1 2,
$$
with probability  $1 - \delta$ . Taking a triangle inequality and union bound, we can see that for sufficiently large  $C$ ,  $\operatorname{Reg}(\{(S'(t), \ell(t))\}^{(1:T)}, \{1 - \mathcal{L}_{D_{(\cdot), (\cdot)}}(h(t))\}^{(1:T)}) \leq T\varepsilon / 4$ .
By Lemma 3.3, the ergodic iterate  $h^*$  is an  $(\varepsilon / 2)$ -optimal solution. By Fact E.2,  $h^*$  is a  $(S, S', \varepsilon, \lambda)$ -conditionally multicalibrated predictor. The sample complexity is exactly  $T$  since the algorithm only samples one datapoint at each iteration.
Stronger multicalibration guarantees for (almost) free. We can also drop the assumption of access to the conditional distributions  $\{D_{S'}\}_{S' \in S'}$  and ask for a weaker guarantee than conditional multicalibration where a subgroup's error tolerance scales with  $\sqrt{\operatorname*{Pr}(x \in S')}$ . For simplicity, we will fix  $S' = S$  here. Note that unlike Theorem E.3, Theorem 4.4 does not assume knowledge of the vector  $[\operatorname*{Pr}_D(x \in S)]_{S \in S}$ .
Theorem 4.4. Fix  $\varepsilon >0$ ,  $\lambda, k \in \mathbb{Z}_{+}$ , two sets of groups  $S \subseteq 2^{\mathcal{X}}$ , and a data distribution  $D$ . There is an algorithm that, with probability at least  $1 - \delta$ , takes  $O(\ln (k) \cdot (\ln (|\mathcal{S}| / \varepsilon \delta) + k \ln (\lambda)) / \varepsilon^{4})$  samples from  $D$  and returns a deterministic  $k$ -class predictor  $h$  satisfying
The sample complexity of Theorem 4.4 should be improvable to a sample complexity of  $\widetilde{O}((\sqrt{\ln(k)}\ln(|S| / \delta) + k\ln(\lambda)) / \varepsilon^3)$  using adaptive data analysis [30, 1]. Even without using adaptive data analysis, Theorem E.3 guarantees a sample complexity that is only a  $1 / \varepsilon$  factor greater (less than a cube-root increase) than the best known sample complexity for deterministic multicalibration. In fact, it matches the best sample complexity for deterministic multicalibration that is known to be possible without adaptive data analysis. This means that we can attain this strictly stronger multicalibration guarantee for free (compared to its non-adaptive data analysis counterpart) or with only a minor cube-root increase in sample complexity (compared to its adaptive data analysis counterpart). We defer a complete proof and algorithm statement to Section E.4.
E.2 Agnostic Multicalibration
An implicit assumption of multicalibration is that one has unlimited freedom to vary their predictions based on which subgroups a datapoint belongs to. This assumption arises because we assume that groups are defined as subsets of the domain,  $S \subseteq \mathcal{X}$ , and predictors are allowed to condition arbitrarily on the domain. This is impractical in many settings including, for example, when subgroups correspond to protected demographics. This is an important concern for fairness applications of multicalibration.
One can extend multicalibration to a more general agnostic setting by assuming that group membership is passed to the predictor separately from the covariate. Note that, in this definition, different groups may not share a Bayes classifier and a fully multicalibrated predictor may not exist.
Definition E.5. Consider a data distribution  $D$  supported on  $\mathcal{X} \times \mathcal{X}' \times \mathcal{Y}$  and where the set  $\mathcal{P}$  of  $k$ -class predictors that are functions of the visible covariates  $\mathcal{X}$  but not the protected covariates  $\mathcal{X}'$ .
Fix  $\varepsilon > 0$ ,  $\lambda \in \mathbb{Z}_{+}$ , and a set of groups  $S \subseteq 2^{\mathcal{X}'}$ . A (possibly non-deterministic)  $k$ -class predictor  $p \in \Delta(\mathcal{P})$  is  $(S, \varepsilon, \lambda)$ -agnostic multicalibrated with respect to a baseline  $B$  if
$$
\forall S \in \mathcal {S}, v \in V _ {\lambda} ^ {k}, j \in [ k ]: \left| \mathbb {E} _ {(x, x ^ {\prime}, y) \sim D, h \sim p} \left[ (h (x) _ {j} - \delta_ {y, j}) \cdot 1 [ h (x) \in v, x ^ {\prime} \in S ] \right] \right| \leq \varepsilon + B.
$$
In the original definition of multicalibration, it is implicitly assumed that one knows, a priori, the probability that a covariate  $x$  belongs to a certain group; this is necessary to attain sample complexity rates independent of domain size  $\mathcal{X}$ . We will similarly allow agnostic multicalibration to take for granted knowledge of the group memberships of each covariate  $x \in \mathcal{X}$ .
Choosing a baseline. One challenge with defining agnostic multicalibration is choosing a proper baseline  $B$ . An immediate option is to choose a min-max baseline:
$$
B_{\text{weak}}:= \min_{h^{*}\in \mathcal{H}^{*}}\max_{j,v,S}\left|\underset {(x,x^{\prime},y)\sim D}{\mathbb{E}}\left[(h^{*}(x)_{j} - y_{j})\cdot 1[h^{*}(x)\in v,x^{\prime}\in S]\right]\right|.
$$
However, achieving this baseline is intractable, with a sample and oracle complexity that depends—potentially linearly—on domain size. Moreover, the predictor attaining the min-max baseline is odd and undesirable: when two groups disagree on the label for a set of covariates, the predictor hedges its losses by spreading its predictions for those covariates into as many bins as possible.
Proposition E.6. Define  $\mathcal{X} = [m]$  for some large  $m\in \mathbb{Z}_{+}$  and uniformly sample a random half of  $\mathcal{X}$  as  $\widetilde{\mathcal{X}}\subseteq \mathcal{X}$  where  $|\widetilde{\mathcal{X}} | = |\mathcal{X}| / 2$  . Let  $\mathcal{X}' = \{1,2\}$ $\lambda = 3$  and  $k = 2$  , and define the marginal distributions  $\operatorname *{Pr}_D(x\mid x' = 1)$  and  $\operatorname *{Pr}_D(x\mid x' = 2)$  to be uniform on  $\mathcal{X}$  . For  $x\notin \widetilde{\mathcal{X}}$  , let labels be random:  $\operatorname *{Pr}_D(y = 1\mid x' = 1,x) = \operatorname *{Pr}_D(y = 1\mid x' = 2,x) = 0.5$  . For  $x\in \widetilde{\mathcal{X}}$  , let labels be deterministic:  $\operatorname *{Pr}_D(y = 1\mid x' = 1,x) = 0$  and  $\operatorname *{Pr}_D(y = 1\mid x' = 2,x) = 1$  . Finding a predictor that is  $(S,\varepsilon ,\lambda)$  -agnostic multicalibrated with respect to the min-max baseline requires at least  $\Theta (|x|)$  samples must be taken from  $D$  for  $\varepsilon = 0.1$
Proof. The min-max optimal multicalibrated predictor divides  $\widetilde{\mathcal{X}}$  into 4 equal parts:  $\widetilde{\mathcal{X}}_1, \ldots, \widetilde{\mathcal{X}}_4$ . It predicts  $h(x) = 0.5$  for all  $x \notin \widetilde{\mathcal{X}}$ ,  $x \in \widetilde{\mathcal{X}}_1$  and  $x \in \widetilde{\mathcal{X}}_2$ . It predicts  $h(x) = 1/3$  for all  $x \in \widetilde{\mathcal{X}}_3$ . It predicts  $h(x) = 2/3$  for all  $x \in \widetilde{\mathcal{X}}_4$ . This attains a multicalibration violation of  $1/6$ . Thus, achieving an  $\varepsilon$ -optimal predictor requires determining the membership (in  $\widetilde{\mathcal{X}}$ ) of at least a  $3/4 - \varepsilon$  fraction of  $\mathcal{X}$ .
To provide a more satisfactory definition of agnostic multicalibration, we ask that one's predictor assigns the same label probabilities to datapoints with the same group memberships and relaxes the baseline so that hedging is not necessary.
Definition E.7. We say that a  $k$ -predictor  $h$  is agnostic multicalibrated if the predictor  $h$  provides the same prediction probabilities  $h(x_{1}) = h(x_{2})$  to datapoints  $x_{1}, x_{2} \in \mathcal{X}$  that share the same group membership distributions  $\forall S \in \mathcal{S}: \operatorname*{Pr}(x' \in S \mid x = x_{1}) = \operatorname*{Pr}(x' \in S \mid x = x_{2})$ ; and (2) is  $(S, \varepsilon, \lambda)$ -agnostic multicalibrated with respect to the multi-accuracy baseline:
$$
B _ {\text {m u l t i - a c c}} := \min  _ {h ^ {*} \in \mathcal {H} ^ {*}} \max  _ {j \in [ k ], S \in \mathcal {S}} \left| \underset {(x, x ^ {\prime}, y) \sim D} {\mathbb {E}} \left[ \left(h ^ {*} (x) _ {j} - \delta_ {y, j}\right) \cdot 1 \left[ x ^ {\prime} \in S \right] \right] \right|. \tag {10}
$$
This baseline is zero when the groups share a Bayes classifier, recovering Definition 2.1. With a slight tweak to Fact 2.5, we can write agnostic multicalibration as the following multi-objective learning problem.
Fact E.8. Let  $D$  be a data distribution for some  $k$ -class prediction problem and fix  $\varepsilon > 0$ ,  $\lambda \in \mathbb{Z}_{+}$  and a set of groups  $S \subseteq 2^{\mathcal{X}'}$ . For every protected feature  $\mathbf{x}' \in (\mathcal{X}')^\mathcal{X}$ , direction  $i \in \{\pm 1\}$ , level set  $v \in V_{\lambda}^{k}$ , group  $S \in S$ , and class  $j \in [k]$ , we define an objective  $\ell_{\mathbf{x}', i, j, S, v}: \mathcal{P} \times (\mathcal{X} \times \mathcal{Y}) \to [0,1]$  where
$$
\ell_ {\mathbf {x} ^ {\prime}, i, j, S, v} (h, (x, y)) = 0. 5 + 0. 5 \cdot i \cdot 1 [ h (x) \in v, \mathbf {x} _ {x} ^ {\prime} \in S ] \cdot (h (x) _ {j} - \delta_ {y, j}), \tag {11}
$$
and  $\mathcal{G}_{\mathrm{ag}} = \{\ell_{\mathbf{x}',i,j,S,v}\}_{\mathbf{x}',i,j,S,v}$  is the set of these objectives. We also define objectives without subscript  $\mathbf{x}'$ , where  $\ell_{i,j,S,v}(h,(x,x',y)) = 0.5 + 0.5\cdot i\cdot 1[h(x)\in v,x'\in S]\cdot (h(x)_j - \delta_{y,j})$  and
$\mathcal{G}_{\mathrm{ag}}^{\prime} = \{\ell_{i,j,S,v}\}_{i,j,S,v}$ . For every objective  $\ell \in \mathcal{G}_{\mathrm{ag}}^{\prime}$ , let  $q_{\ell} \in \Delta(\mathcal{G}_{\mathrm{ag}})$  denote the objective mixture where  $\mathbb{E}_{x \sim D, \ell \sim q_{\ell}}[\ell(\cdot)] = \mathbb{E}_{x', x \sim D}[\ell(\cdot)]$ . If a predictor  $p \in \Delta(\mathcal{P})$  is a  $\varepsilon$ -optimal solution to the multi-objective learning problem ( $\{D\}, \mathcal{G}_{\mathrm{ag}}^{\prime}, \mathcal{P}$ ) with respect to the multi-accuracy baseline, then  $p$  is  $(S, S', 2\varepsilon, \lambda)$ -agnostic multicalibrated for  $D$  with respect to the multi-accuracy baseline.
A simple modification to our batch multicalibration algorithm suffices to achieve agnostic multicalibration.
Theorem E.9. Fix  $\varepsilon > 0$ ,  $\lambda, k \in \mathbb{Z}_{+}$ , a set of groups  $S \subseteq 2^{\mathcal{X}'}$ , and a data distribution  $D$ . The following algorithm returns a deterministic  $k$ -class predictor that is  $(S, \varepsilon, \lambda)$ -agnostic multicalibrated on  $D$  with respect to the multi-accuracy baseline. It makes  $O(\ln(k) / \varepsilon^{2})$  calls to an agnostic learning oracle and, with probability  $1 - \delta$ , these oracle calls can be implemented with  $\widetilde{O}\left(\frac{1}{\varepsilon^{3}} (\sqrt{\ln(k)} \ln(k | S| / \delta) + k \ln(\lambda))\right)$  samples from  $D$ .
Construct the problem  $(\{D\}, \mathcal{G}_{\mathrm{ag}}, \mathcal{P})$  from Fact 2.5 and let  $T = C\varepsilon^{-2}\ln(|S| \lambda^k \delta)$  for some universal constant  $C$ . Over  $T$  rounds, have an adversary choose  $\ell^{(1:T)} \in \mathcal{G}_{\mathrm{ag}}'$  by calling an agnostic learning oracle at each  $t \in [T]$ :  $\ell^{(t)} = \mathcal{A}_{\varepsilon/8}(1 - \mathcal{L}_{D,(\cdot)}(h^{(t)}), \mathcal{G}_{\mathrm{ag}}')$ . Have a learner choose predictors  $h^{(1:T)} \in \mathcal{P}$  by applying the no-regret learning algorithm of Theorem C.6 to the stochastic costs  $\{q_{\ell^{(t)}}\}^{(1:T)} \in \Delta(\mathcal{G}_{\mathrm{ag}})$ . Using  $C\ln(T/\delta)/\varepsilon^2$  samples from  $D$ , return the predictor  $h^{(t^*)}$  with the lowest empirical multicalibration error. This algorithm is written explicitly in Algorithm 6.
Proof. By Theorem C.6, if  $T \geq 256\varepsilon^{-2}\ln (k)$ , the predictors  $h^{(1:T)}$  satisfy  $\operatorname{Reg}_{B^*}(h^{(1:T)},\{q_{\ell(t)}\}^{(1:T)}) \leq T\varepsilon /8$ . Note that, since predictors  $h^{(1:T)}$  can only observe  $x$  and not  $x'$ , we applied the algorithm of Theorem C.6 to (mixtures of) objectives from  $\mathcal{G}_{\mathrm{ag}}$  that depend only on  $(x,y)$ . Since  $\mathbb{E}_{x \sim D,\ell_{\mathbf{x}',i,j,S,v} \sim q_{\ell(t)}}[\ell_{\mathbf{x}',i,j,S,v}(\cdot)] = \mathbb{E}_{x',x \sim D}[\ell^{(t)}(\cdot)]$  however, we can still bound regret with respect to the loss functions in  $\mathcal{G}_{\mathrm{ag}}'$ , which do depend on  $x'$ , by  $\operatorname{Reg}_{B^*}(h^{(1:T)},\ell^{(1:T)}) = \operatorname{Reg}_{B^*}(h^{(1:T)},\{q_{\ell(t)}\}^{(1:T)}) \leq T\varepsilon /8$ ; we define both of these regrets with respect to the set of all predictors from  $\mathcal{X} \to \mathcal{Y}$ . Also note that the  $B^*$  baseline bounded by the multiaccuracy baseline (10):
$$
\begin{array}{l} B ^ {*} = \min _ {h ^ {*} \in \mathcal {H} ^ {*}} \frac {1}{T} \sum_ {t = 1} ^ {T} \max _ {h ^ {\prime} \in \mathcal {H} ^ {*}} \underset {(x, y) \sim D} {\mathbb {E}} \left[ f ^ {(t)} (x, h ^ {\prime} (x)) \cdot (h ^ {*} (x) - g ^ {(t)} (y)) \right] \\ \leq \min _ {h ^ {*} \in \mathcal {H} ^ {*}} \max _ {\ell^ {*} \in \mathcal {G}} \max _ {h ^ {\prime} \in \mathcal {H} ^ {*}} \underset {(x, y) \sim D} {\mathbb {E}} \left[ f ^ {*} (x, h ^ {\prime} (x)) \cdot (h ^ {*} (x) - g ^ {*} (y)) \right] \\ \leq \min _ {h ^ {*} \in \mathcal {H} ^ {*}} \max _ {j, S} \left| \underset {(x, x ^ {\prime}, y) \sim D} {\mathbb {E}} \left[ (h ^ {*} (x) _ {j} - y _ {j}) \cdot 1 [ x ^ {\prime} \in S ] \right] \right| \\ = B _ {\text {m u l t i - a c c}}. \\ \end{array}
$$
Thus, we have regret  $\mathrm{Reg}_{B_{\mathrm{multi - acc}}}(h^{(1:T)},\ell^{(1:T)})\leq \mathrm{Reg}_{B^*}(h^{(1:T)},\ell^{(1:T)})\leq T\frac{\varepsilon}{8}$ . By construction, every  $\ell^{(t)}$  is an  $(\varepsilon /8)$  best-response to the cost  $1 - \mathcal{L}_{D,(\cdot)}(h^{(t)})$ . By Lemma 3.4, there must exist a timestep where  $h^{(t)}$  is  $(\varepsilon /4)$ -optimal with respect to  $B_{\mathrm{multiacc}}$ . By Lemma 3.5, the  $h^{(t^*)}$  found by the algorithm is  $(\varepsilon /2)$ -optimal with respect to  $B_{\mathrm{multiacc}}$  with probability at least  $1 - \delta$ . By Fact E.8,  $h^{(t^*)}$  is a  $(S,\varepsilon ,\lambda)$ -agnostic multicalibrated predictor. Lastly, we verify that datapoints with identical group membership probabilities must have the same label probabilities as the update rules for their predicted labels are identical.
The best existing sample complexity bounds for agnostic multicalibration come from uniform convergence [38], and scale with  $\log (|\mathcal{H}|)\approx |\mathcal{X}|$  -that is, sample complexity scales with the size of one's domain. In contrast, Theorem E.9 provides a sample complexity that still depends only on the complexity of the groups on which one desires multicalibration. This can be an exponential (in  $|\mathcal{X}|$ ) reduction in sample complexity.
Similarly to Algorithm 6, we can use the same modification to scale our non-deterministic multicalibration algorithm (Algorithm 1) to agnostic settings.
Theorem E.10. Fix  $\varepsilon > 0$ ,  $\lambda \in \mathbb{Z}_{+}$  and a set of groups  $S \subseteq 2^{\mathcal{X}}$ . There exists an algorithm guarantees, with probability at least  $1 - \delta$ , the algorithm returns a randomized  $(S, \varepsilon, \lambda)$ -agnostic multicalibrated predictor while taking no more than  $O\left((\ln (|S| / \delta) + k\ln (\lambda)) / \varepsilon^{2}\right)$  samples from  $D$ .
E.3 Moment Multicalibration
Our treatment of multicalibration, and extensions of multicalibration, hold more generally for any problem that can be written as multi-objective learning with objectives that look separable. As an example, we demonstrate that we can use the same approach to devise algorithms for moment multicalibration, which concerns not only learning a multicalibrated label predictor but also a second predictor that estimates the higher-order moments of the label distribution [22]. As in online multicalibration, we will work with only binary classification problems in this section, and so will say that predictors output real-valued  $h_{\mu}(x), h_m(x) \in [0,1]$ .
Definition E.11. Let  $D$  be a data distribution for a binary classification problem and fix  $\varepsilon > 0$ ,  $\lambda, m \in \mathbb{Z}_{+}$ , and a set of groups  $S \subseteq 2^{\mathcal{X}}$ . A pair of predictors  $p = (p_{\mu}, p_m)$  where  $p_{\mu}, p_m : \mathcal{X} \to [0,1]$  is  $(\mathcal{S}, \varepsilon, \lambda, m)$ -mean-conditioned moment-multicalibrated on  $D$  if, for all  $S \in \mathcal{S}, v_{\mu}, v_m \in V_{\lambda}$ ,
$$
\begin{array}{l} \varepsilon \geq \left| \mathbb {E} _ {(x, y) \sim D, (h _ {\mu}, h _ {m}) \sim p} [ (h _ {\mu} (x) - y) \cdot 1 [ h _ {\mu} (x) \in v _ {\mu}, h _ {m} \in v _ {m}, x \in S ] ] \right|, \\ \varepsilon \geq \left| \mathbb {E} _ {(x, y) \sim D, (h _ {\mu}, h _ {m}) \sim p} \left[ \left(\left(h _ {\mu} (x) - y\right) ^ {m} - h _ {m} (x)\right) \cdot 1 \left[ h _ {\mu} (x) \in v _ {\mu}, h _ {m} \in v _ {m}, x \in S \right] \right] \right|. \\ \end{array}
$$
We will write  $h = (h_{\mu}, h_m)$  as shorthand, where  $h(x) = (h_{\mu}(x), h_m(x))$ . Moment multicalibration can also be expressed as a multi-objective learning problem.
Fact E.12. Consider the set of all pairs of predictors  $\mathcal{H} = \{(h_{\mu}, h_m) : h_{\mu}, h_m : \mathcal{X} \to [0,1]\}$ . Fix  $\varepsilon > 0$  and  $\lambda, m \in \mathbb{Z}_+$ , and a set of groups  $S \subseteq 2^{\mathcal{X}}$ . Define the objectives  $\mathcal{G}_{\mathrm{mc}}^{\mu} := \{\ell_{\mu, i, S, v}\}_{i \in \{\pm 1\}, S \in S, v \in V_{\lambda}^{2}}$  and  $\mathcal{G}_{\mathrm{mc}}^{m} := \{\ell_{m, i, S, v}\}_{i \in \{\pm 1\}, S \in S, v \in V_{\lambda}^{2}}$  where
$$
\begin{array}{l} \ell_ {m, i, S, v} (h, (x, y)) = 0. 5 + 0. 5 \cdot i \cdot 1 [ h (x) \in v, x \in S ] \cdot \left(h _ {m} (x) - (y - h _ {\mu} (x)) ^ {m}\right), \\ \ell_ {\mu , i, S, v} (h, (x, y)) = 0. 5 + 0. 5 \cdot i \cdot 1 [ h (x) \in v, x \in S ] \cdot (h _ {\mu} (x) - y). \\ \end{array}
$$
A pair of predictors is an  $\varepsilon$ -optimal solution to the single-distribution multi-objective learning problem  $(\{D\}, \mathcal{G}_{\mathrm{mc}}^{m} \cup \mathcal{G}_{\mathrm{mc}}^{\mu}, \mathcal{P})$  if and only if the pair is also  $(S, 2\varepsilon, \lambda, m)$  mean-conditioned moment multicalibrated.
Thus, we can follow the same steps as before to derive a moment multicalibration algorithm. The algorithm we derive for Theorem E.13 recovers the sample complexity of [22] for moment multicalibration. Moreover, the algorithm is parallelized in that it learns the mean and moment estimators  $h_{\mu}, h_{m}$  simultaneously, in contrast with [22]'s algorithm which uses nested optimization.
Theorem E.13. Fix  $\varepsilon > 0$ ,  $\lambda, r \in \mathbb{Z}_{+}$ , a set of groups  $S \subseteq 2^{\mathcal{X}}$ , and a data distribution  $D$ . The following algorithm returns a pair of predictors that are  $(S, \varepsilon, \lambda, m)$ -mean-conditioned moment multicalibrated on  $D$  and makes  $O(1 / \varepsilon^{2})$  calls to an agnostic learning oracle. Moreover, with probability  $1 - \delta$ , the oracle calls can be implemented with  $\widetilde{O}(m \ln(|S| \lambda / \delta) / \varepsilon^{4})$  samples from  $D$ .
Construct the problem  $(\{D\}, \mathcal{G}_{\mathrm{mc}}^m \cup \mathcal{G}_{\mathrm{mc}}^\mu, \mathcal{P})$  from Fact E.12 and let  $T = C\varepsilon^{-2}\ln(|S| \lambda^k \delta)$  for some universal constant  $C$ . Over  $T$  rounds, have a learner choose predictors  $h_m^{(1:T)} \in \mathcal{P}$  by applying the no-regret learning algorithm of Theorem C.6 to the stochastic costs  $\ell_m^{(1:T)}$ , instantiating  $\mathcal{A}$  to be Hedge with learning  $T^{-3/4}$ . In parallel, have a learner choose predictors  $h_\mu^{(1:T)} \in \mathcal{P}$  by applying the no-regret learning algorithm of Theorem C.6 to the stochastic costs  $\ell_\mu^{(1:T)}$ , instantiating  $\mathcal{A}$  to be the strongly adaptive Hedge algorithm (Lemma A.3). Have an adversary choose  $\ell_m^{(t)}, \ell_m^{(t)}$  by calling an agnostic learning oracle at each  $t \in [T]$ :  $\ell_m^{(t)} = \mathcal{A}_{\varepsilon/8}(1 - \ell_{(\cdot)}(h^{(t)}), \mathcal{G}_{\mathrm{mc}}^m)$  and