SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark C.8 (Closed-form of distribution-free best-responses). For calibration and multicalibration problems, the best-response implicitly defined in Theorem C.7 takes a clean closed-form: for any given  $x \in \mathcal{X}$ ,  $p(x)$  randomizes on two neighboring actions. This simple closed form was first derived by [10] and independently rediscovered by [18].
In Theorem C.7, we referenced the notion of a function being of bounded variation. This is a common notion in analysis which says that a function cannot go up and down too many times; thus reasonable objectives (including every finite loss function you can think of) should all be of bounded variation. For completeness, we prove below that bounded variation implies finite domain coverings.
Lemma C.9. Consider a function  $f: K \to [0,1]$  where  $K$  is a convex compact subset of  $\mathbb{R}^n$ . Suppose that  $f$  has pathwise bounded variation on  $K$ : that is, there exists a finite constant  $M$  such that for any linear path  $\gamma: [0,1] \to K$ ,  $V_{\gamma}(f) = \sup_{0=t_0 \leq t_1 \leq \dots \leq t_N=1} \sum_{i=1}^{N} \|f(\gamma(t_i)) - f(\gamma(t_{i-1}))\| \leq M$ . Then for any  $\varepsilon > 0$ , there exists a finite subset  $S \subseteq K$  such that for every  $y \in f(K)$ , there exists an  $x_i \in S$  with  $\|f(x_i) - y\| < \varepsilon$ .
Proof. Since  $K$  is compact, it is totally bounded. For any  $\delta > 0$ , there exists a finite set  $T$  such that  $K$  is covered by balls of radius  $\delta$  centered at points in  $T$ . Let  $\delta = \frac{\varepsilon}{2M}$ , where  $M$  is the constant from the pathwise bounded variation condition. For each point  $t \in T$ , choose a point  $s_t \in K$  such that  $\| s_t - t \| < \delta$ . Define  $S = \{s_t : t \in T\}$ . Let  $y \in f(K)$ . Then there exists an  $x \in K$  with  $f(x) = y$ . Since  $K$  is covered by balls of radius  $\delta$  centered at points in  $T$ , there exists a  $t \in T$  with  $\| x - t \| < \delta$ . By the construction of  $S$ , we also have  $\| s_t - t \| < \delta$ . Then, by the triangle inequality,  $\| s_t - x \| \leq \| s_t - t \| + \| t - x \| < 2\delta = \frac{\varepsilon}{M}$ . Consider a linear continuous path  $\gamma : [0,1] \to K$  with  $\gamma(0) = x$  and  $\gamma(1) = s_t$ ; such a path must exist by the convexity of  $K$ . By the pathwise bounded variation condition, we have  $V_{\gamma}(f) \leq M$ . For the partition  $0 = t_0 \leq t_1 = 1$ , we have  $\| f(s_t) - y \| = \| f(s_t) - f(x) \| \leq V_{\gamma}(f) \cdot \| s_t - x \| \leq M \cdot \frac{\varepsilon}{M} = \varepsilon$ . Thus, for every  $y \in f(K)$ , there exists an  $s_t \in S$  such that  $\| f(s_t) - y \| < \varepsilon$ .
D Proofs and Algorithms for Section 4
In the algorithms throughout this section, given an objective  $\ell^{(t)}\in \mathcal{G}_{\mathrm{mc}}$ , we write  $i^{(t)},j^{(t)},S^{(t)},v^{(t)}$  such that  $\ell^{(t)} = \ell_{i,j,S,v}$ .
D.1 Batch Multicalibration
Multicalibration with non-deterministic predictors. Our first algorithm uses no-regret no-regret (NRNR) dynamics to find non-deterministic multicalibrated predictors. We now restate and prove its guarantees from Theorem 4.1.
Algorithm 1 Non-Deterministic Multicalibration Algorithm (Theorem 4.1)
1: Input:  $S \subseteq 2^{\mathcal{X}}$ ,  $\varepsilon \in (0,1)$ ,  $k, \lambda, T \in \mathbb{Z}_{+}$ , and distribution  $D$ ;
2: Initialize Hedge iterate  $q^{(1)} = \mathrm{Uniform}(\mathcal{G}_{\mathrm{mc}})$  and Hedge iterate  $h^{(1)} = [1 / k,\dots ,1 / k]^{\chi}$
3: for  $t = 1$  to  $T$  do
4: Sample objective  $\ell^{(t)}\sim q^{(t)}$  and datapoint  $(x^{(t)},y^{(t)})\sim D$
5: For every  $x \in \mathcal{X}$ , let  $h^{(t + 1)}(x) \coloneqq \operatorname{Hedge}(c_x^{(1:t)})$ , where
$$
c _ {x} ^ {(t)} (\widehat {y}) := \frac {1}{2} (1 + i ^ {(t)} \cdot 1 [ h (x) \in v ^ {(t)}, x \in S ^ {(t)} ] \cdot \widehat {y} _ {j ^ {(t)}});
$$
6: Let  $q^{(t + 1)} \coloneqq \mathrm{Hedge}(c_{\mathrm{adv}}^{(1:t)})$ , where  $c_{\mathrm{adv}}^{(t)}(\ell) \coloneqq 1 - \ell(h^{(t)}, (x^{(t)}, y^{(t)}))$ ;
7: end for
8: Return:  $p^*$ , a uniform distribution over  $h^{(1)},\ldots ,h^{(T)}$
Theorem 4.1. Fix  $\varepsilon >0$ ,  $\lambda, k\in \mathbb{Z}_{+}$ , set of groups  $S\subseteq 2^{\mathcal{X}}$ , and data distribution  $D$ . The below algorithm, with probability  $1 - \delta$ , returns a non-deterministic  $k$ -class predictor that is  $(S,\varepsilon,\lambda)$ -multicalibrated on  $D$  and takes no more than  $O\left(\varepsilon^{-2}(\ln (|S| / \delta) + k\ln (\lambda))\right)$  samples from  $D$ .
Construct the problem  $\{\{D\}, \mathcal{G}_{\mathrm{mc}}, \mathcal{P}\}$  from Fact 2.5 and let  $T = C\varepsilon^{-2}\ln(|S| \lambda^k / \delta)$  for some universal constant  $C$ . Over  $T$  rounds, have an adversary choose  $q^{(1:T)} \in \Delta(\mathcal{G}_{\mathrm{mc}})$  by applying Hedge to the costs  $\{1 - \ell_{(\cdot)}(h^{(t)}, (x^{(t)}, y^{(t)}))\}^{(1:T)}$  where  $(x^{(t)}, y^{(t)}) \stackrel{\mathrm{i.i.d.}}{\sim} D$ . In parallel, have a learner choose predictors  $h^{(1:T)} \in \mathcal{P}$  by applying the no-regret learning algorithm of Theorem 3.7 to the stochastic costs  $\ell^{(1:T)}$ , where  $\ell^{(t)} \stackrel{\mathrm{i.i.d.}}{\sim} q^{(t)}$ . Return the predictor  $p = \mathrm{Uniform}(h^{(1:T)})$ . This algorithm is written explicitly in Algorithm 1.
Proof. By Theorem 3.7, if  $T \geq 64\varepsilon^{-2}\ln(k)$ , the predictors  $h^{(1:T)}$  guarantee the learner a regret bound of  $\mathrm{Reg}_{\mathrm{weak}}(h^{(1:T)}, \{\mathcal{L}_{\ell(t)}\}^{(1:T)}) \leq T\varepsilon/4$ . Similarly, by Lemma 3.1, if  $T \geq 576\varepsilon^{-2}\ln(2k\lambda^k |S|)$ , the objective mixtures  $q^{(1:T)}$  guarantee the adversary a regret bound of  $\mathrm{Reg}(q^{(1:T)}, \{1 - \ell_{(\cdot)}(h^{(t)}, (x^{(t)}, y^{(t)}))\}^{(1:T)}) \leq T\frac{\varepsilon}{12}$ .
We now argue that the adversary's regret with respect to the costs  $1 - \ell_{(\cdot)}(h^{(t)},(x^{(t)},y^{(t)}))$  approximates its regret with respect to the costs  $1 - \mathcal{L}_{D,(\cdot)}(h^{(t)})$ . Since  $\mathbb{E}_{x^{(t)},y^{(t)}\sim D}\left[\ell_{(\cdot)}(h^{(t)},(x^{(t)},y^{(t)}))\right] = \mathcal{L}_{D,(\cdot)}(h^{(t)})$ , by Lemma 3.2, there is a universal constant  $C$  such that, if  $T\geq C\varepsilon^{-2}\ln (2k\lambda^k |\mathcal{S}| / \delta)$
$$
\left. \right.\left| \mathrm {R e g} \left(q ^ {(1: T)}, \{1 - \ell_ {(\cdot)} (h ^ {(t)}, (x ^ {(t)}, y ^ {(t)})) \} ^ {(1: T)}\right) - \mathrm {R e g} \left(q ^ {(1: T)}, \{1 - \mathcal {L} _ {D, (\cdot)} (h ^ {(t)}) \} ^ {(1: T)}\right)\right| \leq T \frac {\varepsilon}{1 2},
$$
with probability  $1 - \delta$ . Similarly, since  $\mathbb{E}_{\ell(t) \sim q^{(t)}}\left[\mathcal{L}_{D,\ell(t)}(h^{(t)})\right] = \mathcal{L}_{D,q^{(t)}}(h^{(t)})$ , Lemma 3.2 guarantees
$$
\left. \right.\left| \operatorname {R e g} \left(q ^ {(1: T)}, \{1 - \mathcal {L} _ {D, (\cdot)} (h ^ {(t)}) \} ^ {(1: T)}\right) - \operatorname {R e g} \left(\ell^ {(1: T)}, \{1 - \mathcal {L} _ {D, (\cdot)} (h ^ {(t)}) \} ^ {(1: T)}\right)\right| \leq T \frac {\varepsilon}{1 2},
$$
with probability  $1 - \delta$ . Taking a triangle inequality and union bound, we can see that for sufficiently large  $C$ ,  $\operatorname{Reg}\left(\ell^{(1:T)}, \{1 - \mathcal{L}_{D,(\cdot)}(h^{(t)})\}^{(1:T)}\right) \leq T\varepsilon / 4$ .
By Lemma 3.3, the ergodic iterate  $h^*$  is an  $(\varepsilon / 2)$ -optimal solution. By Fact 2.5,  $h^*$  is therefore a  $(S, \varepsilon, \lambda)$ -multicalibrated predictor. The sample complexity is exactly  $T$  since the algorithm only samples one datapoint at each iteration.
Multicalibration with deterministic predictors. We are often specifically interested in finding deterministic multicalibrated predictors. This is usually because non-deterministic predictors can be multicalibrated in a very weak sense, as we show in the following example.
Example D.1. Consider a data distribution  $D$  supported uniformly on  $\mathcal{X} = \{x_1, x_2\}$ , where  $\mathcal{Y} = [0, 1]$ ,  $\operatorname*{Pr}(Y = 1 \mid X = x_1) = 0$  and  $\operatorname*{Pr}(Y = 1 \mid X = x_2) = 1$ . The non-deterministic predictor that is supported uniformly on predictors  $h_1$  and  $h_2$ , where  $h_1(x_1) = 0$  and  $h_1(x_2) = 0.5$  and  $h_2(x_1) = 0.5$  and  $h_2(x_2) = 1$ , is technically multicalibrated. However, neither  $h_1$  nor  $h_2$  are calibrated.
Our second algorithm uses no-regret best-response (NRBR) dynamics to find deterministic multicalibrated predictors. We now restate and prove its guarantees from Theorem 4.2.
Algorithm 2 Deterministic Multicalibration Algorithm (Theorem 4.2)
1: Input:  $S \subseteq 2^{\mathcal{X}}, \varepsilon \in (0,1), k, \lambda, T, C \in \mathbb{Z}_{+}$ , distribution  $D$ , and agnostic learning oracle  $\mathcal{A}$ ;
2: Initialize Hedge iterate  $h^{(1)} = [1 / k, \dots, 1 / k]^{\mathcal{X}}$ ;
3: for  $t = 1$  to  $T$  do
4: Let  $\ell^{(t)} = \mathcal{A}_{\varepsilon /8}(c_{\mathrm{adv}}^{(t)},\mathcal{G}_{\mathrm{mc}})$  where  $c_{\mathrm{adv}}^{(t)}(\ell)\coloneqq 1 - \mathcal{L}_{D,\ell}(h^{(t)})$
5: For every  $x \in \mathcal{X}$ , let  $h^{(t + 1)}(x) \coloneqq \operatorname{Hedge}(c_x^{(1:t)})$ , where
6: end for
7: Take  $C\ln (T / \delta) / \varepsilon^2$  samples  $\mathbf{x} \sim D$  and let  $t^{*} = \arg \min \sum \ell^{(t)}(h^{(t)},(x,y))$ ;
8: Return the predictor  $h(t^{*})$
Theorem 4.2. Fix  $\varepsilon >0$ ,  $\lambda ,k\in \mathbb{Z}_{+}$ , a set of groups  $S\subseteq 2^{\mathcal{X}}$ , and a data distribution  $D$ . The following algorithm returns a deterministic  $k$ -class predictor that is  $(S,\varepsilon ,\lambda)$ -multicalibrated on  $D$  and makes  $O(\ln (k) / \varepsilon^{2})$  calls to an agnostic learning oracle. Moreover, with probability  $1 - \delta$ , the oracle calls can be implemented with  $\widetilde{O}\left(\frac{1}{\varepsilon^3} (\sqrt{\ln(k)}\ln (k|\mathcal{S}| / \delta) + k\ln (\lambda))\right)$  samples from  $D$ .
Proof. By Theorem 3.7, if  $T \geq 256\varepsilon^{-2}\ln (k)$ , the predictors  $h^{(1:T)}$  guarantee the learner a regret bound of  $\mathrm{Reg}_{\mathrm{weak}}(h^{(1:T)},\{\mathcal{L}_{D,\ell^{(t)}}\}^{(1:T)}) \leq T\varepsilon /8$ . Moreover, by construction, every  $\ell^{(t)}$  is an  $(\varepsilon /8)$  best-response to the cost  $1 - \mathcal{L}_{D,\cdot}(\cdot)(h^{(t)})$ . By Lemma 3.4, there must exist a timestep where  $h^{(t)}$  is  $(\varepsilon /4)$ -optimal. By Lemma 3.5, the  $h^{(t^*)}$  found by the algorithm is  $(\varepsilon /2)$ -optimal with probability at least  $1 - \delta$ . By Fact 2.5,  $h^{(t^*)}$  is a  $(S,\varepsilon ,\lambda)$ -multicalibrated predictor. The oracle complexity is exactly  $T$ , while the sample complexity of oracle  $\mathcal{A}$  is a standard adaptive data analysis result (Lemma A.6).
We note that our use of no-regret best-response dynamics recovers a multicalibration algorithm similar to the original multicalibration algorithm of [21] and which can also be found in [7, 25]. We also remark that the guarantees of Theorem 4.2 hold in weaker settings. In particular, Theorem 4.2 holds with the same analysis even if we only asked that our agnostic learning oracle  $\mathcal{A}$  to be best-responding with respect to a min-max baseline. Furthermore, the last step of Algorithm 2, which explicitly samples datapoints to find timestep  $t^*$ , can be removed if one assumes that our agnostic learning oracle  $\mathcal{A}$  signals to us when it cannot find a greater than  $\varepsilon$  violation of multicalibration in the current predictor. This assumption would allow Algorithm 2 to terminate early and return the current predictor, and is assumed by prior multicalibration literature.
D.2 Online Multicalibration
Our next algorithm uses best-response no-regret (BRNR) dynamics for online multicalibration. We now restate and prove its guarantees from Theorem 4.3.
Algorithm 3 Online Multicalibration Algorithm
1: Input:  $S \subseteq 2^{\mathcal{X}}$ ,  $\varepsilon \in (0,1)$ ,  $\lambda, T \in \mathbb{Z}_{+}$ ;
2: Initialize Hedge iterate  $q^{(1)} = \mathrm{Uniform}(\mathcal{G}_{\mathrm{mc}})$ ;
3: for  $t = 1$  to  $T$  do
4:  $p^{(t)}(x)\coloneqq \min_{p^* (x)\in \Delta ([0,\varepsilon /4\lambda ,\ldots ,1])}\max_{y\in [0,1]}\frac{1}{2}\mathbb{E}_{\widehat{y}\sim p^* (x)}\left[1 + \mathbb{E}_{\ell_{i,S,v}\sim q^{(t)}}\bigl [i\cdot 1[x\in S,y\in v]\cdot (\widehat{y} -y)\bigr ]\right];$
5: Announce predictor  $p^{(t)}$  to Nature and observe Nature's data distribution  $D^{(t)}$ ;
6: Sample  $(x^{(t)},y^{(t)})\sim D^{(t)}$  and let  $q^{(t + 1)}\coloneqq \mathrm{Hedge}(c_{\mathrm{adv}}^{(1:t)})$  where  $c_{\mathrm{adv}}^{(t)}(\ell)\coloneqq 1 - \ell (h^{(t)},(x^{(t)},y^{(t)}))$
7: end for
Construct the online multi-objective learning problem  $(\mathcal{D},\mathcal{G}_{\mathrm{mc}},\mathcal{P})$  in Fact 2.6 and let  $T = C\varepsilon^{-2}\ln (|\mathcal{S}|\lambda \delta)$  for some universal constant  $C$ . Over  $T$  rounds, have an adversary choose  $q^{(1:T)}\in \Delta (\mathcal{G}_{\mathrm{mc}})$  by applying Hedge to the costs  $\{1 - \ell_{(\cdot)}(p^{(t)},(x^{(t)},y^{(t)}))\}^{(1:T)}$ , where  $(x^{(t)},y^{(t)})^{\mathrm{i.i.d.}}\sim D^{(t)}$ . Have a learner best-respond to each stochastic cost  $q^{(t)}$  with the  $(\varepsilon /2)$ -distribution-free best-response  $p^{(t)}\in \Delta (\mathcal{P})$  of Theorem 3.8. This algorithm is written explicitly in Algorithm 3.
Proof. By Lemma 3.1, if  $T \geq 576\varepsilon^{-2}\ln (2k\lambda |\mathcal{S}|)$ , the objective mixtures  $q^{(1:T)}$  guarantee the adversary a regret bound of  $\operatorname{Reg}\left(q^{(1:T)},\{1 - \ell_{(\cdot)}(p^{(t)},(x^{(t)},y^{(t)}))\}^{(1:T)}\right) \leq T\frac{\varepsilon}{12}$ . By Lemma 3.2, there is a universal constant  $C$  such that, if  $T \geq C\varepsilon^{-2}\ln (2k\lambda |\mathcal{S}| / \delta)$ ,
$$
\left. \right.\left| \operatorname {R e g} \left(q ^ {(1: T)}, \{1 - \ell_ {(\cdot)} (p ^ {(t)}, (x ^ {(t)}, y ^ {(t)})) \} ^ {(1: T)}\right) - \operatorname {R e g} \left(q ^ {(1: T)}, \{1 - \mathcal {L} _ {D ^ {(t)}, (\cdot)} (p ^ {(t)}) \} ^ {(1: T)}\right)\right| \leq T \frac {\varepsilon}{1 2},
$$
with probability  $1 - \delta$ . Thus,  $\operatorname{Reg}\left(q^{(1:T)},\{1 - \mathcal{L}_{D^{(t)},(\cdot)}(h^{(t)})\}^{(1:T)}\right) \leq T\varepsilon /4$  with probability  $1 - \delta$ .
Since each  $p^{(t)}$  is a  $(\varepsilon / 4)$  distribution-free best-response to  $q^{(t)}$ , whose existence is proven by Theorem 3.8, by Lemma 3.6, the predictors  $p^{(1:T)}$  are  $(\varepsilon / 2)$ -optimal on  $D^{(1:T)}$  with probability  $1 - \delta$ . By Fact 2.6,  $p^{(1:T)}$  are also  $(S, \varepsilon, \lambda)$ -online multicalibrated on  $D^{(1:T)}$ .
An online-to-batch reduction. The online multicalibration algorithm of Theorem 4.3 can also be used to obtain a non-deterministic multicalibrated predictor in batch settings through an online-to-batch reduction. This exactly recovers the non-deterministic multicalibration algorithm of [18, 35].
Theorem D.2 (Analysis of Algorithm 3 for batch multicalibration). Fix  $\varepsilon >0$ ,  $\lambda \in \mathbb{Z}_{+}$ , a set of groups  $S\subseteq 2^{\mathcal{X}}$ , and a data distribution  $D$ . Simulate Algorithm 3 by having Nature choose  $D$  at every timestep and return a uniform distribution  $\overline{p}$  over its outputs  $p^{(1:T)}$ . With probability  $1 - \delta$ ,  $\overline{p}$  is  $(S,\varepsilon ,\lambda)$ -multicalibrated on  $D$ . This algorithm takes  $O(\ln(|S|\lambda/\delta)/\varepsilon^2)$  samples from  $D$ .
Proof. By Theorem 4.3,  $\max_{\ell^{*}\in \mathcal{G}_{\mathrm{mc}}}\sum_{t = 1}^{T}\mathcal{L}_{D,\ell^{*}}(p^{(t)})\leq T\varepsilon$ . Thus, the non-deterministic predictor  $\overline{p}$  given by taking a uniform distribution over  $p^{(1)},\ldots ,p^{(T)}$  is a  $(S,\varepsilon ,\lambda)$ -multicalibrated predictor.
Unlike the algorithm of Theorem 4.1, the predictor  $\bar{p}$  output by Theorem D.2 is neither guaranteed to be succinct nor guaranteed to be of small circuit size. The lack of succinctness is because, even if  $\bar{p}$ 's
predicted label distribution on any feature  $x \in \mathcal{X}$  is succinct,  $\overline{p}$  itself can require an exponentially large support over  $\mathcal{P}$ . The large circuit size is because, at each timestep  $t$ , the learner is best-responding to a distribution  $q^{(t)}$  over objectives  $\mathcal{G}$  with a non-zero weight on every objective. In contrast, in the algorithm of Theorem 4.1, the learner only interacts with a single new objective  $\ell^{(t)}$  at each timestep  $t$ .
E Additional Multicalibration Considerations
In this section, we present additional results on multicalibration that all follow from the same game dynamics presented in Section 3.
E.1 Conditional Multicalibration
One shortcoming of existing definitions of multicalibration is that they measure violations marginally over the entire distribution  $D$ . That is, the amount that a predictor is allowed to violate calibration on a subgroup is inversely proportional to the probability mass of the subgroup, as reflected by the use of the indicator function in Definition 2.1. This can lead to predictors that are certifiably multicalibrated, but still poorly calibrated on underrepresented or minority groups. In contrast, the original definition of multicalibration proposed by [21], for which no non-trivial guarantee is known, is a conditional notion of multicalibration.
Below, we generalize this notion of conditional multicalibration so that setting  $S = S'$  recovers the original multicalibration definition of [21].
Definition E.1. Fix  $\varepsilon > 0$ ,  $\lambda \in \mathbb{Z}_{+}$ , and two sets of groups  $S, S' \subseteq 2^{\mathcal{X}}$ . A  $k$ -class predictor  $p \in \Delta(\mathcal{P})$  is  $(S, S', \varepsilon, \lambda)$ -conditionally multicalibrated for some data distribution  $D$  if
$$
\forall S \in \mathcal {S}, S ^ {\prime} \in \mathcal {S} ^ {\prime}, v \in V _ {\lambda} ^ {k}, j \in [ k ]: \left| \underset {(x, y) \sim D, h \sim p} {\mathbb {E}} \left[ (h (x) _ {j} - \delta_ {y, j}) \cdot 1 [ h (x) \in v, x \in S ] \mid x \in S ^ {\prime} \right] \right| \leq \varepsilon .
$$
It is not possible to obtain conditional multicalibration generally, so we will assume sample access to the conditional distributions  $\{D_{S'}\}_{S' \in S'}$ , where  $D_{S'} \coloneqq D \mid x \in S'$ . In practice, this assumption means that we are able to sample data from certain protected groups that may otherwise be underrepresented. For convenience, in this section, we will assume prior knowledge of  $\{\operatorname*{Pr}_D(x \in S')\}_{S' \in S'}$  but note these probabilities can be cheaply estimated beforehand.
To derive conditional multicalibration algorithms from game dynamics, we first write conditional multicalibration as a multi-distribution multi-objective learning problem.
Fact E.2. Let  $D$  be a data distribution for some  $k$ -class prediction problem and fix  $\varepsilon > 0$ ,  $\lambda \in \mathbb{Z}_{+}$  and two sets of groups  $S, S' \subseteq 2^{\mathcal{X}}$ . Let  $\mathcal{G}_{\mathrm{mc}}$  be the set of objectives  $\{\ell_{i,j,S,v}\}$  as defined in Fact 2.5 and let  $\mathcal{D} = \{D_S'\}_{S' \in S'}$ . Predictor  $p \in \Delta(\mathcal{P})$  is a  $\varepsilon$ -optimal solution to the multi-objective learning problem  $(\mathcal{D}, \mathcal{G}_{\mathrm{mc}}, \mathcal{P})$  if and only if  $p$  is  $(S, S', 2\varepsilon, \lambda)$ -conditionally multicalibrated for  $D$ .
Proof. In the multi-objective learning problem  $(\{D_{\mathcal{S}}^{\prime}\}_{S^{\prime}\in \mathcal{S}^{\prime}},\mathcal{G}_{\mathrm{mc}},\mathcal{P})$ , the multi-objective value of a predictor  $p$ ,  $\mathcal{L}^{*}(p)\coloneqq \max_{D^{*}\in \mathcal{D},\ell^{*}\in \mathcal{G}}\mathcal{L}_{D^{*},\ell^{*}}(p)$ , is exactly the (rescaled and shifted) magnitude of the predictor's multicalibration violation. Formally,
$$
\mathcal {L} ^ {*} (p) = \frac {1}{2} + \frac {1}{2} \max _ {{j \in [ k ], S \in \mathcal {S}, S ^ {\prime} \in \mathcal {S} ^ {\prime}, v \in V _ {\lambda} ^ {k}}} | \underset {(x, y) \sim D _ {S} ^ {\prime}, h \sim p} {\mathbb {E}} [ (h (x) _ {j} - \delta_ {y, j}) \cdot 1 [ h (x) \in v, x \in S ] ] |.
$$
The optimal multi-objective value of the problem  $(\mathcal{D},\mathcal{G}_{\mathrm{mc}},\mathcal{P})$  is 0.5, as  $\mathcal{G}_{\mathrm{mc}}$  are symmetric around 0.5 and the Bayes classifier still achieves a loss of 0.5.
Thus,  $\mathcal{L}^*(p) - \min_{h^* \in \mathcal{P}} \mathcal{L}^*(h^*) = \varepsilon$  if and only if our conditional multicalibration violation is  $2\varepsilon$ .
Conditional multicalibration algorithms. The following Theorem E.3 is, to the best of our knowledge, the first non-trivial guarantee for conditional multicalibration.
Theorem E.3. Fix  $\varepsilon > 0$ ,  $\lambda, k \in \mathbb{Z}_{+}$ , two sets of groups  $S, S' \subseteq 2^{\mathcal{X}}$ , and a data distribution  $D$ . The following algorithm returns a deterministic  $k$ -class predictor that is  $(S, S', \varepsilon, \lambda)$ -conditionally multicalibrated on  $D$  and makes  $O(\ln(k) / \varepsilon^{2})$  calls to an agnostic learning oracle. Moreover, with probability  $1 - \delta$ , the oracle calls can be implemented with  $\widetilde{O}\left(\frac{1}{\varepsilon^{3}} (\sqrt{\ln(k)} \ln(k |S| |S'| / \delta) + k \ln(\lambda))\right)$  samples from each conditional distribution  $\{D_{S}'\}_{S' \in S'}$ .
Construct the problem  $(\mathcal{D},\mathcal{G}_{\mathrm{mc}},\mathcal{P})$  from Fact E.2 and let  $T = C\varepsilon^{-2}|\mathcal{S}'|\ln (k)$  for some universal constant  $C$ . Over  $T$  rounds, have a learner choose predictors  $h^{(1:T)}$  by applying the no-regret learning algorithm of Theorem C.6 to the stochastic costs  $\ell^{(1:T)}$  and distributions  $\{D_{S'(t)}\}^{(1:T)}$ . Have an adversary choose  $S^{\prime}(1:T)$  and  $\ell^{(1:T)}$  by calling an agnostic learning oracle at each  $t\in [T]$ :  $S^{\prime}(t),\ell^{(t)} = \mathcal{A}_{\varepsilon /8}(1 - \mathcal{L}_{D_{(\cdot),(\cdot)}}(h^{(t)}))$ . Using  $C\ln (T|S'|\delta) / \varepsilon^2$  samples from each distribution in  $\{D_S^{\prime}\}_{S^{\prime}\in S^{\prime}}$ , return the predictor  $h^{(t^{*})}$  with the lowest empirical multicalibration error. This algorithm is written explicitly in Algorithm 4.
Proof. In order to apply Theorem C.6, we first observe that every objective in  $\mathcal{G}_{\mathrm{mc}}$  is separable and every distribution  $D_S'$  is absolutely continuous with respect to  $D$ . By definition, for any  $S' \in S'$ , the Radon-Nikodym derivative of  $D_S'$  with respect to  $D$  at  $x \in \mathcal{X}$  is  $1[x \in S'] / \operatorname*{Pr}_D(x \in S')$ . Thus, the sequence  $h^{(1:T)}$  corresponds to running the no-regret algorithm of Theorem C.6 when  $\mathcal{A}$  is chosen to be Prod. By Lemma A.4, we know that choosing the Prod algorithm as  $\mathcal{A}$  satisfies the conditions of Theorem C.6 with  $R(T) \in O(\sqrt{\ln(k)T})$ . Theorem C.6 therefore guarantees that  $\operatorname{Reg}_{B^*}(h^{(1:T)}, \{\mathcal{L}_{D_{S'(t)}, \ell^{(t)}}(\cdot)\}^{(1:T)}) \leq O(\sqrt{|S'| \ln(k)T})$ , where  $B^*$  is as defined in (8). Since we can lower bound the baseline  $B^* \geq 0.5$  by plugging the Bayes classifier  $h^*(x) = \mathbb{E}_{(x,y) \sim D}[y \mid x]$  into (8),  $\operatorname{Reg}_{\mathrm{weak}}(h^{(1:T)}, \{\mathcal{L}_{D_{S'(t)}, \ell^{(t)}}(\cdot)\}^{(1:T)}) \leq \operatorname{Reg}_{B^*}(h^{(1:T)}, \{\mathcal{L}_{D_{S'(t)}, \ell^{(t)}}(\cdot)\}^{(1:T)}) \leq O(\sqrt{|S'| \ln(k)T})$ . Thus, if  $T \geq C|S'| \ln(k)/\varepsilon^2$ ,  $\operatorname{Reg}_{\mathrm{weak}}(h^{(1:T)}, \{\mathcal{L}_{D_{S'(t)}, \ell^{(t)}}(\cdot)\}^{(1:T)}) \leq T\varepsilon/8$ .