SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ \left| \psi^ {j} \left(t _ {j}, q _ {j}, p _ {j}\right) - \psi^ {j} \left(\tilde {t} _ {j}, q _ {j}, \tilde {p} _ {j}\right) \right| \right] \leq L _ {\mathrm {t}} ^ {j} \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ \left| t _ {j} - \tilde {t} _ {j} \right| \right] + L _ {\mathrm {p}} ^ {j} \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ \left| p _ {j} - \tilde {p} _ {j} \right| \right] \\ = L _ {\mathfrak {t}} ^ {j} \mathbb {E} _ {\mathbf {Y} | \mathbf {X}} \left[ \sqrt {(t _ {j} - \tilde {t} _ {j}) ^ {2}} \right] + L _ {\mathfrak {p}} ^ {j} \mathbb {E} _ {\mathbf {Y} | \mathbf {X}} \left[ \sqrt {(p _ {j} - \tilde {p} _ {j}) ^ {2}} \right] \\ \leq L _ {\mathrm {t}} ^ {j} \sqrt {\mathbb {E} _ {\boldsymbol {Y} \mid \boldsymbol {X}} \left[ \left(t _ {j} - \tilde {t} _ {j}\right) ^ {2} \right]} + L _ {\mathrm {p}} ^ {j} \sqrt {\mathbb {E} _ {\boldsymbol {Y} \mid \boldsymbol {X}} \left[ \left(p _ {j} - \tilde {p} _ {j}\right) ^ {2} \right]}, \tag {54} \\ \end{array}
$$
where the last inequality follows from Jensen's inequality applied to a concave function  $x \mapsto \sqrt{x}$ . Using the fact that  $\tilde{t_j} = \mathbb{E}_{\mathbf{Y}|\mathbf{X}}[t_j]$  and  $\tilde{p_j} = \mathbb{E}_{\mathbf{Y}|\mathbf{X}}[p_j]$ , we have
$$
\mathbb {E} _ {\boldsymbol {Y} \mid \boldsymbol {X}} \left[ \left(t _ {j} - \tilde {t} _ {j}\right) ^ {2} \right] = \operatorname {V a r} _ {\boldsymbol {Y} \mid \boldsymbol {X}} \left(t _ {j}\right) \leq \frac {1}{4 n}, \tag {55}
$$
as  $t_j = n^{-1} \sum_{i=1}^{n} y_{ij} \hat{y}_{ij}$  is an average of  $n$  Bernoulli i.i.d. random variables  $y_{ij} \hat{y}_{ij}$ , each having variance at most  $\frac{1}{4}$ ; and using the same argument,  $\mathbb{E}_{\mathbf{Y}|\mathbf{X}}[(p_j - \tilde{p}_j)^2] \leq \frac{1}{4n}$ . This gives
$$
\mathbb {E} _ {\boldsymbol {Y} \mid \boldsymbol {X}} \left[ \left| \psi^ {j} \left(\tilde {t} _ {j}, q _ {j}, \tilde {p} _ {j}\right) - \psi^ {j} \left(t _ {j}, q _ {j}, p _ {j}\right) \right| \right] \leq \frac {L _ {\mathrm {t}} ^ {j} + L _ {\mathrm {p}} ^ {j}}{2 \sqrt {n}}, \tag {56}
$$
and finishes the proof.
C.3 Regret of semi-ETU under model misspecification
In this section, we quantify the influence of the estimation error of marginal probabilities, proving Theorem 5.3.
Notation To emphasize the dependence of  $\varPsi_{\mathrm{ETU}}$  on the label probability estimates, in this section we will write
$$
\varPsi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {X}) = \mathbb {E} _ {\boldsymbol {Y} \sim \boldsymbol {\eta} (\boldsymbol {X})} [ \varPsi (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) ] =: \varPsi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {\eta}), \tag {57}
$$
This notation is well-defined, as we have shown in Appendix A.2 that in fact, the dependence on  $\mathbf{X}$  is mediated only through the marginal label probabilities  $\eta(\mathbf{X}) = (\eta(\mathbf{x}_1), \dots, \eta(\mathbf{x}_n))$ , and we abbreviate  $\eta(\mathbf{X})$  as  $\eta$ . Similarly, we will write
$$
\tilde {t} _ {j} (\boldsymbol {\eta}) = \mathbb {E} _ {\boldsymbol {y}: j \sim \eta_ {j} (\boldsymbol {X})} [ t (\boldsymbol {y}: j, \hat {\boldsymbol {y}}: j) ] = n ^ {- 1} \sum_ {i = 1} ^ {n} \eta_ {j} (\boldsymbol {x} _ {i}) \hat {y} _ {i j},
$$
$$
\tilde {p} _ {j} (\boldsymbol {\eta}) = \mathbb {E} _ {\boldsymbol {y}: j \sim \eta_ {j} (\boldsymbol {X})} [ p (\boldsymbol {y}: j) ] = n ^ {- 1} \sum_ {i = 1} ^ {n} \eta_ {j} \left(\boldsymbol {x} _ {i}\right). \tag {58}
$$
Note that  $q$  is independent of  $\pmb{\eta}$ . This allows us to write the semi-ETU objective as
$$
\tilde {\Psi} _ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {\eta}) := \Psi (\tilde {\boldsymbol {t}}, \boldsymbol {q}, \tilde {\boldsymbol {p}}), \tag {59}
$$
where we dropped the dependence of  $\tilde{t}$  and  $\tilde{\pmb{p}}$  on  $\eta$  as clear from the context. The optimal (Bayes) predictor  $\hat{Y}^{\star}$  is the one which maximizes the expected utility with regard to the true label probabilities. In the notation of this chapter, (1) reads
$$
\hat {\boldsymbol {Y}} ^ {\star} = \underset {\hat {\boldsymbol {Y}} \in \mathcal {Y} _ {k} ^ {n}} {\operatorname {a r g m a x}} \Psi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {\eta}). \tag {60}
$$
Unfortunately, the learning algorithm does not know the true label marginals  $\pmb{\eta}(\pmb{X})$ , but has only access to the estimates  $\hat{\pmb{\eta}}(\pmb{X}) = (\hat{\pmb{\eta}}(\pmb{x}_1), \dots, \hat{\pmb{\eta}}(\pmb{x}_n))$  for the considered set of instances. The algorithm computes its predictions  $\hat{\pmb{Y}}^\dagger$  by using the estimates in place of the true marginals. Thus, it can only generate
$$
\hat {\boldsymbol {Y}} ^ {\dagger} = \underset {\hat {\boldsymbol {Y}} \in \hat {\mathcal {Y}} _ {k} ^ {n}} {\operatorname {a r g m a x}} \Psi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \hat {\boldsymbol {\eta}}). \tag {61}
$$
We can make the same definitions also for the semi-empirical ETU optimization, leading to
$$
\tilde {\mathbf {Y}} ^ {\star} = \underset {\hat {\mathbf {Y}} \in \hat {\mathcal {Y}} _ {k} ^ {n}} {\operatorname {a r g m a x}} \tilde {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \boldsymbol {\eta}) \quad \text {a n d} \quad \tilde {\mathbf {Y}} ^ {\dagger} = \underset {\tilde {\mathbf {Y}} \in \hat {\mathcal {Y}} _ {k} ^ {n}} {\operatorname {a r g m a x}} \tilde {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \hat {\boldsymbol {\eta}}). \tag {62}
$$
Regret for semi-ETU approximation First, we show that for metrics with  $p$ -Lipschitz components, the  $\Psi$ -regret of the resulting semi-ETU predictor (62), which is the suboptimality of  $\hat{Y}^{\dagger}$  (with respect to  $\hat{Y}^{\star}$ ) in terms of  $\bar{\Psi}_{\mathrm{ETU}}$ , is well-controlled and upper-bounded by the estimation error of the marginals. As the resulting expression is somewhat unwieldy, we then apply some further bounding to arrive at the much simpler result stated in the main paper in Theorem 5.3.
Lemma C.3 (Misspecification for semi-ETU approximation). Let  $\Psi \in \mathcal{L}_{\mathrm{OI}}$  be an instance-order invariant linearly decomposable loss function that is  $p$ -Lipschitz, and  $\hat{\mathbf{Y}}$  an arbitrary set of predictions. Then, the difference of the semi-empirical ETU risk when using two different versions of the marginals,  $\eta$  and  $\eta'$ , is bounded by the difference  $\eta - \eta'$  through
$$
\left| \tilde {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \boldsymbol {\eta}) - \tilde {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \boldsymbol {\eta} ^ {\prime}) \right| \leq n ^ {- 1} \sum_ {j = 1} ^ {m} \left(L _ {\mathrm {t}} ^ {j} \left(\hat {t} ^ {j} (\boldsymbol {\eta})\right) + L _ {\mathrm {p}} ^ {j} \left(\hat {p} ^ {j} (\boldsymbol {\eta})\right)\right) \sum_ {i = 1} ^ {n} \left| \eta_ {j} \left(\boldsymbol {x} _ {i}\right) - \eta_ {j} ^ {\prime} \left(\boldsymbol {x} _ {i}\right) \right|. \tag {63}
$$
Proof. Plugging in the definitions, we have
$$
\begin{array}{l} |\tilde{\Psi}_{\text{ETU}}(\hat{Y};\boldsymbol {\eta}) - \tilde{\Psi}_{\text{ETU}}(\hat{Y};\boldsymbol {\eta})| = \Big|\sum_{j = 1}^{m}\psi^{j}\left(\tilde{t}^{j}(\boldsymbol {\eta}),q^{j},\tilde{p}^{j}(\boldsymbol {\eta})\right) - \sum_{j = 1}^{m}\psi^{j}\left(\tilde{t}^{j}(\boldsymbol{\eta}^{\prime}),q^{j},\tilde{p}^{j}(\boldsymbol{\eta}^{\prime})\right)\Big| \\ \leq \sum_ {j = 1} ^ {m} \left| \psi^ {j} \left(\tilde {t} ^ {j} (\boldsymbol {\eta}), q ^ {j}, \tilde {p} ^ {j} (\boldsymbol {\eta})\right) - \psi^ {j} \left(\tilde {t} ^ {j} \left(\boldsymbol {\eta} ^ {\prime}\right), q ^ {j}, \tilde {p} ^ {j} \left(\boldsymbol {\eta} ^ {\prime}\right)\right) \right| \\ \leq \sum_ {j = 1} ^ {m} L _ {\mathrm {t}} ^ {j} \left(\tilde {t} ^ {j} (\boldsymbol {\eta})\right) \left| \tilde {t} ^ {j} (\boldsymbol {\eta}) - \tilde {t} ^ {j} \left(\boldsymbol {\eta} ^ {\prime}\right) \right| + L _ {\mathrm {p}} ^ {j} \left(\tilde {p} ^ {j} (\boldsymbol {\eta})\right) \left| \tilde {p} ^ {j} (\boldsymbol {\eta}) - \tilde {p} ^ {j} \left(\boldsymbol {\eta} ^ {\prime}\right) \right|, \tag {64} \\ \end{array}
$$
where the last line used the definition of  $p$ -Lipschitzness.
The terms under the sum can be bounded by the estimation error of  $\eta^{\prime}$ :
$$
| \tilde {t} _ {j} (\boldsymbol {\eta}) - \tilde {t} _ {j} (\boldsymbol {\eta} ^ {\prime}) | = \left| n ^ {- 1} \sum_ {i = 1} ^ {n} \hat {y} _ {i j} \left(\eta_ {j} \left(\boldsymbol {x} _ {i}\right) - \eta_ {j} ^ {\prime} \left(\boldsymbol {x} _ {i}\right)\right) \right| \leq n ^ {- 1} \sum_ {i = 1} ^ {n} \left| \eta_ {j} \left(\boldsymbol {x} _ {i}\right) - \eta_ {j} ^ {\prime} \left(\boldsymbol {x} _ {i}\right) \right|,
$$
$$
| \tilde {p} _ {j} (\boldsymbol {\eta}) - \tilde {p} _ {j} \left(\boldsymbol {\eta} ^ {\prime}\right) | = \left| n ^ {- 1} \sum_ {i = 1} ^ {n} \left(\eta_ {j} \left(\boldsymbol {x} _ {i}\right) - \eta_ {j} ^ {\prime} \left(\boldsymbol {x} _ {i}\right)\right) \right| \leq n ^ {- 1} \sum_ {i = 1} ^ {n} \left| \eta_ {j} \left(\boldsymbol {x} _ {i}\right) - \eta_ {j} ^ {\prime} \left(\boldsymbol {x} _ {i}\right) \right|, \tag {65}
$$
where in the first line we used  $\hat{y}_{ij} \in \{0,1\}$ .
Lemma C.4 (Regret bound for semi-ETU approximation). Let  $\varPsi\in\mathcal{L}_{\mathrm{OI}}$  be an instance-order invariant linearly decomposable loss function that is  $p$ -Lipschitz. Then we have
$$
\left. \right. \Psi_ {\mathrm {E T U}} \left(\hat {\boldsymbol {Y}} ^ {\star}; \boldsymbol {X}\right) - \Psi_ {\mathrm {E T U}} \left(\tilde {\boldsymbol {Y}} ^ {\dagger}; \boldsymbol {X}\right) \leq \frac {1}{\sqrt {n}} \sum_ {j = 1} ^ {m} \left(L _ {\mathrm {t}} ^ {j} \left(\tilde {p} _ {j} (\boldsymbol {\eta})\right) + L _ {\mathrm {p}} ^ {j} \left(\tilde {p} _ {j} (\boldsymbol {\eta})\right)\right) +
$$
$$
2 \sum_ {j = 1} ^ {m} \frac {L _ {\mathrm {t}} ^ {j} (\tilde {p} _ {j} (\boldsymbol {\eta})) + L _ {\mathrm {p}} ^ {j} (p _ {j} (\boldsymbol {\eta}))}{n} \sum_ {i = 1} ^ {n} | \eta_ {j} (\boldsymbol {x} _ {i}) - \hat {\eta} _ {j} (\boldsymbol {x} _ {i}) |. \tag {66}
$$
Proof. Using the optimality of  $\tilde{\mathbf{Y}}^{\dagger}$  and a supremum bound, we get
$$
\psi_ {\mathrm {E T U}} (\hat {Y} ^ {\star}; \eta) - \psi_ {\mathrm {E T U}} (\tilde {Y} ^ {\dagger}; \eta) = \psi_ {\mathrm {E T U}} (\hat {Y} ^ {\star}; \eta) - \tilde {\psi} _ {\mathrm {E T U}} (\hat {Y} ^ {\star}; \hat {\eta})
$$
$$
+ \tilde {\Psi} _ {\mathrm {E T U}} (\hat {Y} ^ {\star}; \hat {\eta}) - \tilde {\Psi} _ {\mathrm {E T U}} \left(\tilde {Y} ^ {\dagger}; \hat {\eta}\right)
$$
$$
+ \tilde {\Psi} _ {\mathrm {E T U}} \left(\tilde {Y} ^ {\dagger}; \hat {\boldsymbol {\eta}}\right) - \Psi_ {\mathrm {E T U}} \left(\tilde {Y} ^ {\dagger}; \boldsymbol {\eta}\right)
$$
$$
\leq \bar {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}} ^ {\star}; \boldsymbol {\eta}) - \bar {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}} ^ {\star}; \hat {\boldsymbol {\eta}}) \tag {67}
$$
$$
+ \tilde {\Psi} _ {\mathrm {E T U}} \left(\tilde {\mathbf {Y}} ^ {\dagger}; \hat {\boldsymbol {\eta}}\right) - \Psi_ {\mathrm {E T U}} \left(\tilde {\mathbf {Y}} ^ {\dagger}; \boldsymbol {\eta}\right)
$$
$$
\leq 2 \sup  _ {\hat {\boldsymbol {Y}}} | \Psi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {\eta}) - \tilde {\Psi} _ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \hat {\boldsymbol {\eta}}) |. \tag {68}
$$
We then use Theorem 5.2 and Lemma C.3 to bound
$$
\begin{array}{l} \left| \Psi_ {\text {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {\eta}) - \tilde {\Psi} _ {\text {E T U}} (\hat {\boldsymbol {Y}}; \hat {\boldsymbol {\eta}}) \right| \\ = \left| \Psi_ {\mathrm {E T U}} (\hat {Y}; \eta) - \tilde {\Psi} _ {\mathrm {E T U}} (\hat {Y}; \eta) + \tilde {\Psi} _ {\mathrm {E T U}} (\hat {Y}; \eta) - \tilde {\Psi} _ {\mathrm {E T U}} (\hat {Y}; \hat {\eta}) \right| \\ \leq \left| \bar {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \boldsymbol {\eta}) - \bar {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \boldsymbol {\eta}) \right| + \left| \bar {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \boldsymbol {\eta}) - \bar {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \hat {\boldsymbol {\eta}}) \right| \\ \leq \frac {1}{2 \sqrt {n}} \left(\sum_ {j = 1} ^ {m} \left(L _ {\mathrm {t}} ^ {j} \left(\tilde {p} _ {j}\right) + L _ {\mathrm {p}} ^ {j} \left(\tilde {p} _ {j}\right)\right)\right) + \sum_ {j = 1} ^ {m} \frac {L _ {\mathrm {t}} ^ {j} \left(\tilde {p} _ {j}\right) + L _ {\mathrm {p}} ^ {j} \left(\tilde {p} _ {j}\right)}{n} \sum_ {i = 1} ^ {n} \left| \eta_ {j} \left(\boldsymbol {x} _ {i}\right) - \hat {\eta} _ {j} \left(\boldsymbol {x} _ {i}\right) \right|, \tag {69} \\ \end{array}
$$
where we use the shorthand  $\tilde{p}_j = \tilde{p}_j(\pmb {\eta})$
At the cost of having a less strict bound, we can simplify this to
$$
\left. \Psi_ {\mathrm {E T U}} \left(\hat {Y} ^ {\star}; \boldsymbol {X}\right) - \Psi_ {\mathrm {E T U}} \left(\tilde {\boldsymbol {Y}} ^ {\dagger}; \boldsymbol {X}\right) \leq \frac {m}{\sqrt {n}} B + 2 \frac {\sqrt {m}}{n} B \sum_ {i = 1} ^ {n} \| \boldsymbol {\eta} \left(\boldsymbol {x} _ {i}\right) - \hat {\boldsymbol {\eta}} \left(\boldsymbol {x} _ {i}\right) \| _ {2}, \right. \tag {27}
$$
Proof. Using Cauchy-Schwartz, we can bound
$$
\begin{array}{l} \sum_ {j = 1} ^ {m} \frac {L _ {\mathrm {t}} ^ {j} (\tilde {p} _ {j} (\boldsymbol {\eta})) + L _ {\mathrm {p}} ^ {j} (p _ {j} (\boldsymbol {\eta}))}{n} \sum_ {i = 1} ^ {n} | \eta_ {j} (\boldsymbol {x} _ {i}) - \hat {\eta} _ {j} (\boldsymbol {x} _ {i}) | \\ \leq n ^ {- 1} \sum_ {i = 1} ^ {n} \sqrt {\sum_ {j = 1} ^ {m} \left(L _ {\mathrm {t}} ^ {j} (\tilde {p} _ {j} (\boldsymbol {\eta})) + L _ {\mathrm {p}} ^ {j} (p _ {j} (\boldsymbol {\eta}))\right) ^ {2}} \cdot \sqrt {\sum_ {j = 1} ^ {m} \left(\eta_ {j} (\boldsymbol {x} _ {i}) - \hat {\eta} _ {j} (\boldsymbol {x} _ {i})\right) ^ {2}} \\ = n ^ {- 1} \sqrt {\sum_ {j = 1} ^ {m} \left(L _ {\mathrm {t}} ^ {j} \left(\tilde {p} _ {j} (\boldsymbol {\eta})\right) + L _ {\mathrm {p}} ^ {j} \left(p _ {j} (\boldsymbol {\eta})\right)\right) ^ {2}} \cdot \sum_ {i = 1} ^ {n} \sqrt {\| \boldsymbol {\eta} (\boldsymbol {x} _ {i}) - \hat {\boldsymbol {\eta}} (\boldsymbol {x} _ {i}) \| _ {2} ^ {2}} \\ = \frac {\sqrt {m}}{n} B \sum_ {i = 1} ^ {n} \| \boldsymbol {\eta} (\boldsymbol {x} _ {i}) - \hat {\boldsymbol {\eta}} (\boldsymbol {x} _ {i}) \| _ {2}. \tag {70} \\ \end{array}
$$
For the other term, we can use the inequality between arithmetic and quadratic mean, so that
$$
\sum_ {j = 1} ^ {m} \left(L _ {\mathrm {t}} ^ {j} \left(\tilde {p} _ {j} (\boldsymbol {\eta})\right) + L _ {\mathrm {p}} ^ {j} \left(\tilde {p} _ {j} (\boldsymbol {\eta})\right)\right) \leq m \sqrt {m ^ {- 1} \sum_ {j = 1} ^ {m} \left(L _ {\mathrm {t}} ^ {j} \left(\tilde {p} _ {j} (\boldsymbol {\eta})\right) + L _ {\mathrm {p}} ^ {j} \left(\tilde {p} _ {j} (\boldsymbol {\eta})\right)\right) ^ {2}} = m B. \tag {71}
$$
C.4 Regret for non-approximated ETU
We can formulate the equivalent of Lemma C.4 also for the maximizer of the true empirical ETU risk,  $\hat{Y}^{\dagger}$ :
Lemma C.5 (Regret bound for ETU maximization). Let  $\varPsi\in\mathcal{L}_{\mathrm{OI}}$  be an instance-order invariant linearly decomposable loss function that is  $p$ -Lipschitz. Then we have
$$
\varPsi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}} ^ {\star}; \boldsymbol {X}) - \varPsi_ {\mathrm {E T U}} \bigl (\hat {\boldsymbol {Y}} ^ {\dagger}; \boldsymbol {X} \bigr) \leq \frac {1}{\sqrt {n}} \sum_ {j = 1} ^ {m} \bigl (L _ {\mathrm {t}} ^ {j} (\tilde {p} _ {j} (\boldsymbol {\eta})) + L _ {\mathrm {t}} ^ {j} (\tilde {p} _ {j} (\hat {\boldsymbol {\eta}})) + L _ {\mathrm {p}} ^ {j} (\tilde {p} _ {j} (\boldsymbol {\eta})) + L _ {\mathrm {p}} ^ {j} (\tilde {p} _ {j} (\hat {\boldsymbol {\eta}})) \bigr) +
$$
$$
2 \sum_ {j = 1} ^ {m} \frac {L _ {\mathrm {t}} ^ {j} (\tilde {p} _ {j} (\boldsymbol {\eta})) + L _ {\mathrm {p}} ^ {j} (p _ {j} (\boldsymbol {\eta}))}{n} \sum_ {i = 1} ^ {n} | \eta_ {j} (\boldsymbol {x} _ {i}) - \hat {\eta} _ {j} (\boldsymbol {x} _ {i}) |. \tag {72}
$$
Proof. Following the same line of argument as for Lemma C.4, we get
$$
\begin{array}{l} \varPsi_ {\mathrm {E T U}} (\hat {Y} ^ {\star}; \boldsymbol {\eta}) - \varPsi_ {\mathrm {E T U}} (\hat {Y} ^ {\dagger}; \boldsymbol {\eta}) = \varPsi_ {\mathrm {E T U}} (\hat {Y} ^ {\star}; \boldsymbol {\eta}) - \varPsi_ {\mathrm {E T U}} (\hat {Y} ^ {\star}; \hat {\boldsymbol {\eta}}) \\ + \Psi_ {\mathrm {E T U}} (\hat {\mathbf {Y}} ^ {\star}; \hat {\boldsymbol {\eta}}) - \Psi_ {\mathrm {E T U}} \left(\hat {\mathbf {Y}} ^ {\dagger}; \hat {\boldsymbol {\eta}}\right) \\ + \Psi_ {\mathrm {E T U}} (\hat {\mathbf {Y}} ^ {\dagger}; \hat {\boldsymbol {\eta}}) - \Psi_ {\mathrm {E T U}} (\hat {\mathbf {Y}} ^ {\dagger}; \boldsymbol {\eta}) \\ \leq \Psi_ {\text {E T U}} (\boldsymbol {Y} ^ {\star}; \boldsymbol {\eta}) - \Psi_ {\text {E T U}} (\hat {\boldsymbol {Y}} ^ {\star}; \hat {\boldsymbol {\eta}}) \\ + \Psi_ {\mathrm {E T U}} (\hat {\mathbf {Y}} ^ {\dagger}; \hat {\boldsymbol {\eta}}) - \Psi_ {\mathrm {E T U}} (\hat {\mathbf {Y}} ^ {\dagger}; \boldsymbol {\eta}) \\ \leq 2 \sup  _ {\hat {Y}} | \Psi_ {\mathrm {E T U}} (\hat {Y}; \boldsymbol {\eta}) - \Psi_ {\mathrm {E T U}} (\hat {Y}; \hat {\boldsymbol {\eta}}) |. \tag {73} \\ \end{array}
$$
Next, we make use of the semi-empirical ETU risk to bound
$$
\begin{array}{l} \left| \Psi_ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \boldsymbol {\eta}) - \Psi_ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \hat {\boldsymbol {\eta}}) \right| \leq \left| \Psi_ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \boldsymbol {\eta}) - \tilde {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \boldsymbol {\eta}) \right| \\ + \left| \tilde {\Psi} _ {\mathrm {E T U}} (\hat {Y}; \boldsymbol {\eta}) - \tilde {\Psi} _ {\mathrm {E T U}} (\hat {Y}; \hat {\eta}) \right| \\ + \left| \tilde {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \hat {\boldsymbol {\eta}}) - \Psi_ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \hat {\boldsymbol {\eta}}) \right| \tag {74} \\ \end{array}
$$
The individual terms can now again be bounded by Theorem 5.2 and Lemma C.3.
Algorithm 5 Sparse BCA  $(X,\hat{\eta}^{\mathrm{csr}},k,\epsilon)$    
1:  $\hat{\pmb{y}}_i^{\mathrm{csr}}\gets$  select-random-  $k(m)$  for all  $i\in [n]$    
2:  $\tilde{\pmb{t}}\gets \frac{1}{n}\sum_{i = 1}^{n}\hat{\eta}^{\mathrm{csr}}(\pmb{x}_i)\odot \hat{\pmb{y}}_i^{\mathrm{csr}}$    
3:  $\pmb {q}\gets \frac{1}{n}\sum_{i = 1}^{n}\hat{\pmb{y}}_i^{\mathrm{csr}}$    
4:  $\tilde{\pmb{p}}\gets \frac{1}{n}\sum_{i = 1}^{n}\hat{\eta}^{\mathrm{csr}}(\pmb{x}_i)$    
5:  $u_{old}\gets -\infty ,u_{new}\gets \Psi (\tilde{t},q,\tilde{p})$    
6: while  $u_{\mathrm{new}} > u_{\mathrm{old}} + \epsilon$  do   
7: for  $s\in$  shuffle([n]) do   
8:  $\tilde{\pmb{t}}\gets \tilde{\pmb{t}} -\frac{1}{n}\hat{\eta}^{\mathrm{csr}}(\pmb{x}_s)\odot \hat{\pmb{y}}_s^{\mathrm{csr}}$    
9:  $\pmb {q}\gets \pmb {q} - \frac{1}{n}\hat{\pmb{y}}_{s}^{\mathrm{csr}}$    
10:  $\pmb{g}^{\mathrm{csr}}\gets \emptyset$    
11: for  $(j,\hat{\eta}_j(\pmb {x}_s))\in \hat{\eta}^{\mathrm{csr}}(\pmb {x}_i)$  do   
12:  $\psi^j (1)\gets \psi (\tilde{t}_j + \frac{1}{n}\hat{\eta}_j(\pmb {x}_s),q_j + \frac{1}{n},\tilde{p}_j)$    
13:  $\psi^j (0)\gets \psi (\tilde{t}_j,q_j,\tilde{p}_j)$    
14:  $\pmb{g}^{\mathrm{csr}}\gets \pmb{g}^{\mathrm{csr}}\cup \{(j,\psi^j (1) - \psi^j (0))\}$    
15:  $\hat{\pmb{y}}_s^{\mathrm{csr}}\gets$  select-top- $k(g^{\mathrm{csr}})$    
16:  $\tilde{\pmb{t}}\gets \tilde{\pmb{t}} +\frac{1}{n}\hat{\eta}^{\mathrm{csr}}(\pmb{x}_s)\odot \hat{\pmb{y}}_s^{\mathrm{csr}}$    
17:  $\pmb {q}\gets \pmb {q} + \frac{1}{n}\hat{\pmb{y}}_s^{\mathrm{csr}}$    
18:  $u_{old}\gets u_{\mathrm{new}},u_{\mathrm{new}}\gets \varPsi (\tilde{t},\pmb {q},\tilde{\pmb{p}})$    
19: return YCSR
Algorithm 6 Sparse BCA for coverage  $(X,\hat{\eta}^{\mathrm{csr}},k,\epsilon)$    
1:  $\hat{y}_i^{\mathrm{csr}}\gets$  select-random-  $k(m)$  for all  $i\in [n]$    
2:  $f\gets 1$    
3: for  $i\in [n]$  do   
4: for  $(j,\hat{\eta}_j(\pmb {x}_i))\in \hat{\eta}^{\mathrm{csr}}(\pmb {x}_i)\odot \hat{\pmb{y}}_i$  do   
5:  $f_{j}\gets f_{j}(1 - \hat{\eta}_{j}(\pmb {x}_{i}))$    
6:  $u_{old}\gets -\infty$ $u_{\mathrm{new}}\gets 1 - (m^{-1}\sum_{j = 1}^{m}f_{j})$    
7: while  $u_{\mathrm{new}} > u_{\mathrm{old}} + \epsilon$  do   
8: for  $s\in$  shuffle([n]) do   
9: for  $(j,\hat{\eta}_j(\pmb {x}_s))\in \hat{\eta}^{\mathrm{csr}}(\pmb {x}_s)\odot \hat{\pmb{y}}_s^{\mathrm{csr}}$  do   
10:  $f_{j}\gets f_{j} / (1 - \hat{\eta}_{j}^{\mathrm{csr}}(\pmb {x}_{s}))$    
11:  $g^{\mathrm{csr}}\gets \emptyset$    
12: for  $(j,\hat{\eta}_j(\pmb {x}_s))\in \hat{\eta}^{\mathrm{csr}}(\pmb {x}_s)$  do   
13:  $g^{\mathrm{csr}}\gets g^{\mathrm{csr}}\cup \{(j,\hat{\eta}_j(\pmb {x}_s)f_j)\}$    
14:  $\hat{y}_s^{\mathrm{csr}}\gets$  select-top-  $k(g^{\mathrm{csr}})$    
15: for  $(j,\hat{\eta}_j(\pmb {x}_s))\in \hat{\eta}^{\mathrm{csr}}(\pmb {x}_s)\odot \hat{\pmb{y}}_s^{\mathrm{csr}}$  do   
16:  $f_{j}\gets f_{j}(1 - \hat{\eta}_{j}(\pmb {x}_{s}))$    
17:  $u_{old}\gets u_{\mathrm{new}},u_{\mathrm{new}}\gets 1 - (m^{-1}\sum_{j = 1}^{m}f_{j})$    
18: return  $\hat{Y}^{\mathrm{csr}}$
Algorithm 7 Sparse greedy  $(X,\hat{\eta}^{\mathrm{csr}},k)$    
1:  $\tilde{t}\gets 0$ $q\gets 0$ $\tilde{p}\gets 0$    
2: for  $i\in [n]$  do   
3:  $\tilde{\pmb{p}}\leftarrow \pmb {p} + \frac{1}{n}\hat{\eta}^{\mathrm{csr}}(\pmb {x}_i)$    
4:  $\pmb{g}^{\mathrm{csr}}\gets \emptyset$    
5: for  $(j,\hat{\eta}_{j}(\pmb{x}_{i}))\in \hat{\eta}^{\mathrm{csr}}(\pmb{x}_{i})$  do   
6:  $\psi^{j}(1)\gets \psi (\tilde{t}_{j} + \frac{1}{n}\hat{\eta}_{j}(\pmb{x}_{i}),q_{j} + \frac{1}{n},\tilde{p}_{j})$    
7:  $\psi^j (0)\gets \psi (\tilde{t}_j,q_j,\tilde{p}_j)$    
8:  $\pmb{g}^{\mathrm{csr}}\gets \pmb{g}^{\mathrm{csr}}\cup \{(j,\psi^j (1) - \psi^j (0))\}$    
9:  $\hat{\pmb{y}}_i^{\mathrm{csr}}\gets$  select top-k  $(g^{\mathrm{csr}})$    
10:  $\tilde{t}\gets \tilde{t} +\frac{1}{n}\hat{\eta}^{\mathrm{csr}}(\pmb{x}_i)\odot \hat{\pmb{y}}_i^{\mathrm{csr}}$    
11:  $q\gets q + \frac{1}{n}\hat{y}_i^{\mathrm{csr}}$    
12: return YCSR
Algorithm 8 Sparse greedy for coverage  $(X,\hat{\eta}^{\mathrm{csr}},k)$    
1:  $f\gets 1$    
2: for  $i\in [n]$  do   
3:  $\pmb{g}^{\mathrm{csr}}\gets \emptyset$    
4: for  $(j,\hat{\eta}_j(\pmb {x}_s))\in \hat{\eta}^{\mathrm{csr}}(\pmb {x}_i)$  do   
5:  $\pmb{g}^{\mathrm{csr}}\leftarrow \mathbf{c s r}\pmb {g}\cup \{(j,\hat{\eta}_j(\pmb {x}_s)f_j)\}$    
6:  $\hat{\pmb{y}}_i^{\mathrm{csr}}\gets$  select top-  $\mathbf{k}(\pmb{g}^{\mathrm{csr}})$    
7: for  $(j,\hat{\eta}_j(\pmb {x}_i))\in \hat{\eta}^{\mathrm{csr}}(\pmb {x}_i)\odot \hat{\pmb{y}}_i^{\mathrm{csr}}$  do   
8:  $f_{j}\gets f_{j}(1 - \hat{\eta}_{j}(\pmb{x}_{i}))$    
9: return  $\hat{Y}^{\mathrm{csr}}$
D Sparse and greedy algorithms
In this section, we present pseudocodes for sparse variants of the introduced algorithms, which we used in the main experiment to compute efficiently the results for large datasets. These variants use compressed sparse row (CSR) representation for row vectors. CSR vectors are represented as a list of tuples  $\boldsymbol{a}_i^{\mathrm{csr}} \coloneqq \{(\mathrm{index}, \mathrm{value}) : \mathrm{value} \neq 0\}$ . In these algorithms, we replace  $\hat{\eta}(\boldsymbol{x}_i)$  and  $\hat{\boldsymbol{y}}_i$  with their sparse variants,  $\hat{\eta}^{\mathrm{csr}}(\boldsymbol{x}_i) \coloneqq \{(j, \hat{\eta}_j(\boldsymbol{x}_i)) : \hat{\eta}_j(\boldsymbol{x}_i) \neq 0\}$  and  $\hat{\boldsymbol{y}}_i^{\mathrm{csr}} \coloneqq \{(j, \hat{y}_{ij}) : \hat{y}_{ij} \neq 0\}$ , respectively. We need to ensure that of both  $\hat{\eta}^{\mathrm{csr}}(\boldsymbol{x}_i)$  and  $\hat{\boldsymbol{y}}_i^{\mathrm{csr}}$  are ordered by their indices for efficient element-wise multiplication (Hadamard product) between them. If we assume that the size of  $|\hat{\boldsymbol{y}}_i^{\mathrm{csr}}| = k$  and  $|\hat{\boldsymbol{\eta}}^{\mathrm{csr}}(\boldsymbol{x}_i)| = k'$ , the resulting time and space complexity of the sparse algorithms are  $O(n(k' + k\log k))$  and  $O(n(k' + k))$ , respectively. Additionally, we present the greedy variants of sparse BCA algorithms introduced so far that do only one pass over the dataset.