SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We show in Theorem A.3 that any utility function  $\varPsi(\mathbf{Y},\hat{\mathbf{Y}})$  that is invariant under instance reordering can be defined in terms of confusion matrices. In this way, we justify our choice in the main paper to focus on losses of the form given in (4).
More precisely, let  $\sigma \in \mathfrak{S}(n)$  be a permutation of rows, that is, for  $\mathbf{Y} = [\pmb{y}_1,\dots ,\pmb{y}_n]^{\top}$ , we define  $\sigma \mathbf{Y} = [\pmb{y}_{\sigma (1)},\dots ,\pmb{y}_{\sigma (n)}]^{\top}$ . Then we can make the following definition:
Definition A.1 (Invariant under instance reordering). Let  $n, m \in \mathbb{N}$  and  $\varPsi: \{0, 1\}^{n \times m} \times \{0, 1\}^{n \times m} \longrightarrow \mathbb{R}$  be a function of discrete labels and predictions. If  $\varPsi$  remains unchanged for every permutation of rows  $\sigma \in \mathfrak{S}(n)$ , i.e.,
$$
\varPsi (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) = \varPsi (\sigma \boldsymbol {Y}, \sigma \hat {\boldsymbol {Y}}), \tag {28}
$$
then we call  $\varPsi$  a function that is invariant under instance reordering. We denote the set of instance-order-invariant losses with  $m$  labels as
$$
\mathcal {I} _ {m} := \left\{\Psi : \{0, 1 \} ^ {n \times m} \times \{0, 1 \} ^ {n \times m} \longrightarrow \mathbb {R}: \Psi \text {i s i n v a r i a n t u n d e r i n s t a n c e r e o r d e r i n g} \right\}. \tag {29}
$$
We further define the set of all possible confusion matrices with  $n$  instances as
$$
\mathcal {C} (n) := \left\{\boldsymbol {C}: n \boldsymbol {C} \in \mathbb {N} ^ {2 \times 2}, \| \boldsymbol {C} \| _ {1, 1} = 1 \right\}, \tag {30}
$$
where  $\mathbb{N}$  is the set of natural numbers including 0. Now we are ready to provide a lemma for the binary case:
Lemma A.2. Let  $\psi \colon \{0,1\}^n \times \{0,1\}^n \longrightarrow \mathbb{R}$  be a binary loss function that is invariant under instances reordering. Then there exists a function  $\phi \colon \mathcal{C}(n) \longrightarrow \mathbb{R}$  such that  $\psi = \phi \circ C$ .
Proof. We provide an explicit construction of  $\psi$ . To that end, let  $\hat{C} \in \mathcal{C}(n)$  be one such confusion matrix, then there exists  $(\pmb{y}, \hat{\pmb{y}}) \in \{0, 1\}^n \times \{0, 1\}^n$  given by
$$
(\boldsymbol {y}, \hat {\boldsymbol {y}}) = [ \underbrace {(1 , 1) , \dots , (1 , 1)} _ {\times n \cdot \tilde {c} _ {1 1}}, \underbrace {(0 , 1) , \dots , (0 , 1)} _ {\times n \cdot \tilde {c} _ {0 1}}, \underbrace {(1 , 0) , \dots , (1 , 0)} _ {\times n \cdot \tilde {c} _ {1 0}}, \underbrace {(0 , 0) , \dots , (0 , 0)} _ {\times n \cdot \tilde {c} _ {0 0}} ] ^ {\top}. \tag {31}
$$
Define  $\phi$  such that  $\phi (\tilde{\pmb{C}}) = \psi (\pmb {y},\hat{\pmb{y}})$
Now, let  $(\pmb{y}', \hat{\pmb{y}}') \in \{0,1\}^n \times \{0,1\}^n$  be an arbitrary label-prediction combination, with  $C(\pmb{y}', \hat{\pmb{y}}') = \tilde{C}$ . Then there exists a permutation  $\sigma$  such that  $\sigma \pmb{y}' = \pmb{y}$  and  $\sigma \hat{\pmb{y}}' = \hat{\pmb{y}}$ . By the invariance assumption, it holds that
$$
\psi \left(\boldsymbol {y} ^ {\prime}, \hat {\boldsymbol {y}} ^ {\prime}\right) = \psi \left(\sigma \boldsymbol {y} ^ {\prime}, \sigma \hat {\boldsymbol {y}} ^ {\prime}\right) = \psi (\boldsymbol {y}, \hat {\boldsymbol {y}}) = \phi (\tilde {\boldsymbol {C}}) = \phi \left(\boldsymbol {C} \left(\boldsymbol {y} ^ {\prime}, \hat {\boldsymbol {y}} ^ {\prime}\right)\right). \tag {32}
$$
As the original  $\tilde{C} \in \mathcal{C}(n)$  was arbitrary, the statement is shown.
We can now extend this lemma to show the equivalence of two definitions of the task losses considered in this paper:
Theorem A.3 (Equivalence of order-invariance and confusion-matrix losses). Let  $n, m \in \mathbb{N}$ , and  $\mathcal{V} = \{0,1\}^{m}$ . Define the set of instance-order invariant, label-averaged losses as
$$
\mathcal {L} _ {\mathrm {O I}} := \left\{\Psi \in \mathcal {I} _ {m}: (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) \mapsto \sum_ {j = 1} ^ {m} \psi^ {j} \left(\boldsymbol {y} _ {: j}, \hat {\boldsymbol {y}} _ {: j}\right) \right\}, \tag {33}
$$
and the set of confusion-matrix based, label-averaged losses as
$$
\mathcal {L} _ {\mathrm {C M}} := \left\{\Phi : \mathcal {C} (n) ^ {m} \longrightarrow \mathbb {R} s. t. \mathbf {C} \mapsto \sum_ {j = 1} ^ {m} \phi^ {j} \left(\boldsymbol {C} ^ {j}\right) \right\}. \tag {34}
$$
Then these two descriptions are equivalent, in the sense that
$$
\mathcal {L} _ {\mathrm {O I}} = \left\{\Phi \circ \mathbf {C}: \Phi \in \mathcal {L} _ {\mathrm {C M}} \right\}, \tag {35}
$$
that is, every instance-order invariant loss can be written as a confusion-matrix loss, and every confusion-matrix loss leads to an instance-order invariant loss.
Proof. As calculating the confusion matrix is in itself an operation that is invariant under instance reordering, each  $\Phi \circ \mathbf{C}$  clearly is instance-order invariant.
On the other hand, let  $\varPsi(\boldsymbol{Y},\hat{\boldsymbol{Y}})=\sum_{j=1}^{m}\psi^{j}(\boldsymbol{y}_{:j},\hat{\boldsymbol{y}}_{:j})$ , then by Lemma A.2 there exist  $\phi^1,\ldots,\phi^m$  such that
$$
\psi^ {j} \left(\boldsymbol {y} _ {: j}, \hat {\boldsymbol {y}} _ {: j}\right) = \phi^ {j} \left(\boldsymbol {C} \left(\boldsymbol {y} _ {: j}, \hat {\boldsymbol {y}} _ {: j}\right)\right). \tag {36}
$$
Summing up the individual  $\phi^j$  gives  $\Phi$  of the correct form.
A.2 Sufficiency of label probability estimates
If  $\Psi \in \mathcal{L}_{\mathrm{OI}}$  is an instance-order invariant decomposable task utility, then its ETU risk is a function of the predictions  $\hat{\pmb{Y}}$  and the marginal label probabilities  $\eta (\pmb {x})$  as given by (10). This means that it is not required to estimate the  $m^2 -1$  values of the full joint probability distribution  $\mathbb{P}[\pmb {Y}\mid \pmb {x}]$
Lemma A.4. Let  $\Psi \in \mathcal{L}_{\mathrm{OI}}$  be an instance-order invariant decomposable task utility, and assume that the labels of different instances are independent, i.e., that  $\mathbb{P}(\boldsymbol {Y}|\boldsymbol {X}) = \prod_{i = 1}^{n}\mathbb{P}(\boldsymbol {y}_i|\boldsymbol {x}_i)$ . Then we have
$$
\mathbb {E} _ {\boldsymbol {Y} \mid \boldsymbol {X}} [ \Psi (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) ] = \sum_ {j = 1} ^ {m} \sum_ {\boldsymbol {y} ^ {\prime} \in \{0, 1 \} ^ {n}} \left(\prod_ {i = 1} ^ {n} \eta_ {j} (\boldsymbol {x} _ {i}) y _ {i} ^ {\prime} + (1 - \eta_ {j} (\boldsymbol {x} _ {i})) (1 - y _ {i} ^ {\prime})\right) \psi^ {j} \left(\boldsymbol {y} ^ {\prime}, \hat {\boldsymbol {y}} _ {: j}\right). \tag {37}
$$
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} [ \Psi (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) ] = \sum_ {j = 1} ^ {m} \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ \psi^ {j} (\boldsymbol {y} _ {: j}, \hat {\boldsymbol {y}} _ {: j}) \right] = \sum_ {j = 1} ^ {m} \sum_ {\boldsymbol {y} ^ {\prime} \in \{0, 1 \} ^ {n}} \mathbb {P} [ \boldsymbol {y} _ {: j} = \boldsymbol {y} ^ {\prime} | \boldsymbol {X} ] \psi^ {j} (\boldsymbol {y} ^ {\prime}, \hat {\boldsymbol {y}} _ {: j}) \\ = \sum_ {j = 1} ^ {m} \sum_ {\boldsymbol {y} ^ {\prime} \in \{0, 1 \} ^ {n}} \left(\prod_ {i = 1} ^ {n} \mathbb {P} \left[ Y _ {i j} = y _ {i} ^ {\prime} \mid \boldsymbol {x} _ {i} \right]\right) \psi^ {j} \left(\boldsymbol {y} ^ {\prime}, \hat {\boldsymbol {y}} _ {: j}\right) \\ = \sum_ {j = 1} ^ {m} \sum_ {\boldsymbol {y} ^ {\prime} \in \{0, 1 \} ^ {n}} \left(\prod_ {i = 1} ^ {n} \eta_ {j} \left(\boldsymbol {x} _ {i}\right) y _ {i} ^ {\prime} + \left(1 - \eta_ {j} \left(\boldsymbol {x} _ {i}\right)\right) \left(1 - y _ {i} ^ {\prime}\right)\right) \psi^ {j} \left(\boldsymbol {y} ^ {\prime}, \hat {\boldsymbol {y}} _ {: j}\right). \tag {38} \\ \end{array}
$$
A.3 Representation change from the instance-wise form to the cost-matrix form
If we start with the linear metric given in the instance-wise form according to (5), the corresponding representation according to (15) may be shifted by a constant value, and can be derived through the following:
$$
\begin{array}{l} c _ {1 1} w _ {1 1} + c _ {0 1} w _ {0 1} + c _ {1 0} w _ {1 0} + c _ {0 0} w _ {0 0} = c _ {1 1} w _ {1 1} + c _ {0 1} w _ {0 1} + (p - t) w _ {1 0} + (1 - p - c _ {0 1}) w _ {0 0} \\ = t \left(w _ {1 1} - w _ {1 0}\right) + c _ {0 1} \left(w _ {0 1} - w _ {0 0}\right) + p w _ {1 0} + (1 - p) w _ {0 0} \\ = t \left(w _ {1 1} - w _ {1 0} - w _ {0 1} + w _ {0 0}\right) + q \left(w _ {0 1} - w _ {0 0}\right) + \text {c o n s t}. \tag {39} \\ \end{array}
$$
Thus, we get  $f_{\mathfrak{t}}(p) = w_{11} - w_{10} - w_{01} + w_{00}$  and  $f_{\mathfrak{q}}(p) = w_{01} - w_{00}$ .
B Block Coordinate Ascent
In this section, we provide additional analysis and variations of the block coordinate ascent algorithm. We first give an intuition into the gain-based selection of labels performed by the algorithm in the case of macro-precision. Then, we will illustrate how, for metrics that are linear in the prediction argument as per (15), the algorithm will recover the closed-form solution given in (19). Afterwards, we present an alternative formulation specifically for coverage, and conclude with greedy versions of the BCA algorithm.
B.1 Example: Macro-Precision
As an example, consider the optimization of macro-precision. In this case,  $\psi^j (t,q,p) = t / q$ . Plugging into the gain formula gives
$$
g _ {j} = \frac {t ^ {j} + \hat {\eta} _ {j} (\boldsymbol {x} _ {s})}{q ^ {j} + 1} - \frac {t ^ {j}}{q ^ {j}} = \frac {p ^ {j} \left(t ^ {j} + \hat {\eta} _ {j} (\boldsymbol {x} _ {s})\right) - t ^ {j} \left(q ^ {j} + 1\right)}{q ^ {j} \left(q ^ {j} + 1\right)} = \frac {q ^ {j} \hat {\eta} _ {j} (\boldsymbol {x} _ {s}) - t ^ {j}}{q ^ {j} \left(q ^ {j} + 1\right)}. \tag {40}
$$
This term is positive only if  $t^j / q^j < \hat{\eta}_j(\pmb{x}_s)$ , i.e., predicting the label has a positive impact on the overall utility if its chance of being relevant,  $\hat{\eta}_j(\pmb{x}_s)$ , is larger than the current estimate of precision for label  $j$ . Of course, the "at- $k$ " constraint means that, for a given instance, it may not be possible to select all labels with positive gain, or necessary to select some labels with negative gain to fulfill the constraint.
B.2 Global optimality for linear metrics
If  $\psi^j$  is a linear function, i.e., it corresponds to an instance-wise measure as in (6), the gain  $g$ , defined through lines 9-11 in Algorithm 1, becomes
$$
\begin{array}{l} \psi^ {j} (t + \hat {\eta} _ {j} (\boldsymbol {x} _ {s}), q + 1, p) - \psi^ {j} (t, q, p) = (t + \hat {\eta} _ {j} (\boldsymbol {x} _ {s}) - t) f _ {\mathrm {t}} (p) + (q + 1 - q) f _ {\mathrm {q}} (p) \\ = \hat {\eta} _ {j} \left(\boldsymbol {x} _ {s}\right) f _ {\mathrm {t}} (p) + f _ {\mathrm {q}} (p), \tag {41} \\ \end{array}
$$
because the influence of the other instances cancels out. This exactly reproduces the gain formula from Section 4.2, which means that for each instance, the block coordinate-ascent algorithm will select the globally (approximately) optimal decision on the first pass. If the metric is fully linear, this will be the Bayes-optimal decision, otherwise, it is approximately optimal in the sense of Theorem 5.2. In both cases, the algorithm will perform a second pass over the entire dataset in order to check the stopping criterion. This second pass will leave all predictions unchanged.
B.3 The ETU approach for coverage
In this section, we derive the ETU risk presented initially in Section 4.4. Then, we present a block coordinate ascent algorithm for optimizing it. The step-by-step derivation of (24) is given below:
$$
\begin{array}{l} \left. \Psi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {X}) = \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ \Psi_ {\operatorname {c o v}} (\boldsymbol {t} (\boldsymbol {Y}, \hat {\boldsymbol {Y}}), \boldsymbol {q} (\boldsymbol {Y}, \hat {\boldsymbol {Y}}), \boldsymbol {p} (\boldsymbol {Y}, \hat {\boldsymbol {Y}})) \right] \right. \\ = \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ \frac {1}{m} \sum_ {j = 1} ^ {m} \mathbb {1} [ t _ {j} > 0 ] \right] = \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ \frac {1}{m} \sum_ {j = 1} ^ {m} (1 - \mathbb {1} [ t _ {j} = 0 ]) \right] \\ = \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ \frac {1}{m} \sum_ {j = 1} ^ {m} \left(1 - \prod_ {i = 1} ^ {j} (1 - y _ {i j} \hat {y} _ {i j})\right) \right] = \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ 1 - \frac {1}{m} \sum_ {j = 1} ^ {m} \prod_ {i = 1} ^ {j} (1 - y _ {i j} \hat {y} _ {i j}) \right]. \tag {42} \\ \end{array}
$$
Because we assume labels for one instance to be independent on all other instances, i.e.,  $\mathbb{P}(\mathbf{Y}|\mathbf{X}) = \prod_{i=1}^{n}\mathbb{P}(\pmb{y}_i|\pmb{x}_i)$ , we obtain:
$$
\begin{array}{l} \varPsi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {X}) = 1 - \frac {1}{m} \sum_ {j = 1} ^ {m} \prod_ {i = 1} ^ {n} \left(1 - \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} [ y _ {i j} \hat {y} _ {i j} ]\right) = 1 - \frac {1}{m} \sum_ {j = 1} ^ {m} \prod_ {i = 1} ^ {n} \left(1 - \mathbb {P} [ y _ {i j} = 1 | \boldsymbol {x} _ {i} ]\right) \\ = 1 - \frac {1}{m} \sum_ {j = 1} ^ {m} \prod_ {i = 1} ^ {n} \left(1 - \eta_ {j} \left(\boldsymbol {x} _ {i}\right) \hat {y} _ {i j}\right). \tag {43} \\ \end{array}
$$
Based on this result we can construct a block coordinate ascent procedure which, for predictions being fixed for all instances except  $\boldsymbol{x}_s$ , optimizes the following problem:
$$
\max  _ {\boldsymbol {z} \in \mathcal {Y} _ {k}} \sum_ {j = 1} ^ {m} \psi_ {\operatorname {c o v}} ^ {j} \left(z _ {j}\right) = \sum_ {j = 1} ^ {m} \left(1 - \left(1 - \eta_ {j} \left(\boldsymbol {x} _ {s}\right) z _ {j}\right) \prod_ {i \in [ n ] \backslash \{s \}} \left(1 - \eta_ {j} \left(\boldsymbol {x} _ {i}\right) \hat {y} _ {i j}\right)\right). \tag {44}
$$
Analogously to Section 4.3, everything except  $z_{j}\in \{0,1\}$  is given and we can define a gain vector with elements  $g_{j} = \psi_{\mathrm{cov}}^{j}(1) - \psi_{\mathrm{cov}}^{j}(0)$ . Again the optimal prediction  $\pmb{z}^*$  is then given by  $\pmb{z}^{*} =$  select-top-  $k(g)$  , and we get  $\hat{\pmb{Y}}^{t + 1}$  by replacing the  $s^{\mathrm{th}}$  row of  $\hat{\pmb{Y}}^t$  with  $\pmb{z}^{*}$  . Then we switch to the next instance  $s\gets s + 1$  , and repeat this process until no more progress is made.
Notice that  $\prod_{i=1}^{n}\left(1 - \eta_j(\pmb{x}_i)\hat{y}_{ij}\right)$  corresponds to the probability of label  $j$  being irrelevant for all instances it was selected for. We denote this value as  $f_j$  and use it to speed up computations in each iteration of the algorithm:
Algorithm 2 BCA for coverage(X,ˆη,k,ε)
1:ˆy_i← select-random-k(m) for all i ∈ [n]
2: f← ∏i=1n(1-ˆη(xi) ⊙ˆy_i)
3: uold← -∞, unew← 1-(m-1∑j=1m f_j)
4: while u_new > u_old + ε do
5: for s ∈ shuffle([n]) do
6: f← f ⊙ (1-ˆη(xi) ⊙ˆy_s)-1
7: g← f ⊙ˆη(xi_s)
8:ˆy_s← select-top-k(g)
9: f← f ⊙ (1-ˆη(xi) ⊙ˆy_s)
10: uold← u_new, u_new← 1-(m-1∑j=1m f_j)
11: returnˆY
$$
f _ {j} ^ {0} := \prod_ {i = 1} ^ {n} \left(1 - \eta_ {j} \left(\boldsymbol {x} _ {i}\right) \hat {y} _ {i j} ^ {0}\right), \quad f _ {j} ^ {t + 1} := f _ {j} ^ {t} \frac {1 - \eta_ {j} \left(\boldsymbol {x} _ {s}\right) \hat {y} _ {s j} ^ {t + 1}}{1 - \eta_ {j} \left(\boldsymbol {x} _ {s}\right) \hat {y} _ {s j} ^ {t}}. \tag {45}
$$
We can then compute the gain vector using the following formula:
$$
g _ {j} = \left(1 - f _ {j} ^ {t} \frac {1 - \eta_ {j} \left(\boldsymbol {x} _ {s}\right)}{1 - \eta_ {j} \left(\boldsymbol {x} _ {s}\right) \hat {\mathcal {Y}} _ {s j} ^ {t}}\right) - \left(1 - f _ {j} ^ {t} \frac {1}{1 - \eta_ {j} \left(\boldsymbol {x} _ {s}\right) \hat {\mathcal {Y}} _ {s j} ^ {t}}\right) = \frac {\eta_ {j} \left(\boldsymbol {x} _ {s}\right) f _ {j} ^ {t}}{1 - \eta_ {j} \left(\boldsymbol {x} _ {s}\right) \hat {\mathcal {Y}} _ {s j} ^ {t}}. \tag {46}
$$
This block coordinate ascent procedure for coverage is presented as Algorithm 2.
B.4 Greedy version of block coordinate-ascent
The inference strategy presented in Algorithm 1 requires multiple passes over the entire data. This might be computationally expensive and requires the whole dataset to be available at once. We, thus, propose a greedy version of the algorithm, that performs just a single pass over the dataset and makes decisions for each instance by taking into account only statistics of previously seen instances instead of all other instances in the dataset. This allows to apply this algorithm to the semi-online setting, where the algorithm observes and predicts for instances one by one, but does not receive immediate feedback for its decisions, in contrast to the usual online learning setting. The procedure is outlined as Algorithm 3, and its greedy variant as Algorithm 4. The results of this algorithm in comparison to the BCA approach are given later in Appendix E.
Algorithm 3 Greedy (X, η, k)
1:	til← 0, q← 0, p← 0
2:	for i ∈ [n] do
3:	til← p + 1/n η(xi)
4:	for j ∈ [m] do
5:	ψj(1)← ψ(til + 1/n ηj(xi), qj + 1/n, pj)
6:	ψj(0)← ψ(til, qj, pj)
7:	gj← ψj(1) - ψj(0)
8:	ŷi← select top-k(g)
9:	til←til + 1/n η(xi) ⊙ŷi
10:	q← q + 1/nŷi
11:	return ŷ
Algorithm 4 Greedy for coverage (X, η, k)
1: f← 1
2: for i ∈ [n] do
3: g← f ⊙ η(xi)
4:帽子i← select top-k(g)
5: f← f ⊙ (1 - η(xi) ⊙帽子i)
6: return Y
C Detailed Regret Analysis
In this section, we discuss the  $p$ -Lipschitzness condition of the metrics of interest, and prove Theorem 5.2 and Theorem 5.3.
C.1  $p$ -Lipschitz utility functions
First, recall the definition from the main paper:
The rationale behind this definition is that while we need to control the change in value of the metric under small changes in its arguments, a standard definition of Lipschitzness (with global constant) would not be satisfied by many popular metrics. For the same reason, we only require stability for non-trivial problems, that is, in cases where the rate of positives  $p$  is neither zero nor one.
Relaxing the definition to allow the constants vary as a function of  $p$  suffices to prove our stability results as well as regret bounds, while it is satisfied by most of the metrics of interest, as shown below. The notable exception not present in Table 4 is the precision metric, which is not  $p$ -Lipschitz due to its behavior for  $q \to 0$ .
Lemma C.1. The linear confusion-matrix measures defined by (6):
Table 4: Examples of  $p$ -Lipschitz metrics. We use tp, fp, fn, and tn to denote true positives, false positives, false negatives, and true negatives.
Metric	Definition	ψ(t,q,p)
Accuracy	tp + tn	1 + 2t - q - p
Recall	tp / tp + fn	t / p
Bal. Acc.	tp/2 / tp + fn + tn + fp	t+p(1-q-p) / 2p(1-p)
Fβ	(1+β2)tp / (1+β2)tp + β2fn + fp	(1+β2)t / β2p + q
Jaccard	tp / tp + fp + fn	p+q-2t / p+q-t
G-Mean	√tp·tn / (tp + fn)(tn + fp)	t(1-q-p+t) / p(1-p)
AUC	fp·fn / (tp + fn)(fp + tn)	(q-t)(p-t) / p(1-p)
$$
\varPsi (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) = \sum_ {j = 1} ^ {m} \left(w _ {0 0} ^ {j} c _ {0 0} ^ {j} + w _ {0 1} ^ {j} c _ {0 1} ^ {j} + w _ {1 0} ^ {j} c _ {1 0} ^ {j} + w _ {1 1} ^ {j} c _ {1 1} ^ {j}\right) \tag {47}
$$
with fixed coefficient matrices  $\{\pmb{W}^j\}_{j=1}^m$  are decomposable functions with  $p$ -Lipschitz components.
Proof. The metric can be rewritten in a decomposable form  $\varPsi=\sum_{j}\psi^{j}$ , where each  $\psi^j$  in the  $(t,q,p)$ -parameterization has the following form:
$$
\psi^ {j} (t, q, p) = T _ {j} \cdot t + Q _ {j} \cdot q + P _ {j} \cdot p + C _ {j} \tag {48}
$$
where  $T_{j}, Q_{j}, P_{j}, C_{j}$  are some combinations of the coefficient matrices  $\left\{\boldsymbol{W}^{j}\right\}_{j=1}^{m}$ . Being a linear function of  $t, q, p, \psi^{j}$  is Lipschitz.
Lemma C.2 ( $p$ -Lipschitzness of common utilities [13]). All metrics in Table 4 are  $p$ -Lipschitz.
Proof. Here, we only prove the lemma for recall, which has not been covered by Proposition 1 of Dembczyński et al. [13].
$$
\begin{array}{l} | \psi (t, q, p) - \psi (t ^ {\prime}, q ^ {\prime}, p ^ {\prime}) | = \left| \frac {t}{p} - \frac {t ^ {\prime}}{p ^ {\prime}} \right| = \left| \frac {t - t ^ {\prime}}{p} + \frac {t ^ {\prime}}{p ^ {\prime}} \frac {p ^ {\prime} - p}{p} \right| \\ \leq \underbrace {\frac {1}{p}} _ {= L _ {\mathrm {t}} (p)} | t - t ^ {\prime} | + \underbrace {\frac {t ^ {\prime}}{p ^ {\prime}}} _ {\leq 1} \cdot \underbrace {\frac {1}{p}} _ {= L _ {\mathrm {p}} (p)} | p - p ^ {\prime} |. \tag {49} \\ \end{array}
$$
C.2 Stability of the semi-ETU approximation
We are now ready to prove that when the metric of interest has  $p$ -Lipschitz components, the semi-ETU approximation  $\bar{\Psi}_{\mathrm{ETU}}$  presented in Section 4.1 remains close to the true objective  $\bar{\Psi}_{\mathrm{ETU}}$ . For the sake of convenience, let us recall the definition of the ETU objective:
$$
\varPsi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {X}) = \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} [ \varPsi (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) ] = \sum_ {j = 1} ^ {m} \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ \psi^ {j} \left(t \left(\boldsymbol {y}: j, \hat {\boldsymbol {y}}: j\right), q \left(\hat {\boldsymbol {y}}: j\right), p \left(\boldsymbol {y}: j\right)\right) \right], \tag {50}
$$
as well as its approximation:
$$
\tilde {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \mathbf {X}) = \sum_ {j = 1} ^ {m} \psi^ {j} \left(\mathbb {E} _ {\mathbf {Y} \mid \mathbf {X}} [ t (\mathbf {y}: j, \hat {\mathbf {y}}: j) ], q (\hat {\mathbf {y}}: j), \mathbb {E} _ {\mathbf {Y} \mid \mathbf {X}} [ p (\mathbf {y}: j) ]\right). \tag {51}
$$
We prove the following result:
$$
\left| \Psi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {X}) - \tilde {\Psi} _ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {X}) \right| \leq \frac {1}{2 \sqrt {n}} \left(\sum_ {j = 1} ^ {m} \left(L _ {\mathrm {t}} ^ {j} \left(\tilde {p} _ {j}\right) + L _ {\mathrm {p}} ^ {j} \left(\tilde {p} _ {j}\right)\right)\right). \tag {26}
$$
Proof. For the sake of the analysis, denote the Lipschitz constants as  $L_{\mathfrak{t}}^{j} \coloneqq L_{\mathfrak{t}}^{j}(\tilde{p}_{j})$  and  $L_{\mathfrak{p}}^{j} \coloneqq L_{\mathfrak{p}}^{j}(\tilde{p}_{j})$ . Using definitions (50) and (51) and applying Jensen's inequality, we have
$$
\begin{array}{l} | \Psi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {X}) - \tilde {\Psi} _ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}; \boldsymbol {X}) | = \left| \sum_ {j = 1} ^ {m} \left(\mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ \psi^ {j} (t _ {j}, q _ {j}, p _ {j}) \right] - \psi^ {j} (\tilde {t} _ {j}, q _ {j}, \tilde {p} _ {j})\right) \right| \\ \leq \sum_ {j = 1} ^ {m} \mathbb {E} _ {\boldsymbol {Y} \mid \boldsymbol {X}} \left[ \left| \psi^ {j} \left(t _ {j}, q _ {j}, p _ {j}\right) - \psi^ {j} \left(\tilde {t} _ {j}, q _ {j}, \tilde {p} _ {j}\right) \right| \right], \tag {52} \\ \end{array}
$$
We now bound each term in the sum by  $(L_{\mathrm{t}}^{j} + L_{\mathrm{p}}^{j}) / (2\sqrt{n})$ , which will prove the theorem.
For each  $j \in [m]$ , using  $p$ -Lipschitzness of  $\psi_j$  we have:
$$
\begin{array}{l} \left| \psi^ {j} \left(t _ {j}, q _ {j}, p _ {j}\right) - \psi^ {j} \left(\tilde {t} _ {j}, q _ {j}, \tilde {p} _ {j}\right) \right| = \left| \psi^ {j} \left(\tilde {t} _ {j}, q _ {j}, \tilde {p} _ {j}\right) - \psi^ {j} \left(t _ {j}, q _ {j}, p _ {j}\right) \right| \\ \leq L _ {\mathrm {t}} ^ {j} \left| t _ {j} - \tilde {t} _ {j} \right| + L _ {\mathrm {p}} ^ {j} \left| p _ {j} - \tilde {p} _ {j} \right|. \tag {53} \\ \end{array}
$$
Taking expectation on both sides gives: