SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Erik Schultheis
erik.schultheis@aalto.fi
Wojciech Kotlowski
Poznan University of Technology
wkotlowski@cs.put.poznan.pl
Marek Wydmuch
mwydmuch@cs.put.poznan.pl
Rohit Babbar
University of Bath / Aalto University
Bath, UK / Helsinki, Finland
rb2608@bath.ac.uk
Krzysztof Dembczyński
Yahoo! Research / Poznan University of Technology
New York, USA / Poznan, Poland
krzysztof.dembczynski@yahooinc.com
Extreme multi-label classification (XMLC) is the task of selecting a small subset of relevant labels from a very large set of possible labels. As such, it is characterized by long-tail labels, i.e., most labels have very few positive instances. With standard performance measures such as precision@k, a classifier can ignore tail labels and still report good performance. However, it is often argued that correct predictions in the tail are more "interesting" or "rewarding," but the community has not yet settled on a metric capturing this intuitive concept. The existing propensity-scored metrics fall short on this goal by confounding the problems of long-tail and missing labels. In this paper, we analyze generalized metrics budgeted "at k" as an alternative solution. To tackle the challenging problem of optimizing these metrics, we formulate it in the expected test utility (ETU) framework, which aims to optimize the expected performance on a fixed test set. We derive optimal prediction rules and construct computationally efficient approximations with provable regret guarantees and robustness against model misspecification. Our algorithm, based on block coordinate ascent, scales effortlessly to XMLC problems and obtains promising results in terms of long-tail performance.
Extreme multi-label classification (XMLC) is a challenging task with a wide spectrum of real-life applications, such as tagging of text documents [10], content annotation for multimedia search [14], or different type of recommendation [5, 1, 33, 44, 53, 28, 7]. Because of the nature of its applications, the typical approach in XMLC is to predict exactly  $k$  labels (e.g., corresponding to  $k$  slots in the user interface) which optimize a standard performance metric such as precision or (normalized) discounted cumulative gain. Given the enormous number of labels in XMLC tasks, which can reach millions or more, it is not surprising that many of them are very sparse, and hence make the label distribution strongly long-tailed [2]. It has been noticed that algorithms can achieve high performance on the standard metrics, but never predict any tail labels [42]. Therefore, there is a need to develop
Table 1: Performance measures (%) on AmazonCat-13k of a classifier trained on the full set of labels and a classifier trained with only 1k head labels.
Metric	full labels	head labels
@1	@3	@5	@1 (diff.)	@3 (diff.)	@5 (diff.)
Precision	93.03	78.51	63.74	93.08 (+0.05%)	76.42 (-2.66%)	58.21 (-8.67%)
nDCG	93.03	87.25	85.35	93.08 (+0.05%)	85.75 (-1.71%)	80.91 (-5.19%)
PS-Precision	49.76	62.63	70.35	49.07 (-1.39%)	57.71 (-7.84%)	57.41 (-18.40%)
Macro-Precision	13.28	32.65	44.16	4.31 (-67.54%)	5.28 (-83.82%)	4.32 (-90.21%)
Macro-Recall	1.38	11.06	30.57	0.47 (-65.61%)	2.69 (-75.71%)	4.10 (-86.59%)
Macro-F1	2.26	14.67	32.84	0.74 (-67.37%)	3.10 (-78.88%)	3.77 (-88.51%)
Coverage	15.19	40.53	60.88	5.11 (-66.32%)	7.37 (-81.82%)	7.52 (-87.65%)
a metric that prefers "rewarding" [48], "diverse" [3], and "rare and informative" [34] labels over frequently-occurring head labels. Currently, the XMLC community attempts to capture this need using propensity-scored performance metrics [15]. These metrics give increased weight to tail labels, but have been derived from the perspective of missing labels, and as such they are not really solving the problem of tail labels [41].
In Table 1 we compare different metrics for budgeted at  $k$  predictions. We train a PLT model [17] on the full AmazonCAT-13K dataset [27] and a reduced version with the 1000 most popular labels only. The test is performed for both models on the full set of labels. The standard metrics are only slightly perturbed by reducing the label space to the head labels. This holds even for propensity-scored precision, which decreases by just  $1\% - 20\%$  despite discarding over  $90\%$  of the label space. In contrast, macro measures and coverage decrease between  $60\%$  and  $90\%$  if tail labels are ignored. These results show that budgeted-at- $k$  macro measures might be very attractive in the context of long tails. Macro-averaging treats all the labels equally important, preventing the labels with a small number of positive examples to be ignored. Furthermore, the budget of  $k$  labels "requires" the presence of long-tail labels in a compact set of predicted labels.
While we can easily use these measures to evaluate and compare different methods, we also would like to make predictions that directly optimize these metrics. The existing approaches to macro-averaged metrics consider the unconstrained case, in which label-wise optimization is possible [47, 11, 23, 16, 22, 26]. Each binary problem can be then solved under one of two frameworks for optimizing complex performance measures, namely population utility (PU) or expected test utility (ETU) [49, 12]. The former aims at optimizing the performance on the population level. The latter optimizes the performance directly on a given test set. Interestingly, in both frameworks the optimal solution is based on thresholding conditional label probabilities [12], but the resulting thresholds are different with the discrepancy diminishing with the size of the test set. The threshold tuning for PU is usually performed on a validation set [47, 26], while the exact optimization for ETU is performed on a test set. It requires cubic time in a general case and quadratic time in some special cases [49, 32]. Approximate solutions can be obtained in linear time [25, 12].
These approaches cannot be directly applied if prediction of exactly  $k$  labels for each instance is required. In such case, the optimization problems for different labels are tightly coupled through this constraint, making the final problem much more difficult. Despite the fact that optimization of complex performance metrics is a well-established problem, considered not only in binary and multi-label classification as discussed above, but also in multi-class classification [30, 31], the results presented in this paper go beyond the state-of-the-art as budgeted-at-  $k$  predictions have not yet been analyzed in this context. Let us underline that the requirement of  $k$  predictions is natural for recommendation systems, in which exactly  $k$  slots are available in the user interface to display recommendations. Even in situations where this does not apply, requiring the prediction to be "at  $k$ " can be advantageous, as it prevents trivial solutions such as predicting nothing (for precision) or everything (for recall).
In this paper, we investigate optimal solutions for the class of utility functions that can be linearly decomposed over labels into binary utilities, which includes both instance-wise weighted measures and macro-averages. We solve the problem in the ETU framework which is well-suited, for example, to recommendation tasks in which recommendations for all users or items are rebuilt in regular intervals. In this case, we can first obtain probability estimates of individual labels for each instance in the test set, and then provide optimal predictions for a given metric based on these estimates. We
derive optimal prediction rules and construct computationally efficient approximations with provable guarantees, formally quantifying the influence of the estimation error of the label probabilities on the suboptimality of the resulting classifier. This result is expressed in the form of a regret bound [4, 30, 22, 13]. It turns out that for most metrics of interest, a small estimation error results in at most a small drop of the performance, which confirms our method is viable for applications. Our general algorithm, based on block coordinate ascent, scales effortlessly to XMLC problems and obtains promising empirical results.
2 Setup and notation
Let  $\pmb{x} \in \mathcal{X}$  denote an input instance, and  $\pmb{y} \in \{0,1\}^m =: \mathcal{Y}$  the vector indicating the relevant labels, distributed according to  $\mathbb{P}(\pmb{y}|\pmb{x})$ . We consider the prediction problem in the expected test utility (ETU) framework, that is, we assume that we are given a known set of  $n$  instances  $\pmb{X} = [\pmb{x}_1,\dots,\pmb{x}_n]^{\top} \in \mathcal{X}^n$  with unknown labels, on which we have to make predictions. Our goal is to assign each instance  $\pmb{x}_i$  a set of exactly  $k$  (out of  $m$ ) labels represented as a  $k$ -hot vector  $\hat{\pmb{y}}_i \in \mathcal{Y}_k := \{\pmb{y} \in \mathcal{Y}: \| \pmb{y}\|_1 = k\}$ , and we let  $\hat{\pmb{Y}} = [\hat{\pmb{y}}_1,\dots,\hat{\pmb{y}}_n]^{\top}$  denote the entire  $n \times m$  prediction matrix for a set of instances  $\pmb{X}$ .
In the ETU framework, we treat  $\mathbf{X}$  as given and only make an assumption about the labeling process for the test sample: the labels  $\pmb{y}_i \in \mathcal{V}$  corresponding to  $\pmb{x}_i \in \mathcal{X}$  do not depend on any other instances, that is  $\mathbb{P}(\pmb{Y}|\pmb{X}) = \prod_{i=1}^{n} \mathbb{P}(\pmb{y}_i|\pmb{x}_i)$ , where we use  $\pmb{Y} = [\pmb{y}_1, \dots, \pmb{y}_n]^{\mathrm{T}} \in \mathcal{V}^n$  to denote the entire label matrix. We assume the quality of predictions  $\hat{\pmb{Y}}$  is jointly evaluated against the observed labels  $\pmb{Y}$  by a task utility  $\Psi(\pmb{Y}, \hat{\pmb{Y}})$ , and define the optimal (Bayes) prediction  $\hat{\pmb{Y}}^\star$  as the one maximizing the expected task utility  $\Psi_{\mathrm{ETU}}$ :
$$
\hat {\boldsymbol {Y}} ^ {\star} = \underset {\tilde {\boldsymbol {Y}} \in \mathcal {Y} _ {k} ^ {n}} {\operatorname {a r g m a x}} \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ \Psi (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) \right] =: \underset {\tilde {\boldsymbol {Y}} \in \mathcal {Y} _ {k} ^ {n}} {\operatorname {a r g m a x}} \Psi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}). \tag {1}
$$
We consider task utilities  $\varPsi(\boldsymbol{Y},\hat{\boldsymbol{Y}})$  that linearly decompose over labels, i.e., there exists  $\psi^j$  such that
$$
\varPsi (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) = \sum_ {j = 1} ^ {m} \psi^ {j} \left(\boldsymbol {y}: _ {j}, \hat {\boldsymbol {y}}: _ {j}\right). \tag {2}
$$
We allow the functions  $\psi^j$  to be non-linear themselves and different for each label  $j$ . This is a large class of functions, which encompasses weighted instance-wise and macro-averaged utilities, the two groups of functions which we thoroughly analyze in the next sections.
Let us next define the binary confusion matrix  $C(\pmb{y}, \hat{\pmb{y}})$  for a vector  $\pmb{y}$  of  $n$  ground truth labels and a corresponding vector  $\hat{\pmb{y}}$  of binary predictions:
$$
\boldsymbol {C} (\boldsymbol {y}, \hat {\boldsymbol {y}}) := \left( \begin{array}{c c} \frac {1}{n} \sum_ {i = 1} ^ {n} (1 - y _ {i}) (1 - \hat {y} _ {i}) & \frac {1}{n} \sum_ {i = 1} ^ {n} (1 - y _ {i}) \hat {y} _ {i} \\ \frac {1}{n} \sum_ {i = 1} ^ {n} y _ {i} (1 - \hat {y} _ {i}) & \frac {1}{n} \sum_ {i = 1} ^ {n} y _ {i} \hat {y} _ {i} \end{array} \right). \tag {3}
$$
By indexing from 0, the entry  $c_{00}$  corresponds to true negatives,  $c_{01}$  to false positives,  $c_{10}$  to false negatives, and  $c_{11}$  to true positives. We define the multi-label confusion tensor $^2$ $\mathbf{C}(\mathbf{Y},\hat{\mathbf{Y}})\coloneqq [C(\pmb{y}_{:1},\hat{\pmb{y}}_{:1}),\dots,C(\pmb{y}_{:m},\hat{\pmb{y}}_{:m})]$  being the concatenation of binary confusion matrices of all  $m$  labels.
Assuming the utility function (2) to be invariant under instance reordering, i.e., its value does not change if rows of both matrices are re-ordered using the same permutation, we can define  $\varPsi$  in terms of confusion matrices, instead of ground-truth labels and predictions (shown in Appendix A.1):
$$
\varPsi (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) = \varPsi (\mathbf {C} (\boldsymbol {Y}, \hat {\boldsymbol {Y}})) = \sum_ {j = 1} ^ {m} \psi^ {j} \left(\boldsymbol {C} \left(\boldsymbol {y}: _ {j}, \hat {\boldsymbol {y}}: _ {j}\right)\right). \tag {4}
$$
Finally, we assume that we have access to a label probability estimator (LPE)  $\hat{\eta}(\pmb{x})$  that estimates the marginal probability of each label given the instance,  $\pmb{\eta}(\pmb{x}) = (\eta_1(\pmb{x}), \dots, \eta_m(\pmb{x})) \coloneqq \mathbb{E}_{\pmb{y}|\pmb{x}}[\pmb{y}]$ . Such an LPE can be attained by fitting a model on an additional training set of  $n'$  examples  $(\pmb{x}_i, \pmb{y}_i)_{i=1}^{n'}$  using a proper composite loss function [37], which is a common approach in XMLC, e.g., [18].
<sup>1</sup>We use calligraphic letters for sets  $S$ , bold font for vectors  $\mathbf{v}$  with entries  $v_{i}$ , bold capital letters for matrices  $\mathbf{Y}$  with entries  $y_{ij}$ , rows  $\mathbf{y}_{i}$ , and columns  $\mathbf{y}_{:j}$ .  $\mathbb{1}[S]$  denotes the indicator of event  $S$ , and  $[s] := \{1, \dots, s\}$
2Notice that the confusion matrix can be computed either for multi-label predictions for a given instance  $x$  or binary predictions for label  $j$  obtained on a set of instances  $\mathbf{X}$ . In the following, we focus on the latter.
3 Performance measures for tail labels
3.1 Instance-wise weighted utility functions
By assigning utility (or cost) to each correct/wrong prediction for each label, we can construct an instance-wise weighted utility  $u_{\mathrm{w}}\colon \mathcal{Y}\times \mathcal{Y}\longrightarrow \mathbb{R}_{\geq 0}$  with labels  $\pmb {y}\in \mathcal{V}$  and predictions  $\hat{\pmb{y}}\in \mathcal{V}$  as
$$
u _ {\mathrm {w}} (\boldsymbol {y}, \hat {\boldsymbol {y}}) = \sum_ {j = 1} ^ {m} w _ {0 0} ^ {j} (1 - y _ {j}) (1 - \hat {y} _ {j}) + w _ {0 1} ^ {j} (1 - y _ {j}) \hat {y} _ {j} + w _ {1 0} ^ {j} y _ {j} (1 - \hat {y} _ {j}) + w _ {1 1} ^ {j} y _ {j} \hat {y} _ {j}. \tag {5}
$$
We use  $w_{00}^{j}, w_{01}^{j}, w_{10}^{j}$ , and  $w_{11}^{j}$  to express the utility of true negatives, false positives, false negatives, and true positives, respectively. The corresponding task loss results from summing over all instances. By interchanging the order of summation, we can see that it is of form (4):
$$
\varPsi (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) = \sum_ {i = 1} ^ {n} u _ {\mathrm {w}} (\boldsymbol {y} _ {i}, \hat {\boldsymbol {y}} _ {i}) = \sum_ {j = 1} ^ {m} w _ {0 0} ^ {j} c _ {0 0} ^ {j} + w _ {0 1} ^ {j} c _ {0 1} ^ {j} + w _ {1 0} ^ {j} c _ {1 0} ^ {j} + w _ {1 1} ^ {j} c _ {1 1} ^ {j} = \sum_ {j = 1} ^ {m} \psi^ {j} \left(\boldsymbol {C} \left(\boldsymbol {y}: _ {j}, \hat {\boldsymbol {y}}: _ {j}\right)\right). \tag {6}
$$
In other words, the instance-wise weighted utilities can be seen as linear confusion-based metrics. By choosing  $w_{00}^{j} = w_{11}^{j} = m^{-1}$  and  $w_{01}^{j} = w_{10}^{j} = 0$ , the expression (5) reduces to the  $@k$ -variant of the Hamming utility. Similarly,  $w_{11}^{j} = k^{-1}$  and  $w_{00}^{j} = w_{01}^{j} = w_{10}^{j} = 0$  yield precision@ $k$ .
Another example is the popular propensity-scoring approach [15], commonly used as a tail-performance metric in XMLC. Here, the weights are computed based on training data through
$$
w _ {1 1} ^ {j, \mathrm {p r o p}} = k ^ {- 1} \left(1 + (\log n ^ {\prime} - 1) (b + 1) ^ {a} \left(n ^ {\prime} \hat {\pi} _ {j} + b\right) ^ {- a}\right), \tag {7}
$$
where  $n'$  is the number of training instances,  $\hat{\pi}_j$  is the empirical prior of label  $j$ , and parameters  $a$  and  $b$  are (potentially) dataset-dependent. This form of weighting has been derived from a missing-labels perspective, so its application to tail labels is not fully justified [41]. Also, it introduces two more hyperparameters, which makes the interpretation and comparison of its values rather difficult. It is not less heuristical than other approaches like power-law or logarithmic weighting, given by:
$$
w _ {1 1} ^ {j, \mathrm {p l}} \propto \hat {\pi} _ {j} ^ {- \beta}, \quad w _ {1 1} ^ {j, \log} \propto - \log (\hat {\pi} _ {j}). \tag {8}
$$
3.2 Macro-average of non-decomposable utilities
Macro-averaging usually concerns non-decomposable binary utilities such as the F-measure. In this case, we set up  $\psi^j (\cdot ,\cdot) = m^{-1}\psi (\cdot ,\cdot)$  for all labels  $j$  in (2), yielding:
$$
\Psi (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) = m ^ {- 1} \sum_ {j = 1} ^ {m} \psi \left(\boldsymbol {y}: j, \hat {\boldsymbol {y}}: j\right) = m ^ {- 1} \sum_ {j = 1} ^ {m} \psi \left(\boldsymbol {C} \left(\boldsymbol {y}: j, \hat {\boldsymbol {y}}: j\right)\right). \tag {9}
$$
By using  $\psi_{\mathrm{pr}}(\pmb {C})\coloneqq c_{11} / (c_{11} + c_{01})$  , this becomes macro-precision, for  $\psi_{\mathrm{rec}}(\pmb {C})\coloneqq c_{11} / (c_{11} + c_{10})$  we get macro-recall, and for  $\psi_{\mathrm{F}_{\beta}}(\pmb {C})\coloneqq (\beta +1)c_{11} / ((1 + \beta)c_{11} + \beta^{2}c_{10} + c_{01})$  the macro-F-measure.
Another measure that is promising for the evaluation of long-tailed performance is coverage. It is sometimes used as an auxiliary measure in XMLC [15, 3, 42, 41]. This metric detects for how many different labels the classifier is able to make at least one correct prediction. In our framework, this is achieved by using an indicator function on the true positives,  $\psi_{\mathrm{cov}}(C) \coloneqq \mathbb{1}[c_{11} > 0]$ .
4 Optimal predictions
Let us start with the observation that the label probabilities  $\pmb{\eta}$  are sufficient to make optimal predictions. With the assumption that  $\mathbb{P}(\pmb {Y}|\pmb {X}) = \prod_{i = 1}^{n}\mathbb{P}(\pmb {y}_i|\pmb {x}_i)$ , we obtain (cf. Appendix A.2):
$$
\mathbb {E} _ {\boldsymbol {Y} \mid \boldsymbol {X}} [ \Psi (\boldsymbol {Y}, \hat {\boldsymbol {Y}}) ] = \sum_ {j = 1} ^ {m} \sum_ {\boldsymbol {y} ^ {\prime} \in \{0, 1 \} ^ {n}} \left(\prod_ {i = 1} ^ {n} \eta_ {j} (\boldsymbol {x} _ {i}) y _ {i} ^ {\prime} + (1 - \eta_ {j} (\boldsymbol {x} _ {i})) (1 - y _ {i} ^ {\prime})\right) \psi^ {j} \left(\boldsymbol {y} ^ {\prime}, \hat {\boldsymbol {y}} _ {: j}\right). \tag {10}
$$
This equation lays out a daunting optimization task, as is requires summing over  $2^{n}$  summands  $\pmb{y}^{\prime}$ . In case of binary classification, there exist methods to solve the problem exactly in  $\mathcal{O}(n^3)$ , or in  $\mathcal{O}(n^2)$
in some special cases [32]. By using semi-empirical quantities (defined below), [12] provides an approximate algorithm that runs in  $O(n)$ . Following this approach, we construct a semi-empirical ETU approximation. If this approximation results in a linear function of the predictions, the problem decomposes over instances and can be solved easily. Otherwise, we use an algorithm that leads to locally optimal predictions. A minor modification of this algorithm can be used for coverage.
4.1 Semi-empirical ETU approximation
Since the entries of the confusion matrix are linearly dependent, it suffices to use three independent combinations. More precisely, we parameterize the confusion matrix by the true positives  $t = c_{11}$ , predicted positives  $q = c_{11} + c_{01}$ , and ground-truth positives  $p = c_{11} + c_{10}$ , and use  $\pmb{t} = (t_1,\dots ,t_m)$ ,  $\pmb{q} = (q_{1},\ldots ,q_{m})$ , and  $\pmb{p} = (p_{1},\dots ,p_{m})$  to reformulate the ETU objective:
$$
\varPsi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}) = \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} [ \varPsi (\mathbf {C} (\boldsymbol {Y}, \hat {\boldsymbol {Y}})) ] = \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} [ \varPsi (\boldsymbol {t}, \boldsymbol {q}, \boldsymbol {p}) ] = \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ \sum_ {j = 1} ^ {m} \psi^ {j} \left(t _ {j}, q _ {j}, p _ {j}\right) \right]. \tag {11}
$$
In order to compute  $\varPsi_{\mathrm{ETU}}$ , one needs to take into account every possible combination of confusion-matrix values, and calculate the corresponding value of  $\varPsi$ , which is then averaged according to the respective probabilities. A computationally easier approach is to take the expectation over the labels first, leading to semi-empirical quantities:
$$
\tilde {\boldsymbol {t}} := \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} [ \boldsymbol {t} ], \quad \tilde {\boldsymbol {q}} := \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} [ \boldsymbol {q} ] = \boldsymbol {q}, \quad \tilde {\boldsymbol {p}} := \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} [ \boldsymbol {p} ], \tag {12}
$$
where  $\tilde{\pmb{q}} = \pmb{q}$  follows because the number of predicted positives depends only on the predictions  $\hat{\pmb{Y}}$ . This allows us to define the semi-empirical ETU risk
$$
\tilde {\Psi} _ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}) := \Psi (\tilde {\boldsymbol {t}}, \boldsymbol {q}, \tilde {\boldsymbol {p}}) \approx \mathbb {E} _ {\boldsymbol {Y} \mid \boldsymbol {X}} [ \Psi (\boldsymbol {t}, \boldsymbol {q}, \boldsymbol {p}) ] = \Psi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}). \tag {13}
$$
In particular, the third argument to  $\varPsi$ ,  $\tilde{\pmb{p}}$ , is a constant that does not depend on predictions.
Note that, if  $\varPsi$  is linear in all arguments depending on the random variable  $\mathbf{Y}$ , then the approximation is exact, due to the linearity of expectations. Aside from instance-wise measures, which we showed to be linear above, the approximation is also exact for the more general class of functions of the form
$$
\psi (t, q, p) = f _ {\mathrm {t}} (q) \cdot t + f _ {\mathrm {q}} (q) + f _ {\mathrm {p}} (q) \cdot p. \tag {14}
$$
An important example is macro-precision, with  $f_{\mathrm{t}}(q) = q^{-1}$  and  $f_{\mathrm{q}} = f_{\mathrm{p}} = 0$ . In the general case,  $\tilde{\Psi}_{\mathrm{ETU}}$  as a surrogate for  $\varPsi_{\mathrm{ETU}}$  leads only to  $\mathcal{O}(1 / \sqrt{n})$  error as will be shown in Theorem 5.2, while substantially simplifying the optimization process.
4.2 Linear confusion-matrix measures
We start the discussion on optimization of (13) with a special case in which  $\tilde{\Psi}_{\mathrm{ETU}}$  is linear in the prediction-dependent arguments  $t, q$ , that is, if
$$
\psi (t, q, p) = f _ {\mathrm {t}} (p) \cdot t + f _ {\mathrm {q}} (p) \cdot q, \tag {15}
$$
i.e., both  $f_{\mathrm{t}}(p)$  and  $f_{\mathrm{q}}(p)$  depend on  $p$  only, and  $f_{\mathrm{p}}(p) \cdot p$  can be dropped as it is a constant.
Aside from instance-wise weighted utilities (cf. Appendix A.3), which are linear in all arguments, this form also holds for weights dependent on the (empirical) label priors, e.g., power law weights of the form  $f_{\mathrm{t}}(p) = p^{-\alpha}$  and  $f_{\mathrm{q}} = 0$ , which reduce to macro-recall for  $\alpha = 1$ . If one defines the weights with respect to externally determined label priors, i.e., approximations to  $\mathbb{E}[y_j]$ , which are fixed and thus independent of the test sample  $X$ , then the power-law metrics turn into instance-wise weighted utilities.
From (6) we know that we can reformulate the optimization problem using an instance-wise weighted utility  $u_{\mathrm{w}}$  with weights:
$$
w _ {1 1} ^ {j} = f _ {\mathrm {t}} \left(p _ {j}\right) + f _ {\mathrm {q}} \left(p _ {j}\right), \quad w _ {0 1} ^ {j} = f _ {\mathrm {q}} \left(p _ {j}\right), \quad w _ {1 0} ^ {j} = 0, \quad w _ {0 0} ^ {j} = 0. \tag {16}
$$
Hence, the optimal predictions can be derived for each instance  $\pmb{x} \in \mathcal{X}$  separately, leading to
$$
\hat {\boldsymbol {y}} ^ {*} = \operatorname {a r g m a x} _ {\hat {\boldsymbol {y}} \in \mathcal {Y} _ {k}} \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {x}} \left[ u _ {\mathrm {w}} (\boldsymbol {y}, \hat {\boldsymbol {y}}) \right]. \tag {17}
$$
Plugging in the definition of  $u_{\mathrm{w}}$  from (5), and collecting terms, the expected loss is of the form