SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} _ {\boldsymbol {y} \mid \boldsymbol {x}} \left[ u _ {\mathrm {w}} (\boldsymbol {y}, \hat {\boldsymbol {y}}) \right] = \sum_ {j = 1} ^ {m} \mathbb {E} _ {\boldsymbol {y} \mid \boldsymbol {x}} \left[ w _ {1 1} ^ {j} y _ {j} \hat {y} _ {j} + w _ {0 1} ^ {j} (1 - y _ {j}) \hat {y} _ {j} \right] = \sum_ {j = 1} ^ {m} \hat {y} _ {j} \left(\eta_ {j} (\boldsymbol {x}) f _ {\mathrm {t}} (p _ {j}) + f _ {\mathrm {q}} (p _ {j})\right), \tag {18}
$$
where we call  $g_{j}(\pmb{x}) \coloneqq \eta_{j}(\pmb{x})f_{\mathrm{t}}(p_{j}) + f_{\mathrm{q}}(p_{j})$  the gain of predicting label  $j$  for a given instance  $\pmb{x}$ . The optimal prediction is to select the  $k$  labels with the largest values  $g_{j}(\pmb{x})$ ,
$$
\hat {\boldsymbol {y}} ^ {*} = \underset {\hat {\boldsymbol {y}} \in \hat {\mathcal {Y}} _ {k}} {\operatorname {a r g m a x}} \sum_ {j = 1} ^ {m} g _ {j} (\boldsymbol {x}) \hat {y} _ {j} = \text {s e l e c t - t o p - k} (\boldsymbol {g} (\boldsymbol {x})). \tag {19}
$$
Here, select-top-  $k(\pmb{g})$  denotes an operation that maps a vector of scores to a binary vector that contains 1s only at the positions of the  $k$  largest elements of  $\pmb{g}$ .
4.3 General non-decomposable macro measures
Finally, we turn to the general problem for budgeted-at-  $k$  macro-measures based on nonlinear  $\varPsi$ . As this can be a very hard discrete optimization problem in general, we use an iterative approach based on block-coordinate ascent that constructs a sequence of predictions,  $\hat{\mathbf{Y}}^0,\hat{\mathbf{Y}}^1,\dots$ , with non-decreasing utility, so that we end up with a solution that is locally optimal.
Assume the predictions are fixed for all instances except  $\pmb{x}_s$ , where they are given by  $z$ . In that case, we can write the semi-empirical quantities from (12) as
$$
\tilde {\boldsymbol {t}} = \frac {1}{n} \left(\eta_ {j} \left(\boldsymbol {x} _ {s}\right) z _ {j} + \sum_ {i \in [ n ] \backslash \{s \}} \eta_ {j} \left(\boldsymbol {x} _ {s}\right) \hat {y} _ {i j}\right), \tag {20}
$$
with analog expansions for  $q$  and  $\tilde{p}$ . Plugging into (13) leads to the following optimization:
Algorithm 1 BCA  $(X,\hat{\eta},k,\epsilon)$    
1:  $\hat{y}_i\gets$  select-random-  $k(m)$  for all  $i\in [n]$    
2:  $\tilde{\pmb{t}}\gets \frac{1}{n}\sum_{i = 1}^{n}\hat{\eta} (\pmb {x}_i)\odot \hat{\pmb{y}}_i$    
3:  $\pmb {q}\gets \frac{1}{n}\sum_{i = 1}^{n}\hat{\pmb{y}}_i$ $\tilde{\pmb{p}}\gets \frac{1}{n}\sum_{i = 1}^{n}\hat{\pmb{\eta}} (\pmb {x}_i)$    
4:  $u_{old}\gets -\infty$ $u_{\mathrm{new}}\gets \Psi (\tilde{\pmb{t}},\pmb {q},\tilde{\pmb{p}})$    
5: while  $u_{\mathrm{new}} > u_{\mathrm{old}} + \epsilon$  do   
6: for  $s\in$  shuffle([n]) do   
7:  $\tilde{\pmb{t}}\gets \tilde{\pmb{t}} -\frac{1}{n}\hat{\eta} (\pmb{x}_s)\odot \hat{\pmb{y}}_s$ $q\gets q - \frac{1}{n}\hat{y}_s$    
8: for  $j\in [m]$  do   
9:  $\psi^j (1)\gets \psi (\tilde{t}_j + \frac{1}{n}\hat{\eta}_j(\pmb {x}_s),q_j + \frac{1}{n},\tilde{p}_j)$    
10:  $\psi^j (0)\gets \psi (\tilde{t}_j,q_j,\tilde{p}_j)$    
11:  $g_j\gets \psi^j (1) - \psi^j (0)$    
12:  $\hat{\pmb{y}}_s\gets$  select-top-  $k(g)$    
13:  $\tilde{\pmb{t}}\gets \tilde{\pmb{t}} +\frac{1}{n}\hat{\eta} (\pmb {x}_s)\odot \hat{\pmb{y}}_s$ $q\gets q + \frac{1}{n}\hat{y}_s$    
14:  $u_{old}\gets u_{\mathrm{new}},u_{\mathrm{new}}\gets \Psi (\tilde{\pmb{t}},\pmb {q},\tilde{\pmb{p}})$    
15: return Y
$$
\max  _ {\boldsymbol {z} \in \mathcal {Y} _ {k}} \sum_ {j = 1} ^ {m} \psi^ {j} \left(\frac {1}{n} \eta_ {j} (\boldsymbol {x} _ {s}) z _ {j} + \frac {1}{n} \sum_ {i \in [ n ] \backslash \{s \}} \eta_ {j} (\boldsymbol {x} _ {i}) \hat {y} _ {i j}, \frac {1}{n} z _ {j} + \frac {1}{n} \sum_ {i \in [ n ] \backslash \{s \}} \hat {y} _ {i j}, \frac {1}{n} \sum_ {i = 1} ^ {n} \eta_ {j} (\boldsymbol {x} _ {i})\right). \tag {21}
$$
As everything except  $z_{j} \in \{0,1\}$  is given, we can interpret  $\psi^{j}$  as a linear function of  $z_{j}$ , and define a gain vector with elements  $g_{j} = \psi^{j}(1) - \psi^{j}(0)$ . The optimal prediction  $z^{*}$  is then given by  $z^{*} = \text{select-top-} k(\pmb{g})$ , in a similar form as in case of linear confusion-matrix measures. We get  $\hat{\mathbf{Y}}^{t+1}$  by replacing the  $s^{\text{th}}$  row of  $\hat{\mathbf{Y}}^{t}$  with  $z^{*}$ , and know that  $\tilde{\Psi}_{\text{ETU}}(\hat{\mathbf{Y}}^{t+1}) \geq \tilde{\Psi}_{\text{ETU}}(\hat{\mathbf{Y}}^{t})$ . Then we switch to the next instance  $s \gets s + 1$ , and repeat this process until no more progress is made.
The algorithm starts by predicting  $k$  random labels for each instance. To speed up the computations we cache two quantities,  $\tilde{t}_j^t$  and  $q_j^t$ , for each label:
$$
\tilde {t} _ {j} ^ {0} := \frac {1}{n} \sum_ {i = 1} ^ {n} \eta_ {j} \left(\boldsymbol {x} _ {i}\right) \hat {y} _ {i j} ^ {0}, \quad \tilde {t} _ {j} ^ {t + 1} := \tilde {t} _ {j} ^ {t} + \frac {1}{n} \eta_ {j} \left(\boldsymbol {x} _ {s}\right) \left(\hat {y} _ {s j} ^ {t + 1} - \hat {y} _ {s j} ^ {t}\right), \tag {22}
$$
$$
q _ {j} ^ {0} := \frac {1}{n} \sum_ {i = 1} ^ {n} \hat {y} _ {i j} ^ {0}, \quad q _ {j} ^ {t + 1} := q _ {j} ^ {t} + \frac {1}{n} \left(\hat {y} _ {s j} ^ {t + 1} - \hat {y} _ {s j} ^ {t}\right).
$$
With this, we can compute (21) in  $\mathcal{O}(m)$  time using the following formulas:
$$
\psi^ {j} (1) := \psi^ {j} \left(\tilde {t} _ {j} ^ {t} + \frac {1}{n} \eta_ {j} \left(\boldsymbol {x} _ {s}\right) \left(1 - \hat {y} _ {s j} ^ {t}\right), q _ {j} ^ {t} + \frac {1}{n} \left(1 - \hat {y} _ {s j} ^ {t}\right), \tilde {p} _ {j}\right), \tag {23}
$$
$$
\psi^ {j} (0) := \psi^ {j} \left(\tilde {t} _ {j} ^ {t} - \frac {1}{n} \eta_ {j} (\boldsymbol {x} _ {s}) \hat {y} _ {s j} ^ {t}, q _ {j} ^ {t} - \frac {1}{n} \tilde {y} _ {s j} ^ {t}, \tilde {p} _ {j}\right).
$$
The block coordinate ascent (BCA) procedure is shown in more detail in Algorithm 1. Obviously, the actual implementation cannot use the unknown values  $\pmb{\eta}$ , but instead has to rely on the LPE estimates  $\hat{\pmb{\eta}}(\pmb{x}_i)$ . The stopping criterion for the algorithm is whether, after going over a whole set of instances, the improvement in the objective value over the previous value is lower than  $\epsilon$ , which ensures that the algorithm terminates for every bounded utility in  $\mathcal{O}(1/\epsilon)$ . In practice, even with small  $\epsilon$ , the algorithm usually terminates after a few iterations. The time and space complexity of the single iteration are both  $\mathcal{O}(nm)$ . If  $\psi$  is a linear function, corresponding to an instance-wise weighted utility (6) such as macrorecall, the algorithm recovers the optimal solution in the first iteration, stopping after the second.
In general, Algorithm 1 requires multiple iterations. This can be computationally expensive and requires all of the data to be available at once. We thus propose a greedy algorithm that takes into account only statistics of previously seen instances while performing a single pass over the dataset, which allows for semi-online optimization. The greedy algorithm is outlined in Appendix B.4.
4.4 Optimization of coverage
One measure for which the ETU approximation is not exact is coverage as  $\psi_{\mathrm{cov}}(t,q,p)\coloneqq \mathbb{1}[t > 0]$  is nonlinear. In this case, we can do better than Algorithm 1, by reformulating  $\varPsi_{\mathrm{ETU}}$  for coverage as
$$
\varPsi_ {\mathrm {E T U}} (\hat {\boldsymbol {Y}}) = \mathbb {E} _ {\boldsymbol {Y} | \boldsymbol {X}} \left[ m ^ {- 1} \sum_ {j = 1} ^ {m} \mathbb {1} [ t _ {j} > 0 ] \right] = 1 - m ^ {- 1} \sum_ {j = 1} ^ {m} \prod_ {i = 1} ^ {n} \left(1 - \eta_ {j} \left(\boldsymbol {x} _ {i}\right) \hat {y} _ {i j}\right), \tag {24}
$$
and performing block coordinate ascent directly on this expression, as detailed in Appendix B.3.
5 Regret bounds
Computing optimal predictions relies on an access to the conditional marginal probabilities  $\pmb{\eta}(\pmb{x})$ . In practice, however,  $\pmb{\eta}(\pmb{x})$  are unknown, and are replaced by the LPE  $\hat{\eta}(\pmb{x})$  to do inference (plug-in approach). Furthermore, we replaced the ETU objective  $\varPsi_{\mathrm{ETU}}$  with an approximation  $\tilde{\varPsi}_{\mathrm{ETU}}$ . As generally  $\hat{\eta}(\pmb{x}) \neq \pmb{\eta}(\pmb{x})$  and  $\varPsi_{\mathrm{ETU}} \neq \tilde{\varPsi}_{\mathrm{ETU}}$ , this procedure may result in sub-optimal predictions, the errors of which we would like to control.
If we are able to control the change of the utility under small changes in its arguments, we can show that the suboptimality of the plug-in predictor relative to the optimal predictor (called  $\varPsi$ -regret) decreases with increasing test size  $n$  and decreasing probability estimation error.
To this end we assume that each  $\psi^j$  is a  $p$ -Lipschitz function, which allows us to bound the approximation error.
Definition 5.1 ( $p$ -Lipschitz [13]). A binary classification metric  $\psi(t, q, p)$  is said to be  $p$ -Lipschitz if
$$
\left| \psi (t, q, p) - \psi \left(t ^ {\prime}, q ^ {\prime}, p ^ {\prime}\right) \right| \leq L _ {t} (p) \left| t - t ^ {\prime} \right| + L _ {q} (p) \left| q - q ^ {\prime} \right| + L _ {p} (p) \left| p - p ^ {\prime} \right|, \tag {25}
$$
for any  $q, q' \in [0,1]$ ,  $p, p' \in (0,1)$ ,  $0 \leq t \leq \min(p, q)$ , and  $0 \leq t' \leq \min(p', q')$ . The constants  $L_{\mathfrak{t}}(p), L_{\mathfrak{q}}(p), L_{\mathfrak{p}}(p)$  are allowed to depend on  $p$ , in contrast to the standard Lipschitz functions.
As shown in Appendix C.3, most of metrics of interest satisfy  $p$ -Lipschitz assumption, including the linear confusion-matrix measures (6) with fixed weights (e.g., Hamming utility, precision), macro-recall, macro-F-measure, etc., with macro-precision and coverage being notable exceptions.
Theorem 5.2. Let each  $\psi^j$  be  $p$ -Lipschitz with constants  $L_{\mathfrak{t}}^{j}(p), L_{\mathfrak{q}}^{j}(p), L_{\mathfrak{p}}^{j}(p)$ . For any  $\hat{\mathbf{Y}}$  it holds:
$$
\left| \Psi_ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \mathbf {X}) - \tilde {\Psi} _ {\mathrm {E T U}} (\hat {\mathbf {Y}}; \mathbf {X}) \right| \leq \frac {1}{2 \sqrt {n}} \left(\sum_ {j = 1} ^ {m} \left(L _ {\mathrm {t}} ^ {j} \left(\tilde {p} _ {j}\right) + L _ {\mathrm {p}} ^ {j} \left(\tilde {p} _ {j}\right)\right)\right). \tag {26}
$$
Thus, using  $\tilde{\varPsi}_{\mathrm{ETU}}$  as a surrogate for  $\varPsi_{\mathrm{ETU}}$  leads only to  $\mathcal{O}(1 / \sqrt{n})$  error, diminishing with the test size, while substantially simplifying the optimization process.
Given a decomposable metric of the form (4), let  $\bar{Y}^{\dagger}$  be the plug-in prediction matrix optimizing the semi-empirical ETU with plugged-in probability estimates,  $\varPsi\left(\mathbb{E}_{\boldsymbol{y}\sim \hat{\eta} (\boldsymbol{X})}[\boldsymbol {t}],\boldsymbol {q},\mathbb{E}_{\boldsymbol{y}\sim \hat{\eta} (\boldsymbol{X})}[\boldsymbol {p}]\right)$ .
Theorem 5.3. Let  $\hat{Y}^{\dagger}$  be defined as above. Under the assumptions of Theorem 5.2:
$$
\left. \Psi_ {\mathrm {E T U}} \left(\hat {Y} ^ {\star}; \boldsymbol {X}\right) - \Psi_ {\mathrm {E T U}} \left(\tilde {Y} ^ {\dagger}; \boldsymbol {X}\right) \leq \frac {m}{\sqrt {n}} B + 2 \frac {\sqrt {m}}{n} B \sum_ {i = 1} ^ {n} \| \boldsymbol {\eta} \left(\boldsymbol {x} _ {i}\right) - \hat {\boldsymbol {\eta}} \left(\boldsymbol {x} _ {i}\right) \| _ {2}, \right. \tag {27}
$$
where  $B \coloneqq \sqrt{m^{-1} \sum_{j=1}^{m} (L_{\mathfrak{t}}^{j}(\tilde{p}_{j}) + L_{\mathfrak{p}}^{j}(p_{j}))^{2}}$  is the quadratic mean of the Lipschitz constants.
A similar statement, presented in Appendix C.4, can be made for the unapproximated ETU case.
Thus, the methods described in Section 4 can be used with probability estimates replacing the true marginals, and as long as the estimator is reliable, the resulting predictions will have small  $\Psi$ -regret. This also justifies the plug-in approach used in the experiments in Section 7.
6 Efficient inference
The optimization algorithms introduced so far need to obtain  $\hat{\eta}_j(\pmb{x}_i)$  for all labels and instances first. Then, the BCA inference is of  $\mathcal{O}(nm)$  time and  $\mathcal{O}(nm)$  space complexity for a single iteration. This
is problematic in the setting of XMLC, where many methods aim to predict probabilities only for top labels in time sublinear in  $m$ . Fortunately, this characteristic of XMLC algorithms can be combined with the introduced algorithms to efficiently obtain an approximate solution. Instead of predicting all  $\eta$ , we can predict probabilities only for top- $k'$  labels with highest  $\hat{\eta}_j$ , where  $k \ll k' \ll m$ . For all other labels we then assume  $\hat{\eta}_j = 0$ . Under the natural assumption that  $\psi$  is non-decreasing in true positives and non-increasing in predicted positives, we can leverage the sparsity and consider labels with non-zero  $\hat{\eta}_j$  only (using sparse vectors to represent  $\hat{\eta}(\boldsymbol{x}_i)$ ) to reduce the time and space complexity to  $O(n(k' + k\log k))$  and  $O(n(k' + k))$ , respectively. As in real-world datasets the number of relevant labels  $\| \boldsymbol{y} \|_1$  is much lower than  $m$ , and most  $\eta_j(\boldsymbol{x}_i)$  are close to 0, with reasonably selected  $k'$ , according to Theorem 5.3, we should only slightly increase the regret. A pseudocode for the sparse variant of Algorithm 1 can be found in Appendix D.
Alternatively, we can leverage probabilistic label trees (PLTs) [16], a popular approach in XMLC [35, 45, 19, 50, 7], to efficiently search for "interesting" labels. Originally, PLTs find top labels with the highest  $\hat{\eta}_j$ . In [46], an  $A^*$ -search algorithm has been introduced for finding  $k$  labels with highest value of  $g_j \eta_j(x)$ , where  $g_j \in [0, \infty)$  is a gain assigned to label  $j$ . In Appendix F, we present a more general version of this procedure that can be used to efficiently obtain an exact solution of presented BCA or Greedy algorithms for some of the metrics considered in this work.
To empirically test the introduced framework, we use popular benchmarks from the XMLC repository [6]. We train the LIGHTXML [18] model (with suggested default hyper-parameters) on provided training sets to obtain  $\hat{\eta}$  for all test instances. We then plug these estimates into different inference strategies and report the results across the discussed measures. To run the optimization algorithm efficiently, we use  $k' = 100$  or  $k' = 1000$  to pre-select for each instance the top  $k'$  labels with the highest  $\hat{\eta}_j$  as described in Section 6. $^3$
We use the following inference strategies:
- TOP-K- the optimal strategy for precision@k: selection of  $k$  labels with the highest  $\eta_j$  (default prediction strategy in many XMLC methods).
- PS-K- the optimal strategy for propensity-scored precision@k: selection of  $k$  labels with the highest  $w_{11}^{j,\mathrm{prop}}\eta_j$ , with  $w_{11}^{j,\mathrm{prop}}$  given by the empirical model of Jain et al. [15] (Equation 7) with values  $a$  and  $b$  recommended by the authors.
- Pow-K, LOG-K—the optimal strategy for power-law and log weighted instance-wise utilities: selection of  $k$  labels with the highest  $w_{11}^{j,\mathrm{pl}}\eta_j$  or  $w_{11}^{j,\log}\eta_j$ . For power-law, we use  $\beta = 0.5$ .
- MACRO- $\mathbf{P}_{\mathrm{BCA}}$ , MACRO- $\mathbf{R}_{\mathrm{BCA}}$ , MACRO- $\mathbf{F1}_{\mathrm{BCA}}$ , COV $_{\mathrm{BCA}}$ - the block coordinate ascent (Algorithm 1) for optimizing macro-precision, -recall, -F1, and coverage,
We expect a strategy suited for a given metric to obtain the best results on this metric. Nevertheless, this might not always be the case, as in the derivation of our algorithms, we needed to apply different types of approximation to scale them to XMLC problems. We are mainly interested in the performance on the general nondecomposable macro measures since they seem to be well-tailored to long tails, and their optimization is the most challenging. The results are presented in Table 2. Notice that in almost all cases, the specialized inference strategies are indeed the best on the measure they aim to optimize and achieve substantial gains on corre
sponding metrics compared to the basic TOP-K inference. The other weighted strategies, PS-K, Pow-K, Log-K, usually provide a much smaller improvement over TOP-K and never beat strategies designed for specific macro measures. As the reported performance depends on three things: the inherent difficulty of the data, the success of the inference algorithm, and the quality of the provided
Figure 1: Results of an inference strategy with a mixed utility on AMAZONCAT-13K and  $k = 3$ . The green line shows the results for different interpolations between two measures.
$^{3}$ A code to reproduce all the experiments: https://github.com/mwydmuch/xCOLUMNS
marginal probabilities, the results might diverge from expectations in some cases. In Appendix E, we provide more details and conduct further experiments to investigate the impact of randomness, stopping conditions, quality of probability estimates, and shortlisting on the results. We also present similar experiments with probabilistic label trees used as an LPE.
Unfortunately, our results show that optimization of macro-measures comes with the cost of a significant drop in performance on instance-wise measures. Ideally, we would like to improve the performance on tail labels without sacrificing too much of general performance. To achieve such a trade-off, we can use straight-forward interpolation of instance-wise precision-at-k, as it is covered by our framework, and a selected macro-measure, since the considered class of utility functions is closed under linear combinations. Such an objective can be optimized by the proposed block-coordinate algorithm without any modification. As an example, we plot in Table 7 the results of optimizing a linear combination of instance-wise precision and the macro-F1 measure:  $\Psi (\mathbf{Y},\hat{\mathbf{Y}}) = (1 - \alpha)\Psi_{\mathrm{Instance - P}}(\mathbf{Y},\hat{\mathbf{Y}}) + \alpha \Psi_{\mathrm{Macro - F1}}(\mathbf{Y},\hat{\mathbf{Y}})$ , using different values of  $\alpha \in [0,1]$ . In Appendix E.5, we provide more plots for different utilities, datasets, and values of  $k$ . All the plots show that the instance-vs-macro curve has a nice concave shape that dominates simple baselines. In particular, the initial significant improvement on macro-measures comes with a minor drop in instance-measures, and only if one wants to optimize more strongly for macro-measures, the drop on instance-wise measures becomes more severe.
The advantage of the ETU framework is that one can use multiple inference strategies without re-training a model. This is especially useful in cases where it is not a-priori clear which performance measure should be optimized, or predictions for different purposes are needed at the same time. On the other hand, as this framework optimizes the performance directly on a given test set, it is not designed to make predictions for single instances independently. The ETU framework has mainly been studied in the case of binary classification problems. We are not aware of any work that focuses on optimizing the complex performance metrics in the ETU framework for multi-label classification. One could try to generalize the results from binary classification, but the existing algorithms might not scale well to the extreme number of labels and they do not take the budget of  $k$  labels into account.
Our paper gives novel and non-trivial results regarding this challenging optimization problem. We have thoroughly analyzed the ETU framework for a wide class of performance metrics, derived optimal prediction rules and constructed their computationally efficient approximations with provable regret guarantees and being robust against model misspecification. Our algorithm, based on block coordinate descent, scales effortlessly to XMLC problems and obtains promising empirical results.
Overall, we identified four categories of utilities, that differ in the complexity of the optimization algorithm—whether to use instance-wise optimization as in Section 4.2, or the block coordinate-ascent (Algorithm 1)—and the guarantees for the result—whether semi-empirical quantities (Section 4.1) lead to an optimal solution, or a suboptimal with error bounded by Theorem 5.2. These are given as follows and summarized in Table 3:
Table 3: Four different classes of utilities
Linear in	Approx.	Algorithm
t, q, p	No	Instance-wise
t, q	Yes	Instance-wise
t, p	No	Block-Coordinate
—	Yes	Block-Coordinate
- Fully linear: Optimal predictions for metrics that are linear in all entries of the confusion matrix, as (6), can be solved exactly in an instance-wise manner. Examples are classical metrics such as instance-wise precision@k, or propensity-scored precision@k.
- Linear in predictions: Approximately optimal predictions for metrics that are linear in the predictions as given in (15) can be obtained using instance-wise optimization, by switching from  $\varPsi_{\mathrm{ETU}}$  to  $\bar{\varPsi}_{\mathrm{ETU}}$ . An example is macro recall@k.
- Linear in labels: If a metric is linear in the label variables as given in (14), then  $\tilde{\Psi}_{\mathrm{ETU}} \equiv \tilde{\Psi}_{\mathrm{ETU}}$ . However, the resulting combinatorial optimization problem for  $\tilde{\Psi}_{\mathrm{ETU}}$  is still complex enough, and we can solve it only locally. An example is macro precision@k.
- Nonlinear metrics: If none of the above apply, we have  $\tilde{\Psi}_{\mathrm{ETU}} \neq \tilde{\Psi}_{\mathrm{ETU}}$ , and have to solve it locally using block-coordinate ascent. This is the case of macro  $F$ -measure@ $k$ , or coverage@ $k$ .
The macro-averaged metrics budgeted at  $k$  are attractive for measuring the long-tail performance. Macro-averaging treats all the labels as equally important, while the budget of  $k$  predictions requires the prediction algorithm to choose the labels "wisely." We believe that this approach is substantially better than the one based on propensity-scored metrics. It is important to make the distinction between a metric used as a surrogate for training or inference, and its use as a performance metric in itself. While the former might be well justified for many metrics, the latter not necessarily as a metric might not have a clear interpretation of the calculated numbers.
Finally, the proposed framework is a plug-in approach that works on estimates of marginal probabilities and can be seamlessly applied to many existing state-of-the-art XMLC algorithms that are able to output such predictions, including XR-Transformer [51], CascadeXML [20], and algorithmic approaches for dealing with missing labels [40, 36]. It can also be combined with recently proposed methods that try to improve predictive performance on the tail labels by, e.g., leveraging labels features to estimate labels-correlations [29, 39, 9, 52] or to do data augmentation and generate new training points for tail labels [43, 21, 8].
A part of computational experiments for this paper had been performed in Poznan Supercomputing and Networking Center. We want to acknowledge the support of Academy of Finland via grants 347707 and 348215, and also thank Mohammadreza Qaraei for providing pre-generated LPEs,  $\hat{\eta}$ , obtained by LightXML using computational resources of CSC - IT Center for Science, Finland.
A Order-invariant linearly decomposable task metrics
In this section, we provide a closer look at the family  $\mathcal{L}_{\mathrm{OI}}$  of order-invariant task utilities that can be decomposed into a sum of label-specific functions as laid out in Section 2 in the main paper.
We first formalize the notion of instance-order invariance, and then show that this implies the possibility of reformulating the utility function based on the confusion matrix. We further prove that it is sufficient to know the marginal label probabilities  $\eta(\pmb{x})$  for each instance  $\pmb{x}$  in order to evaluate any utility  $\psi \in \mathcal{L}_{\mathrm{OI}}$  in the ETU-framework.
A.1 Order-invariant task utilities as confusion-matrix metrics