Title: Generalized test utilities for long-tail performance in extreme multi-label classification
Venue: NeurIPS 2023
Accepted: yes
Keywords: extreme multi-label classification

long-tail labels performance

complex performance measures
Abstract: Extreme multi-label classification (XMLC) is the task of selecting a small subset of relevant labels from a very large set of possible labels. 
As such, it is characterized by long-tail labels, i.e., most labels have very few positive instances. With standard performance measures such as precision@k, a classifier can ignore tail labels and still report good performance. However, it is often argued that correct predictions in the tail are more "interesting" or "rewarding," but the community has not yet settled on a metric capturing this intuitive concept. The existing propensity-scored metrics fall short on this goal by confounding the problems of long-tail and missing labels. In this paper, we analyze generalized metrics budgeted "at k" as an alternative solution. To tackle the challenging problem of optimizing these metrics, we formulate it in the expected test utility (ETU) framework, which aims to optimize the expected performance on a given test set. We derive optimal prediction rules and construct their computationally efficient approximations with provable regret guarantees and being robust against model misspecification. Our algorithm, based on block coordinate descent, scales effortlessly to XMLC problems and obtains promising results in terms of long-tail performance.

Generalized test utilities for long-tail performance in extreme multi-label classification

Abstract

Extreme multi-label classification (XMLC) is the task of selecting a small subset of relevant labels from a very large set of possible labels. As such, it is characterized by long-tail labels, i.e., most labels have very few positive instances. With standard performance measures such as precision@k, a classifier can ignore tail labels and still report good performance. However, it is often argued that correct predictions in the tail are more 'interesting' or 'rewarding,' but the community has not yet settled on a metric capturing this intuitive concept. The existing propensity-scored metrics fall short on this goal by confounding the problems of long-tail and missing labels. In this paper, we analyze generalized metrics budgeted 'at k' as an alternative solution. To tackle the challenging problem of optimizing these metrics, we formulate it in the expected test utility (ETU) framework, which aims to optimize the expected performance on a fixed test set. We derive optimal prediction rules and construct computationally efficient approximations with provable regret guarantees and robustness against model misspecification. Our algorithm, based on block coordinate ascent, scales effortlessly to XMLC problems and obtains promising results in terms of long-tail performance.

1 Introduction

Extreme multi-label classification (XMLC) is a challenging task with a wide spectrum of real-life applications, such as tagging of text documents [10], content annotation for multimedia search [14], or different type of recommendation [5, 1, 33, 44, 53, 28, 7]. Because of the nature of its applications, the typical approach in XMLC is to predict exactly k labels (e.g., corresponding to k slots in the user interface) which optimize a standard performance metric such as precision or (normalized) discounted cumulative gain. Given the enormous number of labels in XMLC tasks, which can reach millions or more, it is not surprising that many of them are very sparse, and hence make the label distribution strongly long-tailed [2]. It has been noticed that algorithms can achieve high performance on the standard metrics, but never predict any tail labels [42]. Therefore, there is a need to develop

a metric that prefers 'rewarding' [48], 'diverse' [3], and 'rare and informative' [34] labels over frequently-occurring head labels. Currently, the XMLC community attempts to capture this need using propensity-scored performance metrics [15]. These metrics give increased weight to tail labels, but have been derived from the perspective of missing labels, and as such they are not really solving the problem of tail labels [41].

In Table 1 we compare different metrics for budgeted at k predictions. We train a PLT model [17] on the full AMAZONCAT-13K dataset [27] and a reduced version with the 1000 most popular labels only. The test is performed for both models on the full set of labels. The standard metrics are only slightly perturbed by reducing the label space to the head labels. This holds even for propensity-scored precision, which decreases by just 1%-20% despite discarding over 90% of the label space. In contrast, macro measures and coverage decrease between 60% and 90% if tail labels are ignored. These results show that budgeted-atk macro measures might be very attractive in the context of long tails. Macro-averaging treats all the labels equally important, preventing the labels with a small number of positive examples to be ignored. Furthermore, the budget of k labels "requires" the presence of long-tail labels in a compact set of predicted labels.

While we can easily use these measures to evaluate and compare different methods, we also would like to make predictions that directly optimize these metrics. The existing approaches to macro-averaged metrics consider the unconstrained case, in which label-wise optimization is possible [47, 11, 23, 16, 22, 26]. Each binary problem can be then solved under one of two frameworks for optimizing complex performance measures, namely population utility (PU) or expected test utility (ETU) [49, 12]. The former aims at optimizing the performance on the population level. The latter optimizes the performance directly on a given test set. Interestingly, in both frameworks the optimal solution is based on thresholding conditional label probabilities [12], but the resulting thresholds are different with the discrepancy diminishing with the size of the test set. The threshold tuning for PU is usually performed on a validation set [47, 26], while the exact optimization for ETU is performed on a test set. It requires cubic time in a general case and quadratic time in some special cases [49, 32]. Approximate solutions can be obtained in linear time [25, 12].

These approaches cannot be directly applied if prediction of exactly k labels for each instance is required. In such case, the optimization problems for different labels are tightly coupled through this constraint, making the final problem much more difficult. Despite the fact that optimization of complex performance metrics is a well-established problem, considered not only in binary and multi-label classification as discussed above, but also in multi-class classification [30, 31], the results presented in this paper go beyond the state-of-the-art as budgeted-atk predictions have not yet been analyzed in this context. Let us underline that the requirement of k predictions is natural for recommendation systems, in which exactly k slots are available in the user interface to display recommendations. Even in situations where this does not apply, requiring the prediction to be 'at k ' can be advantageous, as it prevents trivial solutions such as predicting nothing (for precision) or everything (for recall).

In this paper, we investigate optimal solutions for the class of utility functions that can be linearly decomposed over labels into binary utilities, which includes both instance-wise weighted measures and macro-averages. We solve the problem in the ETU framework which is well-suited, for example, to recommendation tasks in which recommendations for all users or items are rebuilt in regular intervals. In this case, we can first obtain probability estimates of individual labels for each instance in the test set, and then provide optimal predictions for a given metric based on these estimates. We

derive optimal prediction rules and construct computationally efficient approximations with provable guarantees, formally quantifying the influence of the estimation error of the label probabilities on the suboptimality of the resulting classifier. This result is expressed in the form of a regret bound [4, 30, 22, 13]. It turns out that for most metrics of interest, a small estimation error results in at most a small drop of the performance, which confirms our method is viable for applications. Our general algorithm, based on block coordinate ascent, scales effortlessly to XMLC problems and obtains promising empirical results.

2 Setup and notation

Let x ∈ X denote an input instance, and y ∈ { 0 , 1 } m =: Y the vector indicating the relevant labels, distributed according to P ( y | x ) . We consider the prediction problem in the expected test utility (ETU) framework, that is, we assume that we are given a known set of n instances X = [ x 1 , . . . , x n ] T ∈ X n with unknown labels, on which we have to make predictions. 1 Our goal is to assign each instance x i a set of exactly k (out of m ) labels represented as a k -hot vector ˆ y i ∈ Y k := { y ∈ Y : ∥ y ∥ 1 = k } , and we let ˆ Y = [ˆ y 1 , . . . , ˆ y n ] T denote the entire n × m prediction matrix for a set of instances X .

In the ETU framework, we treat X as given and only make an assumption about the labeling process for the test sample: the labels y i ∈ Y corresponding to x i ∈ X do not depend on any other instances, that is P ( Y | X ) = ∏ n i =1 P ( y i | x i ) , where we use Y = [ y 1 , . . . , y n ] T ∈ Y n to denote the entire label matrix. We assume the quality of predictions ˆ Y is jointly evaluated against the observed labels Y by a task utility Ψ ( Y , ˆ Y ) , and define the optimal (Bayes) prediction ˆ Y ⋆ as the one maximizing the expected task utility Ψ ETU:

We consider task utilities Ψ ( Y , ˆ Y ) that linearly decompose over labels, i.e., there exists ψ j such that

We allow the functions ψ j to be non-linear themselves and different for each label j . This is a large class of functions, which encompasses weighted instance-wise and macro-averaged utilities, the two groups of functions which we thoroughly analyze in the next sections.

Let us next define the binary confusion matrix C ( y , ˆ y ) for a vector y of n ground truth labels and a corresponding vector ˆ y of binary predictions:

By indexing from 0, the entry c 00 corresponds to true negatives, c 01 to false positives, c 10 to false negatives, and c 11 to true positives.We define the multi-label confusion tensor 2 C ( Y , ˆ Y ) := [ C ( y :1 , ˆ y :1 ) , . . . , C ( y : m , ˆ y : m )] being the concatenation of binary confusion matrices of all m labels.

Assuming the utility function (2) to be invariant under instance reordering, i.e., its value does not change if rows of both matrices are re-ordered using the same permutation, we can define Ψ in terms of confusion matrices, instead of ground-truth labels and predictions (shown in Appendix A.1):

Finally, we assume that we have access to a label probability estimator (LPE) ˆ η ( x ) that estimates the marginal probability of each label given the instance, η ( x ) = ( η 1 ( x ) , . . . , η m ( x )) := E y | x [ y ] . Such an LPE can be attained by fitting a model on an additional training set of n ′ examples ( x i , y i ) n ′ i =1 using a proper composite loss function [37], which is a common approach in XMLC, e.g., [18].

3 Performance measures for tail labels

3.1 Instance-wise weighted utility functions

By assigning utility (or cost) to each correct/wrong prediction for each label, we can construct an instance-wise weighted utility u w : Y × Y -→ R ≥ 0 with labels y ∈ Y and predictions ˆ y ∈ Y as

We use w j 00 , w j 01 , w j 10 , and w j 11 to express the utility of true negatives, false positives, false negatives, and true positives, respectively. The corresponding task loss results from summing over all instances. By interchanging the order of summation, we can see that it is of form (4):

In other words, the instance-wise weighted utilities can be seen as linear confusion-based metrics. By choosing w j 00 = w j 11 = m -1 and w j 01 = w j 10 = 0 , the expression (5) reduces to the @ k -variant of the Hamming utility . Similarly, w j 11 = k -1 and w j 00 = w j 01 = w j 10 = 0 yield precision @ k .

Another example is the popular propensity-scoring approach [15], commonly used as a tailperformance metric in XMLC. Here, the weights are computed based on training data through

where n ′ is the number of training instances, ˆ π j is the empirical prior of label j , and parameters a and b are (potentially) dataset-dependent. This form of weighting has been derived from a missing-labels perspective, so its application to tail labels is not fully justified [41]. Also, it introduces two more hyperparameters, which makes the interpretation and comparison of its values rather difficult. It is not less heuristical than other approaches like power-law or logarithmic weighting, given by:

3.2 Macro-average of non-decomposable utilities

Macro-averaging usually concerns non-decomposable binary utilities such as the F-measure. In this case, we set up ψ j ( · , · ) = m -1 ψ ( · , · ) for all labels j in (2), yielding:

By using ψ pr ( C ) := c 11 / ( c 11 + c 01 ) , this becomes macro-precision , for ψ rec ( C ) := c 11 / ( c 11 + c 10 ) we get macro-recall , and for ψ F β ( C ) := ( β +1) c 11 / ((1+ β ) c 11 + β 2 c 10 + c 01 ) the macro-F-measure .

Another measure that is promising for the evaluation of long-tailed performance is coverage . It is sometimes used as an auxiliary measure in XMLC [15, 3, 42, 41]. This metric detects for how many different labels the classifier is able to make at least one correct prediction. In our framework, this is achieved by using an indicator function on the true positives, ψ cov ( C ) := 1 [ c 11 > 0] .

4 Optimal predictions

Let us start with the observation that the label probabilities η are sufficient to make optimal predictions. With the assumption that P ( Y | X ) = ∏ n i =1 P ( y i | x i ) , we obtain (cf. Appendix A.2):

This equation lays out a daunting optimization task, as is requires summing over 2 n summands y ′ . In case of binary classification, there exist methods to solve the problem exactly in O ( n 3 ) , or in O ( n 2 )

in some special cases [32]. By using semi-empirical quantities (defined below), [12] provides an approximate algorithm that runs in O ( n ) . Following this approach, we construct a semi-empirical ETU approximation. If this approximation results in a linear function of the predictions, the problem decomposes over instances and can be solved easily. Otherwise, we use an algorithm that leads to locally optimal predictions. A minor modification of this algorithm can be used for coverage.

4.1 Semi-empirical ETU approximation

Since the entries of the confusion matrix are linearly dependent, it suffices to use three independent combinations. More precisely, we parameterize the confusion matrix by the true positives t = c 11 , predicted positives q = c 11 + c 01 , and ground-truth positives p = c 11 + c 10 , and use t = ( t 1 , . . . , t m ) , q = ( q 1 , . . . , q m ) , and p = ( p 1 , . . . , p m ) to reformulate the ETU objective:

In order to compute Ψ ETU, one needs to take into account every possible combination of confusionmatrix values, and calculate the corresponding value of Ψ , which is then averaged according to the respective probabilities. A computationally easier approach is to take the expectation over the labels first, leading to semi-empirical quantities:

where ˜ q = q follows because the number of predicted positives depends only on the predictions ˆ Y . This allows us to define the semi-empirical ETU risk

In particular, the third argument to Ψ , ˜ p , is a constant that does not depend on predictions.

Note that, if Ψ is linear in all arguments depending on the random variable Y , then the approximation is exact, due to the linearity of expectations. Aside from instance-wise measures, which we showed to be linear above, the approximation is also exact for the more general class of functions of the form

An important example is macro-precision, with f t ( q ) = q -1 and f q = f p = 0 . In the general case, ˜ Ψ ETU as a surrogate for Ψ ETU leads only to O (1 / √ n ) error as will be shown in Theorem 5.2, while substantially simplifying the optimization process.

4.2 Linear confusion-matrix measures

We start the discussion on optimization of (13) with a special case in which ˜ Ψ ETU is linear in the prediction-dependent arguments t, q , that is, if

i.e., both f t ( p ) and f q ( p ) depend on p only, and f p ( p ) · p can be dropped as it is a constant.

Aside from instance-wise weighted utilities (cf. Appendix A.3), which are linear in all arguments, this form also holds for weights dependent on the (empirical) label priors, e.g., power law weights of the form f t ( p ) = p -α and f q = 0 , which reduce to macro-recall for α = 1 . If one defines the weights with respect to externally determined label priors, i.e., approximations to E [ y j ] , which are fixed and thus independent of the test sample X , then the power-law metrics turn into instance-wise weighted utilities.

From (6) we know that we can reformulate the optimization problem using an instance-wise weighted utility u w with weights:

Hence, the optimal predictions can be derived for each instance x ∈ X separately, leading to

Plugging in the definition of u w from (5), and collecting terms, the expected loss is of the form

where we call g j ( x ) := η j ( x ) f t ( p j ) + f q ( p j ) the gain of predicting label j for a given instance x . The optimal prediction is to select the k labels with the largest values g j ( x ) ,

Here, select-topk ( g ) denotes an operation that maps a vector of scores to a binary vector that contains 1s only at the positions of the k largest elements of g .

4.3 General non-decomposable macro measures

Finally, we turn to the general problem for budgeted-atk macro-measures based on nonlinear Ψ . As this can be a very hard discrete optimization problem in general, we use an iterative approach based on block-coordinate ascent that constructs a sequence of predictions, ˆ Y 0 , ˆ Y 1 , . . . , with non-decreasing utility, so that we end up with a solution that is locally optimal.

Assume the predictions are fixed for all instances except x s , where they are given by z . In that case, we can write the semi-empirical quantities from (12) as

with analog expansions for q and ˜ p . Plugging into (13) leads to the following optimization:

As everything except z j ∈ { 0 , 1 } is given, we can interpret ψ j as a linear function of z j , and define a gain vector with elements g j = ψ j (1) -ψ j (0) . The optimal prediction z ∗ is then given by z ∗ = select-topk ( g ) , in a similar form as in case of linear confusion-matrix measures. We get ˆ Y t +1 by replacing the s th row of ˆ Y t with z ∗ , and know that ˜ Ψ ETU ( ˆ Y t +1 ) ≥ ˜ Ψ ETU ( ˆ Y t ) . Then we switch to the next instance s ← s +1 , and repeat this process until no more progress is made.

The algorithm starts by predicting k random labels for each instance. To speed up the computations we cache two quantities, ˜ t t j and q t j , for each label:

With this, we can compute (21) in O ( m ) time using the following formulas:

The block coordinate ascent (BCA) procedure is shown in more detail in Algorithm 1. Obviously, the actual implementation cannot use the unknown values η , but instead has to rely on the LPE estimates ˆ η ( x i ) . The stopping criterion for the algorithm is whether, after going over a whole set of instances, the improvement in the objective value over the previous value is lower than ϵ , which ensures that the algorithm terminates for every bounded utility in O (1 /ϵ ) . In practice, even with small ϵ , the algorithm usually terminates after a few iterations. The time and space complexity of the single iteration are both O ( nm ) . If ψ is a linear function, corresponding to an instance-wise weighted utility (6) such as macrorecall, the algorithm recovers the optimal solution in the first iteration, stopping after the second.

In general, Algorithm 1 requires multiple iterations. This can be computationally expensive and requires all of the data to be available at once. We thus propose a greedy algorithm that takes into account only statistics of previously seen instances while performing a single pass over the dataset, which allows for semi-online optimization. The greedy algorithm is outlined in Appendix B.4.

4.4 Optimization of coverage

One measure for which the ETU approximation is not exact is coverage as ψ cov ( t, q, p ) := 1 [ t > 0] is nonlinear . In this case, we can do better than Algorithm 1, by reformulating Ψ ETU for coverage as

and performing block coordinate ascent directly on this expression, as detailed in Appendix B.3.

5 Regret bounds

̸

Computing optimal predictions relies on an access to the conditional marginal probabilities η ( x ) . In practice, however, η ( x ) are unknown, and are replaced by the LPE ˆ η ( x ) to do inference ( plug-in approach). Furthermore, we replaced the ETU objective Ψ ETU with an approximation ˜ Ψ ETU. As generally ˆ η ( x ) = η ( x ) and Ψ ETU = ˜ Ψ ETU, this procedure may result in sub-optimal predictions, the errors of which we would like to control.

̸

If we are able to control the change of the utility under small changes in its arguments, we can show that the suboptimality of the plug-in predictor relative to the optimal predictor (called Ψ -regret ) decreases with increasing test size n and decreasing probability estimation error.

To this end we assume that each ψ j is a p -Lipschitz function, which allows us to bound the approximation error.

Definition 5.1 ( p -Lipschitz [13]) . A binary classification metric ψ ( t, q, p ) is said to be p -Lipschitz if

for any q, q ′ ∈ [0 , 1] , p, p ′ ∈ (0 , 1) , 0 ≤ t ≤ min( p, q ) , and 0 ≤ t ′ ≤ min( p ′ , q ′ ) . The constants L t ( p ) , L q ( p ) , L p ( p ) are allowed to depend on p , in contrast to the standard Lipschitz functions.

As shown in Appendix C.3, most of metrics of interest satisfy p -Lipschitz assumption, including the linear confusion-matrix measures (6) with fixed weights (e.g., Hamming utility, precision), macro-recall, macro-F-measure, etc., with macro-precision and coverage being notable exceptions.

Theorem 5.2. Let each ψ j be p -Lipschitz with constants L j t ( p ) , L j q ( p ) , L j p ( p ) . For any ˆ Y it holds:

Thus, using ˜ Ψ ETU as a surrogate for Ψ ETU leads only to O (1 / √ n ) error, diminishing with the test size, while substantially simplifying the optimization process.

Given a decomposable metric of the form (4), let ˜ Y † be the plug-in prediction matrix optimizing the semi-empirical ETU with plugged-in probability estimates, Ψ ( E y ∼ ˆ η ( X ) [ t ] , q , E y ∼ ˆ η ( X ) [ p ] ) .

Theorem 5.3. Let ˜ Y † be defined as above. Under the assumptions of Theorem 5.2:

A similar statement, presented in Appendix C.4, can be made for the unapproximated ETU case.

Thus, the methods described in Section 4 can be used with probability estimates replacing the true marginals, and as long as the estimator is reliable, the resulting predictions will have small Ψ -regret. This also justifies the plug-in approach used in the experiments in Section 7.

6 Efficient inference

The optimization algorithms introduced so far need to obtain ˆ η j ( x i ) for all labels and instances first. Then, the BCA inference is of O ( nm ) time and O ( nm ) space complexity for a single iteration. This

is problematic in the setting of XMLC, where many methods aim to predict probabilities only for top labels in time sublinear in m . Fortunately, this characteristic of XMLC algorithms can be combined with the introduced algorithms to efficiently obtain an approximate solution. Instead of predicting all η , we can predict probabilities only for topk ′ labels with highest ˆ η j , where k ≪ k ′ ≪ m . For all other labels we then assume ˆ η j = 0 . Under the natural assumption that ψ is non-decreasing in true positives and non-increasing in predicted positives, we can leverage the sparsity and consider labels with non-zero ˆ η j only (using sparse vectors to represent ˆ η ( x i ) ) to reduce the time and space complexity to O ( n ( k ′ + k log k )) and O ( n ( k ′ + k )) , respectively. As in real-world datasets the number of relevant labels ∥ y ∥ 1 is much lower than m , and most η j ( x i ) are close to 0, with reasonably selected k ′ , according to Theorem 5.3, we should only slightly increase the regret. A pseudocode for the sparse variant of Algorithm 1 can be found in Appendix D.

Alternatively, we can leverage probabilistic label trees (PLTs) [16], a popular approach in XMLC [35, 45, 19, 50, 7], to efficiently search for 'interesting' labels. Originally, PLTs find top labels with the highest ˆ η j . In [46], an A ∗ -search algorithm has been introduced for finding k labels with highest value of g j η j ( x ) , where g j ∈ [0 , ∞ ) is a gain assigned to label j . In Appendix F, we present a more general version of this procedure that can be used to efficiently obtain an exact solution of presented BCA or Greedy algorithms for some of the metrics considered in this work.

7 Experiments

To empirically test the introduced framework, we use popular benchmarks from the XMLC repository [6]. We train the LIGHTXML [18] model (with suggested default hyper-parameters) on provided training sets to obtain ˆ η for all test instances. We then plug these estimates into different inference strategies and report the results across the discussed measures. To run the optimization algorithm efficiently, we use k ′ = 100 or k ′ = 1000 to pre-select for each instance the top k ′ labels with the highest ˆ η j as described in Section 6. 3

We use the following inference strategies:

TOP-K- the optimal strategy for precision @ k : selection of k labels with the highest η j (default prediction strategy in many XMLC methods).

PS-K- the optimal strategy for propensity-scored precision @ k : selection of k labels with the highest w j, prop 11 η j , with w j, prop 11 given by the empirical model of Jain et al. [15] (Equation 7) with values a and b recommended by the authors.

POW-K, LOG-K- the optimal strategy for power-law and log weighted instance-wise utilities: selection of k labels with the highest w j, pl 11 η j or w j, log 11 η j . For power-law, we use β = 0 . 5 .

MACRO-PBCA, MACRO-RBCA, MACRO-F1BCA, COVBCA- the block coordinate ascent (Algorithm 1) for optimizing macro-precision, -recall, -F1, and coverage,

We expect a strategy suited for a given metric to obtain the best results on this metric. Nevertheless, this might not always be the case, as in the derivation of our algorithms, we needed to apply different types of approximation to scale them to XMLC problems. We are mainly interested in the performance on the general nondecomposable macro measures since they seem to be well-tailored to long tails, and their optimization is the most challenging. The results are presented in Table 2. Notice that in almost all cases, the specialized inference strategies are indeed the best on the measure they aim to optimize and achieve substantial gains on corre-

10

20

30

40

50

Macro-F1@3

70

80

Instance-P@3

PS-K

Log-K

Macro-F1

BCA

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

sponding metrics compared to the basic TOP-K inference. The other weighted strategies, PS-K, POW-K, LOG-K, usually provide a much smaller improvement over TOP-K and never beat strategies designed for specific macro measures. As the reported performance depends on three things: the inherent difficulty of the data, the success of the inference algorithm, and the quality of the provided

marginal probabilities, the results might diverge from expectations in some cases. In Appendix E, we provide more details and conduct further experiments to investigate the impact of randomness, stopping conditions, quality of probability estimates, and shortlisting on the results. We also present similar experiments with probabilistic label trees used as an LPE.

Unfortunately, our results show that optimization of macro-measures comes with the cost of a significant drop in performance on instance-wise measures. Ideally, we would like to improve the performance on tail labels without sacrificing too much of general performance. To achieve such a trade-off, we can use straight-forward interpolation of instance-wise precision-at-k, as it is covered by our framework, and a selected macro-measure, since the considered class of utility functions is closed under linear combinations. Such an objective can be optimized by the proposed block-coordinate algorithm without any modification. As an example, we plot in Table 7 the results of optimizing a linear combination of instance-wise precision and the macro-F1 measure: Ψ ( Y , ˆ Y ) = (1 -α ) Ψ Instance-P ( Y , ˆ Y ) + αΨ Macro-F1 ( Y , ˆ Y ) , using different values of α ∈ [0 , 1] . In Appendix E.5, we provide more plots for different utilities, datasets, and values of k . All the plots show that the instance-vs-macro curve has a nice concave shape that dominates simple baselines. In particular, the initial significant improvement on macro-measures comes with a minor drop in instance-measures, and only if one wants to optimize more strongly for macro-measures, the drop on instance-wise measures becomes more severe.

8 Discussion

The advantage of the ETU framework is that one can use multiple inference strategies without re-training a model. This is especially useful in cases where it is not a-priori clear which performance measure should be optimized, or predictions for different purposes are needed at the same time. On the other hand, as this framework optimizes the performance directly on a given test set, it is not designed to make predictions for single instances independently. The ETU framework has mainly been studied in the case of binary classification problems. We are not aware of any work that focuses on optimizing the complex performance metrics in the ETU framework for multi-label classification. One could try to generalize the results from binary classification, but the existing algorithms might not scale well to the extreme number of labels and they do not take the budget of k labels into account.

Our paper gives novel and non-trivial results regarding this challenging optimization problem. We have thoroughly analyzed the ETU framework for a wide class of performance metrics, derived optimal prediction rules and constructed their computationally efficient approximations with provable regret guarantees and being robust against model misspecification. Our algorithm, based on block coordinate descent, scales effortlessly to XMLC problems and obtains promising empirical results.

Overall, we identified four categories of utilities, that differ in the complexity of the optimization algorithm-whether to use instancewise optimization as in Section 4.2, or the block coordinate-ascent (Algorithm 1)- and the guarantees for the result-whether semi-empirical quantities (Section 4.1) lead to an optimal solution, or a suboptimal with error bounded by Theorem 5.2. These are given as follows and summarized in Table 3:

Fully linear : Optimal predictions for metrics that are linear in all entries of the confusion matrix, as (6), can be solved exactly in an instance-wise manner. Examples are classical metrics such as instance-wise precision @ k , or propensity-scored precision @ k .

Linear in predictions : Approximately optimal predictions for metrics that are linear in the predictions as given in (15) can be obtained using instance-wise optimization, by switching from Ψ ETU to ˜ Ψ ETU. An example is macro recall @ k .

Linear in labels : If a metric is linear in the label variables as given in (14), then ˜ Ψ ETU ≡ Ψ ETU. However, the resulting combinatorial optimization problem for ˜ Ψ ETU is still complex enough, and we can solve it only locally . An example is macro precision @ k .

̸

Nonlinear metrics : If none of the above apply, we have ˜ Ψ ETU = Ψ ETU, and have to solve it locally using block-coordinate ascent . This is the case of macro F -measure @ k , or coverage @ k .

The macro-averaged metrics budgeted at k are attractive for measuring the long-tail performance. Macro-averaging treats all the labels as equally important, while the budget of k predictions requires the prediction algorithm to choose the labels 'wisely." We believe that this approach is substantially better than the one based on propensity-scored metrics. It is important to make the distinction between a metric used as a surrogate for training or inference, and its use as a performance metric in itself. While the former might be well justified for many metrics, the latter not necessarily as a metric might not have a clear interpretation of the calculated numbers.

Finally, the proposed framework is a plug-in approach that works on estimates of marginal probabilities and can be seamlessly applied to many existing state-of-the-art XMLC algorithms that are able to output such predictions, including XR-Transformer [51], CascadeXML [20], and algorithmic approaches for dealing with missing labels [40, 36]. It can also be combined with recently proposed methods that try to improve predictive performance on the tail labels by, e.g., leveraging labels features to estimate labels-correlations [29, 39, 9, 52] or to do data augmentation and generate new training points for tail labels [43, 21, 8].

Acknowledgments

A part of computational experiments for this paper had been performed in Poznan Supercomputing and Networking Center. We want to acknowledge the support of Academy of Finland via grants 347707 and 348215, and also thank Mohammadreza Qaraei for providing pre-generated LPEs, ˆ η , obtained by LightXML using computational resources of CSC - IT Center for Science, Finland.

References

Rahul Agrawal, Archit Gupta, Yashoteja Prabhu, and Manik Varma. Multi-label learning with millions of labels: Recommending advertiser bid phrases for web pages. In 22nd International World Wide Web Conference, WWW '13, Rio de Janeiro, Brazil, May 13-17, 2013 , pages 13-24. International World Wide Web Conferences Steering Committee / ACM, 2013.

Rohit Babbar and Bernhard Schölkopf. Dismec: Distributed sparse machines for extreme multi-label classification. In Proceedings of the tenth ACM international conference on web search and data mining , pages 721-729, 2017.

Rohit Babbar and Bernhard Schölkopf. Data scarcity, robustness and extreme multi-label classification. Machine Learning , 108(8):1329-1351, 2019.

Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

Alina Beygelzimer, John Langford, Yury Lifshits, Gregory B. Sorkin, and Alexander L. Strehl. Conditional probability tree estimation analysis and algorithms. In UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009 , pages 51-58. AUAI Press, 2009.

Kush. Bhatia, Kunal. Dahiya, Himanshu Jain, Anshul Mittal, Yashoteja Prabhu, and Manik Varma. The extreme classification repository: Multi-label datasets and code, 2016. URL http://manikvarma.org/downloads/XC/XMLRepository.html .

Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, and Inderjit S. Dhillon. Taming pretrained transformers for extreme multi-label text classification. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020 , pages 3163-3171. ACM, 2020.

Eli Chien, Jiong Zhang, Cho-Jui Hsieh, Jyun-Yu Jiang, Wei-Cheng Chang, Olgica Milenkovic, and Hsiang-Fu Yu. Pina: Leveraging side information in extreme multi-label classification via predicted instance neighborhood aggregation. arXiv preprint arXiv:2305.12349 , 2023.

Kunal Dahiya, Ananye Agarwal, Deepak Saini, K Gururaj, Jian Jiao, Amit Singh, Sumeet Agarwal, Purushottam Kar, and Manik Varma. Siamesexml: Siamese networks meet extreme classifiers with 100m labels. In International Conference on Machine Learning , pages 23302340. PMLR, 2021.

Ofer Dekel and Ohad Shamir. Multiclass-multilabel classification with more classes than examples. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010 , volume 9 of JMLR Proceedings , pages 137-144. JMLR.org, 2010.

Krzysztof Dembczynski, Arkadiusz Jachnik, Wojciech Kotlowski, Willem Waegeman, and Eyke Hüllermeier. Optimizing the f-measure in multi-label classification: Plug-in rule approach versus structured loss minimization. In International conference on machine learning , pages 1130-1138. PMLR, 2013.

Krzysztof Dembczynski, Wojciech Kotlowski, Oluwasanmi Koyejo, and Nagarajan Natarajan. Consistency analysis for binary classification revisited. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 961-969. PMLR, 2017.

Krzysztof Dembczy´ nski, Wojciech Kotłowski, Oluwasanmi Koyejo, and Nagarajan Natarajan. Consistency analysis for binary classification revisited. In Proceedings of the 34th International Conference on Machine Learning (ICML 2017) , volume 70 of Proceedings of Machine Learning Research , pages 961-969. PMLR, 2017.

Jia Deng, Sanjeev Satheesh, Alexander C. Berg, and Fei-Fei Li. Fast and balanced: Efficient label tree learning for large scale object recognition. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain. , pages 567-575, 2011.

Himanshu Jain, Yashoteja Prabhu, and Manik Varma. Extreme multi-label loss functions for recommendation, tagging, ranking and other missing label applications. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '16, page 935-944. Association for Computing Machinery, 2016. ISBN 9781450342322.

Kalina Jasinska, Krzysztof Dembczynski, Róbert Busa-Fekete, Karlson Pfannschmidt, Timo Klerx, and Eyke Hüllermeier. Extreme F-measure maximization using sparse probability estimates. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , volume 48 of JMLR Workshop and Conference Proceedings , pages 1435-1444. JMLR.org, 2016.

Kalina Jasinska-Kobus, Marek Wydmuch, Krzysztof Dembczy´ nski, Mikhail Kuznetsov, and Róbert Busa-Fekete. Probabilistic label trees for extreme multi-label classification. CoRR , abs/2009.11218, 2020.

Ting Jiang, Deqing Wang, Leilei Sun, Huayi Yang, Zhengyang Zhao, and Fuzhen Zhuang. Lightxml: Transformer with dynamic negative sampling for high-performance extreme multilabel text classification. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7987-7994, 2021.

Sujay Khandagale, Han Xiao, and Rohit Babbar. Bonsai: diverse and shallow trees for extreme multi-label classification. Machine Learning , 109(11):2099-2119, 2020.

Siddhant Kharbanda, Atmadeep Banerjee, Erik Schultheis, and Rohit Babbar. Cascadexml: Rethinking transformers for end-to-end multi-resolution training in extreme multi-label classification. Advances in Neural Information Processing Systems , 35:2074-2087, 2022.

Siddhant Kharbanda, Devaansh Gupta, Erik Schultheis, Atmadeep Banerjee, Vikas Verma, and Rohit Babbar. Gandalf: Data augmentation is all you need for extreme classification. 2022.

Wojciech Kotłowski and Krzysztof Dembczy´ nski. Surrogate regret bounds for generalized classification performance metrics. Machine Learning , 10:549-572, 2017.

Oluwasanmi Koyejo, Nagarajan Natarajan, Pradeep Ravikumar, and Inderjit S. Dhillon. Consistent multilabel classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 3321-3329, 2015.

Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. Numba: A llvm-based python jit compiler. In Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC , pages 1-6, 2015.

David D. Lewis. Evaluating and optimizing autonomous text classification systems. In SIGIR'95, Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Seattle, Washington, USA, July 9-13, 1995 (Special Issue of the SIGIR Forum) , pages 246-254. ACM Press, 1995.

Yu-Jen Lin and Chih-Jen Lin. On the thresholding strategy for infrequent labels in multi-label classification. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , CIKM '23, page 1441-1450, New York, NY, USA, 2023. Association for Computing Machinery.

Julian McAuley, Rahul Pandey, and Jure Leskovec. Inferring networks of substitutable and complementary products. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 785-794, 2015.

Tharun Kumar Reddy Medini, Qixuan Huang, Yiqiu Wang, Vijai Mohan, and Anshumali Shrivastava. Extreme classification in log memory using count-min sketch: A case study of amazon search with 50m products. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' AlchéBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 13265-13275. Curran Associates, Inc., 2019.

Anshul Mittal, Noveen Sachdeva, Sheshansh Agrawal, Sumeet Agarwal, Purushottam Kar, and Manik Varma. Eclare: Extreme classification with label graph correlations. In Proceedings of the Web Conference 2021 , pages 3721-3732, 2021.

Harikrishna Narasimhan, Harish Ramaswamy, Aadirupa Saha, and Shivani Agarwal. Consistent multiclass algorithms for complex performance measures. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 2398-2407, Lille, France, 07 2015. PMLR.

Harikrishna Narasimhan, Harish G. Ramaswamy, Shiv Kumar Tavker, Drona Khurana, Praneeth Netrapalli, and Shivani Agarwal. Consistent multiclass algorithms for complex metrics and constraints, 2022. URL https://arxiv.org/abs/2210.09695 .

Nagarajan Natarajan, Oluwasanmi Koyejo, Pradeep Ravikumar, and Inderjit Dhillon. Optimal classification with multivariate losses. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 1530-1538, New York, New York, USA, 20-22 Jun 2016. PMLR.

Yashoteja Prabhu and Manik Varma. Fastxml: a fast, accurate and stable tree-classifier for extreme multi-label learning. In The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, New York, NY, USA - August 24 - 27, 2014 , pages 263-272. ACM, 2014.

Yashoteja Prabhu, Anil Kag, Shilpa Gopinath, Kunal Dahiya, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. Extreme multi-label learning with label features for warm-start tagging, ranking & recommendation. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining , pages 441-449, 2018.

Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018 , pages 993-1002. ACM, 2018.

Mohammadreza Qaraei, Erik Schultheis, Priyanshu Gupta, and Rohit Babbar. Convex surrogates for unbiased loss functions in extreme classification with missing labels. In Proceedings of The Web Conference 2021 , WWW'21, New York, NY, USA, 2021. Association for Computing Machinery. doi:10.1145/3442381.3450139. URL https://doi.org/10.1145/3442381. 3450139 .

Mark Reid and Robert Williamson. Convexity of proper composite binary losses. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , volume 9 of Proceedings of Machine Learning Research , pages 637-644, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https: //proceedings.mlr.press/v9/reid10a.html .

Stuart J. Russell and Peter Norvig. Artificial Intelligence: a modern approach . Pearson, 3 edition, 2009.

Deepak Saini, Arnav Kumar Jain, Kushal Dave, Jian Jiao, Amit Singh, Ruofei Zhang, and Manik Varma. Galaxc: Graph neural networks with labelwise attention for extreme classification. In Proceedings of the Web Conference 2021 , pages 3733-3744, 2021.

Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. Unbiased recommender learning from missing-not-at-random implicit feedback. In Proceedings of the 13th International Conference on Web Search and Data Mining , WSDM'20, page 501-509, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450368223. doi:10.1145/3336191.3371783. URL https://doi.org/10.1145/3336191.3371783 .

Erik Schultheis, Marek Wydmuch, Rohit Babbar, and Krzysztof Dembczynski. On missing labels, long-tails and propensities in extreme multi-label classification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1547-1557, 2022.

Tong Wei and Yu-Feng Li. Does tail label help for large-scale multi-label learning? IEEE Transactions on Neural Networks and Learning Systems , 31(7):2315-2324, 2020.

Tong Wei, Wei-Wei Tu, Yu-Feng Li, and Guo-Ping Yang. Towards robust prediction on tail labels. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1812-1820, 2021.

Jason Weston, Ameesh Makadia, and Hector Yee. Label partitioning for sublinear ranking. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013 , volume 28 of JMLR Workshop and Conference Proceedings , pages 181-189. JMLR.org, 2013.

Marek Wydmuch, Kalina Jasinska, Mikhail Kuznetsov, Róbert Busa-Fekete, and Krzysztof Dembczynski. A no-regret generalization of hierarchical softmax to extreme multi-label classification. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31 , pages 6355-6366. Curran Associates, Inc., 2018.

Marek Wydmuch, Kalina Jasinska-Kobus, Rohit Babbar, and Krzysztof Dembczynski. Propensity-scored probabilistic label trees. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '21, page 2252-2256, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380379.

Yiming Yang. A study of thresholding strategies for text categorization. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '01, page 137-145, New York, NY, USA, 2001. Association for Computing Machinery. ISBN 1581133316.

Hui Ye, Zhiyu Chen, Da-Han Wang, and Brian Davison. Pretrained generalized autoregressive model with adaptive probabilistic label clusters for extreme multi-label text classification. In International Conference on Machine Learning , pages 10809-10819. PMLR, 2020.

Nan Ye, Kian Ming Adam Chai, Wee Sun Lee, and Hai Leong Chieu. Optimizing F-measure: A tale of two approaches. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26 - July 1, 2012 . icml.cc / Omnipress, 2012.

Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, and Shanfeng Zhu. Attentionxml: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 5812-5822. Curran Associates, Inc., 2019.

Jiong Zhang, Wei-Cheng Chang, Hsiang-Fu Yu, and Inderjit Dhillon. Fast multi-resolution transformer fine-tuning for extreme multi-label text classification. Advances in Neural Information Processing Systems , 34:7267-7280, 2021.

Ruohong Zhang, Yau-Shian Wang, Yiming Yang, Donghan Yu, Tom Vu, and Likun Lei. Longtailed extreme multi-label text classification with generated pseudo label descriptions. arXiv preprint arXiv:2204.00958 , 2022.

Jingwei Zhuo, Ziru Xu, Wei Dai, Han Zhu, Han Li, Jian Xu, and Kun Gai. Learning optimal tree models under beam search. In Proceedings of the 37th International Conference on Machine Learning , Vienna, Austria, 2020. PMLR.

A Order-invariant linearly decomposable task metrics

In this section, we provide a closer look at the family L OI of order-invariant task utilities that can be decomposed into a sum of label-specific functions as laid out in Section 2 in the main paper.

We first formalize the notion of instance-order invariance, and then show that this implies the possibility of reformulating the utility function based on the confusion matrix. We further prove that it is sufficient to know the marginal label probabilities η ( x ) for each instance x in order to evaluate any utility ψ ∈ L OI in the ETU-framework.

A.1 Order-invariant task utilities as confusion-matrix metrics

We show in Theorem A.3 that any utility function Ψ ( Y , ˆ Y ) that is invariant under instance reordering can be defined in terms of confusion matrices. In this way, we justify our choice in the main paper to focus on losses of the form given in (4).

More precisely, let σ ∈ S ( n ) be a permutation of rows, that is, for Y = [ y 1 , . . . , y n ] T , we define σ Y = [ y σ (1) , . . . , y σ ( n ) ] T . Then we can make the following definition:

Definition A.1 (Invariant under instance reordering) . Let n , m ∈ N and Ψ : { 0 , 1 } n × m × { 0 , 1 } n × m -→ R be a function of discrete labels and predictions. If Ψ remains unchanged for every permutation of rows σ ∈ S ( n ) , i.e.,

then we call Ψ a function that is invariant under instance reordering . We denote the set of instanceorder-invariant losses with m labels as

We further define the set of all possible confusion matrices with n instances as

where N is the set of natural numbers including 0. Now we are ready to provide a lemma for the binary case:

Lemma A.2. Let ψ : { 0 , 1 } n × { 0 , 1 } n -→ R be a binary loss function that is invariant under instances reordering. Then there exists a function ϕ : C ( n ) -→ R such that ψ = ϕ ◦ C .

Proof. We provide an explicit construction of ψ . To that end, let ˜ C ∈ C ( n ) be one such confusion matrix, then there exists ( y , ˆ y ) ∈ { 0 , 1 } n ×{ 0 , 1 } n given by

Define ϕ such that ϕ ( ˜ C ) = ψ ( y , ˆ y ) .

Now, let ( y ′ , ˆ y ′ ) ∈ { 0 , 1 } n ×{ 0 , 1 } n be an arbitrary label-prediction combination, with C ( y ′ , ˆ y ′ ) = ˜ C . Then there exists a permutation σ such that σ y ′ = y and σ ˆ y ′ = ˆ y . By the invariance assumption, it holds that

As the original ˜ C ∈ C ( n ) was arbitrary, the statement is shown.

Wecan now extend this lemma to show the equivalence of two definitions of the task losses considered in this paper:

Theorem A.3 (Equivalence of order-invariance and confusion-matrix losses) . Let n , m ∈ N , and Y = { 0 , 1 } m . Define the set of instance-order invariant, label-averaged losses as

and the set of confusion-matrix based, label-averaged losses as

Then these two descriptions are equivalent, in the sense that

that is, every instance-order invariant loss can be written as a confusion-matrix loss, and every confusion-matrix loss leads to an instance-order invariant loss.

Proof. As calculating the confusion matrix is in itself an operation that is invarant under instance reordering, each Φ ◦ C clearly is instance-order invariant.

On the other hand, let Ψ ( Y , ˆ Y ) = ∑ m j =1 ψ j ( y : j , ˆ y : j ) , then by Lemma A.2 there exist ϕ 1 , . . . , ϕ m such that

Summing up the individual ϕ j gives Φ of the correct form.

A.2 Sufficiency of label probability estimates

If Ψ ∈ L OI is an instance-order invariant decomposable task utility, then its ETU risk is a function of the predictions ˆ Y and the marginal label probabilities η ( x ) as given by (10). This means that it is not required to estimate the m 2 -1 values of the full joint probability distribution P [ Y | x ] .

Lemma A.4. Let Ψ ∈ L OI be an instance-order invariant decomposable task utility, and assume that the labels of different instances are independent, i.e., that P ( Y | X ) = ∏ n i =1 P ( y i | x i ) . Then we have

Proof.

A.3 Representation change from the instance-wise form to the cost-matrix form

If we start with the linear metric given in the instance-wise form according to (5), the corresponding representation according to (15) may be shifted by a constant value, and can be derived through the following:

Thus, we get f t ( p ) = w 11 -w 10 -w 01 + w 00 and f q ( p ) = w 01 -w 00 .

B Block Coordinate Ascent

In this section, we provide additional analysis and variations of the block coordinate ascent algorithm. We first give an intuition into the gain-based selection of labels performed by the algorithm in the case of macro-precision. Then, we will illustrate how, for metrics that are linear in the prediction argument as per (15), the algorithm will recover the closed-form solution given in (19). Afterwards, we present an alternative formulation specifically for coverage, and conclude with greedy versions of the BCA algorithm.

B.1 Example: Macro-Precision

As an example, consider the optimization of macro-precision. In this case, ψ j ( t, q, p ) = t/q . Plugging into the gain formula gives

This term is positive only if t j /q j < ˆ η j ( x s ) , i.e., predicting the label has a positive impact on the overall utility if its chance of being relevant, ˆ η j ( x s ) , is larger than the current estimate of precision for label j . Of course, the 'atk ' constraint means that, for a given instance, it may not be possible to select all labels with positive gain, or necessary to select some labels with negative gain to fulfill the constraint.

B.2 Global optimality for linear metrics

If ψ j is a linear function, i.e., it corresponds to an instance-wise measure as in (6), the gain g , defined through lines 9-11 in Algorithm 1, becomes

because the influence of the other instances cancels out. This exactly reproduces the gain formula from Section 4.2, which means that for each instance, the block coordinate-ascent algorithm will select the globally (approximately) optimal decision on the first pass. If the metric is fully linear, this will be the Bayes-optimal decision, otherwise, it is approximately optimal in the sense of Theorem 5.2. In both cases, the algorithm will perform a second pass over the entire dataset in order to check the stopping criterion. This second pass will leave all predictions unchanged.

B.3 The ETU approach for coverage

In this section, we derive the ETU risk presented initially in Section 4.4. Then, we present a block coordinate ascent algorithm for optimizing it. The step-by-step derivation of (24) is given below:

Because we assume labels for one instance to be independent on all other instances, i.e., P ( Y | X ) = ∏ n i =1 P ( y i | x i ) , we obtain:

Based on this result we can construct a block coordinate ascent procedure which, for predictions being fixed for all instances except x s , optimizes the following problem:

Analogously to Section 4.3, everything except z j ∈ { 0 , 1 } is given and we can define a gain vector with elements g j = ψ j cov (1) -ψ j cov (0) . Again the optimal prediction z ∗ is then given by z ∗ = select-topk ( g ) , and we get ˆ Y t +1 by replacing the s th row of ˆ Y t with z ∗ . Then we switch to the next instance s ← s +1 , and repeat this process until no more progress is made.

Notice that ∏ n i =1 (1 -η j ( x i )ˆ y ij ) corresponds to the probability of label j being irrelevant for all instances it was selected for. We denote this value as f j and use it to speed up computations in each iteration of the algorithm:

Algorithm 2 BCA for coverage ( X , ˆ η , k, ϵ )

1: ˆ y i ← select-randomk ( m ) for all i ∈ [ n ] 2: f ← ∏ n i =1 ( 1 -ˆ η ( x i ) ⊙ ˆ y i ) 3: u old ←-∞ , u new ← 1 -( m -1 ∑ m j =1 f j ) 4: while u new > u old + ϵ do 5: for s ∈ shuffle ([ n ]) do 6: f ← f ⊙ ( 1 -ˆ η ( x s ) ⊙ ˆ y s ) -1 7: g ← f ⊙ ˆ η ( x s ) 8: ˆ y s ← select-topk ( g ) 9: f ← f ⊙ ( 1 -ˆ η ( x s ) ⊙ ˆ y s ) 10: u old ← u new, u new ← 1 -( m -1 ∑ m j =1 f j ) 11: return ˆ Y

We can then compute the gain vector using the following formula:

This block coordinate ascent procedure for coverage is presented as Algorithm 2.

B.4 Greedy version of block coordinate-ascent

The inference strategy presented in Algorithm 1 requires multiple passes over the entire data. This might be computationally expensive and requires the whole dataset to be available at once. We, thus, propose a greedy version of the algorithm, that performs just a single pass over the dataset and makes decisions for each instance by taking into account only statistics of previously seen instances instead of all other instances in the dataset. This allows to apply this algorithm to the semi-online setting, where the algorithm observes and predicts for instances one by one, but does not receive immediate feedback for its decisions, in contrast to the usual online learning setting. The procedure is outlined as Algorithm 3, and its greedy variant as Algorithm 4. The results of this algorithm in comparison to the BCA approach are given later in Appendix E.

Algorithm 3 Greedy ( X , ˆ η , k )

Algorithm 4 Greedy for coverage ( X , ˆ η , k )

1: f ← 1

2: for i ∈ [ n ] do

3: g ← f ⊙ ˆ η ( x i )

4: ˆ y i ← select top-k ( g )

5: f ← f ⊙ ( 1 -ˆ η ( x i ) ⊙ ˆ y i )

6: return ˆ Y

C Detailed Regret Analysis

In this section, we discuss the p -Lipschitzness condition of the metrics of interest, and prove Theorem 5.2 and Theorem 5.3.

C.1 p -Lipschitz utility functions

First, recall the definition from the main paper:

Definition 5.1 ( p -Lipschitz [13]) . A binary classification metric ψ ( t, q, p ) is said to be p -Lipschitz if

for any q, q ′ ∈ [0 , 1] , p, p ′ ∈ (0 , 1) , 0 ≤ t ≤ min( p, q ) , and 0 ≤ t ′ ≤ min( p ′ , q ′ ) . The constants L t ( p ) , L q ( p ) , L p ( p ) are allowed to depend on p , in contrast to the standard Lipschitz functions.

The rationale behind this definition is that while we need to control the change in value of the metric under small changes in its arguments, a standard definition of Lipschitzness (with global constant) would not be satisfied by many popular metrics. For the same reason, we only require stability for non-trivial problems, that is, in cases where the rate of positives p is neither zero nor one.

Relaxing the definition to allow the constants vary as a function of p suffices to prove our stability results as well as regret bounds, while it is satisfied by most of the metrics of interest, as shown below. The notable exception not present in Table 4 is the precision metric, which is not p -Lipschitz due to its behavior for q → 0 .

Lemma C.1. The linear confusion-matrix measures defined by (6) :

with fixed coefficient matrices { W j } m j =1 are decomposable functions with p -Lipschitz components.

Proof. The metric can be rewritten in a decomposable form Ψ = ∑ j ψ j , where each ψ j in the ( t, q, p ) -parameterization has the following form:

where T j , Q j , P j , C j are some combinations of the coefficient matrices { W j } m j =1 . Being a linear function of t, q, p , ψ j is Lipschitz.

Lemma C.2 ( p -Lipschitzness of common utilities [13]) . All metrics in Table 4 are p -Lipschitz.

Proof. Here, we only prove the lemma for recall, which has not been covered by Proposition 1 of Dembczy´ nski et al. [13].

C.2 Stability of the semi-ETU approximation

We are now ready to prove that when the metric of interest has p -Lipschitz components, the semi-ETU approximation ˜ Ψ ETU presented in Section 4.1 remains close to the true objective Ψ ETU. For the sake of convenience, let us recall the definition of the ETU objective:

as well as its approximation:

We prove the following result:

Theorem 5.2. Let each ψ j be p -Lipschitz with constants L j t ( p ) , L j q ( p ) , L j p ( p ) . For any ˆ Y it holds:

Proof. For the sake of the analysis, denote the Lipschitz constants as L j t := L j t (˜ p j ) and L j p := L j p (˜ p j ) . Using definitions (50) and (51) and applying Jensen's inequality, we have

We now bound each term in the sum by ( L j t + L j p ) / (2 √ n ) , which will prove the theorem.

For each j ∈ [ m ] , using p -Lipschitzness of ψ j we have:

Taking expectation on both sides gives:

where the last inequality follows from Jensen's inequality applied to a concave function x ↦→ √ x . Using the fact that ˜ t j = E Y | X [ t j ] and ˜ p j = E Y | X [ p j ] , we have

as t j = n -1 ∑ n i =1 y ij ˆ y ij is an average of n Bernoulli i.i.d. random variables y ij ˆ y ij , each having variance at most 1 4 ; and using the same argument, E Y | X [ ( p j -˜ p j ) 2 ] ≤ 1 4 n . This gives

and finishes the proof.

C.3 Regret of semi-ETU under model misspecification

In this section, we quantify the influence of the estimation error of marginal probabilities, proving Theorem 5.3.

Notation To emphasize the dependence of Ψ ETU on the label probability estimates, in this section we will write

This notation is well-defined, as we have shown in Appendix A.2 that in fact, the dependence on X is mediated only through the marginal label probabilities η ( X ) = ( η ( x 1 ) , . . . , η ( x n )) , and we abbreviate η ( X ) as η . Similarly, we will write

Note that q is independent of η . This allows us to write the semi-ETU objective as

where we dropped the dependence of ˜ t and ˜ p on η as clear from the context. The optimal (Bayes) predictor ˆ Y ⋆ is the one which maximizes the expected utility with regard to the true label probabilities. In the notation of this chapter, (1) reads

Unfortunately, the learning algorithm does not know the true label marginals η ( X ) , but has only access to the estimates ˆ η ( X ) = (ˆ η ( x 1 ) , . . . , ˆ η ( x n )) for the considered set of instances. The algorithm computes its predictions ˆ Y † by using the estimates in place of the true marginals. Thus, it can only generate

We can make the same definitions also for the semi-empirical ETU optimization, leading to

Regret for semi-ETU approximation First, we show that for metrics with p -Lipschitz components, the Ψ -regret of the resulting semi-ETU predictor (62), which is the suboptimality of ˜ Y † (with respect to ˆ Y ⋆ ) in terms of Ψ ETU, is well-controlled and upper-bounded by the estimation error of the marginals. As the resulting expression is somewhat unwieldy, we then apply some further bounding to arrive at the much simpler result stated in the main paper in Theorem 5.3.

Lemma C.3 (Misspecification for semi-ETU approximation) . Let Ψ ∈ L OI be an instance-order invariant linearly decomposable loss function that is p -Lipschitz, and ˆ Y an arbitrary set of predictions. Then, the difference of the semi-empirical ETU risk when using two different versions of the marginals, η and η ′ , is bounded by the difference η -η ′ through

Proof. Plugging in the definitions, we have

where the last line used the definition of p -Lipschitzness.

The terms under the sum can be bounded by the estimation error of η ′ :

where in the first line we used ˆ y ij ∈ { 0 , 1 } .

Lemma C.4 (Regret bound for semi-ETU approximation) . Let Ψ ∈ L OI be an instance-order invariant linearly decomposable loss function that is p -Lipschitz. Then we have

Proof. Using the optimality of ˜ Y † and a supremum bound, we get

We then use Theorem 5.2 and Lemma C.3 to bound

where we use the shorthand ˜ p j = ˜ p j ( η ) .

At the cost of having a less strict bound, we can simplify this to

Theorem 5.3. Let ˜ Y † be defined as above. Under the assumptions of Theorem 5.2:

where B := √ m -1 ∑ m j =1 ( L j t (˜ p j ) + L j p ( p j )) 2 is the quadratic mean of the Lipschitz constants.

Proof. Using Cauchy-Schwartz, we can bound

For the other term, we can use the inequality between arithmetic and quadratic mean, so that

C.4 Regret for non-approximated ETU

We can formulate the equivalent of Lemma C.4 also for the maximizer of the true empirical ETU risk, ˆ Y † :

Lemma C.5 (Regret bound for ETU maximization) . Let Ψ ∈ L OI be an instance-order invariant linearly decomposable loss function that is p -Lipschitz. Then we have

Proof. Following the same line of argument as for Lemma C.4, we get

Next, we make use of the semi-empirical ETU risk to bound

The individual terms can now again be bounded by Theorem 5.2 and Lemma C.3.

Algorithm 5 Sparse BCA ( X , ˆ η csr , k, ϵ )

Algorithm 7 Sparse greedy ( X , ˆ η csr , k )

D Sparse and greedy algorithms

In this section, we present pseudocodes for sparse variants of the introduced algorithms, which we used in the main experiment to compute efficiently the results for large datasets. These variants use compressed sparse row (CSR) representation for row vectors. CSR vectors are represented as a list of tuples a csr i := { ( index , value ) : value = 0 } . In these algorithms, we replace ˆ η ( x i ) and ˆ y i with their sparse variants, ˆ η csr ( x i ) := { ( j, ˆ η j ( x i )) : ˆ η j ( x i ) = 0 } and ˆ y csr i := { ( j, ˆ y ij ) : ˆ y ij = 0 } , respectively. We need to ensure that of both ˆ η csr ( x i ) and ˆ y csr i are ordered by their indices for efficient element-wise multiplication (Hadamard product) between them. If we assume that the size of | ˆ y csr i | = k and | ˆ η csr ( x i ) | = k ′ , the resulting time and space complexity of the sparse algorithms are O ( n ( k ′ + k log k )) and O ( n ( k ′ + k )) , respectively. Additionally, we present the greedy variants of sparse BCA algorithms introduced so far that do only one pass over the dataset.

(

̸

̸

We present the sparse variant of BCA (Algorithm 1) in Algorithm 5 and the sparse variant of Greedy (Algorithm 3) in Algorithm 7. The sparse versions of their specialized counterparts for coverage are presented in Algorithm 6 and Algorithm 8.

̸

E Extended results, experiments and details of empirical comparison

E.1 Datasets characteristics

In Table 5 we include an overview of the main characteristics of the datasets used in this paper.

E.2 Extended results of the main experiment

In this section, we present the extended results of our main experiment from Section 7. Here in Table 6 and Table 7, we present additional results for k = 1 , as well as results for additional methods:

MACRO-PGREED, MACRO-F1GREED, COVGREED- the greedy algorithms (Algorithm 3) for optimizing macro-precision, -F1, and coverage (Algorithm 4),

MACRO-RPRIOR- the optimal strategy for macro-recall: selection of k labels with the highest p -1 j η j , with p estimated on the training set.

Because both greedy and block coordinate-ascent algorithms depend on the order in which examples are presented, we check if this has a significant influence on the results. We ran the procedures 5 times with different seeds and presented both mean and standard deviation, which is mostly less than 0 . 25 for the macro measures.

E.3 Results on true labels

In the main experiment, the reported performance depends on three things: the inherent difficulty of the data, the success of the inference algorithm and the quality of the provided marginal probabilities. To be able to judge these terms, we also ran the inference algorithms on probabilities generated from the test set labels: η j ( x i ) = 1 iff Y ij = 1 to rest only our inference strategy. We present the result of this experiment in Table 8. Notice that in this experiment, the specialized inference strategies are almost always the best on measures they aim to optimize. Also, the obtained results for macro measures are significantly below 100%. This is because, in this datasets, not all labels are represented with even one positive training example in the test set. Still, there remains a significant gap between these results and the results of the main experiments, with probability estimates coming from the LIGHTXML model.

E.4 Impact of stopping criterion and using only topk ′ predictions on predictive and computational performance.

In this section, we investigate the impact of stopping criterion and using precalculated topk ′ predictions on the predictive performance as well as computational performance.

To materialize all marginal probability estimates for datasets like WIKIPEDIALARGE-500K and AMAZON-670K, one requires an enormous amount of memory and time. Because of that, we are not able to calculate the exact results for these datasets, and instead, we used pre-select topk ′ labels with the highest ˆ η j as described in Section 6. In the main experiment in Section 7, we used k ′ = 100 for smaller datasets (EURLEX-4K, WIKI-31K, AMAZONCAT-13K) and k ′ = 1000 for larger datasets (WIKIPEDIALARGE-500K and AMAZON-670K). Here, we additionally compare the results for k ′ = 100 and k ′ = 1000 for WIKIPEDIALARGE-500K and AMAZON-670K investigating the impact of predictive and computational performance.

Inference

Instance @1

Macro @1

Instance @3

Macro @3

strategy

P

±

std

R

±

std

P

±

std

R

±

std

F1

±

std

±

std

P

±

std

R

±

std

P

±

std

R

±

std

F1

±

std

±

std

EURLEX-4K

TOP-K

85.77

±

0.00

17.41

±

0.00

15.02

±

0.00

5.10

±

0.00

6.98

±

0.00

16.63

±

0.00

74.20

±

0.00

44.21

±

0.00

24.85

±

0.00

16.45

±

0.00

18.63

±

0.00

31.27

±

0.00

PS-K

74.39

±

0.00

14.99

±

0.00

24.76

±

0.00

13.00

±

0.00

15.22

±

0.00

28.82

±

0.00

69.01

±

0.00

41.06

±

0.00

30.12

±

0.00

23.72

±

0.00

25.08

±

0.00

40.29

±

0.00

POW-K

β

=0

.

25

76.40

±

0.00

15.42

±

0.00

24.58

±

0.00

12.50

±

0.00

14.86

±

0.00

28.29

±

0.00

70.08

±

0.00

41.73

±

0.00

29.43

±

0.00

22.87

±

0.00

24.34

±

0.00

38.93

±

0.00

POW-K

β

=0

.

5

70.19

±

0.00

14.07

±

0.00

24.72

±

0.00

13.48

±

0.00

15.55

±

0.00

29.07

±

0.00

65.35

±

0.00

38.95

±

0.00

30.49

±

0.00

25.26

±

0.00

25.93

±

0.00

42.11

±

0.00

LOG-K

78.60

±

0.00

15.88

±

0.00

23.67

±

0.00

11.61

±

0.00

14.02

±

0.00

27.20

±

0.00

71.03

±

0.00

42.30

±

0.00

29.04

±

0.00

22.10

±

0.00

23.76

±

0.00

37.99

±

0.00

MACRO-PGREED

50.82

±

0.47

10.04

±

0.06

29.02

±

0.20

10.82

±

0.10

13.48

±

0.11

33.85

±

0.28

49.15

±

0.32

29.16

±

0.16

31.18

±

0.32

22.63

±

0.27

23.18

±

0.29

44.30

±

0.20

MACRO-PBCA

47.16

±

0.34

9.38

±

0.08

33.76

±

0.21

10.85

±

0.18

14.24

±

0.19

34.18

±

0.21

41.13

±

0.41

24.29

±

0.22

38.78

±

0.09

16.77

±

0.05

20.63

±

0.05

40.60

±

0.08

MACRO-RPRIOR

57.59

±

0.00

11.49

±

0.00

23.52

±

0.00

14.08

±

0.00

15.44

±

0.00

28.64

±

0.00

53.09

±

0.00

31.55

±

0.00

29.85

±

0.00

26.93

±

0.00

25.59

±

0.00

44.87

±

0.00

MACRO-RBCA

44.55

±

0.00

8.87

±

0.00

22.49

±

0.00

12.91

±

0.00

14.59

±

0.00

26.14

±

0.00

41.47

±

0.00

24.61

±

0.00

30.36

±

0.00

25.16

±

0.00

24.37

±

0.00

44.26

±

0.00

MACRO-F1GREED

63.90

±

0.11

12.78

±

0.02

28.35

±

0.13

12.77

±

0.14

15.77

±

0.14

33.29

±

0.20

60.95

±

0.24

36.30

±

0.17

31.60

±

0.11

24.93

±

0.11

26.26

±

0.12

43.98

±

0.20

MACRO-F1BCA

62.24

±

0.27

12.49

±

0.04

29.67

±

0.11

13.55

±

0.05

16.78

±

0.07

33.87

±

0.11

61.96

±

0.06

36.94

±

0.05

34.42

±

0.06

25.13

±

0.05

27.59

±

0.05

44.52

±

0.06

COVGREED

45.44

±

0.64

8.96

±

0.13

27.92

±

0.26

10.79

±

0.13

13.11

±

0.11

34.55

±

0.28

26.64

±

0.12

15.54

±

0.08

26.78

±

0.19

21.23

±

0.06

18.38

±

0.07

46.27

±

0.07

COVBCA

43.77

±

0.31

8.61

±

0.06

30.89

±

0.21

11.65

±

0.17

14.06

±

0.14

36.14

±

0.24

24.00

±

0.17

14.03

±

0.09

31.46

±

0.18

21.30

±

0.05

19.05

±

0.06

47.19

±

0.12

AMAZONCAT-13K

TOP-K

96.38

±

0.00

27.49

±

0.00

11.29

±

0.00

1.81

±

0.00

2.79

±

0.00

13.08

±

0.00

83.35

±

0.00

63.01

±

0.00

25.71

±

0.00

11.37

±

0.00

13.83

±

0.00

33.05

±

0.00

PS-K

86.41

±

0.00

24.26

±

0.00

42.58

±

0.00

25.12

±

0.00

28.17

±

0.00

55.90

±

0.00

79.09

±

0.00

60.15

±

0.00

43.89

±

0.00

38.64

±

0.00

38.56

±

0.00

62.06

±

0.00

POW-K

β

=0

.

25

79.34

±

0.00

21.47

±

0.00

40.09

±

0.00

23.99

±

0.00

27.09

±

0.00

53.74

±

0.00

74.40

±

0.00

56.59

±

0.00

41.87

±

0.00

39.07

±

0.00

38.17

±

0.00

61.39

±

0.00

POW-K

β

=0

.

5

71.88

±

0.00

19.08

±

0.00

40.09

±

0.00

28.57

±

0.00

30.02

±

0.00

59.59

±

0.00

66.52

±

0.00

50.75

±

0.00

38.18

±

0.00

46.42

±

0.00

39.26

±

0.00

67.76

±

0.00

LOG-K

80.12

±

0.00

21.30

±

0.00

37.42

±

0.00

19.75

±

0.00

23.32

±

0.00

47.55

±

0.00

75.30

±

0.00

56.79

±

0.00

41.09

±

0.00

33.92

±

0.00

35.04

±

0.00

56.05

±

0.00

MACRO-PGREED

84.10

±

0.15

23.96

±

0.03

50.08

±

0.04

27.65

±

0.08

30.90

±

0.05

73.70

±

0.09

65.58

±

0.94

49.76

±

0.72

44.69

±

0.12

48.79

±

0.06

41.63

±

0.07

79.96

±

0.06

MACRO-PBCA

92.29

±

0.03

26.30

±

0.01

63.33

±

0.07

22.58

±

0.05

29.98

±

0.04

73.35

±

0.10

54.97

±

0.35

41.61

±

0.24

64.27

±

0.03

29.22

±

0.03

35.76

±

0.02

76.18

±

0.05

MACRO-RPRIOR

61.09

±

0.00

16.00

±

0.00

36.45

±

0.00

34.11

±

0.00

31.01

±

0.00

64.32

±

0.00

49.82

±

0.00

38.32

±

0.00

30.64

±

0.00

54.38

±

0.00

34.06

±

0.00

75.91

±

0.00

MACRO-RBCA

59.70

±

0.00

15.80

±

0.00

37.98

±

0.00

37.71

±

0.00

32.90

±

0.00

69.52

±

0.00

47.74

±

0.00

37.13

±

0.00

31.40

±

0.00

58.32

±

0.00

34.68

±

0.00

80.71

±

0.00

MACRO-F1GREED

71.87

±

0.02

19.14

±

0.00

46.48

±

0.08

30.92

±

0.04

34.45

±

0.05

70.44

±

0.09

69.04

±

0.02

52.71

±

0.01

42.55

±

0.06

49.15

±

0.07

43.27

±

0.05

75.52

±

0.08

MACRO-F1BCA

72.50

±

0.00

19.30

±

0.00

50.42

±

0.04

31.67

±

0.02

36.33

±

0.02

71.90

±

0.04

70.61

±

0.00

53.86

±

0.00

51.95

±

0.01

48.22

±

0.02

47.93

±

0.01

77.83

±

0.02

COVGREED

10.14

±

0.02

1.78

±

0.00

32.17

±

0.09

22.70

±

0.05

15.13

±

0.04

74.83

±

0.15

4.83

±

0.01

2.44

±

0.00

23.11

±

0.12

33.89

±

0.04

12.83

±

0.02

80.86

±

0.07

COVBCA

9.79

±

0.01

1.73

±

0.00

41.23

±

0.03

24.94

±

0.04

17.08

±

0.03

77.28

±

0.03

4.53

±

0.00

2.29

±

0.00

34.93

±

0.08

35.16

±

0.02

15.91

±

0.03

82.67

±

0.03

WIKI-31K

TOP-K

87.85

±

0.00

5.27

±

0.00

0.99

±

0.00

0.14

±

0.00

0.23

±

0.00

1.11

±

0.00

77.17

±

0.00

13.48

±

0.00

2.01

±

0.00

0.50

±

0.00

0.72

±

0.00

2.54

±

0.00

PS-K

73.00

±

0.00

4.33

±

0.00

3.01

±

0.00

0.68

±

0.00

1.00

±

0.00

3.78

±

0.00

67.95

±

0.00

11.89

±

0.00

3.89

±

0.00

1.47

±

0.00

1.93

±

0.00

5.47

±

0.00

POW-K

β

=0

.

25

72.93

±

0.00

4.29

±

0.00

2.80

±

0.00

0.61

±

0.00

0.91

±

0.00

3.50

±

0.00

66.00

±

0.00

11.50

±

0.00

3.62

±

0.00

1.35

±

0.00

1.79

±

0.00

5.15

±

0.00

POW-K

β

=0

.

5

62.12

±

0.00

3.65

±

0.00

3.44

±

0.00

0.90

±

0.00

1.27

±

0.00

4.57

±

0.00

55.06

±

0.00

9.59

±

0.00

4.49

±

0.00

2.14

±

0.00

2.59

±

0.00

6.98

±

0.00

LOG-K

73.17

±

0.00

4.32

±

0.00

2.52

±

0.00

0.53

±

0.00

0.81

±

0.00

3.16

±

0.00

65.20

±

0.00

11.34

±

0.00

3.40

±

0.00

1.24

±

0.00

1.66

±

0.00

4.84

±

0.00

MACRO-PGREED

37.14

±

0.49

2.15

±

0.04

5.17

±

0.09

1.04

±

0.02

1.53

±

0.03

6.17

±

0.07

32.65

±

0.13

5.62

±

0.02

6.89

±

0.06

2.78

±

0.05

3.44

±

0.05

10.02

±

0.06

MACRO-PBCA

31.46

±

0.24

1.80

±

0.01

6.48

±

0.05

1.24

±

0.02

1.80

±

0.02

6.48

±

0.05

32.86

±

0.66

5.66

±

0.12

9.13

±

0.01

2.55

±

0.03

3.38

±

0.03

9.21

±

0.02

MACRO-RPRIOR

45.30

±

0.00

2.63

±

0.00

3.91

±

0.00

1.51

±

0.00

1.85

±

0.00

5.37

±

0.00

36.67

±

0.00

6.35

±

0.00

5.34

±

0.00

3.55

±

0.00

3.60

±

0.00

9.38

±

0.00

MACRO-RBCA

12.95

±

0.00

0.74

±

0.00

2.29

±

0.00

1.27

±

0.00

1.41

±

0.00

2.61

±

0.00

13.78

±

0.00

2.36

±

0.00

4.77

±

0.00

3.24

±

0.00

3.10

±

0.00

7.26

±

0.00

MACRO-F1GREED

43.15

±

0.46

2.53

±

0.02

5.05

±

0.08

1.23

±

0.02

1.78

±

0.03

6.22

±

0.06

38.12

±

0.15

6.65

±

0.02

6.56

±

0.03

2.91

±

0.03

3.63

±

0.03

9.91

±

0.02

MACRO-F1BCA

39.82

±

0.20

2.33

±

0.01

5.67

±

0.04

1.41

±

0.01

2.03

±

0.02

6.43

±

0.04

37.32

±

0.07

6.52

±

0.01

7.81

±

0.02

3.15

±

0.02

4.10

±

0.02

10.42

±

0.02

COVGREED

40.24

±

0.44

2.33

±

0.03

4.34

±

0.05

0.93

±

0.01

1.36

±

0.02

6.15

±

0.06

27.39

±

0.14

4.66

±

0.03

5.21

±

0.03

2.72

±

0.01

2.97

±

0.01

10.07

±

0.02

COVBCA

40.52

±

0.38

2.35

±

0.02

5.10

±

0.02

1.11

±

0.00

1.60

±

0.00

6.72

±

0.03

27.00

±

0.12

4.63

±

0.03

6.24

±

0.03

3.18

±

0.01

3.41

±

0.01

11.04

±

0.04

WIKIPEDIALARGE-500K

TOP-K

74.94

±

0.00

24.26

±

0.00

12.21

±

0.00

6.74

±

0.00

7.73

±

0.00

15.36

±

0.00

56.02

±

0.00

45.70

±

0.00

20.15

±

0.00

18.87

±

0.00

17.14

±

0.00

32.24

±

0.00

PS-K

72.00

±

0.00

23.90

±

0.00

18.99

±

0.00

10.96

±

0.00

12.47

±

0.00

23.58

±

0.00

54.91

±

0.00

45.61

±

0.00

23.33

±

0.00

22.68

±

0.00

20.41

±

0.00

37.85

±

0.00

POW-K

β

=0

.

25

71.75

±

0.00

23.73

±

0.00

17.59

±

0.00

9.90

±

0.00

11.37

±

0.00

21.89

±

0.00

54.24

±

0.00

45.09

±

0.00

22.48

±

0.00

21.57

±

0.00

19.54

±

0.00

36.29

±

0.00

POW-K

β

=0

.

5

69.11

±

0.00

23.26

±

0.00

19.80

±

0.00

11.59

±

0.00

13.14

±

0.00

24.96

±

0.00

51.81

±

0.00

43.94

±

0.00

23.73

±

0.00

23.67

±

0.00

21.13

±

0.00

39.36

±

0.00

LOG-K

72.89

±

0.00

23.86

±

0.00

15.56

±

0.00

8.57

±

0.00

9.88

±

0.00

19.32

±

0.00

54.82

±

0.00

45.21

±

0.00

21.43

±

0.00

20.18

±

0.00

18.38

±

0.00

34.28

±

0.00

MACRO-PGREED

54.81

±

0.03

18.49

±

0.01

26.59

±

0.02

12.99

±

0.01

15.55

±

0.01

34.05

±

0.03

44.19

±

0.01

37.85

±

0.02

26.33

±

0.01

25.49

±

0.02

23.29

±

0.01

46.04

±

0.02

MACRO-PBCA

47.68

±

0.01

15.82

±

0.00

32.70

±

0.02

13.38

±

0.01

16.94

±

0.01

36.25

±

0.02

25.19

±

0.05

21.81

±

0.03

37.69

±

0.00

20.18

±

0.01

23.44

±

0.00

45.08

±

0.01

MACRO-RPRIOR

63.83

±

0.00

22.18

±

0.00

22.59

±

0.00

14.26

±

0.00

15.65

±

0.00

29.25

±

0.00

46.16

±

0.00

40.99

±

0.00

24.82

±

0.00

26.93

±

0.00

22.92

±

0.00

44.12

±

0.00

MACRO-RBCA

60.61

±

0.00

21.47

±

0.00

24.76

±

0.00

14.91

±

0.00

16.71

±

0.00

32.61

±

0.00

43.40

±

0.00

39.60

±

0.00

25.35

±

0.00

27.55

±

0.00

23.72

±

0.00

46.30

±

0.00

MACRO-F1GREED

62.48

±

0.02

21.42

±

0.01

25.28

±

0.02

12.84

±

0.01

15.39

±

0.01

31.99

±

0.03

48.66

±

0.01

41.43

±

0.01

25.52

±

0.01

24.85

±

0.01

22.76

±

0.01

43.68

±

0.01

MACRO-F1BCA

61.41

±

0.01

20.98

±

0.00

29.23

±

0.01

13.61

±

0.00

16.84

±

0.00

34.32

±

0.01

43.82

±

0.00

36.40

±

0.00

35.42

±

0.01

23.69

±

0.00

26.01

±

0.00

46.36

±

0.01

COVGREED

47.41

±

0.03

15.55

±

0.01

23.70

±

0.02

13.07

±

0.01

14.76

±

0.01

33.36

±

0.02

28.67

±

0.01

25.48

±

0.01

23.56

±

0.00

25.44

±

0.01

20.65

±

0.00

47.46

±

0.02

COVBCA

46.84

±

0.02

15.47

±

0.01

26.16

±

0.01

14.30

±

0.01

16.05

±

0.01

36.06

±

0.01

27.31

±

0.00

24.55

±

0.00

25.92

±

0.00

26.76

±

0.01

21.60

±

0.00

50.16

±

0.01

AMAZON-670K

TOP-K

46.86

±

0.00

9.68

±

0.00

5.05

±

0.00

3.34

±

0.00

3.68

±

0.00

6.12

±

0.00

41.71

±

0.00

24.08

±

0.00

10.77

±

0.00

9.68

±

0.00

9.51

±

0.00

14.35

±

0.00

PS-K

44.95

±

0.00

9.37

±

0.00

5.87

±

0.00

3.97

±

0.00

4.35

±

0.00

6.88

±

0.00

41.05

±

0.00

23.77

±

0.00

11.86

±

0.00

10.54

±

0.00

10.42

±

0.00

15.50

±

0.00

POW-K

β

=0

.

25

45.49

±

0.00

9.45

±

0.00

5.76

±

0.00

3.86

±

0.00

4.25

±

0.00

6.79

±

0.00

41.30

±

0.00

23.89

±

0.00

11.65

±

0.00

10.35

±

0.00

10.24

±

0.00

15.27

±

0.00

POW-K

β

=0

.

5

44.68

±

0.00

9.33

±

0.00

5.94

±

0.00

4.04

±

0.00

4.41

±

0.00

6.93

±

0.00

40.90

±

0.00

23.71

±

0.00

11.99

±

0.00

10.65

±

0.00

10.53

±

0.00

15.64

±

0.00

LOG-K

46.09

±

0.00

9.54

±

0.00

5.55

±

0.00

3.68

±

0.00

4.06

±

0.00

6.61

±

0.00

41.54

±

0.00

24.00

±

0.00

11.34

±

0.00

10.10

±

0.00

9.98

±

0.00

14.94

±

0.00

MACRO-PGREED

43.47

±

0.04

9.03

±

0.01

7.94

±

0.01

3.86

±

0.00

4.69

±

0.01

8.51

±

0.01

38.80

±

0.02

22.48

±

0.01

14.18

±

0.01

10.63

±

0.01

11.34

±

0.01

17.20

±

0.01

MACRO-PBCA

42.37

±

0.06

8.83

±

0.01

8.73

±

0.01

3.99

±

0.01

4.91

±

0.01

8.77

±

0.01

33.79

±

1.46

19.75

±

0.80

17.27

±

0.37

10.53

±

0.11

12.12

±

0.02

17.75

±

0.01

MACRO-RPRIOR

43.08

±

0.00

9.08

±

0.00

6.09

±

0.00

4.26

±

0.00

4.61

±

0.00

6.98

±

0.00

39.85

±

0.00

23.20

±

0.00

12.48

±

0.00

11.07

±

0.00

10.91

±

0.00

16.12

±

0.00

MACRO-RBCA

42.33

±

0.00

8.92

±

0.00

6.82

±

0.00

4.33

±

0.00

4.86

±

0.00

7.46

±

0.00

39.42

±

0.70

22.92

±

0.36

13.71

±

0.61

11.17

±

0.21

11.47

±

0.34

17.12

±

0.41

MACRO-F1GREED

44.04

±

0.03

9.17

±

0.01

7.74

±

0.01

3.96

±

0.01

4.75

±

0.01

8.37

±

0.01

39.76

±

0.02

23.02

±

0.01

13.79

±

0.01

10.61

±

0.01

11.22

±

0.01

16.93

±

0.01

MACRO-F1BCA

43.36

±

0.03

9.05

±

0.01

8.32

±

0.01

4.14

±

0.00

5.01

±

0.00

8.56

±

0.01

37.34

±

0.63

21.70

±

0.35

16.49

±

0.36

10.75

±

0.03

12.17

±

0.08

17.65

±

0.07

COVGREED

42.47

±

0.04

8.58

±

0.01

7.21

±

0.01

3.75

±

0.00

4.43

±

0.00

8.25

±

0.00

35.88

±

0.03

20.57

±

0.02

13.05

±

0.01

10.53

±

0.01

10.75

±

0.01

17.08

±

0.01

COVBCA

42.31

±

0.04

8.56

±

0.01

7.69

±

0.00

3.86

±

0.00

4.61

±

0.00

8.52

±

0.00

35.38

±

0.01

20.32

±

0.01

14.04

±

0.00

10.85

±

0.00

11.23

±

0.00

17.70

±

0.00

Inference

Instance @1

Macro @1

Instance @3

Macro @3

Instance @5

Macro @5

Instance @10

Macro @10

strategy

P

R

P

R

F1

P

R

P

R

F1

P

R

P

R

F1

P

R

P

R

F1

EURLEX-4K

WEIGHTED-TOP-K

100.00

20.53

50.05

33.19

36.83

99.12

60.06

65.95

58.09

60.43

92.58

89.95

66.43

65.11

65.66

52.93

99.96

66.43

66.42

66.42

MACRO-PGREED

100.00

20.53

55.47

28.26

33.76

99.12

60.06

65.54

56.65

59.69

92.58

89.95

66.36

64.93

65.54

52.93

99.96

66.43

66.42

66.42

MACRO-PBCA

100.00

20.53

60.13

31.89

37.57

99.12

60.06

66.35

58.33

61.03

92.58

89.95

66.43

65.18

65.71

52.93

99.96

66.43

66.42

66.42

MACRO-RPRIOR

100.00

20.53

50.05

33.19

36.83

99.12

60.06

65.95

58.09

60.43

92.58

89.95

66.43

65.11

65.66

52.93

99.96

66.43

66.42

66.42

MACRO-RBCA

100.00

20.53

53.01

35.24

39.11

99.12

60.06

66.34

58.69

61.00

92.58

89.95

66.43

65.19

65.71

52.93

99.96

66.43

66.42

66.42

MACRO-F1GREED

100.00

20.53

55.17

31.35

36.27

99.12

60.06

65.64

57.17

59.99

92.58

89.95

66.36

64.94

65.55

52.93

99.96

66.43

66.42

66.42

MACRO-F1BCA

100.00

20.53

57.23

34.84

39.65

99.12

60.06

66.35

58.57

61.12

92.58

89.95

66.43

65.18

65.71

52.93

99.96

66.43

66.42

66.42

AMAZONCAT-13K

WEIGHTED-TOP-K

100.00

29.06

93.05

74.51

78.21

91.73

69.95

99.58

95.56

96.87

77.27

88.81

99.58

98.37

98.82

47.43

97.80

99.58

99.44

99.49

MACRO-PGREED

100.00

29.06

98.40

70.10

77.46

91.73

69.95

99.55

95.27

96.78

77.27

88.81

99.58

98.31

98.80

47.43

97.80

99.58

99.44

99.49

MACRO-PBCA

100.00

29.06

99.14

72.14

78.95

91.73

69.95

99.58

95.48

96.90

77.27

88.81

99.58

98.36

98.83

47.43

97.80

99.58

99.44

99.49

MACRO-RPRIOR

100.00

29.06

93.05

74.51

78.21

91.73

69.95

99.58

95.56

96.87

77.27

88.81

99.58

98.37

98.82

47.43

97.80

99.58

99.44

99.49

MACRO-RBCA

100.00

29.06

94.52

74.75

78.72

91.73

69.95

99.58

95.58

96.89

77.27

88.81

99.58

98.37

98.82

47.43

97.80

99.58

99.43

99.49

MACRO-F1GREED

100.00

29.06

98.29

72.53

78.68

91.73

69.95

99.56

95.38

96.82

77.27

88.81

99.58

98.32

98.80

47.43

97.80

99.58

99.43

99.49

MACRO-F1BCA

100.00

29.06

98.38

74.03

79.68

91.73

69.95

99.58

95.55

96.92

77.27

88.81

99.58

98.36

98.83

47.43

97.80

99.58

99.43

99.49

WIKI-31K

WEIGHTED-TOP-K

100.00

6.19

17.43

13.91

14.84

99.99

18.55

47.67

37.75

39.94

99.93

30.82

64.23

52.71

55.27

98.10

58.83

71.10

67.39

68.56

MACRO-PGREED

100.00

6.19

21.35

10.62

12.43

99.99

18.55

53.77

32.17

36.77

99.93

30.82

65.64

49.18

53.46

98.10

58.83

70.75

66.52

67.96

MACRO-PBCA

100.00

6.19

21.38

10.65

12.46

99.99

18.55

61.79

37.54

42.47

99.93

30.82

70.39

54.16

58.24

98.10

58.83

71.26

67.68

68.87

MACRO-RPRIOR

100.00

6.19

17.43

13.91

14.84

99.99

18.55

47.67

37.75

39.94

99.93

30.82

64.23

52.71

55.27

98.10

58.83

71.10

67.39

68.56

MACRO-RBCA

100.00

6.19

21.06

18.28

19.02

99.99

18.55

54.62

42.65

45.41

99.93

30.82

68.59

55.60

58.55

98.10

58.83

71.26

67.74

68.86

MACRO-F1GREED

100.00

6.19

21.20

14.60

16.09

99.99

18.55

54.49

37.19

40.99

99.93

30.82

66.47

51.53

55.19

98.10

58.83

70.80

66.73

68.09

MACRO-F1BCA

100.00

6.19

21.36

18.26

19.09

99.99

18.55

59.04

42.23

46.11

99.93

30.82

70.10

55.37

58.86

98.10

58.83

71.27

67.72

68.89

WIKIPEDIALARGE-500K

WEIGHTED-TOP-K

100.00

37.10

72.36

49.26

54.43

84.40

73.79

96.74

83.98

87.72

68.87

87.71

98.99

93.53

95.42

44.11

97.48

99.52

98.49

98.90

MACRO-PGREED

100.00

37.10

81.14

44.25

52.36

84.40

73.79

97.45

82.22

87.17

68.87

87.71

99.06

92.92

95.19

44.11

97.48

99.55

98.40

98.86

MACRO-PBCA

100.00

37.10

87.70

48.46

57.03

84.40

73.79

99.19

85.09

89.67

68.87

87.71

99.53

94.04

96.07

44.11

97.48

99.61

98.58

98.99

MACRO-RPRIOR

100.00

37.10

72.36

49.26

54.43

84.40

73.79

96.74

83.98

87.72

68.87

87.71

98.99

93.53

95.42

44.11

97.48

99.52

98.49

98.90

MACRO-RBCA

100.00

37.10

78.55

53.86

59.39

84.40

73.79

98.52

85.87

89.60

68.87

87.71

99.42

94.22

96.04

44.11

97.48

99.58

98.60

98.99

MACRO-F1GREED

100.00

37.10

80.53

48.26

55.46

84.40

73.79

97.63

83.29

87.80

68.87

87.71

99.08

93.19

95.33

44.11

97.48

99.54

98.43

98.88

MACRO-F1BCA

100.00

37.10

85.18

53.26

60.45

84.40

73.79

99.08

85.69

89.88

68.87

87.71

99.49

94.17

96.11

44.11

97.48

99.59

98.60

99.00

AMAZON-670K

WEIGHTED-TOP-K

100.00

25.99

17.51

14.64

15.41

93.39

60.62

40.14

35.43

36.71

87.65

87.65

49.91

47.71

48.41

51.70

100.00

51.81

51.81

51.81

MACRO-PGREED

100.00

25.99

21.37

12.72

14.61

93.39

60.62

43.83

34.21

37.10

87.65

87.65

50.45

47.56

48.63

51.70

100.00

51.81

51.81

51.81

MACRO-PBCA

100.00

25.99

21.84

12.96

14.88

93.39

60.62

46.14

35.91

38.92

87.65

87.65

51.13

48.27

49.33

51.70

100.00

51.81

51.81

51.81

MACRO-RPRIOR

100.00

25.99

17.51

14.64

15.41

93.39

60.62

40.14

35.43

36.71

87.65

87.65

49.91

47.71

48.41

51.70

100.00

51.81

51.81

51.81

MACRO-RBCA

100.00

25.99

19.58

16.63

17.43

93.39

60.62

43.45

37.54

39.18

87.65

87.65

50.88

48.44

49.23

51.70

100.00

51.81

51.81

51.81

MACRO-F1GREED

100.00

25.99

21.15

14.45

16.03

93.39

60.62

44.10

35.16

37.71

87.65

87.65

50.46

47.65

48.67

51.70

100.00

51.81

51.81

51.81

MACRO-F1BCA

100.00

25.99

21.41

16.50

17.73

93.39

60.62

46.17

37.24

39.70

87.65

87.65

51.14

48.38

49.36

51.70

100.00

51.81

51.81

51.81

Another factor that impacts the running time is stopping criterion ϵ , which indicates minimal expected utility gain to continue BCA algorithm. Here we test and report results for 3 values of ϵ ∈ { 10 -3 , 10 -5 , 10 -7 } on AMAZONCAT-13K, WIKIPEDIALARGE-500K, and AMAZON-670K datasets that are the largest in terms of number of samples and labels.

The results of BCA methods with different values of ϵ and k ′ are presented in Table 9. In this table, we additionally report the number of iterations (number of passes over datasets performed by the BCA algorithm) and time in seconds. The numbers are the mean results over 5 runs with different seeds. For all values of ϵ and k ′ , the results are very similar. In the case of WIKIPEDIALARGE-500K dataset, k ′ = 1000 achieves slightly better results than k ′ = 100 in a few cases. In the case of all datasets, the smallest ϵ = 10 -7 is usually the best. However, even the largest tested ϵ = 10 -3 that is greater than the inverse of a number of labels and samples is only slightly worse at the same time, requiring a much smaller number of iterations. For all values of ϵ and k ′ , the BCA algorithm terminates in just a few iterations, finishing always in less than 20 minutes. Please note that we implemented our algorithms in Python with some parts optimized using Numba [24] - LLVM-based just-in-time (JIT) compiler for Python. Because of that, we believe that the running time can be further reduced by using a more performant programming language.

E.5 Results with mixed utilities

As we already discussed in Section 7, the optimization of macro-measures comes with the cost of a significant drop in performance on instance-wise measures, which in some cases may not be acceptable. To achieve the desired trade-off between tail and head label performance, one can optimize

25

30

35

40

Macro-P@3

40

50

60

70

80

Instance-P@3

PS-K

Log-K

Macro-P

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Eurlex-4K

18

20

22

24

26

28

Macro-F1@3

60

.

0

62

.

5

65

.

0

67

.

5

70

.

0

72

.

5

75

.

0

Instance-P@3

PS-K

Log-K

Macro-F1

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Eurlex-4K

30

35

40

45

20

30

40

50

60

70

80

Instance-P@3

PS-K

Log-K

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Eurlex-4K

20

30

40

50

60

70

Macro-P@3

50

60

70

80

90

Instance-P@3

PS-K

Log-K

Macro-P

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

AmazonCat-13K

10

20

30

40

50

Macro-F1@3

65

70

75

80

85

Instance-P@3

PS-K

Log-K

Macro-F1

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

AmazonCat-13K

30

40

50

60

70

80

90

0

20

40

60

80

100

Instance-P@3

PS-K

Log-K

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

AmazonCat-13K

2

4

6

8

10

Macro-P@3

30

40

50

60

70

80

Instance-P@3

PS-K

Log-K

Macro-P

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Wiki10-31K

1

2

3

4

Macro-F1@3

30

40

50

60

70

80

Instance-P@3

PS-K

Log-K

Macro-F1

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Wiki10-31K

2

4

6

8

10

12

20

30

40

50

60

70

80

Instance-P@3

PS-K

Log-K

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Wiki10-31K

20

25

30

35

40

Macro-P@3

20

30

40

50

60

Instance-P@3

PS-K

Log-K

Macro-P

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Wikipedia-500K

16

18

20

22

24

26

Macro-F1@3

42

.

5

45

.

0

47

.

5

50

.

0

52

.

5

55

.

0

57

.

5

Instance-P@3

PS-K

Log-K

Macro-F1

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Wikipedia-500K

30

35

40

45

50

30

40

50

60

Instance-P@3

PS-K

Log-K

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Wikipedia-500K

10

12

14

16

18

Macro-P@3

27

.

5

30

.

0

32

.

5

35

.

0

37

.

5

40

.

0

42

.

5

Instance-P@3

PS-K

Log-K

Macro-P

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Amazon-670K

10

11

12

Macro-F1@3

36

37

38

39

40

41

42

Instance-P@3

PS-K

Log-K

Macro-F1

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Amazon-670K

14

15

16

17

18

34

36

38

40

42

Instance-P@3

PS-K

Log-K

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Amazon-670K

27

.

5

30

.

0

32

.

5

35

.

0

37

.

5

40

.

0

Macro-P@5

30

40

50

60

70

Instance-P@5

PS-K

Log-K

Macro-P

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Eurlex-4K

25

26

27

28

29

30

Macro-F1@5

56

58

60

62

Instance-P@5

PS-K

Log-K

Macro-F1

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Eurlex-4K

40

42

44

46

48

50

Coverage@5

10

20

30

40

50

60

70

Instance-P@5

PS-K

Log-K

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Eurlex-4K

30

40

50

60

70

Macro-P@5

30

40

50

60

70

Instance-P@5

PS-K

Log-K

Macro-P

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

AmazonCat-13K

30

35

40

45

50

Macro-F1@5

55

.

0

57

.

5

60

.

0

62

.

5

65

.

0

67

.

5

70

.

0

Instance-P@5

PS-K

Log-K

Macro-F1

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

AmazonCat-13K

50

60

70

80

Coverage@5

0

20

40

60

80

Instance-P@5

PS-K

Log-K

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

AmazonCat-13K

2

4

6

8

10

Macro-P@5

30

40

50

60

70

Instance-P@5

PS-K

Log-K

Macro-P

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Wiki10-31K

1

2

3

4

5

Macro-F1@5

40

50

60

70

Instance-P@5

PS-K

Log-K

Macro-F1

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Wiki10-31K

4

6

8

10

12

14

Coverage@5

10

20

30

40

50

60

70

80

Instance-P@5

PS-K

Log-K

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Wiki10-31K

20

25

30

35

40

Macro-P@5

10

20

30

40

Instance-P@5

PS-K

Log-K

Macro-P

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Wikipedia-500K

20

22

24

26

28

Macro-F1@5

32

34

36

38

40

42

44

Instance-P@5

PS-K

Log-K

Macro-F1

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Wikipedia-500K

40

45

50

55

Coverage@5

15

20

25

30

35

40

45

Instance-P@5

PS-K

Log-K

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Wikipedia-500K

14

16

18

20

22

Macro-P@5

20

25

30

35

40

Instance-P@5

PS-K

Log-K

Macro-P

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Amazon-670K

13

.

5

14

.

0

14

.

5

15

.

0

15

.

5

16

.

0

16

.

5

Macro-F1@5

30

32

34

36

38

Instance-P@5

PS-K

Log-K

Macro-F1

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Amazon-670K

20

.

5

21

.

0

21

.

5

22

.

0

22

.

5

23

.

0

Coverage@5

30

32

34

36

38

Instance-P@5

PS-K

Log-K

BCA

α

= 0

.

9

α

= 0

.

7

α

= 0

.

5

α

= 0

.

3

α

= 0

.

1

Top-K

Pow-K

β

=0

.

25

Pow-K

β

=0

.

5

Amazon-670K

a mixed utility that is a linear combination of instance-wise measures and selected macro-measures. Here, we present the results for three such mixed utilities (combinations of instance-precision with macro-precision, macro-f1-measure, and coverage):

In Figure 2 and Figure 3, we present the plots with results on two combined measures for different values of α ∈ { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 , 0 . 95 , 0 . 99 , 0 . 995 , 0 . 999 } . Once again, the presented results are the mean values over 5 runs with different seeds. The plots show that the instance-vs-macro curve has a nice concave shape that dominates simple baselines. In particular, we can initially improve macro-measures significantly with only a minor drop in instance-measures, and only if we want to optimize even more strongly for macro-measures, we get larger drops in instance-wise measures. A particularly notable feature of the plug-in approach is that the curves in the figure are cheap to produce since there is no requirement for expensive re-training of the entire architecture, so one can easily select an optimal interpolation constant according to some criteria, such as a maximum decrease of instance-wise performance.

E.6 Hardware

The LIGHTXML model was trained on a workstation with a single Nvidia Tesla V100 GPU with 32 GB of memory and 64 GB of RAM. All the inference strategies were then run on the workstation with 64 GB of RAM. However, to run them, one requires only 16 GB of memory.

F Efficient inference with Probabilistic Label Trees

In this section, we describe probabilistic labels trees (PLTs) [16], which allow for efficient retrieval of only the topk labels with the highest conditional probabilities. Then, we introduce a weighted version and show how to modify it to achieve efficient and exact optimal prediction for any g j ( η j ( x )) that is monotonous in η j ( x ) . Thus, if the LPE η is modeled by a PLT, this provides an alternative to the sparse inference introduced in Appendix D that also scales to XMLC problems.

F.1 Probabilistic labels trees

We denote a tree by T , the set of all its nodes by V T , the root node by r T , and the set of its leaves by L T . The leaf l j ∈ L T corresponds to the label j ∈ [ m ] . The parent node of v is denoted by pa( v ) , and the set of child nodes by ch( v ) . The set of leaves of a (sub)tree rooted in node v is denoted by L v , and path from node v to the root by path( v ) .

A PLT uses a tree T to factorize conditional probabilities of labels, η j ( x ) = P [ y j = 1 | x ] , j ∈ [ m ] , by using the chain rule. Let us define an event that L v contains at least one relevant label in y , denoting z v := 1 [ ∃ j : l j ∈ L v ∧ y j = 1] . Now for every node v ∈ V T , the conditional probability of containing at least one relevant label is given by:

where η ( x , v ) := P [ z v = 1 ∣ ∣ z pa( v ) = 1 , x ] for non-root nodes, and η ( x , v ) := P [ z v = 1 | x ] for the root. Notice that (76) can also be stated recursively as

and that for leaf nodes we get the conditional probabilities of labels

To obtain a PLT, it suffices, for a given T , to train probabilistic classifiers estimating η ( x , v ) for all v ∈ V T . Analogously to the main paper, we denote estimates of η by ˆ η .

F.2 Weighted PLTs

[46] introduced an A ∗ -search based algorithm for efficiently finding the k leaves with the highest gain g ( l j , x ) = w j η j ( x ) , where w j ∈ [0 , ∞ ) is the weight given to label j . The outline of the search method presented below is an adapted description from [46].

For the gain function g ( l j , x ) = w j ˆ η j ( x ) , the procedure uses a cost function c ( l j , x ) for each path from the root to a leaf. Notice that the following holds:

v ∈ path( l j ) Because exp is monotonous, we can define the following cost function for a selected label j , which the algorithm will aim to minimize:

We can then guide the A ∗ -search with the function c ( v , x ) = p ( v , x ) + h ( v , x ) , estimating the value of the optimal path in the subtree of node v , where

is the cost of reaching tree node v from the root, and

is a heuristic function estimating the cost of reaching the best leaf from node v . The A ∗ -search in our procedure evaluates nodes in ascending order of their estimated cost values c ( l j , x ) .

This approach has been proven by [46] to guarantee that A ∗ -search finds the optimal solution-topk labels with the highest c ( l j , x ) and thereby topk labels with the highest w j η j ( x ) -in the optimally efficient way, i.e., there is no other algorithm used with this heuristic that expands fewer nodes [38].

F.3 PLTs for monotonous gain functions

Notice that in Weighted PLTs instead of minimizing cost, one can directly optimize gain by using a tree-search procedure that evaluates nodes in descending order of their estimated gain g ( v , x ) = p ( v , x ) · h ( v , x ) , where p ( v , x ) = ∑ v ′ ∈ path( v ) ˆ η ( x , v ′ ) and h ( v , x ) = max j ∈L v w j . To generalize this formulation and apply it to the proposed algorithms, we need to be able to find the topk with any gain function g j ( η j ( x )) that is monotonous with η j ( x ) .

We can guide the tree search using the estimated gain function g ( v , x ) = h ( v , p ( v , x )) , where

the same as in the case of Weighted PLT, it simply corresponds to the conditional probability of finding at least one positive label in the subtree of node v , and the heuristic part is:

As in the case of Weighted PLT, we would like to guarantee that this search procedure finds the optimal solution-the topk labels with the highest g ( l j , x ) . To do that, we need to ensure that g ( v , x ) is admissible, i.e., in case of maximization, it never underestimates the gain from reaching a leaf node [38]. We also would like g ( v , x ) to be consistent, making the proposed tree search procedure optimally efficient, i.e., there is no other algorithm used with the same g ( v , x ) that expands fewer nodes [38]. Algorithm 9 outlines this procedure for finding the topk labels with highest values of g j ( η j ( x )) that is monotonous with η j ( x ) , and that can replace the select top-k ( g ) operation in the proposed BCA and Greedy algorithms. Unfortunately, not all gain functions g j ( η j ( x )) can be calculated efficiently. To calculate the semi-empirical quantity ˜ p exactly, one needs to know all the η j ( x ) . To obtain them, the whole tree need to be evaluated, which prevents this method from being efficient. Because of that, we use this technique to get exact predictions for the measures that are defined solely on ˜ t and ˜ q , like macro-precision and coverage.

F.4 Experimental comparison of inference with PLTs

In this experiment, we evaluate the inference using probabilistic label trees that can efficiently perform the exact select top-k ( g ) operation. We compare inference times of these variants of the methods that require only one pass over the dataset: TOP-K, PS-K, POW-K, LOG-K, MACRO-PGREED, MACRORPRIOR , and COVGREED, for k = { 3 , 5 } . As a baseline, we use the same PLT to obtain the k ′ = 100 and k ′ = 1000 highest η ( x i ) , and then run the sparse algorithms on top of these predictions. We use a modified implementation of PS-PLT [46, 17], trained with default settings.

We present the results in Table 10. As in the case of the main experiment, we again observe that the specialized inference strategies are indeed the best on the measure they aim to optimize. Compared to the cost of obtaining marginals for k ′ = 100 and k ′ = 1000 labels, which can be later used with one of the proposed inference algorithms (the cost of running this step is not included in the table), the greedy methods combined with PLT-based search consistently achieve speed-up when compared to

top k ′ = 1000 inference. Unfortunately, they are also often slower than k ′ = 100 inference. As we showed in the previous section, k ′ = 100 is usually enough to get good predictive performance on macro measures using BCA and Greedy algorithms. However, here, the PLT-based methods obtain the exact solution for finding the prediction with the highest expected gain. While it is worth noting that tree structure and the type of node estimator have an impact on the performance of PLT-based algorithms and we conducted this experiment with just a single setup, we believe that this experiment shows that PLT-based inference, in some cases, can be an alternative to inference with sparse marginal estimates.

Peer reviews:
comments: SUMMARY:
This paper focuses on the long-tail problem in extreme multi-label classification. To address this problem,  the authors propose a new method to optimize performance metrics for extreme multi-label tasks via the expected test utility (ETU) framework. Experimental and theoretical results are also provided.

STRENGTHS:
1. The long-tail problem in the extreme multi-label classification task is important. It is good to explore how to fairly evaluate the performance of multi-label learning algorithms on tail labels.
2. The proposal is technically sound. Directly optimizing performance measures for tail labels is make sense.

WEAKNESSES:
The main concern about the paper is the experiments. There are also some other novel algorithms that are proposed for the long-tail problem in multi-label classification tasks and can achieve good performance, but the authors did not compare with these studies in the experiments.

Moreover, there are some minor issues, such as the related works should be discussed.

QUESTIONS:
1. There are also some other works that focus on the long-tailed multi-label classification tasks, instead of directly optimizing the performance metrics, how about the performance of these works?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
This paper analyzes generalized metrics budgeted “at k” by formulating it in the expected test utility (ETU) framework. They derive optimal prediction rules and construct their computationally efficient approximations with provable regret guarantees and being robust against model misspecification.

STRENGTHS:
This paper is sound and clearly written.

WEAKNESSES:
Limited contribution:

The paper investigates optimal solutions for the class of utility functions that can be linearly decomposed over labels into binary utilities. Is it possible to extend this method to a more broad class of utility functions.

The algorithm block coordinate ascent looks not new to me.

QUESTIONS:
Table 1 shows that the macro measures are suitable for long tails experimentally. Would you please explain the reason for it in theory?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors studies the evaluation metrics for the long tailed extreme multilabel classification problems. Compared with existing heuristics, such as PSP, they formulate the metrics in the expected test utility framework. Inference rules are derived to obtain optimal metrics. Approximations are construct for the metrics that are difficult to compute.\n\nSTRENGTHS:\nThis paper picks an interesting but not well studied direction and gives thorough study with reliable derivations and proofs. It is well known that the existing metrics for XMC problems cannot measure the performance on tail labels because of the data distribution. I agree with the authors that the widely used proensity scores are heuristics that are not to be used as metrics, especially it requires hyper-parameters which would introduce much consistency issues when used for comparison. The authors express the metrics in the ETU framework thus unifying the existing and potential metrics.\n\nThe paper gives detailed derivations for the optimal prediction rules for each of the derived metrics and provides cheap and easily applicable approximations for metrics that are hard to compute. Empirical results on 4 public XMC benchmark datasets indicates that the optimal rules can actually obtain the best score for the corresponding metrics.\n\nWEAKNESSES:\nThe paper is overall well presented but some of the terms are used without introducing/explaining, such as \"macro-average\".\n\nMinor:\n* Line 82: $C_{11}$ to true positives\n\nQUESTIONS:\nInference with a global budget k is easy to implement but not always optimal in terms of utility-budget trade-off. If there's no top-k constraint, how can we design the metrics/inference rules to determine the number of predictions for each instance?\n\nLIMITATIONS:\nIt would be great to see the complexity analysis of the optimal prediction rules to understand the inference overhead.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper critiques that the existing extreme classification evaluation metrics don't give the complete picture with respect to all labels (more specifically performance on head labels overpowers performance on tail labels), hence it recommends using macro-averaged metrics which are more favorable to tail labels. The authors use expected test utility framework to come up with BCA based methods to get modified prediction rules given a prediction matrix from a trained XMC model. Experiments on benchmark datasets indicate that just taking the top-k predictions is not ideal for marco averaged metrics and the proposed approach performs significantly better.\n\nSTRENGTHS:\n- The proposed approach works as a plugin method that can be used with any existing XMC method without any need for retraining\n- Proposed approach is well reasoned\n- Approximate versions scalable to large datasets without the need for specialized hardware\n\nWEAKNESSES:\n- Not sure how important are macro measures for evaluating performance on XMC datasets as the data imbalance is inherent to XMC applications and in most scenarios we do want to mimic this imbalance in model predictions as well (for e.g. in ad recommendation problem with a billion ads in corpus, certain products are popular, so giving equal weightage to each product during evaluation might not be very useful). I agree that we do want to improve quality of predictions on tail labels as well but not at the cost of making worse overall predictions. Results indicate that trying to do well on macro-averaged metrics heavily impacts the performance on standard metrics which is not desirable.\n- Section 4 is a bit hard to follow\n\nMinor typo:\nline 82, \"c11 to true negatives.\" -> \"c11 to true positives.\"\n\nQUESTIONS:\nis it possible to optimize for a weighted combination of standard precision and macro precision, so that we can smoothly interpolate between these two metrics?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses evaluation metrics to measure the long-tail labels' performance of the extreme multi-label classification (XMLC) problems. The author proposed that macro-average based metrics (e.g., macro-avg Precision/Recall/F1 at k) are more suitable to measure the performance of long-tail labels compared to conventional instance-wised precision/recall at k. The author further discuss how to optimize these metrics on a given test set, with theoretical justification on the objective functions and deviation bounds. The empirical results verified the effectiveness of their proposed optimization methods for the given target metric.\n\nSTRENGTHS:\n- Leverage the ETU framework to approximately optimize Macro-average metrics seems novel\n- Theoretical justifications sounds reasonable and technical derivations are solid\n\nWEAKNESSES:\n- Selecting thresholds to optimize Marco-average metrics are not new in multi-label community. The author seems not aware of some  classical/heuristic approaches. For example, the sCut and SCutFBR methods studied in [1,2,3]. The author should also compare these methods in the experiment sections.\n\n**Reference**\n- [1] Y. Yang. A Study on Thresholding Strategies for Text Categorization. SIGIR 2001\n- [2] Lewis et al. RCV1: A New Benchmark Collection for Text Categorization Research. JMLR 2004\n- [3] Fan et al. A Study on Threshold Selection for Multi-label Classification. Technical Report 2007\n\nQUESTIONS:\n**Major Questions**\n- Q1: How does the proposed method relate and connect to other conventional threshold tuning approaches (e.g., [1,2,3])? \n- Q2: How does the proposed method compare empirically to other conventional threshold tuning approaches (e.g., [1,2,3])?\n- Q3: For the block coordinate ascent algorithm, how sensitive is it to the initial point? If it is a non convex function, I suppose the initialization problem matters? Any better heuristic than \"predicting k random labels\" that described at line 185?\n- Q4: For Line 195-198, monotonicity alone is not enough to ensure convergence [4]. Any more thorough/technical statement for the convergence analysis of the proposed block coordinate ascent algorithm?\n\n**Minor issues**\n- In Eq(5), the first summation has index $i$ in the subscript, but not index $i$ seems not being used in the function of $u_w()$. \n- At Line 154: \"the proof is given in Appendix F\". However, it seems to me that the proof appeared in Appendix G (Theorem G.4)?\n\n**Reference**\n- [1] Y. Yang. A Study on Thresholding Strategies for Text Categorization. SIGIR 2001\n- [2] Lewis et al. RCV1: A New Benchmark Collection for Text Categorization Research. JMLR 2004\n- [3] Fan et al. A Study on Threshold Selection for Multi-label Classification. Technical Report 2007\n- [4] https://web.eecs.umich.edu/~fessler/course/600/l/lmono.pdf\n\nLIMITATIONS:\nTo my knowledge, there's no potential negative societal impact for this submission.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}