Consistent algorithms for multi-label classification with macro-at-$k$ metrics
Authors: Erik Schultheis
Wojciech Kotlowski
Marek Wydmuch
Rohit Babbar
Strom Borman
Krzysztof Dembczynski
Venue: ICLR 2024 poster (2024)
Keywords: multi-label classification
complex performance metrics
macro-at-k
extreme classification
Primary area: general machine learning (i.e., none of the above)

Abstract: We consider the optimization of complex performance metrics in multi-label classification under the population utility framework. We mainly focus on metrics linearly decomposable into a sum of binary classification utilities applied separately to each label with an additional requirement of exactly $k$ labels predicted for each instance. These "macro-at-$k$" metrics possess desired properties for extreme classification problems with long tail labels. Unfortunately, the at-$k$ constraint couples the otherwise independent binary classification tasks, leading to a much more challenging optimization problem than standard macro-averages. We provide a statistical framework to study this problem, prove the existence and the form of the optimal classifier, and propose a statistically consistent and practical learning algorithm based on the Frank-Wolfe method. Interestingly, our main results concern even more general metrics being non-linear functions of label-wise confusion matrices. Empirical results provide evidence for the competitive performance of the proposed approach.

Published as a conference paper at ICLR 2024
CONSISTENT ALGRRIHSS FOR MULTI-LABEL CLASSI-
FICATION WITH MACRO-AT-K METRICS
Erik Schultheis
Wojciech Kotlowski
Aalto University
Pozman University of Technology
Helsinki, Finland
Poznan, Poland
erik. www.shhhhhiesaaaeeeee..
wkotlowse cesskiecs.ppp pwwnan...an..p.p.
Marek Wydmuch
Rohit Babbar
Poznan University of Technology
University of Bath / Aalto University Bath, UK Helsinki, Finland
Poznan, Poland
mwydmach@cs. cs htt.:mmcchcccspppttpoo.aaa. pwwnan.
6026088000 wtt.eeaaath........ uck
Strom Borman
Krzysztof Dembezyáski
Yahoo Research
Yahoo Research / Pozman University of Technology New York, USA Poznan, Poland
Champaign, USA
strom. rrzomboorani@yaoonnn..oo
krzysztof cenbooyynssdddyyhooiicccoooooo.
ABSTRACT
We consider the optimization of complex performance metrics in multi-label classification under the population utility framework. We mainly focus metrics linearly decomposable into sum of binary classification utilities applied separately each label with an additional requirement of exactly k labels predicted for each instance. These "macro-at-k" metrics possess desired properties for extreme classification problems with long tail labels. Unfortunately, the at-k constraint couples the otherwise independent binary classification tasks, leading to a much more challenging optimization problem than standard macro-averages. We provide a statistical framework to study this problem, prove the existence and the form of the optimal classifier, and propose a statistically consistent and practical learning algorithm based the Frank-Wolfe method. Interestingly, our main results concern even more general metrics being non-linear functions of label-wise confusion matrices. Empirical results provide evidence for the competitive performance of the proposed approach.
INTRODUCTION
Various real-world applications of machine learning require performance measures of complex structure, which, unlike misclassification error, do not decompose into an expectation over instance- wise quantities. Examples of such performance measures include the area under the ROC curve (AUC) (Drummond & Holte 22055), geometric mean (Drummond & Holte 20055 Wang & Yao 2012) Menon al.] 2013) Cao et al. 2019), the f'-masuree (Lewis) 1995) or precision the top (Kar et 2015). The theoretical analysis of such measures, as well as the design of consistent and efficient algorithms for them, non-trivial task.
In multi-label classification, one can consider wide spectrum of measures that are usually divided into three categories based on the averaging scheme, namely instance-wise, micro, and macro averaging. Instance-wise measures are defined, as the name suggests, on the level of a single instance. Typical examples are Hamming loss, precision@k, recall@A, and the instance-wise F'-measure. Micro-averages are defined on confusion matrix that accumulates true positives, false positives, false negative, and true negatives from all the labels. Macro-are avero-averaggs require a binary metric to be applied to each label separately and then averaged over the labels. In general, any binary metric can be applied in any of the above averaging schemes. Not surprisingly, some of the metrics, for example misclassification error, lead the same form of the final metric regardless of the scheme

---

Published as a conference paper at ICLR 2024
used. One can also consider the wider class of measures that are defined as general aggregation functions of label-wise confusion matrices. This includes the measures described above, but also, e.g., the geometric mean of label-wise metrics specific variant of the F-measure (Opitz & Burst 2021] being a harmonic mean of macro-precision and macro-recall.
In this paper, target the setting of prediction with budget. Specifically, require the predictions to be "budgeted-at-hk." meaning that for each instance, exactly & labels need to be predicted. The budget of & requires the prediction algorithm to choose the labels "wisely". It is also important in many real-world scenarios. For instance, in recommendation systems or extreme classification, there is fixed number of slots (e.g., indicated by a user interface) required to be filled with related products/searche//ae
(Cremonesi et al. 2010) Chang et al.] (02T). Furthermore, having a fixed prediction budget also interesting from a methodological perspective, as various metrics which lead to degenerate solutions without budget, e.g., predict nothing imacro-precision or everything (macro-recall), become meaningful when restricted to predict i labels per instance.
While all our theoretical results and algorithms apply to general class of multi-label measures, we focus in this paper on macro-averaged metrics. If no additional requirements are imposed on the classi- fier, the linear nature of the macro-averaging means that a binary problem for each label can be solved independently, and existing techniques (Koyejo et al. 2015 Kottowski & Dembezyński 2017) are suf- ficient. In turn, if we require predictions to be atteeted-at-k, the task becomes much more difficult, as this constraint tightly couples the different binary problems together. In general, they cannot be solved independently for each label, requiring instead more involved techniques to find the optimal classifier.
The macro-at-k metrics seem to be very attractive in the context of multi-label classification. Macro- averaging treats all the labels equally important. This prevents ignoring labels with small number of positive examples (Schultheis et al. 2022), so-called tail labels, which are very common in applications of multi-label classification, particularly in the extreme setting when the number of all labels is very large (Jain et al. 2016 Babbar & Schölkopf] 2019). Furthermore, it can be shown that one can remove tail labels from the training set with almost no drop of performance in terms of popular metrics, such as precision @ k and nDCG@k, k, on extreme multi-label data sets (Wei & Li 2019, Schultheis et al., 2023). The macro-at-k metrics, on the other hand, are sensitive to the lack of tail labels in the training set]
We aim at delivering consistent algorithms for macro-at-k metrics, i.e., algorithms that converge in the limit of infinite training data to the optimal classifier for the metrics. Our main theoretical results are stated in a very general form, concerning the large class of aggregation functions of label-wise confusion matrices. Our starting point of the analysis results obtained in the multi-class setting (Narasimhan et al. 2015 2022), concerning consistent algorithms for complex performance measures with additional constraints. Nevertheless, they do not consider budgeted-at-k predictions, which do not apply to multi-class classification, while they play an important role in the multi-labe setting. Furthermore, using arguments from functional analysis, we managed significantly simplify the line of reasoning in the proofs. We first show that the problem can be transformed from optimizing over classifiers to optimizing over the set of feasible confusion matrices, and that the optimal classifier optimizes an unknown linear confusion-mateee matrix metric. In the multi-label setting, interestingly. such a classifier corresponds to a prediction rule, which has the appealingly simple form: selecting the l highest-scoring labels based on an affine transformation of the marginal label probabilities. Combining this result with the optimization of confusion matrices, we state a Frank-Wolfe based algorithm that is consistent for finding the optimal classifier also for nonlinear metrics. Empirical studies provide evidence that the proposed approach can be applied in practical settings and obtains competitive performance in terms of the macro-at-k metries.
2 RELATED WORK
The problem of optimizing complex performance metrics is well-known, with many articles pub- lished for a variety of metrics and different classification problems. It has been considered for
binary (Ye et al. 2012; Koyejo et al. 2014 Busa-Fekete et al. 2015 Dembezynski et al.) (017).
multi-class (Narasimhan et al. 2015 (022). multi-label (Waegeman et al. 2014 Koyejo et al. 2015 Kotlowski & Dembczyński 2017), and multi-outpu (Wang et al. 2019 c classificatin..
Results and description of such an experiment are given in Appendix
2

---

Published as a conference paper at ICLR 2024
Initially, the main focus was on designing algorithms, without conscious emphasis statistical consequences of choosing models and their asymptotic behavior. Notable examples of such con- tributions are the SVMperf algorithm (Joachims 2005). approaches suited to different types of the F-measure (Dembezynski et al.) 2011; Natarajan et al.) 2016 Jasinska et al., 2016), or precision at the top (Kar et al. 2015). Wide use of such complex metrics has caused an increasing interest in investigating their theoretical properties, which can then serve as guide to design practical algorithms.
The consistency of learning algorithms is a well-established problem. The seminal work of Bartlet et al. (2006) was studying this problem for binary classification under the misclassification error. Since then a wide spectrum of learning problems and performance metrics has been analyzed in
terms of consistency. These results concern ranking (Duchi et al., 2010, Ravikumar et al. 201
Calauzenes et al. 2012) Yang & Koyejo, 2020), multi-class (Zhang, 2004 Tewari & Bartlett 2007
and multi-label classification (Koyejo et al. 2015) | Kotlowski & Dembezynskij
classification
with abstention (Yuan & Wegkamp/ [010 Ramaswamy al. [2018] 8), or constrained classification
problems (Agarwal et al.] 2018: Kearns et al. 2018: Narasimhan et al. 2022). Nevertheless, the
problem of designing consistent algorithms for budgeted-ate at-k macro averages is relatively new
Optimizing non-decomposable metrics can be considered in two distinct frameworks (Dembczynski et al. 2017): : population utility (PU) and expected test utility (ETU). The PU framework focuses on estimation, in the sense that a consistent PU classifier is one which correctly estimates the population optimal utility the size of the training set increases. A consistent ETU classifier is one which optimizes the expected prediction error over a given test set. The latter might get better results, as the optimization performed on the test set directly. Optimization of budgeted-at-k metries in this framework has been recently considered in Schultheis et al. (2023). The former framework, which we focus on in this paper, has the advantage that prediction can be made for each test example separately. without knowing the entire test set in advance.
3 PROBLEM STATEMENT
Let æ X' denote an input instance, and y E 10.1)m the vector indicating the relevant labels.
jointly distributed according to (z, y) ~ P. Let h: X' + 10 be randomized multi-label classifter which, given instance æ, predicts a possibly randomized class label vector y E {0, 13 such that Egiz D h(x)). We assume that the predictions are budgeted at ki that is exactly k labels are always predicted relevant, which means that || -E
m Dy k with probability turns out that this equivalent to assuming [1/(()) |l ΣE (x) k for all x E X. Indeed, [|/(())]1 = k is necessary, because & = |g = [h(x)] |I but it also suffices as for any real-valued vector π E [0, 1] with ||T||| ks one can construct distribution over binary vectors D E {0. 13ThM with | 1 = k and marginals Ey]0] = π.; this can be accomplished using, e.g., Madow's sampling scheme (see Appendix A for the actual efficient algorithm). Thus, using notation A = (v E [0, m= || = ). the randomized classifiers budgeted at k are then all (measurable) functions of the form h: X + A m We denote the set of such functions as H.
For any E X, let (=x) == Ey [y] denote the vector of conditional label marginals. Given a random-
ized classifier h H. we define its malti-label confusion tensor C(h) (C' (h1)........ C'T (fm)) ))
as a sequence of m binary classification confusion matrices associated with each label j e [m]. that is Ch,,(hh = Plys u. Øj = u) for u, U. E {0, Note that using the marginals and the definition of the randomized classifier,
(h₂,) = Ex ny Ex i ()) h=))
7) (x) )1 h,
Ex ny (x) (1))
Ez In (z) M, (x)
The set of all possible binary confusion matrices is written as = (C E [0, 1 1|a=2 ||C||, 1,, = 1), and is used to define the set of possible confusion tensors for predictions at k through Ck = Vj E [m] : C CC, ΣØ Ca1 + +4, = k}
{C E [o, 1 x
In this work, we are interested in optimizing performance metrics that do not decompose over individ- ual instances, but are general functions of the confusion tensor of the classifier h. While in general, given two confusion matrices, specific application, it is possible to impose a partial order that any reasonable performance metric should respect. To that end, define:
cannot say which one is better than the other without knowing the
3

---

Published as a conference paper at ICLR 2024
Table I: Examples of binary confusion matrix measures, which be used building blocks for
confusion tensor measures. For clarity, we denote tn = Coo, fp = CoL, fn C1., to.tp-- = C11
Metric
(C)
Metric
/ (C)
Accuracy
tp + tn
Recall
tp
fp-fn fn
Precision
Balanced accuracy
tp+lp
2 2(t++ fa)
2(tu+(pp)
Fg
11+ (T+9 t
G-Mean
tp-ta
V (sp (pp
Jaccard
tp 1p+fp+fn
AUC
2 2-tp tettnnt---fe fp fn-tn
(tp+tn)((p+t))
Definition 3.1 (Binary Confusion Matrix Measure). Let C {CE
[0, 1|2x2
||C||1, = =1} be the
of all possible binary confusion matrices, and C, C' C. Then we say that C' is at least as good as C, C' 2 C, if there exists constants c1, 62 such that
C' (C-(Cro
Coo + éj
Co Co1-61 -6 Cn + 62)
C1o 62
i.e., C' can be generated from by turning some false positives to true negutives and false negatives to true positives. A function :h: C - [0, 1] is called a binary confusion matrix measure (Singh & Khim 2022 if it respects that ordering, i.e., for C' 2 C we have (CC)) 2 (CC).
Similarly, in the multi-label case we cannot compare arbitrary confusion tensors, where one is better on some labels than on others DetrssBuburr we can recognize if one is better on all labels:
Definition 3.2 (Confusion Tensor Measure). For given number of labels m E N, and two confusion tensors C, C' C. we say that C' is least as good as C, C' C, if for all labels j E [m] holds that C = C³. A function Vr: c. - [0. 1] is called a confusion tensor measure if respects this ordering, i.e., for C' > C have v (CC) (C).
Of particular interest in this paper are functions which linearly decompose over the labels, that is macro-averaged multi-label metrics (Manning et al. 2008 Parambath al., 2014 Koyejo et al.
2013 Kotlowski & Dembezyński, 2017) of the form:
(h) - (CC(h) = -1 Σ (()))))
where v is some binary confusion matrix measure. If one takes a binary confusion matrix measure (e.g., any of those define in Table I), then the resulting macro-average will be a valid confusion tensor measure. A more thorough discussion of these conditions can be found in (Appendix H
Macro-averaged metrics find numerous applications in multi-label classification, mainly due their balanced emphasis across labels independent of their frequencies, and thus can potentially alleviate the "long-tail" issues problems with many rare labels (Schultheis et al. 2022).
Denote the optimal value of the metric among all classifiers budgeted at k as:
V == sup (h).
hEW
and let h* E argmax, (h) be an optimal (Bayes) classifier for which (h+) VV, if exists. For any classifier h. define its V-regret as AV (h) = -ト+ (h) to measure the suboptimality of h with respect to W: from the definition, A (h) 2 0 for every classifier h, and AV (h) 0 if and only if h optimal. If the -rerrett of learning algorithm converges to zero with the sample size tending to infinity, it is called (statistically) consistent. We consider such algorithms in Section 53 Even though the objective (3) decomposes onto m binary problems, these are still coupled by the budget constraint, [[((2))]1 is for all X' and cannot be optimized independently
show later in the paper.
This specifically the trade-off want achieve for tail labels!
4

---

Published as a conference paper at ICLR 2024
4 THE OPTIMAL CLASSIFIER
Finding the form of the optimal classifier for general maero-averaged performance metrics is difficult. For instance, when (CC) is the F3 measure, the objective to be optimized is a sum of linear fractional functions, which is known to be NP-hard in general (Schaible & Shi, 2003). We are, however, able to fully determine the optimal classifier for the specific class lineur utilities, which are metrics depending linearly on the confusion tensor of the classifier. Furthermore, we also show that for general class of macro-ar -veeraed metrics, under mild assumptions on the data distribution, the optimal classifier exists and turns out to also be the maximizer of some linear utility, whose coefficients, however, depend on its (unknown a priori) confusion tensor.
We start with metric of the form F (C) G G-C Σj= G C for some vector of gain matrices (gain tensor) G (G' ,..,,..CC G ), possibly depending on the data distribution P. We call such a utility linear as it linearly depends on the confusion matrices of the classifier. Note that we allow the gain matrix G to be different for each label, making this more general than linear macro-averages We need to consider this more general case, because it will appear as subproblem when finding optimal predictions for non- non-linear macro-averages as presented below.
Linear metrics are decomposable over instances, and thus the optimal classifier has an appealingly simple form: It boils down to simply sorting the labels by an affine function of the marginals, and returning the top & elements.
Theorem 4.1. The optimal classifier ht == argmaX 4 (h) for (h) G C(h) is given by
h" h²(x) top (a C m((x) b).
(5)
where: denotes the coordinate-wise product of vectors, while the vectors and b given by: == + Gi - -
b-= = Go Gooo-
and (v) returns a k-hot vector extracting the top k largest entries of v (ties broken arbitrarily).
(sketch,, full proof in Appendix Bp. After simple algebraic manipulations, the objective can be written as (0) - (x) b1) (x) + Rt where jj and b, are as stated in the theorem, while R does not depend on the classifier. For each æ E X' the objective can thus be independently maximized by the choice of h(x) E A which maximizes Σ== (a,7) (x) by (,y)) But this is achieved by sorting a, (x) +b in descending order, and setting hy (z) = 1 for the top & coordinates. and hy (x) = 0 for the remaining coordinates (with ties broken arbitrarily).
Examples of binary metrics for which their macro averages can be written in the linear form include:
the accuracy (CC) Coo + Cn (which leads the Hamming utility after macro-averaging) with aj = 2, bj = 1, and thus for any æ E X, the optimal prediction h (xx) returns k labels with the largest marginals (3x);
the same prediction rule is obtained for the TP metric (C)) Coo (that leads to precision@k) with aj- = 1, by = 0;
the recall (CC = Pl CCn (macro-averaged to recall@k) has a, = P(y 7:(m)
Cn 1)-1.b, O, so that the optimal classifiers returns top labels sorted according P(
* the balanced accuracy (C(C)
Cu + 2T
Con of gives aj 20 + (w) 1 1- (w)
21 - by =
1
00 with the optimal prediction sorting labels according P(,,=I]
P-;;)
We now switch to general case, in which the base binary metrics are not necessarily decomposable over instances, and optimizing their macro averages with budgeted predictors is challenging task. We make the following mild assumptions on the data distribution and performance metric:
We use A B= Em. Aue to denote a dot product over matrices, and a concise notation A B Σ, A'. B² for 'dot product' over matrix sequences A (A) ,..., A An) and B (B' ...... B").
5

---

Published as a conference paper at ICLR 2024
Assumption 4.2. The label conditional marginal vector n(z) Eyjo[[)] is absolutely continuous with respect to the Lebesgue measure on [0, 1m (i.e., has a density over [0, 1)
similar assumption was commonly used in the past works (Koyejo al. 2014 Narasimhan et al. 2015 Dembezynski et al. 2017). Assumption 4.3. The performance metric y/ is differentiable and fulfills for all labels j E [m]]
dc V ( ... ('......,C+ + ( (-1
-.)....cm C
0.
Assumption 4.3 is essentially 'strictly monotonic and differentiable' version of Definition 3.2 and satisfied by all averr-avereged metries given in Table
Our first main result concerns the form of the optimal classifier for general confusion tensor measures, of which macro-averaged binary confusion matrix measures are special cases. To state the result, we define Cp == (C(h) h E H), the set of confusion tensors achievable by randomized k-budgeted classifiers on distribution P. Clearly, maximizing (h) over E H. is equivalent to maximizing V (C) over C E Cp.
Theorem 4.4. Let the data distribution P and metric v satisfy Assumption 4.2 Assumption 4.3 respectively. Then, there exists an optimal C' E Cr, that is V ((C)) VV" Moreover, any classifier h. maximizing the linear utility G C(h) over h E H with G = (G) ".....,.'' Gm)) given by G CC'"((")), also maximizes v (h) over h E H.
Proof (sketch. full proof in Appendix We first prove that Cp is a compact set by using certain properties of continuous linear operators in Hilbert space. Due to continuity of V and the compactness of Cr, there exists maximizer C' argmax (C). By the first order optimality and convexity
of Cp. VV C* 2 (C*) C for all C E Cp, so C' maximizes a linear utility G- C with gain matrices given by G = (C*) A careful analysis under Assumption 4.2 shows that C* uniquely maximizes G C over Cx.
Theorem 4.4 reveals that v V-optimmll classifier exists and can be found by maximizing a linear utility, that is, by predicting the top k labels sorted according to an affine function of the label marginals: h" = (=t)= (a" @ ()x))= + b*) for vectors a* and b* defined for gain matrices G VV(C)) as in Theorem 4.1 Unfortunately, since C* is unknown in advance, the coefficients a* b* are also unknown, and the optimal classifier is not directly available. However, knowing that h" optimizes a linear utility indueed by the gradient of y leads to consistent algorithm described in the next section.
Although the optimal solution is expressed by affine functions of label marginals, in general, cannot be obtained by solving the problem independently for each label, i.e., the values of aj and b, may depend on labels other than j. Let h. (a:) and h² (a:) be optimal for distributions IP and P.. respectively. Let P' differ from P only on single label j. If we could solve the problem independently for each label, then h (xx) and (xx) would be the same up to label j. in the sense that the ordering relation between all other labels would not change. In Appendixx Elwe show that this is not the case, presenting a simple counterexample showing that a different distribution on a single label changes the solution with respect to the other labels.
5 CONSISTENT ALGRITHMS
As any algorithm we propose has to operate on a finite sample, we need to introduce empirical counter- parts for our quantities of interest. For example, we use ()(x)) to denote the estimate of (>x)) given by a label probability estimator trained on some set of training data S = ((x1, y1),... (x yn)) ). We also define the empirical multi-label confusion tensor C(h,, S) of classifier h with respect to some set S of n instances. In this case, we have:
O (hy. 5) Σ [m = u, (x,) = w].
(8)
Following Narasimhan et al. (015)), 3) we use the Frank-Wolfe algorithm (Frank& & Wolfe 1956) to perform implicit optimization over feasible confusion tensors Cp, without having to explicitly

---

Published as a conference paper at ICLR 2024
construct Cp. This is possible, because Frank-Wood only requires us to be able solve two sub- problems: First, given a classifier h, we need to calculate its empirical confusion tensor, which is straight-forward. Second, given classifier and its corresponding confusion tensor, we need to solve a linearized version of the optimization problem, which is possible due to [Theorem 4.1
Consequently, our algorithm, presented in Algorithm 1 proceeds follows: In the beginning, split the available training data into two subsets. One for estimating label probabilities 77. and one for tuning the actual classifier. After determining 77. we initialize h to be the standard top-k classifier, which will get iteratively refined as follows. For the confusion tensor of the current classifier, we can determine a linear objective based on its gradient. Because we can linearly interpolate stochastic classifiers, which will lead to linearly interpolated confusion tensors, this gives over which we can optimize a step-size and the confusion tensor this classifier. Based on this confusion tensor, we can do the next linearized optimization step, until we reach fixed limit for the iteration count. We represent the randomized classifier as set of deterministic classifiers h², and corresponding sampling weights a' obtained across all iterations of the algorithm. The Frank-Wolfe algorithm scales to the larger problems as it only requires Onm)) time per iteration.
descent direction
Algorithm 1 Multi-label Frank- algorithm for complex performance measures
Require: Dataset = {(a) 1/)) )...................... (I yn) number of iterations C N, stopping condition E
I: Split dataset S into S and S
2: Learn label marginals model ñz X - I2" on S1
3: Initialize h" :X'' 3'k
> Initial deterministic classifier Initial probability selecting the initial classifier he & Calculate the initial confusion tensor b Perform iterations D Calculate tensor gradients of Ci-1 in respect to v (gain tensor)
Initialize a {
5: C + C(h",
6: i [1.....,t] do
7:
G' - Vcl (C-
8:
a' < Gh Gúo
Gh
Gio b' <- G Gúo
9:
h' ('(x)) t top, (a' C (σx) + b¹)
b Construct the next classifier h b Calculate the confusion tensor of the next classifier h ((1-a)C' +aC") & Find the best combination C and C' (step-size) D Stop if the step-size a² smaller then e b Calculate a new confusion tensor based on the best a & Update all the previous probabilities of selecting corresponding h Return randomized classifier
10:
C' C(h'
11:
a' - argmax
12:
a < e then break
13:
a²)C (1-a')C C''C'
14:
for (o,.....,
15:
a' a'ta'(l a')
16:
return ((h"....,,h') h1), (a a'))
This algorithm can consistently optimize confusion tensor measure if it fulfills certain conditions:
Theorem 5.1 (Consistency of Frank-Wolfe), Assume the utility function : [0. 1] 1]mxXx2 - R
concave over Cp, L-Lipschitz. and 3-smooth w.r.t. the I-norm. Let S = S2) be a sample drawn i..dd from P.. Further, let î be a label probability estimator learned from S,, and h be the classifier obtained after En iterations. Then, for any 8 (0, 1), with probability of least 1 8 over draws of S,
from
log log n llog 5
8,3m
AT (h S ((E= (1))) Ô
m
+
n
kn + 2
The proof of this theorem, given in Appendix D broadly follows Narasimhan et al. 2015): First, we show that linear metrics can be estimated consistently with regret growing with the L1 error of the LPE. Then, we prove a uniform convergence result for estimating the multi-label confusion tensor. As prerequisite, we derive the VC-dimension of the class of classifiers based on top-k scoring, i.c. those classifiers that minimize some linear confusion tensor metric as shown in Theorem 4.1
Lemma 5.2 (VC dimension for linear top-k classifiers). For m X
[0, 1)" define
Hi = U (h: X - (0,1): h(x)= = 1 e top (a 3 n b)]}
(10)
a.bERm
The VC-complexity of this class is VC(H2,) 6m log(em).
"The classical version of FW uses fixed step-size schedule of _ instead of an inner optimization, but we find the latter give better results empirically. However, for the convergence result, fixed steps assumed.
7

---

Published as a conference paper at ICLR 2024
Table 2: Results of different inference strategies on measure calculated at (3. 5. 10). Notation:
P precisionn, R-recall, FI-FI-measure The green color indicates cells in which the strategy matches the metric. The best results are in bold and the second best are in italic. We additionally report basic statistics of the benchmarks: number of labels and instances in train and test sets, and average number of positive labels per instance, average number of positive instances per label.
Inference stritegy
Instance @33
Macto 3
Instance @85
Macro 5
Instance @10
Macro 6 10
R
F1
P
R
Fl
P
R
R
F1
MEDIAMILL (m1
101,
30993,
12914.
1,
Elllyl
1.36,
Ely Tummn
1338.8 8)
TOP-K
66.25
49.55
8.96
4.81
4,95
51.96
62.04
12.85
8.75
7.71
33.63
76.60
11,46
19.68
11.28
Top-K+w>tn
57.36
42.51
15.31
11.84
10.54
47.68
56.62
13.00
17.37
12.64
32.18
72.98
9.64
29.43
13.07
ToP-K+u K 190
39.72
27.32
14.43
10,10
9.41
35.40
39.96
11.38
15.33
10.95
28.45
63.36
9.86
26.25
12.26
TOP TOP-K+AA K+froe..
65.87
49.60
10.08
4.87
4.94
$2.08
62.16
11.99
8.93
7.90
33.61
76.65
10.76
20.08
11.37
ToP-KelAsT
65,88
49.48
10.31
4.58
4.80
51.55
61.87
11.10
8.50
7.48
33.54
76.75
10.73
19.55
11.16
MACRO-Poc
7.94
6.13
19.33
6.06
2.87
6.99
8.96
17.29
8.79
3.17
6.02
14.14
17.38
17.24
5.23
MACRO-Reuox
6.37
3.67
8.81
19.82
5.31
7,38
7.25
8.91
26.50
6.71
8.31
17.42
10.53
39.24
8.85
MACRO-Rew
6.37
3.67
8.81
5.31
7.38
7.25
8.91
26.50
6.71
8,31
17:42
10.53
39.24
8.85
MACRU-Flaw
45,20
33:05
15.42
11.17
12.21
43.57
$1.60
15.20
15.05
13.82
28.12
64.23
13.93
23.32
14.81
FLICKR (im
195, Phasi
56339,
21154,
Elll//l
1.31,
Ely
X Ruain
412.6)
TOP- K
23.94
56.96
23.04
38.41
26.56
16.99
66.01
17.12
47.03
23.49
10.16
77.35
10.72
59.37
17.24
TOP-K "ow
22.35
53.44
17.96
44.26
24.21
16.10
62.80
13.76
52.39
20.68
9.77
74.54
9:08
63.98
15.08
Top-K+w-m
23.57
56.17
19.86
41.36
25.49
16.76
65.21
15.05
49.75
22.00
10.06
76.63
9.79
61.80
16.10
ToP-K+fruw
23.64
56.27
24.90
26.42
16.89
63.62
18.53
45.67
24.16
10.05
76.63
11.77
57.90
18.14
ToP-K+fAMM
23,37
55.65
23.09
37,00
26.12
16.74
65.04
17.39
45.61
23.60
10.06
76.63
10.91
58.36
17.48
MACRO-PoW
4.65
11.49
39.34
6.63
8.06
5.66
22.75
41.74
9.70
10.57
2.83
22.26
37,59
10.68
8.50
MACRO-ER Rvain
16.14
38.62
17.58
45.50
22.27
12.17
13.98
53.83
19.72
7.89
60.42
64.66
15.07
MACRO-R. Row
16.14
38.62
17.58
45.50
22.27
12.17
47.48
13.98
53,83
19.72
7.89
60.42
9.57
64.66
15.07
MACRO-Flaw
17.59
41.60
35.28
29.28
29.43
12:22
47.31
34.13
32.70
29.43
5.92
45.77
34.55
33.08
29.02
RCVIX IX (m
2456, Plesis
623847, Tloa
155062,
4.80,
Elgy X Pheass]
1218.6)
TOP-K
72.99
75.32
13.06
4.67
5.43
52.30
81.96
12.77
7.61
7.64
32.98
89.70
11.35
14.75
10.28
TOP-K+w"" Km
65.99
69.11
18.58
12.78
13.09
48.48
77.18
14.69
17.66
13.64
31.43
87.14
10.63
26.05
/2.82
Top-K+w>
70.70
73.37
19.97
8.10
9.80
51.18
80.49
16.03
11.75
11.29
32.66
89.14
11,96
19.01
12.06
Top-k+
71.99
74.38
14.06
4.83
5.76
51.46
80.94
12.49
7.65
7.75
32.38
88.75
10.59
14.42
10.06
TOP K+EAsrn
71.14
73.60
14.40
5.44
6.46
50.81
80.13
12.27
8.52
8.41
31.88
87.85
9.64
15.16
10.03
MACRO-Pwr Pow:
46.36
50.11
21.11
5.61
5.84
29.40
49.81
69
5.72
5.31
19.45
60.40
21.66
6.03
5.78
MACRO-Ream
44.26
46.10
14.60
18:24
12.04
34.77
56.28
13.13
24.39
12.77
24.08
70.51
10.66
34.34
12.39
MACRO-Rew Rew
43.28
44.99
14.56
18.41
11.95
34.15
55.24
13.15
24.89
12.73
23.78
69.71
10.76
34.66
12.44
MACRO-F1
58.20
61.22
21.45
10.37
12.09
44.42
71.86
21.96
12.25
13.68
27.26
78.88
22.10
14.86
15.12
AMAZONCAT (m
13330, Phesss
862339
Thoer
306782,
16 |
5.04, 15
Meaar]
448.6)
TOP-K
78.29
59.29
35.73
12.44
16.52
63.63
74.54
46.43
32.72
35.06
39.16
85.18
39.52
$1.69
40.39
66.32
49.76
50.21
45.79
45.70
57.12
67.49
44,85
53.78
46.30
37.31
82,20
30.13
63.53
37.15
TOP-K+w
72.56
54.56
50.30
32.06
36.94
61.15
71.83
48.93
42.87
43.05
38.71
84.49
36.84
56.71
40.60
MACRO-PP. Prw
47.00
35.57
23.74
29.62
41.04
50.74
$5.85
27.45
30.66
69.67
55.27
29.09
34.51
MACRO-ReaaR
48.58
34.93
37.16
59.97
42.02
40.67
47.35
28.17
66.98
35.75
28.06
62.91
17.62
73.98
25.04
MACRO-Rew Row
48.58
34.93
37.15
59.97
42.02
40.67
47.35
28.17
66.98
35.75
28.06
62.91
17.62
73.98
25.04
MACRD-FL.e
68.59
51.49
56.75
34.68
40.90
55.73
65.60
56.62
36.40
41.92
35.30
78.34
54.67
39.93
43,26
6 EXPERIMENTS
In this section, we empirically evaluate the proposed Frank- algorithm on a variety of multi- label benchmark tasks that differ substantially in the number of labels and imbalance of the label distribution: MEDIAMILL (Snock et al. 2006)_ FLICKR (Tang & Liu| 2009). RCVIX (Lewis) et al. 2004), and AMAZONCAT (Mcyuly & Leskovee 2013 Bhatia et al. 2016). For the first three datasets
multi-layer neural network for estimating x))) For the last and largest dataset, we use sparse linear label tree model, which is a common bascline in extreme multi-label classification (Jasinska-Kobbe setup of probability estimators.
al. 2020 In Appendix F we include all the details regarding the
We evaluate the following classifiers optimizing the macro-at-k: measures:
MACRO-PEw. rw. MACRO-Rpw. MACRO-FI randomized classifiers found by the Frank- Wolfe algorithm (Algorithm for optimizing macro precision. recall, and F1, respectively. based on (x) coming from the model trained with binary cross-entropy loss (BCE).
*Code reproduce the experiments:
8

---

Published as a conference paper at ICLR 2024
MACRO- -ACRO-RPRIOR- implements the optimal strategy for macro recall, which selects & labels with the highest Pj 5-16 カ, P,s are estimates of label priors obtained on training set and ()x) are given by the model trained with BCE loss.
As baselines, we use the following algorithms:
TOP-K: K: selects labels with the highest 77 coming from the model trained with BCE loss; the optimal strategy for instance- wise precision at k (Wydmuch et al. (0118).
TOP- TOP-K+W"OM Pow TOP-K+ www similarly to ToP-K, selects labels with the highest wym where wy calculated
function of label priors corresponding
the power-law,
w/ = =D and log weights, w log Py. with P estimated on the training set. For power-law weights, we use 0.5. This kind of weighting aims to put more emphasis on less frequent labels.
TOP-K+(FOCAL- ToP-K+ multi-label focal loss and asymmetric loss (Lin et al. 2017] Ridnik et al. (02TT) are variants of BCE loss, commonly used in multi-label classification to improve classification performance on harder, less frequent labels. Here, we train models using these losses and select l labels with the highest output scores.
For all baselines and MACRO- RYRIOR- we always train the label probability estimator on the whole training set. For MACRO-P Pww MACRO- Few, and MACRO-F1 fw, we tested different ratios (50/50 or 75/25) of splitting training data into sets used for training the label probability estimators and estimating confusion matrix C, well variant where We also investigated two strategies for initialization of classifier h by either using equal weights (resulting TOP-K classifier) or random weights. Finally, we terminate the algorithm if we do observe sufficient improvement in the objective. In practice, we found that Frank- converges within 3-10 iterations. Because of the nature of the random classifier, we repeat the inference on the test set 10 times and report the mean results. In Table 2 we present the variant achieving the best results, and report all the results including standard deviations, running times, number of Frank- iterations in Appendix G
the whole training set for both steps.
The randomized classifiers obtained via the Frank-Wole algorithm achieve, in most cases, the best results for the measure they aim to optimize, at the cost of loosing some instance-wise measures. However, they sometimes fail to obtain the best results on the largest dataset, where the majority of labels have only few (less than 10) positive instances in the training set, preventing them from obtaining accurate estimates of 77 and C. In this case, simple heuristics like ToP- K+w Pow might work better. Popular Focal loss and Asymmetric loss preserve the performance metrics, but improvement on the macro measures is usually small. It is also worth noting that, as expected, MACRO-RA RRw recovers the solution of MACRO-Rpwe in all cases.
instance-wise
CONCLUSIONS
In this paper, we have focused on developing consistent algorithm for complex macro-at-k metrics in the framework of population utility (PU). Our main results have been obtained by following the line of research conducted in the context of multi-class classification with additional constraints. However, these previous works do not address the specific scenario of budgeted predictions k. which commonly arises in multi-label classification problems. For the complex macro-at-k metrics, we have introduced a consistent Frank-Wolfe algorithm, which capable of finding an optimal randomized classifier by transforming the problem of optimizing over classifiers to optimizing over the set of feasible confusion matrices and using the fact that the optimal classifier optimizes (unknown) linear confusion-matric Our empirical studies show that the introduced approach effectively optimizes macro-measures and it scales to even larger datasets with thousands of labels.
ACKNOWLEDGMENTS
A part of computational experiments for this paper had been performed in Poznan Supercomputing and Networking Center. We want to acknowledge the support of Academy of Finland via grants 347707 and 348215.

---

Published as a conference paper at ICLR 2024
REFERENCES
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach fair classification. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 60-69. PMLR, 10-15 Jul 2018.
Rohit Babbar and Bernhard Schölkopf. Dismec: Distributed sparse machines for extreme multi-label classification. In Proceedings of the tenth ACM international conference web search and data mining, pp. 721-729, 2017.
Rohit Babbar and Bernhard Schölkopf. Data scarcity, robustness and extreme multi-label classification. Machine Learning, 108(3):139--351, 2019.
Peter I. Bartlett, Michael Jordan, and Jon D McAuliffè. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-155 2006.
Eric Baum and David Haussler. What size net gives valid generalization? Advances neural information processing systems, 1, 1988.
Kush. Bhatia, Kunal. Dahiya, Himanshu Jain, Anshul Mittal, Yashoteja Prabhu, and Manik Varma.
The extreme classification repository: Multi-label datasets and code, 2016. URL http://
manikvarma org. wwwpg//dwwoaadds.oooomooooo....m XC/ XMMRRepssitory
Róbert Busa-Fekete, Balázs Szörényi, Krzysztof Dembezynski, and Eyke Hüllermeier. Online F-measure optimization. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015. Montreal, Quebec, Canada, pp. 595-603, 2015.
Clément Calauzenes, Nicolas Usunier, and Patrick Gallinari. On the (non-) existence of convex, calibrated surrogate losses for ranking. Advances in Neural Information Processing Systems, 25, 2012.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with fabel-distribution-aaaa margin loss. Advances in neural information processing systems, 32, 2019.
Wei-Cheng Chang, Daniel Jiang, Hs Yu, Choon Hui Teo, Jiong Zhang, Kai Zhong, Kedarnath Kolluri, Qie Hu, Nikhil Shandilya, Vyacheslav levgrafov, et al. Extreme multi-label learning for semantic matching in product search. In Proceedings the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pp. 2643-2651, 2021.
Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. Performance of recommender algorithms on top-n recommendation tasks. In Proceedings of the fourth ACM conference on Recommender systems, pp. 39-46, 2010.
Krzysztof Dembezynski, Willem Waegeman, Weiwei Cheng, and Eyke Hüllermeier. An exact algorithm for F-meauure maximization. In Advances Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of meeting held 12-14 December 2011, Granada, Spain., pp. 1404-1412, 2011.
Krzysztof Dembezynski, Wojciech Kotlowski, Oluwasanmi Koyejo, and Nagarajan Natarajan. Con- sistency analysis for binary classification revisited. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-1 1 August 2017, vol- ume 70 of Proceedings of Machine Learning Research, pp. 961-969, PMLR, 2017.
Chris Drummond and Robert C. Holte. Severe class imbalance: Why better algorithms aren't 't the answer. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML. PKDD), pp. 539-546. Springer, 2005.
John C. Duchi, Lester W. Mackey, and Michael I. Jordan. On the consistency of ranking algorithms. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 327-334, 2010.
10

---

Published as a conference paper at ICLR 2024
Charles Elkan. The foundations of cost-sensitive learning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence Volume 2, IJCATO1. pp. 973-978, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608123 125.
Stack Exchange. Continuous linear image of closed, bounded, and convex set of a hilbert space is compact. URL httthtt///mmahhtti wtwpaaac.stccccccccaaaaeaa com/ /tt.o/////011/// Accessed: 2023-09-
27.
Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research, 9:1871-1874, 2008.
Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval Research Logisties Quarterly, 3(1-2)955-10, 1956. doi
Muhammad Hanif and K. R. W. Brewer. Sampling with unequal probabilities without replacement: review. International Statistical Review, 48:317-335, 1980.
Martin Jaggi. Revisiting frafk-wolfe: Projection-free sparse convex optimization. In International conference on machine learning, pp. 427-433 PMLR, 2013.
Himanshu Jain, Yashoteja Prabhu, and Manik Varma. Extreme multi-label loss functions for rec- ommendation, tagging, ranking and other missing label applications. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '16, pp. 935-944. Association for Computing Machinery, 2016. ISBN 9781450342322.
Kalina Jasinska, Krzysztof Dembczynski, Róbert Busa-Fekete, Karlson Pfannschmidr, Timo Klerx, and Eyke Hüllermeier. Extreme F-measure maximization using sparse probability estimates. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY. USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1444 JMLR.org, 2016.
Kalina Jasinska-Kowl Kobus, Marek Wydmuch, Krzysztof Dembczyński,, Mikhail Kuznetsov, and Róbert Sekete. Probabilistic label trees for extreme multi-label classification. CoRR, 121/2009.11218. 2020.
Thorsten Joachims. A support vector method for multivariate performance measures. In Proceedings of the 22nd International Conference on Machine Learning (ICML 2005), August 7-11, 2005, Bonn, Germany, 2005.
Purushottam Kar, Harikrishna Narasimhan, and Prateek Jain. Surrogate functions for maximizing precision the top. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 189-198, 2015.
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In Jennifer Dy and Andreas Krause (eds.), ), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2564-2572. PMLR, 10-15 Jul 2018
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Wojeiech Kotlowski and Krzysztof Dembczyáski. Surrogate regret bounds for generalized classifica- tion performance metrics. Machine Learning, 10:549-572, 2017.
Oluwasanmi Koyejo, Nagarajan Natarajan, Pradeep Ravikumar, and Inderjit S. Dhillon. Consistent binary classification with generalized performance metrics. In Advances in Neural Information Processing Systems 27: Annual Conference Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 2744-2752, 2014.
Oluwasammi Koyejo, Nagarajan Natarajan, Pradeep Ravikumar, and Inderjit S. Dhillon, Consistent multilabel classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015. Montreal, Quebec, Canada, pp. 3321-3329, 2015.
11

---

Published as a conference paper at ICLR 2024
David D Lewis. Evaluating and optimizing autonomous text classification systems. In Proceedings of the 18th Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval, pp. 246-254. ACM, 1995.
David D Lewis, Yiming Yang, Tony Russell-Rose, and Fan Li. Rev1: A new benchmark collection for text categorization research. Journal of machine learning research, 5(Ap7):361-397, 2004.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
William G. Madow. On the theory of systematic sampling, II. The Annals of Mathematical Statistics, 20())333-354, 1949.
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topies: understanding rating dimen- sions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pp. 165-172, 2013.
Julian McAuley, Rahul Pandey, and Jure Leskovec. Inferring networks of substitutable and com- plementary products. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 785-794, 2015.
Aditya K. Menon, Harikrishna Narasimhan, Shivani Agarwal, and Sanjay Chawla. On the statis- tical consistency of algorithms for binary classification under class imbalance. In International Conference on Machine Learning (ICML), 2013.
Samrat Mukhopadhyay, Sourav Sahoo, and Abhishek Sinha. k-experts mental limits. In Proceedings The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 342-365. PMLR, 2022.
online policies and funda-
Harikrishna Narasimhan, Harish Ramaswamy, Aadirupa Saha, and Shivani Agarwal. Consistent multiclass algorithms for complex performance measures. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2398-2407, Lille, France, 07 2015. PMLR.
Harikrishna Narasimhan, Harish G. Ramaswamy, Shiv Kumar Tavker, Drona Khurana, Praneeth Netrapalli, and Shivani Agarwal. Consistent multiclass algorithms for complex metrics and constraints, 2022.
Nagarajan Natarajan, Oluwasanmi Koyejo, Pradeep Ravikumar, and Inderjit Dhillon. Optimal classification with multivariate losses. In Proceedings The 33rd International Conference on Machine Learning (ICML), pp. 1530-1538, 2016.
Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Cost-sensitive learning with noisy labels. Journal of Machine Learning Research, 18(155):1-33, 2018
Juri Opitz and Sebastian Burst. Macro fl and macro fl, 2021.
Shameem Puthiya Parambath, Nicolas Usunier, and Yves Grandvalet. Optimizing F-measures by cost-sensitive classification. In Neural Information Processing Systems (NIPS), 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-BB Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035, Curran Associates, Inc., 2019.
12

---

Published as a conference paper at ICLR 2024
Harish G. Ramaswamy, Ambuj Tewari, and Shivani Agarwal. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics, 12(1):530 - 554, 2018. dbi100.21447--ES13333
Pradeep Ravikumar, Ambuj Tewari, and Eunho Yang. On ndcg consistency of listwise ranking methods. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 618-626. JMLR Workshop and Conference Proceedings, 2011.
Tal Ridnik, Emanuel Ben-Ban Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi-label classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 82-91, 2021.
Siegfried Schaible and Jianming Shi. Fractional programming: The sum-of-ratios case. Optimization Methods and Software, 18(2):219-229, 2003.
Erik Schultheis, Marek Wydmuch, Rohit Babbar, and Krzysztof Dembezynski. On missing labels, long-tails and propensities in extreme multi-label classification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1547-1557, 2022.
Erik Schultheis. Marek Wydmuch, Wojeiech Kotlowski, Rohit Babbar, and Krzysztof Dembezynski. Generalized test utilities for long-tail performance in extreme multi-label classification. 36: 22269- 22303, 2023.
Shashank Singh and Justin T Khim. Optimal binary classification beyond accuracy. In Advances in Neural Information Processing Systems, volume 35, pp. 18240. Curran Associates, Inc., 2022.
Cees G. M. Snoek, Marcel Worring, Jan C. van Gemert, Jan-Mark Geusebroek, and Arnold W. M. Smeulders. The challenge problem for automated detection of 101 semantic concepts in multime- dia. In ACM International Conference on Multimedia, pp. 421 121-430. Association for Computing Machinery, 2006.
Lei Tang and Huan Liu. Relational learning via latent social dimensions. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '09, pp. 817-826, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605584959. doi [0. 11/01//1///570099557709999
Ambuj Tewari and Peter L. Bartlett. On the consistency of multiclass classification methods. Journal of Machine Learning Research, 8(5), 2007.
Willem Waegeman, Krzysztof Dembezynski, Arkadiusz Jachnik, Weiwei Cheng, and Eyke Hüller- meier. On the baycs-optimality of F-measure maximizers. Journal of Machine Learning Research, 15(1))3333-3388 2014.
Shuo Wang and Xin Yao. Multiclass imbalance problems: Analysis and potential solutions. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 42(4): 119-1130, 2012.
Xiaoyan Wang, Ran Li, Bowei Yan, and Oluwasanmi Koyejo. Consistent classification with general- ized metrics, 2019.
Tong Wei and Yu-Feng Li. Does tail label help for large-scale multi-labed learning? IEEE transactions neural networks and learning systems, 31(7):2315-232 2019.
Marek Wydmuch, Kalina Jasinska, Mikhail Kuznetsov, Róbert Busa-Fekete, and Krzysztof Dem- bezynski. A no-regret generalization of hierarchical softmax extreme multi-label classification. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Disa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 6355-6366. Curran Associates, Inc., 2018.
Forest Yang and Sanmi Koyejo. On the consistency of top-k surrogate losses. In International Conference Machine Learning, pp. 10727-10735. PMLR, 2020.
13

---

Published as a conference paper at ICLR 2024
Nan Ye, Kian Ming Adam Chai, Wee Sun Lee, and Hai Leong Chieu. Optimizing F-measure: A tale of two approaches. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh. Scotland, UK. June 26 July 1, 2012. icml.ce / Omnipress, 2012.
Ming Yuan and Marten Wegkamp. Classification methods with reject option based on convex risk minimization. J. Mach. Learn. Res., 11:111-130, mar 2010. ISSN 1532-4435.
Tong Zhang. Statistical analysis of some multi multi-category large margin classification methods. Journal of Machine Learning Research, 5(Oct):1225 (:cc:)::222-1111 1251, 2004.
14

---

Published as a conference paper at ICLR 2024
A MADOW'S S SAMPLING SCHEME
In this section we present sampling scheme for the following sampling problem: given real-valued
vector π E [0, 1|TH marginal probabilities with || | k. sample binary vectors y E {0, 1
such that the distribution of has as the marginals, [[ =
Theorem A.1. Let 2 1. Given vector E [0, satisfying || k. Algorithm 2 returns randomized binary vector y E {0, 1]"h of size | k, with marginals given by n,, [lg n. The algorithum runs in (m) time.
Algorithm 2 Madow's 's sampling scheme
Require: Vector of marginals π E [0, 1)"h with 1/|1 k
Ensure: random vector ÿ E {0, } with | such that E = π
I: Compute TT 0 and IT = 51 + Ty for 1....,,..
2: Sample uniformly distributed random variable U from the interval [0.
y 0
4: for E (..... A- - 1) do
5: Select such that - < U +U
6: Set ŪJ =
7:
return y
The algorithm is due to Madow (Madow 1949 Mukhopadhyay et al. 2022). and the considered sampling problem has been studied in the statistical literature under the name unequal probability sampling design (Hanif & Brewer 1980). Below give simple proof of correctness of the algorithm for completeness.
Proof. First note that for any E 10.1,...,4 1), there exists unique j for which IT, < U Uis m 11,. This is because due to ビj= k.. = k, the intervals (Ilo, 11,], (11, I12]...., (11m-1. [1m] are disjoint and cover (0, k], whereas U
(0, with probability one. Furthermore, the algorithm will select distinct j's for distinct i's. This is because the condition II < U ++ < 11, is equivalent E jMi- 1 U, 1 U}, and the interval (11 U, II UJ] have width ny and thus can contain at most one integer. So the algorithm will return ÿ with exactly is ones.
Since E [
P
1]. we need show that this probability is equal to и for each j. We have
P = 1 e t=0
U i, - -
h-1
(0, 1 (1,-1 -1 -1==10 =I,,- -1-111 = my
(11)
The theorem and the algorithm from its proof can then be used generate prediction vectors independently for any instance of interest æ by setting π h(x).
B THE OPTIMAL CLASSIFIER FOR LINEAR METRICS
Theorem 4.1. The optimal classifier ht argmax (h) for (h) G C(h) is given by
(x) ta @
(5)
where 3 denotes the coordinate-wise product of vectors, while the vectors a and b are given by:
aj = G G - - - by = Go G%
(6)
and (w) returns k-hot vector extracting the top k largest entries of u (ties broken arbitrarily).
15

---

Published as a conference paper at ICLR 2024
Proof. The linear metrie is decomposable over instances as:
G G),C((hh)) = - -
-
+ G - (ny
= b1)hh, + Tj.
(12)
where aj and by are as stated in the theorem, while
= Go El - 1[1-7,(a))) (x)] + G (z)
(13)
Thus, we can rewrite the objective as:
((h) E - =E E (2) + (2))
+ R.
(14)
where R = r1 +... mm does not depend on h. For each E X, the objective can be independently maximized by the choice of hxx)= E A which maximizes ΣØ=1 (a,m, (ar) + (x)) But this is achieved by sorting (z)) bb in descending order, and setting h, (a) = 1 for the top & coordinates, and (ar) 0 for the remaining coordinates (with ties broken arbitrarily).
Let us notice that coefficients analogous to our a; and b, can also be found in the cost-sensitive prediction rule in binary classification (Elkan 2001 Natarajan et al. 2018).
C THE OPTIMAL CLASSIFIER FOR GENERAL METRICS
In this section, we prove the existence and the form of the optimal classifier. Our results extend past results on binary classification (Koyejo et al. 2014)) and multi-class classification (Narasimhan et al. 2015). We first show that the set of confusion tensors achievable by randomized k-budgeted classifiers is a compact set. Then, the statement of the main theorem follows from the first-order optimality conditions as well as the absolute continuity of marginal vector (7x). We stress that the results here are general and applicable to any mutli-label utility satisfying Assumption 4.3 which need not necessarily be a macro- averrged utility.
We remind that the set of confusion tensors achievable by randomized k-budgeted classifiers on distribution P, is denoted as
Cp = (C(()) h E H}. he
(15)
and that optimizing the metric (h) over h H is equivalent to optimizing (CC over C Cp. Lemma C.1. Cp is convex set.
Proof. Take any C1, C2 E Cp and any À E [0, 1]. and we show that C3 = XC1 + +1 E Cp. Since C1, C2 E Cp, there exist k-budgeted randomized classifiers h and hgg such that C₁ = C(h)) and C2 = C(h2). Take hy defined as hy (x) Xh (x) + (1- 2)h2(z)) for any x E X. Since A i convex and h (x), hz(x) E A for all X,, it also holds that (x) A for all x E X, so h2 is also k- budgeeeed randomized classifier. Since the confusion tensor is linear in predictions, we
have C(hx)= AC(h,) + (1 (h)((h)) = C.. which proves that C₃ E Cp.
We now argue that for the analysis of Cp, it suffices to consider classifiers of the form h f n, i.e. (x) f(n(#)) for some function f: [o, 11mh + Ak..
Lemma C.2. For any E H, there exists function f: [0, 1) - A such that and f o n have the same confusion tensors.
Proof. If h is not of the form f o n, we define function f as:
f(n) [Th(=)))(<>>> =n]
(16)
16

---

Published as a conference paper at ICLR 2024
Due to convexity of A we have f(n) E A Moreover, it is easy to see that C(h) C(f on);
for instance,
(22
n/xx) =
Eln, (lhy(m) = [ ([()) = (f,
(17)
Hence, any confusion tensor achievable by some h is also achievable by some f o 77, so that the set of achievable confusion tensors can be written as Cp [C(f n): f E F), where we denote == (f: [0. 1m + A From this moment on, we thus, without loss of generality, consider optimizing the metrics over functions f E F of random vector n, and make the relation h = f implicit, writing (f) for (f o n7) and using C(f) to denote the confusion tensors C(f o n). that
c(f) = (C
C (f)))
(18)
where
(t [En [[m[[[] - 7,)(1 fn() 1yy((1 7s)fs (n)
En (1 (1
(19)
En (n)
Lemma C.3. The mapping f + C(f) is continuous: for any f.f F
[[C(S) C(f')l - 2 - f
(20)
where -C(f))C([P)) C( == - VI [[))((s) C
Proof. Fix j E [m]. Using dy(n) = fy(1) - fj)()), we have from the definition:
Enl- 77; 78 (1))
Ey [,
-
En [1 175 ())
(n)
=En (y())) (-(1 - fl
(21)
Since the squared Frobenious norm X + | is convex, we can use Jensen's inequality ||[]]||
< [|||
x 12e to get
((1 - 173) 1--11 7i5 Ti
|C'(1,) ( <
Fon 85(11)
(-(? m))))
< En
(8, (n))
- 2 2, (6,,())))
(22)
where we used
(- m3)
-mp 2 = 2 2(1 -n,)² + 2 max ((1 -
2.
(23)
=c(0.1]
Summing the inequality over j 1, 1,..., m and taking square root on both sides finishes the proof.
We will now show that the set of achievable confusion tensors Op is compact. To this end, we first prove result from the functional analysis (which is false without the convexity assumption).
Lemma C.4. Let L H + V be continuous affine operator between Hilbert space IH and finite dimensional vector space V. If S c. is closed, bounded, and convex, then L(S) is compact.
Proof. Observe that it suffices to prove this when C is linear since being compact is translation invariant.
17

---

Published as a conference paper at ICLR 2024
The proof is inspired by answer to related question Mathematics Stack Exchange It suffices to prove every [(zn) sequence in C has a convergent subsequence whose limit is in ((S).
By the Manach-Alaogle theorem, balls in Hilbert spaces weakly compact. For the convenience of the reader we will sketch this proof. Recall that weak convergence 3n + x in H means that for all linear functionals every sequence has a subsequence that converges weakly. Now onto the proof.
H" there is convergence <(x,)) 6(x). Likewise weakly compact means
Let H., be bounded sequence in H. Let e2........ be Hilbert basis for HI and the dual vectors {or, 02....} a Hilbert basis for HI where (x) = (ar, ex). Now apply the diagonal proof method, as in the bounded we know that Φ1 (z,, is bounded in IR and hence we can extract subsequence 2n so that &1 (z,,) + a1 where we may keep æ1. Now on this subsequence do the same for Φ2(x,) - ag but keep 21 and 2/2. Continue this process, where at the m-th step one keeps the first m terms from the previous subsequence. The resulting diagonal subsequence Iy, is such that (x,,) - a, for each اi dze; is in IH (by Bessel's inequality, the weak convergence
Ascoli theorem, of successively passing subsequences. Since the sequence is
i i= 2, The element results, and the fact that the original sequence was bounded). remains to verify that Fn - x weakly, but for this it suffices to check (,,,), + (ar) and by design this is the case.
Now, returning to the proof of the lemma since C H is bounded, is contained in a ball, and hence by passing subsequence we have In + in the weak topology for some H. Furthermore I E S. x wasn't it, then since S is closed and convex, by the Hahn-Banach separation theorem there is a separating hyperplane 6 E HI (xx) < inf ((S). But this contradicts that the weak convergence In + x since In E S.
So it remains to prove convergence C(x,) - C(x). Since C is continuous we have convergence [(x,) - f(x) in the weak topology, but this implies normal convergence since V is finite dimen- sional.
Lemma C.5. Cp is compact set.
Proof. To show that Cp is compact, we will invoke Lemma C.4 To place ourselves in its setting, let the Hilbert space be H T,² (10, 1]m,, RT ()) where fa is the probability measure on [0, 1 1m associated with random vector (+)) The inner product for f.g: [0, 10m + R² in H is
S ((()).(() dp(n)
(f.9) - 10.11-
(24)
where the inner product inside the integral is the normal dot product in R We have the affine map defined via
L THE
R mx2x2
where =(f)=(ffl n);
(25)
and let the subset S C H be
S S= 1[0, A almost everywhere).
(26)
Since Cp = L(S), it suffices to verify the assumptions in Lemma C.4
The map L is continuous by Lemma C.3 The set S is convex since the set of A Rm is convex. Likewise for bounded using also that the we are working with a probability measure: If f E S, then (1)) < for all n [0, 1]mh and hence
|| = / 15()))dp())
J
S Ju..) m du(n)) = m.
(27)
|0.11
Similarly the closedness of A C Rmm translates into the closedness of S as we now prove. Suppose there is sequence of f, E S with fn + f and f & S. This means the set of points
A= = (n E 00 11mh f & +
(28)
that maps out of A& has positive measure /a( (4) function d(2, A = inf PeAt || l|| and for > 0 define the set
In Rm there is well defined distance
A (n E (0, 1] d(f())) A >
(29)
18

---

Published as a conference paper at ICLR 2024
Note that A = U A1g3 since A is closed and hence there is some 0 such that ju(Ae) O.
Therefore for all
|l fnll 1 Mp.1p lf(n) (
/ (mm)))((())()) S e dd((n) = =Ep(AA) >
(30)
where the second inequality uses that (T) E Am almost everywhere. That |l f.ll2 is uniformly bounded away from 0 contradicts that f
iimHH H.
Theorem 4.4. Let the data distribution P and metric v satisfy Assumption 4.2 and Assumption 4.3 respectively. Then, there exists an optimal C* E Cp, that is W (C') = Vr, Moreover, any classifier
h' maximizing the linear utility G C(h) over h E H with G (G', ...., G given by G (C*)(C')), also maximizes (h) over h E H.
Proof. Let C* argmaX (C), which exists by the compactness of Cp (Lemma C.5 and the continuity of v. By the first order optimality and convexity of Cp, for all C Cp
(C(C") C² VV(CC) C.
(31)
which implies:
C = argmax G C
(32)
CECP
for G = (C'(
We now show that C* is the unique optimizer of (32) j E [m]:
Using Assumption 4.3 applied to C*, for all
0 v s...,, "...,, +e "*,,,.,,,*'+ ( -1) ....,.,,C*' C Cmm
)
-VoVV(C")
Oe
===
G + Gi - Gi G = a, > 0,
(33)
with coefficients J E [m] defined in Theorem
Now, since we just showed that a; * 0 for all j, and v7 has density, coordinates of n b all distinct with probability one. This means that, with probability one, top, (a @ + b) is a singleton, and thus the optimizers of the linear utility G C(h) can only differ all have the same confusion tensor. Thus, C* uniquely maximizes linear utility G C over C E Cp.
measure set, they
This means, however, that any classifier h* maximizing G C(h) over h H has C(h*) C+ and thus maximizes V.
D CONSISTENCY OF WOLFE
In this section, we provide the formal proof of consistency for the Frank- Wolfe algorithm. We prove convergence for a slightly modified version of Algorithm 1, in which we replace the line-search in line 13 with fixed schedule, setting
a' a't 2
(34)
1+1
For the experiments, we used the line-search instead, as we found it to give slightly better results.
D.1 VC-DIMENSION LEMMA
Lemma 5.2 (VC dimension for linear top-k classifiers). For m. X - [0, 1) define
Hi
(h: X
10.1): h(x) =1I topı(a 3 n- b)]}
(10)
am
The VC-complexity of this class is VC(HG) 6m loggem))
19

---

Published as a conference paper at ICLR 2024
Proof. For any given a, b, the hypothesis predicts one, (²)x)) iff exists set of k indices
C [m] with [Z] m - j. j & I, such that for all i E I the score a.?) + b, S a, + by is not
greater than the score of label
This computation can be realized as a two-layer network. In the first layer 2, we calculate an indicator
to determine which labels' scores are below the threshold, that is 2; 1[(a; ag) )n; (b: by) ) Then, for the output, we threshold the sum of all the intermediate units to determine if j is predicted:
h( = o()) = Σ
m k
(35)
The resulting network has 2(m 1) edges and m 1 computation nodes. If we allow the output node to be more general-a generic linear threshhold function-, the VC-dimension of this extended function class H" can only grow. For this extended class, we can apply (Baum & Haussler 1988 Corollary 3), which gives an upper bound for the VC-dimension of
VC(H)) < VC(H') s 2(m 1 + 2( m 1)) log(e(m- - 1)) s 6m log(em).
(36)
D.2 ADDITIONAL LEMMAS
Before going into the main proof of Theorem 5.1
we provide two more helper lemmas:
Lemma D.1 (Regret for Linear Macro Measures). Let G be a linear macro-measure, that is,
m
G(h;; n) =
Ex (1 - 11, -
f=1 +
(1- (11-))
(z) hy(x)) ((nn()))((())
(37)
Let hx()))))+))))))))()))) = argmax, G(h; n). and h()) == argmax, G(h; 77. Then
G(h";n) G Gh; m) S m
max | 1.1 E (=x) ]]
(38)
Proof. As Gh; is an affine function in its second argument, we can simplify differences to
m
G(::n) Gh: n) =
m
Ex
Goo 71)(1 - h2) - Go (15 93}h, + Gio(n, - 71 - h,) + Ci - 7T hg
j=1
m
Ex [n, 1
- Gi + (Gi (1 - -n,))]
(39)
j=1
We can use this property to bound the regret of as - = G(h';n) + -G(mm)
G(h":n) G(h + G(: -(h.))
Ex (11 - 1)) (G)) - - -T,) + GG - Gin - (hj])))))))
m
m Σ
(G), 0% - Gio + G/ E.n [10 - 71) - -ī,]]
(40)
As h2 E [0, 1]. we can bound (n= - -m;) - -T,)) - [11 - 11 resulting in
G(h": G(h: n) < m - ( 1/ - G + G EE [1m - 7il
(41)
Using the notation of Theorem 4.1 we set ay G/ G/n Cio + Gooo so that can further bound
G(h':n) - G(h:: n) I m max [ag]
Σ
Emg [m - 7 = m max |a,1 Ean 11 711 (42)
1=1
20

---

Published as a conference paper at ICLR 2024
Using
max layl S max || 11 j
(43)
yields the claim.
Lemma D.2 (Uniform Convergence of Multi-label Confusion Tensors). For 17: X - [0, 1 let Hn ==
U a
hh: X' - {0, 10 h(x) = top, a ④ 7 b)
(44)
and let S (X x {0, n be an i.i.d. sample. Then for any & (0, 1], with probability at least 1 6, we have
[C(h, - s 001) log log log 6
sup hEHn
(45)
n
Proof. Instead of showing uniform convergence for the entries of the confusion tensor directly, show it for accuracy (0-1-error) ace C + C.o- condition positive rate q' Co Cf, and predicted positive rate p Ci C₁,, first for fixed j [m]]
To handle accuracy and predicted positives, consider
lace (h,P) P)) acc (h, hEHn
Σ 1[ h,(x;)]] - Ply; h hz())) i=1
sup heH,
sup
sup NEH,
Σ 2=1
1== = h(xx)) - Plys = h[>)]
(46)
From Lemma 5.2
we know the VC-dimension of HJ is some finite number d, thus, we can employ standard bound for the 0-1 error get, with probability
that
2d 2d (2en/d) + 1lg((4/0)
sup laco' (h,P) ace (h, 5) h
(47)
n
As this bound holds for all distributions of targets y. it holds in particular also for y 1, in which case accuracy turns into predicted positive rate.
Finally, bound the error the condition positive simply using Hoeffding's inequality, as does not depend on the hypothesis h. We get, with probability
log
lq(h, P) (h, S)I hE
sup
(48)
2n
Now we can reconstruct the actual entries of the confusion matrix. For example, the true positive rate as Ip 1-ace-9-P Thus, we can union bound, with probability
2dlog(2en/d) + (8/8)
log(3/5)
sup (h, 6p' (h. <
(49)
he
2n
Similar bounds can be constructed for the other entries. Taking a union bound over all labels:
2d 2og(ogeen + log(8m/
1og(3m/0)
sup ||C(h, PP C(h, -
he
2n
12m log(em) Dog(en/(3mllogg))
log(8m/0)
log (3m/o)
(50)
n
2n
In order to combine the two square-root terms, we can apply the arithmetic-quadratic mean inequality, arrive at the claimed bound
48m (omm + 10 ( \/3 17.8 mm/)
sup ||C(h, (C - heR
2n
21

---

Published as a conference paper at ICLR 2024
Finally, using 3m(log(em) 2 1, we simplify
(og(en((3((lg((em))) I log(en),
(51)
which results in
Olm O(mllg log log + (log(m//))
sup heHn
IC 8lb,
2 (((,,))))
(52)
D.3 BOUND FOR LINEAR OPTIMIZATION STEP
The preceding results allow to prove a bound on the approximation emror for each linear optimization step that is performed part of the Frank- Wollff algorithm:
Lemma D.3. Let V - Rz0 be concave over Cp, L-Lipschitz, and 3-smooth w.r.t. the & n-ror. Let h & H be some classifier, and denote G = (CC(C( S)). Let 9 be the deterministic classifier that empirically optimizes the linear objective induced by v according to Theorem 4.1 For two classifiers h' and g', define
Lp CC''')) (C
(53)
Then for any E (0, 1]. with probability at least 1 5 (over draws of from IP),, we have
Lr 9) 2 max Lr (h,,, g cs
(54)
where
m log m log log 6
Es = 86m - 77x))111 + 8m²3 sup δ h'EH
(55)
Proof. Define empirical counterpart to Lp, the population-level utility of classifier for empirically estimated gradient, as
Es (h', ==)== = C(g' VV SS),
(56)
and the (population-level) optimal classifier g" e argmax,' Lp Lp(h, g') for the exact gradient, whose existence is guaranteed by Theorem 4.1 Then write
max (r g (h, 9) Ee(h,gg) Lr(h,g)
= -rr(h.g') Es(h.g) g') Es g") sh. g) + Es(h, Lr(h, (57)
Now we turn to bounding each of these terms. For the second, we get
Es(h, g") (h C(g", P). C(g, G
I max C(g', PP G C(ğ. P) G < 2m- max || [=(()(()) - (x))))) [1], (58)
where the last step used that 9 is the empirical maximizer of linear measure corresponding to G, in order to apply Lemma D.I Now, if v is Ê-Lipschitz w.r.t. the -ormm, then
VC': (V((C- C' < [['|l2
(59)
Let j E [m], and applying (59 to C' Č for which C = 0 for all i j. and C = 0.25 1, we get
0.25G <
G ||11 < 48.
(60)
As this holds for all the upper bound turns into
Sshh.g g²) Ls(h,g) 86m F (I])()))) 71x)]11
(61)
22

---

Published as a conference paper at ICLR 2024
To bound the other two terms, use Hölder's inequality:
Ep (h. - Ls C'''. VC(C P)) - C(g', VV((Ch,
C V
- [V - - ,
(Hölder)
ml (C,
(Normalization of C) (B-smoothness)
S P)) S(,,
S 4m? - S)lo
S 4m²3 3 sup
CC P)) CCh',
(62)
W'ER
The same argument can be employed to bound the third term. Thus, applying Lemma D.2 with probability at least 5
we get
gh) Lp Ø) 86m E. (ll((a)):)) 7(x))111++
(1
8m2 B sup Ô NEW
-log log n log ò
(63)
n
D.4 CONSISTENCY OF DXED-STEEPSSHEDDULE FRANK-WOL.R
Theorem 5.1 (Consistency of molkk)..)..... Assume the utility function :: 10.1] mm
R
concave over Cr, L-Liipscitzz and 3-smooth w.rl. the I-norm. Let S= S2) be sumple drawn ii.dd from P. Further, let ij be a label probability estimator learned from S1, and h be the classifier obtained after en iterations. Then, for any 8 (0, 1), with probability of least over draws of S,
m log m log n log
8,
AV (h55 & O0Eaalllme (1))) Ô m²
+
(9)
n
Kn + 2
Proof. Define a curvature constant for the loss v
C== =
sup C Cl.
72
CCC)
sup
2 7
2
22 - -c'1R) = sup
|C22 C² 12 <
43m,
(64)
C C0.
CHECEEC
and let be defined as in Lemma D.3 Set 6apx = (t + 1)es/ /Cp and ht as in Algorithm
Let f
be the classifier implicitly dehned in iteration i, that is,
(65)
1=1
For sis I, we can apply Lemma D.3 to fi1 and ht', which gives
c(i V (C(2-1 P)) max C(g', V (c(fi ,P) - -65
max C. Vei C( 10 .P) (p))
Es max C. Veb (c(f-1, .P)) Es
CECP
CECP
max C. Voh V (((f-1,P))) OеC₁
2
Sapx
++1
2 max C. (w(c((--1,)) CEe
dapx
2
C
(66)
+ 1
23

---

Published as a conference paper at ICLR 2024
As consider, for the proof, Frank-Wolfs implementation with fixed step schedule T-1. the confusion tensors are related through
2 c(f.P) (1 i+1 c(f--
.P) i+1 C)h'P).
(67)
With results (66) and (7). we now have the exact same situation as Narasimhan et al. (2013) Proof
of Theorem 16). In particular, application of Jaggi (0033 Theorem 1) gives the desired result. O
E LABEL DEPENDENCE AND OPTIMIZATION OF MACRO-AT-K METRICS
The "budgeted-at-#" constraint couples the label-wise binary problems, resulting in their inability to be independently optimized. To demonstrate this coupling effect, we present simple example. We consider the macro Jaccard similarity, defined below, and assume budget 2:
C₁ ==1 C + C. C%
Jaccard C(h)) = m Σ
(68)
Let us consider two simple distributions, both with two different instances æ of equal probability and three labels:
Distribution A:
Distribution B:
P(æ) x
771 (x)
77(x))
773 (z)
P(x) x
7/1 (a)
72(x))
773 (x)
31
0.5
0.4
0.2
0.6
31
0.5
0.4
0.2
0.6
222
0.5
0.8
0.4
0.4
32/2
0.5
0.8
0.4
0.8
Notice that both distributions only differ on the marginal conditional probability of the third label of the second instance 22 (1z())2). We find the optimal randomized classifiers for both distributions:
Optimal h (az) for distribution A:
Optimal (ar) for distribution B:
[z)
T2(#)
11(x)
T2(
T3(x)
01
1.0
0.0
1.0
0.0
1.0
1.0
1.0
1.0
0.0
32
1.0
0.0
1.0
1/ Iaccard
(C(h A)) Rs 0.453962
Jaceard
C(hig B)) Rs 0.471423
We can notice that despite changing only one marginal conditional probability, the optimal solution is different on the other instance for the two other labels. it were possible to find solution for each label separately, the change in the distribution on one label would not affect the order of other labels, as it happened in the above example.
F EXPERIMENTAL SETUP
TRAINING AND SELECTION OF MARGINAL PROBABILITY ESTIMATORS
In our experiments, use two types of models for the estimation of marginal conditional probabilities of labels (z))
1. For MEDIAMILLL, FLICKR, and RCVIX datasets, we use multi-layer fully connected neural network (ranging from to 3 layers with hidden layer size from (128 to 2048) implemented in Pytorch Paszke al. (011)). We perform a search for the best hyper-parameters (number and size of layers, learning rate, number of epochs) using a validation set created from the train set for each loss used (binary cross-entropy, focal and asymmetric loss). Then, the model is retrained on the whole training set. We use Adam optimizer (Kingma & Ba 2013). For Focal and Asymmetric loss, we use default parameters suggested by the authors in (Ridnik et al. 2021)
24

---

Published as a conference paper at ICLR 2024
Table 3: Mean results with standard deviation of different inference strategies on measure calculated at {3, 5, ] Notation: P --rrecision, R-recall, FI-Fl-measure. The green color indicates cells in which the strategy matches the metric. The best results are bold and the second best are italic.
Incaaze M5
Marro
Marro 10l0
MERDIAHILA
6:05
Turk Ka
57.39
42.51
15.23
11.54
11354
47.58
56.53
1210
12.6
12.15
Ton Kewr"
36.22 2008
11.32
14.43 300
10.10-3
9:4
35400.30
1080
1300
15.33
2000
10.55 1000
28.45 2800
56
2300
0.5
26.35
12:26
Kefnos
and
1000
5300
1100
1300
Tur Kun-a
45.
11133
4.58
4.90
0157
8850
7.46
76.55
10.73
15.55
Maran-P..
794 z21
6.13
11:33 mm0
2:87
6.9 mm
8.9 20
17.29
54P cam
17:38
11:34
5.7125
Macan-8- 8ks
5.7 L800
3,67
18.83
5.31
7:56 1000
0.55
1080
5.1 1300
26.58
6.31 1800
1:31 1500
1742
1300
10.33
19.24
1100
5,35 LC.6
Maran-Fille
11:00200
11.05
2011
15.47
24/44
1117
13:31
(155
21:00-00m
15.05
2111
00
1761
16:1
Too-
2304 LE0
56.96
23.04 JooK
15:41
16.36
25.35
.0X
16.59 Lum
06.81
1080
17.12
1300
47.00
100
23.40
1000
18:16 1800
77.35
1300
10.72
-630
55:37
1606
11:24 LC.6
Tee-Kin""
44:25
1000
1300
1810
13:00
Tur Keatt
13.57
56.17
19:00
16.04
16:76
65.31
15
76.60
0.74
Tur Kefacca
15:27 -0-
16.07
76-17
(5:00
mm 65.0
mm
0300
24
76.01
mmm
11:77
11:40
18:24
Top-KiCuns
55,65
23.09
57.00
36.12
1000
1300
160
1300
10/1
55.30
LC.
Macan-Py
468 s8u
11:01
(9654
-120 6:03
1:06
5.56
22.15
41:76
41:4
1470
10.57
2.55 681
11:26
6310
37.59
10:65
1.50 km
Macan-lle Rew
15.18 =a1
16.02
11:38 u83
66.99
12:27
12:11
47.48
300
1358
4.30
2585 1000
10.12
1.20
4000
00
8.30
0.33
6486
15,77 +1.0
Flye
17.99
5.0
77
16.55
RCVIX
2,83
Tan Keanne
65.99 zar 60.11 -00
18.59 sni
12.76
1589
15
mmm
14:00
5301
17
13.02 5000
11:33 2m00
87
16 =300
1043
36.05
12:53
Tow
10200200 zare
11.17 -00
19:97 mm0
8.00:3
0:80
11
mm
05
28m
mm
11:45
15:01
Te Ke
P1.14 L818 13.00
14.40
5.44
1.45
99.13
1080
12.7
8.52
1540
57.85
+330
064
15.10
1023
15.36-am
S1.11
221.21 m0:0
24:00
16.1
22:00-300
5.13 exe
5.11
-8:21
30.51
Maram-llss
13:28 ca1
11:94
[8.50
18:41
11:45
51:12
50.11
mm
1215 0300
24,89
12.15
51.75
08 71
mmm
10.76
34,6
stod
12:41
sau
61.23
Lean
21.45
10.37
pm
71.6 ev8
21.06 0111
31.36
2010
22.19
18.12
AMAZONCAT
Tank
11,29
59.29
125:2 man
13 eum
16.22
54.54
smn 45.05 eaxo 12.02
23:00
sum
35.16
15.18
=310
16.52
51.09
40..9
Ton Keakk
=::0 40:76 --x 9021
2000 79
2mm 45.00 2000
1111
130
67.40
zam
4485 2800 53 23m
46.50 23m
31.31
18.00
52.20
13:0
50.13
00:0
65.53 :m3
17.15 20
22.56
sam
Macan-B-
47.00 Le:2 35.57
15647 Jus 23.74 1a06 23.62 1567
41.04 1201 50.74 Jau 355A5
1808
27.45 1300
30.23 1800
30/90 180
59:67
1100 55,27
-033 20:99
34.51 11:5
Maram-lla
25 =5
=
mmn 89.77
K867
=30
pam
16.17
2910
66.08 2318
2810
57.91
11:03
73.8
Macan-Fle
68.99 L03) 51.40
5635 Jap-
3416
1800
2323
55.75
1300
63.00
1800
$6A2
18X
36.40
41.2
186
55,30
18/0
78.34
1300
5467
.031
36:33
1091
2. For AMAZONCAT dataset, we use probabilistic label tree (PLT) with LIBLINEAR models trained with Ly-regularized logistic loss (Fan et al. 2008). We make it sparse by truncating all the weights whose absolute value is above the threshold of 0.01, as introduced in (Babbar & Schölkopf (017). to reduce the model size and inference time. Use the implementation provided in NAPKINXC library (Jasinska- Kobuus et al. 2020) and use the library's s default parameters.
F.2 SPARSE MARGINALS IN BRANK-WOLFF ALGORITHM
Materializing the (z)) for all instances in the form of dense matrix requires a considerable amount of memory for datasets like AMAZONCAT (over 58 Gb using 32-bit bit floats). Because of that, we instead use sparse matrix in compressed sparse row (CSR) format with only top-k' values of marginals kept for each instance, where k < k' < m. All the other marginals are being treated as zeros. In CSR format, the row vectors are represented as a list of tuples a. := {(index, value). value * 0} and allow for efficient element- wise multiplication between both dense and sparse vectors needed in Frank-Wore Wolfe procedure. By using the sparse matrix of marginals with exactly k' non-zero values, we effectively reduce the complexity of one iteration from O ((nk)) instead of (( We use k' 200 for both RCVI3 1X and AMAZONCAT datasets. We found that using ke > 100 has no negative impact on predictive performance compared to using full dense matrix.
F.3 HARDWARE
All the experiments were conducted on a workstation with 64 GB of RAM and Nvidia V100 16Gb GPU. However, the experiments can be also reproduced with smaller amount of memory.
G EXTENDED RESULTS
In this section, include extended results of our empirical experiments. Here, we include tables with standard deviations. In Table 3] we present the results of the main experiment from |Section 6 with standard deviations. In Table 4. present the results on balanced accuracy, for which the
25

---

Published as a conference paper at ICLR 2024
Table 4: Comparison of two classifiers for macro-balanced aceuracy calculated at (3,5, 10) a closed form classifier MMACRO-BAPRIO and 2) classifier found using Frank-Wolfe algorithm (MACRO- BArw). The green color indicates cells in which the strategy matches the metric.
Inference stralogy
Instance 93
Macro
Instunce
Macro
Instance #10
Macro @
MEDIAMILL
MACRO- BArRo-BAsuna BAvana
7.43
4.41
10.70
19.86
5.65
58.54
9.05
9.34
11.30
26.54
7.25
60.98
11.53
26.05
10.65
39.33
9.88
65.15
MACHO-
7.45
4.41
9,05
9.34
26.54
11.53
26,05
65.15
FLICKR
MACHO- BAqum
16.33
39:10
17.56
45.50
22.31
72.10
12.35
48.20
13.96
53.84
19:77
75,80
7.98
61.19
9.57
64.67
15.09
79.97
MACHO- BAo
16.33
39.10
17.56
45.50
22.31
72.10
12.35
48.20
13.96
53.84
19.77
75,80
1.98
61.19
9.57
64.07
15.09
79.97
RCVIX
MACRO- BAvas
44.15
46.00
14.62
18.40
12.01
59.17
34.71
56.27
13.18
24
12.78
62.37
24.07
70.61
10.77
34.64
12.47
67.17
MACRO-BA, BAi
44.15
46.00
1462
18:40
12.01
59.17
34.71
56.27
13.18
24.86
12.78
62.37
24.07
70.61
10.77
34.64
12.47
67.17
AMAZONCAT
MACRO-BA BA
47.02
33.74
35.27
39.28
45.74
26.77
27.45
61.62
MACHO- BAw
47,02
33.74
35:27
61.70
40.42
39.28
45.74
26.77
67.95
83.97
27.45
61.62
17.36
73.37
24.55
86.66
Table 5: Means with standard deviations of running times and numbers of iterations performed by the Frank- Wolfe algorithm for different objective measures calculated 3, 5, 10. Notation: T-tootl time in seconds, number of iterations.
Inference stratogy
T33 and
103 Asid
T05 Asıd
/ 1sid
T 1isdd
1@10 1isd
MEDIAMILL
MACRO-Pew
1.21 10.19
9.90
1.15
8.70 11.27
1.57
10.70 22:00
MACRO-Rew
0.52 1000
3.00 10.00
0.53 10000
3,00 1000
0.58 10.10
3.00 10.00
MACRO-Fla>
1.44 20.14
10:40
+1.02
1.42 +0:27
8,90 #1.14
9.10 #291
FLICKR
MACRO-Prw
2.37 10:8
8.30 11.35
2.40 10:52
10:20
1.447
2.44 1042
9,40 1:100
MACRO-Rew
1.17 20.19
3.00 10.00
1.07 10.00
3,00 10000
1.17 10.15
3.00 40000
MACRO- Flow
1.39 #0.19
4.60
2.00 #0.34
7.60 +1.43
2.01
7.20
RCVIX
MACRO-Pex
25.33 2:113
7.40 10.92
23:08 1.50
6.70 10.46
20.04 1:338
5.20 1:0.0
MACRO-Rex Rew
20.70 #1.95
+0.00
21.94
#2249
+0.00
22.41
6.00 #0,00
MACRO- Fl Fl.e
15.05 #1.0
4.20 +0.40
15:03
+1.00
4.00 1000
15.75 #1.20
4.00 20:00
AMAZONCAT
MACRO-Pew
21.97 #2.47
4.50 +0.50
20.80 #3.91
3,80 +0.40
23.44 #2.78
4.40 #0.66
MACRO Row
30.89 #3.86
6.00
29.70 #:333
6.00 #0.00
32.25
6.00 20000
MACRO-FI bow
15.76 20.53
3.00 10000
18.18 18
1:144
3.40 109:
34.02 1684
6.60 1:100
introduced -ollfe algorithm retrieves the same solution as the closed form classifier described in Section 4] similarly to the case with macro-recall. In Table 5] we present the mean number of iterations and running time of the Frank- Wolfe algorithm for different objective measures, as well as different values of k. The values were calculated based on 10 runs of the algorithm. We use the same value of stopping condition = 0.001 for all the experiments. The Frank Wolfe algorithm requires only a small number of 3- iterations, and thanks to the usage of sparse matrices as described in [Section F.2 needs, most cases, less than a minute even for larger benchmark datasets like AMAZONCAT. In further subsections, we also present the results for different splits and initialization strategies for the Frank-Waon Wolfe algorithm.
G.1 IMPACT OF SPLITTING STRATEGY IN THE WOLFE ALGORITHM
In this experiment, we test different ratios of splitting training datasets into the sets used for training the 7) estimator and estimating confusion matrix C' (50/50 or 75/25 split), as well variant where we use the whole training set for both training the estimator and estimating C (100/100 split). The initial classifier hd is initialized with the top-k î classifier for all the experiments here. We present the result of this comparison in Table 6) The results suggest that more data used for training is beneficial for the quality of the final randomized classifier.
26

---

Published as a conference paper at ICLR 2024
Table 6: Comparison of different splitting strategies for the Frank- Wolfe algorithm on measures
calculated at {3, 5, 10}. Notation: P precision, R R-recall F1 F1--a-meauure. The green color indicates cells in which the strategy matches the metric.
Inference stralegy
Instance
Macro
Macro
Macro R
R
F1
R
F1
R
MACRO-Pie Pro spi
7.76
5.87
15.41
6.65
3.89
9.22
11.35
13.60
11.27
5.38
3,56
8.58
15.22
19.08
5.44
MACKO-Fry + 1825
6.74
5.03
9.00
10.70
13.79
4.54
10.56
14.47
MACKO-Pre
yit
7.94
6.13
19.33
6.06
2.87
6.99
8.96
17.29
8.79
3.17
6.02
14.14
17.38
17.24
5.23
MACRO-RI, RiN
5.14
3.05
6.04
6.06
5.93
6.75
14.26
9:97
MACKO-Rre -Rx---------
4.41
2.52
7.89
15.30
4.30
4.93
4.75
7.02
21.83
5:40
5.83
12.01
9.40
34.91
7.37
MACRO-RA RRv 300000
6.37
3.67
8.81
19.82
5.31
7.38
7.25
8.91
26.50
6.71
8.31
17.42
10.53
39.24
8.85
MACRO-F1y Fly 2030
45.25
33.24
14.19
10:67
90
41.36
49.55
12.97
15.24
12.59
26.
62. 90
12. 19
23.43
13:50
MACRO-F),, 7505
43.07
31.33
13.80
10.65
10.91
40.06
64
12.61
15.06
12.49
27.84
64.08
11.86
23.77
13.64
MACRO-FF Fl, 100000 ppi
46.77
34.21
15.61
11.16
12.35
43.48
51.65
14.93
14.98
13.69
27.92
64.11
12.14
28.42
14.63
FLICKR
MACRO-Pre
5.66
13.32
37.66
12.69
12.84
4.32
17.11
37,80
15.02
13.99
2.26
17:43
36.93
16.40
12.71
MACRO-Pre e 1825 go
6.73
16.18
39.33
16:33
3.84
15.28
38.07
2.22
17:40
37.73
17.95
MACRO-PA, Prw 3002000 gt
4.65
11.49
39.34
6.63
8.06
5.66
22.75
41.74
9.70
10.57
2.83
22:26
37.59
10.68
8.50
MACRO-R,, R,v qui
14,90
35.66
18.30
42.72
21
11.41
44.65
14.71
51,16
19.80
7.50
57 /6
9.97
62.17
15.34
MACRO-B Riv 1825 qut
15.34
36.81
17.65
.3.89
21
11.62
45.44
13.78
52.03
19.10
7.58
58.07
9.33
63.70
14.50
MACRO-RA Ryv
16.14
38.62
17.58
45.50
22.27
12.17
47.48
13:98
53.83
19.72
7.89
60.42
9.57
64.66
15.07
MACRO-FI Fl. qut
19.06
45.28
31.30
31.55
28:63
12:24
47.51
31.10
34.24
29.02
6.18
47.5 82
31.37
35.89
28.73
MACRO-FIe Flr wet
17.33
41.28
31.93
32.64
29.65
11,46
44.54
31,62
34.78
29.74
5,86
45.22
30.52
38.03
29.37
MACRO-FI mnconn yiu
18.21
43.06
34.89
29:51
29.41
11.78
45.91
34.68
30.87
7.14
55.21
34.00
33.17
Table 7: Comparison of different initialization strategies for the Frank-Wolfe algorithm on measures calculated at {3, 5, 10} Notation: P P-rrecision, R-recall, F1-F1 -1-easure The green color indicates cells which the strategy matches the metric.
Inference stratogy
Instance Ø3
Macro @3
Instance 65
Macro @5
Instance @ 10 P
Macro @10
Fl
R
P
R
MEDIAMILL
MACRO-PP PyN aptiat
7.49
5.35
16.54
9.43
3.54
9.61
11.18
17.10
11.30
4.37
5.66
12.90
17.05
17.31
5.72
MACRO-Prx miii
6.06
2.87
6.99
8.96
17.29
8.79
3.17
MACRU-R Ryw inskii
6.37
3.67
8.81
19.82
5.31
7.38
7.25
8.91
26.50
6.71
8.31
17.42
10.53
39.24
8.85
MACRO-R R.. tit
6.37
5.31
7.38
7:25
8.91
26.50
6.71
17.42
10.53
39.24
MACRO-FF Flax
46.77
34.21
15.61
11.16
12.35
43.48
51.65
14.93
14.98
13.69
27.92
64.11
12.14
28.42
14.63
MACRO-FF CERR---Fire.... Flim bel
45.20
33.05
15:42
11.17
12.21
43.57
51.60
15.20
15.05
13.82
28.12
64.23
13.93
23.32
14.81
FLICKR
MACRO- Pas: sok
4.46
10.84
40.41
7.87
9.33
2.53
10.44
38.F1
7.84
8.04
1.75
13.99
37.381
10.69
8.42
MACRO-Pre miiis
8.06
5.66
22.75
41.74
10.57
22.26
37.59
MACRU-R Ryw inn
16.14
38.62
17.58
45.50
22. 27
12.17
47.48
13.98
53.83
19.72
7.89
60.442
9.57
64.66
15.07
MACRO-R Row tow-+iaa tot
38.62
17.58
22.27
12.17
47.48
13.98
53.83
19.72
9.57
MACRO- Flax top
18.21
43.06
34.89
29.51
29.41
11.78
45.91
34
30.87
29.45
7.14
55.21
34,00
33.17
29.11
MACRO-FF SIR0O-Flye, Flow and
17.59
41.60
35.28
29.28
29.43
12.22
47.31
34.13
32,70
29.43
5.92
45.77
34.55
33.06
29.02
RCVIX
MACRO-PP P.x: tipk
32.66
34.63
20.70
3.62
3.77
31.39
52. $2.39
21.41
5,80
6.58
16.15
49.66
21.45
7.84
5.96
MACRO-P. PaN maiin
5.84
29.40
49.81
21.69
5.72
19.45
MACRU-R, Ryw inpi
43.28
44.99
14.56
18:4
11.95
34.15
55
13:15
24.89
12.73
23.78
69.71
10.76
34.66
12.44
MACRO-B R.,. rad
11.95
34.15
55.24
13.15
24.89
12.73
23.78
10.76
34.66
MACRO- Flas sppiin
58.14
61.31
21,58
10.36
12.03
44.29
71.93
22.37
12.26
13.65
27.96
80.26
22.53
13.76
15.20
MACRO-FF Flax andias.
58.20
61.22
21.45
10.37
12.09
44.42
71.86
21.95
12.25
13.68
27.26
78.88
22.10
14.86
15.12
27

---

Published as a conference paper at ICLR 2024
MEDIANILL
MEDIANIELL
A-
Par
23
Kag
25-
Miarro F143
Maazo-F103
PLRUR
PLacIo
24
P03
24-
18.
RCVIX
RCVIX
RCVIX
32-
Pun Rauns
60
Figure 1: Comparison of the baseline algorithms with the PU inference with mixed objectives for
E (3,5, 103. The green line shows the results for different interpolations between two measures.
G.2 TOP-K VS RANDOM-INTIAALIAATI
In this experiment, we investigate the impact of the initialization strategy in the Frank Wolfe algorithm on the results. They consider two strategies for the initialization of initial classifier h.; one initialize the classifier that weights îJ by a random positive number (rnd init.), second strategy initialize ho with the top-k îi classifier (top-k init.). For all the experiments here, use the same dataset for both training the 17 estimator and estimating confusion tensor C (100/100). We present the result of this comparison in Table 7] The results show that the initialization strategy has impact the results for MACRO-Pa4 Pew variant of the algorithm. However it is not clear from the results which initialization variant is better and in which circumstances for MACRO-PPw.
G.3 RESULTS WITH MIXED UTILITIES
It can be noticed in the presented results that the optimization of macro-measune comes with the cost of sigmificant drop performance on instance wise measures, which in some cases may not be acceptable. To achieve the desired trade-off between tail and head label performance, one can optimize a mixed utility that is linear combination of instance- wise measures and selected macro-measures As example, we present the results for such mixed utility that is combination of instance-wise precision@k &k with macro Fl F1-measure@k:
(CC (= X)V (C) + AU Ma (C)
m
Σ (1 A) Insttanee pl (C*) + Macro (Cr)
(69)
In Figure we present the plots with results on two combined measures for different values of Once again, the presented results the mean values over 10 runs of the inference. The plots show that the iistancevss-mace curve has a nice concave shape that dominates simple baselines in most cases. In particular, we can initially improve macro-measures significantly with only minor drop in
28

---

Published as a conference paper at ICLR 2024
and only if we want to optimize even more strongly for macro-measures. get larger drops in instance-wist measures. A particularly notable feature of the plug-in approach is that the curves in the figure are cheap to produce since there is requirement for expensive re- re-training of the entire architecture, so one can easily select an optimal interpolation constant according to some criteria, such as maximum decrease of instance- wise performance.
H CONFUSION TENSOR MEASURES
In this section, we will take closer look at the definitions of confusion tensor metrics, and provide some structural results. First, let us recall the definitions from the main text:
Definition 3.1 (Binary Confusion Matrix Measure). Let C = {cc E [0, 1|2x2 ||C|| = 1 be the
set of all possible binary confusion matrices, and C, C' E C. Then we say that C" is at least as good as C, C' ž C, if there exists constants é1, 62 such that
C' Coo +
Coi -1
(2)
C1 - 62
Cn 11+
i.e. C' can be generated C by turning some false positives to true negatives and negatives to true posilives. A function :h: C - [0, 1] is called a binary confusion matrix measure (Singh & Khim 2022) if it respects that ordering, i.e., for C' 2 C we have vp(C") 2 (((C).
Definition 3.2 (Confusion Tensor Measure). For given number of labels m E N, and two confusion tensors C, C' E C we say that C' is at least as good as C, C' ž C, if for all labels j E [m] it holds that C 2 C³. A function V C.
[0. 1] is called confusion tensor measure if respects
this ordering, i.e., for C' > C we have 4/ (CC) 2 (C).
Our first claim is that these form partial orders. Lemma H.1 (Partial order of confusion matrices). The relation ł introduced in Definition 3. forms a partial order on C. Similarly, ž from Definition 3.2) forms a partial order on C
Proof. We start with the binary case. We need to show reflexivity, antisymmetry, and transitivity: Reflexivity: By choosing 6 = 62 O, we see that C' = C. Antisymmetry: Let C C" and C' z C. This implies = = 0, meaning C' = C". Transitivity:
C C" with coefficients e1, E2, and C" > O" with é éz then C > C" by choosing e1 + é 62 + 62
The multi-label case follows directly, as just an m-fold Cartesian product of the binary case.
Next, show systematic way of turning binary confusion matrix measures into confusion tensor measures, using either micro- or macro-aggregation.
Definition H.2 (Aggregation function). For N, call function f: [0. 1J" [0, 1] an
aggregation function if it is nondecreasing in each of its arguments. Theorem H.3 MMacro-Aggregalin
Let WI,..., be collection of binary confusion matrix funccionnnn Then the macro- aggroggggre
measures, and o: [0, 1]Th - [0, 1] be an aggregation function.
v (C) = σ(v1 (C.)...., .....,, t'm (C"))))
(70)
a confusion tensor measure.
Proof. Let C' 2 C. Then, for all labels j. C > C³, which implies dh (C 2 (C"). As is nondecreasing in all of its arguments, this implies & (C') 2 (C),, concluding the proof.
Theorem H.4 Micro-Averaging Let be a binary confusion matrix measures, and o be a lincar aggregation function. Define the averaged confusion matrix by applying aggregation to each entry separately,
(
Cma) m
Cm
C dC))
(71)
@ 10:
@C CT))
Then the micro-average
VCC) = (d(C)),
(72)
confusion tensor measure.
29

---

Published as a conference paper at ICLR 2024
Proof. Let C' C. Then, for all labels j, C > C³ i.e., there exists a collection
(G.62)... (Ef", eTr) which transform C into C'. Denote C σ (C'), and similarly C = (C).
Due to the linearity of Φ, we have
Coo = Coo') = o(Cho + + t....., + cr))
= 6CCo C00 + .l....... +c7)
Similar calculations can be done for the other components. This implies that
0o + + +(1)
Co o(e) +(
C = Cio Cio
(73)
+
+E Cn ( +
i... C² > C, and thus (CC) 2 (C).
If the aggregation function is chosen to be the arithmetic mean, the two cases above reduce regular macro- and micro-averaging. This justifies our choice of (3) in the main paper, proving that this indeed does result in an admissible confusion tensor metric.
Note that for migro-aggregation, we had to be much more strict in what aggregation functions to admit, essentially limiting to weighted arithmetic mean, because we need to ensure that the component-wise averaging of confusion matrices results in matrices that are comparable using the partial order.
Finally, we can also provide the following structural result:
Theorem H.5. Let vp I,.... L n be collection of confusion tensor losses, and an aggregation function. Then
v (C) (VV
(74)
a confusion tensor loss.
Proof. Let C' C, then V (C()") 2 (C) Thus, by monotonicity of o, we get (CC) 2 (C).
This latter result implies that, e.g., calculating the harmonic mean of macro-precision and macro-recall also a confusion tensor loss.
I SENSITIVITY OF MACRO-AT-K METRICS TO TAIL LABELS
The experiment described in this section has been motivated by similar one given in (Wei & Li 2019). In Table 8 we compare different metrics for budgeted at l predictions. We train a probabilistic label tree (PLT) model on the full AMAZONCAT dataset (McAuley al., al., 2015) and on reduced version with the 1000 most popular labels only. The test is performed for both models on the full set of labels. The standard metrics are only slightly perturbed by reducing the label space to the head labels. This holds even for rropensity-scores precision (Jain et al., 2016). a popular measure for evaluating tail labels in extreme multi-label classification, which decreases by just I 1%-20% despite discarding over 90% of the label space. In contrast, the macro-at-k measures decrease between 60% and 90% if tail labels are ignored.
Table 8: Performance measures (%) on AmazonCat- 13k a classifier trained on the full set of labels and classifier trained with only Ik head labels.
Metric
full labels @3
head labels @ (diff)
@
@5
@ (diff.
@5 (diff.)
Precision
93.03
78.51
63.74
93.08 (+0.05%)
76.42 (-2.66%)
58.21 (.8.67%)
nDCG
93.03
87.25
85.35
93.08 (40.05%)
85.75 (-171%)
80.91 (-5.19%)
PS-Precision
49.76
62.63
70.35
49.07 (-1.99%
57.71 (-7.84%)
$7.41 (-18.40%)
Macro-Prcoee Precision
13.28
32.65
44.16
4.31 (467.54%)
5.28 (-83.825)
4.32 (-90.21%)
Macro-Recall
1.38
11.06
30.57
0.47 (65.61%)
2.69 (-75.71%)
4.10 (.86.59%)
Macro-FI
2.26
14.67
32.84
0.74 (67.37%)
3.10 (-88888))
3.77 (88.5115)
30


Reviews:
[{"review_id": "oFJtLAbRgM", "reviewer": "Reviewer_gTuy", "summary": "Paper presents an approach for multi-label classification with a budget using so called macro-at-k metrics which are linearly decomposable into a sum of binary classification utilities, and could be useful in the case of extreme classification with large number of imbalanced labels. This leads to challenging optimisation problem which is tackled using Frank-Wolfe method on label-wise confusion matrices. Theoretical underpinnings of producing consistent classifier are analysed, and performance improvement of macro-at-k against top-k strategies are shown on four different multi-label benchmark datasets with different number of labels and label distribution.", "questions": "- What would be the conclusions or \"rule of thumb\" of selecting particular heuristics from the practitioners' point of view for certain applications or multi-label classification problem?\n- What are the computational costs of proposed approach and how these relate  the size of k?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposal is theoretical sound, and detailed analysis of the properties of proposed metrics as well as how to build the consistent classifier based on confusion matrix measure and tensor representation, are shown. One practical solution for optimising the classifiers with proposed macro metrics, is derived. In empirical evaluations, four benchmark dataset with two base classifiers (MLP, Sparse linear model) are used to compare macro-at-k metrics with more straightforward top-k heuristics. In most cases, proposed approach could improve the precision and recall (and F1 score) in different k-budget levels, whereas on the largest dataset more simple baseline heuristics seems to work better. To my knowledge, the setting of considering budgeted macro-at-k in multi-label classification is novel, providing interesting approach and new knowledge to extreme multi-label problems.", "weaknesses": "Although paper shows good theoretical background and promising results, it lacks some of the detailed analysis and discussion of the results, especially in a broader sense. For instance, the discussion which of proposed metrics and macro-at-k approach should be chosen for different problems from practitioners perspectives, would strengthen the presentation. Also, manuscript is missing the analysis of computational complexity and computational times (of optimising the classifiers) and how these relate to size of the budget and other chosen parameters, as well as how these compare between different heuristics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "72ttiyu1rP", "reviewer": "Reviewer_XNZA", "summary": "This paper aims to find consistent algorithms for macro-at-k metrics, which is widely-used in many long-tailed multi-label classification problems. For the multi-label problem, the author shows such optimal classifiers can be derived by selecting top-k scoring labels based on an affine transformation of the marginal label probabilities (which is unknown in practice). They further presents a Frank-Wolfe algorithm to empirically find the optimal classifiers.", "questions": "- Q1: From the results of Table 2, the proposed method seems to greatly sacrifice instance-wise metrics to trade for gains in macro-average metrics. Is it possible for the proposed method to optimize a interpolated version of the objective that flexibly control the performance tradeoff between instance-wise metrics and macro-averaged metrics?\n\n- Q2: Some baseline methods that claim to also perform good on tail-labels are not discussed in related work, or compared in the experiment section. For example [1] and [2], to name just a few.\n\n- Q3: To improve the clarity of the proposed methods, the author may consider a toy synthetic dataset where data distributions are known, and show derivations of the proposed method, and verify the consistency property through simulations.\n\n- Q4: This submission also seems highly related to [3]. The author should discuss what's the difference, and compare it empirically.\n\n\n### Reference\n- [1] Menon et al. Long-Tail Learning via Logit Adjustment. ICLR 2020.\n- [2] Zhang et al. Long-tailed Extreme Multi-label Text Classification by the Retrieval of Generated Pseudo Label Descriptions. EACL 2023.\n- [3] Schultheis et al. Generalized test utilities for long-tail performance in extreme multi-label classification. NeurIPS 2023.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The technical derivation of this paper seems solid", "weaknesses": "- The writing of this paper is not easy to follow\n- Some baseline methods are missing in experiments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GYPSsyfHOp", "reviewer": "Reviewer_gGLD", "summary": "In the paper \"Consistent algorithms for multi-label classification with macro-at-k metrics\", the authors propose a framework of consistent multi-label learning algorithms for targeting macro-averaged metrics that are however budgeted for a k-subset of labels. The presented approach is based on the Frank-Wolfe algorithm and represents a principled extension towards multi-label classification with corresponding theoretical guarantees. An empirical study confirms the consistency with the targeted metric.", "questions": "- In https://doi.org/10.1007/s10994-021-06107-2 an @k metric interpolating between  Hamming and subset 0/1 loss is presented. To what extent would this relate to the considered @k-metrics in this paper?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Theoretical sound and underpinned approach for targeting macro-averaged at k metrics for multi-label classification.\n- An empirical study confirms the theoretical findings for four datasets and various metrics.\n- The overall presentation and language of the paper are very good.", "weaknesses": "- The empirical evaluation is relatively limited as only four datasets are considered. However, the main focus should remain on the theoretical findings here.\n- The theoretical results could be accompanied by an intuition to ease understanding of the results.\n- A comparison to something like binary relevance learning treating each label independently for demonstrating also empirically that these measures cannot be sufficiently tackled by such an approach would be desirable.\n\nminor:\np. 4 \"define in Table 1\"\np. 7 \"tensor measure measure\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cjj3jIPiDb", "reviewer": "Reviewer_s4jL", "summary": "The manuscript studies the problem of multi-label classification with budgeted (top-k) prediction metrics. The problem is very well established by now, and of much practical relevance. The paper is overall well motivated, well-written, and the contributions are par for the venue.\n\nI see two key contributions: 1) formalizing the empirical uitility maximation notion of '@k' metrics as that of optimizing over certain classes of confusion matrices, which decomposes over individual labels, but the optimization problem itself doesn't decompose easily because of the constraint that any classifier in the hypothesis space must output exactly k labels per instance; 2) deriving a neat form for the Bayes optimal, which yields interpretable closed form solutions for well-known metrics like recall and balanced accuracy; the most-general closed form solution is not very useful because it depends on the optimal values --- this observation by itself is not novel, as pretty much every paper that talks about non-decomposable losses for binary/multi-label classification problems (several of which are clearly cited in the paper) have developed similar results.  \n\nGiven the form of the Bayes optimal, Frank-Wolfe based algorithm for estimating the classifiers can be written down, following the work of Narasimhan et al. (2015). The empirical results demonstrate the effectiveness of the algorithm, and the consistency between optimal Bayes rule and the algorithmic convergence for certain measures, compared to several natural baselines.\n\nOverall, I like the work and I am inclined to accept. I've some minor concerns which I outline under 'weaknesses' section. It would be good to hear from the authors on these questions in their rebuttal.\n\n-- Post rebuttal --\nMost of my concerns are addressed. I'm more positive about the paper now.", "questions": "It would be good to address the points raised in the weaknesses section. And a note on the challenges or novelty in the proofs for Thm 4.4 over and above known work.\n\nIt's nice to see the consistency between Macro-R_prior and Macro-R_FW in the experiments for the recall metric. Did the authors also experiment with balanced accuracy in the experiments, where the closed form solution is also easy to compute?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clearly formulated problem -- it's non-trivial to set up the problem, and I really like the simplicity of the formulation in terms of confusion matrices/tensors, and the authors have done a great job of presenting notation-heavy material with careful development of ideas. \n2. Technical rigor -- the theoretical results are well established, and the supporting key lemmas are included in the main paper, which is very helpful.", "weaknesses": "The intro neatly positions the problem in the context of several closely-related work in this space, but I felt those connections didn't surface as much as I'd have loved to see in the main paper. \n\nFor instance, the first main result in Theorem 4.1. Just looking at the constants a_j and b_j, one can see, not surprisingly, that these are the same constants one would see for weighted binary classification problems, see Lemma 2 of https://www.jmlr.org/papers/volume18/15-226/15-226.pdf (of course, there's no top-k notion here in binary classification). It would be good to draw these connections, position the main results in the context of known results, and tease apart new observations and insights.\n\nSimilarly, with respect to, Theorem 4.4, it would be good to see some form of the result, say special cases, that can be dotted lined to known results for multi-label problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)