sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification
Authors: Gabriel Bénédict
Hendrik Vincent Koops
Daan Odijk
Maarten de Rijke
Venue: Accepted by TMLR ()

Abstract: Multilabel classification is the task of attributing multiple labels to examples via predictions. Current models formulate a reduction of the multilabel setting into either multiple binary classifications or multiclass classification, allowing for the use of existing loss functions (sigmoid, cross-entropy, logistic, etc.). These multilabel classification reductions do not accommodate for the prediction of varying numbers of labels per example. Moreover, the loss functions are distant estimates of the performance metrics. We propose sigmoidF1, a loss function that is an approximation of the F1 score that (i) is smooth and tractable for stochastic gradient descent, (ii) naturally approximates a multilabel metric, and (iii) estimates both label suitability and label counts. We show that any confusion matrix metric can be formulated with a smooth surrogate. We evaluate the proposed loss function on text and image datasets, and with a variety of metrics, to account for the complexity of multilabel classification evaluation. sigmoidF1 outperforms other loss functions on one text and two image datasets over several metrics. These results show the effectiveness of using inference-time metrics as loss functions for non-trivial classification problems like multilabel classification.

Published in Transactions on Machine Learning Research (09/2022)
sigmoidF1: A Smooth F1 Score Surrogate Loss for Multi- label Classification
Gabriel Bénédict
g. benedict@@w....
RTL Nederland V. University of Amsterdam
Hendrik Vincent Koops
Kooss
RTL Nederlond
Daan Odijk RTL Nederlond B.V.
daan, odin..ddkkk@rll..
Maarten de Rijke
m. m.derijke@uvo.rr ml
University of Amsterdam
Reviewed OpenReview: https: // openreview.net. forum? id= guSHaaD2uq
Abstract
Multilabel classification the task of attributing multiple labels to examples via predic- tions. Current models formulate reduction of the multilabel setting into either multiple binary classifications or multiclass classification, allowing for the use of existing loss func- tions (sigmoid, cross-entropy, logistic, etc.). These multilabel classification reductions do not accommodate for the prediction of varying numbers of labels per example. Morcover, the loss functions are distant estimales of the performance metrics. We propose sigmoidF1, loss function that is an approximation of the macro F1 score that (i) is smooth and tractable for stochastic gradient descent at training time, (ii) naturally approximates mul- tilabel metric, and (iii) estimates both label suitability and label counts. We show that any confusion matrix metric can be formulated with smooth surrogate. We evaluate the proposed loss function on text and image datasets, and with a variety of metrics, to account for the complexity multilabel classification evaluation. sigmoidF1 outperforms other loss functions on one lext and three image datasets over several metrics. These results show the effectiveness of using inference-time metrics as loss functions for non-trivial classification problems like multilabel classification.
1 Introduction
Many real-world classification problems are challenging because of unclear (or overlapping) class-boundaries, subjectivity issues, and disagreement between annotators. Multilabel learning tasks common, e.g,, docu-
ment and text classification often deal with multilabel problems (uull 1994 Bruno et al. 2013 Yang 2004
Blosseville et al. 1992), as do query classification (Kang & Kim 2003 Manning et al.) 2008 image classi-
fication (Shen et al. 2017 Xiao et al. 2010)) and product classification (Amoualian et al. 2020 Existing
optimization frameworks typically split the task into known problems and sum over existing losses Σ LMC.
with LMC any multiclass classification loss oftentimes variations of the cross-entropy or logistic loss. Wyd- much et al. (2018) define these frameworks malttilabel reduction techniques; MMeon et al. (2019) put emphasis two: One- Vel (OVA) T and Pick-All-Labels (PAL). For example, if C is the number of possible classes, OVA and PAL reformulate the multilabel problem to C' binary classification and C' multi-
class classification problems, respectively (see Section 2.3 These methods assume that, for one example,
This was already described in Dembrsyríski et
2010 and further formalized in Wydmmch
2018

---

Published in Transactions on Machine Learning Research (09/2022)
classification head
sigmoidF1
smoothly bounded predictions
groundIruth labels
hidden layer
logits
aigmold
Drama
Western
pretrained neural network
/ -
Horer
L F
Comedy
2001
(frozen weighta)
Science-fiction
Romance
s(y)
y
E 20)
E (0.
{00 13
Experimental setup for sigmoidF1 loss function for multilabel classification. Here, movie poster image is fed to pre-trained network with custom classification head that outputs logits (i.e., unbounded values) for each class (i.c., movie genre). At training time, sigmoid function forces logits towards either -1 or 1, respectively negative and positive predictions (illustrated by the darker colors). Confusion matrix metrics and macro Fl can then subsequently be computed. Here, S(horrrar) is close to 1, but the ground truth data claims that 2001: a space odyssey is not a horror movie; this approximately corresponds false positive. Note that - is computed over whole batch training time as a macro mcasure with the formulas in Sections 4.3 and 4.4 With this setup, one can optimize directly for the metric of interest at training time. Our image and text classification tasks below show improved results when compared to existing losses.
Figure 1:
label probabilities (a.k.a. Bayes Optimal Classifier (Dembczyíski al. 2010)) are marginally independent of other label probabilities. Menon et. al.) show iaall and empirically that reduction methods (OVA and PAL) can optimize for precision recall, but not for both precision and recall at once. More generally, shortcoming shared by OVA and PAL is their reliance on the binary multiclass classification setting and the lack of pure multilabel approach - inspired by binary classification literature (see most recently (Gai et al. 2019) and their F1 surrogate loss functions on 3-layer neural networks). We are not aware of a metric surrogate loss function that deals with multilabel classification in a modern deep learning setting in single task. Figure illustrates our approach with concrete example of classifying a movie poster into movie genres with single loss function: sigmoidF1.
Proposed solution to multilabel problems. We propose loss function 4F; that (i) naturally approx- imates the macro Fl classification metric (see Table 3) (ii) estimates label probabilities and label counts (soe Eq. 7 and (fii) is decomposable for stochastic gradient descent at training time (see Scction 4.1 and Figure Our proposed solution is to minimize surrogate of the Fl metric as loss. Strictly speaking, F1, where F1 smooth version of F1. Using a metric as a loss function unpopular for metrics that require a form of thresholding (e.g., counting the number of true positives), as minimizing step loss function (a.k.a. 0-1 loss) is intractable. The soft margin for support vector machines is early example, where the intractability of the direct 0-1 loss optimization is overcome with the hinge loss (Cortes) & Vapnik 1995). We resolve this by approximating the step function by a sigmoid curve (see Figure
we minimize
Main contributions. We introduce sigmoidF1, an Fl score surrogate, with a sigmoid function acting as surrogate thresholding step function. sigmoidF1 allows for the use of the F1 metric that simultancously optimizes for label prediction and label counts in single task. sigmoidF1 is benchnarked against loss functions commonly used in multilabel learning and other existing multilabel models. We show that
2

---

Published in Transactions on Machine Learning Research (09/2022)
custom losses improve predictions current solutions several different metrics, across text and image classification tasks. PyTorch and TensorFlow source code are made available
2 Background
We traditional statistical framework a guideline for multilabel classification methods Tukeey 1977 We distinguish the desired theoretical statistic (the estimand), its functional form (the estimator) and its approximation (the estimate); estimates can be benchmarked with metrics. We show how multilabel reduction estimators tend to reformulate the estimand and treat labels as marginally independent. For example, by treating a multilabel problem as a succession of binary classification tasks. However, with a proper estimator, possible directly model the estimand. If F1 score is indeed the statistic of interest (i.c. estimand), our proposed loss function, sigmoidF1, accommodates for the truc estimand.
We define a learning algorithm F (i.e., class of estimators) that maps imputs to outputs given set of hyperparameters F(; X' - V. We consider particular case, with the input vector x {=1...., znl and each observation is assigned labels (one or more) = (h...., lc} out of a set of C classes. of are binary variables, indicating presence of a label for each observation i and class j. Together, they form the matrix output Y. This is our multilabel setting. Note that multiclass classification can be considered as an instance of multilabel classification, where a single label is attributed an example.
2.1 Estimand and definition of the risk
We distinguish between two scenarios: the mulliclass and the mullilabee scenario. In the multiclass scenario, a single example attributed one class label (e-g., classification of an animal on picture). In the multilabel scenario, a single example can be assigned more than one class label (e.g., movie genres). We focus on the latter. For particular set of inputs (c.g.. movie posters) and outputs Y (e.g., movic genre(s)), the risk formulation is the same in (Menon et al. 2019):
RML (F) = (x [CML (Yx))
(1)
The learning algorithm F is the estimand, the theoretical statistic. For one item 3i, the theoretical risk defines how close the estimand can get to that deterministic output vector y,. In practice, statistical models do output probabilitics &r via an estimator and its estimate (also called propensities or suitabilitics (Menon) ee al. ). The solution to that sochhsstiedeeeemiiiisie incompatibility is either to convert the estimator to deterministic measure via decision thresholds (e-g., traditional cross-entropy loss), to treat the estimand as stochastic measure (our sigmoidF1 loss proposal).
2.2 Estimator: the functional form
The estimator f F is any minimizer of the risk RML- Predicting multiple labels per example comes with the assumption that labels are non mutually-exclussve.
Definition. The multilabed estimator of vl is dependent on the imput and other ground truth labels for that czample, oi f(x, ". P(v! = llw!...... ..)) x.).
By proposing this general formulation, we entrench that autuulll-innclusie characteristic in the estimator. Contrary to Menon et al. (2019), our definition above models interdependence between labels and deals with thresholding for the estimate training time for free. Waegeman et al. (2014) show that an estimator of an F-score can be used at inference time for multilabel classification, when using probabilistic models where parameter estimation is possible (c.g., decision trees, probabilistic classifier chains). When it is not possible, resort to defining loss function.
3

---

Published in Transactions on Machine Learning Research (09/2022)
2.3
Estimate: approximation via loss function
Most of the literature on multilabel classification can be characterized as multilabel reductions (eeoo el al. 2019]: an approximation of the original multilabel problem via loss function s(ys: f). It can take different forms.
One-versus-all (OVA) is a reformulation of the multilabel classification lask to a sequence of binary classifications (f....,f{') with C' the number of classes, Cova () LBC (vf, f where LBC is a binary classification loss (binary relevance (Brinker et al.) 2006; Tsoumakas & Katakis 2007 Dembczyúski et al. 2010)): , most often logistic loss. Minimizing binary cross-entropy is equivalent to maximizing for bog-liccihood (Bishop) 2007 $4.3.4).
Pick-all-labels (PAL) gives the loss function (PAL(y., -c=1 yf CMC f), with LMC a multiclass loss (c.g., softmax cross-cntropy). In this formulation, each example (I,, converted a multiclass framework, with one observation per positive label. The sum of inherently multiclass losses is used to represent the multilabel estimand.
Multilabel reduction methods are characterized by their way of reformulating the estimand, the resulting estimator, and the estimate. This allows the use of existing losses: logistic loss (for binary classification formulations), sigmoid or softmax cross-entropy loss (for multiclass formulations). These reductions imply a reformulation of the estimator (a.k.a. Bayes Optimal) as follows:
1 = f()) = p( = 1[2;).
(2)
Contrary to our definition of the original multilabel estimator (Section 2.2 marginal independence of label propensities is assumed. In other words. the loss function becomes any monotone transformation of the marginal label probabilities P(u! (z)) (Dembozyrskii et al. 2010 Koyejo et al. 2015 Wu & Zhou 2017). In literature reviews, the multilabel reductions OVA and PAL have been coined as di-data-to-algriine opposed to fit-algorithm-to- data (Zhang & Zhou 2014). originally framed as problem tronsformation and algorithm adaptation respectively (Tsoumakas & Katakis 2007)). For the purpose of our narrative, we propose the following formalization of this dichotomy: fit-data-to-algorte existing losses Le, with Le any classification loss and oftentimes sum over all classes. This can be contrasted with sttalgorithmm-to-dee discuss this in Section 3 and Table
as
formulates an additive loss over
where custom loss L is built for the multilabel task. We further
2.4 Metrics: evaluation at inference time
There is consensus on the usefulness of a confusion matrix and ranking metrics to evaluate multilabel classifi-
cation models at inference time (Koyejo et al. 2015 Behera et al.) 2019 Wu & Zhou 2017). Confusion matrix
metrics come with caveats: most of these measures (i) require hard thresholding, which makes them non-dif-
ferentiable for stochastic gradient descent; (ii) they are very sensitive to the number top labels to include k
Chen et al.] 2006);; and (iii) they require aggregation choices to be made in terms of micro/ micro/macro/ /macro/
wwightte /tt/////cwc///eeeee////////////
metrics. Common confusion matrix metrics are Precision, Recall, Fi-score or one-error-loss; see (Wu &
Zhou 2017 for others.
2.5 Multilabel estimate: F1 metric as loss
A modcl's out-of-sample accuracy commonly measured on metrics such as AUROC, F1 score, etc. These reflect an objective catered towards evaluating the model over an entire ranking. Due to the lack of differ- entiability, these metrics cannot be directly used as loss functions at training time (in-sample). Eban et al.) (2017) propose theoretical framework for deriving decomposable surrogates to some of these metrics. We propose our own decomposable surrogates tailored for multilabel classification (see Appendix A
In a typical machine learning classification task, ground truth binary labels are compared to a probabilistic mcasure (or a reversible transformation of a probabilistic mcasure such as a sigmoid or a softmax func- tion) (Bishop) 2007). If the number n, of labels to be predicted per example known a priori,
natural
at training time to assign the topn; predictions to that example (Lapin et al. 2016 2015 Tf the number
4

---

Published in Transactions on Machine Learning Research (09/2022)
Table 1: SigmoidF1 and related loss formulations ordered by publication date. The solution column refers to our proposed formalization of the literature review on how to conduct multilabel classification: D2A refers fit-dota-to to-algorithm (sum over existing or cross- cooss-etrrop--liee CE-like, classification losses refers to ftt-algorthmm-to-data (custom loss (*)
Σ Le) and A2D
Method
Solation
Model lype
Context
Impieteentation
Surrogated metric
Modality
ACE
Fisher
D2A
Any
Any
CV-like
Any
ronking
Zhang & Zhou
DEA
Any
Any
pair-rank
tabular
MFC
Huang et Al
Gaussian mixtures
Mispronuciatioon detection
sigmold
F
Text
opt
Eban e6 al.
A2D
Any
Any
Fi
Theoretical
Ji
D24
Neural net
Imbelanced-muttiin
CE-like
Imaage
deepl'
Decubber of al.
420
Neural net
multilahel
CE.like
P
Imaage
softF1
Liang et BL.
A2D
Newal net
Multilabel
Unbounded
Fi
Image
ASL
Baruch cl al.
D2A
Neural nel
Multilabel
CE-like
Image
RSS
A20
Neural net
Similarity
sigmoid
Rocall
Image
polyl oss
Leng
420
Neural net
Imbsllanddd-miiitlesss,,,
CV-like
Image
sigmoidF
A2D
Neural net
Multilabel
sigmoid
Pi
Text & Image
of labels per example is not known a priori, the question remains at both training and at inference time as to how to decide on the number of labels to assign to each example. This is generally done via a decision threshold, that can be set globally for all examples (Lipton et al., 2014]. This threshold can optimize for specificity or sensitivity (Chen et al. 2006) for per-class thresholding see Chu & Guo (0017). In Section 4 we propose an approach where this threshold is implicitly defined at training time, by using loss function that penalizes explicitly for wrong label counts and fits to the original estimand in Definition 2.2 the Fl metric. In Section
q we show how Fi is formulated into a surrogate loss pp. Our contribution is thus in the fil-algorithm-to
the continuation
trend, because we propose a custom loss function. That loss function also the first to directly approximate the F1 score with non-divergent estimates (see Sections 4.1
and 4.2 on boundedness).
3 Related Work
In Section 2.3 we mentioned how existing solutions for multilabel tasks can be divided into fit-data-to- algorithm solutions, which map multilabel problems to a known problem formulation like multiclass classi- fication, and fit-olgorithm--tor at hand (Madjarov al.] 2012). In most of this work, the term multilabel classification excludes extreme
solutions, which adapt existing classification algorithms to the problem
(tens of thousands of labels) (e.g.. Jernite et al. 2017 Agrawal et al.] 2013 Jain et al.] 2019). hierarchical
(parent and children labels) (e.g., Lehmann et al. 2015 Yang et al.
2019 Howard & Ruder 2018) or
multiclass (single label per example) subfields. These subfields call for their own solutions, including label
embeddings Bhatia et al. 2015 or negative mining (Reddi et al. 2019 for the extreme usecase.
tit-data-to-allorith In fit-data-to-algorith solutions, cross-entropy losses (Fisher 1912 Good 1952)
are used at training time and thresholding is done at inference time to determine how many labels should be assigned to an instance. This has also been called multilabel reduction (Menon et al.) 2019) and differs from
multiclass-to-binar classifications (Zhang) 2004 Tewari & Bartlett 2005 Ramaswany et al. 2014). We can
further distinguish between One-versus-al (OVA) and Pick-all-labels (PAL) solutions (Menon et al. 2019
(see Section 2 In OVA, one reduces the classification problem to independent binary classifications (Brinker
et al. 2006 Tsoumakas & Katakis 2007 Dembezyúski et al.) 2010 Wydmuch et al. 2018). In PAL, one
reformulates the task independent multiclass classifications (Boutell et al. 2004 Jernite et al., 2017 Joulin 2017). The label powerset approach considers each set of labels as a class (Boutell al. 20044). In
ett al.
Pick- One-Onoeonnn Label (POL), single multiclass example is created by randomly sampling a positive label (Joulin
et al.] 2017 Jernite et al., 2017). Alternatively, ranking by pairwise comparison is a solution where the dataset is duplicated for each possible label pair. Each duplicated datasct has therefore two classes and only contains instances lhat. have at least one of the labels in the label pair. Different ranking met
exist (Zhang & Zhou 2006 Hillermeier et al.) 2008 Loza Mencia & Furnkranz 2008p. Ranking loss has
been shown to optimize for two Learning To Rank metrics (Chen et al., (209)). More recently, hicrarchical datascts such as DBpodia [Lehmann et al.) 20155] are used to fine-tune BERT-basee models (Yang et al.
2019 Zaheer et al. 2020); the latter publications use cross-entropy to predict the labels.
5

---

with @ the component-wise multiplication sign. For simplicity, in the formulation above and the ones that follow scores are calculated for a single class, therefore the sum is implicitly over all examples Σ.- This applies to the binary classification problem but also to our multilabel sotting, when micro metrics are calculated Published in Transactions on Machine Learning Research (09/2022)
Fit-algorithm-to-ddta. In ft-allgrithhm-o-datta solutions, clements of the learning algorithm are changed (e.g.: the back propagation procedure). Before focusing on the multilabel case, the multiclass literature has some examples of F1 surrogate loss functions in particular: in the context of SVMs, via pseudo linear functions (Narasimhan et al.) 2015) or by learning a feasible confusion matrix @Nrasimhan et al.) 2015 in the context of deep networks, by learning the surrogale loss function via a dedicated neural network in the binary classification case (Grabocka et al. 2019), by optimizing performance measures composed of true positive and true negative rates (Sanyal et al., 2018) or via empirical utility maximization of F1 on 3-layer neural networks (Gai et al. 2019). Early representatives of multilabel fttalgorithm-to-datt solutions stem from heterogenous domains of machine learning. MultiLabel k-Nearest Neighbors (Zhang & Zhou 2007p, MultiLabel Decision Tree (Clare & King 2001), , Ranking Support Vector Machine (SVM) (Elisseeff & Weston 2001} and backpropagation for multiLabel learning with a ranking loss (Zhang & Zhou, 2006). More recently, the idea of multi- multi-task learning for label prediction and label count prediction was introduced
(MINET, Du et al.] 2019; Li et al.) 2017 Wu et al. 2019). The literature has been clearly hinting at the usefulness of single lask loss function that approximates metric. A formulation similar to our loss unboundedF1 was proposed in an unpublished blog post, which was referred to softF'l (Chang et al. 2019) A similar proposal was to use the hinge loss as decomposable surrogate for confusion matrix entries for binary classification (Eban et al. 2017). Outside of the context of neural networks, the Mazimum Fi-score crilerion for automatic mispronunciation detection was proposed as an objective function to a Gaussian Mixture Model-hidden Markov model (GMM-HMM) (Huang et al. 2015 A recent paper used recall as a loss function for image similarity (Patel et al. 2022 In parallel, there is a growing consensus that the original enosopy loss data-to-aal-orttiiie cannot solve all our problems. A variation of the cross- entropy loss adapted to multilabel classification has been proposed (Baruch et al., 2020, Wu et al.) 2019); ; it extends the multiclass sparse class representation setting (Lin et al., 2017 Leng et al. 2022). In the ranking domain, LambdaLoss has been proposed to optimize directly for the lambdaRank metric (Wang et al. 2018). In the theoretical space, Eban et al. (2017) have proposed a generic framework for decomposable metrics, including F1 as a theoretical fractional lincar program. Table illustrates how sigmoidF1 differs from the methods listed in this paragraph.
An important limitation shared by existing reddaa-tto-lgoriiee and fit-algorithm-toode approaches the lack of unified loss framework that deals with multilabel classification and can approximate a metric of interest. sigmoidF1 compules an Fl surrogale loss over the aggregation of examples in a batch at training time.
4 Method
We introduce our approach for multilabel problems, with a smoothed confusion matrix metric as loss (the original confusion matrix metrics rely on step functions and are therefore intractable, see for example the blue step function in Figure 2). We first briefly define our learning setting and define the confusion matrix metrics in this setting more formally.
We use the binary classification setting (two classes) to simplify notation, without loss of generalization to the multilabel case. Tn typical binary classification problem with the label vector y (y1,-.-, [um) }n predictions are probabilistic and it is necessary to define threshold t, at which prediction is binarized. With 1 as an indicator function, y = =E Ly22, y = Σ 1yes are thus the count of positive and negative predictions at threshold 1. Tet lp. fp, fn, In be number of true positives, false positives, false negatives and true negatives respectively:
tp = I1= Oy fp = 1y2 @ (1-y) fn = -1 Oy tn = 1yet o (1 yy),
(i.e., compute the metric value for each class, and then averagod over all classes). In the multilabel setting y can be substituted by yd for each class j. Note that vectors could be trivially substituted by matrices
6

---

Published in Transactions on Machine Learning Research (09/2022)
(Y) in Eq. to obtain the macro formulation. Given the four confusion matrix quadrants, generate further metrics like precision and recall (see Table 5 in Appendix A). However, none of these metrics are decomposable due to the hard thresholding, which is, in effect, a step function (see Figure 2
Next, we define desirable properties for decomposable thresholding, unbounded confusion matrix entries, and a sigmoid transformation that. renders confusion matrix entries decomposable. Finally, we focus on a smooth Fl score.
4.1 Desirable properties of decomposable thresholding
We define desirable properties for dccomposable sign function f(u) surrogate of the above indicator function lyer-
Property 1. Boundedness:
[f(w) < M, where M is an upper and lower bound.
The ground truth y is bounded between [0, 1] and thus it must be compared to bounded prediction 8; preferably bounded by [o, 1], to avoid further scaling.
Property 2. Saturation: So (-)(w) - f(u) e, with number close to zero and a saluration bound.
For the surrogate to be a proper sign function substitute, it is important to often return values close to 1 or O. Saturation is defined in the context neural network activation functions and refers to the propensity of iterative backpropagation to progressively lead to values very close to or after a long enough training period. Our aim to reach that convergence quickly in order to force decisions towards 0 1 in order to be comparable to step function. This highlights tension: the sigmoid function should contrast. outputs towards 0 1 but should not be too saturated, in order for the derivative at point to be non-null and information to flow back to the network (Krizhevsky et al. 2017).
Property 3. Dynamic Gradient: f'()) > E [-s, s], where s is the saturation bound.
Inside the saturation bounds [-s, the derivative should be significantly higher than zero in order to facilitate stochastic gradient descent and backpropagation. Note that the upper and lower limits of fu) are interchangeably [] or 1] in this paper and in the literature. The conditions above still apply after linear transformation. Next, we show how our formalization of an unbounded F1 surrogate would not fulfill these propertics and how our proposition of a smooth bounded alternative does.
4.2 Unbounded confusion matrix entries
A first trivial remedy to allow for derivation of the sign function f(u), is to define unbounded confusion matrix entries by retaining the original logits (scores) when counting true positives, false negatives, etc. Countrary to the original confusion matrix definition in Eq. 3 tp. fp, fn and in not natural numbers anymore:
Tp = E Fm = -1- in (1)
f- -=
(4)
where ip, fp, fn and in are now replaced by rough surrogates. The disadvantages are that the desirable properties mentioned above are not fulfilled, namely (i) is unbounded and thus certain examples have over- pee-proppotional effects the loss; (ii) it is non-saturated; while non-saturation is desirable for activation functions (Krizhevsky et al. 2017), here it would be desirable to tend towards saturation (i.c., tend to values close to 0 or 1, so as lo give the most accurale predictions al any thresholding values at inference time); and (iii) the gradient of that linear function is 1 and therefore backpropagation will not learn depending on different inputs this stage of the loss function. However, this method has the advantage of resulting in linear loss function that avoids the concept of thresholding altogether and trivial to decompose for stochastic gradient descent.

---

Published in Transactions on Machine Learning Research (09/2022)
mapping
10 mp
P: 10 rp
(:: 5 5p
m 0 stop
unbounded
Figure Different thresholding regimes: the step function (original Ff metric) not decomposable, the linear function is unbounded (ppz) and tends produce divergent gradients, whereas the sigmoid function (Ly)) is bounded and allows for differentiation due to its smooth curvature, tunable at different parametriza- tions.
4.3 Smooth confusion matrix entries
We propose a sigmoid- based transformation of the confusion matrix that renders its entries decomposable and fulfills the three desirable properties above:
Tp = >==
pp S(9) -
(5)
f= o(t))oy in = ((11--(())0(1)))
with S(.) the vectorial form of the sigmoid function S(.):
S(w: B, m) 1 exp(-B(u n))
(6)
with B and lunable parameters for slope and offset, respectively. Higher 3 results in steeper slope al the center of the sigmoid and thus more stringent thresholding. At its extreme, limg-soe S(u; B3 17) corresponds to the step function used in Eq. 3 Note that negative values of 3 geometrically reflect the sigmoid function across the horizontal line at 0.5 and thus invert predictions. These smooth confusion matrix entries allow to build any related metric (see Table in Appendix(A Ap.. Furthermore, the surrogate entries are decomposable, bounded, saturated and have dynamic gradient.
4.4 Smooth macro F1 scores
macro and micro level. Macro-averaging regards all classes equally important, whereas micro-averaging reflects within-class frequency. unboundedF1 and sigmoidF/ below are thought of as macro scores (aggregated over all classes). These scores require a high enough number of representatives in the four confusion matrix quadrants to learn from batch to batch. Ideally, cach training epoch would have only one batch, so as to have the most representatives. Following Eq. 4 is possible to define an unbounded F1 score:
Fl scores can be calculated
21p
-1- FI, where =
(7)
2tp fn fp
While this alternative abstracts the thresholding away, which is convenient for fine-tuning purposes, it does not fulfill the desirable properties of binarization threshold surogate (see Section 4.2 unboundedF1 will be used to benchmark against our proposed sigmoidF1 loss. Given the definitions smooth confusion matrix metrics above, we can now write Fi
2tp
F1 = F1,
where
Fi
(8)
2tp fn fp
8

---

Published in Transactions on Machine Learning Research (09/2022)
sigmoidF1 is particularly suited for the multilabel setting because it is proper hard thresholding surrogate as defined in the previous sections and because it contains significant amount of information about label prediction accuracy: tp, fn and fo are indicative of the number of predicted labels in each category of the confusion matrix but also contain a notion of certainty, given that they are rational numbers. The built in sigmoid function ensures that certainly increases along training epochs, as outlined by Property Finally, as the harmonic mean of precision and recall (a property of F1 in general); weighs in both relevance metrics.
In the next section, we implement Eq.
g in PyTorch and TensorFlow custom loss as follows:
with y the ground truth and the outcome of the last layer sig - 1 (1 + exp - B (z n)))
tp (sig y, dim=0)
(sig (1 yy, din=0)
((1 sig) y, din=0)
sigmoid_f1 2+tp (2+tp
fp 1e-16)
minimize(1 sigmoid_f1)
The pseudocode above illustrates the elementwise multiplication of matrices S(9) and 9 over all examples in the batch and all possible classes.
5 Experimental Setup
We test multilabel learning using our proposed sigmoidF1 loss function on four datasets across different modalities (image and text). For each modality we take state-of-the-ard model that generates an embedding layer and append a sigmoid activation and different losses. Multilabel deep learning is usually implemented with sigmoid binary cross-entropy directly on the last neural layer (a simplification of the OVA and PAL reductions). We follow this approach for our experiments (e.g., in (large) language models (Zaheer et al. 2020
Devlin et al. 2019)) ) Some baselines include multilabel reformulation choices: only keeping the top-n occurring classes (often 4 4-10) (e.g.. Zhang et al.) 2015 Cunha et al. 2021), multiclass classification on each entity within an example (objects in an image, expressions in text) (e.g., Lin al. 2014
Wang et al.
2016
Wei et al. 2016 Zhu et al. 2017). We refrain from doing so.
Table Descriptive statistics of our experimental datasets.
Dataset
Type
Classes
Average label count
Number of examples
movicPosters
image
28
2.2
37,6 37,632
arXiv2020
text
155
1.9
26,558
Pascal-VOC
image
20
1.6
9,963
MS-CÓCO
image
80
2.9
122,218
5.1 Datasets
Table lists the datasets we use. Two of the datasets are multilabel in nature. moviePosters is related movies [Neha] 2018) and arXiv2020 relates arXiv paper abstracts Cfornell-nnivessit (2021). We the image segmentation datasets Pascal-VOC (Everingham et al.] 2007) and MS-COCO (Lin et al. 2014). with bounding boxes and one label per box. By altributing all box labels to the image a whole, it has been used as a reference benchmark for multilabel classification. We refer to Appendix D for further descriptions of the datasets and references.
5.2 Learning framework
Our proposed learning framework consists of two parts: a pretrained deep neural network and a classification head (see Figure D); different. loss functions are computed in the classification head.
Neural network architecture. For the movicPoster image dataset, we use MobileNet (Sandler et al. 2018) architecture that was pretrained on ImageNet (Deng et al. 2009). This network architecture is typically used for inference small computing devices (e.g., smartphones). We use version of MobileNet
9

---

Published in Transactions on Machine Learning Research (09/2022)
2021 p. For the three text datasets, use DistiBERRT (Sanh et al. 2019) as implemented in Hugging Face. This is particularly efficient instance of the BERT model (Huggingface) 2021). For the Pascal-VOC and MS-COCO datasets, we use the recent state-of the--art resnet TresNet (Ridnik et al. 2021) pretrained on ImageNet (Deng et al.] 2009) and some of the best practices for Pascal- and MOCC collected in a recent. benchmark (Baruch et al. 2020).
already stripped off of its original classification head Google
We use Tre:Ket- 21K stands for Imagenet21K, the larger ImageNet corpus. In all cases, we use the final pre-trained layer as an embedding of the input. To ensure that the results of different loss functions are comparable, we fix the model weights of the pretrained MobileNet DistilBERT and TresNet and keep the hyperparameter values that were used to be trained from scratch. At training time, we optimize with Adam for all three architectures and use In-Place Activated BatchNorm (InplaceAABN) for TresNet (Rota Bulò et al. 2018).
The classification head is a latent. representation layer (the final pretrained layer mentioned above) con- nected with a RELU activation. This layer is linked to a final classification layer with a linear activation. The dimension of the final layer is equal to the number of classes in the dataset. The attached loss function is either BCE (Binary
focalLoss (Lin et al. 2017). ASL (Baruch et al. 2020), unboundedF or sigmoidF1 (ours). When computing the loss at training time, sigmoid transforms the unbounded last layer to a [-1,1] 1] bounded vector that contrasts positive and negative predictions. These values are then used as inputs to any of the loss functions above over all classes and the entire batch of examples. In the case of Cpy, this corresponds to a surrogate macro F1. Given the vectorized computation of Lii (see Section 4.3) the computational burden is only marginally affected. At inference time, the last layer is used for prediction and is bounded with sigmoid function. A threshold must. then be chosen at evaluation time compute different metrics. Figure I depicts this learning framework.
Metrics. In our experiments, we report on microF1, macroF1, Precision, mAP (used in some recent multilabel benchmarks; see Appendix A) and [miere-JweighttedF1 (where within-class scores are weighted by their reprosentation in the dataset). We focus our discussion around weightedF1 as it is the most comprehensive F1 measure we could find on multilabel problems: is a micro measure, thus accounts for differences between classes, and has a reweighing argument, thus accounting for class imbalance. Given limited resources we rerun each model on each loss with 5 random sceds. With only 5 runs per loss function, hypothesis testing results would have been particularly sensitive to the choice of distribution F Instead, we show the distribution of results in Appendix F. which show robust statistics (median and interquartile range). Note that cross-validation cannot be performed as Poscal-VOC and MS-COCO have fixed train- validation-test sets. There is an interaction between our optimization on sigmoidF1 and our evaluation using (weighted) Fl metrics. We expect higher values on F1-related metrics during evaluation and thus report on alternative metrics too.
5.3 Hyperparameters and reproducibility
We implemented all losses in Pytorch and Tensorflow. Batch size is set at a relatively high value of 256 to increase accuracy over traditional losses (Smith et al. 2017), but also allow hetcrogencity in the examples within the batch, thus collecting enough values in each quadrant of the confusion matrix (see Section 4.4 for discussion). Regarding the sigmoidF1 hyperparameters 3 and y, we performed a grid search with the values in the range [1] for 3 and [0, 2] for 17. In our experiments, we evaluate the sensitivity of our method to these hyperparameters (see Figure 2 and Appendix D for optimal values). We made sure to split the data the same training, validation and test sets for each loss function. We trained for 60 (Pos MS-COCO) to 100 (arXiv2020, moviePosters) epochs, depending on convergence. Our code, dataset splits and other settings are shared to ensure reproducibility of our results.
6 Experimental Results
The goal of sigmoidF1 (Crz) ) is to optimize for the Fl score directly at training time in the context of multilabel classification. In this section, we test whether C can outperform existing loss functions on
We found that, given some unstable results on unboundedF1, even a conservative student t distribution would imply that the 95% confidence interval covers metric values over 100%
10

---

Published in Transactions on Machine Learning Research (09/2022)
multiple classification metries. We present multilabel classification results for Ly on four datasets, movie- Posters, ar arXiv2020, PoS and MS-COCO in Table 3
We recall Table I in which we highlight that LBC is originally designed for binary classification, LFL. for imbalanced multiclass, CASL to optimize mAP for multilabel classification. They are computed over each class at training time, as opposed to per batch for our p,y and 4pT- The latter two explicitly account for label dependencies in the loss function.
In general, Table 8 shows that Ep outperforms other loss functions on three possible formulations of the F1 metric (weightedF1, microF11 and macroF11). We also confirm that the recent ASL loss outperforms other losses on the precision and mAP metrics. Lpj is designed as an Fl surrogale, it is thus not surprising for it to perform best on F1 metrics and comes at no noticeable additional computational cost (see Appendix We first analyze the Fl metrics before interpreting the precision and mAP results in more detail.
Measured on the F1 metrics (weightedF1, microF1 and macroF1), Coz and ΕвCE always share the top 2 in performance, oftentimes far ahcad of other losses. This highlights that losses inspired by BCE are not yet tailored to optimize for the Fl score in multilabel classification, and also that BCE is a good default choice in general. However, in certain settings, and in particular with our standard datasets Pascal- VOC and MS-COCO, CFj can provide clear improvements over the original BCE, macroFl on the movicPosters dataset is a counter-intuitive exception to that obscrvation: BCE outperforms Cpj only on the macro measure, although L,; is essentially macro F1 loss function, as it is calculated across all classes and over each entire batch. Similarly focalLoss is dominant on MS-COCO macroF1,, but not significantly (see Figure There is room for improvement on MS-COCO because we did not finetune the sigmoidF1 hyperparameterss (B and 11) and instead reused the Pascal- VOC ones, due to resource constraints.
On precision and mAP, no top 2 losses emerge. Instead, results are dataset and modality dependent. Sur- prisingly, the traditional BCE loss outperforms other losses by far in precision on a thoroughly benchmarked dataset like POC. focalLoss delivers best results for MS-COCO on precision, probably because the original paper used MS-COCO as benchmark to design their loss function (Lin et al.) 2017). Precision performance gains are less clear on the two smaller datasets (arXiv2020 and movicPosters); LTT performs reasonably well [ Regarding mAP, LASL. expectedly oulperforms other methods on Pascal- VOC, confirming their own benchmarks (Baruch et al. (0200) and their ability to beat focalLoss and BCE on MS-COCO and PASCAL-VOC. Notably, LASL is also first on mAP on text data. This is the first time that ASL is tested on text data to the best of our knowledge. Overall, these mitigated results for precision and mAP motivate further research in optimizing directly for precision and mAP at training time.
A note on thresholding and zero values. For the bigger and more standard datascts Pascal-VOC and MS-COCO our neutral metric threshold of 0.5 provides results in line with the literature. With our own fine-tuning regime on a smaller model (see Section 5.2), our mAP scores are 1-2% away from the current state of the art (Baruch et al.] 2020). On smaller datasets like arXiv2020, moviePosters and others (see Appendix F . the sigmoid activation per class at inference time are closer to zero. To a certain extent, this can be interpreted as the model having less confidence in its predictions (Guo et al.] 2017p. As a result, a ncutral 0.5 threshold resulted in zero values on almost all losses and metrics for small datasets. Given the range of values in these predictions, 0.05 seems like the next. best neutral threshold. We refrain from further finetuning the threshold for each dataset, loss and metric B As result of the absence of finetuning, moviePosters display zero values for Lr. and LASL on most metrics. This can be explained by the higher average label count for moviePosters. This is in opposition to the propensity of LPL and LASL to deal with sparser label representation.
The analysis above highlights that sigmoidF1 can indcod optimize for F1 metrics (weightedF1, microF1 and macroF1) reliably and consistently, over six datasets in total (see Appendix E. Given the more mitigated
"Coy was found particularly unstable for Pas over 5 different seeds (see the extended results in Appendix D Provided it is unbounded, predictions can diverge towards (positive or negative) infinite values.
The classes in Pascal-VOG and MS-COCO are lot more concrete (e.g., car, person, bicycle) and are directly related to the original classes of ImageNet on which the TresNet and MobileNetV2 were trained, opposed to movic genres for moviePosters or arXiv paper scientific domain.
While optimixing the threshold at inference time is an interesting research topic, we refrain from doing so here, so as to disentangle the lass function benchmarking from the thresholding regime benchmarking.
11

---

Published in Transactions on Machine Learning Research (09/2022)
Table 3: Multilabel classification mean performance in percent over 5 random seeds. The Fl metric variants are the focus here (weightodF1, microF1 and macroF1), since we aim directly optimize for F1 training time. precision and mAP are displayed for reference, as they are often used in the literature that context. Metric formally defined in Appendix A and thresholds indicated there for each datasct. We reused
fine-tuned Pascal-VOG sigmoidF1 hyperparaaeterrs (3 and 7) for MS-COCO due to resource constraints.
Loss
weightedF1
microF1 macroF1
precision
mAP
LBE
79.02
75.81
79.55
82.52
81.21
Tres2Netm2KK on MS-COCO 30.5
2021
CrL 2017
81.28
79.18
81.76
85.73
84.88
LASL.
2020
73.48
70.36
70.81
60.16
85.59
(CNN)
C
PT
otrrs
79.90
77.51
79.74
81.05
78.33
C [ours]
81.82
79.93
81.67
80.62
81.98
CACF 1912
87.52
85.85
87.76
90.75
91.54
TreKNetm21KK 2021 on Pascal-VOC G0.5
CFL 2017
72.54
59.24
76.82
84.70
76.19
ASL
2020
77.85
76.53
75.98
65.36
93.11
(CNN)
FT
ouurs
77.24
74.84
75.31
75.53
79.36
L
[ours]
88.20
87.70
87.87
85.36
92.36
Fj
CBCB 1912
20.59
18.19
18.42
10.15
10.50
DestilBert 2019 on arXiv2020 00.05
CFL 2017
18.85
16.59
18.01
10.10
10.43
LASL. 2020
19.15
16.90
18.16
10.32
10.53
(NLP)
[ours]
15.23
13.74
14.50
10.27
10.49
FT
C
[ours]
20.60
18.20
18.43
10.15
10.50
Fi
CвCE 1912
13.79
9.47
12.94
5.51
5.78
MobileNetV2 2018 on moviePosters $0.05
CPL 2017
0.00
0.00
0.00
0.00
5.80
CAST. 2020
0.00
0.00
0.00
0.00
5.80
(CNN)
F7 [ours]
13.97
9.84
10.11
5.59
5.90
Fj [ours]
14.81
10.33
10.57
5.58
5.81
results for precision and mAP, it seems relevant to further explore opportunities of metrics-as-losses Finally, BCE, which was designed with binary classification in mind, is a good first approximation.
Sensitivity analysis. In Figure 3 we show the sensitivity of sigmoidF1 to different parametrizations of 7) and B. Within the chosen values (see Section 5.3), we chose to display parameter space similar to the one illustrated in Figure 2 Moving the sigmoid to the left allows the learning algorithm to tend to (local) optimum. In general and across dalasets, when sampling for y, not how the optimun lended towards positive values. Offsetting the sigmoid curve to the left has the effect of pushing more candidate predictions to the rank of positive instance (or at least close to 1). We also note how 3 (which cannot be negative or otherwise the sigmoid function would flip around the horizontal axis) is at best close to a value close to 0 on this dataset (we show discrete values here for display purposes). The sigmoid thus relatively smooth, which involves dynamic gradients over different batches. The idea is similar a high learning rate. In our experiments, this rarely gave rise to divergent behavior in the loss function (learning curve). We learn that it is necessary to tune hyperparameters for each dataset, as it is for LFL, LASL and others in Table
The results in this section show that, in general, multilabel classification results measured on Fl metrics can be improved using sigmoidF1 independently of the dataset, its modality or the neural network architecture.
7 Discussion
multilabel classification, and more generally the context of deep neural networks, losses are formulated be decomposable for gradient descent. At inference time, however, end-users tend to look for clear-cut actionable decisions from the model (e.g., automize the arXiv keywords selection, one needs obtain clear-cut set of keywords given each abstract). This is probably why most. evaluation metrics in the
12

---

Published in Transactions on Machine Learning Research (09/2022)
macroF 100055
microF 12005
weightedF- 1020.5
0.100
eta
0.5
-
0.075
0.050
0.025
0.000
6 beta
10
Figure 3: DistilBERT (NIP) on XrXi20020 - different weightedF1 scores at a 0.5 threshold for different values of 1y and & in a sampling region similar to Figure
multilabel literature, with the notable exception of mAP, are also reliant on clear-cut counts (e.g. tp, fn, [p, In). Although models are benchmarked on these values, we found little discussions on how to retrieve clear-cut counts from final softmax sigmoid activations bounded by [0, 1]. Among benchmarked losses, the authors of FocalLoss (Lin et al. 2017 use a global 0.5 threshold at inference-time. The authors of ASL (Wu et al. 2019) do not mention thresholding in the paper but GitHub issue hints at the fact that they used 0.8 as a global threshold for MS-COCO | We feel that defining clear-cut counts deserves more attention.
Such clear-cut counts are usually achieved via a decision threshold. Decubber et al. (2018) distinguish between utility marimization (at inference-time) and decision-theorctiic (at training and at inference time) approaches.
Utility maximization methods. At inference time, a threshold can be set globally for all examples to optimize on the training data, before using it on the test data (Lipton et al., 2014 Decubber et al. 2018). This threshold can optimize for specificity, sensitivity (Chen et al. 2006) or directly for F1 (Decubber et al. 2018). Alternatively, different thresholds can be set per-olass (Chu & Guo 2017).
Decision-theoretic methods. Decision-theoretic methods operate both at training and at inference time. They stem from shallow learning fields and have multiple steps: (i) encoding the original item-label matrix to submatrices that each can be (ii) fit traditional loss function (cross-entropy variations), before (iii) de- coding the submatrices back to the original item-label matrix format via an inference-phase optimization solver. This methodology can be found across the shallow learning fields of SVMs (Ye et al. 2012), logistic regession Dembczyuski et al., 2011; Zhang et al. 2020), multinomial regression Dembczyyski et al. 2013
and Bayesian networks (Gasse & Aussem| (0166). These methods were implemented in a deep learning set- ting, where they had more success than udility marimization or fixed thresholding methods (Decubber et al. 2018). Decision-theoretie methods have at least 3 moving parts mentioned above and are thus complicated to benchmark against each other, let alone against inference-time thresholding or fix thresholding.
As hinted before with ASL and focalLoss, modern deep learning models tend to not tune the decision threshold, either with ulility mazimizalion or decision--heerreiic methods. We propose to take a first step in this direction in the following.
For the sake of this discussion, we focus on the simplest utility mazimization (inference-time) thresholding implementation. Threshold Averaging (Decubber et al., 2018) is method that uses the training set to tune global threshold, before applying it to the test set. Take Yi, set of label predictions for one example i. We select each Diy as possible thresholding candidate to binarise the vector ý,. We then calculate instanco-wise F1 scores over 84- The value fiz that results in the highest F1 score for an instance is chosen as the instance's threshold. This process is repeated for each instance i in the training data. The average threshold over all instances in the training data then chosen as the final global threshold for the test data.
See https://githubb.cmm htwpaabbooooooollbbaaa--aaa Aww.abbaa-l AA.////////S and also insightful learning tricks at https://githnb..com/ Alibaha- MIII// AStt/ssees///
13

---

Published in Transactions on Machine Learning Research (09/2022)
Table 4: Multilabel classification mean performance in percent over' 5 random seeds. Global thresholds were found using a threshold-moving technique. Results are systematically lower than with a fixed threshold (see third row of Table 3). Metric are formally defined Appendix [
Loss
weightedF1
microF1
macroF1
precision
mAP
BbE
1912
15.89
13.98
15.73
10.11
10.35
DistilBert 2019 on arXiv2020 (NLP)
LFL
2017
16.40
14.14
17.22
9.83
10.42
LASI. 2020
17.49
14.86
17.77
10.33
10.51
Threshold moving
Cpy [ours]
16.52
14.27
16.70
9.98
10.43
Lrj [ours]
15.11
13.19
15.20
10.05
10.41
In Table we show results of Threshold Averaging (Decubber et al., 2018) on the arXiv dataset. Tt is notable here that ASL's mean results always outperform other losses. This time around, however, almost all boxplots IQRs intersect, thus results are very inconclusive (see Figure (). We thus refrain from bold numbers like in Table 3 Most importantly, metrics are consistently below results from the original neutral fixed 0.05 threshold in Table g This is consistent with some of the results in (Decubber et al. 2018), showing that simple thresholding methods based on ulilily marimization are not sullicient to consistently beat fixed thresholds or
methods.
Inferenco-time decisions can completely change the outcome of a prediction set, of its resulting evaluation metrics, and, thus, even of the winning model. We hope that thresholding will be more broadly discussed in the future or at least. for the thresholding method to be openly stated in research papers; we chose fixed neutral thresholds, to focus on the benchmarking of losses at training-time.
Together, utility maximization (inference-ting time) thresholding methods and decision-theoretic methods (at training and at inference time) form an under-explored research domain, with several open questions: (i) Which data split should be used for thresholding? With the entire training dataset (Decubber et al. 2018), there is a risk of overfitting the threshold. Maybe it is worth introducing a holdout set that is only used for threshold tuning. (ii) Should we threshold globally for interpretability or have per-class or even per-instance threshold? (iii) Are decision theoretic (a.k.a. at training and at inference time) approaches also not prone to overfitting and are they elficient large neural networks for large datasets? (iv) Can other losses than the classical cross-entropy loss be used to train decision theoretic models?
8 Conclusions
To solve multilabel learning Lasks, existing optimization frameworks are typically based on variations of the cross-entropy loss. Instead inspired by the binary classification literature (see most recently (Gai et al., 2019) and their F1 surrogate loss functions 3-layer neural networks) we propose the sigmoidF1 loss, part of general loss framework for confusion matrix metrics. sigmoidF1 loss can achieve significantly better results for most metrics on four diverse datasets and outperforms other losses on the weightedF1 metric. We thereby provide evidence that sigmoidF1 is robust to modality, model architecture and dataset size, when optimizing for F1 metrics. Generally, our smooth formulation of confusion matrix metrics allows us optimize directly for these metrics that are usually reserved for the evaluation phase. The proposed unboundedF1 counterpart does not require hyperparameter tuning and delivered better results than existing multiclass losses on most metrics; it can act as mathematically less robust approximation of sigmoidF1.
In future work and within the generic multilabel setting, first incremental step could be to train bigger dataset like MS-COCO (Lin et al. 2014) (if provided with more resources) and use more robust transfer Baruing/finetunnig procedures, for example with dynamic weight freezing for finetuning (Howard & Ruder 2018). Alternatively, we could train a CNN or a BERT model for multilabel tasks with our smooth losses from scratch (cf., (Wu et al. 2019) and (Lin al. 2017)). If training from scratch, this can be combined with representation learning Milbich model abstract relationships.
2020 Wang et al. 2020) or supervied learning, in order
14

---

Published in Transactions on Machine Learning Research (09/2022)
can tackle other multilabel settings, such as hierarchical multilabel classification (Benites & Sapozhnikova 2015 active learning NNakano et al. 2020), multi-instance learning (e.g., Soleimani & Miller 2017 Zhou et al. 2012), holistic label learning
Next, we could validate if F1 or another
(see dataset Large Scale Holistic Video Understanding (Diba et al., 2019 (). or extreme multilabel prediction
CChang el al. 2020 Liu et al. 2017 Babbar Schölkopl 2017] Yen el al. 2017 Prabhu et al. 2018 (with
Jain al.] 2016)), where the number of classes ranges the tens of thousands. Beyond the multilabel setting, sigmoidP1 could be tested on any model that uses F1 score as an evaluation
missing labels (Yu et al. 2014
metric such AC-SUM-GAN (Apostolidis et al. 2020).
One limitation of sigmoidF1 is that it is computed at macro level over the whole batch and ignores (micro) per class Fl scores. Given our limited GPU memory, we could not load enough examples in each batch to represent each confusion matrix quadrant of each class reliably. such a route is followed, we could eventually finetune or learn 3. and ?e the parameters of the sigmoid function per class
We believe that smooth metric surrogates should inform future research on multilabel classification tasks. There is evidence of growing interest in the literature (Chang et al.) 2019 Huang et al. 2015 Patel et al. 2022) for metries as losses and the objective of this paper is to further highlight their relevance, across modalities, architectures and dataset sizes. Based on the results presented in this paper, we consider metrics-as-losses (e.g., Jaccard, confusion matrix metrics, ranking metries) as the next step in the evolution of multilabel classification algorithms.
Acknowledgments
We thank our reviewers and the action editor for their valuable comments and suggestions, especially for pointing out to the thresholding literalure, which led us to add discussion section. We thank Nanne Noord for his valuable suggestions on the paper structure and Maurits Bleeker for reviewing the text.
This research was (partially) funded by Bertelsmann SE & Co. KGaA; by the Hybrid Intelligence Center, a 10-year program funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research,
centrenn1
All content represents the opinion of the authors, which is not necessarily shared endorsed by their respective employers and /or sponsors.
Author Contributions
All authors participated in the ideation process and discussions. Gabriel Bénédict was responsible for first drafts, the code and visualizations (except Figure 1, which is mostly attributable to Daan Odijk). Hendrik Vincent Koops, Daan Odijk and Maarten de Rijke revised initial drafts, proposed changes and feedback in content and form.
References
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Gcoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Leven- berg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasude- van, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.
Rahul Agrawal, Archit Gupta, Yashoteja Prabhu, and Manik Varma. Multi-label learning with millions of labels: Recommending advertiser bid phrases for web pages. In Proceedings of the 22nd International Conference on World Wide Web, WWW '13, pp. 13-24, New York, NY, USA, 2013. Association for Computing Machinery.
15

---

Published in Transactions on Machine Learning Research (09/2022)
Hesam Amoualian, Parantapa Goswami, Laurent Ach, Pradipto Das, and Pablo Montalvo. Sigir 2020 commerce workshop data challenge overview. In Proceediiggs Proceedings of ACM SIGIR Workshop eCommerce (SIGIR eCom cC). ACM. ACM, 2020.
Evlampios Apostolidis, Eleni Adamantidou, Alexandros I. Metsai, Vasileios Mezaris, and loannis Patras. AC-SUM-GAN: Connecting actor-critic and generative adversarial networks for unsupervised video sum- marization, IEEE Transactions on Circuits and Systems for Video Technology, 2020.
Rohit Babbar and Bernhard Schölkopf. Dismec: Distributed sparse machines for extreme multi-label clas- sification. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM '17, pp. 721-729, New York, NY, USA, 2017. Association for Computing Machinery.
Emanuel Ben Baruch, Tal Ridník, Nadav Zamir, Asaf Noy, Itamar Friodman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi-label classification. CoRR, abs/2009.1411 14119, 2020.
Bichitrananda Behera, G. Kumaravelan, and P. Kumar B. Performance evaluation of deep learning algo- rithms in biomedical document classification. In 2019 11th International Conference on Advanced Com- puting (ICoAC), pp. 220 224, 2019.
Fernando Benites and Elena Sapozhnikova. HARAM: A hicrarchical ARAM neural network for largo-scale text classification. In 2015 IEEE International Conference on Data Mining Workshop (ICDMW), pp. 847-854, 2015.
Kush Bhatia, Himanshu Jain, Purushottam Kar, Manik Varma, and Prateok Jain. Sparse local embeddings for extreme multi-label classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Christopher M. Bishop. Pottern Recognition and Machine Learning. Information science and statistics.
Springer, 5 edition, 2007.
M. J. Blosseville, Georges Hébrail, Maric-Gaëla Monteil, and N. Pénot. Automatic document classifica- tion: Natural language processing, statistical analysis, and expert system techniques used together. In Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR. '92, pp. 51 58, New York, NY, USA, 1992. Association for Computing Machinery.
Matthew R. Boutell, Jicbo Luo, Xipeng Shen, and Christopher M. Brown. Learning multi-label scene classification. Pollern Recognition, 37(9):1757-11 1771, 2004.
Klaus Brinker, Johannes Fürnkranz, and Eyke Hüllermeier. A unified model for multilabel classification and ranking. In Proceedings of the 2006 Conference on ECAI 2006: 17th European Conference on Artificial Intelligence August 29 September 1, 2006, Riva Del Garda, Italy, pp. 489-493, NLD, 2006. IOS Press.
Trstenjak Bruno, Mikac Sasa, and Dzenana Donko. KNN with TF-IDF based framework for text catego- rization. Procedia Engineering, 69:1356 1364, 11 2013.
Wei-Cheng Chang, Fu Yu, Kai Zhong, Yiming Yang, and Inderjit S. Dhillon. The un- known benefits of using a soft-fl loss in classification systems. Towards Data Science, Dec 2019. URL http:::/Eooarrddaaaacceeeee www.n/c.....tteeeeeeeeeeeeeeeeee ttoo/the-aae wwwwo/tteennnonneaeeeeeeeeeeeeee ttwpmeeeeeeiittt---ooeeeee bet.eettttt----sssss------------ a-soft-f--loss- in otwp:/ffaattooo-stttteemee5----0
Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, and Inderjit S. Dhillon. Taming pretrained transformers for extreme multi-label text classification. Proceedings of the 26th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Mining, Jul 2020.
James J. Chen, Chen-An Tsai, Hojin Moon, Hongshik Ahn, John J. Young, and Chun-Houh Chen. Decision threshold adjustment in class prediction. SAR and QSAR in Environmental Research, 17(3):337-353 Jun 2006.
16

---

Published in Transactions on Machine Learning Research (09/2022)
Wei Chen, Tie-yan Liu, Yanyan Lan, Zhi-ming Ma, and Hang Li. Ranking measures and loss functions in learning to rank. Tn Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (eds.), Aduances
in Newral Information Processing Systems, volume 22. Curran Associates, Inc., 2009. URL https://
Wei-Ta Chu and Hung-Jui Guo. Movie genre classification based on poster images with deep neural networks. Proceedings of the Workshop on Multimodal Understanding of Social, Affectiue and Subjective Attributes, Oct 2017.
Amanda Clare and Ross D. King. Knowledge discovery in multi-label phenotype data. Lecture Noles in Computer Science, pp. 42-53, 2001.
Cornell-University. arXiv datasct and metadata of 1.7M+ scholarly papers across STEM. kaggle.com/ Cornell--biiversittyyarar 2021.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Tn Machine Learning, pp. 273-297, 1995.
Washington Cunha, Vitor Mangaravite, Christian Gomes, Sérgio Canuto, Elaine Resende, Cecilia Nasci- mento, Felipe Viegas, Celso França, Wellington Santos Martins, Jussara M. Almeida, Thierson Rosa, Leonardo Rocha, and Marcos André Gonçalves. On the cost-efeectivenes of neural and non-neural ap- proaches and representations for text classification: A comprchensive comparative study. Information Processing & Management, 58(3):10248,, 2021.
Stijn Decubber, Thomas Mortier, Krzysstof Dembezynski, and Willem Waegeman. Deep f-measure maxi- mization in multi-label classification: A comparative study. In ECML/PKDD, pp. 290 305, 2018. URL
Krzysætof Dembozyíski, Weiwei Cheng, and Eyke Hüllermeier. Bayes optimal multilabel classification via probabilistic classifier chains. In Proceedings of the 27th International Conference on International Con- ference on Machine Learning, ICML'10, pp. 779,286, Madison, WI, USA, 2010. Omnipress.
Krzysztof Dembezynski, Willem Waegeman, Weiwei Cheng, and Eyke Hillermeier. An ex- act algorithm for f-measure maximization. In J. Shawe- R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, vol- une 24. Curran Associates, Inc., 2011. URL http://ppoceedinggnneurrrppppppp httperpppppcccppppp//////.1/. Tlallabbcce1134000.f664662220 Paper..p
Krzysztof Dembczynski, Arkadiusz Jachnik, Wojeiech Kotlowski, Willem Waegeman, and Eyke Hueller- meier. Optimizing the f-measure in multi-label classification: Plug-in rule approach versus structured loss mininization. Tn Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th Interna- tional Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1130 1138, Atlanta, Georgia, USA, 17 19 Jun 2013. PMLR. URL http:://procedingg.ll...esss v28 hts//debccynssk11..tm
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei- ImageNet: A large-scale hierarchical image databasc. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long and Short Papers), pp. 1171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Ali Diba, Mohsen Fayyaz, Vivek Sharma, Manohar Paluri, Jurgen Gall, Rainer Stiefelhagen, and Luc Van Gool. Large Scale Holistic Video Understanding. Xiv preprint arXio: 11451v3, 2019.
17

---

Published in Transactions on Machine Learning Research (09/2022)
Jingcheng Du, Qingyu Chen, Yifan Peng, Yang Xiang, Cui Tao, and Zhiyong Lu. Ml-net: Multi-label classification of biomedical texts with deep neural networks. Journal of the American Medical Informalics Association, 26(11):1279 1285, Jun 2019.
Elad Eban, Mariano Schain, Alan Mackey, Ariel Gordon, Ryan Rifkin, and Gal Elidan. Scalable Learning of Non-Decomposable Objectives. In Aarti Singh and Jerry Zhu (cds.), Proceedings of the 20th Interna- lional Conference on Arlificial Intelligence and Stalistics, volune 54 of Proceedings of Machine Learning Research, pp. 832-840. 840. PMLR, 20-22 Apr 2017.
André Elisseeff and Jason Weston. A kernel method for multi-labelled classification. In Proceedings of the 14th International Conference Newral Information Processing Systems: Natural and Synthetic, NIPS'01, pp. 681 687, Cambridge, MA, USA, 2001. MIT Press.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes challenge 2007 (VOC2007) results.
2007.
Ronald A. Fisher. On an absolute criterion for fitting frequency curves. Messenger of Mathematics, 41: 155-160, 1912.
Yu Gai, Zheng Zhang, and Kyunghyun Cho. Gradient-bassed learning for f-measure and other performance metrics, 2019. URL httppp//pppaarreeeeeerre///oeiep
Maxime Gasse and Alex Aussem. F-measure maximization in multi-label classification with conditionally independent label subsets. CoRR, abs/ 1755, 2016. URL http:://arxx...rg/bbb///////eeet
Irving J. Good. Rational decisions. Journal of the Royal Statistical Society: Series (Methodological)) 14 (1):107-114, Jan 1952.
Google. Feature vectors of images with MobileNet V2 (depth multiplier 1.00) trained on ImageNet (ILSVRC- 2012-CLS). http::///hhbbbeeeeeooo//////eeee
cacpeet_vv21____ w2__100222/feeeeeeeeeeeeeeeeeaee 224/feature //24/eeature_vector//. 2021.
vector/4
Josif Grabocka, Randolf Scholz, and Lars Schmidt-Thieme. Learning surrogate losses. CoRR, abs/1905.10100 2019. URL.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 06-11 Aug 2017.
Douglas Hanahan and Robert A. Weinberg, Hallmarks of cancer: The next generation. Cell, 144(5):646 674, Mar 2011.
Jun He, Liqun Wang, Liu Liu, Jiao Feng, and Hao Wu. Long document classification from local word glimpses via recurrent attention learning. IEEE Access, 7:40707-40718, 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. Proceedings of the 56th Araual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018.
Hao Huang, Haihua Xu, Xianhui Wang, and Wushour Silamu. Maximum fl-score discriminative training criterion for automatic mispromunciation detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(4):787-797, Apr 2015.
Huggingface. DistiBBERT. http::////uiiiiieeeeeeaaaaaeeeea
2021. David A Hull Information retrieval using statistical classification. PhD thesis, Stanford University, 1994.
Eyke Hüllermeier, Johannes Fürnkranz, Weiwei Cheng, and Klaus Brinker. Label ranking by learning pairwise preferences. Artificial Intelligence, 172(16):1897-1116, 2008.
18

---

Published in Transactions on Machine Learning Research (09/2022)
Himanshu Jain, Yashoteja Prabhu, and Manik Varma. Extreme multi-label loss functions for recommen- dation, tagging, ranking; other missing label applications. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pp. 935 944, New York, NY, USA, 2016. Association for Computing Machinery.
Himanshu Jain, Venkatesh Balasubramanian, Bhanu Chunduri, and Manik Varma. Slice: Scalable linear extreme classifiers trained on 100 million labels for related searches. In WSDM '19, February 11-15, 2019, Melbowrne, VTC, Australia. ACM, February 2019. Best Paper Award at WSDM '19.
Yacine Jernite, Anna Choromanska, and David Sontag. Simultaneous learning of trees and representations for extreme classification and density estimation. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 1663-1674. JMLR.org, 2017.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chopter of the Association for Computational Linguisties: Volume 2, Short Papers, pp. 427-431, Valencia, Spain, April 2017. Association for Computational Linguistics.
In-Ho Kang and GilChang Kim. Query type classification for web document retrieval. In Proceedings of the 26th Ansuual International ACM SIGIR Conference on Research and Development in Informaion Relrieval, SIGIR '03, pp. 64-71, New York, NY, USA, 2003. Association for Computing Machinery.
Olwwasammi O. Koyejo, Nagarajan Natarajan, Pradeep K. Ravikumar, and Inderjit S. Dhillon. Consistent multilabel classification. In Advances in Neural Information Processing Systems, volume 28, pp. 3321-3329. Curran Associates, Inc., 2015.
Alex Krizhevsky; Llya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84 90, May 2017.
Maksim Lapin, Matthias Hein, and Bernt Schiele. Top-k multiclass SVM. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS 15, pp. 325-333. Cambridge, MA, USA, 2015. MIT Press.
Maksim Lapin, Matthias Hein, and Bernt Schicle. Loss functions for top-k error: Analysis and insights. 2016 IEEE Conference on Computer Vision and Pattern Recognation (CVPR), Jun 2016.
Kristin Larsson, Πooa Silins, Yufan Guo, Anna Korhonen, Ulla Stenius, and Marika Berglund. Text mining for improved human exposure assessment. Tovicology Lellers, 229:S119, Sep 2014.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, and Christian Bizer. DBpedia- large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web, 6(2):167-199) 2015.
Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Jay Shi, Shuyang Cheng, and Dragomir Anguclov. PolyLoss: A polynomial expansion perspective of classification loss functions. In International Conference on Learning Represenlations, 2022.
Yuncheng Li, Yale Song, and Jiebo Luo. Improving pairwise ranking for multi-label image classification. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. Lecture Notes in Computer Science, pp. 740-755, 2014.
Tsung-Y: Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollar. Focal loss for dense object detection. 2017 IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Zachary C. Lipton, Charles Elkan, and Balakrishnan Naryanaswany. Optimal thresholding of classifiers maximize I measure. Lecture Noles in Compuler Science, pp. 225-239. 2014.
19

---

Published in Transactions on Machine Learning Research (09/2022)
Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, and Yiming Yang. Deep learning for extreme multi-label text classification. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Aug 2017.
Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu. Query2label: simple transformer way to multi-label classification. CoRR, Abs/2107.10834 2021.
Eneldo Loza Mencia and Johannes Furnkranz. Pairwise learning of multilabel classifications with percep- trons. In 2008 IEEE International Joinl. Conference on Neural Nelworks (TEEE World Congress Computational Intelligence), pp. 2899 2906, 2008.
Madjarrv, Dragi Kocev, Dejan Gjorgieviki, and Sašo Džeroski. An extensive experimental comparison of methods for multi-label learning. Pattern recognation, 45(9):3084-3104, 2012.
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Multilabel reductions: what is my loss optimising? In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché Buc, E. Fox, and R. Garnett (ods.), Adeances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Timo Milbich, Omair Ghori, Ferran Diego, and Björn Ommer. Unsupervised representation learning by discovering reliable image relations. Pallern Recognilion, 102:107107, Jun 2020.
Felipe Kenji Nakano, Ricardo Cerri, and Celine Vens. Active learning for hierarchical multi-label classifica- tion. Data Mining and Knowledge Discovery, 34(5):1496-1553) 1530, 2020.
Harikrishna Narasimhan, Purushottam Kar, and Prateek Jain. Optimizing non-decomposable performance measures: A tale of two classes. In Proceedings of the 32nd International Conference on International Conference Machine Learning Volume 37, ICML'15, pp. 199 208. JMLR.org, 2015.
Neha. Movie genre from its poster. http:://ww..aaaggocooooooooloo/p 2018.
Rafacl Padilla, Sergio L. Netto, and Eduardo A. B. da Silva. A survey on performance metrics for object- detection algorithms. In 2020 International Conference on Systems, Signals and Image Processing (IWS- SIP), pp. 237-242, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, perfooraforeecce deep learn- ing library. In H. Wallach, Н. Larochelle, A. Beygelzimer, F. d'AlchéBBuc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Asso- ciates, Inc., 2019. URL Learning library.pdf
an- an-ineeaaiive--ttlllee
high peggh-peffrmance-ooee
Yash Patel, Giorgos Tolias, and Jiri Matas. Recall@k surrogate loss with large batches and similarity mixup. CVPR, 2022.
Adler Perotte, Rimma Pivovarov, Karthik Natarajan, Nicole Weiskopf, Frank Wood, and Noémic Elhadad. Diagnosis code assignment: Models and evaluation metrics. Journal of the American Medical Informalics Association, 21(2):231-223 Mar 2014.
Yashoteja Prablu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. In Proceedings of the 2018 World Wide Web Conference, WWW '18, pp. 993-1002, Republic and Canton of Geneva, CHE, 2018. International World Wide Web Conferences Steering Committee.
20

---

Published in Transactions on Machine Learning Research (09/2022)
Harish G. Ramaswamy, Balaji Srinivasan Babu, Shivani Agarwal, and Robert C. Williamson. On the consistency of output code based learning algorithms for multiclass learning problems. In Maria Florina Balcan, Vitaly Feldman, and Csaba Szepesvári (eds.), Proceedings of The 27th Conference on Learning Theory, volume 35 of Proceedings of Machine Learning Research, pp. 902, Barcelona, Spain, 13-15 Jun 2014. PMLR
Sashank J. Reddi, Satyen Kale, Felix Yu, Daniel Holtmann-Rice, Jiecao Chen, and Sanjiv Kumar. Stochastic negative mining for learning with large output spaces. In Kamalika. Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1940-1949. PMLR, 16-18 Apr 2019.
Tal Ridnik, Hussam Lawen, Asaf Noy, Emanuel Ben Baruch, Gilad Sharir, and Itamar Friedman. Tresnet: High performance gpu-dedicated architecture. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4400-1409, January 2021.
Samuel Rota Bulò, Lorenzo Porzi, and Peter Kontschieder. In-place activated batchnorm for memory- optimized training of dnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. MobileNet Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520, 06 2018
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/ Abs/1190.001008, 2019.
Amartya Sanyal, Pawan Kumar, Purushottam Kar, Sanjay Chawla, and Fabrizio Sebastiani. Optimizing non-decomposable measures with deep networks. Machine Learning, 107, 09 2018. doi: 10. ss09994- 018-5736-y.
Fumin Shen, Yadong Mu, Yang Yang, Wei Liu, Li Liu, Jingkuan Song, and Heng Tao Shen. Classification by retrieval: Binarizing data and classifiers. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, pp. 595 604, New York, NY, USA, 2017. Association for Computing Machinery.
Samuel L. Smith, Picter-Jan Kindermans, and Quoe V. Le. Don't decay the learning rate, increase the batch size. CoRR, abs/1711 1.00499, 2017.
Hossein Soleimani and David J. Miller. Semisupervised, multilabel, multi-instance learning for structured data. Neural Computation, 1054
2017.
Ambuj Tewari and Peter L. Bartlett. On the consistency of multiclass classification methods. In Peter Auer and Ron Meir (eds.), Learning Theory, pp. 147, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg.
Grigorios Tsoumakas and Ioannis Katakis. Multi-label classification: An overview. International Journal of Data Warehousing and Mining (IJDWM), 3(3):1-13, 2007.
John Wilder Tukey. Eaploratory dala analysis. Addison-Wesley series in behavioral science: quantitative methods. Addison-Wesley, Wesley, Reading, Mass., 1977.
Willem Waegeman, Krzysztof Dembezyíki, Arkadiusz Jachnik, Weiwei Cheng, and Eyke Hüillermeer. On the bayes-optimality of f-measure maximizers. J. Mach. Learn. Res., 15(1):3333 3388, January 2014.
Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. Con-rnn: A unified framework for multi-label image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, and et al. Deep high-resolution representation learning for visual recognition. IEFE Transactions on Pottern Analysis and Machine Intelligence, pp. 1-1, 2020.
21

---

Published in Transactions on Machine Learning Research (09/2022)
Xuanhui Wang, Cheng Li, Nadav Golbandi, Mike Bendersky, and Marc Najork. The LambdaLoss framework for ranking metric optimization. In Proceedings of The 27th ACM International Conference on Information and Knowledge Management (CIKM '1), pp. 1313 1322, 2018.
Yunchao Wei, Wei Xia, Min Lin, Junshi Huang, Bingbing Ni, Jian Dong, Yao Zhao, and Shuicheng Yan. Hep: A flexible emn framework for multi-label image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(9):1901-199) 19907 2016.
Baoyuan Wu, Weidong Chen, Yanbo Fan, Yong Zhang, Jinlong Hou, Jie Liu, and Tong Zhang. Tencent ml-images: A large-scale multi-label image database for visual representation learning. IEEE Access, 7: 172683 1726933 2019.
Xi-Zhu Wu and Zhi-Hua Zhou. A unified view of multi-label performance mcasures. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proccedings of Machine Learning Research, pp. 3780- 3788. PMLR, 06-11 Aug 2017.
Marek Wydmuch, Kalina Jasinska, Mikhail Kuznetsov, Róbert Susa-Feketee and Krzysztof Dembezyúski. A no-regret generalization of hierarchical softmax to extreme multi-label classification. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, pp. 6358 6368, Red Hook, NY, USA, 2018. Curran Associates Inc.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3483-3492, 2010.
Yiming Yang. An evaluation of statistical approaches to text categorization. Information Retrieval, 1:69-90, 2004.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. XI- net: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Process- ing Systems, volume 32, pp. 5753 5763. Curran Associates, Inc., 2019.
Nan Ye, Kian Ming Adam Chai, Wee Sun Lee, and Hai Leong Chieu. Optimizing f-measure: A tale of two approaches. In ICML, 2012. URL http://icml.ce http://ccccccccc///////////////e 2///20022////// 100o1//apprs//7.ppdd
Ian E.H. Yen, Xiangru Huang, Wei Dai, Pradeep Ravikumar, Inderjit Dhillon, and Eric Xing. Ppdsparse: A parallel primal-dual sparse method for extreme classification. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, pp. 545-553, New York, NY, USA, 2017. Association for Computing Machinery.
Hsiang-Fu Yu, Pratock Jain, Purushottam Kar, and Inderjit S. Dhillon. Large-scale multi-label learning with missing labels. In Proceedings of the 31st International Conference on International Conference on Machine Learning Volume 32, ICML'14, pp. 593 1-601. JMLR.org, 2014.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems, 2020.
Min-Ling Zhang and Zhi-Hua Zhou. Multilabel neural networks with applications to functional genomics and text categorization. IEFE Transactions on Knowledge and Data Engineering, 18(10):1338-1333 1351, Oct 2006.
Min-Ling Zhang and Zhi-Hua Zhou. MI-knn: A lazy learning approach to multi-label learning. Pattern Recognition, 2003) 2048, Jul 2007.
Min-Ling Zhang and Hua Zhou. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26(8):1819-1888 1837, Aug 2014.
22

---

Published in Transactions on Machine Learning Research (09/2022)
Mingyuan Zhang, Harish G. Ramaswamy, and Shivani Agarwal. Convex calibrated surrogates for the multi- label f-measure. In Proceedings of the 37th International Conference on Machine Learning, ICMT,'00. Jrg, 2020.
Tong Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics, 32(1):56 85, 2004.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS'15, pp- 649 657, Cambridge, MA, USA, 2015. MIT Press.
Zhi-Huu Hua Zhou, Min-Ling Zhang, Sheng-Jun Huang, and Yu-Feng Li. Multi-instance multi-label learning. Artificial Intelligence, (76(1):229) 2320, 2012.
Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, and Xiaogang Wang. Learning spatial regularization with image-level supervisions for multi-label image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Appendix A
Evaluation Metrics
In our experimental evaluation, we consider a suite of metrics that are commonly used in the evaluation of multilabel classification to measure the eflectiveness of multilabel prediction. These metrics are based on the confusion matrix and for which we provided smoothed surrogates to optimize directly (see Table 5
Table 5: Confusion matrix with our proposed smoothed confusion matrix entries, fp, fo, fu and in and six derived loss functions that use these smoothed confusion matrix entries. L is used in our experiments.
Condition
Condition negative
tp
positive
Accuracy
Predicted
True positive
False positive
fp C Precision 1p+/p
positive
ip S()) y
fp S()) @ y))
Predicted
False negalive
True Negative
C NNPV fi
negative
fm (1 - S())) oy
in = (1 S())) o y1
Tp
L Spectficity
in
C
2hp
Recall
bp+fv
fp
When true positives and false positives are used, recall that tp 1y21 y and fp 1y21 ② (1 y), and thus a threshold t must be set. For Pascal-VOC and MS-COCO, we set t = 0.5, as is commonly done in the early literature (Zhang & Zhou] 2014 Clare & King. 2001). In the recent literature, the chosen threshold at inference time can vary but was not found to be justified, we thus decide on neutral thresholds before training.
Extending F1 to multiclass binary classification means deciding whether to pool classes. In first pooled iteration, macro Fi Koyeejo et al. 2015 equates to creating single 2x2 confusion matrix for all classes:
Σ 2tp; 2 2 tpp + - fnz + + fPj
Fmacoo
(9)
()), when summing over each class up to the C classes. Miero Fi Lipton et al. 2014 Koyejo et al. 2015) amounts to creating one confusion matrix per class unpooling:
with as short form of E --)((),
pmiiroo
1
2tp,
1 C
Σ
P.
(10)
C
2tp + fn; + fpj
23

---

Published in Transactions on Machine Learning Research (09/2022)
Weighted micro F1 Behera et al. 2019) is similar but includes weighing to account for class imbalance, i.c., weighing each class by the number of ground truth positives:
F1
Frncighted
1 Pi
Σ Pj
where Pj =
Σ y/=1"
(11)
C j=1
We also define miero precision
pinter
Σ tp fPj
Ipj
(12)
mean Average Precision (mAP) has different definitions. We use mAP as defined for the MS-COCO and Pascal-VOC datasets (Padilla et al. 2020). Traditionally, Precision and Recall is computed section of object detection boxes. We use slightly modified mAP (e.g., in (Baruch et al. 2020)), where precision and recall are computed over the predictions of labels on the whole image. We first obtain the average precision over each class:
the inter-
AP all
E(itt -R;) Pinterp (R++1)
Σ (Ret1
(13)
Pinterp (R++1) = max P(R),
R: Ru+1
and then compute mean Average Precision:
C
mApmicro
0 AP
Σ
(14)
1=1
We write micro here to be explicit, but it seems to be mostly computed at the micro level in the literature.
Appendix B Focal Loss definition
We write down the focalLoss (Lin al. (017),, it deals specifically with class imbalance and is used as a baseline due to its popularity in the multiclass domain.
- (1 83)² log (f*),
(15)
with a² and y hyperparameters. In the next section, we further specify the setup for focal loss and cross entropy as benchmarks for undoundedF1 and sigmoidF1.
Appendix C
Compute Time
Table 6 shows compute times in minutes for different on losses and different datasets on single GPU g4dn.12xlarge 12xlarge AWS instance. The run-time is not particularly long, given that we freeze model weights of the pretrained image / text model.
Table 6: Average training time over 5 seeds in minutes (60 epochs for MS-COCO and Fascal-VOCL 100
epochs for the reminder two).
MS-COCO
Pascal-VOC
arXiv2020
moviePosters
1912
856
112
341
58
CPL
2017
851
108
428
59
LAST.
2020
856
109
427
59
Cpr [ours]
858
116
381
58
L
fuurs]
858
111
351
52
24

---

Published in Transactions on Machine Learning Research (09/2022)
Appendix D
Experimental Setup Details
moviePasters consists of images of movic posters and their genres (e.g., action, comedy) (Chu & Guo 2017 8 The posters and labels have been extracted from IMDB and the dataset was previously used for per- class, post-training thresholding (see Section 3)_ The genre labels in this dataset are not mutually exclusive and of varying counts per movie.
Xi is a subset of the newly created orXiv datase 9 with over 1.7 million open source articles and their metadata. Our experiments use the abstracts and categories that are suitably non-mutually exclusive and of varying counts per example. The limited number of labeled classes render the older dataset unsuitable for our experiments. We write arXiv2020 for the subset of the arXiv dataset that only contains documents published in 2020. This results in around 26k documents. There is longer history of using arXiv to create research datascts; the datasct we use is not to be confused with an carlier long document dataset that only features 11 classes (He al.] 2019), and was used in a recent long transformer publication Zaheer et al. 2020
Lascal-VOC and MS-COCO stand for Pascal Visual Object Classos Challenge (VOC 2007) (Evering- ham et al. 2007 and Microsoft Common Objects in Context (Lin et al. 2014), respectively. They are object recognition/seen reoogution/segnenaaiion segmentattin datasets. The earlier Pascal- VOC dataset has 20 possible object classes and around 10K examples. The later MS-COCO dataset has 80 possible object classes and around 200K class-annotated examples. Some multilabel classification literature for the image domain use object detec- tion / segmentation datasets to perform multilabel classification To MS-COCO, Pascal-VOC, NUS-WIDE, etc. (note that transformer models, which effectively distinguish the original objects on the image while predicting labels, perform better on this task (Liu et al., 2021 Regarding Tresnet-m-21k (Ridnik et al., 2021), while L and XL version of the model exist, the code available online did not allow for correct loading of the weights
We choose to ignore classes that are underrepresented, in order to give the model a fair chance at learning from at least a few examples. We define underrepresentationn as a global irrelevance threshold b for classes: any class that represented less than times considered irrelevant. We decided to set an irrelevance threshold b on all datasets prior to conducting experiments, set to 1000 for both arXiv2020 (145 of the original 155
to not fine-tune for that feature. It was classes remaining) and moviePosters (14 of the 28 classes remaining) and at 10 for chemicalEaposure (all 38 classes remaining) and cancerHallmarks (all 33 classes remaining), in proportion to the number of classes and labels in cach datasct. We used all classes for PaS and MS-COCO since we are comparing with benchmarks that also do so.
Hyperparameters. For Pascal-VOC, we found { = 17 r = 10.25} to work best on weightedF11. Given the similarity of the two datasots and the potentially resource- hungry hyperparameter tuning of MS-COCO, we used the same hyperparameters for MS-COCO. For arXiv2020 and moviePosters, (3 = 1; 9} works best weightedF1. These hyperparameters where tuned on the validation set and we report out test set. It would be hard to give general recommendation of hyperparametens, but it seems that {3 = -0.75; = 10.25) is good basis for image and that {B 1: 9 9} is good basis for text.
the held
Setup. We performed our experiments on Amazon Web Services cloud machines with data parallelization on up to 4 GPUs 14dn..2xlorge with TensorFlow 2 (Abadi et al. 2015) and PyTorch Paszke et al. 2019 as gradient-descent backend.
8Labcls at httpe://tiiyurll.om//yydyyu and images at httg:://stnyuul..oo/lllppppp
9 Available attttpes//iannurccc//yyssss
10 http::///ggesstttttocccc///////s
25

---

Published in Transactions on Machine Learning Research (09/2022)
Appendix E Extended Results
Table 3 shows our results as point estimates over 5 training random seeds. This section contains the distributional counterpart Table 3 namcly boxplots (Figure 6 and 7 with median and inter quartile
range in the blue box. Figure is distributional counterpart Table Ghreshold-moving technique on the arviv dataset) and outlines less conclusive results than for fixed thresholds.
g
weightedft
microt1
macrofa
Prec sdon
mAP
AS.
+
unze.nde-dF1
sigmosh111
74
76
76
82
72
70.0
72.5
75.0
60
78
60
82
Figure 4: Tresnetm21K (CNN) on MS-COCO @0.5.
crossEnrspy
focallass
sigmoiaF1
75
Figure Tresnetm21K (CNN) on Pascal-VOC @0.5 (one outlier (<40) for unboundedF1 on cach metric ignored for better visualization).
weightedd1]
microfi
Prechionn
mAp
crosat
unzoundesdF)
Figure 6: DistilBERT (NLP) on arXiv2020 @0.05.
Weighted
microl's
macrofts
Preck
focalloss
ASL
%
unsoundedF1
sigmoliFl
13.50 15.75 16.00
14:23
2450 14.75
922
950
9.75
10.00
10.25
10
12
13
5.40
5.45
5.50
5:35
5.00
5.5
3.7
5.8
5.2
6.5
Figure 7: MobileNet (CNN) moviePosters @0.05 (zero values for focalLoss and ASL ignored for better visualization).
26

---

Published in Transactions on Machine Learning Research (09/2022)
Hiighto3F
micrF1
macroF1
Procslan
HAP
351
unzounde-4F1
sgmatd
18
10
16
18
18
20
10
10.1
10.2
10.3
Figure 8: DistilBERT (NLP) arXiv2020 Threshold moving.
Appendix F Additional Experiments
This section details additional experiments on two further text datasets from the medical domain. Given that they are relatively small compared to our other benchmark datasets, we keep this discussion in the appendix. Table iilustraats the difference between our main paper datasets and the 2 appendix datasets. Results on the latter are displayed Tables 7a and
7b
ML-NET (Du et al. 2019) has an interesting multitask approach to fil-agorrihmm-to-daaa methods for multi- label learning with unknown label count text. The cancerHallmark (Hanahan & Weinberg 2011 IT and chemical Exposure [Larsson et al. 2014 datasets were used. The third datasel diagnosisCodes could not be obtained (neither from the authors of ML-NET nor of the original paper (Perotte et al., 2014 We aggregate sentence labels to the whole description for cancorHallmarks and chemicalExposure, as was done for ML-NET.
Table Multilabel classification performance@0.0s on single run.
(a) DistiBBERT (NLP) + classification head on cancer
Loss
weightedF1
microF1
macroF1
Precision
LECE
0.0
0.0
0.0
0.0
CFL
10.8
19.0
4.4
7.1
17.0
17.6
9.8
8.9
Dpi
20.2
31.3
9.5
5.9
(b) DistilBERT (NLP) + classification head on chemicalExposure.
Loss
weightedF1
microF1
macroF1
Precision
LBCE
5.1
5.8
1.2
4.7
CPL
26.8
34.8
9.3
13.0
C F1
21.8
19.4
13.3
15.5
C
C
31.9
43.2
11.3
9.1
For arXiv2020, moviePosters, cancerHallmarks and chemicalFxposure, we saw after a few preparatory train- ing rounds that almost only sigmoidF1 had non-zero results for t = 0.5. Class representation is a lot more sparse for these dataset, we thus set the evaluation metrics threshold to a reasonable value of 0.05 and train for 100 (arXiv2020, moviePosters) or 500 (cancerHallmarks and chemicalExposure) epochs until reaching convergence. Once thresholds were decided upon, no further fhreshold-hacking was performed. Note that a threshold of 0.8 POS, used by [Baucch al. 20000), does not alter the results.
On the smaller chemicalExpposure and cancerHallmarss datasets (see Tables and 7b respectively), the unboundedF1 loss delivers good results for macroF1 and Precision and the sigmoidF] loss Icads to higher
Available http:://sttuaaaaa////////llllaaa com.
Available http:///gthhaaccm/////////aaaaaa
27

---

Published in Transactions on Machine Learning Research (09/2022)
scores the remainder of the metries. We observe that unboundedF1 scores higher than sigmoidF1 macroF1 on the two small text datasets (chemicalFxposure and cancerHallmarks). Since unboundedF1 forgoes thresholding altogether, we hypothesize that unboundedF1 develops tolerance for sparse datasets with low number of class instances.
Table 8 Descriptive statistics of all datasets.
Type
Classes
Average label count
Number of examples
moviePosters arXiv2020
image
28
2.2
37,632
text
155
1.9
26,558
chemExposure
text
38
6.1
3,661
cancerHallmarks
text
33
3.5
1,582
Pascal-VOC
image
20
1.6
9,963
MS-COCO
image
80
2.9
122,218
Notably for the CancerHallman dataset, predictions from a model trained with cross-entropy do not reach high enough values to surpass the threshold and thus all metrics return zero values. This was further observed during experimenlation, thus cross-entropy loss might not be good fit. for solving small-dataset multilabel problems.
28


Reviews:
[{"review_id": "N7DojynGea", "reviewer": "Reviewer_2Ux7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "##### Strong aspects\n\n1. The paper focuses on an important question: finding a direct surrogate loss for the F1 metric which can be optimised directly by the neural networks. \n\n2. The paper proposes a relaxation of the step function, in the same flavour as the sigmoid function, which is straightforward and easy to implement. \n\n3. Some empirical results have demonstrated that the proposed surrogate loss could lead to better performance than other losses on F1. \n\n#### Weak points\n\n1. The biggest concern is the significance of the current study. There are many works focusing on optimising the F1 metric in general [1], or specially for multi-label learning [2,3,4]. Moreover, some recent works solve exactly the same research question as the current paper: proposing a surrogate loss function for the F1 metric which can be employed by neural networks in multi-label learning [5,6]. These works are highly related to the current study but have not been discussed or compared yet. Given that the same problem has been solved in [5], which provides an in-depth theoretical study, I do not see the significance in the current work, at least in the current version.\n[1] Nan Ye, Kian Ming Adam Chai, Wee Sun Lee, Hai Leong Chieu. Optimizing F-measure: A Tale of Two Approaches. ICML 2012\n[2] Krzysztof Dembczynski, Willem Waegeman, Weiwei Cheng, Eyke Hüllermeier. An Exact Algorithm for F-Measure Maximization. NIPS 2011: 1404-1412\n[3] Krzysztof Dembczynski, Arkadiusz Jachnik, Wojciech Kotlowski, Willem Waegeman, Eyke Hüllermeier. Optimizing the F-Measure in Multi-Label Classification: Plug-in Rule Approach versus Structured Loss Minimization. ICML (3) 2013: 1130-1138\n[4] Maxime Gasse, Alex Aussem. F-Measure Maximization in Multi-Label Classification with Conditionally Independent Label Subsets. ECML/PKDD (1) 2016: 619-631\n[5] Mingyuan Zhang, Harish Guruprasad Ramaswamy, Shivani Agarwal. Convex Calibrated Surrogates for the Multi-Label F-Measure. ICML 2020: 11246-11255\n[6] Stijn Decubber, Thomas Mortier, Krzysztof Dembczynski, Willem Waegeman. Deep F-Measure Maximization in Multi-label Classification: A Comparative Study. ECML/PKDD (1) 2018: 290-305\n\n2. The proposed surrogate loss is a straightforward use of the sigmoid loss function in the step function in the F1 metric. Since this is a \"surrogate\" loss of the original (non-differentiable) loss function, it is highly recommended for the paper to do a theoretical study on how optimising the surrogate loss can impact the final performance of the desired loss. Specially, two hyperparameters are included in the proposed loss function to add more flexibility to the original sigmoid loss function. It is also interesting to see how such two hyperparameters can impact the final performance from a theoretical perspective. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a surrogate loss for the F1 metric or any other metric based on the confusion matrix for multi-label learning. Specially, it relaxes the step function with a threshold to a sigmoid-style loss with two tunable parameters. Such a loss function can be optimised directly by neural networks. It is a direct surrogate to the F1 metric, so it is estimated that a better F1 metric can be achieved in the testing phase. Experiments demonstrated that a better testing F1 is achieved on three datasets compared to other loss functions setting as the learning objective. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Add references [1-6]. Discuss and compare with them.\n\n2. Add a theoretical study on how optimising the surrogate loss can impact the final performance on the non-differentiable loss (metric) F1, and study theoretically the impact of $\\beta$ and $\\eta$. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TxrcBHxxSs", "reviewer": "Reviewer_rBR1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - The key idea of this paper is straightforward and easy to grasp.  \n\nWeaknesses:\n- The setting of the proposed approach and its explanation need improvements.\n- Overall, the writing needs polishing. \n- As a journal paper, some aspects of the proposed approach aren't discussed in sufficient details.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies multi-label learning and proposes *sigmoidF1* as a surrogate of the conventional F1 measure. The proposed approach is based on the confusion matrix, so it could be used for other measures defined on the confusion matrix such as precision and Jaccard index. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Below are some detailed comments:\n\n- To be rigorous, a loss is a function we want to *minimize*, so F1 cannot be a loss function. I would suggest to revise the wording throughout the paper. \n\n- When talking about optimizing an objective function (e.g., F1) in a multilabel learning problem, we typically see two different setups: (a) optimization at the training time, and (b) optimization at the inference time. This paper belongs to the former. This point becomes clear, however, only late in the paper. I think it is much better if this is clearly stated early. \n\n- Several variants of F1 measures are widely studied in the multilabel learning literature, for example, micro F1, macro F1, sample F1 (i.e., instance-wise F1). This paper addresses macro F1, but not the other variants. This should be specified, if not in the title, at least in the abstract or in the introduction. \n\n- It isn't super clear what the difference between D2A and A2D (fit-data-to-algorithm vs. fit-algorithm-to-data) is. Is it correct to say that the key difference is if a multi-label objective (such as F1) is directly optimized during the training time? What is the significance to differentiate these two, if both share the same issue that you are trying to address? Two follow-up questions: \n  - There is an existing categorization of multi-label learning approaches: problem transformation vs. algorithm adaptation (Tsoumakas, 2007). What does your categorization differ from the existing one?\n  - In the description of fit-data-to-algorithm, it says that cross-entropy losses are used. This is inconsistent with Table 1, where many methods using other losses are labeled as D2A. \n\n- \"We focus our discussion around weightedF1 as it is the most comprehensive F1 measure we could find on multilabel problems.\" Could you explain why weightedF1 is the most comprehensive F1 measure? What do you mean by *comprehensive*?\n\n- Since the proposed approach is based on the confusion matrix, can't it be also applied for other measures such as Jaccard or precision/recall with ease? I think it might be worth some discussions. ", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4vEGh00Qmo", "reviewer": "Reviewer_zFBc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed sigmoid-based transformation fulfills three important and desirable properties of decomposable thresholding.\n\n2. With the help of the proposed sigmoid-based function, training with the proposed loss function in multi-label learning is quite simple. As shown in the end of Section 4, only five lines of code are needed for calculation.\n\n3. The experiments results show that the proposed sigmoidF1 loss can indeed optimize for F1 metrics.\n\n\nWeakness:\n\nIt would be more convincing if more comparison methods could be used in experiments. There are many existing multi-label learning methods which could be considered as \"fit-algorithm-to-data\" methods.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new multi-label loss function called sigmoidF1 loss, which uses a sigmoid-based transformation of the confusion matrix that renders its entries decomposable and fulfills the three desirable properties of decomposable thresholding in multi-label learning.\n.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. More datasets and comparison methods could be used for evaluation.\n\n2. Minor issues. \"classificaton\" ==> \"classification\" \"to a 100 epochs\" ==> \"to 100 epochs\"\n", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the problem of optimizing the multi-label F1 score with deep networks. The proposal is to construct a surrogate loss for this metric, by replacing the TP/FP rate terms with smooth (sigmoid-based) approximations.

A primary concern raised by reviewers was regarding the technical depth of the proposal. The idea of using a sigmoid approximation to the 0-1 loss has a long history in standard binary classification. The present paper essentially extends this idea to the multi-label setting, in a natural way (replacing each occurrence of TP/FP with a sigmoid surrogate). Given the large literature on principled approaches to F-score maximization, such critiques appear reasonable.

At the same time, the express purpose of TMLR is to judge papers based on whether the claims are well-supported, and whether a subset of the audience would find the results interesting. On these points, the paper is on better ground. To properly frame the contributions, the paper should more explicitly acknowledge upfront (e.g., in the Introduction and Conclusion) that the idea of using smooth approximations to the TP/FP rate being standard (see also references below), _but_ that the application of such an idea to multi-label problems has not been systematically explored, particularly in the case of modern deep networks. This would emphasize that the main message is that this strategy is practically effective for multi-label (not just binary) problems.

Further to the reviewers' detailed comments, below are a few additional comments from my own reading. The authors are suggested to incorporate these as appropriate.

**Threshold-based methods** The discussion period touched on threshold-based approaches as encountered in (Decubber et al., 2018), but apparently these were discounted owing to not being performant compared to the best-performing methods in that paper. This is fine if the present paper was then to compare against these best-performing methods, but it is aruged that "they include multiple moving parts and are therefore out of scope of a pure F1-score-as-a-loss paper". Given however that threshold-based methods are fairly simple to implement and tend to be competitive (if not the best-performing) baselines, it would be of interest to compare them against the present approach.

**Bibliography** It is appreciated that the authors incorporated reviewers' suggestions in the initial review phase. However, given the problem being considered is hardly new and features a vast supporting literature, it behooves the paper to properly situate itself with respect to relevant work; please see suggested references below.

- "Menon et al. (2019) define these as multilabel reduction techniques" -> I believe the idea (and terminology) of multilabel reductions is far earlier than this reference. e.g., (Dembczynski et al., 2010) provided one early analysis of the OvA reduction, and (Wydmuch et al., 2018) discuss multilabel reductions.
    
- "minimizing a step loss function is intractable (Reddi et al., 2019)" -> the latter paper seems an unusual choice for this citation; the idea of using surrogates to overcome the intractability of direct 0-1 loss minimisation has been explored since at least the soft margin SVM.

- missing many relevant citations on optimizing non-decomposable performance measures; e.g., Narasimhan et al., "Optimizing Non-decomposable Performance Measures: A Tale of Two Classes"; Narasimhan et al., "Consistent Multiclass Algorithms for Complex Performance Measures"; Sanyal et al., "Optimizing non-decomposable measures with deep networks" (amongst many others). These are mostly limited to the multi-class case, but there is strong relevance to the motivation of the present paper (which anyway relies on an appropriate reduction to binary or multi-class classification).

- a missing relevant citation is Grabocka et al., "Learning Surrogate Losses". This paper also discusses optimizing generic performance measures (for multi-class tasks), and takes the strategy of trying to *learn* a suitable surrogate loss for such a generic measure. The paper presents results with neural network architectures.

- another missing relevant citation is Gai et al., "Gradient-based learning for F-measure and other performance metrics". This paper also considers gradient-amenable surrogates for non-decomposable performance measures (in the multi-class setting).

**Hyperparameters** The method requires specifying two hyperparameters that control the shape of the sigmoid. These are chosen based on a holdout set. Different values appear to be optimal for different datasets. It would be ideal if a single choice was uniformly performant across datasets, since otherwise effectively one is using a different loss function for different tasks. Similar to Figure 3, it would be of interest to see whether there is a single choice that is uniformly performant (even if not optimal) across the datasets.

**Connection to (Eban et al., 2017)** The idea has quite some conceptual similarity to that of Section 3 in (Eban et al., 2017). The latter is for binary rather than multi-label problems, but it certainly appears as if the proposed algorithm is a direct translation of the idea of using a surrogate loss approximation to the TPR/FPR.

**Minor comments**

- Definition 1, the notation "P(yji | y1i, . . . , yj−1i = 1|x)" is confusing; why is there double conditioning?
- Table 1, it would be clearer to include author names rather than years; or, if space is a consideration, to just remove the year


### Summary of requested changes

- Add text in Introduction to acknowledge upfront that the idea of using smooth approximations to the TP/FP rate being standard (see also references below), *but* that the application of such an idea to multi-label problems has not been systematically explored, particularly in the case of modern deep networks

- Update bibliography with provided references, and discuss as appropriate

- Discuss similarity to Section 3 of (Eban et al., 2017)

- Discuss viability of using a consistent set of hyperparameters across datasets; potentially augment Figure 3 with such analysis

- Discuss more the viability of threshold-based methods; ideally, compare against these as well

Decision: Accept (poster)