Two-way Multi-Label Loss
Takumi Kobayashi†‡†National Institute of Advanced Industrial Science and Technology, Japan‡
University of Tsukuba, Japan
takumi.kobayashi@aist.go.jp

Abstract
A natural image frequently contains multiple classification targets, accordingly providing multiple class labels
rather than a single label per image. While the single-label
classification is effectively addressed by applying a softmax
cross-entropy loss, the multi-label task is tackled mainly ina binary cross-entropy (BCE) framework. In contrast to thesoftmax loss, the BCE loss involves issues regarding imbalance as multiple classes are decomposed into a bunch ofbinary classifications; recent works improve the BCE loss
to cope with the issue by means of weighting. In this paper, we propose a multi-label loss by bridging a gap between the softmax loss and the multi-label scenario. The
proposed loss function is formulated on the basis of relative
comparison among classes which also enables us to further improve discriminative power of features by enhancing classification margin. The loss function is so flexible
as to be applicable to a multi-label setting in two ways fordiscriminating classes as well as samples. In the experiments on multi-label classification, the proposed method
exhibits competitive performance to the other multi-label
losses, and it also provides transferrable features on singlelabel ImageNet training. Codes are available at https://github.com/tk1980/TwowayMultiLabelLoss.

1. Introduction
Deep neural networks are successfully applied to supervised learning [12, 26, 38] through back-propagation basedon a loss function exploiting plenty of annotated samples.
In the supervised learning, classification is one of primarytasks to utilize as annotation a class label to which an image
sample belongs. As in ImageNet [7], most of image datasets
provide a single class label per image, and a softmax loss is
widely employed to deal with the single-label annotation,
producing promising performance on various tasks.
The single-label setting, however, is a limited scenario
from practical viewpoints. An image frequently contains
multiple classification targets [39], such as objects, requir-

ing laborious cropping to construct single-label annotations.
There are also targets, such as visual attributes [25], which
are hard to be disentangled and thereby incapable of producing single-label instances. Those realistic situations poseso-called multi-label classification where an image sample
is equipped with multiple labels beyond a single label.
While a softmax loss works well in a single-label learning, the multi-label tasks are addressed mainly by applying
a binary cross-entropy (BCE) loss. Considering multiple
labels are drawn from C class categories, the multi-label
classification can be decomposed into C binary classification tasks, each of which focuses on discriminating samples
in a target class category [28]; the BCE loss is well coupledwith the decomposition approach. Such a decomposition,
however, involves an imbalance issue. Even in a case ofbalanced class distribution, the number of positive samples
is much smaller than that of negatives, as small portion ofwhole C-class categories are assigned to each sample as annotation (positive) labels. The biased distribution is problematic in a naive BCE loss. To cope with the imbalanceissue in BCE, a simple weighting approach based on classfrequencies [25] is commonly applied and in recent years it
is further sophisticated by incorporating adaptive weighting
scheme such as in Focal loss [18] and its variant [2]. On
the other hand, the softmax loss naturally copes with multiple classes without decomposition nor bringing the abovementioned imbalance issue; it actually works well in thebalanced (single-label) class distribution. The softmax loss
is intrinsically based on relative comparison among classes(3) which is missed in the BCE-based losses, though being
less applicable to multi-label classification.
In this paper, we propose a multi-label loss to effectively deal with multiple labels in a manner similar to thesoftmax loss. Through analyzing the intrinsic loss function of the softmax loss, we formulate an efficient multilabel loss function to exploit relative comparison between
positive and negative classes. The relative comparison isrelated to classification margin between positive and negative classes, and we propose an approach to enlarge themargin by simply introducing temperature on logits for fur-ther boosting performance. The proposed loss function isregarded as a generalization of the softmax loss, being reduced into the softmax loss in case of a single-label task.
Thus, the general loss formulation enables us to measurelosses in two ways for computing multi-label classification
loss not only at each sample but also for each class to discriminate samples on that class as the BCE focuses on. Insummary, our contributions are three-fold as follows.

learning both of which cope with multiple positive and negative pairs. The point is that we derive our multi-label loss
through theoretically analyzing a single-label softmax loss,
while the similarity losses [27, 31] are formulated based onrather heuristic pair-wise comparison among positive andnegative similarities; we discuss the difference in Sec. 3.

• We formulate a new loss function to deal with multiple
labels per sample while enhancing classification margin.
• By using the loss function, we propose a two-way approach for measuring multi-label loss.
• The loss is thoroughly analyzed from various aspects inthe experiments and exhibits competitive performance,compared to the other multi-label losses. It also provides
transferrable features on single-label ImageNet training.

We first analyze a softmax cross-entropy from a viewpoint of a single-label loss function and then derive a newloss to effectively cope with multiple labels.

1.1. Related works
Multi-label classification: In recent years, the multi-label
task is addressed in a deep learning framework by improving models from an architectural viewpoint. Relationships
among multiple labels assigned to an image are exploited
such as by recurrent neural network [30] and graph neuralnetworks [8] which also leverage external word-related information [4, 32] to extract semantic characteristics of class
categories. Regional features are also utilized [39] whiletransformer-based spatial attention mechanism is incorporated to detect label-related region [33]. In this paper, we focus on a loss function which is an orthogonal direction to thearchitectural approaches; it would compensate the abovementioned methods which simply employ a BCE loss.
BCE loss: In the multi-label framework, a binary crossentropy (BCE) loss plays a key role through decomposing multi-class classification into multiple class-wise binarytasks. For establishing multi-label losses, research effort ismainly devoted to improving the BCE especially in termsof the imbalance issue mentioned above. Frequency-based
weighting [25] is widely applied as a naive extension of
BCE, and in recent years, sophisticated adaptive weighting
schemes are proposed [2, 18]. There are also works to improve BCE so as to cope with particular situations such asclass imbalance [35] and partial labeling [1]. In contrast,
we derive the proposed method in a framework of a softmax
loss, apart from the BCE formulation. Thus, our method bypasses the imbalance issue tackled by [2, 18] while enhancing discriminativity in terms of both classes and samples.
Metric learning: From a viewpoint of formulation, ourmethod is related to the loss functions applied to optimize
similarity in the literature of metric learning which is different from the multi-label classification. By regarding a classifier vector as a proxy of the class, one could find resemblance between the similarity learning and the multi-label

2. Method

2.1. Softmax cross-entropy loss for single label
In single-label supervised learning, a softmax crossentropy provides an effective loss function. Suppose an image sample I equipped with a class label y ∈ {1, · · · , C}.
The image is processed by a (neural network) model fθ parameterized by θ to produce a logit vector x = fθ (I) ∈RC . The softmax loss is formulated by means of crossentropy between one-hot label y and softmax of logits x asexy
`sm = − log PC

c=1 e

xc

,

(1)

where xc indicates the c-th element of a logit vector x. Following [15], we reformulate it toward a loss-like form akinto hinge loss [6, 29];




C
X
X
exc = log 1 + e−xyexc`sm = log e−xy
c=1

(2)

c6=y

 X


= softplus log
exc − xy ,

(3)

c6=y

where softplus(·) = log[1 + exp(·)] is a softplus function,smooth approximation of a hinge function. In this form,
the log-sum-exp resembles a maximum operator
to providePa hard logit over negative classes as log c6=y exp(xc ) ≈maxc6=y xc [15]. This reformulation (3) reveals that a softmax loss measures difference
between the positive xy andPthe hard negative log c6=y exp(xc ) via a softplus function.

2.2. Multi-label loss
We then consider a multi-label setting where a logit vector x is associated with multiple labels P, a set of positive labels assigned to x; the number of positive labels is
1 ≤ |P| < C and we denote a set of negative labels by N
such that |P ∪ N | = C and P ∩ N = ∅. For correctly identifying the positive classes P on multi-label classification,
we encourage the logits x to have the relationship of
xp > xn , ∀p ∈ P, ∀n ∈ N ⇔ min xp > max xn , (4)
p∈P

n∈N

where xp and xn indicate logits of positive and negativeclasses, respectively; we refer to them as positive and negative logits. The analysis in Sec. 2.1 inspires us to formulate
a multi-label loss by measuring the difference between thehard positive and hard negative logits in (4) via softplus.
Since the softmax loss (3) leverages log-sum-exp to represent negative logits, our particular interest is to effectively
describe the hard positive of minp∈P xp . We follow the approach to replace the min operator by log-sum-exp. The
minimum of positive logits can be written by using max
operator so that log-sum-exp is applied as follows.
X(5)
min xp = − max(−xp ) ≈ − log
e−xp .
p∈P

p∈P

p∈P

We embed this hard positive into a softplus loss function(3) by replacing the single positive logit xy to construct themulti-label loss of
 X

X

xn
−xp
˜
` = softplus log
e
+ log
e
(6)
n∈N

(7)

To enhance the discriminative relationship (4), a margin
for multi-label classification is naturally defined as∆ , min xp − max xn .
p∈P

(8)

n∈N

The multi-label loss (6) implicitly enlarges the margin due
to log-sum-exp which has the following property.
Proposition 1. The difference between hard positive andnegative in (6) is larger than the negative margin, −∆:XX
log
exn + loge−xp > max xn − min xp . (9)
p∈P

n∈N

p∈P

Proof. A log-sum-exp function is rewritten as


X
X
log
exn = xn∗ + log 1+
exn −xn∗ > xn∗ , (10)
n6=n∗

n∈N

P
where n∗ = arg maxn∈N xn and n6=n∗ exn−xn∗ > 0. Similarly, we have
X
(11)
log
e−xp > max(−xp ) = − min xp .
p∈Pnegative

Figure 1. Our multi-label loss function enlarges a margin between
positive and negative logits by a temperature parameter T .

Pshown
in (10), that is, log c exc − maxc xc = log(1 +P
xc −xc∗). We thus introduce temperature T into
c6=c∗ elog-sum-exp for enlarging the gap as follows.
Proposition 2. A temperature
T > 0 reformuP parameterxc
lates log-sum-exp as T log c e T which has
max xc < T log
c

X

xc

e T ≶ logcX

exc for T ≶ 1. (12)cp∈P

2.3. Classification marginn∈N

positive

p∈P


X
 X

xn
−xp
= log 1 +
e
·
e
.
n∈N

marginp∈Pp∈P

Thus, the margin in the multi-label loss (6) is underestimated toward larger margin through minimizing the loss as
in large-margin methods [6, 29].
Based on the margin-based analysis, we further enhancelarge-margin effect in the multi-label loss for improving discrimination. The implicit margin in (6) resorts to the gap

Proof. The tempered log-sum-exp is written byX xc −xc∗ 
X xc
e T
,
T log
e T = xc∗ + T log 1 +

(13)

c6=c∗

c

where the second term monotonically increases w.r.t. T > 0
as c∗ = arg maxc xc and xc − xc∗ ≤ 0, ∀c 6= c∗ .
Thereby, we propose the following multi-label loss.
X xn
X − xp ` = softplus TN log
e TN +TP log
e TP , (14)
n∈N

p∈P

where two temperatures TP and TN are applied to positive
and negative logits, respectively. While such a temperature
is incorporated into softmax in the field of knowledge distillation [13], in this study, the temperature controls a margin in the multi-label loss as shown in Fig. 1. Particularly,
a margin is further underestimated by T > 1 for inducing larger-margin classification. As to negative logits, however, the number of negatives is generally larger than thatof positives,P|N |  |P|, and thereby sufficiently large gapof log(1 + n6=n∗ exn −xn∗ ) can be given in (10) even for
TN = 1, inducing large margin on the side of negatives.
Therefore, the temperature parameters can be reduced into
only a positive one TP by TN = 1 in (14) for considering the larger margin only on positive logits via TP ≥ 1 asshown in Fig. 1; it is empirically discussed in Sec. 4.2. Itshould be noted that the proposed loss (14) enhances classification margin without imposing regularization on logits [15] nor manipulating logits [20].

2.4. Two-way multi-label loss

classes

We have analyzed a loss function from a viewpoint ofsample-wise classification and then formulated the multilabel loss function (14) to work at each sample. On theother hand, a BCE loss commonly applied in the multilabel classification considers rather to discriminate samples
at each class by means of a sigmoid function, which canbe regarded as class-wise classification. These two types ofapproaches deal with classification in orthogonal directions,and in this study we unify them into two-way loss for multilabel classification on the basis of a logit matrix shown in
Fig. 2; in the matrix, the sample-wise classification is performed at each row while the class-wise one is along eachcolumn, which are thus regarded as two-way classifications.
Notation. Suppose a bucket1 of M image samples. As
shown in Fig. 2, it produces a logit matrix X ∈ RM ×C inwhich the (i, c)-th element xic indicates the logit producedby fθ (Ii ) for the c-th class. Accordingly, a set of label yic ∈{0, 1} constitutes a binary label matrix Y ∈ {0, 1}M ×C .
Sample-wise way: In Sec. 2.2, we have considered asample-wise loss by applying the multi-label loss function
to row-wise logits {xic }C
c=1 as


X
X
x
xin
− Tip
`i = softplus log
e + T loge
, (15)
n|yin =0p|yip =1

which enhances a discrimination among classes enforcing(4) to improve classification accuracy of top-K prediction
at the i-th sample. On the other hand, it pays less attention
to the relationship among samples such as xic and xjc at
the c-th class, due to which a learnt model might fail todiscriminate samples on the c-th class. It leads to inferior
performance on the metric of mAP across classes, a popular
performance metric in multi-label classification. This is themain reason why the popular softmax loss is less frequently
applied to multi-label classification.
Class-wise way: To enhance the discrimination among
samples, we apply the multi-label loss function (14) in aclass-wise way, i.e., column-wise manner in Fig. 2, to produce the loss of


X
X
x
c
xic
− Tjc
` = softplus log
e + T loge
. (16)
i|yic =0j|yjc =1

It measures separation between positive and negative samples in the c-th class, promoting the relationship ofminj|yjc =1 xjc > maxi|yic =0 xic to improve precisionat each class. It is noteworthy that we apply the sameloss function as (15) just by feeding column-wise samples{xic }M
i=1 . While a pair-wise comparison xjc > xic is considered in a SVM framework for shallow models [36], we
1 We use a term of bucket to distinguish from batch in deep learning. B
samples in a batch can be divided into k buckets of M = B/k samples.

positive logit

sample-wisesamplesclass-wise

Figure 2. Logit matrix on a bucket of M images.

leverage our multi-label function (14) to effectively formulate the column-wise classification loss with a proper margin induced by a temperature T , working in end-to-end
learning of deep models.
We finally formulate our two-way multi-label loss `¯ by
M
C
1X c
1 X
`i ({xic , yic }C;T)+` ({xic , yic }M`¯ =c=1
i=1 ; T ).
M i=1
C c=1
(17)
The proposed two-way loss contributes to improving both
sample-wise and class-wise discrimination. It shouldbeP
noted that even in a case of single (one-hot) label c yic =1, ∀i, the class-wise samples demand multi-label classification (column in Fig. 2). Thus, this two-way formulation is
intrinsically coupled with the multi-label loss function (14).

3. Discussion
3.1. Loss function
Softmax loss: It is possible to apply a softmax loss in a
multi-label scenario through describing labels in a probabilistic way,Xexcỹc log P x 0 ,(18)`ce = −cc0 ecwhere ỹc = P y0cy 0 is a normalized label. This enforcesccthe positive logits {xp }p∈P to exhibit the same value so as
xp
1
to produce Pe exc = |P|, ∀p ∈ P. It, however, is lesscrelevant to the multi-label classification (4), being a meaningless constraint on feature representation. In contrast, theproposed loss (14) lets the positive logits take any higher
values xp > xp∗ by paying attention to the hard positive
logit. It also coincides with the softmax loss in case of
single-label classification, i.e., |P| = 1.
BCE loss: The multi-label classification is addressed
mainly by a binary cross-entropy (BCE) loss ofX
11−(1−yc ) log, (19)`bce = −yc log
−xc
1
+
e
1
+
exc
c
which can be contrasted with our multi-label loss in the following three points.

First, the BCE loss is faced with imbalance issue due
to |P|  |N |. To mitigate it, some weighting schemes
are proposed in a simple statistical manner [25] and a sophisticated adaptive manner [2, 18]. On the other hand,the proposed loss (17) is based on comparison of logits
among classes and samples in two ways through excavating
the hard negatives and positives by means of log-sum-exp,
which naturally alleviates the imbalance issue.
Second, BCE (19) is based on a point-wise loss without paying much attention to relationships among classes
and samples, which thus leads to the above-mentioned issue. The proposed loss effectively exploits the discriminative characteristics through relative comparison amongclasses and samples.
Third, the point-wise loss in BCE stems from a sigmoid function which provides classification on the basis ofzero, i.e., x ≶ 0, which is regarded as a constraint in theclassification model. The proposed loss based on relative¯
comparison is actually invariant to logit shift2 , `(X,Y) =¯`(X + , Y ), without imposing constraints on logits, which
thereby enables flexible feature representation learning.
Metric learning loss: In the literature of metric learning,a multi-similarity loss [31] is formulated through relative
similarity weighting as




X
X
`ms = log 1 +e−xp + log 1 +exn , (20)
p∈P

n∈N

where a logit xc is regarded as a similarity to the c-th class.
It is rewritten by using log-sum-exp with a BCE loss (19) to
n

o
X
X
−xp
xn
`ms = `bce − log
e
, log
e
, {+1, −1} .
p∈P

n∈N

(21)
This reformulation clarifies that the multi-similarity
loss [31] considers logit relationships within the respective
groups of positives and negatives to partially mitigate theabove-mentioned issues of BCE, though inheriting the zerobasis constraint.
In the metric learning, a circle loss [27] to learn similarities is also presented in a similar form to our loss (7).
The circle loss
from maximizing pair-wise disP is derived
P
crepancy of p∈P n∈N exn −xp in a rather heuristic manner. In contrast, we theoretically formulate the multi-labelloss (14) through analyzing a softmax loss from the viewpoint of a margin-aware loss function. Thus, the tempered
log-sum-exp is introduced into the loss to adaptively control
margin based on the logit distribution for enhancing classification margin. In [27], large-margin discrimination is realized by means of logit bias as xp −P and/or xn +N . In themulti-label scenario, however, due to the above-mentioned
invariance against logit shift, the logit biases are reduced
2 Details are shown in the supplementary material.

into a constant bias2  = P + N in a softplus function (6);such a bias simply increases the loss in disregard of logits,
thus less contributing to larger-margin classification.

3.2. Two-way formulation
Class-wise way: In the proposed two-way loss, the classwise loss is especially distinctive in comparison to a standard softmax loss (1) which considers only sample-wise
relationships. It resembles contrastive learning in selfsupervised learning [11]; from that view, the target c-thclass vector is trained so as to be close enough to the samples associated with the c-th class while being away fromthe other samples. Thus, the proposed two-way loss would
contribute to favorable feature representation learning.
Joint way: The proposed loss (17) compares logits in
respective two ways of class- and sample-wise directions (Fig. 2). It is also conceivable to contrast positive
logits with negatives in a joint manner for encouraging
minj,p|yjp =1 xjp > maxi,n|yin =0 xin , which leads to


X
X
xjp
`j = softplus log
exin + T log
e− T . (22)i,n|yin =0j,p|yjp =1

The joint loss is different from (15, 16) in that summationindexes run over all the elements in a logit matrix. As thejoint loss measures two-way discrimination at once by using
only a single loss function, it would be vulnerable to irregular logit values, such as too large negative and/or small positive logits, which dominate the loss via log-sum-exp function. We empirically compare the proposed two-way loss
with the joint loss (22) in Sec. 4.2.

4. Results
We apply the proposed loss to train CNNs on multi-label
image classification. Performance of multi-label classification is measured based on the following two metrics.
mAP@class: Average precision over N samples is computed at each class and then is aggregated across classes byC

1 X
AP({xic , yic }Ni=1 ),
C c=1(23)

where AP(·) computes average precision of input samples.
This is a popular metric in the multi-label classification.
mAP@sample: In contrast to mAP@class, we measuremean average precision over samples byN

1 X
AP({xic , yic }Cc=1 ).
N i=1(24)

It is connected to classification accuracy, a standard metricfor single-label classification; AP at each sample summarizes top-K accuracies over Ks.

Table 1. Performance results with various temperatures. The performances of vanilla setting, TP = 1 and TN = 1, are underlined.

TN

0.5

mAP@class
1
2

→0
0.5
TP 1
2
4

71.03
72.20
72.85
73.42
73.56

71.58
72.30
72.93
73.66
74.11

71.88
72.17
72.56
73.23
73.90

4

0.5

mAP@sample
1
2

71.86
71.95
72.20
72.53
73.24

85.01
85.72
86.07
86.27
86.26

85.37
85.81
86.14
86.48
86.66

85.55
85.68
85.94
86.34
86.66

4
85.47
85.51
85.66
85.85
86.23

Table 2. Performance results by applying logit bias  [27].


0.5

1

2

4

8

mAP@classmAP@sample

73.08
86.19

73.16
86.22

73.28
86.25

73.44
86.21

73.18
86.02

4.1. Training procedure
We finetune ImageNet-pretrained CNN models on a target dataset with a multi-label loss. SGD with momentum of
0.9 and weight decay of 10−4 is applied to a batch of 512
samples over 40 training epochs with a cosine-scheduledlearning rate starting from 0.01 for the FC classifier and0.001 for the other layers. We apply distributed training
across 4 GPUs of NVIDIA V100, which produces 4 buckets
of M = 512/4 = 128 samples to construct a logit matrix(Fig. 2); the bucket-level losses are averaged in the batch.

4.2. Ablation study
We first analyze the method from various aspects
through the following ablation studies, and then in Sec. 4.3
compare the method to the other loss functions on various
multi-label tasks. The ablation study is conducted on MSCOCO [19] using ResNet-50 [12].
Temperature T : In the loss (14), temperature parameters
TP and TN are applied to enhance classification margin on
positive and negative logits, respectively. Tab. 1 shows performance results across various temperatures. As discussed
in Sec. 2.3, performance is improved by increasing the positive temperature TP to enlarge margin on the positive side.
On the other hand, the negative temperature provides favorable performance by TN = 1, as the number of negative logits is larger than that of positives, in this case ×30(Tab. 6), to provide a sufficient margin even by TN = 1.
Thus, we apply TN = 1 and TP = T = 4 in (15, 16).
We also evaluate the naive hard positive logit ofminp∈P xp by simply setting TP → 0. It significantly
degrades performance as shown in the top row of Tab. 1.
Such an extreme operator produces sparse backward update
as well as provides no margin in the loss. This comparison clarifies that log-sum-exp function effectively produces
hard positive logit with a proper margin via T .

Table 3. Performance regarding ways to compute multi-label loss.

way

both

(a) Ours
class sample

bothmAP@classmAP@sample

74.11
86.66

73.06
82.75

69.19
84.33

67.18
86.07

(b) Softmax
class sample
68.13
69.15

58.00
83.60

Table 4. Performance results by other related loss functions.

TP
mAP@classmAP@sample(a) Multi-sim loss (20)
1
2
4
70.36
85.42

70.94
85.75

71.27
85.95

(b) Joint-way (22)
1
2
4
70.19
84.82

71.71
85.57

71.82
85.64

Table 5. Performance about bucket size M in a batch size of 512.
M

16

32

64

128

256

512

mAP@classmAP@sample

70.76
85.79

72.84
86.33

73.68
86.60

74.11
86.66

74.15
86.58

74.01
86.47

Logit bias: In [27], the margin is also discussed throughadding bias to logits, which is eventually described by asingle bias parameter  to shift the logit difference in a softplus function as described in Sec. 3.1. Tab. 2 shows performance results of various bias . The logit bias slightly
improves performance by  = 4, though being inferior to
our temperature approach of TP = 4 (Tab. 1). As shownin (13), the temperature T adaptively controls margin basedon the logit distribution while a bias  constantly affects logits in the softplus function, less contributing to large-marginclassification during end-to-end learning.
Ways: We then analyze the ways to apply the multi-labelloss function (14) to a logit matrix. The proposed two-way
approach (17) is compared to class-wise (16) and samplewise (15) losses as shown in Tab. 3a. Those two approaches
improve class-wise and sample-wise performances, respectively. By combining those two ways into the loss (17),performance is significantly improved to outperform the respective approaches. The feature representation is effectively learned from these two perspectives to discriminate
both samples and classes.
Loss function: The proposed loss is compared to the related loss functions of softmax loss (18), multi-similarity
loss (20) [31] and joint-way loss (22), which are mentioned
in Sec. 3. As in our loss, the softmax loss is applicablein two ways to improve performance as shown in Tab. 3b,
though being inferior to ours (Tab. 3a). Particularly, oneway softmax loss significantly degrades performance on the
counter-metric; e.g., performance of class-way loss deteriorates at the metric of mAP@sample. This result indicates that the implicit constraint of uniform positive logits(Sec. 3.1) in the softmax loss would make the feature repre-

Table 6. Multi-label image datasets.
Dataset# classes# training samples# test samples# label per sample

MSCOCO [19]VISPR [23]VAW [25]WIDER [17]

VOC2007 [9]VOC2012 [9]8082,08140,1372.9

68
14,167
8,000
5.2

620
229,076
31,8191.8

1428,34029,117
2.9

20
5,011
4,952
1.4

20
5,717
5,823
1.4

sentation overly fit the target performance metric while impeding generalization. In contrast, the proposed multi-labelloss is suitable for the two-way approach without imposing
any constraints, to improve performance.
As discussed in Sec. 3.1, the multi-similarity loss(20) [31] evaluates logits on the basis of zero point andthus is directly applicable to multi-label classification as ina BCE loss. For fair comparison, we apply the proposed
temperature scaling to the loss as shown in Tab. 4a. The
multi-similarity loss works well on both the performance
metrics due to the zero-basis measurement. It, however, issignificantly inferior to our two-way loss. This comparison
clarifies the efficacy of our two-way approach to relatively
compare logits without introducing a fixed zero-basis.
The joint loss (22), a variant of our loss, produces poor
performance as shown in Tab. 4b. It would be hard to compare diverse logits including inconsistent ones on different
samples for different classes, xic and xjd (i 6= j, c 6= d), ina single loss function at once. On the other hand, the proposed two-way approach measures losses along either ofclass-wise and sample-wise ways by fixing the other way,
which contributes to consistent loss measurement with robustness against irregular logits.

4.3. Performance comparison

Bucket size M : The class-wise loss (16) is computed ona bucket of M samples which are randomly drawn trhoughrandom mini-batch sampling. We investigate performancesacross various bucket sizes. In this case, 512 samples ina batch are grouped into k buckets each of which contains
M = 512/k samples where k ∈ {1, 2, 4, 8, 16, 32} produces M ∈ {512, 256, 128, 64, 32, 16}; it could be naturally and efficiently implemented by distributed training on
k GPUs. It should be noted that we fix the batch size to 512
so that the bucket size M affects only the class-wise loss(16) in the two-way loss (17). As shown in Tab. 5, the scoreof mAP@sample is less sensitive to M since the samplewise loss (15) responsible for the metric is irrelevant to thebucket size. On the other hand, mAP@class is improved bylarger M , implying that it would be somehow difficult to explore discriminative features from a smaller number of samples; as M → 1, the proposed two-way loss is reduced to thesample-wise loss disregarding discrimination over samples.
Discriminativity among samples is enhanced by moderate
size of bucket to boost performance on both metrics in our
two-way framework. We use M = 128 throughout the experiments while M ≥ 64 produces favorable performance.

4.4. ImageNet of single-label dataset

We compare the proposed loss with the others by using various CNN models on diverse datasets. We employ
ResNet-50 [12], ResNeXt-50 [38], DenseNet-169 [14] andRegNetY-32gf [26]. Performances are reported on datasetsof MS-COCO [19], VISPR [23], VAW [25], WIDER Attribute [17] and VOC-2007/2012 [9], which are summarizedin Tab. 6. The comparison methods include softmax crossentropy loss (18), binary cross-entropy (BCE) loss weighted
by class frequencies [25], Focal loss [18] and asymmetricloss (ASL) [2]; we set the hyper-parameters to γ = 2 inFocalLoss and {γ+ = 0, γ− = 4, m = 0.05} in ASL as suggested in [2, 18]. For fair comparison, we apply the same
training protocol as shown in Sec. 4.1; only the loss functions are compared by fixing the other components.
The performance results are shown in Tab. 7, demonstrating efficacy of our loss on diverse datasets with various
CNN models. While the softmax loss works well only on
the metric of mAP@sample, the proposed two-way loss issuperior on both metrics, outperforming the others. Someother experimental results are shown in the supplementarymaterial.

Finally, we apply the loss to train ResNet-50 [12] on ImageNet [7] posing single-label classification; the training
protocol is the same as Sec. 4.1 except that the model istrained from scratch over 90 epochs with an initial learning
rate of 0.1. In the ImageNet, the proposed two-way loss differs from a softmax loss in that the class-wise loss (16) isincorporated, while the sample-wise loss (15) is reduced tothe softmax loss (1) in this single-label learning.
To evaluate the feature representation learnt by thelosses, we measure classification accuracy not only on ImageNet validation set but also on the other datasets by following the transfer learning scheme; pretrained ResNet-50
backbone is applied as frozen feature extractor and only an
FC classifier is tuned on the downstream tasks, details of
which are shown in supplementary material. Tab. 8 reports
classification accuracies of ASL [2], softmax (1) and ourloss (17) on various datasets of single-label classification.
While the three losses produce similar performances on
ImageNet, one can find difference among the transfer learning performances. ASL is inferior to the other two losses

Table 7. Classification accuracies (%) on diverse datasets with various CNNs.
mAP@class
CNN ResNet50 ResNeXt50 DenseNet169 RegNetY32gf

mAP@sample
ResNet50 ResNeXt50 DenseNet169 RegNetY32gf

MSCOCO [19]Softmax
BCE
Focal [18]ASL [2]
Ours

58.00
67.71
69.42
70.92
74.11

59.53
69.68
71.33
73.04
75.44

58.21
64.04
67.18
69.25
73.51

64.14
73.38
74.99
76.70
79.57

83.60
79.65
84.38
85.05
86.66

84.46
80.62
85.22
86.06
87.11

83.13
76.14
83.33
84.40
86.62

86.92
83.21
87.51
88.29
89.54

VISPR [23]Softmax
BCE
Focal [18]ASL [2]
Ours

36.61
44.22
46.89
48.53
51.89

36.97
45.34
47.76
49.53
52.79

28.64
39.73
40.78
42.61
48.57

36.79
46.11
48.75
51.03
53.75

85.23
72.39
84.35
84.81
85.64

85.43
73.14
84.26
84.99
85.40

83.75
69.31
82.91
83.99
85.88

85.90
73.75
85.29
86.15
86.67

VAW [25]

Softmax
BCE
Focal [18]ASL [2]
Ours

52.59
51.21
54.38
55.39
56.42

53.33
51.31
54.50
55.72
57.00

47.30
44.53
48.94
48.17
54.28

55.02
52.25
56.77
57.88
59.33

77.68
72.43
77.66
78.05
78.81

78.09
72.29
77.70
78.32
78.95

75.97
66.50
75.81
75.91
78.36

78.99
72.43
78.71
79.03
80.07

WIDER [17]Softmax
BCE
Focal [18]ASL [2]
Ours

63.91
70.16
65.88
67.99
72.28

65.14
71.40
67.29
69.71
72.77

63.61
70.16
64.49
67.34
73.03

66.47
73.26
68.72
71.11
74.92

83.09
77.62
82.27
83.44
85.43

83.74
78.56
82.92
84.12
85.43

83.02
77.36
81.89
83.38
85.87

84.65
79.94
83.66
85.00
86.97

VOC2007 [9]Softmax
BCE
Focal [18]ASL [2]
Ours

83.49
85.58
85.59
86.70
89.04

84.31
86.65
86.27
87.53
89.57

82.53
81.96
78.33
81.44
88.67

86.63
88.25
87.87
89.26
91.44

93.23
91.44
93.04
93.42
94.44

93.66
91.91
93.24
93.80
94.53

92.61
87.55
89.83
91.05
94.13

94.99
92.62
94.56
95.08
95.72

VOC2012 [9]Softmax
BCE
Focal [18]ASL [2]
Ours

82.46
85.56
85.59
86.56
88.12

83.37
86.56
86.04
87.09
88.72

81.56
81.94
79.41
81.83
87.95

86.48
88.27
87.99
89.06
91.01

93.65
92.60
93.56
94.01
94.38

93.81
93.01
93.74
94.19
94.73

92.72
89.63
90.65
92.18
94.28

95.23
94.05
95.17
95.46
96.04

Table 8. Classification accuracies (%) on various datasets by transferring ImageNet-pretrained ResNet-50 features.
Dataset ImageNet Aircraft [21] Caltech101 [10] Car [16] CUB [34] DTD [5] Flower [22] Food101 [3] Pets [24] SUN [37]
ASL [2]
Softmax
Ours

76.76
76.32
76.29

27.42
39.18
44.01

85.22
88.17
88.36

32.99
45.13
46.50

of softmax and ours; it degrades performance especially on
Aircraft dataset3 . As discussed in Sec. 3.1, ASL which isa variant of BCE loss imposes the zero-basis constraint on
the logits in a sigmoid function. It might lead to over-fitting
toward the primary (ImageNet) task, hampering generalization of the learnt features. On the other hand, the losses ofsoftmax and ours based on relative comparison among logits let features be flexibly learned to enhance generalization
performance. The proposed loss further enhances the discriminative power of feature representation through comparison along two directions of classes and samples in the
3 Aircraft dataset [21] poses fine-grained discrimination of aircraft ap-

pearances. Since the ImageNet pre-training task pays less attention to thosevisual features, the performance comparison on that dataset might highlight difference in general discriminative power of feature representations.

55.26
63.15
65.96

64.89
70.85
72.71

75.88
85.75
87.72

57.08
65.44
66.83

91.33
92.12
92.18

53.38
58.85
59.49

two-way formulation, which leads to better performance,such as on Aircraft dataset, as shown in Tab. 8.

5. Conclusion
We have proposed a novel loss to cope with multiple labels. The multi-label loss function is theoretically formulated in a margin-aware form through analyzing the softmax
loss. Then, it is effectively applied in the two-way manner to finally construct multi-label loss for improving bothclass-wise and sample-wise performance. The experimental results show that the proposed loss is effective not only
for improving performance on multi-label classification but
also for providing transferrable features on single-label ImageNet pre-training.

References[1] Emanuel Ben-Baruch, Tal Ridnik, Itamar Friedman, Avi
Ben-Cohen, Nadav Zamir, Asaf Noy, and Lihi ZelnikManor. Multi-label classification with partial annotations using class-aware selective loss. In CVPR, pages 4764–4772,
2022. 2[2] Emanuel Ben-Baruch, Tal Ridnik, Nadav Zamir, Asaf Noy,
Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor.
Asymmetric loss for multi-label classification. In ICCV,
pages 82–91, 2021. 1, 2, 5, 7, 8
[3] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.
Food-101 – mining discriminative components with random
forests. In ECCV, pages 446–461, 2014. 8[4] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen
Guo. Multi-label image recognition with graph convolutional networks. In CVPR, pages 5177–5186, 2019. 2[5] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy
Mohamed, and Andrea Vedaldi. Describing textures in thewild. In CVPR, 2014. 8[6] Koby Crammer and Yoram Singer. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of Machine Learning Research, 2:265–292, 2001. 2, 3
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,
and Li Fei-Fei. Imagenet: A large-scale hierarchical image
database. In CVPR, pages 248–255, 2009. 1, 7
[8] Thibaut Durand, Nazanin Mehrasa, and Greg Mori. Learning a deep convnet for multi-label classification with partial
labels. In CVPR, pages 647–657, 2019. 2[9] Mark Everingham, Luc Gool, Christopher K. Williams, John
Winn, and Andrew Zisserman. The pascal visual object
classes (voc) challenge. International Journal of Computer
Vision, 88(2):303–338, 2010. 7, 8
[10] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In
Computer Vision and Pattern Recognition Workshop, 2004.
8[11] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR,
2006. 5[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In CVPR,
pages 770–778, 2016. 1, 6, 7
[13] Geoffrey E. Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NIPS 2014 Deep
Learning Workshop, 2014. 3[14] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, pages 2261–2269, 2017. 7[15] Takumi Kobayashi. Large margin in softmax cross-entropy
loss. In BMVC, 2019. 2, 3
[16] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.
3d object representations for fine-grained categorization. In
Workshop on 3D Representation and Recognition, 2013. 8[17] Yining Li, Chen Huang, Chen Change Loy, and Xiaoou
Tang. Human attribute recognition by deep hierarchical contexts. In ECCV, 2016. 7, 8

[18] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He,
and Piotr Dollár. Focal loss for dense object detection. In
ICCV, pages 2980–2988, 2017. 1, 2, 5, 7, 8
[19] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir
Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva
Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft
coco: Common objects in context. arXiv, 1405.0312, 2014.
6, 7, 8[20] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang.
Large-margin softmax loss for convolutional neural networks. In ICML, pages 507–516, 2016. 3[21] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B.
Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv:1306.5151, 2013. 8[22] Maria-Elena Nilsback and Andrew Zisserman. Automatedflower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image
Processing, 2008. 8[23] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Towards a visual privacy advisor: Understanding and predicting privacy risks in images. In ICCV, pages 3706–3715,
2017. 7, 8
[24] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, andC. V. Jawahar. Cats and dogs. In CVPR, pages 3498–3505,
2012. 8[25] Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. In CVPR, pages 13018–
13028, 2021. 1, 2, 5, 7, 8
[26] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick,Kaiming He, and Piotr Dollár. Designing network design
spaces. In CVPR, pages 10428–10436, 2020. 1, 7
[27] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang,
Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle
loss: A unified perspective of pair similarity optimization.
In CVPR, pages 6398–6407, 2020. 2, 5, 6
[28] Grigorios Tsoumakas and Ioannis Katakis. Multi-label classification: An overview. International Journal of Data Warehousing and Mining, 3(3):1–13, 2007. 1[29] Vladimir N. Vapnik. Statistical Learning Theory. Wiley,1998. 2, 3[30] Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang
Huang, and Wei Xu. Cnn-rnn: A unified framework for
multi-label image classification. In CVPR, pages 2285–2294,
2016. 2[31] Xun Wang, Xintong Han, Weiling Huang, Dengke Dong, andMatthew R. Scott. Multi-similarity loss with general pairweighting for deep metric learning. In CVPR, pages 5022–
5030, 2019. 2, 5, 6, 7
[32] Ya Wang, Dongliang He, Fu Li, Xiang Long, Zhichao Zhou,
Jinwen Ma, and Shilei Wen. Multi-label classification withlabel graph superimposing. In AAAI, pages 12265–12272,
2020. 2[33] Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, andLiang Lin. Multi-label image recognition by recurrently discovering attentional regions. In ICCV, pages 464–472, 2017.
2

[34] Peter Welinder, Steve Branson, Takashi Mita, Catherine
Wah, Florian Schroff, Serge Belongie, and Pietro Perona.
Caltech-UCSD Birds 200. Technical Report CNS-TR-2010001, California Institute of Technology, 2010. 8[35] Tong Wu, Qingqiu Huang, Ziwei Liu, Yu Wang, and Dahua
Lin. Distribution-balanced loss for multi-label classification
in long-tailed datasets. In ECCV, 2020. 2[36] Xi-Zhu Wu and Zhi-Hua Zhou. A unified view of multi-label
performance measures. In ICML, 2017. 4[37] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva,
and Antonio Torralba. Sun database: Large-scale scene
recognition from abbey to zoo. In CVPR, 2010. 8[38] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, andKaiming He. Aggregated residual transformations for deepneural networks. In CVPR, pages 5987–5995, 2017. 1, 7
[39] Hao Yang, Joey Tianyi Zhou, Yu Zhang, Bin-Bin Gao,
Jianxin Wu, and Jianfei Cai. Exploit bounding box annotations formulti-label object recognition. In CVPR, pages
280–288, 2016. 1, 2