Leveraged Asymmetric Loss with Disambiguation for Multi-label Recognition with One-Positive Annotations
Authors: Jingyi Cui
Tao Huang
Hanyuan Hang
Yisen Wang
James Kwok
Venue: Submitted to ICLR 2023 (2023)

Abstract: In the problem of multi-label learning from single positive labels (SPL), we learn the potential multiple labels from one observable single positive annotation. Despite many efforts to solve this problem, an effective algorithm with sound theoretical understanding is still in need. In this paper,  we propose a novel loss function for the SPL problem, called leveraged asymmetric loss with disambiguation (LASD), where we introduce a pair of leverage parameters to address the severe negative-positive imbalance. From the theoretical perspective, we analyze the SPL problem, for the first time, from the perspective of risk consistency, which links the SPL loss with losses for ordinary multi-label classification. We prove the consistency of our proposed LASD loss to the cost-sensitive Hamming loss, which provides guidance to the empirical choice of our proposed leverage parameters. In experiments, we demonstrate the effectiveness of our proposed LASD loss function over other state-of-the-art methods and empirically verify our theoretical results.

Under review as a conference paper at ICLR 2023
LEVERAGED ASYMMETRIC Loss WITH DISAMBIGUA- TION FOR MULTI-LABEL RECOGNITION WITH ONE- POSITIVE ANNOTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
In the problem of multi-label learning from single positive labels (SPL), we learn the potential multiple labels from one observable single positive annotation. Despite many efforts solve this problem, an effective algorithm with sound theoretical understanding is still in need. In this paper, we propose a novel loss function for the SPL problem, called leveraged asymmetric loss with disambiguation (LASD), where we introduce a pair of leverage parameters to address the severe negative- positive imbalance. From the theoretical perspective, we analyze the SPL problem, for the first time, from the perspective of risk consistency, which links the SPL. loss with losses for ordinary multi-label classification. We prove the consistency of our proposed LASD loss to the cost-sensitive Hamming loss, which provides guidance to the empirical choice of our proposed leverage parameters. In experiments, we demonstrate the effectiveness of our proposed LASD loss function over other state-of-the-at methods and empirically verify our theoretical results.
INTRODUCTION
Different from standard multi-class classification, where each instance is tagged with one target label,
multi-label classification (Liu al. 2022 Li et al. 2022) allows instance have multiple labels
and thus is applicable to wider real real- -world scenarios. For example, a picture can contain multiple
objects (Lanchantin et al.) [202]] Hu et al., (021), sentence can express multiple emotions (Huang
it al. 2021; et al. (2020), and song can belong to multiple genres (Shrivastava et al. 2020
Pellegrini & Masquelier 2021). Despite the wide applications of multi-label learning, the existence of multi-labels further increases the difficulty of annotating high-qualit labels (Deng et al. 2014).
On the one hand, label annotation can be extremely laborious and costly (Deng et al., 2014). On the other band, the small objects rare classes are often inevitably ignored by the human annotators (Liu et al. 202T] Wolfe et al. 2005). To deal with such problems, researchers loosen the requirements of label and propose the Parrtill Multi-Label (PML) paradigm, where the label of each instance can
be the subset its complete label set (Xic & Huang 2021] Yan & Guo 2021 Lyu et al.] 2021 Li
al. 20221. Recently, based PMI. classifiitia,,,, Cole al. (2021) take step further and present the paradigm called Single Positive Labels (SPL., where the data only provides one correct (positive)
label for each instance. simple example is provided to illustrate the the difference between the PML and SPL paradigms: for an image containing a sofa, chair, and a potted plant but without person or car, for PML paradigm, we are given the label: sofa (yes), chair (yes), person (no). others (unknown), while for SPL paradigm, we only know the label: sofa (yes), others (unknown). In fact, many real-world multi-class datasets are potentially multi-labeled, where SPL algorithms can be used directly to explore their underlying multi-labels. For example, the empirical study by Stock & Cisse (20T8) presents the multi-label nature images of ImageNet dateset (Russakovsky et al. (015). By using the previous datasets for multi-class problems, SPL can save massive amount of money, time, and labor resources on data collection.
The algorithmic study of SPL is relatively under-exploited. Cole et al. (222 first formally define the problem of SPL and proposes the adaptive SPL loss, which shows satisfactory experimental performance. However, it suffers from the shortcoming that one of its parammtaramstrss, the average number of positive labels, is hard to define and may vary according to data selection. Subsequently, Verelst al. (2022) and Zhou et al. (2022) also aim solve the SPL. problem, and introduce the

---

Under review as a conference paper at ICLR 2023
of spatial consistency loss and entropy maximization, respectively. However, there exists great demand for theoretical properties for SPL problems.
An alternative idea for solving SPL problems is utilize PML classifiers. The first category explores the relationship among labels and models the label correlations (Chen et al.) 2019: Durand et al. 2019 Huynhh & Elhamifar 2020). However, this category requires at least two labels per instance and thus is incompatible with the SPL problem. The second category turns the PML problem into solving an optimization problem (Sun al. 2010; Bucak et al., 2011 Cabral al. 2011] Xu et al. 2013). However, most of these algorithms only show promising performances on conventional datasets when there are sufficient positive labels per instance. In other words, directly utilizing these algorithms on SPL may induce severe performance degradation. Besides, one can also solve the SPL problem through positive-unlant unlabeled (PU) frameworks for multi-label learning (Sun et al., 2010 Hsieh et al.. 2015,Han et al. 20188] Kanehira & Harada 2016). Nonetheless, these methods are rarely explored in the SPL setting, and most of the works cannot be directly applied to large-scale multi-labeled image classification.
Under such conditions, aiming at solving the SPL problem, we propose a new loss function called leveraged asymmetric loss with disambiguation (LASD), which explicitly copes with the challenges of extreme label imbalance and label disabbiiiiiguttinnn To address the effectiveness of our proposed loss function, we for the first time use the concept of risk consistency show the relationship between loss functions for SPI. and that for fully supervised multi-label data. The contribution of this paper can be summarized follows:
We propose a novel loss function for the SPL problem, where we introduce pair of leverage parameters address the severer negative-positive label imbalance that occurred in SPL than in ordinary multi-label learning. Moreover, we resort to the self-labeling mechanism to disambiguate the unobserved labels and alleviate the bad impact of false negatives.
We for the first time analyze an SPL loss function from the perspective of risk consistency. Under mild sampling assumptions, we first show the theoretical link between arbitrary SPL losses and losses for ordinary multilabel learning. Then we prove the risk consistency of our LASD loss to the cost-sensitive Hamming loss, which guarantees the effectiveness of LASD in dealing with severe label imbalance. This theoretical result also provides theoretical guidance to the choice of the leverage parameters.
In experiments, we compare our proposed loss with other state-of ofte---f-heeettee the SPL methods on multiple multi-label image classification datasets and show the effectiveness of our method. Empirical understandings of the leverage parameter and ablation studies are also conducted.
RELATED WORKS
2.
METHODS TARGETING AT SPL PROBLEM
Cole et al. (2021) first proposed the idea of SPL, which regards the false-negative labels as label noise, decreases the weights of the negative labels in the loss function, and uses the label smoothing (Szegedy et al. 2016) to trim the parameters. However, their loss function contains a hyper-paramett describing the average number of positive labels for each instance, which hard obtain and can vary across datasets. Verelst et al. (2022) utilized a consistency loss to keep feature map output the same across training epochs, making the model for multi-label learning act in synergy with the ubiquitous random resize crop data- auttaaugmentation. However, they focus less on the high negative- positive label imbalance in the SPL problem. Moreover, both the works mentioned above failed to justify their proposed methods theoretically. ZZhou et al. (2022) for the first time introduced the idea of entropy maximization on the unlabeled data and utilized asymmetric pseudo-labeling to address the agative-positive imbalance. Kim et al) (2022) discussed the loss correction method to solve the SPL problem, while SPL algorithms with sound thcoretical understandings from the statistical view are still yet to be exploited.
2.2 PML AND PU METHODS FOR SPL. PROBLEM
The majority can be summarized in two branches: The first branch establishes a new setting with positive and negative labels and solves the problems by designing efficient loss and optimization
2

---

Under review as a conference paper at ICLR 2023
procedures. The other branch pays more attention to label completion, which initially aims at completing unknown labels from the test set. Missing labels on the training set can also be jointly reconstructed in the optimization frameworks. We omit here the discussions of branches incompatible with SPL,, including the requirement of at least two positive labels or one positive and one negative label for each instance and the need for fraction of fully-labeled instances.
Positive- and negative-label methods. Sun et al. (010)) formed a convex optimization problem to predict the unknown labels, which assumes the positive instances will be as sparse as possible for each label. Bucak et al. (201T) similarly presented a ranking- based optimization problem using group lasso to obtain the sparse solution of instances for each label. Zhang et al. (2021) redesigned the loss for negative labels to focus on semi-hard samples and introduced self- loss correction based on the maximum likelihood criterion.
Label matrix completion method. As for label completion methods, et al. (2011) Xu et al. (2013) constructed data matrix composed of training and testing features with training labels and then use ranking minimization to complete unknown testing labels. [siich et al. (2015) conducted matrix completion in the aspect of PU learning, and Han al. (2018) used logistic matrix factorization for label matrix reconstruction.
To the best of our knowledge, few theoretical works have been done on the SPI. problem, especially from the statistical view. In this work, propose new loss function for the SPL problem and for the first time conduct an analysis on the risk consistency of the SPL problem. To be specific, under the mild sampling assumption of labels, losses for fully supervised multi-label learning. Based on this result, we are able to provide further theoretical guidance on the choice of the leverage parameter in our proposed loss function.
provide the theoretical link between SPL losses and the
3 PRELIMINARY
Firstly we will formulate the fully observed multi-label setting from the perspective of the positive label set and then introduce the corresponding single-positive-labll setting.
3.1 FULLY OBSERVED LABELS
In the standard multi-class classification problem, each input variable X from the input space X' is assigned a single label from [K] := (1....,K] where is the number of classes. In the multi-label classification problem under the fully observed label setting, each X is associated with a set of positive labels Y C [K] named the positive label set. For any y E [K], the y-th class is relevant to X, then E y and if the y-th class is not relevant, then & Y. Moreover, the positive multi-label space is denoted as y == (Ý: y c [K]]- We denote as the probability distribution of the fully labeled data (X, Y) defined on X x y and its corresponding probability density function as p. The goal is to find a function f X -> [0, 1] that predicts the applicable labels for each X. The
formal objective is to find an f that minimizes the risk
R(,) = EUx,)))--
where L: [0, 1] x x R+ some multi-label metric such as the average precision 0-1 error.
3.2 SINGLE POSITIVE LABEL
In the multi-class classification problem under single positive label setting, each X is associated with only one single positive label Z E 2 = [K] and obviously Z is one element in the positive label set Y. i.e. Z E Y. We denote P as the probability distribution of the single positive labeled data (X, Z) defined on X' x Z and its corresponding probability density function as p(x, z). Denote the single positive label loss function as L. [0, 1 x [K] + R+ and its corresponding risk
R( = P(x,))=
3

---

Under review as a conference paper at ICLR 2023
3.3 RISK CONSISTENCY
In the single positive label setting, we say single positive label loss function L satisfies the risk consistency, if for any function J, the fully observed label risk R(F) w.r.t. P equals the single positive label risk R(f) w.r.t P. i.e. R(() = R(.).
Through the concept of risk consistency, we are able to relate the SPL problem to the standard multi-label classification. To be specific, risk consistency guarantees the empirical risk w.r.t an SPL loss be an unbiased estimator of the risk w.r.t. multi-label loss. Further, if SPL loss risk consistent to multi-label loss, which is proved to be Bayes consistent (Gao & Zhou 2011), then the SPL loss is also Bayes consistent. In this case, the minimizer of the SPL risk is theoretically guaranteed to be consistent with the Bayes optimum.
4 THE PROPOSED LEVERAGED ASYMMETRIC Loss WITH DISAMBIGUATION
4.1 LEVERAGED ASYMMETRIC LOss
Recall that under the fully supervised multi- learning, Ridnik et al. (2021) proposed the asymmet- ric loss (ASL) to address the high regative-positive imbalance problem concomitant with multi-label learning. specific, the ASI. loss without probability shift is
CASL CASL((()).)) (f(z)
Σ
1kk (fa(x)) + 1kke (x
(3)
where we take the binary focal loss
(1 - fa((x)))))))))))))>)))) log(fa I, (fk(zr) = -fx(x))~ log(1 - fx(x)),
L7+ (f((z)) -(1
(4)
and 7+. 7- 2 0.
In order to apply the loss for multi-label learning to the SPL problem, the vanilla idea to assume that unobserved labels are negative. However, note that the imbalance of fully supervised multi-label learning originates from the existence of more negative labels over positives. the SPL problem, the geeaatve--psitivv label is even more imbalanced, since we have only positive label, and all unobserved labels are assumed to be negative. Therefore, we have to introduce an additional leverage parameter À to tackle the extreme label imbalance, and propose the leveraged asymmetric loss
K
(LAS(f()),
1, by, (fa(z) + A 1 1144 L, ((u(o))
(5)
where À E [0, 1].
4.2
PSEUDO LABEL DISAMBIGUNTION
One thing should pay attention is that the assumed negative labels may indeed be positive. Therefore, we attach pseudo labels to the unobserved labels and treat the pseudo positives and pseudo negatives differently. To this end, reformulate equation 5 with disambiguation, and achieve the leveraged asymmetric loss with disambiguation (LASD), i.e.
K
CLASD (LAs)((()),2)
Σ
|
Lyg (fa(z) d+ 1 -1 Ly (fa(z))
1 JesE [(()))))) Tyy (Aa(z))
(6)
where 7 E [0, is the disambiguation threshold, 7+, >_ 0, and Ly (-) and L,, (-) are defined in equation theoretical guarantees. For the sake of readability, Section The LASD loss equation 6 can decomposed into three terms: the single positive term, the pseudo positive term, and the pseudo negative term.
experiments, we select the leverage parameters A and x_ based on sound
delay the justifications Theorem 5.4 in
4

---

Under review as a conference paper at ICLR 2023
4.3 CONSISTENCY REGULARIZATION
To further increase the precision of pseudo positive and negative annotations introduced in Section 4.2 we adopt the technique of data augmentation and introduce consistency regularization encourage the network's output to be invariant to different randomly augmented images. To be specific, given the original image z and its two random augmentations x' and x", we minimize the &g distance of each output pair. The consistency regularization is therefore formulated as
e Consist (z) = [lf()) f(x')12 + lx) f(=")11 + - f(("))
(7)
The final loss function we used is then formulated as
LLASD (f(), + a * PConsisc(x),
(8)
where a is the weight parameter used to balance the contributions of the loss term LLASD (s()), and the regularization term Consist (z).
5
THEORETICAL ANALYSIS
5.1 ASSUMPTION
The annotator randomly assigns one of the positive labels Y be the single positive label Z. We formulate this kind of annotation as the following assumption.
Assumption 5.1 (Sampling of Labels). Let p and p be the joint probability density functions of the fully labeled data (X, Y) and the single positively labeled data (X, Z). Assume that
p(x,
Σ p(x,
DEV.
where V/- E. DI.
Note that P(=[æ) Egj P(T=))(2207 where P and P denote the corresponding probability distributions of and p. Therefore, Assumption 5. inheerenty assumes the single positive label is uniformly sampled from the fully observed labels, i.e. x(=
Assumption 5.1 is inspired by the conventional modeling of weakly supervised learning problems.
For example, in learning from complementary labels, the complementary label is assumed to be independently and uniformly drawn given the true label (Ishida et al. 2017 2019]] Gao Zhang 202T); in learning from partial labels, the partial labels are also assumed to be uniformly drawn (Feng ct al. 2020). The intuition behind is straightforward: with no prior knowledge given, the simplest and most intuitive way is to assume that the single positive labels are uniformly sampled. Therefore, in this paper, we naturally assume that the single positive label is uniformly sampled, given the ground truth multi-labels.
Here we also propose a practical and efficient way of annotating mulli-labeled instances with single positive labels, which could reduce the cost and labor of annotating large datasets, while satisfies the sampling procedure in Assumption 5.1] In order to get samples with the fully observed labels, annotators often asked to label whether each class of the sample is positive or negative. However, this process is highly time-consuming when the number of classes K is large. Instead, we may choose one of the classes uniformly at random and ask the annotator whether the class is positive or not. Such a yes/no question can be much easier and quicker to be answered than labeling all classes. the answer is yes, this class will be treated as a single positive label; otherwise, we will select the other class again until a true label is obtained.
5.2 RISK CONSISTENCY
The concept of risk consistency a widely used tool in analyzing loss functions for weakly supervised learning problems (see Ishida al. (0017 2019); Gao & Zhang (2021)). In learning from single positive labels, we say an SPL loss function C is risk-consistent to a multi-class loss function they have the same classification risk, given the same classifier f. Note that risk consistency implies the coincidence of the optimal classifiers, i.c., learning from single positive labels with L results in the same optimal classifier learned from fully labeled data with L.
5

---

Under review as a conference paper at ICLR 2023
Theorem 5.2 (Risk Consistency). Under Assumption 5.1
if and L satisfy
E((()),0) = 3 =E
Σ (()()))
(9)
then we have risk consistency, i.e.
R(f)
(10)
we derive the general condition of risk consistency under the setting of multi- learning with single positive labels, which holds for arbitrary loss functions for the SPL problem. To the best of our knowledge, this the first time that single positive loss analyzed from the perspective of risk consistency. By Theorem [.2 risk consistency requires the average of the single positive loss [(f(x), 2) over all potential positive labels E ğ to coincide with the fully supervised multi-label loss ((f(x),)) 37).
In Theorem 5.2]
Now we consider the risk consistency of the single positive loss function introduced in Equation equation 0
Theorem 5.3. we assume that [[ - 1 ken] & E, for some > O, then as E + O, the single positive label loss L defined in equation is risk consistent to the following fully observed label loss L given by
[(f(x), = 77
+
1 1
Σ KET
L,4 (fa())) (tn)))
(1I)
Theorem 5.3 shows that as the model is nearly well trained and the pscudo labels are nearly correctly labeled, 1.e. E - 0, the SPL loss defined in equation 6 is risk consistent to the multi-class loss defined in equation 11 consisting of binary losses on the observed single positive label, the pseudo positive labels, and the pseudo negative labels, respectively.
We show in Theorem 5.3 that our LASD loss is risk consistent to the weighted binary focal loss, which has a similar form to the widely used binary cross-entropy (BCE) loss for multi-label classification. However, equation II uses focal loss instead of cross-entropy loss address the high label imbalance in multi-label learning. Further, when [ 1, where the SPL problem reduced to the ordinary multi-class classification, equation reduces the one-verses-all loss function (Zhang 2004).
Next, we prove that our SPL loss defined in equation risk consistent to the cost-sensitive Hamming loss, and thus can solve the high positive-negative imbalance.
Theorem 5.4. Taking the same assumptions in Theorem 5.3 If ( 1) X- M
then as E + the single positive label loss defined in equation 0 is consistent the cost-sensitive Hamming loss
Dham
(12)
17
kEy
k E
As is known, cost-sensitive learning (ilan 2001 Zhou & Liu 2010)
values false-positive classi-
fication errors differently from false negatives and is widely applied in solving tlas--iballanced
classification problems (Japkowicz & Stephen 2002). In Theorem 5.4 we show that equation 12
the cost-sensitive version of Hamming loss (Gao Zhou| 2011 Wu & Zhou 2017). a widely used loss function for multi-labe classification, where the weights for the false positives and false
negatives are inversely proportional the number of positive and negative labels, respectively. In other words, Theorem 5.4 guarantees the effectiveness of our LASD for the SPL problem in solving severe positive-negative label imbalance by properly selecting the leverage parameters X4, X.
5.3 DISCUSSIONS
In this part, we compare LASD with related works for the SPL problem.
Firstly, we show that our proposed LASD loss an inclusive form, and thus the theoretical analysis in Theorems 5.3 and 5.4 also apply the previous related works. To be specific, as special case,

---

Under review as a conference paper at ICLR 2023
Table 1: Details of four image classification multi-label datasets.
Dataset
Pltect
Rolnss
Avg. pos. per Image
PASCAL VOC 2012 (VOC12) (Everinhham et al.)
5,717
5,823
20
1.46
MS-COCO 2014 (COCO) (Lin ct al.
2014)
82,081
40,137
80
2.94
NUS-WIDE (NUS) (Chua et al.
2009
150,000
60,260
81
1.89
CUB-200-2011 Datasct (CUB)
Wah al. 2011)
5,994
5,794
312
31.4
when taking A, = A_ = 1, our LASD loss reduces to the SPLC loss (without probability shifting) proposed by (Zhang al., (021). Besides, when further taking X x_ loss reduces to the ASL loss without probability shifting (Ridnik et al.] 2021p. p. In order to address the additional high label imbalance brought by the missing labels, SPLC uses the probability shift technique to increase the weight of the semi-hard pseudo positive labels, whereas ASI. adopts the probability shift technique to erase the weight of (assumed) easy negatives. By contrast, we propose to use pair of leverage parameters X and x_ to address this problem, which uniformly down weight all unobserved labels (including both pseudo positives and pseudo negatives). In Theorem [.4 we theoretically prove that properly selected leverage parameters help to balance the weights among the observed single positive label, pseudo positives, and pseudo negatives. Moreover, when taking = and Y+ 7- 0, our LASD loss reduces to the weak assume negative (WAN) loss, which intuitively justifies lowering the weight of the assumed negatives in the binary cross-entropy loss. Nonetheless, byproduct of this work, theory also complements the choice of the parameter in the WAN loss from the perspective of risk consistency.
1. our LASD
Secondly, we compare our LASD with EM loss proposed in (Zhou et al., 2022). which also uses pseudo labels to solve the SPL problem. Zhou et al. (0022) only annotate pseudo negatives to avoid the use of noisy negative pseudo-labels and resort to entropy maximization for the rest of the
unobserved labels. On the contrary, we use both pseudo positives and pseudo negatives in this work.
In Appendix A.3 we empirically verify that when the model is well trained, the precision of pseudo annotations is sufficiently high.
EXPERIMENTS
6.1 DATASET DESCRIPTIONS
We evaluate the performance of our proposed LASD loss on four benchmark multi-label image datasets. We summarize their details in [able including the number of samples for training and for evaluation, and the number of classes. More detailed descriptions are shown in Appendix 1 We generate the single positive multi-label datasets for training by randomly choosing one positive label and discarding other annotations for each training sample from fully labeled multi-label image datasets. The generation process applied only once per dataset, and the generated single positive multi-label dataset is used for all comparisons on that dataset. We split 20% of the training set for validation. Note that different from the training set, the validation and test sets are fully labeled.
6.2
COMPARED METHODS
To evaluate our method on large-scale multi-label image databases, we compare with deep neural network based SPL algorithms and run experiments on their publicly available codes. Specifically, we compare the following loss functions with the formulations of loss functions listed in Appendix A.2.2 CA and frAN are the vanilla ideas that assume all unobserved labels as negatives in the binary cross-entropy and binary focal loss functions, respectively. LAN -LS is variant of LAN where the label smoothing technique (Szegedy et al., 2016 Müller et al. 2019) is incorporated to tackle the issue of "false-negative"" while LWAN down weights all assumed negatives to tackle this problem. LASL is an ordinary multi-label classification method proposed in (Ridnik et al. 2021), where we also assume the unobserved labels to be negative. LASL-PS is variant of LASL where the technique of probability shifting is used. LEPR and CROLE (Cole et al. 202I) introduce the expected number of positive labels per image named k regularization. Different from LERR, LROLE jointly trains
7

---

Under review as a conference paper at ICLR 2023
Table 2: The test mAP (%) evaluated on four image classification multi-label datasets. The best results are marked in bold and the second best marked in underline.
Settings
Losses
VOC12
COCO
NUSWI
LAN
85.89 + 0.38
64.92 + 0.19
42.27 + 0.56
18.31 +: 0.47
LPAN
87.03 + 0.43
68.24 0.04
46.22 0.33
19.42 0.38
CWAN
86.98 + 0.36
0.11
45.71 0.23
19.15 0.56
LAN-LS
87.90 + 0.21
6..15
43.77 = 0.29
16.26 1126+004 0.45
Single
CASE
86.99 + 0.05
0.19
46.68 =
19.72
Positive
LASL-PS
87.32 0.06
0.24
46.96 - 0.08
20.11
Labeled
CEPR
85.15 + 0.58
0.12
45.41 0.05
19.53
PROLE
87.77 0.22
0.19
11.63 + 0.35
13.66 0.24
LLL-R
89.23 + 0.13
7.002
47.44 0.28
19.52
CLI--CI
89.06 0.17
0.14
48.04
20.43 +
CLL-CPP
88.40 0.24
7.07
48.30
0.37
LEM
89.09 0.17
70.70
47.15
20.85 0.42
CEMAPL
89.19 + 0.31
70.87 0.23
47.59 + 0.22
21.84 +
0.34 0.34
CLASD
89.53 + 0.21
72.22
50.52 0.03
22.01 0.48
Two networks and uses the co-training technique, where one network provides pseudo labels for the supervision of the other network. LT.I. R, LIA.-Ct, LLI.- are three variants of the large loss proposed in Kim al. 2022)) and LEM, LEMAPL variants of the entropy maximization loss proposed in Zhou et al. (2022)
6.3 PERFORMANCE EVALUATION
The backbone network the ImageNet pre- trairraned ResNet-50 provided in PyTorch (Paszke et al. 2019). We add linear layer after the backbone network. We run experiments NVIDIA Tesla v v100 32GB GPU. We present results under the scenario fine-tuning the entire network end-to-end for ten epochs. For each method, search the learning rate in {1e 2, le 3, le 4, 1e 5) and the batch size {8, 16). We select the hyper-parametes whose mean average precision (MAP) is the highest on the validation set. The selection of ttype--prameeten for all methods shown Appendix A..22 We report the test mean average precision (MAP) performance on the multi-label image classification datasets. We repeat the experiments ten times and also report the standard deviations. The best results are marked in bold and the second-best are marked in underline.
Table@coompan the mAPs of our proposed LASD loss with other losses under the SPL paradigm. We conduct significant tests and empirically verify that our proposed LASD outperforms other published methods significantly the significance level of 0.05 on all four benchmark datasets. Compared with the best performance of other stae-off-the-an algorithms for the SPL problem, the improvements of LASD are significant especially on two large multi-label datasets COCO and NUSWIDE: the LASD loss achieves +1.2 mAP improvement on the COCC dataset and +2.2 mAp improvement NUSWIDE dataset. Compared with gains obtained by former SOTA methods LLL (Kim et al., 2022) and CEMAPL (Zhou et al. (022) in their original papers where average +1.4 mAP gains are achieved, the improvements of our LASD method are also remarkable enough. These achievements attribute to the learning strategy of leveraged asymmetric loss with pseudo label disambiguation and consistency regularization.
the
6.4 PARAMETER ANALYSIS
In this part, we conduct parameter analysis on essential parameters our LASD loss, including the leverage parameter X4, the disambiguation threshold and the strength of the regularization term a. We change one parameter while fixing other parameters to the best parameters chosen from the validation set. Note that the other leverage parameter X_ is theoretically calculated based Theorem
Parameter Analysis on the leverage parameter A7: From Figure ((a) observe that the perfor- mance reaches the best mAP when X 0.10. When X becomes too small, the pscudo positive term will be neglected, whereas large X means that might be overconfident the pseudo
8

---

Under review as a conference paper at ICLR 2023
DRAF Lew
(a) the leverage parameter A (b) the disambiguation threshold T (c) regularization strength a
Figure 1: Parameter analysis on three parameters of our proposed LASD loss on the VOC12 dataset.
Table 3: Ablation study the LASD loss the voC12 dataset.
Leverage
Disambiguation
Consistency Regularization
VOC12
x
X
X 85.89 + 0.38
X
86.99 + 0.05
88. 70 + 0.14
89.53 0.25
positive labels. Note that our method outperforms other competing methods in a wide range of X4+ which verifies the robustness of this parameter.
Parameter Analysis on the disambiguation threshold т: Figure I(b) shows that when T is close to I, it tends to ignore the potential positive labels, while small is at the risk of false positive. For this reason, suitable r slightly smaller than one beneficial in the SPL problem. Figure [(b) also verifies the robustness of the selection T, our method outperforms in wide range of T.
Parameter Analysis on the weight parameter the regularization term at From Figure T(c) we observe that our method outperforms other competitors in a wide range between 0.2 to 0.8. When the regularization strength a is close to zero, its performance decreases because the model fails to utilize the power of consistency regularization, whereas when a is too large, the regularization term will be dominated in the loss function and is harmful for the training process.
We discuss the effectiveness of three key components of LASD, including the asymmetric leverage, the pseudo label disambiguation and the regularization term. We compare the following cases: case A) without the leveraged term, the disambiguation term, and the regularization term; case B) with the leveraged term but without the disambiguation term and the regularization term; case C) with the leveraged term and the disambiguation term, but without the regularization term; and case D) with the leveraged term, the disambiguation term, and the regularization term. The results in Table S show that all these three terms attribute the performance improvement. For one thing, leveraging negative and positive samples with two leverage parameter X and A_ is essential tackle the problem of highly negative- -ossitvvee imbalance problem with the theoretically understanding w.r.t the risk consistency to cost-sensitive Hamming loss. For another, the pseudo probleem label disambiguation further helps to alleviate the false negative problem and the regularization term further increases the reliability of pseudo-labeling.
Ablation Study:
7 CONCLUSION
In this paper, we proposed a novel loss function called leveraged asymmetric loss with disambiguation (LASD) to address the problem of multi-label learning from single positive labels (SPL). this loss. we introduce a pair of leverage parameters weighting the single positive label and the unobserved labels, so as deal with the high negative-positive imbalance. Moreover, we incorporate pseudo labels to disambiguate potential positive and negative labels. From the theoretical perspective, we introduce the concept of risk consistency, and derive the link between an arbitrary SPL loss to the losses for ordinary multi-label learning. We prove that our proposed LASD loss is consistent with the sessitves Hamming loss, which can guide the empirical choice of our proposed leverage parameter under sound theoretical guarantees. In experiments, compare LASD with other state-of-te of-the-art SPL algorithms, and demonstrate the outstanding performance of our method on multiple real-world datasets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Serhat Selcuk Bucak, Rong Jin, and Anil K. Jain. Multi-label learning with incomplete class assignments. In CVPR, 2011.
Ricardo Silveira Cabral, Fernando De la Torre, João Paulo Costeira, and Alexandre Bernardino. Matrix completion for multi-label image classification. In NIPS, 2011.
Zhao-Min Chen, Xiu-Shen Wei, Peng Wang. and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In CVPR, 2019.
Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. real-world web image database from national university of singapore. In C/VR, 2009.
Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic. Multi-label learning from single positive labels. In CVPR, 2021.
Jia Deng, Olga Russakovsky, Jonathan Krause, Michael S Bernstein, Alex Berg, and Li Fei-Fei. Scalable multi-label annotation. In SIGCHI, 2014.
Thibaut Durand, Nazanin Mehrasa, and Greg Mori. Learning deep convnet for multi-label classifi- cation with partial labels. In CVPR, 2019.
Charles Elkan. The foundations of cost-sensitive learning. In IJCAI, volume 17, pp. 973-978, 2001.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal
Hao Fei, Yue Zhang, Yafeng Ren, and Donghong Ji. Latent emotion memory for multi-label emotion classification. In AAAI, 2020.
Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. In NeurlPS, volume 33, pp. 10948-10960, 2020.
Wei Gao and Zhi Hua Zhou. On the consistency of multi-label learning. In COLT, 2011. 2011.
Yi Gao and MingLing Zhang. Discriminative complementary-labe learning with weighted loss. In International Conference Machine Learning, pp. 3587-3597, 2021.
Yunchao Gong, Yangqing Jia, Thomas Leung, Alexander Toshev, and Sergey loffe. Deep convolu- tional ranking for multilabel image annotation. In ICLR, 2014.
Yufei Han, Guolei Sun, Yun Shen, and Xiangliang Zhang. Multi-label learning with highly incomplete data via collaborative embedding. In KDD, 2018
Cho-Jui Hsieh, Nagarajan Natarajan, and Inderjit S. Dhillon. PU learning for matrix completion. In ICML, 2015.
Bin Hu, Kehua Guo, Xiaokang Wang. Jian Zhang, and Di Zhou. Rrl-gat: Graph attention network- driven multi-label image robust representation learning. IEEE Internet of Things Journal, 2021.
Chenyang Huang, Amine Trabelsi, Xuebin Qin, Nawshad Farruque, Lili Mou, and Osmar Zaiane. Seq2emo: A sequence multi-label emotion classification model. In NAACL, 2021.
Dat Huynh and Ehsan Ellhamifar. Interactive multi-label CNN learning with partial labels. In CVPR, 2020.
Takashi Ishida, Gang Niu, Weihua Hu, and Masashi Sugiyama. Learning from complementary labels. In NeurlPS, volume 30, 2017.
Takashi Ishida. Gang Niu, Aditya Menon, and Masashi Sugiyama. Complementary-lan labell learning for arbitrary losses and models. International Conference on Machine Learning, pp. 2971-2980, 2019.
10

---

Under review as a conference paper at ICLR 2023
Nathalie Japkowicz and Shaju Stephen. The class imbalance problem: A systematic study. Intelligent data analysis, 6(5):429 449, 2002.
Atsushi Kanehira and Tatsuya Harada. Multi-label ranking from positive and unlabeled data. In CVPR, 2016.
Youngwook Kim, Jae Myung Kim, Zeynep Akata, and Jungwoo Lee. Large loss matters in weakly supervised multi-label classification. In CVPR, pp. 14156-14165, 2022.
Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi. General multi-label image classifi- cation with transformers. CVPR, 2021.
Junlong Li, Peipei Li, Xuegang Hu, and Kui Yu. Learning common and label-specific features for multi-label classification with correlation information. Pattern Recognition, 121:108259, 2022.
Ziwei Li, Gengyu Lyu, and Songhe Feng. Partial multi-label learning via multi-subbspace representa- tion. In IJCAI, 2021.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. In ICCV, 2017.
Siyu Liu, Xuehua Song, Zhongchen Ma, Ernest Domanaanmwi Ganaa, and XiangJun Shen. More: Multi-output residual embedding for multi-label classification. Pattern Recognition, 126: 108584, 2022.
Weiwei Liu, Haobo Wang, Xiaobo Shen, and Ivor Tsang. The emerging trends of multi-label learning, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Gengyu Lyu, Songhe Feng, and Yidong Li. Noisy label tolerance: A new perspective of partial multi-label learning. Information Sciences, 543:454-466, 2021.
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In NeurlPS, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high geg--performncc deep learning library. In NeurlPS, 2019.
Thomas Pellegrini and Timothée Masquelier. Fast threshold optimization for multi-label audio tagging using surrogate gradient learning. In ICASSP, 2021.
Tal Ridnik, Emanuel Ben- Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi-label classification. In ICCV, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 1 115(3), 2015.
Harsh Shrivastava, Yfang Yin, Rajiv Ratn Shah, and Roger Zimmermann. Mt-gen for multi-label audio-tagging with noisy labels. In ICASSP. 2020.
Pierre Stock and Moustapha Cisse. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. In ECCV, 2018.
Yu-Yin Sun, Yin Zhang, and Zhi-Hua Zhou. Multi-labe learning with weak label. In AAAI, 2010.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, 2016.
11

---

Under review as a conference paper at ICLR 2023
Thomas Verelst, Paul K Rubenstein, Marcin Eichner, Tinne Tuytelaars, and Maxim Berman. Spatial consistency loss for training multi-label classifiers from single-label annotations. arXiv preprint arXiv:2203.061177, 2022.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The Caltech-UCSD 200d2-200-2011 Dataset. Technical report, California Institute of Technology, 2011.
Jeremy M Wolfe, Todd S Horowitz, and Naomi M Kenner. Rare items often missed in visual searches. Nature, 435(7041):435 440, 2005.
Xi-Zhu Wu and Zhi-Hua Zhou. A unified view multi-label performance measures, In international conference on machine learning, pp. 3780-3788, 2017.
Ming-Kul Kung-KKnnuu Xie and Sheng-Jun Huang. Partial multi-label learning with noisy label identification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Miao Xu, Rong Jin, and Zhi- Hua Zhou. Speedup matrix completion with side information: Applica- tion to multi-label learning. NIPS, 2013.
Yan Yan and Yuhong Guo. Adversarial partial multi-label learning with label disambiguation. In AAAL, 2021.
Tong Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research, 5(0ct): 2004.
Youcai Zhang, Yuhao Cheng, Xinyu Huang, Fei Wen, Rui Feng, Yaqian Li, and Yandong Guo. Simple and robust loss design for multi-label learning with missing labels. arXiv preprint arXiv:21/2.07368, 2021.
Donghao Zhou, Pengfei Chen, Qiong Wang, Guangyong Chen, and Pheng- Ann Heng. Acknowledging the unknown for multi-label learning with single positive labels. arXiv preprint 2tt.::222031.2111 2022.
Zhi-Hua Zhou and Xu- Ying Liu, On multi-class cost-sensitive learning. Computational Intelligence, 26(3):232-257, 2010.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 PROOFS
Proof of Theorem 5.2 We compare the risk in the fully-labeled setting and the risk in the single positive setting as follows.
the one hand, the risk in fully label setting
E - (UX))
=Ex Σ ((
(13)
JEY
On the other hand, the risk in the single positive setting is
[))((()), = [x,,,2(((()),
= =ExByyx E218 [ff((),
= -xx I
[(f(x)),2
ZEY
Ex
Σ p() Σ (()()))))
(14)
VEY
EEG
Combining Equation equation 13 and equation
14
we find out that, if
((f())
((f()),)
(15)
ZEG
is satisfied, then risk consistency holds, which finishes the proof.
Proof of Theorem 5.3
Combining equation 6 and equation we get the risk consistent fully-
observed label loss
[(f(x),) = Σ ((f()),)
2E17
[| + x+ - by + A 1 1
=
ZE 1=
(M|= + x+ 11444 by + A_ 1 (1
ZEN7
K
La (f+)+ Σ
(A+A (fe) + X (( On
=EN
K
Ly (f2)++++ Σ (r((
(16)
2E17
k=1.kzz
where denote L > 1+)1 (fr) X I (fe) for notational simplicity. Then we have
[(f(z),) Σ4 (fz)
Σ
Σ (fi++ Σ L.((f)
kg
13

---

Under review as a conference paper at ICLR 2023
Σ =E
b (f:)
EE
(ffe)
1
ΣΣ
(f(e)
(17)
= K
SE kE
Note that Eeeg
| and Eggg K 1//-
Then the second term of equation becomes
k Σ
(fw)
((W)
(fx).
(18)
k
R
The third term of equation becomes
K
(fx
(Tke)...
(19)
SE
kEy #
KE
Combining equation
equation 18
and equation 19
we have
1
Z(S(z),))
LAnt (fx) +
Lr (fx)
(1aa)
KET
KEE
K
Lay (Jx) +
(Jar
(Jk))
KET
k=1
KEN
K
L++ (fa)
+
Σ
(fA)
Σ (fr))
KEY
1=1
KEJ
K
Lay (J&) +
x Uc((a)s) I (fk) + A In by (Js))
KEY
REN
(20)
By assumption, 1 [x)))) 2 1RkE i.c. [[f((1)>7]] - 1 = E, then there holds
[((())))) - [k:0]] = (1 1((())2))) - (1 -11e0)) [kea] [x(x))+]] = -6.
(21)
Thus, by equation 20 we have
K
1
[(f(z),))
Iay (fk)
10
RET
KEy
( IKET Ixp (fi) x_ [ Tay (fic ed (f1) ed by (fk)
K
1 Σ Le (ft) +
Ly, (fk) + x [R -- (fk)
37
KET
KEy
K
e(A -n (ft)
KEY
-(1
1
1
+ X+ (1
Σ
Toy (Ja(z)) + Σ (ma
M
7
REy
KET
E
Σ
Σ
In, (fk) I (fi))
(22)
KEY
By similar induction, equation 22 also holds for the case where [[((())27] I [[NEo Then as ++0. 0. there holds
[(f()),) +
1 $11 + A (1
Ly (f((z)) + Σ by (fu(z)).
(23)
17
77
KEW
RET
14

---

Under review as a conference paper at ICLR 2023
Proof of Theorem 5.4 We first consider the loss function
Σ
1
+ x-
Σ
1 (ux+
(24)
KEØ
which replaces Tny and Toy- with indicator functions.
Denote Py P(Y x) We have the inner risk w.r.t. C
W [w,,]) = Σ P)f(((x))
JEY
Σ Pj DEY
+ +
Σ
Σ Py A_ Σ
Betual
1
KEV
DEY
ΣΣ Pj
+ x+
1
110
+
Σ
Ps A_ 1 [8a
JEY KEY
JEY KgB
= Σ
Σ
P3
7
A+ (1
1
1OLAUEl Σ
Σ
P3 d_ [00
KE[K] JEY:
KE(K JEVi
Σ
Σ
PJ
X+ (1 127)
1000
Σ
Pū x [00
(25)
KE[K]
JEVE
DEV'S'
For [K], if for some constant
0, 1 + (107
1) = A- - 107) = C, then we have
W(p = Σ
Σ
[0==1] +
Σ
K
1
17
10==11)
(26)
kE[K]
JEV
DE
and the set corresponding Bayes predictors of L
}
A(p) f f(x) Fff) with OS sgn
PT
Σ TE3
1r Ps
(27)
K
Note that A(p) in equation 27 the same as the set Bayes predictors of the cost-sensitive Hamming loss equation| 12. By similar induction of Theorem 15 in Gao & Zhou (201T), one can derive that
LASD equation 6 with surrogate losses I,,,, I,_ is consistent w.r.t. the cost-sensitive Hamming loss
equation 12 if L,4, L,_ are convex and 0L,, (fx).
aL, (-fe)
0f. 11=00 = Ly, and by the above conditions are satisfied and we thus finish the proof.
öfs I < O. By the definition of
A.2 EXPERIMENTAL DETAILS
A.2.1 DATASET DESCRIPTION
We provide the details of four benchmark multi-label image datasets below:
The PASCAL VOC 2012 Dataset (VOC12) 2) (Everingham et al.) includes 5,717 training images and 20 classes. We use the official validation set (5,823 images) as the test set. The VOC2012 data includes images obtained from the "Flickr" website under the "Flickr" terms use. The identity of the images in the database, e.g., source and name of the owner, has been obscured.
The MS-COCO 2014 Dataset (COCO) (Lin et al. 2014) contains 82,081 training images and 80 classes. We also use the official validation set (40, images) the test set.
The NUS-WIDE (NUS) Dataset (Chua et al. 2009) consists of 126,034 and 84,226 images from the official training and test sets respectively. The dataset includes 81 classes. Following the standard practice (Gong et al. 2014 Durand et al. 2019). we combine the official training and test sets and randomly split the whole dataset into the training set with 150,000 images and the non-overlapping test set with 60,260 images. This web image dataset is created by Lab for Media Search at the National University of Singapore and should be only used for non-commercial research educational purposes.
15

---

Under review as a conference paper at ICLR 2023
Dataset (CUB) (Wah al. 2011 contains 5,994 training images and 312 classes. The number of images in the test set is 5,794. Images in this dataset are overlapped with images in ImageNet. The use is restricted non-commercial research and educational purposes.
The
We carefully check the descriptions these data sets and how these data sets were collected to make sure that no personally identifiable information offensive content is contained.
A.2.2 COMPARED METHODS
LAN is from the vanilla idea that unobserved labels are assumed negative. Thus, the assumed negative strategy with the binary cross coss-entropy loss is given by
(AN(f(o),
Σ [ pa log(fa) + 1 [awen] log(1 fu]]
LfAN is also from the vanilla idea that unobserved labels assumed negative. Thus, the assumed negative strategy with the focal loss is given by
(FAN(f(Z),
1 [1 fx)" log(fs) + 1 log(1 fu)] K
Σ
where is suggested in (Lin al. 2017).
LAN-I. the label smoothing version to tackle the issue of "false negative". Label smoothing is proposed in (Szegedy et al. 2016 Müller et al. 2019). CAN-LS combines the assume negative strategy and the label smoothing training technique. LAN-LS given by
K
(AN)
[111 - e/2) 1|==a] + 8/2 1 [ 1og(fx)+
K
k=1
- ((1-E/2) [ + E/2 [|==]] x| [a* log(1 - fx)],
where is the hyper-parameter of label smoothing and is set 0.1 in (Cole et al. 2021
LwAN proposed based on the idea that the false negative term in the loss function down-weighted. LWAN is given by
(wAN(f(=), = Σ b (og(f)) + X [ log(1 fk)]
where E [o, 1] is the weight
and set to 1/(K - in practice.
LASL is also from the vanilla idea that unobserved labels are assumed negative. Thus, the assumed negative strategy with the asymmetrie focal loss is given by
K
(Ast(f(z),
1
Σ
[ - ) log(fe) + 12 log(1 - fw)]
K
where and 7- are suggested in RRidikk et al. 2021
CASL.PPS is also from the vanilla idea that unobserved labels are assumed negative. CASL-PS a variant of LASL that probability shifting added as in (Ridnik al. (022).
Therefore, denote p as max(fe m, 0), LASLPS is given by
() = Σ
1 1 (1 fo) log(fz) 12 log(1 (1 Pm.
where m the probability shifting parameter. We follow the suggestion Riddik et al. 2021 that 7+ 0, 7- and 0.05.
16

---

Under review as a conference paper at ICLR 2023
LEpr and CROLE are proposed in (Cole et al. 2021) Both of them introduced the expected number of positive labels per image named i as regularization. The penalty is applied at the batch level, and the hyper-paramann peeemeerreeter & is specified based on the fully labeled training set. Different from LEPR, PROLE jointly trains two networks using the co-training technique, where one network provides pseudo labels for the supervision of the other network. The expected positive regularization loss LEPR is formulated as
LEPR BB (B) =
Σ BEB
LèCE (fn,zn) + XRt (FB),
B
where
LEC (fx, kn) = Σ [ (z=M) log (fa),
km1
k ((BB) k
R& (FB)
L
i (FB) EnEB Els fui
[B]
and J. g are the image classifier and the label estimator respectively. The regularized online label estimation loss LROLE defined as
C'
FB
YB
+ + Y (Y
FB
LROLE
where
C'
FB YB
Σ nEB
LBCE (f,, sg (Fn)) + LEPR ZB),
[
and sg is the stop-gradient operation.
CLASD is proposed in our paper. We fix 7+ 0, 7- and search the disambiguation
threshold 5 (0.85, 1), the leverage parameter X E (0, 1]. and the weight parameter of
the regularization term a. Another leverage parameter X_ then calculated according to Theorem 5.4 We provide the final selected hyper- hyper-paramen parameters of our proposed LASD as in Table
Table 4: Selected hyper-parameters of method each dataset.
VOC12
COCO
NUS
CUB
X+
0.10
0.03
0.015
0,005
0.90
0.90
0.95
0.93
0.70
0.10
0.15
0.03
A.3 PRECISION OF PSEUDO LABELS
In order to verify that the assumption in Theorem 5.3 that [((x))>r] = likes holds when fa (x) trained good enough, we present the averaged Positive Predictive Value (PPV) and the averaged Negative Predictive Value (NPV) of the proposed LASD loss per epoch on the test sets of vOC12 and COCO under the best hyper-parameters. In each epoch, we use the disambiguation threshold to classify the instance for the k-th label: we classify the instance x positive for the k-th label (x) > т and otherwise negative. Then, we calculate the True Positive (TP), False Positive (FP), False Negative (FN), and True Negative (TN). Positive Predictive Value (PPV) is calculated by TP/(TPFFP), and Negative Predictive Value (NPV) is calculated by TN((TN+ We obtained the averaged PPV and NPV for all labels per epoch.
As shown in Figure 2 the PPVs and the NPVs of our LASD loss are signifitantly higher than 90% and even 95%, which means that most of the classified positive labels are true positives and most of the classified negative labels are true negatives. In other words, [(f(())) = 1 and empirically verified.
(z) [
17

---

Under review as a conference paper at ICLR 2023
1.0
3.0
5.8
5.6
PPV/NPV
0.6
PPV/NPV
0.6
0.4
0.4
6.2
6.2
PPV
PPV
5.0
0.0
Epech
Epech
(a) VOC12
(b) COCO
Figure 2: Positive Predictive Value (PPV) and Negative Predictive Value (NPV). The high PPV and NPV results empirically verify the assumption of Theorem 5.3
18


Reviews:
[{"review_id": "kHDDVE8knEB", "reviewer": "Reviewer_k9E5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a loss function to learn from multi-label data with only one-positive annotation. Specially, the proposed loss function considers three factors: i) the existing asymmetric loss; ii) it uses a threshold to differentiate true-positive and true-negative in the unlabeled candidate labels. The self-labeled data is further used in the asymmetric loss; iii) a regularizer which ensures the data augmentation will not change the model too much. These items are weighted and added together. The paper analyzes some theoretical properties of the problem and the proposed loss function, and finally, shows sufficient experimental results to demonstrate the effectiveness of the proposed method. ", "summary_of_the_review": "The paper has shown a simple, straightforward, and effective solution to a kind of weakly supervised multi-label learning problem. The studied problem is important and the solution is empirically effective, although the proposed method is not novel. The theoretical results may also need to be strengthened. ", "strength_and_weaknesses": "Strength\n1. It proposes an effective loss function for learning from multi-labeled data with only one positive annotation.\n2. It has shown sufficient experimental results and overall, the performance of the proposed method is superior\n3. The paper is clearly written and easy to follow\n\nWeakness\n1. The proposed technique is a combination of existing techniques, may lack technical novelty\n2. The theoretical results may not be convincing (will explain in detail later)  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "S7vWgBnGfD", "reviewer": "Reviewer_PKX1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper seeks to study a weakly supervised problem called Single Positive Labels (SPL) in multi-label learning where only one positive label is observed for each examples. It proposes a loss function named leveraged asymmetric loss with disambiguation (LASD) following multi-label version of focal loss for SPL problem, and tries to prove that learning with the proposed loss is consistent.", "summary_of_the_review": "The theoretical analysis of the paper needs further revision and improvement.", "strength_and_weaknesses": "Strengths:\n1. The topic is interesting.\n2. The method is easy to understand.\n3. Although unsuccessful, the theoretical analysis of the proposed method is to be encouraged.\n\nWeaknesses:\n1. The theoretical analysis is deeply flawed. This paper proves the risk consistency of LASD by proving the risk of the classifier learned with LASD approaches to the risk of the classifier **learned with a specifically defined and  (most likely, at least, unproven) non-classification-calibrated multi-label loss function**. Such \"consistency\" is meaningless because classification-calibrated is the most basic requirement for a loss function.\n2. The assumption of Theorem 5.3 is unrealistic. It assumes that if the classifier f can already successfully classify (i.e., the optimal classifier), then LASD is spurious risk-consistent. One wants the learning to be consistent so that the learned classifier is guaranteed to be (asymptotically) equivalent to the optimal classifier, rather than assuming that the classifier is the optimal one first and then proving something else. It's a complete reversal of cause and effect.\n3. Assumption 5.1 and Theorem 5.2 are disconnected from the main content. They are neither an original contributions nor relevant to the approach presented in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EDzv_MS5oo", "reviewer": "Reviewer_k1k2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the single positive labels problem in multi-label learning. A new loss function called leveraged asymmetric loss with disambiguation (LASD) is proposed, which is an improved version of the existing leveraged asymmetric loss (ASL) with an explicit pseudo label disambiguation. The proposed algorithm further adds a consistency regularization to the LASD loss. Theoretic analysis provides a link between the proposed LASD and Hamming loss. Empirical evaluations are reported using four benchmark datasets against the state-of-the-art literature.", "summary_of_the_review": "This work addresses the single positive labels problem in multi-label learning. The paper proposes a new loss function explicitly addressing the asymmetric imbalance between positive and negative labels. The theoretical analysis makes sense and the proposed method is supported by evaluations. I have concern on the novelty and the method.", "strength_and_weaknesses": "Strengths:\n-\tthe presentation is clear\n-\tthe proposed solution appears to be correct\n-\tthe theoretic analysis appears to make sense\n-\tthe evaluations as well as the ablation studies appear to be extensive\nWeaknesses:\n\nI have two major concerns with this work. First, the novelty. The proposed loss function is an extension from the existing LAS loss. While there is arguably something new with the extension, novelty-wise it is incremental at best.\n\nSecond, the proposed consistency regularization, while conceptually it makes sense, lacks justification in terms of the current form. Why do we have to consider two augmented samples, not one, or not three? It appears to be rather arbitrary and ad hoc.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5EzZO5_EIx7", "reviewer": "Reviewer_HXy7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel loss function called leveraged asymmetric loss with disambiguation (LASD) for the problem of multi-label learning from single positive labels (SPL) where only one positive annotation is available. A pair of leverage parameters are employed to address the severe negative-positive imbalance. This paper links the SPL loss with losses for ordinary multi-label classification from the perspective of risk consistency and proves the consistency of the proposed LASD loss to Hamming loss. Finally, experiments including comparing it with other state-of-art methods and ablation are conducted to validate the effectiveness of their method.", "summary_of_the_review": "A pair of leverage parameters are employed to address the severe negative-positive imbalance of the SPL problem and the usage of consistency regularization can further improve the performance. However, the assumption that the pseudo-positive labels selected by the threshold are just real positive labels is unreasonable and there is a lack of experiments to verify the assumption.", "strength_and_weaknesses": "Strengths:\n1. A pair of leverage parameters are employed to address the severe negative-positive imbalance of the SPL problem and the usage of consistency regularization can further improve the performance.\n2. The proposed method is easy to understand and reproduce.\n\nWeakness:\n1. The most significant contribution of this work is analyzing an SPL loss function from the perspective of risk consistency. However, the assumption of Theorem 5.3 is unreasonable, in the inequation $ \\left|1_{\\left[f_k(x) \\geq \\tau\\right]}-1_{[k \\in \\vec{y}]}\\right| \\leq \\varepsilon $, the left part only has the value of 0 and 1, letting $\\varepsilon \\rightarrow 0$ is equivalent to assuming that the pseudo positive labels selected by the threshold are just real positive labels. In this case, the proof of the consistency between LASD loss and Hamming loss makes no sense.\n2. Theorem 5.4 guarantees the effectiveness of LASD for the SPL problem in solving the severe positive-negative label imbalance by properly selecting the leverage parameters $\\lambda_{+}, \\lambda_{-}$. But another import hyper-parameter $\\tau$ which is used to select pseudo-positive labels in LASD loss is neglected. It would be better if there is theoretical guidance for the selection of the threshold rather than assuming that the pseudo-positive labels selected by the threshold are just real positive labels.\n3. Leveraged asymmetric loss (LAS) has been proposed to alleviate the extreme imbalance of negative-positive labels in the SPL problem. LASD loss just adds a threshold to select the pseudo-positive labels which also has been widely used in previous work. There is not enough innovation in this work.\n4. there is a lack of experiments to verify the assumption of Theorem 5.3 and Theorem 5.4.\n5. In Table 2, the improvement of the performance on datasets VOC12 and CUB is not obvious enough.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject