One Positive Label is Sufficient: Single-Positive Multi-Label Learning with Label Enhancement
Authors: Ning Xu
Congyu Qiao
Jiaqi Lv
Xin Geng
Min-Ling Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Label Enhancement
Multi-Label Learning

Abstract: Multi-label learning (MLL) learns from the examples each associated with multiple labels simultaneously, where the high cost of annotating all relevant labels for each training example is challenging for real-world applications. To cope with the challenge, we investigate single-positive multi-label learning (SPMLL) where each example is annotated with only one relevant label and show that one can successfully learn a theoretically grounded multi-label classifier for the problem. In this paper,  a novel  SPMLL method named SMILE, i.e., Single-positive MultI-label learning with Label Enhancement, is proposed. Specifically, an unbiased risk estimator is derived, which could be guaranteed to approximately converge to the optimal risk minimizer of fully supervised learning and shows that one positive label of each instance is sufficient to train the predictive model. Then, the corresponding empirical risk estimator is established via recovering the latent soft label as a label enhancement process, where the posterior density of the latent soft labels is approximate to the variational Beta density parameterized by an inference model. Experiments on benchmark datasets validate the effectiveness of the proposed method.

One Positive Label is Sufficient: Single-Positive Multi-Label Learning with Label Enhancement
Ning Xu', Congyu Qiao', Jiaqi Lv², Xin Geng", Min-Ling Zhang²
1 School of Computer Science and Engineering. Southeast University, Nanjing 210096, China RIKEN Center for Advanced Intelligence Project, Tokyo 103-0027, Japan (xming, @ssu. edu. cn, is i aogg.vv@@mmall (xgeng, zhangml) @sen @seu. edu.
com,
Abstract
Multi-label learning (MLL) learns from the examples each associated with multiple labels simultaneouslly, where the high cost of annotating all relevant labels for each training example is challenging for real-world applications. To cope with the challenge, we investigate single-positive multi-label learning (SPMLL) where each example is annotated with only one relevant label and show that one can successfully learn theoretically grounded multi-label classifier on SPMLL training examples. In this paper, novel SPMLL method named SMILE, i.e., Single- positive MultI-label learning with Label Enhancement, is proposed. Specifically, an unbiased risk estimator derived, which could be guaranteed to approximately converge to the optimal risk minimizer in fully supervised learning and shows that one positive label of each instance sufficient to train a model. Then, the corresponding empirical risk estimator is established via recovering the latent soft label as a label enhancement process, where the posterior density of the latent soft labels is approximate to the variational Beta density parameterized by inference model. Experiments on twelve corrupted MLL datasets show the effectiveness of SMILE over several existing SPMLL approaches. Source code is available htttps://githi cww/pa//aaaaooc com/palm-mil ml/smile
1 Introduction
Multi-label learning (MLL)) aims to build a predictive model to assign a set of relevant labels for the unseen instance via learning from the training examples associated with multiple class labels
simultaneously (30 46). During the past decade, MLL. has been widely applied to learn from the
data containing rich semantics, such multimedia content annotation [40 [3), text categorization
29 27,,
music emotion analysis (5) and bioinformatics analysis
3 etc.
However, in practice, obtaining ground-truth multiple labels for MLL training datasets is costly due to the expensive and consuminn manual annotations. Comparing with multi-class learning where an example is associated with only one positive label, multi-labed learning requires the complete positive label set for each example. On this account, the annotation cost of multi-label learning is significantly higher than multi-class classification, which limits its application especially when the number categories large.
To mitigate this problem, the setting of single-positive multi-label learning (SPMLL) 5 allows for significantly reduced annotations costs for the datasets, where each example is annotated with
"Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Avouge Presikot
Boss
The the sublanp coampls over the total crampls
Figure I: Test average precision tmc2007. Each curve is generated by randomly subsampling the examples from the training set, where BCE is trained on fully labeled examples via binary cross-entropy loss while the SPMLL methods (ROLE (5) and the proposed SMILE) are trained on single-positive case.
only one relevant label. In Figure 1 comparing with fully labeled case, the SPMLL approaches single-positive labeled examples only incur tolerable drop in the performance but drastically reduce the amount of supervision required to train multi-label classifiers. By establishing the SPMLL methods with the learning power DNNs, recent work [5] has also empirically validated that SPMLL would reduce the annotations costs while achieving good performance in practice. However, no method could provide theoretical insights as why the model trained on the SPMLL examples can converge to an ideal one.
In this paper, propose Multl-label learning with Label Enhancement. Specifically, we first derive an unbiased risk estimator, which suggests that positive label of each instance sufficient to train the predictive models for multi-label learning. Besides, an estimation error bound is derived, which guarantees the risk- consistency 25 of the proposed method. Then could design benchmark solution via recovering the soft labels corresponding to each example in a label enhancement process [39 36). where the
guaaratte method named SMILE, i.e.,
posterior density of the latent soft labels is inferred by leveraging an approximate Beta density. The contributions are summarized as follows:
Theoretically, for the first time derive unbiased risk estimator for SPMLL. Based this, an estimation eтror bound is established that guarantees the css-consistency and demonstrates that the obtained risk minimizer in SPMLL would approximately converge the optimal risk minimizer in fully supervised MIL.
Practically, we propose the method SMILE for SPMLL via adopting the latent soft labels recovered by label enhancement. The posterior density of the latent soft label is inferred by leveraging an approximate Beta density and the evidence lower bound (ELBO) [88 for the optimization is deduced.
Experiments on twelve corrupted MI.L datasets show the effectiveness of SMILE over several existing SPMLL approcchss
Related Work
In multi-label learning, each example is associated with multiple class labels simultancously. As the output space in MLL exponential in size to the number of class labels, numerous approaches are proposed to exploit the label correlations to promote the learning process [14 44 31. The first-order
approaches disassemble the MLL problem into number of binary classification problems [2 45)
The recond-order approaches consider the label correlations between pairs of labels [8] [1.. The
high-order approaches further focus on the label correlations among label set (26 32 Another
line of research focuses manipulating the feature space via formalizing label-specific feature to
each class label facilitate multi-label classification 23 15 42 In addition, some work focus on
dealing with MLL via deep models. A directed graph over the labels is established via employing

---

GCN to propagate information among all label nodes [. Transformer is leveraged for exploring the label dependency by introducing a ternary encoding scheme to represent the state of labels (0).
In practice, the labeling information is often incomplete in training data, since acquiring exhaustive supervision is extremely difficult. Numerous approaches have been proposed handle the MILL with missing labels [12], which is also termed as MLL with partial labels [7]. A transductive learning method is proposed to concatenate features and labels and apply the matrix completion technique [1]. Then, the inductive learning method is proposed to exploit the structure of specific loss functions to offer efficient algorithms for learning with missing labels 41 Wu [34 recovers the full labeling information of each training sample via enforeing consistency with available assignmnttt and smoothness of aubel-assignment The global and local label correlations are exploited simultaneously
via learning a latent label representation in the missing label cases [47] Durand [ empirically
compare different label-assggmment strategies to show the potential employ partial labels for MIL.. Another method induces a cost function that measures the smoothness of labels and features to alleviate the overfitting issue when training data contains missing labels [6].
Comparing with multi-label learning with missing labels, SPMLL (5) considers the hardest version of this problem, where annotators are only asked provide a single positive label for each training example and no additional negative or positive labels. When collecting multi-label annotations, it may be more efficient to annotate only one label rather than multiple labels for each example. To learn from SPMLL examples, an intuitive solution is "assume negative" (AN) [5), which assumes that unobserved labels are negative and trains the predictive model with binary cross-entropy loss on observed positive labels. Recent work [5] proposes some methods to reduce the damaging effects of false-negative labels. An expected positive regularization [5] is proposed to avoid the problem but the expected number of positive labels of each example should be given. Label smoothing (2X is employed to reduce the impact of the incorrect labels. Another approach (5) estimates the unobserved labels and encourages the classifier predictions to match the estimated labels via binary entropy loss. However, no methods can provide theoretical insights as to why the model trained on SPMLL examples can converge to an ideal one.
3 Problem Setup
3.1 Multi-Label Learning
In MLL, each example is associated with multiple labels, and aims build predictive model which can assign a set of relevant labels for the unseen instance. Let X' = R2 be the q- g-dimensional instance space and y (1.2,.... 6) be the label space with e class labels. Given the MLL training set D [x;; Y:)l1 < i S n where a; E X' denotes the q-dimensional instance and Y, E C is the set of relevant labels associated with æ; where
2 The task of multi-label learning is to induce a multi-label classifier f X + 2 that minimizes the following classification risk:
R(f) Ep(a,y) y) (f (x),
Here, L: RR x 2d + R2 is multi-label loss function that measures how well the model fits the data. Note that a method is risk-consistent if the method possesses classification risk estimator that is equivalent to R(f) given the same classifier f (25] 9
3.2 Single-Positive Multi-Label Learning
Given the SPMLL training set D = {(x,,, 7i)11 < n} where 3 denotes the observed single-positive label of æ; Note that 7/2 E Y, while its relevant label set Y; is not directly accessible to the learning algorithms. For each SPMLL training example (k), we use the observed single-
positive vector 4 17 (2........ [9]T E {0, 13 to represent whether j-th label is the observed
positive label, i.e., 1 1 if Th otherwise E = O. The multi-label vector is denoted by y: = [b] v² .......... y E {0, 13 where y 1 the j-th label is relevant to x; and y 0 if the j-th label is irrelevant. The task of SPMLL is to induce a multi-label classifier f : X + 2 from D, which can assign set of relevant label set for the unseen instance.
Recent work 5 empirically validates that SPMLL would reduce the amount of supervision with a tolerable damage in classification performance. The intuitive solution AN is assuming that unobserved
3

---

labels are negative, which leads to the drawback that introduces some number of false negative labels. Therefore, the SOTA approaches aim to reduce the damaging effects of false-negative labels via employing the learning power of DNNs to achieve good performance in practice. However, there is no existing method that can provide theoretical insights.
4 The Proposed Method
4.1 Risk-Consistent Estimator
To deal with single-positive multi-label learning, the classification risk R()) in Eq. ( could be rewritten as
Ex( Y) [f ())
Σ (f (1), pY((())))())) da
YEc
ΣE (ff p
ply 1|2) p(1" = -1 1(x))(())) dæ
(2)
TEY YEC
-Ep(a.)
ppy) = 1(x)c YEC
-fap(f).
Additionally, employ the widely used loss function in multi-label learning, i.e, binary cross-entropy loss, as the loss function L (ʃ (Y)
(f (2), Y) = Σ
log
fz()) +
Σ
log (1 - fz(x))
jEY
JEY
(3)
Σ e Σ
@
jEY
where e =log (xx) and @ =1 (f) Then, Evec
in Eq. 2 could be
calculated a
E
Eee ) Ø
(4)
YEC
Here, d' p(y = 1/a=) [0. would be regarded the soft label corresponding to class j for By substituting Eq. (4) into Eq. 2 we obtain the following risk-consistent estimator for SPMLL
Rxp(S) (( ply" = 1 Jx)
1
Σ dl e + (1 d8 ) j=1
(5)
Therefore, we could express the empirical risk estimator via
Rap( n
Σ dee + (1-
(6)
ply = 1|x;)cc
j=1
Then, could design benchmark solution via applying the sigmoid function on fy, (x;) to approximate " 1|2:;) and recovering the soft label de corresponding to each example via the label enhancement process in the following subsection.
4.2 Training with Label Enhancement
To recover the soft label vector d; [d] d; ... di E [0, 1Js, SMILE considers the topological information of the feature space and estimates adjacency matrix A [asylaxn with
if (xxx), otherwise
aij =
'The detail is provided in Appendix A.I.

---

where (2yy is the set for k-nearest neighbors of z.
We assume that the latent soft label matrix D [d1; d2...... d., generates the observed logical
label matrix L = [4, 12.....Au,, and the adjacency matrix A. Besides, the observed instance matrix
X [21,, 22.... 2nn] is generated from D and the latent feature matrix Z 1 22; ................................ zn We assume that the prior density p(d) is a Beta density with the minor values & = [ā ā² &e]
and 3 = - 32. Jr i..., p (d) = II;= Beta (di | d ( . Then the prior density pDD) could be the product of each p (d). In addition, We assume that the prior density p(z) standard Gaussian and prior density p(Z) can be represented as the product of each Gaussian p(Z) = Gau(z; 2, 0, 1). Then, the posterior density p (D, L X, A) can be decomposed as follows:
DDD 2 X, A) p(D|L, X, () L. X, A) pDD X, ())(Z)
(8)
where and A can be removed from the condition of pZ L. L, A) because of the indepen- dence between Z and L, A when latent variable D is given in the condition. Here we employ (D X, A) and (Z) X) to approximate the true posterior p p(D X, A) and p p( X) re- spectively. The approximate posterior (D X, A) could be the product of Beta parameterized by a; = Ia at...,,.,,, af and B = [8] "..
qw D D 1
Beta (d'la,.
(9)
Here, the parameters A [an, 022 O.n and = 32, G.] are the outputs of the inference model parameterized by wT GCN [9] with adjacency matrix by A. Let qwy (Z X) be the product of Gaussian parameterized by the mean vector μa, and standard deviation vector o;;
9w> (Z|D. X II Gau(z; pi: o.),
(10)
The parameters A [11, M2...... MLP parameterized by w2.
Mn: Mn 01 ........ Tn] are the outputs of the inference model with
We derive the evidence lower bound (ELBO) 18 on the marginal likelihood of the model to ensure that qw (D, Z[, X, A) as close as possible to p PD, Z X, A
LELBO (.. [lo p( DD Z) log (L + logr(A KL (D)L, X, |l KL( (ZID, X)])P(Z)).
(II)
We further assume that p(X|D, Z) is a product of each Gaussian with means & and p DL is a product of each multivariate Bernoulli with probabilities t, In order to simplify the observation model,
T(m) (T(") T2 (m)
Tn (m)) computed from m-th sampling (mm) with a MLP parameterized
by m and E (m) (m)
(m)
...... {m)) is computed from m7-th sampling ((m) and ((m) with
MLP parameterized by 722 Then the first part of Eq. (TT) can tractable as
M
E (D. (...II.X..A (og..)))loo(((DD).) Z) + logp(L|D)) + logp(A|
Σ tr (I. log T(m))
M
(12)
m=1
+ (I (I-L) log (I-T(m)) - |AA (D (m))
(m)) D (m)]]
| + (m)
X|
Here, S(.) is the logistic sigmoid function, and implicit reparameterization trick 10 and MC sampling 18 37 38 are employed.
The second part of Eq. can be analytically calculated as
T(a + 8))((())((3))
KT (ywe,
log T(a + (T) + (af - a()(a))
(13)
(a a + 3/ () + +B))+ + (8) ())08))
'The detail is provided in Appendix A.2.
5

---

Algorithm SMILE Algorithm
Input: The SPMLL training set D = ( 7/1 ))))+=
Pi the number of iteration and the number of
epoch T;
1: Warm-up 0 by using AN solution, and initialize the reference model w), W2 and observation model ry;
2: Estimate the adjacency matrix A by Eq.
3: for 1, T do
4: Shuffle training set D { %) into 7 mini-batches;
5: for = 1..... do
6:
Update w1, u2 and 17 by forward computation and back pack-propagation by Eq. (16):
7:
Obtain the soft label d; for each example by Eq.
8: Apply the sigmoid function on f+s (æx) to approximate ply 1 l[x:);;
9: Update 0 by forward computation and back- proppapppagationn by Eq. 6
end for
11: end for
Output: The predictive model
Here, r'(+) and (-) are Gamma function and Digamma function, respectively. The third part of Eq. can be analytically calculated follows:
(1 1 log (a))) - (xx))2 (o))²)
(L (Z),, X
p ()) =
(14)
Besides, we could promote the label enhancement process via enforcing that the estimated should of observed labels:
inherit the
M log di (1 (1-1) log dt.
Tc
(15)
Finally, the objective label enhancement TLæ is obtained:
TLE -XCELBO + To,
(16)
where À is a syper-parameter.
SMILE first initializes the predictive network by warm-up training with AN solution, which would attain a fine network before it starts fitting noise [3 Then we could sample the soft label from fixed Beta after label enhancement and the sigmoid function on fr, (æx) to approximate p(y" 1|æ;,) to make Eq. (6) accessible, and train the predictive model O by minimizing the risk estimator. In each epoch, SMILE alternately operates label enhancement process and classifier training process. Algorithm 1 shows the algorithmic description of SMILE.
4.3 Estimation Error Bound
In this subsection, we establish an estimation error bound of the proposed method. The empirical risk estimator according to Eq. can be rewritten as:
(f)
(17)
a! where w/ m. and np!
1-
Then the loss function Lap is
=1y.=1|m..e
Cxp
W + wt
(18)

---

Table 1: Predictive performance of each comparing approach std) in terms of Average precision t. The best performance (the larger the better) is shown in bold face.
Datsiers
SMILET
AN
AX-15
WAN
RoLE
GLoCAL
MEME.
TD2MI.
CAL,50
0.401:10.001
0.382:20.064
0.253:10.031
0.301-100001
0.288:20.000
0.227:100.00
0.233:10.000
0.223:00.00
image
0.784-1-0.044
0.615=0.081
0.621-000073
00855-000088
0.696-0.009
4:7714-0.00
0682-+00001
0.224-0.005
scene
0.740=00:127
08144:0.000
yeast
1.758:10.003
0.755:20.003
0.753:10.000
0.357:100003
0.753:20.003
0.6661:0.002
0.456:10.022
0.323:10.001
corelsk
1.383-1-0.000
0.299=00.005
0.272-00.005
00302+00.004
0.215=0.011
0.218-+-0.00
0.072+000001
0.028=0.001
rovl-sl
0.6161100.0
0.600:00.004
0.581 0:0.02
0.600:10.005
0.570:30.004
0.2291:0.000
0.201:10.003
0.001
sorellkkss]
0.344:100003
0.337:20.003
0.316::0.002
0.341:1:0003
0.288:10.004
0:80110:001
0.029
delicious
0.319-1:0.001
0.297
0.1955-0.005
0.320-30.001
0.199=000004
0:027++000001
0.066-000001
0.028-00.00
1pre122
0.314 1: 0.003
0.292
0.204
0:266:10.006
0.243
0,035 0.0002
0.100:00.001
0.001
espgame
0.259-1-0.00.
0.248:00.002
0.205
0:259:000002
0.216:0.0044
0.:00000000
0.066::0.002
0.00
mirtickr
0.629=00.03
0611-+00004
0.53-400003
sm:2007
0.520 0.002
0.815:20.003
0.802
080011:0.001
0.798:0.005
0.619
0.000:10.000
0 0.001
We define a function space as:
Gap
(z)
Σ (w + +TE)
(19)
and denote the expected Rademacher complexity [ of Gap as:
Ryp (Gap) Ex
sup
oig (2 vi)
(20)
gEGup
where {01, (2,,02..... .,,) is Rademacher variables with 0r independently uniform variable taking value in [+1,-1] Then we have
Lemma 1 We suppose that the SPMLL loss function Lep could be bounded by M, i.e. M JEE Cep (f(x), y), and for any 8 > O, with probability at least 1 then we have
M
log 3
())(s) - Rmp(s) S 25R, (Gxp) +
sup fEr
2
2n
The proof of Lemma [ could be founded in Appendix A.3.
We suppose that the loss function É (f(x),, v) and (f(x). y) are -Lischhitz and p-; Lipschitz with respect to fxx (0 < p < oo and p fo) all y E e. respectively, and wi and ui are both bounded in [0, k). Then, we have
Lemma 2
(G.p) S V2nelp* + p-) Σ
Ri (Hyy);
j=1
where Hy = (h: x0: x+ [y(x)) F) and R,,(H..) = Ex,
(suphet,, 1. E h (2.))
The proof of Lemma 2 could be founded in Appendix A.4.
Based one Lemma I and
2
we could obtain the following theorem:
Theorem 1 Assume the loss function E (f(x), u) and E (f((),)) y) are p² -Lipschiitz and p -Lipschitz with respect to f(x) (0 < p= < oo and 0 < p < c0)) for all y E y and the loss function Exp are bounded by M, i.e., M (EP Cop (f(x), with probability at least 1 - 5,
log 2
R( - - < 4V2xxppp* p Σ Rn M
2n
Here, Jap= = min (f) and f min R()) are the empirical risk minimizer and the true risk minimizer, respectively. The proof could be founded in Appendix A.5. Theorem shows that fap would converge to f" + co and R,, (Hy) 0.
7

---

Table 2: Predictive performance of each comparing approach (tda in terms of One-error 4. The best performance (the smaller the better) is shown in bold face.
Datsiers
SMILET
AN
AX-15
WAN
RoLE
GLOCAL
MLME.
DD2MI.
CAL,50
0.358:1:0.156
0.325.20.133
0.271:0.188
0.420:00115 152
0.557:20003
0.813:1:00011
0.809:100032
0.033
image
0.350+-00066
0.597=00.102
0,577=00.05
0.516-00.000
0.488-000055
0.565--0..01
0.200-1-0023
0.600=00.019
sccne
0.417:00:152
0,412400.145
0.3441-00033
0.477:80.111
0.286--0.024
0.167:10.011
yeast
0.236:1.0.066
0.212:20.012
0.204::00009
0.242-100011
0.239:000010
0.276:1000022
0.285:10.003
0.500::00022
corelsk
0.648 1-0000
0.685=-0.019
0.674=000013
0.656--00003
0.696=00.022
0,76444-0.011
0.947-+00005
0
rovl-sl
0.438 1.0.07
0.405:100011
0.907::0.008
0.419100001
0.465:00.003
0.8001000029
0.782 0.002
0,941 0.004
vorellkkss]
0.65::00.005
0.62:110000
0.667:00.099
0.989110.003
0.300:10.001
0.987
delicious
0.410-2-0.005
0.45::0.002
0.516-00:021
0,405-1-0.00
0.498=00.012
0:996-1-0.003
0.004-000011
0.967
1pma12
0.579:1
0.606:00.022
0.01
0.662:1000055
0.659:0..15
0.907:00.000
0.605 10.012
0.897
espgame
0.662-1-00007
0.686:200012
0.702200:111
06731-00000
0,707=000099
0.000000.000
0.009:00000
0.734=0.0044
mirtickr
0.343-0.017
0.800+00009
0.49:-000000
0.670--0.054
0.447=000011
sm:2007
0.201:10.00
0.215:20.003
0.221::0.000
0.220:100003
0.225:20.004
0.313-1:0.001
022002000002
0.409
Table 3: Summary of the Wilcoxon signed-ranks test for SMILE against other comparing approaches at 0.05 significance level. The p-values are shown in the brackets.
SMILE
AN-15
WAN
ROLE
GLOCAL
MLML
Average precition
0000
win 0.0005
win[0.0005
win
win]O.0005]
One-ertor
win wim(0.0005]
win
win[0.0005]
win(0.0342)
win[0.0005]
Banking lots
0269)
w0000.0005]
00000.0005)
w00(0/0024)
winjn 0005)
Hommiing lnss
027]0.0277]
wamj0.0178)
wl0[0.027]]
win(0.0277]
Caverage
win[0.0425]
wim[0.0005]
(8.(0.1819]
w0.00.0005]
win[0.0005]
win[0.0024]
001[0
5 Experiments
5.1 Experimental Configurations
In the experiments, we adopt twelve widely-used MLL datasets [3], which cover a broad range of cases with diversified multi-label properties. To evaluate the performance of SPMLL methods, we generate the single positive training data by randomly selecting one positive label to keep for each training example in the MLL datasets. For each dataset, we run the comparing methods with 80%/10%%/10 train/validation/tes split. The validation and test sets are always fully labeled. The detailed descriptions of these datasets provided in Appendix A.6. Five popular multi-label metries Ranking loss, Hamming loss, One-error, Coverage, and Average precision [46] are employed for performance evaluation. Furthermore, for Average precision, the larger the values the better the performance. While for the other four metrics, the smaller the values the better the performance.
In this paper, SMILE is compared against four well-established SPMLL approaches including 1) AN 5 which assumes unobserved labels are negative and employs binary entropy loss for the training examples with the modified labels, 2) AN-LS [5] which assumes unobserved labels negative and employs label smoothing (28) to reduce the impact of the incorrect labels (i.c. those labels incorreetly assumed to be negative). 3) WAN (5] which reduces the impact of false negatives by employing the wown-weight terms in the loss corresponding to negative labels, and 4) ROLE [5] which online estimates of the unobserved labels throughout training and encourages the classifier predictions to match the estimated labels via binary entropy loss.
As SPMLL could be regarded as the hardest version of MLL with missing labels, three well- established approaches of MIL with missing labels (also termed as MLL with partial labels) are adopted as the comparing approaches including 1) GL.OCAL [47 which exploits global and local label correlations simultancously via learning a latent label representation in the missing label cases, 2) MLML (34] which recovers the full label assignment for each sample by enforcing consistency with available label assignments and smoothness of labels, and 3) D2ML [24] which utilizes both local low-rank label structures and label discriminant information for learning from missing labels.
For all the DNN-based approaches (AN, AN-LS, WAN, ROLE and SMILE), we adopt three-layer MLP as the predictive model for fair comparisons and use the Adam optimizer [I7]. The mini-batch size and the number of epochs are set to 16 and 25, respectively. The learning rate and weight decay are selected from (10-1, 10-3 10-2) with validation set. Other comparing methods are also selected based on the validation. All the comparing methods run trials (with 80% 109%//0%00% train/validation/tet split) on each dataset.
for all the
8

---

Table 4: Predictive performance of SMILE and its variant (mean-tstd) in terms of Average precision. One-error, and Ranking loss.
Averuge precision
Ome-ermr
Ranking love)
Datasets
SMILE
SMILE-SI
SMILE
SMILE-SI
SMILE
SMILF-SI
CAL500
0.401:10:011
0.409100000
0.358:0.1166
0.343:0.010
0.200-0.011
0.24410.000
image
0.784-90044
0.547=+0.022
0.300=2.004
0.675+00.035
0.170=20.055
0,428==00000
scene
0.841:10.070
0.711+000057
0.278:10.112
0.483-2:0.93
0.086:10.045
0.168:20.057
yeusi
0.758 0.003
0.738.10.00
0.236 1.008
0.246.0.006
0.1611
0.176:10.001
corel5k
0.303-10.007
0.302:10.003
0.648-110008
0.655
0. 0.04:0.0003
0.
rovi-st
0.616:10.001
0.577:+0.004
0.48:=00007
0.477=+00015
0,042:+:00000
0.055==0000
core116k-s1
0.344:1-0003
0.336=20.001
0.641:
0.649:20.002
0.133
0.144:200.000
delicious
0.319110.001
0.293:10.004
0.410
0.43410.023
0.0000
0.146
iaprte12
0,314=t-0,03
0.287=000001
0.579==0.022
00607
0.115=20.002
0.143++0000
espgame
0.259:10.003
0.249110.03
0.602:10.007
0..0
0.170:10.
mirflickr
0.635-1-0.04
0.608:100001
0.335
0.349:10012
0.117:10.002
0.122:1:0.002
tmc2007
0.820:10.002
0.814:0.0011
0.204
0.210:20.003
0.049::00011
0.048:10.000
Table 5: Wilcoxon signed ranks test(at 0.05 significance level).
SMILE against SMILE-SI
Evaluation metric
performance
p-value
Average precision
win
0.00:29
One-error
0.0092
Ranking loss
win
0.0161
Eooch
Hamming loss
win
0.0277
Coverage
win
0.0122
Figure 2: Average precision. Figure 3: Convergence of D.
5.2 Experimental Results
Tables and
and report the results of all approaches on Average precision and One-error, respectively. For each evaluation metric, "1" indicates "the smaller the better" while "J" indicates "the larger the better". The results on other metries are similar and could be seen in Appendix A.6. In addition, Wilcoxon aggned-ranks test [6] is employed to show whether SMILE has a significant performance than other comparing approaches. Table 3 reports the p-values for the corresponding tests and the statistical test results at 0.05 significance level.
Table 3 shows that SMILE achieves superior performance against all the comparing approaches all evaluation metrics (except on Rranking loss and Coverage where SMILE achieves comparable performance against WAN). The superior performance of SMILE provides strong evidence for the effectiveness of risk-consistent estimator for SPMLL. Tables | and 2 show that the performance advantage of SMILE over comparing approaches is stable under varying the number of class labels. In summary, these experimental results clearly validate the effectiveness of SMILE.
5.3 Further Analysis
To show the helpfulness of label enhancement SMILE, vanilla variant of SMILE (named SMILE- s1) is adopted. Here, label enhancement is replaced by approximating df with the confidence of the current model fy (æ,), which is a widely-used technique (9] [22] to approximate the soft label in weakly supervised learning. Table 4 reports detailed experimental results terms of Average precision, One-error, and Ranking loss, respectively. The detailed experimental results in terms of other metries are reported in Appendix A.6. Besides, the performance of each approach with the number of epochs on scene is shown in Figure 2 Wilcoxon signed-ranks test [6] in Table s that SMILE achieves superior performance against SMILE-SI on all evaluation metrics, which clearly validates the usefulness of label enhancement. the number of epochs on delicious, which shows that the estimated soft label could converge efficiently.
the estimated D converges with

---

Conclusion
In this paper, we study single-positive multi-label learning and propose a novel approach SMILE. We derive an unbiased risk estimator, which suggests that one positive label of each instance is sufficient to train predictive models for multi-label learning, and design a benchmark solution via estimating the soft label corresponding to each example in a label enhancement process. The effectiveness of proposed method validated on twelve corrupted MLL datasets.
7 Acknowledgments
This research was supported by the National Key Research & Development Plan of China (No. 202 14202), the National Science Foundation of China (62206050, 62125602, and 62076063), China Postdoctoral Science Foundation (2021M700023). Jiangsu Province Science Foundation for Youths (BK20210220).
References
[1] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463 482, 2002.
[2] Matthew R Boutell, Jiebo Luo, Xipeng Shen, and Christopher M Brown. Learning multi-label scene classification. Pattern Recognition, (7(9)115571177) 2004.
[3] Di Chen, Yexiang Xue, Shuo Chen, Daniel Fink, and Carla Gomes, Deep multi-species embedding, arXiv preprint arXiv:1609.088
2016.
[4] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang. and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 5177-51 -66,,1666666 2019.
[5] Elijah Cole. Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic. Multi-label learning from single positive labels. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 933-942 2021.
[6] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 1(Jan):1-30 2006.
[7] Thibaut Durand, Nazanin Mchrasa, and Greg Mori. Learning deep convnet for multi-label classification with partial labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 647-657, 2019.
[8] André Elisseeff and Jason Weston. A kernel method for multi-labelled classification. In Advances in Neural Information Processing Systems 14 (NIPS 2002), pages 681-687, Vancouver, British Columbia, Canada, 2002.
[9] Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. Advances in Neural Information Processing Systems, 2020.
[10] Michael Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameteriizatoon gradients. Advances in Neural Information Processing Systems, 2018.
[11] Johannes Fürnkranz, Eyke Hüllermeier, Eneldo Loza Mencía, and Klaus Brinker. Multilabel classification via calibrated label ranking. Machine Learning, 73(2):133-153, 2008.
[12] Andrew B. Goldberg, Xiaojin Zhu, Ben Recht, Jun-Ming Xu, and Robert D. Nowak. Transduc- tion with matrix completion: Three birds with one stone. In Advances in Neural Information Processing Systems 23, pages 757-765. Curran Associates, Inc., 2010.
[13] Jun-Yi Hang and Min-Ling Zhang. Collaborative learning of label semantics and deep label- specific features for multi-label classification. IEEE Transactions on Patlern Analysis and Machine Intelligence, 2021.
[14] Thomas Hartvigsen, Cansu Sen, Xiangnan Kong, and Elke Rundensteiner. Recurrent halting chain for early multi-label classification. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1382-1392, 2020.
10

---

[15] Jun Huang, Guorong Li, Qingming Huang, and Xindong Wu. Learning label-specifee features and class-dependent labels for multi-label classification. IEEE transactions on knowledge and data engineering, (1(1))333993333 -323,, 2016.
[16] Dat Huynh and Ehsan Elhamifar. Interactive multi-label enn learning with partial labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9423-9432, 2020.
[17] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, 2015, San Diego, CA, 2015.
[18] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference Leurning Representations, Banff, AB, Canada, 2014.
[19] Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint 1rXiv=1611.07308, 2016.
[20] Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi. General multi-label image classification with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16478-16488, 2021.
[21] Hung-Yo Yi Lo, Ju-Chiang Wang. Hsin-Min Wang, and Shou-De Lin. Cost-sensitive multi-label learning for audio tag annotation and retrieval. IEEE Transactions Multimedia, 13(3):518-
529, 2011.
[22] Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In International Conference on Machine Learning, pages 6500-6510. PMLR, 2020.
[23] Jianghong Ma and Tommy WS Chow. Topic-based instance and feature selection in multilabel classification. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[24] Zhongehen Ma and Songcan Chen. Expand globally, shrink locally: Discriminant multi-label learning with missing labels. Pattern Recognition, 111:107675, 2021.
[25] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
[26] Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. Classifier chains for multi- label classification. Machine Learning, 85(3):333, 2011.
[27] Timothy N Rubin, America Chambers, Padhraic Smyth, and Mark Steyvers. Statistical topic models for multi-label document classification. Machine learning. 88(1-2):157-208, 2012.
[28] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2822222 2016.
(29] Pingjie Tang. Meng Jiang, Bryan Ning Xia, Jed W Pitera, Jeffrey Welser, and Nitesh V Chawla. Multi-label patent categorization with non-local attention-based graph convolutional network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9024-9031, 2020.
[30] Grigorios Tsoumakas and loannis Katakis. Multi-label classification: An overview. Interna- tional Journal of Data Worehousing and Mining, 3(3):1113, 2006.
[31] Grigorios Tsoumakas, Ioannis Katakis, and loannis Vlahavas. Mining lulti-label data. In Data mining and knowledge discovery handbook, pages 667-685. Springer, 2009.
[32] Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. Random k- kabelsssss for multilabel classification. IEEE Transactions on Knowledge and Data Engineering, 23(7):1099-1099 2011.
[33] Alexis Vallet and Hiroyasu Sakamoto. A multi- convolutional neural network for automatic image annotation. Journal of information processing. 23(6):767-775, 2015.
[34] Baoyuan Wu, Zhilei Liu, Shangfei Wang, Bao-Gang Hu, and Qiang Ji. Multi-label learning with missing labels. In 2014 22nd International Conference on Pattern Recognition, pages 1964-1968. JEEE, 2014.
[35] Bin Wu, Erheng Zhong, Andrew Horner, and Qiang Yang. Music emotion recognition by multi-label multi-layer multi-instance multi-view learning. In Proceedings of the 22nd ACM international conference Multimedia, pages 117-126, 2014.

---

[36] Ning Xu, Yun-Peng Liu, and Xin Geng. Label enhancement for label distribution learning. IEEE Transactions on Knowledge and Data Engineering, 33(4):1632 1643, 2021.
[37] Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. Advances in Neural Information Processing Systems, 34, 2021.
[38] Ning Xu, Jun Shu, Yun-Pen Lun- Liu, and Xin Geng. Variational label enhancement, In Proceedings of the International Conference on Machine Learning, pages 10597-10606, Vienna, Austria, 2020.
[39] Ning Xu, Jun Shu, Ren-Y Zheng, Xin Geng, Deyu Meng, and Min-Ling Zhang. Variational label enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence, page in press, 2021.
[40] Renchun You, Zhiyao Guo, Lei Cui, Xiang Long, Yingze Bao, and Shilei Wen. Cross-modality attention with semantic graph embedding for multi-label classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12709-12716, 2020.
[41] Hsiang-Fu Yu, Prateck Jain, Purushottam Kar, and Inderjit Dhillon. Large-scale multi-label learning with missing labels. In International conference on machine learning, pages 593-601. PMLR, 2014.
[42] Ze-Bang Yu and Min-Ling Zhang. Multi-label classification with label-specific feature genera- tion: wrapped approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[43] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In Sth International Conference on Learning Representations, Toulon, France.
[44] Jing Zhang and Xindong Wu. Multi-label inference for crowdsourcing. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2738-2747, 2018.
[45] Min-Ling Zhang and Zhi-Hua Zhou. MI-knn: A lazy learning approach to multi-label learning. Pattern Recognition, 40(7):2038 2048, 2007.
[46] Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26(8):1819 2014.
[47] Yue Zhu, James T Kwok, and Zhi-Hua Zhou Multi-label learning with global and local label correlation. IEEE Transactions on Knowledge and Data Engineering, 30(6): 1081-19981 2017.
12


Reviews:
[{"review_id": "e0KodBGU6ZL", "reviewer": "Reviewer_dP1Q", "summary": "This paper considers a cost-effective way on multi-label learning, i.e., single-positive multi-label learning (SPMLL). Theoretically, an unbiased risk estimator is derived and an estimation error bound based on it is established. Practically, a method called SMILE is proposed, which leverages the latent soft labels recovered in a label enhancement process. Experiments on the benchmark datasets validate the effectiveness of the proposed method.", "questions": "1. In the real world, if data annotators are required to annotate only one relevant label for multi-label instances, a common phenomenon is that people tend to choose the most common and easy to see label for annotation, while the probability of those uncommon and hidden labels being annotated may be greatly reduced. This fact is somewhat different from the experimental setup in which a positive label of randomly selected instances is used to construct a dataset. How to deal with this difference in practical application? The authors should consider and add reasonable discussion.\n2. Noting that the performance of proposed method SMILE on one-error metric is not ideal on image and scene datasets, the authors should add some explanation. As far as I am known, the label cardinality of these two datasets is close to 1. Is this related to this phenomenon?\n", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper studies single-positive multi-label learning (SPMLL), which is a cost-effective way to solve the problem of high annotation cost of multi-label learning. The authors propose an unbiased risk estimator of SPMLL, which is shown to be guaranteed to approximately converge to the optimal risk minimizer of fully supervised learning. The authors give in-depth theoretical analysis to derive the error bound to demonstrate its learning consistency. \nExtensive experimental results on multiple datasets and evaluation metrics demonstrate the effectiveness of the proposed method.\nThere are some questions:\n1. In the real world, if data annotators are required to annotate only one relevant label for multi-label instances, a common phenomenon is that people tend to choose the most common and easy to see label for annotation, while the probability of those uncommon and hidden labels being annotated may be greatly reduced. This fact is somewhat different from the experimental setup in which a positive label of randomly selected instances is used to construct a dataset. How to deal with this difference in practical application? The authors should consider and add reasonable discussion.\n2. Noting that the performance of proposed method SMILE on one-error metric is not ideal on image and scene datasets, the authors should add some explanation. As far as I am known, the label cardinality of these two datasets is close to 1. Is this related to this phenomenon?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RUHoxTdYSvn", "reviewer": "Reviewer_rmev", "summary": "Nowadays, the weakly-supervised problem has become more and more popular since the difficulty in annotation acquiring process. Single-positive multi-label learning, where each example is annotated with only one relevant label, can draw the attention of the AI audience. This paper proposes a novel single-positive multi-label learning method via adopting an unbiased risk estimator enhancing and recovering the latent soft label as a label enhancement process. In addition, this paper establishes an estimation error bound that guarantees the learning consistency of the proposed method and proves that the risk minimizer could converge to the Bayes risk minimizer of fully supervised learning. Extensive experiments are conducted and show significant results. ", "questions": "1.How to calculate the second part of Eq. (9) analytically?", "limitations": "The paper has addressed the limitations that the employment of data annotators might be decreased which could lead to a negative societal impact.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper for the first time proposes an unbiased risk estimator for single-positive multi-label learning and shows that one can successfully learn a theoretically grounded multi-label classifier for the single-positive multi-label learning problem, which is novel and solid. I think this point motivates the novel solution of solving the single-positive multi-label learning problem by recovering the latent soft label as a label enhancement process. The latent soft label is recovered by an inference model and the topological structure in the data, where the posterior density of the latent soft label is inferred via taking on the approximate Beta density and an evidence lower bound for the optimization is deduced. This is novel and interesting. The established estimation error bound guarantees the learning consistency of the proposed method and demonstrates that the obtained risk minimizer would converge to the optimal risk minimizer of fully supervised learning as the number of training data tends to infinity. According to the experiment results, the proposed method achieves superior performance to other comparing methods. The paper is very well organized and easy to follow. The paper can be improved by refining some problems:\n1.\tThe authors should give more details about how to analytically calculate the second part of Eq. (9). \n2.\tThe paper should add more details about label enhancement in related work.\n3.\tThe paper could show the recovered soft label matrix D over all training examples could converge on other datasets. \n4.\tThe charts and figures in the paper take up too much space (like Table 5, Figure 2 and Figure 3), the authors should try to adjust them.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MjgjEAlDyo_", "reviewer": "Reviewer_aB62", "summary": "The paper proposes a novel method for single-positive multi-label learning (SPMLL). The method derives an unbiased risk estimator and establishes a corresponding empirical risk estimator by generating latent soft labels with label enhancement. The authors prove that the proposed method could approximately converge to the optimal risk minimizer of fully supervised multi-label learning, which shows that a single-positive label of each instance is sufficient to train the predictive multi-label model in SPMLL. Experiments on benchmark datasets validate the effectiveness of the proposed method. ", "questions": "why the proposed method could achieve superior performance against the vanilla variant by approximating d_ij with the confidence of the predictive model?", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper proposes a novel single-positive multi-label learning method, which derives an unbiased risk estimator and generates the soft labels in the corresponding empirical risk estimator. The authors prove that the proposed method adopted with the unbiased risk estimator is risk-consistent for SPMLL. To the best of my knowledge, this is the first attempt to consider the learning consistency in SPMLL problem. The theoretical results show that training a fine predictive multi-label model on single-positive labeled data is possible, which is vital for the SPMLL setting. I really appreciate the motivation and the theoretical results of the proposed method. This paper generates soft labels via approximating the Beta density parameterized by GCN, which could be optimized by the deduced evidence lower bound. This is novel and effective to estimate the soft label in the empirical risk estimator. The authors analyzed the experiments in detail and the Wilcoxon signed-ranks show that the proposed method achieves significant performance against other methods. Besides, the authors conduct the experiments to validate the helpfulness and the convergence of label enhancement. The paper is clearly written and well organized. \n\nThis paper could be further improved. The authors should add more details about learning consistency in section 3 since it is vital for SPMLL. In addition, it would be good to add some more discussion about why the proposed method could achieve superior performance against the vanilla variant by approximating d_ij with the confidence of the predictive model.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7vCh5llaTku", "reviewer": "Reviewer_JbLM", "summary": "This paper studies the Single-Positive Multi-Label Learning problem in which the learner only observes one label in the label set of each instance. Based on the estimated unobserved labels for each instance, the authors rewrite the expected risk to obtain an unbiased estimator for the SPMLL problem. The authors propose a label enhancement process to estimate the unobserved labels by enforcing that the recovered label should not be completely different from the confidence estimated by the current prediction. Experiments show the effectiveness of the proposed approach.", "questions": "Is there any analysis that how the correctness of the recovered unobserved labels affects the estimation error bound? It is also suggested to provide some experimental results for the accuracy of recovered unobserved labels.", "limitations": "The authors have adequately addressed the limitations and potential negative social impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n+ This paper studies an interesting but critical problem, which appears in many real-world applications.\n+ The proposed label enhancement process seems to give a nice approximation for the unobserved labels in the SPMLL problem.\n+ The empirical studies seem convincing and show the effectiveness of their proposed method.\n\nWeaknesses\n- The proposed theoretical analysis is established on the assumption that the proposed label enhancement process accurately recovers the unobserved labels. Based on this premise, the theoretical analysis seems standard and lacks new insights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept