Understanding Partial Multi-Label Learning via Mutual Information
Authors: Xiuwen Gong
Dong Yuan
Wei Bao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Partial Multi-label Learning，Mutual Information

Abstract: To deal with ambiguities in partial multilabel learning (PML), state-of-the-art methods perform disambiguation by identifying ground-truth labels directly. However, there is an essential question:“Can the ground-truth labels be identified precisely?". If yes, “How can the ground-truth labels be found?". This paper provides affirmative answers to these questions. Instead of adopting hand-made heuristic strategy, we propose a novel Mutual Information Label Identification for Partial Multilabel Learning (MILI-PML), which is derived from a clear probabilistic formulation and could be easily interpreted theoretically from the mutual information perspective, as well as naturally incorporates the feature/label relevancy considerations. Extensive experiments on synthetic and real-world datasets clearly demonstrate the superiorities of the proposed MILI-PML.

Understanding Partial Multi-label Learning via Mutual Information
Xiuwen Gong 1st Dong Yuan', Wei Bao f Faculty of Engineering, The University of Sydney Hunan Huishiwei Intelligent Technology Co., Ltd. (xiuwen, gong, dong-yuan, yuan, wei. bsydney edu. au
Abstract
To deal with ambiguities in partial multi-label learning (PML), state-of-the-art methods perform disambiguation by identifying ground-truth labels directly. How- ever, there an essential question:" the ground-truth labels be identified precisely?". If yes, "How can the ground-truth labels be found?". This paper provides affirmative answers to these questions. Instead of adopting hade heuristic strategy, we propose novel Mutual Information Label Identification for Partial Multi-Label Learning (MILI-PML), which derived from clear probabilistic formulation and could be easily interpreted theoretically from the mutual information perspective, as well as naturally incorporates the feature/label relevancy into consideration. Extensive experiments synthetic and real-world datasets clearly demonstrate the superiorities of the proposed MILI-PML.
1 Introduction
Partial multi-label learning (PML) 2 a weakly supervised learning problem, where each instance associated with set of candidate labels, but only part of them the ground-truth labels while others are false positive labels.
In recent years, many real-world applications arising due the growing demand for identifying ground- -rrth labels from partially labeled data, which are easier and less costly to obtain. For example, in crowdsoureing image annotation, web image might be annotated online by potential unreliable annotator with many specific labels, but only some of them are accurate.
PML aims to train classifier from partially labeled data so as to predict the ground-truth labels for an unseen instance automatically. The main challenge how to deal with the ambiguities caused by false positive labels in candidate label set. One straightforward way is simply treat all candidate labels equally as the ground-truth labels, and then solve the PML problem by standard multi-label classification methods [3] 4) However, these methods can be easily misled by the noisy false-positive labels the candidate set, and fail to generalize well in testing. To deal with the problem, the state-of-the- teee----t-e-rtttt ahr PML research attempts identify the ground- truth labels directly from the candidate label set, which becomes a popular and effective disambiguation strategy.
However, there are two fundamental issues, "Can the ground-truth labels be identified precisely?"" If yes, "How can the ground-trult labels be found?". Moreover, state-of-the-ar PML methods are constructed by hand-designed heuristic modeling under considerations like feature/label correlations with no theoretical interpretation.
In this paper, instead of hand-made heuristic modeling, we propose a novel MILI-PML model, which derived from clear probabilistic formulation and can be easily interpreted theoretically by infor- mation theory. Besides, MILI-PML naturally incorporates the feature/label relevancy considerations rather than that of heuristic modeling.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

To answer the first question "CAN". we define an e-identifiable score function to evaluate whether PML method is workable for identifying the ground-trus mound-truth labels precisely. Moreover, we demonstrate that the derived MILII-MII is o e-identifiable under certain assumptions and conditions theoretically.
To answer the second question "HOW", we design efficient optimization algorithm called MILII Algorithm to optimize the objective function.
Extensive experiments are conducted on six synthetic datasets well four real-world datasets. The experimental results demonstrate that the proposed MILI-PML consistently outperforms the state-of of-the-art methods in terms of five widely-useds multi-label metrics including ranking loss, hamming loss, one error, coverage, and average precision.
2 Related Work
Partial multi-label learning (PML) 5 6 is different from multi-label learning 7 (88) or partial label learning [10 (3) In PML. each instance is associated with set of candidate labels, which contains multiple ground truth labels and the others are false positive labels. The state-of-the-ar PML studies attempt identify the ground-truth labels directly from the candidate label set.
14 propose PARTICLE to extract credible labels with high confidence via propagation matrix and use the identified labels to train multi-label classifiers. [15] propose DRAMA to get the reliable labels with high confidence by employing the feature manifold, and then use the identified labels to train multi-label classifier.
Moreover, ① propose PML-Ic and PML-fp to optimize the label ranking confidence matrix in training classifiers which considers the label correlations and the feature prototype respectively, [16] develop fPML to optimize the label confidentce matrix by considering feature and label correlations. 17 propose PML--RS to get label ranking which utilizes the low-rank and sparse decomposition to train classifiers while considering the feature and label interdependencies. [88 develop MUSER to train classifiers by decreasing the feature noise and label redundancy via mapping with orthogonality constraint and graph Laplacian regularization, which considers the feature correlation and label correlation simultaneously.
However, existing methods have two common issues. First, models are hand-made heuristic with no theoretical interpretation. Second, existing partial multi-label learning (PML) methods assume that the ground-trutt truuh labels exist in the candidate label set and identify them directly. We then have the question:" Can the ground-truth labels be identified precisely?", ". If yes, "How can the ground-truth labels be found?".
The Model Derivation
In this section, we derive our method, i.e., MILI-PML, from the following probabilistic formulation.
Assume S = {x'', (x) i== ......,]. n.. are drawn i.i.d. times from a probability distribution P. Each observation is a pairwise example (x, y). consisting of d-dimensional feature vector x = "2,.".",,,]""" and a g-dimensional candidate label vector = [y1..... Uql"" drawn from the underlying random variables X = [X Xa)" and Y = [Y,....,
..l.
We adopt g-dimensional binary vector discarded. Notation y,, indicates the vector of selected labels, that is, the full vector projected onto the dimensions specified by Notation y, is the complement, i.e., the vector of remaining candidate labels. The full label vector y can therefore be expressed as = {y Y. which means is the concatenation of y, and yg- In PML. y. is selected from y.
indicating the label is selected, 0 indicating is
Assume that Q a hypothetical predictive model with two parameters: e representing which labels selected and w representing parameter used for prediction. In order identify the ground-truth labels from the candidate label set for PML, we can maximize the conditional likelihood of training dataset with respect to parameters [v. w/. The conditional likelihood given all training examples = ((x), yt) i 1, 1,...,n, nn. can be expressed as follows:
L(v, w|8) = II Q(yl
(1)
i=1

---

For convenience, we use the conditional log-likelihood corresponding to Eq. ① as follows:
E(v, w -
- Σ log Q(y"|xx, ww)
By multiplying and dividing classifier Q by the distribution of identified ground-truth labels given features, i.e., Pxv,,x), we can re-write the above as follows:
Σ Σ log
Q x(y.
e(U, 1 -S)
log Ply x²)
(3)
Similarly, we can expand the second term in Eq. P(ylx), and get the following formulation:
33 by multiplying and dividing the probability
" Σ
Q p(y.
1 a n
p (x) p(y
feu w|S)
log
log
+
Σ
log P(y' (x))
(4)
71
72
Furthermore, we (x denote the expectation of the random variables (X, Y). n + +00, Eq. (4) can be reformated as follows:
When
Q(y..x, P(y,/x) + log P(y,lx)) Bxx.r) flog (log P(y)x)}
{(v. 1/ =E (x.)
(5)
P(y|x)
The first term of Eq. (5) equals to the following KI.-divergence:
E Ex log Q(y,lx,w),
P(y.(x) DKL(
(6)
As y No), the second term of Eq. can be derived into the following formulation:
E(x,x) { 1og P(y, } = - E(x,) log P(y[x)
P(y,, xly.)
Σ
P(x y) log P(x(y.))
P(y|x)
P(y,1x)
(x.)
=- y(x,)) log P(y., x(yo)
Σ
F.) log Ply.ly.)
P(xly,))(yaa) P(yoly.)
(x.y)
(FoN.)
-,,,,, X(V.) + H(Y./V.) -YY (Y,,,
In Eq. (7). x(Y.) is the conditional mutual information and H(Y,|Yo is the conditional entropy. The last equality holds as result of the conditional entropy H H Y.) 0. This is because in PML setting, Y. is selected from Y, once Y. is known, then Y is of course known in advance, meanwhile, Y, is complementary to Y,.. thus, the uncertainty remaining in Ye is zero. i... H (Y,,) Y.) = 0. The detailed derivation of Eq. (7) can be found in the supplementary materials.
Moreover, the third term of Eq. (5) equals the conditional entropy - -H(Y)), i.e.,
n
log Ply Ply'lx) = -H((())
(8)
Combining together, we get the derived objective function as follows:
E(v, w|S)= - (x_(
w)) - I(Y XIX.)
(9)
The first term KL-divergence between the true and predicted ground-truth label distributions given features. The value of this term depends on how well the model Q can approximate P. The second term is the conditional mutual information between the remaining candidate labels and the features, given the identified ground-truth labels. The value of this term depends solely on the choice of ground-truth labels, and will decrease the selected ground-truit tround-truth label set Y, explains more about features X, until eventually becoming zero when the remaining candidate labels Y: are all false positive labels, which contain no additional information about X. The last term is the conditional entropy of all candidate labels given features. This term quantifies the uncertainty remaining in the candidate labels after we know all features of training data, which independent of parameters v and as constant.
3

---

Theoretical Analysis
To answer the first question: the ground-truth labels be identified precisely?", we define an c-identifiable score function to decide whether a PML method can identify the ground-truth label precisely. Intuitively, the score of an identifiable score function increases when ground-truth label is added to the identified label set compared to that of a false positive label. Thus, the c-identifiable score function can be defined as follows,
Definition 1 (c-identifiable Score Function), Let S be the identified label set; yk be a ground-truth label and be a false positive label. Let e be a non-negalive constant. A score function g 2/ + IB is said to be e-identifiable if the following property holds: for every yk & S and yk & S, we have g(S U {ya}) g( U { 2
We will use this definition to derive the following theorem, which demonstrates that our objective function is e-identifiable score function and is capable of identifying the ground-trun trutt label from the candidate label set.
Theorem 1. Let yk and Vs denote any unidentified ground-truth label and any false positive label respectively. Let Yş and Yg represent the random variables of yk and yı respectively, and that yr. V% E E and yu- bh & v. Then, for any identified ground- truth label set v, our objective function f(u, w/ S) is an e-identifible score function on condition that,
I(YR, X) - I(YK, Y.) + I(Y,, X. - DRLι 2 I(Y T(V V.) + I(Y Y. - DKL2
where DKLI = DKL. ((Y. (Y.K)))X)))0X0.. UVAAX,
and DKL2 DKL U
(10)
Proof.
g(Su (uct) -0(SS (vx))
{
DKL. -DRL UYIY,UYNXIIIO UV/Ix, + DKL UPY.U..YYYMMI.
(11)
Firstly, we derive the first part Eq. [1], i.c., -/ (x. +1 +XXx,XX..XXx,Ux.) Due to the chain rule that I(Y, I(Y,, X) + I(Y,, X(Y.). we have
-1 Ye.
=[(1(X,) 1(Y, -__XX,X))((X,UXX, x) 1(Y, UY,,X) 1(Y,X)
= =(Y,..X) + I.Y,,, - -(Y,))
(12)
I(Y,,) + I(Y,, -IVY,, + I(Y,, - I(Y,X)
Similarly, can get,
VV. ((Y,, x) 1( - 7(8 Y.) Y - XYY (13)
Therefore, we have the following expression,
The first part of Eq. - (Yo Yv, X|Y UY,) + +XXoXXX.XXX.UU
(14)
[(Y X) - I( Y.) + I(Y,, x)) [((Y,, X) I(Y, Y.) + I(Y2, Y./))
where I(Yx, X) the mutual information between a ground- tround--ruuth label yı and features, which means the correlation between the two items; I(Y2. Y.) is the mutual information between ground-truth label y/ and the existing identified ground-truti label set, which means the redundancy between the redundancy, which means the good information
two items; 7(Y,, Y,, X) is the still remaining between ground-truth label yh and the existing identified label set after features are known.

---

Thus, the interpretation of the first part of Eq. (14,, i.e., I(Y X) - I(Y,, + Y, is the overall correlation of a ground truth label y/ and features x, which equals the correlation between ground-truth label yv and features x minus the overall redundancy of label ye with existing identified label set, based on which we can evaluate the effectiveness include the label y/ into the ground-truth label set.
Similarly, the second part of Eq. [14] stands for the overall correlation between a false-positive label yg and features x, which equals the correlation between a false-positive label y/ and features x minus the overall redundancy of label y/ with existing identified label set.
is naturally hold that the first part should be larger than the sencond part as the overall correlation between ground ground-truth label ys and features ought to be stronger than that of false- balse-positive label yE and features.
After that, we illustrate the second part of Eq. (T),, i.e., DRI. (P(Y. XOII
DEL ((YY, YX/X) Yg/X, w)) The interpretaion of D P(Y. Yz/X))IQ(Y,
Yx|X, w)) is how well can the predictive model Q approximate the true probability distribution P when a ground truth label yı is added to the existing identified label set that is, the information loss when using Q to approximate P. The more precise the approximation is, the smaller is the KL-divergence value.
Similarly, DKL (XXIIXX w)) can be interpreted as how well can the predictive model Q approximate the probability distribution P when false-positive label yı is added to the existing identified label set v. The more imprecise the approximation is, the bigger is the KL-divergence value.
naturally hold that the KL-divergence value should be smaller when the ground-truth label ye is added to the existing identified label set than that of the false-positive label yg. that is, DEL P(Y, U (YY XY) vx)) 2 DEL P(Y, U.Y.UY.&JJN (Y/ U Xx, w)).
Therefore, combined with the derivation of the first part of Eq. (1T),
get the condition for objective function E(V, w/ w|S) to be an e-identifiable score function as follows,
I(YR, X) L(Y Y.) I(Yx, Y. X. DKLI 1(Y,, X) I(Y Y.) + +(Y,, Y, DKE2 where DKLL = DKL P(Y, U YXIOXX, u)) and DKL2 DKL P(Y. U
YE X X,) which concludes the proof.
The condition of Eq. (10) requires that the overall correlation between a ground-truth label yk and features should be stronger than the overall correlation between a false-positive label yı and features. From the above theoretical analysis, we can see that the ground-truth labels can be identified correctly by the proposed MILI-PML under given conditions and assumptions in Theorem
Remark.
5
Optimization
To answer the second question: the ground-truth labels be found?", we design efficient algorithm to optimize the proposed MILI-PMI.
The goal of partial multi-label learning (PML) is to maximize L(v, w S of Eq. 9 which is equivalent
to minimizing -((,, w w(S). Consequently, we can rewrite the objective function of MILI-PML as follows:
-((u, w/ S) = DDrr,P(Y, X) w)) I(Y X|Y.) H(Y
(15)
As the last term HY is a constant, we only need to consider optimizing the first term and the second term together in Eq. (5). Thus, minimizing -l(u, w|S) is equivalent to minimizing DEL (P((Y
X, w)) I(Y, x(Y,), that is,
arg min - (v, w S) = arg min D
X, w)) + ((,,)
(16)
0,10
1,20
5

---

Algorithm Alternating Optimization (MILI Algorithm)
Goal: Solve the Optimization Problem of Eq. (16) identifying the ground-truth label set well as training predictive model parameter w:;
Input: training data S {(x', .y') -=....... 1,...,n].
Output: the predictive model parameter w.
I: Initialize the ground-truth label set v² for each instance by randomly choosing from the corre- sponding candidate label set 5"; t 0; (u, w) =
2: repeat
3: Fix v, update in Eq.
according to Gradient Descent Algorithm.
4: Fix w, update v in Eq. I9
according to Algorithm 2
5: t=t+1;
6:
Calculate Ft+1(1, (v, w):
(v..)_ |
7:
until
8;
(0,w)
8: Output the optimal solution v² for each instance x {I....,n) and the predictive model parameter wo
We design alternating optimization algorithm called Mutual Information Label Identification Algorithm (i.c., MILI Algorithm) to optimize the two parameters v (the ground-truth label set parameter) and w (the predictive model parameter) in Eq. (16) by fixing and updating the other. Complete procedures of MILI Algorithm are summarized in Algorithm
For convenience, (P(Y, Algorithm
denote the objective function in Eq. (16) as F(v, w)
w)) I(Y,, X(Y,), which is used to set the stopping creteria in
Specifically, when is fixed update w. Eq. (66 is equivalent optimize the following problem as the second term I(Y, e. x)),) becomes a constant.
arg min Ex),)) log Pry, |x) (x) log Q(y. xx, w) arg min - Exx,,) log Q(v,/x,, w) (17)
Moreover, use the sample estimation approximate (xx) log Q |x, w) based on the Law of Large Numbers as follows:
E E(x,x) log
RJ log (xy"
(18)
To this end, we can employ the gradient descent algorithm optimize Eq. 17
Next, we fix w to update The first term of Eq. (T6) can be decomposed be Ex,y) log P(y,(x)
Exx,) log ((y,[x)) Meanwhile, due to the chain rule that I(Y, I(Y,. X) I(Y,, X(),), min-
imizing I (Y,,, X(Y,) is equivalent to maximizing I I(Y.,) X). Therefore, Eq. (6) can be reformulated as follows:
arg max - E( log Ply, + (x.v) log Q(y,,x) + I(Y, X)
(19)
Moreover, we can identify ground-truth labels yı: from one by one and add to Thus, Eq. (19 is cquivalent to the following formulation:
arg max - -Ex,x) log P(ynlx,y.) + Exxx; log Q(ulxx,y.) + ((x,, EE
(20)
In practice, we can the empirical distribution and sample estimation approximate the three terms in Eq. (20) based on the Law of Large Numbers. To this end, we can employ greedy strategy to optimize Eq. 19 Procedures of this implementation can be found in Algorithm
Here, we explain some main steps Algorithm I Step 2-7 summarizes the core iteration procedures, among which Step 3 and Step 4 are the alternating strategy to optimize the two parameters and v, respectively. Step 6, the objective function value F(r, w) caleulated. Finally, the stopping condition is set to 8-optimal and we choose 6 to be 10-5 55 in practice (Step 7).

---

Algorithm 2 Fix w, update v.
Goal:
Solve the Optimization Problem of Eq. (9
Input: training data S {(x',y" : - ,....,,:: Output:
the ground-truth truth label set v² for each instance E fl....,n]. n..
Initialize 0; oi
and let oy be the initial candidate label set for each instance x².
i (l..... n):
2: while T(Yx. x) do
3: Identifying the ground-truth label y by Eq. (20):
4: Update the ground-truth label set by o/+1 = of y
5: Update the remaining candidate label set by 8/41 = 0f V/
6: 1= +++1; 1;
7: end while
8: Output o² by
Experiments
In this section, we conduct experiments to evaluate the classification performance the proposed MILI-PML and compare it with six state-of-the PML methods.
6.1 Datasets
Experiments are conducted six synthetic PML datasets and four real-world PML datasets (i.e. YeastBP [16]. Music-emotion [9] Music-style [19], MIRFlickr [99 14] of different scales, the characteristics of which are summarized in the supplementary materials. For real-world PML datasets, candidate labels are collected from web users which are further examined by human labelers to specify the ground-truth labels. For synthetic datasets, given the configuration strategy multi- label datasets in [1], we construct the candidate label set by randomly choosing irrelevant labels together with the ground-truth labels (GLs) for each multi-label instance. Specifically, different settings are considered by varying the average number of candidate labels (#CLs (avg.)) for each multi-label dataset. Accordingly, in this paper, we generate thirty synthetic PML datasets. For brevity, we report the detailed results of two configurations for each dataset, i.c. avg. #CLs (avg.) being 7 and 11 for Enron, Corel5k, Eurlex-sm; 9 and 13 for Eurlex-ed and Mediamill; 45 and 65 for CAL500.
Table 1:
Experimental results of the proposed MILI-PML with six state-of-the art PML baselines on four real-world as well as six synthetic PML datasets in terms of ranking loss. The best result (the smaller the better) is in bold.
Dataset
#CLs (avg.)
MILLPML
PAR-VLS
DRAMA
PML-kc
IPML
PML-LRS
MUSER
YeastBP
30.43
.3772.017
436:.032
.004.021
382=-036
A142.012
4182.031
.341:.015
Music-emution
5.29
.1711.008
260 000
2184.013
254-0006
.3522.015
281110005
.1941.021
Music-style
6.04
.1621.007
122
.1781.013
213==013
238-.019
.1792.007
007
.1712.006
MIRHicckr
3.35
.881 1. 000
203- 1007
11941.10
.14
008
.163 025
1074 002
.0934.000
Ennon
0822-0002
297=
.194+.012
-338-1004
-128+.017
207
.114:.003
11
.096:..04
312=-006
210::.015
.31--002
.127=.008
008
.215
.12:+.004
Corel5k
.0121.015
345
.1931.052
.161=0013
.1322 027
.1932.016
.11:1.004
11
.055+.007
383-
.201+:065
.171 771-014
1338+-013
202+ 021
.017+
Eurlex-sm
7
.41:1.003
058::014
.0512.007
079--017
.313= 072
.1832.006
043:1.055
.047-1.006
.067.2.009
.063:.016
.081-.016
.3682.016
.1974.008
.0491.003
1033
Eurlx--ed
13
0022+.017
.069=0025
.071+:013
.92--018
412+.024
.277-011
.043:.002
45
.175-1.007
353=.014
.3351.007
316--008 008
268 015
2811:013
.1741.024
CAL.500
65
.203+-027
471
.317+.016
365--014
.114
019
.367+-016
.213+
.127:1.099
.13520008
2011:0007
216=:015
203=.007
.13
.1734.002
Mediamill
13
.1511.004
198 004
.0111.012
.316-000
009
2252.008
2222-023
.1741.021
6.2 Baselines
We compare the proposed MILI-PML method with the six tfat-oof-the-an PML approaches.

---

Table 2: Experimental results of the proposed MILI-PML with six state-of-the-arr PML baselines four real-world as well as six synthetic PML datasets terms of average precision. The best result (the larger the better) is in bold.
Dataset
(avg.)
MILL-PMI.
PAR-VES
DRAMA
PMI-le
TPMI.
PMI-LRS
MUSER
YeastBP
30.43
151+:027
0824.031
0834.017
140+..03
.994F.021
.0344.023
.154=003
Music-emotion
5.29
6081.018
.5274.006
5822.012
54111.23
.3884.015
.5162.014
.5982.033
Music-style
6.04
.722-0017
2714.0031
.6932.013
.2772.010
.69:4.017
716:2.018
.718=0013
MIRFlickr
807
.6851.017
.7074.014
.7434.018
.731::015
.7961.012
.8011.016
.7841.002
60111.066
.6331.02
.6794.003
.7514.012
.7824.011
.771=.003
Enron
11
683=.005
5877.0066
.3562.012
.6001.004
.6701.006
.8831.007
.6812.005
Corel5k
203:-0012
2012
23351.014
25331.023
.264:.0117
237
-280=000
2799=0007
1964.006
2184.005
.2222.013
.258:.015
.217
.276=.015
752--013
7411.024
.7441.017
.7182.037
.8551.022
6991.016
751
Farlex-sm
11
749:1.016
.721-.016
.728+.013
.7162.006
.6284.027
6154.017
.748--023
.13=:0016
733+2.013
.7274.016
.7192.024
6864.010
696
-23=0023
Eurlex-ed
.748+0027
.728+.015
.725+.023
.715+.017
.664+.014
6811-.016
.752+-012
.5634.027
.5115.025
.16:5.023
CAL500
122
4324:012
.4811.015
434.1.015
A12:1002
441
-18
Mediamill
.769:.019
756+.018
.6984.017
6655..025
.954.0117
6894.010
.716=0012
13
.728:.014
.699..024
.6874.014
.6851.019
.744.0188
.8661.013
.702-.021
Table 3: Experimental results of the proposed MILI-PML with six state-of-the-arl PML baselines on four real-world as well as six synthetic PML datasets in terms of coverage. The best result (the smaller the better) in bold.
Dataset
#CLs (avg.)
MILI-PML
PAR-VLS
DRAMA
PML-Ic
1PML
PML-LRS
MUSER
YeastBP
30.43
422+ 014
7314.032
.4174.025
4894.036
4944.012
.66
031
431+.015
Music-emotion
5.29
.373 008
4.08
4284.013
4344.006
4514.015
4834 0055
.37++002
Music-style
6.44
182= 007
2084.021
2174.032
3674:013
2384.019
2792 007
2T6:2.000
MIRFlickr
3,35
20851005
263.1007
289.1.010
248.1.088
.2671.025
2887:0002
233:1.016
7
.281 022
397.1.00 007
394.1.012
4311.004
.4211.017
.151.021
.3141.05
Enron
11
293=.014
416:1.026
.1154.015
4461.022
.4261.018
.174.017
.323:.002
7
.255 225-.017
A15+.070
4394.002
4.01.
34244007
293+:.015
.214:.014
Corelōk
11
.275-1027
46311.26
45111.025
.3731.014
.3381.023
.372..021
.27.11005
Furlex-sm
7
.1522.003
.1222.014
.1711.007
.3881.017
.3631.012
.1834.006
.174:.005
11
11871.006
263
256
4711:016
.4261.016
266
.219:.003
Eurlex-ed
9
239- 0166
244.1.013
278.1.018
4831.002
.3661.029
238810018
.2111.013
13
246 017
2794.025
27741013
4921.018
.1124.024
2471.010
.245:1.012
45
5857.027
8724.024
8381.027
9361.038
0254.025
865
67941.27
CAL.50
65
.93--0027
953+.012
.17720016
.9541.014
.8814.019
8784.016
.7124.012
Mediamill
20091.019
2112.018
.28.1.024
3171.012
.2171.013
2911.019
.212:.022
13
215=-021
.21
3622.015
3761.007
.2661.017
.222.021
.2571.110
PARTICLE [14]: An identifying method, which tries to extract credible labels with high- confidence values by label propagation procedure, and then trains classifiers by applying two exisiting multi-label models, which are PAR-VLS and PAR-MAP for short. Here, we choose PAR-VLS for comparison.
An identifying method, which tries get the reliable labels with high- confidence by considering the structure of feature space, and then induces gradient boosting model to train classifiers.
DRAMA 15
PML-fp and PML-Ic D: An embedding method, which attempts to figure out the label confidence by minimizing the ranking loss and exploiting data structure information with two models: one considering feature prototype (i.e., PML-fp) and the other considering label correlations (i.e., PMI.-lc). Here, we choose PMI.-Ic for comparison.
fPML [16]: : An embedding method, which figures out the label confidence by adopting feature and label coherent matrix to factorize the original matrix for prediction.
PML-LRS [7 An embedding method, which utilizes low-rank and sparse decomposition to capture the ground-truth label matrix and irrelevant label matrix from the observed candidate label matrix.
MUSER [18]: An embedding method, which considers redundant labels together with noisy features and figures out the label confidence via optimizing correlation matrix.

---

For all PML baselines, we set the orade-off parameters suggested in the original papers. Details can be found in the supplementary materials.
For MILI-PMI,, we employ the simple Binary Relevance 220 21 as the predictive classifier Q in the MILI-PMI. model. In addition, LIBLINEAR [22] with 1.2rregularized square hinge loss is also employed to train the binary classifiers in Binary Relevance (BR).
Evaluation metrics: We employ five widely-used multi-label metrics including ranking loss, hamming loss, one error, coverage, and average precision to evaluate the performance of all methods, More details about these evaluation metries can be found in (3 24 25]. Besides, each dataset, five-fold cross validation is performed where the mean metric value as well as standard deviation are recorded for each comparing method.
6.3 Experimental Results
Due to the page limit, only report the performance comparisons of the proposed MILI-PML with six state-of-the-an art PML methods on four real-world datasets and six synthetic datasets terms of ranking loss, average precision and coverage in Tables I 2 3 respectively. Besides, similar results can be observed in other metrics and we report the detailed results of hamming loss and one error metrics in the supplementary materials. From the overall results, make the following observations:
The proposed MILI-PMI. consistently outperforms all baselines on most real- world datasets, like Music-emotion, Music-style and MIRFlickr datasets, while is comparable to the best performance on YeastBP dataset. For example, MILI-PML is comparable to MUSER in terms of ranking loss and average precision, while comparable to DRAMA in terms of coverage.
MILI-PML is superior all baselines on most synthetic datasets, like Enron, Corel5k, Eurlex-sm, CAL,50 and Mediamill while is comparable to the best performance on Eurlex- ed dataset. Specifically, MILI-PML is comparable MUSER in terms of ranking loss and average precision, comparable to PML-LRS and MUSER in terms of coverage.
It is obvious to observe that MILI-PML performs the best on most real-world as well as most synthetic datasets, except on YeastBP and Eurlex-ed. This is maybe because the number of feature and class combinations is relatively large compared with the number of instances in YeastBP and Eurlex-ed datasets, which increases the difficulty for identifying the ground- truth labels.
The overall results demonstrate the superiorities of MILI-PML, which mainly owes the natural consideration of the relevancy between features and labels as well as the overall redundancy of including an identified label into the ground-truthe label set in the derived MILI- PML. This aligns with our theoretical analysis that MILI-PML is e-identifiable and capable of identifying the ground-truth labels correctly on the given conditions and assumptions of the data distribution.
7 Conclusion
This paper provides new insight into partial multi-label learning problem from the perspective of mutual information. Instead of using hand-made heuristic modeling, we propose a novel method called MILI-PMI,, which is derived from clear probabilistic formulation directly and can be easily interpreted theoretically by information theory as a result of naturally incorporating the relevancy considerations. Moreover, we present two fundamental issues about the assumption of existing PML research. To answer the first question "CAN", we define an e-identifiable score function to evaluate whether PML method is workable for identifying the ground-truth labels precisely. Furthermore. we demonstrate that the derived MILI-PML is c-identifiable under certain assumptions and conditions theoretically. To answer the second question "HOW". we design an efficient optimization algorithm called MILI Algorithm to optimize the objective function. Extensive experiments are conducted on six synthetic PML datasets and four real-world PML datasets. The experimental results clearly demonstrate that the proposed MILI-PML consistently outperforms the state-of-the-arr methods in terms of five widely-used multi-label metrics including ranking loss, hamming loss, one error, coverage, and average precision.

---

References
[1] Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning, In AAAI, pages 4302-4309) 2018.
[2] Gengyu Lyu, Songhe Feng. Yi Jin, Tao Wang. Congyan Lang, and Yidong Li. Prior knowledge regularized self-representation model for partial multilabel learning. IEEE Transactions on Cybernetics, 2021.
[3] Weiwei Liu, Ivor W. Tsang, and Rohurt Müller. An easy-to-hard learning paradigm for multiple classes and multiple labels. Journal of Machine Learning Research, 18:94:1-998 18::31199:38, 2017.
Matthew R. Boutell, Jicho Luo, Xipeng Shen, and Christopher M. Brown. Learning multi-label scene classification. Pattern Recognition, 1777)7177711771 2004.
[5] Ming-Kun Xie, Feng Sun, and Sheng-Jun Huang. Partial multi-lahel learning with meta disambiguation. In SIGKDD, pages 1904-1912, 2021.
[6] Gengyu Lyu, Songhe Feng, and Yidong Li. Partial multi-label learning via probabilistic graph matching mechanism. In SIGKDD, pages 135-113,50 2020.
[7] Weiwei Liu and Ivor W. Tsang, Making decision trees feasible in ultrahigh feature and label dimensions. Journal of Machine Learning Research, 18:81:1-81:36, 2017.
[8] Min-Ling Zhang and Zhi-Hua Zhou. ML-KNN: A lazy learning approach multi-label Iearning. Pattern Recognition, 40(7):2038-204) 2007.
[9] Weiwei Liu, Donna Xu, Ivor W. Tsang, and Wenjie Zhang. Metric learning for multi outpit tasks. IEEE Transactions Pattern Analysis and Machine Intelligence, 41(2):408 422, 2019.
[10] Xiuwen Gong, Dong Yuan, and Wei Bao. Top-k partial label machine. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[11] Yan Yan and Yuhong Guo. Partial label learning with batch label correction. In AAAI, pages 6575-6582, 2020.
[12] Jiaqi Lv. Miao Lei Feng, Gang Niu, Xin Geng. and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In ICML, pages 6500-65 6510, 2020.
[13] Xiuwen Gong, Dong Yuan, and Wei Bao. Discriminative metric learning for partial label learning. IEEE Transactions Neural Networks and Learning Systems, 2021.
[14] Jun-Peng Fang and Min-Ling Zhang, Partial multi-label learning via credible label elicitation. AAAI, pages 3518-3525, 2019.
[15] Haobo Wang, Weiwei Liu, Yang Zhao, Chen Zhang, Tianlei Hu, and Gang Chen. Discriminative and correlative partial multi-label learning. In IJCAI, pages 3691-3697, 2019.
[16] Guoxian Yu, Xia Chen, Carlotta Domeniconi, Jun Wang, Zhao Li, Zili Zhang, and Xindong Wu. Feature- induced partial multi-label learning. In ICDM, pages 1903, 2018.
[17] Lijuan Sun, Songhe Feng, Tao Wang, Congyan Lang, and Yi Jin. Partial multi-label learning by low-rank and sparse decomposition. In AAAI, pages 5016 5023, 2019.
[18] Ziwei Li, Gengyu Lyu, and Songhe Feng. Partial multi-label learning via multi-subspace representation. In IJCAI, pages 261 1-22118, 2020.
[19] Mark J. Huiskes and Michael S. Lew. The MIR flickr retrieval evaluation. In SIGMM, pages 39-43, 2008
[20] Min-Ling Zhang, Yu-Kun Li, Xu- YunY Liu, and Xin Geng. Binary relevance for multi-label learning: an overview. Frontiers of Computer Science, 12(2):191-202, 2018.
[21] Grigorios Tsoumakas, loannis Katakis, and loannis P. Vlahavas. Mining multi-label data. In Data Mining and Knowledge Discovery Handbook, 2nd ed, pages 667-685. Springer, 2010.
[22] Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsich, Xiang-Rui Wang, and Chih-Jen Lin. LIBLINEAR: A library for large lincar classification. Journal of Machine Learning Research,
2008
[23] Johannes Fürnkranz, Fyke Hüllermeier, Encldo Loza Mencía, and Klaus Brinker. Multilahel classification via calibrated label ranking. Machine Learning, 73(2): 133-153, 2008.
[24] Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-labe learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26(8):1819-18 2014.
[25] Zhi-Hus Zhou and Min-Ling Zhang, Multi-label learning. In Encyclopedia of Machine Learning and Data Mining, pages 875-881 881. Springer, 2017.
2017.
10


Reviews:
[{"review_id": "ickSDwRtBtx", "reviewer": "Reviewer_gNGy", "summary": "This paper tackles the partial multi-label learning problem from a probabilistic formulation with theoretical interpretation and derives a novel objective function based on the concept of Mutual Information (MI). To optimize the objective function, the authors propose an optimization algorithm to identify ground-truth labels and update classifier alternatively.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of studying the PML problem with tools from information theory is novel. The training objective function derived by the authors is reasonable and can be well explained theoretically. The technique and theoretical parts are solid and seem sound to me, which can benefit the following research in PML. The contribution is highly related and important to the community.\n\nThe overall structure and writing are clear and easy to follow.\n\nI have some questions: \n- Why there are no related references following the statement “the state-of-the-art PML methods” in the first section?\n- In the paper, the authors claim that y, y_v are the vectors, but describe that “y_v is selected from y”. I believe there are some notation abuses, and the authors should modify it to make the mathematical expression more strict.\n- The notation v is used as vector in section 3, but used as the identified ground-truth label set in section 4. Different notations are suggested to use.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Why there are no related references following the statement “the state-of-the-art PML methods” in the first section?\n- In the paper, the authors claim that y, y_v are the vectors, but describe that “y_v is selected from y”. I believe there are some notation abuses, and the authors should modify it to make the mathematical expression more strict.\n- The notation v is used as vector in section 3, but used as the identified ground-truth label set in section 4. Different notations are suggested to use.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "a8uI-gD3uWl", "reviewer": "Reviewer_JowG", "summary": "The paper addresses the problem setting of partial multi-label learning (PML). The task is to select the ground-truth labels from a set of label candidates and disregard false-positive or noisy labels that may occur in the labelling process. The authors propose MILI-PML that exploits dependencies between labels and features for selection and relies on the assumption that ground-truth labels have a higher overall dependency between labels and features than false-positive ones. The paper provides theoretical analysis to show that the model can identify ground-truth labels from a candidate set correctly and propose a corresponding optimization procedure. Further, the authors conduct experiments to demonstrate the effectiveness of MILI-PML and claim that the method achieves superior performance compared to other SOTA methods.\n\n** Thanks for the response to this review. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper considers the problem setting of PML from an information-theoretic perspective and justifies the proposed method MILI-PML accordingly. With this perspective, the authors provide a contribution to the field of PML. The authors include recent methods and compare the performance of the proposed method against those in the experimental part.\n\nThe quality of the paper is not sufficient yet and needs improvement. Some details of the theoretical analysis need to be clarified, and the experimental part needs to be extended by additional assessments, especially concerning the limitations of the proposed method.\n\nThe paper makes two assumptions on which MILI-PML is based: (1) \"that the overall correlation between a ground-truth label $y_k$ and features should be stronger than the overall correlation between a false-positive label $y_{\\bar{k}}$ and features\" and (2) \"It is naturally hold that the KL-divergence value should be smaller when the ground-truth label $y_k$ is added to the existing identified label set $v$ than that of the false-positive label $y_{\\bar{k}}$\".\n\nThe first assumption seems plausible on an intuitive level. However, it is unclear to what extent this assumption is fulfilled in the datasets used. The reported results give little information about the assumption, as only the performance on different evaluation metrics is reported. Whether the assumption tends to be fulfilled within a dataset could be shown by providing measures of the dependency between individual features and labels, such as a suitable correlation coefficient. Further, it is unclear to what extent MILI-PML exploits dependencies between features and labels. Therefore, it would strengthen the paper to assess how the performance of MILI-PML to select or reject a label correctly is related to the level of dependency between the features and a ground-truth or false-positive label. This influence could be analyzed by generating artificial data in which the dependency between a label and the features is controllable and evaluating the performance of MILI-PML under different configurations. This might also indicate potentials limitations of the proposed method.\n\nRegarding the second assumption, I suppose that the relation between $D_{KL}(P(Y_v \\cup Y_{\\bar{k}} | X) || Q(Y_v \\cup Y_{\\bar{k}} | X,w))$ and $D_{KL}(P(Y_v \\cup Y_{k} | X) || Q(Y_v \\cup Y_{k} | X,w))$ depends on both the choice of model $Q$ and its parameterization $w$. I expect that depending on the choice of $Q$ and $w$, the relation of the two KL-divergences can take on any value. Please clarify why your assumption that in general $Q$ approximates the true probability distribution $P$ better when a ground-truth label is added to $v$ rather than a false-positive one holds.\n\nIn Chapter 6.2 (and Chapter 3 of the Supplementary Material), it is stated that the hyper-parameters of the baselines were set to the corresponding values of the original papers. Optimize the hyper-parameters for the baselines for a fair comparison with MILI-PML and state the range of hyper-parameters considered and methods used for selection.\n\nThe submission is generally well written and organized. Mathematical statements are presented in (great) detail and are mostly easy to follow. Please see section „Questions and minor remarks“ for minor issues. However, I would recommend a language review before publication.\n\nWith the provided information about the implementation and the experiments, it is not yet possible to reproduce the results. Therefore, please provide the following details to improve reproducibility:\n\nNumber of evaluation runs and calculation of error bars in the results tables.\nDetailed descriptions of how the first and third quantities are calculated in Eq. 21.\nDetailed descriptions of how the real world datasets are used in the experimental part. For example, the MIRFlickr dataset [1] contains Flickr tags (raw form and processed form) and annotations. It is unclear which of these are used as labels, as the relevant statistics do not match the information on the website.\nThe experimental section indicates promising results for MILI-PML and could be of significance for the area of PML due to both the performance improvement and the theoretical analysis. However, the experimental part does not yet evaluate the method thoroughly enough, and therefore the significance cannot yet be assessed finally.\n\nQuestions and minor remarks:\n\nCan you explain the change of distribution in Eq. 7 in which the joint probability distribution of features and labels $P(x,y)$ is exchanged with the joint probability distribution of disregarded labels and selected labels $P(y_{\\bar{v}, y_v})$?\n\nMinus is missing at the last line of Eq. 1 in the supplementary material.\n\nTypos or missing words in line: 56, 79, 136, 147, 151, 175, 184, 258, 278, 283.\n\n[1] http://press.liacs.nl/mirflickr/", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper partly addresses the limitations of the proposed method by stating assumptions that need to be fulfilled. It does not address the potential negative social impacts of the application of the proposed method. The authors state in the theoretical analysis that the algorithm only works under given conditions and certain assumptions. Limitations due to configurations (for example, which predictive model $Q$ is used) are not described. The experimental part does not examine whether the central assumptions about dependencies in the data hold and to what extent MILI-PML exploits them. I would appreciate it if the authors would comment in the paper on the potential impacts of their method. For example, I would think the method could be of interest in the context of automatic filtering of \"noisy\" or \"false\" labels and could lead to bias or discrimination depending on the domain of the data (e.g. personal data). Moreover, there might be a risk considering the quality of the data used. For example, if a large amount of structurally misannotated data is included in the underlying data set, the central assumption could be affected.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "WmpkcDTV8Z", "reviewer": "Reviewer_9mLo", "summary": "Partial multi-label learning has appeared to be more useful in many applications. This paper first investigates some basic problems existed in the current research, and proposes a new method based on the mutual information, which appears to be interesting to the community.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- This paper first investigates two basic issues existed in the current research of partial multi-label learning and provides the solid technical solutions with theoretical guarantee.\n\n- A new method based on mutual information is proposed in this paper. The motivation and derivation are clear. The idea is interesting, which could motivate the community to bring some new tools from other domains to address partial multi-label learning tasks.\n\n- The extensive experiments are conducted on various data sets to verify the superior performance of the proposed method.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Some state-of-the-art partial multi-label references are missing, such as 1) Partial Multi-Label Learning with Label Distribution 2) Noisy label tolerance: A new perspective of Partial Multi-Label Learning 3) Partial multi-label learning with mutual teaching.\n\n- The explanation of Theorem 1 is weak; the author should provide more explanations.\n\n- Can the author do the experiments on the image data set?\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hours"}, "extra_text": {}}, {"review_id": "-U0i5vmb1s", "reviewer": "Reviewer_yXKv", "summary": "This paper presents two fundamental questions about the assumption of existing PML research. The authors derive a novel PML method from a probabilistic formulation which meets the $\\epsilon$-identifiable definition. They provide theoretical analysis and algorithms to verify the proposed two questions. Thorough comparative experiments validate the superiorities of the proposed method.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \nThis paper provides a new insight into partial multi-label learning problem from mutual information perspective, which is novel and highly related to the NeurIPS community. The presentation of the paper is clear and the relation with prior work is well-explained. A new objective function for PML is derived from a probabilistic formulation and a novel definition to evaluate whether a PML method can identify the ground-truth labels precisely is presented. The theoretical analysis is sound and solid. Thorough comparative experiments have been conducted to validate the superiorities of the proposed method, which has good academic value. \n\nCons:\n1.\tFrom Table 3 to 5, we can see that the proposed method does not always perform the best, such as, onYeastBP and Eurlex-ed datasets. How do the authors view this?\n2.\tSome details need elaboration. How to calculate the probabilities in Eq. (18) and Eq. (21)? Besides, it is pointed out that LIBLINEAR with L2-regularized square hinge loss is employed to train the binary classifiers. How is the L2-regularized square hinge loss incorporated into the optimization algorithm designed in section 5?\n3.\tIn Algorithm 2, the while loop of the condition should be explicitly explained.\n4.\tThere are some typos, such as “fundermental”, “ojective”. The authors should carefully proofread this paper and correct all the typos.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1.\tFrom Table 3 to 5, we can see that the proposed method does not always perform the best, such as, onYeastBP and Eurlex-ed datasets. How do the authors view this?\n2.\tHow to calculate the probabilities in Eq. (18) and Eq. (21)? \n3.\tHow is the L2-regularized square hinge loss incorporated into the optimization algorithm designed in section 5?\n4.\tIn Algorithm 2, the while loop of the condition should be explicitly explained.\n5.\tThere are some typos, such as “fundermental”, “ojective”. The authors should carefully proofread this paper and correct all the typos.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hour"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The author response clarified the main concerns regarding the paper, notably regarding the assumptions and the presentation. The reviewers agree that the paper makes a strong contribution to the PML setting, and that the experiments convincingly support the approach.
Decision: Accept (Poster)