Adversarial Paritial Multi-label Learning
Authors: Yan Yan
Yuhong Guo
Venue:  (2020)

Abstract: Partial multi-label learning (PML), which tackles the problem of learning multi-label prediction models from instances with overcomplete noisy annotations, has recently started gaining attention from the research community. In this paper, we propose a novel adversarial learning model, PML-GAN, under a generalized encoder-decoder framework for partial multi-label learning. The PML-GAN model uses a disambiguation network to identify noisy labels and uses a multi-label prediction network to map the training instances to the disambiguated label vectors, while deploying a generative adversarial network as an inverse mapping from label vectors to data samples in the input feature space. The learning of the overall model corresponds to a minimax adversarial game, which enhances the correspondence of input features with the output labels. Extensive experiments are conducted on multiple datasets, while the proposed model demonstrates the state-of-the-art performance for partial multi-label learning.

Under review as a conference paper at ICLR 2020
ADVERSARIAL PARTIAL MULTI-LABEL LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Partial multi-label learning (PML), which tackles the problem of learning multi- label prediction models from instances with overcomplete noisy annotations, has recently started gaining attention from the research community. In this paper, we propose novel ad adversarial learning model, PML-GAN, under a generalized encoder-decoder framework for partial multi-label learning. The PML-GAN model uses disambiguation network to identify noisy labels and uses multi- prediction network to map the training instances to the disambiguated label vectors, while deploying a generative adversarial network as an inverse mapping from label vectors to data samples in the input feature space. The learning of the overall model corresponds to minimax adversarial game, which enhances the correspondence of input features with the output labels. Extensive experiments are conducted multiple datasets, while the proposed model demonstrates the state-of-the-art performance for partial multi-label learning.
INTRODUCTION
In partial multi-label learning (PML), each training instance is assigned multiple candidate labels which are only partially relevant; that is, some assigned labels are irrelevant noise. As it is typically difficult and costly precisely annotate instances for multi-label data (Xie & Huang 2018), the lask of PML naturally arises in many real-world scenarios with crowdsource annotations. Figure T presents an example of training images for object recognition under the PML setting, where the union of candidate labels provided by crowdsource annotators is overcomplete and contains both ground truth labels (in black color) and irrelevant noise labels (in red color). PML much more challenging than standard multi-label learning the true labels are hidden among irrelevant labels and the number of true labels unknown. The goal of PMI. is to learn a good multi-label prediction model from such partial label training set, and hence reduce the annotation cost.
An intuitive strategy of PML is to treat all candidate labels as relevant ground truth, thus any off-the- shelf multi-label classification methods can be adapted to induce an expected multi-label predictor (Zhang& & Zhou 2014). This strategy, though simple, cannot work well since taking the noisy labels as part of the true labels will mislead the multi-labe training and induce inferior prediction models. The PML work in (Xie & Huang 2018) assumes that each candidate label has a confidence score of being true label, and learns the confidence scores and classifier in alternative manner by minimizing confidence weighted ranking loss between the candidate and non-candidate labels. Although this work yields some reasonable results, the estimation of label confidence scores is error-prone, especially when noisy labels dominate, which can seriously impair the classifier's 's performance. Another recent work in (Fang& & Zhang 2019) also exploits label confidence values of candidate labels for PML. estimates the confidence values using iterative label propagation and chooses the candidate labels with high confidence values as credible labels, which are then used to induce multi- label prediction model. This work however suffers from the cumulative errors induced in propagation, which can impact the estimation of the credible labels and consequently impair the prediction model.
In this paper, we propose a novel adversarial learning model, PML-GAN, under generalized encoder-decoder framework to tackle the partial multi-label learning problem. The PML-GAN model comprises four component networks: disambiguation network that predicts the probability of each candidate label being an additive noise for a training instance; a prediction network that predicts the disambiguated true labels of each instance from its input features; generation network that generates samples in the feature space given latent vectors in the label space; and a discrimination network that separates the generated samples from the real data. The prediction network and

---

Under review as a conference paper at ICLR 2020
Candidate
Labels:
X
D
building
G
Generation
river
Loss
tree
D
F(x)
Valletta
Classification Loss
F
Ioaland
An annotated im- age under the partial multi-label learning (PML) setting.
Figure 1:
Figure 2: The proposed PML-GAN model. It has four com--
ponent networks: generator G, disambiguator D, predictor F, and discriminator D.
disambiguation network together form encoder that maps data samples in the input feature space to the disambiguated label vectors, while the generation network and discrimination network forms a generative adversarial network (GAN) (Goodfellow al.] (014)) from vectors in the multi-labe space to samples in the imput feature space. The learning of the overall model corresponds a minimax adversarial game, which enhances the correspondence of input features with the output labels, and consequently boosts multi-label prediction performance. To the best of our knowledge, this is the first work that exploits GANs for PML. We conduct extensive experiments on multi-label datasets under partial multi-label learning setting. The empirical results show the proposed PML-GAN yields the state-of-the-art PML performance.
inverse decoding mapping
2 RELATED WORK
Multi-label learning is prevalent classification problem in many real world domains, where each instance can be assigned into multiple classes simultaneously. Many multi-label learning methods developed in the literature exploit label correlations different degrees to produce multi-label classifiers (Zhang & Zhou, 2014), including the first order methods (Zhang et al. 2018), second order methods (Li al. 20144, and high-order methods (Burkhardt & Kramer 2018 Nevertheless, standard multi-label learning methods all assume each training instance is annotated with complete of ground truth labels, which can be impractical in many domains, where the annotations are obtained through crowdsourcing. With the union of annotations produced by multiple noisy labelers under the crowdsourcing setting, the partial multi-label learning (PML.) problem arises naturally in real world scenarios, where the set of labels assigned to each training instance not only contain the ground truth labels, but also some additional irrelevant labels.
PML is more challenging than standard multi-label learning. The previous PML work in (Xie & Huang 2018) proposes two methods, PML-FP and PML-LC, to estimate the label confidence values and optimize the relevance ordering of labels each training instance by exploring the structural information in both feature and label spaces. However, due to the inherent property of alternative optimization, in these methods, the estimation error of labeling confidence values negatively impact the coupled multi-label predictor. The work in (Sun et al. 2019) denoises the observed label another recent work (Fang & Zhang 2019), the authors propose to address PML problem using a two-stage strategy. It first estimates the label confidence value of each candidate label with iterative label propagation, and then performs multi-label learning over selected credible labels based on the confidence values by using pairwise label ranking (PARTICLE-V VLS) or maximum posteriori reasoning (PARTICLE-MAP) However, their credible label estimation can be impaired by the cumulative error induced in the propagation process, which can consequently degrade the multi-label learning performance, especially when there many noisy labels.
matrix based on low-rank and sparse matrix decomposition.
Studies on weak learning and partial label learning have some connections with PML, but address different problems. Weak label learning tackles the problem of multi-label learning with incomplete labels (Sun et al. 2010 Wei et al. 2018). ) where some ground truth labels are missed out from the annotations. Partial label learning (PLL) tackles multi-class classification under the setting where for each training instance there is one ground-trie truth label among the given candidate label set (Cour
al. 2011 Liu & Dietterich 2012 Zhang & Yu 2015
Yu & Zhang 2016 Chen et al. 2018 PL.
2

---

Under review as a conference paper at ICLR 2020
methods cannot be directly applied on the more challenging PML problems, under PML one has unknown numbers of ground truth labels among the candidate label set for each training instance.
Generative adversarial networks (GANs) (Goodfellow al. 2014), which perform minimax adver- sarial training over a generation network and a discrimination network, are one of the most popular generative models since its introduction. During the past years, vast range of GAN--asse adversarial learning methods have been developed to address different tasks, including semi suni-suprviised learning (Kumar al. [2177] Lecouat et al. 2018), unsupervised learning (Jakab et al.) 2018), and learning with noisy labels (Thekumparampi et al. 2018). The proposed work in this paper however is the first one that exploits GAN models for PML.
PROPOSED APPROACH
In this section, we present the proposed adversarial partial multi-label learning model, PML-GAN, under the following setting. Assume we have a training set S = (X, Y) = yx), 2 where X; E [R denotes the input feature vector for the i-th instance, and y; E 10, 4 annotated label indicator vector. The multiple values in each y; indicate either the ground truth labels or the additional mis-annotated labels. We aim to learn a good multi-label prediction model from this partially labeled training set.
the corresponding
The proposed PMI--ANN model is illustrated in Figure Z which comprises four component networks, disambiguation network D, prediction network F, generation network G and discrimination network D. The four components coordinate with and enhance each other under an encoder-decoder learning framework, which forms inverse mappings between the instance vectors in the input feature space and the label vectors in the output class label space. Below we present these model components, the learning objective and training algorithm in details.
3.1 PREDICTION WITH DISAMBIGUATED LABELS
Comparing to standard multi-label learning, the main difficulty of PML is that the annotated labels {y:} in the training data contain additive noisy labels. The main challenge lies in identifying the ground truth labels z from each annotated candidate label vectors y;: that is dropping the additional Is from each candidate label vector y:. We propose to tackle this challenge by using disambiguation network D 22 - aa (S2. denotes the corresponding domain space), which predicts the irrelevant labels for a given instance. Hence the true label vector z can be recovered z ReLU(y: A.), where ReLU(-) = max 0) denotes the commonly used rectified linear unit activation function for deep neural networks. Then classifier, to predict the disambiguated ground truth labels for each instance.
learn prediction network F f2x Ω2z i.e., multi-label
Although the label indicator vectors in the training data are provided as discrete values, it is difficult for either the disambiguation network or the prediction network to directly produce discrete output values. Instead, by using a sigmoid activation function on the last layer of each network, D(x) and F(x) can predict the probability of each class label being the additive irrelevant label and the ground truth label respectively. With the disambiguation network and prediction network, we can perform partial multi-label learning by minimizing the classification loss on training data S:
min F.D
Le (x,
Σ (a(F(xx); 2:)
(1)
(x:Y1)~88
s.t. 2, Re-RReLUy;:AA
A = D(x)), Vix,,ss)~5 ~
where z; denotes the disambiguated label confidence vector with continuous values in [0, 1], which can be viewed as relaxation of a true label indicator vector, while (. denotes the cross-entropy loss between the predicted probability of each label and its confidence of being ground-truth label.
3.2 INVERSE MAPPING WITH GANS
The prediction network and disambiguation network together form encoder that maps data samples in the input feature space to the disambiguated label vectors. To enhance the label disambiguation and hence improve multi-label classification, we propose to conduct inverse decoding mapping
3

---

Under review as a conference paper at ICLR 2020
from label vectors å E {0, deploy a generative adversarial network (GAN) model to generate samples in the imput feature space given label vectors in the label space. The GAN model comprises generation network G and discrimination network D. Given a label vector 2 sampled from a prior binomial distribution P(2), can generate sample x using the generation network, is used to discriminate the generated samples from the real samples in S. The training of the GAN model is a minimax optimization problem over an adversarial loss function:
1 to samples in the input feature space. In particular, we propose to
( A two-class discriminator D
min max D
Ladv (G, D) Ex [(x)) + Egg ((G(
where the discriminator D tries to maximally distinguish the generated samples G( from the real data samples in S, and the generator G' tries to generate samples that are similar the real data as much as possible such that the discriminator cannot tell the difference.
In theory, the samples generated by the adversarially trained generator G have identical distribution with the real data S (Goodfellow et al. (014).. But this does not guarantee the generated samples can match the real training samples. To ensure the generator G can provide an inverse mapping function relative to the predictor F, we further propose to decode the disambiguated training label vectors into the training samples S with G by deploying generation loss:
(g(
Σ (g(G(z,),,;)
where ReLUUyy -D(0))
(x..)))~8
where (.(
measures the generation loss each training instance and can be least squares function. This generation loss can enhance the label disambiguation and improve multi-label learning.
3.3 LEARNING WITH PML-GANS
By integrating the classification loss in Eq. the adversarial loss in Eq. and the generation loss in 33 together, we obtain the following minimax optimization problem for the proposed PML-GAN model:
min max E(xiyows (lo(P(x)), ( G.D.F D
(G((z)) +
3 ([xxss(lg + E P(y)ll)) -D(G(2)))
(4)
s.t. = ReLUUy; D(x:)), V(x,,y:) S
where B is trade-off hyperparameter that controls the relative importance of the generation loss and adversarial loss respectively: the objective function can be denoted as L(G. D, F, D). The learning of the overall model corresponds to a minimax adversarial game, which enhances the correspondence of input features with the output labels in inverse encoder-decoder manner, and consequently boosts multi-label prediction performance.
We perform training using minibatch based stochastic gradient descent algorithm. In each iteration of the training, the minimization over G, D, and the maximization over D are conducted alternatively. The overall training algorithm is presented in Algorithm |
4 THEORETICAL RESULTS
In the proposed PMI-GAN model, given the gen- crator G, the discriminator D is conditionally inde- pendent from the predictor and disambiguator D.
F
Between G, F and D, G' and F' are conditionally independent from each other given D. Their indepen- dence relationship can be illustrated using the undi- rected dependence graph in Figure Based these conditional independence relationships, we have the following optimality results.
D
Figure 3: Dependence graph of PML-GAN.
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Minibatch stochastic gradient descent training of PML-GAN
Input: training set St trade-off parameter 9; k- of update steps for the discriminator. for number training iterations do
Sample minibatch of samples {(X1, yr), ... {m)) from training set S. Sample n lahel vectors (21.- ... 2n from a prior P(2).
Update the network parameters of G, D, F by descending with their stochastic gradients:
E Afoo(1 - ((G(2)))) + L ReL ReLU D(x:))) + (G - D(x:))), S
Vec.D.F
for 1::: do
Sample label vectors (21,... ...... 2n Update the parameters the discrimination network by ascending with its stochastic gradient:
from a prior P(2).
Σ [xx)( + 1oge(1 D(G())))
Ven 8
end for end for
Proposition 1. For any G, D, and F, the optimal discriminator D is given by
D ( Di (x) pPsx)/ (ps(x)((ss(x))) + Pg
(5)
where ps()) and P.(.) denote the probability distributions of real and generated data respectively.
Proof. Due to the conditional independence relationship between D and {F, Đ}, the optimal dis- criminator only depends on the generator Given fixed G, the optimal discriminator can be derived in the same way as in the standard GAN model (Goodfellow al. 2014 Proposition 1), O
Proposition 2. Assume the model has sufficient capacity. Let C(G, D, F) = max L(G. D, Given fixed D, the minimum of C D, P') is lower bounded by E. H (ReLU(y,-D(xx)) 33 log 4, which can be achieved when F (xx) = Rel.U(y - (xx))), G(F(x:)) X, and Pg = ps-
Here (-) denotes an entropy function. Proof is provided in the appendix. This proposition suggests that F and G should be inverse mapping functions for each other in the ideal optimal case.
EXPERIMENTS
5.1 EXPERIMENTAL SETTING
Datasets. We conducted experiments on twelve multi-label classification datasets. Three of them have existing partial multi-label learning settings (mirflickr, music_styll and susic_emotion (Fang & Zhang 2019)). For each of the other nine datasets (Zhang & Zhou| 2014). we transformed into PML dataset by randomly adding irrelevant labels into the candidate label set of each training instance. By adding different numbers of irrelevant labels, for each dataset we can create multiple
PML variants with different average numbers of candidate labels. Following the setting of (Xie & Huang 2018), we also filtered out the rare labels and kept at most 15 classes in each dataset. The detailed characteristics of the processed datasets are given in Table
Comparison Methods. We compared our proposed method with four state-of-the-an PML methods and one baseline multi-label learning method. We adopted a simple but effective neural network based multi-label learning method, ML-RBF Zhang (2009), derived from the radial basis function (RBF), as a baseline method, which performs PML by treating all the candidate labels as ground -tuth labels. Then we used four recently developed PML methods for comparison, including the PML-LC and PML-FP methods from (Xie & Huang 2018) and the PARTICLE- VLS and PARTICLE-MAP methods from (Fang & Zhang 2019).
Implementation. The proposed PML-GAN model has four component networks, all of which designed multilayer perceptrons with Leaky ReLu activation for the middle layers. The
5

---

Under review as a conference paper at ICLR 2020
Table Information of the experimental data sets. The number of instances, features and classes are recorded. The Lavg.#CLs column lists the average number of candidate labels in each PML set.
Dataset
#inst.
#Fcats #Classes
avg.#CLs
Dataset
#inst.
#Fcctt #C CClasses
#CL
music_ cestiie_eotionn
6833
98
11
5.29
music_style
6839
98
10
6.04
mirllickr
10433
100
7
3.35
image
2000
294
5
2,3,4
scene
2407
294
3,4,5
yeast
2417
103
14
9,10,11,12
enron
1702
1001
15
8.9,10.11,12.13
corel5k
5000
499
15
8.9,10.11,12.13
eurlex_de dc
8636
100
15
8.9.10,11,12,13
eurlex sm
12679
100
15
8,9.10,11,12.13
delicious
14000
500
15
8,9.10.11.12.13
tmc2007
28596
49060
15
8.9.10.11,12.13
disambiguator, predictor, and discriminator are all three-layer networks with sigmoid activation in the output layer, while the generator is five layer network with Tanh activation in the output layer. Batch normalization also deployed in the middle three layers of the generation network. We used the Adam (Kingma & Bal (2014) optimizer in our implementation. The mini-batch size, m, is set to 64. The hyperparameters (the number of steps for discriminator update) and (the number of label vectors sampled) in Algorithm| [ are set to and 21th respectively. The hyperparameter is chosen from {0.001, 0.01, 0.1, 1, 10 based on the classification loss value Le in the training objective function; that is, the 3 value that leads to the smallest training Le loss will be chosen.
5.2 COMPARISON RESULTS
We compared the proposed PML-GAN method with the five comparison methods on the twelve datasets. For each dataset, we randomly select 80% of the data for training and use the remaining 20% for testing. We repeat each experiment 10 times with different random partitions of the datasets. The comparison test results in terms of four commonly used evaluation metrics (Hamming loss, ranking loss, average precision, and macro-averaging AUC) (Zhang & Zhou 2014) are reported in Table 2 The results are the means and standard deviations over the 10 repeated runs. We can see that the methods specially developed for PML problems all outperform the baseline multi-label neural network classifier, ML-RBF, in most cases. But it is difficult to beat the baseline competitor on all the datasets with different evaluation metrics, Among the total 48 cases over 12 datasets and 4 evaluation metrics, PARTICLE-VIS, PARTICLE-MAP, PML-LC and PML-FP outperform ML-RBF in 44, 48, 37 and 45 cases respectively. By contrast, the proposed PMI-GAN method outperforms ML-RBF consistently across all the 48 cases with remarkably performance gains. Even comparing with all the other four PML methods, PML-GAN produced the best results 43 out of the total 48 cases. Moreover, the performance gains yield by PML-GAN over all the other methods are quite notable in many cases. For example, in terms of average precision. PML-GAN outperforms the best alternative comparison method by 4.6%, 4.1% and 3.3% eurlex_d _c, scene and image respectively. These results clearly demonstrate the effectiveness of the proposed PMI-GAN model.
The results reported above are produced each dataset with selected average number of candidate labels. As shown in Table [ we have multiple PML variants with different numbers of candidate labels for each of the nine datasets in the list (except -esoiion_ In total this provides us 49 PML datasets. We hence also conducted experiments on each of these 49 variant datasets, by comparing the proposed PML-GAN with each of the other methods in terms of the 4 evaluation metrics. In total there are 196 comparison cases for each pair of methods. For the comparison of "PMI-GAN vs other method" in each case, we conducted pairwise t-test at significance level of 0.05. The win/tie/loss counts in all cases are reported in Table We can see that overall the proposed PML-GAN significantly outperforms PARTICLE- VLS, PARTICLE-MAP, PML-LC, PML-FP, and ML-RBF in 77% 76%, 78.5%, 81.6%, and 90.3% of the cases respectively. This again validates the efficacy of the proposed method.
and mirflickr).
Impact of Irrelevant Labels To demonstrate how would the number of irrelevant labels affect the performance of PML. methods, we plotted the experimental results on the delicious dataset with different average numbers candidate labels in Figure We can see with the increase of the number of irrelevant labels, consequently the average number of candidate labels, the performance of each method in general degrades. Nevertheless, the proposed PML-GAN consistently outperforms all the other methods. Moreover, in terms of Hamming loss, the performance of PML-GAN actually is quite stable with the increase of the noisy labels. This validates the effectiveness of PML-GAN in irrelevant noisy label disambiguation.

---

Under review as a conference paper at ICLR 2020
Table 2: Comparison results of in terms of Hamming loss, ranking loss, average precision, and
micro-averaging AUC. The best results are presented in bold font. The average number of candidate labels is presented under the column "avg.#C.Ls"
Data set
avg. .C.Ls
PMI.-GAN
PARTICLE- VLS
PARTICLE-MA
PML-LC
PML-FFP
ML.-BFF
Hamming loss (the smaller, the better)
music_ eenotiin
5.29
.1971.004
.2122.00
2151.004
2361.003
.2511.004
.7791.004
music_sp style
6.04
.116+.005
.121+ 003
.175+-005
112++004
.126+.004
.856+0001
mirllickr
3.35
.171:1.02
.178.1033
1891..81
202.1.057
.0021.057
.748
image
3
.202-+008
.2342.065
269+.096
264+.072
.2674.063
.754+.003
.137+-007
.184+0037
.174+.035
.178+0029
.187+.038
.80++0001
yeast
.2162.005
2262.004
2202.008
2264.008
2194:0099
694
enron
.186=.004
.17=
.190+0036
206+.027
.206+.027
.13+.004
corel5k
.119++001
189+.012
269+-027
151+-008
.152+-008
.88++0001
eurlex_ds
10
.044+.001
061++0011
0644-004
096+.000
.0714.001
9334-001
curlex sm
0822+.001
.067-.001
076+.002
.199.0006
.1224.002
8854.001
delicious
-248=0003
.200 003
290+:005
290+:004
290+:0004
.712+0002
tmc2007
085+-001
.090 .003
.110+003
103+-002
.00++0002
.857+0001
Ranking loss (the smaller, the better)
mmusic_emotien
5.29
.2431.000 009
.2401.007
2671.009
.75110000
3651.010
music_sto style
6.04
.1412.001
16632.007
.154..05
215+.005
.1501.005
2424..006
mirtlickr
3.35
.1272.014
.2272.029
.129=.108
.60++.029
.1434.028
.195+.015
imaage
.1921.015
.2311.077 007
2911..134
.2171.120
22111.019
.1111.014
17771.049
1674.060
1924.032
2381.056
1881.014
yeast
.193+.008
203+ .07
201
219+.011
203
270+
enron
.1791.013
240.1.088
182.1.099
2391..08
23911.047
244.1.010
corel5k
.293+.012
367=+0332
311++008
366+.035
.38++.025
404+.082
eurlex
.065+.003
.150+0004
085+-004
.337+008
.1314-001
.135+.003
10
eurlex_sm
.1191.005
.1291.007
.1771.009
282.1.007
.182...08
.1831.003
delicious
.2561.006
3141.005
27611.004
2771.005
.2661.005
31611.033
tmc2007
#711.003
.96:2.008
.055.0007
08224.05
.8804.005
.1331.002
Average precision (the larger, the betler)
music cmotion
5.29
621=.012
603=.012
.124.009
5744.013
5684.014
.00+.0012
music music_style style
6.04
773++0015
.715+-009
.709+0009
.702+0088
703+-008
.646+.010
mirflickr
3.35
.7712.026
.6782.027
.7914.202
.7362.043
.7884.039
.6764.048
imtrage
3741.003
J411
7291.086
7251.119
7.331.02
4
.7941.014
75011.074
7531.064
6891.047
.7101.079
.7281.015
yeast
7308
714+-010
714+-010
721+-012
.78++0010
634+-008
enron
1.00
.95
6611.047
55611.04
.5551.041
$60.1.099
core15k
.4011.014
.77
4151.008
3451.027
38441021
3341.008
eurlex_
.7971.009
.6922.013
.7514.088
6934.019
.7161.014
.7104.000
10
eurlex
.722=.002
7055.009
.883.011
4384.016
.679+.011
.556+.000
delicious
630==0077
.907
.01+-008
607+.007
608++0006
576+.004
tmc2007
8201.004
7994.013
.7591.013
.7931.012
.7941.012
6624.003
Macro-averaging AUC (the larger, the better)
music cmotion
5.29
776=2.066
.73:2.008
.764.004
6324.008
636+.007
504+.003
music style
6.04
.706+0016
.694+0007
.715+.005
754+-006
.755+.006
503+.003
mirflickr
3.35
.873+0002
806+0000
816+-057
842+.030
840+.031
808+0000
image
3
.7982.015
7352.119
75742.103
.2742.111
.7844.130
.7094.000
seene
4
874#.012
789++005
814+.049
.34+.041
.735+.038
.730+.010
yeast
.6661.026
.1002.017
6501.014
6311.019
.6361.020
6001.012
enron
.6881.088
60772.024
660+.031
.311.0299
.6314.029
5964.016
corelSk
.628=.007
5422-039
618+.031
550+.033
.5514.034
5274.007
eurlex_
.8721.009
8671.011
8354.008
8404.005
8534.007
8254.006
10
eurlex
887 008
88432.008
8234.008
.799.0077
.7801.004
.7741.003
delicious
7024
.666-.000
634+-006
688+.006
.68++.006
.332-002
tmc2007
-802
836+-010
830+-009
861+-008
.89+-008
.765+.003
5.3 ABLATION STUDY
As shown in Eq. the objective of PML-GAN contains three parts: classification loss, generation
loss and adversarial loss. The generation loss and adversarial loss are integrated to assist the predictor training. To investigate and validate the contribution of the generation loss and adversarial loss, we conducted an ablation study by comparing PML-GAN with three of its ablation variants: (1)
CLS-GEN, which drops the adversarial loss; (2) CLS-GAN, which drops the generation loss; and (3) CLS-ML, which only uses the classification loss by dropping both the adversarial loss and generation loss. The comparison results reported in Table 4 We
that comparing the full model, all three variants produced inferior results in general. Among the three variants, both CLS- GEN and CLS-GAN outperform CL.S-M. in most cases. This suggests that both generation loss and adversarial
7

---

Under review as a conference paper at ICLR 2020
PML-GAN
PARTICLE-VLS
PARTICLE-MAN
PML-LC
PML-FP
0.5r
0.35
0.70
0.75
0.65
0.60
0.3
0.25
0.65
0.55
9
10
11
12
8
10
11
12
13
0.50
8
10
11 12
13
8
9
10
12
(a) Hamming Loss
(b) Ranking Loss
(c) Average Precision
(d) Macro-AUC
Figure 4: Test results with different average numbers of candidate labels on the delicious dataset.
Table 3: Win/tie/loss counts of pairwise t-test (with p < 0.05 between PML-GAN and each
comparison method over all dataset variants with different numbers of candidate labels.
PML-GAN vs
Evaluation Metric
PARTICLE- VLS
PARTICLE- MAP
PML-LC
PML-FP
ML-RBF
Hamming loss
39/6/4
38/9/2
40/7/2
40/3/6
45/4/0
Ranking loss
38/9/2
38/8/3
38/8/3
40/5/4
44/3/2
Average precision
37/8/4
36/10/3
40/6/3
42/4/3
46/3/0
Macro-averaging AUC
37/7/5
37/8/4
36/7/6
38/5/6
42/7/0
Total
151/30/15
149/357T2
154/28/14
160/17/19
177/17/2
Table 4: Comparison results of PML-GAN and its three ablation variants.
Data set
PMI-GAN CLS-GEN
CLS-GAN
CLS-ML.
PMI-GAN
CLS-GEN
CLS-GAN CLS-ML.
Hamming loss (the smaller, the better)
Ranking loss (the smaller, better)
music emotion
.1971.004
.2001.003
.1994.005
20224.003
.2431.009
.2472.008
.2441.007
2481.007
music style
.116::0005
.1174.003
.118+.003
.120+4003
.1412.001
.146:.004
.1434.007
.1644.007
mirflickr
.171::002
.174:1.05
.1724.005
.176:1.004
.1271.014
.132:.022
.128:.020
.1355.015
image
.202:4.008
2205..006
+208
.330+.008
.192+.015
.196+.013
.196:.013
.206+.014
.1371.007
.1401:013
.1431.009
.1524.007
.131:.014
.1362.009
.1311.007
.1371.006
yeast
.216:.0005
21191.004
.2174.006
.2881.005
.1931.008
.1981.010
.196:1.010
.205..012
.186::.004
2811:.014
.273+.018
.811:0012
.179:.013
.1844.013
.1844.008
.1894.010
core15k
.119:.001
.119
.121+.002
.1224.002
.293+.012
.302++0099
.302+.017
308
curlex de
.0441.001
.052+1003
.0461.001
.0541.001
.0651.003
.085#.012
.0681.003
0071
eurlex_sm
.082+.001
0083+.001
.084+-001
.8864.001
.119+.005
.120+.003
.123+.005
.125+.004
delicious
228::.003
2211+001
.2444.002
.2554.002
256.1.006
.2594.006
.2571.004
.323+.006
tmc2007
.88:1.001
.088:1001
.8851.001
091110011
.0711.003
0744.002
.072:.002
0755
Average precision (the larger, the better)
Macro-averaging AUC
(the larger, the better)
music emotion
.6211.012
.68.1.012
.2001.010
6141.010
.72611006
.7241.007
.72411009
.6154.010
music music_style style
.734::015
.726:.008
.7304.010
7004+0022
.766:.016
.700+.011
.703:1.00
6911t.014
mirflickr
.771:::026
4763
.7694.038
.330+.010
.873+.002
.872+.003
.8724.002
860+.003
image
.7742.013
.7711.015
.7661.017
.784..018
.7981.015
.7942.009
.7941.012
.7902.000
scene
.794:.014
.7944.013
.7894.011
.7834.009
.844.012
87244006
.8344.009
8524.010
yeast
.733:.008
.2284.014
.7314.015
.7084.012
.6661.026
.288.0015
.6272.011
6254.011
.665+.002
.6634.025
.6574.014
.6554.023
.6681.018
566+.0022
.5734.013
561
corelSk
.4401.014
.3441.011
.3611.015
.3324.008
.6281.007
.5794.004
.5791.006
.70110006
eurlex_de de
7797+.009
7814.019
.7974.008
.779+.011
.872+.009
8399+.016
.8724.009
822+.007
curlex sm
.722+1.022
.719+.003
.714:1.009
.7124.004
$277..088
.24:1.005
.8254.008
820+.005
delicious
.630+.007
628-+.004
.630+1006
.628+.005
.712+.004
.7102.003
.7114:002
.709+.002
tmc2007
.820
818++0044
.820+-004
815+.0003
.886 0002
.84+-003
8886+.002
802
loss are critical terms for the proposed model. Moreover, even the baseline variant CL.S-ML. still produces some reasonable PML results. This suggests the integration of our proposed prediction network and disambiguation network is also effective.
6 CONCLUSION
In this paper, we proposed a novel adversarial model for partial multi-label learning. The proposed model comprises four component networks, which form an encoder-decoder framework to improve noisy label disambiguation and boost multi-label learning performance. The training problem forms a minimax adversarial optimization, which solved using an alternative min-max procedure with minibatch-based stochastic gradient descent. We conducted extensive experiments on multiple PML datasets. The results show that the proposed model outperforms all the comparison methods and achieves the of-the-ar PML performance.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
S. Burkhardt and S. Kramer. Online multi-label dependency topic models for text classification. Machine Learning, 107(5):859-888) 2018.
C.-H. Chen, V. Patel, and R. Chellappa. Learning from ambiguously labeled face images. IEEE transactions on pattern analysis and machine intelligence, 40(7):1653-166) 2018.
Cour, B. Sapp. and Taskar. Learning from partial labels. Journal of Machine Learning Research, 120May): 2011.
J.PP. Fang and M.-L., Zhang. Partial multi-label learning via credible label elicitation. In AAAI Conference on Artificial Intelligence (AAAI), 2019.
I. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley. S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems (NeurIPS), 2014.
Jakab, Gupta, H. Bilen, and Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Diederik P. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1e/11/42.6,0... 2014.
A. Kumar, P. Sattigeri, and P. T. Fletcher. Semi-supervised learning with GANs: Manifold invariance with improved inference. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
B. Lecouat, C.-S. Foo, H. Zenati, and Vijay R. Chandrasekhar. supervise learning with GANs: Revisiting manifold regularization. arXiv preprint urXiv:1805.08957 2018.
X. Li, F.-P. Zhao, and Y.-H. Guo. Multi-label image classification with a probabilistic label enhance- ment model. In the Conference Uncertainty in Artificial Intelligence (UAI), 2014.
L.P.. Liu and T. Dietterich. A conditional multinomial mixture model for superset label learning. In Advances in neural information processing systems (NeurIPS), 2012.
L.-J. Sun, S.-H. Feng. Wang, C.-Y. Lang, and Y. Jin. Partial multi-label learning by low-rank and sparse decomposition. In AAAI Conference on Artificial Intelligence (AAAI), 2019.
Y.-Y. Sun, Y. Zhang, and 7.-H. Zhou. Multi-label learning with weak label. In AAAI Conference Artificial Intelligence (AAAI), 2010.
K. Thekumparampil, Khetan, Z. Lin, and S. Oh. Robustness of conditional GANs to noisy labels. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
T. Wei, L.-Z. Guo, Y-F. Li, and W. Gao. Learning safe multi-label prediction for weakly labeled data. Machine Learning, 107(4):03-725, 2018.
M.-K. Xie and S.-J. Huang. Partial multi-label learning. In AAAI Conference Artificial Intelligence (AAAI), 2018.
and M.-L. Zhang. Maximum margin partial label learning. In Asian Conference on Machine Learning, 2016.
M.-L. Zhang. ML-RBF: rbf neural networks for multi-labe learning, Neural Processing Letters, 29 (2):61-74, 2009.
M.-L. Zhang and Yu. Solving the partial label learning problem: instance-based approach. In International Join! Conference on Artificial Intelligence (IJCAI), 2015.
M.-L. Zhang and Z.HH. Zhou. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering. 26(8):1819 1837, 2014.
M.-L. Zhang, Y.-K Li, X.-Y. Liu, and X. Geng. Binary relevance for multi-label learning: an overview. Frontiers of Computer Science, 1912):191-202, 2018.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
In this appendix, we provide proof for Proposition 2, details of the network structure for the PML- GAN model, and additional experimental results. For additional experimental results, first reported results with two additional evaluation metrics, and results of parameter sensitivity analysis. Then we empirically compared PML-GAN with its variant that considers additional classification loss on the generated data. Finally we conducted extended experiments on few datasets where we constructed PML data by adding large number of label noise.
A. PROOF OF PROPOSITION 2
Proof. As shown in section 3.3, [(G D, D) denotes the objective function PML-GAN. Based on the solution for optimal discriminator D in Proposition I, we have:
C(G,D,F) D,F) max [(G, D,F,D,F,D)
E(x,y))~s ReLL(y D(x:))) (CC((RLL( D(xx))), +
[xxps (x) + Ez- ( ((((())))))
E(x.y))~S lo (x()(x)), ReLU(y: - D(x:))) + (G(ReLU(y; ((xx))). +
Exxps (1 (x)] + Ex-pp [log(1 - Di (x)))
E(xexi)
(x(F(xx)), ReLU D(xx))) Eo (G((eLUy D(x.))), +
(xxxpslogg (xx
ps(x)
+ Ex-p, [log Px(x)
pg (x))
Note given fixed D, F is conditionally independent from G and D. Hence the minimization of D,G,D,F) over F can be independently conducted from the minimization over G.
Let z; ReLU(y, D(x:)), With the cross-entropy loss function ( we have:
min D, min ((x,))) Le ReLU(y, - -((x)))
= min E -2) log F(x:) - (1 1z.)) log(1 F(x)))
= min Exx H(z:) KL(Z; F(x:))
Ex) H(z))
where H()) denotes the entropy over binomial distriubtion vector and KL(-) denotes the KL- divergence between two sets of binomial distributions. Assume sufficient capacity for F, the minimum can be reached when the predictor obtains the same distributions as the 2, that is
F F"xx)) ReLU(y, D(x.)), V(x..)) E
Next let's consider the minimization problem over G. Note G is involved in both the generation loss and adversarial loss. If we could find solutions that lead to minimals both losses separately, we can guarantee minimal in the united loss. The adversarial loss part in C(G. D, F) can be rewritten as
(x) ps (x) + Pe (x) Ex-pp [log
Ps (x)
BLads 8 (Ex-ps [log
ps (x Py (x)
3 kps,
ps Ps log 2 + KL(Pg,
ps + Pa log 2
2
2
= (ps,
+ Pg ) + kT,
ps + P2j - log 4
2
2
2 -9 log
10

---

Under review as a conference paper at ICLR 2020
Table 5: The network architecture of PMI-GAN. BN: Batch normalization; LReLU: Leaky rectified
unit; Act.; Activation function; dim: Feature dimension of training samples x; class num: the number of class labels.
Generator
Discriminator
Input 2
Output
BN
Act.
Input data
Output
Act.
512
LRcLU
512 256
LReLU LReLU
512
1024
LReLU
512
1024
256
LReLU
256
Sigmoid
256
128
LRcLU
128
dim
Tanh
Disambiguator D
Predictor F
512
LReLU
512
LReLU I.ReIUU
512
256
LReLU
$12
256 class num
256
class num
Sigmoid
256
Sigmoid
where the minimal can be achieved when ps = Py which leads to zeros as the KL-divergence values. The generation loss part (with least squares loss function) C(G, D, F) can be rewritten as -x (far)))s((((((eea)ee (C((eLL) D(x:)))
(x [| [(x,)) - x;
20
where the minimal 0 can only be achieved when
G(ReLU(y: D(x))) = xx V(x,y:) S
It is obvious the optimal condition above can be satisfied simultaneously together with the condition Pg = Ps. Hence the proposition proved.
A.2 NETWORK STRUCTURE THE PML-GAN MODEL
The proposed PMI--GN model has four component networks, all of them are designed multilayer perceptrons with LeakyReLu activation function for the middle layers. The disambiguator, predictor, and discriminator are all three-layer networks with sigmoid activation in the output layer, while the generator is a five layer network with Tanh activation in the output layer. Batch normalization is also deployed in the middle three layers of the generation network. The detailed input and output setup information for each layer of the networks is given in Table
A.3 ADDITIONAL EXPERIMENTAL RESULTS IN MORE EVALUATION METRICS
For the comparison results, evaluated the test performance using six commonly used metries from Zhang & Zhou (2014). Due to space limitation, we previously reported the results in terms of four metrics. The results terms of the remaining two metrics, one error and coverage, reported in Table 6 We can see that our proposed PML-GAN again produced the best results in majority cases.
In Table 7 we reported the ablation study results in terms of coverage and one error. The relative comparison performance is quite similar to the results reported in the paper. It suggests the component networks in our model are necessary and useful.
A.4 PARAMETER SENSITIVITY ANALYSIS
We also conducted experiments on the eurlex de data set to study the impact of the trade-off hyperparameter 3 on the performance of PMI-GAN, We repeated the experiments in the same setting as before with different 3 values from the range {0.001, 0.01, 0.1, 1, 10}. Note larger & provides
larger weight to the adversarial loss.
Figure 5 reports the average test results over 10 runs for different 3 values. We can when 3 is very small, the performance poor. With the increase of 3, the performance improves. This suggests the adversarial loss is important. When 3 0.1 the best result is obtained. When 3 is too large, performance degrades as the adversarial loss dominates. This makes sense since the adversarial loss expected to help the classification model, rather than dominating the learning.
11

---

Under review as a conference paper at ICLR 2020
Table 6: Comparison results for PMI-GAN and the other methods in terms of two additional metrics: coverage and one eπror. The best results are in bold font.
avg.#C..ss
PMI.-GAN
PARTICLE- VLS
PARTICLE-MAT
PML-LC
PML-FP
ML-RBF
Coverage smaller, the hetter)
music emotion
5.29
.072.008
4132.007
.362.0007
4282.010
4341.010
5061.000
music style
6.04
.202-.011
228+.000
.05+-0077
203+.008
203+.008
305+0006
mirflickr
3.35
227+.010
270+-079
.27+-0577
234+-073
.233+0074
.288+0002
image
3
2112.0014
2272.079
2632.085
3092.109
2981.095
2299:.015
secne
.144.010
.14420054
.10++0050
.180+.033
.2144.045
.1674.012
ycast
5022.0144
506+.000
528+.003
5244.016
.95+.006
583+.007
.402+0015
116+
.65+-075
496+.089
.96+
458+.017
curelSk
.384:.011
122
3931.017
4971.042
189.1.035
49211.009
eurlex_de
.6311.00
136.1.003
082.1.04
3181.017
162.1.01
.1171.003
eurlex_sm sm
.1222.006
.1922.009
1832.012
4031.009
4081.009
2431.000
delicious
5644.008
6034.004
5844.004
591+.005
.50+H.005
.610+.004
tmc2007
1181=.006
2055.0088
.114:011
.16+.007
.194
.283+0003
One smaller, the better)
music emotion
5.29
.44 +.27
473+.016
475+.018
556+.028
$401.027
587+.019
music style
6.04
.366=.022
.3442005
3999.019
409+.013
4084.013
385+.006
mirflickr
3.35
389
.655-.150
229++3306
300+.129
228
.33++0002
image
3
.3001.020
.391.114
3872.147
5421.191
.4911.174
3981.034
seene
.329=.020
.4002008
349+.082
4974.089
.234..188
428+.022
yeast
.2011.019
.2481.019
2521.018
2571.017
2631.027
408.1.023
.3071.039
A112.101
3311
4941.039
4981.038
4951.019
core15k
.697=.020
835=.025
.7212.035
.744.009
.7872.024
8094.015
curlex
.305=.016
390=-006
3344.014
7074.014
.188.011
.32+.008
10
eurlex sm
.341+.003
.50+-014
360+0015
506+.031
5424.018
.30++0005
delicious
.3722.012
.6622.115
A144.018
4014.015
.3994.013
4504.009
tmc2007
.200= 005
.194#.029
.26720018
235+.019
.236+.019
388+.006
Table 7: Comparison of PML-GAN with its ablation variants in terms of coverage and error. The best results are in bold font.
PML-GAN
CLS-GEN
CLS-GAN
CLS-ML
PML-GAN
CLS-GEN
CLS-GAN
CLS-ML
Coverage (the smaller, better)
smaller, the better)
music emotion
.4071.008
.1011008
4081.006
.101.005
4442.027
460=002
.4461.000
4631.024
_tyle
.202:-011
.2044.004
204.1.008
207.1008
.359=001
.3114.017
4.01
mirllickr
.2271.010
.3011.017
22771.014
2424.013
.3372.059
.388-092
.3704.085
4054.094
image
.211::014
2144.011
2124.011
2201.011
.340-020
.344=023
.3001.032
3681.022
.124:.010
.1274.008
.125:.006
.128#.010
.229-020
.331=023
.3411.022
348:.0188
5002:.014
.09440066
5074.013
5094:023
2500
227
.211:.029
254+.024
enrnn
.002+.015
.10+-0199
408+
A14+-015
.39+-039
029
.31++027
333+.028
corel5k
.384::011
3944.008
.3964.017
3992:011
.697-.020
.699-017
.6984.020
.7012.016
eurlex_de
.063+-0003
.082+.011
.063+.003
.085+.005
.05=-0166
312=0024
.307+.013
312+.017
eurlex_sm
.172+-006
.174+004
.176+0005
.178+0004
341+-003
343 347-
350+-015
351+.007
delicious
5641.008
566+.008
.5441.006
.688+.005
.322-012
373=010
.3334.008
375+.011
tmc2007
1811-006
.184+.004
.181+:.04
.185+=004
.200=0005
-211-0005
.203+0007
.205+0004
A.5 VARIANT OF PML-GAN WITH CLASSIFICATION LOss ON GENERATED DATA
The classification loss in the PML-GAN model of Eq. takes the real data into account, as that is the primary concern of PML. natural extension is to consider an auxiliary classification loss on the generated data as well. This extension leads to the following variant of PML-GAN, which we denote as PML-GAN':
min max Exx.. (a(((( ((x.)))(((((((x).)). (G(()), + (a G.D.F D
(Bg B log(1 DDGG()))
(6)
s.t. 2, = ReLU(y, D(x:), ((x.y:)~55
where and 3 are trade-off hrude-offhyperreeee hyperparameters. For the parameter selection of PML-GAN', both and are chosen from (0.001, 0.01, 0.1, 1, 10) based on the classification loss value L, on the real data in the training objective function.
12

---

Under review as a conference paper at ICLR 2020
0.050
0.075
0.82
0.88
0.070
0.80
0.86
0.065
0.70
0.84
0.040
0.060
10'
10
0.76
10
0.02
1000
tow
10
(a) Hamming Loss
(b) Ranking Loss
(c) Average Precision
(d) Macro-AUC
Figure 5: Test performance with different & values the eurlex_de data set.
The comparison results between PML-GAN and PML-GAN are reported in Table We can see that PML-GAN' has similar performance with PML-GAN, and the difference between their results are very small in most cases. To statistically compare PML-GAN with PML-GAN'. we conducted pairwise t-test at significance level of 0.05 over all the 49 PML variant datasets in terms of the evaluation metrics. The win/tie/loss counts in all 196 cases are reported in Table 9 can see that overall the PML-GAN' ties with PMI-GAN in 99.5% cases. This suggests that an addition auxiliary classification loss on the generated data cannot improve PML-GAN, which makes sense given the goal is to learn good classification model on the sufficient real data. Meanwhile, the additional hyperparameter a of PML-GAN' induces much higher computational cost due to additional parameter selection. Overall, PML-GAN is more suitable choice than PML-GAN'.
Table 8: Comparison results for PML-GAN and PML-GAN' in terms of six evaluation metrics. The best results are presented bold font. The average number candidate labels presented under column "avg.#C.Ls"
Data set
av..#C.LLs
PML-GAN
PML-GAN
PML-GAN Ranking loss (the small the better)
PML-GAN
Hamming loss (the smaller, the hetler)
music_ music_ermotion _emotion
5.29
.197:1.04
.20011.033
243:1.009 0.0
.239 2,09
muyle_style
.1161..05
.1151.002
.141..001
1.061.006
mirflickr
3.35
.171+-002
.171+-004
.127+0014
.122+:025
image
20224.000 008
.2000E.0008
1922.005
.1912.010
.1371.007
.133.1.007
.131_1014
.1191..00
yeast
216+t.005
21655.005
.13==0008
.194=0007
enron
.186.1.004
.18611.04
.1792.013
.178:.012
corel5k
.119::001
.113:t.001
.293::012
.298:.014
eurlex_da
.044.1.011
.0441.001
.065 0.511.003
.0641.006
eurlex_sm
.08211.001
.8821.001
.11911.05
.1191.007
delicious
+248+.003
+.03
2256+.006
257++005
Imc2007
1.051.001
086_1.001
071.1.003
.0691..00
Average precision
larger, the better)
Macro-averaging AUC (the larger, the betler)
music emotion
5.29
.221=.012
6221=.014
.726+-0006
.722#:006
music_sty _stlle
6.04
.734:1001
.728:.009
706=.0066
.731::009
mirflickr
3.35
.771#.026
.781+.046
.873-0002
.871=.003
image
.774+.013
.775+.012
.798=0015
.790=t011
scene
.794+-0014
888++.009
.874+-012
.882+-008
yeast
.733:5.088
F.35
.666 .266
.6661.006
enron
.665.1.020
.6701.024
.668
1.681.018
corel5k
4440+-014
.35+-013
.628=.007
.628=.007
10
eurlex_ de
-797+-009
800+-0112
872+-009
887+-007
curlex sm delicious
.722+-002
720=
028
831=
.30-+-007
630-+.006
.712-004
.71
tmc2007
.20.1.004
.8201.004
.8861.002
.8261.002
Table 9: Win/tie/loss counts of pairwise t-test (with < 0.05) about PML-GAN vs PML-GAN' over all 49 PML variant datasets with different numbers of candidate labels.
Hamming loss 0/49/0
Ranking loss 0/49/0
Average precision 0/49/0
Macro-averaging 00/48/1
Total 0/195/1
13

---

Under review as a conference paper at ICLR 2020
Table 10: Characteristics of the multi-label experimental datasets.
Dataset
#lnst.
#Feature
#Classes
Domain
CAL500
500
68
174
music
emotions
72
music
image
2000
294
images
scene
2407
294
images
tmc2007
28596
49060
text
Table 11: Comparison of PMI--AN with the comparison methods. The best results are presented in
bold font.
Dataset
PML-GAN
PARTICLE-VLS
PARTICLE-
PML-LC
PML-FP
ML-RBE
Hamming
loss (the smaller, the hetter)
CAL500
.143:1002
.1501.005
1544.004
1611.005
.1611.005
.2302.014
emotions
228:20088
226:2.066
2342:024
.244
.241:.033
.23=20077
image
.122.1005
222.1.056
2521.093
.2381.048
.6111.052
252.1.003
scene
.1301.005
.1324.037
.1334.088
.1462.031
.1611.037
.2002.002
tmc2007
.079=.001
.0802.003
.082.0003
.0792.002
.0801.002
.1092.001
Ranking loss (the smaller, the better)
CAL500
19011.003
19311.014
2301:017
19511.099
195.1.009
2001
emotions
.1782.013
.2321.031
.1812.023
.2001.029
.206:.025
.2362.028
image
.0872.010
.99:t.015
.102.013
.111=.020
.128:.023
.18::.012
scene
.0821.005
.0961.007
.123.010
.1001.15
.152.1.20
.411.007
tmc2007
.0451.001
.0591.008
.555..005
.0511.006
.0511.007
.5662.002
Average precision (the larger, better)
CAL500
499==0088
4642.019
460-.020
4812.016
480EE.015
433=.017
emotions
78911.08
76411.02
78801.026
.7881.023
.7771.030
.7701.034
image
8882.002
.7871.062
.7772.082
.7872.086
.7554.090
.7702.012
secne
.85==0008
810:t0881
802+.055
.5522.099
.7174.075
.7532.010
tmc2007
.8261.002
.14.1.022
8041.017
8131.016
8141..16
.79811003
Macro-averaging AUC (the larger, the hetter)
CAL500
.511..014
3191.011
5142.010
3401.009
.5431.010
.802.012
emotions
.822=.008
758+..021
.775=.030
810=.023
820+
.622.022
image
91911.010
8411.111
8061.104
866.1.119
8711.117
8061.008
seene
.2002.006
836:1.045
8312.075
810=.029
.795+.024
8242.045
tmc2007
.908=.002
877+.007
881=:005
893+.010
8934.011
.74
Coverage
(the smaller, the better)
CAL500
.8031.014
804
8661:007
8091.025
808.L.25
878.1.004
emotions
.355.001
343:1.048
316=.034
320:1.033
.3224.011
.3002.003
image
.089=.009
.134:.022
146+.020
.11
.218+.073
.144.011
.0831.005
.0981.011
1031.015
.0011.011
.1171.041
100110008
tmc2007
.1272.001
143+1.008
145+.008
.1001.010
.1394.010
.1252.002
One emrur (the smaller, the betler)
CAL500
1401..36
156.1.027
1701.020
.1251.023
.1331.023
.600L.021
emotions
.811:.031
.76:1.038
316::034
3391:035
33011.036
.3311.03
image
.2522.019
.05:4.094
3194.147
.3822.188
.36:1.133
.372.018
scene
.2431.012
.27.1.012
2751.081
303.1098
347.L.177
.3341.017
tmc2007
.1962.0044
197110011
.1961.022
2091.024
208.1.025
.2011.005
A.6 EXTENDED EXPERIMENTS
To further validate the effectiveness of the proposed approach, we conducted extended experiment on five datasets by adding a large portion of label noise into the true labels. Specifically, for each dataset with a total of C class labels, we created the PML data in the following way: For each training instance, if it has & true labels, we randomly select min(k, (k, C C- k) irrelevant labels to add and form its candidate label set. When of relevant true labels; when k C/2, the whole label set is used as the candidate label set.
C/2, the number irrelevant noisy labels is identical to the number
this setting, we performed experiments on five data sets with the different sizes class labels. The number of instances, number of classes and domain for each dataset are illustrated in Table [0] We compared the proposed PML-GAN method with the five comparison methods on the five datasets. For each dataset, randomly select 80% of the data for training and the remaining 20% for testing. We repeat each experiment 10 times with different random partitions of the datasets. The comparison test results in terms of six commonly used evaluation metrics (hamming loss, ranking loss,
14

---

Under review as a conference paper at ICLR 2020
average precision, avero-averaging AUC, coverage, error) Zhang & Zhou (2014) reported in Table 11] We can see that the proposed PML-GAN outperforms the comparison methods in general. Specifically, among the 30 cases over datasets and 6 evaluation metrics, the proposed approach outperforms PARTICLE-VI.S PARTICLE-MAP PMI.-IC, PMI.-FP and ML-RBBF 28, 29, 28, 29, 30 cases respectively. Moreover, the performance gains yield by the proposed approach over all the other methods are quite notable. For example, in terms of average precision, the proposed approach outperforms the best alternative comparison method by 6.1% and 4.5% on image and scene respectively. These results again validated the effectiveness of the proposed approach.
15


Reviews:
[{"review_id": "S1xtGfhRFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The motivation of this paper is to handle noise candidate multi-labels by co-training two networks. The work trains the disambiguation network, which learns to predict the probability of each class label being the additive irrelevant label and then is used to get the disambiguated label confidence vector, and the prediction network, which learns to predict the probability of the disambiguated labels.\nThe additional adversarial loss and generation loss is aimed to enhance the label disambiguation by learning the mapping from the disambiguated labels to the input features. \n\nPros:\nBy minimizing the disagreement between the confidence of being the disambiguated label and the predicted probability of each label, the partial multi-label learning gets better results. \n\nThe experiments in this paper are complete and thorough. The authors have tested the model in many datasets and designed the ablation study to verify the effect of each loss. And the proposed model achieved the state-of-art results.\n\nCons:\nHowever, the effect of the adversarial loss and the generation loss is doubtful because:\n1) The mapping from the labels to the input features is hard to learn since the label space does not contain the complete information of input features. It is doubtful that the generation is helpful.\n\n2) The results of the ablation study do not show consistently significant improvements.\n\n3) In the appendix, the variant of PML-GAN, which considers an auxiliary classification loss on the generated data, has little improvement compared to PML-GAN. The author claims that it is because of sufficient training data. What if considering the insufficient training data and testing the variant of PML-GAN? I think it can somehow verify the effect of the generation.\n\nAbout the writing of this paper, the motivation of the work is not clearly defined. Although we can get what the work was done, we cannot get why the work did this.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gWhqG2Yr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new method for partial multi-label learning, based on the idea of generative adversarial networks. The partial multi-label learning is the problem that one instance is associated with several ground truth labels simultaneously, but we are given a superset of the ground truth labels for training. To solve the problem, the paper proposes a learning model composing of four deep neural networks, two of them for prediction the ground truth labels (the prediction network, and the disambiguation network), one of them to generate instances based on the disambiguated labels, and one discriminative network to discriminated generated instance versus true instances. The four network are concatenated, and the paper proposed to optimize the sum of generation loss, the classification loss, and the adversarial loss. Theoretical results like the theories in the original GAN paper are also given. Finally, the paper does thorough empirical studies compared the proposed method with four other baselines and three variants on fourteen datasets with various settings.  \n\nThe paper is the first paper to proposes a GAN-style algorithm for the partial multi-label learning problem. The empirical results also validate its superior performance compared to other baselines on the data sets used. On the other hand, the paper fails to give enough intuition on why the GAN can naturally be used to solve the partial multi-label learning problem, and why the proposed method can perform better than other baselines. There is also some inaccuracy in discussing related work. Overall, given the contribution to introduce GAN to partial multi-label learning problem and the thorough empirical studies, I think this paper can be weakly accepted. \n\nMain arguments\nThe paper argues that existing methods may not be good on this problem is due to the fact that they generally learn a “confidence score” for every candidate labels, while the learned “confidence score” may not be accurate enough, and the errors may accumulate through these process. I agree with the paper on this aspect. However, in the paper, it also learns something similar to the “confidence” score in the disambiguation network \\tilde D, so why the proposed method is better than previous baselines? I guess the reason is due to the adversarial training part in the latter steps. But the paper fails to give clear intuition why the adversarial training part can lead to better performance.\n\nI am also wondering will the proposed model leads to a trivial solution. For example, one possible solution may be that both the prediction network F and the disambiguation network \\tilde D get the same solution: the given partial label set. In this way, the classification error can always be zero, and we can train the generation network to give a ground truth instance x, given that the generation network G is strong enough. So I also guess the discrimination network is the key part for the success of the proposed method empirically. So I am curious if we only use the adversarial loss without training the prediction network F, what the results will be. In the Ablation Study Sec. 5.3, the paper does not compare the results optimizing only the adversarial loss. I believe the results of such a study can help us understand what the key part of the proposed method is.\n\nIn the related work part, the arguments on “weak label learning” cannot be used for partial multi-label learning is not accurate. In fact, “weak label learning” studies the problem when positive labels are missing, and partial multi-label learning studies the problem when negative labels are missing. So in general, the methods for “weak label learning” can be used for “partial multi-label learning” if we exchange the role of positive labels and negative labels in both problems. So why we need to study partial multi-label learning? I think the reason is that, generally, in multi-label learning, the number of negative labels will be much larger than the number of positive labels. So “partial multi-label” is actually more strong supervision information than “weak labels” because positive labels are more important. Such arguments and related empirical studies (when the number of negative labels is much larger than positive labels) are suggested to be added into the further version of the paper.\n\nAlthough the paper has some deficiencies, it does a good job of introducing GAN into partial multi-label learning, and it is also the first paper to use the powerful deep neural networks into this problem, which may trigger some interesting studies given the booming of neural networks these days. And thorough empirical studies are done by comparing to not only baselines but also different loss parts of the proposal it owes. It is worth reading especially it may have an impact on further studies. \n\n---------------------------------------\nI am satisfied with the rebuttal and tend to increase my score. Although the paper is not perfect, it does a good job of introducing GAN into partial multi-label learning, and it is also the first paper (thus it is novel) to use the powerful deep neural networks into this problem, which may trigger some interesting studies given the booming of neural networks these days.\n\nActually, I think partial multi-label learning problem is difficult and it may not possible to have perfect solution nowadays. Partial multi-class learning may be easier since you know only one label is true, but partial multi-label learning is difficult since you do not know how many true labels there are. It makes no point to require a perfect solution for such a dirty problem. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkeQUY2aYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel GAN-based method to tackle PML problem. By using Encoder-Decoder architecture, the authors combine four neural networks including disambiguator, predictor, generator, and discriminator to implement the integrated model and get an overall good performance in various experiment settings and datasets.\n\nPros:\n- It's novel to introduce GAN to solve the PML problem.\n-The model works better than other state-of-art models overall.\n\nCons:\n- The novelty of this paper is limited to some extent. It seems that the model just combines ideas of GAN and PML. \n- I wonder if GAN still works in this architecture in some situations. Under the condition when the dimension of labels is small enough, it will be hard to generate good samples because the information which can be utilized for the generator mightn't be sufficient. However, when the dimension of labels is large, there will be some combinations of labels that aren't in the training set, which may harm the performance of the generator.\n- Because the relations between instances and labels aren't one-to-one in most cases, I wonder whether the five-layer perceptron still works well as a generator if one setting of labels can correspond to various kinds of instances.\n- It will be better to compare the generator part proposed in this paper with a simple interpolation method in the process of extending the dataset.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers a problem of clearly practical importance: multi-label classification where the ground truth label sets are noisy, specifically they are known (or at least assumed) to be a superset of the true ground truth labels. Learning a classifier in this setting require simultaneous identification of irrelevant labels. The proposed solution is a 4-part neural architecture, wherein a multi-label classifier is composed with a disambiguation or "cleanup" network, which is used as conditioning input to a conditional GAN which learns an inverse mapping, trained via an adversarial loss and also a least squares reconstruction loss ("generation loss"). 

Reviews were split 2 to 1 in favour of rejection, and the discussion phase did not resolve this split, as two reviewers did not revisit their assessments. R2 and R3 were concerned about the overall novelty and degeneracy of the inverse mapping problem. R1 increased their score after the rebuttal phase as they felt their concerns were addressed in comments (regarding issues surrounding the related work, the possibility of trivial solutions, and intuition for why the adversarial objective helps), but these were not addressed in the text as no updates were made.

I agree with the authors that PML is an important problem (one that receives perhaps less attention than it should from our community), and their empirical validation seems to support that their method outperforms (marginally, in many cases) methods from the literature. While the ablation study offers preliminary evidence that the inverse mapping is responsible for some of the gains, there are a lot of moving parts here and the authors haven't done a great job of motivating why this should help, or investigating why it in fact does. Based on the scores and my own reading of the paper, I'd recommend rejection at this time.

My own comments for the authors: I'd urge efforts to clarify the motivation for learning the inverse mapping, in particular adversarially (rather than just with the generation loss) in the text of the paper as you have in your rebuttals, and to improve the notation (the use of both D-tilde and D is confusing, and the omega notation seems unnecessary). I'm also not entirely clear whether the generator is stochastic or not, as the notation doesn't mention a randomly sampled latent variable (the traditional "z" here is a conditioning vector). Either way, the answer should be made more explicit.
Decision: Reject