Title: ADVERSARIAL PARTIAL MULTI-LABEL LEARNING

Abstract: Partial multi-label learning (PML), which tackles the problem of learning multilabel prediction models from instances with overcomplete noisy annotations, has recently started gaining attention from the research community. In this paper, we propose a novel adversarial learning model, PML-GAN, under a generalized encoder-decoder framework for partial multi-label learning. The PML-GAN model uses a disambiguation network to identify noisy labels and uses a multi-label prediction network to map the training instances to the disambiguated label vectors, while deploying a generative adversarial network as an inverse mapping from label vectors to data samples in the input feature space. The learning of the overall model corresponds to a minimax adversarial game, which enhances the correspondence of input features with the output labels. Extensive experiments are conducted on multiple datasets, while the proposed model demonstrates the state-of-the-art performance for partial multi-label learning.

1. INTRODUCTION:

In partial multi-label learning (PML), each training instance is assigned multiple candidate labels which are only partially relevant; that is, some assigned labels are irrelevant noise. As it is typically difficult and costly to precisely annotate instances for multi-label data (Xie & Huang, 2018) , the task of PML naturally arises in many real-world scenarios with crowdsource annotations. Figure 1 presents an example of training images for object recognition under the PML setting, where the union of candidate labels provided by crowdsource annotators is overcomplete and contains both ground truth labels (in black color) and irrelevant noise labels (in red color). PML is much more challenging than standard multi-label learning as the true labels are hidden among irrelevant labels and the number of true labels is unknown. The goal of PML is to learn a good multi-label prediction model from such a partial label training set, and hence reduce the annotation cost.

An intuitive strategy of PML is to treat all candidate labels as relevant ground truth, thus any off-theshelf multi-label classification methods can be adapted to induce an expected multi-label predictor (Zhang & Zhou, 2014) . This strategy, though simple, cannot work well since taking the noisy labels as part of the true labels will mislead the multi-label training and induce inferior prediction models. The PML work in (Xie & Huang, 2018) assumes that each candidate label has a confidence score of being a true label, and learns the confidence scores and classifier in an alternative manner by minimizing a confidence weighted ranking loss between the candidate and non-candidate labels. Although this work yields some reasonable results, the estimation of label confidence scores is error-prone, especially when noisy labels dominate, which can seriously impair the classifier's performance. Another recent work in (Fang & Zhang, 2019 ) also exploits label confidence values of candidate labels for PML. It estimates the confidence values using iterative label propagation and chooses the candidate labels with high confidence values as credible labels, which are then used to induce a multi-label prediction model. This work however suffers from the cumulative errors induced in propagation, which can impact the estimation of the credible labels and consequently impair the prediction model.

In this paper, we propose a novel adversarial learning model, PML-GAN, under a generalized encoder-decoder framework to tackle the partial multi-label learning problem. The PML-GAN model comprises four component networks: a disambiguation network that predicts the probability of each candidate label being an additive noise for a training instance; a prediction network that predicts the disambiguated true labels of each instance from its input features; a generation network that generates samples in the feature space given latent vectors in the label space; and a discrimination network that separates the generated samples from the real data. The prediction network and disambiguation network together form an encoder that maps data samples in the input feature space to the disambiguated label vectors, while the generation network and discrimination network forms a generative adversarial network (GAN) (Goodfellow et al., 2014) as an inverse decoding mapping from vectors in the multi-label space to samples in the input feature space. The learning of the overall model corresponds to a minimax adversarial game, which enhances the correspondence of input features with the output labels, and consequently boosts multi-label prediction performance. To the best of our knowledge, this is the first work that exploits GANs for PML. We conduct extensive experiments on multi-label datasets under partial multi-label learning setting. The empirical results show the proposed PML-GAN yields the state-of-the-art PML performance.

2. RELATED WORK:

Multi-label learning is a prevalent classification problem in many real world domains, where each instance can be assigned into multiple classes simultaneously. Many multi-label learning methods developed in the literature exploit label correlations at different degrees to produce multi-label classifiers (Zhang & Zhou, 2014) , including the first order methods (Zhang et al., 2018) , second order methods (Li et al., 2014) , and high-order methods (Burkhardt & Kramer, 2018) . Nevertheless, standard multi-label learning methods all assume each training instance is annotated with a complete set of ground truth labels, which can be impractical in many domains, where the annotations are obtained through crowdsourcing. With the union of annotations produced by multiple noisy labelers under the crowdsourcing setting, the partial multi-label learning (PML) problem arises naturally in real world scenarios, where the set of labels assigned to each training instance not only contain the ground truth labels, but also some additional irrelevant labels.

PML is more challenging than standard multi-label learning. The previous PML work in (Xie & Huang, 2018) proposes two methods, PML-FP and PML-LC, to estimate the label confidence values and optimize the relevance ordering of labels on each training instance by exploring the structural information in both feature and label spaces. However, due to the inherent property of alternative optimization, in these methods, the estimation error of labeling confidence values can negatively impact the coupled multi-label predictor. The work in (Sun et al., 2019) denoises the observed label matrix based on low-rank and sparse matrix decomposition. In another recent work (Fang & Zhang, 2019) , the authors propose to address PML problem using a two-stage strategy. It first estimates the label confidence value of each candidate label with iterative label propagation, and then performs multi-label learning over selected credible labels based on the confidence values by using pairwise label ranking (PARTICLE-VLS) or maximum a posteriori reasoning (PARTICLE-MAP). However, their credible label estimation can be impaired by the cumulative error induced in the propagation process, which can consequently degrade the multi-label learning performance, especially when there are many noisy labels.

Studies on weak learning and partial label learning have some connections with PML, but address different problems. Weak label learning tackles the problem of multi-label learning with incomplete labels (Sun et al., 2010; Wei et al., 2018) , where some ground truth labels are missed out from the annotations. Partial label learning (PLL) tackles multi-class classification under the setting where for each training instance there is one ground-truth label among the given candidate label set (Cour et al., 2011; Liu & Dietterich, 2012; Zhang & Yu, 2015; Yu & Zhang, 2016; Chen et al., 2018) . PLL methods cannot be directly applied on the more challenging PML problems, as under PML one has unknown numbers of ground truth labels among the candidate label set for each training instance.

Generative adversarial networks (GANs) (Goodfellow et al., 2014) , which perform minimax adversarial training over a generation network and a discrimination network, are one of the most popular generative models since its introduction. During the past years, a vast range of GAN-based adversarial learning methods have been developed to address different tasks, including semi-supervised learning (Kumar et al., 2017; Lecouat et al., 2018) , unsupervised learning (Jakab et al., 2018) , and learning with noisy labels (Thekumparampil et al., 2018) . The proposed work in this paper however is the first one that exploits GAN models for PML.

3. PROPOSED APPROACH:

In this section, we present the proposed adversarial partial multi-label learning model, PML-GAN, under the following setting. Assume we have a training set S = (X, Y ) = {(x i , y i )} N i=1 , where x i ∈ R d denotes the input feature vector for the i-th instance, and y i ∈ {0, 1} L is the corresponding annotated label indicator vector. The multiple 1 values in each y i indicate either the ground truth labels or the additional mis-annotated labels. We aim to learn a good multi-label prediction model from this partially labeled training set.

The proposed PML-GAN model is illustrated in Figure 2 , which comprises four component networks, disambiguation network D, prediction network F , generation network G and discrimination network D. The four components coordinate with and enhance each other under an encoder-decoder learning framework, which forms inverse mappings between the instance vectors in the input feature space and the label vectors in the output class label space. Below we present these model components, the learning objective and training algorithm in details.

3.1. PREDICTION WITH DISAMBIGUATED LABELS:

Comparing to standard multi-label learning, the main difficulty of PML is that the annotated labels {y i } in the training data contain additive noisy labels. The main challenge lies in identifying the ground truth labels z * i from each annotated candidate label vectors y i ; that is dropping the additional 1s from each candidate label vector y i . We propose to tackle this challenge by using a disambiguation network D : Ω x → Ω ∆ (Ω • denotes the corresponding domain space), which predicts the irrelevant labels for a given instance. Hence the true label vector z * i can be recovered as z * i = ReLU(y i -∆ i ), where ReLU(•) = max(•, 0) denotes the commonly used rectified linear unit activation function for deep neural networks. Then we can learn a prediction network F : Ω x → Ω z , i.e., a multi-label classifier, to predict the disambiguated ground truth labels for each instance.

Although the label indicator vectors in the training data are provided as discrete values, it is difficult for either the disambiguation network or the prediction network to directly produce discrete output values. Instead, by using a sigmoid activation function on the last layer of each network, D(x) and F (x) can predict the probability of each class label being the additive irrelevant label and the ground truth label respectively. With the disambiguation network and prediction network, we can perform partial multi-label learning by minimizing the classification loss on training data S:

min F, D L c (X, Y ; F, D) = (xi,yi)∼S c (F (x i ), z i ) (1) s.t. z i = ReLU(y i -∆ i ), ∆ i = D(x i ), ∀(x i , y i ) ∼ S

where z i denotes the disambiguated label confidence vector with continuous values in [0, 1], which can be viewed as a relaxation of a true label indicator vector, while c (•, •) denotes the cross-entropy loss between the predicted probability of each label and its confidence of being a ground-truth label.

3.2. INVERSE MAPPING WITH GANS:

The prediction network and disambiguation network together form an encoder that maps data samples in the input feature space to the disambiguated label vectors. To enhance the label disambiguation and hence improve multi-label classification, we propose to conduct an inverse decoding mapping from label vectors ẑ ∈ {0, 1} L to samples in the input feature space. In particular, we propose to deploy a generative adversarial network (GAN) model to generate samples in the input feature space given label vectors in the label space. The GAN model comprises a generation network G and a discrimination network D. Given a label vector ẑ sampled from a prior binomial distribution P (ẑ), one can generate a sample x using the generation network, x = G(ẑ). A two-class discriminator D is used to discriminate the generated samples from the real samples in S. The training of the GAN model is a minimax optimization problem over an adversarial loss function:

EQUATION

where the discriminator D tries to maximally distinguish the generated samples G(ẑ) from the real data samples in S, and the generator G tries to generate samples that are similar to the real data as much as possible such that the discriminator cannot tell the difference.

In theory, the samples generated by the adversarially trained generator G can have an identical distribution with the real data S (Goodfellow et al., 2014) . But this does not guarantee the generated samples can match the real training samples. To ensure the generator G can provide an inverse mapping function relative to the predictor F , we further propose to decode the disambiguated training label vectors into the training samples S with G by deploying a generation loss:

EQUATION

where g (•, •) measures the generation loss on each training instance and can be a least squares function. This generation loss can enhance the label disambiguation and improve multi-label learning.

3.3. LEARNING WITH PML-GANS:

By integrating the classification loss in Eq.( 1), the adversarial loss in Eq.( 2), and the generation loss in Eq.( 3) together, we obtain the following minimax optimization problem for the proposed PML-GAN model:

min G, D,F max D E (xi,yi)∼S c (F (x i ), z i ) + g (G(z i ), x i ) + β E xi∼S [logD(x i )] + E ẑ∼P (ẑ) [log(1 -D(G(ẑ)))] (4) s.t. z i = ReLU(y i -D(x i )), ∀(x i , y i ) ∼ S

where β is a trade-off hyperparameter that controls the relative importance of the generation loss and adversarial loss respectively; the objective function can be denoted as L(G, D, F, D). The learning of the overall model corresponds to a minimax adversarial game, which enhances the correspondence of input features with the output labels in an inverse encoder-decoder manner, and consequently boosts multi-label prediction performance.

We 

{(x 1 , y i ), • • • , (x m , y m )} from training set S. Sample n label vectors {ẑ 1 , • • • , ẑn } from a prior P (ẑ).

Update the network parameters of G, D, F by descending with their stochastic gradients:

∇Θ G, D,F 1 n n i=1 β[log(1 -D(G(ẑi)))] + 1 m m i=1 c F (xi), ReLU(yi -D(xi)) + g G(ReLU(yi -D(xi))), xi for r=1:k do Sample n label vectors {ẑ 1 , • • • , ẑn } from a prior P (ẑ).

Update the parameters of the discrimination network by ascending with its stochastic gradient:

∇Θ D β 1 m m i=1 [logD(xi)] + 1 n n i=1 [log(1 -D(G(ẑ)))]

end for end for Proposition 1. For any G, D, and F , the optimal discriminator D is given by

EQUATION

where p S (•) and p g (•) denote the probability distributions of real and generated data respectively.

Proof. Due to the conditional independence relationship between D and {F, D}, the optimal discriminator D only depends on the generator G. Given fixed G, the optimal discriminator can be derived in the same way as in the standard GAN model (Goodfellow et al., 2014 , Proposition 1). Here H(•) denotes an entropy function. Proof is provided in the appendix. This proposition suggests that F and G should be inverse mapping functions for each other in the ideal optimal case.

5. EXPERIMENTS:



5.1. EXPERIMENTAL SETTING:

Datasets. We conducted experiments on twelve multi-label classification datasets. Three of them have existing partial multi-label learning settings (mirflickr, music_style and music_emotion (Fang & Zhang, 2019) ). For each of the other nine datasets (Zhang & Zhou, 2014) , we transformed it into a PML dataset by randomly adding irrelevant labels into the candidate label set of each training instance. By adding different numbers of irrelevant labels, for each dataset we can create multiple PML variants with different average numbers of candidate labels. Following the setting of (Xie & Huang, 2018) , we also filtered out the rare labels and kept at most 15 classes in each dataset. The detailed characteristics of the processed datasets are given in Table 1 .

Comparison Methods. We compared our proposed method with four state-of-the-art PML methods and one baseline multi-label learning method. We adopted a simple but effective neural network based multi-label learning method, ML-RBF Zhang (2009) , derived from the radial basis function (RBF), as a baseline method, which performs PML by treating all the candidate labels as ground-truth labels. Then we used four recently developed PML methods for comparison, including the PML-LC and PML-FP methods from (Xie & Huang, 2018) and the PARTICLE-VLS and PARTICLE-MAP methods from (Fang & Zhang, 2019) .

Implementation. disambiguator, predictor, and discriminator are all three-layer networks with sigmoid activation in the output layer, while the generator is a five layer network with Tanh activation in the output layer. Batch normalization is also deployed in the middle three layers of the generation network. We used the Adam (Kingma & Ba, 2014) optimizer in our implementation. The mini-batch size, m, is set to 64. The hyperparameters k (the number of steps for discriminator update) and n (the number of label vectors sampled) in Algorithm 1 are set to 1 and 2 10 respectively. The hyperparameter β is chosen from {0.001, 0.01, 0.1, 1, 10} based on the classification loss value L c in the training objective function; that is, the β value that leads to the smallest training L c loss will be chosen.

5.2. COMPARISON RESULTS:

We compared the proposed PML-GAN method with the five comparison methods on the twelve datasets. For each dataset, we randomly select 80% of the data for training and use the remaining 20% for testing. We repeat each experiment 10 times with different random partitions of the datasets. The comparison test results in terms of four commonly used evaluation metrics (Hamming loss, ranking loss, average precision, and macro-averaging AUC) (Zhang & Zhou, 2014) are reported in Table 2 . The results are the means and standard deviations over the 10 repeated runs. We can see that the methods specially developed for PML problems all outperform the baseline multi-label neural network classifier, ML-RBF, in most cases. But it is difficult to beat the baseline competitor on all the datasets with different evaluation metrics. Among the total 48 cases over 12 datasets and 4 evaluation metrics, PARTICLE-VLS, PARTICLE-MAP, PML-LC and PML-FP outperform ML-RBF in 44, 48, 37 and 45 cases respectively. By contrast, the proposed PML-GAN method outperforms ML-RBF consistently across all the 48 cases with remarkably performance gains. Even comparing with all the other four PML methods, PML-GAN produced the best results in 43 out of the total 48 cases. Moreover, the performance gains yield by PML-GAN over all the other methods are quite notable in many cases. For example, in terms of average precision, PML-GAN outperforms the best alternative comparison method by 4.6%, 4.1%, and 3.3% on eurlex_dc, scene and image respectively. These results clearly demonstrate the effectiveness of the proposed PML-GAN model.

The results reported above are produced on each dataset with a selected average number of candidate labels. As shown in Table 1 , we have multiple PML variants with different numbers of candidate labels for each of the nine datasets in the list (except music_emotion, music_style, and mirflickr).

In total this provides us 49 PML datasets. We hence also conducted experiments on each of these 49 variant datasets, by comparing the proposed PML-GAN with each of the other methods in terms of the 4 evaluation metrics. In total there are 196 comparison cases for each pair of methods. For the comparison of "PML-GAN vs other method" in each case, we conducted pairwise t-test at significance level of 0.05. The win/tie/loss counts in all cases are reported in Table 3 . We can see that overall the proposed PML-GAN significantly outperforms PARTICLE-VLS, PARTICLE-MAP, PML-LC, PML-FP, and ML-RBF in 77%, 76%, 78.5%, 81.6%, and 90.3% of the cases respectively. This again validates the efficacy of the proposed method.

Impact of Irrelevant Labels:

To demonstrate how would the number of irrelevant labels affect the performance of PML methods, we plotted the experimental results on the delicious dataset with different average numbers of candidate labels in Figure 4 . We can see with the increase of the number of irrelevant labels, consequently the average number of candidate labels, the performance of each method in general degrades. Nevertheless, the proposed PML-GAN consistently outperforms all the other methods. Moreover, in terms of Hamming loss, the performance of PML-GAN actually is quite stable with the increase of the noisy labels. This validates the effectiveness of PML-GAN in irrelevant noisy label disambiguation. 

5.3. ABLATION STUDY:

As shown in Eq.( 6), the objective of PML-GAN contains three parts: classification loss, generation loss and adversarial loss. The generation loss and adversarial loss are integrated to assist the predictor training. To investigate and validate the contribution of the generation loss and adversarial loss, we conducted an ablation study by comparing PML-GAN with three of its ablation variants: (1) CLS-GEN, which drops the adversarial loss;

(2) CLS-GAN, which drops the generation loss; and (3) CLS-ML, which only uses the classification loss by dropping both the adversarial loss and generation loss. The comparison results are reported in Table 4 . We can see that comparing to the full model, all three variants produced inferior results in general. Among the three variants, both CLS-GEN and CLS-GAN outperform CLS-ML in most cases. This suggests that both generation loss and adversarial loss are critical terms for the proposed model. Moreover, even the baseline variant CLS-ML still produces some reasonable PML results. This suggests the integration of our proposed prediction network and disambiguation network is also effective.

6. CONCLUSION:

In this paper, we proposed a novel adversarial model for partial multi-label learning. The proposed model comprises four component networks, which form an encoder-decoder framework to improve noisy label disambiguation and boost multi-label learning performance. The training problem forms a minimax adversarial optimization, which is solved using an alternative min-max procedure with minibatch-based stochastic gradient descent. We conducted extensive experiments on multiple PML datasets. The results show that the proposed model outperforms all the comparison methods and achieves the state-of-the-art PML performance.

A APPENDIX:

In this appendix, we provide proof for Proposition 2, details of the network structure for the PML-GAN model, and additional experimental results. For additional experimental results, we first reported results with two additional evaluation metrics, and results of parameter sensitivity analysis. Then we empirically compared PML-GAN with its variant that considers additional classification loss on the generated data. Finally we conducted extended experiments on a few datasets where we constructed PML data by adding large number of label noise.

A.1 PROOF OF PROPOSITION 2

Proof. As shown in section 3.3, L(G, D, F, D) denotes the objective function of PML-GAN. Based on the solution for optimal discriminator D in Proposition 1, we have:

C(G, D, F ) = max D L(G, D, F, D) =    E (x i ,y i )∼S c(F (xi), ReLU(yi -D(xi))) + g (G(ReLU(yi -D(xi))), xi) + β Ex∼p S [logD * G (x)] + E ẑ∼P (ẑ) [log(1 -D * G (G(ẑ)))]    =    E (x i ,y i )∼S c(F (xi), ReLU(yi -D(xi))) + g (G(ReLU(yi -D(xi))), xi) + β Ex∼p S [logD * G (x)] + Ex∼p g [log(1 -D * G (x))]    =    E (x i ,y i )∼S c(F (xi), ReLU(yi -D(xi))) + g (G(ReLU(yi -D(xi))), xi) + β Ex∼p S [log p S (x) p S (x)+pg (x) ] + Ex∼p g [log pg (x) p S (x)+pg (x) ]   

Note given fixed D, F is conditionally independent from G and D. Hence the minimization of C(G, D, F ) over F can be independently conducted from the minimization over G.

Let z i = ReLU(y i -D(x i )). With the cross-entropy loss function c (•, •) we have:

min F C(G, D, F ) ≡ min F E (x i ,y i )∼S c F (xi), ReLU(yi -D(xi)) ≡ min F E (x i ,y i )∼S -z i log F (xi) -(1 -zi) log(1 -F (xi)) ≡ min F E (x i ,y i )∼S H(zi) + KL(zi F (xi)) ≥ E (x i ,y i )∼S H(zi)

where H(•) denotes the entropy over a binomial distriubtion vector and KL(•) denotes the KLdivergence between two sets of binomial distributions. Assume sufficient capacity for F , the minimum can be reached when the predictor obtains the same distributions as the z i ; that is

F * (xi) = zi = ReLU(yi -D(xi)), ∀(xi, yi) ∈ S

Next let's consider the minimization problem over G. Note G is involved in both the generation loss and adversarial loss. If we could find solutions that lead to minimals in both losses separately, we can guarantee a minimal in the united loss. The adversarial loss part in C(G, D, F ) can be rewritten as where the minimal can be achieved when p S = p g which leads to zeros as the KL-divergence values.

βL adv = β Ex∼p S [log pS(x) pS(x) + pg(x) ] + Ex∼p g [log pg(x) pS(x) + pg(x) ] = β KL(pS, pS + pg 2 ) -log 2 + KL(pg, pS + pg 2 ) -log 2 = β KL(pS, pS + pg 2 ) + KL(pg, pS + pg 2 ) -log 4 ≥ -β log 4

The generation loss part (with least squares loss function) in C(G, D, F ) can be rewritten as

E (x i ,y i )∼S g (G(ReLU(yi -D(xi))), xi) =E (x i ,y i )∼S G(ReLU(yi -D(xi))) -xi 2 ≥0

where the minimal 0 can only be achieved when

G(ReLU(yi -D(xi))) = xi, ∀(xi, yi) ∈ S

It is obvious the optimal condition above can be satisfied simultaneously together with the condition p g = p S . Hence the proposition is proved.

A.2 NETWORK STRUCTURE OF THE PML-GAN MODEL:

The proposed PML-GAN model has four component networks, all of them are designed as multilayer perceptrons with LeakyReLu activation function for the middle layers. The disambiguator, predictor, and discriminator are all three-layer networks with sigmoid activation in the output layer, while the generator is a five layer network with Tanh activation in the output layer. Batch normalization is also deployed in the middle three layers of the generation network. The detailed input and output setup information for each layer of the networks is given in Table 5 .

A.3 ADDITIONAL EXPERIMENTAL RESULTS IN MORE EVALUATION METRICS:

For the comparison results, we evaluated the test performance using six commonly used metrics from Zhang & Zhou (2014) . Due to space limitation, we previously reported the results in terms of four metrics. The results in terms of the remaining two metrics, one error and coverage, are reported in Table 6 . We can see that our proposed PML-GAN again produced the best results in majority cases.

In Table 7 , we reported the ablation study results in terms of coverage and one error. The relative comparison performance is quite similar to the results reported in the paper. It suggests the component networks in our model are necessary and useful.

A.4 PARAMETER SENSITIVITY ANALYSIS:

We also conducted experiments on the eurlex_dc data set to study the impact of the trade-off hyperparameter β on the performance of PML-GAN. We repeated the experiments in the same setting as before with different β values from the range {0.001, 0.01, 0.1, 1, 10}. Note a larger β provides larger weight to the adversarial loss.

Figure 5 reports the average test results over 10 runs for different β values. We can see when β is very small, the performance is poor. With the increase of β, the performance improves. This suggests the adversarial loss is important. When β = 0.1 the best result is obtained. When β is too large, performance degrades as the adversarial loss dominates. This makes sense since the adversarial loss is expected to help the classification model, rather than dominating the learning. The classification loss in the PML-GAN model of Eq.( 6) takes the real data into account, as that is the primary concern of PML. A natural extension is to consider an auxiliary classification loss on the generated data as well. This extension leads to the following variant of PML-GAN, which we denote as PML-GAN : The comparison results between PML-GAN and PML-GAN are reported in Table 8 . We can see that PML-GAN has similar performance with PML-GAN, and the difference between their results are very small in most cases. To statistically compare PML-GAN with PML-GAN , we conducted pairwise t-test at significance level of 0.05 over all the 49 PML variant datasets in terms of the 4 evaluation metrics. The win/tie/loss counts in all 196 cases are reported in Table 9 . We can see that overall the PML-GAN ties with PML-GAN in 99.5% cases. This suggests that an addition auxiliary classification loss on the generated data cannot improve PML-GAN, which makes sense given the goal is to learn a good classification model on the sufficient real data. Meanwhile, the additional hyperparameter α of PML-GAN induces much higher computational cost due to additional parameter selection. Overall, PML-GAN is a more suitable choice than PML-GAN . 11 . We can see that the proposed PML-GAN outperforms the comparison methods in general. Specifically, among the 30 cases over 5 datasets and 6 evaluation metrics, the proposed approach outperforms PARTICLE-VLS, PARTICLE-MAP, PML-LC, PML-FP and ML-RBF in 28, 29, 28, 29, 30 cases respectively. Moreover, the performance gains yield by the proposed approach over all the other methods are quite notable. For example, in terms of average precision, the proposed approach outperforms the best alternative comparison method by 6.1% and 4.5% on image and scene respectively. These results again validated the effectiveness of the proposed approach.

min G, D,F max D E (xi,yi)∼S c (F (x i ), z i ) + g (G(z i ), x i ) + α E ẑ∼P (ẑ) c (F (G(ẑ)), ẑ) + β E xi∼S [logD(x i )] + E ẑ∼P (ẑ) [log(1 -D(G(ẑ)))] (6) s.t. z i = ReLU(y i -D(x i )), ∀(x i , y i ) ∼ S

:



Figures and Tables:

Figure 1: An annotated image under the partial multi-label learning (PML) setting.

Figure 2: The proposed PML-GAN model. It has four component networks: generator G, disambiguator D, predictor F , and discriminator D.

Figure 3: Dependence graph of PML-GAN.In the proposed PML-GAN model, given the generator G, the discriminator D is conditionally independent from the predictor F and disambiguator D. Between G, F and D, G and F are conditionally independent from each other given D. Their independence relationship can be illustrated using the undirected dependence graph in Figure3. Based on these conditional independence relationships, we have the following optimality results.

Assume the model has sufficient capacity. Let C(G, D, F ) = max D L(G, D, F, D). Given fixed D, the minimum of C(G, D, F ) is lower bounded by E (xi,yi)∼S H ReLU(y i -D(x i ))β log 4, which can be achieved when F (x i ) = ReLU(y i -D(x i )), G(F (x i )) = x i , and p g = p S .

Figure 4: Test results with different average numbers of candidate labels on the delicious dataset.

where α and β are trade-off hyperparameters. For the parameter selection of PML-GAN , both α and β are chosen from {0.001, 0.01, 0.1, 1, 10} based on the classification loss value L c on the real data in the training objective function.

Figure 5: Test performance with different β values on the eurlex_dc data set.

Information of the experimental data sets. The number of instances, features and classes are recorded. The "avg.#CLs" column lists the average number of candidate labels in each PML set.

Table content: <table><tr><td>Dataset</td><td colspan="2">#Inst. #Feats #Classes</td><td>avg.#CLs</td><td>Dataset</td><td colspan="2">#Inst. #Feats #Classes</td><td>avg.#CLs</td></tr><tr><td colspan="2">music_emotion 6833 98</td><td>11</td><td>5.29</td><td colspan="2">music_style 6839 98</td><td>10</td><td>6.04</td></tr><tr><td>mirflickr</td><td>10433 100</td><td>7</td><td>3.35</td><td>image</td><td>2000 294</td><td>5</td><td>2,3,4</td></tr><tr><td>scene</td><td>2407 294</td><td>6</td><td>3,4,5</td><td>yeast</td><td>2417 103</td><td>14</td><td>9,10,11,12</td></tr><tr><td>enron</td><td>1702 1001</td><td>15</td><td colspan="2">8,9,10,11,12,13 corel5k</td><td>5000 499</td><td>15</td><td>8,9,10,11,12,13</td></tr><tr><td>eurlex_dc</td><td>8636 100</td><td>15</td><td colspan="3">8,9,10,11,12,13 eurlex_sm 12679 100</td><td>15</td><td>8,9,10,11,12,13</td></tr><tr><td>delicious</td><td>14000 500</td><td>15</td><td colspan="2">8,9,10,11,12,13 tmc2007</td><td>28596 49060</td><td>15</td><td>8,9,10,11,12,13</td></tr></table>

Comparison results of in terms of Hamming loss, ranking loss, average precision, and micro-averaging AUC. The best results are presented in bold font. The average number of candidate labels is presented under the column "avg.#C.Ls".

Table content: <table><tr><td>Data set</td><td colspan="4">avg.#C.Ls PML-GAN PARTICLE-VLS PARTICLE-MAP</td><td>PML-LC</td><td>PML-FP</td><td>ML-RBF</td></tr><tr><td colspan="3">Hamming loss (the smaller, the better)</td><td/><td/><td/></tr><tr><td>music_emotion</td><td>5.29</td><td>.197±.004</td><td>.212±.004</td><td>.215±.004</td><td colspan="2">.236±.003 .245±.004 .779±.004</td></tr><tr><td>music_style</td><td>6.04</td><td>.116±.005</td><td>.121±.003</td><td>.175±.005</td><td colspan="2">.126±.004 .126±.004 .856±.001</td></tr><tr><td>mirflickr</td><td>3.35</td><td>.171±.002</td><td>.178±.035</td><td>.189±.081</td><td colspan="2">.202±.057 .202±.057 .748±.002</td></tr><tr><td>image</td><td>3</td><td>.202±.008</td><td>.234±.065</td><td>.269±.096</td><td colspan="2">.264±.072 .267±.063 .754±.003</td></tr><tr><td>scene</td><td>4</td><td>.137±.007</td><td>.184±.037</td><td>.174±.035</td><td colspan="2">.178±.029 .187±.038 .820±.001</td></tr><tr><td>yeast</td><td/><td>.216±.005</td><td>.226±.004</td><td>.220±.008</td><td colspan="2">.226±.008 .219±.009 .694±.003</td></tr><tr><td>enron</td><td/><td>.186±.004</td><td>.197±.032</td><td>.190±.036</td><td colspan="2">.206±.027 .206±.027 .813±.004</td></tr><tr><td>corel5k eurlex_dc</td><td>10</td><td>.119±.001 .044±.001</td><td>.189±.012 .061±.001</td><td>.269±.027 .064±.004</td><td colspan="2">.151±.008 .152±.008 .886±.001 .096±.001 .071±.001 .933±.001</td></tr><tr><td>eurlex_sm</td><td/><td>.082±.001</td><td>.067±.001</td><td>.076±.002</td><td colspan="2">.119±.006 .122±.002 .885±.001</td></tr><tr><td>delicious</td><td/><td>.248±.003</td><td>.260±.003</td><td>.290±.005</td><td colspan="2">.290±.004 .290±.004 .712±.002</td></tr><tr><td>tmc2007</td><td/><td>.085±.001</td><td>.090±.003</td><td>.110±.003</td><td colspan="2">.103±.002 .103±.002 .857±.001</td></tr><tr><td colspan="3">Ranking loss (the smaller, the better)</td><td/><td/><td/></tr><tr><td>music_emotion</td><td>5.29</td><td>.243±.009</td><td>.263±.008</td><td>.240±.007</td><td colspan="2">.267±.009 .275±.010 .365±.010</td></tr><tr><td>music_style</td><td>6.04</td><td>.141±.001</td><td>.163±.007</td><td>.147±.005</td><td colspan="2">.215±.005 .150±.005 .242±.006</td></tr><tr><td>mirflickr</td><td>3.35</td><td>.127±.014</td><td>.227±.029</td><td>.129±.108</td><td colspan="2">.160±.029 .143±.028 .195±.015</td></tr><tr><td>image</td><td>3</td><td>.192±.015</td><td>.239±.077</td><td>.250±.085</td><td colspan="2">.291±.134 .217±.120 .251±.019</td></tr><tr><td>scene</td><td>4</td><td>.131±.014</td><td>.177±.049</td><td>.167±.060</td><td colspan="2">.192±.032 .238±.056 .188±.014</td></tr><tr><td>yeast</td><td/><td>.193±.008</td><td>.203±.007</td><td>.208±.012</td><td colspan="2">.219±.011 .203±.008 .270±.007</td></tr><tr><td>enron</td><td/><td>.179±.013</td><td>.240±.078</td><td>.182±.029</td><td colspan="2">.239±.048 .239±.047 .244±.010</td></tr><tr><td>corel5k</td><td/><td>.293±.012</td><td>.367±.032</td><td>.311±.008</td><td colspan="2">.366±.035 .398±.025 .404±.082</td></tr><tr><td>eurlex_dc eurlex_sm</td><td>10</td><td>.065±.003 .119±.005</td><td>.150±.004 .129±.007</td><td>.085±.004 .127±.009</td><td colspan="2">.137±.008 .131±.001 .135±.003 .282±.007 .182±.008 .183±.003</td></tr><tr><td>delicious</td><td/><td>.256±.006</td><td>.314±.005</td><td>.276±.004</td><td colspan="2">.277±.005 .276±.005 .316±.003</td></tr><tr><td>tmc2007</td><td/><td>.071±.003</td><td>.096±.008</td><td>.095±.007</td><td colspan="2">.082±.005 .080±.005 .153±.002</td></tr><tr><td colspan="3">Average precision (the larger, the better)</td><td/><td/><td/></tr><tr><td>music_emotion</td><td>5.29</td><td>.621±.012</td><td>.605±.012</td><td>.612±.009</td><td colspan="2">.574±.013 .568±.014 .506±.012</td></tr><tr><td>music_style</td><td>6.04</td><td>.734±.015</td><td>.715±.009</td><td>.709±.009</td><td colspan="2">.702±.008 .703±.008 .646±.010</td></tr><tr><td>mirflickr</td><td>3.35</td><td>.771±.026</td><td>.678±.027</td><td>.791±.202</td><td colspan="2">.736±.043 .758±.039 .676±.048</td></tr><tr><td>image</td><td>3</td><td>.774±.013</td><td>.741±.090</td><td>.729±.086</td><td colspan="2">.644±.131 .725±.119 .723±.021</td></tr><tr><td>scene</td><td>4</td><td>.794±.014</td><td>.750±.074</td><td>.753±.064</td><td colspan="2">.689±.047 .710±.079 .728±.015</td></tr><tr><td>yeast</td><td/><td>.733±.008</td><td>.724±.010</td><td>.714±.010</td><td colspan="2">.721±.012 .728±.010 .634±.008</td></tr><tr><td>enron</td><td/><td>.665±.020</td><td>.595±.099</td><td>.661±.047</td><td colspan="2">.556±.041 .575±.041 .560±.009</td></tr><tr><td>corel5k</td><td/><td>.440±.014</td><td>.377±.025</td><td>.415±.008</td><td colspan="2">.345±.027 .384±.021 .334±.008</td></tr><tr><td>eurlex_dc eurlex_sm</td><td>10</td><td>.797±.009 .722±.002</td><td>.692±.013 .705±.009</td><td>.751±.008 .683±.011</td><td colspan="2">.693±.019 .716±.014 .710±.000 .438±.016 .679±.011 .656±.000</td></tr><tr><td>delicious</td><td/><td>.630±.007</td><td>.596±.007</td><td>.601±.008</td><td colspan="2">.607±.007 .608±.006 .576±.004</td></tr><tr><td>tmc2007</td><td/><td>.820±.004</td><td>.799±.013</td><td>.759±.013</td><td colspan="2">.793±.012 .794±.012 .662±.003</td></tr><tr><td colspan="3">Macro-averaging AUC (the larger, the better)</td><td/><td/><td/></tr><tr><td>music_emotion</td><td>5.29</td><td>.726±.006</td><td>.673±.008</td><td>.676±.004</td><td colspan="2">.632±.008 .636±.007 .504±.003</td></tr><tr><td>music_style</td><td>6.04</td><td>.706±.016</td><td>.694±.007</td><td>.715±.005</td><td colspan="2">.754±.006 .755±.006 .503±.003</td></tr><tr><td>mirflickr</td><td>3.35</td><td>.873±.002</td><td>.806±.040</td><td>.816±.057</td><td colspan="2">.842±.030 .840±.031 .808±.000</td></tr><tr><td>image</td><td>3</td><td>.798±.015</td><td>.735±.119</td><td>.757±.103</td><td colspan="2">.727±.131 .728±.130 .729±.000</td></tr><tr><td>scene</td><td>4</td><td>.874±.012</td><td>.789±.054</td><td>.814±.049</td><td colspan="2">.734±.041 .735±.038 .730±.010</td></tr><tr><td>yeast</td><td/><td>.666±.026</td><td>.610±.017</td><td>.650±.014</td><td colspan="2">.631±.019 .636±.020 .600±.012</td></tr><tr><td>enron</td><td/><td>.668±.018</td><td>.607±.024</td><td>.660±.031</td><td colspan="2">.631±.029 .631±.029 .596±.016</td></tr><tr><td>corel5k</td><td/><td>.628±.007</td><td>.542±.039</td><td>.618±.031</td><td colspan="2">.550±.033 .551±.034 .527±.007</td></tr><tr><td>eurlex_dc eurlex_sm</td><td>10</td><td>.872±.009 .827±.008</td><td>.867±.011 .843±.008</td><td>.835±.008 .823±.008</td><td colspan="2">.840±.005 .853±.007 .825±.006 .799±.007 .780±.004 .774±.003</td></tr><tr><td>delicious</td><td/><td>.712±.004</td><td>.666±.006</td><td>.634±.006</td><td colspan="2">.688±.006 .688±.006 .632±.002</td></tr><tr><td>tmc2007</td><td/><td>.886±.002</td><td>.836±.010</td><td>.830±.009</td><td colspan="2">.861±.008 .859±.008 .765±.003</td></tr></table>

Win/tie/loss counts of pairwise t-test (with p < 0.05 ) between PML-GAN and each comparison method over all dataset variants with different numbers of candidate labels.

Table content: <table><tr><td>Evaluation Metric</td><td>PARTICLE-VLS</td><td colspan="2">PML-GAN vs -PARTICLE-MAP PML-LC</td><td>PML-FP</td><td>ML-RBF</td></tr><tr><td>Hamming loss</td><td>39/6/4</td><td>38/9/2</td><td>40/7/2</td><td>40/3/6</td><td>45/4/0</td></tr><tr><td>Ranking loss</td><td>38/9/2</td><td>38/8/3</td><td>38/8/3</td><td>40/5/4</td><td>44/3/2</td></tr><tr><td>Average precision</td><td>37/8/4</td><td>36/10/3</td><td>40/6/3</td><td>42/4/3</td><td>46/3/0</td></tr><tr><td>Macro-averaging AUC</td><td>37/7/5</td><td>37/8/4</td><td>36/7/6</td><td>38/5/6</td><td>42/7/0</td></tr><tr><td>Total</td><td>151/30/15</td><td>149/35/12</td><td>154/28/14</td><td>160/17/19</td><td>177/17/2</td></tr></table>

Comparison results of PML-GAN and its three ablation variants.

Table content: <table><tr><td>Data set</td><td colspan="2">PML-GAN CLS-GEN CLS-GAN CLS-ML PML-GAN CLS-GEN CLS-GAN CLS-ML</td></tr><tr><td colspan="2">Hamming loss (the smaller, the better)</td><td>Ranking loss (the smaller, the better)</td></tr><tr><td colspan="3">music_emotion .197±.004 .200±.003 .199±.005 .202±.003 .243±.009 .247±.008 .244±.007 .248±.007</td></tr><tr><td>music_style</td><td colspan="2">.116±.005 .117±.003 .118±.003 .120±.003 .141±.001 .146±.004 .143±.007 .164±.007</td></tr><tr><td>mirflickr</td><td colspan="2">.171±.002 .174±.005 .172±.005 .176±.004 .127±.014 .132±.022 .128±.020 .135±.015</td></tr><tr><td>image</td><td colspan="2">.202±.008 .205±.006 .208±.008 .230±.008 .192±.015 .196±.013 .196±.013 .206±.014</td></tr><tr><td>scene</td><td colspan="2">.137±.007 .140±.013 .143±.009 .152±.007 .131±.014 .136±.009 .131±.007 .137±.006</td></tr><tr><td>yeast</td><td colspan="2">.216±.005 .219±.004 .217±.006 .228±.005 .193±.008 .198±.010 .196±.010 .205±.012</td></tr><tr><td>enron</td><td colspan="2">.186±.004 .281±.014 .273±.018 .281±.012 .179±.013 .184±.013 .184±.008 .189±.010</td></tr><tr><td>corel5k</td><td colspan="2">.119±.001 .119±.002 .121±.002 .122±.002 .293±.012 .302±.009 .302±.017 .308±.009</td></tr><tr><td>eurlex_dc</td><td colspan="2">.044±.001 .052±.003 .046±.001 .054±.001 .065±.003 .085±.012 .068±.003 .071±.005</td></tr><tr><td>eurlex_sm</td><td colspan="2">.082±.001 .083±.001 .084±.001 .086±.001 .119±.005 .120±.003 .123±.005 .125±.004</td></tr><tr><td>delicious</td><td colspan="2">.248±.003 .251±.001 .249±.002 .255±.002 .256±.006 .259±.006 .257±.004 .323±.006</td></tr><tr><td>tmc2007</td><td colspan="2">.085±.001 .088±.001 .085±.001 .091±.001 .071±.003 .074±.002 .072±.002 .075±.003</td></tr><tr><td colspan="2">Average precision (the larger, the better)</td><td>Macro-averaging AUC (the larger, the better)</td></tr><tr><td colspan="3">music_emotion .621±.012 .608±.012 .620±.010 .614±.010 .726±.006 .724±.007 .724±.009 .615±.010</td></tr><tr><td>music_style</td><td colspan="2">.734±.015 .726±.008 .730±.010 .700±.012 .706±.016 .700±.011 .703±.000 .691±.014</td></tr><tr><td>mirflickr</td><td colspan="2">.771±.026 .763±.042 .769±.038 .730±.010 .873±.002 .872±.003 .872±.002 .860±.003</td></tr><tr><td>image</td><td colspan="2">.774±.013 .771±.015 .766±.017 .758±.018 .798±.015 .797±.009 .794±.012 .790±.000</td></tr><tr><td>scene</td><td colspan="2">.794±.014 .794±.013 .789±.011 .783±.009 .874±.012 .872±.006 .873±.009 .852±.010</td></tr><tr><td>yeast</td><td colspan="2">.733±.008 .728±.014 .731±.015 .708±.012 .666±.026 .628±.015 .627±.011 .625±.011</td></tr><tr><td>enron</td><td colspan="2">.665±.020 .663±.025 .657±.014 .655±.023 .668±.018 .566±.012 .573±.013 .561±.008</td></tr><tr><td>corel5k</td><td colspan="2">.440±.014 .434±.011 .436±.015 .432±.008 .628±.007 .579±.004 .579±.006 .570±.006</td></tr><tr><td>eurlex_dc</td><td colspan="2">.797±.009 .781±.019 .797±.008 .779±.011 .872±.009 .839±.016 .872±.009 .822±.007</td></tr><tr><td>eurlex_sm</td><td colspan="2">.722±.002 .719±.003 .714±.009 .712±.004 .827±.008 .824±.005 .825±.008 .820±.005</td></tr><tr><td>delicious</td><td colspan="2">.630±.007 .628±.004 .630±.006 .628±.005 .712±.004 .710±.003 .711±.002 .709±.002</td></tr><tr><td>tmc2007</td><td colspan="2">.820±.004 .818±.004 .820±.004 .815±.003 .886±.002 .884±.003 .886±.002 .883±.002</td></tr></table>

The network architecture of PML-GAN. BN: Batch normalization; LReLU: Leaky rectified unit; Act.: Activation function; dim: Feature dimension of training samples x; class num: the number of class labels.

Table content: <table><tr><td/><td>Generator G</td><td/><td/><td/><td>Discriminator D</td><td/></tr><tr><td>Input</td><td>Output</td><td>BN</td><td>Act.</td><td>Input</td><td>Output</td><td>Act.</td></tr><tr><td>ẑ 512 1024 256</td><td>512 1024 256 128</td><td>× √ √ √</td><td>LReLU LReLU LReLU LReLU</td><td>data 512 256</td><td>512 256 1</td><td>LReLU LReLU Sigmoid</td></tr><tr><td>128</td><td>dim</td><td>×</td><td>Tanh</td><td/><td/><td/></tr><tr><td/><td colspan="2">Disambiguator D</td><td/><td/><td>Predictor F</td><td/></tr><tr><td>x</td><td>512</td><td>-</td><td>LReLU</td><td>x</td><td>512</td><td>LReLU</td></tr><tr><td>512</td><td>256</td><td>-</td><td>LReLU</td><td>512</td><td>256</td><td>LReLU</td></tr><tr><td>256</td><td>class num</td><td>-</td><td>Sigmoid</td><td>256</td><td>class num</td><td>Sigmoid</td></tr></table>

Comparison results for PML-GAN and the other methods in terms of two additional metrics: coverage and one error. The best results are in bold font.

Table content: <table><tr><td>Data set</td><td colspan="4">avg.#C.Ls PML-GAN PARTICLE-VLS PARTICLE-MAP</td><td>PML-LC</td><td>PML-FP</td><td>ML-RBF</td></tr><tr><td colspan="3">Coverage (the smaller, the better)</td><td/><td/><td/></tr><tr><td>music_emotion</td><td>5.29</td><td>.407±.008</td><td>.413±.007</td><td>.396±.007</td><td colspan="2">.428±.010 .434±.010 .508±.006</td></tr><tr><td>music_style</td><td>6.04</td><td>.202±.011</td><td>.208±.006</td><td>.205±.007</td><td colspan="2">.203±.008 .203±.008 .305±.006</td></tr><tr><td>mirflickr</td><td>3.35</td><td>.227±.010</td><td>.270±.079</td><td>.227±.057</td><td colspan="2">.234±.073 .233±.074 .258±.002</td></tr><tr><td>image</td><td>3</td><td>.211±.014</td><td>.227±.079</td><td>.263±.085</td><td colspan="2">.305±.109 .298±.095 .249±.015</td></tr><tr><td>scene</td><td>4</td><td>.124±.010</td><td>.144±.054</td><td>.190±.050</td><td colspan="2">.180±.033 .214±.045 .167±.012</td></tr><tr><td>yeast</td><td/><td>.502±.014</td><td>.506±.010</td><td>.528±.003</td><td colspan="2">.524±.016 .495±.016 .583±.007</td></tr><tr><td>enron</td><td/><td>.402±.015</td><td>.457±.116</td><td>.369±.075</td><td colspan="2">.496±.089 .496±.088 .458±.017</td></tr><tr><td>corel5k</td><td/><td>.384±.011</td><td>.541±.022</td><td>.393±.017</td><td colspan="2">.497±.042 .489±.035 .492±.009</td></tr><tr><td>eurlex_dc eurlex_sm</td><td>10</td><td>.063±.003 .172±.006</td><td>.136±.003 .192±.009</td><td>.082±.004 .183±.012</td><td colspan="2">.318±.017 .162±.015 .117±.003 .403±.009 .408±.009 .243±.005</td></tr><tr><td>delicious</td><td/><td>.564±.008</td><td>.603±.004</td><td>.584±.004</td><td colspan="2">.591±.005 .590±.005 .610±.004</td></tr><tr><td>tmc2007</td><td/><td>.181±.006</td><td>.205±.008</td><td>.214±.011</td><td colspan="2">.196±.007 .194±.008 .283±.003</td></tr><tr><td colspan="3">One error (the smaller, the better)</td><td/><td/><td/></tr><tr><td>music_emotion</td><td>5.29</td><td>.444±.027</td><td>.473±.016</td><td>.475±.018</td><td colspan="2">.556±.028 .540±.027 .587±.019</td></tr><tr><td>music_style</td><td>6.04</td><td>.346±.022</td><td>.374±.005</td><td>.399±.019</td><td colspan="2">.409±.013 .408±.013 .385±.006</td></tr><tr><td>mirflickr</td><td>3.35</td><td>.337±.059</td><td>.165±.150</td><td>.229±.306</td><td colspan="2">.300±.129 .298±.121 .338±.002</td></tr><tr><td>image</td><td>3</td><td>.340±.020</td><td>.369±.134</td><td>.387±.147</td><td colspan="2">.542±.191 .549±.174 .398±.034</td></tr><tr><td>scene</td><td>4</td><td>.329±.020</td><td>.340±.078</td><td>.349±.082</td><td colspan="2">.497±.089 .523±.118 .428±.022</td></tr><tr><td>yeast</td><td/><td>.250±.019</td><td>.248±.019</td><td>.252±.018</td><td colspan="2">.257±.017 .263±.027 .408±.023</td></tr><tr><td>enron</td><td/><td>.307±.039</td><td>.411±.101</td><td>.351±.040</td><td colspan="2">.494±.039 .498±.038 .495±.019</td></tr><tr><td>corel5k</td><td/><td>.697±.020</td><td>.835±.025</td><td>.721±.035</td><td colspan="2">.784±.029 .787±.024 .809±.015</td></tr><tr><td>eurlex_dc eurlex_sm</td><td>10</td><td>.305±.016 .341±.003</td><td>.390±.016 .350±.014</td><td>.374±.014 .360±.015</td><td colspan="2">.707±.014 .518±.011 .342±.008 .506±.031 .542±.018 .340±.005</td></tr><tr><td>delicious</td><td/><td>.372±.012</td><td>.366±.015</td><td>.414±.018</td><td colspan="2">.401±.015 .399±.013 .450±.009</td></tr><tr><td>tmc2007</td><td/><td>.200±.005</td><td>.194±.029</td><td>.267±.018</td><td colspan="2">.235±.019 .236±.019 .388±.006</td></tr></table>

Comparison of PML-GAN with its ablation variants in terms of coverage and one error. The best results are in bold font.

Table content: <table><tr><td>Data set</td><td>PML-GAN CLS-GEN CLS-GAN</td><td>CLS-ML</td><td>PML-GAN CLS-GEN</td><td>CLS-GAN</td><td>CLS-ML</td></tr><tr><td colspan="2">Coverage (the smaller, the better)</td><td/><td colspan="2">One error (the smaller, the better)</td><td/></tr><tr><td colspan="6">music_emotion .407±.008 .410±.008 .408±.006 .410±.005 .444±.027 .460±.022 .446±.000 .463±.024</td></tr><tr><td>music_style</td><td colspan="5">.202±.011 .204±.004 .204±.008 .207±.008 .346±.022 .359±.014 .351±.017 .361±.017</td></tr><tr><td>mirflickr</td><td colspan="5">.227±.010 .230±.017 .227±.014 .242±.013 .337±.059 .388±.092 .370±.085 .405±.094</td></tr><tr><td>image</td><td colspan="5">.211±.014 .214±.011 .212±.011 .220±.011 .340±.020 .344±.023 .360±.032 .368±.024</td></tr><tr><td>scene</td><td colspan="5">.124±.010 .127±.008 .125±.006 .128±.010 .329±.020 .331±.023 .341±.022 .348±.018</td></tr><tr><td>yeast</td><td colspan="5">.502±.014 .509±.016 .507±.013 .509±.023 .250±.019 .260±.027 .251±.029 .254±.024</td></tr><tr><td>enron</td><td colspan="5">.402±.015 .410±.019 .408±.016 .414±.015 .307±.039 .309±.029 .321±.027 .333±.028</td></tr><tr><td>corel5k</td><td colspan="5">.384±.011 .394±.008 .396±.017 .399±.011 .697±.020 .699±.017 .698±.020 .700±.016</td></tr><tr><td>eurlex_dc</td><td colspan="5">.063±.003 .082±.011 .063±.003 .085±.005 .305±.016 .312±.024 .307±.013 .312±.017</td></tr><tr><td>eurlex_sm</td><td colspan="5">.172±.006 .174±.004 .176±.005 .178±.004 .341±.003 .343±.007 .350±.015 .351±.007</td></tr><tr><td>delicious</td><td colspan="5">.564±.008 .566±.008 .564±.006 .628±.005 .372±.012 .373±.010 .373±.008 .375±.011</td></tr><tr><td>tmc2007</td><td colspan="5">.181±.006 .184±.004 .181±.004 .185±.004 .200±.005 .201±.005 .203±.007 .205±.004</td></tr><tr><td colspan="5">A.5 VARIANT OF PML-GAN WITH CLASSIFICATION LOSS ON GENERATED DATA</td><td/></tr></table>

Comparison results for PML-GAN and PML-GAN in terms of six evaluation metrics. The best results are presented in bold font. The average number of candidate labels is presented under column "avg.#C.Ls".

Table content: <table><tr><td>Data set</td><td colspan="2">avg.#C.Ls PML-GAN</td><td>PML-GAN</td><td>PML-GAN</td><td>PML-GAN</td></tr><tr><td/><td/><td colspan="2">Hamming loss (the smaller, the better)</td><td colspan="2">Ranking loss (the smaller, the better)</td></tr><tr><td>music_emotion</td><td>5.29</td><td>.197±.004</td><td>.200±.003</td><td>.243±.009</td><td>.239±.009</td></tr><tr><td>music_style</td><td>6.04</td><td>.116±.005</td><td>.115±.002</td><td>.141±.001</td><td>.145±.006</td></tr><tr><td>mirflickr</td><td>3.35</td><td>.171±.002</td><td>.171±.004</td><td>.127±.014</td><td>.122±.025</td></tr><tr><td>image</td><td>3</td><td>.202±.008</td><td>.200±.008</td><td>.192±.015</td><td>.191±.010</td></tr><tr><td>scene</td><td>4</td><td>.137±.007</td><td>.133±.007</td><td>.131±.014</td><td>.119±.008</td></tr><tr><td>yeast</td><td/><td>.216±.005</td><td>.216±.005</td><td>.193±.008</td><td>.194±.007</td></tr><tr><td>enron</td><td/><td>.186±.004</td><td>.186±.004</td><td>.179±.013</td><td>.178±.012</td></tr><tr><td>corel5k eurlex_dc</td><td>10</td><td>.119±.001 .044±.001</td><td>.113±.001 .044±.001</td><td>.293±.012 .065±.003</td><td>.298±.014 .064±.006</td></tr><tr><td>eurlex_sm</td><td/><td>.082±.001</td><td>.082±.001</td><td>.119±.005</td><td>.119±.007</td></tr><tr><td>delicious</td><td/><td>.248±.003</td><td>.248±.003</td><td>.256±.006</td><td>.257±.005</td></tr><tr><td>tmc2007</td><td/><td>.085±.001</td><td>.086±.001</td><td>.071±.003</td><td>.069±.003</td></tr><tr><td/><td/><td colspan="4">Average precision (the larger, the better) Macro-averaging AUC (the larger, the better)</td></tr><tr><td>music_emotion</td><td>5.29</td><td>.621±.012</td><td>.621±.014</td><td>.726±.006</td><td>.722±.006</td></tr><tr><td>music_style</td><td>6.04</td><td>.734±.015</td><td>.728±.009</td><td>.706±.016</td><td>.731±.009</td></tr><tr><td>mirflickr</td><td>3.35</td><td>.771±.026</td><td>.781±.046</td><td>.873±.002</td><td>.871±.003</td></tr><tr><td>image</td><td>3</td><td>.774±.013</td><td>.775±.012</td><td>.798±.015</td><td>.790±.010</td></tr><tr><td>scene</td><td>4</td><td>.794±.014</td><td>.808±.009</td><td>.874±.012</td><td>.882±.008</td></tr><tr><td>yeast</td><td/><td>.733±.008</td><td>.735±.009</td><td>.666±.026</td><td>.666±.026</td></tr><tr><td>enron</td><td/><td>.665±.020</td><td>.670±.024</td><td>.668±.018</td><td>.668±.018</td></tr><tr><td>corel5k eurlex_dc</td><td>10</td><td>.440±.014 .797±.009</td><td>.435±.013 .800±.012</td><td>.628±.007 .872±.009</td><td>.628±.007 .882±.007</td></tr><tr><td>eurlex_sm</td><td/><td>.722±.002</td><td>.720±.010</td><td>.827±.008</td><td>.831±.005</td></tr><tr><td>delicious</td><td/><td>.630±.007</td><td>.630±.006</td><td>.712±.004</td><td>.711±.003</td></tr><tr><td>tmc2007</td><td/><td>.820±.004</td><td>.820±.004</td><td>.886±.002</td><td>.886±.002</td></tr></table>

Win/tie/loss counts of pairwise t-test (with p < 0.05 ) about PML-GAN vs PML-GAN over all 49 PML variant datasets with different numbers of candidate labels. average precision, macro-averaging AUC, coverage, one error)Zhang & Zhou (2014) are reported in Table

Table content: <table><tr><td>Hamming loss</td><td>Ranking loss</td><td>Average precision</td><td>Macro-averaging</td><td>Total</td></tr><tr><td>0/49/0</td><td>0/49/0</td><td>0/49/0</td><td>00/48/1</td><td>0/195/1</td></tr></table>

Peer review (iclr2020):
The motivation of this paper is to handle noise candidate multi-labels by co-training two networks. The work trains the disambiguation network, which learns to predict the probability of each class label being the additive irrelevant label and then is used to get the disambiguated label confidence vector, and the prediction network, which learns to predict the probability of the disambiguated labels. The additional adversarial loss and generation loss is aimed to enhance the label disambiguation by learning the mapping from the disambiguated labels to the input features. Pros: By minimizing the disagreement between the confidence of being the disambiguated label and the predicted probability of each label, the partial multi-label learning gets better results. The experiments in this paper are complete and thorough. The authors have tested the model in many datasets and designed the ablation study to verify the effect of each loss. And the proposed model achieved the state-of-art results. Cons: However, the effect of the adversarial loss and the generation loss is doubtful because: 1) The mapping from the labels to the input features is hard to learn since the label space does not contain the complete information of input features. It is doubtful that the generation is helpful. 2) The results of the ablation study do not show consistently significant improvements. 3) In the appendix, the variant of PML-GAN, which considers an auxiliary classification loss on the generated data, has little improvement compared to PML-GAN. The author claims that it is because of sufficient training data. What if considering the insufficient training data and testing the variant of PML-GAN? I think it can somehow verify the effect of the generation. About the writing of this paper, the motivation of the work is not clearly defined. Although we can get what the work was done, we cannot get why the work did this.