Multi-Level Generative Models for Partial Label Learning with Non-random Label Noise
Authors: Yan Yan
Yuhong Guo
Venue:  (2021)

Abstract: Partial label (PL) learning tackles the problem where each training instance is associated with a set of candidate labels that include both the true label and irrelevant noise labels. In this paper, we propose a novel multi-level generative model for partial label learning (MGPLL), which tackles the PL problem by learning both a label level adversarial generator and a feature level adversarial generator under a bi-directional mapping framework between the label vectors and the data samples. MGPLL uses a conditional noise label generation network to model the non-random noise labels and perform label denoising, and uses a multi-class predictor to map the training instances to the denoised label vectors, while a conditional data feature generator is used to form an inverse mapping from the denoised label vectors to data samples. Both the noise label generator and the data feature generator are learned in an adversarial manner to match the observed candidate labels and data features respectively. We conduct extensive experiments on both synthesized and real-world partial label datasets. The proposed approach demonstrates the state-of-the- art performance for partial label learning.

Under review as a conference paper at ICLR 2021
MULTI-LEVEL GENERATIVE MODELS FOR PARTIAL LABEL LEARNING WITH NON-RANDOM LABEL NOISE
Anonymous authors Paper under double-blind review
ABSTRACT
Partial label (PL) learning tackles the problem where each training instance is as- sociated with set of candidate labels that include both the true label and irrelevant noise labels. In this paper, we propose a novel multi-level generative model for partial label learning (MGPLL), which tackles the PL problem by learning both label level adversarial generator and a feature level adversarial generator under a bi-directional mapping framework between the label vectors and the data sam- ples. MGPLL uses a conditional noise label generation network to model the non- random noise labels and perform label denoising, and uses multi-class predictor to map the training instances to the denoised label vectors, while conditional data feature generator is used to form inverse mapping from the denoised label vectors to data samples. Both the noise label generator and the data feature gener- ator are learned in an adversarial manner match the observed candidate labels and data features respectively. We conduct extensive experiments on both synthe- sized and real-world partial label datasets. The proposed approach demonstrates the state-of -fte--o--hh-rrr the-art performance for partial label learning.
INTRODUCTION
Partial label (PL) learning is weakly supervised learning problem with ambiguous labels
(Hüllermeier & Beringer (006) Zeng et al. (203)), where each training instance is assigned a set of
candidate labels, among which only one the true label. Since typically difficult and costly to annotate instances precisely, the task of partial label learning naturally arises in many real-world
learning scenarios, including automatic face naming (Hüllermeier & Beringer 2006 Zeng et al., 2013). and web mining (Luo & Orabona 2010).
As the true label information is hidden in the candidate label set, the main challenge of PL lies in identifying the ground truth labels from the candidate noise labels, aiming to learn a good pre- diction model. Some previous works have made effort on adjusting the existing effective learning techniques to directly handle the candidate label sets and perform label disambiguation implicitly Gong et al., 20188 Nguyen & Caruana 2008 Wu & Zhang 2018). These methods are good at ex- ploiting the strengths of the standard classification techniques and have produced promising results on PL learning. Another set of works pursue explicit label disambiguation by trying to identify the true labels from the noise labels in the candidate label sets. For example, the work in (Feng & An 2018) tries to estimate the latent label distribution with iterative label propagations and then induce prediction model by fitting the learned latent label distribution. Another work in (Lei & An 2019) exploits a self-training strategy to induce label confidence values and learn classifiers in an alter- native manner by minimizing the squared loss between the model predictions and the learned label confidence matrix. However, these methods suffer from the cumulative errors induced in either the separate label distribution estimation steps or the error-prone label confidence estimation process. Moreover, all these methods have a common drawback: they automatically assumed random noise in the label space that is, they assume the noise labels are randomly distributed in the label space for each instance. However, in real world problems the appearance of noise labels is usually depen- dent on the target true label. For example, when the object contained in an image is a "computer", noise label "TV" could be added due to recognition mistake image ambiguity, but it is less likely to annotate the object as "lamp" or "curtain", while the probability of getting noise labels such "tree" or "bike" is even smaller.

---

Under review as a conference paper at ICLR 2021
In this paper, we propose a novel multi-level adversarial generative model, MGPLL, for partial label learning. The MGPLL model comprises of conditional data generators at both the label level and feature level. The noise label generator directly models non-random appearances of noise labels conditioning on the true label by adversarially matching the candidate label observations, while the data feature generator models the data samples conditioning on the corresponding true labels by adversarially matching the observed data sample distribution. Moreover, prediction network is incorporated to predict the denoised true label of each instance from its input features, which forms inverse mappings between labels and features, together with the data feature generator. The learning of the overall model corresponds to a minimax adversarial game, which simultaneously identifies true labels of the training instances from both the observed data features and the observed candidate labels, while inducing accurate prediction networks that map input feature vectors to (denoised) true label vectors. To the best of our knowledge, this is the first work that exploits multi-level generative models to model non-random noise labels for partial label learning. We conduct extensive experiments on real-world and synthesized PL datasets. The empirical results show the proposed MGPLL achieves the state-of-the-ar PL. performance.
2 RELATED WORK
Partial label (PL) learning a popular weakly supervised learning framework (hou 2018) in many real-world domains, where the true label of each training instance is hidden within a given candidate label set. The challenge of PL learning lies in disambiguating the true labels from the candidate label sets to induce good prediction models.
One strategy towards PI. learning is to adjust the standard learning techniques and implicitly disam- biguate the noise candidate labels through the statistical prediction pattern of the data. For example, with the maximum likelihood techniques, the likelihood of each PL training sample can be defined over its candidate label set instead of its implicit ground-truth label (Jin & Ghahramani 2003 Liu & Dietterich 2012). For the k-nearest neighbor technique, the candidate labels from neighbor instances can be aggregated to induce the final prediction on test instance (Hällermeier & Beringer 2006 Gong et al. 2018 Zhang & Yu 2013). For the maximum margin technique, the classification mar- gin can be defined over the predictive difference between the candidate labels and the non-candidate labels for each PL training sample (Nguyen & Caruana 2008 Yu & ZZang 2016). For the boost- ing technique, the weight of each PL training instance and the confidence value of each candidate label being ground-truth label can be refined via each boosting round (Tang & Zhang] (017)). For the error-correctin output codes (ECOC) technique, multiple binary classifier corresponding to the ECOC coding matrix are built based on the transformed binary training sets (Zhang et al.) 2017). For the binary decomposition techniques, a one-vs-one decomposition strategy has been adopted to address PL learning by considering the relevance of each label pair (Wu & Zhang 2018).
Recently, there have been increasing attentions in designing explicit feature-aware disambiguation strategies (Feng & An 2018. Xu et al., 2019a] Feng & An 2019; Wang et al.] (009a)) The authors of (Feng& & An | 2018) attempt to refine the latent label distribution using iterative label propagations and then induce predictive model based on the Iearned latent label distribution. However, the la- tent label distribution estimation in this approach can be impaired by the cumulative error induced in the propagation process, which can consequently degrade the PL learning performance, especially when the noisy labels dominate. Another work in (Lei & An 2019) tries to refine the label con- fidence values with self-training strategy and induce the prediction model over the refined label confidence scores via alternative optimization. Its estimation error on confidence values however can negatively impact the coupled partial label classifier due to the nature of alternative optimiza- tion. A recent work in (Yao et al. (0200) proposes to address the PL learning problem by enhancing the representation ability via deep features and improving the discrimination ability through mar- gin maximization between the candidate labels and the non-candidate labels. Another recent work in (Yan & Guo 2020) proposes to dynamically correct label confidence values with a batch-wise label correction strategy and induce robust predictive model based on the MixUp enhanced data. Although these works demonstrate good empirical performance, they are subject to one common drawback of assuming random distributions of noise labels by default, which does not hold in many real-world learning scenarios. This paper presents the first work that explicitly model non-rame non-random noise labels for partial label learning.
2

---

Under review as a conference paper at ICLR 2021
Gn (F(x),))
Gx
Vn
Dn
Dx
2p $
F(x)
g
Label Level
Gn (Zp
Feature Level
The proposed MGPLL model. It consists of an adversarial generative model at the label level with a conditional noise label generator G,, and discriminator D,,, and an adversarial gener- ative model at the feature level with conditional sample generator GN and a discriminator D,. The prediction network F builds connections between these two level generative models while providing an inverse mapping for Gz.
Figure 1:
PL learning related other types of weakly supervised learning problems, including noise label learning (NLL) (Xu et al. 2019b] [Thekumparampil et al., 2018 Arazo et al. 2019) and partial multi-label learning (PML) (Wang et al., 2019b] Fang & Zhang 2019 Xie & Huang 2018), but addresses different problems from them. The main difference between the PL learning and the other two well-established learning problems lies in the assumption on the label information provided by the training samples. Both PL learning and NLL aim to induce multi-class prediction model from the training instances with woise-corrupter labels. However NLL assumes the true labels on some training instances replaced by the noise labels, while PL assumes the true-label coexists with the noise labels in the candidate label set of each training instance. Hence the off-the-shelf NI.L learning methods cannot be directly applied solve the PL learning problem. Both PL learning and PML learn from training samples with ambiguous candidate label sets, which contains the true labels and additional noise labels. But PL learning addresses a multi-class learning problem where each candidate label set contains only one true label, while PML learning addresses multi learning problem where each candidate label set contains all but unknown number of true labels.
The Wasserstein Generative Adversarial Networks (WGANs) (Arjovsky et al. 2017), which per- form minimax adversarial training with generator and discriminator, is a popular alternative to the standard GANs (Goodfellow et al. 2014b) due to its effective and stable training of GANs. Dur- ing the past few years, WGANs have been proposed as a successful tool for various applications, including adversarial sample generation (Zhao et al. 2017). domain adaption (Dou et al.) 2018)) and learning with noisy labels (Chen et al. 2018). This paper presents the first work that exploits WGAN model non-random noise labels for partial label learning.
PROPOSED APPROACH
Given a partial label training set S yx) where Rd is a d-dimensional feature vector for the i-th instance, and y; E {0, 1)2 denotes the candidate label indicator vector associated with x,, which has multiple 1 values corresponding to the ground-truth label and the additional noise labels, the task of PL learning is to learn a good multi-class prediction model from S. In real world scenarios, the irrelevant noise labels are typically not presented in a random manner, but rather correlated with the ground-truth label. In this section, we present a novel multi-level generative model for partial label learning, MGPLL, which models non-random noise labels using an adver- sarial conditional noise label generator, and builds connections between the denoised label vectors and instance features using a label-conditioned feature generator and label prediction network.
The overall model learning problem corresponds to minimax adversarial game, which conducts multi-level generator learning by matching the observed data both the feature and label spaces, while boosting the correspondence relationships between features and labels to induce accurate multi-class prediction model.
Figure [ Tillustrates the proposed multi-level generative model, MGPLL, which attempts address the partial label learning problem from both the label level and feature level under a bi-directional mapping framework. The MGPLL model comprises five component networks: the conditional noise
3

---

Under review as a conference paper at ICLR 2021
label generator, G, which models the noise labels conditioning on the ground- truth label at the label level; the conditional data generator, Gg. which generates data samples at the feature level condition- ing on the denoised label vectors; the discriminator, Da, which separates the generated candidate label vectors from the observed candidate label vectors in the real training data; the discriminator, D2. which separates the generated samples from the real data in the feature space; and the prediction network, F. which predicts the denoised label for each sample from its input features. Zp denotes one-hot label indicator vector sampled from a multinomial distribution P'z. The conditional noise label generator G,, induces the denoised prediction target for the prediction network F, while the conditional data generator G learns an inverse mapping the feature level that maps the denoised label vectors in the label space to the data samples in the feature space. Below we present the details of the two level generations and the overall learning algorithm.
3.1 CONDITIONAL NOISE LABEL GENERATION
The key challenge of partial label learning lies in the fact that the ground-truth label is hidden among the noise labels in the given candidate label set. As aforementioned, in real world partial label learning problems, the presence of noise labels typically does not happen at random, but rather correlates with the ground-truth labels. Hence we propose conditional noise label generation model to model the appearances of the target-label dependent noise labels by adversarially matching the observed candidate label distribution in the training data, aiming to help identify the true labels later.
Specifically, given noise value sampled from uniform distribution dicator vector sampled from a multinomial distribution Pz. we use noise label generator Gn, (z, e) generate a noise label vector conditioning on the true label z, which can be combined with z in rectified sum, "0" to form a generated candidate label vector y, such that
P. and one-hot label in-
y = Gn (z, e) @ z = min(G.. (2, + z. 1).
(1)
Here we assume the generator G. generates non-negative values. We then adopt the adversarial learning principle to learn such a noise label generation model by introducing discriminator D, (y), which is a two-class classifier and predicts how likely a given label vector y comes from the real data instead of the generated data. By adopting the adversarial loss of the Wasserstein Generative Adversarial Network (WGAN), our adversarial learning problem can be formulated as the following minimax optimization problem:
min max Gn D2
Cadv Dr) s (y:) Ezn.ng (G, c) (z) c-P,
Here the discriminator D, attempts to maximally distinguish the generated candidate label vectors from the observed candidate label indicator vectors in the real training data, while the generator G, tries to generate noise label vectors and hence candidate label vectors that are similar to the real data in order to maximally confuse the discriminator Dn, By playing a minimax game between the generator Gn and the discriminator D.. the adversarial learning is expected to induce generator G such that the generated candidate label distribution can match the observed candidate label distribution in the training data. We adopt the training loss of the WGAN here, as WGANs can overcome the mode collapse problem and have improved learning stability comparing to the standard GAN models (Arjovsky et af., 2017).
Note although the proposed generator G, is designed to model true-label dependent noise labels, it can be easily modified to model random noise label distributions by simply dropping the label vector input from the generator, which yields (t)
3.2 PREDICTION NETWORK
The ultimate goal of partial label learning is to learn an accurate prediction network F. To train a good predictor, need to obtain denoised labels on the training data. For candidate label indicator vector y, if the noise label indicator vector yn is given, one can simply perform label denoising as follows to obtain the corresponding true label vector z:
y θ yn max(y yn. 0)
Here the rectified minus operator "e" is introduced to generalize the standard minus "_" operator into the non-ideal case, where the noise label indicator vector yn is not properly contained in the candidate label indicator vector.
4

---

Under review as a conference paper at ICLR 2021
The generator G., presented in the previous section provides mechanism to generate noise labels and denoise candidate label sets, but requires the true target label vector as its input. We propose to use the outputs of the prediction network F to approximate the target true label vectors of the training data for the purpose of denoising the candidate labels with G.. while using the denoised labels as the prediction target for F. Specifically, with the noise label generator G, and the predictor F, we perform partial label learning by minimizing the following classification loss on the training data S:
min F.G.
C,(F, = E Sn (5,0000
Le (F(xs)) Y e (F(())))
(4)
Although in the ideal case, the output vectors of G. and F would be indicator label vectors, it error-prone and difficult for neural networks to output discrete values. To pursue more reliable predictions and avoid overconfident outputs, we use G, and F to predict the probability of each class label being noise label and the ground-truth label respectively. Hence the loss function E,(-,) in (4. above denotes a mean square emor loss between the predicted probability of each label being the true label (through F) and its denoised confidence of being a ground-truth label (through G.).
3.3 CONDITIONAL FEATURE LEVEL DATA GENERATION
With the noise label generation model and the prediction network above, the observed training data in both the label and feature spaces are exploited to recognize the true labels and induce good prediction models. Next, we incorporate a conditional data generator Gz e) at the feature level to map (denoised) label vectors in the label space into instances in the feature space, aiming to further strengthen the mapping relations between data samples and the corresponding labels, enhance label denoising and hence improve the partial label learning performance. Specifically, given noise value e sampled from a uniform distribution P. and a one-hot label vector z sampled from a multinomial distribution Pz. Gz(z,e) E) generates an instance in the feature space that is corresponding to label z. Given the training label vectors in S denoised with Gn, the data generator Gg is also expected to regenerate the corresponding training instances in the feature space. This assumption can be captured using the following generation loss:
Eg Gn, Gz) = -EG lg (2i, 11.22-Pc
(5)
with
Z, @ G. (F(x),,
where z, denotes the denoised label vector for the i-th training instance, and (. is a mean square error loss function.
Moreover, by introducing a discriminator Dx which predicts how likely a given instance x is real, we deploy an adversarial learning scheme to learn the generator G2 through the following minimax optimization problem with the WGAN loss:
min max Gx
Cadv (Gu, Dz) = Exx....)
Ex-Py Dz (G c-
(6)
Dr
By playing minimax game between G and Dr, this adversarial learning is expected to induce generator G that can generate samples with the same distribution as the observed training instances. Together with the generation loss in (5. we expect the mapping relation from label vectors to samples induced by G can be consistent with the observed data. Moreover, the consistency of the mapping relation induced by G, and the inverse mapping from samples label vectors through the prediction network F can be further strengthened by enforcing an auxiliary classification loss on the generated data:
Le (e, Gz) Ex-P, la (F(G ()) cMPe
where (...)) can be a cross-entropy loss between the label prediction probability vector and the sampled true label indicator vector.
3.4 LEARNING THE MGPLL MODEL
By integrating the classification loss in Eq.4 the adversarial losses in Eq. and Eq. the gener- ation loss in Eq. (5) and the auxiliary classification loss in Eq. (7) together, MGPLL learning can be
5

---

Under review as a conference paper at ICLR 2021
Table 1: Win/tie/loss counts of pairwise t-test (at 0.05 significance level) between MGPI.L and each comparison method.
MGPLL vs
SURE
PALOC
CLPL
PL-SVM
PL-KNN
varying p r
25/14/3
32/10/0
36/6/0
37/5/0
37/5/0
varying p 2
27/13/2
33/9/0
33/9/0
38/4/0
35/7/0
varying 3)
26/14/2
32/10/0
32/10/0
36/6/0
34/8/0
varying 12, 1]
25/17/0
30/12/0
32/10/0
35/7/0
33/9/0
Total
103/58/7
127/41/0
133/35/0
146/22/0
139/29/0
formulated the following min-max optimizationn problem:
min G..Gu.P Da.Da
G.)+ Cado (Gn:; Dn) tafa (G,, Dz) D2)+ Gu (F,
where 8 and y are trade-off hyperparameters. The learning of the overall model corresponds to minimax adversarial game. We develop batch-based stochastic gradient descent algorithm solve it by conducting minimization over {Gn. Ga, F) and maximization over (Dn, Dz alternatively. The overall training algorithm is provided in the appendix.
EXPERIMENT
We conducted extensive experiments both controlled synthetic PL datasets and real-world PL. datasets to investigate the empirical performance the proposed model. In this section, we present our experimental settings, comparison results and discussions.
4.1 EXPERIMENT SETTING
The synthetic datasets are generated from six UCI datasets, ecoli, deter, vehicle, segment, satimage and letter. From each UCI dataset, we generated synthetic PL. datasets using three con- trolling parameters p. and c, following the controlling protocol in previous studies (Wu & Zhang, 2018 Xu et al., 2019a| Lei An 2019). Among the three parameters, p controls the proportion of instances that have noise candidate labels, r controls the number of false positive labels, and controls the probability of a specific false positive label co-occurring with the true label. Under different parameter configurations, multiple PL variants can be generated from each UCI dataset. Given that both random noise labels and target sabel-dependent noise labels may exist in real-world applications, considered two types of settings. In the first type of setting, noise labels with the following three groups of configurations: (I) = 1. E (0.1, .......... 0..7); (II) 2. p E {0.1, 0.2, 0.7]: and (III) setting, we consider the target tabel-dependent noise labels with the following configuration: (IV) p r 1, {0.1, 0.2...................... 0.7). In total, the four groups of configurations provide us 168 (28 configurations x 6 UCI datasets) synthetic PL datasets.
Datasets
consider random
p {0.1, -.. 0.77.. 0...0...00.7...
In the second type of
We used five world PL datasets that are collected from several application domains. including FG-NET (Panis & Lanitis 2014) for facial age estimation, Lost (Cour et al., 201T). Yahoo! News (Gillaumin al.] 2010) for automatic face naming in images or videos, MSRCv2 (Dietterich &
Bakiri 1994) for object classification, and BirdSong (Briggs et al. (012) for bird song classification.
compared the proposed MGPLL approach with the following meth- ods, each configured with the suggested parameters according to the respective literature: PL.-KNN (Hüllermeier Beringer (0066). PL-SVM (Nguyen Caruana 2008), CLPL (Cour et al. 201
Comparison Methods
PALOC (Wu & Zhang 2018), and SURE (Lei & An 2019).
4.2 RESULTS ON SYNTHETIC PI. DATASETS
We conducted experiments on two types of synthetic PL datasets generated from the UCI datasets, with random noise labels and target lahel-dependent noise labels, respectively. For each PI. dataset,

---

Under review as a conference paper at ICLR 2021
MGPLL
SURE
PALOC
CLPL
PL-VV
PL-KNN
0.90
1.00
0.95
1.00
0.90
0.90
0.5
0.80
6.70
0400
0.40
0.75
0.60
0.70
0.50
0.70
04 (co-oorartt oxpling Linct
0.65
03
0.6 0.7
mebiii bad!
(a) vehicle
(b) segment
satimage
(d) letter
Test accuracy of each comparison method as increases from 0.1 to 0.7 (with 100% partially labeled examples (p 1] and one false positive candidate label [r 1)).
Figure 2:
Test accuracy (td) of each comparison method on the real-world PL datasets. / indicates whether MGPLL is statistically superior/inferion to the comparison method on each dataset (pairwise t-test 0.05 significance level).
Table 2:
MGPLL
SURE
PALOC
CLPL
PL-SVM
PL-KNN
FG-NET
0.079
0.068
10.011
0.063.1..02
0.063
0.038 0.038 +0.025
FG-NET(MAE3)
0.458 0..1620.027
00.04
0.445
0.458
0.356_0.022
1..6940.0455
FG-NET(MAES)
0, 0.626=o.0m
0.615=0.019
0.609
0.596
017
0.479=u.016
0.438
Lost
0. 0.798 033
0.780 036
0.629
0.742
0.729
0.424
MSRCv2
0.533 021
0.481 =0.030
0.413 0.419++0.042
20.011
0.161
0.4-8
BirdSong
0.748
0.728
000024
0.71 1 1.0016
0. 10.021
0.660
0.61 0.614
Yahoo! News
0.678 40.088 0.644
0.462
0.629
0.457
ten-fold cross-validation is performed and the average test accuracy results are recorded. Figure 2 presents the comparison results for the configuration setting (IV) on four datasets. We can see that the proposed MGPLL consistently outperforms all the other methods.
To statistically study the significance of the performance gains achieved by MGPLL over the other comparison methods, we conducted pairwise t-test at 0.05 significance level based on the compari- son results ten-fold cross-validation over all the 168 synthetic PL datasets obtained from all the different configuration settings. The detailed win/tie/loss counts between MGPLL and each com- parison method are reported in Table T From the results, we have the following observations: (1) MGPLL achieves superior at least comparable performance against PALOC, CLPL, PL-SVM and PL-KNN in all cases, which is not easy given the comparison methods have different strengths across different datasets. (2) MGPLL significantly outperforms PALOC, CLPLL, PL-SVM and PL-KKNN in 75.6%, 79.1 1 %, 86.9% and 82.7% of the cases respectively, and produces ties in the remaining cases. (3) MGPLL significantly outperforms SURE in 61.3% of the cases, achieves comparable perfor- mance with SURE in 34.5% of the cases, while being outperformed by SURE in only 4.2% of the cases. (4) On the PL datasets with target aabel-dependent noise labels, we can see that MGPLL significantly outperforms SURE PALOC, CLPL, PL-SVM, PL-KNN in 59.5%, 71.4%, 76.2%, 83.3%. 78.6% of the cases respectively. (5) It is worth noting that MGPLL is never significantly outperformed by any comparison method on datasets with label-dependent noise labels. In sum- mary, these results on the controlled PI. datasets clearly demonstrate the effectiveness of MGPLI. for partial label learning under different settings.
4.3 RESULTS ON REAL-WORLL PI. DATASETS
We compared the proposed MGPLL method with the comparison methods on five real-world PL datasets. For each dataset, ten-fold roos-validattin is conducted. The mean test accuracy and the standard deviation results are reported in Table 2 Moreover, statistical pairwise t-test at 0.05 significance level conducted to compare MGPLL with each comparison method based on the results of ten-fold cross-validation. The significance results are indicated in Table 2 as well. Note that the average number of candidate labels (avg.#CLs) of FG-NET dataset is quite large, which causes poor performance for all the comparison methods. For better evaluation of this facial age estimation task, we employ the conventional mean absolute error (MAE) (Zhang et al.) 2016) to conduct two extra experiments. Two extra test accuracies are reported on the FG-NET dataset where
7

---

Under review as a conference paper at ICLR 2021
Table 3: Comparison results of MGPLL and its five ablation variants.
MGPLL
CLS-w/o-ddv
CLS-w/o-adv
CLS-w/o-au CLS-w/og
CLS
FG-NET
0.029
0.061 +0.024
0.072
0.068
0.076
0.057
FG-NET(MAE3)
0.468:10.027
0. 200
0.451
0.436440 038
0.456440.08
0.420+0.42 420
FG-NET(MAES)
0.62640.022
0, 00.05
0, 0.605++0.05
0,590
0.612
0.570 034
Lost
0.798
0.6233 0.007
0. 0.754 032
0,687 026
0.782
0.609
MSRCv2
0. 00.772=0.088 0.533=20.02
48.0+00.03
0.499700.000
0.526
0.450
BirdSong
0.748=.0.220
0. 0.728 00.000
0.732 10.011
0.716
0..42
0.674
Yahoo! News
0.678.40.008
4..45
0.675+0.000
0.648
0.671 +0.012
0.61 0.610
a test sample is considered to be correctly predicted if the difference between the predicted age and the ground-truth age less than years (MAE3) or 5 years (MAE5). From Table 2 we have the following observations: (1) Comparing with all the other five PI. methods, MGPL.I consistently produces the best results on all the datasets, with remarkable performance gains in many cases. For example, MGPLL outperforms the best alternative comparison methods by 5.2%. 3.4% and 2.0% on MSRCv2, Yahoo! News and Birdsong respectively. (2) Out of the total 35 comparison cases (5 comparison methods 7 datasets), MGPLL significantly outperforms all the comparison methods across 77.1% 1% of the cases, and achieves competitive performance in the remaining 22.9% of cases. (3) It is worth noting that the performance of MGPLL is never significantly inferior to any other comparison method. These results again validate the efficacy of the proposed method.
4.4 ABLATION STUDY
The objective function of MGPLL contains five loss terms: classification loss, adversarial loss the label level, adversarial loss at the feature level, generation loss and auxiliary classification loss. To assess the contribution of each part, we conducted an ablation study by comparing MGPLL with the following ablation variants: (1) CLS-w/o-adyn, which drops the adversarial loss at the label level. (2)
which drops the adversarial loss at the feature level. (3) CLS- w/o-g, which drops the generation loss. (4) CLS-w/o-uux, which drops the auxiliary classification loss. (5) CLS, which only uses the classification loss by dropping all the other loss terms. The comparison results are reported in Table 3 We can see that comparing to the full model, all five variants produce inferior results in general and have performance degradations to different degrees. This demonstrates that the different components in MGPLL all contribute to the proposed model to some extend. From T we can also see that the variant CL.S-w/o-ade performance degradation by dropping the adversarial loss at the label level, while the variant CLSS w/o-aux has a small performance degradation by dropping the auxiliary classification loss. This makes sense as by dropping the adversarial loss for learning noise label generator, the generator can produce poor predictions and seriously impact the label denoising of the MGPLL model. This suggests that our non-random noise label generation through adversarial learning is very effective and important component for MGPLL. For CLS-w/o-aux, as we have already got the classification loss on real data, it reasonable to see that the auxiliary classification loss on generated data can help but is not critical. Overall, the ablation results suggest that the proposed MGPLL is effective.
has a relatively larger
CONCLUSION
In this paper, we proposed a novel multi-level generative model, MGPLL, for partial label learning. MGPLL uses a conditional label level generator to model the target label dependent non-random noise label appearances, which directly performs candidate label denoising, while using condi- tional feature level generator to generate data samples from denoised label vectors. Moreover, a prediction network is incorporated to predict the denoised true label of each instance from its input features, which forms bi-directional inverse mappings between labels and features, together with the data feature generator. The adversarial learning of the overall model simultancously identifies true labels of the training instances from both the observed data features and the observed candidate labels, while inducing accurate prediction networks that map input feature vectors to (denoised) true label vectors. We conducted extensive experiments on real-world and synthesized PL datasets. The proposed MGPLL model demonstrates the otate-of-the-ar PL. performance.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. arXiv preprint arXiv:1904. 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
Forrest Briggs, Xiaoli Fern, and Raviv Raich. Rank-loss support instance machines for miml instance annotation. In Proceedings of the ACM SIGKDD international conference on Knowledge discovery and data mining (KDD), 2012.
Jingwen Chen, Jiawei Chen, Hongyang Chao, and Ming Yang. Image blind denoising with gen- erative adversarial network based noise modeling. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 3155-3164, 2018.
Timothee Cour, Ben Sapp. and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research, 12(May):1501-153:: 2011.
Thomas G Dietterich and Ghulum Bakiri. Solving multiclass learning problems via error-corecting output codes. Journal of artificial intelligence research, 2:263-286, 1994.
Qi Dou, Cheng Ouyang, Cheng Chen, Hao Chen, and Pheng-Ann Heng. Unsupervised cross- modality domain adaptation of convnets for biomedical image segmentations with adversarial loss. arXiv preprint arXiv:1804. 10916, 2018.
Jun-Peng Fang and Min-Ling Zhang. Partial multi-label learning via credible label elicitation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3518-3525, 2019.
Lei Feng and Bo An. Leveraging latent label distributions for partial label learning. In International Joint Conference on Artificial Intelligence (IJCAI), 2018.
Lei Feng and Bo An. Partial label learning by semantic difference maximization. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
Chen Gong, Tongliang Liu, Yuanyan Tang, Jian Yang. Jie Yang, and Dacheng Tao. A regularization approach for instance-based superset label learning. IEEE transactions on cybernetics, 48(3): 967-978, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2014a.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, 2014b.
Matthieu Guillaumin, Jakob Verbeek, and Cordelia Schmid. Multiple instance metric learning from automatically labeled bags of faces. In European Conference on Computer Vision (ECCV), 2010.
Eyke Hüllermeier and Jürgen Beringer. Learning from ambiguously labeled examples. Intelligent Data Analysis, 10(5):419-49) 439, 2006.
Rong Jin and Zoubin Ghahramani. Learning with multiple labels. In Advances in neural information processing systems (NeurIPS), 2003.
Feng Lei and Bo An. Partial label learning with self-guided retraining. In AAAI Conference Artificial Intelligence (AAAI), 2019.
Liping Liu and Thomas G Dietterich. A conditional multinomial mixture model for superset label learning. Advances in neural information processing systems, pp. 548-556, 2012.
Jie Luo and Francesco Orabona. Learning from candidate labeling sets. In Advances in Neural Information Processing Systems (NeurIPS), 2010.

---

Under review as a conference paper at ICLR 2021
Nam Nguyen and Rich Caruana. Classification with partial labels. In Proceedings of the ACM SIGKDD international conference on Knowledge discovery and data mining (KDD), 2008.
Gabriel Panis and Andreas Lanitis. An overview of research activities in facial age estimation using the fg-net aging database. In European Conference on Computer Vision (ECCV), 2014.
Cai-Zhi Tang and Min-Ling Zhang. Confidence-rated discriminative partial label learning. In AAAI Conference on Artificial Intelligence (AAAI), 2017.
Kiran K Thekumparampil, Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans to noisy labels. In Advances in neural information processing systems, pp. 10271-10282 2018.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26- 31, 2012.
Deng-Bao Bao Wang, Li Li, and Min-Ling Zhang. Adaptive graph guided disambiguation for partial label learning. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 83 93, 2019a.
Haobo Wang. Weiwei Liu, Yang Zhao, Chen Zhang, Tianlei Hu, and Gang Chen. Discriminative and correlative partial multi-label learning. In Proceedings of the Twenty-Fighth International Joint Conference on Artificial Intelligence, IJCAI, pp. 10-16, 2019b.
Xuan Wu and Min-Ling Zhang. Towards enabling binary decomposition for partial label learning. In International Joint Conference on Artificial Intelligence (IJCAI), 2018.
Ming-Kun Kun Xie and Sheng-Jun Huang. Partial multi-label learning. In Thirty-Second AAAI Confer- ence on Artificial Intelligence, 2018.
Ning Xu, Jiaqi Lv, and Xin Geng. Partial label learning via label enhancement. In AAAI Conference on Artificial Intelligence (AAAI), 2019a.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L dmi: A novel information-theoretic loss function for training deep nets robust to label noise. In Advances in Neural Information Process- ing Systems, pp. 2222,,233, 2019b.
Yan Yan and Yuhong Guo. Partial label learning with batch label correction. In AAAI Conference on Artificial Intelligence (AAAI), 2020.
Yao Yao, Chen Gong, Jiehui Deng, Xiuhua Chen, Jianxin Wu, and Jian Yang. Deep diseriminative enn with temporal ensembling for ambiguously-labeled image classification. In AAAI Conference on Artificial Intelligence (AAAI), 2020.
Fei Yu and Min-Ling Zhang. Maximum margin partial label learning. In Asian Conference on Machine Learning, pp. 96,111,20 2016.
Zinan Zeng. Shijie Xiao, Kui Jia, Tsung-Har Chan, Shenghua Gao, Dong Xu, and Yi Ma. Learning by associating ambiguously labeled images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
Min-Ling Zhang and Fei Yu. Solving the partial label learning problem: An restance-basedd ap- proach. In International Joint Conference on Artificial Intelligence (IJCAI), 2015.
Min-Ling Zhang, Bin-Bin Bin Zhou, and Xu- YunYing Liu. Partial label learning via feature- disam- biguation. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discov- ery and Data Mining (KDD), 2016.
Min-Ling Zhang, Fei Yu, and Cai-Zhi Tang. Disambiguatioon-rre partial label learning. IEEE Transactions Knowledge and Data Engineering, 29(10):155-2167, 2017.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. arXiv 2017.
preprint arXiv:
Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1): 44-53, 2018.
10

---

Under review as a conference paper at ICLR 2021
Algorithm Minibatch stochastic gradient descent.
Input: S the PL training set; a, 5, y. the trade-off hyperparameters:
c: the clipping parameter; m. minibatch size.
for number of training iterations do
Sample minibatch B yi)] of m samples from Sample noise values {E1, ,,,'" Em from prior P(e).
Sample m label vectors {Z1,-.. 2; ... Zm from prior Pa.
Update Dny D2 by ascending their stochastic gradients:
VeD.. Dy
1
Σ (D Da(c + 1=1
- - -Da(GG((,)))
Perform WGAN adjustment: Op,, clip(( cc,)
Sample m noise values (. .q,,... 2m from a prior P(c).
Update Gn: Gx, F by stochastic gradient descent:
m
Dn, @2.) ( (Gs(GG((2; Bll ((,
(F.
VeGa Gz.F
m
end for
Table 4: Characteristics of the UCI datasets (left side) and the real-world PI. datasets (right side).
Dataset
#Example
#Feature
#Class
Dataset
#Example
#Feature
#Class
aVg,#CLs
ecoli
336
7
FG-NET
1,002
262
78
7.48
deter
358
23
6
Lost
1,122
108
16
2.23
vehicle
846
18
MSRCv2
1,758
48
23
3.16
segment
2310
18
BirdSong
4.998
38
13
2.18
satimage
6,345
36
7
Yahoo! News
22,991
163
219
1.91
letter
20,000
16
26
A
APPENDIX
A.I
THE OVERALL TRAINING ALGORITHM
The overall training algorithm for solving the formulated min-max optimization problem in E outlined in Algorithm
A.2
THE CHARACTERISTICS OF THE DATASETS
The characteristics of the UCI datasets and the real-world PL datasets are summaized in Table
A.3 IMPLEMENTATION DETAILS
The proposed MGPLL model has five component networks, all of which are designed as multilayer perceptrons with Leaky ReLu activation for the middle layers. The noise label generator is four- layer network with sigmoid activation in the output layer. The conditional data generator a five- layer network with tanh activation in the output layer, while batch normalization is deployed in its three middle layers. The predictor is a three-layer network with softmax activation in the output layer. Both the noise label discriminator and the data discriminator are three-layer networks without activation in the output layer. We used the RMSProp Tieleman & Hinton, 2012) optimizer in our implementation and the mini-batch size is set to 32. We selected the hyperparameters a. 3 and y from (0.001, 0.01, 0.1, 1, 10) in a heuristic way based on the classification loss value C. in the training objective function; that is, chose their values that lead to the smallest training L. loss.

---

Under review as a conference paper at ICLR 2021
MGPLL
SURE
PALOC
CTPL
SVM
PL KNN
0.90
1.00
0.90
0.85
0.95
0.70 0.1 0.2 033 0.4 0.5 0.6 0.7
0.75
01 02 0.3 0.
03
U.6 0.7
0.1 02 033 0.4 03
06 07
100 (00-0c3ETing protohilty coopting liee
(00-ocaming prbbhilly ofth cooplng stell
(00 oocooccamtg probanirty oc cooping 1800
(a) ccoli
(b) deter
(c) vehicle
1.00
0.95
1.00
0.90
0.50
0.75
0.70
0.70 0.2 0.3 04 05 Db 0.7
0.69 0.7 0.3 0.4 05 0. 0.7
0.1 0.2 0.3 044 0. D. 0.7 koo-occomii probabil cooplina
(onvoccumee prebabilt ofth couplitx aball
Eicoversmo mrataiiiity soupling line)
(d) segment
(e) satimage
(I) letter
Figure 3: Test accuracy of each comparison method changes as (co-oceurring probability of the coupling label) increases from 0.1 0.7 (with 100% partially labeled examples [p 1] and one false positive candidate label [r 1)..
MGPLL
SURE
PALOC
CLPL
PL-SVM
PL-KNN
0.90
0.90
0.88
0.80
0.80
0.84
0.75
0.88
0.82
0.70
0.80
3.1 0.2 03 04 0.5 05 077
0.84
02 0.3 044 0.5
0.7
0.65
6:1 0.2
0.3 0.4 033
0.6 07
P [proppetiia pariiillly exampias)
(peppottionn el purtilly exampies)
(pmporticn pantillyy comple))
(a) ecoli
(b) deter
(c) vehicle
0.95
0.90
0.90
0.80
0.85
0.70
0.00
0.60
0.70 0.2 0.3 u.a 0.5 8.6 d.7
0.79
0.1 0.2 0.1 0.4 0.5 ob 0.7
0.50
0.1 0.2 0.3 0.4 0.5 Bb
pipropotioon or p'onially eamplec
( /'imppotion panisity laheled comple))
Pipropatioon popunially labeled exinglic
(d) segment
(c) satimage
(f) letter
Figure 4: Test accuracy of each comparison method changes as p (proportion of partially labeled examples) increases from 0.1 to 0.7 (with one false positive candidate label [r = =1).
A.4 MORE RESULTS ON SYNTHETIC PL DATASETS
We conducted experiments on two types of synthetic PL datasets generated from the UCI datasets, with random noise labels and target lahel-dependent noise labels, respectively. For each PI. dataset, ten-fold cross-validation is performed and the average test accuracy results are recorded. First we study the comparison results over the synthetic PL datasets with target label-dependent noise labels under the PL configuration setting (IV). In this setting, specific label is selected as the coupled label that co-occurs with the ground -tuth label with probability c, and any other label can randomly chosen noisy label with probability 1-6 Figure 3 presents the comparison results for the configuration setting (IV), where e increases from 0.1 to 0.7 with = and r 1. From Figure 3 we that the proposed MGPLI produces impressive results. It consistently outperforms
12

---

Under review as a conference paper at ICLR 2021
MGPLL
SURE
PALOX
CTPL
SVM
PL KNN
0.90
1.00
0.90
0.85
0.90
0.80
0.1 0.7 0.3 0.4 0.5
055 0.7
0.70
0.7 0.5 0.4
055 0.6.
0.60 0.2 0.3 0.4 0.5
n. 0.7
?.gropomron parisily eaarpin
P-progontion of ponially eample)s
/ of pf lahelad somple)
ecoli
(b) deter
(c) vehicle
1.00
0.95
1.00
0.30
0.80
0.90
0.85
0.60
0.80
0.75
0.1 0.2 0.3
0.6 0.7
0.70
0.1
0.4
0.5
0.20
0.3 0.4 0.7 (prropaation of perially labeled evamples
0 (proponioon of periilly labeled cxample)
(sroporiio panially labeled ouanplss)
(d) segment
(c) satimage
(f) letter
Figure 5: Test accuracy of each comparison method changes as p (proportion of partially labeled examples) increases from 0.1 to 0.7 (with two false positive candidate label [r = 2D.
MGPLL
SURE
PALOC
CIP.
PL.VVM
KLKNN
0.90
1.00
0.90
0.85
0.90
0.80
0.80
0.75
0.70
0.1
0.5
0.70
0.3
0.7
0.60
0.1
0.5 0.6
[ssopentiins wtriiilllyylllllooolllllla kisied campio)
(peopontion ol purtillly lalded examples)
(pmpanicc pantally labelod cumple)
(a) ecoli
(b) deter
(c) vehicle
1.00
0.95
1.00
0.90
0.00
0.90
0.85
0.80
0.80
0.70
0.7 P.prgpotion.r sabeeed exmmppe)
0.70
0.1
0.5
0.20
0.7
panisily labelad P-gropatioon pfoniallly laheled exmppto
(d) segment
(c) satimage
(f) letter
Figure 6: Test accuracy of each comparison method changes as p (proportion of partially labeled examples) increases from 0.1 to 0.7 (with three false positive candidate label [r 3)).
all the other methods across different values on four datasets, vehicle, segment, satimage and letter, while achieving remarkable performance gains on segment and satimage. On the other two datasets, ecoli and deter, MGPLL also produces the best results in most cases and remains to be the most effective method. By contrast, the performance of the other comparison methods varies largely across different datasets. For example, CLPL and SURE demonstrate good performance ecoli, deter and vehicle, but presents inferior results than PL-KNN many cases the other three datasets. PALOC and PL-SVM have the same drawback of producing poor results some datasets. Our proposed MGPLL demonstrates good overall performance across these varying cases.
We also conducted experiments the PL datasets with random noise labels produced under the PL configuration settings (I), (II) and (III). The comparison results in these three sets of configura- tions reported in Figure 4 Figure 5 and Figure 6 respectively. From these figures can
13

---

Under review as a conference paper at ICLR 2021
BirdSong
-
Yahoo! Nows
Birdsong
Yahoo! NewS
-
Birdsog
Yahoo! News
0.01
0.00
0.00
chosificaion sccuracy
chasiiicaion accuracy
chooiiictoon
0.40
1000
100
Figure 7: Parameter sensitivity analysis for MGPLL on the Lost and MSRCv2 datasets.
that the proposed MGPLL (with noise label generator G2 (e)) achieves similar positive comparison results as in the configuration setting (IV), In particular, the proposed method achieves remarkable performance gains four of the overall six datasets, segment, satimage, vehicle and letter.
A.5 PARAMETER SENSITIVITY ANALYSIS
We also conducted parameter sensitivity analysis two well-world PL datasets BirdSong and Ya- hoo! News datasets to study how the trade-off hyperparameters a, B and influence the performance of MGPLL. We conducted the experiments by using different combination settings of the a, 1j and y values from {0.001, 0.01, 0.1, 1, 10}. We vary each parameter's value by keeping the other two fixed at their best setting. Note that larger value for a, 8 and 7 will provide larger weight to the feature level WGAN loss, generation loss and auxiliary classification loss respectively.
The three figures in Figure the average test results as well standard deviations for different a 3 and y values respectively. We can see that when a is very small, the performance of MGPLL is very good since the feature level WGAN loss is not allowed to contribute much to the learning. With the increase of a, the performance improves, which suggests that the WGAN loss is important. When a is large, the performance degrades as the WGAN loss dominates. This reasonable since the WGAN loss is expected to help the predictive model, rather than dominate the learning process. similar phenomenon can be observed for y. For the parameter 3, the proposed method performs bad when 3 is very small. With the increase of ;3, the performance of MGPLL improves and remains relatively stable in broader range, i.c., 3 E [0.01, 1]. It shows that the proposed model not very sensitivity to the 3 parameter within the considered range of values.
14


Reviews:
[{"review_id": "uXskmJ-G1LL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a multi-level generative model for partial label learning. The basic idea is to use a conditional noise label generation network to model the label noise. The noise label generator and the data feature generator are learned in an adversarial manner. Experiments on synthesized and real-world data sets show competitive performance. \n\nOverall the idea modeling conditional noise label is interesting. The paper is easy to follow. My concerns are mainly as following. \n\n1.\tIt is not unclear for me for the motivation that previous methods are based on the separate label distribution estimation steps or the error-prone label confidence estimation process. Such kinds of approaches do not mean that they are not good methods. Actually using the separate label distribution estimation steps or the error-prone label confidence estimation process may be benefit to be more accurate and more efficient. Therefore, such motivation is not so convincing, and leads to an important new contribution. \n2.\tThe second motivation is based on non-random noise. This is an interesting observation. The proposed method involves a conditional probability to model the correlation. However, this might be sensitive and restrict. In practical, such kind of information may be not correct for infrequent patterns. For frequent patterns, this could be trivial to hold. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "qCVdGfq2TIJ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I like the idea of this paper and it is well-written. In this paper, the authors propose multi-level generative models for partial label learning with non-random label noise. It consists of five components: the conditional noise label generator which models the noise labels conditioning on the ground-truth label at the label level; the conditional data generator which generates data samples at the feature level conditioning on the denoised label vectors; the discriminator which separates the generated candidate label vectors from the observed candidate label vectors in the real training data; the discriminator which separates the generated samples from the real data in the feature space; and the prediction network which predicts the denoised label for each sample from its input features. With the proposed minmax adversarial loss, the proposed framework achieved state-of-the-art performance for partial label learning.\\\n\\\nI think this paper has the following advantages:\n1. It is novel to exploit multi-level generative models to model non-random noise labels for the partial label learning problem.\n2. The experiments in this paper are complete and thorough. The authors have tested the model in many datasets and designed the ablation study to verify the effect of each loss.\n3. The proposed model achieved the state-of-art results.\n\nDespite the above advantages, I still have the following questions:\n1. Is the true label vector $\\mathbf{z}$ given for each training sample as the ground truth or it is sampled from the multinomial distribution?\n2. In Algorithm 1, why the parameter $\\Theta$ needs to be limited in $[-c, c]$?\n3. In equation 3, it is not clear what the “$n$” in $\\mathbf{y}_n$ stands for? In the previous context, $n$ is the size of the training set.\n4. In Fig. 1, the input for $\\mathcal{L}_c$ is $\\mathbf{z}$ and $F(\\mathbf{x})$, while in equation 4 it is $F(\\mathbf{x})$ and $\\mathbf{y}\\ominus G_n(F(\\mathbf{x}),\\epsilon))$\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2JXf5FnvDWg", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes a new method of learning from data with partially observed labels. In this problem, every instance has a label candidate set, which contains the true label. This submission introduces adversarial learning to improve the disambiguation of inexact labels. Particularly, there are two adversarial learning component. In the first component, a generator tries to match the distribution of label candidate sets given the \"true\" label of an instance. In the second component, a generator tries to learn the distribution of instances give their \"true\" labels. Since the the \"true\" label is not accessible, the \"true\" label is actually from a predictive model. \n\nThe submission has done extensive experiments and shows that the proposed method outperforms several baseline methods. \n\nIn a summary, the experiment results of this submission is strong, but I feel the motivation of the model design is not clear. \n\nMy biggest question is the motivation behind adversarial training. To make it simple, adversarial training aims to match a generative distribution to the data distribution. \n\n1. Loss (4) is somewhat reasonable to me. Can I interpret it as: F(x) predict z, and z generate a label vector y'; if y' does not match the partial label y, then there is a loss? Then the model needs to learn both the predictive distribution p(z | x) through F, and the label distribution p(y | z).  \n\nIf my interpretation is correct, have you considered to use alternative distributions for p(y | z) instead of a generator? There are many choices such as Restricted Boltzmann machine. Since you treat y as continuous variables, there are even more choices such as a decoder in a conditional variational autoencoder.  I don't mean to say that GAN distribution is not a good choice, but if you could make this task clear so people may consider alternative choices. \n\n2.The generative distribution p(x | z) is more confusing. The structure in (5) seems like an encoder-decoder: decoding x, y gives z, and z should recover x. With this loss, the model has some flavor of generative modeling because it models the feature vector x. As a principle in machine learning, generative modeling is less powerful than discriminative models in classification tasks, so I don't see why we want to model input features. \n\n3. In the ablation study, have you tried to use only the classification model and losses from adversarial labels? \n\n4. Since the model has better performance than baseline methods, there might be several explanations. \n\n1) differences in base models. What are base predictive models of F in your algorithm and equivalent in competing methods? Is the difference significant? In the results from synthetic datasets, p(y | z) does not contain much information since noise labels are randomly sampled. The proposed method still outperform baseline methods by a good margin. Is this an evidence that the model is still better without much contribution from the label generator? In another word, if you replace G_n(z, \\epsilon) by your groundtruth distribution, the model should perform even better? \n\n2) The combination of the generative model Gx(Z, epsilon), which works like a generative model. When it works with F(x) and other loss terms, the entire model is like a combination of two neural networks. This might provide extra classification power? \n\n5. The loss in (4) is unnatural to me: the loss is computed from probabilities, and square loss may not be the best choice. \n\nI am asking a lot of questions, trying to explain why the experiment results. In another work, I wish the submission could answer some of these questions. \n\nThe proposed model also have many parameters and neural network components. I don't how easy it is for others to tune the model. \n\nThe writing of the submission also has many issues. A lot of symbols are not well defined. A few examples. \n1. n has two meaning: number of instances and \"noisy\" labels. \n2. what is p_z? \n3. The minimization problem in 2 is invalid since the true label z is unknown. \n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Dear Authors,

Thank you very much for your detailed feedback to the reviewers in the rebuttal phase. This certainly clarified some of the concerns raised by the reviewers and contributed highly to deepen their understanding of your work.

We positively evaluated the novelty and the superior empirical performance of the proposed method. However, we still have concern about the justification since the proposed model is so complex that it is not clear what was the key for the good performance. 

For this reason, I suggest rejection of this submission, in comparison with many other strong submissions. I hope that the reviewers' feedback is useful for improving your work for future publication.

Best,
AC
Decision: Reject