Instance-Dependent Partial Label Learning
Authors: Ning Xu
Congyu Qiao
Xin Geng
Min-Ling Zhang
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: partial label learning
label enhancement
label distribution
pseudo label

Abstract: Partial label learning (PLL) is a typical weakly supervised learning problem, where each training example is associated with a set of candidate labels among which only one is true. Most existing PLL approaches assume that the incorrect labels in each training example are randomly picked as the candidate labels. However, this assumption is not realistic since the candidate labels are always instance-dependent. In this paper, we consider instance-dependent PLL and assume that each example is associated with a latent label distribution constituted by the real number of each label, representing the degree to each label describing the feature. The incorrect label with a high degree is more likely to be annotated as the candidate label. Therefore, the latent label distribution is the essential labeling information in partially labeled examples and worth being leveraged for predictive model training. Motivated by this consideration, we propose a novel PLL method that recovers the label distribution as a label enhancement (LE) process and trains the predictive model iteratively in every epoch. Specifically, we assume the true posterior density of the latent label distribution takes on the variational approximate Dirichlet density parameterized by an inference model. Then the evidence lower bound is deduced for optimizing the inference model and the label distributions generated from the variational posterior are utilized for training the predictive model. Experiments on benchmark and real-world datasets validate the effectiveness of the proposed method. Source code is available at https://github.com/palm-ml/valen.

Instance-Dependent Partial Label Learning
Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China MOE Key Laboratory of Computer Network and Information Integration, Ministry of Education, China (xning, qiaocy, xgeng, zhangml @seu. edu.cn cn
Abstract
Partial label learning (PLL) typical weakly supervised learning problem, where each training example is associated with a set of candidate labels among which only one true. Most existing PLL approaches assume that the incorrect labels in each training example are randomly picked as the candidate labels. However, this assumption is realistic since the candidate labels are always instance- dependent. In this paper, we consider instance- destance-dependent PLL and assume that each example is associated with latent label distribution constituted by the real number of each label, representing the degree to each label describing the feature. The incorrect label with a high degree more likely as the candidate label. Therefore, the latent label distribution is the essential labeling information in partially labeled examples and worth being leveraged for predictive model training. Motivated by this consideration, we propose a novel PLL method that recovers the label distribution as label enhancement (LE) process and trains the predictive model iteratively in every epoch. Specifically, we assume the true posterior density the latent label distribution takes on the variational approximate Dirichlet density parameterized by inference model. Then the evidence lower bound is deduced for optimizing the inference model and the label distributions generated from the variational posterior utilized for training the predictive model. Experiments on benchmark and real-world datasets validate the effectiveness of the proposed method. Source code is available https:://qitiei com/ pa mamallmmmmaae coom/palmmmm/vaeess vvalen
annotated
Introduction
Partial label learning (PI..) deals with the problem where each training example associated with set of candidate labels, among which only one label is valid 7 5 37]. Due to the difficulty in collecting exactly labeled data in many real-world scenarios, PLL leverages inexact supervision instead of exact labels. The need to learn from the inexact supervision leads to wide range of applications for PLL techniques, such as web mining 25), multimedia content analysis [38
ecoinformatics
24 31,, etc.
To accomplish the task of learning from partial label data, many approaches have been proposed. Identification-basedd PI.I approaches [17 28 24
37 regard the ground-truth label as latent variable and try to identify it. Average-based approaches 15] 40) treat all the candidate labels equally and average the modeling outputs as the prediction. For confidence-based approaches
8 34 42], the confidence of each label is estimated instead of identifying the ground-truth label. These approaches always adopt the randomly picked candidate labels to corrupt benchmark data into partially labeled version despite having no explicit generation process of candidate label sets. To depict the iistance-indepencee
generation process of candidate label sets, Feng [9] proposes statistical model and deduces a risk-consistent method and a classifier-consisten method. Under the
"Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

deogigtiionneeeee
descripiionngs
06
06
04
01
02
02
bid
labe
00
Ship
Alrplane
Frog
(a) Handwritten digits in MNIST 22
(b) Color image in CIFAR-10
Figure I: The examples about the latent label distributions for partial label learning. The candidate labels are in the box and the red one is valid.
same generation process, another classifier-cossiseent risk estimator is proposed for deep model and stochastic optimizers [26].
The previous methods assume that the candidate labels are randomly sampled with the uniform generating procedure [2. which is commonly adopted to corrupt benchmark datasets into partially labeled versions in their experiments. However, the candidate labels are always instance-dependent (feature-dependent) in practice the incorrect labels related to the feature picked as candidate label set for each instance. These methods usually do not perform as well as expected due the unrealistic assumption the generating procedure of candidate label sets.
more likely to be
In this paper, we consider instance-dependent PLL and assume that each instance in PLL associated with latent label distribution B3 35 [1] constituted by the real number of each label, representing
the degree to each label describing the feature. Then, the incorrect label with a high degree in the latent label distribution is more likely be annotated candidate label set of the handwritten digits in Figure contains "1", , "3" and "5", where "I" and "3" not ground-truth but selected candidate labels due to their high degrees in the latent label distribution of the instance. The object Figure [(b) is annotated with "bird" and "airplane" as the degrees of these two labels are much higher than others in the label distribution. The intrinsical ambiguity increases the difficulty of annotating, which leads to the result that annotators pick the candidate labels with high degrees in the latent label distribution of each instance instead of annotating the ground-truth label directly in PLL. Therefore, the latent label distribution is the essential labeling information in partially labeled examples and worth being leveraged for predictive model training.
the candidate label. For example, the
Motivated by the above consideration, we deal with the PLL problem from two aspects. First, we enhance the labeling information by recovering the latent label distribution for each training example as a label enhancement process 33 35 Second, we run label enhancement and train the predictive model with recovered label distributions iteratively. The proposed method named VALEN, i.e., VAriational Label ENhancement for instance-dependers partial label learning, uses the candidate labels to initialize the predictive model in the warm-up training stage, then recovers the latent label distributions via inferring the variational posterior density parameterized by an inference model with the deduced evidence lower bound, and trains the predictive model with a risk estimator by leveraging the candidate labels as well as the label distributions. Our contributions can be summarized as follows:
We for the first time consider the instance-dependent PLI and assume that each partially laheled example is associated with latent label distribution, which is the essential labeling information and worth being recovered for predictive model training.
We infer the posterior density of the latent label distribution via taking on the approximate Dirichlet density parameterized by an inference model and deduce the evidence lower bound for optimization, in which the topological information and the features extracted from the predictive model are leveraged.
We train predictive model with a proposed empirical risk estimator by leveraging the candidate labels well the label distributions. We iteratively recover the latent label distributions and train the predictive model in every epoch. After the network has been fully trained, the predictive model can perform predictions for future test examples alone.
Experiments on the corrupted benchmark datasets and ness of the proposed method.
PI.I. datasets validate the effective-

---

2 Proposed Method
First of all, we briefly introduce some necessary notations. Let X R be the g-dimensional instance space and y {v,, U2, ..., Ne be the label space with e class labels. Given the PLL training set D D= (x,, S.) S i S n} where æ; denotes the g-dimensional instance and S; C 3 denotes the candidate label set associated with æ,, Note that S; contains the correct label of æ; and the task of PLL is to induce a multi-clase classifier f 7: X' H y from D. For each PLL training example (a,,, we use the logical label vector L [1'" .192 ...... [ {0, 1] to represent whether y/ is the candidate label, i.e., 1 1 if yj S,, otherwise 1 0. The label distribution of is denoted by d; [d di ..... d E [0, 1]e where Σj=1 d = 1. Then L [1 11,22.... Ln] and D [d.. d2 ... dn) represent the logical label matrix and label distribution matrix, respectively.
2.1
Overview
To deal with PLL problem, we iteratively recover the latent label distribution for each example æ and train the predictive model by leveraging the recovered label distribution. We start with a warm-up period, in which we train the predictive model with the PI. minimal loss [26]. This allows us to attain a reasonable predictive model before it starts fitting incorrect labels. After the warm-up period,
the features extracted from the predictive model can help for recovering the latent label distribution.
Benefited from the essential labeling information in the recovered label distribution, the performance of the predictive model could be further improved.
VALEN implements label enhancement and classifier training iteratively in every epoch. In label enhancement, we assume the true posterior density of the latent label distribution takes on the variational approximate Dirichlet density parameterized by an inference model. Then the evidence lower bound is deduced for optimizing the inference model and the label distributions can be generated from the variational posterior. In classifier training, the predictive model is trained by leveraging the recovered label distributions and candidate labels with an empirical risk estimator. After the models has been fully trained, the predictive model can perform prediction for future test instances alone.
2.2 Warm-up Training
The predictive model 0 is trained partially labeled examples by minimizing the following PLL minimal loss function 26)
Lmin Σ
min ex()))) Uo ES,
(1)
where é is cross-entropy loss and e = {e y) E } denotes the standard canonical vector in RS, i.e. the j-element e equals and others equal O. Similar to (66, the min operator in Eq. [ is replaced by using the current predictions for slightly weighting on the possible labels in warm-up training. Then we could extract the feature & of each via using the predictive model.
2.3 Label Enhancement
We assume that the prior density p(d) is Dirichlet with &, i.e. p pdd Dir (d () where á = (e,e,...,e) e) is c-dimensional vector with minor value e. Then we let the prior density p(D) be the product of each Dirichlet
DD) II (ir
(2)
We consider the topological information of the feature space, which is represented by the affinity graph G = (V, E, A). Here, the feature vector Φ; of each example could be extracted from the predictive model 0 in current epoch, V {Φ: 1 < i < n) corresponds the vertex set consisting of feature vectors, F = [(0i, Φ5) 1 S i + j < n) corresponds to the edge set, and sparse adjacency matrix A [aijlaxn be obtained by
aij = { 1
if Φb E (( otherwise
(3)
3

---

where (),,) is the set for k-nearest neighbors of Φ3 and the diagonal elements of A are set to 1.
Let features matrix + [ Si,,,,...... Øn). adjacency matrix A and logical labels L be observed matrix, VALEN aims to infer the posterior density p(D L. Φb A). As the computation the exact posterior density PDD[L, + A) is intractable, fixed-form density ((D[L, + A) is employed to approximate the true posterior. We let the approximate posterior be the product of each Dirichlet parameterized by vector a; [at az. "
Iw D A)
II Dir (d.lo)). ==1
(4)
Here, the parameters A [0i, O.,,.... Onn] are outputs of the inference model parameterized by w, which is defined as two-layer GCN [20] by GCN(L,
D, A) A ReLU (AZwo) W1, with
[ L] and weight Wo. W1. Here A Å AÁ is the symmetrically normalized weight matrix where is the degree matrix A.
By following the Variational Bayes techniques, a lower bound on the marginal likelihood of the model derived which ensures that fw [D $ A) is as close as possible to p(D & A). For logical label matrix L. feature matrix L, and the corresponding A, the log marginal probability can be decomposed follows
log plp(L, (L, $ A) LELBO KL (D +, A)]L $, A)]-
(5)
where
CELBO = E. (.L. 4, AJD)] kL (D]L, A, AD)))
(6)
Due to the non-negative property of KL divergence, the first term LELBO constitutes lower bound of p(L, Φb A), which is often called as evidence lower bound (ELBO), i.e., logp(L, Φ A) 2 LELBO.
According to Eq. and Eq. (4 the KL divergence in Eq. (6) can be analytically calculated as follows:
KL (gm (DIL, $, A)(PP()) - logg (Σ of) Ei= log (of)
- log l (c.-) + clog r (e) + + (ag -e) Wb ao)) (Σi- of) of))
where r(-) and (-) are Gamma function and Digamma function, respectively.
As the first part of Eq. (6) intractable, we employ the implicit reparameterization trick [0 to approximate it by Monte Carlo (MC) estimation. Inspired by [20], we simply drop the dependence on Φ:
n
PLL A, D)
II (1, A,D),
(8)
p D = IIII =1
(aij | d:, d,), with p (a.j = 1 = (d, d,)-
Here, s(.) is the logistic sigmoid function. We further assume that p (4, D) is multi- variate Bernoulli with probabilities т, In order simplify the observation model, ((m) (m) (m)
[T] .22 ...., (m²) is computed from m-th sampling (m)) with a three-layer MLP parame- terized by n,. Then the first part of Eq. (6) can be tractable:
E (L Φ, AJD)
Σ m=1
tr (I- L) log (I T(()))
M
[ log T(m)
- ||A -D(D
+ ir
Mooe detailed caleulations can be seen in Appendix A.I.

---

Algorithm VALEN Algorithm
Input: The PLL training set {(x,, St) epoch 7'' and iteration
1: Initialize the predictive model 0 by warm-up training, the reference model w and observation model ny:
2: Extract the features + from predictive model 0 and calculate the adjacency matrix A;
3: for ........................... T do
4: Shuffle training set D { 5.) into 7 mini-batches;
5: for ,,....
7 do
6:
Obtain label distribution d, for each example æ; by Eq.
7:
Update 0, w and ry by forward computation and back-propngation by fusing Eq. and
Eq. 11
8:
end for
9:
end for
Output: The predictive model
Note that we can use only one MC sample in Eq.
9 during the training process as suggested in
19 35.
In addition, VALEN improves the label enhancement by employing the compatibility loss, which enforces that the recovered label distributions should not be completely different from the confidence (xx)) 9 26 estimated by current prediction (x 0):
1 n
L.
Gax;) log d.
(10)
=1
where
(f())) =
fs (a 0) Emes, fx (wx;
yf E S otherwise
(11)
Now we can easily get the objective of label enhancement LL as follows:
LLE Xc. LELBO
(12)
where is hhper-parametee The label distribution matrix D is sampled from ((D)L, 4 A), i.e. d; ~ Dir(a.). Note that the implicit reparameterization gradient [10] is employed, which avoids the inversion of the standardization function, which makes the gradients can be computed analytically in backward pass.
2.4 Classifier Training
To train the predictive model, we minimize the following empirical risk estimator by levering the recovered label distributions:
d
Rv)
Enses, dhd (((aa)),e
(13)
ES...
Here adopt the average value of d, sampled by d, Dir(a,). We any deep neural network as the predictive model, and then equip it with the VALEN framework to deal with PLL. Note that we could train the predictive model and update the label distributions principled end-to-end manner by fusing the objective Eq. and Eq. (3).. The algorithmic description of the VALEN is shown in Algorithm
Let Fv minjer (f) be the empirical risk minimizer and f min/ez Ry R be the optimal risk minimizer where Rv(f) is the risk estimator. Besides, we define the function space Hy, for the label yj E ) as {h= : + fay (æ: ʃ E F}. Let R., (Hyy) be the expected Rademacher complexity Hyg with sample size n, then we have the following theorem.
Theorem Assume the loss function E( f(((x)) ed)) is I--Lipschitz with respect to f(x) < I. < oo) for all y ) and bounded by M, i.e., M supzex ((f()) ((f()), Then, for any
5

---

5> O, with probability at least 1 8,
log
( -R( =4 Σ Nn (Hy,) + My 5=1
2n
The proof of Theorem T is provided in Appendix A.2. Theorem T shows that the empirical risk minimizer fv converges to the optimal risk minimizer f* and Rn (Hys) - for all parametric models with a bounded norm.
3 Related Work
As shown in Section supervision information conveyed by partially labeled training examples implicit as the ground-truth label hidden within the candidate label set. Therefore, partial label learning can be regarded as a weak supervision learning framework [18] with implicit labeling information. Intuitively, the basic strategy for handling partial label learning disambiguation, i.c.. trying to identify the ground-truth label from the candidate label set associated with each training example, where existing strategies include disambiguation by identification or disambiguation by averaging. For identification-bol based disambiguation, the ground-truth label is regarded as latent variable and identified [17 28 24 37 For averaging- -based disambiguation, all the candidate labels are treated equally and the prediction is made by averaging their modeling outputs [15 40)
Most existing algorithms aim to fulfill the learning task by fitting widely-used learning techniques to partial label data. For maximum likelihood techniques, the likelihood of observing each partially labeled training example is defined over its candidate label set instead of the unknown ground-truth label [1 24]. K-nearest neighbor techniques determine the class label of unseen instances via For maximum margin techniques, the classification margins over the partially labeled training examples are defined by discriminating modeling outputs from candidate labels and non-candidate labels [28 37. For boosting techniques, the weight over each partially labeled training example and the confidence over the candidate labels are updated in each boosting round [31) For disambiguation-fie strategies, the generalized description degree is estimated by using graph Laplacian and induce multi-output regression 34 The confidence each candidate label estimated by using the manifold structure of feature space 42) However, these methods just estimate the soft labeling information and train the predictive models in separate stages without considering the feedback of the predictive models.
voting among the candidate labels of its neighboring examples [5 40].
The above-mentioned works were solved in specific low-efficiency manners and incompatible with high-efficient stochastic optimization. To handle large-scale datasets, the deep networks are employed with an entropy- based regularizer to maximize the margin between the potentially correct label and the risk estimator and a progressive identification,
unlikely ones [36]. (26) proposes a maassffir-consiister which is compatible with deep models and stochastic optimizers. [ proposes statistical model to depict the generation process of candidate label sets, which deduces a risk-consistent method and a classifier-oonsistent method.
The previous methods assume that the candidate labels are randomly sampled with the uniform gen- erating procedure. However, the candidate labels are always intance-deppndeen (feature-dependem)) in practice as the incorrect labels related the feature are more likely to be picked candidate label set for each instance. In this paper, we consider Destance-depedemn PLL and assume that each instance in PLL is associated with a latent label distribution [33] 35 II constituted by the real number of each label, representing the degree to each label describing the feature. Label enhancement
(LE) 12 33 35 recovers the latent label distribution from the observed logical labels, in which the recovered label distribution is a kind of pseudo label 20 23 actually. Note that the generation
process of soft labels in label smoothing 30 27 and distillation label enhancement process.
14
39 could also regarded as

---

Table 1: Classification accuracy (meandstd) of cach comparing approach on benchmark datasets
corrupted by the instance-dependent generating procedure.
MNIST
Kuzushiji-MNIST
Fashion-MNIST
CIFAR-I0
VALEN
97.85+0.05%
86.19:0.14%
86.171.0.9%
80.38:0..52%
PRODEN
97.69-0.04%
85.71
85.5410.09%
79.80%
RC
97.60=0.05% 84.86=0.11%
85.51=0.10%
79.46=0.25%
CC
97.44=0.03% 82.67=1.82%
85.19=0.04%
78. 0.6%%
D2CNN
94.63=0.16% 83.03=0.78%
82.42=0.21%%
73.11=0.11%
GA
95.25=0.0%%
82.45=0.63%
80.41=0.24%
77.57
Table 2:
Classification accuracy (tea of each comparing approach on benchmark datasets corrupted by the uniform generating procedure.
MNIST
Kuzushiji-MNIST
Fashion-MNIST
CIFAR-10 10
VALEN
97.93+00.055
88.76+0.224
81.93+ 1.01 %
PRODEN
0..97 88.55=0.109
88.9420. 12%
81.53-10.53%
RC
97.8610.03%% 86.6520.1099
88.59=10.08%
81.30 1.30%
CC
97.73=0.02% 87.99:0.03%%
88.93=0000%%
80.17=1.099 1.09%
D2CNN
95.12:00.6%% 84.03=00.8%%
80.42:#0221% %
75.11=0.11%
GA
96.29=0.19%
82.3610.98%
81.91
60.14+1..5%
4 Experiments
4.1
Datasets
We adopt four widely used benchmark datasets including MNIST [22 Fashion-MNIS7 32 Kuzushi jiushiji-Min -MNMSST (), and CIFAR-10 [21], and five datasets from the UCI Machine Learn- ing Repository [1, including Yeast, Texture, Dermatol oogy Synthetic Control, and 20Neggooups.
We manually corrupt these datasets into partially labeled versions by using flipping probability == = P( 1 0, 0;);) where ?n1 is the original clean label. To synthesize the instance- dependent candidate labels, we set the flipping probability of each incorrect label corresponding to an example æ; by using the confidence prediction of a clean neural network 0 (trained with the fm((;;)
original clean labels) 43 with _ max f,(:0))) where Y; is the incorrect label of æ,, The
uniform corrupted version adopts the uniform generating procedure 26 to flip the incorrect label into candidate label, where e.th
In addition, five real-world PLL datasets are adopted, which are collected from several application domains including Lost (7]. Soccer Player [38] and Yahoo! News 133 for automatic face naming from images or videos, MSRCv2 (24] for object classification, and Bi rdSong (3] for bird song classification. The detailed descriptions of these datasets are provided in Appendix A.3.
We run 5 trials on the four benchmark datasets and perform five-fold cross-validation on UCI datasets and real-world PLL datasets. The mean accuracy well standard deviation are recorded for all comparing approaches.
4.2 Baselines
The performance of VALEN is compared against five DNN based approaches: 1) PRODEN [26]
progressive identification partial label learning approach which approximately minimizes a risk estimator and identifies the true labels in seamless manner; 2) RC [9]: A risk-consistent partial
label learning approach which employs the importance reweighting strategy to converges the true risk
"The datasets corrupted by the instance- genaaeceed procedure are available at awtps/bbaaatttttts///s/aaaiii
google. com/ com//aiivee/ooaeerr// wtt.::/www..aaee///ccccoeeeeaae.

---

Table 3: Classification accuracy (meanistd) of each comparing approach on UCI datasets corrupted by the instance- dependent generating procedure.
Yeast
Texture
Synthetic Control
Dermatology
20Newsgroup
VALEN
57.57-+-1.0%
9.976%0.93%
82.86++0.66%
89.86+11.3 1.11%
81.88-00.47%
PRODEN
54.78+ +
89.87-22. 14% 14%
71.16+6.1955
88.53++..875
78.06:t:0.74%
RC
54.77+1.27%
89.57+2.37%%
65.99+2272%%
88.53+:3.4495
78.02:+00.99%
CC
54.98 % 44.94:1..87%% 69.52:15579%%
88.92+7.556%
66.99+2.445%
88.26+44.1 11%
77. 77.88++-0339%
D2CNN
62.66.18.92%%
81,95.1..
73.55:1:0.22%
GA
25.86:13.17% 74.84
56,43:1 1.2%%
84.8511 1.4%%
49.49:13.42%%
CLPL
54.92.1..38%
81.2719.099%
66.33.13.25%
92.07-13.429
77.62.L0.23%
PL-SVM
41.85.15.92% 39.03
50.33:1.5.33%
84.98.14.56%
72.89-10.41%
PL-KNN
47.44.12.66%%
80.50-11.26%
83.6113.1194
33.28 1.0%%
IPAL
56.40+t2.07%
93.49=0.89%
77.66+3.60%
88.94-8.34%%
67.38=t0.95%
PLLE
55.53
84.45=1.07%
16.16%7.96%
93.16.t..58% %0
75.54=0..6%
Table 4: Classification accuracy (mean of each comparing approach on UCI datasets corrupted by the uniform generating procedure.
Yeast
Texture
Synthetic Control
Dermatology
20Newsgroup
VALEN
58.18.1 1.46%
97.30-10.57%
97.1710.47%
97.07 %
71.75.13.02%
PRODEN
56.32+ 1.98%
97.75=+0.53%
95.83+1.955 1.95%
95.07 .84%
68.28:+0.91%%
RC
56.39 1.85%
9..77=00.55%
11.99=1.80%
95.62++.55 1.51%
68.44+1. 1.09%
CC
16.25.11..994
97.79110.57%
96.33.11.39%
95.90-1 .69%
67.95
D2CNN
54.04-1 .0%%
97.2310.72%%
81.16.1.. 1
90,43
65.88.1..566%
GA
1.2.5%%
1.009
1.53%
51,95 3.89%
18.29%
CLPL
56.54
98.14
94.66:16.41%
96.72++0.76%
70.45
PL-SVM
46,23-47.2159 %
39.74=2.11%
76.50 %
92.37-+5.08%
70.44:10.37%
PL-KNN
44.40:t..50% 33.86%3.39%%
9..31=0.85%
95.33
92.91
27.10=:0.99%
IPAL
98.71=0.37%
96.83+1.909
95.35+12.085
6..39
PLLE
53.58+2.86%
98.40=0.40%%
1.966=
1.8%%
53.88
minimizer; 3) CC (:: A classifier-consistent partial label learning approach which uses a transition matrix to form empirical risk estimator; 4) D2CNN [36] A deep partial label learning approach which design an entropy-
regularizer to maximize margin between the potentially correct label and the unlikely ones; 5) GA [16]: An unbiased risk estimator approach which can be applied for partial label learning.
For all the DNN based approaches, we adopt the same predictive model for fair comparisons. Specifically, the 32-layer ResNet is trained on CIF AR-10 in which the learning rate, weight decay and bicc- size are set to 0.05, 10 and 256, respectively. The three-layer MLP is trained on MNIST, Fashi ON-MNIST and Kuzushi ioaijji-MMi --MIIST where the learning rate, weight decay and mini-batch size are set to 10-2, 10-4 and 256, respectively. The linear model is trained on UCI and real-world PLL datasets where the learning rate, weight decay and mini-batch size are set to 10-2 10-1 and 100, respectively. We implement the comparing methods with PyTorch. The number of epochs is set to 500, in which the first 10 epochs are warm-up training.
In addition, we also compare with five classical partial label learning approaches, each configured with parameters suggested respective literatures: 1) CLPL 77 convex partial label learning approach which uses averaging-based disambiguation; 2) PL-KNN [5]: An instance- based partial label learning approach which works by k-nearest neighbor weighted voting: 3) PL-SVM (8]: maximum margin partial label learning approach which works by identification sertifiiaatio-bbaedd disambiguation; 4) IPAL 41: A nor-paraettric method that applies the label propagation strategy to iteratively update the confidence of each candidate label; 5) PLLE 34 A two-stage partial label learning approach which estimates the generalized description degree of each class label values via graph Laplacian and induces multi-label predictive model with the generalized description degree in separate stages.
8

---

Table 5: Classification accuracy (mean=istd) of each comparing approach on the real-world datasets.
Lost
MSRCv2
BirdSong
Soccer Player
Yahoo!News
VALEN
70.28:12.299
17.61:11.79%
72.02:10.37%
55.90.0.58%%
67.52-10.19%
PRODEN
68.62-t.486%%
44.47+42.33%
71.6%+0..3%
54.40=+0.85%
67.12++0.97%
RC
68.89+550029 44.59++2.65%
71.56+0.88%%
54.23++0.89%
67.04++0.889
CC
62.2111..7%% 47.49.12.31%
68.42.10.99%
53.50:10.96%%
61.92.10.96%
D2CNN
68.56116.6855 43.27-12.98%
65.48.12.57%%
48.16110.625
52.46.11..1%
GA
50.21113.62% 30.91 %
34.5713.41%%
50.65110.94%
45.72.11.75%
CLPL
74.15.13.036
44.47.12.588%
65.76111.95%
1..01
53.25
PL-SVM
71.56:1..71%
38.25
50.6611.23%
36.39-11.0394
51.24:10.72%
PL-KNN
33.87+2.488%
43.28+2..55%
64.34=0.75%
1..24+1.23%
0..3%+0.377%
IPAL
72.10=2.75% 1..66=11.66% %
70.32=0..1%%
5..4110668%
66.04=t0.85%
PLLE
72.55=3.55%%
47.54
70.63=1.24%%
13.38
49.45%
4.3 Experimental Results
Table 1 reports the classification accuracy of each DNN-based method on benchmark datasets corrupted by the instance-dependent generating procedure. The best results are highlighted bold. In addition, / indicates whether VALEN is statistically superior/inferior to the comparing approach on each dataset (pairwise t-test at 0.05 significance level). From the table, we can observe that VALEN always achieves the best performance and significantly outperforms other compared methods in most cases, In addition, we also validate the effectiveness of our approach on uniform corrupted versions that is commonly adopted in previous works. From Table 2 we observe that VALEN achieves superior or at least comparable performance to other approaches on uniform corrupted versions.
Table 3 and Table repoort the classification accuracy of each method on UCI datasets corrupted by the destanceddeeendent generating procedure and the uniform generating procedure, respectively. VALEN always achieves the best performance and significantly outperforms other DNN-based methods in most cases on instance-dependen corrupted versions while achieves superior at least comparable performance to other approaches uniform corrupted versions. We further compare VALEN with five classical PLL methods that can hardly be implemented by DNNs on large-scale datasets. Despite the small scale of most UCI datasets, VALEN always achieve the best performance in most cases against the classical PLL methods as VALEN can deal with the high average number of candidate labels (can be seen in Appendix A.3) in the corrupted UCI datasets.
Table rreports the experimental results on real-world PLL datasets. We can find that VALEN achieves best performance against other DNN-based methods on the real-world PLI. datasets. Note that VALEN achieves best performance against classical methods on all datasets except Lost and MSRCv2 as these datasets are small-scale and the average number of candidate labels in each dataset is low (can be seen Appendix A.3), which leads to the result that DNN-based methods cannot take full advantage.
Figure 2(a) and Figure 2(b) illustrate the performance of VALEN on KMNIST corrupted by the destence-dependent generating procedure and the uniform generating procedure under different flipping probability, respectively. Besides. the performance of the ablation version that removes the label enhancement and trains the predictive model with PLL minimal loss (denoted by VALEN-NON) recorded. These results clearly validate the usefulness of recovered label distributions for improving predictive performance. Figure 2(c) illustrates the recovered label distribution matrix over all training examples converges as the number of epoch (after warm-up training) on Kuzushi ji-ijjj-MMII ST. We can see that the recovered label distributions converge fast with the increasing number of epoch.
5 Conclusion
In this paper, the problem of partial label learning is studied where a novel approach VALEN proposed. We for the first time consider the instance-dependent PLL and assume that each partially labeled example is associated with latent label distribution, which is the essential labeling information and worth being recovered for predictive model training. VALEN recovers the latent label distribution via inferring the true posterior density of the latent label distribution by Dirichlet density

---

VALEN VALEN NDN
VALEN VALEN MON
135
Test Accurayyte
Test Accuracyiis
Average Fllpping Probabilky
Average Flipping Probability
Epoch
(a) Instance-dependent version
(c) Convergence of Figure 2: Further analysis of VALEN KMNIST.
(b) Uniform version
parameterized with an inference model and deduce the evidence lower bound for optimization. In addition, VALEN iteratively recovers latent label distributions and trains the predictive model in every epoch. The effectiveness of the proposed approach validated via comprehensive experiments on both synthesis datasets and real-world PLL datasets.
Acknowledgments
This research was supported by the National Key Research & Development Plan of China (No. 2018AAA0100104, No. 2018AAA0100100) National Science Foundation China (62125602, 62076063), China Postdoctoral Science Foundation (200020))) Jiangsu Province Science Foundation for Youths (BK20210220).
References
[1] Arthur Asuncion and David Newman. UCI machine learning repository, 2007.
[2] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463 482, 2002.
[3] Forrest Briggs, Xiaoli Z Fern, and Raviv Raich. Rank-loss support instance machines for MIML instance annotation. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 534-542, Beijing, China, 2012.
[4] Ching-Hui Chen, Vishal M Patel, and Rama Chellappa. Learning from ambiguously labeled face images. IEEE Transactions on Pattern Analysis and Machine Intelligence, press.
[5] Yi-Chen Chen, Vishal M Patel, Rama Chellappa, and P Jonathon Phillips. Ambiguously labeled learning using dictionaries. IEEE Transactions on Information Forensics and Security, ((12):076-20088 2014.
[6] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv: 1812.0111 2018.
[7] Timothee Cour, Ben Sapp, and Ben Taskar, Learning from partial labels. Journal of Machine Learning Research, 1501a)):1011-1536 201
[8] Lei Feng and Bo An. Leveraging latent label distributions for partial label learning. In IJCAI. pages 2107-2113, 2018.
[9] Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-lahel learning. Advances in Neural Information Processing Systems, 2020.
[10] Michael Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterization gradients. Advances in Neural Information Processing Systems, 2018.
10

---

[11] Xin Geng, Label distribution learning. IEEE Transactions on Knowledge and Data Engineering. 28 2016.
[12] Xin Geng, Chao Yin, and Zhi-Hua Zhou. Facial age estimation by learning from label distri- butions. IEEE transactions pattern analysis and machine intelligence, 35(10):2401-241) 2013.
[13] Matthieu Guillaumin, Jakob Verbeek, and Cordelia Schmid. Multiple instance metric learning from automatically labeled bags of faces. In Lecture Notes in Computer Science 6311, pages 634 647. Springer, Berlin, 2010.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1eXiv::503.0253 2015.
[15] Eyke Hüllermeier and Jürgen Beringer. Learning from ambiguously labeled examples. Intelli- gent Data Analysis, 10(5):419 439, 2006.
[16] Takashi Ishida. Gang Niu. Aditya Menon, and Masashi Sugiyama. Complementary-label learning for arbitrary losses and models. In International Conference on Machine Learning. pages
PMLR, 2019.
[17] Rong Jin and Zoubin Ghahramani. Learning with multiple labels. In Advances in Neural Information Processing Systems 15, pages 897-904, Cambridge, MA, 2003.
[18] Julian Katz-Samuels, Gilles Blanchard, and Clayton Scott. Decontamination of mutual contami- nation models. Journal of machine learning research, 20(41), 2019.
[19] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, Banff, AB, Canada, 2014.
[20] Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07303 2016.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[22] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applicd to document recognition. Proceedings of the IEEE, 86(1 (6(11))2278-2)) 1998.
[23] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896, 2013.
[24] Liping Liu and Thomas G Dietterich. A conditional multinomial mixture model for superset label learning. Advances in Neural Information Processing Systems 25, pages 557-565, Cambridge, MA, 2012.
[25] Jie Luo and Francesco Orabona. Learning from candidate labeling sets. In Advances in Neural Information Processing Systems 23, pages 1504-1512. Cambridge, MA, 2010.
[26] Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. International Conference on Machine Learning, pages 6500-6510. PMLR, 2020.
[27] Rafael Müller, Simon Korbblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint arXiv: 02629, 2019.
[28] Nam Nguyen and Rich Caruana. Classification with partial labels. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 381 381-389, Las Vegas, NV, 2008.
[29] Hieu Pham, Zihang Dai, Qizhe Xie, and Quoe V Le. Meta pseudo labels. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11557-11568, 2.21.

---

[30] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2888 18-6826, 2016.
[31] Cai-Zhi Tang and Min-Ling Zhang. Confidence-rated discriminative partial label learning. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 2611 2611-2617, San Francisco, CA, 2017.
[32] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist. minis:: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: /
2017.
[33] Ning Xu, Yun-Peng Liu, and Xin Geng. Label enhancement for label distribution learning. IEEE Transactions on Knowledge and Data Engineering, 33(4):1632 1643, 2021.
[34] Ning Xu, Jiaqi Lvv, and Xin Geng. Partial label learning via label enhancement. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5557-5564, 2019.
[35] Ning Xu, Jun Shu, Yun-Peng Liu, and Xin Geng. Variational label enhancement. In Proceedings of the International Conference on Machine Learning, pages 10597-10606, Vienna, Austria, 2020.
[36] Yao Yao, Jiehui Deng, Xiuhua Chen, Chen Gong, Jianxin Wu, and Jian Yang. Deep discrimina- tive cnn with temporal ensembling for ambiguously-labeke image classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12669-12676, 2020.
[37] Fei Yu and Min- Zhang. Maximum margin partial label learning. Machine Learning. 106(4):573-535 -93, 2017.
[38] Zinan Zeng, Shijie Xiao, Kui Jia, Tsung-Han Chan, Shenghua Gao, Dong Xu, and Yi Ma. Learning by associating ambiguously labeled images. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 708-715, Portland, OR, 2013.
[39] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3713-3722, 2019.
[40] Min-Ling Zhang and Fei Yu. Solving the partial label learning problem: An instance-based approach. In Proceedings of the 24th International Joint Conference on Artificial Intelligence, pages 4048 4054, Buenos Aires, Argentina, 2015.
[41] Min-Ling Zhang, Fei Yu, and Cai-Zhi Tang. Disambiguation-fres partial label learning. IEEE Transactions on Knowledge and Data Engineering, 29(10):2155-2166 2017.
[42] Min-Ling Zhang, Bin-Bin Zhou, and Xu- Ying Liu. Partial label learning via feature-aware disambiguation. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1335-1344, 2016.
[43] Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature dependent label noise: a progressive approach. arXiv preprint arXiv:2103.077 2021.
12


Reviews:
[{"review_id": "JCvtdavw7Ws", "reviewer": "Reviewer_BDBH", "summary": "This work approaches the feature-dependent partial-label learning problem where each training example has a set of candidate labels that are generated by feature-dependent procedure rather than random procedure. This paper assumes that each instance in PLL is associated with a latent label distribution which is the essential labeling information in partially labeled examples and worth being leveraged for predictive model training. Inspired by this, the authors propose a novel method to enhance the labeling information by recovering the latent label distribution and train the predictive model with recovered label distributions. In the experiment, the authors show the effectiveness of the method in the benchmark and real-world datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a novel partial label learning method for the feature-dependent generating procedure with latent label distribution. The example in Figure 1 motivates the problem well, which shows that the incorrect label with a high degree in the latent label distribution is more likely to be annotated as the candidate label. This is the first attempt to consider the cause of the candidate labels in feature-dependent partial label learning datasets to motivate the novel solution of solving the problem with the latent label distribution. Besides, the authors propose a posterior inference model to approximate the posterior density of latent label distribution with Dirichlet density and optimize the model with the derived evidence lower bound. The predictive network is trained on the proposed empirical risk estimator by leveraging the candidate labels as well as the label distributions. \n\nThe paper is clearly written and well organized. Comprehensive experiments are performed on synthetic as well as real-world data sets to show the effectiveness of the proposed approach. Besides, the authors conduct further experiments to validate the helpfulness of the latent label distribution for partial label learning. \n\nNevertheless, the literature review should be more focused on describing the works that are more related and should give more details about label distribution learning. As label distributions might be more interesting to those people working on other more common machine learning problems, the authors could give more discussion about the latent label distribution in the conclusion.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See my main review. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "IEUupr3CIq", "reviewer": "Reviewer_2yTr", "summary": "This paper studies the problem of feature-dependent partial label learning, which is an interesting weakly supervised learning problem where the candidate labels of each instance are feature-dependent. Accordingly, the first attempt to feature-dependent partial label learning with latent label distribution is proposed.  The performance advantage of the proposed approach over state-of-the-art approaches is clearly validated via extensive experimental studies.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The problem studies in this paper, i.e., feature-dependent partial label learning, is interesting and practical since the incorrect labels related to the feature are more likely to be picked as candidate label in partially labeled data. As far as I know, this is the first study that focuses on the feature-dependent setting in partial label learning. \n2. The latent label distribution developed in this paper is well motivated and clearly presented. Based on the latent label distribution, the paper recovers the latent label distributions and train the predictive model alternately. \n3. This paper infers the posterior density of the latent label distribution by leveraging the approximate Dirichlet density with the evidence lower bound.  \n4. Comprehensive experiments are performed on synthetic as well as real-world data sets to show the effectiveness of the proposed approach. \nGenerally, the paper is very well organized and is easy to follow. It studied an important problem for partial label learning and proposed a novel method. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The paper could give more discussions on the different results on the datasets corrupted by the feature-dependent generating procedure and the uniform generating procedure. \n2. It seems that classical partial label learning approaches achieve better performance than DNN-based approaches on Lost and MSRCv2. The authors should give more discussion about the reasons why DNN-based methods achieve poor performance on these two datasets.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "AzyFOqO6Vnq", "reviewer": "Reviewer_tK2X", "summary": "This paper proposes an algorithm for learning from feature-dependent partial labels. The paper is based on a latent distribution over the weak labels which is estimated during training. A variational approach is used to make approximate inference computationally tractable. The experiments show that the proposed algorithms outperform many algorithm in the state of the art. Experiments are done using real data with synthetic label corruptions and also with real weak labels.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Clarity: despite the mathematical complexity of the inference process, the main ideas of the paper are clear.\n\nOriginality: the majority of papers about PLL and, in general, weak label learning, assume (explicitly or not) that the corruption process is independent on the features. This paper is a serious attempt to overcome this limitation.\n\nQuality: the paper is technically correct.\n\nSignificance: the experimental results show that removing the feature independent assumption, the performance of the classifiers based on weak labels can be improved. The advantages of the proposed method are more clear with the synthetic corruptions, as expected. The experiments using real-world PLL datasets are diverse, but the proposed method shows some advantage over many state of the art method in three out of five datasets.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A brief mention to a potential negative societal impact is included in the paper, related to the potential employment destruction as a consequence of reducing the need of human labelers.", "review_text": "", "extra_scores": {"ethical_concerns": "No relevant issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is the first to address partial label learning in the setting where the partial label depends on the feature vector. The authors introduce a principled approach to solve this problem and demonstrate some state-of-the-art empirical performance. This paper will be well-cited in the PLL literature and serve as an important benchmark for future research. The reviewers make a handful of comments that should be reflected in the final version. A pertinent missing reference is Katz-Samuels et al., Decontamination of Mutual Contamination Models, JMLR 2019.
Decision: Accept (Spotlight)