Partial Multi-Label Learning with Probabilistic Graphical Disambiguation
Authors: Jun-Yi Hang
Min-Ling Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Machine learning
multi-label learning
partial multi-label learning
label disambiguation

Abstract: In partial multi-label learning (PML), each training example is associated with a set of candidate labels, among which only some labels are valid. As a common strategy to tackle PML problem, disambiguation aims to recover the ground-truth labeling information from such inaccurate annotations. However, existing approaches mainly rely on heuristics or ad-hoc rules to disambiguate candidate labels, which may not be universal enough in complicated real-world scenarios. To provide a principled way for disambiguation, we make a first attempt to explore the probabilistic graphical model for PML problem, where a directed graph is tailored to infer latent ground-truth labeling information from the generative process of partial multi-label data. Under the framework of stochastic gradient variational Bayes, a unified variational lower bound is derived for this graphical model, which is further relaxed probabilistically so that the desired prediction model can be induced with simultaneously identified ground-truth labeling information. Comprehensive experiments on multiple synthetic and real-world data sets show that our approach outperforms the state-of-the-art counterparts.

Partial Multi-Label Learning with Probabilistic Graphical Disambiguation
Jun-Yi Hang, Min-Ling Zhang"
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China Key Laboratory of Computer Network and Information Integration (Southeast University). Ministry of Education, China {hangjy, zhangml (seul edu...edu..ccm
Abstract
In partial multi-label learning (PML), each training example is associated with a set of candidate labels, among which only some labels are valid. As a common strategy to tackle PML problem, disumbiguation aims to recover the ground-truth label- ing information from such inaccurate annotations. However, existing approaches mainly rely on heuristics or ad-hoc rules to disambiguate candidate labels, which may not be universal enough in complicated real-world scenarios. To provide a principled way for disambiguation, bilistic graphical model for PML problem, where a directed graph is tailored to infer latent ground-truth labeling information from the generative process of partial multi-label data. Under the framework of stochastic gradient variational Bayes, unified variational lower bound is derived for this graphical model, which is further relaxed probabilistically so that the desired prediction model can be induced with simultaneously identified ground-truth labeling information. Comprehensive experiments on multiple synthetic and real-world data sets show that our approach outperforms the state-of-the-a counterparts.
make a first attempt explore the proba-
Introduction
Partial multi-label learning aims to induce multi-label predictor from inaccurately annotated examples, where a set of candidate labels is assigned to each training example but only some of these candidate ones are valid. The problem of PML naturally arises in many real-world applications [39, 51, 52, 20). For instance, as shown in Figure 1, the data collected from the crowdsoureing platform is inevitable to contain inaccurate supervision information due to potential unreliable annotators, where invalid labels exist in the set of candidate labels.
Formally, let X = Rd denote the input space and y (2,, le) denote the label space with / class labels. partial multi-label example denoted as (x, S), where x X' is feature vector and S S' is its set of candidate labels. As a basic assumption of PMI., the ground-truth labels reside in the candidate label set, i.c. Y C S, and are concealed from the learning algorithm. Given partial multi-label data set D [x,, S,) < m) the goal of PML is to derive multi-label predictor h: X + 2 which can accurately predict all the ground -tuth labels for an unseen instance.
A straightforward strategy for tackling the problem of PML is to simply treat all the candidate labels as valid ones and then apply off-the off-th multi-labee label learning approaches [50, 23] induce the predictor. This naïve strategy is obviously suboptimal as the learning process will be significantly misled by the noisy labels in the candidate label set.
*Corresoondigg author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Candidate labels (valid ones in red)
free
flower light
mountain
car
road
people
An exemplary partial multi-label learning scenario. Among the set of 7 candidate labels given by crowdsourcing annotators, only labels are valid including tree, light, mountain, and road.
Figure 1:
Considering that the ground-truth labels concealed in the candidate label set, it is common strategy of existing approaches to learn from partial multi-label data by disambiguation, i.e. recovering the ground-truth labeling information from candidate labels. The basic idea of disambiguation is to estimate the labeling confidence for each candidate label from observed inaccurately annotated examples, which reflects how likely the candidate label can be a ground-truth one. For example, some approaches introduce the smoothness assumption to directly elicit the labeling confidence via label propagation [48, 26], label enhancement [31, 41], or cluster assignment [39, 3]. etc. While some other works heuristically believe that the noisy labels in the candidate label set are generally sparse, so that the noisy label matrix [30. 29] or the noisy label identifier [35] can be learned with sparsity regularization. Although feasible results have been achieved in these approaches, a potential limitation that they rely on heuristics or ad-hoe rules for disambiguation, which may not widely applicable in some challenging scenarios. For example, sparsity on underlying noisy labels can hardly be satisfied in some extremely low-quality annotation scenarios.
To improve this, we explore a principled manner for disambiguation, which is regarded as a task of latent variable inference in this paper. Accordingly, novel partial multi-label learning approach named PARD, i.e. Partial multi-label learning with probAbilistic-ge gRaphical Disambiguation, is pre- sented. By regarding the concealed ground-truth labels as latent variable, a directed graphical model is tailored to describe the generative process of partial multi-label data. With the directed graphical model, unified variational lower bound of the data log- likelihoodd is derived via variational inference. which allows to disambiguate the candidate labels and induce the prediction model simultaneously. Specifically, the ground-trutt labeling information identified by fitting the generative model of PML. data, where the intrinsic structure information of data can be captured for principled disambiguation. While the desired prediction model optimized with a confidence-smoother cross entropy loss. Comprehensive experimental studies show that our approach performs better than well-established PML algorithms.
The rest of this paper is organized as follows. Section 2 briefly reviews related works. Section presents details of the proposed PARD approach. Section reports experimental results over wide range of synthetic and real-world data sets. Section 5 concludes this paper.
2 Related Work
Partial multi-label learning is an emerging weakly-supervised learning problem, which has received wide attention in recent years [39, 40, 36). Here, would like to first make brief review on two closely related learning problems, i.c. multi-label learning [50, 23] and partial label learning [6. 24].
Multi-label learning (MLL) deals with the problem where an example can be associated with multiple labels simultaneously. According to the order of label correlations considered, existing MLL approaches can be grouped into three categories, including first-order approaches which treat each class label independently [2. 14], second-order approaches which explore label correlations between pairwise class labels [8, 53], and high-order approuches which consider label correlations among subset or the whole set of class labels [32, 44]. With the same goal, both MILL and PMI. aim to learn nulli predictor from multi-semantic objects. Nevertheless, the key difference between these

---

two learning problems lies in whether the ground-truth labels are accessible or not during the learning process, which also makes PML more challenging than MLL since the supervision information is inaccurate.
Partial label learning (PI..) deals with the problem where each example is associated with multiple candidate labels, among which only one is valid. The problem of PLL can be tackled in several different strategies, such as transforming into well-studied supervised learning problems [49, 34], disambiguating the candidate labels [9, 45, 43), or learning with theoretically consistent algorithms [24, 10), etc. Conceptually speaking, PLL and PML possess similar form of supervision, where false positive labels reside in the candidate label set. However, the task of PML is more complicated than PLL since the desired predictor a multi- one in PML instead of single-label one in PLL and the number of ground truth labels is further unknown.
To solve the PML problem, the most commonly employed strategy is disambiguation, i.e. recovering the ground-truth labeling information from candidate labels. Existing approaches mainly rely on some heuristics or ad-hoc rules to disambiguate candidate labels, which can be roughly grouped into three categories in terms of the heuristics or rules exploited, including smoothness assumption- approaches, low-rank constraint-based approaches, and sparsity regularization-bused approaches. The first category of approaches rely on smoothness assumption that close instances may share similar labels for disambiguation. As a seminal work, an iterative procedure is proposed in [39] to optimize the labeling confidence of each candidate label and the prediction model alternatively, where the labeling confidence scores are obtained by assuming cluster structures exist in the feature space. Follow-up works employ smoothness assumption to design different disambiguation mechanisms, such as propagating labels in a graph based on instance similarity [48, 26), recovering numerical labels with label enhancement [31, 41, 42], or performing implicit disambiguation by jointly embedding [12]. Low-rank constraint-based approaches turn to the low-rank property for disambiguation. For example, low-rank matrix decomposition is exploited to recover the ground-truth labeling information in [47] and a low-rank subspace is learned in [25] to elicit credible labels from improved subspace representations. While sparsity regularization-
approaches [30, 29, 35] impose sparsity on noises hiding in candidate label set, which facilitate the disambiguation in a roundabout manner. A potential limitation of these approaches is that the above heuristics or ad-hoc rules can be hardly held in some challenging scenarios, which may lead to suboptimal performance of induced prediction model.
Therefore, quite natural to consider whether it is possible to perform disambiguation in a principled manner, which is still an underexplored direction with only a few attempts. PML-GAN [46] recovers ground-truth labeling information with an instance reconstruction process, corresponding to a minimax adversarial game to implicitly model instance distribution. While MILI-PML [11] disambiguates candidate labels by maximizing the mutual information between features and identified ground-truth labels. With totally different methodology, our approach regards disambiguation as a task of latent variable inference, which is formalized in a maximum likelihood framework with a concise variational lower bound of the log- likellhooh on observed PML data. As another pioneer work towards this direction, PML-MD [37] devises a meta-learning framework, where disambiguation is performed with the guidance from an accurately annotated validation set. Moving one step forward, we make a first attempt to tackle the problem via modelling the generative process of partial multi- label data with a tailored directed graphical model. Compared with PML-MD, our approach is more flexible since there is no need to include a clean validation set for disambiguation, which may not be easily accessible in many real-world scenarios. We will detail our approach in the next section.
3
The PARD Approach
3.1
Overview
To tackle the problem of partial multi-label learning, PARD disambiguates the candidate labels and induces the prediction model simultaneously by optimizing towards unified variational lower bound of the data liglliiklooodd. In the following content, we will present the tailored directed graphical model which describes the generative process of partial multi-label data. Then, we derive variational lower bound of the data log-likelihood and further explain how to relate it with disambiguation and predictor induction. Finally, essential implementation issues w.r.t. the optimization procedure are detailed.
3

---

Some necessary notations have been introduced in section 1. For notation briefness, a t-dimensional indicator vector s {0, 13' is utilized denote the set of candidate labels S, where Sy. indicates l E S and 8; = 0 otherwise. Similarly, I-dimensional indicator vector E {0, 1 is utilized to denote the set of ground-truth labels Y.
3.2 Probabilistic Graphical Disambiguation Framework
Considering that only the instance and its associated candidate labels can be observed in the partial multi-label training set, we tailor directed graphical model to describe the generative process of partial multi-label data, which involves an unobserved latent variable as the ground-truth labels. The generative process consists of three steps: (I) sample instance x from the marginal distribution pe(x); (2) sample a latent y as the ground-truth labels of the instance x from the ground-truth class posterior distribution pe ( x): (3) corrupt the instance's 's ground-truth labels to obtain its candidate labels by pe (s|x, y). Accordingly, the joint probability pe y, y, s) can be factorized as
po (x y, y, s) (x)))po (y)x))osssxx))
Given the partial multi-label training set D, the above directed graphical model can be learned via maximizing the log-likelihood on the observed data. However, it is generally challenging to directly estimate the parameter of a directed graphical model by likelihood maximization due to computational intractability [17]. To make the optimization tractable, the variational lower bound of the log-likelihood derived as follows
[s(s|x, (issxx,)jun ())p(y))))))))))))
logpo(s)x)) 2 L(x, (;,)) g flog
( x)
where 4o (y (x. s) is the introduced variational posterior to approximate the true posterior po (y|x, s). L(x, ()) is the derived variational lower bound surrogate loss function of the nieliiol,oo.d, which can be rewritten as
L(x, s; 0) = (yxs(y/xss) (o y)] - KT[ga s) (polpo
(3)
where KE[-I1-] denotes the KL-divergence between two distributions. Here, the meaning of the ground-truth class posterior distribution (y x) is comprehensible, which is actually the desired prediction model?. While the variational posterior qo (o(y/x, s) (a.k.a. inference model) attempts to disambiguate the candidate label set by inferring the most probable ground-truth labels from which the candidate labels could have been corrupted given the instance x. Correspondingly, Po(s|x,) y) (a.ka. generative model) corrupts the identified ground labels to recover the observed candidate labels.
To move one step further for comprehending the learning behavior of the above three models, we reformulate the variational lower bound via unfolding the KL-divergence term
L(x, s: 0 ②) = EEo Po (s (, + H (ys (x)ss)]
+ Ex xxx))xxx [log Po (y/xx)]
(4)
where H denotes the entropy of a distribution. In the above objective, the first two terms present an retropy-regularizee autoencode process (w.r.t. s), with which the intrinsic structure information of data can be captured and utilized for disambiguation. While the last term actually a cross entropy loss for inducing the prediction model with identified ground-truth labeling information go ( |x, s). By optimizing all these terms in a unified variational lower bound, the models can gradually capture the underlying generative process of partial multi-label data, so that the candidate label set is disambiguated and the desired prediction model induced simultancously.
3.3 Additional Training Techniques
In this section, we present additional techniques that allow the derived variational lower bound to be optimized efficiently.
We omit the optimization for the marginal distribution on x, as it is not the focus of our work. Detailed derivation process can be found in Appendix B.
'With a little abuse of notations, distribution, e.g. Pe (y xx), is also called a model which actually denotes the model to parameterize the distribution. when the context is clear. All the distributions involved in Eq. (3) are instantiated as multivariate Bemoulli distributions and are parameterized by neural networks.

---

Algorithm Pseudocode of the Optimization Procedure for PARD
Inputs: Training set D {(x,, S S m), batch size and maximal iteration T.
Process:
1: Initialize model parameters 0, d;;
2: for = 1: T do
3: Sample batch of training examples B = [xxx, S 6) from training set D;
4: Compute unbiased estimator of the variational lower bound on by Eq. (6) and Eq. (7);
5: Update model parameters 8, & via gradient ascent.
6: end for
Outputs: Model parameters 0,
Continuous Relaxing with Gumbel-Softmax Trick. is quite challenging to compute the first expectation term in Eq. (3). Actually, it has high complexity of O22") analytically compute the first expectation term in Eq. (3), since the ground -truh labels can take 2 different values. In addition, estimating it by Monte Carlo sampling
con-iifferentiable, since sampling from discrete variational posterior (y(x,, prevents gradients from flowing through the inference model.
x,
To circumvent this problem, exploit Gumbel-Softmax trick [13, 27] to smooth the sampling process so that the expectation term can be efficiently estimated by Monte Carlo sampling a differentiable manner. Specifically, the Bemoulli variable y is relaxed by the binary Concrete variable c BinConnceeeep..T which is a continuous alternative with the reparameterization form as
+ expl expppp(loge (log + 1)/T]
where P__. and p is the parameter of the multivariate Bernoulli distribution ( 8). 1 sampling from Logistic distribution and 0 is temperature parameter. In the limit - 0, a binary Concrete variable smoothly converges to its Bernoulli counterpart. With the above sampling trick, an unbiased estimation of the expectation term can be obtained and gradients w.r.t. the distribution parameter (or w.r.t. the parameter of the inference model equivalently) are well-defined by the chain rule
(xo(iix.)) log poggo(s(xx ~ log Po (0)
where ( ~ BinComcrete(p.r)
Closed-Form Solution of the KL-Divergence Term. The KL-divergence term in Eq. (3) can not be integrated analytically in general cases. To make tractable, we exploit mean-fich field approximation technique and derive closed-form solution of the KL-divergence term as follows
2 log uE +11 1 log
KTlgo(y(xxs))lpo(yss
Σ p k=1
yk
where pht (yn 1|x, and pd po (y=e 1/x). Detailed derivation process be found in Appendix C.
With these training techniques, the derived variational lower bound can straightforwardly opti- mized with standard stochastic gradient methods. Algorithm summarizes the pseudocode of the optimization procedure, where the candidate labels and the prediction model are updated simultane- ously.
Experiments
4.1
Experimental Setup
4.1.1
Data Sets
For comprehensive performance evaluation, five real-world and number of synthetic PMI data sets employed in this paper. Table summarizes detailed characteristies of cach data set. Specifically,
5

---

Table I: Characteristies of the experimental data sets. The fisrt five ones real-worlls PML data
sets and the last five ones are multi-label data sets employed to generate synthetic PML data sets.
Dataset
#Examples
#Fcatures
#Class Labels
Cardinality
Domain
YeastBP
6139
61.99
217
5.54
Biology
YeastCC
6139
6139
1.35
Biology'
YeastMF
6139
6139
39
1.01
Biology
Music_ cerii__emtionn
6833
98
2.42
Music
Music Musie_style
6839
98
10
1.44
Music
core15k
5000
499
374
3.52
Images?
revl-sl
6000
944
101
2.88
Text"
Corellók-sl
13766
500
153
2.86
Images?
12pric
19627
1000
291
5.72
Images?
espgame 20770
1000
268
4.69
Images?
http://palm... seu edu. cdu..c//zhangm//
2 http::/mulan..eee laa.llaanssuuceffrrrree www/dattstee....ee.e html
the first five data sets real-world PML data sets, where candidate labels are collected from web users and manually examined to specify the ground-truth labels. While the last five data sets, including corel5k, revl-sl, Corel16k-sl. iaprtc12 and espgame, are multi-label data sets.
Following [35], synthetic PML data set is generated from one multi-label data set by adding random labeling noise. Accordingly, for data sets with more than 100 class labels, filter out their rare labels to keep the 15 most frequent labels and remove instances without relevant labels. Then, we randomly flip the irrelevant labels of an instance until the number of irrelevant labels in the set of candidate labels equals to 7% number of ground-truth labels. We vary the value of y in the range 20 synthetic PML data sets are generated for
of (100, 150, 200, 250), that total of experimental studies.
4.1.2 Evaluation Metries
Five widely-used evaluation metrics for multi-label learning are employed to evaluate the performance of each approach, including Average precision, Hamming loss, One-error, Coverage and Ranking loss. Detailed definitions on these metrics can be found [50].
4.1.3 Implementation Details
The inference model (y|xx, s) and the generative model ps(s)x, are instantiated by fully- connected neural networks with ReLU activations, where the hidden dimensionalities are set to [256; 512; 256] and [256; 512) respectively. For fair comparison with existing PML approaches, the prediction model is implemented as a linear model. To compute the objective function in Eq. (3), trade-off parameter is introduced for the KL-divergence term and Monte Carlo sampling with sampling number L = is conducted to estimate the first expectation term, where the temperature parameter 2/3 as suggested by [27]. In the following experiments, we objective function is still a valid lower bound of the data log-likelihood. For network optimization, Adam with batch size of 128, weight decay of 10-4, momentums of 0.999 and 0.9 employed. In this paper, all experiments are conducted on one V100 GPU.
1 so that the
4.2 Comparative Studies
PARD* is compared against six well-established PML approaches with parameter configurations suggested in respective literatures:
FPMI. [47]: FPML. employs the low-rank approximation of the instance-label association matrix to estimate the labeling confidence and then trains multi-label predictor. [} 0.1, = 1, A3 = 10]
*Code package of PARD is publicly available
seu, edu, wwwzhangml/fille/Peeee

---

Table 2: Predictive performance of each comparing approach (mean deviation) in terms of Average precision, where /o indicates whether PARD is significantly superior/ifferior to one comparing approach via paired t-test at 0.05 significance level. (J) indicates the larger (smaller) the value, the better the performance. Best results are shown in boldface.
Average precision
Data sets
7%
FIML
PARVLS
PMI-NI
PMI.-M
UPML-HI
UPMI-RL
PARD
YeusiBP
0.202.10.013
0.0591:0000
0.43010.018:
0.31010.016
0.392:10:200
0.17010.00%
0.417
YeastCC
0.375=00.019
0.145:-0.009
0.610=0.022
0.527+-0.027
0.596=0.0022
0.459+0.021
0.622-0.020
YeasMMF
0.279:10.020
0.114::0.007
0.471:0.022+
0.42210.027+
0.451=0.016
0.329:100233
0.491=00028
Music_ Muxic_emmtionn
0.501
0.603
0.600.10.012+
0.646.10.011
0.656_0.012:
0.63910.0112
0.650.00007
style
0.702=000114
0.720=0.012
0.733=0.012+
0.71620.012
0.734=0.010+
0,705:00012+
0.742=00009
100
0.005:0.0014
0:001::0.017
0.473:0.001.
0.491:.0.019
0.516::0.015.
0,514:10.015
0.533-0.013
150
0.478-+0.013+
0.408+00001
0.453+0.013+
0.479+0.015
0.514-0.015
0.512+0.015
0.013
core15k
200
0.474:00.013
0.395100021
0.438:0.0022
0.472:0.011+
0.503.0.009
0.511
0.5161100011
250
0.470:10.012
0:0022
0.426:10.01
0.4701.00019
0.502:00:013
0.507=00014
0.013-00.02
100
0.682::0.010
0.520:0.021
0.693:0.0144
0.678:0.010
0.712:40.013+
0.689
150
0.678-t0.011
0.513+0.0022+
0.644-0.11.
0.656+0.011
0:705-0.013+
0.690+0.008
0.716-0.010
rev1-sl
200
0.673.10.010+
0.0951:0.021
0.64810.010+ 0.64910.000
0.704.00012+
0.85110.005
0.010
250
0.666::0.01 I
0,488 1:0016
0.630:00.01 I
0.640:00001+
0.692::001.5
0.672:..009
0.706 0.009
100
0.535::0.010+
0.430:0.010+
0.522=0.011+ 0.518=0.0111
0.539=0.012+
0.5251000077
0,551=0.012
Corc116k-s1
150
0.529=00.008
0.423++0.012 12
0.499+0.001 1
0.512+0.011+
0:532=0001
0.522+0.009+
0.545
200
0.523=00010+
0.404:0.012+
0.487=0.013+
0.507=0.012
0.504=0.011
0.519:00009.
0.536=0.014
250
0.514:0.011
0.389
0.476:0.0133
0.501
0.507=000114+
0,516++0.00$
0.014
100
0.603:10.006
0,5911:0.008
0.59920.009
0.60010.008
0.60120.009
0.563110.008+
0.621=0.011
150
0.60010.0006
0.005
0.580.10.010+
0.003
0.597.0.0077
0.559
0,009
iaprte12
200
0.597=00.00$
0.88
0.563=0.006-
0.569=0.009+
0.600=0.008
0,554+0.008
0.610=0.008
250
0.588:000009
0.561=00008
0.543=000009
0.56020.009
0.582=00008
0,547100006+
0.601=0.009
100
0.498
0.472:0.007+
0.478:00.00$
0.49110.009
0.001=0.008
0.066:10.009
0.515=00008
150
0.496.00.007
0.456.0.007
0.45910.0077
0.48110.007
0.50910.008
0.40110.006+
0.511
ospgame
200
0.495=00.007
0.445:0.007
0.446:000005
0,473=0.007
0:475:0.012+
0.4661:0006
0.508=0.008
250
0.489:200009
0.427
0.43110.000
0.46710.007
0.49020.008.
0.46010.006*
0.501=0.009
PARVLS [48]: A two-step approach which elicits credible labels via label propagation and then induces multi-label predictor by virtual label splitting. [k- 10, 0.95, and thr = 0.9]
PML-NI [35]: PML-NI performs disambiguation by learning a noisy label identifier with sparsity regularization. [3 = 0.5, 0.5, and search for A E {1, 10, 100}]
PML-MD [37]: PML-MD disambiguates between ground -rrut and noisy labels in meta- learning fashion and learns multi-label predictor with a confidence-weighted ranking loss.
UPML-HL [36]: A disambiguation-fiee approach which directly induces the prediction model by optimizing an unbiased estimator of the Hamming loss.
UPML-RL [36]: A disambiguation-free approach which directly induces the prediction model by optimizing an unbiased estimator of the Ranking loss.
Following [37], we take out 10% examples in each data set as hold-out validation set, so that PML-MD can use this clean validation set to perform meta-disambiguation. The remaining 90% examples are randomly splitted into training set and test set with ratio of 9:1 for training and evaluation repectively. During training, PML-MD performs meta-learning on the noisy training set and the clean validation set. While other approaches including ours only employ this validation set to search hyperparameters. In PARD, the trade-off paramter a is searched in {1, 2,5, 5, 10, 20, 50, 100]. For fair comparison, all deep approaches adopt linear model to instantiate the predictor and share the same optimizer with the learning rate searched in 1, For each data set, we repeat the random splitting process for ten times and record the average predicitve performance across ten training/test trials.
Due to page limit, Table 2 and Table 3 report detailed experimental results in terms of Average precision and Ranking loss. Results on other metries can found in Appendix A. Furthermore, the Wilcoxon higned-ranks test [33] at 0.05 significance level is conducted to analyze whether PARD achieves statistically superior performance to other comparing approaches. Table summarizes the p-luuue statistics each evaluation metric. Based on these results, it is impressive to observe that:
7

---

Table 3: Predictive performance of each comparing approach (mean=std. deviation) in terms of Ranking loss, where / /o indicates whether PARD is sigmificantly superior/iferior to one comparing approach via paired t-test at 0.05 significance level. f (L) indicates the larger (smaller) the value, the better the performance. Best results are shown in boldface.
Ranking 4
Data sets
7%
FIML
PARVLS
PMI-NI
PML--M
UPML-HI
UPMI-RL
PARD
YeusiBP
0.308.00.007
0.4821:0000
0.21810.008 0.19710.012
0.222:100015
0.30910.011
0.175-10.01
YeastCC
0.305--0.016
0.400:-0.010+
0.156:0.001
0.190=0.0177
0.167:00.17
0,212+00015
0.015
YeasMMF
0.373:000019
0.533::00100
0.226:0.018
0.23510.011
0.239-0.023
0,286:10017
0.192-000155
Music_ Munii_emmttion
0.274.10.012
0.252.00.00
0.24910.010
0.231
0.223.10.099
0.23610.011e
0.228.10.007
style_style
0.160:000010
0.150=0.007
0.139:0.008
0.143=0.007
0.13840.007
0.149:0.007
0.137=0.009
100
0.003:
0.000
0.200:00.012
0.251::0.01
0.216::0.0144
0,229:10.013
0.223-0.011
150
0.260+0..133+
0.327+0.011
0.296+0.0115
0.260+0.0144
0.235+0.014
0.235-00013
0.233-0.010
core15k
200
0.263:10.011
0.338:00021
0.312:100033+
0.26410.0122
0.254.10:10
0.240-0.012
0.242:10.010
250
0.268.10.012
0.002
0.322:10.013
0.26710.015
0.257:000133
0.242 0.013
0.040
100
0.004:00000
0.222::00113+
0.00:
0.093
0.086:00.00$
0.087:10.033
0.003
150
0. 0.10-+0.005
0.240+0.0200
0.12++0.006 0066
0,105++0.005
0.089-0.004
0.089+0.004+
0.079
rev1-sl
200
0.107.10.000
0.262::00022+
0.11110.004+ 0.000
0.092:10.007
0.0931000033
0.084
250
0.004
0.207::0.017
0. 0.0000
0.115
0.090:000006
0.100::0.004
0.090
100
0.219:00000*
0,289:100077
0.247:0.000*
0.201
0.212:00006
0,216::00066
0.205=000006
Corc116k-s1
150
0.226:+0.006+
0.300=0.011
0,262:+0.00 000+
0.228+0..006
0.224=0.007
0.220+0.007+
0.212:0.007
200
0.235=00.007
0.329=:0011
0.274=0.011
0,232+0.0088
0.250=0.000
0.223=00006
0.221=0.009
250
0.245=00.009
0.009:0.010
0.286:0.0112
0.238
0.249:0.011
0.208
0.023-0.008
0.18910.000
0.207
0.192:10.05 0.206
0.19920.007
0.21010.004
0.180
150
0.19110.0004
0.212::0.005
0.222.10.007
0.205.10.005
0.202.10.006
0.20110.005
0,086
iaprte12
200
0. 19622.0004
0.227=0.0006
0.239=000006
0,214=0.005
0.203=0.006m
0.21620.005
0,191=0.005
250
0.204
0,241200.00$
0.257=0.000*
0.222:00000+
0.218=0.0077
0.221::0.066+
0.199=00005
100
0.252:00.005
0.2812:.006
00005
0.25
0.256:20.006
0.277:100008
0.241=00000
150
0.256.10.006
0.300
0.302.10.055
0.206.00.006
0.248.10.006
0.265.10.006
0.245-100006
ospgame
200
0.258=00.006
0.313::0.006
0.314=0.005
0,272:0.006
0.277=0.008
0.2691:0005.
0.248=0.006
250
0.266:20.006
0.33010.007+
0.3291100007
0.27410.006
0.269=0.007
0.275
0.255=00005
Table 4: Summary of the Wilcoxon signed-ranks test for PARD against other comparing approaches at 0.05 significance level. p-values are shown in the brackets.
PARD against
FPML
PARVLS
PML-NI win [1.8e-5]
PML-MD
UPML-HL
UPML-RL
Average precision
win [1.2e-5]
win| [1 [1.2c-5]
win [1.2e-5]
win [1.7e-5]
win [1.2e-5]
Homming loss
win [8.2e-3]
win [4.6e-5]
win [2.1e-5]
win [8.5e-5]
tie [7.0e-1]
win [3.5e-5]
One-error
win [1.2e-5]
win [1.2e-5]
win [4.6e-5]
win [1.2e-5]
win [1.8e-4]
win [1.2e-5]
Coverage
win [1.2c-5]
win [1.2c-5]
win [1.4e-5]
win [1,2c-5]
win [2.4e-5]
win [4.0e-5]
Ranking loss
win [1.2c-5]
win [1.2c-5]
win [1.2e-5]
win [1.2e-5]
win [2.0e-5]
win [2.5e-5]
Across all evaluation metrics, PARD achieves the best performance in 82,4% cases over all the 25 data sets.
As shown in Table 4, PARD achieves statistically better performance against other approaches which rely on heuristics or ad-hoc rules for disambiguation. The superior performance of PARD is consistent across almost all real-world data sets and synthetic data sets under varied noise levels, which provides a strong evidence for the effectiveness of probabilistic graphical disambiguation to facilitate partial multi-label learning.
Mcantime, PARD significantly outperforms PML-MD which also disambiguates PML data in a principled manner. Note that PML-MD requires an accurately annotated validation set to perform disambiguation during the learning process. The superior performance of PARD against PML-MD indicates that PARD is an effective and more flexible approach for principled disambiguation.
4.3 Further Analyses
4.3.1
Ablation Study
To validate that the disambiguation process in PARD is effective for facilitating partial multi-label learning, we implement a baseline which induces prediction model with cross-entropy loss by directly treating all the candidate labels as valid ones. As shown in Figure 2, clear performance degradation
8

---

24
626
PARD
Baseline
PARO
Baseline
Average procition
Ronking less
WewstUP
YeastCCC
Yeasthi
Music_emation
Music_style
YeastUP
YeastOC
YeastMf
Mask_emotion
Musi_style
Figure Predictive performance of PARD and the baseline terms Averuge precision and Ranking loss on real-world data sets.
Ivvverage precision
1.Average procision
Harriing lows
One-ernor
0.5
Oné-error
Performance
2.-
Ranking
Coverage
Performance
0.4
Caverage
Rancing loss
34-
6.2
100
Trade-off parmmeter
Trade-off
(a) YeastCC
(b) Music Music_style
Figure 3: Validation performance of PARD with varying trade-off parameter
is witnessed in the baseline model, which demonstrates the disambiguation process induced from the variational lower bound can facilitate partial multi-label learning.
4.3.2 Parameter Sensitivity
Figure gives some illustrative examples on how the performance PARD changes when the value of the trade-off parameter a changes. The performance of PARD is relatively stable as the value of changes within a reasonable range, which is desirable property when deploying PARD in real-worke applications. Similar results can be observed other data sets.
4.3.3 Running Time Comparision
For PARD, the training phase only involves pair of forward and backward computations among the inference, generative and prediction models, which is efficient thanks to the training techniques presented in section 3.3. Figure 4 illustrates the empirical training and test time of each comparing approach, which shows that PARD competitive with exisiting approaches in time overhead.
FPML
PARVLS
Test sole
PHL-NI
URML-11
UPML-RL
PARD
Trunisg
Training time of comparing approaches
Test time of comparing approaches
Figure 4: Running time (training/test) of each comparing approach on real-world data sets. For histogram illustration, the y-axis corresponds to the logarithm of running time.

---

Limitation
Our primary focus is on partial multi-label learning in this paper. However, there are other weakly- supervised multi-label learning problems that receive increasing attention from related community. The probabilistic graphical disambiguation framework of PARD currently has been tested on these problems, such as learning with partial labels which deals with incompletely annotated data 17, 19, 1. 4]. learning with single positive label which aims to induce an accurate multi-label predictor from multi-labe data annotated with only one positive labels [5, 28, 15, 38, 16, 22], and learning with general noisy labels where noise may exist in every label of inaccurately annotated data [18, 21, 36]. In addition, we do not implement PARD and investigate its behavior in large model environment, which are also important to practitioners. We will explore these points in the future.
Conclusion
In this paper, attempt towards principled disambiguation for partial multi-label learning is presented, where a directed graphical model is tailored to describe the generative process of partial multi-labod data. By maximizing the data log-likelihood on given partial multi-label data set with unified surrogate objective derived by variational inference, PARD achieves to disambiguate the candidate label set and induce the prediction model simultaneously. Comprehensive experiments show the superiority of our approach. Since disambiguation lies the heart of partial multi-label learning, we hope that PARD will encourage more future researches explore alternative implementations for principled disambiguation.
Acknowledgments
The authors wish to thank the anonymous reviewers for their helpful comments and suggestions. This work was supported by the National Science Foundation of China (62225602), and the Big Data Computing Center of Southeast University.
References
[1] Emanuel Ben Baruch, Tal Ridnik, Itamar Friedman, Avi Ben-Cohen, Nadav Zamir, Asaf Noy, and Lihi Zelnik-Manor. Multi-label classification with partial annotations using class- aware selective loss. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4754-4762, New Orleans, LA, 2022.
[2] Matthew R. Boutell, Jiebo Luo, Xipeng Shen, and Christopher M. Brown. Learning multi-label scene classification. Pattern Recognition, (7(9):1757-17) 2004.
[3] Nan Cao, Teng Zhang, and Hai Jin. Partial multi-label optimal margin distribution machine. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, pages 2198-2204, Montreal, Canada, 2021.
[4] Tianshui Chen, Tao Pu, Hefeng Wu, Yuan Xie, and Liang Lin. Structured semantic transfer for multi-label recognition with partial labels. Proceedings of the 36th AAAI Conference on Artificial Intelligence, pages 339-346, Virtual Event, 2022.
[5] Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic. Multi-label learning from single positive labels. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 933-942, virtual, 2021.
[6] Timothée Cour, Benjamin Sapp, and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research, 12:1501-1536, 2011.
[7] Thibaut Durand, Nazanin Mehrasa, and Greg Mori. Learning a deep convnet for multi-label classification with partial labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 647-657, Long Beach, CA, 2019.
[8] A. Flisseeff and J. Weston. kernel method for multi-labelled classification. Advances in Neural Information Processing Systems 14, pages 681-687, Vancouver, Canada, 2001.
10

---

[9] Lei Feng and Bo An. Partial label learning with self self-guided retraining. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, pages 3542-3549, Honolulu, HA, 2019.
[10] Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. In Advances in Neural Information Processing Systems 33, pages 10960, virtual, 2020.
[11] Xiuwen Gong, Dong Yuan, and Wei Bao. Understanding partial multi-label learning via mutual information. In Advances in Neural Information Processing Systems 34, pages 4147-4156, virtual, 2021.
[12] Xiuwen Gong, Dong Yuan, and Wei Bao. Partial multi-label learning via large margin nearest neighbour embeddings. In Proceedings of the 36th AAAI Conference on Artificial Intelligence, pages 6729-6736, Virtual Event, 2022.
[13] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-sofuue In Proceedings of the 5th International Conference on Learning Representations, Toulon, France, 2017.
[14] Bin-Bin Jia and Min-Ling Zhang. Multi-dimensional classification via selective feature aug- mentation. Machine Intelligence Research, 19(1):38-51, 2022.
[15] Youngwook Kim, Jae-Myung Kim, Zeynep Akata, and Jungwoo Lee. Large loss matters in weakly supervised multi-label classification. In Proceedings of the IEEE Conference on Computer Vision and Pattenn Recognition, pages 14136-14145, New Orleans, LA,, 2022.
[16] Youngwook Kim, Jae-Myung Kim, Jieun Jeong, Cordelia Schmid, Zeynep Akata, and Jungwoo Lee. Bridging the gap between model explanations in partially annotated multi-label classifi- cation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 1408-34177 Vancouver, Canada, 2023.
[17] Diederik Kingma and Max Welling. Auto-encoding variational bayes. Proceedings of the 2nd International Conference on Learning Representations, Banff, Canada, 2014.
[18] Himanshu Kumar. Naresh Manwani, and P. S. Sastry. Robust learning of multi-label classifiers under label noise. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pages 90-97, Hyderabad, India, 2020.
[19] Kaustav Kundu and Joseph Tighe. Exploiting weakly supervised visual patterns to learn from partial annotations. In Advances in Neural Information Processing Systems 33, pages 561-572, virtual, 2020.
[20] Jiaran Li, Richong Zhang, Samuel Mensah, Wenyi Qin, and Chunming Hu. Classification- oriented dawid skene model for transferring intelligence from crowds to machines. Frontiers of Computer Science, 17553177333,, 2023.
[21] Shikun Li, Xiaobo Xia, Hansong Zhang, Yibing Zhan, Shiming Ge, and Tongliang Liu. Estimat- ing noise transition matrix with label correlations for noisy multi-label learning. In Advances in Neural Information Processing Systems 35, pages 24199844198, New Orleans, LA, 2022.
[22] Biao Liu, Ning Xu, Jiaqi Lv, and Xin Geng. Revisiting pseudo-label for single-positive multi- label learning. In Proceedings of the 40th International Conference on Machine Learning. pages 19249-22265 Honolulu, HA, 2023.
[23] Weiwei Liu, Xiaobo Shen, Haobo Wang, and Ivor W. Tsang. The emerging trends of multi-label learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1 1)77555-7974 2022.
[24] Jiaqi Lv, Miao Xu, Lei Feng. Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In Proceedings of the 37th International Conference Machine Learning, pages 6500 6510, Virtual Event, 2020.
[25] Gengyu Lyu, Songhe Feng, Yi Jin, Tao Wang, Congyan Lang, and Yidong Li. Prior knowledge regularized self-representattion model for partial multilabel learning. IEEE Trunsactions on Cybernetics, 53(3):1618-1622 2023,
11

---

[26] Gengyu Lyu, Songhe Feng, and Yidong Li. Partial multi-label learning via probabilistic graph matching mechanism. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 105-113, Virtual Event, 2020.
[27] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete distribution: A continuous relaxation of discrete random variables. Proceedings of the Sth International Conference on Learning Representations, Toulon, France, 2017.
[28] Sai Rajeswar, Pau Rodríguez, Soumye Singhal, David Vázquez, and Aaron C. Courville. Multi- label iterated learning for image classification with label ambiguity. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4773 4783, New Orleans, LA, 2022.
[29] Lijuan Sun, Songhe Feng, Jun Liu, Gengyu Lyu, and Congyan Lang. Global-local label correlation for partial multi-label learning. IEEE Transactions on Multimedia, 24:581-593, 2022.
[30] Lijuan Sun, Songhe Feng, Tao Wang, Congyan Lang, and Yi Jin. Partial multi-label learning by low-rank and sparse decomposition. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, pages 5016-5023 Honolulu, HA, 2019.
[31] Haobo Wang, Weiwei I Liu, Yang Zhao, Chen Zhang, Tianlei Hu, and Gang Chen. Discriminative and correlative partial multi-label learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 3691-3697, Macao, China, 2019.
[32] Jonatas Wehrmann, Ricardo Cerri, and Rodrigo C. Barros. Hierarchical multi-label classification networks. In Proceedings of the 35th International Conference on Machine Learning, pages Stockholm, Sweden, 2018
[33] F. Wilcoxon. Individual Comparisons by Ranking Methods, pages 196-202. Springer, Berlin, Germany, 1992.
[34] Xuan Wu and Min-Ling Zhang. Towards enabling binary decomposition for partial label learning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2868-2874, Stockholm, Sweden, 2018.
[35] Ming-Kun Xie and Sheng- Jun Huang. Partial multi-label learning with noisy label identification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3676-3677 2022.
[36] Ming-Kul Kun Xie and Sheng-Jun Huang. CCMN: A general framework for learning with class- conditional multi-label noise. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):154-1666 2023.
[37] Ming-Kup Kun Xie, Feng Sun, and Sheng-Jun Huang. Partial multi-label learning with meta disambiguation. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 1904-1912, Virtual Event, 2021.
[38] Ming-Kul Kun Xie, Jiahao Xiao, and Sheng-Jun Huang. Label-aware global consistency for multi- label learning with single positive labels. In Advances in Neural Information Processing Systems 35, pages 18430-1844 New Orleans, LA, 2022.
[39] Ming-Kung Xie and Sheng-Jun Huang. Partial multi-label learning. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pages 4302-4309, New Orleans, LA, 2018
[40] Miao Xu and Lan-Zhe Guo. Learning from group supervision: The impact of supervision deficiency on multi-label learning. Science China Information Sciences, 64(3), 2021.
[41] Ning Xu, Yun-Peng Liu, and Xin Geng, Partial multi-label learning with label distribution. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, pages 6510-6517, New York, NY, 2020.
[42] Ning Xu, Yun-Peng Liu, Yan Zhang, and Xin Geng. Progressive enhancement of label distri- butions for partial multilabel learning. IEEE Transactions on Neural Networks and Learning Systems, 34(8):4856-448) 2023.
12

---

[43] Ning Xu, Congyu Qiao. Xin Geng, and Min-Ling Zhang. instance-dependee partial label learning. In Advances in Neural Information Processing Systems 34, pages 27119-27130, virtual, 2021.
[44] Ning Xu, Jun Shu, Yun-Peng Liu, and Xin Geng. Variational label enhancement. In Proceedings of the 37th International Conference on Machine Learning, pages 10597-10606, Virtual Event, 2020.
[45] Yan Yan and Yuhong Guo. Partial label learning with batch label correction. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, pages 6575-6582, New York, NY, 2020.
[46] Yan Yan and Yuhong Guo. Adversarial partial multi-label learning with label disambiguation. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, pages 10568-10576. Virtual Event, 2021.
[47] Guoxian Yu, Xia Chen, Carlotta Domeniconi, Jun Wang, Zhao Li, Zili Zhang, and Xindong Wu. Feature-induced partial multi-label learning- In Proceedings of the IEEE International Conference Data Mining, pages 1398-1403, Singapore, 2018.
[48] Min-Ling Zhang and Jun-Peng Peng Fang. Partial multi- label learning via credible label elicitation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10):3587-359 2020.
[49] Min-Ling Zhang, Fei Yu, and Cai-Zhi Tang. Disambiguattinn-free partial label learning. IEEE Transactions Knowledge and Data Engineering, 29(10):115-2266) 2017.
[50] Min-Ling Zhang and Zhi-Hua Zhou A review on multi-labs learning algorithms. IEEE Transactions on Knowledge and Data Engineering, ((18):1819-1888 2014.
[51] Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1):44-53, 2018.
[52] Zhi-Hua Zhou. Open-environment machine learning. National Science Review, 98)): 123, 2022.
[53] Yue Zhu, James T. Kwok, and Zui-Hua Zhou. Multi-label learning with global and local label correlation. IEEE Transactions on Knowledge and Data Engineering, 30(6): 1081-194, 2018.
13


Reviews:
[{"review_id": "bAtZs2F3Dz", "reviewer": "Reviewer_uhwx", "summary": "This paper attempts to tackle the Partial Multi-Label Learning task by employing the probabilistic graph method. The proposed method employs the directed graph to model the generative process of partial multi-label data, then derives a variational lower bound objective with a gumbel-softmax trick and closed form of KL-divergence. Empirical results on serval benchmarks demonstrate the effectiveness of the proposed method.", "questions": "-\tPlease see the weakness for details.", "limitations": "There seem to be no limitations or potential negative social impact of this paper.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-\tThis paper is well-written and quite easy to follow.\n-\tThe idea of tackling partial multi-label learning with probabilistic graphical model seems to be interesting.\n-\tEmpirical results demonstrate the effectiveness of the proposed method.", "weaknesses": "-\tThe motivation of this paper is not clear. In this paper, the authors attempt to employ the probabilistic graphical model to perform a principled way for disambiguation and learn the classifier. However, why can the probabilistic graphical model achieve principled disambiguation? The simple entropy-regularized autoencode process seems to be not enough.\n-\tSome baseline methods in recent years are missing in the comparison experiment. You can consider some baselines published in 2022 and 2023, such as Global [1] and PAKS [2]. And these two papers are mentioned in the references, but the relevant citations are missing in the text.\n-\tWhat is the baseline model in the ablation study? Do you mean that it learns the prediction model with cross-entropy loss by directly treating all the candidate labels as valid ones?\n-\tIn formula 6, there is an extra '] '.\n\n[1] Lijuan Sun, Songhe Feng, Jun Liu, Gengyu Lyu, and Congyan Lang. Global-local label correlation for partial multi-label learning. IEEE Transactions on Multimedia, 24:581–593, 2022.\n\n[2] Gengyu Lyu, Songhe Feng, and Yidong Li. Partial multi-label learning via probabilistic graph matching mechanism. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 105–113, Virtual Event, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "------\nThank you for your reviews and insightful comments. The following are our responses to the **Questions**:\n\n------\n1. *The motivation of this paper is not clear. In this paper, the authors attempt to employ the probabilistic graphical model to perform a principled way for disambiguation and learn the classifier. However, why can the probabilistic graphical model achieve principled disambiguation? The simple entropy-regularized autoencode process seems to be not enough.*\n\n* **Response 1**: Thanks to the comments. The probabilistic graphical model is known to be an effective tool to infer latent variables based on observed variables. For example, in representation disentanglement literature [1-3], probabilistic graphical model has been widely investigated to recover latent factors of variation in the data which inherently control what the observed data looks like following the data generative process. Inspired by it, we regard the disambiguation as a task of latent variable inference in our work, thus leading to a first attempt to introduce probabilistic graphical model for partial multi-label learning (PML). Under a maximum likelihood framework, we aim to infer underlying ground-truth labeling information by fitting the generative model of PML data. Empirical results on multiple synthetic and real-world data sets show that the probabilistic graphical model is a promising tool for tackling PML problem. We will make it clearer in the revised version.\n\n  [1] Higgins, I., et al. Beta-VAE: Learning basic visual concepts with a constrained variational framework. In Proceedings of the 5th International Conference on Learning Representations.\n\n  [2] Kim, H. and Mnih, A. Disentangling by factorising. In Proceedings of the 35th International Conference on Machine Learning.\n\n  [3] Van Steenkiste, S., et al. Are disentangled representations helpful for abstract visual reasoning? In Advances in Neural Information Processing Systems 32.\n\n  ------\n2. *Some baseline methods in recent years are missing in the comparison experiment. You can consider some baselines published in 2022 and 2023, such as Global and PAKS. And these two papers are mentioned in the references, but the relevant citations are missing in the text.*\n\n* **Response 2**: Thanks to the suggestions. We have included more recent approaches for comparative studies. Global and PAKS are two important works and have already been cited in the Related Work section (pp. 3, citation indices [16] and [13] respectively). We will add further discussions on these two works in the revised version. However, their codes are not publicly available and we fail to reproduce their results due to limited time. Instead, we further compare our approach with two SOTA algorithms uPMLHL and uPMLRL which are published in 2023 [4]. Representative results (in terms of Average precision$\\uparrow$) are shown in the following table and detailed results will be included in the revised version. Our approach still shows superior performance.\n\n  |      Data sets      |       PML-NI        |     PML-MD      |       uPMLHL        |     uPMLRL      |        PARD         |\n  | :----------------- | :-----------------: | :-------------: | :-----------------: | :-------------: | :-----------------: |\n  |       YeastBP       | **0.430$\\pm$0.018** | 0.310$\\pm$0.016 |   0.392$\\pm$0.020   | 0.170$\\pm$0.009 |   0.417$\\pm$0.015   |\n  |       YeastCC       |   0.610$\\pm$0.022   | 0.527$\\pm$0.027 |   0.596$\\pm$0.022   | 0.459$\\pm$0.021 | **0.622$\\pm$0.020** |\n  |       YeastMF       |   0.471$\\pm$0.022   | 0.422$\\pm$0.027 |   0.451$\\pm$0.016   | 0.329$\\pm$0.023 | **0.491$\\pm$0.028** |\n  |    Music_emotion    |   0.600$\\pm$0.012   | 0.646$\\pm$0.011 | **0.656$\\pm$0.012** | 0.639$\\pm$0.012 |   0.650$\\pm$0.007   |\n  |     Music_style     |   0.733$\\pm$0.012   | 0.716$\\pm$0.012 |   0.734$\\pm$0.010   | 0.703$\\pm$0.012 | **0.742$\\pm$0.009** |\n  |   corel5k ($\\gamma$%=250)   |   0.426$\\pm$0.011   | 0.470$\\pm$0.019 |   0.502$\\pm$0.013   | 0.507$\\pm$0.014 | **0.513$\\pm$0.012** |\n  |   rcv1-s1 ($\\gamma$%=250)   |   0.630$\\pm$0.011   | 0.640$\\pm$0.011 |   0.693$\\pm$0.015   | 0.672$\\pm$0.009 | **0.706$\\pm$0.009** |\n  | Corel16k-s1 ($\\gamma$%=250) |   0.476$\\pm$0.013   | 0.501$\\pm$0.011 |   0.507$\\pm$0.014   | 0.516$\\pm$0.008 | **0.531$\\pm$0.014** |\n  |  iaprtc12 ($\\gamma$%=250)   |   0.543$\\pm$0.009   | 0.560$\\pm$0.009 |   0.582$\\pm$0.008   | 0.547$\\pm$0.006 | **0.601$\\pm$0.009** |\n  |   espgame ($\\gamma$%=250)   |   0.431$\\pm$0.007   | 0.467$\\pm$0.007 |   0.490$\\pm$0.008   | 0.460$\\pm$0.006 | **0.501$\\pm$0.009** |\n\n\n  [4] Xie, M.-K. and Huang, S.-J. CCMN: A general framework for learning with class-conditional multi-label noise. IEEE Transactions on Pattern Analysis and Machine Intelligence.\n\n  ------\n3. *What is the baseline model in the ablation study? Do you mean that it learns the prediction model with cross-entropy loss by directly treating all the candidate labels as valid ones?*\n\n* **Response 3**: Thanks to the comments. The baseline model learns the prediction model with cross-entropy loss by directly treating all the candidate labels as valid ones. The only difference between the baseline model and our approach is that our approach performs additional probabilistic graphical disambiguation. Therefore, this baseline model provides a solid basis to evaluate the effectiveness of the probabilistic graphical disambiguation method. We will make it clearer in the revised version.\n\n  ------\n4. *In formula 6, there is an extra '] '.*\n\n* **Response 4**: Thanks for the reviewer's patience in reading and checking our manuscript. We will carefully proofread the manuscript and correct typos in the revised version.\n\n------"}, {"review_id": "gjs7orEjcX", "reviewer": "Reviewer_mCzx", "summary": "This paper presents a novel, systematic approach to partial multi-label learning (PML) and label disambiguation using a probabilistic graphical model and the stochastic gradient variational Bayes framework. The authors offer a reliable method tested extensively on synthetic and real-world datasets. The results underscore its performance and demonstrate its potential for future PML advancements.", "questions": "- Could the authors provide further clarification regarding the omission of the marginal distribution on x?\n\n- Why is there no discussion and comparison with the Yan and Guo paper [1], which also explores similar dependencies in label disambiguation?\n\n- While the paper claims to outperform state-of-the-art approaches, it does not specify which ones are considered state-of-the-art in the comparisons.\n\n\n[1] Yan, Yan, and Yuhong Guo. \"Adversarial partial multi-label learning with label disambiguation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.", "limitations": "limitations were not explicitly addressed", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents an interesting and effective technique for PML using graphical models. The approach is explained clearly, although incorporating an illustration of the graphical model would enhance the paper. The extensive experiments validate the method's effectiveness. The overall structure and writing style are lucid and easy to comprehend.", "weaknesses": "- The authors make frequent assertions about existing approaches being heuristic and ad-hoc without providing any explanation or justification. Furthermore, there are similar approaches to the one proposed that are not adequately discussed or compared in the paper [1].\n\n- The related work section could be improved by including brief descriptions of some recent related methods.\n\n- The experimental section lacks a comparison with more recent and state-of-the-art papers such as MILI-PML [2] and PML-GAN [1].\n\n[1] Yan, Yan, and Yuhong Guo. \"Adversarial partial multi-label learning with label disambiguation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.\n\n[2] Gong, Xiuwen, Dong Yuan, and Wei Bao. \"Understanding partial multi-label learning via mutual information.\" Advances in Neural Information Processing Systems 34 (2021): 4147-4156.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "------\nThank you for your reviews and insightful comments. The following are our responses to the **Questions** and **Weaknesses**:\n\n------\n1. *Could the authors provide further clarification regarding the omission of the marginal distribution on $\\mathbf{x}$?*\n\n* **Response 1**: Thanks to the comments. The whole log-likelihood on observed PML data $(\\mathbf{x},\\mathbf{s})$ can be factorized as follows\n  $$\n  \\log p_\\theta(\\mathbf{x},\\mathbf{s})=\\log p_{\\theta_1}(\\mathbf{x})+\\log p_{\\theta_2}(\\mathbf{s}|\\mathbf{x}),\n  $$\n  where $\\theta,\\theta_1,\\theta_2$ parametrize distributions $p_\\theta(\\mathbf{x},\\mathbf{s}),p_{\\theta_1}(\\mathbf{x}),p_{\\theta_2}(\\mathbf{s}|\\mathbf{x})$ respectively, and $\\theta=\\\\{\\theta_1,\\theta_2\\\\}$.\n\n  Therefore, maximizing the whole log-likelihood is actually equivalent to maximizing the marginal distribution on $\\mathbf{x}$ and the conditional log-likelihood separately\n  $$\n  \\max_\\theta \\log p_\\theta(\\mathbf{x},\\mathbf{s})\\equiv \\max_{\\theta_1,\\theta_2} \\log p_{\\theta_1}(\\mathbf{x})+\\log p_{\\theta_2}(\\mathbf{s}|\\mathbf{x}).\n  $$\n  Since the focus of our work is to induce the prediction model $p(\\mathbf{y}|\\mathbf{x})$ which only involves in the conditional likelihood $p_{\\theta_2}(\\mathbf{s}|\\mathbf{x})$ according to Eq. (1), we omit the optimization for the marginal distribution on $\\mathbf{x}$.\n\n  ------\n2. *Why is there no discussion and comparison with the Yan and Guo paper, which also explores similar dependencies in label disambiguation?*\n\n* **Response 2**: Thanks to the comments. In PML-GAN, the disambiguation process is regularized by instance reconstruction, which corresponds to a minimax adversarial game to implicitly model instance distribution $p(\\mathbf{x})$. While in our approach, the disambiguation process is learned with a unified variational lower bound of the log-likelihood on observed PML data $(\\mathbf{x},\\mathbf{s})$, which provides solid theoretical support for its effectiveness. Though both PML-GAN and PARD perform disambiguation in an autoencode process, the underlying methodologies and implementations are different. We will include these discussions in the revised version. However, codes of PML-GAN are not publicly available and we fail to reproduce their results due to limited time. Instead, we further compare our approach with two SOTA algorithms uPMLHL and uPMLRL which are published in 2023 [1]. Representative results (in terms of Average precision$\\uparrow$) are shown in the following table and detailed results will be included in the revised version. Our approach still shows superior performance.\n\n  |          Data sets          |       uPMLHL        |     uPMLRL      |        PARD         |\n  | :------------------------- | :-----------------: | :-------------: | :-----------------: |\n  |           YeastBP           |   0.392$\\pm$0.020   | 0.170$\\pm$0.009 |   **0.417$\\pm$0.015**   |\n  |           YeastCC           |   0.596$\\pm$0.022   | 0.459$\\pm$0.021 | **0.622$\\pm$0.020** |\n  |           YeastMF           |   0.451$\\pm$0.016   | 0.329$\\pm$0.023 | **0.491$\\pm$0.028** |\n  |        Music_emotion        | **0.656$\\pm$0.012** | 0.639$\\pm$0.012 |   0.650$\\pm$0.007   |\n  |         Music_style         |   0.734$\\pm$0.010   | 0.703$\\pm$0.012 | **0.742$\\pm$0.009** |\n  |   corel5k ($\\gamma$%=250)   |   0.502$\\pm$0.013   | 0.507$\\pm$0.014 | **0.513$\\pm$0.012** |\n  |   rcv1-s1 ($\\gamma$%=250)   |   0.693$\\pm$0.015   | 0.672$\\pm$0.009 | **0.706$\\pm$0.009** |\n  | Corel16k-s1 ($\\gamma$%=250) |   0.507$\\pm$0.014   | 0.516$\\pm$0.008 | **0.531$\\pm$0.014** |\n  |  iaprtc12 ($\\gamma$%=250)   |   0.582$\\pm$0.008   | 0.547$\\pm$0.006 | **0.601$\\pm$0.009** |\n  |   espgame ($\\gamma$%=250)   |   0.490$\\pm$0.008   | 0.460$\\pm$0.006 | **0.501$\\pm$0.009** |\n  \n  [1] Xie, M.-K. and Huang, S.-J. CCMN: A general framework for learning with class-conditional multi-label noise. TPAMI.\n\n  ------\n3. *While the paper claims to outperform state-of-the-art approaches, it does not specify which ones are considered state-of-the-art in the comparisons.*\n\n* **Response 3**: Thanks to the comments. In comparative studies, we have compared our approach against 6 existing PML algorithms, four of which are published after 2020. Furthermore, we include two more recent and SOTA algorithms uPMLHL and uPMLRL published in 2023. Comparative results can be found in the table of **Response 2**. Our approach still shows superior performance.\n\n  ------\n4. *The authors make frequent assertions about existing approaches being heuristic and ad-hoc without providing any explanation or justification.*\n\n* **Response 4**: Thanks to the comments. Existing approaches mainly rely on some heuristics or ad-hoc rules to disambiguate candidate labels. Generally speaking, these approaches can be roughly grouped into three categories in terms of the heuristics or rules exploited, including smoothness assumption-based approaches [2, 3], sparsity regularization-based approaches [4, 5], and low-rank constraint-based approaches [6, 7]. These heuristics or rules can be too strong to be satisfied in real-world scenarios, which may lead to suboptimal performance of induced prediction model. For example, some approaches rely on sparsity property for disambiguation, although the underlying noisy labels in the set of candidate labels may not be sparse in some extremely low-quality annotation scenarios. We will make it clearer in the revised version.\n\n  [2] Wang, H., et al. Discriminative and correlative partial multi-label learning. IJCAI.\n\n  [3] Zhang, M.-L. and Fang, J.-P. Partial multi-label learning via credible label elicitation. TPAMI.\n\n  [4] Xie, M.-K. and Huang, S.-J. Partial multi-label learning with noisy label identification. TPAMI.\n\n  [5] Sun, L., et al. Global-local label correlation for partial multi-label learning. TMM.\n\n  [6] Yu, G., et al. Feature-induced partial multi-label learning. ICDM.\n\n  [7] Sun, L., et al. Partial multi-label learning by low-rank and sparse decomposition. AAAI.\n\n------"}, {"review_id": "Cv0U9SvC88", "reviewer": "Reviewer_fZq4", "summary": "The paper introduces PARD , a new approach for addressing the problem of partial multi\u0002label learning. PARD uses a probabilistic graphical model to recover ground-truth labeling information. By leveraging the framework of stochastic gradient variational Bayes, PARD simultaneously identifies labeling information and induces a prediction model. Experimental  results on synthetic and real-world datasets demonstrate that PARD outperforms existing methods, providing a more effective and universal solution for partial multi-label learning.", "questions": "1.The derivation from Eq1 to Eq2 can be more detailed explanation. ", "limitations": "The article deals with the situation of partial multi label.  It seems that for general weakly supervised multi-label \nlearning like label noise or single positive label, the proposed method is not generalizable. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The article presents a novel approach for addressing the problem of partial multi-label learning. It introduces a probabilistic graphical model and leverages stochastic gradient variational Bayes to simultaneously identify labeling information and induce a prediction model.\n2.The article is well-structured and effectively communicates the proposed approach, experimental setup, and results. The authors provide sufficient details, making it easier to understand and replicate the methodology.\n3.The article provides extensive experimental results on both synthetic and real-world datasets. It demonstrates that PARD outperforms existing methods, indicating its effectiveness in improving the accuracy of partial multi-label learning tasks.", "weaknesses": "Many papers on weakly supervised multi-label have been proposed recently, the author are  encouraged to provide a detailed discussion and if possible, comparing with them.\n\n[1] Y. Kim, J. M. Kim, Z. Akata, and J. Lee, “Large loss matters in weakly supervised multi-label classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14 156–14 165.\n[2] S. Rajeswar, P. Rodriguez, S. Singhal, D. Vazquez, and A. Courville, “Multi-label iterated learning for image classification with label ambiguity,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4783–4793.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "------\nThank you for your reviews and insightful comments. The following are our responses to the **Questions** and **Weaknesses**:\n\n------\n1. *The derivation from Eq1 to Eq2 can be more detailedly explained.*\n\n* **Response 1**: Thanks to the comments. Detailed derivation process has been provided in Appendix B. We will make it clearer in the revised version.\n\n  ------\n2. *Many papers on weakly supervised multi-label have been proposed recently, the authors are encouraged to provide a detailed discussion and if possible, comparing with them.*\n\n* **Response 2**: Thanks to the suggestions. We will include the discussion on recent progress in more settings of weakly supervised multi-label learning (e.g. learning with single positive label, learning with partial labels, and learning with general noisy labels) in the revised version. And we will list it as an important future work to investigate whether the probabilistic graphical model can be adapted to tackle these interesting weakly supervised learning problems.\n\n------"}, {"review_id": "lsUT7OUgCy", "reviewer": "Reviewer_65gX", "summary": "In the manuscript, the authors adopt the probabilistic graphical model to solve the disambiguation problem in partial multi-label learning. Specifically, a directed graph is tailored to describe the generative process of partial multi-label data, and a partial multi-label learning approach named PARD is proposed. PARD attempts to disambiguate the candidate label set and induce the prediction model simultaneously by maximizing the data log-likelihood with a unified surrogate objective derived by variational inference. The authors conduct extensive experiments to validate the effectiveness of the proposed method.", "questions": "The motivation is not clear. Why do authors apply probabilistic graphical model for disambiguation in PML?\n\nThe authors claim that existing approaches are not universal enough in complicated real-world scenarios; however, in real world applications, the data distribution is hard to follow the assumptions claimed when directly tailoring the graph model, which leads to the unreasonability of the proposed approach. \n\nThe technical part is not solid, e.g., it is not clear to write the variational posterior q in the subscript of expectation through all the equations.", "limitations": "N\\A", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea to use probabilistic graphical model for PML problem is novel. The manuscript is easy to follow.", "weaknesses": "The motivation is not clear. Why do authors apply probabilistic graphical model for disambiguation in PML?\n\nThe authors claim that existing approaches are not universal enough in complicated real-world scenarios; however, in real world applications, the data distribution is hard to follow the assumptions claimed when directly tailoring the graph model, which leads to the unreasonability of the proposed approach. \n\nThe technical part is not solid, e.g., it is not clear to write the variational posterior q in the subscript of expectation through all the equations.\n\n\n-------\nThanks the author for the reply. However, I still have the concerns about this paper.\n\n1, The motivation is quite weak.  Why do we need directed graphical model for PML? What are the limitations of existing methods? Why the proposed directed graphical model achieve better performance than other methods? The author did not show the necessity of introducing the directed graphical model for PML.  Moreover, The authors make frequent assertions in the abstract and introduction about existing approaches being heuristic and ad-hoc as the SOTA limitation that motivates the paper. However, the fact is that existing work exactly solves the heuristic or ad-hoc problem.\n\n2,The key issue of this paper is the lack of the theoretical and technical contribution. The paper directly applies the very known technique graphical model for PML. I can not find any new theoretical and technical contributions in this paper.\n\nTherefore, I vote for the rejection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "------\nThank you for your reviews and insightful comments. The following are our responses to the **Questions**:\n\n------\n1. *The motivation is not clear. Why do authors apply probabilistic graphical model for disambiguation in PML?*\n\n* **Response 1**: Thanks to the comments. The probabilistic graphical model is known to be an effective tool to infer latent variables based on observed variables. For example, in representation disentanglement literature [1-3], probabilistic graphical model has been widely investigated to recover latent factors of variation in the data which inherently control what the observed data looks like following the data generative process. Inspired by it, we regard the disambiguation as a task of latent variable inference in our work, thus leading to a first attempt to introduce probabilistic graphical model for PML. Under a maximum likelihood framework, we aim to infer underlying ground-truth labeling information by fitting the generative model of PML data. Empirical results on multiple synthetic and real-world data sets show that the probabilistic graphical model is a promising tool for tackling PML problem. We will make it clearer in the revised version.\n\n  [1] Higgins, I., et al. Beta-VAE: Learning basic visual concepts with a constrained variational framework. In Proceedings of the 5th International Conference on Learning Representations.\n\n  [2] Kim, H. and Mnih, A. Disentangling by factorising. In Proceedings of the 35th International Conference on Machine Learning.\n\n  [3] Van Steenkiste, S., et al. Are disentangled representations helpful for abstract visual reasoning? In Advances in Neural Information Processing Systems 32.\n\n  ------\n2. *The authors claim that existing approaches are not universal enough in complicated real-world scenarios; however, in real world applications, the data distribution is hard to follow the assumptions claimed when directly tailoring the graph model, which leads to the unreasonability of the proposed approach.*\n\n* **Response 2**: Thanks to the comments. Existing approaches mainly rely on some heuristics or ad-hoc rules to disambiguate candidate labels. These heuristics or rules can be too strong to be satisfied in real-world scenarios. For example, some approaches rely on sparsity property for disambiguation, although the underlying noisy labels in the set of candidate labels may not be sparse in some extremely low-quality annotation scenarios. Instead, our approach only makes mild assumption on the generative process of PML data (the first paragraph in section 3.2, pp.3) and keeps enough capacity in the hypothesis space of the graphical model to approximate the real data distribution.\n\n  ------\n3. *The technical part is not solid, e.g., it is not clear to write the variational posterior q in the subscript of expectation through all the equations.*\n\n* **Response 3**: Thanks to the comments. Actually, we follow the writing style in [4] to write the variational posterior in the subscript of expectation. For example, $\\mathbb{E}\\_{q\\_\\phi(\\mathbf{y}|\\mathbf{x},\\mathbf{s})} [\\log p\\_\\theta(\\mathbf{s}|\\mathbf{x},\\mathbf{y})]$ denotes $\\int q\\_\\phi(\\mathbf{y}|\\mathbf{x},\\mathbf{s}) \\cdot \\log p\\_\\theta(\\mathbf{s}|\\mathbf{x},\\mathbf{y}) d\\mathbf{y}$​. We will make it clearer in the revised version.\n\n  [4] Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations.\n\n------"}, {"review_id": "rESIpwVtyo", "reviewer": "Reviewer_GJDR", "summary": "The paper focuses on a specific problem called partial multi-label learning (PML), which is an emerging weakly-supervised learning problem. PML involves inaccurate annotations, where each training example is associated with a set of candidate labels, among which only some labels are valid. The paper proposes a probabilistic graphical model for PML and develops a principled approach for disambiguating candidate labels. The approach is evaluated through comprehensive experiments on multiple synthetic and real-world datasets, showing superior performance compared to existing methods.", "questions": "1. Any idea about how to explore alternative implementations for principled disambiguation?\n2. The proposed approach PARD outperforms the state-of-the-art counterparts on multiple synthetic and real-world datasets, demonstrating its effectiveness in improving the accuracy of partial multi-label learning tasks. But I'd like to see how this approach behaves in a large model environment, or how it might be combined\n3. The results of the comprehensive experiments on multiple synthetic and real-world datasets show that PARD outperforms the state-of-the-art counterparts, which indicates that this approach can be used to improve the performance of many practical applications that involve partial multi-label learning. But why does this model achieve the best experimental results on all tasks? What about other tasks?", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a novel approach called PARD for partial multi-label learning, which aims to recover the groundtruth labeling information from inaccurate annotations.\n2. PARD uses a directed graphical model to describe the generative process of partial multi-label data and a unified variational lower bound derived by variational inference to maximize the data log-likelihood on given partial multi-label data set.\n3. PARD achieves to disambiguate the candidate label set and induce the prediction model simultaneously, and comprehensive experiments show the superiority of this approach.", "weaknesses": "1. PARD provides a principled way for disambiguation, which is the process of recovering the groundtruth labeling information from inaccurate annotations. This can be useful in many real-world scenarios where the annotations are not perfect.However, the number of labels required may not meet the requirements of the actual task, and further work can be done in this regard.\n2. The use of a directed graphical model and variational inference in PARD can be extended to other machine learning tasks beyond partial multi-label learning, where probabilistic modeling and inference are required. But the construction of modeling and inference rules is very dependent on the nature of the specific task, is there a good transfom method to take advantage of the previous work?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "------\nThank you for your reviews and insightful comments. The following are our responses to the **Questions** and **Weaknesses**:\n\n------\n1. *Any idea about how to explore alternative implementations for principled disambiguation?*\n\n* **Response 1**: Thanks to the comments. We believe that new ideas on alternative implementations for principled disambiguation can be inspired from related problems of partial multi-label learning (PML). For example, some methods in noisy label learning, e.g. those based on sample selection or transition matrix, could be adapted for tackling PML problem.\n\n  ------\n2. *The proposed approach PARD outperforms the state-of-the-art counterparts on multiple synthetic and real-world datasets, demonstrating its effectiveness in improving the accuracy of partial multi-label learning tasks. But I'd like to see how this approach behaves in a large model environment, or how it might be combined.*\n\n* **Response 2**: Thanks to the comments. Actually, PARD can be implemented flexibly with different network structures. To combine with large models, a feasible way is to use the large model as a shared feature extractor and attach three heads (i.e. inference model, generative model and prediction model) on top of the large model. Then, the whole model can be trained with the unified variational lower bound in Eq. (3). However, training a good large model is not an easy work due to limited time and may involve many tricks. We believe it is interesting to investigate the behavior of PARD in the large model environment and will explore it in the future.\n\n  ------\n3. *The results of the comprehensive experiments on multiple synthetic and real-world datasets show that PARD outperforms the state-of-the-art counterparts, which indicates that this approach can be used to improve the performance of many practical applications that involve partial multi-label learning. But why does this model achieve the best experimental results on all tasks? What about other tasks?*\n\n* **Response 3**: Thanks to the comments. Existing approaches mainly rely on heuristics or ad-hoc rules to disambiguate candidate labels. These heuristics or rules can be too strong to be satisfied in real-world scenarios. For example, some approaches rely on sparsity property for disambiguation, although the underlying noisy labels in the set of candidate labels may not be sparse in some extremely low-quality annotation scenarios. While our approach has no such requirement, which directly disambiguates and induces prediction model by maximizing the log-likelihood on observed data. Therefore, it can handle diversified PML tasks more flexibly and outperform existing approaches in multiple benchmark data sets.\n\n  ------\n4. *The use of a directed graphical model and variational inference in PARD can be extended to other machine learning tasks beyond partial multi-label learning, where probabilistic modeling and inference are required. But the construction of modeling and inference rules is very dependent on the nature of the specific task, is there a good transform method to take advantage of the previous work?*\n\n* **Response 4**: Thanks to the comments. We agree with the reviewer that the directed graphical model and variational inference are useful tools to tackle various machine learning tasks. And the graphical model and inference rules do need some flexible customization according to the properties of the specific task. For example, in PARD, we tailor the graphical model to describe the generative process of partial multi-label data and exploit additional techniques (e.g. Gumbel-Softmax relaxing) to make inference efficient. In our opinion, it is exactly the point making these tools fascinating to machine learning community, since it allows to introduce inductive bias or expert knowledge from humans.\n\n------"}]

Author rebuttal:

Meta-review: This paper proposes an approach for partial multi-label learning, in which each example can have multiple binary ground truth labels and the training annotations are a superset of the true labels. The paper's approach is to construct a Bayesian network that describes the generative process of the partial labels. The true labels are modeled as latent variables. The paper presents several training techniques to more efficiently optimize a variational lower bound on the data log-likelihood. Experiments show that this approach compares favorably with previously published methods on synthetic and real-world datasets.

The majority of reviewers favored acceptance. They identified as strengths the technical contribution of a novel, principled approach, convincing experiments, and quality of presentation. One reviewer raised a strong concern that the paper had substantial overlap in motivation with "Understanding partial multi-label learning via mutual information," NeurIPS 2021. The other reviewers and the area chair reviewed this reference for similarities. While the other paper applies to the same problem, it takes an approach based on maximizing mutual information between features and identified ground truth labels. This paper instead takes a latent variable approach. While this paper would be strengthened by an empirical comparison with that prior work (in addition to all the other comparisons already done), other reviewers and the area chair did not think that the overlap was that significant.
Decision: Accept (poster)