Title: Partial Multi-Label Learning with Probabilistic Graphical Disambiguation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Machine learning

multi-label learning

partial multi-label learning

label disambiguation
Abstract: In partial multi-label learning (PML), each training example is associated with a set of candidate labels, among which only some labels are valid. As a common strategy to tackle PML problem, disambiguation aims to recover the ground-truth labeling information from such inaccurate annotations. However, existing approaches mainly rely on heuristics or ad-hoc rules to disambiguate candidate labels, which may not be universal enough in complicated real-world scenarios. To provide a principled way for disambiguation, we make a first attempt to explore the probabilistic graphical model for PML problem, where a directed graph is tailored to infer latent ground-truth labeling information from the generative process of partial multi-label data. Under the framework of stochastic gradient variational Bayes, a unified variational lower bound is derived for this graphical model, which is further relaxed probabilistically so that the desired prediction model can be induced with simultaneously identified ground-truth labeling information. Comprehensive experiments on multiple synthetic and real-world data sets show that our approach outperforms the state-of-the-art counterparts.

Partial Multi-Label Learning with Probabilistic Graphical Disambiguation

Abstract

In partial multi-label learning (PML), each training example is associated with a set of candidate labels, among which only some labels are valid. As a common strategy to tackle PML problem, disambiguation aims to recover the ground-truth labeling information from such inaccurate annotations. However, existing approaches mainly rely on heuristics or ad-hoc rules to disambiguate candidate labels, which may not be universal enough in complicated real-world scenarios. To provide a principled way for disambiguation, we make a first attempt to explore the probabilistic graphical model for PML problem, where a directed graph is tailored to infer latent ground-truth labeling information from the generative process of partial multi-label data. Under the framework of stochastic gradient variational Bayes, a unified variational lower bound is derived for this graphical model, which is further relaxed probabilistically so that the desired prediction model can be induced with simultaneously identified ground-truth labeling information. Comprehensive experiments on multiple synthetic and real-world data sets show that our approach outperforms the state-of-the-art counterparts.

1 Introduction

Partial multi-label learning aims to induce a multi-label predictor from inaccurately annotated examples, where a set of candidate labels is assigned to each training example but only some of these candidate ones are valid. The problem of PML naturally arises in many real-world applications [39, 51, 52, 20]. For instance, as shown in Figure 1, the data collected from the crowdsourcing platform is inevitable to contain inaccurate supervision information due to potential unreliable annotators, where invalid labels exist in the set of candidate labels.

Formally, let X = R d denote the input space and Y = { l 1 , l 2 , . . . , l t } denote the label space with t class labels. A partial multi-label example is denoted as ( x , S ) , where x ∈ X is its feature vector and S ⊆ Y is its set of candidate labels. As a basic assumption of PML, the ground-truth labels reside in the candidate label set, i.e. Y ⊆ S , and are concealed from the learning algorithm. Given a partial multi-label data set D = { ( x i , S i ) | 1 ≤ i ≤ m } , the goal of PML is to derive a multi-label predictor h : X → 2 Y which can accurately predict all the ground-truth labels for an unseen instance.

A straightforward strategy for tackling the problem of PML is to simply treat all the candidate labels as valid ones and then apply off-the-shelf multi-label learning approaches [50, 23] to induce the predictor. This naïve strategy is obviously suboptimal as the learning process will be significantly misled by the noisy labels in the candidate label set.

Candidate labels

(valid ones in red) tree flower light mountain car road people

Considering that the ground-truth labels are concealed in the candidate label set, it is a common strategy of existing approaches to learn from partial multi-label data by disambiguation, i.e. recovering the ground-truth labeling information from candidate labels. The basic idea of disambiguation is to estimate the labeling confidence for each candidate label from observed inaccurately annotated examples, which reflects how likely the candidate label can be a ground-truth one. For example, some approaches introduce the smoothness assumption to directly elicit the labeling confidence via label propagation [48, 26], label enhancement [31, 41], or cluster assignment [39, 3], etc. While some other works heuristically believe that the noisy labels in the candidate label set are generally sparse, so that the noisy label matrix [30, 29] or the noisy label identifier [35] can be learned with sparsity regularization. Although feasible results have been achieved in these approaches, a potential limitation is that they rely on heuristics or ad-hoc rules for disambiguation, which may not be widely applicable in some challenging scenarios. For example, sparsity on underlying noisy labels can hardly be satisfied in some extremely low-quality annotation scenarios.

To improve this, we explore a principled manner for disambiguation, which is regarded as a task of latent variable inference in this paper. Accordingly, a novel partial multi-label learning approach named PARD, i.e. P artial multi-label learning with prob A bilistic-g R aphical D isambiguation, is presented. By regarding the concealed ground-truth labels as latent variable, a directed graphical model is tailored to describe the generative process of partial multi-label data. With the directed graphical model, a unified variational lower bound of the data log-likelihood is derived via variational inference, which allows to disambiguate the candidate labels and induce the prediction model simultaneously. Specifically, the ground-truth labeling information is identified by fitting the generative model of PML data, where the intrinsic structure information of data can be captured for principled disambiguation. While the desired prediction model is optimized with a confidence-smoothed cross entropy loss. Comprehensive experimental studies show that our approach performs better than well-established PML algorithms.

The rest of this paper is organized as follows. Section 2 briefly reviews related works. Section 3 presents details of the proposed PARD approach. Section 4 reports experimental results over a wide range of synthetic and real-world data sets. Section 5 concludes this paper.

2 Related Work

Partial multi-label learning is an emerging weakly-supervised learning problem, which has received wide attention in recent years [39, 40, 36]. Here, we would like to first make a brief review on two closely related learning problems, i.e. multi-label learning [50, 23] and partial label learning [6, 24].

Multi-label learning (MLL) deals with the problem where an example can be associated with multiple labels simultaneously. According to the order of label correlations considered, existing MLL approaches can be grouped into three categories, including first-order approaches which treat each class label independently [2, 14], second-order approaches which explore label correlations between pairwise class labels [8, 53], and high-order approaches which consider label correlations among a subset or the whole set of class labels [32, 44]. With the same goal, both MLL and PML aim to learn a multi-label predictor from multi-semantic objects. Nevertheless, the key difference between these

two learning problems lies in whether the ground-truth labels are accessible or not during the learning process, which also makes PML more challenging than MLL since the supervision information is inaccurate.

Partial label learning (PLL) deals with the problem where each example is associated with multiple candidate labels, among which only one is valid. The problem of PLL can be tackled in several different strategies, such as transforming into well-studied supervised learning problems [49, 34], disambiguating the candidate labels [9, 45, 43], or learning with theoretically consistent algorithms [24, 10], etc. Conceptually speaking, PLL and PML possess similar form of supervision, where false positive labels reside in the candidate label set. However, the task of PML is more complicated than PLL since the desired predictor is a multi-label one in PML instead of a single-label one in PLL and the number of ground-truth labels is further unknown.

To solve the PML problem, the most commonly employed strategy is disambiguation, i.e. recovering the ground-truth labeling information from candidate labels. Existing approaches mainly rely on some heuristics or ad-hoc rules to disambiguate candidate labels, which can be roughly grouped into three categories in terms of the heuristics or rules exploited, including smoothness assumption-based approaches , low-rank constraint-based approaches , and sparsity regularization-based approaches . The first category of approaches rely on smoothness assumption that close instances may share similar labels for disambiguation. As a seminal work, an iterative procedure is proposed in [39] to optimize the labeling confidence of each candidate label and the prediction model alternatively, where the labeling confidence scores are obtained by assuming cluster structures exist in the feature space. Follow-up works employ smoothness assumption to design different disambiguation mechanisms, such as propagating labels in a graph based on instance similarity [48, 26], recovering numerical labels with label enhancement [31, 41, 42], or performing implicit disambiguation by jointly embedding [12]. Low-rank constraint-based approaches turn to the low-rank property for disambiguation. For example, low-rank matrix decomposition is exploited to recover the ground-truth labeling information in [47] and a low-rank subspace is learned in [25] to elicit credible labels from improved subspace representations. While sparsity regularization-based approaches [30, 29, 35] impose sparsity on noises hiding in candidate label set, which facilitate the disambiguation in a roundabout manner. A potential limitation of these approaches is that the above heuristics or ad-hoc rules can be hardly held in some challenging scenarios, which may lead to suboptimal performance of induced prediction model.

Therefore, it is quite natural to consider whether it is possible to perform disambiguation in a principled manner, which is still an underexplored direction with only a few attempts. PML-GAN [46] recovers ground-truth labeling information with an instance reconstruction process, corresponding to a minimax adversarial game to implicitly model instance distribution. While MILI-PML [11] disambiguates candidate labels by maximizing the mutual information between features and identified ground-truth labels. With a totally different methodology, our approach regards disambiguation as a task of latent variable inference, which is formalized in a maximum likelihood framework with a concise variational lower bound of the log-likelihood on observed PML data. As another pioneer work towards this direction, PML-MD [37] devises a meta-learning framework, where disambiguation is performed with the guidance from an accurately annotated validation set. Moving one step forward, we make a first attempt to tackle the problem via modelling the generative process of partial multilabel data with a tailored directed graphical model. Compared with PML-MD, our approach is more flexible since there is no need to include a clean validation set for disambiguation, which may not be easily accessible in many real-world scenarios. We will detail our approach in the next section.

3 The PARD Approach

3.1 Overview

To tackle the problem of partial multi-label learning, PARD disambiguates the candidate labels and induces the prediction model simultaneously by optimizing towards a unified variational lower bound of the data log-likelihood. In the following content, we will present the tailored directed graphical model which describes the generative process of partial multi-label data. Then, we derive a variational lower bound of the data log-likelihood and further explain how to relate it with disambiguation and predictor induction. Finally, essential implementation issues w.r.t. the optimization procedure are detailed.

Some necessary notations have been introduced in section 1. For notation briefness, a t -dimensional indicator vector s ∈ { 0 , 1 } t is utilized to denote the set of candidate labels S , where s k = 1 indicates l k ∈ S and s k = 0 otherwise. Similarly, a t -dimensional indicator vector y ∈ { 0 , 1 } t is utilized to denote the set of ground-truth labels Y .

3.2 Probabilistic Graphical Disambiguation Framework

Considering that only the instance and its associated candidate labels can be observed in the partial multi-label training set, we tailor a directed graphical model to describe the generative process of partial multi-label data, which involves an unobserved latent variable y as the ground-truth labels. The generative process consists of three steps: (1) sample an instance x from the marginal distribution p θ ( x ) ; (2) sample a latent y as the ground-truth labels of the instance x from the ground-truth class posterior distribution p θ ( y | x ) ; (3) corrupt the instance's ground-truth labels to obtain its candidate labels s by p θ ( s | x , y ) . Accordingly, the joint probability p θ ( x , y , s ) can be factorized as

Given the partial multi-label training set D , the above directed graphical model can be learned via maximizing the log-likelihood on the observed data. However, it is generally challenging to directly estimate the parameter of a directed graphical model by likelihood maximization due to computational intractability [17]. To make the optimization tractable, the variational lower bound of the log-likelihood 2 is derived as follows

where q φ ( y | x , s ) is the introduced variational posterior to approximate the true posterior p θ ( y | x , s ) . L ( x , s ; θ, φ ) is the derived variational lower bound as a surrogate loss function of the log-likelihood, which can be rewritten as

where KL [ ·||· ] denotes the KL-divergence between two distributions. Here, the meaning of the ground-truth class posterior distribution p θ ( y | x ) is comprehensible, which is actually the desired prediction model 3 . While the variational posterior q φ ( y | x , s ) (a.k.a. inference model) attempts to disambiguate the candidate label set by inferring the most probable ground-truth labels from which the candidate labels could have been corrupted given the instance x . Correspondingly, p θ ( s | x , y ) (a.k.a. generative model) corrupts the identified ground-truth labels to recover the observed candidate labels.

To move one step further for comprehending the learning behavior of the above three models, we reformulate the variational lower bound via unfolding the KL-divergence term

where H [ · ] denotes the entropy of a distribution. In the above objective, the first two terms present an entropy-regularized autoencode process (w.r.t. s ), with which the intrinsic structure information of data can be captured and utilized for disambiguation. While the last term is actually a cross entropy loss for inducing the prediction model with identified ground-truth labeling information q φ ( y | x , s ) . By optimizing all these terms in a unified variational lower bound, the models can gradually capture the underlying generative process of partial multi-label data, so that the candidate label set is disambiguated and the desired prediction model is induced simultaneously.

3.3 Additional Training Techniques

In this section, we present additional techniques that allow the derived variational lower bound to be optimized efficiently.

Algorithm 1 Pseudocode of the Optimization Procedure for PARD

Inputs: Training set D = { ( x i , S i ) | 1 ≤ i ≤ m } , batch size b and maximal iteration T . Process:

1: Initialize model parameters θ, φ ;

2: for t = 1 : T do

3: Sample a batch of training examples B = { ( x i k , S i k ) | 1 ≤ k ≤ b } from training set D ;

4: Compute unbiased estimator of the variational lower bound on B by Eq. (6) and Eq. (7);

5: Update model parameters θ, φ via gradient ascent.

6: end for

Outputs : Model parameters θ, φ

.

Continuous Relaxing with Gumbel-Softmax Trick. It is quite challenging to compute the first expectation term in Eq. (3). Actually, it has a high complexity of O (2 t ) to analytically compute the first expectation term in Eq. (3), since the ground-truth labels y can take 2 t different values. In addition, estimating it by Monte Carlo sampling is non-differentiable, since sampling from discrete variational posterior q φ ( y | x , s ) prevents gradients from flowing through the inference model.

To circumvent this problem, we exploit Gumbel-Softmax trick [13, 27] to smooth the sampling process so that the expectation term can be efficiently estimated by Monte Carlo sampling in a differentiable manner. Specifically, the Bernoulli variable y is relaxed by the binary Concrete variable c ∼ BinConcrete ( p , τ ) , which is a continuous alternative with the reparameterization form as

where α = p 1 -p and p is the parameter of the multivariate Bernoulli distribution q φ ( y | x , s ) . l is a sampling from Logistic distribution and τ > 0 is a temperature parameter. In the limit τ → 0 , a binary Concrete variable smoothly converges to its Bernoulli counterpart. With the above sampling trick, an unbiased estimation of the expectation term can be obtained and gradients w.r.t. the distribution parameter p (or w.r.t. the parameter of the inference model equivalently) are well-defined by the chain rule

Closed-Form Solution of the KL-Divergence Term. The KL-divergence term in Eq. (3) can not be integrated analytically in general cases. To make it tractable, we exploit mean-field approximation technique and derive a closed-form solution of the KL-divergence term as follows

where p y k φ = q φ ( y k = 1 | x , s ) and p y k θ = p θ ( y k = 1 | x ) . Detailed derivation process can be found in Appendix C.

With these training techniques, the derived variational lower bound can be straightforwardly optimized with standard stochastic gradient methods. Algorithm 1 summarizes the pseudocode of the optimization procedure, where the candidate labels and the prediction model are updated simultaneously.

4 Experiments

4.1 Experimental Setup

4.1.1 Data Sets

For comprehensive performance evaluation, five real-world and a number of synthetic PML data sets are employed in this paper. Table 1 summarizes detailed characteristics of each data set. Specifically,

the first five data sets are real-world PML data sets, where candidate labels are collected from web users and manually examined to specify the ground-truth labels. While the last five data sets, including corel5k , rcv1-s1 , Corel16k-s1 , iaprtc12 and espgame , are multi-label data sets.

Following [35], a synthetic PML data set is generated from one multi-label data set by adding random labeling noise. Accordingly, for data sets with more than 100 class labels, we filter out their rare labels to keep the 15 most frequent labels and remove instances without relevant labels. Then, we randomly flip the irrelevant labels of an instance until the number of irrelevant labels in the set of candidate labels equals to γ % number of ground-truth labels. We vary the value of γ in the range of { 100 , 150 , 200 , 250 } , so that a total of 5 × 4 = 20 synthetic PML data sets are generated for experimental studies.

4.1.2 Evaluation Metrics

Five widely-used evaluation metrics for multi-label learning are employed to evaluate the performance of each approach, including Average precision , Hamming loss , One-error , Coverage and Ranking loss . Detailed definitions on these metrics can be found in [50].

4.1.3 Implementation Details

The inference model q φ ( y | x , s ) and the generative model p θ ( s | x , y ) are instantiated by fullyconnected neural networks with ReLU activations, where the hidden dimensionalities are set to [256; 512; 256] and [256; 512] respectively. For fair comparison with existing PML approaches, the prediction model is implemented as a linear model. To compute the objective function in Eq. (3), a trade-off parameter α is introduced for the KL-divergence term and Monte Carlo sampling with sampling number L = 1 is conducted to estimate the first expectation term, where the temperature parameter τ = 2 / 3 as suggested by [27]. In the following experiments, we set α ≥ 1 so that the objective function is still a valid lower bound of the data log-likelihood. For network optimization, Adam with a batch size of 128, weight decay of 10 -4 , momentums of 0.999 and 0.9 is employed. In this paper, all experiments are conducted on one V100 GPU.

4.2 Comparative Studies

PARD 4 is compared against six well-established PML approaches with parameter configurations suggested in respective literatures:

FPML [47]: FPML employs the low-rank approximation of the instance-label association matrix to estimate the labeling confidence and then trains multi-label predictor. [ λ 1 = 0 . 1 , λ 2 = 1 , λ 3 = 10 ]

Data sets

γ

%

Average precision

↑

FPML

PARVLS

PML-NI

PML-MD

UPML-HL

UPML-RL

PARD

YeastBP

0.202

±

0.013

•

0.059

±

0.003

•

0.430

±

0.018

◦

0.310

±

0.016

•

0.392

±

0.020

•

0.170

±

0.009

•

0.417

±

0.015

YeastCC

0.375

±

0.019

•

0.145

±

0.009

•

0.610

±

0.022

•

0.527

±

0.027

•

0.596

±

0.022

•

0.459

±

0.021

•

0.622

±

0.020

YeastMF

0.279

±

0.020

•

0.114

±

0.007

•

0.471

±

0.022

•

0.422

±

0.027

•

0.451

±

0.016

•

0.329

±

0.023

•

0.491

±

0.028

Music_emotion

0.571

±

0.013

•

0.603

±

0.011

•

0.600

±

0.012

•

0.646

±

0.011

0.656

±

0.012

◦

0.639

±

0.012

•

0.650

±

0.007

Music_style

0.702

±

0.014

•

0.720

±

0.012

•

0.733

±

0.012

•

0.716

±

0.012

•

0.734

±

0.010

•

0.703

±

0.012

•

0.742

±

0.009

corel5k

100

0.485

±

0.014

•

0.391

±

0.017

•

0.473

±

0.016

•

0.491

±

0.019

•

0.516

±

0.015

•

0.514

±

0.015

•

0.533

±

0.013

150

0.478

±

0.013

•

0.408

±

0.011

•

0.453

±

0.013

•

0.479

±

0.015

•

0.514

±

0.015

•

0.512

±

0.015

•

0.523

±

0.013

200

0.474

±

0.013

•

0.395

±

0.021

•

0.438

±

0.012

•

0.472

±

0.011

•

0.503

±

0.009

•

0.511

±

0.013

0.516

±

0.011

250

0.470

±

0.012

•

0.391

±

0.022

•

0.426

±

0.011

•

0.470

±

0.019

•

0.502

±

0.013

•

0.507

±

0.014

0.513

±

0.012

rcv1-s1

100

0.682

±

0.010

•

0.520

±

0.021

•

0.693

±

0.014

•

0.678

±

0.010

•

0.712

±

0.013

•

0.689

±

0.007

•

0.720

±

0.008

150

0.678

±

0.011

•

0.513

±

0.022

•

0.664

±

0.013

•

0.656

±

0.011

•

0.705

±

0.013

•

0.690

±

0.008

•

0.716

±

0.010

200

0.673

±

0.010

•

0.495

±

0.021

•

0.649

±

0.010

•

0.648

±

0.010

•

0.704

±

0.012

•

0.685

±

0.005

•

0.710

±

0.010

250

0.666

±

0.011

•

0.488

±

0.016

•

0.630

±

0.011

•

0.640

±

0.011

•

0.693

±

0.015

•

0.672

±

0.009

•

0.706

±

0.009

100

0.535

±

0.010

•

0.430

±

0.010

•

0.518

±

0.011

•

0.522

±

0.011

•

0.539

±

0.012

•

0.525

±

0.007

•

0.551

±

0.012

150

0.529

±

0.008

•

0.423

±

0.012

•

0.499

±

0.011

•

0.512

±

0.011

•

0.532

±

0.011

•

0.522

±

0.009

•

0.545

±

0.012

200

0.523

±

0.010

•

0.404

±

0.012

•

0.487

±

0.013

•

0.507

±

0.012

•

0.504

±

0.011

•

0.519

±

0.009

•

0.536

±

0.014

250

0.514

±

0.011

•

0.389

±

0.011

•

0.476

±

0.013

•

0.501

±

0.011

•

0.507

±

0.014

•

0.516

±

0.008

•

0.531

±

0.014

iaprtc12

100

0.603

±

0.006

•

0.591

±

0.008

•

0.599

±

0.009

•

0.600

±

0.008

•

0.601

±

0.009

•

0.563

±

0.008

•

0.621

±

0.011

150

0.600

±

0.006

•

0.585

±

0.008

•

0.580

±

0.010

•

0.583

±

0.010

•

0.597

±

0.007

•

0.559

±

0.007

•

0.615

±

0.009

200

0.597

±

0.008

•

0.574

±

0.008

•

0.563

±

0.008

•

0.569

±

0.009

•

0.600

±

0.008

•

0.554

±

0.008

•

0.610

±

0.008

250

0.588

±

0.009

•

0.561

±

0.008

•

0.543

±

0.009

•

0.560

±

0.009

•

0.582

±

0.008

•

0.547

±

0.006

•

0.601

±

0.009

espgame

100

0.498

±

0.007

•

0.472

±

0.007

•

0.478

±

0.008

•

0.491

±

0.009

•

0.501

±

0.008

•

0.466

±

0.009

•

0.515

±

0.008

150

0.496

±

0.007

•

0.456

±

0.007

•

0.459

±

0.007

•

0.481

±

0.007

•

0.509

±

0.008

0.471

±

0.006

•

0.511

±

0.009

200

0.495

±

0.007

•

0.445

±

0.007

•

0.446

±

0.005

•

0.473

±

0.007

•

0.475

±

0.012

•

0.466

±

0.006

•

0.508

±

0.008

250

0.489

±

0.009

•

0.427

±

0.009

•

0.431

±

0.007

•

0.467

±

0.007

•

0.490

±

0.008

•

0.460

±

0.006

•

0.501

±

0.009

PARVLS [48]: A two-step approach which elicits credible labels via label propagation and then induces multi-label predictor by virtual label splitting. [ k = 10 , α = 0 . 95 , and thr = 0 . 9 ]

PML-NI [35]: PML-NI performs disambiguation by learning a noisy label identifier with sparsity regularization. [ β = 0 . 5 , γ = 0 . 5 , and search for λ ∈ { 1 , 10 , 100 } ]

PML-MD [37]: PML-MD disambiguates between ground-truth and noisy labels in a metalearning fashion and learns multi-label predictor with a confidence-weighted ranking loss.

UPML-HL [36]: A disambiguation-free approach which directly induces the prediction model by optimizing an unbiased estimator of the Hamming loss.

UPML-RL [36]: A disambiguation-free approach which directly induces the prediction model by optimizing an unbiased estimator of the Ranking loss.

Following [37], we take out 10% examples in each data set as hold-out validation set, so that PML-MD can use this clean validation set to perform meta-disambiguation. The remaining 90% examples are randomly splitted into training set and test set with a ratio of 9:1 for training and evaluation repectively. During training, PML-MD performs meta-learning on the noisy training set and the clean validation set. While other approaches including ours only employ this validation set to search hyperparameters. In PARD, the trade-off paramter α is searched in { 1 , 2 , 5 , 10 , 20 , 50 , 100 } . For fair comparison, all deep approaches adopt a linear model to instantiate the predictor and share the same optimizer with the learning rate searched in {1e-3,3e-3,1e-2,3e-2}. For each data set, we repeat the random splitting process for ten times and record the average predicitve performance across ten training/test trials.

Due to page limit, Table 2 and Table 3 report detailed experimental results in terms of Average precision and Ranking loss . Results on other metrics can be found in Appendix A. Furthermore, the Wilcoxon signed-ranks test [33] at 0.05 significance level is conducted to analyze whether PARD achieves statistically superior performance to other comparing approaches. Table 4 summarizes the p -value statistics on each evaluation metric. Based on these results, it is impressive to observe that:

Across all evaluation metrics, PARD achieves the best performance in 82.4% cases over all the 25 data sets.

As shown in Table 4, PARD achieves statistically better performance against other approaches which rely on heuristics or ad-hoc rules for disambiguation. The superior performance of PARD is consistent across almost all real-world data sets and synthetic data sets under varied noise levels, which provides a strong evidence for the effectiveness of probabilistic graphical disambiguation to facilitate partial multi-label learning.

Meantime, PARD significantly outperforms PML-MD which also disambiguates PML data in a principled manner. Note that PML-MD requires an accurately annotated validation set to perform disambiguation during the learning process. The superior performance of PARD against PML-MD indicates that PARD is an effective and more flexible approach for principled disambiguation.

4.3 Further Analyses

4.3.1 Ablation Study

To validate that the disambiguation process in PARD is effective for facilitating partial multi-label learning, we implement a baseline which induces prediction model with cross-entropy loss by directly treating all the candidate labels as valid ones. As shown in Figure 2, a clear performance degradation

0.8 -

0.26 -

PARD

Baseline

PARD

Baseline

0.7

0.24 =

sion

0.22

precis

0.6

loss

0.20

nking

Average

0.5

0.18

anl

0.16

0.4

R

0.14

0.3

0.12

0.2

0.10

YeastBP

YeastCC

YeastMF

Music_emotion Music_style

YeastBP

YeastCC

YeastMF

Music_emotion Music_style

0.6

1.6

1-Average precision

1-Average precision

1.4

Hamming loss

One-error

0.5

Hamming loss

One-error

Performance

1.2

Coverage

Performance

Coverage

1.0

Ranking loss

0.4

Ranking loss

0.8

0.3

P

0.6

0.4

0.2

0.2

0.0

0.1

1

2

5

10

20

50

100

1

2

5

10

20

50

100

Trade-offparameterα

Trade-offparameterα

(a)YeastCC

(b) Music_style

is witnessed in the baseline model, which demonstrates the disambiguation process induced from the variational lower bound can facilitate partial multi-label learning.

4.3.2 Parameter Sensitivity

Figure 3 gives some illustrative examples on how the performance of PARD changes when the value of the trade-off parameter α changes. The performance of PARD is relatively stable as the value of α changes within a reasonable range, which is a desirable property when deploying PARD in real-world applications. Similar results can be observed on other data sets.

4.3.3 Running Time Comparision

For PARD, the training phase only involves a pair of forward and backward computations among the inference, generative and prediction models, which is efficient thanks to the training techniques presented in section 3.3. Figure 4 illustrates the empirical training and test time of each comparing approach, which shows that PARD is competitive with exisiting approaches in time overhead.

4.0

FPML

PARVLS

PML-NI

scale

log

S

S'E

3.0

scale

(su)

3.0

PML-MD

2.5

Test time in log s

2.0

日

UPML-HL

time

2.0

S°t

PARD

UPML-RL

Training

1.5

1.0

1.0

0.5

0.5

0.0

YeastBP

YeastCC

YeastMF

Music_emotion

Music_style

0.0

YeastBP

YeastCC

YeastMF

Music_emotion

Music_style

(a) Training time of comparing approaches

(b) Test time of comparing approaches

5 Limitation

Our primary focus is on partial multi-label learning in this paper. However, there are other weaklysupervised multi-label learning problems that receive increasing attention from related community. The probabilistic graphical disambiguation framework of PARD currently has not been tested on these problems, such as learning with partial labels which deals with incompletely annotated data [7, 19, 1, 4], learning with single positive label which aims to induce an accurate multi-label predictor from multi-label data annotated with only one positive labels [5, 28, 15, 38, 16, 22], and learning with general noisy labels where noise may exist in every label of inaccurately annotated data [18, 21, 36]. In addition, we do not implement PARD and investigate its behavior in large model environment, which are also important to practitioners. We will explore these points in the future.

6 Conclusion

In this paper, an attempt towards principled disambiguation for partial multi-label learning is presented, where a directed graphical model is tailored to describe the generative process of partial multi-label data. By maximizing the data log-likelihood on given partial multi-label data set with a unified surrogate objective derived by variational inference, PARD achieves to disambiguate the candidate label set and induce the prediction model simultaneously. Comprehensive experiments show the superiority of our approach. Since disambiguation lies in the heart of partial multi-label learning, we hope that PARD will encourage more future researches to explore alternative implementations for principled disambiguation.

Acknowledgments

The authors wish to thank the anonymous reviewers for their helpful comments and suggestions. This work was supported by the National Science Foundation of China (62225602), and the Big Data Computing Center of Southeast University.

References

Emanuel Ben Baruch, Tal Ridnik, Itamar Friedman, Avi Ben-Cohen, Nadav Zamir, Asaf Noy, and Lihi Zelnik-Manor. Multi-label classification with partial annotations using classaware selective loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4754-4762, New Orleans, LA, 2022.

Matthew R. Boutell, Jiebo Luo, Xipeng Shen, and Christopher M. Brown. Learning multi-label scene classification. Pattern Recognition , 37(9):1757-1771, 2004.

Nan Cao, Teng Zhang, and Hai Jin. Partial multi-label optimal margin distribution machine. In Proceedings of the 30th International Joint Conference on Artificial Intelligence , pages 2198-2204, Montreal, Canada, 2021.

Tianshui Chen, Tao Pu, Hefeng Wu, Yuan Xie, and Liang Lin. Structured semantic transfer for multi-label recognition with partial labels. In Proceedings of the 36th AAAI Conference on Artificial Intelligence , pages 339-346, Virtual Event, 2022.

Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic. Multi-label learning from single positive labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 933-942, virtual, 2021.

Timothée Cour, Benjamin Sapp, and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research , 12:1501-1536, 2011.

Thibaut Durand, Nazanin Mehrasa, and Greg Mori. Learning a deep convnet for multi-label classification with partial labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 647-657, Long Beach, CA, 2019.

A. Elisseeff and J. Weston. A kernel method for multi-labelled classification. In Advances in Neural Information Processing Systems 14 , pages 681-687, Vancouver, Canada, 2001.

Lei Feng and Bo An. Partial label learning with self-guided retraining. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence , pages 3542-3549, Honolulu, HA, 2019.

Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. In Advances in Neural Information Processing Systems 33 , pages 10948-10960, virtual, 2020.

Xiuwen Gong, Dong Yuan, and Wei Bao. Understanding partial multi-label learning via mutual information. In Advances in Neural Information Processing Systems 34 , pages 4147-4156, virtual, 2021.

Xiuwen Gong, Dong Yuan, and Wei Bao. Partial multi-label learning via large margin nearest neighbour embeddings. In Proceedings of the 36th AAAI Conference on Artificial Intelligence , pages 6729-6736, Virtual Event, 2022.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In Proceedings of the 5th International Conference on Learning Representations , Toulon, France, 2017.

Bin-Bin Jia and Min-Ling Zhang. Multi-dimensional classification via selective feature augmentation. Machine Intelligence Research , 19(1):38-51, 2022.

Youngwook Kim, Jae-Myung Kim, Zeynep Akata, and Jungwoo Lee. Large loss matters in weakly supervised multi-label classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 14136-14145, New Orleans, LA, 2022.

Youngwook Kim, Jae-Myung Kim, Jieun Jeong, Cordelia Schmid, Zeynep Akata, and Jungwoo Lee. Bridging the gap between model explanations in partially annotated multi-label classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3408-3417, Vancouver, Canada, 2023.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations , Banff, Canada, 2014.

Himanshu Kumar, Naresh Manwani, and P. S. Sastry. Robust learning of multi-label classifiers under label noise. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD , pages 90-97, Hyderabad, India, 2020.

Kaustav Kundu and Joseph Tighe. Exploiting weakly supervised visual patterns to learn from partial annotations. In Advances in Neural Information Processing Systems 33 , pages 561-572, virtual, 2020.

Jiaran Li, Richong Zhang, Samuel Mensah, Wenyi Qin, and Chunming Hu. Classificationoriented dawid skene model for transferring intelligence from crowds to machines. Frontiers of Computer Science , 17(5):175332, 2023.

Shikun Li, Xiaobo Xia, Hansong Zhang, Yibing Zhan, Shiming Ge, and Tongliang Liu. Estimating noise transition matrix with label correlations for noisy multi-label learning. In Advances in Neural Information Processing Systems 35 , pages 24184-24198, New Orleans, LA, 2022.

Biao Liu, Ning Xu, Jiaqi Lv, and Xin Geng. Revisiting pseudo-label for single-positive multilabel learning. In Proceedings of the 40th International Conference on Machine Learning , pages 22249-22265, Honolulu, HA, 2023.

Weiwei Liu, Xiaobo Shen, Haobo Wang, and Ivor W. Tsang. The emerging trends of multi-label learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(11):7955-7974, 2022.

Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In Proceedings of the 37th International Conference on Machine Learning , pages 6500-6510, Virtual Event, 2020.

Gengyu Lyu, Songhe Feng, Yi Jin, Tao Wang, Congyan Lang, and Yidong Li. Prior knowledge regularized self-representation model for partial multilabel learning. IEEE Transactions on Cybernetics , 53(3):1618-1628, 2023.

Gengyu Lyu, Songhe Feng, and Yidong Li. Partial multi-label learning via probabilistic graph matching mechanism. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 105-113, Virtual Event, 2020.

Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete distribution: A continuous relaxation of discrete random variables. In Proceedings of the 5th International Conference on Learning Representations , Toulon, France, 2017.

Sai Rajeswar, Pau Rodríguez, Soumye Singhal, David Vázquez, and Aaron C. Courville. Multilabel iterated learning for image classification with label ambiguity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4773-4783, New Orleans, LA, 2022.

Lijuan Sun, Songhe Feng, Jun Liu, Gengyu Lyu, and Congyan Lang. Global-local label correlation for partial multi-label learning. IEEE Transactions on Multimedia , 24:581-593, 2022.

Lijuan Sun, Songhe Feng, Tao Wang, Congyan Lang, and Yi Jin. Partial multi-label learning by low-rank and sparse decomposition. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence , pages 5016-5023, Honolulu, HA, 2019.

Haobo Wang, Weiwei Liu, Yang Zhao, Chen Zhang, Tianlei Hu, and Gang Chen. Discriminative and correlative partial multi-label learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages 3691-3697, Macao, China, 2019.

Jonatas Wehrmann, Ricardo Cerri, and Rodrigo C. Barros. Hierarchical multi-label classification networks. In Proceedings of the 35th International Conference on Machine Learning , pages 5225-5234, Stockholm, Sweden, 2018.

F. Wilcoxon. Individual Comparisons by Ranking Methods , pages 196-202. Springer, Berlin, Germany, 1992.

Xuan Wu and Min-Ling Zhang. Towards enabling binary decomposition for partial label learning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence , pages 2868-2874, Stockholm, Sweden, 2018.

Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning with noisy label identification. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(7):3676-3687, 2022.

Ming-Kun Xie and Sheng-Jun Huang. CCMN: A general framework for learning with classconditional multi-label noise. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(1):154-166, 2023.

Ming-Kun Xie, Feng Sun, and Sheng-Jun Huang. Partial multi-label learning with meta disambiguation. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1904-1912, Virtual Event, 2021.

Ming-Kun Xie, Jiahao Xiao, and Sheng-Jun Huang. Label-aware global consistency for multilabel learning with single positive labels. In Advances in Neural Information Processing Systems 35 , pages 18430-18441, New Orleans, LA, 2022.

Ming-Kung Xie and Sheng-Jun Huang. Partial multi-label learning. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence , pages 4302-4309, New Orleans, LA, 2018.

Miao Xu and Lan-Zhe Guo. Learning from group supervision: The impact of supervision deficiency on multi-label learning. Science China Information Sciences , 64(3), 2021.

Ning Xu, Yun-Peng Liu, and Xin Geng. Partial multi-label learning with label distribution. In Proceedings of the 34th AAAI Conference on Artificial Intelligence , pages 6510-6517, New York, NY, 2020.

Ning Xu, Yun-Peng Liu, Yan Zhang, and Xin Geng. Progressive enhancement of label distributions for partial multilabel learning. IEEE Transactions on Neural Networks and Learning Systems , 34(8):4856-4867, 2023.

Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. In Advances in Neural Information Processing Systems 34 , pages 27119-27130, virtual, 2021.

Ning Xu, Jun Shu, Yun-Peng Liu, and Xin Geng. Variational label enhancement. In Proceedings of the 37th International Conference on Machine Learning , pages 10597-10606, Virtual Event, 2020.

Yan Yan and Yuhong Guo. Partial label learning with batch label correction. In Proceedings of the 34th AAAI Conference on Artificial Intelligence , pages 6575-6582, New York, NY, 2020.

Yan Yan and Yuhong Guo. Adversarial partial multi-label learning with label disambiguation. In Proceedings of the 35th AAAI Conference on Artificial Intelligence , pages 10568-10576, Virtual Event, 2021.

Guoxian Yu, Xia Chen, Carlotta Domeniconi, Jun Wang, Zhao Li, Zili Zhang, and Xindong Wu. Feature-induced partial multi-label learning. In Proceedings of the IEEE International Conference on Data Mining , pages 1398-1403, Singapore, 2018.

Min-Ling Zhang and Jun-Peng Fang. Partial multi-label learning via credible label elicitation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(10):3587-3599, 2020.

Min-Ling Zhang, Fei Yu, and Cai-Zhi Tang. Disambiguation-free partial label learning. IEEE Transactions on Knowledge and Data Engineering , 29(10):2155-2167, 2017.

Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering , 26(8):1819-1837, 2014.

Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review , 5(1):44-53, 2018.

Zhi-Hua Zhou. Open-environment machine learning. National Science Review , 9(8):nwac123, 2022.

Yue Zhu, James T. Kwok, and Zhi-Hua Zhou. Multi-label learning with global and local label correlation. IEEE Transactions on Knowledge and Data Engineering , 30(6):1081-1094, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on a specific problem called partial multi-label learning (PML), which is an emerging weakly-supervised learning problem. PML involves inaccurate annotations, where each training example is associated with a set of candidate labels, among which only some labels are valid. The paper proposes a probabilistic graphical model for PML and develops a principled approach for disambiguating candidate labels. The approach is evaluated through comprehensive experiments on multiple synthetic and real-world datasets, showing superior performance compared to existing methods.\n\nSTRENGTHS:\n1. The paper presents a novel approach called PARD for partial multi-label learning, which aims to recover the groundtruth labeling information from inaccurate annotations.\n2. PARD uses a directed graphical model to describe the generative process of partial multi-label data and a unified variational lower bound derived by variational inference to maximize the data log-likelihood on given partial multi-label data set.\n3. PARD achieves to disambiguate the candidate label set and induce the prediction model simultaneously, and comprehensive experiments show the superiority of this approach.\n\nWEAKNESSES:\n1. PARD provides a principled way for disambiguation, which is the process of recovering the groundtruth labeling information from inaccurate annotations. This can be useful in many real-world scenarios where the annotations are not perfect.However, the number of labels required may not meet the requirements of the actual task, and further work can be done in this regard.\n2. The use of a directed graphical model and variational inference in PARD can be extended to other machine learning tasks beyond partial multi-label learning, where probabilistic modeling and inference are required. But the construction of modeling and inference rules is very dependent on the nature of the specific task, is there a good transfom method to take advantage of the previous work?\n\nQUESTIONS:\n1. Any idea about how to explore alternative implementations for principled disambiguation?\n2. The proposed approach PARD outperforms the state-of-the-art counterparts on multiple synthetic and real-world datasets, demonstrating its effectiveness in improving the accuracy of partial multi-label learning tasks. But I'd like to see how this approach behaves in a large model environment, or how it might be combined\n3. The results of the comprehensive experiments on multiple synthetic and real-world datasets show that PARD outperforms the state-of-the-art counterparts, which indicates that this approach can be used to improve the performance of many practical applications that involve partial multi-label learning. But why does this model achieve the best experimental results on all tasks? What about other tasks?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the manuscript, the authors adopt the probabilistic graphical model to solve the disambiguation problem in partial multi-label learning. Specifically, a directed graph is tailored to describe the generative process of partial multi-label data, and a partial multi-label learning approach named PARD is proposed. PARD attempts to disambiguate the candidate label set and induce the prediction model simultaneously by maximizing the data log-likelihood with a unified surrogate objective derived by variational inference. The authors conduct extensive experiments to validate the effectiveness of the proposed method.\n\nSTRENGTHS:\nThe idea to use probabilistic graphical model for PML problem is novel. The manuscript is easy to follow.\n\nWEAKNESSES:\nThe motivation is not clear. Why do authors apply probabilistic graphical model for disambiguation in PML?\n\nThe authors claim that existing approaches are not universal enough in complicated real-world scenarios; however, in real world applications, the data distribution is hard to follow the assumptions claimed when directly tailoring the graph model, which leads to the unreasonability of the proposed approach. \n\nThe technical part is not solid, e.g., it is not clear to write the variational posterior q in the subscript of expectation through all the equations.\n\n\n-------\nThanks the author for the reply. However, I still have the concerns about this paper.\n\n1, The motivation is quite weak.  Why do we need directed graphical model for PML? What are the limitations of existing methods? Why the proposed directed graphical model achieve better performance than other methods? The author did not show the necessity of introducing the directed graphical model for PML.  Moreover, The authors make frequent assertions in the abstract and introduction about existing approaches being heuristic and ad-hoc as the SOTA limitation that motivates the paper. However, the fact is that existing work exactly solves the heuristic or ad-hoc problem.\n\n2,The key issue of this paper is the lack of the theoretical and technical contribution. The paper directly applies the very known technique graphical model for PML. I can not find any new theoretical and technical contributions in this paper.\n\nTherefore, I vote for the rejection.\n\nQUESTIONS:\nThe motivation is not clear. Why do authors apply probabilistic graphical model for disambiguation in PML?\n\nThe authors claim that existing approaches are not universal enough in complicated real-world scenarios; however, in real world applications, the data distribution is hard to follow the assumptions claimed when directly tailoring the graph model, which leads to the unreasonability of the proposed approach. \n\nThe technical part is not solid, e.g., it is not clear to write the variational posterior q in the subscript of expectation through all the equations.\n\nLIMITATIONS:\nN\\A", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces PARD , a new approach for addressing the problem of partial multi\u0002label learning. PARD uses a probabilistic graphical model to recover ground-truth labeling information. By leveraging the framework of stochastic gradient variational Bayes, PARD simultaneously identifies labeling information and induces a prediction model. Experimental  results on synthetic and real-world datasets demonstrate that PARD outperforms existing methods, providing a more effective and universal solution for partial multi-label learning.\n\nSTRENGTHS:\n1. The article presents a novel approach for addressing the problem of partial multi-label learning. It introduces a probabilistic graphical model and leverages stochastic gradient variational Bayes to simultaneously identify labeling information and induce a prediction model.\n2.The article is well-structured and effectively communicates the proposed approach, experimental setup, and results. The authors provide sufficient details, making it easier to understand and replicate the methodology.\n3.The article provides extensive experimental results on both synthetic and real-world datasets. It demonstrates that PARD outperforms existing methods, indicating its effectiveness in improving the accuracy of partial multi-label learning tasks.\n\nWEAKNESSES:\nMany papers on weakly supervised multi-label have been proposed recently, the author are  encouraged to provide a detailed discussion and if possible, comparing with them.\n\n[1] Y. Kim, J. M. Kim, Z. Akata, and J. Lee, “Large loss matters in weakly supervised multi-label classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14 156–14 165.\n[2] S. Rajeswar, P. Rodriguez, S. Singhal, D. Vazquez, and A. Courville, “Multi-label iterated learning for image classification with label ambiguity,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4783–4793.\n\nQUESTIONS:\n1.The derivation from Eq1 to Eq2 can be more detailed explanation.\n\nLIMITATIONS:\nThe article deals with the situation of partial multi label.  It seems that for general weakly supervised multi-label \nlearning like label noise or single positive label, the proposed method is not generalizable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel, systematic approach to partial multi-label learning (PML) and label disambiguation using a probabilistic graphical model and the stochastic gradient variational Bayes framework. The authors offer a reliable method tested extensively on synthetic and real-world datasets. The results underscore its performance and demonstrate its potential for future PML advancements.\n\nSTRENGTHS:\nThe paper presents an interesting and effective technique for PML using graphical models. The approach is explained clearly, although incorporating an illustration of the graphical model would enhance the paper. The extensive experiments validate the method's effectiveness. The overall structure and writing style are lucid and easy to comprehend.\n\nWEAKNESSES:\n- The authors make frequent assertions about existing approaches being heuristic and ad-hoc without providing any explanation or justification. Furthermore, there are similar approaches to the one proposed that are not adequately discussed or compared in the paper [1].\n\n- The related work section could be improved by including brief descriptions of some recent related methods.\n\n- The experimental section lacks a comparison with more recent and state-of-the-art papers such as MILI-PML [2] and PML-GAN [1].\n\n[1] Yan, Yan, and Yuhong Guo. \"Adversarial partial multi-label learning with label disambiguation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.\n\n[2] Gong, Xiuwen, Dong Yuan, and Wei Bao. \"Understanding partial multi-label learning via mutual information.\" Advances in Neural Information Processing Systems 34 (2021): 4147-4156.\n\nQUESTIONS:\n- Could the authors provide further clarification regarding the omission of the marginal distribution on x?\n\n- Why is there no discussion and comparison with the Yan and Guo paper [1], which also explores similar dependencies in label disambiguation?\n\n- While the paper claims to outperform state-of-the-art approaches, it does not specify which ones are considered state-of-the-art in the comparisons.\n\n\n[1] Yan, Yan, and Yuhong Guo. \"Adversarial partial multi-label learning with label disambiguation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.\n\nLIMITATIONS:\nlimitations were not explicitly addressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper attempts to tackle the Partial Multi-Label Learning task by employing the probabilistic graph method. The proposed method employs the directed graph to model the generative process of partial multi-label data, then derives a variational lower bound objective with a gumbel-softmax trick and closed form of KL-divergence. Empirical results on serval benchmarks demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n-\tThis paper is well-written and quite easy to follow.\n-\tThe idea of tackling partial multi-label learning with probabilistic graphical model seems to be interesting.\n-\tEmpirical results demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n-\tThe motivation of this paper is not clear. In this paper, the authors attempt to employ the probabilistic graphical model to perform a principled way for disambiguation and learn the classifier. However, why can the probabilistic graphical model achieve principled disambiguation? The simple entropy-regularized autoencode process seems to be not enough.\n-\tSome baseline methods in recent years are missing in the comparison experiment. You can consider some baselines published in 2022 and 2023, such as Global [1] and PAKS [2]. And these two papers are mentioned in the references, but the relevant citations are missing in the text.\n-\tWhat is the baseline model in the ablation study? Do you mean that it learns the prediction model with cross-entropy loss by directly treating all the candidate labels as valid ones?\n-\tIn formula 6, there is an extra '] '.\n\n[1] Lijuan Sun, Songhe Feng, Jun Liu, Gengyu Lyu, and Congyan Lang. Global-local label correlation for partial multi-label learning. IEEE Transactions on Multimedia, 24:581–593, 2022.\n\n[2] Gengyu Lyu, Songhe Feng, and Yidong Li. Partial multi-label learning via probabilistic graph matching mechanism. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 105–113, Virtual Event, 2020.\n\nQUESTIONS:\n-\tPlease see the weakness for details.\n\nLIMITATIONS:\nThere seem to be no limitations or potential negative social impact of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}