Title: Partial Label Learning with Dissimilarity Propagation guided Candidate Label Shrinkage
Venue: NeurIPS 2023
Accepted: yes
Keywords: partial label learning

dissimilarity propagation

candidate label shrinkage
Abstract: In partial label learning (PLL), each sample is associated with a group of candidate labels, among which only one label is correct. The key of PLL is to disambiguate the candidate label set to find the ground-truth label. To this end, we first construct a constrained regression model to capture the confidence of the candidate labels, and multiply the label confidence matrix by its transpose to build a second-order similarity matrix, whose elements indicate the pairwise similarity relationships of samples globally. Then we develop a semantic dissimilarity matrix by considering the complement of the intersection of the candidate label set, and further propagate the initial dissimilarity relationships to the whole data set by leveraging the local geometric structure of samples. The similarity and dissimilarity matrices form an adversarial relationship, which is further utilized to shrink the solution space of the label confidence matrix and promote the dissimilarity matrix. We finally extend the proposed model to a kernel version to exploit the non-linear structure of samples and solve the proposed model by the inexact augmented Lagrange multiplier method. By exploiting the adversarial prior, the proposed method can significantly outperform
state-of-the-art PLL algorithms when evaluated on 10 artificial and 7 real-world partial label data sets. We also prove the effectiveness of our method with some theoretical guarantees. The code is publicly available at https://github.com/Yangfc-ML/DPCLS.

Partial Label Learning with Dissimilarity Propagation guided Candidate Label Shrinkage

Abstract

In partial label learning (PLL), each sample is associated with a group of candidate labels, among which only one label is correct. The key of PLL is to disambiguate the candidate label set to find the ground-truth label. To this end, we first construct a constrained regression model to capture the confidence of the candidate labels, and multiply the label confidence matrix by its transpose to build a second-order similarity matrix, whose elements indicate the pairwise similarity relationships of samples globally. Then we develop a semantic dissimilarity matrix by considering the complement of the intersection of the candidate label set, and further propagate the initial dissimilarity relationships to the whole data set by leveraging the local geometric structure of samples. The similarity and dissimilarity matrices form an adversarial relationship, which is further utilized to shrink the solution space of the label confidence matrix and promote the dissimilarity matrix. We finally extend the proposed model to a kernel version to exploit the non-linear structure of samples and solve the proposed model by the inexact augmented Lagrange multiplier method. By exploiting the adversarial prior, the proposed method can significantly outperform state-of-the-art PLL algorithms when evaluated on 10 artificial and 7 real-world partial label data sets. We also prove the effectiveness of our method with some theoretical guarantees. The code is publicly available at https://github.com/Yangfc-ML/DPCLS .

1 Introduction

Partial label learning (PLL) [14, 15, 17] is an emerging weakly supervised learning framework. In PLL, each sample is associated with a set of candidate labels, among which only one is the groundtruth label. Different from conventional supervised learning, PLL avoids precisely annotating label on each sample, which greatly reduces the labeling cost. Accordingly, PLL has been applied to many real-world scenarios such as automatic image annotation [1, 20], web mining [9], and ecoinformatics [8].

Formally speaking, let X = R d be the d -dimensional feature space and Y = { 1 , 2 , ..., q } be the label space with q labels. Given the partial label training set D = { ( x i , S i ) | 1 ≤ i ≤ m } , where x i ∈ X is a d -dimensional feature vector to represent the i -th sample and S i ⊆ Y is the associated candidate label set, among which only one label is correct. PLL aims to induce a multi-class classifier f : X → Y from D , which is very challenging as the ground-truth label of a sample is concealed in the candidate label set.

The key to solving PLL is label disambiguation, i.e., identifying the ground-truth label of a sample from its candidate label set. For example, [6, 12] considered the ground-truth label as a latent variable and identified it through an iterative refining procedure. [3] narrowed the candidate label set through a sparsity-based self-training procedure. Some works [19, 22, 23] leveraged the similarity relationship of samples in the feature space to disambiguation, i.e., samples are similar to each other in the feature space are likely to share the same ground-truth label. Recently, some researches exploited the label space to achieve label disambiguation [2, 7, 13]. For example, SDIM [2] first built a pairwise dissimilarity matrix through the candidate label sets, and then maximized the difference of the label confidence between two samples if their pairwise dissimilarity between them is large according to the constructed dissimilarity matrix. However, the dissimilarity matrix constructed by SDIM is predefined and relatively sparse, which depresses its effectiveness.

Realizing the effectiveness of the dissimilarity relationship in PLL, we propose a novel PLL method named DPCLS (partial label learning with Dissimilarity Propagation guided Candidate Label Shrinkage). Specifically, we first construct a semantic dissimilarity matrix by considering the complement of the intersection of the candidate label set, i.e., if two samples do not share any common candidate labels, they must belong to different classes, and their semantic dissimilarity is large. The above constructed dissimilarity relationships are still sparse and fixed, we therefore propose to propagate the initial dissimilarity relationships to the whole data set by leveraging the local geometric structure of samples, i.e., if two samples are similar to each other in feature space, they are expected to share the similar dissimilarity codings. Second, to include the enhanced semantic dissimilarity matrix in label disambiguation, we design a second-order similarity matrix by multiplying the label confidence matrix with its transpose. Under the ideal condition, the enhanced semantic dissimilarity matrix and the similarity matrix naturally form an adversarial relationship, i.e., a larger (resp. smaller) dissimilarity between two samples means a smaller (resp. larger) similarity between them. By exploiting this adversarial prior, the enhanced dissimilarity matrix can shrink the solution space of the label confidence matrix, and meanwhile, the similarity matrix induced from the label confidence matrix also contributes to build a better dissimilarity matrix. We theoretically confirm the above statement under some general conditions. Furthermore, we extend our method to a kernel version to model the non-linear structure of samples. The proposed model is finally formulated as a constrained regression problem with adversarial learning and graph regularization, which is optimized by inexact augmented Lagrange multiplier (IALM). Extensive experiments on artificial and real-world partial label data sets demonstrate the effectiveness of the proposed PLL method.

2 The Proposed Method

Basic Model

Let X = [ x 1 , x 2 , · · · , x m ] T ∈ R m × d denote the feature matrix, where m and d represent the number of samples and the dimension of features. Y = [ y 1 , y 2 , · · · , y m ] T ∈ { 0 , 1 } m × q represents the partial label matrix, where q is the number of classes. Moreover, y ij = 1 indicates that the j -th label is one of the candidate labels of the sample x i . Note that in the candidate label set, only one label is correct.

To fulfill PLL, we first build the following constrained regression model

where F ∈ R m × q is the label confidence matrix with F ij representing the probability of the j -th label being the ground-truth label for the i -th sample. W ∈ R d × q is the coefficient matrix that maps the feature matrix to the label confidence matrix F . To avoid over-fitting, we impose the widely-used squared Frobenius norm on W as the regularization term, which is introduced by λ > 0 . 1 q ∈ R q and 1 m ∈ R m are two all ones vectors, 0 m × q ∈ R m × q is an all zeroes matrix. The first constraint F1 q = 1 m normalizes the label confidence vector for all samples. The second constraint 0 m × q ≤ F ≤ Y means each element of F is no less than 0, and no more than the corresponding element of Y , which implies the ground-truth label of each sample must reside in the candidate label set and the label confidence of the non-candidate labels must be 0. We initialize the label confidence matrix as F ij = 1 / ∑ j y ij if y ij =1, otherwise F ij = 0 .

By minimizing Eq. (1), we construct a linear regression model that maps the feature space to the label confidence F . We assume that the mapping from the features to the ground-truth label may

be easier, while that to the false-positive label residing in the candidate label set is relatively harder. Accordingly, optimizing Eq. (1) will help disambiguate the candidate labels and produce a preliminary label confidence matrix by exploring the useful information in the feature space.

Dissimilarity Propagation guided Candidate Label Shrinkage

To further exploit the valuable information in the label space, we first use candidate labels to construct a dissimilarity matrix D 0 ∈ R m × m , i.e.,

If y i y T j = 0 , the i -th sample x i and the j -th sample x j do not share any common candidate labels, which means they must belong to the different classes. Otherwise, x i and x j have a chance to belong to the same class. Therefore, D 0 indicates the semantic dissimilarity of samples. We then multiply the label confidence matrix with its transpose to create a similarity matrix termed as FF T , whose ( i, j ) -th element indicates the similarity between x i and x j . As the semantic dissimilarity matrix D 0 and similarity matrix FF T form an adversarial relationship, i.e., a larger (resp. smaller) element in D 0 implies a smaller (resp. larger) element in FF T , we use this adversarial prior to shrink the solution space of F by:

̸

where ⊙ denotes the element-wise product of two matrices and || · || 1 represents the l 1 norm (i.e., the sum of absolute values of all elements in a matrix). Minimizing Eq. (3) ensues that the positive elements of D 0 and FF T will lie in the different locations. Unfortunately, directly minimizing Eq. (3) cannot help produce a better label confidence matrix F , because D 0 is inferred from the candidate label set, and the ( i, j )-th element of D 0 is positive only when y i y T j = 0 , while F is upper bounded by Y , the ( i, j )-th element of FF T is positive only when y i y T j = 0 . That is the locations of the positive elements of D 0 and FF T are complementary.

To solve this problem, we propose to promote the initial dissimilarity matrix D 0 to produce a denser dissimilarity matrix D ∈ R m × m by a novel dissimilarity propagation method. Specifically, we leverage the local geometric structure of samples to enhance D 0 . Note that each column of D (e.g., D .i ) can represent the dissimilarity relationships between a sample (e.g., x i ) and the other samples. If two samples x i and x j are close to each other in the feature space, their dissimilarity relationships ( D .i and D .j ) should also be similar. To capture the feature similarity, we build a local geometric matrix S ∈ R m × m using a radial basis function (RBF) kernel:

where j ∈ N i indicates that x j is a k -nearest neighbor of x i , and σ is a hyper-parameter controlling the bandwidth of the RBF kernel. Based on S , the dissimilarity propagation guided candidate label shrinkage module becomes

The second term makes the highly similar samples in the feature samples share the similar dissimilarity codings. Moreover, D ij = D 0 ij if D 0 ij = 1 means the reliable semantic relationship should be retained in D . In order to make D a well-defined dissimilarity matrix, each element in D should lie in the range of [0,1]. Note that in Eq. (5), both F and D are optimization variables. By minimizing Eq. (5), the enhanced semantic dissimilarity matrix will shrink the solution space of F and at the same time F will also help promote the quality of D by the adversarial term.

Taking all the above considerations into account, the proposed model finally becomes:

where L ∈ R m × m = D S -S is a graph Laplacian matrix, and D S ∈ R m × m is a diagonal matrix with the i -th diagonal element D S ii = ∑ m j =1 S ij . Tr ( · ) returns the trace of a matrix. α , β ≥ 0 are two hyper-parameters to balance different terms. By solving Eq. (6), the semantic dissimilarity is enhanced by a dissimilarity propagation process, and further utilized to shrink the solution space of the label confidence matrix by a novel adversarial prior.

3 Optimization and Setting

We adopt IALM to solve the problem in Eq. (6). To simplify Eq. (6), we introduce an auxiliary matrix A = D ∈ R m × m and equivalently rewrite it as

The solution of Eq. (7) can be obtained by solving the following augmented Lagrange equation:

where Φ ∈ R m × m is the Lagrange multiplier matrix, µ ≥ 0 is a penalty parameter, and ⟨· , ·⟩ returns the inner product of two matrices. We can optimize Eq. (8) by solving the following subproblems alternatively and iteratively.

1) W subproblem is formulated as

Kernel Extension

The linear mapping in Eq. (9) may fail to model the nonlinear relationship. Therefore, we extend the above model to a kernel-based non-linear version. Let ϕ ( · ) : R d → R h denote the feature transformation that maps the origin feature space X to a higher dimensional Hilbert space ϕ ( X ) . According to the Representer Theorem, W can be expressed as a linear combination of the input features, i.e., W = ϕ ( X ) T H , where H ∈ R m × q stores the combination weights. Then, we have ϕ ( X ) W = KH , where K = ϕ ( X ) ϕ ( X ) T ∈ R m × m is the kernel matrix and each element K ij = K ( x i , x j ) . Finally, the nonlinear version is represented as:

where b ∈ R q is the bias term. When the first derivatives of H and b reach 0, Eq. (10) is solved, i.e.,

where I m × m ∈ R m × m denotes an identity matrix. In the experiments, we use the RBF kernel as the kernel function, i.e., K ( x i , x j ) = exp ( -|| x i -x j || 2 2 /σ 2 ) , for our method and the compared ones. 2) F subproblem is written as

where P = KH + 1 m b T ∈ R m × q is the output matrix of the model. Eq. (12) can be formulated as a standard quadratic programming (QP) problem, and solved by any QP tools. The detailed derivation process can be found in Section A of the supplementary file. 3) D subproblem is represented as

Eq. (13) reaches the minimum when its first-order derivative with respect to D vanishes, leading to

4) A subproblem is expressed as

Eq. (15) can solved element-wisely, i.e.,

Algorithm 1 The Pseudo Code of the Proposed Method

Input : D : the partial label training set; λ, α, β, k : the parameters of model; ̂ x : an unseen test sample Output : y : the predicted label for sample x

1: Construct the dissimilarity matrix D 0 according to Eq. (2) and the kernel matrix K = [ K ( x i , x j )] m × m

-8

̂ ̂ 2: Initialize D = A = Φ = 0 m × m , µ = 10 -4 3: while not converged do 4: Update H and b by Eq. (11) 5: Update F by solving Eq. (12) 6: Update D by Eq. (14) 7: Update A by Eq. (16) 8: Update Φ , µ by Eq. (17) 9: Check the convergence condition ∥ D -A ∥ ∞ < 10 10: end while 11: Return the predicted label ̂ y according to Eq. (18).

where T , T 1 , T 0 are three thresholding operators in elementwise, i.e., T ( C ij ) = 1 , if D 0 ij = 1 , T 1 ( C ij ) := min (1 , C ij ) , T 0 ( C ij ) := max (0 , C ij ) .

Finally, the Lagrangian multiplier matrix and µ are updated by

where µ max= 10 10 is a predefined upper bound for µ .

Model Prediction

Given an unseen test example ̂ x , our method predicts its label by

The overall pseudo code of our method is summarized in Algorithm 1, where it stops when ∥ D -A ∥ ∞ < 10 -8 , where ∥ · ∥ ∞ denotes the infinity norm of a matrix.

Hyper-parameter Settings of Our Method

Parameter λ is used to control the model complexity. A too large (resp. small) value of λ will lead to under-fitting (resp. over-fitting). Therefore, we set λ =0.05 for our method. Parameters α and β are used to control the importance of the adversarial term and dissimilarity propagation term respectively. According to a number of experiments, we fix β = 0 . 001 and select α from { 0 . 001 , 0 . 01 } . Parameter k controls the number of k -nearest neighbors. Following the previous works [16, 22], we set k =10.

Computational Complexity Analysis

The computational complexity of Algorithm 1 is mainly determined by steps 4-7. Specifically, steps 4 and 6 involve the inversion of m × m matrices with the complexity of O ( m 3 ) . Step 5 solves a QP problem, leading to the complexity of O ( m 3 q 3 ) . Step 7 can be efficiently solved by linear thresholding operations with the complexity of O ( m 2 ) . Therefore, the overall computational complexity of Algorithm 1 in each iteration is O (2 m 3 + m 2 + m 3 q 3 ) . More analysis about computational complexity can be found in Section A of the supplementary file.

4 Theoretical Analysis

Theorem 1. The square loss function ℓ of DPCLS can be rewritten as ∥ F G + N -XW ∥ 2 F , where F G ∈ R m × q and N ∈ R m × q are ground-truth label matrix and false-positive label matrix respectively. Let H = W × N represent the family of functions for DPCLS, where the linear function ( W , N ) ∈ H . Suppose the complexity of W and the sparsity of N are upper bounded by ϵ 1 and ϵ 2 respectively, i.e., ∥ W ∥ F ≤ ϵ 1 and ∥ N ∥ 1 ≤ ϵ 2 . The Rademacher complexity of DPCLS with square loss ℓ is upper bounded as follow

Lemma 1. [11] Denote H be a family of functions and S = { x 1 , x 2 , ..., x m } is a set of fixed samples. Loss function ℓ is upper bounded by Θ ≥ 0 , then for any δ > 0 , with probability at least 1 -δ , for all h ∈ H we have

where L ( h ) and L S ( h ) are generalization error and empirical error to h respectively.

The detailed proof of Theorem 1 is given in the Section B of the supplementary file. According to Lemma 1 and Theorem 1 , we have

The Rademacher complexity is bounded by the sum of the complexity of classifier W and the sparsity of false-positive label matrix N . When the number of false-positive labels is small, which leads to a better generalization performance. Moreover, more training samples (a larger m ) will also promote the generalization performance.

Theorem 2. Denote F ∈ { 0 , 1 } m × q and D ∈ { 0 , 1 } m × m the partial label matrix and the to be optimized semantic dissimilarity matrix. Let F G and ˆ D be the ground-truth label matrix and the ground-truth dissimilarity matrix. Suppose the smallest eigenvalue of ˆ D and L are λ ˆ D and λ L respectively ( λ ˆ D ≥ 0 , λ L ≥ 0 ). Let ∥ ∥ ¯ ∆ F ∥ ∥ F be the average distance of each sample between F G and F (i.e., ∥ ∥ ¯ ∆ F ∥ ∥ F = 1 m ∥ F G -F ∥ F ) and ∥ ∥ ¯ ∆ D ∥ ∥ F be the average distance of each corresponding position between ˆ D and D (i.e., ∥ ∥ ¯ ∆ D ∥ ∥ F = 1 m 2 ∥ ∥ ∥ ˆ D -D ∥ ∥ ∥ F ). Then we have

The proof can be found in Section B of the supplementary file. From Theorem 2 , we can find that as the number of samples m increases, the upper bound of ∥ ∥ ¯ ∆ F ∥ ∥ F decreases, which indicates that more training samples will push the partial label matrix to be close to the ground-truth one and achieve better PLL performance. Moreover, a smaller error between D and ˆ D implies a smaller upper bound of ∥ ∥ ¯ ∆ F ∥ ∥ F , which indicates a better dissimilarity matrix can help achieve a better label matrix. Similarly, a larger number of training samples will reduce the distance between D and ˆ D , and a smaller error between F and F G implies a smaller upper bound of ∥ ∥ ¯ ∆ D ∥ ∥ F , suggesting a better dissimilarity matrix. As a summary, we prove that, under some general assumptions, a better dissimilarity matrix will help produce a better label matrix, and vice versa. Therefore, the rationality of the proposed adversarial prior is theoretically proved.

5 Experiment and Analysis

To demonstrate the effectiveness of the proposed model, we compared DPCLS with eight shallow PLL algorithms, which were configured by the suggested parameters in the literature, i.e., CLPL [1], PL-SVM [12], PL-KNN [5], PL-DA [18], IPAL [22], AGGD [16], PL-CLA [13], SDIM [2]. Those methods were evaluated on 10 synthetic data sets and 7 real-world data sets, whose details can be found in Section C of the supplementary file. Three deep learning PLL methods RC [4], PRODEN [10], and CAVL [21] were also evaluated as comparison on the real-world data sets. Ten runs of 50% / 50% random train/test splits were performed on each data set, and the average classification accuracy and the standard deviation were recorded.

5.1 Performance on Synthetic Data Sets

Following the widely-used partial label data generation protocol [1], 10 synthetic data sets were used to generate artificial partial label data sets. Specifically, three parameters r, p, ϵ control the generation process, i.e., p controls the proportion of partial label examples, r controls the number of false-positive labels, and ϵ controls the probability of a specific false-positive label co-occurs with the ground-truth label.

Fig. 1 illustrates the classification accuracy of six data sets (Glass, Steel, Ecoli, Yeast, Optdigits and Usps) as the co-occurring probability ϵ varies from 0.1 to 0.7 with step-size 0.1 ( p =1, r =1). In general, the proposed model clearly exceeds the compared methods with different ϵ , and achieves the highest accuracy in 33 out of 36 cases. Moreover, as ϵ increases, it is more difficult to distinguish the

DPCLS

CLPL

PL-SVM

PL-KNN

PL-DA

IPAL

AGGD

PL-CLA

SDIM

0.3

0.4

0.5

0.6

0.7

0.8

(co-occurring probability of the coupling label)

0.45

0.5

0.55

0.6

0.65

Classification Accuracy

p

=1,

r

=1)

0.3

0.4

0.5

0.6

0.7

0.8

(co-occurring probability of the coupling label)

0.55

0.6

0.65

0.7

Classification Accuracy

(b) Steel (

p

=1,

r

=1)

0.3

0.4

0.5

0.6

0.7

0.8

(co-occurring probability of the coupling label)

0.75

0.8

0.85

Classification Accuracy

p

=1,

r

=1)

0.3

0.4

0.5

0.6

0.7

0.8

(co-occurring probability of the coupling label)

0.45

0.5

0.55

Classification Accuracy

(d) Yeast (

p

=1,

r

=1)

0.3

0.4

0.5

0.6

0.7

0.8

(co-occurring probability of the coupling label)

0.8

0.85

0.9

0.95

Classification Accuracy

(e) Optdigits (

p

=1,

r

=1)

0.3

0.4

0.5

0.6

0.7

0.8

(co-occurring probability of the coupling label)

0.85

0.9

0.95

Classification Accuracy

(f) Usps (

p

=1,

r

=1)

false-positive label and the ground-truth label, and PLL becomes harder. Accordingly, the accuracies of all the PLL algorithms decrease, but the performance advantage of the proposed method over the compared ones becomes more salient such as on Glass and Ecoli, which proves that our method is skilled in more challenging PLL tasks.

Table 1 reports the classification accuracies of different algorithms as r varies from 3 to 6 on four data sets (Isolet, Orl, Amazon, Bookmark) with more than 20 classes. Our method ranks first in 11 out of 12 cases, and ranks second in the remaining case. The improvements brought by our method are significant. For example, on Isolet with r = 6 , our method improves the classification accuracy of the best comparison from 0.824 to 0.858. According to the pairwise t-test, our method significantly surpasses the compared ones in 103 out of 128 cases. Moreover, different methods have different characteristics and may fit different data sets. For example, CLPL performs satisfactorily on Orl and Amazon, but badly on Isolet and Bookmark; AGGD is good at Bookmark, but not at Orl.

0.7

Different from them, the proposed method produces excellent performance on all the evaluated data sets, suggesting its robustness to different data sets.

Table 3 (row (I)) reports win/tie/loss counts between DPCLS and eight comparing algorithms on the synthetic data sets according to the pairwise t-test at the significance level of 0.05, where we can find that DPCLS statistically outperforms other algorithms in 79.6% cases (331 out of 416) and none of the algorithms can beat DPCLS significantly.

5.2 Performance on Real-world Data Sets

Real-world data sets were collected from various tasks and domains. As the average size of the candidate label set of FG-NET is large, which will cause low classification accuracy on the test set. Following [2, 16] we employed the mean absolute error (MAE) to further calculate two extra evaluation indicators MAE3 and MAE5 on FG-NET, i.e., the test examples are considered to be correctly classified if the difference between the predicted age and the ground-truth age is no more than 3/5 years.

Table 2 demonstrates the classification accuracies of different methods on the real-world data sets. It is obvious that our method ranks first in all cases when compared with shallow PLL algorithms. Moreover, according to the pairwise t-test, our method statistically outperforms PL-SVM, PL-KNN and PL-DA on all real-world data sets and statistically surpasses AGGD, PL-CLA and SDIM on six data sets except FG-NET. Furthermore, compared with the best comparisons, our method improves the classification accuracy from 0.494 to 0.557 on MSRCv2 and from 0.747 to 0.770 on Lost.

We also compare the classification accuracies of our method with three deep learning PLL methods RC, PRODEN, CAVL on the real-world data sets. Our method ranks first on six the data sets except Yahoo! News. Specifically, according to the pairwise t-test, our method statistically outperforms RC, PRODEN and CAVL on three real-world data sets (Lost, MSRCv2 and BirdSong). Furthermore, compared with the best comparisons of deep leraning PLL methods, our method improves the classification accuracy from 0.446 to 0.557 on MSRCv2 and from 0.715 to 0.751 on BirdSong. Therefore, we can conclude that our method shows competitive performance compared with the deep learning PLL methods.

20

40

60

80

100

120

140

160

20

40

60

80

100

120

140

160

(a)

Ideal

FF

T

20

40

60

80

100

120

140

160

20

40

60

80

100

120

140

160

(b)

Original

FF

T

20

40

60

80

100

120

140

160

20

40

60

80

100

120

140

160

(c)

FF

T

by DPCLS

20

40

60

80

100

120

140

160

20

40

60

80

100

120

140

160

(d)

Ideal

D

20

40

60

80

100

120

140

160

20

40

60

80

100

120

140

160

(e)

Original

D

0

20

40

60

80

100

120

140

160

20

40

60

80

100

120

140

160

(f)

D

by DPCLS

Figure 2: Visual comparison of the similarity matrix and the semantic dissimilarity matrix on data set Ecoli ( p =1, r =1, ϵ =0.8). (a), (d) Ideal similarity matrix and semantic dissimilarity matrix; (b), (e) Initial similarity matrix FF T and semantic dissimilarity matrix D 0 ; (c), (f) The similarity matrix FF T and the dissimilarity matrix D produced by DPCLS. The brighter color indicates a larger value.

Table 3 (row (II)) presents the win/tie/loss counts of the proposed method against the compared ones on real-world data sets according to the pairwise t-test at significance level of 0.05. On all the real-world data sets, DPCLS achieves significantly superior performance in 85.9% cases, and gets comparable performance in the remaining cases against the comparing algorithms. Taking both the synthetic data sets and real-world data sets into account, our method accomplishes significantly better performance in 80.8% cases, which indicates the excellent classification ability of our method.

5.3 Further Analysis

Visualization Fig. 2 visually compares the ideal, the original and the enhanced similarity matrix and semantic dissimilarity matrices on data set Ecoli ( p =1, r =1, ϵ =0.8). Compared with the ideal similarity matrix, the value of each element in the original similarity matrix FF T is relatively small with many incorrect connections. Compared with the ideal dissimilarity matrix, the original dissimilarity matrix D 0 is relatively sparse, and many positive elements are missed in D 0 . On the contrary, the similarity matrix and the dissimilarity matrix produced by DPCLS become denser, which are quite close to the ideal ones. Especially, when checking the areas highlighted by the red rectangle boxes, DPCLS can effectively recover the similarity relationship between samples in FF T , and many positive elements missed by D 0 have been recovered by our method. The visual comparison illustrates that our method can produce much higher-quality similarity and dissimilarity matrices, which is useful to find a high-quality label confidence matrix F and promote the performance of PLL.

Ablation Study In Table 4, we conduct an ablation study on the real-world data sets to check the necessity of the involved terms of our method. Specifically, DPCLS-LM denotes DPCLS without Dissimilarity Propagation and Kernel Extension, and DPCLS-KE and DPCLS-DP indicate DPCLS without Kernel Extension and without Dissimilarity Propagation respectively. From Table 4, we can find that both the kernel extension and the dissimilarity propagation are helpful in improving classification accuracy and taking both of them into account is the best choice.

6 Conclusion

In this paper, we have presented a novel PLL method. Specifically, we first construct a similarity matrix based on the multiplication of the label confidence matrix and its transpose. Then, we develop a dissimilarity matrix by exploiting the label space, and further utilize the local geometric structure of the samples to enhance the dissimilarity matrix, i.e., propagating the initial semantic dissimilarity relationships to the whole data set. The similarity and dissimilarity matrices form an adversarial relation, and the proposed model takes advantage of this novel adversarial prior to shrink the solution space of the label confidence matrix, which contributes to find the correct label in the candidate label set. Extensive experiments and comparisons on artificial and real-world partial label data sets have validated the effectiveness of our approach.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62106044 and 62072100, in part by the Natural Science Foundation of Jiangsu Province under Grant BK20210221, and in part by the ZhiShan Youth Scholar Program from Southeast University under Grant 2242022R40015.

References

Timothée Cour, Benjamin Sapp, and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research , 12:1501-1536, 2011.

Lei Feng and Bo An. Partial label learning by semantic difference maximization. In International Joint Conference on Artificial Intelligence , pages 2294-2300, 2019.

Lei Feng and Bo An. Partial label learning with self-guided retraining. In AAAI Conference on Artificial Intelligence , pages 3542-3549, 2019.

Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems , 2020.

Eyke Hüllermeier and Jürgen Beringer. Learning from ambiguously labeled examples. Intelligent Data Analysis , 10(5):419-439, 2006.

Rong Jin and Zoubin Ghahramani. Learning with multiple labels. In Advances in Neural Information Processing Systems 15 , pages 897-904, 2002.

Changchun Li, Ximing Li, and Jihong Ouyang. Learning with noisy partial labels by simultaneously leveraging global and local consistencies. In ACM International Conference on Information and Knowledge Management , pages 725-734, 2020.

Li-Ping Liu and Thomas G. Dietterich. A conditional multinomial mixture model for superset label learning. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems , pages 557-565, 2012.

Jie Luo and Francesco Orabona. Learning from candidate labeling sets. In Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems , pages 1504-1512, 2010.

Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In International Conference on Machine Learning , volume 119, pages 6500-6510, 2020.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . The MIT Press, 2nd edition, 2018.

Nam Nguyen and Rich Caruana. Classification with partial labels. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 551-559, 2008.

Peng Ni, Suyun Zhao, Zhi-Gang Dai, Hong Chen, and Cui-Ping Li. Partial label learning via conditional-label-aware disambiguation. J. Comput. Sci. Technol. , 36(3):590-605, 2021.

Cai-Zhi Tang and Min-Ling Zhang. Confidence-rated discriminative partial label learning. In Satinder Singh and Shaul Markovitch, editors, AAAI Conference on Artificial Intelligence , pages 2611-2617, 2017.

Yingjie Tian, Xiaotong Yu, and Saiji Fu. Partial label learning: Taxonomy, analysis and outlook. Neural Networks , 161:708-734, 2023.

Deng-Bao Wang, Min-Ling Zhang, and Li Li. Adaptive graph guided disambiguation for partial label learning. IEEE Trans. Pattern Anal. Mach. Intell. , 44(12):8796-8811, 2022.

Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico: Contrastive label disambiguation for partial label learning. In International Conference on Learning Representations , 2022.

Wei Wang and Min-Ling Zhang. Partial label learning with discrimination augmentation. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1920-1928, 2022.

Ning Xu, Jiaqi Lv, and Xin Geng. Partial label learning via label enhancement. In AAAI Conference on Artificial Intelligence , pages 5557-5564, 2019.

Zinan Zeng, Shijie Xiao, Kui Jia, Tsung-Han Chan, Shenghua Gao, Dong Xu, and Yi Ma. Learning by associating ambiguously labeled images. In IEEE Conference on Computer Vision and Pattern Recognition , pages 708-715, 2013.

Fei Zhang, Lei Feng, Bo Han, Tongliang Liu, Gang Niu, Tao Qin, and Masashi Sugiyama. Exploiting class activation value for partial-label learning. In International Conference on Learning Representations , 2022.

Min-Ling Zhang and Fei Yu. Solving the partial label learning problem: An instance-based approach. In International Joint Conference on Artificial Intelligence , pages 4048-4054, 2015.

Min-Ling Zhang, Bin-Bin Zhou, and Xu-Ying Liu. Partial label learning via feature-aware disambiguation. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1335-1344, 2016.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this submission, authors propose a novel partial label learning method named DPCLS by realizing the effectiveness of the dissimilarity relationship. They develop a semantic similarity and dissimilarity matrix which form an adversarial relationship, which is further utilized to shrink the solution space of the label confdence matrix and promote the dissimilarity matrix.\n\nSTRENGTHS:\n1. Partial label learning is an interesting topic.\n2. A novel partial label learning approach is proposed.\n3. Experiments validate the effectiveness of the proposed approach.\n\nWEAKNESSES:\n1. The motivation is unclear and the novelty might be limited.\n2. The technical details of proposed approach can be further discussed.\n3. More experiments can be conducted to validate the effectiveness of the proposed approach.\n\nQUESTIONS:\n1. The submission does propose a partial label learning approach, but the motivation for proposing it is unclear. Why similarity and dissimilarity matrix? From this pespective, the novelty is very limited.\n2. Why $l_1$ norm in Eq.(3)? Some comparative experiments?\n3. The hyper-parameter $\\sigma$ in Eq.(4) is not given.\n4. According to Figure S1(a) and Figure S1(b) in supplementary material, the smaller the value of $\\alpha$, the better the performance (with some $\\beta$). Thus, it is suggested to do further ablation studies where $\\lambda$, $\\alpha$, $\\beta$ in Eq.(6) are respectively set to zeros.\n5. As shown in Algorithm 1, the proposed approach works in an iterative manner, so the convergence analysis should be conducted? How many rounds the approach will converge?\n6. As shown in Table 1, the traditional baseline CLPL achieves relative superior performance while the newest baselines only achieves relative general performance? Data sets matters?\n\nLIMITATIONS:\nLimitations are not discussed in the current version.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach to partial label learning, called DPCLS, which learns the similarity and dissimilarity matrices to improve labeling accuracy in an adversarial relationship. The proposed method is compared to several existing methods on a variety of datasets, and the results demonstrate its superior performance in most cases. The paper also includes theoretical proof of the rationality of the proposed adversarial prior and a visualization of the enhanced similarity and dissimilarity matrices on a real-world dataset.\n\nSTRENGTHS:\n1. This paper introduces a new partial-label learning method called DPCLS, which leverages similarity and dissimilarity matrices an adversarial relationship to effectively tackle the challenges associated with partial-label learning.\n\n2. A theoretical proof of the rationality of the proposed adversarial prior is included to further validate the proposed method’s effectiveness.\n\n3. Extensive experiments conducted on both real-world and artificial partial label datasets showcase the efficacy of the proposed method.\n\nWEAKNESSES:\nThe paper focus on an interesting partial label learning problem and has several issues that can be improved: the datasets adopted in this paper are small-scale and the motivation of this paper should be explained in detail.\n\nQUESTIONS:\n1. In Fig.1 (c), compared with the co-occurring probability of 0.7, it confused me that the proposed method seems to get a better performance in the co-occurring probability of 0.8.\n\n2. The datasets adopted in the paper are simple and small-scale which is hardly convincing, and the moderately sized datasets are suggested to adopt to validate the effectiveness of the proposed method. For example, CIFAR-100.\n\n3. I am confused about the reported comparison results in Table 2: for the comparison method CAVL, there is a big performance drop between the reported experimental results and the CAVL paper. And can you compare the proposed method with the representative PLL method PiCO?\n\n4. For hyper-parameters sensitivity, the author should provide the variance in the figure.\n\n5. The motivation of the paper should be further explained in detail.\n\nLIMITATIONS:\nPlease refer to the weakness and limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper constructs a second-order similarity matrix and a semantic dissimilarity matrix. The similarity matrix is obtained by leveraging the confidence obtained from the underlying model, while the semantic dissimilarity matrix is determined based on the label candidate set and the distribution of samples in the feature space. An objective function is formulated using the adversarial relationship between the similarity matrix and the dissimilarity matrix. Additionally, the paper proposes a version of the algorithm to handle nonlinear problems by mapping the original feature space to a high-dimensional reproducing kernel Hilbert space (RKHS). Finally, the proposed method is compared with several state-of-the-art PLL algorithms and validated on ten  synthetic datasets and seven real-world datasets.

STRENGTHS:
1.	An adversarial relationship is constructed for label disambiguation, and in addition, two versions are proposed: linear separable and non-linear separable.
2.	Solid theoretical analysis and ample amount of work.

WEAKNESSES:
1.	The research motivation of the paper is not clearly stated.
2.	The paper mainly focuses on describing the proposed method, without summarizing and extracting issues from existing PLL research.
3.	The principle behind the construction of the loss term based on adversarial priors is not explained.

QUESTIONS:
1.	What was the motivation behind the author's paper? What problems in PLL did it solve?
2.	If the relationship between two samples is characterized by both low confidence and low dissimilarity, can the loss term based on adversarial priors handle this situation?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method for partial label learning called Dissimilarity Propagation guided Candidate Label Shrinkage（DPCLS）. The method captures the confidence of candidate labels by constructing a constrained regression model and uses the product of the label confidence matrix and its transpose to build a second-order similarity matrix. Additionally, the method constructs a semantic dissimilarity matrix by considering the complement of the intersection of candidate label sets and propagating the initial dissimilarity relationships throughout the entire dataset using the local geometric structure of the samples. The adversarial relationship between the similarity and dissimilarity matrices is further utilized to narrow down the solution space of the label confidence matrix and facilitate the construction of the dissimilarity matrix. The method is evaluated on artificial datasets and real-world partial label datasets, demonstrating superior performance compared to existing partial label learning algorithms.\n\nSTRENGTHS:\n1. The paper introduces a unique combination of dissimilarity propagation and guided candidate label shrinkage for PLL, offering a fresh perspective on the problem.\n2. The authors present a detailed framework, including the construction of similarity and dissimilarity matrices, leveraging local geometric structures, and extension to a kernel version, providing a comprehensive solution for PLL.\n3. The proposed method is extensively evaluated on multiple artificial and real-world datasets, demonstrating its effectiveness and outperforming existing algorithms.\n\nWEAKNESSES:\n1. The structure and logic of the paper need further improvement. I recommend that the authors provide a clearer background and motivation in the introduction, enumerate the contributions of the paper, and provide an overview of the overall framework of the paper.\n2. In the setting of hyperparameters of the method, the authors do not provide a specific method or principle for parameter selection. For parameters λ, α, β, and k, the authors only mention fixed values or ranges without explaining how to choose these parameters to obtain optimal performance. It is recommended that the authors provide guidance or experimental results regarding hyperparameter selection.\n3. The paper has a limited number of references and lacks a comprehensive review of the latest research in the relevant field. I recommend that the authors conduct a more thorough literature search on the relevant field and provide more background and explanations of related studies in the introduction and related work sections.\n4. The proposed model involves constructing similarity and dissimilarity matrices and solving the problem using the augmented Lagrange multiplier method, which may result in higher computational complexity compared to simpler approaches. It is recommended to perform complexity analysis on the algorithm.\n5. In the conclusion section, the authors can further discuss the limitations of the proposed method and directions for future improvements to enhance the completeness of the conclusion.\n\nQUESTIONS:\nSee the above.\n\nLIMITATIONS:\nSee the above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript delineates an innovative method, termed as DPCLS, which is designed for partial label learning. In addressing the issue  - namely label disambiguation - the DPCLS method exhibits an amalgamation of similarity relationship and dissimilarity relationship in an adversarial manner that endows the method with superior performance in comparison to baseline methods. \n\nThe manuscript manifests commendable quality through its comprehensive and well-crafted approach toward partial label learning. The clarity is evident in the lucid and well-structured exposition of the proposed methodology. The originality lies in the adversarial learning of similarity and dissimilarity relationships, addressing the challenges posed in partial label learning. The paper's significance is underscored by its superior performance over baseline methods, along with its potential to inspire future research in this domain.\n\nSTRENGTHS:\nS1. The DPCLS method's novelty is encapsulated in its adversarial learning manner of similarity and dissimilarity relationships, effectively addressing the unique challenges in partial label learning.\n\nS2. The paper employs a sound theoretical approach with clear and well-structured explanations of the methodology and its components.\n\nS3. The evaluation is comprehensive, with comparisons to baseline methods providing a compelling demonstration of the superior performance of the DPCLS method.\n\nWEAKNESSES:\nW1.  Although the paper provides a comprehensive explanation of the methodology, further technical insights regarding the implementation and specific algorithms within the DPCLS method would be beneficial. For example, does the order of Steps. 4-9 have an influence on the final performance? The auxiliary matrix $A$ needs more clarification, which is not easy to follow.\n\nW2.  The paper falls short in providing a detailed analysis of the limitations of the proposed approach, a factor that could be significant for future research and practical applications.\n\nW3.  Too many hyper-parameters in a method would somewhat degrade its quality.\n\nW4.  A more detailed exposition of existing methods related to this work, including their characteristics and potential biases or weaknesses, would enrich the manuscript.\n\nW5.  A clear motivation about real-world applications or potential application scenarios, would strengthen the practical significance.\n\nW6.  Some claims in Section 4 Theoretical Analysis need more clarifications, e.g., the number of training samples, and dissimilarity matrix.\n\nQUESTIONS:\nPlease find my comments on weaknesses.\n\nLIMITATIONS:\nPlease find my comments on weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}