Title: Class-Distribution-Aware Pseudo-Labeling for Semi-Supervised Multi-Label Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Semi-supervised multi-label learning

pseudo labeling.
Abstract: Pseudo-labeling has emerged as a popular and effective approach for utilizing unlabeled data. However, in the context of semi-supervised multi-label learning (SSMLL), conventional pseudo-labeling methods encounter difficulties when dealing with instances associated with multiple labels and an unknown label count. These limitations often result in the introduction of false positive labels or the neglect of true positive ones. To overcome these challenges, this paper proposes a novel solution called Class-Aware Pseudo-Labeling (CAP) that performs pseudo-labeling in a class-aware manner. The proposed approach introduces a regularized learning framework incorporating class-aware thresholds, which effectively control the assignment of positive and negative pseudo-labels for each class. Notably, even with a small proportion of labeled examples, our observations demonstrate that the estimated class distribution serves as a reliable approximation. Motivated by this finding, we develop a class-distribution-aware thresholding strategy to ensure the alignment of pseudo-label distribution with the true distribution. The correctness of the estimated class distribution is theoretically verified, and a generalization error bound is provided for our proposed method. Extensive experiments on multiple benchmark datasets confirm the efficacy of CAP in addressing the challenges of SSMLL problems.

Class-Distribution-Aware Pseudo-Labeling for Semi-Supervised Multi-Label Learning

Abstract

Pseudo-labeling has emerged as a popular and effective approach for utilizing unlabeled data. However, in the context of semi-supervised multi-label learning (SSMLL), conventional pseudo-labeling methods encounter difficulties when dealing with instances associated with multiple labels and an unknown label count. These limitations often result in the introduction of false positive labels or the neglect of true positive ones. To overcome these challenges, this paper proposes a novel solution called Class-Aware Pseudo-Labeling (CAP) that performs pseudolabeling in a class-aware manner. The proposed approach introduces a regularized learning framework incorporating class-aware thresholds, which effectively control the assignment of positive and negative pseudo-labels for each class. Notably, even with a small proportion of labeled examples, our observations demonstrate that the estimated class distribution serves as a reliable approximation. Motivated by this finding, we develop a class-distribution-aware thresholding strategy to ensure the alignment of pseudo-label distribution with the true distribution. The correctness of the estimated class distribution is theoretically verified, and a generalization error bound is provided for our proposed method. Extensive experiments on multiple benchmark datasets confirm the efficacy of CAP in addressing the challenges of SSMLL problems. The implementation is available at https://github.com/milkxie/SSMLL-CAP .

1 Introduction

In single-label supervised learning, each instance is assumed to be associated with only one class label, while many realistic scenarios may be multi-labeled , where each instance consists of multiple semantics. For example, an image of a nature landscape often contains the objects of sky , cloud , and mountain . Multi-label learning (MLL) is a practical and effective paradigm for handling examples with multiple labels. It trains a classifier that can predict all the relevant labels for unseen instances based on the given training examples. A large number of recent works have witnessed the great progress that MLL has made in many practical applications [27], e.g. , image annotation [21], protein subcellular localization [28], and visual attribute recognition [32].

Thanks to its powerful capacity, the deep neural network (DNN) has become a prevalent learning model for handling MLL examples [34]. Unfortunately, it requires a large number of precisely labeled

examples to achieve favorable performance. This leads to a high cost of manual annotation, especially when the dataset is large and the labeling task must be carried out by an expert. Given that it is hard to train an effective DNN based on a small subset of training examples, it is rather important to exploit the information from unlabeled instances. The problem has been formalized as a learning framework called semi-supervised multi-label learning (SSMLL), which aims to train a classifier based on a small set of labeled MLL examples and a large set of unlabeled ones.

Compared to semi-supervised learning (SSL) that has made great progress [4, 38], SSMLL has received relatively less attention in the context of deep learning. Generally, there are still three main challenges towards the development of SSMLL. Firstly, since each instance is associated with multiple labels and the number is unknown, the commonly used pseudo-labeling strategy that selects the most probable label or the topk probable labels cannot be applied to the SSMLL problems. It would face the dilemma of either introducing false positive labels or neglecting true positive ones. Secondly, due to the intrinsic class-imbalance property of MLL data, it is hard to achieve favorable performance by using a fixed threshold for each instance. Thirdly, recent studies have mainly focused on multi-label learning with missing labels (MLML) [11, 17, 2] scenarios, where each training instance is assumed to be assigned with a subset of true labels. Unfortunately, these methods often fail to achieve favorable performance, or cannot even be applied to the SSMLL scenarios, since most of them were designed under the assumption of MLML.

To solve these challenges, in this paper, we propose a novel Class-Aware Pseudo-labeling (CAP) method for handling the SSMLL problems. Unlike the existing methods, we perform pseudo-labeling in a class-aware manner to avoid estimating the number of true labels for each instance, which can be very hard in practice. Specifically, a regularized learning framework is proposed to determine the numbers of positive and negative pseudo-labels for each class based on the class-aware thresholds. Given that the true class distribution is unknown, we alternatively determine the thresholds based on the estimated class distribution of labeled data, which can be a tight approximation according to our observation. Our theoretical results show the correctness of estimated class distribution and provide a generalization error bound for CAP. Extensive experimental results on multiple benchmark datasets with a variety of comparing methods validate that the proposed method can achieve state-of-the-art performance.

2 Related Work

Thanks to the powerful learning capacity of DNNs, MLL has made great advances in the context of deep learning. Some methods designed architectures [6] or training strategies [22, 49] to exploit the label correlations. Some other methods designed sophisticated loss functions to improve the performance of MLL [34]. The last group of methods designed specific architectures to capture the objects related to semantic labels. Global-average-pooling (GAP) based models [33] and attentionbased models [22, 26] are two groups of representative methods.

There are relatively few works that study how to improve the performance of deep models in SSMLL scenarios. Instead of end-to-end training, the only deep SSMLL method [43] performed the twostage training, which first used a DNN to extract features, and then used a linear model to perform classification. [37] proposed a deep sequential generative model to handle the noisy labels collected by crowdsourcing and unlabeled data simultaneously. [20] focused on the transductive and non-deep scenario, and thus cannot be applied to our setting. The method proposed by [39] utilized the graph neural network (GNN) to deal with SSMLL data with graph structures. In contrast, there are many works that trained linear models to solve the SSMLL problems [5, 15, 42, 52, 50, 40]. M3DN was proposed to deal with SSMLL data in multi-modal multi-instance learning scenarios by adopting optimal transport technique [48].

Pseudo-labeling has become a popular method in semi-supervised learning (SSL). The idea was firstly applied to semi-supervised training of deep neural networks [23]. Subsequently, a great deal of works have been devoted to improving the quality of pseudo-labels either by adopting consistency regularization [4, 38], or by using distribution alignment [30, 3]. The contrastive learning technique has been applied to improve the performance of SSL [24]. To improve the reliability of pseudolabels, an uncertainty-aware pseudo-labeling method proposed in [35] selected reliable pseudo-labels based on the prediction uncertainty. Recent studies have also paid attention to dealing with the class-imbalance problem of pseudo-labeling in SSL scenarios [18, 45, 14]. Unlike FixMatch that

selects unlabeled examples with a fixed threshold, Dash [47] selected unlabeled examples with a dynamic threshold, with the goal of achieving better pseudo-labeling performance. FlexMatch [51] was proposed to select unlabeled examples for every class according to the current learning status of the model. Several works have been explored the idea of selecting different thresholds for different classes to improve the performance of SLL [13, 14, 44].However, these methods are designed for the multi-class single-label scenario, and cannot be directly applied to the multi-label scenario.

In order to reduce the annotation cost, a cost-effective strategy is to assign a subset of true labels to each instance. For example, [11] designed a partial binary cross entropy (BCE) loss that re-weights the losses of known labels. As an extreme case of MLML, single positive multi-label learning (SPML) [8, 53, 41] assumes that only one of multiple true labels can be observed during the training stage. The pioneering work [8] trains DNNs by simply treating unobserved labels as negative ones and utilizes the regularization to alleviate the harmfulness of false negative labels. [53] propose asymmetric pseudo labeling technique to recover true labels.

3 The Method

In the SSMLL problem, let x ∈ X be a feature vector and y ∈ Y be its corresponding label vector, where X = R d is the feature space and Y = { 0 , 1 } q is the label space with q possible class labels. Here, y k = 1 indicates the k -th label is relevant to the instance, while y k = 0 , otherwise. Suppose that we are given a labeled dataset with n training examples D l = { ( x i , y i ) } n i =1 and an unlabeled dataset with m training instances D u = { x j } j m =1 . Our goal is to train a DNN f ( x ; θ ) based on the labeled dataset D l and unlabeled dataset D u , where θ is the parameter of the network. For notational simplicity, we omit the notation θ and let f ( x ) be the predicted probability distribution over classes and f k ( x ) be the predicted probability of the k -th class for input x .

Typical multi-label learning methods usually train a DNN with the commonly used binary cross entropy (BCE) loss, which decomposes the original task into multiple binary classification problems. Unfortunately, BCE loss often suffers from positive-negative imbalance issue. To mitigate this problem, we adopt the asymmetric loss (ASL) [34], which is a variant of focal loss with different focusing parameters for positive and negative instances. In our experiment, we found it works better than BCE loss. Formally, given the predicted probabilities f ( x ) on instance x , the ASL loss is defined as

Here, ℓ 1 ( f k ) = -(1 -f k ) λ 1 log( f k ) and ℓ 0 ( f k ) = -( f k ) λ 0 log(1 -f k ) represent the losses calculated on positive and negative labels, where λ 1 and λ 0 are positive and negative focusing parameters.

3.1 Instance-Aware Pseudo-Labeling

The loss function may not be the best choice to solve the SSMLL problem, since besides the labeled training examples, there still exist a large number of unlabeled training examples. To exploit the information of unlabeled data, inspired by recent SSL works [4, 38], an intuitive strategy is assigning the unlabeled instances with pseudo-labels based on the model outputs. Formally, we define the unlabeled loss L u as

where ˆ y = [ˆ y 1 , · · · , ˆ y j ] ⊤ represents the pseudo-label vector for instance x .

In the above formulation, the most significant element is how to obtain the pseudo-labels ˆ y that significantly affects the final performance of SSMLL. Most of existing pseudo-labeling methods are performed in an instance-aware manner by assigning pseudo-labels to each unlabeled instance based on its probability distribution. Below, we briefly review three instance-aware pseudo-labeling strategies that can be applied to the SSMLL problems. The most commonly used strategy adopted by the SSL method called FixMatch [38] is to select the most probable label as the ground-truth one:

One advantage of the strategy is that it is likely to safely identify a true label for each unlabeled training instance. Unfortunately, it is obvious that the strategy would neglect multiple true labels. Generally, it transforms the unlabeled dataset into another learning scenario called single positive multi-label learning (SPML) [8], where only one of multiple positive labels is available for each instance. A straightforward strategy is to simply treat unobserved labels as negative ones. Although this strategy enables us to train a classifier based on SPML data, it would introduce a large number of false negative labels, leading to unfavorable performance.

The second choice is an improved version of the above strategy, which selects the top l probable labels as the true ones:

where τ l is the l -th predicted probability in a descending order. The strategy conducts a competition among labels, and selects top l winners. The optimal solution is to set l as the true number of positive labels for each unlabeled instance. Unfortunately, since the true number is unknown in practice, as a compromise, we set l as the average number of positive labels per instance. Given that the true number does not always equal to the average number, it would be caught in a dilemma of either introducing false positive labels or neglecting true positive ones.

The last choice is to adopt an instance-aware threshold τ j that separates positive and negative labels for each unlabeled instance.

Compared to the above methods, this strategy achieves a strong flexibility that allows it to assign different numbers of positive labels to different instances. A potential limitation is that it is hard to find the optimal thresholds for different instances. In practice, a feasible solution is to adopt a global threshold, that is ∀ j ∈ [ m ] , τ j = τ . Obviously, it is impossible to adopt a global threshold that is optimal for all instances, especially considering the class-imbalance property of MLL data. In general, a large threshold often leads to a small recall score of tail classes, which indicates that less positive labels would be identified. While a small threshold often results in a small precision score of head classes, which indicates besides positive labels, a great deal of negative ones would be treated as positive ones. The dilemma prevents the model from obtaining favorable performance.

3.2 Class-Aware Pseudo-Labeling

As discussed above, in many real-world scenarios, it is really difficult to acquire the true number of positive labels for each instance. This leads the instance-aware pseudo-labeling methods to be caught in the dilemma of either mislabeling false positive labels or neglecting true positive labels, resulting in a noticeable decrease of the model performance.

To solve this issue, we propose a regularized learning framework to assign pseudo-labels in a class-aware manner. Formally, we reformulate the optimization problem of SSMLL as

where α k and β k are class-aware regularized parameters to control how many positive and negative labels would be included into model training for class k . Below, we primarily provide a solution of the optimization problem Eq.(5), and then discuss how to set parameters α k and β k to capture the true class distribution of unlabeled examples.

Alternative Search It is hard to directly solve the optimization problem Eq.(5), since there are two sets of variables. A feasible solution is to adopt the alternative convex search [1, 54] strategy that optimizes a group of variables by fixing the other group of variables.

Confidence

Confidence

Top-1

Top-

k

Instance-aware thresholding

Class-distribution-aware

thresholding

(a) Different Pseudo-Labling Strategies

COCO

VOC

(b) Class Proportions

0.6

0.5

Estimated

Proportion

0.4

True

0.3

0.2

0.1

0.0

0

10

20

304050

607080

Class Index

0.4

Proportion

0.3

0.2

0.1

0.0

0

2

4

6

81012141618

Class Index

丁

丁

Suppose that pseudo labels ˆ y are given, then the optimization problem Eq.(5) can be transformed into an ordinary loss by treating the pseudo labels as the true ones:

which can be solved by applying the stochastic gradient decent (SGD) method.

With the parameters θ fixed, we reformulate the optimization problem with respect to ˆ y as

Consider that ˆ y k is assume to be one or zero, we can obtain the following solution:

where τ ( α k ) = exp( -α k ) and τ ( β k ) = 1 -exp( -β k ) are two class-aware thresholds, and ˆ y k = -1 means that the label ˆ y k would not be used for model training.

3.3 Class-Distribution-Aware Thresholding

An important problem is how to set the thresholds τ ( α k ) and τ ( β k ) , which determine the numbers of positive and negative pseudo-labels for every class k . In order to capture the true class distribution, we propose the Class-distribution-Aware Thresholding (CAT) strategy to determine τ ( α k ) and τ ( β k ) . Suppose that we are given y j , ∀ j ∈ [ m ] , i.e. , the true label vectors of unlabeled training instances. By solving the following equation, we can obtain τ ( α k ) and τ ( β k ) that capture the true class distribution of unlabeled data.

where γ ∗ k = ∑ j m =1 I ( y jk =1) m and ρ ∗ k = ∑ j m =1 I ( y jk =0) m are respectively the proportions of positive and negative labels in unlabeled data for class k . Although during the training process, the true labels of unlabeled instances are inaccessible, our observation shows that the estimated class distribution, i.e.,

the class proportions of positive and negative labels in labeled examples, can tightly approximate the true class distribution. As shown Figure 1 (b), we illustrate the proportions of positive labels in labeled examples and unlabeled examples for every class k on two benchmark datasets COCO and VOC. The proportions of labeled examples are respectively p = 0 . 05 and p = 0 . 1 for COCO and VOC. From the figures, it can be observed that even with a small proportion of labeled examples ( p = 0 . 05 ), it achieves a nearly complete overlap between the estimated and true curves, which validates that the estimated class distribution can be a tight approximation of the true one. This motivate us to alternatively utilize the estimated class distribution to solve the solutions for τ ( α k ) and τ ( β k ) :

where ˆ γ k = ∑ n i =1 I ( y ik =1) n and ˆ ρ k = ∑ n i =1 I ( y ik =0) n are respectively the proportions of positive and negative labels in labeled data for class k . Figure 1 provides an illustration of the comparison between three instance-aware pseudo-labeling methods and the CAP method. By utilizing CAT strategy, CAP is expected to assign pseudo-labels with the class distribution that approximates the true one.

In practice, to further improve the performance of CAP, one feasible solution is to discard a fraction of unreliable pseudo-labels with relatively low confidences, which may have a negative impact on the model training. Specifically, for any class k ∈ [ q ] , we select top η 1 · ˆ γ k and η 0 · ˆ ρ k proportion probable pseudo-labels, where η 1 , η 0 ∈ [0 , 1] are two parameters to control the reliable intervals of pseudo-labels. By substituting the two terms into the right sides of Eq.(9), we can obtain the thresholds correspondingly. In Section 5.4, we perform ablation experiments to study the influence of reliable intervals on the model performance.

4 Theoretical Analysis

In this section, we perform theoretical analyses for the proposed method. In general, the performance of pseudo-labeling depends mainly on two factors, i.e., the quality of the model predictions and the correctness of estimated class distribution. Our work focuses on the latter. Consider an extreme case, where the model predictions are perfect, i.e., the confidences of positive labels are always greater than that of negative labels. In such a case, we still need an appropriate threshold to precisely separate the positive and negative labels. This implies that we need to capture the true class distribution of unlabeled data in order to achieve desirable pseudo-labeling performance.

4.1 Correctness of the Estimated Class Distribution

To study the correctness of estimated class distribution, we provide the following theorem, which gives an upper bound on the difference between the estimated class proportion ˆ γ k and the true class proportion γ ∗ k (its proof is given in Appendix A). A similar result can be derived for ˆ ρ k .

Theorem 1. Assume the estimated class proportion ˆ γ k = 1 n ∑ n i =1 I ( y ik = 1) , and the true class proportion γ ∗ k = 1 m ∑ m j =1 I ( y jk = 1) for any k ∈ [ q ] , where n and m are the numbers of labeled and unlabeled examples that satisfy m>>n . Then, with the probability larger than 1 -2 n -1 -2 m -1 , we have, ∀ k ∈ [ q ] , | ˆ γ k -γ ∗ k | ≤ √ log n √ 2 n + √ log m √ 2 m .

Theorem 1 tells us that the correctness of the estimated class distribution mainly depends on the number of labeled and unlabeled data. In general, the bound is dominated by the first term, since it always satisfies m>>n . By neglecting the second term, we can see that ∀ k ∈ [ q ] , γ ∗ k → ˆ γ k in the parametric rate O p (1 / √ n ) , where O p denotes the order in probability. Obviously, as the number of training examples increase, the estimated class distribution would quickly converge to the true one.

4.2 Generalization Bound

Moreover, we study the generalization performance of CAP. Before providing the main results, we first define the true risk with respect to the classification model f ( x ; θ ) :

Our goal is to learn a good classification model by minimizing the empirical risk ̂ R ( f ) = ̂ R l ( f ) + ̂ R u ( f ) , where ̂ R l ( f ) and ̂ R u ( f ) are respectively the empirical risk of the labeled loss L l ( f ( x ) , y ) and unlabeled loss L u ( f ( x ) , y ) :

Note that during the training, we cannot train a model directly by optimizing ̂ R u ( f ) , since the labels of unlabeled data are inaccessible. Instead, we train the model with ̂ R ′ u ( f ) = 1 m ∑ m j =1 L u ( f ( x j ) , ˆ y j ) , where ˆ y j represents the pseudo-label vector of the instance x j .

Let ℓ ( f k ( x )) = y k ℓ 1 ( f k ( x )) + (1 -y k ) ℓ 0 ( f k ( x )) be the loss for the class k , and L ℓ be any (not necessarily the best) Lipschitz constant of ℓ . Let R N ( F ) be the expected Rademacher complexity [31] of F with N = m + n training points. Let ˆ f = arg min f ∈F ̂ R ( f ) be the empirical risk minimizer, where F is a function class, and f ⋆ = arg min f ∈F R ( f ) be the true minimizer. We derive the following theorem, which provides a generalization error bound for the proposed method (its proof is given in Appendix B).

Theorem 2. Suppose that ℓ ( · ) is bounded by B . For some ϵ > 0 , if ∑ m j =1 | I ( f k ( x j ) ≥ τ ( α k )) -I ( y jk = 1) | /m ≤ ϵ for any k ∈ [ q ] , for any δ > 0 , with probability at least 1 -δ , we have

From Theorem 4, it can be observed that the generalization performance of ˆ f mainly depends on two factors, i.e. , the pseudo-labeling error ϵ and the number of training examples N . Apparently, a smaller pseudo-labeling error ϵ often leads to better generalization performance. Thanks to its ability to capture the true class distribution, CAP can achieve a much smaller pseudo-labeling error ϵ than existing instance-aware pseudo-labeling methods, which is beneficial for obtaining better classification performance. This can be further validated by our empirical results in Section 5.3. The second factor is the number of training examples. As N →∞ and ϵ → 0 , Theorem 4 shows that the empirical risk minimizer ˆ f converges to the true risk minimizer f ⋆ .

5 Experiments

In this section, we first perform experiments to validate the effectiveness of the proposed method; then, we perform ablation studies to analyze the mechanism behind CAP.

5.1 Experimental Settings

Datasets To evaluate the performance of the propose method, we conduct experiments on three benchmark image datasets, including Pascal VOC-2012 (VOC for short) 2 [12], MS-COCO-2014 (MS-COCO for short) 3 [25], and NUS-WIDE (NUS for short) 4 [7]. The detailed information of these datasets can be found in the appendix. For each dataset, we randomly sample a proportion p ∈ { 0 . 05 , 0 . 1 , 0 . 15 , 0 . 2 } of examples with full labels while the others without any supervised information. Following the previous works [8, 53], we report the mean average precision (mAP) on the test set for each method.

Comparing methods To validate the effectiveness of the proposed method, we compare it with five groups of methods: 1) three instance-aware pseudo-labeling methods: Top-1 (Eq.(2)), Topk (Eq.(3)), IAT (Eq.(4)); 2) two state-of-the-art MLML methods: LL [19] (includes three variants LL-R , LL-Ct , and LL-Cp ), PLC [46]; 3) Two state-of-the-art SSL methods: Adsh [14], FreeMatch [44]; 4) One state-of-the-art SSMLL method: DRML [43]; 5) Two baseline methods, BCE , ASL [34]. DRML is the only deep SSMLL method whose source code could be found on the Internet.

Furthermore, most MLML methods cannot be applied to the SSMLL scenario, since they assume that a subset of labels have been annotated for each training instance. The detailed information of these methods can be found in the appendix.

Implementation We employ ResNet-50 [16] pre-trained on ImageNet [36] for training the classification model. We adopt RandAugment [9] and Cutout [10] for data augmentation. We employ AdamW [29] optimizer and one-cycle policy scheduler [10] to train the model with maximal learning rate of 0.0001. The number of warm-up epochs is set as 12 for all datasets. The batch size is set as 32, 64, and 64 for VOC, MS-COCO, and NUS. Furthermore, we perform exponential moving average (EMA) for the model parameter θ with a decay of 0.9997. For all methods, we use the ASL loss as the base loss function, since it shows superiority to BCE loss [34]. We perform all experiments on GeForce RTX 3090 GPUs. The random seed is set to 1 for all experiments.

5.2 Comparison Results

Table 1 and Table 2 report the comparison results between CAP and the comparing methods in terms of mAP on VOC, COCO, and NUS. From the tables, we can see that: 1) DRML obtains unfavorable performance, even worse than baselines BCE and ASL, since it performs two-stage training, which may destroy its representation learning. The original paper did not report the results on these three datasets. Therefore, it is rather important to design an effective SSMLL method in deep learning paradigm. 2) CAP outperforms three instance-aware pseudo-labeling methods, which demonstrates that by utilizing CAT strategy, CAP can precisely estimate the class distribution of unlabeled data and thus obtain desirable pseudo-labeling performance. 3) The performance of CAP is better than that of two state-of-the-art SSL methods. To achieve better performance, we have made several modifications for these two methods not limited to the following: a) use the ASL loss ; b) adopt stronger data augmentations; c) change the training scheme to make them more suitable for the multi-label scenario. 4) CAP achieves the best performance in all cases and significantly outperforms







k













  



























  

































  





























  































  

















(a)

p

= 0

.

05











  



















(b)

p

= 0

.

1











  





















(c)

p

= 0

.

15











  





















(d)

p

= 0

.

2











1















p

=0.05

p

=0.1

p

=0.15

p

=0.2

(a) VOC,

η

0

= 1

     

0















(b) VOC,

η

1

= 1











1















(c) COCO,

η

0

= 1

     

0

















(d) COCO,

η

1

= 1

the comparing methods, especially when the number of labeled examples is small. These results convincingly validate the effectiveness of the proposed method.

5.3 Study on the Performance of Pseudo-Labeling

In this section, we explain why CAP is better than the conventional instance-aware pseudo-labeling methods. Figure 2 illustrates the performance of different pseudo-labeling methods in terms of CF1 score on VOC and COCO. From the figures, we can see that CAP achieves the best performance in all cases. As discussed above, the pseudo-labeling performance mainly depends on two factors, i.e. , the quality of model predictions and the correctness of estimated class proportions. CAP improves the pseudo-labeling performance by precisely estimating the class distribution. An interesting observation is that at the first epoch, when the model predictions are the same for four methods, our method significantly outperforms the comparing methods, since it is able to capture the true class proportions. These results validate that CAP can achieve better pseudo-labeling performance.

5.4 Study on the Influence of Reliable Interval

As mentioned above, to improve the performance, instead of using all pseudo-labels, we can train the model with only reliable pseudo-labels within the reliable intervals that are controlled by η 1 , η 0 . Figure 3 illustrates the performance of CAP as η 1 and η 0 change in the ranges of [0 . 8 , 0 . 85 , 0 . 9 , 0 . 95 , 1] and [0 . 95 , 0 . 96 , 0 . 97 , 0 . 98 , 0 . 99 , 1] . From the figures, it can be observed that discarding the unreliable positive pseudo-labels would improve the performance, but discarding the unreliable negative pseudo-labels would degrade the performance. One possible reason behinds the phenomenon is due to the significant positive-negative imbalance in MLL data, i.e. , the number of negative labels is often much greater than that of positive labels. This leads the model to be sensitive to false positive labels, while be robust to false negative labels. In our main experiments (Table 1 and Table 2), we set

η 1 = 1 , η 0 = 1 . In practice, we are expected to achieve better performance by tuning the parameter η 1 .

6 Conclusion

The paper studies the problem of semi-supervised multi-label learning, which aims to train a multilabel classifier by leveraging the information of unlabeled data. Different from the conventional instance-aware pseudo-labeling methods, we propose to assign pseudo-labels to unlabeled instances in a class-aware manner, with the aim of capturing the true class distribution of unlabeled data. Towards this goal, we propose the CAT strategy to obtain an estimated class distribution, which has been proven to be a desirable estimation of the true class distribution based on our observations. Theoretically, we first perform an analysis on the correctness of estimated class distribution; then, we provide the generalization error bound for CAP and show its dependence to the pseudo-labeling performance. Extensive experimental results on multiple benchmark datasets validate that CAP can achieve state-of-the-art performance.

In general, the performance of pseudo-labeling depends mainly on two factors, i.e., the quality of the model predictions and the correctness of the estimated class distribution. This work focuses on the latter. In future, we plan to boost the performance of SSMLL by improving the quality of model predictions.

Acknowledgments and Disclosure of Funding

Sheng-Jun Huang was supported by Natural Science Foundation of Jiangsu Province of China (BK20222012, BK20211517), the National Key R&D Program of China (2020AAA0107000), and NSFC (62222605). Masashi Sugiyama was supported by JST CREST Grant Number JPMJCR18A2.

References

Mokhtar S Bazaraa, Hanif D Sherali, and Chitharanjan M Shetty. Nonlinear programming: theory and algorithms . John Wiley & Sons, 2013.

Emanuel Ben-Baruch, Tal Ridnik, Itamar Friedman, Avi Ben-Cohen, Nadav Zamir, Asaf Noy, and Lihi Zelnik-Manor. Multi-label classification with partial annotations using class-aware selective loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4764-4772, 2022.

David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785 , 2019.

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems , 32, 2019.

Gang Chen, Yangqiu Song, Fei Wang, and Changshui Zhang. Semi-supervised multi-label learning by solving a sylvester equation. In Proceedings of the 2008 SIAM international conference on data mining , pages 410-419. SIAM, 2008.

Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5177-5186, 2019.

Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nuswide: a real-world web image database from national university of singapore. In Proceedings of the ACM international conference on image and video retrieval , pages 1-9, 2009.

Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic. Multi-label learning from single positive labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 933-942, 2021.

Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems , 2020.

Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Thibaut Durand, Nazanin Mehrasa, and Greg Mori. Learning a deep convnet for multi-label classification with partial labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 647-657, 2019.

Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision , 111(1):98-136, 2015.

Vasilii Feofanov, Emilie Devijver, and Massih-Reza Amini. Transductive bounds for the multiclass majority vote classifier. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 3566-3573, 2019.

Lan-Zhe Guo and Yu-Feng Li. Class-imbalanced semi-supervised learning with adaptive thresholding. In International Conference on Machine Learning , pages 8082-8094, 2022.

Yuhong Guo and Dale Schuurmans. Semi-supervised multi-label classification: a simultaneous large-margin, subspace learning approach. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23 , pages 355-370. Springer, 2012.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Dat Huynh and Ehsan Elhamifar. Interactive multi-label cnn learning with partial labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9423-9432, 2020.

Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. Advances in neural information processing systems , 33:14567-14579, 2020.

Youngwook Kim, Jae Myung Kim, Zeynep Akata, and Jungwoo Lee. Large loss matters in weakly supervised multi-label classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14156-14165, 2022.

Xiangnan Kong, Michael K Ng, and Zhi-Hua Zhou. Transductive multilabel learning via label set propagation. IEEE Transactions on Knowledge and Data Engineering , 25(3):704-719, 2011.

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision , 128(7):1956-1981, 2020.

Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi. General multi-label image classification with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16478-16488, 2021.

Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , volume 3, page 896, 2013.

Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: Semi-supervised learning with contrastive graph regularization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9475-9484, 2021.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755, 2014.

Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu. Query2label: A simple transformer way to multi-label classification. arXiv preprint arXiv:2107.10834 , 2021.

Weiwei Liu, Haobo Wang, Xiaobo Shen, and Ivor W Tsang. The emerging trends of multi-label learning. IEEE transactions on pattern analysis and machine intelligence , 44(11):7955-7974, 2021.

Ziyi Liu, Zengmao Wang, and Bo Du. Multi-marginal contrastive learning for multi-label subcellular protein localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 20626-20635, June 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations , 2019.

Gideon S Mann and Andrew McCallum. Simple, robust, scalable semi-supervised learning via expectation regularization. In Proceedings of the 24th international conference on Machine learning , pages 593-600, 2007.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13018-13028, 2021.

Kirill Prokofiev and Vladislav Sovrasov. Combining Metric Learning and Attention Heads For Accurate and Efficient Multilabel Image Classification. arXiv e-prints , 2022.

Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi-label classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 82-91, 2021.

Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations , 2020.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015.

Wanli Shi, Victor S Sheng, Xiang Li, and Bin Gu. Semi-supervised multi-label learning from crowds via deep sequential generative model. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1141-1149, 2020.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

Zixing Song, Ziqiao Meng, Yifei Zhang, and Irwin King. Semi-supervised multi-label learning for graph-structured data. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 1723-1733, 2021.

Qiaoyu Tan, Yanming Yu, Guoxian Yu, and Jun Wang. Semi-supervised multi-label classification using incomplete label information. Neurocomputing , 260:192-202, 2017.

Thomas Verelst, Paul K Rubenstein, Marcin Eichner, Tinne Tuytelaars, and Maxim Berman. Spatial consistency loss for training multi-label classifiers from single-label annotations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3879-3889, 2023.

Bo Wang, Zhuowen Tu, and John K Tsotsos. Dynamic label propagation for semi-supervised multi-class multi-label classification. In Proceedings of the IEEE international conference on computer vision , pages 425-432, 2013.

Lichen Wang, Yunyu Liu, Can Qin, Gan Sun, and Yun Fu. Dual relation semi-supervised multilabel learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 6227-6234, 2020.

Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. Freematch: Selfadaptive thresholding for semi-supervised learning. In The Eleventh International Conference on Learning Representations , 2023.

Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A classrebalancing self-training framework for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10857-10866, 2021.

Ming-Kun Xie, Jia-Hao Xiao, and Sheng-Jun Huang. Label-aware global consistency for multi-label learning with single positive labels. In Advances in Neural Information Processing Systems , 2022.

Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference on Machine Learning , pages 11525-11536. PMLR, 2021.

Yang Yang, Zhao-Yang Fu, De-Chuan Zhan, Zhi-Bin Liu, and Yuan Jiang. Semi-supervised multi-modal multi-instance multi-label deep network with optimal transport. IEEE Transactions on Knowledge and Data Engineering , 33(2):696-709, 2019.

Yang Yang, Yi-Feng Wu, De-Chuan Zhan, Zhi-Bin Liu, and Yuan Jiang. Complex object classification: A multi-modal multi-instance multi-label deep network with optimal transport. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &Data Mining , pages 2594-2603, 2018.

Wang Zhan and Min-Ling Zhang. Inductive semi-supervised multi-label learning with cotraining. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1305-1314, 2017.

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems , 34:18408-18419, 2021.

Feipeng Zhao and Yuhong Guo. Semi-supervised multi-label learning with incomplete labels. In Twenty-fourth international joint conference on artificial intelligence , 2015.

Donghao Zhou, Pengfei Chen, Qiong Wang, Guangyong Chen, and Pheng-Ann Heng. Acknowledging the unknown for multi-label learning with single positive labels, 2022.

Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5982-5991, 2019.

A Proof of Theorem 1

Theorem 3. Assume the estimated class proportion ˆ γ k = 1 n ∑ n i =1 I ( y ik = 1) , and the true class proportion γ ∗ k = 1 m ∑ m j =1 I ( y jk = 1) for any k ∈ [ q ] , where n and m are the numbers of labeled and unlabeled examples that satisfy m>>n . Then, with the probability larger than 1 -2 n -1 -2 m -1 , we have, ∀ k ∈ [ q ] , | ˆ γ k -γ ∗ k | ≤ √ log n √ 2 n + √ log m √ 2 m .

Proof. The proof is mainly based on Hoeffding's inequality that can be defined as follows.

Lemma 1. (Hoeffding's inequality). Let z 1 , ..., z N be independent random variables bounded by [ a i , b i ] . Then ˆ z = 1 N ∑ N i =1 z i obeys for any ν > 0

Let ¯ γ k = p ( y k = 1) represents the expected class proportion. According to Hoeffding's inequality, for any k ∈ [ q ] , we have √

or equivalently, with the probability at least 1 -2 n -1 , we have | ˆ γ k -¯ γ k | ≤ √ log n/ √ 2 n . Similarly, with the probability at least 1 -2 m -1 , for any k ∈ [ q ] , we have | γ ∗ k -¯ γ k | ≤ √ log m/ √ 2 m . By applying the triangle inequality, with the probability at least 1 -2 n -1 -2 m -1 , for any k ∈ [ q ] , we have √ √

which completes the proof.

B Proof of Theorem 2

Theorem 4. Suppose that ℓ ( · ) is bounded by B . For some ϵ > 0 , if ∑ m j =1 | I ( f k ( x j ) ≥ τ ( α k )) -I ( y jk = 1) | /m ≤ ϵ for any k ∈ [ q ] , for any δ > 0 , with probability at least 1 -δ , we have

Proof. Before proving the theorem, we first provide two useful lemmas as follows.

Weprimarily derive the uniform deviation bound between R ( f ) and ̂ R ( f ) , which is a simple extension of the result in the binary setting [31].

Lemma 2. Suppose that the loss function ℓ is L ℓ -Lipschitz continuous w.r.t. θ . For any δ > 0 , with probability at least 1 -δ , we have

Proof. In order to prove this lemma, we define the Rademacher complexity of L and F with m + n training examples as follows:

Considering that L ( f ( x ) , y ) = ∑ q ℓ ( f k , y k ) , we have

where the second line is due to the Lipschitz continuity of the loss function ℓ .

Then, we proceed the proof by showing that the one direction sup f ∈F R ( f ) -̂ R ( f ) is bounded with probability at least 1 -δ/ 2 , and the other direction can be proved similarly. Note that replacing an example ( x j , y j ) with another ( x ′ j , y ′ j ) leads to a change of sup f ∈F R ( f ) -̂ R ( f ) at most qB n + m due to the fact that ℓ is bounded by B . According to McDiarmid's inequality [31], for any δ > 0 , with probability at least 1 -δ/ 2 , we have

According to the result in [31] (Theorem 3.3) that shows E [sup f ∈F R ( f ) -̂ R ( f )] ≤ 2 R m ( F ) , by further considering the other direction sup f ∈F ̂ R ( f ) -R ( f ) , with probability at least 1 -δ , we have

which completes the proof.

Then, we can bound the difference between ̂ R ( f ) and ̂ R ′ ( f ) as follows

Lemma 3. Suppose that ℓ ( · ) is bounded by B . For some ϵ > 0 , if ∑ m j =1 | I ( f k ( x j ) ≥ τ ( α k )) -I ( y jk = 1) | /m ≤ ϵ for any k ∈ [ q ] , for any f ∈ F , we have:

Proof. Without loss of generality, assume that ϵ is the largest pseudo-labeling error among q classes, i.e. , ϵ = max k ∈ [ q ] ∑ m j =1 | I ( f k ( x j ) ≥ τ ( α k )) -I ( y jk = 1) | /m . Obviously, ϵ consists of exactly two types of pseudo-labeling error:

where ϵ 1 calculates the proportion of positive labels being treated as negative ones, and ϵ 0 calculates the proportion of negative labels being treated as positive ones. Then, we prove the following two sides, which provide the bounds for ̂ R ′ u ( f ) . Firstly, we prove its upper bound:

where the second line holds based on Eq.(14). Then, we prove its low bound:

By combining these two sides, we can obtain the following result:

which concludes the proof.

For any δ > 0 , with probability at least 1 -δ , we have:

where the first and fifth lines are based on Eq.(1), and second and fourth lines are due to Lemma 2. The third line is by the definition of ˆ f .

C Details of Experimental Settings

Table 3 reports the detailed characteristics of four benchmark datasets. VOC is a popular multi-label dataset that has been divided into a trainval set containing 5,011 examples and a test set containing 4,952 examples from 20 object categories. MS-COCO is another widely used multi-label dataset, which consists of 82,081 training examples and 40,504 validation examples belonging to 80 different categories. In our experiments, the validation set is used for testing. NUS is incomplete online due to many invalid URLs. Our collection consists of 126,034 training images and 84,226 testing images from 81 classes.

Comparing methods To validate the effectiveness of the proposed method, we compare it with four groups of methods: 1) Three instance-aware pseudo labeling methods: Top-1 , which is similar to

FixMatch [38] that selects the most probable label as the ground-truth one; Topk , which selects the top k rank labels according to the predicted probabilities as the true labels; IAT , which utilizes the instance-aware thresholding strategy to separates positive and negative labels. 2) Two state-of-the-art MLML methods: LL [19] (includes three variants LL-R , LL-Ct , and LL-Cp ), which rejects or corrects the large-loss examples to prevent model from memorizing noisy labels; PLC [46], which performs the pseudo-labeling consistency regularization for recovering the labeling information of potential labels. 3) Two state-of-the-art SSL methods: Adsh [14], which solves the class-imbalance issue in SSL by performing adaptive thresholding; FreeMatch [44], which improves the performance of FixMatch by introducing a self-adaptive class fairness regularization. 4) One state-of-the-art SSMLL method: DRML [43], which performs pseudo labeling by exploring the feature distribution and the label correlation simultaneously. 5) Two baseline methods, BCE , which is the most commonly used loss function for multi-label classification; ASL [34], which improves BCE by utilizing the dynamic down-weighting and hard-thresholding techniques. DRML is the only deep SSMLL method whose source code could be found on the Internet. Furthermore, most MLML methods cannot be applied to the SSMLL scenario, since they assume that a subset of labels have been annotated for each training instance.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a class-distribution-aware method to deal with the semi-supervised multi-label learning (SSMLL) problem. The main motivation is that the conventional pseudo-labeling methods cannot be applied to SSMLL scenarios, since instance is assigned with more than one ground-truth labels. To solve this challenge, the paper proposes a class-aware pseudo-labeling strategy, which is free of estimating the number of true labels for each instance. To capture the class distribution of unlabeled data, authors propose a regularized learning framework to determine the pseudo-labels by exploiting the estimated class distribution of labeled data.\n\nSTRENGTHS:\nThe motivation of this paper is clear. Although pseudo-labeling is a commonly used strategy in single-label case, it cannot be applied to multi-label case due to the unknown label count.\n\nThe proposed method is intuitive and effective. Authors propose the class-aware pseudo-labeling strategy to cleverly avoid the estimation of the label count for each instance. Consequently, the problem has been transformed to the task of estimating the class distribution of unlabeled data, which can be easily solved by the proposed class-distribution-aware thresholding method.\n\nTheoretical and empirical studies comprehensively verify the effectiveness of the proposed method. Specifically, the paper studies the correctness of the estimated class distributions theoretically and conduct comparisons with sota methods.\n\nWEAKNESSES:\nIt seems that only one threshold can determine the positive and negative pseudo-labels for each class. It is suggested to explain the reasons why two thresholds are used in the paper.\n\nIt seems that the paper did not discuss how to set the parameter of ASL.\n\nThere are some language mistakes. The paper is suggested to proofread the paper carefully.\n\nQUESTIONS:\nThe paper considers the estimation of class distribution of unlabeled data. Is there any other factor can affect the pseudo-labeling performance?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

comments: SUMMARY:
This work studies pseudo-labeling for multi-label semi-supervised learning. Differing from the traditional instance-aware pseudo-labeling methods, they propose to assign pseudo-labels to unlabeled data in a class-aware manner to capture the true class distribution of the unlabeled data. This work proposes CAT strategy to estimate the class distribution. It proves it is a desirable estimation by performing an analysis of the correctness of the estimation and providing the generalization error bound for CAP.

STRENGTHS:
This paper is well-structured and easy to follow. The methods are clearly presented in Section 3. They also provide a theoretical analysis in Section 4 of the proposed method. Solid experiments are conducted to show the effectiveness of the proposed methods. Multi-label classification is an important task, and pseudo-labeling for this scenario will be helpful to this community.

WEAKNESSES:
* The key idea relies on the observation that the class proportions of positive and negative labels in labeled examples can tightly approximate the true class distribution. I am concerned if the unlabeled data have totally different distribution, will the assumption relying on such observation still hold true? Therefore, it would be helpful to show some examples when there is a distribution mismatch.  

* The technical novelty seems not that clear. It would be great if the authors can re-state and present the novelty of the propose approach.

QUESTIONS:
1. Adsh and FreeMatch are designed for traditional SSL, how do the authors implement these two algorithms for the SSMLL setting? 
2. This method can be used widely. I would re-consider rating if the authors are willing to make the codebase public.

LIMITATIONS:
They do not claim/discuss any limitation in their main manuscript.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Class-Aware Pseudo-labeling (CAP) method to solve semi-supervised multi-label learning (SSMLL) problem by controlling the assignment of positive and negative pseudo-labels for each class through a class-distribution-aware thresholding (CAT) strategy.\n\nSTRENGTHS:\n1. This paper proposes a pseudo-labeling strategy for semi-supervised multi-label learning in a class-aware manner.\n\n2. This paper is easy to read and well organized.\n\n3. The experimental results and the proof of Theorems in this paper seem to be solid.\n\nWEAKNESSES:\n1. Lack of some references. This paper is devoted to solving the SSMLL problem. Therefore, the authors should review and compare some SSMLL methods and more SSL methods. For example, \n[1] Rizve et al., In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning, ICLR, 2021. \n[2] Xu et al., Dash: Semi-supervised learning with dynamic thresholding, ICML, 2021.\n[3] Zhang et al., Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling, NeurIPS, 2021. \n\n2. What is the motivation for designing the operation $\\tau(\\alpha_{k})=exp(-\\alpha_{k})$? This does not seem to be mentioned in this paper.\n\n3. It is not enough clear about the Class-Distribution-Aware Thresholding. Also, it is suggested that the author gives a complete methodological illustration about CAP and CAT for each class to facilitate the readers' understanding. \n\n4. The random seed in this paper is set to 1 for all experiments. Following the SSL setting, it is necessary to report the mean value with different random seeds as the final performance.\n\n5. The authors should check if the title of the figure corresponds to the content. For example, Figure 2 is titled OF1 score while the results are CF1 scores. In addition, it is suggested to give a number for each formula.\n\nQUESTIONS:\nPlease refer to the Weaknesses.\n\nLIMITATIONS:\nThe limitations of this paper should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis papers introduces a novel method called Class-Aware Pseudo-Labeling (CAP) to address the challenges of semi-supervised multi-label learning (SSMLL). Traditional pseudo-labeling methods struggle with instances associated with multiple labels and an unknown label count, often leading to the introduction of false positive labels or the omission of true positive ones. The CAP method overcomes these issues by performing pseudo-labeling in a class-aware manner. It introduces a regularized learning framework that incorporates class-aware thresholds, effectively controlling the assignment of positive and negative pseudo-labels for each class. The paper highlights that even with a small proportion of labeled examples, the estimated class distribution can serve as a reliable approximation. This observation led to the development of a class-distribution-aware thresholding strategy to align the pseudo-label distribution with the true distribution. The paper provides theoretical verification of the correctness of the estimated class distribution and offers a generalization error bound for the proposed method. Extensive experiments on multiple benchmark datasets confirm the effectiveness of the CAP method in addressing the challenges of SSMLL problems.\n\nSTRENGTHS:\n1. The Class-Aware Pseudo-Labeling (CAP) method presents a significant innovation in the field of semi-supervised multi-label learning. Its unique approach of performing pseudo-labeling in a class-aware manner addresses the limitations of traditional pseudo-labeling methods, which often struggle with instances associated with multiple labels and an unknown label count.\n2. The authors provide a theoretical verification of the correctness of the estimated class distribution, which is a crucial aspect of the Class-Aware Pseudo-Labeling (CAP) method. This theoretical grounding not only validates the method's approach but also enhances its credibility and reliability. Furthermore, the authors offer a generalization error bound for the proposed method. This is an important contribution as it quantifies the expected performance of the CAP method and provides a measure of its robustness.\n3. The performance of the Class-Aware Pseudo-Labeling (CAP) method across all datasets and settings in the experiments is a strength of this paper. The fact that the proposed method consistently achieves optimal results demonstrates its effectiveness and robustness.\n\nWEAKNESSES:\nThe paper does not provide a clear explanation for the choice of using an exponential transformation in line 167 on the threshold value for each class in the Class-Aware Pseudo-Labeling (CAP) method.\n\nQUESTIONS:\nIn line 167 of the paper, the paper mention the use of an exponential transformation on the threshold value for each class in the Class-Aware Pseudo-Labeling (CAP) method. Why not just use a number as the threshold? Could the authors provide more details on the rationale behind this choice?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}